Infrastructure · Advanced · 30-minute read

Scaling a Git Backend for AI Development

AI development changes the shape of version-control traffic. The difficult question is not whether a Git-compatible service can store more data; it is whether the entire read, write, reference, indexing, and synchronization path remains predictable when many agents work at once.

Updated 3 November 2026 · Reproducible architecture experiment

Traditional developer workflows produce a relatively small number of human-paced repository operations. An automated coding system can produce many short-lived branches, repeated clones, small commits, frequent pushes, worktree refreshes, status checks, and cleanup operations. Agents also retry. They may inspect the same repository several times while deciding what to do, and an orchestrator may run many of them concurrently.

This article describes an experiment for finding the bottleneck before choosing a storage design. It does not report benchmark results, because no benchmark was run for this article. The commands and configuration below are a test plan and reference implementation. Replace the example values with measurements from your environment.

The workload is different before the backend is different

The first mistake in Git capacity planning is to estimate only repository size. For agentic development, the more useful unit is an operation mix:

• Discovery: reference advertisement, capability negotiation, shallow fetches, and status-like reads.
• Materialization: clone, fetch, checkout, sparse checkout, and worktree creation.
• Mutation: blob and tree uploads, commit creation, ref updates, and push-pack processing.
• Coordination: locks, leases, branch creation, merge checks, and garbage-collection bookkeeping.
• Retention: reachability scans, repacking, pruning, snapshots, and deletion of temporary branches.

These operations stress different resources. A fetch can be network-bound or pack-generation-bound. A push can be limited by CPU while objects are compressed, by metadata latency while objects are checked, or by serialized ref updates. A large clone can hide a high request rate behind a single long transfer. A thousand small agent operations can create the opposite profile: modest bytes but intense metadata and concurrency pressure.

Define the target workload explicitly. A useful model is:

operations_per_second =
  agents
  × operations_per_agent
  ÷ active_seconds

This is only a planning equation. It does not predict backend capacity. The measured result depends on operation size, repository topology, cache state, client behavior, retries, and the number of requests generated by each protocol operation.

Architecture under test

Test the system as four separable planes:

1. Git protocol plane: the service that accepts smart HTTP or SSH Git requests and manages repository semantics.
2. Object plane: durable storage for immutable Git objects, packfiles, indexes, and related repository data.
3. Metadata plane: mutable references, repository configuration, leases, locks, and indexes.
4. Control plane: authentication, scheduling, rate limits, observability, garbage collection, and lifecycle policy.

The experiment should compare at least two layouts:

Keep the Git protocol implementation constant while comparing layouts. Otherwise a result may reflect a server change rather than a storage change. Likewise, do not compare an empty repository with a mature monorepo and call the difference a storage effect.

Build a controlled repository corpus

Use synthetic repositories when customer data is unavailable. Synthetic data is acceptable for a controlled experiment, but it is not evidence about a particular production workload. Generate several shapes rather than one “average” repository:

• Small active repository: many small commits and branches, representative of agent task repositories.
• Large history repository: enough commits and pack data to exercise long fetches and repacking.
• Binary-heavy repository: larger blobs and fewer files, useful for transfer and object-store throughput.
• High-branch repository: many refs with short lifetimes, useful for metadata contention and cleanup.
• Shared-base repository: many agents starting from the same commit and producing nearby descendants.

Record the corpus manifest in version control. At minimum, include repository identifier, commit count, ref count, reachable object count, pack count, total loose-object bytes, packed bytes, largest blob size, and generation method. Do not include secrets or private source material.

# Example corpus manifest shape
repository: small-active
seed: 20261103-a
commits: <measure after generation>
refs: <measure after generation>
reachable_objects: <measure after generation>
packed_bytes: <measure after generation>
largest_blob_bytes: <measure after generation>

Use placeholders until the values are measured. Never copy example values into a report as if they were observations.

Define the protocol scenarios

Each scenario should have a fixed request sequence, a fixed repository selection rule, and a fixed concurrency level. A minimal suite might contain the following cases.

Scenario A: cold clone

Start a clean client directory for every run. Clone a repository over the production protocol. Measure time to first byte, time to complete transfer, total bytes received, client CPU, server CPU, object-store requests, and errors.

git -c protocol.version=2 \
    clone --no-checkout \
    "$GIT_URL/small-active.git" \
    "$RUN_DIR/client-$WORKER"

Run this with an empty cache and then with a warm cache. Report the cache state explicitly; warm and cold runs answer different questions.

Scenario B: repeated fetch

Clone once, then run repeated fetches while another process creates new commits. This exposes pack reuse, ref advertisement cost, cache invalidation, and the impact of a rapidly changing default branch.

for iteration in $(seq 1 "$FETCH_ITERATIONS"); do
  git -C "$CLIENT_DIR" fetch --prune origin
done

Scenario C: concurrent task branches

Start from a common base commit. Each worker creates a unique branch, changes a disjoint file set, commits, and pushes. Keep branch names unique so the scenario measures concurrent writes rather than accidental collisions.

branch="agent/$RUN_ID/$WORKER_ID"
git -C "$CLIENT_DIR" switch --create "$branch"
git -C "$CLIENT_DIR" add .
git -C "$CLIENT_DIR" -c user.name="experiment" \
  -c user.email="experiment@example.invalid" \
  commit -m "experiment: worker $WORKER_ID"
git -C "$CLIENT_DIR" push origin "HEAD:refs/heads/$branch"

The identity above is an example test identity, not a credential. Use a reserved non-routable email domain such as example.invalid for synthetic data.

Scenario D: same-reference contention

Have multiple workers attempt to update the same test reference. This measures conflict handling, lock duration, retry behavior, and failure clarity. It is intentionally different from the disjoint-branch scenario. A backend that performs well with independent refs may still serialize or reject same-ref updates.

Scenario E: cleanup and reachability

Create and delete temporary branches, then run the backend’s documented maintenance procedure. Measure foreground request latency while cleanup runs. Do not assume that deleting a ref immediately removes its objects; retention and garbage collection are implementation-specific and must be verified from the system’s documented behavior and observed metrics.

Drive concurrency without hiding failures

A load generator should preserve per-operation outcomes. Do not reduce a run to one aggregate success rate. Save one record per operation with scenario, worker, repository, operation, start time, end time, status, bytes, retry count, and error category.

A shell harness can be enough for a first pass:

#!/usr/bin/env bash
set -Eeuo pipefail

: "${GIT_URL:?set GIT_URL}"
: "${REPO:?set REPO}"
: "${RUN_ID:?set RUN_ID}"
: "${WORKERS:?set WORKERS}"
: "${RUN_DIR:?set RUN_DIR}"

mkdir -p "$RUN_DIR/logs"

run_worker() {
  worker_id="$1"
  log="$RUN_DIR/logs/worker-$worker_id.log"
  client="$RUN_DIR/client-$worker_id"
  start_ns=$(date +%s%N)

  if git -c protocol.version=2 clone --no-checkout \
      "$GIT_URL/$REPO.git" "$client" >"$log" 2>&1; then
    status=ok
  else
    status=failed
  fi

  end_ns=$(date +%s%N)
  printf '%s,%s,%s,%s,%s\n' \
    "$RUN_ID" "$worker_id" "$status" "$start_ns" "$end_ns" \
    >> "$RUN_DIR/results.csv"
}

export GIT_URL REPO RUN_ID RUN_DIR
export -f run_worker

seq 1 "$WORKERS" | xargs -n1 -P"$WORKERS" bash -c 'run_worker "$@"' _

This harness is deliberately incomplete: it does not implement authentication, cleanup, retries, or every scenario. Extend it only after the single-worker path is correct. A more mature runner should use a structured output format, monotonic timers, bounded retries, and a coordinator that can schedule scenarios without overwhelming the test host.

Measure the system at each boundary

Throughput alone is insufficient. Capture latency distributions and resource saturation at the same time.

Use a histogram or equivalent distribution view for latency. Report at least median, p95, and p99 when the sample size supports those percentiles. Also report the number of observations. A percentile from a very small sample can look precise while being statistically weak.

Test object storage assumptions explicitly

Remote object storage is attractive because immutable objects can be durable, horizontally scalable, and independently retained. The risk is that Git operations are not simply “write one file, read one file.” A single logical operation may involve listing, reading metadata, fetching pack ranges, uploading temporary data, committing an index, and updating refs.

For the object plane, test:

• many small object reads;
• large sequential pack reads;
• parallel range reads, if supported by the implementation;
• multipart or staged uploads, if supported;
• temporary object cleanup after failed pushes;
• read-after-write behavior required by the protocol path;
• request-rate limits and backoff behavior;
• cache hit and miss behavior.

Do not infer consistency from a successful happy-path test. Include a test that writes an object, records the returned identifier, immediately attempts the dependent read, and then exercises the corresponding ref or commit path. The acceptance condition should be defined by the Git backend’s correctness requirements, not by a generic assumption about the storage service.

Test concurrency as a correctness property

Concurrency is not only a performance variable. It changes correctness risks. A backend must preserve the relationship between refs and reachable objects, avoid lost updates, and handle retries without corrupting state or creating ambiguous outcomes.

Include these checks:

1. Two workers push different branches from the same base.
2. Two workers update the same branch with incompatible expected old values.
3. A worker retries after a client-side timeout where the server may have completed the update.
4. A worker uploads objects but fails before the ref update.
5. Maintenance runs while reads and writes are active.
6. A client reconnects during a long transfer.

For every failure, distinguish “operation definitely failed” from “operation outcome unknown.” An unknown outcome must be reconciled by reading the ref or querying an operation record before a retry is issued. Blind retries can duplicate work, amplify load, or overwrite a valid update.

Compatibility is a separate test track

High throughput does not compensate for an incompatible Git implementation. Validate protocol and repository behavior with a compatibility matrix that names the client version, transport, repository state, and expected result.

Use ordinary Git clients as the compatibility oracle where possible. If the backend exposes an internal API in addition to Git protocol endpoints, test both paths but do not treat internal success as proof of Git compatibility.

Analyze bottlenecks with a capacity model

After each run, classify the limiting resource. Useful categories include:

• CPU-bound: compression, delta search, hashing, pack indexing, or encryption consumes available CPU.
• Metadata-bound: ref locks, repository lookups, object indexes, or database transactions queue.
• Object-request-bound: the system issues too many small remote requests or encounters service throttling.
• Bandwidth-bound: transfer volume saturates a network link before compute or storage.
• Connection-bound: pools, file descriptors, proxies, or worker limits cap concurrency.
• Maintenance-bound: compaction, repacking, scans, or deletion competes with foreground traffic.
• Client-bound: the load generator or agent runtime cannot create enough pressure to test the backend.

Run a concurrency sweep rather than one large run. For example, use worker counts of 1, 2, 4, 8, 16, and 32 only if those values are appropriate for the test environment. Stop when error rates or latency become unacceptable, and record the stopping rule before looking at results. The interesting point is often where throughput stops increasing while queueing and tail latency continue to rise.

For each point, retain the raw records and a summary:

run_id,scenario,workers,successes,failures,
throughput_ops_per_sec,p50_ms,p95_ms,p99_ms,
bytes_read,bytes_written,retries,notes

The summary is derived data. Keep it alongside the raw operation records so another engineer can recalculate it.

Failure cases the experiment must expose

Hot references

A single default branch or shared coordination ref can become a serialization point. Compare many independent refs with repeated updates to one ref. If the second case collapses, scaling stateless protocol workers will not solve the primary problem.

Small-request amplification

An object layout can appear efficient for large transfers while performing poorly for many metadata requests. Count requests per logical Git operation and inspect the distribution, not only total bytes.

Retry storms

Transient failures become dangerous when every client retries immediately. Add bounded exponential backoff with jitter in the test client, then compare it with an intentionally aggressive retry policy to expose amplification.

Cache invalidation

A cache that serves stale refs can produce incorrect clones, confusing push failures, or repeated retries. Test cache invalidation after a ref update and after a failed update. A cache hit is useful only if it preserves the required correctness semantics.

Maintenance interference

Run maintenance during foreground load. A backend can pass isolated read and write tests but fail when pack generation or garbage collection competes for CPU, disk bandwidth, locks, or object-store requests.

Partial failure

Kill a client at different points in a push, interrupt a network path, and stop a worker during maintenance. Then verify that recovery is observable, bounded, and safe. “The service eventually recovered” is not enough; record what state remained and what operator action was required.

Recommended experiment sequence

1. Freeze versions, hardware, network topology, and configuration.
2. Generate and manifest the repository corpus.
3. Validate one clone, one fetch, one branch push, and one branch deletion manually.
4. Run each scenario with one worker and inspect raw logs.
5. Run the concurrency sweep with the local baseline.
6. Repeat with remote object storage and identical client behavior.
7. Repeat with cache enabled, documenting cache policy and invalidation.
8. Run the failure and maintenance cases separately from the throughput sweep.
9. Run the compatibility matrix against every supported transport and client profile.
10. Publish raw data, derived summaries, configuration, and limitations together.

Keep each run reproducible by assigning a run identifier and storing the exact command line, environment variables, configuration files, client version, server version, repository manifest, and clock source. Redact tokens, passwords, private endpoints, and personal identifiers before sharing artifacts.

What a useful conclusion looks like

A strong result does not say “the backend scales” in the abstract. It says something conditional and measurable, such as:

Under the recorded repository corpus, client version, protocol, cache state, and concurrency range, the observed limiting resource was X. Increasing workers beyond Y did not improve completed operations per second and increased tail latency or failure rate. The next experiment is Z.

That form keeps the conclusion tied to evidence. It also makes the next engineering decision clearer: reduce object-request fan-out, separate hot metadata, tune pack generation, enlarge a connection pool, isolate maintenance, add a cache, shard repositories, or change the agent scheduler.

Limitations

This experiment cannot establish universal performance, durability, cost, or operational suitability. Synthetic repositories may not reproduce real source distributions. A single load generator may under-drive the backend. A local deployment may omit production proxies, identity systems, backups, replication, or failure domains. Results can change with Git client versions, server versions, kernel settings, object-store behavior, compression settings, repository topology, and cache state.

It also does not replace a disaster-recovery exercise. Restore a representative repository, verify refs and reachable objects, and test client access after restoration before making durability claims.

Conclusion

Scaling a Git backend for AI development starts with workload decomposition, not with a larger machine or a more fashionable storage system. Separate protocol, immutable objects, mutable metadata, and control-plane behavior in the experiment. Measure request amplification, concurrency, tail latency, retries, maintenance interference, and compatibility. Preserve raw evidence, label examples as examples, and make every conclusion conditional on the recorded test environment.

The practical outcome is an architecture experiment that can tell you where the backend breaks and why. That is more valuable than a single headline throughput number, because agent workloads will continue to change as orchestration, repository lifecycles, and coding patterns evolve.

Continue with the guides or look up terms in the glossary.