The previous lesson solved one of the three ways in which a repository becomes unmanageable: very large files. Two remain, and they are independent of each other. A repository can have a colossal amount of history even if every file is tiny, and it can have a colossal number of files even if the history is short. Each problem has its symptom and its remedy, and applying the wrong remedy fixes nothing.

This lesson delivers on two of the course's promises. Lesson 06-05 left open the comparison between submodules, subtree, packages and monorepo, deferring the monorepo until here. 08-06 taught how to measure and optimise an ordinary repository that has become slow, and referred genuinely enormous repositories here. And case 3 of 10-01 described the enterprise monorepo, explaining the why and leaving the how for this lesson.

Let us start by making one thing clear, because it is the most important of all: almost nothing in here is needed by task-manager, and you probably do not need it either. This is material for the day you inherit a repository of fifty thousand files, and so that you can recognise when that day has come. Applying these techniques to a healthy repository adds complexity, oddities and a new surface for mistakes, in exchange for nothing.

1. The three dimensions of growth
2. Measure before optimising (and what to measure in each case)
3. Partial clones: --filter=blob:none and --filter=tree:0
4. Partial clone versus shallow clone
5. Sparse checkout and the sparse index
6. Client-side accelerators
7. Monorepo versus multi-repo
8. What it takes for a monorepo to work
9. When none of this is needed
10. A recipe per symptom

The most common mistake when faced with a slow repository is to treat it as one problem. It is not. There are three, they show up in different ways and they are fixed with different tools.

The three are independent. A repository can suffer one, two or all three at once, and each one has to be diagnosed separately.

An example that clarifies the independence: a repository with twenty years of history and sixty text files suffers dimension 1 and nothing else. git clone takes ten minutes and git status is instantaneous. Applying sparse-checkout to it would achieve absolutely nothing.

The rule from lesson 08-06 still stands and matters more here: intuition about what is slow is almost always wrong. Before applying any of these techniques, measure.

Three numbers that steer the decision, with approximate orders of magnitude:

They are reference points, not exact thresholds: they depend heavily on the hardware, the filesystem and the operating system. A repository of 30,000 files runs sweetly on a modern laptop with Linux and can be miserable on Windows with an antivirus checking every access.

Reading the diagnosis: 1.8 seconds reading the index and 9.1 seconds walking directories looking for untracked files. It is a pure dimension 2 problem. Neither the commit-graph nor a partial clone would do anything for it; what is needed is untrackedCache, fsmonitor and, if appropriate, sparse-checkout.

And task-manager, for reference:

None of the three dimensions. Nothing in this lesson applies to it.

The partial clone is, by a wide margin, the most useful and least known technique in this lesson.

A normal clone downloads every reachable object: every commit, every tree and every blob of the whole history. A partial clone downloads the graph but omits certain objects, and asks the server for them on demand when some command needs them.

This requires the server to support it — the major platforms and modern versions of Git do — and it works over a persistent connection called a promisor remote: the remote "promises" to have the missing objects.

It downloads every commit and every tree, but no blobs other than those needed to check out the working revision.

What carries on working without downloading anything:

What triggers an on-demand download:

Seeing it in action:

Git has gone off to fetch the 47 versions of the file it needed for the blame, and then answered. It is slower the first time and stays cached for subsequent ones.

The objects that have already been fetched stay put:

That --missing=print lists, with a ? in front, the objects that are missing locally. It is the way to know how much is left to download.

More aggressive still: it omits the trees as well. Only commit objects are downloaded.

With this, git log --oneline, git log --graph, git rev-list and any purely topological or message-based query carry on working. But almost any operation on files triggers downloads, including git log --name-only, because the names live in the trees.

It is the option for very specific cases: analysing the graph of an enormous repository, counting commits, extracting authorship statistics, generating a changelog. For day-to-day work, --filter=blob:none is almost always the right choice.

blob:limit=1m is an interesting middle ground: it brings down all the code (which is small) and leaves out the heavy binaries. Without needing LFS.

For a hypothetical very large repository, with approximate orders of magnitude:

The row that matters is the second: 8% of the download while keeping the complete history. That is what makes the partial clone the reasonable default for large repositories.

In lesson 02-02 we saw the shallow clone --depth=1. It is worth contrasting them, because they solve different things and the shallow one has traps the partial one does not.

1. It does not break Git's semantics. A shallow clone is an incomplete repository: there are commits that simply do not exist, and any operation that needs to reach them fails. A partial clone is a complete repository with deferred content: everything exists, some things get downloaded when you ask for them.

This is the underlying argument. With --depth, Git will lie to you about the history and you will have to remember that it is truncated. With --filter, Git will tell you the truth and download what it is missing.

2. Recovery is transparent. If a shallow clone needs something old, you have to run git fetch --unshallow, which downloads the entire repository in one go. A partial clone fetches only the specific objects it needs, without you intervening.

3. The shallow clone's errors are confusing. This is the classic one in a CI pipeline:

Or worse, in a proposal comparison:

Both fail because the divergence point is outside the downloaded depth. It is the reason so many CI configurations carry fetch-depth: 0 (download everything), which solves the problem the expensive way. The good alternative:

Complete history for the comparisons and for describe, without downloading gigabytes of old blobs. This is the recommended CI configuration for large repositories.

• The server does not support partial clones.
• A CI job that only builds the current revision and does not query anything from the history: --depth=1 is minimal and sufficient.
• A deployment container where you only want the files of a specific version. Although for that, git archive is usually better still:

That creates no repository: it downloads only the files of that tag.

The partial clone attacks what gets downloaded. sparse-checkout attacks what appears on disk, which is a different problem: dimension 2.

A monorepo with 300,000 files. Ana works only in services/tasks/, which is 400 of them. But her working copy has all 300,000, and consequently:

• git status walks 300,000 entries.
• The editor indexes 300,000 files.
• Every git checkout between branches checks 300,000 paths.
• Text searches trawl through code that is of no interest to her.

After this, the working copy contains only those directories (plus the files at the root), and git status goes from walking 300,000 entries to walking a few thousand.

And a more direct way of cloning with this already switched on:

That combination — partial clone + sparse checkout — is the standard setup for working in a large monorepo: you download little and materialise less.

sparse-checkout has two modes, and the difference is crucial.

Cone mode gives up expressiveness to gain a structural property: because the patterns are always whole directories, Git can reason about complete subtrees instead of about individual files. And that enables the genuinely important thing.

Recall from lesson 01-04 and from 08-06 that the index (.git/index) contains one entry per tracked file. In a monorepo of 300,000 files, the index has 300,000 entries and takes up tens of megabytes. Reading and writing it is the primary cause of git status being slow, and it happens even when the files are not on disk: the index still lists them.

The sparse index solves that. Instead of one entry per file, it stores a single entry per directory that lies outside the cone, pointing straight at its tree object.

It is switched on like this:

And checked with:

The effect on git status in a large monorepo is an order of magnitude. And the reason is purely structural: a tree entry represents an entire subtree, thanks to the fact that a tree's hash summarises all its content (lesson 01-04). If the hash of services/billing/'s tree has not changed, Git knows with mathematical certainty that nothing in that subdirectory has changed, without looking at a single file.

This is a lovely example of why understanding the data model matters: the sparse index is not a trick, it is a direct consequence of content addressing.

With sparse-checkout active, git status includes a notice:

That notice is important and you have to know how to read it. You are not seeing the whole repository. If you look for a file and it does not appear, perhaps it is not that it does not exist: it is outside your cone.

Commands that behave differently:

And the oddity that disconcerts most: if you git checkout a branch that deletes a file outside your cone, you will see nothing, because that file was never on your disk. It is all consistent, but it requires having the model clear.

These already appeared in lesson 08-06; here they come with the nuance of scale.

It is an auxiliary index file that stores, precomputed, the topology of the history: for each commit, its parents, its date and its generation number (the distance to the root commit).

Without it, answering "is A an ancestor of B?" forces Git to read commit objects from disk and walk the graph. With it, the generation number allows whole branches of the walk to be discarded without reading anything.

The --changed-paths deserves special attention in enormous repositories: it stores a Bloom filter with the paths modified in each commit. It serves to drastically speed up:

In a monorepo, git log on a specific file goes from examining millions of commits to discarding almost all of them with a bit check. It is the difference between thirty seconds and half a second.

It is the dimension 1 optimisation with the best cost-to-benefit ratio, and it has no drawback whatsoever: it is a regenerable cache file that alters nothing in the repository.

fsmonitor starts a daemon that listens to filesystem notifications and keeps the list of what has changed. That way git status does not have to walk the directories: it asks the daemon.

untrackedCache stores in the index the last scan of each directory along with its timestamp, so as not to rescan the ones that have not been touched.

The combined effect in a repository of 100,000 files, as an order of magnitude: git status can go from several seconds to fractions of a second. The combination with the sparse index is multiplicative, because they attack different parts of the cost: fsmonitor reduces what has to be looked at on disk, the sparse index reduces what has to be read from the index.

A platform note: the built-in fsmonitor works on macOS and Windows out of the box; on Linux it depends on the Git version and may require additional configuration. Measure it before and after in your specific environment.

It schedules periodic tasks in the system scheduler: updating the commit-graph, packing loose objects incrementally, prefetching from the remotes and cleaning up references.

Compared with the traditional automatic gc, it has two advantages: it runs when you are not working rather than interrupting you, and it includes tasks gc does not do, such as the prefetch (which downloads in the background what others publish, so that your git fetch is instantaneous).

Recommended configuration for a large repository:

Git offers two configuration "packages" that switch several things on at once:

feature.manyFiles switches on index.version=4 (a more compact index format), core.untrackedCache=true and index.skipHash=true. It is the reasonable shortcut for dimension 2. Exactly what it does may change between Git versions, so it is worth checking with git help config.

Here the comparison that lesson 06-05 left open and that case 3 of 10-01 left pending is closed.

Multi-repo: each project or library has its own repository. The relationship between them is established with published versions (packages), with submodules or with subtree.

Monorepo: a single repository contains many projects, which relate to each other by directory path and are built together.

Notice the versions in the left-hand diagram: three consumers using three different versions of the same library. That is normal in a multi-repo, and it is at once its greatest advantage (autonomy) and its greatest problem (drift).

A monorepo makes sense when:

• The projects change together often: the unmistakable symptom is that a proposal needs coordinated changes in several repositories.
• There are shared internal libraries and version drift already hurts.
• Everybody can see all the code.
• There is capacity to invest in tooling and to maintain it.
• You want global refactoring to be possible.

A multi-repo makes sense when:

• The projects are genuinely independent and evolve at different rates.
• There are access requirements that force separation.
• The teams are autonomous and release on their own account.
• Some components are published outside the organisation.
• There is no capacity to maintain in-house tooling.
• You prefer the solution that works on its own.

Of all the criteria, one weighs more than the rest:

How often does a change need to touch several repositories at once?

If the answer is "hardly ever", multi-repo, without hesitation. If it is "constantly, and it is our biggest bottleneck", the monorepo solves exactly that problem and is worth the investment.

And remember the conclusion of lesson 06-05: submodules do not solve this. They tie a specific version of a library to a consumer — they give traceability — but every update is still a commit in every consumer. They are a multi-repo with traceability, not a cheap monorepo.

It is neither a binary nor an irreversible decision. The most common intermediate patterns:

1. Monorepo by domain. One repository per large area (platform, product, data), not one per service nor one for everything. It captures most of the atomicity benefit without reaching scales that demand exotic tooling. It is the sweet spot for most medium-sized organisations.

2. Monorepo for the shared libraries. Only the internal libraries are grouped, which are the ones suffering version drift, and the products stay separate. It is a reversible step with immediate benefit.

3. Gradual migration. One repository can be merged into another preserving its history:

The combination of merge -s ours + read-tree --prefix is the classic mechanism: the merge connects the two histories without bringing in content, and the read-tree places the other repository's tree under the given prefix. The result is that git log --follow libraries/ui-components/button.js carries on working backwards. It is also, essentially, what git subtree add does underneath (lesson 06-05).

If the decision is a monorepo, these pieces are not optional. A monorepo without them is simply a slow and chaotic repository.

This is best put into an onboarding script in the repository itself, because nobody is going to remember six commands.

The CODEOWNERS file we saw in lesson 10-01. Without it, the monorepo loses the clear ownership a multi-repo gives away for free.

We already saw the skeleton in 10-01. The version that matters here is the one based on the dependency graph, not on directories:

Notice the sparse-checkout inside the CI job: each run materialises only the project it is going to test. It is the same technique as section 5, applied to the pipeline.

What affected.sh does is what distinguishes a good selective CI from a naive one: if libraries/ui-components changes, it is not enough to test that library; every project that depends on it has to be tested. That demands an explicit dependency graph, which the build system provides.

In a monorepo, building everything on every change is unfeasible. You need a system that understands the dependency graph and that caches results by the hash of the inputs: if nothing going into a target has changed, the previous result is reused.

It is, curiously, the same principle that underpins Git: identify by the hash of the content and do not recompute what has not changed.

With many projects in one repository, strong conventions are needed: a homogeneous directory structure, predictable names, and tools for applying mass changes ("rename this function in 200 places") automatically and reviewably.

This section is as important as the previous ones.

task-manager needs nothing from this lesson. No partial clones, no sparse-checkout, no fsmonitor, no monorepo. It has nine files, 412 commits and weighs 2 MB. git status answers in milliseconds.

And it is not an exceptional case: the vast majority of the world's repositories are in that situation. A project with 5,000 files, 30,000 commits and 200 MB runs perfectly with Git out of the box on any modern computer.

The most dangerous technique on the list is sparse-checkout. A newcomer who inherits a sparse configuration without knowing it can spend hours failing to understand why a file they can see on the web is not on their disk.

If git status answers in less than a second and git clone takes less than a minute, you do not have any problem these techniques solve.

Measure first (section 2). Apply the remedy of the dimension that hurts. Measure afterwards. And if nothing hurts, do nothing.

A quick reference table.

Mistake 1: optimising without measuring. The central mistake of the lesson. Applying sparse-checkout to a history problem does nothing. Diagnose the dimension first.

Mistake 2: using --depth=1 in CI and then needing the history. It produces fatal: no merge base and fatal: No names found. Use fetch-depth: 0 with filter: blob:none.

Mistake 3: sparse-checkout without --cone. Pattern mode is slower, does not allow a sparse index and is far easier to misconfigure. Use --cone unless you have a very specific reason.

Mistake 4: forgetting that sparse-checkout is active. Document the sparse configuration in the README.md and remember that git status warns with "You are in a sparse checkout with N% of tracked files present".

Mistake 5: believing a partial clone works offline. If you work on a plane with a partial clone and run git log -p over old history, it will fail: it needs to go to the server. Before disconnecting, fetch what you are going to need:

Mistake 6: choosing a monorepo because it is fashionable. Without selective CI, without CODEOWNERS and without a build system with a cache, a monorepo is worse than what you had. The question is not "what do the big companies do?", but "how often does one of our changes touch several repositories?".

Mistake 7: believing submodules are a cheap monorepo. They are not: they do not give atomicity. Every update of the library still demands a commit in every consumer (lesson 06-05).

Tip 1: commit-graph for everybody. It is the only optimisation in the lesson with no drawback at all. Switch it on even if your repository is medium-sized:

Tip 2: git maintenance start on your large repositories. Better than waiting for the automatic gc, and without interrupting you.

Tip 3: measure with hyperfine or with a simple loop. Before and after, several times:

Tip 4: put an onboarding script in the repository. If your monorepo needs six configuration commands, put them in tools/setup.sh and mention it in the README.md.

Tip 5: CI can use sparse checkout too. Modern workflows allow you to declare which directories to materialise. In a monorepo, that reduces each job from minutes to seconds.

For each repository, say which dimension it suffers from, what would NOT fix it and what would:

A.

B.

C.

Ana joins a team with a monorepo: 340,000 files, 2.1 million commits, 12 GB. She will work only in services/tasks/ and will use libraries/ui-components/.

Write the complete sequence of commands, from the clone to a working setup, explaining what each one attacks and which dimension. Include also what you would tell her about the oddities she is going to run into.

A 120-person company has 23 repositories: 8 services, 6 client applications, 9 internal libraries.

The symptoms they report:

• Changing the interface of the authentication library takes six weeks to reach all its consumers.
• There are four different versions of ui-components in production at once.
• Nobody knows which services use which libraries.
• A new contributor takes three days to clone and configure what they need.
• Two of the services are maintained by an external company that must not see the rest of the code.
• The charts and utils libraries are published publicly as open source.

Decide what you would recommend, with which steps and in which order. Point out explicitly what you would not put into the monorepo and why.

Solution 1

A. Pure dimension 1: lots of history.

Evidence: 1.8 million commits but only 1,204 files. git status answers in 89 ms (dimension 2 does not exist). The repository weighs 1.84 GB, which is consistent with 1.8 million commits of text code, not with binaries.

The decisive symptom: git log on a file takes 47 seconds. Git has to walk millions of commits checking whether each one touched src/core.c.

What would NOT fix it: sparse-checkout (there are only 1,204 files and status is already instantaneous), LFS (there are no binaries), fsmonitor (nothing to walk on disk).

What would:

With --changed-paths, that 47-second git log -- src/core.c should drop to under a second: the Bloom filter discards the vast majority of the commits without opening their trees.

B. Dimension 2 dominant: lots of files.

Evidence: 285,000 files, only 8,420 commits (the history is short), and git status takes 42 seconds. The 3.21 GB is consistent with many files, not necessarily with large binaries.

What would NOT fix it: commit-graph (with 8,420 commits, the history is not the problem), a shallow clone (nor that).

What would, in order from least to most intrusive:

The order matters: steps 1 and 2 do not change what you see on disk and may resolve it. Step 3 does change it and has a comprehension cost for the whole team.

It is also worth checking whether those 3.21 GB hide a dimension 3 problem:

C. Pure dimension 3: very large files.

Unmistakable evidence: 89 files, 1,204 commits, 6.72 GB. The arithmetic leaves no room for doubt: 6.72 GB across 89 files is about 75 MB per file on average. They are binaries.

What would NOT fix it: absolutely nothing from sparse-checkout or commit-graph. Not even a shallow clone would help much, because the large blobs of the latest revision already weigh what they weigh.

What would:

Step 4 is a useful relief: the partial clone avoids downloading all the historical versions of the binaries, even though it does not fix the size of the repository on the server side.

Solution 2

--filter=blob:none avoids downloading the historical content of 2.1 million commits: from 12 GB we go down to a small fraction, keeping the complete graph. --sparse makes the initial checkout materialise only the root, instead of writing 340,000 files to disk.

--cone allows the sparse index; --sparse-index and index.sparse make the index store one entry per directory outside the cone instead of one per file. From ~340,000 entries down to a few thousand.

What I would tell Ana about the oddities:

1. "You are not going to see most of the files." git status will warn you: "You are in a sparse checkout with 2% of tracked files present". They are not deleted: they are not materialised. git show HEAD:services/billing/main.js works perfectly.

2. "If you need another area, add it." git sparse-checkout add services/notifications. There is no need to clone again.

3. "Some operations will go to the network." git log -p over old history, git blame on a file with many versions, or git checkout of an old tag will download blobs. The first time is slow; after that it stays cached.

4. "Before going offline, fetch what you are going to need." A partial clone needs the server for what it does not have.

5. "Your editor and your searches only see your cone." That is good for performance, but if you search for a function and it does not appear, perhaps it is outside your area. git grep --no-index or searching on the repository's website are the way out.

6. "If something behaves oddly, look at git sparse-checkout list first." It is the prime suspect for nearly any oddity.

And I would hand her a tools/setup-workstation.sh script with all of the above, because nobody remembers fourteen commands.

Solution 3

Recommendation: a partial monorepo, in three phases, starting with the libraries.

The dominant symptom is unmistakable: six weeks to propagate a library change and four versions of ui-components in production. That is exactly the problem the monorepo eliminates at the root, and it confirms there is strong coupling between the libraries and their consumers.

But there are two constraints that rule out a total monorepo, and they have to be respected.

What does NOT go into the monorepo, and why:

Phase 1: a monorepo of internal libraries (7 libraries).

It is the step with the best benefit-to-risk ratio, and it is reversible.

Immediate benefit: a change that touches authentication and data at once becomes one commit, and git bisect works across libraries.

These libraries carry on being published as versioned packages for their consumers, so nothing changes for the 12 products. That is why it is reversible.

Phase 2: absorb the most tightly coupled consumers.

Measure first which they are:

The ones that turn up repeatedly alongside library changes are the candidates. Those get absorbed, not all of them.

Phase 3: consolidate the rest, if the experience has been good.

Compulsory investment before phase 2 (without this, the monorepo makes things worse):

1. CODEOWNERS by directory — recovers the ownership the multi-repo gave away for free.
2. Selective CI with a dependency graph — without it, every proposal runs the entire suite.
3. A build system with caching by input hash — or the build times go through the roof.
4. A workstation setup script — with --filter=blob:none --sparse, as in solution 2.

What each reported symptom becomes:

Final result: 1 monorepo (7 libraries + whichever internal products get absorbed), 2 private repositories for the external company, 2 public repositories. From 23 repositories down to 5, without violating any constraint.

What I would NOT do: put it all in at once. Phase 1 is reversible and gives benefit within weeks; a total migration of 23 repositories without tooling is a well-established way of ending up with a 40 GB repository, a two-hour CI and a furious team.

Scaling Git starts with no longer treating it as a single problem.

• There are three independent dimensions: lots of history (git log and blame slow), lots of files (git status slow) and very large files (a repository of gigabytes). Each one has its symptom, its measurement and its remedy, and applying the wrong remedy fixes nothing. They are diagnosed with git rev-list --count --all, git ls-files | wc -l and git count-objects -vH, and refined with GIT_TRACE2_PERF.
• Partial clones are the most useful and least known technique: --filter=blob:none downloads the complete graph and omits the historical content, requesting it on demand; --filter=tree:0 omits the trees as well. They preserve log, bisect, describe and merge-base.
• Compared with the shallow clone --depth, the partial one is almost always better: --depth produces an incomplete repository in which commits are missing and operations fail (fatal: no merge base), whereas the partial one produces a complete repository with deferred content. For CI: fetch-depth: 0 + filter: blob:none.
• sparse-checkout --cone limits what gets materialised on disk, and the sparse index stores one entry per directory outside the cone instead of one per file. Cone mode gives up expressiveness in order to be able to reason about complete subtrees, something only possible because a tree's hash summarises all its content (lesson 01-04).
• The client-side accelerators — commit-graph --changed-paths, fsmonitor, untrackedCache, feature.manyFiles and git maintenance — attack different parts of the cost and combine well. The commit-graph is the only one with no drawbacks: always switch it on.
• Monorepo versus multi-repo is decided with a single question: how often does a change need to touch several repositories at once? The monorepo buys atomicity and global refactoring; it pays with in-house tooling, CODEOWNERS, selective CI and Git scaling. And there are middle roads: monorepo by domain, a monorepo of libraries only, gradual migration with merge -s ours + read-tree --prefix.
• And most importantly: most repositories need none of this. task-manager has nine files, 412 commits and 2 MB. Applying sparse-checkout to it would be adding confusion in exchange for nothing.

The rule that sums up the lesson, and which is the one from 08-06 taken to another scale:

Measure, diagnose the dimension, apply that dimension's remedy, and measure again. If git status answers in less than a second, you do not have any problem these techniques solve.

We now know how to operate Git in repositories of any size. One last piece of the real world remains, and it is the one that turns Git into something more than a developers' tool.

In lesson 07-06 we drew a boundary: there we talked about continuous integration — automatically checking that what gets integrated works — and left for later how that code reaches users. In 07-04 too we deferred the mechanics of environments and promotion, and in 05-05, when tagging v1.4.0, we said that annotated tags would be the piece that triggers a deployment.

All of that converges here. In many organisations, Git is no longer used only by one person at a terminal: it is used by a hundred pipelines that clone, tag, build and publish with no human intervention. The repository stops containing only code and comes to contain the definition of the environment, the configuration and the infrastructure. And a git push to the right branch stops meaning "I have saved my work" and comes to mean "this is in production in four minutes".

Lesson 10-05: Git in DevOps delivers on that promise: Git as the single source of truth, the three rungs of CI, delivery and deployment, what triggers each one, why the artefact must be identified by the commit hash, what GitOps is, where secrets live in an automated world, and why rolling back in production is almost never a git revert.