Git Monorepos: Scaling Workflows and VFS for Large Codebases

As engineering organizations grow, the strategy for managing source code becomes a critical architectural decision. Historically, organizations split codebases into hundreds of individual repositories—the Polyrepo model.

Today, many technology companies (such as Google, Meta, and Microsoft) use the Monorepo model, storing their entire catalog of projects, libraries, and microservices inside a single Git repository.

While monorepos simplify dependency management, code sharing, and cross-project refactoring, they push the Git version control system to its limits. When a repository grows to millions of files and gigabytes of history, standard Git operations (such as git status, git checkout, or git fetch) become painfully slow. This systems audit analyzes how to scale Git monorepos using Virtual File Systems (VFS), sparse checkouts, and cache-optimized build pipelines.

---

The Monorepo vs. Polyrepo Dilemma

Choosing between monorepos and polyrepos involves several software engineering trade-offs:

• Polyrepos (Multi-repo):
  • Pros: Isolated security permissions, small repository sizes, and fast local Git performance.
  • Cons: Complex dependency management, difficult cross-project refactoring, and fragmented CI/CD configurations.
• Monorepos (Single-repo):
  • Pros: Single source of truth, simplified dependency management, atomic commits across projects, and centralized CI/CD configs.
  • Cons: Version control performance degradation, access control challenges, and high local build times.

For large engineering teams, the collaborative benefits of a monorepo often outweigh the drawbacks, provided the performance issues can be managed.

---

Git Scaling Bottlenecks at Scale

Standard Git was designed for medium-sized projects (like the Linux kernel). In a massive monorepo, several performance bottlenecks emerge:

1. Object Graph Traversals

Git tracks history as a directed acyclic graph (DAG) of commit, tree, and blob objects. As the number of commits reaches millions, operations that traverse the graph (such as git log or git branch) consume significant memory and CPU.

2. The Index File

The Git index (.git/index) is a binary file listing every file tracked in the working directory. In a repository with 500,000 files, the index can exceed 100MB. Reading and writing this file during standard operations adds delay.

3. File System Scanning (lstat)

When running git status, Git must execute an lstat() system call on every file in the working directory to detect modifications. On slow filesystems or repositories with millions of files, this scan can take minutes.

---

Mitigating Scale: Sparse Checkouts and Cone Mode

To prevent developers from downloading and scanning millions of files they do not need, Git supports Sparse Checkouts.

Sparse checkout allows a developer to configure Git to only write a subset of the repository’s files to the local working directory. The remaining files are kept hidden in the object database without being populated on disk.

                  Git Monorepo Directory Tree
                            [ Root ]
                               |
         +---------------------+---------------------+
         |                     |                     |
   [ Shared Libs ]       [ Frontend App ]      [ Backend API ]
   (Checked Out)         (Checked Out)         (Hidden on Disk)

By enabling Git’s Cone Mode (git sparse-checkout set --cone), developers specify directories rather than complex file patterns. This allows Git to optimize directory scanning, reducing lstat() overhead and accelerating status checks.

---

Virtual File Systems (VFS) and Scalar

For extremely large codebases (such as Windows or Office, which contain millions of files), sparse checkouts alone are insufficient. These scale challenges are addressed using Virtual File Systems (such as VFS for Git) and Scalar.

1. VFS for Git (Virtualization)

Developed by Microsoft, VFS for Git virtualizes the underlying filesystem. When a developer clones a repository, VFS for Git downloads only the repository metadata. The working directory appears to contain all files, but the files are actually empty placeholders. When an editor or tool reads a file, the virtual filesystem driver intercepts the call and downloads the file contents from the server in real time.

2. Scalar

To standardize monorepo scaling, Microsoft and the Git community created Scalar. Scalar is a command-line tool built into Git that configures optimal performance settings for large repositories:

• Enforces sparse checkouts.
• Enables the commit-graph file format to accelerate graph traversals.
• Runs background maintenance tasks (such as pre-fetching objects and packing refs) to keep local repositories optimized.

---

Monorepo Build Systems: Caching and Compilation Graphs

Enabling fast Git performance is only half the battle; developers must also be able to compile and test code without waiting hours. Traditional build tools (like Make or Gradle) are not designed for monorepos. Modern monorepos use monorepo-aware build engines (like Bazel, Buck, or Turborepo) that rely on two core concepts:

1. Directed Acyclic Graph (DAG) Compilation

The build tool analyzes the monorepo’s dependency tree, constructing a DAG of compile tasks. This allows the compiler to run independent tasks in parallel across all available CPU cores.

2. Remote Build Caching

Build engines calculate cryptographic hashes of all input source files, compiler versions, and configuration flags. If the hash for a build task matches a previously cached run, the engine skips compilation and pulls the pre-built output directly from a local or remote cache, reducing build times.

---

Conclusion & Key Takeaways

Scaling a Git monorepo requires a combination of Git optimizations and specialized build tooling. By leveraging sparse checkouts, Scalar configurations, and cache-optimized build engines, organizations can maintain a unified codebase while keeping developer workflows fast.

• Use Sparse Checkouts: Configure developers to check out only the directories they are actively working on.
• Run Scalar: Enable Scalar configurations to automate background repository maintenance.
• Implement Cache Build Tools: Use Bazel or Turborepo to enable compile caching and parallel execution.

---

FAQ

What is the commit-graph file in Git?

The commit-graph file is a performance feature that stores Git’s commit history in a binary, column-oriented format, allowing Git to skip parsing text-based object files during history traversals.

Can a monorepo support project-specific access permissions?

Standard Git does not support directory-level read permissions; cloning a repository grants access to the entire history. To restrict access, companies use proxy servers, Git virtual file systems, or split directories into submodules.

How does Git’s filesystem monitor (FSMonitor) work?

FSMonitor is a hook that integrates Git with operating system filesystem watchers (such as FSEvents on macOS or inotify on Linux), allowing Git to immediately receive a list of modified files rather than scanning the entire directory tree.

---

Related Inquiries

• Explore container isolation security.
• Learn about CSS layout engines performance rendering.