Sharing rust-analyzer across Git worktrees: in-memory overlay architecture and concurrency lessons

In a multi-agent coding environment, every task runs in an isolated Git worktree. The primary checkout remains undisturbed while delegated tasks branch into dedicated worktrees. That isolation is essential: without it, concurrent edits in the same checkout clobber build outputs, invalidate compiler caches, and risk committing unvetted changes.

The bottleneck is the language server. In Rust, running an independent rust-analyzer instance per worktree means loading a complete RootDatabase—the crate graph, standard library metadata, third-party dependency HIR, and parsed ASTs—for each working tree. In a non-trivial workspace, a single analysis host typically consumes several gigabytes of resident memory. When multiple agents operate concurrently on sibling tasks, running separate processes quickly compounds host memory pressure.

Yet for worktree checkouts derived from the same base repository, the vast majority of the code is identical: the lockfile, external dependencies, and unchanged workspace crates share the exact same syntax trees and semantic models.

We explored serving multiple worktree checkouts from a single in-memory rust-analyzer instance. Here is the architecture of the overlay, the subtle concurrency and isolation bugs discovered during integration, and the status of our upstream proposal.

---

Architecture: Shared Database with VFS Overlays

Instead of launching separate server processes, the multi-worktree model embeds load_cargo::worktrees::Worktrees, maintaining a single shared RootDatabase and Vfs for a base checkout plus any number of overlay worktree copies:

[ Shared Overlay Model ]
Worktree A ──(ViewId A)──┐
Worktree B ──(ViewId B)──┼──> [ Shared Analysis Host ] ──> Shared RootDatabase
...                      │                                  ├─ Shared stdlib & dependencies
Worktree N ──(ViewId N)──┘                                  └─ Per-worktree VFS overlay deltas

Fast-Path Metadata and Conditional Fallbacks

Evaluating cargo metadata parses manifests, resolves feature flags, and builds the dependency graph. When an agent creates a worktree to write unit tests or implement an isolated feature, the workspace manifest, package manifests, lockfile, and .cargo/config.toml typically remain identical to the base checkout.

Worktrees::workspace_of_copy(&overlay) inspects these inputs:

• When package manifests, lockfiles, and cargo configs match, it relocates the already-parsed ProjectWorkspace directly to the copy’s root, skipping redundant cargo metadata execution.
• If manifests, feature sets, or toolchains diverge in the worktree, the fast path conditionally falls back to loading an independent workspace model.

Selective Build-Script Output Inheritance

Running custom_build scripts for external crates produces output directories and proc-macro dynamic libraries. Worktrees::inherit_build_scripts(&vfs, &mut workspace, &overlay) selectively passes build-script outputs from the base checkout to unchanged, compatible packages in the overlay, avoiding redundant compilation of identical proc-macro crates while ensuring that packages with modified sources or feature flags receive a fresh build.

---

Concurrency and Isolation Lessons

Integrating a shared database across concurrent worktrees revealed two critical edge cases that single-client setups never encounter.

1. View-Scoped Path Containment

In rust-analyzer, path resolution historically queried the virtual filesystem globally (Vfs::file_id). When Worktree B created an untracked scratch file or received an in-memory didOpen buffer, that file became known to the global VFS.

When a query in Worktree A resolved file paths, a raw VFS lookup for Worktree B’s file could return Some(file_id), allowing Worktree A’s analysis snapshot to traverse private files belonging to Worktree B.

The VFS view model provides isolation via ViewId. We added view-scoped lookup (Views::file_in_view):

impl Views {
    pub fn file_in_view(
        &self,
        vfs: &Vfs,
        view: ViewId,
        path: &VfsPath,
        is_in_a_crate: bool,
    ) -> Option<FileId> {
        let file_id = self.file(vfs, path, is_in_a_crate)?;
        if self.in_view(file_id, view) {
            Some(file_id)
        } else {
            None
        }
    }
}

By ensuring that file resolution explicitly checks in_view(file_id, view), untracked files and draft edits remain strictly partitioned within their originating worktree.

2. Salsa Cancellation Across Worktrees

In Salsa, database consistency dictates that any write transaction (such as saving a modified file) increments the database revision. Active read queries executing on background threads that encounter a concurrent write may return salsa::Cancelled.

In single-developer environments, cancelling a query when the user types a new character is expected behavior. In a multi-worktree environment sharing one database, however:

• An edit in Worktree A triggers a write transaction.
• An in-flight definition or hover query in Worktree B encounters that write and returns salsa::Cancelled.
• If the embedder treats Cancelled as a fatal failure, queries in sibling worktrees abruptly abort.

The solution requires explicit cancellation tolerance:

1. Batch file change notifications where possible to minimize revision bumps.
2. Provide a retry_cancelled execution helper in the embedder API: when a query is cancelled because of a concurrent write in a sibling worktree, the runner drops the invalidated snapshot, acquires a fresh snapshot of the updated database, and retries the query transparently.

---

Memory Footprint and Benchmark Evidence

In one release-build benchmark on a 328-crate workspace, the base process used about 1.8 GB RSS. With build scripts and proc macros disabled, sixteen worktree copies with a leaf-crate edit reached about 2.07 GB total RSS, roughly 16 MB per copy. This result is workload- and configuration-specific.

Sub-linear scaling holds as long as the worktrees share common base dependencies and toolchain configurations, avoiding full duplicate database allocations for each working tree.

---

Upstream Proposal Status

We have separated our internal deployment from the upstream contribution path:

1. Production Fork:
   • The shared overlay is maintained in our fork on branch prod/worktree-overlay, pinned to an immutable revision in our engine gateway for deterministic builds.
2. Upstream rust-analyzer:
   • We submitted an initial proposal pull request to the upstream project: rust-lang/rust-analyzer#23454 (feat: in-memory worktree overlay sharing and multi-client analysis host).
   • This submission is currently awaiting review by the rust-analyzer working group.
   • In alignment with upstream contribution guidelines, our next step is to coordinate with maintainers on Zulip to determine the best approach for decomposing the feature into smaller, focused PRs (such as standalone VFS view-scoping, multi-client session handling, and project inheritance) that can be integrated incrementally into the compiler core.

---

The Rule

Share immutable AST and semantic caches eagerly, isolate filesystem views at the VFS boundary strictly, and make every read transaction resilient to sibling cancellations. Without all three, an in-memory compiler overlay either leaks state across tasks or falls over on the first concurrent edit.

Alexander Panasenko (2026-10-02). Sharing rust-analyzer across Git worktrees: in-memory overlay architecture and concurrency lessons. https://prod.codes/blog/one-engine-ten-worktrees/