The Registry Listing Cache

A plugin registry answers GET /api/plugins with the catalog an installation may install from. Until #4222, every one of those requests read the whole source repository from GitHub and parsed every manifest in it, and about sixty a day did not finish inside the client's 30-second attempt budget.

What was measured

Neither is an outage: the 90 s total budget covers a retry, so the listing usually lands on a later attempt. What it costs is that every catalog open and every /Store render can wait out a 30 s attempt before anything is even retried, and that sixty red lines a day have to be recognised and discounted by every reader of the log.

The mechanism

GET /api/plugins → PluginRegistryEndpoints.List does three things per request, and the first dominates:

1. Sources(hub, config) resolves the sources FRESH. PackageSources.FromConfiguration → PackageSources.FromRepo constructs a brand-new IPackageSource on every request. That is the detail that decides the design: a cache held by the source instance would be discarded with it.
2. ListPackages(gitRef) reads the repository. For a URL source that is IGitHubRepoClient.Fetch — a shallow clone into a temp directory — followed by parsing every package.json / index.json in the tree. ~70 modules.
3. Then a mesh read of the pushed publications, and IsGranted(caller, source, package) per package.

What is cached, and what is deliberately not

The SOURCE snapshot, keyed by (repoUrl, subdir, format, gitRef) — never the response.

🚨 This is a security property, not an implementation detail. The key carries no caller, because the value carries no per-caller decision. The grant filter, the plan-tier refusals and the pushed-artifact stamp all still run per request over the cached list. Caching the RESPONSE instead would be #3768's shape — the registry answering with an identity that is not the caller's — and a latency fix would have become a disclosure bug. PackageListingCacheTest pins it from the other side: two instances with different grants, served off one cached listing, each see only their own packages.

File fetches are not cached. FetchPackageFiles is a per-package read on the INSTALL path; it is not repeated per catalog render, and a stale one would install stale bytes. It is NARROWED instead — see the narrow-fetch section below.

🚨 A LOCAL directory source is not cached either, and that is a correctness boundary rather than a tuning choice. The defect is a repository fetched OVER THE NETWORK per request; a local read is cheap, has no webhook to invalidate it, and its listed version is contractually a function of the files on disk right now — a mounted source must report an edit on the very next listing (LocalSourceContentVersionTest pins exactly that, and it is the test that caught the first version of this change caching one). A local-dev or air-gapped registry would otherwise have reported five-minute-old content.

A fault is not cached. The entry lives in a PromiseCache, which evicts on the terminal OnError, so a transient GitHub failure is retried by the next caller rather than replayed for the life of the pod (#1369). Concurrent first callers still share the one read — a burst of catalog opens used to be a burst of clones, which is most of how the 30 s budget was reached at all.

Invalidation, in two layers

Primary — the green build. The registry already receives the plugins repo's webhook; a green build lands as Admin/_Build/{owner}.{repo}, and the listing for that repository is forgotten — so a merge is visible on the next request rather than after a wait.

🚨 Both source lists are covered, and they are genuinely different lists. PluginUpdateWatcher historically watched only repositories named by a PluginCatalog NODE (SourceRepoPath), while /api/plugins serves what PackageSources.FromConfiguration reads from PluginCatalog:Sources:N:RepoPath — which is how the fleet registry is wired. Evicting only on the first would have left the freshness window as the only invalidation for the sources that actually matter, silently. The watcher now also opens an evict-only build watch per configured source, read through FromConfiguration itself so it cannot disagree with what the endpoint serves; a repository that is on both lists is watched once and does both.

Safety net — a freshness window. PluginCatalog:ListingCacheSeconds (default 5 minutes, 0 switches caching off) bounds how stale a listing may be when a broadcast never arrives. This is the same role, and the same justification, as PluginCatalogOptions.ReconcileSafetyNetInterval one layer up — it bounds the staleness of a cache; it is not a bound placed over anything that hangs. A malformed value falls back to the default rather than to "off": a typo must not silently restore the per-request read.

An expired listing is still the answer — the re-read runs behind it (#4963)

🚨 A request never waits for a re-read of a listing the registry already holds. Past the window, or after a green build of its repository, the held listing answers at once and ONE revalidating read is started behind it; its answer replaces the held one when it lands. Only a key that has never been read makes its caller wait, and concurrent first callers share that read.

What forced it, measured on the fleet registry on 2026-10-09 (governed Logs on the control instance, Ops/Actions/sevm-20261009-sourcelisting, Ops/Actions/sevm-20261009-2006-cloud-registry-deadline-105m): the answer deadline refused 123 GET /api/plugins requests in 110 minutes with every source listing still running, for example "Still waiting on: source listing 'Plugins' (running 24.9 s); source listing 'Education' (running 24.9 s); … 'SocialMedia' (running 24.9 s)", and 124 index and bundle requests with "Still waiting on: package origin anchor (running 25.0 s)". In both cases authentication and every other stage had finished in well under a second. Each expired listing had been dropped, so the request that found it waited for the re-read. A re-read of all five sources queues its git processes on the shared Process pool beside GitSync, and that wait outlasted the 25 s budget. The data was there the whole time.

The entitlement anchor (PackageOriginAnchor), which the bundle index decides grants against, had the same shape one layer up. Every read past its 60 s window listed every source with the request waiting, and a non-authoritative snapshot was never reused at all. It now answers from the snapshot it holds the same way: one shared re-listing, released when it settles. A window of 0 still means no reuse.

What it does not loosen. The window says how long a read may be reused before the source is asked again. It never meant that a caller must wait for the source. The worst staleness is unchanged in kind: the window plus one read's duration, which a waiting caller also got, minus the wait. Three behaviours hold it to that:

• A read that started before a green build lands as stale, so the next request reads again.
• A failed re-read keeps the held listing in service and is logged with the listing's age. The entry stays expired, so the next request asks again.
• A re-read still in flight after a whole window is named in the log, and a new one starts beside it.

PackageListingCacheTest and PackageOriginAnchorAnswersFromItsSnapshotTest pin this behaviour. Both are red against the previous code ("emitted nothing at all" in 12 s). Their negative controls show that a key with nothing held does wait for its source.

Where it lives

A mesh-scoped singleton, registered by AddPluginCatalog — never static (No Static State), so a test mesh cannot inherit another's catalog. It has to be mesh-scoped for a second reason too: the thing that must outlive a request is precisely what the per-request source factory cannot hold.

PackageSources.FromRepo wraps at one site, with the three source shapes built in a separate Build, so a source shape added later is cached by construction rather than by whoever adds it remembering. A mesh without the plugin catalog registered — a bare test mesh, the CLI — resolves no cache and gets the source unchanged.

The other half — what ONE read costs

A cache changes how often the repository is read. It does not change what a read costs. Before #4963 the first request after every eviction or window expiry paid that cost in full, on a page-facing request; now only the first read of a key does, and every re-read pays it in the background. So the same issue has a second half, and it is the one the 2026-08-26 comment did not name.

The listing was transferring the whole repository to read its manifests. ListPackages asked for an unfiltered snapshot — IGitHubRepoClient.Fetch(url, ref, subdir, token) — which GitProtocolRepoClient served as git fetch --depth 1 plus a full checkout, and then it kept the two files per plugin it parses and discarded everything else.

Measured against MeshWeaver.Plugins at 25dc26d7 on 2026-09-16:

Two readings matter in that table. The first is that 13 s lands exactly in the 12–19 s the production portals measured, which is what identifies the transfer as the dominant term rather than anything the endpoint computes per request. The second is that the parse was never the problem: reading all 4,995 files off a warm local disk takes 83 ms, so a filter applied while reading the worktree — which is what the client used to do — saves nothing a caller can feel.

Nor was it entitlement. The same comment blamed "entitlement per package and a mesh query on every request". That is true of the sibling BUNDLE index, which really does run InstalledPackages and HeldPartitions mesh queries per request — and false of this endpoint: IsGranted is caller.Allows(...), pure in-memory over a grant the authenticator already resolved, and Artifacts(hub) is NoPublicationArtifacts on every deployment in the fleet, i.e. Observable.Return([]). A measurement taken on one endpoint had been carried to another.

How the narrow read works

IGitHubRepoClient's filtered Fetch overload has always stated this contract — "downloads ONLY the blobs whose path satisfies pathFilter … without pulling the rest of the repo" — and OctokitGitHubRepoClient honoured it. GitProtocolRepoClient did not: it fetched everything and filtered at read time, on the reasoning that the git protocol costs the same number of REST calls (zero) either way. True of calls, false of bytes. And nothing called the overload, so the listing never even asked.

The git-protocol client now selects before it transfers:

1. git fetch --depth 1 --filter=blob:none — a blobless partial clone. Commits and trees only, so the whole path list is known before one file's bytes move (1.5 s / 372 KB for the repo above).
2. git ls-tree -r --name-only -z FETCH_HEAD — that list, for free. -z because ls-tree quotes unusual paths otherwise, and a quoted path would be selected and read under a name the worktree does not have.
3. The caller's predicate selects from it, and the survivors become --no-cone sparse-checkout patterns, so the checkout materialises exactly those in one batched lazy fetch.

A filter that matches nothing skips the checkout entirely and moves no blobs at all. The worktree read still applies the same predicate, so the ANSWER is identical however the transfer went.

🚨 A remote that cannot serve partial clones does not fail — it warns and sends everything. uploadpack.allowFilter is a server capability; when it is off, git prints "filtering not recognized by server, ignoring" and exits 0 having transferred the whole pack. The answer stays correct, but the saving is gone, and an unannounced loss of it is exactly how this latency would return unnoticed. The client detects that line and says so in its log. Only a hard fetch failure takes the fallback branch, and a failure of the fallback propagates — nothing is swallowed.

🚨 The INSTALL read is narrowed too — to the package's whole folder, never to the listing's predicate (MeshWeaver#5826). It used to call the unfiltered Fetch with no subdirectory and keep <Plugin>/ out of the snapshot afterwards, so every POST /api/plugins/files cost the registry a whole-repository transfer of MeshWeaver.Plugins (47.8 MB / 13 s on its own) before the first byte of the answer — against the consumer's 30 s per-attempt budget. On memex.systemorph.com the boot's fetch for Anthropic (2026-09-27) and AppleIntelligence (2026-09-28) spent all three attempts waiting for response headers (HttpConnection.InitialFillAsync) and the 90 s pipeline failed the package. The reason it had been left whole — "narrowing that one would install an empty package" — holds for the listing's manifest-only predicate and not for one that keeps the folder: NodeRepoPackageSource now asks for path.StartsWith("<Plugin>/") (and, on the manifest-diff fast path, only the named paths inside it), GitHubPackageSource for everything under the package's subdirectory. The answer is byte-identical; a client with no narrow implementation falls back to the old fetch-then-filter. The unfiltered Fetch is untouched and still transfers everything, which is what a content sync wants. PackageListingCacheIsArmedTest pins that the install takes the narrow read AND returns the whole folder, binary blob included, and nothing from a prefix-sharing neighbour. GitProtocolNarrowFetchTest pins both halves, and pins which path ran rather than only what it answered: a narrow fetch that silently regressed to a whole checkout still returns the right files, so asserting the files alone would pass over the defect the test exists to prevent.

Why not simply raise the client budget

The comment that already sits in ServiceDefaults says it, from 2026-08-26: "a registry this slow to serve 8.7 KB is its own defect and wants caching". The budget raise it documents was needed — a client whose budget cannot cover the server's observed latency keeps failing after the server improves — but the listing's inputs change at the cadence of a merge, not of a request, and reading them per request is the defect.

Where this sits

Plugin Bundles in the Registry — what the registry serves beside the listing. Controlled IO Pooling → "Promise-cache for idempotent one-shots" — the PromiseCache contract this uses, including why a fault must evict. No Static State — why the cache is an instance on a mesh-scoped singleton.