The sync/ Hub Population

A 26 h production replica held 9 398 MessageHub instances, of which 6 925 were sync/ hubs at RunLevel=Started — about 2.7 GB of a 3.5 GB live heap (#3432). This page is what the code says about where they come from and what is supposed to take them away. It is a narrowing, not a diagnosis: the sections marked 🚨 NOT ESTABLISHED are still open.

🚨 The cause was established on 2026-09-10, by experiment on a live replica: the READ path minted one sync/ hub per GetDataRequest. See The Read Path Minted a Hub Per Read, which also carries the live census that decomposes the population into its holders — and re-scopes the evicted-stream mechanism to 1 parked stream in 475.

1. A sync/ hub is not a population — it is one field of a stream

SynchronizationStream<T>'s constructor mints exactly one, unconditionally, for every stream:

// src/MeshWeaver.Data/Serialization/SynchronizationStream.cs
var syncHub = Host.RunLevel > MessageHubRunLevel.Started
    ? null
    : Host.GetHostedHub(
        SynchronizationAddress.Create(ClientId), ConfigureSynchronizationHub, HostedHubCreation.Always);
…
Hub = syncHub;
syncHub.RegisterForDisposal(_ => ReleaseHub());

And one cross-hub subscription builds two streams, hence two hubs:

Both carry the same ClientId, so the two hubs share an address string while living under different parents.

This settles the first of #3432's open questions. The dump's own arithmetic agrees:

6 925 Started sync/  +  1 495 Dead sync/  =  8 420
SynchronizationStream total                =  8 461     (0.5 % apart)

So there is no separate "sync hub leak" to explain. The question is entirely "why are there 8 461 live SynchronizationStreams", and each one costs its own Autofac ILifetimeScope and TypeRegistry (~390 KB, of which ~173 KB is framework metadata duplicated per hub).

2. What retires one — three paths, and only one of them applies here

🚨 This table is about the STREAM's lifetime. The registry-level account — which collection roots a sync/ hub, what takes it out, and which (RunLevel, IsDisposing) states can persist there — is §8 below, written against the code after the retire-and-replace change, and it is the half that says which of these hubs cannot be removed versus is removed only when X happens.

The middle row is #3321 step 3, and it is why the Dead bucket exists at all. It does not touch a hub whose parent is alive. So for a Started sync/ hub under a live parent — which is this issue's entire 6 925 — the idle sweep is the only reaper in the process.

MeshNodeStreamHandle.DetachUpstreams() is reachable from nowhere else, and it is also the only thing that collects the change-feed-parked predecessors described below.

3. The sweep's predicate, and why traffic defeats it

// src/MeshWeaver.Hosting/MeshNodeStreamCache.cs — Entry
public bool TryMarkIdleEvicted(TimeSpan idleWindow)
{
    lock (gate)
    {
        if (evicted || subscribers > 0
            || Environment.TickCount64 - lastActiveAt < (long)idleWindow.TotalMilliseconds)
            return false;
        evicted = true;
        return true;
    }
}

Window 10 minutes, swept every minute (MeshNodeStreamCacheOptions). Two conjuncts, and the option's own summary states what refreshes the second: "Every GetStream subscription, unsubscription and Update on the path refreshes the window."

🚨 The subscribers count is NOT the Rx-subscriber trap Workspace warns about. That warning ("the reduce chain subscribes to the stream ITSELF, so an evicted stream measures 2–3 subscribers and never reaches zero — implemented, measured and reverted") is about a different, abandoned mechanism at the workspace layer. Entry.subscribers is a purpose-built refcount moved only by TryAddSubscriber/RemoveSubscriber from the cache's own SharedView, and the internal hydration subscription bypasses it. It genuinely can reach zero. Anyone reasoning about #3432 from the Workspace comment alone will mis-attribute this.

What actually holds the sweep off is the other conjunct.

4. The change-feed orphan — the write both creates it and postpones its collection

🚨 Since #1174 this runs only for change events no mirror can be held to (a delete, a recreate, a version-less recycle broadcast); a versioned Updated keeps the stream — see Live Mirrors and the Change Feed. The measurement below predates that.

Workspace.EvictForPath runs on every change-feed event for an owner path and removes every identity's cached stream for that owner:

_evictedRemoteStreams[removed.Value] = 0;
ReclaimIfUnheld(removed.Value);        // disposes ONLY if no declared lease remains

The mesh-node cache's hydration is a declared holder for the entry's whole life — that is deliberate, and MeshNodeStreamHandle.Subscribe says so: "The lease lives exactly as long as this subscription. The shared mesh-node cache's hydration comes through here, so its entry IS the declared holder of the path's upstream for the entry's whole life — which is what makes every OTHER (write-scoped) stream for the same path reclaimable on eviction."

So on the first write to a cached path:

1. the entry's stream S1 is removed from _remoteStreamCache and parked;
2. ReclaimIfUnheld(S1) finds the hydration's lease and does not dispose it;
3. nothing re-points the hydration — the sweep "only ever CLOSES idle entries; it never re-subscribes anything (the 2026-06-08 rule)" — so the entry keeps reading S1 forever;
4. the next reader or writer misses the cache and builds S2, which becomes the cached one.

From then on the path holds two live streams — four sync/ hubs, ~1.5 MB — where the design intends one. Write-scoped streams still churn correctly (an unleased evicted stream is reclaimed at once), so the accumulation is a floor of one orphaned pair per cached, written path, not growth per write.

🚨 And the same act does both halves: the Update that fires the change-feed event which orphans S1 is also an Update that Touch()es the entry, pushing the 10-minute window out. A path written at least once every ten minutes can never be released, so its orphan can never be collected. The only reaper is switched off by the traffic that feeds it.

That is the answer to "is the retirement path missing, or present-but-not-firing?" — present, and structurally unable to fire on exactly the paths that generate the garbage.

5. Bounding

Neither cache has memory-pressure eviction or any other TTL:

• Workspace._remoteStreamCache — a plain ConcurrentDictionary keyed by (Owner, Reference, Identity), so a path read under N identities holds N streams.
• MeshNodeStreamCache._streams — a plain ConcurrentDictionary keyed by path. (The MemoryCache in the same file is the write-queue's, not this one's.)

So the live-stream count is bounded only by (distinct paths × identities) touched since process start, with the per-path floor above added on top.

6. 🚨 What is NOT established

🚨 Partly superseded. The Evicted-Stream Retention closes the "what holds them" half for one named mechanism, with a controlled two-arm test rather than a dump: a change-feed eviction parks a remote stream, and ReclaimIfUnheld returns without disposing whenever the stream carries no lease entry at all — so every unleased call site (including LayoutExtensions.GetControlStream, i.e. every rendered layout area) retains one stream, and two sync/ hubs, per change event. It also rules out static state and timers as roots, and replaces the referrer walk below with two field reads on the singleton Workspace. What stays open is the magnitude, not the mechanism.

• Whether the 8 461 are garbage. A portal with thousands of open Blazor circuits, each holding live layout-area streams, is supposed to look like this. Sections 1–5 explain the shape of the population, not that it is waste.
• The split between the per-path floor of §4 and legitimately-live streams. That is the number that decides whether #3432 is a leak with a fix or the per-hub DI cost the issue's own third outcome row describes.
• Whether #3427 moved this number. Still not re-measured; no deployment carried f41f8bda at the time of the last attempt.

7. The discriminator — a ratio, from data the existing plan already collects

#3432's plan calls for a ClrMD referrer walk, which is the expensive half and needs a dump that (measured) restarts the replica. §1 makes a much cheaper reading decisive, and it comes off dumpheap -stat, which that plan already takes:

count MeshNodeStreamCache+Entry alongside SynchronizationStream.

No referrer walk, no field traversal, no second dump — one extra line of a histogram the plan already produces. 🚨 It does not replace the referrer walk for the third row; it tells you whether you need one.

8. The registry account: what roots a sync/ hub, and what takes it out

Sections 1–7 are about the STREAM. This section is about the HUB, read off HostedHubsCollection and MessageHub as they stand today. It answers the question the dump's histogram poses — what can a Started sync/ hub be? — by construction rather than by inference, and it distinguishes cannot be removed from is removed only when X happens.

8.1 One root, and it is the parent's registry

A sync/{clientId} hub is a HOSTED hub. There is exactly one creation site (§1), and every hosted hub enters its parent's registry through one method:

// src/MeshWeaver.Messaging.Hub/HostedHubsCollection.cs — Track
private void Track(IMessageHub hub)
{
    messageHubs[hub.Address] = hub;
    hub.RegisterForDisposal(h =>
        messageHubs.TryRemove(new KeyValuePair<Address, IMessageHub>(h.Address, h)));
}

messageHubs is an instance ConcurrentDictionary<Address, IMessageHub> on the collection, whose lifetime is the parent hub's. There is a second root beside it, retiring, a reference-keyed dictionary of hubs taken out from under their address but still tearing down; Hubs enumerates both, which is why a census walking Hubs sees a retired hub too.

8.2 Three removals, all of them self-driven

Nothing else removes a hosted hub from its parent's registry. There is no sweep, no idle eviction, no cap, and nothing anywhere asks "is this hub still needed". Removal 1 is the hub disposing itself; removals 2 and 3 only move a hub that is already disposing. So the registry is a pure consequence of who calls Dispose(), and for a sync/ hub that is: the stream's own Dispose(), the parent's teardown, or a routed DisposeRequest — §2's table.

🚨 Removal 2 is lookup-triggered, not self-triggered. Retire-and-replace made the hand-out safe; it did not make the registry self-cleaning. A hub wedged at ShutDown that no Always lookup ever asks for again stays in messageHubs for the parent's life. That is #4883's wedge showing through, deliberately left there.

8.3 The (RunLevel, IsDisposing) cross-tab — which cells can persist

IsDisposing is disposalStarted, set as the FIRST statement of Dispose(), before it posts ShutdownRequest(Quiescing). The RunLevel only moves when the action block dequeues that request.

8.4 A second retention class the histogram cannot see: the registrant composite

A hub's registered cleanups live in one CompositeDisposable:

// src/MeshWeaver.Messaging.Hub/MessageHub.cs
private readonly CompositeDisposable disposables = new();
public IMessageHub RegisterForDisposal(IDisposable disposable)
{
    disposables.Add(GuardRegistrant(disposable));
    return this;
}

A plain registration is append-only. CompositeDisposable.Add never prunes, and nothing removes an entry added through RegisterForDisposal, so such a registrant is held — with everything its closure captured — for that hub's whole life. The ONE way out before the hub dies is a DETACHABLE registration (RegisterForDisposalDetachable, or SubscribeHeldUntilTerminal built on it): its handle removes the entry without disposing it, and it is to be used only once the registrant's disposal can no longer do anything (see the table below). A PLAIN registrant whose own subject is SHORTER-lived than the hub is therefore a monotone root, and it retains an object graph rather than a hub, which is why no MessageHub histogram can see it.

MessageHub.DisposalRegistrantCount is the reading that can, and it is printed in both disposal diagnostics as Registrants=. A hub whose registrant count climbs with the traffic it has served is retaining one object graph per unit of that traffic.

Two classes, both measured in MeshWeaver.Data.Test:

🚨 Still held by design, and NOT a leak this fix touches: a GetDataRequest over a LIVE workspace reference keeps shipping every change to its requester, so its subscription — and its registrant — live until the owner goes down. Whether anything should end such a read earlier (the requester's own Observe completes on the first answer) is a separate question, not measured here. A read that ends EMPTY on a healthy hub also stays registered on purpose: its caller is still owed the teardown NACK.

9. The 2026-10-11 census: every sync/ hub is the fixed complement of a live owner

Three in-process, read-only censuses on images carrying every fix this page names (core e438ab89e7, image 3.1.10517). They walk the live hosted-hub tree by field reads and resolve nothing from DI. This is the reading #3432 was closed on, and it is the issue's own third outcome row: the sync/ population is per-hub cost, not retention of sync/ hubs. No sync/ hub was found outliving or lacking its owner. That moves the lifetime question to the owners and does not answer it: whether the node hubs themselves retire once idle is NOT established here (end of this section), so this reading does not say the owners are all needed.

Attribution, instance A (the v7.2 census, identity exact — a stream counts only when its ClientId equals the hub's Address.Id): sync=228 attributed=228, every other bucket zero. All 228 read Started / <not started> except one Starting. Each is one of the owner hub's own data-source reduces (MeshNodeReference, CollectionReference, CollectionsReference on host=X, owner=ds/X) or one mesh-node cache hydration (host=cache/…, 68 of them). The duplicate discriminator reads streams=228 distinctPairs=197 excess=31, and every duplicate group reads refsEqual=YES hash=SAME, the configured (deliberately uncached) branch. No group reads refsEqual=NO: the reference that could not be a cache key (ContentCollectionReference, 24 of the 80 excess on 2026-09-16) is gone.

Shape, instance B, the busy replica — sync/ hubs per parent:

980 of 1 047 parents hold exactly three or four. A hub that accumulated streams per read, per write or per subscriber would stand out of that histogram, and the only parent above twelve is the cache hub, whose count is its entry count. So the number of sync/ hubs is bounded per live node hub — three or four for 980 of the 1 046 non-cache parents, at most twelve for any — plus the cache entries. The 66 parents outside three-or-four (one with a single child, 65 with five to twelve, 354 hubs) were NOT attributed stream by stream on this replica; the per-stream attribution was run on instance A, where the whole excess over the base set (31 of 228) was the configured, deliberately uncached branch. That is the likely reading of the 65 here and it is an inference, not a measurement.

🚨 The working hypothesis is that the heap follows the number of live node hubs, and that on the first replica of a generation the boot sets that number. It is a correlation across two replicas, not a root attribution. Of the 1 059 node hubs on 8f2g5 thirty minutes after it started, 930 fall in buckets that are not request traffic: 403 are two-segment addresses (NodeType definitions, the bake), 128 are partition roots, 115 are …/_Policy, 115 …/_Entitlements, 43 …/_GitSync, 82 Admin/Build/…, 44 MeshWeaver/samples/…. The other 129 were not classified. Its sibling, booted ninety seconds later and serving the same instance, holds 101 node hubs. The two readings differ by 958 node hubs and 4 260 MiB of last-GC heap, about 4.4 MiB per node hub (its own scope, type registry, workspace and its three or four sync/ hubs would be in that figure). That fan-out is the candidate for the multi-GiB [HEAPSTEP] series every roll prints in its first minutes: the sampled allocation of those steps is System.String, System.Byte[], Entry[System.String][] and Autofac's IResolvePipelineBuilder handlers and MiddlewareDeclaration, which is what hub construction allocates. [HEAPSTEP] names allocated types, not retained roots, and this section has neither a quiet-process root attribution nor a before/after on one process, so the per-hub figure and the link to the heap step are to be confirmed, not cited.

🚨 NOT established: whether those boot-activated hubs retire once idle. The replica was replaced by the next roll 20 minutes after the reading (the instance rolled twice inside one hour), so no second reading on the same process exists, and at 33 minutes an idle activation is only just past the keep-alive window plus the collection age. The 2026-09-25 reading on a 30 h process (post-GC floor level at about 3.3 GiB from hour 18) is the newest evidence that the population plateaus. A replica that lives a day is the instrument; HubCensusByOwner (a Code node in the operator's own partition, the script behind the tables above) is the reading to repeat on it.

The standing census names the satellite shape (#6632)

meshweaver.hubs.live (PlatformMetrics) is tagged kind (the address type), satellite and runlevel. satellite is the LAST address segment that starts with _ — _DocumentPart, _Activity, _Thread — or none; never an id. The 2026-10-04/05 OOM burst was ~5,000 Admin/content/_Documents/…/_DocumentPart/{index} hubs, which the kind tag alone folded into kind=Admin; with the second tag the next burst names its address shape while it happens. The bound is the meter's own, and it holds for the meter's LIFETIME: at most MaxSatelliteValues (32) satellite values are ever named. A value is admitted the first time it is among the most populous of a scrape while a slot is free and keeps its name afterwards; every value never admitted folds into _other. A per-scrape top-N would bound one scrape only — values rotating through the top would each leave a series behind in the metric store. Nothing enforces that a _ segment is a declared satellite, so cardinality is not left to the deployment (PlatformMetricsTest.SatelliteValues_StayBounded_AcrossScrapesThatRotateTheirValues).

10. Which boot step builds each class of node hub (#6623)

The rule: a boot step that only needs a node's CONTENT reads it without activating that node's per-node hub. GetMeshNodeStream(path) is the authoritative LIVE read, and it gets there by subscribing at the node's own address, which builds the hub. For a one-shot read of many nodes the store answers with the DURABLE snapshot: exactly what the next activation would load. StoredNodeReads.ReadStoredNode / ReadStoredNodes (MeshWeaver.Mesh.Contract) is that read: storage first, the served static node second. It is the same order ReadNodeAuthoritative has always used for the partition bootstrap, and the access-rule gate's ReadSubjectNode now delegates to it.

🚨 The durable snapshot is not the live record. Where an owner is live, the row trails it by one 200 ms persistence-sampler window plus the write, and with no bound after a store write that FAILED: MeshDataSource.WriteSampledNode logs a faulted sampled save and does not retry it, and PendingOwnSave counts the accepted request as saved, so the owner stays ahead of its row until its next change is written (and a recycle loses the acknowledged state — an owner-side durability defect, separate from this read). So a caller may use this read only where its decision is safe on a trailing row, and says why. The two GitSync sweeps are: BundleKeyedHold.Decide holds only where it REPRODUCES the definition's fingerprint from the current Code nodes, so a trailing fingerprint abstains, and a row trailing only in provenance either misses a hold (the documented StaleAdopted direction) or holds a type's sources in place (nothing written or deleted) until the row catches up or the shelf releases the hold; SyncedPartitionDrift.Measure either misses drift (abstain) or reports drift that is gone, whose action re-imports at the commit the source already claims and moves nothing backwards (NodeTypeDefinitionReads carries the full argument).

The step → call → class mapping from reading the code. The census counts address shapes only, so each attribution below is an inference from code, not a per-address measurement:

Pinned by NodeTypeSweepActivatesNoHubTest (MeshWeaver.Documentation.Test). It creates twelve NodeTypes, runs the sweep's read and watches HostedHubsCollection.HubAdded: no hub is built. The negative control runs the replaced GetMeshNodeStream read over the same types with the same instrument and sees twelve, one per type.

Not established: that the 403 two-segment hubs on 8f2g5 were built by these two sweeps rather than by enrichment watches or by plugin-side code. A per-address census on the first replica of the next generation answers that. The same goes for whether boot-activated hubs retire once idle (§9).

Related

• Stream Liveness and the Hub Reference — #3321 step 3, the parent-teardown hook that reclaimed the Dead half
• MeshNode Stream Cache — the entry, the refcount and the idle sweep
• Portal Heap Is Hubs — the five dumps, the measurement hazard, and the ClrMD recipe
• The Evicted-Stream Retention — the named retainer, the controlled experiment that established it, and the two-field-read measurement that closes #3432