Reading a Silo Eviction

An Orleans silo that reads its own Dead row stops itself and says so:

fail: Orleans.Runtime.MembershipService.MembershipTableManager[100627]
  I have been told I am dead, so this silo will stop! Reason: I should be Dead according to
  membership table (in CleanupTableEntries): entry = [SiloAddress=S10.244.4.154:11111:147256594
  … Status=Dead … IAmAliveTime=2026-09-02 03:12:40.984 GMT
  Suspecters=[S10.244.3.122:…, S10.244.5.16:…, S10.244.3.122:…]
  SuspectTimes=[2026-09-02 03:12:40.394 GMT, …, …]]

Two details in that line look like proof of a false-positive kill, and both are misreadings:

Getting either wrong sends you to fix Orleans' failure detector, which is fine, instead of the process that stopped answering — and the process that stopped answering is still doing it.

Why a live heartbeat proves nothing

IAmAliveTime is refreshed by a single cheap row upsert on its own timer, and one gap between pauses is enough for it. A probe (IMembershipService.Ping) has to be answered inside its timeout, several times in a row. Orleans 10 answers it in the connection's receive loop (SiloConnection.HandlePingMessage), not in a grain turn (the second shape below turns on exactly that), but the receive loop is ordinary managed code on thread-pool threads: it does not run while the GC has the process suspended, and under pool delays it waits for a worker like everything else. A process deep in GC-stall territory therefore does the first and not the second, so during a stall the membership row keeps looking fresh while every probe times out. That is not a contradiction — it is the signature of a stalled host.

The victim's own log is where this is decided, and on 2026-09-02 it settled the question in one line. The heartbeat cited as proof of liveness — IAmAliveTime=03:12:40.984 — carries the same timestamp as:

2026-09-02T03:12:40.984  .NET Runtime Platform stalled for 00:00:06.5910746.
                         Total GC Pause duration during that period: 00:00:06.5384310.
                         We are now using a total of 11886MB memory.
                         Collection counts per generation: 0: 10594, 1: 5443, 2: 2880

The heartbeat did not land because the silo was healthy. It landed at the instant the process came out of a 6.6-second GC pause — one of the gaps between pauses, which is exactly enough for a row upsert and nowhere near enough for an RPC.

And the silo was measuring itself, too. Orleans' LocalSiloHealthMonitor, on the victim, minutes before the vote:

03:11:04  .NET Thread Pool is exhibiting delays of 10.8451907s
03:11:36  .NET Thread Pool is exhibiting delays of 38.1501219s
03:12:34  Self-monitoring determined that local health is degraded. Degradation score is 6/8

A silo with a 38-second thread-pool delay is, for failure-detection purposes, dead. The eviction was correct.

The control arm: what would falsify it

A rule that cannot say no is worth nothing, so this one has an explicit discriminator. Count the probe-failure targets across the suspecters' whole retained log:

az aks command invoke -g <rg> -n <cluster> --command \
  "kubectl logs <suspecter-pod> -n <ns> \
   | grep -oE 'Did not get response for probe #[0-9]+ to silo S[0-9.]+:[0-9]+' \
   | grep -oE 'S[0-9.]+:[0-9]+' | sort | uniq -c"

On 2026-09-02 the two suspecters had 12 and 59 probe failures between them and every single one named S10.244.4.154 — zero against any other silo, over 17 h and 9 h of retained log. That is not a shared stale snapshot and not a network event; it is one process.

Three further checks, each of which could have flipped the verdict and did not:

• Was a suspecter itself shutting down? (#3053's hypothesis.) kubectl get pod … -o custom-columns=…,RESTARTS:.status.containerStatuses[0].restartCount. Both suspecters had restartCount=0 and were still running — neither was stopping.
• Was a rollout in progress? kubectl get rs -n <ns>. One active ReplicaSet, 5/5 ready, 22 h old, every other scaled to zero. No surge, so no shutting-down peer to blame.
• Was it OOMKilled by the kernel? No — and this is worth reading carefully, because the exit code lies about the cause. lastState.terminated showed exitCode: 139, reason: Error, and the dump header reads signo 6 (SIGABRT), i.e. the Environment.FailFast from FatalErrorHandler — the self-kill the log already announced, not a segfault. See Debugging Native Crashes.

The kill is the cleanup, not the fault

The managed-heap trace is where the actual defect is:

No DOTNET_GCHeapHardLimit-family variable is set, so .NET takes 75 % of the 16 GiB cgroup limit → a 12 GiB managed hard limit. 11 850 / 12 288 = 96.4 %. That is why 43 managed OutOfMemoryExceptions appeared between 02:53:29 and 03:12:45 (the last one 7 s before the kill) while the kernel never OOMKilled anything — the GC refused to grow past its own ceiling with ~1.4 GiB of cgroup slack still free.

🚨 Raising the limit is the band-aid here. The heap grew ~6.6 GB in four hours; a bigger ceiling buys hours, not a fix. The OOM sites already filed against that window — #3044, #3045, #3046, #3049 — are unbounded message/response serialisation, which is both a consequence of the pressure and a plausible amplifier of it: an OOM raised inside ReportFailure while reporting an OOM is a cascade.

The same stall filed a second, unrelated-looking incident

A stalled host does not only fail Orleans probes. On this pod, in the same window, 89.4 % of all Npgsql.NpgsqlException lines were emitted within 100 ms of a GC-stall report — 294 of 329, in windows covering 1.77 % of the log's span (≈ 50× enrichment). Those became #3050 / #3051, both of which concluded "database-host unavailability" with high confidence. The database was fine; the connect timeout is wall-clock and does not pause for GC, so a frozen client produces the identical exception.

🚨 When a host stalls, expect a scatter of incidents that each name a different innocent subsystem. Before accepting any of them, check the pod's own stall lines in the same window. See An Unreachable Store Is Not a Refusal for that measurement and the reading rule it produces.

The second shape: a silo wedged from boot, with no stall at all

The 2026-10 recurrences (#6395 on memex, #6432 twice on memex-cloud) had no memory-pressure signature, so they reopened the membership question (procedure step 5 below). They resolve the same way, as a correct eviction of a silo that was not serving, but the evidence looks different.

The boot fingerprint. About 30 s after the silo starts, its own hosted client gets no answer from a grain placed on that same silo:

Response did not arrive on time in '00:00:30' for message: 'Request
[S<ip> sys.client/hosted-<ip>]->[S<ip> messagehub/Agent/_Activity/import-manifest] … DeliverMessage'

The first minute then reads the same way every time. GetMeshNode('Agent/_Activity/import-manifest') timed out … NO LOCAL HUB … it never activated here, the platform-startup CreateNode reports "post-creation handlers had not finished within 30s", and the static-repo import fails partition by partition. The doomed process also never logs Application started: on the pod read in full (2026-10-10 18:10Z), the last hosted service to log its start did so 5 ms before that CreateNode was written, and the process that replaced it logged Application started 40 ms after the same point. From the image that carries MeshWeaver#6391's refinement on, that CreateNode line also says what the leg was waiting at (Write Verdict Totality). Only the … matching <handler> [module <name>, registration <i>] form names a module container the create could not get an answer from. post-creation-handlers: resolving names none: it says the handler registrations were still being constructed, which narrows the wait to that construction and no further. Measured: 3 of 3 evicted pods carried it. None of the healthy new pods in the same rolls did; they created their import manifest within 3 s. The query is did not arrive on time.*_Activity/import- over the roll window.

Threads are consumed, not starved. [LIVENESS] on the doomed pod keeps a 10.00 s gap and a flat GC, so it shows no pause and no thread-pool stall, and LocalSiloHealthMonitor stays quiet. But poolThreads climbs steadily (20 → 55 over 16 minutes on 5j9gx) while the work completed per 10 s collapses (about 500 against about 47 000 on a healthy pod in the same roll, which ran on 16 threads). That is what the counters establish: workers stay blocked long enough for the pool to keep injecting new ones. It is not established that any individual thread never returns. poolThreads is the number of workers the pool retains, so it stays high after a worker comes back, and the completed count never reaches zero, so some work does finish. Whether the blocking is permanent, and on what, is the question the heap dump below answers.

Why it takes minutes, not seconds, to be evicted. Orleans 10 answers a membership ping inside the connection's receive loop (SiloConnection.HandlePingMessage). Every other message goes through MessageCenter.ReceiveMessage, which runs synchronously on that loop and calls Catalog.GetOrCreateActivation, where the grain is constructed. A silo whose activation or dispatch path blocks therefore keeps passing probes on every connection whose loop has not yet hit a blocking message. Meanwhile it answers nothing else in either direction, writes IAmAlive to the membership table normally, and logs no connection error. The measured eviction times were 8, 15 and 20 minutes. A silo failing probes from the start would be gone in about 2 to 3 minutes under the deployed ProbeTimeout of 15 s and NumMissedProbesLimit of 5.

Those two values are the portal host's own (Memex.Portal.Distributed, MeshWeaver.Plugins, with indirect probes enabled), in its code since before any of the three incidents, #6395 included; the running options were not read back from the pods. Core's ClusterMembershipTolerance baseline (10 s × 3, added for #6395) is inserted first so that a host's explicit configuration wins, so it changes nothing on these deployments; it applies to a host that configures no membership options. #6395 was therefore not a silo evicted after 15 s of Orleans defaults. It was this shape, evicted under the wider vote, and no tolerance setting addresses it.

So "requests reach it, nothing comes back" is not, by itself, evidence of a broken pod network path. Check the thread trend first.

The scatter it files. Peers see 30 s timeouts against the silo: a stream RegisterConsumer against a rendezvous grain on a remote silo (#6429) and grain-call cancellation batches that all target one silo (#6392). Each names a different innocent subsystem, as in the stall case above. Two more incidents from the same roll windows are correlated, not established: a placement timeout (#6394) whose sample does not say which silo was chosen, and a memory-stream dequeue that timed out while 5j9gx called a queue grain on another silo (#6431). Neither shows that this silo received a response and failed to dispatch it. Count them as the same failure only once the heap dump or the remote silo's own log shows the same path.

What is still open. Which call blocks is not in any log. The instrument is the FailFast heap dump the self-kill writes (step 4). Read it with the governed AnalyzeDump action (Debugging Native Crashes, "Reading a production dump"): filed on the control instance against Deployments/<id> with dumpPod or dumpAround naming the evicted pod, it runs threadpool, syncblk, clrthreads, clrstack -all and dumpasync in the cluster and lands the grouped stacks at Ops/Dumps/<action id>. No direct cluster access is needed.

A self-kill does not always leave a dump, so check before filing. The verdict is the last [crash-dumps] line of that run of the script (init on a pod's first start, postStart on every start): each run ends on its arming outcome. Earlier lines are retention (parked, deleted, kept, and ERROR: could not park … or could not delete …), after which the script carries on, so neither the first line nor an arbitrary ERROR is the verdict. As the last line, armed means the next crash writes a dump; HELD: dumps are DISABLED means it wrote nothing, which is what every pod logs where the shared claim is smaller than the dump gate requires; and a final ERROR says what stopped the script: dumps REMAIN ARMED, no dump can be written, or a MEMEX_CRASHDUMP_ROOT it refused to touch, which changed nothing and must be read as "not established", never as armed. An instance whose dump root is still an emptyDir lost the dump with the pod. In both cases there is nothing for AnalyzeDump to read from that incident: fix the storage first (declare a dumps volume on the record and Reconcile; see Debugging Native Crashes), and the next recurrence is the one that can be read. In the result, look for receive-loop threads (SiloConnection, MessageCenter.ReceiveMessage) or activation threads parked on one lock or one Wait. Until it has been read, any tolerance change or watchdog is a band-aid over a blocking call nobody has seen.

What the first dump said: one container lock, and Orleans queued behind it

The first read (Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-2 on the control instance: the pod evicted on 2026-10-10 at 18:16Z, 91 managed threads) names the lock, and with it why the probes failed. It does not yet name the call that holds it.

• syncblk shows two held monitors and no other. Monitor #2, a System.Object created in the process's first moments, is owned by OS thread 0x1e, one of the first pool workers, with MonitorHeld 43: 21 threads waiting. Monitor #169 is owned by thread 0xf1 with 4 waiting.
• Every waiter shown is in the same frame: Monitor.Enter under Autofac.Core.Lifetime.LifetimeScope.CreateSharedInstance. That is the lock a container scope holds while it constructs one of its single-instance services, and it is one lock per scope, not one per service. The owner of #169 is itself such a waiter: it holds one scope's lock while it constructs a service and waits for another scope's lock for that service's constructor argument.
• The waiters are the whole process, not one feature. The mesh's own: the post-creation handlers of three creates, two MessageHubGrain activations resolving their hub configuration, a node-stream update, the provider credential seed, ASP.NET's route matcher. And Orleans' own message path: threads in MessageSerializer.Write and ReadBodyObject constructing the codec for IndirectProbeResponse, for the grain-call cancellation system target and for the grain directory, each through ActivatorUtilities into the same lock. A silo that cannot construct the codec for a probe response cannot answer the probe. The thread pool agrees with the liveness lines above: 35 workers, 30 running, CPU 0 %.

So the eviction was correct for a concrete reason: the root container's single-instance lock was held for the life of the process, and everything that had not been resolved before it was taken, Orleans' codecs included, waited on it. IAmAlive kept being written because that path had been constructed already.

Not established: what thread 0x1e was doing. Its stack was among the groups that read cut "to fit" (the oldest thread, dropped first), so whether it waits on monitor #169 (two scopes locked in opposite orders by two threads) or on something else while it holds #2 is not known from that read. The read was repaired for exactly this: a thread syncblk names as a monitor's owner is now always shown, first, with its whole stack. The next AnalyzeDump of the same dump is the reading that names the call. The lock order is worth knowing before it arrives: since each module's services live in a container of their own (Live Module Update), a module service constructed there may take a root service as a constructor argument (module lock, then root lock), and a root service under construction may ask for a module's service (root lock, then module lock). Both directions exist by design; whether they are this incident's two threads is what the owner's stack decides.

What the second read said: a scoped service built in the root

The repaired read of the same dump (Ops/Actions/analyzedump-memex-cloud-20261011-ltlwc-3) shows both owners whole, and the pair is NOT the root and a module container. It is the root and the mesh hub's own scope:

PortalApplication is a SCOPED registration. The module container's fallback resolved every root registration in the root scope itself, so a scoped service asked for from a module scope was built — and then shared, for the life of the process — in the root, under the root's lock, and its constructor reached into a child scope. Before modules had containers of their own (core #6127) the same call resolved it from a root child scope, under that scope's lock, which is why the first wedge of this shape is two days younger than #6127.

The fix: the fallback resolves a root registration in a root child scope owned by the asking module scope and disposed with it (RootResolutionScope). A singleton is still the root's one instance; a scoped service is that scope's; a transient's disposal ends with the module scope. AScopedRootServiceIsNeverBuiltInTheRootTest pins it with this dump's two threads forced to overlap, and a negative control that builds the scoped service in the root and deadlocks.

The sibling shape, found while reading this dump and fixed separately (MeshWeaver#6542): a root SINGLETON whose construction calls a module's service (root lock, then module lock) while a module singleton's construction takes a root singleton (module lock, then root lock) deadlocked the same way. It was reproduced with real containers; no dump has shown it. Taking the root's lock first on the module side cured the repro and was rejected — a module constructor then ran under the root's lock, so one slow module constructor stalled every first-time root resolve, which is this incident's shape again. The fix is built before published: every shared instance of a module's container is built before the root can reach it — at boot before the root provider is handed out, on a live swap before it commits — so the root only ever reads a module's built instances and its lock never waits on a module's (Live Module Update → "Built before published"). RootAndModuleSingletonsBuildInOneLockOrderTest forces the two first-time builds to overlap; its negative control publishes the same container unbuilt and deadlocks.

Procedure

1. Read the victim's own log first, not the membership table. LocalSiloHealthMonitor complaints, .NET Runtime Platform stalled, Thread Pool is exhibiting delays, and the managed heap number those lines carry. A degraded self-score is the silo agreeing with its peers.
2. Count the probe-failure targets on each suspecter (above). One target ⇒ the target. Many ⇒ membership/network.
3. Check the suspecters' restartCount and the ReplicaSet state before believing any shutting-down-peer or rolling-deploy story.
4. Read the signal, not the exit code — signo 6 in the dump header is a FailFast, not a crash.
5. Count the fingerprint's occurrences. One occurrence with a full memory-pressure signature is an incident about the memory, not about membership. A recurrence without that signature is the sample that reopens the membership question — name it explicitly rather than closing on "could not reproduce".
6. No stall and no memory pressure? Look for the boot fingerprint (above): unanswered _Activity/import-* deliveries to the silo's own grains in its first minute, and poolThreads climbing while throughput falls. Then read the FailFast dump (AnalyzeDump) for the blocked threads before touching membership options.

See also

• The Pod-Hub Claim Must Be Re-Asserted — what a membership change costs the mesh once a silo really does leave
• Pod-Hub Delivery Roll Plan — the stranded-address shape during membership churn
• Debugging Native Crashes — reading exit codes, createdump, and why 139 is not proof of a segfault
• Error Propagation & Wedges — the accumulation shapes that grow a heap like this