Roll churn is recorded, not ticketed

Systemorph/MeshWeaver#4802 (canonical), #2906 (the decision it implements), #5300 and #5336. In one day, 2026-09-23 on the public instance, the same Orleans log site filed #5597, #5598, #5602, #5606, #5607 and #5619 — on top of the dozens already folded into #4802.

The site

Orleans' memory stream keeps each queue in a MemoryStreamQueueGrain that lives in the RAM of one silo. When that silo leaves, every PersistentStreamPullingAgent reading the queue logs fail: —

• Orleans.Runtime.SiloUnavailableException once membership has declared the silo gone, or
• a 30 s System.TimeoutException on IMemoryStreamQueueGrain.Dequeue while it is still stopping,

under Orleans.Providers.MemoryAdapterReceiver…Memory.memory-N or Orleans.Streams.Memory — and then recovers on its own once the queue re-activates on a surviving silo. Orleans owns the line and its level; there is no MeshWeaver frame on it. The real remedy is to retire AddMemoryStreams, and that is blocked on the stream's two remaining production producers (see #4802's 2026-09-22 comment), so until then every roll ticketed a fault nobody can act on.

Three more sites, the same shape

The rule below was first written for the memory stream alone. Three other Orleans sites have exactly the same shape — logged at fail: against a peer silo that is leaving, retried by Orleans itself, no MeshWeaver frame — and kept reopening tickets on every roll (#2906, #5300, #5712, #5336):

The last row joined on a measurement: on the public instance, 2026-10-10 19:10Z to 2026-10-11 11:10Z, that line was logged 62 times in nine bursts at nine different instants, while the sites of the two rows above it were logged zero times. Only the connection-refused shape is covered. The same event with a placement TimeoutException (the directory lookup hanging against a silo that is mute but still listed) is terminal for the sender and is never classified.

The classification, chosen on the standing instruction to follow the issues' recommendation: churn expected inside a declared roll window is recorded with a count and not ticketed; anything outside a roll, or persisting past it, keeps filing as an incident. "Inside a roll" is the same observable as for the memory stream — a shutdown line of the same workload in the five minutes before — so the two failure modes those issues said must stay visible still file: the stalled-but-Active silo (#2906's 2026-09-01 and 2026-09-24 samples) and the silo that died without shutting down, leaving its row Active (#5300's open question). A silo voted dead by its peers (#6395) is never classified, in or out of a roll — that is a failure, not churn.

The rule — and what it refuses to classify

A burst is roll churn only when both hold (RollChurnClassifier, in MeshWeaver.Observability.Contract):

1. it is one of those sites (RollChurnClassifier.IsRollChurnSite) — for the memory stream: a memory-stream reader category, a SiloUnavailableException or TimeoutException, and a call to a memorystreamqueue/ grain; for the other three, the shapes in the table above; and
2. a pod of the same workload (the pod name without its ReplicaSet hash and id) logged Application is shutting down... within five minutes BEFORE the burst.

The window is derived, not tuned: the host's ShutdownTimeout (90 s, the silo holds its queue grains until it has stopped), a Dequeue in flight to it timing out (30 s), membership declaring an unclean stop dead (ProbeTimeout 15 s × NumMissedProbesLimit 5), plus the agent's retry of that read.

🚨 The same failure with no preceding exit keeps filing. That is a silo that stalled, or was killed without shutting down — an OOM kill logs no shutdown line — which is exactly the case #2906's analysis said must stay visible. Measured on the public instance on 2026-09-23 (Logs actions Ops/logs-memexcloud-20260924-memstream-rollchurn-shutdown and …-started):

The 18:11Z pair is the instructive one: pod …-4d84f9-n7g6b logged a fresh Application started at 18:13:12Z with no shutdown line before it — it was killed, not stopped. The classifier leaves those lines on their ordinary fingerprint, where they are the only red evidence of that kill.

Recorded, never dropped — and never able to hide a steady-state occurrence

A classified burst is not discarded. It goes to its own incident, Admin/_LogIncident/roll-churn-{siteFold} — one per log site — which the portal records as Suppressed from its first sighting, with classification: RollChurn and the evidence that classified the latest burst ("<pod> logged 'Application is shutting down...' at <t>, N s before this read failed"). It is counted and sampled like any incident, and it asks for nothing, so no triage round and no issue.

Because it is a different node, a later occurrence of the same site outside a roll cannot fold into the suppressed incident and disappear: it keeps the site's ordinary identity and is triaged as before. That is why the classification changes the fingerprint rather than the status of the existing incident — suppressing the ordinary incident would have silenced both.

The watcher also logs one Information line per classified report, naming the fingerprint and the evidence, so the watcher's own log shows what it did not triage.

Where the exit comes from

The watcher's query is deliberately unfiltered, so the host's own Microsoft.Hosting.Lifetime line (Information in the portal's appsettings.json) is in the same stream as the failure. A pass reads one page per poll interval and the failure trails its exit by up to minutes, so the watcher carries the exits it has seen forward, per namespace, for one window. They are held in memory by design: a watcher restart forgets them, which can only make a churn burst file as an ordinary fault — never hide one.

The contract is additive in both directions: an older watcher sends no classification and everything files as before; an older portal ignores the field, and the roll-churn-… incident is triaged as an ordinary one.

An incident opened before the portal knew the classification is retired

The watcher can be rolled AHEAD of the portal (Plugins#3512). In that window the watcher stamps a classification the running portal does not know, and the portal opens the incident New, asking for Triage — the classification itself is still stored on the node. Suppression used to be decided only at creation, so after the portal rolled nothing ever moved that incident: every later burst folded into it and asked for triage again.

The portal now applies the same rule to an incident that ALREADY exists (LogIncidentRetirement), in the two places that write it:

• the fold of a repeat — the burst's classification, or the stored one when the burst carries none; and
• the control plane's claim of a Triage or File request — so an incident that never sees another burst is retired the moment the current build reaches the request the older one left.

An incident that qualifies becomes Suppressed, asks for nothing, and keeps its count, samples and evidence — exactly what it would have been had the current build created it. It qualifies only while it has no issue and is New, Triaging, Failed or already Suppressed. An unknown classification still retires nothing (open vocabulary).

A ticketed incident is annotated once, then retired

The maintainer's decision on Plugins#3512: "annotate and retire." An incident that already has a GitHub issue, and whose fingerprint the running build no longer triages, gets ONE comment on that issue with the evidence — the classification, what classified it, the occurrence counts and window, the latest samples — and then no further recurrence comments. The issue is not closed, and not reopened either: the classification says the platform stops REPORTING this fingerprint, not that the defect behind the ticket is gone, so closing it stays a person's call.

• LogIncidentRetirement.ShouldAnnotate decides it: a recorded-not-triaged classification, an issue link, no RetirementAnnouncedAt yet, status New/Triaging/Failed/Filed (a Filing incident is left alone until its link is known).
• The fold of a repeat asks for Comment regardless of the recurrence comment window; the control plane's claim grants any Triage/File/Comment request on such an incident as that comment.
• LogIncidentFiler.AnnounceRetirement posts it at the ticket's CURRENT home (a transferred ticket is followed; a deleted one takes the same proof-before-clearing path a recurrence takes, after which the incident has no issue and is retired unticketed). It never reopens or closes.
• Only once the comment LANDED is the incident Suppressed with RetirementAnnouncedAt stamped. From then on every repeat is counted and asks for nothing, and the comment-debt clock skips it. A refused post parks it Failed and rings the platform bell like any refused comment, and the next repeat asks again.

Related

• Incident identity — who computes a fingerprint; a non-hex one such as roll-churn-… passes through the portal's resolution unchanged.
• One log site, one ticket — the site fold the roll-churn node is keyed on.