When the ticketing pipeline tickets itself

Measured 2026-09-19 on memex. MeshWeaver#3138 — one failed Postgres query stopped red logs being ticketed for the life of the pod — was fixed by 2e1d498338 : OpenIncidentWatch now re-opens the watch with backoff, unbounded in count and bounded in rate, instead of ending the sequence with .Catch(… ⇒ Observable.Empty).

Then the fix reported itself as a production incident:

No restart in between. That sequence is itself the proof the fix works — pre-fix, a restart was the only way the watch could ever answer again — and it is also a self-referential noise loop.

The loop, in one sentence per hop

1. The retry arm reported at LogError.
2. A .NET console LogError renders with the prefix fail:.
3. mw-log-watcher ingests fail:/crit: and nothing else (Red-log watching and automatic ticketing).
4. So the pipeline that turns red logs into tickets ticketed its OWN retry — and the fingerprint it landed on was the one already mapped to the issue about the defect that retry had fixed.

This is the rule #4257 landed one layer up, arrived at from the other direction: never assert a severity on a DETECTION when a remedy is about to run. Log the detection at Warning, and put the Error on the outcome.

Why the fix is a DURATION and not a demotion

The retry is unbounded in count by design — the watch's lifetime is the service's, not the first fault's — so onGiveUp may never fire at all. Demoting every arm to Warning would have removed the noise and, with it, any report of a ticketing blackout that lasts all night. That is a worse defect than the one being fixed: the pipeline that reports outages is the one thing whose own outage nothing else reports.

So the levelling is:

Five minutes because every occurrence ever measured on this pipeline was a single Postgres CONNECT timeout the next attempt healed — 18 of them over ten days in #3138, and the 2026-09-19 sample recovered in one second — while the backoff reaches its one-minute ceiling at attempt 7 (63 s cumulative). Five minutes is roughly ten attempts: unambiguously "the store is down", and short enough that an operator hears about a real blackout inside one on-call cycle.

The escalation rides on a fault rather than on a timer, so it is observed within one ceiling (≤ 1 min) of the threshold. That is deliberate: a timer whose only job is to watch a clock would be a second subscription to keep alive, and the down store is already producing a fault a minute.

Two decisions that are easy to get backwards

The reset happens AFTER the handler, not on arrival. "Recovered" has to mean the whole emission path worked. A query that keeps completing into an enqueue that always throws is a ticketing outage too, and resetting on arrival would hide exactly that case behind an endless row of warnings with no error anywhere. AnEmissionWhoseHandlerThrows_IsNotARecovery pins it.

The escalation carries NO exception object, though it names the fault inline. This is the part that actually kills the reopen loop, and demoting the message alone would not have: StructuralLogIncidentIdentity keys a burst on its top application FRAME when there is one and discards the reporter's prose, so an Error here that passed the store's TimeoutException would fingerprint as the Npgsql connect fault — the very identity that made the old retry line reopen #3138. With no stack trace the burst is keyed on the log SITE instead, which is the ticket an operator wants: the log-incident watch is blind, not Postgres timed out, which every other component reports better. Nothing is lost — the warning carries the exception in full, at every attempt including the escalating one, at the same instant.

The control

IncidentWatchSurvivesAFailedQueryTest, on a TestScheduler, no clock and no mesh — the composition stays internal static and scheduler-parameterised so the policy is an assertion rather than a timing observation. The three original facts (re-open, unbounded in count, bounded in rate) are unchanged; four were added:

• AShortBlip_IsWarnedAboutAndNeverEscalated — the production case. Before the levelling this fact's zero was a two.
• AnOutageThatLasts_IsEscalatedExactlyOnce_HoweverLongItRuns — one report per hour-long outage, and its duration is within one backoff ceiling of the threshold.
• ARecovery_ArmsTheEscalationAgain_SoTheNextOutageIsReportedToo — a latched error is a one-shot alarm dressed as a continuous one.
• AnEmissionWhoseHandlerThrows_IsNotARecovery — the reset ordering above.

The same loop, one layer down: a roll ticketing its own shutdown (MeshWeaver#4962)

The incident writes had the same shape. From ApplicationStopping on, the router refuses every outbound delivery of the process (Host is shutting down, cannot route to …). The control plane kept starting transitions into that window, and it reported each refusal at Error: "Incident X: Comment could not write …, so nothing was recorded on the incident. It still carries its request and is picked up again …". The sentence itself said the work was retained, so the Error described a remedy that was about to run. 6 of the 10 retained samples on MeshWeaver#4962 are that terminal, and every roll filed incidents about its own shutdown.

Two changes, both keyed on core's one predicate for this, IMessageHub.IsLeaving() (hub shutting down or the process's ApplicationStopping, MeshWeaver#3129):

• A leaving host starts no transition. Run checks at subscribe time and returns without touching the node, logging at Information what it did not start. Nothing is lost, because nothing has claimed the request: it is still on the incident, and the replica that replaces this one picks it up from the collection.
• A transition that fails after the host began leaving records nothing and is reported at Information when the work is still recoverable, meaning a reconcile or a failure before the claim. This covers ANY failure, the operation's own as well as the incident write's: the recovery write that would record an operation failure goes into the same refusing router and came back as an Error ("could not record the failure"). A stranded transition (the claim had already consumed the request) stays Error whether or not the host is leaving, because nothing resumes it.

ALeavingHostStartsNoIncidentWriteTest pins both. With the start guard disabled, the test fails because a write was attempted (an exception-carrying line appears). With both guards disabled, it fails on the Error line. LogIncidentUnwritableIncidentTest's first case is the control: the same unwritable incident on a host that is not leaving is still exactly one report that carries the fault (a Warning, see the next section).

And once more: a recoverable write on a host that is staying (MeshWeaver#4962)

The leaving-host guards left one more copy of the loop. A write that fails before the claim, or a reconcile's write, is re-attempted by design: the request is still on the node, or the reconcile's trigger survives. Each such attempt was still reported at Error.

Measured on the control instance, fingerprint f1e8bef1a3761721. 34 occurrences between 2026-10-05 and 2026-10-11, every one the 30 s base-state abort ("Update aborted: no initial state arrived … NO change item at all reached the mirror"). The retained samples fall in two clusters, and each sits inside the minutes a wedged silo was being evicted (MeshWeaver#6432):

So none of the three suspects in the abort message applies: the nodes existed, the store answered, and no owner was failing to hydrate. A silo whose threads are parked on a container lock dispatches nothing, whether the write starts on it or needs an answer from it. All six incidents named in those samples were written again afterwards (each carries later versions on the control instance).

Not established: which silo the 20:43Z owners were waiting on. That the owner of 90565fcee7617b58 sat on a healthy pod and came unstuck 23 s before pod A was evicted is a correlation. (The pod names are in MeshWeaver#4962's evidence, not here.)

The change. LogIncidentControlPlane.RecordFailure reports a recoverable incident-write failure at Warning, with the fault. The Error is on the outcome: an incident still unwritable after IncidentWriteOutageEscalationAfter (five minutes, the watch's own threshold), once per outage, with no exception object for the reason given above. It is raised by the next failed attempt past the threshold, and by the comment-debt clock for an incident nothing re-attempts, since "picked up on the next change" may never come for a quiet one. A write that lands ends the outage — and so does the incident collection showing that the work is no longer owed (the request left the node, the stranded triage was repaired, the comment debt is settled, the incident is gone): the outage ledger is one process's, and on a portal with several replicas another one may have served the request this one failed on (ForgetServedWriteOutages). The same rule covers the NESTED write: when the operation fails first (a GitHub call, a thread query) and the write recording that failure on the incident then fails too, that write joins the outage at Warning while the work is recoverable, instead of the immediate "could not record the failure" Error. A stranded transition is unchanged: Error on the first attempt, nested write included.

A failed reconcile's warning names the ONE trigger that re-attempts its kind (Plugins#3414), because each kind has exactly one. A Comment reconcile is enqueued only by the comment-debt clock, so it is re-attempted on the next tick that still finds the debt unsettled. A Triage reconcile (the stranded-triage repair) is enqueued only by a collection emission that still shows the triage stranded; the clock's tick never re-attempts it and only escalates a lasting outage. The warning used to name both triggers for every reconcile, which promised a quiet incident a retry its next tick would not make.

ARecoverableIncidentWriteFailureIsNotATicketTest pins each of those, with the stranded case as its control.

What this does not fix is the wedge. That is MeshWeaver#6432, and its own lines (the eviction, the stalled creates, the unanswered cancellation calls) still file.