Reading a Write Verdict

A cross-hub write through GetMeshNodeStream(path).Update(...) fails closed: on silence past its confirmation window it faults rather than optimistically succeeding (see MeshNodeStreamCache). Every user-visible failure therefore arrives as the same thing — a MeshNodeStreamException in a log line — and the investigation it should trigger depends entirely on which site minted the error inside it.

Get that wrong and you spend the day on the wrong hub. This page is the decision procedure.

The exception is a formatted verdict, not a description

MeshNodeStreamException renders its structured MeshNodeError verbatim:

// src/MeshWeaver.Data.Contract/MeshNodeError.cs
public MeshNodeStreamException(MeshNodeError error)
    : base($"MeshNode {error.Code} at '{error.Path}': {error.Message}")

So a log line has the shape MeshNode {Code} at '{Path}': {Message}, and Code alone separates the two investigations.

The shapes, and what each one means

The Unknown row and the classified owner refusals come from one classifier on the owner:

// src/MeshWeaver.Data/DataExtensions.cs — ClassifyPatchException
_ => (MeshNodeErrorCode.Unknown, ex.GetType().Name),
…
return new MeshNodeError(code, path, $"{prefix}: {ex.Message}", ex.StackTrace);

That default arm is the only site in src/ that mints a MeshNodeError whose message is {TypeName}: {message}. A MeshNode Unknown at '…': TimeoutException: … line is therefore proof that the owner answered — the writer's own no-verdict path cannot produce it, because it always stamps OwnerUnreachable and always writes a long sentence with a request trail.

🚨 TimeoutException in the message does not mean the write timed out. It means the owner hit one of its own bounds and told you so. The word is the same; the failing component is not.

The owner's two internal ack bounds

When the owner accepts a PatchDataRequest for a MeshNode, it does not ack immediately. It watches for its own commit echo and then flushes durably — two bounded steps, and both surface as Unknown + TimeoutException:

// src/MeshWeaver.Data/DataExtensions.cs
var postSub = ArmPatchAckWatcher(
    stream
        .Where(c => ChangeContainsStampedWrite(...))   // identity-based echo detection
        .Take(1)
        .Timeout(TimeSpan.FromSeconds(20)),            // ① commit echo
    committed => hub.ServiceProvider.GetService<IPostCommitFlush>()?.Flush(committed.Value!),
    TimeSpan.FromSeconds(10),                          // ② flush bound — a WAIT bound on the ack, not a fate
    AckOnce, d => hub.RegisterForDisposal(d), hubPath, () => hub.IsShuttingDown, patchAckLogger);

ArmPatchAckWatcher subscribes the echo with all three arms (see "The ack watcher's completion arm" below). Only bound ① can mint the timeout NACK. Bound ② no longer produces a verdict at all — see "The flush bound is not a verdict" below.

Which one fired is readable from the elapsed time between the user's action and the log line:

• ~20 s and a NACK Unknown + TimeoutException → ① the commit echo never arrived. The merge turn did not reach the stream, or the reduced stream's fan-out is starved. The write's fate is genuinely unknown: read the node's Version before retrying anything that accumulates.
• ~10 s after the merge, a SUCCESS, and an owner-side FLUSH_OUTLIVED_BOUND warning → ② the commit landed and IPostCommitFlush (i.e. StoragePostCommitFlush → storage.WriteAndPublishUpdated) had not finished in 10 s. The write applied; the storage behind that owner is slow. This is the signal to read when a bulk operation drags — never a reason to retry the write.
• ~31 s → neither: this is the writer's OwnerUnreachable window, and the Code will say so.

The flush bound is not a verdict (#3112)

Reaching the flush at all means the owner's reduced stream emitted the echo containing this write: the merge landed, the Version advanced, every mirror is already receiving the new state. Whatever happens to the durable flush afterwards decides when the ack is posted, never whether the write applied. Until #3112 the code disagreed with that: durable.Take(1).Timeout(10 s) fed the fault arm, so a flush still queued at the bound became a NACK Unknown + TimeoutException — read by the writer as LATE_NACK_TERMINAL … the write did NOT apply and is not auto-retryable and by its caller as a failed write — and the Timeout also disposed the flush, cancelling a storage write that had been queued for the whole bound and handing the row to the persistence sampler, which queued it again. Under the congestion that made the flush slow, every slow flush became two writes and one false failure.

The measurement. MeshWeaver.Manufacturing PR #48, run 33623113056, test-repos / Compile + render node repos: the gate's disposable mesh installs 37 upstream packages and, per installed NodeType, re-seeds the sealed bake's prebuilt assembly through one cross-hub Update on the type's node. For Radzen/Gallery the trace reads ADVANCE_WITHOUT_HANDOFF … bound=5000ms at +5 s, then at +10.02 s LATE_NACK_TERMINAL … code=Unknown … msg=TimeoutException — the owner answered, ten seconds after the post, which is the flush bound measured from an echo that arrived within milliseconds. The seed logged seeding Radzen/Gallery … did not complete — the sweep compiles it instead, the sweep compiled the type over the adoption stamp that had in fact committed, and the consumption postcondition reported adopted 101 of 102 … DECLINED: Radzen/Gallery. The control arm — Reinsurance run 33623801785, same seal, same bytes, minutes later — adopted Radzen.zip 1/1. Same store, same code; only the queue depth differed. In the same failing log, portal/nodeops held a CreateOrUpdateNodeRequest for 60 s ([STALE-CALLBACK] … AWAITING), which is the depth of the queue the flush was sitting in.

The shape now. On the bound the owner acks Success — the commit is what "saved" means (#2661), so this is the truthful verdict — logs [PatchAck] FLUSH_OUTLIVED_BOUND path=… bound=10000ms on the MeshWeaver.Data.PatchAck channel, and leaves the flush running. The flush's own terminal then finds the once-only gate claimed: an emission is a no-op; a fault is logged as FLUSH_FAULTED_AFTER_ACK and the sampler — whose claim StoragePostCommitFlush releases in Finally — stays the writer of record for that version. A flush that faults inside the bound still NACKs with its classified code: a storage refusal is a fact the caller should hear; slowness is not a verdict. PatchAckTotalityTest pins all three paths on a virtual clock.

Under a genuinely slow store the writer therefore sees the ack arrive late — LATE_ACK, the caller completes with success — and ADVANCE_WITHOUT_HANDOFF at 5 s still says the successor diffed against the mirror. That warning is now the one to read for storage pressure; it is not a failed write.

The writer of record does not drop a failed write

The persistence sampler is that writer of record, and until this fix its own store write was the one place a committed state could still vanish. The sampler marks a state requested the moment the hub accepts its SaveMeshNodeRequest, and the dispose-time flush (FlushPendingOwnSave) persists only states that were never requested. So a sampled write that failed in the store — an outage, a connect timeout — logged one warning and left its state marked as saved while the row stayed behind: the change lived only in RAM, and a recycle reloaded the older row.

The shape now. A failed sampled write logs [SaveMeshNode] the store write of … FAILED — the state is not durable at Error, and un-marks its own save (PendingOwnSave.MarkWriteFailed, matched by reference, so a newer save that superseded it is left alone). The state reads as unsaved again and is re-attempted by the next sampled save or, at the latest, by the dispose flush — which first waits for the hub's last store write to settle, so a write that fails after teardown began is still seen. This is not a retry loop: one re-attempt per later save or teardown, and a failure of the dispose flush's attempt surfaces as a [DISPOSE-ACTION] … faulted warning on the hub. Pinned by AFailedOwnSaveSurvivesARecycleTest (MeshWeaver.Graph.Test): the store fails the own save once, the hub is recycled, and the fresh activation must serve the change from the durable row — with a healthy store as the negative control.

One write at a time, in order. A re-attempt follows a failed write quickly, so an OLDER write still in flight could otherwise land after the newer one — or after the dispose flush — and roll the row back: the file-system adapter's pool runs writes concurrently and its backend has no conditional write to refuse the stale one. Every own write and the dispose flush are therefore queued on PendingOwnSave.Enqueue, which starts a write only once the previous one has terminated (either way) and replays the write's own outcome to its caller. The queue is a chain of eagerly fed ReplaySubjects — the IoPoolExtensions.Run shape — so no connection handle is left unowned, and the flush joins it simply by being the last entry. Pinned by OwnWritesAreSerializedTest (same file).

The ack watcher's completion arm (the gap closed by #3033)

postSub used to subscribe with onNext and onError only. If the filtered stream completed without emitting — which SynchronizationStream.Dispose does by calling Store.OnCompleted(), and which mirror eviction can do while the hub still lives — then Take(1) ended empty, the Timeout was cancelled, neither arm ran, and no ack was ever posted. The writer then waited out its full window and reported OwnerUnreachable for a write the owner may already have committed. This was the exact defect fixed on the writer side by RequireBaseState (Write Verdict Totality), one hub over; RegisterOwnerDisposingNack covered a disposing hub, not a stream that completes while the hub lives.

The watcher is now armed through one pure composition, DataExtensions.ArmPatchAckWatcher, built on the WhenCompletesEmpty operator — "run this callback if the source completes without ever having emitted" — so every path posts exactly one terminal:

Three things about that shape are load-bearing:

• 🚨 The completion arm is guarded on "no emission was ever observed". A bare onCompleted => AckOnce(false) would NACK every successful write: on the happy path Take(1) emits the echo and completes immediately, while the inner flush is still in flight, and AckOnce latches — the completion arm would win the race against the flush's later AckOnce(true). WhenCompletesEmpty fires only for a completion that no emission preceded.
• OwnerDisposing, not OwnerUnreachable and not a new code. A stream ending under a live patch is the owner's stream going away, which is what the code means; it is the code the writer auto-retries (bounded — MaxOwnerDisposingReenqueues), and a re-enqueue re-runs the update lambda against the fresh state and re-diffs, so a merge that did commit before the stream ended becomes a no-op. "Fate unknown, safe to retry" is the honest verdict — and because the timeout verdict stays Unknown, the two are separable by Code.
• 🚨 The completion arm stands aside when the OWNER is shutting down (hub.IsShuttingDown, passed to ArmPatchAckWatcher as ownerIsShuttingDown). An owner hub's teardown completes the very same stream — its sync hub disposes in the DisposeHostedHubs phase and Store.OnCompleted()s — but that completion is not the eviction case, and answering it from the watcher broke both teardown-NACK tests on 2026-09-02 (LateNackReenqueueTest, NackReachesTheWaiterDuringTeardownTest). Two defects in one claim of the once-only ack gate: one phase too early — a hosted hub leaves its parent's registry only in its ShutDown phase (Hub Disposal Model), so a "safe to retry against the fresh activation" minted at DisposeHostedHubs sent the writer's immediate re-enqueue into the dying activation, which rejected it ShuttingDown, and the write failed Unknown; and the wrong transport — hub.Post from a hub past Quiescing is dropped under a whole-mesh teardown, and with the gate already claimed the disposal registrant's direct ILatePatchVerdictSink.Dispatch (the one route that still reaches the armed waiter) was skipped, so the caller burned its whole verdict budget in silence (#2778's shape again). The ShutDown-phase RegisterOwnerDisposingNack — registered on the same hub, total for a disposing hub — therefore owns the verdict for a patch in flight at owner teardown, and the watcher's arm fires only while the owner lives.

The generic deferred path (ApplyJsonMergePatchAndUpdate, non-MeshNode data hubs) had the same gap twice over — its initial stream.Take(1) read had neither an error nor a completion arm, and its Skip(1).Take(1) echo watcher had no completion arm — and is closed by the same two seams. An empty initial read NACKs OwnerDisposing with the patch was NOT applied (there the write provably never ran).

A tool's "landed" is the verdict, never a read-back (#6635)

The MCP patch and update verbs (MeshOperations) answer Patched: / Updated: on the owner's commit verdict — what IMeshService.UpdateNode emits — and nothing else. That verdict is total: the owner acks only after its identity-gated commit echo and only when every key merged (a partial or total three-way refusal is a Conflict NACK), and every refusal, early or late, faults the call. The version printed beside the answer is read from the live mirror for display only and can never turn a landed write into a failure.

Until #6635 the verbs confirmed instead by re-reading the caller's own fields from the LIVE node. That asks a different question — "does the node still hold my values?" — and a field another writer legitimately consumes answers it "no". Measured on the control instance: a patch carrying a review and requestedAction: PostReview committed at v279, the item's control plane cleared the intent at v281 63 ms later, and the tool answered "did not land within the confirmation window … retry" after 10 s — inviting a retry of a non-idempotent request that had already been accepted. AConsumedIntentIsNotALostWriteTest pins it with an owner-side watcher that clears the intent the moment it lands; on the read-back confirmation it is red on its first round with that exact message.

Procedure

1. Read the Code, not the prose. OwnerUnreachable and Unknown lead to different hubs.
2. Measure the elapsed time. It names which bound expired.
3. Read the node's Version. If it advanced, the write landed and only the ack failed — do not retry a non-idempotent write.
4. Only then look at the owner. For OwnerUnreachable, ask whether the activation was alive at all; for Unknown + TimeoutException, ask what kept the commit echo from the owner's reduced stream for 20 s (the flush bound cannot produce this code any more — a slow flush is a FLUSH_OUTLIVED_BOUND warning beside a successful write).

See also

• MeshNodeStreamCache — the write path, its serial queue, the late-response registry and the fail-closed verdict.
• Durable But Unreadable — the other way a write's durable state and its reported outcome disagree.
• Debugging Message Flow — finding the exact broken edge once you know which hub to inspect.
• CQRS and Content Access — why the index can answer while the owning per-node hub cannot.