A silent cell owner must not mislabel a round

Every cell of a thread ({thread}/{cellId}, a ThreadMessage) is its own per-node hub. On the control instance those owners were regularly slow to answer: a version-1 user cell nobody had read since it was written, the cells of a thread whose hub the supervisor had just recycled, every cell of a thread on a pod fresh out of a roll, and a response cell whose first writes were never acknowledged inside the write queue's 5 s bound ([UpdateQueue] ADVANCE_WITHOUT_HANDOFF → [MergeGuard] refused stale/reordered cross-hub write → [UpdateRemote] LATE_NACK_REENQUEUE). Why an owner is that slow is a platform question (the query fan-in stall family, core #5315 / #5344 / #5478) and is NOT answered here. What this page records is where the AI engine let one silent owner decide a whole round, and what each of the three places does now.

1. History: a witnessed cell comes from one listing, not from its owner (MeshWeaver#5862)

Before. ThreadExecution.LoadFullConversationHistoryFromMesh read every prior cell from its OWNER (GetMeshNodeStream("{thread}/{cell}")), in parallel, each under a 5 s budget, and an Unseen cell fails the round (HISTORY_LOAD_FAILED, #982 — a history with an unknown hole is never handed to the agent). So a thread with N prior cells asked N per-node hubs, every one of which had to be active on the serving replica before it could answer. Measured on the control instance: 27 of 31 prior cells … did not answer within 5s on a pod fresh out of a roll, 1 of 272 … on the fleet coordinator, 1 of 1 … on a relaunch 30 s after a recycle. In every sample the store held the cell's text.

Why not simply read the cells from a listing. That was tried and withdrawn. A listing's Content is read-side data that may trail a committed write (Doc/Architecture/CqrsAndContentAccess), and a cell is NOT write-once: ResubmitMessage rewrites a user cell's text, and a daily-limit re-queue runs the next round INTO its Error cell. A trailing row would put an old prompt or an old refusal into the model's history with nothing reporting it. Thread.UserMessageFingerprints cannot tell either: a resubmit deliberately keeps the original fingerprint.

Now: the thread holds a witness per cell, and a row is taken only when it produces that witness. Thread.HistoryWitnesses maps a cell id to the full SHA-256 digest of the cell's role and text (HistoryWitness, 64 hex characters; Plugins#3413). The loader already reads the thread node live from its owner, and the round's own two cells prove that snapshot current (#982). With it in hand the loader asks for ONE children listing of the thread and takes a row for a cell only when the row is settled (it has text and is not streaming or queued) AND its role and text hash to the witness the thread holds. A row that trails its cell cannot produce the witness of the newer text. Every other cell is read from its owner exactly as before, with the same Unseen refusal:

the cell read from
witnessed, and the listed row produces the witness the listing — no hub is asked
nothing has witnessed it yet (a thread from before this, a cell written by a path that records none) its owner; the load then records the witness
the listed row does not produce the witness (the row trails, or the cell was edited) its owner
not in the listing, or the listing fails or does not answer inside the cell budget its owner

The listing is never trusted for a negative, and a thread with no witness at all is not listed.

Who writes a witness, always in a thread write that already happens:

A wrong or missing witness therefore costs one owner read, never a wrong history. No witness is recorded for empty text. The witness is the WHOLE digest: a 64-bit truncation (the first cut) let a crafted collision make a trailing row pass, so that guarantee held only up to about 2^32 attempts. A thread that still carries a truncated witness from before this matches no row, so each such cell costs one owner read, and the load then records the full digest.

What a witness does not cover. An edit of a settled cell made OUTSIDE the engine (a direct update of the cell node) changes no witness. While the listing trails that edit the round sees the text the thread witnessed, and from the moment the listing shows the edit the row no longer matches and the owner is asked. The engine's own rewrites are all covered above.

Reading it on a deployed portal. Read the thread node, not the log: get {threadPath} and compare content.historyWitnesses with content.messages. Every cell id in historyWitnesses is one the next load takes from the listing, as long as its row still produces that witness; every prior cell missing from it goes to its owner. A load that fails because an owner it had to ask was silent logs HISTORY_LOAD_FAILED at Error with N of N prior cells … did not answer, and that line does reach Loki.

Reading it in the log (local runs only). [ThreadExec] HISTORY_SOURCES … fromListing=… fromOwner=… witnessed=… is logged once per load, and HISTORY_LISTING_UNAVAILABLE when the listing did not answer and every cell went to its owner. Both are Information lines in the category MeshWeaver.AI.AgentChatClient. Every shipped portal configuration (Memex.Portal.Distributed and Memex.Portal.Monolith, Development included) keeps MeshWeaver at Warning and pins no MeshWeaver.AI category, so neither line reaches Loki: their absence there says nothing about which path a load took. To see them on a local run, raise that category to Information in the appsettings.json of the BUILD OUTPUT (bin/…, reloaded on change), never in src/. The production levels are a cost decision, not a debugging switch.

Pinned by HistoryFromWitnessedListingTest on a real mesh with SilentOwnerSwitch, which also counts every delivery that reaches a silenced hub: a witnessed thread of a dozen turns loads in order while all its cell owners answer nothing and NO delivery reaches any of them; a cell appended just before the load is in the history and is witnessed for the next load; a row that does not produce the thread's witness is refused while its owner is silent. Negative control: the same thread without witnesses is the loader as it was — every owner is asked and the load fails with the production message, N of N prior cells … did not answer within 1s.

A cell read ends on the cell's verdict, never on the timer. A cell read from its owner used to end only when it carried TEXT, so a settled text-less cell (a cancelled round's cell, an attachment-only turn) ran out the whole per-cell budget on every load: five seconds added to every round of every thread that carries one, and its Empty verdict decided by a timer firing rather than by what the cell said. The read now ends at the first content that settles the question: text (Loaded), or no text with a status that is neither Streaming nor Queued (Empty, logged HISTORY_CELL_EMPTY). A text-less cell that is still streaming or queued may yet get text, so the read keeps waiting for it, exactly as before.

An unseen cell says how it went unseen. The refusal used to read N of N prior cells … did not answer within Ns for every unseen cell, including a read that FAULTED in milliseconds (a phantom cell's read ends on a routing error long before a 1 s budget). It now names each cell's fate: {cell} did not answer within Ns when the budget ran out with nothing seen, {cell} read faulted ({error}) when the read ended with an error; HISTORY_CELL_DROP carries the same distinction (ended=). A slow owner and a failing read are different defects, and the message has to tell them apart.

Pinned by LoadConversationHistoryTest: SettledTextlessCell_IsAnsweredOnItsVerdict_NotOnTheBudget loads under a 60 s budget and must answer inside 20 s, StreamingTextlessCell_IsNotTakenAsAnAnswer is the other arm, and SomeCellsUnseen_FailsTheLoad_InsteadOfAnsweringOnAHole asserts the phantom cell is reported as a fault. Negative control: on the previous loader the first and the third fail (the load answers only at the budget; the phantom is reported as "did not answer within 1s"), and the second passes on both.

What the round does when an owner it must ask is silent is unchanged: it ends in a NAMED state — the response cell carries the localized "history could not be loaded" error, the thread settles Idle with that Summary, and a delegating parent is told the round failed. During the thread hub's own teardown it is not failed at all (HISTORY_LOAD_INTERRUPTED, #2618) and the next activation resumes it.

2. Error rounds: the thread settles on its own write (MeshWeaver#6137)

Before. The Error branch of ThreadExecution chained the thread's terminal write onto the error CELL's write under a bare .Timeout(10 s). When the cell's owner did not acknowledge in time, Rx's message-less TimeoutException: The operation has timed out. faulted the round, and the outer catch handled it as an initialization stall:

Now. As in the Completed and Cancelled branches, the cell push and the thread's terminal write are independent writes to two nodes. The cell push reports its own verdict (the write pipeline bounds it; a failure is logged); the thread settles on its own write — Idle, the classified Summary, the inbox fold, the re-queue — whatever the cell's owner is doing. The consumed-inbox snapshot is now taken synchronously in the branch, before the round's finally resets the channel; it used to be taken in the cell push's callback, after that reset.

Pinned by RefusedRoundRequeueTest.ARefusedRound_SettlesTheThread_WhileTheErrorCellsOwnerIsSilent: the response cell's owner is silenced from the moment the provider is called; the thread must settle with the input pending again and a Summary naming the refusal, and the cell, once released, carries no "stalled" suffix. Negative control: before the change the thread settles Idle with nothing pending and an empty Summary — the production shape exactly.

3. A post to a node that is not a thread is refused by name (MeshWeaver#6003)

Before. ThreadInput.AppendUserInput's update lambda read the target with a logging ContentAs<Thread> and returned the node unchanged when it was something else. A post into the TRIAGE ITEM Hosting/Triage/issue/systemorph-meshweaver-6002 (whose threads are paths recorded in its content) therefore logged As<Thread> … value is TriageItemContent … not convertible at Error and vanished; the poster was told nothing — submit_message read an onError that a cross-hub write can only call later, so it answered "Message queued" for every post.

Now. ThreadInput.AppendUserInputObservable is the append as a cold observable that FAULTS, naming the node and saying to post to the thread path the node records, when the target declares another NodeType or holds content that is not a thread. hub.ObserveSubmitMessage(...) is SubmitMessage as an observable (SubmitMessage subscribes it and reports the reason through its onError), and the submit_message tool answers from the write's verdict through ToolTask.Bridge, so the agent hears the refusal and can follow the item's thread path. A signed-in user's post already went through ThreadContinuation, which refused a non-thread target.

Pinned by SubmitToANonThreadTest. Negative control: on the old lambda the test reproduces the production Error line verbatim and the refusal never arrives.

The signed-in principal's refusal was opaque, and the tool now words the refusal it gets back. Measured on the control instance (2026-10-11, response cell rbuergi/_Thread/verification-probe-for-meshweaver-6003-d-e1820ba71e72d3f3b763/1321e1a6): an agent running as a signed-in user called submit_message with a Markdown node's path. The post was refused and the node untouched, but the tool result read Message NOT delivered to …: SubmitMessage failed: No handler found for message type SubmitThreadMessageRequest. For a real user ObserveSubmitMessage sends a SubmitThreadMessageRequest to the target's own hub, and a hub that is not a thread has no handler for it — so the refusal that names the node and points at its thread path is never reached on that route, and the model is left with a transport error it cannot act on.

ThreadMessageTool therefore answers that refusal in words the model can act on. When the target path has no _Thread segment (ThreadMessageTool.IsThreadPath), the refused post is answered with the owner's own refusal, that nothing was sent, and how to reach the conversation: read the node and post to the thread path it records (threadPath). It deliberately names no thread search: a node's thread need not live under that node — the triage item of the measured case records its thread under Hosting/Triage/_Thread/… — so namespace:{node}/_Thread would come back empty for exactly that case and read as "this node has no thread".

The advice is added ONLY to a refusal that identifies a non-thread target (ThreadMessageTool.IdentifiesNonThreadTarget, Plugins#3415): the append's own refusal (NotAThreadException, or its words once a hop flattened it) or the owner's "No handler found for message type SubmitThreadMessageRequest". Every other refusal (no write access, a closed conversation, a timeout) passes through unchanged, because a refused LEGACY thread also lacks the segment and the advice to treat it as an about-node would mislead the model there.

The path's shape only chooses that WORDING; the tool sends first and words the refusal afterwards, and never refuses before sending. Not every thread path carries _Thread: legacy conversations are Thread nodes stored elsewhere in a user's home, which ThreadQueries.InboxThreads keeps in the inbox, and a post to one of those is delivered — the write reaches the node's type, which the path cannot. (A first version of this change refused every path without the segment before sending and so cut those conversations off.) The answer therefore hedges ("most likely not a thread") and keeps the owner's refusal verbatim, which is what tells a refused legacy thread from a node that is not one. Pinned by SubmitToANonThreadTest (SubmitMessageTool_DeliversToALegacyThreadOutsideTheThreadSegment, SubmitMessageTool_ToANonThreadNode_AnswersWithTheRecordedThreadPath, SubmitMessageTool_ADeniedPostToALegacyThread_PassesTheRefusalThroughWithoutTheAdvice) and ThreadMessageToolTest (The_answer_for_a_refused_non_thread_path_names_the_recorded_thread_path).

What is NOT established