CD reconciles the Plugins seal

🚨 Retired — this page is history, not the current design. Core commit 08d37d38ac (#6143, policy platform-module-deploy-separate, see Platform and Module Deploy) removed the plugins-modules, plugins-bake and report-plugins-seal jobs from main-cd.yml, together with gate's seal probe (plugins_seal_due) and the cd-plugins-seal ledger.

Today:

• Core CD's publish-bake publishes meshweaver-content only — the platform's own content.
• The only writer of the plugins publication is MeshWeaver.Plugins, through its own CI calling node-repo-publish-bake.yml. Core CD merely reads that sealed publication (published-modules) as compatibility evidence, and a publication absent for the run's identity is red there, never rebuilt.
• A missing plugins publication is therefore the Plugins lane's to publish, not core CD's to re-attempt. Sealing it per package rather than per commit is in flight (core #6609, MeshWeaver.Plugins#3507) and not yet landed.

The rest of the page records the mechanism as it ran from #4539 to #6143, and why the platform seal is the trio alone — that part still holds.

A core CD run published two different things for one commit, and until MeshWeaver#4539 only one of them could heal.

• The platform set — the images, plus the platform's own content bake. It is sealed when the trio Promote: tag the full set, Verify every image shipped and Bake platform content in the shipped image + publish have all succeeded. That trio is the seal and nothing else is (maintainer rule, 2026-09-12; it is stated in resolve-platform.py's module docstring and read by every satellite's copy of it).
• The plugins publication — the NodeType bundles baked from MeshWeaver.Plugins against that set's portal image, sealed to the artifact store under the set's framework identity.

Every satellite resolves the newest sealed set and then fetches plugins under that set's identity. So a set can seal while its plugins publication does not exist, and every satellite holds — correctly — on an upstream nobody is going to publish.

What it looked like

On 2026-09-16 core CD #8765 promoted, verified and baked 3.0.0-ci.8765. Then Plugins: pack the module bundles the bake composes / Warm the shared build environment died pulling the tester image:

Error response from daemon: Get "https://meshweaver.azurecr.io/v2/":
dial tcp 51.12.25.82:443: connect: connection refused

Plugins: bake + seal the publication for this identity was therefore skipped, and framework identity s5ec352bb102e5a2275e3831a08ac0c8d got no plugins publication. At 20:14:20Z four satellites resolved 8765 — the newest sealed set — and all four failed in the same second with

upstream 'plugins' answers 404 … no sealed publication for source 'plugins'
under framework identity 's5ec352bb…'

Nothing re-attempted it. MeshWeaver#2452 had given the platform bake a reconcile path (bake_only), but the three plugins-* jobs were still gated on publish == 'true', so every hourly tick re-asserted the platform bake and left the publication missing. What healed it was a coincidence: MeshWeaver.Plugins merged, which moved plugins HEAD, which made the image set read INCOMPLETE, which produced a full re-publish.

It is not a one-off

Sweeping the 200 core CD runs of 2026-09-14 → 09-17, nine sets sealed with their Plugins seal not success:

That split is what decides the design: three of nine are transient, six are a genuine Plugins red.

Why the set still seals — the shape that was rejected

The obvious alternative is to make the seal all-or-nothing: no platform set until its plugins publication exists. That removes the broken state rather than repairing it, and it was rejected on the measurement above.

• Six of the nine were Plugins' own red. Under an all-or-nothing seal, core would have sealed nothing for each of those windows. A MeshWeaver.Plugins unit test would freeze the portal image that every AKS install self-updates to — a strictly worse outage than the one being fixed.
• It reverses a dated maintainer rule (the trio is the seal, 2026-09-12) and hands MeshWeaver.Plugins a veto over the platform's delivery. That is the dependency direction PlatformNeverDependsOnPluginsGuard forbids, and the one satellite-compat already refuses in the same words: "Blocking the publish on a satellite's compile would hand every satellite a veto over the platform's delivery."
• It could not be an invariant anyway. MeshWeaver.Plugins' own CI publishes for the same identity at any later time, so "sealed ⇒ published" is not a property core can maintain alone. It would hold at seal time only — which is exactly the moment it already holds when nothing fails.

So the platform seal is unchanged, and the reconciler learned to finish a set that is missing its publication.

How the reconciler decided (retired by #6143)

The decision lived in gate, which owns every other publish decision in main-cd.yml, and it runs only on the bake_only path — the reconcile tick that finds a complete image set.

1. bake_version recovers the promoted set's release version off memex-portal-ai:<short-sha>'s tag set. (This read used to live in publish-bake's release step; it moved here because a second consumer appeared, and two independent resolutions of one fact is the shape gate.image_tag's own comment records getting wrong seven times.)
2. seal runs check-release-availability.sh <version> plugins — the same script, against the same store, that publish-bake already uses for meshweaver-content and that release.yml already uses for both sources. CD asking about only half of what it ships was the whole gap.
3. Its answer becomes gate.plugins_seal_due, and the three plugins-* jobs gain a second way in, exactly as publish-bake did in #2452.

The probe names the identity

A publication seal is per framework identity. An "is it sealed?" check that does not name one is answering confidently about something else.

The identity is the API-surface hash s<hash> of the portal's reference assemblies. It is a property of the binaries, not of the source (#1725), so it cannot be computed from a tag or a commit — resolving it means running mw-plugin-test framework-identity against the portal's /app. The one thing outside the image that knows it is the _releases/<version> marker that publish-bake-bundles.sh writes on every run, and check-release-availability.sh reads it. That is why the probe is keyed by release version and why the version had to be resolved first.

Three answers, and only one licenses a re-attempt

check-release-availability.sh deliberately keeps its refusals distinguishable, and the step keeps them distinguishable too:

🚨 The benign row is matched by its OWN wording, never by the CANNOT RESOLVE prefix. The empty-marker refusal opens with the same prefix, so a step keyed on the prefix answers a producer defect with a green. And the script asks whether the marker exists before reading it: until the #4539 review fix, a failed read — auth, throttling, network — was reported as "has no marker" and landed in the benign row, so a storage outage read as a pending bake. .github/scripts/test-cd-steps.py pins all three rows, and asserts each phrase still exists in the real script so a stub cannot describe an outcome the script no longer produces.

An absence that printed no identity is also a red: the finding names no identity and cannot be acted on.

Why a re-attempt is repair and not a retry loop

• Idempotent. publish-bake-bundles.sh skips a sealed directory whose content × framework identity already match, so a re-attempt that races a publication into place is a no-op.
• It cannot publish backwards, structurally. bake_only means check-image-set.sh found the pair tag memex-portal-ai:<core-short>-p<plugins-short> for plugins HEAD now — so the promoted set was built from exactly the commit a re-attempt would bake. A merge in MeshWeaver.Plugins moves HEAD, the pair tag stops resolving, the set reads INCOMPLETE, and the ordinary full-publish path takes over instead. On top of that the publisher's own never-seal-backwards compare still runs, unchanged — including the #4152 fix that gives it the content repository's token, without which the compare answers 404 when core bakes the private MeshWeaver.Plugins.
• Bounded. For a fixed (core sha, plugins sha) pair the bake is deterministic: it failed on infrastructure and will likely pass, or on Plugins' content and will fail identically for as long as that commit is HEAD. A fourth attempt buys no information and costs ~30 minutes of runner time an hour, so the budget is 3 per pair, the slot is consumed when an attempt starts, and a new pair resets it. The ledger is marker comments on an issue labelled cd-plugins-seal — its own label, because the ci-failure ledger closes itself when an image heal succeeds (#3176) and an image heal says nothing about a publication.
• The budget is atomic: claim, then rank. Reading the count and then appending a marker is not: schedule runs in the reconcile concurrency group but workflow_dispatch in the ref-based one, so two reconciles can overlap, both read 2, and both launch "3/3". So a run writes its claim first, then re-reads every claim for the pair in the order GitHub recorded them and proceeds only if its own falls inside the budget. Both racers compute the same order, so at most three proceed, under any overlap and without changing the workflow's concurrency model. A losing claim still counts.
• The ledger fails closed. The step runs with set -e. Without it, a failed gh issue list read as "no ledger yet" and created a second ledger at count zero every hour, and a failed marker write launched a repair that consumed no attempt — either way the bound was gone, silently.
• A failed preflight stops the legs. The three plugins-* jobs carry always() so they are not skipped behind promote on the reconcile path. always() also stops them inheriting a failed need's skip, so each asserts needs.preflight.result == 'success' — though nothing reads preflight's outputs, it is the gate that proves the run's external inputs exist.

.github/scripts/test-cd-steps.py pins all of the above, and each case was shown to fail against the defect it names: read-then-append, a rank read placed before the claim write, a seal step without -e, and a leg whose if: lost the preflight check.

And the reconcile is now judged

delivery-verdict's every-leg check was publish == 'true'-only, and its reconcile branch fires only on an incomplete set — while bake_only is by definition the complete-set branch. So a bake-only tick was judged by nothing. It now carries its own arm for publish-bake, always. The three plugins-* legs — on a publishing run and on a reconcile whose plugins_seal_due was true — are judged by report-plugins-seal, a job no platform job needs: a due leg that skipped or failed is red THERE, and recorded on the cd-plugins-seal issue, and never on the platform verdict (policy platform-backwards-compatibility; see Continuous Delivery Contract → The ladder).

The resolver waits for a seal that is still running — and has no opt-out

Since #6143 this wait is inert. resolve-platform.py still names the core job Plugins: bake + seal the publication for this identity (PLUGINS_SEAL_JOB), but no core CD run carries that job any more, so _group reads it as absent and nothing is ever held back for it. Likewise the resolver's plugins-run / plugins-sealed-at outputs can only name a core run from before #6143, not the publication MeshWeaver.Plugins seals today. The evaluation below is kept as written for that era; the plugins publication a consumer fetches is now Plugins' own.

The other side of the same seam is in resolve-platform.py. A set whose platform trio is sealed but whose own Plugins: bake + seal … job is still running is passed over (or waited for, on a release trigger with --wait-for-seal). A seal that failed, was skipped or was cancelled is terminal and holds nothing back; only the in-flight one does. The exception exists because taking the set in that window made every consumer red on upstream 'plugins' has no SEALED publication … for identity <id> (measured on MeshWeaver.Manufacturing, 2026-09-10).

The wait has a price. Measured on the core CD runs of 2026-09-27, the platform bake finished about 5 minutes after promote and the Plugins seal a further 13 to 23 minutes later (run 36299288833: promote 06:59–07:02, platform bake 07:02–07:06, Plugins seal 07:17–07:34). So a new core set becomes resolvable about 20 minutes later than its platform seal.

MeshWeaver#6041 asked whether a consumer that does not read the plugins publication could opt out of that wait. The answer is no second resolver mode, and the reason is who the consumers are. Read from each repository's default branch through the REST contents API on 2026-10-09, and re-read on 2026-10-11:

MeshWeaver.Plugins is the only consumer the opt-out could serve, and for it the opt-out is not clearly safe:

1. The same resolution feeds its publish-bake. One run resolves one set, and that set's image digest is what Plugins' own publish-bake uses to write prebuilt-bundles/<identity>/plugins/ — the same prefix core's Plugins: bake + seal job is writing in that window. Resolving before core's seal lands makes Plugins the second writer on an identity whose first writer is mid-publish. That two-publisher interleave is handled (publish-bake-bundles.sh, the two-writer postcondition, proven by test-publish-bake-overlap.py), but as a rare case. An opt-out would make it the ordinary one. Scoping the opt-out to the gate and not the bake would give one run two platform sets, which breaks the "one resolved set per run" rule the lanes assert.
2. The gain is small. Plugins no longer rebuilds per platform build. It runs on its own pushes, on a manual dispatch and once a day, so the 20 minutes only matter to a push that lands between a core promote and that set's seal — and such a run is not wrong, it tests the previous sealed set and the next run takes the new one.

So one resolver mode stays, and it is the one every reader of the publication needs. If the wait is ever to shrink, shrink it at its source — the duration of core's Plugins: bake + seal job — which helps every consumer and adds no mode. What would change this evaluation: a consumer appearing that calls the resolver, reads no upstream, and publishes nothing under the resolved identity.

What this does not cover

• The fleet-registry copy. check-release-availability.sh probes the Azure Files share, which is what /api/plugins/bundles/prebuilt/{identity}/{source} serves and therefore what the satellites' gates read. The ORAS copy pushed to cr.meshweaver.cloud is not probed, so a seal that completed the share publication and then failed its registry push (the 8711 shape) reads as available here.
• A satellite's own upstreams. A satellite that declares several upstreams still waits for each of them; that is the documented wait, not this defect.

Reading the result

The gate's step summary carries the verdict in one line, always naming the identity and the release:

🔁 re-attempt 1/3 — `plugins` is NOT sealed for `s5ec352bb…` (release `3.0.0-ci.8765`),
   so this reconcile bakes and seals it.

A satellite holding on a missing publication is the gate working. The bug was always the missing publication, never the hold.

Related

• Reading CI Signals — how to read the satellites' 404 when it happens.
• CI Content Bake — what a framework identity is and how a bake is addressed.
• Bake Publication Receipt — what a sealed publication contains.
• Module Build Architecture — the one build shape every repo runs.