Where a model's configuration lives

A model's configuration used to live in three places that could not see each other, and the symptoms — a pinned model declared unusable, a config key no template rendered, duplicate provider nodes nobody could repair, a funded key on a node and a dead one in Key Vault — were four faces of that one gap rather than four bugs. This page is the resolution: one administered home per fact, and a named path from every other place into it.

Read this before touching BuiltInLanguageModelProvider, ProviderCredentialSeed, ChatClientCredentialResolver, ModelStaticRepoSource or ModelProviderService. The operational procedures — set a key, rotate one, decouple a partition, the AKS Key Vault layering — are the /provider-keys skill; this page is why they are shaped that way.

The decision

The NODE is the administered home. Deployment configuration is SEED-ONLY. Maintainer, 2026-08-22.

resolve:  the node, and only the node
seed:     configuration → node, converging, never read again at resolve time

One place to look, one place to rotate. That is not a preference, it is what removes a whole class of failure: two live sources that disagree silently, which is exactly what cost an afternoon on 2026-08-21 when a funded OpenRouter key sat on the node while env and Key Vault carried a stale unfunded one, and every full round on ns memex returned HTTP 402.

One administered source per fact

A deployment's {Section}:* values are the bootstrap fallback for a FRESH deployment, and nothing else. Nothing in the resolution path reads them. ChatClientCredentialResolver walks node rungs only — providerRef → Provider/{name} → the legacy per-model fields — and a model whose provider node has no key is honestly unusable. (A driver factory still applies its own IOptions fallback when it actually calls out; that is the driver's business, not a source of truth this platform arbitrates.)

The seam that had to change: a seed allowed to run TWICE

The catalog's static-repo import marks each Provider/{name} SyncBehavior.ExcludeThisOnly, so the importer CREATES it once and never revisits it. That claim is what protects an admin's key edit from the next boot's re-seed — and it is also why a key added to a deployment AFTER the node existed reached every factory and never reached the node.

That is measured, not hypothetical: on the control instance, Provider/Anthropic was created keyless by system-security on 2026-08-14, Anthropic__ApiKey was configured afterwards, and the node stayed keyless until a human pasted the key in by hand on 2026-08-21. For that week the deployment could serve claude-* rounds and the platform believed it could not.

A seeder that only runs at CREATION cannot converge that. So the credential is no longer that seam's business at all: the import strips it (ModelStaticRepoSource.StripApiKey) and ProviderCredentialSeed runs on every boot of a DB-synced deployment. Its rules:

• Fill-if-absent, per field. A field is written only when the node's own value is empty. An administered value — pasted, rotated, or seeded by an earlier boot — is never touched. Running twice is not merely safe, it is the point.

• Never an unprotected key. Whatever is written goes through IProviderKeyProtector.Protect and is verified to carry the enc: tag on the exact bytes about to be persisted.

• Refuse rather than downgrade. With no Ai:KeyProtection:MasterKey the pass reports RefusedUnprotected and logs at Error. The provider stays keyless and its models are honestly unusable; it does not quietly store a live credential in the clear.

• One snapshot, sequential writes, then complete. It never reads its own writes — the reconcile-write-storm rule.

• One write per provider: the field patch, never the node written back. Each write is GetMeshNodeStream(path).Update<ModelProviderConfiguration>(…), re-checking the field under the write, and nothing follows it.

  It used to be followed by a SaveMeshNodeRequest carrying the node the update emitted, "to force persistence". Since core #5985 (1dfa0cd6ee, platform set 3.0.0-ci.9803) that message is a raw storage write only when the owning hub posts it to itself; from any other hub the owner forwards it as a checked CreateOrUpdateNodeRequest — a second, whole-node write of a snapshot, issued off-turn and with no version guard, so it is unordered against everything written after the patch. Two things followed, both measured 2026-10-02:

  • A rotation was undone by the pass. A key rotated straight after the repair was replaced by the pass's own older snapshot — node versions 3 (re-protected), 4 (rotated), 5 (the snapshot again). ARotationRightAfterTheRepair_IsNotRevertedByThePass fails 6 runs of 6 with the write-back in place and passes 40 of 40 without it.
  • main went red. The test helper that stores a plaintext key posted the same kind of save; its plaintext snapshot could land after the repair and put the key back in the clear, so the second pass repaired again and AKeyStoredInPlaintext_IsEncryptedInPlace_AndKeepsWorking reported "has not converged" — intermittently, on every main run from 15:43Z.

  That the patch alone is durable is held by reading the storage adapter in that test, on the in-memory store the suite runs on. A persist the owner did lose would heal on the next boot, since the pass decides from what the node holds.

Two replicas booting together both run it, and that is fine: whichever writes first wins and the other reports AlreadyAdministered. In the narrow window where both decide to write, both values decrypt to the same key.

The read boundary — cryptographic, because nothing else can hold

Moving a secret onto a node changes its security class. A Helm value is a deployment secret; a node's content is application data, readable by whoever can read the node — through the GUI, the REST API, MCP, exports, sync targets and any GitSync binding on the partition.

And the Provider partition must be publicly readable. Its _Policy sets publicRead: true because the model picker queries namespace:Provider nodeType:LanguageModel under the user's own identity; without it RLS filters out every model and the picker is empty. On the public instance that policy was measured still set on 2026-09-01, over seven ModelProvider nodes.

So the question is not "who may read the node" but "what does the node hand back". Three candidate controls were examined and two of them cannot work here:

The rule, stated once: a secret's administered home may be a node; its VALUE may never be what a read returns. On this platform that is enforced by encryption, not by an access rule and not by a projection.

[Browsable(false)] on ModelProviderConfiguration.ApiKey is not part of this. It suppresses the GUI property grid and nothing else — its own doc comment says the value still serializes for the factory and for persistence. A GUI that stops rendering a field is not a read boundary.

The invariant, and how it is kept

ModelProvider.apiKey is enc:-tagged at rest. Always, on every path.

Two halves, and refusing to write plaintext only ever covered the first:

1. Nothing new writes plaintext. Protect THROWS with no master key rather than passing the value through — the passthrough is what put a live OpenRouter key in cleartext into production on 2026-08-24. NoPlaintextCredentialFallbackGuard fails the build on the protector?.Protect(k) ?? k shape.
2. What is ALREADY stored converges. Refusing future writes says nothing about the past, and the platform wrote plenty before the refusal existed: on the public instance, Provider/Azure and Provider/AzureFoundry have carried the same key in the clear since 2026-08-03/04. So ProviderCredentialSeed's second pass encrypts a stored plaintext key in place — the same credential, a different storage form, so Unprotect returns exactly what it returned before and the resolver's next read serves the same key.

That second pass walks every provider node in the catalog, whether or not a catalog source still names it. The qualifier is load-bearing: the duplicate Provider/Azure node is precisely one a source-driven pass walks past, and duplicates are why a rotation's blast radius is not fixed but grows with the duplication.

Its boundary is explicit. With no master key the pass can neither repair the node nor pretend it is fine, so it logs at Error naming the exposure and leaves the credential working — a repair that took chat down to fix a storage form would be worse than the exposure, and the actual answer to an exposure is a rotation.

🚨 A key that has been exposed is rotated — encrypting it is not the remedy

Re-protecting repairs the storage defect. It does not undo the window in which the value was readable. Rotate any provider key that has been stored in the clear on a publicly-readable partition, on the exposure window alone — evidence of use is not the test. The re-protect pass logs at Warning saying exactly this, per node, so an operator finds it rather than having to know.

Thinking level — a per-model datum, sent on every call

A reasoning model's provider decides how long every round thinks before its first token, and that default is the provider's, not ours. z-ai/glm-5.3 on OpenRouter ships with reasoning always on at max; an Executive Assistant round of nine cheap tool calls took 3 min 11 s on it, 74 s of thinking before the first tool call, four times over — and the platform had no knob at all. The composer's "effort" reaches the CLI harnesses only; nothing sent a reasoning effort to a provider.

The knob is reasoningEffort on the LanguageModel node — none, low, medium, high or max (aliases minimal, xhigh). The round reads it through ChatClientCredentialResolver.ReasoningEffortFor and sets ChatOptions.Reasoning once, in the one place the round builds its options, so every provider call of the round — including the tool-loop iterations the function invoker makes — carries it; the OpenAI adapter puts it on the wire as reasoning_effort, which OpenRouter forwards to the model. Absent, nothing is sent and the provider's default stays in force, which is exactly the pre-field behaviour. An unknown spelling parses to "send nothing" and never throws, for the reason tier gives: a typo on a node costs the setting, never the model.

Tuning a model down is therefore a mesh edit, not a deploy:

patch @Provider/OpenRouter/z-ai/glm-5.3  {"content":{"reasoningEffort":"low"}}

Provider support varies: a model without a reasoning phase ignores the field; one whose reasoning cannot be disabled treats none as its lowest level. Pinned by ReasoningEffortRoundTest (node → request) and ReasoningEffortWireTest (request → wire).

Output cap — always declared, sized by the task, bounded by the model

OpenRouter reserves a call's requested max_tokens against the key's remaining daily budget before the call starts, and a call that sends no max_tokens is reserved at the model's whole output maximum (131072 for GLM-5.3). Rounds that sent nothing were refused with HTTP 402 — You requested up to 131072 tokens, but can only afford 12427 — while most of the day's budget was left, and on 2026-10-12 every agent round on two instances failed that way (#3498).

So every round sends a DECLARED cap (OutputCap), set once with the round's options, so it holds for every provider call of the round, tool-loop iterations included:

1. The task's cap: the round's own (ThreadMessage.MaxOutputTokens, e.g. the review ensemble), else the answering agent's (maxOutputTokens front matter), else the agent tier's maxOutputTokens on its ModelTier node. An agent with no tier is a chat round. A tier node that leaves the field blank inherits the shipped value for its id: utility 4096, chat 16384, reasoning 32768, review 65536, coding 32768.
2. Bounded by the model: a LanguageModel node's maxOutputTokens is the model's output limit, and no rung exceeds it.

The model's context window is never used to size a cap. Pinned by OutputCapTest and ReasoningEffortRoundTest.DeclaredOutputCap_* (node → request).

A credit refusal that names what the key can still afford is re-issued once. ProviderCallRetryChatClient reads the can only afford N figure off a 402 that arrives before any output, and when N is below what was asked it issues the call again with the cap bounded to N. There is no back-off and the transient-retry budget is untouched, but the call must still start inside the retry window. A second 402, or one without the figure, ends the round as quota-exhausted. Pinned by ProviderBudgetBoundedRetryTest.

Context window — the request is sized in tokens against the model's declared window

A round on memex sent a request of about 1.3M tokens to a model whose context window is 1,048,576, and the provider refused it after the round had paid for the attempt (#3517). Two holes let it through: the history cap counted CHARACTERS of message text only — a tool call's arguments and a tool result are not text — and it ran once per turn, while the tool loop (FunctionInvokingChatClient) appends a call and a result per iteration inside the turn.

• Declared, not derived. A LanguageModel node declares contextWindowTokens (the provider's context_length, e.g. 1048576) — data on the node, never a per-model table in code. Without it the round does no window check and the provider's refusal remains the only bound.
• Measured in tokens, every content kind. ContextBudget estimates a request from its text, reasoning, tool-call arguments, tool results, binary parts (a flat 2,000 tokens each) and tool definitions, at a pessimistic 3 characters per token: JSON, code and log lines tokenize denser than prose, and an under-estimate sends a request the provider must refuse.
• The history is trimmed to fit. The round keeps the system prompt and the newest messages within the cost cap (21,000 tokens — the old 64,000-character cap at the same rate) and, when the model declares a window, within what the window leaves after the output the round reserves and the tool definitions it sends.
• The tool loop is bounded. ContextWindowGuardChatClient sits below FunctionInvokingChatClient, outside the transient-fault retry, so it sees every request of the round. One that no longer fits is refused BEFORE it is sent, and the round ends with a message naming the model, the request's estimate, the reserved output and the window (English and German, ContextWindowTexts). The round's stamp that carries the window is removed before the provider sees the options, so it never reaches the wire.

Pinned by ContextWindowBudgetTest — composed with the real FunctionInvokingChatClient, its negative control is the same loop without the guard sending a request past the window — and TruncateToContextBudgetTest.

Customer-governed instances: the record is a seed and a ceiling

Everything above is how an instance reads its record by default: the record's {Section}:Models list IS the child set of Provider/{name}. On a customer-governed instance, policy model-settings-seed-and-ceiling (MeshWeaver.Plugins#2604) reads model settings the other way: the record SEEDS the instance's model settings and CAPS them. The instance's own administrator may narrow within the ceiling and never exceed it, and the record wins whenever the two meet, as it does for the update policy (self-update-record-authoritative).

How an instance knows. Core renders AI__ModelGovernance=SeedAndCeiling for a record that sets operator.customerGoverned: true and names at least one model in its ai block (DeploymentPortalConfig.IsModelCeiling). Every other record renders no key, and the instance reads it as Record, which changes nothing. The key is reserved for the typed fields, so an extraPortalConfig entry can neither impose a ceiling nor lift one. The vocabulary is ModelGovernance (open string constants).

The ceiling (ModelCeiling) is what the record renders, read from the same effective catalog sources the catalog seeds from: for Provider/{p}, the {Section}:Models list of the source that seeds {p}. A source whose section is named differently from its provider (CustomOpenAI seeding Provider/OpenAI), or is nested (AI:Providers:X), caps exactly the provider it seeds. The seed and the ceiling are therefore one declaration and cannot disagree. A provider the record names no models for has an empty ceiling. requiredDataResidency stays a separate, fail-closed ceiling beside this one.

A model is judged by what it runs, never by its address alone: its path, its wire id (ModelDefinition.Id, what is sent to the provider) and the provider its credential binds through (ProviderRef, else Provider/{Provider}). A node at an admitted address whose content names another id, or binds a provider the record names nothing for, is refused. Two kinds of model pass:

• a router, which only dispatches to admitted models: the platform's built-in Auto (label-pinned or not), or a node marked isRouter. Any other model filed under Provider/Auto is an ordinary model and is judged like one.
• a person's own bring-your-own-key model ({user}/_Memex/…), while the person's own key serves it. Where the credential chain resolves it through one of the instance's providers (a ProviderRef to Provider/{p}, or the conventional fallback), it runs on the instance's credential and is judged as that provider's model: the record's list and the switch-offs apply. The resolver judges with the credential it actually resolves (CredentialResolution.BoundProvider); the picker and the write guard, which have no credential snapshot, judge with the provider the content binds (ModelCeiling.BoundProvider).

What the instance may change, all of it data on nodes the instance owns, so a re-import never overwrites it:

Enforcement, on write and on read.

• Write. ModelCeilingWriteGuard refuses a person's create or update of a LanguageModel in Provider/… that the record does not name, and a tier whose model lies outside the ceiling. The refusal names the model, the provider and what the record allows, in the writer's language (ModelCeilingText, English and German). The platform's own writers pass: the catalog import seeds exactly the record's models.
• Read. A raw stream.Update patch never meets a node validator, so the ceiling is also applied wherever a model is chosen. The picker hides what lies outside (AgentPickerProjection.AdmittedByInstance with the snapshot, and the command picker through AdmittedByCeiling). The default model, tier resolution and the Auto router skip it (ChatClientCredentialResolver.CeilingAdmits). A round on a stale selection is healed onto a model inside the ceiling, or refused before any prompt is sent when none is (AgentChatClient.ModelCeilingRefusal). The image generator's guard (InstanceModelRefusal) refuses it too.
• Seed. BuiltInLanguageModelProvider seeds only the record's models in this mode. A provider's bootstrap defaults that the record does not name lie outside the ceiling, so they are not seeded.

Tier resolution gains one rung ahead of the others (ModelTierCatalog.ResolveWithInstanceChoice): the instance's choice, then the label on a model node, then the record's seed (ModelTier:*), then the deployment default. Every rung sees the same candidates, those the ceiling and the switch-offs admit. A choice the record no longer covers is therefore clamped: the next read skips it and the record's seed answers. Nothing is rewritten. If the record names the model again, the choice is honoured again.

When the record changes, the next read rebuilds the ceiling from live configuration. A model the record drops is pruned from the catalog by the import and refused at once by every read. A model it adds is seeded and offered until the administrator switches it off.

Pinned by ModelSeedAndCeilingTest. Each test has a negative control, the same instance read as Record. The tests cover the pure rule with named reasons in both languages, narrowing, the tier choice and its clamp, seeding only the record's models, the person's refused write, the resolver through seed, narrow and clamp, and a round on a model outside the ceiling that never reaches it. They also cover the ceiling keyed by the provider a source seeds (distinct and nested section names), a node whose content sends an id the record does not name (a real round that never reaches it), an ordinary model under Provider/Auto, and a personal model on the instance's credential. The fleet app /Hosting/Models says on such a record's page that it edits the ceiling, and links the instance's own providers.

What is still open

This page describes the seam. Three pieces of the administered surface are not built yet, and they are independent of everything above:

• Provenance, rendered. The resolver already tags which rung answered (providerRef:Provider/Anthropic); the Providers & Models admin surface does not show it. Since the configuration rung was removed the interesting question changed from "node or config" to "WHICH node" — the shared catalog, the user's own {user}/_Memex/…, or a legacy per-model field — and that is what the surface should name, per model.
• Per-model health. The last round's outcome per model (OpenRouter 402 credits-exhausted, DeepSeek 410 deprecated, a 401 on a rotated key) has nowhere to live and nowhere to render, so a dead model is discovered by a user mid-thread instead of by an operator.
• Write-time consistency between a provider's dialect and its endpoint. { "provider": "Anthropic", "endpoint": "https://…services.ai.azure.com/anthropic/" } saves cleanly today: the wire dialect says direct-Anthropic, the address is Azure AI Foundry, and every round ends in Azure's 404. ProviderModelLister now honours a configured endpoint so listing at least exercises the host chat will call, but nothing refuses the inconsistent pair at write time.

Three known gaps in the invariant itself, each narrower than the above and stated here so they are not rediscovered:

• The in-memory path still projects a plaintext key into served node content. BuiltInLanguageModelProvider stamps {Section}:ApiKey verbatim, and on that path the projection IS the served node. It is re-read from configuration on every read so it can never go stale, and ModelStaticRepoSource strips it before anything is persisted — but on a deployment serving Provider from memory, the publicly-readable node carries the literal. Encrypting the projection needs a stable ciphertext per read (a fresh nonce per enumeration would churn the served node), and refusing to project it without a master key would take local development offline. That trade-off is a maintainer call, not a detail.
• Two key writes still post the node back after their patch — ModelProviderService.RotateKey and ModelProviderLayoutAreas.SaveKey (and the provider-keys skill teaches the shape). It is the same whole-node write-back the boot pass dropped, so a write landing between the patch and the forwarded upsert is replaced by the older snapshot. They are left as they are for now: the post is there because "a sync-driven update does not always fire the per-node save", that claim has not been re-measured on Postgres or Orleans, and a key an administrator saved and the store never got cannot heal itself the way a boot pass does. Measure durability of the patch alone on both backends, then drop the post.
• A raw MCP update/patch of apiKey stores plaintext, because it does not route through Protect. The boot pass heals it on the next restart; until then it is exposed. Healing it at write time means a reconcile that writes into the subtree it watches, which is the write-storm shape — it would have to be built with the convergence proof that rule demands.

Related

• /provider-keys — the operational skill: set, rotate, decouple, the master key, the AKS layering.
• Doc/Architecture/PartitionSyncGuide, Doc/Architecture/StaticRepoImport — what the import claims and what "Not synced" does.
• Doc/Architecture/AccessControl — why a global admin has no data access, and why publicRead on a partition is a grant rather than a cap.