Operator Credentials Travel by Name

A hosting-operator step is never handed a secret value. It is handed the Key Vault object's NAME, and it reads the value itself with the identity it already runs as. That is the one rule; the rest of this page is why it is the rule and what enforces it.

The channel a step has, and what it can carry

A plan step — hosting-backup, hosting-kv-ensure, any of the hosting-* commands — runs on one of two executors, and both give it exactly one environment:

Neither is a secret channel, and neither can become one without a second mechanism. Both are the right channel for a name: a resource group, an ingress IP, a vault object's name.

What went wrong (Memex#132)

hosting-backup, hosting-restore and hosting-verify-restore opened with

hosting::need_env PGPASSWORD "the admin password for the flexible server (mounted from Key Vault)"

and need_env's refusal ended "It is supplied by Hosting:Operator:Environment on the control instance". So the first real Backup that reached the operator failed at step 1 — and the sentence it failed with named a channel that cannot carry the value, because "supplying it the documented way" meant putting the flexible server's memexadmin password (the one credential that reaches every instance database on the server) into a ConfigMap. Every Backup, Restore, Suspend and Teardown-with-backup was blocked on that for weeks, on both executors.

The remedy that was built first was a secret channel: hosting-operator-secrets.yaml renders a SecretProviderClass and a CSI-synced Secret into the operator's namespace and publishes three coordinates so a Job could mount it and read the value from a file. It was consumed by nobody — nothing opted in, HostingOperator.Resolve never attached the class to a Job, and the scripts kept demanding the env var. Measured on the thread three times over two weeks: the chart half existed, the wiring half did not, and a values-only "opt-in" would have rendered a class no pod mounts, which syncs nothing, silently. And when the executor decision (P3a, the aks-ops lane replacing the Job) was checked, the lane had the same hole with no class to opt into at all.

The fix: the same read hosting-kv-ensure already makes

hosting-kv-ensure — two steps earlier in every Provision — already reads that exact object, by name, with that exact identity: a VALUE read of --db-password-secret in --vault, captured into a variable and never printed.

🚨 That value read is one of the four the secrets migration removes: under policy secrets-write-only-entry the identity that writes the vault may never read a value (Secrets: Write-Only Entry, Split Identities → Where the fleet stands). Until the connection string is composed without reading the admin password back, these steps keep running as hosting-operator, not as the writer.

The plan hands it --vault {record.keyVault} and --db-password-secret "$AZ_POSTGRES_PASSWORD_SECRET" (a name the record's operator.environment carries, correctly, in the plaintext channel), and it composes the instance's connection string from what it reads. So the identity holds the grant, the name is already delivered, and the read is already trusted. The three Postgres steps now do the same thing through one shared primitive in _common.sh:

hosting::pg_password <vault> <object>     # sets and exports PGPASSWORD for this process

• Both flags, or neither. --vault V --password-secret O reads the value; with neither, a PGPASSWORD already in the environment is honoured (the by-hand shape: an operator at a shell), and its absence is refused naming the two flags and the record key that supplies the object name — never the ConfigMap.
• Three answers, not two. ABSENT (SecretNotFound, or an EMPTY value) and REFUSED (Forbidden) are different sentences, through hosting::probe: a vault that refused this identity has ruled nothing out about the object (Denied Is Not Absent). An empty value is refused as loudly as an unreadable one — psql with an empty PGPASSWORD prompts, and a Job that prompts hangs to its deadline saying nothing that names the step.
• The value never leaves the process. az writes it to stdout, which is captured; it reaches pg_dump/psql/pg_restore through their environment, never an argv; nothing prints it. Both names are validated as plain identifiers before they reach a command line, at the top of each script, before anything is reported — the same boundary every other name crosses.
• A dry run reads nothing. The rehearsal reads no secret, as hosting-kv-ensure's does not.

It works on both executors because both run the same scripts under the same identity: the Job as the hosting-operator workload identity (run.sh's az login --federated-token), the lane as the hosting-operator OIDC session (aks-ops-session.sh, which hands run.sh the same token file). No class, no mount, no opt-in, no lane step, and nothing to decide between the two executors.

What holds it

deploy/aks/operator/test/run-tests.sh runs the three scripts against stub az, pg_dump, psql and pg_restore that record argv and whether PGPASSWORD reached them (as set/unset, never the value). A case on each side: the read happens by name and BEFORE pg_dump; an absent object, an empty object, a refusing vault, no channel at all, and one flag without the other each refuse before anything is dumped, saying which; a hand-set PGPASSWORD with no flags is honoured without a vault read; a dry run reads nothing; a metacharacter in either name is refused before the script reports anything. The control on the other side is the pre-change script, which refused --vault as an unknown argument and demanded the env var.

The two halves that are not in this repository

Two facts only an operator can verify, and this page does not assert: that the object the record names actually holds memexadmin's password (a Provision that composed a connection string proved it once; the record naming it is not that proof), and that the operator identity holds secret get on it in the vault the record names (the same Provision exercised it).

The login is declared too: --user is the record's databaseUsername

The password is read by name, and the LOGIN it belongs to is declared beside it. The three scripts take --user <login>; the plan passes the record's databaseUsername — the same pair hosting-kv-ensure composes a connection string from (--db-user {databaseUsername} --db-password-secret "$AZ_POSTGRES_PASSWORD_SECRET"), so a record that provisions a working portal declares a working dump login by construction. There is no built-in default. The scripts used to fall back to PGUSER, else a hard-coded memexadmin, so the step logged in as a user the record never declared — and a record whose databaseUsername disagreed with the object's owner (memex-cloud declared postgres while its vault object holds the server admin's password) could only be found on a failed run. Now: no --user and no hand-set PGUSER refuses before the vault is read; a hand-set PGUSER is honoured only when --user is absent; a --user that is not a plain name refuses before anything runs. The plan refuses a dump-based action whose record does not STATE databaseUsername (the derived fallback postgres is a guess, not a declaration).

The Azure session is re-established before any az call that follows long work

az signs in with a federated client assertion and presents that SAME assertion to Entra for every token it later needs — Key Vault, Storage, ARM. The assertion is short-lived: a GitHub Actions OIDC token is valid for five minutes. So one sign-in at the start of a run cannot carry a step that works for longer and then calls az. Measured on Ops/Actions/backup-memex-cloud-memex132-i: the vault read and the login as memexadmin succeeded, pg_dump wrote 8.4 GB in 16.6 minutes, and the upload then failed with AADSTS700024: Client assertion is not within its valid time range. The workload-identity Job has the same shape — the kubelet refreshes the projected token file, but az had read it once.

_common.sh therefore signs in from a FRESH assertion (hosting::az_signin, hosting::az_refresh):

run.sh signs in that way at the start and again before every step; hosting-backup refreshes between the dump and the upload, and hosting-restore before its download. A refresh that cannot get a fresh assertion refuses, naming where and why, rather than letting the next az call fail on a dead session with a message that names neither. A by-hand run (no run.sh session) uses the caller's own az login and is not touched. The assertion goes from its source straight into az's argument and is never printed. run-tests.sh holds it with a stub az that rejects every call on an expired assertion (AADSTS700024) and a pg_dump that expires it: both routes upload after the long dump; the controls — no fresh assertion to be had, and a single sign-in for the whole run — fail exactly as the measured run did.

The refusal has to survive the shell it is written in

"Naming where and why" was true of one failure and false of the other. A refresh fails in two places: az login rejects the assertion, or no assertion can be had at all (the OIDC endpoint does not answer, the projected token file is gone). The second was fetched inside a command substitution — assertion="$(hosting::az_assertion)" — and a command substitution is a subshell, so the cause it recorded in HOSTING_AZ_ERR never reached the shell that quotes it. Under set -u an unset cause is not an empty sentence: it ends the script. Measured on Memex run 38038325130, a Roll of memex-cloud: the migration ran, the chart was applied, and the refresh before step 3/3 died on _common.sh: line 269: HOSTING_AZ_ERR: unbound variable — the reason no assertion could be had is not recoverable from that run.

Two things follow, and both are in _common.sh:

• A helper that reports through a variable sets it IN PLACE. hosting::az_assertion puts the assertion in HOSTING_AZ_ASSERTION and prints nothing, exactly as hosting::safe_name validates in place for the same reason. hosting::az_signin clears it on every path out. Both variables are initialised when the file is sourced.
• A script that bash stops still reports a cause. The control plane quotes the first <command>: ERROR: … line of a failed run's log tail as its cause, and run.sh's step i/n '<name>' failed line as its step; bash's own line N: X: unbound variable is neither, so that run reached the control instance as a verdict with nothing behind it. hosting::on_exit is the EXIT trap of every script that sources _common.sh: on an exit that is not zero, not a hosting::die and not a deliberate exit N, it prints the refusal-shaped line — the file and line it stopped at and the command it was running, as source text (never an expanded value) — and, in run.sh, the step line. It can be the only trace: a command that dies under a 2>file redirect takes bash's own message with it.

A script with an EXIT trap of its own replaces that one, so its trap calls hosting::on_exit FIRST (trap 'hosting::on_exit; rm -f "$tmp"' EXIT); run-tests.sh refuses a trap under bin/ that does not. The suite fails the fetch on both routes, stops run.sh on a shell error mid-step, and checks the three exits that must stay silent (a refusal, a deliberate exit, success). Against the scripts as they were, the two fetch cases reproduce the measured line verbatim.

The chart's secretEnvironment block, after this

hostingOperator.secretEnvironment and hosting-operator-secrets.yaml remain in the chart for a credential a script cannot fetch by name itself. Nothing in the fleet declares a key there, and no HostingOperator.Resolve attaches the class to a Job, so a key declared today renders a class no pod mounts. Whether that block stays or goes is a separate decision; what this page settles is that the Postgres admin password is not what it is for.

The other direction: a rendered manifest is data, never log text (Memex#626)

Names flow IN. Values must not flow OUT either, and the way out is the step's own output, which becomes the job log. The mesh copies that log onto the action node, so every reader of Ops/Actions can read it. A rendered manifest carries every value the record and the Key Vault half render: Secret payloads, connection strings and, on a hand-applied object, whatever a person put in an env var. Memex#626 is that leak. An ApplyChart dry-run diff printed a live Deployment's env, an Azure Storage SAS included, into the run log and the node's callbackLogTail.

So the rule holds in both repositories:

• hosting-deploy captures helm template and the kubectl create --dry-run name listing with stdout and stderr APART, and a failure names ONE line of the tool's stderr. It used to capture with 2>&1, which folded whatever the tool had already written to stdout into the die line. test/run-tests.sh ("the rendered manifest never reaches the log") feeds a render carrying a fake SAS through a failing render, a failing rehearsal render and a failing parse. Its negative control fails all three cases against the old capture.
• The ApplyChart diff (Memex aks-ops-chart.py) reads both manifests as objects and redacts them structurally, default-deny: a value prints only when it is proven harmless, and everything else prints as a per-run keyed fingerprint.
• The sink (MeshWeaver.Plugins CredentialMask) masks credential-shaped text in every log it copies onto a node. It is a regex, so it is the second line of defence, not the first.

See also

• Denied Is Not Absent — the three answers a read owes its reader, which hosting::pg_password gives.
• Operating from the portal, not the cluster — the actions these steps run under.
• In-cluster databases — the database shape these steps do not yet serve.