hyb-oa/.trellis/spec/hyb-oa/deployment-trialrun.md

Deployment / Trial Run

Convention: Local Release Candidate Receipt

• bash scripts/qa/run-release-gate.sh verify-candidate reuses existing static and managed full-local stages and never calls deployment. The approved R1 local integration requires this receipt in every write entry; actual push/deployment still needs separate authorization.
• release-gate-result.mjs owns an exclusive empty directory under artifacts/release-gate/, records INCOMPLETE before execution, and derives final results from fixed ordered phases, source/index/tracked and untracked input digests, current policy/toolchain, owned environment/cleanup, G19 full discovery and every local result plus witnesses/acceptance specs. Missing, truncated, linked, changed, interrupted or partial evidence is not PASS.
• qa-stage-record.mjs records full-local phases through failure cleanup. Keep plain shell calls under errexit: wrapping compound helpers in if/|| can suppress their internal failures. Preserve the original failing exit code if evidence cleanup also fails.
• Persist actual browser selection separately from discovery: exact derived spec arguments, supplementary arguments, mode, retries, forbid-only and worker count. Ordinary release verification rejects supplementary filters, shards, missing projects, changed budgets, skips, retries, expected failures and inconsistent statistics/time ranges.
• Native environment probes retain separate Chromium/WebKit observations even on launch or cleanup failure. Record the failing stage and allowlisted error category, never raw browser messages, endpoints or paths. Cleanup failures append without replacing the primary cause; every failure still rejects the environment. A Chromium trace cannot diagnose WebKit. A standalone blank-page diagnostic is not a managed gate receipt or evidence of business coverage.
• Prepare owned browser caches with python3 scripts/qa/copy-browser-cache.py <source> <new-destination> <new-receipt>. Use an explicitly selected physical cache directory. Preserve internal relative links to regular files; reject absolute, external, broken, cyclic or directory links and special files. Verify source before/after, copied bytes/modes/link structure and independent file inodes; matching expanded bytes alone is insufficient for native library aliases. Never flatten an entire bundle or execute from a historical shared cache. Keep Unix-socket runtime roots short and private; evidence directories need not be runtime directories. Copy receipts confer no release eligibility.
• A dirty development tree may have localChecksPassed=true, but returns PARTIAL/exit 2 and deploymentEligible=false. Offline verify re-reads and hashes evidence and recomputes against the current checkout. Local proof does not discharge the 12 same-candidate public obligations, independent reproduction, or operator approval.
• Existing project-wide skip flags accept only exact 0/1. Each or both require nonempty reason/operator/approval reference, persist affected stages, and end PARTIAL/exit 2. Other release modes reject them before stage side effects. These metadata do not grant authorization; no approval time is invented.
• Validate with pure receipt counterexamples, actual shell/fake-tool invocation, source mutations, and existing deployment lifecycle/safety regressions. Fake tools must intercept every external action. Evidence is auditable, not a cryptographic signature.
• Explicit consumption uses RELEASE_GATE_RECEIPT_DIR (absolute or repository-relative, confined beneath artifacts/release-gate/); RELEASE_GATE_ARTIFACT_DIR only selects new output. Never search for a latest receipt. release-gate-result.mjs require re-reads all evidence and current source/policy/toolchain; caller PASS/skip flags do not grant eligibility. The three deployment entries are part of the policy digest.
• ci-release/all reuse a supplied eligible receipt or run the verification-only child once, then call production identity bootstrap. all retains its extra smoke-local before that verification. Explicit invalid/empty receipts fail without fallback tests. deploy and direct wrappers require a receipt and never start full tests themselves. Repeated lower-level assertions only revalidate evidence.
• Pin the returned commit across bootstrap and wrapper preflights with RELEASE_GATE_EXPECTED_COMMIT; it is an additional equality constraint, never an override. Match provenance/archived SOURCE_COMMIT to it. Recheck before each lower entry and before the first remote write (the mkdir preceding rsync), not merely before container work. Initial rejection must precede SSH/config/provenance-fetch access; do not invoke failure collectors that read real configuration for a rejected publication.
• Both read-only flags accept only exact 0/1. Bootstrap identity-only exits before database lookup; bootstrap/existing preflight-only exits before deploy.sh. Release-runner publication modes and raw deploy.sh reject either read-only flag rather than treating it as write permission or publication success.
• Add boundary checks with synthetic disposable repos, private child HOME/PATH, audited fake config and fail-closed SSH/rsync/curl/git-fetch/Docker tools. Execute actual local entry scripts; remote-body lifecycle tests remain separate. Source mutations must expose an observable violation, not just remove a matching string. Freeze and commit the final reviewed candidate before the single full release run; task-only records and old partial receipts cannot substitute for it.

Convention: Managed OCR And Concrete Visual Coverage

• After a successfully owned local server start, run-web-gates.sh recalculates PLAYWRIGHT_OCR_MOCK from that startup's OCR_PROVIDER=mock and JOBS_ENABLED=true; a caller's flag cannot enable unsupported capability. Each startup appends non-secret provider/job facts to server-capabilities.jsonl in the run report. Actual missing-field correction still has to pass; configuration alone is not behavior evidence.
• R108 uses an active synthetic reimbursement submitted through formal APIs, with a valid PDF and explicit item binding. Initialization fails the test rather than skipping it. Preserve the employee approval refusal, current approve/return and no-reject contract, offline upload queue, 390/414 layouts, and independent output paths. An incomplete visual report is not PASS just because findings is empty.
• The existing project visual matrix explicitly lists all 14 current system routes. Unknown wildcards/parameters and unowned system routes fail route-plan preparation. AP uses finance, system/audit uses admin, approval uses manager, and login uses an anonymous context. Require exact target URL plus a page-specific title or workspace anchor; 403/404/login fallback and same-origin API 403/404 are failures, not target page coverage. Mock response controls must declare UTF-8 so Chinese error headings exercise the intended guard.
• Per-route arrival and full audit completion are separate: record completion only after the screenshot and all DOM checks. The approved matrix is 25 PC + 8 mobile pages, grouped into seven role/viewport tests with 240 seconds each (per-test override; the Playwright project default stays 180 seconds and remains a receipt check) and one timed synthetic preparation. Preserve all assertions, screenshots, configured workers and zero retries; only this file's groups remain serial as before. Compare exact planned/completed route-role-viewport identities, reject missing/duplicate/unexpected pages and wrong arrivals, and require all seven actual test statuses passed. Failed or skipped serial successors must leave the report incomplete; unrelated guard tests cannot overwrite it. Matrix coverage does not mean every product page. Live/frozen obligations remain in full discovery; focused passes do not change ordinary eligibility.

Public OA Boundary

Public OA is quasi-production. Deployments, remote config changes and live data operations require explicit approval and evidence.

Rules

• Existing installations use DEPLOY_SERVER=<deploy-ssh-target> DEPLOY_REFRESH_SERVER=<forced-command-ssh-target> bash deploy/trialrun/deploy-existing.sh only when deployment is approved; the wrapper reads the frozen allowlisted contract, keeps SYNC_ENV=0, performs aggregate read-only preflight and refreshes fixed backup build drop-ins after release. First initialization or controlled recovery without that unit uses deploy.sh with explicit absolute data paths and DEPLOY_DATA_MODE; neither path overwrites remote .env by default.
• When server/src/config.js reads a new environment key, add the same key to deploy/trialrun/.env.example before deployment; deploy/trialrun/deploy.sh runs node deploy/trialrun/validate-env-contract.js at step 0 and must fail if the example contract drifts.
• Current backup policy is manual Tencent Lighthouse snapshots only, as selected in 08-28-production-arm-backup-transfer-incident/prd.md. Do not introduce object storage, ARM/hyinda-20c backup or automatic snapshot rotation by default. Before each approved production deployment, confirm the OA target and coverage of database/attachment disks, then wait for snapshot creation to succeed. Record target/coverage confirmation, creation time and success without resource IDs. Check current console quota before any separately authorized old-snapshot deletion; preserve an existing successful recovery point during rotation. Recoverability is limited to existing snapshot times and covered disks; this policy provides neither a daily recovery-point guarantee nor an independent copy.
• Policy selection does not prove automatic backups have stopped. Follow the same task's reviewed transition plan: preserve the service-based deployment identity contract, adapt backup timer checks before disabling those timers, and separately approve remote configuration changes. Preserve historical backups and independent public-health monitoring. Legacy backup contracts below describe retained tooling; they do not authorize re-enabling backup services.
• After deployment, report commit hash, /ready, public smoke results and whether the user can test.
• Real business writes are manual unless the current Trellis task explicitly authorizes a controlled test write.
• Do not execute real payments, import bank statements, connect payment providers or write ERP without a separate approved Trellis task.

Scenario: Deployment Seed Separation

1. Scope / Trigger

Daily releases preserve operator-maintained configuration. This local candidate contract does not assert production has been deployed.

2. Signatures

• Shared entry: cd server && npm run deploy:init.
• DEPLOY_DATA_MODE=existing|initialize is mandatory; no default.
• deploy.sh passes mode via Compose run -e; Compose init calls the same entry. With entrypoint: ["sh", "-c"], command must be a one-element list containing the entire shell expression, not a folded scalar subject to argv splitting. Verify entry reachability and propagation of a nonzero child exit, not only the YAML text or Compose config rendering. Approved direct calls use run --rm --no-deps -T -e DEPLOY_DATA_MODE=existing init with established project/file/env selectors and an already healthy database.

3. Contracts

• Validate mode before subprocess/DB access; then config precheck, migrate deploy, initialize-only seed:config/seed:trialrun, and read-only configuration checks.
• Existing never invokes either seed, even on missing configuration. Migrations are not read-only. No implicit repair or mode fallback.
• Require nonzero counts of enabled expenseType, enabled approvalRule and active reimbursement workflowDefinition. Mirror current reimbursement predicates. No forced v1, roles, thresholds or values equal to seed defaults.
• Bounded repeatable-read transaction explicitly sets READ ONLY. Output fixed check codes/counts/maintenance hints only, never row payloads.
• Empty core configuration blocks switching, including intentional disablement. Nonempty checks do not prove amount coverage, assignees or business readiness.
• Workspace/sequence fallbacks, optional flags/calendar and employee/account readiness stay outside this gate; do not create missing records.

4. Validation & Error Matrix

• Missing/invalid mode → deployment.data_mode_required before migrate/seed.
• Subcommand failure → deployment.init_command_failed; no later stages.
• Missing core → REQUIRED_CONFIGURATION_MISSING with safe check list, then deployment.required_configuration_missing; never auto-seed existing data.
• Query/connection errors → deployment.init_failed, nonzero, no raw exception.
• INIT_PASS proves only this init/check stage, not public release success.

5. Good / Base / Bad Cases

• Good: existing custom flags/workflows/calendars and versions survive.
• Base: explicitly initialize an empty synthetic database with seed in order.
• Bad: repair missing records by replaying seed, or default to initialize.

6. Tests Required

• deploy-init.test.js mode/order/errors/read-only checks join the static gate.
• Keep deployment-db-lifecycle and deployment-safety-contract regressions.
• Opt-in deploy-init.integration.test.js uses DEPLOY_INIT_TEST_DATABASE_URL: loopback, uniquely named hyb_seed_test_* empty synthetic DB. No production default URL, reset, real archive or remote access. Operator owns exact cleanup. Missing URL means skipped, not integration PASS.
• Test actual initialize, repeated existing execution, hashes of 17 config models including timestamps/versions, and each missing-core failure.

7. Wrong vs Correct

Wrong: values differ from defaults → overwrite them on every deployment. Correct: preserve values; stop only for approved minimum missing-core checks. Expense types require authorized configuration repair; do not invent a UI route. Approval rules and workflow definitions have existing admin pages. Raw seed commands remain available for separately approved initialization; this does not prevent an administrator explicitly running seed. Old-image rollback cannot reverse migrations or initialization data.

Scenario: Existing Application Release Must Not Manage Database Containers

1. Scope / Trigger

• Applies to deploy-existing.sh → deploy.sh and direct deploy.sh execution.
• Separate application publication from database container lifecycle management; this does not make migrations or seed operations read-only.

2. Signatures

• DEPLOY_DATA_MODE=existing|initialize; the existing wrapper pins existing.
• Only initialize may explicitly run Compose up -d db.
• One-shot init: run --rm --no-deps -T --entrypoint sh init ....
• Application: explicit build server, then up -d --no-build --no-deps server; optional worker and explicitly selected MinIO services also isolate dependencies.
• Database readiness: timeout --signal=KILL 120 bash -s -- <compose arguments>; poll every 2 seconds with no environment override for the 120-second deadline.

3. Contracts

• Existing releases may query database state but must not start, stop, restart, recreate, replace or remove db, explicitly or via dependency propagation. A project-wide lifecycle operation is not safe merely because it omits db.
• Resolve exactly one container through the approved Compose project/service. Inspect its running state and Docker health, replacing the service_healthy startup guarantee otherwise lost when dependency startup is disabled.
• The deadline encloses both polling and potentially stuck Compose/inspect calls. Missing, stopped, unhealthy, ambiguous or uninspectable db must fail; only running|starting waits, and only running|healthy proceeds. No automatic initialize fallback, restart, permission repair or health-bypass option.
• Initialize starts db explicitly, then passes the same health gate before init. Reuse the gate after init build, after init and after server build, before each application switch; a healthy snapshot before a long build is not sufficient. Prepare worker before switching server: resolve its existing container through Compose, inspect Config.Image and verify that image reference, not just a retained image ID whose tag may be gone. Build explicitly when no container/reference is available, then check health and start with --no-build --no-deps. A cached image without a container may incur an extra cached build. This assumes the current build-only worker definition; future image overrides need their own compatibility review, rather than treating a previous container reference as new configuration. Preserve migrations and initialize-only seed order through the shared deploy:init entry above, safe logs, identity/path preflights, rollback window, build markers and backup-contract refresh.
• Missing timeout tooling or unsupported isolation options fail closed. Do not retry a command without isolation to accommodate an older Compose executable.

4. Validation & Error Matrix

• Missing/invalid mode → nonzero before container operations.
• Missing timeout tool → db_health_timeout_unavailable, no db start or init.
• Missing/non-unique db → db_container_missing_or_ambiguous.
• Query failure → db_health_query_failed; invalid/unhealthy state → db_not_healthy.
• Deadline → db_health_timeout, nonzero; never continue init/application switch.
• Init failure → preserve failure status and stop before application switch.

5. Good / Base / Bad Cases

• Good: existing healthy db → isolated init → isolated application switch; captured effective lifecycle targets never include db.
• Base: initialize → explicit db start → bounded healthy check → isolated init.
• Bad: remove only the explicit up db but leave init run or application up free to start dependencies; grep for one --no-deps cannot prove safety.

6. Tests Required

• Execute the actual remote shell from current source in a private temporary fixture; capture argv and resolve effective service targets from the real Compose dependency graph. Assert execution reaches init and expected services.
• Cover V1/V2, both modes, worker/MinIO profiles and failure cases; mutate explicit db startup and each init/server/worker dependency isolation independently.
• Change db health after build/init/server-switch stages and prove the next guarded action never runs; verify image builds cannot follow the last health check implicitly inside application up.
• Cover a retained worker container/image ID with a missing launch tag; rebuild must happen before the server switch, not fail later at isolated worker startup.
• Test the real deadline mechanism with a shortened deadline only in a temporary source copy, including a hung inspect; never shorten the production constant.
• Fake binaries and whitelisted PATH/env exist only in test subprocesses. Unknown commands fail, real Docker/SSH/production credentials remain inaccessible.
• Avoid a cold Node runtime per fake command: a per-test private Unix socket reuses the same command parser and trace semantics. The isolated Python stdlib client preserves argv, requested stdin and status; transport failure exits 97 without a real-command fallback. Record timeout before directly executing GNU timeout, and close all owned connections before removing the temporary fixture. Keep the original workers, deadlines and full-chain/mutation assertions.
• Run server/tests/unit/scripts/deployment-db-lifecycle.test.js in static gates and retain existing deployment-safety-contract.test.js coverage.

7. Wrong vs Correct

• Wrong: database unavailable → application deploy silently runs up db.
• Correct: database unavailable → stop and request separate fault recovery.
• First authorized real release needs an operator, pre/post db container identity comparison and separate application-image/tool recovery references. Keep the old tool source as evidence, not a ready-to-run command sequence that reintroduces unapproved db lifecycle operations or replays migrations/seeds.

Scenario: Exact Production Identity Bootstrap Before Migration Status

1. Scope / Trigger

• Trigger: the public environment has a legacy name that resembles a trial or staging environment, or the production database identity is intentionally absent from the repository.
• Purpose: identify the one approved deployment by independent exact values without treating a name token as proof for or against production.

2. Signatures

• Bootstrap entry: node deploy/trialrun/production-identity-bootstrap.mjs.
• Identity-only stop: PRODUCTION_IDENTITY_ONLY=1 node deploy/trialrun/production-identity-bootstrap.mjs.
• Read-only stop: PRODUCTION_READ_ONLY_PREFLIGHT=1 node deploy/trialrun/production-identity-bootstrap.mjs.
• Exact local inputs passed to the existing wrapper: EXPECTED_DEPLOY_SERVER, EXPECTED_REMOTE_DIR, EXPECTED_COMPOSE_PROJECT_NAME, EXPECTED_PRODUCTION_ORIGIN and EXPECTED_PRODUCTION_DB_NAME_SHA256.

3. Contracts

• Recover deployment target and Compose project from reviewed Git history. A shell ${REMOTE_DIR:-...} default that operations routinely override is a fallback, not proof of the production repository root.
• Select the refresh identity only when exactly one concrete SSH candidate returns the exact forced-command probe token. Do not print candidate aliases.
• Compare target and project against their independent history. Use a structurally validated systemd contract only to locate the remote env/runtime, verify both origin samples, then freeze its directory as the exact expected value for later preflight. Missing or empty expected values are errors, not reasons to skip a comparison.
• A legacy token such as trialrun is neutral. Continue to reject loopback/zero addresses, local regression, staging, shadow and placeholder identities.
• Identity-only mode must stop after env/runtime origin agreement and contract-directory freeze; it must not read POSTGRES_DB, run aggregate checks or enter Prisma status.
• Read POSTGRES_DB only after history target/project, systemd contract structure and both public-origin samples agree. Keep the database name remote; pass only a hash comparison baseline to the preflight.
• Read-only mode must exit after target/pre-status and rollback-window checks and before deploy.sh, rsync, migration or application switching.

4. Validation & Error Matrix

• Zero refresh probe matches -> REFRESH_IDENTITY_NOT_FOUND; stop before target gate.
• Multiple exact probe matches -> REFRESH_IDENTITY_NOT_UNIQUE; do not select the first candidate.
• Historical target/project or contract-relative-path disagreement -> an explicit HISTORICAL_* or SYSTEMD_*_MISMATCH; stop before database identity read.
• Any expected exact value missing/empty -> EXPECTED_*_MISSING; no SSH preflight fallback.
• Env origin or running-container origin differs from the approved origin -> PROD_TARGET_*_ORIGIN_MISMATCH; do not read migration status.
• Read-only mode reaches deploy.sh -> contract failure; tests and env validation must reject the ordering.

5. Good/Base/Bad Cases

• Good: a legacy-named deployment passes target/project checks, both origin samples match, the contract directory becomes the exact in-process baseline, then identity-only mode exits before DB/status reads.
• Base: an explicitly supplied, unambiguous production identity passes the same exact comparisons.
• Bad: reject or accept solely because the alias/project contains or omits trialrun.
• Bad: database name lookup fails and the wrapper continues without comparing database identity.

6. Tests Required

• Positive fixture for the approved legacy naming pattern.
• Separate negative assertions and stable error codes for deploy target, Compose project, contract-relative env path and later preflight directory mismatches.
• Missing-value tests proving exact comparison cannot be skipped.
• Zero/multiple refresh-probe fixtures.
• Ordering assertion: contract structure -> env/runtime origin -> contract-directory freeze -> identity-only exit; only a later approved run may read DB -> target/pre-status/window -> read-only exit -> deploy.sh when read-only mode is off.
• Evidence assertions proving target/path/database values are absent from stdout and committed reports.

7. Wrong vs Correct

Wrong

name does not contain staging -> assume production -> run migrate deploy

Correct

reviewed target/project + unique refresh probe + contract structure
-> env/runtime origin agreement -> freeze contract directory -> identity-only stop
-> separately approved remote-only DB identity hash -> target/pre-status/window
-> explicit read-only stop or separately approved deploy

Scenario: Existing Deployment SSH Identity Disambiguation

1. Scope / Trigger

• Trigger: DEPLOY_SERVER / DEPLOY_REFRESH_SERVER are not injected and local SSH configuration exposes more than one alias that can read the frozen deployment contract.
• Purpose: select the existing least-privilege deployment account without guessing from alias names or accidentally publishing as unrestricted root.

2. Signatures

• Contract reader: deployment_contract_load <ssh-target> <systemd-unit> <assoc-array>.
• Aggregate preflight: DEPLOY_SERVER=<target> DEPLOY_REFRESH_SERVER=<target> bash deploy/trialrun/check-existing-deployment.sh.
• Refresh probe: ssh <refresh-target> probe must return exactly [deployment-contract-refresh][PASS] probe.
• Deployment entry: bash deploy/trialrun/deploy-existing.sh with the two resolved targets kept only in process environment.

3. Contracts

• Never choose a target from its alias name, apparent hostname or ordering in SSH config, and never print candidate targets while probing.
• A normal deployment candidate must be non-root, own the contracted REMOTE_DIR, REMOTE_ENV_FILE and synchronized source files, and have Docker access. These checks use numeric UID equality on the remote host.
• An unrestricted root alias is not a valid normal deployment target even if it can read the same contract and pass the aggregate checks. Publishing as root can change synchronized-file ownership and break the intended least-privilege path.
• Multiple aliases may point to the same machine and contract. Select a target automatically only when exactly one candidate satisfies the full non-root ownership contract; otherwise stop for explicit local configuration guidance.
• The refresh identity is separate and valid only when the exact forced-command probe response matches. Do not reuse an unrestricted root login as the refresh identity.
• After deployment, assert the source files still belong to the selected deployment UID before recording success.

4. Validation & Error Matrix

• No contract-readable candidate -> stop before remote mutation.
• Exactly one non-root owner candidate + one exact refresh probe -> continue to 13/13 aggregate preflight.
• Several contract-readable aliases, but only one non-root owner -> use the owner candidate; do not treat alias count alone as ambiguity.
• Several non-root owner candidates -> stop and request local SSH guidance; do not choose by config order or connection speed.
• Root candidate passes 13/13 -> exclude it from normal deployment selection.
• Post-deploy ownership differs from the selected UID -> fail closeout and investigate; do not report the release as clean.

5. Good/Base/Bad Cases

• Good: two aliases reach the same host and contract; the directory-owning non-root account is uniquely selected and ownership remains unchanged.
• Base: one normal deployment identity and one forced-command refresh identity are injected explicitly; no discovery is needed.
• Bad: use an unrestricted root alias because it also returns 13/13 PASS.
• Bad: select the first matching Host entry or expose aliases, usernames, paths or SSH configuration in task evidence.

6. Tests Required

• Confirm the selected normal identity has a non-zero UID and numerically owns the contracted repository, environment file and a synchronized source file.
• Confirm Docker access succeeds without sudo and the aggregate preflight is exactly PASS=13 FAIL=0.
• Confirm the refresh identity returns the exact probe token and the normal identity is not substituted for it.
• After release, compare the synchronized source owner UID with the selected identity UID and require equality.
• Persist only PASS/FAIL counts and the selection rationale; run the sensitive content gate before commit.

7. Wrong vs Correct

Wrong

# Both aliases connect, so choose the first one (which may be root).
DEPLOY_SERVER="${candidates[0]}" bash deploy/trialrun/deploy-existing.sh

Correct

# Resolve exactly one non-root candidate that owns the frozen contract paths,
# then run the aggregate gate before the controlled deployment wrapper.
test "${#owner_candidates[@]}" -eq 1
DEPLOY_SERVER="${owner_candidates[0]}" \
DEPLOY_REFRESH_SERVER="$forced_refresh_target" \
bash deploy/trialrun/check-existing-deployment.sh

Scenario: Deployment Env Contract

1. Scope / Trigger

• Trigger: any backend config, deployment compose, or trialrun env wiring change.
• Purpose: prevent deploy-time surprises where local code compiles but the deployment contract rejects the release or silently misses a runtime default.

2. Signatures

• Command: node deploy/trialrun/validate-env-contract.js
• Deployment entry requires SYNC_ENV=0, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR and DEPLOY_DATA_MODE=existing|initialize.
• Contract sources:
  • server/src/config.js
  • deploy/trialrun/.env.example
  • deploy/trialrun/docker-compose.yml

3. Contracts

• Every process.env.KEY or requiredEnv("KEY") in server/src/config.js must be present in deploy/trialrun/.env.example.
• server, worker, and init services must load ./.env through env_file.
• Compose service-level environment overrides are limited to the explicit allowlist in validate-env-contract.js.
• SYNC_ENV=0 is the default public deploy mode; it does not overwrite remote .env.
• Relative data-path defaults are allowed only for direct local Compose. deploy.sh requires explicit absolute host paths and a data mode; existing verifies the PGDATA directory contains PG_VERSION, while initialize is the only mode allowed to create missing directories.

4. Validation & Error Matrix

• Missing key in .env.example -> validate-env-contract.js exits non-zero.
• Service omits env_file: ./.env -> contract exits non-zero.
• Unauthorized compose override -> contract exits non-zero.
• Remote .env missing -> deploy step 4 exits before migration.
• Missing/relative/unsafe/overlapping data paths or missing data mode -> deploy exits locally before provenance, synchronization or SSH.
• existing target missing PG_VERSION or storage directory -> remote deploy exits before Compose starts DB.

5. Good/Base/Bad Cases

• Good: add EMAIL_LOGIN_CODE_TTL_SECONDS to config.js, document it in .env.example, then run contract validation before deploy.
• Base: code-only feature without env changes still runs the contract gate.
• Bad: add a config key only in config.js; deploy fails at step 0.

6. Tests Required

• Always run node deploy/trialrun/validate-env-contract.js after env-related changes and before public deployment.
• For public deploys, also run or verify /ready, runtime smoke, external deps smoke, public bundle copy scan, and live smoke when credentials exist.

7. Wrong vs Correct

Wrong

# Config key exists only in server/src/config.js.
SYNC_ENV=0 bash deploy/trialrun/deploy.sh

Correct

# Add the key to deploy/trialrun/.env.example first.
node deploy/trialrun/validate-env-contract.js
SYNC_ENV=0 POSTGRES_DATA_DIR=/srv/hyb-oa/pgdata \
  STORAGE_HOST_DIR=/srv/hyb-oa/storage DEPLOY_DATA_MODE=existing \
  bash deploy/trialrun/deploy.sh

Scenario: Candidate Build Identity Single Source

1. Scope / Trigger

• Trigger: changing candidate provenance, deploy.sh, production preflight or Compose build-SHA wiring.
• Purpose: prevent the full commit, its 12-character public prefix and the two Compose environment keys from drifting between deployment entry points.

2. Signatures

• Shared function: deriveCandidateBuildEnv(commit) -> Object.freeze({ BUILD_SHA, VITE_BUILD_SHA }).
• Provenance CLI keeps commit and shortCommit and adds the same buildEnv object.
• Consumers: deploy/trialrun/deploy.sh and deploy/trialrun/production-preflight.mjs.

3. Contracts

• The shared function accepts only a lowercase 40–64 character hexadecimal commit and returns both environment values in one object; VITE_BUILD_SHA is exactly the first 12 characters of BUILD_SHA.
• Consumers traverse and inject the complete object. They must not call git rev-parse HEAD, slice the commit again, define an independent key list or selectively consume one build key.
• production-preflight-remote.sh remains an independent consumption guard: before docker compose config and Prisma status, it requires both keys, validates both formats and checks the prefix relationship. A single local source does not replace transport and caller validation.
• Preflight writes the stable redacted summary and report reference before it classifies a remote failure. A non-zero remote exit takes precedence over a missing Prisma marker.

4. Validation & Error Matrix

• Invalid candidate commit -> CANDIDATE_COMMIT_INVALID locally; no SSH.
• Missing or malformed remote build value -> BUILD_SHA_MISSING, PROD_TARGET_BUILD_SHA_INVALID or PROD_TARGET_VITE_BUILD_SHA_INVALID; no Compose/Prisma call.
• Valid values with a mismatched prefix -> PROD_TARGET_BUILD_SHA_MISMATCH; no Compose/Prisma call.
• Compose config failure -> PROD_TARGET_COMPOSE_CONFIG_INVALID; no Prisma status or migration command.
• Remote non-zero with code=... -> propagate that code even without a Prisma marker or with an unreadable marker body; remote non-zero without a parseable code -> PRODUCTION_PREFLIGHT_REMOTE_FAILED.
• Remote zero without a Prisma marker -> PRISMA_STATUS_OUTPUT_MISSING.

5. Good/Base/Bad Cases

• Good: provenance derives one frozen object; both entry points inject all of it; the remote helper independently validates what arrived.
• Base: the existing provenance CLI fields remain compatible and equal the two object values.
• Bad: preflight uses candidateCommit.slice(0, 12), or deployment reads only one property and reconstructs the other.
• Bad: check the Prisma marker before a non-zero remote status and thereby hide the remote safety code.

6. Tests Required

• Unit-test fixed keys, immutability, the 12-character relationship and invalid commit inputs.
• Execute the real production-preflight Node subprocess behind fake SSH for all four status/marker quadrants: non-zero with code and no marker, non-zero with code and marker, non-zero without code, and zero without marker. Assert the stable summary and report reference exist on every failure.
• Run the remote helper behind an allowlisted fake Docker and prove build-value and Compose failures occur before any Prisma exec or status call.
• Mutation-check both historical defects: reversing status/marker precedence and removing either shared build key must make the relevant tests fail.

7. Wrong vs Correct

Wrong

const buildSha = candidateCommit;
const viteBuildSha = candidateCommit.slice(0, 12);

Correct

const candidateBuildEnv = deriveCandidateBuildEnv(candidateCommit);
const assignments = Object.entries(candidateBuildEnv);

Scenario: Public Smoke Navigation Labels

1. Scope / Trigger

• Trigger: frontend shell, navigation labels, sidebar grouping, workbench copy, or live smoke assertions change.
• Purpose: prevent public deployment from being marked failed only because smoke tests look for stale navigation copy while the page itself is healthy.

2. Signatures

• Live smoke command: E2E_MODE=live LIVE_SMOKE_BASE_URL=https://oa.hyinda.com npx playwright test tests/e2e/live-smoke.spec.ts
• Shared assertions: client/tests/e2e/helpers/smokeAssertions.ts
• Workbench route: /portal
• Current primary workbench nav label: 工作台
• Stale user-facing label that must not be accepted by current mocks or smoke: Portal

3. Contracts

• assertPortalLoaded(page) must assert route and page content first: /portal URL and the 今日待处理 heading.
• Navigation-label assertions for the workbench link must require the exact current label 工作台; mocks and fixtures must use the same contract.
• Navigation assertions must be role-aware. Admin/manager smoke may assert team/admin entries such as 团队, but ordinary employee smoke must not require links that NavLinks intentionally hides for that role.
• Smoke tests must not require a copy string that is not a product contract for public availability.
• If the product intentionally renames the primary workbench label again, update the product contract, fixtures and shared smoke helper together rather than adding an ad hoc fallback in one live smoke test.

4. Validation & Error Matrix

• Page reaches /portal, 今日待处理 is visible, but nav label is 工作台 -> smoke should pass.
• Page reaches /portal and renders Portal as the user-facing nav label -> smoke should fail as a stale-copy regression.
• Employee reaches /portal, role-aware navigation hides 团队, and employee pages such as 项目 remain reachable -> smoke should pass.
• Page reaches /portal, but 今日待处理 is missing -> smoke should fail.
• Page stays on login, /403, error fallback or another route -> smoke should fail.
• New shell label appears only in one live smoke test -> stop and align the product contract, fixtures and shared helper before changing smoke.

5. Good/Base/Bad Cases

• Good: assertPortalLoaded checks /portal, 今日待处理, and an exact 工作台 workbench link.
• Good: assertMainNavVisible(page, { expectTeam: false }) is used for ordinary employee live smoke.
• Base: the internal route and helper name may retain portal, while all user-facing navigation fixtures and assertions use 工作台.
• Bad: a mock emits Portal and smoke accepts it as a valid current label, masking a user-facing English-copy regression.
• Bad: employee live smoke requires the 团队 sidebar link after role-aware navigation intentionally hides it.

6. Tests Required

• After changing live smoke assertions, run: cd client && npm run typecheck
• Load the focused visual and shell specs with Playwright --list when changing fixture/report contracts.
• For public deploy verification, run live smoke or the readiness wrapper: LIVE_SMOKE_BASE_URL=https://oa.hyinda.com bash scripts/qa/run-live-smoke-readiness.sh
• When changing the shell or Portal page, include a focused Playwright check that proves the workbench route and heading still load.

7. Wrong vs Correct

Wrong

await expect(
  page.getByRole("link", { name: "工作台", exact: true })
    .or(page.getByRole("link", { name: "Portal", exact: true }))
    .first(),
).toBeVisible();

Correct

await expect(page.getByRole("link", { name: "工作台", exact: true }).first()).toBeVisible();

Scenario: Trial Run Runtime Navigation Baseline

1. Scope / Trigger

• Trigger: changing sidebar grouping, workspace module visibility defaults, server/src/services/workspaceSettings.js, or server/src/scripts/smoke-trialrun-runtime.js.
• Purpose: keep deployment smoke aligned with the current small-team navigation model instead of blocking deployment on retired menu assumptions.

2. Signatures

• Runtime smoke command inside deployment: docker compose exec -T server node src/scripts/smoke-trialrun-runtime.js
• Source of truth for sidebar grouping: server/src/services/workspaceSettings.js
• Runtime smoke assertion file: server/src/scripts/smoke-trialrun-runtime.js

3. Contracts

• Runtime smoke must distinguish primary sidebar entries from collapsed 更多 entries.
• Core day-to-day ledgers such as 报销, 采购, 合同, 项目 and 资产 are expected in main when the active role has access.
• Low-frequency entries such as 周报 and 公告 may be enabled but should be checked in backup / 更多, not forced into main.
• 待办 follows the current workspace and role-aware navigation baseline; do not move it between main, backup or hidden as part of a deployment-smoke fix unless a product task explicitly changes that rule.
• Management pages should remain represented by the single 管理 entry unless a current task explicitly changes the management navigation model.

4. Validation & Error Matrix

• Core ledger missing from main -> runtime smoke fails with 系统导航 main 缺少 <key> 入口.
• 周报 or 公告 missing from backup while enabled -> runtime smoke fails with 系统导航 backup 缺少 <key> 入口.
• Management children are hidden behind 管理, but smoke still expects workspace, people, featureFlags or auditLogs in admin -> update the smoke baseline; do not re-expand the sidebar for the smoke.
• systemSettings missing from admin -> runtime smoke fails because the management entry is no longer reachable.

5. Good/Base/Bad Cases

• Good: main includes portal, reimbursements, purchaseRequests, contractsRegistry, projects and assets; backup includes weeklyReports and announcements; admin includes systemSettings.
• Base: 待办 is in main for a configured admin/owner trial account; smoke accepts it there because it follows the current workspace baseline.
• Bad: smoke expects weeklyReports in main after the sidebar rule moved low-frequency modules to 更多.
• Bad: smoke expects every management child as a sidebar row after the product rule consolidated management into one 管理 entry.

6. Tests Required

• Run node --check server/src/scripts/smoke-trialrun-runtime.js after editing runtime smoke.
• Run SYNC_ENV=0 bash deploy/trialrun/deploy.sh for public deployment so the remote runtime smoke validates the actual workspace and feature-flag state.

7. Wrong vs Correct

Wrong

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "weeklyReports",
  "announcements",
];
const REQUIRED_ADMIN_NAV_KEYS = ["workspace", "people", "featureFlags"];

Correct

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "purchaseRequests",
  "contractsRegistry",
  "projects",
  "assets",
];
const REQUIRED_BACKUP_NAV_KEYS = ["weeklyReports", "announcements"];
const REQUIRED_ADMIN_NAV_KEYS = ["systemSettings"];

Scenario: Immutable Deployment Provenance And Dual Build SHA

1. Scope / Trigger

• Trigger: changing the trialrun deployment entry, Docker build, Compose wiring, /health or /ready, public bundle scanning or live readiness.

2. Signatures

• Provenance gate: node deploy/trialrun/check-source-provenance.mjs.
• Standard release additionally supplies the explicit absolute data-path and data-mode contract required by deploy.sh.
• Build/runtime inputs: BUILD_SHA=<full-commit> and VITE_BUILD_SHA=<12-char-commit>.
• Runtime outputs: /health.data.buildSha, /ready.data.buildSha and the frontend html[data-build-sha] marker.

3. Contracts

• Deployment rejects tracked and untracked dirty files before synchronization.
• By default HEAD must be contained in its upstream. The explicit unpublished override is for controlled offline recovery and never relaxes clean-tree.
• Standard deployment refreshes the configured upstream remote before the ancestor check; a stale local remote-tracking ref is not publication proof.
• Unpublished override requires an allowlisted DEPLOY_UNPUBLISHED_REASON and must emit an UNPUBLISHED-OVERRIDE audit line.
• Synchronization comes from git archive <verified-commit>, not the working tree.
• Full BUILD_SHA is injected at build and runtime; VITE_BUILD_SHA is normalized before bundling so only 12 characters become public.
• Neither build key is persisted in deploy/trialrun/.env; direct Compose use must explicitly provide both and must not publish unknown.
• Deployment compares public /ready and the public bundle marker with the verified commit before reporting success.
• /health.data.buildSha and /ready.data.buildSha are normalized by server/src/config.js to the first 12 characters. A probe with a pinned full commit must compare the public value to exactly those 12 characters, not the full commit or an arbitrary-length prefix. Runtime environment and backup manifest identities remain full-SHA checks. A public identity mismatch is not by itself proof that production changed.

4. Validation & Error Matrix

• Dirty tree -> dirty_worktree, deployment stops.
• No upstream or commit absent from upstream -> stop unless the explicit unpublished override is approved.
• Upstream refresh fails -> stop in standard mode; override mode requires an allowlisted reason and records remoteVerified=false.
• Missing runtime/build SHA input -> Compose interpolation fails.
• /ready SHA mismatch, bundle marker missing or full SHA exposed -> deployment fails.
• Standalone readiness missing/mismatching observed SHA -> non-clean finding.

5. Good/Base/Bad Cases

• Good: clean published commit -> Git archive -> dual-SHA build -> matching backend and public bundle checks.
• Base: local development build uses unknown but is never represented as a release.
• Bad: rsync the current working tree or let .env override the image's SHA with unknown.

6. Tests Required

• Temp-repo provenance tests for dirty, missing upstream, published and unpublished-override cases, including a stale remote-tracking ref after a forced remote update.
• node deploy/trialrun/validate-env-contract.js and Compose config with both SHA inputs.
• Synthetic 40-character build: bundle contains the matching 12-character SHA and not the full value.
• Local-stub readiness test for expected/observed SHA propagation.
• Readiness fixtures must derive the public SHA from the actual getConfig contract; cover wrong, short, empty and unexpectedly full public values. Before a privileged restore handoff, exercise the same public probe from the target host without privileges; a stub that echoes the input full SHA cannot substitute for this read-only check.
• Public deployment still requires separate approval and post-deploy checks.

7. Wrong vs Correct

Wrong

rsync -az --delete ./ remote:/app
docker compose up -d --build server

Correct

node deploy/trialrun/check-source-provenance.mjs
BUILD_SHA=<verified-full-sha> VITE_BUILD_SHA=<matching-short-sha> \
  docker compose -f deploy/trialrun/docker-compose.yml build server

Scenario: Managed Local Gates With Synthetic Password Preparation

1. Scope / Trigger

• Prepare ordinary full-local or revalidate explicitly selected browser/server contracts in a fresh, run-owned local database. Focused diagnosis has no normal release eligibility; none of these entries activates R1 or authorizes production access.

2. Signatures

• Entries: ordinary bash scripts/qa/run-web-gates.sh full-local; focused retained-local with explicit QA_RETAINED_SPECS and optional QA_RETAINED_GREP; retained-server-local with required QA_RETAINED_SERVER_SUITE=workflow-actions|server-regressions. The shared shell helpers are qa_seed_synthetic_credentials and qa_prepare_synthetic_accounts.
• Account owner: scripts/qa/prepare-local-e2e-accounts.mjs; validators validateAccountPreparation and prepareAccounts.
• Receipt: synthetic-account-preparation.json, selection, complete Playwright JSON, managed environment and before/after source/repository snapshots.

3. Contracts

• The existing managed namespace, native-online/no-public-egress topology, exclusive PostgreSQL and owner-verified cleanup apply. Accept only existing local spec basenames; no traversal or implicit full-suite selection.
• Seed all eight synthetic roles once with a per-run random initial password and the real must-change flag. Prove the initial browser change-password gate, then serially use official login/change-password endpoints for each role; the old credential must fail and the new public test fixture credential must log in without another change requirement.
• The preparation CLI verifies the managed context, exact loopback HTTP origin and synthetic database prefix before writes. Do not read operator credentials, print auth payloads, clear flags with SQL, alter production seed behavior or reset shared accounts from concurrent login helpers.
• All three entries seed per-run initial credentials and prepare once before business cases. Ordinary full-local qualifies the guest entry after server startup, then prepares before workflow-actions/server regressions; the later job-enabled restart does not reset accounts. Remove initial-password and seed-password environment variables after preparation. Keep full discovery/PWA inputs, fail-fast, zero retries and configured file concurrency. The diagnostic inventory mode remains a separate contract and is not silently migrated.
• Build Vite clients with NODE_ENV=production while the isolated application server remains in test mode. No real notifications, bank calls or external OCR. Ordinary full-local and the focused server-regressions profile retain IM_NOTIFIER_ENABLED=true for the existing mock/loopback-failure notifier contract; focused browser/workflow-only profiles keep it false. The fresh seeded database contains no real channels; managed no-egress and disabled real WeCom/email remain required. The managed inner runner fixes SAMPLE_FILE to a nonexistent run-owned scratch path so the smoke script generates its synthetic fallback instead of reading a shared temporary sample.

4. Validation & Error Matrix

• Missing/invalid spec selection or managed environment, remote origin or invalid database identity -> fail before account preparation.
• Initial change gate missing, change rejected, old credential accepted or fresh login still requiring change -> fail before business cases, including ordinary server regressions. Cleanup still owns this run’s database. Removing preparation must make the command-capture failure test fail.
• Missing final browser JSON, interrupted run, changed source/repository or failed cleanup -> no complete pass. Owner-verified container removal is recorded independently from an interrupted inner database-drop failure.
• Focused pass -> selected contracts only; normalReleaseEligible=false remains mandatory. A focused server selection cannot replace full-local; only the two fixed script mappings are accepted, with no shell command interpolation. Ordinary discovery/browser calls use --forbid-only and --retries=0.

5. Good/Base/Bad Cases

• Good: one owned empty database, serial official password lifecycle, then unchanged shared login helpers and complete selected results.
• Base: browser runtime absent -> environment limitation; preserve the original observation and record NOT_RUN.
• Bad: disable must-change, mutate the same user's password from parallel tests, or promote a green filtered run into a full release receipt.

6. Tests Required

• scripts/qa/tests/prepare-local-e2e-accounts.test.mjs: accepted lifecycle, old-password rejection and invalid origin/database/initial-password behavior; no credential values in receipts.
• Managed browser exercise: first-login change page, eight successful account lifecycle records, at least one full real-auth business case, zero retries and verified cleanup/source/repository invariants.
• Keep PWA production-build command regression and the real A/B upgrade witness separate from mocked page checks.
• Mock API specs must explicitly block service workers in their actual contexts; production workers can bypass page.route. Keep real PWA A/B witnesses enabled and retain their original budgets. A focused pass after a concurrent failure is not proof of the performance root cause.
• Regression fixtures use current contracts: explicit non-travel expense type for a notification-only reimbursement, explicit invoice item association, management scope for employee lifecycle fields, and save/precheck/submit responses observed separately. Keep blocked prechecks rejected and avoid logging raw payloads.
• Synthetic notifier-secret checks use wecom_webhook with a fixed loopback failure address: encrypted storage, redacted return and provider failure are distinct from mock delivery. Never use real channel configuration.
• Retire a duplicate entry only after proving assertion containment and retaining its canonical manifest/runner entry. Preserve original discovery/results. Visual wildcard route tokens must not be reported as coverage of every matching page; record actual routes and unresolved ownership.
• Copy or hash temporary fixture inputs before the owner cleanup can remove them. A successful list/coverage stage followed by a receipt-collection failure must retain both outcomes; discovery alone never confers release eligibility.

7. Wrong vs Correct

• Wrong: turn off the seed change-password flag so the shared login helper reaches the portal.
• Correct: retain the flag, change the password once through the official API in the owned pre-test stage, verify both rejection and re-login, then leave authentication unchanged during concurrent tests.

Scenario: Isolated Authentication Security Gate

1. Scope / Trigger

• Trigger: changing authentication challenges, credential/session transactions, trusted proxy handling or the deployment order for their schema migration.
• The gate proves local PostgreSQL behavior only; public deployment and edge changes remain separately approved operations.

2. Signatures

• Complete local gate: bash scripts/qa/run-web-gates.sh auth-security-local.
• Database allowlist: loopback PostgreSQL and a fresh hyb_oa_auth_* database whose exact name is confirmed to the checker.
• Runtime guard: RUN_AUTH_SECURITY_REGRESSION=1; concurrency mode accepts only 0 or 1 and reports FULL only for 1.
• Public release/rollback gate: four challenge endpoints marked EXTERNAL_GATED in the runbook and checklist.

3. Contracts

• Validate loopback API, loopback PostgreSQL, isolated database name, explicit confirmation and test mode before the first HTTP write.
• The authentication gate generates per-run JWT, pepper and field-encryption test keys; it must not inherit production values. Email, IM, WeCom and OIDC side effects are disabled or injected with a no-network test service.
• PID/log paths are unique per run. Cleanup may stop only the server spawned by the current run, and a database drop failure makes the gate fail.
• The runner removes only a database that this invocation successfully created; failed creation must never authorize deletion of an existing database.
• Deployment or rollback to old challenge code requires an operator to close /api/auth/forgot-password, /api/auth/reset-password, /api/auth/email-login/code and /api/auth/email-login/verify. Scripts do not claim this external action occurred.

4. Validation & Error Matrix

• Remote API/DB, wrong database prefix/confirmation or production mode -> fail before login, seed-user mutation or challenge requests.
• Code-display logging, enabled external adapter or inherited proxy hop -> fail the test-server guard.
• Concurrency value other than 0|1 -> fail; 0 -> BASE_ONLY/PARTIAL; 1 plus all PostgreSQL scenarios -> FULL.
• Owned server cleanup failure or temporary database drop failure -> non-zero gate result.
• Four-endpoint edge confirmation absent -> public release/rollback remains EXTERNAL_GATED, regardless of local test success.

5. Good/Base/Bad Cases

• Good: disposable loopback PostgreSQL, synthetic secrets, no-network auth email seam, five concurrency scenarios, zero residual databases and an automatically removed container.
• Base: the HTTP regression runs without PostgreSQL concurrency and reports partial evidence rather than a complete security pass.
• Bad: run several mutating HTTP steps and only then discover that the target database or API is remote.
• Bad: reuse a shared PID file or ignore dropdb failure while returning exit code zero.

6. Tests Required

• Static guard tests for remote authority, query-host override, exact database confirmation, secret isolation, external-adapter disablement, owned cleanup and redacted failure output.
• bash -n for the runner and regression scripts; Node syntax checks for the checker and test-server entry.
• Prisma validate, clean migration deployment, the full isolated gate and an explicit post-run query proving no hyb_oa_auth_* database remains.
• node deploy/trialrun/validate-env-contract.js must enforce loopback API binding, XFF overwrite templates and four-endpoint EXTERNAL_GATED wording.

7. Wrong vs Correct

Wrong

run_mutating_auth_regression
validate_target_afterwards
dropdb --if-exists "$DB" || true

Correct

validate_isolated_target_before_writes
run_mutating_auth_regression
drop_owned_database_or_fail

Scenario: Repository Local Candidate Gate

The independent all-spec owner is bash scripts/qa/run-web-gates.sh full-local, not this route-manifest subset. Full-local prepares temporary PWA A/B production builds before Playwright collection using the shared PWA fixture helper; all non-manual spec/test files under the current test directory remain assigned to full-local discovery, including dedicated-input specs and filenames containing live. The approved G19 contract below partitions execution; public obligations stay mandatory after deployment. The existing local/live runtime mode restrictions remain intact; ownership is not authorization to use live accounts or contact production.

verify-full-local-discovery.mjs compares the actual collection with source files rather than trusting the report or acceptanceSpecs as its expected set. Missing/unknown files, collection errors or files without cases fail before DB preparation. FULL_LOCAL_DISCOVERY_ONLY proves enumeration, not execution or release eligibility. A later fail-fast stage must leave browser cases marked not run, never passed.

bash scripts/qa/run-web-gates.sh full-local-diagnostic is a separately approved local inventory, never a release gate. Require non-empty QA_DIAGNOSTIC_REASON, QA_DIAGNOSTIC_OPERATOR and QA_DIAGNOSTIC_APPROVAL_REF; preserve them in its dedicated, non-overwritten report directory. It always finishes PARTIAL with exit 2, even if every executed test passes. Input/runner errors and interruption remain nonzero. The normal full-local entry retains fail-fast behavior; it never falls back to diagnostics. No boolean environment marker grants release eligibility.

Both full-local entries own the Linux isolated-online-v1 environment through full-local-environment.mjs: a fresh network namespace, an up internal veth pair with no external path, and an owned offline PostgreSQL container connected only through its Unix socket and loopback proxy. No host fallback, image pull, arbitrary outer wrapper or boolean online assertion is accepted. Unsupported capabilities are environment failures, not business test failures. Reports separate inherited, runner-created and browser namespace fingerprints; live ownership/topology and current policy digests are revalidated between stages and at completion. The only default route is default dev qaonline scope link: no gateway/via/multipath, no IPv6 default, and no other interfaces. Verify both veth endpoints are in the owned namespace by reciprocal peer names/indexes, info_kind=veth, and no external namespace/master link. This closed link lets Linux WebKit detect availability without an external path. The native blank-page probe must observe actual online=true in both Chromium and WebKit; each browser process must match the owned namespace. Never patch navigator or count browser network emulation as a kernel observation. Before business browser tests, a fresh unmocked context opens the real candidate login page: listeners precede navigation, cookies/storage are empty, the form appears, no refresh or login request occurs, and native online samples are true. Failed/401 refresh attempts still count. Form visibility alone cannot prove the guest branch: terminal refresh errors also render it. Samples remain independent observations, never an internal AuthContext measurement; missing and false differ. Legitimate per-context offline scenarios remain untouched. Environment failure stops the diagnostic's remaining branches; successful diagnostics remain PARTIAL/2. The requested report base contains environment/ and results/; preserve both, including traces and source snapshots. Local receipts are auditable evidence, not cryptographic authenticity guarantees; third-party reproduction stays required. R1 consumption/activation remains separately authorized, not enabled by this helper.

Candidate snapshots retain two independent scopes: the existing selected-source digest and a repository snapshot containing complete porcelain-v1 NUL-delimited status, readable status, every tracked worktree file's kind/mode/content digest, and index stage entries/flags/digest. Symlink content is the link target string, never the target file; unsupported directory/ancestor-link coverage is unqualified. Capture the final snapshot on failure/cleanup paths as well as success. sourceUnchanged does not imply repositoryUnchanged; unchanged does not imply cleanAtStart or cleanAtEnd. A dirty-to-dirty overwrite must remain detectable. Untracked paths are retained in status; their contents outside the selected source set are not covered by the tracked-file digest. Do not call a dirty diagnostic a byte-for-byte freeze of all untracked materials. Future clean release candidates cannot contain these untracked paths in the first place. Environment qualification stays separate from candidate integrity: a modified tracked screenshot does not retroactively falsify native-online observations, but it cannot qualify a release. Ordinary runs fail on candidate mutation; diagnostics retain PARTIAL/2 and explicit CHANGED/UNKNOWN evidence. R1 activation and its required clean candidate remain separately authorized.

Discovery and full-browser commands supply QA_VISUAL_OUTPUT_DIR beneath the run's report directory. Screenshot-only specs use unique per-spec subdirectories, including standalone defaults; they must not write into registered historical evidence paths. .gitignore never exempts already-tracked output from these checks.

Diagnostic stages run in separate strict shell processes. Failed assertions do not block independent suites; failed preparation blocks dependent work with explicit NOT_RUN reasons. Every write-suite and the browser receive a newly created, uniquely named local database and separate storage/server; only owned resources may be removed, with cleanup failures recorded. Keep PWA inputs shared for this run, preserve workers and zero retries, and retain original per-case outcomes (including serial skips and missing results). The diagnostic browser stage must explicitly pass --trace=on; the shared browser helper forwards only its caller-supplied internal arguments. Zero retries combined with on-first-retry yields no trace, so a retry-dependent default is not diagnostic evidence coverage. Preserve trace archives in the existing per-run output directory together with screenshots and result JSON. This does not add a public scope/worker override or change ordinary full-local's trace policy. A future ordinary-gate retain-on-failure change requires its own review. Captured synthetic traces are local debugging artifacts, not files to publish or commit without a separate sensitive-content review. The nested regression-all invocation retains its default fail-fast: do not pass its --json-report option, which continues mutating suites on the same database after failure. Its repeated workflow-actions is a second source of the same finding, not a second defect; subsequent suites remain NOT_RUN. A stage PASS is not proof every subcase ran, including disabled modules. Use neutral temporary-root names that do not trigger production-identity guards in synthetic fixtures; verify with the actual guard, never relax it. Never disable an inner suite's fail-fast to claim complete coverage. The diagnostic duration is not a measurement of the normal all-green release path.

Full-local explicitly passes --retries=0 regardless of CI's configured retry default. This is a release coverage requirement independent of PWA. Keep the configured file-level worker count; PWA owns an isolated origin/context and declares serial execution internally. No silent exclusion or global serial fallback is permitted. If a future split needs exceptions, each requires a reason, mandatory runner, owner and evidence obligation before approval.

QA_FULL_REPORT_DIR holds persistent build logs, collection JSON and source coverage, then full browser JSON/HTML/results when that stage is reached. Cleanup only reclaims the current run's temporary PWA builds and owned DB/server; it must preserve these reports and not overwrite client/dist for A/B fixtures. Validate the shared standalone/full-local paths with node --test scripts/qa/tests/pwa-full-local-inputs.test.mjs scripts/qa/tests/verify-full-local-discovery.test.mjs, including CI zero retries, no worker override, failed/missing A/B inputs, omitted PWA collection, browser failure and cleanup boundaries.

1. Scope / Trigger

• Trigger: declaring a checkout ready for internal deployment planning without deploying it or using public, production or real-account evidence.

2. Signatures

• Full gate: bash scripts/qa/run-local-candidate.sh full.
• Browser-only rerun: bash scripts/qa/run-local-candidate.sh focused.
• Parse-only check: bash scripts/qa/run-local-candidate.sh list.
• Route contract: scripts/qa/route-manifest.json.
• Result verifier: scripts/qa/verify-local-candidate-results.mjs.

3. Contracts

• full alone may print LOCAL_CANDIDATE_PASS; focused prints FOCUSED_BROWSER_PASS, and list prints LIST_ONLY.
• The gate accepts only loopback HTTP origins and canonical TCP ports. It uses local synthetic fixtures, excludes manual/live specs and does not consume historical Issue CSV as a candidate prerequisite.
• Every registered route is classified. Every core bundle has one stable, source-bound witness; candidate spec basenames are unique.
• Every selected Playwright result must pass. Missing, duplicate, unknown, skipped, multi-tag or wrong-source witnesses and any failed/skipped acceptance test make the gate fail.
• The script's full pass proves only its local scope. A task changing payment concurrency also requires the isolated PostgreSQL payment gate; deployment, public checks and real-device behavior remain separate states.

4. Validation & Error Matrix

• Non-loopback URL, credentialed URL, path/query target or invalid port -> fail before Vite/Playwright use.
• Unclassified/orphan route or missing core behavior/visual witness -> non-zero scanner result.
• Playwright suite error, unexpected, failed or skipped test -> non-zero result gate with basename-only redacted diagnostics.
• focused succeeds -> useful rerun evidence, but not a full candidate claim.

5. Good/Base/Bad Cases

• Good: full base checks pass, all source-bound witnesses and acceptance specs execute, and the runner prints one full-pass label.
• Base: use list before a long run to prove assets parse, while recording that no behavior has run.
• Bad: reuse a prior browser pass after parser changes, or report focused/list output as LOCAL_CANDIDATE_PASS.

6. Tests Required

• Negative tests for route gaps, manual/live evidence, duplicate basenames, invalid loopback inputs, witness result/source drift and mode labels.
• Current-checkout full run with zero skipped/unexpected results.
• For payment-state changes, run the disposable PostgreSQL gate and prove cleanup left no matching temporary databases.

7. Wrong vs Correct

Wrong

bash scripts/qa/run-local-candidate.sh list
echo LOCAL_CANDIDATE_PASS

Correct

bash scripts/qa/run-local-candidate.sh full
# Only this successful branch emits LOCAL_CANDIDATE_PASS.

Scenario: Compose-Resolved Loopback Host Port

1. Scope / Trigger

• Trigger: changing Compose port publication, deployment health checks, Caddy routing or operator commands for the server container.
• Purpose: support a non-default host port without changing the container's fixed application port or accidentally checking a stale service.

2. Signatures

• Env: APP_HOST_PORT=<1-65535>; default 3000.
• Compose mapping: 127.0.0.1:${APP_HOST_PORT:-3000}:3000.
• Runtime resolution: docker compose port server 3000 or the compatible docker-compose command selected by the deployment script.
• Backend container still listens on PORT=3000.

3. Contracts

• Publication remains loopback-only. APP_HOST_PORT changes only the host side; Caddy points to the same resolved host port.
• Deployment health checks must parse the running Compose mapping after startup; do not hardcode 3000 and do not assume .env variables entered the current shell merely because Compose read --env-file.
• The parsed port must be numeric before it is used in curl. A missing, multi-line or non-numeric mapping stops deployment before readiness success.
• server, worker and init retain the shared env-file contract; only server publishes an HTTP host port.
• Changing the real host port or Caddy target is production configuration and requires separate approval plus rollback evidence.

4. Validation & Error Matrix

• Missing APP_HOST_PORT -> Compose uses loopback port 3000.
• Valid alternate port -> Compose publishes it and deployment checks that port.
• Non-numeric/unresolved compose port output -> deployment exits non-zero.
• Caddy target differs from Compose mapping -> deployment is not accepted even if one local /ready endpoint answers.
• Host port already occupied -> Compose startup fails; do not start a second deployment on a guessed port.

5. Good/Base/Bad Cases

• Good: .env selects an alternate loopback port, Compose reports it, deploy checks the matching build SHA there and Caddy targets the same port.
• Base: default port 3000 remains compatible without extra configuration.
• Bad: Compose publishes 3001 while deploy/runbook curls 3000 and reports an old container as the new release.

6. Tests Required

• bash -n deploy/trialrun/deploy.sh deploy/trialrun/setup-server.sh.
• node deploy/trialrun/validate-env-contract.js must assert the exact loopback-only variable mapping and env example key.
• Review runbook/checklist commands for Compose resolution rather than shell expansion of an unloaded .env.
• Public deployment additionally proves Caddy, /ready and frontend marker use the approved build on the same resolved port.

7. Wrong vs Correct

Wrong

curl -fsS http://127.0.0.1:3000/ready

Correct

SERVER_HOST_PORT="$(docker compose port server 3000)"
SERVER_HOST_PORT="${SERVER_HOST_PORT##*:}"
curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"

Scenario: Compose Project And Data-Path Isolation

1. Scope / Trigger

• Trigger: changing Compose identity, host data mounts, database publication, deployment entrypoints, deployment-time seeds or operational timers.
• Purpose: ensure one approved Compose project owns one verified DB/storage pair without fixed global container names, source synchronization touching runtime data, or deployment seeds overwriting approved production rules.

2. Signatures

• Every deploy/timer Compose command uses -p "$COMPOSE_PROJECT_NAME".
• PostgreSQL mount: ${POSTGRES_DATA_DIR:-./data/postgres}.
• Application storage mount: ${STORAGE_HOST_DIR:-./data/storage}.
• Formal deploy inputs: absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, absolute REMOTE_DIR, absolute REMOTE_ENV_FILE, DEPLOY_DATA_MODE=existing|initialize.
• Deployment init includes npm run seed:config; approval-rule defaults are inserted only when approval_rules is empty.
• Timer install inputs: COMPOSE_PROJECT_NAME, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, full 40–64 hex BUILD_SHA and its exact 12-hex VITE_BUILD_SHA prefix.
• DB has no host ports; server remains loopback-only.

3. Contracts

• Compose files must not set container_name. Operational commands resolve services through the approved project, never a historical global name.
• A project-name change and its DB/storage path selection are one atomic, separately approved downtime operation. Never start two PostgreSQL instances against one PGDATA.
• Freeze the old project name from docker compose ls, not from a directory guess. Record immutable image IDs/digests, successful migration count, fixed business-table counts, resolved mounts and Caddy target before any write.
• Relative defaults preserve local trialrun compatibility, but production must use explicit absolute command inputs and verify them against active container mounts before any start, backup or migration.
• REMOTE_DIR must not equal, contain or be contained by either production data path. Validate this locally before provenance, SSH or rsync --delete. If REMOTE_ENV_FILE is inside the synchronization tree, derive its actual root-relative path and add explicit rsync protect/exclude filters; a fixed default .env exclusion is insufficient.
• Deployment-time seed:config may create the three default approval rules for an empty database. If any approval rule exists, preserve all existing rules; production rule changes require their own approved transaction, drift checks and audit rows.
• existing is the normal migration mode and refuses a missing PGDATA marker or storage directory. initialize is only for an approved empty first start; only that mode may create data directories.
• Remote readiness may use host curl, but build-SHA JSON validation runs with Node inside the server container; the Docker host does not need Node.
• The timer installer must validate the same non-root, non-overlapping absolute data-path shape as formal deploy, require an existing PG_VERSION and storage directory, and write both paths plus both build identifiers into the systemd service Environment= entries. Project name alone is insufficient: a later timer invocation would otherwise interpolate Compose defaults and operate against a non-active data pair.
• Timer installation may enable/start the timer unit, but verification must not manually start the daily service because that path can send real alerts. Verify the unit environment against active container mounts, then run Compose config --quiet and exec -T server true with the same values.
• For an existing cutover, stop the old PostgreSQL completely, copy PGDATA and storage into pre-verified empty absolute targets while preserving ownership, permissions, links and extended attributes, and retain the old directories untouched and unmounted for the observation period. Do not rename away the only pre-migration rollback data and do not copy a running PGDATA.

4. Validation & Error Matrix

• DB ports appears -> environment contract fails.
• Any fixed container_name appears -> environment contract fails.
• Required data mount loses its parameter -> environment contract fails.
• Deploy/timer omits -p -> environment contract fails.
• Timer install omits either data path/build identifier, receives nested paths, a PGDATA without PG_VERSION, a missing storage directory, an invalid full SHA or a 12-character value that is not its prefix -> fail before writing the systemd unit.
• A path-segment regex can match an empty string at the leading / -> contract failure; use explicit root, trailing slash, //, /./ and /../ checks so valid absolute paths remain installable.
• Formal deploy omits paths/mode, uses a relative/broad/overlapping path, or an existing PGDATA lacks PG_VERSION -> fail before DB start.
• REMOTE_DIR overlaps PostgreSQL or attachment storage in either direction -> fail before provenance, SSH and rsync.
• A custom REMOTE_ENV_FILE lies inside the sync tree without dynamic protect/exclude filters -> deployment contract fails.
• approval_rules contains one or more rows -> seed:config reports them as preserved and performs no approval-rule create, update or delete.
• New project resolves the same PGDATA while the old DB runs -> stop before Compose start; do not use service health as permission to continue.
• Old project name was inferred, immutable image ID or baseline counts are missing, or target copy is non-empty -> stop the cutover.

5. Tests Required

• node deploy/trialrun/validate-env-contract.js.
• The isolated backup/restore drill mutates Compose with a DB port and fixed name, invokes formal deploy with each data path nested below REMOTE_DIR, and proves every unsafe shape is rejected before network access.
• bash -n covers deploy, timer, backup and restore scripts.
• Seed unit tests assert: an empty table receives exactly three defaults; any non-empty table receives zero approval-rule writes and no delete call.
• Timer contract assertions cover all four explicit inputs, path isolation, PGDATA marker, SHA/prefix validation and the four systemd Environment= entries. On the deployment host, assert timer enabled/active, daily service inactive, exact active-environment equality and non-mutating Compose resolve.

6. Good/Base/Bad Cases

• Good: one approved project resolves one stopped-and-migrated PGDATA and the matching active storage path; deploy, timer and backup all use that identity.
• Good: source lives under one absolute root, data lives under separate absolute roots, a custom env file is protected, and existing approval rules survive deployment init byte-for-byte until an approved rule transaction runs.
• Good: the timer unit contains the current project, active mount sources and deployed full/short SHA pair; Compose config and a non-mutating server exec resolve under those exact values while the daily service stays inactive.
• Base: direct local Compose uses the documented relative defaults under one project; those defaults are never accepted by formal deploy.sh.
• Bad: start a renamed project before the old DB stops, or let it create a new empty directory and accept a green health check as proof of data continuity.
• Bad: move the only PGDATA into the new path, migrate it in place, then assume the old container name alone is a rollback point.
• Bad: reinstall the timer with only COMPOSE_PROJECT_NAME, allowing systemd Compose interpolation to fall back to repository-relative data directories.
• Bad: place PGDATA or attachments below REMOTE_DIR and rely on one hard-coded rsync exclusion, or delete/recreate configurable approval rules in a routine deployment seed.

7. Wrong vs Correct

Wrong

docker-compose up -d db
DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh

Correct

docker-compose -p "$COMPOSE_PROJECT_NAME" -f deploy/trialrun/docker-compose.yml \
  --env-file deploy/trialrun/.env up -d db
mapfile -t DB_CONTAINERS < <(docker-compose -p "$COMPOSE_PROJECT_NAME" \
  -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env ps -q db)
test "${#DB_CONTAINERS[@]}" -eq 1
DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \
  STORAGE_DIR_HOST=<verified-active-storage> ./scripts/backup-all.sh

For timer installation, keep the active deployment identity explicit:

# Wrong: project identity alone still permits relative data-path fallback.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

# Correct: use the verified active mounts and deployed SHA pair.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
POSTGRES_DATA_DIR=<verified-active-pgdata> \
STORAGE_HOST_DIR=<verified-active-storage> \
BUILD_SHA=<verified-full-build-sha> \
VITE_BUILD_SHA=<verified-12-char-prefix> \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

For source synchronization and approval-rule defaults:

# Wrong: runtime data sits below the --delete synchronization root, and seed
# replaces operator-managed rules during every deployment.
REMOTE_DIR=/srv/oa POSTGRES_DATA_DIR=/srv/oa/data/postgres \
  bash deploy/trialrun/deploy.sh
# seedApprovalRules: deleteMany -> createMany

# Correct: data roots are disjoint; an in-tree env file is dynamically
# protected; seed creates defaults only for an empty approval_rules table.
REMOTE_DIR=/srv/oa-app POSTGRES_DATA_DIR=/srv/oa-data/postgres \
STORAGE_HOST_DIR=/srv/oa-data/storage DEPLOY_DATA_MODE=existing \
  bash deploy/trialrun/deploy.sh
# seedApprovalRules: count > 0 -> preserve; count = 0 -> create defaults

Scenario: Paired Backup And Shadow-Only Restore

1. Scope / Trigger

• Trigger: changing or operating database/file backups, restore scripts, retention instructions or recovery-readiness evidence.
• Purpose: prevent a database-only recovery point, a mixed attachment restore or an accidental destructive restore into the active internal instance.

2. Signatures

• Paired backup: DB_TOOL_MODE=docker DB_CONTAINER=<compose-resolved-id> STORAGE_DIR_HOST=<verified-active-directory> BUILD_SHA=<actual-build-sha> SOURCE_IMAGE_DIGEST=<actual-sha256-image-id> ./scripts/backup-all.sh; at least one source identity is required.
• Optional safe batch identifier: BACKUP_RUN_ID=<safe-id>.
• Manifest writer: host node is preferred; host python3 is an equivalent fallback for Docker-only runtime machines.
• DB shadow restore requires RESTORE_SCOPE=shadow, RESTORE_CONFIRM_DATABASE=<actual-current-database> and DATABASE_URL.
• Storage shadow restore: RESTORE_SCOPE=shadow ./scripts/restore-storage.sh <tar> <empty-target>.
• Local synthetic drill: ./scripts/qa/run-backup-restore-drill.sh.
• DB_TOOL_MODE is auto, host or docker; use docker with Compose to avoid host/server PostgreSQL tool-version drift.

3. Contracts

• A complete recovery point is one batch directory containing database.dump, storage.tar.gz and a final manifest.json. The manifest contains only safe relative names, byte sizes, SHA-256 hashes, batch time, actual build SHA when known and actual source image digest when supplied.
• Bind the canonical batch directory to the verified manifest runId, never derive it from createdAt: run start and manifest creation can differ by seconds. A cross-host manifest hash match does not replace checking the selected directory identity on the actual restore host.
• An observation used to prepare a restore plan must return the observed runId and relative batchName, not just a timestamp/hash. Pin the intended manifest and reject a changed newest canonical candidate; do not skip a newer symlink or silently fall back to an older matching batch. A successful directory observation is not execution-time preflight or restore approval.
• A separately approved cross-platform image rebuild pins the source commit and tree, native target platform and verified local base digest. Export only the approved tracked paths, reject secrets/unsafe archive entries, and record source/lock/context hashes plus output BUILD_SHA and OCI revision. Build cache reuse is not a clean dependency install; same source is not the original image's byte identity. Building alone authorizes no application startup.
• Use an exclusively created attempt receipt before a bounded remote build; an existing receipt or uncertain timeout stops without automatic retry. Keep prior images/caches and immutable review bundles. Attach new image/source observations outside the frozen bundle; they do not clear pending content review, source-directory binding or execution-time freshness gates.
• Rebinding reviewed recovery code uses distinct reviewManifestSha256 and executionManifestSha256 identities. Verify embedded code/helpers against the original review manifest and restrict bundle changes to the approved source/image binding fields before importing the runner. Preserve the full migration map and historical exception, not merely their counts. Review of the old code does not imply independent review of a new binding envelope.
• BUILD_SHA supplied by the caller takes precedence over local Git fallback. On a non-Git runtime host, at least SOURCE_IMAGE_DIGEST=sha256:<64-hex> is mandatory. Never label a legacy backup with the commit planned for deployment.
• Manifest generation uses exclusive creation and preserves the same schema under Node and Python 3; if neither runtime exists, fail before publishing the final batch directory.
• For an approved live backup, do not assume the deployment .env still matches the running database. Confirm that the selected PostgreSQL container's database and user match at least one active application DATABASE_URL, then pass the verified runtime values without logging their contents.
• Docker backup/restore has no default container name. Resolve the unique DB container with compose -p <approved-project> ... ps -q db and fail when the result is empty or ambiguous. Storage backup likewise requires an explicit STORAGE_DIR_HOST; it must never infer STORAGE_DIR or a repository default.
• The DB dump must pass pg_restore --list and the storage archive must pass tar -tzf before the manifest may say complete.
• DB restore queries current_database() and compares it exactly with RESTORE_CONFIRM_DATABASE before pg_restore --clean.
• The actual DB name must also contain a separate shadow, drill or restore_test marker; setting RESTORE_SCOPE=shadow alone is not trust.
• Storage restore requires an explicit empty target and rejects absolute paths, .. path segments, symbolic links and hard links before extraction.
• Current scripts intentionally reject production restore. A real incident requires an approved Trellis task, stop window, paired batch, two responsible people, rollback point and post-restore /ready/permission checks.
• Copy the whole completed batch to approved off-host/off-disk storage; a batch left only on the application host is not accepted recovery evidence.
• A real-format shadow restore compares successful migration count and a fixed set of business-table counts with the pre-backup baseline, compares extracted regular-file count with the active storage source, and runs /ready using a schema-compatible image. /ready alone is not recovery evidence.
• When shadow SQL is supplied through stdin or a heredoc, docker exec must use -i and psql must use ON_ERROR_STOP=1; otherwise an empty result can be mistaken for a data mismatch even though no verification query ran.
• Compare fixed table counts as a keyed set, not by database-returned row order. Require the expected unique key count, then normalize both table=count collections with LC_ALL=C sort (or an equivalent map comparison) before equality checking; database collation order is not an evidence contract.
• A documented historical missing attachment is not an implicit restore exception. A separately approved exception must pin record/path identities and disposition evidence, recheck uniqueness, archive time and absence of business associations inside the restored database, and prove actual file absence. Keep every reference in the original strict validator input.
• Report totalReferences, rawMismatches, acceptedKnownMissing and unexplainedMismatches separately. An accepted missing file stays incomplete; it may allow only the explicitly approved later read-only stage and a PARTIAL outcome, never storage-integrity PASS. Preserve earlier NO_GO.

4. Validation & Error Matrix

• One backup component fails or integrity listing fails -> remove the temporary batch; do not write a complete manifest.
• Existing batch identifier -> fail without overwriting.
• Missing/non-shadow restore scope -> fail before any restore write.
• Actual DB name differs from confirmation -> fail before pg_restore --clean.
• Confirmed DB name lacks a shadow/drill/restore-test marker -> fail before pg_restore --clean.
• Storage target is implicit, root/repository root or non-empty -> fail before extraction.
• Corrupt dump/tar or dangerous/link archive member -> fail before restore.
• Host PostgreSQL tools mismatch in auto mode -> docker fallback is allowed only when the explicit target container and required DB variables exist.
• Neither Node nor Python 3 is available -> fail manifest generation and remove the temporary batch; do not publish a partial recovery point.
• Invalid build SHA/image digest, or neither identity available on a non-Git host -> fail before DB or storage backup starts.
• Deployment .env, PostgreSQL container and active application connection do not identify the same database/user -> stop the live backup; never guess credentials or expose values in diagnostics.
• Shadow migration count, any fixed table count, attachment regular-file count or compatible-image /ready differs from baseline -> reject the recovery point and do not enter production cutover.
• Empty, duplicate or missing table-count output -> classify as a verification execution failure first; do not report production-data drift until stdin was delivered and every expected key was returned exactly once.
• Historical target identity/archive/association drift, duplicate target, file substitution or any additional unexplained mismatch -> stop before the application stage. Never use archived_at IS NOT NULL as a blanket exclusion.
• Expired or not-yet-rebound source -> do not render an executable restore entry; code review and synthetic tests may continue with an explicit binding gate and a non-executable review bundle.
• Build source/base/platform/resource mismatch -> stop before docker build; a successful build with wrong output revision/platform remains unusable for restore. A rebuild never extends the source backup's freshness window.

5. Good/Base/Bad Cases

• Good: one command creates a validated pair, the complete batch is copied off host, and a disposable shadow drill restores the DB marker and attachment hash.
• Good: a Docker-only host uses Python 3 to write the same manifest contract after the active application connection is matched to the database container.
• Good: a legacy container without /ready.data.buildSha records its inspected sha256: image ID and leaves buildSha null rather than inventing provenance.
• Good: a heredoc query uses docker exec -i ... psql -v ON_ERROR_STOP=1, then compares the complete keyed count set under deterministic C sorting.
• Base: backup succeeds locally but off-host copy and deployed /ready evidence remain explicitly pending; do not claim formal recovery readiness.
• Bad: source a stale deployment .env and treat a failed dump as a prompt to guess or print credentials.
• Bad: source the active trial .env and run a cleaning restore directly, or restore attachments into a non-empty live directory.

6. Tests Required

• Shell syntax checks for all backup/restore scripts.
• Synthetic Docker/PostgreSQL and temporary-storage round trip.
• For a narrowly approved missing-file exception, assert exact identity, archive time, every association and actual absence against synthetic SQL; assert new missing files, wrong size/hash, duplicate targets and symlinks still block, and that raw mismatch counts and PARTIAL are preserved.
• Negative assertions for wrong DB confirmation, corrupt DB/tar, non-empty target and dangerous tar path.
• Validate manifest relative paths, absence of connection secrets and matching SHA-256 values, explicit build-SHA precedence and image-digest preservation.
• Run the synthetic drill with Node available and with a PATH that excludes Node but includes Python 3; both paths must complete the same round trip.
• For real-format count checks, assert that stdin queries return the full unique key set, and prove that a different database collation/order does not change the pass result while a changed count still fails.
• Real-instance restore and /ready remain separately approved evidence and must never be fabricated from the local drill.
• For a cross-platform build wrapper, assert exact base/revision/platform, minimal Dockerfile substitution, secret/path/link rejection, resource thresholds, failed-preflight zero builds, exclusive receipt/no retry, and successful image inspection without docker run or deletion of old images.

7. Wrong vs Correct

Wrong

source deploy/trialrun/.env
./scripts/backup-all.sh
./scripts/restore-db.sh some.dump
./scripts/restore-storage.sh some.tar.gz deploy/trialrun/data/storage

Correct

./scripts/qa/run-backup-restore-drill.sh
# Before an approved live backup, compare the selected DB container identity
# with an active app connection using redacted boolean diagnostics.
DB_TOOL_MODE=docker DB_CONTAINER=<verified-container> \
  STORAGE_DIR_HOST=<verified-active-storage> \
  SOURCE_IMAGE_DIGEST=<verified-sha256-image-id> ./scripts/backup-all.sh
# Real restore remains an approved incident operation after shadow evidence.

For a containerized count check, the input and comparison contract is:

# Wrong: stdin is not attached, and raw row order becomes part of the gate.
actual="$(docker exec shadow-db psql -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts ORDER BY table_key;
SQL
)"
test "$actual" = "$expected"

# Correct: deliver stdin, fail on SQL errors and compare keyed sets deterministically.
actual="$(docker exec -i shadow-db psql -v ON_ERROR_STOP=1 -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts;
SQL
)"
test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
  "$(printf '%s\n' "$expected" | LC_ALL=C sort)"

Scenario: Off-Host Archive And Private-Copy Identity Validation

1. Scope / Trigger

• Trigger: an off-host recovery drill copies a formally archived backup batch into a private temporary directory before database or attachment restore.
• Purpose: preserve the archive placement evidence at the source boundary while validating a renamed private copy by its source-bound content identity.

2. Signatures

• Source validation: validate_source_batch(<archive-run-directory>) -> source_metadata.
• Private-copy validation: validate_copied_batch(<private-copy-directory>, <source_metadata>).
• source_metadata contains only redacted comparison values: verified run id, manifest/build/time identity, exact three-file set, byte sizes and SHA-256 values. It must not contain credentials, business payloads or full paths.

3. Contracts

• A formal archive run directory must have a basename that exactly equals the validated manifest run id. This detects a valid manifest placed under the wrong archive run and is mandatory before any copy or restore write.
• A private copy may use an operator-owned neutral basename such as batch; that basename is not archive provenance and must not be compared with the manifest run id.
• The copied manifest run id must exactly equal the run id already verified at the source. Internal self-consistency inside the copy is insufficient.
• The copy must repeat manifest schema/status, run-id format, exact manifest.json/database.dump/storage.tar.gz regular-file set, build and age gates, byte-size checks and SHA-256 checks.
• After repeated validation, every redacted copied metadata field must equal source_metadata. Copy validation must never disable a source-side gate or accept a newly invented identity.
• Formal archives remain read-only. A validation failure stops before creating PostgreSQL, extracting attachments or modifying the archive, and cleanup must prove zero private runtime and privilege-entry residue.

4. Validation & Error Matrix

• Source basename differs from manifest run id -> batch_identity; NO_GO before copy.
• Copied manifest run id differs from the verified source run id, even when the copy is internally self-consistent -> copied_batch_invalid/batch_identity; NO_GO before restore runtime creation.
• Copied file set, bytes, SHA-256, build, time or other redacted metadata differs from the source -> copied_batch_invalid; remove only the owned private runtime and retain the archive unchanged.
• Private basename differs from the run id but all source-bound content checks pass -> valid; continue to the separately approved restore step.
• Any cleanup residue remains -> attempt stays non-clean and cannot be reported as recovery GO.

5. Good/Base/Bad Cases

• Good: source directory basename matches its manifest run id; the same batch is copied to a private directory named batch, revalidated, and its redacted metadata exactly equals the source record.
• Base: source and copy basenames happen to match; both still use their proper validation modes and the copy remains bound to source_metadata.
• Bad: reuse source validation on the private directory and reject it only because the operator chose a neutral basename.
• Bad: skip the copy basename check and then trust any internally consistent replacement manifest without comparing it with the verified source run id.

6. Tests Required

• Positive: a formal source whose basename matches its manifest run id passes.
• Positive: an unchanged copy under a different private basename passes and produces metadata equal to the source record.
• Negative: source basename/run-id mismatch fails before copy.
• Negative: copied content or checksum mutation fails before restore.
• Negative: a copied manifest with an internally consistent but source-different run id fails, proving the source binding cannot be bypassed.
• Regression: run the existing synthetic backup/restore contract and assert no PostgreSQL object, extracted attachment, private runtime or privilege entry is left after every pre-runtime failure.

7. Wrong vs Correct

Wrong

validate_batch(source, require_basename_match=true)
copy source -> private/batch
validate_batch(private/batch, require_basename_match=true)

Correct

source_metadata = validate_source_batch(source_run_directory)
copy source_run_directory -> private/batch
validate_copied_batch(private/batch, source_metadata)

Scenario: Shadow-Restore Failure Diagnostics Before Teardown

1. Scope / Trigger

• Trigger: a one-shot off-host shadow restore keeps raw tool output only in a private runtime that must be deleted on every exit path.
• Purpose: preserve enough allowlisted stage evidence to locate a restore failure without retaining or publishing raw stderr, paths, credentials, SQL or restored object names.

2. Signatures

• Required stages: dump_list, target_confirmation, restore_apply, each with the enum PASS|FAIL|NOT_RUN.
• Required execution fields: restore_execution_path=DOCKER_STDIN|HOST_FILE|NOT_RUN, dump_list_exit_code, target_confirmation_exit_code, target_name_match=yes|no|NOT_RUN, target_confirmation_attempt_count, target_confirmation_success_streak, restore_script_exit_code, pg_restore_exit_code, pg_restore_error_count and restore_error_category.
• Exit-code fields are non-negative integers or NOT_RUN; pg_restore_error_count additionally permits UNKNOWN when no safe numeric summary is available.
• Required data-restore timing fields: rto_batch_copy_seconds, rto_postgres_container_start_seconds, rto_dump_list_seconds, rto_target_confirmation_seconds, rto_restore_apply_seconds, rto_database_verify_seconds, rto_storage_attachment_verify_seconds, rto_final_integrity_seconds and data_restore_rto_seconds. A measured value is a non-negative integer; an unstarted stage is NOT_RUN.
• Tracked supply-chain entries: scripts/qa/offhost-shadow-restore-runner.template.sh, scripts/qa/validate-offhost-shadow-restore-attachments.py and bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh.
• Candidate render: bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh render <private-config.json> <new-output>; it prints only PASS and the candidate SHA-256, never config values or paths.
• One-shot installer manifest: the candidate LIB_DIR, runner entry, dependency basenames and sudoers entry form one exact private mapping. The installer must validate that mapping before its first privileged write; a sibling lib / libexec path reconstructed independently is not equivalent.

3. Contracts

• Emit the fixed diagnostic fields on the failure path before runtime teardown. Teardown remains in a trap/finally-equivalent path and must still run after diagnostic emission; callers retain stdout, not the private raw log.
• Before an attachment-reference failure, preserve completed DB checks, extraction/file-count results and per-source reference/mismatch counts. Classify missing files, size/hash mismatches, unsafe paths, malformed rows, unreadable files and external storage without outputting identifiers or paths. Check diagnostic totals against the unchanged validator; disagreement also fails closed. Do not skip archived, test or duplicate references to gain PASS.
• An ARM restore runner fixes restore_execution_path=DOCKER_STDIN. A private Docker proxy may observe the real pg_restore --list and apply calls and persist only their stage name and numeric exit code; it must not record argv, environment values or stderr.
• Run dump_list and target confirmation independently before the existing restore script. A failed stage leaves every later stage NOT_RUN.
• Measure the eight data-restore stages in this exact continuous order: batch_copy -> postgres_container_start -> dump_list -> target_confirmation -> restore_apply -> database_verify -> storage_attachment_verify -> final_integrity. One timestamp closes the current stage and starts the next, so no elapsed time is hidden between fields. The total is computed from the measured stage fields and must equal their exact sum.
• On failure, finalize the active stage and emit the completed/current numeric durations plus later NOT_RUN fields before runtime teardown. A pre-copy failure emits NOT_RUN for the total and every stage. Do not emit the total a second time on the success-only summary path.
• For the official PostgreSQL image, target readiness must use a real query over container-local TCP loopback (psql -h 127.0.0.1), not default Unix-socket pg_isready. The initdb temporary server listens only on its socket, while the final server accepts loopback TCP even under --network none. Require two consecutive exact current_database() matches inside a bounded retry window as additional stability evidence; a successful wrong name fails immediately and a non-zero query resets the streak.
• Define target confirmation in the rendered runner and make the contract extract or directly drive that same implementation. A fixture-only wait loop that stabilizes PostgreSQL before testing a different runner path is forbidden: it can make the contract pass while the deployed candidate still contains the startup race.
• Raw restore output remains mode 0600, never leaves the off-host node and is deleted with the owned runtime on success and failure.
• Any non-zero pg_restore apply exit is FAIL/NO_GO, even when a numeric "errors ignored" count can be extracted. There is no partial-success allowlist. Relaxing that rule requires a separately approved design proving the exact restored object set.
• A diagnostic failure is an exercise-runner finding until evidence identifies a dump or restore-content defect. It must not be summarized as proof that the latest backup is unrecoverable.
• The tracked runner is a template, not a site-specific script. Remote address, account, paths, container/database names, timer units, build identity and migration fingerprint arrive only through a private JSON with an exact key set. Rendering validates every value's type, charset and safety shape before exclusively creating a candidate; runtime environment variables cannot override those rendered constants.
• The exact rendered candidate must pass the tracked contract and preserve a deterministic SHA-256 before upload. A workspace-only runner, hand-edited candidate or contract failure is a hard upload blocker.
• Private render JSON and site-specific candidates must live outside the repository or under the dedicated ignored /.restore-drill-private/ directory. The renderer checks both input and output with Git ignore rules and fails before candidate creation when an in-repository path is not ignored; operator memory is not a control.
• The mode-0600 raw log is truncated exactly once at initialization. Container start, dump list, target confirmation, restore apply and storage restore each append a fixed stage marker and their private output, so a later failure does not erase earlier-stage diagnostics before final teardown.
• A source migration fingerprint enumerates only direct directory objects under server/prisma/migrations at the source commit, validates every directory name, sorts the names and hashes that exact non-empty set. Files such as migration_lock.toml are not migrations and must never enter the fingerprint.
• A one-shot installer must install every helper into the exact LIB_DIR embedded in the already approved rendered runner, or derive both values from the same validated private render input and compare them before sudo install. It must use the same exact destinations for preflight, installation, residue accounting and cleanup; a path that differs only by lib versus libexec is an entry-contract failure, not an ARM environment exception.
• A root/sudo helper must not use $HOME to locate a caller-owned upload, installer or diagnostic script: sudo normally changes it to the root home. Self-cleanup may use $0 only after validating the fixed basename, regular file type and absence of a symlink; otherwise a non-root outer wrapper owns cleanup of the exact caller-side path.
• A runner's temporary_access_cleanup=PASS or equivalent self-report proves only that the destinations encoded in that runner's cleanup contract are absent. It is not a zero-residue claim: an installer can place an object outside that destination set and the runner cannot observe it. Before an attempt is called clean, an independent post-run audit must enumerate the installer manifest, runner/probe entries, sudoers candidates, intended and rejected dependency destinations, user staging, owned runtime and container identities; every count must be zero and the audit must clean itself.

4. Validation & Error Matrix

• Dump list non-zero -> dump_list=FAIL, later stages NOT_RUN, category DUMP_LIST_NONZERO / dump_list_failed.
• Bounded TCP target queries remain non-zero -> target_confirmation=FAIL, apply NOT_RUN, category TARGET_QUERY_NONZERO; preserve the last numeric exit, attempt count and zero success streak. All query retries and their one-second intervals accrue only to rto_target_confirmation_seconds.
• Target query succeeds but the exact shadow name differs -> fail immediately with target_name_match=no, category TARGET_NAME_MISMATCH; do not retry a confirmed wrong target.
• Independent list and target pass, but the restore script never invokes apply -> category APPLY_EXIT_CODE_MISSING; fail closed.
• Apply exit non-zero -> restore_apply=FAIL, preserve the exact numeric exit and safe numeric/UNKNOWN error count, category RESTORE_APPLY_NONZERO.
• Apply and script exit zero -> restore_apply=PASS; later migration, table and attachment checks still decide the final recovery result.
• Diagnostics emitted only after cleanup, or cleanup residue non-zero -> the attempt is non-clean and cannot be reported as GO.
• A measured stage is negative/non-numeric, a completed stage is NOT_RUN, a later stage is numeric after an earlier failure, or the stage sum differs from data_restore_rto_seconds -> runner contract failure; do not render or upload the candidate.
• Missing/extra render key, newline/shell syntax, unsafe path/name, malformed digest or pre-existing output -> render fails without publishing a partial candidate.
• In-repository private config or candidate path not covered by Git ignore -> render fails before the candidate exists. The dedicated ignored directory remains valid for a locally protected render.
• Any raw-log truncating redirect after initialization, or a missing fixed stage marker -> tracked contract failure; do not publish a candidate.
• A source-tree listing that mixes migration directories with ordinary files -> reject the generator result or filter by Git directory object type before hashing; never reinterpret the extra file as schema drift.
• Candidate differs from the deterministic render or has not passed the tracked contract -> do not upload or install it.
• Installer dependency destination differs from the rendered runner LIB_DIR -> fail before the first privileged file write; do not install into the similar sibling directory and rely on runner cleanup to discover it later.
• A sudo helper resolves caller staging through root $HOME, or cannot prove the fixed identity of $0 -> leave caller-side cleanup to the non-root outer wrapper and report it explicitly; do not guess or unlink a different home.
• Runner cleanup reports PASS but an installer/outer residue count is non-zero or not independently measured -> the attempt remains non-clean; preserve the runner self-report as contract-scoped evidence and do not rewrite it as zero residue.

5. Good/Base/Bad Cases

• Good: dump list and target pass, apply exits 23; fixed fields show DOCKER_STDIN, 23 and RESTORE_APPLY_NONZERO before teardown, then cleanup reports zero residue without exposing the raw error.
• Base: all three stages pass with exit code 0; the runner continues to migration and attachment verification.
• Good: Unix-socket availability alone does not pass; loopback TCP becomes available, two exact target queries succeed consecutively, then apply runs.
• Good: an apply failure records numeric copy/container/list/target/apply durations, leaves database/storage/final verification NOT_RUN, emits their exact partial sum, and only then tears down the private runtime.
• Base: a successful run emits all eight numeric durations once and their sum exactly equals data_restore_rto_seconds; the value remains a data-only exercise RTO, not a complete OA recovery RTO.
• Bad: emit only database_restore_failed after deleting the sole private log.
• Bad: parse and publish arbitrary pg_restore lines, or treat a non-zero exit as success because the restored database appears partly populated.
• Bad: commit one host's concrete runner or reconstruct a supposedly equivalent runner from chat/workspace immediately before an exercise.
• Bad: add two successful queries only to the fixture while the rendered runner still accepts the first socket pg_isready, or use a fixed sleep as an ARM timing workaround.
• Good: the installer mechanically compares its dependency destination with the rendered runner LIB_DIR, installs all three pinned helpers there and checks that same exact directory during residue accounting.
• Bad: type a site path twice, install helpers below lib while the candidate reads libexec, or use sudo $HOME to delete a script uploaded by the calling user.
• Good: runner cleanup reports its contract destinations clean, then a separate fixed-target audit independently returns zero for entries, both intended and previously rejected dependency directories, staging, runtime and containers.
• Bad: summarize temporary_access_cleanup=PASS as “all temporary objects are zero” without checking objects that only the installer knew how to create.

6. Tests Required

• Directly drive the repository scripts/restore-db.sh with a real custom dump through Docker stdin and host-file execution paths; do not replace the script with an equivalent test implementation.
• Inject list failure, target mismatch and list-pass/apply-failure. Assert exact stage enums, later-stage NOT_RUN, execution path and numeric exit code.
• Drive the rendered runner's timing functions with deterministic timestamps. Assert all eight successful stage durations and their exact total; inject target-confirmation and restore-apply failures and assert completed/current numeric values, later NOT_RUN, one total field and timing output before the runtime cleanup marker.
• Statically assert that the bounded TCP confirmation call is enclosed by the dump_list -> target_confirmation and target_confirmation -> restore_apply timing transitions; a retry sleep must not be charged to container start or restore apply.
• Extract the target-confirmation function from the rendered runner and drive it with: socket-only availability/TCP failure, first-query recovery, streak reset, exact-name mismatch and permanent bounded failure. Assert every probe contains -h 127.0.0.1, name mismatch is not retried, and the real PostgreSQL 16 fixture uses this same function before both restore paths.
• Inject a non-zero apply exit and assert the diagnostic is emitted before teardown, teardown passes, and the calling test can still read the fixed fields afterwards.
• Scan captured diagnostics for fixture credentials, connection strings, SQL, temporary paths and free-form tool output.
• Run the existing synthetic paired backup/restore and automatic off-host contract suites after changing the diagnostic runner.
• Render the same private fixture twice and require byte equality; inject shell syntax and require failure before the output file exists.
• Assert an unignored in-repository private config and an unignored candidate path both fail before output creation; assert the dedicated ignored directory accepts synthetic input and output without becoming trackable.
• Statically require exactly one raw-log truncation at initialization and fixed append markers for container start, dump list, target confirmation, restore apply and storage restore.
• Build a source-tree fixture containing migration directories plus migration_lock.toml; assert the fingerprint includes only sorted directory names and still matches the restored successful migration set.
• Directly test source/copy identity, dynamic migration fingerprint and the tracked attachment validator. Run check-sensitive-content.mjs explicitly on all three tracked runner assets and require zero findings.
• Build an installer fixture from a rendered candidate and assert runner entry, exact LIB_DIR, dependency basenames and sudoers destination are one mapping. Mutate only libexec to lib and require failure before any simulated privileged install; verify residue accounting uses the same destinations.
• Exercise a diagnostic helper with sudo/root HOME: $0 plus fixed-basename, regular-file and non-symlink checks may self-clean, while an unverified path must remain owned by a non-root outer cleanup. Assert no root-home path is inferred or emitted.
• Inject an installer destination outside the runner cleanup set. Assert runner cleanup may truthfully pass for its own targets while the independent audit reports non-zero and blocks a clean conclusion; after exact cleanup, rerun the audit and require all enumerated counts plus audit self-cleanup to pass.

7. Wrong vs Correct

Wrong

fixture: wait for two queries
runner: first socket pg_isready -> continue
restore-db > private.log || fail database_restore_failed
RTO = finish - start; stage gaps are inferred later
finally: delete private runtime
print result=NO_GO

Correct

tracked template + exact private config -> deterministic candidate + SHA-256
candidate LIB_DIR == installer dependency destination -> privileged install
contract drives rendered TCP target function -> list -> target -> real restore path
continuous stage transitions -> exact stage sum + one data-only RTO
capture only stage/path/exit/category/duration allowlists
finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime
sudo helper: validated $0 self-cleanup or exact non-root outer cleanup; never sudo $HOME
runner cleanup PASS + independent fixed-target zero-residue audit -> clean attempt
non-zero apply remains FAIL

Scenario: Daily Restricted Off-Host Backup

1. Scope / Trigger

• Trigger: installing or changing automatic paired backups, off-host transfer, retention, freshness alerts or quarterly restore reminders.
• Purpose: keep a 24-hour recovery point without giving the production host ordinary shell or delete access to the ARM archive.

2. Signatures

• Production runner: scripts/run-automatic-backup.sh with run, check-stale and remind-drill modes.
• ARM forced receiver: scripts/receive-offhost-backup.sh.
• Shared host alert helper: scripts/send-wecom-group-alert.py with explicit --webhook-file, --summary-file and --state-file paths.
• ARM watchdog: scripts/check-offhost-backup-deadman.py with archive root, state root, alert helper, Webhook file and a default 93600 second threshold.
• Installers: install-automatic-backup-timer.sh and install-offhost-backup-receiver.sh.
• Local contract: bash scripts/qa/run-automatic-backup-contract.sh.
• Production required environment: REMOTE_DIR, REMOTE_ENV_FILE, COMPOSE_PROJECT_NAME, POSTGRES_DATA_DIR, STORAGE_HOST_DIR, BUILD_SHA, VITE_BUILD_SHA, AUTOMATIC_BACKUP_DIR, AUTOMATIC_BACKUP_STATE_DIR, OFFHOST_TARGET, OFFHOST_SSH_KEY, OFFHOST_KNOWN_HOSTS, ALERT_WEBHOOK_FILE and ALERT_HELPER.
• Receiver config: independent absolute BACKUP_INBOX_DIR, BACKUP_ARCHIVE_DIR, BACKUP_WEEKLY_DIR, RRSYNC_BIN, PYTHON_BIN, DAILY_KEEP and WEEKLY_KEEP.

3. Contracts

• Resolve one active Compose server/database pair and verify its build, PGDATA, storage mount and database identity before backup. Do not use fixed container names or repository-relative runtime data.
• Parse Docker mount identity from docker inspect JSON and require exactly one destination match. Do not split human-formatted Go-template output on tabs or spaces: println inserts argument spacing that can make a visually correct line fail exact path comparison on a real daemon.
• Database connection values used by the automatic wrapper must be exported in an isolated child shell before capture_command; do not invoke env with a complete DATABASE_URL or password assignment in its argv. Docker backup and shadow-restore commands inherit PGPASSWORD with PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...; never use a value-bearing -e PGPASSWORD=... Docker argument.
• Transfer only a complete auto-<UTC timestamp> three-file batch to ARM inbox. The forced key allows write-only rrsync or an exact finalize command; it rejects shell, sender/read and destructive rsync options.
• ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256, then atomically moves the batch to archive. Archive and weekly directories are outside the rsync root. A different run ID for the same UTC day may replace that day's archive only after full candidate validation: rename the old daily point to a same-filesystem rollback name, publish the candidate, rebuild that ISO week's hard-link checkpoint, then remove the rollback. Any publication or weekly-checkpoint failure restores the prior daily point.
• After taking the finalize lock and before reading the incoming partial, ARM reconciles interrupted same-filesystem transactions. A valid .daily-replacement-<UTC-day>.* is deleted only when one valid canonical archive for that day exists; when the canonical is absent it is renamed back from its own verified manifest identity. Invalid, linked, duplicate or otherwise ambiguous daily state fails closed and preserves every object.
• Weekly .rollback.* is authoritative only after full batch validation and is restored when the canonical checkpoint is absent. Weekly .new.* is never an authoritative source; it may be removed only after a valid canonical, rollback or unique valid daily source is established. Invalid canonical, rollback, duplicate candidate, week mismatch or non-unique rebuild source fails closed without deleting evidence. A reconciled daily point forces its ISO-week checkpoint to be rebuilt before finalize continues. When a valid canonical daily point and its old rollback coexist, defer deleting that daily rollback until every required weekly reconciliation has succeeded; a weekly validation or rebuild failure must leave the daily rollback in place.
• Write production last-success.json and prune source copies only after ARM finalize succeeds. Keep seven automatic source/ARM daily points and four ARM weekly checkpoints; unrelated manual batches are outside automatic retention.
• Daily, freshness and quarterly reminder timers are separate. Alert summaries contain categories only. A shared Python helper reads the complete WeCom group robot URL only from a regular, executing-account-owned 0600 file and never prints it. It sends the first failure, worsening/new findings, one recovery notice and the first quarterly reminder, while suppressing an identical repeated state. It does not call the OA server container or resolve an employee.
• ARM runs its own 26-hour dead-man as root. It verifies the newest archive's exact file set, manifest identity, sizes, checksums and timestamp, so a stopped production host/server can still cause a group alert. Missing, corrupt, stale or future-dated archives fail; recovery sends once.
• Production command diagnostics for paired backup, rsync and finalize stay only in the state directory as last-error.log mode 0600. The summary and group message contain only the safe phase category. A later successful off-host run removes the retained diagnostic.
• The production freshness check hard-fails missing, malformed, future, stale or Compose-project-mismatched state. A fresh recovery point with a different build SHA is PASS_WITH_FINDINGS, because deploy-time code identity and recoverable data freshness are separate facts.
• Automatic-backup units freeze the active Compose project, build, PGDATA and storage mount when installed. After any deployment, rollback or direct container rebuild that changes the active container identity or host mounts, compare all frozen values with the unique running server/database pair. If the units are already installed and any value differs, rerun the existing installer with the complete current contract and recheck all identities before declaring the deployment complete; do not patch one unit in place.
• Installer install is disabled-by-default; enable, real transfer, real WeCom alert and all server/account/key changes remain EXTERNAL_GATED.

4. Validation & Error Matrix

• Source mount/build/project drift, lock conflict or incomplete backup -> fail before success state and source retention.
• Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no off-host success state, preserve the source batch and retain a root-only local diagnostic. A corrupt same-day candidate leaves the previous archive intact.
• Valid same-day retry -> replace only that UTC day's point and rebuild its ISO weekly checkpoint; concurrent finalize -> flock rejection.
• Interrupted daily replacement with valid canonical -> remove the verified old rollback only after rebuilding the weekly point; canonical missing -> restore the verified rollback; canonical invalid, weekly rebuild failure or candidates ambiguous -> preserve the remaining rollback evidence and fail.
• Interrupted weekly replacement with valid canonical -> remove safe staging and verified rollback; canonical missing + valid rollback -> restore it; staging-only + one valid daily source -> rebuild; any invalid or ambiguous authority -> preserve all objects and fail.
• Missing, corrupt, stale, future or project-mismatched success record -> hard watchdog failure. Fresh build mismatch -> one finding without a failing exit.
• phase_stale_check_failed can also mean the watchdog failed its runtime identity precheck before reading the success record. Diagnose the nested safe category and active-unit identity first; do not infer that the last archive is stale or corrupt from this outer phase alone.
• Missing/non-regular/symlink/non-0600 Webhook file -> fail closed without printing the file value. Identical alert state -> update/evaluate without send.
• Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver rejection without changing archive.
• ARM host/network/WeCom all unavailable at once -> the group robot path cannot prove its own outage; this physical boundary is not a green state.

5. Good/Base/Bad Cases

• Good: one verified active source creates a complete batch, write-only rsync resumes inside inbox, ARM finalize verifies and archives it, then production writes success and prunes old automatic source points. If production later stops, ARM independently raises one dead-man alert and one recovery notice.
• Good: a deployment rebuilds the server with a different storage mount, then refreshes the installed backup units through the installer and proves every frozen identity matches before closing the deployment.
• Base: local implementation and synthetic contracts pass while both server installers remain disabled and explicitly externally gated.
• Bad: send through compose exec server, hard-code a personal employee number, persist a robot URL in .env/unit/repository, give production a normal ARM login key, put archive below inbox, accept a new host fingerprint automatically, prune local points before target-side finalize, or leave installed backup units bound to a pre-deployment mount until the next scheduled alert.

6. Tests Required

• Shell syntax plus run-automatic-backup-contract.sh using temporary files and mocked Docker/SSH boundaries; it must never contact real hosts.
• Docker mocks must return the real {{json .Mounts}} shape consumed by the runner. A simplified tab-delimited mock is not valid evidence for real mount parsing.
• Contract assertions for same-day replacement, corrupt-candidate preservation, weekly hard-link rebuild, daily/weekly interrupted-transaction recovery, rollback preservation when weekly reconciliation fails, ambiguous-state preservation, source diagnostic mode, build mismatch finding, Webhook mode/symlink rejection and group alert behavior without a server container. Test transport must be explicitly gated and local-only.
• Mock Docker assertions must prove synthetic connection values are absent from wrapper/backup/restore argv while an inherited PGPASSWORD reaches the mocked container command. Test-fixture env KEY=value launch arrays do not need mechanical rewriting because they contain only synthetic values.
• Environment contract assertions for both helpers, both installers, production timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file binding, disabled-by-default installation and absence of a literal complete robot URL in tracked or untracked repository files.
• Deployment and rebuild review must assert that installed production units still match the unique active project, build, PGDATA and storage mount. A changed mount with otherwise healthy /ready is not a completed deployment until installer refresh, timer state and runtime identity all pass.
• Real ARM setup, first paired batch, deliberate source-timer stop or transfer failure, dead-man first/repeated/recovery behavior and quarterly shadow restore are separately approved operational evidence. A direct robot send test alone is not dead-man evidence.

7. Wrong vs Correct

Wrong

# Rebuild the application, observe /ready, but leave installed backup units
# frozen to the pre-deployment runtime identity.
docker-compose <approved-compose-arguments> up -d --build server

compose exec server send-alert --recipient=<personal-employee-number>
rsync -a <batch>/ backup-host:<archive>/
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ...

Correct

# After a deployment/rebuild, compare the installed unit contract with the
# unique active server/database pair. If any frozen identity changed, refresh
# all units through the installer before verifying timers and identity again.
<complete-current-contract-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh enable

# Install only with explicit active identity and a root-only Webhook file path;
# default remains disabled.
<explicit-project-data-build-ssh-and-webhook-file-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh install
# ARM separately verifies archive freshness even when production is unavailable.
python3 scripts/check-offhost-backup-deadman.py <explicit-safe-path-arguments>
# The runner writes success and prunes only after restricted target finalize.
bash scripts/run-automatic-backup.sh run
# The Docker CLI inherits the value without placing it in argv.
PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...

Scenario: Independent Public Application Health Monitor

1. Scope / Trigger

• Trigger: installing or changing the off-host public /ready probe, its ARM timer, application-health alert states or the shared WeCom group-alert helper.
• Purpose: detect an unavailable OA host, edge/TLS path, application or database readiness signal without adding a time-series platform or relying on traffic.
• Topology boundary: the Tencent Cloud server is the OA production host; ARM is the off-host backup and external-health-monitor node and does not run OA. Changing the ARM probe is an operations-monitor rollout, not an OA application deployment.

2. Signatures

• Probe: scripts/check-public-app-health.py --ready-url <https-/ready> --state-dir <root-0700-dir> --webhook-file <root-0600-file> --alert-helper <installed-helper> --failure-threshold 2 --timeout-seconds <1..30>.
• Installer: deploy/trialrun/install-offhost-app-health-monitor.sh install|enable|uninstall.
• Units: hyb-oa-public-app-health.service and hyb-oa-public-app-health.timer; the timer uses OnCalendar=*:0/5.
• Runtime direction: ARM monitor -> public HTTPS /ready -> Tencent Cloud OA. DEPLOY_SERVER continues to identify the OA production host and must never be inferred from the ARM monitoring alias.
• Local contract: bash scripts/qa/run-app-health-monitor-contract.sh.
• Application statuses: FAIL_APP_HEALTH, PASS_APP_HEALTH_READY and FAIL_APP_HEALTH_FLAPPING.
• Probe-state schema v2 phases: healthy, failure_pending, alerted, recovery_pending, recovery_watch and flapping.

3. Contracts

• Run the probe from the existing ARM host against an exact HTTPS /ready URL. Success requires trusted TLS, HTTP 2xx, bounded JSON and data.ok === true.
• Classify the remote role before any write. ARM may receive only the fixed probe/helper/monitor units and its independent state; OA application, database, Caddy, Compose and migrations remain on the Tencent Cloud production host and follow their separate deployment workflow.
• Classify DNS resolution and TCP connection failures separately before the shared fallback catches OSError; keep TLS, timeout, HTTP, response-size and response-contract categories stable. Count any stable failure category toward the same consecutive-failure threshold. The first failure is pending and the second evaluates a failure alert; identical failures remain deduplicated.
• After an alert, require two consecutive healthy samples before emitting one recovery. A single healthy sample followed by another failure stays in the existing incident and sends no recovery noise.
• Keep the recovered category under a 60-minute observation window. The same category reaching the two-failure threshold in that window emits one FAIL_APP_HEALTH_FLAPPING reminder. Later switches stay local until health has remained continuous for 30 minutes, then the probe exits flapping silently. A different category in the recovery window remains a new normal incident and still uses the two-failure threshold.
• Persist only schema version, phase, bounded counts, stable categories and necessary Unix timestamps in a root-owned 0700 directory with atomic 0600 files. Schema v1 migrates conservatively without backfilling a notification. Never persist or send the URL, resolved address, response payload, credential or raw exception.
• The probe returns a notification candidate on repeated samples while the corresponding state is active; the shared helper owns final deduplication. This lets a failed send retry on a later sample without repeating a successfully sent message. Healthy flapping samples may retry the stable flapping reminder but never produce a recovery candidate.
• Application messages use the explicit titles OA 公网健康告警, OA 公网健康恢复 and OA 公网健康链路反复抖动. Shared backup/dead-man statuses retain their generic operations titles.
• Application-health probe state and alert state are separate from backup and dead-man state. The installer must deploy the helper revision that passed the backup contract; an older installed helper may reject the new status tokens.
• install writes only the exact managed probe/helper/state/unit targets and leaves the timer disabled. enable, ARM writes, real robot messages and a controlled failure/recovery drill remain separately approved operations.
• Before any remote write, prove the selected SSH identity can run sudo -n and that no application-health artifact already exists. A password prompt is a stop condition, not permission to request or transmit a password in chat. If the operator creates a temporary privilege rule, remove its exact file at the end, validate sudoers in the same root invocation, invalidate the sudo timestamp and prove the original non-interactive access is gone.
• uninstall removes only the exact application-health units and probe. It retains the shared helper, webhook, state evidence and all backup components.

4. Validation & Error Matrix

• Non-HTTPS, userinfo, query, fragment or non-/ready URL -> reject before a request with ready_url_invalid.
• DNS, TCP connection, TLS, timeout, HTTP, oversized response or invalid JSON/contract -> distinct stable redacted category and non-zero oneshot exit.
• Invalid owner/mode/link/state or alert-helper failure -> fail closed with a fixed local error; do not include the offending value.
• A first isolated failure followed by health -> no failure or recovery message.
• One healthy sample after an alerted failure -> recovery_pending, no recovery message; the second consecutive sample -> one recovery.
• Same-category recurrence before 60 minutes -> one flapping reminder; at exactly 60 minutes -> a new normal incident, not flapping.
• Flapping health shorter than 30 minutes -> remain flapping; at 30 continuous minutes -> return to healthy without a message.
• Invalid/unknown schema, out-of-range counter/timestamp or inconsistent phase fields -> local_state_invalid, with no raw state in output.
• Any overridden managed target or service name -> installer stops before systemctl, file replacement or removal.
• SSH succeeds but sudo -n requests a password -> stop before upload or systemd mutation; resume only after an operator establishes a separately approved, non-chat root path and rerun the complete preflight.
• Temporary privilege rule remains after final health verification -> rollout is incomplete even when the timer is healthy.
• ARM itself or its network/robot path unavailable -> the monitor can be silent; do not report this design as having an independent third-party dead-man.
• Treating ARM as the OA production server, or selecting it as DEPLOY_SERVER for an application rollout -> stop before upload or mutation and correct the topology classification in the active task.

5. Good/Base/Bad Cases

• Good: ARM observes two consecutive unavailable checks, sends one explicit OA-public-health alert, suppresses repeats, confirms recovery with two healthy samples, and emits only one flapping reminder for a same-category recurrence.
• Base: repository contracts pass while units remain disabled; report the code as ready for an approved rollout, not as an active monitor.
• Bad: run the timer only on the OA host, send raw /ready errors, install and enable in one unapproved step, or deploy the probe without its helper revision.
• Bad: describe ARM as the OA production host or deploy OA application artifacts there because the monitor itself is production-facing.

6. Tests Required

• The local HTTPS stub must assert healthy baseline, pending first failure, second-failure alert, identical deduplication, changed-category alert, two-sample recovery, transient suppression, same-category flapping, different-category recurrence, silent 30-minute exit, exact 30/60-minute boundaries, schema-v1 migration, separate DNS/TCP classification, TLS/timeout/HTTP/size/JSON failures, corrupt private state, URL rejection and absence of sensitive output.
• Captured application messages must assert the three explicit titles and stable status/category only. The automatic-backup contract must prove shared helper titles, recovery and deduplication remain unchanged.
• Run Python compile and shell syntax checks, validate the five-minute calendar, and assert disabled-by-default plus exact managed targets and retained state.
• For a remote rollout, record boolean evidence for pre-write sudo -n, exact temporary-upload removal, temporary privilege removal and post-removal non-interactive-sudo failure; never record the password, host address or rule contents.
• Re-run bash scripts/qa/run-automatic-backup-contract.sh whenever the shared helper allowlist or behavior changes.
• ARM install, enabled/active timer status and real failure/recovery messages are external evidence and may not be inferred from the local contract.
• The rollout task and evidence must name the two roles without addresses or accounts: Tencent Cloud = OA production; ARM = off-host monitoring/backup. Assert that the ARM command set contains no OA deploy, Compose, migration or database mutation entrypoint.

7. Wrong vs Correct

Wrong

# One healthy sample arrived, so announce recovery immediately and keep
# alternating alert/recovery messages during a flapping link.
bash scripts/qa/run-app-health-monitor-contract.sh
bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
# Wrong target classification: the off-host monitor is not DEPLOY_SERVER.
DEPLOY_SERVER=<arm-monitor-target> bash deploy/trialrun/deploy-existing.sh

Correct

bash scripts/qa/run-app-health-monitor-contract.sh
bash scripts/qa/run-automatic-backup-contract.sh
# Runtime contract: fail x2 -> alert; healthy x2 -> recovery; same fail x2
# within 60m -> one flapping reminder; continuous health 30m -> silent exit.
# After separate ARM-write approval: install disabled, inspect, then seek
# enable/real-message/drill approval before claiming the monitor is active.
<explicit-controlled-paths> \
  bash deploy/trialrun/install-offhost-app-health-monitor.sh install
# OA application deployment keeps its separately approved Tencent Cloud target.
DEPLOY_SERVER=<tencent-oa-production-target> bash deploy/trialrun/deploy-existing.sh

Scenario: Existing Deployment Preflight And Cache Stability

1. Scope / Trigger

• Trigger: deploying a new published commit onto an existing single-host trialrun installation.
• Purpose: prevent a wrong repository-root interpretation from reaching rsync --delete, and prevent per-commit build identity from invalidating stable dependency layers.

2. Signatures

• Existing deployment entry: DEPLOY_SERVER=<deploy-ssh-target> DEPLOY_REFRESH_SERVER=<forced-command-ssh-target> bash deploy/trialrun/deploy-existing.sh.
• Existing deployment root-refresh identity: DEPLOY_REFRESH_SERVER=<forced-command-ssh-target>; it is a separate dedicated key/alias, not sudo and not the ordinary deploy identity.
• Aggregate read-only preflight: deploy/trialrun/check-existing-deployment.sh.
• Root-owned forced-command source/installer: scripts/refresh-deployment-contract.sh and deploy/trialrun/install-deployment-contract-refresh.sh.
• Remote preflight: deploy/trialrun/preflight-remote.sh executed over SSH before rsync.
• Persistent long-stage evidence: repo-relative artifacts/deployment-runtime/<stage>-<12-char-sha>.log for build-init, run-init and build-switch-server.
• Canonical parameter source: allowlisted Environment= values from hyb-oa-automatic-backup-daily.service.
• Stable image stages: base, client_deps, client_build, server_deps, server_build.

3. Contracts

• Never derive REMOTE_DIR from Compose com.docker.compose.project.working_dir; it may name the Compose-file directory rather than the repository root.
• The existing-deployment wrapper reads only REMOTE_DIR, REMOTE_ENV_FILE, COMPOSE_FILE, COMPOSE_PROJECT_NAME, POSTGRES_DATA_DIR and STORAGE_HOST_DIR; it does not echo their values.
• Before mkdir or rsync, existing mode requires repository markers, PG_VERSION, storage directory and a regular non-symlink env file when SYNC_ENV=0.
• Before the fail-fast remote preflight, the existing-deployment wrapper runs all aggregate checks without set -e: each check executes in an isolated conditional/subshell, results are emitted in a fixed order, and any failure blocks all writes only after the complete report is produced. Output contains check names and PASS/FAIL only.
• The ordinary deploy identity and the root maintenance identity are separate. Root maintenance uses a dedicated key whose only authorized entry is a forced command with forwarding, PTY and user rc disabled. If the same key blob has an unmanaged or unrestricted root entry, installation fails.
• The forced-command target is root-owned, non-symlink and self-contained under /usr/local/sbin. It accepts exact probe and refresh commands through SSH_ORIGINAL_COMMAND; it never executes the repository, accepts no caller path/unit/SHA, and uses fixed service/timer names.
• If the frozen unit points to a missing or non-repository root, recover the canonical source tree from an exact published Git commit into a new dedicated root. Do not choose among stale source directories by timestamp, and do not use --delete or carry an environment file into the new root during this bootstrap.
• When the frozen REMOTE_ENV_FILE is missing, locate candidates from the running Compose configuration and compare each candidate's service/database values with the running containers by per-key digest. Output only key names and PASS/FAIL; never print values or digests. A candidate must pass every container-observable key, including the field-encryption and database identity keys, before root copies it (not moves it) to the canonical root with deployment-user ownership and mode 0600. If no candidate fully passes, stop and use the environment recovery classification; never use SYNC_ENV=1 to guess.
• When recovering a broken deployment contract or changing frozen paths/project identity, re-run the automatic-backup and WARCH3 installers with the active full BUILD_SHA and matching 12-character frontend prefix. Validate all managed service working directories/env paths/build identities and enabled timers, then run one complete backup plus the freshness watchdog; application /ready alone does not prove that a controlled recovery is complete.
• For normal existing deployments after forced-command installation, the wrapper replaces the manual per-release installer step: after application, smoke and public checks, refresh derives the full SHA from the unique active server, atomically writes root-owned build-only drop-ins for the three automatic-backup services and WARCH3, reloads systemd, and verifies four final service SHAs plus four enabled/active timers. Full installers remain authoritative for path or complete-unit changes.
• A post-refresh failure does not roll back a healthy application. The wrapper emits DEPLOYED_BUT_CONTRACT_REFRESH_FAILED, classifies root transport, missing program, stale unit build or inactive timer, and exits non-zero; only successful refresh may print the complete existing-deployment PASS.
• After that partial-success state, repair the classified root/program/unit/timer problem, invoke the forced-command refresh directly, then rerun aggregate preflight. The application already passed release smoke; do not redeploy it merely to repair the backup build contract, and do not bypass the expected backup_build failure while it is stale.
• apk add and both lockfile npm ci steps precede BUILD_SHA / VITE_BUILD_SHA. Build identity may invalidate application output, never stable dependency installation.
• Seed and smoke output reports readiness, counts and booleans; it does not print full login identifiers, contact values, WeCom identifiers, redirect URI or generated credentials.
• The three long remote mutation stages write stdout/stderr directly to a remote file instead of relying on the SSH stream as the only copy. The log directory is mode 0700; each fresh log is mode 0600. Stage names are a fixed allowlist at call sites, and the filename uses the provenance-verified 12-character candidate prefix.
• A stage success/failure emits only stage, exact child exit code and a repository-relative logRef. Full raw logs remain remote and uncommitted; after failure, retrieve only the needed tail read-only and redact any sensitive content before writing task/chat evidence. Do not automatically dump init/seed logs into ordinary output.
• Logs are created after the verified-source rsync. A later rsync may remove old runtime artifacts, so retrieve a failed stage log before any approved retry. Keeping an interactive execution handle is not a substitute for this second copy.
• An SSH transport loss after a stage starts makes the remote deployment state UNKNOWN; it does not prove that the remote shell or later stages stopped. Before choosing retry or rollback, inspect every candidate-stage log in order, then run independent read-only migration reconciliation, active runtime SHA and public readiness checks. If migration/application switching completed but contract refresh did not, classify partial success and repair only the refresh contract; never replay the deployment.

4. Validation & Error Matrix

• Config directory passed as REMOTE_DIR -> preflight fails before rsync.
• Remote env missing or symlinked with SYNC_ENV=0 -> preflight fails before rsync.
• Multiple environment candidates or any per-key mismatch -> stop before copying, syncing or rebuilding containers; do not select the newest-looking file.
• Environment copy digest, owner or mode mismatch -> stop before formal deploy and keep the original candidate unchanged.
• Existing PGDATA missing PG_VERSION or storage missing -> preflight fails before rsync.
• Canonical systemd unit missing an allowlisted value -> wrapper fails without calling deploy.sh.
• An early aggregate item fails -> all independent items still run, the fixed summary reports every FAIL, and no remote write begins.
• Root SSH returns transport 255 -> root_identity_unavailable; remote 126/127 or missing probe marker -> refresh_program_unavailable.
• Helper runs but final service build differs -> unit_build_not_refreshed; builds match but any timer is disabled/inactive -> timer_not_active.
• Backup mode file /etc/hyb-oa-backup-mode absent -> automatic mode, all backup and ops timers must be enabled/active. Content exactly manual_snapshot (root-owned regular file) -> the three automatic-backup timers must be disabled/inactive, otherwise backup_timer_not_disabled (45); hyb-oa-warch3-ops.timer stays required (timer_not_active). Any other content, symlink or ownership -> backup_mode_invalid (44). In manual mode every backup .service must also be inactive or failed; an in-flight run -> 45. probe runs the same mode and timer/service checks, so a mismatch fails the pre-deploy aggregate as refresh_program FAIL (run the forced-command probe directly to read the category on stderr) before any release write; refresh re-checks after the release. Never mask or delete backup units: deploy identity still reads the daily backup service environment.
• A similarly named environment key such as OLD_BUILD_SHA matches the active value while exact BUILD_SHA remains stale -> treat the service as stale; build verification parses exact keys and applies the last systemd override.
• Post-deploy installer still embeds the previous build SHA, any managed timer is inactive, or the real backup/watchdog fails runtime identity or off-host freshness -> deployment recovery remains incomplete even when /ready is green.
• A later deployment attempt is blocked by backup_build after DEPLOYED_BUT_CONTRACT_REFRESH_FAILED -> expected fail-closed behavior; repair with direct forced-command refresh, not another application release.
• Lockfile unchanged but a new SHA reruns dependency installation -> Dockerfile contract test fails.
• Long stage exits non-zero -> preserve its original status and relative log reference, stop under set -e, and do not enter the next stage.
• SSH stream disappears -> use the already-written remote log for diagnosis; treat remote completion as unknown, inspect all stage logs plus post-status, and do not repeat the deployment merely to recreate console output.
• Runtime log directory/file mode is broader than 0700/0600, stage output has no log reference, or a bare build/init/server-switch call bypasses the wrapper -> deployment evidence contract fails locally.

5. Good/Base/Bad Cases

• Good: wrapper reads the frozen unit contract, preflight passes, Git archive syncs and unchanged dependency layers are cached.
• Good: a long stage writes its complete remote log, returns a stable relative reference and preserves the exact non-zero child status even if the interactive stream is incomplete.
• Good: SSH closes after the first stage marker; all three remote logs and post-status prove migration/application completion, so operations run only the exact contract refresh and finish with aggregate read-only preflight.
• Good: ordinary and forced-command identities pass separately, all aggregate items are reported, deployment passes, root helper updates only fixed build drop-ins, and four timers remain enabled/active.
• Good: a broken-root recovery creates a clean published-commit source tree, proves one environment candidate against running containers without exposing values, copies it with mode 0600, deploys with SYNC_ENV=0, then updates the backup contracts and completes a real off-host backup.
• Base: first initialization uses reviewed explicit parameters and DEPLOY_DATA_MODE=initialize; it does not pretend an existing unit is available.
• Bad: inspect a running container's working_dir, append another deploy/trialrun, then discover the mistake only after rsync.
• Bad: choose one of several stale .env files by path or modification time, use SYNC_ENV=1, or stop after /ready while the backup unit still expects the previous build.
• Bad: grant NOPASSWD to a repository installer, point forced-command at a deploy-user-writable script, reuse an unrestricted root key, stop aggregation after its first failure, or downgrade refresh failure to a warning.
• Bad: run docker compose build/run/up as bare SSH-stream-only calls, lose the terminal tail, then retry production solely to recover diagnostics.
• Bad: interpret a local SSH 255/disconnect as proof that the remote command stopped, then replay migration/application deployment without checking later stage logs or the active runtime.

6. Tests Required

• bash -n deploy/trialrun/deploy.sh deploy/trialrun/deploy-existing.sh deploy/trialrun/preflight-remote.sh deploy/trialrun/production-preflight-remote.sh.
• bash scripts/qa/run-deployment-contract-refresh-contract.sh must prove multi-failure aggregation reaches the final item, output redaction, forced-command exact matching, four failure categories, four build drop-ins and four timer checks.
• node deploy/trialrun/validate-env-contract.js.
• cd server && npm test -- tests/unit/scripts/deployment-safety-contract.test.js.
• Synthetic fixtures must prove wrong root, missing/symlink env and invalid data fail; static ordering must prove preflight precedes rsync and dependency stages precede build args.
• Static deployment tests must prove all three long stages use the persistent wrapper, output is redirected to the mode-0600 file, the error branch captures $? immediately, and the former bare calls are absent.
• A disconnect recovery fixture must model missing local PASS output while later remote stage logs exist; the expected decision is read-only reconciliation and partial-success refresh, with no second deploy command.
• A controlled recovery must additionally record: exact published source commit, per-key PASS set/date without values, byte-copy/owner/mode PASS, new-SHA service/timer contract PASS, a successful complete backup, and a successful off-host freshness check. Old candidates remain untouched until a separately approved cleanup.

7. Wrong vs Correct

Wrong

# working_dir may be the Compose config directory, not the repository root.
  REMOTE_DIR="$(docker inspect <server> --format '<working_dir-label>')" \
    bash deploy/trialrun/deploy.sh

Correct

DEPLOY_SERVER=<approved-ssh-target> \
  DEPLOY_REFRESH_SERVER=<approved-forced-command-target> \
  bash deploy/trialrun/deploy-existing.sh

# A normal release ends here: the wrapper refreshes the fixed build drop-ins
# and verifies all four timers. Do not manually start backup services on every
# release. If contract recovery was required, follow the separately approved
# recovery runbook to prove one complete backup and off-host freshness.

# Wrong: the SSH stream is the only build evidence.
"${compose_cmd[@]}" build init

# Correct: the remote file is authoritative and the stream carries a safe ref.
run_logged_deploy_stage "build-init" "${compose_cmd[@]}" build init

# A disconnect means UNKNOWN, not failed-before-write:
# inspect build-init/run-init/build-switch-server logs and post-status first.
# If app/schema are current and only backup_build is stale, run refresh only.

Scenario: Production Migration-First Target Gate

1. Scope / Trigger

• Trigger: an approved existing-host release whose candidate commit contains a Prisma migration, especially the contract terminated workspace migration.
• Purpose: prove the remote target identity and migration history before any schema write, then preserve the state-based rollback boundary.

2. Signatures

• Wrapper: deploy/trialrun/deploy-existing.sh.
• Local orchestrator: deploy/trialrun/production-preflight.mjs.
• Remote read-only helper: deploy/trialrun/production-preflight-remote.sh.
• Required process inputs: EXPECTED_PRODUCTION_ORIGIN, EXPECTED_PRODUCTION_DB_NAME_SHA256, EXPECTED_PENDING_MIGRATIONS, and the exact deployment target/directory/Compose identity values. The strict wrapper must also pass its provenance-verified CANDIDATE_COMMIT; preflight must not fall back to git rev-parse HEAD.
• Candidate Compose interpolation: BUILD_SHA=<full candidate commit> and VITE_BUILD_SHA=<first 12 hex characters of BUILD_SHA>.

3. Contracts

• Production R2 is the inverse of the local migration task: the gate must prove the target is the expected production HTTPS origin and database identity, and reject loopback, local regression, staging, shadow and placeholder identities. A legacy trialrun token is neutral; the gate must never infer production or non-production from an SSH alias, Compose label or directory name alone.
• The strict production-validation wrapper rejects unpublished-commit overrides; disaster-recovery exceptions use a separately approved recovery flow.
• Expected target values remain process-only. The local orchestrator sends only one-way digests to the remote helper and records stable codes, not values, credentials, paths or raw environment content.
• The candidate manifest is generated from the verified commit. Before deploy.sh, the remote helper confirms the runtime DATABASE_URL host and database identity, then runs prisma migrate status. Raw stdout/stderr stays in the controlled execution environment; the local report contains only a redacted status copy. The helper compares _prisma_migrations with the candidate manifest. Pending must exactly equal the explicit allowlist; pending IDs reported by status must belong to the candidate manifest; failed, rolled-back, unfinished, diverged or unknown history fails closed.
• docker compose config validates the candidate configuration that would be deployed, not the currently running container configuration. The local orchestrator derives both Compose build variables only from the provenance-verified candidate: the full commit becomes BUILD_SHA and its first 12 characters become VITE_BUILD_SHA. The remote helper requires both, validates their hexadecimal lengths and asserts the prefix relationship. Never use --no-interpolate, which would bypass required ${VAR:?...} constraints instead of checking the candidate configuration.
• The preflight helper is read-only: it does not run migrate dev, reset, db push, migrate resolve, seed, DDL or business writes. deploy.sh is reached only after target and pre-status gates pass.
• The post-deploy status is a separate evidence item and must be manually read; the wrapper re-runs the read-only helper with an empty pending allowlist before backup-contract refresh, and a successful exit code alone is insufficient.
• Each preflight run removes only its two known report files before SSH; a reused report directory must never leave an old PASS status artifact in place.
• After the stable remote summary and report reference are emitted, a non-zero remote exit is evaluated before Prisma marker completeness. A parsed remote code=... remains the primary stop reason; a non-zero exit without a safe code becomes PRODUCTION_PREFLIGHT_REMOTE_FAILED. Only a zero remote exit without the required marker becomes PRISMA_STATUS_OUTPUT_MISSING. This ordering preserves both the original safety diagnosis and failure evidence.
• When the strict wrapper invokes deploy.sh, it sets DEPLOY_STATUS_OUTPUT_MODE=quiet; the legacy status check must not echo raw datasource output. Detailed evidence comes from the controlled pre/post helper artifacts.
• terminated count defines a state window: zero rows means OPEN; the first committed row closes it. Once closed, an old application that does not know terminated cannot be restored and the new columns are never auto-dropped; recovery is roll-forward or a compatible hotfix.
• Window evidence has two explicit bases. If the termination migration is not in the applied set, do not issue the status query; record rollbackWindow=OPEN terminatedRows=0 basis=migration_not_applied. If it is applied, query contract_registry.status = 'terminated' and record basis=terminated_status_count. Any query/parse failure in the applied branch fails closed; it must never be caught and converted into OPEN.
• The target identity must remain unchanged between the read-only preflight and the schema write; if a controlled change freeze cannot be established, stop before deploy.sh rather than relying on post-status to repair a wrong-target write.

4. Validation & Error Matrix

• Missing or invalid expected target input -> stop before SSH mutation.
• Target hash, runtime mode, cookie security or DB identity mismatch -> stop before deploy.sh.
• Missing/invalid provenance candidate -> CANDIDATE_COMMIT_MISSING or CANDIDATE_COMMIT_INVALID; do not infer a candidate from local HEAD.
• Invalid full/short build SHA or a short SHA that is not the first 12 characters of the full SHA -> PROD_TARGET_BUILD_SHA_INVALID, PROD_TARGET_VITE_BUILD_SHA_INVALID or PROD_TARGET_BUILD_SHA_MISMATCH; stop before Compose config and Prisma status.
• Status output missing/unreadable, failed/rolled-back/diverged history or an unknown pending migration -> PAUSED; do not reset or resolve in place.
• Status predicates use one status_matches helper with direct here-string input, not printf | grep -q under pipefail. An early-exiting consumer can yield producer=141/consumer=0 and falsely skip unknown-pending rejection. Matcher exit 0 means match, 1 means no match, any other status immediately fails with PROD_MIGRATION_STATUS_UNREADABLE. Preserve candidate/history reconciliation; do not infer safety from the absence of parsed pending IDs.
• Non-zero remote exit with a safe code=..., with or without a Prisma marker -> propagate that code and stop. Non-zero exit without a parseable code -> PRODUCTION_PREFLIGHT_REMOTE_FAILED; never continue to marker validation.
• Zero remote exit without the Prisma marker -> PRISMA_STATUS_OUTPUT_MISSING; do not treat the transport success as a complete pre-status result.
• Pending set differs from the approved allowlist -> PAUSED and retain the status artifact for human review.
• Termination migration absent from the candidate manifest -> PROD_TERMINATION_MIGRATION_NOT_IN_CANDIDATE; the window basis is unknown.
• Termination migration not applied -> explicit OPEN with basis=migration_not_applied, without querying the status value.
• Termination migration applied but the count query fails or returns a non-integer -> PROD_TERMINATED_COUNT_UNAVAILABLE or PROD_TERMINATED_COUNT_INVALID; never default to OPEN.
• terminated count greater than zero -> record rollbackWindow=CLOSED and disallow compatibility rollback; this task never creates such a row.

5. Good/Base/Bad Cases

• Good: the remote helper exits non-zero with a stable database-identity code; the wrapper writes the stable evidence summary, returns that exact code and stops before migration or deployment.
• Good: the verified candidate commit is injected as the full build SHA and its 12-character prefix, so Compose validates exactly the configuration that a later approved deployment would activate.
• Base: the remote helper exits zero with a readable Prisma marker and the wrapper continues into migration reconciliation.
• Base: the approved termination migration is pending, so reconciliation passes and the window is explicitly OPEN with basis=migration_not_applied; no status query is attempted.
• Bad: check for a missing marker before checking the remote exit and replace the real remote safety code with PRISMA_STATUS_OUTPUT_MISSING.
• Bad: omit build SHA interpolation, use a seven-character Git abbreviation, re-read HEAD inside preflight, or pass --no-interpolate merely to make Compose config return zero.

6. Tests Required

• node --check deploy/trialrun/production-preflight.mjs.
• bash -n deploy/trialrun/production-preflight-remote.sh.
• node deploy/trialrun/validate-env-contract.js.
• cd server && npm test -- tests/unit/scripts/deployment-safety-contract.test.js.
• Inject the observed producer 141 into the actual status parsing/rejection source; restored old-predicate mutation must demonstrate lost rejection. Cover safe/allowlisted/unknown/unsafe/unreadable states, 96 KiB trailing output and matcher failure. The multi-subprocess remote fixture has a bounded 15 s test budget (not a production timeout change); assertions are unchanged.
• Static ordering must prove production preflight precedes deploy.sh, and the remote status marker precedes migration-history reconciliation. Fixtures must reject loopback/staging target identities without invoking SSH.
• Cover all remote-result precedence combinations: non-zero with a safe code and missing marker; non-zero with a safe code and present marker; non-zero without a safe code; zero with a missing marker. Also assert the stable summary and report reference are emitted before either failure check.
• Assert the SSH command receives the provenance candidate as BUILD_SHA and its exact 12-character prefix as VITE_BUILD_SHA; reject malformed and individually valid-but-mismatched pairs. Static checks must prove there is no local-HEAD candidate fallback.
• Cover both window bases: a pending termination migration must return explicit OPEN without issuing the status query; an applied migration must query and distinguish zero/positive counts. Force the applied-branch query to fail and assert PROD_TERMINATED_COUNT_UNAVAILABLE, not OPEN.

7. Wrong vs Correct

Wrong

missing Prisma marker -> PRISMA_STATUS_OUTPUT_MISSING
-> inspect remote non-zero exit afterward

Correct

write stable evidence -> inspect remote non-zero exit and propagate/fallback
-> only after remote success require the Prisma marker

# Wrong: validate a non-interpolated or unrelated working-tree configuration.
git rev-parse HEAD -> docker compose config --no-interpolate

# Correct: validate the exact published candidate configuration.
provenance-verified commit -> BUILD_SHA + first 12 chars as VITE_BUILD_SHA
-> interpolated docker compose config

# Wrong: a missing column/query error happens -> catch -> assume OPEN.

# Correct: migration not applied -> explicit OPEN without querying;
# migration applied -> required count query, with any error stopping closed.

Scenario: Environment Recovery Classification And Secret Custody

1. Scope / Trigger

• Trigger: adding/removing a trialrun environment key, documenting host-loss recovery, or preparing an environment-file migration.
• Purpose: make environment loss recoverable without placing Secret values in Git, ordinary backup, logs or AI context.

2. Signatures

• Template contract: deploy/trialrun/.env.example.
• Recovery classification: deploy/trialrun/env-recovery-classification.json.
• Recovery card: the opening section of deploy/trialrun/runbook.md.
• Validator: node deploy/trialrun/validate-env-contract.js.
• Fixed categories: template_rebuild, external_reissue, direct_rotate, downtime_reset, preserve_original.

3. Contracts

• Every .env.example key appears in exactly one fixed recovery category; the classification file contains key names only, never values, defaults, hashes, accounts or platform addresses.
• template_rebuild comes from reviewed templates/runbooks; external_reissue comes from the owning platform; direct_rotate records the invalidation consequence; downtime_reset requires an isolated approved outage.
• POSTGRES_PASSWORD is a reset case. PostgreSQL stores a verifier in pg_authid, not recoverable plaintext. Any temporary authentication relaxation is local-socket only, never a trusted host rule, and the original authentication file must be restored byte-for-byte before reopening the application.
• FIELD_ENCRYPTION_KEY is preserve_original and the highest-risk class. Its current AES-256-GCM payload is v1:<iv>:<tag>:<ciphertext> without a key id; replacing a lost key does not recover existing bank, identity or notification-channel ciphertext.
• preserve_original custody must be available to both the system administrator and a company responsible person through a repository-external shared vault or encrypted sealed copy. Evidence records roles and status only.
• Original-key custody is a current recovery constraint, not a permanent rotation design. Key id, keyring, lazy/bulk re-encryption and old-key retirement require a separate approved security task.
• Database/attachment backups do not absorb plaintext environment files. Phase 1 recovery-contract work never reads production env or contacts production systems.

4. Validation & Error Matrix

• New .env.example key missing from classification -> validator fails.
• Key appears in two categories -> validator fails and names the duplicated key.
• Unknown category, extra key, illegal key name or value-like entry -> validator fails.
• FIELD_ENCRYPTION_KEY outside preserve_original -> validator fails.
• POSTGRES_PASSWORD outside downtime_reset -> validator fails.
• Original encryption key unavailable -> stop recovery and report ciphertext impact; never generate a replacement and claim success.
• Temporary database trust remains or a DB host port is published -> recovery fails and the external entry remains closed.

5. Good/Base/Bad Cases

• Good: a TTL is rebuilt from the template, a WeCom Secret is reissued by its platform owner, JWT is rotated with session invalidation, and the field-encryption key is restored from dual-access custody.
• Base: the production key set differs from the example contract; a separately approved read-only check explains key names/counts without reading values before migration.
• Bad: put .env into the paired data backup, print a secret hash as evidence, assume pg_authid reveals the old password, or replace FIELD_ENCRYPTION_KEY with a new random value.

6. Tests Required

• node deploy/trialrun/validate-env-contract.js.
• Focused unit tests must cover the complete contract plus missing, duplicate, extra, unknown-category, illegal/value-like, field-key and database-password misclassification cases.
• Run node scripts/qa/check-sensitive-content.mjs, Trellis governance, process contract, docs registry audit and git diff --check after classification/runbook changes.
• Production key-name comparison, shared-vault custody, env migration, systemd updates, restarts and old-copy deletion remain separately approved Phase 2 evidence.

7. Wrong vs Correct

Wrong

{"direct_rotate":["FIELD_ENCRYPTION_KEY"]}

Correct

{"preserve_original":["FIELD_ENCRYPTION_KEY"]}

Scenario: Formal Feature-Flag Baseline And Reversible QA Window

1. Scope / Trigger

• Trigger: a feature is formally enabled after an earlier short-window rollout, or a QA runner temporarily changes an existing feature flag.
• Purpose: prevent a new database, stale .env or QA cleanup path from silently returning a formal capability to its old disabled state.

2. Signatures

• Hard gate: PAYMENT_BATCH_ENABLED=true in deploy/trialrun/.env.example and the active production env.
• Database baseline: payment_batch_v1 is first-created as force_on/global; seed:config still uses create plus P2002 preservation and never updates an existing row.
• Deployment assertion: smoke-trialrun-runtime.js requires both the env value and effective FEATURE_FLAG_KEYS.paymentBatch result.
• QA snapshot helper: node scripts/qa/payment-batch-flag-snapshot.mjs capture|restore --snapshot-file <private-json>.
• Mobile entry: /mobile/payment-batches; PC remains /payment-batches.

3. Contracts

• Formal enablement is a three-layer contract: env hard gate, database feature flag and runtime smoke. Changing only one layer is incomplete.
• A short QA window captures overrideMode, rolloutType, departmentIds and note before its first flag write and restores those fields in finally; it must work when the caller starts either enabled or disabled.
• A formal-enable runner keeps force_on only after every acceptance step passes. Any failed step restores its caller snapshot, not a hard-coded force_off.
• Snapshot files use mode 0600, contain configuration only and remain under the ignored QA report directory. Credentials and access tokens never enter the snapshot or output.
• Mobile links must target the mobile route. Enterprise-messenger routing maps a PC-only payment path back to the mobile workbench, so a PC path is not a valid mobile deep link even when the underlying page is responsive.
• Mobile and PC payment routes share the same permission guard; adding reachability must not broaden payment.batch.manage|payment.batch.review|risk.review.manage.

4. Validation & Error Matrix

• Env false or absent -> runtime smoke fails before deployment is accepted.
• Database flag absent/disabled -> runtime smoke fails with the feature-flag reason; do not hide the failure by changing navigation only.
• Seed meets an existing flag -> P2002 is preserved with zero update/upsert calls.
• QA capture fails -> stop before the temporary window; do not run with an unknown rollback target.
• QA action or later matrix fails -> restore the caller snapshot and keep the run failed.
• Snapshot restore version conflicts -> fail visibly; never fall back to force_off.
• Mobile finance entry points at /payment-batches -> small-screen routing returns to /mobile; treat this as an unreachable entry and use /mobile/payment-batches.
• User lacks payment permission -> both routes remain forbidden even if the feature is enabled.

5. Good/Base/Bad Cases

• Good: env true + seed first-create force-on + effective runtime smoke pass; a temporary off/on test restores an original allowlist and note byte-for-byte.
• Base: a database already contains an operator-managed flag; deployment seed preserves it and runtime smoke reports whether it matches the formal baseline.
• Bad: change only the DB flag while env remains false, or make the mobile card visible but link it to a path that the mobile shell redirects home.
• Bad: every QA cleanup writes force_off, erasing a formal-on state that existed before the test.

6. Tests Required

• Seed unit test asserts first-create force_on/global and proves P2002 performs no update/upsert.
• Snapshot helper tests cover an original force_off allowlist and an original force_on global configuration, including note restoration.
• Deployment contract test asserts the env baseline, runtime-smoke flag assertion and absence of fixed-off cleanup helpers.
• Mobile Playwright covers zero batches/zero exceptions with the feature enabled, a visible “进入付款” link, successful /mobile/payment-batches navigation and no finance card for an unrelated role.
• Run client typecheck/build, Prisma validate, shell/Node syntax, task/governance/process/sensitive gates and git diff --check.
• Public rollout additionally verifies active env, effective flag reason, finance navigation and eligible candidate count without creating, reviewing or executing a real batch.

7. Wrong vs Correct

Wrong

# Cleanup assumes disabled was always the caller's state.
PUT payment_batch_v1 '{"overrideMode":"force_off"}'

<Link to="/payment-batches">进入付款</Link>

Correct

node scripts/qa/payment-batch-flag-snapshot.mjs capture --snapshot-file "$REPORT_DIR/flag-before.json"
# temporary test writes
node scripts/qa/payment-batch-flag-snapshot.mjs restore --snapshot-file "$REPORT_DIR/flag-before.json"

<Link to="/mobile/payment-batches">进入付款</Link>

Scenario: Production Configuration Write Observation And Uncertainty

1. Scope / Trigger

• Trigger: an approved production runner changes feature flags or another versioned/runtime configuration through an API or environment file, then uses browser/network observation, response parsing, container health metadata or local counters to decide whether the write landed or whether rollback is needed.
• Purpose: prevent an observer defect from turning a known successful write into an unnecessary compensating write, while preserving zero retry and exact audit evidence.

2. Signatures

• Forward write: the approved mutation endpoint, for example POST /api/system/feature-flags/batch, with frozen expectedVersion values.
• Authoritative readback: GET the changed resources and compare version, configured value and effective state with both the PRE snapshot and the expected POST snapshot.
• Independent evidence: action-specific audit total and per-target changelog deltas.
• Observer sources must be named separately: page requests, browser-context requests, APIRequestContext, server response and audit/readback.
• Container restart witness: inspect whether the service actually defines a Docker healthcheck, then combine running state, restart count, public /ready and the changed capability's read-only response. An absent healthcheck is health=none, not an unhealthy result.

3. Contracts

• A local request counter is evidence only for the transport it actually observes. A Playwright page.on("request") listener does not observe writes sent through context.request.fetch or another APIRequestContext.
• Missing observer events, contradictory counters, missing output, non-zero transport status or invalid JSON enters APPLY_UNCERTAIN. The first action is read-only resource readback; audit and changelog are the second independent source.
• Before readback classifies the result, prohibit every further mutation: no forward retry, rollback, compensation, single-item repair or workspace/config cleanup. Rollback is a state change and follows the same uncertainty rule as retry.
• Readback equals expected POST (version=PRE+1, desired value/effective state on every target) -> classify the write as applied and continue POST verification. A zero local counter cannot override authoritative state.
• Readback equals the PRE snapshot -> classify as not applied and stop. A later attempt requires a fresh approval/version lock unless the original approval explicitly defines another attempt.
• Partial or mixed readback -> return STOP_FOR_MANUAL_DECISION, record a contract/product finding and stop. Do not continue POST, auto-rollback or conceal a broken atomicity contract with manual repair.
• Execute an approved rollback only after the forward result is known and a defined POST acceptance condition actually fails. Capture rollback response, final readback and audit deltas separately from the forward evidence.
• Make POST validation proportional to the changed dependency graph and failure cost. A reversible, zero-use feature flag with statically verified soft consumers does not justify probing unrelated domains merely because they are available.
• For a confirmed low-risk, reversible configuration state, a POST witness failure defaults to preserving the applied state for read-only diagnosis. Rollback requires evidence of a real product regression, not merely a runner assertion, stale error accumulator or probe-contract mismatch.
• Every HTTP probe must use the real route query contract. Page-error evidence must be sliced from immediately before one navigation until that navigation completes; errors accumulated during login, PRE or earlier pages cannot fail a later page witness.
• A restart waiter may require Docker healthy only when the resolved running service defines a healthcheck. If none is defined, it must not manufacture a failure; use running state, stable restart count, public /ready and dependency-specific read-only capability checks before deciding whether rollback is warranted.

4. Validation & Error Matrix

• Response count is correct but local observed-write count is zero -> APPLY_UNCERTAIN; GET versions/state, do not rollback from the counter alone.
• Response missing or JSON invalid -> APPLY_UNCERTAIN; GET versions/state and audit, never retry blindly.
• All targets at expected POST -> continue read-only POST checks; rollback only for a real approved acceptance failure.
• All targets at PRE -> stop with zero compensating writes.
• Some targets changed -> STOP_FOR_MANUAL_DECISION; stop, preserve evidence and request an explicit human decision. Neither continue nor rollback is automatically correct.
• Readback itself fails -> state remains unknown; stop with zero further mutation.
• Expected POST is confirmed and the only witness fails -> preserve the applied state, open a fresh read-only context and classify the witness. Confirmed product regression may enter an explicitly approved rollback; runner/probe defect must not.
• Container is running with health=none, restart count is stable, public /ready passes and the changed capability is ready -> classify the health waiter as incompatible with the runtime contract; preserve the applied state and fix the observer separately.
• Container exits, restart count increases, public /ready fails or the changed capability remains unavailable -> treat as a real POST acceptance failure and follow the approved rollback/stop contract.

5. Good/Base/Bad Cases

• Good: APIRequestContext writes a three-item batch while a page listener reports zero; GET shows all three at PRE+1, audit is +3, so the runner continues POST verification without rollback.
• Base: the batch response and observer agree, readback confirms the expected state, and the audit delta matches target count.
• Base: a configuration-only restart targets a service without Docker healthcheck; running state, restart count, public readiness and the changed capability all pass, so the applied state is retained.
• Bad: treat a page-only counter as authoritative for a context request and immediately restore the PRE values before checking current versions.
• Bad: call rollback because it is pre-authorized while the forward result is still unknown.
• Bad: add unrelated domain probes to a three-flag soft-disable, send unsupported query parameters, or let errors from an earlier page fail the current Portal witness.
• Bad: require .State.Health.Status=healthy without first proving a healthcheck exists, then rollback a healthy service because Docker reports no health object.

6. Tests Required

• Local integration/harness case where the write is sent through APIRequestContext, the page listener observes zero, the server returns success and readback shows PRE+1; assert no rollback and no retry occur.
• Cases for response success/counter agreement, response missing with POST state, response missing with PRE state, readback failure and partial/mixed state.
• Assert mutation ordering: forward at most once -> uncertain readback/audit only -> either POST verification or stop -> rollback at most once only after a concrete approved acceptance failure.
• Assert final evidence reconciles response count, readback versions, global audit delta and per-target changelog delta; no one source may silently override contradictions.
• Derive every probe path/query from the actual route/client contract. For browser witnesses, reset or snapshot the page-error index before each navigation and assert only the new slice.
• For low-risk reversible changes, test that a witness failure preserves POST state for diagnosis and that rollback is unreachable until a fresh read-only check confirms product regression.
• Cover both resolved service shapes: healthcheck present must reach healthy; healthcheck absent must require running state, unchanged restart count, public /ready and the dependency-specific capability. Assert health=none alone cannot trigger rollback.

7. Wrong vs Correct

Wrong

forward response count=3 + page observer count=0
-> assume failure -> rollback immediately

Correct

forward response/counter disagree -> APPLY_UNCERTAIN
-> GET every target version and state -> reconcile audit/changelog
-> expected POST means continue verification
-> PRE means stop
-> minimal dependency-matched witness
-> witness failure preserves state for fresh read-only diagnosis
-> rollback only after confirmed product regression and explicit approval

Wrong

restart service -> always wait for Docker healthy
-> no healthcheck object -> declare failure -> rollback

Correct

resolve service healthcheck contract
-> healthcheck present: require healthy
-> healthcheck absent: running + stable restart count + public /ready
   + dependency-specific capability
-> rollback only for a confirmed POST acceptance failure

Scenario: Unique Production Administrator Break-Glass Reset

1. Scope / Trigger

• Trigger: the only active production administrator cannot authenticate, self-service reset is unavailable, and a separately approved task restores access with the existing break-glass script.
• Purpose: make the single irreversible reset attempt observable without exposing credentials or treating historical audit rows as proof of the current run.

2. Signatures

• Read-only commands: --preflight, --verify, --verify-password-changed, --diagnose-wecom-binding on server/src/scripts/reset-active-admin-password.js.
• Single write command: node src/scripts/reset-active-admin-password.js --apply ACTIVE_ADMIN_BREAK_GLASS, with the temporary password on stdin only.
• Browser lifecycle: /login -> /change-password -> /portal, one login submission and one password-change submission.

3. Contracts

• Compare the running-container script SHA with the reviewed deployed candidate before selecting a target. A workspace copy must never be pushed into the container to make the check pass.
• Create a local apply-attempt sentinel before opening the SSH/container channel. Missing output, non-zero transport status or invalid JSON enters APPLY_UNCERTAIN; only --verify may classify the result, and apply is never retried.
• Capture the break-glass audit count before apply and require an increment of exactly one afterward. auditPresent=true alone is insufficient because a historical reset may already satisfy that boolean.
• --verify immediately after apply must prove the unique target, must-change=true, cleared locks, zero active sessions, zero active password-reset codes, zero active email-login codes and a break-glass audit.
• Atomically replace the repository-external smoke secret first with the temporary credential and later with the final credential. Before consuming the one allowed login, read the file back in-process and compare both identifier and password to the controlled source; a successful rename without content agreement is not readiness.
• The browser runner receives credentials through controlled file descriptors/stdin, blocks non-authentication business APIs, submits each form once, clears browser state and does not enter business pages.
• A prior hash may be kept only as approved 0700/0600 short-lived recovery material. Destroy it and all temporary credentials after --verify-password-changed passes; evidence stores booleans/counts, never values or paths.

4. Validation & Error Matrix

• Running script SHA mismatch, target count other than one or binding conflict -> stop before recovery material and apply.
• Active session with recent activity -> stop and confirm operator ownership; an old inactive session may be revoked only when that side effect is already approved.
• Apply channel/result uncertain -> write no second password; run --verify only.
• Apply PASS but --verify fails -> do not log in; retain controlled materials until a new decision or the custody deadline.
• Secret update content mismatch -> do not consume the login attempt; fix the local atomic update and re-verify without another apply.
• Login or change-password submission fails -> zero retry; preserve the known credential state and request a new decision.
• Recovery-material deletion fails -> task remains incomplete even when the final password works.

5. Good/Base/Bad Cases

• Good: audit baseline 1 becomes 2 after one apply, immediate verify passes, one login completes forced change, final verify passes, and all temporary material is destroyed.
• Base: audit baseline 0 becomes 1 with the same state-machine checks.
• Bad: accept auditPresent=true when the baseline was already positive, blindly retry after an SSH interruption, or attempt login before proving the temporary secret contains the new credential.

6. Tests Required

• Source review must trace every session/challenge update to where.employee_id=target.id. When this script's write behavior changes, focused unit tests must additionally assert those exact where arguments; call-count assertions alone do not prove target scoping.
• Unit tests cover non-unique target refusal, transaction-time target change, apply side effects and both verify modes.
• Execution evidence asserts runtime-script SHA match, apply-attempt count 1, audit delta exactly 1, the complete post-apply verify fields, login/change submit counts 1/1 and cleanup booleans.
• Run focused script tests, Node syntax, task/process/governance/sensitive gates and git diff --check; an authentication-state-machine task additionally receives a read-only reviewer.

7. Wrong vs Correct

Wrong

apply output missing -> retry apply
verify.auditPresent=true -> current audit write proved
atomic rename succeeded -> consume the only login attempt

Correct

create attempt sentinel -> apply once -> uncertain output means verify only
capture audit baseline -> require baseline + 1
atomic rename -> compare identifier/password in-process -> login once
-> change password once -> final verify -> destroy temporary material

Scenario: Dedicated Employee Smoke Break-Glass Reset

1. Scope / Trigger

• Trigger: a dedicated OA-only smoke employee is the approved negative-permission sample, its identifier is uniquely discoverable but its password is unusable, and changing a real employee is not acceptable.
• Purpose: restore only that smoke account's authentication state while preserving the exact employee-only role and proving that no contract-management permission is introduced.

2. Signatures

• Read-only commands on server/src/scripts/reset-employee-smoke-password.js: --preflight, --verify, and --verify-password-changed.
• Single write command: node src/scripts/reset-employee-smoke-password.js --apply EMPLOYEE_SMOKE_BREAK_GLASS, with the temporary password on stdin only.
• server/src/scripts/repair-oa-only-smoke-accounts.js is a read-only cross-check for this scenario. Its --apply path is not a password-reset substitute.

3. Contracts

• Discover the target from production state; do not accept or emit a complete identifier. The unique target must be active, employed, unarchived, identifier-ready, unlocked, not awaiting a password change, matched by the OA-only smoke predicate, and have roles exactly equal to ["employee"].
• Resolve permissions independently from the exact role list and require *, project.manage, project.view_all, and system.settings all to be absent. Role equality alone is insufficient if the role-permission mapping later expands.
• Run the repair helper in dry-run mode before apply and require actions=[], passwordReset=false, and a safe smoke target. A stale built-in default target, a non-empty action plan, or an unsafe target is a stop condition; never repair forward to make reset preflight pass.
• Use a Serializable transaction that locks the employee and role rows, then rechecks identity version, status, smoke classification, exact roles, and permissions before the sole employee update. Role tables are read-only throughout this script.
• The write set is limited to the target's password and authentication counters/locks/token version, target-scoped sessions and challenges, and one dedicated audit event. It must not change roles, status, identifiers, WeCom binding, schema, or business data.
• After apply, both script verification and an independent read-only snapshot must recompute exact roles and all four permission booleans before any login is allowed. A successful preflight is not reusable as post-write proof.
• Apply-attempt, audit-delta, zero-retry browser, repository-external secret, recovery-material, and redacted-evidence rules are the same as the administrator break-glass scenario.

4. Validation & Error Matrix

• Target count other than one, role list not exactly employee, any forbidden permission, lock/status/smoke drift, or repair-helper action -> stop before apply.
• Unknown CLI arguments or an inexact confirmation tuple -> explicit_confirmation_required or the allowlisted unsupported-arguments error; default behavior remains read-only.
• Transaction-time employee/token/role/permission drift -> rollback with zero writes; do not broaden selection or modify roles.
• Apply channel/result uncertain -> run --verify only and never apply a second password.
• Post-write role/permission mismatch -> do not log in; permission repair requires a separate production-write decision.
• Login or forced password change fails -> zero retry and no second apply; preserve the known credential state for a new decision.

5. Good/Base/Bad Cases

• Good: helper dry-run has no actions, preflight finds one exact employee-only smoke target, audit baseline increases by one after one apply, post-write permission snapshots agree, one login/forced change succeeds, and sensitive material is destroyed.
• Base: target-scoped session or challenge update counts are zero because no active rows existed; exact where.employee_id scoping is still asserted.
• Bad: use the repair helper because its name says repair, accept an employee role as merely present among extra roles, reuse the preflight permission result after apply, or select an arbitrary non-admin employee when target discovery is ambiguous.

6. Tests Required

• Unit tests cover zero/multiple targets; non-smoke, identifier, status, employment, archive, lock and must-change rejection; extra/missing roles; each forbidden permission; transaction-time token/role drift; confirmation parsing; apply write keys; target-scoped sessions/challenges; and both verify modes.
• Regression tests freeze the repair helper's actual contract: it can restore lifecycle state, clear a login lock and add missing roles, but does not reset a password, delete extra roles, or revoke sessions/challenges.
• Source review requires one employee-update block, no role write path, Serializable isolation, role-row locking, and target-scoped authentication writes.
• Production evidence requires runtime-script SHA match, applyExecutionCount=1, audit delta exactly one, exact-role and four-permission booleans at every checkpoint, login/change counts 1/1, zero non-authentication mutation, and cleanup booleans.

7. Wrong vs Correct

Wrong

repair helper default target looks familiar -> apply helper
roles include employee -> reset password -> assume permissions stayed safe
apply output missing -> retry with another temporary password

Correct

helper dry-run -> actions=[] and safe target
-> employee reset preflight -> exact roles + permission booleans
-> apply once -> verify + independent read-only permission snapshot
-> login/change once -> final verify -> destroy sensitive material

Scenario: Zero-Retry Production Authenticated Readiness

1. Scope / Trigger

• Trigger: a separately approved production-readiness task uses one real login attempt to inspect an already deployed feature without business writes.
• Purpose: preserve the only authentication attempt and produce a precise safe result when production data does not contain the samples needed by the UI assertions.

2. Signatures

• Browser execution contract: workers=1, retries=0, one login submission per approved role.
• Product status-label source: features/contractsRegistry/presentation.ts (or the current shared mapping imported by the production page under test).
• Required evidence fields: stage, loginAttempts, loginHttpStatus, sampleCategory, sampleCount, businessMutationCount, and the relevant endpoint-specific request count such as terminatePostCount.
• A finance smoke window additionally records authorizationRef, accountAlias, observedRoleCodes, disableReason, disableAuditRef and postDisableStatus. Use aliases and redacted references, never credentials or full identifiers.

3. Contracts

• Define fine-grained safe stages before execution, including authentication, route reachability, list loading, sample selection, detail loading and each optional state-specific assertion. A single catch-all "browser failed" stage is not acceptable evidence for a zero-retry run.
• UI labels and state words must come from the current product source or shared presentation mapping. Do not retype remembered Chinese labels in the runner; a stale label can consume the only login while testing an assertion that is not part of the deployed product contract.
• Query or observe the available sample categories without creating data. Report each missing category and count independently. Missing active, archived or terminated data makes only its dependent assertions PAUSED; it is not an authentication failure and does not authorize a retry.
• Install network mutation counters before navigation. A stopped run may claim safety only from explicit counts such as terminatePostCount=0 and businessMutationCount=0, not from the fact that a detail page was never reached.
• Record authentication side effects separately from prohibited business writes. A successful login normally creates a session and login audit; do not summarize that run as "production writes = 0". If logout is not reached, record the residual session and follow the approved session-lifecycle policy rather than revoking it directly without authority.
• Recheck any state-based rollback window before and after the browser run. Missing samples and a still-open window remain independent facts.

Finance-role acceptance checklist

1. Resolve exactly one configured OA-only finance smoke in the isolated test department. Before login, read its status, employment_status, archived_at, last_working_date, lock state and effective role grants. Require exactly employee + finance, with no extra roles or wildcard grant. The database uses status=archived for disabled accounts; employees have no left_at field. A disabled account is a blocked acceptance prerequisite, not permission to weaken the login guard.
2. Obtain explicit authorization for the complete enable -> acceptance -> disable window before any account write. Record the operator alias, purpose, deadline and abort/cleanup responsibility. Re-read status and role grants after enabling. Neither an earlier window nor an unexpectedly active account grants permission for a new window or permanent activation.
3. Run the approved checks as that finance actor. Administrator results prove only administrator behavior: wildcard privileges cannot establish finance visibility, projections or action boundaries. A different read-only actor likewise cannot replace exact finance-role acceptance. Stop at the precise failed stage; do not replace the actor and mark the finance criterion PASS.
4. Immediately disable the smoke through the approved account-management path when checks finish or abort. Read back status=archived, non-null archived_at and target session revocation, and link the disable audit event. A failed cleanup or missing verification leaves the window unclosed and must be handed off explicitly; it is not a successful closeout. Do not submit an extra login merely to prove that disabling worked.

• Record a concrete disable reason, such as finance_acceptance_complete or finance_acceptance_aborted, linked to the authorization and disable event. Current server/src/routes/users.js writes SYSTEM_USER_DISABLED with extra: {}: audit-native reason persistence is not implemented. Until a separately approved code change adds it, retain the reason in controlled acceptance evidence linked to the audit reference, operator alias and event time. Do not claim the audit row contains a reason, invent historical reasons, or directly patch audit rows to satisfy this checklist.
• Do not treat a permanently active read-only finance account as an account-only shortcut. Current grants are fixed by RoleCode/EmployeeRole and resolvePermissionsForRoles, not arbitrary per-account permission strings; there is no finance-read-only role, and employee itself includes writes. Both payment candidates and combination readiness require payment.batch.manage; expense.view_all alone cannot cover them. A future read/write permission split needs separate approval and projection/denial verification. Read-only access still exposes confidential financial data and is not a zero-risk standing identity.

4. Validation & Error Matrix

• Login fails once -> stop with the authentication stage and re-read lock/failure state; no second login.
• Finance smoke is disabled -> record the prerequisite as PAUSED(finance_account_disabled); finance-role acceptance remains unverified. If login was attempted, distinguish 403/AUTH_003 at /api/auth/login from 403/PERM_001 at a later protected endpoint; never infer the layer from 403 alone.
• Disable reason, linked audit reference or post-disable state proof is missing -> acceptance-window closeout is incomplete, even when all UI checks passed.
• Login succeeds but the required sample count is zero -> authentication PASS, sample-dependent UI checks PAUSED, mutation counters must remain zero.
• Runner label differs from the current shared product mapping -> runner contract defect; stop without relabeling the product and do not rerun the consumed login solely to repair evidence.
• Mutation counter is missing or positive -> no safe read-only claim; classify the exact endpoint/stage and stop.
• Browser exits before logout after a successful login -> record the new session as an authentication side effect; do not conceal it or revoke it by an unapproved database write.
• Rollback window closes while the runner sent no relevant mutation -> record a concurrent real-user event and switch to the approved roll-forward policy.

5. Good/Base/Bad Cases

• Good: one login returns 200, the contract table is empty, the runner reports sampleCategory=contracts, sampleCount=0, all mutation counters are zero, and detail/termination assertions remain explicitly PAUSED.
• Base: suitable existing samples are present, so the runner checks only the approved read-only UI stages and exits with zero business mutations.
• Bad: hard-code a remembered label, fail at list selection, then report only "browser failed" and spend a second login to discover that production had no sample.
• Bad: say "no production writes" after a successful login created a session and audit row.

6. Tests Required

• Before a live run, statically compare every runner-owned status label with the current shared presentation mapping, or import that mapping directly.
• Exercise a local zero-sample fixture and assert it emits the independent safe category/count, preserves authentication PASS and marks only dependent UI checks PAUSED.
• Assert workers=1, retries=0, login submission count at most one per role, and mutation listeners installed before the first application navigation.
• Evidence review must reconcile login attempts with session/audit changes and reconcile every unexecuted UI assertion with its sample count and stage.
• For a finance window, verify the unique actor and exact grants before login, and the reason/audit linkage, disabled state and session revocation afterward. Administrator or alternative-role checks must not satisfy finance-role criteria.
• Production execution still requires explicit approval, pre/post rollback window snapshots, sensitive-content scanning and a read-only reviewer.

7. Wrong vs Correct

Wrong

remembered UI label -> one broad browser stage -> sample lookup fails
-> retry login -> report "no production writes"

Correct

current shared mapping -> fine-grained stages + mutation listeners
-> login once -> classify sample categories/counts
-> zero sample means dependent checks PAUSED, not auth failure
-> reconcile session/audit side effects -> post-window snapshot

Scenario: Read-Only Automatic Backup Chain Investigation

1. Scope / Trigger

• Trigger: the latest off-host archive exceeds its RPO/freshness gate, an automatic-backup or dead-man service fails, or a restore drill discovers a stale latest batch.
• Purpose: distinguish scheduler failure, source production failure, interrupted transfer, finalize failure and alert-delivery behavior without triggering a backup, changing a timer, reading business contents or modifying an archive.

2. Signatures

• Production units: hyb-oa-automatic-backup-daily.service/.timer and hyb-oa-automatic-backup-watchdog.service/.timer.
• ARM dead-man unit basename is the installed DEADMAN_SERVICE contract from deploy/trialrun/install-offhost-backup-receiver.sh; probes must derive or mechanically compare that value instead of typing a remembered unit name.
• Safe state fields: unit ActiveState, Result, ExecMainStatus, timer last trigger, effective max-age seconds, summary status/category/findingCount, alert state status/category, journal evaluated sent=true|false counts, archive/partial counts and age aggregates, and filesystem percent/free GiB.

3. Contracts

• A timer being active proves scheduling remains enabled; it does not prove its last service run passed. Always pair timer state/trigger with service Result and exit status.
• Discover the remote account's actual capabilities before installing a probe. The least-privilege production deployment account is not assumed to have sudo; a production probe must run unprivileged or stop with an explicit authority requirement. It must never attempt to widen sudoers itself.
• File absence and access denial are different evidence. Before reporting a root-owned state file as missing, prove the parent directory is traversable and readable; otherwise emit ACCESS_DENIED and keep the fact unknown.
• Classify receiver inbox objects without outputting names or paths. A partial with the exact three regular files, valid manifest/run identity and matching artifact byte counts indicates transfer completion followed by missing or failed finalize. A partial that fails that contract indicates transfer did not complete; SHA-256 may remain uncomputed in this diagnostic classification.
• A receiver partial proves that the transfer authenticated and opened its destination before interruption. Initial private-key rejection, authorized_keys rejection and host-key refusal cannot create that object; keep investigation on mid-transfer network/session limits, source read, receiver write and transport I/O. A later SSH-layer disconnect may still surface as rsync/SSH failure and is not excluded by successful startup.
• For multiple incomplete partials, compare only aggregate byte totals/ratios, the wrapped rsync exit code and elapsed transfer seconds. Similar stopping offsets or durations indicate a deterministic limit; materially different offsets/durations favor intermittent timeout or network interruption. Do not infer this split from service exit 1, which is the wrapper's generic code.
• A stopping-offset pair is similar only when its absolute difference is no more than both 50000 ppm (5% of the larger aggregate) and 67108864 bytes (64 MiB), equivalently difference <= min(relative_limit, absolute_limit). Using max(...) is forbidden: it misclassifies a small nonzero partial and a zero-byte partial as similar merely because both are below 64 MiB.
• last-deadman.json proves the detector decision. deadman-alert-state.json proves the deduplication state. A journal allowlist line with evaluated sent=true proves the helper attempted and accepted one send; it does not prove a human noticed the message. Repeated sent=false for the same failure is normal deduplication, not a monitoring blind spot.
• Keep the two JSON schemas distinct: the dead-man summary contains status/category/findingCount/findings, while the alert deduplication state contains only schemaVersion/status/category. A probe must not request findingCount from the alert state or report its contractual absence as an invalid remote state.
• Read-only investigation output is an allowlist of status/category/count/age and capacity fields. Never print unit command lines, raw JSON, raw journal, remote identities, paths, run ids, webhook data or diagnostic stderr.
• Before classifying an off-host transport failure, validate the newest source batch with the same manifest schema/status, exact three-file set, run-id pairing, byte-count and SHA-256 contract used by the receiver. Report only whether a complete local recovery point exists, its age/count/aggregate bytes and one of production_single_point or no_proven_recovery_point. Off-host staleness alone does not distinguish those two materially different production risks.
• On an online production host, enumerate older local batches with manifest, file-set, identity and byte metadata only. Fully hash exactly the newest metadata-valid candidate; report older hashes as NOT_RUN. If that latest candidate fails SHA-256, keep no_proven_recovery_point and do not fall back to multi-GiB reads of older candidates during the same probe. ARM archive validation remains full-contract because it is the backup node.
• Inspect cleanup semantics as executable ordering, not a remembered claim. Source publication must precede transfer; local pruning must follow verified off-host finalize and success-state publication; generation staging must be protected by an exit cleanup. Also report current valid/invalid/staging counts so repeated transfer failures cannot silently consume source space.
• A retained last-error.log may provide an rsync code only when the matching safe run summary says phase_offhost_transfer_failed. Never parse a log from another phase or substitute service exit 1. If the wrapper did not persist transfer start/end clocks, emit NOT_AVAILABLE; service duration, mtime and conversation time are not transfer-duration evidence.
• A read-only inspector must prevalidate every output as a fixed key plus an enum/integer value. Malformed manifests, unreadable files and unexpected runtime exceptions collapse to stable categories without a traceback, because tracebacks can expose absolute paths even when normal output is redacted.
• If existing read-only state cannot converge on a cause, stop expanding the read surface. The next step is a separately approved, single-attempt observational retransmit with frozen input, fixed telemetry, bounded timeout and no automatic retry. It is a diagnostic write, not part of read-only authority.
• After a transport repair, one fresh atomic batch may unlock a blocked restore drill, but incident closeout requires two consecutive natural schedules. Each cycle records source and receiver capacity plus receiver partial count and aggregate bytes; a manual run does not replace either natural sample.

4. Validation & Error Matrix

• Timer inactive or never triggered -> scheduler/configuration branch.
• Timer triggered and service failed before a new local batch is visible -> source/common-preflight branch; if root state is inaccessible, preserve the exact phase as unknown instead of guessing.
• New incomplete receiver partial -> offhost_transfer_incomplete; do not call it a finalize failure.
• Partial exists -> initial SSH authentication/host-key acceptance succeeded; do not spend the first incident cycle rotating keys or rewriting authorized_keys without contrary evidence.
• Receiver partial satisfies the complete byte/manifest contract but no atomic archive exists -> offhost_finalize_not_completed.
• Atomic archive stale + dead-man state stale + sent=true -> detector and send invocation worked; investigate human/channel reception separately.
• Atomic archive stale but dead-man state fresh/missing, service not run, or timer failed -> monitoring execution blind spot.
• Path cannot be traversed by the approved account -> ACCESS_DENIED; request separate read authority only if the unresolved field materially changes the decision.
• Complete source batch valid + off-host point stale -> production has a local recovery point but remains a single point; prefer the next safe natural run unless a separately approved transition run is justified.
• No source batch passes the complete contract -> no_proven_recovery_point; raise the transition-recovery-point priority, but do not trigger a backup until root cause, capacity and stop conditions are approved.
• last-error.log contains rsync-like text but the safe last-run phase is not offhost_transfer -> exit code remains NOT_AVAILABLE rather than attaching stale diagnostics to the current failure.

5. Good/Base/Bad Cases

• Good: daily timer triggered, service failed, ARM has a recent incomplete partial, dead-man records archive_stale and the alert helper records one sent=true; conclude transfer interruption plus successful detector/send invocation while keeping the production raw phase unknown.
• Base: latest archive is fresh, services pass, inbox has zero partials and the alert state is healthy.
• Bad: infer “timer active means backup succeeded”, call an inaccessible state file missing, or describe any .partial as finalize failure without checking whether its three-file byte/manifest contract completed.
• Bad: hard-code a near-match dead-man unit name in a one-off probe and treat an empty systemctl show response as a parser-format problem.

6. Tests Required

• Extract the receiver dead-man unit basename from the installer and compare it with every rendered probe before upload; a near-match unit must fail locally.
• Fixture systemd output for active timer/pass service, active timer/failed service, nonexistent unit and explicit max-age versus script-default max-age.
• Fixture root-owned state directories and assert an unprivileged probe emits ACCESS_DENIED, never present=no.
• Fixture incomplete partial, complete-contract partial and archive transaction temporary entries; assert only aggregate classifications and ages are emitted.
• Fixture two partials with equal and unequal aggregate stopping offsets plus rsync codes 23/30/12/11/255; assert the incident summary preserves code and elapsed-time evidence separately from wrapper exit 1 and does not output file names.
• Assert 2.68 MB versus zero is different, a large pair within both 5% and 64 MiB is similar, and exceeding either threshold is different.
• Fixture one complete source batch, no complete source batch, checksum/size corruption and malformed manifest. Assert the risk classification and transition recommendation change without emitting paths, run ids, artifact names, raw diagnostics or tracebacks.
• Fixture two metadata-valid production batches with a same-size hash mutation in the older and then newest candidate. Assert only the newest candidate is fully hashed, older hashes remain NOT_RUN, and a newest-candidate hash failure does not trigger fallback reads.
• Mechanically mutate the wrapper ordering so pruning appears before transfer; the cleanup/retention contract must turn false. A prose claim that failure retains data is not sufficient regression evidence.
• Sensitive scan the probe and assert raw ExecStart, journal, JSON, paths, remote identities and run ids cannot appear in stdout.
• Every root probe must self-remove its fixed entry/sudoers and an independent non-root wrapper must report entry and staging residue separately.

7. Wrong vs Correct

Wrong

timer active -> backup healthy
state path test returned false -> state missing
partial count > 0 -> finalize failed
off-host stale -> assume one complete backup still exists on production
service exit 1 -> call it the rsync exit code and use service duration
partial difference <= max(5% limit, 64 MiB) -> similar

Correct

derive unit identity from installer -> timer trigger + service Result/exit
-> distinguish ACCESS_DENIED from missing
-> partial three-file/manifest/byte classification
-> partial proves initial SSH acceptance; compare byte offset + rsync code + duration
-> dead-man decision + dedup state + sent flag
-> safe scheduler/source/transfer/finalize/alert conclusion
-> validate the source batch first and classify single-point vs no proven point
-> bind retained diagnostics to the safe failure phase; unavailable clocks stay unavailable
-> partial difference <= min(5% limit, 64 MiB) before calling it similar

G19 layered browser acceptance (approved 2026-09-18)

• verify-full-local-discovery.mjs still discovers every non-manual source spec. full-local-coverage-contract.json fixes 12 post-deployment obligations and two retained frozen travel chains by file/case identity and source digest; do not edit the old sources to hide failure.
• full-local-coverage.mjs plan derives local files/cases from actual full discovery. Missing sources/protections, duplicate identities, changed exception source or an unregistered case in an exception file fail before business execution. Other discovered cases are locally mandatory.
• The ordinary runner uses exact derived file regexes and zero retries/default concurrency; callers cannot narrow it. verify-local checks source identity and every actual case, rejecting missing/unknown/duplicate/skipped/retried/expected-failure results even when Playwright returns zero. Failure receipts remain on disk.
• Local browser PASS always carries post-deploy NOT_RUN_LOCAL, delivery INCOMPLETE and no release eligibility. Frozen protection PASS means the new boundary assertions passed, never that the old travel chain ran.
• The offline post-deploy validator needs live profile, the same source candidate/deployed candidate, and complete original Playwright results for all public obligations. It does not establish remote identity independently, grant release authority or run network calls. R1 stage/source/environment enforcement remains a separate open task.
• Focused diagnosis stays focused. Keep original failed ordinary runs; later focused passes do not rewrite them. Actual owners and public evidence remain pending until supplied through authorized work.
• Travel is retained/frozen, not retired: current employee/finance mutation permissions and shared payment blocker are protected without restoring access. The managed synthetic runner enables travel sync for existing low-level regressions; do not claim all integration APIs are disabled or that finance managers lack integration.manage.