hyb-oa/.trellis/spec/hyb-oa/deployment-trialrun.md

Deployment / Trial Run

Public OA Boundary

Public OA is quasi-production. Deployments, remote config changes and live data operations require explicit approval and evidence.

Rules

• Existing installations use DEPLOY_SERVER=<ssh-target> bash deploy/trialrun/deploy-existing.sh only when deployment is approved; the wrapper reads the frozen allowlisted contract and keeps SYNC_ENV=0. First initialization or controlled recovery without that unit uses deploy.sh with explicit absolute data paths and DEPLOY_DATA_MODE; neither path overwrites remote .env by default.
• When server/src/config.js reads a new environment key, add the same key to deploy/trialrun/.env.example before deployment; deploy/trialrun/deploy.sh runs node deploy/trialrun/validate-env-contract.js at step 0 and must fail if the example contract drifts.
• After deployment, report commit hash, /ready, public smoke results and whether the user can test.
• Real business writes are manual unless the current Trellis task explicitly authorizes a controlled test write.
• Do not execute real payments, import bank statements, connect payment providers or write ERP without a separate approved Trellis task.

Scenario: Deployment Env Contract

1. Scope / Trigger

• Trigger: any backend config, deployment compose, or trialrun env wiring change.
• Purpose: prevent deploy-time surprises where local code compiles but the deployment contract rejects the release or silently misses a runtime default.

2. Signatures

• Command: node deploy/trialrun/validate-env-contract.js
• Deployment entry requires SYNC_ENV=0, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR and DEPLOY_DATA_MODE=existing|initialize.
• Contract sources:
  • server/src/config.js
  • deploy/trialrun/.env.example
  • deploy/trialrun/docker-compose.yml

3. Contracts

• Every process.env.KEY or requiredEnv("KEY") in server/src/config.js must be present in deploy/trialrun/.env.example.
• server, worker, and init services must load ./.env through env_file.
• Compose service-level environment overrides are limited to the explicit allowlist in validate-env-contract.js.
• SYNC_ENV=0 is the default public deploy mode; it does not overwrite remote .env.
• Relative data-path defaults are allowed only for direct local Compose. deploy.sh requires explicit absolute host paths and a data mode; existing verifies the PGDATA directory contains PG_VERSION, while initialize is the only mode allowed to create missing directories.

4. Validation & Error Matrix

• Missing key in .env.example -> validate-env-contract.js exits non-zero.
• Service omits env_file: ./.env -> contract exits non-zero.
• Unauthorized compose override -> contract exits non-zero.
• Remote .env missing -> deploy step 4 exits before migration.
• Missing/relative/unsafe/overlapping data paths or missing data mode -> deploy exits locally before provenance, synchronization or SSH.
• existing target missing PG_VERSION or storage directory -> remote deploy exits before Compose starts DB.

5. Good/Base/Bad Cases

• Good: add EMAIL_LOGIN_CODE_TTL_SECONDS to config.js, document it in .env.example, then run contract validation before deploy.
• Base: code-only feature without env changes still runs the contract gate.
• Bad: add a config key only in config.js; deploy fails at step 0.

6. Tests Required

• Always run node deploy/trialrun/validate-env-contract.js after env-related changes and before public deployment.
• For public deploys, also run or verify /ready, runtime smoke, external deps smoke, public bundle copy scan, and live smoke when credentials exist.

7. Wrong vs Correct

Wrong

# Config key exists only in server/src/config.js.
SYNC_ENV=0 bash deploy/trialrun/deploy.sh

Correct

# Add the key to deploy/trialrun/.env.example first.
node deploy/trialrun/validate-env-contract.js
SYNC_ENV=0 POSTGRES_DATA_DIR=/srv/hyb-oa/pgdata \
  STORAGE_HOST_DIR=/srv/hyb-oa/storage DEPLOY_DATA_MODE=existing \
  bash deploy/trialrun/deploy.sh

Scenario: Public Smoke Navigation Labels

1. Scope / Trigger

• Trigger: frontend shell, navigation labels, sidebar grouping, workbench copy, or live smoke assertions change.
• Purpose: prevent public deployment from being marked failed only because smoke tests look for stale navigation copy while the page itself is healthy.

2. Signatures

• Live smoke command: E2E_MODE=live LIVE_SMOKE_BASE_URL=https://oa.hyinda.com npx playwright test tests/e2e/live-smoke.spec.ts
• Shared assertions: client/tests/e2e/helpers/smokeAssertions.ts
• Workbench route: /portal
• Current primary workbench nav label: 工作台
• Stale user-facing label that must not be accepted by current mocks or smoke: Portal

3. Contracts

• assertPortalLoaded(page) must assert route and page content first: /portal URL and the 今日待处理 heading.
• Navigation-label assertions for the workbench link must require the exact current label 工作台; mocks and fixtures must use the same contract.
• Navigation assertions must be role-aware. Admin/manager smoke may assert team/admin entries such as 团队, but ordinary employee smoke must not require links that NavLinks intentionally hides for that role.
• Smoke tests must not require a copy string that is not a product contract for public availability.
• If the product intentionally renames the primary workbench label again, update the product contract, fixtures and shared smoke helper together rather than adding an ad hoc fallback in one live smoke test.

4. Validation & Error Matrix

• Page reaches /portal, 今日待处理 is visible, but nav label is 工作台 -> smoke should pass.
• Page reaches /portal and renders Portal as the user-facing nav label -> smoke should fail as a stale-copy regression.
• Employee reaches /portal, role-aware navigation hides 团队, and employee pages such as 项目 remain reachable -> smoke should pass.
• Page reaches /portal, but 今日待处理 is missing -> smoke should fail.
• Page stays on login, /403, error fallback or another route -> smoke should fail.
• New shell label appears only in one live smoke test -> stop and align the product contract, fixtures and shared helper before changing smoke.

5. Good/Base/Bad Cases

• Good: assertPortalLoaded checks /portal, 今日待处理, and an exact 工作台 workbench link.
• Good: assertMainNavVisible(page, { expectTeam: false }) is used for ordinary employee live smoke.
• Base: the internal route and helper name may retain portal, while all user-facing navigation fixtures and assertions use 工作台.
• Bad: a mock emits Portal and smoke accepts it as a valid current label, masking a user-facing English-copy regression.
• Bad: employee live smoke requires the 团队 sidebar link after role-aware navigation intentionally hides it.

6. Tests Required

• After changing live smoke assertions, run: cd client && npm run typecheck
• Load the focused visual and shell specs with Playwright --list when changing fixture/report contracts.
• For public deploy verification, run live smoke or the readiness wrapper: LIVE_SMOKE_BASE_URL=https://oa.hyinda.com bash scripts/qa/run-live-smoke-readiness.sh
• When changing the shell or Portal page, include a focused Playwright check that proves the workbench route and heading still load.

7. Wrong vs Correct

Wrong

await expect(
  page.getByRole("link", { name: "工作台", exact: true })
    .or(page.getByRole("link", { name: "Portal", exact: true }))
    .first(),
).toBeVisible();

Correct

await expect(page.getByRole("link", { name: "工作台", exact: true }).first()).toBeVisible();

Scenario: Trial Run Runtime Navigation Baseline

1. Scope / Trigger

• Trigger: changing sidebar grouping, workspace module visibility defaults, server/src/services/workspaceSettings.js, or server/src/scripts/smoke-trialrun-runtime.js.
• Purpose: keep deployment smoke aligned with the current small-team navigation model instead of blocking deployment on retired menu assumptions.

2. Signatures

• Runtime smoke command inside deployment: docker compose exec -T server node src/scripts/smoke-trialrun-runtime.js
• Source of truth for sidebar grouping: server/src/services/workspaceSettings.js
• Runtime smoke assertion file: server/src/scripts/smoke-trialrun-runtime.js

3. Contracts

• Runtime smoke must distinguish primary sidebar entries from collapsed 更多 entries.
• Core day-to-day ledgers such as 报销, 采购, 合同, 项目 and 资产 are expected in main when the active role has access.
• Low-frequency entries such as 周报 and 公告 may be enabled but should be checked in backup / 更多, not forced into main.
• 待办 follows the current workspace and role-aware navigation baseline; do not move it between main, backup or hidden as part of a deployment-smoke fix unless a product task explicitly changes that rule.
• Management pages should remain represented by the single 管理 entry unless a current task explicitly changes the management navigation model.

4. Validation & Error Matrix

• Core ledger missing from main -> runtime smoke fails with 系统导航 main 缺少 <key> 入口.
• 周报 or 公告 missing from backup while enabled -> runtime smoke fails with 系统导航 backup 缺少 <key> 入口.
• Management children are hidden behind 管理, but smoke still expects workspace, people, featureFlags or auditLogs in admin -> update the smoke baseline; do not re-expand the sidebar for the smoke.
• systemSettings missing from admin -> runtime smoke fails because the management entry is no longer reachable.

5. Good/Base/Bad Cases

• Good: main includes portal, reimbursements, purchaseRequests, contractsRegistry, projects and assets; backup includes weeklyReports and announcements; admin includes systemSettings.
• Base: 待办 is in main for a configured admin/owner trial account; smoke accepts it there because it follows the current workspace baseline.
• Bad: smoke expects weeklyReports in main after the sidebar rule moved low-frequency modules to 更多.
• Bad: smoke expects every management child as a sidebar row after the product rule consolidated management into one 管理 entry.

6. Tests Required

• Run node --check server/src/scripts/smoke-trialrun-runtime.js after editing runtime smoke.
• Run SYNC_ENV=0 bash deploy/trialrun/deploy.sh for public deployment so the remote runtime smoke validates the actual workspace and feature-flag state.

7. Wrong vs Correct

Wrong

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "weeklyReports",
  "announcements",
];
const REQUIRED_ADMIN_NAV_KEYS = ["workspace", "people", "featureFlags"];

Correct

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "purchaseRequests",
  "contractsRegistry",
  "projects",
  "assets",
];
const REQUIRED_BACKUP_NAV_KEYS = ["weeklyReports", "announcements"];
const REQUIRED_ADMIN_NAV_KEYS = ["systemSettings"];

Scenario: Immutable Deployment Provenance And Dual Build SHA

1. Scope / Trigger

• Trigger: changing the trialrun deployment entry, Docker build, Compose wiring, /health or /ready, public bundle scanning or live readiness.

2. Signatures

• Provenance gate: node deploy/trialrun/check-source-provenance.mjs.
• Standard release additionally supplies the explicit absolute data-path and data-mode contract required by deploy.sh.
• Build/runtime inputs: BUILD_SHA=<full-commit> and VITE_BUILD_SHA=<12-char-commit>.
• Runtime outputs: /health.data.buildSha, /ready.data.buildSha and the frontend html[data-build-sha] marker.

3. Contracts

• Deployment rejects tracked and untracked dirty files before synchronization.
• By default HEAD must be contained in its upstream. The explicit unpublished override is for controlled offline recovery and never relaxes clean-tree.
• Standard deployment refreshes the configured upstream remote before the ancestor check; a stale local remote-tracking ref is not publication proof.
• Unpublished override requires an allowlisted DEPLOY_UNPUBLISHED_REASON and must emit an UNPUBLISHED-OVERRIDE audit line.
• Synchronization comes from git archive <verified-commit>, not the working tree.
• Full BUILD_SHA is injected at build and runtime; VITE_BUILD_SHA is normalized before bundling so only 12 characters become public.
• Neither build key is persisted in deploy/trialrun/.env; direct Compose use must explicitly provide both and must not publish unknown.
• Deployment compares public /ready and the public bundle marker with the verified commit before reporting success.

4. Validation & Error Matrix

• Dirty tree -> dirty_worktree, deployment stops.
• No upstream or commit absent from upstream -> stop unless the explicit unpublished override is approved.
• Upstream refresh fails -> stop in standard mode; override mode requires an allowlisted reason and records remoteVerified=false.
• Missing runtime/build SHA input -> Compose interpolation fails.
• /ready SHA mismatch, bundle marker missing or full SHA exposed -> deployment fails.
• Standalone readiness missing/mismatching observed SHA -> non-clean finding.

5. Good/Base/Bad Cases

• Good: clean published commit -> Git archive -> dual-SHA build -> matching backend and public bundle checks.
• Base: local development build uses unknown but is never represented as a release.
• Bad: rsync the current working tree or let .env override the image's SHA with unknown.

6. Tests Required

• Temp-repo provenance tests for dirty, missing upstream, published and unpublished-override cases, including a stale remote-tracking ref after a forced remote update.
• node deploy/trialrun/validate-env-contract.js and Compose config with both SHA inputs.
• Synthetic 40-character build: bundle contains the matching 12-character SHA and not the full value.
• Local-stub readiness test for expected/observed SHA propagation.
• Public deployment still requires separate approval and post-deploy checks.

7. Wrong vs Correct

Wrong

rsync -az --delete ./ remote:/app
docker compose up -d --build server

Correct

node deploy/trialrun/check-source-provenance.mjs
BUILD_SHA=<verified-full-sha> VITE_BUILD_SHA=<matching-short-sha> \
  docker compose -f deploy/trialrun/docker-compose.yml build server

Scenario: Isolated Authentication Security Gate

1. Scope / Trigger

• Trigger: changing authentication challenges, credential/session transactions, trusted proxy handling or the deployment order for their schema migration.
• The gate proves local PostgreSQL behavior only; public deployment and edge changes remain separately approved operations.

2. Signatures

• Complete local gate: bash scripts/qa/run-web-gates.sh auth-security-local.
• Database allowlist: loopback PostgreSQL and a fresh hyb_oa_auth_* database whose exact name is confirmed to the checker.
• Runtime guard: RUN_AUTH_SECURITY_REGRESSION=1; concurrency mode accepts only 0 or 1 and reports FULL only for 1.
• Public release/rollback gate: four challenge endpoints marked EXTERNAL_GATED in the runbook and checklist.

3. Contracts

• Validate loopback API, loopback PostgreSQL, isolated database name, explicit confirmation and test mode before the first HTTP write.
• The authentication gate generates per-run JWT, pepper and field-encryption test keys; it must not inherit production values. Email, IM, WeCom and OIDC side effects are disabled or injected with a no-network test service.
• PID/log paths are unique per run. Cleanup may stop only the server spawned by the current run, and a database drop failure makes the gate fail.
• The runner removes only a database that this invocation successfully created; failed creation must never authorize deletion of an existing database.
• Deployment or rollback to old challenge code requires an operator to close /api/auth/forgot-password, /api/auth/reset-password, /api/auth/email-login/code and /api/auth/email-login/verify. Scripts do not claim this external action occurred.

4. Validation & Error Matrix

• Remote API/DB, wrong database prefix/confirmation or production mode -> fail before login, seed-user mutation or challenge requests.
• Code-display logging, enabled external adapter or inherited proxy hop -> fail the test-server guard.
• Concurrency value other than 0|1 -> fail; 0 -> BASE_ONLY/PARTIAL; 1 plus all PostgreSQL scenarios -> FULL.
• Owned server cleanup failure or temporary database drop failure -> non-zero gate result.
• Four-endpoint edge confirmation absent -> public release/rollback remains EXTERNAL_GATED, regardless of local test success.

5. Good/Base/Bad Cases

• Good: disposable loopback PostgreSQL, synthetic secrets, no-network auth email seam, five concurrency scenarios, zero residual databases and an automatically removed container.
• Base: the HTTP regression runs without PostgreSQL concurrency and reports partial evidence rather than a complete security pass.
• Bad: run several mutating HTTP steps and only then discover that the target database or API is remote.
• Bad: reuse a shared PID file or ignore dropdb failure while returning exit code zero.

6. Tests Required

• Static guard tests for remote authority, query-host override, exact database confirmation, secret isolation, external-adapter disablement, owned cleanup and redacted failure output.
• bash -n for the runner and regression scripts; Node syntax checks for the checker and test-server entry.
• Prisma validate, clean migration deployment, the full isolated gate and an explicit post-run query proving no hyb_oa_auth_* database remains.
• node deploy/trialrun/validate-env-contract.js must enforce loopback API binding, XFF overwrite templates and four-endpoint EXTERNAL_GATED wording.

7. Wrong vs Correct

Wrong

run_mutating_auth_regression
validate_target_afterwards
dropdb --if-exists "$DB" || true

Correct

validate_isolated_target_before_writes
run_mutating_auth_regression
drop_owned_database_or_fail

Scenario: Repository Local Candidate Gate

1. Scope / Trigger

• Trigger: declaring a checkout ready for internal deployment planning without deploying it or using public, production or real-account evidence.

2. Signatures

• Full gate: bash scripts/qa/run-local-candidate.sh full.
• Browser-only rerun: bash scripts/qa/run-local-candidate.sh focused.
• Parse-only check: bash scripts/qa/run-local-candidate.sh list.
• Route contract: scripts/qa/route-manifest.json.
• Result verifier: scripts/qa/verify-local-candidate-results.mjs.

3. Contracts

• full alone may print LOCAL_CANDIDATE_PASS; focused prints FOCUSED_BROWSER_PASS, and list prints LIST_ONLY.
• The gate accepts only loopback HTTP origins and canonical TCP ports. It uses local synthetic fixtures, excludes manual/live specs and does not consume historical Issue CSV as a candidate prerequisite.
• Every registered route is classified. Every core bundle has one stable, source-bound witness; candidate spec basenames are unique.
• Every selected Playwright result must pass. Missing, duplicate, unknown, skipped, multi-tag or wrong-source witnesses and any failed/skipped acceptance test make the gate fail.
• The script's full pass proves only its local scope. A task changing payment concurrency also requires the isolated PostgreSQL payment gate; deployment, public checks and real-device behavior remain separate states.

4. Validation & Error Matrix

• Non-loopback URL, credentialed URL, path/query target or invalid port -> fail before Vite/Playwright use.
• Unclassified/orphan route or missing core behavior/visual witness -> non-zero scanner result.
• Playwright suite error, unexpected, failed or skipped test -> non-zero result gate with basename-only redacted diagnostics.
• focused succeeds -> useful rerun evidence, but not a full candidate claim.

5. Good/Base/Bad Cases

• Good: full base checks pass, all source-bound witnesses and acceptance specs execute, and the runner prints one full-pass label.
• Base: use list before a long run to prove assets parse, while recording that no behavior has run.
• Bad: reuse a prior browser pass after parser changes, or report focused/list output as LOCAL_CANDIDATE_PASS.

6. Tests Required

• Negative tests for route gaps, manual/live evidence, duplicate basenames, invalid loopback inputs, witness result/source drift and mode labels.
• Current-checkout full run with zero skipped/unexpected results.
• For payment-state changes, run the disposable PostgreSQL gate and prove cleanup left no matching temporary databases.

7. Wrong vs Correct

Wrong

bash scripts/qa/run-local-candidate.sh list
echo LOCAL_CANDIDATE_PASS

Correct

bash scripts/qa/run-local-candidate.sh full
# Only this successful branch emits LOCAL_CANDIDATE_PASS.

Scenario: Compose-Resolved Loopback Host Port

1. Scope / Trigger

• Trigger: changing Compose port publication, deployment health checks, Caddy routing or operator commands for the server container.
• Purpose: support a non-default host port without changing the container's fixed application port or accidentally checking a stale service.

2. Signatures

• Env: APP_HOST_PORT=<1-65535>; default 3000.
• Compose mapping: 127.0.0.1:${APP_HOST_PORT:-3000}:3000.
• Runtime resolution: docker compose port server 3000 or the compatible docker-compose command selected by the deployment script.
• Backend container still listens on PORT=3000.

3. Contracts

• Publication remains loopback-only. APP_HOST_PORT changes only the host side; Caddy points to the same resolved host port.
• Deployment health checks must parse the running Compose mapping after startup; do not hardcode 3000 and do not assume .env variables entered the current shell merely because Compose read --env-file.
• The parsed port must be numeric before it is used in curl. A missing, multi-line or non-numeric mapping stops deployment before readiness success.
• server, worker and init retain the shared env-file contract; only server publishes an HTTP host port.
• Changing the real host port or Caddy target is production configuration and requires separate approval plus rollback evidence.

4. Validation & Error Matrix

• Missing APP_HOST_PORT -> Compose uses loopback port 3000.
• Valid alternate port -> Compose publishes it and deployment checks that port.
• Non-numeric/unresolved compose port output -> deployment exits non-zero.
• Caddy target differs from Compose mapping -> deployment is not accepted even if one local /ready endpoint answers.
• Host port already occupied -> Compose startup fails; do not start a second deployment on a guessed port.

5. Good/Base/Bad Cases

• Good: .env selects an alternate loopback port, Compose reports it, deploy checks the matching build SHA there and Caddy targets the same port.
• Base: default port 3000 remains compatible without extra configuration.
• Bad: Compose publishes 3001 while deploy/runbook curls 3000 and reports an old container as the new release.

6. Tests Required

• bash -n deploy/trialrun/deploy.sh deploy/trialrun/setup-server.sh.
• node deploy/trialrun/validate-env-contract.js must assert the exact loopback-only variable mapping and env example key.
• Review runbook/checklist commands for Compose resolution rather than shell expansion of an unloaded .env.
• Public deployment additionally proves Caddy, /ready and frontend marker use the approved build on the same resolved port.

7. Wrong vs Correct

Wrong

curl -fsS http://127.0.0.1:3000/ready

Correct

SERVER_HOST_PORT="$(docker compose port server 3000)"
SERVER_HOST_PORT="${SERVER_HOST_PORT##*:}"
curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"

Scenario: Compose Project And Data-Path Isolation

1. Scope / Trigger

• Trigger: changing Compose identity, host data mounts, database publication, deployment entrypoints, deployment-time seeds or operational timers.
• Purpose: ensure one approved Compose project owns one verified DB/storage pair without fixed global container names, source synchronization touching runtime data, or deployment seeds overwriting approved production rules.

2. Signatures

• Every deploy/timer Compose command uses -p "$COMPOSE_PROJECT_NAME".
• PostgreSQL mount: ${POSTGRES_DATA_DIR:-./data/postgres}.
• Application storage mount: ${STORAGE_HOST_DIR:-./data/storage}.
• Formal deploy inputs: absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, absolute REMOTE_DIR, absolute REMOTE_ENV_FILE, DEPLOY_DATA_MODE=existing|initialize.
• Deployment init includes npm run seed:config; approval-rule defaults are inserted only when approval_rules is empty.
• Timer install inputs: COMPOSE_PROJECT_NAME, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, full 40–64 hex BUILD_SHA and its exact 12-hex VITE_BUILD_SHA prefix.
• DB has no host ports; server remains loopback-only.

3. Contracts

• Compose files must not set container_name. Operational commands resolve services through the approved project, never a historical global name.
• A project-name change and its DB/storage path selection are one atomic, separately approved downtime operation. Never start two PostgreSQL instances against one PGDATA.
• Freeze the old project name from docker compose ls, not from a directory guess. Record immutable image IDs/digests, successful migration count, fixed business-table counts, resolved mounts and Caddy target before any write.
• Relative defaults preserve local trialrun compatibility, but production must use explicit absolute command inputs and verify them against active container mounts before any start, backup or migration.
• REMOTE_DIR must not equal, contain or be contained by either production data path. Validate this locally before provenance, SSH or rsync --delete. If REMOTE_ENV_FILE is inside the synchronization tree, derive its actual root-relative path and add explicit rsync protect/exclude filters; a fixed default .env exclusion is insufficient.
• Deployment-time seed:config may create the three default approval rules for an empty database. If any approval rule exists, preserve all existing rules; production rule changes require their own approved transaction, drift checks and audit rows.
• existing is the normal migration mode and refuses a missing PGDATA marker or storage directory. initialize is only for an approved empty first start; only that mode may create data directories.
• Remote readiness may use host curl, but build-SHA JSON validation runs with Node inside the server container; the Docker host does not need Node.
• The timer installer must validate the same non-root, non-overlapping absolute data-path shape as formal deploy, require an existing PG_VERSION and storage directory, and write both paths plus both build identifiers into the systemd service Environment= entries. Project name alone is insufficient: a later timer invocation would otherwise interpolate Compose defaults and operate against a non-active data pair.
• Timer installation may enable/start the timer unit, but verification must not manually start the daily service because that path can send real alerts. Verify the unit environment against active container mounts, then run Compose config --quiet and exec -T server true with the same values.
• For an existing cutover, stop the old PostgreSQL completely, copy PGDATA and storage into pre-verified empty absolute targets while preserving ownership, permissions, links and extended attributes, and retain the old directories untouched and unmounted for the observation period. Do not rename away the only pre-migration rollback data and do not copy a running PGDATA.

4. Validation & Error Matrix

• DB ports appears -> environment contract fails.
• Any fixed container_name appears -> environment contract fails.
• Required data mount loses its parameter -> environment contract fails.
• Deploy/timer omits -p -> environment contract fails.
• Timer install omits either data path/build identifier, receives nested paths, a PGDATA without PG_VERSION, a missing storage directory, an invalid full SHA or a 12-character value that is not its prefix -> fail before writing the systemd unit.
• A path-segment regex can match an empty string at the leading / -> contract failure; use explicit root, trailing slash, //, /./ and /../ checks so valid absolute paths remain installable.
• Formal deploy omits paths/mode, uses a relative/broad/overlapping path, or an existing PGDATA lacks PG_VERSION -> fail before DB start.
• REMOTE_DIR overlaps PostgreSQL or attachment storage in either direction -> fail before provenance, SSH and rsync.
• A custom REMOTE_ENV_FILE lies inside the sync tree without dynamic protect/exclude filters -> deployment contract fails.
• approval_rules contains one or more rows -> seed:config reports them as preserved and performs no approval-rule create, update or delete.
• New project resolves the same PGDATA while the old DB runs -> stop before Compose start; do not use service health as permission to continue.
• Old project name was inferred, immutable image ID or baseline counts are missing, or target copy is non-empty -> stop the cutover.

5. Tests Required

• node deploy/trialrun/validate-env-contract.js.
• The isolated backup/restore drill mutates Compose with a DB port and fixed name, invokes formal deploy with each data path nested below REMOTE_DIR, and proves every unsafe shape is rejected before network access.
• bash -n covers deploy, timer, backup and restore scripts.
• Seed unit tests assert: an empty table receives exactly three defaults; any non-empty table receives zero approval-rule writes and no delete call.
• Timer contract assertions cover all four explicit inputs, path isolation, PGDATA marker, SHA/prefix validation and the four systemd Environment= entries. On the deployment host, assert timer enabled/active, daily service inactive, exact active-environment equality and non-mutating Compose resolve.

6. Good/Base/Bad Cases

• Good: one approved project resolves one stopped-and-migrated PGDATA and the matching active storage path; deploy, timer and backup all use that identity.
• Good: source lives under one absolute root, data lives under separate absolute roots, a custom env file is protected, and existing approval rules survive deployment init byte-for-byte until an approved rule transaction runs.
• Good: the timer unit contains the current project, active mount sources and deployed full/short SHA pair; Compose config and a non-mutating server exec resolve under those exact values while the daily service stays inactive.
• Base: direct local Compose uses the documented relative defaults under one project; those defaults are never accepted by formal deploy.sh.
• Bad: start a renamed project before the old DB stops, or let it create a new empty directory and accept a green health check as proof of data continuity.
• Bad: move the only PGDATA into the new path, migrate it in place, then assume the old container name alone is a rollback point.
• Bad: reinstall the timer with only COMPOSE_PROJECT_NAME, allowing systemd Compose interpolation to fall back to repository-relative data directories.
• Bad: place PGDATA or attachments below REMOTE_DIR and rely on one hard-coded rsync exclusion, or delete/recreate configurable approval rules in a routine deployment seed.

7. Wrong vs Correct

Wrong

docker-compose up -d db
DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh

Correct

docker-compose -p "$COMPOSE_PROJECT_NAME" -f deploy/trialrun/docker-compose.yml \
  --env-file deploy/trialrun/.env up -d db
mapfile -t DB_CONTAINERS < <(docker-compose -p "$COMPOSE_PROJECT_NAME" \
  -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env ps -q db)
test "${#DB_CONTAINERS[@]}" -eq 1
DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \
  STORAGE_DIR_HOST=<verified-active-storage> ./scripts/backup-all.sh

For timer installation, keep the active deployment identity explicit:

# Wrong: project identity alone still permits relative data-path fallback.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

# Correct: use the verified active mounts and deployed SHA pair.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
POSTGRES_DATA_DIR=<verified-active-pgdata> \
STORAGE_HOST_DIR=<verified-active-storage> \
BUILD_SHA=<verified-full-build-sha> \
VITE_BUILD_SHA=<verified-12-char-prefix> \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

For source synchronization and approval-rule defaults:

# Wrong: runtime data sits below the --delete synchronization root, and seed
# replaces operator-managed rules during every deployment.
REMOTE_DIR=/srv/oa POSTGRES_DATA_DIR=/srv/oa/data/postgres \
  bash deploy/trialrun/deploy.sh
# seedApprovalRules: deleteMany -> createMany

# Correct: data roots are disjoint; an in-tree env file is dynamically
# protected; seed creates defaults only for an empty approval_rules table.
REMOTE_DIR=/srv/oa-app POSTGRES_DATA_DIR=/srv/oa-data/postgres \
STORAGE_HOST_DIR=/srv/oa-data/storage DEPLOY_DATA_MODE=existing \
  bash deploy/trialrun/deploy.sh
# seedApprovalRules: count > 0 -> preserve; count = 0 -> create defaults

Scenario: Paired Backup And Shadow-Only Restore

1. Scope / Trigger

• Trigger: changing or operating database/file backups, restore scripts, retention instructions or recovery-readiness evidence.
• Purpose: prevent a database-only recovery point, a mixed attachment restore or an accidental destructive restore into the active internal instance.

2. Signatures

• Paired backup: DB_TOOL_MODE=docker DB_CONTAINER=<compose-resolved-id> STORAGE_DIR_HOST=<verified-active-directory> BUILD_SHA=<actual-build-sha> SOURCE_IMAGE_DIGEST=<actual-sha256-image-id> ./scripts/backup-all.sh; at least one source identity is required.
• Optional safe batch identifier: BACKUP_RUN_ID=<safe-id>.
• Manifest writer: host node is preferred; host python3 is an equivalent fallback for Docker-only runtime machines.
• DB shadow restore requires RESTORE_SCOPE=shadow, RESTORE_CONFIRM_DATABASE=<actual-current-database> and DATABASE_URL.
• Storage shadow restore: RESTORE_SCOPE=shadow ./scripts/restore-storage.sh <tar> <empty-target>.
• Local synthetic drill: ./scripts/qa/run-backup-restore-drill.sh.
• DB_TOOL_MODE is auto, host or docker; use docker with Compose to avoid host/server PostgreSQL tool-version drift.

3. Contracts

• A complete recovery point is one batch directory containing database.dump, storage.tar.gz and a final manifest.json. The manifest contains only safe relative names, byte sizes, SHA-256 hashes, batch time, actual build SHA when known and actual source image digest when supplied.
• BUILD_SHA supplied by the caller takes precedence over local Git fallback. On a non-Git runtime host, at least SOURCE_IMAGE_DIGEST=sha256:<64-hex> is mandatory. Never label a legacy backup with the commit planned for deployment.
• Manifest generation uses exclusive creation and preserves the same schema under Node and Python 3; if neither runtime exists, fail before publishing the final batch directory.
• For an approved live backup, do not assume the deployment .env still matches the running database. Confirm that the selected PostgreSQL container's database and user match at least one active application DATABASE_URL, then pass the verified runtime values without logging their contents.
• Docker backup/restore has no default container name. Resolve the unique DB container with compose -p <approved-project> ... ps -q db and fail when the result is empty or ambiguous. Storage backup likewise requires an explicit STORAGE_DIR_HOST; it must never infer STORAGE_DIR or a repository default.
• The DB dump must pass pg_restore --list and the storage archive must pass tar -tzf before the manifest may say complete.
• DB restore queries current_database() and compares it exactly with RESTORE_CONFIRM_DATABASE before pg_restore --clean.
• The actual DB name must also contain a separate shadow, drill or restore_test marker; setting RESTORE_SCOPE=shadow alone is not trust.
• Storage restore requires an explicit empty target and rejects absolute paths, .. path segments, symbolic links and hard links before extraction.
• Current scripts intentionally reject production restore. A real incident requires an approved Trellis task, stop window, paired batch, two responsible people, rollback point and post-restore /ready/permission checks.
• Copy the whole completed batch to approved off-host/off-disk storage; a batch left only on the application host is not accepted recovery evidence.
• A real-format shadow restore compares successful migration count and a fixed set of business-table counts with the pre-backup baseline, compares extracted regular-file count with the active storage source, and runs /ready using a schema-compatible image. /ready alone is not recovery evidence.
• When shadow SQL is supplied through stdin or a heredoc, docker exec must use -i and psql must use ON_ERROR_STOP=1; otherwise an empty result can be mistaken for a data mismatch even though no verification query ran.
• Compare fixed table counts as a keyed set, not by database-returned row order. Require the expected unique key count, then normalize both table=count collections with LC_ALL=C sort (or an equivalent map comparison) before equality checking; database collation order is not an evidence contract.

4. Validation & Error Matrix

• One backup component fails or integrity listing fails -> remove the temporary batch; do not write a complete manifest.
• Existing batch identifier -> fail without overwriting.
• Missing/non-shadow restore scope -> fail before any restore write.
• Actual DB name differs from confirmation -> fail before pg_restore --clean.
• Confirmed DB name lacks a shadow/drill/restore-test marker -> fail before pg_restore --clean.
• Storage target is implicit, root/repository root or non-empty -> fail before extraction.
• Corrupt dump/tar or dangerous/link archive member -> fail before restore.
• Host PostgreSQL tools mismatch in auto mode -> docker fallback is allowed only when the explicit target container and required DB variables exist.
• Neither Node nor Python 3 is available -> fail manifest generation and remove the temporary batch; do not publish a partial recovery point.
• Invalid build SHA/image digest, or neither identity available on a non-Git host -> fail before DB or storage backup starts.
• Deployment .env, PostgreSQL container and active application connection do not identify the same database/user -> stop the live backup; never guess credentials or expose values in diagnostics.
• Shadow migration count, any fixed table count, attachment regular-file count or compatible-image /ready differs from baseline -> reject the recovery point and do not enter production cutover.
• Empty, duplicate or missing table-count output -> classify as a verification execution failure first; do not report production-data drift until stdin was delivered and every expected key was returned exactly once.

5. Good/Base/Bad Cases

• Good: one command creates a validated pair, the complete batch is copied off host, and a disposable shadow drill restores the DB marker and attachment hash.
• Good: a Docker-only host uses Python 3 to write the same manifest contract after the active application connection is matched to the database container.
• Good: a legacy container without /ready.data.buildSha records its inspected sha256: image ID and leaves buildSha null rather than inventing provenance.
• Good: a heredoc query uses docker exec -i ... psql -v ON_ERROR_STOP=1, then compares the complete keyed count set under deterministic C sorting.
• Base: backup succeeds locally but off-host copy and deployed /ready evidence remain explicitly pending; do not claim formal recovery readiness.
• Bad: source a stale deployment .env and treat a failed dump as a prompt to guess or print credentials.
• Bad: source the active trial .env and run a cleaning restore directly, or restore attachments into a non-empty live directory.

6. Tests Required

• Shell syntax checks for all backup/restore scripts.
• Synthetic Docker/PostgreSQL and temporary-storage round trip.
• Negative assertions for wrong DB confirmation, corrupt DB/tar, non-empty target and dangerous tar path.
• Validate manifest relative paths, absence of connection secrets and matching SHA-256 values, explicit build-SHA precedence and image-digest preservation.
• Run the synthetic drill with Node available and with a PATH that excludes Node but includes Python 3; both paths must complete the same round trip.
• For real-format count checks, assert that stdin queries return the full unique key set, and prove that a different database collation/order does not change the pass result while a changed count still fails.
• Real-instance restore and /ready remain separately approved evidence and must never be fabricated from the local drill.

7. Wrong vs Correct

Wrong

source deploy/trialrun/.env
./scripts/backup-all.sh
./scripts/restore-db.sh some.dump
./scripts/restore-storage.sh some.tar.gz deploy/trialrun/data/storage

Correct

./scripts/qa/run-backup-restore-drill.sh
# Before an approved live backup, compare the selected DB container identity
# with an active app connection using redacted boolean diagnostics.
DB_TOOL_MODE=docker DB_CONTAINER=<verified-container> \
  STORAGE_DIR_HOST=<verified-active-storage> \
  SOURCE_IMAGE_DIGEST=<verified-sha256-image-id> ./scripts/backup-all.sh
# Real restore remains an approved incident operation after shadow evidence.

For a containerized count check, the input and comparison contract is:

# Wrong: stdin is not attached, and raw row order becomes part of the gate.
actual="$(docker exec shadow-db psql -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts ORDER BY table_key;
SQL
)"
test "$actual" = "$expected"

# Correct: deliver stdin, fail on SQL errors and compare keyed sets deterministically.
actual="$(docker exec -i shadow-db psql -v ON_ERROR_STOP=1 -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts;
SQL
)"
test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
  "$(printf '%s\n' "$expected" | LC_ALL=C sort)"

Scenario: Daily Restricted Off-Host Backup

1. Scope / Trigger

• Trigger: installing or changing automatic paired backups, off-host transfer, retention, freshness alerts or quarterly restore reminders.
• Purpose: keep a 24-hour recovery point without giving the production host ordinary shell or delete access to the ARM archive.

2. Signatures

• Production runner: scripts/run-automatic-backup.sh with run, check-stale and remind-drill modes.
• ARM forced receiver: scripts/receive-offhost-backup.sh.
• Shared host alert helper: scripts/send-wecom-group-alert.py with explicit --webhook-file, --summary-file and --state-file paths.
• ARM watchdog: scripts/check-offhost-backup-deadman.py with archive root, state root, alert helper, Webhook file and a default 93600 second threshold.
• Installers: install-automatic-backup-timer.sh and install-offhost-backup-receiver.sh.
• Local contract: bash scripts/qa/run-automatic-backup-contract.sh.
• Production required environment: REMOTE_DIR, REMOTE_ENV_FILE, COMPOSE_PROJECT_NAME, POSTGRES_DATA_DIR, STORAGE_HOST_DIR, BUILD_SHA, VITE_BUILD_SHA, AUTOMATIC_BACKUP_DIR, AUTOMATIC_BACKUP_STATE_DIR, OFFHOST_TARGET, OFFHOST_SSH_KEY, OFFHOST_KNOWN_HOSTS, ALERT_WEBHOOK_FILE and ALERT_HELPER.
• Receiver config: independent absolute BACKUP_INBOX_DIR, BACKUP_ARCHIVE_DIR, BACKUP_WEEKLY_DIR, RRSYNC_BIN, PYTHON_BIN, DAILY_KEEP and WEEKLY_KEEP.

3. Contracts

• Resolve one active Compose server/database pair and verify its build, PGDATA, storage mount and database identity before backup. Do not use fixed container names or repository-relative runtime data.
• Parse Docker mount identity from docker inspect JSON and require exactly one destination match. Do not split human-formatted Go-template output on tabs or spaces: println inserts argument spacing that can make a visually correct line fail exact path comparison on a real daemon.
• Database connection values used by the automatic wrapper must be exported in an isolated child shell before capture_command; do not invoke env with a complete DATABASE_URL or password assignment in its argv. Docker backup and shadow-restore commands inherit PGPASSWORD with PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD ...; never use a value-bearing -e PGPASSWORD=... Docker argument.
• Transfer only a complete auto-<UTC timestamp> three-file batch to ARM inbox. The forced key allows write-only rrsync or an exact finalize command; it rejects shell, sender/read and destructive rsync options.
• ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256, then atomically moves the batch to archive. Archive and weekly directories are outside the rsync root. A different run ID for the same UTC day may replace that day's archive only after full candidate validation: rename the old daily point to a same-filesystem rollback name, publish the candidate, rebuild that ISO week's hard-link checkpoint, then remove the rollback. Any publication or weekly-checkpoint failure restores the prior daily point.
• After taking the finalize lock and before reading the incoming partial, ARM reconciles interrupted same-filesystem transactions. A valid .daily-replacement-<UTC-day>.* is deleted only when one valid canonical archive for that day exists; when the canonical is absent it is renamed back from its own verified manifest identity. Invalid, linked, duplicate or otherwise ambiguous daily state fails closed and preserves every object.
• Weekly .rollback.* is authoritative only after full batch validation and is restored when the canonical checkpoint is absent. Weekly .new.* is never an authoritative source; it may be removed only after a valid canonical, rollback or unique valid daily source is established. Invalid canonical, rollback, duplicate candidate, week mismatch or non-unique rebuild source fails closed without deleting evidence. A reconciled daily point forces its ISO-week checkpoint to be rebuilt before finalize continues. When a valid canonical daily point and its old rollback coexist, defer deleting that daily rollback until every required weekly reconciliation has succeeded; a weekly validation or rebuild failure must leave the daily rollback in place.
• Write production last-success.json and prune source copies only after ARM finalize succeeds. Keep seven automatic source/ARM daily points and four ARM weekly checkpoints; unrelated manual batches are outside automatic retention.
• Daily, freshness and quarterly reminder timers are separate. Alert summaries contain categories only. A shared Python helper reads the complete WeCom group robot URL only from a regular, executing-account-owned 0600 file and never prints it. It sends the first failure, worsening/new findings, one recovery notice and the first quarterly reminder, while suppressing an identical repeated state. It does not call the OA server container or resolve an employee.
• ARM runs its own 26-hour dead-man as root. It verifies the newest archive's exact file set, manifest identity, sizes, checksums and timestamp, so a stopped production host/server can still cause a group alert. Missing, corrupt, stale or future-dated archives fail; recovery sends once.
• Production command diagnostics for paired backup, rsync and finalize stay only in the state directory as last-error.log mode 0600. The summary and group message contain only the safe phase category. A later successful off-host run removes the retained diagnostic.
• The production freshness check hard-fails missing, malformed, future, stale or Compose-project-mismatched state. A fresh recovery point with a different build SHA is PASS_WITH_FINDINGS, because deploy-time code identity and recoverable data freshness are separate facts.
• Automatic-backup units freeze the active Compose project, build, PGDATA and storage mount when installed. After any deployment, rollback or direct container rebuild that changes the active container identity or host mounts, compare all frozen values with the unique running server/database pair. If the units are already installed and any value differs, rerun the existing installer with the complete current contract and recheck all identities before declaring the deployment complete; do not patch one unit in place.
• Installer install is disabled-by-default; enable, real transfer, real WeCom alert and all server/account/key changes remain EXTERNAL_GATED.

4. Validation & Error Matrix

• Source mount/build/project drift, lock conflict or incomplete backup -> fail before success state and source retention.
• Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no off-host success state, preserve the source batch and retain a root-only local diagnostic. A corrupt same-day candidate leaves the previous archive intact.
• Valid same-day retry -> replace only that UTC day's point and rebuild its ISO weekly checkpoint; concurrent finalize -> flock rejection.
• Interrupted daily replacement with valid canonical -> remove the verified old rollback only after rebuilding the weekly point; canonical missing -> restore the verified rollback; canonical invalid, weekly rebuild failure or candidates ambiguous -> preserve the remaining rollback evidence and fail.
• Interrupted weekly replacement with valid canonical -> remove safe staging and verified rollback; canonical missing + valid rollback -> restore it; staging-only + one valid daily source -> rebuild; any invalid or ambiguous authority -> preserve all objects and fail.
• Missing, corrupt, stale, future or project-mismatched success record -> hard watchdog failure. Fresh build mismatch -> one finding without a failing exit.
• phase_stale_check_failed can also mean the watchdog failed its runtime identity precheck before reading the success record. Diagnose the nested safe category and active-unit identity first; do not infer that the last archive is stale or corrupt from this outer phase alone.
• Missing/non-regular/symlink/non-0600 Webhook file -> fail closed without printing the file value. Identical alert state -> update/evaluate without send.
• Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver rejection without changing archive.
• ARM host/network/WeCom all unavailable at once -> the group robot path cannot prove its own outage; this physical boundary is not a green state.

5. Good/Base/Bad Cases

• Good: one verified active source creates a complete batch, write-only rsync resumes inside inbox, ARM finalize verifies and archives it, then production writes success and prunes old automatic source points. If production later stops, ARM independently raises one dead-man alert and one recovery notice.
• Good: a deployment rebuilds the server with a different storage mount, then refreshes the installed backup units through the installer and proves every frozen identity matches before closing the deployment.
• Base: local implementation and synthetic contracts pass while both server installers remain disabled and explicitly externally gated.
• Bad: send through compose exec server, hard-code a personal employee number, persist a robot URL in .env/unit/repository, give production a normal ARM login key, put archive below inbox, accept a new host fingerprint automatically, prune local points before target-side finalize, or leave installed backup units bound to a pre-deployment mount until the next scheduled alert.

6. Tests Required

• Shell syntax plus run-automatic-backup-contract.sh using temporary files and mocked Docker/SSH boundaries; it must never contact real hosts.
• Docker mocks must return the real {{json .Mounts}} shape consumed by the runner. A simplified tab-delimited mock is not valid evidence for real mount parsing.
• Contract assertions for same-day replacement, corrupt-candidate preservation, weekly hard-link rebuild, daily/weekly interrupted-transaction recovery, rollback preservation when weekly reconciliation fails, ambiguous-state preservation, source diagnostic mode, build mismatch finding, Webhook mode/symlink rejection and group alert behavior without a server container. Test transport must be explicitly gated and local-only.
• Mock Docker assertions must prove synthetic connection values are absent from wrapper/backup/restore argv while an inherited PGPASSWORD reaches the mocked container command. Test-fixture env KEY=value launch arrays do not need mechanical rewriting because they contain only synthetic values.
• Environment contract assertions for both helpers, both installers, production timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file binding, disabled-by-default installation and absence of a literal complete robot URL in tracked or untracked repository files.
• Deployment and rebuild review must assert that installed production units still match the unique active project, build, PGDATA and storage mount. A changed mount with otherwise healthy /ready is not a completed deployment until installer refresh, timer state and runtime identity all pass.
• Real ARM setup, first paired batch, deliberate source-timer stop or transfer failure, dead-man first/repeated/recovery behavior and quarterly shadow restore are separately approved operational evidence. A direct robot send test alone is not dead-man evidence.

7. Wrong vs Correct

Wrong

# Rebuild the application, observe /ready, but leave installed backup units
# frozen to the pre-deployment runtime identity.
docker-compose <approved-compose-arguments> up -d --build server

compose exec server send-alert --recipient=<personal-employee-number>
rsync -a <batch>/ backup-host:<archive>/
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ...

Correct

# After a deployment/rebuild, compare the installed unit contract with the
# unique active server/database pair. If any frozen identity changed, refresh
# all units through the installer before verifying timers and identity again.
<complete-current-contract-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh enable

# Install only with explicit active identity and a root-only Webhook file path;
# default remains disabled.
<explicit-project-data-build-ssh-and-webhook-file-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh install
# ARM separately verifies archive freshness even when production is unavailable.
python3 scripts/check-offhost-backup-deadman.py <explicit-safe-path-arguments>
# The runner writes success and prunes only after restricted target finalize.
bash scripts/run-automatic-backup.sh run
# The Docker CLI inherits the value without placing it in argv.
PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...

Scenario: Independent Public Application Health Monitor

1. Scope / Trigger

• Trigger: installing or changing the off-host public /ready probe, its ARM timer, application-health alert states or the shared WeCom group-alert helper.
• Purpose: detect an unavailable OA host, edge/TLS path, application or database readiness signal without adding a time-series platform or relying on traffic.

2. Signatures

• Probe: scripts/check-public-app-health.py --ready-url <https-/ready> --state-dir <root-0700-dir> --webhook-file <root-0600-file> --alert-helper <installed-helper> --failure-threshold 2 --timeout-seconds <1..30>.
• Installer: deploy/trialrun/install-offhost-app-health-monitor.sh install|enable|uninstall.
• Units: hyb-oa-public-app-health.service and hyb-oa-public-app-health.timer; the timer uses OnCalendar=*:0/5.
• Local contract: bash scripts/qa/run-app-health-monitor-contract.sh.

3. Contracts

• Run the probe from the existing ARM host against an exact HTTPS /ready URL. Success requires trusted TLS, HTTP 2xx, bounded JSON and data.ok === true.
• Count any stable failure category toward the same consecutive-failure threshold. The first failure is pending; the second evaluates a failure alert. Once alerting, a changed category may send a new alert, identical failures are deduplicated, and one later success sends one recovery notice.
• Persist only schema version, count, stable category and the alert state in a root-owned 0700 directory with atomic 0600 files. Never persist or send the URL, resolved address, response payload, credential or raw exception.
• Application-health probe state and alert state are separate from backup and dead-man state. The installer must deploy the helper revision that passed the backup contract; an older installed helper may reject the new status tokens.
• install writes only the exact managed probe/helper/state/unit targets and leaves the timer disabled. enable, ARM writes, real robot messages and a controlled failure/recovery drill remain separately approved operations.
• Before any remote write, prove the selected SSH identity can run sudo -n and that no application-health artifact already exists. A password prompt is a stop condition, not permission to request or transmit a password in chat. If the operator creates a temporary privilege rule, remove its exact file at the end, validate sudoers in the same root invocation, invalidate the sudo timestamp and prove the original non-interactive access is gone.
• uninstall removes only the exact application-health units and probe. It retains the shared helper, webhook, state evidence and all backup components.

4. Validation & Error Matrix

• Non-HTTPS, userinfo, query, fragment or non-/ready URL -> reject before a request with ready_url_invalid.
• DNS/connect, TLS, timeout, HTTP, oversized response or invalid JSON/contract -> stable redacted category and non-zero oneshot exit.
• Invalid owner/mode/link/state or alert-helper failure -> fail closed with a fixed local error; do not include the offending value.
• A first isolated failure followed by health -> no failure or recovery message.
• Any overridden managed target or service name -> installer stops before systemctl, file replacement or removal.
• SSH succeeds but sudo -n requests a password -> stop before upload or systemd mutation; resume only after an operator establishes a separately approved, non-chat root path and rerun the complete preflight.
• Temporary privilege rule remains after final health verification -> rollout is incomplete even when the timer is healthy.
• ARM itself or its network/robot path unavailable -> the monitor can be silent; do not report this design as having an independent third-party dead-man.

5. Good/Base/Bad Cases

• Good: ARM observes two consecutive unavailable checks, sends one redacted category alert, suppresses repeats and sends one recovery after health returns.
• Base: repository contracts pass while units remain disabled; report the code as ready for an approved rollout, not as an active monitor.
• Bad: run the timer only on the OA host, send raw /ready errors, install and enable in one unapproved step, or deploy the probe without its helper revision.

6. Tests Required

• The local HTTPS stub must assert healthy baseline, pending first failure, second-failure alert, identical deduplication, changed-category alert, one recovery, transient suppression, TLS/connect/timeout/HTTP/size/JSON failures, corrupt private state, URL rejection and absence of sensitive output.
• Run Python compile and shell syntax checks, validate the five-minute calendar, and assert disabled-by-default plus exact managed targets and retained state.
• For a remote rollout, record boolean evidence for pre-write sudo -n, exact temporary-upload removal, temporary privilege removal and post-removal non-interactive-sudo failure; never record the password, host address or rule contents.
• Re-run bash scripts/qa/run-automatic-backup-contract.sh whenever the shared helper allowlist or behavior changes.
• ARM install, enabled/active timer status and real failure/recovery messages are external evidence and may not be inferred from the local contract.

7. Wrong vs Correct

Wrong

# Local contract passed, so install and enable remotely and call it active.
bash deploy/trialrun/install-offhost-app-health-monitor.sh enable

Correct

bash scripts/qa/run-app-health-monitor-contract.sh
bash scripts/qa/run-automatic-backup-contract.sh
# After separate ARM-write approval: install disabled, inspect, then seek
# enable/real-message/drill approval before claiming the monitor is active.
<explicit-controlled-paths> \
  bash deploy/trialrun/install-offhost-app-health-monitor.sh install

Scenario: Existing Deployment Preflight And Cache Stability

1. Scope / Trigger

• Trigger: deploying a new published commit onto an existing single-host trialrun installation.
• Purpose: prevent a wrong repository-root interpretation from reaching rsync --delete, and prevent per-commit build identity from invalidating stable dependency layers.

2. Signatures

• Existing deployment entry: DEPLOY_SERVER=<ssh-target> bash deploy/trialrun/deploy-existing.sh.
• Remote preflight: deploy/trialrun/preflight-remote.sh executed over SSH before rsync.
• Canonical parameter source: allowlisted Environment= values from hyb-oa-automatic-backup-daily.service.
• Stable image stages: base, client_deps, client_build, server_deps, server_build.

3. Contracts

• Never derive REMOTE_DIR from Compose com.docker.compose.project.working_dir; it may name the Compose-file directory rather than the repository root.
• The existing-deployment wrapper reads only REMOTE_DIR, REMOTE_ENV_FILE, COMPOSE_FILE, COMPOSE_PROJECT_NAME, POSTGRES_DATA_DIR and STORAGE_HOST_DIR; it does not echo their values.
• Before mkdir or rsync, existing mode requires repository markers, PG_VERSION, storage directory and a regular non-symlink env file when SYNC_ENV=0.
• apk add and both lockfile npm ci steps precede BUILD_SHA / VITE_BUILD_SHA. Build identity may invalidate application output, never stable dependency installation.
• Seed and smoke output reports readiness, counts and booleans; it does not print full login identifiers, contact values, WeCom identifiers, redirect URI or generated credentials.

4. Validation & Error Matrix

• Config directory passed as REMOTE_DIR -> preflight fails before rsync.
• Remote env missing or symlinked with SYNC_ENV=0 -> preflight fails before rsync.
• Existing PGDATA missing PG_VERSION or storage missing -> preflight fails before rsync.
• Canonical systemd unit missing an allowlisted value -> wrapper fails without calling deploy.sh.
• Lockfile unchanged but a new SHA reruns dependency installation -> Dockerfile contract test fails.

5. Good/Base/Bad Cases

• Good: wrapper reads the frozen unit contract, preflight passes, Git archive syncs and unchanged dependency layers are cached.
• Base: first initialization uses reviewed explicit parameters and DEPLOY_DATA_MODE=initialize; it does not pretend an existing unit is available.
• Bad: inspect a running container's working_dir, append another deploy/trialrun, then discover the mistake only after rsync.

6. Tests Required

• bash -n deploy/trialrun/deploy.sh deploy/trialrun/deploy-existing.sh deploy/trialrun/preflight-remote.sh.
• node deploy/trialrun/validate-env-contract.js.
• cd server && npm test -- tests/unit/scripts/deployment-safety-contract.test.js.
• Synthetic fixtures must prove wrong root, missing/symlink env and invalid data fail; static ordering must prove preflight precedes rsync and dependency stages precede build args.

7. Wrong vs Correct

Wrong

# working_dir may be the Compose config directory, not the repository root.
REMOTE_DIR="$(docker inspect <server> --format '<working_dir-label>')" \
  bash deploy/trialrun/deploy.sh

Correct

DEPLOY_SERVER=<approved-ssh-target> \
  bash deploy/trialrun/deploy-existing.sh

Scenario: Environment Recovery Classification And Secret Custody

1. Scope / Trigger

• Trigger: adding/removing a trialrun environment key, documenting host-loss recovery, or preparing an environment-file migration.
• Purpose: make environment loss recoverable without placing Secret values in Git, ordinary backup, logs or AI context.

2. Signatures

• Template contract: deploy/trialrun/.env.example.
• Recovery classification: deploy/trialrun/env-recovery-classification.json.
• Recovery card: the opening section of deploy/trialrun/runbook.md.
• Validator: node deploy/trialrun/validate-env-contract.js.
• Fixed categories: template_rebuild, external_reissue, direct_rotate, downtime_reset, preserve_original.

3. Contracts

• Every .env.example key appears in exactly one fixed recovery category; the classification file contains key names only, never values, defaults, hashes, accounts or platform addresses.
• template_rebuild comes from reviewed templates/runbooks; external_reissue comes from the owning platform; direct_rotate records the invalidation consequence; downtime_reset requires an isolated approved outage.
• POSTGRES_PASSWORD is a reset case. PostgreSQL stores a verifier in pg_authid, not recoverable plaintext. Any temporary authentication relaxation is local-socket only, never a trusted host rule, and the original authentication file must be restored byte-for-byte before reopening the application.
• FIELD_ENCRYPTION_KEY is preserve_original and the highest-risk class. Its current AES-256-GCM payload is v1:<iv>:<tag>:<ciphertext> without a key id; replacing a lost key does not recover existing bank, identity or notification-channel ciphertext.
• preserve_original custody must be available to both the system administrator and a company responsible person through a repository-external shared vault or encrypted sealed copy. Evidence records roles and status only.
• Original-key custody is a current recovery constraint, not a permanent rotation design. Key id, keyring, lazy/bulk re-encryption and old-key retirement require a separate approved security task.
• Database/attachment backups do not absorb plaintext environment files. Phase 1 recovery-contract work never reads production env or contacts production systems.

4. Validation & Error Matrix

• New .env.example key missing from classification -> validator fails.
• Key appears in two categories -> validator fails and names the duplicated key.
• Unknown category, extra key, illegal key name or value-like entry -> validator fails.
• FIELD_ENCRYPTION_KEY outside preserve_original -> validator fails.
• POSTGRES_PASSWORD outside downtime_reset -> validator fails.
• Original encryption key unavailable -> stop recovery and report ciphertext impact; never generate a replacement and claim success.
• Temporary database trust remains or a DB host port is published -> recovery fails and the external entry remains closed.

5. Good/Base/Bad Cases

• Good: a TTL is rebuilt from the template, a WeCom Secret is reissued by its platform owner, JWT is rotated with session invalidation, and the field-encryption key is restored from dual-access custody.
• Base: the production key set differs from the example contract; a separately approved read-only check explains key names/counts without reading values before migration.
• Bad: put .env into the paired data backup, print a secret hash as evidence, assume pg_authid reveals the old password, or replace FIELD_ENCRYPTION_KEY with a new random value.

6. Tests Required

• node deploy/trialrun/validate-env-contract.js.
• Focused unit tests must cover the complete contract plus missing, duplicate, extra, unknown-category, illegal/value-like, field-key and database-password misclassification cases.
• Run node scripts/qa/check-sensitive-content.mjs, Trellis governance, process contract, docs registry audit and git diff --check after classification/runbook changes.
• Production key-name comparison, shared-vault custody, env migration, systemd updates, restarts and old-copy deletion remain separately approved Phase 2 evidence.

7. Wrong vs Correct

Wrong

{"direct_rotate":["FIELD_ENCRYPTION_KEY"]}

Correct

{"preserve_original":["FIELD_ENCRYPTION_KEY"]}

Scenario: Formal Feature-Flag Baseline And Reversible QA Window

1. Scope / Trigger

• Trigger: a feature is formally enabled after an earlier short-window rollout, or a QA runner temporarily changes an existing feature flag.
• Purpose: prevent a new database, stale .env or QA cleanup path from silently returning a formal capability to its old disabled state.

2. Signatures

• Hard gate: PAYMENT_BATCH_ENABLED=true in deploy/trialrun/.env.example and the active production env.
• Database baseline: payment_batch_v1 is first-created as force_on/global; seed:config still uses create plus P2002 preservation and never updates an existing row.
• Deployment assertion: smoke-trialrun-runtime.js requires both the env value and effective FEATURE_FLAG_KEYS.paymentBatch result.
• QA snapshot helper: node scripts/qa/payment-batch-flag-snapshot.mjs capture|restore --snapshot-file <private-json>.
• Mobile entry: /mobile/payment-batches; PC remains /payment-batches.

3. Contracts

• Formal enablement is a three-layer contract: env hard gate, database feature flag and runtime smoke. Changing only one layer is incomplete.
• A short QA window captures overrideMode, rolloutType, departmentIds and note before its first flag write and restores those fields in finally; it must work when the caller starts either enabled or disabled.
• A formal-enable runner keeps force_on only after every acceptance step passes. Any failed step restores its caller snapshot, not a hard-coded force_off.
• Snapshot files use mode 0600, contain configuration only and remain under the ignored QA report directory. Credentials and access tokens never enter the snapshot or output.
• Mobile links must target the mobile route. Enterprise-messenger routing maps a PC-only payment path back to the mobile workbench, so a PC path is not a valid mobile deep link even when the underlying page is responsive.
• Mobile and PC payment routes share the same permission guard; adding reachability must not broaden payment.batch.manage|payment.batch.review|risk.review.manage.

4. Validation & Error Matrix

• Env false or absent -> runtime smoke fails before deployment is accepted.
• Database flag absent/disabled -> runtime smoke fails with the feature-flag reason; do not hide the failure by changing navigation only.
• Seed meets an existing flag -> P2002 is preserved with zero update/upsert calls.
• QA capture fails -> stop before the temporary window; do not run with an unknown rollback target.
• QA action or later matrix fails -> restore the caller snapshot and keep the run failed.
• Snapshot restore version conflicts -> fail visibly; never fall back to force_off.
• Mobile finance entry points at /payment-batches -> small-screen routing returns to /mobile; treat this as an unreachable entry and use /mobile/payment-batches.
• User lacks payment permission -> both routes remain forbidden even if the feature is enabled.

5. Good/Base/Bad Cases

• Good: env true + seed first-create force-on + effective runtime smoke pass; a temporary off/on test restores an original allowlist and note byte-for-byte.
• Base: a database already contains an operator-managed flag; deployment seed preserves it and runtime smoke reports whether it matches the formal baseline.
• Bad: change only the DB flag while env remains false, or make the mobile card visible but link it to a path that the mobile shell redirects home.
• Bad: every QA cleanup writes force_off, erasing a formal-on state that existed before the test.

6. Tests Required

• Seed unit test asserts first-create force_on/global and proves P2002 performs no update/upsert.
• Snapshot helper tests cover an original force_off allowlist and an original force_on global configuration, including note restoration.
• Deployment contract test asserts the env baseline, runtime-smoke flag assertion and absence of fixed-off cleanup helpers.
• Mobile Playwright covers zero batches/zero exceptions with the feature enabled, a visible “进入付款” link, successful /mobile/payment-batches navigation and no finance card for an unrelated role.
• Run client typecheck/build, Prisma validate, shell/Node syntax, task/governance/process/sensitive gates and git diff --check.
• Public rollout additionally verifies active env, effective flag reason, finance navigation and eligible candidate count without creating, reviewing or executing a real batch.

7. Wrong vs Correct

Wrong

# Cleanup assumes disabled was always the caller's state.
PUT payment_batch_v1 '{"overrideMode":"force_off"}'

<Link to="/payment-batches">进入付款</Link>

Correct

node scripts/qa/payment-batch-flag-snapshot.mjs capture --snapshot-file "$REPORT_DIR/flag-before.json"
# temporary test writes
node scripts/qa/payment-batch-flag-snapshot.mjs restore --snapshot-file "$REPORT_DIR/flag-before.json"

<Link to="/mobile/payment-batches">进入付款</Link>