hyb-oa/.trellis/spec/hyb-oa/deployment-trialrun.md

Deployment / Trial Run

Public OA Boundary

Public OA is quasi-production. Deployments, remote config changes and live data operations require explicit approval and evidence.

Rules

• Use SYNC_ENV=0 POSTGRES_DATA_DIR=<absolute> STORAGE_HOST_DIR=<absolute> DEPLOY_DATA_MODE=<existing|initialize> bash deploy/trialrun/deploy.sh only when deployment is approved; do not overwrite remote .env.
• When server/src/config.js reads a new environment key, add the same key to deploy/trialrun/.env.example before deployment; deploy/trialrun/deploy.sh runs node deploy/trialrun/validate-env-contract.js at step 0 and must fail if the example contract drifts.
• After deployment, report commit hash, /ready, public smoke results and whether the user can test.
• Real business writes are manual unless the current Trellis task explicitly authorizes a controlled test write.
• Do not execute real payments, import bank statements, connect payment providers or write ERP without a separate approved Trellis task.

Scenario: Deployment Env Contract

1. Scope / Trigger

• Trigger: any backend config, deployment compose, or trialrun env wiring change.
• Purpose: prevent deploy-time surprises where local code compiles but the deployment contract rejects the release or silently misses a runtime default.

2. Signatures

• Command: node deploy/trialrun/validate-env-contract.js
• Deployment entry requires SYNC_ENV=0, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR and DEPLOY_DATA_MODE=existing|initialize.
• Contract sources:
  • server/src/config.js
  • deploy/trialrun/.env.example
  • deploy/trialrun/docker-compose.yml

3. Contracts

• Every process.env.KEY or requiredEnv("KEY") in server/src/config.js must be present in deploy/trialrun/.env.example.
• server, worker, and init services must load ./.env through env_file.
• Compose service-level environment overrides are limited to the explicit allowlist in validate-env-contract.js.
• SYNC_ENV=0 is the default public deploy mode; it does not overwrite remote .env.
• Relative data-path defaults are allowed only for direct local Compose. deploy.sh requires explicit absolute host paths and a data mode; existing verifies the PGDATA directory contains PG_VERSION, while initialize is the only mode allowed to create missing directories.

4. Validation & Error Matrix

• Missing key in .env.example -> validate-env-contract.js exits non-zero.
• Service omits env_file: ./.env -> contract exits non-zero.
• Unauthorized compose override -> contract exits non-zero.
• Remote .env missing -> deploy step 4 exits before migration.
• Missing/relative/unsafe/overlapping data paths or missing data mode -> deploy exits locally before provenance, synchronization or SSH.
• existing target missing PG_VERSION or storage directory -> remote deploy exits before Compose starts DB.

5. Good/Base/Bad Cases

• Good: add EMAIL_LOGIN_CODE_TTL_SECONDS to config.js, document it in .env.example, then run contract validation before deploy.
• Base: code-only feature without env changes still runs the contract gate.
• Bad: add a config key only in config.js; deploy fails at step 0.

6. Tests Required

• Always run node deploy/trialrun/validate-env-contract.js after env-related changes and before public deployment.
• For public deploys, also run or verify /ready, runtime smoke, external deps smoke, public bundle copy scan, and live smoke when credentials exist.

7. Wrong vs Correct

Wrong

# Config key exists only in server/src/config.js.
SYNC_ENV=0 bash deploy/trialrun/deploy.sh

Correct

# Add the key to deploy/trialrun/.env.example first.
node deploy/trialrun/validate-env-contract.js
SYNC_ENV=0 POSTGRES_DATA_DIR=/srv/hyb-oa/pgdata \
  STORAGE_HOST_DIR=/srv/hyb-oa/storage DEPLOY_DATA_MODE=existing \
  bash deploy/trialrun/deploy.sh

Scenario: Public Smoke Navigation Labels

1. Scope / Trigger

• Trigger: frontend shell, navigation labels, sidebar grouping, workbench copy, or live smoke assertions change.
• Purpose: prevent public deployment from being marked failed only because smoke tests look for stale navigation copy while the page itself is healthy.

2. Signatures

• Live smoke command: E2E_MODE=live LIVE_SMOKE_BASE_URL=https://oa.hyinda.com npx playwright test tests/e2e/live-smoke.spec.ts
• Shared assertions: client/tests/e2e/helpers/smokeAssertions.ts
• Workbench route: /portal
• Current primary workbench nav label: 工作台
• Stale user-facing label that must not be accepted by current mocks or smoke: Portal

3. Contracts

• assertPortalLoaded(page) must assert route and page content first: /portal URL and the 今日待处理 heading.
• Navigation-label assertions for the workbench link must require the exact current label 工作台; mocks and fixtures must use the same contract.
• Navigation assertions must be role-aware. Admin/manager smoke may assert team/admin entries such as 团队, but ordinary employee smoke must not require links that NavLinks intentionally hides for that role.
• Smoke tests must not require a copy string that is not a product contract for public availability.
• If the product intentionally renames the primary workbench label again, update the product contract, fixtures and shared smoke helper together rather than adding an ad hoc fallback in one live smoke test.

4. Validation & Error Matrix

• Page reaches /portal, 今日待处理 is visible, but nav label is 工作台 -> smoke should pass.
• Page reaches /portal and renders Portal as the user-facing nav label -> smoke should fail as a stale-copy regression.
• Employee reaches /portal, role-aware navigation hides 团队, and employee pages such as 项目 remain reachable -> smoke should pass.
• Page reaches /portal, but 今日待处理 is missing -> smoke should fail.
• Page stays on login, /403, error fallback or another route -> smoke should fail.
• New shell label appears only in one live smoke test -> stop and align the product contract, fixtures and shared helper before changing smoke.

5. Good/Base/Bad Cases

• Good: assertPortalLoaded checks /portal, 今日待处理, and an exact 工作台 workbench link.
• Good: assertMainNavVisible(page, { expectTeam: false }) is used for ordinary employee live smoke.
• Base: the internal route and helper name may retain portal, while all user-facing navigation fixtures and assertions use 工作台.
• Bad: a mock emits Portal and smoke accepts it as a valid current label, masking a user-facing English-copy regression.
• Bad: employee live smoke requires the 团队 sidebar link after role-aware navigation intentionally hides it.

6. Tests Required

• After changing live smoke assertions, run: cd client && npm run typecheck
• Load the focused visual and shell specs with Playwright --list when changing fixture/report contracts.
• For public deploy verification, run live smoke or the readiness wrapper: LIVE_SMOKE_BASE_URL=https://oa.hyinda.com bash scripts/qa/run-live-smoke-readiness.sh
• When changing the shell or Portal page, include a focused Playwright check that proves the workbench route and heading still load.

7. Wrong vs Correct

Wrong

await expect(
  page.getByRole("link", { name: "工作台", exact: true })
    .or(page.getByRole("link", { name: "Portal", exact: true }))
    .first(),
).toBeVisible();

Correct

await expect(page.getByRole("link", { name: "工作台", exact: true }).first()).toBeVisible();

Scenario: Trial Run Runtime Navigation Baseline

1. Scope / Trigger

• Trigger: changing sidebar grouping, workspace module visibility defaults, server/src/services/workspaceSettings.js, or server/src/scripts/smoke-trialrun-runtime.js.
• Purpose: keep deployment smoke aligned with the current small-team navigation model instead of blocking deployment on retired menu assumptions.

2. Signatures

• Runtime smoke command inside deployment: docker compose exec -T server node src/scripts/smoke-trialrun-runtime.js
• Source of truth for sidebar grouping: server/src/services/workspaceSettings.js
• Runtime smoke assertion file: server/src/scripts/smoke-trialrun-runtime.js

3. Contracts

• Runtime smoke must distinguish primary sidebar entries from collapsed 更多 entries.
• Core day-to-day ledgers such as 报销, 采购, 合同, 项目 and 资产 are expected in main when the active role has access.
• Low-frequency entries such as 周报 and 公告 may be enabled but should be checked in backup / 更多, not forced into main.
• 待办 follows the current workspace and role-aware navigation baseline; do not move it between main, backup or hidden as part of a deployment-smoke fix unless a product task explicitly changes that rule.
• Management pages should remain represented by the single 管理 entry unless a current task explicitly changes the management navigation model.

4. Validation & Error Matrix

• Core ledger missing from main -> runtime smoke fails with 系统导航 main 缺少 <key> 入口.
• 周报 or 公告 missing from backup while enabled -> runtime smoke fails with 系统导航 backup 缺少 <key> 入口.
• Management children are hidden behind 管理, but smoke still expects workspace, people, featureFlags or auditLogs in admin -> update the smoke baseline; do not re-expand the sidebar for the smoke.
• systemSettings missing from admin -> runtime smoke fails because the management entry is no longer reachable.

5. Good/Base/Bad Cases

• Good: main includes portal, reimbursements, purchaseRequests, contractsRegistry, projects and assets; backup includes weeklyReports and announcements; admin includes systemSettings.
• Base: 待办 is in main for a configured admin/owner trial account; smoke accepts it there because it follows the current workspace baseline.
• Bad: smoke expects weeklyReports in main after the sidebar rule moved low-frequency modules to 更多.
• Bad: smoke expects every management child as a sidebar row after the product rule consolidated management into one 管理 entry.

6. Tests Required

• Run node --check server/src/scripts/smoke-trialrun-runtime.js after editing runtime smoke.
• Run SYNC_ENV=0 bash deploy/trialrun/deploy.sh for public deployment so the remote runtime smoke validates the actual workspace and feature-flag state.

7. Wrong vs Correct

Wrong

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "weeklyReports",
  "announcements",
];
const REQUIRED_ADMIN_NAV_KEYS = ["workspace", "people", "featureFlags"];

Correct

const REQUIRED_MAIN_NAV_KEYS = [
  "portal",
  "reimbursements",
  "purchaseRequests",
  "contractsRegistry",
  "projects",
  "assets",
];
const REQUIRED_BACKUP_NAV_KEYS = ["weeklyReports", "announcements"];
const REQUIRED_ADMIN_NAV_KEYS = ["systemSettings"];

Scenario: Immutable Deployment Provenance And Dual Build SHA

1. Scope / Trigger

• Trigger: changing the trialrun deployment entry, Docker build, Compose wiring, /health or /ready, public bundle scanning or live readiness.

2. Signatures

• Provenance gate: node deploy/trialrun/check-source-provenance.mjs.
• Standard release additionally supplies the explicit absolute data-path and data-mode contract required by deploy.sh.
• Build/runtime inputs: BUILD_SHA=<full-commit> and VITE_BUILD_SHA=<12-char-commit>.
• Runtime outputs: /health.data.buildSha, /ready.data.buildSha and the frontend html[data-build-sha] marker.

3. Contracts

• Deployment rejects tracked and untracked dirty files before synchronization.
• By default HEAD must be contained in its upstream. The explicit unpublished override is for controlled offline recovery and never relaxes clean-tree.
• Standard deployment refreshes the configured upstream remote before the ancestor check; a stale local remote-tracking ref is not publication proof.
• Unpublished override requires an allowlisted DEPLOY_UNPUBLISHED_REASON and must emit an UNPUBLISHED-OVERRIDE audit line.
• Synchronization comes from git archive <verified-commit>, not the working tree.
• Full BUILD_SHA is injected at build and runtime; VITE_BUILD_SHA is normalized before bundling so only 12 characters become public.
• Neither build key is persisted in deploy/trialrun/.env; direct Compose use must explicitly provide both and must not publish unknown.
• Deployment compares public /ready and the public bundle marker with the verified commit before reporting success.

4. Validation & Error Matrix

• Dirty tree -> dirty_worktree, deployment stops.
• No upstream or commit absent from upstream -> stop unless the explicit unpublished override is approved.
• Upstream refresh fails -> stop in standard mode; override mode requires an allowlisted reason and records remoteVerified=false.
• Missing runtime/build SHA input -> Compose interpolation fails.
• /ready SHA mismatch, bundle marker missing or full SHA exposed -> deployment fails.
• Standalone readiness missing/mismatching observed SHA -> non-clean finding.

5. Good/Base/Bad Cases

• Good: clean published commit -> Git archive -> dual-SHA build -> matching backend and public bundle checks.
• Base: local development build uses unknown but is never represented as a release.
• Bad: rsync the current working tree or let .env override the image's SHA with unknown.

6. Tests Required

• Temp-repo provenance tests for dirty, missing upstream, published and unpublished-override cases, including a stale remote-tracking ref after a forced remote update.
• node deploy/trialrun/validate-env-contract.js and Compose config with both SHA inputs.
• Synthetic 40-character build: bundle contains the matching 12-character SHA and not the full value.
• Local-stub readiness test for expected/observed SHA propagation.
• Public deployment still requires separate approval and post-deploy checks.

7. Wrong vs Correct

Wrong

rsync -az --delete ./ remote:/app
docker compose up -d --build server

Correct

node deploy/trialrun/check-source-provenance.mjs
BUILD_SHA=<verified-full-sha> VITE_BUILD_SHA=<matching-short-sha> \
  docker compose -f deploy/trialrun/docker-compose.yml build server

Scenario: Isolated Authentication Security Gate

1. Scope / Trigger

• Trigger: changing authentication challenges, credential/session transactions, trusted proxy handling or the deployment order for their schema migration.
• The gate proves local PostgreSQL behavior only; public deployment and edge changes remain separately approved operations.

2. Signatures

• Complete local gate: bash scripts/qa/run-web-gates.sh auth-security-local.
• Database allowlist: loopback PostgreSQL and a fresh hyb_oa_auth_* database whose exact name is confirmed to the checker.
• Runtime guard: RUN_AUTH_SECURITY_REGRESSION=1; concurrency mode accepts only 0 or 1 and reports FULL only for 1.
• Public release/rollback gate: four challenge endpoints marked EXTERNAL_GATED in the runbook and checklist.

3. Contracts

• Validate loopback API, loopback PostgreSQL, isolated database name, explicit confirmation and test mode before the first HTTP write.
• The authentication gate generates per-run JWT, pepper and field-encryption test keys; it must not inherit production values. Email, IM, WeCom and OIDC side effects are disabled or injected with a no-network test service.
• PID/log paths are unique per run. Cleanup may stop only the server spawned by the current run, and a database drop failure makes the gate fail.
• The runner removes only a database that this invocation successfully created; failed creation must never authorize deletion of an existing database.
• Deployment or rollback to old challenge code requires an operator to close /api/auth/forgot-password, /api/auth/reset-password, /api/auth/email-login/code and /api/auth/email-login/verify. Scripts do not claim this external action occurred.

4. Validation & Error Matrix

• Remote API/DB, wrong database prefix/confirmation or production mode -> fail before login, seed-user mutation or challenge requests.
• Code-display logging, enabled external adapter or inherited proxy hop -> fail the test-server guard.
• Concurrency value other than 0|1 -> fail; 0 -> BASE_ONLY/PARTIAL; 1 plus all PostgreSQL scenarios -> FULL.
• Owned server cleanup failure or temporary database drop failure -> non-zero gate result.
• Four-endpoint edge confirmation absent -> public release/rollback remains EXTERNAL_GATED, regardless of local test success.

5. Good/Base/Bad Cases

• Good: disposable loopback PostgreSQL, synthetic secrets, no-network auth email seam, five concurrency scenarios, zero residual databases and an automatically removed container.
• Base: the HTTP regression runs without PostgreSQL concurrency and reports partial evidence rather than a complete security pass.
• Bad: run several mutating HTTP steps and only then discover that the target database or API is remote.
• Bad: reuse a shared PID file or ignore dropdb failure while returning exit code zero.

6. Tests Required

• Static guard tests for remote authority, query-host override, exact database confirmation, secret isolation, external-adapter disablement, owned cleanup and redacted failure output.
• bash -n for the runner and regression scripts; Node syntax checks for the checker and test-server entry.
• Prisma validate, clean migration deployment, the full isolated gate and an explicit post-run query proving no hyb_oa_auth_* database remains.
• node deploy/trialrun/validate-env-contract.js must enforce loopback API binding, XFF overwrite templates and four-endpoint EXTERNAL_GATED wording.

7. Wrong vs Correct

Wrong

run_mutating_auth_regression
validate_target_afterwards
dropdb --if-exists "$DB" || true

Correct

validate_isolated_target_before_writes
run_mutating_auth_regression
drop_owned_database_or_fail

Scenario: Repository Local Candidate Gate

1. Scope / Trigger

• Trigger: declaring a checkout ready for internal deployment planning without deploying it or using public, production or real-account evidence.

2. Signatures

• Full gate: bash scripts/qa/run-local-candidate.sh full.
• Browser-only rerun: bash scripts/qa/run-local-candidate.sh focused.
• Parse-only check: bash scripts/qa/run-local-candidate.sh list.
• Route contract: scripts/qa/route-manifest.json.
• Result verifier: scripts/qa/verify-local-candidate-results.mjs.

3. Contracts

• full alone may print LOCAL_CANDIDATE_PASS; focused prints FOCUSED_BROWSER_PASS, and list prints LIST_ONLY.
• The gate accepts only loopback HTTP origins and canonical TCP ports. It uses local synthetic fixtures, excludes manual/live specs and does not consume historical Issue CSV as a candidate prerequisite.
• Every registered route is classified. Every core bundle has one stable, source-bound witness; candidate spec basenames are unique.
• Every selected Playwright result must pass. Missing, duplicate, unknown, skipped, multi-tag or wrong-source witnesses and any failed/skipped acceptance test make the gate fail.
• The script's full pass proves only its local scope. A task changing payment concurrency also requires the isolated PostgreSQL payment gate; deployment, public checks and real-device behavior remain separate states.

4. Validation & Error Matrix

• Non-loopback URL, credentialed URL, path/query target or invalid port -> fail before Vite/Playwright use.
• Unclassified/orphan route or missing core behavior/visual witness -> non-zero scanner result.
• Playwright suite error, unexpected, failed or skipped test -> non-zero result gate with basename-only redacted diagnostics.
• focused succeeds -> useful rerun evidence, but not a full candidate claim.

5. Good/Base/Bad Cases

• Good: full base checks pass, all source-bound witnesses and acceptance specs execute, and the runner prints one full-pass label.
• Base: use list before a long run to prove assets parse, while recording that no behavior has run.
• Bad: reuse a prior browser pass after parser changes, or report focused/list output as LOCAL_CANDIDATE_PASS.

6. Tests Required

• Negative tests for route gaps, manual/live evidence, duplicate basenames, invalid loopback inputs, witness result/source drift and mode labels.
• Current-checkout full run with zero skipped/unexpected results.
• For payment-state changes, run the disposable PostgreSQL gate and prove cleanup left no matching temporary databases.

7. Wrong vs Correct

Wrong

bash scripts/qa/run-local-candidate.sh list
echo LOCAL_CANDIDATE_PASS

Correct

bash scripts/qa/run-local-candidate.sh full
# Only this successful branch emits LOCAL_CANDIDATE_PASS.

Scenario: Compose-Resolved Loopback Host Port

1. Scope / Trigger

• Trigger: changing Compose port publication, deployment health checks, Caddy routing or operator commands for the server container.
• Purpose: support a non-default host port without changing the container's fixed application port or accidentally checking a stale service.

2. Signatures

• Env: APP_HOST_PORT=<1-65535>; default 3000.
• Compose mapping: 127.0.0.1:${APP_HOST_PORT:-3000}:3000.
• Runtime resolution: docker compose port server 3000 or the compatible docker-compose command selected by the deployment script.
• Backend container still listens on PORT=3000.

3. Contracts

• Publication remains loopback-only. APP_HOST_PORT changes only the host side; Caddy points to the same resolved host port.
• Deployment health checks must parse the running Compose mapping after startup; do not hardcode 3000 and do not assume .env variables entered the current shell merely because Compose read --env-file.
• The parsed port must be numeric before it is used in curl. A missing, multi-line or non-numeric mapping stops deployment before readiness success.
• server, worker and init retain the shared env-file contract; only server publishes an HTTP host port.
• Changing the real host port or Caddy target is production configuration and requires separate approval plus rollback evidence.

4. Validation & Error Matrix

• Missing APP_HOST_PORT -> Compose uses loopback port 3000.
• Valid alternate port -> Compose publishes it and deployment checks that port.
• Non-numeric/unresolved compose port output -> deployment exits non-zero.
• Caddy target differs from Compose mapping -> deployment is not accepted even if one local /ready endpoint answers.
• Host port already occupied -> Compose startup fails; do not start a second deployment on a guessed port.

5. Good/Base/Bad Cases

• Good: .env selects an alternate loopback port, Compose reports it, deploy checks the matching build SHA there and Caddy targets the same port.
• Base: default port 3000 remains compatible without extra configuration.
• Bad: Compose publishes 3001 while deploy/runbook curls 3000 and reports an old container as the new release.

6. Tests Required

• bash -n deploy/trialrun/deploy.sh deploy/trialrun/setup-server.sh.
• node deploy/trialrun/validate-env-contract.js must assert the exact loopback-only variable mapping and env example key.
• Review runbook/checklist commands for Compose resolution rather than shell expansion of an unloaded .env.
• Public deployment additionally proves Caddy, /ready and frontend marker use the approved build on the same resolved port.

7. Wrong vs Correct

Wrong

curl -fsS http://127.0.0.1:3000/ready

Correct

SERVER_HOST_PORT="$(docker compose port server 3000)"
SERVER_HOST_PORT="${SERVER_HOST_PORT##*:}"
curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"

Scenario: Compose Project And Data-Path Isolation

1. Scope / Trigger

• Trigger: changing Compose identity, host data mounts, database publication, deployment entrypoints or operational timers.
• Purpose: ensure one approved Compose project owns one verified DB/storage pair without fixed global container names or accidental shared PGDATA.

2. Signatures

• Every deploy/timer Compose command uses -p "$COMPOSE_PROJECT_NAME".
• PostgreSQL mount: ${POSTGRES_DATA_DIR:-./data/postgres}.
• Application storage mount: ${STORAGE_HOST_DIR:-./data/storage}.
• Formal deploy inputs: absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, DEPLOY_DATA_MODE=existing|initialize.
• Timer install inputs: COMPOSE_PROJECT_NAME, absolute POSTGRES_DATA_DIR, absolute STORAGE_HOST_DIR, full 40–64 hex BUILD_SHA and its exact 12-hex VITE_BUILD_SHA prefix.
• DB has no host ports; server remains loopback-only.

3. Contracts

• Compose files must not set container_name. Operational commands resolve services through the approved project, never a historical global name.
• A project-name change and its DB/storage path selection are one atomic, separately approved downtime operation. Never start two PostgreSQL instances against one PGDATA.
• Freeze the old project name from docker compose ls, not from a directory guess. Record immutable image IDs/digests, successful migration count, fixed business-table counts, resolved mounts and Caddy target before any write.
• Relative defaults preserve local trialrun compatibility, but production must use explicit absolute command inputs and verify them against active container mounts before any start, backup or migration.
• existing is the normal migration mode and refuses a missing PGDATA marker or storage directory. initialize is only for an approved empty first start; only that mode may create data directories.
• Remote readiness may use host curl, but build-SHA JSON validation runs with Node inside the server container; the Docker host does not need Node.
• The timer installer must validate the same non-root, non-overlapping absolute data-path shape as formal deploy, require an existing PG_VERSION and storage directory, and write both paths plus both build identifiers into the systemd service Environment= entries. Project name alone is insufficient: a later timer invocation would otherwise interpolate Compose defaults and operate against a non-active data pair.
• Timer installation may enable/start the timer unit, but verification must not manually start the daily service because that path can send real alerts. Verify the unit environment against active container mounts, then run Compose config --quiet and exec -T server true with the same values.
• For an existing cutover, stop the old PostgreSQL completely, copy PGDATA and storage into pre-verified empty absolute targets while preserving ownership, permissions, links and extended attributes, and retain the old directories untouched and unmounted for the observation period. Do not rename away the only pre-migration rollback data and do not copy a running PGDATA.

4. Validation & Error Matrix

• DB ports appears -> environment contract fails.
• Any fixed container_name appears -> environment contract fails.
• Required data mount loses its parameter -> environment contract fails.
• Deploy/timer omits -p -> environment contract fails.
• Timer install omits either data path/build identifier, receives nested paths, a PGDATA without PG_VERSION, a missing storage directory, an invalid full SHA or a 12-character value that is not its prefix -> fail before writing the systemd unit.
• A path-segment regex can match an empty string at the leading / -> contract failure; use explicit root, trailing slash, //, /./ and /../ checks so valid absolute paths remain installable.
• Formal deploy omits paths/mode, uses a relative/broad/overlapping path, or an existing PGDATA lacks PG_VERSION -> fail before DB start.
• New project resolves the same PGDATA while the old DB runs -> stop before Compose start; do not use service health as permission to continue.
• Old project name was inferred, immutable image ID or baseline counts are missing, or target copy is non-empty -> stop the cutover.

5. Tests Required

• node deploy/trialrun/validate-env-contract.js.
• The isolated backup/restore drill mutates Compose with a DB port and fixed name and proves the contract rejects it.
• bash -n covers deploy, timer, backup and restore scripts.
• Timer contract assertions cover all four explicit inputs, path isolation, PGDATA marker, SHA/prefix validation and the four systemd Environment= entries. On the deployment host, assert timer enabled/active, daily service inactive, exact active-environment equality and non-mutating Compose resolve.

6. Good/Base/Bad Cases

• Good: one approved project resolves one stopped-and-migrated PGDATA and the matching active storage path; deploy, timer and backup all use that identity.
• Good: the timer unit contains the current project, active mount sources and deployed full/short SHA pair; Compose config and a non-mutating server exec resolve under those exact values while the daily service stays inactive.
• Base: direct local Compose uses the documented relative defaults under one project; those defaults are never accepted by formal deploy.sh.
• Bad: start a renamed project before the old DB stops, or let it create a new empty directory and accept a green health check as proof of data continuity.
• Bad: move the only PGDATA into the new path, migrate it in place, then assume the old container name alone is a rollback point.
• Bad: reinstall the timer with only COMPOSE_PROJECT_NAME, allowing systemd Compose interpolation to fall back to repository-relative data directories.

7. Wrong vs Correct

Wrong

docker-compose up -d db
DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh

Correct

docker-compose -p "$COMPOSE_PROJECT_NAME" -f deploy/trialrun/docker-compose.yml \
  --env-file deploy/trialrun/.env up -d db
mapfile -t DB_CONTAINERS < <(docker-compose -p "$COMPOSE_PROJECT_NAME" \
  -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env ps -q db)
test "${#DB_CONTAINERS[@]}" -eq 1
DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \
  STORAGE_DIR_HOST=<verified-active-storage> ./scripts/backup-all.sh

For timer installation, keep the active deployment identity explicit:

# Wrong: project identity alone still permits relative data-path fallback.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

# Correct: use the verified active mounts and deployed SHA pair.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
POSTGRES_DATA_DIR=<verified-active-pgdata> \
STORAGE_HOST_DIR=<verified-active-storage> \
BUILD_SHA=<verified-full-build-sha> \
VITE_BUILD_SHA=<verified-12-char-prefix> \
  bash deploy/trialrun/install-warch3-ops-timer.sh install

Scenario: Paired Backup And Shadow-Only Restore

1. Scope / Trigger

• Trigger: changing or operating database/file backups, restore scripts, retention instructions or recovery-readiness evidence.
• Purpose: prevent a database-only recovery point, a mixed attachment restore or an accidental destructive restore into the active internal instance.

2. Signatures

• Paired backup: DB_TOOL_MODE=docker DB_CONTAINER=<compose-resolved-id> STORAGE_DIR_HOST=<verified-active-directory> BUILD_SHA=<actual-build-sha> SOURCE_IMAGE_DIGEST=<actual-sha256-image-id> ./scripts/backup-all.sh; at least one source identity is required.
• Optional safe batch identifier: BACKUP_RUN_ID=<safe-id>.
• Manifest writer: host node is preferred; host python3 is an equivalent fallback for Docker-only runtime machines.
• DB shadow restore requires RESTORE_SCOPE=shadow, RESTORE_CONFIRM_DATABASE=<actual-current-database> and DATABASE_URL.
• Storage shadow restore: RESTORE_SCOPE=shadow ./scripts/restore-storage.sh <tar> <empty-target>.
• Local synthetic drill: ./scripts/qa/run-backup-restore-drill.sh.
• DB_TOOL_MODE is auto, host or docker; use docker with Compose to avoid host/server PostgreSQL tool-version drift.

3. Contracts

• A complete recovery point is one batch directory containing database.dump, storage.tar.gz and a final manifest.json. The manifest contains only safe relative names, byte sizes, SHA-256 hashes, batch time, actual build SHA when known and actual source image digest when supplied.
• BUILD_SHA supplied by the caller takes precedence over local Git fallback. On a non-Git runtime host, at least SOURCE_IMAGE_DIGEST=sha256:<64-hex> is mandatory. Never label a legacy backup with the commit planned for deployment.
• Manifest generation uses exclusive creation and preserves the same schema under Node and Python 3; if neither runtime exists, fail before publishing the final batch directory.
• For an approved live backup, do not assume the deployment .env still matches the running database. Confirm that the selected PostgreSQL container's database and user match at least one active application DATABASE_URL, then pass the verified runtime values without logging their contents.
• Docker backup/restore has no default container name. Resolve the unique DB container with compose -p <approved-project> ... ps -q db and fail when the result is empty or ambiguous. Storage backup likewise requires an explicit STORAGE_DIR_HOST; it must never infer STORAGE_DIR or a repository default.
• The DB dump must pass pg_restore --list and the storage archive must pass tar -tzf before the manifest may say complete.
• DB restore queries current_database() and compares it exactly with RESTORE_CONFIRM_DATABASE before pg_restore --clean.
• The actual DB name must also contain a separate shadow, drill or restore_test marker; setting RESTORE_SCOPE=shadow alone is not trust.
• Storage restore requires an explicit empty target and rejects absolute paths, .. path segments, symbolic links and hard links before extraction.
• Current scripts intentionally reject production restore. A real incident requires an approved Trellis task, stop window, paired batch, two responsible people, rollback point and post-restore /ready/permission checks.
• Copy the whole completed batch to approved off-host/off-disk storage; a batch left only on the application host is not accepted recovery evidence.
• A real-format shadow restore compares successful migration count and a fixed set of business-table counts with the pre-backup baseline, compares extracted regular-file count with the active storage source, and runs /ready using a schema-compatible image. /ready alone is not recovery evidence.
• When shadow SQL is supplied through stdin or a heredoc, docker exec must use -i and psql must use ON_ERROR_STOP=1; otherwise an empty result can be mistaken for a data mismatch even though no verification query ran.
• Compare fixed table counts as a keyed set, not by database-returned row order. Require the expected unique key count, then normalize both table=count collections with LC_ALL=C sort (or an equivalent map comparison) before equality checking; database collation order is not an evidence contract.

4. Validation & Error Matrix

• One backup component fails or integrity listing fails -> remove the temporary batch; do not write a complete manifest.
• Existing batch identifier -> fail without overwriting.
• Missing/non-shadow restore scope -> fail before any restore write.
• Actual DB name differs from confirmation -> fail before pg_restore --clean.
• Confirmed DB name lacks a shadow/drill/restore-test marker -> fail before pg_restore --clean.
• Storage target is implicit, root/repository root or non-empty -> fail before extraction.
• Corrupt dump/tar or dangerous/link archive member -> fail before restore.
• Host PostgreSQL tools mismatch in auto mode -> docker fallback is allowed only when the explicit target container and required DB variables exist.
• Neither Node nor Python 3 is available -> fail manifest generation and remove the temporary batch; do not publish a partial recovery point.
• Invalid build SHA/image digest, or neither identity available on a non-Git host -> fail before DB or storage backup starts.
• Deployment .env, PostgreSQL container and active application connection do not identify the same database/user -> stop the live backup; never guess credentials or expose values in diagnostics.
• Shadow migration count, any fixed table count, attachment regular-file count or compatible-image /ready differs from baseline -> reject the recovery point and do not enter production cutover.
• Empty, duplicate or missing table-count output -> classify as a verification execution failure first; do not report production-data drift until stdin was delivered and every expected key was returned exactly once.

5. Good/Base/Bad Cases

• Good: one command creates a validated pair, the complete batch is copied off host, and a disposable shadow drill restores the DB marker and attachment hash.
• Good: a Docker-only host uses Python 3 to write the same manifest contract after the active application connection is matched to the database container.
• Good: a legacy container without /ready.data.buildSha records its inspected sha256: image ID and leaves buildSha null rather than inventing provenance.
• Good: a heredoc query uses docker exec -i ... psql -v ON_ERROR_STOP=1, then compares the complete keyed count set under deterministic C sorting.
• Base: backup succeeds locally but off-host copy and deployed /ready evidence remain explicitly pending; do not claim formal recovery readiness.
• Bad: source a stale deployment .env and treat a failed dump as a prompt to guess or print credentials.
• Bad: source the active trial .env and run a cleaning restore directly, or restore attachments into a non-empty live directory.

6. Tests Required

• Shell syntax checks for all backup/restore scripts.
• Synthetic Docker/PostgreSQL and temporary-storage round trip.
• Negative assertions for wrong DB confirmation, corrupt DB/tar, non-empty target and dangerous tar path.
• Validate manifest relative paths, absence of connection secrets and matching SHA-256 values, explicit build-SHA precedence and image-digest preservation.
• Run the synthetic drill with Node available and with a PATH that excludes Node but includes Python 3; both paths must complete the same round trip.
• For real-format count checks, assert that stdin queries return the full unique key set, and prove that a different database collation/order does not change the pass result while a changed count still fails.
• Real-instance restore and /ready remain separately approved evidence and must never be fabricated from the local drill.

7. Wrong vs Correct

Wrong

source deploy/trialrun/.env
./scripts/backup-all.sh
./scripts/restore-db.sh some.dump
./scripts/restore-storage.sh some.tar.gz deploy/trialrun/data/storage

Correct

./scripts/qa/run-backup-restore-drill.sh
# Before an approved live backup, compare the selected DB container identity
# with an active app connection using redacted boolean diagnostics.
DB_TOOL_MODE=docker DB_CONTAINER=<verified-container> \
  STORAGE_DIR_HOST=<verified-active-storage> \
  SOURCE_IMAGE_DIGEST=<verified-sha256-image-id> ./scripts/backup-all.sh
# Real restore remains an approved incident operation after shadow evidence.

For a containerized count check, the input and comparison contract is:

# Wrong: stdin is not attached, and raw row order becomes part of the gate.
actual="$(docker exec shadow-db psql -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts ORDER BY table_key;
SQL
)"
test "$actual" = "$expected"

# Correct: deliver stdin, fail on SQL errors and compare keyed sets deterministically.
actual="$(docker exec -i shadow-db psql -v ON_ERROR_STOP=1 -At -F = <<'SQL'
SELECT table_key, row_count FROM shadow_counts;
SQL
)"
test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
  "$(printf '%s\n' "$expected" | LC_ALL=C sort)"

Scenario: Daily Restricted Off-Host Backup

1. Scope / Trigger

• Trigger: installing or changing automatic paired backups, off-host transfer, retention, freshness alerts or quarterly restore reminders.
• Purpose: keep a 24-hour recovery point without giving the production host ordinary shell or delete access to the ARM archive.

2. Signatures

• Production runner: scripts/run-automatic-backup.sh with run, check-stale and remind-drill modes.
• ARM forced receiver: scripts/receive-offhost-backup.sh.
• Installers: install-automatic-backup-timer.sh and install-offhost-backup-receiver.sh.
• Local contract: bash scripts/qa/run-automatic-backup-contract.sh.
• Production required environment: REMOTE_DIR, REMOTE_ENV_FILE, COMPOSE_PROJECT_NAME, POSTGRES_DATA_DIR, STORAGE_HOST_DIR, BUILD_SHA, VITE_BUILD_SHA, AUTOMATIC_BACKUP_DIR, AUTOMATIC_BACKUP_STATE_DIR, OFFHOST_TARGET, OFFHOST_SSH_KEY and OFFHOST_KNOWN_HOSTS.
• Receiver config: independent absolute BACKUP_INBOX_DIR, BACKUP_ARCHIVE_DIR, BACKUP_WEEKLY_DIR, RRSYNC_BIN, PYTHON_BIN, DAILY_KEEP and WEEKLY_KEEP.

3. Contracts

• Resolve one active Compose server/database pair and verify its build, PGDATA, storage mount and database identity before backup. Do not use fixed container names or repository-relative runtime data.
• Transfer only a complete auto-<UTC timestamp> three-file batch to ARM inbox. The forced key allows write-only rrsync or an exact finalize command; it rejects shell, sender/read and destructive rsync options.
• ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256, then atomically moves the batch to archive. Archive and weekly directories are outside the rsync root; an archived name is never overwritten.
• Write production last-success.json and prune source copies only after ARM finalize succeeds. Keep seven automatic source/ARM daily points and four ARM weekly checkpoints; unrelated manual batches are outside automatic retention.
• Daily, freshness and quarterly reminder timers are separate. Alert summaries contain categories only. AUTO_BACKUP sends the first failure, worsening/new findings and the first quarterly reminder, while suppressing an identical repeated failure or reminder.
• Installer install is disabled-by-default; enable, real transfer, real WeCom alert and all server/account/key changes remain EXTERNAL_GATED.

4. Validation & Error Matrix

• Source mount/build/project drift, lock conflict or incomplete backup -> fail before success state and source retention.
• Transfer/finalize failure, wrong/extra/link artifact, bad hash or second batch for one UTC day -> leave no off-host success state and preserve source batch.
• Missing, stale or project/build-mismatched success record -> watchdog finding.
• Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver rejection without changing archive.
• Host/network/WeCom all unavailable at once -> self-hosted alert cannot prove its own outage; this is a documented physical boundary, not a green state.

5. Good/Base/Bad Cases

• Good: one verified active source creates a complete batch, write-only rsync resumes inside inbox, ARM finalize verifies and archives it, then production writes success and prunes old automatic source points.
• Base: local implementation and synthetic contracts pass while both server installers remain disabled and explicitly externally gated.
• Bad: give production a normal ARM login key, put archive below inbox, accept a new host fingerprint automatically, or prune local recovery points after rsync but before target-side finalize.

6. Tests Required

• Shell syntax plus run-automatic-backup-contract.sh using temporary files and mocked Docker/SSH boundaries; it must never contact real hosts.
• Focused WeCom alert tests for stdin path redaction, preserved AUTO_BACKUP issue identity, first/repeated failure and quarterly reminder behavior.
• Environment contract assertions for all four scripts, three timers, explicit project/data/build/SSH binding and disabled-by-default installation.
• Real ARM setup, first paired batch, failure alert and quarterly shadow restore are separately approved operational evidence.

7. Wrong vs Correct

Wrong

rsync -a <batch>/ backup-host:<archive>/
rm -rf <old-local-batches>

Correct

# Install only with explicit active identity; default remains disabled.
<explicit-project-data-build-and-ssh-env> \
  bash deploy/trialrun/install-automatic-backup-timer.sh install
# The runner writes success and prunes only after restricted target finalize.
bash scripts/run-automatic-backup.sh run