增强(08-23-isolated-backup-restore-drill): 固化八阶段RTO合同

.trellis/spec/hyb-oa/deployment-trialrun.md

@@ -1330,6 +1330,13 @@ validate_copied_batch(private/batch, source_metadata)
 - Exit-code fields are non-negative integers or `NOT_RUN`;
   `pg_restore_error_count` additionally permits `UNKNOWN` when no safe numeric
   summary is available.
+- Required data-restore timing fields:
+  `rto_batch_copy_seconds`, `rto_postgres_container_start_seconds`,
+  `rto_dump_list_seconds`, `rto_target_confirmation_seconds`,
+  `rto_restore_apply_seconds`, `rto_database_verify_seconds`,
+  `rto_storage_attachment_verify_seconds`, `rto_final_integrity_seconds` and
+  `data_restore_rto_seconds`. A measured value is a non-negative integer;
+  an unstarted stage is `NOT_RUN`.
 - Tracked supply-chain entries:
   `scripts/qa/offhost-shadow-restore-runner.template.sh`,
   `scripts/qa/validate-offhost-shadow-restore-attachments.py` and
@@ -1349,6 +1356,17 @@ validate_copied_batch(private/batch, source_metadata)
   environment values or stderr.
 - Run `dump_list` and target confirmation independently before the existing
   restore script. A failed stage leaves every later stage `NOT_RUN`.
+- Measure the eight data-restore stages in this exact continuous order:
+  `batch_copy` -> `postgres_container_start` -> `dump_list` ->
+  `target_confirmation` -> `restore_apply` -> `database_verify` ->
+  `storage_attachment_verify` -> `final_integrity`. One timestamp closes the
+  current stage and starts the next, so no elapsed time is hidden between
+  fields. The total is computed from the measured stage fields and must equal
+  their exact sum.
+- On failure, finalize the active stage and emit the completed/current numeric
+  durations plus later `NOT_RUN` fields before runtime teardown. A pre-copy
+  failure emits `NOT_RUN` for the total and every stage. Do not emit the total
+  a second time on the success-only summary path.
 - For the official PostgreSQL image, target readiness must use a real query over
   container-local TCP loopback (`psql -h 127.0.0.1`), not default Unix-socket
   `pg_isready`. The initdb temporary server listens only on its socket, while
@@ -1399,7 +1417,8 @@ validate_copied_batch(private/batch, source_metadata)
   `DUMP_LIST_NONZERO` / `dump_list_failed`.
 - Bounded TCP target queries remain non-zero -> `target_confirmation=FAIL`,
   apply `NOT_RUN`, category `TARGET_QUERY_NONZERO`; preserve the last numeric
-  exit, attempt count and zero success streak.
+  exit, attempt count and zero success streak. All query retries and their
+  one-second intervals accrue only to `rto_target_confirmation_seconds`.
 - Target query succeeds but the exact shadow name differs -> fail immediately
   with `target_name_match=no`, category `TARGET_NAME_MISMATCH`; do not retry a
   confirmed wrong target.
@@ -1411,6 +1430,10 @@ validate_copied_batch(private/batch, source_metadata)
   attachment checks still decide the final recovery result.
 - Diagnostics emitted only after cleanup, or cleanup residue non-zero -> the
   attempt is non-clean and cannot be reported as `GO`.
+- A measured stage is negative/non-numeric, a completed stage is `NOT_RUN`, a
+  later stage is numeric after an earlier failure, or the stage sum differs
+  from `data_restore_rto_seconds` -> runner contract failure; do not render or
+  upload the candidate.
 - Missing/extra render key, newline/shell syntax, unsafe path/name, malformed
   digest or pre-existing output -> render fails without publishing a partial
   candidate.
@@ -1434,6 +1457,12 @@ validate_copied_batch(private/batch, source_metadata)
   migration and attachment verification.
 - Good: Unix-socket availability alone does not pass; loopback TCP becomes
   available, two exact target queries succeed consecutively, then apply runs.
+- Good: an apply failure records numeric copy/container/list/target/apply
+  durations, leaves database/storage/final verification `NOT_RUN`, emits their
+  exact partial sum, and only then tears down the private runtime.
+- Base: a successful run emits all eight numeric durations once and their sum
+  exactly equals `data_restore_rto_seconds`; the value remains a data-only
+  exercise RTO, not a complete OA recovery RTO.
 - Bad: emit only `database_restore_failed` after deleting the sole private log.
 - Bad: parse and publish arbitrary pg_restore lines, or treat a non-zero exit as
   success because the restored database appears partly populated.
@@ -1450,6 +1479,15 @@ validate_copied_batch(private/batch, source_metadata)
   with an equivalent test implementation.
 - Inject list failure, target mismatch and list-pass/apply-failure. Assert exact
   stage enums, later-stage `NOT_RUN`, execution path and numeric exit code.
+- Drive the rendered runner's timing functions with deterministic timestamps.
+  Assert all eight successful stage durations and their exact total; inject
+  target-confirmation and restore-apply failures and assert completed/current
+  numeric values, later `NOT_RUN`, one total field and timing output before the
+  runtime cleanup marker.
+- Statically assert that the bounded TCP confirmation call is enclosed by the
+  `dump_list -> target_confirmation` and
+  `target_confirmation -> restore_apply` timing transitions; a retry sleep
+  must not be charged to container start or restore apply.
 - Extract the target-confirmation function from the rendered runner and drive
   it with: socket-only availability/TCP failure, first-query recovery, streak
   reset, exact-name mismatch and permanent bounded failure. Assert every probe
@@ -1485,6 +1523,7 @@ validate_copied_batch(private/batch, source_metadata)
 fixture: wait for two queries
 runner: first socket pg_isready -> continue
 restore-db > private.log || fail database_restore_failed
+RTO = finish - start; stage gaps are inferred later
 finally: delete private runtime
 print result=NO_GO
 ```
@@ -1494,8 +1533,9 @@ print result=NO_GO
 ```text
 tracked template + exact private config -> deterministic candidate + SHA-256
 contract drives rendered TCP target function -> list -> target -> real restore path
-capture only stage/path/exit/category enums
-emit fixed diagnostics -> finally teardown raw private runtime
+continuous stage transitions -> exact stage sum + one data-only RTO
+capture only stage/path/exit/category/duration allowlists
+finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime
 non-zero apply remains FAIL
 ```
 

.trellis/tasks/08-23-isolated-backup-restore-drill/implement.md

@@ -130,14 +130,16 @@
 - [x] 最新批次来源与当前生产构建相同，探针时年龄 `64625` 秒且不超过 24 小时；run id 配对、DB/附件大小与 SHA-256、source digest 均通过。
 - [x] 来源/当前 migration 目录为 `65/65`、差异 `0`；仓库 HEAD 相对生产构建的应用代码漂移文件数 `0`。
 - [x] 探针 root 入口与普通用户暂存残留均为 `0`；按已验证私有配置取得的 dead-man/健康 timer 实际 unit 状态为 `active/active`。
-- [ ] RTO 八阶段实现尚未批准，因此未渲染 Attempt 5 最终候选、未生成最终候选 SHA-256，也未执行实际恢复。
+- [x] 本次探针完成时 RTO 八阶段实现尚未批准，因此当时未渲染 Attempt 5 最终候选、未生成最终候选 SHA-256，也未执行实际恢复；后续本地实现授权单独追加，不改写该时点事实。
 
-### Attempt 5 前 RTO 分阶段提案（待单独批准实现）
+### Attempt 5 前 RTO 分阶段实施（已批准仅本地实现）
 
 - [x] 设计口径已固定为总 RTO 加八段串行秒数：批次复制、容器启动、dump list、目标确认、restore apply、migration/12 类表、附件恢复与关联校验、最终源/副本复核。
-- [ ] runner 增加固定 allowlist 阶段字段，成功时八段均为非负整数且精确加总为 `data_restore_rto_seconds`；失败时已完成段和当前失败段保留数字，后续段为 `NOT_RUN`。
-- [ ] 合同注入各阶段成功/失败，验证最长 TCP 轮询只进入目标确认阶段、后续 `NOT_RUN`、总数精确加总、输出不含路径/SQL/凭据；原有清理与恢复合同继续通过。
-- [ ] 未经单独批准，不修改 runner、重新渲染 Attempt 5 候选或执行实际恢复。
+- [x] runner 增加固定 allowlist 阶段字段，成功时八段均为非负整数且精确加总为 `data_restore_rto_seconds`；失败时已完成段和当前失败段保留数字，后续段为 `NOT_RUN`。
+- [x] 合同注入各阶段成功/失败，验证最长 TCP 轮询只进入目标确认阶段、后续 `NOT_RUN`、总数精确加总、输出不含路径/SQL/凭据；原有清理与恢复合同继续通过。
+- [x] 用户批准的当前边界仅为本地修改 runner、更新可复跑合同并执行回归；不连接 ARM、不重新渲染 Attempt 5 现场候选、不执行实际恢复。
+- [ ] 本轮八阶段计时与完整合同通过后，下一份满足 24 小时门禁的新鲜批次直接进入 Attempt 5 的候选渲染、SHA-256 核对、只读探针和实际恢复审批流程。此后发现的改进项，除非会导致 Attempt 5 失败或产生错误结论，否则只登记为后续任务，不再作为 Attempt 5 前置增强插队。
+- [x] 八阶段状态必须在失败路径清理前输出；阶段边界沿用 `dump_list`、`target_confirmation`、`restore_apply` 等既有诊断词汇，不建立第二套同义命名。
 
 ## 验证命令类别
 

.trellis/tasks/08-23-isolated-backup-restore-drill/research/restore-drill-evidence.md

@@ -452,3 +452,17 @@ Attempt 4 前后探针的最新构建短标识、run id 配对、产物完整性
 timer 状态只采用已验证私有配置中的实际 unit 名只读取得；一次使用猜测名称得到的无效读数已当场撤销，未进入证据结论。探针结果表明该最新批次继续满足构建、新鲜度、身份和完整性前置门禁，但仍不是恢复成功证明。
 
 Attempt 5 最终候选尚未渲染。原因是用户提出的 RTO 分阶段记录已进入设计、但 runner 实现尚未另行批准；若先渲染，后续计时代码会改变候选字节和 SHA-256。必须先裁决并完成 RTO 八阶段本地合同，再从跟踪模板重新确定性渲染候选并核对 SHA-256，实际恢复仍须单独批准。
+
+## Attempt 5 前八阶段 RTO 本地合同
+
+日期：2026-08-25
+
+用户随后批准的范围仅为本地实现八阶段 RTO 计时并执行合同回归；本阶段未连接 ARM、未渲染现场候选、未生成 Attempt 5 候选 SHA-256，也未执行实际恢复。此前探针批次的 `64625` 秒新鲜度只属于探针时点，后续不得复用，实际恢复前必须重新选择最新正式批次并重跑只读身份门禁。
+
+runner 现在以连续边界依次记录 `batch_copy`、`postgres_container_start`、`dump_list`、`target_confirmation`、`restore_apply`、`database_verify`、`storage_attachment_verify` 和 `final_integrity`。同一个时间点关闭前一段并开启后一段；成功路径八段均为非负整数且精确加总为唯一的 `data_restore_rto_seconds`。失败路径先结算当前阶段，保留已完成和当前阶段的数字，后续阶段为 `NOT_RUN`，然后才清理私有 runtime。最长 60 次 TCP 目标确认及其 1 秒间隔只位于 `target_confirmation` 边界内。
+
+可复跑合同直接驱动 rendered runner 的同一计时函数。确定性成功夹具得到八段 `3/5/2/11/7/13/17/3` 秒，总计 `61` 秒且总字段只输出一次；目标确认失败夹具保留 `2/4/1/9` 秒、后四段 `NOT_RUN`、部分总计 `16` 秒，并证明计时输出位于 runtime cleanup 之前；restore apply 失败夹具记录当前段 `12` 秒、后续三段 `NOT_RUN`、部分总计 `28` 秒。捕获输出不含路径、连接串、凭据或 SQL。
+
+完整本地门禁结果：runner 合同全部 `PASS`；真实 PostgreSQL 16 fixture 和未修改的 `scripts/restore-db.sh` Docker stdin/host-file 双路径继续通过；既有成对备份/恢复演练 `PASS`；自动异机备份合同 `74/74 PASS`；task validate、Trellis governance、process baseline、docs registry、敏感扫描和 `git diff --check` 通过。文档审计错误 `0`，保留既有未登记文件 warning；本地测试容器残留 `0`，`scripts/restore-db.sh` 相对生产构建仍无字节漂移。
+
+Attempt 5 前增强停止线从本节起生效：八阶段计时完成并通过合同后，下一份满足 24 小时门禁的新鲜批次直接进入最终候选渲染、SHA-256 核对、只读探针和实际恢复审批。此后发现的改进项，除非会导致 Attempt 5 失败或产生错误结论，否则只登记为后续任务，不再作为 Attempt 5 前置增强插队。本地合同通过仍只证明演练器合同，不回答最新正式备份是否可恢复。