From cba74cf97d3ff80cd061f1b095680b5cccd1b02c Mon Sep 17 00:00:00 2001 From: shunli Date: Tue, 28 Jul 2026 07:54:37 +0800 Subject: [PATCH] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E8=87=AA=E5=8A=A8=E5=A4=87?= =?UTF-8?q?=E4=BB=BD=E8=BF=90=E8=A1=8C=E8=BA=AB=E4=BB=BD=E6=BC=82=E7=A7=BB?= =?UTF-8?q?=EF=BC=8807-28-automatic-backup-runtime-identity-failure?= =?UTF-8?q?=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .trellis/spec/hyb-oa/deployment-trialrun.md | 31 +++++++++- .../check.jsonl | 1 + .../design.md | 37 +++++++++++ .../implement.jsonl | 1 + .../implement.md | 37 +++++++++++ .../prd.md | 62 +++++++++++++++++++ .../task.json | 26 ++++++++ deploy/trialrun/checklist.md | 1 + deploy/trialrun/runbook.md | 3 + 9 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/check.jsonl create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/design.md create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.jsonl create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.md create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/prd.md create mode 100644 .trellis/tasks/07-28-automatic-backup-runtime-identity-failure/task.json diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index d689b490..8eb83cdd 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -1005,6 +1005,13 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ Compose-project-mismatched state. A fresh recovery point with a different build SHA is `PASS_WITH_FINDINGS`, because deploy-time code identity and recoverable data freshness are separate facts. +- Automatic-backup units freeze the active Compose project, build, PGDATA and + storage mount when installed. After any deployment, rollback or direct + container rebuild that changes the active container identity or host mounts, + compare all frozen values with the unique running server/database pair. If + the units are already installed and any value differs, rerun the existing + installer with the complete current contract and recheck all identities + before declaring the deployment complete; do not patch one unit in place. - Installer `install` is disabled-by-default; `enable`, real transfer, real WeCom alert and all server/account/key changes remain `EXTERNAL_GATED`. @@ -1027,6 +1034,10 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ authority -> preserve all objects and fail. - Missing, corrupt, stale, future or project-mismatched success record -> hard watchdog failure. Fresh build mismatch -> one finding without a failing exit. +- `phase_stale_check_failed` can also mean the watchdog failed its runtime + identity precheck before reading the success record. Diagnose the nested safe + category and active-unit identity first; do not infer that the last archive is + stale or corrupt from this outer phase alone. - Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without printing the file value. Identical alert state -> update/evaluate without send. - Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver @@ -1040,12 +1051,16 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ resumes inside inbox, ARM finalize verifies and archives it, then production writes success and prunes old automatic source points. If production later stops, ARM independently raises one dead-man alert and one recovery notice. +- Good: a deployment rebuilds the server with a different storage mount, then + refreshes the installed backup units through the installer and proves every + frozen identity matches before closing the deployment. - Base: local implementation and synthetic contracts pass while both server installers remain disabled and explicitly externally gated. - Bad: send through `compose exec server`, hard-code a personal employee number, persist a robot URL in `.env`/unit/repository, give production a normal ARM login key, put archive below inbox, accept a new host fingerprint automatically, - or prune local points before target-side finalize. + prune local points before target-side finalize, or leave installed backup + units bound to a pre-deployment mount until the next scheduled alert. ### 6. Tests Required @@ -1068,6 +1083,10 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file binding, disabled-by-default installation and absence of a literal complete robot URL in tracked or untracked repository files. +- Deployment and rebuild review must assert that installed production units + still match the unique active project, build, PGDATA and storage mount. A + changed mount with otherwise healthy `/ready` is not a completed deployment + until installer refresh, timer state and runtime identity all pass. - Real ARM setup, first paired batch, deliberate source-timer stop or transfer failure, dead-man first/repeated/recovery behavior and quarterly shadow restore are separately approved operational evidence. A direct robot send test alone @@ -1078,6 +1097,10 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ #### Wrong ```bash +# Rebuild the application, observe /ready, but leave installed backup units +# frozen to the pre-deployment runtime identity. +docker-compose up -d --build server + compose exec server send-alert --recipient= rsync -a / backup-host:/ docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ... @@ -1086,6 +1109,12 @@ docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" db pg_dump ... #### Correct ```bash +# After a deployment/rebuild, compare the installed unit contract with the +# unique active server/database pair. If any frozen identity changed, refresh +# all units through the installer before verifying timers and identity again. + \ + bash deploy/trialrun/install-automatic-backup-timer.sh enable + # Install only with explicit active identity and a root-only Webhook file path; # default remains disabled. \ diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/check.jsonl b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/check.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/check.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/design.md b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/design.md new file mode 100644 index 00000000..869e6406 --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/design.md @@ -0,0 +1,37 @@ +# 自动备份运行身份失败修复设计 + +## 1. Root Cause + +自动备份安装时把附件宿主目录冻结进 systemd unit。后续 server 容器重建使用了另一实际宿主目录，但 unit 未同步刷新。runner 在 `runtime_identity` 阶段比较 Docker 实际 mount source 与 unit 的 `STORAGE_HOST_DIR`，发现不一致后在 DB dump、附件归档和传输前失败关闭。 + +严格检查本身正确：两个目录均存在、非符号链接、realpath 和 inode 不同，因此不能通过路径规范化或放宽校验修复。 + +## 2. Safe Refresh + +从 daily unit 读取现有环境合同，从唯一运行 server 容器解析实际附件 mount source，仅在内存中替换 `STORAGE_HOST_DIR`。验证所有路径、权限、build 对应关系后，调用仓库安装器 `enable` 原子重写三个 service/timer 文件并 daemon-reload；写入前保存三个 services/timers 的 root-only 临时副本用于失败回滚，成功验证后删除副本。 + +不在命令输出、task 或文档中记录实际路径、容器 ID、连接串或 webhook。 + +## 3. Validation Sequence + +```text +local 73-check contract + -> production read-only recheck + -> snapshot six unit files + -> installer enable with actual storage mount + -> six-dimension runtime identity check + -> manual daily service start + -> verify new success state + ARM finalize + recovery alert state + -> verify timers and unrelated monitors + -> remove temporary unit snapshot +``` + +## 4. Rollback + +- 安装器或 unit 验证失败：恢复六个 unit 快照、daemon-reload、恢复原 timer 状态；不运行备份。 +- unit 正确但补跑备份后续阶段失败：不恢复旧 `STORAGE_HOST_DIR`，因为旧值已证实错误；保留新配置、失败摘要和本地诊断，按具体 phase 另行修复。 +- 新批次只有在 ARM finalize 后才更新 success state 和触发本地保留策略，失败不会删除现有最后成功恢复点。 + +## 5. Prevention + +把“部署后 active mount 与自动备份 unit 合同一致”写入部署/试运行规约和 checklist。涉及 data path 的部署不能只验证应用 `/ready`；若自动备份 units 已安装，必须在交付结束前核对或刷新冻结的 project/build/PGDATA/storage 身份。 diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.jsonl b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.md b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.md new file mode 100644 index 00000000..1ca48ee2 --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/implement.md @@ -0,0 +1,37 @@ +# 自动备份运行身份失败实施清单 + +## 1. Pre-fix + +- [x] 确认 daily service 02:30 非零退出且告警类别为 `phase_runtime_identity_failed`。 +- [x] 按 runner 的 Compose 选择逻辑确认唯一根因为附件宿主挂载不匹配。 +- [x] 确认不是 symlink/realpath/inode 等价，其他五项 runtime identity 通过。 +- [x] 确认最近成功恢复点约 11 小时，尚未超过 watchdog/dead-man 窗口。 +- [x] 确认 07:10 `phase_stale_check_failed` 由 watchdog 复用同一 runtime identity 检查触发，不是独立的归档损坏结论。 +- [x] 运行本地 `bash scripts/qa/run-automatic-backup-contract.sh`（73 项通过）。 + +## 2. Unit Refresh + +- [x] 重新只读解析当前实际附件 mount source并验证安全边界。 +- [x] 保存六个自动备份 unit 的 root-only 临时快照。 +- [x] 沿用现有 unit 合同，仅替换实际附件宿主目录，调用安装器 `enable`。 +- [x] 验证 daily/watchdog/drill timers enabled/active 和六项 runtime identity 全部匹配。 + +## 3. Recovery Backup + +- [x] 手工启动 daily service；记录稳定 phase/result，不输出原始日志。 +- [x] 验证新的 last-success 时间、项目/build 身份和 manifest 哈希格式。 +- [x] 通过仅在 ARM finalize 成功后写入的 success state 验证新归档闭环；普通 ARM 账号无 archive 读取权限，未扩大权限取证。 +- [x] 验证 alert state 为 `PASS_AUTOMATIC_BACKUP_COMPLETE`。 +- [x] 用户确认群内只收到一条 `PASS_AUTOMATIC_BACKUP_COMPLETE / healthy` 恢复通知。 +- [x] 验证应用健康 timer、备份 watchdog/drill 和 ARM dead-man/receiver 不受影响。 + +## 4. Cleanup and Governance + +- [x] 删除临时 unit 快照；不删除旧目录、旧批次或失败证据。 +- [x] 更新部署规约/checklist 的自动备份 unit 漂移门禁。 +- [x] 运行 task validate、governance、sensitive、docs audit 和 `git diff --check`；全部无错误，docs audit 仅保留既有 22 项未登记文档 warning。 + +## 5. Rollback + +- unit 刷新或 identity 复检失败：恢复六个 unit 快照并 daemon-reload，不补跑备份。 +- 补跑在 paired/offhost phase 失败：保留已纠正 unit，依据安全 phase 诊断继续处理，不回退旧附件目录。 diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/prd.md b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/prd.md new file mode 100644 index 00000000..7f39e146 --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/prd.md @@ -0,0 +1,62 @@ +# 自动备份运行身份失败 + +## Goal + +恢复每日自动成对备份，使生产 unit 冻结的存储路径重新与当前运行容器一致，补跑并完成一次数据库、附件和 ARM 异机归档闭环，同时保留失败告警的脱敏证据。 + +## Scope + +- 只读确认 02:30 `phase_runtime_identity_failed` 的精确失败维度和最近成功恢复点。 +- 使用当前运行 server 容器的实际附件挂载源刷新自动备份三个 systemd services/timers；其他已验证环境值沿用现有 unit，禁止打印路径或秘密。 +- 刷新前后验证 active server/db 数量、运行状态、构建 SHA、附件挂载、PGDATA 和数据库身份。 +- 手工运行一次 daily backup，要求本地完整批次、ARM finalize、success state 和一条恢复通知全部成功。 +- 保持应用健康监控、备份 receiver、ARM dead-man 和业务服务不变。 + +## Confirmed Facts + +- 2026-07-28 02:30 daily service 以非零状态失败，脱敏状态为 `FAIL_AUTOMATIC_BACKUP / phase_runtime_identity_failed`。 +- 生产机使用 standalone `docker-compose`；最初诊断命令误用 `docker compose`，已按 runner 相同选择逻辑更正，未把诊断工具错误当作根因。 +- 当前 server/db 各唯一且运行中；unit 冻结的构建 SHA、PGDATA、应用/数据库身份均与运行容器匹配。 +- 唯一不匹配项是 `STORAGE_HOST_DIR`：unit 配置目录和当前容器实际挂载源都存在，但 realpath/inode 均不同，不是符号链接或路径规范化差异。 +- unit 早于当前 server 容器启动，说明部署/重建后容器使用了新附件目录，而自动备份 unit 仍保留旧目录。 +- 最近一次成功异机恢复点约 11 小时前，项目和构建身份均匹配；当前未超过 24/26 小时窗口，但本次 02:30 没有产生新恢复点。 +- runtime identity 阶段不写 `last-error.log`，因此没有原始诊断残留；脚本按设计在任何备份写入前失败关闭。 +- 2026-07-28 07:10 watchdog 又发送 `phase_stale_check_failed`。`check-stale` 在读取 success state 前同样调用 `resolve_active_runtime`，因此该消息由同一个附件挂载漂移触发；它不是第二个独立故障，也不证明最后成功批次本身损坏或过期。 + +## Requirements + +- R1：刷新前再次从运行容器解析唯一实际附件挂载源，要求安全绝对目录、存在、非符号链接，且不得输出路径。 +- R2：不得直接手改 unit 单行；使用已提交安装器以完整合同重写 daily/watchdog/drill units，并保持三个 timers enabled/active。 +- R3：除 `STORAGE_HOST_DIR` 改为当前实际挂载源外，Compose project、PGDATA、build、远端目标、SSH、webhook 和备份目录等合同必须与现有 unit 一致。 +- R4：刷新后先只读复跑六项 runtime identity；任何一项不匹配均不得补跑备份。 +- R5：手工补跑必须使用现有 runner，完成数据库 dump、附件归档、manifest、受限传输和 ARM finalize 后才写 success state。 +- R6：成功后应把当前告警状态（最新类别为 `phase_stale_check_failed`）切换为 `PASS_AUTOMATIC_BACKUP_COMPLETE` 并发送一次恢复通知；相同健康状态不得重复发送。 +- R7：不得删除旧成功批次、旧目录或失败状态；正常保留策略只在新批次已由 ARM finalize 后运行。 +- R8：如果 unit 刷新失败，恢复原 unit 文件并 daemon-reload；如果补跑失败，保留刷新后的正确 identity 配置和脱敏失败状态，不回退到旧附件目录。 + +## Acceptance Criteria + +- [x] AC1：根因证据明确为非规范化的附件宿主目录漂移，其他 runtime identity 维度全部匹配。 +- [x] AC2：三个自动备份 timer 仍为 enabled/active，刷新后的 daily unit 与当前附件挂载、PGDATA、build、项目和数据库身份全部匹配。 +- [x] AC3：手工 daily backup 成功生成并在 ARM finalize 一个新的完整批次，生产 success state 时间更新。 +- [x] AC4：群内只收到一次自动备份恢复通知，状态为成功；无秘密、路径或原始日志进入消息/任务证据。 +- [x] AC5：应用健康 timer、ARM backup dead-man 和备份 receiver 未受影响。 +- [x] AC6：补充防复发合同：部署后若容器宿主挂载变化，必须刷新/核对自动备份 unit，不能等次日告警发现。 + +## Out of Scope + +- 修改数据库、附件内容、OA/Caddy、备份保留数量或 ARM 归档结构。 +- 删除旧附件目录或判断其业务处置。 +- 重置数据库、恢复数据、改 webhook、SSH key 或真实业务数据。 +- 将路径、容器 ID、数据库身份、连接串或原始日志写入文档/聊天。 + +## Test Plan + +- 刷新前后运行脱敏 runtime identity 矩阵。 +- 运行本地自动备份 73 项合同，确保安装器/helper/runner 行为无回归。 +- 生产补跑后检查 service result、last-success 新鲜度、ARM 新批次 finalize、alert state 和三个 timers。 +- 运行 task validate、Trellis governance、敏感内容扫描、文档审计和 `git diff --check`。 + +## Key Decision + +立即修复 unit 的附件路径并补跑备份，不修改严格相等检查。两个目录不是同一 inode，放宽为 realpath 或“目录存在即可”会掩盖真实数据源漂移。 diff --git a/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/task.json b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/task.json new file mode 100644 index 00000000..16df7f0f --- /dev/null +++ b/.trellis/tasks/07-28-automatic-backup-runtime-identity-failure/task.json @@ -0,0 +1,26 @@ +{ + "id": "automatic-backup-runtime-identity-failure", + "name": "automatic-backup-runtime-identity-failure", + "title": "自动备份运行身份失败", + "description": "诊断 02:30 自动备份 phase_runtime_identity_failed 告警，确认影响范围、根因与安全修复路径。", + "status": "in_progress", + "dev_type": "operations", + "scope": "deployment", + "package": "hyb-oa", + "priority": "P2", + "creator": "shunli", + "assignee": "shunli", + "createdAt": "2026-07-28", + "completedAt": null, + "branch": null, + "base_branch": "main", + "worktree_path": null, + "commit": null, + "pr_url": null, + "subtasks": [], + "children": [], + "parent": null, + "relatedFiles": [], + "notes": "02:30 daily 与 07:10 watchdog 均由自动备份 unit 的旧附件宿主目录漂移触发，其他运行身份正常。已用现有安装器刷新 units，三个 timer 与全部运行身份通过；手工补跑、ARM finalize、成功状态、单条群恢复通知和关联监控复核通过，临时快照已删除。73 项合同及治理、敏感、流程、文档和格式门禁无错误，全部验收项闭环。", + "meta": {} +} diff --git a/deploy/trialrun/checklist.md b/deploy/trialrun/checklist.md index a32d1292..2058a9de 100644 --- a/deploy/trialrun/checklist.md +++ b/deploy/trialrun/checklist.md @@ -93,6 +93,7 @@ - [ ] 公网 live smoke（只读，不做写操作）：`bash scripts/qa/run-web-gates.sh smoke-live` - [ ] 若本地已配置 `LIVE_SMOKE_*`，确认 `bash deploy/trialrun/deploy.sh` 输出中包含 live smoke 结果 - [ ] 已确认 release gate / live smoke 结果为通过；关键链路不再作为每次发布的手工必做项 +- [ ] 若自动备份 units 已安装，已脱敏核对其冻结的 Compose project、build、PGDATA、附件宿主挂载与当前唯一活动 server/db；任一项变化均已用现有安装器按完整合同刷新，三个 timer 仍为 enabled/active，未直接手改 unit - [ ] 查看 server 日志（无异常错误风暴）： - [ ] `docker-compose -p "$COMPOSE_PROJECT_NAME" -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env logs --tail=200 server` - [ ] 已确认 smoke 账号为独立账号（`smoke-admin` / `smoke-employee`），且不参与真实业务 diff --git a/deploy/trialrun/runbook.md b/deploy/trialrun/runbook.md index b02bc20f..96ad4039 100644 --- a/deploy/trialrun/runbook.md +++ b/deploy/trialrun/runbook.md @@ -236,6 +236,8 @@ manifest 必须至少记录一个真实来源标识：活动 `/ready` 有构建 - 生产与 ARM 都直接使用独立企微运维群机器人，不依赖 OA server、数据库或个人工号。Webhook 正文只由用户手工写入两台主机的 root-only `0600` 文件；仓库、普通 `.env`、systemd unit、日志和聊天只能出现受控文件路径，不能出现 URL 或 key。 - 失败告警只包含阶段、状态、finding 数和类别建议；原始命令诊断只保留在源端状态目录的 `0600` 最近失败日志。同一失败降噪；首次失败、状态恶化、新 finding、恢复通知和每季度首次恢复提醒才发送。新部署构建与最近恢复点 build SHA 不同只记 finding，缺失、损坏、超龄、未来时间或 Compose project 不同仍硬失败。 +自动备份 unit 会冻结安装时的 Compose project、build、PGDATA 和附件宿主挂载。每次发布、回滚或直接重建容器后，若这些 units 已安装，必须把 unit 中的四项身份与唯一活动 server/db 逐项脱敏核对；任一项变化都要使用现有安装器和完整当前合同刷新 units，再复核三个 timer 与全部运行身份，不能直接手改单个 unit，也不能等次日定时告警后再处理。`phase_stale_check_failed` 是 watchdog 外层阶段，它也可能表示读取成功状态前的运行身份检查失败；应先看安全子类别和 unit/活动容器身份，不能仅凭该阶段认定归档已经超龄或损坏。 + 创建机器人后，由用户分别登录两台主机，在 root shell 中用静默输入把完整 Webhook 写入受控路径；不要把值作为命令参数、环境文件内容或聊天消息交给 Codex： ```bash @@ -431,6 +433,7 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT##*:}/ready" 重建后必须补做： - `smoke-external-deps.js` - `smoke-trialrun-runtime.js` +- 若自动备份 units 已安装，核对其 Compose project、build、PGDATA、附件宿主挂载与当前唯一活动 server/db；不一致时用安装器完整刷新并复核三个 timer - 公网页面视觉检查：至少覆盖 `/invoice-pool`、`/reimbursements`、`/mobile/capture` 的 390px 视口 发布/回滚 Checklist：