From 706941eb5cfdee9ce0186a41d577d92043c76292 Mon Sep 17 00:00:00 2001 From: shunli Date: Sun, 26 Jul 2026 20:41:22 +0800 Subject: [PATCH] =?UTF-8?q?=E5=8A=A0=E5=9B=BA=E5=A4=87=E4=BB=BD=E6=81=A2?= =?UTF-8?q?=E5=A4=8D=E6=9C=AC=E5=9C=B0=E5=9F=BA=E7=BA=BF=EF=BC=8807-26-int?= =?UTF-8?q?ernal-ops-recovery-baseline=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .trellis/spec/hyb-oa/deployment-trialrun.md | 92 +++++++++++ .../check.jsonl | 1 + .../design.md | 61 +++++++ .../implement.jsonl | 1 + .../implement.md | 35 ++++ .../prd.md | 77 +++++++++ .../task.json | 37 +++++ deploy/trialrun/checklist.md | 14 +- deploy/trialrun/runbook.md | 66 +++++--- scripts/backup-all.sh | 93 +++++++++++ scripts/backup-db.sh | 51 +++++- scripts/backup-storage.sh | 24 ++- scripts/qa/run-backup-restore-drill.sh | 150 ++++++++++++++++++ scripts/restore-db.sh | 83 ++++++++-- scripts/restore-storage.sh | 60 +++++-- server/README.md | 8 +- 16 files changed, 792 insertions(+), 61 deletions(-) create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/check.jsonl create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/design.md create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/implement.jsonl create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/prd.md create mode 100644 .trellis/tasks/07-26-internal-ops-recovery-baseline/task.json create mode 100755 scripts/backup-all.sh create mode 100755 scripts/qa/run-backup-restore-drill.sh diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index 8afb0f85..ab07cc3a 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -516,3 +516,95 @@ echo LOCAL_CANDIDATE_PASS bash scripts/qa/run-local-candidate.sh full # Only this successful branch emits LOCAL_CANDIDATE_PASS. ``` + +## Scenario: Paired Backup And Shadow-Only Restore + +### 1. Scope / Trigger + +- Trigger: changing or operating database/file backups, restore scripts, + retention instructions or recovery-readiness evidence. +- Purpose: prevent a database-only recovery point, a mixed attachment restore or + an accidental destructive restore into the active internal instance. + +### 2. Signatures + +- Paired backup: `DB_TOOL_MODE=docker DB_CONTAINER= ./scripts/backup-all.sh`. +- Optional safe batch identifier: `BACKUP_RUN_ID=`. +- DB shadow restore requires `RESTORE_SCOPE=shadow`, + `RESTORE_CONFIRM_DATABASE=` and `DATABASE_URL`. +- Storage shadow restore: + `RESTORE_SCOPE=shadow ./scripts/restore-storage.sh `. +- Local synthetic drill: `./scripts/qa/run-backup-restore-drill.sh`. +- `DB_TOOL_MODE` is `auto`, `host` or `docker`; use `docker` with Compose to + avoid host/server PostgreSQL tool-version drift. + +### 3. Contracts + +- A complete recovery point is one batch directory containing `database.dump`, + `storage.tar.gz` and a final `manifest.json`. The manifest contains only safe + relative names, byte sizes, SHA-256 hashes, batch time and optional build SHA. +- The DB dump must pass `pg_restore --list` and the storage archive must pass + `tar -tzf` before the manifest may say `complete`. +- DB restore queries `current_database()` and compares it exactly with + `RESTORE_CONFIRM_DATABASE` before `pg_restore --clean`. +- The actual DB name must also contain a separate `shadow`, `drill` or + `restore_test` marker; setting `RESTORE_SCOPE=shadow` alone is not trust. +- Storage restore requires an explicit empty target and rejects absolute paths, + `..` path segments, symbolic links and hard links before extraction. +- Current scripts intentionally reject production restore. A real incident + requires an approved Trellis task, stop window, paired batch, two responsible + people, rollback point and post-restore `/ready`/permission checks. +- Copy the whole completed batch to approved off-host/off-disk storage; a batch + left only on the application host is not accepted recovery evidence. + +### 4. Validation & Error Matrix + +- One backup component fails or integrity listing fails -> remove the temporary + batch; do not write a complete manifest. +- Existing batch identifier -> fail without overwriting. +- Missing/non-shadow restore scope -> fail before any restore write. +- Actual DB name differs from confirmation -> fail before `pg_restore --clean`. +- Confirmed DB name lacks a shadow/drill/restore-test marker -> fail before + `pg_restore --clean`. +- Storage target is implicit, root/repository root or non-empty -> fail before + extraction. +- Corrupt dump/tar or dangerous/link archive member -> fail before restore. +- Host PostgreSQL tools mismatch in `auto` mode -> docker fallback is allowed + only when the explicit target container and required DB variables exist. + +### 5. Good/Base/Bad Cases + +- Good: one command creates a validated pair, the complete batch is copied off + host, and a disposable shadow drill restores the DB marker and attachment hash. +- Base: backup succeeds locally but off-host copy and deployed `/ready` evidence + remain explicitly pending; do not claim formal recovery readiness. +- Bad: source the active trial `.env` and run a cleaning restore directly, or + restore attachments into a non-empty live directory. + +### 6. Tests Required + +- Shell syntax checks for all backup/restore scripts. +- Synthetic Docker/PostgreSQL and temporary-storage round trip. +- Negative assertions for wrong DB confirmation, corrupt DB/tar, non-empty + target and dangerous tar path. +- Validate manifest relative paths, absence of connection secrets and matching + SHA-256 values. +- Real-instance restore and `/ready` remain separately approved evidence and + must never be fabricated from the local drill. + +### 7. Wrong vs Correct + +#### Wrong + +```bash +source deploy/trialrun/.env +./scripts/restore-db.sh some.dump +./scripts/restore-storage.sh some.tar.gz deploy/trialrun/data/storage +``` + +#### Correct + +```bash +./scripts/qa/run-backup-restore-drill.sh +# Real restore remains an approved incident operation after shadow evidence. +``` diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/check.jsonl b/.trellis/tasks/07-26-internal-ops-recovery-baseline/check.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/check.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/design.md b/.trellis/tasks/07-26-internal-ops-recovery-baseline/design.md new file mode 100644 index 00000000..77b31b1c --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/design.md @@ -0,0 +1,61 @@ +# 内部 OA 运行与恢复底线设计 + +## 总体边界 + +本任务将现有四个底层脚本保留为单一职责工具,在其上增加一个成对备份入口和一个隔离演练入口。所有默认值面向本地/影子环境;真实环境只能沿 runbook 的人工门禁执行。 + +```text +backup pair entry + ├─ backup-db.sh → .db.dump + ├─ backup-storage.sh → .storage.tar.gz + └─ integrity checks → .manifest + +isolated drill + ├─ disposable PostgreSQL/database + ├─ disposable source/restore directories + ├─ synthetic marker + synthetic attachment + └─ restore → compare → cleanup exact targets +``` + +## 备份合同 + +- 一个批次只使用一个 `run_id`,由上层入口传递明确的 `OUT_FILE` 给现有底层脚本。 +- 先把产物写入批次临时目录;DB 和附件均成功并通过完整性检查后,再写 manifest 并把批次标记为完整。 +- manifest 使用稳定、脱敏的键:`schemaVersion`、`runId`、`createdAt`、`buildSha`(如可安全取得)、两份相对文件名、字节数和 SHA-256。 +- 不把连接串、容器密码、存储源路径、真实业务文件清单或主机绝对路径写入 manifest。 +- 本任务不负责异机传输实现;runbook 要求操作者把完整批次复制到异机/异盘,并记录保留周期。 + +## DB 恢复安全合同 + +- 恢复前通过数据库连接读取 `current_database()`,不得只相信 URL 字符串或调用者描述。 +- 调用者必须提供与实际数据库名一致的确认值,并显式选择 `shadow`。 +- 实际目标库名还必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记,不能仅靠调用者把模式写成 `shadow`。 +- 确认缺失、连接失败、数据库名不匹配或模式不是 `shadow` 时,在 `pg_restore --clean` 之前退出。 +- Docker fallback 也必须执行同样的数据库名探测和确认,不得成为绕过路径。 +- 真实/生产恢复不在本任务脚本默认路径内;未来如确需开放,应另建任务定义双人确认和事故编号。 + +## 附件恢复安全合同 + +- 必须显式传入恢复目标,不再默认落入试运行数据目录。 +- 明确拒绝仓库的默认试运行附件目录,即使该目录碰巧为空。 +- 影子恢复目标必须为空;非空时退出,不执行覆盖或合并。 +- 解压前列出归档成员并拒绝绝对路径、空成员和包含 `..` 路径段的成员。 +- 只对已验证的归档执行解压;演练通过内容 SHA-256 比较证明往返一致。 + +## 隔离演练 + +- 数据库只使用一次性本地容器或明确命名的 disposable database,插入不含业务信息的 marker 表/行。 +- 附件只使用临时目录和合成文本/二进制样件。 +- 演练记录开始/结束时间、各阶段通过状态和相对产物名;不记录连接串、容器密码或临时绝对路径。 +- 清理只针对演练创建的明确容器、数据库和临时目录;不得使用宽泛路径或工作区根目录。 + +## 兼容与回滚 + +- `backup-db.sh` 和 `backup-storage.sh` 的现有单独调用继续可用;推荐运维入口改为成对备份命令。 +- `restore-db.sh`/`restore-storage.sh` 的安全收紧属于有意的命令合同变化,runbook 和 checklist 必须同步更新。 +- 回滚代码可恢复旧脚本,但已经生成的 dump/tar 仍保持 PostgreSQL/tar 标准格式,不被新 manifest 锁定。 +- 如演练环境缺少 PostgreSQL 客户端和 Docker,停止 DB 动态演练;仍可完成 shell 静态检查和附件演练,不得转向真实环境补测。 + +## 外部门禁 + +以下事项不因本地演练通过而自动获批:读取真实运行环境、复制真实备份、建立异机定时任务、停机、恢复真实库/附件、部署和公网 `/ready`。进入这些步骤前再次列出目标、只读/写入范围和回滚方式并取得明确批准。 diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.jsonl b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.jsonl new file mode 100644 index 00000000..9dd3234a --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.jsonl @@ -0,0 +1 @@ +{"_example": "Fill with {\"file\": \"\", \"reason\": \"\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."} diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md new file mode 100644 index 00000000..8379b011 --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/implement.md @@ -0,0 +1,35 @@ +# 内部 OA 运行与恢复底线执行计划 + +## 实施步骤 + +- [x] 审计现有备份/恢复脚本、部署卷、runbook 和 checklist,冻结现有兼容入口与风险点。 +- [x] 增加 DB + 附件成对备份入口、统一批次号、完整性检查、SHA-256 和脱敏 manifest。 +- [x] 收紧 DB 恢复:实际数据库名探测、影子模式和精确确认必须在破坏性命令前通过。 +- [x] 收紧附件恢复:显式目标、空目录门禁和 tar 路径安全检查。 +- [x] 增加隔离合成数据演练,覆盖成功往返及关键负向场景。 +- [x] 更新 runbook/checklist 和部署规约,区分普通备份、影子演练和真实事故恢复。 +- [x] 运行任务测试计划和 Trellis 治理门禁;按实际结果记录本地 RTO,不伪造 `/ready` 或异机证据。 +- [ ] 在单独授权后再执行真实运行来源、职责分离、异机副本和 `/ready` 验收。 + +## 推荐文件边界 + +- 复用并按需修改:`scripts/backup-db.sh`、`scripts/backup-storage.sh`、`scripts/restore-db.sh`、`scripts/restore-storage.sh`。 +- 最多新增两个小脚本:一个成对备份入口、一个 QA-only 隔离演练入口;不创建通用备份框架。 +- 更新现有 `deploy/trialrun/runbook.md`、`deploy/trialrun/checklist.md` 和 `.trellis/spec/hyb-oa/deployment-trialrun.md`,不新建常驻平行业务文档。 + +## 验证命令 + +- `bash -n scripts/backup-db.sh scripts/backup-storage.sh scripts/restore-db.sh scripts/restore-storage.sh scripts/backup-all.sh scripts/qa/run-backup-restore-drill.sh` +- `bash scripts/qa/run-backup-restore-drill.sh` +- `node deploy/trialrun/validate-env-contract.js` +- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-26-internal-ops-recovery-baseline` +- `node scripts/qa/check-trellis-governance.mjs` +- `node scripts/qa/check-sensitive-content.mjs` +- `git diff --check` + +## 回滚点 + +- 成对入口失败:保留四个底层脚本,不写完整 manifest。 +- DB/附件安全门禁与现有操作冲突:先修 runbook 和调用方,不放宽默认门禁。 +- 动态演练环境不可用:记录缺失依赖并停止,不使用真实数据替代。 +- 任何命令解析到生产、远程或含真实数据的目标:立即停止并请求明确授权。 diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/prd.md b/.trellis/tasks/07-26-internal-ops-recovery-baseline/prd.md new file mode 100644 index 00000000..a49e8a96 --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/prd.md @@ -0,0 +1,77 @@ +# 内部OA运行与恢复底线 + +## Goal + +为 5–10 人内部 OA 建立一个主要开发者可执行的备份与恢复底线:数据库和附件始终成对备份、产物可校验、恢复默认只能进入明确的影子目标,并用合成数据证明可恢复;同时把真实内部实例的运行来源、角色职责、异机副本和 `/ready` 验收留在明确的外部授权门后。 + +## Background + +- 当前可信状态仍是 `LOCAL_CANDIDATE_PASS`,不能据此宣称已部署或可以正式内用。 +- 已有 `backup-db.sh`、`backup-storage.sh`、`restore-db.sh` 和 `restore-storage.sh`,runbook 也描述了 DB + 文件恢复,但尚无当前恢复演练证据。 +- DB 恢复会执行 `pg_restore --clean --if-exists`;当前脚本没有影子目标确认,误指向真实库会产生破坏性后果。 +- 文件恢复会直接解压到目标目录;当前不会拒绝非空目录,可能把旧文件和恢复文件混合。 +- DB 与附件备份分别生成时间戳,没有共同批次清单、完整性校验和,无法稳定证明两份产物属于同一次恢复点。 +- 本任务以小团队低维护为目标,不引入备份平台、对象存储、Kubernetes、云厂商锁定或复杂灾备编排。 + +## Scope + +- 本地备份/恢复 shell 脚本的安全合同、成对备份入口、完整性校验与脱敏 manifest。 +- 使用一次性 PostgreSQL 目标和临时附件目录完成合成数据恢复演练及负向测试。 +- 更新现有试运行 runbook、checklist 和部署规约,明确影子恢复与真实恢复边界。 +- 规划并保留真实运行 commit、访问边界、角色职责、异机副本及恢复后 `/ready` 的外部授权门;未授权前不执行这些动作。 + +## Requirements + +- 提供一个简单的成对备份入口,复用现有 DB/附件脚本,以同一批次标识产生两份产物和脱敏 manifest。 +- manifest 只包含批次号、相对文件名、大小、SHA-256、创建时间及安全的版本信息,不包含数据库 URL、Secret、真实附件名或完整绝对路径。 +- 备份完成前验证 DB dump 可列出、附件 tar 可读取;任一部分失败时不得产出“完整批次”结论。 +- DB 恢复必须先解析并显示目标数据库名,并要求调用者显式确认相同目标;默认仅允许 `shadow` 演练模式,且实际库名必须带 `shadow`、`drill` 或 `restore_test` 独立标记。本任务不开放生产恢复快捷开关。 +- 附件恢复必须要求显式目标目录;默认拒绝非空目录,并在解压前检查归档条目,禁止绝对路径和 `..` 越界路径。 +- 用完全隔离的合成数据库和合成附件完成本地往返演练,不读取或复制真实发票、合同、银行资料或生产数据。 +- 恢复演练至少证明:DB 标记数据恢复、附件内容哈希一致、损坏备份被拒绝、错误确认值被拒绝、非空附件目标被拒绝。 +- 更新现有 runbook/checklist,使正常备份、影子演练、真实恢复三种场景不再共用含糊命令;保持现有本地存储默认,不启用 MinIO。 +- 将真实内部实例核查列为后续同任务外部门禁:运行 commit/访问方式/TLS、真实职责分配、异机副本和恢复后 `/ready` 均需另行批准和脱敏留证。 + +## Acceptance Criteria + +- [x] 一条命令生成同批 DB、附件和脱敏 manifest,且完整性校验通过。 +- [x] DB 恢复在缺少或不匹配目标确认时失败,不会执行 `pg_restore --clean`。 +- [x] 附件恢复拒绝隐式目标、非空目录和危险 tar 路径。 +- [x] 隔离合成数据演练证明 DB 与附件可往返恢复,并记录实际耗时作为本地 RTO 参考。 +- [x] runbook/checklist 明确影子演练命令、生产停止条件、DB + 附件配对要求和异机副本要求。 +- [x] 相关 shell 语法、QA 脚本、Trellis 治理、敏感扫描和 `git diff --check` 通过。 +- [ ] 在获得单独授权后,确认真实运行 commit、访问边界、职责分离、异机备份及恢复后 `/ready`;未授权前保持未勾选且不宣称正式可用。 + +## Out of Scope + +- 不连接生产数据库,不读取真实附件,不执行远程恢复、部署或公网检查。 +- 不修改 Prisma schema、业务 API、权限模型、审批链或付款状态机。 +- 不引入 MinIO、云备份 SDK、全量灾备平台、自动跨地域复制或 24×7 值守。 +- 不在仓库提交备份文件、恢复数据、环境变量、凭证、真实账号或完整本地路径。 +- 不在本任务完成企微调休、导航、报销或模块退役;它们属于后续批次。 + +## Test Plan + +- `bash -n scripts/backup-db.sh scripts/backup-storage.sh scripts/restore-db.sh scripts/restore-storage.sh <新增脚本>` +- 使用临时目录和一次性本地 PostgreSQL 目标运行合成数据恢复演练;若本机缺少 PostgreSQL/Docker,则记录为环境阻塞,不改用真实库。 +- 覆盖损坏 dump/tar、危险 tar 路径、非空目标目录、缺少确认和确认不匹配等负向场景。 +- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-26-internal-ops-recovery-baseline` +- `node scripts/qa/check-trellis-governance.mjs` +- `node scripts/qa/check-sensitive-content.mjs` +- `git diff --check` + +## Key Decision + +先改进可恢复性和防误操作,再接触真实环境。默认工作流只支持影子恢复;真实恢复不是普通脚本路径,而是需要停机、负责人确认、成对备份、回滚点和独立授权的事故操作。 + +## Local Evidence + +- 已使用一次性 PostgreSQL 16 容器和合成附件完成多轮往返演练,最近一次本地参考 RTO 为 6 秒。 +- 演练覆盖 manifest/校验和、错误确认、正式风格库名、损坏 DB、损坏 tar、非空附件目录、默认试运行附件目录和危险 tar 路径拒绝。 +- 该证据不包含真实业务数据、真实运行实例、异机副本或部署后 `/ready`,因此最后一项验收保持未完成。 + +## Notes + +- Keep `prd.md` focused on requirements, constraints, and acceptance criteria. +- Lightweight tasks can remain PRD-only. +- For complex tasks, add `design.md` for technical design and `implement.md` for execution planning before `task.py start`. diff --git a/.trellis/tasks/07-26-internal-ops-recovery-baseline/task.json b/.trellis/tasks/07-26-internal-ops-recovery-baseline/task.json new file mode 100644 index 00000000..5a90976e --- /dev/null +++ b/.trellis/tasks/07-26-internal-ops-recovery-baseline/task.json @@ -0,0 +1,37 @@ +{ + "id": "internal-ops-recovery-baseline", + "name": "internal-ops-recovery-baseline", + "title": "内部OA运行与恢复底线", + "description": "为5至10人内部OA建立低维护、可验证、防误操作的数据库与附件成对备份恢复基线,并明确真实环境授权门。", + "status": "in_progress", + "dev_type": null, + "scope": "backup-restore-safety-local-drill-operational-baseline", + "package": null, + "priority": "P0", + "creator": "shunli", + "assignee": "shunli", + "createdAt": "2026-07-26", + "completedAt": null, + "branch": null, + "base_branch": "main", + "worktree_path": null, + "commit": null, + "pr_url": null, + "subtasks": [], + "children": [], + "parent": null, + "relatedFiles": [ + "scripts/backup-db.sh", + "scripts/backup-storage.sh", + "scripts/backup-all.sh", + "scripts/restore-db.sh", + "scripts/restore-storage.sh", + "scripts/qa/run-backup-restore-drill.sh", + "deploy/trialrun/runbook.md", + "deploy/trialrun/checklist.md", + "server/README.md", + ".trellis/spec/hyb-oa/deployment-trialrun.md" + ], + "notes": "本地阶段完成:成对备份、脱敏 manifest、影子库/空目录恢复门禁和合成数据演练已通过,最近本地参考 RTO 6 秒;真实运行来源、职责分离、异机副本和部署后 /ready 仍待单独授权。", + "meta": {} +} diff --git a/deploy/trialrun/checklist.md b/deploy/trialrun/checklist.md index c02d31fa..a3d1c9e2 100644 --- a/deploy/trialrun/checklist.md +++ b/deploy/trialrun/checklist.md @@ -33,8 +33,9 @@ - [ ] 全量回归(建议):`cd server && RUN_AUDIT_REGRESSION=1 RUN_OCR_REGRESSION=1 ./scripts/regression-all.sh` - [ ] 备份(DB + 文件存储): - [ ] `set -a && source deploy/trialrun/.env && set +a` - - [ ] `./scripts/backup-db.sh`(记录输出文件名) - - [ ] `./scripts/backup-storage.sh`(记录输出文件名) + - [ ] `DB_TOOL_MODE=docker DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh` + - [ ] 同一批次包含 `database.dump`、`storage.tar.gz`、`manifest.json`,两份 SHA-256 校验通过 + - [ ] 完整批次已复制到获批异机/异盘,并记录保留周期 ## B. 发布(Deploy) - [ ] 本地工作树 clean,目标 commit 已进入 upstream;部署日志中的 commit 与 `/ready.data.buildSha` 一致 @@ -71,8 +72,8 @@ - [ ] **EXTERNAL_GATED**:若回滚目标仍包含旧认证挑战逻辑,人工确认边缘继续关闭 `/api/auth/forgot-password`、`/api/auth/reset-password`、`/api/auth/email-login/code`、`/api/auth/email-login/verify`;脚本不会自动改边缘,未确认不得回滚开放,直到安全版本恢复 - [ ] 停止 server:`docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env stop server` -- [ ] 恢复 DB:`./scripts/restore-db.sh ` -- [ ] 恢复存储:`./scripts/restore-storage.sh deploy/trialrun/data/storage` +- [ ] 恢复 DB/存储属于获批事故操作;当前脚本仅允许 `RESTORE_SCOPE=shadow`,不得绕过后直接指向真实目标 +- [ ] 已先在独立数据库和空附件目录完成同批次影子恢复,数据库名精确确认且 manifest/SHA-256 匹配 - [ ] 回到上个版本:`git checkout ` - [ ] 重启并探活: - [ ] `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d --build server` @@ -111,8 +112,9 @@ - [ ] 管理员账号仍可进入工作区设置、人员、组织、流程定义、系统设置、功能开关和审计日志。 - [ ] 已确认不删除备用模块数据;恢复某模块只通过工作区设置或功能开关完成。 - [ ] 已生成 `week-01.md` 到 `week-04.md` 的观测计划,至少完成第 1 周导出。 -- [ ] 已完成 DB + 文件存储备份,并记录输出文件名。 -- [ ] 已在同机或临时环境完成一次恢复演练,恢复后 `/ready` 通过。 +- [ ] 已完成 DB + 文件存储同批次备份,manifest/SHA-256 通过并复制到异机/异盘。 +- [ ] 已运行 `./scripts/qa/run-backup-restore-drill.sh` 完成本地合成数据演练。 +- [ ] 已在获批临时环境完成真实格式影子恢复,恢复后 `/ready` 通过。 - [ ] 已核对管理员账号清单和 smoke 账号,自动化不使用真实业务账号。 - [ ] 已执行 `cd server && npm audit --omit=dev --audit-level=high` 并完成高危/严重漏洞评审;若仍失败,不进入公网正式试点。 - [ ] 已执行 `bash scripts/qa/run-release-gate.sh r66-trialrun`;若数据库或公网账号条件不足,已记录手工替代步骤。 diff --git a/deploy/trialrun/runbook.md b/deploy/trialrun/runbook.md index 0e89fa33..fc2b5bcc 100644 --- a/deploy/trialrun/runbook.md +++ b/deploy/trialrun/runbook.md @@ -164,8 +164,8 @@ npx playwright test tests/e2e/manual/live-invoice-ocr.spec.ts --workers=1 --repo 2. 执行 `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d server`。 3. 在 Issue/发布记录中写明降级原因、预计恢复时间和影响范围。 -## 1.5 备份与恢复演练(强烈建议至少做一次) -> 备份建议做到“DB + 文件存储(票据上传目录)”双份齐全。 +## 1.5 备份与恢复演练(正式启用前必须完成) +> 数据库和文件存储必须使用同一批次成对备份;只有两份产物、完整性检查和 manifest 全部成功,才算一个可用恢复点。 ### 1.5.1 备份 ```bash @@ -174,47 +174,66 @@ set -a source deploy/trialrun/.env set +a -./scripts/backup-db.sh -./scripts/backup-storage.sh +DB_TOOL_MODE=docker \ +DB_CONTAINER=hyb-oa-trialrun-db \ +./scripts/backup-all.sh ``` -输出默认在: -- `deploy/trialrun/backups/` +输出默认在 `deploy/trialrun/backups/hyb-oa-backup-/`,其中包含: + +- `database.dump` +- `storage.tar.gz` +- `manifest.json`(相对文件名、大小和 SHA-256,不含连接串或源目录) + +整批复制到获批的异机/异盘位置并记录保留周期;不能只复制 DB 或附件中的一份。 + +### 1.5.2 本地隔离恢复演练 + +开发机具备 Docker、PostgreSQL 客户端和 Node.js 时,使用合成数据运行: -### 1.5.2 恢复(到同一台机器或新机器) -1) 先停止 server: ```bash -docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env stop server +./scripts/qa/run-backup-restore-drill.sh ``` -2) 恢复 DB: +该命令创建一次性 PostgreSQL 容器和临时附件目录,验证 DB/附件往返、manifest 校验和、错误数据库确认、损坏备份、非空目标目录和危险 tar 路径,结束后只清理本次创建的明确目标。它不读取 `deploy/trialrun/.env`,也不接触真实数据。 + +手工影子恢复必须使用独立数据库和空附件目录,并显式确认实际数据库名: + ```bash -set -a -source deploy/trialrun/.env -set +a - -./scripts/restore-db.sh deploy/trialrun/backups/ +RESTORE_SCOPE=shadow \ +RESTORE_CONFIRM_DATABASE= \ +DB_TOOL_MODE=docker \ +DB_CONTAINER= \ +DATABASE_URL= \ +POSTGRES_DB= \ +POSTGRES_USER= \ +POSTGRES_PASSWORD= \ +./scripts/restore-db.sh deploy/trialrun/backups//database.dump ``` -3) 恢复文件存储: ```bash -./scripts/restore-storage.sh deploy/trialrun/backups/ deploy/trialrun/data/storage +RESTORE_SCOPE=shadow \ +./scripts/restore-storage.sh \ + deploy/trialrun/backups//storage.tar.gz \ + ``` -4) 重启并验收: -```bash -docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d server -curl -fsS http://127.0.0.1:3000/ready -``` +`restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记;数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。 -### 1.5.3 R66 小团队上线前演练记录 +### 1.5.3 真实事故恢复 + +真实库/附件恢复不是普通脚本路径。必须另建或更新获批 Trellis task,记录停机窗口、恢复批次、实际目标、两名负责人、当前备份、回滚点和验证清单。当前恢复脚本不会接受 `production` 模式,不得通过修改环境变量或脚本临时绕过。恢复后必须验证 `/ready`、运行 commit、核心权限和 DB/附件关联完整性。 + +### 1.5.4 小团队上线前演练记录 上线前至少记录一次演练结果: - 演练日期: - 执行人: - DB 备份文件: - 文件存储备份文件: +- manifest 与 SHA-256 校验: - 恢复目标机器或目录: +- 本地合成数据 RTO: - 恢复后 `/ready` 结果: - 上个可回滚 tag: - 回滚负责人: @@ -223,6 +242,7 @@ curl -fsS http://127.0.0.1:3000/ready - 提供或确认真实 `deploy/trialrun/.env`,特别是 `JWT_SECRET`、`APP_ORIGIN`、`COOKIE_SECURE`、`OPENAPI_DOCS_ENABLED`。 - 确认备份文件保存位置和保留周期。 +- 确认完整批次已经复制到异机/异盘,而不是仍只保存在应用主机。 - 准备独立 smoke 账号,避免使用老板、管理员或真实员工账号跑自动化。 ## 2. 升级流程(试运行) diff --git a/scripts/backup-all.sh b/scripts/backup-all.sh new file mode 100755 index 00000000..d0eca120 --- /dev/null +++ b/scripts/backup-all.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +BACKUP_DIR="${BACKUP_DIR:-$ROOT_DIR/deploy/trialrun/backups}" +BACKUP_RUN_ID="${BACKUP_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}" + +if [[ ! "$BACKUP_RUN_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$ ]]; then + echo "[ERR] BACKUP_RUN_ID must use 1-80 safe characters" >&2 + exit 1 +fi + +mkdir -p "$BACKUP_DIR" +FINAL_DIR="$BACKUP_DIR/hyb-oa-backup-$BACKUP_RUN_ID" +if [[ -e "$FINAL_DIR" ]]; then + echo "[ERR] backup batch already exists: hyb-oa-backup-$BACKUP_RUN_ID" >&2 + exit 1 +fi + +TEMP_DIR="$(mktemp -d "$BACKUP_DIR/.hyb-oa-backup-$BACKUP_RUN_ID.tmp.XXXXXX")" +cleanup() { + if [[ -n "${TEMP_DIR:-}" && -d "$TEMP_DIR" && "$TEMP_DIR" == "$BACKUP_DIR"/.hyb-oa-backup-*.tmp.* ]]; then + rm -rf -- "$TEMP_DIR" + fi +} +trap cleanup EXIT + +DB_NAME="database.dump" +STORAGE_NAME="storage.tar.gz" +MANIFEST_NAME="manifest.json" +DB_FILE="$TEMP_DIR/$DB_NAME" +STORAGE_FILE="$TEMP_DIR/$STORAGE_NAME" + +BACKUP_DIR="$TEMP_DIR" OUT_FILE="$DB_FILE" "$ROOT_DIR/scripts/backup-db.sh" +BACKUP_DIR="$TEMP_DIR" OUT_FILE="$STORAGE_FILE" "$ROOT_DIR/scripts/backup-storage.sh" + +DB_VALIDATED="false" +if [[ "${DB_TOOL_MODE:-auto}" != "docker" ]] && command -v pg_restore >/dev/null 2>&1; then + if pg_restore --list "$DB_FILE" >/dev/null; then + DB_VALIDATED="true" + elif [[ "${DB_TOOL_MODE:-auto}" == "host" ]]; then + echo "[ERR] host pg_restore could not validate the DB dump" >&2 + exit 1 + fi +fi +if [[ "$DB_VALIDATED" != "true" ]] && command -v docker >/dev/null 2>&1 && [[ -n "${DB_CONTAINER:-}" ]]; then + docker exec -i "$DB_CONTAINER" pg_restore --list <"$DB_FILE" >/dev/null + DB_VALIDATED="true" +fi +if [[ "$DB_VALIDATED" != "true" ]]; then + echo "[ERR] pg_restore or DB_CONTAINER is required to validate the DB dump" >&2 + exit 1 +fi + +tar -tzf "$STORAGE_FILE" >/dev/null +while IFS= read -r listing_line; do + if [[ "${listing_line:0:1}" == "l" || "${listing_line:0:1}" == "h" ]]; then + echo "[ERR] storage backup contains links that the safe restore path will reject" >&2 + exit 1 + fi +done < <(tar -tvzf "$STORAGE_FILE") + +DB_SHA256="$(sha256sum "$DB_FILE" | awk '{print $1}')" +STORAGE_SHA256="$(sha256sum "$STORAGE_FILE" | awk '{print $1}')" +DB_BYTES="$(wc -c <"$DB_FILE" | tr -d ' ')" +STORAGE_BYTES="$(wc -c <"$STORAGE_FILE" | tr -d ' ')" +BUILD_SHA="$(git -C "$ROOT_DIR" rev-parse --short=12 HEAD 2>/dev/null || true)" +CREATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +node - "$TEMP_DIR/$MANIFEST_NAME" "$BACKUP_RUN_ID" "$CREATED_AT" "$BUILD_SHA" \ + "$DB_NAME" "$DB_BYTES" "$DB_SHA256" "$STORAGE_NAME" "$STORAGE_BYTES" "$STORAGE_SHA256" <<'NODE' +const fs = require("node:fs"); +const [manifestPath, runId, createdAt, buildSha, dbFile, dbBytes, dbSha256, storageFile, storageBytes, storageSha256] = process.argv.slice(2); +const manifest = { + schemaVersion: 1, + runId, + createdAt, + buildSha: buildSha || null, + status: "complete", + artifacts: { + database: { file: dbFile, bytes: Number(dbBytes), sha256: dbSha256 }, + storage: { file: storageFile, bytes: Number(storageBytes), sha256: storageSha256 }, + }, +}; +fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, { encoding: "utf8", flag: "wx" }); +NODE + +mv "$TEMP_DIR" "$FINAL_DIR" +TEMP_DIR="" +trap - EXIT + +echo "[OK] Paired backup complete: hyb-oa-backup-$BACKUP_RUN_ID" +echo "[INFO] Copy the complete batch directory to approved off-host/off-disk storage" diff --git a/scripts/backup-db.sh b/scripts/backup-db.sh index e930709e..337cd409 100755 --- a/scripts/backup-db.sh +++ b/scripts/backup-db.sh @@ -9,17 +9,46 @@ mkdir -p "$BACKUP_DIR" TS="$(date +%Y%m%d_%H%M%S)" OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-db-${TS}.dump}" +if [[ -e "$OUT_FILE" ]]; then + echo "[ERR] backup output already exists: $OUT_FILE" >&2 + exit 1 +fi + +mkdir -p "$(dirname "$OUT_FILE")" + if [[ -z "${DATABASE_URL:-}" ]]; then echo "[ERR] DATABASE_URL is required" >&2 exit 1 fi +DB_TOOL_MODE="${DB_TOOL_MODE:-auto}" +if [[ "$DB_TOOL_MODE" != "auto" && "$DB_TOOL_MODE" != "host" && "$DB_TOOL_MODE" != "docker" ]]; then + echo "[ERR] DB_TOOL_MODE must be auto, host, or docker" >&2 + exit 1 +fi + echo "[INFO] DB backup start: $OUT_FILE" -if command -v pg_dump >/dev/null 2>&1; then - pg_dump "$DATABASE_URL" --format=c --no-owner --no-acl --file "$OUT_FILE" - echo "[OK] DB backup done (pg_dump): $OUT_FILE" - exit 0 +if [[ "$DB_TOOL_MODE" != "docker" ]] && command -v pg_dump >/dev/null 2>&1; then + if pg_dump "$DATABASE_URL" --format=c --no-owner --no-acl --file "$OUT_FILE"; then + if [[ ! -s "$OUT_FILE" ]]; then + echo "[ERR] DB backup is empty: $OUT_FILE" >&2 + exit 1 + fi + echo "[OK] DB backup done (pg_dump): $OUT_FILE" + exit 0 + fi + rm -f -- "$OUT_FILE" + if [[ "$DB_TOOL_MODE" == "host" ]]; then + echo "[ERR] host pg_dump failed" >&2 + exit 1 + fi + echo "[WARN] host pg_dump failed; trying docker fallback" >&2 +fi + +if [[ "$DB_TOOL_MODE" == "host" ]]; then + echo "[ERR] DB_TOOL_MODE=host requires pg_dump" >&2 + exit 1 fi if ! command -v docker >/dev/null 2>&1; then @@ -37,9 +66,17 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th exit 1 fi -docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ +if ! docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --format=c --no-owner --no-acl \ - >"$OUT_FILE" + >"$OUT_FILE"; then + rm -f -- "$OUT_FILE" + echo "[ERR] docker pg_dump failed" >&2 + exit 1 +fi + +if [[ ! -s "$OUT_FILE" ]]; then + echo "[ERR] DB backup is empty: $OUT_FILE" >&2 + exit 1 +fi echo "[OK] DB backup done (docker exec): $OUT_FILE" - diff --git a/scripts/backup-storage.sh b/scripts/backup-storage.sh index 70c8a732..2bcbc5b7 100755 --- a/scripts/backup-storage.sh +++ b/scripts/backup-storage.sh @@ -9,6 +9,11 @@ mkdir -p "$BACKUP_DIR" TS="$(date +%Y%m%d_%H%M%S)" OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-storage-${TS}.tar.gz}" +if [[ -e "$OUT_FILE" ]]; then + echo "[ERR] backup output already exists: $OUT_FILE" >&2 + exit 1 +fi + STORAGE_DIR_HOST="${STORAGE_DIR_HOST:-}" if [[ -z "$STORAGE_DIR_HOST" ]]; then if [[ -n "${STORAGE_DIR:-}" && -d "${STORAGE_DIR:-}" ]]; then @@ -21,10 +26,25 @@ if [[ -z "$STORAGE_DIR_HOST" ]]; then fi fi +if [[ ! -d "$STORAGE_DIR_HOST" ]]; then + echo "[ERR] storage dir not found: $STORAGE_DIR_HOST" >&2 + exit 1 +fi + +mkdir -p "$(dirname "$OUT_FILE")" + echo "[INFO] Storage backup start: $OUT_FILE" echo "[INFO] Storage dir: $STORAGE_DIR_HOST" -tar -czf "$OUT_FILE" -C "$STORAGE_DIR_HOST" . +if ! tar -czf "$OUT_FILE" -C "$STORAGE_DIR_HOST" .; then + rm -f -- "$OUT_FILE" + echo "[ERR] storage backup failed" >&2 + exit 1 +fi + +if [[ ! -s "$OUT_FILE" ]]; then + echo "[ERR] storage backup is empty: $OUT_FILE" >&2 + exit 1 +fi echo "[OK] Storage backup done: $OUT_FILE" - diff --git a/scripts/qa/run-backup-restore-drill.sh b/scripts/qa/run-backup-restore-drill.sh new file mode 100755 index 00000000..c5e6c437 --- /dev/null +++ b/scripts/qa/run-backup-restore-drill.sh @@ -0,0 +1,150 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +DRILL_RUN_ID="drill-$(date -u +%Y%m%dT%H%M%SZ)-$$" +CONTAINER_NAME="hyb-oa-backup-drill-$$" +DRILL_DIR="$(mktemp -d)" +POSTGRES_USER="drill_user" +POSTGRES_PASSWORD="drill_password" +POSTGRES_DB="hyb_oa_restore_shadow" + +cleanup() { + docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true + if [[ -n "${DRILL_DIR:-}" && -d "$DRILL_DIR" && "$DRILL_DIR" == /tmp/* ]]; then + rm -rf -- "$DRILL_DIR" + fi +} +trap cleanup EXIT + +require_command() { + if ! command -v "$1" >/dev/null 2>&1; then + echo "[ERR] required command missing: $1" >&2 + exit 1 + fi +} + +for command_name in docker pg_dump pg_restore psql tar sha256sum node; do + require_command "$command_name" +done + +START_EPOCH="$(date +%s)" +SOURCE_STORAGE="$DRILL_DIR/source-storage" +RESTORED_STORAGE="$DRILL_DIR/restored-storage" +NONEMPTY_STORAGE="$DRILL_DIR/nonempty-storage" +BACKUP_ROOT="$DRILL_DIR/backups" +mkdir -p "$SOURCE_STORAGE/nested" "$NONEMPTY_STORAGE" "$BACKUP_ROOT" +printf 'synthetic attachment for restore drill\n' >"$SOURCE_STORAGE/nested/marker.txt" +printf 'must not be overwritten\n' >"$NONEMPTY_STORAGE/existing.txt" + +docker run -d --rm --name "$CONTAINER_NAME" \ + -e "POSTGRES_USER=$POSTGRES_USER" \ + -e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \ + -e "POSTGRES_DB=$POSTGRES_DB" \ + -p 127.0.0.1::5432 postgres:16-alpine >/dev/null + +for _ in $(seq 1 60); do + if docker exec "$CONTAINER_NAME" pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" >/dev/null 2>&1; then + break + fi + sleep 1 +done +docker exec "$CONTAINER_NAME" pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" >/dev/null + +HOST_PORT="$(docker port "$CONTAINER_NAME" 5432/tcp | awk -F: 'NR==1 {print $NF}')" +DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@127.0.0.1:$HOST_PORT/$POSTGRES_DB" +export DATABASE_URL STORAGE_DIR_HOST="$SOURCE_STORAGE" BACKUP_DIR="$BACKUP_ROOT" BACKUP_RUN_ID="$DRILL_RUN_ID" +export DB_TOOL_MODE="docker" DB_CONTAINER="$CONTAINER_NAME" POSTGRES_USER POSTGRES_PASSWORD POSTGRES_DB + +psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc 'CREATE TABLE restore_drill_marker (value text NOT NULL)' +psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc "INSERT INTO restore_drill_marker(value) VALUES ('synthetic-round-trip')" + +"$ROOT_DIR/scripts/backup-all.sh" +BATCH_DIR="$BACKUP_ROOT/hyb-oa-backup-$DRILL_RUN_ID" +DB_BACKUP="$BATCH_DIR/database.dump" +STORAGE_BACKUP="$BATCH_DIR/storage.tar.gz" + +node - "$BATCH_DIR/manifest.json" "$DB_BACKUP" "$STORAGE_BACKUP" <<'NODE' +const crypto = require("node:crypto"); +const fs = require("node:fs"); +const path = require("node:path"); +const [manifestPath, dbPath, storagePath] = process.argv.slice(2); +const manifestText = fs.readFileSync(manifestPath, "utf8"); +const manifest = JSON.parse(manifestText); +if (manifest.status !== "complete" || manifest.schemaVersion !== 1) throw new Error("manifest contract mismatch"); +if (path.isAbsolute(manifest.artifacts.database.file) || path.isAbsolute(manifest.artifacts.storage.file)) { + throw new Error("manifest contains absolute artifact path"); +} +if (/postgresql:|DATABASE_URL|drill_password/.test(manifestText)) throw new Error("manifest contains sensitive connection data"); +const sha256 = (file) => crypto.createHash("sha256").update(fs.readFileSync(file)).digest("hex"); +if (sha256(dbPath) !== manifest.artifacts.database.sha256) throw new Error("database checksum mismatch"); +if (sha256(storagePath) !== manifest.artifacts.storage.sha256) throw new Error("storage checksum mismatch"); +NODE + +psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc 'DROP TABLE restore_drill_marker' + +docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$CONTAINER_NAME" \ + createdb -U "$POSTGRES_USER" hyb_oa_prod +if DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@127.0.0.1:$HOST_PORT/hyb_oa_prod" \ + POSTGRES_DB=hyb_oa_prod RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=hyb_oa_prod \ + "$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP" >/dev/null 2>&1; then + echo "[ERR] production-like DB name was accepted as a shadow target" >&2 + exit 1 +fi + +CORRUPT_DB="$DRILL_DIR/corrupt.dump" +head -c 32 "$DB_BACKUP" >"$CORRUPT_DB" +if RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE="$POSTGRES_DB" \ + "$ROOT_DIR/scripts/restore-db.sh" "$CORRUPT_DB" >/dev/null 2>&1; then + echo "[ERR] corrupt DB dump was accepted" >&2 + exit 1 +fi + +if RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=wrong_target \ + "$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP" >/dev/null 2>&1; then + echo "[ERR] mismatched DB confirmation was accepted" >&2 + exit 1 +fi + +RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE="$POSTGRES_DB" \ + "$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP" + +RESTORED_VALUE="$(psql "$DATABASE_URL" -Atqc 'SELECT value FROM restore_drill_marker')" +if [[ "$RESTORED_VALUE" != "synthetic-round-trip" ]]; then + echo "[ERR] DB round-trip marker mismatch" >&2 + exit 1 +fi + +if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$STORAGE_BACKUP" "$NONEMPTY_STORAGE" >/dev/null 2>&1; then + echo "[ERR] non-empty storage target was accepted" >&2 + exit 1 +fi + +if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" \ + "$STORAGE_BACKUP" "$ROOT_DIR/deploy/trialrun/data/storage" >/dev/null 2>&1; then + echo "[ERR] default live storage path was accepted as a shadow target" >&2 + exit 1 +fi + +RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$STORAGE_BACKUP" "$RESTORED_STORAGE" +cmp "$SOURCE_STORAGE/nested/marker.txt" "$RESTORED_STORAGE/nested/marker.txt" + +CORRUPT_TAR="$DRILL_DIR/corrupt.tar.gz" +head -c 32 "$STORAGE_BACKUP" >"$CORRUPT_TAR" +if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$CORRUPT_TAR" "$DRILL_DIR/corrupt-target" >/dev/null 2>&1; then + echo "[ERR] corrupt storage archive was accepted" >&2 + exit 1 +fi + +DANGEROUS_TAR="$DRILL_DIR/dangerous.tar.gz" +tar -czf "$DANGEROUS_TAR" --transform='s|nested/marker.txt|../escape.txt|' \ + -C "$SOURCE_STORAGE" nested/marker.txt +if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$DANGEROUS_TAR" "$DRILL_DIR/dangerous-target" >/dev/null 2>&1; then + echo "[ERR] dangerous storage archive path was accepted" >&2 + exit 1 +fi + +END_EPOCH="$(date +%s)" +ELAPSED_SECONDS="$((END_EPOCH - START_EPOCH))" +echo "[PASS] isolated backup/restore drill completed" +echo "[INFO] local synthetic RTO seconds: $ELAPSED_SECONDS" diff --git a/scripts/restore-db.sh b/scripts/restore-db.sh index 45fcd5c6..3369ac30 100755 --- a/scripts/restore-db.sh +++ b/scripts/restore-db.sh @@ -1,8 +1,6 @@ #!/usr/bin/env bash set -euo pipefail -ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" - BACKUP_FILE="${1:-${BACKUP_FILE:-}}" if [[ -z "${BACKUP_FILE:-}" ]]; then echo "Usage: $0 " >&2 @@ -19,16 +17,69 @@ if [[ -z "${DATABASE_URL:-}" ]]; then exit 1 fi -echo "[INFO] DB restore start: $BACKUP_FILE" +RESTORE_SCOPE="${RESTORE_SCOPE:-}" +RESTORE_CONFIRM_DATABASE="${RESTORE_CONFIRM_DATABASE:-}" +DB_TOOL_MODE="${DB_TOOL_MODE:-auto}" -if command -v pg_restore >/dev/null 2>&1; then +if [[ "$DB_TOOL_MODE" != "auto" && "$DB_TOOL_MODE" != "host" && "$DB_TOOL_MODE" != "docker" ]]; then + echo "[ERR] DB_TOOL_MODE must be auto, host, or docker" >&2 + exit 1 +fi + +if [[ "$RESTORE_SCOPE" != "shadow" ]]; then + echo "[ERR] RESTORE_SCOPE=shadow is required; production restore is not available from this command path" >&2 + exit 1 +fi + +if [[ -z "$RESTORE_CONFIRM_DATABASE" ]]; then + echo "[ERR] RESTORE_CONFIRM_DATABASE is required" >&2 + exit 1 +fi + +validate_dump_with_host() { + pg_restore --list "$BACKUP_FILE" >/dev/null +} + +validate_shadow_database_name() { + local database_name="$1" + if [[ ! "$database_name" =~ (^|[_-])(shadow|drill|restore_test)([_-]|$) ]]; then + echo "[ERR] restore target name is not an explicit shadow/drill database" >&2 + exit 1 + fi +} + +restore_with_host() { + local actual_database + actual_database="$(psql "$DATABASE_URL" -Atqc 'SELECT current_database()')" + if [[ -z "$actual_database" || "$actual_database" != "$RESTORE_CONFIRM_DATABASE" ]]; then + echo "[ERR] restore target confirmation mismatch: actual=${actual_database:-unknown}" >&2 + exit 1 + fi + validate_shadow_database_name "$actual_database" + echo "[INFO] DB restore target confirmed: $actual_database (shadow)" pg_restore --clean --if-exists --no-owner --no-acl -d "$DATABASE_URL" "$BACKUP_FILE" - echo "[OK] DB restore done (pg_restore): $BACKUP_FILE" - exit 0 +} + +if [[ "$DB_TOOL_MODE" != "docker" ]] && command -v pg_restore >/dev/null 2>&1 && command -v psql >/dev/null 2>&1; then + if validate_dump_with_host; then + restore_with_host + echo "[OK] DB restore done (pg_restore): $BACKUP_FILE" + exit 0 + fi + if [[ "$DB_TOOL_MODE" == "host" ]]; then + echo "[ERR] host pg_restore could not validate the dump" >&2 + exit 1 + fi + echo "[WARN] host pg_restore could not validate the dump; trying docker fallback" >&2 +fi + +if [[ "$DB_TOOL_MODE" == "host" ]]; then + echo "[ERR] DB_TOOL_MODE=host requires pg_restore and psql" >&2 + exit 1 fi if ! command -v docker >/dev/null 2>&1; then - echo "[ERR] pg_restore not found and docker not available" >&2 + echo "[ERR] pg_restore/psql not found and docker not available" >&2 exit 1 fi @@ -42,8 +93,20 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th exit 1 fi -cat "$BACKUP_FILE" | docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ - pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB" +ACTUAL_DATABASE="$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ + psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atqc 'SELECT current_database()')" + +if [[ -z "$ACTUAL_DATABASE" || "$ACTUAL_DATABASE" != "$RESTORE_CONFIRM_DATABASE" ]]; then + echo "[ERR] restore target confirmation mismatch: actual=${ACTUAL_DATABASE:-unknown}" >&2 + exit 1 +fi +validate_shadow_database_name "$ACTUAL_DATABASE" + +docker exec -i "$DB_CONTAINER" pg_restore --list <"$BACKUP_FILE" >/dev/null + +echo "[INFO] DB restore target confirmed: $ACTUAL_DATABASE (shadow)" +docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \ + pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB" \ + <"$BACKUP_FILE" echo "[OK] DB restore done (docker exec): $BACKUP_FILE" - diff --git a/scripts/restore-storage.sh b/scripts/restore-storage.sh index 5c658b38..5a79268c 100755 --- a/scripts/restore-storage.sh +++ b/scripts/restore-storage.sh @@ -16,19 +16,61 @@ fi STORAGE_DIR_HOST="${2:-${STORAGE_DIR_HOST:-}}" if [[ -z "$STORAGE_DIR_HOST" ]]; then - if [[ -n "${STORAGE_DIR:-}" && -d "${STORAGE_DIR:-}" ]]; then - STORAGE_DIR_HOST="$STORAGE_DIR" - else - STORAGE_DIR_HOST="$ROOT_DIR/deploy/trialrun/data/storage" - fi + echo "[ERR] explicit restore target is required as the second argument or STORAGE_DIR_HOST" >&2 + exit 1 fi -mkdir -p "$STORAGE_DIR_HOST" +if [[ "${RESTORE_SCOPE:-}" != "shadow" ]]; then + echo "[ERR] RESTORE_SCOPE=shadow is required; production restore is not available from this command path" >&2 + exit 1 +fi + +TARGET_PATH="$(realpath -m "$STORAGE_DIR_HOST")" +LIVE_STORAGE_PATH="$(realpath -m "$ROOT_DIR/deploy/trialrun/data/storage")" +if [[ "$TARGET_PATH" == "/" || "$TARGET_PATH" == "$ROOT_DIR" || "$TARGET_PATH" == "$LIVE_STORAGE_PATH" ]]; then + echo "[ERR] unsafe storage restore target" >&2 + exit 1 +fi + +if [[ -d "$TARGET_PATH" && -n "$(find "$TARGET_PATH" -mindepth 1 -print -quit)" ]]; then + echo "[ERR] storage restore target must be empty: $TARGET_PATH" >&2 + exit 1 +fi + +validate_archive_entries() { + local entry normalized part listing_line + tar -tzf "$BACKUP_FILE" >/dev/null + while IFS= read -r entry; do + if [[ -z "$entry" || "$entry" == /* ]]; then + echo "[ERR] unsafe archive path" >&2 + return 1 + fi + normalized="${entry#./}" + [[ -z "$normalized" || "$normalized" == "." ]] && continue + IFS='/' read -r -a path_parts <<<"$normalized" + for part in "${path_parts[@]}"; do + if [[ "$part" == ".." ]]; then + echo "[ERR] unsafe archive path segment" >&2 + return 1 + fi + done + done < <(tar -tzf "$BACKUP_FILE") + + tar -tvzf "$BACKUP_FILE" >/dev/null + while IFS= read -r listing_line; do + if [[ "${listing_line:0:1}" == "l" || "${listing_line:0:1}" == "h" ]]; then + echo "[ERR] storage archive links are not allowed" >&2 + return 1 + fi + done < <(tar -tvzf "$BACKUP_FILE") +} + +validate_archive_entries +mkdir -p "$TARGET_PATH" echo "[INFO] Storage restore start: $BACKUP_FILE" -echo "[INFO] Storage dir: $STORAGE_DIR_HOST" +echo "[INFO] Storage target confirmed: shadow" -tar -xzf "$BACKUP_FILE" -C "$STORAGE_DIR_HOST" +tar -xzf "$BACKUP_FILE" --no-same-owner --no-same-permissions -C "$TARGET_PATH" echo "[OK] Storage restore done: $BACKUP_FILE" - diff --git a/server/README.md b/server/README.md index 38d4673d..7590e9ca 100644 --- a/server/README.md +++ b/server/README.md @@ -298,12 +298,12 @@ SERVE_CLIENT_DIST=true npm run dev 备份/恢复脚本(E14): ```bash cd -./scripts/backup-db.sh -./scripts/restore-db.sh deploy/trialrun/backups/ -./scripts/backup-storage.sh -./scripts/restore-storage.sh deploy/trialrun/backups/ deploy/trialrun/data/storage +DB_TOOL_MODE=docker DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh +./scripts/qa/run-backup-restore-drill.sh ``` +成对备份会生成 DB、附件和脱敏 manifest。恢复脚本只允许显式影子目标;真实事故恢复、异机副本和 `/ready` 验收按 `deploy/trialrun/runbook.md` 的授权门执行,不直接在 README 复制破坏性命令。 + 优雅停机(E21): - `docker stop` / `Ctrl+C` 会触发 `SIGTERM/SIGINT`,server 会调用 `app.close()` 完成收敛关闭(定时任务/HTTP/DB)。