加固备份恢复本地基线（07-26-internal-ops-recovery-baseline）
这个提交包含在：
@@ -516,3 +516,95 @@ echo LOCAL_CANDIDATE_PASS
|
|||||||
bash scripts/qa/run-local-candidate.sh full
|
bash scripts/qa/run-local-candidate.sh full
|
||||||
# Only this successful branch emits LOCAL_CANDIDATE_PASS.
|
# Only this successful branch emits LOCAL_CANDIDATE_PASS.
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Scenario: Paired Backup And Shadow-Only Restore
|
||||||
|
|
||||||
|
### 1. Scope / Trigger
|
||||||
|
|
||||||
|
- Trigger: changing or operating database/file backups, restore scripts,
|
||||||
|
retention instructions or recovery-readiness evidence.
|
||||||
|
- Purpose: prevent a database-only recovery point, a mixed attachment restore or
|
||||||
|
an accidental destructive restore into the active internal instance.
|
||||||
|
|
||||||
|
### 2. Signatures
|
||||||
|
|
||||||
|
- Paired backup: `DB_TOOL_MODE=docker DB_CONTAINER=<name> ./scripts/backup-all.sh`.
|
||||||
|
- Optional safe batch identifier: `BACKUP_RUN_ID=<safe-id>`.
|
||||||
|
- DB shadow restore requires `RESTORE_SCOPE=shadow`,
|
||||||
|
`RESTORE_CONFIRM_DATABASE=<actual-current-database>` and `DATABASE_URL`.
|
||||||
|
- Storage shadow restore:
|
||||||
|
`RESTORE_SCOPE=shadow ./scripts/restore-storage.sh <tar> <empty-target>`.
|
||||||
|
- Local synthetic drill: `./scripts/qa/run-backup-restore-drill.sh`.
|
||||||
|
- `DB_TOOL_MODE` is `auto`, `host` or `docker`; use `docker` with Compose to
|
||||||
|
avoid host/server PostgreSQL tool-version drift.
|
||||||
|
|
||||||
|
### 3. Contracts
|
||||||
|
|
||||||
|
- A complete recovery point is one batch directory containing `database.dump`,
|
||||||
|
`storage.tar.gz` and a final `manifest.json`. The manifest contains only safe
|
||||||
|
relative names, byte sizes, SHA-256 hashes, batch time and optional build SHA.
|
||||||
|
- The DB dump must pass `pg_restore --list` and the storage archive must pass
|
||||||
|
`tar -tzf` before the manifest may say `complete`.
|
||||||
|
- DB restore queries `current_database()` and compares it exactly with
|
||||||
|
`RESTORE_CONFIRM_DATABASE` before `pg_restore --clean`.
|
||||||
|
- The actual DB name must also contain a separate `shadow`, `drill` or
|
||||||
|
`restore_test` marker; setting `RESTORE_SCOPE=shadow` alone is not trust.
|
||||||
|
- Storage restore requires an explicit empty target and rejects absolute paths,
|
||||||
|
`..` path segments, symbolic links and hard links before extraction.
|
||||||
|
- Current scripts intentionally reject production restore. A real incident
|
||||||
|
requires an approved Trellis task, stop window, paired batch, two responsible
|
||||||
|
people, rollback point and post-restore `/ready`/permission checks.
|
||||||
|
- Copy the whole completed batch to approved off-host/off-disk storage; a batch
|
||||||
|
left only on the application host is not accepted recovery evidence.
|
||||||
|
|
||||||
|
### 4. Validation & Error Matrix
|
||||||
|
|
||||||
|
- One backup component fails or integrity listing fails -> remove the temporary
|
||||||
|
batch; do not write a complete manifest.
|
||||||
|
- Existing batch identifier -> fail without overwriting.
|
||||||
|
- Missing/non-shadow restore scope -> fail before any restore write.
|
||||||
|
- Actual DB name differs from confirmation -> fail before `pg_restore --clean`.
|
||||||
|
- Confirmed DB name lacks a shadow/drill/restore-test marker -> fail before
|
||||||
|
`pg_restore --clean`.
|
||||||
|
- Storage target is implicit, root/repository root or non-empty -> fail before
|
||||||
|
extraction.
|
||||||
|
- Corrupt dump/tar or dangerous/link archive member -> fail before restore.
|
||||||
|
- Host PostgreSQL tools mismatch in `auto` mode -> docker fallback is allowed
|
||||||
|
only when the explicit target container and required DB variables exist.
|
||||||
|
|
||||||
|
### 5. Good/Base/Bad Cases
|
||||||
|
|
||||||
|
- Good: one command creates a validated pair, the complete batch is copied off
|
||||||
|
host, and a disposable shadow drill restores the DB marker and attachment hash.
|
||||||
|
- Base: backup succeeds locally but off-host copy and deployed `/ready` evidence
|
||||||
|
remain explicitly pending; do not claim formal recovery readiness.
|
||||||
|
- Bad: source the active trial `.env` and run a cleaning restore directly, or
|
||||||
|
restore attachments into a non-empty live directory.
|
||||||
|
|
||||||
|
### 6. Tests Required
|
||||||
|
|
||||||
|
- Shell syntax checks for all backup/restore scripts.
|
||||||
|
- Synthetic Docker/PostgreSQL and temporary-storage round trip.
|
||||||
|
- Negative assertions for wrong DB confirmation, corrupt DB/tar, non-empty
|
||||||
|
target and dangerous tar path.
|
||||||
|
- Validate manifest relative paths, absence of connection secrets and matching
|
||||||
|
SHA-256 values.
|
||||||
|
- Real-instance restore and `/ready` remain separately approved evidence and
|
||||||
|
must never be fabricated from the local drill.
|
||||||
|
|
||||||
|
### 7. Wrong vs Correct
|
||||||
|
|
||||||
|
#### Wrong
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source deploy/trialrun/.env
|
||||||
|
./scripts/restore-db.sh some.dump
|
||||||
|
./scripts/restore-storage.sh some.tar.gz deploy/trialrun/data/storage
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Correct
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/qa/run-backup-restore-drill.sh
|
||||||
|
# Real restore remains an approved incident operation after shadow evidence.
|
||||||
|
```
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# 内部 OA 运行与恢复底线设计
|
||||||
|
|
||||||
|
## 总体边界
|
||||||
|
|
||||||
|
本任务将现有四个底层脚本保留为单一职责工具，在其上增加一个成对备份入口和一个隔离演练入口。所有默认值面向本地/影子环境；真实环境只能沿 runbook 的人工门禁执行。
|
||||||
|
|
||||||
|
```text
|
||||||
|
backup pair entry
|
||||||
|
├─ backup-db.sh → <run-id>.db.dump
|
||||||
|
├─ backup-storage.sh → <run-id>.storage.tar.gz
|
||||||
|
└─ integrity checks → <run-id>.manifest
|
||||||
|
|
||||||
|
isolated drill
|
||||||
|
├─ disposable PostgreSQL/database
|
||||||
|
├─ disposable source/restore directories
|
||||||
|
├─ synthetic marker + synthetic attachment
|
||||||
|
└─ restore → compare → cleanup exact targets
|
||||||
|
```
|
||||||
|
|
||||||
|
## 备份合同
|
||||||
|
|
||||||
|
- 一个批次只使用一个 `run_id`，由上层入口传递明确的 `OUT_FILE` 给现有底层脚本。
|
||||||
|
- 先把产物写入批次临时目录；DB 和附件均成功并通过完整性检查后，再写 manifest 并把批次标记为完整。
|
||||||
|
- manifest 使用稳定、脱敏的键：`schemaVersion`、`runId`、`createdAt`、`buildSha`（如可安全取得）、两份相对文件名、字节数和 SHA-256。
|
||||||
|
- 不把连接串、容器密码、存储源路径、真实业务文件清单或主机绝对路径写入 manifest。
|
||||||
|
- 本任务不负责异机传输实现；runbook 要求操作者把完整批次复制到异机/异盘，并记录保留周期。
|
||||||
|
|
||||||
|
## DB 恢复安全合同
|
||||||
|
|
||||||
|
- 恢复前通过数据库连接读取 `current_database()`，不得只相信 URL 字符串或调用者描述。
|
||||||
|
- 调用者必须提供与实际数据库名一致的确认值，并显式选择 `shadow`。
|
||||||
|
- 实际目标库名还必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记，不能仅靠调用者把模式写成 `shadow`。
|
||||||
|
- 确认缺失、连接失败、数据库名不匹配或模式不是 `shadow` 时，在 `pg_restore --clean` 之前退出。
|
||||||
|
- Docker fallback 也必须执行同样的数据库名探测和确认，不得成为绕过路径。
|
||||||
|
- 真实/生产恢复不在本任务脚本默认路径内；未来如确需开放，应另建任务定义双人确认和事故编号。
|
||||||
|
|
||||||
|
## 附件恢复安全合同
|
||||||
|
|
||||||
|
- 必须显式传入恢复目标，不再默认落入试运行数据目录。
|
||||||
|
- 明确拒绝仓库的默认试运行附件目录，即使该目录碰巧为空。
|
||||||
|
- 影子恢复目标必须为空；非空时退出，不执行覆盖或合并。
|
||||||
|
- 解压前列出归档成员并拒绝绝对路径、空成员和包含 `..` 路径段的成员。
|
||||||
|
- 只对已验证的归档执行解压；演练通过内容 SHA-256 比较证明往返一致。
|
||||||
|
|
||||||
|
## 隔离演练
|
||||||
|
|
||||||
|
- 数据库只使用一次性本地容器或明确命名的 disposable database，插入不含业务信息的 marker 表/行。
|
||||||
|
- 附件只使用临时目录和合成文本/二进制样件。
|
||||||
|
- 演练记录开始/结束时间、各阶段通过状态和相对产物名；不记录连接串、容器密码或临时绝对路径。
|
||||||
|
- 清理只针对演练创建的明确容器、数据库和临时目录；不得使用宽泛路径或工作区根目录。
|
||||||
|
|
||||||
|
## 兼容与回滚
|
||||||
|
|
||||||
|
- `backup-db.sh` 和 `backup-storage.sh` 的现有单独调用继续可用；推荐运维入口改为成对备份命令。
|
||||||
|
- `restore-db.sh`/`restore-storage.sh` 的安全收紧属于有意的命令合同变化，runbook 和 checklist 必须同步更新。
|
||||||
|
- 回滚代码可恢复旧脚本，但已经生成的 dump/tar 仍保持 PostgreSQL/tar 标准格式，不被新 manifest 锁定。
|
||||||
|
- 如演练环境缺少 PostgreSQL 客户端和 Docker，停止 DB 动态演练；仍可完成 shell 静态检查和附件演练，不得转向真实环境补测。
|
||||||
|
|
||||||
|
## 外部门禁
|
||||||
|
|
||||||
|
以下事项不因本地演练通过而自动获批：读取真实运行环境、复制真实备份、建立异机定时任务、停机、恢复真实库/附件、部署和公网 `/ready`。进入这些步骤前再次列出目标、只读/写入范围和回滚方式并取得明确批准。
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# 内部 OA 运行与恢复底线执行计划
|
||||||
|
|
||||||
|
## 实施步骤
|
||||||
|
|
||||||
|
- [x] 审计现有备份/恢复脚本、部署卷、runbook 和 checklist，冻结现有兼容入口与风险点。
|
||||||
|
- [x] 增加 DB + 附件成对备份入口、统一批次号、完整性检查、SHA-256 和脱敏 manifest。
|
||||||
|
- [x] 收紧 DB 恢复：实际数据库名探测、影子模式和精确确认必须在破坏性命令前通过。
|
||||||
|
- [x] 收紧附件恢复：显式目标、空目录门禁和 tar 路径安全检查。
|
||||||
|
- [x] 增加隔离合成数据演练，覆盖成功往返及关键负向场景。
|
||||||
|
- [x] 更新 runbook/checklist 和部署规约，区分普通备份、影子演练和真实事故恢复。
|
||||||
|
- [x] 运行任务测试计划和 Trellis 治理门禁；按实际结果记录本地 RTO，不伪造 `/ready` 或异机证据。
|
||||||
|
- [ ] 在单独授权后再执行真实运行来源、职责分离、异机副本和 `/ready` 验收。
|
||||||
|
|
||||||
|
## 推荐文件边界
|
||||||
|
|
||||||
|
- 复用并按需修改：`scripts/backup-db.sh`、`scripts/backup-storage.sh`、`scripts/restore-db.sh`、`scripts/restore-storage.sh`。
|
||||||
|
- 最多新增两个小脚本：一个成对备份入口、一个 QA-only 隔离演练入口；不创建通用备份框架。
|
||||||
|
- 更新现有 `deploy/trialrun/runbook.md`、`deploy/trialrun/checklist.md` 和 `.trellis/spec/hyb-oa/deployment-trialrun.md`，不新建常驻平行业务文档。
|
||||||
|
|
||||||
|
## 验证命令
|
||||||
|
|
||||||
|
- `bash -n scripts/backup-db.sh scripts/backup-storage.sh scripts/restore-db.sh scripts/restore-storage.sh scripts/backup-all.sh scripts/qa/run-backup-restore-drill.sh`
|
||||||
|
- `bash scripts/qa/run-backup-restore-drill.sh`
|
||||||
|
- `node deploy/trialrun/validate-env-contract.js`
|
||||||
|
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-26-internal-ops-recovery-baseline`
|
||||||
|
- `node scripts/qa/check-trellis-governance.mjs`
|
||||||
|
- `node scripts/qa/check-sensitive-content.mjs`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
|
## 回滚点
|
||||||
|
|
||||||
|
- 成对入口失败：保留四个底层脚本，不写完整 manifest。
|
||||||
|
- DB/附件安全门禁与现有操作冲突：先修 runbook 和调用方，不放宽默认门禁。
|
||||||
|
- 动态演练环境不可用：记录缺失依赖并停止，不使用真实数据替代。
|
||||||
|
- 任何命令解析到生产、远程或含真实数据的目标：立即停止并请求明确授权。
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# 内部OA运行与恢复底线
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
为 5–10 人内部 OA 建立一个主要开发者可执行的备份与恢复底线：数据库和附件始终成对备份、产物可校验、恢复默认只能进入明确的影子目标，并用合成数据证明可恢复；同时把真实内部实例的运行来源、角色职责、异机副本和 `/ready` 验收留在明确的外部授权门后。
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
- 当前可信状态仍是 `LOCAL_CANDIDATE_PASS`，不能据此宣称已部署或可以正式内用。
|
||||||
|
- 已有 `backup-db.sh`、`backup-storage.sh`、`restore-db.sh` 和 `restore-storage.sh`，runbook 也描述了 DB + 文件恢复，但尚无当前恢复演练证据。
|
||||||
|
- DB 恢复会执行 `pg_restore --clean --if-exists`；当前脚本没有影子目标确认，误指向真实库会产生破坏性后果。
|
||||||
|
- 文件恢复会直接解压到目标目录；当前不会拒绝非空目录，可能把旧文件和恢复文件混合。
|
||||||
|
- DB 与附件备份分别生成时间戳，没有共同批次清单、完整性校验和，无法稳定证明两份产物属于同一次恢复点。
|
||||||
|
- 本任务以小团队低维护为目标，不引入备份平台、对象存储、Kubernetes、云厂商锁定或复杂灾备编排。
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- 本地备份/恢复 shell 脚本的安全合同、成对备份入口、完整性校验与脱敏 manifest。
|
||||||
|
- 使用一次性 PostgreSQL 目标和临时附件目录完成合成数据恢复演练及负向测试。
|
||||||
|
- 更新现有试运行 runbook、checklist 和部署规约，明确影子恢复与真实恢复边界。
|
||||||
|
- 规划并保留真实运行 commit、访问边界、角色职责、异机副本及恢复后 `/ready` 的外部授权门；未授权前不执行这些动作。
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- 提供一个简单的成对备份入口，复用现有 DB/附件脚本，以同一批次标识产生两份产物和脱敏 manifest。
|
||||||
|
- manifest 只包含批次号、相对文件名、大小、SHA-256、创建时间及安全的版本信息，不包含数据库 URL、Secret、真实附件名或完整绝对路径。
|
||||||
|
- 备份完成前验证 DB dump 可列出、附件 tar 可读取；任一部分失败时不得产出“完整批次”结论。
|
||||||
|
- DB 恢复必须先解析并显示目标数据库名，并要求调用者显式确认相同目标；默认仅允许 `shadow` 演练模式，且实际库名必须带 `shadow`、`drill` 或 `restore_test` 独立标记。本任务不开放生产恢复快捷开关。
|
||||||
|
- 附件恢复必须要求显式目标目录；默认拒绝非空目录，并在解压前检查归档条目，禁止绝对路径和 `..` 越界路径。
|
||||||
|
- 用完全隔离的合成数据库和合成附件完成本地往返演练，不读取或复制真实发票、合同、银行资料或生产数据。
|
||||||
|
- 恢复演练至少证明：DB 标记数据恢复、附件内容哈希一致、损坏备份被拒绝、错误确认值被拒绝、非空附件目标被拒绝。
|
||||||
|
- 更新现有 runbook/checklist，使正常备份、影子演练、真实恢复三种场景不再共用含糊命令；保持现有本地存储默认，不启用 MinIO。
|
||||||
|
- 将真实内部实例核查列为后续同任务外部门禁：运行 commit/访问方式/TLS、真实职责分配、异机副本和恢复后 `/ready` 均需另行批准和脱敏留证。
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] 一条命令生成同批 DB、附件和脱敏 manifest，且完整性校验通过。
|
||||||
|
- [x] DB 恢复在缺少或不匹配目标确认时失败，不会执行 `pg_restore --clean`。
|
||||||
|
- [x] 附件恢复拒绝隐式目标、非空目录和危险 tar 路径。
|
||||||
|
- [x] 隔离合成数据演练证明 DB 与附件可往返恢复，并记录实际耗时作为本地 RTO 参考。
|
||||||
|
- [x] runbook/checklist 明确影子演练命令、生产停止条件、DB + 附件配对要求和异机副本要求。
|
||||||
|
- [x] 相关 shell 语法、QA 脚本、Trellis 治理、敏感扫描和 `git diff --check` 通过。
|
||||||
|
- [ ] 在获得单独授权后，确认真实运行 commit、访问边界、职责分离、异机备份及恢复后 `/ready`；未授权前保持未勾选且不宣称正式可用。
|
||||||
|
|
||||||
|
## Out of Scope
|
||||||
|
|
||||||
|
- 不连接生产数据库，不读取真实附件，不执行远程恢复、部署或公网检查。
|
||||||
|
- 不修改 Prisma schema、业务 API、权限模型、审批链或付款状态机。
|
||||||
|
- 不引入 MinIO、云备份 SDK、全量灾备平台、自动跨地域复制或 24×7 值守。
|
||||||
|
- 不在仓库提交备份文件、恢复数据、环境变量、凭证、真实账号或完整本地路径。
|
||||||
|
- 不在本任务完成企微调休、导航、报销或模块退役；它们属于后续批次。
|
||||||
|
|
||||||
|
## Test Plan
|
||||||
|
|
||||||
|
- `bash -n scripts/backup-db.sh scripts/backup-storage.sh scripts/restore-db.sh scripts/restore-storage.sh <新增脚本>`
|
||||||
|
- 使用临时目录和一次性本地 PostgreSQL 目标运行合成数据恢复演练；若本机缺少 PostgreSQL/Docker，则记录为环境阻塞，不改用真实库。
|
||||||
|
- 覆盖损坏 dump/tar、危险 tar 路径、非空目标目录、缺少确认和确认不匹配等负向场景。
|
||||||
|
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/07-26-internal-ops-recovery-baseline`
|
||||||
|
- `node scripts/qa/check-trellis-governance.mjs`
|
||||||
|
- `node scripts/qa/check-sensitive-content.mjs`
|
||||||
|
- `git diff --check`
|
||||||
|
|
||||||
|
## Key Decision
|
||||||
|
|
||||||
|
先改进可恢复性和防误操作，再接触真实环境。默认工作流只支持影子恢复；真实恢复不是普通脚本路径，而是需要停机、负责人确认、成对备份、回滚点和独立授权的事故操作。
|
||||||
|
|
||||||
|
## Local Evidence
|
||||||
|
|
||||||
|
- 已使用一次性 PostgreSQL 16 容器和合成附件完成多轮往返演练，最近一次本地参考 RTO 为 6 秒。
|
||||||
|
- 演练覆盖 manifest/校验和、错误确认、正式风格库名、损坏 DB、损坏 tar、非空附件目录、默认试运行附件目录和危险 tar 路径拒绝。
|
||||||
|
- 该证据不包含真实业务数据、真实运行实例、异机副本或部署后 `/ready`，因此最后一项验收保持未完成。
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Keep `prd.md` focused on requirements, constraints, and acceptance criteria.
|
||||||
|
- Lightweight tasks can remain PRD-only.
|
||||||
|
- For complex tasks, add `design.md` for technical design and `implement.md` for execution planning before `task.py start`.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
{
|
||||||
|
"id": "internal-ops-recovery-baseline",
|
||||||
|
"name": "internal-ops-recovery-baseline",
|
||||||
|
"title": "内部OA运行与恢复底线",
|
||||||
|
"description": "为5至10人内部OA建立低维护、可验证、防误操作的数据库与附件成对备份恢复基线，并明确真实环境授权门。",
|
||||||
|
"status": "in_progress",
|
||||||
|
"dev_type": null,
|
||||||
|
"scope": "backup-restore-safety-local-drill-operational-baseline",
|
||||||
|
"package": null,
|
||||||
|
"priority": "P0",
|
||||||
|
"creator": "shunli",
|
||||||
|
"assignee": "shunli",
|
||||||
|
"createdAt": "2026-07-26",
|
||||||
|
"completedAt": null,
|
||||||
|
"branch": null,
|
||||||
|
"base_branch": "main",
|
||||||
|
"worktree_path": null,
|
||||||
|
"commit": null,
|
||||||
|
"pr_url": null,
|
||||||
|
"subtasks": [],
|
||||||
|
"children": [],
|
||||||
|
"parent": null,
|
||||||
|
"relatedFiles": [
|
||||||
|
"scripts/backup-db.sh",
|
||||||
|
"scripts/backup-storage.sh",
|
||||||
|
"scripts/backup-all.sh",
|
||||||
|
"scripts/restore-db.sh",
|
||||||
|
"scripts/restore-storage.sh",
|
||||||
|
"scripts/qa/run-backup-restore-drill.sh",
|
||||||
|
"deploy/trialrun/runbook.md",
|
||||||
|
"deploy/trialrun/checklist.md",
|
||||||
|
"server/README.md",
|
||||||
|
".trellis/spec/hyb-oa/deployment-trialrun.md"
|
||||||
|
],
|
||||||
|
"notes": "本地阶段完成：成对备份、脱敏 manifest、影子库/空目录恢复门禁和合成数据演练已通过，最近本地参考 RTO 6 秒；真实运行来源、职责分离、异机副本和部署后 /ready 仍待单独授权。",
|
||||||
|
"meta": {}
|
||||||
|
}
|
||||||
@@ -33,8 +33,9 @@
|
|||||||
- [ ] 全量回归（建议）：`cd server && RUN_AUDIT_REGRESSION=1 RUN_OCR_REGRESSION=1 ./scripts/regression-all.sh`
|
- [ ] 全量回归（建议）：`cd server && RUN_AUDIT_REGRESSION=1 RUN_OCR_REGRESSION=1 ./scripts/regression-all.sh`
|
||||||
- [ ] 备份（DB + 文件存储）：
|
- [ ] 备份（DB + 文件存储）：
|
||||||
- [ ] `set -a && source deploy/trialrun/.env && set +a`
|
- [ ] `set -a && source deploy/trialrun/.env && set +a`
|
||||||
- [ ] `./scripts/backup-db.sh`（记录输出文件名）
|
- [ ] `DB_TOOL_MODE=docker DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh`
|
||||||
- [ ] `./scripts/backup-storage.sh`（记录输出文件名）
|
- [ ] 同一批次包含 `database.dump`、`storage.tar.gz`、`manifest.json`，两份 SHA-256 校验通过
|
||||||
|
- [ ] 完整批次已复制到获批异机/异盘，并记录保留周期
|
||||||
|
|
||||||
## B. 发布（Deploy）
|
## B. 发布（Deploy）
|
||||||
- [ ] 本地工作树 clean，目标 commit 已进入 upstream；部署日志中的 commit 与 `/ready.data.buildSha` 一致
|
- [ ] 本地工作树 clean，目标 commit 已进入 upstream；部署日志中的 commit 与 `/ready.data.buildSha` 一致
|
||||||
@@ -71,8 +72,8 @@
|
|||||||
|
|
||||||
- [ ] **EXTERNAL_GATED**：若回滚目标仍包含旧认证挑战逻辑，人工确认边缘继续关闭 `/api/auth/forgot-password`、`/api/auth/reset-password`、`/api/auth/email-login/code`、`/api/auth/email-login/verify`；脚本不会自动改边缘，未确认不得回滚开放，直到安全版本恢复
|
- [ ] **EXTERNAL_GATED**：若回滚目标仍包含旧认证挑战逻辑，人工确认边缘继续关闭 `/api/auth/forgot-password`、`/api/auth/reset-password`、`/api/auth/email-login/code`、`/api/auth/email-login/verify`；脚本不会自动改边缘，未确认不得回滚开放，直到安全版本恢复
|
||||||
- [ ] 停止 server：`docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env stop server`
|
- [ ] 停止 server：`docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env stop server`
|
||||||
- [ ] 恢复 DB：`./scripts/restore-db.sh <db.dump>`
|
- [ ] 恢复 DB/存储属于获批事故操作；当前脚本仅允许 `RESTORE_SCOPE=shadow`，不得绕过后直接指向真实目标
|
||||||
- [ ] 恢复存储：`./scripts/restore-storage.sh <storage.tar.gz> deploy/trialrun/data/storage`
|
- [ ] 已先在独立数据库和空附件目录完成同批次影子恢复，数据库名精确确认且 manifest/SHA-256 匹配
|
||||||
- [ ] 回到上个版本：`git checkout <previous-tag>`
|
- [ ] 回到上个版本：`git checkout <previous-tag>`
|
||||||
- [ ] 重启并探活：
|
- [ ] 重启并探活：
|
||||||
- [ ] `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d --build server`
|
- [ ] `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d --build server`
|
||||||
@@ -111,8 +112,9 @@
|
|||||||
- [ ] 管理员账号仍可进入工作区设置、人员、组织、流程定义、系统设置、功能开关和审计日志。
|
- [ ] 管理员账号仍可进入工作区设置、人员、组织、流程定义、系统设置、功能开关和审计日志。
|
||||||
- [ ] 已确认不删除备用模块数据；恢复某模块只通过工作区设置或功能开关完成。
|
- [ ] 已确认不删除备用模块数据；恢复某模块只通过工作区设置或功能开关完成。
|
||||||
- [ ] 已生成 `week-01.md` 到 `week-04.md` 的观测计划，至少完成第 1 周导出。
|
- [ ] 已生成 `week-01.md` 到 `week-04.md` 的观测计划，至少完成第 1 周导出。
|
||||||
- [ ] 已完成 DB + 文件存储备份，并记录输出文件名。
|
- [ ] 已完成 DB + 文件存储同批次备份，manifest/SHA-256 通过并复制到异机/异盘。
|
||||||
- [ ] 已在同机或临时环境完成一次恢复演练，恢复后 `/ready` 通过。
|
- [ ] 已运行 `./scripts/qa/run-backup-restore-drill.sh` 完成本地合成数据演练。
|
||||||
|
- [ ] 已在获批临时环境完成真实格式影子恢复，恢复后 `/ready` 通过。
|
||||||
- [ ] 已核对管理员账号清单和 smoke 账号，自动化不使用真实业务账号。
|
- [ ] 已核对管理员账号清单和 smoke 账号，自动化不使用真实业务账号。
|
||||||
- [ ] 已执行 `cd server && npm audit --omit=dev --audit-level=high` 并完成高危/严重漏洞评审；若仍失败，不进入公网正式试点。
|
- [ ] 已执行 `cd server && npm audit --omit=dev --audit-level=high` 并完成高危/严重漏洞评审；若仍失败，不进入公网正式试点。
|
||||||
- [ ] 已执行 `bash scripts/qa/run-release-gate.sh r66-trialrun`；若数据库或公网账号条件不足，已记录手工替代步骤。
|
- [ ] 已执行 `bash scripts/qa/run-release-gate.sh r66-trialrun`；若数据库或公网账号条件不足，已记录手工替代步骤。
|
||||||
|
|||||||
+43
-23
@@ -164,8 +164,8 @@ npx playwright test tests/e2e/manual/live-invoice-ocr.spec.ts --workers=1 --repo
|
|||||||
2. 执行 `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d server`。
|
2. 执行 `docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d server`。
|
||||||
3. 在 Issue/发布记录中写明降级原因、预计恢复时间和影响范围。
|
3. 在 Issue/发布记录中写明降级原因、预计恢复时间和影响范围。
|
||||||
|
|
||||||
## 1.5 备份与恢复演练（强烈建议至少做一次）
|
## 1.5 备份与恢复演练（正式启用前必须完成）
|
||||||
> 备份建议做到“DB + 文件存储（票据上传目录）”双份齐全。
|
> 数据库和文件存储必须使用同一批次成对备份；只有两份产物、完整性检查和 manifest 全部成功，才算一个可用恢复点。
|
||||||
|
|
||||||
### 1.5.1 备份
|
### 1.5.1 备份
|
||||||
```bash
|
```bash
|
||||||
@@ -174,47 +174,66 @@ set -a
|
|||||||
source deploy/trialrun/.env
|
source deploy/trialrun/.env
|
||||||
set +a
|
set +a
|
||||||
|
|
||||||
./scripts/backup-db.sh
|
DB_TOOL_MODE=docker \
|
||||||
./scripts/backup-storage.sh
|
DB_CONTAINER=hyb-oa-trialrun-db \
|
||||||
|
./scripts/backup-all.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
输出默认在：
|
输出默认在 `deploy/trialrun/backups/hyb-oa-backup-<run-id>/`，其中包含：
|
||||||
- `deploy/trialrun/backups/`
|
|
||||||
|
- `database.dump`
|
||||||
|
- `storage.tar.gz`
|
||||||
|
- `manifest.json`（相对文件名、大小和 SHA-256，不含连接串或源目录）
|
||||||
|
|
||||||
|
整批复制到获批的异机/异盘位置并记录保留周期；不能只复制 DB 或附件中的一份。
|
||||||
|
|
||||||
|
### 1.5.2 本地隔离恢复演练
|
||||||
|
|
||||||
|
开发机具备 Docker、PostgreSQL 客户端和 Node.js 时，使用合成数据运行：
|
||||||
|
|
||||||
### 1.5.2 恢复（到同一台机器或新机器）
|
|
||||||
1) 先停止 server：
|
|
||||||
```bash
|
```bash
|
||||||
docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env stop server
|
./scripts/qa/run-backup-restore-drill.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
2) 恢复 DB：
|
该命令创建一次性 PostgreSQL 容器和临时附件目录，验证 DB/附件往返、manifest 校验和、错误数据库确认、损坏备份、非空目标目录和危险 tar 路径，结束后只清理本次创建的明确目标。它不读取 `deploy/trialrun/.env`，也不接触真实数据。
|
||||||
|
|
||||||
|
手工影子恢复必须使用独立数据库和空附件目录，并显式确认实际数据库名：
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
set -a
|
RESTORE_SCOPE=shadow \
|
||||||
source deploy/trialrun/.env
|
RESTORE_CONFIRM_DATABASE=<shadow_database_name> \
|
||||||
set +a
|
DB_TOOL_MODE=docker \
|
||||||
|
DB_CONTAINER=<shadow_postgres_container> \
|
||||||
./scripts/restore-db.sh deploy/trialrun/backups/<hyb-oa-db-*.dump>
|
DATABASE_URL=<shadow_database_url> \
|
||||||
|
POSTGRES_DB=<shadow_database_name> \
|
||||||
|
POSTGRES_USER=<shadow_database_user> \
|
||||||
|
POSTGRES_PASSWORD=<shadow_database_password> \
|
||||||
|
./scripts/restore-db.sh deploy/trialrun/backups/<batch>/database.dump
|
||||||
```
|
```
|
||||||
|
|
||||||
3) 恢复文件存储：
|
|
||||||
```bash
|
```bash
|
||||||
./scripts/restore-storage.sh deploy/trialrun/backups/<hyb-oa-storage-*.tar.gz> deploy/trialrun/data/storage
|
RESTORE_SCOPE=shadow \
|
||||||
|
./scripts/restore-storage.sh \
|
||||||
|
deploy/trialrun/backups/<batch>/storage.tar.gz \
|
||||||
|
<empty_shadow_storage_directory>
|
||||||
```
|
```
|
||||||
|
|
||||||
4) 重启并验收：
|
`restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记；数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。
|
||||||
```bash
|
|
||||||
docker-compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d server
|
|
||||||
curl -fsS http://127.0.0.1:3000/ready
|
|
||||||
```
|
|
||||||
|
|
||||||
### 1.5.3 R66 小团队上线前演练记录
|
### 1.5.3 真实事故恢复
|
||||||
|
|
||||||
|
真实库/附件恢复不是普通脚本路径。必须另建或更新获批 Trellis task，记录停机窗口、恢复批次、实际目标、两名负责人、当前备份、回滚点和验证清单。当前恢复脚本不会接受 `production` 模式，不得通过修改环境变量或脚本临时绕过。恢复后必须验证 `/ready`、运行 commit、核心权限和 DB/附件关联完整性。
|
||||||
|
|
||||||
|
### 1.5.4 小团队上线前演练记录
|
||||||
上线前至少记录一次演练结果：
|
上线前至少记录一次演练结果：
|
||||||
|
|
||||||
- 演练日期：
|
- 演练日期：
|
||||||
- 执行人：
|
- 执行人：
|
||||||
- DB 备份文件：
|
- DB 备份文件：
|
||||||
- 文件存储备份文件：
|
- 文件存储备份文件：
|
||||||
|
- manifest 与 SHA-256 校验：
|
||||||
- 恢复目标机器或目录：
|
- 恢复目标机器或目录：
|
||||||
|
- 本地合成数据 RTO：
|
||||||
- 恢复后 `/ready` 结果：
|
- 恢复后 `/ready` 结果：
|
||||||
- 上个可回滚 tag：
|
- 上个可回滚 tag：
|
||||||
- 回滚负责人：
|
- 回滚负责人：
|
||||||
@@ -223,6 +242,7 @@ curl -fsS http://127.0.0.1:3000/ready
|
|||||||
|
|
||||||
- 提供或确认真实 `deploy/trialrun/.env`，特别是 `JWT_SECRET`、`APP_ORIGIN`、`COOKIE_SECURE`、`OPENAPI_DOCS_ENABLED`。
|
- 提供或确认真实 `deploy/trialrun/.env`，特别是 `JWT_SECRET`、`APP_ORIGIN`、`COOKIE_SECURE`、`OPENAPI_DOCS_ENABLED`。
|
||||||
- 确认备份文件保存位置和保留周期。
|
- 确认备份文件保存位置和保留周期。
|
||||||
|
- 确认完整批次已经复制到异机/异盘，而不是仍只保存在应用主机。
|
||||||
- 准备独立 smoke 账号，避免使用老板、管理员或真实员工账号跑自动化。
|
- 准备独立 smoke 账号，避免使用老板、管理员或真实员工账号跑自动化。
|
||||||
|
|
||||||
## 2. 升级流程（试运行）
|
## 2. 升级流程（试运行）
|
||||||
|
|||||||
可执行文件
+93
@@ -0,0 +1,93 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
BACKUP_DIR="${BACKUP_DIR:-$ROOT_DIR/deploy/trialrun/backups}"
|
||||||
|
BACKUP_RUN_ID="${BACKUP_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||||
|
|
||||||
|
if [[ ! "$BACKUP_RUN_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,79}$ ]]; then
|
||||||
|
echo "[ERR] BACKUP_RUN_ID must use 1-80 safe characters" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$BACKUP_DIR"
|
||||||
|
FINAL_DIR="$BACKUP_DIR/hyb-oa-backup-$BACKUP_RUN_ID"
|
||||||
|
if [[ -e "$FINAL_DIR" ]]; then
|
||||||
|
echo "[ERR] backup batch already exists: hyb-oa-backup-$BACKUP_RUN_ID" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
TEMP_DIR="$(mktemp -d "$BACKUP_DIR/.hyb-oa-backup-$BACKUP_RUN_ID.tmp.XXXXXX")"
|
||||||
|
cleanup() {
|
||||||
|
if [[ -n "${TEMP_DIR:-}" && -d "$TEMP_DIR" && "$TEMP_DIR" == "$BACKUP_DIR"/.hyb-oa-backup-*.tmp.* ]]; then
|
||||||
|
rm -rf -- "$TEMP_DIR"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
DB_NAME="database.dump"
|
||||||
|
STORAGE_NAME="storage.tar.gz"
|
||||||
|
MANIFEST_NAME="manifest.json"
|
||||||
|
DB_FILE="$TEMP_DIR/$DB_NAME"
|
||||||
|
STORAGE_FILE="$TEMP_DIR/$STORAGE_NAME"
|
||||||
|
|
||||||
|
BACKUP_DIR="$TEMP_DIR" OUT_FILE="$DB_FILE" "$ROOT_DIR/scripts/backup-db.sh"
|
||||||
|
BACKUP_DIR="$TEMP_DIR" OUT_FILE="$STORAGE_FILE" "$ROOT_DIR/scripts/backup-storage.sh"
|
||||||
|
|
||||||
|
DB_VALIDATED="false"
|
||||||
|
if [[ "${DB_TOOL_MODE:-auto}" != "docker" ]] && command -v pg_restore >/dev/null 2>&1; then
|
||||||
|
if pg_restore --list "$DB_FILE" >/dev/null; then
|
||||||
|
DB_VALIDATED="true"
|
||||||
|
elif [[ "${DB_TOOL_MODE:-auto}" == "host" ]]; then
|
||||||
|
echo "[ERR] host pg_restore could not validate the DB dump" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if [[ "$DB_VALIDATED" != "true" ]] && command -v docker >/dev/null 2>&1 && [[ -n "${DB_CONTAINER:-}" ]]; then
|
||||||
|
docker exec -i "$DB_CONTAINER" pg_restore --list <"$DB_FILE" >/dev/null
|
||||||
|
DB_VALIDATED="true"
|
||||||
|
fi
|
||||||
|
if [[ "$DB_VALIDATED" != "true" ]]; then
|
||||||
|
echo "[ERR] pg_restore or DB_CONTAINER is required to validate the DB dump" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
tar -tzf "$STORAGE_FILE" >/dev/null
|
||||||
|
while IFS= read -r listing_line; do
|
||||||
|
if [[ "${listing_line:0:1}" == "l" || "${listing_line:0:1}" == "h" ]]; then
|
||||||
|
echo "[ERR] storage backup contains links that the safe restore path will reject" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done < <(tar -tvzf "$STORAGE_FILE")
|
||||||
|
|
||||||
|
DB_SHA256="$(sha256sum "$DB_FILE" | awk '{print $1}')"
|
||||||
|
STORAGE_SHA256="$(sha256sum "$STORAGE_FILE" | awk '{print $1}')"
|
||||||
|
DB_BYTES="$(wc -c <"$DB_FILE" | tr -d ' ')"
|
||||||
|
STORAGE_BYTES="$(wc -c <"$STORAGE_FILE" | tr -d ' ')"
|
||||||
|
BUILD_SHA="$(git -C "$ROOT_DIR" rev-parse --short=12 HEAD 2>/dev/null || true)"
|
||||||
|
CREATED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
|
||||||
|
node - "$TEMP_DIR/$MANIFEST_NAME" "$BACKUP_RUN_ID" "$CREATED_AT" "$BUILD_SHA" \
|
||||||
|
"$DB_NAME" "$DB_BYTES" "$DB_SHA256" "$STORAGE_NAME" "$STORAGE_BYTES" "$STORAGE_SHA256" <<'NODE'
|
||||||
|
const fs = require("node:fs");
|
||||||
|
const [manifestPath, runId, createdAt, buildSha, dbFile, dbBytes, dbSha256, storageFile, storageBytes, storageSha256] = process.argv.slice(2);
|
||||||
|
const manifest = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
runId,
|
||||||
|
createdAt,
|
||||||
|
buildSha: buildSha || null,
|
||||||
|
status: "complete",
|
||||||
|
artifacts: {
|
||||||
|
database: { file: dbFile, bytes: Number(dbBytes), sha256: dbSha256 },
|
||||||
|
storage: { file: storageFile, bytes: Number(storageBytes), sha256: storageSha256 },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
fs.writeFileSync(manifestPath, `${JSON.stringify(manifest, null, 2)}\n`, { encoding: "utf8", flag: "wx" });
|
||||||
|
NODE
|
||||||
|
|
||||||
|
mv "$TEMP_DIR" "$FINAL_DIR"
|
||||||
|
TEMP_DIR=""
|
||||||
|
trap - EXIT
|
||||||
|
|
||||||
|
echo "[OK] Paired backup complete: hyb-oa-backup-$BACKUP_RUN_ID"
|
||||||
|
echo "[INFO] Copy the complete batch directory to approved off-host/off-disk storage"
|
||||||
+44
-7
@@ -9,17 +9,46 @@ mkdir -p "$BACKUP_DIR"
|
|||||||
TS="$(date +%Y%m%d_%H%M%S)"
|
TS="$(date +%Y%m%d_%H%M%S)"
|
||||||
OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-db-${TS}.dump}"
|
OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-db-${TS}.dump}"
|
||||||
|
|
||||||
|
if [[ -e "$OUT_FILE" ]]; then
|
||||||
|
echo "[ERR] backup output already exists: $OUT_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$OUT_FILE")"
|
||||||
|
|
||||||
if [[ -z "${DATABASE_URL:-}" ]]; then
|
if [[ -z "${DATABASE_URL:-}" ]]; then
|
||||||
echo "[ERR] DATABASE_URL is required" >&2
|
echo "[ERR] DATABASE_URL is required" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
DB_TOOL_MODE="${DB_TOOL_MODE:-auto}"
|
||||||
|
if [[ "$DB_TOOL_MODE" != "auto" && "$DB_TOOL_MODE" != "host" && "$DB_TOOL_MODE" != "docker" ]]; then
|
||||||
|
echo "[ERR] DB_TOOL_MODE must be auto, host, or docker" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
echo "[INFO] DB backup start: $OUT_FILE"
|
echo "[INFO] DB backup start: $OUT_FILE"
|
||||||
|
|
||||||
if command -v pg_dump >/dev/null 2>&1; then
|
if [[ "$DB_TOOL_MODE" != "docker" ]] && command -v pg_dump >/dev/null 2>&1; then
|
||||||
pg_dump "$DATABASE_URL" --format=c --no-owner --no-acl --file "$OUT_FILE"
|
if pg_dump "$DATABASE_URL" --format=c --no-owner --no-acl --file "$OUT_FILE"; then
|
||||||
echo "[OK] DB backup done (pg_dump): $OUT_FILE"
|
if [[ ! -s "$OUT_FILE" ]]; then
|
||||||
exit 0
|
echo "[ERR] DB backup is empty: $OUT_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "[OK] DB backup done (pg_dump): $OUT_FILE"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
rm -f -- "$OUT_FILE"
|
||||||
|
if [[ "$DB_TOOL_MODE" == "host" ]]; then
|
||||||
|
echo "[ERR] host pg_dump failed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "[WARN] host pg_dump failed; trying docker fallback" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DB_TOOL_MODE" == "host" ]]; then
|
||||||
|
echo "[ERR] DB_TOOL_MODE=host requires pg_dump" >&2
|
||||||
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if ! command -v docker >/dev/null 2>&1; then
|
if ! command -v docker >/dev/null 2>&1; then
|
||||||
@@ -37,9 +66,17 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
if ! docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||||
pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --format=c --no-owner --no-acl \
|
pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --format=c --no-owner --no-acl \
|
||||||
>"$OUT_FILE"
|
>"$OUT_FILE"; then
|
||||||
|
rm -f -- "$OUT_FILE"
|
||||||
|
echo "[ERR] docker pg_dump failed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -s "$OUT_FILE" ]]; then
|
||||||
|
echo "[ERR] DB backup is empty: $OUT_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
echo "[OK] DB backup done (docker exec): $OUT_FILE"
|
echo "[OK] DB backup done (docker exec): $OUT_FILE"
|
||||||
|
|
||||||
|
|||||||
+22
-2
@@ -9,6 +9,11 @@ mkdir -p "$BACKUP_DIR"
|
|||||||
TS="$(date +%Y%m%d_%H%M%S)"
|
TS="$(date +%Y%m%d_%H%M%S)"
|
||||||
OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-storage-${TS}.tar.gz}"
|
OUT_FILE="${OUT_FILE:-$BACKUP_DIR/hyb-oa-storage-${TS}.tar.gz}"
|
||||||
|
|
||||||
|
if [[ -e "$OUT_FILE" ]]; then
|
||||||
|
echo "[ERR] backup output already exists: $OUT_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
STORAGE_DIR_HOST="${STORAGE_DIR_HOST:-}"
|
STORAGE_DIR_HOST="${STORAGE_DIR_HOST:-}"
|
||||||
if [[ -z "$STORAGE_DIR_HOST" ]]; then
|
if [[ -z "$STORAGE_DIR_HOST" ]]; then
|
||||||
if [[ -n "${STORAGE_DIR:-}" && -d "${STORAGE_DIR:-}" ]]; then
|
if [[ -n "${STORAGE_DIR:-}" && -d "${STORAGE_DIR:-}" ]]; then
|
||||||
@@ -21,10 +26,25 @@ if [[ -z "$STORAGE_DIR_HOST" ]]; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
if [[ ! -d "$STORAGE_DIR_HOST" ]]; then
|
||||||
|
echo "[ERR] storage dir not found: $STORAGE_DIR_HOST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$OUT_FILE")"
|
||||||
|
|
||||||
echo "[INFO] Storage backup start: $OUT_FILE"
|
echo "[INFO] Storage backup start: $OUT_FILE"
|
||||||
echo "[INFO] Storage dir: $STORAGE_DIR_HOST"
|
echo "[INFO] Storage dir: $STORAGE_DIR_HOST"
|
||||||
|
|
||||||
tar -czf "$OUT_FILE" -C "$STORAGE_DIR_HOST" .
|
if ! tar -czf "$OUT_FILE" -C "$STORAGE_DIR_HOST" .; then
|
||||||
|
rm -f -- "$OUT_FILE"
|
||||||
|
echo "[ERR] storage backup failed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ! -s "$OUT_FILE" ]]; then
|
||||||
|
echo "[ERR] storage backup is empty: $OUT_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
echo "[OK] Storage backup done: $OUT_FILE"
|
echo "[OK] Storage backup done: $OUT_FILE"
|
||||||
|
|
||||||
|
|||||||
+150
@@ -0,0 +1,150 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||||
|
DRILL_RUN_ID="drill-$(date -u +%Y%m%dT%H%M%SZ)-$$"
|
||||||
|
CONTAINER_NAME="hyb-oa-backup-drill-$$"
|
||||||
|
DRILL_DIR="$(mktemp -d)"
|
||||||
|
POSTGRES_USER="drill_user"
|
||||||
|
POSTGRES_PASSWORD="drill_password"
|
||||||
|
POSTGRES_DB="hyb_oa_restore_shadow"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
|
||||||
|
if [[ -n "${DRILL_DIR:-}" && -d "$DRILL_DIR" && "$DRILL_DIR" == /tmp/* ]]; then
|
||||||
|
rm -rf -- "$DRILL_DIR"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
require_command() {
|
||||||
|
if ! command -v "$1" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] required command missing: $1" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
for command_name in docker pg_dump pg_restore psql tar sha256sum node; do
|
||||||
|
require_command "$command_name"
|
||||||
|
done
|
||||||
|
|
||||||
|
START_EPOCH="$(date +%s)"
|
||||||
|
SOURCE_STORAGE="$DRILL_DIR/source-storage"
|
||||||
|
RESTORED_STORAGE="$DRILL_DIR/restored-storage"
|
||||||
|
NONEMPTY_STORAGE="$DRILL_DIR/nonempty-storage"
|
||||||
|
BACKUP_ROOT="$DRILL_DIR/backups"
|
||||||
|
mkdir -p "$SOURCE_STORAGE/nested" "$NONEMPTY_STORAGE" "$BACKUP_ROOT"
|
||||||
|
printf 'synthetic attachment for restore drill\n' >"$SOURCE_STORAGE/nested/marker.txt"
|
||||||
|
printf 'must not be overwritten\n' >"$NONEMPTY_STORAGE/existing.txt"
|
||||||
|
|
||||||
|
docker run -d --rm --name "$CONTAINER_NAME" \
|
||||||
|
-e "POSTGRES_USER=$POSTGRES_USER" \
|
||||||
|
-e "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" \
|
||||||
|
-e "POSTGRES_DB=$POSTGRES_DB" \
|
||||||
|
-p 127.0.0.1::5432 postgres:16-alpine >/dev/null
|
||||||
|
|
||||||
|
for _ in $(seq 1 60); do
|
||||||
|
if docker exec "$CONTAINER_NAME" pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" >/dev/null 2>&1; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
docker exec "$CONTAINER_NAME" pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" >/dev/null
|
||||||
|
|
||||||
|
HOST_PORT="$(docker port "$CONTAINER_NAME" 5432/tcp | awk -F: 'NR==1 {print $NF}')"
|
||||||
|
DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@127.0.0.1:$HOST_PORT/$POSTGRES_DB"
|
||||||
|
export DATABASE_URL STORAGE_DIR_HOST="$SOURCE_STORAGE" BACKUP_DIR="$BACKUP_ROOT" BACKUP_RUN_ID="$DRILL_RUN_ID"
|
||||||
|
export DB_TOOL_MODE="docker" DB_CONTAINER="$CONTAINER_NAME" POSTGRES_USER POSTGRES_PASSWORD POSTGRES_DB
|
||||||
|
|
||||||
|
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc 'CREATE TABLE restore_drill_marker (value text NOT NULL)'
|
||||||
|
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc "INSERT INTO restore_drill_marker(value) VALUES ('synthetic-round-trip')"
|
||||||
|
|
||||||
|
"$ROOT_DIR/scripts/backup-all.sh"
|
||||||
|
BATCH_DIR="$BACKUP_ROOT/hyb-oa-backup-$DRILL_RUN_ID"
|
||||||
|
DB_BACKUP="$BATCH_DIR/database.dump"
|
||||||
|
STORAGE_BACKUP="$BATCH_DIR/storage.tar.gz"
|
||||||
|
|
||||||
|
node - "$BATCH_DIR/manifest.json" "$DB_BACKUP" "$STORAGE_BACKUP" <<'NODE'
|
||||||
|
const crypto = require("node:crypto");
|
||||||
|
const fs = require("node:fs");
|
||||||
|
const path = require("node:path");
|
||||||
|
const [manifestPath, dbPath, storagePath] = process.argv.slice(2);
|
||||||
|
const manifestText = fs.readFileSync(manifestPath, "utf8");
|
||||||
|
const manifest = JSON.parse(manifestText);
|
||||||
|
if (manifest.status !== "complete" || manifest.schemaVersion !== 1) throw new Error("manifest contract mismatch");
|
||||||
|
if (path.isAbsolute(manifest.artifacts.database.file) || path.isAbsolute(manifest.artifacts.storage.file)) {
|
||||||
|
throw new Error("manifest contains absolute artifact path");
|
||||||
|
}
|
||||||
|
if (/postgresql:|DATABASE_URL|drill_password/.test(manifestText)) throw new Error("manifest contains sensitive connection data");
|
||||||
|
const sha256 = (file) => crypto.createHash("sha256").update(fs.readFileSync(file)).digest("hex");
|
||||||
|
if (sha256(dbPath) !== manifest.artifacts.database.sha256) throw new Error("database checksum mismatch");
|
||||||
|
if (sha256(storagePath) !== manifest.artifacts.storage.sha256) throw new Error("storage checksum mismatch");
|
||||||
|
NODE
|
||||||
|
|
||||||
|
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -qc 'DROP TABLE restore_drill_marker'
|
||||||
|
|
||||||
|
docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$CONTAINER_NAME" \
|
||||||
|
createdb -U "$POSTGRES_USER" hyb_oa_prod
|
||||||
|
if DATABASE_URL="postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@127.0.0.1:$HOST_PORT/hyb_oa_prod" \
|
||||||
|
POSTGRES_DB=hyb_oa_prod RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=hyb_oa_prod \
|
||||||
|
"$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] production-like DB name was accepted as a shadow target" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
CORRUPT_DB="$DRILL_DIR/corrupt.dump"
|
||||||
|
head -c 32 "$DB_BACKUP" >"$CORRUPT_DB"
|
||||||
|
if RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE="$POSTGRES_DB" \
|
||||||
|
"$ROOT_DIR/scripts/restore-db.sh" "$CORRUPT_DB" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] corrupt DB dump was accepted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE=wrong_target \
|
||||||
|
"$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] mismatched DB confirmation was accepted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
RESTORE_SCOPE=shadow RESTORE_CONFIRM_DATABASE="$POSTGRES_DB" \
|
||||||
|
"$ROOT_DIR/scripts/restore-db.sh" "$DB_BACKUP"
|
||||||
|
|
||||||
|
RESTORED_VALUE="$(psql "$DATABASE_URL" -Atqc 'SELECT value FROM restore_drill_marker')"
|
||||||
|
if [[ "$RESTORED_VALUE" != "synthetic-round-trip" ]]; then
|
||||||
|
echo "[ERR] DB round-trip marker mismatch" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$STORAGE_BACKUP" "$NONEMPTY_STORAGE" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] non-empty storage target was accepted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" \
|
||||||
|
"$STORAGE_BACKUP" "$ROOT_DIR/deploy/trialrun/data/storage" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] default live storage path was accepted as a shadow target" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$STORAGE_BACKUP" "$RESTORED_STORAGE"
|
||||||
|
cmp "$SOURCE_STORAGE/nested/marker.txt" "$RESTORED_STORAGE/nested/marker.txt"
|
||||||
|
|
||||||
|
CORRUPT_TAR="$DRILL_DIR/corrupt.tar.gz"
|
||||||
|
head -c 32 "$STORAGE_BACKUP" >"$CORRUPT_TAR"
|
||||||
|
if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$CORRUPT_TAR" "$DRILL_DIR/corrupt-target" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] corrupt storage archive was accepted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
DANGEROUS_TAR="$DRILL_DIR/dangerous.tar.gz"
|
||||||
|
tar -czf "$DANGEROUS_TAR" --transform='s|nested/marker.txt|../escape.txt|' \
|
||||||
|
-C "$SOURCE_STORAGE" nested/marker.txt
|
||||||
|
if RESTORE_SCOPE=shadow "$ROOT_DIR/scripts/restore-storage.sh" "$DANGEROUS_TAR" "$DRILL_DIR/dangerous-target" >/dev/null 2>&1; then
|
||||||
|
echo "[ERR] dangerous storage archive path was accepted" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
END_EPOCH="$(date +%s)"
|
||||||
|
ELAPSED_SECONDS="$((END_EPOCH - START_EPOCH))"
|
||||||
|
echo "[PASS] isolated backup/restore drill completed"
|
||||||
|
echo "[INFO] local synthetic RTO seconds: $ELAPSED_SECONDS"
|
||||||
+73
-10
@@ -1,8 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
||||||
|
|
||||||
BACKUP_FILE="${1:-${BACKUP_FILE:-}}"
|
BACKUP_FILE="${1:-${BACKUP_FILE:-}}"
|
||||||
if [[ -z "${BACKUP_FILE:-}" ]]; then
|
if [[ -z "${BACKUP_FILE:-}" ]]; then
|
||||||
echo "Usage: $0 <backup-file.dump>" >&2
|
echo "Usage: $0 <backup-file.dump>" >&2
|
||||||
@@ -19,16 +17,69 @@ if [[ -z "${DATABASE_URL:-}" ]]; then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "[INFO] DB restore start: $BACKUP_FILE"
|
RESTORE_SCOPE="${RESTORE_SCOPE:-}"
|
||||||
|
RESTORE_CONFIRM_DATABASE="${RESTORE_CONFIRM_DATABASE:-}"
|
||||||
|
DB_TOOL_MODE="${DB_TOOL_MODE:-auto}"
|
||||||
|
|
||||||
if command -v pg_restore >/dev/null 2>&1; then
|
if [[ "$DB_TOOL_MODE" != "auto" && "$DB_TOOL_MODE" != "host" && "$DB_TOOL_MODE" != "docker" ]]; then
|
||||||
|
echo "[ERR] DB_TOOL_MODE must be auto, host, or docker" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$RESTORE_SCOPE" != "shadow" ]]; then
|
||||||
|
echo "[ERR] RESTORE_SCOPE=shadow is required; production restore is not available from this command path" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -z "$RESTORE_CONFIRM_DATABASE" ]]; then
|
||||||
|
echo "[ERR] RESTORE_CONFIRM_DATABASE is required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
validate_dump_with_host() {
|
||||||
|
pg_restore --list "$BACKUP_FILE" >/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
validate_shadow_database_name() {
|
||||||
|
local database_name="$1"
|
||||||
|
if [[ ! "$database_name" =~ (^|[_-])(shadow|drill|restore_test)([_-]|$) ]]; then
|
||||||
|
echo "[ERR] restore target name is not an explicit shadow/drill database" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
restore_with_host() {
|
||||||
|
local actual_database
|
||||||
|
actual_database="$(psql "$DATABASE_URL" -Atqc 'SELECT current_database()')"
|
||||||
|
if [[ -z "$actual_database" || "$actual_database" != "$RESTORE_CONFIRM_DATABASE" ]]; then
|
||||||
|
echo "[ERR] restore target confirmation mismatch: actual=${actual_database:-unknown}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
validate_shadow_database_name "$actual_database"
|
||||||
|
echo "[INFO] DB restore target confirmed: $actual_database (shadow)"
|
||||||
pg_restore --clean --if-exists --no-owner --no-acl -d "$DATABASE_URL" "$BACKUP_FILE"
|
pg_restore --clean --if-exists --no-owner --no-acl -d "$DATABASE_URL" "$BACKUP_FILE"
|
||||||
echo "[OK] DB restore done (pg_restore): $BACKUP_FILE"
|
}
|
||||||
exit 0
|
|
||||||
|
if [[ "$DB_TOOL_MODE" != "docker" ]] && command -v pg_restore >/dev/null 2>&1 && command -v psql >/dev/null 2>&1; then
|
||||||
|
if validate_dump_with_host; then
|
||||||
|
restore_with_host
|
||||||
|
echo "[OK] DB restore done (pg_restore): $BACKUP_FILE"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if [[ "$DB_TOOL_MODE" == "host" ]]; then
|
||||||
|
echo "[ERR] host pg_restore could not validate the dump" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "[WARN] host pg_restore could not validate the dump; trying docker fallback" >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$DB_TOOL_MODE" == "host" ]]; then
|
||||||
|
echo "[ERR] DB_TOOL_MODE=host requires pg_restore and psql" >&2
|
||||||
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if ! command -v docker >/dev/null 2>&1; then
|
if ! command -v docker >/dev/null 2>&1; then
|
||||||
echo "[ERR] pg_restore not found and docker not available" >&2
|
echo "[ERR] pg_restore/psql not found and docker not available" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -42,8 +93,20 @@ if [[ -z "$POSTGRES_DB" || -z "$POSTGRES_USER" || -z "$POSTGRES_PASSWORD" ]]; th
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
cat "$BACKUP_FILE" | docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
ACTUAL_DATABASE="$(docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||||
pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB"
|
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atqc 'SELECT current_database()')"
|
||||||
|
|
||||||
|
if [[ -z "$ACTUAL_DATABASE" || "$ACTUAL_DATABASE" != "$RESTORE_CONFIRM_DATABASE" ]]; then
|
||||||
|
echo "[ERR] restore target confirmation mismatch: actual=${ACTUAL_DATABASE:-unknown}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
validate_shadow_database_name "$ACTUAL_DATABASE"
|
||||||
|
|
||||||
|
docker exec -i "$DB_CONTAINER" pg_restore --list <"$BACKUP_FILE" >/dev/null
|
||||||
|
|
||||||
|
echo "[INFO] DB restore target confirmed: $ACTUAL_DATABASE (shadow)"
|
||||||
|
docker exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" "$DB_CONTAINER" \
|
||||||
|
pg_restore --clean --if-exists --no-owner --no-acl -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
|
||||||
|
<"$BACKUP_FILE"
|
||||||
|
|
||||||
echo "[OK] DB restore done (docker exec): $BACKUP_FILE"
|
echo "[OK] DB restore done (docker exec): $BACKUP_FILE"
|
||||||
|
|
||||||
|
|||||||
+51
-9
@@ -16,19 +16,61 @@ fi
|
|||||||
|
|
||||||
STORAGE_DIR_HOST="${2:-${STORAGE_DIR_HOST:-}}"
|
STORAGE_DIR_HOST="${2:-${STORAGE_DIR_HOST:-}}"
|
||||||
if [[ -z "$STORAGE_DIR_HOST" ]]; then
|
if [[ -z "$STORAGE_DIR_HOST" ]]; then
|
||||||
if [[ -n "${STORAGE_DIR:-}" && -d "${STORAGE_DIR:-}" ]]; then
|
echo "[ERR] explicit restore target is required as the second argument or STORAGE_DIR_HOST" >&2
|
||||||
STORAGE_DIR_HOST="$STORAGE_DIR"
|
exit 1
|
||||||
else
|
|
||||||
STORAGE_DIR_HOST="$ROOT_DIR/deploy/trialrun/data/storage"
|
|
||||||
fi
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
mkdir -p "$STORAGE_DIR_HOST"
|
if [[ "${RESTORE_SCOPE:-}" != "shadow" ]]; then
|
||||||
|
echo "[ERR] RESTORE_SCOPE=shadow is required; production restore is not available from this command path" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
TARGET_PATH="$(realpath -m "$STORAGE_DIR_HOST")"
|
||||||
|
LIVE_STORAGE_PATH="$(realpath -m "$ROOT_DIR/deploy/trialrun/data/storage")"
|
||||||
|
if [[ "$TARGET_PATH" == "/" || "$TARGET_PATH" == "$ROOT_DIR" || "$TARGET_PATH" == "$LIVE_STORAGE_PATH" ]]; then
|
||||||
|
echo "[ERR] unsafe storage restore target" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -d "$TARGET_PATH" && -n "$(find "$TARGET_PATH" -mindepth 1 -print -quit)" ]]; then
|
||||||
|
echo "[ERR] storage restore target must be empty: $TARGET_PATH" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
validate_archive_entries() {
|
||||||
|
local entry normalized part listing_line
|
||||||
|
tar -tzf "$BACKUP_FILE" >/dev/null
|
||||||
|
while IFS= read -r entry; do
|
||||||
|
if [[ -z "$entry" || "$entry" == /* ]]; then
|
||||||
|
echo "[ERR] unsafe archive path" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
normalized="${entry#./}"
|
||||||
|
[[ -z "$normalized" || "$normalized" == "." ]] && continue
|
||||||
|
IFS='/' read -r -a path_parts <<<"$normalized"
|
||||||
|
for part in "${path_parts[@]}"; do
|
||||||
|
if [[ "$part" == ".." ]]; then
|
||||||
|
echo "[ERR] unsafe archive path segment" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
done < <(tar -tzf "$BACKUP_FILE")
|
||||||
|
|
||||||
|
tar -tvzf "$BACKUP_FILE" >/dev/null
|
||||||
|
while IFS= read -r listing_line; do
|
||||||
|
if [[ "${listing_line:0:1}" == "l" || "${listing_line:0:1}" == "h" ]]; then
|
||||||
|
echo "[ERR] storage archive links are not allowed" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
done < <(tar -tvzf "$BACKUP_FILE")
|
||||||
|
}
|
||||||
|
|
||||||
|
validate_archive_entries
|
||||||
|
mkdir -p "$TARGET_PATH"
|
||||||
|
|
||||||
echo "[INFO] Storage restore start: $BACKUP_FILE"
|
echo "[INFO] Storage restore start: $BACKUP_FILE"
|
||||||
echo "[INFO] Storage dir: $STORAGE_DIR_HOST"
|
echo "[INFO] Storage target confirmed: shadow"
|
||||||
|
|
||||||
tar -xzf "$BACKUP_FILE" -C "$STORAGE_DIR_HOST"
|
tar -xzf "$BACKUP_FILE" --no-same-owner --no-same-permissions -C "$TARGET_PATH"
|
||||||
|
|
||||||
echo "[OK] Storage restore done: $BACKUP_FILE"
|
echo "[OK] Storage restore done: $BACKUP_FILE"
|
||||||
|
|
||||||
|
|||||||
+4
-4
@@ -298,12 +298,12 @@ SERVE_CLIENT_DIST=true npm run dev
|
|||||||
备份/恢复脚本（E14）：
|
备份/恢复脚本（E14）：
|
||||||
```bash
|
```bash
|
||||||
cd <repo-root>
|
cd <repo-root>
|
||||||
./scripts/backup-db.sh
|
DB_TOOL_MODE=docker DB_CONTAINER=hyb-oa-trialrun-db ./scripts/backup-all.sh
|
||||||
./scripts/restore-db.sh deploy/trialrun/backups/<db.dump>
|
./scripts/qa/run-backup-restore-drill.sh
|
||||||
./scripts/backup-storage.sh
|
|
||||||
./scripts/restore-storage.sh deploy/trialrun/backups/<storage.tar.gz> deploy/trialrun/data/storage
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
成对备份会生成 DB、附件和脱敏 manifest。恢复脚本只允许显式影子目标；真实事故恢复、异机副本和 `/ready` 验收按 `deploy/trialrun/runbook.md` 的授权门执行，不直接在 README 复制破坏性命令。
|
||||||
|
|
||||||
优雅停机（E21）：
|
优雅停机（E21）：
|
||||||
- `docker stop` / `Ctrl+C` 会触发 `SIGTERM/SIGINT`，server 会调用 `app.close()` 完成收敛关闭（定时任务/HTTP/DB）。
|
- `docker stop` / `Ctrl+C` 会触发 `SIGTERM/SIGINT`，server 会调用 `app.close()` 完成收敛关闭（定时任务/HTTP/DB）。
|
||||||
|
|
||||||
|
|||||||
在新工单中引用
屏蔽一个用户