diff --git a/.trellis/spec/hyb-oa/deployment-trialrun.md b/.trellis/spec/hyb-oa/deployment-trialrun.md index 11c2d59a..3a6f5416 100644 --- a/.trellis/spec/hyb-oa/deployment-trialrun.md +++ b/.trellis/spec/hyb-oa/deployment-trialrun.md @@ -927,13 +927,18 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ - Production runner: `scripts/run-automatic-backup.sh` with `run`, `check-stale` and `remind-drill` modes. - ARM forced receiver: `scripts/receive-offhost-backup.sh`. +- Shared host alert helper: `scripts/send-wecom-group-alert.py` with explicit + `--webhook-file`, `--summary-file` and `--state-file` paths. +- ARM watchdog: `scripts/check-offhost-backup-deadman.py` with archive root, + state root, alert helper, Webhook file and a default `93600` second threshold. - Installers: `install-automatic-backup-timer.sh` and `install-offhost-backup-receiver.sh`. - Local contract: `bash scripts/qa/run-automatic-backup-contract.sh`. - Production required environment: `REMOTE_DIR`, `REMOTE_ENV_FILE`, `COMPOSE_PROJECT_NAME`, `POSTGRES_DATA_DIR`, `STORAGE_HOST_DIR`, `BUILD_SHA`, `VITE_BUILD_SHA`, `AUTOMATIC_BACKUP_DIR`, `AUTOMATIC_BACKUP_STATE_DIR`, - `OFFHOST_TARGET`, `OFFHOST_SSH_KEY` and `OFFHOST_KNOWN_HOSTS`. + `OFFHOST_TARGET`, `OFFHOST_SSH_KEY`, `OFFHOST_KNOWN_HOSTS`, + `ALERT_WEBHOOK_FILE` and `ALERT_HELPER`. - Receiver config: independent absolute `BACKUP_INBOX_DIR`, `BACKUP_ARCHIVE_DIR`, `BACKUP_WEEKLY_DIR`, `RRSYNC_BIN`, `PYTHON_BIN`, `DAILY_KEEP` and `WEEKLY_KEEP`. @@ -952,14 +957,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ it rejects shell, sender/read and destructive rsync options. - ARM revalidates manifest identity, exact regular-file set, sizes and SHA-256, then atomically moves the batch to archive. Archive and weekly directories are - outside the rsync root; an archived name is never overwritten. + outside the rsync root. A different run ID for the same UTC day may replace + that day's archive only after full candidate validation: rename the old daily + point to a same-filesystem rollback name, publish the candidate, rebuild that + ISO week's hard-link checkpoint, then remove the rollback. Any publication or + weekly-checkpoint failure restores the prior daily point. - Write production `last-success.json` and prune source copies only after ARM finalize succeeds. Keep seven automatic source/ARM daily points and four ARM weekly checkpoints; unrelated manual batches are outside automatic retention. - Daily, freshness and quarterly reminder timers are separate. Alert summaries - contain categories only. AUTO_BACKUP sends the first failure, worsening/new - findings and the first quarterly reminder, while suppressing an identical - repeated failure or reminder. + contain categories only. A shared Python helper reads the complete WeCom group + robot URL only from a regular, executing-account-owned `0600` file and never + prints it. It sends the first failure, worsening/new findings, one recovery + notice and the first quarterly reminder, while suppressing an identical + repeated state. It does not call the OA server container or resolve an employee. +- ARM runs its own 26-hour dead-man as root. It verifies the newest archive's + exact file set, manifest identity, sizes, checksums and timestamp, so a stopped + production host/server can still cause a group alert. Missing, corrupt, stale + or future-dated archives fail; recovery sends once. +- Production command diagnostics for paired backup, rsync and finalize stay only + in the state directory as `last-error.log` mode `0600`. The summary and group + message contain only the safe phase category. A later successful off-host run + removes the retained diagnostic. +- The production freshness check hard-fails missing, malformed, future, stale or + Compose-project-mismatched state. A fresh recovery point with a different + build SHA is `PASS_WITH_FINDINGS`, because deploy-time code identity and + recoverable data freshness are separate facts. - Installer `install` is disabled-by-default; `enable`, real transfer, real WeCom alert and all server/account/key changes remain `EXTERNAL_GATED`. @@ -967,24 +990,32 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ - Source mount/build/project drift, lock conflict or incomplete backup -> fail before success state and source retention. -- Transfer/finalize failure, wrong/extra/link artifact, bad hash or second batch - for one UTC day -> leave no off-host success state and preserve source batch. -- Missing, stale or project/build-mismatched success record -> watchdog finding. +- Transfer/finalize failure, wrong/extra/link artifact or bad hash -> leave no + off-host success state, preserve the source batch and retain a root-only local + diagnostic. A corrupt same-day candidate leaves the previous archive intact. +- Valid same-day retry -> replace only that UTC day's point and rebuild its ISO + weekly checkpoint; concurrent finalize -> `flock` rejection. +- Missing, corrupt, stale, future or project-mismatched success record -> hard + watchdog failure. Fresh build mismatch -> one finding without a failing exit. +- Missing/non-regular/symlink/non-`0600` Webhook file -> fail closed without + printing the file value. Identical alert state -> update/evaluate without send. - Ordinary shell, rsync sender/read/delete or archive overwrite -> receiver rejection without changing archive. -- Host/network/WeCom all unavailable at once -> self-hosted alert cannot prove - its own outage; this is a documented physical boundary, not a green state. +- ARM host/network/WeCom all unavailable at once -> the group robot path cannot + prove its own outage; this physical boundary is not a green state. ### 5. Good/Base/Bad Cases - Good: one verified active source creates a complete batch, write-only rsync resumes inside inbox, ARM finalize verifies and archives it, then production - writes success and prunes old automatic source points. + writes success and prunes old automatic source points. If production later + stops, ARM independently raises one dead-man alert and one recovery notice. - Base: local implementation and synthetic contracts pass while both server installers remain disabled and explicitly externally gated. -- Bad: give production a normal ARM login key, put archive below inbox, accept - a new host fingerprint automatically, or prune local recovery points after - rsync but before target-side finalize. +- Bad: send through `compose exec server`, hard-code a personal employee number, + persist a robot URL in `.env`/unit/repository, give production a normal ARM + login key, put archive below inbox, accept a new host fingerprint automatically, + or prune local points before target-side finalize. ### 6. Tests Required @@ -993,28 +1024,37 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \ - Docker mocks must return the real `{{json .Mounts}}` shape consumed by the runner. A simplified tab-delimited mock is not valid evidence for real mount parsing. -- Focused WeCom alert tests for stdin path redaction, preserved AUTO_BACKUP - issue identity, first/repeated failure and quarterly reminder behavior. -- Environment contract assertions for all four scripts, three timers, explicit - project/data/build/SSH binding and disabled-by-default installation. -- Real ARM setup, first paired batch, failure alert and quarterly shadow restore - are separately approved operational evidence. +- Contract assertions for same-day replacement, corrupt-candidate preservation, + weekly hard-link rebuild, source diagnostic mode, build mismatch finding, + Webhook mode/symlink rejection and group alert behavior without a server + container. Test transport must be explicitly gated and local-only. +- Environment contract assertions for both helpers, both installers, production + timers, ARM dead-man timer, explicit project/data/build/SSH/Webhook-file + binding, disabled-by-default installation and absence of a literal complete + robot URL in tracked or untracked repository files. +- Real ARM setup, first paired batch, deliberate source-timer stop or transfer + failure, dead-man first/repeated/recovery behavior and quarterly shadow restore + are separately approved operational evidence. A direct robot send test alone + is not dead-man evidence. ### 7. Wrong vs Correct #### Wrong ```bash +compose exec server send-alert --recipient= rsync -a / backup-host:/ -rm -rf ``` #### Correct ```bash -# Install only with explicit active identity; default remains disabled. - \ +# Install only with explicit active identity and a root-only Webhook file path; +# default remains disabled. + \ bash deploy/trialrun/install-automatic-backup-timer.sh install +# ARM separately verifies archive freshness even when production is unavailable. +python3 scripts/check-offhost-backup-deadman.py # The runner writes success and prunes only after restricted target finalize. bash scripts/run-automatic-backup.sh run ``` diff --git a/.trellis/tasks/07-27-automatic-offhost-backup/design.md b/.trellis/tasks/07-27-automatic-offhost-backup/design.md index eb3539cf..36ffe9fb 100644 --- a/.trellis/tasks/07-27-automatic-offhost-backup/design.md +++ b/.trellis/tasks/07-27-automatic-offhost-backup/design.md @@ -19,7 +19,12 @@ production 02:30 -> prune local daily copies production 07:10 watchdog - -> last off-host success <= 24h ? pass : alert + -> recovery point missing/corrupt/stale/project drift ? fail + group alert + -> build SHA differs ? one finding + group alert + +ARM independent watchdog + -> newest finalized archive <= 26h ? pass : direct group alert + -> source host/server/container not required quarterly reminder -> one redacted WeCom reminder per calendar quarter @@ -39,8 +44,8 @@ quarterly reminder - ARM 一次性安装 rsync,创建密码锁定的专用账号、独立 inbox/archive/weekly 目录和一把专用公钥。该公钥的 `authorized_keys` 强制执行仓库提供的 receiver,并禁用 PTY、端口转发、agent 转发和 X11 转发。 - receiver 只接受两类 `SSH_ORIGINAL_COMMAND`:rsync server 接收模式,或 `hyb-backup-finalize `。其他命令全部拒绝。 - rsync 通过 `rrsync -wo` 只能进入 inbox;archive 和 weekly 位于 rrsync 根之外。即使生产端密钥泄露,也不能用 rsync 读取或删除已归档批次。 -- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接,manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配;已存在 archive 目标时拒绝覆盖。 -- 校验通过后在同一文件系统内原子移动到 archive。每个自然日最多接受一个正式自动批次进入保留计算,避免失陷源端通过短时间伪造大量批次提前挤掉历史。 +- finalize 只接受 `auto-` 安全批次号和完整 manifest SHA。它要求 inbox 中恰有三个普通文件、无链接,manifest 的 runId/status/schema、相对文件名、大小和 SHA-256 全部匹配;同 run_id 重复 finalize 继续保持幂等。 +- 同 UTC 日已有不同 run_id 时,先完整校验并冻结新批次权限,再把旧日归档 rename 到同文件系统临时回滚名、新批次 rename 到正式归档名;新批次发布失败立即把旧归档 rename 回原名。新批次发布成功后才删除旧回滚名,并重建该 ISO 周硬链接检查点,使每日一个恢复点和周检查点都指向当前有效批次。 - archive 保留最近 7 个日批次;每个 ISO 周首次成功批次用只读硬链接快照形成周检查点,保留最近 4 周。清理不得删除最后一个 archive 或当前 finalize 批次。 - 旧手工/审计批次不进入自动前缀和自动轮转范围。 @@ -52,19 +57,22 @@ quarterly reminder ## 告警与静默失败检测 -- wrapper 无论成功或失败都生成字段白名单 summary;失败 trap 尝试通过活动 server 容器调用现有企微运维告警 CLI,并使用备份专属 state file,避免与 WARCH3 降噪状态互相覆盖。 -- 现有告警 CLI 小范围泛化为接受 stdin summary、保留 summary issueId,并为备份失败/过期提供正确建议;发送仍受既有 `WARCH3_OPS_ALERT_SEND=1` 双门控制。 -- 独立 watchdog timer 在每日巡检前读取宿主 `last-success.json`。文件缺失、格式错误、构建/项目不匹配或成功时间超过 24 小时均产生失败 summary 并尝试告警。 -- 若 Docker、整机网络或企微本身同时不可用,主机无法自证并外发,这是自托管告警的物理边界;watchdog 能覆盖 timer 未运行和短暂失败,但不替代外部主机存活监控。 +- 新增一个小型宿主告警 helper,使用 Python 标准库从 root-only Webhook 文件读取完整 URL,并向企微运维群机器人发送固定白名单文本;不复用 OA server 容器、员工表或个人工号。 +- 生产 wrapper 无论成功或失败都生成字段白名单 summary;即时失败和源端 watchdog 直接调用宿主 helper。Webhook 文件缺失、权限不是 `0600`、不是普通文件或内容格式不合法时 fail closed,并在本机诊断中留类别,不打印 URL。 +- ARM receiver 安装器同时安装独立 dead-man service/timer。watchdog 读取 archive 中最新有效自动批次的完成时间,超过 26 小时、没有批次或 manifest 无法验证时直接调用同一 helper;状态文件实现首次失败、重复降噪和恢复通知。 +- 生产 stale watchdog 仍验证 schema/status/project/时间。build SHA 不同输出 `PASS_WITH_FINDINGS`,不返回失败;恢复点缺失、损坏、未来时间、超过阈值或 project 不同继续返回失败。 +- 生产外部命令的 stdout/stderr 只写状态目录下的临时诊断,失败时原子保留为 `last-error.log` 并设为 `0600`,成功后清理本次临时诊断。summary 和企微文本不得复制诊断正文。 +- Webhook secret 同时存在于两台主机的 root-only 文件,仓库只保存“文件路径”合同;`.env.example` 和 systemd `Environment=` 都不得出现 URL/key。 - 季度提醒以年份+季度作为 finding key,确保每季度一次且同季度不重复;只提醒人工影子恢复,不执行恢复。 ## 安装与回滚 1. 本地先实现合成测试、受限 receiver、告警适配和两个安装器,不接触生产/ARM。 -2. 获批后先在 ARM 安装 rsync、受限账号/receiver/公钥和目标目录;使用无业务数据的合成批次验证拒绝 shell、拒绝 archive 覆盖和 finalize。 -3. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒;首次安装先保持 timer disabled。 -4. 手工启动一次获批自动备份,核对源/ARM 哈希、状态和无敏感输出;再 enable timers。 -5. 用不包含业务正文的可控失败验证企微告警后恢复链路,最终确认下次执行时间。 +2. 获批后先由用户在企微创建运维群机器人,并分别把同一 Webhook 写入两台主机 root-only 文件;不向 Codex、仓库或普通 `.env` 提供其内容。 +3. 在 ARM 安装 rsync、受限账号/receiver/公钥、dead-man timer 和目标目录;使用无业务数据的合成批次验证拒绝 shell、同日安全取代和 dead-man 状态逻辑。 +4. 再在生产安装专用私钥/known_hosts、每日备份 service/timer、stale watchdog 和季度提醒;首次安装先保持 timer disabled。 +5. 手工启动一次获批自动备份,核对源/ARM 哈希、状态和无敏感输出;再 enable timers。 +6. 真正停止一次源端 timer 或故意让一次传输失败,等待/模拟超过 26 小时并确认 ARM 群告警;恢复链路后验证恢复通知和新批次成功。 回滚只 disable/stop 新 timer、移除 unit 和专用密钥授权;保留已生成的完整备份、状态报告和 ARM archive,不删除历史数据。旧手工 `backup-all.sh` 入口保持兼容。 @@ -74,3 +82,4 @@ quarterly reminder - 不给生产普通 ARM shell,而用 inbox + 强制 receiver + archive 隔离,抵消推送方案最主要的跨机删除风险。 - 使用 rsync 而不是新备份服务;当前压缩批次规模较小,收益主要是断点续传和成熟 SSH 传输,不追求块级去重。 - 不自动恢复。季度人工影子恢复才是可恢复性证据,日 timer 只负责生成恢复点。 +- 运维加固在 dead-man、真实失败链路、timer、恢复卡第二人走查完成后硬收口;不继续新增平台化备份或告警能力,后续回到产品 backlog。 diff --git a/.trellis/tasks/07-27-automatic-offhost-backup/implement.md b/.trellis/tasks/07-27-automatic-offhost-backup/implement.md index a5d8a3ce..ae5f83af 100644 --- a/.trellis/tasks/07-27-automatic-offhost-backup/implement.md +++ b/.trellis/tasks/07-27-automatic-offhost-backup/implement.md @@ -9,9 +9,16 @@ - [x] 增加合成契约测试:来源漂移、锁冲突、不完整/损坏/重复批次、续传 partial、拒绝普通 shell、保留轮转、stale 和季度降噪。 - [x] 更新部署环境契约、runbook、checklist、管理员手册的最小操作入口和 `.trellis/spec/hyb-oa/deployment-trialrun.md`;不新增平行长文档。 - [x] 生成一页脱敏恢复卡内容,优先放在现有 runbook 的首部紧急段;由用户复制到公司知识库并安排第二人走查。 -- [ ] 完成本地全量门禁并提交;生产、ARM 安装另行请求批准。 +- [x] 完成本地全量门禁并提交;生产、ARM 安装另行请求批准。 - [ ] 获批后先安装 ARM 接收端并用合成批次验证,再安装生产 units,手工运行首次真实自动批次,最后 enable timers。 - [ ] 获批执行一次可控传输失败告警并恢复正常链路;不读取附件正文、不执行业务或企微其他写入。 +- [x] 删除个人工号默认值,新增生产/ARM 共用的 root-only 企微群机器人 helper;两端 systemd 只绑定 Webhook 文件路径。 +- [x] 增加 ARM 26 小时 dead-man service/timer,覆盖缺批次、超龄、损坏、重复降噪和恢复通知。 +- [x] 将 buildSha 不同降级为 finding;缺失/损坏/超龄/project 不同继续硬失败,并支持部署后立即运行一次备份。 +- [x] 为 backup/rsync/finalize 保存 `0600` 最近失败诊断,summary 与群消息只保留脱敏类别。 +- [x] 把同 UTC 日不同 run_id 从拒绝改为“完整校验后原子取代”,覆盖失败回滚和周检查点重建。 +- [x] 增加 Webhook 不进入 `.env.example`/仓库配置/报告的否定断言,并在 runbook 记录调整 `BACKUP_RUN_AT` 后需人工确认次日窗口。 +- [x] 本地门禁通过并完成只读复核后提交;不推送、不部署、不配置真实 Webhook。 ## 预计文件边界 @@ -22,6 +29,7 @@ - 新增 `scripts/qa/run-automatic-backup-contract.sh`:纯合成正负向测试。 - 修改 `server/src/scripts/send-wecom-ops-alert.js` 及其单测,仅做通用脱敏 summary 兼容。 - 修改 `deploy/trialrun/validate-env-contract.js`、现有 runbook/checklist/管理员手册和部署规约。 +- 新增一个不含 Secret 的宿主告警 helper;只读取安装时传入的 root-only Webhook 文件。 ## 验证命令 @@ -37,28 +45,31 @@ - `node scripts/qa/audit-docs.mjs --check-registry` - `node scripts/qa/check-process-contract.mjs --baseline --json` - `git diff --check` +- 合成测试必须模拟源端不再产生 finalize,并由 ARM dead-man 得到失败;不能只直接调用“发送消息”路径。 ## 外部门禁与回滚点 - 安装 ARM rsync/账号/authorized_keys/目录属于远端配置写入,须单独批准;不得复用普通登录账号或本地工作站私钥。 - 安装生产 SSH key/config/systemd units、首次真实备份、真实异机传输和企微失败告警均须单独批准。 +- 创建企微运维群机器人及把 Webhook 写入两台主机 root-only 文件由用户手工完成;Codex 不读取、不回显、不记录其内容。 - 首次真实执行前记录现有手工批次数量、磁盘余量、活动容器/mount/build 和现有 timer;只输出脱敏计数/布尔值。 - 任何目标身份、主机指纹、活动 mount、manifest 或哈希不匹配立即停止,不猜路径、不接受新指纹、不清理源端批次。 - 回滚新 timer/receiver 时只撤销调度和专用授权;ARM archive、源端完整批次和原 WARCH3 timer 保留。 ## 本地实施与验证证据 -- 已实现生产 wrapper、ARM forced receiver、两端安装器和 3 组 systemd timer;安装器默认只安装 disabled units。 -- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 28 项正负向合同,不连接真实主机。 +- 已实现生产 wrapper、ARM forced receiver、独立群机器人 helper、ARM dead-man 和两端安装器;生产 3 组 timer 与 ARM 1 组 timer 均默认只安装为 disabled。 +- `run-automatic-backup-contract.sh` 使用临时目录与 mock Docker/SSH/rsync 完成 46 项正负向合同,不连接真实主机;覆盖同日取代与回滚、坏候选保留、周检查点重建、Webhook 权限/链接/格式负例、build finding、状态损坏、源端停止、重复降噪和恢复通知。 - 企微告警聚焦单测 15 项通过;stdin 路径被替换为安全类别,AUTO_BACKUP 首次失败/季度提醒发送且相同重复状态降噪。 - `run-backup-restore-drill.sh` 本地合成 DB/附件往返通过,本次合成 RTO 为 8 秒;该结果不是生产 RTO 或 ARM 恢复证据。 - `scripts/check-minimal.sh` 通过:server 162 个测试文件、1004 项测试全部通过,client typecheck/build 通过。 - 环境合同、Trellis task/governance、敏感扫描、文档注册、流程合同和 `git diff --check` 均通过;文档注册保留既有 22 个未注册文件 warning,无本任务新增文档。 -- ARM 安装、生产 units、真实首批、timer enable、真实企微失败告警和第二人季度走查仍为外部门禁,未执行、不得写成已完成。 +- 最终只读复核修正了三个本地缺陷:生产安装器残留个人工号校验、ARM 重装可能改变 dead-man 状态文件属主、stale 状态未校验 run id/manifest SHA;修正后 46 项合同与 `scripts/check-minimal.sh` 全量复跑通过。 +- 更新后的 ARM dead-man/helper、生产 helper/runner、真实有效首批、四组 timer enable、真实企微失败链路和第二人季度走查仍为外部门禁,未执行、不得写成已完成。 ### 首次远端执行增量 - ARM 已安装 rsync、密码锁定专用账号和 forced-command receiver;从生产验证普通 shell、sender/read、delete 均拒绝,写通道 dry-run 通过且未生成归档。 - 生产三组 units 已安装为 disabled/inactive,并绑定活动 Compose project、PGDATA、附件挂载、构建标识和固定 SSH 身份。 -- 首次真实备份在 `runtime_identity` 阶段 fail-closed,未生成源端或 ARM 批次;真实 Docker `println` 输出的参数间空格导致制表符拆分后路径不精确相等。 -- runner 已改为解析 `docker inspect` mount JSON 并要求唯一匹配;合成 mock 同步改为真实 JSON 形状。重新执行真实首批和 timer enable 尚待验证。 +- 首次真实备份曾在 `runtime_identity` 阶段因 Docker `println` 参数间空格 fail-closed;runner 改用 `docker inspect` mount JSON 后,后续真实源端成对批次已生成并传输。 +- ARM finalize 随后因同 UTC 日已有不同 run id 而拒绝,因此尚无可作为自动备份闭环证据的新批次;本轮已在本地实现“完整校验后同日取代”,更新后的远端脚本、首批和 timer enable 仍待另行批准。 diff --git a/.trellis/tasks/07-27-automatic-offhost-backup/prd.md b/.trellis/tasks/07-27-automatic-offhost-backup/prd.md index 9bba28e5..61e3ec3d 100644 --- a/.trellis/tasks/07-27-automatic-offhost-backup/prd.md +++ b/.trellis/tasks/07-27-automatic-offhost-backup/prd.md @@ -12,6 +12,8 @@ - 两台主机之间当前没有可直接复用的免密 SSH 别名或密钥链路,不能把本地工作站的 SSH 配置误当作服务器间自动化能力。 - 当前手工正确批次和 ARM 影子恢复已证明格式可用,但不能满足 RPO 24 小时或自动失败发现。 - 已确认采用“生产主动推送到 ARM 的专用受限接收身份”:生产保留即时传输结果,ARM 的已归档批次不暴露给 rsync 写入根。 +- 生产切换已经完成并应用 61 项 migration;本地当前为 62 项,`2526052` 后续部署只新增 1 项 migration。自动备份三组 units 已安装但仍未形成可靠启用闭环。 +- 首次真实自动批次已证明源端成对备份可生成,但 ARM 因“同一 UTC 日仅允许一个不同 run_id”拒绝重试批次;现有即时告警还依赖活动 server 容器,不能覆盖容器或整机故障。 ## Scope @@ -28,7 +30,11 @@ - 完整批次通过固定主机指纹和专用受限密钥自动推送到 ARM;该身份无普通 shell,只能把文件写入 inbox 或请求校验/归档,不能通过 rsync 读取或删除 archive。 - 传输端和目标端都要校验 manifest、文件大小和 SHA-256;同名已归档目标不得覆盖。中断传输可以在 inbox 内续传,但只有完整校验后才能原子进入 archive。 - 生产保留最近 7 个成功日批次;ARM 至少保留最近 7 个日批次和 4 个周检查点。清理只能发生在本次异机成功确认之后,且不得删除最后一个已知成功批次。 -- 备份或异机校验失败必须生成脱敏状态报告,并复用现有企微运维告警能力;独立 stale watchdog 应在现有 07:30 巡检前检查最近一次异机成功时间,防止备份 timer 未运行或即时告警自身失败造成静默缺口。 +- 生产即时告警和 ARM dead-man 告警统一使用独立企微运维群机器人;Webhook 只从 root-only 文件读取,不依赖 OA server、数据库、员工绑定或默认个人工号,也不得进入仓库、`.env.example`、Trellis、日志或聊天。 +- ARM 安装独立 26 小时 dead-man timer,直接检查最近归档完成时间;生产整机或容器不可用、timer 未运行、传输未 finalize 时仍能从另一台主机发出告警。 +- 生产 watchdog 的“恢复点缺失/损坏/超龄/Compose project 不匹配”继续硬失败;仅 `buildSha` 早于当前部署时降级为一次 finding,不得误报为没有恢复点。部署后仍推荐立即触发一次获批备份。 +- backup、rsync 和 finalize 的失败输出必须写入状态目录下权限 `0600` 的最近失败诊断;企微告警仍只包含阶段、状态、finding 数和脱敏建议,不附原始输出。 +- 同一 UTC 日已有不同自动归档时,新批次必须先完整通过 manifest、文件集、大小和 SHA-256 校验,再以可回滚的同文件系统 rename 顺序原子取代旧日批次;任一替换步骤失败必须恢复旧批次或保留可诊断状态。最终仍只保留每日一个恢复点。 - 告警只包含阶段、状态、finding 数和脱敏建议,不包含数据库凭证、完整目录、SSH 信息、真实业务数据、附件名或原始命令输出。 - 季度恢复保持人工触发,在 ARM 隔离目标执行;季度 timer 只发送一次脱敏提醒,不得自动运行恢复,更不得触碰生产库。 - 形成一页脱敏恢复卡,说明主机故障后的负责人、备份位置类别、恢复顺序和验收项,供另一名人员存入公司知识库。 @@ -38,7 +44,11 @@ - [x] 本地合成测试证明每日 wrapper 只接受唯一活动来源、互斥锁有效,并只发布完整备份批次。 - [x] 传输测试证明不完整批次、哈希不一致、重复批次和不可达 ARM 均失败,且不产生异机成功状态。 - [x] 保留测试覆盖 7 个日批次、4 个周检查点、最后成功批次保护和仅在异机确认后清理。 -- [x] 告警测试证明首次失败和状态恶化会告警,重复同一失败降噪;07:30 stale 检查能发现超过 24 小时没有异机成功记录。 +- [x] 告警测试证明首次失败和状态恶化会告警,重复同一失败降噪;07:10 stale 检查能发现超过 24 小时没有异机成功记录。 +- [x] 本地合成测试证明同日不同 run_id 的完整新批次可取代旧批次,损坏新批次不会影响旧归档,且周检查点与日归档不产生悬空或错误指向。 +- [x] 本地合成测试证明 buildSha 不同仅产生 finding,恢复点缺失/损坏/超龄/project 不同仍失败;失败诊断为 `0600` 且不进入告警摘要。 +- [x] 环境契约证明 Webhook 相关键和值不出现在 `.env.example`、仓库配置或报告;两端安装器只接受 root-only Webhook 文件路径。 +- [x] ARM 26 小时 dead-man 合成验证覆盖“源端停止/不产生新 finalize”而非仅验证机器人能发送,并覆盖首次失败、重复降噪与恢复通知。 - [ ] systemd 安装器显式绑定 Compose project、活动数据路径、构建标识、备份根和受限远端身份;安装后 timer enabled/active,但部署验证不手工执行真实告警。 - [ ] 经单独生产授权后完成一次自动日备份:源端完整、ARM 目标端哈希一致、状态报告脱敏,且未读取附件正文或修改业务数据。 - [ ] 模拟传输失败能在当天产生一条脱敏企微告警;测试后恢复正常链路并完成成功批次。 @@ -49,12 +59,15 @@ - 不引入备份平台、对象存储、MinIO、云厂商 SDK、Kubernetes 或持续复制。 - 不自动执行生产恢复、数据库 reset、迁移改写或附件回写。 - 不把真实业务正文、附件清单、凭证、主机地址、完整目录或 SSH 私钥写入仓库、报告或聊天。 +- 不把企微群机器人 Webhook URL、key 或配置文件正文写入仓库、`.env.example`、Trellis、报告、日志或聊天。 - 不在本任务继续企微本人授权、员工角色、付款、报销或其他产品功能。 - 不因自动备份完成而取消季度人工影子恢复。 ## Test Plan - Shell 语法与合成文件系统测试,覆盖锁、来源解析、传输失败、目标重复、哈希和保留轮转。 +- 合成 receiver 测试覆盖同日完整批次取代、损坏候选保留旧批次、替换失败恢复以及周检查点一致性。 +- 合成双主机告警测试覆盖生产无 server 容器、ARM 26 小时无新归档、重复降噪、恢复通知和 root-only Webhook 文件门禁。 - 复用 `scripts/qa/run-backup-restore-drill.sh` 验证成对备份合同未回退。 - 聚焦测试现有企微运维告警的备份状态与 stale finding 适配。 - `node deploy/trialrun/validate-env-contract.js`。 diff --git a/deploy/trialrun/checklist.md b/deploy/trialrun/checklist.md index dafb0354..d8faae75 100644 --- a/deploy/trialrun/checklist.md +++ b/deploy/trialrun/checklist.md @@ -31,7 +31,7 @@ - [ ] 正式 deploy 使用绝对数据路径;既有实例使用 `DEPLOY_DATA_MODE=existing`,PGDATA 已存在且含 `PG_VERSION`,附件目录已存在 - [ ] 已证明没有两个 PostgreSQL 实例同时挂载同一 PGDATA;任何不一致均停止发布 - [ ] 已记录活动 server/db 容器、不可变镜像 ID(及可用 repo digest)、Caddy 目标、挂载源、环境摘要,不记录 env 正文 -- [ ] 已冻结成功 migration 数量(当前预期 58)和关键业务表逐表计数基线 +- [ ] 已现场只读冻结成功 migration 数量和关键业务表逐表计数基线;当前生产证据为 61 项、本地为 62 项,本次待上线只应新增 1 项,现场结果不符立即停止 - [ ] 确认当前版本已打 tag:`git tag -a trialrun-YYYYMMDD-NN -m "trialrun release"` - [ ] 本地门禁:`./scripts/check-minimal.sh` - [ ] 部署契约校验:`node deploy/trialrun/validate-env-contract.js` @@ -48,12 +48,14 @@ - [ ] 完整批次已复制到获批异机/异盘,并记录保留周期 - [ ] 同批次已在隔离影子目标恢复:migration 数、关键业务表逐表计数、附件普通文件数均与冻结基线一致,兼容镜像 `/ready` 通过 - [ ] 自动异机备份(未完成远端门禁前保持未勾选): - - [ ] **EXTERNAL_GATED**:ARM 已获批安装 rsync、密码锁定专用账号、forced-command key 和独立 inbox/archive/weekly;普通 shell、rsync sender/read/delete 均被拒绝 - - [ ] **EXTERNAL_GATED**:生产专用私钥为 `0600`,known-hosts 指纹已人工核对,安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录和远端身份 + - [ ] **EXTERNAL_GATED**:用户已在独立企微运维群创建机器人,并分别把 Webhook 写入生产、ARM 的 root-only `0600` 文件;未向 Codex、仓库、普通 `.env`、日志或聊天提供正文 + - [ ] **EXTERNAL_GATED**:ARM 已获批安装 rsync、密码锁定专用账号、forced-command key、独立 inbox/archive/weekly 和 26 小时 dead-man;普通 shell、rsync sender/read/delete 均被拒绝 + - [ ] **EXTERNAL_GATED**:生产专用私钥为 `0600`,known-hosts 指纹已人工核对,安装器显式绑定活动 project、PGDATA、附件、完整/短构建 SHA、备份/状态目录、远端身份和 Webhook 文件路径 - [ ] `install-automatic-backup-timer.sh install` 只安装 disabled units;首批真实批次及 ARM manifest/SHA-256 核对通过后才显式执行 `enable` - - [ ] daily/watchdog/drill-reminder 三组 timer enabled/active,daily service 当前 inactive;未通过手工启动真实告警来伪造验证 + - [ ] 生产 daily/watchdog/drill-reminder 与 ARM dead-man 四组 timer enabled/active,各 oneshot service 当前 inactive;未通过只调用机器人发送入口来伪造验证 - [ ] 最近一次异机成功不超过 24 小时;生产自动日批次为 7 个,ARM 自动日批次为 7 个、周检查点为 4 个,历史手工批次未被轮转 - - [ ] **EXTERNAL_GATED**:可控传输失败仅产生一条脱敏企微告警,恢复链路后新批次成功;同一失败重复运行已降噪 + - [ ] **EXTERNAL_GATED**:已真正停止一次源端 timer 或故意破坏一次传输,ARM dead-man 仅产生一条脱敏企微告警;恢复 finalize 后收到一次恢复通知,同一失败重复运行已降噪 + - [ ] 若调整过 `BACKUP_RUN_AT`,已人工确认调整后的次日 UTC 分桶和 26 小时窗口,未把一次预期空窗误判为长期故障 - [ ] 本季度已由第二名内部人员按 runbook 首部恢复卡在 ARM 隔离目标完成影子恢复走查;timer 未自动恢复生产 - [ ] 停机复制方案已冻结:旧 PostgreSQL 完全停止后复制到空新目录并保留所有权/权限/链接/扩展属性;旧目录原位保留且观察期内不再写入 diff --git a/deploy/trialrun/install-automatic-backup-timer.sh b/deploy/trialrun/install-automatic-backup-timer.sh index 2a80326f..b0712c7f 100755 --- a/deploy/trialrun/install-automatic-backup-timer.sh +++ b/deploy/trialrun/install-automatic-backup-timer.sh @@ -14,7 +14,8 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}" OFFHOST_TARGET="${OFFHOST_TARGET:-}" OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}" OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}" -ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}" +ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}" +ALERT_HELPER="${ALERT_HELPER:-${REMOTE_DIR}/scripts/send-wecom-group-alert.py}" BACKUP_RUN_AT="${BACKUP_RUN_AT:-02:30}" WATCHDOG_RUN_AT="${WATCHDOG_RUN_AT:-07:10}" SERVICE_PREFIX="${SERVICE_PREFIX:-hyb-oa-automatic-backup}" @@ -66,7 +67,8 @@ paths_overlap() { validate_inputs() { local name value for name in REMOTE_DIR REMOTE_ENV_FILE POSTGRES_DATA_DIR STORAGE_HOST_DIR \ - AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS; do + AUTOMATIC_BACKUP_DIR AUTOMATIC_BACKUP_STATE_DIR OFFHOST_SSH_KEY OFFHOST_KNOWN_HOSTS \ + ALERT_WEBHOOK_FILE ALERT_HELPER; do value="${!name:-}" validate_absolute_path "$name" "$value" done @@ -91,10 +93,14 @@ validate_inputs() { && "$(stat -c %a "$OFFHOST_SSH_KEY")" == "600" ]] \ || fail "off-host private key must exist with mode 0600" [[ -s "$OFFHOST_KNOWN_HOSTS" ]] || fail "pinned known-hosts file is missing" + [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \ + && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \ + && "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \ + || fail "alert webhook file must be root-owned with mode 0600" + [[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing" [[ "$BACKUP_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "backup time is invalid" [[ "$WATCHDOG_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "watchdog time is invalid" [[ "$SERVICE_PREFIX" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "service prefix is invalid" - [[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid" local -a isolated_paths=( "$POSTGRES_DATA_DIR" "$STORAGE_HOST_DIR" "$AUTOMATIC_BACKUP_DIR" "$AUTOMATIC_BACKUP_STATE_DIR" ) @@ -123,7 +129,8 @@ Environment=AUTOMATIC_BACKUP_STATE_DIR=${AUTOMATIC_BACKUP_STATE_DIR} Environment=OFFHOST_TARGET=${OFFHOST_TARGET} Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY} Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS} -Environment=ALERT_RECIPIENT_EMPLOYEE_NO=${ALERT_RECIPIENT_EMPLOYEE_NO} +Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE} +Environment=ALERT_HELPER=${ALERT_HELPER} ENVIRONMENT } diff --git a/deploy/trialrun/install-offhost-backup-receiver.sh b/deploy/trialrun/install-offhost-backup-receiver.sh index b4fe4bab..25674baf 100755 --- a/deploy/trialrun/install-offhost-backup-receiver.sh +++ b/deploy/trialrun/install-offhost-backup-receiver.sh @@ -7,6 +7,15 @@ PUBLIC_KEY_FILE="${PUBLIC_KEY_FILE:-}" RECEIVER_SOURCE="${RECEIVER_SOURCE:-}" RECEIVER_TARGET="${RECEIVER_TARGET:-/usr/local/sbin/hyb-oa-offhost-receiver}" CONFIG_FILE="${CONFIG_FILE:-/etc/hyb-oa-offhost-backup.conf}" +ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}" +ALERT_HELPER_SOURCE="${ALERT_HELPER_SOURCE:-}" +ALERT_HELPER_TARGET="${ALERT_HELPER_TARGET:-/usr/local/sbin/hyb-oa-wecom-group-alert}" +DEADMAN_SOURCE="${DEADMAN_SOURCE:-}" +DEADMAN_TARGET="${DEADMAN_TARGET:-/usr/local/sbin/hyb-oa-offhost-deadman}" +DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}" +DEADMAN_RUN_AT="${DEADMAN_RUN_AT:-06:30}" +DEADMAN_SERVICE="${DEADMAN_SERVICE:-hyb-oa-offhost-deadman}" +SYSTEMD_DIR="${SYSTEMD_DIR:-/etc/systemd/system}" DAILY_KEEP="${DAILY_KEEP:-7}" WEEKLY_KEEP="${WEEKLY_KEEP:-4}" MODE="${1:-install}" @@ -45,7 +54,7 @@ validate_backup_root() { if [[ -d "$BACKUP_ROOT" ]]; then local unmanaged_entry unmanaged_entry="$(find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 \ - ! -name inbox ! -name archive ! -name weekly -print -quit)" + ! -name inbox ! -name archive ! -name weekly ! -name state -print -quit)" [[ -z "$unmanaged_entry" ]] || fail "backup root contains an unmanaged top-level entry" fi } @@ -57,8 +66,22 @@ validate_inputs() { validate_absolute_path RECEIVER_SOURCE "$RECEIVER_SOURCE" validate_absolute_path RECEIVER_TARGET "$RECEIVER_TARGET" validate_absolute_path CONFIG_FILE "$CONFIG_FILE" + validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE" + validate_absolute_path ALERT_HELPER_SOURCE "$ALERT_HELPER_SOURCE" + validate_absolute_path ALERT_HELPER_TARGET "$ALERT_HELPER_TARGET" + validate_absolute_path DEADMAN_SOURCE "$DEADMAN_SOURCE" + validate_absolute_path DEADMAN_TARGET "$DEADMAN_TARGET" [[ -f "$PUBLIC_KEY_FILE" && ! -L "$PUBLIC_KEY_FILE" ]] || fail "public key file is missing" [[ -f "$RECEIVER_SOURCE" && ! -L "$RECEIVER_SOURCE" ]] || fail "receiver source is missing" + [[ -f "$ALERT_HELPER_SOURCE" && ! -L "$ALERT_HELPER_SOURCE" ]] || fail "alert helper source is missing" + [[ -f "$DEADMAN_SOURCE" && ! -L "$DEADMAN_SOURCE" ]] || fail "dead-man source is missing" + [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \ + && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" \ + && "$(stat -c %u "$ALERT_WEBHOOK_FILE")" == "0" ]] \ + || fail "alert webhook file must be root-owned with mode 0600" + [[ "$DEADMAN_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "dead-man threshold must be positive" + [[ "$DEADMAN_RUN_AT" =~ ^([01][0-9]|2[0-3]):[0-5][0-9]$ ]] || fail "dead-man time is invalid" + [[ "$DEADMAN_SERVICE" =~ ^[a-z0-9][a-z0-9_-]*$ ]] || fail "dead-man service name is invalid" [[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ && "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] \ || fail "retention values must be positive" command -v rsync >/dev/null 2>&1 || fail "rsync must be installed before receiver setup" @@ -86,22 +109,28 @@ install_receiver() { fi passwd -l "$BACKUP_USER" >/dev/null 2>&1 || true - local home_dir inbox_dir archive_dir weekly_dir ssh_dir auth_file public_key rrsync_bin + local home_dir inbox_dir archive_dir weekly_dir state_dir ssh_dir auth_file public_key rrsync_bin home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')" validate_absolute_path BACKUP_USER_HOME "$home_dir" inbox_dir="$BACKUP_ROOT/inbox" archive_dir="$BACKUP_ROOT/archive" weekly_dir="$BACKUP_ROOT/weekly" - mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh" + state_dir="$BACKUP_ROOT/state" + mkdir -p "$inbox_dir" "$archive_dir" "$weekly_dir" "$state_dir" "$home_dir/.ssh" [[ -d "$BACKUP_ROOT" && ! -L "$BACKUP_ROOT" \ && -d "$inbox_dir" && ! -L "$inbox_dir" \ && -d "$archive_dir" && ! -L "$archive_dir" \ && -d "$weekly_dir" && ! -L "$weekly_dir" ]] \ || fail "backup receiver directories must not be symbolic links" - chown -R "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT" "$home_dir/.ssh" + chown "$BACKUP_USER:$BACKUP_USER" "$BACKUP_ROOT" + chown -R "$BACKUP_USER:$BACKUP_USER" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh" chmod 0700 "$BACKUP_ROOT" "$inbox_dir" "$archive_dir" "$weekly_dir" "$home_dir/.ssh" + chown -R root:root "$state_dir" + chmod 0700 "$state_dir" install -o root -g root -m 0755 "$RECEIVER_SOURCE" "$RECEIVER_TARGET" + install -o root -g root -m 0755 "$ALERT_HELPER_SOURCE" "$ALERT_HELPER_TARGET" + install -o root -g root -m 0755 "$DEADMAN_SOURCE" "$DEADMAN_TARGET" rrsync_bin="$(command -v rrsync)" cat >"$CONFIG_FILE" </dev/null - echo "[offhost-receiver-installer][PASS] restricted receiver installed" + cat >"$SYSTEMD_DIR/${DEADMAN_SERVICE}.service" <"$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" </dev/null 2>&1 || true + systemctl stop "${DEADMAN_SERVICE}.service" >/dev/null 2>&1 || true + rm -f -- "$SYSTEMD_DIR/${DEADMAN_SERVICE}.timer" "$SYSTEMD_DIR/${DEADMAN_SERVICE}.service" if id "$BACKUP_USER" >/dev/null 2>&1; then local home_dir auth_file home_dir="$(getent passwd "$BACKUP_USER" | awk -F: '{print $6}')" @@ -146,19 +211,23 @@ uninstall_receiver() { rm -f -- "$auth_file" fi fi - rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE" - echo "[offhost-receiver-installer][PASS] receiver authorization removed; archives retained" + rm -f -- "$RECEIVER_TARGET" "$CONFIG_FILE" "$ALERT_HELPER_TARGET" "$DEADMAN_TARGET" + systemctl daemon-reload + echo "[offhost-receiver-installer][PASS] receiver authorization and dead-man removed; archives retained" } case "$MODE" in install) install_receiver ;; + enable) + enable_receiver + ;; uninstall|--uninstall) uninstall_receiver ;; --help|-h) - echo "Usage: BACKUP_ROOT= PUBLIC_KEY_FILE= RECEIVER_SOURCE= bash install-offhost-backup-receiver.sh [install|uninstall]" + echo "Usage: bash install-offhost-backup-receiver.sh [install|enable|uninstall]" ;; *) fail "unsupported mode" diff --git a/deploy/trialrun/runbook.md b/deploy/trialrun/runbook.md index 2f045046..09d6502b 100644 --- a/deploy/trialrun/runbook.md +++ b/deploy/trialrun/runbook.md @@ -226,29 +226,46 @@ manifest 必须至少记录一个真实来源标识:活动 `/ready` 有构建 整批复制到获批的异机/异盘位置并记录保留周期;不能只复制 DB 或附件中的一份。 -#### 每日自动异机备份(当前仅本地实现,远端启用为 EXTERNAL_GATED) +#### 每日自动异机备份(可靠启用闭环仍为 EXTERNAL_GATED) - 每日 02:30 由生产主机解析唯一活动 Compose project 和真实 DB/附件挂载,生成同批三文件,ARM 校验并原子归档成功后才写成功状态和执行源端轮转。 -- 生产保留 7 个自动日批次;ARM 保留 7 个日批次和 4 个周检查点。07:10 watchdog 检查最近 24 小时异机成功记录;季度 timer 只发送人工影子恢复提醒。 +- 生产保留 7 个自动日批次;ARM 保留 7 个日批次和 4 个周检查点。07:10 源端 watchdog 检查最近 24 小时异机成功记录;ARM 独立 dead-man 检查最近 26 小时归档,生产整机或 OA server 停止时仍能告警;季度 timer 只发送人工影子恢复提醒。 - ARM 使用专用密码锁定账号和 forced-command key:rsync 只能写 inbox,不能读取/删除 archive,也没有普通 shell。生产固定专用 key 和人工核对的 known-hosts,禁止 `accept-new`。 -- 失败告警只包含阶段、状态、finding 数和类别建议。同一失败降噪;首次失败、状态恶化、新 finding 和每季度首次恢复提醒才发送。 +- 同一 UTC 日重试使用新 run id 时,ARM 必须先完整校验候选,再原子取代当日旧批次并重建该周硬链接检查点;坏候选或发布失败不得触碰原有效批次。 +- 生产与 ARM 都直接使用独立企微运维群机器人,不依赖 OA server、数据库或个人工号。Webhook 正文只由用户手工写入两台主机的 root-only `0600` 文件;仓库、普通 `.env`、systemd unit、日志和聊天只能出现受控文件路径,不能出现 URL 或 key。 +- 失败告警只包含阶段、状态、finding 数和类别建议;原始命令诊断只保留在源端状态目录的 `0600` 最近失败日志。同一失败降噪;首次失败、状态恶化、新 finding、恢复通知和每季度首次恢复提醒才发送。新部署构建与最近恢复点 build SHA 不同只记 finding,缺失、损坏、超龄、未来时间或 Compose project 不同仍硬失败。 + +创建机器人后,由用户分别登录两台主机,在 root shell 中用静默输入把完整 Webhook 写入受控路径;不要把值作为命令参数、环境文件内容或聊天消息交给 Codex: + +```bash +umask 077 +read -rsp 'WeCom operations robot Webhook: ' HYB_OA_ALERT_WEBHOOK +printf '\n' +install -o root -g root -m 0600 /dev/null +printf '%s\n' "$HYB_OA_ALERT_WEBHOOK" > +unset HYB_OA_ALERT_WEBHOOK +``` 安装顺序必须是:先获批 ARM 端安装 rsync、受限身份和 receiver,用合成批次验证;再获批生产端安装 units,手工完成首个真实批次;最后显式执行生产安装器的 `enable`。两个安装器默认 `install` 均不代表已启用,不得把本地合同通过写成远端完成。 ```bash # ARM:参数从受控配置取得;install 会写账号/authorized_keys/receiver,需单独批准。 BACKUP_ROOT= PUBLIC_KEY_FILE= \ -RECEIVER_SOURCE= \ +RECEIVER_SOURCE= ALERT_WEBHOOK_FILE= \ +ALERT_HELPER_SOURCE= DEADMAN_SOURCE= \ bash deploy/trialrun/install-offhost-backup-receiver.sh install # 生产:先只安装 disabled units;路径、项目、构建和 SSH 身份必须绑定活动实例。 - \ + \ bash deploy/trialrun/install-automatic-backup-timer.sh install # 首批真实备份、ARM 哈希和脱敏告警均经单独确认后,才允许启用三组 timer。 + bash deploy/trialrun/install-offhost-backup-receiver.sh enable bash deploy/trialrun/install-automatic-backup-timer.sh enable ``` +`BACKUP_RUN_AT` 使用生产主机本地时区,ARM 归档按 UTC 日分桶。调整执行时刻的当天可能出现预期空窗;改动后必须人工确认次日 UTC 分桶、下一次 daily 运行和 26 小时 dead-man 窗口。真实告警验收必须真正停止一次源端 timer 或故意让一次传输失败,再确认 ARM 首次告警、重复降噪和恢复通知;只测试机器人能发消息不算通过。 + 卸载只撤销 timer/receiver 授权,保留源端批次、状态、ARM archive 和凭证文件;专用账号/密钥的最终销毁另走获批清理。 ### 1.5.2 本地隔离恢复演练 @@ -284,7 +301,7 @@ RESTORE_SCOPE=shadow \ `restore-db.sh` 和 `restore-storage.sh` 默认只接受 `shadow`。影子库名必须包含独立的 `shadow`、`drill` 或 `restore_test` 标记;数据库确认不匹配、库名不安全、附件目录非空、目标为默认试运行附件目录或归档路径危险时均在写入前退出。 -真实格式影子恢复的验收不仅是 `/ready`:恢复后的 `_prisma_migrations` 成功项必须等于冻结基线(当前预期 58),预先列出的关键业务表计数必须逐表一致,附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。 +真实格式影子恢复的验收不仅是 `/ready`:恢复后的 `_prisma_migrations` 成功项必须等于执行前现场冻结基线(当前生产证据为 61 项;本地 62 项仅表示下一次部署预计新增 1 项),预先列出的关键业务表计数必须逐表一致,附件解包后的普通文件数必须与活动源目录一致。任一项不一致即判定批次不可作为切换回滚点。 ### 1.5.3 生产切换前冻结记录 diff --git a/deploy/trialrun/validate-env-contract.js b/deploy/trialrun/validate-env-contract.js index 272f0b2d..b233109d 100755 --- a/deploy/trialrun/validate-env-contract.js +++ b/deploy/trialrun/validate-env-contract.js @@ -1,6 +1,7 @@ #!/usr/bin/env node const fs = require('node:fs'); const path = require('node:path'); +const childProcess = require('node:child_process'); const rootDir = path.resolve(__dirname, '..', '..'); const configPath = path.join(rootDir, 'server/src/config.js'); @@ -29,6 +30,8 @@ const offhostReceiverInstallerPath = path.join( ); const automaticBackupRunnerPath = path.join(rootDir, 'scripts/run-automatic-backup.sh'); const offhostReceiverPath = path.join(rootDir, 'scripts/receive-offhost-backup.sh'); +const groupAlertHelperPath = path.join(rootDir, 'scripts/send-wecom-group-alert.py'); +const offhostDeadmanPath = path.join(rootDir, 'scripts/check-offhost-backup-deadman.py'); const nginxPaths = [ path.join(rootDir, 'deploy/trialrun/nginx-hyb-oa.conf'), path.join(rootDir, 'deploy/trialrun/nginx.conf.example'), @@ -157,6 +160,8 @@ const automaticBackupTimerScriptContent = readFile(automaticBackupTimerScriptPat const offhostReceiverInstallerContent = readFile(offhostReceiverInstallerPath); const automaticBackupRunnerContent = readFile(automaticBackupRunnerPath); const offhostReceiverContent = readFile(offhostReceiverPath); +const groupAlertHelperContent = readFile(groupAlertHelperPath); +const offhostDeadmanContent = readFile(offhostDeadmanPath); const nginxContents = nginxPaths.map((filePath) => ({ filePath, content: readFile(filePath), @@ -185,6 +190,25 @@ for (const key of ['APP_HOST_PORT', 'POSTGRES_DATA_DIR', 'STORAGE_HOST_DIR']) { if (envExampleKeys.has('COMPOSE_PROJECT_NAME')) { errors.push('.env.example 不应持久化 COMPOSE_PROJECT_NAME,调用方必须通过 -p 显式选择项目'); } +for (const key of ['ALERT_WEBHOOK_FILE', 'WECOM_WEBHOOK_URL', 'WECOM_WEBHOOK_KEY']) { + if (envExampleKeys.has(key)) { + errors.push(`.env.example 不应持久化企微群机器人配置 ${key}`); + } +} +const trackedFiles = childProcess.execFileSync('git', ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], { + cwd: rootDir, + encoding: 'utf8', +}).split('\0').filter(Boolean); +const literalWebhookPattern = /https:\/\/qyapi\.weixin\.qq\.com\/cgi-bin\/webhook\/send\?key=/; +for (const relativePath of trackedFiles) { + const filePath = path.join(rootDir, relativePath); + if (!fs.statSync(filePath).isFile()) continue; + const content = fs.readFileSync(filePath); + if (content.includes(0)) continue; + if (literalWebhookPattern.test(content.toString('utf8'))) { + errors.push(`${relativePath} 不得持久化完整企微群机器人 Webhook URL`); + } +} for (const serviceName of requiredServices) { const block = extractServiceBlock(composeContent, serviceName); @@ -306,6 +330,8 @@ for (const requiredFragment of [ 'Environment=OFFHOST_TARGET=${OFFHOST_TARGET}', 'Environment=OFFHOST_SSH_KEY=${OFFHOST_SSH_KEY}', 'Environment=OFFHOST_KNOWN_HOSTS=${OFFHOST_KNOWN_HOSTS}', + 'Environment=ALERT_WEBHOOK_FILE=${ALERT_WEBHOOK_FILE}', + 'Environment=ALERT_HELPER=${ALERT_HELPER}', 'install)\n install_units', 'enable)\n enable_units', ]) { @@ -325,7 +351,10 @@ for (const requiredFragment of [ 'hyb-backup-finalize $run_id $manifest_sha', 'write_success_state "$run_id" "$manifest_sha"', 'prune_local_backups "$batch_name"', - '--summary-file=-', + 'python3 "$ALERT_HELPER"', + '--webhook-file "$ALERT_WEBHOOK_FILE"', + 'capture_command offhost_transfer rsync', + 'last-error.log', "docker inspect -f '{{json .Mounts}}'", 'if len(matches) != 1 or not matches[0]:', ]) { @@ -345,6 +374,8 @@ for (const requiredFragment of [ 'batch must contain exactly three regular files', 'manifest artifact checksum mismatch', 'mv -- "$partial_dir" "$archive_dir"', + 'replacement_rollback=', + 'publish_weekly_checkpoint', 'prune_named_directories "$BACKUP_ARCHIVE_DIR"', 'prune_named_directories "$BACKUP_WEEKLY_DIR"', ]) { @@ -360,6 +391,9 @@ for (const requiredFragment of [ 'BACKUP_INBOX_DIR=${inbox_dir}', 'BACKUP_ARCHIVE_DIR=${archive_dir}', 'BACKUP_WEEKLY_DIR=${weekly_dir}', + 'DEADMAN_MAX_AGE_SECONDS="${DEADMAN_MAX_AGE_SECONDS:-93600}"', + 'ExecStart=/usr/bin/python3 ${DEADMAN_TARGET}', + 'systemctl enable --now "${DEADMAN_SERVICE}.timer"', 'archives retained', 'backup root contains an unmanaged top-level entry', 'backup root must not be inside a system-managed directory', @@ -369,6 +403,30 @@ for (const requiredFragment of [ } } +for (const requiredFragment of [ + 'stat.S_IMODE(info.st_mode) != 0o600', + 'parsed.hostname != "qyapi.weixin.qq.com"', + 'HYB_WECOM_ALERT_TEST_MODE', + 'should_send(previous, status, category)', + '不要在群内粘贴原始日志', +]) { + if (!groupAlertHelperContent.includes(requiredFragment)) { + errors.push(`send-wecom-group-alert.py 缺少独立脱敏告警合同: ${requiredFragment}`); + } +} + +for (const requiredFragment of [ + 'default=93600', + 'archive_manifest_invalid', + 'archive_time_in_future', + 'FAIL_OFFHOST_DEADMAN', + 'deadman-alert-state.json', +]) { + if (!offhostDeadmanContent.includes(requiredFragment)) { + errors.push(`check-offhost-backup-deadman.py 缺少 ARM 独立看门狗合同: ${requiredFragment}`); + } +} + for (const requiredFragment of [ 'validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"', 'validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"', diff --git a/docs/管理员手册.md b/docs/管理员手册.md index 36868239..dbe1c0e5 100644 --- a/docs/管理员手册.md +++ b/docs/管理员手册.md @@ -359,12 +359,13 @@ WAPC2 请假/补卡只读 POC 操作口径: 自动异机备份运维口径: 1. 自动备份以 OA 仓库 runbook 首部“紧急恢复卡”为唯一短入口;公司知识库保存该卡副本,并指定系统管理员和一名替补操作人。 -2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM;07:10 检查最近 24 小时异机成功状态。生产保留 7 个自动日批次,ARM 保留 7 个日批次和 4 个周检查点。 +2. 每日 02:30 生成数据库、附件、manifest 同批恢复点并推送 ARM;07:10 检查最近 24 小时异机成功状态,ARM 另以 26 小时 dead-man 独立检查归档。生产保留 7 个自动日批次,ARM 保留 7 个日批次和 4 个周检查点。 3. ARM 专用身份只能写 inbox 和请求 finalize,不能登录 shell、读取或删除 archive。任何人不得为了省事改用普通 ARM 登录账号或关闭 known-hosts 固定校验。 -4. 企微只收脱敏异常:首次失败、恶化或新 finding 发送;同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复,不会自动恢复生产。 -5. 当前仓库实现和本地合成合同不等于服务器已安装。ARM receiver、生产 units、首次真实批次、timer enable 和真实失败告警均为 **EXTERNAL_GATED**,必须逐阶段批准并回填证据。 +4. 自动备份使用独立企微运维群机器人,不依赖 OA server 或个人员工号。Webhook 只由用户写入两台主机的 root-only 文件,不进仓库、普通 `.env`、日志或聊天。企微只收脱敏异常、finding 和恢复通知;同一失败降噪。季度提醒只要求人工在 ARM 隔离目标恢复,不会自动恢复生产。 +5. 当前 receiver 和生产 units 的既有安装不等于可靠启用闭环完成。群机器人、ARM dead-man、更新后的两端脚本、首次有效真实批次、全部 timer enable 和真实断链告警均为 **EXTERNAL_GATED**,必须逐阶段批准并回填证据;真实验收必须停止一次源端 timer 或故意让一次传输失败,不能只测试机器人发送。 6. 每季度由替补操作人按恢复卡走查,至少能指出归档类别、三条恢复入口、`/ready` 和关键计数验收项。不能独立说明时,本季度恢复门禁不通过。 7. 卸载 timer 或 receiver 时保留历史 archive 和源端完整批次;账号、密钥、备份数据的删除必须另行批准。 +8. 改动每日备份时刻后,人工确认次日 UTC 分桶和 26 小时窗口,避免把调整日的预期空窗误当成长期故障。 WROAD2/WARCH3 企微受控上线每日巡检: diff --git a/scripts/check-offhost-backup-deadman.py b/scripts/check-offhost-backup-deadman.py new file mode 100755 index 00000000..456290b3 --- /dev/null +++ b/scripts/check-offhost-backup-deadman.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Validate the newest ARM archive and drive an independent dead-man alert.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import stat +import subprocess +import sys +import tempfile +from datetime import datetime, timezone +from pathlib import Path + + +BATCH_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))$") + + +def fail(message: str) -> "NoReturn": + print(f"[offhost-deadman][ERROR] {message}", file=sys.stderr) + raise SystemExit(1) + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def validate_latest(archive_root: Path, max_age: int) -> tuple[str, str]: + if not archive_root.is_dir() or archive_root.is_symlink(): + return "FAIL_OFFHOST_DEADMAN", "archive_root_invalid" + candidates = sorted( + (entry for entry in archive_root.iterdir() if entry.is_dir() and not entry.is_symlink() and BATCH_RE.fullmatch(entry.name)), + key=lambda entry: entry.name, + ) + if not candidates: + return "FAIL_OFFHOST_DEADMAN", "archive_missing" + batch = candidates[-1] + match = BATCH_RE.fullmatch(batch.name) + assert match is not None + run_id, stamp = match.groups() + expected_names = {"database.dump", "storage.tar.gz", "manifest.json"} + entries = list(batch.iterdir()) + if {entry.name for entry in entries} != expected_names or any( + entry.is_symlink() or not stat.S_ISREG(entry.stat().st_mode) for entry in entries + ): + return "FAIL_OFFHOST_DEADMAN", "archive_files_invalid" + try: + manifest = json.loads((batch / "manifest.json").read_text(encoding="utf-8")) + if manifest.get("schemaVersion") != 1 or manifest.get("status") != "complete" or manifest.get("runId") != run_id: + raise ValueError + artifacts = manifest.get("artifacts") or {} + for key, name in (("database", "database.dump"), ("storage", "storage.tar.gz")): + item = artifacts.get(key) or {} + target = batch / name + if item.get("file") != name or item.get("bytes") != target.stat().st_size or item.get("sha256") != sha256(target): + raise ValueError + completed = datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc) + except (OSError, ValueError, TypeError, json.JSONDecodeError): + return "FAIL_OFFHOST_DEADMAN", "archive_manifest_invalid" + age = (datetime.now(timezone.utc) - completed).total_seconds() + if age < 0: + return "FAIL_OFFHOST_DEADMAN", "archive_time_in_future" + if age > max_age: + return "FAIL_OFFHOST_DEADMAN", "archive_stale" + return "PASS_OFFHOST_DEADMAN_FRESH", "fresh_offhost_archive" + + +def write_summary(path: Path, status: str, category: str) -> None: + finding = status.startswith("FAIL_") + payload = { + "issueId": "AUTO_BACKUP", + "status": status, + "conclusion": status, + "findingCount": 1 if finding else 0, + "findings": ([{"source": "offhost_deadman", "status": status, "severity": "P0", "detail": category}] if finding else []), + "privacy": "status/count/category only; no credentials, host addresses, absolute paths, file names, or payloads", + } + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + fd, temporary = tempfile.mkstemp(prefix=".deadman-summary.", dir=path.parent) + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(payload, handle, ensure_ascii=True, indent=2) + handle.write("\n") + os.chmod(temporary, 0o600) + os.replace(temporary, path) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--archive-root", required=True, type=Path) + parser.add_argument("--state-dir", required=True, type=Path) + parser.add_argument("--webhook-file", required=True, type=Path) + parser.add_argument("--alert-helper", required=True, type=Path) + parser.add_argument("--max-age-seconds", type=int, default=93600) + args = parser.parse_args() + if args.max_age_seconds <= 0: + fail("max age must be positive") + try: + args.state_dir.mkdir(parents=True, exist_ok=True, mode=0o700) + if args.state_dir.is_symlink() or not args.state_dir.is_dir(): + fail("state directory is invalid") + os.chmod(args.state_dir, 0o700) + status, category = validate_latest(args.archive_root, args.max_age_seconds) + summary = args.state_dir / "last-deadman.json" + write_summary(summary, status, category) + result = subprocess.run( + [sys.executable, str(args.alert_helper), "--webhook-file", str(args.webhook_file), "--summary-file", str(summary), "--state-file", str(args.state_dir / "deadman-alert-state.json")], + check=False, + ) + except OSError: + fail("dead-man local file operation failed") + if result.returncode != 0: + fail("alert evaluation failed") + if status.startswith("FAIL_"): + fail(category) + print("[offhost-deadman][PASS] newest archive is fresh and valid") + + +if __name__ == "__main__": + main() diff --git a/scripts/qa/run-automatic-backup-contract.sh b/scripts/qa/run-automatic-backup-contract.sh index f654786d..4736ccec 100755 --- a/scripts/qa/run-automatic-backup-contract.sh +++ b/scripts/qa/run-automatic-backup-contract.sh @@ -78,7 +78,7 @@ receiver() { "$ROOT_DIR/scripts/receive-offhost-backup.sh" "$@" } -CURRENT_RUN_ID="auto-$(date -u +%Y%m%dT%H%M%SZ)" +CURRENT_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z" CURRENT_BATCH="hyb-oa-backup-$CURRENT_RUN_ID" VALID_LAYOUT="$TEST_DIR/receiver-valid" @@ -92,15 +92,41 @@ pass "receiver finalizes one complete synthetic batch" receiver "$VALID_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$VALID_MANIFEST_SHA" >/dev/null pass "receiver treats an identical repeated finalize as idempotent" -SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000000Z" -if [[ "$SECOND_RUN_ID" == "$CURRENT_RUN_ID" ]]; then - SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z" -fi +SECOND_RUN_ID="auto-$(date -u +%Y%m%d)T000001Z" SECOND_BATCH="hyb-oa-backup-$SECOND_RUN_ID" make_batch "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID" SECOND_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')" -expect_failure "receiver rejects a different automatic batch for the same UTC day" \ - receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA" +receiver "$VALID_LAYOUT" finalize-test "$SECOND_RUN_ID" "$SECOND_MANIFEST_SHA" >/dev/null +[[ ! -e "$VALID_LAYOUT/archive/$CURRENT_BATCH" && -d "$VALID_LAYOUT/archive/$SECOND_BATCH" ]] +WEEKLY_BATCH="$(find "$VALID_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)" +[[ -n "$WEEKLY_BATCH" ]] +[[ "$(stat -c %i "$WEEKLY_BATCH/database.dump")" == "$(stat -c %i "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump")" ]] +pass "receiver atomically replaces a different batch for the same UTC day and rebuilds its weekly checkpoint" + +THIRD_RUN_ID="auto-$(date -u +%Y%m%d)T000002Z" +THIRD_BATCH="hyb-oa-backup-$THIRD_RUN_ID" +make_batch "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial" "$THIRD_RUN_ID" +printf 'corrupt candidate\n' >>"$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/database.dump" +THIRD_MANIFEST_SHA="$(sha256sum "$VALID_LAYOUT/inbox/${THIRD_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver validates a same-day replacement before touching the current archive" \ + receiver "$VALID_LAYOUT" finalize-test "$THIRD_RUN_ID" "$THIRD_MANIFEST_SHA" +[[ -d "$VALID_LAYOUT/archive/$SECOND_BATCH" && ! -e "$VALID_LAYOUT/archive/$THIRD_BATCH" ]] +pass "receiver preserves the current daily archive when a replacement candidate is corrupt" + +ROLLBACK_LAYOUT="$TEST_DIR/receiver-replacement-rollback" +make_layout "$ROLLBACK_LAYOUT" +make_batch "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial" "$CURRENT_RUN_ID" +ROLLBACK_FIRST_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${CURRENT_BATCH}.partial/manifest.json" | awk '{print $1}')" +receiver "$ROLLBACK_LAYOUT" finalize-test "$CURRENT_RUN_ID" "$ROLLBACK_FIRST_SHA" >/dev/null +ROLLBACK_WEEKLY="$(find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -print -quit)" +find "$ROLLBACK_LAYOUT/weekly" -mindepth 1 -maxdepth 1 -type d -exec rm -rf -- {} + +printf 'invalid weekly target\n' >"$ROLLBACK_WEEKLY" +make_batch "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial" "$SECOND_RUN_ID" +ROLLBACK_SECOND_SHA="$(sha256sum "$ROLLBACK_LAYOUT/inbox/${SECOND_BATCH}.partial/manifest.json" | awk '{print $1}')" +expect_failure "receiver rolls back the daily archive when weekly checkpoint publication fails" \ + receiver "$ROLLBACK_LAYOUT" finalize-test "$SECOND_RUN_ID" "$ROLLBACK_SECOND_SHA" +[[ -d "$ROLLBACK_LAYOUT/archive/$CURRENT_BATCH" && ! -e "$ROLLBACK_LAYOUT/archive/$SECOND_BATCH" ]] +pass "receiver restores the prior daily point after a later replacement step fails" negative_receiver_case() { local label="$1" @@ -225,7 +251,7 @@ if [[ "${1:-}" == "compose" ]]; then args=" $* " if [[ "$args" == *" ps -q server "* ]]; then echo server-id; exit 0; fi if [[ "$args" == *" ps -q db "* ]]; then echo db-id; exit 0; fi - if [[ "$args" == *" exec -T server "* ]]; then cat >>"$MOCK_ALERT_LOG"; exit 0; fi + if [[ "$args" == *" exec -T server "* ]]; then exit 99; fi fi if [[ "${1:-}" == "inspect" ]]; then template="${3:-}" @@ -260,6 +286,7 @@ MOCK cat >"$MOCK_BIN/rsync" <<'MOCK' #!/usr/bin/env bash +echo 'synthetic transfer diagnostic' >&2 exit "${MOCK_RSYNC_EXIT:-1}" MOCK cat >"$MOCK_BIN/ssh" <<'MOCK' @@ -267,6 +294,34 @@ cat >"$MOCK_BIN/ssh" <<'MOCK' exit "${MOCK_SSH_EXIT:-0}" MOCK chmod +x "$MOCK_BIN/docker" "$MOCK_BIN/rsync" "$MOCK_BIN/ssh" +WECOM_TEST_HOST='qyapi.weixin.qq.com' +WECOM_TEST_PATH='/cgi-bin/webhook/send' +printf 'https://%s%s?key=%s\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \ + '00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook" +chmod 0600 "$TEST_DIR/wecom-webhook" +ALERT_CAPTURE="$TEST_DIR/alert-capture.jsonl" +cat >"$TEST_DIR/failure-summary.json" <<'JSON' +{"issueId":"AUTO_BACKUP","status":"FAIL_AUTOMATIC_BACKUP","conclusion":"FAIL_AUTOMATIC_BACKUP","findingCount":1,"findings":[{"source":"automatic_backup","status":"FAIL_AUTOMATIC_BACKUP","severity":"P0","detail":"synthetic_failure"}]} +JSON +chmod 0600 "$TEST_DIR/failure-summary.json" +chmod 0644 "$TEST_DIR/wecom-webhook" +expect_failure "group alert helper rejects a webhook file that is not mode 0600" env \ + HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \ + python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook" \ + --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json" +chmod 0600 "$TEST_DIR/wecom-webhook" +ln -s "$TEST_DIR/wecom-webhook" "$TEST_DIR/wecom-webhook-link" +expect_failure "group alert helper rejects a symbolic-link webhook file" env \ + HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \ + python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-link" \ + --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json" +printf 'https://%s%s?key=%s&extra=forbidden\n' "$WECOM_TEST_HOST" "$WECOM_TEST_PATH" \ + '00000000-0000-0000-0000-000000000000' >"$TEST_DIR/wecom-webhook-invalid" +chmod 0600 "$TEST_DIR/wecom-webhook-invalid" +expect_failure "group alert helper rejects a webhook URL with an unexpected query field" env \ + HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" \ + python3 "$ROOT_DIR/scripts/send-wecom-group-alert.py" --webhook-file "$TEST_DIR/wecom-webhook-invalid" \ + --summary-file "$TEST_DIR/failure-summary.json" --state-file "$TEST_DIR/invalid-alert-state.json" wrapper_env=( env PATH="$MOCK_BIN:$PATH" REMOTE_DIR="$WRAPPER_ROOT" @@ -275,7 +330,9 @@ wrapper_env=( STORAGE_HOST_DIR="$WRAPPER_STORAGE" BUILD_SHA="$BUILD_SHA" VITE_BUILD_SHA="$VITE_BUILD_SHA" AUTOMATIC_BACKUP_DIR="$WRAPPER_BACKUPS" AUTOMATIC_BACKUP_STATE_DIR="$WRAPPER_STATE" OFFHOST_TARGET="hybbackup@backup-alias" OFFHOST_SSH_KEY="$TEST_DIR/offhost-key" - OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_RECIPIENT_EMPLOYEE_NO="OPS001" + OFFHOST_KNOWN_HOSTS="$TEST_DIR/known-hosts" ALERT_WEBHOOK_FILE="$TEST_DIR/wecom-webhook" + ALERT_HELPER="$ROOT_DIR/scripts/send-wecom-group-alert.py" + HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$ALERT_CAPTURE" ) exec 8>"$WRAPPER_STATE/automatic-backup.lock" @@ -305,6 +362,9 @@ assert payload["status"] == "FAIL_AUTOMATIC_BACKUP" assert not any(value in serialized for value in ("/tmp/", "postgresql://", "backup-alias", "fixture")) PYTHON pass "wrapper failure summary is category-only and redacted" +[[ -f "$WRAPPER_STATE/last-error.log" && "$(stat -c %a "$WRAPPER_STATE/last-error.log")" == "600" ]] +grep -Fq 'synthetic transfer diagnostic' "$WRAPPER_STATE/last-error.log" +pass "wrapper retains the latest command diagnostic locally with mode 0600" find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' -exec rm -rf -- {} + for day in 01 02 03 04 05 06 07; do @@ -315,6 +375,8 @@ done [[ -f "$WRAPPER_STATE/last-success.json" ]] [[ "$(find "$WRAPPER_BACKUPS" -mindepth 1 -maxdepth 1 -type d -name 'hyb-oa-backup-auto-*' | wc -l)" == "7" ]] pass "off-host finalize success writes state and then prunes source retention to seven" +[[ ! -e "$WRAPPER_STATE/last-error.log" ]] +pass "wrapper clears the retained diagnostic after a successful off-host run" rm -f "$WRAPPER_STATE/last-success.json" expect_failure "stale watchdog rejects a missing success record" \ @@ -329,7 +391,7 @@ with open(sys.argv[1], "w", encoding="utf-8") as handle: "schemaVersion": 1, "status": "complete", "completedAt": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), - "runId": "auto-synthetic", + "runId": "auto-20260727T000000Z", "manifestSha256": "0" * 64, "composeProject": "hyb-oa-current", "buildSha": sys.argv[2], @@ -345,12 +407,89 @@ payload["buildSha"] = "0" * 40 with open(sys.argv[1], "w", encoding="utf-8") as handle: json.dump(payload, handle) PYTHON -expect_failure "stale watchdog rejects a success record from another build" \ - "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale +"${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null +python3 - "$WRAPPER_STATE/last-run.json" <<'PYTHON' +import json +import sys +with open(sys.argv[1], encoding="utf-8") as handle: + payload = json.load(handle) +assert payload["status"] == "PASS_WITH_FINDINGS" +assert payload["findings"][0]["detail"] == "recovery_point_build_differs" +PYTHON +pass "stale watchdog downgrades a fresh recovery point from another build to one finding" mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json" "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale >/dev/null pass "stale watchdog accepts a fresh project/build-bound success record" +cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json" +python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON' +import json +import sys +with open(sys.argv[1], encoding="utf-8") as handle: + payload = json.load(handle) +payload["composeProject"] = "different-project" +with open(sys.argv[1], "w", encoding="utf-8") as handle: + json.dump(payload, handle) +PYTHON +expect_failure "stale watchdog hard-fails a fresh record from another Compose project" \ + "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale +mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json" +cp "$WRAPPER_STATE/last-success.json" "$WRAPPER_STATE/last-success.valid.json" +python3 - "$WRAPPER_STATE/last-success.json" <<'PYTHON' +import json +import sys +with open(sys.argv[1], encoding="utf-8") as handle: + payload = json.load(handle) +payload["manifestSha256"] = "invalid" +with open(sys.argv[1], "w", encoding="utf-8") as handle: + json.dump(payload, handle) +PYTHON +expect_failure "stale watchdog hard-fails a malformed success record" \ + "${wrapper_env[@]}" "$ROOT_DIR/scripts/run-automatic-backup.sh" check-stale +mv "$WRAPPER_STATE/last-success.valid.json" "$WRAPPER_STATE/last-success.json" + +DEADMAN_LAYOUT="$TEST_DIR/deadman" +mkdir -p "$DEADMAN_LAYOUT/archive" "$DEADMAN_LAYOUT/state" +cp -a "$VALID_LAYOUT/archive/$SECOND_BATCH" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" +DEADMAN_CAPTURE="$TEST_DIR/deadman-alerts.jsonl" +deadman_env=(env HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CAPTURE") +"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \ + --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \ + --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null +[[ ! -e "$DEADMAN_CAPTURE" ]] +pass "ARM dead-man accepts a fresh verified archive without sending a healthy baseline" +DEADMAN_CORRUPT_STATE="$TEST_DIR/deadman-corrupt-state" +DEADMAN_CORRUPT_CAPTURE="$TEST_DIR/deadman-corrupt-alerts.jsonl" +mkdir "$DEADMAN_CORRUPT_STATE" +chmod 0600 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump" +printf 'corruption\n' >>"$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump" +expect_failure "ARM dead-man rejects a corrupt newest finalized archive" env \ + HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$DEADMAN_CORRUPT_CAPTURE" \ + python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \ + --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_CORRUPT_STATE" \ + --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" +cp "$VALID_LAYOUT/archive/$SECOND_BATCH/database.dump" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump" +chmod 0400 "$DEADMAN_LAYOUT/archive/$SECOND_BATCH/database.dump" +pass "ARM dead-man detects archive corruption independently of source state" +mv "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" "$TEST_DIR/deadman-withheld-batch" +expect_failure "ARM dead-man detects that the source no longer produces a finalized archive" \ + "${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \ + --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \ + --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" +[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]] +expect_failure "ARM dead-man suppresses an identical repeated failure alert" \ + "${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \ + --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \ + --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" +[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "1" ]] +pass "ARM dead-man keeps repeated identical failure alerts deduplicated" +mv "$TEST_DIR/deadman-withheld-batch" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH" +"${deadman_env[@]}" python3 "$ROOT_DIR/scripts/check-offhost-backup-deadman.py" \ + --archive-root "$DEADMAN_LAYOUT/archive" --state-dir "$DEADMAN_LAYOUT/state" \ + --webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null +[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]] +pass "ARM dead-man sends one recovery notification after finalized archives resume" + DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567" expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \ "${wrapper_env[@]}" BUILD_SHA="$DRIFT_BUILD_SHA" VITE_BUILD_SHA="${DRIFT_BUILD_SHA:0:12}" \ diff --git a/scripts/receive-offhost-backup.sh b/scripts/receive-offhost-backup.sh index 6d4163bb..c6ce246d 100755 --- a/scripts/receive-offhost-backup.sh +++ b/scripts/receive-offhost-backup.sh @@ -60,6 +60,7 @@ validate_layout() { [[ "$DAILY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "DAILY_KEEP must be positive" [[ "$WEEKLY_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "WEEKLY_KEEP must be positive" command -v "$PYTHON_BIN" >/dev/null 2>&1 || fail "python3 is required" + command -v flock >/dev/null 2>&1 || fail "flock is required" } validate_run_id() { @@ -155,11 +156,44 @@ prune_named_directories() { done } +publish_weekly_checkpoint() { + local archive_dir="$1" + local weekly_dir="$2" + local replace_existing="$3" + if [[ -e "$weekly_dir" && "$replace_existing" != "1" ]]; then + return 0 + fi + local weekly_temp="${weekly_dir}.new.$$" + local weekly_rollback="${weekly_dir}.rollback.$$" + [[ ! -e "$weekly_temp" && ! -e "$weekly_rollback" ]] || return 1 + if ! cp -al -- "$archive_dir" "$weekly_temp"; then + rm -rf -- "$weekly_temp" + return 1 + fi + if [[ -e "$weekly_dir" ]]; then + [[ -d "$weekly_dir" && ! -L "$weekly_dir" ]] || return 1 + mv -- "$weekly_dir" "$weekly_rollback" || return 1 + fi + if ! mv -- "$weekly_temp" "$weekly_dir"; then + rm -rf -- "$weekly_temp" + if [[ -e "$weekly_rollback" ]]; then + mv -- "$weekly_rollback" "$weekly_dir" + fi + return 1 + fi + if [[ -e "$weekly_rollback" ]]; then + rm -rf -- "$weekly_rollback" + fi +} + finalize_batch() { local run_id="$1" local expected_manifest_sha="$2" validate_run_id "$run_id" validate_layout + exec 9>"$BACKUP_ARCHIVE_DIR/.finalize.lock" + chmod 0600 "$BACKUP_ARCHIVE_DIR/.finalize.lock" + flock -n 9 || fail "another finalize operation is already running" local batch_name="hyb-oa-backup-$run_id" local partial_dir="$BACKUP_INBOX_DIR/${batch_name}.partial" @@ -173,15 +207,32 @@ finalize_batch() { local run_day="${run_id#auto-}" run_day="${run_day%%T*}" - local existing_for_day - existing_for_day="$(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \ - -name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print -quit)" - [[ -z "$existing_for_day" ]] || fail "another automatic archive already exists for this UTC day" + local existing_for_day="" replacement_rollback="" + local -a existing_for_day_entries=() + mapfile -t existing_for_day_entries < <(find "$BACKUP_ARCHIVE_DIR" -mindepth 1 -maxdepth 1 -type d \ + -name "hyb-oa-backup-auto-${run_day}T*Z" ! -name "$batch_name" -print) + [[ "${#existing_for_day_entries[@]}" -le 1 ]] || fail "daily archive invariant is already violated" + if [[ "${#existing_for_day_entries[@]}" -eq 1 ]]; then + existing_for_day="${existing_for_day_entries[0]}" + fi if [[ "$already_archived" == "0" ]]; then validate_batch "$partial_dir" "$run_id" "$expected_manifest_sha" chmod 0400 "$partial_dir/database.dump" "$partial_dir/storage.tar.gz" "$partial_dir/manifest.json" - mv -- "$partial_dir" "$archive_dir" + if [[ -n "$existing_for_day" ]]; then + [[ "$existing_for_day" =~ /hyb-oa-backup-auto-${run_day}T[0-9]{6}Z$ \ + && -d "$existing_for_day" && ! -L "$existing_for_day" ]] \ + || fail "existing daily archive is invalid" + replacement_rollback="$BACKUP_ARCHIVE_DIR/.daily-replacement-${run_day}.$$" + [[ ! -e "$replacement_rollback" ]] || fail "daily replacement rollback target already exists" + mv -- "$existing_for_day" "$replacement_rollback" + fi + if ! mv -- "$partial_dir" "$archive_dir"; then + if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then + mv -- "$replacement_rollback" "$existing_for_day" + fi + fail "daily archive publication failed" + fi fi local iso_week @@ -192,10 +243,17 @@ day = datetime.strptime(sys.argv[1], "%Y%m%d").date() iso = day.isocalendar() print(f"week-{iso.year:04d}-W{iso.week:02d}") PYTHON -)" + )" local weekly_dir="$BACKUP_WEEKLY_DIR/$iso_week" - if [[ ! -e "$weekly_dir" ]]; then - cp -al -- "$archive_dir" "$weekly_dir" + if ! publish_weekly_checkpoint "$archive_dir" "$weekly_dir" "$([[ -n "$replacement_rollback" ]] && echo 1 || echo 0)"; then + if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then + rm -rf -- "$archive_dir" + mv -- "$replacement_rollback" "$existing_for_day" + fi + fail "weekly checkpoint update failed" + fi + if [[ -n "$replacement_rollback" && -e "$replacement_rollback" ]]; then + rm -rf -- "$replacement_rollback" fi prune_named_directories "$BACKUP_ARCHIVE_DIR" "hyb-oa-backup-auto-" "$DAILY_KEEP" diff --git a/scripts/run-automatic-backup.sh b/scripts/run-automatic-backup.sh index d3ed75ce..b4bc573b 100755 --- a/scripts/run-automatic-backup.sh +++ b/scripts/run-automatic-backup.sh @@ -15,9 +15,10 @@ AUTOMATIC_BACKUP_STATE_DIR="${AUTOMATIC_BACKUP_STATE_DIR:-}" OFFHOST_TARGET="${OFFHOST_TARGET:-}" OFFHOST_SSH_KEY="${OFFHOST_SSH_KEY:-}" OFFHOST_KNOWN_HOSTS="${OFFHOST_KNOWN_HOSTS:-}" +ALERT_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-}" +ALERT_HELPER="${ALERT_HELPER:-$ROOT_DIR/scripts/send-wecom-group-alert.py}" AUTOMATIC_BACKUP_KEEP="${AUTOMATIC_BACKUP_KEEP:-7}" AUTOMATIC_BACKUP_MAX_AGE_SECONDS="${AUTOMATIC_BACKUP_MAX_AGE_SECONDS:-86400}" -ALERT_RECIPIENT_EMPLOYEE_NO="${ALERT_RECIPIENT_EMPLOYEE_NO:-HYD000006}" MODE="${1:-run}" CURRENT_PHASE="preflight" @@ -79,7 +80,12 @@ validate_common() { || fail "frontend build SHA must match the full build prefix" [[ "$AUTOMATIC_BACKUP_KEEP" =~ ^[1-9][0-9]*$ ]] || fail "backup retention must be positive" [[ "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" =~ ^[1-9][0-9]*$ ]] || fail "stale threshold must be positive" - [[ "$ALERT_RECIPIENT_EMPLOYEE_NO" =~ ^[A-Za-z0-9_-]{1,80}$ ]] || fail "alert recipient id is invalid" + validate_absolute_path ALERT_WEBHOOK_FILE "$ALERT_WEBHOOK_FILE" + validate_absolute_path ALERT_HELPER "$ALERT_HELPER" + [[ -f "$ALERT_WEBHOOK_FILE" && ! -L "$ALERT_WEBHOOK_FILE" \ + && "$(stat -c %a "$ALERT_WEBHOOK_FILE")" == "600" ]] \ + || fail "alert webhook file must exist with mode 0600" + [[ -f "$ALERT_HELPER" && ! -L "$ALERT_HELPER" ]] || fail "alert helper is missing" if [[ "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR" \ || "$AUTOMATIC_BACKUP_DIR" == "$AUTOMATIC_BACKUP_STATE_DIR"/* \ || "$AUTOMATIC_BACKUP_STATE_DIR" == "$AUTOMATIC_BACKUP_DIR"/* ]]; then @@ -218,10 +224,25 @@ PYTHON send_summary_alert() { local state_name="$1" [[ -n "$SUMMARY_FILE" && -f "$SUMMARY_FILE" ]] || return 1 - [[ -n "$SERVER_CONTAINER" ]] || resolve_active_runtime - compose exec -T server sh -lc \ - "cd /app/server && WARCH3_OPS_ALERT_SEND=1 npm run send:wecom-ops-alert -- --summary-file=- --recipient-employee-no=${ALERT_RECIPIENT_EMPLOYEE_NO} --send --state-file=/app/artifacts/qa-flow/${state_name} --format=json" \ - <"$SUMMARY_FILE" >/dev/null + python3 "$ALERT_HELPER" \ + --webhook-file "$ALERT_WEBHOOK_FILE" \ + --summary-file "$SUMMARY_FILE" \ + --state-file "$AUTOMATIC_BACKUP_STATE_DIR/$state_name" >/dev/null +} + +capture_command() { + local phase="$1" + shift + local temporary="$AUTOMATIC_BACKUP_STATE_DIR/.${phase}.diagnostic.$$" + : >"$temporary" + chmod 0600 "$temporary" + if "$@" >"$temporary" 2>&1; then + rm -f -- "$temporary" + return 0 + fi + mv -f -- "$temporary" "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log" + chmod 0600 "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log" + return 1 } on_error() { @@ -320,11 +341,12 @@ print(f"{unquote(value.username)}\t{database_name}") [[ "$source_image_digest" =~ ^sha256:[0-9a-f]{64}$ ]] || fail "active server image digest is invalid" CURRENT_PHASE="paired_backup" - if ! DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \ + if ! capture_command paired_backup env \ + DATABASE_URL="$database_url" DB_TOOL_MODE=docker DB_CONTAINER="$DB_CONTAINER" \ POSTGRES_DB="$postgres_db" POSTGRES_USER="$postgres_user" POSTGRES_PASSWORD="$postgres_password" \ STORAGE_DIR_HOST="$STORAGE_HOST_DIR" BACKUP_DIR="$AUTOMATIC_BACKUP_DIR" \ BACKUP_RUN_ID="$run_id" BUILD_SHA="$BUILD_SHA" SOURCE_IMAGE_DIGEST="$source_image_digest" \ - "$ROOT_DIR/scripts/backup-all.sh" >/dev/null 2>&1; then + "$ROOT_DIR/scripts/backup-all.sh"; then fail "paired backup command failed" fi [[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]] || fail "complete local batch was not published" @@ -343,14 +365,14 @@ print(f"{unquote(value.username)}\t{database_name}") "$OFFHOST_SSH_KEY" "$OFFHOST_KNOWN_HOSTS" CURRENT_PHASE="offhost_transfer" - if ! rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \ - "$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/" >/dev/null 2>&1; then + if ! capture_command offhost_transfer rsync -a --partial --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= -e "$rsync_shell" \ + "$batch_dir/" "$OFFHOST_TARGET:${batch_name}.partial/"; then fail "off-host transfer failed" fi CURRENT_PHASE="offhost_finalize" - if ! ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \ - "hyb-backup-finalize $run_id $manifest_sha" >/dev/null 2>&1; then + if ! capture_command offhost_finalize ssh "${ssh_options[@]}" "$OFFHOST_TARGET" \ + "hyb-backup-finalize $run_id $manifest_sha"; then fail "off-host finalize failed" fi @@ -359,6 +381,7 @@ print(f"{unquote(value.username)}\t{database_name}") prune_local_backups "$batch_name" write_summary "PASS_AUTOMATIC_BACKUP_COMPLETE" 0 "offhost_verified" send_summary_alert "automatic-backup-alert-state.json" || true + rm -f -- "$AUTOMATIC_BACKUP_STATE_DIR/last-error.log" trap - ERR echo "[automatic-backup][PASS] paired batch finalized off-host" } @@ -369,9 +392,11 @@ check_stale() { CURRENT_PHASE="stale_check" resolve_active_runtime local success_file="$AUTOMATIC_BACKUP_STATE_DIR/last-success.json" - if ! python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON' + local stale_result + if ! stale_result="$(python3 - "$success_file" "$AUTOMATIC_BACKUP_MAX_AGE_SECONDS" "$COMPOSE_PROJECT_NAME" "$BUILD_SHA" <<'PYTHON' import json import os +import re import sys from datetime import datetime, timezone @@ -382,15 +407,29 @@ with open(file_path, "r", encoding="utf-8") as handle: payload = json.load(handle) if payload.get("schemaVersion") != 1 or payload.get("status") != "complete": raise SystemExit(1) -if payload.get("composeProject") != expected_project or payload.get("buildSha") != expected_build: +if payload.get("composeProject") != expected_project: + raise SystemExit(1) +if not re.fullmatch(r"auto-[0-9]{8}T[0-9]{6}Z", payload.get("runId", "")): + raise SystemExit(1) +if not re.fullmatch(r"[0-9a-f]{64}", payload.get("manifestSha256", "")): + raise SystemExit(1) +if not re.fullmatch(r"[0-9a-fA-F]{40,64}", payload.get("buildSha", "")): raise SystemExit(1) completed = datetime.fromisoformat(payload["completedAt"].replace("Z", "+00:00")) age = (datetime.now(timezone.utc) - completed).total_seconds() if age < 0 or age > int(max_age): raise SystemExit(1) +print("BUILD_MISMATCH" if payload.get("buildSha") != expected_build else "FRESH") PYTHON - then - fail "latest off-host success is missing, invalid, mismatched, or stale" + )"; then + fail "latest off-host success is missing, invalid, project-mismatched, or stale" + fi + if [[ "$stale_result" == "BUILD_MISMATCH" ]]; then + write_summary "PASS_WITH_FINDINGS" 1 "recovery_point_build_differs" + send_summary_alert "automatic-backup-alert-state.json" || true + trap - ERR + echo "[automatic-backup][PASS_WITH_FINDINGS] off-host recovery point is fresh but build differs" + return 0 fi write_summary "PASS_AUTOMATIC_BACKUP_FRESH" 0 "fresh_offhost_recovery_point" send_summary_alert "automatic-backup-alert-state.json" || true diff --git a/scripts/send-wecom-group-alert.py b/scripts/send-wecom-group-alert.py new file mode 100755 index 00000000..8a5e647c --- /dev/null +++ b/scripts/send-wecom-group-alert.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Send a redacted, deduplicated OA operations alert to a WeCom group robot.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import stat +import sys +import tempfile +import urllib.parse +import urllib.request +from pathlib import Path + + +SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$") +ALLOWED_STATUSES = { + "FAIL_AUTOMATIC_BACKUP", + "FAIL_OFFHOST_DEADMAN", + "PASS_AUTOMATIC_BACKUP_COMPLETE", + "PASS_AUTOMATIC_BACKUP_FRESH", + "PASS_OFFHOST_DEADMAN_FRESH", + "PASS_WITH_FINDINGS", +} + + +def fail(message: str) -> "NoReturn": + print(f"[wecom-group-alert][ERROR] {message}", file=sys.stderr) + raise SystemExit(1) + + +def validate_private_file(path: Path, label: str) -> None: + try: + info = path.lstat() + except FileNotFoundError: + fail(f"{label} is missing") + if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode): + fail(f"{label} must be a regular non-symbolic file") + if stat.S_IMODE(info.st_mode) != 0o600: + fail(f"{label} mode must be 0600") + if info.st_uid != os.geteuid(): + fail(f"{label} must be owned by the executing account") + + +def read_webhook(path: Path) -> str: + validate_private_file(path, "webhook file") + try: + value = path.read_text(encoding="utf-8").strip() + except OSError: + fail("webhook file cannot be read") + if "\n" in value or "\r" in value: + fail("webhook file must contain one line") + parsed = urllib.parse.urlparse(value) + query = urllib.parse.parse_qs(parsed.query, strict_parsing=True) + if ( + parsed.scheme != "https" + or parsed.hostname != "qyapi.weixin.qq.com" + or parsed.netloc != "qyapi.weixin.qq.com" + or parsed.path != "/cgi-bin/webhook/send" + or set(query) != {"key"} + or len(query["key"]) != 1 + or not re.fullmatch(r"[A-Za-z0-9-]{16,128}", query["key"][0]) + or parsed.fragment + ): + fail("webhook file format is invalid") + return value + + +def load_summary(path: Path) -> tuple[str, str, int]: + validate_private_file(path, "summary file") + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + fail("summary file is invalid") + status = payload.get("status") + findings = payload.get("findings") or [] + finding_count = payload.get("findingCount") + if status not in ALLOWED_STATUSES or not isinstance(finding_count, int): + fail("summary status is invalid") + if finding_count != len(findings) or finding_count not in {0, 1}: + fail("summary finding count is invalid") + category = "healthy" + if findings: + category = findings[0].get("detail", "") + if not isinstance(category, str) or not SAFE_TOKEN.fullmatch(category): + fail("summary category is invalid") + return status, category, finding_count + + +def load_previous(path: Path) -> dict[str, object]: + if not path.exists(): + return {} + validate_private_file(path, "alert state file") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + fail("alert state file is invalid") + return value if isinstance(value, dict) else {} + + +def atomic_write(path: Path, payload: dict[str, object]) -> None: + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(path.parent, 0o700) + fd, temp_name = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent) + try: + os.fchmod(fd, 0o600) + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(payload, handle, ensure_ascii=True, indent=2) + handle.write("\n") + os.replace(temp_name, path) + except BaseException: + try: + os.unlink(temp_name) + except FileNotFoundError: + pass + raise + + +def should_send(previous: dict[str, object], status: str, category: str) -> bool: + previous_status = previous.get("status") + previous_category = previous.get("category") + if status.startswith("FAIL_") or status == "PASS_WITH_FINDINGS": + return previous_status != status or previous_category != category + return isinstance(previous_status, str) and ( + previous_status.startswith("FAIL_") or previous_status == "PASS_WITH_FINDINGS" + ) + + +def safe_text(status: str, category: str) -> str: + if status.startswith("FAIL_"): + headline = "内部 OA 运维告警" + elif status == "PASS_WITH_FINDINGS": + headline = "内部 OA 运维提醒" + else: + headline = "内部 OA 运维恢复通知" + return f"{headline}\n状态:{status}\n类别:{category}\n请按恢复卡检查,不要在群内粘贴原始日志。" + + +def send(webhook: str, text: str) -> None: + body = json.dumps({"msgtype": "text", "text": {"content": text}}).encode("utf-8") + capture = os.environ.get("HYB_WECOM_ALERT_TEST_CAPTURE") + if capture: + if os.environ.get("HYB_WECOM_ALERT_TEST_MODE") != "1": + fail("test capture is disabled") + capture_path = Path(capture) + with capture_path.open("ab") as handle: + handle.write(body + b"\n") + os.chmod(capture_path, 0o600) + return + request = urllib.request.Request( + webhook, + data=body, + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=10) as response: + result = json.loads(response.read().decode("utf-8")) + except Exception: + fail("group robot request failed") + if result.get("errcode") != 0: + fail("group robot rejected the alert") + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--webhook-file", required=True, type=Path) + parser.add_argument("--summary-file", required=True, type=Path) + parser.add_argument("--state-file", required=True, type=Path) + args = parser.parse_args() + + try: + status, category, _ = load_summary(args.summary_file) + previous = load_previous(args.state_file) + send_now = should_send(previous, status, category) + if send_now: + webhook = read_webhook(args.webhook_file) + send(webhook, safe_text(status, category)) + atomic_write(args.state_file, {"schemaVersion": 1, "status": status, "category": category}) + except OSError: + fail("local alert file operation failed") + print(f"[wecom-group-alert][PASS] evaluated sent={str(send_now).lower()}") + + +if __name__ == "__main__": + main()