备份策略改为部署前手动快照：部署合同增加手动快照模式 [08-28-production-arm-backup-transfer-incident]
- 用户决定仅用部署前手动轻量快照，不再推进对象存储、ARM/hyinda-20c 备份 - refresh-deployment-contract 新增 /etc/hyb-oa-backup-mode 显式模式： 无文件保持原自动模式；manual_snapshot 要求三个备份 timer 停用且备份 service 已退出，ops timer 仍须运行；内容/属主/符号链接异常 fail-closed - probe 部署前与 refresh 部署后均校验模式与 timer/service 一致性 - 新增 44 backup_mode_invalid / 45 backup_timer_not_disabled 分类及合同测试 - 记录 10-05 ARM 与 hyinda-20c 只读核验结果、远端切换顺序与回滚 - 远端切换未执行，需另行批准 Co-Authored-By: Claude Opus 5.5 <[email protected]>
这个提交包含在：
Claude Opus 5.5
@@ -37,6 +37,22 @@ operations require explicit approval and evidence.
|
|||||||
`deploy/trialrun/.env.example` before deployment; `deploy/trialrun/deploy.sh`
|
`deploy/trialrun/.env.example` before deployment; `deploy/trialrun/deploy.sh`
|
||||||
runs `node deploy/trialrun/validate-env-contract.js` at step 0 and must fail
|
runs `node deploy/trialrun/validate-env-contract.js` at step 0 and must fail
|
||||||
if the example contract drifts.
|
if the example contract drifts.
|
||||||
|
- Current backup policy is manual Tencent Lighthouse snapshots only, as selected
|
||||||
|
in `08-28-production-arm-backup-transfer-incident/prd.md`. Do not introduce
|
||||||
|
object storage, ARM/hyinda-20c backup or automatic snapshot rotation by default.
|
||||||
|
Before each approved production deployment, confirm the OA target and coverage
|
||||||
|
of database/attachment disks, then wait for snapshot creation to succeed.
|
||||||
|
Record target/coverage confirmation, creation time and success without resource
|
||||||
|
IDs. Check current console quota before any separately authorized old-snapshot
|
||||||
|
deletion; preserve an existing successful recovery point during rotation.
|
||||||
|
Recoverability is limited to existing snapshot times and covered disks; this
|
||||||
|
policy provides neither a daily recovery-point guarantee nor an independent copy.
|
||||||
|
- Policy selection does not prove automatic backups have stopped. Follow the
|
||||||
|
same task's reviewed transition plan: preserve the service-based deployment
|
||||||
|
identity contract, adapt backup timer checks before disabling those timers,
|
||||||
|
and separately approve remote configuration changes. Preserve historical
|
||||||
|
backups and independent public-health monitoring. Legacy backup contracts below
|
||||||
|
describe retained tooling; they do not authorize re-enabling backup services.
|
||||||
- After deployment, report commit hash, `/ready`, public smoke results and
|
- After deployment, report commit hash, `/ready`, public smoke results and
|
||||||
whether the user can test.
|
whether the user can test.
|
||||||
- Real business writes are manual unless the current Trellis task explicitly
|
- Real business writes are manual unless the current Trellis task explicitly
|
||||||
@@ -2489,6 +2505,7 @@ DEPLOY_SERVER=<tencent-oa-production-target> bash deploy/trialrun/deploy-existin
|
|||||||
- An early aggregate item fails -> all independent items still run, the fixed summary reports every FAIL, and no remote write begins.
|
- An early aggregate item fails -> all independent items still run, the fixed summary reports every FAIL, and no remote write begins.
|
||||||
- Root SSH returns transport 255 -> `root_identity_unavailable`; remote 126/127 or missing probe marker -> `refresh_program_unavailable`.
|
- Root SSH returns transport 255 -> `root_identity_unavailable`; remote 126/127 or missing probe marker -> `refresh_program_unavailable`.
|
||||||
- Helper runs but final service build differs -> `unit_build_not_refreshed`; builds match but any timer is disabled/inactive -> `timer_not_active`.
|
- Helper runs but final service build differs -> `unit_build_not_refreshed`; builds match but any timer is disabled/inactive -> `timer_not_active`.
|
||||||
|
- Backup mode file `/etc/hyb-oa-backup-mode` absent -> automatic mode, all backup and ops timers must be enabled/active. Content exactly `manual_snapshot` (root-owned regular file) -> the three automatic-backup timers must be disabled/inactive, otherwise `backup_timer_not_disabled` (45); `hyb-oa-warch3-ops.timer` stays required (`timer_not_active`). Any other content, symlink or ownership -> `backup_mode_invalid` (44). In manual mode every backup `.service` must also be `inactive` or `failed`; an in-flight run -> 45. `probe` runs the same mode and timer/service checks, so a mismatch fails the pre-deploy aggregate as `refresh_program FAIL` (run the forced-command `probe` directly to read the category on stderr) before any release write; `refresh` re-checks after the release. Never mask or delete backup units: deploy identity still reads the daily backup service environment.
|
||||||
- A similarly named environment key such as `OLD_BUILD_SHA` matches the active value while exact `BUILD_SHA` remains stale -> treat the service as stale; build verification parses exact keys and applies the last systemd override.
|
- A similarly named environment key such as `OLD_BUILD_SHA` matches the active value while exact `BUILD_SHA` remains stale -> treat the service as stale; build verification parses exact keys and applies the last systemd override.
|
||||||
- Post-deploy installer still embeds the previous build SHA, any managed timer is inactive, or the real backup/watchdog fails runtime identity or off-host freshness -> deployment recovery remains incomplete even when `/ready` is green.
|
- Post-deploy installer still embeds the previous build SHA, any managed timer is inactive, or the real backup/watchdog fails runtime identity or off-host freshness -> deployment recovery remains incomplete even when `/ready` is green.
|
||||||
- A later deployment attempt is blocked by `backup_build` after `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED` -> expected fail-closed behavior; repair with direct forced-command `refresh`, not another application release.
|
- A later deployment attempt is blocked by `backup_build` after `DEPLOYED_BUT_CONTRACT_REFRESH_FAILED` -> expected fail-closed behavior; repair with direct forced-command `refresh`, not another application release.
|
||||||
|
|||||||
@@ -1,5 +1,17 @@
|
|||||||
# 生产到 ARM 备份传输中断设计
|
# 生产到 ARM 备份传输中断设计
|
||||||
|
|
||||||
|
## 2026-10-05 手动快照切换方案（本地适配已完成，远端切换待批准）
|
||||||
|
|
||||||
|
用户已选定仅手动快照，本节替代下方历史修复路径。用户 2026-10-05 批准后本地适配已实施并验证（结果见 implement.md）；远端切换尚未执行，需另行批准。
|
||||||
|
|
||||||
|
1. 发布适配：`scripts/refresh-deployment-contract.sh` 的 `verify_timers` 当前强制要求三个备份 timer enabled/active；`deploy-existing.sh`、生产身份引导器和聚合检查仍从 daily service 读取部署身份。已增加显式、受控的手动快照模式（根属 `/etc/hyb-oa-backup-mode` 内容为 `manual_snapshot`），校验备份 timer disabled/inactive 且备份 service 已退出（inactive/failed），`probe` 部署前与 `refresh` 部署后均检查，保留身份、构建、数据目录及非备份运维 timer 的验证。不得删除单位、全局跳过 timer 检查或靠错误吞并放行。
|
||||||
|
2. 本地验证：扩展既有发布合同测试，覆盖手动模式三个备份 timer 停用可通过、意外启用失败、非备份 timer 失败仍阻断及旧模式回归。受控入口的模式配置与候选版本一并绑定，实际执行前完成可用的独立只读复核；当前 inline 限制下标记待独立复核，不虚报。
|
||||||
|
3. 远端切换范围：生产 `hyb-oa-automatic-backup-{daily,watchdog,drill-reminder}.timer` 与 ARM `hyb-oa-offhost-deadman.timer`。先核实际单位及既有运行状态；先禁用定时器阻止新任务启动，再有界等待在途 service 退出（上限预先写定），超时停止切换并报告、不强杀；最终核验含对应 service inactive/failed。ARM dead-man 由用户 sudo 停用并单独核验，生产侧 `refresh` 成功不代替 ARM 核验。源码默认名称须与现场一致，不猜测替代单位。
|
||||||
|
4. 保留 unit 文件及其环境、历史归档/partial、状态、凭据和共享告警工具，不调用带删除行为的 installer uninstall。公网健康探针与 `hyb-oa-warch3-ops.timer` 不在备份停用范围，hyinda-20c 未安装本项目备份任务，无迁移或清理动作。
|
||||||
|
5. 顺序为先完成兼容适配、再获批部署适配并停用定时器、最后只读核验与记录。执行前保留旧模式/单位启用状态作为回退依据；出现异常停止，恢复旧模式和定时器需在本次获批回滚范围内执行。不得出现脚本仍强制 active 而定时器已关闭的未处理状态。
|
||||||
|
|
||||||
|
本方案不购买存储、不创建或删除快照、不删备份、不恢复业务数据。远端具体配置、版本和回滚清单须在实施后形成可审阅结果再申请执行批准；当前选择不等于远端切换已完成。
|
||||||
|
|
||||||
## 设计结论
|
## 设计结论
|
||||||
|
|
||||||
采用“先只读判因、再最小变更、最后两个自然周期验证”的事件处置路径。传输修复和容量处置属于同一个 P1 operations 任务，但每项远端写操作维持独立审批；恢复演练作为下游消费者保持阻塞，不参与本任务的根因修复。
|
采用“先只读判因、再最小变更、最后两个自然周期验证”的事件处置路径。传输修复和容量处置属于同一个 P1 operations 任务，但每项远端写操作维持独立审批；恢复演练作为下游消费者保持阻塞，不参与本任务的根因修复。
|
||||||
|
|||||||
@@ -1,5 +1,57 @@
|
|||||||
# 生产到 ARM 备份传输中断实施计划
|
# 生产到 ARM 备份传输中断实施计划
|
||||||
|
|
||||||
|
## 2026-10-05 最新决定与本地落实：仅手动快照
|
||||||
|
|
||||||
|
用户明确选择仅手动快照，不用其他厂商对象存储，也不再选 ARM/hyinda-20c 承担备份。此前对象存储选型、ARM 修复或替换、两个自然周期验收不再作为当前推进目标；原始结果保留，不勾成通过、不自动归档。
|
||||||
|
|
||||||
|
- [x] 复用本 task，更新 PRD 当前目标、design 切换方案及部署规则；不再询问备份产品选择。
|
||||||
|
- [x] 本地确认停用依赖：`scripts/refresh-deployment-contract.sh:verify_timers` 要求三个自动备份 timer 启用且运行；生产身份仍取自 daily service。不能直接卸载或停 timer 后宣称发布兼容。
|
||||||
|
- [x] 本地 task validate、Trellis governance、process baseline、敏感扫描及 diff 检查通过；docs registry 无错误，保留 22 个未登记文档的既有警告，本轮未改动 docs 或处理这些文件。
|
||||||
|
- [x] 用户 2026-10-05 批准适配后本地实施并验证：
|
||||||
|
- `scripts/refresh-deployment-contract.sh`：新增根属普通文件 `/etc/hyb-oa-backup-mode` 显式模式。文件不存在 = 原自动模式（行为不变）；内容恰为 `manual_snapshot` 时要求三个自动备份 timer `disabled:inactive`（否则 exit 45 `backup_timer_not_disabled`），`hyb-oa-warch3-ops.timer` 仍须 `enabled:active`（43）；其他内容/符号链接/非 root 属主 → exit 44 `backup_mode_invalid`，`probe` 同样校验。不删除、不 mask 备份 unit，部署身份仍取自 daily service。
|
||||||
|
- `deploy/trialrun/deployment-contract-lib.sh` 增加 44/45 分类；`validate-env-contract.js` 增加必需片段；部署规约 Validation Matrix 增加对应条目。
|
||||||
|
- 验证：`run-deployment-contract-refresh-contract.sh` PASS（新增自动模式下停用失败、手动模式通过/全部运行/部分运行/ops 停用/内容非法/多行/符号链接/移除模式文件回归共 10 组断言）；两个源码变异（手动模式接受运行中 timer、跳过内容校验）均使测试变红；`validate-env-contract.js` exit 0；`deployment-safety-contract` 39/39、`deployment-db-lifecycle` 52/52。
|
||||||
|
- [x] 复核补充（同日）：①`probe` 也执行模式与 timer/service 一致性检查，部署前即拦截，`refresh` 部署后复检；②手动模式还要求三个备份 `.service` 为 `inactive`/`failed`，在途运行 → 45。新增 7 条断言（probe 拦截运行中/部分运行、自动模式 probe 拦截停用、服务 active/activating 拒绝、failed 放行）；`probe` 去掉 timer 检查、去掉 service 检查两个变异均使测试变红；合同测试、env 合同、部署安全 + DB 生命周期 91/91 通过。注意：自动模式下 timer 异常现在部署前就失败（原先只在部署后失败），属有意收紧。
|
||||||
|
- [ ] 远端切换（另批；生产需 root 安装权限，ARM 需用户 sudo）。顺序：
|
||||||
|
1. 用 `install-deployment-contract-refresh.sh install` 安装新 helper；无模式文件时仍为自动模式，forced-command `probe` 须 PASS。
|
||||||
|
2. 先 `systemctl disable --now` 三个生产备份 **timer**（只停定时器，阻止新任务启动，不动 service）。
|
||||||
|
3. 有界等待三个备份 service 变为 `inactive`/`failed`（上限预先写定，例如 90 分钟）；超时则停止切换、保持现状并报告，**不强杀**。
|
||||||
|
4. 写入 `/etc/hyb-oa-backup-mode`（root:root 0644，内容 `manual_snapshot`），forced-command `refresh` 返回 PASS。
|
||||||
|
5. 由用户 sudo 在 ARM 上 `disable --now hyb-oa-offhost-deadman.timer` 并**单独核验** timer `disabled:inactive`、service 非 active；生产侧 `refresh` 成功不能代替 ARM 核验。未停 dead-man 时，生产停传后 ARM 会发一次 `archive_stale`。
|
||||||
|
- 回滚：删除模式文件，`enable --now` 三个 timer 与 dead-man，再跑 `refresh`。历史备份、unit 文件、公网健康探针均不动。
|
||||||
|
- [ ] 完成受限只读独立复核；当前 inline 模式下仅主会话检查，独立复核待完成。
|
||||||
|
- [ ] 形成固定版本、模式配置、四个备份 timer 及回滚清单，另获生产/ARM 配置变更批准后执行与核验。
|
||||||
|
|
||||||
|
运行状态：本轮未连接生产或 ARM，自动备份/告警可能仍按旧配置运行；未停用定时器、未删除历史备份、未创建快照，未提交或归档。当前选择不自动停用独立公网健康监控。手动快照执行时记录目标确认、覆盖确认、创建时间及成功状态，不记录真实资源标识。
|
||||||
|
|
||||||
|
## 2026-10-05 告警频繁的只读核验（用户当轮授权，ARM 普通账号）
|
||||||
|
|
||||||
|
背景：用户反馈群内仍频繁出现“内部 OA 运维恢复通知”及“OA 公网健康告警/恢复”。授权范围为生产与 ARM 只读核验；生产主机读取被本机工具权限策略拦截，未执行，生产侧数据全部为 NOT_RUN。
|
||||||
|
|
||||||
|
ARM 侧结果（13:30–13:45 CST，只记聚合值，不记地址/路径/日志原文）：
|
||||||
|
|
||||||
|
- dead-man：timer active/enabled，10-05 06:30 service success/0。说明当日 06:30 时 ARM 最新归档未超 26h 阈值；归档完整性未复核（目录及 sudo 仍不可用）。
|
||||||
|
- 公网健康探针：timer active/enabled，13:30 那次 service 以 exit-code/1 失败，耗时约 11s（与 10s 超时一致）；具体类别在 root-only 状态/journal，普通账号读不到（journal 返回“无条目”）。
|
||||||
|
- ARM → OA 公网入口：ping 三轮丢包 67% / 74% / 52%，RTT 约 143ms；HTTPS `/ready` 40 次中 11 次超时（10s）、13 次 TCP 建连 >1s、平均 3.35s。
|
||||||
|
- 对照：ARM → 其他三个国内目标 ping 0% 丢包；ARM 本机 DNS 解析约 1ms（已缓存），当前不是 DNS 阶段问题。本机开发环境经代理（假 IP 段）访问，40/40 成功，但路径不同，不能证明生产入口对普通用户正常。
|
||||||
|
|
||||||
|
判断（假设，未结案）：当前公网健康告警的主要成因是 ARM 到生产入口这条线路持续丢包导致请求超时，不是 OA 应用不可用。同一条线路的丢包也可能是历史 rsync 传输中断（partial）的成因，需生产侧传输记录印证。丢包发生在 ARM 运营商一侧还是生产入口一侧尚未区分；若在生产入口一侧，真实用户也可能受影响。
|
||||||
|
|
||||||
|
未完成 / 下一步：
|
||||||
|
- 生产侧（备份/watchdog 服务结果、告警状态、容量、近 7 天告警类别与部署时间对照）：需用户放开本机对生产主机只读命令的权限，或由用户手动执行。
|
||||||
|
- ARM root-only 公网健康状态文件和 journal 中的类别统计：需用户以 sudo 手动执行固定只读命令。
|
||||||
|
- 区分线路问题在哪一侧：需要第三个直连观测点（不经代理）对生产入口测丢包。
|
||||||
|
|
||||||
|
### 2026-10-05 候选替代节点 hyinda-20c 只读摸底（用户当轮授权）
|
||||||
|
|
||||||
|
- 形态：Windows 主机上的 WSL2（Ubuntu 22.04，x86_64，20 核，15 GiB 内存），经动态域名 + 非标端口映射访问；普通账号 sudo 需密码；无 Docker。
|
||||||
|
- 可用性：开机约 9 天；近两个月重启记录集中在 09-26（多次）与 08-22（多次），生命周期受 Windows 开关机/更新/WSL 实例回收影响。
|
||||||
|
- 存储：WSL ext4 显示 1 TiB 虚拟盘，实际受宿主 Windows 盘余量约束（宿主两盘分别约 167G / 88G 可用）；未见 Linux 层磁盘加密，宿主 BitLocker 未知。
|
||||||
|
- 网络：节点流量（含指定公共 DNS 的解析）均被本机透明代理接管为假 IP 段，到 OA 入口 HTTPS 70/70 成功、平均约 0.13s，但不是直连测量，不能作为线路证据。
|
||||||
|
- 容量需求参照：既有记录单批次聚合约 11 MB 量级，容量和带宽对任何候选节点都不是约束；选型关键是常在线、独立、可控访问与恢复能力。
|
||||||
|
- 用户决定（2026-10-05）：不购买轻量“自动备份”（备份点按 60GB 盘容量预付至实例到期，约 983 元）；采用**每次部署前手动创建轻量免费快照**（配额 2，满则删最旧），已写入 `.trellis/spec/hyb-oa/deployment-trialrun.md` Rules。快照随实例销毁而删除，不替代实例外副本；实例外副本（另一厂商对象存储）与 ARM 链路退役待用户另行决定。用户确认控制台中名为“Windows Server”的轻量实例即 OA 生产（实际为 Linux，仅实例名称历史遗留），快照对象即此实例。
|
||||||
|
- 结论：不建议 hyinda-20c 作为主要异机备份节点（WSL 生命周期、代理接管网络、虚拟盘依赖宿主、无加密证据）；可作为额外副本候选。主要节点建议改评国内云的独立厂商对象存储或小型云主机，待用户决策。
|
||||||
|
|
||||||
|
|
||||||
## 2026-09-17 结案缺口与最小补证方案（本地整理，未执行远端观察）
|
## 2026-09-17 结案缺口与最小补证方案（本地整理，未执行远端观察）
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,18 @@
|
|||||||
# 生产到 ARM 备份传输中断
|
# 生产到 ARM 备份传输中断
|
||||||
|
|
||||||
|
## 2026-10-05 当前决策：仅使用手动快照
|
||||||
|
|
||||||
|
用户明确决定不使用其他厂商对象存储、ARM 或 hyinda-20c 备份，只使用已确认 OA 生产轻量实例的手动快照。此决策取代本任务继续修复异机链路、选型迁移及两个自然备份周期验收的目标；以下原始要求和 AC 保留为历史，未完成项不追认为通过。关联 `08-23-isolated-backup-restore-drill` 的历史部分恢复结果不变，本决策不授权继续真实恢复。
|
||||||
|
|
||||||
|
- 当前备份方式为部署前手动快照；确认目标、磁盘覆盖和创建成功后再部署，不新增自动快照脚本或其他副本方案。恢复范围受现存成功快照的时间点与覆盖磁盘限制，不承诺每日恢复点或实例外副本。
|
||||||
|
- 选择备份方式与实际停用分开记录。本轮落实本地合同；发布脚本适配、生产/ARM 定时任务停用方案见 design/implement，非微小代码实施和远端配置变更待方案批准。
|
||||||
|
- 历史备份、快照、凭据和 unit 文件不因本决策自动删除；公网健康监控是独立功能，不把“停用备份”解释为关闭全部监控。
|
||||||
|
|
||||||
|
### 当前验收
|
||||||
|
|
||||||
|
- [x] S1：当前选择已同步到任务及部署规约，本地文档/治理检查通过；旧未验项与当前决策明确区分。文档审计保留既有未登记文件警告，详见 implement。
|
||||||
|
- [ ] S2：获批的发布适配与备份任务停用完成，停用状态及发布兼容性有据；此前不得称“已经只运行手动快照方案”。
|
||||||
|
|
||||||
## Goal
|
## Goal
|
||||||
|
|
||||||
恢复腾讯云生产到 ARM 的自动异机备份链，消除持续扩大的异机恢复点缺口，同时把生产备份落盘容量恢复到可持续范围，并证明修复不是一次偶然成功。
|
恢复腾讯云生产到 ARM 的自动异机备份链，消除持续扩大的异机恢复点缺口，同时把生产备份落盘容量恢复到可持续范围，并证明修复不是一次偶然成功。
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
"assignee": "shunli",
|
"assignee": "shunli",
|
||||||
"createdAt": "2026-08-28",
|
"createdAt": "2026-08-28",
|
||||||
"completedAt": null,
|
"completedAt": null,
|
||||||
"branch": null,
|
"branch": "main",
|
||||||
"base_branch": "main",
|
"base_branch": "main",
|
||||||
"worktree_path": null,
|
"worktree_path": null,
|
||||||
"commit": null,
|
"commit": null,
|
||||||
|
|||||||
@@ -81,6 +81,8 @@ deployment_contract_refresh_category() {
|
|||||||
255) printf 'root_identity_unavailable' ;;
|
255) printf 'root_identity_unavailable' ;;
|
||||||
126|127) printf 'refresh_program_unavailable' ;;
|
126|127) printf 'refresh_program_unavailable' ;;
|
||||||
43) printf 'timer_not_active' ;;
|
43) printf 'timer_not_active' ;;
|
||||||
|
44) printf 'backup_mode_invalid' ;;
|
||||||
|
45) printf 'backup_timer_not_disabled' ;;
|
||||||
*) printf 'unit_build_not_refreshed' ;;
|
*) printf 'unit_build_not_refreshed' ;;
|
||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -751,6 +751,9 @@ for (const requiredFragment of [
|
|||||||
'refresh_program_integrity_invalid',
|
'refresh_program_integrity_invalid',
|
||||||
'unit_build_not_refreshed',
|
'unit_build_not_refreshed',
|
||||||
'timer_not_active',
|
'timer_not_active',
|
||||||
|
'backup_mode_invalid',
|
||||||
|
'backup_timer_not_disabled',
|
||||||
|
'manual_snapshot',
|
||||||
'systemctl daemon-reload',
|
'systemctl daemon-reload',
|
||||||
]) {
|
]) {
|
||||||
if (!deploymentRefreshHelperContent.includes(requiredFragment)) {
|
if (!deploymentRefreshHelperContent.includes(requiredFragment)) {
|
||||||
|
|||||||
@@ -133,9 +133,42 @@ case "$action" in
|
|||||||
esac
|
esac
|
||||||
;;
|
;;
|
||||||
daemon-reload) exit 0 ;;
|
daemon-reload) exit 0 ;;
|
||||||
is-enabled) echo enabled ;;
|
is-enabled)
|
||||||
|
unit="${2:-}"
|
||||||
|
if [[ "$unit" == hyb-oa-automatic-backup-* ]]; then
|
||||||
|
case "${FIXTURE_BACKUP_TIMERS:-running}" in
|
||||||
|
disabled) echo disabled; exit 1 ;;
|
||||||
|
mixed)
|
||||||
|
if [[ "$unit" == *watchdog* ]]; then echo enabled; else echo disabled; exit 1; fi
|
||||||
|
;;
|
||||||
|
*) echo enabled ;;
|
||||||
|
esac
|
||||||
|
else
|
||||||
|
echo enabled
|
||||||
|
fi
|
||||||
|
;;
|
||||||
is-active)
|
is-active)
|
||||||
|
unit="${2:-}"
|
||||||
|
if [[ "$unit" == *.service ]]; then
|
||||||
|
if [[ "$unit" == hyb-oa-automatic-backup-daily.service \
|
||||||
|
&& -n "${FIXTURE_DAILY_SERVICE_STATE:-}" ]]; then
|
||||||
|
echo "$FIXTURE_DAILY_SERVICE_STATE"
|
||||||
|
[[ "$FIXTURE_DAILY_SERVICE_STATE" == "active" ]] || exit 3
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo inactive; exit 3
|
||||||
|
fi
|
||||||
if [[ "${FIXTURE_TIMER_INACTIVE:-0}" == "1" ]]; then echo inactive; exit 3; fi
|
if [[ "${FIXTURE_TIMER_INACTIVE:-0}" == "1" ]]; then echo inactive; exit 3; fi
|
||||||
|
if [[ "$unit" == hyb-oa-automatic-backup-* ]]; then
|
||||||
|
case "${FIXTURE_BACKUP_TIMERS:-running}" in
|
||||||
|
disabled) echo inactive; exit 3 ;;
|
||||||
|
mixed)
|
||||||
|
if [[ "$unit" == *watchdog* ]]; then echo active; else echo inactive; exit 3; fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
elif [[ "${FIXTURE_OPS_TIMER_INACTIVE:-0}" == "1" ]]; then
|
||||||
|
echo inactive; exit 3
|
||||||
|
fi
|
||||||
echo active
|
echo active
|
||||||
;;
|
;;
|
||||||
*) exit 1 ;;
|
*) exit 1 ;;
|
||||||
@@ -162,6 +195,57 @@ unsupported_status=$?
|
|||||||
set -e
|
set -e
|
||||||
[[ "$unit_status" -eq 42 && "$timer_status" -eq 43 && "$unsupported_status" -eq 64 ]]
|
[[ "$unit_status" -eq 42 && "$timer_status" -eq 43 && "$unsupported_status" -eq 64 ]]
|
||||||
|
|
||||||
|
run_refresh_status() {
|
||||||
|
local command="$1"
|
||||||
|
shift
|
||||||
|
set +e
|
||||||
|
env HYB_CONTRACT_REFRESH_TEST_ROOT="$FIXTURE_ROOT" "$@" \
|
||||||
|
SSH_ORIGINAL_COMMAND="$command" bash "$HELPER" >/dev/null 2>&1
|
||||||
|
local status=$?
|
||||||
|
set -e
|
||||||
|
printf '%s' "$status"
|
||||||
|
}
|
||||||
|
|
||||||
|
mode_file="$FIXTURE_ROOT/etc/hyb-oa-backup-mode"
|
||||||
|
# Automatic mode (no mode file): disabled backup timers stay fail-closed,
|
||||||
|
# both before (probe) and after (refresh) a release.
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled)" -eq 43 ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_BACKUP_TIMERS=disabled)" -eq 43 ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_TIMER_INACTIVE=1)" -eq 43 ]]
|
||||||
|
|
||||||
|
printf 'manual_snapshot\n' >"$mode_file"
|
||||||
|
manual_output="$(HYB_CONTRACT_REFRESH_TEST_ROOT="$FIXTURE_ROOT" FIXTURE_BACKUP_TIMERS=disabled \
|
||||||
|
SSH_ORIGINAL_COMMAND=refresh bash "$HELPER")"
|
||||||
|
[[ "$manual_output" == '[deployment-contract-refresh][PASS] refreshed' ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_BACKUP_TIMERS=disabled)" -eq 0 ]]
|
||||||
|
# Manual mode must not silently accept running or partially running backup timers,
|
||||||
|
# and the pre-deploy probe must reject the mismatch before any release write.
|
||||||
|
[[ "$(run_refresh_status refresh)" -eq 45 ]]
|
||||||
|
[[ "$(run_refresh_status probe)" -eq 45 ]]
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=mixed)" -eq 45 ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_BACKUP_TIMERS=mixed)" -eq 45 ]]
|
||||||
|
# Disabled timers with an in-flight backup run are not a finished switch.
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled FIXTURE_DAILY_SERVICE_STATE=active)" -eq 45 ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_BACKUP_TIMERS=disabled FIXTURE_DAILY_SERVICE_STATE=activating)" -eq 45 ]]
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled FIXTURE_DAILY_SERVICE_STATE=failed)" -eq 0 ]]
|
||||||
|
# Non-backup ops timer is still required in manual mode.
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled FIXTURE_OPS_TIMER_INACTIVE=1)" -eq 43 ]]
|
||||||
|
|
||||||
|
printf 'manual\n' >"$mode_file"
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled)" -eq 44 ]]
|
||||||
|
[[ "$(run_refresh_status probe FIXTURE_BACKUP_TIMERS=disabled)" -eq 44 ]]
|
||||||
|
printf 'manual_snapshot\nautomatic\n' >"$mode_file"
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled)" -eq 44 ]]
|
||||||
|
rm -f -- "$mode_file"
|
||||||
|
printf 'manual_snapshot\n' >"$FIXTURE_ROOT/backup-mode-target"
|
||||||
|
ln -s "$FIXTURE_ROOT/backup-mode-target" "$mode_file"
|
||||||
|
[[ "$(run_refresh_status refresh FIXTURE_BACKUP_TIMERS=disabled)" -eq 44 ]]
|
||||||
|
rm -f -- "$mode_file"
|
||||||
|
[[ "$(run_refresh_status refresh)" -eq 0 ]]
|
||||||
|
|
||||||
|
[[ "$(deployment_contract_refresh_category 44 '')" == "backup_mode_invalid" ]]
|
||||||
|
[[ "$(deployment_contract_refresh_category 45 '')" == "backup_timer_not_disabled" ]]
|
||||||
|
|
||||||
public_key_file="$FIXTURE_ROOT/refresh.pub"
|
public_key_file="$FIXTURE_ROOT/refresh.pub"
|
||||||
key_blob='AAAAC3NzaC1lZDI1NTE5AAAAIFixturerefreshkeymaterial000000000000'
|
key_blob='AAAAC3NzaC1lZDI1NTE5AAAAIFixturerefreshkeymaterial000000000000'
|
||||||
printf 'ssh-ed25519 %s fixture-refresh\n' "$key_blob" >"$public_key_file"
|
printf 'ssh-ed25519 %s fixture-refresh\n' "$key_blob" >"$public_key_file"
|
||||||
|
|||||||
@@ -12,10 +12,17 @@ SERVICES=(
|
|||||||
hyb-oa-automatic-backup-drill-reminder.service
|
hyb-oa-automatic-backup-drill-reminder.service
|
||||||
hyb-oa-warch3-ops.service
|
hyb-oa-warch3-ops.service
|
||||||
)
|
)
|
||||||
TIMERS=(
|
BACKUP_TIMERS=(
|
||||||
hyb-oa-automatic-backup-daily.timer
|
hyb-oa-automatic-backup-daily.timer
|
||||||
hyb-oa-automatic-backup-watchdog.timer
|
hyb-oa-automatic-backup-watchdog.timer
|
||||||
hyb-oa-automatic-backup-drill-reminder.timer
|
hyb-oa-automatic-backup-drill-reminder.timer
|
||||||
|
)
|
||||||
|
BACKUP_SERVICES=(
|
||||||
|
hyb-oa-automatic-backup-daily.service
|
||||||
|
hyb-oa-automatic-backup-watchdog.service
|
||||||
|
hyb-oa-automatic-backup-drill-reminder.service
|
||||||
|
)
|
||||||
|
OPS_TIMERS=(
|
||||||
hyb-oa-warch3-ops.timer
|
hyb-oa-warch3-ops.timer
|
||||||
)
|
)
|
||||||
CONTRACT_SERVICE="hyb-oa-automatic-backup-daily.service"
|
CONTRACT_SERVICE="hyb-oa-automatic-backup-daily.service"
|
||||||
@@ -47,6 +54,11 @@ if [[ "$(id -u)" != "0" ]]; then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
UNIT_ROOT="${ROOT_PREFIX}/etc/systemd/system"
|
UNIT_ROOT="${ROOT_PREFIX}/etc/systemd/system"
|
||||||
|
# Absent file = automatic backup mode (all backup timers must run). The only
|
||||||
|
# accepted content is "manual_snapshot": backup timers must then be disabled
|
||||||
|
# and inactive, because backups are manual Lighthouse snapshots before deploys.
|
||||||
|
BACKUP_MODE_FILE="${ROOT_PREFIX}/etc/hyb-oa-backup-mode"
|
||||||
|
BACKUP_MODE=""
|
||||||
LOCK_FILE="${ROOT_PREFIX}/run/lock/hyb-oa-deployment-contract-refresh.lock"
|
LOCK_FILE="${ROOT_PREFIX}/run/lock/hyb-oa-deployment-contract-refresh.lock"
|
||||||
EXPECTED_TARGET="${ROOT_PREFIX}/usr/local/sbin/hyb-oa-deployment-contract-refresh"
|
EXPECTED_TARGET="${ROOT_PREFIX}/usr/local/sbin/hyb-oa-deployment-contract-refresh"
|
||||||
|
|
||||||
@@ -160,25 +172,69 @@ verify_builds() {
|
|||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
|
resolve_backup_mode() {
|
||||||
|
local content
|
||||||
|
if [[ ! -e "$BACKUP_MODE_FILE" && ! -L "$BACKUP_MODE_FILE" ]]; then
|
||||||
|
BACKUP_MODE="automatic"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
[[ -f "$BACKUP_MODE_FILE" && ! -L "$BACKUP_MODE_FILE" ]] \
|
||||||
|
|| fail 44 backup_mode_invalid
|
||||||
|
if [[ "$TEST_MODE" == "0" ]]; then
|
||||||
|
[[ "$(stat -c %u:%g "$BACKUP_MODE_FILE")" == "0:0" ]] \
|
||||||
|
&& [[ "$(stat -c %a "$BACKUP_MODE_FILE")" =~ ^(644|600|444|400)$ ]] \
|
||||||
|
|| fail 44 backup_mode_invalid
|
||||||
|
fi
|
||||||
|
content="$(<"$BACKUP_MODE_FILE")"
|
||||||
|
[[ "$content" == "manual_snapshot" ]] || fail 44 backup_mode_invalid
|
||||||
|
BACKUP_MODE="manual_snapshot"
|
||||||
|
}
|
||||||
|
|
||||||
|
timer_state() {
|
||||||
|
local timer="$1"
|
||||||
|
printf '%s:%s' \
|
||||||
|
"$(systemctl is-enabled "$timer" 2>/dev/null || true)" \
|
||||||
|
"$(systemctl is-active "$timer" 2>/dev/null || true)"
|
||||||
|
}
|
||||||
|
|
||||||
verify_timers() {
|
verify_timers() {
|
||||||
local timer
|
local timer
|
||||||
for timer in "${TIMERS[@]}"; do
|
for timer in "${OPS_TIMERS[@]}"; do
|
||||||
[[ "$(systemctl is-enabled "$timer")" == "enabled" \
|
[[ "$(timer_state "$timer")" == "enabled:active" ]] || fail 43 timer_not_active
|
||||||
&& "$(systemctl is-active "$timer")" == "active" ]] \
|
done
|
||||||
|| fail 43 timer_not_active
|
for timer in "${BACKUP_TIMERS[@]}"; do
|
||||||
|
if [[ "$BACKUP_MODE" == "manual_snapshot" ]]; then
|
||||||
|
[[ "$(timer_state "$timer")" == "disabled:inactive" ]] \
|
||||||
|
|| fail 45 backup_timer_not_disabled
|
||||||
|
else
|
||||||
|
[[ "$(timer_state "$timer")" == "enabled:active" ]] || fail 43 timer_not_active
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
[[ "$BACKUP_MODE" == "manual_snapshot" ]] || return 0
|
||||||
|
# A disabled timer does not stop an in-flight run; the switch is complete
|
||||||
|
# only after every backup service has exited (a failed last run still counts).
|
||||||
|
local service state
|
||||||
|
for service in "${BACKUP_SERVICES[@]}"; do
|
||||||
|
state="$(systemctl is-active "$service" 2>/dev/null || true)"
|
||||||
|
[[ "$state" == "inactive" || "$state" == "failed" ]] \
|
||||||
|
|| fail 45 backup_timer_not_disabled
|
||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
probe() {
|
probe() {
|
||||||
validate_self
|
validate_self
|
||||||
validate_fixed_units
|
validate_fixed_units
|
||||||
|
resolve_backup_mode
|
||||||
resolve_active_build
|
resolve_active_build
|
||||||
|
# Fail before any release write when timers disagree with the backup mode.
|
||||||
|
verify_timers
|
||||||
echo "[deployment-contract-refresh][PASS] probe"
|
echo "[deployment-contract-refresh][PASS] probe"
|
||||||
}
|
}
|
||||||
|
|
||||||
refresh() {
|
refresh() {
|
||||||
validate_self
|
validate_self
|
||||||
validate_fixed_units
|
validate_fixed_units
|
||||||
|
resolve_backup_mode
|
||||||
resolve_active_build
|
resolve_active_build
|
||||||
mkdir -p "$(dirname "$LOCK_FILE")"
|
mkdir -p "$(dirname "$LOCK_FILE")"
|
||||||
exec 9>"$LOCK_FILE"
|
exec 9>"$LOCK_FILE"
|
||||||
|
|||||||
在新工单中引用
屏蔽一个用户