加固生产运维定时器绑定(07-26-internal-ops-recovery-baseline)
ci-e2e / e2e (push) Waiting to run
ci-minimal / check-minimal (push) Waiting to run

这个提交包含在:
2026-07-27 07:59:48 +08:00
父节点 3c4086555a
当前提交 325c592aba
共修改 7 个文件,包含 147 行新增和 5 行删除
@@ -619,6 +619,9 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
- Application storage mount: `${STORAGE_HOST_DIR:-./data/storage}`.
- Formal deploy inputs: absolute `POSTGRES_DATA_DIR`, absolute
`STORAGE_HOST_DIR`, `DEPLOY_DATA_MODE=existing|initialize`.
- Timer install inputs: `COMPOSE_PROJECT_NAME`, absolute
`POSTGRES_DATA_DIR`, absolute `STORAGE_HOST_DIR`, full 40–64 hex
`BUILD_SHA` and its exact 12-hex `VITE_BUILD_SHA` prefix.
- DB has no host `ports`; server remains loopback-only.
### 3. Contracts
@@ -639,6 +642,16 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
only that mode may create data directories.
- Remote readiness may use host curl, but build-SHA JSON validation runs with
Node inside the server container; the Docker host does not need Node.
- The timer installer must validate the same non-root, non-overlapping absolute
data-path shape as formal deploy, require an existing `PG_VERSION` and
storage directory, and write both paths plus both build identifiers into the
systemd service `Environment=` entries. Project name alone is insufficient:
a later timer invocation would otherwise interpolate Compose defaults and
operate against a non-active data pair.
- Timer installation may enable/start the timer unit, but verification must not
manually start the daily service because that path can send real alerts.
Verify the unit environment against active container mounts, then run
Compose `config --quiet` and `exec -T server true` with the same values.
- For an existing cutover, stop the old PostgreSQL completely, copy PGDATA and
storage into pre-verified empty absolute targets while preserving ownership,
permissions, links and extended attributes, and retain the old directories
@@ -651,6 +664,13 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
- Any fixed `container_name` appears -> environment contract fails.
- Required data mount loses its parameter -> environment contract fails.
- Deploy/timer omits `-p` -> environment contract fails.
- Timer install omits either data path/build identifier, receives nested paths,
a PGDATA without `PG_VERSION`, a missing storage directory, an invalid full
SHA or a 12-character value that is not its prefix -> fail before writing the
systemd unit.
- A path-segment regex can match an empty string at the leading `/` -> contract
failure; use explicit root, trailing slash, `//`, `/./` and `/../` checks so
valid absolute paths remain installable.
- Formal deploy omits paths/mode, uses a relative/broad/overlapping path, or an
`existing` PGDATA lacks `PG_VERSION` -> fail before DB start.
- New project resolves the same PGDATA while the old DB runs -> stop before
@@ -664,17 +684,26 @@ curl -fsS "http://127.0.0.1:${SERVER_HOST_PORT}/ready"
- The isolated backup/restore drill mutates Compose with a DB port and fixed
name and proves the contract rejects it.
- `bash -n` covers deploy, timer, backup and restore scripts.
- Timer contract assertions cover all four explicit inputs, path isolation,
PGDATA marker, SHA/prefix validation and the four systemd `Environment=`
entries. On the deployment host, assert timer enabled/active, daily service
inactive, exact active-environment equality and non-mutating Compose resolve.
### 6. Good/Base/Bad Cases
- Good: one approved project resolves one stopped-and-migrated PGDATA and the
matching active storage path; deploy, timer and backup all use that identity.
- Good: the timer unit contains the current project, active mount sources and
deployed full/short SHA pair; Compose config and a non-mutating server exec
resolve under those exact values while the daily service stays inactive.
- Base: direct local Compose uses the documented relative defaults under one
project; those defaults are never accepted by formal `deploy.sh`.
- Bad: start a renamed project before the old DB stops, or let it create a new
empty directory and accept a green health check as proof of data continuity.
- Bad: move the only PGDATA into the new path, migrate it in place, then assume
the old container name alone is a rollback point.
- Bad: reinstall the timer with only `COMPOSE_PROJECT_NAME`, allowing systemd
Compose interpolation to fall back to repository-relative data directories.
### 7. Wrong vs Correct
@@ -697,6 +726,22 @@ DB_TOOL_MODE=docker DB_CONTAINER="${DB_CONTAINERS[0]}" \
STORAGE_DIR_HOST=<verified-active-storage> ./scripts/backup-all.sh
```
For timer installation, keep the active deployment identity explicit:
```bash
# Wrong: project identity alone still permits relative data-path fallback.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
bash deploy/trialrun/install-warch3-ops-timer.sh install
# Correct: use the verified active mounts and deployed SHA pair.
COMPOSE_PROJECT_NAME="$COMPOSE_PROJECT_NAME" \
POSTGRES_DATA_DIR=<verified-active-pgdata> \
STORAGE_HOST_DIR=<verified-active-storage> \
BUILD_SHA=<verified-full-build-sha> \
VITE_BUILD_SHA=<verified-12-char-prefix> \
bash deploy/trialrun/install-warch3-ops-timer.sh install
```
## Scenario: Paired Backup And Shadow-Only Restore
### 1. Scope / Trigger
@@ -131,4 +131,4 @@ GET /auth/wecom/login?mode=onboard
2. 本地先修复 Compose 可配置回环端口及部署脚本动态端口探测,通过全部门禁并提交、发布批准提交。
3. 停机后把活动 PGDATA 和附件复制到新的显式绝对目录,复制前目标必须为空;保留旧目录原位且不再挂载写入,直到观察期结束。复制仅在旧 PostgreSQL 完全停止后执行,并保留所有权、权限、链接和扩展属性。
4. 用新 Compose 重建 db/init/server,应用已提交 migrations;旧容器和旧目录在验收前保留,不清理。
5. 公网构建标识、数据计数、角色、企微入口和关键只读检查通过后更新运维 timer;失败则停止新容器并恢复旧目录/容器和原 Caddy 目标。
5. 公网构建标识、数据计数和关键部署只读检查通过后更新运维 timer;角色与企微入口仍按各自真实写入门单独验收,不阻塞部署运维基线收口。部署检查失败则停止新容器并恢复旧目录/容器和原 Caddy 目标。
@@ -20,13 +20,13 @@
- [ ] 复用既有 HR 同步与角色语义写入 `employee + finance`、`employee + ceo`,复核会话失效、审计和 maker-checker,不部署、不迁移。
- [x] 增加默认关闭的一次性企微敏感资料授权入口,保持普通登录静默;实现 `user_ticket -> getuserdetail`、active/字段/唯一性门禁和仅 `employee` 自助开通。
- [x] 补齐 OAuth 路由、服务、配置契约和单元测试,证明票据/原始敏感响应不进入日志、Cookie、数据库或审计。
- [ ] 修复部署脚本的现场端口解析,确定唯一部署目录,并在活动 DB/附件挂载、环境摘要和旧镜像回滚点全部确认后才执行获批切换。
- [x] 修复部署脚本的现场端口解析,确定唯一部署目录,并在活动 DB/附件挂载、环境摘要和旧镜像回滚点全部确认后执行获批切换。
- [x] 完成本地 Compose/备份加固:移除固定容器名和 DB 宿主端口,参数化数据目录,deploy/timer 显式 project,备份来源缺失硬失败,远端构建标识改由容器内 Node 核验。
- [x] 增加隐式附件源、隐式 DB 容器、DB 端口与固定容器名回退的负向契约演练;未部署或启用新 Compose。
- [x] 收紧正式 deploy 数据路径:显式绝对路径、`existing|initialize` 模式、既有 PGDATA 标记和禁止 Docker 静默建空目录。
- [x] 让成对 manifest 尊重显式 `BUILD_SHA` 并支持活动镜像 digest;补 Node/Python 等价与非法来源标识负向测试。
- [x] 把旧 Compose project 实测、镜像 digest、58 项迁移与关键表计数、影子逐表/附件计数验收、停机后复制并保留旧目录写入 runbook/checklist/spec。
- [ ] 部署后验证唯一 Compose 来源、构建标识、迁移状态、Caddy 公网 `/ready` 和前端 marker;不因服务可访问而忽略来源漂移。
- [x] 部署后验证唯一 Compose 来源、构建标识、迁移状态、Caddy 公网 `/ready` 和前端 marker;不因服务可访问而忽略来源漂移。
- [ ] 用户在企微应用详情选择手机、邮箱、企业邮箱敏感字段;部署开关开启后,两名目标分别通过一次性链接本人确认授权。
- [ ] 两人自助开通成功后写入获批 `finance` / `ceo` 职责,复核不同身份、maker-checker、会话撤销、审计和付款零写入。
@@ -71,6 +71,15 @@
- 已从活动 DB 容器与活动 server 附件挂载生成正确批次并复制到 ARM 备份机;DB catalog、manifest、源/目标产物 SHA-256、附件 tar 与 0 个普通文件计数均通过,且未伪造旧实例 build SHA。ARM 只承担异机备份与恢复演练职责,不运行 OA 服务。
- 已在 ARM 备份机用不发布端口的一次性 PostgreSQL 16 容器恢复正确批次:58 项成功迁移、12 张关键表计数和 0 个附件普通文件均与冻结基线一致,包含镜像拉取的实测 RTO 为 33 秒。演练后容器、临时附件目录和本轮新增镜像均已删除,未运行 OA server;应用 `/ready` 留待生产部署环境验证。
## 生产部署与 timer 收口证据(2026-07-27)
- 已在获批停机窗口把生产切换到唯一 Compose project;活动 DB 与附件从停机冻结副本进入新的独立目录,旧容器保持停止,旧数据目录原位保留作回滚点。
- 生产已应用 61 项 migration;12 张关键表计数与冻结基线一致,活动附件普通文件数仍为 0。server 仅回环发布,DB 不发布宿主端口。
- 容器内与公网 `/ready`、前端 marker 均返回批准提交的同一 12 位构建标识;runtime smoke、external-deps smoke 和公网 bundle/marker 检查通过。
- 运维 timer 安装器已要求显式绑定 Compose project、活动 PGDATA、活动附件目录、完整后端 SHA 和 12 位前端 SHA。生产 timer 已重新安装并保持 enabled/active,daily service 未被手工触发;unit 环境与活动挂载/构建标识一致,同环境 Compose config 与 server exec 解析通过。
- timer 首次收口时发现从 deploy 复制的旧路径正则会误拒绝所有绝对路径;安装在写 unit 前安全停止。现已与 deploy 的根目录、尾随斜杠、双斜杠及点路径段门禁保持一致,并由环境契约阻止旧正则回退。
- 本轮没有启动真实告警任务,没有修改员工、角色、付款、企微后台或企微开通开关;职责分离与两人本人授权继续保持未完成。
## 回滚点
- 成对入口失败:保留四个底层脚本,不写完整 manifest。
@@ -62,7 +62,7 @@
- [ ] 获批写入后,两名活跃员工分别具有 `employee + finance` 与 `employee + ceo`,企微绑定不同,角色审计存在且未产生付款业务写入。
- [x] 普通企微登录仍生成 `snsapi_base`,一次性开通入口生成 `snsapi_privateinfo`;缺少开关、`user_ticket`、成员授权字段、active 状态或唯一性时均拒绝创建(本地 synthetic 验证)。
- [ ] 一次性授权通过 `getuserdetail` 获取成员同意的手机和邮箱,自动创建或补齐 OA 员工与企微绑定;新建账号角色严格等于 `employee`,日志和审计不含票据或完整敏感响应。
- [ ] 部署目录、Compose、环境文件、运维 timer 和活动容器来源统一;活动 DB/附件连续,远端 migrations 与批准提交一致,公网 `/ready` 和前端 marker 返回同一短构建标识。
- [x] 部署目录、Compose、环境文件、运维 timer 和活动容器来源统一;活动 DB/附件连续,远端 migrations 与批准提交一致,公网 `/ready` 和前端 marker 返回同一短构建标识。
- [x] 本地 Compose、部署、timer 和备份合同完成加固并覆盖负向回退;未把本地通过误报为生产切换完成。
- [x] 部署入口拒绝隐式/相对数据目录及缺少数据模式,成对 manifest 可记录显式 build SHA 或活动镜像 digest,并有 Node/Python 等价测试。
- [ ] 两名目标成员分别完成一次本人授权后,账号唯一、企微绑定不同,再由管理员写入 `employee + finance` 与 `employee + ceo`;maker-checker、会话撤销和角色审计通过。
@@ -111,3 +111,12 @@
- 已使用正确异机批次在不发布端口的一次性 PostgreSQL 16 容器中完成真实备份格式影子恢复;58 项成功迁移、12 张关键表计数与冻结基线逐项一致,附件归档安全检查和解包后的 0 个普通文件也一致,包含镜像拉取的实测 RTO 为 33 秒。
- 恢复演练后,一次性数据库容器、临时附件目录和本轮新增 PostgreSQL 镜像均已删除;ARM 继续只承担备份与恢复演练职责,没有运行 OA server 或开放 OA 端口。
- 当前证据证明 DB+附件异机恢复点可用,但不替代部署环境的应用 `/ready`、构建标识、Caddy 和前端 marker 验收。生产服务未停止,数据库、附件、容器、Caddy、timer、角色和企微均未修改。
## 2026-07-27 获批生产切换与运维收口
- 已完成获批生产切换:新 Compose project 成为唯一活动来源,旧项目容器保持停止;旧数据目录和旧容器均保留,正式活动数据使用停机后复制的新目录。
- 生产共 61 项 migration 成功,12 张关键表计数与冻结基线一致,活动附件普通文件数为 0;server 只绑定回环地址,DB 未发布宿主端口。
- 容器内与公网 `/ready`、前端 marker 均观察到批准提交的同一 12 位构建标识;runtime、外部依赖和公网资源标识检查通过。
- 运维 timer 已重新安装并显式绑定当前 Compose project、活动 DB/附件目录和前后端构建标识;timer 为 enabled/active,daily service 保持 inactive,未手工触发真实告警。使用同一组脱敏环境事实执行 Compose config 与 server exec 均通过。
- timer 安装前的路径与构建门禁先在本地通过;一次旧正则误拒绝绝对路径后,安装在 unit 写入前停止,随后修复并增加防回退契约。全程未输出或写入 Secret、完整宿主路径和环境原文。
- 部署与恢复基线现已完成;真实职责分配、两名成员本人敏感资料授权和企微开通仍是独立未完成门禁,因此整体仍不得宣称付款 maker-checker 或企微自助开通已可用。
+49 -1
查看文件
@@ -5,6 +5,10 @@ REMOTE_DIR="${REMOTE_DIR:-/opt/hyb-oa-trialrun}"
REMOTE_ENV_FILE="${REMOTE_ENV_FILE:-${REMOTE_DIR}/deploy/trialrun/.env}"
COMPOSE_FILE="${COMPOSE_FILE:-deploy/trialrun/docker-compose.yml}"
COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-hyb-oa-trialrun}"
POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"
STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"
BUILD_SHA="${BUILD_SHA:-}"
VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"
SERVICE_NAME="${SERVICE_NAME:-hyb-oa-warch3-ops}"
RUN_AT="${RUN_AT:-07:30}"
MODE="${1:-install}"
@@ -19,6 +23,20 @@ need_root() {
fi
}
validate_data_path() {
local name="$1"
local value="$2"
if [[ -z "$value" || "$value" != /* || ! "$value" =~ ^/[A-Za-z0-9._/-]+$ ]]; then
echo "[ERROR] $name 必须显式提供受限字符集的绝对路径" >&2
exit 1
fi
if [[ "$value" == "/" || "$value" == */ || "$value" == *"//"* \
|| "/$value/" == *"/../"* || "/$value/" == *"/./"* ]]; then
echo "[ERROR] $name 不得是根目录、尾随斜杠或包含空、.、.. 路径段" >&2
exit 1
fi
}
uninstall_timer() {
need_root
systemctl disable --now "${SERVICE_NAME}.timer" >/dev/null 2>&1 || true
@@ -42,6 +60,32 @@ install_timer() {
echo "[ERROR] COMPOSE_PROJECT_NAME 格式无效" >&2
exit 1
fi
validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"
validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"
if [[ "$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR" \
|| "$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR"/* \
|| "$STORAGE_HOST_DIR" == "$POSTGRES_DATA_DIR"/* ]]; then
echo "[ERROR] PostgreSQL 与附件宿主目录必须相互独立" >&2
exit 1
fi
if [[ ! -f "$POSTGRES_DATA_DIR/PG_VERSION" ]]; then
echo "[ERROR] POSTGRES_DATA_DIR 必须是包含 PG_VERSION 的现存 PGDATA" >&2
exit 1
fi
if [[ ! -d "$STORAGE_HOST_DIR" ]]; then
echo "[ERROR] STORAGE_HOST_DIR 必须是现存目录" >&2
exit 1
fi
if [[ ! "$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$ ]]; then
echo "[ERROR] BUILD_SHA 必须是 40-64 位十六进制完整提交标识" >&2
exit 1
fi
BUILD_SHA="${BUILD_SHA,,}"
VITE_BUILD_SHA="${VITE_BUILD_SHA,,}"
if [[ ! "$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$ || "${BUILD_SHA:0:12}" != "$VITE_BUILD_SHA" ]]; then
echo "[ERROR] VITE_BUILD_SHA 必须是 BUILD_SHA 的 12 位十六进制前缀" >&2
exit 1
fi
local compose_exec
if docker compose version >/dev/null 2>&1; then
@@ -66,6 +110,10 @@ Environment=REMOTE_DIR=${REMOTE_DIR}
Environment=REMOTE_ENV_FILE=${REMOTE_ENV_FILE}
Environment=COMPOSE_FILE=${COMPOSE_FILE}
Environment=COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME}
Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}
Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}
Environment=BUILD_SHA=${BUILD_SHA}
Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}
Environment=LIVE_SMOKE_BASE_URL=https://oa.hyinda.com
Environment=WARCH3_SEND_ALERT=1
Environment=WARCH3_OPS_ALERT_SEND=1
@@ -99,7 +147,7 @@ case "$MODE" in
uninstall_timer
;;
--help|-h)
echo "Usage: REMOTE_DIR=/opt/hyb-oa-trialrun bash deploy/trialrun/install-warch3-ops-timer.sh [install|uninstall]"
echo "Usage: REMOTE_DIR=/opt/hyb-oa-trialrun COMPOSE_PROJECT_NAME=<project> POSTGRES_DATA_DIR=<absolute-pgdata> STORAGE_HOST_DIR=<absolute-storage> BUILD_SHA=<full-sha> VITE_BUILD_SHA=<12-char-prefix> bash deploy/trialrun/install-warch3-ops-timer.sh [install|uninstall]"
;;
*)
echo "[ERROR] unsupported mode: $MODE" >&2
+31
查看文件
@@ -241,6 +241,37 @@ for (const [name, content] of [
if (!content.includes('-p "$COMPOSE_PROJECT_NAME"')) {
errors.push(`${name} 必须向 Compose 显式传入独立 project name`);
}
if (!content.includes('"$value" == *"//"*')
|| !content.includes('"/$value/" == *"/../"*')
|| !content.includes('"/$value/" == *"/./"*')) {
errors.push(`${name} 缺少绝对数据路径的空段与点路径段门禁`);
}
if (content.includes('[[:space:]]*)($|/)')) {
errors.push(`${name} 不得以可匹配空字符串的路径段正则拒绝全部绝对路径`);
}
}
for (const requiredFragment of [
'POSTGRES_DATA_DIR="${POSTGRES_DATA_DIR:-}"',
'STORAGE_HOST_DIR="${STORAGE_HOST_DIR:-}"',
'BUILD_SHA="${BUILD_SHA:-}"',
'VITE_BUILD_SHA="${VITE_BUILD_SHA:-}"',
'validate_data_path POSTGRES_DATA_DIR "$POSTGRES_DATA_DIR"',
'validate_data_path STORAGE_HOST_DIR "$STORAGE_HOST_DIR"',
'"$POSTGRES_DATA_DIR" == "$STORAGE_HOST_DIR"/*',
'"$STORAGE_HOST_DIR" == "$POSTGRES_DATA_DIR"/*',
'"$POSTGRES_DATA_DIR/PG_VERSION"',
'"$BUILD_SHA" =~ ^[0-9a-fA-F]{40,64}$',
'"$VITE_BUILD_SHA" =~ ^[0-9a-f]{12}$',
'"${BUILD_SHA:0:12}" != "$VITE_BUILD_SHA"',
'Environment=POSTGRES_DATA_DIR=${POSTGRES_DATA_DIR}',
'Environment=STORAGE_HOST_DIR=${STORAGE_HOST_DIR}',
'Environment=BUILD_SHA=${BUILD_SHA}',
'Environment=VITE_BUILD_SHA=${VITE_BUILD_SHA}',
]) {
if (!timerScriptContent.includes(requiredFragment)) {
errors.push(`install-warch3-ops-timer.sh 缺少活动部署环境绑定: ${requiredFragment}`);
}
}
for (const requiredFragment of [