优化(wecom-public-health-alert-diagnostics-and-debounce): 增强公网健康告警诊断与防抖
这个提交包含在：
@@ -1443,18 +1443,42 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
|||||||
- Units: `hyb-oa-public-app-health.service` and
|
- Units: `hyb-oa-public-app-health.service` and
|
||||||
`hyb-oa-public-app-health.timer`; the timer uses `OnCalendar=*:0/5`.
|
`hyb-oa-public-app-health.timer`; the timer uses `OnCalendar=*:0/5`.
|
||||||
- Local contract: `bash scripts/qa/run-app-health-monitor-contract.sh`.
|
- Local contract: `bash scripts/qa/run-app-health-monitor-contract.sh`.
|
||||||
|
- Application statuses: `FAIL_APP_HEALTH`, `PASS_APP_HEALTH_READY` and
|
||||||
|
`FAIL_APP_HEALTH_FLAPPING`.
|
||||||
|
- Probe-state schema v2 phases: `healthy`, `failure_pending`, `alerted`,
|
||||||
|
`recovery_pending`, `recovery_watch` and `flapping`.
|
||||||
|
|
||||||
### 3. Contracts
|
### 3. Contracts
|
||||||
|
|
||||||
- Run the probe from the existing ARM host against an exact HTTPS `/ready` URL.
|
- Run the probe from the existing ARM host against an exact HTTPS `/ready` URL.
|
||||||
Success requires trusted TLS, HTTP 2xx, bounded JSON and `data.ok === true`.
|
Success requires trusted TLS, HTTP 2xx, bounded JSON and `data.ok === true`.
|
||||||
- Count any stable failure category toward the same consecutive-failure
|
- Classify DNS resolution and TCP connection failures separately before the
|
||||||
threshold. The first failure is pending; the second evaluates a failure
|
shared fallback catches `OSError`; keep TLS, timeout, HTTP, response-size and
|
||||||
alert. Once alerting, a changed category may send a new alert, identical
|
response-contract categories stable. Count any stable failure category toward
|
||||||
failures are deduplicated, and one later success sends one recovery notice.
|
the same consecutive-failure threshold. The first failure is pending and the
|
||||||
- Persist only schema version, count, stable category and the alert state in a
|
second evaluates a failure alert; identical failures remain deduplicated.
|
||||||
root-owned `0700` directory with atomic `0600` files. Never persist or send
|
- After an alert, require two consecutive healthy samples before emitting one
|
||||||
the URL, resolved address, response payload, credential or raw exception.
|
recovery. A single healthy sample followed by another failure stays in the
|
||||||
|
existing incident and sends no recovery noise.
|
||||||
|
- Keep the recovered category under a 60-minute observation window. The same
|
||||||
|
category reaching the two-failure threshold in that window emits one
|
||||||
|
`FAIL_APP_HEALTH_FLAPPING` reminder. Later switches stay local until health
|
||||||
|
has remained continuous for 30 minutes, then the probe exits flapping
|
||||||
|
silently. A different category in the recovery window remains a new normal
|
||||||
|
incident and still uses the two-failure threshold.
|
||||||
|
- Persist only schema version, phase, bounded counts, stable categories and
|
||||||
|
necessary Unix timestamps in a root-owned `0700` directory with atomic
|
||||||
|
`0600` files. Schema v1 migrates conservatively without backfilling a
|
||||||
|
notification. Never persist or send the URL, resolved address, response
|
||||||
|
payload, credential or raw exception.
|
||||||
|
- The probe returns a notification candidate on repeated samples while the
|
||||||
|
corresponding state is active; the shared helper owns final deduplication.
|
||||||
|
This lets a failed send retry on a later sample without repeating a
|
||||||
|
successfully sent message. Healthy flapping samples may retry the stable
|
||||||
|
flapping reminder but never produce a recovery candidate.
|
||||||
|
- Application messages use the explicit titles `OA 公网健康告警`,
|
||||||
|
`OA 公网健康恢复` and `OA 公网健康链路反复抖动`. Shared backup/dead-man
|
||||||
|
statuses retain their generic operations titles.
|
||||||
- Application-health probe state and alert state are separate from backup and
|
- Application-health probe state and alert state are separate from backup and
|
||||||
dead-man state. The installer must deploy the helper revision that passed the
|
dead-man state. The installer must deploy the helper revision that passed the
|
||||||
backup contract; an older installed helper may reject the new status tokens.
|
backup contract; an older installed helper may reject the new status tokens.
|
||||||
@@ -1474,11 +1498,19 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
|||||||
|
|
||||||
- Non-HTTPS, userinfo, query, fragment or non-`/ready` URL -> reject before a
|
- Non-HTTPS, userinfo, query, fragment or non-`/ready` URL -> reject before a
|
||||||
request with `ready_url_invalid`.
|
request with `ready_url_invalid`.
|
||||||
- DNS/connect, TLS, timeout, HTTP, oversized response or invalid JSON/contract
|
- DNS, TCP connection, TLS, timeout, HTTP, oversized response or invalid
|
||||||
-> stable redacted category and non-zero oneshot exit.
|
JSON/contract -> distinct stable redacted category and non-zero oneshot exit.
|
||||||
- Invalid owner/mode/link/state or alert-helper failure -> fail closed with a
|
- Invalid owner/mode/link/state or alert-helper failure -> fail closed with a
|
||||||
fixed local error; do not include the offending value.
|
fixed local error; do not include the offending value.
|
||||||
- A first isolated failure followed by health -> no failure or recovery message.
|
- A first isolated failure followed by health -> no failure or recovery message.
|
||||||
|
- One healthy sample after an alerted failure -> `recovery_pending`, no recovery
|
||||||
|
message; the second consecutive sample -> one recovery.
|
||||||
|
- Same-category recurrence before 60 minutes -> one flapping reminder; at
|
||||||
|
exactly 60 minutes -> a new normal incident, not flapping.
|
||||||
|
- Flapping health shorter than 30 minutes -> remain flapping; at 30 continuous
|
||||||
|
minutes -> return to `healthy` without a message.
|
||||||
|
- Invalid/unknown schema, out-of-range counter/timestamp or inconsistent phase
|
||||||
|
fields -> `local_state_invalid`, with no raw state in output.
|
||||||
- Any overridden managed target or service name -> installer stops before
|
- Any overridden managed target or service name -> installer stops before
|
||||||
`systemctl`, file replacement or removal.
|
`systemctl`, file replacement or removal.
|
||||||
- SSH succeeds but `sudo -n` requests a password -> stop before upload or
|
- SSH succeeds but `sudo -n` requests a password -> stop before upload or
|
||||||
@@ -1491,8 +1523,9 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
|||||||
|
|
||||||
### 5. Good/Base/Bad Cases
|
### 5. Good/Base/Bad Cases
|
||||||
|
|
||||||
- Good: ARM observes two consecutive unavailable checks, sends one redacted
|
- Good: ARM observes two consecutive unavailable checks, sends one explicit
|
||||||
category alert, suppresses repeats and sends one recovery after health returns.
|
OA-public-health alert, suppresses repeats, confirms recovery with two healthy
|
||||||
|
samples, and emits only one flapping reminder for a same-category recurrence.
|
||||||
- Base: repository contracts pass while units remain disabled; report the code
|
- Base: repository contracts pass while units remain disabled; report the code
|
||||||
as ready for an approved rollout, not as an active monitor.
|
as ready for an approved rollout, not as an active monitor.
|
||||||
- Bad: run the timer only on the OA host, send raw `/ready` errors, install and
|
- Bad: run the timer only on the OA host, send raw `/ready` errors, install and
|
||||||
@@ -1501,9 +1534,15 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
|||||||
### 6. Tests Required
|
### 6. Tests Required
|
||||||
|
|
||||||
- The local HTTPS stub must assert healthy baseline, pending first failure,
|
- The local HTTPS stub must assert healthy baseline, pending first failure,
|
||||||
second-failure alert, identical deduplication, changed-category alert, one
|
second-failure alert, identical deduplication, changed-category alert,
|
||||||
recovery, transient suppression, TLS/connect/timeout/HTTP/size/JSON failures,
|
two-sample recovery, transient suppression, same-category flapping,
|
||||||
corrupt private state, URL rejection and absence of sensitive output.
|
different-category recurrence, silent 30-minute exit, exact 30/60-minute
|
||||||
|
boundaries, schema-v1 migration, separate DNS/TCP classification,
|
||||||
|
TLS/timeout/HTTP/size/JSON failures, corrupt private state, URL rejection and
|
||||||
|
absence of sensitive output.
|
||||||
|
- Captured application messages must assert the three explicit titles and
|
||||||
|
stable status/category only. The automatic-backup contract must prove shared
|
||||||
|
helper titles, recovery and deduplication remain unchanged.
|
||||||
- Run Python compile and shell syntax checks, validate the five-minute calendar,
|
- Run Python compile and shell syntax checks, validate the five-minute calendar,
|
||||||
and assert disabled-by-default plus exact managed targets and retained state.
|
and assert disabled-by-default plus exact managed targets and retained state.
|
||||||
- For a remote rollout, record boolean evidence for pre-write `sudo -n`, exact
|
- For a remote rollout, record boolean evidence for pre-write `sudo -n`, exact
|
||||||
@@ -1520,7 +1559,9 @@ PGPASSWORD="$POSTGRES_PASSWORD" docker exec -e PGPASSWORD db pg_dump ...
|
|||||||
#### Wrong
|
#### Wrong
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Local contract passed, so install and enable remotely and call it active.
|
# One healthy sample arrived, so announce recovery immediately and keep
|
||||||
|
# alternating alert/recovery messages during a flapping link.
|
||||||
|
bash scripts/qa/run-app-health-monitor-contract.sh
|
||||||
bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
|
bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1529,6 +1570,8 @@ bash deploy/trialrun/install-offhost-app-health-monitor.sh enable
|
|||||||
```bash
|
```bash
|
||||||
bash scripts/qa/run-app-health-monitor-contract.sh
|
bash scripts/qa/run-app-health-monitor-contract.sh
|
||||||
bash scripts/qa/run-automatic-backup-contract.sh
|
bash scripts/qa/run-automatic-backup-contract.sh
|
||||||
|
# Runtime contract: fail x2 -> alert; healthy x2 -> recovery; same fail x2
|
||||||
|
# within 60m -> one flapping reminder; continuous health 30m -> silent exit.
|
||||||
# After separate ARM-write approval: install disabled, inspect, then seek
|
# After separate ARM-write approval: install disabled, inspect, then seek
|
||||||
# enable/real-message/drill approval before claiming the monitor is active.
|
# enable/real-message/drill approval before claiming the monitor is active.
|
||||||
<explicit-controlled-paths> \
|
<explicit-controlled-paths> \
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# 企微公网健康告警防抖与诊断增强设计
|
||||||
|
|
||||||
|
## 1. Architecture and Boundaries
|
||||||
|
|
||||||
|
本任务保持现有“ARM systemd timer → 公网探测器 → 脱敏 summary → 共用企微 helper”的边界，不引入守护进程、数据库或外部状态服务。
|
||||||
|
|
||||||
|
- 探测器负责网络阶段分类、连续样本计数、恢复确认和抖动周期判定。
|
||||||
|
- 告警 helper 继续负责 summary 校验、消息文案、Webhook 安全读取和发送侧去重。
|
||||||
|
- 安装脚本继续固定 5 分钟 timer、受控路径、root 所有权和默认 disabled。
|
||||||
|
- 本地合同测试通过 stub、受控 socket/DNS 失败和测试 capture 验证，不连接真实机器人。
|
||||||
|
|
||||||
|
## 2. Failure Classification Contract
|
||||||
|
|
||||||
|
分类优先级固定为：
|
||||||
|
|
||||||
|
1. `socket.gaierror` 或等价解析失败 → `dns_resolution_failed`；
|
||||||
|
2. TCP 建连拒绝、重置、不可达及其他建连类错误 → `tcp_connection_failed`；
|
||||||
|
3. TLS 证书或握手校验失败 → `tls_validation_failed`；
|
||||||
|
4. socket/请求超时 → `request_timeout`；
|
||||||
|
5. 非 2xx HTTP → `http_unavailable`；
|
||||||
|
6. 响应超过上限 → `response_too_large`；
|
||||||
|
7. JSON、编码或 `{data: {ok: true}}` 合同不满足 → `response_contract_invalid`。
|
||||||
|
|
||||||
|
异常类型按从具体到通用的顺序匹配，避免 `OSError` 提前吞掉 DNS、TLS 或超时。对无法进一步安全细分的连接阶段异常归入 `tcp_connection_failed`，但不把异常文本写入状态、summary 或日志。
|
||||||
|
|
||||||
|
## 3. State Model
|
||||||
|
|
||||||
|
### 3.1 State schema
|
||||||
|
|
||||||
|
探测状态升级为显式版本，字段只包含：
|
||||||
|
|
||||||
|
- `phase`：`healthy`、`failure_pending`、`alerted`、`recovery_pending`、`recovery_watch`、`flapping`；
|
||||||
|
- `activeCategory` / `lastRecoveredCategory`：稳定类别 token 或 `null`；
|
||||||
|
- `consecutiveFailures` / `consecutiveSuccesses` / `flapTransitions`：有界非负整数；
|
||||||
|
- `recoveredAt` / `healthySince`：必要时存在的非负 Unix 秒时间戳。
|
||||||
|
|
||||||
|
旧版 schema 只含连续失败数和最后类别。部署时采用保守兼容迁移：有效旧状态映射为等价的 `healthy` 或 `failure_pending`，不凭旧状态补发消息；非法或未知 schema 继续 fail closed。告警 helper 自身状态 schema 保持向后兼容。
|
||||||
|
|
||||||
|
### 3.2 Normal incident flow
|
||||||
|
|
||||||
|
```text
|
||||||
|
healthy --失败1--> failure_pending --失败2--> alerted（发送公网健康告警）
|
||||||
|
alerted --成功1--> recovery_pending --成功2--> recovery_watch（发送公网健康恢复）
|
||||||
|
recovery_pending --失败--> alerted（不发送恢复；持续事件仍去重）
|
||||||
|
```
|
||||||
|
|
||||||
|
失败阈值统计持续不可用样本，告警使用达到阈值时的最新稳定类别。已告警期间相同类别不重复发送；类别发生变化时沿用现有最多一次更新通知语义。
|
||||||
|
|
||||||
|
### 3.3 Recovery watch and flapping flow
|
||||||
|
|
||||||
|
```text
|
||||||
|
recovery_watch
|
||||||
|
--60分钟内同类失败达到2次--> flapping（发送一次链路反复抖动提醒）
|
||||||
|
--不同类别失败达到2次-----> alerted（作为新事件发送普通告警）
|
||||||
|
--观察窗到期且健康--------> healthy（静默）
|
||||||
|
|
||||||
|
flapping
|
||||||
|
--任意失败/短时成功-------> flapping（只更新脱敏计数）
|
||||||
|
--连续健康满30分钟--------> healthy（静默退出）
|
||||||
|
```
|
||||||
|
|
||||||
|
“连续健康满 30 分钟”以第一条连续成功样本的时间为起点，期间任何失败都会清空 `healthySince`。状态转换实现为可直接传入当前时间的纯函数，使合同测试可使用合成时间，不依赖等待真实 30/60 分钟，也不增加生产测试参数。
|
||||||
|
|
||||||
|
## 4. Notification Contract
|
||||||
|
|
||||||
|
应用健康事件使用专用标题：
|
||||||
|
|
||||||
|
- `FAIL_APP_HEALTH`：`OA 公网健康告警`；
|
||||||
|
- `PASS_APP_HEALTH_READY`：`OA 公网健康恢复`；
|
||||||
|
- `FAIL_APP_HEALTH_FLAPPING`：`OA 公网健康链路反复抖动`。
|
||||||
|
|
||||||
|
正文只保留状态、稳定类别和操作提示。helper 根据应用健康状态选择专用文案，其他备份/dead-man/提醒状态继续使用原有通用标题。抖动事件采用独立、白名单内的稳定状态 token，防止与普通失败去重键互相覆盖。
|
||||||
|
|
||||||
|
状态转换返回“通知候选”而不是自行发送：处于普通故障、已确认恢复或抖动故障时，可在后续同类样本上重复交给 helper 评估，由 helper 的现有状态完成去重；如果上次发送失败且 helper 状态未落盘，下一次样本可以重试。抖动期的健康样本和静默退出不产生恢复候选，由此保证抖动退出不产生额外恢复消息。
|
||||||
|
|
||||||
|
## 5. Data Flow and Failure Handling
|
||||||
|
|
||||||
|
1. 校验 URL、状态目录、状态文件及阈值。
|
||||||
|
2. 读取并严格校验/迁移探测状态。
|
||||||
|
3. 执行一次 `/ready` 探测并生成成功或稳定失败类别。
|
||||||
|
4. 用当前时间和样本驱动纯状态转换，得到新状态及可选通知候选。
|
||||||
|
5. 原子写入 `0600` 探测状态；有通知候选时才写脱敏 summary 并调用 helper，由 helper 去重或重试。
|
||||||
|
6. 探测失败仍返回非零，健康样本返回零；通知发送或本地状态失败继续 fail closed。
|
||||||
|
|
||||||
|
状态先落盘再通知，保持当前“状态不可安全保存则不发送误导消息”的边界。若通知失败，命令返回失败并保留可诊断的稳定错误；后续同类样本再次提交候选，helper 仅在自身发送状态尚未落盘时重试，合同测试必须证明成功发送后不会重复。
|
||||||
|
|
||||||
|
## 6. Compatibility and Trade-offs
|
||||||
|
|
||||||
|
- 2 次恢复确认会把恢复消息延后约 5 分钟，这是减少短时成功噪声的明确代价。
|
||||||
|
- 同类抖动期不逐次通知，会降低每次瞬断的可见性；本地计数和最终稳定退出状态保留诊断能力。
|
||||||
|
- 不同类别复发仍作为新事件处理，消息可能多于“全部静默”，但避免把新根因错误吞掉。
|
||||||
|
- 不修改 `/ready` 合同、timer 周期、Webhook 文件、summary 隐私边界或非应用健康告警语义。
|
||||||
|
|
||||||
|
## 7. Rollout and Rollback
|
||||||
|
|
||||||
|
- 本地阶段仅提交代码、合同测试和必要文档，不触碰 ARM/DNS/真实企微。
|
||||||
|
- rollout 获批后先以 `install` 模式部署 disabled unit，核对文件权限、unit 内容、旧状态迁移和只读探测，再单独批准 `enable`。
|
||||||
|
- 启用后先观察分类计数和消息频率；DNS 调整只以实际 `dns_resolution_failed` 证据为依据。
|
||||||
|
- 回滚时停用 timer，恢复上一版探测器/helper/unit；保留并备份脱敏状态。若旧版不能读取新 schema，则在明确目标文件后将新状态移出状态目录，由旧版建立健康基线，避免伪造恢复通知。
|
||||||
|
|
||||||
|
## 8. Security and Privacy
|
||||||
|
|
||||||
|
- 禁止写入 URL、域名、IP、文件路径、异常文本、响应 payload、Webhook 或 Secret。
|
||||||
|
- capture 仅在既有显式测试模式下可用，测试输出权限保持 `0600`。
|
||||||
|
- 所有状态枚举和类别经过白名单/token 校验；计数与时间戳必须有类型和范围检查。
|
||||||
|
- 生产、企业微信和 DNS 写入继续受独立批准门禁控制。
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# 企微公网健康告警防抖与诊断增强实施计划
|
||||||
|
|
||||||
|
## Preconditions
|
||||||
|
|
||||||
|
- 用户审核并明确批准本规划后，才运行 `task.py start`。
|
||||||
|
- Phase 2 开始前加载 `trellis-before-dev`，读取 operations 相关 spec、当前任务三份规划材料及引用的运维事实源。
|
||||||
|
- 本任务采用 Codex inline 实施，不整理或依赖 `implement.jsonl` / `check.jsonl`。
|
||||||
|
- 保持现有报销任务未提交改动不动，不纳入本任务 diff、校验或提交。
|
||||||
|
|
||||||
|
## Ordered Implementation Checklist
|
||||||
|
|
||||||
|
1. **探测分类**
|
||||||
|
- 在 `scripts/check-public-app-health.py` 按具体异常优先级拆分 DNS 与 TCP 类别。
|
||||||
|
- 保持 TLS、超时、HTTP、响应过大和响应合同分类稳定，日志只打印 token。
|
||||||
|
2. **状态 schema 与纯状态转换**
|
||||||
|
- 引入显式新版探测状态、严格校验、有界计数和旧 schema 保守迁移。
|
||||||
|
- 将“样本 + 当前时间 → 新状态 + 可选通知候选”提取为纯逻辑。
|
||||||
|
- 实现 2 次失败、2 次成功、60 分钟同类复发提醒、抖动期静默计数和连续健康 30 分钟静默退出。
|
||||||
|
3. **专用消息文案与兼容**
|
||||||
|
- 为应用健康普通告警、恢复和抖动事件增加明确的“OA 公网健康”标题。
|
||||||
|
- 扩展 helper 白名单及发送判断，但保持备份、dead-man 和其他既有状态文案/去重行为不变。
|
||||||
|
4. **合同测试**
|
||||||
|
- 扩充 `scripts/qa/run-app-health-monitor-contract.sh`：分别模拟 DNS、TCP、TLS、超时、HTTP、格式、过大响应。
|
||||||
|
- 使用纯状态转换或合成时间覆盖全部正常、恢复确认、同类抖动、不同类别复发、30/60 分钟边界和旧状态迁移。
|
||||||
|
- 断言消息数量、顺序、专用标题、静默退出及敏感内容禁入。
|
||||||
|
- 回归损坏状态 fail-closed、文件权限、固定路径、默认 disabled 和 5 分钟 timer。
|
||||||
|
5. **共享合同与事实源**
|
||||||
|
- 运行自动备份合同，证明共用 helper 无回归。
|
||||||
|
- 仅在形成稳定可复用规则时，更新既有 `.trellis/spec/hyb-oa/` 或 `docs/试运行与自动化验证手册.md`；不新建无必要文档。
|
||||||
|
6. **本地质量门禁**
|
||||||
|
- 逐项运行下列验证，限定敏感扫描和 diff 审核在本任务相关文件。
|
||||||
|
- 对权限/状态机变更完成主会话只读复核；inline 模式不派发 implement/check 子代理。
|
||||||
|
7. **提交与交付**
|
||||||
|
- 只暂存当前 Trellis task 和本任务实现/测试/必要事实源文件。
|
||||||
|
- 使用中文 commit message，并引用 `wecom-public-health-alert-diagnostics-and-debounce`。
|
||||||
|
- 本地完成后仅报告“本地验证通过”；未获 ARM 部署与公网检查批准前，不告知用户可在企业微信测试。
|
||||||
|
|
||||||
|
## Validation Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m py_compile scripts/check-public-app-health.py scripts/send-wecom-group-alert.py
|
||||||
|
bash scripts/qa/run-app-health-monitor-contract.sh
|
||||||
|
bash scripts/qa/run-automatic-backup-contract.sh
|
||||||
|
bash -n deploy/trialrun/install-offhost-app-health-monitor.sh
|
||||||
|
node scripts/qa/check-process-contract.mjs --baseline --json
|
||||||
|
node scripts/qa/audit-docs.mjs --check-registry
|
||||||
|
node scripts/qa/check-trellis-governance.mjs
|
||||||
|
python3 ./.trellis/scripts/task.py validate 08-23-wecom-public-health-alert-diagnostics-and-debounce
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
敏感扫描采用稳定关键词和本任务文件清单，不打印潜在 Secret 值或原始 payload。
|
||||||
|
|
||||||
|
## Risky Files and Rollback Points
|
||||||
|
|
||||||
|
- `scripts/check-public-app-health.py`：状态迁移和通知时机会改变；每一步先以合同测试锁定消息数量，再进入下一步。
|
||||||
|
- `scripts/send-wecom-group-alert.py`：被备份/dead-man 共用；任何改动必须先通过应用健康合同，再通过完整自动备份合同。
|
||||||
|
- `deploy/trialrun/install-offhost-app-health-monitor.sh`：若无需新增生产参数则尽量不改；如需改动，必须保持默认 disabled 和固定安全目标。
|
||||||
|
- 状态 schema：必须支持有效旧状态保守迁移；回滚前先停 timer，并只对已解析的精确状态文件执行保留/移出操作。
|
||||||
|
|
||||||
|
## External Approval Gates
|
||||||
|
|
||||||
|
以下动作均不属于本地实施批准，执行前必须分别再次获得明确授权：
|
||||||
|
|
||||||
|
1. ARM 文件、systemd 或生产配置写入；
|
||||||
|
2. 启用/重启公网健康 timer；
|
||||||
|
3. 向真实企业微信群发送测试或生产消息；
|
||||||
|
4. 修改 DNS、resolver 或任何网络策略；
|
||||||
|
5. 写入真实业务数据或执行恢复操作。
|
||||||
@@ -0,0 +1,92 @@
|
|||||||
|
# 企微公网健康告警防抖与诊断增强
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
让 ARM 公网健康探测准确区分 DNS、TCP、TLS、HTTP 和响应合同故障，以明确来源的企业微信文案报告真实事件，并在链路反复抖动时减少无意义的“告警—恢复”消息，同时保持约 10 分钟量级的持续 OA 不可用发现能力。
|
||||||
|
|
||||||
|
## Background and Confirmed Facts
|
||||||
|
|
||||||
|
- 上一排查任务已确认群内反复消息来自 ARM 对公网 `/ready` 的健康探测，不是备份失败或数据恢复；生产自动备份和新鲜度 watchdog 近 14 天均为 14/14 成功、0 条失败。
|
||||||
|
- 实施前基线 `986668a58762` 每 5 分钟探测，连续 2 次失败即告警、1 次成功即恢复；基线实现见该提交的 `scripts/check-public-app-health.py:220-279` 和 `deploy/trialrun/install-offhost-app-health-monitor.sh:94-113`。
|
||||||
|
- 实施前基线 `scripts/check-public-app-health.py:131-170` 将 DNS 与 TCP 故障合并为 `dns_or_connect_failed`。
|
||||||
|
- 实施前基线 `scripts/send-wecom-group-alert.py:133-140` 为所有健康恢复统一使用“内部 OA 运维恢复通知”，接收人无法直接判断这是公网健康监控。
|
||||||
|
- 当前 ARM 健康。单一静态 DNS 是可疑单点，但现有证据不足以断言历史故障均由 DNS 导致，因此本任务先增强诊断和防抖，不直接修改 DNS。
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### R1 — 故障阶段分类
|
||||||
|
|
||||||
|
- 将 `dns_or_connect_failed` 至少拆分为 `dns_resolution_failed` 与 `tcp_connection_failed`。
|
||||||
|
- 保留并稳定区分 `tls_validation_failed`、`request_timeout`、`http_unavailable`、`response_contract_invalid` 和 `response_too_large`。
|
||||||
|
- 分类只记录稳定类别，不记录解析结果、目标地址、异常原文、响应正文或请求 payload。
|
||||||
|
|
||||||
|
### R2 — 公网健康专用文案
|
||||||
|
|
||||||
|
- `FAIL_APP_HEALTH`、`PASS_APP_HEALTH_READY` 和新增的 `FAIL_APP_HEALTH_FLAPPING` 抖动事件必须明确显示“OA 公网健康”。
|
||||||
|
- 恢复文案不得使用可能被理解为数据库、附件或备份恢复的泛化标题。
|
||||||
|
- 自动备份、异机 dead-man、季度提醒等共用告警 helper 的既有标题、发送条件和脱敏合同保持兼容。
|
||||||
|
|
||||||
|
### R3 — 均衡防抖状态机
|
||||||
|
|
||||||
|
- 保持 5 分钟探测频率和连续 2 次失败告警，持续不可用仍在约 10 分钟量级发送一次告警。
|
||||||
|
- 已发送故障告警后，必须连续 2 次成功才发送一次恢复；单次短暂成功只进入恢复待确认状态，不发送恢复。
|
||||||
|
- 正常恢复后的 60 分钟观察窗内，同一故障类别再次连续达到失败阈值时，只发送一次“OA 公网健康链路反复抖动”提醒并进入抖动状态。
|
||||||
|
- 抖动状态内后续失败、成功及同类切换只更新本地脱敏计数，不逐次发送告警或恢复。
|
||||||
|
- 抖动状态在连续健康满 30 分钟后静默退出；退出本身不另发消息。之后的新故障按新的正常周期处理。
|
||||||
|
- 恢复观察窗内出现不同故障类别时，按新的故障事件执行连续 2 次失败门槛，避免隐藏不同根因；同一持续故障期间类别变化仍按既有“类别变化最多通知一次”的兼容行为处理。
|
||||||
|
|
||||||
|
### R4 — 状态与隐私边界
|
||||||
|
|
||||||
|
- 本地状态仅允许保存 schema 版本、稳定状态/类别、连续成功或失败计数、必要的 Unix 时间戳和抖动计数。
|
||||||
|
- 状态文件继续使用原子写入、`0600` 权限和严格 schema 校验；损坏或越界状态 fail closed，并只输出稳定脱敏错误类别。
|
||||||
|
- 不保存主机名、IP、URL、路径、异常原文、Webhook、Secret 或原始响应；测试捕获不得访问真实企业微信。
|
||||||
|
|
||||||
|
### R5 — 发布门禁
|
||||||
|
|
||||||
|
- 本地实现和合同验证不授权 ARM 写入、systemd 变更、DNS 修改或真实企业微信消息测试。
|
||||||
|
- 后续 ARM rollout 必须再次取得明确批准，并按“disabled 安装 → 只读检查 → 受控启用”执行；DNS 调整必须依据增强分类后的观测证据另行决策。
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
- `scripts/check-public-app-health.py` 的故障分类、恢复确认、抖动状态机和脱敏状态 schema。
|
||||||
|
- `scripts/send-wecom-group-alert.py` 的公网健康专用文案及共享行为兼容。
|
||||||
|
- `scripts/qa/run-app-health-monitor-contract.sh`、`scripts/qa/run-automatic-backup-contract.sh` 和 ARM 安装合同的相关回归。
|
||||||
|
- 必要的既有运维事实源更新，以及后续 ARM rollout/rollback 操作设计。
|
||||||
|
|
||||||
|
## Out of Scope
|
||||||
|
|
||||||
|
- 规划阶段修改产品代码、ARM 配置、systemd 或 DNS。
|
||||||
|
- 自动重启 OA、切换流量、数据库修复或任何数据恢复。
|
||||||
|
- 部署 Prometheus/Alertmanager 或引入新的消息供应商。
|
||||||
|
- 未取得证据前指定公共 DNS、覆盖主机网络策略或停用健康监控。
|
||||||
|
- 在本任务内执行生产部署、真实企微消息发送或真实数据写入。
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] **AC1 / R1**：无外网依赖的合同测试可稳定区分 DNS、TCP、TLS、超时、HTTP、响应格式及响应过大故障。
|
||||||
|
- [x] **AC2 / R2**：应用健康告警、恢复及抖动提醒的标题均明确包含“OA 公网健康”，测试证明消息不含地址、路径、异常原文或 payload。
|
||||||
|
- [x] **AC3 / R3**：5 分钟频率与 2 次失败阈值保持不变；持续故障只发送一次告警，相同故障持续期间不重复发送。
|
||||||
|
- [x] **AC4 / R3**：状态机测试证明 1 次成功不恢复、2 次连续成功只恢复一次；恢复后 60 分钟内同类复发只提醒一次，抖动期切换静默计数，连续健康 30 分钟后静默退出。
|
||||||
|
- [x] **AC5 / R3**：恢复观察窗内的不同类别故障不会被同类抖动规则吞掉，并仍经过连续 2 次失败门槛。
|
||||||
|
- [x] **AC6 / R4**：本地状态 schema、权限、原子写入、损坏状态 fail-closed 和敏感字段禁入均有自动化覆盖。
|
||||||
|
- [x] **AC7 / R2**：自动备份、异机 dead-man 和既有共享 helper 合同全部通过，非应用健康消息行为无回归。
|
||||||
|
- [x] **AC8 / R5**：安装合同保持默认 disabled、5 分钟周期、固定安全目标及独立启用门禁；本地验证不产生真实企微或远端写入。
|
||||||
|
|
||||||
|
## Test Plan
|
||||||
|
|
||||||
|
- `python3 -m py_compile scripts/check-public-app-health.py scripts/send-wecom-group-alert.py`
|
||||||
|
- `bash scripts/qa/run-app-health-monitor-contract.sh`
|
||||||
|
- `bash scripts/qa/run-automatic-backup-contract.sh`
|
||||||
|
- `bash -n deploy/trialrun/install-offhost-app-health-monitor.sh`
|
||||||
|
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||||
|
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||||
|
- `node scripts/qa/check-trellis-governance.mjs`
|
||||||
|
- 针对本任务变更执行敏感信息扫描和 `git diff --check`。
|
||||||
|
- ARM 写入获批后才执行 disabled 安装、只读状态检查、受控启用和回滚验证；真实消息测试与 DNS 修改仍需分别批准。
|
||||||
|
|
||||||
|
## Local Verification Result
|
||||||
|
|
||||||
|
- `bash scripts/qa/run-app-health-monitor-contract.sh`：PASS，36 项检查。
|
||||||
|
- `bash scripts/qa/run-automatic-backup-contract.sh`：PASS，74 项检查，共享备份/dead-man 标题和去重无回归。
|
||||||
|
- Python/Shell 语法、process baseline、文档注册审计、敏感扫描、Trellis governance、task validate 和 `git diff --check` 均通过；文档审计仅保留既有的未登记文件 warning。
|
||||||
|
- 当前结论为 `LOCAL_FIX_VERIFIED`；ARM 部署、真实群消息、DNS 调整和外部行为验证仍为 `EXTERNAL_GATED`。
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
{
|
||||||
|
"id": "wecom-public-health-alert-diagnostics-and-debounce",
|
||||||
|
"name": "wecom-public-health-alert-diagnostics-and-debounce",
|
||||||
|
"title": "企微公网健康告警防抖与诊断增强",
|
||||||
|
"description": "细分 ARM 公网健康探测故障阶段，优化企微告警来源文案与防抖状态机，并为后续 DNS 调整建立证据门槛。",
|
||||||
|
"status": "in_progress",
|
||||||
|
"dev_type": null,
|
||||||
|
"scope": "operations",
|
||||||
|
"package": null,
|
||||||
|
"priority": "P2",
|
||||||
|
"creator": "shunli",
|
||||||
|
"assignee": "shunli",
|
||||||
|
"createdAt": "2026-08-23",
|
||||||
|
"completedAt": null,
|
||||||
|
"branch": null,
|
||||||
|
"base_branch": "main",
|
||||||
|
"worktree_path": null,
|
||||||
|
"commit": null,
|
||||||
|
"pr_url": null,
|
||||||
|
"subtasks": [],
|
||||||
|
"children": [],
|
||||||
|
"parent": null,
|
||||||
|
"relatedFiles": [
|
||||||
|
"scripts/check-public-app-health.py",
|
||||||
|
"scripts/send-wecom-group-alert.py",
|
||||||
|
"scripts/qa/run-app-health-monitor-contract.sh",
|
||||||
|
"scripts/qa/run-automatic-backup-contract.sh",
|
||||||
|
".trellis/spec/hyb-oa/deployment-trialrun.md",
|
||||||
|
"docs/试运行与自动化验证手册.md"
|
||||||
|
],
|
||||||
|
"notes": "本地实现与合同验证完成；ARM 部署、真实企微消息、DNS 调整和外部行为验证仍需独立批准。",
|
||||||
|
"meta": {}
|
||||||
|
}
|
||||||
+5
-3
@@ -154,9 +154,11 @@ cd client && E2E_BASE_URL=https://oa.hyinda.com npx playwright test tests/e2e/dg
|
|||||||
|
|
||||||
### ARM 异机应用健康监控
|
### ARM 异机应用健康监控
|
||||||
|
|
||||||
`07-27-app-health-monitoring` 为公网 `/ready` 增加轻量异机探测：ARM 每 5 分钟经标准 HTTPS 校验访问一次；连续两次失败才发送一条脱敏企微群机器人告警，相同类别持续失败不重复发送，恢复后只通知一次。它覆盖 OA 应用、数据库就绪、OA 主机不可达、DNS/Caddy/TLS 已失效等基础故障，不替代登录、审批或报销等业务 smoke，也不执行自动重启或恢复。
|
`07-27-app-health-monitoring` 为公网 `/ready` 增加轻量异机探测：ARM 每 5 分钟经标准 HTTPS 校验访问一次；连续两次失败才发送一条脱敏企微群机器人告警，相同类别持续失败不重复发送。它覆盖 OA 应用、数据库就绪、OA 主机不可达、DNS/Caddy/TLS 已失效等基础故障，不替代登录、审批或报销等业务 smoke，也不执行自动重启或恢复。
|
||||||
|
|
||||||
2026-07-27 已完成 ARM 安装、启用和真实故障/恢复演练：timer 为 enabled/active，首次失败 pending、第二次发送、重复失败降噪、正式恢复发送、重复健康降噪，用户确认群内实际收到一条故障与一条恢复通知；备份 dead-man 未受影响，临时上传和临时 sudoers 已删除。后续重装、真实消息或再次演练仍需单独批准。先运行本地无真实消息合同：
|
2026-07-27 已完成 ARM 安装、启用和真实故障/恢复演练：timer 为 enabled/active，首次失败 pending、第二次发送、重复失败降噪、正式恢复发送、重复健康降噪，用户确认群内实际收到一条故障与一条恢复通知；备份 dead-man 未受影响，临时上传和临时 sudoers 已删除。
|
||||||
|
|
||||||
|
`08-23-wecom-public-health-alert-diagnostics-and-debounce` 已完成本地候选增强：DNS 与 TCP 分类分离；应用消息标题明确为“OA 公网健康”；恢复改为连续 2 次健康确认；正常恢复后 60 分钟内同类故障再次达到 2 次失败时只发送一次链路反复抖动提醒，后续切换只保留脱敏计数，连续健康 30 分钟后静默退出。该结论目前只代表本地合同通过，尚未部署到 ARM，不能据此声称群消息已经改变。ARM rollout、真实消息和 DNS 修改仍需分别批准。先运行本地无真实消息合同：
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash scripts/qa/run-app-health-monitor-contract.sh
|
bash scripts/qa/run-app-health-monitor-contract.sh
|
||||||
@@ -171,7 +173,7 @@ systemctl list-timers hyb-oa-public-app-health.timer --no-pager
|
|||||||
systemctl status hyb-oa-public-app-health.service --no-pager
|
systemctl status hyb-oa-public-app-health.service --no-pager
|
||||||
```
|
```
|
||||||
|
|
||||||
受控演练必须证明：第一次失败仅进入 pending；第二次持续失败只发一条；重复失败降噪；恢复后只发一条。群消息只允许状态和稳定类别，不粘贴地址、路径、响应、凭证或 journal 原文。ARM 自身宕机仍会令该 timer 失声，首版保留为人工 timer 巡检风险，不据此宣称具备第三方 dead-man。
|
增强版受控演练必须证明：第一次失败仅进入 pending；第二次持续失败只发一条；重复失败降噪；第一次健康不恢复、第二次连续健康只恢复一条；60 分钟内同类复发只产生一条抖动提醒；连续健康 30 分钟后静默退出。群消息只允许状态和稳定类别，不粘贴地址、路径、响应、凭证或 journal 原文。ARM 自身宕机仍会令该 timer 失声，首版保留为人工 timer 巡检风险，不据此宣称具备第三方 dead-man。
|
||||||
|
|
||||||
发布候选进入公网或小范围试运行前，先按 [`docs/trialrun/发布试运行准备清单.md`](trialrun/发布试运行准备清单.md) 完成证据整理、权限直连审计、部署预检、live smoke 和试运行分级。
|
发布候选进入公网或小范围试运行前，先按 [`docs/trialrun/发布试运行准备清单.md`](trialrun/发布试运行准备清单.md) 完成证据整理、权限直连审计、部署预检、live smoke 和试运行分级。
|
||||||
|
|
||||||
|
|||||||
+413
-46
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Probe the public OA readiness endpoint and drive a deduplicated alert."""
|
"""Probe the public OA readiness endpoint and drive a debounced alert."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -14,6 +14,7 @@ import stat
|
|||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import time
|
||||||
import urllib.error
|
import urllib.error
|
||||||
import urllib.parse
|
import urllib.parse
|
||||||
import urllib.request
|
import urllib.request
|
||||||
@@ -23,8 +24,44 @@ from pathlib import Path
|
|||||||
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
||||||
SAFE_HOST = re.compile(r"^[A-Za-z0-9.-]{1,253}$")
|
SAFE_HOST = re.compile(r"^[A-Za-z0-9.-]{1,253}$")
|
||||||
MAX_RESPONSE_BYTES = 64 * 1024
|
MAX_RESPONSE_BYTES = 64 * 1024
|
||||||
|
MAX_COUNTER = 1_000_000
|
||||||
|
MAX_TIMESTAMP = (1 << 53) - 1
|
||||||
|
STATE_SCHEMA_VERSION = 2
|
||||||
|
RECOVERY_THRESHOLD = 2
|
||||||
|
RECOVERY_WATCH_SECONDS = 60 * 60
|
||||||
|
FLAPPING_STABLE_SECONDS = 30 * 60
|
||||||
HEALTHY_STATUS = "PASS_APP_HEALTH_READY"
|
HEALTHY_STATUS = "PASS_APP_HEALTH_READY"
|
||||||
FAILURE_STATUS = "FAIL_APP_HEALTH"
|
FAILURE_STATUS = "FAIL_APP_HEALTH"
|
||||||
|
FLAPPING_STATUS = "FAIL_APP_HEALTH_FLAPPING"
|
||||||
|
FAILURE_CATEGORIES = {
|
||||||
|
"dns_resolution_failed",
|
||||||
|
"tcp_connection_failed",
|
||||||
|
"tls_validation_failed",
|
||||||
|
"request_timeout",
|
||||||
|
"http_unavailable",
|
||||||
|
"response_contract_invalid",
|
||||||
|
"response_too_large",
|
||||||
|
}
|
||||||
|
LEGACY_FAILURE_CATEGORIES = FAILURE_CATEGORIES | {"dns_or_connect_failed"}
|
||||||
|
PHASES = {
|
||||||
|
"healthy",
|
||||||
|
"failure_pending",
|
||||||
|
"alerted",
|
||||||
|
"recovery_pending",
|
||||||
|
"recovery_watch",
|
||||||
|
"flapping",
|
||||||
|
}
|
||||||
|
STATE_KEYS = {
|
||||||
|
"schemaVersion",
|
||||||
|
"phase",
|
||||||
|
"activeCategory",
|
||||||
|
"lastRecoveredCategory",
|
||||||
|
"consecutiveFailures",
|
||||||
|
"consecutiveSuccesses",
|
||||||
|
"recoveredAt",
|
||||||
|
"healthySince",
|
||||||
|
"flapTransitions",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
class ProbeFailure(Exception):
|
class ProbeFailure(Exception):
|
||||||
@@ -89,9 +126,139 @@ def validate_ready_url(value: str) -> str:
|
|||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def new_probe_state() -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"schemaVersion": STATE_SCHEMA_VERSION,
|
||||||
|
"phase": "healthy",
|
||||||
|
"activeCategory": None,
|
||||||
|
"lastRecoveredCategory": None,
|
||||||
|
"consecutiveFailures": 0,
|
||||||
|
"consecutiveSuccesses": 0,
|
||||||
|
"recoveredAt": None,
|
||||||
|
"healthySince": None,
|
||||||
|
"flapTransitions": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def is_bounded_int(value: object, minimum: int = 0, maximum: int = MAX_COUNTER) -> bool:
|
||||||
|
return isinstance(value, int) and not isinstance(value, bool) and minimum <= value <= maximum
|
||||||
|
|
||||||
|
|
||||||
|
def is_category(value: object, allow_none: bool = True) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return allow_none
|
||||||
|
return isinstance(value, str) and value in LEGACY_FAILURE_CATEGORIES and bool(SAFE_TOKEN.fullmatch(value))
|
||||||
|
|
||||||
|
|
||||||
|
def is_timestamp(value: object, allow_none: bool = True) -> bool:
|
||||||
|
if value is None:
|
||||||
|
return allow_none
|
||||||
|
return is_bounded_int(value, maximum=MAX_TIMESTAMP)
|
||||||
|
|
||||||
|
|
||||||
|
def validate_v2_state(payload: dict[str, object]) -> bool:
|
||||||
|
if set(payload) != STATE_KEYS or payload.get("schemaVersion") != STATE_SCHEMA_VERSION:
|
||||||
|
return False
|
||||||
|
phase = payload.get("phase")
|
||||||
|
active = payload.get("activeCategory")
|
||||||
|
recovered = payload.get("lastRecoveredCategory")
|
||||||
|
failures = payload.get("consecutiveFailures")
|
||||||
|
successes = payload.get("consecutiveSuccesses")
|
||||||
|
recovered_at = payload.get("recoveredAt")
|
||||||
|
healthy_since = payload.get("healthySince")
|
||||||
|
transitions = payload.get("flapTransitions")
|
||||||
|
if (
|
||||||
|
phase not in PHASES
|
||||||
|
or not is_category(active)
|
||||||
|
or not is_category(recovered)
|
||||||
|
or not is_bounded_int(failures)
|
||||||
|
or not is_bounded_int(successes)
|
||||||
|
or not is_timestamp(recovered_at)
|
||||||
|
or not is_timestamp(healthy_since)
|
||||||
|
or not is_bounded_int(transitions)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
if phase == "healthy":
|
||||||
|
return (
|
||||||
|
active is None
|
||||||
|
and recovered is None
|
||||||
|
and failures == 0
|
||||||
|
and successes == 0
|
||||||
|
and recovered_at is None
|
||||||
|
and healthy_since is None
|
||||||
|
and transitions == 0
|
||||||
|
)
|
||||||
|
if phase == "failure_pending":
|
||||||
|
return (
|
||||||
|
active is not None
|
||||||
|
and failures >= 1
|
||||||
|
and successes == 0
|
||||||
|
and healthy_since is None
|
||||||
|
and transitions == 0
|
||||||
|
and ((recovered is None and recovered_at is None) or (recovered is not None and recovered_at is not None))
|
||||||
|
)
|
||||||
|
if phase == "alerted":
|
||||||
|
return (
|
||||||
|
active is not None
|
||||||
|
and failures >= 1
|
||||||
|
and successes == 0
|
||||||
|
and recovered is None
|
||||||
|
and recovered_at is None
|
||||||
|
and healthy_since is None
|
||||||
|
and transitions == 0
|
||||||
|
)
|
||||||
|
if phase == "recovery_pending":
|
||||||
|
return (
|
||||||
|
active is not None
|
||||||
|
and failures == 0
|
||||||
|
and successes >= 1
|
||||||
|
and recovered is None
|
||||||
|
and recovered_at is None
|
||||||
|
and healthy_since is None
|
||||||
|
and transitions == 0
|
||||||
|
)
|
||||||
|
if phase == "recovery_watch":
|
||||||
|
return (
|
||||||
|
active is None
|
||||||
|
and recovered is not None
|
||||||
|
and failures == 0
|
||||||
|
and successes == 0
|
||||||
|
and recovered_at is not None
|
||||||
|
and healthy_since is None
|
||||||
|
and transitions == 0
|
||||||
|
)
|
||||||
|
if phase != "flapping" or active is None or active != recovered or recovered_at is None or transitions < 1:
|
||||||
|
return False
|
||||||
|
if healthy_since is None:
|
||||||
|
return failures >= 1 and successes == 0
|
||||||
|
return failures == 0 and successes >= 1
|
||||||
|
|
||||||
|
|
||||||
|
def migrate_v1_state(payload: dict[str, object]) -> dict[str, object] | None:
|
||||||
|
if set(payload) != {"schemaVersion", "consecutiveFailures", "lastCategory"}:
|
||||||
|
return None
|
||||||
|
count = payload.get("consecutiveFailures")
|
||||||
|
category = payload.get("lastCategory")
|
||||||
|
if payload.get("schemaVersion") != 1 or not is_bounded_int(count) or not is_category(category):
|
||||||
|
return None
|
||||||
|
state = new_probe_state()
|
||||||
|
if count == 0:
|
||||||
|
return state if category is None else None
|
||||||
|
if category is None:
|
||||||
|
return None
|
||||||
|
state.update(
|
||||||
|
{
|
||||||
|
"phase": "failure_pending",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": count,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
def load_probe_state(path: Path) -> dict[str, object]:
|
def load_probe_state(path: Path) -> dict[str, object]:
|
||||||
if not path.exists():
|
if not path.exists():
|
||||||
return {"schemaVersion": 1, "consecutiveFailures": 0, "lastCategory": None}
|
return new_probe_state()
|
||||||
validate_regular_file(path, "local_state", private=True)
|
validate_regular_file(path, "local_state", private=True)
|
||||||
try:
|
try:
|
||||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||||
@@ -99,17 +266,14 @@ def load_probe_state(path: Path) -> dict[str, object]:
|
|||||||
fail("local_state_invalid")
|
fail("local_state_invalid")
|
||||||
if not isinstance(payload, dict):
|
if not isinstance(payload, dict):
|
||||||
fail("local_state_invalid")
|
fail("local_state_invalid")
|
||||||
count = payload.get("consecutiveFailures")
|
if payload.get("schemaVersion") == STATE_SCHEMA_VERSION:
|
||||||
category = payload.get("lastCategory")
|
if not validate_v2_state(payload):
|
||||||
if (
|
fail("local_state_invalid")
|
||||||
payload.get("schemaVersion") != 1
|
return payload
|
||||||
or not isinstance(count, int)
|
migrated = migrate_v1_state(payload)
|
||||||
or isinstance(count, bool)
|
if migrated is None:
|
||||||
or count < 0
|
|
||||||
or (category is not None and (not isinstance(category, str) or not SAFE_TOKEN.fullmatch(category)))
|
|
||||||
):
|
|
||||||
fail("local_state_invalid")
|
fail("local_state_invalid")
|
||||||
return payload
|
return migrated
|
||||||
|
|
||||||
|
|
||||||
def atomic_write(path: Path, payload: dict[str, object]) -> None:
|
def atomic_write(path: Path, payload: dict[str, object]) -> None:
|
||||||
@@ -128,13 +292,193 @@ def atomic_write(path: Path, payload: dict[str, object]) -> None:
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
|
|
||||||
def classify_url_error(error: urllib.error.URLError) -> str:
|
def capped(value: int, maximum: int = MAX_COUNTER) -> int:
|
||||||
reason = error.reason
|
return min(value, maximum)
|
||||||
|
|
||||||
|
|
||||||
|
def elapsed(now: int, started_at: object) -> int:
|
||||||
|
if not isinstance(started_at, int) or isinstance(started_at, bool):
|
||||||
|
return 0
|
||||||
|
return max(0, now - started_at)
|
||||||
|
|
||||||
|
|
||||||
|
def transition_probe_state(
|
||||||
|
state: dict[str, object],
|
||||||
|
category: str | None,
|
||||||
|
now: int,
|
||||||
|
failure_threshold: int,
|
||||||
|
) -> tuple[dict[str, object], tuple[str, str] | None]:
|
||||||
|
"""Return the next redacted state and an optional notification candidate."""
|
||||||
|
if not validate_v2_state(state) or not is_timestamp(now, allow_none=False):
|
||||||
|
raise ValueError("invalid transition input")
|
||||||
|
if category is not None and category not in FAILURE_CATEGORIES:
|
||||||
|
raise ValueError("invalid transition category")
|
||||||
|
phase = str(state["phase"])
|
||||||
|
|
||||||
|
if category is None:
|
||||||
|
if phase == "healthy":
|
||||||
|
return state.copy(), None
|
||||||
|
if phase == "failure_pending":
|
||||||
|
if state["recoveredAt"] is not None:
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "recovery_watch",
|
||||||
|
"activeCategory": None,
|
||||||
|
"consecutiveFailures": 0,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, None
|
||||||
|
return new_probe_state(), None
|
||||||
|
if phase == "alerted":
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "recovery_pending",
|
||||||
|
"consecutiveFailures": 0,
|
||||||
|
"consecutiveSuccesses": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, None
|
||||||
|
if phase == "recovery_pending":
|
||||||
|
successes = capped(int(state["consecutiveSuccesses"]) + 1)
|
||||||
|
if successes < RECOVERY_THRESHOLD:
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state["consecutiveSuccesses"] = successes
|
||||||
|
return next_state, None
|
||||||
|
recovered_category = str(state["activeCategory"])
|
||||||
|
next_state = new_probe_state()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "recovery_watch",
|
||||||
|
"lastRecoveredCategory": recovered_category,
|
||||||
|
"recoveredAt": now,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (HEALTHY_STATUS, "public_ready_healthy")
|
||||||
|
if phase == "recovery_watch":
|
||||||
|
if elapsed(now, state["recoveredAt"]) >= RECOVERY_WATCH_SECONDS:
|
||||||
|
return new_probe_state(), None
|
||||||
|
return state.copy(), (HEALTHY_STATUS, "public_ready_healthy")
|
||||||
|
if phase == "flapping":
|
||||||
|
next_state = state.copy()
|
||||||
|
if state["healthySince"] is None:
|
||||||
|
next_state["healthySince"] = now
|
||||||
|
next_state["consecutiveFailures"] = 0
|
||||||
|
next_state["consecutiveSuccesses"] = 1
|
||||||
|
next_state["flapTransitions"] = capped(int(state["flapTransitions"]) + 1)
|
||||||
|
else:
|
||||||
|
next_state["consecutiveSuccesses"] = capped(int(state["consecutiveSuccesses"]) + 1)
|
||||||
|
if elapsed(now, next_state["healthySince"]) >= FLAPPING_STABLE_SECONDS:
|
||||||
|
return new_probe_state(), None
|
||||||
|
return next_state, (FLAPPING_STATUS, str(state["activeCategory"]))
|
||||||
|
|
||||||
|
if phase == "flapping":
|
||||||
|
next_state = state.copy()
|
||||||
|
if state["healthySince"] is not None:
|
||||||
|
next_state["flapTransitions"] = capped(int(state["flapTransitions"]) + 1)
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"consecutiveFailures": capped(int(state["consecutiveFailures"]) + 1),
|
||||||
|
"consecutiveSuccesses": 0,
|
||||||
|
"healthySince": None,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (FLAPPING_STATUS, str(state["activeCategory"]))
|
||||||
|
|
||||||
|
if phase == "alerted":
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": capped(int(state["consecutiveFailures"]) + 1),
|
||||||
|
"consecutiveSuccesses": 0,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (FAILURE_STATUS, category)
|
||||||
|
|
||||||
|
if phase == "recovery_pending":
|
||||||
|
next_state = new_probe_state()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "alerted",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (FAILURE_STATUS, category)
|
||||||
|
|
||||||
|
if phase == "recovery_watch":
|
||||||
|
within_watch = elapsed(now, state["recoveredAt"]) < RECOVERY_WATCH_SECONDS
|
||||||
|
next_state = new_probe_state()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "failure_pending",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": 1,
|
||||||
|
"lastRecoveredCategory": state["lastRecoveredCategory"] if within_watch else None,
|
||||||
|
"recoveredAt": state["recoveredAt"] if within_watch else None,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, None
|
||||||
|
|
||||||
|
if phase == "healthy":
|
||||||
|
next_state = new_probe_state()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "failure_pending",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, None
|
||||||
|
|
||||||
|
failures = capped(int(state["consecutiveFailures"]) + 1)
|
||||||
|
if failures < failure_threshold:
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state.update({"activeCategory": category, "consecutiveFailures": failures})
|
||||||
|
return next_state, None
|
||||||
|
same_recent_failure = (
|
||||||
|
state["lastRecoveredCategory"] == category
|
||||||
|
and state["recoveredAt"] is not None
|
||||||
|
and elapsed(now, state["recoveredAt"]) < RECOVERY_WATCH_SECONDS
|
||||||
|
)
|
||||||
|
if same_recent_failure:
|
||||||
|
next_state = state.copy()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "flapping",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": failures,
|
||||||
|
"consecutiveSuccesses": 0,
|
||||||
|
"healthySince": None,
|
||||||
|
"flapTransitions": 1,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (FLAPPING_STATUS, category)
|
||||||
|
next_state = new_probe_state()
|
||||||
|
next_state.update(
|
||||||
|
{
|
||||||
|
"phase": "alerted",
|
||||||
|
"activeCategory": category,
|
||||||
|
"consecutiveFailures": failures,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return next_state, (FAILURE_STATUS, category)
|
||||||
|
|
||||||
|
|
||||||
|
def classify_connection_error(reason: object) -> str:
|
||||||
if isinstance(reason, (ssl.SSLError, ssl.CertificateError)):
|
if isinstance(reason, (ssl.SSLError, ssl.CertificateError)):
|
||||||
return "tls_validation_failed"
|
return "tls_validation_failed"
|
||||||
if isinstance(reason, (socket.timeout, TimeoutError)):
|
if isinstance(reason, (socket.timeout, TimeoutError)):
|
||||||
return "request_timeout"
|
return "request_timeout"
|
||||||
return "dns_or_connect_failed"
|
if isinstance(reason, socket.gaierror):
|
||||||
|
return "dns_resolution_failed"
|
||||||
|
return "tcp_connection_failed"
|
||||||
|
|
||||||
|
|
||||||
|
def classify_url_error(error: urllib.error.URLError) -> str:
|
||||||
|
return classify_connection_error(error.reason)
|
||||||
|
|
||||||
|
|
||||||
def probe(ready_url: str, timeout_seconds: float, ca_file: Path | None) -> None:
|
def probe(ready_url: str, timeout_seconds: float, ca_file: Path | None) -> None:
|
||||||
@@ -166,8 +510,10 @@ def probe(ready_url: str, timeout_seconds: float, ca_file: Path | None) -> None:
|
|||||||
raise ProbeFailure("request_timeout") from None
|
raise ProbeFailure("request_timeout") from None
|
||||||
except (ssl.SSLError, ssl.CertificateError):
|
except (ssl.SSLError, ssl.CertificateError):
|
||||||
raise ProbeFailure("tls_validation_failed") from None
|
raise ProbeFailure("tls_validation_failed") from None
|
||||||
|
except socket.gaierror:
|
||||||
|
raise ProbeFailure("dns_resolution_failed") from None
|
||||||
except (http.client.HTTPException, ConnectionError, OSError):
|
except (http.client.HTTPException, ConnectionError, OSError):
|
||||||
raise ProbeFailure("dns_or_connect_failed") from None
|
raise ProbeFailure("tcp_connection_failed") from None
|
||||||
if len(body) > MAX_RESPONSE_BYTES:
|
if len(body) > MAX_RESPONSE_BYTES:
|
||||||
raise ProbeFailure("response_too_large")
|
raise ProbeFailure("response_too_large")
|
||||||
try:
|
try:
|
||||||
@@ -179,7 +525,7 @@ def probe(ready_url: str, timeout_seconds: float, ca_file: Path | None) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def write_summary(path: Path, status: str, category: str) -> None:
|
def write_summary(path: Path, status: str, category: str) -> None:
|
||||||
finding = status == FAILURE_STATUS
|
finding = status in {FAILURE_STATUS, FLAPPING_STATUS}
|
||||||
payload = {
|
payload = {
|
||||||
"issueId": "APP_HEALTH",
|
"issueId": "APP_HEALTH",
|
||||||
"status": status,
|
"status": status,
|
||||||
@@ -217,6 +563,26 @@ def evaluate_alert(alert_helper: Path, webhook_file: Path, summary_file: Path, a
|
|||||||
fail("alert_evaluation_failed")
|
fail("alert_evaluation_failed")
|
||||||
|
|
||||||
|
|
||||||
|
def persist_transition(
|
||||||
|
state_file: Path,
|
||||||
|
summary_file: Path,
|
||||||
|
alert_state_file: Path,
|
||||||
|
state: dict[str, object],
|
||||||
|
candidate: tuple[str, str] | None,
|
||||||
|
alert_helper: Path,
|
||||||
|
webhook_file: Path,
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
atomic_write(state_file, state)
|
||||||
|
if candidate is not None:
|
||||||
|
status, category = candidate
|
||||||
|
write_summary(summary_file, status, category)
|
||||||
|
except OSError:
|
||||||
|
fail("local_state_invalid")
|
||||||
|
if candidate is not None:
|
||||||
|
evaluate_alert(alert_helper, webhook_file, summary_file, alert_state_file)
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser()
|
parser = argparse.ArgumentParser()
|
||||||
parser.add_argument("--ready-url", required=True)
|
parser.add_argument("--ready-url", required=True)
|
||||||
@@ -238,45 +604,46 @@ def main() -> None:
|
|||||||
summary_file = args.state_dir / "last-app-health.json"
|
summary_file = args.state_dir / "last-app-health.json"
|
||||||
alert_state_file = args.state_dir / "app-health-alert-state.json"
|
alert_state_file = args.state_dir / "app-health-alert-state.json"
|
||||||
state = load_probe_state(probe_state_file)
|
state = load_probe_state(probe_state_file)
|
||||||
|
now = int(time.time())
|
||||||
|
|
||||||
try:
|
try:
|
||||||
probe(ready_url, args.timeout_seconds, args.ca_file)
|
probe(ready_url, args.timeout_seconds, args.ca_file)
|
||||||
except ProbeFailure as error:
|
except ProbeFailure as error:
|
||||||
failures = min(int(state["consecutiveFailures"]) + 1, args.failure_threshold)
|
|
||||||
try:
|
try:
|
||||||
atomic_write(
|
next_state, candidate = transition_probe_state(state, error.category, now, args.failure_threshold)
|
||||||
probe_state_file,
|
except ValueError:
|
||||||
{"schemaVersion": 1, "consecutiveFailures": failures, "lastCategory": error.category},
|
|
||||||
)
|
|
||||||
except OSError:
|
|
||||||
fail("local_state_invalid")
|
fail("local_state_invalid")
|
||||||
if failures >= args.failure_threshold:
|
persist_transition(
|
||||||
try:
|
probe_state_file,
|
||||||
write_summary(summary_file, FAILURE_STATUS, error.category)
|
summary_file,
|
||||||
except OSError:
|
alert_state_file,
|
||||||
fail("local_state_invalid")
|
next_state,
|
||||||
evaluate_alert(args.alert_helper, args.webhook_file, summary_file, alert_state_file)
|
candidate,
|
||||||
print(
|
args.alert_helper,
|
||||||
f"[app-health][ERROR] health_check_failed category={error.category} consecutive={failures} alert=evaluated",
|
args.webhook_file,
|
||||||
file=sys.stderr,
|
)
|
||||||
)
|
alert_result = "evaluated" if candidate is not None else "pending"
|
||||||
else:
|
print(
|
||||||
print(
|
f"[app-health][ERROR] health_check_failed category={error.category} "
|
||||||
f"[app-health][ERROR] health_check_failed category={error.category} consecutive={failures} alert=pending",
|
f"consecutive={next_state['consecutiveFailures']} alert={alert_result}",
|
||||||
file=sys.stderr,
|
file=sys.stderr,
|
||||||
)
|
)
|
||||||
raise SystemExit(1)
|
raise SystemExit(1)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
atomic_write(
|
next_state, candidate = transition_probe_state(state, None, now, args.failure_threshold)
|
||||||
probe_state_file,
|
except ValueError:
|
||||||
{"schemaVersion": 1, "consecutiveFailures": 0, "lastCategory": None},
|
|
||||||
)
|
|
||||||
write_summary(summary_file, HEALTHY_STATUS, "public_ready_healthy")
|
|
||||||
except OSError:
|
|
||||||
fail("local_state_invalid")
|
fail("local_state_invalid")
|
||||||
evaluate_alert(args.alert_helper, args.webhook_file, summary_file, alert_state_file)
|
persist_transition(
|
||||||
print("[app-health][PASS] public readiness is healthy")
|
probe_state_file,
|
||||||
|
summary_file,
|
||||||
|
alert_state_file,
|
||||||
|
next_state,
|
||||||
|
candidate,
|
||||||
|
args.alert_helper,
|
||||||
|
args.webhook_file,
|
||||||
|
)
|
||||||
|
print(f"[app-health][PASS] public readiness is healthy phase={next_state['phase']}")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -137,6 +137,7 @@ MAIN_CAPTURE="$TEST_DIR/main-capture.jsonl"
|
|||||||
new_state_dir "$MAIN_STATE"
|
new_state_dir "$MAIN_STATE"
|
||||||
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
[[ "$(capture_lines "$MAIN_CAPTURE")" == "0" ]]
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "0" ]]
|
||||||
|
[[ "$(stat -c %a "$MAIN_STATE/app-health-probe-state.json")" == "600" ]]
|
||||||
pass "healthy baseline writes state without sending an alert"
|
pass "healthy baseline writes state without sending an alert"
|
||||||
|
|
||||||
set_mode unavailable
|
set_mode unavailable
|
||||||
@@ -155,14 +156,74 @@ pass "changed failure category sends one updated alert"
|
|||||||
|
|
||||||
set_mode healthy
|
set_mode healthy
|
||||||
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "2" ]]
|
||||||
|
pass "first healthy sample keeps recovery pending"
|
||||||
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
[[ "$(capture_lines "$MAIN_CAPTURE")" == "3" ]]
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "3" ]]
|
||||||
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
[[ "$(capture_lines "$MAIN_CAPTURE")" == "3" ]]
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "3" ]]
|
||||||
pass "recovery sends exactly one notification"
|
pass "second healthy sample sends exactly one recovery notification"
|
||||||
|
|
||||||
|
set_mode invalid-json
|
||||||
|
expect_failure "first same-category recurrence stays pending" probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null 2>&1
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "3" ]]
|
||||||
|
expect_failure "second same-category recurrence enters flapping" probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null 2>&1
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "4" ]]
|
||||||
|
expect_failure "continued flapping failure remains deduplicated" probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null 2>&1
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "4" ]]
|
||||||
|
set_mode healthy
|
||||||
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "4" ]]
|
||||||
|
python3 - "$MAIN_STATE/app-health-probe-state.json" <<'PY'
|
||||||
|
import json, os, sys, time
|
||||||
|
path = sys.argv[1]
|
||||||
|
payload = json.load(open(path, encoding="utf-8"))
|
||||||
|
assert payload["phase"] == "flapping", payload
|
||||||
|
payload["healthySince"] = int(time.time()) - 1800
|
||||||
|
with open(path, "w", encoding="utf-8") as handle:
|
||||||
|
json.dump(payload, handle)
|
||||||
|
handle.write("\n")
|
||||||
|
os.chmod(path, 0o600)
|
||||||
|
PY
|
||||||
|
probe "$MAIN_STATE" "$MAIN_CAPTURE" >/dev/null
|
||||||
|
[[ "$(capture_lines "$MAIN_CAPTURE")" == "4" ]]
|
||||||
|
python3 - "$MAIN_STATE/app-health-probe-state.json" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
|
assert payload["phase"] == "healthy", payload
|
||||||
|
assert payload["flapTransitions"] == 0, payload
|
||||||
|
PY
|
||||||
|
pass "flapping sends once and exits silently after thirty healthy minutes"
|
||||||
|
|
||||||
|
DIFFERENT_STATE="$TEST_DIR/different-state"
|
||||||
|
DIFFERENT_CAPTURE="$TEST_DIR/different-capture.jsonl"
|
||||||
|
new_state_dir "$DIFFERENT_STATE"
|
||||||
|
set_mode healthy
|
||||||
|
probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null
|
||||||
|
set_mode unavailable
|
||||||
|
expect_failure "different-case first failure stays pending" probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null 2>&1
|
||||||
|
expect_failure "different-case second failure alerts" probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null 2>&1
|
||||||
|
set_mode healthy
|
||||||
|
probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null
|
||||||
|
probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null
|
||||||
|
[[ "$(capture_lines "$DIFFERENT_CAPTURE")" == "2" ]]
|
||||||
|
set_mode invalid-json
|
||||||
|
expect_failure "new-category recurrence first sample stays pending" probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null 2>&1
|
||||||
|
[[ "$(capture_lines "$DIFFERENT_CAPTURE")" == "2" ]]
|
||||||
|
expect_failure "new-category recurrence becomes a normal alert" probe "$DIFFERENT_STATE" "$DIFFERENT_CAPTURE" >/dev/null 2>&1
|
||||||
|
[[ "$(capture_lines "$DIFFERENT_CAPTURE")" == "3" ]]
|
||||||
|
python3 - "$DIFFERENT_STATE/app-health-probe-state.json" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
|
assert payload["phase"] == "alerted", payload
|
||||||
|
assert payload["activeCategory"] == "response_contract_invalid", payload
|
||||||
|
PY
|
||||||
|
pass "different failure category is not swallowed by same-category flapping debounce"
|
||||||
|
|
||||||
TRANSIENT_STATE="$TEST_DIR/transient-state"
|
TRANSIENT_STATE="$TEST_DIR/transient-state"
|
||||||
TRANSIENT_CAPTURE="$TEST_DIR/transient-capture.jsonl"
|
TRANSIENT_CAPTURE="$TEST_DIR/transient-capture.jsonl"
|
||||||
new_state_dir "$TRANSIENT_STATE"
|
new_state_dir "$TRANSIENT_STATE"
|
||||||
|
set_mode healthy
|
||||||
probe "$TRANSIENT_STATE" "$TRANSIENT_CAPTURE" >/dev/null
|
probe "$TRANSIENT_STATE" "$TRANSIENT_CAPTURE" >/dev/null
|
||||||
set_mode unavailable
|
set_mode unavailable
|
||||||
expect_failure "single transient outage is recorded" probe "$TRANSIENT_STATE" "$TRANSIENT_CAPTURE" >/dev/null 2>&1
|
expect_failure "single transient outage is recorded" probe "$TRANSIENT_STATE" "$TRANSIENT_CAPTURE" >/dev/null 2>&1
|
||||||
@@ -184,10 +245,48 @@ expected = {
|
|||||||
"timeout": "request_timeout",
|
"timeout": "request_timeout",
|
||||||
}[sys.argv[2]]
|
}[sys.argv[2]]
|
||||||
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
assert payload["lastCategory"] == expected, payload
|
assert payload["activeCategory"] == expected, payload
|
||||||
PY
|
PY
|
||||||
done
|
done
|
||||||
|
|
||||||
|
python3 - "$ROOT_DIR/scripts/check-public-app-health.py" <<'PY'
|
||||||
|
import importlib.util
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location("app_health", sys.argv[1])
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
assert module.classify_url_error(urllib.error.URLError(socket.gaierror(-2, "private-dns-text"))) == "dns_resolution_failed"
|
||||||
|
assert module.classify_url_error(urllib.error.URLError(ConnectionRefusedError())) == "tcp_connection_failed"
|
||||||
|
|
||||||
|
watch = module.new_probe_state()
|
||||||
|
watch.update({
|
||||||
|
"phase": "recovery_watch",
|
||||||
|
"lastRecoveredCategory": "http_unavailable",
|
||||||
|
"recoveredAt": 1000,
|
||||||
|
})
|
||||||
|
pending, candidate = module.transition_probe_state(watch, "http_unavailable", 4599, 2)
|
||||||
|
assert pending["phase"] == "failure_pending" and candidate is None, (pending, candidate)
|
||||||
|
flapping, candidate = module.transition_probe_state(pending, "http_unavailable", 4599, 2)
|
||||||
|
assert flapping["phase"] == "flapping", flapping
|
||||||
|
assert candidate == ("FAIL_APP_HEALTH_FLAPPING", "http_unavailable"), candidate
|
||||||
|
|
||||||
|
expired, candidate = module.transition_probe_state(watch, "http_unavailable", 4600, 2)
|
||||||
|
assert expired["phase"] == "failure_pending" and expired["recoveredAt"] is None, expired
|
||||||
|
alerted, candidate = module.transition_probe_state(expired, "http_unavailable", 4600, 2)
|
||||||
|
assert alerted["phase"] == "alerted", alerted
|
||||||
|
assert candidate == ("FAIL_APP_HEALTH", "http_unavailable"), candidate
|
||||||
|
|
||||||
|
flapping["healthySince"] = 5000
|
||||||
|
flapping["consecutiveFailures"] = 0
|
||||||
|
flapping["consecutiveSuccesses"] = 1
|
||||||
|
stable, candidate = module.transition_probe_state(flapping, None, 6800, 2)
|
||||||
|
assert stable["phase"] == "healthy" and candidate is None, (stable, candidate)
|
||||||
|
PY
|
||||||
|
pass "DNS/TCP classification and exact thirty/sixty-minute boundaries are deterministic"
|
||||||
|
|
||||||
UNUSED_PORT="$(python3 - <<'PY'
|
UNUSED_PORT="$(python3 - <<'PY'
|
||||||
import socket
|
import socket
|
||||||
sock = socket.socket()
|
sock = socket.socket()
|
||||||
@@ -207,9 +306,9 @@ expect_failure "connection failure is rejected" env \
|
|||||||
python3 - "$CONNECT_STATE/app-health-probe-state.json" <<'PY'
|
python3 - "$CONNECT_STATE/app-health-probe-state.json" <<'PY'
|
||||||
import json, sys
|
import json, sys
|
||||||
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
assert payload["lastCategory"] == "dns_or_connect_failed", payload
|
assert payload["activeCategory"] == "tcp_connection_failed", payload
|
||||||
PY
|
PY
|
||||||
pass "connection failure is stored as a stable category"
|
pass "TCP connection failure is stored separately from DNS resolution failure"
|
||||||
|
|
||||||
TLS_STATE="$TEST_DIR/tls-state"
|
TLS_STATE="$TEST_DIR/tls-state"
|
||||||
new_state_dir "$TLS_STATE"
|
new_state_dir "$TLS_STATE"
|
||||||
@@ -222,10 +321,39 @@ expect_failure "untrusted TLS certificate is rejected" env \
|
|||||||
python3 - "$TLS_STATE/app-health-probe-state.json" <<'PY'
|
python3 - "$TLS_STATE/app-health-probe-state.json" <<'PY'
|
||||||
import json, sys
|
import json, sys
|
||||||
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
assert payload["lastCategory"] == "tls_validation_failed", payload
|
assert payload["activeCategory"] == "tls_validation_failed", payload
|
||||||
PY
|
PY
|
||||||
pass "TLS failure is stored as a stable category"
|
pass "TLS failure is stored as a stable category"
|
||||||
|
|
||||||
|
MIGRATION_STATE="$TEST_DIR/migration-state"
|
||||||
|
MIGRATION_CAPTURE="$TEST_DIR/migration-capture.jsonl"
|
||||||
|
new_state_dir "$MIGRATION_STATE"
|
||||||
|
printf '%s\n' '{"schemaVersion":1,"consecutiveFailures":0,"lastCategory":null}' \
|
||||||
|
>"$MIGRATION_STATE/app-health-probe-state.json"
|
||||||
|
chmod 0600 "$MIGRATION_STATE/app-health-probe-state.json"
|
||||||
|
set_mode healthy
|
||||||
|
probe "$MIGRATION_STATE" "$MIGRATION_CAPTURE" >/dev/null
|
||||||
|
python3 - "$MIGRATION_STATE/app-health-probe-state.json" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
payload = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||||
|
assert payload["schemaVersion"] == 2, payload
|
||||||
|
assert payload["phase"] == "healthy", payload
|
||||||
|
PY
|
||||||
|
[[ "$(capture_lines "$MIGRATION_CAPTURE")" == "0" ]]
|
||||||
|
pass "valid legacy state migrates without backfilling a notification"
|
||||||
|
|
||||||
|
INVALID_SCHEMA_STATE="$TEST_DIR/invalid-schema-state"
|
||||||
|
new_state_dir "$INVALID_SCHEMA_STATE"
|
||||||
|
printf '%s\n' '{"schemaVersion":2,"phase":"healthy","activeCategory":null,"lastRecoveredCategory":null,"consecutiveFailures":1000001,"consecutiveSuccesses":0,"recoveredAt":null,"healthySince":null,"flapTransitions":0}' \
|
||||||
|
>"$INVALID_SCHEMA_STATE/app-health-probe-state.json"
|
||||||
|
chmod 0600 "$INVALID_SCHEMA_STATE/app-health-probe-state.json"
|
||||||
|
set_mode healthy
|
||||||
|
expect_failure "out-of-range v2 state fails closed" probe "$INVALID_SCHEMA_STATE" \
|
||||||
|
"$TEST_DIR/invalid-schema-capture.jsonl" >"$TEST_DIR/invalid-schema-stdout" 2>"$TEST_DIR/invalid-schema-stderr"
|
||||||
|
grep -Fxq '[app-health][ERROR] local_state_invalid' "$TEST_DIR/invalid-schema-stderr"
|
||||||
|
! grep -Fq '1000001' "$TEST_DIR/invalid-schema-stderr"
|
||||||
|
pass "v2 state schema rejects inconsistent or out-of-range fields without disclosure"
|
||||||
|
|
||||||
CORRUPT_STATE="$TEST_DIR/corrupt-state"
|
CORRUPT_STATE="$TEST_DIR/corrupt-state"
|
||||||
new_state_dir "$CORRUPT_STATE"
|
new_state_dir "$CORRUPT_STATE"
|
||||||
printf '%s\n' 'private-payload-must-not-appear' >"$CORRUPT_STATE/app-health-probe-state.json"
|
printf '%s\n' 'private-payload-must-not-appear' >"$CORRUPT_STATE/app-health-probe-state.json"
|
||||||
@@ -241,15 +369,19 @@ pass "local state error output is stable and redacted"
|
|||||||
python3 - "$MAIN_CAPTURE" <<'PY'
|
python3 - "$MAIN_CAPTURE" <<'PY'
|
||||||
import json, sys
|
import json, sys
|
||||||
messages = [json.loads(line)["text"]["content"] for line in open(sys.argv[1], encoding="utf-8")]
|
messages = [json.loads(line)["text"]["content"] for line in open(sys.argv[1], encoding="utf-8")]
|
||||||
assert len(messages) == 3, messages
|
assert len(messages) == 4, messages
|
||||||
assert "FAIL_APP_HEALTH" in messages[0]
|
assert "FAIL_APP_HEALTH" in messages[0]
|
||||||
assert "http_unavailable" in messages[0]
|
assert "http_unavailable" in messages[0]
|
||||||
assert "response_contract_invalid" in messages[1]
|
assert "response_contract_invalid" in messages[1]
|
||||||
assert "PASS_APP_HEALTH_READY" in messages[2]
|
assert "PASS_APP_HEALTH_READY" in messages[2]
|
||||||
|
assert "FAIL_APP_HEALTH_FLAPPING" in messages[3]
|
||||||
|
assert messages[0].startswith("OA 公网健康告警\n"), messages[0]
|
||||||
|
assert messages[2].startswith("OA 公网健康恢复\n"), messages[2]
|
||||||
|
assert messages[3].startswith("OA 公网健康链路反复抖动\n"), messages[3]
|
||||||
for forbidden in ("do-not-leak", "localhost", "https://", "/tmp/", "not-json-private-payload"):
|
for forbidden in ("do-not-leak", "localhost", "https://", "/tmp/", "not-json-private-payload"):
|
||||||
assert all(forbidden not in message for message in messages), (forbidden, messages)
|
assert all(forbidden not in message for message in messages), (forbidden, messages)
|
||||||
PY
|
PY
|
||||||
pass "captured alerts contain only stable status and category text"
|
pass "captured application alerts use explicit source titles and stable redacted categories"
|
||||||
|
|
||||||
expect_failure "plain HTTP readiness URL is rejected" env \
|
expect_failure "plain HTTP readiness URL is rejected" env \
|
||||||
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$TEST_DIR/http-capture.jsonl" \
|
HYB_WECOM_ALERT_TEST_MODE=1 HYB_WECOM_ALERT_TEST_CAPTURE="$TEST_DIR/http-capture.jsonl" \
|
||||||
|
|||||||
@@ -701,6 +701,16 @@ mv "$TEST_DIR/deadman-withheld-batch" "$DEADMAN_LAYOUT/archive/$SECOND_BATCH"
|
|||||||
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
--webhook-file "$TEST_DIR/wecom-webhook" --alert-helper "$ROOT_DIR/scripts/send-wecom-group-alert.py" >/dev/null
|
||||||
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]]
|
[[ "$(wc -l <"$DEADMAN_CAPTURE")" == "2" ]]
|
||||||
pass "ARM dead-man sends one recovery notification after finalized archives resume"
|
pass "ARM dead-man sends one recovery notification after finalized archives resume"
|
||||||
|
python3 - "$DEADMAN_CAPTURE" <<'PYTHON'
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
messages = [json.loads(line)["text"]["content"] for line in open(sys.argv[1], encoding="utf-8")]
|
||||||
|
assert messages[0].startswith("内部 OA 运维告警\n"), messages
|
||||||
|
assert messages[1].startswith("内部 OA 运维恢复通知\n"), messages
|
||||||
|
assert all("OA 公网健康" not in message for message in messages), messages
|
||||||
|
PYTHON
|
||||||
|
pass "shared helper keeps dead-man alert and recovery titles unchanged"
|
||||||
|
|
||||||
DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567"
|
DRIFT_BUILD_SHA="0123456789abcdef0123456789abcdef01234567"
|
||||||
expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \
|
expect_failure "wrapper rejects an active runtime build that drifted from its installed contract" \
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ from pathlib import Path
|
|||||||
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
SAFE_TOKEN = re.compile(r"^[A-Za-z0-9_-]{1,160}$")
|
||||||
ALLOWED_STATUSES = {
|
ALLOWED_STATUSES = {
|
||||||
"FAIL_APP_HEALTH",
|
"FAIL_APP_HEALTH",
|
||||||
|
"FAIL_APP_HEALTH_FLAPPING",
|
||||||
"FAIL_AUTOMATIC_BACKUP",
|
"FAIL_AUTOMATIC_BACKUP",
|
||||||
"FAIL_OFFHOST_DEADMAN",
|
"FAIL_OFFHOST_DEADMAN",
|
||||||
"PASS_APP_HEALTH_READY",
|
"PASS_APP_HEALTH_READY",
|
||||||
@@ -131,7 +132,13 @@ def should_send(previous: dict[str, object], status: str, category: str) -> bool
|
|||||||
|
|
||||||
|
|
||||||
def safe_text(status: str, category: str) -> str:
|
def safe_text(status: str, category: str) -> str:
|
||||||
if status.startswith("FAIL_"):
|
if status == "FAIL_APP_HEALTH":
|
||||||
|
headline = "OA 公网健康告警"
|
||||||
|
elif status == "FAIL_APP_HEALTH_FLAPPING":
|
||||||
|
headline = "OA 公网健康链路反复抖动"
|
||||||
|
elif status == "PASS_APP_HEALTH_READY":
|
||||||
|
headline = "OA 公网健康恢复"
|
||||||
|
elif status.startswith("FAIL_"):
|
||||||
headline = "内部 OA 运维告警"
|
headline = "内部 OA 运维告警"
|
||||||
elif status == "PASS_WITH_FINDINGS":
|
elif status == "PASS_WITH_FINDINGS":
|
||||||
headline = "内部 OA 运维提醒"
|
headline = "内部 OA 运维提醒"
|
||||||
|
|||||||
在新工单中引用
屏蔽一个用户