比较提交
20
次代码提交
41da4c2638
...
59dc5f66a4
| 作者 | SHA1 | 提交日期 | |
|---|---|---|---|
|
|
59dc5f66a4 | ||
|
|
1055fea246 | ||
|
|
8c28e15a74 | ||
|
|
e8d676634f | ||
|
|
174cb7acde | ||
|
|
46bfa7e00f | ||
|
|
f178aa0de3 | ||
|
|
be057e665d | ||
|
|
1bb066ddb7 | ||
|
|
d6a2b85dc2 | ||
|
|
573c46fbe7 | ||
|
|
a1c59e2a78 | ||
|
|
f84e6cf4e3 | ||
|
|
7bfaa92bf1 | ||
|
|
0b94b97a3a | ||
|
|
e0e2465e0c | ||
|
|
6d7dfbb1eb | ||
|
|
18233d3433 | ||
|
|
f216387837 | ||
|
|
5ec7db13b2 |
+3
@@ -27,6 +27,9 @@ artifacts/
|
||||
# Local sensitive business samples; never commit originals
|
||||
/input/
|
||||
|
||||
# Private off-host restore render inputs and site-specific candidates
|
||||
/.restore-drill-private/
|
||||
|
||||
# Vitest coverage artifacts
|
||||
server/coverage
|
||||
server/.vitest
|
||||
|
||||
@@ -1210,6 +1210,394 @@ test "$(printf '%s\n' "$actual" | LC_ALL=C sort)" = \
|
||||
"$(printf '%s\n' "$expected" | LC_ALL=C sort)"
|
||||
```
|
||||
|
||||
## Scenario: Off-Host Archive And Private-Copy Identity Validation
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: an off-host recovery drill copies a formally archived backup batch
|
||||
into a private temporary directory before database or attachment restore.
|
||||
- Purpose: preserve the archive placement evidence at the source boundary while
|
||||
validating a renamed private copy by its source-bound content identity.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Source validation:
|
||||
`validate_source_batch(<archive-run-directory>) -> source_metadata`.
|
||||
- Private-copy validation:
|
||||
`validate_copied_batch(<private-copy-directory>, <source_metadata>)`.
|
||||
- `source_metadata` contains only redacted comparison values: verified run id,
|
||||
manifest/build/time identity, exact three-file set, byte sizes and SHA-256
|
||||
values. It must not contain credentials, business payloads or full paths.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- A formal archive run directory must have a basename that exactly equals the
|
||||
validated manifest run id. This detects a valid manifest placed under the
|
||||
wrong archive run and is mandatory before any copy or restore write.
|
||||
- A private copy may use an operator-owned neutral basename such as `batch`;
|
||||
that basename is not archive provenance and must not be compared with the
|
||||
manifest run id.
|
||||
- The copied manifest run id must exactly equal the run id already verified at
|
||||
the source. Internal self-consistency inside the copy is insufficient.
|
||||
- The copy must repeat manifest schema/status, run-id format, exact
|
||||
`manifest.json`/`database.dump`/`storage.tar.gz` regular-file set, build and
|
||||
age gates, byte-size checks and SHA-256 checks.
|
||||
- After repeated validation, every redacted copied metadata field must equal
|
||||
`source_metadata`. Copy validation must never disable a source-side gate or
|
||||
accept a newly invented identity.
|
||||
- Formal archives remain read-only. A validation failure stops before creating
|
||||
PostgreSQL, extracting attachments or modifying the archive, and cleanup must
|
||||
prove zero private runtime and privilege-entry residue.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Source basename differs from manifest run id -> `batch_identity`; `NO_GO`
|
||||
before copy.
|
||||
- Copied manifest run id differs from the verified source run id, even when the
|
||||
copy is internally self-consistent -> `copied_batch_invalid/batch_identity`;
|
||||
`NO_GO` before restore runtime creation.
|
||||
- Copied file set, bytes, SHA-256, build, time or other redacted metadata differs
|
||||
from the source -> `copied_batch_invalid`; remove only the owned private
|
||||
runtime and retain the archive unchanged.
|
||||
- Private basename differs from the run id but all source-bound content checks
|
||||
pass -> valid; continue to the separately approved restore step.
|
||||
- Any cleanup residue remains -> attempt stays non-clean and cannot be reported
|
||||
as recovery `GO`.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: source directory basename matches its manifest run id; the same batch
|
||||
is copied to a private directory named `batch`, revalidated, and its redacted
|
||||
metadata exactly equals the source record.
|
||||
- Base: source and copy basenames happen to match; both still use their proper
|
||||
validation modes and the copy remains bound to `source_metadata`.
|
||||
- Bad: reuse source validation on the private directory and reject it only
|
||||
because the operator chose a neutral basename.
|
||||
- Bad: skip the copy basename check and then trust any internally consistent
|
||||
replacement manifest without comparing it with the verified source run id.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Positive: a formal source whose basename matches its manifest run id passes.
|
||||
- Positive: an unchanged copy under a different private basename passes and
|
||||
produces metadata equal to the source record.
|
||||
- Negative: source basename/run-id mismatch fails before copy.
|
||||
- Negative: copied content or checksum mutation fails before restore.
|
||||
- Negative: a copied manifest with an internally consistent but source-different
|
||||
run id fails, proving the source binding cannot be bypassed.
|
||||
- Regression: run the existing synthetic backup/restore contract and assert no
|
||||
PostgreSQL object, extracted attachment, private runtime or privilege entry is
|
||||
left after every pre-runtime failure.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```text
|
||||
validate_batch(source, require_basename_match=true)
|
||||
copy source -> private/batch
|
||||
validate_batch(private/batch, require_basename_match=true)
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```text
|
||||
source_metadata = validate_source_batch(source_run_directory)
|
||||
copy source_run_directory -> private/batch
|
||||
validate_copied_batch(private/batch, source_metadata)
|
||||
```
|
||||
|
||||
## Scenario: Shadow-Restore Failure Diagnostics Before Teardown
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: a one-shot off-host shadow restore keeps raw tool output only in a
|
||||
private runtime that must be deleted on every exit path.
|
||||
- Purpose: preserve enough allowlisted stage evidence to locate a restore
|
||||
failure without retaining or publishing raw stderr, paths, credentials, SQL
|
||||
or restored object names.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Required stages: `dump_list`, `target_confirmation`, `restore_apply`, each
|
||||
with the enum `PASS|FAIL|NOT_RUN`.
|
||||
- Required execution fields: `restore_execution_path=DOCKER_STDIN|HOST_FILE|NOT_RUN`,
|
||||
`dump_list_exit_code`, `target_confirmation_exit_code`,
|
||||
`target_name_match=yes|no|NOT_RUN`, `target_confirmation_attempt_count`,
|
||||
`target_confirmation_success_streak`, `restore_script_exit_code`,
|
||||
`pg_restore_exit_code`, `pg_restore_error_count` and
|
||||
`restore_error_category`.
|
||||
- Exit-code fields are non-negative integers or `NOT_RUN`;
|
||||
`pg_restore_error_count` additionally permits `UNKNOWN` when no safe numeric
|
||||
summary is available.
|
||||
- Required data-restore timing fields:
|
||||
`rto_batch_copy_seconds`, `rto_postgres_container_start_seconds`,
|
||||
`rto_dump_list_seconds`, `rto_target_confirmation_seconds`,
|
||||
`rto_restore_apply_seconds`, `rto_database_verify_seconds`,
|
||||
`rto_storage_attachment_verify_seconds`, `rto_final_integrity_seconds` and
|
||||
`data_restore_rto_seconds`. A measured value is a non-negative integer;
|
||||
an unstarted stage is `NOT_RUN`.
|
||||
- Tracked supply-chain entries:
|
||||
`scripts/qa/offhost-shadow-restore-runner.template.sh`,
|
||||
`scripts/qa/validate-offhost-shadow-restore-attachments.py` and
|
||||
`bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh`.
|
||||
- Candidate render:
|
||||
`bash scripts/qa/run-offhost-shadow-restore-runner-contract.sh render <private-config.json> <new-output>`;
|
||||
it prints only PASS and the candidate SHA-256, never config values or paths.
|
||||
- One-shot installer manifest: the candidate `LIB_DIR`, runner entry, dependency
|
||||
basenames and sudoers entry form one exact private mapping. The installer must
|
||||
validate that mapping before its first privileged write; a sibling `lib` /
|
||||
`libexec` path reconstructed independently is not equivalent.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- Emit the fixed diagnostic fields on the failure path before runtime teardown.
|
||||
Teardown remains in a trap/finally-equivalent path and must still run after
|
||||
diagnostic emission; callers retain stdout, not the private raw log.
|
||||
- An ARM restore runner fixes `restore_execution_path=DOCKER_STDIN`. A private
|
||||
Docker proxy may observe the real `pg_restore --list` and apply calls and
|
||||
persist only their stage name and numeric exit code; it must not record argv,
|
||||
environment values or stderr.
|
||||
- Run `dump_list` and target confirmation independently before the existing
|
||||
restore script. A failed stage leaves every later stage `NOT_RUN`.
|
||||
- Measure the eight data-restore stages in this exact continuous order:
|
||||
`batch_copy` -> `postgres_container_start` -> `dump_list` ->
|
||||
`target_confirmation` -> `restore_apply` -> `database_verify` ->
|
||||
`storage_attachment_verify` -> `final_integrity`. One timestamp closes the
|
||||
current stage and starts the next, so no elapsed time is hidden between
|
||||
fields. The total is computed from the measured stage fields and must equal
|
||||
their exact sum.
|
||||
- On failure, finalize the active stage and emit the completed/current numeric
|
||||
durations plus later `NOT_RUN` fields before runtime teardown. A pre-copy
|
||||
failure emits `NOT_RUN` for the total and every stage. Do not emit the total
|
||||
a second time on the success-only summary path.
|
||||
- For the official PostgreSQL image, target readiness must use a real query over
|
||||
container-local TCP loopback (`psql -h 127.0.0.1`), not default Unix-socket
|
||||
`pg_isready`. The initdb temporary server listens only on its socket, while
|
||||
the final server accepts loopback TCP even under `--network none`. Require
|
||||
two consecutive exact `current_database()` matches inside a bounded retry
|
||||
window as additional stability evidence; a successful wrong name fails
|
||||
immediately and a non-zero query resets the streak.
|
||||
- Define target confirmation in the rendered runner and make the contract
|
||||
extract or directly drive that same implementation. A fixture-only wait loop
|
||||
that stabilizes PostgreSQL before testing a different runner path is
|
||||
forbidden: it can make the contract pass while the deployed candidate still
|
||||
contains the startup race.
|
||||
- Raw restore output remains mode `0600`, never leaves the off-host node and is
|
||||
deleted with the owned runtime on success and failure.
|
||||
- Any non-zero pg_restore apply exit is `FAIL/NO_GO`, even when a numeric
|
||||
"errors ignored" count can be extracted. There is no partial-success
|
||||
allowlist. Relaxing that rule requires a separately approved design proving
|
||||
the exact restored object set.
|
||||
- A diagnostic failure is an exercise-runner finding until evidence identifies
|
||||
a dump or restore-content defect. It must not be summarized as proof that the
|
||||
latest backup is unrecoverable.
|
||||
- The tracked runner is a template, not a site-specific script. Remote address,
|
||||
account, paths, container/database names, timer units, build identity and
|
||||
migration fingerprint arrive only through a private JSON with an exact key
|
||||
set. Rendering validates every value's type, charset and safety shape before
|
||||
exclusively creating a candidate; runtime environment variables cannot
|
||||
override those rendered constants.
|
||||
- The exact rendered candidate must pass the tracked contract and preserve a
|
||||
deterministic SHA-256 before upload. A workspace-only runner, hand-edited
|
||||
candidate or contract failure is a hard upload blocker.
|
||||
- Private render JSON and site-specific candidates must live outside the
|
||||
repository or under the dedicated ignored `/.restore-drill-private/`
|
||||
directory. The renderer checks both input and output with Git ignore rules
|
||||
and fails before candidate creation when an in-repository path is not
|
||||
ignored; operator memory is not a control.
|
||||
- The mode-`0600` raw log is truncated exactly once at initialization. Container
|
||||
start, dump list, target confirmation, restore apply and storage restore each
|
||||
append a fixed stage marker and their private output, so a later failure does
|
||||
not erase earlier-stage diagnostics before final teardown.
|
||||
- A source migration fingerprint enumerates only direct directory objects under
|
||||
`server/prisma/migrations` at the source commit, validates every directory
|
||||
name, sorts the names and hashes that exact non-empty set. Files such as
|
||||
`migration_lock.toml` are not migrations and must never enter the fingerprint.
|
||||
- A one-shot installer must install every helper into the exact `LIB_DIR`
|
||||
embedded in the already approved rendered runner, or derive both values from
|
||||
the same validated private render input and compare them before `sudo
|
||||
install`. It must use the same exact destinations for preflight, installation,
|
||||
residue accounting and cleanup; a path that differs only by `lib` versus
|
||||
`libexec` is an entry-contract failure, not an ARM environment exception.
|
||||
- A root/sudo helper must not use `$HOME` to locate a caller-owned upload,
|
||||
installer or diagnostic script: sudo normally changes it to the root home.
|
||||
Self-cleanup may use `$0` only after validating the fixed basename, regular
|
||||
file type and absence of a symlink; otherwise a non-root outer wrapper owns
|
||||
cleanup of the exact caller-side path.
|
||||
- A runner's `temporary_access_cleanup=PASS` or equivalent self-report proves
|
||||
only that the destinations encoded in that runner's cleanup contract are
|
||||
absent. It is not a zero-residue claim: an installer can place an object
|
||||
outside that destination set and the runner cannot observe it. Before an
|
||||
attempt is called clean, an independent post-run audit must enumerate the
|
||||
installer manifest, runner/probe entries, sudoers candidates, intended and
|
||||
rejected dependency destinations, user staging, owned runtime and container
|
||||
identities; every count must be zero and the audit must clean itself.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Dump list non-zero -> `dump_list=FAIL`, later stages `NOT_RUN`, category
|
||||
`DUMP_LIST_NONZERO` / `dump_list_failed`.
|
||||
- Bounded TCP target queries remain non-zero -> `target_confirmation=FAIL`,
|
||||
apply `NOT_RUN`, category `TARGET_QUERY_NONZERO`; preserve the last numeric
|
||||
exit, attempt count and zero success streak. All query retries and their
|
||||
one-second intervals accrue only to `rto_target_confirmation_seconds`.
|
||||
- Target query succeeds but the exact shadow name differs -> fail immediately
|
||||
with `target_name_match=no`, category `TARGET_NAME_MISMATCH`; do not retry a
|
||||
confirmed wrong target.
|
||||
- Independent list and target pass, but the restore script never invokes apply
|
||||
-> category `APPLY_EXIT_CODE_MISSING`; fail closed.
|
||||
- Apply exit non-zero -> `restore_apply=FAIL`, preserve the exact numeric exit
|
||||
and safe numeric/`UNKNOWN` error count, category `RESTORE_APPLY_NONZERO`.
|
||||
- Apply and script exit zero -> `restore_apply=PASS`; later migration, table and
|
||||
attachment checks still decide the final recovery result.
|
||||
- Diagnostics emitted only after cleanup, or cleanup residue non-zero -> the
|
||||
attempt is non-clean and cannot be reported as `GO`.
|
||||
- A measured stage is negative/non-numeric, a completed stage is `NOT_RUN`, a
|
||||
later stage is numeric after an earlier failure, or the stage sum differs
|
||||
from `data_restore_rto_seconds` -> runner contract failure; do not render or
|
||||
upload the candidate.
|
||||
- Missing/extra render key, newline/shell syntax, unsafe path/name, malformed
|
||||
digest or pre-existing output -> render fails without publishing a partial
|
||||
candidate.
|
||||
- In-repository private config or candidate path not covered by Git ignore ->
|
||||
render fails before the candidate exists. The dedicated ignored directory
|
||||
remains valid for a locally protected render.
|
||||
- Any raw-log truncating redirect after initialization, or a missing fixed stage
|
||||
marker -> tracked contract failure; do not publish a candidate.
|
||||
- A source-tree listing that mixes migration directories with ordinary files ->
|
||||
reject the generator result or filter by Git directory object type before
|
||||
hashing; never reinterpret the extra file as schema drift.
|
||||
- Candidate differs from the deterministic render or has not passed the tracked
|
||||
contract -> do not upload or install it.
|
||||
- Installer dependency destination differs from the rendered runner `LIB_DIR`
|
||||
-> fail before the first privileged file write; do not install into the
|
||||
similar sibling directory and rely on runner cleanup to discover it later.
|
||||
- A sudo helper resolves caller staging through root `$HOME`, or cannot prove
|
||||
the fixed identity of `$0` -> leave caller-side cleanup to the non-root outer
|
||||
wrapper and report it explicitly; do not guess or unlink a different home.
|
||||
- Runner cleanup reports `PASS` but an installer/outer residue count is non-zero
|
||||
or not independently measured -> the attempt remains non-clean; preserve the
|
||||
runner self-report as contract-scoped evidence and do not rewrite it as zero
|
||||
residue.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: dump list and target pass, apply exits `23`; fixed fields show
|
||||
`DOCKER_STDIN`, `23` and `RESTORE_APPLY_NONZERO` before teardown, then cleanup
|
||||
reports zero residue without exposing the raw error.
|
||||
- Base: all three stages pass with exit code `0`; the runner continues to
|
||||
migration and attachment verification.
|
||||
- Good: Unix-socket availability alone does not pass; loopback TCP becomes
|
||||
available, two exact target queries succeed consecutively, then apply runs.
|
||||
- Good: an apply failure records numeric copy/container/list/target/apply
|
||||
durations, leaves database/storage/final verification `NOT_RUN`, emits their
|
||||
exact partial sum, and only then tears down the private runtime.
|
||||
- Base: a successful run emits all eight numeric durations once and their sum
|
||||
exactly equals `data_restore_rto_seconds`; the value remains a data-only
|
||||
exercise RTO, not a complete OA recovery RTO.
|
||||
- Bad: emit only `database_restore_failed` after deleting the sole private log.
|
||||
- Bad: parse and publish arbitrary pg_restore lines, or treat a non-zero exit as
|
||||
success because the restored database appears partly populated.
|
||||
- Bad: commit one host's concrete runner or reconstruct a supposedly equivalent
|
||||
runner from chat/workspace immediately before an exercise.
|
||||
- Bad: add two successful queries only to the fixture while the rendered runner
|
||||
still accepts the first socket `pg_isready`, or use a fixed sleep as an ARM
|
||||
timing workaround.
|
||||
- Good: the installer mechanically compares its dependency destination with the
|
||||
rendered runner `LIB_DIR`, installs all three pinned helpers there and checks
|
||||
that same exact directory during residue accounting.
|
||||
- Bad: type a site path twice, install helpers below `lib` while the candidate
|
||||
reads `libexec`, or use sudo `$HOME` to delete a script uploaded by the
|
||||
calling user.
|
||||
- Good: runner cleanup reports its contract destinations clean, then a separate
|
||||
fixed-target audit independently returns zero for entries, both intended and
|
||||
previously rejected dependency directories, staging, runtime and containers.
|
||||
- Bad: summarize `temporary_access_cleanup=PASS` as “all temporary objects are
|
||||
zero” without checking objects that only the installer knew how to create.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Directly drive the repository `scripts/restore-db.sh` with a real custom dump
|
||||
through Docker stdin and host-file execution paths; do not replace the script
|
||||
with an equivalent test implementation.
|
||||
- Inject list failure, target mismatch and list-pass/apply-failure. Assert exact
|
||||
stage enums, later-stage `NOT_RUN`, execution path and numeric exit code.
|
||||
- Drive the rendered runner's timing functions with deterministic timestamps.
|
||||
Assert all eight successful stage durations and their exact total; inject
|
||||
target-confirmation and restore-apply failures and assert completed/current
|
||||
numeric values, later `NOT_RUN`, one total field and timing output before the
|
||||
runtime cleanup marker.
|
||||
- Statically assert that the bounded TCP confirmation call is enclosed by the
|
||||
`dump_list -> target_confirmation` and
|
||||
`target_confirmation -> restore_apply` timing transitions; a retry sleep
|
||||
must not be charged to container start or restore apply.
|
||||
- Extract the target-confirmation function from the rendered runner and drive
|
||||
it with: socket-only availability/TCP failure, first-query recovery, streak
|
||||
reset, exact-name mismatch and permanent bounded failure. Assert every probe
|
||||
contains `-h 127.0.0.1`, name mismatch is not retried, and the real
|
||||
PostgreSQL 16 fixture uses this same function before both restore paths.
|
||||
- Inject a non-zero apply exit and assert the diagnostic is emitted before
|
||||
teardown, teardown passes, and the calling test can still read the fixed
|
||||
fields afterwards.
|
||||
- Scan captured diagnostics for fixture credentials, connection strings, SQL,
|
||||
temporary paths and free-form tool output.
|
||||
- Run the existing synthetic paired backup/restore and automatic off-host
|
||||
contract suites after changing the diagnostic runner.
|
||||
- Render the same private fixture twice and require byte equality; inject shell
|
||||
syntax and require failure before the output file exists.
|
||||
- Assert an unignored in-repository private config and an unignored candidate
|
||||
path both fail before output creation; assert the dedicated ignored directory
|
||||
accepts synthetic input and output without becoming trackable.
|
||||
- Statically require exactly one raw-log truncation at initialization and fixed
|
||||
append markers for container start, dump list, target confirmation, restore
|
||||
apply and storage restore.
|
||||
- Build a source-tree fixture containing migration directories plus
|
||||
`migration_lock.toml`; assert the fingerprint includes only sorted directory
|
||||
names and still matches the restored successful migration set.
|
||||
- Directly test source/copy identity, dynamic migration fingerprint and the
|
||||
tracked attachment validator. Run `check-sensitive-content.mjs` explicitly on
|
||||
all three tracked runner assets and require zero findings.
|
||||
- Build an installer fixture from a rendered candidate and assert runner entry,
|
||||
exact `LIB_DIR`, dependency basenames and sudoers destination are one mapping.
|
||||
Mutate only `libexec` to `lib` and require failure before any simulated
|
||||
privileged install; verify residue accounting uses the same destinations.
|
||||
- Exercise a diagnostic helper with sudo/root `HOME`: `$0` plus fixed-basename,
|
||||
regular-file and non-symlink checks may self-clean, while an unverified path
|
||||
must remain owned by a non-root outer cleanup. Assert no root-home path is
|
||||
inferred or emitted.
|
||||
- Inject an installer destination outside the runner cleanup set. Assert runner
|
||||
cleanup may truthfully pass for its own targets while the independent audit
|
||||
reports non-zero and blocks a clean conclusion; after exact cleanup, rerun
|
||||
the audit and require all enumerated counts plus audit self-cleanup to pass.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```text
|
||||
fixture: wait for two queries
|
||||
runner: first socket pg_isready -> continue
|
||||
restore-db > private.log || fail database_restore_failed
|
||||
RTO = finish - start; stage gaps are inferred later
|
||||
finally: delete private runtime
|
||||
print result=NO_GO
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```text
|
||||
tracked template + exact private config -> deterministic candidate + SHA-256
|
||||
candidate LIB_DIR == installer dependency destination -> privileged install
|
||||
contract drives rendered TCP target function -> list -> target -> real restore path
|
||||
continuous stage transitions -> exact stage sum + one data-only RTO
|
||||
capture only stage/path/exit/category/duration allowlists
|
||||
finalize active stage -> emit fixed diagnostics -> finally teardown raw private runtime
|
||||
sudo helper: validated $0 self-cleanup or exact non-root outer cleanup; never sudo $HOME
|
||||
runner cleanup PASS + independent fixed-target zero-residue audit -> clean attempt
|
||||
non-zero apply remains FAIL
|
||||
```
|
||||
|
||||
## Scenario: Daily Restricted Off-Host Backup
|
||||
|
||||
### 1. Scope / Trigger
|
||||
@@ -2417,3 +2805,213 @@ current shared mapping -> fine-grained stages + mutation listeners
|
||||
-> zero sample means dependent checks PAUSED, not auth failure
|
||||
-> reconcile session/audit side effects -> post-window snapshot
|
||||
```
|
||||
|
||||
## Scenario: Read-Only Automatic Backup Chain Investigation
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
- Trigger: the latest off-host archive exceeds its RPO/freshness gate, an
|
||||
automatic-backup or dead-man service fails, or a restore drill discovers a
|
||||
stale latest batch.
|
||||
- Purpose: distinguish scheduler failure, source production failure, interrupted
|
||||
transfer, finalize failure and alert-delivery behavior without triggering a
|
||||
backup, changing a timer, reading business contents or modifying an archive.
|
||||
|
||||
### 2. Signatures
|
||||
|
||||
- Production units: `hyb-oa-automatic-backup-daily.service/.timer` and
|
||||
`hyb-oa-automatic-backup-watchdog.service/.timer`.
|
||||
- ARM dead-man unit basename is the installed `DEADMAN_SERVICE` contract from
|
||||
`deploy/trialrun/install-offhost-backup-receiver.sh`; probes must derive or
|
||||
mechanically compare that value instead of typing a remembered unit name.
|
||||
- Safe state fields: unit `ActiveState`, `Result`, `ExecMainStatus`, timer last
|
||||
trigger, effective max-age seconds, summary `status/category/findingCount`,
|
||||
alert state `status/category`, journal `evaluated sent=true|false` counts,
|
||||
archive/partial counts and age aggregates, and filesystem percent/free GiB.
|
||||
|
||||
### 3. Contracts
|
||||
|
||||
- A timer being `active` proves scheduling remains enabled; it does not prove
|
||||
its last service run passed. Always pair timer state/trigger with service
|
||||
`Result` and exit status.
|
||||
- Discover the remote account's actual capabilities before installing a probe.
|
||||
The least-privilege production deployment account is not assumed to have
|
||||
sudo; a production probe must run unprivileged or stop with an explicit
|
||||
authority requirement. It must never attempt to widen sudoers itself.
|
||||
- File absence and access denial are different evidence. Before reporting a
|
||||
root-owned state file as missing, prove the parent directory is traversable
|
||||
and readable; otherwise emit `ACCESS_DENIED` and keep the fact unknown.
|
||||
- Classify receiver inbox objects without outputting names or paths. A partial
|
||||
with the exact three regular files, valid manifest/run identity and matching
|
||||
artifact byte counts indicates transfer completion followed by missing or
|
||||
failed finalize. A partial that fails that contract indicates transfer did
|
||||
not complete; SHA-256 may remain uncomputed in this diagnostic classification.
|
||||
- A receiver partial proves that the transfer authenticated and opened its
|
||||
destination before interruption. Initial private-key rejection,
|
||||
`authorized_keys` rejection and host-key refusal cannot create that object;
|
||||
keep investigation on mid-transfer network/session limits, source read,
|
||||
receiver write and transport I/O. A later SSH-layer disconnect may still
|
||||
surface as rsync/SSH failure and is not excluded by successful startup.
|
||||
- For multiple incomplete partials, compare only aggregate byte totals/ratios,
|
||||
the wrapped rsync exit code and elapsed transfer seconds. Similar stopping
|
||||
offsets or durations indicate a deterministic limit; materially different
|
||||
offsets/durations favor intermittent timeout or network interruption. Do not
|
||||
infer this split from service exit `1`, which is the wrapper's generic code.
|
||||
- A stopping-offset pair is `similar` only when its absolute difference is no
|
||||
more than both `50000 ppm` (5% of the larger aggregate) and `67108864` bytes
|
||||
(64 MiB), equivalently `difference <= min(relative_limit, absolute_limit)`.
|
||||
Using `max(...)` is forbidden: it misclassifies a small nonzero partial and a
|
||||
zero-byte partial as similar merely because both are below 64 MiB.
|
||||
- `last-deadman.json` proves the detector decision. `deadman-alert-state.json`
|
||||
proves the deduplication state. A journal allowlist line with
|
||||
`evaluated sent=true` proves the helper attempted and accepted one send; it
|
||||
does not prove a human noticed the message. Repeated `sent=false` for the same
|
||||
failure is normal deduplication, not a monitoring blind spot.
|
||||
- Keep the two JSON schemas distinct: the dead-man summary contains
|
||||
`status/category/findingCount/findings`, while the alert deduplication state
|
||||
contains only `schemaVersion/status/category`. A probe must not request
|
||||
`findingCount` from the alert state or report its contractual absence as an
|
||||
invalid remote state.
|
||||
- Read-only investigation output is an allowlist of status/category/count/age
|
||||
and capacity fields. Never print unit command lines, raw JSON, raw journal,
|
||||
remote identities, paths, run ids, webhook data or diagnostic stderr.
|
||||
- Before classifying an off-host transport failure, validate the newest source
|
||||
batch with the same manifest schema/status, exact three-file set, run-id
|
||||
pairing, byte-count and SHA-256 contract used by the receiver. Report only
|
||||
whether a complete local recovery point exists, its age/count/aggregate
|
||||
bytes and one of `production_single_point` or `no_proven_recovery_point`.
|
||||
Off-host staleness alone does not distinguish those two materially different
|
||||
production risks.
|
||||
- On an online production host, enumerate older local batches with manifest,
|
||||
file-set, identity and byte metadata only. Fully hash exactly the newest
|
||||
metadata-valid candidate; report older hashes as `NOT_RUN`. If that latest
|
||||
candidate fails SHA-256, keep `no_proven_recovery_point` and do not fall back
|
||||
to multi-GiB reads of older candidates during the same probe. ARM archive
|
||||
validation remains full-contract because it is the backup node.
|
||||
- Inspect cleanup semantics as executable ordering, not a remembered claim.
|
||||
Source publication must precede transfer; local pruning must follow verified
|
||||
off-host finalize and success-state publication; generation staging must be
|
||||
protected by an exit cleanup. Also report current valid/invalid/staging
|
||||
counts so repeated transfer failures cannot silently consume source space.
|
||||
- A retained `last-error.log` may provide an rsync code only when the matching
|
||||
safe run summary says `phase_offhost_transfer_failed`. Never parse a log from
|
||||
another phase or substitute service exit `1`. If the wrapper did not persist
|
||||
transfer start/end clocks, emit `NOT_AVAILABLE`; service duration, mtime and
|
||||
conversation time are not transfer-duration evidence.
|
||||
- A read-only inspector must prevalidate every output as a fixed key plus an
|
||||
enum/integer value. Malformed manifests, unreadable files and unexpected
|
||||
runtime exceptions collapse to stable categories without a traceback,
|
||||
because tracebacks can expose absolute paths even when normal output is
|
||||
redacted.
|
||||
- If existing read-only state cannot converge on a cause, stop expanding the
|
||||
read surface. The next step is a separately approved, single-attempt
|
||||
observational retransmit with frozen input, fixed telemetry, bounded timeout
|
||||
and no automatic retry. It is a diagnostic write, not part of read-only
|
||||
authority.
|
||||
- After a transport repair, one fresh atomic batch may unlock a blocked restore
|
||||
drill, but incident closeout requires two consecutive natural schedules.
|
||||
Each cycle records source and receiver capacity plus receiver partial count
|
||||
and aggregate bytes; a manual run does not replace either natural sample.
|
||||
|
||||
### 4. Validation & Error Matrix
|
||||
|
||||
- Timer inactive or never triggered -> scheduler/configuration branch.
|
||||
- Timer triggered and service failed before a new local batch is visible ->
|
||||
source/common-preflight branch; if root state is inaccessible, preserve the
|
||||
exact phase as unknown instead of guessing.
|
||||
- New incomplete receiver partial -> `offhost_transfer_incomplete`; do not call
|
||||
it a finalize failure.
|
||||
- Partial exists -> initial SSH authentication/host-key acceptance succeeded;
|
||||
do not spend the first incident cycle rotating keys or rewriting
|
||||
`authorized_keys` without contrary evidence.
|
||||
- Receiver partial satisfies the complete byte/manifest contract but no atomic
|
||||
archive exists -> `offhost_finalize_not_completed`.
|
||||
- Atomic archive stale + dead-man state stale + `sent=true` -> detector and send
|
||||
invocation worked; investigate human/channel reception separately.
|
||||
- Atomic archive stale but dead-man state fresh/missing, service not run, or
|
||||
timer failed -> monitoring execution blind spot.
|
||||
- Path cannot be traversed by the approved account -> `ACCESS_DENIED`; request
|
||||
separate read authority only if the unresolved field materially changes the
|
||||
decision.
|
||||
- Complete source batch valid + off-host point stale -> production has a local
|
||||
recovery point but remains a single point; prefer the next safe natural run
|
||||
unless a separately approved transition run is justified.
|
||||
- No source batch passes the complete contract -> `no_proven_recovery_point`;
|
||||
raise the transition-recovery-point priority, but do not trigger a backup
|
||||
until root cause, capacity and stop conditions are approved.
|
||||
- `last-error.log` contains rsync-like text but the safe last-run phase is not
|
||||
`offhost_transfer` -> exit code remains `NOT_AVAILABLE` rather than attaching
|
||||
stale diagnostics to the current failure.
|
||||
|
||||
### 5. Good/Base/Bad Cases
|
||||
|
||||
- Good: daily timer triggered, service failed, ARM has a recent incomplete
|
||||
partial, dead-man records `archive_stale` and the alert helper records one
|
||||
`sent=true`; conclude transfer interruption plus successful detector/send
|
||||
invocation while keeping the production raw phase unknown.
|
||||
- Base: latest archive is fresh, services pass, inbox has zero partials and the
|
||||
alert state is healthy.
|
||||
- Bad: infer “timer active means backup succeeded”, call an inaccessible state
|
||||
file missing, or describe any `.partial` as finalize failure without checking
|
||||
whether its three-file byte/manifest contract completed.
|
||||
- Bad: hard-code a near-match dead-man unit name in a one-off probe and treat an
|
||||
empty `systemctl show` response as a parser-format problem.
|
||||
|
||||
### 6. Tests Required
|
||||
|
||||
- Extract the receiver dead-man unit basename from the installer and compare it
|
||||
with every rendered probe before upload; a near-match unit must fail locally.
|
||||
- Fixture systemd output for active timer/pass service, active timer/failed
|
||||
service, nonexistent unit and explicit max-age versus script-default max-age.
|
||||
- Fixture root-owned state directories and assert an unprivileged probe emits
|
||||
`ACCESS_DENIED`, never `present=no`.
|
||||
- Fixture incomplete partial, complete-contract partial and archive transaction
|
||||
temporary entries; assert only aggregate classifications and ages are emitted.
|
||||
- Fixture two partials with equal and unequal aggregate stopping offsets plus
|
||||
rsync codes `23/30/12/11/255`; assert the incident summary preserves code and
|
||||
elapsed-time evidence separately from wrapper exit `1` and does not output
|
||||
file names.
|
||||
- Assert `2.68 MB` versus zero is `different`, a large pair within both 5% and
|
||||
64 MiB is `similar`, and exceeding either threshold is `different`.
|
||||
- Fixture one complete source batch, no complete source batch, checksum/size
|
||||
corruption and malformed manifest. Assert the risk classification and
|
||||
transition recommendation change without emitting paths, run ids, artifact
|
||||
names, raw diagnostics or tracebacks.
|
||||
- Fixture two metadata-valid production batches with a same-size hash mutation
|
||||
in the older and then newest candidate. Assert only the newest candidate is
|
||||
fully hashed, older hashes remain `NOT_RUN`, and a newest-candidate hash
|
||||
failure does not trigger fallback reads.
|
||||
- Mechanically mutate the wrapper ordering so pruning appears before transfer;
|
||||
the cleanup/retention contract must turn false. A prose claim that failure
|
||||
retains data is not sufficient regression evidence.
|
||||
- Sensitive scan the probe and assert raw `ExecStart`, journal, JSON, paths,
|
||||
remote identities and run ids cannot appear in stdout.
|
||||
- Every root probe must self-remove its fixed entry/sudoers and an independent
|
||||
non-root wrapper must report entry and staging residue separately.
|
||||
|
||||
### 7. Wrong vs Correct
|
||||
|
||||
#### Wrong
|
||||
|
||||
```text
|
||||
timer active -> backup healthy
|
||||
state path test returned false -> state missing
|
||||
partial count > 0 -> finalize failed
|
||||
off-host stale -> assume one complete backup still exists on production
|
||||
service exit 1 -> call it the rsync exit code and use service duration
|
||||
partial difference <= max(5% limit, 64 MiB) -> similar
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```text
|
||||
derive unit identity from installer -> timer trigger + service Result/exit
|
||||
-> distinguish ACCESS_DENIED from missing
|
||||
-> partial three-file/manifest/byte classification
|
||||
-> partial proves initial SSH acceptance; compare byte offset + rsync code + duration
|
||||
-> dead-man decision + dedup state + sent flag
|
||||
-> safe scheduler/source/transfer/finalize/alert conclusion
|
||||
-> validate the source batch first and classify single-point vs no proven point
|
||||
-> bind retained diagnostics to the safe failure phase; unavailable clocks stay unavailable
|
||||
-> partial difference <= min(5% limit, 64 MiB) before calling it similar
|
||||
```
|
||||
|
||||
@@ -14,6 +14,22 @@ duplicate detection or attachment behavior.
|
||||
- Reimbursement flow tests should cover upload/OCR, duplicate handling,
|
||||
submission, approval, payment batch, receipt attachment and AP export when the
|
||||
Trellis task scope touches those areas.
|
||||
- Reimbursement object maintenance is limited to the form owner or submitter
|
||||
while the status permits editing. `expense.view_all`, an admin role and an
|
||||
approval assignment grant read or workflow-specific abilities only; they do
|
||||
not authorize editing another employee's form, items, attachments, project
|
||||
link, submission, or reimbursement invoice binding/unbinding.
|
||||
- PC and mobile reimbursement details must project the same boundary. A
|
||||
nonparticipant remains able to read the detail and permitted originals or
|
||||
merged materials, but sees a clearly labeled read-only view with no owner
|
||||
maintenance actions. Bound-invoice helper text must describe facts (for
|
||||
example `已关联` / `未关联`) rather than advertise removal or internal
|
||||
auto-matching behavior to a viewer who cannot perform it.
|
||||
- Reimbursement precheck exposes budget, expense-standard and risk information,
|
||||
so it is an object read even though it uses POST. Resolve the target form and
|
||||
require the same own-or-`expense.view_all` guard as reimbursement detail reads
|
||||
before invoking the precheck service; an ordinary nonparticipant must receive
|
||||
403 without service-side evaluation.
|
||||
|
||||
## Scenario: OCR Disabled Without Mock Data
|
||||
|
||||
@@ -689,10 +705,18 @@ reply.header(
|
||||
- PDF originals contribute every original page; JPG/PNG/WebP materials are
|
||||
normalized with `sharp` and embedded one per page. The merged PDF is a
|
||||
derived approval view, never an original or digital-signature fact source.
|
||||
- Image page geometry must use the width and height returned by Sharp's final
|
||||
normalized output (`toBuffer({ resolveWithObject: true }).info`), not input
|
||||
`metadata()`. `rotate()` applies EXIF orientation during output, so input
|
||||
metadata may describe the opposite portrait/landscape shape.
|
||||
- A stable collection hash covers the explicitly ordered material type, ID,
|
||||
file hash, MIME and byte count. The cache key also includes reimbursement ID
|
||||
and merger version. Fix CreationDate, ModificationDate, Creator, Producer and
|
||||
save options so identical input produces byte-identical output.
|
||||
- Any visible merge-algorithm change, including corrected image page geometry,
|
||||
increments the merger version. Byte determinism is required across repeated
|
||||
generation within one version; different versions may intentionally produce
|
||||
different bytes for the same ordered source collection.
|
||||
- The in-memory bundle is bounded to 20 materials, 50 MiB source bytes, 40
|
||||
pages, 40 megapixels per image, 50 MiB output and 100 MiB process cache.
|
||||
Worker execution has a fixed timeout; concurrent requests for one collection
|
||||
@@ -732,6 +756,7 @@ reply.header(
|
||||
### 6. Tests Required
|
||||
|
||||
- Service tests assert deterministic bytes, all-page copy, image embedding,
|
||||
EXIF orientation 6/8 output geometry, ordinary JPG/PNG/WebP geometry,
|
||||
20–30-page worker behavior, concurrent deduplication, expiry and limits.
|
||||
- Route tests assert authorization, reason gate, token binding, cache miss,
|
||||
issue/access audit, audit-failure fail-closed behavior and exact PDF size.
|
||||
@@ -748,6 +773,12 @@ reply.header(
|
||||
const pages = await rasterizeEveryPdfOnThePhone(); // slow, blank-prone and duplicates PDF rendering
|
||||
```
|
||||
|
||||
```js
|
||||
const metadata = await sharp(source).rotate().metadata();
|
||||
// metadata describes the input and can disagree with the rotated output.
|
||||
const page = output.addPage([metadata.width, metadata.height]);
|
||||
```
|
||||
|
||||
#### Correct
|
||||
|
||||
```tsx
|
||||
@@ -755,6 +786,14 @@ const bundle = await fetchReimbursementMaterialPreviewBundle(reimbursementId);
|
||||
await previewFileInWecom({ url: bundle.signedUrl, name: bundle.originalName, size: bundle.fileSize });
|
||||
```
|
||||
|
||||
```js
|
||||
const { data, info } = await sharp(source)
|
||||
.rotate()
|
||||
.jpeg()
|
||||
.toBuffer({ resolveWithObject: true });
|
||||
const page = output.addPage([info.width, info.height]);
|
||||
```
|
||||
|
||||
## Scenario: Lightweight Travel Summary On Reimbursements
|
||||
|
||||
### 1. Scope / Trigger
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,145 @@
|
||||
# 隔离备份恢复演练设计
|
||||
|
||||
## 设计结论
|
||||
|
||||
采用“ARM 最新正式日归档 + 本轮私有临时副本 + 无宿主端口的一次性 PostgreSQL 16 + 空附件目录”的数据恢复方案。腾讯云生产只做演练前后公网 `/ready` 读取,不执行任何远端生产写操作;开发机不接收真实备份。
|
||||
|
||||
## 环境选择
|
||||
|
||||
| 方案 | 数据移动与风险 | 对生产影响 | 结论 |
|
||||
| --- | --- | --- | --- |
|
||||
| ARM 一次性影子恢复 | 备份已在 ARM,只在同机复制到私有临时区;明文恢复范围最小 | 无生产容器操作 | 采用 |
|
||||
| 开发机本地 Docker | 需要把真实 DB/附件下载到个人开发机,扩大敏感数据副本 | 无运行影响,但数据治理风险高 | 不采用 |
|
||||
| 腾讯云同机影子 Compose | 不需跨机下载,但会在真实生产宿主创建容器、目录和资源负载 | 生产宿主存在误操作和资源竞争风险 | 不采用 |
|
||||
|
||||
ARM 在本设计中仍是备份与恢复演练节点,不是生产部署目标。一次性 PostgreSQL 不发布端口;本轮不启动 OA server。
|
||||
|
||||
## 数据流与信任边界
|
||||
|
||||
```text
|
||||
ARM 正式日归档(只读)
|
||||
│ 选择最新 complete 批次并校验
|
||||
▼
|
||||
本轮 0700 临时副本
|
||||
├── database.dump ──> 一次性 PostgreSQL 16 影子库 ──> migration/表计数检查
|
||||
└── storage.tar.gz ─> 本轮空附件目录 ───────────────> 文件数/引用/哈希检查
|
||||
│
|
||||
▼
|
||||
仅数量与状态的脱敏证据
|
||||
```
|
||||
|
||||
正式归档不被解包、改名、删除或直接用作写入目标。临时副本、影子数据库和恢复附件目录都由本轮唯一 run id 标识,清理只允许命中该标识。
|
||||
|
||||
## 恢复点合同
|
||||
|
||||
1. 从正式日归档中按 manifest `createdAt` 选择最新批次,而不是按目录 mtime 猜测。
|
||||
2. 候选目录只能包含三个普通文件;manifest 必须为 schema v1、`complete`,并通过 run id、文件名、字节数和 SHA-256 自校验。
|
||||
3. `buildSha` 必须唯一解析为本仓库已知提交,并且是演练开始时公网 `/ready` 实测当前构建的祖先或相同;同时保留 source image digest 是否存在的布尔证据,不输出完整 digest。相等状态仅用于度量发布后何时形成同源恢复点,不再决定原始备份是否可恢复。
|
||||
4. 演练开始时批次年龄不得超过 24 小时。候选不合格直接 `HOLD`,不自动降级到旧批次。
|
||||
5. 临时副本复制完成后再次验证三文件和 manifest,证明恢复输入与正式归档一致。
|
||||
|
||||
目录身份校验只适用于 ARM 正式归档源目录:其 basename 必须与 manifest `runId` 配对。临时副本允许使用本轮固定私有名称;对临时副本必须继续验证 manifest 合同、run id 格式、三文件集合、字节数、SHA-256、构建和时间,并与源目录首次校验得到的脱敏元数据完全一致,但不得要求临时目录 basename 冒充正式归档名称。
|
||||
|
||||
## 隔离恢复合同
|
||||
|
||||
### 数据库
|
||||
|
||||
- 使用随机一次性容器名和含 `drill` 的数据库名;容器无 `ports`,不加入现有业务网络,不挂载生产或 ARM 归档目录。
|
||||
- 使用 PostgreSQL 16,与现有恢复基线和脚本一致。
|
||||
- 调用 `scripts/restore-db.sh`,同时满足 `RESTORE_SCOPE=shadow`、精确数据库确认和显式 Docker 容器身份。
|
||||
- 恢复后只运行 `SELECT`:migration 成功/失败状态、固定关键表行数和附件引用聚合。不执行 Prisma migrate、seed、DDL 或修复语句。
|
||||
- 从 manifest 来源提交动态取得 migration 目录名称集合,与恢复库成功且未回滚的 `_prisma_migrations.migration_name` 做精确集合核对;不硬编码数量。再静态比较来源提交与当前构建的 migration 集合,记录“相同”或“当前有待升级 migration”。
|
||||
|
||||
固定 12 类表为:部门、员工、员工角色、采购申请、报销单、发票池、付款批次、合同、项目、采购附件、报销附件、付款回单附件。本次已知来源和当前构建现场均为 65 个 migration,但执行器必须使用动态名称集合,不能把 65 写成长期断言。
|
||||
|
||||
### 附件
|
||||
|
||||
- 调用 `scripts/restore-storage.sh` 恢复到不存在或为空的本轮目录。
|
||||
- 恢复前验证 tar 可读、无绝对路径、无 `..`、无软/硬链接;恢复后比较普通文件数。
|
||||
- 对使用本地存储的发票、采购、报销、付款回单和请假附件记录,只在影子库内读取 `storage_path`、大小和哈希用于机器校验,输出仅保留总数和不一致数。
|
||||
- `s3:` 引用不属于当前 tar 恢复范围;若最新备份中出现则 `FAIL` 并登记类别,不尝试访问外部对象存储。
|
||||
|
||||
## RTO/RPO 口径
|
||||
|
||||
- `RPO = drill_started_at - manifest.createdAt`。它表示使用该最新批次恢复时最多回退的数据时间窗口。
|
||||
- `data_restore_RTO = all_data_checks_passed_at - restore_copy_started_at`。起点位于全部只读前置门禁通过后、复制正式批次到临时工作区之前;终点为 DB、附件及关联检查全部通过。
|
||||
- `data_restore_RTO` 不只输出总秒数,还必须按 runner 的实际串行边界输出八个非负整数阶段:`batch_copy`、`postgres_container_start`、`dump_list`、`target_confirmation`、`restore_apply`、`database_verify`(migration 与 12 类表)、`storage_attachment_verify`(附件恢复、文件数与业务关联)和 `final_integrity`(源/副本末次身份复核)。总数应与八段之和一致;若因秒级取整存在误差,差值必须为 `0`,不另设无法归属的隐藏开销。
|
||||
- 阶段计时以 runner 内连续边界为准:每一段结束时立即关闭本段并开始下一段。最长 60 秒的 TCP 目标确认轮询只计入 `target_confirmation`;1 秒轮询粒度是当前演练器实现成本,不与实际 restore apply 混算。失败时已完成阶段输出实测秒数,当前失败阶段输出从其起点到失败的秒数,未运行阶段为 `NOT_RUN`。
|
||||
- 阶段分解用于区分批次复制、容器与演练器等待、数据库/附件实际恢复及一致性校验成本;它们仍共同属于“数据恢复演练 RTO”。本轮未启动 OA 应用、未验证登录或业务 smoke,因此总数和任何阶段都不得称为完整 OA 灾难恢复 RTO,也不形成 SLA。
|
||||
- `same_build_recovery_point_lag` 的正式口径需要可审计的发布完成时刻与 ARM 原子归档完成时刻;当前证据不具备后者,因此不得输出精确正式值。内部诊断可记录 `manifest_creation_after_contract_dropin = manifest.createdAt - automatic-backup service 构建 drop-in 最大 mtime`,但必须明确它既不是应用发布时刻,也不是 ARM 到达时刻。首次严格构建门禁失败仍证明发布后存在一段没有当前构建同源恢复点的窗口;该画像独立于数据 RPO 和数据恢复 RTO,即使恢复没有开始也保留。
|
||||
- 前置访问准备、等待人工输入和任务文档整理不计入 RTO;镜像拉取、批次复制、实际恢复和验证命令计入。
|
||||
- 本轮不启动应用,因此不得称为“完整 OA 恢复 RTO”;生产 `/ready` 只证明演练未影响在线服务。
|
||||
|
||||
## 状态判定
|
||||
|
||||
- `GO`:AC1–AC7 全部满足,可证明最新成对数据备份按其来源 schema 可恢复;若来源与当前 migration 集合不同,必须同时标注当前 schema 升级未验证,不能宣称完整灾难恢复 GO。
|
||||
- `PAUSE`:访问权限、磁盘空间、Docker/PostgreSQL 16 运行条件或清理验证不足,但未发现备份内容损坏。
|
||||
- `NO_GO`:最新批次超龄/损坏、来源提交未知或不是当前构建祖先/相同、DB restore 失败、来源 migration 集合或关键表异常、附件缺失/哈希不一致、DB 与附件不配对,或生产受到影响。
|
||||
|
||||
重试不得覆盖首次失败证据。若修正环境性问题后重试,必须使用新 run id 并保留首次脱敏 finding。
|
||||
|
||||
## Attempt 1 后的只读身份探针
|
||||
|
||||
首次恢复入口因最新 manifest `build_mismatch` 失败关闭后,不直接重装恢复入口。该失败证据保留,但严格相等门禁已被证明会在发布后自然备份到达前产生必然假阻断。后续先安装一个固定无参数、运行后自删除的 ARM root-owned 只读探针,权限仅用于读取 receiver 的路径配置和最新正式日归档:
|
||||
|
||||
- 按 manifest `createdAt` 选择最新正式日批次,不自动改选旧批次。
|
||||
- 只输出 manifest 构建短标识、与当前生产短标识的相等状态、批次年龄、run id/目录配对状态、source image digest 是否存在,以及两份产物的文件类型、大小和 SHA-256 聚合校验状态;祖先关系在本地仓库只读验证,不要求 ARM 持有 Git 历史。
|
||||
- 不输出完整 run id、完整 digest、完整路径、字节数、文件名之外的业务信息或原始 manifest。
|
||||
- 不复制、不解包、不启动 Docker、不读取数据库内容或附件内容,不修改正式归档。
|
||||
- 成功或失败均删除探针自身和对应 sudoers;普通用户暂存文件在 SHA-256 精确匹配后另行清理。
|
||||
|
||||
探针结果不能替代恢复验收。最新 manifest 与当前生产不相等时,先判断其是否为当前构建祖先:是祖先且批次完整、新鲜时可进入原始恢复重试审批,同时记录同源恢复点滞后;未知或非祖先才维持 `NO_GO` 并转入备份链调查。手工触发生产备份始终是可撤回备选项,不是完成项。
|
||||
|
||||
## Attempt 3 后的数据库恢复诊断合同
|
||||
|
||||
Attempt 3 只留下 `database_restore_failed` 聚合类别,失败清理同时删除了权限 `0600` 的私有原始日志。删除潜在敏感日志是正确安全边界,但下一轮 runner 必须在清理前生成只含 allowlist 状态的阶段诊断,不能再次盲重试。
|
||||
|
||||
- 数据库恢复拆成 `dump_list`、`target_confirmation`、`restore_apply` 三个独立阶段;每个阶段只输出 `PASS`、`FAIL` 或 `NOT_RUN`,并保留稳定错误类别,不输出 pg_restore 原始 stderr、对象名、路径、凭据或 SQL。诊断必须在失败路径进入清理前写到标准输出,清理仍由 trap/finally 等价路径保证执行;输出先于清理不表示保留私有原始日志。
|
||||
- `dump_list` 使用与实际恢复相同的 PostgreSQL 16 容器和临时副本,只解析 custom dump 目录,不执行数据库写入。失败时分类为 `dump_list_failed` 并停止。
|
||||
- `target_confirmation` 只确认当前数据库名等于含 `drill` 标记的本轮目标,并确认容器连接可用;失败时分类为 `restore_target_confirmation_failed`。
|
||||
- 只有前两项通过才允许调用现有 `restore-db.sh`。调用必须记录执行路径枚举 `DOCKER_STDIN` 或 `HOST_FILE`、脚本退出码和真实 pg_restore apply 退出码;ARM runner 固定使用 `DOCKER_STDIN`,两条路径都由本地集成回归直接驱动仓库中的 `scripts/restore-db.sh`。此后非零分类为 `restore_apply_failed_after_list_pass`,证明 dump 可解析但实际应用失败;它仍不能自动等同于备份内容损坏。
|
||||
- 对 pg_restore 非零不设置“可忽略对象错误”白名单:即使私有日志末尾可脱敏提取到对象错误计数,任何非零仍为 `FAIL/NO_GO`。计数只输出非负整数或 `UNKNOWN`,不得据此把部分恢复升级为 `PASS`;未来若要允许部分成功,必须另行列出错误类别、证明对象集合完整并重新批准设计。
|
||||
- 私有原始日志继续保持 `0600` 且不得复制出 ARM;成功或失败都随本轮 runtime 删除。清理前将三项阶段状态写入最终标准输出,保证失败原因至少可定位到合同边界。
|
||||
- 合成回归必须分别制造 dump 列表失败、目标确认失败和列表通过但 apply 失败,并断言后续阶段保持 `NOT_RUN`、输出不含 fixture Secret/路径/SQL、容器与 runtime 清理为零;至少一条注入失败用例必须同时证明诊断在清理前已输出、清理完成后该诊断仍可从调用方捕获。成功回归直接以 PostgreSQL 16 custom dump 驱动真实 `scripts/restore-db.sh` 的 Docker stdin 与宿主文件两条路径,不重写等价恢复逻辑。
|
||||
- 该修复只改变演练器诊断能力,不放宽正式归档、源/副本身份、24 小时新鲜度、shadow 数据库、无端口、附件或清理门禁。任何 Attempt 4 实际恢复仍须使用新 run id、重新探测最新批次并另行批准。
|
||||
|
||||
## Attempt 4 前的 runner 持久化门禁提案
|
||||
|
||||
个人 `.trellis/workspace/` 中的 runner 和回归不会提交,不能作为 Attempt 4 的可重复证据。该限制不接受;实际恢复前应在另行批准的本地代码阶段完成以下持久化,不与 ARM 探针或恢复操作混在一起:
|
||||
|
||||
- 将当前一次性恢复 runner 收敛为受版本控制、只含占位符和校验逻辑的 `scripts/qa/offhost-shadow-restore-runner.template.sh`,将附件关联校验器作为 `scripts/qa/validate-offhost-shadow-restore-attachments.py`。Attempt 4 候选必须由合同脚本使用私有 JSON 参数机械渲染,并与现有 `scripts/restore-db.sh`、`scripts/restore-storage.sh` 一起打包;不得从聊天或个人 workspace 重新生成等价实现。
|
||||
- 新增 `scripts/qa/run-offhost-shadow-restore-runner-contract.sh`,直接校验候选 runner 的 Bash 语法、依赖 SHA-256、固定无参数入口、源/副本身份重绑、动态 migration 集合、三阶段脱敏诊断、退出码、失败先诊断后清理及敏感输出边界。
|
||||
- 合同脚本直接以 PostgreSQL 16 custom dump 驱动真实 `scripts/restore-db.sh` 的 Docker stdin 和 host-file 两条路径,并覆盖 dump-list 失败、目标确认失败、apply 非零、后续阶段 `NOT_RUN` 和零残留。测试失败不得上传候选。
|
||||
- 远端地址、现场路径、远端账号、一次性容器/数据库命名、timer、构建前缀和 migration 集合指纹均不进入模板;由私有 JSON 提供并按字段类型、字符集和安全命名规则验证后替换。渲染器拒绝额外/缺失字段、换行、shell 语法、非绝对或逃逸路径和错误摘要格式,不从不受信环境变量覆盖。
|
||||
- 私有 JSON 和渲染后的现场候选只能位于仓库外,或位于仓库内专用的 `/.restore-drill-private/` ignored 目录。渲染器必须对输入和输出分别执行 Git ignore 判定;任一路径在仓库内但未被忽略时,在创建候选前失败关闭,不能依赖人工避免 `git add -A`。
|
||||
- 权限 `0600` 的私有原始日志只允许初始化时截断一次;容器启动、dump list、目标确认、restore apply 和附件恢复各写固定阶段标记并使用追加模式。这样后续阶段失败时仍能在本轮 runtime 内交叉核对之前阶段,最终清理和脱敏输出边界不变。
|
||||
- 上传前由合同脚本对实际候选文件生成/校验 SHA-256;ARM 安装前再次校验同一摘要。证据记录合同脚本提交、候选摘要和 PASS,不记录地址、路径或业务数据。
|
||||
|
||||
## Attempt 4 后的 PostgreSQL 稳定就绪修订提案
|
||||
|
||||
Attempt 4 在 `dump_list=PASS/0` 后、实际 restore apply 前以 `restore_target_confirmation_failed` 失败关闭。只读环境差异调查确认:runner 以第一次 `pg_isready` 成功作为容器就绪,而合同夹具因曾观察到 PostgreSQL 首次初始化临时服务关闭窗口,已改为连续两次真实查询成功;两者没有复用同一就绪实现。该问题按 `local_contract_vs_arm_environment_difference_investigation` 登记,高置信度根因为 `postgres_initialization_readiness_race`,不能归因于备份不可恢复。
|
||||
|
||||
修订只允许改变跟踪 runner 与其可复跑合同,不修改 `scripts/restore-db.sh`,不放宽目标身份、归档、shadow、无端口、恢复非零失败关闭或清理门禁:
|
||||
|
||||
1. 删除 runner 的“首次 `pg_isready` 成功即通过”判定。`pg_isready` 最多可作为诊断,不能成为允许 restore 的充分条件。
|
||||
2. `dump_list` 保持不连接目标数据库的独立阶段;其成功只证明 custom dump 目录可解析,不计入数据库稳定就绪次数。
|
||||
3. `target_confirmation` 在容器内显式通过 TCP 回环 `127.0.0.1` 执行真实 `SELECT current_database()`,不用默认 Unix socket。官方 PostgreSQL 镜像的 initdb 临时服务器不监听 TCP,而正式服务器监听容器内回环;因此 TCP 成功是排除临时初始化服务器的结构性判据。`--network none` 不影响容器自己的 loopback,实际 restore 仍可继续沿现有 socket 路径,不修改 `scripts/restore-db.sh`。
|
||||
4. TCP 目标确认使用最多 60 次、每次间隔 1 秒的有界窗口作为额外保险。每次查询退出码非零时把连续成功计数重置为 `0`;查询成功且数据库名精确等于本轮 shadow 目标时累计,只有连续两次成功才 `PASS`;查询成功但名称不匹配立即失败,不继续重试。连续成功不是排除临时服务器的唯一依据,不能移除 TCP 回环判据后单独保留时序启发式。
|
||||
5. 脱敏输出增加 `target_confirmation_exit_code`、`target_name_match=yes|no|NOT_RUN`、`target_confirmation_attempt_count` 和 `target_confirmation_success_streak`。退出码取最后一次实际查询的非负整数;未执行为 `NOT_RUN`。错误类别区分 `TARGET_QUERY_NONZERO` 与 `TARGET_NAME_MISMATCH`,不得输出查询文本、数据库名、路径、凭据或原始 stderr。
|
||||
6. 稳定就绪函数必须定义在被渲染 runner 中;合同从 rendered runner 提取或直接驱动该同一实现,不得在 fixture 外围另写一套更宽松或更稳定的等待逻辑。fixture 自身如需等待,也必须通过同一实现进入后续真实脚本双路径回归。
|
||||
7. 实施前审计 runner 合同的每个覆盖机制,区分“直接执行跟踪脚本/校验器”“从 rendered runner 提取同一实现”“只做静态结构断言”和“合同自行重写等价逻辑”。最后一类必须消除或登记为明确限制,不能以组件测试通过推断未执行的 runner 编排行为。
|
||||
8. 回归至少覆盖:Unix socket 可用但 TCP 仍失败时不得通过;首次 TCP 查询失败后连续两次成功可继续;一次成功后失败会重置 streak;数据库名错配立即失败;查询持续非零在第 60 次后失败;每条失败路径的 apply 保持 `NOT_RUN`、诊断先于清理且残留为 `0`;原有 dump、restore 双路径、身份、migration、附件和敏感输出合同继续全绿。
|
||||
9. 禁止用固定长 sleep、增加 ARM 特例、降低连续成功次数、跳过 TCP 回环或名称精确比较、只修改合同夹具来修复。若相同实现的本地合同通过而 ARM 仍在该阶段失败,保留环境差异 finding,依据新退出码和尝试计数继续调查,不现场放宽。
|
||||
|
||||
本节已于 2026-08-25 获准仅在本地实施和回归;授权不包含连接 ARM 或执行 Attempt 5。修复通过后仍须提交可追溯变更、重新选择当时最新批次、重跑只读身份探针、重新渲染并核对候选 SHA-256,再单独申请 Attempt 5 实际恢复。
|
||||
|
||||
## 清理与回滚
|
||||
|
||||
- 生产无变更,无生产回滚步骤。
|
||||
- 成功或失败都只删除本轮 run id 对应的一次性容器、临时附件目录和临时批次副本;归档目录、receiver 管理目录和现有监控容器不在清理匹配范围。
|
||||
- 清理前再次解析容器名和临时根目录;任何对象不满足本轮标识、类型或父目录约束时停止自动清理。
|
||||
- 清理后确认无本轮容器、宿主端口、临时网络和临时目录残留;失败则结论至少为 `PAUSE`,交由管理员精确处置。
|
||||
|
||||
## 证据边界
|
||||
|
||||
执行证据写入当前 task 的 `research/restore-drill-evidence.md`,随后只在现有 OPS7 Runbook 或人工试运行报告中补一条脱敏状态摘要;不新建 `docs/` 平行文档。不得保存主机地址、SSH 别名、完整远端路径、Secret、连接串、真实账号、表内容、附件名或原始日志。
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,213 @@
|
||||
# 隔离备份恢复演练实施计划
|
||||
|
||||
## 执行前门禁
|
||||
|
||||
- [x] 确认任务已在用户审阅最终规划摘要后由 `task.py start` 进入 `in_progress`。
|
||||
- [x] 只读确认本地 `main`、当前生产构建事实、来源/当前 migration 集合和现有恢复脚本;现场两构建均为 65 且目录无变化,但后续执行不硬编码数量。
|
||||
- [x] 只读确认生产公网 `/ready` 正常;不读取生产数据库或执行生产写入。
|
||||
- [ ] 连接 ARM 后只读核对 receiver 配置有效、最新正式日批次身份、dead-man 状态、Docker 可用性和磁盘余量;不列出完整目录或原始 manifest。
|
||||
- [ ] 磁盘安全预算至少满足“批次总字节数 5 倍 + 2 GiB”;不足则 `PAUSE`。
|
||||
- [ ] 若当前账号没有对正式归档只读和本轮临时区/Docker 的最小权限,停止并给用户一条窄权限操作步骤;不得申请通用 root shell 或扩大既有健康监控 sudo 入口。
|
||||
|
||||
## 有序执行清单
|
||||
|
||||
1. 生成不含主机信息的本轮 run id,记录 UTC 开始时间和生产前置 `/ready` 状态。
|
||||
2. 在 ARM 正式日归档中选择 manifest 时间最新的 complete 批次,验证三文件、字节数、SHA-256 与 24 小时 RPO 门禁;在本地仓库确认来源提交是当前构建祖先或相同,并单独记录是否严格相等。
|
||||
3. 创建权限 `0700` 的本轮临时工作区;启动 RTO 计时,复制整个批次并对临时副本重新执行同一校验。
|
||||
4. 创建无宿主端口、无业务网络、无持久 volume 的一次性 PostgreSQL 16 容器;数据库名含 `drill`,凭据随机且仅存在本轮进程环境。
|
||||
5. 使用 `scripts/restore-db.sh` 的 shadow 合同恢复数据库;任何警告、确认不匹配或 restore 非零立即停止后续验证并进入清理。
|
||||
6. 从来源提交动态冻结 migration 名称集合;通过错误即停的只读 SQL 验证恢复库成功且未回滚的 migration 集合精确匹配来源、无失败/未知项,并静态记录来源与当前构建的 migration 差异;随后验证固定 12 类关键表并记录行数。
|
||||
7. 使用 `scripts/restore-storage.sh` 恢复附件到本轮空目录;比较归档与解包普通文件数。
|
||||
8. 在影子库中聚合本地附件引用,以不输出路径的校验程序验证路径安全、文件存在及适用的大小/SHA-256;记录引用总数和不一致数。
|
||||
9. 所有数据检查通过时停止 RTO 计时;按 manifest 时间计算 RPO,记录当前发布到首个同源恢复点的可用滞后,形成区分“原始数据可恢复性”和“当前 schema 升级状态”的 `GO/PAUSE/NO_GO` 脱敏结论。
|
||||
10. 清理本轮容器、临时网络(若有)、临时附件目录、临时批次副本和临时凭据;确认无本轮残留。
|
||||
11. 再次只读检查生产公网 `/ready` 和 ARM dead-man/健康 timer 状态,确认演练没有影响在线生产与现有监控。
|
||||
12. 将证据写入当前 task,并在既有事实源补充当前状态;不提交任何真实备份、日志或业务内容。
|
||||
|
||||
## Attempt 1 执行状态
|
||||
|
||||
- 最新 ARM 候选在 `select_latest_batch` 阶段因 manifest `build_mismatch` 失败关闭;复制、容器创建、数据库恢复和附件恢复均未开始。
|
||||
- 失败后的只读复核确认:公网 `/ready` 与生产自动备份 unit 的构建短标识一致,两个生产侧事实均为 `7b86ffb3`;因此当前不是计划事实源过期或备份 unit 冻结构建漂移。
|
||||
- ARM 一次性 root 入口和本轮 runtime 均为零残留,两个监控 timer 仍为 active;本轮 5 个普通用户暂存文件已按 SHA-256 精确匹配并删除,未宽泛删除目录。
|
||||
- Attempt 1 保持 `NO_GO`。在当时取得最新 ARM manifest 的脱敏构建短标识、run id 配对关系和 manifest 摘要一致性前,不进入第 3 步及之后的恢复流程。
|
||||
|
||||
## Attempt 1 后只读调查清单
|
||||
|
||||
1. 生成固定无参数的 root-owned ARM 只读身份探针和最小 sudoers,只允许读取 receiver 路径配置及最新正式日归档。
|
||||
2. 本地执行 `bash -n`、候选 SHA-256、sudoers candidate `visudo`(环境可用时)和禁止写入/恢复关键字人工复核。
|
||||
3. 上传到 ARM 普通用户私有暂存目录;由用户输入 sudo 密码安装固定入口,安装前再次核对全部候选 SHA-256 和系统 sudoers 语法。
|
||||
4. 运行一次固定入口,记录构建短标识、批次年龄、manifest 配对、产物聚合完整性和入口自清理状态。
|
||||
5. 对照公网 `/ready` 与生产自动备份 unit 的当前短标识:相等时记录同源恢复点已形成;不相等时在本地仓库核对是否为当前构建祖先,祖先且完整/新鲜仍可形成原始恢复重试方案,未知或非祖先才 `NO_GO`。
|
||||
6. 按 SHA-256 精确删除普通用户暂存文件;不得删除 ARM 正式归档或宽泛清理目录。
|
||||
|
||||
### 只读调查执行结果
|
||||
|
||||
- [x] 探针候选通过本地/ARM SHA-256、Bash 语法和 ARM sudoers candidate 校验。
|
||||
- [x] 固定入口只读读取最新正式批次并自删除;未复制、解包、启动容器或恢复数据。
|
||||
- [x] 最新批次 `ec682565` 在 24 小时内,run id/目录配对及 DB/附件大小、SHA-256 均通过,但与当前 `7b86ffb3` 不匹配。
|
||||
- [x] 只读对时确认自动备份合同刷新晚于最新批次生成;finding 收敛为“发布后尚未产生新构建恢复点”。
|
||||
- [x] root 入口残留为 0,3 个普通用户候选及暂存目录已按精确身份清理,两个 ARM 监控 timer 仍 active。
|
||||
- [x] 08-24 最新自然批次只读探针通过:来源与当前构建相同、年龄不超过 24 小时、run id 配对及 DB/附件 SHA-256 均通过;临时入口和暂存零残留。当前可提交恢复重试审批,尚未获得恢复授权。
|
||||
|
||||
### Attempt 2 恢复重试结果
|
||||
|
||||
- [x] 用户批准后安装并执行动态 migration 集合版固定恢复入口。
|
||||
- [x] ARM 正式归档最新批次首次门禁通过,并开始复制到本轮私有临时区。
|
||||
- [x] 临时副本复检因 runner 错把正式源目录 basename 身份规则应用到固定名 `batch` 而触发 `batch_identity`;失败阶段为 `create_isolated_runtime`,数据库容器和恢复均未开始。
|
||||
- [x] runtime、root 入口、6 个普通用户候选和暂存目录均精确清理为零残留,两个 ARM 监控 timer 保持 active。
|
||||
- [x] 修订 runner:正式源目录继续强制 basename/run id 配对;临时副本 manifest run id 重绑到源侧已验证 run id,并继续校验完整内容合同及源/副本元数据精确一致。补齐 2 正例 + 3 负例后重新执行只读身份探针,实际恢复另行批准。
|
||||
|
||||
### Attempt 3 准备批准边界
|
||||
|
||||
- [x] 用户已批准修复 runner,并重新执行 08-25 最新正式批次只读身份探针。
|
||||
- [x] 回归已覆盖:正式源通过、同内容改名副本通过、内容篡改副本失败、源 basename/run id 错配失败、副本 run id 与源侧已验证 run id 不同失败。
|
||||
- [x] Attempt 3 未复用 Attempt 2 新鲜度:08-25 最新批次年龄约 6 小时 11 分,来源与当前构建相同,run id 配对、大小与 SHA-256 均通过。
|
||||
- [x] 用户复核探针后明确批准 Attempt 3 实际恢复,范围仅限 ARM 隔离影子恢复。
|
||||
|
||||
### Attempt 3 恢复结果与停止点
|
||||
|
||||
- [x] 修复后的候选在本地和 ARM 通过 SHA-256、Bash 与 sudoers 语法校验后安装;正式归档门禁和临时副本源侧身份重绑通过。
|
||||
- [x] runner 在 `restore_database` 阶段因 `restore-db.sh` 返回非零而以 `database_restore_failed` 失败关闭;migration、12 类关键表、附件恢复与关联验证均未执行。
|
||||
- [x] 一次性容器、本轮 runtime、临时 root 入口和普通用户候选均精确清理为零残留;ARM 两个监控 timer 保持 active,生产演练后 `/ready` 仍为相同构建的 `PASS`。
|
||||
- [x] 在任何 Attempt 4 前修复失败诊断合同:分别记录 dump 列表校验与实际 restore 的脱敏阶段码,并确保失败清理前只保留安全聚合诊断;本地回归已通过,远端探针和实际恢复仍需重新批准。
|
||||
|
||||
### Attempt 4 前的修复清单
|
||||
|
||||
1. runner 在调用 `restore-db.sh` 前,以同一 PostgreSQL 16 容器对临时副本执行独立 `pg_restore --list`,记录 `dump_list=PASS|FAIL|NOT_RUN`。
|
||||
2. 独立确认影子目标数据库名与连接状态,记录 `target_confirmation=PASS|FAIL|NOT_RUN`;不输出数据库名或连接信息。
|
||||
3. 仅在前两项通过后执行实际 restore,记录 `restore_apply=PASS|FAIL|NOT_RUN`;失败类别区分为 `dump_list_failed`、`restore_target_confirmation_failed` 或 `restore_apply_failed_after_list_pass`。
|
||||
4. 原始 stderr 只进入本轮 `0600` 私有日志并随 runtime 删除;最终输出只含 allowlist 阶段状态和既有脱敏聚合。
|
||||
5. 补三条阶段负例、敏感输出负例和每条失败路径零残留回归;通过后重新执行最新批次身份探针,再请求 Attempt 4 实际恢复批准。
|
||||
|
||||
### Attempt 3 后本地修复结果
|
||||
|
||||
- [x] `dump_list`、`target_confirmation`、`restore_apply` 固定输出 `PASS/FAIL/NOT_RUN`,并增加执行路径、脚本/pg_restore 退出码、对象错误计数与稳定类别。
|
||||
- [x] 诊断在清理前输出;注入 apply 退出码 `23` 后,调用方在 runtime/access 清理完成后仍可读取阶段状态和退出码。
|
||||
- [x] partial-success 规则保持失败关闭:任何 pg_restore 非零均为 `FAIL/NO_GO`,不设置可忽略对象错误白名单。
|
||||
- [x] PostgreSQL 16 custom dump 直接驱动真实 `scripts/restore-db.sh` 的 Docker stdin 与 host-file 两条路径,均恢复成功。
|
||||
- [x] dump-list 失败、目标确认失败、apply 失败及敏感输出负例通过;既有成对备份/恢复回归和自动异机备份合同通过。
|
||||
- [x] 用户另行批准并完成 Attempt 4 前最新批次只读探针:构建 `SAME`、年龄约 7 小时 24 分、run id/产物完整性与 migration 集合门禁通过,入口及暂存零残留;尚未执行实际恢复。
|
||||
|
||||
### Attempt 4 前的持久化门禁
|
||||
|
||||
1. 将已验证的 runner 和附件校验器从个人 workspace 收敛为 `scripts/qa/` 下的跟踪文件,不改变恢复边界或生产脚本。
|
||||
2. 将八类 runner 回归收敛为可独立执行的 `run-offhost-shadow-restore-runner-contract.sh`,直接驱动真实 `restore-db.sh` 双路径。
|
||||
3. 合同脚本必须机械验证三阶段诊断、执行路径、两个退出码、非零失败关闭、源/副本身份、dynamic migration、敏感输出和清理顺序;对候选生成稳定 SHA-256。
|
||||
4. 运行新合同、既有成对备份/恢复、自动异机备份 74 项、Trellis 治理与敏感扫描;全部通过后才允许把跟踪文件打包为 Attempt 4 候选。
|
||||
5. 本阶段仅本地修改与测试;新批次探针结果和持久化门禁均通过后,再单独申请 Attempt 4 实际恢复。
|
||||
|
||||
### Attempt 4 前持久化门禁结果
|
||||
|
||||
- [x] runner 已收敛为 `scripts/qa/offhost-shadow-restore-runner.template.sh`;现场路径、账号、容器/数据库命名、timer、构建和 migration 指纹均为经严格校验的私有渲染字段。
|
||||
- [x] 附件关联校验器已收敛为 `scripts/qa/validate-offhost-shadow-restore-attachments.py`。
|
||||
- [x] `scripts/qa/run-offhost-shadow-restore-runner-contract.sh` 可独立复跑并提供 `render` 模式;相同私有配置生成确定性候选 SHA-256,shell 注入在输出候选前失败关闭。
|
||||
- [x] 源/副本身份、内容篡改、dynamic migration、附件安全、三阶段诊断、真实 restore 双路径、NOT_RUN、退出码、清理顺序和敏感输出合同全部通过。
|
||||
- [x] 新增三个文件由 `check-sensitive-content.mjs` 定向扫描,`3/3` 文件、finding `0`;本地测试容器残留 `0`。
|
||||
- [x] `.gitignore` 增加 `/.restore-drill-private/` 专用目录;合同分别证明仓库内未忽略的私有 JSON、未忽略的现场候选均在创建输出前失败,ignored 输入/输出正例通过。
|
||||
- [x] 私有原始日志保留唯一初始化截断,五个固定阶段均以标记分隔并追加写;静态合同拒绝后续阶段再次截断。
|
||||
- [x] 完整复跑 runner 合同、既有隔离备份/恢复演练和自动异机备份 74 项;task、Trellis、process、默认/定向敏感扫描及空白门禁通过,测试容器和本地私有路径残留均为 `0`。
|
||||
- [x] Attempt 4 最新批次只读探针重新执行并通过;候选已由提交 `5ec7db1` 的跟踪模板确定性渲染,完整合同、依赖钉住和 SHA-256 复核通过。
|
||||
- [x] 用户批准 Attempt 4 后,候选在 ARM 安装前再次通过 SHA 清单与语法校验并执行;`dump_list=PASS/0`,随后在 `target_confirmation` 阶段失败,实际 restore apply、migration、关键表和附件均未运行。
|
||||
- [x] Attempt 4 runtime、临时 root 入口和普通用户上传暂存均清理为 `PASS/0`;按预先裁决规则转入本地合同路径与 ARM 实际环境的只读差异调查,不直接修补 runner 或创建 Attempt 5。
|
||||
- [x] 只读差异矩阵与本地精确复现收敛到 PostgreSQL 初始化就绪竞态:合同夹具等待连续两次真实查询,runner 仅等待首次 `pg_isready`;本地稳定数据库上的同一 target query 为 `PASS/0`,ARM 所有临时对象复核残留 `0`。
|
||||
- [x] Attempt 4 后正式归档只读探针与执行前为同一构建和同一 manifest 脱敏摘要,run id 配对、产物 SHA-256、24 小时新鲜度均通过;探针入口与暂存残留 `0`,正式归档内容级零修改已补证。
|
||||
- [x] 演练后腾讯云生产 `/ready` 已由执行方沿既有只读路径补证:状态 `PASS`、12 位构建前缀 `7b86ffb3c9ef`,与演练前一致;未记录地址或原始响应。
|
||||
- [x] 用户已批准仅在本地实施和回归 PostgreSQL 稳定就绪修复;该批准不包含连接 ARM 或执行 Attempt 5。
|
||||
|
||||
### Attempt 5 前仅本地修复结果
|
||||
|
||||
- [x] 在跟踪 runner 中以容器内 TCP 回环 `127.0.0.1` 的真实 `SELECT current_database()` 作为结构性就绪判据,并以最多 60 次、间隔 1 秒、连续两次成功且名称精确匹配作为额外保险,替换首次 Unix socket `pg_isready` 即通过逻辑;名称错配立即失败,查询非零重置连续成功计数。
|
||||
- [x] 增加 target query 退出码、名称匹配、尝试次数和连续成功次数的 allowlist 诊断,区分查询非零与名称错配;私有原始日志、失败先诊断后清理和零残留边界不变。
|
||||
- [x] 可复跑合同直接提取并驱动 rendered runner 中同一稳定就绪函数;fixture 外围与 runner 不同源的等待循环已删除。
|
||||
- [x] “socket 可用但 TCP 不可用仍失败”、首次失败后恢复、streak 重置、名称错配立即失败、永久查询失败有界退出五类回归通过;真实 `restore-db.sh` Docker stdin/host-file、源/副本身份、dynamic migration、附件安全、敏感输出和清理合同继续通过。
|
||||
- [x] 已审计合同全部覆盖机制,未发现第二套自写等价恢复逻辑;完整 runner 的 migration/12 类表/附件编排仍只能由真实隔离恢复验证,已作为明确限制保留,不能由本地组件合同替代。
|
||||
- [x] `scripts/restore-db.sh` 相对生产构建字节未改;runner、成对备份/恢复、自动异机备份 74 项、task、Trellis、process、docs、敏感扫描和空白门禁均通过。
|
||||
- [x] 本地修复阶段未连接 ARM、未生成或安装 Attempt 5 现场入口;提交后已重新选择最新批次、执行只读探针、重新渲染候选并核对 SHA-256,实际恢复另行取得了明确批准。
|
||||
|
||||
### Attempt 5 前只读身份探针结果
|
||||
|
||||
- [x] 用户批准后仅安装并执行最新正式批次只读探针;未复制、解包、启动容器、读取业务数据或执行恢复。
|
||||
- [x] 最新批次来源与当前生产构建相同,探针时年龄 `64625` 秒且不超过 24 小时;run id 配对、DB/附件大小与 SHA-256、source digest 均通过。
|
||||
- [x] 来源/当前 migration 目录为 `65/65`、差异 `0`;仓库 HEAD 相对生产构建的应用代码漂移文件数 `0`。
|
||||
- [x] 探针 root 入口与普通用户暂存残留均为 `0`;按已验证私有配置取得的 dead-man/健康 timer 实际 unit 状态为 `active/active`。
|
||||
- [x] 本次探针完成时 RTO 八阶段实现尚未批准,因此当时未渲染 Attempt 5 最终候选、未生成最终候选 SHA-256,也未执行实际恢复;后续本地实现授权单独追加,不改写该时点事实。
|
||||
|
||||
### Attempt 5 前 RTO 分阶段实施(已批准仅本地实现)
|
||||
|
||||
- [x] 设计口径已固定为总 RTO 加八段串行秒数:批次复制、容器启动、dump list、目标确认、restore apply、migration/12 类表、附件恢复与关联校验、最终源/副本复核。
|
||||
- [x] runner 增加固定 allowlist 阶段字段,成功时八段均为非负整数且精确加总为 `data_restore_rto_seconds`;失败时已完成段和当前失败段保留数字,后续段为 `NOT_RUN`。
|
||||
- [x] 合同注入各阶段成功/失败,验证最长 TCP 轮询只进入目标确认阶段、后续 `NOT_RUN`、总数精确加总、输出不含路径/SQL/凭据;原有清理与恢复合同继续通过。
|
||||
- [x] 用户批准的当前边界仅为本地修改 runner、更新可复跑合同并执行回归;不连接 ARM、不重新渲染 Attempt 5 现场候选、不执行实际恢复。
|
||||
- [x] 八阶段计时与完整合同通过后,满足 24 小时门禁的批次已直接进入 Attempt 5 候选渲染、SHA-256 核对、只读探针和实际恢复审批流程;期间未插入新的非阻断增强。
|
||||
- [x] 八阶段状态必须在失败路径清理前输出;阶段边界沿用 `dump_list`、`target_confirmation`、`restore_apply` 等既有诊断词汇,不建立第二套同义命名。
|
||||
|
||||
### Attempt 5 最终候选与执行前探针
|
||||
|
||||
- [x] 从提交 `0b94b97` 的跟踪模板和 ignored 私有配置完成两次确定性渲染;候选字节一致、完整 runner 合同通过,候选 SHA-256 为 `7904052aa5a8c31d54734bc2fbf2d9caf2dac8b0116135119519f8467b3dae27`。
|
||||
- [x] 候选固定的是生产构建前缀和来源 migration 集合指纹,不固定探针时的 source run id;实际执行会在 ARM 上重新选择当时最新正式批次,并对该批次重新执行完整身份、产物 SHA-256、24 小时 RPO 和 migration 集合门禁。
|
||||
- [x] 最终只读探针重新选择执行时最新正式批次:来源构建与当前生产相同,年龄 `68889` 秒且不超过 24 小时,run id/目录配对、DB/附件完整性和 source digest 均通过。
|
||||
- [x] 本地复核来源/当前 migration 为 `65/65`、差异 `0`,候选 migration 指纹与来源集合一致,应用代码漂移和 `scripts/restore-db.sh` 字节漂移均为 `0`。
|
||||
- [x] 探针 root 入口、sudoers 和普通用户暂存残留均为 `0`;本地 runner 合同容器残留为 `0`。本阶段未复制、解包或恢复正式归档,未连接腾讯云生产。
|
||||
- [x] Attempt 5 实际隔离恢复已在新鲜度门禁成立时另行批准并执行;紧邻执行前探针通过,但 runner 在选择实际恢复批次之前即因依赖库入口合同失败,因此本轮不存在“实际选择批次”,不能宣称恢复输入与探针批次相同。
|
||||
|
||||
### Attempt 5 实际结果与安装器差异调查
|
||||
|
||||
- [x] 安装前候选 SHA-256 与已批准值精确一致;紧邻执行前只读探针通过,批次年龄 `70444` 秒,来源构建与当前生产相同,run id 配对、产物完整性和 source digest 门禁通过。
|
||||
- [x] runner 在 `entry_contract` 阶段以 `restore_library_missing` 失败关闭;批次选择、正式归档复制、PostgreSQL 容器启动、数据库/附件恢复和全部八阶段 RTO 均未运行,原始结果保留为 `NO_GO`。
|
||||
- [x] 失败分类为 `installer_path_mismatch`:已批准 runner 从固定 `libexec` 位置加载三项依赖,本轮一次性安装器却把同一组依赖安装到相似但不同的 `lib` 位置。该结论指向演练安装器,不证明 runner 或备份不可用。
|
||||
- [x] 对错误安装目录做精确只读检查:三项依赖均为 root 拥有的普通文件、SHA-256 前缀与本地包一致、额外条目数 `0`;随后只清理该已确认目录并得到 `runner_lib_cleanup=PASS`。
|
||||
- [x] 诊断脚本在 sudo 上下文中用 `$HOME` 推断普通用户暂存位置,导致自删除误指向 root 家目录;随后由普通用户外层对已知暂存脚本执行精确清理。后续安装器不得在 sudo 脚本中用 `$HOME` 推断调用用户路径。
|
||||
- [x] Attempt 5 后腾讯云生产 `/ready` 为 `PASS`,12 位构建前缀仍为 `7b86ffb3c9ef`;失败发生在归档选择之前,未触碰腾讯云生产或正式归档内容。
|
||||
- [x] 最终只读复核确认 runner/probe 入口与 sudoers、正确 `libexec`、错误 `lib`、普通用户暂存、Docker 容器和 runtime 六类残留全部为 `0`,dead-man/健康 timer 为 `active/active`;复核脚本自身清理 `PASS`。
|
||||
- [x] ignored/workspace 安装器已把三项依赖安装目标对齐 runner 已批准的 `libexec` 合同;诊断脚本改用经 basename、普通文件和非链接校验的调用路径自清理,不再以 sudo `$HOME` 推断普通用户路径。runner 与候选字节未改、无需重新渲染。
|
||||
- [ ] 再次实际恢复前重新选择满足 24 小时门禁的新鲜批次,执行只读身份探针并单独申请实际恢复批准;不得复用 Attempt 5 的批次新鲜度或既有实际恢复授权。
|
||||
|
||||
### 08-27 最新批次只读身份探针
|
||||
|
||||
- [x] 安装器与 rendered runner/probe 的共享路径完成本地静态核对:runner `SELF_PATH/LIB_DIR/SUDOERS_PATH`、probe `SELF_PATH/SUDOERS_PATH` 均在首次特权写入前精确比对;`WORK_ROOT` 未在安装器重复编码,sudoers candidate 从最终路径推导。
|
||||
- [x] 探针上传准备中一次 basename 截取错误被远端 SHA-256 校验在普通用户暂存阶段拦截;未执行 sudo、未安装入口、未读取归档。三个已确认截短文件逐个精确清理后,以真实 basename 重传并通过三文件 SHA-256 与 Bash 语法校验。
|
||||
- [x] 最新正式批次只读探针执行成功:来源构建与当前生产相同,run id/目录配对、DB/附件产物完整性和 source digest 均通过;探针入口与普通用户暂存残留为 `0/0`。
|
||||
- [x] 最新批次年龄 `124137` 秒,约 34 小时 29 分,超过 24 小时门禁 `37737` 秒,约 10 小时 29 分;本次恢复前置判定为 `NO_GO/stale_latest_batch`,不得改选旧批次或执行实际恢复。
|
||||
- [x] 用户批准后完成生产与 ARM 两侧只读调查:daily/watchdog timer 均按计划触发但最近 service 均 `exit-code/1`;ARM 最新原子归档超龄且存在两个不完整 partial,定位为生产到 ARM 的传输未完成,不是调度漏跑或完整传输后的 finalize 失败。
|
||||
- [x] dead-man 实际阈值为 `93600` 秒并正确记录 `FAIL_OFFHOST_DEADMAN/archive_stale`;journal 聚合存在一次 `sent=true`,证明 detector 与发送调用未被去抖静默抑制。健康 timer 保持 active,两端调查入口和暂存均清理为零残留。
|
||||
- [x] 两个 partial 证明 SSH 会话已完成初始认证并开始写入,后续排查不再以密钥失效、authorized_keys 或 host key 初始拒绝为首要假设;独立事件调查优先采集 partial 聚合字节偏移、真实 rsync 退出码和传输持续时间。
|
||||
- [ ] 在生产自动备份传输恢复、ARM 形成新的 24 小时内原子批次之前,恢复演练保持阻塞;补跑备份、修复传输配置/网络、修改 timer 或发送测试告警均需新的明确批准。
|
||||
- [ ] 将生产到 ARM 的传输中断拆为独立 Trellis 生产事件任务,当前演练任务只保留交叉引用和阻塞关系;创建任务、生产侧只读深查与实际修复均另行取得批准。
|
||||
|
||||
## 验证命令类别
|
||||
|
||||
- 本地/仓库:`bash -n` 覆盖备份、DB 恢复、附件恢复和 receiver 脚本;只读统计 migration 目录。
|
||||
- ARM 前置:receiver `validate-config`、systemd 公共状态、Docker 版本/容量、候选 manifest 脱敏校验。
|
||||
- 恢复输入:`sha256sum`、manifest schema/文件集/字节数校验、`pg_restore --list`、`tar -tzf`/`tar -tvzf`。
|
||||
- DB 验证:`psql -v ON_ERROR_STOP=1` 的 migration 名称/状态集合和固定表计数查询;本地用 Git 比较来源/当前 migration 目录,不执行 migrate。
|
||||
- 附件验证:`find` 普通文件计数及不输出路径的 Python 只读校验;只输出引用数和 mismatch 数。
|
||||
- 任务门禁:
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/08-23-isolated-backup-restore-drill`
|
||||
- `node scripts/qa/check-trellis-governance.mjs`
|
||||
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||
- `node scripts/qa/check-sensitive-content.mjs`
|
||||
- `git diff --check`
|
||||
|
||||
具体远端命令在执行阶段依据 ARM 已安装的 root-owned receiver 配置生成,不把真实地址、账号、目录或凭据写入本计划。
|
||||
|
||||
## 停止条件
|
||||
|
||||
- 最新批次不存在、超过 24 小时、manifest 或 SHA-256 不一致,或来源提交未知/不是当前构建祖先或相同。仅“不严格相等”不再单独停止。
|
||||
- 权限、Docker、PostgreSQL 16 或磁盘预算不满足。
|
||||
- 命令目标可能指向腾讯云生产、ARM 正式归档、receiver 管理目录或既有容器/网络。
|
||||
- DB restore、migration、固定表、附件安全、文件引用、哈希/大小任一检查失败。
|
||||
- 生产 `/ready` 在演练期间异常,或现有 ARM dead-man/健康监控状态改变。
|
||||
- 清理对象身份不明确。此时保留现场并报告,不使用宽泛删除命令。
|
||||
|
||||
## 变更与证据文件边界
|
||||
|
||||
- 允许写入:当前 Trellis task、既有 OPS7 Runbook/人工试运行报告中的脱敏状态摘要、本轮 ARM 私有临时区和一次性 Docker 对象。
|
||||
- 禁止写入:产品代码、migration、生产/ARM 正式配置、生产 DB/附件、ARM 正式归档、企业微信。
|
||||
- 计划新增的 task 内证据:`research/restore-drill-evidence.md`;只在实施实际发生后创建。
|
||||
|
||||
## 审查与收口
|
||||
|
||||
- 本任务涉及真实备份数据和远端运维边界。执行完成后做一次只读复核,重点确认:生产零变更、归档未修改、RTO/RPO 口径正确、附件验证没有漏项、敏感内容未进入证据、临时对象已清理。
|
||||
- 若现有脚本需要修改,回到规划阶段补充设计和新的批准;不得在远端临时放宽 shadow/路径/确认门禁。
|
||||
- 门禁通过后用中文 commit 引用 `08-23-isolated-backup-restore-drill`,是否推送仍按用户当时授权执行。
|
||||
@@ -0,0 +1,89 @@
|
||||
# 隔离备份恢复演练
|
||||
|
||||
## Goal
|
||||
|
||||
在不覆盖、不暂停腾讯云生产 OA 的前提下,使用 ARM 备份机上最新的完整异机批次完成一次数据库与附件的隔离影子恢复,证明“备份存在”进一步达到“最新成对数据备份可恢复”,并记录实测数据恢复 RTO、RPO 和脱敏证据。
|
||||
|
||||
## Scope
|
||||
|
||||
- 在 ARM 上选择并校验最新正式日归档,使用本轮临时副本恢复数据库与附件。
|
||||
- 验证 migration、固定关键表、附件数量与 DB/附件引用一致性,记录数据恢复 RTO/RPO。
|
||||
- 复核生产前后 `/ready`、ARM 监控状态和临时对象清理结果,形成当前 task 内脱敏证据并更新既有事实源摘要。
|
||||
- 不修改产品运行代码;允许把演练 runner 模板、附件校验器和可复跑合同收敛到 `scripts/qa/`。生产、正式归档、企业微信和现有监控均保持只读或零变更。
|
||||
|
||||
## Background
|
||||
|
||||
- 腾讯云服务器是真实 OA 生产;ARM 只承担异机备份、外部健康监控和受控恢复演练,不是 OA 生产服务器。
|
||||
- 2026-08-23 只读复核确认:生产最近 14 天每日异机备份和新鲜度 watchdog 均成功,ARM dead-man 最近成功;企业微信中的“恢复通知”来自公网健康状态恢复,不代表执行过数据恢复。
|
||||
- 2026-07-27 曾在 ARM 使用当时的正确异机批次完成真实格式影子恢复,但不能替代对当前最新自动备份的可恢复性验证。
|
||||
- 首次演练时生产产品构建为 `7b86ffb`,已通过公网 `/ready` 和 migration pending=none;当时仓库与来源构建 `ec682565` 均有 65 个 Prisma migration 目录,二者之间 migration 目录无变化。该数字是本次现场事实,不再作为跨构建硬编码合同。
|
||||
- 现有恢复脚本默认只接受带 `shadow`、`drill` 或 `restore_test` 标记的数据库名,并拒绝生产式目标、确认值不一致、非空附件目标和危险归档路径。
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1. 恢复点选择与前置门禁
|
||||
|
||||
- 只选择 ARM 正式日归档中按 UTC 时间最新的一个完整批次,不触碰 inbox、临时事务目录、旧错误批次或周检查点。
|
||||
- 批次必须恰含 `database.dump`、`storage.tar.gz`、`manifest.json` 三个普通文件;manifest 必须为 schema v1、`complete`,相对文件名、字节数、SHA-256 和来源构建身份全部一致。
|
||||
- 批次年龄必须不超过 24 小时。manifest `buildSha` 必须唯一解析为本仓库已知提交,且该提交是演练开始时公网 `/ready` 实测当前构建的祖先或与其相同;未知提交、旁支/后继提交、超龄、损坏或文件集歧义判定 `NO_GO`,磁盘空间不足判定 `PAUSE`。两者都不得自动改选更旧批次掩盖问题。
|
||||
- 构建相等只表示“当前发布后已经形成同源恢复点”,不再作为数据备份可恢复性的必要条件。若来源是当前构建的祖先,必须单独记录发布后同源恢复点滞后 finding,不能把它误报为归档损坏。
|
||||
- 执行前后各做一次公网 `/ready` 只读检查,仅用于证明生产未受影响;该结果不得充当影子恢复后的应用启动证据。
|
||||
|
||||
### R2. ARM 隔离和数据保护
|
||||
|
||||
- 演练仅在 ARM 上创建本轮专属、权限 `0700` 的临时工作区、一次性 PostgreSQL 16 容器和空附件目录;不得在开发机下载真实备份。
|
||||
- PostgreSQL 不发布宿主端口,不加入现有 OA/监控网络,不复用任何现有容器名、数据库名、volume 或目录;影子数据库名必须含 `drill` 标记。
|
||||
- 先把所选批次复制到临时工作区并重新校验 manifest 与三文件 SHA-256,再对临时副本执行恢复;正式 ARM 归档始终只读,不直接解包或改名。
|
||||
- 不读取或复用生产 `.env`、JWT、Webhook、企业微信 Secret、真实数据库口令或业务账号;影子数据库只使用本轮随机临时凭据,凭据不进入聊天、仓库、报告或命令参数证据。
|
||||
- 含现场路径、命名或构建值的私有渲染 JSON 和渲染后候选不得进入 Git 历史;若位于仓库内,必须置于专用 ignored 目录,合同脚本须在创建候选前机械验证该 ignore 状态,未忽略即失败关闭。
|
||||
|
||||
### R3. 数据库恢复验证
|
||||
|
||||
- 使用现有 `scripts/restore-db.sh` 的 `RESTORE_SCOPE=shadow` 和精确数据库名确认门禁,将最新 `database.dump` 恢复到一次性 PostgreSQL 16 容器。
|
||||
- `pg_restore --list` 和实际 restore 必须成功。从 manifest 来源提交动态冻结其 Prisma migration 目录集合;恢复后 `_prisma_migrations` 中成功且未回滚的 migration 名称必须与该来源集合一致,并且不存在失败、回滚中或来源未知的 migration。不得用硬编码数量替代集合核对。
|
||||
- 同时比较来源提交与当前构建的 migration 目录集合:集合相同则记录“当前 schema 无额外升级步骤”;当前构建新增 migration 时,原始备份恢复仍可继续验证,但“升级到当前 schema”标为 `PAUSE/未验证`,另行设计仅针对影子库的 migration 演练,不在本轮临时执行。
|
||||
- 固定核对组织、人员、角色、采购、报销、发票池、付款批次、合同、项目及三类核心附件表的存在性和行数;只记录表名、数量和 PASS/FAIL,不记录业务字段或正文。
|
||||
- 所有验证 SQL 必须只读并开启错误即停;不执行 migration、seed、DDL、数据修补或业务写入。
|
||||
|
||||
### R4. 附件恢复与关联验证
|
||||
|
||||
- 使用现有 `scripts/restore-storage.sh` 的 shadow 门禁恢复 `storage.tar.gz` 到本轮空目录;归档路径、链接类型和目标空目录检查必须通过。
|
||||
- 归档普通文件数必须与解包后普通文件数一致;对数据库中本地存储型的发票、采购附件、报销附件、付款回单和请假附件引用,验证相对路径安全、目标文件存在,并在已有 SHA-256/大小字段时核对哈希或大小。
|
||||
- 发现 `s3:` 引用、缺失文件、路径逃逸、哈希/大小不一致或 DB 与附件批次不配对时判定 `NO_GO`;不输出文件名、原始路径、票据内容或附件内容。
|
||||
|
||||
### R5. 计时、证据和清理
|
||||
|
||||
- `RPO` 定义为演练开始 UTC 时间减 manifest `createdAt`;必须小于等于 24 小时才通过。本次实测值只用于当前恢复演练,不形成 SLA 承诺。
|
||||
- “数据恢复 RTO”从通过前置门禁后、开始复制批次到临时工作区之前计时,到 DB/附件恢复和全部数据一致性检查通过时结束;不得把历史合成演练耗时或生产 `/ready` 计入本轮 RTO。
|
||||
- 另行记录“同源恢复点可用滞后”:当前发布合同刷新时间到首个相同构建的 ARM 完整批次到达时间。该指标与数据 RPO、数据恢复 RTO 分开,不形成 SLA;即使恢复未执行,观测到的发布后空窗仍是有效演练发现。
|
||||
- 证据写入当前 Trellis task,只记录日期、角色、批次脱敏标识、构建短标识、年龄、耗时、数量、状态、finding 类别和清理结果;不保存远端地址、完整目录、凭据、真实账号、业务正文或原始 payload。
|
||||
- 成功或失败后只清理本轮明确创建的一次性容器、临时工作区和临时凭据;清理后确认无容器、端口、临时目录残留。任何目标身份不明确时停止清理并人工复核,绝不删除正式归档。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] AC1:ARM 最新正式日批次通过文件集、manifest、字节数和 SHA-256 校验,来源构建是当前构建的祖先或相同,且实测 RPO 不超过 24 小时;是否已形成当前构建同源恢复点单独记录。
|
||||
- [ ] AC2:腾讯云生产在演练前后 `/ready` 均正常,生产容器、数据库、附件、配置、timer 和业务数据零写入、零重启、零切换。
|
||||
- [ ] AC3:一次性 PostgreSQL 16 影子库无宿主端口,数据库恢复成功;恢复 migration 集合与 manifest 来源提交一致且无失败/未知项,来源与当前 migration 差异已明确;固定 12 类关键表均可读并形成仅含数量的脱敏摘要。
|
||||
- [ ] AC4:附件安全恢复成功,归档与解包普通文件数一致;所有纳入范围的本地附件引用均安全且存在,适用的哈希/大小核对无不一致。
|
||||
- [ ] AC5:记录本轮数据恢复 RTO、RPO、同源恢复点可用滞后、开始/结束时间、失败点或 finding、Go/Pause/No-Go 结论;即使恢复未开始也保留已观测空窗,不把任何指标表述为完整应用灾备 RTO 或 SLA。
|
||||
- [ ] AC6:演练结束后本轮容器、端口、网络和临时数据全部清理,ARM 正式归档与现有 dead-man/健康监控不受影响。
|
||||
- [ ] AC7:任务校验、Trellis 治理、流程合同、文档审计、敏感扫描和 `git diff --check` 通过;证据不含敏感信息。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 不在腾讯云生产执行 restore、停机、重启、数据库 reset、migration、DDL、seed、配置修改、timer 修改或业务写入。
|
||||
- 不触发新的生产备份,不删除、替换或轮转 ARM 正式归档,不发送企业微信测试消息。
|
||||
- 不把真实备份下载到开发机,不把 ARM 变成 OA 部署目标或长期运行 OA server。
|
||||
- 本轮不启动隔离 OA 应用容器、不做登录、报销详情或附件下载页面验证,也不对恢复库执行 Prisma migration;因此只形成“原始数据恢复 RTO”和静态 migration 兼容判断,完整应用恢复、schema 升级和切换 RTO 仍需未来灾难恢复专项验证。
|
||||
- 不修改备份/恢复产品代码;若现有脚本在真实最新批次上暴露缺陷,先停止并另行回到规划,不现场放宽安全门禁。
|
||||
|
||||
## Blocking Open Questions
|
||||
|
||||
- 无。构建门禁已裁决为“来源提交是当前构建祖先或相同 + 批次年龄不超过 24 小时”;实施仍需用户审阅本次修订后的最终规划摘要并再次明确批准。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 校验最新批次 manifest、三文件集合、字节数、SHA-256、构建身份和 24 小时新鲜度。
|
||||
- 通过现有 shadow 恢复脚本完成 PostgreSQL 16 和附件恢复,验证来源提交的 migration 集合、与当前构建的 migration 差异、固定 12 类表及附件引用聚合。
|
||||
- 复核无宿主端口、生产前后 `/ready`、ARM 监控未受影响和本轮临时对象零残留。
|
||||
- 运行 task validate、Trellis governance、process contract、docs registry、敏感扫描和 `git diff --check`。
|
||||
@@ -0,0 +1,594 @@
|
||||
# 隔离备份恢复演练证据
|
||||
|
||||
## 证据边界
|
||||
|
||||
- 仅记录日期、角色、短构建标识、状态、finding 类别和清理结果。
|
||||
- 不记录远端地址、SSH 身份、完整目录、凭据、账号、原始 manifest、业务字段或附件内容。
|
||||
- 腾讯云是 OA 生产;ARM 仅是异机备份、外部健康探测和隔离恢复演练节点。
|
||||
|
||||
## Attempt 1:前置门禁失败关闭
|
||||
|
||||
日期:2026-08-23
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 生产公网 `/ready`(演练前) | `PASS` |
|
||||
| 计划基线构建短标识 | `7b86ffb` |
|
||||
| 最新 ARM 正式日批次选择 | 已选择最新候选,未降级到旧批次 |
|
||||
| manifest 来源构建门禁 | `FAIL`:`build_mismatch` |
|
||||
| 失败阶段 | `select_latest_batch` |
|
||||
| finding 类别 | `latest_batch_invalid` |
|
||||
| 数据库/附件复制 | 未开始 |
|
||||
| 一次性 PostgreSQL / 恢复写入 | 未创建、未执行 |
|
||||
| runtime 清理 | `PASS` |
|
||||
| 临时 root 入口清理 | `PASS` |
|
||||
| 结论 | `NO_GO` |
|
||||
|
||||
该结果只证明最新候选未通过来源构建门禁,不能单独证明备份内容损坏。首次失败证据必须保留,禁止自动改选更旧批次或放宽门禁。
|
||||
|
||||
## 失败后只读复核
|
||||
|
||||
日期:2026-08-23
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 生产公网 `/ready` | `PASS`,构建短标识 `7b86ffb3` |
|
||||
| 生产自动备份 timer | `active` |
|
||||
| 生产自动备份 unit 冻结构建 | `7b86ffb3` |
|
||||
| ARM dead-man timer | `active` |
|
||||
| ARM 公网健康 timer | `active` |
|
||||
| ARM 一次性 root 入口残留 | `0` |
|
||||
| ARM 本轮 runtime 残留 | `0` |
|
||||
| ARM 普通用户暂存候选 | 窄名称初扫发现 `1`;随后按本轮 5 个文件的 SHA-256 精确匹配并全部删除 |
|
||||
|
||||
只读复核排除了“当前生产构建事实源过期”和“生产自动备份 unit 仍冻结旧构建”两种解释。ARM 上本轮 5 个普通用户暂存文件均经 SHA-256 与本地候选逐一匹配后精确删除,未执行宽泛目录删除。剩余调查重点是:最新 ARM 正式日批次是否来自当前生产自动备份链,以及 manifest 构建身份在生产生成、异机传输或 ARM 日归档选择中的哪一环发生漂移。
|
||||
|
||||
## Attempt 1 当时判定与重试门禁
|
||||
|
||||
- Attempt 1 仍为 `NO_GO`,当时 AC1、AC3、AC4、AC5 未满足;该次没有形成数据恢复 RTO/RPO 实测值。
|
||||
- 不重试恢复,不触碰 ARM 正式归档,不修改腾讯云生产数据或配置。
|
||||
- 下一步只允许以窄权限读取以下脱敏事实:最新 ARM manifest 的构建短标识、批次时间与最新生产成功状态是否指向同一 run id、manifest 摘要是否一致。
|
||||
- 若确认只是 ARM 最新批次来源/归档选择异常,另行规划修复备份链;若确认归档身份正确且仅演练器口径有误,先更新任务设计并重新批准。
|
||||
- 任何第二次恢复尝试前,必须更新规划、重新生成一次性入口并再次取得用户明确批准。
|
||||
|
||||
## 只读身份探针结果
|
||||
|
||||
日期:2026-08-23
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 最新 ARM manifest 构建短标识 | `ec682565` |
|
||||
| 当前生产/自动备份 unit 短标识 | `7b86ffb3` |
|
||||
| 构建匹配 | `NO` |
|
||||
| 最新批次年龄 | 约 18 小时,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件 manifest 大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 摘要 | 已取得 12 位脱敏前缀,不保存完整摘要 |
|
||||
| 探针临时 root 入口清理 | `PASS`,残留 `0` |
|
||||
| 普通用户暂存清理 | 3 个候选均按 SHA-256 精确删除,目录残留 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
|
||||
`ec682565` 是仓库可识别的历史提交,也是当前生产提交的祖先。随后只读对时确认:当前生产自动备份合同约 1.3 小时前才刷新,而最新 ARM 批次约 18 小时前生成;下一次自然调度约 14 小时后执行。因此最新 ARM 批次是在当前发布之前由当时活动构建生成,不支持“错误来源批次”或“传输损坏”的判断。
|
||||
|
||||
当前 finding 应归类为“发布后尚未产生新构建恢复点”。首次恢复入口按当时已批准的精确构建门禁正确失败关闭;该 `NO_GO` 是 Attempt 1 的真实历史结论,不覆盖也不回写。后续设计复盘确认,严格相等不是原始备份可恢复性的必要条件,新的准入合同见 2026-08-24 规划修订。
|
||||
|
||||
当时用户决策:不手工触发生产自动备份,等待既有 timer 自然生成并异机归档下一批恢复点。该决策不是完成项,也不永久排除手工备份;若后续自然批次异常,可重新作为需明确批准的备选项。
|
||||
|
||||
## 2026-08-24 设计复盘与有效产出
|
||||
|
||||
- 自 2026-08-23 对时到当前日期,下一次自然调度窗口已经过去;当前唯一可立即执行的动作是按新设计重新只读探测 ARM 最新批次,而不是继续等待。
|
||||
- Attempt 1 已经观测到:生产合同刷新后,到下一个自然备份抵达前,没有与当前构建严格相等的异机恢复点。该“同源恢复点可用滞后”是独立于数据 RPO/RTO 的有效恢复画像,即使数据库恢复未开始也必须保留。
|
||||
- `ec682565` 与 `7b86ffb3` 的 migration 目录均为 65 个,目录差异为 0;这证明本次历史批次与当前 schema 没有 migration 增量,但不能证明“65”适合作为长期硬编码断言。
|
||||
- 修订后的恢复点门禁为:manifest 来源提交是当前生产构建的已知祖先或相同,批次年龄不超过 24 小时,且 manifest/两份产物完整。严格相等只用于度量同源恢复点何时形成。
|
||||
- migration 验证改为来源提交目录名称集合与恢复库成功集合精确一致;来源/当前集合不同则把“升级到当前 schema”单独标为未验证,不在本轮临时执行 migration。
|
||||
|
||||
## 2026-08-24 最新自然批次只读探针
|
||||
|
||||
本节仅追加,不改写 Attempt 1 及 2026-08-23 既有证据。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 演练开始时公网 `/ready` | `PASS` |
|
||||
| 公网当前构建唯一解析 | `PASS`,短标识 `7b86ffb3` |
|
||||
| 生产自动备份 unit 与公网构建 | `MATCH`,timer `active` |
|
||||
| ARM 最新批次构建短标识 | `7b86ffb3` |
|
||||
| 来源与当前构建关系 | `SAME`;因此同时满足“祖先或相同” |
|
||||
| 批次年龄 | `68491` 秒,约 19 小时 2 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件 manifest 大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 摘要 | 已取得 12 位脱敏前缀,不保存完整摘要 |
|
||||
| 来源/当前 migration 目录 | 均为 65 个,差异文件 `0` |
|
||||
| 探针临时 root 入口 | 自清理 `PASS`,复核残留 `0` |
|
||||
| 普通用户暂存 | 3 个候选按 SHA-256 精确删除,目录残留 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
|
||||
同源恢复点画像:生产备份合同刷新距探针约 `94465` 秒;最新同源 manifest 距探针 `68491` 秒,因此该同源批次的 manifest 创建时间约在合同刷新后 `25974` 秒(约 7 小时 13 分)。当前证据没有直接采集 ARM 原子归档完成时刻,不能把该差值表述为精确“到达 ARM”耗时;只能证明同源批次创建滞后约 7 小时 13 分,并且在本次探针时已经可用。
|
||||
|
||||
执行方结论:最新自然批次满足修订后的来源关系、新鲜度、run id 配对和产物完整性门禁,可进入恢复重试审批。该结论不是恢复授权;本次只读探针没有复制或解包备份、创建 Docker 对象、恢复数据库/附件或修改生产与 ARM 正式归档。
|
||||
|
||||
## 2026-08-24 同源滞后锚点更正
|
||||
|
||||
本节追加说明前节 `25974` 秒的锚点,不改写既有记录。
|
||||
|
||||
- 起点实际是生产 `automatic-backup` service 构建 drop-in 的最大文件 mtime;终点是最新 manifest `createdAt`。
|
||||
- 该差值可称为内部诊断指标“合同 drop-in 文件刷新到同源 manifest 创建”,不能称为应用发布完成到恢复点到达 ARM 的精确耗时。
|
||||
- 当前证据没有采集 ARM 原子归档完成时刻,因此正式 `same_build_recovery_point_lag` 只能保留为“存在发布后空窗”的定性 finding,`7h13m` 不对外引用为精确恢复画像。
|
||||
- 2026-08-23 “下一次自然调度约 14 小时后”的估算存在时区处理错误:探针把 systemd 输出的本地时区时间直接附为 UTC,导致约多算 8 小时。修正后自然调度约在当时探针后 6 小时,与 2026-08-24 最新 manifest 创建时间一致;没有证据表明中间发生非计划手工备份。
|
||||
|
||||
## Attempt 2:临时副本目录身份误用
|
||||
|
||||
日期:2026-08-24
|
||||
|
||||
本节仅追加,不改写 Attempt 1、只读探针或锚点更正记录。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 最新正式批次前置门禁 | `PASS` |
|
||||
| 正式归档读取/修改 | 只读使用、零修改 |
|
||||
| 临时副本创建 | 已复制到本轮私有临时区 |
|
||||
| 失败阶段 | `create_isolated_runtime` |
|
||||
| finding 类别 | `copied_batch_invalid` / `batch_identity` |
|
||||
| PostgreSQL 容器 | 未创建 |
|
||||
| DB/附件恢复 | 未开始 |
|
||||
| migration/关键表/附件关联验证 | 未执行 |
|
||||
| runtime 清理 | `PASS`,复核残留 `0` |
|
||||
| 临时 root 入口清理 | `PASS`,复核残留 `0` |
|
||||
| 普通用户暂存清理 | 6 个候选按 SHA-256 精确删除,目录残留 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
| 结论 | `NO_GO` |
|
||||
|
||||
根因:runner 的 `validate_batch` 同时承担正式源目录身份和批次内容校验。正式源目录 basename 必须与 manifest `runId` 配对是正确安全门禁;复制后的私有临时目录固定名为 `batch`,内容仍保持同一 manifest/run id,却被同一个 basename 断言错误拒绝。该 finding 不表示备份损坏,也没有进入数据库或附件恢复。
|
||||
|
||||
修订边界:正式源目录继续执行 basename/run id 强校验;临时副本只跳过目录 basename 断言,manifest、run id 格式、文件集、构建、时间、大小、SHA-256 和与源元数据相等比较全部保留。任何重试前必须补“正式源通过、同内容改名副本通过、任一内容篡改失败”的合成回归并再次取得明确批准。
|
||||
|
||||
## 2026-08-25 Attempt 2 finding 分类补充
|
||||
|
||||
本节只追加分类,不改写 Attempt 2 原始结果。
|
||||
|
||||
- Attempt 2 finding 类别明确为 `runner_defect`,具体子类为“源目录身份断言误用于临时副本”。
|
||||
- Attempt 1 finding 是“发布后尚未形成同源恢复点”;Attempt 2 finding 是演练器缺陷。两者均未证明备份不可恢复,不能合并表述为“备份连续失败两次”。
|
||||
- Attempt 2 的正式批次前置门禁 `PASS`、临时复制完成、恢复前失败关闭、正式归档零修改和所有临时对象零残留,证明安全门禁在错误发生后按设计收口。
|
||||
- 临时副本修订不是简单放宽:副本 manifest `runId` 必须重绑到源侧已验证的 run id,并继续通过 manifest、文件集、构建、时间、字节数、SHA-256 及源/副本元数据相等校验。
|
||||
- 新增强制负例:源目录 basename 与 manifest run id 不配对必须 `NO_GO`;副本 manifest run id 即使内部自洽,但与源侧已验证 run id 不同也必须 `NO_GO`。
|
||||
|
||||
## 2026-08-25 runner 修复验证与最新批次探针
|
||||
|
||||
本节仅追加,不改写 Attempt 1、Attempt 2 或既有探针记录。
|
||||
|
||||
### Runner 修复验证
|
||||
|
||||
| 回归 | 结果 |
|
||||
| --- | --- |
|
||||
| 正式源 basename 与 manifest run id 配对 | `PASS` |
|
||||
| 同内容改名副本重绑源侧 run id | `PASS` |
|
||||
| 副本内容篡改 | 预期失败,`PASS` |
|
||||
| 源 basename 与 manifest run id 错配 | 预期失败,`PASS` |
|
||||
| 副本 manifest run id 内部自洽但不同于源 run id | 预期失败,`PASS` |
|
||||
| 动态 migration 集合正/反例 | `PASS` |
|
||||
| 既有隔离备份/恢复合同回归 | `PASS` |
|
||||
|
||||
实现结果:正式源目录继续校验 basename/run id;临时副本 manifest run id 必须等于源侧已验证 run id,并继续验证 manifest、文件集、构建、时间、大小、SHA-256 及源/副本脱敏元数据完全一致。
|
||||
|
||||
### 08-25 最新自然批次
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 演练开始时公网 `/ready` | `PASS` |
|
||||
| 公网构建与自动备份 unit | `MATCH`,短标识 `7b86ffb3`,timer `active` |
|
||||
| ARM 最新批次构建短标识 | `7b86ffb3` |
|
||||
| 来源与当前构建关系 | `SAME` |
|
||||
| 批次年龄 | `22249` 秒,约 6 小时 11 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件 manifest 大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 摘要 | 已取得 12 位脱敏前缀,不保存完整摘要 |
|
||||
| 来源/当前 migration 目录 | 均为 65 个,差异文件 `0` |
|
||||
| 探针 root 入口及普通用户暂存 | 残留均为 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
|
||||
该批次不是 Attempt 2 的旧候选,新鲜度和全部身份/完整性门禁已重新计算。当前满足 Attempt 3 恢复审批前提,但本次批准不包含实际恢复,未复制、解包或恢复该批次。
|
||||
|
||||
## Attempt 3:数据库恢复阶段失败关闭
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
本节仅追加,不改写 Attempt 1、Attempt 2 或既有探针记录。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 候选文件 SHA-256、Bash 与 sudoers 语法 | `PASS` |
|
||||
| 最新正式批次前置门禁 | `PASS`;使用 08-25 新候选 |
|
||||
| 正式归档读取/修改 | 只读取用;runner 没有归档写入路径 |
|
||||
| 临时副本创建及源侧身份重绑 | `PASS`;否则不会进入数据库恢复阶段 |
|
||||
| 失败阶段 | `restore_database` |
|
||||
| runner 聚合类别 | `database_restore_failed` |
|
||||
| migration/12 类表/附件恢复与关联验证 | 未执行 |
|
||||
| runtime 清理 | `PASS`,复核目录残留 `0` |
|
||||
| 一次性 PostgreSQL 容器清理 | `PASS`,复核残留 `0` |
|
||||
| 临时 root 入口与库清理 | `PASS`,复核残留 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
| 腾讯云生产公网 `/ready`(演练后) | `PASS`,构建短标识仍为 `7b86ffb3` |
|
||||
| 结论 | `NO_GO` |
|
||||
|
||||
Attempt 3 在数据库恢复命令返回非零后正确停止,未进入 migration、关键表、附件恢复或附件关联检查。当前输出不能区分 dump 列表校验失败、恢复命令失败、工具/参数合同不匹配或其他 runner 调用问题,因此该次 `NO_GO` 不能被改写为“备份不可恢复”,也不能据此给出数据恢复 RTO 或完整恢复结论。
|
||||
|
||||
本轮还暴露了独立的演练器可观测性缺陷:`restore-db.sh` 的详细输出仅写入权限 `0600` 的本轮私有日志,而失败清理随后删除该日志,最终只保留 `database_restore_failed` 聚合类别。安全清理是正确的,但失败关闭前必须先生成不含路径、凭据、业务内容或原始 SQL 的阶段化脱敏诊断;在该合同补齐并通过回归、重新审批前,不执行 Attempt 4,也不改选旧批次或放宽门禁。
|
||||
|
||||
措辞对齐说明:既有 Attempt 2 段落中的“临时副本只跳过目录 basename 断言”仅描述不把自建目录名当归档证据,不构成放宽。生效合同始终是副本 manifest run id 必须等于源侧已验证 run id,并且副本全部脱敏元数据与源侧首次校验结果完全一致。
|
||||
|
||||
## 2026-08-25 Attempt 3 后本地诊断修复
|
||||
|
||||
本节只追加本地 runner 与合成回归结果;未连接 ARM、未探测新批次、未执行 Attempt 4,也未修改腾讯云生产或 ARM 正式归档。
|
||||
|
||||
- runner 将数据库恢复拆为 `dump_list`、`target_confirmation`、`restore_apply` 三个阶段,只输出 `PASS/FAIL/NOT_RUN`、执行路径枚举、数字退出码、数字或 `UNKNOWN` 错误计数和稳定错误类别。
|
||||
- allowlist 诊断在 trap/finally 等价清理之前输出;原始 stderr 仍只存在于权限 `0600` 的本轮私有日志并随 runtime 删除。注入退出码 `23` 的回归证明:阶段诊断先输出,runtime/access 清理随后均为 `PASS`,调用方在清理后仍能读取数字退出码。
|
||||
- PostgreSQL 16 custom dump 直接驱动仓库 `scripts/restore-db.sh` 的 Docker stdin 路径成功;host-file 路径通过 PostgreSQL 16 工具执行同一真实脚本成功。没有在测试中重写等价恢复逻辑。
|
||||
- dump-list 失败、目标确认失败和 list/target 通过后 apply 失败三条负例均通过;后续阶段保持 `NOT_RUN`,apply 非零不会因对象错误计数较小而降级为成功。
|
||||
- 捕获诊断不含 fixture 凭据、连接串、SQL、临时路径或自由文本工具错误;既有合成成对备份/恢复回归通过,自动异机备份合同 `74` 项通过。
|
||||
|
||||
截至 Attempt 3,最新备份是否真的可恢复这个核心问题仍然没有答案。目前验证到的是演练器、恢复点身份门禁和每次失败后的安全收口,不是最新备份的数据可恢复性。Attempt 1 是发布后恢复点缺口;Attempt 2 是临时副本身份规则误用;Attempt 3 目前只能确定演练器可观测性不足,底层数据库恢复失败原因仍未知。三次均为 `NO_GO`,但不得把次数解释为“最新备份连续三次不可恢复”。
|
||||
|
||||
## 2026-08-25 Attempt 4 前最新批次只读探针
|
||||
|
||||
本节仅追加探针与本地 Git 静态核对结果;未复制、解包或恢复批次,未安装恢复 runner,未执行 Attempt 4。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 演练前公网 `/ready` | `PASS`,构建短标识 `7b86ffb3c9ef` |
|
||||
| ARM 最新批次探针 | `PASS` |
|
||||
| 最新批次构建短标识 | `7b86ffb3` |
|
||||
| 来源提交唯一解析 | `PASS` |
|
||||
| 来源与当前生产构建关系 | `SAME` |
|
||||
| 批次年龄 | `26652` 秒,约 7 小时 24 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ee1f97a5b42d` |
|
||||
| 来源/当前 migration 集合 | `65/65`,差异条目 `0` |
|
||||
| 仓库 HEAD 相对生产构建的应用代码漂移 | 文件数 `0` |
|
||||
| 探针 root 入口清理 | `PASS`,残留 `0` |
|
||||
| 普通用户探针暂存 | 残留 `0` |
|
||||
| ARM 两个监控 timer | `active` |
|
||||
|
||||
该批次满足 Attempt 4 的构建关系、新鲜度、run id 配对、产物完整性和 schema 集合前置门禁;该结论仍不是恢复授权,也不回答最新备份是否可恢复。
|
||||
|
||||
恢复 runner 的持久化限制不予接受:截至本节时,修复后的 runner 与八类本地回归仍位于不提交的个人 workspace,仓库尚无可由其他执行者独立复跑的 runner 合同门禁。Attempt 4 实际恢复前必须先把 runner 的稳定来源和对应合同回归纳入 `scripts/qa/`,由合同脚本机械验证候选 SHA-256、三阶段诊断、源/副本身份、真实 `restore-db.sh` 双路径和失败清理;未完成前不安装 Attempt 4 恢复入口。
|
||||
|
||||
## 2026-08-25 runner 持久化合同结果
|
||||
|
||||
本节仅追加本地跟踪文件与合同回归结果;未连接 ARM、未渲染现场候选、未执行 Attempt 4。
|
||||
|
||||
新增跟踪文件:
|
||||
|
||||
- `scripts/qa/offhost-shadow-restore-runner.template.sh`
|
||||
- `scripts/qa/validate-offhost-shadow-restore-attachments.py`
|
||||
- `scripts/qa/run-offhost-shadow-restore-runner-contract.sh`
|
||||
|
||||
| 合同类别 | 结果 |
|
||||
| --- | --- |
|
||||
| 私有参数渲染、确定性候选 SHA-256、shell 注入拒绝 | `PASS` |
|
||||
| 正式源身份与改名私有副本源侧重绑正例 | `PASS` |
|
||||
| 源 basename/run id 错配、副本篡改、副本 run id 与源不同负例 | `PASS` |
|
||||
| dynamic migration 集合指纹正/反例,无硬编码数量 | `PASS` |
|
||||
| 附件本地路径/大小/SHA-256 正例,路径逃逸与 S3 负例 | `PASS` |
|
||||
| 三阶段诊断、执行路径、两个退出码、后续 `NOT_RUN` | `PASS` |
|
||||
| 真实 `scripts/restore-db.sh` Docker stdin 与 PostgreSQL 16 host-file 路径 | `PASS/PASS` |
|
||||
| apply 非零失败关闭、诊断先于清理、清理后可读、无敏感自由文本 | `PASS` |
|
||||
|
||||
持久化模板不含远端地址、个人用户目录、远端账号、现场路径、实际容器/数据库名、生产构建值或 migration 指纹;这些值只能来自私有 JSON,并在候选写出前通过精确字段集、类型、字符集、绝对路径、shadow 命名和摘要格式校验。`check-sensitive-content.mjs` 对三个新增文件定向扫描结果为 `PASS`,扫描文件 `3`、finding `0`;本地测试容器残留 `0`。
|
||||
|
||||
Attempt 4 的 runner 产生过程已从个人 workspace 人工产物改为“跟踪模板 + 跟踪合同 + 私有参数确定性渲染 + 候选 SHA-256”。合同未通过不得生成或上传候选。该结果只补齐演练器可重复性,仍不构成最新备份可恢复性结论或 Attempt 4 恢复授权。
|
||||
|
||||
## 2026-08-25 私有渲染与阶段日志门禁补强
|
||||
|
||||
本节仅追加本地门禁修订与合成回归结果;未连接 ARM、未创建现场私有配置、未渲染或上传 Attempt 4 候选,也未执行实际恢复。
|
||||
|
||||
- `.gitignore` 已增加专用 `/.restore-drill-private/` 目录;合同分别注入仓库内未忽略的私有 JSON 和未忽略的候选输出路径,两者均在候选创建前按预期失败。ignored 目录内的合成输入/输出正例通过,测试文件在退出时清理。
|
||||
- 私有原始日志仍为 `0600` 且只在初始化时截断一次;PostgreSQL 容器启动、dump list、目标确认、restore apply 和附件恢复使用固定阶段标记追加。合同静态拒绝后续截断,最终 runtime 清理与只输出 allowlist 诊断的边界不变。
|
||||
- PostgreSQL 16 fixture 增加连续成功查询的稳定就绪门禁,避免镜像首次初始化临时服务关闭窗口造成合成回归假失败;真实 `scripts/restore-db.sh` 字节未修改,Docker stdin 和 host-file 两条路径仍直接执行仓库脚本并通过。
|
||||
- 本轮新增门禁回归全部 `PASS`;完成完整质量门禁并形成可追溯 Git 提交前,仍不得准备或上传 Attempt 4 候选。
|
||||
|
||||
完整本地门禁随后通过:runner 合同全部 `PASS`,既有隔离备份/恢复合成演练 `PASS`,自动异机备份合同 `74/74 PASS`;task validate、Trellis governance、process baseline、默认敏感扫描和三个新增文件定向敏感扫描均 `PASS`。文档审计错误 `0`,仅保留既有未登记文件 warning;`shellcheck` 在本机不可用,已以 Bash 语法、真实双路径集成回归和仓库合同覆盖。测试容器、本地私有渲染目录及候选残留均为 `0`。
|
||||
|
||||
## Attempt 4 环境差异预先裁决规则
|
||||
|
||||
本节在 Attempt 4 新批次探针和实际恢复之前追加,不能按执行结果事后改写。
|
||||
|
||||
- Attempt 2 的源/副本身份重绑和 Attempt 3 的三阶段诊断已经由提交 `5ec7db1` 下的可复跑合同机械把关;合同失败时不得生成或上传候选。
|
||||
- 若同一提交的合同、候选 SHA-256 和 ARM 安装前摘要均通过,但 Attempt 4 再次在恢复完成前暴露演练器侧失败,则默认 finding 升级为“本地合同环境与 ARM 实际执行环境存在系统性差异”,不再直接归为又一个孤立 runner 缺陷。
|
||||
- 后续调查必须先形成脱敏环境差异矩阵,至少比较 Docker/Podman 实现与版本、PostgreSQL 16 镜像摘要和工具版本、shell/coreutils/openssl/python 行为、文件系统与权限/SELinux/AppArmor、可用 PATH、恢复脚本及依赖摘要、网络模式和资源约束。未取得差异证据前,不继续逐点放宽或修补 runner。
|
||||
- 该规则不把任何失败预判为 ARM 环境问题,也不改变备份内容判定。若阶段诊断明确证明 dump、数据库对象、migration 或附件内容不满足既有合同,仍按对应数据恢复 finding 如实分类。
|
||||
|
||||
## 2026-08-25 Attempt 4 执行前最终探针与候选
|
||||
|
||||
本节仅追加最新批次只读探针、候选本地渲染与合同结果;未上传或安装恢复候选,未执行 Attempt 4 实际恢复,未写腾讯云生产或 ARM 正式归档。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| ARM 最新正式批次探针 | `PASS`;动态重选后仍为同一最新批次 |
|
||||
| 最新批次构建短标识 | `7b86ffb3` |
|
||||
| 来源与当前生产构建关系 | `SAME` |
|
||||
| 批次年龄 | `38572` 秒,约 10 小时 43 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ee1f97a5b42d` |
|
||||
| 探针临时 root 入口/sudoers 残留 | `0/0` |
|
||||
| 普通用户探针暂存 | 再次通过 SHA 清单后逐一删除,残留 `0` |
|
||||
| ARM dead-man/健康 timer | `active/active` |
|
||||
| 来源/当前 migration 目录 | `65/65`,差异条目 `0` |
|
||||
|
||||
候选由提交 `5ec7db1` 的 `scripts/qa/offhost-shadow-restore-runner.template.sh` 和 ignored 私有 JSON 确定性渲染。私有 JSON 权限为 `0600`、精确 20 个字段,runner 权限为 `0700`,输入与输出均命中仓库根锚定 ignore。两次独立渲染的字节和 SHA-256 完全一致;候选 SHA-256 为 `81f4eca2cf173196e1a57b781d28629412de7b7fe1a09b001b02d2025950391c`,合同报告值与本地 `sha256sum` 重算一致。
|
||||
|
||||
完整 runner 合同再次全部 `PASS`:源/副本身份、三条身份负例、dynamic migration、附件安全、五阶段追加日志、失败先诊断后清理、真实 `restore-db.sh` Docker stdin/host-file 双路径、退出码和敏感输出边界均通过。私有候选包共 6 个文件(runner、附件校验器、两份现有恢复脚本、sudoers、SHA 清单),包内 SHA 清单复核 `PASS`;runner 钉住的三份依赖摘要全部匹配,sudoers 固定命令与 runner 自身路径一致,`scripts/restore-db.sh` 相对生产构建仍无字节漂移,本地测试容器残留 `0`。
|
||||
|
||||
准备过程有一次安全失败:首次来源 migration 枚举误把 `migration_lock.toml` 当作 migration 名称,现场显示 `66/65`。该错误发生在候选渲染前,没有产生候选;随后改为只读取来源提交 `server/prisma/migrations` 树下的目录对象,并与当前工作树目录集合独立比较,结果为 `65/65`、差异 `0` 后才生成私有配置。Attempt 4 若获批准,安装前仍须在 ARM 复核上述候选包 SHA 清单;批次年龄和最新批次身份也须以实际执行开始时为准,超过 24 小时或最新批次变化时重新探针,不沿用本节结论。
|
||||
|
||||
## Attempt 4:目标数据库确认阶段失败关闭
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
本节仅追加 Attempt 4 原始脱敏结果,不改写前置探针、候选合同或环境差异预先裁决规则。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| ARM 安装前候选包 SHA 清单 | `PASS` |
|
||||
| 正式批次与私有副本前置门禁 | 已通过,否则不会进入 `restore_database` |
|
||||
| dump list | `PASS`,退出码 `0` |
|
||||
| target confirmation | `FAIL` |
|
||||
| 失败阶段/类别 | `restore_database` / `restore_target_confirmation_failed` |
|
||||
| 执行路径 | `DOCKER_STDIN` |
|
||||
| restore apply | `NOT_RUN` |
|
||||
| restore 脚本/真实 pg_restore 退出码 | `NOT_RUN/NOT_RUN` |
|
||||
| pg_restore 错误计数 | `NOT_RUN` |
|
||||
| 稳定错误类别 | `TARGET_CONFIRMATION_FAILED` |
|
||||
| migration/12 类关键表 | `NOT_RUN/NOT_RUN` |
|
||||
| 附件恢复与业务关联 | `NOT_RUN` |
|
||||
| runtime/临时 root 入口清理 | `PASS/PASS` |
|
||||
| 普通用户候选暂存 | `PASS`,残留 `0` |
|
||||
| 结论 | `NO_GO` |
|
||||
|
||||
`dump_list=PASS` 只证明最新 custom dump 可由 ARM 的 PostgreSQL 16 工具读取目录;`target_confirmation=FAIL` 发生在实际恢复调用之前,因此本次没有数据库数据写入、migration 查询、关键表读取、附件解包或附件关联校验。数据恢复 RTO 未形成,实际执行时的精确 RPO 秒数未被失败输出保留;只能确认 runner 已在创建 runtime 前通过不超过 24 小时的硬门禁,不能用前置探针年龄冒充执行时精确 RPO。
|
||||
|
||||
本轮输出没有分别保留 target query 的退出码和“查询成功但数据库名不等”布尔值,现有证据尚不能区分连接/认证/工具调用失败与目标名称比较失败。正式归档在 runner 中只有读取和复制路径,未观察到归档写入;但失败发生在末尾源批次复校验之前,因此“正式归档零修改”仍需后续只读摘要复核,不以代码静态结论替代现场证据。演练后生产 `/ready` 也须另行补充,不能从 ARM timer 或清理 PASS 推断。
|
||||
|
||||
按 Attempt 4 开始前已提交的裁决规则,本次先登记为 `local_contract_vs_arm_environment_difference_investigation`,不是新的孤立 runner 缺陷,也不是备份不可恢复证据。下一步仅允许建立脱敏环境差异矩阵和补齐生产/归档只读复核;在差异证据、修订设计和新批准之前,不修改 runner、不创建 Attempt 5、不复用本批次盲重试。
|
||||
|
||||
## Attempt 4 失败后的只读环境差异调查
|
||||
|
||||
本节仅追加失败后的只读 ARM 信息与本地合成复现;未在 ARM 启动调查容器、未读取业务数据、未修改 runner 或正式归档。
|
||||
|
||||
| 维度 | 本地合同环境 | ARM 实际环境 | 判定 |
|
||||
| --- | --- | --- | --- |
|
||||
| CPU/容器架构 | `x86_64` / `linux/amd64` | `aarch64` / `linux/arm64` | 不同 |
|
||||
| Docker client/server | `25.0.4/25.0.4` | `28.0.2/28.0.2` | 不同 |
|
||||
| PostgreSQL 16 镜像实例摘要前缀 | `fff3594bf464` | `c05eced0bdb4` | 多架构实例不同 |
|
||||
| 存储/cgroup/rootless | `overlay2/systemd/no` | `overlay2/systemd/no` | 相同 |
|
||||
| Bash/Python/OpenSSL/coreutils | `5.1.4/3.9.2/1.1.1w/8.32` | `5.1.4/3.9.2/1.1.1w/8.32` | 相同 |
|
||||
| AppArmor/SELinux | `Y/unavailable` | `Y/unavailable` | 相同 |
|
||||
| 工作文件系统 | `ext2/ext3` 类 | `ext2/ext3` 类 | 相同 |
|
||||
| 必需命令 | 合同执行通过 | `PASS` | 无缺失证据 |
|
||||
|
||||
差异矩阵本身不能证明 Docker 28 或 arm64 有缺陷。真正收敛根因的是启动时序覆盖不一致:tracked runner 在启动 PostgreSQL 后以第一次 `pg_isready` 成功结束等待;本地合同夹具曾真实出现“初始化临时服务正在关闭”的失败,之后夹具改成连续两次真实 `SELECT 1` 成功才继续,但 runner 的等待实现没有同步修改。于是合同环境稳定化后绕过了 runner 的竞态,而 ARM 仍执行首次 `pg_isready` 路径。
|
||||
|
||||
失败序列与该机制一致:dump list 不连接目标数据库,因此可在 PostgreSQL 初始化服务切换窗口中 `PASS/0`;紧随其后的 target query 需要真实连接,命中关闭/重启窗口即进入 `TARGET_CONFIRMATION_FAILED`,actual restore 尚未调用。本地使用相同镜像配置、相同随机凭据和 runner 中完全相同的 target query,在连续两次真实查询确认稳定后得到退出码 `0`、数据库名精确匹配、stderr 为空;本地测试容器随后清理为残留 `0`。
|
||||
|
||||
当前高置信度根因类别为 `postgres_initialization_readiness_race`,上位 finding 仍为开跑前定义的 `local_contract_vs_arm_environment_difference_investigation`。系统性差异具体表现为:合同夹具和 runner 没有复用同一稳定就绪路径,架构/容器版本造成的时序差异把该覆盖缺口暴露在 ARM。修复方向应是 runner 与合同共同执行“连续成功的真实目标数据库查询”并保留 target query 数字退出码与名称匹配布尔值,而不是为 ARM 增加一次固定 sleep、放宽目标确认或再次只修测试夹具。该修复和任何 Attempt 5 均须先更新设计、补可复跑负例并重新批准。
|
||||
|
||||
失败后只读残留复核:一次性容器、runtime、root 入口、恢复库目录、sudoers 和普通用户暂存均为 `0`;ARM dead-man 与公网健康 timer 均为 `active`。演练后生产 `/ready` 与正式归档摘要在此时仍待独立只读补证。
|
||||
|
||||
## Attempt 4 后正式归档只读复核
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
本节追加 Attempt 4 后正式归档只读探针结果,不回写前置探针或 Attempt 4 原始失败记录。探针不复制、不解包、不恢复、不启动数据库容器,也不修改正式归档。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 只读探针 | `PASS`,退出码 `0` |
|
||||
| 最新批次构建短标识 | `7b86ffb3`,与当前生产构建相同 |
|
||||
| 探针时批次年龄 | `42007` 秒,约 11 小时 40 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件产物完整性 | 大小与 SHA-256 聚合 `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ee1f97a5b42d` |
|
||||
| 探针临时访问清理 | `PASS` |
|
||||
| 探针入口/暂存残留 | `0/0` |
|
||||
|
||||
Attempt 4 前后探针的最新构建短标识、run id 配对、产物完整性和 manifest 脱敏摘要前缀一致,后置探针仍在 24 小时内通过。由此正式归档的内容级零修改已取得现场只读补证;该结论仅覆盖正式归档,不替代恢复成功证明,也不回答最新备份是否可恢复。
|
||||
|
||||
腾讯云生产演练后 `/ready` 仍未独立补证。ARM 的 timer active、清理 PASS 和归档探针均不能推导生产在线状态;在取得 `/ready` 正常状态及 12 位构建前缀前,AC2 保持未满足,证据不得表述为生产前后均正常。
|
||||
|
||||
## Attempt 4 后生产只读补证与本地稳定就绪修复
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
执行方沿既有审计窄路径完成腾讯云生产演练后 `/ready` 只读核验:状态 `PASS`,12 位构建前缀 `7b86ffb3c9ef`,与演练前记录一致。地址和原始响应未写入本证据;该读取没有登录、写库、重启、切换或配置修改。至此 Attempt 4 的生产前后 `/ready` 均正常已补证,但不改变本轮 restore apply 未执行和最新备份可恢复性仍未知的结论。
|
||||
|
||||
用户随后批准的范围仅为本地 runner 修复与回归,不连接 ARM、不执行 Attempt 5。修复把 PostgreSQL 就绪从默认 Unix socket 的首次 `pg_isready` 改为容器内 TCP 回环 `127.0.0.1` 的真实 `SELECT current_database()`:initdb 临时服务器不监听 TCP,因此该判据结构性排除临时服务器;连续两次成功和最多 60 次有界重试只作为额外稳定保险。目标名称成功但不匹配立即失败,查询非零重置 streak;实际 `restore-db.sh` 仍走原路径且字节不改。
|
||||
|
||||
合同夹具与 runner 分叉审计结果:
|
||||
|
||||
| 覆盖对象 | 合同机制 | 是否存在等价逻辑分叉 |
|
||||
| --- | --- | --- |
|
||||
| 私有配置渲染与候选 SHA | 直接使用跟踪模板和合同内唯一渲染器 | 否 |
|
||||
| 正式源/临时副本身份 | 从 rendered runner 提取 `validate_batch` 同一实现 | 否 |
|
||||
| PostgreSQL 目标就绪 | 从 rendered runner 提取同一 TCP 目标确认函数;真实 PostgreSQL 16 fixture 也调用该函数 | 原分叉已消除 |
|
||||
| DB restore Docker stdin/host-file | 直接执行仓库 `scripts/restore-db.sh` | 否 |
|
||||
| 附件引用安全 | 直接执行跟踪附件校验器 | 否 |
|
||||
| 失败诊断与 finally 清理 | 从 rendered runner 提取初始化和 finalization 同一实现 | 否 |
|
||||
| migration 指纹 | 对 rendered 值、目录集合变化和禁止硬编码做静态/合成断言 | 无等价重写;真实恢复集合仍待远端演练 |
|
||||
| 12 类表、storage 与完整编排 | 本地只做组件合同,不执行含真实备份的完整 runner | 明确覆盖限制,不能推断远端成功 |
|
||||
|
||||
就绪回归全部 `PASS`:socket 路径假定可用但 TCP 永久失败仍在第 60 次失败;首次查询失败后连续两次成功在第 3 次通过;一次成功后失败会清零 streak 并在第 4 次通过;目标名不匹配第 1 次立即失败;永久查询非零以 `TARGET_QUERY_NONZERO` 失败。真实 PostgreSQL 16 fixture 使用 extracted rendered-runner 函数就绪后,`restore-db.sh` Docker stdin 与 host-file 两条路径均成功;dump-list、目标错配、apply 非零、诊断先于清理、敏感输出负例继续通过。本地测试容器残留 `0`。
|
||||
|
||||
完整复跑结果:runner 合同全部 `PASS`,既有成对备份/恢复演练 `PASS`,自动异机备份合同 `74/74 PASS`;task validate、Trellis governance、process baseline、默认/定向敏感扫描和 `git diff --check` 均通过。文档审计错误 `0`,仅保留既有未登记文件 warning;`scripts/restore-db.sh` 相对生产构建无字节漂移,测试容器和生成缓存残留均为 `0`。
|
||||
|
||||
本地合同通过只证明演练器修复与组件路径,不回答最新正式备份是否可恢复。下一步仍须先提交可追溯变更,再动态选择最新正式批次、执行只读身份探针并重新渲染候选;Attempt 5 实际恢复必须另行批准。
|
||||
|
||||
## Attempt 5 前最新正式批次只读身份探针
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
本节仅追加用户批准的最新正式批次只读探针与本地仓库关系核对;未执行恢复、未复制或解包正式归档、未启动 Docker、未读取数据库或附件内容。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 探针结果 | `PASS`;退出码未单独输出 |
|
||||
| 最新批次构建短标识 | `7b86ffb3` |
|
||||
| 来源与当前生产构建关系 | `SAME` |
|
||||
| 探针时批次年龄 | `64625` 秒,约 17 小时 57 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ee1f97a5b42d` |
|
||||
| 来源/当前 migration 目录 | `65/65`,差异 `0` |
|
||||
| 仓库 HEAD 相对生产构建的应用代码漂移 | 文件数 `0` |
|
||||
| 探针 root 入口/普通用户暂存残留 | `0/0` |
|
||||
| ARM dead-man/健康 timer | `active/active` |
|
||||
|
||||
timer 状态只采用已验证私有配置中的实际 unit 名只读取得;一次使用猜测名称得到的无效读数已当场撤销,未进入证据结论。探针结果表明该最新批次继续满足构建、新鲜度、身份和完整性前置门禁,但仍不是恢复成功证明。
|
||||
|
||||
Attempt 5 最终候选尚未渲染。原因是用户提出的 RTO 分阶段记录已进入设计、但 runner 实现尚未另行批准;若先渲染,后续计时代码会改变候选字节和 SHA-256。必须先裁决并完成 RTO 八阶段本地合同,再从跟踪模板重新确定性渲染候选并核对 SHA-256,实际恢复仍须单独批准。
|
||||
|
||||
## Attempt 5 前八阶段 RTO 本地合同
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
用户随后批准的范围仅为本地实现八阶段 RTO 计时并执行合同回归;本阶段未连接 ARM、未渲染现场候选、未生成 Attempt 5 候选 SHA-256,也未执行实际恢复。此前探针批次的 `64625` 秒新鲜度只属于探针时点,后续不得复用,实际恢复前必须重新选择最新正式批次并重跑只读身份门禁。
|
||||
|
||||
runner 现在以连续边界依次记录 `batch_copy`、`postgres_container_start`、`dump_list`、`target_confirmation`、`restore_apply`、`database_verify`、`storage_attachment_verify` 和 `final_integrity`。同一个时间点关闭前一段并开启后一段;成功路径八段均为非负整数且精确加总为唯一的 `data_restore_rto_seconds`。失败路径先结算当前阶段,保留已完成和当前阶段的数字,后续阶段为 `NOT_RUN`,然后才清理私有 runtime。最长 60 次 TCP 目标确认及其 1 秒间隔只位于 `target_confirmation` 边界内。
|
||||
|
||||
可复跑合同直接驱动 rendered runner 的同一计时函数。确定性成功夹具得到八段 `3/5/2/11/7/13/17/3` 秒,总计 `61` 秒且总字段只输出一次;目标确认失败夹具保留 `2/4/1/9` 秒、后四段 `NOT_RUN`、部分总计 `16` 秒,并证明计时输出位于 runtime cleanup 之前;restore apply 失败夹具记录当前段 `12` 秒、后续三段 `NOT_RUN`、部分总计 `28` 秒。捕获输出不含路径、连接串、凭据或 SQL。
|
||||
|
||||
完整本地门禁结果:runner 合同全部 `PASS`;真实 PostgreSQL 16 fixture 和未修改的 `scripts/restore-db.sh` Docker stdin/host-file 双路径继续通过;既有成对备份/恢复演练 `PASS`;自动异机备份合同 `74/74 PASS`;task validate、Trellis governance、process baseline、docs registry、敏感扫描和 `git diff --check` 通过。文档审计错误 `0`,保留既有未登记文件 warning;本地测试容器残留 `0`,`scripts/restore-db.sh` 相对生产构建仍无字节漂移。
|
||||
|
||||
Attempt 5 前增强停止线从本节起生效:八阶段计时完成并通过合同后,下一份满足 24 小时门禁的新鲜批次直接进入最终候选渲染、SHA-256 核对、只读探针和实际恢复审批。此后发现的改进项,除非会导致 Attempt 5 失败或产生错误结论,否则只登记为后续任务,不再作为 Attempt 5 前置增强插队。本地合同通过仍只证明演练器合同,不回答最新正式备份是否可恢复。
|
||||
|
||||
## Attempt 5 最终候选与执行前只读探针
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
本阶段只执行用户批准的三件事:重新选择最新正式批次、从跟踪模板渲染最终候选并核对 SHA-256、运行一次自清理只读身份探针。未复制或解包正式归档、未启动恢复容器、未执行数据库或附件恢复,也未连接腾讯云生产。
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| runner 跟踪提交 | `0b94b97` |
|
||||
| 两次确定性渲染 | 字节一致,完整合同 `PASS` |
|
||||
| 最终候选 SHA-256 | `7904052aa5a8c31d54734bc2fbf2d9caf2dac8b0116135119519f8467b3dae27` |
|
||||
| 探针结果 | `PASS` / exit `0` |
|
||||
| 最新批次构建短标识 | `7b86ffb3`,与当前生产 `SAME` |
|
||||
| 探针时批次年龄 | `68889` 秒,约 19 小时 08 分,`within_24h=yes` |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ee1f97a5b42d` |
|
||||
| 来源/当前 migration | `65/65`,差异 `0`,候选指纹匹配 |
|
||||
| 应用代码/restore-db 字节漂移 | `0/0` |
|
||||
| 探针 access/entry/staging 残留 | `PASS/0/0` |
|
||||
| 本地合同容器残留 | `0` |
|
||||
|
||||
探针时距 24 小时门禁约剩 `17511` 秒,即约 4 小时 52 分。该余量不被静态批准延长;Attempt 5 runner 在实际开始时仍会动态选择最新正式批次并重新校验构建、run id、产物 SHA-256 和年龄。若执行被推迟到候选超龄且尚无更新批次,runner 必须在复制前失败关闭,不能沿用本节探针结论。
|
||||
|
||||
候选钉住的是 `EXPECTED_BUILD_PREFIX` 和 `EXPECTED_MIGRATION_SET_SHA`,不是本次探针批次的 source run id。runner 会在 ARM 上于实际执行时重新选择当时最新正式批次,再对实际批次执行构建前缀、目录/run id、三文件集合、大小、SHA-256、RPO 和恢复后 migration 集合门禁。如果探针后有更新批次原子落地,实际恢复输入可以合法切换到更新批次;最终证据必须记录实际批次的脱敏身份,并明确它与本次探针批次是否一致,不能仅因候选 SHA-256 不变就默认恢复了探针批次。
|
||||
|
||||
至此 Attempt 5 的前置批次身份、演练器合同、八阶段 RTO 和失败诊断均已具备;最新正式备份是否实际可恢复仍未得到答案,必须由另行批准的 Attempt 5 隔离恢复给出结论。
|
||||
|
||||
## Attempt 5 实际结果与安装器路径根因
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
用户明确批准 Attempt 5 后,执行方在 ARM 安装前再次核对候选 SHA-256,完整值与已批准候选一致。紧邻执行前的只读探针在演练调用同一秒观测到最新正式批次年龄 `70444` 秒,来源构建与当前生产相同,run id/目录配对、DB/附件产物完整性、source digest 和 24 小时门禁均为 `PASS`。探针反推的批次创建时刻已作为操作侧脱敏时间证据保留。
|
||||
|
||||
runner 随后在 `entry_contract` 阶段以 `restore_library_missing` 失败关闭:`dump_list`、目标确认和 restore apply 均为 `NOT_RUN`,八个 RTO 阶段与总计也全部为 `NOT_RUN`,最终结果原样保留为 `NO_GO`。失败发生在 runner 选择实际恢复批次之前,因此本轮不存在可对外记录的“实际恢复批次”,也不能用紧邻探针批次代替实际恢复输入。正式归档复制、PostgreSQL 容器启动、数据库恢复、migration/12 类表核对、附件恢复与关联验证均未开始;这次结果没有证明最新备份可恢复,也没有证明备份不可恢复。
|
||||
|
||||
本次 finding 分类为 `installer_path_mismatch`,属于演练安装器缺陷。已批准 runner 的依赖合同指向固定 `libexec` 目录,本轮一次性安装器却把三个依赖安装到名称相似但不同的 `lib` 目录。runner 因而在正确位置找不到依赖并按入口合同失败关闭;其 `temporary_access_cleanup=PASS` 清理的是合同内目标,而错误目录未被该清理函数命中,这解释了初始 `attempt5_entry_residue=1` 与 runner 自报清理通过并存的原因。
|
||||
|
||||
对错误目录的定向只读诊断得到:`restore-db.sh`、`restore-storage.sh` 和附件校验器均存在、均为 root 拥有的普通文件,三者 SHA-256 脱敏前缀与本地安装包一致,额外条目数为 `0`。在目标身份和文件集合全部明确后,仅对该错误安装目录执行精确清理,结果 `runner_lib_cleanup=PASS`;该动作未使用宽泛目录匹配,也未触碰正式归档。
|
||||
|
||||
诊断脚本的自删除另外暴露出一个安装器层问题:脚本在 sudo 上下文中以 `$HOME` 推断普通用户暂存路径,实际解析为 root 家目录,因目标不存在而报 `unlink` 失败。这不影响错误依赖目录已清理的结论;随后由非 root 外层对普通用户暂存中的已知诊断脚本做精确删除。后续安装器不得在 sudo 脚本中使用 `$HOME` 推断调用用户路径,应由安装器预先固定并验证普通用户暂存目标,或由非 root 外层负责自清理。
|
||||
|
||||
演练后腾讯云生产 `/ready` 由既有审计窄路径只读取得 `PASS`,12 位构建前缀仍为 `7b86ffb3c9ef`,与演练前一致。Attempt 5 未进入正式归档选择和复制路径,也未启动容器或恢复写入,因此腾讯云生产与正式归档未被本次失败修改。
|
||||
|
||||
在任何下一次实际恢复前仍需做一次最终只读零残留复核,范围为:runner 入口、合同内 `libexec`、错误 `lib`、runner/probe sudoers 及 candidate、probe 入口、普通用户安装器/候选/诊断暂存、Docker 容器和 runtime,并确认 dead-man 与健康 timer 仍为 `active/active`。在该复核完成前,不把 Attempt 5 表述为“全部临时对象零残留”。
|
||||
|
||||
最小修订方案只涉及 ignored/workspace 安装器:将三项依赖的安装目标从错误 `lib` 对齐到 runner 已批准的 `libexec`,并修正普通用户暂存脚本的清理所有权;不修改已批准 runner,不重新渲染候选,候选 SHA-256 可保持不变。该安装器修订、本地校验、最新批次只读探针和任何再次实际恢复均需按既有批准边界分别推进。核心问题——最新正式备份能否完成数据库与附件隔离恢复——在 Attempt 5 后仍无答案。
|
||||
|
||||
## Attempt 5 最终零残留复核与安装器最小修订
|
||||
|
||||
日期:2026-08-25
|
||||
|
||||
用户批准后,执行方沿既有 ARM 窄路径运行固定目标的最终复核入口。该入口只读取本轮已知入口、依赖目录、sudoers、暂存、容器/runtime 和两个监控 timer 的聚合状态,不读取正式归档或业务数据,不执行恢复。结果如下:
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| runner/probe 入口、sudoers 与 candidate 残留 | `0` |
|
||||
| runner 合同内 `libexec` 残留 | `0` |
|
||||
| 安装器误用的 `lib` 残留 | `0` |
|
||||
| 普通用户安装器/候选/诊断暂存残留 | `0` |
|
||||
| 本轮容器残留 | `0` |
|
||||
| 本轮 runtime 残留 | `0` |
|
||||
| dead-man/健康 timer | `active/active` |
|
||||
| 最终复核 | `PASS` |
|
||||
| 复核脚本自身清理 | `PASS` |
|
||||
|
||||
至此 Attempt 5 初始 `entry_residue=1` 已由定向诊断、精确清理和独立最终复核闭环为全部已知临时对象残留 `0`。该清理结论不改变 Attempt 5 的 `NO_GO`,也不补造任何未运行的 RTO、数据库或附件恢复证据。
|
||||
|
||||
已批准的本地最小修订只改变 ignored/workspace 安装器,不进入 Git 历史:三项依赖安装目标由错误 `lib` 对齐到已批准 runner 中同一 `libexec`,静态检查确认安装器与 rendered runner 的目录类别一致;诊断脚本以自身调用路径作为自清理目标,并在删除前验证固定 basename、普通文件且非链接,不再读取 sudo `$HOME`。三个私有脚本通过 Bash 语法检查和 Git ignore 守卫;已批准 runner、附件校验器、恢复脚本和候选字节均未修改,候选 SHA-256 不变。
|
||||
|
||||
本地完整复核结果:影子恢复 runner 合同全部 `PASS`,真实 PostgreSQL 16 fixture 的 Docker stdin/host-file 路径继续通过,合成成对备份/恢复演练 `PASS`,自动异机备份合同 `74/74 PASS`;task validate、Trellis governance、process baseline、敏感扫描和 `git diff --check` 通过。文档审计错误 `0`,只保留既有未登记文件 warning;`scripts/restore-db.sh` 相对生产构建字节漂移 `0`,本地测试容器残留 `0`。
|
||||
|
||||
任何再次实际恢复仍须重新选择满足 24 小时门禁的新鲜批次、重新执行只读身份探针并取得独立批准。本节只闭环 Attempt 5 的环境清理和安装器路径缺陷,不回答最新正式备份是否可恢复。
|
||||
|
||||
## 08-27 最新正式批次只读身份探针
|
||||
|
||||
日期:2026-08-27
|
||||
|
||||
本节只追加用户批准的最新正式批次只读探针。未复制、解包或恢复正式归档,未启动 Docker,未修改腾讯云生产、ARM 正式归档、timer 或企业微信。
|
||||
|
||||
探针准备前完成了安装器共享路径静态核对:runner 的 `SELF_PATH`、`LIB_DIR`、`SUDOERS_PATH` 与安装器目标精确一致,probe 的 `SELF_PATH`、`SUDOERS_PATH` 与探针安装器精确一致;`WORK_ROOT` 只由 runner 持有,安装器没有第二份编码;sudoers candidate 由最终 sudoers 路径推导。上述比较均位于首次 `sudo install` 之前,失败即不产生特权写入。
|
||||
|
||||
第一次上传准备因本地编排器错误截取 basename,三个文件以截短名称进入普通用户暂存;远端 SHA-256 门禁立即失败。该失败发生在 sudo 和探针执行之前,未安装 root 入口、未读取归档。确认暂存目录恰含三个预期的截短普通文件后,逐个精确删除并移除本轮暂存和安装器,再使用真实 basename 重新上传;重传包的两项 SHA-256、Bash 语法和三文件集合均通过。该 finding 属于 `local_probe_upload_orchestration_defect`,未影响 probe/runner 字节或 ARM 正式状态。
|
||||
|
||||
现场探针结果:
|
||||
|
||||
| 检查项 | 结果 |
|
||||
| --- | --- |
|
||||
| 探针执行 | `PASS` / exit `0` |
|
||||
| 最新批次构建短标识 | `7b86ffb3`,与当前生产相同 |
|
||||
| 最新批次年龄 | `124137` 秒,约 34 小时 29 分 |
|
||||
| 24 小时门禁 | `FAIL`;超出 `37737` 秒,约 10 小时 29 分 |
|
||||
| run id 与目录配对 | `PASS` |
|
||||
| DB/附件大小与 SHA-256 | `PASS` |
|
||||
| source image digest | 存在 |
|
||||
| manifest 脱敏摘要前缀 | `ede8a77d9912` |
|
||||
| 探针合同内临时访问清理 | `PASS` |
|
||||
| 探针入口/普通用户暂存残留 | `0/0` |
|
||||
|
||||
`probe_result=PASS` 只表示只读探针完整执行并成功校验所见批次,不等于恢复前置门禁通过。按 PRD 的 24 小时硬门禁,本次结论为 `NO_GO/stale_latest_batch`:不得执行实际恢复,也不得改选更旧批次掩盖最新恢复点不新鲜。构建、身份和产物完整性通过说明当前证据不指向归档损坏;但仅凭本次探针不能区分源端自动备份未产出、传输/finalize 失败或调度时点变化。
|
||||
|
||||
下一步须另行批准只读调查生产自动备份最近运行状态、ARM receiver 最新原子归档时刻以及 dead-man/健康 timer 状态。该调查不得手工触发备份或修改 timer。只有新的正式批次满足 24 小时门禁并重新通过身份探针后,才能再次申请实际隔离恢复批准;既有恢复授权不延续。
|
||||
|
||||
## 08-27 自动备份链与 dead-man 只读调查
|
||||
|
||||
日期:2026-08-27
|
||||
|
||||
用户批准的调查只读取生产自动备份 systemd 聚合状态、ARM receiver 原子归档/inbox 聚合状态、dead-man 状态和两端文件系统容量。未触发备份、未修改 timer、未执行恢复、未改正式归档、未发送测试告警,也未输出远端身份、现场路径、原始 JSON、journal、凭据或完整 run id。
|
||||
|
||||
生产侧结果:daily timer 与 watchdog timer 均为 `active`,最近分别在当日计划时点触发;对应 service 最近一次均为 `failed/result=exit-code/exit=1`。因此“timer 未调度”被排除,失败位于两项服务共用的备份运行合同或其后续路径。批准的部署账号不具备 sudo,也不能穿越 root-owned 状态/本地备份目录;调查明确记录 `ACCESS_DENIED`,没有把无权读取改写成状态文件或本地批次不存在。该权限边界使生产侧原始失败 phase 保持未知,不据此猜测具体 stderr。
|
||||
|
||||
ARM 侧结果:dead-man timer 与公网健康 timer 均为 `active`;dead-man service 最近一次按失败状态退出。实际 dead-man 阈值为显式参数 `93600` 秒(26 小时)。`last-deadman` 为 `FAIL_OFFHOST_DEADMAN/archive_stale/findingCount=1`,alert state 同为 `FAIL_OFFHOST_DEADMAN/archive_stale`。最近 72 小时 journal 仅做 allowlist 聚合,得到 `sent=true` 一次、`sent=false` 两次,最近一次为 `sent=true`。该次 dead-man timer 触发锚点为 2026-08-27 06:30:06(Asia/Shanghai),alert state 在同轮随即更新,可作为本次发送调用时刻。恢复演练探针在同日稍后首先促成人工响应,但当时未独立采集“人员实际知晓”的精确墙钟时刻,因此本轮不能伪造精确响应延迟,只能登记为“发送调用早于人工响应、精确差值未测”。后续事件任务必须同时记录 `sent=true` 与首次确认时刻,直接计算真实响应延迟。
|
||||
|
||||
检测侧结论为正面:真实超龄被正确检测,告警 helper 至少完成了一次发送调用,且相同失败的 `sent=false` 属于正常去抖;08-23 去抖改造没有造成检测或发送调用盲区。响应侧仍有 finding:本次备份中断最终由恢复演练探针触发处置,而不是由既有告警形成及时响应。证据不证明具体人员已阅读消息,后续应调查群消息送达、可见性、责任人确认和升级流程,不能把检测调用成功等同于闭环响应。
|
||||
|
||||
receiver 聚合结果:正式原子日归档共 7 个,最新归档在复核时年龄约 43 小时 40 分;inbox 有 2 个 `.partial`,正式归档事务临时对象为 0、inbox 其他对象为 0。两个 partial 均不满足“三个普通文件 + manifest/run 身份 + 产物字节数”的完整传输合同,完整合同数量为 0、不完整数量为 2;最新 partial 年龄约 19 小时 40 分,最旧 partial 约 31 天 10 小时。由此将当前事故分类为 `offhost_transfer_incomplete`:生产 daily 确实触发并在 ARM 留下未完成传输,不能归为源端完全未产出、timer 漏调度或“传输已经完整但 finalize 失败”。partial 的存在同时证明 rsync 已通过初始 SSH 认证、host key 校验和 receiver 授权并开始写入,因此密钥失效、authorized_keys 初始拒绝和 host key 初始拒绝不是本次首要候选;排查应收缩到传输中段的超时/网络中断、源端读取、目标端写入或会话/传输限制。最旧 partial 同时是历史清理/观测 finding,但本轮没有删除它。
|
||||
|
||||
当前只读字段还不足以在上述传输中段候选之间继续裁决。下一轮独立生产事件调查应比较两个 partial 的聚合字节大小/停止比例,读取 wrapper 私有诊断中的真实 rsync 退出码,并记录每轮传输持续时间:相近停止偏移或相近时长支持确定性限制,不同偏移/时长更支持网络或超时;service 的通用 exit `1` 不得替代 rsync 原始码。该调查仍只输出字节聚合、退出码和耗时,不输出文件名、路径或 stderr 原文。
|
||||
|
||||
容量必须与传输修复并行处理:生产侧相关路径位于同一可识别文件系统,使用率约 82%、可用约 10 GiB;ARM archive/inbox 文件系统使用率约 53%、可用约 89 GiB。现有证据仍不足以把这次 rsync 中断直接归因于磁盘不足,但自动备份会先在生产生成完整 dump 与 storage 归档,持续失败还会反复生成/保留本地产物;10 GiB 对该流程属于薄余量。若继续下降,故障可能从“传输中断”升级为“源端无法产出”,因此容量治理不是低优先级后续项,而是独立生产事件中的并行处置项。
|
||||
|
||||
调查器本身出现并安全关闭了两类本地缺陷:首次生产探针错误假设部署账号可 sudo,安装前即被权限拒绝;前两次 ARM 探针把 dead-man unit 写成近似但不存在的名称,均在状态读取前以合同不可读失败。每次 root 入口、sudoers candidate、普通用户暂存均为 0。最终版本从仓库 receiver 安装器机械提取并比较 unit basename,并将生产探针改为无提权只读执行;Bash/Python 语法、两种 ExecStart 解析 fixture 和敏感扫描通过。上述缺陷不改变生产、ARM、归档、timer 或告警状态,也没有伪造调查结果。
|
||||
|
||||
当前独立生产 finding 为:最新可用异机恢复点已超过 24 小时目标,自动备份传输链正在失败;恢复演练因此保持阻塞。该事件不再作为恢复演练的普通 finding 排队处理,建议另建 Trellis 生产事件任务承载传输/容量调查与修复,当前演练任务仅保留证据交叉引用和阻塞关系。后续优先顺序是另行批准修复生产到 ARM 的传输链并形成新的原子批次,确认自动备份/两端 dead-man 与 watchdog 恢复,再对新批次重新执行身份探针。手工补跑备份、修改网络/SSH/timer、清理 partial 或发送测试告警均不在本次只读批准内。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "isolated-backup-restore-drill",
|
||||
"name": "isolated-backup-restore-drill",
|
||||
"title": "隔离备份恢复演练",
|
||||
"description": "在不覆盖生产的隔离环境验证最新数据库与附件备份可恢复性,并记录RTO/RPO和脱敏证据。",
|
||||
"status": "in_progress",
|
||||
"dev_type": null,
|
||||
"scope": "operations",
|
||||
"package": null,
|
||||
"priority": "P2",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-23",
|
||||
"completedAt": null,
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "2026-08-28:恢复演练保持 in_progress/P2,但被 P1 operations 事件 08-28-production-arm-backup-transfer-incident 阻塞。该事件负责生产到 ARM 传输链修复、生产容量处置、告警响应计时和连续两个自然调度周期验证;在新原子批次形成并重新通过 24 小时身份门禁前,本任务不得继续实际恢复。既有恢复候选 SHA 仍有效,无需因事件任务重新渲染。",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,105 @@
|
||||
# 生产到 ARM 备份传输中断设计
|
||||
|
||||
## 设计结论
|
||||
|
||||
采用“先只读判因、再最小变更、最后两个自然周期验证”的事件处置路径。传输修复和容量处置属于同一个 P1 operations 任务,但每项远端写操作维持独立审批;恢复演练作为下游消费者保持阻塞,不参与本任务的根因修复。
|
||||
|
||||
## 证据链与判定矩阵
|
||||
|
||||
```text
|
||||
生产 daily timer 触发
|
||||
-> 本地成对批次生成
|
||||
-> rsync 建立 SSH/receiver 会话
|
||||
-> ARM inbox/<batch>.partial 写入
|
||||
-> rsync 完成
|
||||
-> receiver finalize 原子发布
|
||||
-> ARM dead-man + 生产 watchdog 判定新鲜
|
||||
```
|
||||
|
||||
当前证据已经到达 `.partial 写入`,所以初始 SSH 认证、host key 与 receiver 授权不是首要分支。后续只读调查按以下矩阵裁决:
|
||||
|
||||
| 证据 | 主要指向 | 保留边界 |
|
||||
| --- | --- | --- |
|
||||
| 两个 partial 停止偏移接近,失败时长也接近 | 确定性大小/读取/会话上限 | 结合 rsync 原始码确认,不直接改超时 |
|
||||
| 停止偏移差异明显,时长不收敛 | 网络或间歇超时 | 读取客户端/服务端安全状态,不抓原始 payload |
|
||||
| rsync `23` | 部分传输/源或目标文件错误 | 结合私有阶段类别,不输出 stderr |
|
||||
| rsync `30` | 超时 | 定位是哪一层超时,禁止无限放宽 |
|
||||
| rsync `12` | 协议/数据流错误 | 核对双方版本和会话中断证据 |
|
||||
| rsync `11` | 文件 I/O | 联查源读取、目标写入和容量 |
|
||||
| SSH/包装层 `255` 且已有 partial | 会话在启动后断开 | 不误判为初始认证失败 |
|
||||
| partial 满足完整三文件/manifest/字节合同 | finalize 未完成 | 当前现场为 0,不作为默认假设 |
|
||||
|
||||
## 只读采集合同
|
||||
|
||||
- 生产侧先用 manifest/三文件/身份/字节元数据枚举全部本地候选,只对最新一个元数据合格候选完整读取 database/storage 并计算 SHA-256;旧候选明确输出 `older_metadata_batch_hash_status=NOT_RUN`,不得为了计数连续读取数 GiB。最新候选只有通过完整 SHA-256 合同才输出 `complete_local_recovery_point=yes`;其哈希失败时不得现场回退扫描旧候选,而应保守标为“无已证明可用备份”。输出年龄、元数据合格数量和聚合大小,不输出 run id、文件名或路径。该结果先于传输根因分析,决定当前是“生产单点”还是“无已证明可用备份”。ARM 仍对正式归档执行完整哈希合同。
|
||||
- 对成功与失败路径分别输出本地完整批次、生成期临时对象和保留策略的聚合结果,明确 `retained_after_failure`、`pruned_after_success` 和连续失败累积趋势;不从单次目录计数猜测脚本语义。
|
||||
- partial 只输出每个匿名样本的总字节数、预期总字节数、安全比例和年龄;不输出名称、路径、run id 或 artifact 名称。
|
||||
- partial 停止偏移的“接近”必须同时满足相对差异不超过 `50000 ppm`(5%)和绝对差异不超过 `67108864` bytes(64 MiB),等价于以两者较小值作为上限。不得用两者较大值,否则小样本与零字节样本会被绝对阈值错误吞并为 `similar`。
|
||||
- 生产私有诊断只输出固定的 `transfer_exit_code`、`failure_category`、开始/结束 epoch 和持续秒数。若现有状态没有持久化真实 rsync 码或开始时刻,记录 `NOT_AVAILABLE`,需要本地设计可观测性修订并另行批准部署,不能现场改脚本。
|
||||
- 容量只输出 filesystem 使用率/可用 GiB、完整/不完整/临时批次数量和聚合字节,不列目录。
|
||||
- 采集器先探测账号能力;无 sudo 时不得尝试临时扩权。需要 root-owned 状态时先提交固定 allowlist 入口设计并单独审批。
|
||||
|
||||
若上述静态/历史只读证据仍无法收敛,停止追加远端读取项。下一候选步骤是一次单独审批的受控观测性重传:冻结一个已校验本地批次,只允许一次传输尝试,实时记录固定阶段、rsync/SSH 原始码、开始/结束 epoch 和聚合字节进度,设置有界超时且失败不重试。它是诊断性写操作,不属于第一阶段只读授权。
|
||||
|
||||
## 容量预算
|
||||
|
||||
生产修复前冻结以下聚合量:最近成功完整本地批次大小、一次备份生成期间的峰值临时增量、当前保留批次总量和文件系统可用量。最低通过线为:
|
||||
|
||||
```text
|
||||
available_before_schedule >= observed_full_batch_peak * 2
|
||||
```
|
||||
|
||||
其中 `observed_full_batch_peak` 包含 database dump、storage archive、manifest 和生成期临时开销;两倍口径提供一个新批次的实际需求和等量安全余量。若现场无法可靠测得生成期峰值,则使用“最近最大完整批次聚合大小 × 2 + 已知最大临时开销”的更保守口径。任何降低该门槛的决定须重新审阅 PRD;扩容优先于删除唯一恢复证据。
|
||||
|
||||
## 变更与回滚边界
|
||||
|
||||
- 根因明确后只设计一个最小变更集,例如有界传输超时、具体 I/O 修复、网络会话修复或容量扩展;不同根因不打包试错上线。
|
||||
- 修改 systemd/SSH/rsync 配置前保存脱敏旧值指纹和固定回滚命令;不输出配置正文。
|
||||
- 清理自动备份自有对象时必须先以固定模式、文件类型、所有者、时间和引用关系验证身份,逐个命中;不使用宽泛递归清理。
|
||||
- 手工补跑仅用于经单独批准的修复后快速验证,不计入两个自然周期;失败时保留新的脱敏 finding,不连续重试。
|
||||
|
||||
## 告警响应计时
|
||||
|
||||
从本任务开始使用两个独立 epoch:
|
||||
|
||||
- `alert_sent_epoch`:helper 返回 `sent=true` 所在调用时刻;
|
||||
- `human_ack_epoch`:指定责任人首次明确确认收到并开始处置的时刻。
|
||||
|
||||
`response_seconds = human_ack_epoch - alert_sent_epoch`。缺任一锚点即 `NOT_MEASURED`,不得用文件 mtime、对话时间或事后回忆替代。该指标只评价响应闭环,不改变 dead-man 的检测正确性。
|
||||
|
||||
## 验收状态机
|
||||
|
||||
```text
|
||||
PLANNING
|
||||
-> READ_ONLY_DIAGNOSIS_APPROVED
|
||||
-> ROOT_CAUSE_EVIDENCED
|
||||
-> CHANGE_PLAN_APPROVED
|
||||
-> FIX_APPLIED
|
||||
-> FIRST_ATOMIC_BATCH_PASS
|
||||
-> NATURAL_CYCLE_1_PASS
|
||||
-> NATURAL_CYCLE_2_PASS
|
||||
-> INCIDENT_GO
|
||||
```
|
||||
|
||||
任何一步失败都停在当前状态并保留证据。`FIRST_ATOMIC_BATCH_PASS` 形成一个满足 24 小时门禁的新鲜原子批次后,即可解除对恢复演练的事件阻塞;恢复演练仍需自己的身份探针和实际恢复审批。P1 事件本身只有到 `NATURAL_CYCLE_2_PASS` 才能结案,两条线不互相等待。
|
||||
|
||||
## 过渡恢复点决策
|
||||
|
||||
- `complete_local_recovery_point=yes`:若下个自然周期在可接受窗口内且容量预算允许,默认 `wait_natural`;仍可基于风险窗口单独申请手工补跑,但不是自动动作。
|
||||
- `complete_local_recovery_point=no`:事件升级为当前无已证明可用备份,默认建议 `request_manual_run`,但必须先保证根因不会使补跑无界失败、容量满足停止条件,并单独审批。
|
||||
- 任何手工补跑都只建立过渡恢复点和验证修复,不替代两个自然周期。
|
||||
|
||||
## 安全与脱敏
|
||||
|
||||
- 腾讯云生产和 ARM 地址、账号、绝对现场路径、完整 run id/digest、密钥、known_hosts、webhook、原始日志和业务内容不进入仓库或聊天。
|
||||
- 远端临时探针若获批,固定目标、SHA-256、语法、sudoers 和自清理均需机械校验,并由外层独立报告入口/暂存残留。
|
||||
- timer `active`、partial 存在、单次备份成功或告警 `sent=true` 都是局部证据,不能扩写为链路整体正常。
|
||||
|
||||
## 2026-08-28 只读调查裁决
|
||||
|
||||
- 生产本地存在一个通过完整哈希合同的最新可用批次,风险分类为 `production_single_point`,不是“当前无已证明可用备份”;元数据合格批次 7 个,失败传输后保留本地批次、异机成功后才 prune,过渡动作保持 `wait_natural`。
|
||||
- ARM 原始调查输出曾将两个 partial 判为 `similar`。该输出保留作为原始证据;现场聚合字节为 `2682168` 与 `0`,按修正后的“双阈值同时满足”合同人工重算为 `different`。一个老 partial 已满足完整合同但未 finalize,另一个较新 partial 为零字节,两者不支持“固定相同偏移的确定性传输上限”,且可能来自不同失败类型。
|
||||
- 现有状态未持久化本次失败的 rsync 原始退出码和传输专属时钟,均保持 `NOT_AVAILABLE`;只读证据不足以证明单一根因。若后续自然周期再次失败,按既定边界转为单独审批的一次受控观测性重传,不扩大只读读取范围来补造结论。
|
||||
- 调查时 ARM 已出现约 12 小时 43 分的新鲜完整原子归档,dead-man 最近 service 为 `success/0` 且状态为 healthy,说明异机链已自然恢复一次;该事实可解除恢复演练的事件阻塞,但不满足本 P1 事件的两个自然周期结案条件。
|
||||
- 私有探针输出的 `arm_deadman_alert_state_finding_count=invalid` 是表示缺陷,不是 ARM 状态损坏:`last-deadman.json` summary 才有 `findingCount`,`deadman-alert-state.json` 按稳定 helper 合同只保存 `schemaVersion/status/category`。后续探针不得对 alert state 索取不存在的 count 字段。
|
||||
- ARM health service 同期为 `failed/1`,而生产公网 `/ready` 调查前后均为 PASS、12 位构建前缀一致。该 finding 指向健康监控执行、状态或通知链,不得扩写为 OA 当前不可用;进一步读取或处置需另行授权。
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,90 @@
|
||||
# 生产到 ARM 备份传输中断实施计划
|
||||
|
||||
## Phase 0:规划与授权
|
||||
|
||||
- [x] 创建 P1、`operations` Trellis task,记录来源事件和与恢复演练的阻塞关系。
|
||||
- [x] 固定只读调查、生产/ARM 写操作、手工补跑和测试告警为不同审批边界。
|
||||
- [x] 用户审阅并批准本规划及补充项后运行 `task.py start`;该批准只授权进入本地实施,不自动授权连接远端。
|
||||
|
||||
## Phase 1:本地只读调查准备
|
||||
|
||||
- [x] 从 `scripts/run-automatic-backup.sh`、receiver、dead-man 和 systemd installer 提取真实单位名、状态路径来源、阶段和退出码合同,禁止手写近似常量。
|
||||
- [x] 实现只输出 allowlist 的跟踪只读采集器:生产本地完整恢复点合同、本地成功/失败清理保留语义、partial 匿名字节比例、rsync 原始码、无法取得的传输时钟、两端容量;timer/service 继续复用既有只读 systemd 聚合合同,不在 Python 中另写一套。
|
||||
- [x] 为本地完整/无完整批次、成功清理/失败保留、相近/不同停止偏移、rsync `23/30/12/11/255`、缺失持续时间、权限拒绝、畸形 manifest 和敏感输出补合成回归。
|
||||
- [x] 本地语法、合同、敏感扫描全部通过后,提交远端只读调查摘要与精确权限边界,单独申请执行批准。
|
||||
|
||||
### 本地准备结果(2026-08-28)
|
||||
|
||||
- 跟踪调查器:`scripts/qa/inspect-backup-transfer-incident.py`;仅接受显式绝对路径参数,不回显参数值,未知异常也只输出稳定类别。
|
||||
- 可复跑合同:`scripts/qa/run-backup-transfer-incident-contract.sh`;初始 19 项合成检查通过;现场分类缺陷修复后扩展为 20 项,fixture 与 Python cache 零残留。权限拒绝不会被误写成“无有效备份”,其他阶段的旧日志不会冒充本轮 rsync 诊断;生产多个批次时只哈希最新元数据合格候选,最新哈希失败不回退扫描旧批次;停止偏移必须同时满足 5% 和 64 MiB 双阈值。
|
||||
- 既有 `run-automatic-backup-contract.sh` 74 项通过;生产 wrapper 未改。现状没有持久化传输专属开始/结束时刻,故持续时间必须为 `NOT_AVAILABLE`。
|
||||
- rsync 原始码只在安全摘要明确为 `phase_offhost_transfer_failed` 时从 root 私有 `last-error.log` 固定正则提取,避免把 paired-backup/finalize 的旧诊断错误归因给传输。
|
||||
|
||||
## Phase 2:获批的远端只读调查
|
||||
|
||||
- [x] 在不修改生产/ARM 的前提下采集四类证据;每个临时入口在同轮精确清理并由外层独立复核零残留。
|
||||
- [x] 先判定“生产单点”或“无已证明可用备份”,再交叉判定主要根因、次要候选、清理/累积语义和容量预算;无法取得的字段标为 `NOT_AVAILABLE/NOT_MEASURED`。
|
||||
- [x] 记录过渡恢复点决策;只读证据无法收敛时停止扩读,提交一次受控观测性重传的独立审批方案,不直接执行。
|
||||
- [ ] 更新本 task 证据,给出单一最小修复方案、回滚方案和生产写操作清单;停止并请求用户审阅。
|
||||
|
||||
### 远端只读调查结果(2026-08-28,追加时间线)
|
||||
|
||||
- 公网 `/ready` 调查前后均 PASS,12 位构建前缀均为 `7b86ffb3c9ef`;生产固定入口与 staging 经外层独立枚举均为 0,ARM `arm_independent_residue=0`。调查未改变在线 OA、正式归档、timer 或业务数据。
|
||||
- 生产:`complete_local_recovery_point=yes`、`backup_risk_classification=production_single_point`;完整哈希通过的最新批次 1 个、元数据完整批次 7 个,最新候选年龄 42885 秒、聚合 11400180 bytes;生产文件系统 82% 使用、10 GiB 可用。源码顺序证明传输失败保留完整本地批次、异机成功后 prune、生成临时对象在失败路径清理,故过渡动作是 `wait_natural`。
|
||||
- 生产最近 daily/watchdog service 均为 `success/0`、timer active;历史失败摘要未保留可绑定到本轮的 rsync 原始码或传输专属时钟,因此退出码和时长均为 `NOT_AVAILABLE`,不得用 service exit、mtime 或对话时刻替代。
|
||||
- ARM 原始输出:partial 2 个,聚合 2682168 bytes,样本总字节分别为 2682168 与 0;调查器当时输出 `arm_partial_stop_offset_class=similar`。该原始结果不回写。
|
||||
- 分类缺陷修订:旧实现以 `max(64 MiB, 5%)` 为接近上限,使 2.68 MB 与 0 字节被 64 MiB 阈值错误吞并。按固定阈值必须同时满足的合同,人工重算为 `different`;一个约 32 日龄 partial 满足完整合同但未 finalize,另一个约 36 小时龄 partial 为零字节,不能据此声称两次都停在同一确定性偏移。
|
||||
- ARM 正式原子归档 7 个、无无效归档,最新完整归档年龄 45756 秒;inbox/archive 文件系统 54% 使用、89 GiB 可用;dead-man service `success/0`、timer active、最近状态 `PASS_OFFHOST_DEADMAN_FRESH/healthy`。链路已自然恢复一个新鲜原子批次,可解除恢复演练的事件阻塞,但 P1 仍需第二个连续自然周期及容量验收后才能结案。
|
||||
- `arm_deadman_alert_state_finding_count=invalid` 不表示远端状态损坏。仓库 helper 明确把 alert state 写成仅含 `schemaVersion/status/category` 的去抖状态;`findingCount` 属于 `last-deadman.json` summary。私有探针错误复用了 summary 读取模板,后续证据不得把这个表示缺陷升级为告警链 finding。
|
||||
- 独立 finding:ARM health service 为 `failed/1`,但公网 `/ready` 实测前后一致且 PASS。这是健康监控执行/状态/通知链问题,不是 OA 当前不可用的证据;本次授权不覆盖进一步调查或修改。
|
||||
- 根因裁决:历史单一 rsync 原始码和可靠传输时长不可得,两个 partial 的真实停止位置明显不同,现有证据不能收敛为单一根因。不得把自然恢复一次冒充“已实施修复”;后续若再失败,候选动作是另行批准的一次受控观测性重传。生产 10 GiB 余量仍需独立容量方案。
|
||||
|
||||
### 分类缺陷复盘
|
||||
|
||||
- 根因类别是“隐含假设 + 测试覆盖缺口”:实现用 `max(absolute, relative)` 把两个独立上限误写成二选一宽松上限;旧正例夹具只证明“绝对差异较小”,没有证明相对差异也在 5% 内,因而为错误实现背书。
|
||||
- 修复机制已闭环:实现改为 `min(...)`;合同直接覆盖现场 `2682168/0`、同时满足双阈值的正例、分别越过相对/绝对阈值的负例;稳定 spec 写明公式、错误写法与断言点。仓库搜索确认该分类公式只有这一个实现位置,不存在需要同步的第二套逻辑。
|
||||
- 原始远端 `similar` 保留,人工修订追加为 `different`;以后任何现场分类缺陷都按同样的不可变时间线处理,不回写旧证据。
|
||||
|
||||
## Phase 3:本地修复与变更审批
|
||||
|
||||
- [ ] 仅修改证据支持的脚本/installer/合同;不轮换已通过初始认证的密钥,不放宽 manifest/finalize 门禁。
|
||||
- [ ] 容量方案优先扩容或精确处置已确认的自动备份自有对象,证明不会触碰唯一恢复点、数据库或附件源。
|
||||
- [ ] 运行自动异机备份、receiver、dead-man、故障注入、回滚和敏感输出合同。
|
||||
- [ ] 列出生产/ARM 的每个写目标、旧值指纹、候选 SHA、执行/回滚命令和停止条件,单独申请实际变更批准。
|
||||
|
||||
## Phase 4:获批的实际修复
|
||||
|
||||
- [ ] 执行最小变更和容量处置;不做未批准的手工备份、timer 修改、partial 清理或测试告警。
|
||||
- [ ] 若另获手工补跑批准,只运行一次并记录真实 rsync 码、时长、原子 finalize 和残留;失败不自动重试。
|
||||
- [ ] 验证至少一个新原子批次、manifest/三文件/字节/SHA-256、三组 service exit `0` 和容量预算。
|
||||
|
||||
## Phase 5:两个自然周期观察
|
||||
|
||||
- [ ] 自然周期 1:daily 新原子批次 PASS;watchdog/dead-man service success;记录生产/ARM 容量、ARM partial 数量/聚合字节趋势和适用的告警响应锚点。
|
||||
- [ ] 自然周期 2:重复同一组断言;不以手工补跑或旧批次替代;ARM partial/容量趋势不得继续恶化。
|
||||
- [ ] 两个周期均通过后标记事件 `GO`,否则保留失败并返回 Phase 2/3,不临时放宽阈值。
|
||||
|
||||
## Phase 6:解除阻塞与收口
|
||||
|
||||
- [ ] 在 `08-23-isolated-backup-restore-drill` 追加本任务结果交叉引用并解除事件阻塞;恢复演练仍需新批次身份探针和单独实际恢复审批。
|
||||
- [ ] 运行 task validate、Trellis governance、process baseline、docs registry、敏感扫描和 `git diff --check`。
|
||||
- [ ] 中文提交引用 `08-28-production-arm-backup-transfer-incident`;不自动 push,不归档未满足两个自然周期的任务。
|
||||
|
||||
## 停止条件
|
||||
|
||||
- 只读操作将输出路径、身份、凭据、原始日志/JSON 或业务内容。
|
||||
- 需要新增 sudo、读取超出 allowlist 的 root 状态或改变生产/ARM 状态但尚未获独立批准。
|
||||
- 清理对象、配置目标、旧值或回滚路径不唯一。
|
||||
- 新原子批次不完整、service 非零、容量低于预算或任一自然周期失败。
|
||||
- 告警发送/人工确认锚点缺失时不得补造响应延迟。
|
||||
|
||||
## 验证命令
|
||||
|
||||
- `bash -n` / `python3 -m py_compile` 覆盖新增或修改的运维脚本。
|
||||
- 运行自动备份、receiver、dead-man 和相关敏感输出合同。
|
||||
- `python3 ./.trellis/scripts/task.py validate .trellis/tasks/08-28-production-arm-backup-transfer-incident`
|
||||
- `node scripts/qa/check-trellis-governance.mjs`
|
||||
- `node scripts/qa/check-process-contract.mjs --baseline --json`
|
||||
- `node scripts/qa/audit-docs.mjs --check-registry`
|
||||
- `node scripts/qa/check-sensitive-content.mjs`
|
||||
- `git diff --check`
|
||||
@@ -0,0 +1,113 @@
|
||||
# 生产到 ARM 备份传输中断
|
||||
|
||||
## Goal
|
||||
|
||||
恢复腾讯云生产到 ARM 的自动异机备份链,消除持续扩大的异机恢复点缺口,同时把生产备份落盘容量恢复到可持续范围,并证明修复不是一次偶然成功。
|
||||
|
||||
## Scope
|
||||
|
||||
- 只读定位 partial 停止偏移、rsync 原始退出码、失败持续时间、生产本地批次累积与容量预算。
|
||||
- 基于证据设计并在另行批准后实施最小传输链修复和生产容量处置。
|
||||
- 验证新 ARM 原子归档、daily/watchdog/dead-man 的 timer 与 service,并连续观察两个自然调度周期。
|
||||
- 建立真实告警发送调用时刻、人工确认时刻和响应延迟的脱敏记录合同。
|
||||
- 维护与 `08-23-isolated-backup-restore-drill` 的阻塞交叉引用;不在本任务内执行恢复演练。
|
||||
|
||||
## Background
|
||||
|
||||
- 腾讯云服务器是真实 OA 生产;ARM 是异机备份、dead-man 和隔离恢复节点,不是生产。
|
||||
- 2026-08-27,任务 `08-23-isolated-backup-restore-drill` 的最新批次探针发现 ARM 最新原子归档已超过 24 小时目标;随后获批的只读调查确认生产 daily/watchdog timer 正常触发,但对应 service 最近均以 `exit-code/1` 失败。
|
||||
- ARM inbox 有两个未完成 `.partial`,其中最新一个与最近自然备份时点对应;两者均未满足三文件、manifest 身份和产物字节数的完整传输合同,正式归档事务临时对象为零。因此当前 finding 为 `offhost_transfer_incomplete`,不是调度漏跑或完整传输后的 finalize 失败。
|
||||
- partial 已证明 rsync 通过初始 SSH 认证、host key 校验和 receiver 授权并开始写入。密钥失效、`authorized_keys` 初始拒绝和 host key 初始拒绝不是首要假设;仍需调查传输中段的源端读取、网络/超时、目标写入和会话/传输限制。
|
||||
- ARM dead-man 以显式 `93600` 秒阈值正确判定 `archive_stale`,且 journal 聚合存在一次 `sent=true`。检测与发送调用没有被去抖静默抑制,但该告警没有形成及时人工响应;本次首次知晓时刻未采集,历史响应延迟不可补造。
|
||||
- 生产相关文件系统只读观测约为 82% 使用、10 GiB 可用。现有证据不能把本次传输中断直接归因于空间不足,但该余量不足以作为持续生成完整 dump、storage 归档和失败重试的安全基线,必须与传输故障并行处置。
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1. 分阶段授权与数据保护
|
||||
|
||||
- 任务创建和规划不授权连接生产或 ARM,也不授权任何远端写入。
|
||||
- 第一阶段只读调查必须另行批准,只允许读取聚合字节数、rsync 原始退出码、固定阶段时刻/耗时、systemd 状态、源端本地批次聚合和文件系统容量;不得输出远端地址、完整路径、run id、文件名、原始 stderr/journal/JSON、凭据或业务内容。
|
||||
- 清理空间、删除或移动本地/ARM partial、调整 SSH/rsync/systemd 参数、修改 timer、重启服务、手工触发备份、重跑传输或发送测试告警均属于生产/ARM 写操作,必须在根因和变更方案审阅后单独明确批准。
|
||||
- 不修改或删除已有 ARM 正式原子归档;诊断不得把真实备份下载到开发机。
|
||||
|
||||
### R2. 只读根因定位
|
||||
|
||||
- 第一项先判断生产本地是否至少存在一个完整可用批次,并对最新本地候选执行与异机归档相同的 manifest schema/status、精确三文件集合、run id 配对、字节数和 SHA-256 校验。输出只含 `complete_local_recovery_point=yes|no`、年龄、数量和聚合大小。若为 `yes`,当前风险是异机副本缺失/生产单点;若为 `no`,当前不存在任何已证明可用的备份,事件风险和过渡恢复点优先级立即升级。
|
||||
- 分别核对成功运行和失败运行对生产本地批次/临时对象的保留与清理行为:成功后是否按保留策略清理,失败后是否保留完整本地批次、是否残留生成期临时对象,以及连续失败是否逐轮累积。不得仅统计“是否累积”而遗漏清理语义。
|
||||
- 比较两个 partial 的聚合总字节数、各预期产物已写入字节数或安全比例,不输出文件名/路径。相同或接近停止偏移作为确定性限制信号,明显不同偏移作为网络/超时等间歇中断信号;阈值和计算口径须在取数前固定。
|
||||
- 从生产备份 wrapper 的私有状态/诊断中提取真实 rsync 退出码和稳定类别,不以 systemd/service 的通用 exit `1` 替代。至少区分 rsync `23`、`30`、`12`、`11` 与 SSH 层 `255`;不得输出原始错误文本。
|
||||
- 记录每次失败的传输阶段开始、失败时刻和持续秒数,判断是否收敛到相近时长;无法从现有状态可靠取得时明确标为未测,不从 mtime 或聊天时间猜测。
|
||||
- 只读统计生产当前可用空间、最近完整本地批次聚合大小、当前本地完整/不完整/临时批次数量及成功/失败路径的清理与累积行为,形成容量预算。
|
||||
- 将 partial 偏移、rsync 原始码、失败时长和两端容量交叉判定,输出一个主要根因类别及仍未排除的候选,不为尽快修复而伪造单一确定根因。
|
||||
- 若现有只读证据不能收敛,尤其是故障疑似位于不可控链路侧,不扩大远端读取范围或反复追加低信号探针;转为设计“一次受控观测性重传”,冻结输入、实时采集允许字段并有界停止,作为独立审批项。该路径不在只读调查授权内。
|
||||
|
||||
### R3. 传输链修复
|
||||
|
||||
- 只修复 R2 证据直接支持的最小故障面;不得顺手轮换已证明可以完成初始认证的密钥、重建 receiver、放宽 manifest/finalize 安全门禁或修改恢复 runner。
|
||||
- 若根因为超时/会话限制,变更须有明确的客户端、服务端或中间链路归属、取值依据、回滚值和有界失败行为;禁止无限超时或无界重试。
|
||||
- 若根因为源端读取、目标写入、网络或 rsync 行为,优先修复可机械验证的具体错误,并保留原子 finalize、partial 隔离和失败关闭。
|
||||
- 生产和 ARM 的实际变更必须先形成脱敏变更清单、备份/回滚点与验证命令,再单独申请执行批准。
|
||||
|
||||
### R4. 生产容量处置
|
||||
|
||||
- 基于最近成功批次和当前生成流程计算至少容纳一个完整新批次的实际峰值预算,并增加保守安全余量;容量阈值必须在生产写操作前写入设计和证据。
|
||||
- 优先识别可安全处置的自动备份自有产物、失败临时对象或保留策略问题;任何删除/移动必须精确匹配已验证身份,禁止宽泛清理、跨任务清理或触碰生产数据库/附件源数据。
|
||||
- 容量处置不得以降低备份保留、删除唯一恢复点或跳过本地完整批次为代价。若空间扩容比删除更安全,应选择扩容方案并单独审批。
|
||||
- 修复后生产必须具备“一个完整新批次实际峰值 + 安全余量”的可验证空间,且两个自然周期内空间趋势不继续逼近失败线。
|
||||
|
||||
### R5. 恢复验证与持续观察
|
||||
|
||||
- 修复后必须形成新的 ARM 原子归档批次,而不是仅出现 partial;该批次须通过 manifest schema/status、精确三文件集合、run id/目录配对、字节数和 SHA-256 校验。
|
||||
- daily、watchdog、dead-man 不得仅以 timer `active` 作为通过。对应 timer 必须正常触发,最近 service 必须 `Result=success`、退出码 `0`,状态摘要须与新鲜归档一致。
|
||||
- 必须连续观察至少两个自然 daily 调度周期,每个周期都形成新的完整原子归档并通过相同合同;手工补跑成功不能替代两个自然周期。
|
||||
- 两个周期都必须记录 ARM inbox partial 数量/聚合字节变化和 ARM 文件系统可用量/使用率趋势;不得只观察生产容量,避免传输失败对象在 ARM 侧继续累积形成新的容量故障。
|
||||
- 观察期间不得为通过验收而删除失败证据、改选旧批次或临时放宽新鲜度/完整性阈值。
|
||||
|
||||
### R5a. 过渡恢复点决策
|
||||
|
||||
- R2 得出本地备份可用性后,必须单独记录 `transition_recovery_point_action=wait_natural|request_manual_run` 及理由。
|
||||
- 若生产本地已有完整可用批次且下一自然周期时间/容量风险可接受,优先等待自然调度;若生产本地也没有完整可用批次,手工补跑建立过渡恢复点的优先级显著上升,但仍须先有根因/容量停止条件并取得单独批准。
|
||||
- 手工补跑即使形成过渡恢复点,也不计入 AC5 的两个自然周期;若没有获批,不得为缩短风险窗口自行执行。
|
||||
|
||||
### R6. 告警送达与响应闭环
|
||||
|
||||
- 保留本次“检测与发送调用正常、响应未及时形成”的双重结论,不把 `sent=true` 等同于人员已阅读。
|
||||
- 从本任务开始,每次真实备份/dead-man 告警须分别记录脱敏的发送调用时刻和首位责任人确认时刻,并计算响应延迟;不得记录 webhook、群标识、人员账号或消息原文。
|
||||
- 若后续告警仍只有 `sent=true` 而无人确认,调查群送达、可见性、责任人和升级流程;发送测试告警须单独批准。
|
||||
|
||||
### R7. 与恢复演练解耦
|
||||
|
||||
- 本任务优先级为 P1、scope 为 `operations`,高于 P2 的 `08-23-isolated-backup-restore-drill`。
|
||||
- 演练任务保持 `in_progress`,明确被本任务阻塞;在新鲜原子批次形成并通过 24 小时身份门禁前不得继续实际恢复。
|
||||
- 既有恢复候选 SHA 保持有效,无需因本事件重新渲染;本任务不修改恢复 runner 或冒充恢复可用性结论。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] AC1:只读调查首先明确生产本地是否存在通过完整合同校验的可用批次,并区分“异机副本缺失/生产单点”与“当前无已证明可用备份”;同时取得成功/失败运行的清理保留语义、两个 partial 的聚合停止偏移、真实 rsync 退出码、可靠失败持续时间(或明确未测原因)、生产空间与累积状态,形成证据支持的根因分类和过渡恢复点建议。
|
||||
- [ ] AC2:经单独批准的最小传输修复与容量处置完成,变更、回滚点和最终容量预算有脱敏证据;未修改正式归档、恢复门禁或业务数据。
|
||||
- [ ] AC3:ARM 形成至少一个新的完整原子归档批次,并通过 manifest、精确文件集、run id 配对、字节数和 SHA-256 校验;partial 不得冒充完成。
|
||||
- [ ] AC4:生产 daily/watchdog 与 ARM dead-man 的 timer 正常触发,最近 service 均 `Result=success`、exit `0`,新鲜度状态与最新原子归档一致。
|
||||
- [ ] AC5:连续两个自然 daily 调度周期均产生新的完整原子归档并通过 AC3/AC4;手工补跑不计入这两个周期;两个周期的 ARM partial 数量/聚合字节和 ARM 容量趋势均已记录且未继续恶化。
|
||||
- [ ] AC6:生产容量达到经审阅的“一个完整新批次实际峰值 + 安全余量”预算,两个自然周期内无继续逼近失败线的趋势。
|
||||
- [ ] AC7:告警响应计时合同已启用;观察窗口内若出现真实告警/恢复通知,则同时记录发送调用时刻、人工确认时刻和响应延迟;若没有自然通知,明确标为未触发,不发送未批准的测试消息,也不把缺样本写成响应闭环已验证。
|
||||
- [ ] AC8:演练任务保留阻塞交叉引用;task validate、Trellis governance、流程合同、文档审计、敏感扫描和 `git diff --check` 通过。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 本轮规划不授权连接生产/ARM、不授权只读调查执行,也不授权任何修复、清理、重跑或告警发送。
|
||||
- 不恢复数据库或附件,不启动隔离 OA,不修改恢复候选、migration、业务代码、生产数据库或业务附件。
|
||||
- 不轮换备份 SSH 密钥、不重建 receiver,除非后续只读证据推翻“初始认证已成功”的当前事实并重新审阅设计。
|
||||
- 不删除 ARM 正式归档或历史 partial;历史 partial 的清理须在身份与保留价值审阅后单独批准。
|
||||
- 不把一次手工成功、timer active、service 通用 exit 或 `sent=true` 单独作为结案依据。
|
||||
|
||||
## Blocking Open Questions
|
||||
|
||||
- 无。第一阶段只读调查、任何本地实现以及生产/ARM 写操作仍分别需要后续明确批准。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 本地使用合成 fixture 验证 partial 聚合比较、rsync 退出码映射、失败时长聚类、容量预算和敏感输出 allowlist。
|
||||
- 获批后对生产/ARM 只读采集 R2 字段,保持原始日志、路径、身份和业务数据不出远端私有边界。
|
||||
- 修复候选先通过 Bash/Python 语法、既有自动异机备份/receiver/dead-man 合同和回滚演练;实际变更另行批准。
|
||||
- 实际修复后校验新原子批次与三组 timer/service,并连续观察两个自然调度周期。
|
||||
- 运行 task validate、Trellis governance、process baseline、docs registry、敏感扫描和 `git diff --check`。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "production-arm-backup-transfer-incident",
|
||||
"name": "production-arm-backup-transfer-incident",
|
||||
"title": "生产到 ARM 备份传输中断",
|
||||
"description": "调查并修复生产到 ARM 的自动异机备份传输中断,同时处置生产容量风险并验证连续两个自然调度周期。",
|
||||
"status": "in_progress",
|
||||
"dev_type": null,
|
||||
"scope": "operations",
|
||||
"package": null,
|
||||
"priority": "P1",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-28",
|
||||
"completedAt": null,
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "来源证据见 08-23-isolated-backup-restore-drill 的 2026-08-27 自动备份链只读调查。创建与规划不授权连接生产/ARM;只读深查、生产/ARM 写操作、手工补跑和测试告警分别审批。",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
# 设计:报销详情修复部署与公网验证
|
||||
|
||||
## 当前依赖
|
||||
|
||||
部署候选必须包含 Task A 和 `08-28-reimbursement-nonparticipant-readonly-projection`。业务提交为 `e8d6766`;本次 PRD 修订提交完成后,以最终干净 `HEAD` 冻结候选,在外部审批记录中写死 SHA,此后不得再产生任何仓库变化。用户必须先单独批准推送;upstream 与严格 provenance 验证通过后,再单独批准连接生产和部署。
|
||||
|
||||
## 发布流
|
||||
|
||||
```text
|
||||
提交本次 PRD 修订并确认工作区干净
|
||||
-> 冻结当前 HEAD,在外部审批记录写死 SHA
|
||||
-> 用户单独批准 push
|
||||
-> push + upstream 祖先关系 + 严格 provenance
|
||||
-> 用户单独批准生产部署
|
||||
-> 生产身份与 existing 部署合同只读 preflight
|
||||
-> 期望 migration 清单核对
|
||||
-> deploy-existing.sh(SYNC_ENV=0)
|
||||
-> init + migrate deploy + post migration status
|
||||
-> /ready + runtime/external smoke + bundle marker
|
||||
-> 备份合同 refresh
|
||||
-> 角色化公网检查和材料预览
|
||||
-> 归档部署子任务与 Task A
|
||||
```
|
||||
|
||||
- 正式入口使用仓库现有 wrapper,不手工 rsync、不 source 或覆盖远端 `.env`。
|
||||
- 本轮代码不包含新 schema,但 init 和 pre/post migration status 仍是必需证据。
|
||||
- `pdf-v2` 缓存仅在进程内,应用重启后自然失效,无缓存迁移或清理动作。
|
||||
- 真实材料检查只记录页数、页序、比例和成功/失败,不保存截图、原件或票面字段。
|
||||
|
||||
## 固定回滚
|
||||
|
||||
- 切换前失败:停止,不进入回滚,因为生产应用尚未改变。
|
||||
- 切换后任一必需门禁失败:停止新版本验证,使用同一 existing 部署入口发布固定提交 `f178aa0`,保持 `SYNC_ENV=0`,不回退或 reset migration。
|
||||
- 回滚后必须确认 `/ready` 和 bundle marker 为 `f178aa0` 的短 SHA,并复核 migration status、runtime/external smoke、DB/附件挂载与公网入口。
|
||||
- 回滚来源同样必须干净、已发布且远端可验证;不得以 unpublished override 绕过正式 wrapper。
|
||||
|
||||
## 授权边界
|
||||
|
||||
任务规划批准不等于推送批准,推送批准也不等于生产部署批准。候选冻结后先取得独立 push 授权并完成 upstream/provenance 验证,再取得独立部署授权;连接生产、应用切换和合成 owner 撤回写入均不得从前一项授权推定。真实报销永不作为撤回测试对象。
|
||||
@@ -0,0 +1,37 @@
|
||||
# 实施清单:报销详情修复部署与公网验证
|
||||
|
||||
## 1. 依赖与候选
|
||||
|
||||
- [x] P0 非参与人只读投影修复已实施、只读复核 PASS 并提交为 `e8d6766`。
|
||||
- [ ] 本次 PRD 修订提交完成后确认工作区干净,以最终 `HEAD` 冻结候选,并在外部审批记录写死 SHA;冻结后不得再提交或修改工作区。
|
||||
- [ ] 用户已审查最终候选、回滚和公网验证清单,并单独明确批准推送冻结候选。
|
||||
- [ ] 推送后确认 upstream 包含候选,严格 provenance 通过且未使用 unpublished override。
|
||||
- [ ] 来源验证通过后,用户另行明确批准连接生产和正式部署。
|
||||
|
||||
## 2. 部署前门禁
|
||||
|
||||
- [ ] 工作区干净且 HEAD 等于外部审批记录中的候选;候选与 `f178aa0` 均已发布且来源可远端验证。
|
||||
- [ ] 生产 identity、部署合同、DB/附件 existing 路径、运行容器和回滚窗口只读 preflight 全部通过。
|
||||
- [ ] 明确期望 pending migrations;本轮若仍无新增 migration,期望值写为空但不跳过 status。
|
||||
|
||||
## 3. 正式发布
|
||||
|
||||
- [ ] 使用 `deploy-existing.sh`、`DEPLOY_DATA_MODE=existing`、`SYNC_ENV=0` 执行发布。
|
||||
- [ ] init、migrate deploy、post migration status、runtime/external smoke、`/ready`、bundle marker 和备份合同 refresh 全部通过。
|
||||
|
||||
## 4. 公网验证
|
||||
|
||||
- [ ] `/ready` 与前端 marker 等于候选短 SHA,迁移状态正常。
|
||||
- [ ] 财务/管理员非参与人无报销人维护入口;owner 视角维护入口正确。
|
||||
- [ ] 仅在最终授权包含合成测试写入且存在专用合成单据时验证 owner 撤回成功;否则停止,不操作真实单据。
|
||||
- [ ] 材料合并预览可打开,页序和比例正常;不保存真实票据截图或原件。
|
||||
|
||||
## 5. 失败与回滚
|
||||
|
||||
- [ ] 切换前失败直接停止;切换后任一必需门禁失败,使用同一入口回滚固定提交 `f178aa0`。
|
||||
- [ ] 回滚不 reset migration、不覆盖环境或数据目录;回滚后复核 build、status、smoke、挂载和公网入口。
|
||||
|
||||
## 6. 收口
|
||||
|
||||
- [ ] 公网全部通过后才报告用户可测试,回填脱敏证据和实际部署 SHA。
|
||||
- [ ] 先归档部署子任务,再按真实完成证据归档 Task A;任何未完成门禁保持未勾选。
|
||||
@@ -0,0 +1,62 @@
|
||||
# 报销详情修复部署与公网验证
|
||||
|
||||
## Goal
|
||||
|
||||
在 Task A 与非参与人只读投影修复都完成后,冻结新的已发布候选,执行受控生产部署、显式迁移状态校验、角色化公网检查和固定回滚验证。
|
||||
|
||||
## Current Planning Gate
|
||||
|
||||
- 原候选 `174cb7a` 只包含撤回投影与 EXIF 尺寸修复,未覆盖已确认的非参与人草稿/驳回维护动作和服务端解绑资格问题,因此不再作为最终部署基线。
|
||||
- `08-28-reimbursement-nonparticipant-readonly-projection` 已实施、只读复核并形成业务提交 `e8d6766`;部署工具只能以干净工作区的当前 `HEAD` 作为来源,因此旧口径 `8c28e15` 作候选失效。
|
||||
- 本次 PRD 修订提交完成后,以当时最终 `HEAD` 冻结候选,并在外部审批记录中写死完整/短 SHA;冻结后到部署收口前不得再产生任何提交(包括文档提交)、untracked 文件或工作区修改。若 HEAD 或工作区变化,原审批失效,必须重新冻结和审查。
|
||||
- Requirements、Acceptance Criteria、`design.md` 与 `implement.md` 已写明回滚到 `f178aa0` 的触发条件/动作、迁移 pre/post status、公网角色和材料预览检查;本任务继续保持 `planning`,等待用户先审查候选并单独批准推送,再单独批准生产部署。
|
||||
- 当前未经批准不得推送、部署、写生产或归档 Task A。
|
||||
|
||||
## Scope
|
||||
|
||||
- 在 P0 只读投影修复完成并提交后,以最终干净 `HEAD` 冻结包含两轮报销修复的候选 SHA;候选冻结后禁止任何仓库变化。
|
||||
- 推送是独立、显式授权的发布门:取得单独推送批准后,只推送已冻结候选所在分支,再确认 upstream 包含该 SHA 且 provenance 未使用 unpublished override。推送成功不等于批准部署。
|
||||
- 使用现有 `deploy/trialrun/deploy-existing.sh` 完成生产只读 preflight、existing 模式部署、init、迁移状态复核、运行时 smoke、构建标识和备份合同刷新。
|
||||
- 部署后核验财务/管理员非参与人只读视图、owner 维护/撤回合同和材料合并预览;不把真实票据内容写入证据。
|
||||
- 固定回滚基线为 `f178aa0`;部署切换后的任一必需门禁失败时执行同一正式入口回滚应用代码,不回退数据库 migration。
|
||||
|
||||
## Requirements
|
||||
|
||||
- 候选冻结时必须确认工作区完全干净,以当前分支 `HEAD` 作为唯一候选并记录 SHA;之后任何提交、tracked/untracked 变化或 HEAD 漂移都使候选失效。
|
||||
- 推送必须另行获得明确批准。推送后必须确认当前分支有 upstream、候选是 upstream 的祖先,并以默认严格模式通过 `check-source-provenance.mjs`;不得使用 `allowUnpublished` 或接受 `overrideUsed: true`。
|
||||
- 只有候选已推送且来源校验通过后,才可请求另一项独立的生产部署批准。部署批准不得从 PRD 审批或推送批准中推定。
|
||||
- 部署前必须显式复核候选提交范围、生产身份、existing 数据目录、待迁移清单和回滚窗口;来源未发布、工作区不干净、HEAD 不等于已批准候选或身份不唯一时停止。
|
||||
- 即使候选没有新增 migration,init 后仍必须显式运行并记录 Prisma migration status;不得因“无 schema 变更”跳过。
|
||||
- `SYNC_ENV=0`,不覆盖生产环境文件;不 reset 数据库、不覆盖 DB/附件目录、不清理 `pdf-v2` 进程内缓存,应用重启自然失效。
|
||||
- 公网 `/ready` 和前端 bundle marker 必须等于最终候选短 SHA;迁移、runtime/external smoke、备份合同刷新和角色化检查全部为必需门禁。
|
||||
- 非参与人只读检查不得写业务数据。owner 撤回成功只能使用专用合成单据并在最终部署授权中单列生产测试写入;没有合格合成单据时该项保持阻断,不得操作真实报销。
|
||||
- 切换前门禁失败只中止部署;切换后必需门禁失败则回滚到 `f178aa0`,再复核该基线 `/ready`、bundle、migration status、runtime smoke 和数据挂载连续性。
|
||||
- 公网检查全部通过前不告知用户可以测试;通过后先归档部署子任务,再按真实完成证据归档 Task A。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] 非参与人只读投影修复已实施、复核并提交为 `e8d6766`。
|
||||
- [ ] 本次 PRD 修订提交完成后的最终干净 `HEAD` 已冻结,具体 SHA 已在外部审批记录中写死,且冻结后无任何仓库变化。
|
||||
- [ ] 完整部署 PRD、设计、实施和回滚清单经用户审查;用户已单独批准推送冻结候选。
|
||||
- [ ] 推送完成,upstream 包含冻结候选,严格 provenance 通过且 `overrideUsed` 不为 true。
|
||||
- [ ] 用户在来源校验通过后另行明确批准生产部署。
|
||||
- [ ] 最终候选的 `/ready`、bundle marker、迁移状态、runtime/external smoke 和备份合同刷新全部通过。
|
||||
- [ ] 财务/管理员非参与人无报销人维护动作,owner 既有维护合同正常,材料合并预览可打开且页序和比例正常。
|
||||
- [ ] 任一必需门禁失败时已按固定动作回滚到 `f178aa0` 并完成回滚后复核,未临场选择其他基线。
|
||||
- [ ] 公网检查通过后才报告可测试并归档相关任务。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 在部署任务内继续修改报销业务代码、权限、状态机、schema、migration 或生产配置。
|
||||
- 处理 `BX-20260828-0017` 票面差额、退回、付款、重传或任何其他真实单据写入。
|
||||
- Task B 产品改造、r102 测试债、企业微信写入、真实付款、数据库 reset 或附件清理。
|
||||
- 为 `pdf-v2` 执行缓存迁移或清理;缓存仅存在进程内。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 冻结与推送:干净工作区、当前分支 `HEAD`、外部审批记录中的固定 SHA、独立 push 批准、upstream 祖先关系和严格 provenance(禁止 unpublished override)。
|
||||
- 部署前:独立部署批准、候选范围/上游来源、`check-existing-deployment.sh`、production identity/preflight、期望待迁移清单和回滚窗口。
|
||||
- 部署入口:`deploy-existing.sh` 的 existing 模式、`SYNC_ENV=0`、init、pre/post Prisma status、runtime/external smoke、`/ready`、bundle marker 和合同 refresh。
|
||||
- 公网角色:财务/管理员非参与人只读、owner 维护/撤回、材料合并预览与原件降级;只保留脱敏结论。
|
||||
- 回滚演练:从受控干净来源发布 `f178aa0`,复核构建标识、迁移状态、运行 smoke、挂载和公网入口。
|
||||
- 治理:task validate、Trellis governance、敏感检查和 `git diff --check`。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "reimbursement-detail-material-preview-deploy",
|
||||
"name": "reimbursement-detail-material-preview-deploy",
|
||||
"title": "报销详情修复部署与公网验证",
|
||||
"description": "在报销详情两轮权限修复完成后,以新的已发布候选执行受控部署、迁移状态校验、固定回滚和公网角色验证。",
|
||||
"status": "planning",
|
||||
"dev_type": null,
|
||||
"scope": "production deployment preflight; migration status; public role and material-preview verification; fixed rollback",
|
||||
"package": null,
|
||||
"priority": "P0",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-28",
|
||||
"completedAt": null,
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": "08-28-reimbursement-detail-material-preview-fixes",
|
||||
"relatedFiles": [],
|
||||
"notes": "旧候选 8c28e15 因后续文档提交导致 HEAD 漂移而失效。以本次 PRD 修订提交完成后的最终干净 HEAD 冻结新候选,并在外部审批记录中写死 SHA;冻结后禁止任何仓库变化。push 与生产部署是两个独立授权门,当前均未批准。",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
# 设计:报销详情权限与材料预览缺陷修复
|
||||
|
||||
## 边界与原则
|
||||
|
||||
本任务是两个小型缺陷修复加一项生产只读诊断,不改变业务权限、审批状态或数据模型。前端修复负责把服务端已有的对象资格正确投影为可见动作;后端修复只校正图片规范化结果的尺寸事实。真实材料诊断与代码修复分离,诊断结果不能成为扩大实现范围的默许授权。
|
||||
|
||||
## R1 动作资格数据流
|
||||
|
||||
```text
|
||||
报销详情 DTO ownerId / submitterId + 当前用户 me.id
|
||||
-> isReimbursementParticipant
|
||||
-> ReimbursementActions.showWithdrawAction
|
||||
-> reviewing 状态下按“参与人 AND 状态”显示撤回
|
||||
```
|
||||
|
||||
- 复用 `ReimbursementDetailPage` 已有 `isReimbursementParticipant`,不新增页面级权限枚举或 API 字段。
|
||||
- `ReimbursementActions.showWithdrawAction` 改为必填 prop 并移除默认值。全仓库当前只有 PC 报销详情一个调用点,由 TypeScript 强制调用方显式传入参与人资格,后续新增调用也默认 fail-closed。
|
||||
- owner/submitter 的既有交互、默认撤回说明、mutation 和反馈不变。
|
||||
- Playwright 使用两类合成身份验证:现有 owner 夹具继续看到撤回;复用 `r102-reimbursement-main-flow.spec.ts` 中 `adminMe + ownerId=employee-other-r102 + status=reviewing + scope=all` 的非参与管理员夹具断言撤回不可见。管理员覆盖当前 fail-open 权限面的最宽视角,服务端既有 403 测试继续作为真正安全边界。
|
||||
|
||||
## R2 图片规范化数据流
|
||||
|
||||
```text
|
||||
源 JPG/PNG/WebP
|
||||
-> sharp(source).rotate() 应用 EXIF
|
||||
-> flatten + JPEG 编码
|
||||
-> toBuffer({ resolveWithObject: true })
|
||||
-> info.width / info.height(实际输出尺寸)
|
||||
-> fitPage
|
||||
-> PDF page + drawImage
|
||||
```
|
||||
|
||||
- 不再在输出前使用 Sharp 输入 metadata 作为页面尺寸。
|
||||
- `resolveWithObject` 同时返回规范化 buffer 和实际输出 info,避免为了取尺寸再解码一次。
|
||||
- 输出宽高仍执行存在性、整数性和 `maxImagePixels` 检查;输入解码继续由 Sharp 的 `limitInputPixels` 限制。若 Sharp 输出缺少合法尺寸,沿用 `material_preview_image_invalid`。
|
||||
- 合并器版本从 `pdf-v1` 升级为下一稳定版本,使集合哈希和 cache key 明确区分旧算法。缓存仍只在进程内、按 grant 生命周期管理,不迁移数据。
|
||||
- 测试用 Sharp 生成无敏感内容的方向 6/8 JPEG。除页数和确定性外,读取合并 PDF page size,断言其纵横方向等于自动旋转后的实际输出;保留普通 PNG/JPG/WebP 基线。
|
||||
|
||||
## R3 真实单据只读诊断
|
||||
|
||||
```text
|
||||
受控生产只读定位业务单号
|
||||
-> 冻结有序材料最小元数据
|
||||
-> 必要时流式读取材料到安全临时目录/内存
|
||||
-> 读取 EXIF 和规范化输出尺寸
|
||||
-> 用当前合并器生成临时派生 PDF
|
||||
-> 核对页序、页数、目标页尺寸与是否存在图像对象
|
||||
-> 删除临时材料
|
||||
-> 仅保留脱敏结论
|
||||
```
|
||||
|
||||
- 现有顺序与单据事实已确认两张单页 PDF 发票在前、两张 JPG 附件在后,因此用户口中的“第一张图”固定对应合并 PDF 第 3 页;诊断直接从第 3 页开始,同时核对总页数,避免把 PDF 第 1 页误作目标图片。
|
||||
- 只在元数据不足以定因时读取最小原件字节;不进行 OCR、不展示内容、不输出完整文件名和存储路径。
|
||||
- 诊断记录只允许:材料类别、相对顺序、MIME、脱敏尺寸/方向、合并页数/页尺寸、错误类别和修复覆盖结论。
|
||||
- 若现象可由旋转后尺寸不一致完整解释,R2 为修复;若 PDF 页本身缺少可复制内容或存在表单/可选内容依赖,停止在证据结论,另行规划兼容方案。
|
||||
|
||||
## 兼容性与回滚
|
||||
|
||||
- 无 API、schema、迁移、权限或状态机变更。
|
||||
- 前端回滚:同时恢复 prop 默认值和调用点合同才会恢复旧投影;正常修复不允许只移除调用点参数,因为必填类型应在编译期阻断该回退错误。服务端安全边界始终不变。
|
||||
- 后端回滚:恢复旧图片尺寸读取并回退 merger version。已签发短时 bundle 本就随进程/TTL 失效,无持久化产物迁移。
|
||||
- 未部署前仅能报告本地验证与只读诊断结论;公网修复需要单独批准部署并完成构建号和关键路径检查。
|
||||
|
||||
## 关键权衡
|
||||
|
||||
- 选择复用现有参与人布尔值,而不是增加新的通用授权抽象:当前只有一个明确调用缺口,扩大抽象不会增加安全性。
|
||||
- 选择读取 Sharp 输出 info,而不是交换 metadata 宽高:EXIF 不只有 6/8,输出事实比枚举推断更可靠。
|
||||
- 不在本任务替换 `pdf-lib`:首图的 PDF 特殊内容兼容仍未被证据确认,且更换渲染链会改变确定性、资源上限和签名事实边界。
|
||||
@@ -0,0 +1,59 @@
|
||||
# 实施清单:报销详情权限与材料预览缺陷修复
|
||||
|
||||
## 1. 实施前只读诊断
|
||||
|
||||
- [x] 确认当前任务已获最终方案实施批准,并加载 `trellis-before-dev`。
|
||||
- [x] 使用受控只读路径冻结 `BX-20260828-0017` 的材料顺序与最小元数据,不打印敏感字段;按“两张单页 PDF 发票在前、两张 JPG 附件在后”的既有事实,将第一张图片锁定为合并 PDF 第 3 页。
|
||||
- [x] 在运行中 server 容器内以只读事务和内存缓冲复现当前合并结果,从第 3 页核对目标图片的 EXIF、规范化输出尺寸、PDF 页序和页尺寸;未写生产、未落临时原件。
|
||||
- [x] 诊断确认两张 JPG 均无 EXIF 方向标记,第 3/4 页比例正确且存在图像内容;两张源 PDF 也均有普通内容流,不存在 AcroForm/可选内容层依赖。当前数据无法复现“第 3 页无内容”,因此 EXIF 修复不宣称解决 `0017` 现象,本轮不扩展 PDF 引擎。
|
||||
|
||||
## 2. PC 撤回动作资格
|
||||
|
||||
- [x] 将 `ReimbursementActions.showWithdrawAction` 改为无默认值的必填 prop,并在 PC 报销详情调用点显式传入 `showWithdrawAction={isReimbursementParticipant}`。
|
||||
- [x] 保留 owner/submitter reviewing 状态的撤回能力、mutation 和反馈,不改组件外的权限或状态判断。
|
||||
- [x] 在 `r102-reimbursement-main-flow.spec.ts` 复用现有 admin 非参与人全范围夹具,断言撤回按钮和意见框均不存在;保留现有 owner 可见回归和 PWA Service Worker 隔离合同。
|
||||
|
||||
## 3. EXIF 合并修复
|
||||
|
||||
- [x] 调整图片规范化函数,从实际 JPEG 输出 info 读取宽高并校验合法性。
|
||||
- [x] 更新材料合并器版本,确保缓存身份与算法版本一致。
|
||||
- [x] 增加 orientation 6/8 合成图片回归,断言合并页方向/比例;保留普通图片、页数、确定性、限额和失败路径测试。
|
||||
|
||||
## 脱敏只读诊断结论
|
||||
|
||||
- 材料顺序为 2 张单页 PDF 发票、2 张 JPEG 附件,共 4 页;用户所称“第一张图”为第 3 页。
|
||||
- 两张 JPEG 的输入与规范化尺寸比例一致,均无 EXIF orientation;当前合并 PDF 第 3、4 页均存在内容,页面比例与图片一致。
|
||||
- 两张源 PDF 均存在普通页面内容流,未发现 AcroForm 或可选内容层;当前合并结果也保留对应页面内容。
|
||||
- 因而:代码中的 EXIF 尺寸读取缺陷成立并应修复,但不是 `BX-20260828-0017` 当前材料的直接原因;该单的“第一张图未显示”在本次只读内存复现中未重现,可能与当时的短时 bundle/阅读器状态有关,不能无证据归因。
|
||||
- 下一步:服务端材料链路已排除。若现象复现,需向用户确认具体页码、浏览器或企业微信 WebView 版本,以及是否稳定复现,再决定是否创建新的客户端诊断任务;本轮不在服务端继续扩面。
|
||||
|
||||
## 4. 验证与只读复核
|
||||
|
||||
- [x] 运行聚焦后端 Vitest、修改文件 `node --check` 和 Prisma validate。
|
||||
- [x] 运行前端 typecheck/build 与聚焦 Playwright。
|
||||
- [x] 运行流程、文档、Trellis 治理及 `git diff --check`。
|
||||
- [x] 只读检查 diff,确认未改变服务端撤回授权、权限矩阵、审批状态机、材料顺序、授权 grant、审计或原件降级合同。
|
||||
- [x] 按 `trellis-update-spec` 将“旋转后以输出尺寸建页”和“可见算法变化升级 merger version”沉淀到报销材料规约。
|
||||
|
||||
## 5. 交付与回滚点
|
||||
|
||||
- [x] 汇总代码变化、合成测试结果和 `BX-20260828-0017` 脱敏诊断结论,不附真实原件或敏感标识。
|
||||
- [x] 将 `BX-20260828-0017` 的真实处置作为单独业务决定提醒:按现行 PDF-only 规则由业务选择退回重传或财务留痕例外接受,本任务不代为处理。
|
||||
- [x] 提交信息使用中文并引用 `08-28-reimbursement-detail-material-preview-fixes`。
|
||||
- [x] 未经单独批准不部署;如后续批准部署,必须完成公网构建号和关键路径只读检查后才能通知用户测试。
|
||||
|
||||
## 关键文件
|
||||
|
||||
- `client/src/pages/ReimbursementDetailPage.tsx`:PC 撤回动作资格投影。
|
||||
- 覆盖 PC 报销详情的现有或聚焦 Playwright spec:owner/财务双视角回归。
|
||||
- `server/src/services/reimbursementMaterialPdf.js`:Sharp 输出尺寸读取。
|
||||
- `server/src/services/reimbursementMaterialPreview.js`:合并器版本。
|
||||
- `server/tests/unit/services/reimbursementMaterialPreview.test.js`:EXIF、页面尺寸和确定性回归。
|
||||
|
||||
## 验证记录
|
||||
|
||||
- 聚焦后端:3 个测试文件、23 项测试全部通过;含材料合并、附件路由和非参与人撤回 403 回归。
|
||||
- 前端:typecheck 与 production build 通过;admin 非参与人撤回不可见、owner 撤回可用两条聚焦 Playwright 均通过。
|
||||
- 完整 `r102-reimbursement-main-flow.spec.ts` 为 5/6:本次新增用例通过;唯一失败为既有用例仍查找旧按钮名“编辑”,当前页面实际为“编辑报销日期”,与本次修改无关,未在本任务扩面修复。
|
||||
- 全量后端为 179 个测试文件通过、2 个失败,1295/1298 项通过;3 项失败均来自既有付款单 BIFF8 生成器在当前环境不可用,本任务未修改付款单代码。
|
||||
- process contract 与 Trellis governance 通过;docs registry 为 0 error、既有 22 个未登记文档 warning;敏感内容检查与 `git diff --check` 通过。
|
||||
@@ -0,0 +1,84 @@
|
||||
# 报销详情权限与材料预览缺陷修复
|
||||
|
||||
## Goal
|
||||
|
||||
修复报销只读查看者误看到“撤回报销”操作,以及带 EXIF 方向信息的图片进入材料合并预览后页面尺寸错误的问题;同时对 `BX-20260828-0017` 的首张图片未正常显示现象做受控、脱敏、只读诊断,让财务只看到职责内动作,并让图片材料在合并 PDF 中保持正确方向和比例。
|
||||
|
||||
## Background
|
||||
|
||||
- PC 报销详情已经计算 `isReimbursementParticipant`,但调用 `ReimbursementActions` 时未传 `showWithdrawAction`;该组件默认允许在 `reviewing` 状态显示撤回,因此具备全量查看权限的财务人员也会看到撤回入口。
|
||||
- 服务端撤回接口仍按 owner/submitter 做对象资格校验,非参与人会被拒绝;当前问题是前端动作资格投影错误,不是服务端越权或数据损坏。
|
||||
- 移动报销详情当前没有撤回操作,本轮只修复存在缺陷的 PC 入口,不增加移动操作。
|
||||
- 材料合并服务先对图片执行 Sharp `rotate()` 自动应用 EXIF 方向,但页面尺寸读取自旋转前 metadata。方向为 6/8 的手机图片会出现“输出像素已转向、PDF 页面仍按原宽高”的不一致。
|
||||
- 现有合并预览按发票在前、补充附件在后排序;`BX-20260828-0017` 有两张 PDF 发票和两张 JPG 补充附件。此前只读核查已确认 JPG 实际为发票图片,但未打开或持久化原件,尚不能仅凭附件列表断定“首张图片未显示”的最终技术原因。
|
||||
- 现行业务规则要求:只读审批人不得看到编辑、解绑、删除等数据变更动作;图片各占合并 PDF 一页,合并失败时保留逐份原件查看路径。
|
||||
|
||||
## Scope
|
||||
|
||||
- 修复 PC 报销详情中 owner/submitter 与财务只读查看者的撤回动作可见性。
|
||||
- 修复服务端材料合并时 EXIF 自动旋转后仍沿用输入尺寸的问题,并更新合并器版本。
|
||||
- 使用合成数据覆盖前后端回归;对 `BX-20260828-0017` 仅执行受控、脱敏、生产只读诊断。
|
||||
- 保持服务端授权、审批状态机、材料顺序、原件查看与失败降级合同不变。
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1 PC 撤回动作资格投影
|
||||
|
||||
- PC 报销详情仅在当前用户是报销单 owner 或 submitter 且状态为 `reviewing` 时显示“撤回报销”。
|
||||
- 财务、审批人、管理员等因 `expense.view_all` 或当前审批职责获得只读详情访问时,不得看到撤回输入框或按钮。
|
||||
- owner/submitter 在 `reviewing` 状态下继续看到并可使用现有撤回入口;草稿、驳回、已通过、已支付等既有状态行为不变。
|
||||
- `ReimbursementActions.showWithdrawAction` 必须改为无默认值的必填 prop,让遗漏资格投影在 TypeScript 编译期失败,不保留 fail-open 默认行为。
|
||||
- 服务端撤回资格校验、权限矩阵、审批状态机和审计合同保持不变,前端不能把隐藏按钮当作安全边界。
|
||||
|
||||
### R2 EXIF 图片合并尺寸修复
|
||||
|
||||
- 图片经 Sharp 自动旋转、扁平化和 JPEG 规范化后,必须使用实际输出的宽高创建 PDF 页面和绘制图片,不得使用旋转前 metadata。
|
||||
- 方向为 6/8 的合成 JPEG 应在输出 PDF 中保持正确的纵横方向和比例;无 EXIF 或无需旋转的 PNG/JPG/WebP 行为保持不变。
|
||||
- 继续执行现有图片像素、页数、输出大小、确定性字节、整包失败和逐份原件降级合同;不引入新依赖、不把合并件当作原件。
|
||||
- 合并算法的可见输出发生变化后应更新 merger version,避免新旧算法产物共用同一集合缓存身份。
|
||||
|
||||
### R3 `BX-20260828-0017` 首图只读诊断
|
||||
|
||||
- 在实施获批后,使用既有受控生产只读访问,确认材料实际顺序、MIME、字节/像素元数据、EXIF 方向、合并页数和对应页尺寸;现有事实已将用户所称“第一张图”锁定为合并 PDF 第 3 页,诊断从该页开始,仅在必要时生成短生命周期的本地临时派生预览。
|
||||
- 诊断不得写生产数据库、报销、审批、付款、对象存储或企业微信;不得把原始票据、图片、完整票号、存储位置、签名 URL、原始 payload 或凭据写入仓库、任务文档或聊天。
|
||||
- 临时原件或渲染页若确有必要,只能放在明确的安全临时目录,诊断完成即删除;交付只记录脱敏元数据和结论。
|
||||
- 若证据确认问题由 EXIF 尺寸错误造成,使用 R2 修复覆盖;若属于 PDF `AcroForm`、可选内容组、损坏源文件或其他超出当前图片合并边界的原因,只形成证据并返回规划,不临时替换 PDF 引擎或扩大任务。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] 复用现有 `adminMe + ownerId=employee-other-r102 + status=reviewing + scope=all` 合成场景;管理员作为非参与只读查看者打开报销详情时,页面不存在 `reimbursement-withdraw` 和撤回意见输入框。
|
||||
- [x] 同一状态下,合成 owner/submitter 仍能看到现有撤回入口;其他状态的既有动作投影不变。
|
||||
- [x] 服务端非 owner/submitter 撤回拒绝合同保持通过,未修改权限矩阵、状态机、工作流或数据库结构。
|
||||
- [x] 带 EXIF orientation 6 和 8 的合成图片合并后,PDF 页面宽高与规范化输出一致,方向和比例正确;普通横图、普通竖图和 PNG/WebP 回归通过。
|
||||
- [x] 相同有序材料在同一 merger version 内重复生成时字节完全一致;跨 merger version 允许因修复产生不同字节。材料顺序、页数上限、整包失败和原件降级合同不变。
|
||||
- [x] merger version 随算法修复更新,新的集合缓存身份不会复用旧算法产物。
|
||||
- [x] `BX-20260828-0017` 首图形成一份不含原件和敏感字段的只读诊断结论,明确材料顺序、问题类别、是否由 EXIF 修复覆盖,以及未覆盖时的下一步。
|
||||
- [x] 前端 typecheck/build、后端聚焦 Vitest/语法检查、相关 Playwright、Trellis 治理和 `git diff --check` 通过。
|
||||
- [x] 未经另行批准不部署公网;部署与公网检查完成前不告知用户线上可测试。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 财务审批页一页式重构、按钮“处理”的改名或待办信息结构调整。
|
||||
- 费用类型、会计科目、发票—明细关联、票面金额差异门禁、多票拆分、图片发票上传提示或附件用途治理;这些属于 Task B。
|
||||
- 修改、撤回、退回、审批或付款任何真实报销;重分类或重新上传 `BX-20260828-0017` 的图片附件。
|
||||
- 修改服务端撤回授权、角色/权限矩阵、审批规则、状态机、schema、迁移或生产配置。
|
||||
- 引入 PDF.js、Poppler、新 PDF 引擎,或在未重新规划时处理 PDF 表单域/可选内容组兼容。
|
||||
- 修改 `ReimbursementActions` 现有撤回默认说明文案“申请撤回并补充说明后重提”。
|
||||
- 对 `BX-20260828-0017` 执行真实单据处置,包括退回、例外接受、重传 PDF、重分类附件或修正票面合计;该单据应按现行规则另行作出业务处理决定。
|
||||
- 公网部署、生产写入、企微写入和真实付款。
|
||||
|
||||
## Risks and Constraints
|
||||
|
||||
- 隐藏动作只能改善正确性和体验,安全仍由服务端对象资格校验负责。
|
||||
- EXIF 修复可以解决旋转后尺寸不一致,但不能预先证明 `BX-20260828-0017` 的首图问题只有一个原因。
|
||||
- 真实材料含敏感信息,诊断必须最小读取、只保留脱敏摘要,并对临时文件 fail closed。
|
||||
- 现有视觉回归夹具中 owner 视角明确期待撤回按钮可见,修复不能简单删除该操作;需要新增非参与高权限查看者断言。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 后端聚焦:`cd server && npx vitest run tests/unit/services/reimbursementMaterialPreview.test.js tests/unit/routes/reimbursements.attachments.test.js`。
|
||||
- 后端基础:`cd server && node --check src/services/reimbursementMaterialPdf.js && node --check src/services/reimbursementMaterialPreview.js && npm run prisma:validate`。
|
||||
- 前端:`cd client && npm run typecheck && npm run build`。
|
||||
- Playwright:复用 `r102-reimbursement-main-flow.spec.ts` 的 admin 非参与人全范围夹具增加撤回不可见断言,并保留现有 owner 可见断言;route mock 场景继续阻断 Service Worker。
|
||||
- 只读诊断:以脱敏元数据核对 `BX-20260828-0017` 材料顺序、EXIF/像素和合并页尺寸,不保留真实材料证据。
|
||||
- 治理:`node scripts/qa/check-process-contract.mjs --baseline --json`、`node scripts/qa/audit-docs.mjs --check-registry`、`node scripts/qa/check-trellis-governance.mjs`、`git diff --check`。
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"id": "reimbursement-detail-material-preview-fixes",
|
||||
"name": "reimbursement-detail-material-preview-fixes",
|
||||
"title": "报销详情权限与材料预览缺陷修复",
|
||||
"description": "修复财务只读查看报销时误显示撤回操作、EXIF 图片材料合并尺寸错误,并对 BX-20260828-0017 首图异常做脱敏只读诊断。",
|
||||
"status": "in_progress",
|
||||
"dev_type": null,
|
||||
"scope": "client reimbursement action projection; server material PDF image normalization; production read-only preview diagnostics",
|
||||
"package": null,
|
||||
"priority": "P0",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-28",
|
||||
"completedAt": null,
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": "46bfa7e",
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [
|
||||
"08-28-reimbursement-detail-material-preview-deploy",
|
||||
"08-28-reimbursement-nonparticipant-readonly-projection"
|
||||
],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "任务范围内聚焦验证通过。完整 r102 为 5/6,唯一失败是既有用例仍查找已不存在的‘编辑’按钮;本任务新增的管理员撤回不可见断言通过,不把该历史失败静默视为绿色基线。",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
# 设计:非参与人报销详情只读投影修复
|
||||
|
||||
## 设计边界
|
||||
|
||||
本任务修正“访问资格”和“维护资格”被混用的问题,不新建报销详情页面、不重构审批工作台,也不改变角色/权限码。详情读取继续支持 owner/submitter、当前审批职责和 `expense.view_all`;报销人自助写入必须额外通过对象参与资格。
|
||||
|
||||
## 资格模型
|
||||
|
||||
```text
|
||||
可查看详情
|
||||
= owner/submitter 的 expense.view_own
|
||||
OR 当前审批/职责访问
|
||||
OR expense.view_all
|
||||
|
||||
可维护报销
|
||||
= 当前用户是 owner/submitter
|
||||
AND 当前状态允许该动作
|
||||
AND 具备对应自助权限码
|
||||
|
||||
职责动作
|
||||
= workflow 审批/退回、财务付款、付款批次等专用接口
|
||||
!= 报销人自助维护
|
||||
```
|
||||
|
||||
- 前端 PC 与移动端都从同一参与资格推导 `canMaintainReimbursement`,所有新增、编辑、删除、绑定、解绑、提交、撤回、催办和项目维护入口使用该事实。
|
||||
- 只读视图不渲染禁用的维护按钮;它保留可访问的事实信息和材料查看动作,并用“只读查看”标明当前上下文。
|
||||
- owner 与 submitter 都是参与人,支持代填/代提交的现有合同;其他角色不因 admin 或全量查看权限获得自助维护资格。
|
||||
|
||||
## 服务端数据流
|
||||
|
||||
```text
|
||||
请求权限码通过
|
||||
-> 锁定/读取目标报销
|
||||
-> 校验 actor.id == owner_id OR submitter_id
|
||||
-> 校验动作状态
|
||||
-> 执行业务写入与副作用
|
||||
```
|
||||
|
||||
- `reimbursements.js` 将 `ensureApplicantOrAdmin` 收紧为无 admin 豁免的参与人检查,统一用于报销头、删除、附件、明细和提交;`link-project` 补 `expense.edit_own`,从可查看检查改为参与人写检查,并增加 `draft/rejected` 状态门。
|
||||
- `invoicePool.js` 的 reimbursement bind/unbind 分支移除 `expense.view_all` 写入豁免,只保留 owner/submitter。travel order 分支继续按 `travel.payment.manage`,不受本任务影响。
|
||||
- 对象资格失败统一返回 `403 PERM_001`,并尽量使用稳定的 `only_owner_or_submitter` 详情码;测试断言事务内没有 update/create/delete、文件落盘、审计、workflow 或通知副作用。
|
||||
- `ensureOwnOrCanViewAll` 继续用于详情、附件原件和材料预览等读取接口,并补到会返回预算、费用标准和风险结果的 precheck;这些读取接口不改为参与人专属。
|
||||
- `expense.finance.pay` 和 workflow 路由是显式职责动作,保持原授权和状态机。
|
||||
|
||||
## 前端角色化呈现
|
||||
|
||||
### 参与人维护视图
|
||||
|
||||
- draft/rejected:保留报销头、明细、项目、发票、附件、提交和删除等现有动作。
|
||||
- reviewing:保留撤回和催办;不恢复草稿编辑动作。
|
||||
- 其他状态:沿用现有只读和付款结果呈现。
|
||||
|
||||
### 非参与人只读视图
|
||||
|
||||
- 概况区标明“只读查看”,下一步只描述状态事实,不给报销人操作指令。
|
||||
- 明细、材料对应情况、处理进度、风险、原件和合并预览继续可见。
|
||||
- 发票卡显示实际关联事实:`关联明细:...`、`已关联`;没有 `linkedItemId` 时明确为 `未关联`,不推断自动生成或匹配结果。`已关联` 适用于所有当前不可维护视角,不只限于非参与人。
|
||||
- 不渲染任何维护控件,避免出现按钮后再依赖 403 纠错。
|
||||
- PC 与移动端使用同一文案和资格含义;交互继续使用现有语义化 Button/Link、焦点样式和移动触控尺寸,不引入新的视觉系统。
|
||||
|
||||
## 兼容性与回滚
|
||||
|
||||
- 无 schema、migration、API payload 或角色定义变化;读取 DTO 不变。
|
||||
- 权限收紧会移除管理员/财务通过通用详情维护他人报销的旧行为,这是本任务的预期破坏性修正。历史数据不迁移,已经发生的审计记录不改写。
|
||||
- 回滚仅恢复本任务前后端代码;回滚会重新开放已确认的不安全资格,因此只有新版本造成更高等级阻断且获得明确回滚批准时才执行。
|
||||
- 本任务提交后仍不部署;部署子任务必须冻结新的候选 SHA,不能继续使用 `174cb7a`。
|
||||
|
||||
## 关键权衡
|
||||
|
||||
- 复用同一详情页面但切换角色化模式,避免复制 PC/移动两套详情和数据查询;资格合同通过共享事实和回归保证,不用 URL 区分安全边界。
|
||||
- 只隐藏动作不足以修复问题,因此同步收紧服务端对象资格。
|
||||
- 不给 admin 保留隐式后门。若未来确有代维护需要,应新建设计明确、带原因与审计的管理动作,而不是复用员工自助接口。
|
||||
- admin 豁免关闭后,离职/长期缺席员工的卡住草稿暂不提供代维护路径;真实需求由 Task B 的专用纠错动作或独立运营修复任务承接。
|
||||
- 本轮保留票据和明细事实展示,为财务核验提供上下文;一页式审批的信息结构和金额门禁留给 Task B。
|
||||
@@ -0,0 +1,54 @@
|
||||
# 实施清单:非参与人报销详情只读投影修复
|
||||
|
||||
## 1. 实施前门禁
|
||||
|
||||
- [x] 用户已审查并明确批准本 PRD、设计和实施清单;运行 `task.py start` 后再修改业务代码。
|
||||
- [x] 加载 `trellis-before-dev`,重读 frontend/backend/approval/reimbursement 规约和当前任务材料。
|
||||
- [x] 冻结工作区与基线;确认部署任务仍为 planning,未推送、未部署、未访问或写入生产。
|
||||
|
||||
## 2. 服务端对象资格
|
||||
|
||||
- [x] 将报销头、删除、附件、明细和提交路由的 `ensureApplicantOrAdmin` 收紧为 owner/submitter 参与人检查,保留原状态与权限码门禁。
|
||||
- [x] 为项目关联补 `expense.edit_own`,从 `expense.view_all` 可写收紧为 owner/submitter 可写,并增加 `draft/rejected` 状态门;不改变现有 feature flag 和 payload。
|
||||
- [x] 为 precheck 补 owner/submitter 或 `expense.view_all` 对象级读守卫;普通非参与人 403 且 precheck 服务零调用,合法查看者保持成功。
|
||||
- [x] 将 reimbursement 发票 bind/unbind 分支从 `isOwn || expense.view_all` 收紧为 owner/submitter;保持 travel order 分支不变。
|
||||
- [x] 统一 403 详情码并检查失败顺序,确保资格失败发生在数据、文件、审计、workflow 和通知副作用之前。
|
||||
|
||||
## 3. PC 与移动端动作投影
|
||||
|
||||
- [x] 建立统一的参与人/可维护资格事实,PC 和移动端都使用“参与人 AND 状态”控制维护动作。
|
||||
- [x] PC 收紧主要动作、明细增改删、项目选择、票据添加/移除、附件上传/删除、提交、删除草稿、撤回和催办。
|
||||
- [x] 移动端收紧 sticky submit、明细增改删、票据添加/移除、附件上传/删除、删除草稿、撤回和催办。
|
||||
- [x] 保留 owner/submitter 既有动作、财务/审批专用流程、材料查看和打印。
|
||||
|
||||
## 4. 只读文案与信息
|
||||
|
||||
- [x] 非参与人显示“只读查看”上下文,并把 draft/rejected 的下一步改为等待报销人处理的事实文案。
|
||||
- [x] `linkedItemId` 为空改为“关联明细:未关联”,无法解析改为“关联明细:无法识别”;移除“自动生成或匹配明细”。
|
||||
- [x] 当前不可维护视角的 bound 发票显示“已关联”,不渲染“草稿或驳回状态可移除”;参与人可维护视角保留必要操作说明。
|
||||
- [x] 检查 PC/移动无禁用的伪操作、无内部权限码或实现术语,继续使用语义化按钮/链接和现有焦点/触控合同。
|
||||
|
||||
## 5. 回归测试
|
||||
|
||||
- [x] 服务端合成矩阵按 S1/S2/S3 分档覆盖:admin、finance/`expense.view_all` 非参与人对 form/items/attachments/submit/project/invoice bind-unbind 的 403 与零副作用;approved/paid 项目关联为 409 且零写入。
|
||||
- [x] 服务端保留 owner/submitter 成功、GET/material preview 读取、workflow/finance pay 和 travel invoice 分支基线。
|
||||
- [x] Playwright 覆盖 PC/移动 nonparticipant draft 的无维护动作和文案,并保留 participant 主流程;rejected/reviewing 由相同资格投影与既有撤回/催办测试覆盖。
|
||||
- [x] 运行前端 typecheck/build、后端 node check/Prisma/聚焦测试及相关报销回归。
|
||||
|
||||
## 6. 复核、规约与提交
|
||||
|
||||
- [x] 按 `trellis-check` 复核跨层资格流,修复发现后重跑聚焦与全局门禁。
|
||||
- [x] 因涉及权限模型,实施完成后启动独立只读 review,不让 reviewer 修改代码;结论为 PASS,无权限阻断。
|
||||
- [x] 将“全量查看不等于对象维护、非参与人报销详情默认只读”的稳定合同更新到报销规约,避免再次 fail-open。
|
||||
- [x] 运行 task validate、process contract、Trellis governance、敏感检查和 `git diff --check`。
|
||||
- [x] 用户批准提交后,使用中文并引用 `08-28-reimbursement-nonparticipant-readonly-projection`;业务提交为 `e8d6766`,未推送、未部署、未归档。
|
||||
- [x] 提交完成后更新部署子任务候选依赖;完整 PRD 已明确固定回滚、迁移状态和公网角色检查,继续等待用户审查部署方案。
|
||||
|
||||
## 7. 实施发现与验证记录
|
||||
|
||||
- `link-project` 原实现调用未定义的 `parseBody`,feature flag 打开后会在写入前返回 500,因此其错误的 `expense.view_all` 写授权历史上不可实际利用。本轮在同一路由范围内改为专用 Zod 解析函数,测试覆盖 draft 参与人 200、非参与人 403、approved/paid 409。
|
||||
- r102 完整文件仍为 5/6:唯一失败保持在 `spec.ts:412`,根因仍是 travel category 夹具使 `showDescription=false`,与本轮 diff 无关;本轮新增的管理员 PC/移动只读投影用例通过。
|
||||
- 后端全量 Vitest 为 1303/1306;3 个失败均在既有付款单 BIFF8 生成链路,错误为 `payment_order_generator_unavailable`,本轮 diff 未触及付款批次或生成器。当前任务相关的聚焦与扩展套件均通过。
|
||||
- 独立只读权限复核结论 PASS:非参与人写入守卫位于副作用前,读取、workflow、财务付款和差旅专用分支未收紧。非阻断建议是后续补附件 DELETE 单独用例及 rejected/reviewing UI 状态矩阵。
|
||||
- 提交前复核发现 precheck 缺少对象级读授权;本轮决定直接修复。路由现在先读取报销并复用 `ensureOwnOrCanViewAll`,普通非参与人 403 且 precheck 服务零调用,owner/submitter 与 `expense.view_all` 保持成功。父 task `task.json` 末字节已再次确认为 `0a`。
|
||||
- 本轮没有 schema 或 migration 变化,没有访问生产、推送、部署或归档;部署任务继续处于 planning 并阻塞于本任务复核与提交决定。
|
||||
@@ -0,0 +1,100 @@
|
||||
# 非参与人报销详情只读投影修复
|
||||
|
||||
## Goal
|
||||
|
||||
把报销详情按对象参与资格收敛为两种明确模式:owner/submitter 在允许编辑的状态维护自己的报销;财务、审批人、管理员和其他因职责或 `expense.view_all` 获得访问权的非参与人只查看、核验材料和执行职责内的独立业务动作,不得复用报销人的维护入口或直接变更他人报销数据。
|
||||
|
||||
## Background
|
||||
|
||||
- `ReimbursementDetailPage.tsx:725` 和 `MobileReimbursementDetailPage.tsx:206` 只按 `draft/rejected` 判断可编辑状态。PC 的新增/编辑/删除明细、项目关联、附件上传/删除和主要提交动作,以及移动端对应动作,没有全部叠加 owner/submitter 资格。
|
||||
- PC 与移动端的 `canUnbindInvoices` 都把 `expense.view_all` 当成他人报销的解绑资格;`invoicePool.js:1082` 的服务端解绑合同也允许该权限直接解除他人报销与发票的关联,因此不是单纯的前端误显。
|
||||
- `ensureApplicantOrAdmin` 允许管理员修改、删除、提交他人草稿,新增/修改/删除他人明细和附件;`link-project` 与发票 bind 也把全量查看权限当成写入资格。全量查看、系统管理或当前审批职责不等于报销人自助维护资格。
|
||||
- 发票卡片在所有视角共用 `resolveInvoiceNextStepText`,向非参与人显示“草稿或驳回状态可移除”;`linkedItemId` 为空时又显示技术性兜底“自动生成或匹配明细”。这些文案不是只读核验事实。
|
||||
- 现有事实源要求页面按当前任务角色收敛,审批人和财务只看到职责内动作;报销材料仍允许授权查看原件和合并预览。Task A 已修复审批中撤回入口,但没有覆盖更广的草稿/驳回维护动作和服务端对象资格。
|
||||
|
||||
## Defect Severity
|
||||
|
||||
- **S1 真实写入能力**:reimbursement 发票 unbind/bind 把 `expense.view_all` 当成对象写资格;`ensureApplicantOrAdmin` 允许 admin 修改、删除、提交他人报销头、明细和附件。`link-project` 的授权设计同样错误(仅认证、复用读守卫且没有状态门),但原实现还调用了未定义的 `parseBody`,因此 feature flag 打开后会在写入前 500,历史上该写入洞不可实际利用;本轮仍须同时修复可用性和授权边界。测试必须断言 403/409 及零副作用,不能只检查按钮不可见。
|
||||
- **S2 仅显示或前后端不一致**:PC 多处明细/附件动作和移动端对应动作只按 editable status 渲染,普通财务可能看到按钮但被部分服务端路由拒绝。测试必须断言 PC/移动 DOM 中不存在无资格动作,并保留直接 API 负向回归。
|
||||
- **S3 已有正确边界**:PC/移动报销头编辑器已经同时检查 editable status 和参与人资格;实施只统一资格事实并证明基线,不重复改写已经正确的组件合同。
|
||||
|
||||
## Scope
|
||||
|
||||
- 统一 PC、移动端的报销参与人和可维护资格计算,修正所有报销人自助维护动作的投影。
|
||||
- 收紧报销与发票路由中把 admin 或 `expense.view_all` 当成他人报销维护资格的服务端分支。
|
||||
- 为报销 precheck 补对象级读取授权,避免普通员工横向读取他人预算、费用标准和风险结果。
|
||||
- 为非参与人提供明确的只读查看语义和事实型票据文案,同时保留授权材料查看、打印、审批和财务专属流程。
|
||||
- 使用合成身份和数据覆盖 owner/submitter、财务、管理员及普通非参与查看者,不读取或修改真实报销。
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1 服务端对象资格必须 fail closed
|
||||
|
||||
- 报销 owner 或 submitter 统称参与人。只有参与人可通过报销人自助接口修改报销头、删除草稿、上传/删除附件、新增/修改/删除明细、提交报销、关联项目,以及为该报销绑定或解绑发票;状态和对应权限码仍需同时满足。
|
||||
- `POST /reimbursements/:id/link-project` 必须补 `expense.edit_own` 权限门,只允许参与人在 `draft/rejected` 状态写入;非参与人返回 403,已通过、已支付等非编辑状态返回 409。不得继续使用 `ensureOwnOrCanViewAll` 读守卫授权写入。
|
||||
- admin、财务、审批人及任何仅持 `expense.view_all` 的非参与人调用上述接口时必须稳定返回 `403 PERM_001`,并在读取目标对象后、任何业务写入、文件持久化、审计或通知副作用前失败。
|
||||
- `expense.view_all` 继续允许授权查看报销详情、发票/附件原件和材料合并预览;不得把读权限降级为无权查看。
|
||||
- `POST /reimbursements/:id/precheck` 虽不写业务数据,但会返回预算、费用标准、审批规则和风险结果,必须先读取目标报销并通过 owner/submitter 或 `expense.view_all` 对象级读守卫;普通非参与人返回 403,且不得调用 precheck 服务。审批页既有 `standardViolations` 快照继续保留。
|
||||
- 显式职责动作保持独立:workflow 审批/退回、财务付款及付款批次接口不改;差旅订单 bind/unbind 分支不改。若未来需要财务纠正票据归属,必须在 Task B 设计专用、可审计的业务动作,不能继续借用报销人解绑接口。
|
||||
- 不改变状态机、权限码集合、角色定义、数据库 schema 或迁移。
|
||||
|
||||
### R2 PC 与移动端统一维护资格
|
||||
|
||||
- 页面使用“参与人 AND 状态允许编辑”计算统一的 `canMaintainReimbursement`,不得再把 `draft/rejected` 或 `expense.view_all` 单独当成维护资格。
|
||||
- 非参与人不得看到或触发:新增/编辑/删除明细、修改报销日期/事由/差旅摘要、关联项目下拉框、添加/移除发票、上传/删除附件、提交报销、删除草稿、撤回、催办等报销人自助动作。
|
||||
- owner/submitter 在 draft/rejected 状态继续看到并可使用既有维护动作;owner/submitter 在 reviewing 状态继续使用 Task A 已修复的撤回和催办合同,其他状态行为不变。
|
||||
- 只读查看者可继续查看报销概要、明细事实、处理进度、材料原件/合并预览、风险与材料对应情况,以及权限允许的打印入口;页面不通过禁用按钮暗示不可用动作,而是完全不渲染无资格动作。
|
||||
- PC 和移动端必须使用同一资格语义,不能只修截图对应的 PC 页面。
|
||||
|
||||
### R3 非参与人使用事实型查看文案
|
||||
|
||||
- 同一详情路由可以复用,但非参与人应显示明确的“只读查看”上下文,不伪装成报销人维护页,也不新建一套重复详情页面。
|
||||
- 非参与人的“下一步”按当前业务状态表达观察事实:草稿为“等待报销人完善”,驳回为“等待报销人修改后重提”,审批中显示当前处理事实,已通过为“等待财务处理”,已支付为“查看付款结果”。不得提示其新增明细、添加发票或提交审批。
|
||||
- 发票存在真实 `linkedItemId` 时显示 `关联明细:<费用类型 · 明细说明>`;为空时显示 `关联明细:未关联`,关联对象无法解析时显示 `关联明细:无法识别`。不得再显示“自动生成或匹配明细”。
|
||||
- bound 发票在当前不可维护视角(包括非参与人,以及 owner/submitter 查看 approved/paid 等非编辑状态)使用事实状态“已关联”,且不显示“草稿或驳回状态可移除”等操作提示。只有参与人的可维护视角可保留必要的移除说明。
|
||||
- 材料对应、票面金额、日期和风险提示继续作为财务/审批核验信息展示;本轮不重构为 Task B 的一页式审批工作台。
|
||||
|
||||
### R4 回归与可部署基线
|
||||
|
||||
- 前端回归至少覆盖 PC 和移动端的非参与管理员/财务查看他人 draft、rejected、reviewing 报销,断言全部维护动作不存在、只读文案正确、原件和合并预览入口仍可用。
|
||||
- owner/submitter 合成场景继续覆盖 draft/rejected 维护动作和 reviewing 撤回,防止把页面整体改成只读。
|
||||
- 服务端对每类维护接口至少覆盖一个非参与高权限身份的 `403` 和零副作用,并保留参与人成功基线;发票 bind/unbind 同时保留差旅分支现有合同。
|
||||
- 权限与审批规则改动实施后必须完成独立只读复核;未经复核和新工作提交,不更新部署候选、不推送、不部署。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] 非参与管理员或财务查看他人 draft/rejected 报销时,PC 和移动端不存在明细、项目、票据、附件、提交、删除、撤回和催办等报销人维护动作。
|
||||
- [ ] 同一非参与人仍能查看报销概要、明细事实、处理进度、材料原件和合并预览;reviewing 审批/财务职责入口仍在原工作流页面正常使用。
|
||||
- [ ] 非参与人看到“只读查看”上下文;未关联票据显示“关联明细:未关联”,bound 状态显示“已关联”,页面不存在“自动生成或匹配明细”和“草稿或驳回状态可移除”。
|
||||
- [ ] 非参与人直接调用报销头修改/删除、附件增删、明细增改删、提交、项目关联、发票 bind/unbind 均返回 `403 PERM_001`,且对应数据、文件、审计、workflow 和通知零副作用。
|
||||
- [ ] 普通非参与人调用 precheck 返回 `403 PERM_001` 且 precheck 服务零调用;owner/submitter 与 `expense.view_all` 查看者保持成功读取基线。
|
||||
- [ ] owner/submitter 在 approved/paid 等非编辑状态调用 `link-project` 返回 409 且项目归属不变;draft/rejected 且具备 `expense.edit_own` 时保持成功基线。
|
||||
- [ ] owner/submitter 在既有允许状态仍可维护、提交、撤回和催办自己的报销;现有状态校验、材料查看、打印、财务付款、workflow 审批和差旅票据分支不变。
|
||||
- [ ] 无 schema、迁移、权限码或角色定义变更;前后端聚焦测试、前端 typecheck/build、相关 Playwright、Trellis 治理、敏感检查和 `git diff --check` 通过。
|
||||
- [ ] 只读权限复核结论为 PASS 后才形成新工作提交;部署子任务随后重新冻结包含 Task A 与本任务的候选 SHA,原 `174cb7a` 不再作为最终部署基线。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Task B 的财务一页式审批、费用口径、`requires_invoice`、票据—明细强制关联、金额差异门禁和多票拆分。
|
||||
- 为财务或管理员新增“代员工修改”“纠正票据归属”或其他管理修复入口。
|
||||
- 修改 workflow 审批/退回、付款批次、财务付款、材料查看授权、打印权限、发票 OCR 或材料合并算法。
|
||||
- 处理 `BX-20260828-0017` 的真实单据、票面差额、退回、付款或附件重传。
|
||||
- 修复 r102 差旅夹具测试债;该项仍按独立任务处理。
|
||||
- 推送、部署、公网验证、生产配置、生产数据写入和任务归档。
|
||||
|
||||
## Risks and Constraints
|
||||
|
||||
- 服务端移除 admin/`expense.view_all` 写入豁免属于真实权限收紧,必须用直接接口测试证明读取能力仍在、写入统一拒绝。
|
||||
- 不能只隐藏前端动作;否则旧客户端或直接 API 仍可变更他人报销。
|
||||
- 不能把所有非参与人操作一刀切:workflow 审批、财务付款和授权材料查看是职责动作,必须与报销人维护动作区分。
|
||||
- 移除 admin 通用写入豁免后,员工离职或长期缺席造成的卡住草稿暂时没有代维护入口;若真实发生,应在 Task B 的专用可审计纠错动作或独立运营修复任务中重新规划,本任务不以保留通用后门兜底。
|
||||
- 同一页面角色化比复制新页面更低风险,但 PC/移动端必须共享资格事实和文案合同,避免再次漂移。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 服务端语法与基础:`cd server && node --check src/routes/reimbursements.js && node --check src/routes/invoicePool.js && npm run prisma:validate`。
|
||||
- 服务端聚焦:新增或扩展报销参与人资格矩阵测试,覆盖 form/items/attachments/submit/project/invoice bind-unbind 的非参与高权限 403、零副作用和参与人成功基线。
|
||||
- 前端:`cd client && npm run typecheck && npm run build`。
|
||||
- Playwright:扩展 `r102-reimbursement-main-flow.spec.ts` 或新增聚焦 spec,使用阻断 Service Worker 的合成 PC/移动身份覆盖 participant 与 nonparticipant 的 draft/rejected/reviewing 投影及只读票据文案。
|
||||
- 全局回归:运行相关报销、材料预览和系统动作确认用例;完整 r102 的既有差旅夹具失败必须如实单列,不得把它当成本任务新失败或默认绿色基线。
|
||||
- 治理:`node scripts/qa/check-process-contract.mjs --baseline --json`、`node scripts/qa/check-trellis-governance.mjs`、`node scripts/qa/check-sensitive-content.mjs`、`git diff --check`。
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"id": "reimbursement-nonparticipant-readonly-projection",
|
||||
"name": "reimbursement-nonparticipant-readonly-projection",
|
||||
"title": "非参与人报销详情只读投影修复",
|
||||
"description": "收紧非 owner/submitter 查看报销详情时的 PC、移动端操作投影与服务端变更资格,并改为面向审批/浏览者的只读票据文案。",
|
||||
"status": "in_progress",
|
||||
"dev_type": null,
|
||||
"scope": "PC/mobile reimbursement read-only projection; reimbursement and invoice-pool object authorization; role-aware invoice copy",
|
||||
"package": null,
|
||||
"priority": "P0",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-28",
|
||||
"completedAt": null,
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": "e8d6766",
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": "08-28-reimbursement-detail-material-preview-fixes",
|
||||
"relatedFiles": [
|
||||
"client/src/pages/ReimbursementDetailPage.tsx",
|
||||
"client/src/pages/MobileReimbursementDetailPage.tsx",
|
||||
"client/src/lib/invoiceStatus.ts",
|
||||
"server/src/routes/reimbursements.js",
|
||||
"server/src/routes/invoicePool.js"
|
||||
],
|
||||
"notes": "实施与独立只读权限复核 PASS,业务提交为 e8d6766;同时补齐 precheck 对象级读授权。财务/审批详情中的行内费用标准提示当前不主动请求 precheck,由 Task B 的 B3 一页式审批恢复,现有 budgetHint、riskFlags 和审批页 standardViolations 快照不受影响。未推送、未部署、未归档,部署仍须单独批准。",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,47 @@
|
||||
# 设计:报销规范核查与审批文案优化
|
||||
|
||||
## 边界
|
||||
|
||||
本任务包含两个相互关联但可独立验证的输出:生产 4 笔报销的只读规范性报告,以及负责人本人财务兜底节点的用户文案修复。前者不写业务数据;后者只调整步骤命名与详情显示投影,不改审批规则、角色、权限或数据库结构。
|
||||
|
||||
## 数据流
|
||||
|
||||
### 报销核查
|
||||
|
||||
```text
|
||||
生产只读数据 → 按提交人/日期冻结 4 个业务编号 → 最小字段核查
|
||||
→ 对照报销/发票/附件事实源 → 脱敏逐笔结论
|
||||
```
|
||||
|
||||
- 优先使用现有受控生产只读连接或已授权 API 身份。
|
||||
- 查询结果只在进程内用于判断;任务文档只登记问题分类、业务编号和必要的非敏感摘要。
|
||||
- 不打开或下载原始附件时,只把“需要财务查看原件”列为保留项,不推断真实性。
|
||||
|
||||
### 审批文案
|
||||
|
||||
```text
|
||||
审批规则解析
|
||||
→ buildRoleApprovalStep(保存步骤 ID、角色与业务步骤名)
|
||||
→ workflow task / workflow_steps_snapshot(历史事实)
|
||||
→ reimbursement processingProgress 显示投影
|
||||
→ 当前处理 / 下一步 / 时间线
|
||||
```
|
||||
|
||||
负责人本人兜底的技术原因继续由步骤 ID 后缀 `_owner_self_fallback`、分支原因和角色规则表达。业务步骤名改为按角色映射后的 `财务复核`,避免新记录继续污染用户入口。针对已经保存旧步骤名的历史记录,`resolveApprovalStageLabel` 将 `负责人本人申请兜底:<role>` 识别为内部标签,再按 `assignee_role` 映射为 `财务复核`。
|
||||
|
||||
`processingProgress` 当前阶段的 `detail` 不再拼接旧步骤名,而使用规范化后的 stage label,因此详情页现有 `nextStepText` 可直接显示 `待财务复核处理`。前端无需引入第二套文案映射。
|
||||
|
||||
## 兼容性
|
||||
|
||||
- 不回写历史工作流实例或任务。
|
||||
- 普通的自定义可读节点名继续保留;只把明确的负责人本人兜底内部标签、序号、通用名和技术标识走角色映射。
|
||||
- `finance` 与 `finance_mgr` 的显示均为 `财务复核`;审批资格仍按原角色执行。
|
||||
- 报销审批历史如直接读取旧 `step_name`,需在同一服务端显示边界复用规范化函数,避免详情页不同区域一处修好、一处仍泄漏。
|
||||
- 待办中心的历史任务若仍直接暴露旧名,本任务只做必要的共享显示归一化,不改变任务 API 的权限与动作模型。
|
||||
|
||||
## 风险与回滚
|
||||
|
||||
- 风险:过宽地判定“不可读节点名”会覆盖合法自定义名称。控制方式是只增加负责人本人兜底前缀的精确识别,并保留现有名称优先级测试。
|
||||
- 风险:仅改生成名无法修复已经提交的 4 笔。控制方式是在读取投影层同时兼容历史名。
|
||||
- 回滚:代码回退即可;无 schema、配置或数据回滚。
|
||||
- 公网部署不在默认执行范围,必须另行明确批准。
|
||||
@@ -0,0 +1 @@
|
||||
{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}
|
||||
@@ -0,0 +1,44 @@
|
||||
# 实施清单:报销规范核查与审批文案优化
|
||||
|
||||
## 1. 只读核查
|
||||
|
||||
- [x] 确认受控生产只读访问方式,不打印或持久化凭据。
|
||||
- [x] 按 `majianyong`、提交日期 `2026-08-28` 查询并冻结 4 个业务编号;数量漂移则停止。
|
||||
- [x] 读取最小核查字段,逐笔完成事由、项目、明细、日期金额、发票对应和补充材料检查。
|
||||
- [x] 形成脱敏结论与建议,不执行任何报销、附件、审批、付款或企微写操作。
|
||||
|
||||
## 2. 文案实现
|
||||
|
||||
- [x] 修改负责人本人兜底步骤的业务名称,保留内部步骤 ID 和审批路由原因。
|
||||
- [x] 扩展报销处理进度的显示归一化,兼容 4 笔已保存的旧快照和任务名。
|
||||
- [x] 让当前阶段 detail 使用规范化的节点标签,得到 `待财务复核处理`。
|
||||
- [x] 检查审批历史、PC 待办和移动待办的直接消费路径,只修复会继续暴露相同内部标签的位置。
|
||||
|
||||
## 3. 测试与复核
|
||||
|
||||
- [x] 补充 workflow 和 reimbursement detail 聚焦单测,覆盖新旧步骤名及普通链路。
|
||||
- [x] 运行后端语法、Prisma、聚焦 Vitest。
|
||||
- [x] 运行前端 typecheck/build 与聚焦 PC 报销详情 Playwright。
|
||||
- [x] 主会话进行只读 diff 复核,确认自审、权限、状态机、任务推进、历史快照和审批规则未改变。
|
||||
- [x] 运行流程、文档、Trellis 治理和 `git diff --check`。
|
||||
|
||||
## 4. 交付边界
|
||||
|
||||
- [x] 汇总 4 笔脱敏核查结论、代码变化和验证结果。
|
||||
- [x] 如需公网部署,先向用户单独申请批准;部署前不执行生产写入,部署后通过 `/ready`、构建号和公网只读检查后才能通知测试。
|
||||
|
||||
## 脱敏核查结论
|
||||
|
||||
- `BX-20260828-0014`:硬件类 5 张发票合并为 1 条明细,发票日期跨 5—7 月但明细只填一个日期;金额超过 2 万元且无订单、验收或资产信息附件。建议撤回后按实际设备/采购批次拆分,补用途、使用人/保管人、订单验收和资产候选说明。
|
||||
- `BX-20260828-0016`:4 张外部技术服务发票合并为 1 条笼统明细,发票日期跨 5—6 月且均未关联具体明细,无服务范围、服务期间、成果或订单/合同摘要材料。建议撤回后按服务事项或期间拆分并补业务依据。
|
||||
- `BX-20260828-0017`:2 张办公用品发票未关联明细,发票合计与申报金额相差 1,949.81 元;虽有 2 张图片附件,但材料类型和名称不能说明是支付、订单、验收或差额说明。建议撤回核对申报金额,逐票关联并明确差额和附件用途。
|
||||
- `BX-20260828-0018`:5 张办公用品发票金额合计与申报一致,但均未关联明细;明细日期晚于全部发票日期,事由和说明仅写“办公用品采购”,无订单/物品清单或用途材料。建议撤回后按购买内容/日期拆分并逐票关联。
|
||||
|
||||
本轮仅执行生产只读 SQL;未打开或下载发票原件和附件,未写生产数据库、审批、付款或企业微信。
|
||||
|
||||
## 关键文件与回滚点
|
||||
|
||||
- `server/src/utils/workflow.js`:新工作流步骤业务名;回退单个命名分支即可。
|
||||
- `server/src/routes/reimbursements.js`:历史显示投影;回退标签识别与 detail 生成即可。
|
||||
- `server/tests/unit/utils/workflow.test.js`、`server/tests/unit/routes/reimbursements.detail.test.js`:聚焦回归。
|
||||
- 如待办入口确需同步,优先复用已有显示函数或服务端投影,不新增页面级散落映射。
|
||||
@@ -0,0 +1,71 @@
|
||||
# 报销规范核查与审批文案优化
|
||||
|
||||
## Goal
|
||||
|
||||
核查 `majianyong` 于 2026-08-28 提交的 4 笔报销是否满足当前报销、发票和附件规范,并将“负责人本人申请兜底:finance”这类内部审批路由说明替换为用户能直接理解的财务处理文案,同时保持禁止自审和历史审批事实不变。
|
||||
|
||||
## Background
|
||||
|
||||
- 用户反馈 `majianyong` 当前提交了 4 笔填写不够规范的报销,并提供了其中 `BX-20260828-0018` 的详情截图。
|
||||
- 截图可确认该单为办公用品报销:整单事由与唯一明细说明均为泛化的“办公用品采购”,未关联项目;5 张发票自动归入 1 条总额明细,材料对应区提示“待关联发票或补充无票说明”。
|
||||
- 当前正式组织口径是 `majianyong` 为公司负责人 `ceo`、不配置直属主管且不得自审;负责人本人提交报销时由唯一 active 非本人 `finance` 处理。
|
||||
- 后端在生成负责人本人兜底步骤时把内部原因和角色代码拼为 `负责人本人申请兜底:finance`,报销详情的处理进度投影把它当作可读节点名,因而在“下一步”“当前处理”和时间线中直接展示。
|
||||
- 现有稳定规约已经规定审批投影不得暴露角色代码或技术标识,`finance|finance_mgr` 应显示为“财务复核”。
|
||||
|
||||
## Scope
|
||||
|
||||
- 生产 4 笔报销的最小字段只读核查与脱敏结论。
|
||||
- 负责人本人财务兜底步骤的新记录业务命名。
|
||||
- 报销详情、PC/移动审批任务等用户入口对历史兜底步骤名的只读显示投影。
|
||||
- 聚焦后端回归、前端构建与报销详情页面验证;不包含公网部署。
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1 生产报销只读核查
|
||||
|
||||
- 只读定位 `majianyong` 在 2026-08-28 提交的 4 笔报销;若实际结果不恰好为 4 笔,应停止逐笔结论并报告范围漂移。
|
||||
- 仅读取核查所需的最小字段:业务编号、状态、报销日期、事由、项目关联、费用类型、明细日期/金额/说明、发票与明细关联状态、补充附件类型/说明、材料完整性提示和审批节点摘要。
|
||||
- 按当前事实源检查:事由和明细说明是否能说明购买/支出内容及业务用途;费用类型和日期是否准确;项目性费用是否关联项目;多张发票是否合理拆分或能与明细对应;发票是否逐条关联;支付截图、订单或情况说明等补充材料是否足以支撑实付与用途。
|
||||
- 输出逐笔脱敏问题清单,区分“应退回补正”“建议补充”“仅需财务核对”;不得在任务文件、仓库或聊天中记录完整发票号码、附件原文、原始 payload、账号、凭据或存储路径。
|
||||
- 本任务不替用户编辑、撤回、审批或付款,不写生产数据库,不写企业微信。
|
||||
|
||||
### R2 用户可见审批文案
|
||||
|
||||
- `owner_self_fallback` 继续作为内部路由/审计语义,仍须保持 active 非本人财务处理和禁止自审。
|
||||
- 报销详情中该节点统一显示为 `财务复核`;“当前处理”按现有结构显示 `财务复核 · <实际处理人姓名>`,不显示内部角色代码;“下一步”显示 `待财务复核处理`。
|
||||
- 处理进度时间线对当前和历史负责人本人兜底快照均显示 `财务复核`,不得要求迁移或重写历史 `workflow_steps_snapshot`。
|
||||
- 新提交报销生成的业务步骤名使用可读文案,内部兜底原因保留在稳定步骤 ID/路由逻辑中,不能因文案调整丢失自审保护或审批人解析依据。
|
||||
- 聚焦核对待办/移动审批等直接使用任务步骤名的用户入口;如它们会继续暴露该内部文案,应复用同一服务端显示语义或确保新任务不再生成内部文案。不得进行通用工作流 UI 重构。
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] 生产只读结果恰好覆盖 `majianyong` 于 2026-08-28 提交的 4 笔报销,并形成不含敏感原文的逐笔规范性结论。
|
||||
- [x] 每笔结论至少覆盖事由、项目、明细拆分/说明、日期与金额、发票对应、补充材料六类检查,并给出明确处理建议。
|
||||
- [x] `BX-20260828-0018` 不再把 5 张发票仅以泛化“办公用品采购”视为充分说明;结论明确指出其明细对应和用途证据缺口。
|
||||
- [x] 负责人本人申请仍只由 active 非本人 `finance` 处理,不产生自审、不改变普通员工审批链。
|
||||
- [x] 报销详情的“当前处理”“下一步”和处理进度不再显示 `负责人本人申请兜底`、`finance` 或其他内部步骤标识,分别使用“财务复核 · 处理人”“待财务复核处理”和“财务复核”。
|
||||
- [x] 已存在的历史兜底快照无需数据迁移即可得到同样的用户可读投影;新提交记录不再生成内部原因作为用户可见步骤名。
|
||||
- [x] 后端聚焦测试覆盖历史步骤名、新步骤名、普通财务节点、非财务节点与禁止自审回归;相关前端类型检查、构建和报销详情 Playwright 通过。
|
||||
- [x] 审批/workflow 变更完成后进行一次只读代码复核,确认权限、状态机、快照和审批规则语义未扩大。
|
||||
- [x] 未经另行批准不部署公网;部署并通过公网只读检查前,不告知用户线上可测试。
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- 修改、撤回、驳回、通过这 4 笔真实报销,或替员工补录材料。
|
||||
- 修改审批角色、负责人直属主管、审批金额档、生产审批规则或历史快照。
|
||||
- 执行真实付款、付款批次操作、附件下载、企业微信写入或数据库迁移。
|
||||
- 新增通用 BPM、重构全部业务域审批文案或改变现有页面布局。
|
||||
|
||||
## Risks and Constraints
|
||||
|
||||
- 生产数据可能已发生状态变化;只读查询必须冻结查询时间并以业务编号为结论锚点。
|
||||
- 发票 PDF 和支付证明需要人工看原件才能判断真实性;本轮自动/字段核查只能判断材料结构与对应关系,不能替代财务验真。
|
||||
- 历史快照是审批事实,文案修复必须在显示投影层兼容,不得回写历史数据。
|
||||
|
||||
## Test Plan
|
||||
|
||||
- 聚焦后端:`cd server && npx vitest run tests/unit/utils/workflow.test.js tests/unit/routes/reimbursements.detail.test.js`。
|
||||
- 基础后端:`cd server && npm run prisma:validate`,修改的 JavaScript 文件执行 `node --check`。
|
||||
- 前端:`cd client && npm run typecheck && npm run build`;运行覆盖 PC 报销详情的聚焦 Playwright。
|
||||
- 流程治理:`node scripts/qa/check-process-contract.mjs --baseline --json`、`node scripts/qa/audit-docs.mjs --check-registry`、`node scripts/qa/check-trellis-governance.mjs`、`git diff --check`。
|
||||
- 人工只读复核:检查 diff 未改变审批资格、任务推进、历史快照持久化或生产数据。
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"id": "reimbursement-review-routing-copy",
|
||||
"name": "reimbursement-review-routing-copy",
|
||||
"title": "报销规范核查与审批文案优化",
|
||||
"description": "只读核查 majianyong 的 4 笔报销规范性,并修正负责人本人财务兜底节点的用户可见文案。",
|
||||
"status": "completed",
|
||||
"dev_type": null,
|
||||
"scope": "server/workflow and reimbursement display projection; production reimbursement read-only review",
|
||||
"package": null,
|
||||
"priority": "P2",
|
||||
"creator": "shunli",
|
||||
"assignee": "shunli",
|
||||
"createdAt": "2026-08-28",
|
||||
"completedAt": "2026-08-28",
|
||||
"branch": null,
|
||||
"base_branch": "main",
|
||||
"worktree_path": null,
|
||||
"commit": null,
|
||||
"pr_url": null,
|
||||
"subtasks": [],
|
||||
"children": [],
|
||||
"parent": null,
|
||||
"relatedFiles": [],
|
||||
"notes": "",
|
||||
"meta": {}
|
||||
}
|
||||
@@ -198,12 +198,12 @@ export function MobileReimbursementDetailPage() {
|
||||
const detail = detailQuery.data;
|
||||
const canUploadInvoice = hasPermission(permissions, "invoice.upload");
|
||||
const canViewOwnInvoice = hasPermission(permissions, "invoice.view_own");
|
||||
const canViewAllExpense = hasPermission(permissions, "expense.view_all");
|
||||
const canDeleteExpense = hasPermission(permissions, "expense.delete_own");
|
||||
const canSubmitExpense = hasPermission(permissions, "expense.submit");
|
||||
const canAccessAssets = hasAnyPermission(permissions, ["asset.view_own", "asset.view_all", "asset.manage", "asset.transfer", "asset.inventory"]);
|
||||
const isParticipant = Boolean(detail && me?.id && [detail.ownerId, detail.submitterId].includes(me.id));
|
||||
const canEdit = isEditableStatus(detail?.status);
|
||||
const canMaintainReimbursement = Boolean(canEdit && isParticipant);
|
||||
const pendingTaskId = detail?.pendingWorkflow?.task?.id ?? null;
|
||||
const canRemindPendingWorkflow = Boolean(
|
||||
detail?.status === "reviewing" &&
|
||||
@@ -211,14 +211,14 @@ export function MobileReimbursementDetailPage() {
|
||||
isParticipant &&
|
||||
detail?.pendingWorkflow?.task?.assignee_id !== me?.id,
|
||||
);
|
||||
const canOpenInvoiceDialog = Boolean(canEdit && isParticipant && canUploadInvoice && canViewOwnInvoice);
|
||||
const canUnbindInvoices = Boolean(canEdit && (isParticipant || canViewAllExpense));
|
||||
const canDeleteDraft = Boolean(detail?.status === "draft" && canDeleteExpense && (isParticipant || canViewAllExpense));
|
||||
const canManageAttachments = Boolean(canEdit && (isParticipant || canViewAllExpense));
|
||||
const canOpenInvoiceDialog = Boolean(canMaintainReimbursement && canUploadInvoice && canViewOwnInvoice);
|
||||
const canUnbindInvoices = canMaintainReimbursement;
|
||||
const canDeleteDraft = Boolean(detail?.status === "draft" && canDeleteExpense && isParticipant);
|
||||
const canManageAttachments = canMaintainReimbursement;
|
||||
const selectedInvoiceEntries = Object.entries(selectedInvoices);
|
||||
|
||||
const reimbursementPrecheckQuery = useQuery({
|
||||
enabled: Boolean(reimbursementId && canSubmitExpense && detail),
|
||||
enabled: Boolean(reimbursementId && canSubmitExpense && canMaintainReimbursement),
|
||||
queryKey: ["reimbursements", reimbursementId, "precheck"],
|
||||
queryFn: () => precheckReimbursement(reimbursementId),
|
||||
});
|
||||
@@ -834,10 +834,10 @@ export function MobileReimbursementDetailPage() {
|
||||
|
||||
const resolveLinkedItemLabel = (linkedItemId: string | null) => {
|
||||
if (!linkedItemId) {
|
||||
return "自动生成或匹配明细";
|
||||
return "未关联";
|
||||
}
|
||||
const item = detail?.items.find((candidate) => candidate.id === linkedItemId);
|
||||
return item ? itemLabel(item) : "明细";
|
||||
return item ? itemLabel(item) : "无法识别";
|
||||
};
|
||||
|
||||
const expenseTypes = expenseTypesQuery.data?.items ?? [];
|
||||
@@ -874,7 +874,7 @@ export function MobileReimbursementDetailPage() {
|
||||
}
|
||||
|
||||
const displayCode = resolveReimbursementDisplayCode(detail);
|
||||
const showSubmitAction = Boolean(canEdit && canSubmitExpense);
|
||||
const showSubmitAction = Boolean(canMaintainReimbursement && canSubmitExpense);
|
||||
const showStickyActions = showSubmitAction;
|
||||
const showSecondaryActions = Boolean(actionError || canDeleteDraft);
|
||||
|
||||
@@ -903,6 +903,16 @@ export function MobileReimbursementDetailPage() {
|
||||
<div className="mobile-section-header mobile-section-header--top">
|
||||
<div>
|
||||
<h2 className="mobile-section-title">报销概况</h2>
|
||||
{!isParticipant ? (
|
||||
<p className="mt-1 text-xs text-zinc-500">
|
||||
只读查看
|
||||
{detail.status === "draft"
|
||||
? " · 等待报销人完善"
|
||||
: detail.status === "rejected"
|
||||
? " · 等待报销人修改后重提"
|
||||
: ""}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
<div data-testid="mobile-reimbursement-summary">
|
||||
@@ -990,7 +1000,7 @@ export function MobileReimbursementDetailPage() {
|
||||
<section className="mobile-card" data-testid="mobile-reimbursement-items">
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
<h2 className="text-base font-semibold text-zinc-900">报销明细</h2>
|
||||
{canEdit ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<Button size="sm" variant="secondary" type="button" onClick={startCreateItem}>
|
||||
新增
|
||||
</Button>
|
||||
@@ -1076,7 +1086,7 @@ export function MobileReimbursementDetailPage() {
|
||||
<div className="shrink-0 text-base font-semibold text-zinc-900">{formatMoney(item.amount)}</div>
|
||||
</div>
|
||||
{item.description ? <p className="mt-2 text-sm leading-5 text-zinc-600">{item.description}</p> : null}
|
||||
{canEdit ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<div className="mobile-two-action-grid mt-3">
|
||||
<Button size="sm" variant="secondary" type="button" onClick={() => startEditItem(item)}>
|
||||
编辑
|
||||
@@ -1220,7 +1230,9 @@ export function MobileReimbursementDetailPage() {
|
||||
<div className="mt-3 space-y-2">
|
||||
<div data-testid={TEST_IDS.reimbursement.invoices} className="space-y-2">
|
||||
{detail.invoices.map((invoice) => {
|
||||
const invoiceHint = resolveInvoiceNextStepText(invoice);
|
||||
const invoiceHint = invoice.status === "bound" && !canMaintainReimbursement
|
||||
? "已关联"
|
||||
: resolveInvoiceNextStepText(invoice);
|
||||
return (
|
||||
<article key={invoice.id} className="mobile-list-card">
|
||||
<button
|
||||
|
||||
@@ -158,6 +158,12 @@ export function ReimbursementDetailPage() {
|
||||
});
|
||||
|
||||
const projectsQuery = useQuery({
|
||||
enabled: Boolean(
|
||||
detailQuery.data &&
|
||||
me?.id &&
|
||||
[detailQuery.data.ownerId, detailQuery.data.submitterId].includes(me.id) &&
|
||||
["draft", "rejected"].includes(detailQuery.data.status),
|
||||
),
|
||||
queryKey: ["projects", "reimbursement-link"],
|
||||
queryFn: () => fetchProjects({ mine: true }),
|
||||
});
|
||||
@@ -165,12 +171,18 @@ export function ReimbursementDetailPage() {
|
||||
const canSubmitExpense = hasPermission(permissions, "expense.submit");
|
||||
const canUploadInvoice = hasPermission(permissions, "invoice.upload");
|
||||
const canViewOwnInvoice = hasPermission(permissions, "invoice.view_own");
|
||||
const canViewAllExpense = hasPermission(permissions, "expense.view_all");
|
||||
const canDeleteExpense = hasPermission(permissions, "expense.delete_own");
|
||||
const canAccessAssets = hasAnyPermission(permissions, ["asset.view_own", "asset.view_all", "asset.manage", "asset.transfer", "asset.inventory"]);
|
||||
|
||||
const reimbursementPrecheckQuery = useQuery({
|
||||
enabled: Boolean(reimbursementId) && canSubmitExpense && Boolean(detailQuery.data),
|
||||
enabled: Boolean(
|
||||
reimbursementId &&
|
||||
canSubmitExpense &&
|
||||
detailQuery.data &&
|
||||
me?.id &&
|
||||
[detailQuery.data.ownerId, detailQuery.data.submitterId].includes(me.id) &&
|
||||
["draft", "rejected"].includes(detailQuery.data.status),
|
||||
),
|
||||
queryKey: ["reimbursements", reimbursementId, "precheck"],
|
||||
queryFn: () => precheckReimbursement(reimbursementId),
|
||||
});
|
||||
@@ -724,6 +736,7 @@ export function ReimbursementDetailPage() {
|
||||
const budgetHint = detail.budgetHint;
|
||||
const canEditItems = ["draft", "rejected"].includes(detail.status);
|
||||
const isReimbursementParticipant = Boolean(me?.id) && [detail.ownerId, detail.submitterId].includes(me?.id ?? "");
|
||||
const canMaintainReimbursement = canEditItems && isReimbursementParticipant;
|
||||
const pendingTaskId = detail.pendingWorkflow?.task?.id ?? null;
|
||||
const canRemindPendingWorkflow = Boolean(
|
||||
detail.status === "reviewing" &&
|
||||
@@ -736,16 +749,16 @@ export function ReimbursementDetailPage() {
|
||||
isReimbursementParticipant &&
|
||||
canUploadInvoice &&
|
||||
canViewOwnInvoice;
|
||||
const canUnbindInvoices = canEditItems && (isReimbursementParticipant || canViewAllExpense);
|
||||
const canUnbindInvoices = canMaintainReimbursement;
|
||||
const canDeleteReimbursement =
|
||||
detail.status === "draft" &&
|
||||
canDeleteExpense &&
|
||||
(isReimbursementParticipant || canViewAllExpense);
|
||||
isReimbursementParticipant;
|
||||
const precheckStandardChecks = reimbursementPrecheckQuery.data?.standardChecks ?? [];
|
||||
const blockStandardChecks = precheckStandardChecks.filter((item) => item.exceeded && item.controlMode === "block");
|
||||
const warnStandardChecks = precheckStandardChecks.filter((item) => item.exceeded && item.controlMode === "warn");
|
||||
const missingInvoiceSubmitMessage = "提交前需先添加至少 1 张发票。";
|
||||
const submitInvoiceNotice = canEditItems && detail.invoices.length < 1 ? missingInvoiceSubmitMessage : "";
|
||||
const submitInvoiceNotice = canMaintainReimbursement && detail.invoices.length < 1 ? missingInvoiceSubmitMessage : "";
|
||||
const selectedInvoiceEntries = Object.entries(selectedInvoices);
|
||||
|
||||
function handleInvoicePdfSelection(files: FileList | null) {
|
||||
@@ -886,12 +899,12 @@ export function ReimbursementDetailPage() {
|
||||
|
||||
const resolveLinkedItemLabel = (linkedItemId: string | null) => {
|
||||
if (!linkedItemId) {
|
||||
return "自动生成或匹配明细";
|
||||
return "未关联";
|
||||
}
|
||||
|
||||
const linkedItem = detail.items.find((item) => item.id === linkedItemId);
|
||||
if (!linkedItem) {
|
||||
return "未识别明细";
|
||||
return "无法识别";
|
||||
}
|
||||
|
||||
return `${linkedItem.expenseType?.name ?? "未识别费用类型"} · ${linkedItem.description ?? "未填写说明"}`;
|
||||
@@ -909,6 +922,9 @@ export function ReimbursementDetailPage() {
|
||||
: detail.status === "paid" ? "已完成" : "待处理";
|
||||
const nextStepText = (() => {
|
||||
if (["draft", "rejected"].includes(detail.status)) {
|
||||
if (!isReimbursementParticipant) {
|
||||
return detail.status === "rejected" ? "等待报销人修改后重提" : "等待报销人完善";
|
||||
}
|
||||
if (!itemCount) {
|
||||
return "补报销明细";
|
||||
}
|
||||
@@ -933,7 +949,7 @@ export function ReimbursementDetailPage() {
|
||||
})();
|
||||
const showReviewHints = Boolean(budgetHint || detail.riskFlags.length);
|
||||
const primaryAction = (() => {
|
||||
if (["draft", "rejected"].includes(detail.status) && canEditItems) {
|
||||
if (canMaintainReimbursement) {
|
||||
if (!itemCount) {
|
||||
return (
|
||||
<Button
|
||||
@@ -1021,6 +1037,9 @@ export function ReimbursementDetailPage() {
|
||||
meta={(
|
||||
<>
|
||||
<StatusBadge status={detail.status} />
|
||||
{!isReimbursementParticipant ? (
|
||||
<span className="rounded-full bg-zinc-100 px-2.5 py-1 text-xs font-medium text-zinc-600">只读查看</span>
|
||||
) : null}
|
||||
</>
|
||||
)}
|
||||
/>
|
||||
@@ -1078,7 +1097,7 @@ export function ReimbursementDetailPage() {
|
||||
editable={Boolean(canEditItems && isReimbursementParticipant)}
|
||||
embedded
|
||||
/>
|
||||
{canEditItems || detail.project || detail.linkedTravelRequest ? (
|
||||
{canMaintainReimbursement || detail.project || detail.linkedTravelRequest ? (
|
||||
<div className="mt-4 border-t border-zinc-200 pt-4 text-sm" data-testid="reimbursement-related-records">
|
||||
<div className="flex flex-wrap items-center justify-between gap-3">
|
||||
<span className="font-medium text-zinc-900">关联事项</span>
|
||||
@@ -1088,7 +1107,7 @@ export function ReimbursementDetailPage() {
|
||||
</Link>
|
||||
) : null}
|
||||
</div>
|
||||
{canEditItems ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<select
|
||||
className="select-std mt-3 max-w-xl"
|
||||
value={detail.projectId ?? ""}
|
||||
@@ -1155,7 +1174,7 @@ export function ReimbursementDetailPage() {
|
||||
<SectionPanel title="报销明细">
|
||||
<div className="space-y-4">
|
||||
<div className="space-y-4" data-testid="reimbursement-items">
|
||||
{canEditItems ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<div className="flex justify-end">
|
||||
<Button
|
||||
data-testid="reimbursement-item-add"
|
||||
@@ -1178,7 +1197,7 @@ export function ReimbursementDetailPage() {
|
||||
</p>
|
||||
) : null}
|
||||
|
||||
{reimbursementPrecheckQuery.isLoading && canEditItems ? (
|
||||
{reimbursementPrecheckQuery.isLoading && canMaintainReimbursement ? (
|
||||
<p className="text-sm text-zinc-500">正在刷新费用标准校验结果...</p>
|
||||
) : null}
|
||||
|
||||
@@ -1241,7 +1260,7 @@ export function ReimbursementDetailPage() {
|
||||
))}
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
{canEditItems ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<div className="flex min-w-36 items-center gap-2">
|
||||
<Button
|
||||
data-testid="reimbursement-item-edit"
|
||||
@@ -1358,7 +1377,9 @@ export function ReimbursementDetailPage() {
|
||||
|
||||
<div className="space-y-2" data-testid={TEST_IDS.reimbursement.invoices}>
|
||||
{detail.invoices.map((invoice) => {
|
||||
const invoiceHint = resolveInvoiceNextStepText(invoice);
|
||||
const invoiceHint = invoice.status === "bound" && !canMaintainReimbursement
|
||||
? "已关联"
|
||||
: resolveInvoiceNextStepText(invoice);
|
||||
return (
|
||||
<div className="detail-evidence-card flex flex-col gap-3 md:flex-row md:items-center" key={invoice.id}>
|
||||
<button
|
||||
@@ -1397,7 +1418,7 @@ export function ReimbursementDetailPage() {
|
||||
</div>
|
||||
|
||||
<div className="space-y-4" data-testid="reimbursement-attachments">
|
||||
{canEditItems ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<div className="flex flex-col gap-3 md:flex-row md:items-center md:justify-between">
|
||||
<p className="text-xs text-zinc-500">支持 PDF/JPG/PNG/WebP,单次最多 5 个。</p>
|
||||
<div className="flex flex-col gap-2 md:flex-row md:items-center">
|
||||
@@ -1461,7 +1482,7 @@ export function ReimbursementDetailPage() {
|
||||
</span>
|
||||
<ChevronRight aria-hidden="true" className="size-4 shrink-0 text-zinc-400" />
|
||||
</button>
|
||||
{canEditItems ? (
|
||||
{canMaintainReimbursement ? (
|
||||
<Button
|
||||
type="button"
|
||||
variant="destructive"
|
||||
@@ -1514,6 +1535,7 @@ export function ReimbursementDetailPage() {
|
||||
<ReimbursementActions
|
||||
status={detail.status}
|
||||
showSubmitAction={false}
|
||||
showWithdrawAction={isReimbursementParticipant}
|
||||
actionError={["reviewing", "approved"].includes(detail.status) ? actionError : ""}
|
||||
submitDisabled={blockStandardChecks.length > 0}
|
||||
submitHint=""
|
||||
|
||||
@@ -10,7 +10,7 @@ export interface ReimbursementActionsProps {
|
||||
submitHint?: string;
|
||||
submitNotice?: string;
|
||||
showSubmitAction?: boolean;
|
||||
showWithdrawAction?: boolean;
|
||||
showWithdrawAction: boolean;
|
||||
isPending: boolean;
|
||||
onSubmit: () => void;
|
||||
onWithdraw: (comment: string) => void;
|
||||
@@ -23,7 +23,7 @@ export function ReimbursementActions({
|
||||
submitHint = "",
|
||||
submitNotice = "",
|
||||
showSubmitAction = true,
|
||||
showWithdrawAction = true,
|
||||
showWithdrawAction,
|
||||
isPending,
|
||||
onSubmit,
|
||||
onWithdraw,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { expect, test, type Page, type Route } from "@playwright/test";
|
||||
|
||||
test.use({ serviceWorkers: "block" });
|
||||
|
||||
function envelope(data: unknown) {
|
||||
return { requestId: "mock-r102-reimbursement", data };
|
||||
}
|
||||
@@ -450,7 +452,7 @@ test.describe("R102 reimbursement main flow", () => {
|
||||
await expect(page.getByTestId("mobile-reimbursement-items")).toContainText("报销合计 ¥168.30");
|
||||
});
|
||||
|
||||
test("移动管理员默认查看本人并可切换全部报销", async ({ page }) => {
|
||||
test("管理员查看他人报销时保持只读并可在移动端切换全部报销", async ({ page }) => {
|
||||
const adminMe = {
|
||||
...me,
|
||||
id: "admin-r102",
|
||||
@@ -473,11 +475,25 @@ test.describe("R102 reimbursement main flow", () => {
|
||||
...reimbursementDetail(),
|
||||
id: otherReimbursementId,
|
||||
formNumber: "BX-20260731-0002",
|
||||
status: "reviewing",
|
||||
status: "draft",
|
||||
ownerId: "employee-other-r102",
|
||||
submitterId: "employee-other-r102",
|
||||
owner: { id: "employee-other-r102", employee_no: "E103", name: "其他员工" },
|
||||
submitter: { id: "employee-other-r102", employee_no: "E103", name: "其他员工" },
|
||||
invoices: [{
|
||||
id: "invoice-other-r102",
|
||||
status: "bound",
|
||||
invoiceCode: "SYNTHETIC",
|
||||
invoiceNumber: "R102-OTHER",
|
||||
invoiceDate: "2026-07-31",
|
||||
sellerName: "合成供应方",
|
||||
buyerName: "合成购买方",
|
||||
totalAmount: "168.30",
|
||||
originalName: "合成发票.pdf",
|
||||
mimeType: "application/pdf",
|
||||
fileSize: 1024,
|
||||
linkedItemId: null,
|
||||
}],
|
||||
});
|
||||
});
|
||||
await page.route((url) => url.pathname === "/api/reimbursements", async (route) => {
|
||||
@@ -508,6 +524,19 @@ test.describe("R102 reimbursement main flow", () => {
|
||||
|
||||
await page.reload();
|
||||
await expect(page).toHaveURL(/\/portal/);
|
||||
await page.setViewportSize({ width: 1440, height: 960 });
|
||||
await page.goto(`/reimbursements/${otherReimbursementId}`);
|
||||
await expect(page.getByTestId("reimbursement-overview")).toBeVisible();
|
||||
await expect(page.getByTestId("reimbursement-withdraw")).toHaveCount(0);
|
||||
await expect(page.getByTestId("reimbursement-withdraw-comment")).toHaveCount(0);
|
||||
await expect(page.getByText("只读查看")).toBeVisible();
|
||||
await expect(page.getByText("等待报销人完善")).toBeVisible();
|
||||
await expect(page.getByTestId("reimbursement-item-add")).toHaveCount(0);
|
||||
await expect(page.getByRole("combobox", { name: "关联项目" })).toHaveCount(0);
|
||||
await expect(page.getByTestId("reimbursement-invoices")).toContainText("明细:未关联");
|
||||
await expect(page.getByTestId("reimbursement-invoices")).toContainText("已关联");
|
||||
await expect(page.getByText("草稿或驳回状态可移除")).toHaveCount(0);
|
||||
|
||||
await page.setViewportSize({ width: 375, height: 812 });
|
||||
await page.goto("/mobile/reimbursements?status=reviewing");
|
||||
|
||||
@@ -535,6 +564,11 @@ test.describe("R102 reimbursement main flow", () => {
|
||||
|
||||
await page.getByText("其他员工").click();
|
||||
await expect(page).toHaveURL(new RegExp(`/mobile/reimbursements/${otherReimbursementId}$`));
|
||||
await expect(page.getByText("只读查看")).toBeVisible();
|
||||
await expect(page.getByTestId("mobile-reimbursement-items").getByRole("button", { name: "新增" })).toHaveCount(0);
|
||||
await expect(page.getByTestId("reimbursement-invoices")).toContainText("明细:未关联");
|
||||
await expect(page.getByTestId("reimbursement-invoices")).toContainText("已关联");
|
||||
await expect(page.getByRole("button", { name: "移除" })).toHaveCount(0);
|
||||
const routeState = await page.evaluate(() => window.history.state?.usr ?? null);
|
||||
expect(routeState?.from).toContain("status=reviewing");
|
||||
expect(routeState?.from).toContain("scope=all");
|
||||
|
||||
@@ -0,0 +1,558 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only, redacted inspection of an interrupted automatic-backup chain."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
|
||||
BATCH_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))$")
|
||||
PARTIAL_RE = re.compile(r"^hyb-oa-backup-(auto-(\d{8}T\d{6}Z))\.partial$")
|
||||
TEMP_RE = re.compile(r"^\.hyb-oa-backup-.+\.tmp\..+$")
|
||||
EXPECTED_FILES = {"database.dump", "storage.tar.gz", "manifest.json"}
|
||||
ARTIFACTS = {"database": "database.dump", "storage": "storage.tar.gz"}
|
||||
RSYNC_CODE_RE = re.compile(r"rsync error:[^\r\n]*\(code (11|12|23|30|255)\)")
|
||||
RSYNC_CATEGORIES = {
|
||||
"11": "file_io_error",
|
||||
"12": "protocol_stream_error",
|
||||
"23": "partial_transfer_error",
|
||||
"30": "transport_timeout",
|
||||
"255": "ssh_session_error",
|
||||
}
|
||||
SIMILAR_RELATIVE_PPM = 50_000
|
||||
SIMILAR_ABSOLUTE_BYTES = 64 * 1024 * 1024
|
||||
SAFE_KEY_RE = re.compile(r"^[a-z0-9_]+$")
|
||||
SAFE_VALUE_RE = re.compile(r"^(?:[A-Za-z][A-Za-z0-9_]*|-?[0-9]+)$")
|
||||
SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||
|
||||
|
||||
class InspectionError(Exception):
|
||||
"""Stable, non-sensitive inspection failure."""
|
||||
|
||||
def __init__(self, category: str):
|
||||
super().__init__(category)
|
||||
self.category = category
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BatchResult:
|
||||
valid: bool
|
||||
total_bytes: int
|
||||
created_epoch: int | None
|
||||
expected_artifact_bytes: int | None
|
||||
received_artifact_bytes: int
|
||||
|
||||
|
||||
def emit(items: Iterable[tuple[str, object]]) -> None:
|
||||
rendered_items: list[tuple[str, str]] = []
|
||||
for key, value in items:
|
||||
if isinstance(value, bool):
|
||||
rendered = "yes" if value else "no"
|
||||
else:
|
||||
rendered = str(value)
|
||||
if not SAFE_KEY_RE.fullmatch(key) or not SAFE_VALUE_RE.fullmatch(rendered):
|
||||
print("probe_result=FAIL")
|
||||
print("probe_category=output_allowlist_violation")
|
||||
raise SystemExit(1)
|
||||
rendered_items.append((key, rendered))
|
||||
for key, rendered in rendered_items:
|
||||
print(f"{key}={rendered}")
|
||||
|
||||
|
||||
def safe_root(path: Path) -> Path:
|
||||
try:
|
||||
if not path.is_absolute() or path.is_symlink() or not path.is_dir():
|
||||
raise InspectionError("root_invalid")
|
||||
list(path.iterdir())
|
||||
except PermissionError as error:
|
||||
raise InspectionError("access_denied") from error
|
||||
except OSError as error:
|
||||
raise InspectionError("root_unreadable") from error
|
||||
return path
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def tree_bytes(path: Path) -> int:
|
||||
total = 0
|
||||
try:
|
||||
for root, directories, files in os.walk(path, followlinks=False):
|
||||
directories[:] = [
|
||||
name for name in directories if not (Path(root) / name).is_symlink()
|
||||
]
|
||||
for name in files:
|
||||
item = Path(root) / name
|
||||
if item.is_symlink():
|
||||
continue
|
||||
total += item.stat().st_size
|
||||
except PermissionError as error:
|
||||
raise InspectionError("access_denied") from error
|
||||
except OSError as error:
|
||||
raise InspectionError("entry_unreadable") from error
|
||||
return total
|
||||
|
||||
|
||||
def run_epoch(stamp: str) -> int | None:
|
||||
try:
|
||||
parsed = datetime.strptime(stamp, "%Y%m%dT%H%M%SZ").replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
return None
|
||||
return int(parsed.timestamp())
|
||||
|
||||
|
||||
def validate_batch(
|
||||
path: Path,
|
||||
expected_run_id: str,
|
||||
stamp: str,
|
||||
*,
|
||||
verify_hashes: bool = True,
|
||||
) -> BatchResult:
|
||||
total = tree_bytes(path)
|
||||
received_artifact_bytes = 0
|
||||
expected_artifact_bytes: int | None = None
|
||||
created_epoch = run_epoch(stamp)
|
||||
if created_epoch is None:
|
||||
return BatchResult(False, total, None, None, 0)
|
||||
try:
|
||||
entries = list(path.iterdir())
|
||||
regular_names = {
|
||||
item.name
|
||||
for item in entries
|
||||
if not item.is_symlink() and stat.S_ISREG(item.stat().st_mode)
|
||||
}
|
||||
if len(entries) != 3 or regular_names != EXPECTED_FILES:
|
||||
return BatchResult(False, total, created_epoch, None, 0)
|
||||
manifest = json.loads((path / "manifest.json").read_text(encoding="utf-8"))
|
||||
if (
|
||||
manifest.get("schemaVersion") != 1
|
||||
or manifest.get("status") != "complete"
|
||||
or manifest.get("runId") != expected_run_id
|
||||
or not (manifest.get("buildSha") or manifest.get("sourceImageDigest"))
|
||||
):
|
||||
return BatchResult(False, total, created_epoch, None, 0)
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
expected_artifact_bytes = 0
|
||||
for key, expected_name in ARTIFACTS.items():
|
||||
item = artifacts.get(key) or {}
|
||||
target = path / expected_name
|
||||
if item.get("file") != expected_name or isinstance(item.get("bytes"), bool):
|
||||
return BatchResult(False, total, created_epoch, None, 0)
|
||||
expected_size = item.get("bytes")
|
||||
if not isinstance(expected_size, int) or expected_size < 0:
|
||||
return BatchResult(False, total, created_epoch, None, 0)
|
||||
expected_sha = item.get("sha256")
|
||||
if not isinstance(expected_sha, str) or not SHA256_RE.fullmatch(expected_sha):
|
||||
return BatchResult(False, total, created_epoch, None, 0)
|
||||
expected_artifact_bytes += expected_size
|
||||
actual_size = target.stat().st_size
|
||||
received_artifact_bytes += actual_size
|
||||
if actual_size != expected_size or (
|
||||
verify_hashes and file_sha256(target) != expected_sha
|
||||
):
|
||||
return BatchResult(
|
||||
False,
|
||||
total,
|
||||
created_epoch,
|
||||
expected_artifact_bytes,
|
||||
received_artifact_bytes,
|
||||
)
|
||||
return BatchResult(
|
||||
True,
|
||||
total,
|
||||
created_epoch,
|
||||
expected_artifact_bytes,
|
||||
received_artifact_bytes,
|
||||
)
|
||||
except PermissionError as error:
|
||||
raise InspectionError("access_denied") from error
|
||||
except (AttributeError, json.JSONDecodeError, OSError, TypeError, ValueError):
|
||||
return BatchResult(False, total, created_epoch, expected_artifact_bytes, received_artifact_bytes)
|
||||
|
||||
|
||||
def inspect_partial(path: Path, run_id: str, stamp: str) -> BatchResult:
|
||||
"""Measure a partial without requiring the complete three-file contract."""
|
||||
total = tree_bytes(path)
|
||||
received = 0
|
||||
expected: int | None = None
|
||||
try:
|
||||
manifest_path = path / "manifest.json"
|
||||
if manifest_path.is_file() and not manifest_path.is_symlink():
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if manifest.get("runId") == run_id and manifest.get("schemaVersion") == 1:
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
sizes: list[int] = []
|
||||
for key, expected_name in ARTIFACTS.items():
|
||||
item = artifacts.get(key) or {}
|
||||
size = item.get("bytes")
|
||||
if (
|
||||
item.get("file") != expected_name
|
||||
or isinstance(size, bool)
|
||||
or not isinstance(size, int)
|
||||
or size < 0
|
||||
):
|
||||
sizes = []
|
||||
break
|
||||
sizes.append(size)
|
||||
target = path / expected_name
|
||||
if target.is_file() and not target.is_symlink():
|
||||
received += target.stat().st_size
|
||||
if len(sizes) == 2:
|
||||
expected = sum(sizes)
|
||||
validated = validate_batch(path, run_id, stamp)
|
||||
return BatchResult(validated.valid, total, run_epoch(stamp), expected, received)
|
||||
except PermissionError as error:
|
||||
raise InspectionError("access_denied") from error
|
||||
except (AttributeError, json.JSONDecodeError, OSError, TypeError, ValueError):
|
||||
return BatchResult(False, total, run_epoch(stamp), expected, received)
|
||||
|
||||
|
||||
def filesystem_fields(prefix: str, root: Path) -> list[tuple[str, object]]:
|
||||
values = os.statvfs(root)
|
||||
total = values.f_blocks * values.f_frsize
|
||||
available = values.f_bavail * values.f_frsize
|
||||
used_percent = 0 if total == 0 else ((total - available) * 100) // total
|
||||
return [
|
||||
(f"{prefix}_disk_used_percent", used_percent),
|
||||
(f"{prefix}_disk_free_gib", available // (1024**3)),
|
||||
]
|
||||
|
||||
|
||||
def source_contract(wrapper: Path, backup_script: Path) -> list[tuple[str, object]]:
|
||||
try:
|
||||
if (
|
||||
wrapper.is_symlink()
|
||||
or not wrapper.is_file()
|
||||
or backup_script.is_symlink()
|
||||
or not backup_script.is_file()
|
||||
):
|
||||
raise InspectionError("source_contract_invalid")
|
||||
wrapper_text = wrapper.read_text(encoding="utf-8")
|
||||
backup_text = backup_script.read_text(encoding="utf-8")
|
||||
except PermissionError as error:
|
||||
raise InspectionError("source_contract_access_denied") from error
|
||||
except OSError as error:
|
||||
raise InspectionError("source_contract_unreadable") from error
|
||||
|
||||
if "run_backup() {" not in wrapper_text or "\ncheck_stale() {" not in wrapper_text:
|
||||
raise InspectionError("source_contract_invalid")
|
||||
run_backup_body = wrapper_text.split("run_backup() {", 1)[1].split("\ncheck_stale() {", 1)[0]
|
||||
markers = {
|
||||
"local_publish": '[[ -d "$batch_dir" && -f "$batch_dir/manifest.json" ]]',
|
||||
"transfer": "capture_command offhost_transfer rsync",
|
||||
"finalize": "capture_command offhost_finalize ssh",
|
||||
"success": 'write_success_state "$run_id" "$manifest_sha"',
|
||||
"prune": 'prune_local_backups "$batch_name"',
|
||||
}
|
||||
positions = {key: run_backup_body.find(marker) for key, marker in markers.items()}
|
||||
all_present = all(position >= 0 for position in positions.values())
|
||||
retained = all_present and positions["local_publish"] < positions["transfer"] < positions["prune"]
|
||||
pruned = all_present and positions["finalize"] < positions["success"] < positions["prune"]
|
||||
temp_cleanup = all(
|
||||
marker in backup_text
|
||||
for marker in (
|
||||
"trap cleanup EXIT",
|
||||
'TEMP_DIR="$(mktemp -d',
|
||||
'rm -rf -- "$TEMP_DIR"',
|
||||
'mv "$TEMP_DIR" "$FINAL_DIR"',
|
||||
'TEMP_DIR=""',
|
||||
"trap - EXIT",
|
||||
)
|
||||
)
|
||||
return [
|
||||
("retained_after_transfer_failure", retained),
|
||||
("pruned_after_offhost_success", pruned),
|
||||
("generation_temp_cleanup_on_failure", temp_cleanup),
|
||||
("cleanup_contract_source_verified", all_present and temp_cleanup),
|
||||
]
|
||||
|
||||
|
||||
def last_backup_phase(state_dir: Path) -> str:
|
||||
state_file = state_dir / "last-run.json"
|
||||
if not state_file.exists():
|
||||
return "NOT_AVAILABLE"
|
||||
try:
|
||||
if state_file.is_symlink() or not state_file.is_file():
|
||||
return "STATE_INVALID"
|
||||
payload = json.loads(state_file.read_text(encoding="utf-8"))
|
||||
findings = payload.get("findings") or []
|
||||
detail = findings[0].get("detail") if len(findings) == 1 and isinstance(findings[0], dict) else None
|
||||
match = re.fullmatch(r"phase_([a-z_]+)_failed", detail or "")
|
||||
allowed = {
|
||||
"runtime_identity",
|
||||
"paired_backup",
|
||||
"offhost_transfer",
|
||||
"offhost_finalize",
|
||||
"success_state",
|
||||
"stale_check",
|
||||
"quarterly_reminder",
|
||||
}
|
||||
return match.group(1) if match and match.group(1) in allowed else "NOT_AVAILABLE"
|
||||
except PermissionError as error:
|
||||
raise InspectionError("diagnostic_access_denied") from error
|
||||
except (AttributeError, json.JSONDecodeError, OSError, TypeError):
|
||||
return "STATE_INVALID"
|
||||
|
||||
|
||||
def parse_transfer_diagnostic(state_dir: Path) -> list[tuple[str, object]]:
|
||||
phase = last_backup_phase(state_dir)
|
||||
diagnostic = state_dir / "last-error.log"
|
||||
if phase != "offhost_transfer" or not diagnostic.exists():
|
||||
code = "NOT_AVAILABLE"
|
||||
else:
|
||||
try:
|
||||
if diagnostic.is_symlink() or not diagnostic.is_file():
|
||||
raise InspectionError("diagnostic_invalid")
|
||||
text = diagnostic.read_text(encoding="utf-8", errors="replace")
|
||||
except PermissionError as error:
|
||||
raise InspectionError("diagnostic_access_denied") from error
|
||||
except OSError as error:
|
||||
raise InspectionError("diagnostic_unreadable") from error
|
||||
matches = RSYNC_CODE_RE.findall(text)
|
||||
code = matches[-1] if matches else "NOT_AVAILABLE"
|
||||
return [
|
||||
("last_backup_failure_phase", phase),
|
||||
("transfer_exit_code", code),
|
||||
("transfer_exit_code_source", "last_error_log_regex" if code != "NOT_AVAILABLE" else "NOT_AVAILABLE"),
|
||||
("transfer_failure_category", RSYNC_CATEGORIES.get(code, "not_available")),
|
||||
("transfer_started_epoch", "NOT_AVAILABLE"),
|
||||
("transfer_failed_epoch", "NOT_AVAILABLE"),
|
||||
("transfer_duration_seconds", "NOT_AVAILABLE"),
|
||||
("transfer_duration_reason", "wrapper_does_not_persist_transfer_clock"),
|
||||
]
|
||||
|
||||
|
||||
def production(args: argparse.Namespace) -> None:
|
||||
backup_root = safe_root(args.backup_root)
|
||||
state_dir = safe_root(args.state_dir)
|
||||
now = args.now_epoch
|
||||
metadata_valid: list[tuple[Path, str, str, BatchResult]] = []
|
||||
invalid_count = invalid_bytes = temp_count = temp_bytes = other_count = 0
|
||||
for entry in backup_root.iterdir():
|
||||
match = BATCH_RE.fullmatch(entry.name)
|
||||
if match and entry.is_dir() and not entry.is_symlink():
|
||||
result = validate_batch(
|
||||
entry,
|
||||
match.group(1),
|
||||
match.group(2),
|
||||
verify_hashes=False,
|
||||
)
|
||||
if result.valid and result.created_epoch is not None and result.created_epoch <= now:
|
||||
metadata_valid.append((entry, match.group(1), match.group(2), result))
|
||||
else:
|
||||
invalid_count += 1
|
||||
invalid_bytes += result.total_bytes
|
||||
elif TEMP_RE.fullmatch(entry.name) and entry.is_dir() and not entry.is_symlink():
|
||||
temp_count += 1
|
||||
temp_bytes += tree_bytes(entry)
|
||||
else:
|
||||
other_count += 1
|
||||
|
||||
metadata_valid.sort(key=lambda item: item[3].created_epoch or -1)
|
||||
latest_metadata = metadata_valid[-1] if metadata_valid else None
|
||||
latest_verified: BatchResult | None = None
|
||||
if latest_metadata is not None:
|
||||
latest_verified = validate_batch(
|
||||
latest_metadata[0],
|
||||
latest_metadata[1],
|
||||
latest_metadata[2],
|
||||
verify_hashes=True,
|
||||
)
|
||||
complete_local_recovery_point = bool(latest_verified and latest_verified.valid)
|
||||
latest_result = latest_metadata[3] if latest_metadata else None
|
||||
output: list[tuple[str, object]] = [
|
||||
("production_probe_result", "PASS"),
|
||||
("complete_local_recovery_point", complete_local_recovery_point),
|
||||
(
|
||||
"backup_risk_classification",
|
||||
"production_single_point"
|
||||
if complete_local_recovery_point
|
||||
else "no_proven_recovery_point",
|
||||
),
|
||||
("fully_verified_local_batch_count", 1 if complete_local_recovery_point else 0),
|
||||
("metadata_complete_local_batch_count", len(metadata_valid)),
|
||||
(
|
||||
"metadata_complete_local_batch_total_bytes",
|
||||
sum(item[3].total_bytes for item in metadata_valid),
|
||||
),
|
||||
(
|
||||
"older_metadata_batch_hash_status",
|
||||
"NOT_RUN" if len(metadata_valid) > 1 else "NOT_APPLICABLE",
|
||||
),
|
||||
(
|
||||
"latest_local_candidate_hash_status",
|
||||
"PASS"
|
||||
if complete_local_recovery_point
|
||||
else ("FAIL" if latest_metadata is not None else "NOT_RUN"),
|
||||
),
|
||||
(
|
||||
"latest_local_candidate_age_seconds",
|
||||
now - latest_result.created_epoch
|
||||
if latest_result and latest_result.created_epoch is not None
|
||||
else "NOT_AVAILABLE",
|
||||
),
|
||||
(
|
||||
"latest_local_candidate_bytes",
|
||||
latest_result.total_bytes if latest_result else "NOT_AVAILABLE",
|
||||
),
|
||||
("invalid_local_batch_count", invalid_count),
|
||||
("invalid_local_batch_total_bytes", invalid_bytes),
|
||||
("generation_temp_count", temp_count),
|
||||
("generation_temp_total_bytes", temp_bytes),
|
||||
("unclassified_local_entry_count", other_count),
|
||||
(
|
||||
"transition_recovery_point_action",
|
||||
"wait_natural" if complete_local_recovery_point else "request_manual_run",
|
||||
),
|
||||
]
|
||||
output.extend(filesystem_fields("production_backup", backup_root))
|
||||
output.extend(source_contract(args.wrapper_script, args.backup_script))
|
||||
output.extend(parse_transfer_diagnostic(state_dir))
|
||||
emit(output)
|
||||
|
||||
|
||||
def stopping_class(samples: list[BatchResult]) -> str:
|
||||
if len(samples) < 2:
|
||||
return "insufficient_samples"
|
||||
sizes = sorted(item.total_bytes for item in samples[-2:])
|
||||
difference = sizes[1] - sizes[0]
|
||||
relative_limit = sizes[1] * SIMILAR_RELATIVE_PPM // 1_000_000
|
||||
similar_limit = min(SIMILAR_ABSOLUTE_BYTES, relative_limit)
|
||||
return "similar" if difference <= similar_limit else "different"
|
||||
|
||||
|
||||
def arm(args: argparse.Namespace) -> None:
|
||||
inbox_root = safe_root(args.inbox_root)
|
||||
archive_root = safe_root(args.archive_root)
|
||||
now = args.now_epoch
|
||||
partials: list[BatchResult] = []
|
||||
other_count = 0
|
||||
for entry in inbox_root.iterdir():
|
||||
match = PARTIAL_RE.fullmatch(entry.name)
|
||||
if match and entry.is_dir() and not entry.is_symlink():
|
||||
partials.append(inspect_partial(entry, match.group(1), match.group(2)))
|
||||
else:
|
||||
other_count += 1
|
||||
partials.sort(key=lambda item: item.created_epoch or -1)
|
||||
|
||||
archives: list[BatchResult] = []
|
||||
invalid_archive_count = 0
|
||||
for entry in archive_root.iterdir():
|
||||
match = BATCH_RE.fullmatch(entry.name)
|
||||
if match and entry.is_dir() and not entry.is_symlink():
|
||||
result = validate_batch(entry, match.group(1), match.group(2))
|
||||
if result.valid and result.created_epoch is not None and result.created_epoch <= now:
|
||||
archives.append(result)
|
||||
else:
|
||||
invalid_archive_count += 1
|
||||
archives.sort(key=lambda item: item.created_epoch or -1)
|
||||
|
||||
output: list[tuple[str, object]] = [
|
||||
("arm_probe_result", "PASS"),
|
||||
("arm_partial_count", len(partials)),
|
||||
("arm_partial_total_bytes", sum(item.total_bytes for item in partials)),
|
||||
("arm_partial_complete_contract_count", sum(1 for item in partials if item.valid)),
|
||||
("arm_partial_incomplete_contract_count", sum(1 for item in partials if not item.valid)),
|
||||
("arm_partial_stop_offset_class", stopping_class(partials)),
|
||||
("arm_partial_similarity_relative_ppm", SIMILAR_RELATIVE_PPM),
|
||||
("arm_partial_similarity_absolute_bytes", SIMILAR_ABSOLUTE_BYTES),
|
||||
("arm_inbox_other_entry_count", other_count),
|
||||
("arm_valid_atomic_archive_count", len(archives)),
|
||||
("arm_invalid_atomic_archive_count", invalid_archive_count),
|
||||
(
|
||||
"arm_latest_atomic_archive_age_seconds",
|
||||
now - archives[-1].created_epoch if archives else "NOT_AVAILABLE",
|
||||
),
|
||||
]
|
||||
for index, sample in enumerate(partials, 1):
|
||||
ratio = (
|
||||
min(sample.received_artifact_bytes, sample.expected_artifact_bytes)
|
||||
* 1_000_000
|
||||
// sample.expected_artifact_bytes
|
||||
if sample.expected_artifact_bytes and sample.expected_artifact_bytes > 0
|
||||
else "NOT_AVAILABLE"
|
||||
)
|
||||
output.extend(
|
||||
[
|
||||
(f"arm_partial_sample_{index}_total_bytes", sample.total_bytes),
|
||||
(
|
||||
f"arm_partial_sample_{index}_expected_artifact_bytes",
|
||||
sample.expected_artifact_bytes
|
||||
if sample.expected_artifact_bytes is not None
|
||||
else "NOT_AVAILABLE",
|
||||
),
|
||||
(f"arm_partial_sample_{index}_received_ratio_ppm", ratio),
|
||||
(
|
||||
f"arm_partial_sample_{index}_age_seconds",
|
||||
now - sample.created_epoch
|
||||
if sample.created_epoch is not None
|
||||
else "NOT_AVAILABLE",
|
||||
),
|
||||
(f"arm_partial_sample_{index}_complete_contract", sample.valid),
|
||||
]
|
||||
)
|
||||
output.extend(filesystem_fields("arm_inbox", inbox_root))
|
||||
output.extend(filesystem_fields("arm_archive", archive_root))
|
||||
emit(output)
|
||||
|
||||
|
||||
def absolute_path(value: str) -> Path:
|
||||
path = Path(value)
|
||||
if not path.is_absolute():
|
||||
raise argparse.ArgumentTypeError("an explicit absolute path is required")
|
||||
return path
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Emit redacted backup-chain incident fields")
|
||||
parser.add_argument("--now-epoch", type=int, default=int(datetime.now(timezone.utc).timestamp()))
|
||||
subparsers = parser.add_subparsers(dest="mode", required=True)
|
||||
production_parser = subparsers.add_parser("production")
|
||||
production_parser.add_argument("--backup-root", type=absolute_path, required=True)
|
||||
production_parser.add_argument("--state-dir", type=absolute_path, required=True)
|
||||
production_parser.add_argument("--wrapper-script", type=absolute_path, required=True)
|
||||
production_parser.add_argument("--backup-script", type=absolute_path, required=True)
|
||||
production_parser.set_defaults(handler=production)
|
||||
arm_parser = subparsers.add_parser("arm")
|
||||
arm_parser.add_argument("--inbox-root", type=absolute_path, required=True)
|
||||
arm_parser.add_argument("--archive-root", type=absolute_path, required=True)
|
||||
arm_parser.set_defaults(handler=arm)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
try:
|
||||
args.handler(args)
|
||||
except InspectionError as error:
|
||||
emit([("probe_result", "FAIL"), ("probe_category", error.category)])
|
||||
return 1
|
||||
except PermissionError:
|
||||
emit([("probe_result", "FAIL"), ("probe_category", "access_denied")])
|
||||
return 1
|
||||
except (OSError, TypeError, ValueError):
|
||||
emit([("probe_result", "FAIL"), ("probe_category", "inspection_io_error")])
|
||||
return 1
|
||||
except Exception:
|
||||
emit([("probe_result", "FAIL"), ("probe_category", "inspection_internal_error")])
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,766 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
umask 077
|
||||
|
||||
CONFIG_FILE="@@RECEIVER_CONFIG_FILE@@"
|
||||
LIB_DIR="@@RESTORE_LIB_DIR@@"
|
||||
SELF_PATH="@@SELF_PATH@@"
|
||||
SUDOERS_PATH="@@SUDOERS_PATH@@"
|
||||
SUDOERS_MAIN_FILE="@@SUDOERS_MAIN_FILE@@"
|
||||
WORK_ROOT="@@WORK_ROOT@@"
|
||||
WORK_PREFIX="@@WORK_PREFIX@@"
|
||||
CONTAINER_PREFIX="@@CONTAINER_PREFIX@@"
|
||||
SHADOW_DB_USER="@@SHADOW_DB_USER@@"
|
||||
SHADOW_DB_NAME="@@SHADOW_DB_NAME@@"
|
||||
POSTGRES_IMAGE="@@POSTGRES_IMAGE@@"
|
||||
DEADMAN_TIMER_UNIT="@@DEADMAN_TIMER_UNIT@@"
|
||||
HEALTH_TIMER_UNIT="@@HEALTH_TIMER_UNIT@@"
|
||||
ARCHIVE_BATCH_PREFIX="@@ARCHIVE_BATCH_PREFIX@@"
|
||||
RUN_ID_PREFIX="@@RUN_ID_PREFIX@@"
|
||||
EXPECTED_DB_RESTORE_SHA="@@EXPECTED_DB_RESTORE_SHA@@"
|
||||
EXPECTED_STORAGE_RESTORE_SHA="@@EXPECTED_STORAGE_RESTORE_SHA@@"
|
||||
EXPECTED_ATTACHMENT_VALIDATOR_SHA="@@EXPECTED_ATTACHMENT_VALIDATOR_SHA@@"
|
||||
EXPECTED_BUILD_PREFIX="@@EXPECTED_BUILD_PREFIX@@"
|
||||
EXPECTED_MIGRATION_SET_SHA="@@EXPECTED_MIGRATION_SET_SHA@@"
|
||||
PHASE="bootstrap"
|
||||
WORK_DIR=""
|
||||
CONTAINER_NAME=""
|
||||
RUNTIME_CLEANED=0
|
||||
RESULT_STATUS="NO_GO"
|
||||
RPO_SECONDS=""
|
||||
RTO_SECONDS="NOT_RUN"
|
||||
RTO_BATCH_COPY_SECONDS="NOT_RUN"
|
||||
RTO_POSTGRES_CONTAINER_START_SECONDS="NOT_RUN"
|
||||
RTO_DUMP_LIST_SECONDS="NOT_RUN"
|
||||
RTO_TARGET_CONFIRMATION_SECONDS="NOT_RUN"
|
||||
RTO_RESTORE_APPLY_SECONDS="NOT_RUN"
|
||||
RTO_DATABASE_VERIFY_SECONDS="NOT_RUN"
|
||||
RTO_STORAGE_ATTACHMENT_VERIFY_SECONDS="NOT_RUN"
|
||||
RTO_FINAL_INTEGRITY_SECONDS="NOT_RUN"
|
||||
RTO_ACTIVE_STAGE=""
|
||||
RTO_STAGE_STARTED_EPOCH=""
|
||||
RTO_FINALIZED=0
|
||||
MIGRATION_SUCCESS=""
|
||||
MIGRATION_OTHER=""
|
||||
MIGRATION_SET_MATCH=""
|
||||
ARCHIVE_FILE_COUNT=""
|
||||
RESTORED_FILE_COUNT=""
|
||||
ATTACHMENT_TOTAL=""
|
||||
ATTACHMENT_MISMATCH=""
|
||||
TABLE_COUNTS=""
|
||||
SOURCE_RUN_ID=""
|
||||
DUMP_LIST_STATUS="NOT_RUN"
|
||||
TARGET_CONFIRMATION_STATUS="NOT_RUN"
|
||||
RESTORE_APPLY_STATUS="NOT_RUN"
|
||||
RESTORE_EXECUTION_PATH="NOT_RUN"
|
||||
DUMP_LIST_EXIT_CODE="NOT_RUN"
|
||||
TARGET_CONFIRMATION_EXIT_CODE="NOT_RUN"
|
||||
TARGET_NAME_MATCH="NOT_RUN"
|
||||
TARGET_CONFIRMATION_ATTEMPT_COUNT="0"
|
||||
TARGET_CONFIRMATION_SUCCESS_STREAK="0"
|
||||
TARGET_CONFIRMATION_FAILURE_REASON="NOT_APPLICABLE"
|
||||
RESTORE_SCRIPT_EXIT_CODE="NOT_RUN"
|
||||
PG_RESTORE_EXIT_CODE="NOT_RUN"
|
||||
PG_RESTORE_ERROR_COUNT="NOT_RUN"
|
||||
RESTORE_ERROR_CATEGORY="NOT_APPLICABLE"
|
||||
FAILURE_CATEGORY="none"
|
||||
DIAGNOSTICS_EMITTED=0
|
||||
|
||||
fail() {
|
||||
FAILURE_CATEGORY="$1"
|
||||
echo "[restore-drill][FAIL] phase=$PHASE category=$1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_command() {
|
||||
command -v "$1" >/dev/null 2>&1 || fail "required_command_missing"
|
||||
}
|
||||
|
||||
rto_now() {
|
||||
date -u +%s
|
||||
}
|
||||
|
||||
rto_set_stage_seconds() {
|
||||
local stage="$1"
|
||||
local seconds="$2"
|
||||
case "$stage" in
|
||||
batch_copy) RTO_BATCH_COPY_SECONDS="$seconds" ;;
|
||||
postgres_container_start) RTO_POSTGRES_CONTAINER_START_SECONDS="$seconds" ;;
|
||||
dump_list) RTO_DUMP_LIST_SECONDS="$seconds" ;;
|
||||
target_confirmation) RTO_TARGET_CONFIRMATION_SECONDS="$seconds" ;;
|
||||
restore_apply) RTO_RESTORE_APPLY_SECONDS="$seconds" ;;
|
||||
database_verify) RTO_DATABASE_VERIFY_SECONDS="$seconds" ;;
|
||||
storage_attachment_verify) RTO_STORAGE_ATTACHMENT_VERIFY_SECONDS="$seconds" ;;
|
||||
final_integrity) RTO_FINAL_INTEGRITY_SECONDS="$seconds" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
rto_begin() {
|
||||
local stage="$1"
|
||||
local now
|
||||
[[ -z "$RTO_ACTIVE_STAGE" && "$RTO_FINALIZED" -eq 0 ]] || return 1
|
||||
now="$(rto_now)"
|
||||
[[ "$now" =~ ^[0-9]+$ ]] || return 1
|
||||
rto_set_stage_seconds "$stage" "NOT_RUN" || return 1
|
||||
RTO_ACTIVE_STAGE="$stage"
|
||||
RTO_STAGE_STARTED_EPOCH="$now"
|
||||
}
|
||||
|
||||
rto_close_active_at() {
|
||||
local now="$1"
|
||||
local elapsed
|
||||
[[ -n "$RTO_ACTIVE_STAGE" && "$RTO_STAGE_STARTED_EPOCH" =~ ^[0-9]+$ \
|
||||
&& "$now" =~ ^[0-9]+$ && "$now" -ge "$RTO_STAGE_STARTED_EPOCH" ]] \
|
||||
|| return 1
|
||||
elapsed="$((now - RTO_STAGE_STARTED_EPOCH))"
|
||||
rto_set_stage_seconds "$RTO_ACTIVE_STAGE" "$elapsed" || return 1
|
||||
}
|
||||
|
||||
rto_transition() {
|
||||
local expected_stage="$1"
|
||||
local next_stage="$2"
|
||||
local now
|
||||
[[ "$RTO_ACTIVE_STAGE" == "$expected_stage" && "$RTO_FINALIZED" -eq 0 ]] \
|
||||
|| return 1
|
||||
now="$(rto_now)"
|
||||
rto_close_active_at "$now" || return 1
|
||||
rto_set_stage_seconds "$next_stage" "NOT_RUN" || return 1
|
||||
RTO_ACTIVE_STAGE="$next_stage"
|
||||
RTO_STAGE_STARTED_EPOCH="$now"
|
||||
}
|
||||
|
||||
rto_finalize() {
|
||||
local now
|
||||
local value
|
||||
local total=0
|
||||
local measured=0
|
||||
if [[ "$RTO_FINALIZED" -eq 1 ]]; then
|
||||
return 0
|
||||
fi
|
||||
if [[ -n "$RTO_ACTIVE_STAGE" ]]; then
|
||||
now="$(rto_now)"
|
||||
rto_close_active_at "$now" || return 1
|
||||
RTO_ACTIVE_STAGE=""
|
||||
RTO_STAGE_STARTED_EPOCH=""
|
||||
fi
|
||||
for value in \
|
||||
"$RTO_BATCH_COPY_SECONDS" \
|
||||
"$RTO_POSTGRES_CONTAINER_START_SECONDS" \
|
||||
"$RTO_DUMP_LIST_SECONDS" \
|
||||
"$RTO_TARGET_CONFIRMATION_SECONDS" \
|
||||
"$RTO_RESTORE_APPLY_SECONDS" \
|
||||
"$RTO_DATABASE_VERIFY_SECONDS" \
|
||||
"$RTO_STORAGE_ATTACHMENT_VERIFY_SECONDS" \
|
||||
"$RTO_FINAL_INTEGRITY_SECONDS"; do
|
||||
if [[ "$value" =~ ^[0-9]+$ ]]; then
|
||||
total="$((total + value))"
|
||||
measured=1
|
||||
fi
|
||||
done
|
||||
if [[ "$measured" -eq 1 ]]; then
|
||||
RTO_SECONDS="$total"
|
||||
fi
|
||||
RTO_FINALIZED=1
|
||||
}
|
||||
|
||||
confirm_postgres_target() {
|
||||
local actual_database=""
|
||||
local query_exit_code=0
|
||||
local attempt=0
|
||||
TARGET_CONFIRMATION_EXIT_CODE="NOT_RUN"
|
||||
TARGET_NAME_MATCH="NOT_RUN"
|
||||
TARGET_CONFIRMATION_ATTEMPT_COUNT="0"
|
||||
TARGET_CONFIRMATION_SUCCESS_STREAK="0"
|
||||
TARGET_CONFIRMATION_FAILURE_REASON="TARGET_QUERY_NONZERO"
|
||||
for attempt in $(seq 1 60); do
|
||||
TARGET_CONFIRMATION_ATTEMPT_COUNT="$attempt"
|
||||
set +e
|
||||
actual_database="$(PGPASSWORD="$postgres_password" docker exec -e PGPASSWORD \
|
||||
"$CONTAINER_NAME" psql -h 127.0.0.1 -v ON_ERROR_STOP=1 \
|
||||
-U "$SHADOW_DB_USER" -d "$SHADOW_DB_NAME" \
|
||||
-Atqc 'SELECT current_database()' 2>>"$private_log")"
|
||||
query_exit_code="$?"
|
||||
set -e
|
||||
TARGET_CONFIRMATION_EXIT_CODE="$query_exit_code"
|
||||
if [[ "$query_exit_code" -ne 0 ]]; then
|
||||
TARGET_NAME_MATCH="NOT_RUN"
|
||||
TARGET_CONFIRMATION_SUCCESS_STREAK="0"
|
||||
[[ "$attempt" -eq 60 ]] || sleep 1
|
||||
continue
|
||||
fi
|
||||
if [[ "$actual_database" != "$SHADOW_DB_NAME" ]]; then
|
||||
TARGET_NAME_MATCH="no"
|
||||
TARGET_CONFIRMATION_FAILURE_REASON="TARGET_NAME_MISMATCH"
|
||||
return 1
|
||||
fi
|
||||
TARGET_NAME_MATCH="yes"
|
||||
TARGET_CONFIRMATION_SUCCESS_STREAK="$((TARGET_CONFIRMATION_SUCCESS_STREAK + 1))"
|
||||
if [[ "$TARGET_CONFIRMATION_SUCCESS_STREAK" -ge 2 ]]; then
|
||||
TARGET_CONFIRMATION_FAILURE_REASON="NONE"
|
||||
return 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
cleanup_runtime() {
|
||||
local cleanup_ok=1
|
||||
local container_suffix=""
|
||||
local work_suffix=""
|
||||
set +e
|
||||
if [[ -n "$CONTAINER_NAME" ]]; then
|
||||
container_suffix="${CONTAINER_NAME#"$CONTAINER_PREFIX"}"
|
||||
if [[ "$CONTAINER_NAME" == "$CONTAINER_PREFIX"* \
|
||||
&& "$container_suffix" =~ ^[0-9TZ-]+$ ]]; then
|
||||
docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
|
||||
docker inspect "$CONTAINER_NAME" >/dev/null 2>&1 && cleanup_ok=0
|
||||
else
|
||||
cleanup_ok=0
|
||||
fi
|
||||
fi
|
||||
if [[ -n "$WORK_DIR" ]]; then
|
||||
work_suffix="${WORK_DIR#"$WORK_ROOT/$WORK_PREFIX"}"
|
||||
if [[ "$WORK_DIR" == "$WORK_ROOT/$WORK_PREFIX"* \
|
||||
&& "$work_suffix" =~ ^[0-9TZ-]+$ \
|
||||
&& -d "$WORK_DIR" && ! -L "$WORK_DIR" ]]; then
|
||||
rm -rf -- "$WORK_DIR"
|
||||
[[ ! -e "$WORK_DIR" ]] || cleanup_ok=0
|
||||
elif [[ -e "$WORK_DIR" ]]; then
|
||||
cleanup_ok=0
|
||||
fi
|
||||
fi
|
||||
set -e
|
||||
[[ "$cleanup_ok" -eq 1 ]]
|
||||
}
|
||||
|
||||
cleanup_access() {
|
||||
local cleanup_ok=1
|
||||
set +e
|
||||
if [[ -f "$SUDOERS_PATH" && ! -L "$SUDOERS_PATH" \
|
||||
&& "$(stat -c %u "$SUDOERS_PATH" 2>/dev/null)" == "0" ]]; then
|
||||
rm -f -- "$SUDOERS_PATH" || cleanup_ok=0
|
||||
elif [[ -e "$SUDOERS_PATH" ]]; then
|
||||
cleanup_ok=0
|
||||
fi
|
||||
visudo -cf "$SUDOERS_MAIN_FILE" >/dev/null 2>&1 || cleanup_ok=0
|
||||
if [[ -d "$LIB_DIR" && ! -L "$LIB_DIR" \
|
||||
&& "$(stat -c %u "$LIB_DIR" 2>/dev/null)" == "0" ]]; then
|
||||
rm -rf -- "$LIB_DIR" || cleanup_ok=0
|
||||
elif [[ -e "$LIB_DIR" ]]; then
|
||||
cleanup_ok=0
|
||||
fi
|
||||
if [[ -f "$SELF_PATH" && ! -L "$SELF_PATH" \
|
||||
&& "$(stat -c %u "$SELF_PATH" 2>/dev/null)" == "0" ]]; then
|
||||
rm -f -- "$SELF_PATH" || cleanup_ok=0
|
||||
elif [[ -e "$SELF_PATH" ]]; then
|
||||
cleanup_ok=0
|
||||
fi
|
||||
set -e
|
||||
[[ "$cleanup_ok" -eq 1 ]]
|
||||
}
|
||||
|
||||
emit_restore_diagnostics() {
|
||||
if [[ "$DIAGNOSTICS_EMITTED" -eq 1 ]]; then
|
||||
return 0
|
||||
fi
|
||||
DIAGNOSTICS_EMITTED=1
|
||||
echo "failure_category=$FAILURE_CATEGORY"
|
||||
echo "dump_list=$DUMP_LIST_STATUS"
|
||||
echo "dump_list_exit_code=$DUMP_LIST_EXIT_CODE"
|
||||
echo "target_confirmation=$TARGET_CONFIRMATION_STATUS"
|
||||
echo "target_confirmation_exit_code=$TARGET_CONFIRMATION_EXIT_CODE"
|
||||
echo "target_name_match=$TARGET_NAME_MATCH"
|
||||
echo "target_confirmation_attempt_count=$TARGET_CONFIRMATION_ATTEMPT_COUNT"
|
||||
echo "target_confirmation_success_streak=$TARGET_CONFIRMATION_SUCCESS_STREAK"
|
||||
echo "restore_execution_path=$RESTORE_EXECUTION_PATH"
|
||||
echo "restore_apply=$RESTORE_APPLY_STATUS"
|
||||
echo "restore_script_exit_code=$RESTORE_SCRIPT_EXIT_CODE"
|
||||
echo "pg_restore_exit_code=$PG_RESTORE_EXIT_CODE"
|
||||
echo "pg_restore_error_count=$PG_RESTORE_ERROR_COUNT"
|
||||
echo "restore_error_category=$RESTORE_ERROR_CATEGORY"
|
||||
echo "rto_batch_copy_seconds=$RTO_BATCH_COPY_SECONDS"
|
||||
echo "rto_postgres_container_start_seconds=$RTO_POSTGRES_CONTAINER_START_SECONDS"
|
||||
echo "rto_dump_list_seconds=$RTO_DUMP_LIST_SECONDS"
|
||||
echo "rto_target_confirmation_seconds=$RTO_TARGET_CONFIRMATION_SECONDS"
|
||||
echo "rto_restore_apply_seconds=$RTO_RESTORE_APPLY_SECONDS"
|
||||
echo "rto_database_verify_seconds=$RTO_DATABASE_VERIFY_SECONDS"
|
||||
echo "rto_storage_attachment_verify_seconds=$RTO_STORAGE_ATTACHMENT_VERIFY_SECONDS"
|
||||
echo "rto_final_integrity_seconds=$RTO_FINAL_INTEGRITY_SECONDS"
|
||||
echo "data_restore_rto_seconds=$RTO_SECONDS"
|
||||
}
|
||||
|
||||
finish() {
|
||||
local exit_code="$?"
|
||||
local runtime_cleanup="PASS"
|
||||
local access_cleanup="PASS"
|
||||
trap - EXIT HUP INT TERM
|
||||
# Emit only allowlisted stage diagnostics while the private runtime still
|
||||
# exists. Raw stderr remains mode 0600 and is removed by cleanup_runtime.
|
||||
if ! rto_finalize; then
|
||||
FAILURE_CATEGORY="rto_stage_finalize_failed"
|
||||
exit_code=1
|
||||
fi
|
||||
emit_restore_diagnostics
|
||||
if [[ "$RUNTIME_CLEANED" -ne 1 ]]; then
|
||||
cleanup_runtime || runtime_cleanup="FAIL"
|
||||
fi
|
||||
cleanup_access || access_cleanup="FAIL"
|
||||
if [[ "$runtime_cleanup" != "PASS" || "$access_cleanup" != "PASS" ]]; then
|
||||
exit_code=1
|
||||
fi
|
||||
echo "runtime_cleanup=$runtime_cleanup"
|
||||
echo "temporary_access_cleanup=$access_cleanup"
|
||||
if [[ "$exit_code" -eq 0 ]]; then
|
||||
echo "result=$RESULT_STATUS"
|
||||
echo "rpo_seconds=$RPO_SECONDS"
|
||||
echo "migration_success=$MIGRATION_SUCCESS"
|
||||
echo "migration_other=$MIGRATION_OTHER"
|
||||
echo "migration_set_match=$MIGRATION_SET_MATCH"
|
||||
echo "archive_regular_files=$ARCHIVE_FILE_COUNT"
|
||||
echo "restored_regular_files=$RESTORED_FILE_COUNT"
|
||||
echo "attachment_reference_count=$ATTACHMENT_TOTAL"
|
||||
echo "attachment_mismatch_count=$ATTACHMENT_MISMATCH"
|
||||
printf '%s\n' "$TABLE_COUNTS"
|
||||
else
|
||||
echo "result=NO_GO"
|
||||
fi
|
||||
exit "$exit_code"
|
||||
}
|
||||
trap finish EXIT HUP INT TERM
|
||||
|
||||
for command_name in docker python3 sha256sum tar find stat df cp openssl visudo systemctl seq; do
|
||||
require_command "$command_name"
|
||||
done
|
||||
|
||||
PHASE="entry_contract"
|
||||
[[ "$(id -u)" -eq 0 ]] || fail "root_required"
|
||||
for absolute_path in "$CONFIG_FILE" "$LIB_DIR" "$SELF_PATH" "$SUDOERS_PATH" \
|
||||
"$SUDOERS_MAIN_FILE" "$WORK_ROOT"; do
|
||||
[[ "$absolute_path" == /* && "$absolute_path" != "/" ]] \
|
||||
|| fail "rendered_path_invalid"
|
||||
done
|
||||
[[ "$WORK_PREFIX" =~ ^[a-z0-9][a-z0-9-]*-$ \
|
||||
&& "$CONTAINER_PREFIX" =~ ^[a-z0-9][a-z0-9-]*-$ \
|
||||
&& "$SHADOW_DB_USER" =~ ^[a-z_][a-z0-9_]*$ \
|
||||
&& "$SHADOW_DB_NAME" =~ (^|_)(shadow|drill|restore_test)(_|$) \
|
||||
&& "$POSTGRES_IMAGE" =~ ^[a-z0-9._/-]+:[a-zA-Z0-9._-]+$ \
|
||||
&& "$DEADMAN_TIMER_UNIT" =~ ^[a-zA-Z0-9_.@-]+[.]timer$ \
|
||||
&& "$HEALTH_TIMER_UNIT" =~ ^[a-zA-Z0-9_.@-]+[.]timer$ \
|
||||
&& "$ARCHIVE_BATCH_PREFIX" =~ ^[a-z0-9][a-z0-9-]*-$ \
|
||||
&& "$RUN_ID_PREFIX" =~ ^[a-z0-9][a-z0-9-]*-$ ]] \
|
||||
|| fail "rendered_identity_invalid"
|
||||
[[ "$EXPECTED_DB_RESTORE_SHA" =~ ^[0-9a-f]{64}$ \
|
||||
&& "$EXPECTED_STORAGE_RESTORE_SHA" =~ ^[0-9a-f]{64}$ \
|
||||
&& "$EXPECTED_ATTACHMENT_VALIDATOR_SHA" =~ ^[0-9a-f]{64}$ \
|
||||
&& "$EXPECTED_BUILD_PREFIX" =~ ^[0-9a-f]{12}$ \
|
||||
&& "$EXPECTED_MIGRATION_SET_SHA" =~ ^[0-9a-f]{64}$ ]] \
|
||||
|| fail "rendered_digest_invalid"
|
||||
[[ -f "$CONFIG_FILE" && ! -L "$CONFIG_FILE" \
|
||||
&& "$(stat -c %u "$CONFIG_FILE")" == "0" ]] || fail "receiver_config_invalid"
|
||||
[[ -f "$LIB_DIR/restore-db.sh" && ! -L "$LIB_DIR/restore-db.sh" \
|
||||
&& -f "$LIB_DIR/restore-storage.sh" && ! -L "$LIB_DIR/restore-storage.sh" \
|
||||
&& -f "$LIB_DIR/validate-attachment-restore.py" \
|
||||
&& ! -L "$LIB_DIR/validate-attachment-restore.py" ]] \
|
||||
|| fail "restore_library_missing"
|
||||
[[ "$(sha256sum "$LIB_DIR/restore-db.sh" | awk '{print $1}')" == "$EXPECTED_DB_RESTORE_SHA" ]] \
|
||||
|| fail "db_restore_script_hash_mismatch"
|
||||
[[ "$(sha256sum "$LIB_DIR/restore-storage.sh" | awk '{print $1}')" == "$EXPECTED_STORAGE_RESTORE_SHA" ]] \
|
||||
|| fail "storage_restore_script_hash_mismatch"
|
||||
[[ "$(sha256sum "$LIB_DIR/validate-attachment-restore.py" | awk '{print $1}')" == "$EXPECTED_ATTACHMENT_VALIDATOR_SHA" ]] \
|
||||
|| fail "attachment_validator_hash_mismatch"
|
||||
|
||||
# Root-owned receiver configuration contains path-only assignments.
|
||||
# shellcheck disable=SC1090
|
||||
source "$CONFIG_FILE"
|
||||
BACKUP_ARCHIVE_DIR="${BACKUP_ARCHIVE_DIR:-}"
|
||||
[[ "$BACKUP_ARCHIVE_DIR" == /* && "$BACKUP_ARCHIVE_DIR" != "/" \
|
||||
&& -d "$BACKUP_ARCHIVE_DIR" && ! -L "$BACKUP_ARCHIVE_DIR" ]] \
|
||||
|| fail "archive_root_invalid"
|
||||
|
||||
PHASE="monitor_preflight"
|
||||
deadman_enabled_before="$(systemctl is-enabled "$DEADMAN_TIMER_UNIT" 2>/dev/null || true)"
|
||||
deadman_active_before="$(systemctl is-active "$DEADMAN_TIMER_UNIT" 2>/dev/null || true)"
|
||||
health_enabled_before="$(systemctl is-enabled "$HEALTH_TIMER_UNIT" 2>/dev/null || true)"
|
||||
health_active_before="$(systemctl is-active "$HEALTH_TIMER_UNIT" 2>/dev/null || true)"
|
||||
[[ "$deadman_enabled_before" == "enabled" && "$deadman_active_before" == "active" \
|
||||
&& "$health_enabled_before" == "enabled" && "$health_active_before" == "active" ]] \
|
||||
|| fail "monitor_preflight_not_ready"
|
||||
|
||||
validate_batch() {
|
||||
local batch_dir="$1"
|
||||
local identity_mode="$2"
|
||||
local expected_run_id="${3:-}"
|
||||
python3 - "$batch_dir" "$EXPECTED_BUILD_PREFIX" "$identity_mode" \
|
||||
"$expected_run_id" "$ARCHIVE_BATCH_PREFIX" "$RUN_ID_PREFIX" <<'PY'
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import sys
|
||||
|
||||
(
|
||||
batch_dir,
|
||||
build_prefix,
|
||||
identity_mode,
|
||||
expected_run_id,
|
||||
archive_batch_prefix,
|
||||
run_id_prefix,
|
||||
) = sys.argv[1:]
|
||||
if identity_mode not in {"source", "copy"}:
|
||||
raise SystemExit("identity_mode")
|
||||
if not os.path.isdir(batch_dir) or os.path.islink(batch_dir):
|
||||
raise SystemExit("batch_type")
|
||||
entries = set(os.listdir(batch_dir))
|
||||
expected = {"database.dump", "storage.tar.gz", "manifest.json"}
|
||||
if entries != expected:
|
||||
raise SystemExit("batch_files")
|
||||
for name in expected:
|
||||
path = os.path.join(batch_dir, name)
|
||||
mode = os.lstat(path).st_mode
|
||||
if not stat.S_ISREG(mode) or stat.S_ISLNK(mode):
|
||||
raise SystemExit("artifact_type")
|
||||
with open(os.path.join(batch_dir, "manifest.json"), encoding="utf-8") as handle:
|
||||
manifest = json.load(handle)
|
||||
run_id = manifest.get("runId")
|
||||
if manifest.get("schemaVersion") != 1 or manifest.get("status") != "complete":
|
||||
raise SystemExit("manifest_contract")
|
||||
run_id_pattern = re.escape(run_id_prefix) + r"[0-9]{8}T[0-9]{6}Z"
|
||||
if not isinstance(run_id, str) or not re.fullmatch(run_id_pattern, run_id):
|
||||
raise SystemExit("run_id")
|
||||
if identity_mode == "source":
|
||||
if expected_run_id:
|
||||
raise SystemExit("unexpected_expected_run_id")
|
||||
if os.path.basename(batch_dir) != archive_batch_prefix + run_id:
|
||||
raise SystemExit("source_batch_identity")
|
||||
else:
|
||||
if not re.fullmatch(run_id_pattern, expected_run_id):
|
||||
raise SystemExit("expected_run_id")
|
||||
if run_id != expected_run_id:
|
||||
raise SystemExit("copied_batch_source_identity")
|
||||
build = manifest.get("buildSha")
|
||||
if not isinstance(build, str) or not re.fullmatch(r"[0-9a-f]{40,64}", build):
|
||||
raise SystemExit("build_identity")
|
||||
if not build.startswith(build_prefix):
|
||||
raise SystemExit("build_mismatch")
|
||||
created = manifest.get("createdAt")
|
||||
try:
|
||||
created_at = dt.datetime.strptime(created, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
|
||||
except Exception as exc:
|
||||
raise SystemExit("created_at") from exc
|
||||
artifacts = manifest.get("artifacts") or {}
|
||||
total_bytes = 0
|
||||
for key, name in (("database", "database.dump"), ("storage", "storage.tar.gz")):
|
||||
meta = artifacts.get(key) or {}
|
||||
if meta.get("file") != name or os.path.isabs(str(meta.get("file") or "")):
|
||||
raise SystemExit("artifact_name")
|
||||
path = os.path.join(batch_dir, name)
|
||||
size = os.path.getsize(path)
|
||||
if size != meta.get("bytes"):
|
||||
raise SystemExit("artifact_size")
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
if digest.hexdigest() != meta.get("sha256"):
|
||||
raise SystemExit("artifact_hash")
|
||||
total_bytes += size
|
||||
print(
|
||||
int(created_at.timestamp()),
|
||||
total_bytes,
|
||||
build[:8],
|
||||
int(bool(manifest.get("sourceImageDigest"))),
|
||||
run_id,
|
||||
)
|
||||
PY
|
||||
}
|
||||
|
||||
PHASE="select_latest_batch"
|
||||
selected_batch="$(python3 - "$BACKUP_ARCHIVE_DIR" \
|
||||
"$ARCHIVE_BATCH_PREFIX" "$RUN_ID_PREFIX" <<'PY'
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
root, archive_batch_prefix, run_id_prefix = sys.argv[1:]
|
||||
batch_pattern = re.escape(archive_batch_prefix + run_id_prefix) + r"[0-9]{8}T[0-9]{6}Z"
|
||||
candidates = []
|
||||
for entry in os.scandir(root):
|
||||
if entry.is_symlink() or not entry.is_dir(follow_symlinks=False):
|
||||
continue
|
||||
if not re.fullmatch(batch_pattern, entry.name):
|
||||
continue
|
||||
candidates.append((entry.name, entry.path))
|
||||
if not candidates:
|
||||
raise SystemExit(1)
|
||||
candidates.sort(key=lambda item: item[0])
|
||||
print(candidates[-1][1])
|
||||
PY
|
||||
)" || fail "latest_batch_not_found"
|
||||
|
||||
batch_metadata="$(validate_batch "$selected_batch" source "")" || fail "latest_batch_invalid"
|
||||
read -r created_epoch batch_total_bytes build_short source_digest_present SOURCE_RUN_ID <<<"$batch_metadata"
|
||||
drill_started_epoch="$(date -u +%s)"
|
||||
RPO_SECONDS="$((drill_started_epoch - created_epoch))"
|
||||
[[ "$RPO_SECONDS" -ge 0 && "$RPO_SECONDS" -le 86400 ]] || fail "latest_batch_outside_rpo"
|
||||
required_bytes="$((batch_total_bytes * 5 + 2147483648))"
|
||||
available_kib="$(df -Pk "$WORK_ROOT" | awk 'NR==2 {print $4}')"
|
||||
[[ "$available_kib" =~ ^[0-9]+$ ]] || fail "disk_capacity_unavailable"
|
||||
available_bytes="$((available_kib * 1024))"
|
||||
[[ "$available_bytes" -ge "$required_bytes" ]] || fail "disk_capacity_insufficient"
|
||||
|
||||
PHASE="create_isolated_runtime"
|
||||
run_id="$(date -u +%Y%m%dT%H%M%SZ)-$$-$RANDOM"
|
||||
[[ "$run_id" =~ ^[0-9TZ-]+$ ]] || fail "run_id_invalid"
|
||||
WORK_DIR="$WORK_ROOT/$WORK_PREFIX$run_id"
|
||||
CONTAINER_NAME="$CONTAINER_PREFIX$run_id"
|
||||
[[ ! -e "$WORK_DIR" ]] || fail "work_dir_exists"
|
||||
mkdir -m 0700 "$WORK_DIR"
|
||||
private_log="$WORK_DIR/private.log"
|
||||
: >"$private_log"
|
||||
chmod 0600 "$private_log"
|
||||
rto_begin batch_copy || fail "rto_stage_begin_failed"
|
||||
cp -a -- "$selected_batch" "$WORK_DIR/batch"
|
||||
copied_metadata="$(validate_batch "$WORK_DIR/batch" copy "$SOURCE_RUN_ID")" \
|
||||
|| fail "copied_batch_invalid"
|
||||
[[ "$copied_metadata" == "$batch_metadata" ]] || fail "copied_batch_identity_mismatch"
|
||||
rto_transition batch_copy postgres_container_start \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
env_file="$WORK_DIR/postgres.env"
|
||||
postgres_password="$(openssl rand -hex 24)"
|
||||
{
|
||||
echo "POSTGRES_USER=$SHADOW_DB_USER"
|
||||
echo "POSTGRES_PASSWORD=$postgres_password"
|
||||
echo "POSTGRES_DB=$SHADOW_DB_NAME"
|
||||
} >"$env_file"
|
||||
chmod 0600 "$env_file"
|
||||
printf '%s\n' '[stage=postgres_container_start]' >>"$private_log"
|
||||
docker run -d --rm --name "$CONTAINER_NAME" --network none --env-file "$env_file" \
|
||||
"$POSTGRES_IMAGE" >>"$private_log" 2>&1 || fail "postgres_container_start_failed"
|
||||
rto_transition postgres_container_start dump_list \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
PHASE="restore_database"
|
||||
DUMP_LIST_STATUS="FAIL"
|
||||
RESTORE_EXECUTION_PATH="DOCKER_STDIN"
|
||||
printf '%s\n' '[stage=dump_list]' >>"$private_log"
|
||||
set +e
|
||||
docker exec -i "$CONTAINER_NAME" pg_restore --list \
|
||||
<"$WORK_DIR/batch/database.dump" >>"$private_log" 2>&1
|
||||
DUMP_LIST_EXIT_CODE="$?"
|
||||
set -e
|
||||
if [[ "$DUMP_LIST_EXIT_CODE" -ne 0 ]]; then
|
||||
RESTORE_ERROR_CATEGORY="DUMP_LIST_NONZERO"
|
||||
fail "dump_list_failed"
|
||||
fi
|
||||
DUMP_LIST_STATUS="PASS"
|
||||
rto_transition dump_list target_confirmation \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
TARGET_CONFIRMATION_STATUS="FAIL"
|
||||
printf '%s\n' '[stage=target_confirmation]' >>"$private_log"
|
||||
if ! confirm_postgres_target; then
|
||||
RESTORE_ERROR_CATEGORY="$TARGET_CONFIRMATION_FAILURE_REASON"
|
||||
fail "restore_target_confirmation_failed"
|
||||
fi
|
||||
TARGET_CONFIRMATION_STATUS="PASS"
|
||||
rto_transition target_confirmation restore_apply \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
real_docker_path="$(command -v docker)"
|
||||
docker_proxy_dir="$WORK_DIR/docker-proxy"
|
||||
docker_proxy="$docker_proxy_dir/docker"
|
||||
restore_diag_file="$WORK_DIR/restore-diag"
|
||||
mkdir -m 0700 "$docker_proxy_dir"
|
||||
cat >"$docker_proxy" <<'PROXY'
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
"$RESTORE_REAL_DOCKER" "$@"
|
||||
command_exit="$?"
|
||||
is_pg_restore=0
|
||||
is_list=0
|
||||
for argument in "$@"; do
|
||||
[[ "$argument" == "pg_restore" ]] && is_pg_restore=1
|
||||
[[ "$argument" == "--list" ]] && is_list=1
|
||||
done
|
||||
if [[ "$is_pg_restore" -eq 1 ]]; then
|
||||
if [[ "$is_list" -eq 1 ]]; then
|
||||
printf 'script_dump_list_exit_code=%s\n' "$command_exit" >>"$RESTORE_DIAG_FILE"
|
||||
else
|
||||
printf 'restore_apply_exit_code=%s\n' "$command_exit" >>"$RESTORE_DIAG_FILE"
|
||||
fi
|
||||
fi
|
||||
exit "$command_exit"
|
||||
PROXY
|
||||
chmod 0700 "$docker_proxy"
|
||||
: >"$restore_diag_file"
|
||||
chmod 0600 "$restore_diag_file"
|
||||
|
||||
RESTORE_APPLY_STATUS="FAIL"
|
||||
printf '%s\n' '[stage=restore_apply]' >>"$private_log"
|
||||
set +e
|
||||
RESTORE_REAL_DOCKER="$real_docker_path" \
|
||||
RESTORE_DIAG_FILE="$restore_diag_file" \
|
||||
PATH="$docker_proxy_dir:$PATH" \
|
||||
RESTORE_SCOPE=shadow \
|
||||
RESTORE_CONFIRM_DATABASE="$SHADOW_DB_NAME" \
|
||||
DB_TOOL_MODE=docker \
|
||||
DB_CONTAINER="$CONTAINER_NAME" \
|
||||
DATABASE_URL="postgresql://shadow.invalid/$SHADOW_DB_NAME" \
|
||||
POSTGRES_DB="$SHADOW_DB_NAME" \
|
||||
POSTGRES_USER="$SHADOW_DB_USER" \
|
||||
POSTGRES_PASSWORD="$postgres_password" \
|
||||
"$LIB_DIR/restore-db.sh" "$WORK_DIR/batch/database.dump" \
|
||||
>>"$private_log" 2>&1
|
||||
RESTORE_SCRIPT_EXIT_CODE="$?"
|
||||
set -e
|
||||
|
||||
script_dump_list_exit_code="$(awk -F= \
|
||||
'$1 == "script_dump_list_exit_code" {print $2}' "$restore_diag_file")"
|
||||
apply_exit_code="$(awk -F= \
|
||||
'$1 == "restore_apply_exit_code" {print $2}' "$restore_diag_file")"
|
||||
if [[ ! "$script_dump_list_exit_code" =~ ^[0-9]+$ \
|
||||
|| "$script_dump_list_exit_code" -ne 0 ]]; then
|
||||
RESTORE_ERROR_CATEGORY="SCRIPT_DUMP_LIST_NONZERO"
|
||||
fail "restore_script_list_failed"
|
||||
fi
|
||||
if [[ ! "$apply_exit_code" =~ ^[0-9]+$ ]]; then
|
||||
RESTORE_ERROR_CATEGORY="APPLY_EXIT_CODE_MISSING"
|
||||
fail "restore_diagnostic_missing"
|
||||
fi
|
||||
PG_RESTORE_EXIT_CODE="$apply_exit_code"
|
||||
if [[ "$PG_RESTORE_EXIT_CODE" -eq 0 ]]; then
|
||||
PG_RESTORE_ERROR_COUNT="0"
|
||||
else
|
||||
PG_RESTORE_ERROR_COUNT="$(sed -nE \
|
||||
's/^pg_restore: (warning: )?errors ignored on restore: ([0-9]+)$/\2/p' \
|
||||
"$private_log" | tail -n 1)"
|
||||
[[ "$PG_RESTORE_ERROR_COUNT" =~ ^[0-9]+$ ]] \
|
||||
|| PG_RESTORE_ERROR_COUNT="UNKNOWN"
|
||||
fi
|
||||
if [[ "$RESTORE_SCRIPT_EXIT_CODE" -ne 0 || "$PG_RESTORE_EXIT_CODE" -ne 0 ]]; then
|
||||
RESTORE_ERROR_CATEGORY="RESTORE_APPLY_NONZERO"
|
||||
fail "restore_apply_failed_after_list_pass"
|
||||
fi
|
||||
RESTORE_APPLY_STATUS="PASS"
|
||||
RESTORE_ERROR_CATEGORY="NONE"
|
||||
rto_transition restore_apply database_verify \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
PHASE="verify_database"
|
||||
migration_names="$(PGPASSWORD="$postgres_password" docker exec -i -e PGPASSWORD \
|
||||
"$CONTAINER_NAME" psql -v ON_ERROR_STOP=1 -U "$SHADOW_DB_USER" -d "$SHADOW_DB_NAME" \
|
||||
-At <<'SQL' | LC_ALL=C sort
|
||||
SELECT migration_name
|
||||
FROM "_prisma_migrations"
|
||||
WHERE finished_at IS NOT NULL AND rolled_back_at IS NULL
|
||||
ORDER BY migration_name;
|
||||
SQL
|
||||
)" || fail "migration_name_query_failed"
|
||||
[[ -n "$migration_names" ]] || fail "migration_set_empty"
|
||||
if printf '%s\n' "$migration_names" | grep -Ev '^[0-9]{14}_[a-z0-9_]+$' >/dev/null; then
|
||||
fail "migration_name_invalid"
|
||||
fi
|
||||
MIGRATION_SUCCESS="$(printf '%s\n' "$migration_names" | wc -l | tr -d ' ')"
|
||||
MIGRATION_OTHER="$(PGPASSWORD="$postgres_password" docker exec -i -e PGPASSWORD \
|
||||
"$CONTAINER_NAME" psql -v ON_ERROR_STOP=1 -U "$SHADOW_DB_USER" -d "$SHADOW_DB_NAME" \
|
||||
-At <<'SQL'
|
||||
SELECT count(*)
|
||||
FROM "_prisma_migrations"
|
||||
WHERE finished_at IS NULL OR rolled_back_at IS NOT NULL;
|
||||
SQL
|
||||
)" || fail "migration_state_query_failed"
|
||||
[[ "$MIGRATION_OTHER" == "0" ]] || fail "migration_state_mismatch"
|
||||
migration_set_sha="$(printf '%s\n' "$migration_names" | sha256sum | awk '{print $1}')"
|
||||
[[ "$migration_set_sha" == "$EXPECTED_MIGRATION_SET_SHA" ]] \
|
||||
|| fail "migration_set_mismatch"
|
||||
MIGRATION_SET_MATCH="PASS"
|
||||
|
||||
TABLE_COUNTS="$(PGPASSWORD="$postgres_password" docker exec -i -e PGPASSWORD \
|
||||
"$CONTAINER_NAME" psql -v ON_ERROR_STOP=1 -U "$SHADOW_DB_USER" -d "$SHADOW_DB_NAME" \
|
||||
-At -F = <<'SQL' | LC_ALL=C sort
|
||||
SELECT key, value FROM (VALUES
|
||||
('departments', (SELECT count(*)::bigint FROM departments)),
|
||||
('employees', (SELECT count(*)::bigint FROM employees)),
|
||||
('employee_roles', (SELECT count(*)::bigint FROM employee_roles)),
|
||||
('purchase_requests', (SELECT count(*)::bigint FROM purchase_requests)),
|
||||
('reimbursement_forms', (SELECT count(*)::bigint FROM reimbursement_forms)),
|
||||
('invoice_pool', (SELECT count(*)::bigint FROM invoice_pool)),
|
||||
('payment_batches', (SELECT count(*)::bigint FROM payment_batches)),
|
||||
('contract_registry', (SELECT count(*)::bigint FROM contract_registry)),
|
||||
('projects', (SELECT count(*)::bigint FROM projects)),
|
||||
('purchase_attachments', (SELECT count(*)::bigint FROM purchase_attachments)),
|
||||
('reimbursement_attachments', (SELECT count(*)::bigint FROM reimbursement_attachments)),
|
||||
('payment_batch_receipt_attachments', (SELECT count(*)::bigint FROM payment_batch_receipt_attachments))
|
||||
) AS counts(key, value);
|
||||
SQL
|
||||
)" || fail "table_count_query_failed"
|
||||
[[ "$(printf '%s\n' "$TABLE_COUNTS" | sed '/^$/d' | wc -l | tr -d ' ')" == "12" ]] \
|
||||
|| fail "table_count_key_mismatch"
|
||||
rto_transition database_verify storage_attachment_verify \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
PHASE="restore_storage"
|
||||
restored_storage="$WORK_DIR/restored-storage"
|
||||
printf '%s\n' '[stage=restore_storage]' >>"$private_log"
|
||||
if ! RESTORE_SCOPE=shadow "$LIB_DIR/restore-storage.sh" \
|
||||
"$WORK_DIR/batch/storage.tar.gz" "$restored_storage" \
|
||||
>>"$private_log" 2>&1; then
|
||||
fail "storage_restore_failed"
|
||||
fi
|
||||
ARCHIVE_FILE_COUNT="$(tar -tvzf "$WORK_DIR/batch/storage.tar.gz" \
|
||||
| awk 'substr($0,1,1)=="-" {count++} END {print count+0}')"
|
||||
RESTORED_FILE_COUNT="$(find "$restored_storage" -type f -printf '.' | wc -c | tr -d ' ')"
|
||||
[[ "$ARCHIVE_FILE_COUNT" == "$RESTORED_FILE_COUNT" ]] || fail "storage_file_count_mismatch"
|
||||
|
||||
PHASE="verify_attachment_references"
|
||||
attachment_result="$(PGPASSWORD="$postgres_password" docker exec -i -e PGPASSWORD \
|
||||
"$CONTAINER_NAME" psql -v ON_ERROR_STOP=1 -U "$SHADOW_DB_USER" -d "$SHADOW_DB_NAME" \
|
||||
-At -F $'\t' <<'SQL' | python3 "$LIB_DIR/validate-attachment-restore.py" "$restored_storage"
|
||||
SELECT encode(convert_to(storage_path, 'UTF8'), 'base64'), file_size::text, coalesce(file_sha256, '') FROM invoice_pool
|
||||
UNION ALL
|
||||
SELECT encode(convert_to(storage_path, 'UTF8'), 'base64'), file_size::text, coalesce(file_sha256, '') FROM purchase_attachments
|
||||
UNION ALL
|
||||
SELECT encode(convert_to(storage_path, 'UTF8'), 'base64'), file_size::text, coalesce(file_sha256, '') FROM reimbursement_attachments
|
||||
UNION ALL
|
||||
SELECT encode(convert_to(storage_path, 'UTF8'), 'base64'), file_size::text, coalesce(file_sha256, '') FROM payment_batch_receipt_attachments
|
||||
UNION ALL
|
||||
SELECT encode(convert_to(attachment_storage_path, 'UTF8'), 'base64'), '', '' FROM leave_requests WHERE attachment_storage_path IS NOT NULL;
|
||||
SQL
|
||||
)" || fail "attachment_reference_query_failed"
|
||||
read -r ATTACHMENT_TOTAL ATTACHMENT_MISMATCH attachment_s3_count <<<"$attachment_result"
|
||||
[[ "$ATTACHMENT_TOTAL" =~ ^[0-9]+$ && "$ATTACHMENT_MISMATCH" == "0" \
|
||||
&& "$attachment_s3_count" == "0" ]] || fail "attachment_reference_mismatch"
|
||||
rto_transition storage_attachment_verify final_integrity \
|
||||
|| fail "rto_stage_transition_failed"
|
||||
|
||||
PHASE="post_validation"
|
||||
copied_metadata_after="$(validate_batch "$WORK_DIR/batch" copy "$SOURCE_RUN_ID")" \
|
||||
|| fail "copied_batch_postcheck_failed"
|
||||
source_metadata_after="$(validate_batch "$selected_batch" source "")" \
|
||||
|| fail "source_archive_postcheck_failed"
|
||||
[[ "$copied_metadata_after" == "$batch_metadata" \
|
||||
&& "$source_metadata_after" == "$batch_metadata" ]] || fail "archive_identity_changed"
|
||||
rto_finalize || fail "rto_stage_finalize_failed"
|
||||
|
||||
deadman_enabled_after="$(systemctl is-enabled "$DEADMAN_TIMER_UNIT" 2>/dev/null || true)"
|
||||
deadman_active_after="$(systemctl is-active "$DEADMAN_TIMER_UNIT" 2>/dev/null || true)"
|
||||
health_enabled_after="$(systemctl is-enabled "$HEALTH_TIMER_UNIT" 2>/dev/null || true)"
|
||||
health_active_after="$(systemctl is-active "$HEALTH_TIMER_UNIT" 2>/dev/null || true)"
|
||||
[[ "$deadman_enabled_after" == "$deadman_enabled_before" \
|
||||
&& "$deadman_active_after" == "$deadman_active_before" \
|
||||
&& "$health_enabled_after" == "$health_enabled_before" \
|
||||
&& "$health_active_after" == "$health_active_before" ]] \
|
||||
|| fail "monitor_state_changed"
|
||||
|
||||
PHASE="runtime_cleanup"
|
||||
cleanup_runtime || fail "runtime_cleanup_failed"
|
||||
RUNTIME_CLEANED=1
|
||||
RESULT_STATUS="GO"
|
||||
PHASE="complete"
|
||||
exit 0
|
||||
@@ -0,0 +1,338 @@
|
||||
#!/usr/bin/env bash
|
||||
set -eEuo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
INSPECTOR="$ROOT_DIR/scripts/qa/inspect-backup-transfer-incident.py"
|
||||
TEST_ROOT="$(mktemp -d)"
|
||||
PASS_COUNT=0
|
||||
|
||||
cleanup() {
|
||||
if [[ -n "${TEST_ROOT:-}" && -d "$TEST_ROOT" && "$TEST_ROOT" == /tmp/* ]]; then
|
||||
rm -rf -- "$TEST_ROOT"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
pass() {
|
||||
PASS_COUNT=$((PASS_COUNT + 1))
|
||||
printf '[PASS] %s\n' "$1"
|
||||
}
|
||||
|
||||
assert_line() {
|
||||
local file="$1"
|
||||
local expected="$2"
|
||||
grep -Fxq -- "$expected" "$file" || {
|
||||
printf '[FAIL] missing field: %s\n' "$expected" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
make_batch() {
|
||||
local target="$1"
|
||||
local run_id="$2"
|
||||
local db_bytes="${3:-1024}"
|
||||
local storage_bytes="${4:-2048}"
|
||||
mkdir -p "$target"
|
||||
head -c "$db_bytes" /dev/zero >"$target/database.dump"
|
||||
head -c "$storage_bytes" /dev/zero >"$target/storage.tar.gz"
|
||||
python3 - "$target" "$run_id" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
target = Path(sys.argv[1])
|
||||
run_id = sys.argv[2]
|
||||
|
||||
def item(name):
|
||||
data = (target / name).read_bytes()
|
||||
return {"file": name, "bytes": len(data), "sha256": hashlib.sha256(data).hexdigest()}
|
||||
|
||||
payload = {
|
||||
"schemaVersion": 1,
|
||||
"runId": run_id,
|
||||
"createdAt": "2026-08-28T00:00:00Z",
|
||||
"buildSha": "a" * 40,
|
||||
"sourceImageDigest": None,
|
||||
"status": "complete",
|
||||
"artifacts": {"database": item("database.dump"), "storage": item("storage.tar.gz")},
|
||||
}
|
||||
(target / "manifest.json").write_text(json.dumps(payload) + "\n", encoding="utf-8")
|
||||
PY
|
||||
}
|
||||
|
||||
run_production() {
|
||||
local fixture="$1"
|
||||
local output="$2"
|
||||
python3 -B "$INSPECTOR" --now-epoch 1787875200 production \
|
||||
--backup-root "$fixture/backups" \
|
||||
--state-dir "$fixture/state" \
|
||||
--wrapper-script "$ROOT_DIR/scripts/run-automatic-backup.sh" \
|
||||
--backup-script "$ROOT_DIR/scripts/backup-all.sh" >"$output"
|
||||
}
|
||||
|
||||
make_production_layout() {
|
||||
local fixture="$1"
|
||||
mkdir -p "$fixture/backups" "$fixture/state"
|
||||
}
|
||||
|
||||
write_transfer_failure_state() {
|
||||
local fixture="$1"
|
||||
printf '%s\n' '{"findings":[{"detail":"phase_offhost_transfer_failed"}]}' >"$fixture/state/last-run.json"
|
||||
}
|
||||
|
||||
PRODUCTION_VALID="$TEST_ROOT/production-valid"
|
||||
make_production_layout "$PRODUCTION_VALID"
|
||||
make_batch "$PRODUCTION_VALID/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
write_transfer_failure_state "$PRODUCTION_VALID"
|
||||
printf 'rsync error: synthetic redacted fixture (code 23) at fixture.c(1)\n' >"$PRODUCTION_VALID/state/last-error.log"
|
||||
run_production "$PRODUCTION_VALID" "$TEST_ROOT/production-valid.out"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "complete_local_recovery_point=yes"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "backup_risk_classification=production_single_point"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "transition_recovery_point_action=wait_natural"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "transfer_exit_code=23"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "transfer_failure_category=partial_transfer_error"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "transfer_duration_seconds=NOT_AVAILABLE"
|
||||
pass "生产存在完整本地恢复点时识别为单点风险并保留未测时长"
|
||||
|
||||
PRODUCTION_MULTIPLE="$TEST_ROOT/production-multiple"
|
||||
make_production_layout "$PRODUCTION_MULTIPLE"
|
||||
make_batch "$PRODUCTION_MULTIPLE/backups/hyb-oa-backup-auto-20260827T000000Z" "auto-20260827T000000Z"
|
||||
make_batch "$PRODUCTION_MULTIPLE/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
printf 'X' | dd of="$PRODUCTION_MULTIPLE/backups/hyb-oa-backup-auto-20260827T000000Z/database.dump" bs=1 seek=0 conv=notrunc status=none
|
||||
run_production "$PRODUCTION_MULTIPLE" "$TEST_ROOT/production-multiple.out"
|
||||
assert_line "$TEST_ROOT/production-multiple.out" "complete_local_recovery_point=yes"
|
||||
assert_line "$TEST_ROOT/production-multiple.out" "fully_verified_local_batch_count=1"
|
||||
assert_line "$TEST_ROOT/production-multiple.out" "metadata_complete_local_batch_count=2"
|
||||
assert_line "$TEST_ROOT/production-multiple.out" "older_metadata_batch_hash_status=NOT_RUN"
|
||||
assert_line "$TEST_ROOT/production-multiple.out" "latest_local_candidate_hash_status=PASS"
|
||||
pass "生产多个批次时只对最新元数据合格候选执行全量 SHA-256"
|
||||
|
||||
PRODUCTION_LATEST_HASH_FAIL="$TEST_ROOT/production-latest-hash-fail"
|
||||
make_production_layout "$PRODUCTION_LATEST_HASH_FAIL"
|
||||
make_batch "$PRODUCTION_LATEST_HASH_FAIL/backups/hyb-oa-backup-auto-20260827T000000Z" "auto-20260827T000000Z"
|
||||
make_batch "$PRODUCTION_LATEST_HASH_FAIL/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
printf 'X' | dd of="$PRODUCTION_LATEST_HASH_FAIL/backups/hyb-oa-backup-auto-20260828T000000Z/database.dump" bs=1 seek=0 conv=notrunc status=none
|
||||
run_production "$PRODUCTION_LATEST_HASH_FAIL" "$TEST_ROOT/production-latest-hash-fail.out"
|
||||
assert_line "$TEST_ROOT/production-latest-hash-fail.out" "complete_local_recovery_point=no"
|
||||
assert_line "$TEST_ROOT/production-latest-hash-fail.out" "fully_verified_local_batch_count=0"
|
||||
assert_line "$TEST_ROOT/production-latest-hash-fail.out" "metadata_complete_local_batch_count=2"
|
||||
assert_line "$TEST_ROOT/production-latest-hash-fail.out" "older_metadata_batch_hash_status=NOT_RUN"
|
||||
assert_line "$TEST_ROOT/production-latest-hash-fail.out" "latest_local_candidate_hash_status=FAIL"
|
||||
pass "最新候选哈希失败时不回退扫描旧批次并保持无已证明恢复点"
|
||||
|
||||
PRODUCTION_EMPTY="$TEST_ROOT/production-empty"
|
||||
make_production_layout "$PRODUCTION_EMPTY"
|
||||
run_production "$PRODUCTION_EMPTY" "$TEST_ROOT/production-empty.out"
|
||||
assert_line "$TEST_ROOT/production-empty.out" "complete_local_recovery_point=no"
|
||||
assert_line "$TEST_ROOT/production-empty.out" "backup_risk_classification=no_proven_recovery_point"
|
||||
assert_line "$TEST_ROOT/production-empty.out" "transition_recovery_point_action=request_manual_run"
|
||||
pass "生产无完整本地恢复点时升级风险与过渡恢复点建议"
|
||||
|
||||
PRODUCTION_CORRUPT="$TEST_ROOT/production-corrupt"
|
||||
make_production_layout "$PRODUCTION_CORRUPT"
|
||||
make_batch "$PRODUCTION_CORRUPT/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
printf 'tamper' >>"$PRODUCTION_CORRUPT/backups/hyb-oa-backup-auto-20260828T000000Z/database.dump"
|
||||
run_production "$PRODUCTION_CORRUPT" "$TEST_ROOT/production-corrupt.out"
|
||||
assert_line "$TEST_ROOT/production-corrupt.out" "complete_local_recovery_point=no"
|
||||
assert_line "$TEST_ROOT/production-corrupt.out" "invalid_local_batch_count=1"
|
||||
pass "manifest 大小或 SHA 篡改不得冒充完整本地恢复点"
|
||||
|
||||
PRODUCTION_MALFORMED="$TEST_ROOT/production-malformed"
|
||||
make_production_layout "$PRODUCTION_MALFORMED"
|
||||
make_batch "$PRODUCTION_MALFORMED/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
printf '%s\n' '[]' >"$PRODUCTION_MALFORMED/backups/hyb-oa-backup-auto-20260828T000000Z/manifest.json"
|
||||
run_production "$PRODUCTION_MALFORMED" "$TEST_ROOT/production-malformed.out"
|
||||
assert_line "$TEST_ROOT/production-malformed.out" "complete_local_recovery_point=no"
|
||||
assert_line "$TEST_ROOT/production-malformed.out" "invalid_local_batch_count=1"
|
||||
pass "畸形 manifest 安全归类为无效批次且不输出 traceback"
|
||||
|
||||
PRODUCTION_UNREADABLE="$TEST_ROOT/production-unreadable"
|
||||
make_production_layout "$PRODUCTION_UNREADABLE"
|
||||
make_batch "$PRODUCTION_UNREADABLE/backups/hyb-oa-backup-auto-20260828T000000Z" "auto-20260828T000000Z"
|
||||
chmod 000 "$PRODUCTION_UNREADABLE/backups/hyb-oa-backup-auto-20260828T000000Z/manifest.json"
|
||||
if [[ "$(id -u)" == 0 ]] && command -v runuser >/dev/null 2>&1 && id nobody >/dev/null 2>&1; then
|
||||
chmod 0755 "$TEST_ROOT" "$PRODUCTION_UNREADABLE" "$PRODUCTION_UNREADABLE/backups" "$PRODUCTION_UNREADABLE/state" \
|
||||
"$PRODUCTION_UNREADABLE/backups/hyb-oa-backup-auto-20260828T000000Z"
|
||||
if runuser -u nobody -- python3 -B "$INSPECTOR" --now-epoch 1787875200 production \
|
||||
--backup-root "$PRODUCTION_UNREADABLE/backups" \
|
||||
--state-dir "$PRODUCTION_UNREADABLE/state" \
|
||||
--wrapper-script "$ROOT_DIR/scripts/run-automatic-backup.sh" \
|
||||
--backup-script "$ROOT_DIR/scripts/backup-all.sh" >"$TEST_ROOT/production-unreadable.out"; then
|
||||
printf '[FAIL] unreadable manifest unexpectedly passed\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
if run_production "$PRODUCTION_UNREADABLE" "$TEST_ROOT/production-unreadable.out"; then
|
||||
printf '[FAIL] unreadable manifest unexpectedly passed\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
chmod 0600 "$PRODUCTION_UNREADABLE/backups/hyb-oa-backup-auto-20260828T000000Z/manifest.json"
|
||||
assert_line "$TEST_ROOT/production-unreadable.out" "probe_category=access_denied"
|
||||
pass "访问拒绝保持为 unknown/ACCESS_DENIED 而不是伪装成无有效备份"
|
||||
|
||||
assert_line "$TEST_ROOT/production-valid.out" "retained_after_transfer_failure=yes"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "pruned_after_offhost_success=yes"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "generation_temp_cleanup_on_failure=yes"
|
||||
assert_line "$TEST_ROOT/production-valid.out" "cleanup_contract_source_verified=yes"
|
||||
pass "生产脚本顺序机械证明失败保留、成功后 prune 与生成临时清理"
|
||||
|
||||
SOURCE_MUTATION="$TEST_ROOT/source-mutation"
|
||||
cp "$ROOT_DIR/scripts/run-automatic-backup.sh" "$SOURCE_MUTATION"
|
||||
python3 - "$SOURCE_MUTATION" <<'PY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
path = Path(sys.argv[1])
|
||||
text = path.read_text(encoding="utf-8")
|
||||
transfer = text.index("capture_command offhost_transfer rsync")
|
||||
text = text[:transfer] + 'prune_local_backups "$batch_name"\n ' + text[transfer:]
|
||||
path.write_text(text, encoding="utf-8")
|
||||
PY
|
||||
python3 -B "$INSPECTOR" --now-epoch 1787875200 production \
|
||||
--backup-root "$PRODUCTION_VALID/backups" \
|
||||
--state-dir "$PRODUCTION_VALID/state" \
|
||||
--wrapper-script "$SOURCE_MUTATION" \
|
||||
--backup-script "$ROOT_DIR/scripts/backup-all.sh" >"$TEST_ROOT/source-mutation.out"
|
||||
assert_line "$TEST_ROOT/source-mutation.out" "retained_after_transfer_failure=no"
|
||||
assert_line "$TEST_ROOT/source-mutation.out" "pruned_after_offhost_success=no"
|
||||
pass "源码顺序漂移会关闭失败保留与成功后 prune 结论"
|
||||
|
||||
for code_category in \
|
||||
23:partial_transfer_error \
|
||||
30:transport_timeout \
|
||||
12:protocol_stream_error \
|
||||
11:file_io_error \
|
||||
255:ssh_session_error; do
|
||||
code="${code_category%%:*}"
|
||||
category="${code_category#*:}"
|
||||
fixture="$TEST_ROOT/rsync-$code"
|
||||
make_production_layout "$fixture"
|
||||
write_transfer_failure_state "$fixture"
|
||||
printf 'rsync error: synthetic redacted fixture (code %s) at fixture.c(1)\n' "$code" >"$fixture/state/last-error.log"
|
||||
run_production "$fixture" "$TEST_ROOT/rsync-$code.out"
|
||||
assert_line "$TEST_ROOT/rsync-$code.out" "transfer_exit_code=$code"
|
||||
assert_line "$TEST_ROOT/rsync-$code.out" "transfer_failure_category=$category"
|
||||
done
|
||||
pass "rsync 23/30/12/11/255 均映射为稳定脱敏类别"
|
||||
|
||||
PRODUCTION_UNKNOWN="$TEST_ROOT/production-unknown"
|
||||
make_production_layout "$PRODUCTION_UNKNOWN"
|
||||
write_transfer_failure_state "$PRODUCTION_UNKNOWN"
|
||||
printf 'unstructured synthetic transport failure\n' >"$PRODUCTION_UNKNOWN/state/last-error.log"
|
||||
run_production "$PRODUCTION_UNKNOWN" "$TEST_ROOT/production-unknown.out"
|
||||
assert_line "$TEST_ROOT/production-unknown.out" "transfer_exit_code=NOT_AVAILABLE"
|
||||
assert_line "$TEST_ROOT/production-unknown.out" "transfer_failure_category=not_available"
|
||||
pass "原始 rsync 码不存在时安全失败为 NOT_AVAILABLE"
|
||||
|
||||
PRODUCTION_PHASE_MISMATCH="$TEST_ROOT/production-phase-mismatch"
|
||||
make_production_layout "$PRODUCTION_PHASE_MISMATCH"
|
||||
printf '%s\n' '{"findings":[{"detail":"phase_offhost_finalize_failed"}]}' >"$PRODUCTION_PHASE_MISMATCH/state/last-run.json"
|
||||
printf 'rsync error: stale synthetic fixture (code 23) at fixture.c(1)\n' >"$PRODUCTION_PHASE_MISMATCH/state/last-error.log"
|
||||
run_production "$PRODUCTION_PHASE_MISMATCH" "$TEST_ROOT/production-phase-mismatch.out"
|
||||
assert_line "$TEST_ROOT/production-phase-mismatch.out" "last_backup_failure_phase=offhost_finalize"
|
||||
assert_line "$TEST_ROOT/production-phase-mismatch.out" "transfer_exit_code=NOT_AVAILABLE"
|
||||
assert_line "$TEST_ROOT/production-phase-mismatch.out" "transfer_exit_code_source=NOT_AVAILABLE"
|
||||
pass "私有诊断只在安全摘要明确为 offhost_transfer 时绑定 rsync 原始码"
|
||||
|
||||
ARM_SIMILAR="$TEST_ROOT/arm-similar"
|
||||
mkdir -p "$ARM_SIMILAR/inbox" "$ARM_SIMILAR/archive"
|
||||
make_batch "$ARM_SIMILAR/inbox/hyb-oa-backup-auto-20260827T000000Z.partial" "auto-20260827T000000Z" 1000 1000
|
||||
make_batch "$ARM_SIMILAR/inbox/hyb-oa-backup-auto-20260828T000000Z.partial" "auto-20260828T000000Z" 1040 1000
|
||||
rm "$ARM_SIMILAR/inbox/hyb-oa-backup-auto-20260827T000000Z.partial/storage.tar.gz"
|
||||
rm "$ARM_SIMILAR/inbox/hyb-oa-backup-auto-20260828T000000Z.partial/storage.tar.gz"
|
||||
python3 -B "$INSPECTOR" --now-epoch 1787875200 arm --inbox-root "$ARM_SIMILAR/inbox" --archive-root "$ARM_SIMILAR/archive" >"$TEST_ROOT/arm-similar.out"
|
||||
assert_line "$TEST_ROOT/arm-similar.out" "arm_partial_count=2"
|
||||
assert_line "$TEST_ROOT/arm-similar.out" "arm_partial_stop_offset_class=similar"
|
||||
assert_line "$TEST_ROOT/arm-similar.out" "arm_partial_incomplete_contract_count=2"
|
||||
pass "两个 partial 接近停止偏移按预先冻结阈值分类为 similar"
|
||||
|
||||
ARM_DIFFERENT="$TEST_ROOT/arm-different"
|
||||
mkdir -p "$ARM_DIFFERENT/inbox" "$ARM_DIFFERENT/archive"
|
||||
make_batch "$ARM_DIFFERENT/inbox/hyb-oa-backup-auto-20260827T000000Z.partial" "auto-20260827T000000Z" 1000 1000
|
||||
make_batch "$ARM_DIFFERENT/inbox/hyb-oa-backup-auto-20260828T000000Z.partial" "auto-20260828T000000Z" 70000000 1000
|
||||
rm "$ARM_DIFFERENT/inbox/hyb-oa-backup-auto-20260827T000000Z.partial/storage.tar.gz"
|
||||
rm "$ARM_DIFFERENT/inbox/hyb-oa-backup-auto-20260828T000000Z.partial/storage.tar.gz"
|
||||
python3 -B "$INSPECTOR" --now-epoch 1787875200 arm --inbox-root "$ARM_DIFFERENT/inbox" --archive-root "$ARM_DIFFERENT/archive" >"$TEST_ROOT/arm-different.out"
|
||||
assert_line "$TEST_ROOT/arm-different.out" "arm_partial_stop_offset_class=different"
|
||||
pass "两个 partial 明显不同停止偏移分类为 different"
|
||||
|
||||
python3 -B - "$INSPECTOR" <<'PY'
|
||||
import importlib.util
|
||||
import sys
|
||||
|
||||
spec = importlib.util.spec_from_file_location("backup_transfer_incident", sys.argv[1])
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
|
||||
def sample(total_bytes):
|
||||
return module.BatchResult(False, total_bytes, None, None, total_bytes)
|
||||
|
||||
cases = [
|
||||
("现场量级的 2.68MB 与 0 字节必须不同", [sample(0), sample(2_682_168)], "different"),
|
||||
("大文件差异同时落在 5% 与 64MiB 内才接近", [sample(1_000_000_000), sample(1_040_000_000)], "similar"),
|
||||
("即使落在 64MiB 内但超过 5% 仍不同", [sample(10_000_000), sample(10_600_000)], "different"),
|
||||
("即使落在 5% 内但超过 64MiB 仍不同", [sample(2_000_000_000), sample(2_080_000_000)], "different"),
|
||||
]
|
||||
for label, samples, expected in cases:
|
||||
actual = module.stopping_class(samples)
|
||||
if actual != expected:
|
||||
raise SystemExit(f"{label}: expected {expected}, got {actual}")
|
||||
PY
|
||||
pass "停止偏移分类同时满足相对与绝对阈值并覆盖现场 2.68MB/0 字节回归"
|
||||
|
||||
ARM_COMPLETE="$TEST_ROOT/arm-complete"
|
||||
mkdir -p "$ARM_COMPLETE/inbox" "$ARM_COMPLETE/archive"
|
||||
make_batch "$ARM_COMPLETE/inbox/hyb-oa-backup-auto-20260828T000000Z.partial" "auto-20260828T000000Z"
|
||||
make_batch "$ARM_COMPLETE/archive/hyb-oa-backup-auto-20260827T000000Z" "auto-20260827T000000Z"
|
||||
python3 -B "$INSPECTOR" --now-epoch 1787875200 arm --inbox-root "$ARM_COMPLETE/inbox" --archive-root "$ARM_COMPLETE/archive" >"$TEST_ROOT/arm-complete.out"
|
||||
assert_line "$TEST_ROOT/arm-complete.out" "arm_partial_complete_contract_count=1"
|
||||
assert_line "$TEST_ROOT/arm-complete.out" "arm_valid_atomic_archive_count=1"
|
||||
pass "完整 partial 与正式原子归档分别按同一三文件身份和哈希合同验证"
|
||||
|
||||
PERMISSION_FIXTURE="$TEST_ROOT/permission"
|
||||
mkdir -p "$PERMISSION_FIXTURE/inbox" "$PERMISSION_FIXTURE/archive"
|
||||
chmod 000 "$PERMISSION_FIXTURE/inbox"
|
||||
if command -v runuser >/dev/null 2>&1 && id nobody >/dev/null 2>&1 && [[ "$(id -u)" == 0 ]]; then
|
||||
if runuser -u nobody -- python3 -B "$INSPECTOR" arm --inbox-root "$PERMISSION_FIXTURE/inbox" --archive-root "$PERMISSION_FIXTURE/archive" >"$TEST_ROOT/permission.out"; then
|
||||
printf '[FAIL] permission denial unexpectedly passed\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
assert_line "$TEST_ROOT/permission.out" "probe_category=access_denied"
|
||||
else
|
||||
chmod 0700 "$PERMISSION_FIXTURE/inbox"
|
||||
ln -s "$PERMISSION_FIXTURE/inbox" "$PERMISSION_FIXTURE/inbox-link"
|
||||
if python3 -B "$INSPECTOR" arm --inbox-root "$PERMISSION_FIXTURE/inbox-link" --archive-root "$PERMISSION_FIXTURE/archive" >"$TEST_ROOT/permission.out"; then
|
||||
printf '[FAIL] invalid root unexpectedly passed\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
assert_line "$TEST_ROOT/permission.out" "probe_category=root_invalid"
|
||||
fi
|
||||
chmod 0700 "$PERMISSION_FIXTURE/inbox"
|
||||
pass "不可读或不安全根目录明确拒绝且不伪装为不存在"
|
||||
|
||||
if rg -n --fixed-strings \
|
||||
-e "$TEST_ROOT" \
|
||||
-e "auto-20260828T000000Z" \
|
||||
-e "database.dump" \
|
||||
-e "storage.tar.gz" \
|
||||
-e "unstructured synthetic transport failure" \
|
||||
"$TEST_ROOT"/*.out >/dev/null; then
|
||||
printf '[FAIL] redacted output leaked fixture identity, path, artifact name, or diagnostic text\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
pass "stdout allowlist 不含路径、run id、产物名或原始诊断"
|
||||
|
||||
PYTHONPYCACHEPREFIX="$TEST_ROOT/pycache" python3 -m py_compile "$INSPECTOR"
|
||||
bash -n "$ROOT_DIR/scripts/qa/run-backup-transfer-incident-contract.sh"
|
||||
pass "Python 与 Bash 语法通过"
|
||||
|
||||
cleanup
|
||||
[[ ! -e "$TEST_ROOT" ]]
|
||||
TEST_ROOT=""
|
||||
trap - EXIT
|
||||
pass "合成 fixture 与 Python cache 零残留"
|
||||
|
||||
printf '[PASS] backup transfer incident contract: %s checks\n' "$PASS_COUNT"
|
||||
文件差异内容过多而无法显示
加载差异
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
import base64
|
||||
import hashlib
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) != 2:
|
||||
return 2
|
||||
|
||||
root = os.path.realpath(sys.argv[1])
|
||||
total = 0
|
||||
mismatch = 0
|
||||
s3_count = 0
|
||||
|
||||
for raw in sys.stdin:
|
||||
raw = raw.rstrip("\n")
|
||||
if not raw:
|
||||
continue
|
||||
parts = raw.split("\t")
|
||||
if len(parts) != 3:
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
total += 1
|
||||
try:
|
||||
storage_path = base64.b64decode(parts[0], validate=True).decode("utf-8")
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
if storage_path.startswith("s3:"):
|
||||
s3_count += 1
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
pure_path = pathlib.PurePosixPath(storage_path)
|
||||
if not storage_path or pure_path.is_absolute() or ".." in pure_path.parts:
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
target = os.path.realpath(os.path.join(root, *pure_path.parts))
|
||||
if not target.startswith(root + os.sep) or not os.path.isfile(target):
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
expected_size, expected_sha = parts[1], parts[2].lower()
|
||||
if expected_size and str(os.path.getsize(target)) != expected_size:
|
||||
mismatch += 1
|
||||
continue
|
||||
|
||||
if expected_sha:
|
||||
digest = hashlib.sha256()
|
||||
with open(target, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
if digest.hexdigest() != expected_sha:
|
||||
mismatch += 1
|
||||
|
||||
print(total, mismatch, s3_count)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1079,8 +1079,7 @@ async function invoicePoolRoutes(fastify) {
|
||||
const isOwn =
|
||||
reimbursement.owner_id === request.employee.id ||
|
||||
reimbursement.submitter_id === request.employee.id;
|
||||
const canViewAll = fastify.hasPermission(request.employee, "expense.view_all");
|
||||
if (!isOwn && !canViewAll) {
|
||||
if (!isOwn) {
|
||||
throw new AppError({
|
||||
code: "PERM_001",
|
||||
message: "无权限",
|
||||
@@ -1591,8 +1590,7 @@ async function invoicePoolRoutes(fastify) {
|
||||
const isOwn =
|
||||
reimbursement.owner_id === request.employee.id ||
|
||||
reimbursement.submitter_id === request.employee.id;
|
||||
const canViewAll = fastify.hasPermission(request.employee, "expense.view_all");
|
||||
if (!isOwn && !canViewAll) {
|
||||
if (!isOwn) {
|
||||
throw new AppError({
|
||||
code: "PERM_001",
|
||||
message: "无权限",
|
||||
|
||||
@@ -57,6 +57,7 @@ const {
|
||||
buildApprovalWorkflowSnapshot,
|
||||
getWorkflowStepsSnapshot,
|
||||
resolveActionableStep,
|
||||
resolveWorkflowStepDisplayName,
|
||||
toMoneyString,
|
||||
} = require("../utils/workflow");
|
||||
|
||||
@@ -99,6 +100,9 @@ const REIMBURSEMENT_ATTACHMENT_PREVIEW_BODY_SCHEMA = z
|
||||
.object({ reason: z.string().trim().max(200).optional() })
|
||||
.strict();
|
||||
const REIMBURSEMENT_MATERIAL_PREVIEW_BODY_SCHEMA = REIMBURSEMENT_ATTACHMENT_PREVIEW_BODY_SCHEMA;
|
||||
const REIMBURSEMENT_PROJECT_LINK_BODY_SCHEMA = z
|
||||
.object({ projectId: z.string().uuid().nullable() })
|
||||
.strict();
|
||||
|
||||
function resolveSignedFileOrigin(config, request) {
|
||||
const configured = String(config.appOrigin ?? "")
|
||||
@@ -355,11 +359,12 @@ function isOrdinalApprovalLabel(value) {
|
||||
}
|
||||
|
||||
function resolveApprovalStageLabel(task) {
|
||||
const stepName = String(task?.step_name ?? "").trim();
|
||||
const stepName = resolveWorkflowStepDisplayName(task);
|
||||
const stepId = String(task?.step_id ?? "").trim();
|
||||
const isTechnicalIdentifier = /^[a-z][a-z0-9]*(?:[_-][a-z0-9]+)+$/i.test(stepName);
|
||||
const isGenericLabel = ["审批", "审核", "复核"].includes(stepName);
|
||||
if (stepName && stepName !== stepId && !isOrdinalApprovalLabel(stepName) && !isTechnicalIdentifier && !isGenericLabel) {
|
||||
if (stepName && stepName !== stepId && !isOrdinalApprovalLabel(stepName)
|
||||
&& !isTechnicalIdentifier && !isGenericLabel) {
|
||||
return stepName;
|
||||
}
|
||||
|
||||
@@ -581,16 +586,12 @@ function buildProcessingProgress({ reimbursement, workflowInstance, paymentBatch
|
||||
};
|
||||
}
|
||||
|
||||
function ensureApplicantOrAdmin(request, reimbursement) {
|
||||
if (request.employee?.roleCodes?.includes("admin")) {
|
||||
return;
|
||||
}
|
||||
|
||||
const isApplicant =
|
||||
function ensureReimbursementParticipant(request, reimbursement) {
|
||||
const isParticipant =
|
||||
reimbursement.owner_id === request.employee.id ||
|
||||
reimbursement.submitter_id === request.employee.id;
|
||||
|
||||
if (!isApplicant) {
|
||||
if (!isParticipant) {
|
||||
throw new AppError({
|
||||
code: "PERM_001",
|
||||
message: "无权限",
|
||||
@@ -678,6 +679,19 @@ function parseReimbursementUpdateBody(body) {
|
||||
return parsed.data;
|
||||
}
|
||||
|
||||
function parseReimbursementProjectLinkBody(body) {
|
||||
const parsed = REIMBURSEMENT_PROJECT_LINK_BODY_SCHEMA.safeParse(body ?? {});
|
||||
if (!parsed.success) {
|
||||
throw new AppError({
|
||||
code: "VALID_001",
|
||||
message: "参数校验失败",
|
||||
statusCode: 400,
|
||||
details: parsed.error.flatten(),
|
||||
});
|
||||
}
|
||||
return parsed.data;
|
||||
}
|
||||
|
||||
function parseReimbursementAttachmentDeleteBody(body) {
|
||||
const parsed = REIMBURSEMENT_ATTACHMENT_DELETE_BODY_SCHEMA.safeParse(body ?? {});
|
||||
if (!parsed.success) {
|
||||
@@ -2756,7 +2770,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
await ensureCutoverAllowed({
|
||||
prisma: tx,
|
||||
config: fastify.config,
|
||||
@@ -2859,7 +2873,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
|
||||
if (reimbursement.status !== "draft") {
|
||||
throw new AppError({
|
||||
@@ -2926,7 +2940,7 @@ async function reimbursementRoutes(fastify) {
|
||||
});
|
||||
}
|
||||
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
code: "VALID_001",
|
||||
@@ -3138,7 +3152,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
code: "VALID_001",
|
||||
@@ -3615,7 +3629,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
@@ -3743,7 +3757,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
@@ -3864,7 +3878,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
@@ -3908,6 +3922,14 @@ async function reimbursementRoutes(fastify) {
|
||||
},
|
||||
async (request, reply) => {
|
||||
const reimbursementId = parseReimbursementId(request.params?.id);
|
||||
const reimbursement = await fastify.prisma.reimbursementForm.findUnique({
|
||||
where: { id: reimbursementId },
|
||||
select: { id: true, owner_id: true, submitter_id: true },
|
||||
});
|
||||
if (!reimbursement) {
|
||||
throw new AppError({ code: "SYS_404", message: "Not Found", statusCode: 404 });
|
||||
}
|
||||
ensureOwnOrCanViewAll(fastify, request, reimbursement);
|
||||
const precheck = await resolveEntityPrecheck({
|
||||
prisma: fastify.prisma,
|
||||
config: fastify.config,
|
||||
@@ -3946,7 +3968,7 @@ async function reimbursementRoutes(fastify) {
|
||||
}
|
||||
|
||||
const reimbursement = reimbursementRows[0];
|
||||
ensureApplicantOrAdmin(request, reimbursement);
|
||||
ensureReimbursementParticipant(request, reimbursement);
|
||||
|
||||
if (!["draft", "rejected"].includes(reimbursement.status)) {
|
||||
throw new AppError({
|
||||
@@ -4484,7 +4506,7 @@ async function reimbursementRoutes(fastify) {
|
||||
fastify.post(
|
||||
"/reimbursements/:id/link-project",
|
||||
{
|
||||
preHandler: fastify.authenticate,
|
||||
preHandler: fastify.requirePermission("expense.edit_own"),
|
||||
},
|
||||
async (request, reply) => {
|
||||
const gate = await evaluateFeatureFlag({
|
||||
@@ -4498,7 +4520,7 @@ async function reimbursementRoutes(fastify) {
|
||||
throw new AppError({ code: "VALID_001", message: "参数校验失败", statusCode: 409, details: { featureFlag: "project_linking_v1_disabled", reason: gate.reason } });
|
||||
}
|
||||
const reimbursementId = parseReimbursementId(request.params?.id);
|
||||
const body = parseBody(z.object({ projectId: z.string().uuid().nullable() }).strict(), request.body);
|
||||
const body = parseReimbursementProjectLinkBody(request.body);
|
||||
const current = await fastify.prisma.reimbursementForm.findUnique({
|
||||
where: { id: reimbursementId },
|
||||
select: { id: true, owner_id: true, submitter_id: true, project_id: true, status: true },
|
||||
@@ -4506,7 +4528,15 @@ async function reimbursementRoutes(fastify) {
|
||||
if (!current) {
|
||||
throw new AppError({ code: "SYS_404", message: "Not Found", statusCode: 404 });
|
||||
}
|
||||
ensureOwnOrCanViewAll(fastify, request, current);
|
||||
ensureReimbursementParticipant(request, current);
|
||||
if (!["draft", "rejected"].includes(current.status)) {
|
||||
throw new AppError({
|
||||
code: "VALID_001",
|
||||
message: "参数校验失败",
|
||||
statusCode: 409,
|
||||
details: { reimbursement: "invalid_status_for_project_link" },
|
||||
});
|
||||
}
|
||||
const updated = await fastify.prisma.reimbursementForm.update({
|
||||
where: { id: reimbursementId },
|
||||
data: { project_id: body.projectId ?? null, updated_by: request.employee.id },
|
||||
|
||||
@@ -8,6 +8,7 @@ const {
|
||||
getWorkflowStepsSnapshot,
|
||||
isBusinessApprovalRoleCode,
|
||||
resolveActionableStep,
|
||||
resolveWorkflowStepDisplayName,
|
||||
toMoneyString,
|
||||
} = require("../utils/workflow");
|
||||
const { consumeReservedLeave, releaseReservedLeave, formatDateOnly } = require("../services/leave");
|
||||
@@ -336,6 +337,7 @@ function isoOrNull(value) {
|
||||
|
||||
function mapTaskSummary(task, allowedActions = []) {
|
||||
const businessSummary = buildTaskBusinessSummary(task);
|
||||
const stepDisplayName = resolveWorkflowStepDisplayName(task);
|
||||
const workflowInstance = task.workflow_instance
|
||||
? {
|
||||
entity_type: task.workflow_instance.entity_type,
|
||||
@@ -348,7 +350,7 @@ function mapTaskSummary(task, allowedActions = []) {
|
||||
id: task.id,
|
||||
instance_id: task.instance_id,
|
||||
step_id: task.step_id,
|
||||
step_name: task.step_name,
|
||||
step_name: stepDisplayName,
|
||||
assignee_id: task.assignee_id,
|
||||
assignee_role: task.assignee_role,
|
||||
status: task.status,
|
||||
@@ -359,7 +361,7 @@ function mapTaskSummary(task, allowedActions = []) {
|
||||
|
||||
instanceId: task.instance_id,
|
||||
stepId: task.step_id,
|
||||
stepName: task.step_name,
|
||||
stepName: stepDisplayName,
|
||||
assigneeId: task.assignee_id,
|
||||
assigneeRole: task.assignee_role,
|
||||
createdAt: isoOrNull(task.created_at),
|
||||
@@ -531,7 +533,7 @@ function buildWorkflowTaskSummaryPayload(tasks, scope = "mine", now = new Date()
|
||||
}
|
||||
|
||||
incrementSummaryBucket(byEntityType, task.workflow_instance?.entity_type ?? "unknown", task, now);
|
||||
incrementSummaryBucket(byStep, task.step_name ?? task.step_id ?? "unknown", task, now);
|
||||
incrementSummaryBucket(byStep, resolveWorkflowStepDisplayName(task) || task.step_id || "unknown", task, now);
|
||||
incrementSummaryBucket(byAssigneeRole, task.assignee_role ?? (task.assignee_id ? "指定审批人" : "未分配"), task, now);
|
||||
}
|
||||
|
||||
@@ -2433,7 +2435,7 @@ async function workflowRoutes(fastify) {
|
||||
id: task.id,
|
||||
instanceId: task.instance_id,
|
||||
stepId: task.step_id,
|
||||
stepName: task.step_name,
|
||||
stepName: resolveWorkflowStepDisplayName(task),
|
||||
assigneeId: task.assignee_id,
|
||||
assigneeRole: task.assignee_role,
|
||||
status: task.status,
|
||||
|
||||
@@ -24,15 +24,14 @@ function materialCollectionHash(materials, mergerVersion) {
|
||||
|
||||
async function normalizeImageForPdf(source, limits) {
|
||||
const image = sharp(source, { failOn: "error", limitInputPixels: limits.maxImagePixels }).rotate();
|
||||
const metadata = await image.metadata();
|
||||
if (!metadata.width || !metadata.height || metadata.width * metadata.height > limits.maxImagePixels) {
|
||||
throw new Error("material_preview_image_invalid");
|
||||
}
|
||||
const normalized = await image
|
||||
const { data: normalized, info } = await image
|
||||
.flatten({ background: "#ffffff" })
|
||||
.jpeg({ quality: 90, chromaSubsampling: "4:4:4", mozjpeg: false })
|
||||
.toBuffer();
|
||||
return { buffer: normalized, width: metadata.width, height: metadata.height };
|
||||
.toBuffer({ resolveWithObject: true });
|
||||
if (!info.width || !info.height || info.width * info.height > limits.maxImagePixels) {
|
||||
throw new Error("material_preview_image_invalid");
|
||||
}
|
||||
return { buffer: normalized, width: info.width, height: info.height };
|
||||
}
|
||||
|
||||
function fitPage(width, height) {
|
||||
|
||||
@@ -3,7 +3,7 @@ const path = require("node:path");
|
||||
const { Worker } = require("node:worker_threads");
|
||||
const { materialCollectionHash } = require("./reimbursementMaterialPdf");
|
||||
|
||||
const MATERIAL_PREVIEW_MERGER_VERSION = "pdf-v1";
|
||||
const MATERIAL_PREVIEW_MERGER_VERSION = "pdf-v2";
|
||||
const MATERIAL_PREVIEW_LIMITS = Object.freeze({
|
||||
maxMaterials: 20,
|
||||
maxSourceBytes: 50 * 1024 * 1024,
|
||||
|
||||
+14
-1
@@ -107,13 +107,25 @@ function getRoleCodeFromStep(step) {
|
||||
return rule?.type === "role" && typeof rule.value === "string" ? rule.value : null;
|
||||
}
|
||||
|
||||
function resolveWorkflowStepDisplayName(step) {
|
||||
const stepName = String(step?.step_name ?? step?.stepName ?? step?.name ?? "").trim();
|
||||
const assigneeRole = String(step?.assignee_role ?? step?.assigneeRole ?? "").toLowerCase();
|
||||
const isLegacyOwnerFallback = /^负责人本人申请兜底\s*[::]/u.test(stepName);
|
||||
|
||||
if (isLegacyOwnerFallback && ["finance", "finance_mgr"].includes(assigneeRole)) {
|
||||
return "财务复核";
|
||||
}
|
||||
|
||||
return stepName;
|
||||
}
|
||||
|
||||
function buildRoleApprovalStep(roleCode, index, sourceStep, reason) {
|
||||
const suffix = reason === "owner_self_fallback" ? "_owner_self_fallback" : "";
|
||||
return {
|
||||
id: `approval_${index + 1}_${roleCode}${suffix}`,
|
||||
name:
|
||||
reason === "owner_self_fallback"
|
||||
? `负责人本人申请兜底:${roleCode}`
|
||||
? "财务复核"
|
||||
: sourceStep?.name ?? `审批节点${index + 1}`,
|
||||
type: sourceStep?.type ?? "approval",
|
||||
assignee_rule: { type: "role", value: roleCode },
|
||||
@@ -565,5 +577,6 @@ module.exports = {
|
||||
resolveActionableStep,
|
||||
resolveAssigneeFromRule,
|
||||
resolveDepartmentManagerAssignee,
|
||||
resolveWorkflowStepDisplayName,
|
||||
toMoneyString,
|
||||
};
|
||||
|
||||
@@ -87,7 +87,7 @@ function buildPrisma({
|
||||
return { prisma, state };
|
||||
}
|
||||
|
||||
async function buildApp({ prisma, permissionCodes = ["travel.payment.manage"] }) {
|
||||
async function buildApp({ prisma, permissionCodes = ["travel.payment.manage"], employeeId = "actor-1" }) {
|
||||
const app = Fastify();
|
||||
|
||||
app.decorate("prisma", prisma);
|
||||
@@ -101,7 +101,7 @@ async function buildApp({ prisma, permissionCodes = ["travel.payment.manage"] })
|
||||
if (!request.headers.authorization) {
|
||||
throw new AppError({ code: "AUTH_001", message: "未登录", statusCode: 401 });
|
||||
}
|
||||
request.employee = { id: "actor-1", permissionCodes };
|
||||
request.employee = { id: employeeId, permissionCodes };
|
||||
});
|
||||
app.decorate("hasPermission", (employee, permissionCode) => (employee?.permissionCodes ?? []).includes("*") || (employee?.permissionCodes ?? []).includes(permissionCode));
|
||||
app.decorate("requirePermission", (permissionCode) => async (request) => {
|
||||
@@ -133,6 +133,84 @@ async function buildApp({ prisma, permissionCodes = ["travel.payment.manage"] })
|
||||
}
|
||||
|
||||
describe("invoicePool travel-order bind routes", () => {
|
||||
it("rejects a view-all nonparticipant before binding a reimbursement invoice", async () => {
|
||||
const reimbursementId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
|
||||
const { prisma, state } = buildPrisma({
|
||||
invoice: {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
status: "available",
|
||||
linked_reimbursement_id: null,
|
||||
linked_item_id: null,
|
||||
travel_order_id: null,
|
||||
total_amount: "1683.00",
|
||||
},
|
||||
reimbursement: {
|
||||
id: reimbursementId,
|
||||
owner_id: "actor-1",
|
||||
submitter_id: "actor-1",
|
||||
status: "draft",
|
||||
reimbursement_date: new Date("2026-05-08T00:00:00.000Z"),
|
||||
},
|
||||
});
|
||||
const app = await buildApp({
|
||||
prisma,
|
||||
employeeId: "finance-viewer",
|
||||
permissionCodes: ["invoice.upload", "expense.view_all"],
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/invoice-pool/11111111-1111-4111-8111-111111111111/bind",
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: { reimbursementId },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(state.invoice.status).toBe("available");
|
||||
expect(prisma.invoicePool.update).not.toHaveBeenCalled();
|
||||
expect(prisma.reimbursementItem.create).not.toHaveBeenCalled();
|
||||
expect(prisma.auditLog.create).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("rejects a view-all nonparticipant before unbinding a reimbursement invoice", async () => {
|
||||
const reimbursementId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
|
||||
const { prisma, state } = buildPrisma({
|
||||
invoice: {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
status: "bound",
|
||||
linked_reimbursement_id: reimbursementId,
|
||||
linked_item_id: "item-1",
|
||||
travel_order_id: null,
|
||||
total_amount: "1683.00",
|
||||
},
|
||||
reimbursement: {
|
||||
id: reimbursementId,
|
||||
owner_id: "actor-1",
|
||||
submitter_id: "actor-1",
|
||||
status: "draft",
|
||||
},
|
||||
});
|
||||
const app = await buildApp({
|
||||
prisma,
|
||||
employeeId: "finance-viewer",
|
||||
permissionCodes: ["expense.view_all"],
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: "/api/invoice-pool/11111111-1111-4111-8111-111111111111/unbind",
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: {},
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(state.invoice.status).toBe("bound");
|
||||
expect(prisma.invoicePool.update).not.toHaveBeenCalled();
|
||||
expect(prisma.auditLog.create).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("adds invoice to reimbursement and auto-creates an item when no detail matches", async () => {
|
||||
const reimbursementId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
|
||||
const { prisma, state } = buildPrisma({
|
||||
|
||||
@@ -44,7 +44,13 @@ function buildMultipartRequest({ files, fields = {} }) {
|
||||
};
|
||||
}
|
||||
|
||||
async function buildApp({ reimbursementStatus = "draft", materialPreviewStore = null } = {}) {
|
||||
async function buildApp({
|
||||
reimbursementStatus = "draft",
|
||||
materialPreviewStore = null,
|
||||
employeeId = ACTOR_ID,
|
||||
roleCodes = ["employee"],
|
||||
permissionCodes = ["expense.edit_own", "expense.view_own"],
|
||||
} = {}) {
|
||||
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "hyb-oa-r111-reimbursement-attachments-"));
|
||||
const reimbursementId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
|
||||
const createdAttachments = [];
|
||||
@@ -160,9 +166,9 @@ async function buildApp({ reimbursementStatus = "draft", materialPreviewStore =
|
||||
throw new AppError({ code: "AUTH_001", message: "未登录", statusCode: 401 });
|
||||
}
|
||||
request.employee = {
|
||||
id: ACTOR_ID,
|
||||
roleCodes: ["employee"],
|
||||
permissionCodes: ["expense.edit_own", "expense.view_own"],
|
||||
id: employeeId,
|
||||
roleCodes,
|
||||
permissionCodes,
|
||||
};
|
||||
});
|
||||
app.decorate("hasPermission", (employee, permissionCode) => {
|
||||
@@ -202,6 +208,38 @@ async function buildApp({ reimbursementStatus = "draft", materialPreviewStore =
|
||||
}
|
||||
|
||||
describe("reimbursement attachment routes", () => {
|
||||
it("rejects an admin view-all nonparticipant before attachment side effects", async () => {
|
||||
const { app, prisma, tempDir, reimbursementId, createdAttachments } = await buildApp({
|
||||
employeeId: "finance-viewer",
|
||||
roleCodes: ["admin"],
|
||||
permissionCodes: ["expense.edit_own", "expense.view_all"],
|
||||
});
|
||||
|
||||
try {
|
||||
const requestPayload = buildMultipartRequest({
|
||||
files: [{
|
||||
filename: "synthetic.pdf",
|
||||
contentType: "application/pdf",
|
||||
content: "%PDF synthetic",
|
||||
}],
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${reimbursementId}/attachments`,
|
||||
headers: { authorization: "Bearer test", ...requestPayload.headers },
|
||||
payload: requestPayload.payload,
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(createdAttachments).toHaveLength(0);
|
||||
expect(prisma.reimbursementAttachment.create).not.toHaveBeenCalled();
|
||||
expect(prisma.auditLog.create).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
await app.close();
|
||||
await fs.rm(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("uploads five attachment materials with the production global one-file baseline", async () => {
|
||||
const { app, prisma, tempDir, reimbursementId, createdAttachments } = await buildApp();
|
||||
|
||||
|
||||
@@ -177,7 +177,26 @@ async function buildApp({
|
||||
paymentActors = [],
|
||||
} = {}) {
|
||||
const app = Fastify();
|
||||
const tx = {
|
||||
$queryRaw: vi.fn(async () => [reimbursement]),
|
||||
reimbursementForm: {
|
||||
update: vi.fn(),
|
||||
delete: vi.fn(),
|
||||
},
|
||||
invoicePool: {
|
||||
updateMany: vi.fn(),
|
||||
},
|
||||
reimbursementItem: {
|
||||
create: vi.fn(),
|
||||
update: vi.fn(),
|
||||
deleteMany: vi.fn(),
|
||||
},
|
||||
auditLog: {
|
||||
create: vi.fn(),
|
||||
},
|
||||
};
|
||||
const prisma = {
|
||||
$transaction: vi.fn(async (callback) => callback(tx)),
|
||||
reimbursementForm: {
|
||||
findUnique: vi.fn(async ({ where }) => {
|
||||
if (where.id !== REIMBURSEMENT_ID) {
|
||||
@@ -185,6 +204,22 @@ async function buildApp({
|
||||
}
|
||||
return reimbursement;
|
||||
}),
|
||||
update: vi.fn(async ({ data }) => ({
|
||||
id: REIMBURSEMENT_ID,
|
||||
project_id: data.project_id ?? null,
|
||||
project: data.project_id
|
||||
? { id: data.project_id, project_no: "P-001", name: "合成项目" }
|
||||
: null,
|
||||
})),
|
||||
},
|
||||
featureFlag: {
|
||||
findUnique: vi.fn(async () => ({
|
||||
flag_key: "project_linking_v1",
|
||||
override_mode: "inherit",
|
||||
rollout_type: "global",
|
||||
department_ids: [],
|
||||
version: 1,
|
||||
})),
|
||||
},
|
||||
workflowInstance: {
|
||||
findFirst: vi.fn(async () => latestWorkflowInstance),
|
||||
@@ -210,6 +245,7 @@ async function buildApp({
|
||||
app.decorate("config", {
|
||||
budgetControlMode: "warn",
|
||||
appTimezone: "Asia/Shanghai",
|
||||
projectLinkingEnabled: true,
|
||||
});
|
||||
app.decorate("authenticate", async (request) => {
|
||||
if (!request.headers.authorization) {
|
||||
@@ -255,7 +291,7 @@ async function buildApp({
|
||||
});
|
||||
|
||||
await app.register(reimbursementRoutes, { prefix: "/api" });
|
||||
return { app, prisma };
|
||||
return { app, prisma, tx };
|
||||
}
|
||||
|
||||
describe("GET /api/reimbursements/:id detail enhancements", () => {
|
||||
@@ -443,6 +479,59 @@ describe("GET /api/reimbursements/:id detail enhancements", () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("projects legacy owner-self fallback task names as finance review", async () => {
|
||||
const { app } = await buildApp({
|
||||
latestWorkflowInstance: {
|
||||
id: PENDING_INSTANCE_ID,
|
||||
current_step_id: "approval_1_finance_owner_self_fallback",
|
||||
current_status: "pending",
|
||||
created_at: new Date("2026-08-28T11:08:00.000Z"),
|
||||
started_at: new Date("2026-08-28T11:08:00.000Z"),
|
||||
context_data: {
|
||||
workflow_steps_snapshot: [{
|
||||
id: "approval_1_finance_owner_self_fallback",
|
||||
name: "负责人本人申请兜底:finance",
|
||||
type: "approval",
|
||||
assignee_rule: { type: "role", value: "finance" },
|
||||
}],
|
||||
},
|
||||
workflow_definition: { definition_json: { steps: [] } },
|
||||
workflow_tasks: [{
|
||||
id: "owner-fallback-task",
|
||||
step_id: "approval_1_finance_owner_self_fallback",
|
||||
step_name: "负责人本人申请兜底:finance",
|
||||
assignee_role: "finance",
|
||||
status: "pending",
|
||||
outcome: null,
|
||||
comment: null,
|
||||
created_at: new Date("2026-08-28T11:08:00.000Z"),
|
||||
completed_at: null,
|
||||
assignee_employee: { name: "财务人员" },
|
||||
completer_employee: null,
|
||||
}],
|
||||
},
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "GET",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}`,
|
||||
headers: { authorization: "Bearer test" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json().data.processingProgress.stages[1]).toEqual(
|
||||
expect.objectContaining({
|
||||
key: "approval_owner-fallback-task",
|
||||
label: "财务复核",
|
||||
state: "current",
|
||||
actorName: "财务人员",
|
||||
detail: "待财务复核处理",
|
||||
}),
|
||||
);
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("appends future snapshot approvals that do not have workflow tasks yet", async () => {
|
||||
const { app } = await buildApp({
|
||||
latestWorkflowInstance: {
|
||||
@@ -833,3 +922,101 @@ describe("GET /api/reimbursements/:id detail enhancements", () => {
|
||||
await app.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("POST /api/reimbursements/:id/link-project authorization", () => {
|
||||
const projectId = "66666666-6666-4666-8666-666666666666";
|
||||
|
||||
it("rejects a view-all nonparticipant without changing the project", async () => {
|
||||
const { app, prisma } = await buildApp({
|
||||
employeeId: "finance-viewer",
|
||||
permissionCodes: ["expense.view_all", "expense.edit_own"],
|
||||
reimbursement: buildReimbursementRecord({ status: "draft" }),
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/link-project`,
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: { projectId },
|
||||
});
|
||||
|
||||
expect(response.statusCode, response.body).toBe(403);
|
||||
expect(prisma.reimbursementForm.update).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it.each(["approved", "paid"])("rejects participant project changes in %s status", async (status) => {
|
||||
const { app, prisma } = await buildApp({
|
||||
employeeId: OWNER_ID,
|
||||
permissionCodes: ["expense.edit_own"],
|
||||
reimbursement: buildReimbursementRecord({ status }),
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/link-project`,
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: { projectId },
|
||||
});
|
||||
|
||||
expect(response.statusCode, response.body).toBe(409);
|
||||
expect(prisma.reimbursementForm.update).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("allows a draft participant to change the project", async () => {
|
||||
const { app, prisma } = await buildApp({
|
||||
employeeId: SUBMITTER_ID,
|
||||
permissionCodes: ["expense.edit_own"],
|
||||
reimbursement: buildReimbursementRecord({ status: "draft" }),
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/link-project`,
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: { projectId },
|
||||
});
|
||||
|
||||
expect(response.statusCode, response.body).toBe(200);
|
||||
expect(prisma.reimbursementForm.update).toHaveBeenCalledTimes(1);
|
||||
await app.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("reimbursement maintenance authorization", () => {
|
||||
const itemId = "77777777-7777-4777-8777-777777777777";
|
||||
const expenseTypeId = "88888888-8888-4888-8888-888888888888";
|
||||
|
||||
it.each([
|
||||
["PATCH", `/api/reimbursements/${REIMBURSEMENT_ID}`, { description: "合成修改" }, "expense.edit_own"],
|
||||
["DELETE", `/api/reimbursements/${REIMBURSEMENT_ID}`, undefined, "expense.delete_own"],
|
||||
["POST", `/api/reimbursements/${REIMBURSEMENT_ID}/items`, {
|
||||
expenseTypeId,
|
||||
expenseDate: "2026-04-10",
|
||||
amount: "10.00",
|
||||
description: "合成明细",
|
||||
}, "expense.edit_own"],
|
||||
["PATCH", `/api/reimbursements/${REIMBURSEMENT_ID}/items/${itemId}`, { amount: "20.00" }, "expense.edit_own"],
|
||||
["DELETE", `/api/reimbursements/${REIMBURSEMENT_ID}/items/${itemId}`, undefined, "expense.edit_own"],
|
||||
])("rejects a view-all nonparticipant for %s %s before mutations", async (method, url, payload, permission) => {
|
||||
const { app, tx } = await buildApp({
|
||||
employeeId: "finance-viewer",
|
||||
permissionCodes: [permission, "expense.view_all"],
|
||||
reimbursement: buildReimbursementRecord({ status: "draft" }),
|
||||
});
|
||||
const response = await app.inject({
|
||||
method,
|
||||
url,
|
||||
headers: { authorization: "Bearer test" },
|
||||
...(payload ? { payload } : {}),
|
||||
});
|
||||
|
||||
expect(response.statusCode, response.body).toBe(403);
|
||||
expect(tx.reimbursementForm.update).not.toHaveBeenCalled();
|
||||
expect(tx.reimbursementForm.delete).not.toHaveBeenCalled();
|
||||
expect(tx.invoicePool.updateMany).not.toHaveBeenCalled();
|
||||
expect(tx.reimbursementItem.create).not.toHaveBeenCalled();
|
||||
expect(tx.reimbursementItem.update).not.toHaveBeenCalled();
|
||||
expect(tx.reimbursementItem.deleteMany).not.toHaveBeenCalled();
|
||||
expect(tx.auditLog.create).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -53,7 +53,7 @@ async function buildApp(options = {}) {
|
||||
id: EMPLOYEE_ID,
|
||||
department_id: DEPARTMENT_ID,
|
||||
roleCodes: [],
|
||||
permissionCodes: ["expense.submit"],
|
||||
permissionCodes: ["expense.submit", "expense.view_own"],
|
||||
};
|
||||
const reimbursement = options.reimbursement ?? {
|
||||
id: REIMBURSEMENT_ID,
|
||||
@@ -176,6 +176,8 @@ async function buildApp(options = {}) {
|
||||
reimbursementForm: {
|
||||
findUnique: vi.fn(async () => ({
|
||||
id: reimbursement.id,
|
||||
owner_id: reimbursement.owner_id,
|
||||
submitter_id: reimbursement.submitter_id,
|
||||
form_number: reimbursement.form_number,
|
||||
total_amount: reimbursement.total_amount,
|
||||
owner: {
|
||||
@@ -239,6 +241,29 @@ describe("reimbursements routes expense standard integration", () => {
|
||||
resolveRecipientsForAssigneeMock.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
it("rejects an admin view-all nonparticipant before creating submission side effects", async () => {
|
||||
const { app, prisma, tx } = await buildApp({
|
||||
actor: {
|
||||
id: "finance-viewer",
|
||||
department_id: DEPARTMENT_ID,
|
||||
roleCodes: ["admin"],
|
||||
permissionCodes: ["expense.submit", "expense.view_all"],
|
||||
},
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/submit`,
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(tx.workflowInstance.create).not.toHaveBeenCalled();
|
||||
expect(tx.workflowTask.create).not.toHaveBeenCalled();
|
||||
expect(tx.reimbursementForm.update).not.toHaveBeenCalled();
|
||||
expect(tx.auditLog.create).not.toHaveBeenCalled();
|
||||
expect(prisma.reimbursementForm.findUnique).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("blocks travel reimbursement submit until the structured summary is complete", async () => {
|
||||
const { app, tx } = await buildApp({ travelItem: true });
|
||||
const response = await app.inject({ method: "POST", url: `/api/reimbursements/${REIMBURSEMENT_ID}/submit` });
|
||||
@@ -347,6 +372,47 @@ describe("reimbursements routes expense standard integration", () => {
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("rejects an ordinary nonparticipant before resolving reimbursement precheck", async () => {
|
||||
const { app } = await buildApp({
|
||||
actor: {
|
||||
id: "ordinary-nonparticipant",
|
||||
department_id: DEPARTMENT_ID,
|
||||
roleCodes: ["employee"],
|
||||
permissionCodes: ["expense.submit"],
|
||||
},
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/precheck`,
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(resolveEntityPrecheckMock).not.toHaveBeenCalled();
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("allows a view-all nonparticipant to read reimbursement precheck", async () => {
|
||||
resolveEntityPrecheckMock.mockResolvedValue(buildPrecheck());
|
||||
const { app } = await buildApp({
|
||||
actor: {
|
||||
id: "finance-viewer",
|
||||
department_id: DEPARTMENT_ID,
|
||||
roleCodes: ["finance"],
|
||||
permissionCodes: ["expense.submit", "expense.view_all"],
|
||||
},
|
||||
});
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/precheck`,
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(resolveEntityPrecheckMock).toHaveBeenCalledTimes(1);
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("blocks submit with 400 and standardViolations when block standard is exceeded", async () => {
|
||||
resolveEntityPrecheckMock.mockResolvedValue(
|
||||
buildPrecheck({
|
||||
|
||||
@@ -162,6 +162,7 @@ function buildPrisma({ processedTaskCount = 0 } = {}) {
|
||||
async function buildApp(options = {}) {
|
||||
const { prisma, tx, state } = buildPrisma(options);
|
||||
const permissionCodes = options.permissionCodes ?? ["expense.withdraw", "invoice.upload"];
|
||||
const employeeId = options.employeeId ?? OWNER_ID;
|
||||
const app = Fastify();
|
||||
|
||||
app.decorate("prisma", prisma);
|
||||
@@ -178,7 +179,7 @@ async function buildApp(options = {}) {
|
||||
throw new AppError({ code: "AUTH_001", message: "未登录", statusCode: 401 });
|
||||
}
|
||||
request.employee = {
|
||||
id: OWNER_ID,
|
||||
id: employeeId,
|
||||
department_id: "dept-1",
|
||||
roleCodes: [],
|
||||
permissionCodes,
|
||||
@@ -222,6 +223,28 @@ async function buildApp(options = {}) {
|
||||
}
|
||||
|
||||
describe("POST /api/reimbursements/:id/withdraw invoice release", () => {
|
||||
it("rejects a non-participant even when the employee has withdraw permission", async () => {
|
||||
const { app, tx, state } = await buildApp({ employeeId: "77777777-7777-4777-8777-777777777777" });
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/api/reimbursements/${REIMBURSEMENT_ID}/withdraw`,
|
||||
headers: { authorization: "Bearer test" },
|
||||
payload: { comment: "不应允许非参与人撤回" },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(response.json().error).toMatchObject({
|
||||
code: "PERM_001",
|
||||
details: { withdraw: "only_owner_or_submitter" },
|
||||
});
|
||||
expect(tx.reimbursementForm.update).not.toHaveBeenCalled();
|
||||
expect(tx.invoicePool.updateMany).not.toHaveBeenCalled();
|
||||
expect(state.reimbursement.status).toBe("reviewing");
|
||||
|
||||
await app.close();
|
||||
});
|
||||
|
||||
it("releases bound invoice on withdraw and allows rebinding to the draft reimbursement", async () => {
|
||||
const { app, tx, state } = await buildApp();
|
||||
|
||||
|
||||
@@ -122,7 +122,7 @@ describe("workflow travel helpers", () => {
|
||||
{
|
||||
id: "task-2",
|
||||
status: "pending",
|
||||
step_name: "财务审批",
|
||||
step_name: "负责人本人申请兜底:finance",
|
||||
assignee_role: "finance",
|
||||
created_at: new Date("2026-05-12T07:00:00.000Z"),
|
||||
due_date: new Date("2026-05-12T07:30:00.000Z"),
|
||||
@@ -148,5 +148,10 @@ describe("workflow travel helpers", () => {
|
||||
{ key: "purchase_request", pendingTotal: 1, overdueTotal: 1 },
|
||||
]),
|
||||
);
|
||||
expect(summary.byStep).toEqual(
|
||||
expect.arrayContaining([
|
||||
{ key: "财务复核", pendingTotal: 1, overdueTotal: 1 },
|
||||
]),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -103,7 +103,7 @@ describe("paymentProcessingProgress", () => {
|
||||
);
|
||||
const reimbursementAssembler = reimbursementsSource.slice(
|
||||
reimbursementsSource.indexOf("function buildProcessingProgress("),
|
||||
reimbursementsSource.indexOf("function ensureApplicantOrAdmin("),
|
||||
reimbursementsSource.indexOf("function ensureReimbursementParticipant("),
|
||||
);
|
||||
const paymentBatchAssembler = paymentBatchesSource.slice(
|
||||
paymentBatchesSource.indexOf("function buildPaymentBatchProcessingProgress("),
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
const { PDFDocument } = require("pdf-lib");
|
||||
const sharp = require("sharp");
|
||||
const { mergeMaterialsToPdf } = require("../../../src/services/reimbursementMaterialPdf");
|
||||
const { MaterialPreviewError, createReimbursementMaterialPreviewStore, runMergeWorker } = require("../../../src/services/reimbursementMaterialPreview");
|
||||
const {
|
||||
MATERIAL_PREVIEW_MERGER_VERSION,
|
||||
MaterialPreviewError,
|
||||
createReimbursementMaterialPreviewStore,
|
||||
runMergeWorker,
|
||||
} = require("../../../src/services/reimbursementMaterialPreview");
|
||||
|
||||
const limits = { maxPages: 40, maxImagePixels: 40_000_000, maxOutputBytes: 10 * 1024 * 1024 };
|
||||
const ENCRYPTED_PDF = Buffer.from("JVBERi0xLjMKJeLjz9MKMSAwIG9iago8PAovUHJvZHVjZXIgPGQ1MTZlOTU0YzA+Cj4+CmVuZG9iagoyIDAgb2JqCjw8Ci9UeXBlIC9QYWdlcwovQ291bnQgMQovS2lkcyBbIDQgMCBSIF0KPj4KZW5kb2JqCjMgMCBvYmoKPDwKL1R5cGUgL0NhdGFsb2cKL1BhZ2VzIDIgMCBSCj4+CmVuZG9iago0IDAgb2JqCjw8Ci9UeXBlIC9QYWdlCi9SZXNvdXJjZXMgPDwKPj4KL01lZGlhQm94IFsgMC4wIDAuMCAyMDAgMjAwIF0KL1BhcmVudCAyIDAgUgo+PgplbmRvYmoKNSAwIG9iago8PAovViAyCi9SIDMKL0xlbmd0aCAxMjgKL1AgNDI5NDk2NzI5MgovRmlsdGVyIC9TdGFuZGFyZAovTyA8YTgxN2MwMzIxY2ViOTJjMDQ3MDkyMDc2MTA0NjJkZjViZWIwMzJjYzBhYTQ4YTY3MGY5YWVmNTM3NTk2MzdkNj4KL1UgPDIyOWI4ZWJjYTUzMTFkZWYzZjhiNzM4MzQ1MzExZTdmMjhiZjRlNWU0ZTc1OGE0MTY0MDA0ZTU2ZmZmYTAxMDg+Cj4+CmVuZG9iagp4cmVmCjAgNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTUgMDAwMDAgbiAKMDAwMDAwMDA1OSAwMDAwMCBuIAowMDAwMDAwMTE4IDAwMDAwIG4gCjAwMDAwMDAxNjcgMDAwMDAgbiAKMDAwMDAwMDI2MSAwMDAwMCBuIAp0cmFpbGVyCjw8Ci9TaXplIDYKL1Jvb3QgMyAwIFIKL0luZm8gMSAwIFIKL0lEIFsgPDM1Mzk2MzMyMzA2MjYyNjE2NTYzMzgzMjY1MzE2MjM1NjMzNjMzNjM2MTYyNjU2NjM1NjE2NjYxNjU2NjMxMzE+IDwzNTM5NjMzMjMwNjI2MjYxNjU2MzM4MzI2NTMxNjIzNTYzMzYzMzYzNjE2MjY1NjYzNTYxNjY2MTY1NjYzMTMxPiBdCi9FbmNyeXB0IDUgMCBSCj4+CnN0YXJ0eHJlZgo0NzYKJSVFT0YK", "base64");
|
||||
@@ -13,6 +18,39 @@ async function samplePdf(pageCount = 2) {
|
||||
}
|
||||
|
||||
describe("reimbursement material PDF preview", () => {
|
||||
it("uses normalized output dimensions for EXIF-rotated and ordinary images", async () => {
|
||||
const oriented6 = await sharp({ create: { width: 400, height: 200, channels: 3, background: "white" } })
|
||||
.jpeg()
|
||||
.withMetadata({ orientation: 6 })
|
||||
.toBuffer();
|
||||
const oriented8 = await sharp({ create: { width: 600, height: 300, channels: 3, background: "white" } })
|
||||
.jpeg()
|
||||
.withMetadata({ orientation: 8 })
|
||||
.toBuffer();
|
||||
const verticalJpeg = await sharp({ create: { width: 120, height: 240, channels: 3, background: "white" } })
|
||||
.jpeg()
|
||||
.toBuffer();
|
||||
const horizontalWebp = await sharp({ create: { width: 160, height: 80, channels: 3, background: "white" } })
|
||||
.webp()
|
||||
.toBuffer();
|
||||
|
||||
const result = await mergeMaterialsToPdf([
|
||||
{ id: "orientation-6", materialType: "attachment", mimeType: "image/jpeg", source: oriented6 },
|
||||
{ id: "orientation-8", materialType: "attachment", mimeType: "image/jpeg", source: oriented8 },
|
||||
{ id: "vertical-jpeg", materialType: "attachment", mimeType: "image/jpeg", source: verticalJpeg },
|
||||
{ id: "horizontal-webp", materialType: "attachment", mimeType: "image/webp", source: horizontalWebp },
|
||||
], limits);
|
||||
const pages = (await PDFDocument.load(result.buffer)).getPages();
|
||||
|
||||
expect(pages.map((page) => [page.getWidth(), page.getHeight()])).toEqual([
|
||||
[200, 400],
|
||||
[300, 600],
|
||||
[120, 240],
|
||||
[160, 80],
|
||||
]);
|
||||
expect(MATERIAL_PREVIEW_MERGER_VERSION).toBe("pdf-v2");
|
||||
});
|
||||
|
||||
it("copies every PDF page, embeds images, and produces deterministic bytes", async () => {
|
||||
const pdf = await samplePdf(2);
|
||||
const png = await sharp({ create: { width: 80, height: 40, channels: 3, background: "white" } }).png().toBuffer();
|
||||
|
||||
@@ -114,6 +114,22 @@ describe("workflow assignee resolution", () => {
|
||||
);
|
||||
});
|
||||
|
||||
describe("resolveWorkflowStepDisplayName", () => {
|
||||
it("hides the internal role code in legacy owner fallback tasks", () => {
|
||||
expect(workflowUtils.resolveWorkflowStepDisplayName({
|
||||
step_name: "负责人本人申请兜底:finance",
|
||||
assignee_role: "finance",
|
||||
})).toBe("财务复核");
|
||||
});
|
||||
|
||||
it("keeps normal readable workflow names unchanged", () => {
|
||||
expect(workflowUtils.resolveWorkflowStepDisplayName({
|
||||
step_name: "高额财务终审",
|
||||
assignee_role: "finance",
|
||||
})).toBe("高额财务终审");
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildApprovalWorkflowSnapshot", () => {
|
||||
it("keeps ordinary employee approval snapshots unchanged", async () => {
|
||||
const tx = buildTx({ ownerRoles: ["employee"], managerId: MANAGER_ID });
|
||||
@@ -153,6 +169,7 @@ describe("workflow assignee resolution", () => {
|
||||
{ type: "role", value: "finance" },
|
||||
]);
|
||||
expect(steps[0].id).toContain("owner_self_fallback");
|
||||
expect(steps[0].name).toBe("财务复核");
|
||||
|
||||
const resolved = await resolveActionableStep({
|
||||
tx,
|
||||
@@ -184,6 +201,7 @@ describe("workflow assignee resolution", () => {
|
||||
{ type: "role", value: "finance" },
|
||||
]);
|
||||
expect(steps[0].id).toContain("owner_self_fallback");
|
||||
expect(steps[0].name).toBe("财务复核");
|
||||
});
|
||||
|
||||
it("removes CEO self node and duplicate fallback role for large CEO owner applications", async () => {
|
||||
|
||||
在新工单中引用
屏蔽一个用户