hyb-oa/.trellis/spec/hyb-oa/docs-and-evidence.md

Docs And Evidence

Source Docs

Start from docs/文档索引.md and docs/文档事实源状态矩阵.md.

Rules

• Prefer updating existing fact sources, module specs, manuals, evidence docs or the current Trellis task.
• New documents need a reason and must be registered in both the index and status matrix.
• Old Issue CSV files are historical audit evidence and should not be rewritten unless the current Trellis task explicitly calls for a legacy/export update.
• For generated evidence, store report paths and conclusions, not raw secrets, raw payloads, full account identifiers, invoice originals or confidential business content.

Scenario: Finding Lifecycle And Audit Closeout

1. Scope / Trigger

• Trigger: a review, audit or trial-run task records findings and later code, deployment or external evidence changes their current state.
• Applies to Trellis PRDs, review reports, finding ledgers and management summaries.

2. Signatures

• Canonical ledger fields: findingId, dimension, discoverySeverity, confidence, discoveryStatus, remediationStatus, deploymentStatus, externalVerificationStatus, evidenceRef, businessImpact, controlObjective, safeVerification, followUpWorkPackage.
• Delta fields: discoveryCommit, reviewedCommit, changed-commit list, validation command, result and explicit environment boundary.

3. Contracts

• Preserve the discovery baseline, discovery severity and discovery status. Later remediation must add a delta; it must not rewrite history as if the finding never existed.
• Record remediation, deployment and external-behavior verification as three independent states. A local test pass does not prove deployment, and a deployed build marker does not prove role-specific or real-device behavior.
• A closeout task must have one canonical finding ledger. Component reports may retain their original wording, but duplicate IDs and reclassified items must point to the canonical ledger.
• When HEAD moves after discovery, record the exact old and new commits, classify the delta and rerun the closeout gates. Do not relabel an old report as evidence for the new HEAD.
• Use repo-relative evidence references and synthetic fixtures. Unknown or unauthorized deployment, public, production-data and real-device checks stay explicit as pending or externally gated.

4. Validation & Error Matrix

• Local fix tested, not deployed -> remediation may pass; deployment remains pending and external behavior remains pending when applicable.
• Finding open with no fix artifact -> deployment is NO_FIX_TO_DEPLOY, not DEPLOYMENT_PENDING.
• Static/tooling-only finding -> deployment may be NOT_APPLICABLE; the remediation state still records whether the contract itself is fixed.
• Duplicate component finding -> preserve its source reference and map it to exactly one canonical ledger entry.

5. Good/Base/Bad Cases

• Good: a historical P0 remains P0/confirmed while local remediation passes, deployment is pending and external behavior is pending.
• Base: an open local-only QA finding uses NO_FIX_TO_DEPLOY, NOT_APPLICABLE deployment or NOT_REQUIRED external verification as appropriate.
• Bad: rename the discovery status to RESOLVED and omit whether the fix was deployed or observed externally.

6. Tests Required

• Task validation and Trellis governance.
• Sensitive-content scan and document registry audit.
• git diff --check plus a manual check that the recorded HEAD matches the evidence-producing checkout.

7. Wrong vs Correct

Wrong

{"findingId":"F-001","status":"resolved"}

Correct

{
  "findingId": "F-001",
  "discoverySeverity": "P0",
  "discoveryStatus": "CONFIRMED",
  "remediationStatus": "LOCAL_FIX_VERIFIED",
  "deploymentStatus": "DEPLOYMENT_PENDING",
  "externalVerificationStatus": "EXTERNAL_BEHAVIOR_PENDING"
}

Scenario: QA Readiness Report Summaries

1. Scope / Trigger

• Trigger: QA shell runners aggregate child reports for launch, patrol, smoke or readiness gates.
• Applies to generated summary.json, step JSON and privacy findings under artifacts/qa-flow/ or Trellis task research directories.

2. Signatures

• Parent shell runner writes summary.json.
• Child runner summary input: <child-report-dir>/summary.json.
• Step summary input/output: <step-name>.step.json.

3. Contracts

• Use repo-relative reportRef / logRef when a path is inside the repo.
• Do not write absolute local workspace paths into summary.json.
• Parent summaries must include child report status and child finding counts.
• Live-readiness summaries must distinguish the local sourceCommit from the public observedBuildSha; a missing or mismatched public build marker cannot produce a clean pass.
• Parent findingCount must become non-zero when any child summary reports PASS_WITH_FINDINGS, SKIP, PARTIAL, PAUSED, MISSING, blocker-like status or child findingCount > 0.
• Machine-readable reports may include status, count, mode, conclusion, boolean operational flags and relative report refs.
• Machine-readable reports must not include amounts, prices, costs, balances, bank details, invoice values, full accounts, tokens, secrets, raw payloads, raw form values or full local paths.
• Credential/readiness presence reports must use safe business labels, not raw env key names. For example, emit adminCredentialReady: true instead of a field named after a credential-value environment variable.
• check-pending-ocr.mjs must reuse the adapter-backed Prisma client factory and close both Prisma and its external PostgreSQL pool.

4. Validation & Error Matrix

• Child PASS + findingCount=0 -> parent may stay pass.
• Child PASS_WITH_FINDINGS -> parent records a finding.
• Child SKIP / PARTIAL / PAUSED / MISSING -> parent records manual_or_config.
• Child FAIL / BLOCKED / ERROR -> parent records a finding or blocker according to whether the parent step is required.
• Privacy smoke match -> write redacted finding metadata and make final summary reflect the failure.
• Secret-like field names in generated reports -> privacy finding, even when the value is only a boolean presence flag.
• Public /ready has no usable build SHA -> manual_or_config finding.
• Expected and observed public SHA differ -> manual_or_config finding or a deployment blocker; never a clean pass attributed to local HEAD.

5. Good/Base/Bad Cases

• Good: summary.json contains reportRef, childReports.<name>.status, childReports.<name>.findingCount, sourceCommit, observedBuildSha and no absolute local path.
• Base: Missing runtime credentials produce PASS_WITH_FINDINGS or manual_or_config, not a clean pass.
• Bad: Parent command exits 0 for a child PASS_WITH_FINDINGS and the parent summary still says clean pass.

6. Tests Required

• bash -n <runner>.sh
• Run the parent runner in a missing-credential scenario and assert:
  • parent status is PASS_WITH_FINDINGS or blocked, not clean pass;
  • child finding count is visible in parent summary;
  • privacy step is present in parent summary;
  • summary.json has no full workspace path or business amount fields.
• For live/readiness env reports, assert the JSON contains safe readiness labels and does not contain raw secret-like env key names.
• Run the matching launch/readiness aggregator when it consumes the report.
• Build a synthetic 40-character SHA and assert the public bundle contains only its 12-character marker; use a local HTTP stub to assert readiness records and matches the observed marker.

7. Wrong vs Correct

Wrong

{
  "status": "PASS_OA_REAL_USE_DAILY_READY",
  "reportDir": "/home/user/project/artifacts/qa-flow/run",
  "childReports": {}
}

{
  "liveSmoke": {
    "RAW_CREDENTIAL_ENV_KEY": true
  }
}

Correct

{
  "status": "PASS_WITH_FINDINGS",
  "sourceCommit": "abcdef1234567890",
  "observedBuildSha": "abcdef123456",
  "buildShaMatches": true,
  "reportRef": "artifacts/qa-flow/run",
  "childReports": {
    "oaOnlySmokeReadiness": {
      "status": "PASS_WITH_FINDINGS",
      "reportRef": "artifacts/qa-flow/run/oa-only-smoke-readiness/summary.json",
      "findingCount": 7
    }
  }
}

{
  "liveSmoke": {
    "adminCredentialReady": true
  }
}