From 5f2788ccacb390ec35b9ff1f7fd10d507a56feec Mon Sep 17 00:00:00 2001 From: shunli Date: Wed, 29 Jul 2026 15:07:58 +0800 Subject: [PATCH] =?UTF-8?q?=E5=A2=9E=E5=8A=A0=E9=93=81=E8=B7=AFOCR?= =?UTF-8?q?=E9=87=91=E9=A2=9D=E5=AD=97=E6=AE=B5=E5=AE=89=E5=85=A8=E8=AF=8A?= =?UTF-8?q?=E6=96=AD=EF=BC=8807-28-travel-reimbursement-guide=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../spec/hyb-oa/reimbursement-invoice-ocr.md | 12 ++++- .../design.md | 2 + .../implement.md | 14 +++++ scripts/qa/check-ocr-provider-samples.mjs | 24 +++++++++ server/src/services/ocr/tencent.js | 20 +++++++ .../unit/scripts/ocr-provider-samples.test.js | 54 +++++++++++++++++++ .../tests/unit/services/ocr.tencent.test.js | 54 +++++++++++++++++++ 7 files changed, 179 insertions(+), 1 deletion(-) diff --git a/.trellis/spec/hyb-oa/reimbursement-invoice-ocr.md b/.trellis/spec/hyb-oa/reimbursement-invoice-ocr.md index d3add75b..0d2853df 100644 --- a/.trellis/spec/hyb-oa/reimbursement-invoice-ocr.md +++ b/.trellis/spec/hyb-oa/reimbursement-invoice-ocr.md @@ -273,7 +273,8 @@ await prisma.ocrResult.create({ data: { result_json: persisted } }); imageBase64, sourceMimeType: "application/pdf" })`. - Output: one JSON object containing only `alias`, `provider`, `status`, `transportMode`, `elapsedMs`, safe document category/subtype/count, - per-field hit booleans, missing field names and a bounded error code. + per-field hit booleans, missing field names, an optional railway amount-field + boolean diagnostic and a bounded error code. ### 3. Contracts @@ -291,6 +292,12 @@ await prisma.ocrResult.create({ data: { result_json: persisted } }); - Never output recognized identifiers, dates, amounts, buyer/seller/passenger names, provider messages, raw response or input bytes. Field presence is a boolean; errors are bounded stable codes. +- When a railway result lacks `totalAmount`, the adapter may expose an + ephemeral diagnostic for the documented Tencent `Total` and `Fare` fields. + It contains only presence, string/number type, non-empty and current-parser + acceptance booleans. The live runner re-allowlists those booleans; the + persistence sanitizer must drop the complete diagnostic object before every + cache or database write. - The JSON summary is the primary evidence even when the probe exits non-zero. Shell callers must capture and persist the bounded stdout summary before interpreting the exit status; do not let `set -e` discard a valid HOLD @@ -331,6 +338,9 @@ await prisma.ocrResult.create({ data: { result_json: persisted } }); the provider mock is called. - Privacy tests seed raw provider and recognizable normalized markers, then assert serialized output contains only booleans and safe metadata. +- Railway amount-diagnostic tests cover absent, present-but-unparseable and + parseable fields, assert source values never reach stdout, and assert + `sanitizeOcrResultForPersistence` removes diagnostics completely. - CLI test pipes one synthetic PDF over stdin with credentials absent and asserts one parseable redacted error object with empty stderr. - Caller capture test asserts a redacted HOLD JSON remains available when the diff --git a/.trellis/tasks/07-28-travel-reimbursement-guide/design.md b/.trellis/tasks/07-28-travel-reimbursement-guide/design.md index 1e4d8a8f..65717d31 100644 --- a/.trellis/tasks/07-28-travel-reimbursement-guide/design.md +++ b/.trellis/tasks/07-28-travel-reimbursement-guide/design.md @@ -51,6 +51,8 @@ - 每个进程只接收一份 PDF，验证文件签名和 7MiB 上限后最多调用一次 `RecognizeGeneralInvoice`。住宿与铁路样票分两次执行，任一失败立即停止，不自动遍历目录中的其他文件。 - 输出只允许匿名别名、provider、票种、安全传输模式、耗时、字段命中布尔值、缺字段名和稳定错误码；不得输出文件名、原始响应、识别值、人员/企业名称、票号、金额或票面内容。 - live runner 的代码、合成测试和生产部署先通过，再单独取得两次真实 API 调用批准；工具可用不等于真实样票已验证。 +- 当铁路票已被腾讯识别但金额缺失时，适配器可生成仅供 live runner 使用的临时字段形态诊断：只包含 `Total`/`Fare` 是否存在、是否为字符串或数字、是否非空及能否被当前金额解析器接受的布尔值。诊断不得携带字段值，必须在持久化白名单处被丢弃，普通 OCR job、缓存和数据库均不得保存。 +- 字段形态诊断先用合成 provider 响应证明不输出金额或原始 payload，再部署；真实调用仍受单次授权和“铁路未 PASS 即停止住宿”约束。 ## 文档口径 diff --git a/.trellis/tasks/07-28-travel-reimbursement-guide/implement.md b/.trellis/tasks/07-28-travel-reimbursement-guide/implement.md index 0de30905..ec142164 100644 --- a/.trellis/tasks/07-28-travel-reimbursement-guide/implement.md +++ b/.trellis/tasks/07-28-travel-reimbursement-guide/implement.md @@ -46,6 +46,8 @@ - [x] 确认实际 provider 为 `tencent`，且只有一个 OCR job executor。 - [x] 修正 live 样票脚本：复用 `RecognizeGeneralInvoice`、按票种必填字段和安全持久化白名单，支持单 PDF 标准输入且仅输出脱敏摘要。 - [x] 使用合成 PDF/provider 响应覆盖住宿、铁路、非 PDF、超限、缺字段和原始响应不输出，并完成本地门禁、提交和生产部署。 +- [x] 为铁路 `Total`/`Fare` 增加只含存在、类型、非空和可解析布尔值的临时诊断；证明持久化白名单丢弃诊断且 live runner 不输出真实值。 +- [ ] 完成本地门禁、提交和生产部署后，使用剩余一次授权诊断铁路金额字段；铁路未 PASS 时仍不调用住宿样票。 - [ ] 经单独样票批准后分别运行住宿电子普通发票和铁路电子客票 live OCR PDF 验收。 - [ ] 只保存脱敏状态、耗时、provider 和字段命中摘要；确认不存在 mock 调用证据。 - [ ] 按正常 API/业务生命周期清理或归档测试记录，并执行只读残留检查。 @@ -85,6 +87,18 @@ - 按“第一份未取得明确 PASS 即停止”约束，未调用住宿样票，也未重试铁路样票。 - 空 stdin 捕获自检证明修正后的调用方式可在 provider 调用前保留 `HOLD/sample_empty` 摘要；该自检未消耗 OCR 次数。 - 本次没有创建 OA 发票、附件或数据库记录；真实样票仍未验收通过，员工上传门禁保持关闭。 +- 经额外授权后，修正包装器重新验收铁路 PDF，明确得到 `HOLD/sample_pdf_invalid`；失败发生在 provider 调用前，本次未消耗腾讯 OCR 次数。 +- 匿名结构预检确认铁路 PDF 的 EOF 后存在非空白尾随数据，不符合现行严格 PDF 终止边界；不截断原件、不放宽安全门禁，需重新取得官方原始电子票据。 +- 住宿 PDF 的 PDF 头部、EOF 终止边界和 7 MiB 上限预检均通过，但因铁路票未 PASS，未调用住宿样票 provider。 +- 用户另行指定的铁路 PDF 通过同一结构预检，证明上一份 `sample_pdf_invalid` 是旧文件特有的 EOF 尾随数据问题，不是所有铁路 PDF 或校验器普遍失败。 +- 新铁路 PDF 经真实腾讯 OCR 识别为单张 `ElectronicTrainTicketFull`，票号、日期和购买方字段命中，但 `totalAmount` 未命中，终态为 `HOLD/missing_required_fields:totalAmount`；未输出或持久化任何识别值、原始响应或票面字段。 +- 当前适配器已按腾讯 SDK 合同读取铁路票 `Total` 与 `Fare` 两个金额字段；现有脱敏摘要不能区分 provider 未返回金额与返回值格式不可解析，需在另行批准后增加字段形态布尔诊断。铁路票未 PASS，因此仍未调用住宿样票 provider。 + +## 2026-07-29 铁路金额字段诊断实现验证 + +- 聚焦测试 19 项、完整后端 168 个文件/1075 项、Prisma validate、前端 typecheck/build、流程合同、Trellis governance、敏感内容和 diff 门禁通过；仅有既有依赖弃用提示与 22 个既有未注册文档 warning。 +- P2 聚合器的后端聚焦测试 94 项及前端门禁通过；其业务流因合成账号首次改密状态失败，live-readiness 因本地旧凭据和部署前构建标识不一致失败。进入 live OCR 步骤时已主动终止，生产最近 10 分钟新增发票和 OCR 尝试记录均为 0。 +- 诊断只存在于 provider 返回到 live runner 的进程内对象，持久化 sanitizer 丢弃整个 diagnostics；合成敏感标记未出现在序列化摘要中。 ## 风险文件与操作 diff --git a/scripts/qa/check-ocr-provider-samples.mjs b/scripts/qa/check-ocr-provider-samples.mjs index 7e1b37cf..214fe464 100644 --- a/scripts/qa/check-ocr-provider-samples.mjs +++ b/scripts/qa/check-ocr-provider-samples.mjs @@ -131,6 +131,28 @@ function buildFieldHits(normalized) { ); } +function buildTrainAmountDiagnostics(providerResult) { + if (providerResult?.documentCategory !== "electronic_train_ticket") { + return null; + } + + const source = providerResult?.diagnostics?.trainAmountFields; + const booleanValue = (field, key) => source?.[field]?.[key] === true; + + return { + totalPresent: booleanValue("total", "present"), + totalIsString: booleanValue("total", "isString"), + totalIsNumber: booleanValue("total", "isNumber"), + totalNonEmpty: booleanValue("total", "nonEmpty"), + totalParseable: booleanValue("total", "parseable"), + farePresent: booleanValue("fare", "present"), + fareIsString: booleanValue("fare", "isString"), + fareIsNumber: booleanValue("fare", "isNumber"), + fareNonEmpty: booleanValue("fare", "nonEmpty"), + fareParseable: booleanValue("fare", "parseable"), + }; +} + function buildErrorSummary(alias, error) { return { alias: SAFE_ALIAS_PATTERN.test(alias) ? alias : "SAMPLE-PDF-UNKNOWN", @@ -174,6 +196,7 @@ async function runLiveSampleCheck({ documentSubtype: safeToken(normalized.documentSubtype), documentCount: Number.isInteger(normalized.documentCount) ? normalized.documentCount : null, fieldHits: buildFieldHits(normalized), + trainAmountDiagnostics: buildTrainAmountDiagnostics(providerResult), missingRequiredFields: qualityError?.startsWith("missing_required_fields:") ? qualityError.slice("missing_required_fields:".length).split(",").filter(Boolean) : [], @@ -211,6 +234,7 @@ if (import.meta.url === invokedPath) { export { LiveOcrSampleError, buildErrorSummary, + buildTrainAmountDiagnostics, parseArgs, readBoundedStdin, runLiveSampleCheck, diff --git a/server/src/services/ocr/tencent.js b/server/src/services/ocr/tencent.js index 2008e762..a3e50a16 100644 --- a/server/src/services/ocr/tencent.js +++ b/server/src/services/ocr/tencent.js @@ -73,6 +73,20 @@ function toNumberString(value) { return num.toFixed(2); } +function buildNumericFieldDiagnostics(value) { + const present = value != null; + const isString = typeof value === "string"; + const isNumber = typeof value === "number" && Number.isFinite(value); + + return { + present, + isString, + isNumber, + nonEmpty: isString ? value.trim().length > 0 : isNumber, + parseable: toNumberString(value) != null, + }; +} + function addNumberStrings(...values) { const normalized = values.map((value) => toNumberString(value)); if (normalized.some((value) => value == null)) { @@ -221,6 +235,12 @@ function normalizeElectronicTrainTicket(info) { taxAmount, totalAmount, buyerName: normalizeString(info?.Buyer), + diagnostics: { + trainAmountFields: { + total: buildNumericFieldDiagnostics(info?.Total), + fare: buildNumericFieldDiagnostics(info?.Fare), + }, + }, }; } diff --git a/server/tests/unit/scripts/ocr-provider-samples.test.js b/server/tests/unit/scripts/ocr-provider-samples.test.js index e4001339..3a2d9859 100644 --- a/server/tests/unit/scripts/ocr-provider-samples.test.js +++ b/server/tests/unit/scripts/ocr-provider-samples.test.js @@ -106,6 +106,60 @@ describe("check-ocr-provider-samples", () => { expect(summary.missingRequiredFields).toEqual(["buyerName"]); }); + it("exposes only boolean railway amount diagnostics", async () => { + const summary = await liveRunner.runLiveSampleCheck({ + alias: "SAMPLE-PDF-TRAIN-DIAGNOSTIC", + buffer: syntheticPdf(), + config: providerConfig(), + runOcr: vi.fn().mockResolvedValue( + normalizedResult({ + documentSubtype: "ElectronicTrainTicketFull", + documentCategory: "electronic_train_ticket", + invoiceCode: null, + totalAmount: null, + sellerName: null, + diagnostics: { + trainAmountFields: { + total: { + present: true, + isString: true, + isNumber: false, + nonEmpty: true, + parseable: false, + value: "forbidden-total-marker", + }, + fare: { + present: false, + isString: false, + isNumber: false, + nonEmpty: false, + parseable: false, + value: "forbidden-fare-marker", + }, + }, + }, + }), + ), + }); + const serialized = JSON.stringify(summary); + + expect(summary.status).toBe("HOLD"); + expect(summary.trainAmountDiagnostics).toEqual({ + totalPresent: true, + totalIsString: true, + totalIsNumber: false, + totalNonEmpty: true, + totalParseable: false, + farePresent: false, + fareIsString: false, + fareIsNumber: false, + fareNonEmpty: false, + fareParseable: false, + }); + expect(serialized).not.toContain("forbidden-total-marker"); + expect(serialized).not.toContain("forbidden-fare-marker"); + }); + it("never exposes provider raw fields or normalized business values", async () => { const summary = await liveRunner.runLiveSampleCheck({ alias: "SAMPLE-PDF-PRIVACY", diff --git a/server/tests/unit/services/ocr.tencent.test.js b/server/tests/unit/services/ocr.tencent.test.js index 188b681e..24fb304e 100644 --- a/server/tests/unit/services/ocr.tencent.test.js +++ b/server/tests/unit/services/ocr.tencent.test.js @@ -139,12 +139,66 @@ describe("Tencent general invoice OCR", () => { totalAmount: "553.00", sellerName: null, buyerName: "合成公司", + diagnostics: { + trainAmountFields: { + total: { + present: true, + isString: true, + isNumber: false, + nonEmpty: true, + parseable: true, + }, + fare: { + present: false, + isString: false, + isNumber: false, + nonEmpty: false, + parseable: false, + }, + }, + }, }); expect(buildMissingRequiredOcrError("vat", normalized)).toBeNull(); expect(shouldDeduplicateByInvoiceNumberOnly(normalized)).toBe(true); expect(JSON.stringify(normalized)).not.toContain("SENSITIVE-"); }); + it("reports only boolean railway amount field diagnostics", () => { + const normalized = normalizeGeneralInvoiceResult({ + MixedInvoiceItems: [ + buildInvoiceItem("ElectronicTrainTicketFull", "ElectronicTrainTicketFull", { + Number: "251234", + Date: "2026-07-26", + Fare: "not-a-number-sensitive-marker", + Buyer: "合成公司", + }), + ], + }); + + expect(normalized.diagnostics).toEqual({ + trainAmountFields: { + total: { + present: false, + isString: false, + isNumber: false, + nonEmpty: false, + parseable: false, + }, + fare: { + present: true, + isString: true, + isNumber: false, + nonEmpty: true, + parseable: false, + }, + }, + }); + expect(JSON.stringify(normalized.diagnostics)).not.toContain("sensitive-marker"); + expect(sanitizeOcrResultForPersistence("tencent", normalized)).not.toHaveProperty( + "diagnostics", + ); + }); + it("returns safe terminal quality results for zero, multiple and unsupported invoices", () => { expect(normalizeGeneralInvoiceResult({ MixedInvoiceItems: [] })).toEqual( expect.objectContaining({ qualityError: "invoice_not_detected", documentCount: 0 }),