增加铁路OCR金额字段安全诊断(07-28-travel-reimbursement-guide)
这个提交包含在:
@@ -273,7 +273,8 @@ await prisma.ocrResult.create({ data: { result_json: persisted } });
|
|||||||
imageBase64, sourceMimeType: "application/pdf" })`.
|
imageBase64, sourceMimeType: "application/pdf" })`.
|
||||||
- Output: one JSON object containing only `alias`, `provider`, `status`,
|
- Output: one JSON object containing only `alias`, `provider`, `status`,
|
||||||
`transportMode`, `elapsedMs`, safe document category/subtype/count,
|
`transportMode`, `elapsedMs`, safe document category/subtype/count,
|
||||||
per-field hit booleans, missing field names and a bounded error code.
|
per-field hit booleans, missing field names, an optional railway amount-field
|
||||||
|
boolean diagnostic and a bounded error code.
|
||||||
|
|
||||||
### 3. Contracts
|
### 3. Contracts
|
||||||
|
|
||||||
@@ -291,6 +292,12 @@ await prisma.ocrResult.create({ data: { result_json: persisted } });
|
|||||||
- Never output recognized identifiers, dates, amounts, buyer/seller/passenger
|
- Never output recognized identifiers, dates, amounts, buyer/seller/passenger
|
||||||
names, provider messages, raw response or input bytes. Field presence is a
|
names, provider messages, raw response or input bytes. Field presence is a
|
||||||
boolean; errors are bounded stable codes.
|
boolean; errors are bounded stable codes.
|
||||||
|
- When a railway result lacks `totalAmount`, the adapter may expose an
|
||||||
|
ephemeral diagnostic for the documented Tencent `Total` and `Fare` fields.
|
||||||
|
It contains only presence, string/number type, non-empty and current-parser
|
||||||
|
acceptance booleans. The live runner re-allowlists those booleans; the
|
||||||
|
persistence sanitizer must drop the complete diagnostic object before every
|
||||||
|
cache or database write.
|
||||||
- The JSON summary is the primary evidence even when the probe exits non-zero.
|
- The JSON summary is the primary evidence even when the probe exits non-zero.
|
||||||
Shell callers must capture and persist the bounded stdout summary before
|
Shell callers must capture and persist the bounded stdout summary before
|
||||||
interpreting the exit status; do not let `set -e` discard a valid HOLD
|
interpreting the exit status; do not let `set -e` discard a valid HOLD
|
||||||
@@ -331,6 +338,9 @@ await prisma.ocrResult.create({ data: { result_json: persisted } });
|
|||||||
the provider mock is called.
|
the provider mock is called.
|
||||||
- Privacy tests seed raw provider and recognizable normalized markers, then
|
- Privacy tests seed raw provider and recognizable normalized markers, then
|
||||||
assert serialized output contains only booleans and safe metadata.
|
assert serialized output contains only booleans and safe metadata.
|
||||||
|
- Railway amount-diagnostic tests cover absent, present-but-unparseable and
|
||||||
|
parseable fields, assert source values never reach stdout, and assert
|
||||||
|
`sanitizeOcrResultForPersistence` removes diagnostics completely.
|
||||||
- CLI test pipes one synthetic PDF over stdin with credentials absent and
|
- CLI test pipes one synthetic PDF over stdin with credentials absent and
|
||||||
asserts one parseable redacted error object with empty stderr.
|
asserts one parseable redacted error object with empty stderr.
|
||||||
- Caller capture test asserts a redacted HOLD JSON remains available when the
|
- Caller capture test asserts a redacted HOLD JSON remains available when the
|
||||||
|
|||||||
@@ -51,6 +51,8 @@
|
|||||||
- 每个进程只接收一份 PDF,验证文件签名和 7MiB 上限后最多调用一次 `RecognizeGeneralInvoice`。住宿与铁路样票分两次执行,任一失败立即停止,不自动遍历目录中的其他文件。
|
- 每个进程只接收一份 PDF,验证文件签名和 7MiB 上限后最多调用一次 `RecognizeGeneralInvoice`。住宿与铁路样票分两次执行,任一失败立即停止,不自动遍历目录中的其他文件。
|
||||||
- 输出只允许匿名别名、provider、票种、安全传输模式、耗时、字段命中布尔值、缺字段名和稳定错误码;不得输出文件名、原始响应、识别值、人员/企业名称、票号、金额或票面内容。
|
- 输出只允许匿名别名、provider、票种、安全传输模式、耗时、字段命中布尔值、缺字段名和稳定错误码;不得输出文件名、原始响应、识别值、人员/企业名称、票号、金额或票面内容。
|
||||||
- live runner 的代码、合成测试和生产部署先通过,再单独取得两次真实 API 调用批准;工具可用不等于真实样票已验证。
|
- live runner 的代码、合成测试和生产部署先通过,再单独取得两次真实 API 调用批准;工具可用不等于真实样票已验证。
|
||||||
|
- 当铁路票已被腾讯识别但金额缺失时,适配器可生成仅供 live runner 使用的临时字段形态诊断:只包含 `Total`/`Fare` 是否存在、是否为字符串或数字、是否非空及能否被当前金额解析器接受的布尔值。诊断不得携带字段值,必须在持久化白名单处被丢弃,普通 OCR job、缓存和数据库均不得保存。
|
||||||
|
- 字段形态诊断先用合成 provider 响应证明不输出金额或原始 payload,再部署;真实调用仍受单次授权和“铁路未 PASS 即停止住宿”约束。
|
||||||
|
|
||||||
## 文档口径
|
## 文档口径
|
||||||
|
|
||||||
|
|||||||
@@ -46,6 +46,8 @@
|
|||||||
- [x] 确认实际 provider 为 `tencent`,且只有一个 OCR job executor。
|
- [x] 确认实际 provider 为 `tencent`,且只有一个 OCR job executor。
|
||||||
- [x] 修正 live 样票脚本:复用 `RecognizeGeneralInvoice`、按票种必填字段和安全持久化白名单,支持单 PDF 标准输入且仅输出脱敏摘要。
|
- [x] 修正 live 样票脚本:复用 `RecognizeGeneralInvoice`、按票种必填字段和安全持久化白名单,支持单 PDF 标准输入且仅输出脱敏摘要。
|
||||||
- [x] 使用合成 PDF/provider 响应覆盖住宿、铁路、非 PDF、超限、缺字段和原始响应不输出,并完成本地门禁、提交和生产部署。
|
- [x] 使用合成 PDF/provider 响应覆盖住宿、铁路、非 PDF、超限、缺字段和原始响应不输出,并完成本地门禁、提交和生产部署。
|
||||||
|
- [x] 为铁路 `Total`/`Fare` 增加只含存在、类型、非空和可解析布尔值的临时诊断;证明持久化白名单丢弃诊断且 live runner 不输出真实值。
|
||||||
|
- [ ] 完成本地门禁、提交和生产部署后,使用剩余一次授权诊断铁路金额字段;铁路未 PASS 时仍不调用住宿样票。
|
||||||
- [ ] 经单独样票批准后分别运行住宿电子普通发票和铁路电子客票 live OCR PDF 验收。
|
- [ ] 经单独样票批准后分别运行住宿电子普通发票和铁路电子客票 live OCR PDF 验收。
|
||||||
- [ ] 只保存脱敏状态、耗时、provider 和字段命中摘要;确认不存在 mock 调用证据。
|
- [ ] 只保存脱敏状态、耗时、provider 和字段命中摘要;确认不存在 mock 调用证据。
|
||||||
- [ ] 按正常 API/业务生命周期清理或归档测试记录,并执行只读残留检查。
|
- [ ] 按正常 API/业务生命周期清理或归档测试记录,并执行只读残留检查。
|
||||||
@@ -85,6 +87,18 @@
|
|||||||
- 按“第一份未取得明确 PASS 即停止”约束,未调用住宿样票,也未重试铁路样票。
|
- 按“第一份未取得明确 PASS 即停止”约束,未调用住宿样票,也未重试铁路样票。
|
||||||
- 空 stdin 捕获自检证明修正后的调用方式可在 provider 调用前保留 `HOLD/sample_empty` 摘要;该自检未消耗 OCR 次数。
|
- 空 stdin 捕获自检证明修正后的调用方式可在 provider 调用前保留 `HOLD/sample_empty` 摘要;该自检未消耗 OCR 次数。
|
||||||
- 本次没有创建 OA 发票、附件或数据库记录;真实样票仍未验收通过,员工上传门禁保持关闭。
|
- 本次没有创建 OA 发票、附件或数据库记录;真实样票仍未验收通过,员工上传门禁保持关闭。
|
||||||
|
- 经额外授权后,修正包装器重新验收铁路 PDF,明确得到 `HOLD/sample_pdf_invalid`;失败发生在 provider 调用前,本次未消耗腾讯 OCR 次数。
|
||||||
|
- 匿名结构预检确认铁路 PDF 的 EOF 后存在非空白尾随数据,不符合现行严格 PDF 终止边界;不截断原件、不放宽安全门禁,需重新取得官方原始电子票据。
|
||||||
|
- 住宿 PDF 的 PDF 头部、EOF 终止边界和 7 MiB 上限预检均通过,但因铁路票未 PASS,未调用住宿样票 provider。
|
||||||
|
- 用户另行指定的铁路 PDF 通过同一结构预检,证明上一份 `sample_pdf_invalid` 是旧文件特有的 EOF 尾随数据问题,不是所有铁路 PDF 或校验器普遍失败。
|
||||||
|
- 新铁路 PDF 经真实腾讯 OCR 识别为单张 `ElectronicTrainTicketFull`,票号、日期和购买方字段命中,但 `totalAmount` 未命中,终态为 `HOLD/missing_required_fields:totalAmount`;未输出或持久化任何识别值、原始响应或票面字段。
|
||||||
|
- 当前适配器已按腾讯 SDK 合同读取铁路票 `Total` 与 `Fare` 两个金额字段;现有脱敏摘要不能区分 provider 未返回金额与返回值格式不可解析,需在另行批准后增加字段形态布尔诊断。铁路票未 PASS,因此仍未调用住宿样票 provider。
|
||||||
|
|
||||||
|
## 2026-07-29 铁路金额字段诊断实现验证
|
||||||
|
|
||||||
|
- 聚焦测试 19 项、完整后端 168 个文件/1075 项、Prisma validate、前端 typecheck/build、流程合同、Trellis governance、敏感内容和 diff 门禁通过;仅有既有依赖弃用提示与 22 个既有未注册文档 warning。
|
||||||
|
- P2 聚合器的后端聚焦测试 94 项及前端门禁通过;其业务流因合成账号首次改密状态失败,live-readiness 因本地旧凭据和部署前构建标识不一致失败。进入 live OCR 步骤时已主动终止,生产最近 10 分钟新增发票和 OCR 尝试记录均为 0。
|
||||||
|
- 诊断只存在于 provider 返回到 live runner 的进程内对象,持久化 sanitizer 丢弃整个 diagnostics;合成敏感标记未出现在序列化摘要中。
|
||||||
|
|
||||||
## 风险文件与操作
|
## 风险文件与操作
|
||||||
|
|
||||||
|
|||||||
@@ -131,6 +131,28 @@ function buildFieldHits(normalized) {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function buildTrainAmountDiagnostics(providerResult) {
|
||||||
|
if (providerResult?.documentCategory !== "electronic_train_ticket") {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const source = providerResult?.diagnostics?.trainAmountFields;
|
||||||
|
const booleanValue = (field, key) => source?.[field]?.[key] === true;
|
||||||
|
|
||||||
|
return {
|
||||||
|
totalPresent: booleanValue("total", "present"),
|
||||||
|
totalIsString: booleanValue("total", "isString"),
|
||||||
|
totalIsNumber: booleanValue("total", "isNumber"),
|
||||||
|
totalNonEmpty: booleanValue("total", "nonEmpty"),
|
||||||
|
totalParseable: booleanValue("total", "parseable"),
|
||||||
|
farePresent: booleanValue("fare", "present"),
|
||||||
|
fareIsString: booleanValue("fare", "isString"),
|
||||||
|
fareIsNumber: booleanValue("fare", "isNumber"),
|
||||||
|
fareNonEmpty: booleanValue("fare", "nonEmpty"),
|
||||||
|
fareParseable: booleanValue("fare", "parseable"),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function buildErrorSummary(alias, error) {
|
function buildErrorSummary(alias, error) {
|
||||||
return {
|
return {
|
||||||
alias: SAFE_ALIAS_PATTERN.test(alias) ? alias : "SAMPLE-PDF-UNKNOWN",
|
alias: SAFE_ALIAS_PATTERN.test(alias) ? alias : "SAMPLE-PDF-UNKNOWN",
|
||||||
@@ -174,6 +196,7 @@ async function runLiveSampleCheck({
|
|||||||
documentSubtype: safeToken(normalized.documentSubtype),
|
documentSubtype: safeToken(normalized.documentSubtype),
|
||||||
documentCount: Number.isInteger(normalized.documentCount) ? normalized.documentCount : null,
|
documentCount: Number.isInteger(normalized.documentCount) ? normalized.documentCount : null,
|
||||||
fieldHits: buildFieldHits(normalized),
|
fieldHits: buildFieldHits(normalized),
|
||||||
|
trainAmountDiagnostics: buildTrainAmountDiagnostics(providerResult),
|
||||||
missingRequiredFields: qualityError?.startsWith("missing_required_fields:")
|
missingRequiredFields: qualityError?.startsWith("missing_required_fields:")
|
||||||
? qualityError.slice("missing_required_fields:".length).split(",").filter(Boolean)
|
? qualityError.slice("missing_required_fields:".length).split(",").filter(Boolean)
|
||||||
: [],
|
: [],
|
||||||
@@ -211,6 +234,7 @@ if (import.meta.url === invokedPath) {
|
|||||||
export {
|
export {
|
||||||
LiveOcrSampleError,
|
LiveOcrSampleError,
|
||||||
buildErrorSummary,
|
buildErrorSummary,
|
||||||
|
buildTrainAmountDiagnostics,
|
||||||
parseArgs,
|
parseArgs,
|
||||||
readBoundedStdin,
|
readBoundedStdin,
|
||||||
runLiveSampleCheck,
|
runLiveSampleCheck,
|
||||||
|
|||||||
@@ -73,6 +73,20 @@ function toNumberString(value) {
|
|||||||
return num.toFixed(2);
|
return num.toFixed(2);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function buildNumericFieldDiagnostics(value) {
|
||||||
|
const present = value != null;
|
||||||
|
const isString = typeof value === "string";
|
||||||
|
const isNumber = typeof value === "number" && Number.isFinite(value);
|
||||||
|
|
||||||
|
return {
|
||||||
|
present,
|
||||||
|
isString,
|
||||||
|
isNumber,
|
||||||
|
nonEmpty: isString ? value.trim().length > 0 : isNumber,
|
||||||
|
parseable: toNumberString(value) != null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function addNumberStrings(...values) {
|
function addNumberStrings(...values) {
|
||||||
const normalized = values.map((value) => toNumberString(value));
|
const normalized = values.map((value) => toNumberString(value));
|
||||||
if (normalized.some((value) => value == null)) {
|
if (normalized.some((value) => value == null)) {
|
||||||
@@ -221,6 +235,12 @@ function normalizeElectronicTrainTicket(info) {
|
|||||||
taxAmount,
|
taxAmount,
|
||||||
totalAmount,
|
totalAmount,
|
||||||
buyerName: normalizeString(info?.Buyer),
|
buyerName: normalizeString(info?.Buyer),
|
||||||
|
diagnostics: {
|
||||||
|
trainAmountFields: {
|
||||||
|
total: buildNumericFieldDiagnostics(info?.Total),
|
||||||
|
fare: buildNumericFieldDiagnostics(info?.Fare),
|
||||||
|
},
|
||||||
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -106,6 +106,60 @@ describe("check-ocr-provider-samples", () => {
|
|||||||
expect(summary.missingRequiredFields).toEqual(["buyerName"]);
|
expect(summary.missingRequiredFields).toEqual(["buyerName"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("exposes only boolean railway amount diagnostics", async () => {
|
||||||
|
const summary = await liveRunner.runLiveSampleCheck({
|
||||||
|
alias: "SAMPLE-PDF-TRAIN-DIAGNOSTIC",
|
||||||
|
buffer: syntheticPdf(),
|
||||||
|
config: providerConfig(),
|
||||||
|
runOcr: vi.fn().mockResolvedValue(
|
||||||
|
normalizedResult({
|
||||||
|
documentSubtype: "ElectronicTrainTicketFull",
|
||||||
|
documentCategory: "electronic_train_ticket",
|
||||||
|
invoiceCode: null,
|
||||||
|
totalAmount: null,
|
||||||
|
sellerName: null,
|
||||||
|
diagnostics: {
|
||||||
|
trainAmountFields: {
|
||||||
|
total: {
|
||||||
|
present: true,
|
||||||
|
isString: true,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: true,
|
||||||
|
parseable: false,
|
||||||
|
value: "forbidden-total-marker",
|
||||||
|
},
|
||||||
|
fare: {
|
||||||
|
present: false,
|
||||||
|
isString: false,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: false,
|
||||||
|
parseable: false,
|
||||||
|
value: "forbidden-fare-marker",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
});
|
||||||
|
const serialized = JSON.stringify(summary);
|
||||||
|
|
||||||
|
expect(summary.status).toBe("HOLD");
|
||||||
|
expect(summary.trainAmountDiagnostics).toEqual({
|
||||||
|
totalPresent: true,
|
||||||
|
totalIsString: true,
|
||||||
|
totalIsNumber: false,
|
||||||
|
totalNonEmpty: true,
|
||||||
|
totalParseable: false,
|
||||||
|
farePresent: false,
|
||||||
|
fareIsString: false,
|
||||||
|
fareIsNumber: false,
|
||||||
|
fareNonEmpty: false,
|
||||||
|
fareParseable: false,
|
||||||
|
});
|
||||||
|
expect(serialized).not.toContain("forbidden-total-marker");
|
||||||
|
expect(serialized).not.toContain("forbidden-fare-marker");
|
||||||
|
});
|
||||||
|
|
||||||
it("never exposes provider raw fields or normalized business values", async () => {
|
it("never exposes provider raw fields or normalized business values", async () => {
|
||||||
const summary = await liveRunner.runLiveSampleCheck({
|
const summary = await liveRunner.runLiveSampleCheck({
|
||||||
alias: "SAMPLE-PDF-PRIVACY",
|
alias: "SAMPLE-PDF-PRIVACY",
|
||||||
|
|||||||
@@ -139,12 +139,66 @@ describe("Tencent general invoice OCR", () => {
|
|||||||
totalAmount: "553.00",
|
totalAmount: "553.00",
|
||||||
sellerName: null,
|
sellerName: null,
|
||||||
buyerName: "合成公司",
|
buyerName: "合成公司",
|
||||||
|
diagnostics: {
|
||||||
|
trainAmountFields: {
|
||||||
|
total: {
|
||||||
|
present: true,
|
||||||
|
isString: true,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: true,
|
||||||
|
parseable: true,
|
||||||
|
},
|
||||||
|
fare: {
|
||||||
|
present: false,
|
||||||
|
isString: false,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: false,
|
||||||
|
parseable: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
});
|
});
|
||||||
expect(buildMissingRequiredOcrError("vat", normalized)).toBeNull();
|
expect(buildMissingRequiredOcrError("vat", normalized)).toBeNull();
|
||||||
expect(shouldDeduplicateByInvoiceNumberOnly(normalized)).toBe(true);
|
expect(shouldDeduplicateByInvoiceNumberOnly(normalized)).toBe(true);
|
||||||
expect(JSON.stringify(normalized)).not.toContain("SENSITIVE-");
|
expect(JSON.stringify(normalized)).not.toContain("SENSITIVE-");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("reports only boolean railway amount field diagnostics", () => {
|
||||||
|
const normalized = normalizeGeneralInvoiceResult({
|
||||||
|
MixedInvoiceItems: [
|
||||||
|
buildInvoiceItem("ElectronicTrainTicketFull", "ElectronicTrainTicketFull", {
|
||||||
|
Number: "251234",
|
||||||
|
Date: "2026-07-26",
|
||||||
|
Fare: "not-a-number-sensitive-marker",
|
||||||
|
Buyer: "合成公司",
|
||||||
|
}),
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(normalized.diagnostics).toEqual({
|
||||||
|
trainAmountFields: {
|
||||||
|
total: {
|
||||||
|
present: false,
|
||||||
|
isString: false,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: false,
|
||||||
|
parseable: false,
|
||||||
|
},
|
||||||
|
fare: {
|
||||||
|
present: true,
|
||||||
|
isString: true,
|
||||||
|
isNumber: false,
|
||||||
|
nonEmpty: true,
|
||||||
|
parseable: false,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(JSON.stringify(normalized.diagnostics)).not.toContain("sensitive-marker");
|
||||||
|
expect(sanitizeOcrResultForPersistence("tencent", normalized)).not.toHaveProperty(
|
||||||
|
"diagnostics",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("returns safe terminal quality results for zero, multiple and unsupported invoices", () => {
|
it("returns safe terminal quality results for zero, multiple and unsupported invoices", () => {
|
||||||
expect(normalizeGeneralInvoiceResult({ MixedInvoiceItems: [] })).toEqual(
|
expect(normalizeGeneralInvoiceResult({ MixedInvoiceItems: [] })).toEqual(
|
||||||
expect.objectContaining({ qualityError: "invoice_not_detected", documentCount: 0 }),
|
expect.objectContaining({ qualityError: "invoice_not_detected", documentCount: 0 }),
|
||||||
|
|||||||
在新工单中引用
屏蔽一个用户