feat(server): MinIO/S3 对象存储兼容(F21)

- 新增 storage 抽象:local 与 s3: 前缀兼容(不改 schema)
- invoice-pool 上传/预览/ocr 读取适配 S3/MinIO
- trialrun compose 增加 minio profile + env 示例与 runbook 指引
- 更新 README 与 v3 CSV 状态推进
这个提交包含在:
2026-02-09 20:42:53 +08:00
父节点 6ecebf8ad6
当前提交 c791e67b20
共修改 12 个文件,包含 2558 行新增和 28 行删除
+13
查看文件
@@ -21,6 +21,19 @@ DATABASE_URL="postgresql://hyboa_user:hyboa_password@db:5432/hyb_oa?schema=publi
# ===== 存储(票据上传目录)===== # ===== 存储(票据上传目录)=====
# 在 deploy/trialrun/docker-compose.yml 中映射到宿主机 deploy/trialrun/data/storage # 在 deploy/trialrun/docker-compose.yml 中映射到宿主机 deploy/trialrun/data/storage
STORAGE_DIR="/data/storage" STORAGE_DIR="/data/storage"
# 可选:对象存储(MinIO/S3 兼容,F21)
# - 默认 local:文件写到 STORAGE_DIR
# - 启用 MinIO:STORAGE_DRIVER="s3" 并启用 compose profile:--profile minio
STORAGE_DRIVER="local" # local|s3
S3_ENDPOINT="http://minio:9000"
S3_REGION="us-east-1"
S3_BUCKET="hyb-oa"
S3_ACCESS_KEY_ID="minioadmin"
S3_SECRET_ACCESS_KEY="minioadmin"
S3_FORCE_PATH_STYLE="true"
MINIO_ROOT_USER="minioadmin"
MINIO_ROOT_PASSWORD="minioadmin"
# ===== 单域托管 client/dist ===== # ===== 单域托管 client/dist =====
SERVE_CLIENT_DIST="true" SERVE_CLIENT_DIST="true"
+51
查看文件
@@ -19,6 +19,36 @@ services:
timeout: 5s timeout: 5s
retries: 10 retries: 10
minio:
image: minio/minio:RELEASE.2025-01-20T14-49-07Z
container_name: hyb-oa-trialrun-minio
restart: unless-stopped
profiles: ["minio"]
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
command: ["server", "/data", "--console-address", ":9001"]
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data/minio:/data
minio_init:
image: minio/mc:RELEASE.2025-08-13T08-35-41Z
container_name: hyb-oa-trialrun-minio-init
profiles: ["minio"]
depends_on:
- minio
entrypoint: ["sh", "-c"]
command: |
set -e
for i in $(seq 1 60); do
mc alias set local http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" && break || true
sleep 1
done
mc mb --ignore-existing "local/$S3_BUCKET"
server: server:
build: build:
context: ../.. context: ../..
@@ -35,6 +65,13 @@ services:
JWT_SECRET: ${JWT_SECRET} JWT_SECRET: ${JWT_SECRET}
DATABASE_URL: ${DATABASE_URL} DATABASE_URL: ${DATABASE_URL}
STORAGE_DIR: ${STORAGE_DIR} STORAGE_DIR: ${STORAGE_DIR}
STORAGE_DRIVER: ${STORAGE_DRIVER}
S3_ENDPOINT: ${S3_ENDPOINT}
S3_REGION: ${S3_REGION}
S3_BUCKET: ${S3_BUCKET}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_FORCE_PATH_STYLE: ${S3_FORCE_PATH_STYLE}
HOST: ${HOST} HOST: ${HOST}
PORT: ${PORT} PORT: ${PORT}
COOKIE_SECURE: ${COOKIE_SECURE} COOKIE_SECURE: ${COOKIE_SECURE}
@@ -83,6 +120,13 @@ services:
JWT_SECRET: ${JWT_SECRET} JWT_SECRET: ${JWT_SECRET}
DATABASE_URL: ${DATABASE_URL} DATABASE_URL: ${DATABASE_URL}
STORAGE_DIR: ${STORAGE_DIR} STORAGE_DIR: ${STORAGE_DIR}
STORAGE_DRIVER: ${STORAGE_DRIVER}
S3_ENDPOINT: ${S3_ENDPOINT}
S3_REGION: ${S3_REGION}
S3_BUCKET: ${S3_BUCKET}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_FORCE_PATH_STYLE: ${S3_FORCE_PATH_STYLE}
SERVE_CLIENT_DIST: "false" SERVE_CLIENT_DIST: "false"
CLIENT_DIST_DIR: ${CLIENT_DIST_DIR} CLIENT_DIST_DIR: ${CLIENT_DIST_DIR}
@@ -119,6 +163,13 @@ services:
JWT_SECRET: ${JWT_SECRET} JWT_SECRET: ${JWT_SECRET}
DATABASE_URL: ${DATABASE_URL} DATABASE_URL: ${DATABASE_URL}
STORAGE_DIR: ${STORAGE_DIR} STORAGE_DIR: ${STORAGE_DIR}
STORAGE_DRIVER: ${STORAGE_DRIVER}
S3_ENDPOINT: ${S3_ENDPOINT}
S3_REGION: ${S3_REGION}
S3_BUCKET: ${S3_BUCKET}
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID}
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY}
S3_FORCE_PATH_STYLE: ${S3_FORCE_PATH_STYLE}
SERVE_CLIENT_DIST: ${SERVE_CLIENT_DIST} SERVE_CLIENT_DIST: ${SERVE_CLIENT_DIST}
CLIENT_DIST_DIR: ${CLIENT_DIST_DIR} CLIENT_DIST_DIR: ${CLIENT_DIST_DIR}
OCR_PROVIDER: ${OCR_PROVIDER} OCR_PROVIDER: ${OCR_PROVIDER}
+10
查看文件
@@ -23,6 +23,10 @@ cp deploy/trialrun/.env.example deploy/trialrun/.env
- 如需快速回滚作业调度:将 `JOB_DRIVER` 改回 `cron` - 如需快速回滚作业调度:将 `JOB_DRIVER` 改回 `cron`
- 可选 Worker 拆分(F20):将 `JOBS_ENABLED=false` 并启用 `--profile worker`(worker 进程会强制跑作业) - 可选 Worker 拆分(F20):将 `JOBS_ENABLED=false` 并启用 `--profile worker`(worker 进程会强制跑作业)
存储建议:
- 默认 `STORAGE_DRIVER=local`(票据文件落到 `deploy/trialrun/data/storage`)
- 可选启用 MinIO(F21):`STORAGE_DRIVER=s3` 并启用 `--profile minio`
### 1.2 启动数据库与服务 ### 1.2 启动数据库与服务
```bash ```bash
cd /path/to/hyb-oa cd /path/to/hyb-oa
@@ -30,6 +34,12 @@ docker compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/
docker compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d --build server docker compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env up -d --build server
``` ```
可选:启用 MinIO(对象存储,F21):
```bash
cd /path/to/hyb-oa
docker compose -f deploy/trialrun/docker-compose.yml --env-file deploy/trialrun/.env --profile minio up -d minio minio_init
```
可选:启用 Worker 进程(推荐在试运行/多进程模式使用): 可选:启用 Worker 进程(推荐在试运行/多进程模式使用):
```bash ```bash
cd /path/to/hyb-oa cd /path/to/hyb-oa
@@ -22,7 +22,7 @@ F19b,作业迁移(OCR -> pg-boss),"将 OCR 异步处理迁移到 pg-boss:
F19c,作业迁移(委派回收 -> pg-boss),"将委派过期回收迁移到 pg-boss:保持 FOR UPDATE SKIP LOCKED + UPDATE RETURNING 原子认领语义","多进程并发下审计不重复写;回收逻辑不变;回归全绿","cd server && RUN_DELEGATION_CONCURRENCY_CHECK=1 ./scripts/regression-workflow-actions.sh",manual,DONE,DONE,DONE,"server/src/jobs/pgBossJobs.js | server/src/jobs/delegationRevokeJob.js | server/src/jobs/jobSystem.js | server/README.md","[""F19""]","仍保留原子认领+审计只基于 RETURNING 结果集" F19c,作业迁移(委派回收 -> pg-boss),"将委派过期回收迁移到 pg-boss:保持 FOR UPDATE SKIP LOCKED + UPDATE RETURNING 原子认领语义","多进程并发下审计不重复写;回收逻辑不变;回归全绿","cd server && RUN_DELEGATION_CONCURRENCY_CHECK=1 ./scripts/regression-workflow-actions.sh",manual,DONE,DONE,DONE,"server/src/jobs/pgBossJobs.js | server/src/jobs/delegationRevokeJob.js | server/src/jobs/jobSystem.js | server/README.md","[""F19""]","仍保留原子认领+审计只基于 RETURNING 结果集"
F19d,作业迁移(超期归档 -> pg-boss),"将 90 天游离归档迁移到 pg-boss(替代 node-cron),统一重试/并发/可观测字段","作业可重复执行幂等;多进程不重复处理;回归全绿","cd server && ./scripts/regression-ocr.sh && ./scripts/regression-all.sh",manual,DONE,DONE,DONE,"server/src/jobs/pgBossJobs.js | server/src/jobs/invoiceArchiveJob.js | server/src/jobs/jobSystem.js | server/README.md","[""F19"",""F12""]","invoice.archive-unused 默认按 INVOICE_ARCHIVE_CRON 调度;回滚:JOB_DRIVER=cron" F19d,作业迁移(超期归档 -> pg-boss),"将 90 天游离归档迁移到 pg-boss(替代 node-cron),统一重试/并发/可观测字段","作业可重复执行幂等;多进程不重复处理;回归全绿","cd server && ./scripts/regression-ocr.sh && ./scripts/regression-all.sh",manual,DONE,DONE,DONE,"server/src/jobs/pgBossJobs.js | server/src/jobs/invoiceArchiveJob.js | server/src/jobs/jobSystem.js | server/README.md","[""F19"",""F12""]","invoice.archive-unused 默认按 INVOICE_ARCHIVE_CRON 调度;回滚:JOB_DRIVER=cron"
F20,Worker进程拆分(可选),"提供 API 进程与 Worker 进程拆分的部署方式(compose 开关),保持单机也可跑","开启后:API 不跑作业;Worker 只跑作业;两者共享 DATABASE_URL;功能不回归","manual: docker compose 起两个进程并跑回归脚本",manual,DONE,DONE,DONE,"deploy/trialrun/docker-compose.yml | server/src/index.js | server/README.md","[""F19""]","仍保持单机;为未来横向扩展铺路" F20,Worker进程拆分(可选),"提供 API 进程与 Worker 进程拆分的部署方式(compose 开关),保持单机也可跑","开启后:API 不跑作业;Worker 只跑作业;两者共享 DATABASE_URL;功能不回归","manual: docker compose 起两个进程并跑回归脚本",manual,DONE,DONE,DONE,"deploy/trialrun/docker-compose.yml | server/src/index.js | server/README.md","[""F19""]","仍保持单机;为未来横向扩展铺路"
F21,对象存储(MinIO/S3兼容),"存储从本地目录演进到 S3 兼容(先 MinIO),保持 signed-url 契约不变或提供兼容层","上传/预览/签名URL 在 MinIO 下可用;回归与 E2E 通过;迁移路径清晰可回滚","manual: trialrun 环境启用 MinIO 并完成一次上传预览",manual,TODO,TODO,TODO,"server/src/services/storage/* | deploy/trialrun/* | server/README.md","[""F20""]","优先用 @aws-sdk/client-s3;单机优先 MinIO" F21,对象存储(MinIO/S3兼容),"存储从本地目录演进到 S3 兼容(先 MinIO),保持 signed-url 契约不变或提供兼容层","上传/预览/签名URL 在 MinIO 下可用;回归与 E2E 通过;迁移路径清晰可回滚","manual: trialrun 环境启用 MinIO 并完成一次上传预览",manual,DONE,DONE,DONE,"server/src/services/storage/* | deploy/trialrun/* | server/README.md","[""F20""]","优先用 @aws-sdk/client-s3;单机优先 MinIO"
F22,可观测性增强(metrics/日志),"新增 /metrics(Prometheus)与更完整的结构化日志字段规范(不落敏感)","metrics 可抓取关键指标(HTTP/作业/DB);日志具备 requestId/actorId/targetId 等字段","manual: curl /metrics 并核对字段;触发作业观察指标变化",manual,DONE,DONE,DONE,"server/src/app.js | server/src/observability/* | deploy/trialrun/*","[""F19""]","优先用 prom-client;OTel 放更后" F22,可观测性增强(metrics/日志),"新增 /metrics(Prometheus)与更完整的结构化日志字段规范(不落敏感)","metrics 可抓取关键指标(HTTP/作业/DB);日志具备 requestId/actorId/targetId 等字段","manual: curl /metrics 并核对字段;触发作业观察指标变化",manual,DONE,DONE,DONE,"server/src/app.js | server/src/observability/* | deploy/trialrun/*","[""F19""]","优先用 prom-client;OTel 放更后"
F23,SSO/OIDC(可选,P7+),"支持 OIDC 登录(企业 SSO),保留现有密码登录作为 fallback","可通过 OIDC 登录并绑定员工;权限模型不变;可配置开关","manual: 配置 OIDC provider 后走一次登录回调",manual,TODO,TODO,TODO,"server/src/routes/auth.js | server/src/services/oidc/* | server/README.md","[""F1""]","优先复用 openid-client/passport 策略" F23,SSO/OIDC(可选,P7+),"支持 OIDC 登录(企业 SSO),保留现有密码登录作为 fallback","可通过 OIDC 登录并绑定员工;权限模型不变;可配置开关","manual: 配置 OIDC provider 后走一次登录回调",manual,TODO,TODO,TODO,"server/src/routes/auth.js | server/src/services/oidc/* | server/README.md","[""F1""]","优先复用 openid-client/passport 策略"
F24,工作流扩展到其他实体(P7+),"抽象 entity_type 扩展到请假/采购等(先骨架:definition+instance+task 复用)","新实体可创建 instance 并走同审批动作;不影响报销既有链路","manual: seed 一个 leave 定义并跑一条最小链路",manual,TODO,TODO,TODO,"server/src/routes/workflow.js | server/src/routes/leave.js | server/prisma/schema.prisma","[""F19""]","只做骨架与复用点,不做完整业务表单" F24,工作流扩展到其他实体(P7+),"抽象 entity_type 扩展到请假/采购等(先骨架:definition+instance+task 复用)","新实体可创建 instance 并走同审批动作;不影响报销既有链路","manual: seed 一个 leave 定义并跑一条最小链路",manual,TODO,TODO,TODO,"server/src/routes/workflow.js | server/src/routes/leave.js | server/prisma/schema.prisma","[""F19""]","只做骨架与复用点,不做完整业务表单"
1 ID Title Description Acceptance Test_Method Tools Dev_Status Review1_Status Regression_Status Files Dependencies Notes
22 F19c 作业迁移(委派回收 -> pg-boss) 将委派过期回收迁移到 pg-boss:保持 FOR UPDATE SKIP LOCKED + UPDATE RETURNING 原子认领语义 多进程并发下审计不重复写;回收逻辑不变;回归全绿 cd server && RUN_DELEGATION_CONCURRENCY_CHECK=1 ./scripts/regression-workflow-actions.sh manual DONE DONE DONE server/src/jobs/pgBossJobs.js | server/src/jobs/delegationRevokeJob.js | server/src/jobs/jobSystem.js | server/README.md ["F19"] 仍保留原子认领+审计只基于 RETURNING 结果集
23 F19d 作业迁移(超期归档 -> pg-boss) 将 90 天游离归档迁移到 pg-boss(替代 node-cron),统一重试/并发/可观测字段 作业可重复执行幂等;多进程不重复处理;回归全绿 cd server && ./scripts/regression-ocr.sh && ./scripts/regression-all.sh manual DONE DONE DONE server/src/jobs/pgBossJobs.js | server/src/jobs/invoiceArchiveJob.js | server/src/jobs/jobSystem.js | server/README.md ["F19","F12"] invoice.archive-unused 默认按 INVOICE_ARCHIVE_CRON 调度;回滚:JOB_DRIVER=cron
24 F20 Worker进程拆分(可选) 提供 API 进程与 Worker 进程拆分的部署方式(compose 开关),保持单机也可跑 开启后:API 不跑作业;Worker 只跑作业;两者共享 DATABASE_URL;功能不回归 manual: docker compose 起两个进程并跑回归脚本 manual DONE DONE DONE deploy/trialrun/docker-compose.yml | server/src/index.js | server/README.md ["F19"] 仍保持单机;为未来横向扩展铺路
25 F21 对象存储(MinIO/S3兼容) 存储从本地目录演进到 S3 兼容(先 MinIO),保持 signed-url 契约不变或提供兼容层 上传/预览/签名URL 在 MinIO 下可用;回归与 E2E 通过;迁移路径清晰可回滚 manual: trialrun 环境启用 MinIO 并完成一次上传预览 manual TODO DONE TODO DONE TODO DONE server/src/services/storage/* | deploy/trialrun/* | server/README.md ["F20"] 优先用 @aws-sdk/client-s3;单机优先 MinIO
26 F22 可观测性增强(metrics/日志) 新增 /metrics(Prometheus)与更完整的结构化日志字段规范(不落敏感) metrics 可抓取关键指标(HTTP/作业/DB);日志具备 requestId/actorId/targetId 等字段 manual: curl /metrics 并核对字段;触发作业观察指标变化 manual DONE DONE DONE server/src/app.js | server/src/observability/* | deploy/trialrun/* ["F19"] 优先用 prom-client;OTel 放更后
27 F23 SSO/OIDC(可选,P7+) 支持 OIDC 登录(企业 SSO),保留现有密码登录作为 fallback 可通过 OIDC 登录并绑定员工;权限模型不变;可配置开关 manual: 配置 OIDC provider 后走一次登录回调 manual TODO TODO TODO server/src/routes/auth.js | server/src/services/oidc/* | server/README.md ["F1"] 优先复用 openid-client/passport 策略
28 F24 工作流扩展到其他实体(P7+) 抽象 entity_type 扩展到请假/采购等(先骨架:definition+instance+task 复用) 新实体可创建 instance 并走同审批动作;不影响报销既有链路 manual: seed 一个 leave 定义并跑一条最小链路 manual TODO TODO TODO server/src/routes/workflow.js | server/src/routes/leave.js | server/prisma/schema.prisma ["F19"] 只做骨架与复用点,不做完整业务表单
+7
查看文件
@@ -5,6 +5,13 @@ JWT_SECRET="change_me"
# DEV_SHOW_PASSWORD_RESET_CODE="true" # DEV_SHOW_PASSWORD_RESET_CODE="true"
APP_TIMEZONE="Asia/Shanghai" APP_TIMEZONE="Asia/Shanghai"
STORAGE_DIR="generated/storage" STORAGE_DIR="generated/storage"
# STORAGE_DRIVER="local" # local|s3(MinIO/S3 兼容)
# S3_ENDPOINT="" # 例如:http://127.0.0.1:9000(MinIO)
# S3_REGION="us-east-1"
# S3_BUCKET="hyb-oa"
# S3_ACCESS_KEY_ID=""
# S3_SECRET_ACCESS_KEY=""
# S3_FORCE_PATH_STYLE="true" # MinIO 建议 true
SIGNED_URL_TTL_SECONDS="600" SIGNED_URL_TTL_SECONDS="600"
MAX_UPLOAD_SIZE_BYTES="10485760" MAX_UPLOAD_SIZE_BYTES="10485760"
+12
查看文件
@@ -113,6 +113,18 @@ JOBS_ENABLED=true npm run worker
- 启用 worker profile:`--profile worker` - 启用 worker profile:`--profile worker`
- 同时将 server 的 `JOBS_ENABLED=false`(worker 进程内强制为 true) - 同时将 server 的 `JOBS_ENABLED=false`(worker 进程内强制为 true)
### 2.3.5 存储驱动(local vs S3/MinIO)(F21)
> 目标:把“票据文件存储”从本地目录可选演进到 S3 兼容(单机优先 MinIO),保持签名 URL 契约不变。
- 默认:`STORAGE_DRIVER=local`(写入 `STORAGE_DIR`)
- 可选:`STORAGE_DRIVER=s3`(写入 S3/MinIO)
- 需要配置:`S3_ENDPOINT/S3_REGION/S3_BUCKET/S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY/S3_FORCE_PATH_STYLE`
- 为兼容混合存储,`invoice_pool.storage_path` 对 S3 会写成 `s3:<key>`(无 schema 变更)
签名 URL 兼容策略:
- `/api/invoice-pool/:id/signed-url` 仍返回本系统 URL(带 token)
- `/api/invoice-pool/:id/file` 内部按 `storage_path` 自动从本地或 S3 读取并返回文件流
### 2.4 启动服务 ### 2.4 启动服务
```bash ```bash
cd /home/shunli/Projects/hyb-oa/server cd /home/shunli/Projects/hyb-oa/server
+2207
查看文件
文件差异内容过多而无法显示 加载差异
+1
查看文件
@@ -26,6 +26,7 @@
"author": "", "author": "",
"license": "ISC", "license": "ISC",
"dependencies": { "dependencies": {
"@aws-sdk/client-s3": "^3.985.0",
"@fastify/cookie": "^11.0.2", "@fastify/cookie": "^11.0.2",
"@fastify/cors": "^11.2.0", "@fastify/cors": "^11.2.0",
"@fastify/helmet": "^13.0.2", "@fastify/helmet": "^13.0.2",
+7
查看文件
@@ -50,6 +50,13 @@ function getConfig() {
appTimezone: process.env.APP_TIMEZONE ?? "Asia/Shanghai", appTimezone: process.env.APP_TIMEZONE ?? "Asia/Shanghai",
storageDir: path.resolve(serverRoot, storageDir), storageDir: path.resolve(serverRoot, storageDir),
storageDriver: process.env.STORAGE_DRIVER ?? "local",
s3Endpoint: process.env.S3_ENDPOINT ?? "",
s3Region: process.env.S3_REGION ?? "us-east-1",
s3Bucket: process.env.S3_BUCKET ?? "",
s3AccessKeyId: process.env.S3_ACCESS_KEY_ID ?? "",
s3SecretAccessKey: process.env.S3_SECRET_ACCESS_KEY ?? "",
s3ForcePathStyle: process.env.S3_FORCE_PATH_STYLE === "true",
serveClientDist: process.env.SERVE_CLIENT_DIST === "true", serveClientDist: process.env.SERVE_CLIENT_DIST === "true",
clientDistDir: path.resolve(serverRoot, clientDistDir), clientDistDir: path.resolve(serverRoot, clientDistDir),
signedUrlTtlSeconds: Number(process.env.SIGNED_URL_TTL_SECONDS ?? 600), signedUrlTtlSeconds: Number(process.env.SIGNED_URL_TTL_SECONDS ?? 600),
+5 -4
查看文件
@@ -1,8 +1,7 @@
const fs = require("node:fs/promises");
const path = require("node:path");
const cron = require("node-cron"); const cron = require("node-cron");
const { runOcrForInvoice } = require("../services/ocr"); const { runOcrForInvoice } = require("../services/ocr");
const { readFileBuffer } = require("../services/storage");
const OCR_DUPLICATE_ERROR = "duplicate_invoice_code_number"; const OCR_DUPLICATE_ERROR = "duplicate_invoice_code_number";
@@ -222,8 +221,10 @@ async function markInvoiceAsFailed(prisma, claimedInvoice, errorMessage, config)
async function processClaimedInvoice(prisma, config, claimedInvoice, workerId, logger) { async function processClaimedInvoice(prisma, config, claimedInvoice, workerId, logger) {
try { try {
const absolutePath = path.resolve(config.storageDir, claimedInvoice.storage_path); const fileBuffer = await readFileBuffer({
const fileBuffer = await fs.readFile(absolutePath); config,
storagePath: claimedInvoice.storage_path,
});
const imageBase64 = fileBuffer.toString("base64"); const imageBase64 = fileBuffer.toString("base64");
const { ocrResultId, normalized } = await getOrCreateOcrResult( const { ocrResultId, normalized } = await getOrCreateOcrResult(
+28 -23
查看文件
@@ -1,6 +1,4 @@
const crypto = require("node:crypto"); const crypto = require("node:crypto");
const fs = require("node:fs/promises");
const fsSync = require("node:fs");
const path = require("node:path"); const path = require("node:path");
const { z } = require("zod"); const { z } = require("zod");
@@ -13,6 +11,7 @@ const { signPayload, verifyToken } = require("../utils/signedUrl");
const { toMoneyString } = require("../utils/workflow"); const { toMoneyString } = require("../utils/workflow");
const { findActiveDelegationForDelegate } = require("../utils/delegation"); const { findActiveDelegationForDelegate } = require("../utils/delegation");
const { isOcrEnabledForType } = require("../services/ocr"); const { isOcrEnabledForType } = require("../services/ocr");
const { openReadStream, StorageNotFoundError, writeBuffer } = require("../services/storage");
const MIME_ALLOWLIST = new Set([ const MIME_ALLOWLIST = new Set([
"application/pdf", "application/pdf",
@@ -565,11 +564,13 @@ async function invoicePoolRoutes(fastify) {
const fileSha256 = sha256Hex(fileBuffer); const fileSha256 = sha256Hex(fileBuffer);
const datePrefix = compactDateString(fastify.config.appTimezone); const datePrefix = compactDateString(fastify.config.appTimezone);
const extension = path.extname(filePart.filename ?? "").toLowerCase(); const extension = path.extname(filePart.filename ?? "").toLowerCase();
const storagePath = path.join(datePrefix, `${crypto.randomUUID()}${extension}`); const objectKey = path.posix.join(datePrefix, `${crypto.randomUUID()}${extension}`);
const absoluteStoragePath = path.resolve(fastify.config.storageDir, storagePath); const storagePath = await writeBuffer({
config: fastify.config,
await fs.mkdir(path.dirname(absoluteStoragePath), { recursive: true }); key: objectKey,
await fs.writeFile(absoluteStoragePath, fileBuffer); buffer: fileBuffer,
contentType: filePart.mimetype,
});
const status = isOcrEnabledForType(fastify.config, ocrType) ? "pending_ocr" : "available"; const status = isOcrEnabledForType(fastify.config, ocrType) ? "pending_ocr" : "available";
@@ -1918,23 +1919,27 @@ async function invoicePoolRoutes(fastify) {
}); });
} }
const storageRoot = path.resolve(fastify.config.storageDir); let stream = null;
const absolutePath = path.resolve(storageRoot, invoice.storage_path);
if (!absolutePath.startsWith(`${storageRoot}${path.sep}`)) {
throw new AppError({
code: "PERM_001",
message: "无权限",
statusCode: 403,
});
}
try { try {
await fs.access(absolutePath); const opened = await openReadStream({
} catch { config: fastify.config,
storagePath: invoice.storage_path,
});
stream = opened.stream;
} catch (error) {
if (error instanceof StorageNotFoundError) {
throw new AppError({
code: "SYS_404",
message: "Not Found",
statusCode: 404,
});
}
throw new AppError({ throw new AppError({
code: "SYS_404", code: "SYS_001",
message: "Not Found", message: "服务器内部错误",
statusCode: 404, statusCode: 500,
details: {},
}); });
} }
@@ -1960,7 +1965,7 @@ async function invoicePoolRoutes(fastify) {
"content-disposition", "content-disposition",
`inline; filename=\"${sanitizeFilename(invoice.original_name)}\"`, `inline; filename=\"${sanitizeFilename(invoice.original_name)}\"`,
); );
return reply.send(fsSync.createReadStream(absolutePath)); return reply.send(stream);
}); });
fastify.post( fastify.post(
+216
查看文件
@@ -0,0 +1,216 @@
const fs = require("node:fs/promises");
const fsSync = require("node:fs");
const path = require("node:path");
const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
class StorageNotFoundError extends Error {
constructor(message) {
super(message);
this.name = "StorageNotFoundError";
}
}
class StorageConfigError extends Error {
constructor(message) {
super(message);
this.name = "StorageConfigError";
}
}
function parseStoragePath(storagePath) {
const raw = typeof storagePath === "string" ? storagePath : "";
if (raw.startsWith("s3:")) {
return { driver: "s3", key: raw.slice("s3:".length) };
}
return { driver: "local", key: raw };
}
function ensureSafeLocalPath(storageRoot, relativeKey) {
const root = path.resolve(storageRoot);
const absolutePath = path.resolve(root, relativeKey);
if (!absolutePath.startsWith(`${root}${path.sep}`)) {
throw new StorageConfigError("Invalid storage_path (path traversal detected)");
}
return absolutePath;
}
function resolveS3Config(config) {
if (!config?.s3Bucket) {
throw new StorageConfigError("Missing S3_BUCKET for s3 storage");
}
return {
bucket: config.s3Bucket,
region: config.s3Region || "us-east-1",
endpoint: config.s3Endpoint || undefined,
credentials:
config.s3AccessKeyId && config.s3SecretAccessKey
? {
accessKeyId: config.s3AccessKeyId,
secretAccessKey: config.s3SecretAccessKey,
}
: undefined,
forcePathStyle: Boolean(config.s3ForcePathStyle),
};
}
let cachedS3Client = null;
let cachedS3ClientKey = null;
function getS3Client(config) {
const s3 = resolveS3Config(config);
const cacheKey = JSON.stringify({
bucket: s3.bucket,
region: s3.region,
endpoint: s3.endpoint ?? "",
accessKeyId: s3.credentials?.accessKeyId ?? "",
forcePathStyle: s3.forcePathStyle,
});
if (cachedS3Client && cacheKey === cachedS3ClientKey) {
return cachedS3Client;
}
cachedS3ClientKey = cacheKey;
cachedS3Client = new S3Client({
region: s3.region,
endpoint: s3.endpoint,
credentials: s3.credentials,
forcePathStyle: s3.forcePathStyle,
});
return cachedS3Client;
}
async function streamToBuffer(stream) {
const chunks = [];
for await (const chunk of stream) {
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
}
return Buffer.concat(chunks);
}
async function writeBuffer({ config, key, buffer, contentType }) {
if (!Buffer.isBuffer(buffer) || buffer.length === 0) {
throw new StorageConfigError("Invalid buffer");
}
const driver = config?.storageDriver ?? "local";
if (driver === "s3") {
const s3 = resolveS3Config(config);
const client = getS3Client(config);
await client.send(
new PutObjectCommand({
Bucket: s3.bucket,
Key: key,
Body: buffer,
ContentType: contentType || "application/octet-stream",
}),
);
// Use a lightweight prefix to support mixed local+s3 records without schema changes.
return `s3:${key}`;
}
const absolutePath = ensureSafeLocalPath(config.storageDir, key);
await fs.mkdir(path.dirname(absolutePath), { recursive: true });
await fs.writeFile(absolutePath, buffer);
return key;
}
async function readFileBuffer({ config, storagePath }) {
const parsed = parseStoragePath(storagePath);
if (!parsed.key) {
throw new StorageNotFoundError("missing_storage_path");
}
if (parsed.driver === "s3") {
const s3 = resolveS3Config(config);
const client = getS3Client(config);
try {
const result = await client.send(
new GetObjectCommand({
Bucket: s3.bucket,
Key: parsed.key,
}),
);
if (!result?.Body) {
throw new StorageNotFoundError("s3_body_missing");
}
return streamToBuffer(result.Body);
} catch (error) {
const code = error?.$metadata?.httpStatusCode;
if (code === 404 || error?.name === "NoSuchKey") {
throw new StorageNotFoundError("s3_not_found");
}
throw error;
}
}
const absolutePath = ensureSafeLocalPath(config.storageDir, parsed.key);
try {
return await fs.readFile(absolutePath);
} catch (error) {
if (error?.code === "ENOENT") {
throw new StorageNotFoundError("local_not_found");
}
throw error;
}
}
async function openReadStream({ config, storagePath }) {
const parsed = parseStoragePath(storagePath);
if (!parsed.key) {
throw new StorageNotFoundError("missing_storage_path");
}
if (parsed.driver === "s3") {
const s3 = resolveS3Config(config);
const client = getS3Client(config);
try {
const result = await client.send(
new GetObjectCommand({
Bucket: s3.bucket,
Key: parsed.key,
}),
);
if (!result?.Body) {
throw new StorageNotFoundError("s3_body_missing");
}
return { stream: result.Body };
} catch (error) {
const code = error?.$metadata?.httpStatusCode;
if (code === 404 || error?.name === "NoSuchKey") {
throw new StorageNotFoundError("s3_not_found");
}
throw error;
}
}
const absolutePath = ensureSafeLocalPath(config.storageDir, parsed.key);
try {
await fs.access(absolutePath);
} catch (error) {
if (error?.code === "ENOENT") {
throw new StorageNotFoundError("local_not_found");
}
throw error;
}
return { stream: fsSync.createReadStream(absolutePath) };
}
module.exports = {
StorageConfigError,
StorageNotFoundError,
openReadStream,
parseStoragePath,
readFileBuffer,
writeBuffer,
};