mirror of
https://github.com/jxxghp/MoviePilot.git
synced 2026-09-01 21:47:50 +08:00
refactor: add durable transfer planning checkpoints
This commit is contained in:
@@ -69,17 +69,17 @@ MoviePilot V3 已经形成较清晰的模块化单体:`foundation`、`domain`
|
||||
|
||||
| 指标 | 当前值 | 解释 |
|
||||
|---|---:|---|
|
||||
| 宿主 Python 模块 / 内部依赖边 | 836 / 6,827 | `dependency-baseline.json` 当前快照 |
|
||||
| 宿主 Python 模块 / 内部依赖边 | 836 / 6,832 | `dependency-baseline.json` 当前快照 |
|
||||
| 非平凡 SCC | 2 | 新增 Chain 包根环;另一个是隔离的 29 模块 TMDB 移植包环 |
|
||||
| 跨层 DB 边界债务 | 0 | Application、Chain、API、Agent、Runtime、Workflow 到 DB 的受控债务均为零 |
|
||||
| Model/Oper 事务债务 | 0 | 自建 Session、自动事务装饰器、直接 commit/rollback 等基线均为零 |
|
||||
| Module Contract | 215 specs / 214 methods / 264 calls | 动态方法名为 0;仍有 50 个结果形状为 `ANY` |
|
||||
| Module Contract | 217 specs / 215 methods / 265 calls | 动态方法名为 0;内部 planning 合同不进入插件调度,旧 transfer 只保留 provider ABI |
|
||||
| Event Contract | 53 | 均已有 payload model,但当前全部是 diagnostic enforcement |
|
||||
| Python 源码量 | 约 271,400 行 | 60 个文件超过 1,000 行,14 个超过 2,000 行 |
|
||||
| 长方法 | 281 个超过 80 行 | 67 个超过 150 行,23 个超过 250 行;大量是私有方法 |
|
||||
| 全量 mypy 历史债务 | 11,983 / 601 文件 | strict frontier 当前只覆盖 41 个文件,且 ratchet 已新增 2 个错误 |
|
||||
| Ruff 历史诊断 | 967 | 低水位门禁通过,但规则集只覆盖 `E4/E7/E9/F/I` |
|
||||
| 覆盖率低水位 | Application 77.85%,Domain 79.24% | Chain、Runtime、Agent、Adapter、Startup 未进入包级覆盖率门禁 |
|
||||
| Ruff 历史诊断 | 937 | 低水位门禁通过,但规则集只覆盖 `E4/E7/E9/F/I` |
|
||||
| 覆盖率低水位 | Application 78.06%,Domain 79.29% | Chain、Runtime、Agent、Adapter、Startup 未进入包级覆盖率门禁 |
|
||||
|
||||
### 3.3 热点文件
|
||||
|
||||
@@ -160,7 +160,8 @@ MoviePilot V3 已经形成较清晰的模块化单体:`foundation`、`domain`
|
||||
- 架构总览此前仍记录 811 模块、6,572 条边和 1 个 SCC,已经落后于当前基线。
|
||||
- Event consumer 扫描曾把任意同名 `.register()` 调用当成事件注册;S0-L2.5 已改为证明
|
||||
canonical EventManager receiver,10 个动态误报归零并保留唯一 workflow 动态注册。
|
||||
- S0-L2.6 已将 producer/consumer 合并为逐调用事实源:99 个 producer(98 静态、1 动态)与
|
||||
- S0-L2.6 已将 producer/consumer 合并为逐调用事实源;本轮统一 Transfer 事件发送点后为
|
||||
97 个 producer(96 静态、1 动态)与
|
||||
17 个 consumer(16 静态、1 动态);consumer 由不可自动写入的精确人工 policy 管理。
|
||||
|
||||
**目标与步骤**
|
||||
@@ -188,7 +189,9 @@ MoviePilot V3 已经形成较清晰的模块化单体:`foundation`、`domain`
|
||||
|
||||
- `S1-L1.1 Durable admission`:`VERIFIED`。已交付 persist-before-enqueue、Application-owned typed Port、
|
||||
DB adapter 与可逆 migration,宿主退出 raw/`Any` `TransferPendingOper` admission 路径。
|
||||
- `S1-L1.2 Planning checkpoint`:`PLANNED`。持久化稳定任务身份、整理模式、规划状态和目标 checkpoint。
|
||||
- `S1-L1.2 Planning checkpoint`:`VERIFIED`。版本化请求与指纹先准入;无 legacy provider 时通过
|
||||
`accepted -> planned` CAS 提交完整目标和有序操作,有 provider 时先提交 `provider_pending`,全部
|
||||
返回空后再以第二次 CAS 提交 `planned`;planned 重放只消费冻结上下文和目标。
|
||||
- `S1-L1.3 Lease 与恢复调度`:`PLANNED`。交付 claim/lease/heartbeat/attempt、过期接管与唯一恢复入口。
|
||||
- `S1-L1.4 幂等执行与终态结算`:`PLANNED`。交付文件/历史幂等、唯一 retry owner 和
|
||||
`manual_review` 语义。
|
||||
@@ -201,28 +204,35 @@ MoviePilot V3 已经形成较清晰的模块化单体:`foundation`、`domain`
|
||||
异常均不会伪装成重复任务成功,失败记录可供恢复。
|
||||
- 宿主 canonical Chain 只取得类型化 `TransferAdmissionRepository`;旧 Oper API 仅保留给统一兼容层,
|
||||
插件公开 `TransferTask.to_dict()` 字段未增加内部任务标识。
|
||||
- worker 未知异常最终也会在 `app/chain/transfer.py:1107-1113,1262-1272` 删除 pending。
|
||||
- `app/db/models/transferpending.py:9-34` 只有 `storage/src_path/created_at`,没有目标、模式、
|
||||
step、lease、attempt、last_error,无法判定“文件已移动、历史未提交”等中间态。
|
||||
- `S1-L1.2` 已消除 checkpoint 前的文件副作用:目标路径、操作顺序及 resolved 识别上下文原子落库后,
|
||||
执行器才允许触发 cleanup、建目录和复制/移动;规划失败保留 `accepted` 并记录 `last_error`。
|
||||
- 旧插件 `transfer` provider 的身份、顺序和原始 ABI 参数先冻结为 `provider_pending`;提交后才精确
|
||||
解析并严格执行,缺失或异常不 fallback。全部返回空后才生成宿主计划,并以第二次 CAS 提升为
|
||||
`planned` 后执行。旧 caller 只经 `ChainBase.transfer` 注入式兼容门面进入同一 durable command,
|
||||
宿主 FileManager/TransHandler 的旧执行入口已删除。
|
||||
- `TransferPending` 现在可区分 `accepted/provider_pending/planned`,但尚无
|
||||
claim/lease/heartbeat/attempt、逐步骤执行结果和 `manual_review`,仍无法判定“文件已移动、历史未提交”
|
||||
等后续中间态。
|
||||
- 这与 `docs/adr/0007-background-action-reliability.md:123-139` 对 E3 的稳定身份、步骤状态、
|
||||
lease/heartbeat 和人工恢复要求不一致。
|
||||
|
||||
**目标与步骤**
|
||||
|
||||
- [ ] 先在独立持久事务中 commit pending,再尝试放入内存队列;数据库事务不能与 `queue.Queue`
|
||||
- [x] 先在独立持久事务中 commit pending,再尝试放入内存队列;数据库事务不能与 `queue.Queue`
|
||||
原子提交,入队失败时必须保留 pending 供重放。
|
||||
- [ ] 初始登记保存稳定源身份、模式、状态和 attempt/lease;目标在规划完成后以 planning checkpoint
|
||||
更新,不能要求任务刚入队时已经具备尚未计算的目标路径。
|
||||
- [x] 初始登记保存稳定源身份、版本化请求和状态;目标与有序操作在纯规划完成后以 planning
|
||||
checkpoint 原子更新,任何文件副作用不得早于该提交。
|
||||
- [ ] 增加 claim/lease/heartbeat/attempt 与过期接管,同一任务同时只能有一个 worker owner。
|
||||
- [ ] 设计幂等文件操作和历史提交;只有所有必要步骤达到持久终态后才能删除记录。
|
||||
- [ ] 在持久状态机与现有失败历史/AI retry 之间指定唯一 retry owner,定义旧记录迁移和兼容规则。
|
||||
- [ ] E3 失败使用持久 `failed/manual_review`、最后稳定 checkpoint 和补偿边界,不直接套用 E2
|
||||
Outbox 的 dead-letter 语义;禁止按年龄通用清理 pending。
|
||||
- [ ] 数据模型变更必须配套 Alembic migration,并验证升级与降级路径。
|
||||
- [x] 当前 admission/planning 数据模型变更均配套 Alembic migration,并验证升级、降级和中断重跑。
|
||||
|
||||
**故障注入验收**
|
||||
|
||||
- [ ] 登记后、内存入队前崩溃,重启可继续。
|
||||
- [ ] 持久登记成功但内存入队失败,重启可继续。
|
||||
- [x] 登记后、内存入队前崩溃,重启可继续。
|
||||
- [x] 持久登记成功但内存入队失败,重启可继续。
|
||||
- [ ] 文件移动后、历史提交前崩溃,在支持稳定身份/幂等操作的存储上不重复移动且可补齐历史。
|
||||
- [ ] worker 未知异常和 lease 超时后保留可诊断状态。
|
||||
- [ ] 重复回放、重复消息和人工重试都保持幂等。
|
||||
|
||||
@@ -369,8 +369,8 @@ collector 只接受 canonical `eventmanager`、`EventManager()` 及其有限别
|
||||
`register`/`add_event_listener`;当前宿主有 16 个静态注册点,另保留 1 个由工作流配置驱动的
|
||||
真实动态注册。`app/plugins/**` 插件副本不进入宿主事实。
|
||||
|
||||
生产者与消费者共用 `scripts/architecture/event_facts.py` 这一份逐调用事实源。当前宿主有 99 个
|
||||
生产调用,其中 98 个静态解析为 100 个事件引用,只有 `Command.send_plugin_event` 的插件事件类型
|
||||
生产者与消费者共用 `scripts/architecture/event_facts.py` 这一份逐调用事实源。当前宿主有 97 个
|
||||
生产调用,其中 96 个静态解析为 98 个事件引用,只有 `Command.send_plugin_event` 的插件事件类型
|
||||
保持动态;17 个消费注册中 16 个静态、1 个动态。生成的
|
||||
`runtime-contract-baseline.json` 保存 line-free 事实、数量和枚举索引;人工维护的
|
||||
`runtime-contract-policy.json` 只批准 consumer 的精确 fingerprint、owner 和理由,任何新增、替换、
|
||||
@@ -690,7 +690,7 @@ flowchart LR
|
||||
stream/vendor/diagnostic/control-plane 事实是精确 containment。每条初始边的指纹由测试独立冻结,
|
||||
bindings/uses 变化、分类互换、通配导入和初始边增长都会失败;债务删除时同步删除冻结项以禁止恢复,
|
||||
`--write-host` 不会改写人工 policy 或冻结上界。
|
||||
- `event_facts` 是生产者/消费者唯一收集源;运行快照记录 99 个生产调用和 17 个消费注册,
|
||||
- `event_facts` 是生产者/消费者唯一收集源;运行快照记录 97 个生产调用和 17 个消费注册,
|
||||
consumer 的 17 个唯一 fingerprint 另由只读人工 policy 精确准入。CI 将语义 policy 与生成快照
|
||||
分成独立步骤,前者不能通过刷新后者绕过。
|
||||
- 任何所有权迁移必须同步更新:canonical 导入、`app/runtime/compat/manifest.py`、
|
||||
@@ -704,13 +704,13 @@ flowchart LR
|
||||
|
||||
| 指标 | 当前值 |
|
||||
|---|---:|
|
||||
| Python 模块 | 835 |
|
||||
| 内部导入边 | 6,827 |
|
||||
| Python 模块 | 836 |
|
||||
| 内部导入边 | 6,832 |
|
||||
| 非平凡 SCC | 2(`ARCH-107` 临时 Chain 包根环;精确 containment 的 TMDB 移植包环) |
|
||||
| Direct egress | 66(12 条待迁移债务,54 条精确 containment) |
|
||||
| Module Contract V2 spec | 215(其中 214 个进入 `run_module` 观察面) |
|
||||
| Module Contract V2 spec | 217(其中 215 个进入 `run_module` 观察面) |
|
||||
| Event Contract | 53 |
|
||||
| Event producer / consumer | 99(98 静态、1 动态)/ 17(16 静态、1 动态) |
|
||||
| Event producer / consumer | 97(96 静态、1 动态)/ 17(16 静态、1 动态) |
|
||||
| Model/Oper 自动事务与自建 Session | 0 |
|
||||
| 组合根外 `SystemConfigOper()` | 0 |
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ G-ARCH 只有在以下条件全部满足后才可完成:
|
||||
| Leaf | 状态 | 依赖 | 完成定义 |
|
||||
|---|---|---|---|
|
||||
| S1-L1.1 Durable admission | `VERIFIED` | S0 | Application-owned typed Port + DB adapter + migration 落地;先持久 commit 再入队,入队失败保留可恢复记录;宿主不再通过 raw/`Any` `TransferPendingOper` 处理 admission |
|
||||
| S1-L1.2 Planning checkpoint | `PLANNED` | S1-L1.1 | 稳定任务身份、整理模式和 planning 状态持久化;目标路径只在规划完成后写入 checkpoint,任何文件副作用前已有可判定状态 |
|
||||
| S1-L1.2 Planning checkpoint | `VERIFIED` | S1-L1.1 | 版本化输入与指纹先持久化;无 legacy provider 时以 `accepted -> planned` CAS 提交完整计划,有 provider 时先提交 `provider_pending`,全部返回空后再以第二次 CAS 提交 `planned`;重放只执行冻结目标,所有文件副作用晚于对应 checkpoint commit |
|
||||
| S1-L1.3 Lease 与恢复调度 | `PLANNED` | S1-L1.2 | claim/lease/heartbeat/attempt 与过期接管规则落地;启动回放和同进程恢复共用唯一调度入口,同一任务同时只有一个 worker owner |
|
||||
| S1-L1.4 幂等执行与终态结算 | `PLANNED` | S1-L1.3 | 文件操作、历史提交和 checkpoint 可重放;唯一 retry owner 生效,未知外部结果进入 `manual_review`,仅完整终态删除 pending |
|
||||
| S1-L1.5 E3 全链收口 | `PLANNED` | S1-L1.4 | 崩溃矩阵、升级/降级、重复回放和插件 ABI 验收完整;旧 fail-open、重复状态与兼容层外旧入口删除,ARCH-102 债务归零 |
|
||||
@@ -144,7 +144,7 @@ G-ARCH 只有在以下条件全部满足后才可完成:
|
||||
| S4-L2 Event strict contract | `PLANNED` | S0-L2.6,S1-L6 | 宿主事件输入/输出按风险 strict,诊断例外只属于第三方插件兼容 |
|
||||
| S4-L3 Complexity v2 | `PLANNED` | S3 | 私有方法、class/file、圈复杂度进入门禁;所有超限通过职责拆分归零 |
|
||||
| S4-L4 全量 mypy 清零 | `PLANNED` | S3,S4-L1,S4-L2 | `mypy-baseline.json` 归零并删除债务接受路径,全宿主 strict 类型通过 |
|
||||
| S4-L5 Ruff 治理债务清零 | `PLANNED` | S3 | 当前受控 967 条诊断归零,规则集扩展经过独立审查且新增诊断为零 |
|
||||
| S4-L5 Ruff 治理债务清零 | `PLANNED` | S3 | 当前受控 937 条诊断归零,规则集扩展经过独立审查且新增诊断为零 |
|
||||
| S4-L6 Coverage/并发/质量证据 | `PLANNED` | S3,S4-L1,S4-L2 | 高风险包纳入 coverage;raw concurrency 分类清零;Module Quality 有真实 evidence test |
|
||||
|
||||
### S5:Plugin、Agent、Domain、Startup 与最终收口
|
||||
@@ -227,3 +227,50 @@ git diff --check
|
||||
`git diff --check` 全部通过。
|
||||
- failure injection 已覆盖 admission 失败不入队、batch/enqueue 失败保留记录、批次返回失败、
|
||||
queue -> worker -> terminal discard 稳定身份,以及 Legacy TransferTask 序列化字段不变。
|
||||
|
||||
### S1-L1.2 Planning checkpoint
|
||||
|
||||
**Status:** `VERIFIED`
|
||||
|
||||
**Outcome**
|
||||
|
||||
把 durable admission 推进为可独立恢复的 `accepted -> provider_pending -> planned` 状态:准入时冻结
|
||||
版本化请求 JSON 和 SHA-256 指纹;存在旧插件 provider 时先 CAS 提交精确身份、顺序和原始 ABI 参数,
|
||||
全部返回空后才由 FileManager 只读规划目标及有序叶操作,并通过第二次 CAS 提交宿主 checkpoint。
|
||||
任何对应 checkpoint 提交前都不允许进入其文件副作用;`provider_pending` 重放只消费冻结调用,
|
||||
`planned` 重放只消费冻结 resolved 上下文、目标和操作,不重新访问在线识别、目录选择或重命名配置。
|
||||
|
||||
**Ownership and compatibility**
|
||||
|
||||
- `app/application/transfer.py` 拥有 planning input、plan item、checkpoint 和状态错误合同;JSON 版本、
|
||||
指纹及 resolved 上下文均可跨进程 round-trip。
|
||||
- `app/modules/filemanager/transhandler.py` 是唯一目标规划与文件执行实现;`FileManagerModule.transfer`
|
||||
与 `TransHandler.transfer_media` 已删除,不保留第二套重命名、覆盖或目录递归逻辑。
|
||||
- `app/db/adapters/transfer.py` 通过短 Session/UoW 提交 checkpoint;Oper 只负责带状态和指纹条件的
|
||||
stage,3.0.14 migration 可升级、降级并在中断后重跑。
|
||||
- cleanup intent 随准入输入冻结。宿主路径由 FileManager 在 `TransferIntercept` 放行后、任何文件写入前
|
||||
执行;legacy provider 路径为保持旧 ABI 顺序,在全部冻结引用解析成功后、调用 provider 前执行。
|
||||
strict 查询确认目标不存在才视为幂等成功,查询或删除失败抛错并保留对应 checkpoint 供重试;provider
|
||||
全空后提升的宿主 checkpoint 会记录 cleanup 已完成,禁止二次查询或删除。
|
||||
- 插件公开 Transfer 方法签名、事件类型和 payload 不变;旧 provider 身份和顺序随 checkpoint
|
||||
冻结,提交后由统一 dispatcher 精确解析并严格执行,缺失或异常时明确失败而不静默换路;全部返回空
|
||||
才生成宿主 plan,并以第二次 CAS 提交 `planned` checkpoint 后执行。`ChainBase.transfer` 仅委托启动
|
||||
组合根注入的 canonical durable
|
||||
command,内部 plan/execute 合同不向插件调度;新 DTO 不从包根重复导出,`app/plugins/**` 插件
|
||||
副本不参与改造。
|
||||
|
||||
**Excluded**
|
||||
|
||||
- 本叶不引入 claim、lease、heartbeat、attempt、执行步骤幂等或 `manual_review`;这些由
|
||||
`S1-L1.3` 和 `S1-L1.4` 交付。
|
||||
- 文件操作成功后到历史结算前的未知结果仍未达到 E3,ARCH-102 父项继续保持执行中。
|
||||
|
||||
**Local verification (2026-08-27)**
|
||||
|
||||
- planning、持久化、迁移、兼容、replay 和 worker 聚焦回归:`224 passed, 2 skipped`;跳过项仅为
|
||||
本机未配置隔离 PostgreSQL,SQLite upgrade/downgrade/re-upgrade 已覆盖。
|
||||
- 完整本地套件:`6,578 passed, 8 skipped`;架构回归:`174 passed`;scoped Pylint `10.00/10`;
|
||||
host baseline、Ruff/mypy ratchet 与 `git diff --check` 通过。
|
||||
- failure injection 覆盖 commit 前零文件副作用、commit 后崩溃重放、离线 resolved context 恢复、
|
||||
配置漂移仍使用冻结 target storage、规划失败留痕、旧 provider 提交后短路、严格异常、空结果两阶段
|
||||
fallback、缺失引用零 cleanup,以及 cleanup 顺序/幂等/瞬时失败。
|
||||
|
||||
@@ -467,10 +467,25 @@ Durable post-commit side effects have a separate boundary:
|
||||
|
||||
Transfer durable admission follows the same ownership direction without using
|
||||
the Outbox as an execution queue: `app/application/transfer.py` owns the typed
|
||||
admission contract and persist-before-enqueue orchestration, while
|
||||
`app/db/adapters/transfer.py` commits it in a short Session/UoW. Canonical host
|
||||
chains never obtain `TransferPendingOper`; its no-Session API remains only for
|
||||
the exact legacy plugin import contract.
|
||||
admission and versioned planning-checkpoint contracts, while
|
||||
`app/db/adapters/transfer.py` commits admission and the
|
||||
`accepted -> provider_pending -> planned` compare-and-set transitions in short
|
||||
Session/UoW scopes. `app/modules/filemanager/` owns the
|
||||
single pure-plan and checkpoint-execution implementation: all file writes occur
|
||||
after checkpoint commit, and planned recovery consumes frozen resolved context,
|
||||
target storage and ordered operations without online recognition or renaming.
|
||||
Legacy plugin `transfer` providers are frozen by exact identity, order, and ABI
|
||||
arguments in a provider-only checkpoint, then executed by the unified module
|
||||
dispatcher only after commit. The dispatcher resolves every frozen reference
|
||||
before the compatibility cleanup hook and propagates provider failures. Missing
|
||||
or failing providers therefore remain `provider_pending`; only an all-empty
|
||||
result permits host planning and a second CAS to `planned`. Host-only `plan_transfer` and
|
||||
`execute_transfer_plan` contracts never dispatch to plugins. `ChainBase.transfer`
|
||||
is the sole legacy caller facade and delegates the startup-injected durable
|
||||
command; `FileManagerModule.transfer` and `TransHandler.transfer_media` must not
|
||||
be recreated.
|
||||
Canonical host chains never obtain `TransferPendingOper`; its no-Session API
|
||||
remains only for the exact legacy plugin import contract.
|
||||
|
||||
## Composition and Compatibility Boundaries
|
||||
|
||||
@@ -604,8 +619,8 @@ driven workflow registration.
|
||||
| `app/application/subscription/write.py` | Subscription media translation and sync/async write-port orchestration |
|
||||
| `app/application/outbox.py` | Durable intent, topic handler and Outbox repository contracts |
|
||||
| `app/db/adapters/outbox.py` | SQLAlchemy Outbox persistence, claim/lease and retry state adapter |
|
||||
| `app/application/transfer.py` | Transfer task, durable admission contract and persist-before-enqueue use case |
|
||||
| `app/db/adapters/transfer.py` | SQLAlchemy durable admission persistence and detached snapshot adapter |
|
||||
| `app/application/transfer.py` | Transfer task, durable admission, versioned planning input/checkpoint contracts and queue use case |
|
||||
| `app/db/adapters/transfer.py` | SQLAlchemy admission/checkpoint persistence, CAS state transition and detached snapshot adapter |
|
||||
| `app/application/scheduling.py` | Runtime scheduler facade for Agent tools and endpoints; `Scheduler` class registered by `app/startup/initializers/scheduler.py` |
|
||||
| `app/application/commands.py` | Command registry facade for Agent tools and endpoints; `Command` class registered by `app/startup/initializers/command.py` |
|
||||
| `app/application/workflow.py` | Workflow use cases plus the runtime port consumed by API and Chain; `WorkFlowManager` is registered by `app/startup/initializers/workflow.py` |
|
||||
@@ -626,7 +641,7 @@ driven workflow registration.
|
||||
| `app/runtime/event/snapshot.py` | Read-only typed payload snapshots for the plugin SDK; never mutates or replaces the event ABI |
|
||||
| `app/runtime/extensions/module/dispatcher.py` | Plugin-first invocation, short-circuit, list merge, signature relay and sync/async execution |
|
||||
| `app/runtime/extensions/module/contracts.py` | High-frequency method families and frozen legacy fallback contract |
|
||||
| `app/application/chain/context.py` | Injectable Chain dependencies and no-argument compatibility provider |
|
||||
| `app/application/chain/context.py` | Injectable Chain dependencies, no-argument compatibility provider and legacy Transfer command Port |
|
||||
| `app/startup/lifecycle/components.py` | Declarative normal/safe-mode lifecycle manifest, ordering and timeout budgets |
|
||||
| `app/runtime/extensions/module_manager.py` | Module discovery and lifecycle |
|
||||
| `app/runtime/extensions/plugin_manager.py` | Plugin discovery and lifecycle |
|
||||
|
||||
Reference in New Issue
Block a user