mirror of
https://github.com/jxxghp/MoviePilot.git
synced 2026-08-22 00:32:50 +08:00
refactor(db): 修复异步连接池无界增长，并完成 SQLAlchemy 2.0 迁移与分层归位 (#6320)
This commit is contained in:
Aqr-K
@@ -112,18 +112,18 @@ eventmanager.send_event(EventType.TransferComplete, data_dict)
|
||||
|
||||
**When to use:** All database reads and writes. Never issue SQLAlchemy queries directly from chain, module, or endpoint code.
|
||||
|
||||
**Convention:** Each SQLAlchemy model in `app/db/models/` has a corresponding `<Model>Oper` class in `app/db/<model>_oper.py`.
|
||||
**Convention:** Each SQLAlchemy model in `app/db/models/` has a corresponding `<Model>Oper` class in `app/db/oper/<model>.py` — the two packages mirror each other file for file, so the module name carries the entity and the package carries the role.
|
||||
|
||||
```
|
||||
app/db/models/subscribe.py → app/db/subscribe_oper.py (SubscribeOper)
|
||||
app/db/models/systemconfig.py → app/db/systemconfig_oper.py (SystemConfigOper)
|
||||
app/db/models/transferhistory.py → app/db/transferhistory_oper.py (TransferHistoryOper)
|
||||
app/db/models/subscribe.py → app/db/oper/subscribe.py (SubscribeOper)
|
||||
app/db/models/systemconfig.py → app/db/oper/systemconfig.py (SystemConfigOper)
|
||||
app/db/models/transferhistory.py → app/db/oper/transferhistory.py (TransferHistoryOper)
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
|
||||
```python
|
||||
from app.db.subscribe_oper import SubscribeOper
|
||||
from app.db.oper.subscribe import SubscribeOper
|
||||
|
||||
oper = SubscribeOper()
|
||||
subscribe = oper.get(sid=1)
|
||||
@@ -180,11 +180,11 @@ Do not introduce new singletons unless the class genuinely manages global shared
|
||||
|
||||
**Enum:** `SystemConfigKey` in `app/schemas/types.py`
|
||||
|
||||
**Oper class:** `SystemConfigOper` in `app/db/systemconfig_oper.py`
|
||||
**Oper class:** `SystemConfigOper` in `app/db/oper/systemconfig.py`
|
||||
|
||||
```python
|
||||
from app.schemas.types import SystemConfigKey
|
||||
from app.db.systemconfig_oper import SystemConfigOper
|
||||
from app.db.oper.systemconfig import SystemConfigOper
|
||||
|
||||
oper = SystemConfigOper()
|
||||
value = oper.get(SystemConfigKey.RssUrls)
|
||||
@@ -199,7 +199,7 @@ oper.set(SystemConfigKey.RssUrls, ["https://..."])
|
||||
|
||||
**When to use:** Per-user settings that must survive across sessions but differ by user.
|
||||
|
||||
**Oper class:** `UserConfigOper` in `app/db/userconfig_oper.py`
|
||||
**Oper class:** `UserConfigOper` in `app/db/oper/userconfig.py`
|
||||
|
||||
Usage mirrors `SystemConfigOper` but scoped to a `user_id`.
|
||||
|
||||
@@ -212,8 +212,8 @@ Usage mirrors `SystemConfigOper` but scoped to a `user_id`.
|
||||
| `module -> chain` coupling | Move orchestration into `chain` and shared logic into its owning canonical package |
|
||||
| `module -> module` direct calls | Use `chain` to orchestrate cross-module workflows |
|
||||
| Lower-level module importing a chain or manager | Register a callback/resolver from `app/startup/` or move orchestration to `chain` |
|
||||
| Raw SQLAlchemy queries in endpoints or chains | Use the corresponding `*_oper.py` class |
|
||||
| Raw SQLAlchemy queries in endpoints or chains | Use the corresponding Oper class in `app/db/oper/` |
|
||||
| Raw string keys for SystemConfig | Define and use a `SystemConfigKey` enum entry |
|
||||
| HTTP requests via `requests` or `httpx` directly | Host code uses `RequestUtils` from `app/foundation/http.py`; plugins use `app.sdk.network` |
|
||||
| HTTP requests via `requests` or `httpx` directly | Host code uses `RequestUtils` from `app/adapters/network/http.py`; plugins use `app.sdk.network` |
|
||||
|
||||
*Last Updated: 2026-08-14*
|
||||
|
||||
@@ -204,10 +204,27 @@ depend on this established runtime root.
|
||||
|
||||
### DB / Oper layer
|
||||
|
||||
SQLAlchemy models stay under `app/db/models/`; `*_oper.py` classes encapsulate
|
||||
queries. Chains, modules, application services and endpoints use Oper classes
|
||||
instead of issuing SQLAlchemy queries directly. Every schema change requires an
|
||||
Alembic migration under `database/versions/`.
|
||||
SQLAlchemy models stay under `app/db/models/`; the data access classes live in
|
||||
`app/db/oper/` and mirror them one-for-one (`models/subscribe.py` ↔
|
||||
`oper/subscribe.py`), so a filename carries only the entity and the package name
|
||||
carries the role. Chains, modules, application services and endpoints use Oper
|
||||
classes instead of issuing SQLAlchemy queries directly. Every schema change
|
||||
requires an Alembic migration under `database/versions/`.
|
||||
|
||||
Oper classes take and return persistence values, not domain objects. Translating
|
||||
`MediaInfo` / `MetaBase` into a row is business logic and belongs in
|
||||
`app/application/` — see `application/subscribe.py` and `application/history.py`
|
||||
for the two write paths. Column-type coercion (numeric year to string, boolean
|
||||
switches to integers) stays in the Oper because it follows the column, not the
|
||||
caller.
|
||||
|
||||
Invariants that must hold for *every* write are enforced at the mapper rather
|
||||
than at each call site: `app/db/models/_identity.py` normalizes
|
||||
`media_source` / `media_id` on `before_insert` / `before_update`, so a new write
|
||||
path cannot forget them. Identity representation rules themselves
|
||||
(alias folding, trimming, rejecting zero) live in `app/schemas/media.py`
|
||||
alongside the two identity mixins; `app/domain/media.py` keeps only source
|
||||
policy. `app/db` therefore has no dependency on `app/domain`.
|
||||
|
||||
## Composition and Compatibility Boundaries
|
||||
|
||||
|
||||
@@ -115,8 +115,8 @@ except:
|
||||
## What Not To Do
|
||||
|
||||
- Do not introduce new third-party libraries without placing them in the correct dependency entry: runtime packages in `requirements.in`, test/lint/build tooling in `requirements-dev.in`.
|
||||
- Do not use `requests` or `httpx` directly for external HTTP calls - host code uses `RequestUtils` from `app/foundation/http.py`; plugins use `app.sdk.network`.
|
||||
- Do not issue raw SQLAlchemy queries from chains, modules, or endpoints — use the `*_oper.py` classes.
|
||||
- Do not use `requests` or `httpx` directly for external HTTP calls - host code uses `RequestUtils` from `app/adapters/network/http.py`; plugins use `app.sdk.network`.
|
||||
- Do not issue raw SQLAlchemy queries from chains, modules, or endpoints — use the Oper classes in `app/db/oper/`.
|
||||
- Do not add TODO or FIXME without context. Only keep one if it is genuinely deferred and cannot be addressed in the current task.
|
||||
- Do not add noisy markers like `# change starts here`, `# important`, or `# this is a fix`.
|
||||
- Do not write comments that restate what the code already clearly says.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## HTTP Client Conventions
|
||||
|
||||
**Rule:** Host outbound HTTP requests must go through `RequestUtils` from `app/foundation/http.py`. Plugins import it from `app.sdk.network`. Do not use `requests`, `httpx`, or `aiohttp` directly.
|
||||
**Rule:** Host outbound HTTP requests must go through `RequestUtils` from `app/adapters/network/http.py`. Plugins import it from `app.sdk.network`. Do not use `requests`, `httpx`, or `aiohttp` directly.
|
||||
|
||||
`RequestUtils` handles:
|
||||
- Proxy configuration (from `settings.PROXY_*`)
|
||||
|
||||
@@ -48,21 +48,38 @@ alembic revision -m "describe the change"
|
||||
|
||||
**Location:** `app/db/`
|
||||
|
||||
Each model has a corresponding `*_oper.py` file containing the data access class. Do not write SQLAlchemy queries directly in chain, module, or endpoint code.
|
||||
Each model has a corresponding file under `app/db/oper/` containing the data access
|
||||
class, mirroring `app/db/models/` one-for-one. Do not write SQLAlchemy queries
|
||||
directly in chain, module, or endpoint code.
|
||||
|
||||
| Oper Class | File |
|
||||
|---|---|
|
||||
| `SubscribeOper` | `subscribe_oper.py` |
|
||||
| `SystemConfigOper` | `systemconfig_oper.py` |
|
||||
| `TransferHistoryOper` | `transferhistory_oper.py` |
|
||||
| `DownloadHistoryOper` | `downloadhistory_oper.py` |
|
||||
| `MediaServerOper` | `mediaserver_oper.py` |
|
||||
| `UserOper` | `user_oper.py` |
|
||||
| `UserConfigOper` | `userconfig_oper.py` |
|
||||
| `MessageOper` | `message_oper.py` |
|
||||
| `SiteOper` | `site_oper.py` |
|
||||
| `PluginDataOper` | `plugindata_oper.py` |
|
||||
| `WorkflowOper` | `workflow_oper.py` |
|
||||
| `AgentChatOper` | `oper/agentchat.py` |
|
||||
| `AgentTaskOper` | `oper/agenttask.py` |
|
||||
| `DownloadFailureOper` | `oper/downloadfailure.py` |
|
||||
| `DownloadHistoryOper` | `oper/downloadhistory.py` |
|
||||
| `MediaServerOper` | `oper/mediaserver.py` |
|
||||
| `MessageOper` | `oper/message.py` |
|
||||
| `PluginDataOper` | `oper/plugindata.py` |
|
||||
| `SiteOper` | `oper/site.py` |
|
||||
| `SubscribeHistoryOper` | `oper/subscribehistory.py` |
|
||||
| `SubscribeOper` | `oper/subscribe.py` |
|
||||
| `SystemConfigOper` | `oper/systemconfig.py` |
|
||||
| `TransferHistoryOper` | `oper/transferhistory.py` |
|
||||
| `TransferPendingOper` | `oper/transferpending.py` |
|
||||
| `UserConfigOper` | `oper/userconfig.py` |
|
||||
| `UserOper` | `oper/user.py` |
|
||||
| `WorkflowOper` | `oper/workflow.py` |
|
||||
|
||||
Import by module (`from app.db.oper.subscribe import SubscribeOper`) — that is the
|
||||
preferred form in this repository. `app/db/oper/__init__.py` also resolves class
|
||||
names lazily for callers that only want a name, but it deliberately does not
|
||||
eagerly re-export: several tests isolate a single Oper by stubbing it in
|
||||
`sys.modules`, and an eager re-export would pull in the other fifteen and bypass
|
||||
the stub.
|
||||
|
||||
Oper classes accept and return persistence values. Turning a `MediaInfo` or
|
||||
`MetaBase` into a row is business logic and lives in `app/application/`.
|
||||
|
||||
**Standard Oper method conventions:**
|
||||
|
||||
@@ -83,11 +100,11 @@ oper.delete(sid=1) # Delete by key
|
||||
|
||||
**Enum:** `SystemConfigKey` in `app/schemas/types.py`
|
||||
|
||||
**Oper:** `SystemConfigOper` in `app/db/systemconfig_oper.py`
|
||||
**Oper:** `SystemConfigOper` in `app/db/oper/systemconfig.py`
|
||||
|
||||
```python
|
||||
from app.schemas.types import SystemConfigKey
|
||||
from app.db.systemconfig_oper import SystemConfigOper
|
||||
from app.db.oper.systemconfig import SystemConfigOper
|
||||
|
||||
oper = SystemConfigOper()
|
||||
|
||||
@@ -107,7 +124,7 @@ oper.set(SystemConfigKey.RssUrls, ["https://example.com/rss"])
|
||||
**Purpose:** Settings that differ per user account. Uses `UserConfigOper`.
|
||||
|
||||
```python
|
||||
from app.db.userconfig_oper import UserConfigOper
|
||||
from app.db.oper.userconfig import UserConfigOper
|
||||
|
||||
oper = UserConfigOper()
|
||||
value = oper.get(user_id=1, key="notification_enabled")
|
||||
|
||||
@@ -105,7 +105,7 @@ The `API_TOKEN` value in `settings` is the source of truth. It is set at initial
|
||||
|
||||
## SQL Injection Prevention
|
||||
|
||||
- All database access goes through SQLAlchemy ORM via the `*_oper.py` classes. No raw SQL string construction.
|
||||
- All database access goes through SQLAlchemy ORM via the Oper classes in `app/db/oper/`. No raw SQL string construction.
|
||||
- If a raw SQL query is ever genuinely necessary, use SQLAlchemy's `text()` with parameterized binds — never string interpolation.
|
||||
|
||||
---
|
||||
|
||||
@@ -21,7 +21,7 @@ python tests/run.py # 等价于 pytest 全量（参数透
|
||||
|
||||
收集任何测试模块、`import app.*` **之前**，conftest 完成两件事：
|
||||
|
||||
1. **临时库**：把 `CONFIG_DIR` 指向临时目录并 `init_db()` 建表。`app.db` 在导入期即按 `CONFIG_PATH` 连接 `user.db`，所以必须早于首个 `import app.*`；空库会让运行期查表报 `no such table`，故必须建表。
|
||||
1. **临时库**：把 `CONFIG_DIR` 指向临时目录并 `init_db()` 建表。引擎本身已惰性创建（`import app.db` 不再连库），但 `settings` 在 `import app.runtime.config` 那一刻就把 `CONFIG_DIR` 读进字段并建好配置子目录，之后再改环境变量对 `settings.CONFIG_PATH` 毫无影响——引擎晚点才建，连的仍是真实 `user.db`。所以隔离必须早于首个牵入 `app.runtime.config` 的 import（`app.db` / `app.chain.*` 都会牵入）；空库会让运行期查表报 `no such table`，故必须建表。
|
||||
2. **`app.application.site.sites` 垫片**：该模块由独立仓库动态拉取、CI 无此文件，conftest 统一补最小垫片（本地存在真实模块时优先用真实模块）。兼容层会把旧插件的 `app.helper.sites` 导入路由到同一模块。
|
||||
|
||||
由此推出两条**硬规范**：
|
||||
|
||||
Reference in New Issue
Block a user