perf: 并行化后端单测并拆分覆盖率任务 (#6369)

.github/workflows/test.yml

@@ -21,8 +21,16 @@ concurrency:
 jobs:
   pytest:
     runs-on: ubuntu-latest
-    name: Unit Tests
+    name: Unit Tests (${{ matrix.shard }})
     timeout-minutes: 20
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - shard: '1/4'
+          - shard: '2/4'
+          - shard: '3/4'
+          - shard: '4/4'
 
     steps:
       - name: Checkout code
@@ -42,18 +50,42 @@ jobs:
         run: uv sync --locked
 
       - name: Run tests
+        timeout-minutes: 10
+        run: uv run --locked --no-sync python tests/run.py --shard "${{ matrix.shard }}"
+
+  coverage:
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    name: Coverage Report
+    timeout-minutes: 20
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v5
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+        with:
+          version: '0.12.5'
+          python-version: '3.12'
+          enable-cache: true
+          cache-dependency-glob: |
+            pyproject.toml
+            uv.lock
+
+      - name: Install dependencies
+        run: uv sync --locked
+
+      - name: Generate coverage reports
         timeout-minutes: 10
         run: |
-          # tests/run.py 以 pytest 跑 tests 全量；tests/conftest.py 在收集前把 CONFIG_DIR
-          # 指向临时库并建表；CI 额外生成覆盖率报告，便于后续补测和回归分析。
           uv run --locked --no-sync python -m coverage erase
-          uv run --locked --no-sync python -m coverage run tests/run.py
+          uv run --locked --no-sync python -m coverage run tests/run.py --serial
           uv run --locked --no-sync python -m coverage report
           uv run --locked --no-sync python -m coverage json
           uv run --locked --no-sync python -m coverage xml
 
       - name: Upload coverage report
-        if: always()
         uses: actions/upload-artifact@v6
         with:
           name: coverage-report

docs/development-setup.md

@@ -176,6 +176,11 @@ Safety 直接识别项目清单和锁文件，不需要生成或维护 requireme
    uv run --locked --no-sync pytest
    ```
 
+   `python tests/run.py` 在本地默认把排序后的测试文件按向上取整的连续区间切成 4 片，
+   并启动 4 个独立 pytest 进程；GitHub Actions 使用同一入口的 `--shard N/TOTAL`
+   参数启动对应分片。需要单进程调试时使用 `python tests/run.py --serial`。覆盖率报告
+   按需通过 `Unit Tests` workflow 的手动触发串行生成，不阻塞常规 PR / push 门禁。
+
 ### 7. 参考资源
 
 - [uv 官方文档](https://docs.astral.sh/uv/)

docs/testing.md

@@ -7,12 +7,18 @@
 pytest 是唯一运行入口。`tests/conftest.py` 在收集前完成隔离引导，因此任何方式启动 pytest 都会自动隔离。
 
 ```bash
-uv run --locked --no-sync pytest tests                              # 全量
+uv run --locked --no-sync pytest tests                              # 串行全量
 uv run --locked --no-sync pytest tests/test_xxx.py                  # 单文件
 uv run --locked --no-sync pytest tests/test_xxx.py::SomeTest::test_y   # 单用例
-uv run --locked --no-sync python tests/run.py                       # 等价于 pytest 全量（参数透传）
+uv run --locked --no-sync python tests/run.py                       # 默认按文件连续切成 4 片并行跑全量
+uv run --locked --no-sync python tests/run.py --serial              # 串行全量，便于调试或生成覆盖率
+uv run --locked --no-sync python tests/run.py --shard 1/4           # 只跑指定分片，供 CI 复用
 ```
 
+`tests/run.py` 的 runner 参数只有 `--serial` 和 `--shard N/TOTAL`；其余参数保持原顺序
+透传给 pytest，例如 `python tests/run.py -q --maxfail=1`。文件先按字典序排序，再以
+`ceil(文件数 / 分片数)` 的大小连续切片，确保本地与 CI 执行相同的文件集合和顺序。
+
 - 不再使用 `python -m unittest discover`：它不导入 `tests` 包、收不到纯函数用例，且绕过 `conftest.py` 的隔离。
 - 不再依赖 `python tests/test_xxx.py` 直跑：所有 `if __name__ == "__main__": unittest.main()` 尾巴已移除。
 - **复现 CI 用干净环境**：使用 `uv sync --locked` 从 `uv.lock` 创建环境，再以
@@ -139,6 +145,7 @@ def test_recognize_prefers_explicit_identity(sample_meta, monkeypatch):
 
 ## CI 与 PR
 
-- **门禁**：`.github/workflows/test.yml` 在指向 `v3` 的 `pull_request` / `push` 及手动触发时，从 `uv.lock` 同步环境并用 `tests/run.py` 跑全量单测。
-- **PR**：产品代码、测试基础设施、依赖或运行行为发生变化时，运行 `uv run --locked --no-sync python tests/run.py`，确认本次改动涉及的路径通过且 socket 探针零真实出站。若存在无关失败，必须在当前 `upstream/v3` 基线上独立复现并在 PR 中如实说明；不得静默扩大当前 PR 去修复基线问题。纯文档变更按实际内容执行文本、结构和 diff 检查，CI 仍会运行全量门禁。
+- **门禁**：`.github/workflows/test.yml` 在指向 `v3` 的 `pull_request` / `push` 及手动触发时，从 `uv.lock` 同步环境，通过 `tests/run.py --shard N/TOTAL` 把全量测试文件稳定分到 4 个独立 pytest job。每个分片都有独立进程和临时 `CONFIG_DIR`，不共用 SQLite 或进程级状态。
+- **PR**：产品代码、测试基础设施、依赖或运行行为发生变化时，运行 `uv run --locked --no-sync python tests/run.py`，默认以 4 个独立 pytest 进程完成全量；需要断点、输出顺序或测试污染诊断时使用 `--serial`。确认本次改动涉及的路径通过且 socket 探针零真实出站。若存在无关失败，必须在当前 `upstream/v3` 基线上独立复现并在 PR 中如实说明；不得静默扩大当前 PR 去修复基线问题。纯文档变更按实际内容执行文本、结构和 diff 检查，CI 仍会运行全量门禁。
+- 覆盖率不参与常规 PR / push 的合并门禁；需要覆盖率制品时手动触发 `Unit Tests` workflow，独立的 `Coverage Report` job 会通过 `tests/run.py --serial` 跑串行全量并上传 JSON / XML 报告。
 - 复现 CI 使用 `uv sync --locked`；主程序运行依赖位于 `[project].dependencies`，pytest 与覆盖率工具位于默认 `dev` 依赖组。

tests/run.py

@@ -1,14 +1,158 @@
-"""全量单测入口：以 pytest 跑 tests 目录全部用例，命令行参数透传给 pytest。
+"""后端单测入口：默认并行执行文件分片，也支持单分片和显式串行调试。"""
+from __future__ import annotations
 
-用 __file__ 推导 tests 目录绝对路径，使脚本不依赖当前工作目录，从任意位置调用均可。
-"""
+import argparse
+import subprocess
 import sys
 from pathlib import Path
+from typing import Sequence
 
 import pytest
 
-# 本文件即位于 tests/ 下，其所在目录即测试根目录
-_TESTS_DIR = Path(__file__).resolve().parent
+TESTS_DIR = Path(__file__).resolve().parent
+RUNNER_PATH = Path(__file__).resolve()
+DEFAULT_SHARD_COUNT = 4
+
+
+def collect_test_files() -> list[Path]:
+    """按稳定路径顺序返回根测试目录中的全部测试文件。"""
+    return sorted(TESTS_DIR.glob("test_*.py"))
+
+
+def split_test_files(
+    test_files: Sequence[Path], shard_count: int
+) -> list[list[Path]]:
+    """把排序后的文件连续均分，保持 CI 分片归属稳定且易于复现。"""
+    if shard_count <= 0:
+        raise ValueError("shard_count 必须大于 0")
+    shard_size = (len(test_files) + shard_count - 1) // shard_count
+    if shard_size == 0:
+        return [[] for _ in range(shard_count)]
+    shards = [
+        list(test_files[start:start + shard_size])
+        for start in range(0, len(test_files), shard_size)
+    ]
+    return shards + [[] for _ in range(shard_count - len(shards))]
+
+
+def parse_shard(value: str) -> tuple[int, int]:
+    """解析一基的 ``N/TOTAL`` 分片标识，供本地与 CI 共享稳定参数。"""
+    try:
+        index_text, count_text = value.split("/", maxsplit=1)
+        index = int(index_text)
+        count = int(count_text)
+    except (TypeError, ValueError) as error:
+        raise argparse.ArgumentTypeError("--shard 必须使用 N/TOTAL 格式") from error
+    if count <= 0 or not 1 <= index <= count:
+        raise argparse.ArgumentTypeError("--shard 必须满足 1 <= N <= TOTAL")
+    return index, count
+
+
+def parse_args(argv: Sequence[str]) -> tuple[argparse.Namespace, list[str]]:
+    """解析 runner 参数，其余参数原样传给 pytest。"""
+    parser = argparse.ArgumentParser(
+        description="默认以 4 个独立 pytest 文件分片并行运行后端全量测试。"
+    )
+    parser.add_argument(
+        "--serial",
+        action="store_true",
+        help="在当前进程串行运行 tests 目录，适合断点和顺序污染调试。",
+    )
+    parser.add_argument(
+        "--shard",
+        type=parse_shard,
+        metavar="N/TOTAL",
+        help="只运行指定文件分片；CI 使用同一参数启动独立 job。",
+    )
+    args, pytest_args = parser.parse_known_args(argv)
+    if args.serial and args.shard is not None:
+        parser.error("--serial 不能与 --shard 同时使用")
+    return args, pytest_args
+
+
+def run_pytest(paths: Sequence[Path], pytest_args: Sequence[str]) -> int:
+    """在当前进程运行完整目录或一个文件分片。"""
+    return pytest.main([*(str(path) for path in paths), *pytest_args])
+
+
+def _worker_command(
+    shard_index: int, shard_count: int, pytest_args: Sequence[str]
+) -> list[str]:
+    """构造与 CI 完全相同的单分片 worker 命令。"""
+    return [
+        sys.executable,
+        str(RUNNER_PATH),
+        "--shard",
+        f"{shard_index}/{shard_count}",
+        *pytest_args,
+    ]
+
+
+def run_parallel_shards(
+    shards: Sequence[Sequence[Path]], pytest_args: Sequence[str]
+) -> int:
+    """启动独立 pytest 进程并等待全部文件分片结束。"""
+    shard_count = len(shards)
+    processes: list[tuple[int, subprocess.Popen]] = []
+    for shard_index, shard in enumerate(shards, start=1):
+        if not shard:
+            continue
+        print(
+            f"启动测试分片 {shard_index}/{shard_count}：{len(shard)} 个文件",
+            flush=True,
+        )
+        processes.append((
+            shard_index,
+            subprocess.Popen(_worker_command(shard_index, shard_count, pytest_args)),
+        ))
+
+    exit_code = 0
+    try:
+        for shard_index, process in processes:
+            return_code = process.wait()
+            if return_code != 0:
+                print(
+                    f"测试分片 {shard_index}/{shard_count} 失败，退出码 {return_code}",
+                    file=sys.stderr,
+                    flush=True,
+                )
+                exit_code = exit_code or return_code
+    except KeyboardInterrupt:
+        for _, process in processes:
+            if process.poll() is None:
+                process.terminate()
+        for _, process in processes:
+            process.wait()
+        return 130
+    return exit_code
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    """执行串行全量、指定单分片或默认四分片并行全量。"""
+    args, pytest_args = parse_args(sys.argv[1:] if argv is None else argv)
+    if args.serial:
+        return run_pytest([TESTS_DIR], pytest_args)
+
+    test_files = collect_test_files()
+    if not test_files:
+        print(f"未在 {TESTS_DIR} 找到 test_*.py", file=sys.stderr)
+        return 2
+
+    if args.shard is not None:
+        shard_index, shard_count = args.shard
+        selected = split_test_files(test_files, shard_count)[shard_index - 1]
+        if not selected:
+            print(f"测试分片 {shard_index}/{shard_count} 为空", file=sys.stderr)
+            return 2
+        print(
+            f"运行测试分片 {shard_index}/{shard_count}：{len(selected)} 个文件",
+            flush=True,
+        )
+        return run_pytest(selected, pytest_args)
+
+    shards = split_test_files(test_files, DEFAULT_SHARD_COUNT)
+    return run_parallel_shards(shards, pytest_args)
+
 
 if __name__ == "__main__":
-    sys.exit(pytest.main([str(_TESTS_DIR), *sys.argv[1:]]))
+    sys.exit(main())

tests/test_agent_image_support.py

@@ -436,6 +436,8 @@ class AgentImageSupportTest(unittest.TestCase):
 
         with patch.object(settings, "AI_AGENT_ENABLE", True), patch.object(
             settings, "LLM_SUPPORT_IMAGE_INPUT", False
+        ), patch(
+            "app.chain.message.supports_image_input", return_value=False
         ), patch.object(chain, "_get_or_create_session_id", return_value="session-1"), patch.object(
             chain, "_download_attachments_to_data_urls"
         ) as download_images, patch.object(

tests/test_test_runner.py

@@ -0,0 +1,95 @@
+"""后端单测统一 runner 的分片与 CI 调用合同。"""
+
+from pathlib import Path
+
+import pytest
+
+from tests import run as test_runner
+
+
+ROOT = Path(__file__).resolve().parents[1]
+WORKFLOW = ROOT / ".github" / "workflows" / "test.yml"
+
+
+def _test_files(count: int) -> list[Path]:
+    """构造按字典序排列的测试文件路径。"""
+    return [Path(f"test_{index:03d}.py") for index in range(count)]
+
+
+def test_split_test_files_uses_stable_contiguous_chunks() -> None:
+    """文件分片必须稳定覆盖全集，且与既有 CI 的连续均分语义一致。"""
+    test_files = _test_files(10)
+
+    shards = test_runner.split_test_files(test_files, shard_count=4)
+
+    assert [len(shard) for shard in shards] == [3, 3, 3, 1]
+    assert [test_file for shard in shards for test_file in shard] == test_files
+
+
+def test_main_defaults_to_four_parallel_shards(monkeypatch) -> None:
+    """无 runner 参数时应并行执行四个独立 pytest 文件分片。"""
+    test_files = _test_files(10)
+    captured = {}
+    monkeypatch.setattr(test_runner, "collect_test_files", lambda: test_files)
+
+    def fake_run_parallel(shards, pytest_args):
+        captured["shards"] = shards
+        captured["pytest_args"] = pytest_args
+        return 0
+
+    monkeypatch.setattr(test_runner, "run_parallel_shards", fake_run_parallel)
+
+    assert test_runner.main(["-q", "--maxfail=1"]) == 0
+    assert [len(shard) for shard in captured["shards"]] == [3, 3, 3, 1]
+    assert captured["pytest_args"] == ["-q", "--maxfail=1"]
+
+
+def test_main_runs_requested_ci_shard_in_current_process(monkeypatch) -> None:
+    """CI 指定分片时只运行该分片，并继续透传 pytest 参数。"""
+    test_files = _test_files(10)
+    captured = {}
+    monkeypatch.setattr(test_runner, "collect_test_files", lambda: test_files)
+
+    def fake_run_pytest(paths, pytest_args):
+        captured["paths"] = paths
+        captured["pytest_args"] = pytest_args
+        return 0
+
+    monkeypatch.setattr(test_runner, "run_pytest", fake_run_pytest)
+
+    assert test_runner.main(["--shard", "2/4", "-q"]) == 0
+    assert captured["paths"] == test_files[3:6]
+    assert captured["pytest_args"] == ["-q"]
+
+
+def test_main_serial_preserves_legacy_full_suite_entry(monkeypatch) -> None:
+    """串行模式必须保留 tests 根目录加 pytest 参数透传的旧入口。"""
+    captured = {}
+
+    def fake_run_pytest(paths, pytest_args):
+        captured["paths"] = paths
+        captured["pytest_args"] = pytest_args
+        return 0
+
+    monkeypatch.setattr(test_runner, "run_pytest", fake_run_pytest)
+
+    assert test_runner.main(["--serial", "-q", "--maxfail=1"]) == 0
+    assert captured["paths"] == [test_runner.TESTS_DIR]
+    assert captured["pytest_args"] == ["-q", "--maxfail=1"]
+
+
+@pytest.mark.parametrize("value", ["0/4", "5/4", "1/0", "invalid"])
+def test_invalid_shard_values_are_rejected(value: str) -> None:
+    """分片参数必须使用有效的一基 N/TOTAL 范围。"""
+    with pytest.raises(SystemExit, match="2"):
+        test_runner.parse_args(["--shard", value])
+
+
+def test_workflow_uses_the_shared_runner_contract() -> None:
+    """CI 不得另行维护 shell 分片算法，覆盖率必须显式使用串行模式。"""
+    workflow = WORKFLOW.read_text(encoding="utf-8")
+
+    assert 'python tests/run.py --shard "${{ matrix.shard }}"' in workflow
+    assert "python -m coverage run tests/run.py --serial" in workflow
+    assert "mapfile" not in workflow
+    assert "SHARD_INDEX" not in workflow