Files
BiliNote/backend/tests/test_bilibili_dm_patch.py
T
pumpkinperson996 3e579b1434 fix(youtube): 修复 YouTube 笔记生成失败 "Requested format is not available"
## 问题

输入 YouTube 链接生成笔记时任务必定失败:

    ERROR: [youtube] <id>: Requested format is not available

即使字幕已经抓取成功(日志里能看到"成功获取 YouTube 字幕,共 N 段"),
任务仍然在下载阶段崩掉。

## 原因

两个独立的问题叠加:

1. `requirements.txt` 把 yt-dlp 钉在 `2025.3.31`。YouTube 之后轮换过
   player,旧版 yt-dlp 解不出 nsig 签名,所有音视频格式都被丢弃,只剩下
   storyboard 图片(日志:`Only images are available for download`),
   格式选择随即抛错。

2. 有字幕时 `NoteGenerator` 走的是"只取元信息"的路径
   (`download(skip_download=True)`,只要 title/duration/cover),但
   `YoutubeDownloader.download` 无条件设置了
   `format='bestaudio[ext=m4a]/bestaudio/best'`。于是一个根本不需要媒体流的
   调用,也会因为选不出格式而失败——把一个字幕已经到手的任务整个带崩。

## 改动

- `youtube_downloader.py`:`skip_download` 时设置
  `ignore_no_formats_error=True`。yt-dlp 落后于 YouTube 时,只取元信息的路径
  降级为"没有音频",而不是让整个笔记失败。
- `youtube_downloader.py`:`ext = info.get("ext") or "m4a"`。跳过下载时
  yt-dlp 返回 `ext=None`,`dict.get` 的默认值对显式 None 不生效,
  会拼出 `xxx.None` 这样的路径。
- `requirements.txt`:`yt-dlp==2025.3.31` → `>=2026.7.4`。yt-dlp 是对抗
  YouTube 变化的滚动依赖,精确钉版本本身就是这个 bug 的成因;用 `>=` 与同文件
  的 `youtube-transcript-api>=1.0.0` 保持一致。
- `bilibili_dm_patch.py`:wrapper 改为透传 `**kwargs`。升级 yt-dlp 后
  `_real_extract` 会以 `fatal=False` 调用 `_download_playinfo`,而 wrapper
  钉死了签名,导致 **所有 B 站下载** 抛
  `TypeError: ... got an unexpected keyword argument 'fatal'`。
- 测试:新增 `test_youtube_metadata_only.py` 覆盖上面两条 YouTube 保证;
  `test_bilibili_dm_patch.py` 新增未知 kwargs 透传用例,并让 fake 响应带上
  `code` 字段(yt-dlp 2026.x 会先校验信封再返回 data)。

## 验证

- 真实跑通:YouTube(有字幕,走元信息路径)与 B 站(无字幕,走完整下载 +
  转写)均能生成笔记。
- `pytest tests/` → 46 passed。唯一失败的
  `test_task_serial_executor` 在升级前后表现一致,与本次改动无关,未作改动。
2026-07-25 07:37:10 -05:00

123 lines
4.7 KiB
Python

"""
TDD coverage for the Bilibili wbi/playurl dm_img risk-control patch.
Background: around 2026-06, Bilibili's `x/player/wbi/playurl` gateway began
rejecting requests that omit the browser fingerprint params
(dm_img_list / dm_img_str / dm_cover_img_str / dm_img_inter + web_location)
with HTTP 412. yt-dlp (incl. latest) does not yet send these for playurl, so
videos whose web page does not inline playinfo (forcing the API call) fail.
These tests verify our yt-dlp monkey-patch injects those params *before* wbi
signing, and that caller-supplied query params still win.
"""
import importlib.util
import pathlib
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
MODULE_PATH = ROOT / "app" / "downloaders" / "bilibili_dm_patch.py"
spec = importlib.util.spec_from_file_location("bilibili_dm_patch", MODULE_PATH)
if spec is None or spec.loader is None:
raise ImportError("bilibili_dm_patch module spec not found")
bilibili_dm_patch = importlib.util.module_from_spec(spec)
spec.loader.exec_module(bilibili_dm_patch)
REQUIRED_KEYS = {
"web_location",
"dm_img_list",
"dm_img_str",
"dm_cover_img_str",
"dm_img_inter",
}
class BuildDmImgParamsTest(unittest.TestCase):
def test_contains_all_required_risk_control_keys(self):
params = bilibili_dm_patch.build_dm_img_params()
self.assertTrue(REQUIRED_KEYS.issubset(params.keys()))
def test_web_location_is_expected_sentinel(self):
self.assertEqual(bilibili_dm_patch.build_dm_img_params()["web_location"], 1550101)
class ApplyPatchTest(unittest.TestCase):
def setUp(self):
try:
import yt_dlp.extractor.bilibili # noqa: F401
except Exception as exc: # pragma: no cover - env without yt-dlp
self.skipTest(f"yt-dlp not importable: {exc}")
def test_patch_is_idempotent(self):
from yt_dlp.extractor.bilibili import BilibiliBaseIE
self.assertTrue(bilibili_dm_patch.apply_bilibili_dm_img_patch())
first = BilibiliBaseIE._download_playinfo
self.assertTrue(bilibili_dm_patch.apply_bilibili_dm_img_patch())
self.assertIs(BilibiliBaseIE._download_playinfo, first)
def test_dm_params_reach_wbi_signing_with_caller_query_preserved(self):
from yt_dlp import YoutubeDL
from yt_dlp.extractor.bilibili import BilibiliBaseIE
bilibili_dm_patch.apply_bilibili_dm_img_patch()
captured = {}
def fake_sign_wbi(params, video_id):
# Capture the exact params handed to wbi signing (just before the
# HTTP request). dm_* must already be present here, pre-signature.
captured.update(params)
return params
def fake_download_json(url, video_id, **kwargs):
# Avoid any network; the real playurl call would 412 without dm_*.
# yt-dlp >= 2026.x checks the envelope's `code` before returning
# `data`, so the fake has to look like a real playurl response.
return {"code": 0, "data": {"ok": True}}
ie = BilibiliBaseIE(YoutubeDL({"quiet": True}))
ie._sign_wbi = fake_sign_wbi
ie._download_json = fake_download_json
ie._download_playinfo("BV1X9L16oEgB", 4242, headers={}, query={"qn": 64})
self.assertTrue(
REQUIRED_KEYS.issubset(captured.keys()),
f"missing dm_* keys, got: {sorted(captured)}",
)
self.assertEqual(captured["web_location"], 1550101)
# caller-supplied query must survive the merge
self.assertEqual(captured["qn"], 64)
# the original method still builds its base params
self.assertEqual(captured["bvid"], "BV1X9L16oEgB")
def test_patch_forwards_unknown_kwargs_to_original(self):
"""
yt-dlp's real call site passes kwargs the wrapper never declared —
`_real_extract` calls `_download_playinfo(..., fatal=False)` since
2026.x. A wrapper with a pinned signature raises TypeError there and
breaks every Bilibili download, so unknown kwargs must pass through.
"""
from yt_dlp import YoutubeDL
from yt_dlp.extractor.bilibili import BilibiliBaseIE
bilibili_dm_patch.apply_bilibili_dm_img_patch()
seen = {}
def fake_download_json(url, video_id, **kwargs):
return {"code": 0, "data": {"ok": True}}
ie = BilibiliBaseIE(YoutubeDL({"quiet": True}))
ie._sign_wbi = lambda params, video_id: seen.update(params) or params
ie._download_json = fake_download_json
# Must not raise TypeError on a kwarg the wrapper does not name.
ie._download_playinfo("BV1X9L16oEgB", 4242, headers={}, query={}, fatal=False)
self.assertTrue(REQUIRED_KEYS.issubset(seen.keys()))
if __name__ == "__main__":
unittest.main()