Compare commits

...

80 Commits

Author SHA1 Message Date
amtoaer
802565e4f6 chore: 发布 bili-sync 2.3.0 2025-01-25 00:34:47 +08:00
amtoaer
4984026017 docs: 更新文档,跟进最新代码变化 2025-01-25 00:29:12 +08:00
amtoaer
2a98359085 chore: 隐藏 target 并调整表述,缩减日志长度 2025-01-25 00:11:22 +08:00
amtoaer
979294bb94 fix: 修复 video path 未正确设置问题 2025-01-24 14:05:16 +08:00
ᴀᴍᴛᴏᴀᴇʀ
40cf22a7fa refactor: 引入 enum_dispatch 静态分发,提升性能 (#232) 2025-01-24 13:44:27 +08:00
ᴀᴍᴛᴏᴀᴇʀ
9e5a8b0573 feat: 确保 video stream 在出现错误时返回 Err (#231) 2025-01-24 13:17:12 +08:00
ᴀᴍᴛᴏᴀᴇʀ
7c220f0d2b refactor: 精简代码,统一逻辑 (#229) 2025-01-24 01:11:59 +08:00
amtoaer
aa88f97eff refactor: 尝试将任务处理部分重构为 stream 写法,增补注释 2025-01-23 17:13:51 +08:00
ᴀᴍᴛᴏᴀᴇʀ
b4177d4ffc feat: 引入更健壮的新视频检测方法 (#228)
* feat: 为各个 video list 表添加 latest_row_at 字段

* chore: 为 model 引入新增的字段

* feat: 实现新版中断条件(待测试)

* test: 更新测试
2025-01-22 23:53:18 +08:00
amtoaer
b888db6a61 refactor: 数据块已经在内存中,直接使用 write_all 2025-01-22 01:52:32 +08:00
amtoaer
6ae87364b4 feat: 为下载加入 flush 与 content-length 检查 2025-01-22 00:18:04 +08:00
amtoaer
18c966a0f9 refactor: 避免一些不必要的 to_string 2025-01-21 22:59:16 +08:00
amtoaer
ab84a8dad1 refactor: 签名时按需使用 String 2025-01-21 22:54:20 +08:00
amtoaer
1a32e38dc3 refactor: 使用 context 代替 ok_or 和 ok_or_else 2025-01-21 18:06:54 +08:00
amtoaer
0f25923c52 refactor: 继续调整优化部分代码,移除主体代码的所有 unwrap 2025-01-21 17:17:14 +08:00
amtoaer
cdc30e1b32 refactor: 优化部分代码,移除一批 unwrap 2025-01-21 03:12:45 +08:00
NKDark
c10c14c125 chore: 修改配置文件写入逻辑 (#222) 2025-01-21 01:39:48 +08:00
amtoaer
60604aeb33 docs: 更新文档描述,简化视频合集/视频列表的配置 2025-01-17 17:53:32 +08:00
amtoaer
276fb5b3e4 chore: 发布 bili-sync 2.2.0 2025-01-14 19:12:47 +08:00
ᴀᴍᴛᴏᴀᴇʀ
e05f58b8a1 docs: 文档跟进最新代码变化 (#217) 2025-01-14 18:16:15 +08:00
amtoaer
8dfc96e1dc chore: 补充一条提示信息 2025-01-14 05:18:04 +08:00
amtoaer
cdc639cf75 fix: 修复代码语义错误,精简一些不必要的代码 2025-01-14 02:21:15 +08:00
amtoaer
847c3115cd chore: 遇到编码不符合的情况不再打印日志 2025-01-14 01:19:03 +08:00
amtoaer
7dc049ffe5 chore: 默认设置请求频率限制,用户可手动调整 2025-01-14 00:08:38 +08:00
amtoaer
265fe630dd fix: 修复 UP 主信息接口的类型问题 2025-01-14 00:07:51 +08:00
ᴀᴍᴛᴏᴀᴇʀ
f31900e6c7 deps: 更新项目依赖 (#214) 2025-01-13 19:39:08 +08:00
ᴀᴍᴛᴏᴀᴇʀ
54b46c150e refactor: 一些边边角角的小重构 (#213) 2025-01-13 18:57:08 +08:00
ᴀᴍᴛᴏᴀᴇʀ
7d9999d6aa feat: 调整并重构视频音频流的选择逻辑,应该可以提升些许性能 (#212)
* feat: 调整并重构视频音频流的选择逻辑,应该可以提升些许性能

* test: 添加少量单元测试
2025-01-13 13:51:16 +08:00
amtoaer
05aa30119e ci: 使用最新 nightly 执行 check 2025-01-12 03:13:59 +08:00
amtoaer
368b9ef735 style: 清空 clippy 提示 2025-01-11 23:36:59 +08:00
ᴀᴍᴛᴏᴀᴇʀ
0113bf704d chore: 支持使用 leaky-bucket 限制请求频率 (#211)
* chore: 移除之前引入的 delay

* feat: 支持为 b 站请求配置频率限制
2025-01-11 23:24:01 +08:00
ᴀᴍᴛᴏᴀᴇʀ
66a7b1394e test: 修复 windows 单元测试错误 (#164) 2024-08-09 00:02:56 +08:00
ᴀᴍᴛᴏᴀᴇʀ
ae05cad22f feat: 允许在 video_name 和 page_name 中使用对应平台的路径分隔符 (#163) 2024-08-08 23:53:22 +08:00
amtoaer
be3abab13f chore: 移除多余的 info 2024-08-08 22:01:52 +08:00
ᴀᴍᴛᴏᴀᴇʀ
c432a282a7 fix: 修复视频 page 过多时数据库插入失败的问题 (#162) 2024-08-03 23:49:00 +08:00
ᴀᴍᴛᴏᴀᴇʀ
e9e20ace93 build: 升级依赖 (#160) 2024-07-28 15:38:42 +08:00
ᴀᴍᴛᴏᴀᴇʀ
6187827e1b fix: 确保无论视频下载结果如何,都在最终删除临时文件 (#159) 2024-07-28 15:34:00 +08:00
ᴀᴍᴛᴏᴀᴇʀ
8a4a95e343 feat: 支持设置 video 和 page 的下载并发 (#157) 2024-07-28 02:32:02 +08:00
ᴀᴍᴛᴏᴀᴇʀ
401fcdc630 refactor: 将 filenamify 移动至本地,将正则表达式设置为 static (#156) 2024-07-28 01:51:37 +08:00
ᴀᴍᴛᴏᴀᴇʀ
b2d22253c5 feat: 支持 up 主投稿视频下载 (#155) 2024-07-27 22:35:20 +08:00
ᴀᴍᴛᴏᴀᴇʀ
29bfc2efce refactor: 重构部分代码,调整函数位置 (#154) 2024-07-25 00:05:29 +08:00
ᴀᴍᴛᴏᴀᴇʀ
75de39dfbb feat: 支持设置时间格式化字符串,支持在 video_name 和 page_name 中使用 time (#152) 2024-07-24 21:06:40 +08:00
ᴀᴍᴛᴏᴀᴇʀ
8f37fdf841 refactor: 把循环拆分到外层,提取公共代码 (#151) 2024-07-24 00:36:19 +08:00
ᴀᴍᴛᴏᴀᴇʀ
20e3ac2129 build: 升级 time 依赖 (#150) 2024-07-23 22:38:52 +08:00
ᴀᴍᴛᴏᴀᴇʀ
3a8f33d273 feat: 支持各种任务结束之后的 delay 配置 (#148) 2024-07-23 22:29:25 +08:00
ᴀᴍᴛᴏᴀᴇʀ
d46881aea6 docs: 支持点击放大文档中的图片 (#149) 2024-07-23 04:13:05 -07:00
ᴀᴍᴛᴏᴀᴇʀ
e25339c53c docs: 将图片转为 webp 并压缩,大幅缩小占用空间 (#147) 2024-07-22 22:12:42 +08:00
ᴀᴍᴛᴏᴀᴇʀ
5102999676 docs: 修复配置文件位置的描述错误 (#145) 2024-07-22 12:53:41 +08:00
amtoaer
991ce3ea3c chore: 发布 bili-sync 2.1.2 2024-07-21 23:40:30 +08:00
amtoaer
e4fb096d0c build: 更新项目依赖 2024-07-21 22:51:56 +08:00
ᴀᴍᴛᴏᴀᴇʀ
28070aa7d8 docs: 添加"工作原理"小节 (#135) 2024-07-21 21:34:52 +08:00
ᴀᴍᴛᴏᴀᴇʀ
33e758bd91 refactor: 移除不必要的标记和代码块,统一 use 格式 (#144) 2024-07-21 19:16:52 +08:00
ᴀᴍᴛᴏᴀᴇʀ
86e858082d feat: 为下载视频接口加入 wbi 签名 (#143) 2024-07-21 18:47:09 +08:00
ᴀᴍᴛᴏᴀᴇʀ
2ffe432f37 feat: 为合集接口实现 wbi 签名 (#140) 2024-07-21 16:49:53 +08:00
A1ca7raz
6ef9ecaee0 chore: 更正许可证文件名错误 (#141) 2024-07-19 20:33:36 +08:00
amtoaer
9ef88e1b2b docs: 更新部分表述,更新当前的功能列表 2024-07-11 19:27:02 +08:00
amtoaer
6e7c6061b2 chore: 发布 bili-sync 2.1.1 2024-07-11 18:09:31 +08:00
amtoaer
40b3f77748 docs: 添加 2.1.1 中稍后再看的文档 2024-07-11 18:08:13 +08:00
ᴀᴍᴛᴏᴀᴇʀ
c27d1a2381 feat: 支持稍后再看的扫描与下载 (#131)
* 暂存

* 写点

* feat: 支持稍后再看

* chore: 干掉 print
2024-07-10 22:46:01 -07:00
ᴀᴍᴛᴏᴀᴇʀ
4c5d1b6ea1 fix: 修复 exist_labels 可能判断错误的问题 (#132) 2024-07-09 22:47:07 +08:00
amtoaer
0b6fd72682 chore: 发布 bili-sync 2.1.0 2024-07-05 22:44:56 +08:00
amtoaer
e65cd36b2e chore: 固定 CNAME 文件,修改 gh-pages 分支的提交信息 2024-07-05 22:35:34 +08:00
ᴀᴍᴛᴏᴀᴇʀ
352282f277 docs: 全局修改描述,在文档中加入版本信息并在发版时自动替换 (#128)
* 暂存

* chore: 修改一些杂项
2024-07-05 22:31:26 +08:00
amtoaer
fa2bc7b5e8 docs: 采用自定义域名方式,移除 base 目录 2024-07-05 18:26:39 +08:00
amtoaer
bb90f0c6f2 docs: 修改文档的 base 目录 2024-07-05 16:58:18 +08:00
amtoaer
90f2a1d4ed docs: 添加在线文档的 workflow 构建流,修复一些问题 2024-07-05 16:45:42 +08:00
amtoaer
e2b65746dd docs: 添加独立的文档页面,移除 README 中的相关描述 (#127) 2024-07-05 02:17:42 +08:00
amtoaer
24d0da0bf3 chore: 修改 as 大小写,避免 warning 2024-07-04 01:49:04 +08:00
ᴀᴍᴛᴏᴀᴇʀ
ff1150e863 fix: 修复重构引入的若干 bug (#126) 2024-07-04 01:00:41 +08:00
amtoaer
940abd4f3b build: 修改现有的版本号,添加 release 相关选项 2024-07-03 22:11:31 +08:00
ᴀᴍᴛᴏᴀᴇʀ
4c9ad2318c feat: 大范围重构,支持视频合集下载 (#97) 2024-07-03 03:57:12 -07:00
ᴀᴍᴛᴏᴀᴇʀ
097f885050 build: 更新依赖 (#125) 2024-06-28 03:07:45 -07:00
ᴀᴍᴛᴏᴀᴇʀ
6ebef0a414 ci: 对处于 draft 状态的 PR 禁用 workflow (#123) 2024-06-28 00:04:30 +08:00
ᴀᴍᴛᴏᴀᴇʀ
4818e62414 refactor: 引入 clap 处理环境变量和命令行参数 (#119) 2024-06-08 10:57:08 +08:00
ᴀᴍᴛᴏᴀᴇʀ
1744f8647b chore: 修改项目路径结构,使用 workspace 组织包 (#118) 2024-06-08 01:56:53 +08:00
ᴀᴍᴛᴏᴀᴇʀ
c4db12b154 fix: 修复类型错误导致的数值溢出 (#115) 2024-06-01 03:21:23 +08:00
ᴀᴍᴛᴏᴀᴇʀ
2ef99a20c9 feat: 支持自定义 NFO 文件中的视频时间,可选加入收藏夹的时间、视频发布的时间 (#114)
* feat: 支持自定义 NFO 文件中的视频时间,可选加入收藏夹的时间、视频发布的时间

* chore: 使用小写
2024-06-01 03:01:39 +08:00
ᴀᴍᴛᴏᴀᴇʀ
67de151234 ci: 使用较旧的 rust nightly 版本,避免语言变更导致的编译失败 (#113) 2024-06-01 01:51:19 +08:00
ᴀᴍᴛᴏᴀᴇʀ
73f97f937f feat: 每次执行前检查登录状态,避免凭据失效导致的非预期行为 (#112)
* feat: 每次执行前检查登录状态,避免凭据失效导致的非预期行为

* refactor: 减少代码长度
2024-06-01 01:46:15 +08:00
ky0utarou
8fee6fb97a Update README.md - compose中指定user,附加简要说明 (#102)
* Update README.md - compose中指定user

* Update README.md - compose中指定user的简要说明
2024-05-08 19:11:32 +08:00
122 changed files with 5855 additions and 7301 deletions

View File

@@ -5,8 +5,7 @@ on:
branches:
- main
pull_request:
branches:
- "**"
types: ['opened', 'reopened', 'synchronize', 'ready_for_review']
concurrency:
# Allow only one workflow per any non-`main` branch.
@@ -22,11 +21,12 @@ jobs:
tests:
name: Run Clippy and tests
runs-on: ubuntu-latest
if: ${{ github.event_name == 'push' || !github.event.pull_request.draft }}
steps:
- name: Checkout repo
uses: actions/checkout@v4
- run: rustup toolchain install nightly && rustup default nightly && rustup component add rustfmt clippy
- run: rustup default nightly && rustup component add rustfmt clippy
- name: Cache dependencies
uses: swatinem/rust-cache@v2

40
.github/workflows/doc.yaml vendored Normal file
View File

@@ -0,0 +1,40 @@
name: Build Docs
on:
push:
paths:
- 'docs/**'
jobs:
doc:
if: ${{ github.ref == 'refs/heads/main' }}
name: Build documentation
runs-on: ubuntu-latest
defaults:
run:
working-directory: docs
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: Setup bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Cache dependencies
uses: actions/cache@v4
with:
path: ~/.bun/install/cache
key: ${{ runner.os }}-bun-${{ hashFiles('docs/bun.lockb') }}
restore-keys: |
${{ runner.os }}-bun-
- name: Build documentation
run: bun run docs:build
- name: Deploy Github Pages
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: docs/.vitepress/dist
force_orphan: true
commit_message: 部署来自 main 的最新文档变更:

4
.gitignore vendored
View File

@@ -1,6 +1,8 @@
**/target
auth_data
*.sqlite
*.json
video
debug*
node_modules
docs/.vitepress/cache
docs/.vitepress/dist

1379
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,58 +1,80 @@
[package]
name = "bili-sync-rs"
version = "2.0.3"
edition = "2021"
[workspace]
members = ["crates/*"]
default-members = ["crates/bili_sync"]
resolver = "2"
[workspace.package]
version = "2.3.0"
authors = ["amtoaer <amtoaer@gmail.com>"]
license = "MIT"
description = "基于 rust tokio 编写的 bilibili 收藏夹同步下载工具"
description = "由 Rust & Tokio 驱动的哔哩哔哩同步工具"
edition = "2021"
publish = false
[dependencies]
anyhow = { version = "1.0.81", features = ["backtrace"] }
arc-swap = { version = "1.7", features = ["serde"] }
async-stream = "0.3.5"
chrono = { version = "0.4.35", features = ["serde"] }
cookie = "0.18.0"
dirs = "5.0.1"
entity = { path = "entity" }
filenamify = "0.1.0"
[workspace.dependencies]
bili_sync_entity = { path = "crates/bili_sync_entity" }
bili_sync_migration = { path = "crates/bili_sync_migration" }
anyhow = { version = "1.0.95", features = ["backtrace"] }
arc-swap = { version = "1.7.1", features = ["serde"] }
assert_matches = "1.5"
async-std = { version = "1.13.0", features = ["attributes", "tokio1"] }
async-stream = "0.3.6"
async-trait = "0.1.85"
chrono = { version = "0.4.39", features = ["serde"] }
clap = { version = "4.5.26", features = ["env"] }
cookie = "0.18.1"
cow-utils = "0.1.3"
dirs = "6.0.0"
enum_dispatch = "0.3.13"
float-ord = "0.3.2"
futures = "0.3.30"
handlebars = "5.1.2"
futures = "0.3.31"
handlebars = "6.3.0"
hex = "0.4.3"
log = "0.4.21"
memchr = "2.5.0"
migration = { path = "migration" }
once_cell = "1.19.0"
prost = "0.12.4"
quick-xml = { version = "0.31.0", features = ["async-tokio"] }
leaky-bucket = "1.1.2"
md5 = "0.7.0"
memchr = "2.7.4"
once_cell = "1.20.2"
prost = "0.13.4"
quick-xml = { version = "0.37.2", features = ["async-tokio"] }
rand = "0.8.5"
regex = "1.10.3"
reqwest = { version = "0.12.4", features = [
"json",
"stream",
regex = "1.11.1"
reqwest = { version = "0.12.12", features = [
"charset",
"cookies",
"gzip",
"charset",
"http2",
"json",
"rustls-tls",
"stream",
], default-features = false }
rsa = { version = "0.9.6", features = ["sha2"] }
sea-orm = { version = "0.12", features = [
"sqlx-sqlite",
"runtime-tokio-rustls",
rsa = { version = "0.9.7", features = ["sha2"] }
sea-orm = { version = "1.1.4", features = [
"macros",
"runtime-tokio-rustls",
"sqlx-sqlite",
] }
serde = { version = "1.0.197", features = ["derive"] }
serde_json = "1.0"
strum = { version = "0.26", features = ["derive"] }
thiserror = "1.0.58"
tokio = { version = "1", features = ["full"] }
toml = "0.8.12"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["chrono"] }
sea-orm-migration = { version = "1.1.4", features = [] }
serde = { version = "1.0.217", features = ["derive"] }
serde_json = "1.0.135"
serde_urlencoded = "0.7.1"
strum = { version = "0.26.3", features = ["derive"] }
thiserror = "2.0.11"
tokio = { version = "1.43.0", features = ["full"] }
toml = "0.8.19"
tracing = "0.1.41"
tracing-subscriber = { version = "0.3.19", features = ["chrono"] }
[workspace]
members = [".", "entity", "migration"]
[workspace.metadata.release]
release = false
tag-message = ""
tag-prefix = ""
pre-release-commit-message = "chore: 发布 bili-sync {{version}}"
publish = false
pre-release-replacements = [
{ file = "../../docs/.vitepress/config.mts", search = "\"v[0-9\\.]+\"", replace = "\"v{{version}}\"", exactly = 1 },
{ file = "../../docs/introduction.md", search = " v[0-9\\.]+", replace = " v{{version}}", exactly = 1 },
]
[profile.release]
strip = true

View File

@@ -1,4 +1,4 @@
FROM alpine as base
FROM alpine AS base
ARG TARGETPLATFORM

View File

@@ -7,4 +7,15 @@ build:
build-docker: build
cp target/x86_64-unknown-linux-musl/release/bili-sync-rs ./Linux-x86_64-bili-sync-rs
docker build . -t bili-sync-rs-local --build-arg="TARGETPLATFORM=linux/amd64"
just clean
just clean
copy-config:
rm -rf ~/.config/bili-sync
cp -r ~/.config/nas/bili-sync-rs ~/.config/bili-sync
sed -i -e 's/\/Bilibilis/\/Test_Bilibilis/g' -e 's/.config\/nas/.config\/test_nas/g' ~/.config/bili-sync/config.toml
run:
cargo run
debug: copy-config
just run

View File

177
README.md
View File

@@ -3,173 +3,38 @@
## 简介
> [!NOTE]
> 此为 v2.x 版本文档v1.x 版本文档请前往[此处](https://github.com/amtoaer/bili-sync/tree/v1.x)查看
为 NAS 用户编写的 BILIBILI 收藏夹同步工具,可使用 EMBY 等媒体库工具浏览。
支持展示视频封面、名称、加入日期、标签、分页等。
> [点击此处](https://bili-sync.allwens.work/)查看文档
bili-sync 是一款专为 NAS 用户编写的哔哩哔哩同步工具,由 Rust & Tokio 驱动。
## 效果演示
**注:因为可能同时存在单页视频和多页视频,媒体库类型请选择“混合内容”。**
### 概览
![概览](./assets/overview.png)
![概览](./assets/overview.webp)
### 详情
![详情](./assets/detail.png)
![详情](./assets/detail.webp)
### 播放(使用 infuse
![播放](./assets/play.png)
![播放](./assets/play.webp)
### 文件排布
![文件](./assets/dir.png)
![文件](./assets/dir.webp)
## 配置文件说明
> [!NOTE]
> 在 Docker 环境中,`~` 会被展开为 `/app`。
## 功能与路线图
程序默认会将配置文件存储于 `~/.config/bili-sync/config.toml`,数据库文件存储于 `~/.config/bili-sync/data.sqlite`,如果发现不存在会新建并写入默认配置。
- [x] 使用用户填写的凭据认证,并在必要时自动刷新
- [x] 支持收藏夹与视频列表/视频合集的下载
- [x] 自动选择用户设置范围内最优的视频和音频流,并在下载完成后使用 FFmpeg 合并
- [x] 使用 Tokio 与 Reqwest对视频、视频分页进行异步并发下载
- [x] 使用媒体服务器支持的文件命名,方便一键作为媒体库导入
- [x] 当前轮次下载失败会在下一轮下载时重试,失败次数过多自动丢弃
- [x] 使用数据库保存媒体信息,避免对同个视频的多次请求
- [x] 打印日志,并在请求出现风控时自动终止,等待下一轮执行
- [x] 提供多平台的二进制可执行文件,为 Linux 平台提供了立即可用的 Docker 镜像
- [x] 支持对“稍后再看”内视频的自动扫描与下载
- [x] 支持对 UP 主投稿视频的自动扫描与下载
- [x] 支持限制任务的并行度和接口请求频率
- [ ] 下载单个文件时支持断点续传与并发下载
配置文件加载时会进行简单校验,默认配置无法通过校验,程序会报错终止运行。
可以下载程序后直接运行程序,看到报错后参考报错信息对默认配置进行修改,修改正确后即可正常运行。
对于配置文件中的 `credential`,请参考[凭据获取流程](https://nemo2011.github.io/bilibili-api/#/get-credential)。
配置文件中的 `video_name``page_name` 支持使用模板,模板的替换语法请参考示例。模板中的内容在执行时会被动态替换为对应的内容。
video_name 支持设置 bvid视频编号、title视频标题、upper_nameup 主名称、upper_midup 主 id
page_name 除支持 video 的全部参数外,还支持 ptitle分 P 标题、pid分 P 页号)。
对于每个 favorite_list 的下载路径,程序会在其下建立如下的文件夹:
1. 单页视频:
```bash
├── {video_name}
│   ├── {page_name}.mp4
│   ├── {page_name}.nfo
│   └── {page_name}-poster.jpg
```
2. 多页视频:
```bash
├── {video_name}
│   ├── poster.jpg
│   ├── Season 1
│   │   ├── {page_name} - S01E01.mp4
│   │   ├── {page_name} - S01E01.nfo
│   │   ├── {page_name} - S01E01-thumb.jpg
│   │   ├── {page_name} - S01E02.mp4
│   │   ├── {page_name} - S01E02.nfo
│   │   └── {page_name} - S01E02-thumb.jpg
│   └── tvshow.nfo
```
对于 filter_option 的可选值,请前往 [analyzer.rs](https://github.com/amtoaer/bili-sync/blob/main/src/bilibili/analyzer.rs) 查看。
对于 danmaku_option 的项含义,请前往 [danmaku/mod.rs](https://github.com/amtoaer/bili-sync/blob/main/src/bilibili/danmaku/canvas/mod.rs) 与 [原项目的说明](https://github.com/gwy15/danmu2ass?tab=readme-ov-file#%E5%91%BD%E4%BB%A4%E8%A1%8C) 查看。
## 配置文件示例
```toml
# 视频所处文件夹的名称
video_name = "{{title}}"
# 视频分页文件的命名
page_name = "{{bvid}}"
# 扫描运行的间隔(单位:秒)
interval = 1200
# emby 演员信息的保存位置
upper_path = "/home/amtoaer/.config/nas/emby/metadata/people/"
[credential]
# Bilibili 的 Web 端身份凭据,需要凭据才能下载高清视频
sessdata = ""
bili_jct = ""
buvid3 = ""
dedeuserid = ""
ac_time_value = ""
[filter_option]
# 视频、音频流的筛选选项,程序会使用范围内质量最高的流
# 注意设置范围过小可能导致无满足条件的流,推荐仅调整质量上限和编码优先级
video_max_quality = "Quality8k"
video_min_quality = "Quality360p"
audio_max_quality = "QualityHiRES"
audio_min_quality = "Quality64k"
codecs = [
"AV1",
"HEV",
"AVC",
]
no_dolby_video = false
no_dolby_audio = false
no_hdr = false
no_hires = false
[danmaku_option]
# 弹幕的一些相关选项,如字体、字号、透明度、停留时间、是否加粗等
duration = 12.0
font = "黑体"
font_size = 25
width_ratio = 1.2
horizontal_gap = 20.0
lane_size = 32
float_percentage = 0.5
bottom_percentage = 0.3
opacity = 76
bold = true
outline = 0.8
time_offset = 0.0
[favorite_list]
# 收藏夹 ID = 存储的位置
52642258 = "/home/amtoaer/HDDs/Videos/Bilibilis/混剪"
```
## Docker Compose 文件示例
该项目为 `Linux/amd64` 与 `Linux/arm64` 提供了 Docker 版本镜像。
Docker 版包含该平台对应版本的可执行文件(位于`/app/bili-sync-rs`),并预装了 FFmpeg其它用法与普通版本完全一致。可查看 [用于构建镜像的 Dockerfile](./Dockerfile)
以下是一个 Docker Compose 的编写示例:
```yaml
services:
bili-sync-rs:
image: amtoaer/bili-sync-rs:v2.0.0
restart: unless-stopped
network_mode: bridge
tty: true # 该选项请仅在日志终端支持彩色输出时启用,否则日志中可能会出现乱码
hostname: bili-sync-rs
container_name: bili-sync-rs
volumes:
- /home/amtoaer/.config/nas/bili-sync-rs:/app/.config/bili-sync
# 以及一些其它必要的挂载,确保此处的挂载与 bili-sync-rs 的配置相匹配
# ...
logging:
driver: "local"
```
## 路线图
- [x] 凭证认证
- [x] 视频选优
- [x] 视频下载
- [x] 支持并发下载
- [x] 支持作为 daemon 运行
- [x] 构建 nfo 和 poster 文件,方便以单集形式导入 emby
- [x] 支持收藏夹翻页,下载全部历史视频
- [x] 对接数据库,提前检查,按需下载
- [x] 支持弹幕下载
- [x] 更好的错误处理
- [x] 更好的日志
- [x] 请求过快出现风控的 workaround
- [x] 提供简单易用的打包(如 docker
- [ ] 支持 UP 主合集下载
## 参考与借鉴
@@ -177,4 +42,4 @@ services:
+ [bilibili-API-collect](https://github.com/SocialSisterYi/bilibili-API-collect) B 站的第三方接口文档
+ [bilibili-api](https://github.com/Nemo2011/bilibili-api) 使用 Python 调用接口的参考实现
+ [danmu2ass](https://github.com/gwy15/danmu2ass) 本项目弹幕下载功能的缝合来源
+ [danmu2ass](https://github.com/gwy15/danmu2ass) 本项目弹幕下载功能的缝合来源

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.1 MiB

BIN
assets/detail.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 342 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1015 KiB

BIN
assets/dir.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.6 MiB

BIN
assets/overview.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 270 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.4 MiB

BIN
assets/play.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 216 KiB

View File

@@ -0,0 +1,56 @@
[package]
name = "bili_sync"
version = { workspace = true }
edition = { workspace = true }
authors = { workspace = true }
license = { workspace = true }
description = { workspace = true }
publish = { workspace = true }
readme = "../../README.md"
[dependencies]
anyhow = { workspace = true }
arc-swap = { workspace = true }
async-stream = { workspace = true }
bili_sync_entity = { workspace = true }
bili_sync_migration = { workspace = true }
chrono = { workspace = true }
clap = { workspace = true }
cookie = { workspace = true }
cow-utils = { workspace = true }
dirs = { workspace = true }
enum_dispatch = { workspace = true }
float-ord = { workspace = true }
futures = { workspace = true }
handlebars = { workspace = true }
hex = { workspace = true }
leaky-bucket = { workspace = true }
md5 = { workspace = true }
memchr = { workspace = true }
once_cell = { workspace = true }
prost = { workspace = true }
quick-xml = { workspace = true }
rand = { workspace = true }
regex = { workspace = true }
reqwest = { workspace = true }
rsa = { workspace = true }
sea-orm = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
serde_urlencoded = { workspace = true }
strum = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true }
toml = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
[dev-dependencies]
assert_matches = { workspace = true }
[package.metadata.release]
release = true
[[bin]]
name = "bili-sync-rs"
path = "src/main.rs"

View File

@@ -0,0 +1,115 @@
use std::path::Path;
use std::pin::Pin;
use anyhow::{Context, Result};
use bili_sync_entity::*;
use futures::Stream;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::{OnConflict, SimpleExpr};
use sea_orm::ActiveValue::Set;
use sea_orm::{DatabaseConnection, Unchanged};
use crate::adapter::{VideoListModel, VideoListModelEnum, _ActiveModel};
use crate::bilibili::{BiliClient, Collection, CollectionItem, CollectionType, VideoInfo};
impl VideoListModel for collection::Model {
fn filter_expr(&self) -> SimpleExpr {
video::Column::CollectionId.eq(self.id)
}
fn set_relation_id(&self, video_model: &mut video::ActiveModel) {
video_model.collection_id = Set(Some(self.id));
}
fn path(&self) -> &Path {
Path::new(self.path.as_str())
}
fn get_latest_row_at(&self) -> DateTime {
self.latest_row_at
}
fn update_latest_row_at(&self, datetime: DateTime) -> _ActiveModel {
_ActiveModel::Collection(collection::ActiveModel {
id: Unchanged(self.id),
latest_row_at: Set(datetime),
..Default::default()
})
}
fn log_refresh_video_start(&self) {
info!("开始扫描{}「{}」..", CollectionType::from(self.r#type), self.name);
}
fn log_refresh_video_end(&self, count: usize) {
info!(
"扫描{}「{}」完成,获取到 {} 条新视频",
CollectionType::from(self.r#type),
self.name,
count,
);
}
fn log_fetch_video_start(&self) {
info!(
"开始填充{}「{}」视频详情..",
CollectionType::from(self.r#type),
self.name
);
}
fn log_fetch_video_end(&self) {
info!("填充{}「{}」视频详情完成", CollectionType::from(self.r#type), self.name);
}
fn log_download_video_start(&self) {
info!("开始下载{}「{}」视频..", CollectionType::from(self.r#type), self.name);
}
fn log_download_video_end(&self) {
info!("下载{}「{}」视频完成", CollectionType::from(self.r#type), self.name);
}
}
pub(super) async fn collection_from<'a>(
collection_item: &'a CollectionItem,
path: &Path,
bili_client: &'a BiliClient,
connection: &DatabaseConnection,
) -> Result<(VideoListModelEnum, Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>)> {
let collection = Collection::new(bili_client, collection_item);
let collection_info = collection.get_info().await?;
collection::Entity::insert(collection::ActiveModel {
s_id: Set(collection_info.sid),
m_id: Set(collection_info.mid),
r#type: Set(collection_info.collection_type.into()),
name: Set(collection_info.name.clone()),
path: Set(path.to_string_lossy().to_string()),
..Default::default()
})
.on_conflict(
OnConflict::columns([
collection::Column::SId,
collection::Column::MId,
collection::Column::Type,
])
.update_columns([collection::Column::Name, collection::Column::Path])
.to_owned(),
)
.exec(connection)
.await?;
Ok((
collection::Entity::find()
.filter(
collection::Column::SId
.eq(collection_item.sid.clone())
.and(collection::Column::MId.eq(collection_item.mid.clone()))
.and(collection::Column::Type.eq(Into::<i32>::into(collection_item.collection_type.clone()))),
)
.one(connection)
.await?
.context("collection not found")?
.into(),
Box::pin(collection.into_video_stream()),
))
}

View File

@@ -0,0 +1,95 @@
use std::path::Path;
use std::pin::Pin;
use anyhow::{Context, Result};
use bili_sync_entity::*;
use futures::Stream;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::{OnConflict, SimpleExpr};
use sea_orm::ActiveValue::Set;
use sea_orm::{DatabaseConnection, Unchanged};
use crate::adapter::{VideoListModel, VideoListModelEnum, _ActiveModel};
use crate::bilibili::{BiliClient, FavoriteList, VideoInfo};
impl VideoListModel for favorite::Model {
fn filter_expr(&self) -> SimpleExpr {
video::Column::FavoriteId.eq(self.id)
}
fn set_relation_id(&self, video_model: &mut video::ActiveModel) {
video_model.favorite_id = Set(Some(self.id));
}
fn path(&self) -> &Path {
Path::new(self.path.as_str())
}
fn get_latest_row_at(&self) -> DateTime {
self.latest_row_at
}
fn update_latest_row_at(&self, datetime: DateTime) -> _ActiveModel {
_ActiveModel::Favorite(favorite::ActiveModel {
id: Unchanged(self.id),
latest_row_at: Set(datetime),
..Default::default()
})
}
fn log_refresh_video_start(&self) {
info!("开始扫描收藏夹「{}」..", self.name);
}
fn log_refresh_video_end(&self, count: usize) {
info!("扫描收藏夹「{}」完成,获取到 {} 条新视频", self.name, count);
}
fn log_fetch_video_start(&self) {
info!("开始填充收藏夹「{}」视频详情..", self.name);
}
fn log_fetch_video_end(&self) {
info!("填充收藏夹「{}」视频详情完成", self.name);
}
fn log_download_video_start(&self) {
info!("开始下载收藏夹「{}」视频..", self.name);
}
fn log_download_video_end(&self) {
info!("下载收藏夹「{}」视频完成", self.name);
}
}
pub(super) async fn favorite_from<'a>(
fid: &str,
path: &Path,
bili_client: &'a BiliClient,
connection: &DatabaseConnection,
) -> Result<(VideoListModelEnum, Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>)> {
let favorite = FavoriteList::new(bili_client, fid.to_owned());
let favorite_info = favorite.get_info().await?;
favorite::Entity::insert(favorite::ActiveModel {
f_id: Set(favorite_info.id),
name: Set(favorite_info.title.clone()),
path: Set(path.to_string_lossy().to_string()),
..Default::default()
})
.on_conflict(
OnConflict::column(favorite::Column::FId)
.update_columns([favorite::Column::Name, favorite::Column::Path])
.to_owned(),
)
.exec(connection)
.await?;
Ok((
favorite::Entity::find()
.filter(favorite::Column::FId.eq(favorite_info.id))
.one(connection)
.await?
.context("favorite not found")?
.into(),
Box::pin(favorite.into_video_stream()),
))
}

View File

@@ -0,0 +1,121 @@
mod collection;
mod favorite;
mod submission;
mod watch_later;
use std::path::Path;
use std::pin::Pin;
use anyhow::Result;
use enum_dispatch::enum_dispatch;
use futures::Stream;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::SimpleExpr;
use sea_orm::DatabaseConnection;
#[rustfmt::skip]
use bili_sync_entity::collection::Model as Collection;
use bili_sync_entity::favorite::Model as Favorite;
use bili_sync_entity::submission::Model as Submission;
use bili_sync_entity::watch_later::Model as WatchLater;
use crate::adapter::collection::collection_from;
use crate::adapter::favorite::favorite_from;
use crate::adapter::submission::submission_from;
use crate::adapter::watch_later::watch_later_from;
use crate::bilibili::{BiliClient, CollectionItem, VideoInfo};
#[enum_dispatch]
pub enum VideoListModelEnum {
Favorite,
Collection,
Submission,
WatchLater,
}
#[enum_dispatch(VideoListModelEnum)]
pub trait VideoListModel {
/// 获取特定视频列表的筛选条件
fn filter_expr(&self) -> SimpleExpr;
// 为 video_model 设置该视频列表的关联 id
fn set_relation_id(&self, video_model: &mut bili_sync_entity::video::ActiveModel);
// 获取视频列表的保存路径
fn path(&self) -> &Path;
/// 获取视频 model 中记录的最新时间
fn get_latest_row_at(&self) -> DateTime;
/// 更新视频 model 中记录的最新时间,此处返回需要更新的 ActiveModel接着调用 save 方法执行保存
/// 不同 VideoListModel 返回的类型不同,为了 VideoListModel 的 object safety 不能使用 impl Trait
/// Box<dyn ActiveModelTrait> 又提示 ActiveModelTrait 没有 object safety因此手写一个 Enum 静态分发
fn update_latest_row_at(&self, datetime: DateTime) -> _ActiveModel;
/// 开始刷新视频
fn log_refresh_video_start(&self);
/// 结束刷新视频
fn log_refresh_video_end(&self, count: usize);
/// 开始填充视频
fn log_fetch_video_start(&self);
/// 结束填充视频
fn log_fetch_video_end(&self);
/// 开始下载视频
fn log_download_video_start(&self);
/// 结束下载视频
fn log_download_video_end(&self);
}
#[derive(Clone, Copy)]
pub enum Args<'a> {
Favorite { fid: &'a str },
Collection { collection_item: &'a CollectionItem },
WatchLater,
Submission { upper_id: &'a str },
}
pub async fn video_list_from<'a>(
args: Args<'a>,
path: &Path,
bili_client: &'a BiliClient,
connection: &DatabaseConnection,
) -> Result<(VideoListModelEnum, Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>)> {
match args {
Args::Favorite { fid } => favorite_from(fid, path, bili_client, connection).await,
Args::Collection { collection_item } => collection_from(collection_item, path, bili_client, connection).await,
Args::WatchLater => watch_later_from(path, bili_client, connection).await,
Args::Submission { upper_id } => submission_from(upper_id, path, bili_client, connection).await,
}
}
pub enum _ActiveModel {
Favorite(bili_sync_entity::favorite::ActiveModel),
Collection(bili_sync_entity::collection::ActiveModel),
Submission(bili_sync_entity::submission::ActiveModel),
WatchLater(bili_sync_entity::watch_later::ActiveModel),
}
impl _ActiveModel {
pub async fn save(self, connection: &DatabaseConnection) -> Result<()> {
match self {
_ActiveModel::Favorite(model) => {
model.save(connection).await?;
}
_ActiveModel::Collection(model) => {
model.save(connection).await?;
}
_ActiveModel::Submission(model) => {
model.save(connection).await?;
}
_ActiveModel::WatchLater(model) => {
model.save(connection).await?;
}
}
Ok(())
}
}

View File

@@ -0,0 +1,95 @@
use std::path::Path;
use std::pin::Pin;
use anyhow::{Context, Result};
use bili_sync_entity::*;
use futures::Stream;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::{OnConflict, SimpleExpr};
use sea_orm::ActiveValue::Set;
use sea_orm::{DatabaseConnection, Unchanged};
use crate::adapter::{VideoListModel, VideoListModelEnum, _ActiveModel};
use crate::bilibili::{BiliClient, Submission, VideoInfo};
impl VideoListModel for submission::Model {
fn filter_expr(&self) -> SimpleExpr {
video::Column::SubmissionId.eq(self.id)
}
fn set_relation_id(&self, video_model: &mut video::ActiveModel) {
video_model.submission_id = Set(Some(self.id));
}
fn path(&self) -> &Path {
Path::new(self.path.as_str())
}
fn get_latest_row_at(&self) -> DateTime {
self.latest_row_at
}
fn update_latest_row_at(&self, datetime: DateTime) -> _ActiveModel {
_ActiveModel::Submission(submission::ActiveModel {
id: Unchanged(self.id),
latest_row_at: Set(datetime),
..Default::default()
})
}
fn log_refresh_video_start(&self) {
info!("开始扫描「{}」投稿..", self.upper_name);
}
fn log_refresh_video_end(&self, count: usize) {
info!("扫描「{}」投稿完成,获取到 {} 条新视频", self.upper_name, count,);
}
fn log_fetch_video_start(&self) {
info!("开始填充「{}」投稿视频详情..", self.upper_name);
}
fn log_fetch_video_end(&self) {
info!("填充「{}」投稿视频详情完成", self.upper_name);
}
fn log_download_video_start(&self) {
info!("开始下载「{}」投稿视频..", self.upper_name);
}
fn log_download_video_end(&self) {
info!("下载「{}」投稿视频完成", self.upper_name);
}
}
pub(super) async fn submission_from<'a>(
upper_id: &str,
path: &Path,
bili_client: &'a BiliClient,
connection: &DatabaseConnection,
) -> Result<(VideoListModelEnum, Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>)> {
let submission = Submission::new(bili_client, upper_id.to_owned());
let upper = submission.get_info().await?;
submission::Entity::insert(submission::ActiveModel {
upper_id: Set(upper.mid.parse()?),
upper_name: Set(upper.name),
path: Set(path.to_string_lossy().to_string()),
..Default::default()
})
.on_conflict(
OnConflict::column(submission::Column::UpperId)
.update_columns([submission::Column::UpperName, submission::Column::Path])
.to_owned(),
)
.exec(connection)
.await?;
Ok((
submission::Entity::find()
.filter(submission::Column::UpperId.eq(upper.mid))
.one(connection)
.await?
.context("submission not found")?
.into(),
Box::pin(submission.into_video_stream()),
))
}

View File

@@ -0,0 +1,92 @@
use std::path::Path;
use std::pin::Pin;
use anyhow::{Context, Result};
use bili_sync_entity::*;
use futures::Stream;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::{OnConflict, SimpleExpr};
use sea_orm::ActiveValue::Set;
use sea_orm::{DatabaseConnection, Unchanged};
use crate::adapter::{VideoListModel, VideoListModelEnum, _ActiveModel};
use crate::bilibili::{BiliClient, VideoInfo, WatchLater};
impl VideoListModel for watch_later::Model {
fn filter_expr(&self) -> SimpleExpr {
video::Column::WatchLaterId.eq(self.id)
}
fn set_relation_id(&self, video_model: &mut video::ActiveModel) {
video_model.watch_later_id = Set(Some(self.id));
}
fn path(&self) -> &Path {
Path::new(self.path.as_str())
}
fn get_latest_row_at(&self) -> DateTime {
self.latest_row_at
}
fn update_latest_row_at(&self, datetime: DateTime) -> _ActiveModel {
_ActiveModel::WatchLater(watch_later::ActiveModel {
id: Unchanged(self.id),
latest_row_at: Set(datetime),
..Default::default()
})
}
fn log_refresh_video_start(&self) {
info!("开始扫描稍后再看..");
}
fn log_refresh_video_end(&self, count: usize) {
info!("扫描稍后再看完成,获取到 {} 条新视频", count);
}
fn log_fetch_video_start(&self) {
info!("开始填充稍后再看视频详情..");
}
fn log_fetch_video_end(&self) {
info!("填充稍后再看视频详情完成");
}
fn log_download_video_start(&self) {
info!("开始下载稍后再看视频..");
}
fn log_download_video_end(&self) {
info!("下载稍后再看视频完成");
}
}
pub(super) async fn watch_later_from<'a>(
path: &Path,
bili_client: &'a BiliClient,
connection: &DatabaseConnection,
) -> Result<(VideoListModelEnum, Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>)> {
let watch_later = WatchLater::new(bili_client);
watch_later::Entity::insert(watch_later::ActiveModel {
id: Set(1),
path: Set(path.to_string_lossy().to_string()),
..Default::default()
})
.on_conflict(
OnConflict::column(watch_later::Column::Id)
.update_column(watch_later::Column::Path)
.to_owned(),
)
.exec(connection)
.await?;
Ok((
watch_later::Entity::find()
.filter(watch_later::Column::Id.eq(1))
.one(connection)
.await?
.context("watch_later not found")?
.into(),
Box::pin(watch_later.into_video_stream()),
))
}

View File

@@ -0,0 +1,388 @@
use anyhow::{bail, Context, Result};
use serde::{Deserialize, Serialize};
use crate::bilibili::error::BiliError;
pub struct PageAnalyzer {
info: serde_json::Value,
}
#[derive(Debug, strum::FromRepr, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
pub enum VideoQuality {
Quality360p = 16,
Quality480p = 32,
Quality720p = 64,
Quality1080p = 80,
Quality1080pPLUS = 112,
Quality1080p60 = 116,
Quality4k = 120,
QualityHdr = 125,
QualityDolby = 126,
Quality8k = 127,
}
#[derive(Debug, Clone, Copy, strum::FromRepr, PartialEq, Eq, Serialize, Deserialize)]
pub enum AudioQuality {
Quality64k = 30216,
Quality132k = 30232,
QualityDolby = 30250,
QualityHiRES = 30251,
Quality192k = 30280,
}
impl Ord for AudioQuality {
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
self.as_sort_key().cmp(&other.as_sort_key())
}
}
impl PartialOrd for AudioQuality {
fn partial_cmp(&self, other: &AudioQuality) -> Option<std::cmp::Ordering> {
Some(self.cmp(other))
}
}
impl AudioQuality {
pub fn as_sort_key(&self) -> isize {
match self {
// 这可以让 Dolby 和 Hi-RES 排在 192k 之后,且 Dolby 和 Hi-RES 之间的顺序不变
Self::QualityHiRES | Self::QualityDolby => (*self as isize) + 40,
_ => *self as isize,
}
}
}
#[allow(clippy::upper_case_acronyms)]
#[derive(Debug, strum::EnumString, strum::Display, strum::AsRefStr, PartialEq, PartialOrd, Serialize, Deserialize)]
pub enum VideoCodecs {
#[strum(serialize = "hev")]
HEV,
#[strum(serialize = "avc")]
AVC,
#[strum(serialize = "av01")]
AV1,
}
// 视频流的筛选偏好
#[derive(Serialize, Deserialize)]
pub struct FilterOption {
pub video_max_quality: VideoQuality,
pub video_min_quality: VideoQuality,
pub audio_max_quality: AudioQuality,
pub audio_min_quality: AudioQuality,
pub codecs: Vec<VideoCodecs>,
pub no_dolby_video: bool,
pub no_dolby_audio: bool,
pub no_hdr: bool,
pub no_hires: bool,
}
impl Default for FilterOption {
fn default() -> Self {
Self {
video_max_quality: VideoQuality::Quality8k,
video_min_quality: VideoQuality::Quality360p,
audio_max_quality: AudioQuality::QualityHiRES,
audio_min_quality: AudioQuality::Quality64k,
codecs: vec![VideoCodecs::AV1, VideoCodecs::HEV, VideoCodecs::AVC],
no_dolby_video: false,
no_dolby_audio: false,
no_hdr: false,
no_hires: false,
}
}
}
// 上游项目中的五种流类型,不过目测应该只有 Flv、DashVideo、DashAudio 三种会被用到
#[derive(Debug, PartialEq, PartialOrd)]
pub enum Stream {
Flv(String),
Html5Mp4(String),
EpositeTryMp4(String),
DashVideo {
url: String,
quality: VideoQuality,
codecs: VideoCodecs,
},
DashAudio {
url: String,
quality: AudioQuality,
},
}
// 通用的获取流链接的方法,交由 Downloader 使用
impl Stream {
pub fn url(&self) -> &str {
match self {
Self::Flv(url) => url,
Self::Html5Mp4(url) => url,
Self::EpositeTryMp4(url) => url,
Self::DashVideo { url, .. } => url,
Self::DashAudio { url, .. } => url,
}
}
}
/// 用于获取视频流的最佳筛选结果,有两种可能:
/// 1. 单个混合流,作为 Mixed 返回
/// 2. 视频、音频分离,作为 VideoAudio 返回,其中音频流可能不存在(对于无声视频,如 BV1J7411H7KQ
#[derive(Debug)]
pub enum BestStream {
VideoAudio { video: Stream, audio: Option<Stream> },
Mixed(Stream),
}
impl PageAnalyzer {
pub fn new(info: serde_json::Value) -> Self {
Self { info }
}
fn is_flv_stream(&self) -> bool {
self.info.get("durl").is_some() && self.info["format"].as_str().is_some_and(|f| f.starts_with("flv"))
}
fn is_html5_mp4_stream(&self) -> bool {
self.info.get("durl").is_some()
&& self.info["format"].as_str().is_some_and(|f| f.starts_with("mp4"))
&& self.info["is_html5"].as_bool().is_some_and(|b| b)
}
fn is_episode_try_mp4_stream(&self) -> bool {
self.info.get("durl").is_some()
&& self.info["format"].as_str().is_some_and(|f| f.starts_with("mp4"))
&& self.info["is_html5"].as_bool().is_none_or(|b| !b)
}
/// 获取所有的视频、音频流,并根据条件筛选
fn streams(&mut self, filter_option: &FilterOption) -> Result<Vec<Stream>> {
if self.is_flv_stream() {
return Ok(vec![Stream::Flv(
self.info["durl"][0]["url"]
.as_str()
.context("invalid flv stream")?
.to_string(),
)]);
}
if self.is_html5_mp4_stream() {
return Ok(vec![Stream::Html5Mp4(
self.info["durl"][0]["url"]
.as_str()
.context("invalid html5 mp4 stream")?
.to_string(),
)]);
}
if self.is_episode_try_mp4_stream() {
return Ok(vec![Stream::EpositeTryMp4(
self.info["durl"][0]["url"]
.as_str()
.context("invalid episode try mp4 stream")?
.to_string(),
)]);
}
let mut streams: Vec<Stream> = Vec::new();
for video in self.info["dash"]["video"]
.as_array()
.ok_or(BiliError::RiskControlOccurred)?
.iter()
{
let (Some(url), Some(quality), Some(codecs)) = (
video["baseUrl"].as_str(),
video["id"].as_u64(),
video["codecs"].as_str(),
) else {
continue;
};
let quality = VideoQuality::from_repr(quality as usize).context("invalid video stream quality")?;
// 从视频流的 codecs 字段中获取编码格式,此处并非精确匹配而是判断包含,比如 codecs 是 av1.42c01e,需要匹配为 av1
let Some(codecs) = [VideoCodecs::HEV, VideoCodecs::AVC, VideoCodecs::AV1]
.into_iter()
.find(|c| codecs.contains(c.as_ref()))
else {
// 少数情况会走到此处,如 codecs 为 dvh1.08.09、hvc1.2.4.L123.90 等,直接跳过,不影响流程
continue;
};
if !filter_option.codecs.contains(&codecs)
|| quality < filter_option.video_min_quality
|| quality > filter_option.video_max_quality
|| (quality == VideoQuality::QualityHdr && filter_option.no_hdr)
|| (quality == VideoQuality::QualityDolby && filter_option.no_dolby_video)
{
continue;
}
streams.push(Stream::DashVideo {
url: url.to_string(),
quality,
codecs,
});
}
if let Some(audios) = self.info["dash"]["audio"].as_array() {
for audio in audios.iter() {
let (Some(url), Some(quality)) = (audio["baseUrl"].as_str(), audio["id"].as_u64()) else {
continue;
};
let quality = AudioQuality::from_repr(quality as usize).context("invalid audio stream quality")?;
if quality < filter_option.audio_min_quality || quality > filter_option.audio_max_quality {
continue;
}
streams.push(Stream::DashAudio {
url: url.to_string(),
quality,
});
}
}
let flac = &self.info["dash"]["flac"]["audio"];
if !(filter_option.no_hires || flac.is_null()) {
let (Some(url), Some(quality)) = (flac["baseUrl"].as_str(), flac["id"].as_u64()) else {
bail!("invalid flac stream");
};
let quality = AudioQuality::from_repr(quality as usize).context("invalid flac stream quality")?;
if quality >= filter_option.audio_min_quality && quality <= filter_option.audio_max_quality {
streams.push(Stream::DashAudio {
url: url.to_string(),
quality,
});
}
}
let dolby_audio = &self.info["dash"]["dolby"]["audio"][0];
if !(filter_option.no_dolby_audio || dolby_audio.is_null()) {
let (Some(url), Some(quality)) = (dolby_audio["baseUrl"].as_str(), dolby_audio["id"].as_u64()) else {
bail!("invalid dolby audio stream");
};
let quality = AudioQuality::from_repr(quality as usize).context("invalid dolby audio stream quality")?;
if quality >= filter_option.audio_min_quality && quality <= filter_option.audio_max_quality {
streams.push(Stream::DashAudio {
url: url.to_string(),
quality,
});
}
}
Ok(streams)
}
pub fn best_stream(&mut self, filter_option: &FilterOption) -> Result<BestStream> {
let streams = self.streams(filter_option)?;
if self.is_flv_stream() || self.is_html5_mp4_stream() || self.is_episode_try_mp4_stream() {
// 按照 streams 中的假设,符合这三种情况的流只有一个,直接取
return Ok(BestStream::Mixed(
streams.into_iter().next().context("no stream found")?,
));
}
let (videos, audios): (Vec<Stream>, Vec<Stream>) =
streams.into_iter().partition(|s| matches!(s, Stream::DashVideo { .. }));
Ok(BestStream::VideoAudio {
video: Iterator::max_by(videos.into_iter(), |a, b| match (a, b) {
(
Stream::DashVideo {
quality: a_quality,
codecs: a_codecs,
..
},
Stream::DashVideo {
quality: b_quality,
codecs: b_codecs,
..
},
) => {
if a_quality != b_quality {
return a_quality.cmp(b_quality);
};
filter_option
.codecs
.iter()
.position(|c| c == b_codecs)
.cmp(&filter_option.codecs.iter().position(|c| c == a_codecs))
}
_ => unreachable!(),
})
.context("no video stream found")?,
audio: Iterator::max_by(audios.into_iter(), |a, b| match (a, b) {
(Stream::DashAudio { quality: a_quality, .. }, Stream::DashAudio { quality: b_quality, .. }) => {
a_quality.cmp(b_quality)
}
_ => unreachable!(),
}),
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bilibili::{BiliClient, Video};
use crate::config::CONFIG;
#[test]
fn test_quality_order() {
assert!([
VideoQuality::Quality360p,
VideoQuality::Quality480p,
VideoQuality::Quality720p,
VideoQuality::Quality1080p,
VideoQuality::Quality1080pPLUS,
VideoQuality::Quality1080p60,
VideoQuality::Quality4k,
VideoQuality::QualityHdr,
VideoQuality::QualityDolby,
VideoQuality::Quality8k
]
.is_sorted());
assert!([
AudioQuality::Quality64k,
AudioQuality::Quality132k,
AudioQuality::Quality192k,
AudioQuality::QualityDolby,
AudioQuality::QualityHiRES,
]
.is_sorted());
}
#[ignore = "only for manual test"]
#[tokio::test]
async fn test_best_stream() {
let testcases = [
// 随便一个 8k + hires 视频
(
"BV1xRChYUE2R",
VideoQuality::Quality8k,
Some(AudioQuality::QualityHiRES),
),
// 一个没有声音的纯视频
("BV1J7411H7KQ", VideoQuality::Quality720p, None),
// 一个杜比全景声的演示片
(
"BV1Mm4y1P7JV",
VideoQuality::Quality4k,
Some(AudioQuality::QualityDolby),
),
];
for (bvid, video_quality, audio_quality) in testcases.into_iter() {
let client = BiliClient::new();
let video = Video::new(&client, bvid.to_owned());
let pages = video.get_pages().await.expect("failed to get pages");
let first_page = pages.into_iter().next().expect("no page found");
let best_stream = video
.get_page_analyzer(&first_page)
.await
.expect("failed to get page analyzer")
.best_stream(&CONFIG.filter_option)
.expect("failed to get best stream");
dbg!(bvid, &best_stream);
match best_stream {
BestStream::VideoAudio {
video: Stream::DashVideo { quality, .. },
audio,
} => {
assert_eq!(quality, video_quality);
assert_eq!(
audio.map(|audio_stream| match audio_stream {
Stream::DashAudio { quality, .. } => quality,
_ => unreachable!(),
}),
audio_quality,
);
}
_ => unreachable!(),
}
}
}
}

View File

@@ -1,10 +1,13 @@
use std::sync::Arc;
use std::time::Duration;
use anyhow::Result;
use anyhow::{Context, Result};
use leaky_bucket::RateLimiter;
use reqwest::{header, Method};
use crate::bilibili::credential::WbiImg;
use crate::bilibili::Credential;
use crate::config::CONFIG;
use crate::config::{RateLimit, CONFIG};
// 一个对 reqwest::Client 的简单封装,用于 Bilibili 请求
#[derive(Clone)]
@@ -31,7 +34,7 @@ impl Client {
.connect_timeout(std::time::Duration::from_secs(10))
.read_timeout(std::time::Duration::from_secs(10))
.build()
.unwrap(),
.expect("failed to build reqwest client"),
)
}
@@ -60,15 +63,32 @@ impl Default for Client {
pub struct BiliClient {
pub client: Client,
limiter: Option<RateLimiter>,
}
impl BiliClient {
pub fn new() -> Self {
let client = Client::new();
Self { client }
let limiter = CONFIG
.concurrent_limit
.rate_limit
.as_ref()
.map(|RateLimit { limit, duration }| {
RateLimiter::builder()
.initial(*limit)
.refill(*limit)
.max(*limit)
.interval(Duration::from_millis(*duration))
.build()
});
Self { client, limiter }
}
pub fn request(&self, method: Method, url: &str) -> reqwest::RequestBuilder {
/// 获取一个预构建的请求,通过该方法获取请求时会检查并等待速率限制
pub async fn request(&self, method: Method, url: &str) -> reqwest::RequestBuilder {
if let Some(limiter) = &self.limiter {
limiter.acquire_one().await;
}
let credential = CONFIG.credential.load();
self.client.request(method, url, credential.as_deref())
}
@@ -85,4 +105,11 @@ impl BiliClient {
CONFIG.credential.store(Some(Arc::new(new_credential)));
CONFIG.save()
}
/// 获取 wbi img用于生成请求签名
pub async fn wbi_img(&self) -> Result<WbiImg> {
let credential = CONFIG.credential.load();
let credential = credential.as_deref().context("no credential found")?;
credential.wbi_img(&self.client).await
}
}

View File

@@ -0,0 +1,281 @@
use std::fmt::{Display, Formatter};
use anyhow::{anyhow, Context, Result};
use async_stream::try_stream;
use futures::Stream;
use reqwest::Method;
use serde::Deserialize;
use serde_json::Value;
use crate::bilibili::credential::encoded_query;
use crate::bilibili::{BiliClient, Validate, VideoInfo, MIXIN_KEY};
#[derive(PartialEq, Eq, Hash, Clone, Debug)]
pub enum CollectionType {
Series,
Season,
}
impl From<CollectionType> for i32 {
fn from(v: CollectionType) -> Self {
match v {
CollectionType::Series => 1,
CollectionType::Season => 2,
}
}
}
impl From<i32> for CollectionType {
fn from(v: i32) -> Self {
match v {
1 => CollectionType::Series,
2 => CollectionType::Season,
_ => panic!("invalid collection type"),
}
}
}
impl Display for CollectionType {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
let s = match self {
CollectionType::Series => "列表",
CollectionType::Season => "合集",
};
write!(f, "{}", s)
}
}
#[derive(PartialEq, Eq, Hash, Debug)]
pub struct CollectionItem {
pub mid: String,
pub sid: String,
pub collection_type: CollectionType,
}
pub struct Collection<'a> {
client: &'a BiliClient,
collection: &'a CollectionItem,
}
#[derive(Debug, PartialEq)]
pub struct CollectionInfo {
pub name: String,
pub mid: i64,
pub sid: i64,
pub collection_type: CollectionType,
}
impl<'de> Deserialize<'de> for CollectionInfo {
fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
#[derive(Deserialize)]
struct CollectionInfoRaw {
mid: i64,
name: String,
season_id: Option<i64>,
series_id: Option<i64>,
}
let raw = CollectionInfoRaw::deserialize(deserializer)?;
let (sid, collection_type) = match (raw.season_id, raw.series_id) {
(Some(sid), None) => (sid, CollectionType::Season),
(None, Some(sid)) => (sid, CollectionType::Series),
_ => return Err(serde::de::Error::custom("invalid collection info")),
};
Ok(CollectionInfo {
mid: raw.mid,
name: raw.name,
sid,
collection_type,
})
}
}
impl<'a> Collection<'a> {
pub fn new(client: &'a BiliClient, collection: &'a CollectionItem) -> Self {
Self { client, collection }
}
pub async fn get_info(&self) -> Result<CollectionInfo> {
let meta = match self.collection.collection_type {
// 没有找到专门获取 Season 信息的接口,所以直接获取第一页,从里面取 meta 信息
CollectionType::Season => self.get_videos(1).await?["data"]["meta"].take(),
CollectionType::Series => self.get_series_info().await?["data"]["meta"].take(),
};
Ok(serde_json::from_value(meta)?)
}
async fn get_series_info(&self) -> Result<Value> {
self.client
.request(Method::GET, "https://api.bilibili.com/x/series/series")
.await
.query(&[("series_id", self.collection.sid.as_str())])
.send()
.await?
.error_for_status()?
.json::<Value>()
.await?
.validate()
}
async fn get_videos(&self, page: i32) -> Result<Value> {
let page = page.to_string();
let (url, query) = match self.collection.collection_type {
CollectionType::Series => (
"https://api.bilibili.com/x/series/archives",
encoded_query(
vec![
("mid", self.collection.mid.as_str()),
("series_id", self.collection.sid.as_str()),
("only_normal", "true"),
("sort", "desc"),
("pn", page.as_str()),
("ps", "30"),
],
MIXIN_KEY.load().as_deref(),
),
),
CollectionType::Season => (
"https://api.bilibili.com/x/polymer/web-space/seasons_archives_list",
encoded_query(
vec![
("mid", self.collection.mid.as_str()),
("season_id", self.collection.sid.as_str()),
("sort_reverse", "true"),
("page_num", page.as_str()),
("page_size", "30"),
],
MIXIN_KEY.load().as_deref(),
),
),
};
self.client
.request(Method::GET, url)
.await
.query(&query)
.send()
.await?
.error_for_status()?
.json::<Value>()
.await?
.validate()
}
pub fn into_video_stream(self) -> impl Stream<Item = Result<VideoInfo>> + 'a {
try_stream! {
let mut page = 1;
loop {
let mut videos = self.get_videos(page).await.with_context(|| {
format!(
"failed to get videos of collection {:?} page {}",
self.collection, page
)
})?;
let archives = &mut videos["data"]["archives"];
if archives.as_array().is_none_or(|v| v.is_empty()) {
Err(anyhow!(
"no videos found in collection {:?} page {}",
self.collection,
page
))?;
}
let videos_info: Vec<VideoInfo> = serde_json::from_value(archives.take()).with_context(|| {
format!(
"failed to parse videos of collection {:?} page {}",
self.collection, page
)
})?;
for video_info in videos_info {
yield video_info;
}
let page_info = &videos["data"]["page"];
let fields = match self.collection.collection_type {
CollectionType::Series => ["num", "size", "total"],
CollectionType::Season => ["page_num", "page_size", "total"],
};
let values = fields
.iter()
.map(|f| page_info[f].as_i64())
.collect::<Vec<Option<i64>>>();
if let [Some(num), Some(size), Some(total)] = values[..] {
if num * size < total {
page += 1;
continue;
}
} else {
Err(anyhow!(
"invalid page info of collection {:?} page {}: read {:?} from {}",
self.collection,
page,
fields,
page_info
))?;
}
break;
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_collection_info_parse() {
let testcases = vec![
(
r#"
{
"category": 0,
"cover": "https://archive.biliimg.com/bfs/archive/a6fbf7a7b9f4af09d9cf40482268634df387ef68.jpg",
"description": "",
"mid": 521722088,
"name": "合集·【命运方舟全剧情解说】",
"ptime": 1714701600,
"season_id": 1987140,
"total": 10
}
"#,
CollectionInfo {
mid: 521722088,
name: "合集·【命运方舟全剧情解说】".to_owned(),
sid: 1987140,
collection_type: CollectionType::Season,
},
),
(
r#"
{
"series_id": 387212,
"mid": 521722088,
"name": "提瓦特冒险记",
"description": "原神沙雕般的游戏体验",
"keywords": [
""
],
"creator": "",
"state": 2,
"last_update_ts": 1633167320,
"total": 3,
"ctime": 1633167320,
"mtime": 1633167320,
"raw_keywords": "",
"category": 1
}
"#,
CollectionInfo {
mid: 521722088,
name: "提瓦特冒险记".to_owned(),
sid: 387212,
collection_type: CollectionType::Series,
},
),
];
for (json, expect) in testcases {
let info: CollectionInfo = serde_json::from_str(json).unwrap();
assert_eq!(info, expect);
}
}
}

View File

@@ -0,0 +1,337 @@
use std::borrow::Cow;
use std::collections::HashSet;
use anyhow::{bail, ensure, Context, Result};
use cookie::Cookie;
use cow_utils::CowUtils;
use regex::Regex;
use reqwest::{header, Method};
use rsa::pkcs8::DecodePublicKey;
use rsa::sha2::Sha256;
use rsa::{Oaep, RsaPublicKey};
use serde::{Deserialize, Serialize};
use crate::bilibili::{Client, Validate};
const MIXIN_KEY_ENC_TAB: [usize; 64] = [
46, 47, 18, 2, 53, 8, 23, 32, 15, 50, 10, 31, 58, 3, 45, 35, 27, 43, 5, 49, 33, 9, 42, 19, 29, 28, 14, 39, 12, 38,
41, 13, 37, 48, 7, 16, 24, 55, 40, 61, 26, 17, 0, 1, 60, 51, 30, 4, 22, 25, 54, 21, 56, 59, 6, 63, 57, 62, 11, 36,
20, 34, 44, 52,
];
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
pub struct Credential {
pub sessdata: String,
pub bili_jct: String,
pub buvid3: String,
pub dedeuserid: String,
pub ac_time_value: String,
}
#[derive(Debug, Deserialize)]
pub struct WbiImg {
img_url: String,
sub_url: String,
}
impl From<WbiImg> for Option<String> {
/// 尝试将 WbiImg 转换成 mixin_key
fn from(value: WbiImg) -> Self {
let key = match (
get_filename(value.img_url.as_str()),
get_filename(value.sub_url.as_str()),
) {
(Some(img_key), Some(sub_key)) => img_key.to_string() + sub_key,
_ => return None,
};
let key = key.as_bytes();
Some(MIXIN_KEY_ENC_TAB.iter().take(32).map(|&x| key[x] as char).collect())
}
}
impl Credential {
pub async fn wbi_img(&self, client: &Client) -> Result<WbiImg> {
let mut res = client
.request(Method::GET, "https://api.bilibili.com/x/web-interface/nav", Some(self))
.send()
.await?
.json::<serde_json::Value>()
.await?
.validate()?;
Ok(serde_json::from_value(res["data"]["wbi_img"].take())?)
}
/// 检查凭据是否有效
pub async fn need_refresh(&self, client: &Client) -> Result<bool> {
let res = client
.request(
Method::GET,
"https://passport.bilibili.com/x/passport-login/web/cookie/info",
Some(self),
)
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()?;
res["data"]["refresh"].as_bool().context("check refresh failed")
}
pub async fn refresh(&self, client: &Client) -> Result<Self> {
let correspond_path = Self::get_correspond_path();
let csrf = self.get_refresh_csrf(client, correspond_path).await?;
let new_credential = self.get_new_credential(client, &csrf).await?;
self.confirm_refresh(client, &new_credential).await?;
Ok(new_credential)
}
fn get_correspond_path() -> String {
// 调用频率很低,让 key 在函数内部构造影响不大
let key = RsaPublicKey::from_public_key_pem(
"-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDLgd2OAkcGVtoE3ThUREbio0Eg
Uc/prcajMKXvkCKFCWhJYJcLkcM2DKKcSeFpD/j6Boy538YXnR6VhcuUJOhH2x71
nzPjfdTcqMz7djHum0qSZA0AyCBDABUqCrfNgCiJ00Ra7GmRj+YCK1NJEuewlb40
JNrRuoEUXpabUzGB8QIDAQAB
-----END PUBLIC KEY-----",
)
.expect("fail to decode public key");
let ts = chrono::Local::now().timestamp_millis();
let data = format!("refresh_{}", ts).into_bytes();
let mut rng = rand::rngs::OsRng;
let encrypted = key
.encrypt(&mut rng, Oaep::new::<Sha256>(), &data)
.expect("fail to encrypt");
hex::encode(encrypted)
}
async fn get_refresh_csrf(&self, client: &Client, correspond_path: String) -> Result<String> {
let res = client
.request(
Method::GET,
format!("https://www.bilibili.com/correspond/1/{}", correspond_path).as_str(),
Some(self),
)
.header(header::COOKIE, "Domain=.bilibili.com")
.send()
.await?
.error_for_status()?;
regex_find(r#"<div id="1-name">(.+?)</div>"#, res.text().await?.as_str())
}
async fn get_new_credential(&self, client: &Client, csrf: &str) -> Result<Credential> {
let mut res = client
.request(
Method::POST,
"https://passport.bilibili.com/x/passport-login/web/cookie/refresh",
Some(self),
)
.header(header::COOKIE, "Domain=.bilibili.com")
.form(&[
// 这里不是 json而是 form data
("csrf", self.bili_jct.as_str()),
("refresh_csrf", csrf),
("refresh_token", self.ac_time_value.as_str()),
("source", "main_web"),
])
.send()
.await?
.error_for_status()?;
// 必须在 .json 前取出 headers否则 res 会被消耗
let headers = std::mem::take(res.headers_mut());
let res = res.json::<serde_json::Value>().await?.validate()?;
let set_cookies = headers.get_all(header::SET_COOKIE);
let mut credential = Self {
buvid3: self.buvid3.clone(),
..Self::default()
};
let required_cookies = HashSet::from(["SESSDATA", "bili_jct", "DedeUserID"]);
let cookies: Vec<Cookie> = set_cookies
.iter()
.filter_map(|x| x.to_str().ok())
.filter_map(|x| Cookie::parse(x).ok())
.filter(|x| required_cookies.contains(x.name()))
.collect();
ensure!(
cookies.len() == required_cookies.len(),
"not all required cookies found"
);
for cookie in cookies {
match cookie.name() {
"SESSDATA" => credential.sessdata = cookie.value().to_string(),
"bili_jct" => credential.bili_jct = cookie.value().to_string(),
"DedeUserID" => credential.dedeuserid = cookie.value().to_string(),
_ => unreachable!(),
}
}
match res["data"]["refresh_token"].as_str() {
Some(token) => credential.ac_time_value = token.to_string(),
None => bail!("refresh_token not found"),
}
Ok(credential)
}
async fn confirm_refresh(&self, client: &Client, new_credential: &Credential) -> Result<()> {
client
.request(
Method::POST,
"https://passport.bilibili.com/x/passport-login/web/confirm/refresh",
// 此处用的是新的凭证
Some(new_credential),
)
.form(&[
("csrf", new_credential.bili_jct.as_str()),
("refresh_token", self.ac_time_value.as_str()),
])
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()?;
Ok(())
}
}
// 用指定的 pattern 正则表达式在 doc 中查找,返回第一个匹配的捕获组
fn regex_find(pattern: &str, doc: &str) -> Result<String> {
let re = Regex::new(pattern)?;
Ok(re
.captures(doc)
.context("no match found")?
.get(1)
.context("no capture found")?
.as_str()
.to_string())
}
fn get_filename(url: &str) -> Option<&str> {
url.rsplit_once('/')
.and_then(|(_, s)| s.rsplit_once('.'))
.map(|(s, _)| s)
}
pub fn encoded_query<'a>(
params: Vec<(&'a str, impl Into<Cow<'a, str>>)>,
mixin_key: Option<impl AsRef<str>>,
) -> Vec<(&'a str, Cow<'a, str>)> {
match mixin_key {
Some(key) => _encoded_query(params, key.as_ref(), chrono::Local::now().timestamp().to_string()),
None => params.into_iter().map(|(k, v)| (k, v.into())).collect(),
}
}
fn _encoded_query<'a>(
params: Vec<(&'a str, impl Into<Cow<'a, str>>)>,
mixin_key: &str,
timestamp: String,
) -> Vec<(&'a str, Cow<'a, str>)> {
let disallowed = ['!', '\'', '(', ')', '*'];
let mut params: Vec<(&'a str, Cow<'a, str>)> = params
.into_iter()
.map(|(k, v)| {
(
k,
match Into::<Cow<'a, str>>::into(v) {
Cow::Borrowed(v) => v.cow_replace(&disallowed[..], ""),
Cow::Owned(v) => v.replace(&disallowed[..], "").into(),
},
)
})
.collect();
params.push(("wts", timestamp.into()));
params.sort_by(|a, b| a.0.cmp(b.0));
let query = serde_urlencoded::to_string(&params)
.expect("fail to encode query")
.replace('+', "%20");
params.push(("w_rid", format!("{:x}", md5::compute(query.clone() + mixin_key)).into()));
params
}
#[cfg(test)]
mod tests {
use assert_matches::assert_matches;
use super::*;
#[test]
fn test_parse_and_find() {
let doc = r#"
<html lang="zh-Hans">
<body>
<div id="1-name">b0cc8411ded2f9db2cff2edb3123acac</div>
</body>
</html>
"#;
assert_eq!(
regex_find(r#"<div id="1-name">(.+?)</div>"#, doc).unwrap(),
"b0cc8411ded2f9db2cff2edb3123acac",
);
}
#[test]
fn test_encode_query() {
let query = vec![
("bar", "五一四".to_string()),
("baz", "1919810".to_string()),
("foo", "one one four".to_string()),
];
assert_eq!(
serde_urlencoded::to_string(query).unwrap().replace('+', "%20"),
"bar=%E4%BA%94%E4%B8%80%E5%9B%9B&baz=1919810&foo=one%20one%20four"
);
}
#[test]
fn test_wbi_key() {
let key = WbiImg {
img_url: "https://i0.hdslb.com/bfs/wbi/7cd084941338484aae1ad9425b84077c.png".to_string(),
sub_url: "https://i0.hdslb.com/bfs/wbi/4932caff0ff746eab6f01bf08b70ac45.png".to_string(),
};
let key = Option::<String>::from(key).expect("fail to convert key");
assert_eq!(key.as_str(), "ea1db124af3c7062474693fa704f4ff8");
// 没有特殊字符
assert_matches!(
&_encoded_query(
vec![("foo", "114"), ("bar", "514"), ("zab", "1919810")],
key.as_str(),
"1702204169".to_string(),
)[..],
[
("bar", Cow::Borrowed(a)),
("foo", Cow::Borrowed(b)),
("wts", Cow::Owned(c)),
("zab", Cow::Borrowed(d)),
("w_rid", Cow::Owned(e)),
] => {
assert_eq!(*a, "514");
assert_eq!(*b, "114");
assert_eq!(c, "1702204169");
assert_eq!(*d, "1919810");
assert_eq!(e, "8f6f2b5b3d485fe1886cec6a0be8c5d4");
}
);
// 有特殊字符
assert_matches!(
&_encoded_query(
vec![("foo", "'1(1)4'"), ("bar", "!5*1!14"), ("zab", "1919810")],
key.as_str(),
"1702204169".to_string(),
)[..],
[
("bar", Cow::Owned(a)),
("foo", Cow::Owned(b)),
("wts", Cow::Owned(c)),
("zab", Cow::Borrowed(d)),
("w_rid", Cow::Owned(e)),
] => {
assert_eq!(a, "5114");
assert_eq!(b, "114");
assert_eq!(c, "1702204169");
assert_eq!(*d, "1919810");
assert_eq!(e, "6a2c86c4b0648ce062ba0dac2de91a85");
}
);
}
}

View File

@@ -0,0 +1,95 @@
use anyhow::{anyhow, Context, Result};
use async_stream::try_stream;
use futures::Stream;
use serde_json::Value;
use crate::bilibili::{BiliClient, Validate, VideoInfo};
pub struct FavoriteList<'a> {
client: &'a BiliClient,
fid: String,
}
#[derive(Debug, serde::Deserialize)]
pub struct FavoriteListInfo {
pub id: i64,
pub title: String,
}
#[derive(Debug, serde::Deserialize)]
pub struct Upper<T> {
pub mid: T,
pub name: String,
pub face: String,
}
impl<'a> FavoriteList<'a> {
pub fn new(client: &'a BiliClient, fid: String) -> Self {
Self { client, fid }
}
pub async fn get_info(&self) -> Result<FavoriteListInfo> {
let mut res = self
.client
.request(reqwest::Method::GET, "https://api.bilibili.com/x/v3/fav/folder/info")
.await
.query(&[("media_id", &self.fid)])
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()?;
Ok(serde_json::from_value(res["data"].take())?)
}
async fn get_videos(&self, page: u32) -> Result<Value> {
self.client
.request(reqwest::Method::GET, "https://api.bilibili.com/x/v3/fav/resource/list")
.await
.query(&[
("media_id", self.fid.as_str()),
("pn", page.to_string().as_str()),
("ps", "20"),
("order", "mtime"),
("type", "0"),
("tid", "0"),
])
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()
}
// 拿到收藏夹的所有权,返回一个收藏夹下的视频流
pub fn into_video_stream(self) -> impl Stream<Item = Result<VideoInfo>> + 'a {
try_stream! {
let mut page = 1;
loop {
let mut videos = self
.get_videos(page)
.await
.with_context(|| format!("failed to get videos of favorite {} page {}", self.fid, page))?;
let medias = &mut videos["data"]["medias"];
if medias.as_array().is_none_or(|v| v.is_empty()) {
Err(anyhow!("no medias found in favorite {} page {}", self.fid, page))?;
}
let videos_info: Vec<VideoInfo> = serde_json::from_value(medias.take())
.with_context(|| format!("failed to parse videos of favorite {} page {}", self.fid, page))?;
for video_info in videos_info {
yield video_info;
}
let has_more = &videos["data"]["has_more"];
if let Some(v) = has_more.as_bool() {
if v {
page += 1;
continue;
}
} else {
Err(anyhow!("has_more is not a bool"))?;
}
break;
}
}
}
}

View File

@@ -0,0 +1,200 @@
use std::sync::Arc;
pub use analyzer::{BestStream, FilterOption};
use anyhow::{bail, ensure, Result};
use arc_swap::ArcSwapOption;
use chrono::serde::ts_seconds;
use chrono::{DateTime, Utc};
pub use client::{BiliClient, Client};
pub use collection::{Collection, CollectionItem, CollectionType};
pub use credential::Credential;
pub use danmaku::DanmakuOption;
pub use error::BiliError;
pub use favorite_list::FavoriteList;
use favorite_list::Upper;
use once_cell::sync::Lazy;
pub use submission::Submission;
pub use video::{Dimension, PageInfo, Video};
pub use watch_later::WatchLater;
mod analyzer;
mod client;
mod collection;
mod credential;
mod danmaku;
mod error;
mod favorite_list;
mod submission;
mod video;
mod watch_later;
static MIXIN_KEY: Lazy<ArcSwapOption<String>> = Lazy::new(Default::default);
pub(crate) fn set_global_mixin_key(key: String) {
MIXIN_KEY.store(Some(Arc::new(key)));
}
pub(crate) trait Validate {
type Output;
fn validate(self) -> Result<Self::Output>;
}
impl Validate for serde_json::Value {
type Output = serde_json::Value;
fn validate(self) -> Result<Self::Output> {
let (code, msg) = match (self["code"].as_i64(), self["message"].as_str()) {
(Some(code), Some(msg)) => (code, msg),
_ => bail!("no code or message found"),
};
ensure!(code == 0, BiliError::RequestFailed(code, msg.to_owned()));
Ok(self)
}
}
#[derive(Debug, serde::Deserialize)]
#[serde(untagged)]
/// 注意此处的顺序是有要求的,因为对于 untagged 的 enum 来说serde 会按照顺序匹配
/// > There is no explicit tag identifying which variant the data contains.
/// > Serde will try to match the data against each variant in order and the first one that deserializes successfully is the one returned.
pub enum VideoInfo {
/// 从视频详情接口获取的视频信息
Detail {
title: String,
bvid: String,
#[serde(rename = "desc")]
intro: String,
#[serde(rename = "pic")]
cover: String,
#[serde(rename = "owner")]
upper: Upper<i64>,
#[serde(with = "ts_seconds")]
ctime: DateTime<Utc>,
#[serde(rename = "pubdate", with = "ts_seconds")]
pubtime: DateTime<Utc>,
pages: Vec<PageInfo>,
state: i32,
},
/// 从收藏夹接口获取的视频信息
Favorite {
title: String,
#[serde(rename = "type")]
vtype: i32,
bvid: String,
intro: String,
cover: String,
upper: Upper<i64>,
#[serde(with = "ts_seconds")]
ctime: DateTime<Utc>,
#[serde(with = "ts_seconds")]
fav_time: DateTime<Utc>,
#[serde(with = "ts_seconds")]
pubtime: DateTime<Utc>,
attr: i32,
},
/// 从稍后再看接口获取的视频信息
WatchLater {
title: String,
bvid: String,
#[serde(rename = "desc")]
intro: String,
#[serde(rename = "pic")]
cover: String,
#[serde(rename = "owner")]
upper: Upper<i64>,
#[serde(with = "ts_seconds")]
ctime: DateTime<Utc>,
#[serde(rename = "add_at", with = "ts_seconds")]
fav_time: DateTime<Utc>,
#[serde(rename = "pubdate", with = "ts_seconds")]
pubtime: DateTime<Utc>,
state: i32,
},
/// 从视频合集/视频列表接口获取的视频信息
Collection {
bvid: String,
#[serde(rename = "pic")]
cover: String,
#[serde(with = "ts_seconds")]
ctime: DateTime<Utc>,
#[serde(rename = "pubdate", with = "ts_seconds")]
pubtime: DateTime<Utc>,
},
// 从用户投稿接口获取的视频信息
Submission {
title: String,
bvid: String,
#[serde(rename = "description")]
intro: String,
#[serde(rename = "pic")]
cover: String,
#[serde(rename = "created", with = "ts_seconds")]
ctime: DateTime<Utc>,
},
}
#[cfg(test)]
mod tests {
use futures::StreamExt;
use super::*;
use crate::utils::init_logger;
#[ignore = "only for manual test"]
#[tokio::test]
async fn test_video_info_type() {
init_logger("None,bili_sync=debug");
let bili_client = BiliClient::new();
// 请求 UP 主视频必须要获取 mixin key使用 key 计算请求参数的签名,否则直接提示权限不足返回空
let Ok(Some(mixin_key)) = bili_client.wbi_img().await.map(|wbi_img| wbi_img.into()) else {
panic!("获取 mixin key 失败");
};
set_global_mixin_key(mixin_key);
// 测试视频合集
let collection_item = CollectionItem {
mid: "521722088".to_string(),
sid: "4523".to_string(),
collection_type: CollectionType::Season,
};
let collection = Collection::new(&bili_client, &collection_item);
let videos = collection
.into_video_stream()
.take(20)
.filter_map(|v| futures::future::ready(v.ok()))
.collect::<Vec<_>>()
.await;
assert!(videos.iter().all(|v| matches!(v, VideoInfo::Collection { .. })));
assert!(videos.iter().rev().is_sorted_by_key(|v| v.release_datetime()));
// 测试收藏夹
let favorite = FavoriteList::new(&bili_client, "3144336058".to_string());
let videos = favorite
.into_video_stream()
.take(20)
.filter_map(|v| futures::future::ready(v.ok()))
.collect::<Vec<_>>()
.await;
assert!(videos.iter().all(|v| matches!(v, VideoInfo::Favorite { .. })));
assert!(videos.iter().rev().is_sorted_by_key(|v| v.release_datetime()));
// 测试稍后再看
let watch_later = WatchLater::new(&bili_client);
let videos = watch_later
.into_video_stream()
.take(20)
.filter_map(|v| futures::future::ready(v.ok()))
.collect::<Vec<_>>()
.await;
assert!(videos.iter().all(|v| matches!(v, VideoInfo::WatchLater { .. })));
assert!(videos.iter().rev().is_sorted_by_key(|v| v.release_datetime()));
// 测试投稿
let submission = Submission::new(&bili_client, "956761".to_string());
let videos = submission
.into_video_stream()
.take(20)
.filter_map(|v| futures::future::ready(v.ok()))
.collect::<Vec<_>>()
.await;
assert!(videos.iter().all(|v| matches!(v, VideoInfo::Submission { .. })));
assert!(videos.iter().rev().is_sorted_by_key(|v| v.release_datetime()));
}
}

View File

@@ -0,0 +1,90 @@
use anyhow::{anyhow, Context, Result};
use arc_swap::access::Access;
use async_stream::try_stream;
use futures::Stream;
use reqwest::Method;
use serde_json::Value;
use crate::bilibili::credential::encoded_query;
use crate::bilibili::favorite_list::Upper;
use crate::bilibili::{BiliClient, Validate, VideoInfo, MIXIN_KEY};
pub struct Submission<'a> {
client: &'a BiliClient,
upper_id: String,
}
impl<'a> Submission<'a> {
pub fn new(client: &'a BiliClient, upper_id: String) -> Self {
Self { client, upper_id }
}
pub async fn get_info(&self) -> Result<Upper<String>> {
let mut res = self
.client
.request(Method::GET, "https://api.bilibili.com/x/web-interface/card")
.await
.query(&[("mid", self.upper_id.as_str())])
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()?;
Ok(serde_json::from_value(res["data"]["card"].take())?)
}
async fn get_videos(&self, page: i32) -> Result<Value> {
self.client
.request(Method::GET, "https://api.bilibili.com/x/space/wbi/arc/search")
.await
.query(&encoded_query(
vec![
("mid", self.upper_id.as_str()),
("order", "pubdate"),
("order_avoided", "true"),
("platform", "web"),
("web_location", "1550101"),
("pn", page.to_string().as_str()),
("ps", "30"),
],
MIXIN_KEY.load().as_deref(),
))
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()
}
pub fn into_video_stream(self) -> impl Stream<Item = Result<VideoInfo>> + 'a {
try_stream! {
let mut page = 1;
loop {
let mut videos = self
.get_videos(page)
.await
.with_context(|| format!("failed to get videos of upper {} page {}", self.upper_id, page))?;
let vlist = &mut videos["data"]["list"]["vlist"];
if vlist.as_array().is_none_or(|v| v.is_empty()) {
Err(anyhow!("no medias found in upper {} page {}", self.upper_id, page))?;
}
let videos_info: Vec<VideoInfo> = serde_json::from_value(vlist.take())
.with_context(|| format!("failed to parse videos of upper {} page {}", self.upper_id, page))?;
for video_info in videos_info {
yield video_info;
}
let count = &videos["data"]["page"]["count"];
if let Some(v) = count.as_i64() {
if v > (page * 30) as i64 {
page += 1;
continue;
}
} else {
Err(anyhow!("count is not an i64"))?;
}
break;
}
}
}
}

View File

@@ -1,4 +1,4 @@
use anyhow::{bail, Result};
use anyhow::{ensure, Result};
use futures::stream::FuturesUnordered;
use futures::TryStreamExt;
use prost::Message;
@@ -6,8 +6,9 @@ use reqwest::Method;
use crate::bilibili::analyzer::PageAnalyzer;
use crate::bilibili::client::BiliClient;
use crate::bilibili::credential::encoded_query;
use crate::bilibili::danmaku::{DanmakuElem, DanmakuWriter, DmSegMobileReply};
use crate::bilibili::Validate;
use crate::bilibili::{Validate, VideoInfo, MIXIN_KEY};
static MASK_CODE: u64 = 2251799813685247;
static XOR_CODE: u64 = 23442827791579;
@@ -39,7 +40,7 @@ impl serde::Serialize for Tag {
}
#[derive(Debug, serde::Deserialize, Default)]
pub struct PageInfo {
pub cid: i32,
pub cid: i64,
pub page: i32,
#[serde(rename = "part")]
pub name: String,
@@ -61,10 +62,28 @@ impl<'a> Video<'a> {
Self { client, aid, bvid }
}
/// 直接调用视频信息接口获取详细的视频信息,视频信息中包含了视频的分页信息
pub async fn get_view_info(&self) -> Result<VideoInfo> {
let mut res = self
.client
.request(Method::GET, "https://api.bilibili.com/x/web-interface/view")
.await
.query(&[("aid", &self.aid), ("bvid", &self.bvid)])
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()?;
Ok(serde_json::from_value(res["data"].take())?)
}
#[allow(unused)]
pub async fn get_pages(&self) -> Result<Vec<PageInfo>> {
let mut res = self
.client
.request(Method::GET, "https://api.bilibili.com/x/player/pagelist")
.await
.query(&[("aid", &self.aid), ("bvid", &self.bvid)])
.send()
.await?
@@ -79,6 +98,7 @@ impl<'a> Video<'a> {
let mut res = self
.client
.request(Method::GET, "https://api.bilibili.com/x/web-interface/view/detail/tag")
.await
.query(&[("aid", &self.aid), ("bvid", &self.bvid)])
.send()
.await?
@@ -92,7 +112,7 @@ impl<'a> Video<'a> {
pub async fn get_danmaku_writer(&self, page: &'a PageInfo) -> Result<DanmakuWriter> {
let tasks = FuturesUnordered::new();
for i in 1..=(page.duration + 359) / 360 {
tasks.push(self.get_danmaku_segment(page, i as i32));
tasks.push(self.get_danmaku_segment(page, i as i64));
}
let result: Vec<Vec<DanmakuElem>> = tasks.try_collect().await?;
let mut result: Vec<DanmakuElem> = result.into_iter().flatten().collect();
@@ -100,23 +120,23 @@ impl<'a> Video<'a> {
Ok(DanmakuWriter::new(page, result.into_iter().map(|x| x.into()).collect()))
}
async fn get_danmaku_segment(&self, page: &PageInfo, segment_idx: i32) -> Result<Vec<DanmakuElem>> {
async fn get_danmaku_segment(&self, page: &PageInfo, segment_idx: i64) -> Result<Vec<DanmakuElem>> {
let mut res = self
.client
.request(Method::GET, "http://api.bilibili.com/x/v2/dm/web/seg.so")
.await
.query(&[("type", 1), ("oid", page.cid), ("segment_index", segment_idx)])
.send()
.await?
.error_for_status()?;
let headers = std::mem::take(res.headers_mut());
let content_type = headers.get("content-type");
if !content_type.is_some_and(|v| v == "application/octet-stream") {
bail!(
"unexpected content type: {:?}, body: {:?}",
content_type,
res.text().await
);
}
ensure!(
content_type.is_some_and(|v| v == "application/octet-stream"),
"unexpected content type: {:?}, body: {:?}",
content_type,
res.text().await
);
Ok(DmSegMobileReply::decode(res.bytes().await?)?.elems)
}
@@ -124,14 +144,18 @@ impl<'a> Video<'a> {
let mut res = self
.client
.request(Method::GET, "https://api.bilibili.com/x/player/wbi/playurl")
.query(&[
("avid", self.aid.as_str()),
("cid", page.cid.to_string().as_str()),
("qn", "127"),
("otype", "json"),
("fnval", "4048"),
("fourk", "1"),
])
.await
.query(&encoded_query(
vec![
("avid", self.aid.as_str()),
("cid", page.cid.to_string().as_str()),
("qn", "127"),
("otype", "json"),
("fnval", "4048"),
("fourk", "1"),
],
MIXIN_KEY.load().as_deref(),
))
.send()
.await?
.error_for_status()?
@@ -148,7 +172,7 @@ fn bvid_to_aid(bvid: &str) -> u64 {
(bvid[4], bvid[7]) = (bvid[7], bvid[4]);
let mut tmp = 0u64;
for char in bvid.into_iter().skip(3) {
let idx = DATA.iter().position(|&x| x == char).unwrap();
let idx = DATA.iter().position(|&x| x == char).expect("invalid bvid");
tmp = tmp * BASE + idx as u64;
}
(tmp & MASK_CODE) ^ XOR_CODE
@@ -158,8 +182,8 @@ fn bvid_to_aid(bvid: &str) -> u64 {
mod tests {
use super::*;
#[tokio::test]
async fn test_bvid_to_aid() {
#[test]
fn test_bvid_to_aid() {
assert_eq!(bvid_to_aid("BV1Tr421n746"), 1401752220u64);
assert_eq!(bvid_to_aid("BV1sH4y1s7fe"), 1051892992u64);
}

View File

@@ -0,0 +1,45 @@
use anyhow::{anyhow, Context, Result};
use async_stream::try_stream;
use futures::Stream;
use serde_json::Value;
use crate::bilibili::{BiliClient, Validate, VideoInfo};
pub struct WatchLater<'a> {
client: &'a BiliClient,
}
impl<'a> WatchLater<'a> {
pub fn new(client: &'a BiliClient) -> Self {
Self { client }
}
async fn get_videos(&self) -> Result<Value> {
self.client
.request(reqwest::Method::GET, "https://api.bilibili.com/x/v2/history/toview")
.await
.send()
.await?
.error_for_status()?
.json::<serde_json::Value>()
.await?
.validate()
}
pub fn into_video_stream(self) -> impl Stream<Item = Result<VideoInfo>> + 'a {
try_stream! {
let mut videos = self
.get_videos()
.await
.with_context(|| "Failed to get watch later list")?;
let list = &mut videos["data"]["list"];
if list.as_array().is_none_or(|v| v.is_empty()) {
Err(anyhow!("No videos found in watch later list"))?;
}
let videos_info: Vec<VideoInfo> =
serde_json::from_value(list.take()).with_context(|| "Failed to parse watch later list")?;
for video_info in videos_info {
yield video_info;
}
}
}
}

View File

@@ -0,0 +1,11 @@
use clap::Parser;
#[derive(Parser)]
#[command(version, about, long_about = None)]
pub struct Args {
#[arg(short, long, env = "SCAN_ONLY")]
pub scan_only: bool,
#[arg(short, long, default_value = "None,bili_sync=info", env = "RUST_LOG")]
pub log_level: String,
}

View File

@@ -0,0 +1,87 @@
use std::path::PathBuf;
use clap::Parser;
use handlebars::handlebars_helper;
use once_cell::sync::Lazy;
use crate::config::clap::Args;
use crate::config::item::PathSafeTemplate;
use crate::config::Config;
/// 全局的 CONFIG可以从中读取配置信息
pub static CONFIG: Lazy<Config> = Lazy::new(load_config);
/// 全局的 TEMPLATE用来渲染 video_name 和 page_name 模板
pub static TEMPLATE: Lazy<handlebars::Handlebars> = Lazy::new(|| {
let mut handlebars = handlebars::Handlebars::new();
handlebars_helper!(truncate: |s: String, len: usize| {
if s.chars().count() > len {
s.chars().take(len).collect::<String>()
} else {
s.to_string()
}
});
handlebars.register_helper("truncate", Box::new(truncate));
handlebars
.path_safe_register("video", &CONFIG.video_name)
.expect("failed to register video template");
handlebars
.path_safe_register("page", &CONFIG.page_name)
.expect("failed to register page template");
handlebars
});
/// 全局的 ARGS用来解析命令行参数
pub static ARGS: Lazy<Args> = Lazy::new(Args::parse);
/// 全局的 CONFIG_DIR表示配置文件夹的路径
pub static CONFIG_DIR: Lazy<PathBuf> =
Lazy::new(|| dirs::config_dir().expect("No config path found").join("bili-sync"));
#[cfg(not(test))]
fn load_config() -> Config {
let config = Config::load().unwrap_or_else(|err| {
if err
.downcast_ref::<std::io::Error>()
.is_none_or(|e| e.kind() != std::io::ErrorKind::NotFound)
{
panic!("加载配置文件失败,错误为: {err}");
}
warn!("配置文件不存在,使用默认配置...");
let default_config = Config::default();
default_config.save().expect("保存默认配置时遇到错误");
info!("已将默认配置写入 {}", CONFIG_DIR.join("config.toml").display());
default_config
});
info!("检查配置文件..");
config.check();
info!("配置文件检查通过");
config
}
#[cfg(test)]
fn load_config() -> Config {
let credential = match (
std::env::var("TEST_SESSDATA"),
std::env::var("TEST_BILI_JCT"),
std::env::var("TEST_BUVID3"),
std::env::var("TEST_DEDEUSERID"),
std::env::var("TEST_AC_TIME_VALUE"),
) {
(Ok(sessdata), Ok(bili_jct), Ok(buvid3), Ok(dedeuserid), Ok(ac_time_value)) => {
Some(std::sync::Arc::new(crate::bilibili::Credential {
sessdata,
bili_jct,
buvid3,
dedeuserid,
ac_time_value,
}))
}
_ => None,
};
Config {
credential: arc_swap::ArcSwapOption::from(credential),
..Default::default()
}
}

View File

@@ -0,0 +1,139 @@
use std::collections::HashMap;
use std::path::PathBuf;
use anyhow::Result;
use serde::de::{Deserializer, MapAccess, Visitor};
use serde::ser::SerializeMap;
use serde::{Deserialize, Serialize};
use crate::bilibili::{CollectionItem, CollectionType};
use crate::utils::filenamify::filenamify;
/// 稍后再看的配置
#[derive(Serialize, Deserialize, Default)]
pub struct WatchLaterConfig {
pub enabled: bool,
pub path: PathBuf,
}
/// NFO 文件使用的时间类型
#[derive(Serialize, Deserialize, Default)]
#[serde(rename_all = "lowercase")]
pub enum NFOTimeType {
#[default]
FavTime,
PubTime,
}
/// 并发下载相关的配置
#[derive(Serialize, Deserialize)]
pub struct ConcurrentLimit {
pub video: usize,
pub page: usize,
pub rate_limit: Option<RateLimit>,
}
#[derive(Serialize, Deserialize)]
pub struct RateLimit {
pub limit: usize,
pub duration: u64,
}
impl Default for ConcurrentLimit {
fn default() -> Self {
Self {
video: 3,
page: 2,
// 默认的限速配置,每 250ms 允许请求 4 次
rate_limit: Some(RateLimit {
limit: 4,
duration: 250,
}),
}
}
}
pub trait PathSafeTemplate {
fn path_safe_register(&mut self, name: &'static str, template: &'static str) -> Result<()>;
fn path_safe_render(&self, name: &'static str, data: &serde_json::Value) -> Result<String>;
}
/// 通过将模板字符串中的分隔符替换为自定义的字符串,使得模板字符串中的分隔符得以保留
impl PathSafeTemplate for handlebars::Handlebars<'_> {
fn path_safe_register(&mut self, name: &'static str, template: &'static str) -> Result<()> {
Ok(self.register_template_string(name, template.replace(std::path::MAIN_SEPARATOR_STR, "__SEP__"))?)
}
fn path_safe_render(&self, name: &'static str, data: &serde_json::Value) -> Result<String> {
Ok(filenamify(&self.render(name, data)?).replace("__SEP__", std::path::MAIN_SEPARATOR_STR))
}
}
/* 后面是用于自定义 Collection 的序列化、反序列化的样板代码 */
pub(super) fn serialize_collection_list<S>(
collection_list: &HashMap<CollectionItem, PathBuf>,
serializer: S,
) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut map = serializer.serialize_map(Some(collection_list.len()))?;
for (k, v) in collection_list {
let prefix = match k.collection_type {
CollectionType::Series => "series",
CollectionType::Season => "season",
};
map.serialize_entry(&[prefix, &k.mid, &k.sid].join(":"), v)?;
}
map.end()
}
pub(super) fn deserialize_collection_list<'de, D>(deserializer: D) -> Result<HashMap<CollectionItem, PathBuf>, D::Error>
where
D: Deserializer<'de>,
{
struct CollectionListVisitor;
impl<'de> Visitor<'de> for CollectionListVisitor {
type Value = HashMap<CollectionItem, PathBuf>;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("a map of collection list")
}
fn visit_map<A>(self, mut map: A) -> Result<Self::Value, A::Error>
where
A: MapAccess<'de>,
{
let mut collection_list = HashMap::new();
while let Some((key, value)) = map.next_entry::<String, PathBuf>()? {
let collection_item = match key.split(':').collect::<Vec<&str>>().as_slice() {
[prefix, mid, sid] => {
let collection_type = match *prefix {
"series" => CollectionType::Series,
"season" => CollectionType::Season,
_ => {
return Err(serde::de::Error::custom(
"invalid collection type, should be series or season",
))
}
};
CollectionItem {
mid: mid.to_string(),
sid: sid.to_string(),
collection_type,
}
}
_ => {
return Err(serde::de::Error::custom(
"invalid collection key, should be series:mid:sid or season:mid:sid",
))
}
};
collection_list.insert(collection_item, value);
}
Ok(collection_list)
}
}
deserializer.deserialize_map(CollectionListVisitor)
}

View File

@@ -5,33 +5,20 @@ use std::sync::Arc;
use anyhow::Result;
use arc_swap::ArcSwapOption;
use once_cell::sync::Lazy;
use serde::{Deserialize, Serialize};
use crate::bilibili::{Credential, DanmakuOption, FilterOption};
mod clap;
mod global;
mod item;
pub static CONFIG: Lazy<Config> = Lazy::new(|| {
let config = Config::load().unwrap_or_else(|err| {
if err
.downcast_ref::<std::io::Error>()
.map_or(true, |e| e.kind() != std::io::ErrorKind::NotFound)
{
panic!("加载配置文件失败,错误为: {err}");
}
warn!("配置文件不存在,使用默认配置...");
Config::new()
});
// 放到外面,确保新的配置项被保存
info!("配置加载完毕,覆盖刷新原有配置");
config.save().unwrap();
// 检查配置文件内容
info!("校验配置文件内容...");
config.check();
config
});
use crate::bilibili::{CollectionItem, Credential, DanmakuOption, FilterOption};
pub use crate::config::global::{ARGS, CONFIG, CONFIG_DIR, TEMPLATE};
use crate::config::item::{deserialize_collection_list, serialize_collection_list, ConcurrentLimit};
pub use crate::config::item::{NFOTimeType, PathSafeTemplate, RateLimit, WatchLaterConfig};
pub static CONFIG_DIR: Lazy<PathBuf> =
Lazy::new(|| dirs::config_dir().expect("No config path found").join("bili-sync"));
fn default_time_format() -> String {
"%Y-%m-%d".to_string()
}
#[derive(Serialize, Deserialize)]
pub struct Config {
@@ -40,38 +27,76 @@ pub struct Config {
#[serde(default)]
pub danmaku_option: DanmakuOption,
pub favorite_list: HashMap<String, PathBuf>,
#[serde(
default,
serialize_with = "serialize_collection_list",
deserialize_with = "deserialize_collection_list"
)]
pub collection_list: HashMap<CollectionItem, PathBuf>,
#[serde(default)]
pub submission_list: HashMap<String, PathBuf>,
#[serde(default)]
pub watch_later: WatchLaterConfig,
pub video_name: Cow<'static, str>,
pub page_name: Cow<'static, str>,
pub interval: u64,
pub upper_path: PathBuf,
#[serde(default)]
pub nfo_time_type: NFOTimeType,
#[serde(default)]
pub concurrent_limit: ConcurrentLimit,
#[serde(default = "default_time_format")]
pub time_format: String,
}
impl Default for Config {
fn default() -> Self {
Self::new()
}
}
impl Config {
fn new() -> Self {
Self {
credential: ArcSwapOption::from(Some(Arc::new(Credential::default()))),
filter_option: FilterOption::default(),
danmaku_option: DanmakuOption::default(),
favorite_list: HashMap::new(),
collection_list: HashMap::new(),
submission_list: HashMap::new(),
watch_later: Default::default(),
video_name: Cow::Borrowed("{{title}}"),
page_name: Cow::Borrowed("{{bvid}}"),
interval: 1200,
upper_path: CONFIG_DIR.join("upper_face"),
nfo_time_type: NFOTimeType::FavTime,
concurrent_limit: ConcurrentLimit::default(),
time_format: default_time_format(),
}
}
}
/// 简单的预检查
impl Config {
pub fn save(&self) -> Result<()> {
let config_path = CONFIG_DIR.join("config.toml");
std::fs::create_dir_all(&*CONFIG_DIR)?;
std::fs::write(config_path, toml::to_string_pretty(self)?)?;
Ok(())
}
#[cfg(not(test))]
fn load() -> Result<Self> {
let config_path = CONFIG_DIR.join("config.toml");
let config_content = std::fs::read_to_string(config_path)?;
Ok(toml::from_str(&config_content)?)
}
#[cfg(not(test))]
pub fn check(&self) {
let mut ok = true;
if self.favorite_list.is_empty() {
if self.favorite_list.is_empty() && self.collection_list.is_empty() && !self.watch_later.enabled {
ok = false;
error!("未设置需监听的收藏夹,程序空转没有意义");
error!("没有配置任何需要扫描的内容,程序空转没有意义");
}
if self.watch_later.enabled && !self.watch_later.path.is_absolute() {
error!(
"稍后再看保存的路径应为绝对路径,检测到:{}",
self.watch_later.path.display()
);
}
for path in self.favorite_list.values() {
if !path.is_absolute() {
@@ -109,7 +134,10 @@ impl Config {
error!("未设置 Credential 信息");
}
}
if !(self.concurrent_limit.video > 0 && self.concurrent_limit.page > 0) {
ok = false;
error!("video 和 page 允许的并发数必须大于 0");
}
if !ok {
panic!(
"位于 {} 的配置文件不合法,请参考提示信息修复后继续运行",
@@ -117,17 +145,4 @@ impl Config {
);
}
}
fn load() -> Result<Self> {
let config_path = CONFIG_DIR.join("config.toml");
let config_content = std::fs::read_to_string(config_path)?;
Ok(toml::from_str(&config_content)?)
}
pub fn save(&self) -> Result<()> {
let config_path = CONFIG_DIR.join("config.toml");
std::fs::create_dir_all(&*CONFIG_DIR)?;
std::fs::write(config_path, toml::to_string_pretty(self)?)?;
Ok(())
}
}

View File

@@ -0,0 +1,25 @@
use anyhow::Result;
use bili_sync_migration::{Migrator, MigratorTrait};
use sea_orm::{ConnectOptions, Database, DatabaseConnection};
use crate::config::CONFIG_DIR;
fn database_url() -> String {
format!("sqlite://{}?mode=rwc", CONFIG_DIR.join("data.sqlite").to_string_lossy())
}
pub async fn database_connection() -> Result<DatabaseConnection> {
let mut option = ConnectOptions::new(database_url());
option
.max_connections(100)
.min_connections(5)
.acquire_timeout(std::time::Duration::from_secs(90));
Ok(Database::connect(option).await?)
}
pub async fn migrate_database() -> Result<()> {
// 注意此处使用内部构造的 DatabaseConnection而不是通过 database_connection() 获取
// 这是因为使用多个连接的 Connection 会导致奇怪的迁移顺序问题,而使用默认的连接选项不会
let connection = Database::connect(database_url()).await?;
Ok(Migrator::up(&connection, None).await?)
}

View File

@@ -1,10 +1,11 @@
use core::str;
use std::path::Path;
use anyhow::{anyhow, Result};
use anyhow::{bail, ensure, Result};
use futures::StreamExt;
use reqwest::Method;
use tokio::fs::{self, File};
use tokio::io;
use tokio::io::AsyncWriteExt;
use crate::bilibili::Client;
pub struct Downloader {
@@ -24,10 +25,22 @@ impl Downloader {
fs::create_dir_all(parent).await?;
}
let mut file = File::create(path).await?;
let mut res = self.client.request(Method::GET, url, None).send().await?.bytes_stream();
while let Some(item) = res.next().await {
io::copy(&mut item?.as_ref(), &mut file).await?;
let resp = self.client.request(Method::GET, url, None).send().await?;
let expected = resp.content_length().unwrap_or_default();
let mut received = 0u64;
let mut stream = resp.bytes_stream();
while let Some(bytes) = stream.next().await {
let bytes = bytes?;
received += bytes.len() as u64;
file.write_all(&bytes).await?;
}
file.flush().await?;
ensure!(
received >= expected,
"received {} bytes, expected {} bytes",
received,
expected
);
Ok(())
}
@@ -35,24 +48,19 @@ impl Downloader {
let output = tokio::process::Command::new("ffmpeg")
.args([
"-i",
video_path.to_str().unwrap(),
video_path.to_string_lossy().as_ref(),
"-i",
audio_path.to_str().unwrap(),
audio_path.to_string_lossy().as_ref(),
"-c",
"copy",
"-y",
output_path.to_str().unwrap(),
output_path.to_string_lossy().as_ref(),
])
.output()
.await?;
if !output.status.success() {
return match String::from_utf8(output.stderr) {
Ok(err) => Err(anyhow!(err)),
_ => Err(anyhow!("ffmpeg error")),
};
bail!("ffmpeg error: {}", str::from_utf8(&output.stderr).unwrap_or("unknown"));
}
let _ = fs::remove_file(video_path).await;
let _ = fs::remove_file(audio_path).await;
Ok(())
}
}

View File

@@ -0,0 +1,83 @@
#[macro_use]
extern crate tracing;
mod adapter;
mod bilibili;
mod config;
mod database;
mod downloader;
mod error;
mod utils;
mod workflow;
use std::path::PathBuf;
use once_cell::sync::Lazy;
use tokio::time;
use crate::adapter::Args;
use crate::bilibili::BiliClient;
use crate::config::{ARGS, CONFIG};
use crate::database::{database_connection, migrate_database};
use crate::utils::init_logger;
use crate::workflow::process_video_list;
#[tokio::main]
async fn main() {
init_logger(&ARGS.log_level);
Lazy::force(&CONFIG);
migrate_database().await.expect("数据库迁移失败");
let connection = database_connection().await.expect("获取数据库连接失败");
let mut anchor = chrono::Local::now().date_naive();
let bili_client = BiliClient::new();
let params = build_params();
loop {
'inner: {
match bili_client.wbi_img().await.map(|wbi_img| wbi_img.into()) {
Ok(Some(mixin_key)) => bilibili::set_global_mixin_key(mixin_key),
Ok(_) => {
error!("解析 mixin key 失败,等待下一轮执行");
break 'inner;
}
Err(e) => {
error!("获取 mixin key 遇到错误:{e},等待下一轮执行");
break 'inner;
}
};
if anchor != chrono::Local::now().date_naive() {
if let Err(e) = bili_client.check_refresh().await {
error!("检查刷新 Credential 遇到错误:{e},等待下一轮执行");
break 'inner;
}
anchor = chrono::Local::now().date_naive();
}
for (args, path) in &params {
if let Err(e) = process_video_list(*args, &bili_client, path, &connection).await {
error!("处理过程遇到错误:{e}");
}
}
info!("本轮任务执行完毕,等待下一轮执行");
}
time::sleep(time::Duration::from_secs(CONFIG.interval)).await;
}
}
fn build_params() -> Vec<(Args<'static>, &'static PathBuf)> {
let mut params = Vec::new();
CONFIG
.favorite_list
.iter()
.for_each(|(fid, path)| params.push((Args::Favorite { fid }, path)));
CONFIG
.collection_list
.iter()
.for_each(|(collection_item, path)| params.push((Args::Collection { collection_item }, path)));
if CONFIG.watch_later.enabled {
params.push((Args::WatchLater, &CONFIG.watch_later.path));
}
CONFIG
.submission_list
.iter()
.for_each(|(upper_id, path)| params.push((Args::Submission { upper_id }, path)));
params
}

View File

@@ -0,0 +1,182 @@
use chrono::{DateTime, NaiveDateTime, Utc};
use sea_orm::ActiveValue::{NotSet, Set};
use sea_orm::IntoActiveModel;
use crate::bilibili::{PageInfo, VideoInfo};
impl VideoInfo {
/// 在检测视频更新时,通过该方法将 VideoInfo 转换为简单的 ActiveModel此处仅填充一些简单信息后续会使用详情覆盖
pub fn into_simple_model(self) -> bili_sync_entity::video::ActiveModel {
let default = bili_sync_entity::video::ActiveModel {
id: NotSet,
created_at: NotSet,
// 此处不使用 ActiveModel::default() 是为了让其它字段有默认值
..bili_sync_entity::video::Model::default().into_active_model()
};
match self {
VideoInfo::Collection {
bvid,
cover,
ctime,
pubtime,
} => bili_sync_entity::video::ActiveModel {
bvid: Set(bvid),
cover: Set(cover),
ctime: Set(ctime.naive_utc()),
pubtime: Set(pubtime.naive_utc()),
category: Set(2), // 视频合集里的内容类型肯定是视频
valid: Set(true),
..default
},
VideoInfo::Favorite {
title,
vtype,
bvid,
intro,
cover,
upper,
ctime,
fav_time,
pubtime,
attr,
} => bili_sync_entity::video::ActiveModel {
bvid: Set(bvid),
name: Set(title),
category: Set(vtype),
intro: Set(intro),
cover: Set(cover),
ctime: Set(ctime.naive_utc()),
pubtime: Set(pubtime.naive_utc()),
favtime: Set(fav_time.naive_utc()),
download_status: Set(0),
valid: Set(attr == 0),
upper_id: Set(upper.mid),
upper_name: Set(upper.name),
upper_face: Set(upper.face),
..default
},
VideoInfo::WatchLater {
title,
bvid,
intro,
cover,
upper,
ctime,
fav_time,
pubtime,
state,
} => bili_sync_entity::video::ActiveModel {
bvid: Set(bvid),
name: Set(title),
category: Set(2), // 稍后再看里的内容类型肯定是视频
intro: Set(intro),
cover: Set(cover),
ctime: Set(ctime.naive_utc()),
pubtime: Set(pubtime.naive_utc()),
favtime: Set(fav_time.naive_utc()),
download_status: Set(0),
valid: Set(state == 0),
upper_id: Set(upper.mid),
upper_name: Set(upper.name),
upper_face: Set(upper.face),
..default
},
VideoInfo::Submission {
title,
bvid,
intro,
cover,
ctime,
} => bili_sync_entity::video::ActiveModel {
bvid: Set(bvid),
name: Set(title),
intro: Set(intro),
cover: Set(cover),
ctime: Set(ctime.naive_utc()),
category: Set(2), // 投稿视频的内容类型肯定是视频
valid: Set(true),
..default
},
_ => unreachable!(),
}
}
/// 填充视频详情时调用,该方法会将视频详情附加到原有的 Model 上
/// 特殊地,如果在检测视频更新时记录了 favtime那么 favtime 会维持原样,否则会使用 pubtime 填充
pub fn into_detail_model(self, base_model: bili_sync_entity::video::Model) -> bili_sync_entity::video::ActiveModel {
match self {
VideoInfo::Detail {
title,
bvid,
intro,
cover,
upper,
ctime,
pubtime,
state,
..
} => bili_sync_entity::video::ActiveModel {
bvid: Set(bvid),
name: Set(title),
category: Set(2),
intro: Set(intro),
cover: Set(cover),
ctime: Set(ctime.naive_utc()),
pubtime: Set(pubtime.naive_utc()),
favtime: if base_model.favtime != NaiveDateTime::default() {
NotSet // 之前设置了 favtime不覆盖
} else {
Set(pubtime.naive_utc()) // 未设置过 favtime使用 pubtime 填充
},
download_status: Set(0),
valid: Set(state == 0),
upper_id: Set(upper.mid),
upper_name: Set(upper.name),
upper_face: Set(upper.face),
..base_model.into_active_model()
},
_ => unreachable!(),
}
}
/// 获取视频的发布时间,用于对时间做筛选检查新视频
pub fn release_datetime(&self) -> &DateTime<Utc> {
match self {
VideoInfo::Collection { pubtime: time, .. }
| VideoInfo::Favorite { fav_time: time, .. }
| VideoInfo::WatchLater { fav_time: time, .. }
| VideoInfo::Submission { ctime: time, .. } => time,
_ => unreachable!(),
}
}
}
impl PageInfo {
pub fn into_active_model(
self,
video_model: &bili_sync_entity::video::Model,
) -> bili_sync_entity::page::ActiveModel {
let (width, height) = match &self.dimension {
Some(d) => {
if d.rotate == 0 {
(Some(d.width), Some(d.height))
} else {
(Some(d.height), Some(d.width))
}
}
None => (None, None),
};
bili_sync_entity::page::ActiveModel {
video_id: Set(video_model.id),
cid: Set(self.cid),
pid: Set(self.page),
name: Set(self.name),
width: Set(width),
height: Set(height),
duration: Set(self.duration),
image: Set(self.first_frame),
download_status: Set(0),
..Default::default()
}
}
}

View File

@@ -0,0 +1,61 @@
macro_rules! regex {
($re:literal $(,)?) => {{
static RE: once_cell::sync::OnceCell<regex::Regex> = once_cell::sync::OnceCell::new();
RE.get_or_init(|| regex::Regex::new($re).expect("invalid regex"))
}};
}
pub fn filenamify<S: AsRef<str>>(input: S) -> String {
let reserved = regex!("[<>:\"/\\\\|?*\u{0000}-\u{001F}\u{007F}\u{0080}-\u{009F}]+");
let windows_reserved = regex!("^(con|prn|aux|nul|com\\d|lpt\\d)$");
let outer_periods = regex!("^\\.+|\\.+$");
let replacement = "_";
let input = reserved.replace_all(input.as_ref(), replacement);
let input = outer_periods.replace_all(input.as_ref(), replacement);
let mut result = input.into_owned();
if windows_reserved.is_match(result.as_str()) {
result.push_str(replacement);
}
result
}
#[cfg(test)]
mod tests {
use super::filenamify;
#[test]
fn test_filenamify() {
assert_eq!(filenamify("foo/bar"), "foo_bar");
assert_eq!(filenamify("foo//bar"), "foo_bar");
assert_eq!(filenamify("//foo//bar//"), "_foo_bar_");
assert_eq!(filenamify("foo\\bar"), "foo_bar");
assert_eq!(filenamify("foo\\\\\\bar"), "foo_bar");
assert_eq!(filenamify(r"foo\\bar"), "foo_bar");
assert_eq!(filenamify(r"foo\\\\\\bar"), "foo_bar");
assert_eq!(filenamify("////foo////bar////"), "_foo_bar_");
assert_eq!(filenamify("foo\u{0000}bar"), "foo_bar");
assert_eq!(filenamify("\"foo<>bar*"), "_foo_bar_");
assert_eq!(filenamify("."), "_");
assert_eq!(filenamify(".."), "_");
assert_eq!(filenamify("./"), "__");
assert_eq!(filenamify("../"), "__");
assert_eq!(filenamify("../../foo/bar"), "__.._foo_bar");
assert_eq!(filenamify("foo.bar."), "foo.bar_");
assert_eq!(filenamify("foo.bar.."), "foo.bar_");
assert_eq!(filenamify("foo.bar..."), "foo.bar_");
assert_eq!(filenamify("con"), "con_");
assert_eq!(filenamify("com1"), "com1_");
assert_eq!(filenamify(":nul|"), "_nul_");
assert_eq!(filenamify("foo/bar/nul"), "foo_bar_nul");
assert_eq!(filenamify("file:///file.tar.gz"), "file_file.tar.gz");
assert_eq!(filenamify("http://www.google.com"), "http_www.google.com");
assert_eq!(
filenamify("https://www.youtube.com/watch?v=dQw4w9WgXcQ"),
"https_www.youtube.com_watch_v=dQw4w9WgXcQ"
);
}
}

View File

@@ -0,0 +1,30 @@
use serde_json::json;
use crate::config::CONFIG;
pub fn video_format_args(video_model: &bili_sync_entity::video::Model) -> serde_json::Value {
json!({
"bvid": &video_model.bvid,
"title": &video_model.name,
"upper_name": &video_model.upper_name,
"upper_mid": &video_model.upper_id,
"pubtime": &video_model.pubtime.and_utc().format(&CONFIG.time_format).to_string(),
"fav_time": &video_model.favtime.and_utc().format(&CONFIG.time_format).to_string(),
})
}
pub fn page_format_args(
video_model: &bili_sync_entity::video::Model,
page_model: &bili_sync_entity::page::Model,
) -> serde_json::Value {
json!({
"bvid": &video_model.bvid,
"title": &video_model.name,
"upper_name": &video_model.upper_name,
"upper_mid": &video_model.upper_id,
"ptitle": &page_model.name,
"pid": page_model.pid,
"pubtime": video_model.pubtime.and_utc().format(&CONFIG.time_format).to_string(),
"fav_time": video_model.favtime.and_utc().format(&CONFIG.time_format).to_string(),
})
}

View File

@@ -0,0 +1,21 @@
pub mod convert;
pub mod filenamify;
pub mod format_arg;
pub mod model;
pub mod nfo;
pub mod status;
use tracing_subscriber::util::SubscriberInitExt;
pub fn init_logger(log_level: &str) {
tracing_subscriber::fmt::Subscriber::builder()
.compact()
.with_env_filter(tracing_subscriber::EnvFilter::builder().parse_lossy(log_level))
.with_target(false)
.with_timer(tracing_subscriber::fmt::time::ChronoLocal::new(
"%b %d %H:%M:%S".to_owned(),
))
.finish()
.try_init()
.expect("初始化日志失败");
}

View File

@@ -0,0 +1,119 @@
use anyhow::{Context, Result};
use bili_sync_entity::*;
use sea_orm::entity::prelude::*;
use sea_orm::sea_query::{OnConflict, SimpleExpr};
use sea_orm::DatabaseTransaction;
use crate::adapter::{VideoListModel, VideoListModelEnum};
use crate::bilibili::{PageInfo, VideoInfo};
use crate::utils::status::STATUS_COMPLETED;
/// 筛选未填充的视频
pub async fn filter_unfilled_videos(
additional_expr: SimpleExpr,
conn: &DatabaseConnection,
) -> Result<Vec<video::Model>> {
video::Entity::find()
.filter(
video::Column::Valid
.eq(true)
.and(video::Column::DownloadStatus.eq(0))
.and(video::Column::Category.eq(2))
.and(video::Column::SinglePage.is_null())
.and(additional_expr),
)
.all(conn)
.await
.context("filter unfilled videos failed")
}
/// 筛选未处理完成的视频和视频页
pub async fn filter_unhandled_video_pages(
additional_expr: SimpleExpr,
connection: &DatabaseConnection,
) -> Result<Vec<(video::Model, Vec<page::Model>)>> {
video::Entity::find()
.filter(
video::Column::Valid
.eq(true)
.and(video::Column::DownloadStatus.lt(STATUS_COMPLETED))
.and(video::Column::Category.eq(2))
.and(video::Column::SinglePage.is_not_null())
.and(additional_expr),
)
.find_with_related(page::Entity)
.all(connection)
.await
.context("filter unhandled video pages failed")
}
/// 尝试创建 Video Model如果发生冲突则忽略
pub async fn create_videos(
videos_info: Vec<VideoInfo>,
video_list_model: &VideoListModelEnum,
connection: &DatabaseConnection,
) -> Result<()> {
let video_models = videos_info
.into_iter()
.map(|v| {
let mut model = v.into_simple_model();
video_list_model.set_relation_id(&mut model);
model
})
.collect::<Vec<_>>();
video::Entity::insert_many(video_models)
// 这里想表达的是 on 索引名,但 sea-orm 的 api 似乎只支持列名而不支持索引名,好在留空可以达到相同的目的
.on_conflict(OnConflict::new().do_nothing().to_owned())
.do_nothing()
.exec(connection)
.await?;
Ok(())
}
/// 尝试创建 Page Model如果发生冲突则忽略
pub async fn create_pages(
pages_info: Vec<PageInfo>,
video_model: &bili_sync_entity::video::Model,
connection: &DatabaseTransaction,
) -> Result<()> {
let page_models = pages_info
.into_iter()
.map(|p| p.into_active_model(video_model))
.collect::<Vec<page::ActiveModel>>();
for page_chunk in page_models.chunks(50) {
page::Entity::insert_many(page_chunk.to_vec())
.on_conflict(
OnConflict::columns([page::Column::VideoId, page::Column::Pid])
.do_nothing()
.to_owned(),
)
.do_nothing()
.exec(connection)
.await?;
}
Ok(())
}
/// 更新视频 model 的下载状态
pub async fn update_videos_model(videos: Vec<video::ActiveModel>, connection: &DatabaseConnection) -> Result<()> {
video::Entity::insert_many(videos)
.on_conflict(
OnConflict::column(video::Column::Id)
.update_columns([video::Column::DownloadStatus, video::Column::Path])
.to_owned(),
)
.exec(connection)
.await?;
Ok(())
}
/// 更新视频页 model 的下载状态
pub async fn update_pages_model(pages: Vec<page::ActiveModel>, connection: &DatabaseConnection) -> Result<()> {
let query = page::Entity::insert_many(pages).on_conflict(
OnConflict::column(page::Column::Id)
.update_columns([page::Column::DownloadStatus, page::Column::Path])
.to_owned(),
);
query.exec(connection).await?;
Ok(())
}

View File

@@ -0,0 +1,307 @@
use anyhow::Result;
use bili_sync_entity::*;
use quick_xml::events::{BytesCData, BytesText};
use quick_xml::writer::Writer;
use quick_xml::Error;
use tokio::io::AsyncWriteExt;
use crate::config::NFOTimeType;
#[allow(clippy::upper_case_acronyms)]
pub enum NFOMode {
MOVIE,
TVSHOW,
EPOSODE,
UPPER,
}
pub enum ModelWrapper<'a> {
Video(&'a video::Model),
Page(&'a page::Model),
}
pub struct NFOSerializer<'a>(pub ModelWrapper<'a>, pub NFOMode);
/// serde xml 似乎不太好用,先这么裸着写
/// (真是又臭又长啊
impl NFOSerializer<'_> {
pub async fn generate_nfo(self, nfo_time_type: &NFOTimeType) -> Result<String> {
let mut buffer = r#"<?xml version="1.0" encoding="utf-8" standalone="yes"?>
"#
.as_bytes()
.to_vec();
let mut tokio_buffer = tokio::io::BufWriter::new(&mut buffer);
let mut writer = Writer::new_with_indent(&mut tokio_buffer, b' ', 4);
match self {
NFOSerializer(ModelWrapper::Video(v), NFOMode::MOVIE) => {
let nfo_time = match nfo_time_type {
NFOTimeType::FavTime => v.favtime,
NFOTimeType::PubTime => v.pubtime,
};
writer
.create_element("movie")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer
.create_element("plot")
.write_cdata_content_async(BytesCData::new(&v.intro))
.await?;
writer.create_element("outline").write_empty_async().await?;
writer
.create_element("title")
.write_text_content_async(BytesText::new(&v.name))
.await?;
writer
.create_element("actor")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer
.create_element("name")
.write_text_content_async(BytesText::new(&v.upper_id.to_string()))
.await?;
writer
.create_element("role")
.write_text_content_async(BytesText::new(&v.upper_name))
.await?;
Ok(writer)
})
.await?;
writer
.create_element("year")
.write_text_content_async(BytesText::new(&nfo_time.format("%Y").to_string()))
.await?;
if let Some(tags) = &v.tags {
let tags: Vec<String> = serde_json::from_value(tags.clone()).unwrap_or_default();
for tag in tags {
writer
.create_element("genre")
.write_text_content_async(BytesText::new(&tag))
.await?;
}
}
writer
.create_element("uniqueid")
.with_attribute(("type", "bilibili"))
.write_text_content_async(BytesText::new(&v.bvid))
.await?;
writer
.create_element("aired")
.write_text_content_async(BytesText::new(&nfo_time.format("%Y-%m-%d").to_string()))
.await?;
Ok(writer)
})
.await?;
}
NFOSerializer(ModelWrapper::Video(v), NFOMode::TVSHOW) => {
let nfo_time = match nfo_time_type {
NFOTimeType::FavTime => v.favtime,
NFOTimeType::PubTime => v.pubtime,
};
writer
.create_element("tvshow")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer
.create_element("plot")
.write_cdata_content_async(BytesCData::new(&v.intro))
.await?;
writer.create_element("outline").write_empty_async().await?;
writer
.create_element("title")
.write_text_content_async(BytesText::new(&v.name))
.await?;
writer
.create_element("actor")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer
.create_element("name")
.write_text_content_async(BytesText::new(&v.upper_id.to_string()))
.await?;
writer
.create_element("role")
.write_text_content_async(BytesText::new(&v.upper_name))
.await?;
Ok(writer)
})
.await?;
writer
.create_element("year")
.write_text_content_async(BytesText::new(&nfo_time.format("%Y").to_string()))
.await?;
if let Some(tags) = &v.tags {
let tags: Vec<String> = serde_json::from_value(tags.clone()).unwrap_or_default();
for tag in tags {
writer
.create_element("genre")
.write_text_content_async(BytesText::new(&tag))
.await?;
}
}
writer
.create_element("uniqueid")
.with_attribute(("type", "bilibili"))
.write_text_content_async(BytesText::new(&v.bvid))
.await?;
writer
.create_element("aired")
.write_text_content_async(BytesText::new(&nfo_time.format("%Y-%m-%d").to_string()))
.await?;
Ok(writer)
})
.await?;
}
NFOSerializer(ModelWrapper::Video(v), NFOMode::UPPER) => {
writer
.create_element("person")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer.create_element("plot").write_empty_async().await?;
writer.create_element("outline").write_empty_async().await?;
writer
.create_element("lockdata")
.write_text_content_async(BytesText::new("false"))
.await?;
writer
.create_element("dateadded")
.write_text_content_async(BytesText::new(
&v.pubtime.format("%Y-%m-%d %H:%M:%S").to_string(),
))
.await?;
writer
.create_element("title")
.write_text_content_async(BytesText::new(&v.upper_id.to_string()))
.await?;
writer
.create_element("sorttitle")
.write_text_content_async(BytesText::new(&v.upper_id.to_string()))
.await?;
Ok(writer)
})
.await?;
}
NFOSerializer(ModelWrapper::Page(p), NFOMode::EPOSODE) => {
writer
.create_element("episodedetails")
.write_inner_content_async::<_, _, Error>(|writer| async move {
writer.create_element("plot").write_empty_async().await?;
writer.create_element("outline").write_empty_async().await?;
writer
.create_element("title")
.write_text_content_async(BytesText::new(&p.name))
.await?;
writer
.create_element("season")
.write_text_content_async(BytesText::new("1"))
.await?;
writer
.create_element("episode")
.write_text_content_async(BytesText::new(&p.pid.to_string()))
.await?;
Ok(writer)
})
.await?;
}
_ => unreachable!(),
}
tokio_buffer.flush().await?;
Ok(String::from_utf8(buffer)?)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_generate_nfo() {
let video = video::Model {
intro: "intro".to_string(),
name: "name".to_string(),
upper_id: 1,
upper_name: "upper_name".to_string(),
favtime: chrono::NaiveDateTime::new(
chrono::NaiveDate::from_ymd_opt(2022, 2, 2).unwrap(),
chrono::NaiveTime::from_hms_opt(2, 2, 2).unwrap(),
),
pubtime: chrono::NaiveDateTime::new(
chrono::NaiveDate::from_ymd_opt(2033, 3, 3).unwrap(),
chrono::NaiveTime::from_hms_opt(3, 3, 3).unwrap(),
),
bvid: "bvid".to_string(),
tags: Some(serde_json::json!(["tag1", "tag2"])),
..Default::default()
};
assert_eq!(
NFOSerializer(ModelWrapper::Video(&video), NFOMode::MOVIE)
.generate_nfo(&NFOTimeType::PubTime)
.await
.unwrap(),
r#"<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<movie>
<plot><![CDATA[intro]]></plot>
<outline/>
<title>name</title>
<actor>
<name>1</name>
<role>upper_name</role>
</actor>
<year>2033</year>
<genre>tag1</genre>
<genre>tag2</genre>
<uniqueid type="bilibili">bvid</uniqueid>
<aired>2033-03-03</aired>
</movie>"#,
);
assert_eq!(
NFOSerializer(ModelWrapper::Video(&video), NFOMode::TVSHOW)
.generate_nfo(&NFOTimeType::FavTime)
.await
.unwrap(),
r#"<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<tvshow>
<plot><![CDATA[intro]]></plot>
<outline/>
<title>name</title>
<actor>
<name>1</name>
<role>upper_name</role>
</actor>
<year>2022</year>
<genre>tag1</genre>
<genre>tag2</genre>
<uniqueid type="bilibili">bvid</uniqueid>
<aired>2022-02-02</aired>
</tvshow>"#,
);
assert_eq!(
NFOSerializer(ModelWrapper::Video(&video), NFOMode::UPPER)
.generate_nfo(&NFOTimeType::FavTime)
.await
.unwrap(),
r#"<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<person>
<plot/>
<outline/>
<lockdata>false</lockdata>
<dateadded>2033-03-03 03:03:03</dateadded>
<title>1</title>
<sorttitle>1</sorttitle>
</person>"#,
);
let page = page::Model {
name: "name".to_string(),
pid: 3,
..Default::default()
};
assert_eq!(
NFOSerializer(ModelWrapper::Page(&page), NFOMode::EPOSODE)
.generate_nfo(&NFOTimeType::FavTime)
.await
.unwrap(),
r#"<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<episodedetails>
<plot/>
<outline/>
<title>name</title>
<season>1</season>
<episode>3</episode>
</episodedetails>"#,
);
}
}

View File

@@ -2,96 +2,85 @@ use anyhow::Result;
static STATUS_MAX_RETRY: u32 = 0b100;
static STATUS_OK: u32 = 0b111;
pub static STATUS_COMPLETED: u32 = 1 << 31;
/// 用来表示下载的状态,不想写太多列了,所以仅使用一个 u32 表示。
/// 从低位开始,固定每三位表示一种数据的状态,从 0b000 开始,每失败一次加一,最多 0b100即重试 4 次),
/// 如果成功,将对应的三位设置为 0b111
/// 当所有任务都成功或者由于尝试次数过多失败,为 status 最高位打上标记 1将来不再继续尝试
/// 从低位开始,固定每三位表示一种子任务的状态。
/// 子任务状态从 0b000 开始,每执行失败一次将状态加一,最多 0b100即允许重试 4 次),该值定义为 STATUS_MAX_RETRY
/// 如果子任务执行成功,将状态设置为 0b111该值定义为 STATUS_OK
/// 子任务达到最大失败次数或者执行成功时,认为该子任务已经完成。
/// 当所有子任务都已经完成时,为最高位打上标记 1表示整个下载任务已经完成。
#[derive(Clone)]
pub struct Status(u32);
impl Status {
/// 如果 status 整体大于等于 1 << 31则表示任务已经被处理过不再需要重试。
/// 数据库可以使用 status < Status::handled() 来筛选需要处理的内容。
pub fn handled() -> u32 {
1 << 31
}
fn new(status: u32) -> Self {
Self(status)
}
/// 一般仅需要被内部调用,用来设置最高位的标记
fn set_flag(&mut self, handled: bool) {
if handled {
/// 设置最高位的完成标记
fn set_completed(&mut self, completed: bool) {
if completed {
self.0 |= 1 << 31;
} else {
self.0 &= !(1 << 31);
}
}
/// 从低到高检查状态,如果该位置的任务应该继续尝试执行,则返回 true否则返回 false
fn should_run(&self, size: usize) -> Vec<bool> {
assert!(size < 10, "u32 can only store 10 status");
(0..size).map(|x| self.check_continue(x)).collect()
// 获取最高位的完成标记
fn get_completed(&self) -> bool {
self.0 >> 31 == 1
}
/// 如果任务的执行次数小于 STATUS_MAX_RETRY说明可以继续运行
/// 获取某个子任务的状态
fn get_status(&self, offset: usize) -> u32 {
let helper = !0u32;
(self.0 & (helper << (offset * 3)) & (helper >> (32 - 3 * offset - 3))) >> (offset * 3)
}
// 将某个子任务的状态加一(在任务失败时使用)
fn plus_one(&mut self, offset: usize) {
self.0 += 1 << (3 * offset);
}
// 设置某个子任务的状态为 STATUS_OK在任务成功时使用
fn set_ok(&mut self, offset: usize) {
self.0 |= STATUS_OK << (3 * offset);
}
/// 检查某个子任务是否还应该继续执行,实际是检查该子任务的状态是否小于 STATUS_MAX_RETRY
fn check_continue(&self, offset: usize) -> bool {
assert!(offset < 10, "u32 can only store 10 status");
self.get_status(offset) < STATUS_MAX_RETRY
}
/// 根据任务结果更新状态,如果任务成功,设置为 STATUS_OK否则加一
/// 依次检查所有子任务是否还应该继续执行,返回一个 bool 数组
fn should_run(&self, size: usize) -> Vec<bool> {
(0..size).map(|x| self.check_continue(x)).collect()
}
/// 根据子任务执行结果更新子任务的状态
/// 如果 Result 是 Ok那么认为任务执行成功将状态设置为 STATUS_OK
/// 如果 Result 是 Err那么认为任务执行失败将状态加一
fn set_result(&mut self, result: &Result<()>, offset: usize) {
if self.get_status(offset) < STATUS_MAX_RETRY {
match result {
Ok(_) => self.set_ok(offset),
Err(_) => self.plus_one(offset),
}
}
}
/// 根据任务结果更新状态,任务结果是一个 Result 数组,需要与子任务一一对应
/// 如果所有子任务都已经完成,那么打上最高位的完成标记
fn update_status(&mut self, result: &[Result<()>]) {
assert!(result.len() < 10, "u32 can only store 10 status");
for (i, res) in result.iter().enumerate() {
self.set_result(res, i);
}
if self.should_run(result.len()).iter().all(|x| !x) {
// 所有任务都成功或者由于尝试次数过多失败,为 status 最高位打上标记,将来不再重试
self.set_flag(true)
self.set_completed(true)
}
}
fn set_result(&mut self, result: &Result<()>, offset: usize) {
if result.is_ok() {
// 如果任务已经执行到最大次数,那么此时 Result 也是 Ok此时不应该更新状态
if self.get_status(offset) < STATUS_MAX_RETRY {
self.set_ok(offset);
}
} else {
self.plus_one(offset);
}
}
/// 根据 mask 设置状态,如果 mask 为 false则清除对应的状态
fn set_mask(&mut self, mask: &[bool]) {
assert!(mask.len() < 10, "u32 can only store 10 status");
for (i, &m) in mask.iter().enumerate() {
if !m {
self.clear(i);
self.set_flag(false);
}
}
}
fn plus_one(&mut self, offset: usize) {
self.0 += 1 << (3 * offset);
}
fn set_ok(&mut self, offset: usize) {
self.0 |= STATUS_OK << (3 * offset);
}
fn clear(&mut self, offset: usize) {
self.0 &= !(STATUS_OK << (3 * offset));
}
fn get_status(&self, offset: usize) -> u32 {
let helper = !0u32;
(self.0 & (helper << (offset * 3)) & (helper >> (32 - 3 * offset - 3))) >> (offset * 3)
}
}
impl From<Status> for u32 {
@@ -100,7 +89,7 @@ impl From<Status> for u32 {
}
}
/// 从前到后分别表示视频封面、视频信息、Up 主头像、Up 主信息、分 P 下载
/// 包含五个子任务,从前到后依次是视频封面、视频信息、Up 主头像、Up 主信息、分 P 下载
#[derive(Clone)]
pub struct VideoStatus(Status);
@@ -109,11 +98,6 @@ impl VideoStatus {
Self(Status::new(status))
}
pub fn set_mask(&mut self, clear: &[bool]) {
assert!(clear.len() == 5, "VideoStatus should have 5 status");
self.0.set_mask(clear)
}
pub fn should_run(&self) -> Vec<bool> {
self.0.should_run(5)
}
@@ -130,7 +114,7 @@ impl From<VideoStatus> for u32 {
}
}
/// 从前到后分别表示:视频封面、视频内容、视频信息
/// 包含四个子任务,从前到后分别:视频封面、视频内容、视频信息、视频弹幕
#[derive(Clone)]
pub struct PageStatus(Status);
@@ -139,11 +123,6 @@ impl PageStatus {
Self(Status::new(status))
}
pub fn set_mask(&mut self, clear: &[bool]) {
assert!(clear.len() == 4, "PageStatus should have 4 status");
self.0.set_mask(clear)
}
pub fn should_run(&self) -> Vec<bool> {
self.0.should_run(4)
}
@@ -152,6 +131,10 @@ impl PageStatus {
assert!(result.len() == 4, "PageStatus should have 4 status");
self.0.update_status(result)
}
pub fn get_completed(&self) -> bool {
self.0.get_completed()
}
}
impl From<PageStatus> for u32 {
@@ -177,6 +160,6 @@ mod test {
}
status.update_status(&[Err(anyhow!("")), Ok(()), Ok(())]);
assert_eq!(status.should_run(3), vec![false, false, false]);
assert_eq!(u32::from(status), 0b111_111_100 | Status::handled());
assert_eq!(u32::from(status), 0b111_111_100 | STATUS_COMPLETED);
}
}

View File

@@ -1,237 +1,226 @@
use std::collections::HashMap;
use std::env::{args, var};
use std::path::{Path, PathBuf};
use std::pin::Pin;
use anyhow::{bail, Result};
use entity::{favorite, page, video};
use filenamify::filenamify;
use anyhow::{anyhow, bail, Context, Result};
use bili_sync_entity::*;
use futures::stream::{FuturesOrdered, FuturesUnordered};
use futures::{pin_mut, Future, StreamExt};
use once_cell::sync::Lazy;
use futures::{Future, Stream, StreamExt};
use sea_orm::entity::prelude::*;
use sea_orm::ActiveValue::Set;
use sea_orm::TransactionTrait;
use serde_json::json;
use tokio::fs;
use tokio::sync::{Mutex, Semaphore};
use crate::bilibili::{BestStream, BiliClient, BiliError, Dimension, FavoriteList, PageInfo, Video};
use crate::config::CONFIG;
use crate::core::status::{PageStatus, VideoStatus};
use crate::core::utils::{
create_video_pages, create_videos, exist_labels, filter_unfilled_videos, handle_favorite_info, total_video_count,
unhandled_videos_pages, update_pages_model, update_videos_model, ModelWrapper, NFOMode, NFOSerializer, TEMPLATE,
};
use crate::adapter::{video_list_from, Args, VideoListModel, VideoListModelEnum};
use crate::bilibili::{BestStream, BiliClient, BiliError, Dimension, PageInfo, Video, VideoInfo};
use crate::config::{PathSafeTemplate, ARGS, CONFIG, TEMPLATE};
use crate::downloader::Downloader;
use crate::error::{DownloadAbortError, ProcessPageError};
use crate::utils::format_arg::{page_format_args, video_format_args};
use crate::utils::model::{
create_pages, create_videos, filter_unfilled_videos, filter_unhandled_video_pages, update_pages_model,
update_videos_model,
};
use crate::utils::nfo::{ModelWrapper, NFOMode, NFOSerializer};
use crate::utils::status::{PageStatus, VideoStatus};
pub static SCAN_ONLY: Lazy<bool> = Lazy::new(|| var("SCAN_ONLY").is_ok() || args().any(|arg| arg == "--scan-only"));
/// 处理某个收藏夹,首先刷新收藏夹信息,然后下载收藏夹中未下载成功的视频
pub async fn process_favorite_list(
/// 完整地处理某个视频列表
pub async fn process_video_list(
args: Args<'_>,
bili_client: &BiliClient,
fid: &str,
path: &Path,
connection: &DatabaseConnection,
) -> Result<()> {
let favorite_model = refresh_favorite_list(bili_client, fid, path, connection).await?;
let favorite_model = fetch_video_details(bili_client, favorite_model, connection).await?;
if *SCAN_ONLY {
warn!("已开启仅扫描模式,跳过视频下载...");
return Ok(());
// 从参数中获取视频列表的 Model 与视频流
let (video_list_model, video_streams) = video_list_from(args, path, bili_client, connection).await?;
// 从视频流中获取新视频的简要信息,写入数据库
refresh_video_list(&video_list_model, video_streams, connection).await?;
// 单独请求视频详情接口,获取视频的详情信息与所有的分页,写入数据库
fetch_video_details(bili_client, &video_list_model, connection).await?;
if ARGS.scan_only {
warn!("已开启仅扫描模式,跳过视频下载..");
} else {
// 从数据库中查找所有未下载的视频与分页,下载并处理
download_unprocessed_videos(bili_client, &video_list_model, connection).await?;
}
download_unprocessed_videos(bili_client, favorite_model, connection).await
Ok(())
}
/// 获取收藏夹 Model从收藏夹列表中获取所有新添加的视频,将其写入数据库
pub async fn refresh_favorite_list(
bili_client: &BiliClient,
fid: &str,
path: &Path,
/// 请求接口,获取视频列表中所有新添加的视频信息,将其写入数据库
pub async fn refresh_video_list<'a>(
video_list_model: &VideoListModelEnum,
video_streams: Pin<Box<dyn Stream<Item = Result<VideoInfo>> + 'a>>,
connection: &DatabaseConnection,
) -> Result<favorite::Model> {
let bili_favorite_list = FavoriteList::new(bili_client, fid.to_owned());
let favorite_list_info = bili_favorite_list.get_info().await?;
let favorite_model = handle_favorite_info(&favorite_list_info, path, connection).await?;
info!("开始扫描收藏夹: {} - {}...", favorite_model.f_id, favorite_model.name);
// 每十个视频一组,避免太多的数据库操作
let video_stream = bili_favorite_list.into_video_stream().chunks(10);
pin_mut!(video_stream);
let mut got_count = 0;
let total_count = total_video_count(&favorite_model, connection).await?;
while let Some(videos_info) = video_stream.next().await {
got_count += videos_info.len();
let exist_labels = exist_labels(&videos_info, &favorite_model, connection).await?;
// 如果发现有视频的收藏时间和 bvid 和数据库中重合,说明到达了上次处理到的地方,可以直接退出
let should_break = videos_info
.iter()
.any(|v| exist_labels.contains(&(v.bvid.clone(), v.fav_time.naive_utc())));
// 将视频信息写入数据库
create_videos(&videos_info, &favorite_model, connection).await?;
if should_break {
info!("到达上一次处理的位置,提前中止");
break;
}
) -> Result<()> {
video_list_model.log_refresh_video_start();
let latest_row_at = video_list_model.get_latest_row_at().and_utc();
let mut max_datetime = latest_row_at;
let mut error = Ok(());
let mut video_streams = video_streams
.take_while(|res| {
match res {
Err(e) => {
error = Err(anyhow!(e.to_string()));
futures::future::ready(false)
}
Ok(v) => {
// 虽然 video_streams 是从新到旧的,但由于此处是分页请求,极端情况下可能发生访问完第一页时插入了两整页视频的情况
// 此时获取到的第二页视频比第一页的还要新,因此为了确保正确,理应对每一页的第一个视频进行时间比较
// 但在 streams 的抽象下,无法判断具体是在哪里分页的,所以暂且对每个视频都进行比较,应该不会有太大性能损失
let release_datetime = v.release_datetime();
if release_datetime > &max_datetime {
max_datetime = *release_datetime;
}
futures::future::ready(release_datetime > &latest_row_at)
}
}
})
.filter_map(|res| futures::future::ready(res.ok()))
.chunks(10);
let mut count = 0;
while let Some(videos_info) = video_streams.next().await {
count += videos_info.len();
create_videos(videos_info, video_list_model, connection).await?;
}
let total_count = total_video_count(&favorite_model, connection).await? - total_count;
info!(
"扫描收藏夹: {} - {} 完成, 获取了 {} 条视频, 其中有 {} 条新视频",
favorite_model.f_id, favorite_model.name, got_count, total_count
);
Ok(favorite_model)
// 如果获取视频分页过程中发生了错误,直接在此处返回,不更新 latest_row_at
error?;
if max_datetime != latest_row_at {
video_list_model
.update_latest_row_at(max_datetime.naive_utc())
.save(connection)
.await?;
}
video_list_model.log_refresh_video_end(count);
Ok(())
}
/// 筛选出所有没有获取到分页信息和 tag 的视频,请求分页信息和 tag 并写入数据库
/// 筛选出所有获取到全部信息的视频,尝试补充其详细信息
pub async fn fetch_video_details(
bili_client: &BiliClient,
favorite_model: favorite::Model,
video_list_model: &VideoListModelEnum,
connection: &DatabaseConnection,
) -> Result<favorite::Model> {
info!(
"开始获取收藏夹 {} - {} 的视频与分页信息...",
favorite_model.f_id, favorite_model.name
);
let videos_model = filter_unfilled_videos(&favorite_model, connection).await?;
) -> Result<()> {
video_list_model.log_fetch_video_start();
let videos_model = filter_unfilled_videos(video_list_model.filter_expr(), connection).await?;
for video_model in videos_model {
let bili_video = Video::new(bili_client, video_model.bvid.clone());
let tags = match bili_video.get_tags().await {
Ok(tags) => tags,
let video = Video::new(bili_client, video_model.bvid.clone());
let info: Result<_> = async { Ok((video.get_tags().await?, video.get_view_info().await?)) }.await;
match info {
Err(e) => {
error!(
"获取视频 {} - {} 的标签失败,错误为:{}",
"获取视频 {} - {} 的详细信息失败,错误为:{}",
&video_model.bvid, &video_model.name, e
);
if let Some(BiliError::RequestFailed(code, _)) = e.downcast_ref::<BiliError>() {
if *code == -404 {
let mut video_active_model: video::ActiveModel = video_model.into();
video_active_model.valid = Set(false);
video_active_model.save(connection).await?;
}
if let Some(BiliError::RequestFailed(-404, _)) = e.downcast_ref::<BiliError>() {
let mut video_active_model: bili_sync_entity::video::ActiveModel = video_model.into();
video_active_model.valid = Set(false);
video_active_model.save(connection).await?;
}
continue;
}
Ok((tags, mut view_info)) => {
let VideoInfo::Detail { pages, .. } = &mut view_info else {
unreachable!()
};
let pages = std::mem::take(pages);
let pages_len = pages.len();
let txn = connection.begin().await?;
// 将分页信息写入数据库
create_pages(pages, &video_model, &txn).await?;
let mut video_active_model = view_info.into_detail_model(video_model);
video_list_model.set_relation_id(&mut video_active_model);
video_active_model.single_page = Set(Some(pages_len == 1));
video_active_model.tags = Set(Some(serde_json::to_value(tags)?));
video_active_model.save(&txn).await?;
txn.commit().await?;
}
};
let pages_info = match bili_video.get_pages().await {
Ok(pages) => pages,
Err(e) => {
error!(
"获取视频 {} - {} 的分页信息失败,错误为:{}",
&video_model.bvid, &video_model.name, e
);
if let Some(BiliError::RequestFailed(code, _)) = e.downcast_ref::<BiliError>() {
if *code == -404 {
let mut video_active_model: video::ActiveModel = video_model.into();
video_active_model.valid = Set(false);
video_active_model.save(connection).await?;
}
}
continue;
}
};
let txn = connection.begin().await?;
// 将分页信息写入数据库
create_video_pages(&pages_info, &video_model, &txn).await?;
// 将页标记和 tag 写入数据库
let mut video_active_model: video::ActiveModel = video_model.into();
video_active_model.single_page = Set(Some(pages_info.len() == 1));
video_active_model.tags = Set(Some(serde_json::to_value(tags).unwrap()));
video_active_model.save(&txn).await?;
txn.commit().await?;
}
info!(
"获取收藏夹 {} - {} 的视频与分页信息完成",
favorite_model.f_id, favorite_model.name
);
Ok(favorite_model)
video_list_model.log_fetch_video_end();
Ok(())
}
/// 下载所有未处理成功的视频
pub async fn download_unprocessed_videos(
bili_client: &BiliClient,
favorite_model: favorite::Model,
video_list_model: &VideoListModelEnum,
connection: &DatabaseConnection,
) -> Result<()> {
info!(
"开始下载收藏夹: {} - {} 中所有未处理过的视频...",
favorite_model.f_id, favorite_model.name
);
let unhandled_videos_pages = unhandled_videos_pages(&favorite_model, connection).await?;
// 对于视频,允许三个同时下载(视频内还有分页、不同分页还有多种下载任务)
let semaphore = Semaphore::new(3);
video_list_model.log_download_video_start();
let semaphore = Semaphore::new(CONFIG.concurrent_limit.video);
let downloader = Downloader::new(bili_client.client.clone());
let mut uppers_mutex: HashMap<i64, (Mutex<()>, Mutex<()>)> = HashMap::new();
let unhandled_videos_pages = filter_unhandled_video_pages(video_list_model.filter_expr(), connection).await?;
for (video_model, _) in &unhandled_videos_pages {
uppers_mutex.insert(video_model.upper_id, (Mutex::new(()), Mutex::new(())));
uppers_mutex
.entry(video_model.upper_id)
.or_insert_with(|| (Mutex::new(()), Mutex::new(())));
}
let mut tasks = unhandled_videos_pages
let tasks = unhandled_videos_pages
.into_iter()
.map(|(video_model, pages_model)| {
let upper_mutex = uppers_mutex.get(&video_model.upper_id).unwrap();
let upper_mutex = uppers_mutex.get(&video_model.upper_id).expect("upper mutex not found");
download_video_pages(
bili_client,
video_list_model,
video_model,
pages_model,
connection,
&semaphore,
&downloader,
&CONFIG.upper_path,
upper_mutex,
)
})
.collect::<FuturesUnordered<_>>();
let mut models = Vec::with_capacity(10);
while let Some(res) = tasks.next().await {
match res {
Ok(model) => {
models.push(model);
let mut download_aborted = false;
let mut stream = tasks
// 触发风控时设置 download_aborted 标记并终止流
.take_while(|res| {
if res
.as_ref()
.is_err_and(|e| e.downcast_ref::<DownloadAbortError>().is_some())
{
download_aborted = true;
}
Err(e) => {
if e.downcast_ref::<DownloadAbortError>().is_some() {
error!("下载视频时触发风控,将终止收藏夹下所有下载任务,等待下一轮执行");
break;
}
}
}
// 满十个就写入数据库
if models.len() == 10 {
update_videos_model(std::mem::replace(&mut models, Vec::with_capacity(10)), connection).await?;
}
}
if !models.is_empty() {
futures::future::ready(!download_aborted)
})
// 过滤掉没有触发风控的普通 Err只保留正确返回的 Model
.filter_map(|res| futures::future::ready(res.ok()))
// 将成功返回的 Model 按十个一组合并
.chunks(10);
while let Some(models) = stream.next().await {
update_videos_model(models, connection).await?;
}
info!(
"下载收藏夹: {} - {} 中未处理过的视频完成",
favorite_model.f_id, favorite_model.name
);
if download_aborted {
error!("下载触发风控,已终止所有任务,等待下一轮执行");
}
video_list_model.log_download_video_end();
Ok(())
}
/// 暂时这样做,后面提取成上下文
#[allow(clippy::too_many_arguments)]
pub async fn download_video_pages(
bili_client: &BiliClient,
video_list_model: &VideoListModelEnum,
video_model: video::Model,
pages: Vec<page::Model>,
connection: &DatabaseConnection,
semaphore: &Semaphore,
downloader: &Downloader,
upper_path: &Path,
upper_mutex: &(Mutex<()>, Mutex<()>),
) -> Result<video::ActiveModel> {
let permit = semaphore.acquire().await;
if let Err(e) = permit {
bail!(e);
}
let _permit = semaphore.acquire().await.context("acquire semaphore failed")?;
let mut status = VideoStatus::new(video_model.download_status);
let seprate_status = status.should_run();
let base_path = Path::new(&video_model.path);
let base_path = video_list_model
.path()
.join(TEMPLATE.path_safe_render("video", &video_format_args(&video_model))?);
let upper_id = video_model.upper_id.to_string();
let base_upper_path = upper_path
.join(upper_id.chars().next().unwrap().to_string())
let base_upper_path = &CONFIG
.upper_path
.join(upper_id.chars().next().context("upper_id is empty")?.to_string())
.join(upper_id);
let is_single_page = video_model.single_page.unwrap();
let is_single_page = video_model.single_page.context("single_page is null")?;
// 对于单页视频page 的下载已经足够
// 对于多页视频page 下载仅包含了分集内容,需要额外补上视频的 poster 的 tvshow.nfo
let tasks: Vec<Pin<Box<dyn Future<Output = Result<()>>>>> = vec![
@@ -272,6 +261,7 @@ pub async fn download_video_pages(
pages,
connection,
downloader,
&base_path,
)),
];
let tasks: FuturesOrdered<_> = tasks.into_iter().collect();
@@ -280,27 +270,23 @@ pub async fn download_video_pages(
results
.iter()
.take(4)
.zip(["封面", "视频 nfo", "up 主头像", "up 主 nfo"])
.zip(["封面", "详情", "作者头像", "作者详情"])
.for_each(|(res, task_name)| match res {
Ok(_) => info!(
"处理视频 {} - {} 的 {} 成功",
&video_model.bvid, &video_model.name, task_name
),
Err(e) => error!(
"处理视频 {} - {} 的 {} 失败: {}",
&video_model.bvid, &video_model.name, task_name, e
),
Ok(_) => info!("处理视频「{}」{}成功", &video_model.name, task_name),
Err(e) => error!("处理视频{}」{}失败: {}", &video_model.name, task_name, e),
});
if let Err(e) = results.into_iter().nth(4).unwrap() {
if let Err(e) = results.into_iter().nth(4).context("page download result not found")? {
if e.downcast_ref::<DownloadAbortError>().is_some() {
return Err(e);
}
}
let mut video_active_model: video::ActiveModel = video_model.into();
video_active_model.download_status = Set(status.into());
video_active_model.path = Set(base_path.to_string_lossy().to_string());
Ok(video_active_model)
}
/// 分发并执行分页下载任务,当且仅当所有分页成功下载或达到最大重试次数时返回 Ok否则根据失败原因返回对应的错误
pub async fn dispatch_download_page(
should_run: bool,
bili_client: &BiliClient,
@@ -308,89 +294,81 @@ pub async fn dispatch_download_page(
pages: Vec<page::Model>,
connection: &DatabaseConnection,
downloader: &Downloader,
base_path: &Path,
) -> Result<()> {
if !should_run {
return Ok(());
}
// 对于视频的分页,允许两个同时下载(绝大部分是单页视频)
let child_semaphore = Semaphore::new(2);
let mut tasks = pages
let child_semaphore = Semaphore::new(CONFIG.concurrent_limit.page);
let tasks = pages
.into_iter()
.map(|page_model| download_page(bili_client, video_model, page_model, &child_semaphore, downloader))
.map(|page_model| {
download_page(
bili_client,
video_model,
page_model,
&child_semaphore,
downloader,
base_path,
)
})
.collect::<FuturesUnordered<_>>();
let mut models = Vec::with_capacity(10);
let (mut should_error, mut is_break) = (false, false);
while let Some(res) = tasks.next().await {
match res {
Ok(model) => {
if let Set(status) = model.download_status {
let status = PageStatus::new(status);
if status.should_run().iter().any(|v| *v) {
// 有一个分页没变成终止状态(即下载成功或者重试次数达到限制),就应该向上层传递 Error
should_error = true;
let (mut download_aborted, mut error_occurred) = (false, false);
let mut stream = tasks
.take_while(|res| {
match res {
Ok(model) => {
// 当前函数返回的是所有分页的下载状态,只要有任何一个分页返回新的下载状态标识位是 false当前函数就应该认为是失败的
if model
.download_status
.try_as_ref()
.is_none_or(|status| !PageStatus::new(*status).get_completed())
{
error_occurred = true;
}
}
models.push(model);
}
Err(e) => {
if e.downcast_ref::<DownloadAbortError>().is_some() {
should_error = true;
is_break = true;
break;
Err(e) => {
if e.downcast_ref::<DownloadAbortError>().is_some() {
download_aborted = true;
}
}
}
}
if models.len() == 10 {
update_pages_model(std::mem::replace(&mut models, Vec::with_capacity(10)), connection).await?;
}
}
if !models.is_empty() {
// 仅在发生风控时终止流,其它情况继续执行
futures::future::ready(!download_aborted)
})
.filter_map(|res| futures::future::ready(res.ok()))
.chunks(10);
while let Some(models) = stream.next().await {
update_pages_model(models, connection).await?;
}
if should_error {
if is_break {
error!(
"下载视频 {} - {} 的分页时触发风控,将异常向上传递...",
&video_model.bvid, &video_model.name
);
bail!(DownloadAbortError());
} else {
error!(
"下载视频 {} - {} 的分页时出现了错误,将在下一轮尝试重新处理",
&video_model.bvid, &video_model.name
);
bail!(ProcessPageError());
}
if download_aborted {
error!("下载视频「{}」的分页时触发风控,将异常向上传递..", &video_model.name);
bail!(DownloadAbortError());
}
if error_occurred {
error!(
"下载视频「{}」的分页时出现错误,将在下一轮尝试重新处理",
&video_model.name
);
bail!(ProcessPageError());
}
Ok(())
}
/// 下载某个分页,未发生风控且正常运行时返回 Ok(Page::ActiveModel),其中 status 字段存储了新的下载状态,发生风控时返回 DownloadAbortError
pub async fn download_page(
bili_client: &BiliClient,
video_model: &video::Model,
page_model: page::Model,
semaphore: &Semaphore,
downloader: &Downloader,
base_path: &Path,
) -> Result<page::ActiveModel> {
let permit = semaphore.acquire().await;
if let Err(e) = permit {
return Err(e.into());
}
let _permit = semaphore.acquire().await.context("acquire semaphore failed")?;
let mut status = PageStatus::new(page_model.download_status);
let seprate_status = status.should_run();
let is_single_page = video_model.single_page.unwrap();
let base_path = Path::new(&video_model.path);
let base_name = filenamify(TEMPLATE.render(
"page",
&json!({
"bvid": &video_model.bvid,
"title": &video_model.name,
"upper_name": &video_model.upper_name,
"upper_mid": &video_model.upper_id,
"ptitle": &page_model.name,
"pid": page_model.pid,
}),
)?);
let is_single_page = video_model.single_page.context("single_page is null")?;
let base_name = TEMPLATE.path_safe_render("page", &page_format_args(video_model, &page_model))?;
let (poster_path, video_path, nfo_path, danmaku_path, fanart_path) = if is_single_page {
(
base_path.join(format!("{}-poster.jpg", &base_name)),
@@ -417,14 +395,13 @@ pub async fn download_page(
None,
)
};
let dimension = if page_model.width.is_some() && page_model.height.is_some() {
Some(Dimension {
width: page_model.width.unwrap(),
height: page_model.height.unwrap(),
let dimension = match (page_model.width, page_model.height) {
(Some(width), Some(height)) => Some(Dimension {
width,
height,
rotate: 0,
})
} else {
None
}),
_ => None,
};
let page_info = PageInfo {
cid: page_model.cid,
@@ -447,7 +424,7 @@ pub async fn download_page(
video_model,
downloader,
&page_info,
video_path.clone(),
&video_path,
)),
Box::pin(generate_page_nfo(seprate_status[2], video_model, &page_model, nfo_path)),
Box::pin(fetch_page_danmaku(
@@ -463,26 +440,26 @@ pub async fn download_page(
status.update_status(&results);
results
.iter()
.zip(["封面", "视频", "视频 nfo", "弹幕"])
.zip(["封面", "视频", "详情", "弹幕"])
.for_each(|(res, task_name)| match res {
Ok(_) => info!(
"处理视频 {} - {} 第 {} 页{} 成功",
&video_model.bvid, &video_model.name, page_model.pid, task_name
"处理视频{}第 {} 页{}成功",
&video_model.name, page_model.pid, task_name
),
Err(e) => error!(
"处理视频 {} - {} 第 {} 页{} 失败: {}",
&video_model.bvid, &video_model.name, page_model.pid, task_name, e
"处理视频{}第 {} 页{}失败: {}",
&video_model.name, page_model.pid, task_name, e
),
});
// 查看下载视频的状态,该状态会影响上层是否 break
if let Err(e) = results.into_iter().nth(1).unwrap() {
// 如果下载视频时触发风控,直接返回 DownloadAbortError
if let Err(e) = results.into_iter().nth(1).context("video download result not found")? {
if let Ok(BiliError::RiskControlOccurred) = e.downcast::<BiliError>() {
bail!(DownloadAbortError());
}
}
let mut page_active_model: page::ActiveModel = page_model.into();
page_active_model.download_status = Set(status.into());
page_active_model.path = Set(Some(video_path.to_str().unwrap().to_string()));
page_active_model.path = Set(Some(video_path.to_string_lossy().to_string()));
Ok(page_active_model)
}
@@ -497,7 +474,7 @@ pub async fn fetch_page_poster(
if !should_run {
return Ok(());
}
let single_page = video_model.single_page.unwrap();
let single_page = video_model.single_page.context("single_page is null")?;
let url = if single_page {
// 单页视频直接用视频的封面
video_model.cover.as_str()
@@ -521,7 +498,7 @@ pub async fn fetch_page_video(
video_model: &video::Model,
downloader: &Downloader,
page_info: &PageInfo,
page_path: PathBuf,
page_path: &Path,
) -> Result<()> {
if !should_run {
return Ok(());
@@ -532,15 +509,11 @@ pub async fn fetch_page_video(
.await?
.best_stream(&CONFIG.filter_option)?;
match streams {
BestStream::Mixed(mix_stream) => {
downloader.fetch(mix_stream.url(), &page_path).await?;
}
BestStream::Mixed(mix_stream) => downloader.fetch(mix_stream.url(), page_path).await,
BestStream::VideoAudio {
video: video_stream,
audio: None,
} => {
downloader.fetch(video_stream.url(), &page_path).await?;
}
} => downloader.fetch(video_stream.url(), page_path).await,
BestStream::VideoAudio {
video: video_stream,
audio: Some(audio_stream),
@@ -549,12 +522,17 @@ pub async fn fetch_page_video(
page_path.with_extension("tmp_video"),
page_path.with_extension("tmp_audio"),
);
downloader.fetch(video_stream.url(), &tmp_video_path).await?;
downloader.fetch(audio_stream.url(), &tmp_audio_path).await?;
downloader.merge(&tmp_video_path, &tmp_audio_path, &page_path).await?;
let res = async {
downloader.fetch(video_stream.url(), &tmp_video_path).await?;
downloader.fetch(audio_stream.url(), &tmp_audio_path).await?;
downloader.merge(&tmp_video_path, &tmp_audio_path, page_path).await
}
.await;
let _ = fs::remove_file(tmp_video_path).await;
let _ = fs::remove_file(tmp_audio_path).await;
res
}
}
Ok(())
}
pub async fn fetch_page_danmaku(
@@ -572,8 +550,7 @@ pub async fn fetch_page_danmaku(
.get_danmaku_writer(page_info)
.await?
.write(danmaku_path)
.await?;
Ok(())
.await
}
pub async fn generate_page_nfo(
@@ -585,7 +562,7 @@ pub async fn generate_page_nfo(
if !should_run {
return Ok(());
}
let single_page = video_model.single_page.unwrap();
let single_page = video_model.single_page.context("single_page is null")?;
let nfo_serializer = if single_page {
NFOSerializer(ModelWrapper::Video(video_model), NFOMode::MOVIE)
} else {
@@ -657,13 +634,18 @@ async fn generate_nfo(serializer: NFOSerializer<'_>, nfo_path: PathBuf) -> Resul
if let Some(parent) = nfo_path.parent() {
fs::create_dir_all(parent).await?;
}
fs::write(nfo_path, serializer.generate_nfo().await?.as_bytes()).await?;
fs::write(
nfo_path,
serializer.generate_nfo(&CONFIG.nfo_time_type).await?.as_bytes(),
)
.await?;
Ok(())
}
#[cfg(test)]
mod tests {
use handlebars::handlebars_helper;
use serde_json::json;
use super::*;
@@ -678,15 +660,49 @@ mod tests {
}
});
template.register_helper("truncate", Box::new(truncate));
let _ = template.register_template_string("video", "test{{bvid}}test");
let _ = template.register_template_string("test_truncate", "哈哈,{{ truncate title 30 }}");
let _ = template.path_safe_register("video", "test{{bvid}}test");
let _ = template.path_safe_register("test_truncate", "哈哈,{{ truncate title 30 }}");
let _ = template.path_safe_register("test_path_unix", "{{ truncate title 7 }}/test/a");
let _ = template.path_safe_register("test_path_windows", r"{{ truncate title 7 }}\\test\\a");
#[cfg(not(windows))]
{
assert_eq!(
template
.path_safe_render("test_path_unix", &json!({"title": "关注/永雏塔菲喵"}))
.unwrap(),
"关注_永雏塔菲/test/a"
);
assert_eq!(
template
.path_safe_render("test_path_windows", &json!({"title": "关注/永雏塔菲喵"}))
.unwrap(),
"关注_永雏塔菲_test_a"
);
}
#[cfg(windows)]
{
assert_eq!(
template
.path_safe_render("test_path_unix", &json!({"title": "关注/永雏塔菲喵"}))
.unwrap(),
"关注_永雏塔菲_test_a"
);
assert_eq!(
template
.path_safe_render("test_path_windows", &json!({"title": "关注/永雏塔菲喵"}))
.unwrap(),
r"关注_永雏塔菲\\test\\a"
);
}
assert_eq!(
template.render("video", &json!({"bvid": "BV1b5411h7g7"})).unwrap(),
template
.path_safe_render("video", &json!({"bvid": "BV1b5411h7g7"}))
.unwrap(),
"testBV1b5411h7g7test"
);
assert_eq!(
template
.render(
.path_safe_render(
"test_truncate",
&json!({"title": "你说得对,但是 Rust 是由 Mozilla 自主研发的一款全新的编译期格斗游戏。\
Cargo\

View File

@@ -0,0 +1,9 @@
[package]
name = "bili_sync_entity"
version = { workspace = true }
edition = { workspace = true }
publish = { workspace = true }
[dependencies]
sea-orm = { workspace = true }
serde_json = { workspace = true }

View File

@@ -0,0 +1,22 @@
//! `SeaORM` Entity. Generated by sea-orm-codegen 0.12.15
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq)]
#[sea_orm(table_name = "collection")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub s_id: i64,
pub m_id: i64,
pub name: String,
pub r#type: i32,
pub path: String,
pub created_at: String,
pub latest_row_at: DateTime,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}

View File

@@ -12,6 +12,7 @@ pub struct Model {
pub name: String,
pub path: String,
pub created_at: String,
pub latest_row_at: DateTime,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]

View File

@@ -2,6 +2,9 @@
pub mod prelude;
pub mod collection;
pub mod favorite;
pub mod page;
pub mod submission;
pub mod video;
pub mod watch_later;

View File

@@ -8,7 +8,7 @@ pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub video_id: i32,
pub cid: i32,
pub cid: i64,
pub pid: i32,
pub name: String,
pub width: Option<u32>,

View File

@@ -0,0 +1,20 @@
//! `SeaORM` Entity. Generated by sea-orm-codegen 0.12.15
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq)]
#[sea_orm(table_name = "submission")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub upper_id: i64,
pub upper_name: String,
pub path: String,
pub created_at: String,
pub latest_row_at: DateTime,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}

View File

@@ -7,7 +7,10 @@ use sea_orm::entity::prelude::*;
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub favorite_id: i32,
pub collection_id: Option<i32>,
pub favorite_id: Option<i32>,
pub watch_later_id: Option<i32>,
pub submission_id: Option<i32>,
pub upper_id: i64,
pub upper_name: String,
pub upper_face: String,

View File

@@ -0,0 +1,18 @@
//! `SeaORM` Entity. Generated by sea-orm-codegen 0.12.15
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, Eq)]
#[sea_orm(table_name = "watch_later")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub path: String,
pub created_at: String,
pub latest_row_at: DateTime,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}

View File

@@ -0,0 +1,9 @@
[package]
name = "bili_sync_migration"
version = { workspace = true }
edition = { workspace = true }
publish = { workspace = true }
[dependencies]
async-std = { workspace = true }
sea-orm-migration = { workspace = true }

View File

@@ -0,0 +1,22 @@
pub use sea_orm_migration::prelude::*;
mod m20240322_000001_create_table;
mod m20240505_130850_add_collection;
mod m20240709_130914_watch_later;
mod m20240724_161008_submission;
mod m20250122_062926_add_latest_row_at;
pub struct Migrator;
#[async_trait::async_trait]
impl MigratorTrait for Migrator {
fn migrations() -> Vec<Box<dyn MigrationTrait>> {
vec![
Box::new(m20240322_000001_create_table::Migration),
Box::new(m20240505_130850_add_collection::Migration),
Box::new(m20240709_130914_watch_later::Migration),
Box::new(m20240724_161008_submission::Migration),
Box::new(m20250122_062926_add_latest_row_at::Migration),
]
}
}

View File

@@ -0,0 +1,179 @@
use sea_orm_migration::prelude::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.create_table(
Table::create()
.table(Collection::Table)
.if_not_exists()
.col(
ColumnDef::new(Collection::Id)
.unsigned()
.not_null()
.auto_increment()
.primary_key(),
)
.col(ColumnDef::new(Collection::SId).unsigned().not_null())
.col(ColumnDef::new(Collection::MId).unsigned().not_null())
.col(ColumnDef::new(Collection::Name).string().not_null())
.col(ColumnDef::new(Collection::Type).small_unsigned().not_null())
.col(ColumnDef::new(Collection::Path).string().not_null())
.col(
ColumnDef::new(Collection::CreatedAt)
.timestamp()
.default(Expr::current_timestamp())
.not_null(),
)
.to_owned(),
)
.await?;
manager
.create_index(
Index::create()
.table(Collection::Table)
.name("idx_collection_sid_mid_type")
.col(Collection::SId)
.col(Collection::MId)
.col(Collection::Type)
.unique()
.to_owned(),
)
.await?;
manager
.drop_index(
Index::drop()
.table(Video::Table)
.name("idx_video_favorite_id_bvid")
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.add_column(ColumnDef::new(Video::CollectionId).unsigned().null())
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.add_column(ColumnDef::new(Video::TempFavoriteId).unsigned().null())
.to_owned(),
)
.await?;
db.execute_unprepared("UPDATE video SET temp_favorite_id = favorite_id")
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.drop_column(Video::FavoriteId)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.rename_column(Video::TempFavoriteId, Video::FavoriteId)
.to_owned(),
)
.await?;
// 在唯一索引中NULL 不等于 NULL所以需要使用 ifnull 函数排除空的情况
db.execute_unprepared("CREATE UNIQUE INDEX `idx_video_cid_fid_bvid` ON `video` (ifnull(`collection_id`, -1), ifnull(`favorite_id`, -1), `bvid`)")
.await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.drop_index(
Index::drop()
.table(Video::Table)
.name("idx_video_cid_fid_bvid")
.to_owned(),
)
.await?;
db.execute_unprepared("DELETE FROM video WHERE favorite_id IS NULL")
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
// 向存在记录的表中添加非空列时,必须提供默认值
.add_column(ColumnDef::new(Video::TempFavoriteId).unsigned().not_null().default(0))
.to_owned(),
)
.await?;
db.execute_unprepared("UPDATE video SET temp_favorite_id = favorite_id")
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.drop_column(Video::FavoriteId)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.rename_column(Video::TempFavoriteId, Video::FavoriteId)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.drop_column(Video::CollectionId)
.to_owned(),
)
.await?;
manager
.create_index(
Index::create()
.table(Video::Table)
.name("idx_video_favorite_id_bvid")
.col(Video::FavoriteId)
.col(Video::Bvid)
.unique()
.to_owned(),
)
.await?;
manager
.drop_table(Table::drop().table(Collection::Table).to_owned())
.await
}
}
#[derive(DeriveIden)]
enum Collection {
Table,
Id,
SId,
MId,
Name,
Type,
Path,
CreatedAt,
}
#[derive(DeriveIden)]
enum Video {
Table,
FavoriteId,
TempFavoriteId,
CollectionId,
Bvid,
}

View File

@@ -0,0 +1,88 @@
use sea_orm_migration::prelude::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.create_table(
Table::create()
.table(WatchLater::Table)
.if_not_exists()
.col(
ColumnDef::new(WatchLater::Id)
.unsigned()
.not_null()
.auto_increment()
.primary_key(),
)
.col(ColumnDef::new(WatchLater::Path).string().not_null())
.col(
ColumnDef::new(WatchLater::CreatedAt)
.timestamp()
.default(Expr::current_timestamp())
.not_null(),
)
.to_owned(),
)
.await?;
manager
.drop_index(
Index::drop()
.table(Video::Table)
.name("idx_video_cid_fid_bvid")
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.add_column(ColumnDef::new(Video::WatchLaterId).unsigned().null())
.to_owned(),
)
.await?;
db.execute_unprepared("CREATE UNIQUE INDEX `idx_video_unique` ON `video` (ifnull(`collection_id`, -1), ifnull(`favorite_id`, -1), ifnull(`watch_later_id`, -1), `bvid`)")
.await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.drop_index(Index::drop().table(Video::Table).name("idx_video_unique").to_owned())
.await?;
db.execute_unprepared("DELETE FROM video WHERE watch_later_id IS NOT NULL")
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.drop_column(Video::WatchLaterId)
.to_owned(),
)
.await?;
db.execute_unprepared("CREATE UNIQUE INDEX `idx_video_cid_fid_bvid` ON `video` (ifnull(`collection_id`, -1), ifnull(`favorite_id`, -1), `bvid`)")
.await?;
manager
.drop_table(Table::drop().table(WatchLater::Table).to_owned())
.await
}
}
#[derive(DeriveIden)]
enum WatchLater {
Table,
Id,
Path,
CreatedAt,
}
#[derive(DeriveIden)]
enum Video {
Table,
WatchLaterId,
}

View File

@@ -0,0 +1,87 @@
use sea_orm_migration::prelude::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.create_table(
Table::create()
.table(Submission::Table)
.if_not_exists()
.col(
ColumnDef::new(Submission::Id)
.unsigned()
.not_null()
.auto_increment()
.primary_key(),
)
.col(ColumnDef::new(Submission::UpperId).unique_key().unsigned().not_null())
.col(ColumnDef::new(Submission::UpperName).string().not_null())
.col(ColumnDef::new(Submission::Path).string().not_null())
.col(
ColumnDef::new(Submission::CreatedAt)
.timestamp()
.default(Expr::current_timestamp())
.not_null(),
)
.to_owned(),
)
.await?;
manager
.drop_index(Index::drop().table(Video::Table).name("idx_video_unique").to_owned())
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.add_column(ColumnDef::new(Video::SubmissionId).unsigned().null())
.to_owned(),
)
.await?;
db.execute_unprepared("CREATE UNIQUE INDEX `idx_video_unique` ON `video` (ifnull(`collection_id`, -1), ifnull(`favorite_id`, -1), ifnull(`watch_later_id`, -1), ifnull(`submission_id`, -1), `bvid`)")
.await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
let db = manager.get_connection();
manager
.drop_index(Index::drop().table(Video::Table).name("idx_video_unique").to_owned())
.await?;
db.execute_unprepared("DELETE FROM video WHERE submission_id IS NOT NULL")
.await?;
manager
.alter_table(
Table::alter()
.table(Video::Table)
.drop_column(Video::SubmissionId)
.to_owned(),
)
.await?;
db.execute_unprepared("CREATE UNIQUE INDEX `idx_video_unique` ON `video` (ifnull(`collection_id`, -1), ifnull(`favorite_id`, -1), ifnull(`watch_later_id`, -1), `bvid`)")
.await?;
manager
.drop_table(Table::drop().table(Submission::Table).to_owned())
.await
}
}
#[derive(DeriveIden)]
enum Submission {
Table,
Id,
UpperId,
UpperName,
Path,
CreatedAt,
}
#[derive(DeriveIden)]
enum Video {
Table,
SubmissionId,
}

View File

@@ -0,0 +1,122 @@
use sea_orm_migration::prelude::*;
use sea_orm_migration::schema::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
// 为四张 video list 表添加 latest_row_at 字段,表示该列表处理到的最新时间
manager
.alter_table(
Table::alter()
.table(Favorite::Table)
.add_column(timestamp(Favorite::LatestRowAt).default("1970-01-01 00:00:00"))
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Collection::Table)
.add_column(timestamp(Collection::LatestRowAt).default("1970-01-01 00:00:00"))
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(WatchLater::Table)
.add_column(timestamp(WatchLater::LatestRowAt).default("1970-01-01 00:00:00"))
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Submission::Table)
.add_column(timestamp(Submission::LatestRowAt).default("1970-01-01 00:00:00"))
.to_owned(),
)
.await?;
// 手动写 SQL 更新这四张表的 latest 字段到当前取值
let db = manager.get_connection();
db.execute_unprepared(
"UPDATE favorite SET latest_row_at = (SELECT IFNULL(MAX(favtime), '1970-01-01 00:00:00') FROM video WHERE favorite_id = favorite.id)",
)
.await?;
db.execute_unprepared(
"UPDATE collection SET latest_row_at = (SELECT IFNULL(MAX(pubtime), '1970-01-01 00:00:00') FROM video WHERE collection_id = collection.id)",
)
.await?;
db.execute_unprepared(
"UPDATE watch_later SET latest_row_at = (SELECT IFNULL(MAX(favtime), '1970-01-01 00:00:00') FROM video WHERE watch_later_id = watch_later.id)",
)
.await?;
db.execute_unprepared(
"UPDATE submission SET latest_row_at = (SELECT IFNULL(MAX(ctime), '1970-01-01 00:00:00') FROM video WHERE submission_id = submission.id)",
)
.await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager
.alter_table(
Table::alter()
.table(Favorite::Table)
.drop_column(Favorite::LatestRowAt)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Collection::Table)
.drop_column(Collection::LatestRowAt)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(WatchLater::Table)
.drop_column(WatchLater::LatestRowAt)
.to_owned(),
)
.await?;
manager
.alter_table(
Table::alter()
.table(Submission::Table)
.drop_column(Submission::LatestRowAt)
.to_owned(),
)
.await
}
}
#[derive(DeriveIden)]
enum Favorite {
Table,
LatestRowAt,
}
#[derive(DeriveIden)]
enum Collection {
Table,
LatestRowAt,
}
#[derive(DeriveIden)]
enum WatchLater {
Table,
LatestRowAt,
}
#[derive(DeriveIden)]
enum Submission {
Table,
LatestRowAt,
}

View File

@@ -2,5 +2,5 @@ use sea_orm_migration::prelude::*;
#[async_std::main]
async fn main() {
cli::run_cli(migration::Migrator).await;
cli::run_cli(bili_sync_migration::Migrator).await;
}

102
docs/.vitepress/config.mts Normal file
View File

@@ -0,0 +1,102 @@
import { defineConfig } from "vitepress";
import taskLists from "markdown-it-task-lists";
// https://vitepress.dev/reference/site-config
export default defineConfig({
title: "bili-sync",
description: "由 Rust & Tokio 驱动的哔哩哔哩同步工具",
lang: "zh-Hans",
sitemap: {
hostname: "https://bili-sync.github.io",
},
lastUpdated: true,
cleanUrls: true,
metaChunk: true,
themeConfig: {
outline: {
label: "页面导航",
level: "deep",
},
// https://vitepress.dev/reference/default-theme-config
nav: [
{ text: "主页", link: "/" },
{
text: "v2.3.0",
items: [
{
text: "程序更新",
link: "https://github.com/amtoaer/bili-sync/releases",
},
{
text: "文档更新",
link: "https://github.com/search?q=repo:amtoaer/bili-sync+docs&type=commits",
},
],
},
],
sidebar: [
{
text: "简介",
items: [
{ text: "什么是 bili-sync", link: "/introduction" },
{ text: "快速开始", link: "/quick-start" },
],
},
{
text: "细节",
items: [
{ text: "配置文件", link: "/configuration" },
{ text: "命令行参数", link: "/args" },
{ text: "工作原理", link: "/design" },
],
},
{
text: "参考",
items: [
{ text: "获取收藏夹信息", link: "/favorite" },
{
text: "获取视频合集/视频列表信息",
link: "/collection",
},
{ text: "获取投稿信息", link: "/submission" },
],
},
],
socialLinks: [
{ icon: "github", link: "https://github.com/amtoaer/bili-sync" },
],
search: {
provider: "local",
},
notFound: {
title: "你来到了没有知识的荒原",
quote: "这里什么都没有",
linkText: "返回首页",
},
docFooter: {
prev: "上一页",
next: "下一页",
},
lastUpdated: {
text: "上次更新于",
},
returnToTopLabel: "回到顶部",
sidebarMenuLabel: "菜单",
darkModeSwitchLabel: "主题",
lightModeSwitchTitle: "切换到浅色模式",
darkModeSwitchTitle: "切换到深色模式",
},
markdown: {
config: (md) => {
md.use(taskLists);
},
theme: {
light: "github-light",
dark: "github-dark",
},
},
head: [
["link", { rel: "icon", type: "image/svg+xml", href: "/icon.svg" }],
["link", { rel: "icon", type: "image/png", href: "/icon.png" }],
],
});

View File

@@ -0,0 +1,7 @@
.medium-zoom-overlay {
z-index: 30;
}
.medium-zoom-image--opened {
z-index: 31;
}

View File

@@ -0,0 +1,22 @@
import DefaultTheme from "vitepress/theme";
import { onMounted, watch, nextTick } from "vue";
import { useRoute } from "vitepress";
import mediumZoom from "medium-zoom";
import "./index.css";
export default {
...DefaultTheme,
setup() {
const route = useRoute();
const initZoom = () => {
mediumZoom(".main img", { background: "var(--vp-c-bg)" });
};
onMounted(() => {
initZoom();
});
watch(
() => route.path,
() => nextTick(() => initZoom()),
);
},
};

27
docs/args.md Normal file
View File

@@ -0,0 +1,27 @@
# 命令行参数
程序支持有限的命令行参数,可以通过执行 `bili-sync-rs --help` 查看说明。
```shell
bili-sync/target/debug main* ⇡
./bili-sync-rs --help
由 Rust & Tokio 驱动的哔哩哔哩同步工具
Usage: bili-sync-rs [OPTIONS]
Options:
-s, --scan-only [env: SCAN_ONLY=]
-l, --log-level <LOG_LEVEL> [env: RUST_LOG=] [default: None,bili_sync=info]
-h, --help Print help
-V, --version Print version
```
可以看到除版本和帮助信息外,程序仅支持两个参数,参数除可以通过命令行设置外,还可通过环境变量设置。
## `--scan-only`
`--scan-only` 参数用于仅扫描列表,而不实际执行下载操作。该参数的主要目的是[方便用户从 v1 迁移](https://github.com/amtoaer/bili-sync/issues/66#issuecomment-2066642481),新用户不需要关注。
## `--log-level`
`--log-level` 参数用于设置日志级别,一般可以维持默认。该参数与 Rust 程序中 `RUST_LOG` 的语义相同,可以查看[相关文档](https://docs.rs/env_logger/latest/env_logger/#enabling-logging)获取详细信息。

Binary file not shown.

After

Width:  |  Height:  |  Size: 33 KiB

BIN
docs/assets/bili_video.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

BIN
docs/assets/detail.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 342 KiB

BIN
docs/assets/dir.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 130 KiB

BIN
docs/assets/favorite.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 258 KiB

BIN
docs/assets/multi_page.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 152 KiB

BIN
docs/assets/overview.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 270 KiB

BIN
docs/assets/play.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 216 KiB

BIN
docs/assets/season.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 289 KiB

BIN
docs/assets/series.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 296 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 174 KiB

BIN
docs/assets/submission.webp Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 279 KiB

BIN
docs/bun.lockb Executable file

Binary file not shown.

28
docs/collection.md Normal file
View File

@@ -0,0 +1,28 @@
# 获取视频合集/视频列表信息
视频合集和视频列表虽然在哔哩哔哩网站交互上行为类似,但在接口层级是两个不同的概念,程序配置中需要对两者做出区分。
## 配置形式与区分方法
在 bili-sync 的设计中,视频合集的 key 为 `season:{mid}:{season_id}`,而视频列表的 key 为 `series:{mid}:{series_id}`
新版本 b 站网页端已经对两种类型做了初步整合,将需要的参数展示在了视频合集/视频列表的 URL 中不再需要手动查看接口。URL 的路径格式为:
```
/{mid}/lists/{id}?type={season/series}
```
点开你想要订阅的视频合集/视频列表详情,查看 URL 即可拼接出对应的 key。
### 视频合集
![image](./assets/season.webp)
该视频合集的 key 为 `season:521722088:1987140`
### 视频列表
![image](./assets/series.webp)
该视频列表的 key 为 `series:521722088:387214`

238
docs/configuration.md Normal file
View File

@@ -0,0 +1,238 @@
# 配置文件
默认的配置文件已经在[快速开始](/quick-start)中给出,该文档对配置文件的各个参数依次详细解释。
## video_name 与 page_name
`video_name``page_name` 用于设置下载文件的命名规则,对于所有下载的内容,将会维持如下的目录结构:
1. 单页视频:
```bash
├── {video_name}
│   ├── {page_name}.mp4
│   ├── {page_name}.nfo
│   └── {page_name}-poster.jpg
```
2. 多页视频:
```bash
├── {video_name}
│   ├── poster.jpg
│   ├── Season 1
│   │   ├── {page_name} - S01E01.mp4
│   │   ├── {page_name} - S01E01.nfo
│   │   ├── {page_name} - S01E01-thumb.jpg
│   │   ├── {page_name} - S01E02.mp4
│   │   ├── {page_name} - S01E02.nfo
│   │   └── {page_name} - S01E02-thumb.jpg
│   └── tvshow.nfo
```
这两个参数支持使用模板,其中用 <code v-pre>{{ }}</code> 包裹的模板变量在执行时会被动态替换为对应的内容。
对于 `video_name`,支持设置 bvid视频编号、title视频标题、upper_nameup 主名称、upper_midup 主 id、pubtime视频发布时间、fav_time视频收藏时间
对于 `page_name`,除支持 video 的全部参数外,还支持 ptitle分 P 标题、pid分 P 页号)。
为了解决文件名可能过长的问题,程序为模板引入了 `truncate` 函数。如 <code v-pre>{{ truncate title 10 }}</code> 表示截取 `title` 的前 10 个字符。
> [!TIP]
> 1. 仅收藏夹视频会区分 `fav_time` 和 `pubtime`,其它类型下载两者的取值是完全相同的;
> 2. `fav_time` 和 `pubtime` 的格式受 `time_format` 参数控制,详情可参考 [time_format 小节](#time-format)。
此外,`video_name` 和 `page_name` 还支持使用路径分割符,如 <code v-pre>{{ upper_mid }}/{{ title }}_{{ pubtime }}</code> 表示视频会根据 UP 主 id 将视频分到不同的文件夹中。
推荐仅在 `video_name` 中使用路径分割符,暂不清楚在 `page_name` 中使用路径分割符导致分页存储到子文件夹后是否还能被媒体服务器正确识别。
> [!CAUTION]
> **路径分隔符**在不同平台定义不同Windows 下为 `\`MacOS 和 Linux 下为 `/`
## `interval`
表示程序每次执行扫描下载的间隔时间,单位为秒。
## `upper_path`
UP 主头像和信息的保存位置。对于使用 Emby、Jellyfin 媒体服务器的用户,需确保此处路径指向 Emby、Jellyfin 配置中的 `/metadata/people/` 才能够正常在媒体服务器中显示 UP 主的头像。
## `nfo_time_type`
表示在视频信息中使用的时间类型,可选值为 `favtime`(收藏时间)和 `pubtime`(发布时间)。
仅收藏夹视频会区分 `fav_time``pubtime`,其它类型下载两者取值相同。
## `time_format`
时间格式,用于设置 `fav_time``pubtime``video_name``page_name` 中使用时的显示格式,支持的格式符号可以参考 [chrono strftime 文档](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)。
## `credential`
哔哩哔哩账号的身份凭据,请参考[凭据获取流程](https://nemo2011.github.io/bilibili-api/#/get-credential)获取并对应填写至配置文件中,后续 bili-sync 会在必要时自动刷新身份凭据,不再需要手动管理。
推荐使用匿名窗口获取,避免潜在的冲突。
## `filter_option`
过滤选项,用于设置程序的过滤规则,程序会从过滤结果中选择最优的视频、音频流下载。
这些内容的可选值可前往 [analyzer.rs](https://github.com/amtoaer/bili-sync/blob/24d0da0bf3ea65fd45d07587e4dcdbb24d11a589/crates/bili_sync/src/bilibili/analyzer.rs#L10-L55) 中查看。
注意将过滤范围设置过小可能导致筛选不到符合要求的流导致下载失败,建议谨慎修改。
### `video_max_quality`
视频流允许的最高质量。
### `video_min_quality`
视频流允许的最低质量。
### `audio_max_quality`
音频流允许的最高质量。
### `audio_min_quality`
音频流允许的最低质量。
### `codecs`
这是 bili-sync 选择视频编码的优先级顺序,优先级按顺序从高到低。此处对编码格式做一个简单说明:
+ AVC 又称 H.264,是目前使用最广泛的视频编码格式,绝大部分设备可以使用硬件解码播放该格式的视频(也因此播放普遍流畅),但是同等画质下视频体积较大。
+ HEV(C) 又称 H.265,与 AV1 都是新一代的视频编码格式。这两种编码相比 AVC 有更好的压缩率,同等画质下视频体积更小,但由于相对较新,硬件解码支持不如 AVC 广泛。如果你的播放设备不支持则只能使用软件解码播放,这种情况下可能导致播放卡顿、机器发热等问题。
建议查阅自己常用播放设备对这三种编码的硬件解码支持情况以选择合适的编码格式,如果硬件支持 HEV 或 AV1那么可以将其优先级调高。
而如果你的设备不支持,或者单纯懒得查询,那么推荐将 AVC 放在第一位以获得最好的兼容性。
### `no_dolby_video`
是否禁用杜比视频流。
### `no_dolby_audio`
是否禁用杜比音频流。
### `no_hdr`
是否禁用 HDR 视频流。
### `no_hires`
是否禁用 Hi-Res 音频流。
## `danmaku_option`
弹幕的设置选项,用于设置下载弹幕的样式,几乎全部取自[上游仓库](https://github.com/gwy15/danmu2ass)。
### `duration`
弹幕在屏幕上的持续时间,单位为秒。
### `font`
弹幕的字体。
### `font_size`
弹幕的字体大小。
### `width_ratio`
计算弹幕宽度的比例,为避免重叠可以调大这个数值。
### `horizontal_gap`
两条弹幕之间最小的水平距离。
### `lane_size`
弹幕所占据的高度,即“行高度/行间距”。
### `float_percentage`
屏幕上滚动弹幕最多高度百分比。
### `bottom_percentage`
屏幕上底部弹幕最多高度百分比。
### `opacity`
透明度,取值范围为 0-255。透明度可以通过 opacity / 255 计算得到。
### `bold`
是否加粗。
### `outline`
描边宽度。
### `time_offset`
时间轴偏移,>0 会让弹幕延后,<0 会让弹幕提前,单位为秒。
## `favorite_list`
你想要下载的收藏夹与想要保存的位置。简单示例:
```toml
3115878158 = "/home/amtoaer/Downloads/bili-sync/测试收藏夹"
```
收藏夹 ID 的获取方式可以参考[这里](/favorite)。
## `collection_list`
你想要下载的视频合集/视频列表与想要保存的位置。注意“视频合集”与“视频列表”是两种不同的类型。在配置文件中需要做区分:
```toml
"series:387051756:432248" = "/home/amtoaer/Downloads/bili-sync/测试视频列表"
"season:1728547:101343" = "/home/amtoaer/Downloads/bili-sync/测试合集"
```
具体说明可以参考[这里](/collection)。
## `submission_list`
你想要下载的 UP 主投稿与想要保存的位置。简单示例:
```toml
9183758 = "/home/amtoaer/Downloads/bili-sync/测试投稿"
```
UP 主 ID 的获取方式可以参考[这里](/submission)。
## `watch_later`
设置稍后再看的扫描开关与保存位置。
如果你希望下载稍后再看列表中的视频,可以将 `enabled` 设置为 `true`,并填写 `path`
```toml
enabled = true
path = "/home/amtoaer/Downloads/bili-sync/稍后再看"
```
## `concurrent_limit`
对 bili-sync 的并发下载进行多方面的限制,避免 api 请求过于频繁导致的风控。其中 video 和 page 表示下载任务的并发数rate_limit 表示 api 请求的流量限制。默认取值为:
```toml
[concurrent_limit]
video = 3
page = 2
[concurrent_limit.rate_limit]
limit = 4
duration = 250
```
具体来说,程序的处理逻辑是严格从上到下的,即程序会首先并发处理多个 video每个 video 内再并发处理多个 page程序的并行度可以简单衡量为 `video * page`(很多 video 都只有单个 page实际会更接近 `video * 1`),配置项中的 `video``page` 两个参数就是控制此处的,调节这两个参数可以宏观上控制程序的并行度。
另一方面,每个执行的任务内部都会发起若干 api 请求以获取信息,这些请求的整体频率受到 `rate_limit` 的限制,使用漏桶算法实现。如默认配置表示的是每 250ms 允许 4 个 api 请求,超过这个频率的请求会被暂时阻塞,直到漏桶中有空间为止。调节 `rate_limit` 可以从微观上控制程序的并行度,同时也是最直接、最显著的控制 api 请求频率的方法。
据观察 b 站风控限制大多集中在主站,因此目前 `rate_limit` 仅作用于主站的各类请求,如请求各类视频列表、视频信息、获取流下载地址等,对实际的视频、图片下载过程不做限制。
> [!TIP]
> 1. 一般来说,`video` 和 `page` 的值不需要过大;
> 2. `rate_limit` 的值可以根据网络环境和 api 请求频率进行调整,如果经常遇到风控可以优先调小 limit。

111
docs/design.md Normal file
View File

@@ -0,0 +1,111 @@
# 工作原理
本节会尽可能简单明了地介绍 `bili-sync` 的工作原理,让用户了解程序的整体执行过程。
## b 站的视频结构
在了解程序工作原理之前,我们需要先对 b 站视频的组织结构有一个大概的了解。简单来说:
- 收藏夹、稍后再看、视频合集、视频列表等结构都是由一系列视频构成的列表;
- 每个视频都有唯一的 bvid包含了封面、描述和标签信息并包含了一个或多个分页
- 每个分页都有一个唯一的 cid包含了封面、视频、音频、弹幕。
为了描述方便,在后文会将收藏夹、稍后再看这类结构统称为 video list将视频称为 video将分页称为 page。不难看出这三者有着很明显的层级关系**video list 包含若干 videovideo 包含若干 page**。
一个非常容易混淆的点是视频合集/视频列表与多页视频的区别:
> [!NOTE]
> ![bili_collection](./assets/bili_collection.webp)
>
>![bili_video](./assets/bili_video.webp)
这两张图中,上图是视频合集,下图是多页视频。这两者在展示上区别较小,但在结构上有相当大的不同。结合上面对 b 站视频结构的介绍,这个区别可以简单总结为:
+ **视频合集是由多个仅包含单个 page 的 video 组成的 video list**
+ **多页视频是由多个 page 组成的 video**
这说明它们是处于两个不同层级的结构,因此程序对其的处理方式也有着相当大的不同。
## 与 EMBY 媒体库的对应关系
EMBY 的一般结构是: `媒体库 - 文件夹 - 电影/电视剧 - 分季/分集`,方便起见,我采用了如下的对应关系:
1. **文件夹**:对应 b 站的 video list
2. **电视剧** 对应 b 站的 video
3. **第一季的所有分集**:对应 b 站的 page。
特别的,当 video 仅有一个 page 时为了避免过多的层级bili-sync 会将 page 展开到第二层级,变成与电视剧同级的电影。
因此,**需要将媒体库类型设置为“混合内容”以支持在同个媒体库中同时显示电视剧与电影**。
### 单 page 的 video
![single_page](./assets/single_page.webp)
### 多 page 的 video
![multi_page](./assets/multi_page.webp)
![multi_page_detail](./assets/multi_page_detail.webp)
## 数据库设计
> [!NOTE]
> 可以[前往此处](https://github.com/amtoaer/bili-sync/tree/main/crates/bili_sync_entity/src/entities)实时查看当前版本的数据库表结构。
既然拥有着明显的层级关系,那数据库表就很容易设计了。为了简化实现,程序没有额外考虑单个 video 被多个 video list 引用的情况(如一个视频同时在收藏夹和稍后再看中)。而是简单的将其设计为了不交叉的层级结构。
### video list 表
从上面的介绍可以看出video list 并不是一个具体的结构,而是拥有多种实现的抽象概念。我选择将其特化实现为多张表:
1. favorite收藏夹
2. watch_later稍后再看
3. collection: 视频合集/视频列表;
4. ....
### video 表
video 表包含了 video 的基本信息,如 bvid、标题、封面、描述、标签等。此外video 表还包含了与 video list 的关联。
具体来说,每一种 video list 都在 video 表中有一个对应的列,指向 video list 表中的 id如 favorite_id、collection_id 等。接下来将这些键与 video 的 bvid 绑在一起建立唯一索引,就可以保证在同一个 video list 中不会有重复的 video。
### page 表
page 表包含了 page 的基本信息,如 cid、标题、封面等。与 video 类似但更简单page 表仅包含了与 video 的关联。
## 执行过程
### 初始化
程序启动时会读取配置文件、迁移数据库、初始化日志等操作。如果发现需要的文件不存在,程序会自动创建。
### 扫描 video list 获取新视频
> [!WARNING]
> b 站实现接口时为了节省资源,通过 video list 获取到的 video 列表通常是分页且不包含详细信息的。
程序会扫描所有配置文件中包含的 video list获取其中包含的 video 的简单信息并填充到数据库。在实现时需要避免频繁的全量扫描。
具体到 bili-sync 的实现中,每个 video list 都有一个 `latest_row_at` 列,用于记录处理过的最新视频时间。程序在请求接口时会设置按时间排序,确保新视频位于前面。排序依据的时间根据 video list 的类型而定:收藏夹按收藏时间,投稿按照投稿时间...
拉取过程会逐页请求,程序会不断将获取到的视频保存到数据库中,直到发现第一个小于等于 `latest_row_at` 的视频时停止。接着将 `latest_row_at` 更新为最新的视频时间。
### 填充 video 详情
将新增视频的简单信息写入数据库后,下一步会填充 video 详情。正如上文所述:**通过 video list 获取到的 video 列表通常是不包含详细信息的**,因此需要额外的请求来填充这些信息。
这一步会筛选出所有未完全填充信息的 video逐个获取 video 的详细信息(如标签、包含的 page 等)并填充到数据库中。
在这个过程中,如果遇到 -404 错误码则说明视频无法被正常访问,程序会将该视频标记为无效并跳过。
### 下载未处理的视频
经过上面处理后,数据库中已经包含了所有需要的 video 和 page 信息,接下来只需要筛选其中“未完全下载”、“成功填充详细信息”的所有视频,并发下载即可。程序在 video 层级最多允许 3 个任务同时下载page 层级最多允许 2 个任务同时下载。
数据库中的 status 字段用于标记 video 和 page 的下载状态视频的各个部分封面、视频、nfo 等)包含在 status 的不同位中。程序会根据 status 的不同位来判断视频的下载状态,以此来决定是否需要下载。
如果某些部分下载失败status 字段会记录这些部分的失败次数,程序会在下次下载时重试。如果重试次数超过了设定的阈值,那么视频会被标记为下载失败,后续直接忽略。
此处程序对风控做了额外的处理,一般风控发生时接下来的所有请求都会失败,因此程序检测到风控时不会认为是某个视频下载失败,而是直接终止 video list 的全部下载任务,等待下次扫描时重试。

5
docs/favorite.md Normal file
View File

@@ -0,0 +1,5 @@
# 获取收藏夹信息
收藏夹的 ID 获取非常简单,在网页端打开自己的收藏夹列表,切换到你想要获取的收藏夹,然后查看 URL 地址栏中的 `fid` 参数内容即可。
![image](./assets/favorite.webp)

58
docs/index.md Normal file
View File

@@ -0,0 +1,58 @@
---
# https://vitepress.dev/reference/default-theme-home-page
layout: home
title: bili-sync
titleTemplate: 由 Rust & Tokio 驱动的哔哩哔哩同步工具
hero:
name: "bili-sync"
text: "由 Rust & Tokio 驱动的哔哩哔哩同步工具"
# tagline: My great project tagline
actions:
- theme: brand
text: 什么是 bili-sync
link: /introduction
- theme: alt
text: 快速开始
link: /quick-start
- theme: alt
text: GitHub
link: https://github.com/amtoaer/bili-sync
image:
src: /logo.webp
alt: bili-sync
features:
- icon: 🤖
title: 无需干预
details: 自动选择最优的视频和音频配置
- icon: 💾
title: 专为 NAS 设计
details: 可被 Emby、Jellyfin 等媒体服务器一键识别
- icon: 🐳
title: 部署简单
details: 提供简单易用的 docker 镜像
---
<style>
:root {
--vp-home-hero-name-color: transparent;
--vp-home-hero-name-background: -webkit-linear-gradient(120deg, #bd34fe 30%, #41d1ff);
--vp-home-hero-image-background-image: linear-gradient(-45deg, #bd34fe 50%, #47caff 50%);
--vp-home-hero-image-filter: blur(44px);
}
@media (min-width: 640px) {
:root {
--vp-home-hero-image-filter: blur(56px);
}
}
@media (min-width: 960px) {
:root {
--vp-home-hero-image-filter: blur(68px);
}
}
</style>

42
docs/introduction.md Normal file
View File

@@ -0,0 +1,42 @@
# bili-sync 是什么?
> [!TIP]
> 当前最新程序版本为 v2.3.0,文档将始终与最新程序版本保持一致。
bili-sync 是一款专为 NAS 用户编写的哔哩哔哩同步工具。
它的基本的工作原理是使用用户填写的凭据定期扫描视频合集、收藏夹等,获取到本地未下载过的内容并保存到本地,维持本地视频库与哔哩哔哩网站的同步。
下载的内容包括视频、封面、弹幕、标签与简介信息等,这些文件整体保持与 Emby、Jellyfin 等媒体服务器软件兼容的文件布局,使得目的文件夹可以直接被作为媒体库添加到这些软件中,无需干预自动识别。
## 使用截图
> [!WARNING]
> 媒体库类型请选择“混合内容”,否则可能导致多页视频无法正常显示。
### 概览
![概览](/assets/overview.webp)
### 详情
![详情](/assets/detail.webp)
### 播放(使用 infuse
![播放](/assets/play.webp)
### 文件排布
![文件](/assets/dir.webp)
## 功能与路线图
- [x] 使用用户填写的凭据认证,并在必要时自动刷新
- [x] 支持收藏夹与视频列表/视频合集的下载
- [x] 自动选择用户设置范围内最优的视频和音频流,并在下载完成后使用 FFmpeg 合并
- [x] 使用 Tokio 与 Reqwest对视频、视频分页进行异步并发下载
- [x] 使用媒体服务器支持的文件命名,方便一键作为媒体库导入
- [x] 当前轮次下载失败会在下一轮下载时重试,失败次数过多自动丢弃
- [x] 使用数据库保存媒体信息,避免对同个视频的多次请求
- [x] 打印日志,并在请求出现风控时自动终止,等待下一轮执行
- [x] 提供多平台的二进制可执行文件,为 Linux 平台提供了立即可用的 Docker 镜像
- [x] 支持对“稍后再看”内视频的自动扫描与下载
- [x] 支持对 UP 主投稿视频的自动扫描与下载
- [x] 支持限制任务的并行度和接口请求频率
- [ ] 下载单个文件时支持断点续传与并发下载

13
docs/package.json Normal file
View File

@@ -0,0 +1,13 @@
{
"dependencies": {},
"devDependencies": {
"markdown-it-task-lists": "^2.1.1",
"medium-zoom": "^1.1.0",
"vitepress": "^1.2.3"
},
"scripts": {
"docs:dev": "vitepress dev",
"docs:build": "vitepress build",
"docs:preview": "vitepress preview"
}
}

Some files were not shown because too many files have changed in this diff Show More