Compare commits

...

115 Commits

Author SHA1 Message Date
krau
15cf81e1bd fix: remove unnecessary chat ID validation and constant usage in Save method 2025-11-06 16:04:31 +08:00
krau
ae48bd52bf fix: handle chat ID parsing correctly by removing unnecessary prefix trimming, close #130 2025-11-06 15:01:18 +08:00
krau
44de871f63 fix: add context to HTTP request in getToken and simplify HTTP client initialization 2025-10-28 17:59:21 +08:00
krau
7a2a530e49 fix: update gotgproto dependency to v1.0.0-beta22 2025-10-28 17:55:22 +08:00
krau
3aa84e89bf fix: upgrade deps 2025-10-24 21:11:18 +08:00
krau
257c292679 fix: correct chat ID parsing by handling negative chat IDs properly 2025-10-12 22:23:08 +08:00
krau
0e2a9cacf2 feat: add lswatch cmd to list all user's watching chats 2025-10-12 21:54:43 +08:00
krau
a7854afb2a refactor: improve client initialization logic and ensure thread safety 2025-10-06 23:32:20 +08:00
krau
0e989cc1a6 fix: add logging for fetching telegraph page and handle image URLs from telegra.ph 2025-10-06 22:13:41 +08:00
krau
76a82a38ee fix: handle images on telegraph server 2025-10-06 21:57:35 +08:00
krau
c7a0076c15 fix: trim space in telegraph dir path , close #119 2025-10-03 09:32:45 +08:00
krau
ea07ff7eca fix: remove unused ChatTitle field from FilenameTemplateData struct 2025-09-30 21:36:23 +08:00
krau
4d837e946c feat: add .chatid variable in file name template 2025-09-30 21:34:35 +08:00
krau
f947ee6fc7 fix: implement filename strategy in userbot listen mode 2025-09-28 16:49:32 +08:00
krau
40ad12a892 fix: update SOCKS5 proxy dialer implementation for consistency 2025-09-18 22:28:25 +08:00
krau
697e419643 feat: refactor command registration to use a centralized handler list 2025-09-13 10:37:43 +08:00
Krau
eef051de3b feat: custom filename template (#110) 2025-09-13 10:25:45 +08:00
krau
6e29442c05 fix: update Docker deployment instructions for userbot integration 2025-09-13 10:21:10 +08:00
krau
a3f1f75caf fix: update initialization error message for clarity, close #108 2025-09-13 10:19:00 +08:00
Krau
f05dd883e3 feat: enhance URL handling by adding utility functions and filters for message entities (#105) 2025-09-09 20:16:56 +08:00
dependabot[bot]
9cb866de8c chore(deps): bump github.com/ulikunitz/xz from 0.5.12 to 0.5.14 (#102)
Bumps [github.com/ulikunitz/xz](https://github.com/ulikunitz/xz) from 0.5.12 to 0.5.14.
- [Commits](https://github.com/ulikunitz/xz/compare/v0.5.12...v0.5.14)

---
updated-dependencies:
- dependency-name: github.com/ulikunitz/xz
  dependency-version: 0.5.14
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-31 12:14:49 +08:00
krau
980455fd24 fix: remove go generate command from build process in Dockerfile and workflow 2025-08-27 11:17:03 +08:00
krau
24978470cd feat: add go generate command to build process and update go.mod dependencies 2025-08-27 11:12:42 +08:00
krau
215e082028 feat: implement internationalization support and update help commands 2025-08-27 11:09:38 +08:00
krau
a7b93e57fc refactor: js api 2025-08-24 22:49:44 +08:00
krau
a4b3b459a9 docs: change service restart policy to always 2025-08-24 14:47:13 +08:00
krau
06f326088a fix: remove redundant check for current version equality in update command 2025-08-24 14:37:01 +08:00
krau
b7d3ec6230 docs: update upgrade command 2025-08-24 14:26:47 +08:00
krau
f812990e1c feat: update help command to include usage and feedback group links 2025-08-24 14:18:43 +08:00
krau
492900bbef feat: add update command and callback for version checking and upgrading 2025-08-24 14:16:26 +08:00
krau
764be2a083 fix: improve error handling in config initialization 2025-08-24 14:16:23 +08:00
krau
46c21b77e9 feat: enhance Save method to validate chat ID and adjust ForceFile logic for image uploads 2025-08-24 12:02:27 +08:00
krau
8b389a58d5 fix: improve chat ID parsing in Save method and fallback to configured chat_id 2025-08-24 11:46:12 +08:00
krau
25ad9befa0 feat: add ForceFile option to TelegramStorageConfig and update Save method 2025-08-24 11:38:44 +08:00
krau
e824b210d1 feat: improve telegram storage chat ID parsing logic 2025-08-24 11:29:19 +08:00
krau
ae0aa7db3f fix: skip overwrite dirpath and storage when rule miss match 2025-08-24 11:28:37 +08:00
krau
226c15ef08 feat: add NormalizePathname function and update task handling for parsed items 2025-08-24 10:28:50 +08:00
krau
9b3f955e48 feat: use default HTTP client from netutil for task creation 2025-08-24 09:37:52 +08:00
krau
4997ec408f docs: update parsers 2025-08-23 20:42:51 +08:00
krau
0756cc9eb1 fix: improve parser configuration handling and default values 2025-08-23 20:40:13 +08:00
krau
37c32a23d4 feat: add Kemono parser with download info extraction and API handling 2025-08-23 20:18:02 +08:00
krau
3aa1e2eaed feat: enhance text message handling and parser configuration 2025-08-23 20:17:56 +08:00
krau
b87dd68880 feat: proxy client for parser 2025-08-23 20:17:24 +08:00
krau
68e5a51300 feat: file name staregy 2025-08-23 17:16:51 +08:00
krau
7300e54c40 refactor: rule package 2025-08-23 16:14:12 +08:00
krau
94f796d0e8 refactor: move version to config package 2025-08-23 16:10:02 +08:00
krau
c023fd869d feat: refactor jsParser to use ParserMethod constants and remove redundant locking in ParseWithContext 2025-08-23 16:04:32 +08:00
krau
e5d1e143e0 feat: configurable parser and refactor config 2025-08-23 14:29:32 +08:00
krau
03eb4f8a18 feat: update parser interface to include context in Parse method 2025-08-23 14:01:00 +08:00
krau
231eb61d25 docs: update parser plugin readme and add example danbooru parser 2025-08-23 12:35:04 +08:00
krau
fd1b586b8d feat: inject ghttp to js vm 2025-08-23 12:34:39 +08:00
krau
d035a3409e feat: support multiple parser additions and include media size in Twitter parser 2025-08-22 23:11:16 +08:00
krau
6112f6c240 feat: truncate item description to 233 characters in parsed text 2025-08-22 15:00:09 +08:00
krau
18eedf2edb docs: update parsers configuration and features 2025-08-22 10:02:44 +08:00
krau
5f9bba9ff7 docs: update contribute and parser help 2025-08-22 09:56:53 +08:00
krau
0d3d2209be docs: update readme 2025-08-22 09:28:24 +08:00
Krau
302db2fe75 feat: parse url with js plugins support (#96)
* feat: WIP. add parser functionality and text message handling

* fix: use json to marshal js result

* feat: add metadata handling and version validation for jsParser

* refactor: rename parser package to parsers and restructure parser handling

* refactor: core code struct and impl parse task handle

* feat: impl parsed download

* fix: seek cache file when processing tph picture

* feat: implement parsed task handling and progress tracking

* feat: enhance task processing with concurrency control and progress tracking

* feat: add resource ID generation and improve resource processing handling

* feat: improve message formatting in parsed text and progress completion

* feat: add example js plugin

* feat: implement Twitter parser

* fix: twitter parse video json decode error

* feat: impl stream mode for parse task
2025-08-21 23:48:17 +08:00
krau
79386bdd7d fix: improve error handling in recovery middleware 2025-08-21 14:39:41 +08:00
krau
f0607de2cc fix: update gotgproto dependency version 2025-08-15 16:49:18 +08:00
krau
b2bfc96a8f fix: get user client panic 2025-08-03 22:25:56 +08:00
krau
0c5bb2ba77 docs(zh): update watch feat 2025-08-03 17:35:47 +08:00
krau
9cc87380ff feat: update bot commands and help 2025-08-03 17:26:47 +08:00
krau
46afc14322 fix: debounce send media event and ignore edit or delete updates 2025-08-03 17:22:55 +08:00
krau
0c16650ea5 fix: watch chat check 2025-08-03 17:04:26 +08:00
krau
133453b5d4 feat: implement watch for monitoring chat messages
- Added a new command handler for /watch that allows users to listen to messages from a specified chat and save them according to storage rules.
- Introduced filtering options for messages using regular expressions.
- Implemented functionality to start and stop watching chats, including error handling for invalid inputs and user settings.
- Created a new utility package for message element handling related to the watch feature.
- Updated the user model to manage watched chats, including methods to add, remove, and check if a chat is being watched.
2025-08-03 16:55:56 +08:00
krau
8f9ef07d1c refactor: replace functions.GetMediaFileNameWithId with tgutil.GetMediaFileName for better media file name handling 2025-08-02 16:55:12 +08:00
krau
36285a0700 feat(storage/telegram): add support for uploading images excluding webp format 2025-08-02 11:41:18 +08:00
krau
ccf206d176 feat(storage/telegram): enhance file upload handling with content length support and media type differentiation 2025-08-02 11:22:14 +08:00
krau
4c851cbbaf ci: fix version inject 2025-08-02 10:58:29 +08:00
krau
b9d14f79c8 refactor: simplify dler client interface 2025-08-02 10:01:59 +08:00
krau
ee5e0b8ff0 fix: use gotgproto fork and upgrade deps 2025-07-30 17:46:36 +08:00
krau
6423fb25a7 fix: update max upload part size to use uploader's maximum value and adjust document upload options 2025-07-28 14:58:01 +08:00
krau
03907f2d32 fix: improve error handling during file download in stream mode 2025-07-28 14:37:08 +08:00
github-actions[bot]
9e5042bda1 docs(contributor): contrib-readme-action has updated readme (#92)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-07-09 20:37:16 +08:00
krau
4ebacb02c1 chore: tidy go mod 2025-07-09 20:35:55 +08:00
Abner
818ac9b240 feat: remote config 2025-07-09 20:30:29 +08:00
krau
fc4a112f08 fix: enhance filename generation by using media file name if available 2025-07-07 14:12:33 +08:00
krau
7a2274baa0 refactor: update configuration file structure and improve comments for clarity 2025-07-07 13:57:58 +08:00
krau
69a3ed6f4e fix: parse chat ID correctly in Save method for Telegram storage 2025-07-07 13:48:16 +08:00
krau
36f3dd83fc refactor: remove error wrapping in retry middleware skip logic 2025-07-07 13:44:32 +08:00
krau
501b9d844a docs: update userbot content 2025-07-04 16:26:17 +08:00
krau
03cec7ec01 docs: add IS-ALBUM rule type 2025-07-04 16:18:01 +08:00
krau
dc0debcd1c fix: add unique filename handling with error logging in Save method for Minio and Webdav 2025-07-04 16:10:42 +08:00
krau
4b136bd41e feat: add test case for nested directory file writing 2025-07-04 16:08:01 +08:00
krau
d703f11ea0 feat: implement album handling rules and refactor related logic 2025-07-04 16:01:29 +08:00
krau
3ce9926967 feat: handle media group message in updates 2025-07-04 14:36:03 +08:00
krau
80146176f0 feat: make retry middleware configurable 2025-07-04 13:38:55 +08:00
krau
14ba2afdf8 chore: update funding url 2025-07-04 13:38:28 +08:00
krau
f4d427a1cb fix(dockerignore): ensure docs directory is ignored correctly 2025-06-30 22:49:01 +08:00
krau
f84c83a7e2 fix(ci): try speed up docker build 2025-06-30 22:47:47 +08:00
krau
cb6540c017 fix(minio): trim leading slash in JoinStoragePath method 2025-06-30 22:19:33 +08:00
krau
e7bab27543 feat: add user client inject in file link handler 2025-06-29 23:08:48 +08:00
krau
f693bd6103 refactor: update file handling to use new downloader interface; remove unused tdler package 2025-06-29 23:00:40 +08:00
krau
75f52569a0 deps: upgrade 2025-06-29 22:26:52 +08:00
krau
c795f957a9 refactor: user client and proxy handling; remove unused auth code 2025-06-29 22:25:05 +08:00
krau
3b85911e3d fix: mimetype nil pointer 2025-06-20 22:42:57 +08:00
krau
336309fad0 chore: remove unuse comment 2025-06-20 22:29:37 +08:00
krau
394cdff865 feat: regex filter for batch save message 2025-06-20 22:25:46 +08:00
krau
40cb3dad9d feat: handle grouped msgs, close #72 2025-06-20 22:07:21 +08:00
krau
2979628cf7 docs(zh): add hook docs 2025-06-20 21:46:00 +08:00
krau
c82c2462bf feat: exec command hook , close #79 2025-06-20 21:30:50 +08:00
krau
88128ecac2 fix: cache init after config 2025-06-20 21:27:45 +08:00
krau
758564d436 feat(config): add cache configuration options for TTL, num counters, and max cost 2025-06-18 10:51:03 +08:00
krau
f5e33472eb feat: add IterMessages function for message iteration with error handling 2025-06-18 10:50:54 +08:00
krau
4df2c5a06d refactor: replace key package with ctxkey for context keys 2025-06-17 22:19:06 +08:00
krau
eb6f8675a4 fix(minio): pass the size to minio puitobject 2025-06-17 22:10:20 +08:00
krau
473a5b9413 chore: update help text 2025-06-16 17:31:34 +08:00
krau
6c2abe3025 chore: remove deprecated config 2025-06-16 17:11:26 +08:00
krau
e7e5b9f434 docs: add repo link 2025-06-16 17:05:31 +08:00
krau
d4d39d1c07 chore: update readme 2025-06-16 16:57:39 +08:00
krau
73b5f1b18e docs: add en translate 2025-06-16 16:30:45 +08:00
krau
837700bf63 docs: adjust font size 2025-06-16 16:08:03 +08:00
krau
53e6d7cc54 ci(docs): fix ci 2025-06-16 16:01:50 +08:00
krau
4206d1fe96 docs: refactor 2025-06-16 15:58:03 +08:00
krau
6566dbbf96 chore: add new storage configuration for Telegram channel 2025-06-16 00:24:38 +08:00
176 changed files with 6241 additions and 1300 deletions

View File

@@ -6,6 +6,6 @@
downloads/
data/
cache/
docs
docs/
config.example.toml
docker-compose.*

2
.github/FUNDING.yml vendored
View File

@@ -1,5 +1,5 @@
# These are supported funding model platforms
custom: [
"https://afdian.com/a/acherkrau"
"https://afdian.com/a/unvapp"
]

View File

@@ -29,13 +29,7 @@ jobs:
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha
type=raw,value=latest
type=ref,event=branch
type=ref,event=tag
labels: |
org.opencontainers.image.title=${{ env.IMAGE_NAME }}
org.opencontainers.image.source=https://github.com/krau/SaveAny-Bot
org.opencontainers.image.url=https://github.com/krau/SaveAny-Bot
type=raw,value=latest,enable={{is_default_branch}}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
@@ -50,23 +44,26 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract version from Git Ref
id: extract_version
- name: Extract Dockerfile args
id: args
run: |
VERSION=$(echo "${{ github.ref }}" | sed 's/refs\/tags\/v//')
echo "VERSION=${VERSION}" >> $GITHUB_ENV
echo "git_commit=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"
echo "build_time=$(git show -s --format=%cI)" >> "$GITHUB_OUTPUT"
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
cache-from: type=gha
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: |
type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
type=gha
cache-to: type=gha,mode=max
build-args: |
VERSION=${{ steps.meta.outputs.version }}
GitCommit=${{ github.sha }}
BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
GitCommit=${{ steps.args.outputs.git_commit }}
BuildTime=${{ steps.args.outputs.build_time }}

View File

@@ -63,9 +63,9 @@ jobs:
README.md
ldflags: >-
-s -w
-X "github.com/krau/SaveAny-Bot/pkg/consts.Version=${{ env.VERSION }}"
-X "github.com/krau/SaveAny-Bot/pkg/consts.BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}"
-X "github.com/krau/SaveAny-Bot/pkg/consts.GitCommit=${{ github.sha }}"
-X "github.com/krau/SaveAny-Bot/config.Version=${{ env.VERSION }}"
-X "github.com/krau/SaveAny-Bot/config.BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}"
-X "github.com/krau/SaveAny-Bot/config.GitCommit=${{ github.sha }}"
binary_name: saveany-bot
env:
VERSION: ${{ env.VERSION }}

View File

@@ -6,17 +6,31 @@ on:
paths:
- "docs/**"
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: actions/cache@v4
submodules: true # Fetch Hugo themes (true OR recursive)
fetch-depth: 0 # Fetch all history for .GitInfo and .Lastmod
- name: Setup Hugo
uses: peaceiris/actions-hugo@v3
with:
key: ${{ github.ref }}
path: .cache
- run: pip install mkdocs-material
- run: cd docs && mkdocs gh-deploy --force
hugo-version: '0.147.8'
extended: true
- name: Build
run: hugo --minify --destination public --source docs
- name: Deploy
uses: peaceiris/actions-gh-pages@v3
if: github.ref == 'refs/heads/main'
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./docs/public
publish_branch: gh-pages

3
.gitignore vendored
View File

@@ -6,4 +6,5 @@ downloads/
session.*
cache.db
.vscode/
temp/
temp/
.hugo_build.lock

View File

@@ -6,22 +6,33 @@ ARG BuildTime="Unknown"
WORKDIR /app
COPY . .
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg \
CGO_ENABLED=0 \
go build -trimpath \
-ldflags "-s -w \
-X github.com/krau/SaveAny-Bot/common.Version=${VERSION} \
-X github.com/krau/SaveAny-Bot/common.GitCommit=${GiTCommit} \
-X github.com/krau/SaveAny-Bot/common.BuildTime=${BuildTime}" \
CGO_ENABLED=0 \
go build -trimpath \
-ldflags=" \
-s -w \
-X 'github.com/krau/SaveAny-Bot/config.Version=${VERSION}' \
-X 'github.com/krau/SaveAny-Bot/config.GitCommit=${GitCommit}' \
-X 'github.com/krau/SaveAny-Bot/config.BuildTime=${BuildTime}' \
" \
-o saveany-bot .
FROM alpine:latest
RUN apk add --no-cache curl
WORKDIR /app
COPY --from=builder /app/saveany-bot .
COPY entrypoint.sh .
ENTRYPOINT ["/app/saveany-bot"]
RUN chmod +x /app/saveany-bot && \
chmod +x /app/entrypoint.sh
ENTRYPOINT ["/app/entrypoint.sh"]

View File

@@ -1,28 +1,81 @@
<div align="center">
# <img src="docs/logo.jpg" width="45" align="center"> Save Any Bot
# <img src="docs/static/logo.png" width="45" align="center"> Save Any Bot
**简体中文** | [English](README_EN.md)
**简体中文** | [English](https://sabot.unv.app/en/)
把 Telegram 的文件存到各类存储端.
> **把 Telegram 的文件存到多种存储端.**
> _就像 PikPak Bot 一样_
[![Release Date](https://img.shields.io/github/release-date/krau/saveany-bot?label=release)](https://github.com/krau/saveany-bot/releases)
[![tag](https://img.shields.io/github/v/tag/krau/saveany-bot.svg)](https://github.com/krau/saveany-bot/releases)
[![Build Status](https://img.shields.io/github/actions/workflow/status/krau/saveany-bot/build-release.yml)](https://github.com/krau/saveany-bot/actions/workflows/build-release.yml)
[![Stars](https://img.shields.io/github/stars/krau/saveany-bot?style=flat)](https://github.com/krau/saveany-bot/stargazers)
[![Downloads](https://img.shields.io/github/downloads/krau/saveany-bot/total)](https://github.com/krau/saveany-bot/releases)
[![Issues](https://img.shields.io/github/issues/krau/saveany-bot)](https://github.com/krau/saveany-bot/issues)
[![Pull Requests](https://img.shields.io/github/issues-pr/krau/saveany-bot?label=pr)](https://github.com/krau/saveany-bot/pulls)
[![License](https://img.shields.io/github/license/krau/saveany-bot)](./LICENSE)
</div>
## [部署](https://sabot.unv.app/deploy/)
## 🎯 Features
## [参与开发](https://sabot.unv.app/contribute/)
- 支持文档/视频/图片/贴纸…甚至还有 [Telegraph](https://telegra.ph/)
- 破解禁止保存的文件
- 批量下载
- 流式传输
- 多用户使用
- 基于存储规则的自动整理
- 监听并自动转存指定聊天的消息, 支持过滤
- 使用 js 编写解析器插件以转存任意网站的文件
- 存储端支持:
- Alist
- S3 (MinioSDK)
- WebDAV
- 本地磁盘
- Telegram (重传回指定聊天)
---
## 📦 Quick Start
## 赞助
创建文件 `config.toml` 并填入以下内容:
```toml
[telegram]
token = "" # 你的 Bot Token, 在 @BotFather 获取
[telegram.proxy]
# 启用代理连接 telegram, 当前只支持 socks5
enable = false
url = "socks5://127.0.0.1:7890"
[[storages]]
name = "本地磁盘"
type = "local"
enable = true
base_path = "./downloads"
[[users]]
id = 114514 # 你的 Telegram 账号 id
storages = []
blacklist = true
```
使用 Docker 运行 Save Any Bot:
```bash
docker run -d --name saveany-bot \
-v ./config.toml:/app/config.toml \
-v ./downloads:/app/downloads \
ghcr.io/krau/saveany-bot:latest
```
请 [**查看文档**](https://sabot.unv.app/) 以获取更多配置选项和使用方法.
## Sponsors
本项目受到 [YxVM](https://yxvm.com/) 与 [NodeSupport](https://github.com/NodeSeekDev/NodeSupport) 的支持.
如果这个项目对你有帮助, 你可以考虑通过以下方式赞助我:
- [爱发电](https://afdian.com/a/acherkrau)
- [爱发电](https://afdian.com/a/unvapp)
## Contributors
@@ -37,6 +90,13 @@
<sub><b>Krau</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/Silentely">
<img src="https://avatars.githubusercontent.com/u/22141172?v=4" width="100;" alt="Silentely"/>
<br />
<sub><b>Abner</b></sub>
</a>
</td>
<td align="center">
<a href="https://github.com/TG-Twilight">
<img src="https://avatars.githubusercontent.com/u/121682528?v=4" width="100;" alt="TG-Twilight"/>
@@ -52,8 +112,8 @@
</a>
</td>
<td align="center">
<a href="https://github.com/ahcorn">
<img src="https://avatars.githubusercontent.com/u/42889600?v=4" width="100;" alt="ahcorn"/>
<a href="https://github.com/AHCorn">
<img src="https://avatars.githubusercontent.com/u/42889600?v=4" width="100;" alt="AHCorn"/>
<br />
<sub><b>安和</b></sub>
</a>
@@ -68,4 +128,11 @@
- [gotd](https://github.com/gotd/td)
- [TG-FileStreamBot](https://github.com/EverythingSuckz/TG-FileStreamBot)
- [gotgproto](https://github.com/celestix/gotgproto)
- [tdl](https://github.com/iyear/tdl)
- All the dependencies
## Contact
- [![Group](https://img.shields.io/badge/ProjectSaveAny-Group-blue)](https://t.me/ProjectSaveAny)
- [![Discussion](https://img.shields.io/badge/Github-Discussion-white)](https://github.com/krau/saveany-bot/discussions)
- [![PersonalChannel](https://img.shields.io/badge/Krau-PersonalChannel-cyan)](https://t.me/acherkrau)

View File

@@ -1,108 +0,0 @@
<div align="center">
# <img src="docs/logo.jpg" width="45" align="center"> Save Any Bot
[简体中文](README.md) | **English**
Save Telegram files to various storage endpoints.
> _Just like PikPak Bot_
</div>
## Deployment
### Deploy from Binary
Download the binary file for your platform from the [Release](https://github.com/krau/SaveAny-Bot/releases) page.
Create a `config.toml` file in the extracted directory, refer to [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) for configuration.
Run:
```bash
chmod +x saveany-bot
./saveany-bot
```
#### Add as systemd Service
Create file `/etc/systemd/system/saveany-bot.service` and write the following content:
```
[Unit]
Description=SaveAnyBot
After=systemd-user-sessions.service
[Service]
Type=simple
WorkingDirectory=/yourpath/
ExecStart=/yourpath/saveany-bot
Restart=on-failure
[Install]
WantedBy=multi-user.target
```
Enable auto-start and start the service:
```bash
systemctl enable --now saveany-bot
```
### Deploy with Docker
#### Docker Compose
Download [docker-compose.yml](https://github.com/krau/SaveAny-Bot/blob/main/docker-compose.yml) file and create a `config.toml` file in the same directory, refer to [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) for configuration.
Run:
```bash
docker compose up -d
```
#### Docker
```shell
docker run -d --name saveany-bot \
-v /path/to/config.toml:/app/config.toml \
-v /path/to/downloads:/app/downloads \
ghcr.io/krau/saveany-bot:latest
```
## Update
Use `upgrade` or `up` command to upgrade to the latest version:
```bash
./saveany-bot upgrade
```
If deployed with Docker, use the following commands to update:
```bash
docker pull ghcr.io/krau/saveany-bot:latest
docker restart saveany-bot
```
## Usage
Send (forward) files to the Bot and follow the prompts.
---
## Sponsors
This project is supported by [YxVM](https://yxvm.com/) and [NodeSupport](https://github.com/NodeSeekDev/NodeSupport).
You can consider sponsoring me if this project helps you:
- [Afdian](https://afdian.com/a/acherkrau)
## Thanks
- [gotd](https://github.com/gotd/td)
- [TG-FileStreamBot](https://github.com/EverythingSuckz/TG-FileStreamBot)
- [gotgproto](https://github.com/celestix/gotgproto)
- All the dependencies

View File

@@ -2,7 +2,6 @@ package bot
import (
"context"
"net/url"
"time"
"github.com/celestix/gotgproto"
@@ -14,31 +13,23 @@ import (
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers"
"github.com/krau/SaveAny-Bot/client/middleware"
"github.com/krau/SaveAny-Bot/common/utils/netutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/ncruces/go-sqlite3/gormlite"
"golang.org/x/net/proxy"
)
var Client *gotgproto.Client
func newProxyDialer(proxyUrl string) (proxy.Dialer, error) {
url, err := url.Parse(proxyUrl)
if err != nil {
return nil, err
}
return proxy.FromURL(url, proxy.Direct)
}
func Init(ctx context.Context) {
func Init(ctx context.Context) <-chan struct{} {
log.FromContext(ctx).Info("初始化 Bot...")
resultChan := make(chan struct {
client *gotgproto.Client
err error
})
shouldRestart := make(chan struct{})
go func() {
var resolver dcs.Resolver
if config.Cfg.Telegram.Proxy.Enable && config.Cfg.Telegram.Proxy.URL != "" {
dialer, err := newProxyDialer(config.Cfg.Telegram.Proxy.URL)
if config.C().Telegram.Proxy.Enable && config.C().Telegram.Proxy.URL != "" {
dialer, err := netutil.NewProxyDialer(config.C().Telegram.Proxy.URL)
if err != nil {
resultChan <- struct {
client *gotgproto.Client
@@ -52,19 +43,24 @@ func Init(ctx context.Context) {
} else {
resolver = dcs.DefaultResolver()
}
client, err := gotgproto.NewClient(config.Cfg.Telegram.AppID,
config.Cfg.Telegram.AppHash,
gotgproto.ClientTypeBot(config.Cfg.Telegram.Token),
client, err := gotgproto.NewClient(
config.C().Telegram.AppID,
config.C().Telegram.AppHash,
gotgproto.ClientTypeBot(config.C().Telegram.Token),
&gotgproto.ClientOpts{
Session: sessionMaker.SqlSession(gormlite.Open(config.Cfg.DB.Session)),
Session: sessionMaker.SqlSession(gormlite.Open(config.C().DB.Session)),
DisableCopyright: true,
Middlewares: middleware.NewDefaultMiddlewares(ctx, 5*time.Minute),
Resolver: resolver,
Context: ctx,
MaxRetries: config.Cfg.Telegram.RpcRetry,
MaxRetries: config.C().Telegram.RpcRetry,
AutoFetchReply: true,
ErrorHandler: func(ctx *ext.Context, u *ext.Update, s string) error {
log.FromContext(ctx).Errorf("Unhandled error: %s", s)
if s == "SAVEANTBOT-RESTART" {
shouldRestart <- struct{}{}
return dispatcher.EndGroups
}
log.FromContext(ctx).Errorf("unhandled error: %s", s)
return dispatcher.EndGroups
},
},
@@ -79,17 +75,13 @@ func Init(ctx context.Context) {
client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
Scope: &tg.BotCommandScopeDefault{},
})
commands := make([]tg.BotCommand, 0, len(handlers.CommandHandlers))
for _, info := range handlers.CommandHandlers {
commands = append(commands, tg.BotCommand{Command: info.Cmd, Description: info.Desc})
}
_, err = client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
Scope: &tg.BotCommandScopeDefault{},
Commands: []tg.BotCommand{
{Command: "start", Description: "开始使用"},
{Command: "help", Description: "显示帮助"},
{Command: "silent", Description: "开启/关闭静默模式"},
{Command: "storage", Description: "设置默认存储端"},
{Command: "save", Description: "保存所回复的文件"},
{Command: "dir", Description: "管理存储文件夹"},
{Command: "rule", Description: "管理规则"},
},
Scope: &tg.BotCommandScopeDefault{},
Commands: commands,
})
resultChan <- struct {
client *gotgproto.Client
@@ -104,8 +96,8 @@ func Init(ctx context.Context) {
if result.err != nil {
log.FromContext(ctx).Fatalf("初始化 Bot 失败: %s", result.err)
}
Client = result.client
handlers.Register(Client.Dispatcher)
handlers.Register(result.client.Dispatcher)
log.FromContext(ctx).Info("Bot 初始化完成")
}
return shouldRestart
}

View File

@@ -3,6 +3,7 @@ package handlers
import (
"errors"
"fmt"
"path"
"strings"
"github.com/celestix/gotgproto/dispatcher"
@@ -11,6 +12,7 @@ import (
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
@@ -72,7 +74,12 @@ func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
}
return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files[0], msgID)
case tasktype.TaskTypeTphpics:
return shortcut.CreateAndAddTphTaskWithEdit(ctx, userID, data.TphPageNode, data.TphDirPath, data.TphPics, selectedStorage, msgID)
return shortcut.CreateAndAddtelegraphWithEdit(ctx, userID, data.TphPageNode, data.TphDirPath, data.TphPics, selectedStorage, msgID)
case tasktype.TaskTypeParseditem:
if len(data.ParsedItem.Resources) > 1 {
dirPath = path.Join(dirPath, fsutil.NormalizePathname(data.ParsedItem.Title))
}
shortcut.CreateAndAddParsedTaskWithEdit(ctx, selectedStorage, dirPath, data.ParsedItem, msgID, userID)
default:
log.FromContext(ctx).Errorf("Unsupported task type: %s", data.TaskType)
}

View File

@@ -0,0 +1,144 @@
package handlers
import (
"fmt"
"strings"
"text/template"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/fnamest"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
)
func handleConfigCmd(ctx *ext.Context, update *ext.Update) error {
ctx.Reply(update, ext.ReplyTextString("请选择要配置的选项"), &ext.ReplyOpts{
Markup: &tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
&tg.KeyboardButtonCallback{
Text: "文件名策略",
Data: fmt.Appendf(nil, "%s %s", tcbdata.TypeConfig, "fnamest"),
},
},
},
},
},
})
return dispatcher.EndGroups
}
func handleConfigCallback(ctx *ext.Context, update *ext.Update) error {
args := strings.Fields(string(update.CallbackQuery.Data))
invaildDataAnswer := func() error {
ctx.AnswerCallback(&tg.MessagesSetBotCallbackAnswerRequest{
QueryID: update.CallbackQuery.GetQueryID(),
Alert: true,
Message: "无效的回调数据",
CacheTime: 5,
})
return dispatcher.EndGroups
}
if len(args) < 2 {
return invaildDataAnswer()
}
switch args[1] {
case "fnamest":
return handleConfigFnameSTCallback(ctx, update)
default:
return invaildDataAnswer()
}
}
func handleConfigFnameSTCallback(ctx *ext.Context, update *ext.Update) error {
userID := update.CallbackQuery.GetUserID()
user, err := database.GetUserByChatID(ctx, userID)
if err != nil {
return err
}
args := strings.Fields(string(update.CallbackQuery.Data))
if len(args) == 3 {
selected := args[2]
st, err := fnamest.ParseFnameST(selected)
if err != nil {
return err
}
user.FilenameStrategy = st.String()
if err := database.UpdateUser(ctx, user); err != nil {
return err
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: update.CallbackQuery.GetMsgID(),
Message: fmt.Sprintf("已将文件名策略设置为: %s", fnamest.FnameSTDisplay[st]),
})
return dispatcher.EndGroups
}
opts := fnamest.FnameSTValues()
buttons := make([]tg.KeyboardButtonClass, 0, len(opts))
for _, opt := range opts {
buttons = append(buttons, &tg.KeyboardButtonCallback{
Text: fnamest.FnameSTDisplay[opt],
Data: fmt.Appendf(nil, "%s %s %s", tcbdata.TypeConfig, "fnamest", opt),
})
}
markup := &tg.ReplyInlineMarkup{Rows: []tg.KeyboardButtonRow{
{Buttons: buttons},
}}
currentStStr := user.FilenameStrategy
if currentStStr == "" {
currentStStr = fnamest.Default.String()
}
currentSt, err := fnamest.ParseFnameST(currentStStr)
if err != nil {
currentSt = fnamest.Default
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: update.CallbackQuery.GetMsgID(),
Message: fmt.Sprintf("请选择文件名策略, 当前策略: %s", fnamest.FnameSTDisplay[currentSt]),
ReplyMarkup: markup,
})
return dispatcher.EndGroups
}
func handleConfigFnameTmpl(ctx *ext.Context, update *ext.Update) error {
userID := update.GetUserChat().GetID()
user, err := database.GetUserByChatID(ctx, userID)
if err != nil {
return err
}
args := strings.Fields(string(update.EffectiveMessage.Text))
if len(args) <= 1 {
text := `使用该命令设置文件名模板, 示例:
/fnametmpl 图片_{{.msgid}}_{{.msgdate}}.jpg
可用变量:
- {{.msgid}}: 消息ID
- {{.msgtags}}: 消息中的标签, 将以下划线分隔输出
- {{.msggen}}: 根据消息生成的文件名
- {{.msgdate}}: 消息日期, 格式 YYYY-MM-DD_HH-MM-SS
- {{.origname}}: 媒体的原始文件名 (如果有)
- {{.chatid}}: 消息的聊天ID
`
if user.FilenameTemplate != "" {
text += fmt.Sprintf("\n\n当前模板: %s", user.FilenameTemplate)
}
text += "\n\n模板仅在文件名策略设置为 '自定义模板' 时生效, 且模板解析错误时会回退到默认文件名"
ctx.Reply(update, ext.ReplyTextString(text), nil)
return dispatcher.EndGroups
}
newTmpl := strings.Join(args[1:], " ")
_, err = template.New("filename").Parse(newTmpl)
if err != nil {
ctx.Reply(update, ext.ReplyTextString("无效的模板, 请检查语法\n"+err.Error()), nil)
return dispatcher.EndGroups
}
user.FilenameTemplate = newTmpl
if err := database.UpdateUser(ctx, user); err != nil {
return err
}
ctx.Reply(update, ext.ReplyTextString("已更新文件名模板"), nil)
return dispatcher.EndGroups
}

View File

@@ -5,26 +5,16 @@ import (
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/krau/SaveAny-Bot/pkg/consts"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/config"
)
func handleHelpCmd(ctx *ext.Context, update *ext.Update) error {
const helpText string = `
Save Any Bot - 转存你的 Telegram 文件
版本: %s , 提交: %s
命令:
/start - 开始使用
/help - 显示帮助
/silent - 开关静默模式
/storage - 设置默认存储位置
/save [自定义文件名] - 保存文件
静默模式: 开启后 Bot 直接保存到收到的文件到默认位置, 不再询问
默认存储位置: 在静默模式下保存到的位置
向 Bot 发送(转发)文件, 或发送一个公开频道的消息链接以保存文件
`
ctx.Reply(update, ext.ReplyTextString(fmt.Sprintf(helpText, consts.Version, consts.GitCommit)), nil)
shortHash := config.GitCommit
if len(shortHash) > 7 {
shortHash = shortHash[:7]
}
ctx.Reply(update, ext.ReplyTextString(fmt.Sprintf(i18n.T(i18nk.BotMsgHelpTextFmt), config.Version, shortHash)), nil)
return dispatcher.EndGroups
}

View File

@@ -38,8 +38,7 @@ func handleMessageLink(ctx *ext.Context, update *ext.Update) error {
editReplied("构建存储选择键盘失败: "+err.Error(), nil)
return dispatcher.EndGroups
}
editReplied(fmt.Sprintf("找到 %d 个文件, 请选择存储位置", len(files)),
markup)
editReplied(fmt.Sprintf("找到 %d 个文件, 请选择存储位置", len(files)), markup)
return dispatcher.EndGroups
}

View File

@@ -1,23 +1,49 @@
package handlers
import (
"fmt"
"sync"
"time"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/charmbracelet/log"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
)
func handleMediaMessage(ctx *ext.Context, update *ext.Update) error {
logger := log.FromContext(ctx)
message := update.EffectiveMessage.Message
groupID, isGroup := message.GetGroupedID()
if isGroup && groupID != 0 {
return handleGroupMediaMessage(ctx, update, message, groupID)
}
logger.Debugf("Got media: %s", message.Media.TypeName())
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message)
userId := update.GetUserChat().GetID()
userDB, err := database.GetUserByChatID(ctx, userId)
if err != nil {
return err
}
userId := update.GetUserChat().GetID()
// tfOpts := make([]tfile.TGFileOption, 0)
// switch userDB.FilenameStrategy {
// case fnamest.Message.String():
// tfOpts = append(tfOpts, tfile.WithName(tgutil.GenFileNameFromMessage(*message)))
// default:
// }
tfOpts := mediautil.TfileOptions(ctx, userDB, message)
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message, tfOpts...)
if err != nil {
return err
}
stors := storage.GetUserStorages(ctx, userId)
req, err := msgelem.BuildAddOneSelectStorageMessage(ctx, stors, file, msg.ID)
if err != nil {
@@ -38,11 +64,119 @@ func handleSilentSaveMedia(ctx *ext.Context, update *ext.Update) error {
return dispatcher.EndGroups
}
message := update.EffectiveMessage.Message
groupID, isGroup := message.GetGroupedID()
if isGroup && groupID != 0 {
return handleGroupMediaMessage(ctx, update, message, groupID)
}
logger.Debugf("Got media: %s", message.Media.TypeName())
userID := update.GetUserChat().GetID()
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message)
userDB, err := database.GetUserByChatID(ctx, userID)
if err != nil {
return err
}
// tfOpts := make([]tfile.TGFileOption, 0)
// switch userDB.FilenameStrategy {
// case fnamest.Message.String():
// tfOpts = append(tfOpts, tfile.WithName(tgutil.GenFileNameFromMessage(*message)))
// default:
// }
tfOpts := mediautil.TfileOptions(ctx, userDB, message)
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message, tfOpts...)
if err != nil {
return err
}
return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, stor, "", file, msg.ID)
}
type MediaGroupHandler struct {
groups map[int64][]tfile.TGFileMessage
timers map[int64]*time.Timer
mu sync.Mutex
timeout time.Duration
}
var mediaGroupHandler = &MediaGroupHandler{
groups: make(map[int64][]tfile.TGFileMessage),
timers: make(map[int64]*time.Timer),
timeout: 1 * time.Second,
}
func handleGroupMediaMessage(ctx *ext.Context, update *ext.Update, message *tg.Message, groupID int64) error {
logger := log.FromContext(ctx)
media := message.Media
supported := mediautil.IsSupported(media)
if !supported {
return dispatcher.EndGroups
}
file, err := tfile.FromMediaMessage(media, ctx.Raw, message, tfile.WithNameIfEmpty(
tgutil.GenFileNameFromMessage(*message),
))
if err != nil {
logger.Errorf("Failed to get file from media: %s", err)
return dispatcher.EndGroups
}
mediaGroupHandler.mu.Lock()
defer mediaGroupHandler.mu.Unlock()
if mediaGroupHandler.groups[groupID] == nil {
mediaGroupHandler.groups[groupID] = make([]tfile.TGFileMessage, 0)
}
mediaGroupHandler.groups[groupID] = append(mediaGroupHandler.groups[groupID], file)
if timer, exists := mediaGroupHandler.timers[groupID]; exists {
timer.Stop()
}
mediaGroupHandler.timers[groupID] = time.AfterFunc(mediaGroupHandler.timeout, func() {
processMediaGroup(ctx, update, groupID)
})
return dispatcher.EndGroups
}
func processMediaGroup(ctx *ext.Context, update *ext.Update, groupID int64) {
logger := log.FromContext(ctx)
mediaGroupHandler.mu.Lock()
items := mediaGroupHandler.groups[groupID]
delete(mediaGroupHandler.groups, groupID)
delete(mediaGroupHandler.timers, groupID)
mediaGroupHandler.mu.Unlock()
if len(items) == 0 {
logger.Warn("No media items to process for group", "groupID", groupID)
return
}
logger.Debugf("Processing media group %d with %d items", groupID, len(items))
userId := update.GetUserChat().GetID()
msg, err := ctx.Reply(update, ext.ReplyTextString("正在保存文件..."), nil)
if err != nil {
logger.Errorf("Failed to reply: %s", err)
return
}
stor := storage.FromContext(ctx)
if stor != nil {
// In silent mode
if len(items) == 1 {
shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userId, stor, "", items[0], msg.ID)
return
}
shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, userId, stor, "", items, msg.ID)
return
}
stors := storage.GetUserStorages(ctx, userId)
markup, err := msgelem.BuildAddSelectStorageKeyboard(stors, tcbdata.Add{
Files: items,
AsBatch: len(items) > 1,
})
if err != nil {
logger.Errorf("构建存储选择键盘失败: %s", err)
ctx.EditMessage(userId, &tg.MessagesEditMessageRequest{
ID: msg.ID,
Message: "构建存储选择键盘失败: " + err.Error(),
})
return
}
ctx.EditMessage(userId, &tg.MessagesEditMessageRequest{
ID: msg.ID,
Message: fmt.Sprintf("共 %d 个文件, 请选择存储位置", len(items)),
ReplyMarkup: markup,
})
}

View File

@@ -11,7 +11,7 @@ import (
func checkPermission(ctx *ext.Context, update *ext.Update) error {
userID := update.GetUserChat().GetID()
if !slice.Contain(config.Cfg.GetUsersID(), userID) {
if !slice.Contain(config.C().GetUsersID(), userID) {
const noPermissionText string = `
您不在白名单中, 无法使用此 Bot.
您可以部署自己的实例: https://github.com/krau/SaveAny-Bot

View File

@@ -0,0 +1,121 @@
// 处理任意文本消息, 用于通用地从外部源下载文件
package handlers
import (
"errors"
"strings"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/charmbracelet/log"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/parsers"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
"github.com/krau/SaveAny-Bot/storage"
)
func handleTextMessage(ctx *ext.Context, u *ext.Update) error {
logger := log.FromContext(ctx)
text := u.EffectiveMessage.Text
entityUrls := tgutil.ExtractMessageEntityUrls(u.EffectiveMessage.Message)
if len(entityUrls) > 0 {
text += "\n" + strings.Join(entityUrls, "\n")
}
ok, pser := parsers.CanHandle(text)
if !ok {
return dispatcher.EndGroups
}
msg, err := ctx.Reply(u, ext.ReplyTextString("正在解析..."), nil)
if err != nil {
return err
}
item, err := pser.Parse(ctx, text)
if errors.Is(err, parsers.ErrNoParserFound) {
return dispatcher.EndGroups
}
if err != nil {
logger.Error("Failed to parse text", "error", err)
ctx.Reply(u, ext.ReplyTextString("Failed to parse text: "+err.Error()), nil)
return dispatcher.EndGroups
}
logger.Debug("Parsed item from text message", "title", item.Title, "url", item.URL)
userID := u.GetUserChat().GetID()
markup, err := msgelem.BuildAddSelectStorageKeyboard(storage.GetUserStorages(ctx, userID), tcbdata.Add{
TaskType: tasktype.TaskTypeParseditem,
ParsedItem: item,
})
if err != nil {
logger.Errorf("Failed to build storage selection keyboard: %s", err)
ctx.Reply(u, ext.ReplyTextString("Failed to build storage selection keyboard: "+err.Error()), nil)
return dispatcher.EndGroups
}
text, entities, err := msgelem.BuildParsedTextEntity(*item)
if err != nil {
logger.Errorf("Failed to build parsed text entity: %s", err)
ctx.Reply(u, ext.ReplyTextString("Failed to build parsed text entity: "+err.Error()), nil)
return dispatcher.EndGroups
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
Message: text,
ReplyMarkup: markup,
Entities: entities,
ID: msg.ID,
})
return dispatcher.EndGroups
}
func handleSilentSaveText(ctx *ext.Context, u *ext.Update) error {
logger := log.FromContext(ctx)
stor := storage.FromContext(ctx)
if stor == nil {
logger.Warn("Context storage is nil")
ctx.Reply(u, ext.ReplyTextString("未找到存储"), nil)
return dispatcher.EndGroups
}
text := u.EffectiveMessage.Text
if text == "" {
return dispatcher.EndGroups
}
item, err := parsers.ParseWithContext(ctx, text)
if errors.Is(err, parsers.ErrNoParserFound) {
return dispatcher.EndGroups
}
if err != nil {
logger.Error("Failed to parse text", "error", err)
ctx.Reply(u, ext.ReplyTextString("Failed to parse text: "+err.Error()), nil)
return dispatcher.EndGroups
}
logger.Debug("Parsed item from text message", "title", item.Title, "url", item.URL)
userID := u.GetUserChat().GetID()
text, entities, err := msgelem.BuildParsedTextEntity(*item)
if err != nil {
logger.Errorf("Failed to build parsed text entity: %s", err)
ctx.Reply(u, ext.ReplyTextString("Failed to build parsed text entity: "+err.Error()), nil)
return dispatcher.EndGroups
}
msg, err := ctx.SendMessage(userID, &tg.MessagesSendMessageRequest{
Message: text,
Entities: entities,
ReplyTo: &tg.InputReplyToMessage{
ReplyToMsgID: u.EffectiveMessage.ID,
ReplyToPeerID: u.GetUserChat().AsInputPeer(),
},
})
if err != nil {
logger.Errorf("Failed to send message: %s", err)
return dispatcher.EndGroups
}
dirPath := ""
if len(item.Resources) > 1 {
dirPath = fsutil.NormalizePathname(item.Title)
}
return shortcut.CreateAndAddParsedTaskWithEdit(ctx, stor, dirPath, item, msg.ID, userID)
}

View File

@@ -1,14 +1,41 @@
package handlers
import (
"regexp"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/dispatcher/handlers"
"github.com/celestix/gotgproto/dispatcher/handlers/filters"
"github.com/celestix/gotgproto/ext"
sabotfilters "github.com/krau/SaveAny-Bot/client/bot/handlers/utils/filters"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/re"
userclient "github.com/krau/SaveAny-Bot/client/user"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
)
type DescCommandHandler struct {
Cmd string
Desc string
handler func(ctx *ext.Context, u *ext.Update) error
}
var CommandHandlers = []DescCommandHandler{
{"start", "开始使用", handleHelpCmd},
{"silent", "切换静默模式", handleSilentCmd},
{"storage", "设置默认存储端", handleStorageCmd},
{"dir", "管理存储文件夹", handleDirCmd},
{"rule", "管理自动存储规则", handleRuleCmd},
{"watch", "监听聊天(UserBot)", handleWatchCmd},
{"unwatch", "取消监听聊天(UserBot)", handleUnwatchCmd},
{"lswatch", "列出监听的聊天(UserBot)", handleLswatchCmd},
{"save", "保存文件", handleSilentMode(handleSaveCmd, handleSilentSaveReplied)},
{"config", "修改配置", handleConfigCmd},
{"fnametmpl", "设置文件命名模板", handleConfigFnameTmpl},
{"update", "检查更新", handleUpdateCmd},
{"help", "显示帮助", handleHelpCmd},
}
func Register(disp dispatcher.Dispatcher) {
disp.AddHandler(handlers.NewMessage(filters.Message.ChatType(filters.ChatTypeChannel), func(ctx *ext.Context, u *ext.Update) error {
return dispatcher.EndGroups
@@ -17,25 +44,20 @@ func Register(disp dispatcher.Dispatcher) {
return dispatcher.EndGroups
}))
disp.AddHandler(handlers.NewMessage(filters.Message.All, checkPermission))
disp.AddHandler(handlers.NewCommand("start", handleHelpCmd))
disp.AddHandler(handlers.NewCommand("help", handleHelpCmd))
disp.AddHandler(handlers.NewCommand("silent", handleSilentCmd))
disp.AddHandler(handlers.NewCommand("storage", handleStorageCmd))
disp.AddHandler(handlers.NewCommand("dir", handleDirCmd))
disp.AddHandler(handlers.NewCommand("rule", handleRuleCmd))
disp.AddHandler(handlers.NewCommand("save", handleSilentMode(handleSaveCmd, handleSilentSaveReplied)))
for _, info := range CommandHandlers {
disp.AddHandler(handlers.NewCommand(info.Cmd, info.handler))
}
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("update"), handleUpdateCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), handleAddCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), handleSetDefaultCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("cancel"), handleCancelCallback))
linkRegexFilter, err := filters.Message.Regex(re.TgMessageLinkRegexString)
if err != nil {
panic("failed to create regex filter: " + err.Error())
}
disp.AddHandler(handlers.NewMessage(linkRegexFilter, handleSilentMode(handleMessageLink, handleSilentSaveLink)))
telegraphUrlRegexFilter, err := filters.Message.Regex(re.TelegraphUrlRegexString)
if err != nil {
panic("failed to create Telegraph URL regex filter: " + err.Error())
}
disp.AddHandler(handlers.NewMessage(telegraphUrlRegexFilter, handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeCancel), handleCancelCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeConfig), handleConfigCallback))
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TgMessageLinkRegexString)), handleSilentMode(handleMessageLink, handleSilentSaveLink)))
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TelegraphUrlRegexString)), handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
disp.AddHandler(handlers.NewMessage(filters.Message.Media, handleSilentMode(handleMediaMessage, handleSilentSaveMedia)))
disp.AddHandler(handlers.NewMessage(filters.Message.Text, handleSilentMode(handleTextMessage, handleSilentSaveText)))
if config.C().Telegram.Userbot.Enable {
go listenMediaMessageEvent(userclient.GetMediaMessageCh())
}
}

View File

@@ -11,7 +11,7 @@ import (
"github.com/duke-git/lancet/v2/slice"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/rule"
"github.com/krau/SaveAny-Bot/pkg/rule"
)
func handleRuleCmd(ctx *ext.Context, update *ext.Update) error {

View File

@@ -2,6 +2,7 @@ package handlers
import (
"fmt"
"regexp"
"strings"
"github.com/celestix/gotgproto/dispatcher"
@@ -11,8 +12,11 @@ import (
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/strutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
"github.com/krau/SaveAny-Bot/pkg/tfile"
@@ -23,25 +27,34 @@ func handleSaveCmd(ctx *ext.Context, update *ext.Update) error {
logger := log.FromContext(ctx)
args := strings.Split(string(update.EffectiveMessage.Text), " ")
if len(args) >= 3 {
return handleBatchSave(ctx, update, args[1], args[2])
return handleBatchSave(ctx, update, args[1:])
}
replyTo := update.EffectiveMessage.ReplyToMessage
if replyTo == nil || replyTo.Message == nil {
ctx.Reply(update, ext.ReplyTextString(msgelem.SaveHelpText), nil)
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgSaveHelpText)), nil)
return dispatcher.EndGroups
}
genFilename := func() string {
if len(args) > 1 {
return args[1]
}
filename := tgutil.GenFileNameFromMessage(*replyTo.Message)
return filename
}()
option := tfile.WithNameIfEmpty(genFilename)
if len(args) > 1 {
option = tfile.WithName(genFilename)
// genFilename := func() string {
// if len(args) > 1 {
// return args[1]
// }
// filename := tgutil.GenFileNameFromMessage(*replyTo.Message)
// return filename
// }()
// option := tfile.WithNameIfEmpty(genFilename)
// if len(args) > 1 {
// option = tfile.WithName(genFilename)
// }
userDB, err := database.GetUserByChatID(ctx, update.GetUserChat().GetID())
if err != nil {
return err
}
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, replyTo.Message, option)
opts := mediautil.TfileOptions(ctx, userDB, replyTo.Message)
if len(args) > 1 {
// custom filename via command arg
opts = append(opts, tfile.WithName(strings.Join(args[1:], " ")))
}
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, replyTo.Message, opts...)
if err != nil {
return err
}
@@ -60,7 +73,7 @@ func handleSaveCmd(ctx *ext.Context, update *ext.Update) error {
func handleSilentSaveReplied(ctx *ext.Context, update *ext.Update) error {
args := strings.Split(string(update.EffectiveMessage.Text), " ")
if len(args) >= 3 {
return handleBatchSave(ctx, update, args[1], args[2])
return handleBatchSave(ctx, update, args[1:])
}
logger := log.FromContext(ctx)
stor := storage.FromContext(ctx)
@@ -71,28 +84,50 @@ func handleSilentSaveReplied(ctx *ext.Context, update *ext.Update) error {
}
replyTo := update.EffectiveMessage.ReplyToMessage
if replyTo == nil || replyTo.Message == nil {
ctx.Reply(update, ext.ReplyTextString(msgelem.SaveHelpText), nil)
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgSaveHelpText)), nil)
return dispatcher.EndGroups
}
genFilename := func() string {
if len(args) > 1 {
return args[1]
}
filename := tgutil.GenFileNameFromMessage(*replyTo.Message)
return filename
}()
option := tfile.WithNameIfEmpty(genFilename)
if len(args) > 1 {
option = tfile.WithName(genFilename)
// genFilename := func() string {
// if len(args) > 1 {
// return args[1]
// }
// filename := tgutil.GenFileNameFromMessage(*replyTo.Message)
// return filename
// }()
// option := tfile.WithNameIfEmpty(genFilename)
// if len(args) > 1 {
// option = tfile.WithName(genFilename)
// }
userDB, err := database.GetUserByChatID(ctx, update.GetUserChat().GetID())
if err != nil {
return err
}
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, replyTo.Message, option)
opts := mediautil.TfileOptions(ctx, userDB, replyTo.Message)
if len(args) > 1 {
// custom filename via command arg
opts = append(opts, tfile.WithName(strings.Join(args[1:], " ")))
}
msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, replyTo.Message, opts...)
if err != nil {
return err
}
return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, update.GetUserChat().GetID(), stor, "", file, msg.GetID())
}
func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgIdRangeArg string) error {
func handleBatchSave(ctx *ext.Context, update *ext.Update, args []string) error {
chatArg := args[0]
msgIdRangeArg := args[1]
var filterStr string
var filter *regexp.Regexp
if len(args) > 2 {
filterStr = args[2]
var err error
filter, err = regexp.Compile(filterStr)
if err != nil {
ctx.Reply(update, ext.ReplyTextString("无效的正则表达式: "+err.Error()), nil)
return dispatcher.EndGroups
}
}
startID, endID, err := strutil.ParseIntStrRange(msgIdRangeArg, "-")
if err != nil {
ctx.Reply(update, ext.ReplyTextString("无效的消息ID范围: "+err.Error()), nil)
@@ -110,7 +145,7 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
return dispatcher.EndGroups
}
// TODO: generator istead of get all messages
// [TODO]: generator istead of get all messages
msgs, err := tgutil.GetMessagesRange(ctx, chatID, int(startID), int(endID))
if err != nil {
ctx.Reply(update, ext.ReplyTextString("获取消息失败: "+err.Error()), nil)
@@ -121,7 +156,11 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
return dispatcher.EndGroups
}
files := make([]tfile.TGFileMessage, 0, len(msgs))
sb := strings.Builder{}
for _, msg := range msgs {
if msg == nil {
continue
}
media, ok := msg.GetMedia()
if !ok {
continue
@@ -130,11 +169,21 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
if !supported {
continue
}
file, err := tfile.FromMediaMessage(media, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
file, err := tfile.FromMediaMessage(media, ctx.Raw, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
if err != nil {
log.FromContext(ctx).Errorf("获取文件失败: %s", err)
continue
}
if filter != nil {
sb.Reset()
sb.WriteString(msg.GetMessage())
sb.WriteString(" ")
fn, _ := tgutil.GetMediaFileName(media)
sb.WriteString(fn)
if !filter.MatchString(sb.String()) {
continue
}
}
files = append(files, file)
}
if len(files) == 0 {
@@ -164,5 +213,4 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
return dispatcher.EndGroups
}
return shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, update.GetUserChat().GetID(), stor, "", files, replied.ID)
}

View File

@@ -71,6 +71,6 @@ func handleSilentSaveTelegraph(ctx *ext.Context, update *ext.Update) error {
return err
}
userID := update.GetUserChat().GetID()
return shortcut.CreateAndAddTphTaskWithEdit(ctx, userID, result.Page, result.TphDir, result.Pics, stor, msg.ID)
return shortcut.CreateAndAddtelegraphWithEdit(ctx, userID, result.Page, result.TphDir, result.Pics, stor, msg.ID)
}

View File

@@ -0,0 +1,102 @@
package handlers
import (
"errors"
"fmt"
"regexp"
"strings"
"github.com/blang/semver"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/gotd/td/telegram/message/html"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/config"
"github.com/rhysd/go-github-selfupdate/selfupdate"
)
func handleUpdateCmd(ctx *ext.Context, u *ext.Update) error {
currentV, err := semver.Parse(config.Version)
if err != nil {
ctx.Reply(u, ext.ReplyTextString(fmt.Sprintf("You are in dev or the version var failed to inject: %v", err)), nil)
return dispatcher.EndGroups
}
latest, ok, err := selfupdate.DetectLatest(config.GitRepo)
if err != nil {
ctx.Reply(u, ext.ReplyTextString(fmt.Sprintf("检测最新版本失败: %v", err)), nil)
return dispatcher.EndGroups
}
if !ok {
ctx.Reply(u, ext.ReplyTextString("没有找到版本信息"), nil)
return dispatcher.EndGroups
}
if latest.Version.LT(currentV) || latest.Version.Equals(currentV) {
ctx.Reply(u, ext.ReplyTextString(fmt.Sprintf("当前已经是最新版本: %s", config.Version)), nil)
return dispatcher.EndGroups
}
ctx.Sender.To(u.GetUserChat().AsInputPeer()).StyledText(ctx, html.String(nil, func() string {
md := latest.ReleaseNotes
md = regexp.MustCompile(`(?m)^###\s+&nbsp;&nbsp;&nbsp;(.+)$`).ReplaceAllString(md, "<b>$1</b>")
md = regexp.MustCompile(`(?m)^#####\s+&nbsp;&nbsp;&nbsp;&nbsp;(.+)$`).ReplaceAllString(md, "<i>$1</i>")
md = regexp.MustCompile(`(?m)^- `).ReplaceAllString(md, "• ")
md = regexp.MustCompile(`\[\((\w{6,})\)\]\((https?://[^\s)]+)\)`).ReplaceAllString(md, `(<a href="$2">$1</a>)`)
md = regexp.MustCompile(`\[(.+?)\]\((https?://[^\s)]+)\)`).ReplaceAllString(md, `<a href="$2">$1</a>`)
md = strings.ReplaceAll(md, "&nbsp;", " ")
return `<blockquote expandable>` + md + `</blockquote>`
}()))
text := fmt.Sprintf(`发现新版本: %s
当前版本: %s
文件大小: %.2f MB
下载链接: %s
发布时间: %s
升级将重启 Bot , 是否升级?`, latest.Version, config.Version,
float64(latest.AssetByteSize)/(1024*1024), latest.AssetURL,
latest.PublishedAt.Format("2006-01-02 15:04:05"),
)
ctx.Reply(u, ext.ReplyTextString(text), &ext.ReplyOpts{
Markup: &tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
&tg.KeyboardButtonCallback{
Text: "升级",
Data: []byte("update"),
},
},
},
},
},
})
return dispatcher.EndGroups
}
func handleUpdateCallback(ctx *ext.Context, u *ext.Update) error {
currentV, err := semver.Parse(config.Version)
if err != nil {
return err
}
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
ID: u.CallbackQuery.GetMsgID(),
Message: fmt.Sprintf("正在升级中, 当前版本: %s", config.Version),
})
latest, err := selfupdate.UpdateSelf(currentV, config.GitRepo)
if err != nil {
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
ID: u.CallbackQuery.GetMsgID(),
Message: fmt.Sprintf("升级失败: %v", err),
})
return dispatcher.EndGroups
}
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
ID: u.CallbackQuery.GetMsgID(),
Message: fmt.Sprintf("已升级至版本 %s\n若 Bot 未自动重启请手动启动", latest.Version),
})
return errors.New("SAVEANTBOT-RESTART")
}

View File

@@ -0,0 +1,26 @@
package filters
import (
"regexp"
"slices"
"github.com/celestix/gotgproto/dispatcher/handlers/filters"
"github.com/celestix/gotgproto/types"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
func RegexUrl(r *regexp.Regexp) filters.MessageFilter {
return func(m *types.Message) bool {
if m.Text == "" {
return false
}
if r.MatchString(m.Text) {
return true
}
urls := tgutil.ExtractMessageEntityUrls(m.Message)
if len(urls) == 0 {
return false
}
return slices.ContainsFunc(urls, r.MatchString)
}
}

View File

@@ -1,6 +1,20 @@
package mediautil
import "github.com/gotd/td/tg"
import (
"context"
"fmt"
"strings"
"text/template"
"time"
"github.com/charmbracelet/log"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/utils/strutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/fnamest"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
func IsSupported(media tg.MessageMediaClass) bool {
switch media.(type) {
@@ -10,3 +24,118 @@ func IsSupported(media tg.MessageMediaClass) bool {
return false
}
}
type FilenameTemplateData struct {
MsgID string `json:"msgid,omitempty"`
MsgTags string `json:"msgtags,omitempty"`
MsgGen string `json:"msggen,omitempty"`
MsgDate string `json:"msgdate,omitempty"`
OrigName string `json:"origname,omitempty"`
ChatID string `json:"chatid,omitempty"`
}
func (f FilenameTemplateData) ToMap() map[string]string {
return map[string]string{
"msgid": f.MsgID,
"msgtags": f.MsgTags,
"msggen": f.MsgGen,
"msgdate": f.MsgDate,
"origname": f.OrigName,
"chatid": f.ChatID,
}
}
func TfileOptions(ctx context.Context, user *database.User, message *tg.Message) []tfile.TGFileOption {
opts := make([]tfile.TGFileOption, 0)
var fnameOpt tfile.TGFileOption
switch user.FilenameStrategy {
case fnamest.Message.String():
fnameOpt = tfile.WithName(tgutil.GenFileNameFromMessage(*message))
case fnamest.Template.String():
if user.FilenameTemplate == "" {
log.FromContext(ctx).Warnf("empty filename template")
fnameOpt = tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message))
break
}
tmpl, err := template.New("filename").Parse(user.FilenameTemplate)
if err != nil {
log.FromContext(ctx).Errorf("failed to parse filename template: %s", err)
fnameOpt = tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message))
break
}
data := BuildFilenameTemplateData(message)
var sb strings.Builder
err = tmpl.Execute(&sb, data)
if err != nil {
log.FromContext(ctx).Errorf("failed to execute filename template: %s", err)
fnameOpt = tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message))
break
}
fnameOpt = tfile.WithName(sb.String())
default:
fnameOpt = tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message))
}
opts = append(opts, fnameOpt, tfile.WithMessage(message))
return opts
}
func BuildFilenameTemplateData(message *tg.Message) map[string]string {
data := FilenameTemplateData{
MsgID: func() string {
id := message.GetID()
if id == 0 {
return ""
}
return fmt.Sprintf("%d", id)
}(),
MsgTags: func() string {
tags := strutil.ExtractTagsFromText(message.GetMessage())
if len(tags) == 0 {
return ""
}
return strings.Join(tags, "_")
}(),
MsgGen: tgutil.GenFileNameFromMessage(*message),
OrigName: func() string {
f, _ := tgutil.GetMediaFileName(message.Media)
return f
}(),
MsgDate: func() string {
date := message.GetDate()
if date == 0 {
return ""
}
t := time.Unix(int64(date), 0)
return t.Format("2006-01-02_15-04-05")
}(),
ChatID: func() string {
// 如果消息是频道的(从消息链接中fetch的) 直接使用其chat id, 无论它是否是从其他来源转发的
if message.GetPost() {
peer := message.GetPeerID()
switch p := peer.(type) {
case *tg.PeerChannel:
return intToStringOmitZero(p.ChannelID)
default: // impossible case
return intToStringOmitZero(tgutil.ChatIdFromPeer(peer))
}
}
fwdHeader, ok := message.GetFwdFrom()
if !ok {
return intToStringOmitZero(tgutil.ChatIdFromPeer(message.GetPeerID()))
}
fwdFrom, ok := fwdHeader.GetFromID()
if !ok {
return intToStringOmitZero(tgutil.ChatIdFromPeer(message.GetPeerID()))
}
return intToStringOmitZero(tgutil.ChatIdFromPeer(fwdFrom))
}(),
}.ToMap()
return data
}
func intToStringOmitZero(i int64) string {
if i == 0 {
return ""
}
return fmt.Sprintf("%d", i)
}

View File

@@ -0,0 +1,39 @@
package msgelem
import (
"fmt"
"github.com/duke-git/lancet/v2/strutil"
"github.com/gotd/td/telegram/message/entity"
"github.com/gotd/td/telegram/message/styling"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/pkg/parser"
)
func BuildParsedTextEntity(item parser.Item) (string, []tg.MessageEntityClass, error) {
eb := entity.Builder{}
if err := styling.Perform(&eb,
styling.Bold(fmt.Sprintf("[%s]%s", item.Site, item.Title)),
styling.Plain("\n链接: "),
styling.Code(item.URL),
styling.Plain("\n作者: "),
styling.Code(item.Author),
styling.Plain("\n描述: "),
styling.Code(strutil.Ellipsis(item.Description, 233)),
styling.Plain("\n文件数量: "),
styling.Code(fmt.Sprintf("%d", len(item.Resources))),
styling.Plain("\n预计总大小: "),
styling.Code(fmt.Sprintf("%.2f MB", func() float64 {
var totalSize int64
for _, res := range item.Resources {
totalSize += res.Size
}
return float64(totalSize) / 1024 / 1024
}())),
styling.Plain("\n请选择存储位置"),
); err != nil {
return "", nil, fmt.Errorf("构建消息失败: %w", err)
}
text, entities := eb.Complete()
return text, entities, nil
}

View File

@@ -1,15 +0,0 @@
package msgelem
const (
SaveHelpText = `
使用方法:
1. 使用该命令回复要保存的文件, 可选文件名参数.
示例:
/save custom_file_name.mp4
2. 设置默认存储后, 发送 /save <频道ID/用户名> <消息ID范围> 来批量保存文件. 遵从存储规则, 若未匹配到任何规则则使用默认存储.
示例:
/save @moreacg 114-514
`
)

View File

@@ -24,6 +24,8 @@ func BuildAddSelectStorageKeyboard(stors []storage.Storage, adddata tcbdata.Add)
taskType = tasktype.TaskTypeTgfiles
} else if adddata.TphPageNode != nil {
taskType = tasktype.TaskTypeTphpics
} else if adddata.ParsedItem != nil {
taskType = tasktype.TaskTypeParseditem
} else {
return nil, fmt.Errorf("unknown task type: %s", taskType)
}
@@ -41,6 +43,8 @@ func BuildAddSelectStorageKeyboard(stors []storage.Storage, adddata tcbdata.Add)
TphPageNode: adddata.TphPageNode,
TphPics: adddata.TphPics,
TphDirPath: adddata.TphDirPath,
ParsedItem: adddata.ParsedItem,
}
dataid := xid.New().String()
err := cache.Set(dataid, data)

View File

@@ -3,10 +3,10 @@ package ruleutil
import (
"context"
"github.com/duke-git/lancet/v2/convertor"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/consts"
ruleenum "github.com/krau/SaveAny-Bot/pkg/enums/rule"
"github.com/krau/SaveAny-Bot/pkg/rule"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
@@ -33,18 +33,29 @@ func (m matchedStorName) String() string {
return string(m)
}
func (m matchedStorName) IsValid() bool {
return m != "" && m != consts.RuleStorNameChosen
// can we use this storage name directly?
func (m matchedStorName) IsUsable() bool {
return m != "" && m != rule.RuleStorNameChosen
}
func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (matchedStorageName matchedStorName, dirPath string) {
type MatchedDirPath string
func (m MatchedDirPath) String() string {
return string(m)
}
func (m MatchedDirPath) NeedNewForAlbum() bool {
return m != "" && m == rule.RuleDirPathNewForAlbum
}
func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (matched bool, matchedStorageName matchedStorName, dirPath MatchedDirPath) {
if inputs == nil || len(rules) == 0 {
return "", ""
return false, "", ""
}
logger := log.FromContext(ctx)
for _, ur := range rules {
switch ur.Type {
case ruleenum.FileNameRegex.String():
case rule.FileNameRegex.String():
ru, err := rule.NewRuleFileNameRegex(ur.StorageName, ur.DirPath, ur.Data)
if err != nil {
logger.Errorf("Failed to create rule: %s", err)
@@ -56,10 +67,10 @@ func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (m
continue
}
if ok {
dirPath = ru.StoragePath()
dirPath = MatchedDirPath(ru.StoragePath())
matchedStorageName = matchedStorName(ru.StorageName())
}
case ruleenum.MessageRegex.String():
case rule.MessageRegex.String():
ru, err := rule.NewRuleMessageRegex(ur.StorageName, ur.DirPath, ur.Data)
if err != nil {
logger.Errorf("Failed to create rule: %s", err)
@@ -71,10 +82,32 @@ func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (m
continue
}
if ok {
dirPath = ru.StoragePath()
dirPath = MatchedDirPath(ru.StoragePath())
matchedStorageName = matchedStorName(ru.StorageName())
}
case rule.IsAlbum.String():
matchAlbum, err := convertor.ToBool(ur.Data)
if err != nil {
matchAlbum = false
}
ru, err := rule.NewRuleMediaType(ur.StorageName, ur.DirPath, matchAlbum)
if err != nil {
logger.Errorf("Failed to create rule: %s", err)
continue
}
ok, err := ru.Match(inputs.File.Message().GroupedID != 0)
if err != nil {
logger.Errorf("Failed to match rule: %s", err)
continue
}
if ok {
dirPath = MatchedDirPath(ru.StoragePath())
matchedStorageName = matchedStorName(ru.StorageName())
}
}
}
return
if matchedStorageName != "" || dirPath != "" {
return true, matchedStorageName, dirPath
}
return false, "", ""
}

View File

@@ -10,27 +10,30 @@ import (
"github.com/celestix/gotgproto/ext"
"github.com/celestix/gotgproto/types"
"github.com/charmbracelet/log"
"github.com/gotd/td/telegram/downloader"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/re"
uc "github.com/krau/SaveAny-Bot/client/user"
"github.com/krau/SaveAny-Bot/common/cache"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/common/utils/tphutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
// 获取消息中的文件并回复等待消息, 返回等待消息, 获取到的文件
func GetFileFromMessageWithReply(ctx *ext.Context, update *ext.Update, message *tg.Message, tfileopts ...tfile.TGFileOptions) (replied *types.Message,
func GetFileFromMessageWithReply(ctx *ext.Context, update *ext.Update, message *tg.Message, tfileopts ...tfile.TGFileOption) (replied *types.Message,
file tfile.TGFileMessage, err error,
) {
logger := log.FromContext(ctx)
media := message.Media
supported := mediautil.IsSupported(media)
if !supported {
ctx.Reply(update, ext.ReplyTextString("不支持的消息类型"), nil)
return nil, nil, dispatcher.EndGroups
return nil, nil, dispatcher.ContinueGroups
}
replied, err = ctx.Reply(update, ext.ReplyTextString("正在获取文件信息..."), nil)
@@ -38,15 +41,15 @@ func GetFileFromMessageWithReply(ctx *ext.Context, update *ext.Update, message *
logger.Errorf("Failed to reply: %s", err)
return nil, nil, dispatcher.EndGroups
}
options := []tfile.TGFileOptions{
tfile.WithMessage(message),
}
if len(tfileopts) > 0 {
options = append(options, tfileopts...)
} else {
options = append(options, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message)))
}
file, err = tfile.FromMediaMessage(media, message, options...)
// options := []tfile.TGFileOption{
// tfile.WithMessage(message),
// }
// if len(tfileopts) > 0 {
// options = append(options, tfileopts...)
// } else {
// options = append(options, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message)))
// }
file, err = tfile.FromMediaMessage(media, ctx.Raw, message, tfileopts...)
if err != nil {
logger.Errorf("Failed to get file from media: %s", err)
ctx.Reply(update, ext.ReplyTextString("获取文件失败: "+err.Error()), nil)
@@ -60,7 +63,7 @@ type EditMessageFunc func(text string, markup tg.ReplyMarkupClass)
// 获取链接中的文件并回复等待消息
func GetFilesFromUpdateLinkMessageWithReplyEdit(ctx *ext.Context, update *ext.Update) (replied *types.Message, files []tfile.TGFileMessage, editReplied EditMessageFunc, err error) {
logger := log.FromContext(ctx)
msgLinks := re.TgMessageLinkRegexp.FindAllString(update.EffectiveMessage.GetMessage(), -1)
msgLinks := re.TgMessageLinkRegexp.FindAllString(tgutil.ExtractMessageEntityUrlsText(update.EffectiveMessage.Message), -1)
if len(msgLinks) == 0 {
logger.Warn("no matched message links but called handleMessageLink")
return nil, nil, nil, dispatcher.EndGroups
@@ -79,31 +82,74 @@ func GetFilesFromUpdateLinkMessageWithReplyEdit(ctx *ext.Context, update *ext.Up
logger.Errorf("failed to edit message: %s", err)
}
}
user, err := database.GetUserByChatID(ctx, update.GetUserChat().GetID())
if err != nil {
logger.Errorf("failed to get user from db: %s", err)
editReplied("获取用户信息失败: "+err.Error(), nil)
return nil, nil, nil, dispatcher.EndGroups
}
files = make([]tfile.TGFileMessage, 0, len(msgLinks))
for _, link := range msgLinks {
chatId, msgId, err := tgutil.ParseMessageLink(ctx, link)
if err != nil {
logger.Errorf("failed to parse message link %s: %s", link, err)
continue
}
msg, err := tgutil.GetMessageByID(ctx, chatId, msgId)
if err != nil {
logger.Errorf("failed to get message by ID: %s", err)
continue
addFile := func(client downloader.Client, msg *tg.Message) {
if msg == nil || msg.Media == nil {
logger.Warn("message is nil, skipping")
return
}
media, ok := msg.GetMedia()
if !ok {
logger.Debugf("message %d has no media", msg.GetID())
continue
return
}
file, err := tfile.FromMediaMessage(media, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
// var opt tfile.TGFileOption
// switch user.FilenameStrategy {
// case fnamest.Message.String():
// opt = tfile.WithName(tgutil.GenFileNameFromMessage(*msg))
// default:
// opt = tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg))
// }
opts := mediautil.TfileOptions(ctx, user, msg)
file, err := tfile.FromMediaMessage(media, client, msg, opts...)
if err != nil {
logger.Errorf("failed to create file from media: %s", err)
continue
return
}
files = append(files, file)
}
tctx := ctx
if config.C().Telegram.Userbot.Enable {
tctx = uc.GetCtx()
}
for _, link := range msgLinks {
linkUrl, err := url.Parse(link)
if err != nil {
logger.Errorf("failed to parse message link %s: %s", link, err)
continue
}
chatId, msgId, err := tgutil.ParseMessageLink(tctx, link)
if err != nil {
logger.Errorf("failed to parse message link %s: %s", link, err)
continue
}
msg, err := tgutil.GetMessageByID(tctx, chatId, msgId)
if err != nil {
logger.Errorf("failed to get message by ID: %s", err)
continue
}
groupID, isGroup := msg.GetGroupedID()
if isGroup && groupID != 0 && !linkUrl.Query().Has("single") {
gmsgs, err := tgutil.GetGroupedMessages(ctx, chatId, msg)
if err != nil {
logger.Errorf("failed to get grouped messages: %s", err)
} else {
for _, gmsg := range gmsgs {
addFile(tctx.Raw, gmsg)
}
}
} else {
addFile(tctx.Raw, msg)
}
}
if len(files) == 0 {
editReplied("没有找到可保存的文件", nil)
return nil, nil, nil, dispatcher.EndGroups
@@ -132,7 +178,7 @@ type TelegraphResult struct {
// return replied message, image urls, telegraph path(unescaped), error
func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*types.Message, *TelegraphResult, error) {
logger := log.FromContext(ctx)
tphurl := re.TelegraphUrlRegexp.FindString(update.EffectiveMessage.GetMessage()) // TODO: batch urls
tphurl := re.TelegraphUrlRegexp.FindString(tgutil.ExtractMessageEntityUrlsText(update.EffectiveMessage.Message))
if tphurl == "" {
logger.Warnf("No telegraph url found but called handleTelegraph")
return nil, nil, dispatcher.ContinueGroups
@@ -144,11 +190,13 @@ func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*type
ctx.Reply(update, ext.ReplyTextString("解析 telegraph 路径失败: "+err.Error()), nil)
return nil, nil, dispatcher.EndGroups
}
tphdir = strings.TrimSpace(tphdir)
msg, err := ctx.Reply(update, ext.ReplyTextString("正在获取 telegraph 页面..."), nil)
if err != nil {
logger.Errorf("Failed to reply to update: %s", err)
return nil, nil, dispatcher.EndGroups
}
logger.Debugf("Fetching telegraph page: %s", pagepath)
page, err := tphutil.DefaultClient().GetPage(ctx, pagepath)
if err != nil {
logger.Errorf("Failed to get telegraph page: %s", err)
@@ -176,6 +224,10 @@ func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*type
}
if node.Tag == "img" {
if src, ok := node.Attrs["src"]; ok {
if strings.HasPrefix(src, "/file/") {
// handle images on telegra.ph server
src = "https://telegra.ph" + src
}
imgs = append(imgs, src)
}
}

View File

@@ -0,0 +1,35 @@
package shortcut
import (
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/charmbracelet/log"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/core/tasks/parsed"
"github.com/krau/SaveAny-Bot/pkg/parser"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
func CreateAndAddParsedTaskWithEdit(ctx *ext.Context, stor storage.Storage, dirPath string, item *parser.Item, msgID int, userID int64) error {
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
task := parsed.NewTask(xid.New().String(), injectCtx, stor, stor.JoinStoragePath(dirPath), item, parsed.NewProgress(msgID, userID))
if err := core.AddTask(injectCtx, task); err != nil {
log.FromContext(ctx).Errorf("Failed to add task: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: msgID,
Message: "任务添加失败: " + err.Error(),
})
return dispatcher.EndGroups
}
text, entities := msgelem.BuildTaskAddedEntities(ctx, item.Title, core.GetLength(ctx))
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: msgID,
Message: text,
Entities: entities,
})
return dispatcher.EndGroups
}

View File

@@ -3,6 +3,7 @@ package shortcut
import (
"fmt"
"path"
"strings"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
@@ -12,15 +13,15 @@ import (
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/ruleutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/core/batchtftask"
"github.com/krau/SaveAny-Bot/core/tftask"
"github.com/krau/SaveAny-Bot/core/tasks/batchtfile"
tftask "github.com/krau/SaveAny-Bot/core/tasks/tfile"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
// 创建一个 tftask.TGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
// 创建一个 tfile.TGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, file tfile.TGFileMessage, trackMsgID int) error {
logger := log.FromContext(ctx)
user, err := database.GetUserByChatID(ctx, userID)
@@ -33,9 +34,14 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
return dispatcher.EndGroups
}
if user.ApplyRule && user.Rules != nil {
matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
dirPath = matchedDirPath
if matchedStorageName.IsValid() {
matched, matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
if !matched {
goto startCreateTask
}
if matchedDirPath != "" {
dirPath = matchedDirPath.String()
}
if matchedStorageName.IsUsable() {
stor, err = storage.GetStorageByUserIDAndName(ctx, user.ChatID, matchedStorageName.String())
if err != nil {
logger.Errorf("Failed to get storage by user ID and name: %s", err)
@@ -47,11 +53,11 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
}
}
}
startCreateTask:
storagePath := stor.JoinStoragePath(path.Join(dirPath, file.Name()))
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
taskid := xid.New().String()
task, err := tftask.NewTGFileTask(taskid, injectCtx, file, ctx.Raw, stor, storagePath,
task, err := tftask.NewTGFileTask(taskid, injectCtx, file, stor, storagePath,
tftask.NewProgressTrack(
trackMsgID,
userID))
@@ -81,7 +87,7 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
return dispatcher.EndGroups
}
// 创建一个 batchtftask.BatchTGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
// 创建一个 batchtfile.BatchTGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, files []tfile.TGFileMessage, trackMsgID int) error {
logger := log.FromContext(ctx)
user, err := database.GetUserByChatID(ctx, userID)
@@ -93,19 +99,30 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
})
return dispatcher.EndGroups
}
useRule := user.ApplyRule && user.Rules != nil
applyRule := func(file tfile.TGFileMessage) (string, string) {
applyRule := func(file tfile.TGFileMessage) (string, ruleutil.MatchedDirPath) {
if !useRule {
return stor.Name(), dirPath
return stor.Name(), ruleutil.MatchedDirPath(dirPath)
}
storName, dirP := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
if !storName.IsValid() {
return stor.Name(), dirP
matched, storName, dirP := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
if !matched {
return stor.Name(), ruleutil.MatchedDirPath(dirPath)
}
return storName.String(), dirP
storname := storName.String()
if !storName.IsUsable() {
storname = stor.Name()
}
return storname, dirP
}
elems := make([]batchtftask.TaskElement, 0, len(files))
elems := make([]batchtfile.TaskElement, 0, len(files))
type albumFile struct {
file tfile.TGFileMessage
storage storage.Storage
}
albumFiles := make(map[int64][]albumFile, 0)
for _, file := range files {
storName, dirPath := applyRule(file)
fileStor := stor
@@ -120,21 +137,59 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
return dispatcher.EndGroups
}
}
storPath := fileStor.JoinStoragePath(path.Join(dirPath, file.Name()))
elem, err := batchtftask.NewTaskElement(fileStor, storPath, file)
if err != nil {
logger.Errorf("Failed to create task element: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: "任务创建失败: " + err.Error(),
if !dirPath.NeedNewForAlbum() {
storPath := fileStor.JoinStoragePath(path.Join(dirPath.String(), file.Name()))
elem, err := batchtfile.NewTaskElement(fileStor, storPath, file)
if err != nil {
logger.Errorf("Failed to create task element: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: "任务创建失败: " + err.Error(),
})
return dispatcher.EndGroups
}
elems = append(elems, *elem)
} else {
groupId, isGroup := file.Message().GetGroupedID()
if !isGroup || groupId == 0 {
logger.Warnf("File %s is not in a group, skipping album handling", file.Name())
continue
}
if _, ok := albumFiles[groupId]; !ok {
albumFiles[groupId] = make([]albumFile, 0)
}
albumFiles[groupId] = append(albumFiles[groupId], albumFile{
file: file,
storage: fileStor,
})
return dispatcher.EndGroups
}
elems = append(elems, *elem)
}
for _, afiles := range albumFiles {
if len(afiles) <= 1 {
continue
}
// 对于需要新建目录的文件, 将第一个文件的文件名(去除扩展名)作为目录名
// 存储以第一个文件的存储为准
albumDir := strings.TrimSuffix(path.Base(afiles[0].file.Name()), path.Ext(afiles[0].file.Name()))
albumStor := afiles[0].storage
for _, af := range afiles {
afstorPath := af.storage.JoinStoragePath(path.Join(dirPath, albumDir, af.file.Name()))
elem, err := batchtfile.NewTaskElement(albumStor, afstorPath, af.file)
if err != nil {
logger.Errorf("Failed to create task element for album file: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: "任务创建失败: " + err.Error(),
})
return dispatcher.EndGroups
}
elems = append(elems, *elem)
}
}
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
taskid := xid.New().String()
task := batchtftask.NewBatchTGFileTask(taskid, injectCtx, elems, ctx.Raw, batchtftask.NewProgressTracker(trackMsgID, userID), true)
task := batchtfile.NewBatchTGFileTask(taskid, injectCtx, elems, batchtfile.NewProgressTracker(trackMsgID, userID), true)
if err := core.AddTask(injectCtx, task); err != nil {
logger.Errorf("Failed to add batch task: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{

View File

@@ -9,19 +9,21 @@ import (
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/common/utils/tphutil"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/core/tphtask"
tphtask "github.com/krau/SaveAny-Bot/core/tasks/telegraph"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
func CreateAndAddTphTaskWithEdit(ctx *ext.Context,
func CreateAndAddtelegraphWithEdit(
ctx *ext.Context,
userID int64,
tphpage *telegraph.Page,
dirPath string, // unescaped ph path for file storage
pics []string,
stor storage.Storage,
trackMsgID int) error {
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
task := tphtask.NewTask(xid.New().String(),
injectCtx,

View File

@@ -0,0 +1,255 @@
package handlers
import (
"fmt"
"path"
"regexp"
"strings"
"text/template"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/ruleutil"
userclient "github.com/krau/SaveAny-Bot/client/user"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/core/tasks/tfile"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/fnamest"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
func handleWatchCmd(ctx *ext.Context, update *ext.Update) error {
logger := log.FromContext(ctx)
args := strings.Split(update.EffectiveMessage.Text, " ")
if len(args) < 2 {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgWatchHelpText)), nil)
return dispatcher.EndGroups
}
userChatID := update.GetUserChat().GetID()
user, err := database.GetUserByChatID(ctx, userChatID)
if err != nil {
logger.Errorf("获取用户失败: %s", err)
ctx.Reply(update, ext.ReplyTextString("获取用户失败"), nil)
return dispatcher.EndGroups
}
if user.DefaultStorage == "" {
ctx.Reply(update, ext.ReplyTextString("请先设置默认存储, 使用 /storage 命令"), nil)
return dispatcher.EndGroups
}
chatArg := args[1]
chatID, err := tgutil.ParseChatID(ctx, chatArg)
if err != nil {
ctx.Reply(update, ext.ReplyTextString("无效的ID或用户名: "+err.Error()), nil)
return dispatcher.EndGroups
}
watching, err := user.WatchingChat(ctx, chatID)
if err != nil {
logger.Errorf("Failed to check if user is watching chat %d: %s", chatID, err)
return dispatcher.EndGroups
}
if watching {
ctx.Reply(update, ext.ReplyTextString("已经在监听此聊天"), nil)
return dispatcher.EndGroups
}
filter := ""
if len(args) > 2 {
filterArg := strings.Join(args[2:], " ")
filterType := strings.Split(filterArg, ":")[0]
filterData := strings.Split(filterArg, ":")[1]
if filterType == "" || filterData == "" {
ctx.Reply(update, ext.ReplyTextString("过滤器格式错误, 请使用 <过滤器类型>:<表达式>"), nil)
return dispatcher.EndGroups
}
switch filterType {
case "msgre":
_, err := regexp.Compile(filterData)
if err != nil {
ctx.Reply(update, ext.ReplyTextString("正则表达式格式错误: "+err.Error()), nil)
return dispatcher.EndGroups
}
filter = filterType + ":" + filterData
default:
ctx.Reply(update, ext.ReplyTextString("不支持的过滤器类型, 请参阅文档"), nil)
return dispatcher.EndGroups
}
}
if err := user.WatchChat(ctx, database.WatchChat{
UserID: user.ID,
ChatID: chatID,
Filter: filter,
}); err != nil {
logger.Errorf("Failed to watch chat %d: %s", chatID, err)
ctx.Reply(update, ext.ReplyTextString("监听聊天失败: "+err.Error()), nil)
return dispatcher.EndGroups
}
ctx.Reply(update, ext.ReplyTextString("已开始监听聊天: "+chatArg), nil)
return dispatcher.EndGroups
}
func handleLswatchCmd(ctx *ext.Context, update *ext.Update) error {
logger := log.FromContext(ctx)
userChatID := update.GetUserChat().GetID()
user, err := database.GetUserByChatID(ctx, userChatID)
if err != nil {
logger.Errorf("获取用户失败: %s", err)
ctx.Reply(update, ext.ReplyTextString("获取用户失败"), nil)
return dispatcher.EndGroups
}
chats := user.WatchChats
if len(chats) == 0 {
ctx.Reply(update, ext.ReplyTextString("当前没有监听任何聊天"), nil)
return dispatcher.EndGroups
}
var sb strings.Builder
sb.WriteString("当前监听的聊天:\n")
for _, chat := range chats {
sb.WriteString("- ")
sb.WriteString(fmt.Sprintf("%d", chat.ChatID))
if chat.Filter != "" {
sb.WriteString(" (过滤器: ")
sb.WriteString(chat.Filter)
sb.WriteString(")")
}
sb.WriteString("\n")
}
ctx.Reply(update, ext.ReplyTextString(sb.String()), nil)
return dispatcher.EndGroups
}
func handleUnwatchCmd(ctx *ext.Context, update *ext.Update) error {
logger := log.FromContext(ctx)
args := strings.Split(update.EffectiveMessage.Text, " ")
if len(args) < 2 {
ctx.Reply(update, ext.ReplyTextString("请提供要取消监听的聊天ID或用户名"), nil)
return dispatcher.EndGroups
}
userChatID := update.GetUserChat().GetID()
user, err := database.GetUserByChatID(ctx, userChatID)
if err != nil {
logger.Errorf("获取用户失败: %s", err)
ctx.Reply(update, ext.ReplyTextString("获取用户失败"), nil)
return dispatcher.EndGroups
}
chatArg := args[1]
chatID, err := tgutil.ParseChatID(ctx, chatArg)
if err != nil {
ctx.Reply(update, ext.ReplyTextString("无效的ID或用户名: "+err.Error()), nil)
return dispatcher.EndGroups
}
if err := user.UnwatchChat(ctx, chatID); err != nil {
logger.Errorf("Failed to unwatch chat %d: %s", chatID, err)
ctx.Reply(update, ext.ReplyTextString("取消监听聊天失败: "+err.Error()), nil)
return dispatcher.EndGroups
}
ctx.Reply(update, ext.ReplyTextString("已取消监听聊天: "+chatArg), nil)
return dispatcher.EndGroups
}
func listenMediaMessageEvent(ch chan userclient.MediaMessageEvent) {
logger := log.FromContext(userclient.GetCtx())
for event := range ch {
logger.Debug("Received media message event", "chat_id", event.ChatID, "file_name", event.File.Name())
ctx := event.Ctx
file := event.File
chats, err := database.GetWatchChatsByChatID(ctx, event.ChatID)
if err != nil {
logger.Errorf("Failed to get watch chats for chat ID %d: %v", event.ChatID, err)
continue
}
msgText := event.File.Message().GetMessage()
for _, chat := range chats {
if chat.Filter != "" {
filter := strings.Split(chat.Filter, ":")
if len(filter) != 2 {
logger.Warnf("Invalid filter format in chat %d, skipping", chat.ChatID)
continue
}
filterType := filter[0]
filterData := filter[1]
switch filterType {
case "msgre": // [TODO] enums for filter types
if ok, err := regexp.MatchString(filterData, msgText); err != nil {
continue
} else if !ok {
continue
}
default:
logger.Warnf("Unsupported filter type %s in chat %d, skipping", filterType, chat.ChatID)
continue
}
}
user, err := database.GetUserByID(ctx, chat.UserID)
if err != nil {
logger.Errorf("Failed to get user by ID %d: %v", chat.UserID, err)
continue
}
if user.DefaultStorage == "" {
logger.Warnf("User %d has no default storage set, skipping media message handling", chat.UserID)
continue
}
stor, err := storage.GetStorageByUserIDAndName(ctx, user.ChatID, user.DefaultStorage)
if err != nil {
logger.Errorf("Failed to get storage by user ID %d and name %s: %v", user.ChatID, user.DefaultStorage, err)
continue
}
switch user.FilenameStrategy {
case fnamest.Message.String():
file.SetName(tgutil.GenFileNameFromMessage(*file.Message()))
case fnamest.Template.String():
if user.FilenameTemplate == "" {
logger.Warnf("Empty filename template for user %d, using default filename", user.ChatID)
break
}
message := file.Message()
tmpl, err := template.New("filename").Parse(user.FilenameTemplate)
if err != nil {
logger.Errorf("Failed to parse filename template for user %d: %s", user.ChatID, err)
break
}
data := mediautil.BuildFilenameTemplateData(message)
var sb strings.Builder
err = tmpl.Execute(&sb, data)
if err != nil {
log.FromContext(ctx).Errorf("failed to execute filename template: %s", err)
break
}
file.SetName(sb.String())
}
var dirPath string
if user.ApplyRule && user.Rules != nil {
matched, matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
if !matched {
goto startCreateTask
}
dirPath = matchedDirPath.String()
if matchedStorageName.IsUsable() {
stor, err = storage.GetStorageByUserIDAndName(ctx, user.ChatID, matchedStorageName.String())
if err != nil {
logger.Errorf("Failed to get storage by user ID and name: %s", err)
continue
}
}
}
startCreateTask:
storagePath := stor.JoinStoragePath(path.Join(dirPath, file.Name()))
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
taskid := xid.New().String()
task, err := tfile.NewTGFileTask(taskid, injectCtx, file, stor, storagePath, nil)
if err != nil {
logger.Errorf("create task failed: %s", err)
continue
}
if err := core.AddTask(injectCtx, task); err != nil {
logger.Errorf("add task failed: %s", err)
continue
}
logger.Infof("Added media message task for user %d in chat %d: %s", chat.UserID, event.ChatID, file.Name())
}
}
}

View File

@@ -9,13 +9,14 @@ import (
"github.com/gotd/td/telegram"
"github.com/krau/SaveAny-Bot/client/middleware/recovery"
"github.com/krau/SaveAny-Bot/client/middleware/retry"
"github.com/krau/SaveAny-Bot/config"
)
// https://github.com/iyear/tdl/blob/master/core/tclient/tclient.go
func NewDefaultMiddlewares(ctx context.Context, timeout time.Duration) []telegram.Middleware {
return []telegram.Middleware{
recovery.New(ctx, newBackoff(timeout)),
retry.New(5),
retry.New(config.C().Telegram.RpcRetry),
floodwait.NewSimpleWaiter(),
}
}

View File

@@ -2,11 +2,11 @@ package recovery
import (
"context"
"fmt"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/charmbracelet/log"
"github.com/go-faster/errors"
"github.com/gotd/td/bin"
"github.com/gotd/td/telegram"
"github.com/gotd/td/tg"
@@ -31,7 +31,7 @@ func (r *recovery) Handle(next tg.Invoker) telegram.InvokeFunc {
return backoff.RetryNotify(func() error {
if err := next.Invoke(ctx, input, output); err != nil {
if r.shouldRecover(ctx, err) {
return errors.Wrap(err, "recover")
return fmt.Errorf("recovery: %w", err)
}
return backoff.Permanent(err)

View File

@@ -5,7 +5,6 @@ import (
"fmt"
"github.com/charmbracelet/log"
"github.com/go-faster/errors"
"github.com/gotd/td/bin"
"github.com/gotd/td/telegram"
"github.com/gotd/td/tg"
@@ -37,7 +36,8 @@ func (r retry) Handle(next tg.Invoker) telegram.InvokeFunc {
retries++
continue
}
return errors.Wrap(err, "retry middleware skip")
// retry middleware skip
return err
}
return nil

View File

@@ -9,9 +9,9 @@ import (
"github.com/fatih/color"
)
type termialAuthConversator struct{}
type terminalAuthConversator struct{}
func (t *termialAuthConversator) AskPhoneNumber() (string, error) {
func (t *terminalAuthConversator) AskPhoneNumber() (string, error) {
phone := ""
err := huh.NewInput().Title("Your Phone Number").
Placeholder("+44 123456").
@@ -29,7 +29,7 @@ func (t *termialAuthConversator) AskPhoneNumber() (string, error) {
return strings.TrimSpace(phone), nil
}
func (t *termialAuthConversator) AskCode() (string, error) {
func (t *terminalAuthConversator) AskCode() (string, error) {
code := ""
err := huh.NewInput().Title("Your Code").
Placeholder("123456").
@@ -45,7 +45,7 @@ func (t *termialAuthConversator) AskCode() (string, error) {
return strings.TrimSpace(code), nil
}
func (t *termialAuthConversator) AskPassword() (string, error) {
func (t *terminalAuthConversator) AskPassword() (string, error) {
pwd := ""
err := huh.NewInput().Title("Your 2FA Password").
@@ -61,7 +61,7 @@ func (t *termialAuthConversator) AskPassword() (string, error) {
return strings.TrimSpace(pwd), nil
}
func (t *termialAuthConversator) AuthStatus(authStatus gotgproto.AuthStatus) {
func (t *terminalAuthConversator) AuthStatus(authStatus gotgproto.AuthStatus) {
switch authStatus.Event {
case gotgproto.AuthStatusPhoneRetrial:
color.Red("The phone number you just entered seems to be incorrect,")

View File

@@ -5,46 +5,87 @@ import (
"time"
"github.com/celestix/gotgproto"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/dispatcher/handlers"
"github.com/celestix/gotgproto/dispatcher/handlers/filters"
"github.com/celestix/gotgproto/ext"
"github.com/celestix/gotgproto/sessionMaker"
"github.com/charmbracelet/log"
"github.com/gotd/td/telegram/dcs"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/middleware"
"github.com/krau/SaveAny-Bot/common/utils/netutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/database"
"github.com/ncruces/go-sqlite3/gormlite"
"golang.org/x/net/proxy"
)
var UC *gotgproto.Client
var uc *gotgproto.Client
var ectx *ext.Context
func GetCtx() *ext.Context {
if uc == nil {
panic("User client is not initialized, please call Login first")
}
if ectx != nil {
// UC.RefreshContext(ectx)
return ectx
}
ectx = UC.CreateContext()
ectx = uc.CreateContext()
return ectx
}
func GetClient() *gotgproto.Client {
if uc == nil {
panic("User client is not initialized, please call Login first")
}
return uc
}
func Login(ctx context.Context) (*gotgproto.Client, error) {
log.FromContext(ctx).Debug("Logging in as user client")
if UC != nil {
return UC, nil
log.FromContext(ctx).Debug("Logging in user client")
if uc != nil {
return uc, nil
}
res := make(chan struct {
client *gotgproto.Client
err error
})
go func() {
var resolver dcs.Resolver
if config.C().Telegram.Proxy.Enable && config.C().Telegram.Proxy.URL != "" {
dialer, err := netutil.NewProxyDialer(config.C().Telegram.Proxy.URL)
if err != nil {
res <- struct {
client *gotgproto.Client
err error
}{nil, err}
return
}
resolver = dcs.Plain(dcs.PlainOptions{
Dial: dialer.(proxy.ContextDialer).DialContext,
})
} else {
resolver = dcs.DefaultResolver()
}
tclient, err := gotgproto.NewClient(
config.Cfg.Telegram.AppID,
config.Cfg.Telegram.AppHash,
config.C().Telegram.AppID,
config.C().Telegram.AppHash,
gotgproto.ClientTypePhone(""),
&gotgproto.ClientOpts{
Session: sessionMaker.SqlSession(gormlite.Open(config.Cfg.Telegram.Userbot.Session)),
AuthConversator: &termialAuthConversator{},
Session: sessionMaker.SqlSession(gormlite.Open(config.C().Telegram.Userbot.Session)),
AuthConversator: &terminalAuthConversator{},
Context: ctx,
DisableCopyright: true,
Resolver: resolver,
MaxRetries: config.C().Telegram.RpcRetry,
AutoFetchReply: true,
Middlewares: middleware.NewDefaultMiddlewares(ctx, 5*time.Minute),
ErrorHandler: func(ctx *ext.Context, u *ext.Update, s string) error {
log.FromContext(ctx).Errorf("Unhandled error: %s", s)
return dispatcher.EndGroups
},
},
)
if err != nil {
@@ -69,7 +110,21 @@ func Login(ctx context.Context) (*gotgproto.Client, error) {
if r.err != nil {
return nil, r.err
}
UC = r.client
return UC, nil
uc = r.client
uc.Dispatcher.AddHandler(handlers.NewMessage(filters.Message.Media, func(ctx *ext.Context, u *ext.Update) error {
switch u.UpdateClass.(type) {
case *tg.UpdateEditChannelMessage, *tg.UpdateEditMessage, *tg.UpdateDeleteChannelMessages, *tg.UpdateDeleteMessages:
return dispatcher.EndGroups
}
chatId := u.EffectiveChat().GetID()
watchChats, err := database.GetWatchChatsByChatID(ctx, chatId)
if err != nil || len(watchChats) == 0 {
return dispatcher.EndGroups
}
return dispatcher.ContinueGroups
}))
uc.Dispatcher.AddHandler(handlers.NewMessage(filters.Message.Media, handleMediaMessage))
log.FromContext(ctx).Infof("User client logged in successfully: %s", uc.Self.FirstName+" "+uc.Self.LastName)
return uc, nil
}
}

100
client/user/watch.go Normal file
View File

@@ -0,0 +1,100 @@
package user
import (
"sync"
"time"
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
type MediaMessageEvent struct {
Ctx *ext.Context
ChatID int64 // from witch the media message was sent
MessageID int
File tfile.TGFileMessage
}
type messageKey struct {
ChatID int64
MessageID int
}
type MediaMessageHandler struct {
events map[messageKey]MediaMessageEvent
timers map[messageKey]*time.Timer
mu sync.Mutex
debounce time.Duration
}
var (
mediaMessageCh = make(chan MediaMessageEvent, 100)
mediaMessageHandler = &MediaMessageHandler{
events: make(map[messageKey]MediaMessageEvent),
timers: make(map[messageKey]*time.Timer),
debounce: 5 * time.Second,
}
)
func GetMediaMessageCh() chan MediaMessageEvent {
return mediaMessageCh
}
func sendMediaMessageEvent(event MediaMessageEvent) {
key := messageKey{ChatID: event.ChatID, MessageID: event.MessageID}
mediaMessageHandler.mu.Lock()
defer mediaMessageHandler.mu.Unlock()
if timer, exists := mediaMessageHandler.timers[key]; exists {
timer.Stop()
} else {
mediaMessageHandler.events[key] = event
}
mediaMessageHandler.timers[key] = time.AfterFunc(mediaMessageHandler.debounce, func() {
mediaMessageHandler.mu.Lock()
event := mediaMessageHandler.events[key]
delete(mediaMessageHandler.events, key)
delete(mediaMessageHandler.timers, key)
mediaMessageHandler.mu.Unlock()
mediaMessageCh <- event
})
}
func handleMediaMessage(ctx *ext.Context, update *ext.Update) error {
message := update.EffectiveMessage
media, ok := message.GetMedia()
if !ok || media == nil {
return dispatcher.EndGroups
}
support := func() bool {
switch media.(type) {
case *tg.MessageMediaDocument, *tg.MessageMediaPhoto:
return true
default:
return false
}
}()
if !support {
return dispatcher.EndGroups
}
file, err := tfile.FromMediaMessage(media, ctx.Raw, message.Message, tfile.WithNameIfEmpty(
tgutil.GenFileNameFromMessage(*message.Message),
))
if err != nil {
return err
}
chatId := update.EffectiveChat().GetID()
sendMediaMessageEvent(MediaMessageEvent{
Ctx: ctx,
ChatID: chatId,
MessageID: message.ID,
File: file,
})
return dispatcher.EndGroups
}

View File

@@ -8,9 +8,10 @@ import (
"io/fs"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"github.com/goccy/go-yaml"
)
func main() {
@@ -20,28 +21,27 @@ func main() {
flag.Parse()
keys := make(map[string]struct{})
re := regexp.MustCompile(`^\s*\[+\s*([^\]\[]+)\s*\]+`)
err := filepath.WalkDir(*dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(d.Name(), ".toml") {
if d.IsDir() || !(strings.HasSuffix(d.Name(), ".yaml") || strings.HasSuffix(d.Name(), ".yml")) {
return nil
}
f, err := os.Open(path)
data, err := os.ReadFile(path)
if err != nil {
return err
}
defer f.Close()
s := bufio.NewScanner(f)
for s.Scan() {
if m := re.FindStringSubmatch(s.Text()); m != nil {
keys[m[1]] = struct{}{}
}
var content map[string]interface{}
if err := yaml.Unmarshal(data, &content); err != nil {
return fmt.Errorf("failed to parse yaml %s: %w", path, err)
}
return s.Err()
collectKeys(content, "", keys)
return nil
})
if err != nil {
fmt.Fprintf(os.Stderr, "Error walking directory: %v\n", err)
@@ -62,23 +62,44 @@ func main() {
defer f.Close()
w := bufio.NewWriter(f)
fmt.Fprintf(w, "// Code generated by cmd/gen_i18n. DO NOT EDIT.\n")
fmt.Fprintf(w, "// Code generated by cmd/geni18n. DO NOT EDIT.\n")
fmt.Fprintf(w, "package %s\n\n", *pkg)
fmt.Fprintf(w, "type Key string\n\n")
fmt.Fprintf(w, "const (\n")
for _, key := range list {
name := toPascal(key)
fmt.Fprintf(w, "\t%s = %q\n", name, key)
fmt.Fprintf(w, "\t%s Key = %q\n", name, key)
}
fmt.Fprintf(w, ")\n")
w.Flush()
}
func collectKeys(node map[string]interface{}, prefix string, keys map[string]struct{}) {
for k, v := range node {
fullKey := k
if prefix != "" {
fullKey = prefix + "." + k
}
switch val := v.(type) {
case map[string]interface{}:
collectKeys(val, fullKey, keys)
default:
keys[fullKey] = struct{}{}
}
}
}
// 转 PascalCase
func toPascal(key string) string {
parts := strings.Split(key, ".")
for i, p := range parts {
if len(p) > 0 {
parts[i] = strings.ToUpper(string(p[0])) + p[1:]
subs := strings.Split(p, "_")
for j, s := range subs {
if len(s) > 0 {
subs[j] = strings.ToUpper(s[:1]) + s[1:]
}
}
parts[i] = strings.Join(subs, "")
}
return strings.Join(parts, "")
}

View File

@@ -12,18 +12,20 @@ import (
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/client/bot"
userclient "github.com/krau/SaveAny-Bot/client/user"
"github.com/krau/SaveAny-Bot/common/cache"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/parsers"
"github.com/krau/SaveAny-Bot/storage"
"github.com/spf13/cobra"
)
func Run(cmd *cobra.Command, _ []string) {
ctx := cmd.Context()
ctx, cancel := context.WithCancel(cmd.Context())
logger := log.NewWithOptions(os.Stdout, log.Options{
Level: log.DebugLevel,
ReportTimestamp: true,
@@ -32,67 +34,84 @@ func Run(cmd *cobra.Command, _ []string) {
})
ctx = log.WithContext(ctx, logger)
initAll(ctx)
exitChan, err := initAll(ctx)
if err != nil {
logger.Fatal("Init failed", "error", err)
}
go func() {
<-exitChan
cancel()
}()
core.Run(ctx)
<-ctx.Done()
logger.Info(i18n.T(i18nk.Exiting))
defer logger.Info(i18n.T(i18nk.Bye))
logger.Info(i18n.T(i18nk.LifetimeExiting))
defer logger.Info(i18n.T(i18nk.LifetimeBye))
cleanCache()
}
func initAll(ctx context.Context) {
func initAll(ctx context.Context) (<-chan struct{}, error) {
if err := config.Init(ctx); err != nil {
fmt.Println("Failed to load config:", err)
os.Exit(1)
return nil, fmt.Errorf("failed to load config: %w", err)
}
cache.Init()
logger := log.FromContext(ctx)
i18n.Init(config.Cfg.Lang)
logger.Info(i18n.T(i18nk.Initing))
if config.Cfg.Telegram.Userbot.Enable {
uc, err := userclient.Login(ctx)
if err != nil {
logger.Fatalf("User client login failed: %s", err)
}
logger.Infof("User client logged in as %s", uc.Self.FirstName)
}
i18n.Init(config.C().Lang)
logger.Info(i18n.T(i18nk.LifetimeIniting))
database.Init(ctx)
storage.LoadStorages(ctx)
bot.Init(ctx)
if config.C().Parser.PluginEnable {
for _, dir := range config.C().Parser.PluginDirs {
if err := parsers.LoadPlugins(ctx, dir); err != nil {
logger.Error(i18n.T(i18nk.ParserPluginLoadFailed), "dir", dir, "error", err)
} else {
logger.Debug(i18n.T(i18nk.ParserPluginLoadedDir), "dir", dir)
}
}
}
if config.C().Telegram.Userbot.Enable {
_, err := userclient.Login(ctx)
if err != nil {
logger.Fatal(i18n.T(i18nk.LifetimeUserLoginFailed, map[string]any{
"Error": err,
}))
}
}
return bot.Init(ctx), nil
}
func cleanCache() {
if config.Cfg.NoCleanCache {
if config.C().NoCleanCache {
return
}
if config.Cfg.Temp.BasePath != "" && !config.Cfg.Stream {
if slices.Contains([]string{"/", ".", "\\", ".."}, filepath.Clean(config.Cfg.Temp.BasePath)) {
log.Error(i18n.T(i18nk.InvalidCacheDir, map[string]any{
"Path": config.Cfg.Temp.BasePath,
if config.C().Temp.BasePath != "" && !config.C().Stream {
if slices.Contains([]string{"/", ".", "\\", ".."}, filepath.Clean(config.C().Temp.BasePath)) {
log.Error(i18n.T(i18nk.ConfigErrInvalidCacheDir, map[string]any{
"Path": config.C().Temp.BasePath,
}))
return
}
currentDir, err := os.Getwd()
if err != nil {
log.Error(i18n.T(i18nk.GetWorkdirFailed, map[string]any{
log.Error(i18n.T(i18nk.ErrGetWorkdirFailed, map[string]any{
"Error": err,
}))
return
}
cachePath := filepath.Join(currentDir, config.Cfg.Temp.BasePath)
cachePath := filepath.Join(currentDir, config.C().Temp.BasePath)
cachePath, err = filepath.Abs(cachePath)
if err != nil {
log.Error(i18n.T(i18nk.GetCacheAbsPathFailed, map[string]any{
log.Error(i18n.T(i18nk.ErrGetCacheAbsPathFailed, map[string]any{
"Error": err,
}))
return
}
log.Info(i18n.T(i18nk.CleaningCache, map[string]any{
log.Info(i18n.T(i18nk.LifetimeCleaningCache, map[string]any{
"Path": cachePath,
}))
if err := fsutil.RemoveAllInDir(cachePath); err != nil {
log.Error(i18n.T(i18nk.CleanCacheFailed, map[string]any{
log.Error(i18n.T(i18nk.ErrCleanCacheFailed, map[string]any{
"Error": err,
}))
}

View File

@@ -4,7 +4,7 @@ import (
"fmt"
"runtime"
"github.com/krau/SaveAny-Bot/pkg/consts"
"github.com/krau/SaveAny-Bot/config"
"github.com/rhysd/go-github-selfupdate/selfupdate"
"github.com/blang/semver"
@@ -16,7 +16,7 @@ var VersionCmd = &cobra.Command{
Aliases: []string{"v"},
Short: "Print the version number of saveany-bot",
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("saveany-bot version: %s %s/%s\nBuildTime: %s, Commit: %s\n", consts.Version, runtime.GOOS, runtime.GOARCH, consts.BuildTime, consts.GitCommit)
fmt.Printf("saveany-bot version: %s %s/%s\nBuildTime: %s, Commit: %s\n", config.Version, runtime.GOOS, runtime.GOARCH, config.BuildTime, config.GitCommit)
},
}
@@ -25,14 +25,14 @@ var upgradeCmd = &cobra.Command{
Aliases: []string{"up"},
Short: "Upgrade saveany-bot to the latest version",
Run: func(cmd *cobra.Command, args []string) {
v := semver.MustParse(consts.Version)
latest, err := selfupdate.UpdateSelf(v, "krau/SaveAny-Bot")
v := semver.MustParse(config.Version)
latest, err := selfupdate.UpdateSelf(v, config.GitRepo)
if err != nil {
fmt.Println("Binary update failed:", err)
fmt.Println("Update failed:", err)
return
}
if latest.Version.Equals(v) {
fmt.Println("Current binary is the latest version", consts.Version)
fmt.Println("Current binary is the latest version", config.Version)
} else {
fmt.Println("Successfully updated to version", latest.Version)
fmt.Println("Release note:\n", latest.ReleaseNotes)

View File

@@ -2,19 +2,22 @@ package cache
import (
"fmt"
"time"
"github.com/charmbracelet/log"
"github.com/dgraph-io/ristretto/v2"
"github.com/krau/SaveAny-Bot/config"
)
var cache *ristretto.Cache[string, any]
// TODO: maybe we should use simple ttl cache instead of ristretto...
func init() {
func Init() {
if cache != nil {
panic("cache already initialized")
}
c, err := ristretto.NewCache(&ristretto.Config[string, any]{
NumCounters: 1e5,
MaxCost: 1e6, // 1000000 / 112 ≈ 8928
NumCounters: config.C().Cache.NumCounters,
MaxCost: config.C().Cache.MaxCost,
BufferItems: 64,
OnReject: func(item *ristretto.Item[any]) {
log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
@@ -27,7 +30,7 @@ func init() {
}
func Set(key string, value any) error {
ok := cache.Set(key, value, 0)
ok := cache.SetWithTTL(key, value, 0, time.Duration(config.C().Cache.TTL)*time.Second)
if !ok {
return fmt.Errorf("failed to set value in cache")
}

View File

@@ -5,12 +5,13 @@ import (
"maps"
"github.com/goccy/go-yaml"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/nicksnyder/go-i18n/v2/i18n"
"github.com/pelletier/go-toml/v2"
"golang.org/x/text/language"
)
//go:embed locale/*.toml
//go:embed locale/*
var localesFS embed.FS
var (
@@ -20,7 +21,7 @@ var (
func Init(lang string) {
bundle = i18n.NewBundle(language.SimplifiedChinese)
bundle.RegisterUnmarshalFunc("toml", toml.Unmarshal)
bundle.RegisterUnmarshalFunc("yaml", yaml.Unmarshal)
files, err := localesFS.ReadDir("locale")
if err != nil {
panic("failed to read locale directory: " + err.Error())
@@ -39,7 +40,7 @@ func Init(lang string) {
}
}
func T(key string, templateData ...map[string]any) string {
func T(key i18nk.Key, templateData ...map[string]any) string {
if localizer == nil || bundle == nil {
panic("localizer or bundle is not initialized, call Init() first")
}
@@ -48,11 +49,11 @@ func T(key string, templateData ...map[string]any) string {
maps.Copy(templateDataMap, data)
}
msg, err := localizer.Localize(&i18n.LocalizeConfig{
MessageID: key,
MessageID: string(key),
TemplateData: templateDataMap,
})
if err != nil {
return key
return string(key)
}
return msg
}
@@ -77,32 +78,32 @@ func TWithLang(lang, key string, templateData ...map[string]any) string {
}
// Only use in tests or packages that load before i18n
func TWithoutInit(lang, key string, templateData ...map[string]any) string {
func TWithoutInit(lang string, key i18nk.Key, templateData ...map[string]any) string {
bundle := i18n.NewBundle(language.SimplifiedChinese)
bundle.RegisterUnmarshalFunc("toml", toml.Unmarshal)
bundle.RegisterUnmarshalFunc("yaml", yaml.Unmarshal)
files, err := localesFS.ReadDir("locale")
if err != nil {
return key
return string(key)
}
for _, file := range files {
if _, err := bundle.LoadMessageFileFS(localesFS, "locale/"+file.Name()); err != nil {
return key
return string(key)
}
}
localizer := i18n.NewLocalizer(bundle, lang)
if localizer == nil {
return key
return string(key)
}
templateDataMap := make(map[string]any)
for _, data := range templateData {
maps.Copy(templateDataMap, data)
}
msg, err := localizer.Localize(&i18n.LocalizeConfig{
MessageID: key,
MessageID: string(key),
TemplateData: templateDataMap,
})
if err != nil {
return key
return string(key)
}
return msg
}

View File

@@ -1,19 +1,24 @@
// Code generated by cmd/gen_i18n. DO NOT EDIT.
// Code generated by cmd/geni18n. DO NOT EDIT.
package i18nk
type Key string
const (
CleanCacheFailed = "CleanCacheFailed"
CleaningCache = "CleaningCache"
ConfigInvalidDuplicateStorageName = "ConfigInvalid.DuplicateStorageName"
ConfigInvalidWorkersOrRetry = "ConfigInvalid.WorkersOrRetry"
CreateRmTimerFailed = "CreateRmTimerFailed"
GetCacheAbsPathFailed = "GetCacheAbsPathFailed"
GetWorkdirFailed = "GetWorkdirFailed"
InvalidCacheDir = "InvalidCacheDir"
LoadedStorages = "LoadedStorages"
RemoveFileAfter = "RemoveFileAfter"
RemoveFileFailed = "RemoveFileFailed"
Bye = "bye"
Exiting = "exiting"
Initing = "initing"
BotMsgHelpTextFmt Key = "bot.msg.help_text_fmt"
BotMsgSaveHelpText Key = "bot.msg.save_help_text"
BotMsgWatchHelpText Key = "bot.msg.watch_help_text"
ConfigErrDuplicateStorageName Key = "config.err.duplicate_storage_name"
ConfigErrInvalidCacheDir Key = "config.err.invalid_cache_dir"
ConfigLoadedStorages Key = "config.loaded_storages"
ErrCleanCacheFailed Key = "err.clean_cache_failed"
ErrGetCacheAbsPathFailed Key = "err.get_cache_abs_path_failed"
ErrGetWorkdirFailed Key = "err.get_workdir_failed"
LifetimeBye Key = "lifetime.bye"
LifetimeCleaningCache Key = "lifetime.cleaning_cache"
LifetimeExiting Key = "lifetime.exiting"
LifetimeInitfailed Key = "lifetime.initfailed"
LifetimeIniting Key = "lifetime.initing"
LifetimeUserLoginFailed Key = "lifetime.user_login_failed"
ParserPluginLoadFailed Key = "parser.plugin.load_failed"
ParserPluginLoadedDir Key = "parser.plugin.loaded_dir"
)

View File

@@ -1,28 +0,0 @@
[initing]
other = "正在启动..."
[exiting]
other = "正在退出..."
[bye]
other = "已退出"
[InvalidCacheDir]
other = "无效的缓存文件夹: {{.Path}}"
[GetWorkdirFailed]
other = "获取工作目录失败: {{.Error}}"
[GetCacheAbsPathFailed]
other = "获取缓存绝对路径失败: {{.Error}}"
[CleaningCache]
other = "正在清理缓存文件夹: {{.Path}}"
[CleanCacheFailed]
other = "清理缓存失败: {{.Error}}"
[CreateRmTimerFailed]
other = "创建清理定时器失败, 路径: {{.Path}}, 错误: {{.Error}}"
[RemoveFileAfter]
other = "将在 {{.Duration}} 后删除文件: {{.Path}}"
[RemoveFileFailed]
other = "删除文件失败: {{.Path}}, 错误: {{.Error}}"
[LoadedStorages]
other = "已加载 {{.Count}} 个存储"
[ConfigInvalid.WorkersOrRetry]
other = "配置无效: workers 或 retry 必须大于 0, 但当前值为: workers={{.Workers}}, retry={{.Retry}}"
[ConfigInvalid.DuplicateStorageName]
other = "存储名称重复: {{.Name}}"

View File

@@ -0,0 +1,62 @@
lifetime:
initing: 正在启动
initfailed: 初始化失败
exiting: 正在退出
user_login_failed: "用户登录失败: {{.Error}}"
cleaning_cache: "正在清理缓存 {{.Path}}"
bye: 已退出
config:
loaded_storages: "已加载 {{.Count}} 个存储后端"
err:
invalid_cache_dir: "无效的缓存目录: {{.Path}},请检查配置文件"
duplicate_storage_name: "存储名称 '{{.Name}}' 重复,请检查配置文件"
err:
get_workdir_failed: "获取工作目录失败: {{.Error}}"
get_cache_abs_path_failed: "获取缓存绝对路径失败: {{.Error}}"
clean_cache_failed: "清理缓存失败: {{.Error}}"
parser:
plugin:
load_failed: 加载解析器插件失败
loaded_dir: 解析器插件已加载
bot:
msg:
help_text_fmt: |
Save Any Bot - 转存你的 Telegram 文件
版本: %s , 提交: %s
命令:
/start - 开始使用
/help - 显示帮助
/silent - 开关静默模式
/storage - 设置默认存储位置
/save [自定义文件名] - 保存文件
/dir - 管理存储目录
/rule - 管理规则
/update - 检查更新并升级
使用帮助: https://sabot.unv.app/usage
反馈群组: https://t.me/ProjectSaveAny
save_help_text: |
使用方法:
1. 使用该命令回复要保存的文件, 可选文件名参数.
示例:
/save custom_file_name.mp4
2. 设置默认存储后, 发送 /save <频道ID/用户名> <消息ID范围> 来批量保存文件. 遵从存储规则, 若未匹配到任何规则则使用默认存储.
示例:
/save @acherkrau 114-514
watch_help_text: |
使用 /watch 命令监听一个聊天的消息, 并自动保存到默认存储中, 遵从存储规则.
命令语法:
/watch <chat_id> [filter]
参数:
- <chat_id>: 聊天的 ID 或用户名
- [filter]: 可选, 格式为 过滤器类型:表达式 , 所有支持类型的过滤器请查看文档
命令示例:
/watch 2229835658 msgre:.*plana.*
这将监听 ID 为 2229835658 的聊天, 并转存所有包含 "plana" 的媒体消息

View File

@@ -1,18 +0,0 @@
package tdler
import (
"github.com/gotd/td/telegram/downloader"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/consts/tglimit"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
type Client interface {
downloader.Client
}
func NewDownloader(client Client, file tfile.TGFile) *downloader.Builder {
return downloader.NewDownloader().WithPartSize(tglimit.MaxPartSize).
Download(client, file.Location()).WithThreads(dlutil.BestThreads(file.Size(), config.Cfg.Threads))
}

View File

@@ -3,6 +3,8 @@ package fsutil
import (
"os"
"path/filepath"
"strings"
"unicode"
"github.com/gabriel-vasile/mimetype"
)
@@ -55,3 +57,21 @@ func CreateFile(fp string) (*File, error) {
}
return &File{File: file}, nil
}
func NormalizePathname(s string) string {
specials := `\/:*?"<>|` + "\n\r\t"
var builder strings.Builder
for _, ch := range s {
if strings.ContainsRune(specials, ch) || unicode.IsControl(ch) {
builder.WriteRune('_')
} else {
builder.WriteRune(ch)
}
}
result := strings.TrimRightFunc(builder.String(), func(r rune) bool {
return r == '.' || r == '_' || unicode.IsSpace(r)
})
return result
}

View File

@@ -0,0 +1,46 @@
package fsutil_test
import (
"testing"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
)
func TestNormalizePathname(t *testing.T) {
tests := []struct {
input string
expected string
}{
{
input: "hello/world?.txt ",
expected: "hello_world_.txt",
},
{
input: "bad|name:\nfile\r.",
expected: "bad_name__file",
},
{
input: "normal.txt",
expected: "normal.txt",
},
{
input: "test.... ",
expected: "test",
},
{
input: "abc<>def",
expected: "abc__def",
},
{
input: "with\tcontrol",
expected: "with_control",
},
}
for _, tc := range tests {
got := fsutil.NormalizePathname(tc.input)
if got != tc.expected {
t.Errorf("NormalizePathname(%q) = %q; want %q", tc.input, got, tc.expected)
}
}
}

View File

@@ -0,0 +1,78 @@
package netutil
import (
"context"
"fmt"
"net"
"net/http"
"net/url"
"sync"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"golang.org/x/net/proxy"
)
func NewProxyDialer(proxyUrl string) (proxy.Dialer, error) {
url, err := url.Parse(proxyUrl)
if err != nil {
return nil, err
}
return proxy.FromURL(url, proxy.Direct)
}
func NewProxyHTTPClient(proxyUrl string) (*http.Client, error) {
if proxyUrl == "" {
return &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
},
}, nil
}
u, err := url.Parse(proxyUrl)
if err != nil {
return nil, err
}
switch u.Scheme {
case "http", "https":
return &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(u),
},
}, nil
case "socks5":
dialer, err := proxy.FromURL(u, proxy.Direct)
if err != nil {
return nil, err
}
return &http.Client{
Transport: &http.Transport{
DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
return dialer.Dial(network, addr)
},
},
}, nil
default:
return nil, fmt.Errorf("unsupported proxy scheme: %s", u.Scheme)
}
}
var (
defaultProxyHttpClient *http.Client
onceLoadDefaultProxyHttpClient sync.Once
)
func DefaultParserHTTPClient() *http.Client {
onceLoadDefaultProxyHttpClient.Do(func() {
client, err := NewProxyHTTPClient(config.C().Parser.Proxy)
if err != nil {
log.Warn("Failed to create default proxy HTTP client, using http.DefaultClient", "error", err)
defaultProxyHttpClient = http.DefaultClient
} else {
defaultProxyHttpClient = client
}
})
return defaultProxyHttpClient
}

View File

@@ -0,0 +1,40 @@
package tgutil
import (
"fmt"
"github.com/gabriel-vasile/mimetype"
"github.com/gotd/td/tg"
)
func GetMediaFileName(media tg.MessageMediaClass) (string, error) {
switch v := media.(type) {
case *tg.MessageMediaPhoto:
f, ok := v.Photo.AsNotEmpty()
if !ok {
return "", fmt.Errorf("unknown type media: %T", media)
}
return fmt.Sprintf("%d.png", f.ID), nil
case *tg.MessageMediaDocument:
f, ok := v.Document.AsNotEmpty()
if !ok {
return "", fmt.Errorf("unknown type media: %T", media)
}
fileName := ""
for _, attribute := range f.Attributes {
if name, ok := attribute.(*tg.DocumentAttributeFilename); ok {
fileName = name.GetFileName()
break
}
}
if fileName == "" {
mmt := mimetype.Lookup(f.GetMimeType())
if mmt != nil {
fileName = fmt.Sprintf("%d.%s", f.GetID(), mmt.Extension())
}
}
return fileName, nil
default:
return "", fmt.Errorf("unsupported type media: %T", media)
}
}

View File

@@ -4,9 +4,12 @@ import (
"fmt"
"strconv"
"strings"
"unicode"
"unicode/utf16"
"github.com/celestix/gotgproto/ext"
"github.com/duke-git/lancet/v2/maputil"
"github.com/duke-git/lancet/v2/mathutil"
"github.com/duke-git/lancet/v2/slice"
lcstrutil "github.com/duke-git/lancet/v2/strutil"
@@ -18,6 +21,9 @@ import (
"github.com/rs/xid"
)
// generate a file name from the message content and media type
//
// it will never return an empty string
func GenFileNameFromMessage(message tg.Message) string {
ext := func(media tg.MessageMediaClass) string {
switch media := media.(type) {
@@ -26,11 +32,11 @@ func GenFileNameFromMessage(message tg.Message) string {
if !ok {
return ""
}
ext := mimetype.Lookup(doc.MimeType).Extension()
if ext == "" {
mmt := mimetype.Lookup(doc.MimeType)
if mmt == nil || mmt.Extension() == "" {
return ""
}
return ext
return mmt.Extension()
case *tg.MessageMediaPhoto:
return ".jpg"
}
@@ -57,16 +63,12 @@ func GenFileNameFromMessage(message tg.Message) string {
return fmt.Sprintf("%s_%s", tagStr, strconv.Itoa(message.GetID()))
}
text = lcstrutil.Substring(strings.Map(func(r rune) rune {
if r < 0x20 || r == 0x7F {
return '_'
}
switch r {
// invalid characters
case '/', '\\',
':', '*', '?', '"', '<', '>', '|':
return '_'
// empty
case ' ', '\t', '\r', '\n':
}
if unicode.IsControl(r) || unicode.IsSpace(r) {
return '_'
}
if validator.IsPrintable(string(r)) {
@@ -81,7 +83,13 @@ func GenFileNameFromMessage(message tg.Message) string {
}()
if filename == "" {
filename = fmt.Sprintf("%d_%s", message.GetID(), xid.New().String())
mname, err := GetMediaFileName(message.Media)
if err != nil {
filename = fmt.Sprintf("%d_%s", message.GetID(), xid.New().String())
} else {
filename = mname
}
}
return filename + ext
}
@@ -159,6 +167,96 @@ func GetMessagesRange(ctx *ext.Context, chatID int64, minId, maxId int) ([]*tg.M
return result, nil
}
type MessageItem struct {
Message *tg.Message
Error error
}
func IterMessages(ctx *ext.Context, chatID int64, minId, maxId int) (<-chan MessageItem, error) {
total := maxId - minId + 1
ch := make(chan MessageItem, 100)
go func() {
defer close(ch)
if !ctx.Self.Bot {
perr := ctx.PeerStorage.GetInputPeerById(chatID)
if perr == nil || perr.(*tg.InputPeerEmpty) != nil {
ch <- MessageItem{
Error: fmt.Errorf("peer not found: %d", chatID),
}
return
}
for i := 0; i < total; i += 100 {
start := minId + i
end := min(start+100, maxId)
msgs, err := ctx.Raw.MessagesGetHistory(ctx, &tg.MessagesGetHistoryRequest{
Peer: perr,
OffsetID: start,
AddOffset: start - end,
Limit: 100,
})
if err != nil {
ch <- MessageItem{
Error: fmt.Errorf("failed to get messages: %w", err),
}
return
}
var msgClass []tg.MessageClass
switch msgsv := msgs.(type) {
case *tg.MessagesMessages:
msgClass = msgsv.GetMessages()
case *tg.MessagesMessagesSlice:
msgClass = msgsv.GetMessages()
case *tg.MessagesChannelMessages:
msgClass = msgsv.GetMessages()
default:
ch <- MessageItem{
Error: fmt.Errorf("unsupported message type: %T", msgsv),
}
continue
}
for _, msg := range msgClass {
msg, ok := msg.AsNotEmpty()
if !ok {
continue
}
switch msg := msg.(type) {
case *tg.Message:
key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msg.GetID())
cache.Set(key, msg)
ch <- MessageItem{
Message: msg,
}
}
}
}
} else {
for i := 0; i < total; i += 100 {
start := minId + i
end := min(start+100, maxId)
msgs, err := GetMessagesRange(ctx, chatID, start, end)
if err != nil {
ch <- MessageItem{
Error: fmt.Errorf("failed to get messages: %w", err),
}
return
}
for _, msg := range msgs {
if msg == nil {
continue
}
ch <- MessageItem{
Message: msg,
}
}
}
}
}()
return ch, nil
}
func GetMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, error) {
key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msgID)
if msg, ok := cache.Get[*tg.Message](key); ok {
@@ -181,3 +279,77 @@ func GetMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, err
cache.Set(key, tgm)
return tgm, nil
}
func GetGroupedMessages(ctx *ext.Context, chatID int64, msg *tg.Message) ([]*tg.Message, error) {
groupID, isGroup := msg.GetGroupedID()
if !isGroup || groupID == 0 {
return nil, fmt.Errorf("message %d is not grouped", msg.GetID())
}
msgID := msg.GetID()
minID := msgID - 10
maxID := msgID + 10
if minID < 1 {
minID = 1
}
msgs, err := GetMessagesRange(ctx, chatID, minID, maxID)
if err != nil {
return nil, fmt.Errorf("failed to get grouped messages: %w", err)
}
groupedMessages := make([]*tg.Message, 0, len(msgs))
for _, m := range msgs {
if m == nil {
continue
}
mgid, isGroup := m.GetGroupedID()
if isGroup && mgid == groupID {
groupedMessages = append(groupedMessages, m)
}
}
return groupedMessages, nil
}
func ExtractMessageEntityUrls(msg *tg.Message) []string {
if len(msg.Entities) == 0 {
return nil
}
msgText := msg.GetMessage()
if msgText == "" {
return nil
}
runes := []rune(msgText)
utf16Codes := utf16.Encode(runes)
var urls []string
for _, entity := range msg.Entities {
switch ent := entity.(type) {
case *tg.MessageEntityTextURL:
urls = append(urls, ent.GetURL())
case *tg.MessageEntityURL:
start := ent.Offset
end := ent.Offset + ent.Length
if start < 0 || end > len(utf16Codes) {
continue
}
subRunes := utf16.Decode(utf16Codes[start:end])
urls = append(urls, string(subRunes))
}
}
return urls
}
func ExtractMessageEntityUrlsText(msg *tg.Message) string {
if msg == nil {
return ""
}
urls := ExtractMessageEntityUrls(msg)
if len(urls) == 0 {
return msg.GetMessage()
}
var sb strings.Builder
for _, url := range urls {
sb.WriteString(url)
sb.WriteString(" ")
}
return sb.String()
}

View File

@@ -0,0 +1,16 @@
package tgutil
import "github.com/gotd/td/tg"
func ChatIdFromPeer(peer tg.PeerClass) int64 {
switch peer := peer.(type) {
case *tg.PeerChannel:
return peer.ChannelID
case *tg.PeerUser:
return peer.UserID
case *tg.PeerChat:
return peer.ChatID
default:
return 0
}
}

View File

@@ -86,7 +86,7 @@ func ParseMessageLink(ctx *ext.Context, link string) (int64, int, error) {
return chatID, msgID, nil
case 3:
// https://t.me/c/123456789/123
// https://t.me/acherkrau/123/456 , 456: message thread ID
// https://t.me/acherkrau/123/456 , 123: topic id
chatPart, msgPart := paths[1], paths[2]
if paths[0] != "c" {
chatPart = paths[0]

View File

@@ -2,28 +2,38 @@ package tphutil
import (
"encoding/json"
"strings"
"sync"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
)
var tphClient *telegraph.Client
var (
tphClient *telegraph.Client
once sync.Once
)
func DefaultClient() *telegraph.Client {
if tphClient != nil {
return tphClient
}
if config.Cfg.Telegram.Proxy.Enable && config.Cfg.Telegram.Proxy.URL != "" {
proxyUrl := config.Cfg.Telegram.Proxy.URL
once.Do(func() {
tphClient = initDefault()
})
return tphClient
}
func initDefault() *telegraph.Client {
var client *telegraph.Client
if config.C().Telegram.Proxy.Enable && config.C().Telegram.Proxy.URL != "" {
proxyUrl := config.C().Telegram.Proxy.URL
var err error
tphClient, err = telegraph.NewClientWithProxy(proxyUrl)
client, err = telegraph.NewClientWithProxy(proxyUrl)
if err != nil {
tphClient = telegraph.NewClient()
client = telegraph.NewClient()
}
} else {
tphClient = telegraph.NewClient()
client = telegraph.NewClient()
}
return tphClient
return client
}
func GetNodeImages(node telegraph.Node) []string {
@@ -41,6 +51,10 @@ func GetNodeImages(node telegraph.Node) []string {
if nodeElement.Tag == "img" {
if src, exists := nodeElement.Attrs["src"]; exists {
if strings.HasPrefix(src, "/file/") {
// handle images on telegra.ph server
src = "https://telegra.ph" + src
}
srcs = append(srcs, src)
}
}

View File

@@ -1,70 +1,34 @@
#创建文件时,若需要保留中文注释,请务必确保本文件编码为 UTF-8 ,否则会无法读取。
workers = 4 # 同时下载文件数
retry = 3 # 下载失败重试次
threads = 4 # 单个任务下载最大线程
stream = false # 使用stream模式, 详情请查看文档
# 创建文件时,若需要保留中文注释,请务必确保本文件编码为 UTF-8 ,否则会无法读取。
# 更详细的配置请在 https://sabot.unv.app/deployment/configuration 查看
workers = 4 # 同时下载文件
retry = 3 # 下载失败重试次
threads = 4 # 单个任务下载使用的最大线程数
stream = false # 使用流式传输模式, 建议仅在硬盘空间十分有限时使用.
[telegram]
# Bot Token
# 更换 Bot Token 后请删除数据库文件 session.db
# 更换 Bot Token 后请删除会话数据库文件 (默认路径为 data/session.db )
token = ""
# Telegram API 配置, 若不配置也可运行, 将使用默认的 API ID 和 API HASH
# 推荐使用自己的 API ID 和 API HASH (https://my.telegram.org)
# app_id = 1025907
# app_hash = "452b0359b988148995f22ff0f4229750"
# 初始化超时时间, 单位: 秒
timeout = 60
# flood_retry = 5
# rpc_retry = 5
[telegram.proxy]
# 启用代理连接 telegram, 只支持 socks5
enable = false
url = "socks5://127.0.0.1:7890"
# 用户列表
[[users]]
# telegram user id
id = 114514
# 使用黑名单模式,开启后下方留空以使用所有存储,反之则为白名单,白名单请在下方输入允许的存储名
blacklist = true
# 将列表留空并开启黑名单模式以允许使用所有存储,此处示例为黑名单模式,用户 114514 可使用所有存储
storages = []
[[users]]
id = 123456
blacklist = false # 使用白名单模式此时用户123456 仅可使用下方列表中的存储
# 此时该用户只能使用名为 本机1 的存储
storages = ["本机1"]
# 存储列表
[[storages]]
# 标识名, 需要唯一
name = "本机1"
# 存储类型, 目前可用: local, alist, webdav, minio
# 存储类型, 目前可用: local, alist, webdav, minio, telegram
type = "local"
# 启用存储
enable = true
# 文件保存根路径
base_path = "./downloads"
[[storages]]
name = "MyAlist"
type = "alist"
enable = false #记得启用
base_path = '/'
url = 'https://alist.com'
username = 'admin'
password = 'password'
# alist token 刷新时间
# 86400--1天 604800--7天 1296000--15天 2592000--30天 15552000--180天
token_exp = 86400
# alist 可直接使用 token 登录, 此时 username, password, token_exp 将被忽略
# 请自行在 alist 侧配置合理的 token 过期时间
# token = ""
[[storages]]
name = "MyWebdav"
type = "webdav"
@@ -74,31 +38,17 @@ url = 'https://example.com/dav'
username = 'username'
password = 'password'
[[storages]]
name = "MyMinio"
type = "minio"
enable = true
endpoint = 'play.min.io'
use_ssl = true
access_key_id = 'Q3AM3UQ867SPQQA43P2F'
secret_access_key = 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG'
bucket_name = 'saveanybot'
base_path = '/path/telegram'
# 其他配置
# [log]
# # 日志等级
# level = "DEBUG"
# [temp]
# # 下载文件临时目录, 请不要在此目录下存放任何其他文件
# base_path = "cache/"
# # 临时文件保存时间, 单位: 秒
# cache_ttl = 30
# [db]
# path = "data/data.db" # 数据库文件路径
# session = "data/session.db"
# 用户列表
[[users]]
# telegram user id
id = 114514
# 存储过滤列表, 元素为存储标识名.
# 将该列表留空并开启黑名单过滤模式以允许使用所有存储,此处示例为黑名单模式,用户 114514 可使用所有存储
storages = []
# 使用列表过滤黑名单模式,反之则为白名单,白名单请在列表中指定可用的存储.
blacklist = true
[[users]]
id = 123456
storages = ["本机1"]
blacklist = false # 使用白名单模式,此时,用户 123456 仅可使用标识名为 '本地1' 的存储

7
config/cache.go Normal file
View File

@@ -0,0 +1,7 @@
package config
type cacheConfig struct {
TTL int64 `toml:"ttl" mapstructure:"ttl" json:"ttl"`
NumCounters int64 `toml:"num_counters" mapstructure:"num_counters" json:"num_counters"`
MaxCost int64 `toml:"max_cost" mapstructure:"max_cost" json:"max_cost"`
}

6
config/db.go Normal file
View File

@@ -0,0 +1,6 @@
package config
type dbConfig struct {
Path string `toml:"path" mapstructure:"path"`
Session string `toml:"session" mapstructure:"session"`
}

22
config/hook.go Normal file
View File

@@ -0,0 +1,22 @@
package config
type hookConfig struct {
Exec hookExecConfig `toml:"exec" mapstructure:"exec" json:"exec"`
}
type hookExecConfig struct {
// command to execute, for all task types
TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
// TaskTypes map[string]hookExecOnTypeConfig `toml:"task_types" mapstructure:"task_types" json:"task_types"` // [TODO]
}
// type hookExecOnTypeConfig struct {
// TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
// TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
// TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
// TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
// }

15
config/parser.go Normal file
View File

@@ -0,0 +1,15 @@
package config
type parserConfig struct {
PluginEnable bool `toml:"plugin_enable" mapstructure:"plugin_enable" json:"plugin_enable"`
PluginDirs []string `toml:"plugin_dirs" mapstructure:"plugin_dirs" json:"plugin_dirs"`
Proxy string `toml:"proxy" mapstructure:"proxy" json:"proxy"`
ParserCfgs map[string]map[string]any `mapstructure:",remain"`
}
func (c Config) GetParserConfigByName(name string) map[string]any {
if c.Parser.ParserCfgs == nil {
return nil
}
return c.Parser.ParserCfgs[name]
}

View File

@@ -9,6 +9,7 @@ import (
type TelegramStorageConfig struct {
BaseConfig
ChatID int64 `toml:"chat_id" mapstructure:"chat_id" json:"chat_id"`
ForceFile bool `toml:"force_file" mapstructure:"force_file" json:"force_file"`
RateLimit int `toml:"rate_limit" mapstructure:"rate_limit" json:"rate_limit"`
RateBurst int `toml:"rate_burst" mapstructure:"rate_burst" json:"rate_burst"`
}

5
config/temp.go Normal file
View File

@@ -0,0 +1,5 @@
package config
type tempConfig struct {
BasePath string `toml:"base_path" mapstructure:"base_path" json:"base_path"`
}

20
config/tg.go Normal file
View File

@@ -0,0 +1,20 @@
package config
type telegramConfig struct {
Token string `toml:"token" mapstructure:"token"`
AppID int `toml:"app_id" mapstructure:"app_id" json:"app_id"`
AppHash string `toml:"app_hash" mapstructure:"app_hash" json:"app_hash"`
Proxy tgProxyConfig `toml:"proxy" mapstructure:"proxy"`
RpcRetry int `toml:"rpc_retry" mapstructure:"rpc_retry" json:"rpc_retry"`
Userbot userbotConfig `toml:"userbot" mapstructure:"userbot" json:"userbot"`
}
type userbotConfig struct {
Enable bool `toml:"enable" mapstructure:"enable"`
Session string `toml:"session" mapstructure:"session"`
}
type tgProxyConfig struct {
Enable bool `toml:"enable" mapstructure:"enable"`
URL string `toml:"url" mapstructure:"url"`
}

View File

@@ -14,7 +14,7 @@ var userIDs []int64
var storages []string
var userStorages = make(map[int64][]string)
func (c *Config) GetStorageNamesByUserID(userID int64) []string {
func (c Config) GetStorageNamesByUserID(userID int64) []string {
us, ok := userStorages[userID]
if ok {
return us
@@ -22,11 +22,11 @@ func (c *Config) GetStorageNamesByUserID(userID int64) []string {
return nil
}
func (c *Config) GetUsersID() []int64 {
func (c Config) GetUsersID() []int64 {
return userIDs
}
func (c *Config) HasStorage(userID int64, storageName string) bool {
func (c Config) HasStorage(userID int64, storageName string) bool {
us, ok := userStorages[userID]
if !ok {
return false

13
config/version.go Normal file
View File

@@ -0,0 +1,13 @@
package config
// inject version by '-X' flag
// go build -ldflags "-X github.com/krau/SaveAny-Bot/config.Version=${{ env.VERSION }}"
var (
Version string = "dev"
BuildTime string = "unknown"
GitCommit string = "unknown"
)
const (
GitRepo = "krau/SaveAny-Bot"
)

View File

@@ -4,7 +4,6 @@ import (
"context"
"errors"
"fmt"
"os"
"strings"
"github.com/duke-git/lancet/v2/slice"
@@ -22,54 +21,22 @@ type Config struct {
Threads int `toml:"threads" mapstructure:"threads" json:"threads"`
Stream bool `toml:"stream" mapstructure:"stream" json:"stream"`
Users []userConfig `toml:"users" mapstructure:"users" json:"users"`
Cache cacheConfig `toml:"cache" mapstructure:"cache" json:"cache"`
Users []userConfig `toml:"users" mapstructure:"users" json:"users"`
Temp tempConfig `toml:"temp" mapstructure:"temp"`
Log logConfig `toml:"log" mapstructure:"log"`
DB dbConfig `toml:"db" mapstructure:"db"`
Telegram telegramConfig `toml:"telegram" mapstructure:"telegram"`
Storages []storage.StorageConfig `toml:"-" mapstructure:"-" json:"storages"`
Parser parserConfig `toml:"parser" mapstructure:"parser" json:"parser"`
Hook hookConfig `toml:"hook" mapstructure:"hook" json:"hook"`
}
type tempConfig struct {
BasePath string `toml:"base_path" mapstructure:"base_path" json:"base_path"`
CacheTTL int64 `toml:"cache_ttl" mapstructure:"cache_ttl" json:"cache_ttl"`
}
var cfg = &Config{}
type logConfig struct {
Level string `toml:"level" mapstructure:"level"`
File string `toml:"file" mapstructure:"file"`
BackupCount uint `toml:"backup_count" mapstructure:"backup_count" json:"backup_count"`
func C() Config {
return *cfg
}
type dbConfig struct {
Path string `toml:"path" mapstructure:"path"`
Session string `toml:"session" mapstructure:"session"`
Expire int64 `toml:"expire" mapstructure:"expire"`
}
type telegramConfig struct {
Token string `toml:"token" mapstructure:"token"`
AppID int `toml:"app_id" mapstructure:"app_id" json:"app_id"`
AppHash string `toml:"app_hash" mapstructure:"app_hash" json:"app_hash"`
Timeout int `toml:"timeout" mapstructure:"timeout" json:"timeout"`
Proxy proxyConfig `toml:"proxy" mapstructure:"proxy"`
RpcRetry int `toml:"rpc_retry" mapstructure:"rpc_retry" json:"rpc_retry"`
Userbot userbotConfig `toml:"userbot" mapstructure:"userbot" json:"userbot"`
}
type userbotConfig struct {
Enable bool `toml:"enable" mapstructure:"enable"`
Session string `toml:"session" mapstructure:"session"`
}
type proxyConfig struct {
Enable bool `toml:"enable" mapstructure:"enable"`
URL string `toml:"url" mapstructure:"url"`
}
var Cfg *Config
func (c Config) GetStorageByName(name string) storage.StorageConfig {
for _, storage := range c.Storages {
if storage.GetName() == name {
@@ -89,28 +56,36 @@ func Init(ctx context.Context) error {
replacer := strings.NewReplacer(".", "_")
viper.SetEnvKeyReplacer(replacer)
viper.SetDefault("lang", "zh-Hans")
defaultConfigs := map[string]any{
// 基础配置
"lang": "zh-Hans",
"workers": 3,
"retry": 3,
"threads": 4,
viper.SetDefault("workers", 3)
viper.SetDefault("retry", 3)
viper.SetDefault("threads", 4)
// 缓存配置
"cache.ttl": 86400,
"cache.num_counters": 1e5,
"cache.max_cost": 1e6,
viper.SetDefault("telegram.app_id", 1025907)
viper.SetDefault("telegram.app_hash", "452b0359b988148995f22ff0f4229750")
viper.SetDefault("telegram.timeout", 60)
viper.SetDefault("telegram.flood_retry", 5)
viper.SetDefault("telegram.rpc_retry", 5)
viper.SetDefault("telegram.userbot.enable", false)
viper.SetDefault("telegram.userbot.session", "data/usersession.db")
// Telegram
"telegram.app_id": 1025907,
"telegram.app_hash": "452b0359b988148995f22ff0f4229750",
"telegram.rpc_retry": 5,
"telegram.userbot.enable": false,
"telegram.userbot.session": "data/usersession.db",
viper.SetDefault("temp.base_path", "cache/")
viper.SetDefault("temp.cache_ttl", 30)
// 临时目录
"temp.base_path": "cache/",
viper.SetDefault("log.level", "INFO")
// 数据库
"db.path": "data/saveany.db",
"db.session": "data/session.db",
}
viper.SetDefault("db.path", "data/saveany.db")
viper.SetDefault("db.session", "data/session.db")
viper.SetDefault("db.expire", 86400*5)
for key, value := range defaultConfigs {
viper.SetDefault(key, value)
}
if err := viper.SafeWriteConfigAs("config.toml"); err != nil {
if _, ok := err.(viper.ConfigFileAlreadyExistsError); !ok {
@@ -120,50 +95,51 @@ func Init(ctx context.Context) error {
if err := viper.ReadInConfig(); err != nil {
fmt.Println("Error reading config file, ", err)
os.Exit(1)
return err
}
Cfg = &Config{}
if err := viper.Unmarshal(Cfg); err != nil {
if err := viper.Unmarshal(cfg); err != nil {
fmt.Println("Error unmarshalling config file, ", err)
os.Exit(1)
return err
}
storagesConfig, err := storage.LoadStorageConfigs(viper.GetViper())
if err != nil {
return fmt.Errorf("error loading storage configs: %w", err)
}
Cfg.Storages = storagesConfig
cfg.Storages = storagesConfig
storageNames := make(map[string]struct{})
for _, storage := range Cfg.Storages {
for _, storage := range cfg.Storages {
if _, ok := storageNames[storage.GetName()]; ok {
return errors.New(i18n.TWithoutInit(Cfg.Lang, i18nk.ConfigInvalidDuplicateStorageName, map[string]any{
return errors.New(i18n.TWithoutInit(cfg.Lang, i18nk.ConfigErrDuplicateStorageName, map[string]any{
"Name": storage.GetName(),
}))
}
storageNames[storage.GetName()] = struct{}{}
}
fmt.Println(i18n.TWithoutInit(Cfg.Lang, i18nk.LoadedStorages, map[string]any{
"Count": len(Cfg.Storages),
fmt.Println(i18n.TWithoutInit(cfg.Lang, i18nk.ConfigLoadedStorages, map[string]any{
"Count": len(cfg.Storages),
}))
for _, storage := range Cfg.Storages {
for _, storage := range cfg.Storages {
fmt.Printf(" - %s (%s)\n", storage.GetName(), storage.GetType())
}
if Cfg.Workers < 1 || Cfg.Retry < 1 {
return errors.New(i18n.TWithoutInit(Cfg.Lang, i18nk.ConfigInvalidWorkersOrRetry, map[string]any{
"Workers": Cfg.Workers,
"Retry": Cfg.Retry,
}))
if cfg.Workers < 1 {
cfg.Workers = 1
}
if cfg.Threads < 1 {
cfg.Threads = 1
}
if cfg.Retry < 1 {
cfg.Retry = 1
}
for _, storage := range Cfg.Storages {
for _, storage := range cfg.Storages {
storages = append(storages, storage.GetName())
}
for _, user := range Cfg.Users {
for _, user := range cfg.Users {
userIDs = append(userIDs, user.ID)
if user.Blacklist {
userStorages[user.ID] = slice.Compact(slice.Difference(storages, user.Storages))
@@ -171,23 +147,5 @@ func Init(ctx context.Context) error {
userStorages[user.ID] = user.Storages
}
}
return nil
}
func Set(key string, value any) {
viper.Set(key, value)
}
func ReloadConfig() error {
if err := viper.WriteConfig(); err != nil {
return err
}
if err := viper.ReadInConfig(); err != nil {
return err
}
if error := viper.Unmarshal(Cfg); error != nil {
return error
}
return nil
}

View File

@@ -2,32 +2,54 @@ package core
import (
"context"
"errors"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/queue"
)
var queueInstance *queue.TaskQueue[Exectable]
type Exectable interface {
Type() tasktype.TaskType
TaskID() string
Execute(ctx context.Context) error
}
func worker(ctx context.Context, qe *queue.TaskQueue[Exectable], semaphore chan struct{}) {
logger := log.FromContext(ctx)
execHooks := config.C().Hook.Exec
for {
semaphore <- struct{}{}
qtask, err := qe.Get()
if err != nil {
logger.Error("Failed to get task from queue:", err)
break // queue closed and empty
}
log.FromContext(ctx).Infof("Processing task: %s", qtask.ID)
task := qtask.Data
logger.Infof("Processing task: %s", task.TaskID())
if err := ExecCommandString(qtask.Context(), execHooks.TaskBeforeStart); err != nil {
logger.Errorf("Failed to execute before start hook for task %s: %v", task.TaskID(), err)
}
if err := task.Execute(qtask.Context()); err != nil {
log.FromContext(ctx).Errorf("Failed to execute task %s: %v", qtask.ID, err)
if errors.Is(err, context.Canceled) {
logger.Infof("Task %s was canceled", task.TaskID())
if err := ExecCommandString(ctx, execHooks.TaskCancel); err != nil {
logger.Errorf("Failed to execute cancel hook for task %s: %v", task.TaskID(), err)
}
} else {
logger.Errorf("Failed to execute task %s: %v", task.TaskID(), err)
if err := ExecCommandString(ctx, execHooks.TaskFail); err != nil {
logger.Errorf("Failed to execute fail hook for task %s: %v", task.TaskID(), err)
}
}
} else {
log.FromContext(ctx).Infof("Task %s completed successfully", qtask.ID)
logger.Infof("Task %s completed successfully", task.TaskID())
if err := ExecCommandString(ctx, execHooks.TaskSuccess); err != nil {
logger.Errorf("Failed to execute success hook for task %s: %v", task.TaskID(), err)
}
}
qe.Done(qtask.ID)
<-semaphore
@@ -36,11 +58,11 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Exectable], semaphore chan
func Run(ctx context.Context) {
log.FromContext(ctx).Info("Start processing tasks...")
semaphore := make(chan struct{}, config.Cfg.Workers)
semaphore := make(chan struct{}, config.C().Workers)
if queueInstance == nil {
queueInstance = queue.NewTaskQueue[Exectable]()
}
for range config.Cfg.Workers {
for range config.C().Workers {
go worker(ctx, queueInstance, semaphore)
}

23
core/hookutil.go Normal file
View File

@@ -0,0 +1,23 @@
package core
import (
"context"
"os"
"os/exec"
"runtime"
)
func ExecCommandString(ctx context.Context, cmd string) error {
if cmd == "" {
return nil
}
var execCmd *exec.Cmd
if runtime.GOOS == "windows" {
execCmd = exec.CommandContext(ctx, "cmd.exe", "/C", cmd)
} else {
execCmd = exec.CommandContext(ctx, "sh", "-c", cmd)
}
execCmd.Stdout = os.Stdout
execCmd.Stderr = os.Stderr
return execCmd.Run()
}

View File

@@ -1,4 +1,4 @@
package batchtftask
package batchtfile
import (
"context"
@@ -9,11 +9,11 @@ import (
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/retry"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/key"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"golang.org/x/sync/errgroup"
)
@@ -21,18 +21,23 @@ func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("batch_file[%s]", t.ID))
logger.Info("Starting batch file task")
t.Progress.OnStart(ctx, t)
workers := config.Cfg.Workers
workers := config.C().Workers
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(workers)
for _, elem := range t.Elems {
elem := elem
eg.Go(func() error {
t.processingMu.RLock()
if t.processing[elem.ID] != nil {
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
}
t.processingMu.RUnlock()
t.processingMu.Lock()
t.processing[elem.ID] = &elem
t.processingMu.Unlock()
defer func() {
t.processingMu.Lock()
delete(t.processing, elem.ID)
t.processingMu.Unlock()
}()
return t.processElement(gctx, elem)
})
@@ -61,10 +66,12 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
t.Progress.OnProgress(ctx, t)
})
errg.Go(func() error {
defer pw.Close()
logger.Info("Starting file download in stream mode")
_, err := tdler.NewDownloader(t.client, elem.File).Stream(uploadCtx, wr)
if closeErr := pw.CloseWithError(err); closeErr != nil {
logger.Errorf("Failed to close pipe writer: %v", closeErr)
_, err := tfile.NewDownloader(elem.File).Stream(uploadCtx, wr)
if err != nil {
logger.Errorf("Failed to download file: %v", err)
pw.CloseWithError(err)
}
return err
})
@@ -88,7 +95,7 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
t.downloaded.Add(int64(n))
t.Progress.OnProgress(ctx, t)
})
_, err = tdler.NewDownloader(t.client, elem.File).Parallel(ctx, wrAt)
_, err = tfile.NewDownloader(elem.File).Parallel(ctx, wrAt)
if err != nil {
return fmt.Errorf("failed to download file: %w", err)
}
@@ -104,7 +111,7 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
if err != nil {
return fmt.Errorf("failed to get file stat: %w", err)
}
vctx := context.WithValue(ctx, key.ContextKeyContentLength, fileStat.Size())
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
err = retry.Retry(func() error {
var file *os.File
file, err = os.Open(elem.localPath)
@@ -117,6 +124,6 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
return err
}
return nil
}, retry.Context(vctx), retry.RetryTimes(uint(config.Cfg.Retry)))
}, retry.Context(vctx), retry.RetryTimes(uint(config.C().Retry)))
return err
}

View File

@@ -1,4 +1,4 @@
package batchtftask
package batchtfile
import (
"context"

View File

@@ -1,13 +1,14 @@
package batchtftask
package batchtfile
import (
"context"
"fmt"
"path/filepath"
"sync"
"sync/atomic"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
@@ -29,10 +30,14 @@ type Task struct {
Progress ProgressTracker
IgnoreErrors bool // if true, errors during processing will be ignored
downloaded atomic.Int64
client tdler.Client
totalSize int64
processing map[string]TaskElementInfo
failed map[string]error // errors for each element
processingMu sync.RWMutex
failed map[string]error // [TODO] errors for each element
}
func (t *Task) Type() tasktype.TaskType {
return tasktype.TaskTypeTgfiles
}
func NewTaskElement(
@@ -42,8 +47,8 @@ func NewTaskElement(
) (*TaskElement, error) {
id := xid.New().String()
_, ok := stor.(storage.StorageCannotStream)
if !config.Cfg.Stream || ok {
cachePath, err := filepath.Abs(filepath.Join(config.Cfg.Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
if !config.C().Stream || ok {
cachePath, err := filepath.Abs(filepath.Join(config.C().Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
if err != nil {
return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
}
@@ -68,14 +73,12 @@ func NewBatchTGFileTask(
id string,
ctx context.Context,
files []TaskElement,
client tdler.Client,
progress ProgressTracker,
ignoreErrors bool,
) *Task {
task := &Task{
ID: id,
Ctx: ctx,
client: client,
Elems: files,
Progress: progress,
downloaded: atomic.Int64{},
@@ -88,6 +91,7 @@ func NewBatchTGFileTask(
}(),
processing: make(map[string]TaskElementInfo),
IgnoreErrors: ignoreErrors,
processingMu: sync.RWMutex{},
failed: make(map[string]error),
}
return task

View File

@@ -1,4 +1,4 @@
package batchtftask
package batchtfile
type TaskElementInfo interface {
FileName() string

View File

@@ -1,4 +1,4 @@
package batchtftask
package batchtfile
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值

View File

@@ -0,0 +1,139 @@
package parsed
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"path"
"path/filepath"
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/retry"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/parser"
"golang.org/x/sync/errgroup"
)
func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx)
logger.Infof("Starting Parsed item task %s", t.item.Title)
if t.progress != nil {
t.progress.OnStart(ctx, t)
}
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(config.C().Workers)
for _, resource := range t.item.Resources {
eg.Go(func() error {
t.processingMu.RLock()
if t.processing[resource.ID()] != nil {
return fmt.Errorf("resource %s is already being processed", resource.ID())
}
t.processingMu.RUnlock()
t.processingMu.Lock()
t.processing[resource.ID()] = &resource
t.processingMu.Unlock()
defer func() {
t.processingMu.Lock()
delete(t.processing, resource.URL)
t.processingMu.Unlock()
}()
err := t.processResource(gctx, resource)
t.downloaded.Add(1)
if errors.Is(err, context.Canceled) {
logger.Debug("Resource processing canceled")
return err
}
if err != nil {
logger.Errorf("Error processing resource %s: %v", resource.URL, err)
return fmt.Errorf("failed to process resource %s: %w", resource.URL, err)
}
return nil
})
}
err := eg.Wait()
if err != nil {
logger.Errorf("Error during Parsed item task execution: %v", err)
} else {
logger.Infof("Parsed item task %s completed successfully", t.item.Title)
}
if t.progress != nil {
t.progress.OnDone(ctx, t, err)
}
return err
}
func (t *Task) processResource(ctx context.Context, resource parser.Resource) error {
logger := log.FromContext(ctx)
err := retry.Retry(func() error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, resource.URL, nil)
if err != nil {
return err
}
if resource.Headers != nil {
for k, v := range resource.Headers {
req.Header.Set(k, v)
}
}
resp, err := t.httpClient.Do(req)
if err != nil {
return fmt.Errorf("failed to download resource %s: %w", resource.URL, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("failed to download resource %s: %s", resource.URL, resp.Status)
}
ctx = context.WithValue(ctx, ctxkey.ContentLength, func() int64 {
if resource.Size > 0 {
return resource.Size
}
return resp.ContentLength
}())
if t.stream {
return t.Stor.Save(ctx, resp.Body, path.Join(t.StorPath, resource.Filename))
}
cacheFile, err := fsutil.CreateFile(filepath.Join(config.C().Temp.BasePath,
fmt.Sprintf("resource_%s_%s", t.ID, resource.Filename)))
if err != nil {
return fmt.Errorf("failed to create cache file for resource %s: %w", resource.URL, err)
}
defer func() {
if err := cacheFile.CloseAndRemove(); err != nil {
logger.Errorf("Failed to close and remove cache file: %v", err)
}
}()
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
t.downloadedBytes.Add(int64(n))
if t.progress != nil {
t.progress.OnProgress(ctx, t)
}
})
copyResultCh := make(chan error, 1)
go func() {
_, err := io.Copy(wr, resp.Body)
copyResultCh <- err
}()
select {
case err := <-copyResultCh:
if err != nil {
return fmt.Errorf("failed to copy resource %s to cache file: %w", resource.URL, err)
}
case <-ctx.Done():
return ctx.Err()
}
_, err = cacheFile.Seek(0, 0)
if err != nil {
return fmt.Errorf("failed to seek cache file for resource %s: %w", resource.URL, err)
}
return t.Stor.Save(ctx, cacheFile, path.Join(t.StorPath, resource.Filename))
}, retry.Context(ctx), retry.RetryTimes(uint(config.C().Retry)))
if ctx.Err() != nil {
return ctx.Err()
}
return err
}

View File

@@ -0,0 +1,209 @@
package parsed
import (
"context"
"errors"
"fmt"
"sync/atomic"
"time"
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/slice"
"github.com/gotd/td/telegram/message/entity"
"github.com/gotd/td/telegram/message/styling"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值
stepPercent int // 每多少 % 更新一次
}{
{10 << 20, 100},
{50 << 20, 50},
{200 << 20, 20},
{500 << 20, 10},
}
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
for _, lvl := range progressUpdatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}
type ProgressTracker interface {
OnStart(ctx context.Context, info TaskInfo)
OnProgress(ctx context.Context, info TaskInfo)
OnDone(ctx context.Context, info TaskInfo, err error)
}
type Progress struct {
MessageID int
ChatID int64
start time.Time
lastUpdatePercent atomic.Int32
}
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
logger := log.FromContext(ctx)
p.start = time.Now()
p.lastUpdatePercent.Store(0)
logger.Debugf("Parsed task progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(fmt.Sprintf("开始下载 %s 的资源\n总大小: ", info.Site())),
styling.Code(fmt.Sprintf("%.2f MB (%d个资源)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
return
}
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
return
}
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
if p.lastUpdatePercent.Load() == int32(percent) {
return
}
p.lastUpdatePercent.Store(int32(percent))
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.DownloadedBytes(), info.TotalBytes())
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain("正在下载\n总大小: "),
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
styling.Plain("\n正在处理:\n"),
func() styling.StyledTextOption {
var lines []string
for _, elem := range info.Processing() {
lines = append(lines, fmt.Sprintf(" - %s (%.2f MB)", elem.FileName(), float64(elem.FileSize())/(1024*1024)))
}
if len(lines) == 0 {
lines = append(lines, " - 无")
}
return styling.Plain(slice.Join(lines, "\n"))
}(),
styling.Plain("\n平均速度: "),
styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(info.DownloadedBytes(), p.start)/(1024*1024))),
styling.Plain("\n当前进度: "),
styling.Bold(fmt.Sprintf("%.2f%%", float64(info.DownloadedBytes())/float64(info.TotalBytes())*100)),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
return
}
}
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
logger := log.FromContext(ctx)
if err != nil {
if errors.Is(err, context.Canceled) {
logger.Infof("Parsed task %s was canceled", info.TaskID())
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, &tg.MessagesEditMessageRequest{
ID: p.MessageID,
Message: fmt.Sprintf("处理已取消: %s", info.TaskID()),
})
}
} else {
logger.Errorf("Parsed task %s failed: %s", info.TaskID(), err)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, &tg.MessagesEditMessageRequest{
ID: p.MessageID,
Message: fmt.Sprintf("处理失败: %s", err.Error()),
})
}
}
return
}
logger.Infof("Parsed task %s completed successfully", info.TaskID())
entityBuilder := entity.Builder{}
if err := styling.Perform(&entityBuilder,
styling.Plain("处理完成, 资源数量: "),
styling.Code(fmt.Sprintf("%d", info.TotalResources())),
styling.Plain("\n保存路径: "),
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
); err != nil {
logger.Errorf("Failed to build entities: %s", err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
}
}
func NewProgress(messageID int, chatID int64) *Progress {
return &Progress{
MessageID: messageID,
ChatID: chatID,
}
}

80
core/tasks/parsed/task.go Normal file
View File

@@ -0,0 +1,80 @@
package parsed
import (
"context"
"net/http"
"sync"
"sync/atomic"
"github.com/krau/SaveAny-Bot/common/utils/netutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/parser"
"github.com/krau/SaveAny-Bot/storage"
)
type Task struct {
ID string
Ctx context.Context
Stor storage.Storage
StorPath string
item *parser.Item
httpClient *http.Client
progress ProgressTracker
stream bool
totalResources int64
downloaded atomic.Int64 // downloaded resources count
totalBytes int64 // total bytes to download
downloadedBytes atomic.Int64 // downloaded bytes count
processing map[string]ResourceInfo
processingMu sync.RWMutex
failed map[string]error // [TODO] errors for each resource
}
func (t *Task) Type() tasktype.TaskType {
return tasktype.TaskTypeParseditem
}
func (t *Task) TaskID() string {
return t.ID
}
func NewTask(
id string,
ctx context.Context,
stor storage.Storage,
storPath string,
item *parser.Item,
progressTracker ProgressTracker,
) *Task {
client := netutil.DefaultParserHTTPClient()
_, ok := stor.(storage.StorageCannotStream)
stream := config.C().Stream && !ok
return &Task{
ID: id,
Ctx: ctx,
Stor: stor,
StorPath: storPath,
item: item,
totalResources: int64(len(item.Resources)),
downloaded: atomic.Int64{},
totalBytes: func() int64 {
var total int64
for _, res := range item.Resources {
if res.Size < 0 {
continue // skip resources with unknown size
}
total += res.Size
}
return total
}(),
stream: stream,
downloadedBytes: atomic.Int64{},
httpClient: client,
progress: progressTracker,
processing: make(map[string]ResourceInfo),
processingMu: sync.RWMutex{},
failed: make(map[string]error),
}
}

View File

@@ -0,0 +1,51 @@
package parsed
type TaskInfo interface {
TaskID() string
Site() string
TotalResources() int64
Downloaded() int64
TotalBytes() int64
DownloadedBytes() int64
Processing() map[string]ResourceInfo
StorageName() string
StoragePath() string
}
func (t *Task) StoragePath() string {
return t.StorPath
}
func (t *Task) TotalResources() int64 {
return t.totalResources
}
func (t *Task) Downloaded() int64 {
return t.downloaded.Load()
}
func (t *Task) StorageName() string {
return t.Stor.Name()
}
func (t *Task) Site() string {
return t.item.Site
}
func (t *Task) TotalBytes() int64 {
return t.totalBytes
}
func (t *Task) DownloadedBytes() int64 {
return t.downloadedBytes.Load()
}
func (t *Task) Processing() map[string]ResourceInfo {
t.processingMu.RLock()
defer t.processingMu.RUnlock()
return t.processing
}
type ResourceInfo interface {
FileName() string
FileSize() int64
}

View File

@@ -1,4 +1,4 @@
package tphtask
package telegraph
import (
"context"
@@ -20,10 +20,8 @@ func (t *Task) Execute(ctx context.Context) error {
logger.Infof("Starting Telegraph task %s", t.PhPath)
t.progress.OnStart(ctx, t)
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(config.Cfg.Workers)
eg.SetLimit(config.C().Workers)
for i, pic := range t.Pics {
pic := pic
i := i
eg.Go(func() error {
err := t.processPic(gctx, pic, i)
if err != nil {
@@ -48,7 +46,7 @@ func (t *Task) Execute(ctx context.Context) error {
func (t *Task) processPic(ctx context.Context, picUrl string, index int) error {
retryOpts := []retry.Option{
retry.Context(ctx),
retry.RetryTimes(uint(config.Cfg.Retry)),
retry.RetryTimes(uint(config.C().Retry)),
}
var lastErr error
err := retry.Retry(func() error {
@@ -61,7 +59,7 @@ func (t *Task) processPic(ctx context.Context, picUrl string, index int) error {
defer body.Close()
filename := fmt.Sprintf("%d%s", index+1, path.Ext(picUrl))
if t.cannotStream {
cacheFile, err := fsutil.CreateFile(filepath.Join(config.Cfg.Temp.BasePath,
cacheFile, err := fsutil.CreateFile(filepath.Join(config.C().Temp.BasePath,
fmt.Sprintf("tph_%s_%s", t.TaskID(), filename),
))
if err != nil {
@@ -79,6 +77,11 @@ func (t *Task) processPic(ctx context.Context, picUrl string, index int) error {
lastErr = fmt.Errorf("failed to copy picture %s to cache file: %w", filename, lastErr)
return lastErr
}
_, err = cacheFile.Seek(0, 0)
if err != nil {
lastErr = fmt.Errorf("failed to seek cache file for picture %s: %w", filename, err)
return lastErr
}
lastErr = t.Stor.Save(ctx, cacheFile, path.Join(t.StorPath, filename))
} else {
lastErr = t.Stor.Save(ctx, body, path.Join(t.StorPath, filename))

View File

@@ -1,4 +1,4 @@
package tphtask
package telegraph
import (
"context"

View File

@@ -1,9 +1,10 @@
package tphtask
package telegraph
import (
"context"
"sync/atomic"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
"github.com/krau/SaveAny-Bot/storage"
)
@@ -23,6 +24,10 @@ type Task struct {
downloaded atomic.Int64
}
func (t *Task) Type() tasktype.TaskType {
return tasktype.TaskTypeTphpics
}
func NewTask(
id string,
ctx context.Context,
@@ -34,7 +39,7 @@ func NewTask(
progress ProgressTracker,
) *Task {
_, cannotStream := stor.(storage.StorageCannotStream)
tphtask := &Task{
telegraph := &Task{
ID: id,
Ctx: ctx,
PhPath: phPath,
@@ -47,5 +52,5 @@ func NewTask(
totalpics: len(pics),
downloaded: atomic.Int64{},
}
return tphtask
return telegraph
}

View File

@@ -1,4 +1,4 @@
package tphtask
package telegraph
type TaskInfo interface {
TaskID() string

View File

@@ -1,4 +1,4 @@
package tphtask
package telegraph
func shouldUpdateProgress(downloaded int64, total int64) bool {
if total <= 0 || downloaded <= 0 {

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
import (
"context"
@@ -8,15 +8,17 @@ import (
"time"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/key"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
func (t *TGFileTask) Execute(ctx context.Context) error {
func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", t.File.Name()))
t.Progress.OnStart(ctx, t)
if t.Progress != nil {
t.Progress.OnStart(ctx, t)
}
if t.stream {
return executeStream(ctx, t)
}
@@ -34,9 +36,11 @@ func (t *TGFileTask) Execute(ctx context.Context) error {
wrAt := newWriterAt(ctx, localFile, t.Progress, t)
defer func() {
t.Progress.OnDone(ctx, t, err)
if t.Progress != nil {
t.Progress.OnDone(ctx, t, err)
}
}()
_, err = tdler.NewDownloader(t.client, t.File).Parallel(ctx, wrAt)
_, err = tfile.NewDownloader(t.File).Parallel(ctx, wrAt)
if err != nil {
return fmt.Errorf("failed to download file: %w", err)
}
@@ -52,8 +56,8 @@ func (t *TGFileTask) Execute(ctx context.Context) error {
if err != nil {
return fmt.Errorf("failed to get file stat: %w", err)
}
vctx := context.WithValue(ctx, key.ContextKeyContentLength, fileStat.Size())
for i := range config.Cfg.Retry + 1 {
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
for i := range config.C().Retry + 1 {
if err = vctx.Err(); err != nil {
return fmt.Errorf("context canceled while saving file: %w", err)
}
@@ -64,7 +68,7 @@ func (t *TGFileTask) Execute(ctx context.Context) error {
}
defer file.Close()
if err = t.Storage.Save(vctx, file, t.Path); err != nil {
if i == config.Cfg.Retry {
if i == config.C().Retry {
return fmt.Errorf("failed to save file: %w", err)
}
logger.Errorf("Failed to save file: %s, retrying...", err)

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
import (
"context"

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
import (
"context"
@@ -6,11 +6,11 @@ import (
"io"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"golang.org/x/sync/errgroup"
)
func executeStream(ctx context.Context, task *TGFileTask) error {
func executeStream(ctx context.Context, task *Task) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", task.File.Name()))
pr, pw := io.Pipe()
@@ -21,16 +21,20 @@ func executeStream(ctx context.Context, task *TGFileTask) error {
})
wr := newWriter(ctx, pw, task.Progress, task)
errg.Go(func() error {
defer pw.Close()
logger.Info("Starting file download in stream mode")
_, err := tdler.NewDownloader(task.client, task.File).Stream(uploadCtx, wr)
if closeErr := pw.CloseWithError(err); closeErr != nil {
logger.Errorf("Failed to close pipe writer: %v", closeErr)
_, err := tfile.NewDownloader(task.File).Stream(uploadCtx, wr)
if err != nil {
logger.Errorf("Failed to download file: %v", err)
pw.CloseWithError(err)
}
return err
})
var err error
defer func() {
task.Progress.OnDone(ctx, task, err)
if task.Progress != nil {
task.Progress.OnDone(ctx, task, err)
}
}()
if err = errg.Wait(); err != nil {
return err

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
type TaskInfo interface {
TaskID() string
@@ -8,22 +8,22 @@ type TaskInfo interface {
StorageName() string
}
func (t *TGFileTask) TaskID() string {
func (t *Task) TaskID() string {
return t.ID
}
func (t *TGFileTask) FileName() string {
func (t *Task) FileName() string {
return t.File.Name()
}
func (t *TGFileTask) FileSize() int64 {
func (t *Task) FileSize() int64 {
return t.File.Size()
}
func (t *TGFileTask) StoragePath() string {
func (t *Task) StoragePath() string {
return t.Path
}
func (t *TGFileTask) StorageName() string {
func (t *Task) StorageName() string {
return t.Storage.Name()
}

View File

@@ -1,59 +1,59 @@
package tftask
package tfile
import (
"context"
"fmt"
"path/filepath"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
)
type TGFileTask struct {
type Task struct {
ID string
Ctx context.Context
File tfile.TGFile
Storage storage.Storage
Path string
Progress ProgressTracker
client tdler.Client
stream bool // true if the file should be downloaded in stream mode
localPath string
}
func (t *Task) Type() tasktype.TaskType {
return tasktype.TaskTypeTgfiles
}
func NewTGFileTask(
id string,
ctx context.Context,
file tfile.TGFile,
client tdler.Client,
stor storage.Storage,
path string,
progress ProgressTracker,
) (*TGFileTask, error) {
) (*Task, error) {
_, ok := stor.(storage.StorageCannotStream)
if !config.Cfg.Stream || ok {
cachePath, err := filepath.Abs(filepath.Join(config.Cfg.Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
if !config.C().Stream || ok {
cachePath, err := filepath.Abs(filepath.Join(config.C().Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
if err != nil {
return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
}
tftask := &TGFileTask{
tfile := &Task{
ID: id,
Ctx: ctx,
client: client,
File: file,
Storage: stor,
Path: path,
Progress: progress,
localPath: cachePath,
}
return tftask, nil
return tfile, nil
}
tfileTask := &TGFileTask{
tfileTask := &Task{
ID: id,
Ctx: ctx,
client: client,
File: file,
Storage: stor,
Path: path,

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值

View File

@@ -1,4 +1,4 @@
package tftask
package tfile
import (
"context"
@@ -20,7 +20,9 @@ func (w *ProgressWriterAt) WriteAt(p []byte, off int64) (int, error) {
if err != nil {
return 0, err
}
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
if w.progress != nil {
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
}
return at, nil
}
@@ -54,7 +56,9 @@ func (w *ProgressWriter) Write(p []byte) (int, error) {
if err != nil {
return 0, err
}
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
if w.progress != nil {
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
}
return at, nil
}

39
database/chat.go Normal file
View File

@@ -0,0 +1,39 @@
package database
import "context"
func (user *User) WatchChat(ctx context.Context, chat WatchChat) error {
if len(user.WatchChats) == 0 {
user.WatchChats = make([]WatchChat, 0)
}
user.WatchChats = append(user.WatchChats, chat)
return db.WithContext(ctx).Save(user.WatchChats).Error
}
func (user *User) UnwatchChat(ctx context.Context, chatID int64) error {
var watchChat WatchChat
err := db.WithContext(ctx).Where("chat_id = ? AND user_id = ?", chatID, user.ID).First(&watchChat).Error
if err != nil {
return err
}
return db.WithContext(ctx).Unscoped().Delete(&watchChat).Error
}
func (user *User) WatchingChat(ctx context.Context, chatID int64) (bool, error) {
var count int64
err := db.WithContext(ctx).Model(&WatchChat{}).Where("chat_id = ? AND user_id = ?", chatID, user.ID).Count(&count).Error
if err != nil {
return false, err
}
return count > 0, nil
}
func GetWatchChatsByChatID(ctx context.Context, chatID int64) ([]*WatchChat, error) {
var watchChats []*WatchChat
err := db.WithContext(ctx).Where("chat_id = ?", chatID).Find(&watchChats).Error
if err != nil {
return nil, err
}
return watchChats, nil
}

View File

@@ -19,11 +19,11 @@ var db *gorm.DB
func Init(ctx context.Context) {
logger := log.FromContext(ctx)
if err := os.MkdirAll(filepath.Dir(config.Cfg.DB.Path), 0755); err != nil {
if err := os.MkdirAll(filepath.Dir(config.C().DB.Path), 0755); err != nil {
logger.Fatal("Failed to create data directory: ", err)
}
var err error
db, err = gorm.Open(gormlite.Open(config.Cfg.DB.Path), &gorm.Config{
db, err = gorm.Open(gormlite.Open(config.C().DB.Path), &gorm.Config{
Logger: glogger.New(logger, glogger.Config{
Colorful: true,
SlowThreshold: time.Second * 5,
@@ -37,7 +37,7 @@ func Init(ctx context.Context) {
logger.Fatal("Failed to open database: ", err)
}
logger.Debug("Database connected")
if err := db.AutoMigrate(&User{}, &Dir{}, &Rule{}); err != nil {
if err := db.AutoMigrate(&User{}, &Dir{}, &Rule{}, &WatchChat{}); err != nil {
logger.Fatal("迁移数据库失败, 如果您从旧版本升级, 建议手动删除数据库文件后重试: ", err)
}
if err := syncUsers(ctx); err != nil {
@@ -60,7 +60,7 @@ func syncUsers(ctx context.Context) error {
}
cfgUserMap := make(map[int64]struct{})
for _, u := range config.Cfg.Users {
for _, u := range config.C().Users {
cfgUserMap[u.ID] = struct{}{}
}

View File

@@ -6,12 +6,22 @@ import (
type User struct {
gorm.Model
ChatID int64 `gorm:"uniqueIndex;not null"`
Silent bool
DefaultStorage string
Dirs []Dir
ApplyRule bool
Rules []Rule
ChatID int64 `gorm:"uniqueIndex;not null"`
Silent bool
DefaultStorage string
Dirs []Dir
ApplyRule bool
Rules []Rule
WatchChats []WatchChat
FilenameStrategy string
FilenameTemplate string
}
type WatchChat struct {
gorm.Model
UserID uint // User's database ID (not chat ID)
ChatID int64
Filter string
}
type Dir struct {

View File

@@ -1,6 +1,10 @@
package database
import "context"
import (
"context"
"gorm.io/gorm/clause"
)
func CreateUser(ctx context.Context, chatID int64) error {
if _, err := GetUserByChatID(ctx, chatID); err == nil {
@@ -11,19 +15,16 @@ func CreateUser(ctx context.Context, chatID int64) error {
func GetAllUsers(ctx context.Context) ([]User, error) {
var users []User
err := db.Preload("Dirs").
WithContext(ctx).
Preload("Rules").
err := db.WithContext(ctx).
Preload(clause.Associations).
Find(&users).Error
return users, err
}
func GetUserByChatID(ctx context.Context, chatID int64) (*User, error) {
var user User
err := db.
Preload("Dirs").
WithContext(ctx).
Preload("Rules").
err := db.WithContext(ctx).
Preload(clause.Associations).
Where("chat_id = ?", chatID).First(&user).Error
return &user, err
}
@@ -36,5 +37,16 @@ func UpdateUser(ctx context.Context, user *User) error {
}
func DeleteUser(ctx context.Context, user *User) error {
return db.WithContext(ctx).Unscoped().Select("Dirs", "Rules").Delete(user).Error
return db.WithContext(ctx).
Unscoped().
Select(clause.Associations).
Delete(user).Error
}
func GetUserByID(ctx context.Context, id uint) (*User, error) {
var user User
err := db.WithContext(ctx).
Preload(clause.Associations).
Where("id = ?", id).First(&user).Error
return &user, err
}

1
docs/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
public/

View File

@@ -0,0 +1,5 @@
+++
date = '{{ .Date }}'
draft = true
title = '{{ replace .File.ContentBaseName "-" " " | title }}'
+++

View File

@@ -0,0 +1 @@
$font-size-base: 18px;

31
docs/content/en/_index.md Normal file
View File

@@ -0,0 +1,31 @@
---
title: Introduction
---
# Save Any Bot
![](https://img.shields.io/github/go-mod/go-version/krau/SaveAny-Bot?style=flat-square)
![](https://img.shields.io/github/license/krau/SaveAny-Bot?style=flat-square)
![](https://img.shields.io/github/v/release/krau/SaveAny-Bot?color=cyan&style=flat-square)
![](https://img.shields.io/github/downloads/krau/SaveAny-Bot/total?style=flat-square)
Save Any Bot is a tool that allows you to save files from Telegram to various storage backends.
## Features
- Supports documents/videos/images/stickers... and even Telegraph
- Breaks restrictions on saving files
- Batch download
- Streaming
- Multi-user
- Automatic organization based on storage rules
- Supports multiple storage backends:
- Alist
- Minio (S3 compatible)
- WebDAV
- Telegram (re-upload to specified chat)
- Local disk
## [Contributors](https://github.com/krau/SaveAny-Bot/graphs/contributors)
![Contributors](https://contrib.rocks/image?repo=krau/SaveAny-Bot&max=750&columns=20)

Some files were not shown because too many files have changed in this diff Show More