feat: parse url with js plugins support (#96)

* feat: WIP. add parser functionality and text message handling

* fix: use json to marshal js result

* feat: add metadata handling and version validation for jsParser

* refactor: rename parser package to parsers and restructure parser handling

* refactor: core code struct and impl parse task handle

* feat: impl parsed download

* fix: seek cache file when processing tph picture

* feat: implement parsed task handling and progress tracking

* feat: enhance task processing with concurrency control and progress tracking

* feat: add resource ID generation and improve resource processing handling

* feat: improve message formatting in parsed text and progress completion

* feat: add example js plugin

* feat: implement Twitter parser

* fix: twitter parse video json decode error

* feat: impl stream mode for parse task

.dockerignore

@@ -6,6 +6,6 @@
 downloads/
 data/
 cache/
-docs
+docs/
 config.example.toml
 docker-compose.*

.github/FUNDING.yml

@@ -1,5 +1,5 @@
 # These are supported funding model platforms
 
 custom: [
-  "https://afdian.com/a/acherkrau"
+  "https://afdian.com/a/unvapp"
 ]

.github/workflows/build-docker.yml

@@ -29,13 +29,7 @@ jobs:
             type=semver,pattern={{version}}
             type=semver,pattern={{major}}.{{minor}}
             type=sha
-            type=raw,value=latest
-            type=ref,event=branch
-            type=ref,event=tag
-          labels: |
-            org.opencontainers.image.title=${{ env.IMAGE_NAME }}
-            org.opencontainers.image.source=https://github.com/krau/SaveAny-Bot
-            org.opencontainers.image.url=https://github.com/krau/SaveAny-Bot
+            type=raw,value=latest,enable={{is_default_branch}}
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3
@@ -50,23 +44,26 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Extract version from Git Ref
-        id: extract_version
+      - name: Extract Dockerfile args
+        id: args
         run: |
-          VERSION=$(echo "${{ github.ref }}" | sed 's/refs\/tags\/v//')
-          echo "VERSION=${VERSION}" >> $GITHUB_ENV
+          echo "git_commit=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT"
+          echo "build_time=$(git show -s --format=%cI)" >> "$GITHUB_OUTPUT"
 
       - name: Build and push Docker image
+        id: build-and-push
         uses: docker/build-push-action@v6
         with:
           context: .
           platforms: linux/amd64,linux/arm64
-          cache-from: type=gha
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: |
+            type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
+            type=gha
           cache-to: type=gha,mode=max
           build-args: |
             VERSION=${{ steps.meta.outputs.version }}
-            GitCommit=${{ github.sha }}
-            BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}
-          push: true
-          tags: ${{ steps.meta.outputs.tags }}
-          labels: ${{ steps.meta.outputs.labels }}
+            GitCommit=${{ steps.args.outputs.git_commit }}
+            BuildTime=${{ steps.args.outputs.build_time }}

.github/workflows/docs.yml

@@ -6,17 +6,31 @@ on:
     paths:
       - "docs/**"
   workflow_dispatch:
+
 jobs:
   deploy:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
-      - uses: actions/cache@v4
+          submodules: true  # Fetch Hugo themes (true OR recursive)
+          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod
+
+      - name: Setup Hugo
+        uses: peaceiris/actions-hugo@v3
         with:
-          key: ${{ github.ref }}
-          path: .cache
-      - run: pip install mkdocs-material
-      - run: cd docs && mkdocs gh-deploy --force
+          hugo-version: '0.147.8'
+          extended: true
+
+      - name: Build
+        run: hugo --minify --destination public --source docs
+
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        if: github.ref == 'refs/heads/main'
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/public
+          publish_branch: gh-pages

.gitignore

@@ -6,4 +6,5 @@ downloads/
 session.*
 cache.db
 .vscode/
-temp/
+temp/
+.hugo_build.lock

Dockerfile

@@ -6,22 +6,33 @@ ARG BuildTime="Unknown"
 
 WORKDIR /app
 
-COPY . .
+COPY go.mod go.sum ./
+RUN --mount=type=cache,target=/go/pkg/mod \
+    go mod download
 
+COPY . .
 RUN --mount=type=cache,target=/root/.cache/go-build \
     --mount=type=cache,target=/go/pkg \
-    CGO_ENABLED=0 \ 
-    go build -trimpath \ 
-    -ldflags "-s -w \ 
-    -X github.com/krau/SaveAny-Bot/common.Version=${VERSION} \
-    -X github.com/krau/SaveAny-Bot/common.GitCommit=${GiTCommit} \
-    -X github.com/krau/SaveAny-Bot/common.BuildTime=${BuildTime}" \
+    CGO_ENABLED=0 \
+    go build -trimpath \
+    -ldflags=" \
+    -s -w \
+    -X 'github.com/krau/SaveAny-Bot/common.Version=${VERSION}' \
+    -X 'github.com/krau/SaveAny-Bot/common.GitCommit=${GitCommit}' \
+    -X 'github.com/krau/SaveAny-Bot/common.BuildTime=${BuildTime}' \
+    " \
     -o saveany-bot .
 
 FROM alpine:latest
 
+RUN apk add --no-cache curl
+
 WORKDIR /app
 
 COPY --from=builder /app/saveany-bot .
+COPY entrypoint.sh .
 
-ENTRYPOINT  ["/app/saveany-bot"]
+RUN chmod +x /app/saveany-bot && \
+    chmod +x /app/entrypoint.sh
+
+ENTRYPOINT ["/app/entrypoint.sh"]

README.md

@@ -1,28 +1,39 @@
 <div align="center">
 
-# <img src="docs/logo.jpg" width="45" align="center"> Save Any Bot
+# <img src="docs/static/logo.png" width="45" align="center"> Save Any Bot
 
-**简体中文** | [English](README_EN.md)
+**简体中文** | [English](https://sabot.unv.app/en/)
 
-把 Telegram 的文件保存到各类存储端.
-
-> _就像 PikPak Bot 一样_
+把 Telegram 上的文件转存到多种存储端.
 
 </div>
 
-## [部署](https://sabot.unv.app/deploy/)
+## 部署
 
-## [参与开发](https://sabot.unv.app/contribute/)
+请参考 [部署文档](https://sabot.unv.app/deployment/installation/)
 
----
+## Features
 
-## 赞助
+- 支持文档/视频/图片/贴纸… 甚至还有 Telegraph
+- 破解禁止保存的文件
+- 批量下载
+- 流式传输
+- 多用户
+- 基于存储规则的自动整理
+- 支持多种存储端:
+  - Alist
+  - Minio (S3 兼容)
+  - WebDAV
+  - Telegram (重传回指定聊天)
+  - 本地磁盘
+
+## Sponsors
 
 本项目受到 [YxVM](https://yxvm.com/) 与 [NodeSupport](https://github.com/NodeSeekDev/NodeSupport) 的支持.
 
 如果这个项目对你有帮助, 你可以考虑通过以下方式赞助我:
 
-- [爱发电](https://afdian.com/a/acherkrau)
+- [爱发电](https://afdian.com/a/unvapp)
 
 ## Contributors
 
@@ -37,6 +48,13 @@
                     <sub><b>Krau</b></sub>
                 </a>
             </td>
+            <td align="center">
+                <a href="https://github.com/Silentely">
+                    <img src="https://avatars.githubusercontent.com/u/22141172?v=4" width="100;" alt="Silentely"/>
+                    <br />
+                    <sub><b>Abner</b></sub>
+                </a>
+            </td>
             <td align="center">
                 <a href="https://github.com/TG-Twilight">
                     <img src="https://avatars.githubusercontent.com/u/121682528?v=4" width="100;" alt="TG-Twilight"/>
@@ -52,8 +70,8 @@
                 </a>
             </td>
             <td align="center">
-                <a href="https://github.com/ahcorn">
-                    <img src="https://avatars.githubusercontent.com/u/42889600?v=4" width="100;" alt="ahcorn"/>
+                <a href="https://github.com/AHCorn">
+                    <img src="https://avatars.githubusercontent.com/u/42889600?v=4" width="100;" alt="AHCorn"/>
                     <br />
                     <sub><b>安和</b></sub>
                 </a>
@@ -68,4 +86,5 @@
 - [gotd](https://github.com/gotd/td)
 - [TG-FileStreamBot](https://github.com/EverythingSuckz/TG-FileStreamBot)
 - [gotgproto](https://github.com/celestix/gotgproto)
+- [tdl](https://github.com/iyear/tdl)
 - All the dependencies

README_EN.md

@@ -1,108 +0,0 @@
-<div align="center">
-
-# <img src="docs/logo.jpg" width="45" align="center"> Save Any Bot
-
-[简体中文](README.md) | **English**
-
-Save Telegram files to various storage endpoints.
-
-> _Just like PikPak Bot_
-
-</div>
-
-## Deployment
-
-### Deploy from Binary
-
-Download the binary file for your platform from the [Release](https://github.com/krau/SaveAny-Bot/releases) page.
-
-Create a `config.toml` file in the extracted directory, refer to [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) for configuration.
-
-Run:
-
-```bash
-chmod +x saveany-bot
-./saveany-bot
-```
-
-#### Add as systemd Service
-
-Create file `/etc/systemd/system/saveany-bot.service` and write the following content:
-
-```
-[Unit]
-Description=SaveAnyBot
-After=systemd-user-sessions.service
-
-[Service]
-Type=simple
-WorkingDirectory=/yourpath/
-ExecStart=/yourpath/saveany-bot
-Restart=on-failure
-
-[Install]
-WantedBy=multi-user.target
-```
-
-Enable auto-start and start the service:
-
-```bash
-systemctl enable --now saveany-bot
-```
-
-### Deploy with Docker
-
-#### Docker Compose
-
-Download [docker-compose.yml](https://github.com/krau/SaveAny-Bot/blob/main/docker-compose.yml) file and create a `config.toml` file in the same directory, refer to [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) for configuration.
-
-Run:
-
-```bash
-docker compose up -d
-```
-
-#### Docker
-
-```shell
-docker run -d --name saveany-bot \
-    -v /path/to/config.toml:/app/config.toml \
-    -v /path/to/downloads:/app/downloads \
-    ghcr.io/krau/saveany-bot:latest
-```
-
-## Update
-
-Use `upgrade` or `up` command to upgrade to the latest version:
-
-```bash
-./saveany-bot upgrade
-```
-
-If deployed with Docker, use the following commands to update:
-
-```bash
-docker pull ghcr.io/krau/saveany-bot:latest
-docker restart saveany-bot
-```
-
-## Usage
-
-Send (forward) files to the Bot and follow the prompts.
-
----
-
-## Sponsors
-
-This project is supported by [YxVM](https://yxvm.com/) and [NodeSupport](https://github.com/NodeSeekDev/NodeSupport).
-
-You can consider sponsoring me if this project helps you:
-
-- [Afdian](https://afdian.com/a/acherkrau)
-
-## Thanks
-
-- [gotd](https://github.com/gotd/td)
-- [TG-FileStreamBot](https://github.com/EverythingSuckz/TG-FileStreamBot)
-- [gotgproto](https://github.com/celestix/gotgproto)
-- All the dependencies

client/bot/bot.go

@@ -2,7 +2,6 @@ package bot
 
 import (
 	"context"
-	"net/url"
 	"time"
 
 	"github.com/celestix/gotgproto"
@@ -14,21 +13,12 @@ import (
 	"github.com/gotd/td/tg"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers"
 	"github.com/krau/SaveAny-Bot/client/middleware"
+	"github.com/krau/SaveAny-Bot/common/utils/netutil"
 	"github.com/krau/SaveAny-Bot/config"
 	"github.com/ncruces/go-sqlite3/gormlite"
 	"golang.org/x/net/proxy"
 )
 
-var Client *gotgproto.Client
-
-func newProxyDialer(proxyUrl string) (proxy.Dialer, error) {
-	url, err := url.Parse(proxyUrl)
-	if err != nil {
-		return nil, err
-	}
-	return proxy.FromURL(url, proxy.Direct)
-}
-
 func Init(ctx context.Context) {
 	log.FromContext(ctx).Info("初始化 Bot...")
 	resultChan := make(chan struct {
@@ -38,7 +28,7 @@ func Init(ctx context.Context) {
 	go func() {
 		var resolver dcs.Resolver
 		if config.Cfg.Telegram.Proxy.Enable && config.Cfg.Telegram.Proxy.URL != "" {
-			dialer, err := newProxyDialer(config.Cfg.Telegram.Proxy.URL)
+			dialer, err := netutil.NewProxyDialer(config.Cfg.Telegram.Proxy.URL)
 			if err != nil {
 				resultChan <- struct {
 					client *gotgproto.Client
@@ -52,7 +42,8 @@ func Init(ctx context.Context) {
 		} else {
 			resolver = dcs.DefaultResolver()
 		}
-		client, err := gotgproto.NewClient(config.Cfg.Telegram.AppID,
+		client, err := gotgproto.NewClient(
+			config.Cfg.Telegram.AppID,
 			config.Cfg.Telegram.AppHash,
 			gotgproto.ClientTypeBot(config.Cfg.Telegram.Token),
 			&gotgproto.ClientOpts{
@@ -79,17 +70,22 @@ func Init(ctx context.Context) {
 		client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
 			Scope: &tg.BotCommandScopeDefault{},
 		})
+		commands := []tg.BotCommand{
+			{Command: "start", Description: "开始使用"},
+			{Command: "help", Description: "显示帮助"},
+			{Command: "silent", Description: "开启/关闭静默模式"},
+			{Command: "storage", Description: "设置默认存储端"},
+			{Command: "save", Description: "保存文件"},
+			{Command: "dir", Description: "管理存储文件夹"},
+			{Command: "rule", Description: "管理规则"},
+		}
+		if config.Cfg.Telegram.Userbot.Enable {
+			commands = append(commands, tg.BotCommand{Command: "watch", Description: "监听聊天"})
+			commands = append(commands, tg.BotCommand{Command: "unwatch", Description: "取消监听聊天"})
+		}
 		_, err = client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
-			Scope: &tg.BotCommandScopeDefault{},
-			Commands: []tg.BotCommand{
-				{Command: "start", Description: "开始使用"},
-				{Command: "help", Description: "显示帮助"},
-				{Command: "silent", Description: "开启/关闭静默模式"},
-				{Command: "storage", Description: "设置默认存储端"},
-				{Command: "save", Description: "保存所回复的文件"},
-				{Command: "dir", Description: "管理存储文件夹"},
-				{Command: "rule", Description: "管理规则"},
-			},
+			Scope:    &tg.BotCommandScopeDefault{},
+			Commands: commands,
 		})
 		resultChan <- struct {
 			client *gotgproto.Client
@@ -104,8 +100,7 @@ func Init(ctx context.Context) {
 		if result.err != nil {
 			log.FromContext(ctx).Fatalf("初始化 Bot 失败: %s", result.err)
 		}
-		Client = result.client
-		handlers.Register(Client.Dispatcher)
+		handlers.Register(result.client.Dispatcher)
 		log.FromContext(ctx).Info("Bot 初始化完成")
 	}
 }

client/bot/handlers/add_task.go

@@ -72,7 +72,9 @@ func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
 		}
 		return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files[0], msgID)
 	case tasktype.TaskTypeTphpics:
-		return shortcut.CreateAndAddTphTaskWithEdit(ctx, userID, data.TphPageNode, data.TphDirPath, data.TphPics, selectedStorage, msgID)
+		return shortcut.CreateAndAddtelegraphWithEdit(ctx, userID, data.TphPageNode, data.TphDirPath, data.TphPics, selectedStorage, msgID)
+	case tasktype.TaskTypeParseditem:
+		shortcut.CreateAndAddParsedTaskWithEdit(ctx, selectedStorage, dirPath, data.ParsedItem, msgID, userID)
 	default:
 		log.FromContext(ctx).Errorf("Unsupported task type: %s", data.TaskType)
 	}

client/bot/handlers/help.go

@@ -12,19 +12,22 @@ func handleHelpCmd(ctx *ext.Context, update *ext.Update) error {
 	const helpText string = `
 Save Any Bot - 转存你的 Telegram 文件
 版本: %s , 提交: %s
+
 命令:
 /start - 开始使用
 /help - 显示帮助
 /silent - 开关静默模式
 /storage - 设置默认存储位置
 /save [自定义文件名] - 保存文件
+/dir - 管理存储目录
+/rule - 管理规则
 
-静默模式: 开启后 Bot 直接保存到收到的文件到默认位置, 不再询问
-
-默认存储位置: 在静默模式下保存到的位置
-
-向 Bot 发送(转发)文件, 或发送一个公开频道的消息链接以保存文件
+使用帮助: https://sabot.unv.app/usage/
 `
-	ctx.Reply(update, ext.ReplyTextString(fmt.Sprintf(helpText, consts.Version, consts.GitCommit)), nil)
+	shortHash := consts.GitCommit
+	if len(shortHash) > 7 {
+		shortHash = shortHash[:7]
+	}
+	ctx.Reply(update, ext.ReplyTextString(fmt.Sprintf(helpText, consts.Version, shortHash)), nil)
 	return dispatcher.EndGroups
 }

client/bot/handlers/link.go

@@ -38,8 +38,7 @@ func handleMessageLink(ctx *ext.Context, update *ext.Update) error {
 		editReplied("构建存储选择键盘失败: "+err.Error(), nil)
 		return dispatcher.EndGroups
 	}
-	editReplied(fmt.Sprintf("找到 %d 个文件, 请选择存储位置", len(files)),
-		markup)
+	editReplied(fmt.Sprintf("找到 %d 个文件, 请选择存储位置", len(files)), markup)
 	return dispatcher.EndGroups
 }
 

client/bot/handlers/media.go

@@ -1,18 +1,32 @@
 package handlers
 
 import (
+	"fmt"
+	"sync"
+	"time"
+
 	"github.com/celestix/gotgproto/dispatcher"
 	"github.com/celestix/gotgproto/ext"
 	"github.com/charmbracelet/log"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+	"github.com/krau/SaveAny-Bot/pkg/tcbdata"
+	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"github.com/krau/SaveAny-Bot/storage"
 )
 
 func handleMediaMessage(ctx *ext.Context, update *ext.Update) error {
 	logger := log.FromContext(ctx)
 	message := update.EffectiveMessage.Message
+	groupID, isGroup := message.GetGroupedID()
+	if isGroup && groupID != 0 {
+		return handleGroupMediaMessage(ctx, update, message, groupID)
+	}
 	logger.Debugf("Got media: %s", message.Media.TypeName())
+
 	msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message)
 	if err != nil {
 		return err
@@ -38,6 +52,10 @@ func handleSilentSaveMedia(ctx *ext.Context, update *ext.Update) error {
 		return dispatcher.EndGroups
 	}
 	message := update.EffectiveMessage.Message
+	groupID, isGroup := message.GetGroupedID()
+	if isGroup && groupID != 0 {
+		return handleGroupMediaMessage(ctx, update, message, groupID)
+	}
 	logger.Debugf("Got media: %s", message.Media.TypeName())
 	userID := update.GetUserChat().GetID()
 	msg, file, err := shortcut.GetFileFromMessageWithReply(ctx, update, message)
@@ -46,3 +64,96 @@ func handleSilentSaveMedia(ctx *ext.Context, update *ext.Update) error {
 	}
 	return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, stor, "", file, msg.ID)
 }
+
+type MediaGroupHandler struct {
+	groups  map[int64][]tfile.TGFileMessage
+	timers  map[int64]*time.Timer
+	mu      sync.Mutex
+	timeout time.Duration
+}
+
+var mediaGroupHandler = &MediaGroupHandler{
+	groups:  make(map[int64][]tfile.TGFileMessage),
+	timers:  make(map[int64]*time.Timer),
+	timeout: 1 * time.Second,
+}
+
+func handleGroupMediaMessage(ctx *ext.Context, update *ext.Update, message *tg.Message, groupID int64) error {
+	logger := log.FromContext(ctx)
+	media := message.Media
+	supported := mediautil.IsSupported(media)
+	if !supported {
+		return dispatcher.EndGroups
+	}
+	file, err := tfile.FromMediaMessage(media, ctx.Raw, message, tfile.WithNameIfEmpty(
+		tgutil.GenFileNameFromMessage(*message),
+	))
+	if err != nil {
+		logger.Errorf("Failed to get file from media: %s", err)
+		return dispatcher.EndGroups
+	}
+	mediaGroupHandler.mu.Lock()
+	defer mediaGroupHandler.mu.Unlock()
+	if mediaGroupHandler.groups[groupID] == nil {
+		mediaGroupHandler.groups[groupID] = make([]tfile.TGFileMessage, 0)
+	}
+	mediaGroupHandler.groups[groupID] = append(mediaGroupHandler.groups[groupID], file)
+
+	if timer, exists := mediaGroupHandler.timers[groupID]; exists {
+		timer.Stop()
+	}
+	mediaGroupHandler.timers[groupID] = time.AfterFunc(mediaGroupHandler.timeout, func() {
+		processMediaGroup(ctx, update, groupID)
+	})
+	return dispatcher.EndGroups
+}
+
+func processMediaGroup(ctx *ext.Context, update *ext.Update, groupID int64) {
+	logger := log.FromContext(ctx)
+	mediaGroupHandler.mu.Lock()
+	items := mediaGroupHandler.groups[groupID]
+	delete(mediaGroupHandler.groups, groupID)
+	delete(mediaGroupHandler.timers, groupID)
+	mediaGroupHandler.mu.Unlock()
+	if len(items) == 0 {
+		logger.Warn("No media items to process for group", "groupID", groupID)
+		return
+	}
+	logger.Debugf("Processing media group %d with %d items", groupID, len(items))
+
+	userId := update.GetUserChat().GetID()
+	msg, err := ctx.Reply(update, ext.ReplyTextString("正在保存文件..."), nil)
+	if err != nil {
+		logger.Errorf("Failed to reply: %s", err)
+		return
+	}
+	stor := storage.FromContext(ctx)
+	if stor != nil {
+		// In silent mode
+		if len(items) == 1 {
+			shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userId, stor, "", items[0], msg.ID)
+			return
+		}
+		shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, userId, stor, "", items, msg.ID)
+		return
+	}
+
+	stors := storage.GetUserStorages(ctx, userId)
+	markup, err := msgelem.BuildAddSelectStorageKeyboard(stors, tcbdata.Add{
+		Files:   items,
+		AsBatch: len(items) > 1,
+	})
+	if err != nil {
+		logger.Errorf("构建存储选择键盘失败: %s", err)
+		ctx.EditMessage(userId, &tg.MessagesEditMessageRequest{
+			ID:      msg.ID,
+			Message: "构建存储选择键盘失败: " + err.Error(),
+		})
+		return
+	}
+	ctx.EditMessage(userId, &tg.MessagesEditMessageRequest{
+		ID:          msg.ID,
+		Message:     fmt.Sprintf("共 %d 个文件, 请选择存储位置", len(items)),
+		ReplyMarkup: markup,
+	})
+}

client/bot/handlers/parse.go

@@ -0,0 +1,104 @@
+// 处理任意文本消息, 用于通用地从外部源下载文件
+
+package handlers
+
+import (
+	"errors"
+
+	"github.com/celestix/gotgproto/dispatcher"
+	"github.com/celestix/gotgproto/ext"
+	"github.com/charmbracelet/log"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
+	"github.com/krau/SaveAny-Bot/parsers"
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
+	"github.com/krau/SaveAny-Bot/pkg/tcbdata"
+	"github.com/krau/SaveAny-Bot/storage"
+)
+
+func handleTextMessage(ctx *ext.Context, u *ext.Update) error {
+	logger := log.FromContext(ctx)
+	text := u.EffectiveMessage.Text
+	item, err := parsers.ParseWithContext(ctx, text)
+	if errors.Is(err, parsers.ErrNoParserFound) {
+		return dispatcher.EndGroups
+	}
+	if err != nil {
+		logger.Error("Failed to parse text", "error", err)
+		ctx.Reply(u, ext.ReplyTextString("Failed to parse text: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	logger.Debug("Parsed item from text message", "text", text, "item", item)
+	userID := u.GetUserChat().GetID()
+	markup, err := msgelem.BuildAddSelectStorageKeyboard(storage.GetUserStorages(ctx, userID), tcbdata.Add{
+		TaskType:   tasktype.TaskTypeParseditem,
+		ParsedItem: item,
+	})
+	if err != nil {
+		logger.Errorf("Failed to build storage selection keyboard: %s", err)
+		ctx.Reply(u, ext.ReplyTextString("Failed to build storage selection keyboard: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	text, entities, err := msgelem.BuildParsedTextEntity(*item)
+	if err != nil {
+		logger.Errorf("Failed to build parsed text entity: %s", err)
+		ctx.Reply(u, ext.ReplyTextString("Failed to build parsed text entity: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	ctx.SendMessage(userID, &tg.MessagesSendMessageRequest{
+		Message:     text,
+		ReplyMarkup: markup,
+		Entities:    entities,
+		ReplyTo: &tg.InputReplyToMessage{
+			ReplyToMsgID:  u.EffectiveMessage.ID,
+			ReplyToPeerID: u.GetUserChat().AsInputPeer(),
+		},
+	})
+
+	return dispatcher.EndGroups
+}
+
+func handleSilentSaveText(ctx *ext.Context, u *ext.Update) error {
+	logger := log.FromContext(ctx)
+	stor := storage.FromContext(ctx)
+	if stor == nil {
+		logger.Warn("Context storage is nil")
+		ctx.Reply(u, ext.ReplyTextString("未找到存储"), nil)
+		return dispatcher.EndGroups
+	}
+	text := u.EffectiveMessage.Text
+	if text == "" {
+		return dispatcher.EndGroups
+	}
+	item, err := parsers.ParseWithContext(ctx, text)
+	if errors.Is(err, parsers.ErrNoParserFound) {
+		return dispatcher.EndGroups
+	}
+	if err != nil {
+		logger.Error("Failed to parse text", "error", err)
+		ctx.Reply(u, ext.ReplyTextString("Failed to parse text: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	logger.Debug("Parsed item from text message", "text", text, "item", item)
+	userID := u.GetUserChat().GetID()
+	text, entities, err := msgelem.BuildParsedTextEntity(*item)
+	if err != nil {
+		logger.Errorf("Failed to build parsed text entity: %s", err)
+		ctx.Reply(u, ext.ReplyTextString("Failed to build parsed text entity: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	msg, err := ctx.SendMessage(userID, &tg.MessagesSendMessageRequest{
+		Message:  text,
+		Entities: entities,
+		ReplyTo: &tg.InputReplyToMessage{
+			ReplyToMsgID:  u.EffectiveMessage.ID,
+			ReplyToPeerID: u.GetUserChat().AsInputPeer(),
+		},
+	})
+	if err != nil {
+		logger.Errorf("Failed to send message: %s", err)
+		return dispatcher.EndGroups
+	}
+	return shortcut.CreateAndAddParsedTaskWithEdit(ctx, stor, "", item, msg.ID, userID)
+}

client/bot/handlers/register.go

@@ -1,12 +1,26 @@
 package handlers
 
 import (
+	"path"
+	"regexp"
+	"strings"
+
 	"github.com/celestix/gotgproto/dispatcher"
 	"github.com/celestix/gotgproto/dispatcher/handlers"
 	"github.com/celestix/gotgproto/dispatcher/handlers/filters"
 	"github.com/celestix/gotgproto/ext"
+	"github.com/charmbracelet/log"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/re"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/ruleutil"
+	userclient "github.com/krau/SaveAny-Bot/client/user"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/core"
+	"github.com/krau/SaveAny-Bot/core/tasks/tfile"
+	"github.com/krau/SaveAny-Bot/database"
 	"github.com/krau/SaveAny-Bot/pkg/tcbdata"
+	"github.com/krau/SaveAny-Bot/storage"
+	"github.com/rs/xid"
 )
 
 func Register(disp dispatcher.Dispatcher) {
@@ -23,6 +37,8 @@ func Register(disp dispatcher.Dispatcher) {
 	disp.AddHandler(handlers.NewCommand("storage", handleStorageCmd))
 	disp.AddHandler(handlers.NewCommand("dir", handleDirCmd))
 	disp.AddHandler(handlers.NewCommand("rule", handleRuleCmd))
+	disp.AddHandler(handlers.NewCommand("watch", handleWatchCmd))
+	disp.AddHandler(handlers.NewCommand("unwatch", handleUnwatchCmd))
 	disp.AddHandler(handlers.NewCommand("save", handleSilentMode(handleSaveCmd, handleSilentSaveReplied)))
 	disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), handleAddCallback))
 	disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), handleSetDefaultCallback))
@@ -38,4 +54,85 @@ func Register(disp dispatcher.Dispatcher) {
 	}
 	disp.AddHandler(handlers.NewMessage(telegraphUrlRegexFilter, handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
 	disp.AddHandler(handlers.NewMessage(filters.Message.Media, handleSilentMode(handleMediaMessage, handleSilentSaveMedia)))
+	disp.AddHandler(handlers.NewMessage(filters.Message.Text, handleSilentMode(handleTextMessage, handleSilentSaveText)))
+
+	if config.Cfg.Telegram.Userbot.Enable {
+		go listenMediaMessageEvent(userclient.GetMediaMessageCh())
+	}
+}
+
+func listenMediaMessageEvent(ch chan userclient.MediaMessageEvent) {
+	logger := log.FromContext(userclient.GetCtx())
+	for event := range ch {
+		logger.Debug("Received media message event", "chat_id", event.ChatID, "file_name", event.File.Name())
+		ctx := event.Ctx
+		file := event.File
+		chats, err := database.GetWatchChatsByChatID(ctx, event.ChatID)
+		if err != nil {
+			logger.Errorf("Failed to get watch chats for chat ID %d: %v", event.ChatID, err)
+			continue
+		}
+		msgText := event.File.Message().GetMessage()
+		for _, chat := range chats {
+			if chat.Filter != "" {
+				filter := strings.Split(chat.Filter, ":")
+				if len(filter) != 2 {
+					logger.Warnf("Invalid filter format in chat %d, skipping", chat.ChatID)
+					continue
+				}
+				filterType := filter[0]
+				filterData := filter[1]
+				switch filterType {
+				case "msgre": // [TODO] enums for filter types
+					if ok, err := regexp.MatchString(filterData, msgText); err != nil {
+						continue
+					} else if !ok {
+						continue
+					}
+				default:
+					logger.Warnf("Unsupported filter type %s in chat %d, skipping", filterType, chat.ChatID)
+					continue
+				}
+			}
+			user, err := database.GetUserByID(ctx, chat.UserID)
+			if err != nil {
+				logger.Errorf("Failed to get user by ID %d: %v", chat.UserID, err)
+				continue
+			}
+			if user.DefaultStorage == "" {
+				logger.Warnf("User %d has no default storage set, skipping media message handling", chat.UserID)
+				continue
+			}
+			stor, err := storage.GetStorageByUserIDAndName(ctx, user.ChatID, user.DefaultStorage)
+			if err != nil {
+				logger.Errorf("Failed to get storage by user ID %d and name %s: %v", user.ChatID, user.DefaultStorage, err)
+				continue
+			}
+			var dirPath string
+			if user.ApplyRule && user.Rules != nil {
+				matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
+				dirPath = matchedDirPath.String()
+				if matchedStorageName.IsUsable() {
+					stor, err = storage.GetStorageByUserIDAndName(ctx, user.ChatID, matchedStorageName.String())
+					if err != nil {
+						logger.Errorf("Failed to get storage by user ID and name: %s", err)
+						continue
+					}
+				}
+			}
+			storagePath := stor.JoinStoragePath(path.Join(dirPath, file.Name()))
+			injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
+			taskid := xid.New().String()
+			task, err := tfile.NewTGFileTask(taskid, injectCtx, file, stor, storagePath, nil)
+			if err != nil {
+				logger.Errorf("create task failed: %s", err)
+				continue
+			}
+			if err := core.AddTask(injectCtx, task); err != nil {
+				logger.Errorf("add task failed: %s", err)
+				continue
+			}
+			logger.Infof("Added media message task for user %d in chat %d: %s", chat.UserID, event.ChatID, file.Name())
+		}
+	}
 }

client/bot/handlers/save.go

@@ -2,6 +2,7 @@ package handlers
 
 import (
 	"fmt"
+	"regexp"
 	"strings"
 
 	"github.com/celestix/gotgproto/dispatcher"
@@ -23,7 +24,7 @@ func handleSaveCmd(ctx *ext.Context, update *ext.Update) error {
 	logger := log.FromContext(ctx)
 	args := strings.Split(string(update.EffectiveMessage.Text), " ")
 	if len(args) >= 3 {
-		return handleBatchSave(ctx, update, args[1], args[2])
+		return handleBatchSave(ctx, update, args[1:])
 	}
 	replyTo := update.EffectiveMessage.ReplyToMessage
 	if replyTo == nil || replyTo.Message == nil {
@@ -60,7 +61,7 @@ func handleSaveCmd(ctx *ext.Context, update *ext.Update) error {
 func handleSilentSaveReplied(ctx *ext.Context, update *ext.Update) error {
 	args := strings.Split(string(update.EffectiveMessage.Text), " ")
 	if len(args) >= 3 {
-		return handleBatchSave(ctx, update, args[1], args[2])
+		return handleBatchSave(ctx, update, args[1:])
 	}
 	logger := log.FromContext(ctx)
 	stor := storage.FromContext(ctx)
@@ -92,7 +93,20 @@ func handleSilentSaveReplied(ctx *ext.Context, update *ext.Update) error {
 	return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, update.GetUserChat().GetID(), stor, "", file, msg.GetID())
 }
 
-func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgIdRangeArg string) error {
+func handleBatchSave(ctx *ext.Context, update *ext.Update, args []string) error {
+	chatArg := args[0]
+	msgIdRangeArg := args[1]
+	var filterStr string
+	var filter *regexp.Regexp
+	if len(args) > 2 {
+		filterStr = args[2]
+		var err error
+		filter, err = regexp.Compile(filterStr)
+		if err != nil {
+			ctx.Reply(update, ext.ReplyTextString("无效的正则表达式: "+err.Error()), nil)
+			return dispatcher.EndGroups
+		}
+	}
 	startID, endID, err := strutil.ParseIntStrRange(msgIdRangeArg, "-")
 	if err != nil {
 		ctx.Reply(update, ext.ReplyTextString("无效的消息ID范围: "+err.Error()), nil)
@@ -121,7 +135,11 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
 		return dispatcher.EndGroups
 	}
 	files := make([]tfile.TGFileMessage, 0, len(msgs))
+	sb := strings.Builder{}
 	for _, msg := range msgs {
+		if msg == nil {
+			continue
+		}
 		media, ok := msg.GetMedia()
 		if !ok {
 			continue
@@ -130,11 +148,21 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
 		if !supported {
 			continue
 		}
-		file, err := tfile.FromMediaMessage(media, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
+		file, err := tfile.FromMediaMessage(media, ctx.Raw, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
 		if err != nil {
 			log.FromContext(ctx).Errorf("获取文件失败: %s", err)
 			continue
 		}
+		if filter != nil {
+			sb.Reset()
+			sb.WriteString(msg.GetMessage())
+			sb.WriteString(" ")
+			fn, _ := tgutil.GetMediaFileName(media)
+			sb.WriteString(fn)
+			if !filter.MatchString(sb.String()) {
+				continue
+			}
+		}
 		files = append(files, file)
 	}
 	if len(files) == 0 {
@@ -164,5 +192,4 @@ func handleBatchSave(ctx *ext.Context, update *ext.Update, chatArg string, msgId
 		return dispatcher.EndGroups
 	}
 	return shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, update.GetUserChat().GetID(), stor, "", files, replied.ID)
-
 }

client/bot/handlers/telegraph.go

@@ -71,6 +71,6 @@ func handleSilentSaveTelegraph(ctx *ext.Context, update *ext.Update) error {
 		return err
 	}
 	userID := update.GetUserChat().GetID()
-	return shortcut.CreateAndAddTphTaskWithEdit(ctx, userID, result.Page, result.TphDir, result.Pics, stor, msg.ID)
+	return shortcut.CreateAndAddtelegraphWithEdit(ctx, userID, result.Page, result.TphDir, result.Pics, stor, msg.ID)
 
 }

client/bot/handlers/utils/msgelem/parse.go

@@ -0,0 +1,38 @@
+package msgelem
+
+import (
+	"fmt"
+
+	"github.com/gotd/td/telegram/message/entity"
+	"github.com/gotd/td/telegram/message/styling"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/pkg/parser"
+)
+
+func BuildParsedTextEntity(item parser.Item) (string, []tg.MessageEntityClass, error) {
+	eb := entity.Builder{}
+	if err := styling.Perform(&eb,
+		styling.Bold(fmt.Sprintf("[%s]%s", item.Site, item.Title)),
+		styling.Plain("\n链接: "),
+		styling.Code(item.URL),
+		styling.Plain("\n作者: "),
+		styling.Code(item.Author),
+		styling.Plain("\n描述: "),
+		styling.Code(item.Description),
+		styling.Plain("\n文件数量: "),
+		styling.Code(fmt.Sprintf("%d", len(item.Resources))),
+		styling.Plain("\n预计总大小: "),
+		styling.Code(fmt.Sprintf("%.2f MB", func() float64 {
+			var totalSize int64
+			for _, res := range item.Resources {
+				totalSize += res.Size
+			}
+			return float64(totalSize) / 1024 / 1024
+		}())),
+		styling.Plain("\n请选择存储位置"),
+	); err != nil {
+		return "", nil, fmt.Errorf("构建消息失败: %w", err)
+	}
+	text, entities := eb.Complete()
+	return text, entities, nil
+}

client/bot/handlers/utils/msgelem/save.go

@@ -10,6 +10,6 @@ const (
 
 	2. 设置默认存储后, 发送 /save <频道ID/用户名> <消息ID范围> 来批量保存文件. 遵从存储规则, 若未匹配到任何规则则使用默认存储.
 	示例:
-	/save @moreacg 114-514
+	/save @acherkrau 114-514
 	`
 )

client/bot/handlers/utils/msgelem/storage.go

@@ -24,6 +24,8 @@ func BuildAddSelectStorageKeyboard(stors []storage.Storage, adddata tcbdata.Add)
 			taskType = tasktype.TaskTypeTgfiles
 		} else if adddata.TphPageNode != nil {
 			taskType = tasktype.TaskTypeTphpics
+		} else if adddata.ParsedItem != nil {
+			taskType = tasktype.TaskTypeParseditem
 		} else {
 			return nil, fmt.Errorf("unknown task type: %s", taskType)
 		}
@@ -41,6 +43,8 @@ func BuildAddSelectStorageKeyboard(stors []storage.Storage, adddata tcbdata.Add)
 			TphPageNode: adddata.TphPageNode,
 			TphPics:     adddata.TphPics,
 			TphDirPath:  adddata.TphDirPath,
+
+			ParsedItem: adddata.ParsedItem,
 		}
 		dataid := xid.New().String()
 		err := cache.Set(dataid, data)

client/bot/handlers/utils/msgelem/watch.go

@@ -0,0 +1,19 @@
+package msgelem
+
+const (
+	WatchHelpText = `
+使用 /watch 命令监听一个聊天的消息, 并自动保存到默认存储中, 遵从存储规则.
+
+命令语法:
+/watch <chat_id> [filter]
+
+参数:
+- <chat_id>: 聊天的 ID 或用户名
+- [filter]: 可选, 格式为 过滤器类型:表达式 , 所有支持类型的过滤器请查看文档
+
+命令示例:
+/watch 2229835658 msgre:.*plana.*
+
+这将监听 ID 为 2229835658 的聊天, 并转存所有包含 "plana" 的媒体消息
+	`
+)

client/bot/handlers/utils/ruleutil/rule.go

@@ -3,6 +3,8 @@ package ruleutil
 import (
 	"context"
 
+	"github.com/duke-git/lancet/v2/convertor"
+
 	"github.com/charmbracelet/log"
 	"github.com/krau/SaveAny-Bot/database"
 	"github.com/krau/SaveAny-Bot/pkg/consts"
@@ -33,11 +35,22 @@ func (m matchedStorName) String() string {
 	return string(m)
 }
 
-func (m matchedStorName) IsValid() bool {
+// can we use this storage name directly?
+func (m matchedStorName) IsUsable() bool {
 	return m != "" && m != consts.RuleStorNameChosen
 }
 
-func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (matchedStorageName matchedStorName, dirPath string) {
+type MatchedDirPath string
+
+func (m MatchedDirPath) String() string {
+	return string(m)
+}
+
+func (m MatchedDirPath) NeedNewForAlbum() bool {
+	return m != "" && m == consts.RuleDirPathNewForAlbum
+}
+
+func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (matchedStorageName matchedStorName, dirPath MatchedDirPath) {
 	if inputs == nil || len(rules) == 0 {
 		return "", ""
 	}
@@ -56,7 +69,7 @@ func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (m
 				continue
 			}
 			if ok {
-				dirPath = ru.StoragePath()
+				dirPath = MatchedDirPath(ru.StoragePath())
 				matchedStorageName = matchedStorName(ru.StorageName())
 			}
 		case ruleenum.MessageRegex.String():
@@ -71,7 +84,26 @@ func ApplyRule(ctx context.Context, rules []database.Rule, inputs *ruleInput) (m
 				continue
 			}
 			if ok {
-				dirPath = ru.StoragePath()
+				dirPath = MatchedDirPath(ru.StoragePath())
+				matchedStorageName = matchedStorName(ru.StorageName())
+			}
+		case ruleenum.IsAlbum.String():
+			matchAlbum, err := convertor.ToBool(ur.Data)
+			if err != nil {
+				matchAlbum = false
+			}
+			ru, err := rule.NewRuleMediaType(ur.StorageName, ur.DirPath, matchAlbum)
+			if err != nil {
+				logger.Errorf("Failed to create rule: %s", err)
+				continue
+			}
+			ok, err := ru.Match(inputs.File.Message().GroupedID != 0)
+			if err != nil {
+				logger.Errorf("Failed to match rule: %s", err)
+				continue
+			}
+			if ok {
+				dirPath = MatchedDirPath(ru.StoragePath())
 				matchedStorageName = matchedStorName(ru.StorageName())
 			}
 		}

client/bot/handlers/utils/shortcut/message.go

@@ -10,13 +10,16 @@ import (
 	"github.com/celestix/gotgproto/ext"
 	"github.com/celestix/gotgproto/types"
 	"github.com/charmbracelet/log"
+	"github.com/gotd/td/telegram/downloader"
 	"github.com/gotd/td/tg"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/mediautil"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/re"
+	uc "github.com/krau/SaveAny-Bot/client/user"
 	"github.com/krau/SaveAny-Bot/common/cache"
 	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
 	"github.com/krau/SaveAny-Bot/common/utils/tphutil"
+	"github.com/krau/SaveAny-Bot/config"
 	"github.com/krau/SaveAny-Bot/pkg/telegraph"
 	"github.com/krau/SaveAny-Bot/pkg/tfile"
 )
@@ -29,8 +32,7 @@ func GetFileFromMessageWithReply(ctx *ext.Context, update *ext.Update, message *
 	media := message.Media
 	supported := mediautil.IsSupported(media)
 	if !supported {
-		ctx.Reply(update, ext.ReplyTextString("不支持的消息类型"), nil)
-		return nil, nil, dispatcher.EndGroups
+		return nil, nil, dispatcher.ContinueGroups
 	}
 
 	replied, err = ctx.Reply(update, ext.ReplyTextString("正在获取文件信息..."), nil)
@@ -46,7 +48,7 @@ func GetFileFromMessageWithReply(ctx *ext.Context, update *ext.Update, message *
 	} else {
 		options = append(options, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*message)))
 	}
-	file, err = tfile.FromMediaMessage(media, message, options...)
+	file, err = tfile.FromMediaMessage(media, ctx.Raw, message, options...)
 	if err != nil {
 		logger.Errorf("Failed to get file from media: %s", err)
 		ctx.Reply(update, ext.ReplyTextString("获取文件失败: "+err.Error()), nil)
@@ -81,29 +83,59 @@ func GetFilesFromUpdateLinkMessageWithReplyEdit(ctx *ext.Context, update *ext.Up
 	}
 
 	files = make([]tfile.TGFileMessage, 0, len(msgLinks))
-	for _, link := range msgLinks {
-		chatId, msgId, err := tgutil.ParseMessageLink(ctx, link)
-		if err != nil {
-			logger.Errorf("failed to parse message link %s: %s", link, err)
-			continue
-		}
-		msg, err := tgutil.GetMessageByID(ctx, chatId, msgId)
-		if err != nil {
-			logger.Errorf("failed to get message by ID: %s", err)
-			continue
+	addFile := func(client downloader.Client, msg *tg.Message) {
+		if msg == nil || msg.Media == nil {
+			logger.Warn("message is nil, skipping")
+			return
 		}
 		media, ok := msg.GetMedia()
 		if !ok {
 			logger.Debugf("message %d has no media", msg.GetID())
-			continue
+			return
 		}
-		file, err := tfile.FromMediaMessage(media, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
+		file, err := tfile.FromMediaMessage(media, client, msg, tfile.WithNameIfEmpty(tgutil.GenFileNameFromMessage(*msg)))
 		if err != nil {
 			logger.Errorf("failed to create file from media: %s", err)
-			continue
+			return
 		}
 		files = append(files, file)
 	}
+
+	tctx := ctx
+	if config.Cfg.Telegram.Userbot.Enable {
+		tctx = uc.GetCtx()
+	}
+
+	for _, link := range msgLinks {
+		linkUrl, err := url.Parse(link)
+		if err != nil {
+			logger.Errorf("failed to parse message link %s: %s", link, err)
+			continue
+		}
+		chatId, msgId, err := tgutil.ParseMessageLink(tctx, link)
+		if err != nil {
+			logger.Errorf("failed to parse message link %s: %s", link, err)
+			continue
+		}
+		msg, err := tgutil.GetMessageByID(tctx, chatId, msgId)
+		if err != nil {
+			logger.Errorf("failed to get message by ID: %s", err)
+			continue
+		}
+		groupID, isGroup := msg.GetGroupedID()
+		if isGroup && groupID != 0 && !linkUrl.Query().Has("single") {
+			gmsgs, err := tgutil.GetGroupedMessages(ctx, chatId, msg)
+			if err != nil {
+				logger.Errorf("failed to get grouped messages: %s", err)
+			} else {
+				for _, gmsg := range gmsgs {
+					addFile(tctx.Raw, gmsg)
+				}
+			}
+		} else {
+			addFile(tctx.Raw, msg)
+		}
+	}
 	if len(files) == 0 {
 		editReplied("没有找到可保存的文件", nil)
 		return nil, nil, nil, dispatcher.EndGroups

client/bot/handlers/utils/shortcut/parsed.go

@@ -0,0 +1,35 @@
+package shortcut
+
+import (
+	"github.com/celestix/gotgproto/dispatcher"
+	"github.com/celestix/gotgproto/ext"
+	"github.com/charmbracelet/log"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+	"github.com/krau/SaveAny-Bot/core"
+	"github.com/krau/SaveAny-Bot/core/tasks/parsed"
+	"github.com/krau/SaveAny-Bot/pkg/parser"
+	"github.com/krau/SaveAny-Bot/storage"
+	"github.com/rs/xid"
+)
+
+func CreateAndAddParsedTaskWithEdit(ctx *ext.Context, stor storage.Storage, dirPath string, item *parser.Item, msgID int, userID int64) error {
+	injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
+	task := parsed.NewTask(xid.New().String(), injectCtx, stor, stor.JoinStoragePath(dirPath), item, parsed.NewProgress(msgID, userID))
+	if err := core.AddTask(injectCtx, task); err != nil {
+		log.FromContext(ctx).Errorf("Failed to add task: %s", err)
+		ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
+			ID:      msgID,
+			Message: "任务添加失败: " + err.Error(),
+		})
+		return dispatcher.EndGroups
+	}
+	text, entities := msgelem.BuildTaskAddedEntities(ctx, item.Title, core.GetLength(ctx))
+	ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
+		ID:       msgID,
+		Message:  text,
+		Entities: entities,
+	})
+	return dispatcher.EndGroups
+}

client/bot/handlers/utils/shortcut/tftask.go

@@ -3,6 +3,7 @@ package shortcut
 import (
 	"fmt"
 	"path"
+	"strings"
 
 	"github.com/celestix/gotgproto/dispatcher"
 	"github.com/celestix/gotgproto/ext"
@@ -12,15 +13,15 @@ import (
 	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/ruleutil"
 	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
 	"github.com/krau/SaveAny-Bot/core"
-	"github.com/krau/SaveAny-Bot/core/batchtftask"
-	"github.com/krau/SaveAny-Bot/core/tftask"
+	"github.com/krau/SaveAny-Bot/core/tasks/batchtfile"
+	tftask "github.com/krau/SaveAny-Bot/core/tasks/tfile"
 	"github.com/krau/SaveAny-Bot/database"
 	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"github.com/krau/SaveAny-Bot/storage"
 	"github.com/rs/xid"
 )
 
-// 创建一个 tftask.TGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
+// 创建一个 tfile.TGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
 func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, file tfile.TGFileMessage, trackMsgID int) error {
 	logger := log.FromContext(ctx)
 	user, err := database.GetUserByChatID(ctx, userID)
@@ -34,8 +35,8 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
 	}
 	if user.ApplyRule && user.Rules != nil {
 		matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
-		dirPath = matchedDirPath
-		if matchedStorageName.IsValid() {
+		dirPath = matchedDirPath.String()
+		if matchedStorageName.IsUsable() {
 			stor, err = storage.GetStorageByUserIDAndName(ctx, user.ChatID, matchedStorageName.String())
 			if err != nil {
 				logger.Errorf("Failed to get storage by user ID and name: %s", err)
@@ -51,7 +52,7 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
 	storagePath := stor.JoinStoragePath(path.Join(dirPath, file.Name()))
 	injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
 	taskid := xid.New().String()
-	task, err := tftask.NewTGFileTask(taskid, injectCtx, file, ctx.Raw, stor, storagePath,
+	task, err := tftask.NewTGFileTask(taskid, injectCtx, file, stor, storagePath,
 		tftask.NewProgressTrack(
 			trackMsgID,
 			userID))
@@ -81,7 +82,7 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
 	return dispatcher.EndGroups
 }
 
-// 创建一个 batchtftask.BatchTGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
+// 创建一个 batchtfile.BatchTGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
 func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, files []tfile.TGFileMessage, trackMsgID int) error {
 	logger := log.FromContext(ctx)
 	user, err := database.GetUserByChatID(ctx, userID)
@@ -93,19 +94,28 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
 		})
 		return dispatcher.EndGroups
 	}
+
 	useRule := user.ApplyRule && user.Rules != nil
-	applyRule := func(file tfile.TGFileMessage) (string, string) {
+
+	applyRule := func(file tfile.TGFileMessage) (string, ruleutil.MatchedDirPath) {
 		if !useRule {
-			return stor.Name(), dirPath
+			return stor.Name(), ruleutil.MatchedDirPath(dirPath)
 		}
 		storName, dirP := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
-		if !storName.IsValid() {
-			return stor.Name(), dirP
+
+		storname := storName.String()
+		if !storName.IsUsable() {
+			storname = stor.Name()
 		}
-		return storName.String(), dirP
+		return storname, dirP
 	}
 
-	elems := make([]batchtftask.TaskElement, 0, len(files))
+	elems := make([]batchtfile.TaskElement, 0, len(files))
+	type albumFile struct {
+		file    tfile.TGFileMessage
+		storage storage.Storage
+	}
+	albumFiles := make(map[int64][]albumFile, 0)
 	for _, file := range files {
 		storName, dirPath := applyRule(file)
 		fileStor := stor
@@ -120,21 +130,59 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
 				return dispatcher.EndGroups
 			}
 		}
-		storPath := fileStor.JoinStoragePath(path.Join(dirPath, file.Name()))
-		elem, err := batchtftask.NewTaskElement(fileStor, storPath, file)
-		if err != nil {
-			logger.Errorf("Failed to create task element: %s", err)
-			ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
-				ID:      trackMsgID,
-				Message: "任务创建失败: " + err.Error(),
+		if !dirPath.NeedNewForAlbum() {
+			storPath := fileStor.JoinStoragePath(path.Join(dirPath.String(), file.Name()))
+			elem, err := batchtfile.NewTaskElement(fileStor, storPath, file)
+			if err != nil {
+				logger.Errorf("Failed to create task element: %s", err)
+				ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
+					ID:      trackMsgID,
+					Message: "任务创建失败: " + err.Error(),
+				})
+				return dispatcher.EndGroups
+			}
+			elems = append(elems, *elem)
+		} else {
+			groupId, isGroup := file.Message().GetGroupedID()
+			if !isGroup || groupId == 0 {
+				logger.Warnf("File %s is not in a group, skipping album handling", file.Name())
+				continue
+			}
+			if _, ok := albumFiles[groupId]; !ok {
+				albumFiles[groupId] = make([]albumFile, 0)
+			}
+			albumFiles[groupId] = append(albumFiles[groupId], albumFile{
+				file:    file,
+				storage: fileStor,
 			})
-			return dispatcher.EndGroups
 		}
-		elems = append(elems, *elem)
 	}
+	for _, afiles := range albumFiles {
+		if len(afiles) <= 1 {
+			continue
+		}
+		// 对于需要新建目录的文件, 将第一个文件的文件名(去除扩展名)作为目录名
+		// 存储以第一个文件的存储为准
+		albumDir := strings.TrimSuffix(path.Base(afiles[0].file.Name()), path.Ext(afiles[0].file.Name()))
+		albumStor := afiles[0].storage
+		for _, af := range afiles {
+			afstorPath := af.storage.JoinStoragePath(path.Join(dirPath, albumDir, af.file.Name()))
+			elem, err := batchtfile.NewTaskElement(albumStor, afstorPath, af.file)
+			if err != nil {
+				logger.Errorf("Failed to create task element for album file: %s", err)
+				ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
+					ID:      trackMsgID,
+					Message: "任务创建失败: " + err.Error(),
+				})
+				return dispatcher.EndGroups
+			}
+			elems = append(elems, *elem)
+		}
+	}
+
 	injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
 	taskid := xid.New().String()
-	task := batchtftask.NewBatchTGFileTask(taskid, injectCtx, elems, ctx.Raw, batchtftask.NewProgressTracker(trackMsgID, userID), true)
+	task := batchtfile.NewBatchTGFileTask(taskid, injectCtx, elems, batchtfile.NewProgressTracker(trackMsgID, userID), true)
 	if err := core.AddTask(injectCtx, task); err != nil {
 		logger.Errorf("Failed to add batch task: %s", err)
 		ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{

client/bot/handlers/utils/shortcut/tphtask.go

@@ -9,19 +9,21 @@ import (
 	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
 	"github.com/krau/SaveAny-Bot/common/utils/tphutil"
 	"github.com/krau/SaveAny-Bot/core"
-	"github.com/krau/SaveAny-Bot/core/tphtask"
+	tphtask "github.com/krau/SaveAny-Bot/core/tasks/telegraph"
 	"github.com/krau/SaveAny-Bot/pkg/telegraph"
 	"github.com/krau/SaveAny-Bot/storage"
 	"github.com/rs/xid"
 )
 
-func CreateAndAddTphTaskWithEdit(ctx *ext.Context,
+func CreateAndAddtelegraphWithEdit(
+	ctx *ext.Context,
 	userID int64,
 	tphpage *telegraph.Page,
 	dirPath string, // unescaped ph path for file storage
 	pics []string,
 	stor storage.Storage,
 	trackMsgID int) error {
+		
 	injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
 	task := tphtask.NewTask(xid.New().String(),
 		injectCtx,

client/bot/handlers/watch.go

@@ -0,0 +1,110 @@
+package handlers
+
+import (
+	"regexp"
+	"strings"
+
+	"github.com/celestix/gotgproto/dispatcher"
+	"github.com/celestix/gotgproto/ext"
+	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+	"github.com/krau/SaveAny-Bot/database"
+)
+
+func handleWatchCmd(ctx *ext.Context, update *ext.Update) error {
+	logger := log.FromContext(ctx)
+	args := strings.Split(string(update.EffectiveMessage.Text), " ")
+	if len(args) < 2 {
+		ctx.Reply(update, ext.ReplyTextString(msgelem.WatchHelpText), nil)
+		return dispatcher.EndGroups
+	}
+	userChatID := update.GetUserChat().GetID()
+	user, err := database.GetUserByChatID(ctx, userChatID)
+	if err != nil {
+		logger.Errorf("获取用户失败: %s", err)
+		ctx.Reply(update, ext.ReplyTextString("获取用户失败"), nil)
+		return dispatcher.EndGroups
+	}
+	if user.DefaultStorage == "" {
+		ctx.Reply(update, ext.ReplyTextString("请先设置默认存储, 使用 /storage 命令"), nil)
+		return dispatcher.EndGroups
+	}
+	chatArg := args[1]
+	chatID, err := tgutil.ParseChatID(ctx, chatArg)
+	if err != nil {
+		ctx.Reply(update, ext.ReplyTextString("无效的ID或用户名: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	watching, err := user.WatchingChat(ctx, chatID)
+	if err != nil {
+		logger.Errorf("Failed to check if user is watching chat %d: %s", chatID, err)
+		return dispatcher.EndGroups
+	}
+	if watching {
+		ctx.Reply(update, ext.ReplyTextString("已经在监听此聊天"), nil)
+		return dispatcher.EndGroups
+	}
+	filter := ""
+	if len(args) > 2 {
+		filterArg := strings.Join(args[2:], " ")
+		filterType := strings.Split(filterArg, ":")[0]
+		filterData := strings.Split(filterArg, ":")[1]
+		if filterType == "" || filterData == "" {
+			ctx.Reply(update, ext.ReplyTextString("过滤器格式错误, 请使用 <过滤器类型>:<表达式>"), nil)
+			return dispatcher.EndGroups
+		}
+		switch filterType {
+		case "msgre":
+			_, err := regexp.Compile(filterData)
+			if err != nil {
+				ctx.Reply(update, ext.ReplyTextString("正则表达式格式错误: "+err.Error()), nil)
+				return dispatcher.EndGroups
+			}
+			filter = filterType + ":" + filterData
+		default:
+			ctx.Reply(update, ext.ReplyTextString("不支持的过滤器类型, 请参阅文档"), nil)
+			return dispatcher.EndGroups
+		}
+	}
+	if err := user.WatchChat(ctx, database.WatchChat{
+		UserID: user.ID,
+		ChatID: chatID,
+		Filter: filter,
+	}); err != nil {
+		logger.Errorf("Failed to watch chat %d: %s", chatID, err)
+		ctx.Reply(update, ext.ReplyTextString("监听聊天失败: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	ctx.Reply(update, ext.ReplyTextString("已开始监听聊天: "+chatArg), nil)
+	return dispatcher.EndGroups
+}
+
+func handleUnwatchCmd(ctx *ext.Context, update *ext.Update) error {
+	logger := log.FromContext(ctx)
+	args := strings.Split(string(update.EffectiveMessage.Text), " ")
+	if len(args) < 2 {
+		ctx.Reply(update, ext.ReplyTextString("请提供要取消监听的聊天ID或用户名"), nil)
+		return dispatcher.EndGroups
+	}
+	userChatID := update.GetUserChat().GetID()
+	user, err := database.GetUserByChatID(ctx, userChatID)
+	if err != nil {
+		logger.Errorf("获取用户失败: %s", err)
+		ctx.Reply(update, ext.ReplyTextString("获取用户失败"), nil)
+		return dispatcher.EndGroups
+	}
+	chatArg := args[1]
+	chatID, err := tgutil.ParseChatID(ctx, chatArg)
+	if err != nil {
+		ctx.Reply(update, ext.ReplyTextString("无效的ID或用户名: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	if err := user.UnwatchChat(ctx, chatID); err != nil {
+		logger.Errorf("Failed to unwatch chat %d: %s", chatID, err)
+		ctx.Reply(update, ext.ReplyTextString("取消监听聊天失败: "+err.Error()), nil)
+		return dispatcher.EndGroups
+	}
+	ctx.Reply(update, ext.ReplyTextString("已取消监听聊天: "+chatArg), nil)
+	return dispatcher.EndGroups
+}

client/middleware/default.go

@@ -9,13 +9,14 @@ import (
 	"github.com/gotd/td/telegram"
 	"github.com/krau/SaveAny-Bot/client/middleware/recovery"
 	"github.com/krau/SaveAny-Bot/client/middleware/retry"
+	"github.com/krau/SaveAny-Bot/config"
 )
 
 // https://github.com/iyear/tdl/blob/master/core/tclient/tclient.go
 func NewDefaultMiddlewares(ctx context.Context, timeout time.Duration) []telegram.Middleware {
 	return []telegram.Middleware{
 		recovery.New(ctx, newBackoff(timeout)),
-		retry.New(5),
+		retry.New(config.Cfg.Telegram.RpcRetry),
 		floodwait.NewSimpleWaiter(),
 	}
 }

client/middleware/recovery/recovery.go

@@ -2,11 +2,11 @@ package recovery
 
 import (
 	"context"
+	"fmt"
 	"time"
 
 	"github.com/cenkalti/backoff/v4"
 	"github.com/charmbracelet/log"
-	"github.com/go-faster/errors"
 	"github.com/gotd/td/bin"
 	"github.com/gotd/td/telegram"
 	"github.com/gotd/td/tg"
@@ -31,7 +31,7 @@ func (r *recovery) Handle(next tg.Invoker) telegram.InvokeFunc {
 		return backoff.RetryNotify(func() error {
 			if err := next.Invoke(ctx, input, output); err != nil {
 				if r.shouldRecover(ctx, err) {
-					return errors.Wrap(err, "recover")
+					return fmt.Errorf("recovery: %w", err)
 				}
 
 				return backoff.Permanent(err)

client/middleware/retry/retry.go

@@ -5,7 +5,6 @@ import (
 	"fmt"
 
 	"github.com/charmbracelet/log"
-	"github.com/go-faster/errors"
 	"github.com/gotd/td/bin"
 	"github.com/gotd/td/telegram"
 	"github.com/gotd/td/tg"
@@ -37,7 +36,8 @@ func (r retry) Handle(next tg.Invoker) telegram.InvokeFunc {
 					retries++
 					continue
 				}
-				return errors.Wrap(err, "retry middleware skip")
+				// retry middleware skip
+				return err
 			}
 
 			return nil

client/user/auth_terminal.go

@@ -9,9 +9,9 @@ import (
 	"github.com/fatih/color"
 )
 
-type termialAuthConversator struct{}
+type terminalAuthConversator struct{}
 
-func (t *termialAuthConversator) AskPhoneNumber() (string, error) {
+func (t *terminalAuthConversator) AskPhoneNumber() (string, error) {
 	phone := ""
 	err := huh.NewInput().Title("Your Phone Number").
 		Placeholder("+44 123456").
@@ -29,7 +29,7 @@ func (t *termialAuthConversator) AskPhoneNumber() (string, error) {
 	return strings.TrimSpace(phone), nil
 }
 
-func (t *termialAuthConversator) AskCode() (string, error) {
+func (t *terminalAuthConversator) AskCode() (string, error) {
 	code := ""
 	err := huh.NewInput().Title("Your Code").
 		Placeholder("123456").
@@ -45,7 +45,7 @@ func (t *termialAuthConversator) AskCode() (string, error) {
 	return strings.TrimSpace(code), nil
 }
 
-func (t *termialAuthConversator) AskPassword() (string, error) {
+func (t *terminalAuthConversator) AskPassword() (string, error) {
 	pwd := ""
 
 	err := huh.NewInput().Title("Your 2FA Password").
@@ -61,7 +61,7 @@ func (t *termialAuthConversator) AskPassword() (string, error) {
 	return strings.TrimSpace(pwd), nil
 }
 
-func (t *termialAuthConversator) AuthStatus(authStatus gotgproto.AuthStatus) {
+func (t *terminalAuthConversator) AuthStatus(authStatus gotgproto.AuthStatus) {
 	switch authStatus.Event {
 	case gotgproto.AuthStatusPhoneRetrial:
 		color.Red("The phone number you just entered seems to be incorrect,")

client/user/userclient.go

@@ -5,46 +5,87 @@ import (
 	"time"
 
 	"github.com/celestix/gotgproto"
+	"github.com/celestix/gotgproto/dispatcher"
+	"github.com/celestix/gotgproto/dispatcher/handlers"
+	"github.com/celestix/gotgproto/dispatcher/handlers/filters"
 	"github.com/celestix/gotgproto/ext"
 	"github.com/celestix/gotgproto/sessionMaker"
+
 	"github.com/charmbracelet/log"
+	"github.com/gotd/td/telegram/dcs"
+	"github.com/gotd/td/tg"
 	"github.com/krau/SaveAny-Bot/client/middleware"
+	"github.com/krau/SaveAny-Bot/common/utils/netutil"
 	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/database"
 	"github.com/ncruces/go-sqlite3/gormlite"
+	"golang.org/x/net/proxy"
 )
 
-var UC *gotgproto.Client
+var uc *gotgproto.Client
 var ectx *ext.Context
 
 func GetCtx() *ext.Context {
+	if uc == nil {
+		panic("User client is not initialized, please call Login first")
+	}
 	if ectx != nil {
-		// UC.RefreshContext(ectx)
 		return ectx
 	}
-	ectx = UC.CreateContext()
+	ectx = uc.CreateContext()
 	return ectx
 }
 
+func GetClient() *gotgproto.Client {
+	if uc == nil {
+		panic("User client is not initialized, please call Login first")
+	}
+	return uc
+}
+
 func Login(ctx context.Context) (*gotgproto.Client, error) {
-	log.FromContext(ctx).Debug("Logging in as user client")
-	if UC != nil {
-		return UC, nil
+	log.FromContext(ctx).Debug("Logging in user client")
+	if uc != nil {
+		return uc, nil
 	}
 	res := make(chan struct {
 		client *gotgproto.Client
 		err    error
 	})
 	go func() {
+		var resolver dcs.Resolver
+		if config.Cfg.Telegram.Proxy.Enable && config.Cfg.Telegram.Proxy.URL != "" {
+			dialer, err := netutil.NewProxyDialer(config.Cfg.Telegram.Proxy.URL)
+			if err != nil {
+				res <- struct {
+					client *gotgproto.Client
+					err    error
+				}{nil, err}
+				return
+			}
+			resolver = dcs.Plain(dcs.PlainOptions{
+				Dial: dialer.(proxy.ContextDialer).DialContext,
+			})
+		} else {
+			resolver = dcs.DefaultResolver()
+		}
 		tclient, err := gotgproto.NewClient(
 			config.Cfg.Telegram.AppID,
 			config.Cfg.Telegram.AppHash,
 			gotgproto.ClientTypePhone(""),
 			&gotgproto.ClientOpts{
 				Session:          sessionMaker.SqlSession(gormlite.Open(config.Cfg.Telegram.Userbot.Session)),
-				AuthConversator:  &termialAuthConversator{},
+				AuthConversator:  &terminalAuthConversator{},
 				Context:          ctx,
 				DisableCopyright: true,
+				Resolver:         resolver,
+				MaxRetries:       config.Cfg.Telegram.RpcRetry,
+				AutoFetchReply:   true,
 				Middlewares:      middleware.NewDefaultMiddlewares(ctx, 5*time.Minute),
+				ErrorHandler: func(ctx *ext.Context, u *ext.Update, s string) error {
+					log.FromContext(ctx).Errorf("Unhandled error: %s", s)
+					return dispatcher.EndGroups
+				},
 			},
 		)
 		if err != nil {
@@ -69,7 +110,21 @@ func Login(ctx context.Context) (*gotgproto.Client, error) {
 		if r.err != nil {
 			return nil, r.err
 		}
-		UC = r.client
-		return UC, nil
+		uc = r.client
+		uc.Dispatcher.AddHandler(handlers.NewMessage(filters.Message.Media, func(ctx *ext.Context, u *ext.Update) error {
+			switch u.UpdateClass.(type) {
+			case *tg.UpdateEditChannelMessage, *tg.UpdateEditMessage, *tg.UpdateDeleteChannelMessages, *tg.UpdateDeleteMessages:
+				return dispatcher.EndGroups
+			}
+			chatId := u.EffectiveChat().GetID()
+			watchChats, err := database.GetWatchChatsByChatID(ctx, chatId)
+			if err != nil || len(watchChats) == 0 {
+				return dispatcher.EndGroups
+			}
+			return dispatcher.ContinueGroups
+		}))
+		uc.Dispatcher.AddHandler(handlers.NewMessage(filters.Message.Media, handleMediaMessage))
+		log.FromContext(ctx).Infof("User client logged in successfully: %s", uc.Self.FirstName+" "+uc.Self.LastName)
+		return uc, nil
 	}
 }

client/user/watch.go

@@ -0,0 +1,100 @@
+package user
+
+import (
+	"sync"
+	"time"
+
+	"github.com/celestix/gotgproto/dispatcher"
+	"github.com/celestix/gotgproto/ext"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+	"github.com/krau/SaveAny-Bot/pkg/tfile"
+)
+
+type MediaMessageEvent struct {
+	Ctx       *ext.Context
+	ChatID    int64 // from witch the media message was sent
+	MessageID int
+	File      tfile.TGFileMessage
+}
+
+type messageKey struct {
+	ChatID    int64
+	MessageID int
+}
+
+type MediaMessageHandler struct {
+	events   map[messageKey]MediaMessageEvent
+	timers   map[messageKey]*time.Timer
+	mu       sync.Mutex
+	debounce time.Duration
+}
+
+var (
+	mediaMessageCh      = make(chan MediaMessageEvent, 100)
+	mediaMessageHandler = &MediaMessageHandler{
+		events:   make(map[messageKey]MediaMessageEvent),
+		timers:   make(map[messageKey]*time.Timer),
+		debounce: 5 * time.Second,
+	}
+)
+
+func GetMediaMessageCh() chan MediaMessageEvent {
+	return mediaMessageCh
+}
+
+func sendMediaMessageEvent(event MediaMessageEvent) {
+	key := messageKey{ChatID: event.ChatID, MessageID: event.MessageID}
+
+	mediaMessageHandler.mu.Lock()
+	defer mediaMessageHandler.mu.Unlock()
+
+	if timer, exists := mediaMessageHandler.timers[key]; exists {
+		timer.Stop()
+	} else {
+		mediaMessageHandler.events[key] = event
+	}
+
+	mediaMessageHandler.timers[key] = time.AfterFunc(mediaMessageHandler.debounce, func() {
+		mediaMessageHandler.mu.Lock()
+		event := mediaMessageHandler.events[key]
+		delete(mediaMessageHandler.events, key)
+		delete(mediaMessageHandler.timers, key)
+		mediaMessageHandler.mu.Unlock()
+
+		mediaMessageCh <- event
+	})
+}
+
+func handleMediaMessage(ctx *ext.Context, update *ext.Update) error {
+	message := update.EffectiveMessage
+	media, ok := message.GetMedia()
+	if !ok || media == nil {
+		return dispatcher.EndGroups
+	}
+	support := func() bool {
+		switch media.(type) {
+		case *tg.MessageMediaDocument, *tg.MessageMediaPhoto:
+			return true
+		default:
+			return false
+		}
+	}()
+	if !support {
+		return dispatcher.EndGroups
+	}
+	file, err := tfile.FromMediaMessage(media, ctx.Raw, message.Message, tfile.WithNameIfEmpty(
+		tgutil.GenFileNameFromMessage(*message.Message),
+	))
+	if err != nil {
+		return err
+	}
+	chatId := update.EffectiveChat().GetID()
+	sendMediaMessageEvent(MediaMessageEvent{
+		Ctx:       ctx,
+		ChatID:    chatId,
+		MessageID: message.ID,
+		File:      file,
+	})
+	return dispatcher.EndGroups
+}

cmd/run.go

@@ -12,12 +12,14 @@ import (
 	"github.com/charmbracelet/log"
 	"github.com/krau/SaveAny-Bot/client/bot"
 	userclient "github.com/krau/SaveAny-Bot/client/user"
+	"github.com/krau/SaveAny-Bot/common/cache"
 	"github.com/krau/SaveAny-Bot/common/i18n"
 	"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
 	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	"github.com/krau/SaveAny-Bot/config"
 	"github.com/krau/SaveAny-Bot/core"
 	"github.com/krau/SaveAny-Bot/database"
+	"github.com/krau/SaveAny-Bot/parsers"
 	"github.com/krau/SaveAny-Bot/storage"
 	"github.com/spf13/cobra"
 )
@@ -46,19 +48,27 @@ func initAll(ctx context.Context) {
 		fmt.Println("Failed to load config:", err)
 		os.Exit(1)
 	}
+	cache.Init()
 	logger := log.FromContext(ctx)
 	i18n.Init(config.Cfg.Lang)
 	logger.Info(i18n.T(i18nk.Initing))
+	database.Init(ctx)
+	storage.LoadStorages(ctx)
+	if config.Cfg.Parser.PluginEnable {
+		for _, dir := range config.Cfg.Parser.PluginDirs {
+			if err := parsers.LoadPlugins(ctx, dir); err != nil {
+				logger.Error("Failed to load parser plugins", "dir", dir, "error", err)
+			} else {
+				logger.Debug("Loaded parser plugins", "dir", dir)
+			}
+		}
+	}
 	if config.Cfg.Telegram.Userbot.Enable {
-		uc, err := userclient.Login(ctx)
+		_, err := userclient.Login(ctx)
 		if err != nil {
 			logger.Fatalf("User client login failed: %s", err)
 		}
-		logger.Infof("User client logged in as %s", uc.Self.FirstName)
 	}
-	database.Init(ctx)
-	storage.LoadStorages(ctx)
-
 	bot.Init(ctx)
 }
 

common/cache/ristretto.go

@@ -2,19 +2,22 @@ package cache
 
 import (
 	"fmt"
+	"time"
 
 	"github.com/charmbracelet/log"
 	"github.com/dgraph-io/ristretto/v2"
+	"github.com/krau/SaveAny-Bot/config"
 )
 
 var cache *ristretto.Cache[string, any]
 
-
-// TODO: maybe we should use simple ttl cache instead of ristretto...
-func init() {
+func Init() {
+	if cache != nil {
+		panic("cache already initialized")
+	}
 	c, err := ristretto.NewCache(&ristretto.Config[string, any]{
-		NumCounters: 1e5,
-		MaxCost:     1e6, // 1000000 / 112 ≈ 8928
+		NumCounters: config.Cfg.Cache.NumCounters,
+		MaxCost:     config.Cfg.Cache.MaxCost,
 		BufferItems: 64,
 		OnReject: func(item *ristretto.Item[any]) {
 			log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
@@ -27,7 +30,7 @@ func init() {
 }
 
 func Set(key string, value any) error {
-	ok := cache.Set(key, value, 0)
+	ok := cache.SetWithTTL(key, value, 0, time.Duration(config.Cfg.Cache.TTL)*time.Second)
 	if !ok {
 		return fmt.Errorf("failed to set value in cache")
 	}

common/tdler/dler.go

@@ -1,18 +0,0 @@
-package tdler
-
-import (
-	"github.com/gotd/td/telegram/downloader"
-	"github.com/krau/SaveAny-Bot/common/utils/dlutil"
-	"github.com/krau/SaveAny-Bot/config"
-	"github.com/krau/SaveAny-Bot/pkg/consts/tglimit"
-	"github.com/krau/SaveAny-Bot/pkg/tfile"
-)
-
-type Client interface {
-	downloader.Client
-}
-
-func NewDownloader(client Client, file tfile.TGFile) *downloader.Builder {
-	return downloader.NewDownloader().WithPartSize(tglimit.MaxPartSize).
-		Download(client, file.Location()).WithThreads(dlutil.BestThreads(file.Size(), config.Cfg.Threads))
-}

common/utils/netutil/proxy.go

@@ -0,0 +1,15 @@
+package netutil
+
+import (
+	"net/url"
+
+	"golang.org/x/net/proxy"
+)
+
+func NewProxyDialer(proxyUrl string) (proxy.Dialer, error) {
+	url, err := url.Parse(proxyUrl)
+	if err != nil {
+		return nil, err
+	}
+	return proxy.FromURL(url, proxy.Direct)
+}

common/utils/tgutil/media.go

@@ -0,0 +1,40 @@
+package tgutil
+
+import (
+	"fmt"
+
+	"github.com/gabriel-vasile/mimetype"
+	"github.com/gotd/td/tg"
+)
+
+func GetMediaFileName(media tg.MessageMediaClass) (string, error) {
+	switch v := media.(type) {
+	case *tg.MessageMediaPhoto:
+		f, ok := v.Photo.AsNotEmpty()
+		if !ok {
+			return "", fmt.Errorf("unknown type media: %T", media)
+		}
+		return fmt.Sprintf("%d.png", f.ID), nil
+	case *tg.MessageMediaDocument:
+		f, ok := v.Document.AsNotEmpty()
+		if !ok {
+			return "", fmt.Errorf("unknown type media: %T", media)
+		}
+		fileName := ""
+		for _, attribute := range f.Attributes {
+			if name, ok := attribute.(*tg.DocumentAttributeFilename); ok {
+				fileName = name.GetFileName()
+				break
+			}
+		}
+		if fileName == "" {
+			mmt := mimetype.Lookup(f.GetMimeType())
+			if mmt != nil {
+				fileName = fmt.Sprintf("%d.%s", f.GetID(), mmt.Extension())
+			}
+		}
+		return fileName, nil
+	default:
+		return "", fmt.Errorf("unsupported type media: %T", media)
+	}
+}

common/utils/tgutil/message.go

@@ -7,6 +7,7 @@ import (
 
 	"github.com/celestix/gotgproto/ext"
 	"github.com/duke-git/lancet/v2/maputil"
+
 	"github.com/duke-git/lancet/v2/mathutil"
 	"github.com/duke-git/lancet/v2/slice"
 	lcstrutil "github.com/duke-git/lancet/v2/strutil"
@@ -18,6 +19,9 @@ import (
 	"github.com/rs/xid"
 )
 
+// generate a file name from the message content and media type
+//
+// it will never return an empty string
 func GenFileNameFromMessage(message tg.Message) string {
 	ext := func(media tg.MessageMediaClass) string {
 		switch media := media.(type) {
@@ -26,11 +30,11 @@ func GenFileNameFromMessage(message tg.Message) string {
 			if !ok {
 				return ""
 			}
-			ext := mimetype.Lookup(doc.MimeType).Extension()
-			if ext == "" {
+			mmt := mimetype.Lookup(doc.MimeType)
+			if mmt == nil || mmt.Extension() == "" {
 				return ""
 			}
-			return ext
+			return mmt.Extension()
 		case *tg.MessageMediaPhoto:
 			return ".jpg"
 		}
@@ -81,7 +85,13 @@ func GenFileNameFromMessage(message tg.Message) string {
 	}()
 
 	if filename == "" {
-		filename = fmt.Sprintf("%d_%s", message.GetID(), xid.New().String())
+		mname, err := GetMediaFileName(message.Media)
+		if err != nil {
+			filename = fmt.Sprintf("%d_%s", message.GetID(), xid.New().String())
+		} else {
+			filename = mname
+		}
+
 	}
 	return filename + ext
 }
@@ -159,6 +169,96 @@ func GetMessagesRange(ctx *ext.Context, chatID int64, minId, maxId int) ([]*tg.M
 	return result, nil
 }
 
+type MessageItem struct {
+	Message *tg.Message
+	Error   error
+}
+
+func IterMessages(ctx *ext.Context, chatID int64, minId, maxId int) (<-chan MessageItem, error) {
+	total := maxId - minId + 1
+	ch := make(chan MessageItem, 100)
+
+	go func() {
+		defer close(ch)
+		if !ctx.Self.Bot {
+			perr := ctx.PeerStorage.GetInputPeerById(chatID)
+			if perr == nil || perr.(*tg.InputPeerEmpty) != nil {
+				ch <- MessageItem{
+					Error: fmt.Errorf("peer not found: %d", chatID),
+				}
+				return
+			}
+
+			for i := 0; i < total; i += 100 {
+				start := minId + i
+				end := min(start+100, maxId)
+				msgs, err := ctx.Raw.MessagesGetHistory(ctx, &tg.MessagesGetHistoryRequest{
+					Peer:      perr,
+					OffsetID:  start,
+					AddOffset: start - end,
+					Limit:     100,
+				})
+				if err != nil {
+					ch <- MessageItem{
+						Error: fmt.Errorf("failed to get messages: %w", err),
+					}
+					return
+				}
+				var msgClass []tg.MessageClass
+				switch msgsv := msgs.(type) {
+				case *tg.MessagesMessages:
+					msgClass = msgsv.GetMessages()
+				case *tg.MessagesMessagesSlice:
+					msgClass = msgsv.GetMessages()
+				case *tg.MessagesChannelMessages:
+					msgClass = msgsv.GetMessages()
+				default:
+					ch <- MessageItem{
+						Error: fmt.Errorf("unsupported message type: %T", msgsv),
+					}
+					continue
+				}
+				for _, msg := range msgClass {
+					msg, ok := msg.AsNotEmpty()
+					if !ok {
+						continue
+					}
+					switch msg := msg.(type) {
+					case *tg.Message:
+						key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msg.GetID())
+						cache.Set(key, msg)
+						ch <- MessageItem{
+							Message: msg,
+						}
+					}
+				}
+			}
+		} else {
+			for i := 0; i < total; i += 100 {
+				start := minId + i
+				end := min(start+100, maxId)
+				msgs, err := GetMessagesRange(ctx, chatID, start, end)
+				if err != nil {
+					ch <- MessageItem{
+						Error: fmt.Errorf("failed to get messages: %w", err),
+					}
+					return
+				}
+				for _, msg := range msgs {
+					if msg == nil {
+						continue
+					}
+					ch <- MessageItem{
+						Message: msg,
+					}
+				}
+			}
+		}
+	}()
+
+	return ch, nil
+}
+
 func GetMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, error) {
 	key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msgID)
 	if msg, ok := cache.Get[*tg.Message](key); ok {
@@ -181,3 +281,31 @@ func GetMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, err
 	cache.Set(key, tgm)
 	return tgm, nil
 }
+
+func GetGroupedMessages(ctx *ext.Context, chatID int64, msg *tg.Message) ([]*tg.Message, error) {
+	groupID, isGroup := msg.GetGroupedID()
+	if !isGroup || groupID == 0 {
+		return nil, fmt.Errorf("message %d is not grouped", msg.GetID())
+	}
+	msgID := msg.GetID()
+	minID := msgID - 10
+	maxID := msgID + 10
+	if minID < 1 {
+		minID = 1
+	}
+	msgs, err := GetMessagesRange(ctx, chatID, minID, maxID)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get grouped messages: %w", err)
+	}
+	groupedMessages := make([]*tg.Message, 0, len(msgs))
+	for _, m := range msgs {
+		if m == nil {
+			continue
+		}
+		mgid, isGroup := m.GetGroupedID()
+		if isGroup && mgid == groupID {
+			groupedMessages = append(groupedMessages, m)
+		}
+	}
+	return groupedMessages, nil
+}

config.example.toml

@@ -1,70 +1,34 @@
-#创建文件时，若需要保留中文注释，请务必确保本文件编码为 UTF-8 ，否则会无法读取。
-workers = 4 # 同时下载文件数
-retry = 3   # 下载失败重试次数
-threads = 4 # 单个任务下载最大线程数
-stream = false # 使用stream模式, 详情请查看文档
+# 创建文件时，若需要保留中文注释，请务必确保本文件编码为 UTF-8 ，否则会无法读取。
+# 更详细的配置请在 https://sabot.unv.app/deployment/configuration 查看
+workers = 4    # 同时下载文件数
+retry = 3      # 下载失败重试次数
+threads = 4    # 单个任务下载使用的最大线程数
+stream = false # 使用流式传输模式, 建议仅在硬盘空间十分有限时使用.
 
 [telegram]
 # Bot Token
-# 更换 Bot Token 后请删除数据库文件和 session.db
+# 更换 Bot Token 后请删除会话数据库文件 (默认路径为 data/session.db )
 token = ""
 # Telegram API 配置, 若不配置也可运行, 将使用默认的 API ID 和 API HASH
 # 推荐使用自己的 API ID 和 API HASH (https://my.telegram.org)
 # app_id = 1025907
 # app_hash = "452b0359b988148995f22ff0f4229750"
-
-# 初始化超时时间, 单位: 秒
-timeout = 60
-
-# flood_retry = 5
-# rpc_retry = 5
-
 [telegram.proxy]
 # 启用代理连接 telegram, 只支持 socks5
 enable = false
 url = "socks5://127.0.0.1:7890"
 
-# 用户列表
-[[users]]
-# telegram user id
-id = 114514
-# 使用黑名单模式，开启后下方留空以使用所有存储，反之则为白名单，白名单请在下方输入允许的存储名
-blacklist = true
-# 将列表留空并开启黑名单模式以允许使用所有存储，此处示例为黑名单模式，用户 114514 可使用所有存储
-storages = []
-
-[[users]]
-id = 123456
-blacklist = false  # 使用白名单模式，此时，用户123456 仅可使用下方列表中的存储
-# 此时该用户只能使用名为 本机1 的存储
-storages = ["本机1"]
-
 # 存储列表
 [[storages]]
 # 标识名, 需要唯一
 name = "本机1"
-# 存储类型, 目前可用: local, alist, webdav, minio
+# 存储类型, 目前可用: local, alist, webdav, minio, telegram
 type = "local"
 # 启用存储
 enable = true
 # 文件保存根路径
 base_path = "./downloads"
 
-[[storages]]
-name = "MyAlist"
-type = "alist"
-enable = false  #记得启用
-base_path = '/'
-url = 'https://alist.com'
-username = 'admin'
-password = 'password'
-# alist token 刷新时间 
-# 86400--1天   604800--7天   1296000--15天   2592000--30天   15552000--180天
-token_exp = 86400  
-# alist 可直接使用 token 登录, 此时 username, password, token_exp 将被忽略
-# 请自行在 alist 侧配置合理的 token 过期时间
-# token = ""
-
 [[storages]]
 name = "MyWebdav"
 type = "webdav"
@@ -74,31 +38,17 @@ url = 'https://example.com/dav'
 username = 'username'
 password = 'password'
 
-[[storages]]
-name = "MyMinio"
-type = "minio"
-enable = true
-endpoint = 'play.min.io'
-use_ssl = true
-access_key_id = 'Q3AM3UQ867SPQQA43P2F'
-secret_access_key = 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG'
-bucket_name = 'saveanybot'
-base_path = '/path/telegram'
-
-
-# 其他配置
-
-# [log]
-# # 日志等级
-# level = "DEBUG"
-
-# [temp]
-# # 下载文件临时目录, 请不要在此目录下存放任何其他文件
-# base_path = "cache/"
-# # 临时文件保存时间, 单位: 秒
-# cache_ttl = 30
-
-# [db]
-# path = "data/data.db" # 数据库文件路径
-# session = "data/session.db"
+# 用户列表
+[[users]]
+# telegram user id
+id = 114514
+# 存储过滤列表, 元素为存储标识名.
+# 将该列表留空并开启黑名单过滤模式以允许使用所有存储，此处示例为黑名单模式，用户 114514 可使用所有存储
+storages = []
+# 使用列表过滤黑名单模式，反之则为白名单，白名单请在列表中指定可用的存储.
+blacklist = true
 
+[[users]]
+id = 123456
+storages = ["本机1"]
+blacklist = false  # 使用白名单模式，此时，用户 123456 仅可使用标识名为 '本地1' 的存储

config/cache.go

@@ -0,0 +1,7 @@
+package config
+
+type cacheConfig struct {
+	TTL         int64 `toml:"ttl" mapstructure:"ttl" json:"ttl"`
+	NumCounters int64 `toml:"num_counters" mapstructure:"num_counters" json:"num_counters"`
+	MaxCost     int64 `toml:"max_cost" mapstructure:"max_cost" json:"max_cost"`
+}

config/db.go

@@ -0,0 +1,6 @@
+package config
+
+type dbConfig struct {
+	Path    string `toml:"path" mapstructure:"path"`
+	Session string `toml:"session" mapstructure:"session"`
+}

config/hook.go

@@ -0,0 +1,22 @@
+package config
+
+type hookConfig struct {
+	Exec hookExecConfig `toml:"exec" mapstructure:"exec" json:"exec"`
+}
+
+type hookExecConfig struct {
+	// command to execute, for all task types
+	TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
+	TaskSuccess     string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
+	TaskFail        string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
+	TaskCancel      string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
+
+	// TaskTypes map[string]hookExecOnTypeConfig `toml:"task_types" mapstructure:"task_types" json:"task_types"` // [TODO]
+}
+
+// type hookExecOnTypeConfig struct {
+// 	TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
+// 	TaskSuccess     string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
+// 	TaskFail        string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
+// 	TaskCancel      string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
+// }

config/parser.go

@@ -0,0 +1,6 @@
+package config
+
+type parserConfig struct {
+	PluginEnable bool     `toml:"plugin_enable" mapstructure:"plugin_enable" json:"plugin_enable"`
+	PluginDirs   []string `toml:"plugin_dirs" mapstructure:"plugin_dirs" json:"plugin_dirs"`
+}

config/temp.go

@@ -0,0 +1,5 @@
+package config
+
+type tempConfig struct {
+	BasePath string `toml:"base_path" mapstructure:"base_path" json:"base_path"`
+}

config/tg.go

@@ -0,0 +1,20 @@
+package config
+
+type telegramConfig struct {
+	Token    string        `toml:"token" mapstructure:"token"`
+	AppID    int           `toml:"app_id" mapstructure:"app_id" json:"app_id"`
+	AppHash  string        `toml:"app_hash" mapstructure:"app_hash" json:"app_hash"`
+	Proxy    tgProxyConfig `toml:"proxy" mapstructure:"proxy"`
+	RpcRetry int           `toml:"rpc_retry" mapstructure:"rpc_retry" json:"rpc_retry"`
+	Userbot  userbotConfig `toml:"userbot" mapstructure:"userbot" json:"userbot"`
+}
+
+type userbotConfig struct {
+	Enable  bool   `toml:"enable" mapstructure:"enable"`
+	Session string `toml:"session" mapstructure:"session"`
+}
+
+type tgProxyConfig struct {
+	Enable bool   `toml:"enable" mapstructure:"enable"`
+	URL    string `toml:"url" mapstructure:"url"`
+}

config/viper.go

@@ -22,53 +22,17 @@ type Config struct {
 	Threads      int    `toml:"threads" mapstructure:"threads" json:"threads"`
 	Stream       bool   `toml:"stream" mapstructure:"stream" json:"stream"`
 
-	Users []userConfig `toml:"users" mapstructure:"users" json:"users"`
-
+	Cache    cacheConfig             `toml:"cache" mapstructure:"cache" json:"cache"`
+	Users    []userConfig            `toml:"users" mapstructure:"users" json:"users"`
 	Temp     tempConfig              `toml:"temp" mapstructure:"temp"`
-	Log      logConfig               `toml:"log" mapstructure:"log"`
 	DB       dbConfig                `toml:"db" mapstructure:"db"`
 	Telegram telegramConfig          `toml:"telegram" mapstructure:"telegram"`
 	Storages []storage.StorageConfig `toml:"-" mapstructure:"-" json:"storages"`
+	Parser   parserConfig            `toml:"parser" mapstructure:"parser" json:"parser"`
+	Hook     hookConfig              `toml:"hook" mapstructure:"hook" json:"hook"`
 }
 
-type tempConfig struct {
-	BasePath string `toml:"base_path" mapstructure:"base_path" json:"base_path"`
-	CacheTTL int64  `toml:"cache_ttl" mapstructure:"cache_ttl" json:"cache_ttl"`
-}
-
-type logConfig struct {
-	Level       string `toml:"level" mapstructure:"level"`
-	File        string `toml:"file" mapstructure:"file"`
-	BackupCount uint   `toml:"backup_count" mapstructure:"backup_count" json:"backup_count"`
-}
-
-type dbConfig struct {
-	Path    string `toml:"path" mapstructure:"path"`
-	Session string `toml:"session" mapstructure:"session"`
-	Expire  int64  `toml:"expire" mapstructure:"expire"`
-}
-
-type telegramConfig struct {
-	Token      string        `toml:"token" mapstructure:"token"`
-	AppID      int           `toml:"app_id" mapstructure:"app_id" json:"app_id"`
-	AppHash    string        `toml:"app_hash" mapstructure:"app_hash" json:"app_hash"`
-	Timeout    int           `toml:"timeout" mapstructure:"timeout" json:"timeout"`
-	Proxy      proxyConfig   `toml:"proxy" mapstructure:"proxy"`
-	RpcRetry   int           `toml:"rpc_retry" mapstructure:"rpc_retry" json:"rpc_retry"`
-	Userbot    userbotConfig `toml:"userbot" mapstructure:"userbot" json:"userbot"`
-}
-
-type userbotConfig struct {
-	Enable  bool   `toml:"enable" mapstructure:"enable"`
-	Session string `toml:"session" mapstructure:"session"`
-}
-
-type proxyConfig struct {
-	Enable bool   `toml:"enable" mapstructure:"enable"`
-	URL    string `toml:"url" mapstructure:"url"`
-}
-
-var Cfg *Config
+var Cfg *Config = &Config{}
 
 func (c Config) GetStorageByName(name string) storage.StorageConfig {
 	for _, storage := range c.Storages {
@@ -89,28 +53,36 @@ func Init(ctx context.Context) error {
 	replacer := strings.NewReplacer(".", "_")
 	viper.SetEnvKeyReplacer(replacer)
 
-	viper.SetDefault("lang", "zh-Hans")
+	defaultConfigs := map[string]any{
+		// 基础配置
+		"lang":    "zh-Hans",
+		"workers": 3,
+		"retry":   3,
+		"threads": 4,
 
-	viper.SetDefault("workers", 3)
-	viper.SetDefault("retry", 3)
-	viper.SetDefault("threads", 4)
+		// 缓存配置
+		"cache.ttl":          86400,
+		"cache.num_counters": 1e5,
+		"cache.max_cost":     1e6,
 
-	viper.SetDefault("telegram.app_id", 1025907)
-	viper.SetDefault("telegram.app_hash", "452b0359b988148995f22ff0f4229750")
-	viper.SetDefault("telegram.timeout", 60)
-	viper.SetDefault("telegram.flood_retry", 5)
-	viper.SetDefault("telegram.rpc_retry", 5)
-	viper.SetDefault("telegram.userbot.enable", false)
-	viper.SetDefault("telegram.userbot.session", "data/usersession.db")
+		// Telegram
+		"telegram.app_id":          1025907,
+		"telegram.app_hash":        "452b0359b988148995f22ff0f4229750",
+		"telegram.rpc_retry":       5,
+		"telegram.userbot.enable":  false,
+		"telegram.userbot.session": "data/usersession.db",
 
-	viper.SetDefault("temp.base_path", "cache/")
-	viper.SetDefault("temp.cache_ttl", 30)
+		// 临时目录
+		"temp.base_path": "cache/",
 
-	viper.SetDefault("log.level", "INFO")
+		// 数据库
+		"db.path":    "data/saveany.db",
+		"db.session": "data/session.db",
+	}
 
-	viper.SetDefault("db.path", "data/saveany.db")
-	viper.SetDefault("db.session", "data/session.db")
-	viper.SetDefault("db.expire", 86400*5)
+	for key, value := range defaultConfigs {
+		viper.SetDefault(key, value)
+	}
 
 	if err := viper.SafeWriteConfigAs("config.toml"); err != nil {
 		if _, ok := err.(viper.ConfigFileAlreadyExistsError); !ok {
@@ -123,8 +95,6 @@ func Init(ctx context.Context) error {
 		os.Exit(1)
 	}
 
-	Cfg = &Config{}
-
 	if err := viper.Unmarshal(Cfg); err != nil {
 		fmt.Println("Error unmarshalling config file, ", err)
 		os.Exit(1)
@@ -171,7 +141,6 @@ func Init(ctx context.Context) error {
 			userStorages[user.ID] = user.Storages
 		}
 	}
-
 	return nil
 }
 

core/core.go

@@ -2,32 +2,54 @@ package core
 
 import (
 	"context"
+	"errors"
 
 	"github.com/charmbracelet/log"
 	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
 	"github.com/krau/SaveAny-Bot/pkg/queue"
 )
 
 var queueInstance *queue.TaskQueue[Exectable]
 
 type Exectable interface {
+	Type() tasktype.TaskType
 	TaskID() string
 	Execute(ctx context.Context) error
 }
 
 func worker(ctx context.Context, qe *queue.TaskQueue[Exectable], semaphore chan struct{}) {
+	logger := log.FromContext(ctx)
+	execHooks := config.Cfg.Hook.Exec
 	for {
 		semaphore <- struct{}{}
 		qtask, err := qe.Get()
 		if err != nil {
+			logger.Error("Failed to get task from queue:", err)
 			break // queue closed and empty
 		}
-		log.FromContext(ctx).Infof("Processing task: %s", qtask.ID)
 		task := qtask.Data
+		logger.Infof("Processing task: %s", task.TaskID())
+		if err := ExecCommandString(qtask.Context(), execHooks.TaskBeforeStart); err != nil {
+			logger.Errorf("Failed to execute before start hook for task %s: %v", task.TaskID(), err)
+		}
 		if err := task.Execute(qtask.Context()); err != nil {
-			log.FromContext(ctx).Errorf("Failed to execute task %s: %v", qtask.ID, err)
+			if errors.Is(err, context.Canceled) {
+				logger.Infof("Task %s was canceled", task.TaskID())
+				if err := ExecCommandString(ctx, execHooks.TaskCancel); err != nil {
+					logger.Errorf("Failed to execute cancel hook for task %s: %v", task.TaskID(), err)
+				}
+			} else {
+				logger.Errorf("Failed to execute task %s: %v", task.TaskID(), err)
+				if err := ExecCommandString(ctx, execHooks.TaskFail); err != nil {
+					logger.Errorf("Failed to execute fail hook for task %s: %v", task.TaskID(), err)
+				}
+			}
 		} else {
-			log.FromContext(ctx).Infof("Task %s completed successfully", qtask.ID)
+			logger.Infof("Task %s completed successfully", task.TaskID())
+			if err := ExecCommandString(ctx, execHooks.TaskSuccess); err != nil {
+				logger.Errorf("Failed to execute success hook for task %s: %v", task.TaskID(), err)
+			}
 		}
 		qe.Done(qtask.ID)
 		<-semaphore

core/hookutil.go

@@ -0,0 +1,23 @@
+package core
+
+import (
+	"context"
+	"os"
+	"os/exec"
+	"runtime"
+)
+
+func ExecCommandString(ctx context.Context, cmd string) error {
+	if cmd == "" {
+		return nil
+	}
+	var execCmd *exec.Cmd
+	if runtime.GOOS == "windows" {
+		execCmd = exec.CommandContext(ctx, "cmd.exe", "/C", cmd)
+	} else {
+		execCmd = exec.CommandContext(ctx, "sh", "-c", cmd)
+	}
+	execCmd.Stdout = os.Stdout
+	execCmd.Stderr = os.Stderr
+	return execCmd.Run()
+}

core/tasks/batchtfile/execute.go

@@ -1,4 +1,4 @@
-package batchtftask
+package batchtfile
 
 import (
 	"context"
@@ -9,11 +9,11 @@ import (
 
 	"github.com/charmbracelet/log"
 	"github.com/duke-git/lancet/v2/retry"
-	"github.com/krau/SaveAny-Bot/common/tdler"
 	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	"github.com/krau/SaveAny-Bot/common/utils/ioutil"
 	"github.com/krau/SaveAny-Bot/config"
-	"github.com/krau/SaveAny-Bot/pkg/enums/key"
+	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
+	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"golang.org/x/sync/errgroup"
 )
 
@@ -25,14 +25,19 @@ func (t *Task) Execute(ctx context.Context) error {
 	eg, gctx := errgroup.WithContext(ctx)
 	eg.SetLimit(workers)
 	for _, elem := range t.Elems {
-		elem := elem
 		eg.Go(func() error {
+			t.processingMu.RLock()
 			if t.processing[elem.ID] != nil {
 				return fmt.Errorf("element with ID %s is already being processed", elem.ID)
 			}
+			t.processingMu.RUnlock()
+			t.processingMu.Lock()
 			t.processing[elem.ID] = &elem
+			t.processingMu.Unlock()
 			defer func() {
+				t.processingMu.Lock()
 				delete(t.processing, elem.ID)
+				t.processingMu.Unlock()
 			}()
 			return t.processElement(gctx, elem)
 		})
@@ -61,10 +66,12 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
 			t.Progress.OnProgress(ctx, t)
 		})
 		errg.Go(func() error {
+			defer pw.Close()
 			logger.Info("Starting file download in stream mode")
-			_, err := tdler.NewDownloader(t.client, elem.File).Stream(uploadCtx, wr)
-			if closeErr := pw.CloseWithError(err); closeErr != nil {
-				logger.Errorf("Failed to close pipe writer: %v", closeErr)
+			_, err := tfile.NewDownloader(elem.File).Stream(uploadCtx, wr)
+			if err != nil {
+				logger.Errorf("Failed to download file: %v", err)
+				pw.CloseWithError(err)
 			}
 			return err
 		})
@@ -88,7 +95,7 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
 		t.downloaded.Add(int64(n))
 		t.Progress.OnProgress(ctx, t)
 	})
-	_, err = tdler.NewDownloader(t.client, elem.File).Parallel(ctx, wrAt)
+	_, err = tfile.NewDownloader(elem.File).Parallel(ctx, wrAt)
 	if err != nil {
 		return fmt.Errorf("failed to download file: %w", err)
 	}
@@ -104,7 +111,7 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
 	if err != nil {
 		return fmt.Errorf("failed to get file stat: %w", err)
 	}
-	vctx := context.WithValue(ctx, key.ContextKeyContentLength, fileStat.Size())
+	vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
 	err = retry.Retry(func() error {
 		var file *os.File
 		file, err = os.Open(elem.localPath)

core/tasks/batchtfile/progress.go

@@ -1,4 +1,4 @@
-package batchtftask
+package batchtfile
 
 import (
 	"context"

core/tasks/batchtfile/task.go

@@ -1,13 +1,14 @@
-package batchtftask
+package batchtfile
 
 import (
 	"context"
 	"fmt"
 	"path/filepath"
+	"sync"
 	"sync/atomic"
 
-	"github.com/krau/SaveAny-Bot/common/tdler"
 	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
 	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"github.com/krau/SaveAny-Bot/storage"
 	"github.com/rs/xid"
@@ -29,10 +30,14 @@ type Task struct {
 	Progress     ProgressTracker
 	IgnoreErrors bool // if true, errors during processing will be ignored
 	downloaded   atomic.Int64
-	client       tdler.Client
 	totalSize    int64
 	processing   map[string]TaskElementInfo
-	failed       map[string]error // errors for each element
+	processingMu sync.RWMutex
+	failed       map[string]error // [TODO] errors for each element
+}
+
+func (t *Task) Type() tasktype.TaskType {
+	return tasktype.TaskTypeTgfiles
 }
 
 func NewTaskElement(
@@ -68,14 +73,12 @@ func NewBatchTGFileTask(
 	id string,
 	ctx context.Context,
 	files []TaskElement,
-	client tdler.Client,
 	progress ProgressTracker,
 	ignoreErrors bool,
 ) *Task {
 	task := &Task{
 		ID:         id,
 		Ctx:        ctx,
-		client:     client,
 		Elems:      files,
 		Progress:   progress,
 		downloaded: atomic.Int64{},
@@ -88,6 +91,7 @@ func NewBatchTGFileTask(
 		}(),
 		processing:   make(map[string]TaskElementInfo),
 		IgnoreErrors: ignoreErrors,
+		processingMu: sync.RWMutex{},
 		failed:       make(map[string]error),
 	}
 	return task

core/tasks/batchtfile/taskinfo.go

@@ -1,4 +1,4 @@
-package batchtftask
+package batchtfile
 
 type TaskElementInfo interface {
 	FileName() string

core/tasks/batchtfile/utils.go

@@ -1,4 +1,4 @@
-package batchtftask
+package batchtfile
 
 var progressUpdatesLevels = []struct {
 	size        int64 // 文件大小阈值

core/tasks/parsed/execute.go

@@ -0,0 +1,139 @@
+package parsed
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"path"
+	"path/filepath"
+
+	"github.com/charmbracelet/log"
+	"github.com/duke-git/lancet/v2/retry"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
+	"github.com/krau/SaveAny-Bot/common/utils/ioutil"
+	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
+	"github.com/krau/SaveAny-Bot/pkg/parser"
+	"golang.org/x/sync/errgroup"
+)
+
+func (t *Task) Execute(ctx context.Context) error {
+	logger := log.FromContext(ctx)
+	logger.Infof("Starting Parsed item task %s", t.item.Title)
+	if t.progress != nil {
+		t.progress.OnStart(ctx, t)
+	}
+	eg, gctx := errgroup.WithContext(ctx)
+	eg.SetLimit(config.Cfg.Workers)
+	for _, resource := range t.item.Resources {
+		eg.Go(func() error {
+			t.processingMu.RLock()
+			if t.processing[resource.ID()] != nil {
+				return fmt.Errorf("resource %s is already being processed", resource.ID())
+			}
+			t.processingMu.RUnlock()
+			t.processingMu.Lock()
+			t.processing[resource.ID()] = &resource
+			t.processingMu.Unlock()
+			defer func() {
+				t.processingMu.Lock()
+				delete(t.processing, resource.URL)
+				t.processingMu.Unlock()
+			}()
+			err := t.processResource(gctx, resource)
+			t.downloaded.Add(1)
+			if errors.Is(err, context.Canceled) {
+				logger.Debug("Resource processing canceled")
+				return err
+			}
+			if err != nil {
+				logger.Errorf("Error processing resource %s: %v", resource.URL, err)
+				return fmt.Errorf("failed to process resource %s: %w", resource.URL, err)
+			}
+			return nil
+		})
+	}
+	err := eg.Wait()
+	if err != nil {
+		logger.Errorf("Error during Parsed item task execution: %v", err)
+	} else {
+		logger.Infof("Parsed item task %s completed successfully", t.item.Title)
+	}
+	if t.progress != nil {
+		t.progress.OnDone(ctx, t, err)
+	}
+	return err
+}
+
+func (t *Task) processResource(ctx context.Context, resource parser.Resource) error {
+	logger := log.FromContext(ctx)
+	err := retry.Retry(func() error {
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, resource.URL, nil)
+		if err != nil {
+			return err
+		}
+		if resource.Headers != nil {
+			for k, v := range resource.Headers {
+				req.Header.Set(k, v)
+			}
+		}
+		resp, err := t.httpClient.Do(req)
+		if err != nil {
+			return fmt.Errorf("failed to download resource %s: %w", resource.URL, err)
+		}
+		defer resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			return fmt.Errorf("failed to download resource %s: %s", resource.URL, resp.Status)
+		}
+		ctx = context.WithValue(ctx, ctxkey.ContentLength, func() int64 {
+			if resource.Size > 0 {
+				return resource.Size
+			}
+			return resp.ContentLength
+		}())
+		if t.stream {
+			return t.Stor.Save(ctx, resp.Body, path.Join(t.StorPath, resource.Filename))
+		}
+		cacheFile, err := fsutil.CreateFile(filepath.Join(config.Cfg.Temp.BasePath,
+			fmt.Sprintf("resource_%s_%s", t.ID, resource.Filename)))
+		if err != nil {
+			return fmt.Errorf("failed to create cache file for resource %s: %w", resource.URL, err)
+		}
+		defer func() {
+			if err := cacheFile.CloseAndRemove(); err != nil {
+				logger.Errorf("Failed to close and remove cache file: %v", err)
+			}
+		}()
+		wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
+			t.downloadedBytes.Add(int64(n))
+			if t.progress != nil {
+				t.progress.OnProgress(ctx, t)
+			}
+		})
+
+		copyResultCh := make(chan error, 1)
+		go func() {
+			_, err := io.Copy(wr, resp.Body)
+			copyResultCh <- err
+		}()
+		select {
+		case err := <-copyResultCh:
+			if err != nil {
+				return fmt.Errorf("failed to copy resource %s to cache file: %w", resource.URL, err)
+			}
+		case <-ctx.Done():
+			return ctx.Err()
+		}
+		_, err = cacheFile.Seek(0, 0)
+		if err != nil {
+			return fmt.Errorf("failed to seek cache file for resource %s: %w", resource.URL, err)
+		}
+		return t.Stor.Save(ctx, cacheFile, path.Join(t.StorPath, resource.Filename))
+	}, retry.Context(ctx), retry.RetryTimes(uint(config.Cfg.Retry)))
+	if ctx.Err() != nil {
+		return ctx.Err()
+	}
+	return err
+}

core/tasks/parsed/progress.go

@@ -0,0 +1,209 @@
+package parsed
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"sync/atomic"
+	"time"
+
+	"github.com/charmbracelet/log"
+	"github.com/duke-git/lancet/v2/slice"
+	"github.com/gotd/td/telegram/message/entity"
+	"github.com/gotd/td/telegram/message/styling"
+	"github.com/gotd/td/tg"
+	"github.com/krau/SaveAny-Bot/common/utils/dlutil"
+	"github.com/krau/SaveAny-Bot/common/utils/tgutil"
+)
+
+var progressUpdatesLevels = []struct {
+	size        int64 // 文件大小阈值
+	stepPercent int   // 每多少 % 更新一次
+}{
+	{10 << 20, 100},
+	{50 << 20, 50},
+	{200 << 20, 20},
+	{500 << 20, 10},
+}
+
+func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
+	if total <= 0 || downloaded <= 0 {
+		return false
+	}
+
+	percent := int((downloaded * 100) / total)
+	if percent <= lastUpdatePercent {
+		return false
+	}
+
+	step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
+	for _, lvl := range progressUpdatesLevels {
+		if total < lvl.size {
+			step = lvl.stepPercent
+			break
+		}
+	}
+
+	return percent >= lastUpdatePercent+step
+}
+
+type ProgressTracker interface {
+	OnStart(ctx context.Context, info TaskInfo)
+	OnProgress(ctx context.Context, info TaskInfo)
+	OnDone(ctx context.Context, info TaskInfo, err error)
+}
+
+type Progress struct {
+	MessageID         int
+	ChatID            int64
+	start             time.Time
+	lastUpdatePercent atomic.Int32
+}
+
+func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
+	logger := log.FromContext(ctx)
+	p.start = time.Now()
+	p.lastUpdatePercent.Store(0)
+	logger.Debugf("Parsed task progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
+	entityBuilder := entity.Builder{}
+	var entities []tg.MessageEntityClass
+	if err := styling.Perform(&entityBuilder,
+		styling.Plain(fmt.Sprintf("开始下载 %s 的资源\n总大小: ", info.Site())),
+		styling.Code(fmt.Sprintf("%.2f MB (%d个资源)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
+	); err != nil {
+		log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
+		return
+	}
+	text, entities := entityBuilder.Complete()
+	req := &tg.MessagesEditMessageRequest{
+		ID: p.MessageID,
+	}
+	req.SetMessage(text)
+	req.SetEntities(entities)
+	req.SetReplyMarkup(&tg.ReplyInlineMarkup{
+		Rows: []tg.KeyboardButtonRow{
+			{
+				Buttons: []tg.KeyboardButtonClass{
+					tgutil.BuildCancelButton(info.TaskID()),
+				},
+			},
+		}},
+	)
+	ext := tgutil.ExtFromContext(ctx)
+	if ext != nil {
+		ext.EditMessage(p.ChatID, req)
+		return
+	}
+}
+
+func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
+	if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
+		return
+	}
+	percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
+	if p.lastUpdatePercent.Load() == int32(percent) {
+		return
+	}
+	p.lastUpdatePercent.Store(int32(percent))
+	log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.DownloadedBytes(), info.TotalBytes())
+	entityBuilder := entity.Builder{}
+	var entities []tg.MessageEntityClass
+	if err := styling.Perform(&entityBuilder,
+		styling.Plain("正在下载\n总大小: "),
+		styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
+		styling.Plain("\n正在处理:\n"),
+		func() styling.StyledTextOption {
+			var lines []string
+			for _, elem := range info.Processing() {
+				lines = append(lines, fmt.Sprintf("  - %s (%.2f MB)", elem.FileName(), float64(elem.FileSize())/(1024*1024)))
+			}
+			if len(lines) == 0 {
+				lines = append(lines, "  - 无")
+			}
+			return styling.Plain(slice.Join(lines, "\n"))
+		}(),
+		styling.Plain("\n平均速度: "),
+		styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(info.DownloadedBytes(), p.start)/(1024*1024))),
+		styling.Plain("\n当前进度: "),
+		styling.Bold(fmt.Sprintf("%.2f%%", float64(info.DownloadedBytes())/float64(info.TotalBytes())*100)),
+	); err != nil {
+		log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
+		return
+	}
+	text, entities := entityBuilder.Complete()
+	req := &tg.MessagesEditMessageRequest{
+		ID: p.MessageID,
+	}
+	req.SetMessage(text)
+	req.SetEntities(entities)
+	req.SetReplyMarkup(&tg.ReplyInlineMarkup{
+		Rows: []tg.KeyboardButtonRow{
+			{
+				Buttons: []tg.KeyboardButtonClass{
+					tgutil.BuildCancelButton(info.TaskID()),
+				},
+			},
+		}},
+	)
+	ext := tgutil.ExtFromContext(ctx)
+	if ext != nil {
+		ext.EditMessage(p.ChatID, req)
+		return
+	}
+}
+
+func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
+	logger := log.FromContext(ctx)
+	if err != nil {
+		if errors.Is(err, context.Canceled) {
+			logger.Infof("Parsed task %s was canceled", info.TaskID())
+			ext := tgutil.ExtFromContext(ctx)
+			if ext != nil {
+				ext.EditMessage(p.ChatID, &tg.MessagesEditMessageRequest{
+					ID:      p.MessageID,
+					Message: fmt.Sprintf("处理已取消: %s", info.TaskID()),
+				})
+			}
+		} else {
+			logger.Errorf("Parsed task %s failed: %s", info.TaskID(), err)
+			ext := tgutil.ExtFromContext(ctx)
+			if ext != nil {
+				ext.EditMessage(p.ChatID, &tg.MessagesEditMessageRequest{
+					ID:      p.MessageID,
+					Message: fmt.Sprintf("处理失败: %s", err.Error()),
+				})
+			}
+		}
+		return
+	}
+	logger.Infof("Parsed task %s completed successfully", info.TaskID())
+
+	entityBuilder := entity.Builder{}
+	if err := styling.Perform(&entityBuilder,
+		styling.Plain("处理完成, 资源数量: "),
+		styling.Code(fmt.Sprintf("%d", info.TotalResources())),
+		styling.Plain("\n保存路径: "),
+		styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
+	); err != nil {
+		logger.Errorf("Failed to build entities: %s", err)
+		return
+	}
+	text, entities := entityBuilder.Complete()
+	req := &tg.MessagesEditMessageRequest{
+		ID: p.MessageID,
+	}
+	req.SetMessage(text)
+	req.SetEntities(entities)
+
+	ext := tgutil.ExtFromContext(ctx)
+	if ext != nil {
+		ext.EditMessage(p.ChatID, req)
+	}
+}
+
+func NewProgress(messageID int, chatID int64) *Progress {
+	return &Progress{
+		MessageID: messageID,
+		ChatID:    chatID,
+	}
+}

core/tasks/parsed/task.go

@@ -0,0 +1,84 @@
+package parsed
+
+import (
+	"context"
+	"net/http"
+	"sync"
+	"sync/atomic"
+
+	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
+	"github.com/krau/SaveAny-Bot/pkg/parser"
+	"github.com/krau/SaveAny-Bot/storage"
+)
+
+type Task struct {
+	ID         string
+	Ctx        context.Context
+	Stor       storage.Storage
+	StorPath   string
+	item       *parser.Item
+	httpClient *http.Client
+	progress   ProgressTracker
+	stream     bool
+
+	totalResources  int64
+	downloaded      atomic.Int64 // downloaded resources count
+	totalBytes      int64        // total bytes to download
+	downloadedBytes atomic.Int64 // downloaded bytes count
+	processing      map[string]ResourceInfo
+	processingMu    sync.RWMutex
+	failed          map[string]error // [TODO] errors for each resource
+}
+
+func (t *Task) Type() tasktype.TaskType {
+	return tasktype.TaskTypeParseditem
+}
+
+func (t *Task) TaskID() string {
+	return t.ID
+}
+
+func NewTask(
+	id string,
+	ctx context.Context,
+	stor storage.Storage,
+	storPath string,
+	item *parser.Item,
+	progressTracker ProgressTracker,
+) *Task {
+	client := &http.Client{
+		Transport: &http.Transport{
+			// [TODO] configure it via config
+			Proxy: http.ProxyFromEnvironment,
+		},
+	}
+	_, ok := stor.(storage.StorageCannotStream)
+	stream := config.Cfg.Stream && !ok
+	return &Task{
+		ID:             id,
+		Ctx:            ctx,
+		Stor:           stor,
+		StorPath:       storPath,
+		item:           item,
+		totalResources: int64(len(item.Resources)),
+		downloaded:     atomic.Int64{},
+		totalBytes: func() int64 {
+			var total int64
+			for _, res := range item.Resources {
+				if res.Size < 0 {
+					continue // skip resources with unknown size
+				}
+				total += res.Size
+			}
+			return total
+		}(),
+		stream:          stream,
+		downloadedBytes: atomic.Int64{},
+		httpClient:      client,
+		progress:        progressTracker,
+		processing:      make(map[string]ResourceInfo),
+		processingMu:    sync.RWMutex{},
+		failed:          make(map[string]error),
+	}
+}

core/tasks/parsed/taskinfo.go

@@ -0,0 +1,51 @@
+package parsed
+
+type TaskInfo interface {
+	TaskID() string
+	Site() string
+	TotalResources() int64
+	Downloaded() int64
+	TotalBytes() int64
+	DownloadedBytes() int64
+	Processing() map[string]ResourceInfo
+	StorageName() string
+	StoragePath() string
+}
+
+func (t *Task) StoragePath() string {
+	return t.StorPath
+}
+func (t *Task) TotalResources() int64 {
+	return t.totalResources
+}
+
+func (t *Task) Downloaded() int64 {
+	return t.downloaded.Load()
+}
+
+func (t *Task) StorageName() string {
+	return t.Stor.Name()
+}
+
+func (t *Task) Site() string {
+	return t.item.Site
+}
+
+func (t *Task) TotalBytes() int64 {
+	return t.totalBytes
+}
+
+func (t *Task) DownloadedBytes() int64 {
+	return t.downloadedBytes.Load()
+}
+
+func (t *Task) Processing() map[string]ResourceInfo {
+	t.processingMu.RLock()
+	defer t.processingMu.RUnlock()
+	return t.processing
+}
+
+type ResourceInfo interface {
+	FileName() string
+	FileSize() int64
+}

core/tasks/telegraph/execute.go

@@ -1,4 +1,4 @@
-package tphtask
+package telegraph
 
 import (
 	"context"
@@ -22,8 +22,6 @@ func (t *Task) Execute(ctx context.Context) error {
 	eg, gctx := errgroup.WithContext(ctx)
 	eg.SetLimit(config.Cfg.Workers)
 	for i, pic := range t.Pics {
-		pic := pic
-		i := i
 		eg.Go(func() error {
 			err := t.processPic(gctx, pic, i)
 			if err != nil {
@@ -79,6 +77,11 @@ func (t *Task) processPic(ctx context.Context, picUrl string, index int) error {
 				lastErr = fmt.Errorf("failed to copy picture %s to cache file: %w", filename, lastErr)
 				return lastErr
 			}
+			_, err = cacheFile.Seek(0, 0)
+			if err != nil {
+				lastErr = fmt.Errorf("failed to seek cache file for picture %s: %w", filename, err)
+				return lastErr
+			}
 			lastErr = t.Stor.Save(ctx, cacheFile, path.Join(t.StorPath, filename))
 		} else {
 			lastErr = t.Stor.Save(ctx, body, path.Join(t.StorPath, filename))

core/tasks/telegraph/progress.go

@@ -1,4 +1,4 @@
-package tphtask
+package telegraph
 
 import (
 	"context"

core/tasks/telegraph/task.go

@@ -1,9 +1,10 @@
-package tphtask
+package telegraph
 
 import (
 	"context"
 	"sync/atomic"
 
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
 	"github.com/krau/SaveAny-Bot/pkg/telegraph"
 	"github.com/krau/SaveAny-Bot/storage"
 )
@@ -23,6 +24,10 @@ type Task struct {
 	downloaded   atomic.Int64
 }
 
+func (t *Task) Type() tasktype.TaskType {
+	return tasktype.TaskTypeTphpics
+}
+
 func NewTask(
 	id string,
 	ctx context.Context,
@@ -34,7 +39,7 @@ func NewTask(
 	progress ProgressTracker,
 ) *Task {
 	_, cannotStream := stor.(storage.StorageCannotStream)
-	tphtask := &Task{
+	telegraph := &Task{
 		ID:           id,
 		Ctx:          ctx,
 		PhPath:       phPath,
@@ -47,5 +52,5 @@ func NewTask(
 		totalpics:    len(pics),
 		downloaded:   atomic.Int64{},
 	}
-	return tphtask
+	return telegraph
 }

core/tasks/telegraph/taskinfo.go

@@ -1,4 +1,4 @@
-package tphtask
+package telegraph
 
 type TaskInfo interface {
 	TaskID() string

core/tasks/telegraph/utils.go

@@ -1,4 +1,4 @@
-package tphtask
+package telegraph
 
 func shouldUpdateProgress(downloaded int64, total int64) bool {
 	if total <= 0 || downloaded <= 0 {

core/tasks/tfile/execute.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 import (
 	"context"
@@ -8,15 +8,17 @@ import (
 	"time"
 
 	"github.com/charmbracelet/log"
-	"github.com/krau/SaveAny-Bot/common/tdler"
 	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	"github.com/krau/SaveAny-Bot/config"
-	"github.com/krau/SaveAny-Bot/pkg/enums/key"
+	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
+	"github.com/krau/SaveAny-Bot/pkg/tfile"
 )
 
-func (t *TGFileTask) Execute(ctx context.Context) error {
+func (t *Task) Execute(ctx context.Context) error {
 	logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", t.File.Name()))
-	t.Progress.OnStart(ctx, t)
+	if t.Progress != nil {
+		t.Progress.OnStart(ctx, t)
+	}
 	if t.stream {
 		return executeStream(ctx, t)
 	}
@@ -34,9 +36,11 @@ func (t *TGFileTask) Execute(ctx context.Context) error {
 	wrAt := newWriterAt(ctx, localFile, t.Progress, t)
 
 	defer func() {
-		t.Progress.OnDone(ctx, t, err)
+		if t.Progress != nil {
+			t.Progress.OnDone(ctx, t, err)
+		}
 	}()
-	_, err = tdler.NewDownloader(t.client, t.File).Parallel(ctx, wrAt)
+	_, err = tfile.NewDownloader(t.File).Parallel(ctx, wrAt)
 	if err != nil {
 		return fmt.Errorf("failed to download file: %w", err)
 	}
@@ -52,7 +56,7 @@ func (t *TGFileTask) Execute(ctx context.Context) error {
 	if err != nil {
 		return fmt.Errorf("failed to get file stat: %w", err)
 	}
-	vctx := context.WithValue(ctx, key.ContextKeyContentLength, fileStat.Size())
+	vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
 	for i := range config.Cfg.Retry + 1 {
 		if err = vctx.Err(); err != nil {
 			return fmt.Errorf("context canceled while saving file: %w", err)

core/tasks/tfile/progress.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 import (
 	"context"

core/tasks/tfile/stream.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 import (
 	"context"
@@ -6,11 +6,11 @@ import (
 	"io"
 
 	"github.com/charmbracelet/log"
-	"github.com/krau/SaveAny-Bot/common/tdler"
+	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"golang.org/x/sync/errgroup"
 )
 
-func executeStream(ctx context.Context, task *TGFileTask) error {
+func executeStream(ctx context.Context, task *Task) error {
 	logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", task.File.Name()))
 
 	pr, pw := io.Pipe()
@@ -21,16 +21,20 @@ func executeStream(ctx context.Context, task *TGFileTask) error {
 	})
 	wr := newWriter(ctx, pw, task.Progress, task)
 	errg.Go(func() error {
+		defer pw.Close()
 		logger.Info("Starting file download in stream mode")
-		_, err := tdler.NewDownloader(task.client, task.File).Stream(uploadCtx, wr)
-		if closeErr := pw.CloseWithError(err); closeErr != nil {
-			logger.Errorf("Failed to close pipe writer: %v", closeErr)
+		_, err := tfile.NewDownloader(task.File).Stream(uploadCtx, wr)
+		if err != nil {
+			logger.Errorf("Failed to download file: %v", err)
+			pw.CloseWithError(err)
 		}
 		return err
 	})
 	var err error
 	defer func() {
-		task.Progress.OnDone(ctx, task, err)
+		if task.Progress != nil {
+			task.Progress.OnDone(ctx, task, err)
+		}
 	}()
 	if err = errg.Wait(); err != nil {
 		return err

core/tasks/tfile/taskinfo.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 type TaskInfo interface {
 	TaskID() string
@@ -8,22 +8,22 @@ type TaskInfo interface {
 	StorageName() string
 }
 
-func (t *TGFileTask) TaskID() string {
+func (t *Task) TaskID() string {
 	return t.ID
 }
 
-func (t *TGFileTask) FileName() string {
+func (t *Task) FileName() string {
 	return t.File.Name()
 }
 
-func (t *TGFileTask) FileSize() int64 {
+func (t *Task) FileSize() int64 {
 	return t.File.Size()
 }
 
-func (t *TGFileTask) StoragePath() string {
+func (t *Task) StoragePath() string {
 	return t.Path
 }
 
-func (t *TGFileTask) StorageName() string {
+func (t *Task) StorageName() string {
 	return t.Storage.Name()
 }

core/tasks/tfile/tftask.go

@@ -1,59 +1,59 @@
-package tftask
+package tfile
 
 import (
 	"context"
 	"fmt"
 	"path/filepath"
 
-	"github.com/krau/SaveAny-Bot/common/tdler"
 	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
 	"github.com/krau/SaveAny-Bot/pkg/tfile"
 	"github.com/krau/SaveAny-Bot/storage"
 )
 
-type TGFileTask struct {
+type Task struct {
 	ID        string
 	Ctx       context.Context
 	File      tfile.TGFile
 	Storage   storage.Storage
 	Path      string
 	Progress  ProgressTracker
-	client    tdler.Client
 	stream    bool // true if the file should be downloaded in stream mode
 	localPath string
 }
 
+func (t *Task) Type() tasktype.TaskType {
+	return tasktype.TaskTypeTgfiles
+}
+
 func NewTGFileTask(
 	id string,
 	ctx context.Context,
 	file tfile.TGFile,
-	client tdler.Client,
 	stor storage.Storage,
 	path string,
 	progress ProgressTracker,
-) (*TGFileTask, error) {
+) (*Task, error) {
 	_, ok := stor.(storage.StorageCannotStream)
 	if !config.Cfg.Stream || ok {
 		cachePath, err := filepath.Abs(filepath.Join(config.Cfg.Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
 		if err != nil {
 			return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
 		}
-		tftask := &TGFileTask{
+		tfile := &Task{
 			ID:        id,
 			Ctx:       ctx,
-			client:    client,
 			File:      file,
 			Storage:   stor,
 			Path:      path,
 			Progress:  progress,
 			localPath: cachePath,
 		}
-		return tftask, nil
+		return tfile, nil
 	}
-	tfileTask := &TGFileTask{
+	tfileTask := &Task{
 		ID:       id,
 		Ctx:      ctx,
-		client:   client,
 		File:     file,
 		Storage:  stor,
 		Path:     path,

core/tasks/tfile/util.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 var progressUpdatesLevels = []struct {
 	size        int64 // 文件大小阈值

core/tasks/tfile/writer.go

@@ -1,4 +1,4 @@
-package tftask
+package tfile
 
 import (
 	"context"
@@ -20,7 +20,9 @@ func (w *ProgressWriterAt) WriteAt(p []byte, off int64) (int, error) {
 	if err != nil {
 		return 0, err
 	}
-	w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
+	if w.progress != nil {
+		w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
+	}
 	return at, nil
 }
 
@@ -54,7 +56,9 @@ func (w *ProgressWriter) Write(p []byte) (int, error) {
 	if err != nil {
 		return 0, err
 	}
-	w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
+	if w.progress != nil {
+		w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
+	}
 	return at, nil
 }
 

database/chat.go

@@ -0,0 +1,39 @@
+package database
+
+import "context"
+
+func (user *User) WatchChat(ctx context.Context, chat WatchChat) error {
+	if len(user.WatchChats) == 0 {
+		user.WatchChats = make([]WatchChat, 0)
+	}
+
+	user.WatchChats = append(user.WatchChats, chat)
+	return db.WithContext(ctx).Save(user.WatchChats).Error
+}
+
+func (user *User) UnwatchChat(ctx context.Context, chatID int64) error {
+	var watchChat WatchChat
+	err := db.WithContext(ctx).Where("chat_id = ? AND user_id = ?", chatID, user.ID).First(&watchChat).Error
+	if err != nil {
+		return err
+	}
+	return db.WithContext(ctx).Unscoped().Delete(&watchChat).Error
+}
+
+func (user *User) WatchingChat(ctx context.Context, chatID int64) (bool, error) {
+	var count int64
+	err := db.WithContext(ctx).Model(&WatchChat{}).Where("chat_id = ? AND user_id = ?", chatID, user.ID).Count(&count).Error
+	if err != nil {
+		return false, err
+	}
+	return count > 0, nil
+}
+
+func GetWatchChatsByChatID(ctx context.Context, chatID int64) ([]*WatchChat, error) {
+	var watchChats []*WatchChat
+	err := db.WithContext(ctx).Where("chat_id = ?", chatID).Find(&watchChats).Error
+	if err != nil {
+		return nil, err
+	}
+	return watchChats, nil
+}

database/db.go

@@ -37,7 +37,7 @@ func Init(ctx context.Context) {
 		logger.Fatal("Failed to open database: ", err)
 	}
 	logger.Debug("Database connected")
-	if err := db.AutoMigrate(&User{}, &Dir{}, &Rule{}); err != nil {
+	if err := db.AutoMigrate(&User{}, &Dir{}, &Rule{}, &WatchChat{}); err != nil {
 		logger.Fatal("迁移数据库失败, 如果您从旧版本升级, 建议手动删除数据库文件后重试: ", err)
 	}
 	if err := syncUsers(ctx); err != nil {

database/model.go

@@ -12,6 +12,14 @@ type User struct {
 	Dirs           []Dir
 	ApplyRule      bool
 	Rules          []Rule
+	WatchChats     []WatchChat
+}
+
+type WatchChat struct {
+	gorm.Model
+	UserID uint // User's database ID (not chat ID)
+	ChatID int64
+	Filter string
 }
 
 type Dir struct {

database/user.go

@@ -1,6 +1,10 @@
 package database
 
-import "context"
+import (
+	"context"
+
+	"gorm.io/gorm/clause"
+)
 
 func CreateUser(ctx context.Context, chatID int64) error {
 	if _, err := GetUserByChatID(ctx, chatID); err == nil {
@@ -11,19 +15,16 @@ func CreateUser(ctx context.Context, chatID int64) error {
 
 func GetAllUsers(ctx context.Context) ([]User, error) {
 	var users []User
-	err := db.Preload("Dirs").
-		WithContext(ctx).
-		Preload("Rules").
+	err := db.WithContext(ctx).
+		Preload(clause.Associations).
 		Find(&users).Error
 	return users, err
 }
 
 func GetUserByChatID(ctx context.Context, chatID int64) (*User, error) {
 	var user User
-	err := db.
-		Preload("Dirs").
-		WithContext(ctx).
-		Preload("Rules").
+	err := db.WithContext(ctx).
+		Preload(clause.Associations).
 		Where("chat_id = ?", chatID).First(&user).Error
 	return &user, err
 }
@@ -36,5 +37,16 @@ func UpdateUser(ctx context.Context, user *User) error {
 }
 
 func DeleteUser(ctx context.Context, user *User) error {
-	return db.WithContext(ctx).Unscoped().Select("Dirs", "Rules").Delete(user).Error
+	return db.WithContext(ctx).
+		Unscoped().
+		Select(clause.Associations).
+		Delete(user).Error
 }
+
+func GetUserByID(ctx context.Context, id uint) (*User, error) {
+	var user User
+	err := db.WithContext(ctx).
+		Preload(clause.Associations).
+		Where("id = ?", id).First(&user).Error
+	return &user, err
+}

docs/.gitignore

@@ -0,0 +1 @@
+public/

docs/archetypes/default.md

@@ -0,0 +1,5 @@
++++
+date = '{{ .Date }}'
+draft = true
+title = '{{ replace .File.ContentBaseName "-" " " | title }}'
++++

docs/assets/_variables.scss

@@ -0,0 +1 @@
+$font-size-base: 18px; 

docs/content/en/_index.md

@@ -0,0 +1,31 @@
+---
+title: Introduction
+---
+
+# Save Any Bot
+
+![](https://img.shields.io/github/go-mod/go-version/krau/SaveAny-Bot?style=flat-square)
+![](https://img.shields.io/github/license/krau/SaveAny-Bot?style=flat-square)
+![](https://img.shields.io/github/v/release/krau/SaveAny-Bot?color=cyan&style=flat-square)
+![](https://img.shields.io/github/downloads/krau/SaveAny-Bot/total?style=flat-square)
+
+Save Any Bot is a tool that allows you to save files from Telegram to various storage backends.
+
+## Features
+
+- Supports documents/videos/images/stickers... and even Telegraph
+- Breaks restrictions on saving files
+- Batch download
+- Streaming
+- Multi-user
+- Automatic organization based on storage rules
+- Supports multiple storage backends:
+    - Alist
+    - Minio (S3 compatible)
+    - WebDAV
+    - Telegram (re-upload to specified chat)
+    - Local disk
+
+## [Contributors](https://github.com/krau/SaveAny-Bot/graphs/contributors)
+
+![Contributors](https://contrib.rocks/image?repo=krau/SaveAny-Bot&max=750&columns=20)

docs/content/en/contribute/_index.md

@@ -0,0 +1,14 @@
+---
+title: "Contributing"
+weight: 20
+---
+
+# Contributing
+
+## Contributing New Storage Backend
+
+1. Fork this repository and clone it to your local machine.
+2. Add the new storage backend type in `pkg/enums/storage/storages.go` and run code generation.
+3. Define the storage backend configuration in the `config/storage` directory and add it to `config/storage/factory.go`.
+4. Create a new package in the `storage` directory, implement the storage backend, and import it in `storage/storage.go`.
+5. Update the documentation to include configuration details for the new storage backend.

docs/content/en/deployment/_index.md

@@ -0,0 +1,4 @@
+---
+title: "Deployment Guide"
+weight: 5
+---

docs/content/en/deployment/configuration/_index.md

@@ -0,0 +1,141 @@
+---
+title: "Configuration Guide"
+---
+
+# Configuration Guide
+
+SaveAnyBot uses the toml format for its configuration files. You can learn more about toml syntax on the [TOML official website](https://toml.io/).
+
+SaveAnyBot needs to read a `config.toml` file in the working directory as its configuration file. If this file is missing, a default file will be created, and the bot will attempt to load configuration from environment variables.
+
+Here is an example of a minimal configuration file:
+
+```toml
+[telegram]
+token = "1234567890:ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
+[[users]]
+# telegram user id
+id = 777000
+blacklist = true
+
+[[storages]]
+name = "Local Storage"
+type = "local"
+enable = true
+base_path = "./downloads"
+```
+
+## Detailed Configuration
+
+### Global Configuration
+
+- `stream`: Whether to enable Stream mode, default is `false`. When enabled, the Bot will stream files directly to storage endpoints (if supported), without downloading them locally.
+{{< hint warning >}}
+Stream mode is very useful for deployment environments with limited disk space, but it also has some drawbacks:
+<br />
+<ul>
+<li>Cannot use multi-threading to download files from Telegram, resulting in slower speeds.</li>
+<li>Higher task failure rate when the network is unstable.</li>
+<li>Cannot process files in the middle layer, such as automatic file type identification.</li>
+<li>Not supported by all storage endpoints; unsupported endpoints may downgrade to normal mode or fail to upload.</li>
+</ul>
+{{< /hint >}}
+- `workers`: Number of tasks to process simultaneously, default is 3.
+- `threads`: Number of threads used when downloading files, default is 4. Only effective when Stream mode is not enabled.
+- `retry`: Number of retries when a task fails, default is 3.
+
+### Telegram Configuration
+
+- `token`: Your Telegram Bot Token, which can be obtained by creating a Bot through [BotFather](https://t.me/botfather).
+- `app_id`, `app_hash`: Telegram API ID & Hash, obtained by creating an application at [Telegram API](https://my.telegram.org/apps). Default values will be used if not provided.
+- `flood_retry`: Number of retries for flood control, default is 5.
+- `rpc_retry`: Number of retries for RPC requests, default is 5.
+- `proxy`: Proxy configuration, optional.
+  - `enable`: Whether to enable the proxy.
+  - `url`: Proxy address, only supports `socks5://`
+
+```toml
+[telegram]
+token = "1234567890:ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+app_id = 1025907
+app_hash = "452b0359b988148995f22ff0f4229750"
+flood_retry = 5
+rpc_retry = 5
+[telegram.proxy]
+enable = false
+url = "socks5://127.0.0.1:7890"
+```
+
+### Storage Endpoints List
+
+The storage endpoints list is used to define the storage locations supported by the Bot. Each storage endpoint needs to specify a name, type, and related configuration, using the double bracket syntax `[[storages]]`.
+
+Each storage endpoint requires at least the following fields:
+
+- `name`: Storage endpoint name, used for identification in the Bot, must be unique.
+- `enable`: Whether to enable this storage endpoint, default is `true`.
+- `type`: Storage endpoint type, currently supports the following types:
+  - `local`: Local disk
+  - `alist`: Alist
+  - `webdav`: WebDAV
+  - `minio`: MinIO (compatible with S3 API)
+  - `telegram`: Upload to Telegram
+
+Example, this is a configuration that includes local storage and webdav storage:
+
+```toml
+[[storages]]
+name = "Local Storage"
+type = "local"
+enable = true
+# Custom configuration for local type storage
+base_path = "./downloads"
+
+[[storages]]
+name = "WebDAV"
+type = "webdav"
+enable = true
+# Custom configuration for webdav type storage
+url = "https://example.com/webdav"
+base_path = "/path/to/webdav"
+username = "your_username"
+password = "your_password"
+```
+
+For custom configuration items for all storage endpoints, see [Storage Configuration](./storages)
+
+### User List
+
+The user list is used to define access control for storage endpoints. Each user needs to specify a Telegram User ID, defined using the double bracket syntax `[[users]]`.
+
+- `id`: The user's Telegram User ID
+- `storages`: Filtered list of storage endpoints, defined by storage endpoint names, default is whitelist mode (i.e., only allows access to storage endpoints in the list)
+- `blacklist`: Whether to enable blacklist mode, default is `false`. If blacklist mode is enabled, the user is allowed to access only storage endpoints that are **not** in the list.
+
+Example, this is a configuration containing three users: user `123123` can only access local storage, user `456456` can only access storage other than WebDAV, and user `789789` has blacklist mode enabled but no storage endpoints specified, so they can access all storage:
+
+```toml
+[[users]]
+id = 123123
+storages = ["Local Storage"]
+
+[[users]]
+id = 456456
+storages = ["WebDAV"]
+blacklist = true
+
+[[users]]
+id = 789789
+storages = []
+blacklist = true
+```
+
+### Miscellaneous
+
+```toml
+no_clean_cache = false # Whether not to clear the cache folder when exiting
+# Temporary download folder configuration
+[temp]
+base_path = "./cache"
+```

docs/content/en/deployment/configuration/storages.md

@@ -0,0 +1,65 @@
+---
+title: "Storage Configuration"
+---
+
+# Storage Configuration
+
+Please first read the [Configuration Guide](../) to understand the basic format of the configuration file.
+
+## Alist
+
+`type=alist`
+
+Stream mode is not supported.
+
+```toml
+url = "https://alist.example.com" # URL of Alist
+username = "your_username"  # Username for Alist
+password = "your_password" # Password for Alist
+base_path = "/path/saveanybot" # Base path in Alist, all files will be stored under this path
+token_exp = 3600 # Auto-refresh time for Alist access token, in seconds
+token = "your_token" 
+# Access token for Alist, optional, if not set, username and password will be used for authentication.
+# When using token authentication, the token cannot be automatically refreshed
+```
+
+## Local Disk
+
+`type=local`
+
+```toml
+base_path = "./downloads" # Base path for local storage, all files will be stored under this path
+```
+
+## WebDAV
+`type=webdav`
+
+```toml
+url = "https://webdav.example.com" # URL of WebDAV
+username = "your_username"  # Username for WebDAV
+password = "your_password" # Password for WebDAV
+base_path = "/path/to/webdav" # Base path in WebDAV, all files will be stored under this path
+```
+
+## MinIO (S3)
+
+`type=minio`
+
+```toml
+endpoint = "minio.example.com" # Endpoint for MinIO or S3
+access_key_id = "your_access_key_id" # Access key ID for MinIO or S3
+secret_access_key = "your_secret_access_key" # Secret access key for MinIO or S3
+bucket_name = "your_bucket_name" # Bucket name for MinIO or S3
+use_ssl = true # Whether to use SSL, default is true
+base_path = "/path/to/minio" # Base path in MinIO, all files will be stored under this path
+```
+
+## Telegram
+
+`type=telegram`
+
+Stream mode is not supported.
+
+```toml
+chat_id = "123456789" # Telegram chat ID, the Bot will send files to this chat
+```

docs/content/en/deployment/installation.md

@@ -0,0 +1,145 @@
+---
+title: "Installation and Updates"
+---
+
+# Installation and Updates
+
+## Deploy from Pre-compiled Files
+
+Download the binary file for your platform from the [Release](https://github.com/krau/SaveAny-Bot/releases) page.
+
+Create a `config.toml` file in the extracted directory, refer to the [Configuration Guide](../configuration) to edit the configuration file.
+
+Run:
+
+```bash
+chmod +x saveany-bot
+./saveany-bot
+```
+
+### Process Monitoring
+
+{{< tabs "daemon" >}}
+{{< tab "systemd (Regular Linux)" >}}
+
+Create a file <code>/etc/systemd/system/saveany-bot.service</code> and write the following content:
+
+{{< codeblock >}}
+[Unit]
+Description=SaveAnyBot
+After=systemd-user-sessions.service
+
+[Service]
+Type=simple
+WorkingDirectory=/yourpath/
+ExecStart=/yourpath/saveany-bot
+Restart=on-failure
+
+[Install]
+WantedBy=multi-user.target
+{{< /codeblock >}}
+
+Enable startup on boot and start the service:
+
+{{< codeblock >}}
+systemctl enable --now saveany-bot
+{{< /codeblock >}}
+
+{{< /tab >}}
+
+{{< tab "procd (OpenWrt)" >}}
+
+<h4>Add Boot Autostart Service</h4>
+
+Create a file <code>/etc/init.d/saveanybot</code>, refer to <a href="https://github.com/krau/SaveAny-Bot/blob/main/docs/confs/wrt_init" target="_blank">wrt_init</a> and modify as needed:
+
+{{< codeblock >}}
+#!/bin/sh /etc/rc.common
+
+#This is the OpenWRT init.d script for SaveAnyBot
+
+START=99 
+STOP=10
+description="SaveAnyBot"
+
+WORKING_DIR="/mnt/mmc1-1/SaveAnyBot"
+EXEC_PATH="$WORKING_DIR/saveany-bot"
+start() {
+    echo "Starting SaveAnyBot..."
+    cd $WORKING_DIR
+    $EXEC_PATH &
+}
+stop() {
+    echo "Stopping SaveAnyBot..."
+    killall saveany-bot
+}
+reload() {
+    stop
+    start
+}
+
+{{< /codeblock >}}
+
+Set permissions:
+
+{{< codeblock >}}
+chmod +x /etc/init.d/saveanybot
+{{< /codeblock >}}
+
+Then copy the file to <code>/etc/rc.d</code> and rename it to <code>S99saveanybot</code>, also set permissions:
+
+{{< codeblock >}}
+chmod +x /etc/rc.d/S99saveanybot
+{{< /codeblock >}}
+
+<h4>Add Shortcut Commands</h4>
+
+Create a file <code>/usr/bin/sabot</code>, refer to <a href="https://github.com/krau/SaveAny-Bot/blob/main/docs/confs/wrt_bin" target="_blank">wrt_bin</a> and modify as needed. Note that the file encoding here only supports ANSI 936.
+
+Then set permissions:
+
+{{< codeblock >}}
+chmod +x /usr/bin/sabot
+{{< /codeblock >}}
+
+Usage: <code>sudo sabot start|stop|restart|status|enable|disable</code>
+
+{{< /tab >}}
+{{< /tabs >}}
+
+
+## Deploy Using Docker
+
+### Docker Compose
+
+Download the [docker-compose.yml](https://github.com/krau/SaveAny-Bot/blob/main/docker-compose.yml) file, create a new `config.toml` file in the same directory, refer to [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) to edit the configuration file.
+
+Start:
+
+```bash
+docker compose up -d
+```
+
+### Docker
+
+```shell
+docker run -d --name saveany-bot \
+    -v /path/to/config.toml:/app/config.toml \
+    -v /path/to/downloads:/app/downloads \
+    ghcr.io/krau/saveany-bot:latest
+```
+
+## Updates
+
+Use `upgrade` or `up` to upgrade to the latest version
+
+```bash
+./saveany-bot upgrade
+```
+
+If you deployed with Docker, use the following commands to update:
+
+```bash
+docker pull ghcr.io/krau/saveany-bot:latest
+docker restart saveany-bot
+```

docs/content/en/help/_index.md

@@ -0,0 +1,18 @@
+---
+title: "Frequently Asked Questions"
+weight: 15
+---
+
+# Frequently Asked Questions
+
+## Upload to AList shows success but actually fails
+
+Adjust the upload chunk size in the AList management page, and deploy AList in a more stable network environment to reduce the occurrence of this issue.
+
+## Bot indicates successful download but files don't show up in AList
+
+AList caches directory structures. Refer to the <a href="https://alist.nn.ci/guide/drivers/common.html#cache-expiration" target="_blank">documentation</a> to adjust cache expiration time.
+
+## Docker deployment still can't connect to Telegram despite proxy configuration (client initialization timeout)
+
+Docker cannot directly access the host network. If you're not familiar with its usage, please set the container to host mode.

docs/content/en/usage/_index.md

@@ -0,0 +1,65 @@
+---
+title: "Usage"
+weight: 10
+---
+
+# Usage
+
+## File Transfer
+
+The bot accepts two types of messages: files and links.
+
+Supported links:
+
+1. Telegram message links, for example: `https://t.me/acherkrau/1097`. **Even if the channel prohibits forwarding and saving, the bot can still download its files.**
+2. Telegra.ph article links, the bot will download all images within.
+
+## Silent Mode
+
+Use the `/silent` command to toggle silent mode.
+
+By default, silent mode is off, and the bot will ask you for the save location of each file.
+
+When silent mode is enabled, the bot will save files directly to the default location without confirmation.
+
+Before enabling silent mode, you need to set the default save location using the `/storage` command.
+
+
+## Storage Rules
+
+Allows you to set some redirection rules for the bot when uploading files to storage, for automatic organization of saved files.
+
+See: <a href="https://github.com/krau/SaveAny-Bot/issues/28" target="_blank">#28</a>
+
+Currently supported rule types:
+
+1. FILENAME-REGEX
+2. MESSAGE-REGEX
+
+Basic syntax for adding rules:
+
+"Rule Type Rule Content Storage Name Path"
+
+Pay attention to the use of spaces; the bot can only parse correctly formatted syntax. Below is an example of a valid rule command:
+
+```
+/rule add FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /videos
+```
+
+Additionally, if "CHOSEN" is used as the storage name in the rule, it means the file will be stored in the path of the storage selected via button click.
+
+Rule descriptions:
+
+### FILENAME-REGEX
+
+Matches based on filename regex. The rule content must be a valid regular expression, such as:
+
+```
+FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /videos
+```
+
+This means files with extensions mp4, mkv, ts, avi, flv will be saved to the /videos directory in the storage named MyAlist (also affected by the `base_path` in the configuration file).
+
+### MESSAGE-REGEX
+
+Similar to the above, but matches based on the text content of the message itself.

docs/content/zh/_index.md

@@ -0,0 +1,31 @@
+---
+title: 介绍
+---
+
+# Save Any Bot
+
+![](https://img.shields.io/github/go-mod/go-version/krau/SaveAny-Bot?style=flat-square)
+![](https://img.shields.io/github/license/krau/SaveAny-Bot?style=flat-square)
+![](https://img.shields.io/github/v/release/krau/SaveAny-Bot?color=cyan&style=flat-square)
+![](https://img.shields.io/github/downloads/krau/SaveAny-Bot/total?style=flat-square)
+
+把 Telegram 上的文件转存到多种存储端.
+
+## 特性
+
+- 支持文档/视频/图片/贴纸... 甚至还有 Telegraph
+- 破解禁止保存的文件
+- 批量下载
+- 流式传输
+- 多用户
+- 基于存储规则的自动整理
+- 支持多种存储端:
+    - Alist
+    - Minio (S3 兼容)
+    - WebDAV
+    - Telegram (重传回指定聊天)
+    - 本地磁盘
+
+## [贡献者](https://github.com/krau/SaveAny-Bot/graphs/contributors)
+
+![Contributors](https://contrib.rocks/image?repo=krau/SaveAny-Bot&max=750&columns=20)

docs/content/zh/contribute/_index.md

@@ -0,0 +1,14 @@
+---
+title: "参与开发"
+weight: 20
+---
+
+# 参与开发
+
+## 贡献新存储端
+
+1. Fork 本项目, 克隆到本地
+2. 在 `pkg/enums/storage/storages.go` 中添加新的存储端类型, 并运行代码生成
+3. 在 `config/storage` 目录下定义存储端配置, 并添加到 `config/storage/factory.go` 中
+4. 在 `storage` 目录下新建一个包, 编写存储端实现, 然后在 `storage/storage.go` 中导入并添加它
+5. 更新文档, 添加配置说明

docs/content/zh/deployment/_index.md

@@ -0,0 +1,4 @@
+---
+title: "部署指南"
+weight: 5
+---

docs/content/zh/deployment/configuration/_index.md

@@ -0,0 +1,174 @@
+---
+title: "配置说明"
+---
+
+# 配置说明
+
+SaveAnyBot 的配置文件使用 toml 格式, 你可以在 [TOML 官方网站](https://toml.io/) 上了解更多关于 toml 的语法.
+
+SaveAnyBot 需要读取工作目录下的 `config.toml` 文件作为配置文件, 若缺少该文件则会创建默认文件, 并尝试从环境变量中加载配置.
+
+以下是一个最简的配置文件示例:
+
+```toml
+[telegram]
+token = "1234567890:ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
+[[users]]
+# telegram user id
+id = 777000
+blacklist = true
+
+[[storages]]
+name = "本机存储"
+type = "local"
+enable = true
+base_path = "./downloads"
+```
+
+## 详细配置
+
+### 全局配置
+
+- `stream`: 是否启用 Stream 模式, 默认为 `false`. 启用后 Bot 将直接将文件流式传输到存储端(若存储端支持), 不需要下载到本地
+{{< hint warning >}}
+Stream 模式对于磁盘空间有限的部署环境十分有用, 但也有一些弊端:
+<br />
+<ul>
+<li>无法使用多线程从 Telegram 下载文件, 速度较慢.</li>
+<li>网络不稳定时, 任务失败率高.</li>
+<li>无法在中间层对文件进行处理, 例如自动文件类型识别.</li>
+<li>并非支持所有存储端, 不支持的存储端可能会降级为普通模式或无法上传.</li>
+</ul>
+{{< /hint >}}
+- `workers`: 同时处理任务数量, 默认为 3
+- `threads`: 下载文件时使用的线程数, 默认为 4. 仅在未启用 Stream 模式时生效.
+- `retry`: 任务失败时的重试次数, 默认为 3.
+
+### Telegram 配置
+
+- `token`: 你的 Telegram Bot Token, 可以通过 [BotFather](https://t.me/botfather) 创建 Bot 并获取 Token.
+- `app_id`, `app_hash`: Telegram API ID & Hash, 在 [Telegram API](https://my.telegram.org/apps) 创建应用获取, 若不提供则使用默认值.
+- `flood_retry`: Flood 控制重试次数, 默认为 5.
+- `rpc_retry`: RPC 请求重试次数, 默认为 5.
+- `proxy`: 代理配置, 可选.
+  - `enable`: 是否启用代理.
+  - `url`: 代理地址, 只支持 `socks5://`
+- `userbot`: userbot 配置, 可选.
+  - `enable`: 启用 userbot 集成, 需要登录用户账号, 此时请务必使用自己的 api id & hash.
+  - `session`: userbot 会话文件路径, 默认为 `data/usersession.db`.
+
+{{< hint warning >}}
+启用 userbot 集成后, bot 可以下载私密频道和群组的文件, 但具有无法避免的账号被封禁的风险.
+<br />
+开启 userbot 集成后第一次启动 bot 时需要通过终端交互输入手机号, 2FA 和验证码, 如果你使用 docker 部署, 请进入容器内执行相关操作.
+{{< /hint >}}
+
+```toml
+[telegram]
+token = "1234567890:ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+app_id = 1025907
+app_hash = "452b0359b988148995f22ff0f4229750"
+flood_retry = 5
+rpc_retry = 5
+[telegram.proxy]
+enable = false
+url = "socks5://127.0.0.1:7890"
+[telegram.userbot]
+enable = false
+session = "data/usersession.db"
+```
+
+### 存储端列表
+
+存储端列表用于定义 Bot 支持的存储位置, 每个存储端需要指定名称、类型和相关配置, 使用双中括号语法 `[[storages]]` 定义.
+
+每一个存储端至少需要以下字段:
+
+- `name`: 存储端名称, 用于在 Bot 中识别, 需要唯一
+- `enable`: 是否启用该存储端, 默认为 `true`
+- `type`: 存储端类型, 目前支持以下类型:
+  - `local`: 本地磁盘
+  - `alist`: Alist
+  - `webdav`: WebDAV
+  - `minio`: MinIO (兼容 S3 API)
+  - `telegram`: 上传到 Telegram
+
+示例, 这是一个包含本地存储和 webdav 存储的配置:
+
+```toml
+[[storages]]
+name = "本地存储"
+type = "local"
+enable = true
+# 以下是 local 类型存储的自定义配置
+base_path = "./downloads"
+
+[[storages]]
+name = "WebDAV"
+type = "webdav"
+enable = true
+# 以下是 webdav 类型存储的自定义配置
+url = "https://example.com/webdav"
+base_path = "/path/to/webdav"
+username = "your_username"
+password = "your_password"
+```
+
+所有存储端的自定义配置项可查看 [存储端配置](./storages) 
+
+### 用户列表
+
+用户列表用于定义对存储端的访问控制, 每个用户需要指定 Telegram 上的用户 ID, 使用双中括号语法 `[[users]]` 定义.
+
+- `id`: 用户的 Telegram User ID
+- `storages`: 过滤的存储端列表, 使用存储端名称定义, 默认为白名单模式 (即只允许访问列表中的存储端)
+- `blacklist`: 是否启用黑名单模式, 默认为 `false`. 若启用黑名单模式, 则仅允许访问**没有**在列表中的存储端.
+
+示例, 这是一个包含三个用户的配置, 用户 `123123` 只能访问本地存储, 用户 `456456` 只能访问除 WebDAV 以外的存储, 用户 `789789` 启用黑名单模式但没有指定存储端, 因此可以访问所有存储:
+
+```toml
+[[users]]
+id = 123123
+storages = ["本地存储"]
+
+[[users]]
+id = 456456
+storages = ["WebDAV"]
+blacklist = true
+
+[[users]]
+id = 789789
+storages = []
+blacklist = true
+```
+
+### 事件触发
+
+事件触发提供了在 Bot 处理任务时根据任务状态执行自定义操作的能力, 目前仅支持任意命令执行. 使用 `[hook.exec]` 配置.
+
+目前具有以下几种事件类型:
+
+- `task_before_start`: 任务即将开始前
+- `task_success`: 任务成功完成后
+- `task_fail`: 任务失败后
+- `task_cancel`: 任务被取消后
+
+提供的配置值需要为完整的命令行命令, Bot 会在事件发生时执行该命令. 示例:
+
+```toml
+[hook.exec]
+task_before_start = "echo '任务即将开始'"
+task_success = "bash /path/to/success_script.sh"
+task_fail = "curl -X POST https://example.com/api/notify -d '任务失败'"
+task_cancel = "bash /path/to/cancel_script.sh"
+```
+
+### 杂项
+
+```toml
+no_clean_cache = false # 是否在退出时不清空缓存文件夹
+# 临时下载文件夹配置
+[temp]
+base_path = "./cache"
+```

docs/content/zh/deployment/configuration/storages.md

@@ -0,0 +1,65 @@
+---
+title: "存储端配置"
+---
+
+# 存储端配置
+
+请先阅读 [配置说明](../) 了解配置文件的基本格式.
+
+## Alist
+
+`type=alist`
+
+不支持 Stream 模式.
+
+```toml
+url = "https://alist.example.com" # Alist 的 URL
+username = "your_username"  # Alist 的用户名
+password = "your_password" # Alist 的密码
+base_path = "/path/saveanybot" # Alist 中的基础路径, 所有文件将存储在此路径下
+token_exp = 3600 # Alist 访问令牌的自动刷新时间, 单位秒
+token = "your_token" 
+# Alist 的访问令牌, 可选, 如果不设置则使用用户名和密码进行身份验证. 
+# 使用 token 验证时无法自动刷新 token
+```
+
+## 本地磁盘
+
+`type=local`
+
+```toml
+base_path = "./downloads" # 本地存储的基础路径, 所有文件将存储在此路径下
+```
+
+## WebDAV
+`type=webdav`
+
+```toml
+url = "https://webdav.example.com" # WebDAV 的 URL
+username = "your_username"  # WebDAV
+password = "your_password" # WebDAV 的密码
+base_path = "/path/to/webdav" # WebDAV 中的基础路径, 所有文件将存储在此路径下
+```
+
+## MinIO (S3)
+
+`type=minio`
+
+```toml
+endpoint = "minio.example.com" # MinIO 或 S3 的端点
+access_key_id = "your_access_key_id" # MinIO 或 S3 的访问密钥 ID
+secret_access_key = "your_secret_access_key" # MinIO 或 S3 的秘密访问密钥
+bucket_name = "your_bucket_name" # MinIO 或 S3 的存储桶名称
+use_ssl = true # 是否使用 SSL, 默认为 true
+base_path = "/path/to/minio" # MinIO 中的基础路径, 所有文件将存储在此路径下
+```
+
+## Telegram
+
+`type=telegram`
+
+不支持 Stream 模式.
+
+```toml
+chat_id = "123456789" # Telegram 聊天 ID, Bot 将把文件发送到这个聊天
+```

docs/content/zh/deployment/installation.md

@@ -0,0 +1,145 @@
+---
+title: "安装与更新"
+---
+
+# 安装与更新
+
+## 从预编译文件部署
+
+在 [Release](https://github.com/krau/SaveAny-Bot/releases) 页面下载对应平台的二进制文件.
+
+在解压后目录新建 `config.toml` 文件, 参考 [配置说明](../configuration) 编辑配置文件
+
+运行:
+
+```bash
+chmod +x saveany-bot
+./saveany-bot
+```
+
+### 进程守护
+
+{{< tabs "daemon" >}}
+{{< tab "systemd (常规 Linux)" >}}
+
+创建文件 <code>/etc/systemd/system/saveany-bot.service</code> 并写入以下内容:
+
+{{< codeblock >}}
+[Unit]
+Description=SaveAnyBot
+After=systemd-user-sessions.service
+
+[Service]
+Type=simple
+WorkingDirectory=/yourpath/
+ExecStart=/yourpath/saveany-bot
+Restart=on-failure
+
+[Install]
+WantedBy=multi-user.target
+{{< /codeblock >}}
+
+设为开机启动并启动服务:
+
+{{< codeblock >}}
+systemctl enable --now saveany-bot
+{{< /codeblock >}}
+
+{{< /tab >}}
+
+{{< tab "procd (OpenWrt)" >}}
+
+<h4>添加开机自启动服务</h4>
+
+创建文件 <code>/etc/init.d/saveanybot</code> ，参考 <a href="https://github.com/krau/SaveAny-Bot/blob/main/docs/confs/wrt_init" target="_blank">wrt_init</a> 并自行修改:
+
+{{< codeblock >}}
+#!/bin/sh /etc/rc.common
+
+#This is the OpenWRT init.d script for SaveAnyBot
+
+START=99 
+STOP=10
+description="SaveAnyBot"
+
+WORKING_DIR="/mnt/mmc1-1/SaveAnyBot"
+EXEC_PATH="$WORKING_DIR/saveany-bot"
+start() {
+    echo "Starting SaveAnyBot..."
+    cd $WORKING_DIR
+    $EXEC_PATH &
+}
+stop() {
+    echo "Stopping SaveAnyBot..."
+    killall saveany-bot
+}
+reload() {
+    stop
+    start
+}
+
+{{< /codeblock >}}
+
+赋予权限:
+
+{{< codeblock >}}
+chmod +x /etc/init.d/saveanybot
+{{< /codeblock >}}
+
+然后将文件复制到 <code>/etc/rc.d</code> 并重命名为 <code>S99saveanybot</code>, 同样赋予权限:
+
+{{< codeblock >}}
+chmod +x /etc/rc.d/S99saveanybot
+{{< /codeblock >}}
+
+<h4>添加快捷指令</h4>
+
+创建文件 <code>/usr/bin/sabot</code> ，参考 <a href="https://github.com/krau/SaveAny-Bot/blob/main/docs/confs/wrt_bin" target="_blank">wrt_bin</a>  并自行修改，注意此处文件编码仅支持 ANSI 936 .
+
+随后赋予权限:
+
+{{< codeblock >}}
+chmod +x /usr/bin/sabot
+{{< /codeblock >}}
+
+使用: <code>sudo sabot start|stop|restart|status|enable|disable</code>
+
+{{< /tab >}}
+{{< /tabs >}}
+
+
+## 使用 Docker 部署
+
+### Docker Compose
+
+下载 [docker-compose.yml](https://github.com/krau/SaveAny-Bot/blob/main/docker-compose.yml) 文件, 在同目录下新建 `config.toml` 文件, 参考 [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) 编辑配置文件.
+
+启动:
+
+```bash
+docker compose up -d
+```
+
+### Docker
+
+```shell
+docker run -d --name saveany-bot \
+    -v /path/to/config.toml:/app/config.toml \
+    -v /path/to/downloads:/app/downloads \
+    ghcr.io/krau/saveany-bot:latest
+```
+
+## 更新
+
+使用 `upgrade` 或 `up` 升级到最新版
+
+```bash
+./saveany-bot upgrade
+```
+
+如果是 Docker 部署, 使用以下命令更新:
+
+```bash
+docker pull ghcr.io/krau/saveany-bot:latest
+docker restart saveany-bot
+```

docs/content/zh/help/_index.md

@@ -1,3 +1,8 @@
+---
+title: "常见问题"
+weight: 15
+---
+
 # 常见问题
 
 ## 上传 alist 失败也会显示成功
@@ -6,11 +11,8 @@
 
 ## Bot 提示下载成功但是 alist 未显示
 
-alist 缓存了目录结构, 参考文档可以调整缓存时间
-
-https://alist.nn.ci/zh/guide/drivers/common.html#缓存过期
+alist 缓存了目录结构, 参考 <a href="https://alist.nn.ci/zh/guide/drivers/common.html#缓存过期" target="_blank">文档</a> 可以调整缓存时间
 
 ## docker部署配置了代理后仍无法连接 telegram (初始化客户端超时)
 
-docker 不能直接访问宿主机网络, 如果你不熟悉其用法, 请将容器设为 host 模式:
-
+docker 不能直接访问宿主机网络, 如果你不熟悉其用法, 请将容器设为 host 模式.

docs/content/zh/usage/_index.md

@@ -0,0 +1,114 @@
+---
+title: "使用帮助"
+weight: 10
+---
+
+# 使用帮助
+
+这里介绍 Save Any Bot 的一些功能和使用方法, 如果你没有在这里找到你需要的内容, 另请参阅 [配置说明](../deployment/configuration) 或前往 Github [Discussions](https://github.com/krau/SaveAny-Bot/discussions) 提问.
+
+## 转存文件
+
+Bot 接受两种消息: 文件和链接.
+
+对于链接, 目前支持以下类型的链接:
+
+1. Telegram 消息链接, 例如: `https://t.me/acherkrau/1097`. **即使频道禁止了转发和保存, Bot 依然可以下载其文件.**
+2. Telegra.ph 的文章链接, Bot 将下载其中的所有图片
+
+## 静默模式 (silent)
+
+使用 `/silent` 命令可以开关静默模式.
+
+默认情况下不开启静默模式, Bot 会询问你每个文件的保存位置.
+
+开启静默模式后, Bot 会直接保存文件到默认位置, 无需确认.
+
+在开启静默模式之前, 需要使用 `/storage` 命令设置默认保存位置.
+
+## 存储规则
+
+允许你为 Bot 在上传文件到存储时设置一些重定向规则, 用于自动整理所保存的文件.
+
+见: <a href="https://github.com/krau/SaveAny-Bot/issues/28" target="_blank">#28</a>
+
+目前支持的规则类型:
+
+1. FILENAME-REGEX
+2. MESSAGE-REGEX
+3. IS-ALBUM
+
+添加规则的基本语法:
+
+"规则类型 规则内容 存储名 路径"
+
+注意空格的使用, 语法正确 bot 才能解析, 以下是一条合法的添加规则命令:
+
+```
+/rule add FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /视频
+```
+
+此外, 规则中的存储名若使用 "CHOSEN" , 则表示存储到点击按钮选择的存储端的路径下
+
+规则类型:
+
+### FILENAME-REGEX
+
+根据文件名正则匹配, 规则内容要求为一个合法的正则表达式, 如
+
+```
+FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /视频
+```
+
+表示将文件名后缀为 mp4,mkv,ts,avi,flv 的文件放到名为 MyAlist 存储下的 /视频 目录内 (同时受配置文件中的 `base_path` 影响)
+
+### MESSAGE-REGEX
+
+同上, 但是是根据消息本身的文本内容正则匹配
+
+### IS-ALBUM
+
+匹配相册消息 (media group), 规则内容只能为 `true` 或 `false`.
+
+规则中的路径若使用 "NEW-FOR-ALBUM" , 则表示为该组消息新建一个文件夹来存储它们. 见: https://github.com/krau/SaveAny-Bot/issues/87
+
+例如:
+
+```
+IS-ALBUM true MyWebdav NEW-FOR-ALBUM
+```
+
+这将会把以 media group 形式发送的消息保存到名为 MyWebdav 的存储下, 并为每个相册新建一个文件夹(由第一个文件生成)来存储它们.
+
+
+## 监听聊天
+
+{{< hint warning >}}
+该功能需开启 UserBot 集成.
+{{< /hint >}}
+
+监听指定聊天的消息, 并自动保存到默认存储中, 遵从存储规则, 并且可以设置过滤器来只保存匹配的消息.
+
+监听聊天:
+
+```
+/watch <chat_id/username> [filter] 
+```
+
+取消监听:
+
+```
+/unwatch <chat_id/username>
+```
+
+过滤器类型:
+
+### msgre
+
+正则匹配消息文本, 例如:
+
+```
+/watch 12345678 msgre:.*hello.*
+```
+
+这将会监听 ID 为 12345678 的聊天, 并且只保存消息文本中包含 "hello" 的消息.

docs/docs/contribute.md

@@ -1,11 +0,0 @@
-# 参与开发
-
-## 贡献新存储端
-
-1. Fork 本项目, 克隆到本地
-2. 在 `config/storage` 目录下定义存储端配置, 并添加到 `config/storage/factory.go` 中
-3. 在 `types/types.go` 中添加新的存储端类型
-4. 在 `storage` 目录下新建一个包, 编写存储端实现, 然后在 `storage/storage.go` 中导入并添加它
-5. 更新 `config.example.toml` 文件, 添加新的示例配置
-
-*可能确实有点麻烦了 = =*

docs/docs/deploy.md

@@ -1,94 +0,0 @@
-# 部署指南
-
-## 从二进制文件部署
-
-在 [Release](https://github.com/krau/SaveAny-Bot/releases) 页面下载对应平台的二进制文件.
-
-在解压后目录新建 `config.toml` 文件, 参考 [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) 编辑配置文件.
-
-运行:
-
-```bash
-chmod +x saveany-bot
-./saveany-bot
-```
-
-### 添加为 systemd 服务
-
-创建文件 `/etc/systemd/system/saveany-bot.service` 并写入以下内容:
-
-```
-[Unit]
-Description=SaveAnyBot
-After=systemd-user-sessions.service
-
-[Service]
-Type=simple
-WorkingDirectory=/yourpath/
-ExecStart=/yourpath/saveany-bot
-Restart=on-failure
-
-[Install]
-WantedBy=multi-user.target
-```
-
-设为开机启动并启动服务:
-
-```bash
-systemctl enable --now saveany-bot
-```
-
-### 为OpenWrt及衍生系统添加开机自启动服务
-
-创建文件 ` /etc/init.d/saveanybot` ，参考[saveanybot](https://github.com/krau/SaveAny-Bot/blob/main/docs/saveanybot)自行修改.
-
-`chmod +x /etc/init.d/saveanybot`
-
-完成后，将文件复制到 `/etc/rc.d`并重命名为`S99saveanybot`.
-
-`chmod +x /etc/rc.d/S99saveanybot`
-
-### 为OpenWrt及衍生系统添加快捷指令
-
-创建文件` /usr/bin/sabot` ，参考[sabot](https://github.com/krau/SaveAny-Bot/blob/main/docs/sabot)自行配置修改，注意此处文件编码仅支持 ANSI 936 .
-
-`chmod +x /usr/bin/sabot`
-
-之后，终端输入`sabot start|stop|restart|status|enable|disable`即可.
-
-
-## 使用 Docker 部署
-
-### Docker Compose
-
-下载 [docker-compose.yml](https://github.com/krau/SaveAny-Bot/blob/main/docker-compose.yml) 文件, 在同目录下新建 `config.toml` 文件, 参考 [config.example.toml](https://github.com/krau/SaveAny-Bot/blob/main/config.example.toml) 编辑配置文件.
-
-启动:
-
-```bash
-docker compose up -d
-```
-
-### Docker
-
-```shell
-docker run -d --name saveany-bot \
-    -v /path/to/config.toml:/app/config.toml \
-    -v /path/to/downloads:/app/downloads \
-    ghcr.io/krau/saveany-bot:latest
-```
-
-## 更新
-
-使用 `upgrade` 或 `up` 升级到最新版
-
-```bash
-./saveany-bot upgrade
-```
-
-如果是 Docker 部署, 使用以下命令更新:
-
-```bash
-docker pull ghcr.io/krau/saveany-bot:latest
-docker restart saveany-bot
-```

docs/docs/experimental.md

@@ -1,46 +0,0 @@
-# 实验性功能
-
-这里的功能不太稳定, 且未来可能会被删除或修改。
-
-## 存储规则
-
-允许你为 Bot 在上传文件到存储时设置一些重定向规则, 用于自动整理所保存的文件.
-
-见: https://github.com/krau/SaveAny-Bot/issues/28
-
-目前支持的规则类型:
-
-1. FILENAME-REGEX
-2. MESSAGE-REGEX
-
-添加规则的基本语法:
-
-"规则类型 规则内容 存储名 路径"
-
-注意空格的使用, 语法正确 bot 才能解析, 以下是一条合法的添加规则命令:
-
-```
-/rule add FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /视频
-```
-
-此外, 规则中的存储名若使用 "CHOSEN" , 则表示存储到点击按钮选择的存储端的路径下
-
-规则介绍:
-
-### FILENAME-REGEX
-
-根据文件名正则匹配, 规则内容要求为一个合法的正则表达式, 如
-
-```
-FILENAME-REGEX (?i)\.(mp4|mkv|ts|avi|flv)$ MyAlist /视频
-```
-
-表示将文件名后缀为 mp4,mkv,ts,avi,flv 的文件放到名为 MyAlist 存储下的 /视频 目录内 (同时受配置文件中的 `base_path` 影响)
-
-### MESSAGE-REGEX
-
-同上, 根据消息文本内容正则匹配
-
-## 复制并发送媒体消息
-
-将接收到的文件(媒体)消息, 或链接对应的消息原样发送到当前聊天, 点击选择存储按钮中的 "发送到当前聊天" 即可.

docs/docs/help.md

@@ -1,38 +0,0 @@
-# 使用帮助
-
-## 保存文件
-
-Bot 接受两种消息: 文件和链接.
-
-支持以下链接:
-
-1. 公开频道 (具有用户名) 的消息链接, 例如: `https://t.me/acherkrau/1097`. **即使频道禁止了转发和保存, Bot 依然可以下载其文件.**
-2. Telegra.ph 的文章链接, Bot 将下载其中的所有图片
-
-## 静默模式 (silent)
-
-使用 `/silent` 命令可以开关静默模式.
-
-默认情况下不开启静默模式, Bot 会询问你每个文件的保存位置.
-
-开启静默模式后, Bot 会直接保存文件到默认位置, 无需确认.
-
-在开启静默模式之前, 需要使用 `/storage` 命令设置默认保存位置.
-
-## Stream 模式
-
-在配置文件中将 `stream` 设置为 `true` 可以开启 Stream 模式.
-
-未开启时, Bot 处理任务分为两步: 下载和上传. Bot 会将文件暂存到本地, 然后上传到对应存储位置, 最后删除本地文件.
-
-开启后, Bot 将直接将文件流式传输到存储端, 不需要下载到本地.
-
-该功能对于硬盘空间有限的部署环境十分有用, 然而相较于普通模式也具有一些弊端:
-
-- 无法使用多线程从 telegram 下载文件, 速度较慢.
-- 网络不稳定时, 任务失败率高.
-- 无法在中间层对文件进行处理, 例如自动文件类型识别.
-
-**不支持** Stream 模式的存储端:
-
-- alist

docs/docs/index.md

@@ -1,7 +0,0 @@
-# SaveAnyBot 文档
-
-SaveAnyBot 是一个可以保存 Telegram 上的文件到云存储的机器人, 就像 PikPak Bot 一样.
-
-不同的是, SaveAnyBot 提供更灵活的存储端选择, 并实现一些更强大的功能.
-
-本项目以 AGPL-3.0 协议开源, 请遵守协议使用.