Compare commits

..

27 Commits

Author SHA1 Message Date
krau
21a519fdbe test(bot): added regression test for callback sender resolution 2026-08-24 08:38:07 +08:00
krau
1d431dbc88 fix(bot): resolved callback sender id in permission check 2026-08-24 08:38:07 +08:00
krau
bd1926d200 chore: go fix codebase 2026-08-22 18:29:55 +08:00
krau
b0850fb5e5 fix(batch): show total size in progress header again
The #228 progress redesign dropped the total size that the old
batch progress message displayed. Add a TotalSize data field
(actual known sizes, formatted via dlutil.FormatSize) to the
batch status header template in both locales.
2026-08-22 17:39:55 +08:00
krau
75d83b0a3e fix: replace nil ctx to context.Background 2026-08-17 19:12:23 +08:00
Krau
c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00
krau
6fd95bbe1b chore: update agents instruction 2026-08-11 09:04:15 +08:00
krau
a1bd2ed762 fix: upgrade deps 2026-08-11 08:57:09 +08:00
krau
7e3e26fcf4 style: format codebase 2026-08-11 08:47:35 +08:00
Haopeng Huo
e49ebef977 fix(telegraph): parse page URLs from message entities (#232) 2026-08-11 08:43:46 +08:00
Haopeng Huo
da38f5bb88 fix: remove cache files after reader closes (#231) 2026-08-11 08:41:40 +08:00
Haopeng Huo
0e6ed66ef5 feat: show upload progress for batch file tasks (#228)
* feat: show upload progress for file tasks

* feat: show upload progress for batch file tasks

* style: distinguish download and upload phases

* style: mark successful task completion

* fix: preserve storage save error context

* fix: serialize single-file progress updates

* fix: stabilize batch upload progress reporting

Correct batch upload totals, completion state, actual file sizes, and synchronized progress snapshots. Add regression coverage for concurrent updates and phase transitions.

* feat: show per-file transfer progress

Format single-file and batch download and upload states with Telegram entities, blockquotes, speeds, transferred sizes, progress bars, concise counters, and accurate confirmation handling. Cover upload retries plus final, error, and cancellation messages with regression tests.

* chore: remove transfer progress tests

* test: restore critical transfer progress coverage

* test: cover interleaved batch transfers

* fix: declare progress styles in locale templates
2026-08-11 08:40:29 +08:00
Haopeng Huo
e4144e73e6 feat(telegram): split oversized videos into playable parts (#229)
* feat(telegram): split oversized videos losslessly

* fix(telegram): respect Premium upload limits

* fix(telegram): keep split videos in one album
2026-08-05 16:57:46 +08:00
Haopeng Huo
1d794a7b9c feat(telegram): preserve source media groups (#226)
* feat(telegram): preserve source media groups

* fix(telegram): rewind batch readers before inspection
2026-08-04 16:46:29 +08:00
irudisca
52f880f0f2 fix(telegram): use seekable file path for thumbnail & metadata extraction (#225)
extractFrameAt() and getVideoMetadata() piped the media into ffmpeg/ffprobe
via pipe:0. A pipe is not seekable, so ffmpeg cannot decode non-faststart MP4s
whose moov atom sits at the END of the file (very common for yt-dlp / HLS-merged
downloads): the frame grab and probe silently fail, and the video is uploaded
with no thumbnail (and, for non-mp4 containers, no dimensions).

Hand ffmpeg/ffprobe a seekable file path instead. When the reader already is an
*os.File we use it directly; otherwise we spool to a temp file and clean it up.

Verified with an A/B upload of a 56 MB non-faststart clip: stock build produced
no thumbnail, patched build produced a correct thumbnail.

Co-authored-by: pennyucloud <valentino@pennyu.co.id>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 18:28:03 +08:00
krau
fc11ca775f fix: update message link parsing to support both t.me and telegram.me domains 2026-07-14 12:53:09 +08:00
krau
056e2fd546 fix: add ca-certificates installation and environment variables for SSL support in Dockerfile, close #222 2026-07-06 11:34:28 +08:00
krau
c9bb6c9e3c feat(docs): add CLI subcommands documentation and file naming strategies
- Introduced new documentation for CLI subcommands: `upload` and `watch`, detailing their usage, flags, and examples.
- Added a section on file naming and conflict strategies, explaining how users can customize file names and handle duplicates via `/config` and `/fnametmpl` commands.
- Translated the new documentation into Chinese for broader accessibility.
2026-06-28 18:00:02 +08:00
krau
2bc460c609 feat: add preset rule import functionality and update related messages 2026-06-26 15:39:14 +08:00
Krau
f02860ff3f feat: implement task event system for progress tracking and reporting (#220) 2026-06-25 21:37:36 +08:00
krau
9c2e70ed43 feat: upgrade deps 2026-06-25 16:30:21 +08:00
krau
3d6cd45909 feat: add configuration options for video download settings 2026-06-25 16:17:03 +08:00
Krau
77ef3154cf feat: add directory watching and auto-upload functionality (#219) 2026-06-25 14:34:59 +08:00
krau
88b170acaa fix: update backoff handling to use a factory function for concurrency safety 2026-06-25 10:22:31 +08:00
Ilham Syahid S
8059e27978 fix(api): sync task lifecycle state and restore GET /api/v1/tasks (#216)
* fix(api): update task route to handle GET and POST methods

Signed-off-by: Ilham Syahid S <ilhamsyahids@gmail.com>

* fix(api): implement ExecutableWrapper to manage task execution and status updates

Signed-off-by: Ilham Syahid S <ilhamsyahids@gmail.com>

* fix(api): refactor task registration and enqueueing into a separate method

Signed-off-by: Ilham Syahid S <ilhamsyahids@gmail.com>

---------

Signed-off-by: Ilham Syahid S <ilhamsyahids@gmail.com>
2026-05-24 23:42:16 +08:00
HLD
bfab4c85c8 fix: correct S3 signature and path handling (#213)
* fix: correct S3 signature and path handling

* fix: preserve existing overwrite behavior
2026-05-22 09:22:09 +08:00
Ryan
62e4a08e28 Merge pull request #204 from Rain-kl/feat/save-strategy
新增功能: 重名文件时选择保存策略 - 重命名,覆盖,跳过
2026-05-20 16:11:00 +08:00
156 changed files with 8099 additions and 1758 deletions

View File

@@ -43,7 +43,7 @@ jobs:
goarch: arm64
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v6
- name: Extract version from Git Ref
id: extract_version
@@ -64,7 +64,7 @@ jobs:
ldflags: >-
-s -w
-X "github.com/krau/SaveAny-Bot/config.Version=${{ env.VERSION }}"
-X "github.com/krau/SaveAny-Bot/config.BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}"
-X "github.com/krau/SaveAny-Bot/config.BuildTime=${{ github.event.repository.updated_at }}"
-X "github.com/krau/SaveAny-Bot/config.GitCommit=${{ github.sha }}"
binary_name: saveany-bot
env:

1
.gitignore vendored
View File

@@ -1,5 +1,6 @@
config.toml
logs/
/cache/
tmp/
data/
downloads/

366
AGENTS.md
View File

@@ -1,301 +1,115 @@
# SaveAny-Bot Agent Guidelines
This document provides essential information for AI coding agents working on the SaveAny-Bot project.
# Repository Guidelines
## Project Overview
SaveAny-Bot is a Telegram bot written in Go that saves files/messages from Telegram and various websites to multiple storage backends (local, S3, MinIO, WebDAV, AList, Telegram). It features a plugin system for parsing web content and extensible storage backends.
SaveAny-Bot is a Telegram bot written in Go that saves files and messages from Telegram and websites to multiple storage backends (Local, S3, MinIO, WebDAV, AList, Rclone, Telegram). It supports single-file saves, batch/album saves, streaming, multi-user access, storage rules, cross-storage transfers, yt-dlp and Aria2 downloads, and a Goja-based JavaScript parser plugin system with optional Playwright browser automation.
**Tech Stack**: Go 1.24.2, gotd/td (Telegram MTProto), Cobra (CLI), Viper (config), GORM (ORM), SQLite, Goja (JS runtime), Playwright (browser automation)
**Tech stack**: Go 1.25, gotgproto + gotd/td v0.149.0 (MTProto), Cobra (CLI), Viper (config), GORM + SQLite, Goja (JS runtime), Playwright, charmbracelet/log. License: AGPL-3.0.
## Build & Test Commands
**Note on gotd versions**: `gotd/td` must stay on `v0.149.0` — v0.150+ breaks `gotgproto` (v1.0.0-beta22) compilation (`AsInputDocumentFileLocation` signature, `gotd/log` Logger interface). Do not bump beyond v0.149.
## Architecture & Data Flow
```
Telegram update → client/bot/handlers → core.AddTask(Executable) → pkg/queue (serial workers)
→ core/tasks/* (download via common/tdler) → storage.Storage → progress feedback
```
- **Startup sequence** (`cmd/run.go::initAll`, keep this order): Config → Cache → i18n → Database → Storage → Parser plugins → Userbot → API → Bot. `bot.Init` returns the exit channel; a `SAVEANTBOT-RESTART` error restarts the process (external supervisor).
- **Task pipeline**: handlers build a task (via `core.AddTask`), the queue executes `Executable{Type, Title, TaskID, Execute(ctx)}` with `config.C().Workers` workers. Lifecycle hooks (`TaskBeforeStart/Success/Fail/Cancel`) run around `Execute`. Cancellation = canceling the task's context; tasks must check `ctx.Err()`.
- **Dual progress channels**: (1) `pkg/taskevent` context bus (consumed by `api/` for HTTP/Webhook consumers — `taskevent.WithSink` must be injected for API tasks), (2) Telegram message edits via `ProgressTracker` + `tgutil.ExtFromContext(ctx)` (bot tasks). Upload progress currently reaches the Telegram channel only.
- **Capability interfaces + type assertion fallback** is the core extensibility pattern: `StorageBatchSaver`, `StorageProgressSaver` (upload progress), `StorageListable`, `StorageReadable` are optional; consumers assert and fall back (e.g. wrap reader with `ioutil.NewProgressReader`). New backends only need to implement the interface and register.
- **Config layering**: CLI flag > env `SAVEANY_*` (dots → underscores, e.g. `SAVEANY_TELEGRAM_TOKEN`) > TOML file (path or http(s) URL). `config.C()` returns a **copy** — never mutate it.
- **Storage registration (3 places)**: `pkg/enums/storage` ENUM comment (go-enum), `config/storage/factory.go::storageFactories` (config struct with `Validate()`), `storage/storage.go::storageConstructors` (implementation).
## Key Directories
| Path | Purpose |
|---|---|
| `cmd/` | CLI: `run` (main bot), `upload`, `watch` (standalone subcommands that do NOT run initAll), `geni18n` (i18n key generator) |
| `core/` | `Executable` interface, queue worker loop, hooks; `core/tasks/{tfile,batchtfile,directlinks,parsed,telegraph,transfer,ytdlp,aria2dl}` |
| `client/bot/` | gotgproto client, `handlers/` (all commands + message/callback handlers), `middleware/`, `client/user/` (userbot) |
| `storage/` | 8 backends + `storage.go` (interfaces/registry) + `load.go` (per-user storage resolution) |
| `parsers/` | `parsers.go` (registry), `js/` (Goja plugins, ghttp/playwright injection, build-tagged), `parsers/` (native: twitter, kemono) |
| `config/` | Viper setup, defaults, `storage/` per-type config structs |
| `database/` | GORM models (User/Dir/Rule/WatchChat), AutoMigrate, `syncUsers` |
| `pkg/` | `queue`, `taskevent`, `tcbdata` (callback data), `rule`, `enums/{tasktype,storage,ctxkey,fnamest}`, `storagetypes`, `tfile`, `parser` |
| `common/` | `tdler` (unified downloader), `utils/{tgutil,dlutil,ioutil,fsutil,strutil,tphutil,netutil}`, `i18n` (embedded locales), `cache` (ristretto) |
| `api/` | HTTP API + webhook (task factory with sink injection) |
| `docs/` | Hugo site (hugo-book theme, zh+en mirrored), separate go.mod |
| `plugins/` | JS parser examples + `README.md` (plugin author contract) |
## Development Commands
### Build
```bash
# Standard build
go build -o saveany-bot .
# Run directly
# Build (standard; CGO_ENABLED=0 for static)
CGO_ENABLED=0 go build -trimpath -o saveany-bot .
go run ./cmd
# Docker build (multi-stage, Alpine-based)
docker build -t saveany-bot .
docker compose up -d
```
### Test
```bash
# Run all tests
# Test — known failures: storage/telegram TestCreateSplitZip/TestExtractThumbFrame/TestGetVideoMetadata
# (need gitignored fixtures tests/testfile.dat, tests/testvideo; ffmpeg/ffprobe)
go test ./...
# Run tests in specific package
go test ./pkg/queue
go test ./storage/telegram
# Run tests with verbose output
go test -v ./...
# Run a single test
go test -race ./core/tasks/... ./storage/... ./pkg/queue/... ./common/...
go test -run TestQueueBasic ./pkg/queue
# Run with coverage
go test -cover ./...
```
# Codegen — run after editing locale YAML or enum comments
go generate ./... # geni18n (i18nk keys) + go-enum (pkg/enums/*)
# go-enum is NOT in go.mod; install externally. geni18n runs via go run.
### Lint & Format
```bash
# Format code (standard Go formatting)
go fmt ./...
# Vet code for common issues
# Verify
go vet ./...
# Generate code (i18n keys)
go generate ./...
go fmt ./...
```
### Other Commands
```bash
# Update dependencies
go mod tidy
**Build variants** (Dockerfile.default/micro/pico): `-tags=no_jsparser,no_playwright,no_minio,no_bubbletea,sqlite_glebarez` — each has a `*_stub.go`/`*_glebarez.go` pairing; keep stubs in sync.
# View documentation
cd docs && hugo server -D
```
Docker: `docker build -t saveany-bot .`, `docker compose up -d` (host network, mounts `./data ./config.toml ./downloads ./cache`). CI (`.github/workflows/`) runs **no tests/lint** — only tag-triggered release/docker builds and docs deployment; run `go test ./...` manually before pushing.
## Code Style Guidelines
## Code Conventions & Common Patterns
### Imports
- Standard library first, then third-party, then project-internal
- Group imports with blank lines between groups
- Use explicit import aliases for clarity when needed (e.g., `storconfig`, `storenum`)
- **Imports**: stdlib → third-party → project-internal, blank-line separated. Aliases for clarity (`storconfig`, `storenum`).
- **Naming**: PascalCase exported, camelCase unexported, files `snake_case.go`; **not** ALL_CAPS constants.
- **Errors**: always wrap with `fmt.Errorf("context: %w", err)`; check with `errors.Is/As`; never ignore.
- **Logging**: `log.FromContext(ctx)` with prefixes (`logger.WithPrefix("component")`); never global logger when ctx is available.
- **Context values** (read from the passed ctx, never globals): `log.FromContext`, `tgutil.ExtFromContext` (Telegram ext — **required for message edits; if nil, edits are silently dropped**), `storage.FromContext`, `storagetypes.WithSourceCaption`, `ctxkey.ContentLength` / `ctxkey.OverwriteExisting`.
- **Progress rendering** (#228 convention): i18n templates declare styles with Telegram HTML (`<b>/<code>/<blockquote>/<i>`); dynamic data MUST go through `i18n.T(key, tgutil.EscapeHTMLTemplateData(data))` before `tgutil.RenderHTML`. Never interpolate user data raw, never render-then-substring-search.
- **Progress tracking**: each task package defines its own small `ProgressTracker` interface; optional `UploadProgressTracker` is probed via type assertion (skip if absent). Serialize state + message edits with a mutex; throttle edits (≥1s); aggregate per-item progress monotonically.
- **i18n**: only edit `common/i18n/locale/{zh-Hans,en}.yaml``go generate ./...` → use `i18nk.<Key>` constants. No raw strings in user-facing messages. zh-Hans and en must stay in sync.
- **Registration points** (never forget): new bot command → `client/bot/handlers/register.go::CommandHandlers` (auto-publishes /help menu); new task type → `pkg/enums/tasktype` + `core/tasks/<name>/` + `api/factory.go::CreateTask`; new storage → 3 places above + `docs/content/{en,zh}/deployment/configuration/storages.md`; new enum value → ENUM comment + `go generate`.
- **Concurrency**: `errgroup.WithContext` + `SetLimit(config.C().Workers)`, `atomic.Int64` counters, `sync.Once` for single-shot events, mutex around render state. No lock-in-callback (callbacks fire after unlock).
- **Cancellation**: queue tasks carry a `WithCancel`-derived ctx; check `ctx.Err()` in loops; classify with `errors.Is(err, context.Canceled)`.
- **JS plugins**: `registerParser({metadata, canHandle, parse})`, `version >= 1.0.0`; per-plugin goja VM is single-goroutine (reqCh buffer 10). Changing `pkg/parser.Item/Resource` JSON fields requires updating `plugins/README.md` and example plugins.
- **Message edits**: `ext.EditMessage(chatID, &tg.MessagesEditMessageRequest{...})`; cancel buttons via `tgutil.BuildCancelButton(taskID)`; callback payloads via `pkg/tcbdata` + `common/cache`.
```go
import (
"context"
"fmt"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/storage"
)
```
## Important Files
### Formatting
- Line length: reasonable (no hard limit, but be sensible)
- Organize code with blank lines between logical sections
- Follow standard Go conventions for braces, spacing, etc.
- `main.go``//go:generate` for i18n keys
- `cmd/run.go` — startup sequence `Run/initAll/cleanCache` (cache cleanup on exit, `NoCleanCache` opt-out)
- `core/core.go` — worker loop, hooks, AddTask/CancelTask
- `pkg/queue/queue.go` — generic serial queue (cond/list; duplicate TaskID rejected)
- `storage/storage.go` — interfaces + registry + compile-time capability assertions
- `config/viper.go`, `config.example.toml` — config schema (authoritative field docs)
- `database/db.go` — GORM init, `GetDialect` (build-tag selectable SQLite driver)
- `client/bot/handlers/register.go` — handler dispatch order and CommandHandlers
- `common/tdler/dler.go` — unified download entry
- `core/tasks/batchtfile/item_progress.go` — per-item phase state machine (Downloading/Transferring/Uploading/Retrying/Confirming, FailureStage)
- `parsers/js/plugin.go` — Goja plugin runtime
- `.github/workflows/` — release/docker/docs (no test gate)
### Types & Interfaces
- Use clear, descriptive type names (PascalCase for exported, camelCase for unexported)
- Define interfaces where abstraction is needed (e.g., `Executable`, `StorageConfig`)
- Embed context in method signatures, not structs: `func (s *Service) Do(ctx context.Context) error`
- Prefer composition over inheritance
## Runtime/Tooling Preferences
```go
// Interfaces define behavior
type Executable interface {
Type() tasktype.TaskType
Title() string
TaskID() string
Execute(ctx context.Context) error
}
- **Go 1.25+**: `t.Context()`, `sync.WaitGroup.Go`, `for range n` are available.
- **Runtime binaries**: ffmpeg/ffprobe (media processing/video split), yt-dlp (ytdlp tasks), aria2 optional; Playwright browsers install on demand to `./playwright` (`playwright.Install(chromium, ...)` at first `pw.get()`); Docker images: default has ffmpeg+yt-dlp, micro only curl, pico is scratch static.
- **No Makefile, no golangci.yml, no test/lint CI** — verification is manual (`go vet`, `go test`).
- **go-enum** required externally for enum generation; **geni18n** is in-repo.
- **Docs**: Hugo site in `docs/` (separate go.mod, hugo-book); edit `docs/content/{zh,en}/` — keep both languages mirrored. `docs/public/` is gitignored build output.
- **gitignored fixtures**: `storage/telegram/tests/` (missing — 3 tests fail locally), `data/`, `config.toml`, `playwright/`, `testplugins/`.
// Structs compose behavior
type Local struct {
config config.LocalStorageConfig
logger *log.Logger
}
```
## Testing & QA
### Naming Conventions
- **Packages**: lowercase, single word when possible (avoid underscores)
- **Files**: lowercase with underscores for multiword (e.g., `auth_terminal.go`, `progress_reader.go`)
- **Variables**: camelCase for unexported, PascalCase for exported
- **Constants**: PascalCase for exported, camelCase for unexported (not ALL_CAPS)
- **Functions/Methods**: PascalCase for exported, camelCase for unexported
- **Test files**: `*_test.go` pattern
### Error Handling
- Always handle errors explicitly; never ignore them
- Wrap errors with context using `fmt.Errorf("context: %w", err)`
- Use `errors.Is()` and `errors.As()` for error checking
- Log errors with appropriate level (Error, Warn, Info)
- Return errors from functions rather than panicking (except for truly unrecoverable situations)
```go
// Good error handling
if err := db.Save(user).Error; err != nil {
return fmt.Errorf("failed to save user %d: %w", user.ChatID, err)
}
// Check specific errors
if errors.Is(err, context.Canceled) {
logger.Info("Operation was canceled")
return nil
}
```
### Logging
- Use `github.com/charmbracelet/log` package
- Get logger from context: `log.FromContext(ctx)`
- Create prefixed loggers for components: `logger.WithPrefix("component")`
- Use appropriate levels: Debug, Info, Warn, Error
- Include context in log messages (e.g., task IDs, file names)
```go
logger := log.FromContext(ctx)
logger.Infof("Processing task: %s", task.ID)
logger.Errorf("Failed to save file %s: %v", filename, err)
```
### Concurrency
- Use channels for communication between goroutines
- Protect shared state with `sync.Mutex` or `sync.RWMutex`
- Use `sync.WaitGroup` for coordinating goroutine completion
- Always pass `context.Context` for cancellation support
- Use `context.WithCancel/WithTimeout` for managing goroutine lifetimes
```go
// Example from queue implementation
func (tq *TaskQueue[T]) Add(task *Task[T]) error {
tq.mu.Lock()
defer tq.mu.Unlock()
// ... critical section
tq.cond.Signal()
return nil
}
```
### Comments
- Document exported types, functions, and packages with doc comments
- Start doc comments with the name being documented
- Use `//` for single-line comments
- Explain *why*, not *what* (code should be self-explanatory for "what")
- Add `[NOTE]`, `[WARN]`, `[IMPORTANT]` tags for important clarifications
```go
// GetUserByChatID retrieves a user by their Telegram chat ID.
// Returns an error if the user is not found.
func GetUserByChatID(ctx context.Context, chatID int64) (*User, error) {
```
## Architecture & Conventions
### Application Structure
- **Entry point**: `main.go``cmd.Execute(ctx)`
- **CLI root**: `cmd/root.go` (Cobra), implementation in `cmd/run.go`
- **Startup sequence**: Config → Cache → i18n → Database → Storage → Parsers → Userbot → Bot → Queue
- Follow this order when adding new initialization steps in `cmd/run.go::initAll`
### Configuration (Viper)
- Config defined in `config/viper.go::Config`
- Read from `config.toml` (see `config.example.toml`)
- Environment variables: `SAVEANY_*` prefix (e.g., `SAVEANY_TELEGRAM_TOKEN`)
- Access via `config.C()` (returns a copy, don't modify the return value)
- Storage configs validated via `config/storage/factory.go::LoadStorageConfigs`
### Telegram Client
- **Bot client**: `client/bot/bot.go::Init` (uses gotgproto)
- **Handlers**: Centralized in `client/bot/handlers/` directory
- **Registration**: All handlers registered in `handlers.Register`
- **Commands**: Add to `CommandHandlers` slice for automatic `/help` and bot command list updates
- **Middleware**: Common middleware in `client/middleware/` (floodwait, retry, etc.)
### Tasks & Queue
- **Task interface**: `core/core.go::Executable` (Type, Title, TaskID, Execute methods)
- **Queue**: `pkg/queue.TaskQueue[Executable]` (generic, thread-safe)
- **Workers**: Count from `config.C().Workers`
- **Task types**: Implementations in `core/tasks/**` (tfile, parsed, telegraph, directlinks, batchtfile)
- **Lifecycle hooks**: `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` (defined in config)
- **Adding tasks**: Use `core.AddTask(ctx, task)`
### Database (GORM + SQLite)
- **Init**: `database.Init` using `config.C().DB.Path`
- **Models**: User, Dir, Rule, WatchChat (in `database/*.go`)
- **Migrations**: Automatic via `db.AutoMigrate`
- **User sync**: `database.syncUsers` syncs DB with `config.C().Users` (don't manually create/delete users)
- **Context**: Always use `db.WithContext(ctx)` for operations
### Storage Backends
- **Interface**: Defined in `config/storage/types.go` and `storage/`
- **Implementations**: local, alist, s3/minio, webdav, telegram (each in subdirectory)
- **Adding new storage**:
1. Add enum to `pkg/enums/storage`
2. Create config struct in `config/storage/` with `Validate()` method
3. Implement storage in `storage/<name>/`
4. Register in `storageFactories` mapping
5. Update `config.example.toml` with example
### Parser Plugins (JavaScript)
- **Runtime**: Goja (JS runtime) + Playwright (browser automation)
- **Plugin API**: `registerParser({ metadata, canHandle, parse })` in JS
- **Integration**: Defined in `parsers/` directory
- **Documentation**: See `plugins/README.md`
- Plugin `parse` returns `Item`/`Resource` which becomes download/transfer task
### Internationalization (i18n)
- **Usage**: `i18n.T(i18nk.SomeKey, map[string]any{"Name": value})`
- **Locale files**: `common/i18n/locale/*.yaml`
- **Key generation**: Run `go generate ./...` to generate `common/i18n/i18nk/keys.go`
- **Adding new strings**: Add to YAML → run `go generate` → use in code
- All user-facing strings should be internationalized
### Context Usage
- Always pass `context.Context` as first parameter
- Use `log.FromContext(ctx)` to get contextual logger
- Respect context cancellation in long-running operations
- Store request-scoped data in context (e.g., `ctxkey.ContentLength`)
## Special Rules from .github/copilot-instructions.md
1. **Never modify `config.C()` return values** - it returns a copy. Modify config in `config.Init` or via Viper.
2. **Handlers must update `CommandHandlers` slice** - ensures `/help` and bot commands stay in sync.
3. **Task execution must preserve hooks** - don't remove `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` hook calls.
4. **User sync is automatic** - don't manually create/delete users in DB; use config-based sync.
5. **Prefer context logger** - use `log.FromContext(ctx)` over global logger when context is available.
6. **Storage factory pattern** - new storage types must register in `storageFactories` mapping.
7. **Plugin API compatibility** - changes to `Item`/`Resource` structures require updating `plugins/README.md`.
## Common Patterns
### Adding a New Command
1. Create handler function in `client/bot/handlers/<name>.go`
2. Add to `CommandHandlers` slice in `register.go`
3. Add i18n key to `common/i18n/locale/*.yaml`
4. Run `go generate ./...`
5. Test with Telegram bot
### Adding a New Task Type
1. Create struct implementing `core.Executable` in `core/tasks/<type>/`
2. Implement `Type()`, `Title()`, `TaskID()`, `Execute(ctx)` methods
3. Add task type enum to `pkg/enums/tasktype`
4. Use `core.AddTask(ctx, task)` to enqueue
### Adding a New Storage Backend
1. Define config struct in `config/storage/<name>.go` with `Validate()` method
2. Implement storage interface in `storage/<name>/<name>.go`
3. Add storage type enum to `pkg/enums/storage`
4. Register factory in `config/storage/factory.go::storageFactories`
5. Update `config.example.toml` with configuration example
## File References
When referencing code locations, use `path/to/file.go:line` format (e.g., `core/core.go:23` for the worker function).
## Testing Guidelines
- Write tests for new functionality (place in `*_test.go` files)
- Test files should be in same package as code being tested
- Use table-driven tests for multiple test cases
- Mock external dependencies (databases, network calls)
- Aim for meaningful tests, not just coverage numbers
## Notes
- Binary size matters: use `CGO_ENABLED=0` for static binaries
- FFmpeg is included in Docker images for media processing
- Build process supports cross-compilation (amd64/arm64, Linux/macOS/Windows)
- Documentation site uses Hugo; edit files in `docs/` directory
- Session data stored in SQLite; delete `data/session.db` if changing bot token
- Pure stdlib `testing` (no testify); table-driven (`[]struct{name...}` + `t.Run`) with `t.Fatalf` got/want assertions. Mock via hand-written interface impls or package-variable replacement (`runMediaTool` in `video_split_test.go`, restored with `t.Cleanup`); in-process services for HTTP (`httptest`), S3 (`gofakes3+s3mem`), WebDAV (`x/net/webdav`).
- **Locale-dependent tests**: pin with `i18n.Init("zh-Hans")` + `t.Cleanup(...)`.
- **Progress/HTML tests**: assert rendered text with `strings.Contains` AND entity counts (`tg.MessageEntityBold/Code/Blockquote/Italic`) — verify style injection stays escaped (`<b>A&B</b>` input must render as literal text).
- **Known failures**: `storage/telegram` `TestCreateSplitZip`, `TestExtractThumbFrame`, `TestGetVideoMetadata` need gitignored fixtures + real ffmpeg — skip with `-skip 'Test(CreateSplitZip|ExtractThumbFrame|GetVideoMetadata)$'`; `api/handlers_test.go` has one `t.Skip` (needs initialized core).
- **Coverage expectations**: pure logic gets table tests (parsers, URL/path utils, progress throttling, grouping); regressions get bug-scenario-named tests (`progress_regression_test.go`). Network/Telegram/Playwright must never be touched by tests.
- When a permanent feature/API change ships: update `config.example.toml` if config, `docs/` if user-facing, `plugins/README.md` if plugin contract, and i18n YAML + `go generate` for new strings.

View File

@@ -7,6 +7,8 @@ ARG BuildTime="Unknown"
WORKDIR /app
RUN apk add --no-cache ca-certificates
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
@@ -31,5 +33,9 @@ FROM scratch
WORKDIR /app
COPY --from=builder /app/saveany-bot .
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_DIR=/etc/ssl/certs
ENTRYPOINT ["/app/saveany-bot"]

View File

@@ -1,7 +1,6 @@
package api
import (
"context"
"crypto/subtle"
"net/http"
"strings"
@@ -9,9 +8,6 @@ import (
"github.com/krau/SaveAny-Bot/config"
)
// tokenContextKey 用于在 context 中存储 token
type tokenContextKey struct{}
// AuthMiddleware 返回认证中间件
func AuthMiddleware() func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
@@ -40,9 +36,7 @@ func AuthMiddleware() func(http.Handler) http.Handler {
return
}
// 将 token 添加到 context
ctx := context.WithValue(r.Context(), tokenContextKey{}, token)
next.ServeHTTP(w, r.WithContext(ctx))
next.ServeHTTP(w, r)
})
}
}

View File

@@ -20,6 +20,7 @@ import (
"github.com/krau/SaveAny-Bot/pkg/aria2"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/parser"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
@@ -38,7 +39,7 @@ func NewTaskFactory(ctx context.Context) *TaskFactory {
// CreateTask 创建任务
func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, error) {
// 验证存储
stor, ok := storage.Storages[req.Storage]
stor, ok := storage.GetStorage(req.Storage)
if !ok {
return nil, fmt.Errorf("storage not found: %s", req.Storage)
}
@@ -66,6 +67,24 @@ func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, e
}
}
func (f *TaskFactory) registerAndEnqueueTask(task core.Executable, taskType tasktype.TaskType, storageName, path, webhook string) error {
taskID := task.TaskID()
info := RegisterTask(taskID, string(taskType), storageName, path, task.Title(), webhook)
// Inject the progress sink into the context so the task's Emit calls update
// the API store (and fire the webhook on terminal states) without the task
// knowing about the API.
taskCtx := taskevent.WithSink(f.ctx, info)
err := core.AddTask(taskCtx, task)
if err != nil {
DeleteTask(taskID)
return fmt.Errorf("failed to add task: %w", err)
}
return nil
}
// createDirectLinksTask 创建直链下载任务
func (f *TaskFactory) createDirectLinksTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params DirectLinksParams
@@ -79,8 +98,9 @@ func (f *TaskFactory) createDirectLinksTask(taskID string, createdAt time.Time,
task := directlinks.NewTask(taskID, f.ctx, params.URLs, stor, req.Path, nil)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err := f.registerAndEnqueueTask(task, tasktype.TaskTypeDirectlinks, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -104,8 +124,9 @@ func (f *TaskFactory) createYTDLPTask(taskID string, createdAt time.Time, req *C
task := ytdlp.NewTask(taskID, f.ctx, params.URLs, params.Flags, stor, req.Path, nil)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err := f.registerAndEnqueueTask(task, tasktype.TaskTypeYtdlp, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -146,8 +167,9 @@ func (f *TaskFactory) createAria2Task(taskID string, createdAt time.Time, req *C
task := aria2dl.NewTask(taskID, f.ctx, gid, params.URLs, aria2Client, stor, req.Path, nil)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeAria2, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -190,8 +212,9 @@ func (f *TaskFactory) createParsedTask(taskID string, createdAt time.Time, req *
task := parsed.NewTask(taskID, f.ctx, stor, req.Path, item, nil)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeParseditem, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -223,15 +246,15 @@ func (f *TaskFactory) createTGFilesTask(taskID string, createdAt time.Time, req
return nil, fmt.Errorf("no files found in provided links")
}
var task core.Executable
if len(files) == 1 {
// 单个文件任务
tfileTask, err := tfile.NewTGFileTask(taskID, f.ctx, files[0], stor, req.Path, nil)
if err != nil {
return nil, fmt.Errorf("failed to create tfile task: %w", err)
}
if err := core.AddTask(f.ctx, tfileTask); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
}
task = tfileTask
} else {
// 批量文件任务
elems := make([]batchtfile.TaskElement, 0, len(files))
@@ -243,10 +266,12 @@ func (f *TaskFactory) createTGFilesTask(taskID string, createdAt time.Time, req
elems = append(elems, *elem)
}
task := batchtfile.NewBatchTGFileTask(taskID, f.ctx, elems, nil, true)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
}
task = batchtfile.NewBatchTGFileTask(taskID, f.ctx, elems, nil, true)
}
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTgfiles, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -281,8 +306,9 @@ func (f *TaskFactory) createTPHPicsTask(taskID string, createdAt time.Time, req
client := telegraph.NewClient()
task := tphtask.NewTask(taskID, f.ctx, phPath, pics, stor, req.Path, client, nil)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTphpics, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
@@ -301,12 +327,12 @@ func (f *TaskFactory) createTransferTask(taskID string, createdAt time.Time, req
}
// 验证源存储和目标存储
sourceStor, ok := storage.Storages[params.SourceStorage]
sourceStor, ok := storage.GetStorage(params.SourceStorage)
if !ok {
return nil, fmt.Errorf("source storage not found: %s", params.SourceStorage)
}
targetStor, ok := storage.Storages[params.TargetStorage]
targetStor, ok := storage.GetStorage(params.TargetStorage)
if !ok {
return nil, fmt.Errorf("target storage not found: %s", params.TargetStorage)
}
@@ -342,8 +368,9 @@ func (f *TaskFactory) createTransferTask(taskID string, createdAt time.Time, req
task := transfer.NewTransferTask(taskID, f.ctx, elems, nil, true)
if err := core.AddTask(f.ctx, task); err != nil {
return nil, fmt.Errorf("failed to add task: %w", err)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTransfer, params.TargetStorage, params.TargetPath, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{

View File

@@ -4,6 +4,7 @@ import (
"encoding/json"
"net/http"
"strings"
"time"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
@@ -117,7 +118,7 @@ func (h *Handlers) CancelTaskHandler(w http.ResponseWriter, r *http.Request) {
return
}
// 取消任务
// Cancel the task; the terminal status is set via the task event stream.
if err := core.CancelTask(r.Context(), taskID); err != nil {
WriteError(w, http.StatusInternalServerError, "cancel_failed", "failed to cancel task: "+err.Error())
return
@@ -134,8 +135,9 @@ func (h *Handlers) ListStoragesHandler(w http.ResponseWriter, r *http.Request) {
return
}
storages := make([]StorageInfo, 0, len(storage.Storages))
for name, stor := range storage.Storages {
all := storage.AllStorages()
storages := make([]StorageInfo, 0, len(all))
for name, stor := range all {
storages = append(storages, StorageInfo{
Name: name,
Type: string(stor.Type()),
@@ -184,27 +186,45 @@ func extractTaskIDFromPath(path string) string {
return parts[3]
}
// convertTaskProgressToResponse 将任务进度转换为响应格式
// convertTaskProgressToResponse renders a task's current state, computing
// percent and speed from the snapshot taken under the task's mutex.
func convertTaskProgressToResponse(task *TaskProgressInfo) TaskInfoResponse {
status, total, downloaded, totalFiles, downloadedFiles, startedAt, errMsg, updatedAt := task.snapshot()
resp := TaskInfoResponse{
TaskID: task.TaskID,
Type: tasktype.TaskType(task.Type),
Status: task.Status,
Status: status,
Title: task.Title,
Storage: task.Storage,
Path: task.Path,
Error: task.Error,
Error: errMsg,
CreatedAt: task.CreatedAt,
UpdatedAt: task.UpdatedAt,
UpdatedAt: updatedAt,
}
// 计算进度
if task.TotalBytes > 0 {
percent := float64(task.DownloadedBytes) * 100 / float64(task.TotalBytes)
var percent float64
var speedMBPS float64
if total > 0 {
percent = float64(downloaded) * 100 / float64(total)
} else if totalFiles > 0 {
percent = float64(downloadedFiles) * 100 / float64(totalFiles)
}
if !startedAt.IsZero() {
elapsed := time.Since(startedAt).Seconds()
if elapsed > 0 {
speedMBPS = float64(downloaded) / elapsed / (1024 * 1024)
}
}
if total > 0 || totalFiles > 0 {
resp.Progress = &TaskProgress{
TotalBytes: task.TotalBytes,
DownloadedBytes: task.DownloadedBytes,
TotalBytes: total,
DownloadedBytes: downloaded,
TotalFiles: totalFiles,
DownloadedFiles: downloadedFiles,
Percent: percent,
SpeedMBPS: speedMBPS,
}
}

View File

@@ -13,6 +13,7 @@ import (
"time"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
)
// setupTestServer creates a test server with handlers
@@ -403,32 +404,38 @@ func TestConcurrentProgressStore(t *testing.T) {
// TestProgressTrackerConcurrentUpdates tests concurrent progress updates
func TestProgressTrackerConcurrentUpdates(t *testing.T) {
tracker := NewProgressTracker("concurrent-progress", "directlinks", "local", "downloads", "Test", "")
tracker.OnStart(10000, 10)
info := RegisterTask("concurrent-progress", "directlinks", "local", "downloads", "Test", "")
info.Emit(taskevent.Event{TaskID: "concurrent-progress", Phase: taskevent.PhaseStart, TotalBytes: 10000})
var wg sync.WaitGroup
numGoroutines := 50
updatesPerGoroutine := 100
// Concurrent progress updates
// Concurrent progress updates via the Sink interface
for i := range numGoroutines {
wg.Add(1)
go func(id int) {
defer wg.Done()
for j := range updatesPerGoroutine {
tracker.OnProgress(int64(id*updatesPerGoroutine+j), j)
info.Emit(taskevent.Event{
TaskID: "concurrent-progress",
Phase: taskevent.PhaseProgress,
DownloadedBytes: int64(id*updatesPerGoroutine + j),
TotalBytes: 10000,
})
}
}(i)
}
wg.Wait()
info := tracker.GetInfo()
if info.Status != TaskStatusRunning {
t.Errorf("expected status Running after concurrent updates, got %s", info.Status)
status, _, downloaded, _, _, _, _, _ := info.snapshot()
if status != TaskStatusRunning {
t.Errorf("expected status Running after concurrent updates, got %s", status)
}
if downloaded <= 0 {
t.Errorf("expected downloaded bytes > 0 after concurrent updates, got %d", downloaded)
}
// Note: Due to race conditions in the simple implementation,
// we can't reliably check exact values without proper synchronization
}
// TestTaskFactoryValidation tests TaskFactory parameter validation
@@ -526,8 +533,7 @@ func TestEdgeCases(t *testing.T) {
{
name: "Progress tracker with empty webhook",
fn: func(t *testing.T) {
tracker := NewProgressTracker("test", "type", "storage", "path", "title", "")
info := tracker.GetInfo()
info := RegisterTask("test-empty-webhook", "type", "storage", "path", "title", "")
if info.Webhook != "" {
t.Error("expected empty webhook")
}

View File

@@ -1,13 +1,18 @@
package api
import (
"context"
"sync"
"sync/atomic"
"time"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
)
// TaskProgressInfo 存储任务的进度信息
// TaskProgressInfo stores the progress of an API-submitted task. All fields are
// guarded by mu. It implements taskevent.Sink so the task layer can update it
// without knowing about the API.
type TaskProgressInfo struct {
mu sync.Mutex
TaskID string
Type string
Status TaskStatus
@@ -21,20 +26,25 @@ type TaskProgressInfo struct {
Error string
CreatedAt time.Time
UpdatedAt time.Time
StartedAt time.Time
Webhook string
webhookNotified bool
}
// progressStore 存储所有 API 任务的进度信息
// progressStore holds all API tasks. Entries are removed a fixed duration after
// they reach a terminal state to bound memory usage.
type progressStore struct {
mu sync.RWMutex
tasks map[string]*TaskProgressInfo
mu sync.RWMutex
tasks map[string]*TaskProgressInfo
retention time.Duration
}
var store = &progressStore{
tasks: make(map[string]*TaskProgressInfo),
tasks: make(map[string]*TaskProgressInfo),
retention: 24 * time.Hour,
}
// RegisterTask 注册一个新的 API 任务
// RegisterTask registers a new API task and returns its progress info.
func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskProgressInfo {
info := &TaskProgressInfo{
TaskID: taskID,
@@ -55,7 +65,7 @@ func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskP
return info
}
// GetTask 获取任务进度信息
// GetTask returns the progress info for a task.
func GetTask(taskID string) (*TaskProgressInfo, bool) {
store.mu.RLock()
defer store.mu.RUnlock()
@@ -63,7 +73,7 @@ func GetTask(taskID string) (*TaskProgressInfo, bool) {
return info, ok
}
// GetAllTasks 获取所有任务
// GetAllTasks returns all tracked tasks.
func GetAllTasks() []*TaskProgressInfo {
store.mu.RLock()
defer store.mu.RUnlock()
@@ -75,76 +85,117 @@ func GetAllTasks() []*TaskProgressInfo {
return tasks
}
// DeleteTask 删除任务记录
// DeleteTask removes a task record.
func DeleteTask(taskID string) {
store.mu.Lock()
defer store.mu.Unlock()
delete(store.tasks, taskID)
}
// UpdateStatus 更新任务状态
func (t *TaskProgressInfo) UpdateStatus(status TaskStatus) {
t.Status = status
t.UpdatedAt = time.Now()
// CleanupExpired removes tasks that reached a terminal state more than the
// store's retention duration ago. It is safe to call periodically.
func CleanupExpired() {
now := time.Now()
store.mu.Lock()
defer store.mu.Unlock()
for id, info := range store.tasks {
info.mu.Lock()
terminal := info.Status == TaskStatusCompleted || info.Status == TaskStatusFailed || info.Status == TaskStatusCancelled
stale := terminal && now.Sub(info.UpdatedAt) > store.retention
info.mu.Unlock()
if stale {
delete(store.tasks, id)
}
}
}
// SetError 设置错误信息
// StartCleanupLoop runs CleanupExpired on a fixed interval until ctx is done.
// It should be started once during API server initialization.
func StartCleanupLoop(ctx interface{ Done() <-chan struct{} }) {
go func() {
ticker := time.NewTicker(10 * time.Minute)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
CleanupExpired()
}
}
}()
}
// UpdateStatus sets the task status.
func (t *TaskProgressInfo) UpdateStatus(status TaskStatus) {
t.mu.Lock()
t.Status = status
t.UpdatedAt = time.Now()
if status == TaskStatusRunning && t.StartedAt.IsZero() {
t.StartedAt = t.UpdatedAt
}
t.mu.Unlock()
}
// SetError marks the task failed with an error message.
func (t *TaskProgressInfo) SetError(err string) {
t.mu.Lock()
t.Error = err
t.Status = TaskStatusFailed
t.UpdatedAt = time.Now()
t.mu.Unlock()
}
// ProgressTracker 用于 API 任务的进度追踪
type ProgressTracker struct {
info *TaskProgressInfo
// snapshot returns a point-in-time copy of the fields needed to render a
// response, so callers never touch the mutex directly.
func (t *TaskProgressInfo) snapshot() (status TaskStatus, total, downloaded int64, totalFiles, downloadedFiles int, startedAt time.Time, err string, updatedAt time.Time) {
t.mu.Lock()
defer t.mu.Unlock()
return t.Status, t.TotalBytes, t.DownloadedBytes, t.TotalFiles, t.DownloadedFiles, t.StartedAt, t.Error, t.UpdatedAt
}
// NewProgressTracker 创建新的进度追踪器
func NewProgressTracker(taskID, taskType, storage, path, title, webhook string) *ProgressTracker {
info := RegisterTask(taskID, taskType, storage, path, title, webhook)
return &ProgressTracker{info: info}
}
// OnStart 任务开始
func (p *ProgressTracker) OnStart(totalBytes int64, totalFiles int) {
p.info.Status = TaskStatusRunning
p.info.TotalBytes = totalBytes
p.info.TotalFiles = totalFiles
p.info.UpdatedAt = time.Now()
}
// OnProgress 进度更新
func (p *ProgressTracker) OnProgress(downloadedBytes int64, downloadedFiles int) {
atomic.StoreInt64(&p.info.DownloadedBytes, downloadedBytes)
p.info.DownloadedFiles = downloadedFiles
p.info.UpdatedAt = time.Now()
}
// OnDone 任务完成
func (p *ProgressTracker) OnDone(err error) {
if err != nil {
p.info.Status = TaskStatusFailed
p.info.Error = err.Error()
} else {
p.info.Status = TaskStatusCompleted
// Emit implements taskevent.Sink. It translates task lifecycle events into
// status/progress updates and fires the webhook on terminal transitions.
func (t *TaskProgressInfo) Emit(e taskevent.Event) {
t.mu.Lock()
switch e.Phase {
case taskevent.PhaseStart:
t.Status = TaskStatusRunning
if t.StartedAt.IsZero() {
t.StartedAt = time.Now()
}
if e.TotalBytes > 0 {
t.TotalBytes = e.TotalBytes
}
case taskevent.PhaseProgress:
t.Status = TaskStatusRunning
if e.TotalBytes > 0 {
t.TotalBytes = e.TotalBytes
}
t.DownloadedBytes = e.DownloadedBytes
if e.TotalFiles > 0 {
t.TotalFiles = e.TotalFiles
}
if e.DownloadedFiles > 0 {
t.DownloadedFiles = e.DownloadedFiles
}
case taskevent.PhaseDone:
if e.Err != nil {
t.Status = TaskStatusFailed
t.Error = e.Err.Error()
} else {
t.Status = TaskStatusCompleted
}
}
p.info.UpdatedAt = time.Now()
}
t.UpdatedAt = time.Now()
notify := t.Webhook != "" && !t.webhookNotified && (t.Status == TaskStatusCompleted || t.Status == TaskStatusFailed)
if notify {
t.webhookNotified = true
}
t.mu.Unlock()
// GetInfo 获取任务信息
func (p *ProgressTracker) GetInfo() *TaskProgressInfo {
return p.info
}
// UpdateProgressBytes 更新下载字节数
func (p *ProgressTracker) UpdateProgressBytes(bytes int64) {
atomic.StoreInt64(&p.info.DownloadedBytes, bytes)
p.info.UpdatedAt = time.Now()
}
// UpdateProgressFiles 更新下载文件数
func (p *ProgressTracker) UpdateProgressFiles(files int) {
p.info.DownloadedFiles = files
p.info.UpdatedAt = time.Now()
if notify {
payload := CreateWebhookPayload(t.TaskID, t.Type, t.Status, t.Storage, t.Path, e.Err)
SendWebhook(context.Background(), payload)
}
}

View File

@@ -3,6 +3,7 @@ package api
import (
"context"
"fmt"
"net"
"net/http"
"time"
@@ -30,16 +31,21 @@ func NewServer(ctx context.Context) *Server {
mux.HandleFunc("/health", handlers.HealthCheckHandler)
// API v1 路由
mux.HandleFunc("/api/v1/tasks", handlers.CreateTaskHandler)
mux.HandleFunc("/api/v1/tasks", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
handlers.ListTasksHandler(w, r)
case http.MethodPost:
handlers.CreateTaskHandler(w, r)
default:
MethodNotAllowedHandler(w, r)
}
})
mux.HandleFunc("/api/v1/tasks/", func(w http.ResponseWriter, r *http.Request) {
// 根据方法和路径分发
switch r.Method {
case http.MethodGet:
if r.URL.Path == "/api/v1/tasks" {
handlers.ListTasksHandler(w, r)
} else {
handlers.GetTaskHandler(w, r)
}
handlers.GetTaskHandler(w, r)
case http.MethodDelete:
handlers.CancelTaskHandler(w, r)
default:
@@ -52,22 +58,19 @@ func NewServer(ctx context.Context) *Server {
// 404 处理
mux.HandleFunc("/", NotFoundHandler)
// 应用中间件
// Apply middleware chain.
var handler http.Handler = mux
// 添加认证中间件
// Apply auth middleware when a token is configured.
token := cfg.Token
if token == "" {
log.FromContext(ctx).Warn("API server is enabled but no token is set, this is insecure!")
}
if token != "" {
handler = AuthMiddleware()(handler)
}
// 添加日志中间件
// Add logging middleware.
handler = loggingMiddleware(handler)
// 添加恢复中间件
// Add recovery middleware.
handler = recoveryMiddleware(handler)
return &Server{
@@ -88,9 +91,15 @@ func (s *Server) Start(ctx context.Context) error {
logger.Infof("Starting API server on %s", s.httpServer.Addr)
// Bind synchronously so listen failures are returned to the caller.
ln, err := net.Listen("tcp", s.httpServer.Addr)
if err != nil {
return fmt.Errorf("failed to listen on %s: %w", s.httpServer.Addr, err)
}
// 在 goroutine 中启动服务器
go func() {
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
if err := s.httpServer.Serve(ln); err != nil && err != http.ErrServerClosed {
logger.Errorf("API server error: %v", err)
}
}()
@@ -146,7 +155,8 @@ func (rw *responseWriter) WriteHeader(code int) {
rw.ResponseWriter.WriteHeader(code)
}
// Start 初始化并启动 API 服务器
// Start initializes and starts the API server. It refuses to start without a
// token, since an open download proxy is a security risk.
func Start(ctx context.Context) error {
cfg := config.C().API
@@ -155,9 +165,13 @@ func Start(ctx context.Context) error {
}
if cfg.Token == "" {
log.FromContext(ctx).Warn("API server is enabled but no token is set, this is insecure!")
return fmt.Errorf("API server is enabled but no token is set; refusing to start insecurely")
}
server := NewServer(ctx)
return server.Start(ctx)
if err := server.Start(ctx); err != nil {
return err
}
StartCleanupLoop(ctx)
return nil
}

View File

@@ -60,6 +60,7 @@ func resolveChatID(_ context.Context, idOrUsername string) (int64, error) {
}
// ParseMessageLink 解析 Telegram 消息链接
// 支持的域名: t.me, telegram.me
// 支持格式:
// - https://t.me/username/123
// - https://t.me/c/123456789/123
@@ -268,5 +269,15 @@ func ExtractFilesFromLinks(ctx context.Context, links []string) ([]tfile.TGFileM
// isValidMessageLink 检查是否是有效的 Telegram 消息链接
func isValidMessageLink(link string) bool {
return strings.HasPrefix(link, "https://t.me/") || strings.HasPrefix(link, "http://t.me/")
for _, prefix := range []string{
"https://t.me/",
"http://t.me/",
"https://telegram.me/",
"http://telegram.me/",
} {
if strings.HasPrefix(link, prefix) {
return true
}
}
return false
}

View File

@@ -40,6 +40,8 @@ type CreateTaskResponse struct {
type TaskProgress struct {
TotalBytes int64 `json:"total_bytes,omitempty"`
DownloadedBytes int64 `json:"downloaded_bytes,omitempty"`
TotalFiles int `json:"total_files,omitempty"`
DownloadedFiles int `json:"downloaded_files,omitempty"`
Percent float64 `json:"percent,omitempty"`
SpeedMBPS float64 `json:"speed_mbps,omitempty"`
}

View File

@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"time"
@@ -30,9 +29,17 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
webhookURL := info.Webhook
// 异步发送 webhook
// Async send with retries.
go func() {
logger := log.FromContext(ctx).With("task_id", payload.TaskID)
var logger *log.Logger
if ctx != nil {
logger = log.FromContext(ctx).With("task_id", payload.TaskID)
} else {
logger = log.Default().With("task_id", payload.TaskID)
}
if ctx == nil {
ctx = context.Background()
}
payloadBytes, err := json.Marshal(payload)
if err != nil {
@@ -40,10 +47,15 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
return
}
// 重试 3 次
for i := range 3 {
req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, webhookURL, bytes.NewBuffer(payloadBytes))
// 重试 3 次, 指数退避 (100ms/400ms/1.6s)
const maxAttempts = 3
const requestTimeout = 30 * time.Second
backoff := 100 * time.Millisecond
for i := range maxAttempts {
reqCtx, cancel := context.WithTimeout(ctx, requestTimeout)
req, err := http.NewRequestWithContext(reqCtx, http.MethodPost, webhookURL, bytes.NewBuffer(payloadBytes))
if err != nil {
cancel()
logger.Errorf("Failed to create webhook request: %v", err)
return
}
@@ -52,9 +64,13 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
req.Header.Set("User-Agent", "SaveAny-Bot/1.0")
resp, err := webhookClient.Do(req)
cancel()
if err != nil {
logger.Warnf("Webhook request failed (attempt %d/3): %v", i+1, err)
time.Sleep(time.Second * time.Duration(i+1))
logger.Warnf("Webhook request failed (attempt %d/%d): %v", i+1, maxAttempts, err)
if i < maxAttempts-1 {
time.Sleep(backoff)
}
backoff *= 4
continue
}
resp.Body.Close()
@@ -64,15 +80,18 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
return
}
logger.Warnf("Webhook returned non-2xx status (attempt %d/3): %d", i+1, resp.StatusCode)
time.Sleep(time.Second * time.Duration(i+1))
logger.Warnf("Webhook returned non-2xx status (attempt %d/%d): %d", i+1, maxAttempts, resp.StatusCode)
if i < maxAttempts-1 {
time.Sleep(backoff)
}
backoff *= 4
}
logger.Errorf("Failed to send webhook after 3 attempts")
logger.Errorf("Failed to send webhook after %d attempts", maxAttempts)
}()
}
// CreateWebhookPayload 创建 Webhook 负载
// CreateWebhookPayload creates a Webhook payload.
func CreateWebhookPayload(taskID string, taskType string, status TaskStatus, storage, path string, err error) *WebhookPayload {
payload := &WebhookPayload{
TaskID: taskID,
@@ -93,38 +112,3 @@ func CreateWebhookPayload(taskID string, taskType string, status TaskStatus, sto
return payload
}
// WrapTaskWithWebhook 包装任务执行,添加 webhook 回调
func WrapTaskWithWebhook(ctx context.Context, taskID string, fn func() error) error {
info, ok := GetTask(taskID)
if !ok {
return fmt.Errorf("task not found: %s", taskID)
}
err := fn()
// 确定任务状态
status := TaskStatusCompleted
if err != nil {
if err == context.Canceled {
status = TaskStatusCancelled
} else {
status = TaskStatusFailed
}
}
// 更新任务状态
if err != nil {
info.SetError(err.Error())
} else {
info.UpdateStatus(TaskStatusCompleted)
}
// 发送 webhook
if info.Webhook != "" {
payload := CreateWebhookPayload(taskID, info.Type, status, info.Storage, info.Path, err)
SendWebhook(ctx, payload)
}
return err
}

View File

@@ -70,9 +70,6 @@ func Init(ctx context.Context) <-chan struct{} {
}{nil, err}
return
}
client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
Scope: &tg.BotCommandScopeDefault{},
})
commands := make([]tg.BotCommand, 0, len(handlers.CommandHandlers))
for _, info := range handlers.CommandHandlers {
commands = append(commands, tg.BotCommand{Command: info.Cmd, Description: i18n.T(info.Desc)})

View File

@@ -23,7 +23,11 @@ import (
)
func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
dataid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
if len(dataParts) < 2 {
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
}
dataid := dataParts[1]
data, err := shortcut.GetCallbackDataWithAnswer[tcbdata.Add](ctx, update, dataid)
if err != nil {
return err
@@ -73,14 +77,16 @@ func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
return dispatcher.EndGroups
}
dirPath = dir.Path
} else if data.SelectedDirPath != "" {
dirPath = data.SelectedDirPath
}
switch data.TaskType {
case tasktype.TaskTypeTgfiles:
if data.AsBatch {
return shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files, msgID)
return shortcut.CreateAndAddBatchTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files, msgID, data.ConflictStrategy)
}
return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files[0], msgID)
return shortcut.CreateAndAddTGFileTaskWithEdit(ctx, userID, selectedStorage, dirPath, data.Files[0], msgID, data.ConflictStrategy)
case tasktype.TaskTypeTphpics:
return shortcut.CreateAndAddtelegraphWithEdit(ctx, userID, data.TphPageNode, data.TphDirPath, data.TphPics, selectedStorage, msgID)
case tasktype.TaskTypeParseditem:

View File

@@ -1,6 +1,7 @@
package handlers
import (
"fmt"
"strings"
"github.com/celestix/gotgproto/dispatcher"
@@ -14,7 +15,11 @@ import (
)
func handleCancelCallback(ctx *ext.Context, update *ext.Update) error {
taskid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
if len(dataParts) < 2 {
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
}
taskid := dataParts[1]
if err := core.CancelTask(ctx, taskid); err != nil {
log.FromContext(ctx).Errorf("Failed to cancel task %s: %v", taskid, err)
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(update.CallbackQuery.GetQueryID(), i18n.T(i18nk.BotMsgCancelErrorCancelFailed, map[string]any{

View File

@@ -8,6 +8,7 @@ import (
"github.com/celestix/gotgproto/dispatcher"
"github.com/celestix/gotgproto/ext"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/conflictutil"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/config"
@@ -26,6 +27,10 @@ func handleConfigCmd(ctx *ext.Context, update *ext.Update) error {
Text: i18n.T(i18nk.BotMsgConfigButtonFilenameStrategy),
Data: fmt.Appendf(nil, "%s %s", tcbdata.TypeConfig, "fnamest"),
},
&tg.KeyboardButtonCallback{
Text: i18n.T(i18nk.BotMsgConfigButtonConflictStrategy),
Data: fmt.Appendf(nil, "%s %s", tcbdata.TypeConfig, "conflictst"),
},
},
},
},
@@ -51,6 +56,8 @@ func handleConfigCallback(ctx *ext.Context, update *ext.Update) error {
switch args[1] {
case "fnamest":
return handleConfigFnameSTCallback(ctx, update)
case "conflictst":
return handleConfigConflictSTCallback(ctx, update)
default:
return invaildDataAnswer()
}
@@ -110,6 +117,55 @@ func handleConfigFnameSTCallback(ctx *ext.Context, update *ext.Update) error {
return dispatcher.EndGroups
}
func handleConfigConflictSTCallback(ctx *ext.Context, update *ext.Update) error {
userID := update.CallbackQuery.GetUserID()
user, err := database.GetUserByChatID(ctx, userID)
if err != nil {
return err
}
args := strings.Fields(string(update.CallbackQuery.Data))
if len(args) == 3 {
selected := args[2]
if !tcbdata.IsConflictStrategy(selected) {
return fmt.Errorf("invalid conflict strategy: %s", selected)
}
user.ConflictStrategy = selected
if err := database.UpdateUser(ctx, user); err != nil {
return err
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: update.CallbackQuery.GetMsgID(),
Message: i18n.T(i18nk.BotMsgConfigInfoConflictStrategySet, map[string]any{
"Strategy": conflictutil.Display(selected),
}),
})
return dispatcher.EndGroups
}
opts := tcbdata.ConflictStrategyValues()
rows := make([]tg.KeyboardButtonRow, 0, len(opts))
for _, opt := range opts {
rows = append(rows, tg.KeyboardButtonRow{
Buttons: []tg.KeyboardButtonClass{
&tg.KeyboardButtonCallback{
Text: conflictutil.Display(opt),
Data: fmt.Appendf(nil, "%s %s %s", tcbdata.TypeConfig, "conflictst", opt),
},
},
})
}
markup := &tg.ReplyInlineMarkup{Rows: rows}
currentSt := conflictutil.EffectiveStrategy(user)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: update.CallbackQuery.GetMsgID(),
Message: i18n.T(i18nk.BotMsgConfigPromptSelectConflictStrategy, map[string]any{
"Strategy": conflictutil.Display(currentSt),
}),
ReplyMarkup: markup,
})
return dispatcher.EndGroups
}
func handleConfigFnameTmpl(ctx *ext.Context, update *ext.Update) error {
userID := update.GetUserChat().GetID()
user, err := database.GetUserByChatID(ctx, userID)

View File

@@ -10,6 +10,7 @@ import (
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/storage"
)
@@ -42,7 +43,9 @@ func handleDirCmd(ctx *ext.Context, update *ext.Update) error {
return dispatcher.EndGroups
}
if _, err := storage.GetStorageByUserIDAndName(ctx, user.ChatID, args[2]); err != nil {
ctx.Reply(update, ext.ReplyTextString(err.Error()), nil)
logger.Errorf("Failed to get storage %q: %s", args[2], err)
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorGetStorageFailed,
tgutil.EscapeHTMLTemplateData(map[string]any{"Error": err.Error()}))), nil)
return dispatcher.EndGroups
}

View File

@@ -13,16 +13,24 @@ import (
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
)
// mediaGroupKey uniquely identifies a media group by chat, sender, and group
// ID so files from different users in the same chat can never be mixed.
type mediaGroupKey struct {
chatID int64
userID int64
groupID int64
}
type MediaGroupHandler struct {
groups map[int64][]tfile.TGFileMessage
timers map[int64]*time.Timer
groups map[mediaGroupKey][]tfile.TGFileMessage
timers map[mediaGroupKey]*time.Timer
mu sync.Mutex
timeout time.Duration
setupOnce sync.Once
@@ -39,8 +47,8 @@ func (m *MediaGroupHandler) SetupTimeout(timeoutSec int) {
var (
mediaGroupHandler = &MediaGroupHandler{
groups: make(map[int64][]tfile.TGFileMessage),
timers: make(map[int64]*time.Timer),
groups: make(map[mediaGroupKey][]tfile.TGFileMessage),
timers: make(map[mediaGroupKey]*time.Timer),
mu: sync.Mutex{},
}
)
@@ -66,32 +74,37 @@ func handleGroupMediaMessage(ctx *ext.Context, update *ext.Update, message *tg.M
}
mediaGroupHandler.mu.Lock()
defer mediaGroupHandler.mu.Unlock()
if mediaGroupHandler.groups[groupID] == nil {
mediaGroupHandler.groups[groupID] = make([]tfile.TGFileMessage, 0)
key := mediaGroupKey{
chatID: update.EffectiveChat().GetID(),
userID: userId,
groupID: groupID,
}
mediaGroupHandler.groups[groupID] = append(mediaGroupHandler.groups[groupID], file)
if mediaGroupHandler.groups[key] == nil {
mediaGroupHandler.groups[key] = make([]tfile.TGFileMessage, 0)
}
mediaGroupHandler.groups[key] = append(mediaGroupHandler.groups[key], file)
if timer, exists := mediaGroupHandler.timers[groupID]; exists {
if timer, exists := mediaGroupHandler.timers[key]; exists {
timer.Stop()
}
mediaGroupHandler.timers[groupID] = time.AfterFunc(mediaGroupHandler.timeout, func() {
processMediaGroup(ctx, update, groupID)
mediaGroupHandler.timers[key] = time.AfterFunc(mediaGroupHandler.timeout, func() {
processMediaGroup(ctx, update, key)
})
return dispatcher.EndGroups
}
func processMediaGroup(ctx *ext.Context, update *ext.Update, groupID int64) {
func processMediaGroup(ctx *ext.Context, update *ext.Update, key mediaGroupKey) {
logger := log.FromContext(ctx)
mediaGroupHandler.mu.Lock()
items := mediaGroupHandler.groups[groupID]
delete(mediaGroupHandler.groups, groupID)
delete(mediaGroupHandler.timers, groupID)
items := mediaGroupHandler.groups[key]
delete(mediaGroupHandler.groups, key)
delete(mediaGroupHandler.timers, key)
mediaGroupHandler.mu.Unlock()
if len(items) == 0 {
logger.Warn("No media items to process for group", "groupID", groupID)
logger.Warn("No media items to process for group", "groupID", key.groupID)
return
}
logger.Debugf("Processing media group %d with %d items", groupID, len(items))
logger.Debugf("Processing media group %d with %d items", key.groupID, len(items))
userId := update.GetUserChat().GetID()
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgMediaGroupInfoSavingFiles, nil)), nil)

View File

@@ -5,6 +5,7 @@ import (
"github.com/celestix/gotgproto/ext"
"github.com/duke-git/lancet/v2/slice"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/dirutil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/config"
@@ -12,16 +13,40 @@ import (
"github.com/krau/SaveAny-Bot/storage"
)
// responsibleUserID returns the sender's ID. Callback queries carry it
// natively; message updates resolve it through the entity map.
func responsibleUserID(u *ext.Update) int64 {
if u.CallbackQuery != nil {
return u.CallbackQuery.GetUserID()
}
return u.GetUserChat().GetID()
}
func checkPermission(ctx *ext.Context, update *ext.Update) error {
userID := update.GetUserChat().GetID()
userID := responsibleUserID(update)
if !slice.Contain(config.C().GetUsersID(), userID) {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorNoPermission, nil)), nil)
if cbq := update.CallbackQuery; cbq != nil {
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(cbq.GetQueryID(), i18n.T(i18nk.BotMsgCommonErrorNoPermission, nil)))
} else {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorNoPermission, nil)), nil)
}
return dispatcher.EndGroups
}
return dispatcher.ContinueGroups
}
// withPermission wraps a callback handler with the same whitelist check used
// for message handlers (checkPermission).
func withPermission(handler func(*ext.Context, *ext.Update) error) func(*ext.Context, *ext.Update) error {
return func(ctx *ext.Context, update *ext.Update) error {
if err := checkPermission(ctx, update); err != nil {
return err
}
return handler(ctx, update)
}
}
func handleSilentMode(next func(*ext.Context, *ext.Update) error, handler func(*ext.Context, *ext.Update) error) func(*ext.Context, *ext.Update) error {
return func(ctx *ext.Context, update *ext.Update) error {
userID := update.GetUserChat().GetID()

View File

@@ -0,0 +1,51 @@
package handlers
import (
"testing"
"github.com/celestix/gotgproto/ext"
"github.com/celestix/gotgproto/types"
"github.com/gotd/td/tg"
)
func TestResponsibleUserID(t *testing.T) {
tests := []struct {
name string
update *ext.Update
want int64
}{
{
name: "callback query uses native user id",
update: &ext.Update{CallbackQuery: &tg.UpdateBotCallbackQuery{UserID: 42}},
want: 42,
},
{
name: "message resolves through entity map",
update: &ext.Update{
EffectiveMessage: &types.Message{Message: &tg.Message{PeerID: &tg.PeerUser{UserID: 7}}},
Entities: &tg.Entities{Users: map[int64]*tg.User{7: {ID: 7}}},
},
want: 7,
},
{
name: "callback query ignores entity map",
update: &ext.Update{
CallbackQuery: &tg.UpdateBotCallbackQuery{UserID: 9},
Entities: &tg.Entities{Users: map[int64]*tg.User{8: {ID: 8}}},
},
want: 9,
},
{
name: "unresolvable update yields zero",
update: &ext.Update{},
want: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := responsibleUserID(tt.update); got != tt.want {
t.Fatalf("responsibleUserID() = %d, want %d", got, tt.want)
}
})
}
}

View File

@@ -56,11 +56,11 @@ func Register(disp dispatcher.Dispatcher) {
for _, info := range CommandHandlers {
disp.AddHandler(handlers.NewCommand(info.Cmd, info.handler))
}
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("update"), handleUpdateCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), handleAddCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), handleSetDefaultCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeCancel), handleCancelCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeConfig), handleConfigCallback))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("update"), withPermission(handleUpdateCallback)))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), withPermission(handleAddCallback)))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), withPermission(handleSetDefaultCallback)))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeCancel), withPermission(handleCancelCallback)))
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeConfig), withPermission(handleConfigCallback)))
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TgMessageLinkRegexString)), handleSilentMode(handleMessageLink, handleSilentSaveLink)))
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TelegraphUrlRegexString)), handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
disp.AddHandler(handlers.NewMessage(filters.Message.Media, handleSilentMode(handleMediaMessage, handleSilentSaveMedia)))

View File

@@ -13,6 +13,7 @@ import (
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/strutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/rule"
)
@@ -84,6 +85,46 @@ func handleRuleCmd(ctx *ext.Context, update *ext.Update) error {
return dispatcher.EndGroups
}
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleInfoCreateRuleSuccess, nil)), nil)
case "preset":
// /rule preset <storage> [base_path]
if len(args) < 3 {
ctx.Reply(update, ext.ReplyTextStyledTextArray(msgelem.BuildRuleHelpStyling(user.ApplyRule, user.Rules)), nil)
return dispatcher.EndGroups
}
storageName := args[2]
if !config.C().HasStorage(user.ChatID, storageName) {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleErrorStorageNotFound, map[string]any{
"Storage": storageName,
})), nil)
return dispatcher.EndGroups
}
basePath := ""
if len(args) >= 4 {
basePath = args[3]
}
presets := rule.PresetCategories(basePath)
imported := 0
for _, p := range presets {
rd := &database.Rule{
Type: rule.FileNameRegex.String(),
Data: p.Regex,
StorageName: storageName,
DirPath: p.Dir,
UserID: user.ID,
}
if err := database.CreateRule(ctx, rd); err != nil {
logger.Errorf("failed to create preset rule %s: %s", p.Name, err)
continue
}
imported++
}
if imported == 0 {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleErrorCreateRuleFailed, nil)), nil)
return dispatcher.EndGroups
}
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleInfoPresetImported, map[string]any{
"Count": imported,
})), nil)
case "del":
// /rule del <id>
if len(args) < 3 {

View File

@@ -1,6 +1,7 @@
package handlers
import (
"fmt"
"strings"
"github.com/celestix/gotgproto/dispatcher"
@@ -43,7 +44,11 @@ func handleSilentCmd(ctx *ext.Context, update *ext.Update) error {
}
func handleSetDefaultCallback(ctx *ext.Context, update *ext.Update) error {
dataid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
if len(dataParts) < 2 {
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
}
dataid := dataParts[1]
data, ok := cache.Get[tcbdata.SetDefaultStorage](dataid)
failedAnswer := func(message string) error {

View File

@@ -89,7 +89,11 @@ func showQueuedTasks(ctx *ext.Context, update *ext.Update) {
styling.Bold(i18n.T(i18nk.BotMsgTasksQueuedTitle)),
styling.Plain(i18n.T(i18nk.BotMsgTasksTotalPrefix, map[string]any{"Count": len(tasks)})),
)
for _, t := range tasks {
const maxShown = 10
for i, t := range tasks {
if i >= maxShown {
break
}
created := t.Created.In(time.Local).Format("2006-01-02 15:04:05")
status := i18n.T(i18nk.BotMsgTasksStatusQueued)
if t.Cancelled {
@@ -105,10 +109,9 @@ func showQueuedTasks(ctx *ext.Context, update *ext.Update) {
styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksFieldStatus)),
styling.Code(status),
)
if len(tasks) > 10 {
opts = append(opts, styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksTruncatedNote, map[string]any{"Count": len(tasks)})))
break
}
}
if len(tasks) > maxShown {
opts = append(opts, styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksTruncatedNote, map[string]any{"Count": len(tasks)})))
}
ctx.Reply(update, ext.ReplyTextStyledTextArray(opts), nil)
}

View File

@@ -10,6 +10,7 @@ import (
"github.com/celestix/gotgproto/ext"
"github.com/gotd/td/telegram/message/html"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/config"
@@ -100,7 +101,10 @@ func handleUpdateCmd(ctx *ext.Context, u *ext.Update) error {
func handleUpdateCallback(ctx *ext.Context, u *ext.Update) error {
currentV, err := semver.Parse(config.Version)
if err != nil {
return err
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(u.CallbackQuery.GetQueryID(), i18n.T(i18nk.BotMsgUpdateErrorVersionVarInvalid, map[string]any{
"Error": err.Error(),
})))
return dispatcher.EndGroups
}
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
ID: u.CallbackQuery.GetMsgID(),

View File

@@ -0,0 +1,55 @@
package conflictutil
import (
"fmt"
"strings"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
)
const maxConflictLines = 10
func EffectiveStrategy(user *database.User) string {
if user != nil && tcbdata.IsConflictStrategy(user.ConflictStrategy) {
return user.ConflictStrategy
}
return tcbdata.ConflictStrategyRename
}
func ResolveStrategy(user *database.User, override string) string {
if tcbdata.IsConflictStrategy(override) {
return override
}
return EffectiveStrategy(user)
}
func Display(strategy string) string {
switch strategy {
case tcbdata.ConflictStrategyRename:
return i18n.T(i18nk.BotMsgConfigConflictStrategyRename, nil)
case tcbdata.ConflictStrategyAsk:
return i18n.T(i18nk.BotMsgConfigConflictStrategyAsk, nil)
case tcbdata.ConflictStrategyOverwrite:
return i18n.T(i18nk.BotMsgConfigConflictStrategyOverwrite, nil)
case tcbdata.ConflictStrategySkip:
return i18n.T(i18nk.BotMsgConfigConflictStrategySkip, nil)
default:
return strategy
}
}
func FormatPaths(conflicts []string) string {
if len(conflicts) <= maxConflictLines {
return strings.Join(conflicts, "\n")
}
return strings.Join(conflicts[:maxConflictLines], "\n") + "\n" + i18n.T(i18nk.BotMsgCommonPromptConflictMoreFiles, map[string]any{
"Count": len(conflicts) - maxConflictLines,
})
}
func FormatPath(storageName, storagePath string) string {
return fmt.Sprintf("[%s]:%s", storageName, storagePath)
}

View File

@@ -24,6 +24,8 @@ func BuildRuleHelpStyling(enabled bool, rules []database.Rule) []styling.StyledT
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpSwitchSuffix, nil)),
styling.Code("add"),
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpAddSuffix, nil)),
styling.Code("preset"),
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpPresetSuffix, nil)),
styling.Code("del"),
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpDelSuffix, nil)),
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpExistingRulesPrefix, nil)),

View File

@@ -38,6 +38,8 @@ func BuildAddSelectStorageKeyboard(stors []storage.Storage, adddata tcbdata.Add)
data := tcbdata.Add{
TaskType: taskType,
SelectedStorName: storage.Name(),
SelectedDirPath: adddata.SelectedDirPath,
ConflictStrategy: adddata.ConflictStrategy,
Files: adddata.Files,
AsBatch: len(adddata.Files) > 1,
@@ -109,6 +111,38 @@ func BuildAddOneSelectStorageMessage(ctx context.Context, stors []storage.Storag
}, nil
}
func BuildConflictStrategyMarkup(adddata tcbdata.Add) (*tg.ReplyInlineMarkup, error) {
type option struct {
text string
strategy string
}
options := []option{
{text: i18n.T(i18nk.BotMsgCommonButtonConflictRename, nil), strategy: tcbdata.ConflictStrategyRename},
{text: i18n.T(i18nk.BotMsgCommonButtonConflictOverwrite, nil), strategy: tcbdata.ConflictStrategyOverwrite},
{text: i18n.T(i18nk.BotMsgCommonButtonConflictSkip, nil), strategy: tcbdata.ConflictStrategySkip},
}
buttons := make([]tg.KeyboardButtonClass, 0, len(options))
for _, opt := range options {
data := adddata
data.ConflictStrategy = opt.strategy
dataid := xid.New().String()
if err := cache.Set(dataid, data); err != nil {
return nil, err
}
buttons = append(buttons, &tg.KeyboardButtonCallback{
Text: opt.text,
Data: fmt.Appendf(nil, "%s %s", tcbdata.TypeAdd, dataid),
})
}
rows := make([]tg.KeyboardButtonRow, 0, len(buttons))
for _, button := range buttons {
rows = append(rows, tg.KeyboardButtonRow{
Buttons: []tg.KeyboardButtonClass{button},
})
}
return &tg.ReplyInlineMarkup{Rows: rows}, nil
}
// Builds the inline keyboard for setting default storage
func BuildSetDefaultStorageMarkup(
ctx context.Context,

View File

@@ -3,8 +3,8 @@ package re
import "regexp"
var (
TgMessageLinkRegexString = `https?://t\.me/(?:c/\d+|[A-Za-z0-9_]+)/\d+(?:/\d+)?(?:\?[^\s#]*[A-Za-z0-9_])?\b`
TgMessageLinkRegexString = `https?://(?:t|telegram)\.me/(?:c/\d+|[A-Za-z0-9_]+)/\d+(?:/\d+)?(?:\?[^\s#]*[A-Za-z0-9_])?\b`
TgMessageLinkRegexp = regexp.MustCompile(TgMessageLinkRegexString)
TelegraphUrlRegexString = `https://telegra.ph/.*`
TelegraphUrlRegexString = `https://telegra\.ph/[^\s]+`
TelegraphUrlRegexp = regexp.MustCompile(TelegraphUrlRegexString)
)

View File

@@ -3,6 +3,7 @@ package shortcut
import (
"encoding/json"
"fmt"
"net/url"
"strings"
@@ -179,21 +180,19 @@ type TelegraphResult struct {
// return replied message, image urls, telegraph path(unescaped), error
func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*types.Message, *TelegraphResult, error) {
logger := log.FromContext(ctx)
tphurl := re.TelegraphUrlRegexp.FindString(tgutil.ExtractMessageEntityUrlsText(update.EffectiveMessage.Message))
tphurl := findTelegraphURL(update.EffectiveMessage.Message)
if tphurl == "" {
logger.Warnf("No telegraph url found but called handleTelegraph")
return nil, nil, dispatcher.ContinueGroups
}
pagepath := strings.Split(tphurl, "/")[len(strings.Split(tphurl, "/"))-1]
tphdir, err := url.PathUnescape(pagepath)
pagepath, err := parseTelegraphPagePath(tphurl)
if err != nil {
logger.Errorf("Failed to unescape telegraph path: %s", err)
logger.Errorf("Failed to parse telegraph path: %s", err)
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorParseTelegraphPathFailed, map[string]any{
"Error": err.Error(),
})), nil)
return nil, nil, dispatcher.EndGroups
}
tphdir = strings.TrimSpace(tphdir)
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonInfoFetchingTelegraphPage, nil)), nil)
if err != nil {
logger.Errorf("Failed to reply to update: %s", err)
@@ -244,7 +243,57 @@ func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*type
}
return msg, &TelegraphResult{
Pics: imgs,
TphDir: tphdir,
TphDir: pagepath,
Page: page,
}, nil
}
func findTelegraphURL(msg *tg.Message) string {
if msg == nil {
return ""
}
var firstMatch string
findValid := func(text string) string {
for _, tphurl := range re.TelegraphUrlRegexp.FindAllString(text, -1) {
if firstMatch == "" {
firstMatch = tphurl
}
if _, err := parseTelegraphPagePath(tphurl); err == nil {
return tphurl
}
}
return ""
}
for _, entityURL := range tgutil.ExtractMessageEntityUrls(msg) {
if tphurl := findValid(entityURL); tphurl != "" {
return tphurl
}
}
if tphurl := findValid(msg.GetMessage()); tphurl != "" {
return tphurl
}
return firstMatch
}
func parseTelegraphPagePath(pageURL string) (string, error) {
u, err := url.Parse(pageURL)
if err != nil {
return "", fmt.Errorf("invalid telegraph URL: %w", err)
}
if u.Scheme != "https" || !strings.EqualFold(u.Hostname(), "telegra.ph") {
return "", fmt.Errorf("invalid telegraph URL host: %s", u.Host)
}
pagepath := strings.Trim(u.EscapedPath(), "/")
if pagepath == "" || strings.Contains(pagepath, "/") {
return "", fmt.Errorf("invalid telegraph URL path: %s", u.Path)
}
pagepath, err = url.PathUnescape(pagepath)
if err != nil {
return "", fmt.Errorf("failed to unescape telegraph path: %w", err)
}
pagepath = strings.TrimSpace(pagepath)
if pagepath == "" || strings.Contains(pagepath, "/") {
return "", fmt.Errorf("invalid telegraph URL path: %s", u.Path)
}
return pagepath, nil
}

View File

@@ -0,0 +1,163 @@
package shortcut
import (
"testing"
"github.com/gotd/td/tg"
)
func TestFindTelegraphURL(t *testing.T) {
tests := []struct {
name string
msg *tg.Message
want string
}{
{
name: "single URL entity",
msg: &tg.Message{
Message: "https://telegra.ph/Example-01-02",
Entities: []tg.MessageEntityClass{
&tg.MessageEntityURL{Offset: 0, Length: 32},
},
},
want: "https://telegra.ph/Example-01-02",
},
{
name: "Telegraph URL before another URL",
msg: &tg.Message{
Message: "https://telegra.ph/Example-01-02 https://example.com/",
Entities: []tg.MessageEntityClass{
&tg.MessageEntityURL{Offset: 0, Length: 32},
&tg.MessageEntityURL{Offset: 33, Length: 20},
},
},
want: "https://telegra.ph/Example-01-02",
},
{
name: "hidden Telegraph URL",
msg: &tg.Message{
Message: "article",
Entities: []tg.MessageEntityClass{
&tg.MessageEntityTextURL{
Offset: 0,
Length: 7,
URL: "https://telegra.ph/Hidden-01-02",
},
},
},
want: "https://telegra.ph/Hidden-01-02",
},
{
name: "URL entity after non-BMP character",
msg: &tg.Message{
Message: "😀 https://telegra.ph/Emoji-01-02",
Entities: []tg.MessageEntityClass{
&tg.MessageEntityURL{Offset: 3, Length: 30},
},
},
want: "https://telegra.ph/Emoji-01-02",
},
{
name: "valid Telegraph URL after invalid candidate",
msg: &tg.Message{
Message: "https://telegra.ph/nested/Bad https://telegra.ph/Valid-01-02",
Entities: []tg.MessageEntityClass{
&tg.MessageEntityURL{Offset: 0, Length: 29},
&tg.MessageEntityURL{Offset: 30, Length: 30},
},
},
want: "https://telegra.ph/Valid-01-02",
},
{
name: "plain message fallback",
msg: &tg.Message{Message: "read https://telegra.ph/Plain-01-02 now"},
want: "https://telegra.ph/Plain-01-02",
},
{
name: "nil message",
msg: nil,
want: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := findTelegraphURL(tt.msg); got != tt.want {
t.Fatalf("findTelegraphURL() = %q, want %q", got, tt.want)
}
})
}
}
func TestParseTelegraphPagePath(t *testing.T) {
tests := []struct {
name string
pageURL string
want string
wantErr bool
}{
{
name: "plain path",
pageURL: "https://telegra.ph/Example-01-02",
want: "Example-01-02",
},
{
name: "escaped path with query and fragment",
pageURL: "https://telegra.ph/%E6%B5%8B%E8%AF%95-01-02?source=telegram#top",
want: "测试-01-02",
},
{
name: "trailing slash",
pageURL: "https://telegra.ph/Example-01-02/",
want: "Example-01-02",
},
{
name: "root URL",
pageURL: "https://telegra.ph/",
wantErr: true,
},
{
name: "wrong host",
pageURL: "https://example.com/Example-01-02",
wantErr: true,
},
{
name: "wrong scheme",
pageURL: "http://telegra.ph/Example-01-02",
wantErr: true,
},
{
name: "invalid percent escape",
pageURL: "https://telegra.ph/Invalid-%zz",
wantErr: true,
},
{
name: "nested path",
pageURL: "https://telegra.ph/nested/Example-01-02",
wantErr: true,
},
{
name: "encoded slash",
pageURL: "https://telegra.ph/nested%2FExample-01-02",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := parseTelegraphPagePath(tt.pageURL)
if tt.wantErr {
if err == nil {
t.Fatalf("parseTelegraphPagePath(%q) returned no error", tt.pageURL)
}
return
}
if err != nil {
t.Fatalf("parseTelegraphPagePath(%q) failed: %v", tt.pageURL, err)
}
if got != tt.want {
t.Fatalf("parseTelegraphPagePath(%q) = %q, want %q", tt.pageURL, got, tt.want)
}
})
}
}

View File

@@ -8,6 +8,7 @@ import (
"github.com/celestix/gotgproto/ext"
"github.com/charmbracelet/log"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/conflictutil"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/ruleutil"
"github.com/krau/SaveAny-Bot/common/i18n"
@@ -17,14 +18,17 @@ import (
"github.com/krau/SaveAny-Bot/core/tasks/batchtfile"
tftask "github.com/krau/SaveAny-Bot/core/tasks/tfile"
"github.com/krau/SaveAny-Bot/database"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
"github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
// 创建一个 tfile.TGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, file tfile.TGFileMessage, trackMsgID int) error {
func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, file tfile.TGFileMessage, trackMsgID int, conflictStrategy ...string) error {
logger := log.FromContext(ctx)
strategy := selectedConflictStrategy(conflictStrategy)
user, err := database.GetUserByChatID(ctx, userID)
if err != nil {
logger.Errorf("Failed to get user by chat ID: %s", err)
@@ -36,6 +40,7 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
})
return dispatcher.EndGroups
}
strategy = conflictutil.ResolveStrategy(user, strategy)
if user.ApplyRule && user.Rules != nil {
matched, matchedStorageName, matchedDirPath := ruleutil.ApplyRule(ctx, user.Rules, ruleutil.NewInput(file))
if !matched {
@@ -60,7 +65,26 @@ func CreateAndAddTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage
}
startCreateTask:
storagePath := path.Join(dirPath, file.Name())
if strategy == tcbdata.ConflictStrategyAsk || strategy == tcbdata.ConflictStrategySkip {
exists := stor.Exists(ctx, storagePath)
if exists && strategy == tcbdata.ConflictStrategyAsk {
return promptTGFileConflictStrategy(ctx, userID, stor.Name(), dirPath, []tfile.TGFileMessage{file}, false, []string{conflictutil.FormatPath(stor.Name(), storagePath)}, trackMsgID)
}
if exists {
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: i18n.T(i18nk.BotMsgCommonInfoAllConflictFilesSkipped, map[string]any{
"Skipped": file.Name(),
}),
ReplyMarkup: nil,
})
return dispatcher.EndGroups
}
}
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
if strategy == tcbdata.ConflictStrategyOverwrite {
injectCtx = storage.WithOverwrite(injectCtx)
}
taskid := xid.New().String()
task, err := tftask.NewTGFileTask(taskid, injectCtx, file, stor, storagePath,
tftask.NewProgressTrack(
@@ -97,8 +121,9 @@ startCreateTask:
}
// 创建一个 batchtfile.BatchTGFileTask 并添加到任务队列中, 以编辑消息的方式反馈结果
func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, files []tfile.TGFileMessage, trackMsgID int) error {
func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor storage.Storage, dirPath string, files []tfile.TGFileMessage, trackMsgID int, conflictStrategy ...string) error {
logger := log.FromContext(ctx)
strategy := selectedConflictStrategy(conflictStrategy)
user, err := database.GetUserByChatID(ctx, userID)
if err != nil {
logger.Errorf("Failed to get user by chat ID: %s", err)
@@ -110,6 +135,7 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
})
return dispatcher.EndGroups
}
strategy = conflictutil.ResolveStrategy(user, strategy)
useRule := user.ApplyRule && user.Rules != nil
@@ -128,14 +154,17 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
return storname, dirP
}
skipped := make([]string, 0)
conflicts := make([]string, 0)
elems := make([]batchtfile.TaskElement, 0, len(files))
type albumFile struct {
file tfile.TGFileMessage
storage storage.Storage
dirPath string
}
albumFiles := make(map[int64][]albumFile, 0)
for _, file := range files {
storName, dirPath := applyRule(file)
storName, matchedDirPath := applyRule(file)
fileStor := stor
if storName != stor.Name() && storName != "" {
fileStor, err = storage.GetStorageByUserIDAndName(ctx, user.ChatID, storName)
@@ -150,8 +179,19 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
return dispatcher.EndGroups
}
}
if !dirPath.NeedNewForAlbum() {
storPath := path.Join(dirPath.String(), file.Name())
if !matchedDirPath.NeedNewForAlbum() {
storPath := path.Join(matchedDirPath.String(), file.Name())
if strategy == tcbdata.ConflictStrategyAsk || strategy == tcbdata.ConflictStrategySkip {
exists := fileStor.Exists(ctx, storPath)
if exists && strategy == tcbdata.ConflictStrategyAsk {
conflicts = append(conflicts, conflictutil.FormatPath(fileStor.Name(), storPath))
continue
}
if exists {
skipped = append(skipped, file.Name())
continue
}
}
elem, err := batchtfile.NewTaskElement(fileStor, storPath, file)
if err != nil {
logger.Errorf("Failed to create task element: %s", err)
@@ -170,12 +210,17 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
logger.Warnf("File %s is not in a group, skipping album handling", file.Name())
continue
}
fileDirPath := matchedDirPath.String()
if matchedDirPath.NeedNewForAlbum() {
fileDirPath = dirPath
}
if _, ok := albumFiles[groupId]; !ok {
albumFiles[groupId] = make([]albumFile, 0)
}
albumFiles[groupId] = append(albumFiles[groupId], albumFile{
file: file,
storage: fileStor,
dirPath: fileDirPath,
})
}
}
@@ -188,7 +233,18 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
albumDir := strings.TrimSuffix(path.Base(afiles[0].file.Name()), path.Ext(afiles[0].file.Name()))
albumStor := afiles[0].storage
for _, af := range afiles {
afstorPath := path.Join(dirPath, albumDir, af.file.Name())
afstorPath := path.Join(af.dirPath, albumDir, af.file.Name())
if strategy == tcbdata.ConflictStrategyAsk || strategy == tcbdata.ConflictStrategySkip {
exists := albumStor.Exists(ctx, afstorPath)
if exists && strategy == tcbdata.ConflictStrategyAsk {
conflicts = append(conflicts, conflictutil.FormatPath(albumStor.Name(), afstorPath))
continue
}
if exists {
skipped = append(skipped, af.file.Name())
continue
}
}
elem, err := batchtfile.NewTaskElement(albumStor, afstorPath, af.file)
if err != nil {
logger.Errorf("Failed to create task element for album file: %s", err)
@@ -204,9 +260,26 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
}
}
if strategy == tcbdata.ConflictStrategyAsk && len(conflicts) > 0 {
return promptTGFileConflictStrategy(ctx, userID, stor.Name(), dirPath, files, true, conflicts, trackMsgID)
}
injectCtx := tgutil.ExtWithContext(ctx.Context, ctx)
if strategy == tcbdata.ConflictStrategyOverwrite {
injectCtx = storage.WithOverwrite(injectCtx)
}
if len(elems) == 0 {
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: i18n.T(i18nk.BotMsgCommonInfoAllConflictFilesSkipped, map[string]any{
"Skipped": strings.Join(skipped, "\n"),
}),
ReplyMarkup: nil,
})
return dispatcher.EndGroups
}
taskid := xid.New().String()
task := batchtfile.NewBatchTGFileTask(taskid, injectCtx, elems, batchtfile.NewProgressTracker(trackMsgID, userID), true)
task := batchtfile.NewBatchTGFileTask(taskid, injectCtx, elems, batchtfile.NewProgressTrackerWithSkipped(trackMsgID, userID, skipped), true)
if err := core.AddTask(injectCtx, task); err != nil {
logger.Errorf("Failed to add batch task: %s", err)
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
@@ -218,11 +291,48 @@ func CreateAndAddBatchTGFileTaskWithEdit(ctx *ext.Context, userID int64, stor st
return dispatcher.EndGroups
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: i18n.T(i18nk.BotMsgCommonInfoBatchTasksAdded, map[string]any{
"Count": len(files),
}),
ID: trackMsgID,
Message: buildBatchAddedMessage(len(elems), skipped),
ReplyMarkup: nil,
})
return dispatcher.EndGroups
}
func promptTGFileConflictStrategy(ctx *ext.Context, userID int64, storageName, dirPath string, files []tfile.TGFileMessage, asBatch bool, conflicts []string, trackMsgID int) error {
markup, err := msgelem.BuildConflictStrategyMarkup(tcbdata.Add{
TaskType: tasktype.TaskTypeTgfiles,
SelectedStorName: storageName,
SettedDir: true,
SelectedDirPath: dirPath,
Files: files,
AsBatch: asBatch,
})
if err != nil {
return err
}
ctx.EditMessage(userID, &tg.MessagesEditMessageRequest{
ID: trackMsgID,
Message: i18n.T(i18nk.BotMsgCommonPromptSelectConflictStrategy, map[string]any{"Files": conflictutil.FormatPaths(conflicts)}),
ReplyMarkup: markup,
})
return dispatcher.EndGroups
}
func selectedConflictStrategy(strategies []string) string {
if len(strategies) == 0 {
return ""
}
return strategies[0]
}
func buildBatchAddedMessage(count int, skipped []string) string {
if len(skipped) == 0 {
return i18n.T(i18nk.BotMsgCommonInfoBatchTasksAdded, map[string]any{
"Count": count,
})
}
return i18n.T(i18nk.BotMsgCommonInfoBatchTasksAddedWithSkipped, map[string]any{
"Count": count,
"Skipped": strings.Join(skipped, "\n"),
})
}

View File

@@ -64,7 +64,7 @@ func handleWatchCmd(ctx *ext.Context, update *ext.Update) error {
filter := ""
if len(args) > 2 {
filterArg := strings.Join(args[2:], " ")
filterType := strings.Split(filterArg, ":")[0]
filterType, _, _ := strings.Cut(filterArg, ":")
filterData := strings.Split(filterArg, ":")[1]
if filterType == "" || filterData == "" {
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgWatchErrorFilterFormatInvalid)), nil)

View File

@@ -15,7 +15,7 @@ import (
// https://github.com/iyear/tdl/blob/master/core/tclient/tclient.go
func NewDefaultMiddlewares(ctx context.Context, timeout time.Duration) []telegram.Middleware {
return []telegram.Middleware{
recovery.New(ctx, newBackoff(timeout)),
recovery.New(ctx, func() backoff.BackOff { return newBackoff(timeout) }),
retry.New(config.C().Telegram.RpcRetry),
floodwait.NewSimpleWaiter(),
}

View File

@@ -14,19 +14,28 @@ import (
)
type recovery struct {
ctx context.Context
backoff backoff.BackOff
ctx context.Context
newBackoff func() backoff.BackOff
}
func New(ctx context.Context, backoff backoff.BackOff) telegram.Middleware {
// New returns a recovery middleware.
//
// newBackoff is a factory that must return a fresh backoff.BackOff on every call: backoff implementations in
// cenkalti/backoff/v4 (notably ExponentialBackOff) are not safe for concurrent
// use, and the Telegram client invokes RPCs from many goroutines in parallel.
//
// Sharing a single instance corrupts its internal counters, breaks the
// exponential interval, and defeats MaxElapsedTime - see issue #218.
func New(ctx context.Context, newBackoff func() backoff.BackOff) telegram.Middleware {
return &recovery{
ctx: ctx,
backoff: backoff,
ctx: ctx,
newBackoff: newBackoff,
}
}
func (r *recovery) Handle(next tg.Invoker) telegram.InvokeFunc {
return func(ctx context.Context, input bin.Encoder, output bin.Decoder) error {
b := r.newBackoff()
return backoff.RetryNotify(func() error {
if err := next.Invoke(ctx, input, output); err != nil {
@@ -38,7 +47,7 @@ func (r *recovery) Handle(next tg.Invoker) telegram.InvokeFunc {
}
return nil
}, r.backoff, func(err error, duration time.Duration) {
}, b, func(err error, duration time.Duration) {
log.FromContext(ctx).Debug("Wait for connection recovery", "error", err, "duration", duration)
})
}

View File

@@ -2,6 +2,7 @@ package user
import (
"context"
"sync"
"time"
"github.com/celestix/gotgproto"
@@ -20,17 +21,18 @@ import (
)
var uc *gotgproto.Client
var ectx *ext.Context
// getEctx lazily creates the user-client ext.Context exactly once. Guarded by
// sync.OnceValue so concurrent GetCtx calls cannot race on ectx creation.
var getEctx = sync.OnceValue(func() *ext.Context {
return uc.CreateContext()
})
func GetCtx() *ext.Context {
if ectx != nil {
return ectx
}
if uc == nil {
return nil
}
ectx = uc.CreateContext()
return ectx
return getEctx()
}
func Login(ctx context.Context) (*gotgproto.Client, error) {

View File

@@ -22,7 +22,13 @@ func main() {
pkg := flag.String("pkg", "i18nk", "Package name for generated file")
flag.Parse()
type localeFile struct {
path string
keys map[string]struct{}
}
keys := make(map[string]struct{})
var localeFiles []localeFile
err := filepath.WalkDir(*dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
@@ -42,7 +48,12 @@ func main() {
return fmt.Errorf("failed to parse yaml %s: %w", path, err)
}
collectKeys(content, "", keys)
fileKeys := make(map[string]struct{})
collectKeys(content, "", fileKeys)
localeFiles = append(localeFiles, localeFile{path: path, keys: fileKeys})
for k := range fileKeys {
keys[k] = struct{}{}
}
return nil
})
if err != nil {
@@ -50,6 +61,25 @@ func main() {
os.Exit(1)
}
// 一致性校验: 每个语言文件必须包含全部 key
invalid := false
for _, f := range localeFiles {
var missing []string
for k := range keys {
if _, ok := f.keys[k]; !ok {
missing = append(missing, k)
}
}
if len(missing) > 0 {
invalid = true
sort.Strings(missing)
fmt.Fprintf(os.Stderr, "Error: locale file %s is missing %d key(s): %s\n", f.path, len(missing), strings.Join(missing, ", "))
}
}
if invalid {
os.Exit(1)
}
var list []string
for k := range keys {
list = append(list, k)

View File

@@ -5,6 +5,7 @@ import (
"fmt"
"github.com/krau/SaveAny-Bot/cmd/upload"
"github.com/krau/SaveAny-Bot/cmd/watch"
"github.com/krau/SaveAny-Bot/config"
"github.com/spf13/cobra"
)
@@ -18,6 +19,7 @@ var rootCmd = &cobra.Command{
func init() {
config.RegisterFlags(rootCmd)
upload.Register(rootCmd)
watch.Register(rootCmd)
}
func Execute(ctx context.Context) {

View File

@@ -61,6 +61,7 @@ func Run(cmd *cobra.Command, _ []string) {
<-ctx.Done()
logger.Info("Exiting...")
defer logger.Info("Exit complete")
core.Close()
cleanCache()
}
@@ -87,7 +88,7 @@ func initAll(ctx context.Context) (<-chan struct{}, error) {
}
}
if err := api.Start(ctx); err != nil {
logger.Error("Failed to start API server", "error", err)
logger.Fatal("Failed to start API server", "error", err)
}
return bot.Init(ctx), nil
}

145
cmd/watch/cmd.go Normal file
View File

@@ -0,0 +1,145 @@
package watch
import (
"fmt"
"time"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/client/bot"
"github.com/krau/SaveAny-Bot/common/cache"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/database"
stortype "github.com/krau/SaveAny-Bot/pkg/enums/storage"
"github.com/krau/SaveAny-Bot/storage"
"github.com/spf13/cobra"
)
var watchCmd = &cobra.Command{
Use: "watch",
Short: "watch a local directory and auto-upload changed files to storage",
Long: `Watch a local directory and automatically upload created or modified files
to the specified storage backend, preserving the relative directory structure.
Example:
saveany-bot watch -p /data/inbox -s mystorage -d backup --recursive`,
RunE: runWatch,
}
func Register(root *cobra.Command) {
flags := watchCmd.Flags()
flags.StringP("path", "p", "", "local directory path to watch")
watchCmd.MarkFlagRequired("path")
flags.StringP("storage", "s", "", "storage name to upload to")
watchCmd.MarkFlagRequired("storage")
flags.StringP("dir", "d", "", "storage dir to upload to, default is the base_path of the storage")
flags.BoolP("recursive", "r", false, "watch subdirectories recursively")
flags.Bool("overwrite", false, "overwrite existing files on storage instead of skipping")
flags.Bool("initial-scan", false, "upload existing files in the directory on startup")
flags.Duration("debounce", 2*time.Second, "wait time after the last change before uploading a file")
flags.Int("upload-workers", 0, "number of concurrent uploads, default is config.workers")
flags.Duration("retry-delay", 3*time.Second, "delay between upload retries")
root.AddCommand(watchCmd)
}
func runWatch(cmd *cobra.Command, _ []string) error {
watchPath, err := cmd.Flags().GetString("path")
if err != nil {
return err
}
storName, err := cmd.Flags().GetString("storage")
if err != nil {
return err
}
destDir, err := cmd.Flags().GetString("dir")
if err != nil {
return err
}
recursive, err := cmd.Flags().GetBool("recursive")
if err != nil {
return err
}
overwrite, err := cmd.Flags().GetBool("overwrite")
if err != nil {
return err
}
initialScan, err := cmd.Flags().GetBool("initial-scan")
if err != nil {
return err
}
debounce, err := cmd.Flags().GetDuration("debounce")
if err != nil {
return err
}
uploadWorkers, err := cmd.Flags().GetInt("upload-workers")
if err != nil {
return err
}
retryDelay, err := cmd.Flags().GetDuration("retry-delay")
if err != nil {
return err
}
ctx := cmd.Context()
logger := log.FromContext(ctx)
configFile := config.GetConfigFile(cmd)
if err := config.Init(ctx, configFile); err != nil {
return fmt.Errorf("failed to load config: %w", err)
}
i18n.Init(config.C().Lang)
cache.Init()
database.Init(ctx)
stor, err := storage.GetStorageByName(ctx, storName)
if err != nil {
return fmt.Errorf("failed to get storage %q: %w", storName, err)
}
// Telegram storage needs the bot client and its ext context injected into ctx.
if stor.Type() == stortype.Telegram {
bot.Init(ctx)
ctx = tgutil.ExtWithContext(ctx, bot.ExtContext())
}
if uploadWorkers < 1 {
uploadWorkers = config.C().Workers
}
uploader := NewUploader(ctx, UploaderOptions{
Storage: stor,
DestDir: destDir,
Overwrite: overwrite,
Workers: uploadWorkers,
Retry: config.C().Retry,
RetryDelay: retryDelay,
})
watcher, err := NewWatcher(ctx, WatcherOptions{
Root: watchPath,
Recursive: recursive,
Debounce: debounce,
Uploader: uploader,
})
if err != nil {
uploader.Close()
return fmt.Errorf("failed to create watcher: %w", err)
}
if initialScan {
watcher.ScanExisting(ctx)
}
logger.Infof("watch started: %s -> storage %q dir %q", watchPath, storName, destDir)
// Run blocks until ctx is cancelled (e.g. SIGINT).
runErr := watcher.Run(ctx)
// Wait for in-flight uploads to finish before exiting.
logger.Info("waiting for in-flight uploads to finish...")
uploader.Close()
logger.Info("watch stopped")
return runErr
}

227
cmd/watch/uploader.go Normal file
View File

@@ -0,0 +1,227 @@
package watch
import (
"context"
"fmt"
"os"
"path"
"path/filepath"
"sync"
"time"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/storage"
)
type uploadJob struct {
// localPath is the absolute path of the local file.
localPath string
// relPath is relative to the watch root, used to preserve directory structure on storage.
relPath string
}
// Uploader uploads local files to the target storage via a worker pool.
// If a file changes while being uploaded, it is re-uploaded once after the
// current upload finishes, instead of being queued multiple times.
type Uploader struct {
stor storage.Storage
destDir string
overwrite bool
retry int
retryDelay time.Duration
logger *log.Logger
jobs chan uploadJob
wg sync.WaitGroup
mu sync.Mutex
// inflight maps in-progress (or queued) file paths. A true value means the
// file changed during upload and must be re-queued once done.
inflight map[string]bool
}
type UploaderOptions struct {
Storage storage.Storage
DestDir string
Overwrite bool
Workers int
Retry int
RetryDelay time.Duration
QueueSize int
}
// NewUploader creates and starts an Uploader. The caller must call Close when done.
func NewUploader(ctx context.Context, opts UploaderOptions) *Uploader {
if opts.Workers < 1 {
opts.Workers = 1
}
if opts.Retry < 1 {
opts.Retry = 1
}
if opts.RetryDelay <= 0 {
opts.RetryDelay = 3 * time.Second
}
if opts.QueueSize < opts.Workers {
opts.QueueSize = opts.Workers * 64
}
u := &Uploader{
stor: opts.Storage,
destDir: opts.DestDir,
overwrite: opts.Overwrite,
retry: opts.Retry,
retryDelay: opts.RetryDelay,
logger: log.FromContext(ctx).WithPrefix("uploader"),
jobs: make(chan uploadJob, opts.QueueSize),
inflight: make(map[string]bool),
}
for i := 0; i < opts.Workers; i++ {
u.wg.Add(1)
go u.worker(ctx)
}
return u
}
// Submit enqueues an upload job. If the same file is already in flight, it is
// marked for re-upload instead of being queued again. Returns false if ctx is
// cancelled before the job can be enqueued.
func (u *Uploader) Submit(ctx context.Context, job uploadJob) bool {
u.mu.Lock()
if _, ok := u.inflight[job.localPath]; ok {
u.inflight[job.localPath] = true
u.mu.Unlock()
u.logger.Debugf("file %s already in flight, marked for re-upload", job.localPath)
return true
}
u.inflight[job.localPath] = false
u.mu.Unlock()
select {
case u.jobs <- job:
return true
case <-ctx.Done():
u.mu.Lock()
delete(u.inflight, job.localPath)
u.mu.Unlock()
return false
}
}
func (u *Uploader) worker(ctx context.Context) {
defer u.wg.Done()
for {
select {
case <-ctx.Done():
return
case job, ok := <-u.jobs:
if !ok {
return
}
u.process(ctx, job)
}
}
}
func (u *Uploader) process(ctx context.Context, job uploadJob) {
if err := u.uploadWithRetry(ctx, job); err != nil {
if ctx.Err() != nil {
u.clearInflight(job.localPath)
return
}
u.logger.Errorf("failed to upload %s after %d attempt(s): %v", job.localPath, u.retry, err)
}
// Re-queue if the file changed again while it was being uploaded.
u.mu.Lock()
needReupload := u.inflight[job.localPath]
if needReupload {
u.inflight[job.localPath] = false
} else {
delete(u.inflight, job.localPath)
}
u.mu.Unlock()
if needReupload {
select {
case u.jobs <- job:
u.logger.Debugf("re-queued %s due to changes during upload", job.localPath)
case <-ctx.Done():
u.clearInflight(job.localPath)
}
}
}
func (u *Uploader) clearInflight(localPath string) {
u.mu.Lock()
delete(u.inflight, localPath)
u.mu.Unlock()
}
func (u *Uploader) uploadWithRetry(ctx context.Context, job uploadJob) error {
var lastErr error
for attempt := 1; attempt <= u.retry; attempt++ {
if ctx.Err() != nil {
return ctx.Err()
}
err := u.upload(ctx, job)
if err == nil {
return nil
}
if ctx.Err() != nil {
return ctx.Err()
}
lastErr = err
u.logger.Warnf("upload %s failed (attempt %d/%d): %v", job.localPath, attempt, u.retry, err)
if attempt < u.retry {
select {
case <-time.After(u.retryDelay):
case <-ctx.Done():
return ctx.Err()
}
}
}
return lastErr
}
func (u *Uploader) upload(ctx context.Context, job uploadJob) error {
file, err := os.Open(filepath.Clean(job.localPath))
if err != nil {
return fmt.Errorf("failed to open file: %w", err)
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return fmt.Errorf("failed to stat file: %w", err)
}
if info.IsDir() {
return fmt.Errorf("path is a directory, not a file")
}
// Keep the relative directory structure on the storage side.
storagePath := path.Join(u.destDir, filepath.ToSlash(job.relPath))
uploadCtx := context.WithValue(ctx, ctxkey.ContentLength, info.Size())
if u.overwrite {
uploadCtx = storage.WithOverwrite(uploadCtx)
} else if u.stor.Exists(uploadCtx, storagePath) {
u.logger.Infof("skip existing file: %s", storagePath)
return nil
}
u.logger.Infof("uploading %s -> %s (%d bytes)", job.localPath, storagePath, info.Size())
if err := u.stor.Save(uploadCtx, file, storagePath); err != nil {
return fmt.Errorf("failed to save to storage: %w", err)
}
u.logger.Infof("uploaded %s", storagePath)
return nil
}
// Close stops accepting jobs and waits for in-flight uploads to finish.
func (u *Uploader) Close() {
close(u.jobs)
u.wg.Wait()
}

269
cmd/watch/watcher.go Normal file
View File

@@ -0,0 +1,269 @@
package watch
import (
"context"
"fmt"
"io/fs"
"os"
"path/filepath"
"sync"
"time"
"github.com/charmbracelet/log"
"github.com/fsnotify/fsnotify"
)
// Watcher watches a local directory and submits stable files to the Uploader.
//
// Write-completion detection: fsnotify emits Write events throughout a write.
// Watcher debounces per file and only uploads once the file size stays
// unchanged across a debounce window, avoiding uploads of partial files.
type Watcher struct {
root string
recursive bool
debounce time.Duration
uploader *Uploader
logger *log.Logger
fsw *fsnotify.Watcher
mu sync.Mutex
pending map[string]*time.Timer
// lastSize is the last observed file size, used to detect a stable write.
lastSize map[string]int64
}
type WatcherOptions struct {
Root string
Recursive bool
Debounce time.Duration
Uploader *Uploader
}
// NewWatcher creates a Watcher.
func NewWatcher(ctx context.Context, opts WatcherOptions) (*Watcher, error) {
if opts.Debounce <= 0 {
opts.Debounce = 2 * time.Second
}
root, err := filepath.Abs(opts.Root)
if err != nil {
return nil, fmt.Errorf("failed to resolve root path: %w", err)
}
info, err := os.Stat(root)
if err != nil {
return nil, fmt.Errorf("failed to stat root path: %w", err)
}
if !info.IsDir() {
return nil, fmt.Errorf("watch path must be a directory: %s", root)
}
fsw, err := fsnotify.NewWatcher()
if err != nil {
return nil, fmt.Errorf("failed to create fsnotify watcher: %w", err)
}
w := &Watcher{
root: root,
recursive: opts.Recursive,
debounce: opts.Debounce,
uploader: opts.Uploader,
logger: log.FromContext(ctx).WithPrefix("watcher"),
fsw: fsw,
pending: make(map[string]*time.Timer),
lastSize: make(map[string]int64),
}
return w, nil
}
// Run starts watching and blocks until ctx is cancelled.
func (w *Watcher) Run(ctx context.Context) error {
if err := w.addDir(w.root); err != nil {
w.fsw.Close()
return fmt.Errorf("failed to watch root: %w", err)
}
w.logger.Infof("watching %s (recursive=%v, debounce=%s)", w.root, w.recursive, w.debounce)
defer w.cleanup()
for {
select {
case <-ctx.Done():
w.logger.Info("stopping watcher")
return nil
case event, ok := <-w.fsw.Events:
if !ok {
return nil
}
w.handleEvent(ctx, event)
case err, ok := <-w.fsw.Errors:
if !ok {
return nil
}
w.logger.Errorf("watch error: %v", err)
}
}
}
func (w *Watcher) handleEvent(ctx context.Context, event fsnotify.Event) {
// Remove/Rename: cancel any pending upload for this path.
if event.Has(fsnotify.Remove) || event.Has(fsnotify.Rename) {
w.cancelPending(event.Name)
return
}
if !event.Has(fsnotify.Create) && !event.Has(fsnotify.Write) {
return
}
info, err := os.Stat(event.Name)
if err != nil {
// File may have been removed or moved; ignore.
return
}
if info.IsDir() {
// New directory: watch it recursively and scan files already inside.
if event.Has(fsnotify.Create) && w.recursive {
if err := w.addDir(event.Name); err != nil {
w.logger.Errorf("failed to watch new dir %s: %v", event.Name, err)
}
w.scanExisting(ctx, event.Name)
}
return
}
w.scheduleUpload(ctx, event.Name)
}
// scheduleUpload schedules a debounced upload for a file.
func (w *Watcher) scheduleUpload(ctx context.Context, file string) {
w.mu.Lock()
defer w.mu.Unlock()
if t, ok := w.pending[file]; ok {
t.Stop()
}
w.pending[file] = time.AfterFunc(w.debounce, func() {
w.maybeUpload(ctx, file)
})
}
// maybeUpload submits the upload once the debounce window passes and the file
// size is stable; otherwise it waits another window.
func (w *Watcher) maybeUpload(ctx context.Context, file string) {
if ctx.Err() != nil {
return
}
info, err := os.Stat(file)
if err != nil {
w.cancelPending(file)
return
}
if info.IsDir() {
w.cancelPending(file)
return
}
w.mu.Lock()
prevSize, seen := w.lastSize[file]
curSize := info.Size()
if !seen || prevSize != curSize {
// Size still changing: likely still being written, wait another window.
w.lastSize[file] = curSize
w.pending[file] = time.AfterFunc(w.debounce, func() {
w.maybeUpload(ctx, file)
})
w.mu.Unlock()
return
}
// Size stable: treat write as complete.
delete(w.pending, file)
delete(w.lastSize, file)
w.mu.Unlock()
relPath, err := filepath.Rel(w.root, file)
if err != nil {
w.logger.Errorf("failed to compute relative path for %s: %v", file, err)
return
}
w.uploader.Submit(ctx, uploadJob{localPath: file, relPath: relPath})
}
func (w *Watcher) cancelPending(file string) {
w.mu.Lock()
defer w.mu.Unlock()
if t, ok := w.pending[file]; ok {
t.Stop()
delete(w.pending, file)
}
delete(w.lastSize, file)
}
// addDir adds a directory to the watch list, recursively when enabled.
func (w *Watcher) addDir(dir string) error {
if !w.recursive {
return w.fsw.Add(dir)
}
return filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err != nil {
w.logger.Warnf("skip path %s: %v", p, err)
return nil
}
if d.IsDir() {
if addErr := w.fsw.Add(p); addErr != nil {
w.logger.Warnf("failed to watch dir %s: %v", p, addErr)
} else {
w.logger.Debugf("watching dir %s", p)
}
}
return nil
})
}
// scanExisting submits files already present under dir (initial sync and new-dir backfill).
func (w *Watcher) scanExisting(ctx context.Context, dir string) {
walkFn := func(p string, d fs.DirEntry, err error) error {
if err != nil {
w.logger.Warnf("skip path %s: %v", p, err)
return nil
}
if ctx.Err() != nil {
return ctx.Err()
}
if d.IsDir() {
if !w.recursive && p != dir {
return fs.SkipDir
}
return nil
}
relPath, relErr := filepath.Rel(w.root, p)
if relErr != nil {
w.logger.Errorf("failed to compute relative path for %s: %v", p, relErr)
return nil
}
w.uploader.Submit(ctx, uploadJob{localPath: p, relPath: relPath})
return nil
}
if err := filepath.WalkDir(dir, walkFn); err != nil && ctx.Err() == nil {
w.logger.Errorf("failed to scan dir %s: %v", dir, err)
}
}
// ScanExisting triggers a one-time scan and upload of existing files under the watch root.
func (w *Watcher) ScanExisting(ctx context.Context) {
w.logger.Info("scanning existing files for initial sync")
w.scanExisting(ctx, w.root)
}
func (w *Watcher) cleanup() {
w.mu.Lock()
for _, t := range w.pending {
t.Stop()
}
w.pending = make(map[string]*time.Timer)
w.lastSize = make(map[string]int64)
w.mu.Unlock()
w.fsw.Close()
}

View File

@@ -2,6 +2,7 @@ package cache
import (
"fmt"
"sync"
"time"
"github.com/charmbracelet/log"
@@ -9,24 +10,26 @@ import (
"github.com/krau/SaveAny-Bot/config"
)
var cache *ristretto.Cache[string, any]
var (
cache *ristretto.Cache[string, any]
initOnce sync.Once
)
func Init() {
if cache != nil {
panic("cache already initialized")
}
c, err := ristretto.NewCache(&ristretto.Config[string, any]{
NumCounters: config.C().Cache.NumCounters,
MaxCost: config.C().Cache.MaxCost,
BufferItems: 64,
OnReject: func(item *ristretto.Item[any]) {
log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
},
initOnce.Do(func() {
c, err := ristretto.NewCache(&ristretto.Config[string, any]{
NumCounters: config.C().Cache.NumCounters,
MaxCost: config.C().Cache.MaxCost,
BufferItems: 64,
OnReject: func(item *ristretto.Item[any]) {
log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
},
})
if err != nil {
log.Fatalf("failed to create ristretto cache: %v", err)
}
cache = c
})
if err != nil {
log.Fatalf("failed to create ristretto cache: %v", err)
}
cache = c
}
func Set(key string, value any) error {

View File

@@ -36,6 +36,9 @@ const (
BotMsgCmdUpdate Key = "bot.msg.cmd.update"
BotMsgCmdWatch Key = "bot.msg.cmd.watch"
BotMsgCmdYtdlp Key = "bot.msg.cmd.ytdlp"
BotMsgCommonButtonConflictOverwrite Key = "bot.msg.common.button_conflict_overwrite"
BotMsgCommonButtonConflictRename Key = "bot.msg.common.button_conflict_rename"
BotMsgCommonButtonConflictSkip Key = "bot.msg.common.button_conflict_skip"
BotMsgCommonCancelButtonText Key = "bot.msg.common.cancel_button_text"
BotMsgCommonErrorBuildDirSelectKeyboardFailed Key = "bot.msg.common.error_build_dir_select_keyboard_failed"
BotMsgCommonErrorBuildStorageSelectKeyboardFailed Key = "bot.msg.common.error_build_storage_select_keyboard_failed"
@@ -63,7 +66,10 @@ const (
BotMsgCommonErrorTaskAddFailed Key = "bot.msg.common.error_task_add_failed"
BotMsgCommonErrorTaskCreateFailed Key = "bot.msg.common.error_task_create_failed"
BotMsgCommonErrorUpdateUserInfoFailed Key = "bot.msg.common.error_update_user_info_failed"
BotMsgCommonInfoAllConflictFilesSkipped Key = "bot.msg.common.info_all_conflict_files_skipped"
BotMsgCommonInfoBatchTasksAdded Key = "bot.msg.common.info_batch_tasks_added"
BotMsgCommonInfoBatchTasksAddedWithSkipped Key = "bot.msg.common.info_batch_tasks_added_with_skipped"
BotMsgCommonInfoConflictFilesSkipped Key = "bot.msg.common.info_conflict_files_skipped"
BotMsgCommonInfoDefaultStorageSet Key = "bot.msg.common.info_default_storage_set"
BotMsgCommonInfoDefaultStorageWithDirSet Key = "bot.msg.common.info_default_storage_with_dir_set"
BotMsgCommonInfoFetchingFileInfo Key = "bot.msg.common.info_fetching_file_info"
@@ -73,16 +79,25 @@ const (
BotMsgCommonInfoSilentModeOff Key = "bot.msg.common.info_silent_mode_off"
BotMsgCommonInfoSilentModeOn Key = "bot.msg.common.info_silent_mode_on"
BotMsgCommonInfoTaskAdded Key = "bot.msg.common.info_task_added"
BotMsgCommonPromptConflictMoreFiles Key = "bot.msg.common.prompt_conflict_more_files"
BotMsgCommonPromptSelectConflictStrategy Key = "bot.msg.common.prompt_select_conflict_strategy"
BotMsgCommonPromptSelectDefaultDir Key = "bot.msg.common.prompt_select_default_dir"
BotMsgCommonPromptSelectDefaultStorage Key = "bot.msg.common.prompt_select_default_storage"
BotMsgCommonPromptSelectDir Key = "bot.msg.common.prompt_select_dir"
BotMsgConfigButtonConflictStrategy Key = "bot.msg.config.button_conflict_strategy"
BotMsgConfigButtonFilenameStrategy Key = "bot.msg.config.button_filename_strategy"
BotMsgConfigConflictStrategyAsk Key = "bot.msg.config.conflict_strategy_ask"
BotMsgConfigConflictStrategyOverwrite Key = "bot.msg.config.conflict_strategy_overwrite"
BotMsgConfigConflictStrategyRename Key = "bot.msg.config.conflict_strategy_rename"
BotMsgConfigConflictStrategySkip Key = "bot.msg.config.conflict_strategy_skip"
BotMsgConfigErrorInvalidCallbackData Key = "bot.msg.config.error_invalid_callback_data"
BotMsgConfigErrorInvalidTemplate Key = "bot.msg.config.error_invalid_template"
BotMsgConfigFnametmplHelp Key = "bot.msg.config.fnametmpl_help"
BotMsgConfigInfoConflictStrategySet Key = "bot.msg.config.info_conflict_strategy_set"
BotMsgConfigInfoCurrentTemplatePrefix Key = "bot.msg.config.info_current_template_prefix"
BotMsgConfigInfoFilenameStrategySet Key = "bot.msg.config.info_filename_strategy_set"
BotMsgConfigInfoTemplateUpdated Key = "bot.msg.config.info_template_updated"
BotMsgConfigPromptSelectConflictStrategy Key = "bot.msg.config.prompt_select_conflict_strategy"
BotMsgConfigPromptSelectFilenameStrategy Key = "bot.msg.config.prompt_select_filename_strategy"
BotMsgConfigPromptSelectOption Key = "bot.msg.config.prompt_select_option"
BotMsgDirButtonDefault Key = "bot.msg.dir.button_default"
@@ -135,28 +150,52 @@ const (
BotMsgProgressAria2Downloading Key = "bot.msg.progress.aria2_downloading"
BotMsgProgressAria2Start Key = "bot.msg.progress.aria2_start"
BotMsgProgressAvgSpeedPrefix Key = "bot.msg.progress.avg_speed_prefix"
BotMsgProgressBatchDonePrefix Key = "bot.msg.progress.batch_done_prefix"
BotMsgProgressBatchProcessingPrefix Key = "bot.msg.progress.batch_processing_prefix"
BotMsgProgressBatchStartPrefix Key = "bot.msg.progress.batch_start_prefix"
BotMsgProgressBatchCanceled Key = "bot.msg.progress.batch_canceled"
BotMsgProgressBatchDone Key = "bot.msg.progress.batch_done"
BotMsgProgressBatchDoneWithSkipped Key = "bot.msg.progress.batch_done_with_skipped"
BotMsgProgressBatchFailedGroup Key = "bot.msg.progress.batch_failed_group"
BotMsgProgressBatchFailedItem Key = "bot.msg.progress.batch_failed_item"
BotMsgProgressBatchFailedTask Key = "bot.msg.progress.batch_failed_task"
BotMsgProgressBatchFailureStageBatchUpload Key = "bot.msg.progress.batch_failure_stage_batch_upload"
BotMsgProgressBatchFailureStageCache Key = "bot.msg.progress.batch_failure_stage_cache"
BotMsgProgressBatchFailureStageConfirm Key = "bot.msg.progress.batch_failure_stage_confirm"
BotMsgProgressBatchFailureStageDownload Key = "bot.msg.progress.batch_failure_stage_download"
BotMsgProgressBatchFailureStageInternal Key = "bot.msg.progress.batch_failure_stage_internal"
BotMsgProgressBatchFailureStageUpload Key = "bot.msg.progress.batch_failure_stage_upload"
BotMsgProgressBatchItemConfirming Key = "bot.msg.progress.batch_item_confirming"
BotMsgProgressBatchItemDownloading Key = "bot.msg.progress.batch_item_downloading"
BotMsgProgressBatchItemDownloadingUnknown Key = "bot.msg.progress.batch_item_downloading_unknown"
BotMsgProgressBatchItemRetrying Key = "bot.msg.progress.batch_item_retrying"
BotMsgProgressBatchItemTransferring Key = "bot.msg.progress.batch_item_transferring"
BotMsgProgressBatchItemTransferringUnknown Key = "bot.msg.progress.batch_item_transferring_unknown"
BotMsgProgressBatchItemUploading Key = "bot.msg.progress.batch_item_uploading"
BotMsgProgressBatchStatusHeader Key = "bot.msg.progress.batch_status_header"
BotMsgProgressBatchSummaryConfirming Key = "bot.msg.progress.batch_summary_confirming"
BotMsgProgressBatchSummaryFailed Key = "bot.msg.progress.batch_summary_failed"
BotMsgProgressBatchSummaryHiddenActive Key = "bot.msg.progress.batch_summary_hidden_active"
BotMsgProgressBatchSummarySkipped Key = "bot.msg.progress.batch_summary_skipped"
BotMsgProgressCurrentProgressPrefix Key = "bot.msg.progress.current_progress_prefix"
BotMsgProgressCurrentSpeedPrefix Key = "bot.msg.progress.current_speed_prefix"
BotMsgProgressDirectDonePrefix Key = "bot.msg.progress.direct_done_prefix"
BotMsgProgressDirectStart Key = "bot.msg.progress.direct_start"
BotMsgProgressDownloadDonePrefix Key = "bot.msg.progress.download_done_prefix"
BotMsgProgressDownloadFailedPrefix Key = "bot.msg.progress.download_failed_prefix"
BotMsgProgressDownloadedPrefix Key = "bot.msg.progress.downloaded_prefix"
BotMsgProgressDownloadingPrefix Key = "bot.msg.progress.downloading_prefix"
BotMsgProgressErrorPrefix Key = "bot.msg.progress.error_prefix"
BotMsgProgressFileNamePrefix Key = "bot.msg.progress.file_name_prefix"
BotMsgProgressFileProcessingPrefix Key = "bot.msg.progress.file_processing_prefix"
BotMsgProgressFileSizePrefix Key = "bot.msg.progress.file_size_prefix"
BotMsgProgressFileStartPrefix Key = "bot.msg.progress.file_start_prefix"
BotMsgProgressParsedDonePrefix Key = "bot.msg.progress.parsed_done_prefix"
BotMsgProgressParsedStartPrefix Key = "bot.msg.progress.parsed_start_prefix"
BotMsgProgressProcessingListPrefix Key = "bot.msg.progress.processing_list_prefix"
BotMsgProgressProcessingNone Key = "bot.msg.progress.processing_none"
BotMsgProgressSavePathPrefix Key = "bot.msg.progress.save_path_prefix"
BotMsgProgressTaskCanceled Key = "bot.msg.progress.task_canceled"
BotMsgProgressSingleCanceled Key = "bot.msg.progress.single_canceled"
BotMsgProgressSingleDone Key = "bot.msg.progress.single_done"
BotMsgProgressSingleDownloading Key = "bot.msg.progress.single_downloading"
BotMsgProgressSingleDownloadingUnknown Key = "bot.msg.progress.single_downloading_unknown"
BotMsgProgressSingleFailed Key = "bot.msg.progress.single_failed"
BotMsgProgressSingleStatusHeader Key = "bot.msg.progress.single_status_header"
BotMsgProgressSingleUploadRetrying Key = "bot.msg.progress.single_upload_retrying"
BotMsgProgressSingleUploading Key = "bot.msg.progress.single_uploading"
BotMsgProgressSizeWithFiles Key = "bot.msg.progress.size_with_files"
BotMsgProgressSizeWithResources Key = "bot.msg.progress.size_with_resources"
BotMsgProgressTaskCanceledWithId Key = "bot.msg.progress.task_canceled_with_id"
BotMsgProgressTaskFailedWithError Key = "bot.msg.progress.task_failed_with_error"
BotMsgProgressTelegraphDonePrefix Key = "bot.msg.progress.telegraph_done_prefix"
@@ -185,6 +224,7 @@ const (
BotMsgRuleErrorGetUserRulesFailed Key = "bot.msg.rule.error_get_user_rules_failed"
BotMsgRuleErrorInvalidRuleId Key = "bot.msg.rule.error_invalid_rule_id"
BotMsgRuleErrorInvalidRuleType Key = "bot.msg.rule.error_invalid_rule_type"
BotMsgRuleErrorStorageNotFound Key = "bot.msg.rule.error_storage_not_found"
BotMsgRuleErrorUpdateUserFailed Key = "bot.msg.rule.error_update_user_failed"
BotMsgRuleHelpAddSuffix Key = "bot.msg.rule.help_add_suffix"
BotMsgRuleHelpAvailableOps Key = "bot.msg.rule.help_available_ops"
@@ -192,18 +232,20 @@ const (
BotMsgRuleHelpCurrentModeEnabled Key = "bot.msg.rule.help_current_mode_enabled"
BotMsgRuleHelpDelSuffix Key = "bot.msg.rule.help_del_suffix"
BotMsgRuleHelpExistingRulesPrefix Key = "bot.msg.rule.help_existing_rules_prefix"
BotMsgRuleHelpPresetSuffix Key = "bot.msg.rule.help_preset_suffix"
BotMsgRuleHelpSwitchSuffix Key = "bot.msg.rule.help_switch_suffix"
BotMsgRuleHelpUsage Key = "bot.msg.rule.help_usage"
BotMsgRuleInfoCreateRuleSuccess Key = "bot.msg.rule.info_create_rule_success"
BotMsgRuleInfoDeleteRuleSuccess Key = "bot.msg.rule.info_delete_rule_success"
BotMsgRuleInfoPresetImported Key = "bot.msg.rule.info_preset_imported"
BotMsgRuleInfoRuleModeDisabled Key = "bot.msg.rule.info_rule_mode_disabled"
BotMsgRuleInfoRuleModeEnabled Key = "bot.msg.rule.info_rule_mode_enabled"
BotMsgRulePromptProvideRuleId Key = "bot.msg.rule.prompt_provide_rule_id"
BotMsgRulePromptProvideStorageName Key = "bot.msg.rule.prompt_provide_storage_name"
BotMsgSaveErrorInvalidIdOrUsername Key = "bot.msg.save.error_invalid_id_or_username"
BotMsgSaveHelpText Key = "bot.msg.save_help_text"
BotMsgStorageInfoFilenamePrefix Key = "bot.msg.storage.info_filename_prefix"
BotMsgStorageInfoPromptSelectStorage Key = "bot.msg.storage.info_prompt_select_storage"
BotMsgSyncpeersDone Key = "bot.msg.syncpeers.done"
BotMsgSyncpeersFailed Key = "bot.msg.syncpeers.failed"
BotMsgSyncpeersStart Key = "bot.msg.syncpeers.start"
BotMsgSyncpeersSuccess Key = "bot.msg.syncpeers.success"

View File

@@ -112,9 +112,17 @@ bot:
error_task_add_failed: "Failed to add task: {{.Error}}"
info_task_added: "Task added"
info_batch_tasks_added: "Batch tasks added, total {{.Count}} files"
info_batch_tasks_added_with_skipped: "Batch tasks added, total {{.Count}} files\nSkipped conflicting files:\n{{.Skipped}}"
info_all_conflict_files_skipped: "All conflicting files were skipped:\n{{.Skipped}}"
info_conflict_files_skipped: "Skipped conflicting files:\n{{.Skipped}}"
error_task_create_failed: "Failed to create task: {{.Error}}"
error_get_dir_failed: "Failed to get directory: {{.Error}}"
prompt_select_dir: "Please select a directory to store to"
prompt_select_conflict_strategy: "Files with the same name already exist. Please select a save strategy:\n{{.Files}}"
prompt_conflict_more_files: "...and {{.Count}} more files"
button_conflict_rename: "Rename"
button_conflict_overwrite: "Overwrite"
button_conflict_skip: "Skip"
prompt_select_default_dir: "Please select a default directory to save to"
info_default_storage_set: "Default storage set to: {{.Name}}"
info_default_storage_with_dir_set: "Default storage set to: {{.Name}}:/{{.Dir}}"
@@ -188,7 +196,11 @@ bot:
help_switch_suffix: " - Toggle rule mode\n"
help_add_suffix: " <type> <data> <storage_name> <path> - Add rule\n"
help_del_suffix: " <rule_id> - Delete rule\n"
help_preset_suffix: " <storage_name> [base_path] - Import built-in filetype rules (video/image/audio/document/archive)\n"
help_existing_rules_prefix: "\nCurrent rules:\n"
prompt_provide_storage_name: "Please provide a storage name"
error_storage_not_found: "Storage not found: {{.Storage}}"
info_preset_imported: "Imported {{.Count}} built-in classification rules into storage {{.Storage}}"
dir:
error_get_user_dirs_failed: "Failed to get user directories"
error_get_user_failed: "Failed to get user"
@@ -266,10 +278,17 @@ bot:
config:
prompt_select_option: "Please select an option to configure"
button_filename_strategy: "Filename strategy"
button_conflict_strategy: "Duplicate file strategy"
error_invalid_callback_data: "Invalid callback data"
error_invalid_template: "Invalid template, please check syntax\n{{.Error}}"
info_filename_strategy_set: "Filename strategy set to: {{.Strategy}}"
info_conflict_strategy_set: "Duplicate file strategy set to: {{.Strategy}}"
prompt_select_filename_strategy: "Please select filename strategy, current strategy: {{.Strategy}}"
prompt_select_conflict_strategy: "Please select duplicate file strategy, current strategy: {{.Strategy}}"
conflict_strategy_rename: "Always rename"
conflict_strategy_ask: "Ask every time"
conflict_strategy_overwrite: "Always overwrite"
conflict_strategy_skip: "Always skip"
fnametmpl_help: |-
Use this command to set filename template, for example:
/fnametmpl Image_{{"{{.msgid}}"}}_{{"{{.msgdate}}"}}.jpg
@@ -332,32 +351,56 @@ bot:
info_filename_prefix: "Filename: "
info_prompt_select_storage: "\nPlease select storage"
progress:
batch_start_prefix: "Starting batch download task\nTotal size: "
batch_processing_prefix: "Processing batch download task\nTotal size: "
batch_status_header: "<b>📦 Processing</b>\n\nFiles: <code>{{.Total}}</code> | Total size: <code>{{.TotalSize}}</code>\nStatus: ✅ <code>{{.Completed}}</code> | 📥 <code>{{.Downloaded}}</code> | ⏳ <code>{{.Waiting}}</code>\nTotal speed: ⬇️ <code>{{.DownloadSpeed}}</code> | ⬆️ <code>{{.UploadSpeed}}</code>"
batch_item_downloading: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} Downloading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_downloading_unknown: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} Downloading</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown</blockquote>"
batch_item_transferring: "<blockquote><b>↕️ {{.Index}}/{{.Total}} Transferring</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_transferring_unknown: "<blockquote><b>↕️ {{.Index}}/{{.Total}} Transferring</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown</blockquote>"
batch_item_uploading: "<blockquote><b>⬆️ {{.Index}}/{{.Total}} Uploading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_retrying: "<blockquote><b>🔁 {{.Index}}/{{.Total}} Retrying upload</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nRetry: <code>{{.Attempt}}/{{.Limit}}</code>\nSpeed before failure: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nReason: <code>{{.Reason}}</code></blockquote>"
batch_item_confirming: "<blockquote><b>⏳ {{.Index}}/{{.Total}} Waiting</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n<i>Uploaded, awaiting remote confirmation</i></blockquote>"
batch_summary_hidden_active: "🔄 <code>{{.Count}}</code> more files are active"
batch_summary_confirming: "☁️ Uploaded, awaiting group send: <code>{{.Count}}</code>"
batch_summary_failed: "❌ Failed: <code>{{.Count}}</code>"
batch_summary_skipped: "⏭️ Skipped: <code>{{.Count}}</code>"
batch_done: "<b>✅ Completed</b>\n\nFiles: <code>{{.Count}}</code>\nTotal size: <code>{{.Size}}</code>"
batch_done_with_skipped: "<b>⚠️ Completed</b>\n\nSucceeded: <code>{{.Success}}</code>\nSkipped: <code>{{.Skipped}}</code>\nTotal size: <code>{{.Size}}</code>"
batch_canceled: "<b>🚫 Task canceled</b>\n\nFiles: <code>{{.Total}}</code>\nCompleted: <code>{{.Completed}}</code>\nIncomplete: <code>{{.Incomplete}}</code>\nSkipped: <code>{{.Skipped}}</code>"
batch_failed_item: "<b>❌ Processing failed</b>\n\nFailed file: <code>{{.Index}}. {{.Name}}</code>\nStage: <code>{{.Stage}}</code>\nProgress: <code>{{.Progress}}</code>\nSpeed before failure: <code>{{.Speed}}</code>\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n❌ Failed: <code>{{.Failed}}</code>\n⏹ Incomplete: <code>{{.Incomplete}}</code>"
batch_failed_group: "<b>❌ Batch upload failed</b>\n\nAffected files: <code>{{.Affected}}</code>\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n❌ Batch failed: <code>{{.Failed}}</code>\n⏹ Incomplete: <code>{{.Incomplete}}</code>"
batch_failed_task: "<b>❌ Processing failed</b>\n\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n⏹ Incomplete: <code>{{.Incomplete}}</code>"
batch_failure_stage_download: "download"
batch_failure_stage_cache: "local cache"
batch_failure_stage_upload: "upload"
batch_failure_stage_confirm: "remote confirmation"
batch_failure_stage_batch_upload: "batch upload"
batch_failure_stage_internal: "internal task"
single_status_header: "<b>📦 Processing</b>"
single_downloading: "<blockquote><b>⬇️ Downloading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
single_downloading_unknown: "<blockquote><b>⬇️ Downloading</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown\nSave to: <code>{{.Destination}}</code></blockquote>"
single_uploading: "<blockquote><b>⬆️ Uploading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
single_upload_retrying: "<blockquote><b>🔁 Retrying upload</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nAttempt: <code>{{.Attempt}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
single_done: "<b>✅ Completed</b>\n\nFilename: <code>{{.Name}}</code>\nTotal size: <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code>"
single_canceled: "<b>🚫 Task canceled</b>\n\nFilename: <code>{{.Name}}</code>"
single_failed: "<b>❌ Processing failed</b>\n\nFilename: <code>{{.Name}}</code>\nReason: <code>{{.Reason}}</code>"
downloading_prefix: "Downloading\nTotal size: "
size_with_files: "{{.Size}} ({{.Count}} files)"
size_with_resources: "{{.Size}} ({{.Count}} resources)"
processing_list_prefix: "\nProcessing:\n"
processing_none: " - None"
avg_speed_prefix: "\nAverage speed: "
current_progress_prefix: "\nCurrent progress: "
task_canceled: "Task canceled"
task_canceled_with_id: "Processing canceled: {{.TaskID}}"
task_failed_with_error: "Processing failed: {{.Error}}"
batch_done_prefix: "Completed\nFile count: "
direct_done_prefix: "Completed, file count: "
parsed_start_prefix: "Starting download from {{.Site}}\nTotal size: "
parsed_done_prefix: "Completed, resource count: "
telegraph_start_prefix: "Starting Telegraph download\nImage count: "
telegraph_progress_prefix: "Downloading\nCurrent progress: "
telegraph_done_prefix: "Completed\nImage count: "
file_start_prefix: "Starting download\nFilename: "
file_processing_prefix: "Processing download task\nFilename: "
download_failed_prefix: "Download failed\nFilename: "
download_done_prefix: "Download completed\nFilename: "
file_size_prefix: "\nFile size: "
save_path_prefix: "\nSave path: "
total_size_prefix: "\nTotal size: "
direct_start: "Starting download, total size: {{.SizeMB}} MB ({{.Count}} files)"
file_name_prefix: "Filename: "
error_prefix: "\nError: "
aria2_start: "Waiting for Aria2 to complete download (GID: {{.GID}})..."
aria2_downloading: "Aria2 downloading (GID: {{.GID}})\n"
@@ -383,7 +426,7 @@ bot:
transfer_failed_files_prefix: "\nFailed files: "
syncpeers:
start: "Starting to sync peers..."
done: "Peer sync completed, total {{.Count}} chats synced"
success: "Peer sync completed, total {{.Count}} chats synced"
failed: "Peer sync failed: {{.Error}}"
aria2:
error_aria2_not_enabled: "Aria2 feature is not enabled in the configuration"

View File

@@ -113,9 +113,17 @@ bot:
error_task_add_failed: "任务添加失败: {{.Error}}"
info_task_added: "任务已添加"
info_batch_tasks_added: "已添加批量任务, 共 {{.Count}} 个文件"
info_batch_tasks_added_with_skipped: "已添加批量任务, 共 {{.Count}} 个文件\n已跳过同名文件:\n{{.Skipped}}"
info_all_conflict_files_skipped: "全部同名文件已跳过:\n{{.Skipped}}"
info_conflict_files_skipped: "已跳过同名文件:\n{{.Skipped}}"
error_task_create_failed: "任务创建失败: {{.Error}}"
error_get_dir_failed: "获取目录失败: {{.Error}}"
prompt_select_dir: "请选择要存储到的目录"
prompt_select_conflict_strategy: "检测到同名文件, 请选择保存策略:\n{{.Files}}"
prompt_conflict_more_files: "...还有 {{.Count}} 个文件"
button_conflict_rename: "重命名"
button_conflict_overwrite: "覆盖"
button_conflict_skip: "跳过"
prompt_select_default_dir: "请选择要保存到的默认文件夹"
info_default_storage_set: "已将默认存储位置设为: {{.Name}}"
info_default_storage_with_dir_set: "已将默认存储位置设为: {{.Name}}:/{{.Dir}}"
@@ -189,7 +197,11 @@ bot:
help_switch_suffix: " - 开关规则模式\n"
help_add_suffix: " <类型> <数据> <存储名> <路径> - 添加规则\n"
help_del_suffix: " <规则ID> - 删除规则\n"
help_preset_suffix: " <存储名> [基础路径] - 导入内置文件类型分类规则(视频/图片/音频/文档/压缩包)\n"
help_existing_rules_prefix: "\n当前已添加的规则:\n"
prompt_provide_storage_name: "请提供存储名称"
error_storage_not_found: "未找到存储: {{.Storage}}"
info_preset_imported: "已导入 {{.Count}} 条内置分类规则到存储 {{.Storage}}"
dir:
error_get_user_dirs_failed: "获取用户文件夹失败"
error_get_user_failed: "获取用户失败"
@@ -226,9 +238,9 @@ bot:
info_install_plugin_success: "插件安装成功: {{.Name}}"
parse:
info_parsing: "正在解析..."
error_parse_text_failed: "Failed to parse text: {{.Error}}"
error_build_storage_select_keyboard_failed: "Failed to build storage selection keyboard: {{.Error}}"
error_build_parsed_text_entity_failed: "Failed to build parsed text entity: {{.Error}}"
error_parse_text_failed: "解析文本失败: {{.Error}}"
error_build_storage_select_keyboard_failed: "构建存储选择键盘失败: {{.Error}}"
error_build_parsed_text_entity_failed: "构建解析文本实体失败: {{.Error}}"
info_link_prefix: "\n链接: "
info_author_prefix: "\n作者: "
info_description_prefix: "\n描述: "
@@ -267,10 +279,17 @@ bot:
config:
prompt_select_option: "请选择要配置的选项"
button_filename_strategy: "文件名策略"
button_conflict_strategy: "重名文件保存策略"
error_invalid_callback_data: "无效的回调数据"
error_invalid_template: "无效的模板, 请检查语法\n{{.Error}}"
info_filename_strategy_set: "已将文件名策略设置为: {{.Strategy}}"
info_conflict_strategy_set: "已将重名文件保存策略设置为: {{.Strategy}}"
prompt_select_filename_strategy: "请选择文件名策略, 当前策略: {{.Strategy}}"
prompt_select_conflict_strategy: "请选择重名文件保存策略, 当前策略: {{.Strategy}}"
conflict_strategy_rename: "始终重命名"
conflict_strategy_ask: "每次询问"
conflict_strategy_overwrite: "始终覆盖"
conflict_strategy_skip: "始终跳过"
fnametmpl_help: |-
使用该命令设置文件名模板, 示例:
/fnametmpl 图片_{{"{{.msgid}}"}}_{{"{{.msgdate}}"}}.jpg
@@ -333,32 +352,56 @@ bot:
info_filename_prefix: "文件名: "
info_prompt_select_storage: "\n请选择存储位置"
progress:
batch_start_prefix: "开始执行批量下载任务\n总大小: "
batch_processing_prefix: "正在处理批量下载任务\n总大小: "
batch_status_header: "<b>📦 正在处理</b>\n\n文件<code>{{.Total}}</code> 总大小:<code>{{.TotalSize}}</code>\n状态✅ <code>{{.Completed}}</code> 📥 <code>{{.Downloaded}}</code> ⏳ <code>{{.Waiting}}</code>\n总速度 <code>{{.DownloadSpeed}}</code> ⬆️ <code>{{.UploadSpeed}}</code>"
batch_item_downloading: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} 下载中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_downloading_unknown: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} 下载中</b>\n<code>{{.Name}}</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / 未知</blockquote>"
batch_item_transferring: "<blockquote><b>↕️ {{.Index}}/{{.Total}} 传输中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_transferring_unknown: "<blockquote><b>↕️ {{.Index}}/{{.Total}} 传输中</b>\n<code>{{.Name}}</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / 未知</blockquote>"
batch_item_uploading: "<blockquote><b>⬆️ {{.Index}}/{{.Total}} 上传中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
batch_item_retrying: "<blockquote><b>🔁 {{.Index}}/{{.Total}} 上传重试</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n重试次数<code>{{.Attempt}}/{{.Limit}}</code>\n失败前速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code>\n原因<code>{{.Reason}}</code></blockquote>"
batch_item_confirming: "<blockquote><b>⏳ {{.Index}}/{{.Total}} 等待中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n<i>文件已上传,正在等待远端确认</i></blockquote>"
batch_summary_hidden_active: "🔄 另有 <code>{{.Count}}</code> 个文件正在处理"
batch_summary_confirming: "☁️ 已上传,等待整组发送:<code>{{.Count}}</code>"
batch_summary_failed: "❌ 失败:<code>{{.Count}}</code>"
batch_summary_skipped: "⏭️ 已跳过:<code>{{.Count}}</code>"
batch_done: "<b>✅ 处理完成</b>\n\n文件数: <code>{{.Count}}</code>\n总大小: <code>{{.Size}}</code>"
batch_done_with_skipped: "<b>⚠️ 处理完成</b>\n\n成功: <code>{{.Success}}</code>\n已跳过: <code>{{.Skipped}}</code>\n总大小: <code>{{.Size}}</code>"
batch_canceled: "<b>🚫 任务已取消</b>\n\n文件数: <code>{{.Total}}</code>\n已完成: <code>{{.Completed}}</code>\n未完成: <code>{{.Incomplete}}</code>\n已跳过: <code>{{.Skipped}}</code>"
batch_failed_item: "<b>❌ 处理失败</b>\n\n失败文件: <code>{{.Index}}. {{.Name}}</code>\n失败阶段: <code>{{.Stage}}</code>\n失败进度: <code>{{.Progress}}</code>\n失败前速度: <code>{{.Speed}}</code>\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n❌ 失败: <code>{{.Failed}}</code>\n⏹ 未完成: <code>{{.Incomplete}}</code>"
batch_failed_group: "<b>❌ 批量上传失败</b>\n\n受影响文件: <code>{{.Affected}} 个</code>\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n❌ 批次失败: <code>{{.Failed}}</code>\n⏹ 未完成: <code>{{.Incomplete}}</code>"
batch_failed_task: "<b>❌ 处理失败</b>\n\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n⏹ 未完成: <code>{{.Incomplete}}</code>"
batch_failure_stage_download: "下载"
batch_failure_stage_cache: "本地缓存"
batch_failure_stage_upload: "上传"
batch_failure_stage_confirm: "云端确认"
batch_failure_stage_batch_upload: "批量上传"
batch_failure_stage_internal: "任务内部"
single_status_header: "<b>📦 正在处理</b>"
single_downloading: "<blockquote><b>⬇️ 下载中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至<code>{{.Destination}}</code></blockquote>"
single_downloading_unknown: "<blockquote><b>⬇️ 下载中</b>\n<code>{{.Name}}</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / 未知\n保存至<code>{{.Destination}}</code></blockquote>"
single_uploading: "<blockquote><b>⬆️ 上传中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至<code>{{.Destination}}</code></blockquote>"
single_upload_retrying: "<blockquote><b>🔁 上传重试</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n尝试次数<code>{{.Attempt}}</code>\n速度<code>{{.Speed}}</code>\n大小<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至<code>{{.Destination}}</code></blockquote>"
single_done: "<b>✅ 处理完成</b>\n\n文件名<code>{{.Name}}</code>\n总大小<code>{{.Size}}</code>\n保存至<code>{{.Destination}}</code>"
single_canceled: "<b>🚫 任务已取消</b>\n\n文件名<code>{{.Name}}</code>"
single_failed: "<b>❌ 处理失败</b>\n\n文件名<code>{{.Name}}</code>\n原因<code>{{.Reason}}</code>"
downloading_prefix: "正在下载\n总大小: "
size_with_files: "{{.Size}} ({{.Count}} 个文件)"
size_with_resources: "{{.Size}} ({{.Count}} 个资源)"
processing_list_prefix: "\n正在处理:\n"
processing_none: " - 无"
avg_speed_prefix: "\n平均速度: "
current_progress_prefix: "\n当前进度: "
task_canceled: "任务已取消"
task_canceled_with_id: "处理已取消: {{.TaskID}}"
task_failed_with_error: "处理失败: {{.Error}}"
batch_done_prefix: "处理完成\n文件数: "
direct_done_prefix: "处理完成, 文件数量: "
parsed_start_prefix: "开始下载 {{.Site}} 的资源\n总大小: "
parsed_done_prefix: "处理完成, 资源数量: "
telegraph_start_prefix: "开始下载Telegraph\n图片数量: "
telegraph_progress_prefix: "正在下载\n当前进度: "
telegraph_done_prefix: "处理完成\n图片数量: "
file_start_prefix: "开始下载\n文件名: "
file_processing_prefix: "正在处理下载任务\n文件名: "
download_failed_prefix: "下载失败\n文件名: "
download_done_prefix: "下载完成\n文件名: "
file_size_prefix: "\n文件大小: "
save_path_prefix: "\n保存路径: "
total_size_prefix: "\n总大小: "
direct_start: "开始下载, 总大小: {{.SizeMB}} MB ({{.Count}} 个文件)"
file_name_prefix: "文件名: "
error_prefix: "\n错误: "
aria2_start: "等待 Aria2 下载完成 (GID: {{.GID}})..."
aria2_downloading: "Aria2 正在下载 (GID: {{.GID}})\n"

View File

@@ -0,0 +1,42 @@
package fsutil_test
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
)
func TestCloseAndRemove(t *testing.T) {
tests := []struct {
name string
preClose bool
}{
{name: "open file"},
{name: "already closed file", preClose: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
filePath := filepath.Join(t.TempDir(), "cache-file")
file, err := fsutil.CreateFile(filePath)
if err != nil {
t.Fatalf("CreateFile() failed: %v", err)
}
if tt.preClose {
if err := file.Close(); err != nil {
t.Fatalf("Close() failed: %v", err)
}
}
if err := file.CloseAndRemove(); err != nil {
t.Fatalf("CloseAndRemove() failed: %v", err)
}
if _, err := os.Stat(filePath); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("cache file still exists after CloseAndRemove(): %v", err)
}
})
}
}

View File

@@ -1,6 +1,7 @@
package fsutil
import (
"errors"
"os"
"path/filepath"
"strings"
@@ -41,10 +42,11 @@ func (f *File) Remove() error {
}
func (f *File) CloseAndRemove() error {
if err := f.Close(); err != nil {
return err
closeErr := f.Close()
if errors.Is(closeErr, os.ErrClosed) {
closeErr = nil
}
return f.Remove()
return errors.Join(closeErr, f.Remove())
}
func CreateFile(fp string) (*File, error) {

View File

@@ -0,0 +1,27 @@
package fsutil
import (
"fmt"
"path"
"strings"
"github.com/rs/xid"
)
// UniquePath returns a non-taken path under basePath: the name itself, then
// numbered variants, then a random suffix.
func UniquePath(basePath, name string, exists func(candidate string) bool, maxAttempts int) string {
candidate := path.Join(basePath, name)
if !exists(candidate) {
return candidate
}
ext := path.Ext(name)
stem := strings.TrimSuffix(name, ext)
for i := 1; i <= maxAttempts; i++ {
candidate = path.Join(basePath, fmt.Sprintf("%s_%d%s", stem, i, ext))
if !exists(candidate) {
return candidate
}
}
return path.Join(basePath, fmt.Sprintf("%s_%s%s", stem, xid.New().String(), ext))
}

View File

@@ -17,7 +17,11 @@ type ProgressReadSeeker struct {
// Seek implements io.ReadSeeker.
func (pr *ProgressReadSeeker) Seek(offset int64, whence int) (int64, error) {
return pr.reader.Seek(offset, whence)
position, err := pr.reader.Seek(offset, whence)
if err == nil {
pr.read.Store(position)
}
return position, err
}
// NewProgressReader creates a new ProgressReader
@@ -54,7 +58,7 @@ func (pr *ProgressReadSeeker) Progress() float64 {
return float64(pr.read.Load()) / float64(pr.total.Load())
}
// Read returns the number of bytes read so far
// BytesRead returns the current tracked reader position.
func (pr *ProgressReadSeeker) BytesRead() int64 {
return pr.read.Load()
}

View File

@@ -0,0 +1,50 @@
package ioutil
import (
"bytes"
"io"
"testing"
)
func TestProgressReadSeekerTracksReads(t *testing.T) {
var gotRead, gotTotal int64
reader := NewProgressReader(bytes.NewReader([]byte("abcdef")), 6, func(read, total int64) {
gotRead = read
gotTotal = total
})
buffer := make([]byte, 4)
if _, err := io.ReadFull(reader, buffer); err != nil {
t.Fatalf("read failed: %v", err)
}
if gotRead != 4 || gotTotal != 6 {
t.Fatalf("progress = %d/%d, want 4/6", gotRead, gotTotal)
}
if reader.BytesRead() != 4 {
t.Fatalf("BytesRead() = %d, want 4", reader.BytesRead())
}
}
func TestProgressReadSeekerResetsPositionOnSeek(t *testing.T) {
reader := NewProgressReader(bytes.NewReader([]byte("abcdef")), 6, nil)
buffer := make([]byte, 4)
if _, err := io.ReadFull(reader, buffer); err != nil {
t.Fatalf("read failed: %v", err)
}
position, err := reader.Seek(0, io.SeekStart)
if err != nil {
t.Fatalf("seek failed: %v", err)
}
if position != 0 || reader.BytesRead() != 0 {
t.Fatalf("position after seek = %d (tracked %d), want 0", position, reader.BytesRead())
}
if _, err := io.ReadFull(reader, buffer[:2]); err != nil {
t.Fatalf("read after seek failed: %v", err)
}
if reader.BytesRead() != 2 {
t.Fatalf("BytesRead() after seek and read = %d, want 2", reader.BytesRead())
}
}

View File

@@ -0,0 +1,50 @@
// Package progressutil provides shared progress-update throttling for task
// progress trackers.
package progressutil
// updatesLevels picks the percent step by file size.
var updatesLevels = []struct {
size int64 // file size threshold
stepPercent int // minimum percent step between updates
}{
{10 << 20, 100},
{50 << 20, 20},
{200 << 20, 10},
{500 << 20, 5},
}
// ShouldUpdate reports whether a byte-based progress update should be shown.
func ShouldUpdate(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := updatesLevels[len(updatesLevels)-1].stepPercent
for _, lvl := range updatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}
// ShouldUpdateCount reports whether a count-based progress update (e.g. files
// downloaded so far) should be shown: every 10 units, or when finished.
func ShouldUpdateCount(downloaded, total int64) bool {
if total <= 0 || downloaded <= 0 {
return false
}
const step = int64(10)
if downloaded < step {
return downloaded == total
}
return downloaded%step == 0 || downloaded == total
}

View File

@@ -0,0 +1,35 @@
package tgutil
import (
stdhtml "html"
"github.com/gotd/td/telegram/message/entity"
messagehtml "github.com/gotd/td/telegram/message/html"
"github.com/gotd/td/telegram/message/styling"
"github.com/gotd/td/tg"
)
// EscapeHTMLTemplateData returns a copy of data with string values escaped for
// interpolation into Telegram HTML templates.
func EscapeHTMLTemplateData(data map[string]any) map[string]any {
escaped := make(map[string]any, len(data))
for key, value := range data {
if text, ok := value.(string); ok {
escaped[key] = stdhtml.EscapeString(text)
continue
}
escaped[key] = value
}
return escaped
}
// RenderHTML renders Telegram-compatible HTML into plain text and message
// entities.
func RenderHTML(markup string) (string, []tg.MessageEntityClass, error) {
var builder entity.Builder
if err := styling.Perform(&builder, messagehtml.String(nil, markup)); err != nil {
return "", nil, err
}
text, entities := builder.Complete()
return text, entities, nil
}

View File

@@ -0,0 +1,54 @@
package tgutil
import (
"testing"
"github.com/gotd/td/tg"
)
func TestEscapeHTMLTemplateDataDoesNotMutateInput(t *testing.T) {
input := map[string]any{
"Text": `<b>A&B</b>`,
"Count": 2,
}
escaped := EscapeHTMLTemplateData(input)
if got, want := escaped["Text"], "&lt;b&gt;A&amp;B&lt;/b&gt;"; got != want {
t.Fatalf("escaped text = %q, want %q", got, want)
}
if got := input["Text"]; got != `<b>A&B</b>` {
t.Fatalf("input was mutated: %q", got)
}
if got := escaped["Count"]; got != 2 {
t.Fatalf("non-string value = %v, want 2", got)
}
}
func TestRenderHTMLUsesTemplateStylesAndDecodesValues(t *testing.T) {
data := EscapeHTMLTemplateData(map[string]any{"Name": `<b>A&B</b>.bin`})
markup := `<blockquote><b>Uploading</b>
<code>` + data["Name"].(string) + `</code></blockquote>`
text, entities, err := RenderHTML(markup)
if err != nil {
t.Fatalf("RenderHTML() failed: %v", err)
}
if want := "Uploading\n<b>A&B</b>.bin"; text != want {
t.Fatalf("rendered text = %q, want %q", text, want)
}
var bold, code, blockquote int
for _, messageEntity := range entities {
switch messageEntity.(type) {
case *tg.MessageEntityBold:
bold++
case *tg.MessageEntityCode:
code++
case *tg.MessageEntityBlockquote:
blockquote++
}
}
if bold != 1 || code != 1 || blockquote != 1 {
t.Fatalf("entity counts = bold:%d code:%d blockquote:%d", bold, code, blockquote)
}
}

View File

@@ -2,6 +2,7 @@ package tgutil
import (
"fmt"
"sort"
"strconv"
"strings"
"unicode"
@@ -193,97 +194,6 @@ func getMessagesRange(ctx *ext.Context, chatID int64, minId, maxId int) ([]*tg.M
return result, nil
}
// [TODO]
// type MessageItem struct {
// Message *tg.Message
// Error error
// }
// func IterMessages(ctx *ext.Context, chatID int64, minId, maxId int) (<-chan MessageItem, error) {
// total := maxId - minId + 1
// ch := make(chan MessageItem, 100)
// go func() {
// defer close(ch)
// if !ctx.Self.Bot {
// perr := ctx.PeerStorage.GetInputPeerById(chatID)
// if perr == nil || perr.(*tg.InputPeerEmpty) != nil {
// ch <- MessageItem{
// Error: fmt.Errorf("peer not found: %d", chatID),
// }
// return
// }
// for i := 0; i < total; i += 100 {
// start := minId + i
// end := min(start+100, maxId)
// msgs, err := ctx.Raw.MessagesGetHistory(ctx, &tg.MessagesGetHistoryRequest{
// Peer: perr,
// OffsetID: start,
// AddOffset: start - end,
// Limit: 100,
// })
// if err != nil {
// ch <- MessageItem{
// Error: fmt.Errorf("failed to get messages: %w", err),
// }
// return
// }
// var msgClass []tg.MessageClass
// switch msgsv := msgs.(type) {
// case *tg.MessagesMessages:
// msgClass = msgsv.GetMessages()
// case *tg.MessagesMessagesSlice:
// msgClass = msgsv.GetMessages()
// case *tg.MessagesChannelMessages:
// msgClass = msgsv.GetMessages()
// default:
// ch <- MessageItem{
// Error: fmt.Errorf("unsupported message type: %T", msgsv),
// }
// continue
// }
// for _, msg := range msgClass {
// msg, ok := msg.AsNotEmpty()
// if !ok {
// continue
// }
// switch msg := msg.(type) {
// case *tg.Message:
// key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msg.GetID())
// cache.Set(key, msg)
// ch <- MessageItem{
// Message: msg,
// }
// }
// }
// }
// } else {
// for i := 0; i < total; i += 100 {
// start := minId + i
// end := min(start+100, maxId)
// msgs, err := GetMessagesRange(ctx, chatID, start, end)
// if err != nil {
// ch <- MessageItem{
// Error: fmt.Errorf("failed to get messages: %w", err),
// }
// return
// }
// for _, msg := range msgs {
// if msg == nil {
// continue
// }
// ch <- MessageItem{
// Message: msg,
// }
// }
// }
// }
// }()
// return ch, nil
// }
func getMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, error) {
key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msgID)
if msg, ok := cache.Get[*tg.Message](key); ok {
@@ -359,9 +269,16 @@ func GetGroupedMessages(ctx *ext.Context, chatID int64, msg *tg.Message) ([]*tg.
groupedMessages = append(groupedMessages, m)
}
}
sortMessagesByID(groupedMessages)
return groupedMessages, nil
}
func sortMessagesByID(messages []*tg.Message) {
sort.Slice(messages, func(i, j int) bool {
return messages[i].GetID() < messages[j].GetID()
})
}
func ExtractMessageEntityUrls(msg *tg.Message) []string {
if len(msg.Entities) == 0 {
return nil

View File

@@ -0,0 +1,18 @@
package tgutil
import (
"testing"
"github.com/gotd/td/tg"
)
func TestSortMessagesByID(t *testing.T) {
messages := []*tg.Message{{ID: 9}, {ID: 3}, {ID: 7}}
sortMessagesByID(messages)
want := []int{3, 7, 9}
for i := range messages {
if messages[i].GetID() != want[i] {
t.Fatalf("message %d has ID %d, want %d", i, messages[i].GetID(), want[i])
}
}
}

View File

@@ -33,6 +33,36 @@ secret = ""
# 转存完成后删除 Aria2 下载的本地文件
remove_after_transfer = true
# yt-dlp 视频下载配置
[ytdlp]
# 默认下载的最高视频清晰度 (按高度限制), 如 1080, 720, 480; 0 表示不限制 (下载最佳画质)
# 仅在使用 /ytdlp 命令且未手动指定任何参数时生效
max_height = 1080
# 直接指定 yt-dlp format 选择表达式, 留空则使用 max_height
# 设置后优先级高于 max_height, 例如: "bv*[height<=720]+ba/b"
format = ""
# 下载后转封装的视频容器格式, 留空则不转封装. 默认 mp4
recode = "mp4"
# 解析器配置
[parser]
# 启用 JS 解析器插件 (Go 内置解析器默认启用)
plugin_enable = false
# 插件目录, 可以是多个目录
plugin_dirs = ["./plugins"]
# 解析器默认代理
proxy = ""
# Twitter/X 解析器配置
[parser.twitter]
# 自定义 API 域名
api_domain = "api.fxtwitter.com"
# 单独为此解析器指定代理 (留空则使用 [parser] 中的 proxy)
# proxy = "http://127.0.0.1:7890"
# Kemono 解析器配置 (暂无可配置项, 留空即可)
[parser.kemono]
# HTTP API 配置
[api]
# 启用 HTTP API

View File

@@ -10,13 +10,4 @@ type hookExecConfig struct {
TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
// TaskTypes map[string]hookExecOnTypeConfig `toml:"task_types" mapstructure:"task_types" json:"task_types"` // [TODO]
}
// type hookExecOnTypeConfig struct {
// TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
// TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
// TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
// TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
// }

View File

@@ -8,14 +8,16 @@ import (
type TelegramStorageConfig struct {
BaseConfig
ChatID int64 `toml:"chat_id" mapstructure:"chat_id" json:"chat_id"`
ForceFile bool `toml:"force_file" mapstructure:"force_file" json:"force_file"`
RateLimit int `toml:"rate_limit" mapstructure:"rate_limit" json:"rate_limit"`
RateBurst int `toml:"rate_burst" mapstructure:"rate_burst" json:"rate_burst"`
SkipLarge bool `toml:"skip_large" mapstructure:"skip_large" json:"skip_large"` // skip files larger than Telegram limit(2GB)
// split files larger than Telegram limit(2GB) into parts of specified size, in MB, leave 0 to set default(2000MB)
ChatID int64 `toml:"chat_id" mapstructure:"chat_id" json:"chat_id"`
ForceFile bool `toml:"force_file" mapstructure:"force_file" json:"force_file"`
RateLimit int `toml:"rate_limit" mapstructure:"rate_limit" json:"rate_limit"`
RateBurst int `toml:"rate_burst" mapstructure:"rate_burst" json:"rate_burst"`
SkipLarge bool `toml:"skip_large" mapstructure:"skip_large" json:"skip_large"` // skip files larger than Telegram limit(2GB)
SplitLargeVideo bool `toml:"split_large_video" mapstructure:"split_large_video" json:"split_large_video"`
// split files larger than the uploader account limit into parts of specified size, in MB
// leave 0 to use the account limit (2000MB for bots/regular users, 4000MB for Premium users)
// only effective when SkipLarge is false
// use zip when splitting
// use zip when splitting non-video files or when lossless video splitting is disabled/unavailable
SplitSizeMB int64 `toml:"split_size_mb" mapstructure:"split_size_mb" json:"split_size_mb"`
}

View File

@@ -9,6 +9,7 @@ import (
"strings"
"time"
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/slice"
"github.com/krau/SaveAny-Bot/config/storage"
"github.com/spf13/viper"
@@ -35,6 +36,7 @@ type Config struct {
Storages []storage.StorageConfig `toml:"-" mapstructure:"-" json:"storages"`
Parser parserConfig `toml:"parser" mapstructure:"parser" json:"parser"`
Hook hookConfig `toml:"hook" mapstructure:"hook" json:"hook"`
Ytdlp YtdlpConfig `toml:"ytdlp" mapstructure:"ytdlp" json:"ytdlp"`
}
type aria2Config struct {
@@ -67,6 +69,13 @@ func (c Config) GetStorageByName(name string) storage.StorageConfig {
}
func Init(ctx context.Context, configFile ...string) error {
logger := log.FromContext(ctx)
// Reset side tables for re-init.
storages = nil
userIDs = nil
userStorages = make(map[int64][]string)
viper.SetConfigType("toml")
viper.SetEnvPrefix("SAVEANY")
viper.AutomaticEnv()
@@ -75,11 +84,13 @@ func Init(ctx context.Context, configFile ...string) error {
// 如果指定了配置文件路径,则使用指定的配置文件
// 配置文件支持传入一个 http(s) URL 地址
loadedFromURL := false
if len(configFile) > 0 && configFile[0] != "" {
cfg := configFile[0]
if strings.HasPrefix(cfg, "http://") || strings.HasPrefix(cfg, "https://") {
// 使用远程配置文件
resp, err := http.Get(cfg)
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Get(cfg)
if err != nil {
return fmt.Errorf("failed to fetch remote config file: %w", err)
}
@@ -90,6 +101,7 @@ func Init(ctx context.Context, configFile ...string) error {
if err := viper.ReadConfig(resp.Body); err != nil {
return fmt.Errorf("failed to read remote config file: %w", err)
}
loadedFromURL = true
} else {
viper.SetConfigFile(cfg)
}
@@ -131,19 +143,24 @@ func Init(ctx context.Context, configFile ...string) error {
"api.host": "0.0.0.0",
"api.port": 8080,
"api.token": "",
// yt-dlp
"ytdlp.recode": "mp4",
}
for key, value := range defaultConfigs {
viper.SetDefault(key, value)
}
if err := viper.ReadInConfig(); err != nil {
fmt.Println("Error reading config file, ", err)
return err
if !loadedFromURL {
if err := viper.ReadInConfig(); err != nil {
logger.Errorf("Error reading config file: %v", err)
return err
}
}
if err := viper.Unmarshal(cfg); err != nil {
fmt.Println("Error unmarshalling config file, ", err)
logger.Errorf("Error unmarshalling config file: %v", err)
return err
}

13
config/ytdlp.go Normal file
View File

@@ -0,0 +1,13 @@
package config
type YtdlpConfig struct {
// MaxHeight limits the video resolution by height in pixels (e.g. 1080, 720).
// 0 means no limit (best available). Ignored when Format is set.
MaxHeight int `toml:"max_height" mapstructure:"max_height" json:"max_height"`
// Format is a raw yt-dlp format selector (-f). When set, it takes precedence
// over MaxHeight and gives the user full control.
Format string `toml:"format" mapstructure:"format" json:"format"`
// Recode is the target video container yt-dlp recodes into (e.g. mp4).
// Empty disables recoding.
Recode string `toml:"recode" mapstructure:"recode" json:"recode"`
}

View File

@@ -3,14 +3,27 @@ package core
import (
"context"
"errors"
"sync"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/queue"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
)
var queueInstance *queue.TaskQueue[Executable]
var (
queueOnce sync.Once
queueInstance *queue.TaskQueue[Executable]
)
// initQueue lazily creates the shared task queue.
func initQueue() *queue.TaskQueue[Executable] {
queueOnce.Do(func() {
queueInstance = queue.NewTaskQueue[Executable]()
})
return queueInstance
}
type Executable interface {
Type() tasktype.TaskType
@@ -30,11 +43,14 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
break // queue closed and empty
}
exe := qtask.Data
taskCtx := qtask.Context()
logger.Infof("Processing task: %s", exe.TaskID())
if err := ExecCommandString(qtask.Context(), execHooks.TaskBeforeStart); err != nil {
taskevent.Emit(taskCtx, taskevent.Event{TaskID: exe.TaskID(), Phase: taskevent.PhaseStart})
if err := ExecCommandString(taskCtx, execHooks.TaskBeforeStart); err != nil {
logger.Errorf("Failed to execute before start hook for task %s: %v", exe.TaskID(), err)
}
if err := exe.Execute(qtask.Context()); err != nil {
err = exe.Execute(taskCtx)
if err != nil {
if errors.Is(err, context.Canceled) {
logger.Infof("Task %s was canceled", exe.TaskID())
if err := ExecCommandString(ctx, execHooks.TaskCancel); err != nil {
@@ -52,6 +68,7 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
logger.Errorf("Failed to execute success hook for task %s: %v", exe.TaskID(), err)
}
}
taskevent.Emit(taskCtx, taskevent.Event{TaskID: exe.TaskID(), Phase: taskevent.PhaseDone, Err: err})
qe.Done(qtask.ID)
<-semaphore
}
@@ -60,17 +77,22 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
func Run(ctx context.Context) {
log.FromContext(ctx).Info("Start processing tasks...")
semaphore := make(chan struct{}, config.C().Workers)
if queueInstance == nil {
queueInstance = queue.NewTaskQueue[Executable]()
}
q := initQueue()
for range config.C().Workers {
go worker(ctx, queueInstance, semaphore)
go worker(ctx, q, semaphore)
}
}
// Close stops the queue and unblocks workers in Get.
func Close() {
if q := initQueue(); q != nil {
q.Close()
}
}
func AddTask(ctx context.Context, task Executable) error {
return queueInstance.Add(queue.NewTask(ctx, task.TaskID(), task.Title(), task))
return initQueue().Add(queue.NewTask(ctx, task.TaskID(), task.Title(), task))
}
func CancelTask(ctx context.Context, id string) error {

View File

@@ -6,12 +6,14 @@ import (
"fmt"
"os"
"path/filepath"
"strconv"
"time"
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/aria2"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
)
// Execute implements core.Executable.
@@ -77,6 +79,12 @@ func (t *Task) waitForDownload(ctx context.Context) error {
if t.Progress != nil {
t.Progress.OnProgress(ctx, t, status)
}
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: parseInt64(status.TotalLength),
DownloadedBytes: parseInt64(status.CompletedLength),
})
// Check if download is complete
if status.IsDownloadComplete() {
@@ -248,3 +256,16 @@ func (t *Task) cancelAria2Download() {
logger.Debugf("Failed to remove download result for %s: %v", t.GID, err)
}
}
// parseInt64 parses an aria2 status string (decimal bytes) into int64,
// returning 0 on failure so it can be used directly in progress events.
func parseInt64(s string) int64 {
if s == "" {
return 0
}
n, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return 0
}
return n
}

View File

@@ -2,10 +2,13 @@ package batchtfile
import (
"context"
"errors"
"fmt"
"io"
"os"
"path"
"sync"
"time"
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/retry"
@@ -14,44 +17,318 @@ import (
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"github.com/krau/SaveAny-Bot/storage"
"golang.org/x/sync/errgroup"
)
type executionGroup struct {
elems []*TaskElement
batchSaver storage.StorageBatchSaver
}
func (g executionGroup) usesBatchSaver() bool {
return g.batchSaver != nil
}
func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("batch_file[%s]", t.ID))
logger.Info("Starting batch file task")
t.Progress.OnStart(ctx, t)
workers := config.C().Workers
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(workers)
for _, elem := range t.elems {
eg.Go(func() error {
t.processingMu.RLock()
if t.processing[elem.ID] != nil {
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
}
t.processingMu.RUnlock()
t.processingMu.Lock()
t.processing[elem.ID] = &elem
t.processingMu.Unlock()
defer func() {
t.processingMu.Lock()
delete(t.processing, elem.ID)
t.processingMu.Unlock()
}()
return t.processElement(gctx, elem)
})
if t.Progress != nil {
t.Progress.OnStart(ctx, t)
}
groups := t.executionGroups()
var err error
for i := 0; i < len(groups); {
if groups[i].usesBatchSaver() {
err = t.processBatch(ctx, groups[i])
i++
} else {
end := i + 1
for end < len(groups) && !groups[end].usesBatchSaver() {
end++
}
elems := make([]*TaskElement, 0, end-i)
for _, group := range groups[i:end] {
elems = append(elems, group.elems...)
}
err = t.processElements(ctx, elems)
i = end
}
if err != nil {
if !t.IgnoreErrors || errors.Is(err, context.Canceled) {
break
}
logger.Warnf("Group processing failed (ignored): %v", err)
err = nil
}
}
err := eg.Wait()
if err != nil {
logger.Errorf("Error during batch file processing: %v", err)
} else {
logger.Info("Batch file task completed successfully")
}
t.Progress.OnDone(ctx, t, err)
t.finishItems(err)
if t.Progress != nil {
t.Progress.OnDone(ctx, t, err)
}
return err
}
// notifyProgress reports a progress update to the optional tracker.
func (t *Task) notifyProgress(ctx context.Context) {
if t.Progress != nil {
t.Progress.OnProgress(ctx, t)
}
}
func (t *Task) executionGroups() []executionGroup {
groups := make([]executionGroup, 0, len(t.elems))
for i := 0; i < len(t.elems); {
elem := &t.elems[i]
batchSaver, batchCapable := elem.Storage.(storage.StorageBatchSaver)
if !batchCapable || elem.sourceGroupKey == "" {
groups = append(groups, executionGroup{elems: []*TaskElement{elem}})
i++
continue
}
end := i + 1
for end < len(t.elems) {
next := &t.elems[end]
if next.Storage != elem.Storage || next.sourceGroupKey != elem.sourceGroupKey {
break
}
end++
}
elems := make([]*TaskElement, 0, end-i)
for j := i; j < end; j++ {
elems = append(elems, &t.elems[j])
}
groups = append(groups, executionGroup{elems: elems, batchSaver: batchSaver})
i = end
}
return groups
}
func (t *Task) processElements(ctx context.Context, elems []*TaskElement) error {
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(config.C().Workers)
for _, elem := range elems {
eg.Go(func() error {
if err := t.markProcessing(ctx, elem); err != nil {
return err
}
defer t.unmarkProcessing(elem.ID)
err := t.processElement(gctx, *elem)
if err != nil && t.IgnoreErrors && !errors.Is(err, context.Canceled) {
// Per-item failure: keep siblings running.
log.FromContext(ctx).Warnf("Element %s failed (ignored): %v", elem.ID, err)
return nil
}
return err
})
}
return eg.Wait()
}
func (t *Task) processBatch(ctx context.Context, group executionGroup) error {
defer func() {
for _, elem := range group.elems {
if err := os.Remove(elem.localPath); err != nil && !os.IsNotExist(err) {
log.FromContext(ctx).Warnf("Failed to cleanup batch cache file %s: %v", elem.localPath, err)
}
}
}()
type downloadResult struct {
elem *TaskElement
err error
}
results := make([]downloadResult, len(group.elems))
var resultsMu sync.Mutex
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(config.C().Workers)
for i, elem := range group.elems {
eg.Go(func() error {
if err := t.markProcessing(ctx, elem); err != nil {
return err
}
defer t.unmarkProcessing(elem.ID)
err := t.downloadElement(gctx, elem)
// Store by original index.
resultsMu.Lock()
results[i] = downloadResult{elem: elem, err: err}
resultsMu.Unlock()
if err != nil && t.IgnoreErrors && !errors.Is(err, context.Canceled) {
// Per-item failure: keep siblings running.
log.FromContext(ctx).Warnf("Element %s failed (ignored): %v", elem.ID, err)
return nil
}
return err
})
}
if err := eg.Wait(); err != nil {
return err
}
// Upload only successfully downloaded elements.
successElems := make([]*TaskElement, 0, len(group.elems))
for _, r := range results {
if r.err == nil {
successElems = append(successElems, r.elem)
}
}
if len(successElems) == 0 {
return fmt.Errorf("all elements failed to download")
}
items := make([]storagetypes.BatchItem, 0, len(successElems))
openFiles := make([]*os.File, 0, len(successElems))
defer func() {
for _, file := range openFiles {
if err := file.Close(); err != nil {
log.FromContext(ctx).Warnf("Failed to close batch cache file %s: %v", file.Name(), err)
}
}
}()
for _, elem := range successElems {
file, err := os.Open(elem.localPath)
if err != nil {
t.markItemFailed(elem.ID, FailureStageCache, err)
return fmt.Errorf("failed to open cache file: %w", err)
}
stat, err := file.Stat()
if err != nil {
file.Close()
t.markItemFailed(elem.ID, FailureStageCache, err)
return fmt.Errorf("failed to get cache file stat: %w", err)
}
openFiles = append(openFiles, file)
items = append(items, storagetypes.BatchItem{
Reader: file,
StoragePath: elem.Path,
Size: stat.Size(),
SourceGroupKey: elem.sourceGroupKey,
Caption: elem.sourceCaption,
PreserveCaption: elem.preserveCaption,
})
}
for index, item := range items {
t.recordDownloadComplete(successElems[index].ID, item.Size)
}
return t.saveBatchItems(ctx, successElems, items)
}
func (t *Task) saveBatchItems(ctx context.Context, successElems []*TaskElement, items []storagetypes.BatchItem) error {
t.startUpload(ctx)
if progressSaver, ok := successElems[0].Storage.(storage.StorageBatchProgressSaver); ok {
err := progressSaver.SaveBatchWithProgress(ctx, items, func(index int, uploaded, total int64) {
if index < 0 || index >= len(successElems) {
return
}
t.uploadCallback(ctx, successElems[index].ID)(uploaded, total)
})
if err != nil {
for _, elem := range successElems {
t.markItemFailed(elem.ID, FailureStageBatchUpload, err)
}
t.notifyStateChange(ctx)
return fmt.Errorf("failed to save batch: %w", err)
}
for index, elem := range successElems {
t.uploadCallback(ctx, elem.ID)(items[index].Size, items[index].Size)
t.markItemCompleted(elem.ID)
}
t.notifyStateChange(ctx)
return nil
}
for i := range items {
items[i].Reader = ioutil.NewProgressReader(
items[i].Reader,
items[i].Size,
t.uploadCallback(ctx, successElems[i].ID),
)
}
if err := successElems[0].Storage.(storage.StorageBatchSaver).SaveBatch(ctx, items); err != nil {
for _, elem := range successElems {
t.markItemFailed(elem.ID, FailureStageBatchUpload, err)
}
t.notifyStateChange(ctx)
return fmt.Errorf("failed to save batch: %w", err)
}
for index, elem := range successElems {
t.uploadCallback(ctx, elem.ID)(items[index].Size, items[index].Size)
t.markItemCompleted(elem.ID)
}
t.notifyStateChange(ctx)
return nil
}
func (t *Task) markProcessing(ctx context.Context, elem *TaskElement) error {
t.processingMu.Lock()
if t.processing[elem.ID] != nil {
t.processingMu.Unlock()
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
}
t.processing[elem.ID] = elem
t.processingMu.Unlock()
t.markItemActive(elem.ID, elem.stream, time.Now())
t.notifyProgress(ctx)
return nil
}
func (t *Task) unmarkProcessing(id string) {
t.processingMu.Lock()
delete(t.processing, id)
t.processingMu.Unlock()
}
func (t *Task) downloadElement(ctx context.Context, elem *TaskElement) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", elem.File.Name()))
logger.Info("Starting file download")
localFile, err := fsutil.CreateFile(elem.localPath)
if err != nil {
t.markItemFailed(elem.ID, FailureStageCache, err)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to create local file: %w", err)
}
wrAt := ioutil.NewProgressWriterAt(localFile, func(n int) {
t.recordItemDownload(elem.ID, int64(n), time.Now())
downloaded := t.downloaded.Add(int64(n))
t.notifyProgress(ctx)
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalSize,
DownloadedBytes: downloaded,
})
})
_, downloadErr := tdler.NewDownloader(elem.File).Parallel(ctx, wrAt)
closeErr := localFile.Close()
if downloadErr != nil {
t.markItemFailed(elem.ID, FailureStageDownload, downloadErr)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to download file: %w", downloadErr)
}
if closeErr != nil {
t.markItemFailed(elem.ID, FailureStageCache, closeErr)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to close cache file: %w", closeErr)
}
logger.Info("File downloaded successfully")
if path.Ext(elem.FileName()) == "" {
if ext := fsutil.DetectFileExt(elem.localPath); ext != "" {
elem.Path += ext
}
}
t.markItemDownloaded(elem.ID)
t.notifyProgress(ctx)
return nil
}
func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", elem.File.Name()))
if elem.stream {
@@ -59,11 +336,23 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
defer pr.Close()
errg, uploadCtx := errgroup.WithContext(ctx)
errg.Go(func() error {
return elem.Storage.Save(uploadCtx, pr, elem.Path)
err := elem.Storage.Save(uploadCtx, pr, elem.Path)
if err != nil {
t.markItemFailed(elem.ID, FailureStageUpload, err)
t.notifyStateChange(ctx)
}
return err
})
wr := ioutil.NewProgressWriter(pw, func(n int) {
t.downloaded.Add(int64(n))
t.Progress.OnProgress(ctx, t)
t.recordItemDownload(elem.ID, int64(n), time.Now())
downloaded := t.downloaded.Add(int64(n))
t.notifyProgress(ctx)
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalSize,
DownloadedBytes: downloaded,
})
})
errg.Go(func() error {
defer pw.Close()
@@ -71,6 +360,8 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
_, err := tdler.NewDownloader(elem.File).Stream(uploadCtx, wr)
if err != nil {
logger.Errorf("Failed to download file: %v", err)
t.markItemFailed(elem.ID, FailureStageDownload, err)
t.notifyStateChange(ctx)
pw.CloseWithError(err)
}
return err
@@ -78,12 +369,22 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
if err := errg.Wait(); err != nil {
return fmt.Errorf("failed to download file in stream mode: %w", err)
}
// Streamed bytes are the uploaded bytes.
var streamedBytes int64
t.updateItem(elem.ID, func(item *itemProgressState) {
streamedBytes = item.downloaded
})
t.recordDownloadComplete(elem.ID, streamedBytes)
t.markItemCompleted(elem.ID)
t.notifyStateChange(ctx)
logger.Info("File downloaded successfully in stream mode")
return nil
}
logger.Info("Starting file download")
localFile, err := fsutil.CreateFile(elem.localPath)
if err != nil {
t.markItemFailed(elem.ID, FailureStageCache, err)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to create local file: %w", err)
}
defer func() {
@@ -92,11 +393,20 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
}
}()
wrAt := ioutil.NewProgressWriterAt(localFile, func(n int) {
t.downloaded.Add(int64(n))
t.Progress.OnProgress(ctx, t)
t.recordItemDownload(elem.ID, int64(n), time.Now())
downloaded := t.downloaded.Add(int64(n))
t.notifyProgress(ctx)
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalSize,
DownloadedBytes: downloaded,
})
})
_, err = tdler.NewDownloader(elem.File).Parallel(ctx, wrAt)
if err != nil {
t.markItemFailed(elem.ID, FailureStageDownload, err)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to download file: %w", err)
}
logger.Info("File downloaded successfully")
@@ -109,21 +419,50 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
var fileStat os.FileInfo
fileStat, err = os.Stat(elem.localPath)
if err != nil {
t.markItemFailed(elem.ID, FailureStageCache, err)
t.notifyStateChange(ctx)
return fmt.Errorf("failed to get file stat: %w", err)
}
t.recordDownloadComplete(elem.ID, fileStat.Size())
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
t.startUpload(vctx)
onProgress := t.uploadCallback(vctx, elem.ID)
attempt := 0
retryLimit := int(config.C().Retry)
lastFailureStage := FailureStageUpload
err = retry.Retry(func() error {
attempt++
var file *os.File
file, err = os.Open(elem.localPath)
if err != nil {
lastFailureStage = FailureStageCache
t.markItemRetry(elem.ID, lastFailureStage, attempt, retryLimit, err)
t.notifyStateChange(vctx)
return fmt.Errorf("failed to open cache file: %w", err)
}
defer file.Close()
if err = elem.Storage.Save(vctx, file, elem.Path); err != nil {
onProgress(0, fileStat.Size())
if progressSaver, ok := elem.Storage.(storage.StorageProgressSaver); ok {
err = progressSaver.SaveWithProgress(vctx, file, elem.Path, onProgress)
} else {
err = elem.Storage.Save(vctx, ioutil.NewProgressReader(file, fileStat.Size(), onProgress), elem.Path)
}
if err != nil {
logger.Errorf("Failed to save file: %s, retrying...", err)
lastFailureStage = t.itemFailureStage(elem.ID)
t.markItemRetry(elem.ID, lastFailureStage, attempt, retryLimit, err)
t.notifyStateChange(vctx)
return err
}
return nil
}, retry.Context(vctx), retry.RetryTimes(uint(config.C().Retry)))
if err == nil {
onProgress(fileStat.Size(), fileStat.Size())
t.markItemCompleted(elem.ID)
t.notifyStateChange(vctx)
} else {
t.markItemFailed(elem.ID, lastFailureStage, err)
t.notifyStateChange(vctx)
}
return err
}

View File

@@ -0,0 +1,57 @@
package batchtfile
import (
"testing"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/pkg/tfile"
tgstorage "github.com/krau/SaveAny-Bot/storage/telegram"
)
func TestExecutionGroupsPreserveSourceAlbums(t *testing.T) {
stor := new(tgstorage.Telegram)
otherStor := new(tgstorage.Telegram)
task := Task{elems: []TaskElement{
{Storage: stor, sourceGroupKey: "album-1"},
{Storage: stor, sourceGroupKey: "album-1"},
{Storage: stor},
{Storage: stor, sourceGroupKey: "album-2"},
{Storage: stor, sourceGroupKey: "album-2"},
{Storage: otherStor, sourceGroupKey: "album-2"},
}}
groups := task.executionGroups()
wantSizes := []int{2, 1, 2, 1}
wantBatch := []bool{true, false, true, true}
if len(groups) != len(wantSizes) {
t.Fatalf("got %d groups, want %d", len(groups), len(wantSizes))
}
for i := range groups {
if got := len(groups[i].elems); got != wantSizes[i] {
t.Errorf("group %d has %d elements, want %d", i, got, wantSizes[i])
}
if got := groups[i].usesBatchSaver(); got != wantBatch[i] {
t.Errorf("group %d batch=%v, want %v", i, got, wantBatch[i])
}
}
}
func TestSourceMetadataPreservesAlbumIdentityAndCaption(t *testing.T) {
msg := &tg.Message{
PeerID: &tg.PeerChannel{ChannelID: 77},
Message: "original caption",
}
msg.SetGroupedID(42)
file := tfile.NewTGFile(nil, nil, 0, "photo.jpg", tfile.WithMessage(msg))
groupKey, caption, preserveCaption := sourceMetadata(file)
if groupKey != "*tg.PeerChannel:77:42" {
t.Fatalf("group key = %q, want %q", groupKey, "*tg.PeerChannel:77:42")
}
if caption != "original caption" {
t.Fatalf("caption = %q, want original caption", caption)
}
if !preserveCaption {
t.Fatal("preserveCaption = false, want true")
}
}

View File

@@ -0,0 +1,360 @@
package batchtfile
import (
"context"
"errors"
"strings"
"time"
)
const (
transferSpeedWindow = 5 * time.Second
transferSamplePeriod = 250 * time.Millisecond
)
// ItemPhase describes the current lifecycle stage of one batch item.
type ItemPhase uint8
const (
ItemPhaseWaiting ItemPhase = iota
ItemPhaseDownloading
ItemPhaseTransferring
ItemPhaseDownloaded
ItemPhaseUploading
ItemPhaseRetrying
ItemPhaseConfirming
ItemPhaseCompleted
ItemPhaseFailed
ItemPhaseStopped
)
// FailureStage identifies the operation that failed for one batch item.
type FailureStage uint8
const (
FailureStageNone FailureStage = iota
FailureStageDownload
FailureStageCache
FailureStageUpload
FailureStageConfirm
FailureStageBatchUpload
FailureStageInternal
)
// TaskItemProgress is an immutable progress snapshot for one batch item.
type TaskItemProgress struct {
Index int
ID string
Name string
Size int64
Downloaded int64
Uploaded int64
DownloadSpeed float64
UploadSpeed float64
Phase ItemPhase
FailureStage FailureStage
RetryAttempt int
RetryLimit int
Error string
}
type transferSample struct {
at time.Time
bytes int64
}
type transferMeter struct {
samples []transferSample
latest transferSample
hasData bool
}
func (m *transferMeter) record(now time.Time, transferred int64) {
if m.hasData && transferred < m.latest.bytes {
m.reset()
}
if m.hasData && now.Before(m.latest.at) {
now = m.latest.at
}
sample := transferSample{at: now, bytes: transferred}
m.latest = sample
m.hasData = true
if len(m.samples) == 0 {
m.samples = append(m.samples, sample)
return
}
if now.Sub(m.samples[len(m.samples)-1].at) >= transferSamplePeriod {
m.samples = append(m.samples, sample)
}
cutoff := now.Add(-transferSpeedWindow)
for len(m.samples) > 1 && m.samples[0].at.Before(cutoff) {
m.samples = m.samples[1:]
}
}
func (m *transferMeter) speed() float64 {
if !m.hasData || len(m.samples) == 0 {
return 0
}
first := m.samples[0]
last := m.latest
elapsed := last.at.Sub(first.at).Seconds()
if elapsed <= 0 || last.bytes <= first.bytes {
return 0
}
return float64(last.bytes-first.bytes) / elapsed
}
func (m *transferMeter) reset() {
m.samples = m.samples[:0]
m.latest = transferSample{}
m.hasData = false
}
type itemProgressState struct {
index int
id string
name string
expectedSize int64
actualSize int64
downloaded int64
uploaded int64
phase ItemPhase
failureStage FailureStage
retryAttempt int
retryLimit int
err string
downloadMeter transferMeter
uploadMeter transferMeter
}
func newItemProgressStates(elems []TaskElement) ([]itemProgressState, map[string]int) {
states := make([]itemProgressState, 0, len(elems))
index := make(map[string]int, len(elems))
for i, elem := range elems {
name := ""
size := int64(0)
if elem.File != nil {
name = elem.File.Name()
size = elem.File.Size()
}
states = append(states, itemProgressState{
index: i + 1,
id: elem.ID,
name: name,
expectedSize: size,
phase: ItemPhaseWaiting,
})
index[elem.ID] = i
}
return states, index
}
func (t *Task) updateItem(id string, update func(*itemProgressState)) bool {
t.itemMu.Lock()
defer t.itemMu.Unlock()
index, ok := t.itemIndex[id]
if !ok || index < 0 || index >= len(t.itemStates) {
return false
}
update(&t.itemStates[index])
return true
}
func (t *Task) markItemActive(id string, stream bool, now time.Time) {
t.updateItem(id, func(item *itemProgressState) {
if stream {
item.phase = ItemPhaseTransferring
} else {
item.phase = ItemPhaseDownloading
}
item.failureStage = FailureStageNone
item.err = ""
item.downloadMeter.record(now, item.downloaded)
if stream {
item.uploadMeter.record(now, item.uploaded)
}
})
}
func (t *Task) recordItemDownload(id string, n int64, now time.Time) {
if n <= 0 {
return
}
t.updateItem(id, func(item *itemProgressState) {
item.downloaded += n
item.downloadMeter.record(now, item.downloaded)
if item.phase == ItemPhaseTransferring {
item.uploaded += n
item.uploadMeter.record(now, item.uploaded)
}
})
}
func (t *Task) markItemDownloaded(id string) {
t.updateItem(id, func(item *itemProgressState) {
item.phase = ItemPhaseDownloaded
})
}
func (t *Task) recordItemDownloaded(id string, actualSize int64) {
t.updateItem(id, func(item *itemProgressState) {
if actualSize > 0 {
item.actualSize = actualSize
}
if item.actualSize == 0 {
item.actualSize = item.downloaded
}
if item.phase != ItemPhaseTransferring {
item.phase = ItemPhaseDownloaded
item.uploadMeter.reset()
}
})
}
func (t *Task) recordItemUpload(id string, uploaded, total int64, now time.Time) bool {
becameConfirming := false
t.updateItem(id, func(item *itemProgressState) {
if uploaded < item.uploaded {
if item.phase != ItemPhaseRetrying {
return
}
item.uploadMeter.reset()
}
if total > 0 {
item.actualSize = total
}
item.uploaded = uploaded
item.uploadMeter.record(now, uploaded)
if total > 0 && uploaded >= total {
becameConfirming = item.phase != ItemPhaseConfirming
item.phase = ItemPhaseConfirming
return
}
item.phase = ItemPhaseUploading
item.failureStage = FailureStageNone
item.err = ""
})
return becameConfirming
}
func (t *Task) markItemRetry(id string, stage FailureStage, attempt, limit int, err error) {
t.updateItem(id, func(item *itemProgressState) {
item.phase = ItemPhaseRetrying
item.failureStage = stage
item.retryAttempt = attempt
item.retryLimit = limit
item.err = compactError(err)
})
}
func (t *Task) markItemFailed(id string, stage FailureStage, err error) {
t.updateItem(id, func(item *itemProgressState) {
if item.phase == ItemPhaseFailed || item.phase == ItemPhaseCompleted {
return
}
if errors.Is(err, context.Canceled) {
item.phase = ItemPhaseStopped
return
}
item.phase = ItemPhaseFailed
item.failureStage = stage
item.err = compactError(err)
})
}
func (t *Task) markItemCompleted(id string) {
t.updateItem(id, func(item *itemProgressState) {
item.phase = ItemPhaseCompleted
item.failureStage = FailureStageNone
item.err = ""
item.retryAttempt = 0
item.retryLimit = 0
if item.actualSize == 0 {
item.actualSize = max(item.downloaded, item.uploaded)
}
})
}
func (t *Task) finishItems(err error) {
if err == nil {
return
}
t.itemMu.Lock()
defer t.itemMu.Unlock()
for i := range t.itemStates {
item := &t.itemStates[i]
if item.phase == ItemPhaseCompleted || item.phase == ItemPhaseFailed {
continue
}
item.phase = ItemPhaseStopped
}
}
func (t *Task) itemFailureStage(id string) FailureStage {
t.itemMu.RLock()
defer t.itemMu.RUnlock()
index, ok := t.itemIndex[id]
if !ok || index < 0 || index >= len(t.itemStates) {
return FailureStageUpload
}
if t.itemStates[index].phase == ItemPhaseConfirming {
return FailureStageConfirm
}
return FailureStageUpload
}
func (t *Task) Items() []TaskItemProgress {
t.itemMu.RLock()
defer t.itemMu.RUnlock()
items := make([]TaskItemProgress, 0, len(t.itemStates))
for i := range t.itemStates {
item := &t.itemStates[i]
size := item.actualSize
if size == 0 {
size = item.expectedSize
}
items = append(items, TaskItemProgress{
Index: item.index,
ID: item.id,
Name: item.name,
Size: size,
Downloaded: item.downloaded,
Uploaded: item.uploaded,
DownloadSpeed: item.downloadMeter.speed(),
UploadSpeed: item.uploadMeter.speed(),
Phase: item.phase,
FailureStage: item.failureStage,
RetryAttempt: item.retryAttempt,
RetryLimit: item.retryLimit,
Error: item.err,
})
}
return items
}
func (t *Task) ActualTotalSize() int64 {
items := t.Items()
var total int64
for _, item := range items {
total += item.Size
}
return total
}
type stateProgressTracker interface {
OnStateChange(ctx context.Context, info TaskInfo)
}
func (t *Task) notifyStateChange(ctx context.Context) {
if tracker, ok := t.Progress.(stateProgressTracker); ok {
tracker.OnStateChange(ctx, t)
}
}
func compactError(err error) string {
if err == nil {
return ""
}
return strings.Join(strings.Fields(err.Error()), " ")
}

View File

@@ -0,0 +1,35 @@
package batchtfile
import (
"context"
"sync/atomic"
"testing"
)
type recordingTracker struct {
calls atomic.Int64
}
func (r *recordingTracker) OnStart(context.Context, TaskInfo) {}
func (r *recordingTracker) OnDone(context.Context, TaskInfo, error) {
}
func (r *recordingTracker) OnProgress(context.Context, TaskInfo) {
r.calls.Add(1)
}
// Regression: notifyProgress must call the tracker, not itself. The previous
// self-call recursed until stack overflow on any batch task with a tracker.
func TestNotifyProgressCallsTracker(t *testing.T) {
tracker := &recordingTracker{}
task := &Task{Progress: tracker}
task.notifyProgress(t.Context())
if tracker.calls.Load() != 1 {
t.Fatalf("expected 1 tracker call, got %d", tracker.calls.Load())
}
}
// The nil tracker path must stay a no-op.
func TestNotifyProgressNilTracker(t *testing.T) {
task := &Task{}
task.notifyProgress(t.Context()) // must not panic
}

View File

@@ -4,19 +4,19 @@ import (
"context"
"errors"
"fmt"
"strconv"
"sync/atomic"
"path"
"strings"
"sync"
"time"
"unicode/utf8"
"github.com/charmbracelet/log"
"github.com/duke-git/lancet/v2/slice"
"github.com/gotd/td/telegram/message/entity"
"github.com/gotd/td/telegram/message/styling"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/config"
)
type ProgressTracker interface {
@@ -26,155 +26,485 @@ type ProgressTracker interface {
}
type Progress struct {
MessageID int
ChatID int64
start time.Time
lastUpdatePercent atomic.Int32
MessageID int
ChatID int64
updateMu sync.Mutex
lastUpdateAt time.Time
lastText string
done bool
skippedFiles []string
}
type renderedBatchMessage struct {
Text string
Entities []tg.MessageEntityClass
Err error
}
const (
progressRenderInterval = time.Second
maxVisibleActiveItems = 5
progressBarWidth = 10
maxDisplayNameRunes = 36
maxDisplayErrorRunes = 240
)
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
p.start = time.Now()
p.lastUpdatePercent.Store(0)
log.FromContext(ctx).Debugf("Batch task progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchStartPrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalSize())/(1024*1024), info.Count())),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
return
}
p.render(ctx, info, true)
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.TotalSize(), info.Downloaded(), int(p.lastUpdatePercent.Load())) {
p.render(ctx, info, false)
}
func (p *Progress) OnStateChange(ctx context.Context, info TaskInfo) {
p.render(ctx, info, true)
}
func (p *Progress) OnUploadStart(ctx context.Context, info TaskInfo, _ int64) {
p.render(ctx, info, true)
}
func (p *Progress) OnUploadProgress(ctx context.Context, info TaskInfo, _, _ int64) {
p.render(ctx, info, false)
}
func (p *Progress) render(ctx context.Context, info TaskInfo, priority bool) {
p.updateMu.Lock()
defer p.updateMu.Unlock()
if p.done {
return
}
percent := int((info.Downloaded() * 100) / info.TotalSize())
if p.lastUpdatePercent.Load() == int32(percent) {
now := time.Now()
if !priority && !p.lastUpdateAt.IsZero() && now.Sub(p.lastUpdateAt) < progressRenderInterval {
return
}
p.lastUpdatePercent.Store(int32(percent))
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.Downloaded(), info.TotalSize())
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchProcessingPrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalSize())/(1024*1024), info.Count())),
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
func() styling.StyledTextOption {
var lines []string
for _, elem := range info.Processing() {
lines = append(lines, fmt.Sprintf(" - %s (%.2f MB)", elem.FileName(), float64(elem.FileSize())/(1024*1024)))
}
if len(lines) == 0 {
lines = append(lines, i18n.T(i18nk.BotMsgProgressProcessingNone, nil))
}
return styling.Plain(slice.Join(lines, "\n"))
}(),
styling.Plain(i18n.T(i18nk.BotMsgProgressAvgSpeedPrefix, nil)),
styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(info.Downloaded(), p.start)/(1024*1024))),
styling.Plain(i18n.T(i18nk.BotMsgProgressCurrentProgressPrefix, nil)),
styling.Bold(fmt.Sprintf("%.2f%%", float64(info.Downloaded())/float64(info.TotalSize())*100)),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
message := buildBatchProgressMessage(info, p.skippedFiles, visibleActiveItems())
if message.Err != nil {
log.FromContext(ctx).Errorf("Failed to render batch progress message: %v", message.Err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
if message.Text == p.lastText {
return
}
p.lastText = message.Text
p.lastUpdateAt = now
p.editMessage(ctx, info.TaskID(), message, true)
}
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
if err != nil {
log.FromContext(ctx).Errorf("Batch task %s failed: %s", info.TaskID(), err)
} else {
log.FromContext(ctx).Debugf("Batch task %s completed successfully", info.TaskID())
}
entityBuilder := entity.Builder{}
var stylingErr error
if err != nil {
if errors.Is(err, context.Canceled) {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskCanceled, nil)),
)
} else {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskFailedWithError, map[string]any{
"Error": "",
})),
styling.Code(err.Error()),
)
}
} else {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchDonePrefix, nil)),
styling.Code(strconv.Itoa(info.Count())),
styling.Plain(i18n.T(i18nk.BotMsgProgressTotalSizePrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB", float64(info.TotalSize())/(1024*1024))),
)
}
if stylingErr != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", stylingErr)
p.updateMu.Lock()
defer p.updateMu.Unlock()
if p.done {
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
p.done = true
message := buildBatchDoneMessage(info, p.skippedFiles, err)
if message.Err != nil {
log.FromContext(ctx).Errorf("Failed to render final batch progress message: %v", message.Err)
return
}
req.SetMessage(text)
req.SetEntities(entities)
p.lastText = message.Text
p.editMessage(ctx, info.TaskID(), message, false)
}
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
func (p *Progress) editMessage(ctx context.Context, taskID string, message renderedBatchMessage, cancellable bool) {
if message.Err != nil {
log.FromContext(ctx).Errorf("Failed to render batch progress message: %v", message.Err)
return
}
req := buildBatchEditMessageRequest(p.MessageID, taskID, message, cancellable)
if ext := tgutil.ExtFromContext(ctx); ext != nil {
if _, err := ext.EditMessage(p.ChatID, req); err != nil {
log.FromContext(ctx).Errorf("Failed to edit batch progress message: %v", err)
}
}
}
func buildBatchEditMessageRequest(messageID int, taskID string, message renderedBatchMessage, cancellable bool) *tg.MessagesEditMessageRequest {
req := &tg.MessagesEditMessageRequest{ID: messageID}
req.SetMessage(message.Text)
if len(message.Entities) > 0 {
req.SetEntities(message.Entities)
}
if cancellable {
req.SetReplyMarkup(&tg.ReplyInlineMarkup{Rows: []tg.KeyboardButtonRow{{
Buttons: []tg.KeyboardButtonClass{tgutil.BuildCancelButton(taskID)},
}}})
}
return req
}
func buildBatchProgressText(info TaskInfo, skipped []string, activeLimit int) string {
return buildBatchProgressMessage(info, skipped, activeLimit).Text
}
func buildBatchProgressMessage(info TaskInfo, skipped []string, activeLimit int) renderedBatchMessage {
items := info.Items()
completed, waiting, downloaded, failed := itemCounts(items)
downloadSpeed, uploadSpeed := aggregateSpeeds(items)
if activeLimit < 1 {
activeLimit = 1
}
total := len(items) + len(skipped)
downloadSpeedText := formatSpeed(downloadSpeed)
uploadSpeedText := formatSpeed(uploadSpeed)
header := localizedProgressMarkup(i18nk.BotMsgProgressBatchStatusHeader, map[string]any{
"Total": total,
"TotalSize": dlutil.FormatSize(info.ActualTotalSize()),
"Completed": completed,
"Downloaded": downloaded,
"Waiting": waiting,
"DownloadSpeed": downloadSpeedText,
"UploadSpeed": uploadSpeedText,
})
var markup strings.Builder
markup.WriteString(header)
visibleItems, hiddenTransfers, summarizedConfirming := visibleBatchItems(items, activeLimit)
for _, item := range visibleItems {
markup.WriteString("\n\n")
markup.WriteString(formatActiveItemMarkup(item, len(items)))
}
if hiddenTransfers > 0 {
markup.WriteString("\n\n")
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryHiddenActive, map[string]any{"Count": hiddenTransfers}))
}
if summarizedConfirming > 0 {
markup.WriteString("\n\n")
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryConfirming, map[string]any{"Count": summarizedConfirming}))
}
if failed > 0 {
markup.WriteString("\n")
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryFailed, map[string]any{"Count": failed}))
}
if len(skipped) > 0 {
markup.WriteString("\n")
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummarySkipped, map[string]any{"Count": len(skipped)}))
}
return completeBatchMessage(markup.String())
}
func buildBatchDoneMarkup(info TaskInfo, skipped []string, err error) string {
items := info.Items()
totalSize := info.ActualTotalSize()
if totalSize == 0 {
totalSize = info.TotalSize()
}
if err == nil {
completed, _, _, failed := itemCounts(items)
// Report per-element failures instead of full completion.
totalSkipped := len(skipped) + failed
if totalSkipped > 0 {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchDoneWithSkipped, map[string]any{
"Success": completed,
"Skipped": totalSkipped,
"Size": dlutil.FormatSize(totalSize),
})
}
return localizedProgressMarkup(i18nk.BotMsgProgressBatchDone, map[string]any{
"Count": len(items),
"Size": dlutil.FormatSize(totalSize),
})
}
completed, _, _, failed := itemCounts(items)
incomplete := max(len(items)-completed-failed, 0)
if errors.Is(err, context.Canceled) {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchCanceled, map[string]any{
"Total": len(items) + len(skipped),
"Completed": completed,
"Incomplete": incomplete,
"Skipped": len(skipped),
})
}
failedItems := make([]TaskItemProgress, 0, failed)
for _, item := range items {
if item.Phase == ItemPhaseFailed {
failedItems = append(failedItems, item)
}
}
if len(failedItems) > 1 && failedItems[0].FailureStage == FailureStageBatchUpload {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedGroup, map[string]any{
"Affected": len(failedItems),
"Reason": displayError(firstError(failedItems), err),
"Completed": completed,
"Failed": len(failedItems),
"Incomplete": incomplete,
})
}
if len(failedItems) == 0 {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedTask, map[string]any{
"Reason": displayError("", err),
"Completed": completed,
"Incomplete": incomplete,
})
}
item := failedItems[0]
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedItem, map[string]any{
"Index": item.Index,
"Name": truncateFilename(item.Name, maxDisplayNameRunes),
"Stage": failureStageLabel(item.FailureStage),
"Progress": failureProgress(item),
"Speed": failureSpeed(item),
"Reason": displayError(item.Error, err),
"Completed": completed,
"Failed": failed,
"Incomplete": incomplete,
})
}
func buildBatchDoneMessage(info TaskInfo, skipped []string, err error) renderedBatchMessage {
return completeBatchMessage(buildBatchDoneMarkup(info, skipped, err))
}
func formatActiveItemMarkup(item TaskItemProgress, total int) string {
data := map[string]any{
"Index": item.Index,
"Total": total,
"Name": truncateFilename(item.Name, maxDisplayNameRunes),
"Speed": formatSpeed(itemSpeed(item)),
"Progress": itemPercent(item),
"Bar": textProgressBar(itemPercent(item)),
"Current": dlutil.FormatSize(itemBytes(item)),
"Size": dlutil.FormatSize(item.Size),
"Attempt": min(max(item.RetryAttempt, 1), max(item.RetryLimit, 1)),
"Limit": max(item.RetryLimit, 1),
"Reason": truncateRunes(item.Error, maxDisplayErrorRunes),
}
switch item.Phase {
case ItemPhaseDownloading:
if item.Size <= 0 {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemDownloadingUnknown, data)
}
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemDownloading, data)
case ItemPhaseTransferring:
if item.Size <= 0 {
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemTransferringUnknown, data)
}
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemTransferring, data)
case ItemPhaseUploading:
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemUploading, data)
case ItemPhaseRetrying:
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemRetrying, data)
case ItemPhaseConfirming:
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemConfirming, data)
default:
return ""
}
}
func localizedProgressMarkup(key i18nk.Key, data map[string]any) string {
return i18n.T(key, tgutil.EscapeHTMLTemplateData(data))
}
func visibleBatchItems(items []TaskItemProgress, limit int) (visible []TaskItemProgress, hiddenTransfers, summarizedConfirming int) {
visible = make([]TaskItemProgress, 0, limit)
transferCount := 0
confirmingCount := 0
for _, item := range items {
switch {
case isTransferPhase(item.Phase):
transferCount++
if len(visible) < limit {
visible = append(visible, item)
}
case item.Phase == ItemPhaseConfirming:
confirmingCount++
}
}
hiddenTransfers = transferCount - len(visible)
if confirmingCount == 1 && len(visible) < limit {
for _, item := range items {
if item.Phase == ItemPhaseConfirming {
visible = append(visible, item)
return visible, hiddenTransfers, 0
}
}
}
return visible, hiddenTransfers, confirmingCount
}
func completeBatchMessage(markup string) renderedBatchMessage {
text, entities, err := tgutil.RenderHTML(markup)
return renderedBatchMessage{Text: text, Entities: entities, Err: err}
}
func itemCounts(items []TaskItemProgress) (completed, waiting, downloaded, failed int) {
for _, item := range items {
switch item.Phase {
case ItemPhaseCompleted:
completed++
case ItemPhaseWaiting:
waiting++
case ItemPhaseDownloaded:
downloaded++
case ItemPhaseFailed:
failed++
}
}
return
}
func aggregateSpeeds(items []TaskItemProgress) (download, upload float64) {
for _, item := range items {
switch item.Phase {
case ItemPhaseDownloading:
download += item.DownloadSpeed
case ItemPhaseTransferring:
download += item.DownloadSpeed
upload += item.UploadSpeed
case ItemPhaseUploading:
upload += item.UploadSpeed
}
}
return
}
func isTransferPhase(phase ItemPhase) bool {
switch phase {
case ItemPhaseDownloading, ItemPhaseTransferring, ItemPhaseUploading, ItemPhaseRetrying:
return true
default:
return false
}
}
func itemBytes(item TaskItemProgress) int64 {
switch item.Phase {
case ItemPhaseDownloading, ItemPhaseTransferring:
return item.Downloaded
default:
return item.Uploaded
}
}
func itemSpeed(item TaskItemProgress) float64 {
switch item.Phase {
case ItemPhaseDownloading, ItemPhaseTransferring:
return item.DownloadSpeed
case ItemPhaseUploading:
return item.UploadSpeed
case ItemPhaseRetrying:
return item.UploadSpeed
default:
return 0
}
}
func itemPercent(item TaskItemProgress) int {
if item.Size <= 0 {
return 0
}
return int(min(itemBytes(item), item.Size) * 100 / item.Size)
}
func textProgressBar(percent int) string {
percent = min(max(percent, 0), 100)
filled := percent * progressBarWidth / 100
return strings.Repeat("🟩", filled) + strings.Repeat("⬜️", progressBarWidth-filled)
}
func formatSpeed(speed float64) string {
if speed <= 0 {
return "0 B/s"
}
return dlutil.FormatSize(int64(speed)) + "/s"
}
func truncateFilename(name string, limit int) string {
if utf8.RuneCountInString(name) <= limit {
return name
}
ext := path.Ext(name)
if utf8.RuneCountInString(ext) >= limit-2 {
return truncateRunes(name, limit-1) + "…"
}
base := strings.TrimSuffix(name, ext)
baseLimit := limit - utf8.RuneCountInString(ext) - 1
return truncateRunes(base, baseLimit) + "…" + ext
}
func truncateRunes(value string, limit int) string {
if limit <= 0 {
return ""
}
runes := []rune(value)
if len(runes) <= limit {
return value
}
return string(runes[:limit])
}
func displayError(itemError string, fallback error) string {
if itemError == "" && fallback != nil {
itemError = compactError(fallback)
}
return truncateRunes(itemError, maxDisplayErrorRunes)
}
func firstError(items []TaskItemProgress) string {
for _, item := range items {
if item.Error != "" {
return item.Error
}
}
return ""
}
func failureStageLabel(stage FailureStage) string {
switch stage {
case FailureStageDownload:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageDownload, nil)
case FailureStageCache:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageCache, nil)
case FailureStageUpload:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageUpload, nil)
case FailureStageConfirm:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageConfirm, nil)
case FailureStageBatchUpload:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageBatchUpload, nil)
default:
return i18n.T(i18nk.BotMsgProgressBatchFailureStageInternal, nil)
}
}
func failureProgress(item TaskItemProgress) string {
if item.Size <= 0 {
return dlutil.FormatSize(failureBytes(item))
}
return fmt.Sprintf("%d%%", min(failureBytes(item), item.Size)*100/item.Size)
}
func failureSpeed(item TaskItemProgress) string {
if item.FailureStage == FailureStageDownload || item.FailureStage == FailureStageCache {
return formatSpeed(item.DownloadSpeed)
}
return formatSpeed(item.UploadSpeed)
}
func failureBytes(item TaskItemProgress) int64 {
if item.FailureStage == FailureStageDownload || item.FailureStage == FailureStageCache {
return item.Downloaded
}
return item.Uploaded
}
func visibleActiveItems() int {
return min(max(config.C().Workers, 1), maxVisibleActiveItems)
}
func NewProgressTracker(messageID int, chatID int64) ProgressTracker {
return NewProgressTrackerWithSkipped(messageID, chatID, nil)
}
func NewProgressTrackerWithSkipped(messageID int, chatID int64, skippedFiles []string) ProgressTracker {
return &Progress{
MessageID: messageID,
ChatID: chatID,
MessageID: messageID,
ChatID: chatID,
skippedFiles: skippedFiles,
}
}

View File

@@ -0,0 +1,335 @@
package batchtfile
import (
"context"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/pkg/tfile"
)
type progressRegressionRecorder struct {
mu sync.Mutex
startTotal int64
notifications []int64
}
func (*progressRegressionRecorder) OnStart(context.Context, TaskInfo) {}
func (*progressRegressionRecorder) OnProgress(context.Context, TaskInfo) {}
func (*progressRegressionRecorder) OnDone(context.Context, TaskInfo, error) {}
func (r *progressRegressionRecorder) OnUploadStart(_ context.Context, _ TaskInfo, total int64) {
r.mu.Lock()
defer r.mu.Unlock()
r.startTotal = total
}
func (r *progressRegressionRecorder) OnUploadProgress(_ context.Context, _ TaskInfo, uploaded, _ int64) {
r.mu.Lock()
defer r.mu.Unlock()
r.notifications = append(r.notifications, uploaded)
}
type orderedProgressRegressionRecorder struct {
firstEntered chan struct{}
releaseFirst chan struct{}
secondEntered chan struct{}
mu sync.Mutex
notifications []int64
}
func (*orderedProgressRegressionRecorder) OnStart(context.Context, TaskInfo) {}
func (*orderedProgressRegressionRecorder) OnProgress(context.Context, TaskInfo) {}
func (*orderedProgressRegressionRecorder) OnDone(context.Context, TaskInfo, error) {}
func (*orderedProgressRegressionRecorder) OnUploadStart(context.Context, TaskInfo, int64) {
}
func (r *orderedProgressRegressionRecorder) OnUploadProgress(_ context.Context, _ TaskInfo, uploaded, _ int64) {
if uploaded == 100 {
close(r.firstEntered)
<-r.releaseFirst
}
if uploaded == 200 {
close(r.secondEntered)
}
r.mu.Lock()
r.notifications = append(r.notifications, uploaded)
r.mu.Unlock()
}
func TestBatchProgressShowsTransferSpeedAndSize(t *testing.T) {
useProgressRegressionLocale(t)
task := newProgressRegressionTask(nil,
progressRegressionFile{"downloading", 1000},
progressRegressionFile{"uploading", 1000},
progressRegressionFile{"waiting", 1000},
)
started := time.Unix(100, 0)
task.markItemActive("downloading", false, started)
task.recordItemDownload("downloading", 500, started.Add(time.Second))
task.recordItemDownloaded("uploading", 1000)
task.recordItemUpload("uploading", 0, 1000, started.Add(time.Second))
task.recordItemUpload("uploading", 250, 1000, started.Add(2*time.Second))
message := buildBatchProgressMessage(task, nil, 2)
if message.Err != nil {
t.Fatalf("buildBatchProgressMessage() failed: %v", message.Err)
}
assertProgressRegressionContains(t, message.Text,
"状态:✅ 0 📥 0 ⏳ 1",
"⬇️ 1/3 下载中",
"速度500 B/s",
"大小500 B / 1000 B",
"⬆️ 2/3 上传中",
"速度250 B/s",
"大小250 B / 1000 B",
)
bold, _, blockquote, _ := batchEntityCounts(message.Entities)
if bold != 3 || blockquote != 2 {
t.Fatalf("entity counts = bold:%d blockquote:%d, want bold:3 blockquote:2", bold, blockquote)
}
}
func TestBatchProgressHeaderShowsTotalSize(t *testing.T) {
useProgressRegressionLocale(t)
task := newProgressRegressionTask(nil,
progressRegressionFile{"first", 1024},
progressRegressionFile{"second", 1024},
)
message := buildBatchProgressMessage(task, nil, 2)
if message.Err != nil {
t.Fatalf("buildBatchProgressMessage() failed: %v", message.Err)
}
assertProgressRegressionContains(t, message.Text,
"文件2 总大小2.00 KB",
)
i18n.Init("en")
english := buildBatchProgressMessage(task, nil, 2)
if english.Err != nil {
t.Fatalf("English batch template failed: %v", english.Err)
}
assertProgressRegressionContains(t, english.Text,
"Files: 2 | Total size: 2.00 KB",
)
}
func TestBatchProgressLimitsRowsWithoutHidingActiveUpload(t *testing.T) {
useProgressRegressionLocale(t)
task := newProgressRegressionTask(nil,
progressRegressionFile{"confirm-01", 100},
progressRegressionFile{"confirm-02", 100},
progressRegressionFile{"uploading", 100},
progressRegressionFile{"downloading", 100},
)
started := time.Unix(100, 0)
task.recordItemUpload("confirm-01", 100, 100, started)
task.recordItemUpload("confirm-02", 100, 100, started)
task.recordItemUpload("uploading", 40, 100, started.Add(time.Second))
task.markItemActive("downloading", false, started)
message := buildBatchProgressText(task, nil, 2)
assertProgressRegressionContains(t, message,
"uploading.bin",
"downloading.bin",
"☁️ 已上传等待整组发送2",
)
if strings.Contains(message, "confirm-01.bin") || strings.Contains(message, "confirm-02.bin") {
t.Fatalf("confirmation rows displaced active transfers:\n%s", message)
}
}
func TestBatchProgressTemplateOwnsStylesAndEscapesValues(t *testing.T) {
useProgressRegressionLocale(t)
fileID := `<b>A&B</b>`
task := newProgressRegressionTask(nil, progressRegressionFile{fileID, 100})
task.markItemRetry(fileID, FailureStageUpload, 1, 3, errors.New(`<i>remote & failed</i>`))
message := buildBatchProgressMessage(task, nil, 1)
if message.Err != nil {
t.Fatalf("buildBatchProgressMessage() failed: %v", message.Err)
}
assertProgressRegressionContains(t, message.Text,
`<b>A&B</b>.bin`,
`<i>remote & failed</i>`,
)
bold, _, blockquote, italic := batchEntityCounts(message.Entities)
if bold != 2 || blockquote != 1 || italic != 0 {
t.Fatalf("entity counts = bold:%d blockquote:%d italic:%d", bold, blockquote, italic)
}
i18n.Init("en")
english := buildBatchProgressMessage(task, nil, 1)
if english.Err != nil {
t.Fatalf("English batch template failed: %v", english.Err)
}
assertProgressRegressionContains(t, english.Text, "📦 Processing", "Retrying upload", `<b>A&B</b>.bin`)
}
func TestDownloadProgressContinuesAfterUploadStarts(t *testing.T) {
useProgressRegressionLocale(t)
progress := new(Progress)
task := newProgressRegressionTask(progress,
progressRegressionFile{"uploading", 100},
progressRegressionFile{"downloading", 100},
)
progress.OnStart(t.Context(), task)
task.recordDownloadComplete("uploading", 100)
task.uploadCallback(t.Context(), "uploading")(50, 100)
started := time.Unix(100, 0)
task.markItemActive("downloading", false, started)
task.recordItemDownload("downloading", 50, started.Add(time.Second))
progress.updateMu.Lock()
progress.lastUpdateAt = time.Now().Add(-progressRenderInterval)
progress.updateMu.Unlock()
progress.OnProgress(t.Context(), task)
progress.updateMu.Lock()
text := progress.lastText
progress.updateMu.Unlock()
assertProgressRegressionContains(t, text,
"uploading.bin",
"🟩🟩🟩🟩🟩⬜️⬜️⬜️⬜️⬜️ 50%",
"总速度:⬇️ 50 B/s ⬆️ 0 B/s",
"🔄 另有 1 个文件正在处理",
)
}
func TestBatchUploadIgnoresOutOfOrderBytesAndAllowsRetryReset(t *testing.T) {
recorder := new(progressRegressionRecorder)
task := newProgressRegressionTask(recorder, progressRegressionFile{"file", 100})
task.recordDownloadComplete("file", 100)
callback := task.uploadCallback(t.Context(), "file")
callback(80, 100)
callback(10, 100)
if got := task.Items()[0].Uploaded; got != 80 {
t.Fatalf("out-of-order callback regressed item to %d, want 80", got)
}
task.markItemRetry("file", FailureStageUpload, 1, 3, context.DeadlineExceeded)
callback(0, 100)
callback(10, 100)
if got := task.Items()[0].Uploaded; got != 10 {
t.Fatalf("retry did not reset item progress: got %d, want 10", got)
}
recorder.mu.Lock()
defer recorder.mu.Unlock()
for index := 1; index < len(recorder.notifications); index++ {
if recorder.notifications[index] < recorder.notifications[index-1] {
t.Fatalf("aggregate progress regressed: %v", recorder.notifications)
}
}
}
func TestUploadProgressNotificationsRemainOrdered(t *testing.T) {
recorder := &orderedProgressRegressionRecorder{
firstEntered: make(chan struct{}),
releaseFirst: make(chan struct{}),
secondEntered: make(chan struct{}),
}
task := newProgressRegressionTask(recorder,
progressRegressionFile{"first", 100},
progressRegressionFile{"second", 100},
)
task.recordDownloadComplete("first", 100)
task.recordDownloadComplete("second", 100)
first := task.uploadCallback(t.Context(), "first")
second := task.uploadCallback(t.Context(), "second")
var wait sync.WaitGroup
wait.Go(func() {
first(100, 100)
})
<-recorder.firstEntered
wait.Go(func() {
second(100, 100)
})
overtook := false
select {
case <-recorder.secondEntered:
overtook = true
case <-time.After(100 * time.Millisecond):
}
close(recorder.releaseFirst)
wait.Wait()
if overtook {
t.Fatal("later aggregate notification overtook the first callback")
}
recorder.mu.Lock()
defer recorder.mu.Unlock()
if got := recorder.notifications; len(got) != 2 || got[0] != 100 || got[1] != 200 {
t.Fatalf("upload notifications = %v, want [100 200]", got)
}
}
func TestBatchUploadUsesActualSizeWhenMetadataIsUnknown(t *testing.T) {
recorder := new(progressRegressionRecorder)
task := newProgressRegressionTask(recorder, progressRegressionFile{"photo", 0})
task.recordDownloadComplete("photo", 25)
task.uploadCallback(t.Context(), "photo")(25, 25)
recorder.mu.Lock()
defer recorder.mu.Unlock()
if recorder.startTotal != 25 {
t.Fatalf("upload start total = %d, want actual size 25", recorder.startTotal)
}
if got := task.ActualTotalSize(); got != 25 {
t.Fatalf("actual total size = %d, want 25", got)
}
}
type progressRegressionFile struct {
id string
size int64
}
func newProgressRegressionTask(progress ProgressTracker, files ...progressRegressionFile) *Task {
elems := make([]TaskElement, 0, len(files))
for _, file := range files {
elems = append(elems, TaskElement{
ID: file.id,
File: tfile.NewTGFile(nil, nil, file.size, file.id+".bin"),
})
}
return NewBatchTGFileTask("progress-regression", context.Background(), elems, progress, true)
}
func useProgressRegressionLocale(t *testing.T) {
t.Helper()
i18n.Init("zh-Hans")
t.Cleanup(func() { i18n.Init("zh-Hans") })
}
func assertProgressRegressionContains(t *testing.T, value string, wants ...string) {
t.Helper()
for _, want := range wants {
if !strings.Contains(value, want) {
t.Fatalf("text does not contain %q:\n%s", want, value)
}
}
}
func batchEntityCounts(entities []tg.MessageEntityClass) (bold, code, blockquote, italic int) {
for _, messageEntity := range entities {
switch messageEntity.(type) {
case *tg.MessageEntityBold:
bold++
case *tg.MessageEntityCode:
code++
case *tg.MessageEntityBlockquote:
blockquote++
case *tg.MessageEntityItalic:
italic++
}
}
return
}

View File

@@ -7,6 +7,7 @@ import (
"sync"
"sync/atomic"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
@@ -18,25 +19,34 @@ import (
var _ core.Executable = (*Task)(nil)
type TaskElement struct {
ID string
Storage storage.Storage
Path string
File tfile.TGFile
localPath string
stream bool
ID string
Storage storage.Storage
Path string
File tfile.TGFile
localPath string
stream bool
sourceGroupKey string
sourceCaption string
preserveCaption bool
}
type Task struct {
ID string
ctx context.Context
elems []TaskElement
Progress ProgressTracker
IgnoreErrors bool // if true, errors during processing will be ignored
downloaded atomic.Int64
totalSize int64
processing map[string]TaskElementInfo
processingMu sync.RWMutex
failed map[string]error // [TODO] errors for each element
ID string
ctx context.Context
elems []TaskElement
Progress ProgressTracker
IgnoreErrors bool // if true, errors during processing will be ignored
downloaded atomic.Int64
totalSize int64
uploadTotalSize atomic.Int64
processing map[string]TaskElementInfo
processingMu sync.RWMutex
itemStates []itemProgressState
itemIndex map[string]int
itemMu sync.RWMutex
uploadOnce sync.Once
uploadMu sync.Mutex
uploaded map[string]int64
}
// Title implements core.Exectable.
@@ -54,6 +64,7 @@ func NewTaskElement(
file tfile.TGFile,
) (*TaskElement, error) {
id := xid.New().String()
groupKey, caption, preserveCaption := sourceMetadata(file)
_, ok := stor.(storage.StorageCannotStream)
if !config.C().Stream || ok {
cachePath, err := filepath.Abs(filepath.Join(config.C().Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
@@ -61,22 +72,42 @@ func NewTaskElement(
return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
}
return &TaskElement{
ID: id,
Storage: stor,
Path: path,
File: file,
localPath: cachePath,
ID: id,
Storage: stor,
Path: path,
File: file,
localPath: cachePath,
sourceGroupKey: groupKey,
sourceCaption: caption,
preserveCaption: preserveCaption,
}, nil
}
return &TaskElement{
ID: id,
Storage: stor,
Path: path,
File: file,
stream: true,
ID: id,
Storage: stor,
Path: path,
File: file,
stream: true,
sourceGroupKey: groupKey,
sourceCaption: caption,
preserveCaption: preserveCaption,
}, nil
}
func sourceMetadata(file tfile.TGFile) (groupKey, caption string, preserveCaption bool) {
messageFile, ok := file.(tfile.TGFileMessage)
if !ok || messageFile.Message() == nil {
return "", "", false
}
msg := messageFile.Message()
groupID, grouped := msg.GetGroupedID()
if !grouped || groupID == 0 {
return "", "", false
}
chatID := tgutil.ChatIdFromPeer(msg.GetPeerID())
return fmt.Sprintf("%T:%d:%d", msg.GetPeerID(), chatID, groupID), msg.GetMessage(), true
}
func NewBatchTGFileTask(
id string,
ctx context.Context,
@@ -84,6 +115,7 @@ func NewBatchTGFileTask(
progress ProgressTracker,
ignoreErrors bool,
) *Task {
itemStates, itemIndex := newItemProgressStates(files)
task := &Task{
ID: id,
ctx: ctx,
@@ -98,9 +130,11 @@ func NewBatchTGFileTask(
return total
}(),
processing: make(map[string]TaskElementInfo),
itemStates: itemStates,
itemIndex: itemIndex,
uploaded: make(map[string]int64),
IgnoreErrors: ignoreErrors,
processingMu: sync.RWMutex{},
failed: make(map[string]error),
}
return task
}

View File

@@ -27,8 +27,10 @@ type TaskInfo interface {
TaskID() string
TotalSize() int64
Downloaded() int64
ActualTotalSize() int64
Count() int
Processing() []TaskElementInfo
Items() []TaskItemProgress
}
func (t *Task) TaskID() string {

View File

@@ -0,0 +1,70 @@
package batchtfile
import (
"context"
"time"
)
// UploadProgressTracker optionally extends a batch progress tracker with a
// distinct aggregate upload phase.
type UploadProgressTracker interface {
OnUploadStart(ctx context.Context, info TaskInfo, total int64)
OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64)
}
func (t *Task) startUpload(ctx context.Context) {
tracker, ok := t.Progress.(UploadProgressTracker)
if !ok {
return
}
t.uploadMu.Lock()
defer t.uploadMu.Unlock()
t.uploadOnce.Do(func() {
tracker.OnUploadStart(ctx, t, t.uploadTotalSize.Load())
})
}
func (t *Task) uploadCallback(ctx context.Context, id string) func(uploaded, total int64) {
return func(uploaded, total int64) {
tracker, ok := t.Progress.(UploadProgressTracker)
if !ok || uploaded < 0 {
return
}
t.startUpload(ctx)
if total > 0 && uploaded > total {
uploaded = total
}
t.uploadMu.Lock()
defer t.uploadMu.Unlock()
becameConfirming := t.recordItemUpload(id, uploaded, total, time.Now())
if t.uploaded == nil {
t.uploaded = make(map[string]int64)
}
previous, tracked := t.uploaded[id]
if !tracked || uploaded > previous {
t.uploaded[id] = uploaded
}
var aggregate int64
for _, current := range t.uploaded {
aggregate += current
}
uploadTotal := t.uploadTotalSize.Load()
if aggregate > uploadTotal {
aggregate = uploadTotal
}
tracker.OnUploadProgress(ctx, t, aggregate, uploadTotal)
if becameConfirming {
t.notifyStateChange(ctx)
}
}
}
func (t *Task) recordDownloadComplete(id string, uploadSize int64) {
t.uploadMu.Lock()
defer t.uploadMu.Unlock()
if uploadSize > 0 {
t.uploadTotalSize.Add(uploadSize)
}
t.recordItemDownloaded(id, uploadSize)
}

View File

@@ -1,32 +0,0 @@
package batchtfile
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值
stepPercent int // 每多少 % 更新一次
}{
{10 << 20, 100},
{50 << 20, 20},
{200 << 20, 10},
{500 << 20, 5},
}
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
for _, lvl := range progressUpdatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}

View File

@@ -15,6 +15,7 @@ import (
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"golang.org/x/sync/errgroup"
)
@@ -75,12 +76,11 @@ func (t *Task) Execute(ctx context.Context) error {
eg.SetLimit(config.C().Workers)
for _, file := range t.files {
eg.Go(func() error {
t.processingMu.RLock()
t.processingMu.Lock()
if _, ok := t.processing[file.URL]; ok {
t.processingMu.Unlock()
return fmt.Errorf("file %s is already being processed", file.URL)
}
t.processingMu.RUnlock()
t.processingMu.Lock()
t.processing[file.URL] = file
t.processingMu.Unlock()
defer func() {
@@ -89,7 +89,6 @@ func (t *Task) Execute(ctx context.Context) error {
t.processingMu.Unlock()
}()
err := t.processLink(gctx, file)
t.downloaded.Add(1)
if errors.Is(err, context.Canceled) {
logger.Debug("Link processing canceled")
return err
@@ -98,6 +97,7 @@ func (t *Task) Execute(ctx context.Context) error {
logger.Errorf("Error processing link %s: %v", file.URL, err)
return fmt.Errorf("failed to process link %s: %w", file.URL, err)
}
t.downloaded.Add(1)
return nil
})
}
@@ -143,10 +143,16 @@ func (t *Task) processLink(ctx context.Context, file *File) error {
}
}()
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
t.downloadedBytes.Add(int64(n))
downloaded := t.downloadedBytes.Add(int64(n))
if t.Progress != nil {
t.Progress.OnProgress(ctx, t)
}
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalBytes,
DownloadedBytes: downloaded,
})
})
copyResultCh := make(chan error, 1)

View File

@@ -15,6 +15,7 @@ import (
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
@@ -102,7 +103,7 @@ func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
// OnProgress implements ProgressTracker.
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
if !progressutil.ShouldUpdate(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
return
}
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
@@ -115,7 +116,10 @@ func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalFiles())),
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithFiles, map[string]any{
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
"Count": info.TotalFiles(),
})),
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
func() styling.StyledTextOption {
var lines []string

View File

@@ -45,7 +45,6 @@ type Task struct {
downloaded atomic.Int64 // downloaded files count
processing map[string]*File // {"url": File}
processingMu sync.RWMutex
failed map[string]error // [TODO] errors for each file
}
// Title implements core.Exectable.
@@ -127,7 +126,6 @@ func NewTask(
client: http.DefaultClient,
processing: make(map[string]*File),
processingMu: sync.RWMutex{},
failed: make(map[string]error),
totalFiles: int64(len(files)),
}
}

View File

@@ -207,34 +207,3 @@ func parseFilenameFallback(cd string) string {
return decodeFilenameParam(value)
}
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值
stepPercent int // 每多少 % 更新一次
}{
{10 << 20, 100},
{50 << 20, 50},
{200 << 20, 20},
{500 << 20, 10},
}
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
for _, lvl := range progressUpdatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}

View File

@@ -16,6 +16,7 @@ import (
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/parser"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"golang.org/x/sync/errgroup"
)
@@ -29,21 +30,20 @@ func (t *Task) Execute(ctx context.Context) error {
eg.SetLimit(config.C().Workers)
for _, resource := range t.item.Resources {
eg.Go(func() error {
t.processingMu.RLock()
if t.processing[resource.ID()] != nil {
return fmt.Errorf("resource %s is already being processed", resource.ID())
}
t.processingMu.RUnlock()
resourceID := resource.ID()
t.processingMu.Lock()
t.processing[resource.ID()] = &resource
if t.processing[resourceID] != nil {
t.processingMu.Unlock()
return fmt.Errorf("resource %s is already being processed", resourceID)
}
t.processing[resourceID] = &resource
t.processingMu.Unlock()
defer func() {
t.processingMu.Lock()
delete(t.processing, resource.URL)
delete(t.processing, resourceID)
t.processingMu.Unlock()
}()
err := t.processResource(gctx, resource)
t.downloaded.Add(1)
if errors.Is(err, context.Canceled) {
logger.Debug("Resource processing canceled")
return err
@@ -52,6 +52,7 @@ func (t *Task) Execute(ctx context.Context) error {
logger.Errorf("Error processing resource %s: %v", resource.URL, err)
return fmt.Errorf("failed to process resource %s: %w", resource.URL, err)
}
t.downloaded.Add(1)
return nil
})
}
@@ -107,10 +108,16 @@ func (t *Task) processResource(ctx context.Context, resource parser.Resource) er
}
}()
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
t.downloadedBytes.Add(int64(n))
downloaded := t.downloadedBytes.Add(int64(n))
if t.progress != nil {
t.progress.OnProgress(ctx, t)
}
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalBytes,
DownloadedBytes: downloaded,
})
})
copyResultCh := make(chan error, 1)

View File

@@ -15,40 +15,10 @@ import (
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值
stepPercent int // 每多少 % 更新一次
}{
{10 << 20, 100},
{50 << 20, 50},
{200 << 20, 20},
{500 << 20, 10},
}
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
for _, lvl := range progressUpdatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}
type ProgressTracker interface {
OnStart(ctx context.Context, info TaskInfo)
OnProgress(ctx context.Context, info TaskInfo)
@@ -73,7 +43,10 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
styling.Plain(i18n.T(i18nk.BotMsgProgressParsedStartPrefix, map[string]any{
"Site": info.Site(),
})),
styling.Code(fmt.Sprintf("%.2f MB (%d个资源)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithResources, map[string]any{
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
"Count": info.TotalResources(),
})),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
return
@@ -101,7 +74,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
if !progressutil.ShouldUpdate(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
return
}
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
@@ -114,7 +87,10 @@ func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithResources, map[string]any{
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
"Count": info.TotalResources(),
})),
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
func() styling.StyledTextOption {
var lines []string

View File

@@ -33,7 +33,6 @@ type Task struct {
downloadedBytes atomic.Int64 // downloaded bytes count
processing map[string]ResourceInfo
processingMu sync.RWMutex
failed map[string]error // [TODO] errors for each resource
}
// Title implements core.Exectable.
@@ -84,6 +83,5 @@ func NewTask(
progress: progressTracker,
processing: make(map[string]ResourceInfo),
processingMu: sync.RWMutex{},
failed: make(map[string]error),
}
}

View File

@@ -11,13 +11,16 @@ import (
"github.com/duke-git/lancet/v2/retry"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"golang.org/x/sync/errgroup"
)
func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx)
logger.Infof("Starting Telegraph task %s", t.PhPath)
t.progress.OnStart(ctx, t)
if t.progress != nil {
t.progress.OnStart(ctx, t)
}
eg, gctx := errgroup.WithContext(ctx)
eg.SetLimit(config.C().Workers)
for i, pic := range t.Pics {
@@ -27,8 +30,16 @@ func (t *Task) Execute(ctx context.Context) error {
logger.Errorf("Error processing picture %s: %v", pic, err)
return fmt.Errorf("failed to process picture %s: %w", pic, err)
}
t.downloaded.Add(1)
t.progress.OnProgress(gctx, t)
downloaded := t.downloaded.Add(1)
if t.progress != nil {
t.progress.OnProgress(gctx, t)
}
taskevent.Emit(gctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalFiles: t.totalpics,
DownloadedFiles: int(downloaded),
})
return nil
})
}
@@ -38,7 +49,9 @@ func (t *Task) Execute(ctx context.Context) error {
} else {
logger.Infof("Telegraph task %s completed successfully", t.PhPath)
}
t.progress.OnDone(ctx, t, err)
if t.progress != nil {
t.progress.OnDone(ctx, t, err)
}
return err
}

View File

@@ -0,0 +1,31 @@
package telegraph_test
import (
"context"
"io"
"testing"
storconfig "github.com/krau/SaveAny-Bot/config/storage"
"github.com/krau/SaveAny-Bot/core/tasks/telegraph"
storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
"github.com/krau/SaveAny-Bot/storage"
)
type mockStorage struct{}
func (mockStorage) Init(context.Context, storconfig.StorageConfig) error { return nil }
func (mockStorage) Type() storenum.StorageType { return storenum.Local }
func (mockStorage) Name() string { return "mock" }
func (mockStorage) Save(context.Context, io.Reader, string) error { return nil }
func (mockStorage) Exists(context.Context, string) bool { return false }
var _ storage.Storage = mockStorage{}
// Regression: API-created tasks run with a nil ProgressTracker (progress goes
// through taskevent); Execute must not panic on the tracker callbacks.
func TestExecuteWithNilTracker(t *testing.T) {
task := telegraph.NewTask("id", t.Context(), "/page", nil, mockStorage{}, "/out", nil, nil)
if err := task.Execute(t.Context()); err != nil {
t.Fatalf("Execute returned error: %v", err)
}
}

View File

@@ -11,6 +11,7 @@ import (
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
@@ -60,7 +61,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.Downloaded(), int64(info.TotalPics())) {
if !progressutil.ShouldUpdateCount(info.Downloaded(), int64(info.TotalPics())) {
return
}
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.Downloaded(), info.TotalPics())

View File

@@ -1,13 +0,0 @@
package telegraph
func shouldUpdateProgress(downloaded int64, total int64) bool {
if total <= 0 || downloaded <= 0 {
return false
}
step := int64(10)
if downloaded < step {
return downloaded == total
}
return downloaded%step == 0 || downloaded == total
}

View File

@@ -0,0 +1,42 @@
package tfile
import (
"testing"
"github.com/gotd/td/tg"
tfilepkg "github.com/krau/SaveAny-Bot/pkg/tfile"
)
func TestSourceCaption(t *testing.T) {
tests := []struct {
name string
file tfilepkg.TGFile
want string
ok bool
}{
{
name: "original caption",
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov", tfilepkg.WithMessage(&tg.Message{Message: "original caption"})),
want: "original caption",
ok: true,
},
{
name: "empty caption suppresses storage fallback",
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov", tfilepkg.WithMessage(&tg.Message{})),
ok: true,
},
{
name: "file without source message",
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov"),
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, ok := sourceCaption(tt.file)
if ok != tt.ok || got != tt.want {
t.Fatalf("sourceCaption() = (%q, %v), want (%q, %v)", got, ok, tt.want, tt.ok)
}
})
}
}

View File

@@ -3,6 +3,7 @@ package tfile
import (
"context"
"fmt"
"io"
"os"
"path"
@@ -10,8 +11,12 @@ import (
"github.com/duke-git/lancet/v2/retry"
"github.com/krau/SaveAny-Bot/common/tdler"
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
tfilepkg "github.com/krau/SaveAny-Bot/pkg/tfile"
"github.com/krau/SaveAny-Bot/storage"
)
func (t *Task) Execute(ctx context.Context) error {
@@ -57,13 +62,34 @@ func (t *Task) Execute(ctx context.Context) error {
return fmt.Errorf("failed to get file stat: %w", err)
}
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
if caption, ok := sourceCaption(t.File); ok {
vctx = storagetypes.WithSourceCaption(vctx, caption)
}
err = retry.Retry(func() error {
file, err := os.Open(t.localPath)
if err != nil {
return fmt.Errorf("failed to open cache file: %w", err)
}
defer file.Close()
if err = t.Storage.Save(vctx, file, t.Path); err != nil {
uploadProgress, tracksUpload := t.Progress.(UploadProgressTracker)
if !tracksUpload {
if err = t.Storage.Save(vctx, file, t.Path); err != nil {
return fmt.Errorf("failed to save file: %w", err)
}
return nil
}
uploadProgress.OnUploadStart(vctx, t, fileStat.Size())
onProgress := func(uploaded, total int64) {
uploadProgress.OnUploadProgress(vctx, t, uploaded, total)
}
if progressSaver, ok := t.Storage.(storage.StorageProgressSaver); ok {
err = progressSaver.SaveWithProgress(vctx, file, t.Path, onProgress)
} else {
var reader io.Reader = ioutil.NewProgressReader(file, fileStat.Size(), onProgress)
err = t.Storage.Save(vctx, reader, t.Path)
}
if err != nil {
return fmt.Errorf("failed to save file: %w", err)
}
return nil
@@ -73,3 +99,11 @@ func (t *Task) Execute(ctx context.Context) error {
}
return nil
}
func sourceCaption(file tfilepkg.TGFile) (string, bool) {
messageFile, ok := file.(tfilepkg.TGFileMessage)
if !ok || messageFile.Message() == nil {
return "", false
}
return messageFile.Message().GetMessage(), true
}

View File

@@ -4,16 +4,17 @@ import (
"context"
"errors"
"fmt"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/charmbracelet/log"
"github.com/gotd/td/telegram/message/entity"
"github.com/gotd/td/telegram/message/styling"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
@@ -23,153 +24,319 @@ type ProgressTracker interface {
OnDone(ctx context.Context, info TaskInfo, err error)
}
// UploadProgressTracker optionally extends a task progress tracker with a
// distinct upload phase. Keeping it separate preserves compatibility with
// custom download-only trackers.
type UploadProgressTracker interface {
OnUploadStart(ctx context.Context, info TaskInfo, total int64)
OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64)
}
type Progress struct {
MessageID int
ChatID int64
start time.Time
lastUpdatePercent atomic.Int32
lastUpdateAt atomic.Int64
updateMu sync.Mutex
uploadAttempt int
uploadedBytes int64
actualSize int64
hasActualSize bool
}
const (
uploadProgressMinInterval = time.Second
uploadProgressMaxInterval = 3 * time.Second
singleProgressBarWidth = 10
maxSingleErrorRunes = 240
)
type singleProgressPhase int
const (
singlePhaseDownloading singleProgressPhase = iota
singlePhaseUploading
singlePhaseRetrying
)
type renderedSingleMessage struct {
Text string
Entities []tg.MessageEntityClass
Err error
}
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
p.updateMu.Lock()
defer p.updateMu.Unlock()
p.start = time.Now()
p.lastUpdatePercent.Store(0)
p.lastUpdateAt.Store(0)
p.uploadAttempt = 0
p.uploadedBytes = 0
p.actualSize = 0
p.hasActualSize = false
log.FromContext(ctx).Debugf("Progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressFileStartPrefix, nil)),
styling.Code(info.FileName()),
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
styling.Plain(i18n.T(i18nk.BotMsgProgressFileSizePrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB", float64(info.FileSize())/(1024*1024))),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
return
}
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(info, singlePhaseDownloading, 0, info.FileSize(), 0, 0), true)
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo, downloaded, total int64) {
if !shouldUpdateProgress(total, downloaded, int(p.lastUpdatePercent.Load())) {
p.updateMu.Lock()
defer p.updateMu.Unlock()
now := time.Now()
elapsed := uploadProgressMaxInterval
if lastUpdateAt := p.lastUpdateAt.Load(); lastUpdateAt > 0 {
elapsed = now.Sub(time.Unix(0, lastUpdateAt))
}
if !shouldUpdateSingleDownloadProgress(total, downloaded, int(p.lastUpdatePercent.Load()), elapsed) {
return
}
percent := int32((downloaded * 100) / total)
if p.lastUpdatePercent.Load() == percent {
return
if total > 0 {
percent := int32((downloaded * 100) / total)
if p.lastUpdatePercent.Load() == percent {
return
}
p.lastUpdatePercent.Store(percent)
}
p.lastUpdatePercent.Store(percent)
p.lastUpdateAt.Store(now.UnixNano())
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.FileName(), downloaded, total)
entityBuilder := entity.Builder{}
var entities []tg.MessageEntityClass
if err := styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressFileProcessingPrefix, nil)),
styling.Code(info.FileName()),
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
styling.Plain(i18n.T(i18nk.BotMsgProgressFileSizePrefix, nil)),
styling.Code(fmt.Sprintf("%.2f MB", float64(total)/(1024*1024))),
styling.Plain(i18n.T(i18nk.BotMsgProgressAvgSpeedPrefix, nil)),
styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(downloaded, p.start)/(1024*1024))),
styling.Plain(i18n.T(i18nk.BotMsgProgressCurrentProgressPrefix, nil)),
styling.Bold(fmt.Sprintf("%.2f%%", float64(downloaded)/float64(total)*100)),
); err != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(
info,
singlePhaseDownloading,
downloaded,
total,
dlutil.GetSpeed(downloaded, p.start),
0,
), true)
}
func shouldUpdateSingleDownloadProgress(total, downloaded int64, lastPercent int, elapsed time.Duration) bool {
if total > 0 {
return progressutil.ShouldUpdate(total, downloaded, lastPercent)
}
return downloaded > 0 && elapsed >= uploadProgressMaxInterval
}
func (p *Progress) OnUploadStart(ctx context.Context, info TaskInfo, total int64) {
p.updateMu.Lock()
defer p.updateMu.Unlock()
p.start = time.Now()
p.lastUpdatePercent.Store(0)
p.lastUpdateAt.Store(p.start.UnixNano())
p.uploadAttempt++
p.uploadedBytes = 0
p.actualSize = max(total, 0)
p.hasActualSize = true
log.FromContext(ctx).Debugf("Upload progress tracking started: %s", info.FileName())
phase := singleUploadPhase(p.uploadAttempt)
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(info, phase, 0, total, 0, p.uploadAttempt), true)
}
func (p *Progress) OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64) {
if total <= 0 || uploaded <= 0 {
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
p.updateMu.Lock()
defer p.updateMu.Unlock()
if uploaded > total {
uploaded = total
}
req.SetMessage(text)
req.SetEntities(entities)
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
Rows: []tg.KeyboardButtonRow{
{
Buttons: []tg.KeyboardButtonClass{
tgutil.BuildCancelButton(info.TaskID()),
},
},
}},
)
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
if uploaded < p.uploadedBytes {
return
}
p.uploadedBytes = uploaded
now := time.Now()
lastUpdateAt := time.Unix(0, p.lastUpdateAt.Load())
lastPercent := int(p.lastUpdatePercent.Load())
if !shouldUpdateUploadProgress(total, uploaded, lastPercent, now.Sub(lastUpdateAt)) {
return
}
percent := int32((uploaded * 100) / total)
p.lastUpdatePercent.Store(percent)
p.lastUpdateAt.Store(now.UnixNano())
log.FromContext(ctx).Debugf("Upload progress update: %s, %d/%d", info.FileName(), uploaded, total)
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(
info,
singleUploadPhase(p.uploadAttempt),
uploaded,
total,
dlutil.GetSpeed(uploaded, p.start),
p.uploadAttempt,
), true)
}
func shouldUpdateUploadProgress(total, uploaded int64, lastPercent int, elapsed time.Duration) bool {
if total <= 0 || uploaded <= 0 {
return false
}
if uploaded >= total {
return lastPercent < 100 && elapsed >= uploadProgressMinInterval
}
percent := int((uploaded * 100) / total)
if percent < lastPercent {
return false
}
if elapsed < uploadProgressMinInterval {
return false
}
if percent == lastPercent {
return elapsed >= uploadProgressMaxInterval
}
return progressutil.ShouldUpdate(total, uploaded, lastPercent) || elapsed >= uploadProgressMaxInterval
}
func singleUploadPhase(attempt int) singleProgressPhase {
if attempt > 1 {
return singlePhaseRetrying
}
return singlePhaseUploading
}
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
p.updateMu.Lock()
defer p.updateMu.Unlock()
if err != nil {
log.FromContext(ctx).Errorf("Progress error for file [%s]: %v", info.FileName(), err)
} else {
log.FromContext(ctx).Debugf("Progress done for file [%s]", info.FileName())
}
entityBuilder := entity.Builder{}
var stylingErr error
p.editMessage(ctx, info.TaskID(), buildSingleDoneMessage(info, p.doneSize(info), err), false)
}
if err != nil {
if errors.Is(err, context.Canceled) {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskCanceled, nil)),
styling.Plain("\n"),
styling.Plain(i18n.T(i18nk.BotMsgProgressFileNamePrefix, nil)),
styling.Code(info.FileName()),
)
} else {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadFailedPrefix, nil)),
styling.Code(info.FileName()),
styling.Plain(i18n.T(i18nk.BotMsgProgressErrorPrefix, nil)),
styling.Bold(err.Error()),
)
}
} else {
stylingErr = styling.Perform(&entityBuilder,
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadDonePrefix, nil)),
styling.Code(info.FileName()),
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
)
func (p *Progress) doneSize(info TaskInfo) int64 {
if p.hasActualSize {
return p.actualSize
}
return max(info.FileSize(), 0)
}
if stylingErr != nil {
log.FromContext(ctx).Errorf("Failed to build entities: %s", stylingErr)
func (p *Progress) editMessage(ctx context.Context, taskID string, message renderedSingleMessage, cancellable bool) {
if message.Err != nil {
log.FromContext(ctx).Errorf("Failed to render file progress message: %v", message.Err)
return
}
text, entities := entityBuilder.Complete()
req := &tg.MessagesEditMessageRequest{
ID: p.MessageID,
req := buildSingleEditMessageRequest(p.MessageID, taskID, message, cancellable)
if ext := tgutil.ExtFromContext(ctx); ext != nil {
if _, err := ext.EditMessage(p.ChatID, req); err != nil {
log.FromContext(ctx).Errorf("Failed to edit file progress message: %v", err)
}
}
req.SetMessage(text)
req.SetEntities(entities)
}
ext := tgutil.ExtFromContext(ctx)
if ext != nil {
ext.EditMessage(p.ChatID, req)
func buildSingleEditMessageRequest(messageID int, taskID string, message renderedSingleMessage, cancellable bool) *tg.MessagesEditMessageRequest {
req := &tg.MessagesEditMessageRequest{ID: messageID}
req.SetMessage(message.Text)
if len(message.Entities) > 0 {
req.SetEntities(message.Entities)
}
if cancellable {
req.SetReplyMarkup(&tg.ReplyInlineMarkup{Rows: []tg.KeyboardButtonRow{{
Buttons: []tg.KeyboardButtonClass{tgutil.BuildCancelButton(taskID)},
}}})
}
return req
}
func buildSingleProgressMessage(
info TaskInfo,
phase singleProgressPhase,
current int64,
total int64,
speed float64,
attempt int,
) renderedSingleMessage {
if current < 0 {
current = 0
}
if total > 0 && current > total {
current = total
}
percent := singleProgressPercent(current, total)
destination := fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())
data := map[string]any{
"Name": info.FileName(),
"Bar": singleProgressBar(percent),
"Progress": percent,
"Speed": singleProgressSpeed(speed),
"Current": dlutil.FormatSize(current),
"Size": dlutil.FormatSize(total),
"Destination": destination,
"Attempt": max(attempt, 1),
}
var key i18nk.Key
switch phase {
case singlePhaseUploading:
key = i18nk.BotMsgProgressSingleUploading
case singlePhaseRetrying:
key = i18nk.BotMsgProgressSingleUploadRetrying
default:
key = i18nk.BotMsgProgressSingleDownloading
if total <= 0 {
key = i18nk.BotMsgProgressSingleDownloadingUnknown
}
}
markup := i18n.T(i18nk.BotMsgProgressSingleStatusHeader, nil) + "\n\n" + localizedProgressMarkup(key, data)
return completeSingleMessage(markup)
}
func buildSingleDoneMessage(info TaskInfo, size int64, err error) renderedSingleMessage {
data := map[string]any{
"Name": info.FileName(),
"Size": dlutil.FormatSize(max(size, 0)),
"Destination": fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath()),
}
var key i18nk.Key
switch {
case err == nil:
key = i18nk.BotMsgProgressSingleDone
case errors.Is(err, context.Canceled):
key = i18nk.BotMsgProgressSingleCanceled
default:
data["Reason"] = truncateSingleError(err.Error())
key = i18nk.BotMsgProgressSingleFailed
}
return completeSingleMessage(localizedProgressMarkup(key, data))
}
func localizedProgressMarkup(key i18nk.Key, data map[string]any) string {
return i18n.T(key, tgutil.EscapeHTMLTemplateData(data))
}
func completeSingleMessage(markup string) renderedSingleMessage {
text, entities, err := tgutil.RenderHTML(markup)
return renderedSingleMessage{Text: text, Entities: entities, Err: err}
}
func singleProgressPercent(current, total int64) int {
if total <= 0 {
return 0
}
return int(min(max(current, 0), total) * 100 / total)
}
func singleProgressBar(percent int) string {
percent = min(max(percent, 0), 100)
filled := percent * singleProgressBarWidth / 100
return strings.Repeat("🟩", filled) + strings.Repeat("⬜️", singleProgressBarWidth-filled)
}
func singleProgressSpeed(speed float64) string {
if speed <= 0 {
return "0 B/s"
}
return dlutil.FormatSize(int64(speed)) + "/s"
}
func truncateSingleError(value string) string {
runes := []rune(value)
if len(runes) <= maxSingleErrorRunes {
return value
}
return string(runes[:maxSingleErrorRunes])
}
type ProgressOption func(*Progress)

View File

@@ -0,0 +1,178 @@
package tfile
import (
"context"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/gotd/td/tg"
"github.com/krau/SaveAny-Bot/common/i18n"
)
type progressTestTaskInfo struct{}
func (progressTestTaskInfo) TaskID() string { return "task" }
func (progressTestTaskInfo) FileName() string { return "file.bin" }
func (progressTestTaskInfo) FileSize() int64 { return 100 << 20 }
func (progressTestTaskInfo) StoragePath() string { return "file.bin" }
func (progressTestTaskInfo) StorageName() string { return "test" }
func TestShouldUpdateUploadProgress(t *testing.T) {
tests := []struct {
name string
total int64
uploaded int64
lastPercent int
elapsed time.Duration
want bool
}{
{name: "invalid total", total: 0, uploaded: 1, want: false},
{name: "no uploaded bytes", total: 100, uploaded: 0, want: false},
{name: "percentage threshold", total: 100 << 20, uploaded: 10 << 20, elapsed: uploadProgressMinInterval, want: true},
{name: "percentage threshold rate limited", total: 100 << 20, uploaded: 10 << 20, elapsed: uploadProgressMinInterval - time.Millisecond, want: false},
{name: "maximum time threshold", total: 100 << 20, uploaded: 1 << 20, elapsed: uploadProgressMaxInterval, want: true},
{name: "below thresholds", total: 100 << 20, uploaded: 1 << 20, elapsed: uploadProgressMaxInterval - time.Millisecond, want: false},
{name: "completion", total: 100, uploaded: 100, lastPercent: 99, elapsed: uploadProgressMinInterval, want: true},
{name: "completion rate limited", total: 100, uploaded: 100, lastPercent: 99, elapsed: uploadProgressMinInterval - time.Millisecond, want: false},
{name: "completion already reported", total: 100, uploaded: 100, lastPercent: 100, elapsed: uploadProgressMinInterval, want: false},
{name: "out of order callback", total: 100, uploaded: 40, lastPercent: 60, elapsed: uploadProgressMaxInterval, want: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := shouldUpdateUploadProgress(tt.total, tt.uploaded, tt.lastPercent, tt.elapsed)
if got != tt.want {
t.Fatalf("shouldUpdateUploadProgress() = %v, want %v", got, tt.want)
}
})
}
}
func TestUploadProgressConcurrentCallbacks(t *testing.T) {
progress := new(Progress)
ctx := context.Background()
info := progressTestTaskInfo{}
const total = int64(100 << 20)
progress.OnUploadStart(ctx, info, total)
progress.lastUpdateAt.Store(time.Now().Add(-uploadProgressMaxInterval).UnixNano())
var wg sync.WaitGroup
for uploaded := int64(1 << 20); uploaded <= total; uploaded += 1 << 20 {
uploaded := uploaded
wg.Go(func() {
progress.OnUploadProgress(ctx, info, uploaded, total)
})
}
wg.Wait()
percent := progress.lastUpdatePercent.Load()
if percent <= 0 || percent > 100 {
t.Fatalf("last upload percentage = %d, want a value in (0, 100]", percent)
}
if progress.uploadedBytes != total {
t.Fatalf("maximum uploaded bytes = %d, want %d", progress.uploadedBytes, total)
}
}
func TestSingleUploadRetryKeepsRichProgressLayout(t *testing.T) {
i18n.Init("zh-Hans")
t.Cleanup(func() { i18n.Init("zh-Hans") })
message := buildSingleProgressMessage(
progressTestTaskInfo{},
singleUploadPhase(2),
25<<20,
100<<20,
5<<20,
2,
)
if message.Err != nil {
t.Fatalf("buildSingleProgressMessage() failed: %v", message.Err)
}
for _, want := range []string{
"🔁 上传重试",
"🟩🟩⬜️⬜️⬜️⬜️⬜️⬜️⬜️⬜️ 25%",
"尝试次数2",
"速度5.00 MB/s",
"大小25.00 MB / 100.00 MB",
} {
if !strings.Contains(message.Text, want) {
t.Fatalf("retry progress does not contain %q:\n%s", want, message.Text)
}
}
}
func TestSingleProgressTemplateOwnsStylesAndEscapesValues(t *testing.T) {
i18n.Init("en")
t.Cleanup(func() { i18n.Init("zh-Hans") })
info := htmlProgressTestTaskInfo{}
message := buildSingleProgressMessage(info, singlePhaseDownloading, 50, 100, 25, 0)
if message.Err != nil {
t.Fatalf("buildSingleProgressMessage() failed: %v", message.Err)
}
for _, want := range []string{
`<b>A&B</b>.bin`,
`[store<&>]:dir/<i>x</i>&`,
"Speed: 25 B/s",
} {
if !strings.Contains(message.Text, want) {
t.Fatalf("progress text does not contain %q:\n%s", want, message.Text)
}
}
bold, code, blockquote, italic := singleEntityCounts(message.Entities)
if bold != 2 || code != 6 || blockquote != 1 || italic != 0 {
t.Fatalf("progress entity counts = bold:%d code:%d blockquote:%d italic:%d", bold, code, blockquote, italic)
}
failure := buildSingleDoneMessage(info, 100, errors.New(`<i>remote & failed</i>`))
if failure.Err != nil {
t.Fatalf("buildSingleDoneMessage() failed: %v", failure.Err)
}
if !strings.Contains(failure.Text, `<i>remote & failed</i>`) {
t.Fatalf("failure reason was not preserved literally:\n%s", failure.Text)
}
bold, code, blockquote, italic = singleEntityCounts(failure.Entities)
if bold != 1 || code != 2 || blockquote != 0 || italic != 0 {
t.Fatalf("failure entity counts = bold:%d code:%d blockquote:%d italic:%d", bold, code, blockquote, italic)
}
}
func TestSingleDoneSizeUsesActualUploadSize(t *testing.T) {
progress := new(Progress)
info := progressTestTaskInfo{}
progress.OnStart(context.Background(), info)
progress.OnUploadStart(context.Background(), info, 2048)
if got := progress.doneSize(info); got != 2048 {
t.Fatalf("done size = %d, want actual upload size 2048", got)
}
}
type htmlProgressTestTaskInfo struct{}
func (htmlProgressTestTaskInfo) TaskID() string { return "html-task" }
func (htmlProgressTestTaskInfo) FileName() string { return `<b>A&B</b>.bin` }
func (htmlProgressTestTaskInfo) FileSize() int64 { return 100 }
func (htmlProgressTestTaskInfo) StoragePath() string { return `dir/<i>x</i>&` }
func (htmlProgressTestTaskInfo) StorageName() string { return `store<&>` }
func singleEntityCounts(entities []tg.MessageEntityClass) (bold, code, blockquote, italic int) {
for _, messageEntity := range entities {
switch messageEntity.(type) {
case *tg.MessageEntityBold:
bold++
case *tg.MessageEntityCode:
code++
case *tg.MessageEntityBlockquote:
blockquote++
case *tg.MessageEntityItalic:
italic++
}
}
return
}

View File

@@ -1,32 +0,0 @@
package tfile
var progressUpdatesLevels = []struct {
size int64 // 文件大小阈值
stepPercent int // 每多少 % 更新一次
}{
{10 << 20, 100},
{50 << 20, 20},
{200 << 20, 10},
{500 << 20, 5},
}
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
if total <= 0 || downloaded <= 0 {
return false
}
percent := int((downloaded * 100) / total)
if percent <= lastUpdatePercent {
return false
}
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
for _, lvl := range progressUpdatesLevels {
if total < lvl.size {
step = lvl.stepPercent
break
}
}
return percent >= lastUpdatePercent+step
}

View File

@@ -4,6 +4,8 @@ import (
"context"
"io"
"sync/atomic"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
)
type ProgressWriterAt struct {
@@ -20,9 +22,16 @@ func (w *ProgressWriterAt) WriteAt(p []byte, off int64) (int, error) {
if err != nil {
return 0, err
}
downloaded := w.downloaded.Add(int64(at))
if w.progress != nil {
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
w.progress.OnProgress(w.ctx, w.info, downloaded, w.total)
}
taskevent.Emit(w.ctx, taskevent.Event{
TaskID: w.info.TaskID(),
Phase: taskevent.PhaseProgress,
TotalBytes: w.total,
DownloadedBytes: downloaded,
})
return at, nil
}
@@ -56,9 +65,16 @@ func (w *ProgressWriter) Write(p []byte) (int, error) {
if err != nil {
return 0, err
}
downloaded := w.downloaded.Add(int64(at))
if w.progress != nil {
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
w.progress.OnProgress(w.ctx, w.info, downloaded, w.total)
}
taskevent.Emit(w.ctx, taskevent.Event{
TaskID: w.info.TaskID(),
Phase: taskevent.PhaseProgress,
TotalBytes: w.total,
DownloadedBytes: downloaded,
})
return at, nil
}

View File

@@ -0,0 +1,75 @@
package transfer_test
import (
"context"
"errors"
"io"
"os"
"path/filepath"
"testing"
"github.com/krau/SaveAny-Bot/config"
storconfig "github.com/krau/SaveAny-Bot/config/storage"
"github.com/krau/SaveAny-Bot/core/tasks/transfer"
storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
"github.com/krau/SaveAny-Bot/storage"
)
// initConfig seeds the global config so task execution reads a sane Workers
// value (the zero default would deadlock errgroup.SetLimit(0)).
func initConfig(t *testing.T) {
t.Helper()
path := filepath.Join(t.TempDir(), "config.toml")
if err := os.WriteFile(path, []byte("workers = 2\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := config.Init(t.Context(), path); err != nil {
t.Fatal(err)
}
}
type cancelSource struct{}
func (cancelSource) Init(context.Context, storconfig.StorageConfig) error { return nil }
func (cancelSource) Type() storenum.StorageType { return storenum.Local }
func (cancelSource) Name() string { return "cancel-source" }
func (cancelSource) Save(context.Context, io.Reader, string) error { return nil }
func (cancelSource) Exists(context.Context, string) bool { return false }
func (cancelSource) ListFiles(context.Context, string) ([]storagetypes.FileInfo, error) {
return nil, nil
}
// OpenFile reports cancellation, as the task context would be cancelled.
func (cancelSource) OpenFile(ctx context.Context, path string) (io.ReadCloser, int64, error) {
return nil, 0, ctx.Err()
}
type voidTarget struct{}
func (voidTarget) Init(context.Context, storconfig.StorageConfig) error { return nil }
func (voidTarget) Type() storenum.StorageType { return storenum.Local }
func (voidTarget) Name() string { return "void-target" }
func (voidTarget) Save(context.Context, io.Reader, string) error { return nil }
func (voidTarget) Exists(context.Context, string) bool { return false }
var (
_ storage.StorageReadable = cancelSource{}
_ storage.Storage = voidTarget{}
)
// Regression: IgnoreErrors must swallow element failures but never a task
// cancellation; Execute must surface context.Canceled.
func TestIgnoreErrorsPropagatesCancel(t *testing.T) {
initConfig(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
elem := transfer.NewTaskElement(cancelSource{}, storagetypes.FileInfo{Path: "/a", Name: "a", Size: 1}, voidTarget{}, "/out")
task := transfer.NewTransferTask("id", ctx, []transfer.TaskElement{*elem}, nil, true)
err := task.Execute(ctx)
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got %v", err)
}
}

View File

@@ -2,6 +2,7 @@ package transfer
import (
"context"
"errors"
"fmt"
"io"
"os"
@@ -11,6 +12,7 @@ import (
"github.com/charmbracelet/log"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"github.com/krau/SaveAny-Bot/storage"
"golang.org/x/sync/errgroup"
)
@@ -19,7 +21,9 @@ import (
func (t *Task) Execute(ctx context.Context) error {
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("transfer[%s]", t.ID))
logger.Info("Starting transfer task")
t.Progress.OnStart(ctx, t)
if t.Progress != nil {
t.Progress.OnStart(ctx, t)
}
workers := config.C().Workers
eg, gctx := errgroup.WithContext(ctx)
@@ -27,14 +31,11 @@ func (t *Task) Execute(ctx context.Context) error {
for _, elem := range t.elems {
eg.Go(func() error {
t.processingMu.RLock()
t.processingMu.Lock()
if t.processing[elem.ID] != nil {
t.processingMu.RUnlock()
t.processingMu.Unlock()
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
}
t.processingMu.RUnlock()
t.processingMu.Lock()
t.processing[elem.ID] = &elem
t.processingMu.Unlock()
@@ -45,7 +46,7 @@ func (t *Task) Execute(ctx context.Context) error {
}()
err := t.processElement(gctx, elem)
if err != nil && !t.IgnoreErrors {
if err != nil && (!t.IgnoreErrors || errors.Is(err, context.Canceled)) {
return err
}
if err != nil {
@@ -65,7 +66,9 @@ func (t *Task) Execute(ctx context.Context) error {
logger.Info("Transfer task completed successfully")
}
t.Progress.OnDone(ctx, t, err)
if t.Progress != nil {
t.Progress.OnDone(ctx, t, err)
}
return err
}
@@ -115,7 +118,15 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
}
t.uploaded.Add(size)
t.Progress.OnProgress(ctx, t)
if t.Progress != nil {
t.Progress.OnProgress(ctx, t)
}
taskevent.Emit(ctx, taskevent.Event{
TaskID: t.ID,
Phase: taskevent.PhaseProgress,
TotalBytes: t.totalSize,
DownloadedBytes: t.uploaded.Load(),
})
logger.Info("File uploaded successfully")
return nil

View File

@@ -14,6 +14,7 @@ import (
"github.com/krau/SaveAny-Bot/common/i18n"
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
)
@@ -83,7 +84,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
}
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
if !shouldUpdateProgress(info.TotalSize(), info.Uploaded(), int(p.lastUpdatePercent.Load())) {
if !progressutil.ShouldUpdate(info.TotalSize(), info.Uploaded(), int(p.lastUpdatePercent.Load())) {
return
}
percent := int((info.Uploaded() * 100) / info.TotalSize())
@@ -221,14 +222,6 @@ func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
}
}
func shouldUpdateProgress(total, current int64, lastPercent int) bool {
if total == 0 {
return false
}
currentPercent := int((current * 100) / total)
return currentPercent > lastPercent && currentPercent%5 == 0
}
func formatDuration(d time.Duration) string {
d = d.Round(time.Second)
h := d / time.Hour

View File

@@ -85,12 +85,10 @@ func (t *Task) downloadFiles(ctx context.Context, tempDir string) ([]string, err
cmd := ytdlp.New().
Output(filepath.Join(tempDir, "%(title)s.%(ext)s"))
// If no custom flags are provided, use default behavior
// Apply config-based format/quality defaults only when the user passes no
// custom flags. Any user flag means they take full control of yt-dlp.
if len(t.Flags) == 0 {
cmd = cmd.
FormatSort("res,ext:mp4:m4a").
RecodeVideo("mp4").
RestrictFilenames()
cmd = applyFormatConfig(cmd, config.C().Ytdlp)
}
// Note: If custom flags are provided, users have full control over format/quality
// The output path is always set above to ensure downloads go to the correct directory

View File

@@ -0,0 +1,40 @@
package ytdlp
import (
"strconv"
ytdlp "github.com/lrstanley/go-ytdlp"
"github.com/krau/SaveAny-Bot/config"
)
// buildFormatSelector translates a max height into a yt-dlp format selector.
// It prefers merging the best video+audio within the height limit, then falls
// back to a single muxed stream. An empty result means "no explicit selector".
func buildFormatSelector(maxHeight int) string {
if maxHeight <= 0 {
return ""
}
h := strconv.Itoa(maxHeight)
return "bv*[height<=" + h + "]+ba/b[height<=" + h + "]/b"
}
// applyFormatConfig configures format/quality on the yt-dlp command according to
// the ytdlp config. It is only meant to be called when the user did not supply
// any custom flags, so config-driven defaults never conflict with user input.
func applyFormatConfig(cmd *ytdlp.Command, cfg config.YtdlpConfig) *ytdlp.Command {
switch {
case cfg.Format != "":
cmd = cmd.Format(cfg.Format)
case cfg.MaxHeight > 0:
cmd = cmd.Format(buildFormatSelector(cfg.MaxHeight))
default:
// Preserve the original default: prefer highest resolution mp4/m4a.
cmd = cmd.FormatSort("res,ext:mp4:m4a")
}
if cfg.Recode != "" {
cmd = cmd.RecodeVideo(cfg.Recode)
}
cmd = cmd.RestrictFilenames()
return cmd
}

View File

@@ -0,0 +1,23 @@
package ytdlp
import "testing"
func TestBuildFormatSelector(t *testing.T) {
tests := []struct {
name string
maxHeight int
want string
}{
{"no limit", 0, ""},
{"negative", -1, ""},
{"1080p", 1080, "bv*[height<=1080]+ba/b[height<=1080]/b"},
{"720p", 720, "bv*[height<=720]+ba/b[height<=720]/b"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := buildFormatSelector(tt.maxHeight); got != tt.want {
t.Errorf("buildFormatSelector(%d) = %q, want %q", tt.maxHeight, got, tt.want)
}
})
}
}

View File

@@ -16,6 +16,7 @@ type User struct {
WatchChats []WatchChat
FilenameStrategy string
FilenameTemplate string
ConflictStrategy string
}
type WatchChat struct {

View File

@@ -113,6 +113,51 @@ secret = "your-rpc-secret"
remove_after_transfer = true
```
### yt-dlp Configuration
Configures the behavior of the `/ytdlp` command and the `ytdlp` HTTP-API task type when no custom flags are passed.
- `max_height`: Default maximum video resolution by height in pixels (e.g. `1080`, `720`). `0` means no limit (best available). Ignored when `format` is set.
- `format`: A raw yt-dlp format selector (`-f`). When set, it takes precedence over `max_height` and gives you full control, e.g. `bv*[height<=720]+ba/b`.
- `recode`: The target video container yt-dlp recodes into after download (e.g. `mp4`). Leave empty to disable recoding.
{{< hint info >}}
These defaults only apply when using the `/ytdlp` command without passing any custom flags. Passing custom flags on the command (or `flags` in the API) overrides them.
{{< /hint >}}
```toml
[ytdlp]
max_height = 1080
format = "" # e.g. "bv*[height<=720]+ba/b"
recode = "mp4" # empty disables recoding
```
### HTTP API Configuration
When enabled, SaveAny-Bot exposes an HTTP API for creating/querying/canceling tasks programmatically. See [HTTP API](../../usage/api) for the full endpoint reference.
- `enable`: Whether to enable the HTTP API server, default is `false`.
- `host`: Bind address, default `0.0.0.0`.
- `port`: Listen port, default `8080`.
- `token`: Authentication token. **Strongly recommended** — if empty, the API is exposed without any authentication.
```toml
[api]
enable = false
host = "0.0.0.0"
port = 8080
token = "your-token"
```
### Log Configuration
- `level`: Log level. One of `debug`, `info`, `warn`, `error`, `fatal`. Default is `info`.
```toml
[log]
level = "info"
```
### Storage Endpoints List
The storage endpoints list is used to define the storage locations supported by the Bot. Each storage endpoint needs to specify a name, type, and related configuration, using the double bracket syntax `[[storages]]`.

View File

@@ -79,7 +79,8 @@ Stream mode is not supported.
chat_id = "123456789" # Telegram chat ID, the bot will send files to this chat
force_file = false # Force sending as file, default is false
skip_large = false # Skip large files, default is false. If enabled, files exceeding Telegram's limit will not be uploaded.
spilt_size_mb = 2000 # Split size in MB, default is 2000 MB (2 GB). Files larger than this will be split into multiple parts (zip format). Ignored when skip_large is true.
split_large_video = false # Losslessly split oversized videos into one album of up to 10 independently playable parts. Falls back to ZIP parts on failure or when more than 10 parts are required.
split_size_mb = 0 # Split size in MB. 0 uses the uploader account limit: 2000 MB for bots/regular users and 4000 MB for Premium users. Oversized non-video files use ZIP parts. Ignored when skip_large is true.
```
## Rclone
@@ -136,4 +137,4 @@ remote = "myremote"
base_path = "/backup"
config_path = "/path/to/rclone.conf"
flags = ["--progress"]
```
```

Some files were not shown because too many files have changed in this diff Show More