mirror of
https://github.com/krau/SaveAny-Bot.git
synced 2026-08-19 11:23:57 +08:00
Compare commits
75 Commits
v0.57.1
...
fix/qualit
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
942022969c | ||
|
|
ab89310325 | ||
|
|
b27bb734be | ||
|
|
fc0c8975a0 | ||
|
|
524a259ae4 | ||
|
|
e8257c46d2 | ||
|
|
6a11b8ac8d | ||
|
|
47d941d851 | ||
|
|
7e502442c5 | ||
|
|
a9425cb5ea | ||
|
|
6ad2a2c884 | ||
|
|
0d42c9f23d | ||
|
|
4c8f35ae80 | ||
|
|
0a67b8bb57 | ||
|
|
196bb9941f | ||
|
|
973f3499a9 | ||
|
|
d9cb5f9cfc | ||
|
|
ccb702148e | ||
|
|
85b1f35dc5 | ||
|
|
9f93a95258 | ||
|
|
189bf9c736 | ||
|
|
534ed7a7c2 | ||
|
|
1d4997ba2d | ||
|
|
f7e532ca7e | ||
|
|
0957d93da3 | ||
|
|
324d8c100f | ||
|
|
4cb23f04b6 | ||
|
|
a92ca75c4f | ||
|
|
4e99b4bdc9 | ||
|
|
389be59371 | ||
|
|
0d49ae94af | ||
|
|
48e739f06d | ||
|
|
b4fbbae068 | ||
|
|
29be3ee90c | ||
|
|
56660a7705 | ||
|
|
0dfb6af153 | ||
|
|
63f53fac7f | ||
|
|
607ec8aced | ||
|
|
cd9886d710 | ||
|
|
3f7f50133e | ||
|
|
c7911cc208 | ||
|
|
b31d628c19 | ||
|
|
60fd2e04d9 | ||
|
|
b72dd67be9 | ||
|
|
aa25eb1510 | ||
|
|
ba0deababc | ||
|
|
d1b2dbfe5f | ||
|
|
bd7da7c31e | ||
|
|
1981d8662b | ||
|
|
51fcd17922 | ||
|
|
610f586d45 | ||
|
|
6bc12d6feb | ||
|
|
ad41aec22c | ||
|
|
4389bfe046 | ||
|
|
832eb27d4f | ||
|
|
b6e981282d | ||
|
|
9ed5b5f962 | ||
|
|
4527022853 | ||
|
|
6fd95bbe1b | ||
|
|
a1bd2ed762 | ||
|
|
7e3e26fcf4 | ||
|
|
e49ebef977 | ||
|
|
da38f5bb88 | ||
|
|
0e6ed66ef5 | ||
|
|
e4144e73e6 | ||
|
|
1d794a7b9c | ||
|
|
52f880f0f2 | ||
|
|
fc11ca775f | ||
|
|
056e2fd546 | ||
|
|
c9bb6c9e3c | ||
|
|
2bc460c609 | ||
|
|
f02860ff3f | ||
|
|
9c2e70ed43 | ||
|
|
3d6cd45909 | ||
|
|
77ef3154cf |
4
.github/workflows/build-release.yml
vendored
4
.github/workflows/build-release.yml
vendored
@@ -43,7 +43,7 @@ jobs:
|
|||||||
goarch: arm64
|
goarch: arm64
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@v4
|
uses: actions/checkout@v6
|
||||||
|
|
||||||
- name: Extract version from Git Ref
|
- name: Extract version from Git Ref
|
||||||
id: extract_version
|
id: extract_version
|
||||||
@@ -64,7 +64,7 @@ jobs:
|
|||||||
ldflags: >-
|
ldflags: >-
|
||||||
-s -w
|
-s -w
|
||||||
-X "github.com/krau/SaveAny-Bot/config.Version=${{ env.VERSION }}"
|
-X "github.com/krau/SaveAny-Bot/config.Version=${{ env.VERSION }}"
|
||||||
-X "github.com/krau/SaveAny-Bot/config.BuildTime=${{ format(github.event.repository.updated_at, 'yyyy-MM-dd HH:mm:ss') }}"
|
-X "github.com/krau/SaveAny-Bot/config.BuildTime=${{ github.event.repository.updated_at }}"
|
||||||
-X "github.com/krau/SaveAny-Bot/config.GitCommit=${{ github.sha }}"
|
-X "github.com/krau/SaveAny-Bot/config.GitCommit=${{ github.sha }}"
|
||||||
binary_name: saveany-bot
|
binary_name: saveany-bot
|
||||||
env:
|
env:
|
||||||
|
|||||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -1,5 +1,6 @@
|
|||||||
config.toml
|
config.toml
|
||||||
logs/
|
logs/
|
||||||
|
/cache/
|
||||||
tmp/
|
tmp/
|
||||||
data/
|
data/
|
||||||
downloads/
|
downloads/
|
||||||
|
|||||||
366
AGENTS.md
366
AGENTS.md
@@ -1,301 +1,115 @@
|
|||||||
# SaveAny-Bot Agent Guidelines
|
# Repository Guidelines
|
||||||
|
|
||||||
This document provides essential information for AI coding agents working on the SaveAny-Bot project.
|
|
||||||
|
|
||||||
## Project Overview
|
## Project Overview
|
||||||
|
|
||||||
SaveAny-Bot is a Telegram bot written in Go that saves files/messages from Telegram and various websites to multiple storage backends (local, S3, MinIO, WebDAV, AList, Telegram). It features a plugin system for parsing web content and extensible storage backends.
|
SaveAny-Bot is a Telegram bot written in Go that saves files and messages from Telegram and websites to multiple storage backends (Local, S3, MinIO, WebDAV, AList, Rclone, Telegram). It supports single-file saves, batch/album saves, streaming, multi-user access, storage rules, cross-storage transfers, yt-dlp and Aria2 downloads, and a Goja-based JavaScript parser plugin system with optional Playwright browser automation.
|
||||||
|
|
||||||
**Tech Stack**: Go 1.24.2, gotd/td (Telegram MTProto), Cobra (CLI), Viper (config), GORM (ORM), SQLite, Goja (JS runtime), Playwright (browser automation)
|
**Tech stack**: Go 1.25, gotgproto + gotd/td v0.149.0 (MTProto), Cobra (CLI), Viper (config), GORM + SQLite, Goja (JS runtime), Playwright, charmbracelet/log. License: AGPL-3.0.
|
||||||
|
|
||||||
## Build & Test Commands
|
**Note on gotd versions**: `gotd/td` must stay on `v0.149.0` — v0.150+ breaks `gotgproto` (v1.0.0-beta22) compilation (`AsInputDocumentFileLocation` signature, `gotd/log` Logger interface). Do not bump beyond v0.149.
|
||||||
|
|
||||||
|
## Architecture & Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Telegram update → client/bot/handlers → core.AddTask(Executable) → pkg/queue (serial workers)
|
||||||
|
→ core/tasks/* (download via common/tdler) → storage.Storage → progress feedback
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Startup sequence** (`cmd/run.go::initAll`, keep this order): Config → Cache → i18n → Database → Storage → Parser plugins → Userbot → API → Bot. `bot.Init` returns the exit channel; a `SAVEANTBOT-RESTART` error restarts the process (external supervisor).
|
||||||
|
- **Task pipeline**: handlers build a task (via `core.AddTask`), the queue executes `Executable{Type, Title, TaskID, Execute(ctx)}` with `config.C().Workers` workers. Lifecycle hooks (`TaskBeforeStart/Success/Fail/Cancel`) run around `Execute`. Cancellation = canceling the task's context; tasks must check `ctx.Err()`.
|
||||||
|
- **Dual progress channels**: (1) `pkg/taskevent` context bus (consumed by `api/` for HTTP/Webhook consumers — `taskevent.WithSink` must be injected for API tasks), (2) Telegram message edits via `ProgressTracker` + `tgutil.ExtFromContext(ctx)` (bot tasks). Upload progress currently reaches the Telegram channel only.
|
||||||
|
- **Capability interfaces + type assertion fallback** is the core extensibility pattern: `StorageBatchSaver`, `StorageProgressSaver` (upload progress), `StorageListable`, `StorageReadable` are optional; consumers assert and fall back (e.g. wrap reader with `ioutil.NewProgressReader`). New backends only need to implement the interface and register.
|
||||||
|
- **Config layering**: CLI flag > env `SAVEANY_*` (dots → underscores, e.g. `SAVEANY_TELEGRAM_TOKEN`) > TOML file (path or http(s) URL). `config.C()` returns a **copy** — never mutate it.
|
||||||
|
- **Storage registration (3 places)**: `pkg/enums/storage` ENUM comment (go-enum), `config/storage/factory.go::storageFactories` (config struct with `Validate()`), `storage/storage.go::storageConstructors` (implementation).
|
||||||
|
|
||||||
|
## Key Directories
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `cmd/` | CLI: `run` (main bot), `upload`, `watch` (standalone subcommands that do NOT run initAll), `geni18n` (i18n key generator) |
|
||||||
|
| `core/` | `Executable` interface, queue worker loop, hooks; `core/tasks/{tfile,batchtfile,directlinks,parsed,telegraph,transfer,ytdlp,aria2dl}` |
|
||||||
|
| `client/bot/` | gotgproto client, `handlers/` (all commands + message/callback handlers), `middleware/`, `client/user/` (userbot) |
|
||||||
|
| `storage/` | 8 backends + `storage.go` (interfaces/registry) + `load.go` (per-user storage resolution) |
|
||||||
|
| `parsers/` | `parsers.go` (registry), `js/` (Goja plugins, ghttp/playwright injection, build-tagged), `parsers/` (native: twitter, kemono) |
|
||||||
|
| `config/` | Viper setup, defaults, `storage/` per-type config structs |
|
||||||
|
| `database/` | GORM models (User/Dir/Rule/WatchChat), AutoMigrate, `syncUsers` |
|
||||||
|
| `pkg/` | `queue`, `taskevent`, `tcbdata` (callback data), `rule`, `enums/{tasktype,storage,ctxkey,fnamest}`, `storagetypes`, `tfile`, `parser` |
|
||||||
|
| `common/` | `tdler` (unified downloader), `utils/{tgutil,dlutil,ioutil,fsutil,strutil,tphutil,netutil}`, `i18n` (embedded locales), `cache` (ristretto) |
|
||||||
|
| `api/` | HTTP API + webhook (task factory with sink injection) |
|
||||||
|
| `docs/` | Hugo site (hugo-book theme, zh+en mirrored), separate go.mod |
|
||||||
|
| `plugins/` | JS parser examples + `README.md` (plugin author contract) |
|
||||||
|
|
||||||
|
## Development Commands
|
||||||
|
|
||||||
### Build
|
|
||||||
```bash
|
```bash
|
||||||
# Standard build
|
# Build (standard; CGO_ENABLED=0 for static)
|
||||||
go build -o saveany-bot .
|
CGO_ENABLED=0 go build -trimpath -o saveany-bot .
|
||||||
|
|
||||||
# Run directly
|
|
||||||
go run ./cmd
|
go run ./cmd
|
||||||
|
|
||||||
# Docker build (multi-stage, Alpine-based)
|
# Test — known failures: storage/telegram TestCreateSplitZip/TestExtractThumbFrame/TestGetVideoMetadata
|
||||||
docker build -t saveany-bot .
|
# (need gitignored fixtures tests/testfile.dat, tests/testvideo; ffmpeg/ffprobe)
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test
|
|
||||||
```bash
|
|
||||||
# Run all tests
|
|
||||||
go test ./...
|
go test ./...
|
||||||
|
go test -race ./core/tasks/... ./storage/... ./pkg/queue/... ./common/...
|
||||||
# Run tests in specific package
|
|
||||||
go test ./pkg/queue
|
|
||||||
go test ./storage/telegram
|
|
||||||
|
|
||||||
# Run tests with verbose output
|
|
||||||
go test -v ./...
|
|
||||||
|
|
||||||
# Run a single test
|
|
||||||
go test -run TestQueueBasic ./pkg/queue
|
go test -run TestQueueBasic ./pkg/queue
|
||||||
|
|
||||||
# Run with coverage
|
# Codegen — run after editing locale YAML or enum comments
|
||||||
go test -cover ./...
|
go generate ./... # geni18n (i18nk keys) + go-enum (pkg/enums/*)
|
||||||
```
|
# go-enum is NOT in go.mod; install externally. geni18n runs via go run.
|
||||||
|
|
||||||
### Lint & Format
|
# Verify
|
||||||
```bash
|
|
||||||
# Format code (standard Go formatting)
|
|
||||||
go fmt ./...
|
|
||||||
|
|
||||||
# Vet code for common issues
|
|
||||||
go vet ./...
|
go vet ./...
|
||||||
|
go fmt ./...
|
||||||
# Generate code (i18n keys)
|
|
||||||
go generate ./...
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Other Commands
|
**Build variants** (Dockerfile.default/micro/pico): `-tags=no_jsparser,no_playwright,no_minio,no_bubbletea,sqlite_glebarez` — each has a `*_stub.go`/`*_glebarez.go` pairing; keep stubs in sync.
|
||||||
```bash
|
|
||||||
# Update dependencies
|
|
||||||
go mod tidy
|
|
||||||
|
|
||||||
# View documentation
|
Docker: `docker build -t saveany-bot .`, `docker compose up -d` (host network, mounts `./data ./config.toml ./downloads ./cache`). CI (`.github/workflows/`) runs **no tests/lint** — only tag-triggered release/docker builds and docs deployment; run `go test ./...` manually before pushing.
|
||||||
cd docs && hugo server -D
|
|
||||||
```
|
|
||||||
|
|
||||||
## Code Style Guidelines
|
## Code Conventions & Common Patterns
|
||||||
|
|
||||||
### Imports
|
- **Imports**: stdlib → third-party → project-internal, blank-line separated. Aliases for clarity (`storconfig`, `storenum`).
|
||||||
- Standard library first, then third-party, then project-internal
|
- **Naming**: PascalCase exported, camelCase unexported, files `snake_case.go`; **not** ALL_CAPS constants.
|
||||||
- Group imports with blank lines between groups
|
- **Errors**: always wrap with `fmt.Errorf("context: %w", err)`; check with `errors.Is/As`; never ignore.
|
||||||
- Use explicit import aliases for clarity when needed (e.g., `storconfig`, `storenum`)
|
- **Logging**: `log.FromContext(ctx)` with prefixes (`logger.WithPrefix("component")`); never global logger when ctx is available.
|
||||||
|
- **Context values** (read from the passed ctx, never globals): `log.FromContext`, `tgutil.ExtFromContext` (Telegram ext — **required for message edits; if nil, edits are silently dropped**), `storage.FromContext`, `storagetypes.WithSourceCaption`, `ctxkey.ContentLength` / `ctxkey.OverwriteExisting`.
|
||||||
|
- **Progress rendering** (#228 convention): i18n templates declare styles with Telegram HTML (`<b>/<code>/<blockquote>/<i>`); dynamic data MUST go through `i18n.T(key, tgutil.EscapeHTMLTemplateData(data))` before `tgutil.RenderHTML`. Never interpolate user data raw, never render-then-substring-search.
|
||||||
|
- **Progress tracking**: each task package defines its own small `ProgressTracker` interface; optional `UploadProgressTracker` is probed via type assertion (skip if absent). Serialize state + message edits with a mutex; throttle edits (≥1s); aggregate per-item progress monotonically.
|
||||||
|
- **i18n**: only edit `common/i18n/locale/{zh-Hans,en}.yaml` → `go generate ./...` → use `i18nk.<Key>` constants. No raw strings in user-facing messages. zh-Hans and en must stay in sync.
|
||||||
|
- **Registration points** (never forget): new bot command → `client/bot/handlers/register.go::CommandHandlers` (auto-publishes /help menu); new task type → `pkg/enums/tasktype` + `core/tasks/<name>/` + `api/factory.go::CreateTask`; new storage → 3 places above + `docs/content/{en,zh}/deployment/configuration/storages.md`; new enum value → ENUM comment + `go generate`.
|
||||||
|
- **Concurrency**: `errgroup.WithContext` + `SetLimit(config.C().Workers)`, `atomic.Int64` counters, `sync.Once` for single-shot events, mutex around render state. No lock-in-callback (callbacks fire after unlock).
|
||||||
|
- **Cancellation**: queue tasks carry a `WithCancel`-derived ctx; check `ctx.Err()` in loops; classify with `errors.Is(err, context.Canceled)`.
|
||||||
|
- **JS plugins**: `registerParser({metadata, canHandle, parse})`, `version >= 1.0.0`; per-plugin goja VM is single-goroutine (reqCh buffer 10). Changing `pkg/parser.Item/Resource` JSON fields requires updating `plugins/README.md` and example plugins.
|
||||||
|
- **Message edits**: `ext.EditMessage(chatID, &tg.MessagesEditMessageRequest{...})`; cancel buttons via `tgutil.BuildCancelButton(taskID)`; callback payloads via `pkg/tcbdata` + `common/cache`.
|
||||||
|
|
||||||
```go
|
## Important Files
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
|
||||||
|
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/storage"
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Formatting
|
- `main.go` — `//go:generate` for i18n keys
|
||||||
- Line length: reasonable (no hard limit, but be sensible)
|
- `cmd/run.go` — startup sequence `Run/initAll/cleanCache` (cache cleanup on exit, `NoCleanCache` opt-out)
|
||||||
- Organize code with blank lines between logical sections
|
- `core/core.go` — worker loop, hooks, AddTask/CancelTask
|
||||||
- Follow standard Go conventions for braces, spacing, etc.
|
- `pkg/queue/queue.go` — generic serial queue (cond/list; duplicate TaskID rejected)
|
||||||
|
- `storage/storage.go` — interfaces + registry + compile-time capability assertions
|
||||||
|
- `config/viper.go`, `config.example.toml` — config schema (authoritative field docs)
|
||||||
|
- `database/db.go` — GORM init, `GetDialect` (build-tag selectable SQLite driver)
|
||||||
|
- `client/bot/handlers/register.go` — handler dispatch order and CommandHandlers
|
||||||
|
- `common/tdler/dler.go` — unified download entry
|
||||||
|
- `core/tasks/batchtfile/item_progress.go` — per-item phase state machine (Downloading/Transferring/Uploading/Retrying/Confirming, FailureStage)
|
||||||
|
- `parsers/js/plugin.go` — Goja plugin runtime
|
||||||
|
- `.github/workflows/` — release/docker/docs (no test gate)
|
||||||
|
|
||||||
### Types & Interfaces
|
## Runtime/Tooling Preferences
|
||||||
- Use clear, descriptive type names (PascalCase for exported, camelCase for unexported)
|
|
||||||
- Define interfaces where abstraction is needed (e.g., `Executable`, `StorageConfig`)
|
|
||||||
- Embed context in method signatures, not structs: `func (s *Service) Do(ctx context.Context) error`
|
|
||||||
- Prefer composition over inheritance
|
|
||||||
|
|
||||||
```go
|
- **Go 1.25+**: `t.Context()`, `sync.WaitGroup.Go`, `for range n` are available.
|
||||||
// Interfaces define behavior
|
- **Runtime binaries**: ffmpeg/ffprobe (media processing/video split), yt-dlp (ytdlp tasks), aria2 optional; Playwright browsers install on demand to `./playwright` (`playwright.Install(chromium, ...)` at first `pw.get()`); Docker images: default has ffmpeg+yt-dlp, micro only curl, pico is scratch static.
|
||||||
type Executable interface {
|
- **No Makefile, no golangci.yml, no test/lint CI** — verification is manual (`go vet`, `go test`).
|
||||||
Type() tasktype.TaskType
|
- **go-enum** required externally for enum generation; **geni18n** is in-repo.
|
||||||
Title() string
|
- **Docs**: Hugo site in `docs/` (separate go.mod, hugo-book); edit `docs/content/{zh,en}/` — keep both languages mirrored. `docs/public/` is gitignored build output.
|
||||||
TaskID() string
|
- **gitignored fixtures**: `storage/telegram/tests/` (missing — 3 tests fail locally), `data/`, `config.toml`, `playwright/`, `testplugins/`.
|
||||||
Execute(ctx context.Context) error
|
|
||||||
}
|
|
||||||
|
|
||||||
// Structs compose behavior
|
## Testing & QA
|
||||||
type Local struct {
|
|
||||||
config config.LocalStorageConfig
|
|
||||||
logger *log.Logger
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Naming Conventions
|
- Pure stdlib `testing` (no testify); table-driven (`[]struct{name...}` + `t.Run`) with `t.Fatalf` got/want assertions. Mock via hand-written interface impls or package-variable replacement (`runMediaTool` in `video_split_test.go`, restored with `t.Cleanup`); in-process services for HTTP (`httptest`), S3 (`gofakes3+s3mem`), WebDAV (`x/net/webdav`).
|
||||||
- **Packages**: lowercase, single word when possible (avoid underscores)
|
- **Locale-dependent tests**: pin with `i18n.Init("zh-Hans")` + `t.Cleanup(...)`.
|
||||||
- **Files**: lowercase with underscores for multiword (e.g., `auth_terminal.go`, `progress_reader.go`)
|
- **Progress/HTML tests**: assert rendered text with `strings.Contains` AND entity counts (`tg.MessageEntityBold/Code/Blockquote/Italic`) — verify style injection stays escaped (`<b>A&B</b>` input must render as literal text).
|
||||||
- **Variables**: camelCase for unexported, PascalCase for exported
|
- **Known failures**: `storage/telegram` `TestCreateSplitZip`, `TestExtractThumbFrame`, `TestGetVideoMetadata` need gitignored fixtures + real ffmpeg — skip with `-skip 'Test(CreateSplitZip|ExtractThumbFrame|GetVideoMetadata)$'`; `api/handlers_test.go` has one `t.Skip` (needs initialized core).
|
||||||
- **Constants**: PascalCase for exported, camelCase for unexported (not ALL_CAPS)
|
- **Coverage expectations**: pure logic gets table tests (parsers, URL/path utils, progress throttling, grouping); regressions get bug-scenario-named tests (`progress_regression_test.go`). Network/Telegram/Playwright must never be touched by tests.
|
||||||
- **Functions/Methods**: PascalCase for exported, camelCase for unexported
|
- When a permanent feature/API change ships: update `config.example.toml` if config, `docs/` if user-facing, `plugins/README.md` if plugin contract, and i18n YAML + `go generate` for new strings.
|
||||||
- **Test files**: `*_test.go` pattern
|
|
||||||
|
|
||||||
### Error Handling
|
|
||||||
- Always handle errors explicitly; never ignore them
|
|
||||||
- Wrap errors with context using `fmt.Errorf("context: %w", err)`
|
|
||||||
- Use `errors.Is()` and `errors.As()` for error checking
|
|
||||||
- Log errors with appropriate level (Error, Warn, Info)
|
|
||||||
- Return errors from functions rather than panicking (except for truly unrecoverable situations)
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Good error handling
|
|
||||||
if err := db.Save(user).Error; err != nil {
|
|
||||||
return fmt.Errorf("failed to save user %d: %w", user.ChatID, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Check specific errors
|
|
||||||
if errors.Is(err, context.Canceled) {
|
|
||||||
logger.Info("Operation was canceled")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Logging
|
|
||||||
- Use `github.com/charmbracelet/log` package
|
|
||||||
- Get logger from context: `log.FromContext(ctx)`
|
|
||||||
- Create prefixed loggers for components: `logger.WithPrefix("component")`
|
|
||||||
- Use appropriate levels: Debug, Info, Warn, Error
|
|
||||||
- Include context in log messages (e.g., task IDs, file names)
|
|
||||||
|
|
||||||
```go
|
|
||||||
logger := log.FromContext(ctx)
|
|
||||||
logger.Infof("Processing task: %s", task.ID)
|
|
||||||
logger.Errorf("Failed to save file %s: %v", filename, err)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Concurrency
|
|
||||||
- Use channels for communication between goroutines
|
|
||||||
- Protect shared state with `sync.Mutex` or `sync.RWMutex`
|
|
||||||
- Use `sync.WaitGroup` for coordinating goroutine completion
|
|
||||||
- Always pass `context.Context` for cancellation support
|
|
||||||
- Use `context.WithCancel/WithTimeout` for managing goroutine lifetimes
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Example from queue implementation
|
|
||||||
func (tq *TaskQueue[T]) Add(task *Task[T]) error {
|
|
||||||
tq.mu.Lock()
|
|
||||||
defer tq.mu.Unlock()
|
|
||||||
// ... critical section
|
|
||||||
tq.cond.Signal()
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Comments
|
|
||||||
- Document exported types, functions, and packages with doc comments
|
|
||||||
- Start doc comments with the name being documented
|
|
||||||
- Use `//` for single-line comments
|
|
||||||
- Explain *why*, not *what* (code should be self-explanatory for "what")
|
|
||||||
- Add `[NOTE]`, `[WARN]`, `[IMPORTANT]` tags for important clarifications
|
|
||||||
|
|
||||||
```go
|
|
||||||
// GetUserByChatID retrieves a user by their Telegram chat ID.
|
|
||||||
// Returns an error if the user is not found.
|
|
||||||
func GetUserByChatID(ctx context.Context, chatID int64) (*User, error) {
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture & Conventions
|
|
||||||
|
|
||||||
### Application Structure
|
|
||||||
- **Entry point**: `main.go` → `cmd.Execute(ctx)`
|
|
||||||
- **CLI root**: `cmd/root.go` (Cobra), implementation in `cmd/run.go`
|
|
||||||
- **Startup sequence**: Config → Cache → i18n → Database → Storage → Parsers → Userbot → Bot → Queue
|
|
||||||
- Follow this order when adding new initialization steps in `cmd/run.go::initAll`
|
|
||||||
|
|
||||||
### Configuration (Viper)
|
|
||||||
- Config defined in `config/viper.go::Config`
|
|
||||||
- Read from `config.toml` (see `config.example.toml`)
|
|
||||||
- Environment variables: `SAVEANY_*` prefix (e.g., `SAVEANY_TELEGRAM_TOKEN`)
|
|
||||||
- Access via `config.C()` (returns a copy, don't modify the return value)
|
|
||||||
- Storage configs validated via `config/storage/factory.go::LoadStorageConfigs`
|
|
||||||
|
|
||||||
### Telegram Client
|
|
||||||
- **Bot client**: `client/bot/bot.go::Init` (uses gotgproto)
|
|
||||||
- **Handlers**: Centralized in `client/bot/handlers/` directory
|
|
||||||
- **Registration**: All handlers registered in `handlers.Register`
|
|
||||||
- **Commands**: Add to `CommandHandlers` slice for automatic `/help` and bot command list updates
|
|
||||||
- **Middleware**: Common middleware in `client/middleware/` (floodwait, retry, etc.)
|
|
||||||
|
|
||||||
### Tasks & Queue
|
|
||||||
- **Task interface**: `core/core.go::Executable` (Type, Title, TaskID, Execute methods)
|
|
||||||
- **Queue**: `pkg/queue.TaskQueue[Executable]` (generic, thread-safe)
|
|
||||||
- **Workers**: Count from `config.C().Workers`
|
|
||||||
- **Task types**: Implementations in `core/tasks/**` (tfile, parsed, telegraph, directlinks, batchtfile)
|
|
||||||
- **Lifecycle hooks**: `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` (defined in config)
|
|
||||||
- **Adding tasks**: Use `core.AddTask(ctx, task)`
|
|
||||||
|
|
||||||
### Database (GORM + SQLite)
|
|
||||||
- **Init**: `database.Init` using `config.C().DB.Path`
|
|
||||||
- **Models**: User, Dir, Rule, WatchChat (in `database/*.go`)
|
|
||||||
- **Migrations**: Automatic via `db.AutoMigrate`
|
|
||||||
- **User sync**: `database.syncUsers` syncs DB with `config.C().Users` (don't manually create/delete users)
|
|
||||||
- **Context**: Always use `db.WithContext(ctx)` for operations
|
|
||||||
|
|
||||||
### Storage Backends
|
|
||||||
- **Interface**: Defined in `config/storage/types.go` and `storage/`
|
|
||||||
- **Implementations**: local, alist, s3/minio, webdav, telegram (each in subdirectory)
|
|
||||||
- **Adding new storage**:
|
|
||||||
1. Add enum to `pkg/enums/storage`
|
|
||||||
2. Create config struct in `config/storage/` with `Validate()` method
|
|
||||||
3. Implement storage in `storage/<name>/`
|
|
||||||
4. Register in `storageFactories` mapping
|
|
||||||
5. Update `config.example.toml` with example
|
|
||||||
|
|
||||||
### Parser Plugins (JavaScript)
|
|
||||||
- **Runtime**: Goja (JS runtime) + Playwright (browser automation)
|
|
||||||
- **Plugin API**: `registerParser({ metadata, canHandle, parse })` in JS
|
|
||||||
- **Integration**: Defined in `parsers/` directory
|
|
||||||
- **Documentation**: See `plugins/README.md`
|
|
||||||
- Plugin `parse` returns `Item`/`Resource` which becomes download/transfer task
|
|
||||||
|
|
||||||
### Internationalization (i18n)
|
|
||||||
- **Usage**: `i18n.T(i18nk.SomeKey, map[string]any{"Name": value})`
|
|
||||||
- **Locale files**: `common/i18n/locale/*.yaml`
|
|
||||||
- **Key generation**: Run `go generate ./...` to generate `common/i18n/i18nk/keys.go`
|
|
||||||
- **Adding new strings**: Add to YAML → run `go generate` → use in code
|
|
||||||
- All user-facing strings should be internationalized
|
|
||||||
|
|
||||||
### Context Usage
|
|
||||||
- Always pass `context.Context` as first parameter
|
|
||||||
- Use `log.FromContext(ctx)` to get contextual logger
|
|
||||||
- Respect context cancellation in long-running operations
|
|
||||||
- Store request-scoped data in context (e.g., `ctxkey.ContentLength`)
|
|
||||||
|
|
||||||
## Special Rules from .github/copilot-instructions.md
|
|
||||||
|
|
||||||
1. **Never modify `config.C()` return values** - it returns a copy. Modify config in `config.Init` or via Viper.
|
|
||||||
2. **Handlers must update `CommandHandlers` slice** - ensures `/help` and bot commands stay in sync.
|
|
||||||
3. **Task execution must preserve hooks** - don't remove `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` hook calls.
|
|
||||||
4. **User sync is automatic** - don't manually create/delete users in DB; use config-based sync.
|
|
||||||
5. **Prefer context logger** - use `log.FromContext(ctx)` over global logger when context is available.
|
|
||||||
6. **Storage factory pattern** - new storage types must register in `storageFactories` mapping.
|
|
||||||
7. **Plugin API compatibility** - changes to `Item`/`Resource` structures require updating `plugins/README.md`.
|
|
||||||
|
|
||||||
## Common Patterns
|
|
||||||
|
|
||||||
### Adding a New Command
|
|
||||||
1. Create handler function in `client/bot/handlers/<name>.go`
|
|
||||||
2. Add to `CommandHandlers` slice in `register.go`
|
|
||||||
3. Add i18n key to `common/i18n/locale/*.yaml`
|
|
||||||
4. Run `go generate ./...`
|
|
||||||
5. Test with Telegram bot
|
|
||||||
|
|
||||||
### Adding a New Task Type
|
|
||||||
1. Create struct implementing `core.Executable` in `core/tasks/<type>/`
|
|
||||||
2. Implement `Type()`, `Title()`, `TaskID()`, `Execute(ctx)` methods
|
|
||||||
3. Add task type enum to `pkg/enums/tasktype`
|
|
||||||
4. Use `core.AddTask(ctx, task)` to enqueue
|
|
||||||
|
|
||||||
### Adding a New Storage Backend
|
|
||||||
1. Define config struct in `config/storage/<name>.go` with `Validate()` method
|
|
||||||
2. Implement storage interface in `storage/<name>/<name>.go`
|
|
||||||
3. Add storage type enum to `pkg/enums/storage`
|
|
||||||
4. Register factory in `config/storage/factory.go::storageFactories`
|
|
||||||
5. Update `config.example.toml` with configuration example
|
|
||||||
|
|
||||||
## File References
|
|
||||||
|
|
||||||
When referencing code locations, use `path/to/file.go:line` format (e.g., `core/core.go:23` for the worker function).
|
|
||||||
|
|
||||||
## Testing Guidelines
|
|
||||||
|
|
||||||
- Write tests for new functionality (place in `*_test.go` files)
|
|
||||||
- Test files should be in same package as code being tested
|
|
||||||
- Use table-driven tests for multiple test cases
|
|
||||||
- Mock external dependencies (databases, network calls)
|
|
||||||
- Aim for meaningful tests, not just coverage numbers
|
|
||||||
|
|
||||||
## Notes
|
|
||||||
|
|
||||||
- Binary size matters: use `CGO_ENABLED=0` for static binaries
|
|
||||||
- FFmpeg is included in Docker images for media processing
|
|
||||||
- Build process supports cross-compilation (amd64/arm64, Linux/macOS/Windows)
|
|
||||||
- Documentation site uses Hugo; edit files in `docs/` directory
|
|
||||||
- Session data stored in SQLite; delete `data/session.db` if changing bot token
|
|
||||||
|
|||||||
@@ -7,6 +7,8 @@ ARG BuildTime="Unknown"
|
|||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
|
RUN apk add --no-cache ca-certificates
|
||||||
|
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN --mount=type=cache,target=/go/pkg/mod \
|
RUN --mount=type=cache,target=/go/pkg/mod \
|
||||||
go mod download
|
go mod download
|
||||||
@@ -31,5 +33,9 @@ FROM scratch
|
|||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
COPY --from=builder /app/saveany-bot .
|
COPY --from=builder /app/saveany-bot .
|
||||||
|
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
|
||||||
|
|
||||||
|
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||||
|
ENV SSL_CERT_DIR=/etc/ssl/certs
|
||||||
|
|
||||||
ENTRYPOINT ["/app/saveany-bot"]
|
ENTRYPOINT ["/app/saveany-bot"]
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package api
|
package api
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
|
||||||
"crypto/subtle"
|
"crypto/subtle"
|
||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -9,9 +8,6 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
// tokenContextKey 用于在 context 中存储 token
|
|
||||||
type tokenContextKey struct{}
|
|
||||||
|
|
||||||
// AuthMiddleware 返回认证中间件
|
// AuthMiddleware 返回认证中间件
|
||||||
func AuthMiddleware() func(http.Handler) http.Handler {
|
func AuthMiddleware() func(http.Handler) http.Handler {
|
||||||
return func(next http.Handler) http.Handler {
|
return func(next http.Handler) http.Handler {
|
||||||
@@ -40,9 +36,7 @@ func AuthMiddleware() func(http.Handler) http.Handler {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 将 token 添加到 context
|
next.ServeHTTP(w, r)
|
||||||
ctx := context.WithValue(r.Context(), tokenContextKey{}, token)
|
|
||||||
next.ServeHTTP(w, r.WithContext(ctx))
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/pkg/aria2"
|
"github.com/krau/SaveAny-Bot/pkg/aria2"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/parser"
|
"github.com/krau/SaveAny-Bot/pkg/parser"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/telegraph"
|
"github.com/krau/SaveAny-Bot/pkg/telegraph"
|
||||||
"github.com/krau/SaveAny-Bot/storage"
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
"github.com/rs/xid"
|
"github.com/rs/xid"
|
||||||
@@ -38,7 +39,7 @@ func NewTaskFactory(ctx context.Context) *TaskFactory {
|
|||||||
// CreateTask 创建任务
|
// CreateTask 创建任务
|
||||||
func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, error) {
|
func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, error) {
|
||||||
// 验证存储
|
// 验证存储
|
||||||
stor, ok := storage.Storages[req.Storage]
|
stor, ok := storage.GetStorage(req.Storage)
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("storage not found: %s", req.Storage)
|
return nil, fmt.Errorf("storage not found: %s", req.Storage)
|
||||||
}
|
}
|
||||||
@@ -68,9 +69,14 @@ func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, e
|
|||||||
|
|
||||||
func (f *TaskFactory) registerAndEnqueueTask(task core.Executable, taskType tasktype.TaskType, storageName, path, webhook string) error {
|
func (f *TaskFactory) registerAndEnqueueTask(task core.Executable, taskType tasktype.TaskType, storageName, path, webhook string) error {
|
||||||
taskID := task.TaskID()
|
taskID := task.TaskID()
|
||||||
RegisterTask(taskID, string(taskType), storageName, path, task.Title(), webhook)
|
info := RegisterTask(taskID, string(taskType), storageName, path, task.Title(), webhook)
|
||||||
|
|
||||||
err := core.AddTask(f.ctx, NewExecutableWrapper(task))
|
// Inject the progress sink into the context so the task's Emit calls update
|
||||||
|
// the API store (and fire the webhook on terminal states) without the task
|
||||||
|
// knowing about the API.
|
||||||
|
taskCtx := taskevent.WithSink(f.ctx, info)
|
||||||
|
|
||||||
|
err := core.AddTask(taskCtx, task)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
DeleteTask(taskID)
|
DeleteTask(taskID)
|
||||||
return fmt.Errorf("failed to add task: %w", err)
|
return fmt.Errorf("failed to add task: %w", err)
|
||||||
@@ -321,12 +327,12 @@ func (f *TaskFactory) createTransferTask(taskID string, createdAt time.Time, req
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 验证源存储和目标存储
|
// 验证源存储和目标存储
|
||||||
sourceStor, ok := storage.Storages[params.SourceStorage]
|
sourceStor, ok := storage.GetStorage(params.SourceStorage)
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("source storage not found: %s", params.SourceStorage)
|
return nil, fmt.Errorf("source storage not found: %s", params.SourceStorage)
|
||||||
}
|
}
|
||||||
|
|
||||||
targetStor, ok := storage.Storages[params.TargetStorage]
|
targetStor, ok := storage.GetStorage(params.TargetStorage)
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, fmt.Errorf("target storage not found: %s", params.TargetStorage)
|
return nil, fmt.Errorf("target storage not found: %s", params.TargetStorage)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/krau/SaveAny-Bot/core"
|
"github.com/krau/SaveAny-Bot/core"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
||||||
@@ -117,7 +118,7 @@ func (h *Handlers) CancelTaskHandler(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 取消任务
|
// Cancel the task; the terminal status is set via the task event stream.
|
||||||
if err := core.CancelTask(r.Context(), taskID); err != nil {
|
if err := core.CancelTask(r.Context(), taskID); err != nil {
|
||||||
WriteError(w, http.StatusInternalServerError, "cancel_failed", "failed to cancel task: "+err.Error())
|
WriteError(w, http.StatusInternalServerError, "cancel_failed", "failed to cancel task: "+err.Error())
|
||||||
return
|
return
|
||||||
@@ -134,8 +135,9 @@ func (h *Handlers) ListStoragesHandler(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
storages := make([]StorageInfo, 0, len(storage.Storages))
|
all := storage.AllStorages()
|
||||||
for name, stor := range storage.Storages {
|
storages := make([]StorageInfo, 0, len(all))
|
||||||
|
for name, stor := range all {
|
||||||
storages = append(storages, StorageInfo{
|
storages = append(storages, StorageInfo{
|
||||||
Name: name,
|
Name: name,
|
||||||
Type: string(stor.Type()),
|
Type: string(stor.Type()),
|
||||||
@@ -184,27 +186,45 @@ func extractTaskIDFromPath(path string) string {
|
|||||||
return parts[3]
|
return parts[3]
|
||||||
}
|
}
|
||||||
|
|
||||||
// convertTaskProgressToResponse 将任务进度转换为响应格式
|
// convertTaskProgressToResponse renders a task's current state, computing
|
||||||
|
// percent and speed from the snapshot taken under the task's mutex.
|
||||||
func convertTaskProgressToResponse(task *TaskProgressInfo) TaskInfoResponse {
|
func convertTaskProgressToResponse(task *TaskProgressInfo) TaskInfoResponse {
|
||||||
|
status, total, downloaded, totalFiles, downloadedFiles, startedAt, errMsg, updatedAt := task.snapshot()
|
||||||
|
|
||||||
resp := TaskInfoResponse{
|
resp := TaskInfoResponse{
|
||||||
TaskID: task.TaskID,
|
TaskID: task.TaskID,
|
||||||
Type: tasktype.TaskType(task.Type),
|
Type: tasktype.TaskType(task.Type),
|
||||||
Status: task.Status,
|
Status: status,
|
||||||
Title: task.Title,
|
Title: task.Title,
|
||||||
Storage: task.Storage,
|
Storage: task.Storage,
|
||||||
Path: task.Path,
|
Path: task.Path,
|
||||||
Error: task.Error,
|
Error: errMsg,
|
||||||
CreatedAt: task.CreatedAt,
|
CreatedAt: task.CreatedAt,
|
||||||
UpdatedAt: task.UpdatedAt,
|
UpdatedAt: updatedAt,
|
||||||
}
|
}
|
||||||
|
|
||||||
// 计算进度
|
var percent float64
|
||||||
if task.TotalBytes > 0 {
|
var speedMBPS float64
|
||||||
percent := float64(task.DownloadedBytes) * 100 / float64(task.TotalBytes)
|
if total > 0 {
|
||||||
|
percent = float64(downloaded) * 100 / float64(total)
|
||||||
|
} else if totalFiles > 0 {
|
||||||
|
percent = float64(downloadedFiles) * 100 / float64(totalFiles)
|
||||||
|
}
|
||||||
|
if !startedAt.IsZero() {
|
||||||
|
elapsed := time.Since(startedAt).Seconds()
|
||||||
|
if elapsed > 0 {
|
||||||
|
speedMBPS = float64(downloaded) / elapsed / (1024 * 1024)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if total > 0 || totalFiles > 0 {
|
||||||
resp.Progress = &TaskProgress{
|
resp.Progress = &TaskProgress{
|
||||||
TotalBytes: task.TotalBytes,
|
TotalBytes: total,
|
||||||
DownloadedBytes: task.DownloadedBytes,
|
DownloadedBytes: downloaded,
|
||||||
|
TotalFiles: totalFiles,
|
||||||
|
DownloadedFiles: downloadedFiles,
|
||||||
Percent: percent,
|
Percent: percent,
|
||||||
|
SpeedMBPS: speedMBPS,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
// setupTestServer creates a test server with handlers
|
// setupTestServer creates a test server with handlers
|
||||||
@@ -403,32 +404,38 @@ func TestConcurrentProgressStore(t *testing.T) {
|
|||||||
|
|
||||||
// TestProgressTrackerConcurrentUpdates tests concurrent progress updates
|
// TestProgressTrackerConcurrentUpdates tests concurrent progress updates
|
||||||
func TestProgressTrackerConcurrentUpdates(t *testing.T) {
|
func TestProgressTrackerConcurrentUpdates(t *testing.T) {
|
||||||
tracker := NewProgressTracker("concurrent-progress", "directlinks", "local", "downloads", "Test", "")
|
info := RegisterTask("concurrent-progress", "directlinks", "local", "downloads", "Test", "")
|
||||||
tracker.OnStart(10000, 10)
|
info.Emit(taskevent.Event{TaskID: "concurrent-progress", Phase: taskevent.PhaseStart, TotalBytes: 10000})
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
var wg sync.WaitGroup
|
||||||
numGoroutines := 50
|
numGoroutines := 50
|
||||||
updatesPerGoroutine := 100
|
updatesPerGoroutine := 100
|
||||||
|
|
||||||
// Concurrent progress updates
|
// Concurrent progress updates via the Sink interface
|
||||||
for i := range numGoroutines {
|
for i := range numGoroutines {
|
||||||
wg.Add(1)
|
wg.Add(1)
|
||||||
go func(id int) {
|
go func(id int) {
|
||||||
defer wg.Done()
|
defer wg.Done()
|
||||||
for j := range updatesPerGoroutine {
|
for j := range updatesPerGoroutine {
|
||||||
tracker.OnProgress(int64(id*updatesPerGoroutine+j), j)
|
info.Emit(taskevent.Event{
|
||||||
|
TaskID: "concurrent-progress",
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
DownloadedBytes: int64(id*updatesPerGoroutine + j),
|
||||||
|
TotalBytes: 10000,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}(i)
|
}(i)
|
||||||
}
|
}
|
||||||
|
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
|
|
||||||
info := tracker.GetInfo()
|
status, _, downloaded, _, _, _, _, _ := info.snapshot()
|
||||||
if info.Status != TaskStatusRunning {
|
if status != TaskStatusRunning {
|
||||||
t.Errorf("expected status Running after concurrent updates, got %s", info.Status)
|
t.Errorf("expected status Running after concurrent updates, got %s", status)
|
||||||
|
}
|
||||||
|
if downloaded <= 0 {
|
||||||
|
t.Errorf("expected downloaded bytes > 0 after concurrent updates, got %d", downloaded)
|
||||||
}
|
}
|
||||||
// Note: Due to race conditions in the simple implementation,
|
|
||||||
// we can't reliably check exact values without proper synchronization
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestTaskFactoryValidation tests TaskFactory parameter validation
|
// TestTaskFactoryValidation tests TaskFactory parameter validation
|
||||||
@@ -526,8 +533,7 @@ func TestEdgeCases(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "Progress tracker with empty webhook",
|
name: "Progress tracker with empty webhook",
|
||||||
fn: func(t *testing.T) {
|
fn: func(t *testing.T) {
|
||||||
tracker := NewProgressTracker("test", "type", "storage", "path", "title", "")
|
info := RegisterTask("test-empty-webhook", "type", "storage", "path", "title", "")
|
||||||
info := tracker.GetInfo()
|
|
||||||
if info.Webhook != "" {
|
if info.Webhook != "" {
|
||||||
t.Error("expected empty webhook")
|
t.Error("expected empty webhook")
|
||||||
}
|
}
|
||||||
|
|||||||
176
api/progress.go
176
api/progress.go
@@ -2,12 +2,16 @@ package api
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TaskProgressInfo 存储任务的进度信息
|
// TaskProgressInfo stores the progress of an API-submitted task. All fields are
|
||||||
|
// guarded by mu. It implements taskevent.Sink so the task layer can update it
|
||||||
|
// without knowing about the API.
|
||||||
type TaskProgressInfo struct {
|
type TaskProgressInfo struct {
|
||||||
|
mu sync.Mutex
|
||||||
TaskID string
|
TaskID string
|
||||||
Type string
|
Type string
|
||||||
Status TaskStatus
|
Status TaskStatus
|
||||||
@@ -21,20 +25,25 @@ type TaskProgressInfo struct {
|
|||||||
Error string
|
Error string
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
|
StartedAt time.Time
|
||||||
Webhook string
|
Webhook string
|
||||||
|
webhookNotified bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// progressStore 存储所有 API 任务的进度信息
|
// progressStore holds all API tasks. Entries are removed a fixed duration after
|
||||||
|
// they reach a terminal state to bound memory usage.
|
||||||
type progressStore struct {
|
type progressStore struct {
|
||||||
mu sync.RWMutex
|
mu sync.RWMutex
|
||||||
tasks map[string]*TaskProgressInfo
|
tasks map[string]*TaskProgressInfo
|
||||||
|
retention time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
var store = &progressStore{
|
var store = &progressStore{
|
||||||
tasks: make(map[string]*TaskProgressInfo),
|
tasks: make(map[string]*TaskProgressInfo),
|
||||||
|
retention: 24 * time.Hour,
|
||||||
}
|
}
|
||||||
|
|
||||||
// RegisterTask 注册一个新的 API 任务
|
// RegisterTask registers a new API task and returns its progress info.
|
||||||
func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskProgressInfo {
|
func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskProgressInfo {
|
||||||
info := &TaskProgressInfo{
|
info := &TaskProgressInfo{
|
||||||
TaskID: taskID,
|
TaskID: taskID,
|
||||||
@@ -55,7 +64,7 @@ func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskP
|
|||||||
return info
|
return info
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetTask 获取任务进度信息
|
// GetTask returns the progress info for a task.
|
||||||
func GetTask(taskID string) (*TaskProgressInfo, bool) {
|
func GetTask(taskID string) (*TaskProgressInfo, bool) {
|
||||||
store.mu.RLock()
|
store.mu.RLock()
|
||||||
defer store.mu.RUnlock()
|
defer store.mu.RUnlock()
|
||||||
@@ -63,7 +72,7 @@ func GetTask(taskID string) (*TaskProgressInfo, bool) {
|
|||||||
return info, ok
|
return info, ok
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetAllTasks 获取所有任务
|
// GetAllTasks returns all tracked tasks.
|
||||||
func GetAllTasks() []*TaskProgressInfo {
|
func GetAllTasks() []*TaskProgressInfo {
|
||||||
store.mu.RLock()
|
store.mu.RLock()
|
||||||
defer store.mu.RUnlock()
|
defer store.mu.RUnlock()
|
||||||
@@ -75,76 +84,117 @@ func GetAllTasks() []*TaskProgressInfo {
|
|||||||
return tasks
|
return tasks
|
||||||
}
|
}
|
||||||
|
|
||||||
// DeleteTask 删除任务记录
|
// DeleteTask removes a task record.
|
||||||
func DeleteTask(taskID string) {
|
func DeleteTask(taskID string) {
|
||||||
store.mu.Lock()
|
store.mu.Lock()
|
||||||
defer store.mu.Unlock()
|
defer store.mu.Unlock()
|
||||||
delete(store.tasks, taskID)
|
delete(store.tasks, taskID)
|
||||||
}
|
}
|
||||||
|
|
||||||
// UpdateStatus 更新任务状态
|
// CleanupExpired removes tasks that reached a terminal state more than the
|
||||||
func (t *TaskProgressInfo) UpdateStatus(status TaskStatus) {
|
// store's retention duration ago. It is safe to call periodically.
|
||||||
t.Status = status
|
func CleanupExpired() {
|
||||||
t.UpdatedAt = time.Now()
|
now := time.Now()
|
||||||
|
store.mu.Lock()
|
||||||
|
defer store.mu.Unlock()
|
||||||
|
for id, info := range store.tasks {
|
||||||
|
info.mu.Lock()
|
||||||
|
terminal := info.Status == TaskStatusCompleted || info.Status == TaskStatusFailed || info.Status == TaskStatusCancelled
|
||||||
|
stale := terminal && now.Sub(info.UpdatedAt) > store.retention
|
||||||
|
info.mu.Unlock()
|
||||||
|
if stale {
|
||||||
|
delete(store.tasks, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetError 设置错误信息
|
// StartCleanupLoop runs CleanupExpired on a fixed interval until ctx is done.
|
||||||
|
// It should be started once during API server initialization.
|
||||||
|
func StartCleanupLoop(ctx interface{ Done() <-chan struct{} }) {
|
||||||
|
go func() {
|
||||||
|
ticker := time.NewTicker(10 * time.Minute)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
CleanupExpired()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// UpdateStatus sets the task status.
|
||||||
|
func (t *TaskProgressInfo) UpdateStatus(status TaskStatus) {
|
||||||
|
t.mu.Lock()
|
||||||
|
t.Status = status
|
||||||
|
t.UpdatedAt = time.Now()
|
||||||
|
if status == TaskStatusRunning && t.StartedAt.IsZero() {
|
||||||
|
t.StartedAt = t.UpdatedAt
|
||||||
|
}
|
||||||
|
t.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetError marks the task failed with an error message.
|
||||||
func (t *TaskProgressInfo) SetError(err string) {
|
func (t *TaskProgressInfo) SetError(err string) {
|
||||||
|
t.mu.Lock()
|
||||||
t.Error = err
|
t.Error = err
|
||||||
t.Status = TaskStatusFailed
|
t.Status = TaskStatusFailed
|
||||||
t.UpdatedAt = time.Now()
|
t.UpdatedAt = time.Now()
|
||||||
|
t.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
// ProgressTracker 用于 API 任务的进度追踪
|
// snapshot returns a point-in-time copy of the fields needed to render a
|
||||||
type ProgressTracker struct {
|
// response, so callers never touch the mutex directly.
|
||||||
info *TaskProgressInfo
|
func (t *TaskProgressInfo) snapshot() (status TaskStatus, total, downloaded int64, totalFiles, downloadedFiles int, startedAt time.Time, err string, updatedAt time.Time) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
return t.Status, t.TotalBytes, t.DownloadedBytes, t.TotalFiles, t.DownloadedFiles, t.StartedAt, t.Error, t.UpdatedAt
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewProgressTracker 创建新的进度追踪器
|
// Emit implements taskevent.Sink. It translates task lifecycle events into
|
||||||
func NewProgressTracker(taskID, taskType, storage, path, title, webhook string) *ProgressTracker {
|
// status/progress updates and fires the webhook on terminal transitions.
|
||||||
info := RegisterTask(taskID, taskType, storage, path, title, webhook)
|
func (t *TaskProgressInfo) Emit(e taskevent.Event) {
|
||||||
return &ProgressTracker{info: info}
|
t.mu.Lock()
|
||||||
}
|
switch e.Phase {
|
||||||
|
case taskevent.PhaseStart:
|
||||||
// OnStart 任务开始
|
t.Status = TaskStatusRunning
|
||||||
func (p *ProgressTracker) OnStart(totalBytes int64, totalFiles int) {
|
if t.StartedAt.IsZero() {
|
||||||
p.info.Status = TaskStatusRunning
|
t.StartedAt = time.Now()
|
||||||
p.info.TotalBytes = totalBytes
|
}
|
||||||
p.info.TotalFiles = totalFiles
|
if e.TotalBytes > 0 {
|
||||||
p.info.UpdatedAt = time.Now()
|
t.TotalBytes = e.TotalBytes
|
||||||
}
|
}
|
||||||
|
case taskevent.PhaseProgress:
|
||||||
// OnProgress 进度更新
|
t.Status = TaskStatusRunning
|
||||||
func (p *ProgressTracker) OnProgress(downloadedBytes int64, downloadedFiles int) {
|
if e.TotalBytes > 0 {
|
||||||
atomic.StoreInt64(&p.info.DownloadedBytes, downloadedBytes)
|
t.TotalBytes = e.TotalBytes
|
||||||
p.info.DownloadedFiles = downloadedFiles
|
}
|
||||||
p.info.UpdatedAt = time.Now()
|
t.DownloadedBytes = e.DownloadedBytes
|
||||||
}
|
if e.TotalFiles > 0 {
|
||||||
|
t.TotalFiles = e.TotalFiles
|
||||||
// OnDone 任务完成
|
}
|
||||||
func (p *ProgressTracker) OnDone(err error) {
|
if e.DownloadedFiles > 0 {
|
||||||
if err != nil {
|
t.DownloadedFiles = e.DownloadedFiles
|
||||||
p.info.Status = TaskStatusFailed
|
}
|
||||||
p.info.Error = err.Error()
|
case taskevent.PhaseDone:
|
||||||
} else {
|
if e.Err != nil {
|
||||||
p.info.Status = TaskStatusCompleted
|
t.Status = TaskStatusFailed
|
||||||
|
t.Error = e.Err.Error()
|
||||||
|
} else {
|
||||||
|
t.Status = TaskStatusCompleted
|
||||||
|
}
|
||||||
}
|
}
|
||||||
p.info.UpdatedAt = time.Now()
|
t.UpdatedAt = time.Now()
|
||||||
}
|
notify := t.Webhook != "" && !t.webhookNotified && (t.Status == TaskStatusCompleted || t.Status == TaskStatusFailed)
|
||||||
|
if notify {
|
||||||
|
t.webhookNotified = true
|
||||||
|
}
|
||||||
|
t.mu.Unlock()
|
||||||
|
|
||||||
// GetInfo 获取任务信息
|
if notify {
|
||||||
func (p *ProgressTracker) GetInfo() *TaskProgressInfo {
|
payload := CreateWebhookPayload(t.TaskID, t.Type, t.Status, t.Storage, t.Path, e.Err)
|
||||||
return p.info
|
SendWebhook(nil, payload)
|
||||||
}
|
}
|
||||||
|
|
||||||
// UpdateProgressBytes 更新下载字节数
|
|
||||||
func (p *ProgressTracker) UpdateProgressBytes(bytes int64) {
|
|
||||||
atomic.StoreInt64(&p.info.DownloadedBytes, bytes)
|
|
||||||
p.info.UpdatedAt = time.Now()
|
|
||||||
}
|
|
||||||
|
|
||||||
// UpdateProgressFiles 更新下载文件数
|
|
||||||
func (p *ProgressTracker) UpdateProgressFiles(files int) {
|
|
||||||
p.info.DownloadedFiles = files
|
|
||||||
p.info.UpdatedAt = time.Now()
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package api
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -57,22 +58,19 @@ func NewServer(ctx context.Context) *Server {
|
|||||||
// 404 处理
|
// 404 处理
|
||||||
mux.HandleFunc("/", NotFoundHandler)
|
mux.HandleFunc("/", NotFoundHandler)
|
||||||
|
|
||||||
// 应用中间件
|
// Apply middleware chain.
|
||||||
var handler http.Handler = mux
|
var handler http.Handler = mux
|
||||||
|
|
||||||
// 添加认证中间件
|
// Apply auth middleware when a token is configured.
|
||||||
token := cfg.Token
|
token := cfg.Token
|
||||||
if token == "" {
|
|
||||||
log.FromContext(ctx).Warn("API server is enabled but no token is set, this is insecure!")
|
|
||||||
}
|
|
||||||
if token != "" {
|
if token != "" {
|
||||||
handler = AuthMiddleware()(handler)
|
handler = AuthMiddleware()(handler)
|
||||||
}
|
}
|
||||||
|
|
||||||
// 添加日志中间件
|
// Add logging middleware.
|
||||||
handler = loggingMiddleware(handler)
|
handler = loggingMiddleware(handler)
|
||||||
|
|
||||||
// 添加恢复中间件
|
// Add recovery middleware.
|
||||||
handler = recoveryMiddleware(handler)
|
handler = recoveryMiddleware(handler)
|
||||||
|
|
||||||
return &Server{
|
return &Server{
|
||||||
@@ -93,9 +91,15 @@ func (s *Server) Start(ctx context.Context) error {
|
|||||||
|
|
||||||
logger.Infof("Starting API server on %s", s.httpServer.Addr)
|
logger.Infof("Starting API server on %s", s.httpServer.Addr)
|
||||||
|
|
||||||
|
// Bind synchronously so listen failures are returned to the caller.
|
||||||
|
ln, err := net.Listen("tcp", s.httpServer.Addr)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to listen on %s: %w", s.httpServer.Addr, err)
|
||||||
|
}
|
||||||
|
|
||||||
// 在 goroutine 中启动服务器
|
// 在 goroutine 中启动服务器
|
||||||
go func() {
|
go func() {
|
||||||
if err := s.httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
if err := s.httpServer.Serve(ln); err != nil && err != http.ErrServerClosed {
|
||||||
logger.Errorf("API server error: %v", err)
|
logger.Errorf("API server error: %v", err)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
@@ -151,7 +155,8 @@ func (rw *responseWriter) WriteHeader(code int) {
|
|||||||
rw.ResponseWriter.WriteHeader(code)
|
rw.ResponseWriter.WriteHeader(code)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Start 初始化并启动 API 服务器
|
// Start initializes and starts the API server. It refuses to start without a
|
||||||
|
// token, since an open download proxy is a security risk.
|
||||||
func Start(ctx context.Context) error {
|
func Start(ctx context.Context) error {
|
||||||
cfg := config.C().API
|
cfg := config.C().API
|
||||||
|
|
||||||
@@ -160,9 +165,13 @@ func Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if cfg.Token == "" {
|
if cfg.Token == "" {
|
||||||
log.FromContext(ctx).Warn("API server is enabled but no token is set, this is insecure!")
|
return fmt.Errorf("API server is enabled but no token is set; refusing to start insecurely")
|
||||||
}
|
}
|
||||||
|
|
||||||
server := NewServer(ctx)
|
server := NewServer(ctx)
|
||||||
return server.Start(ctx)
|
if err := server.Start(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
StartCleanupLoop(ctx)
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ func resolveChatID(_ context.Context, idOrUsername string) (int64, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ParseMessageLink 解析 Telegram 消息链接
|
// ParseMessageLink 解析 Telegram 消息链接
|
||||||
|
// 支持的域名: t.me, telegram.me
|
||||||
// 支持格式:
|
// 支持格式:
|
||||||
// - https://t.me/username/123
|
// - https://t.me/username/123
|
||||||
// - https://t.me/c/123456789/123
|
// - https://t.me/c/123456789/123
|
||||||
@@ -268,5 +269,15 @@ func ExtractFilesFromLinks(ctx context.Context, links []string) ([]tfile.TGFileM
|
|||||||
|
|
||||||
// isValidMessageLink 检查是否是有效的 Telegram 消息链接
|
// isValidMessageLink 检查是否是有效的 Telegram 消息链接
|
||||||
func isValidMessageLink(link string) bool {
|
func isValidMessageLink(link string) bool {
|
||||||
return strings.HasPrefix(link, "https://t.me/") || strings.HasPrefix(link, "http://t.me/")
|
for _, prefix := range []string{
|
||||||
|
"https://t.me/",
|
||||||
|
"http://t.me/",
|
||||||
|
"https://telegram.me/",
|
||||||
|
"http://telegram.me/",
|
||||||
|
} {
|
||||||
|
if strings.HasPrefix(link, prefix) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -40,6 +40,8 @@ type CreateTaskResponse struct {
|
|||||||
type TaskProgress struct {
|
type TaskProgress struct {
|
||||||
TotalBytes int64 `json:"total_bytes,omitempty"`
|
TotalBytes int64 `json:"total_bytes,omitempty"`
|
||||||
DownloadedBytes int64 `json:"downloaded_bytes,omitempty"`
|
DownloadedBytes int64 `json:"downloaded_bytes,omitempty"`
|
||||||
|
TotalFiles int `json:"total_files,omitempty"`
|
||||||
|
DownloadedFiles int `json:"downloaded_files,omitempty"`
|
||||||
Percent float64 `json:"percent,omitempty"`
|
Percent float64 `json:"percent,omitempty"`
|
||||||
SpeedMBPS float64 `json:"speed_mbps,omitempty"`
|
SpeedMBPS float64 `json:"speed_mbps,omitempty"`
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -30,9 +29,17 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
|
|||||||
|
|
||||||
webhookURL := info.Webhook
|
webhookURL := info.Webhook
|
||||||
|
|
||||||
// 异步发送 webhook
|
// Async send with retries.
|
||||||
go func() {
|
go func() {
|
||||||
logger := log.FromContext(ctx).With("task_id", payload.TaskID)
|
var logger *log.Logger
|
||||||
|
if ctx != nil {
|
||||||
|
logger = log.FromContext(ctx).With("task_id", payload.TaskID)
|
||||||
|
} else {
|
||||||
|
logger = log.Default().With("task_id", payload.TaskID)
|
||||||
|
}
|
||||||
|
if ctx == nil {
|
||||||
|
ctx = context.Background()
|
||||||
|
}
|
||||||
|
|
||||||
payloadBytes, err := json.Marshal(payload)
|
payloadBytes, err := json.Marshal(payload)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -40,10 +47,15 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 重试 3 次
|
// 重试 3 次, 指数退避 (100ms/400ms/1.6s)
|
||||||
for i := range 3 {
|
const maxAttempts = 3
|
||||||
req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, webhookURL, bytes.NewBuffer(payloadBytes))
|
const requestTimeout = 30 * time.Second
|
||||||
|
backoff := 100 * time.Millisecond
|
||||||
|
for i := range maxAttempts {
|
||||||
|
reqCtx, cancel := context.WithTimeout(ctx, requestTimeout)
|
||||||
|
req, err := http.NewRequestWithContext(reqCtx, http.MethodPost, webhookURL, bytes.NewBuffer(payloadBytes))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
cancel()
|
||||||
logger.Errorf("Failed to create webhook request: %v", err)
|
logger.Errorf("Failed to create webhook request: %v", err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -52,9 +64,13 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
|
|||||||
req.Header.Set("User-Agent", "SaveAny-Bot/1.0")
|
req.Header.Set("User-Agent", "SaveAny-Bot/1.0")
|
||||||
|
|
||||||
resp, err := webhookClient.Do(req)
|
resp, err := webhookClient.Do(req)
|
||||||
|
cancel()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logger.Warnf("Webhook request failed (attempt %d/3): %v", i+1, err)
|
logger.Warnf("Webhook request failed (attempt %d/%d): %v", i+1, maxAttempts, err)
|
||||||
time.Sleep(time.Second * time.Duration(i+1))
|
if i < maxAttempts-1 {
|
||||||
|
time.Sleep(backoff)
|
||||||
|
}
|
||||||
|
backoff *= 4
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
resp.Body.Close()
|
resp.Body.Close()
|
||||||
@@ -64,15 +80,18 @@ func SendWebhook(ctx context.Context, payload *WebhookPayload) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.Warnf("Webhook returned non-2xx status (attempt %d/3): %d", i+1, resp.StatusCode)
|
logger.Warnf("Webhook returned non-2xx status (attempt %d/%d): %d", i+1, maxAttempts, resp.StatusCode)
|
||||||
time.Sleep(time.Second * time.Duration(i+1))
|
if i < maxAttempts-1 {
|
||||||
|
time.Sleep(backoff)
|
||||||
|
}
|
||||||
|
backoff *= 4
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.Errorf("Failed to send webhook after 3 attempts")
|
logger.Errorf("Failed to send webhook after %d attempts", maxAttempts)
|
||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
// CreateWebhookPayload 创建 Webhook 负载
|
// CreateWebhookPayload creates a Webhook payload.
|
||||||
func CreateWebhookPayload(taskID string, taskType string, status TaskStatus, storage, path string, err error) *WebhookPayload {
|
func CreateWebhookPayload(taskID string, taskType string, status TaskStatus, storage, path string, err error) *WebhookPayload {
|
||||||
payload := &WebhookPayload{
|
payload := &WebhookPayload{
|
||||||
TaskID: taskID,
|
TaskID: taskID,
|
||||||
@@ -93,38 +112,3 @@ func CreateWebhookPayload(taskID string, taskType string, status TaskStatus, sto
|
|||||||
|
|
||||||
return payload
|
return payload
|
||||||
}
|
}
|
||||||
|
|
||||||
// WrapTaskWithWebhook 包装任务执行,添加 webhook 回调
|
|
||||||
func WrapTaskWithWebhook(ctx context.Context, taskID string, fn func() error) error {
|
|
||||||
info, ok := GetTask(taskID)
|
|
||||||
if !ok {
|
|
||||||
return fmt.Errorf("task not found: %s", taskID)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := fn()
|
|
||||||
|
|
||||||
// 确定任务状态
|
|
||||||
status := TaskStatusCompleted
|
|
||||||
if err != nil {
|
|
||||||
if err == context.Canceled {
|
|
||||||
status = TaskStatusCancelled
|
|
||||||
} else {
|
|
||||||
status = TaskStatusFailed
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 更新任务状态
|
|
||||||
if err != nil {
|
|
||||||
info.SetError(err.Error())
|
|
||||||
} else {
|
|
||||||
info.UpdateStatus(TaskStatusCompleted)
|
|
||||||
}
|
|
||||||
|
|
||||||
// 发送 webhook
|
|
||||||
if info.Webhook != "" {
|
|
||||||
payload := CreateWebhookPayload(taskID, info.Type, status, info.Storage, info.Path, err)
|
|
||||||
SendWebhook(ctx, payload)
|
|
||||||
}
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,58 +0,0 @@
|
|||||||
package api
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
|
|
||||||
"github.com/krau/SaveAny-Bot/core"
|
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
|
||||||
)
|
|
||||||
|
|
||||||
// ExecutableWrapper wraps core.Executable to track task status in the API store and send webhooks.
|
|
||||||
type ExecutableWrapper struct {
|
|
||||||
inner core.Executable
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewExecutableWrapper(inner core.Executable) *ExecutableWrapper {
|
|
||||||
return &ExecutableWrapper{inner: inner}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *ExecutableWrapper) Type() tasktype.TaskType { return w.inner.Type() }
|
|
||||||
func (w *ExecutableWrapper) Title() string { return w.inner.Title() }
|
|
||||||
func (w *ExecutableWrapper) TaskID() string { return w.inner.TaskID() }
|
|
||||||
|
|
||||||
func (w *ExecutableWrapper) Execute(ctx context.Context) error {
|
|
||||||
taskID := w.inner.TaskID()
|
|
||||||
|
|
||||||
if info, ok := GetTask(taskID); ok {
|
|
||||||
info.UpdateStatus(TaskStatusRunning)
|
|
||||||
}
|
|
||||||
|
|
||||||
err := w.inner.Execute(ctx)
|
|
||||||
|
|
||||||
info, ok := GetTask(taskID)
|
|
||||||
if !ok {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
var status TaskStatus
|
|
||||||
if err != nil {
|
|
||||||
if errors.Is(err, context.Canceled) {
|
|
||||||
status = TaskStatusCancelled
|
|
||||||
info.UpdateStatus(TaskStatusCancelled)
|
|
||||||
} else {
|
|
||||||
status = TaskStatusFailed
|
|
||||||
info.SetError(err.Error())
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
status = TaskStatusCompleted
|
|
||||||
info.UpdateStatus(TaskStatusCompleted)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.Webhook != "" {
|
|
||||||
payload := CreateWebhookPayload(taskID, info.Type, status, info.Storage, info.Path, err)
|
|
||||||
SendWebhook(ctx, payload)
|
|
||||||
}
|
|
||||||
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
@@ -70,9 +70,6 @@ func Init(ctx context.Context) <-chan struct{} {
|
|||||||
}{nil, err}
|
}{nil, err}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
client.API().BotsSetBotCommands(ctx, &tg.BotsSetBotCommandsRequest{
|
|
||||||
Scope: &tg.BotCommandScopeDefault{},
|
|
||||||
})
|
|
||||||
commands := make([]tg.BotCommand, 0, len(handlers.CommandHandlers))
|
commands := make([]tg.BotCommand, 0, len(handlers.CommandHandlers))
|
||||||
for _, info := range handlers.CommandHandlers {
|
for _, info := range handlers.CommandHandlers {
|
||||||
commands = append(commands, tg.BotCommand{Command: info.Cmd, Description: i18n.T(info.Desc)})
|
commands = append(commands, tg.BotCommand{Command: info.Cmd, Description: i18n.T(info.Desc)})
|
||||||
|
|||||||
@@ -23,7 +23,11 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
|
func handleAddCallback(ctx *ext.Context, update *ext.Update) error {
|
||||||
dataid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
|
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
|
||||||
|
if len(dataParts) < 2 {
|
||||||
|
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
|
||||||
|
}
|
||||||
|
dataid := dataParts[1]
|
||||||
data, err := shortcut.GetCallbackDataWithAnswer[tcbdata.Add](ctx, update, dataid)
|
data, err := shortcut.GetCallbackDataWithAnswer[tcbdata.Add](ctx, update, dataid)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package handlers
|
package handlers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/celestix/gotgproto/dispatcher"
|
"github.com/celestix/gotgproto/dispatcher"
|
||||||
@@ -14,7 +15,11 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func handleCancelCallback(ctx *ext.Context, update *ext.Update) error {
|
func handleCancelCallback(ctx *ext.Context, update *ext.Update) error {
|
||||||
taskid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
|
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
|
||||||
|
if len(dataParts) < 2 {
|
||||||
|
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
|
||||||
|
}
|
||||||
|
taskid := dataParts[1]
|
||||||
if err := core.CancelTask(ctx, taskid); err != nil {
|
if err := core.CancelTask(ctx, taskid); err != nil {
|
||||||
log.FromContext(ctx).Errorf("Failed to cancel task %s: %v", taskid, err)
|
log.FromContext(ctx).Errorf("Failed to cancel task %s: %v", taskid, err)
|
||||||
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(update.CallbackQuery.GetQueryID(), i18n.T(i18nk.BotMsgCancelErrorCancelFailed, map[string]any{
|
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(update.CallbackQuery.GetQueryID(), i18n.T(i18nk.BotMsgCancelErrorCancelFailed, map[string]any{
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
|
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
"github.com/krau/SaveAny-Bot/database"
|
"github.com/krau/SaveAny-Bot/database"
|
||||||
"github.com/krau/SaveAny-Bot/storage"
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
)
|
)
|
||||||
@@ -42,7 +43,9 @@ func handleDirCmd(ctx *ext.Context, update *ext.Update) error {
|
|||||||
return dispatcher.EndGroups
|
return dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
if _, err := storage.GetStorageByUserIDAndName(ctx, user.ChatID, args[2]); err != nil {
|
if _, err := storage.GetStorageByUserIDAndName(ctx, user.ChatID, args[2]); err != nil {
|
||||||
ctx.Reply(update, ext.ReplyTextString(err.Error()), nil)
|
logger.Errorf("Failed to get storage %q: %s", args[2], err)
|
||||||
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorGetStorageFailed,
|
||||||
|
tgutil.EscapeHTMLTemplateData(map[string]any{"Error": err.Error()}))), nil)
|
||||||
return dispatcher.EndGroups
|
return dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -13,16 +13,24 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
|
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/shortcut"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/database"
|
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
|
"github.com/krau/SaveAny-Bot/database"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
|
"github.com/krau/SaveAny-Bot/pkg/tcbdata"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/tfile"
|
"github.com/krau/SaveAny-Bot/pkg/tfile"
|
||||||
"github.com/krau/SaveAny-Bot/storage"
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// mediaGroupKey uniquely identifies a media group by chat, sender, and group
|
||||||
|
// ID so files from different users in the same chat can never be mixed.
|
||||||
|
type mediaGroupKey struct {
|
||||||
|
chatID int64
|
||||||
|
userID int64
|
||||||
|
groupID int64
|
||||||
|
}
|
||||||
|
|
||||||
type MediaGroupHandler struct {
|
type MediaGroupHandler struct {
|
||||||
groups map[int64][]tfile.TGFileMessage
|
groups map[mediaGroupKey][]tfile.TGFileMessage
|
||||||
timers map[int64]*time.Timer
|
timers map[mediaGroupKey]*time.Timer
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
timeout time.Duration
|
timeout time.Duration
|
||||||
setupOnce sync.Once
|
setupOnce sync.Once
|
||||||
@@ -39,8 +47,8 @@ func (m *MediaGroupHandler) SetupTimeout(timeoutSec int) {
|
|||||||
|
|
||||||
var (
|
var (
|
||||||
mediaGroupHandler = &MediaGroupHandler{
|
mediaGroupHandler = &MediaGroupHandler{
|
||||||
groups: make(map[int64][]tfile.TGFileMessage),
|
groups: make(map[mediaGroupKey][]tfile.TGFileMessage),
|
||||||
timers: make(map[int64]*time.Timer),
|
timers: make(map[mediaGroupKey]*time.Timer),
|
||||||
mu: sync.Mutex{},
|
mu: sync.Mutex{},
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
@@ -66,32 +74,37 @@ func handleGroupMediaMessage(ctx *ext.Context, update *ext.Update, message *tg.M
|
|||||||
}
|
}
|
||||||
mediaGroupHandler.mu.Lock()
|
mediaGroupHandler.mu.Lock()
|
||||||
defer mediaGroupHandler.mu.Unlock()
|
defer mediaGroupHandler.mu.Unlock()
|
||||||
if mediaGroupHandler.groups[groupID] == nil {
|
key := mediaGroupKey{
|
||||||
mediaGroupHandler.groups[groupID] = make([]tfile.TGFileMessage, 0)
|
chatID: update.EffectiveChat().GetID(),
|
||||||
|
userID: userId,
|
||||||
|
groupID: groupID,
|
||||||
}
|
}
|
||||||
mediaGroupHandler.groups[groupID] = append(mediaGroupHandler.groups[groupID], file)
|
if mediaGroupHandler.groups[key] == nil {
|
||||||
|
mediaGroupHandler.groups[key] = make([]tfile.TGFileMessage, 0)
|
||||||
|
}
|
||||||
|
mediaGroupHandler.groups[key] = append(mediaGroupHandler.groups[key], file)
|
||||||
|
|
||||||
if timer, exists := mediaGroupHandler.timers[groupID]; exists {
|
if timer, exists := mediaGroupHandler.timers[key]; exists {
|
||||||
timer.Stop()
|
timer.Stop()
|
||||||
}
|
}
|
||||||
mediaGroupHandler.timers[groupID] = time.AfterFunc(mediaGroupHandler.timeout, func() {
|
mediaGroupHandler.timers[key] = time.AfterFunc(mediaGroupHandler.timeout, func() {
|
||||||
processMediaGroup(ctx, update, groupID)
|
processMediaGroup(ctx, update, key)
|
||||||
})
|
})
|
||||||
return dispatcher.EndGroups
|
return dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
|
|
||||||
func processMediaGroup(ctx *ext.Context, update *ext.Update, groupID int64) {
|
func processMediaGroup(ctx *ext.Context, update *ext.Update, key mediaGroupKey) {
|
||||||
logger := log.FromContext(ctx)
|
logger := log.FromContext(ctx)
|
||||||
mediaGroupHandler.mu.Lock()
|
mediaGroupHandler.mu.Lock()
|
||||||
items := mediaGroupHandler.groups[groupID]
|
items := mediaGroupHandler.groups[key]
|
||||||
delete(mediaGroupHandler.groups, groupID)
|
delete(mediaGroupHandler.groups, key)
|
||||||
delete(mediaGroupHandler.timers, groupID)
|
delete(mediaGroupHandler.timers, key)
|
||||||
mediaGroupHandler.mu.Unlock()
|
mediaGroupHandler.mu.Unlock()
|
||||||
if len(items) == 0 {
|
if len(items) == 0 {
|
||||||
logger.Warn("No media items to process for group", "groupID", groupID)
|
logger.Warn("No media items to process for group", "groupID", key.groupID)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
logger.Debugf("Processing media group %d with %d items", groupID, len(items))
|
logger.Debugf("Processing media group %d with %d items", key.groupID, len(items))
|
||||||
|
|
||||||
userId := update.GetUserChat().GetID()
|
userId := update.GetUserChat().GetID()
|
||||||
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgMediaGroupInfoSavingFiles, nil)), nil)
|
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgMediaGroupInfoSavingFiles, nil)), nil)
|
||||||
|
|||||||
@@ -22,6 +22,17 @@ func checkPermission(ctx *ext.Context, update *ext.Update) error {
|
|||||||
return dispatcher.ContinueGroups
|
return dispatcher.ContinueGroups
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// withPermission wraps a callback handler with the same whitelist check used
|
||||||
|
// for message handlers (checkPermission).
|
||||||
|
func withPermission(handler func(*ext.Context, *ext.Update) error) func(*ext.Context, *ext.Update) error {
|
||||||
|
return func(ctx *ext.Context, update *ext.Update) error {
|
||||||
|
if err := checkPermission(ctx, update); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return handler(ctx, update)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func handleSilentMode(next func(*ext.Context, *ext.Update) error, handler func(*ext.Context, *ext.Update) error) func(*ext.Context, *ext.Update) error {
|
func handleSilentMode(next func(*ext.Context, *ext.Update) error, handler func(*ext.Context, *ext.Update) error) func(*ext.Context, *ext.Update) error {
|
||||||
return func(ctx *ext.Context, update *ext.Update) error {
|
return func(ctx *ext.Context, update *ext.Update) error {
|
||||||
userID := update.GetUserChat().GetID()
|
userID := update.GetUserChat().GetID()
|
||||||
|
|||||||
@@ -56,11 +56,11 @@ func Register(disp dispatcher.Dispatcher) {
|
|||||||
for _, info := range CommandHandlers {
|
for _, info := range CommandHandlers {
|
||||||
disp.AddHandler(handlers.NewCommand(info.Cmd, info.handler))
|
disp.AddHandler(handlers.NewCommand(info.Cmd, info.handler))
|
||||||
}
|
}
|
||||||
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("update"), handleUpdateCallback))
|
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix("update"), withPermission(handleUpdateCallback)))
|
||||||
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), handleAddCallback))
|
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeAdd), withPermission(handleAddCallback)))
|
||||||
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), handleSetDefaultCallback))
|
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeSetDefault), withPermission(handleSetDefaultCallback)))
|
||||||
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeCancel), handleCancelCallback))
|
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeCancel), withPermission(handleCancelCallback)))
|
||||||
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeConfig), handleConfigCallback))
|
disp.AddHandler(handlers.NewCallbackQuery(filters.CallbackQuery.Prefix(tcbdata.TypeConfig), withPermission(handleConfigCallback)))
|
||||||
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TgMessageLinkRegexString)), handleSilentMode(handleMessageLink, handleSilentSaveLink)))
|
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TgMessageLinkRegexString)), handleSilentMode(handleMessageLink, handleSilentSaveLink)))
|
||||||
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TelegraphUrlRegexString)), handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
|
disp.AddHandler(handlers.NewMessage(sabotfilters.RegexUrl(regexp.MustCompile(re.TelegraphUrlRegexString)), handleSilentMode(handleTelegraphUrlMessage, handleSilentSaveTelegraph)))
|
||||||
disp.AddHandler(handlers.NewMessage(filters.Message.Media, handleSilentMode(handleMediaMessage, handleSilentSaveMedia)))
|
disp.AddHandler(handlers.NewMessage(filters.Message.Media, handleSilentMode(handleMediaMessage, handleSilentSaveMedia)))
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/strutil"
|
"github.com/krau/SaveAny-Bot/common/utils/strutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/database"
|
"github.com/krau/SaveAny-Bot/database"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/rule"
|
"github.com/krau/SaveAny-Bot/pkg/rule"
|
||||||
)
|
)
|
||||||
@@ -84,6 +85,46 @@ func handleRuleCmd(ctx *ext.Context, update *ext.Update) error {
|
|||||||
return dispatcher.EndGroups
|
return dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleInfoCreateRuleSuccess, nil)), nil)
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleInfoCreateRuleSuccess, nil)), nil)
|
||||||
|
case "preset":
|
||||||
|
// /rule preset <storage> [base_path]
|
||||||
|
if len(args) < 3 {
|
||||||
|
ctx.Reply(update, ext.ReplyTextStyledTextArray(msgelem.BuildRuleHelpStyling(user.ApplyRule, user.Rules)), nil)
|
||||||
|
return dispatcher.EndGroups
|
||||||
|
}
|
||||||
|
storageName := args[2]
|
||||||
|
if !config.C().HasStorage(user.ChatID, storageName) {
|
||||||
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleErrorStorageNotFound, map[string]any{
|
||||||
|
"Storage": storageName,
|
||||||
|
})), nil)
|
||||||
|
return dispatcher.EndGroups
|
||||||
|
}
|
||||||
|
basePath := ""
|
||||||
|
if len(args) >= 4 {
|
||||||
|
basePath = args[3]
|
||||||
|
}
|
||||||
|
presets := rule.PresetCategories(basePath)
|
||||||
|
imported := 0
|
||||||
|
for _, p := range presets {
|
||||||
|
rd := &database.Rule{
|
||||||
|
Type: rule.FileNameRegex.String(),
|
||||||
|
Data: p.Regex,
|
||||||
|
StorageName: storageName,
|
||||||
|
DirPath: p.Dir,
|
||||||
|
UserID: user.ID,
|
||||||
|
}
|
||||||
|
if err := database.CreateRule(ctx, rd); err != nil {
|
||||||
|
logger.Errorf("failed to create preset rule %s: %s", p.Name, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
imported++
|
||||||
|
}
|
||||||
|
if imported == 0 {
|
||||||
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleErrorCreateRuleFailed, nil)), nil)
|
||||||
|
return dispatcher.EndGroups
|
||||||
|
}
|
||||||
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgRuleInfoPresetImported, map[string]any{
|
||||||
|
"Count": imported,
|
||||||
|
})), nil)
|
||||||
case "del":
|
case "del":
|
||||||
// /rule del <id>
|
// /rule del <id>
|
||||||
if len(args) < 3 {
|
if len(args) < 3 {
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package handlers
|
package handlers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/celestix/gotgproto/dispatcher"
|
"github.com/celestix/gotgproto/dispatcher"
|
||||||
@@ -43,7 +44,11 @@ func handleSilentCmd(ctx *ext.Context, update *ext.Update) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func handleSetDefaultCallback(ctx *ext.Context, update *ext.Update) error {
|
func handleSetDefaultCallback(ctx *ext.Context, update *ext.Update) error {
|
||||||
dataid := strings.Split(string(update.CallbackQuery.Data), " ")[1]
|
dataParts := strings.Split(string(update.CallbackQuery.Data), " ")
|
||||||
|
if len(dataParts) < 2 {
|
||||||
|
return fmt.Errorf("invalid callback data: %q", update.CallbackQuery.Data)
|
||||||
|
}
|
||||||
|
dataid := dataParts[1]
|
||||||
data, ok := cache.Get[tcbdata.SetDefaultStorage](dataid)
|
data, ok := cache.Get[tcbdata.SetDefaultStorage](dataid)
|
||||||
|
|
||||||
failedAnswer := func(message string) error {
|
failedAnswer := func(message string) error {
|
||||||
|
|||||||
@@ -89,7 +89,11 @@ func showQueuedTasks(ctx *ext.Context, update *ext.Update) {
|
|||||||
styling.Bold(i18n.T(i18nk.BotMsgTasksQueuedTitle)),
|
styling.Bold(i18n.T(i18nk.BotMsgTasksQueuedTitle)),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgTasksTotalPrefix, map[string]any{"Count": len(tasks)})),
|
styling.Plain(i18n.T(i18nk.BotMsgTasksTotalPrefix, map[string]any{"Count": len(tasks)})),
|
||||||
)
|
)
|
||||||
for _, t := range tasks {
|
const maxShown = 10
|
||||||
|
for i, t := range tasks {
|
||||||
|
if i >= maxShown {
|
||||||
|
break
|
||||||
|
}
|
||||||
created := t.Created.In(time.Local).Format("2006-01-02 15:04:05")
|
created := t.Created.In(time.Local).Format("2006-01-02 15:04:05")
|
||||||
status := i18n.T(i18nk.BotMsgTasksStatusQueued)
|
status := i18n.T(i18nk.BotMsgTasksStatusQueued)
|
||||||
if t.Cancelled {
|
if t.Cancelled {
|
||||||
@@ -105,10 +109,9 @@ func showQueuedTasks(ctx *ext.Context, update *ext.Update) {
|
|||||||
styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksFieldStatus)),
|
styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksFieldStatus)),
|
||||||
styling.Code(status),
|
styling.Code(status),
|
||||||
)
|
)
|
||||||
if len(tasks) > 10 {
|
}
|
||||||
opts = append(opts, styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksTruncatedNote, map[string]any{"Count": len(tasks)})))
|
if len(tasks) > maxShown {
|
||||||
break
|
opts = append(opts, styling.Plain("\n"+i18n.T(i18nk.BotMsgTasksTruncatedNote, map[string]any{"Count": len(tasks)})))
|
||||||
}
|
|
||||||
}
|
}
|
||||||
ctx.Reply(update, ext.ReplyTextStyledTextArray(opts), nil)
|
ctx.Reply(update, ext.ReplyTextStyledTextArray(opts), nil)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ import (
|
|||||||
"github.com/celestix/gotgproto/ext"
|
"github.com/celestix/gotgproto/ext"
|
||||||
"github.com/gotd/td/telegram/message/html"
|
"github.com/gotd/td/telegram/message/html"
|
||||||
"github.com/gotd/td/tg"
|
"github.com/gotd/td/tg"
|
||||||
|
"github.com/krau/SaveAny-Bot/client/bot/handlers/utils/msgelem"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
@@ -100,7 +101,10 @@ func handleUpdateCmd(ctx *ext.Context, u *ext.Update) error {
|
|||||||
func handleUpdateCallback(ctx *ext.Context, u *ext.Update) error {
|
func handleUpdateCallback(ctx *ext.Context, u *ext.Update) error {
|
||||||
currentV, err := semver.Parse(config.Version)
|
currentV, err := semver.Parse(config.Version)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
ctx.AnswerCallback(msgelem.AlertCallbackAnswer(u.CallbackQuery.GetQueryID(), i18n.T(i18nk.BotMsgUpdateErrorVersionVarInvalid, map[string]any{
|
||||||
|
"Error": err.Error(),
|
||||||
|
})))
|
||||||
|
return dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
|
ctx.EditMessage(u.GetUserChat().GetID(), &tg.MessagesEditMessageRequest{
|
||||||
ID: u.CallbackQuery.GetMsgID(),
|
ID: u.CallbackQuery.GetMsgID(),
|
||||||
|
|||||||
@@ -24,6 +24,8 @@ func BuildRuleHelpStyling(enabled bool, rules []database.Rule) []styling.StyledT
|
|||||||
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpSwitchSuffix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpSwitchSuffix, nil)),
|
||||||
styling.Code("add"),
|
styling.Code("add"),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpAddSuffix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpAddSuffix, nil)),
|
||||||
|
styling.Code("preset"),
|
||||||
|
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpPresetSuffix, nil)),
|
||||||
styling.Code("del"),
|
styling.Code("del"),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpDelSuffix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpDelSuffix, nil)),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpExistingRulesPrefix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgRuleHelpExistingRulesPrefix, nil)),
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ package re
|
|||||||
import "regexp"
|
import "regexp"
|
||||||
|
|
||||||
var (
|
var (
|
||||||
TgMessageLinkRegexString = `https?://t\.me/(?:c/\d+|[A-Za-z0-9_]+)/\d+(?:/\d+)?(?:\?[^\s#]*[A-Za-z0-9_])?\b`
|
TgMessageLinkRegexString = `https?://(?:t|telegram)\.me/(?:c/\d+|[A-Za-z0-9_]+)/\d+(?:/\d+)?(?:\?[^\s#]*[A-Za-z0-9_])?\b`
|
||||||
TgMessageLinkRegexp = regexp.MustCompile(TgMessageLinkRegexString)
|
TgMessageLinkRegexp = regexp.MustCompile(TgMessageLinkRegexString)
|
||||||
TelegraphUrlRegexString = `https://telegra.ph/.*`
|
TelegraphUrlRegexString = `https://telegra\.ph/[^\s]+`
|
||||||
TelegraphUrlRegexp = regexp.MustCompile(TelegraphUrlRegexString)
|
TelegraphUrlRegexp = regexp.MustCompile(TelegraphUrlRegexString)
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package shortcut
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"net/url"
|
"net/url"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -179,21 +180,19 @@ type TelegraphResult struct {
|
|||||||
// return replied message, image urls, telegraph path(unescaped), error
|
// return replied message, image urls, telegraph path(unescaped), error
|
||||||
func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*types.Message, *TelegraphResult, error) {
|
func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*types.Message, *TelegraphResult, error) {
|
||||||
logger := log.FromContext(ctx)
|
logger := log.FromContext(ctx)
|
||||||
tphurl := re.TelegraphUrlRegexp.FindString(tgutil.ExtractMessageEntityUrlsText(update.EffectiveMessage.Message))
|
tphurl := findTelegraphURL(update.EffectiveMessage.Message)
|
||||||
if tphurl == "" {
|
if tphurl == "" {
|
||||||
logger.Warnf("No telegraph url found but called handleTelegraph")
|
logger.Warnf("No telegraph url found but called handleTelegraph")
|
||||||
return nil, nil, dispatcher.ContinueGroups
|
return nil, nil, dispatcher.ContinueGroups
|
||||||
}
|
}
|
||||||
pagepath := strings.Split(tphurl, "/")[len(strings.Split(tphurl, "/"))-1]
|
pagepath, err := parseTelegraphPagePath(tphurl)
|
||||||
tphdir, err := url.PathUnescape(pagepath)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logger.Errorf("Failed to unescape telegraph path: %s", err)
|
logger.Errorf("Failed to parse telegraph path: %s", err)
|
||||||
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorParseTelegraphPathFailed, map[string]any{
|
ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonErrorParseTelegraphPathFailed, map[string]any{
|
||||||
"Error": err.Error(),
|
"Error": err.Error(),
|
||||||
})), nil)
|
})), nil)
|
||||||
return nil, nil, dispatcher.EndGroups
|
return nil, nil, dispatcher.EndGroups
|
||||||
}
|
}
|
||||||
tphdir = strings.TrimSpace(tphdir)
|
|
||||||
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonInfoFetchingTelegraphPage, nil)), nil)
|
msg, err := ctx.Reply(update, ext.ReplyTextString(i18n.T(i18nk.BotMsgCommonInfoFetchingTelegraphPage, nil)), nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logger.Errorf("Failed to reply to update: %s", err)
|
logger.Errorf("Failed to reply to update: %s", err)
|
||||||
@@ -244,7 +243,57 @@ func GetTphPicsFromMessageWithReply(ctx *ext.Context, update *ext.Update) (*type
|
|||||||
}
|
}
|
||||||
return msg, &TelegraphResult{
|
return msg, &TelegraphResult{
|
||||||
Pics: imgs,
|
Pics: imgs,
|
||||||
TphDir: tphdir,
|
TphDir: pagepath,
|
||||||
Page: page,
|
Page: page,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func findTelegraphURL(msg *tg.Message) string {
|
||||||
|
if msg == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var firstMatch string
|
||||||
|
findValid := func(text string) string {
|
||||||
|
for _, tphurl := range re.TelegraphUrlRegexp.FindAllString(text, -1) {
|
||||||
|
if firstMatch == "" {
|
||||||
|
firstMatch = tphurl
|
||||||
|
}
|
||||||
|
if _, err := parseTelegraphPagePath(tphurl); err == nil {
|
||||||
|
return tphurl
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
for _, entityURL := range tgutil.ExtractMessageEntityUrls(msg) {
|
||||||
|
if tphurl := findValid(entityURL); tphurl != "" {
|
||||||
|
return tphurl
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if tphurl := findValid(msg.GetMessage()); tphurl != "" {
|
||||||
|
return tphurl
|
||||||
|
}
|
||||||
|
return firstMatch
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseTelegraphPagePath(pageURL string) (string, error) {
|
||||||
|
u, err := url.Parse(pageURL)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("invalid telegraph URL: %w", err)
|
||||||
|
}
|
||||||
|
if u.Scheme != "https" || !strings.EqualFold(u.Hostname(), "telegra.ph") {
|
||||||
|
return "", fmt.Errorf("invalid telegraph URL host: %s", u.Host)
|
||||||
|
}
|
||||||
|
pagepath := strings.Trim(u.EscapedPath(), "/")
|
||||||
|
if pagepath == "" || strings.Contains(pagepath, "/") {
|
||||||
|
return "", fmt.Errorf("invalid telegraph URL path: %s", u.Path)
|
||||||
|
}
|
||||||
|
pagepath, err = url.PathUnescape(pagepath)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to unescape telegraph path: %w", err)
|
||||||
|
}
|
||||||
|
pagepath = strings.TrimSpace(pagepath)
|
||||||
|
if pagepath == "" || strings.Contains(pagepath, "/") {
|
||||||
|
return "", fmt.Errorf("invalid telegraph URL path: %s", u.Path)
|
||||||
|
}
|
||||||
|
return pagepath, nil
|
||||||
|
}
|
||||||
|
|||||||
163
client/bot/handlers/utils/shortcut/message_telegraph_test.go
Normal file
163
client/bot/handlers/utils/shortcut/message_telegraph_test.go
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
package shortcut
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFindTelegraphURL(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
msg *tg.Message
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "single URL entity",
|
||||||
|
msg: &tg.Message{
|
||||||
|
Message: "https://telegra.ph/Example-01-02",
|
||||||
|
Entities: []tg.MessageEntityClass{
|
||||||
|
&tg.MessageEntityURL{Offset: 0, Length: 32},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: "https://telegra.ph/Example-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "Telegraph URL before another URL",
|
||||||
|
msg: &tg.Message{
|
||||||
|
Message: "https://telegra.ph/Example-01-02 https://example.com/",
|
||||||
|
Entities: []tg.MessageEntityClass{
|
||||||
|
&tg.MessageEntityURL{Offset: 0, Length: 32},
|
||||||
|
&tg.MessageEntityURL{Offset: 33, Length: 20},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: "https://telegra.ph/Example-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "hidden Telegraph URL",
|
||||||
|
msg: &tg.Message{
|
||||||
|
Message: "article",
|
||||||
|
Entities: []tg.MessageEntityClass{
|
||||||
|
&tg.MessageEntityTextURL{
|
||||||
|
Offset: 0,
|
||||||
|
Length: 7,
|
||||||
|
URL: "https://telegra.ph/Hidden-01-02",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: "https://telegra.ph/Hidden-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "URL entity after non-BMP character",
|
||||||
|
msg: &tg.Message{
|
||||||
|
Message: "😀 https://telegra.ph/Emoji-01-02",
|
||||||
|
Entities: []tg.MessageEntityClass{
|
||||||
|
&tg.MessageEntityURL{Offset: 3, Length: 30},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: "https://telegra.ph/Emoji-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "valid Telegraph URL after invalid candidate",
|
||||||
|
msg: &tg.Message{
|
||||||
|
Message: "https://telegra.ph/nested/Bad https://telegra.ph/Valid-01-02",
|
||||||
|
Entities: []tg.MessageEntityClass{
|
||||||
|
&tg.MessageEntityURL{Offset: 0, Length: 29},
|
||||||
|
&tg.MessageEntityURL{Offset: 30, Length: 30},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
want: "https://telegra.ph/Valid-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "plain message fallback",
|
||||||
|
msg: &tg.Message{Message: "read https://telegra.ph/Plain-01-02 now"},
|
||||||
|
want: "https://telegra.ph/Plain-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nil message",
|
||||||
|
msg: nil,
|
||||||
|
want: "",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
if got := findTelegraphURL(tt.msg); got != tt.want {
|
||||||
|
t.Fatalf("findTelegraphURL() = %q, want %q", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseTelegraphPagePath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
pageURL string
|
||||||
|
want string
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "plain path",
|
||||||
|
pageURL: "https://telegra.ph/Example-01-02",
|
||||||
|
want: "Example-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "escaped path with query and fragment",
|
||||||
|
pageURL: "https://telegra.ph/%E6%B5%8B%E8%AF%95-01-02?source=telegram#top",
|
||||||
|
want: "测试-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "trailing slash",
|
||||||
|
pageURL: "https://telegra.ph/Example-01-02/",
|
||||||
|
want: "Example-01-02",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "root URL",
|
||||||
|
pageURL: "https://telegra.ph/",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "wrong host",
|
||||||
|
pageURL: "https://example.com/Example-01-02",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "wrong scheme",
|
||||||
|
pageURL: "http://telegra.ph/Example-01-02",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid percent escape",
|
||||||
|
pageURL: "https://telegra.ph/Invalid-%zz",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "nested path",
|
||||||
|
pageURL: "https://telegra.ph/nested/Example-01-02",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "encoded slash",
|
||||||
|
pageURL: "https://telegra.ph/nested%2FExample-01-02",
|
||||||
|
wantErr: true,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := parseTelegraphPagePath(tt.pageURL)
|
||||||
|
if tt.wantErr {
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("parseTelegraphPagePath(%q) returned no error", tt.pageURL)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parseTelegraphPagePath(%q) failed: %v", tt.pageURL, err)
|
||||||
|
}
|
||||||
|
if got != tt.want {
|
||||||
|
t.Fatalf("parseTelegraphPagePath(%q) = %q, want %q", tt.pageURL, got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ package user
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/celestix/gotgproto"
|
"github.com/celestix/gotgproto"
|
||||||
@@ -20,17 +21,18 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var uc *gotgproto.Client
|
var uc *gotgproto.Client
|
||||||
var ectx *ext.Context
|
|
||||||
|
// getEctx lazily creates the user-client ext.Context exactly once. Guarded by
|
||||||
|
// sync.OnceValue so concurrent GetCtx calls cannot race on ectx creation.
|
||||||
|
var getEctx = sync.OnceValue(func() *ext.Context {
|
||||||
|
return uc.CreateContext()
|
||||||
|
})
|
||||||
|
|
||||||
func GetCtx() *ext.Context {
|
func GetCtx() *ext.Context {
|
||||||
if ectx != nil {
|
|
||||||
return ectx
|
|
||||||
}
|
|
||||||
if uc == nil {
|
if uc == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
ectx = uc.CreateContext()
|
return getEctx()
|
||||||
return ectx
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func Login(ctx context.Context) (*gotgproto.Client, error) {
|
func Login(ctx context.Context) (*gotgproto.Client, error) {
|
||||||
|
|||||||
@@ -22,7 +22,13 @@ func main() {
|
|||||||
pkg := flag.String("pkg", "i18nk", "Package name for generated file")
|
pkg := flag.String("pkg", "i18nk", "Package name for generated file")
|
||||||
flag.Parse()
|
flag.Parse()
|
||||||
|
|
||||||
|
type localeFile struct {
|
||||||
|
path string
|
||||||
|
keys map[string]struct{}
|
||||||
|
}
|
||||||
|
|
||||||
keys := make(map[string]struct{})
|
keys := make(map[string]struct{})
|
||||||
|
var localeFiles []localeFile
|
||||||
|
|
||||||
err := filepath.WalkDir(*dir, func(path string, d fs.DirEntry, err error) error {
|
err := filepath.WalkDir(*dir, func(path string, d fs.DirEntry, err error) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -42,7 +48,12 @@ func main() {
|
|||||||
return fmt.Errorf("failed to parse yaml %s: %w", path, err)
|
return fmt.Errorf("failed to parse yaml %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
collectKeys(content, "", keys)
|
fileKeys := make(map[string]struct{})
|
||||||
|
collectKeys(content, "", fileKeys)
|
||||||
|
localeFiles = append(localeFiles, localeFile{path: path, keys: fileKeys})
|
||||||
|
for k := range fileKeys {
|
||||||
|
keys[k] = struct{}{}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -50,6 +61,25 @@ func main() {
|
|||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 一致性校验: 每个语言文件必须包含全部 key
|
||||||
|
invalid := false
|
||||||
|
for _, f := range localeFiles {
|
||||||
|
var missing []string
|
||||||
|
for k := range keys {
|
||||||
|
if _, ok := f.keys[k]; !ok {
|
||||||
|
missing = append(missing, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
invalid = true
|
||||||
|
sort.Strings(missing)
|
||||||
|
fmt.Fprintf(os.Stderr, "Error: locale file %s is missing %d key(s): %s\n", f.path, len(missing), strings.Join(missing, ", "))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if invalid {
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
|
||||||
var list []string
|
var list []string
|
||||||
for k := range keys {
|
for k := range keys {
|
||||||
list = append(list, k)
|
list = append(list, k)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"github.com/krau/SaveAny-Bot/cmd/upload"
|
"github.com/krau/SaveAny-Bot/cmd/upload"
|
||||||
|
"github.com/krau/SaveAny-Bot/cmd/watch"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
)
|
)
|
||||||
@@ -18,6 +19,7 @@ var rootCmd = &cobra.Command{
|
|||||||
func init() {
|
func init() {
|
||||||
config.RegisterFlags(rootCmd)
|
config.RegisterFlags(rootCmd)
|
||||||
upload.Register(rootCmd)
|
upload.Register(rootCmd)
|
||||||
|
watch.Register(rootCmd)
|
||||||
}
|
}
|
||||||
|
|
||||||
func Execute(ctx context.Context) {
|
func Execute(ctx context.Context) {
|
||||||
|
|||||||
@@ -61,6 +61,7 @@ func Run(cmd *cobra.Command, _ []string) {
|
|||||||
<-ctx.Done()
|
<-ctx.Done()
|
||||||
logger.Info("Exiting...")
|
logger.Info("Exiting...")
|
||||||
defer logger.Info("Exit complete")
|
defer logger.Info("Exit complete")
|
||||||
|
core.Close()
|
||||||
cleanCache()
|
cleanCache()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,7 +88,7 @@ func initAll(ctx context.Context) (<-chan struct{}, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := api.Start(ctx); err != nil {
|
if err := api.Start(ctx); err != nil {
|
||||||
logger.Error("Failed to start API server", "error", err)
|
logger.Fatal("Failed to start API server", "error", err)
|
||||||
}
|
}
|
||||||
return bot.Init(ctx), nil
|
return bot.Init(ctx), nil
|
||||||
}
|
}
|
||||||
|
|||||||
145
cmd/watch/cmd.go
Normal file
145
cmd/watch/cmd.go
Normal file
@@ -0,0 +1,145 @@
|
|||||||
|
package watch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/charmbracelet/log"
|
||||||
|
"github.com/krau/SaveAny-Bot/client/bot"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/cache"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
|
"github.com/krau/SaveAny-Bot/database"
|
||||||
|
stortype "github.com/krau/SaveAny-Bot/pkg/enums/storage"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
)
|
||||||
|
|
||||||
|
var watchCmd = &cobra.Command{
|
||||||
|
Use: "watch",
|
||||||
|
Short: "watch a local directory and auto-upload changed files to storage",
|
||||||
|
Long: `Watch a local directory and automatically upload created or modified files
|
||||||
|
to the specified storage backend, preserving the relative directory structure.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
saveany-bot watch -p /data/inbox -s mystorage -d backup --recursive`,
|
||||||
|
RunE: runWatch,
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(root *cobra.Command) {
|
||||||
|
flags := watchCmd.Flags()
|
||||||
|
flags.StringP("path", "p", "", "local directory path to watch")
|
||||||
|
watchCmd.MarkFlagRequired("path")
|
||||||
|
flags.StringP("storage", "s", "", "storage name to upload to")
|
||||||
|
watchCmd.MarkFlagRequired("storage")
|
||||||
|
flags.StringP("dir", "d", "", "storage dir to upload to, default is the base_path of the storage")
|
||||||
|
flags.BoolP("recursive", "r", false, "watch subdirectories recursively")
|
||||||
|
flags.Bool("overwrite", false, "overwrite existing files on storage instead of skipping")
|
||||||
|
flags.Bool("initial-scan", false, "upload existing files in the directory on startup")
|
||||||
|
flags.Duration("debounce", 2*time.Second, "wait time after the last change before uploading a file")
|
||||||
|
flags.Int("upload-workers", 0, "number of concurrent uploads, default is config.workers")
|
||||||
|
flags.Duration("retry-delay", 3*time.Second, "delay between upload retries")
|
||||||
|
root.AddCommand(watchCmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
func runWatch(cmd *cobra.Command, _ []string) error {
|
||||||
|
watchPath, err := cmd.Flags().GetString("path")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
storName, err := cmd.Flags().GetString("storage")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
destDir, err := cmd.Flags().GetString("dir")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
recursive, err := cmd.Flags().GetBool("recursive")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
overwrite, err := cmd.Flags().GetBool("overwrite")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
initialScan, err := cmd.Flags().GetBool("initial-scan")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
debounce, err := cmd.Flags().GetDuration("debounce")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
uploadWorkers, err := cmd.Flags().GetInt("upload-workers")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
retryDelay, err := cmd.Flags().GetDuration("retry-delay")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx := cmd.Context()
|
||||||
|
logger := log.FromContext(ctx)
|
||||||
|
|
||||||
|
configFile := config.GetConfigFile(cmd)
|
||||||
|
if err := config.Init(ctx, configFile); err != nil {
|
||||||
|
return fmt.Errorf("failed to load config: %w", err)
|
||||||
|
}
|
||||||
|
i18n.Init(config.C().Lang)
|
||||||
|
cache.Init()
|
||||||
|
database.Init(ctx)
|
||||||
|
|
||||||
|
stor, err := storage.GetStorageByName(ctx, storName)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to get storage %q: %w", storName, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Telegram storage needs the bot client and its ext context injected into ctx.
|
||||||
|
if stor.Type() == stortype.Telegram {
|
||||||
|
bot.Init(ctx)
|
||||||
|
ctx = tgutil.ExtWithContext(ctx, bot.ExtContext())
|
||||||
|
}
|
||||||
|
|
||||||
|
if uploadWorkers < 1 {
|
||||||
|
uploadWorkers = config.C().Workers
|
||||||
|
}
|
||||||
|
|
||||||
|
uploader := NewUploader(ctx, UploaderOptions{
|
||||||
|
Storage: stor,
|
||||||
|
DestDir: destDir,
|
||||||
|
Overwrite: overwrite,
|
||||||
|
Workers: uploadWorkers,
|
||||||
|
Retry: config.C().Retry,
|
||||||
|
RetryDelay: retryDelay,
|
||||||
|
})
|
||||||
|
|
||||||
|
watcher, err := NewWatcher(ctx, WatcherOptions{
|
||||||
|
Root: watchPath,
|
||||||
|
Recursive: recursive,
|
||||||
|
Debounce: debounce,
|
||||||
|
Uploader: uploader,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
uploader.Close()
|
||||||
|
return fmt.Errorf("failed to create watcher: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if initialScan {
|
||||||
|
watcher.ScanExisting(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.Infof("watch started: %s -> storage %q dir %q", watchPath, storName, destDir)
|
||||||
|
|
||||||
|
// Run blocks until ctx is cancelled (e.g. SIGINT).
|
||||||
|
runErr := watcher.Run(ctx)
|
||||||
|
|
||||||
|
// Wait for in-flight uploads to finish before exiting.
|
||||||
|
logger.Info("waiting for in-flight uploads to finish...")
|
||||||
|
uploader.Close()
|
||||||
|
logger.Info("watch stopped")
|
||||||
|
|
||||||
|
return runErr
|
||||||
|
}
|
||||||
227
cmd/watch/uploader.go
Normal file
227
cmd/watch/uploader.go
Normal file
@@ -0,0 +1,227 @@
|
|||||||
|
package watch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/charmbracelet/log"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type uploadJob struct {
|
||||||
|
// localPath is the absolute path of the local file.
|
||||||
|
localPath string
|
||||||
|
// relPath is relative to the watch root, used to preserve directory structure on storage.
|
||||||
|
relPath string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Uploader uploads local files to the target storage via a worker pool.
|
||||||
|
// If a file changes while being uploaded, it is re-uploaded once after the
|
||||||
|
// current upload finishes, instead of being queued multiple times.
|
||||||
|
type Uploader struct {
|
||||||
|
stor storage.Storage
|
||||||
|
destDir string
|
||||||
|
overwrite bool
|
||||||
|
retry int
|
||||||
|
retryDelay time.Duration
|
||||||
|
logger *log.Logger
|
||||||
|
|
||||||
|
jobs chan uploadJob
|
||||||
|
wg sync.WaitGroup
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
// inflight maps in-progress (or queued) file paths. A true value means the
|
||||||
|
// file changed during upload and must be re-queued once done.
|
||||||
|
inflight map[string]bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type UploaderOptions struct {
|
||||||
|
Storage storage.Storage
|
||||||
|
DestDir string
|
||||||
|
Overwrite bool
|
||||||
|
Workers int
|
||||||
|
Retry int
|
||||||
|
RetryDelay time.Duration
|
||||||
|
QueueSize int
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewUploader creates and starts an Uploader. The caller must call Close when done.
|
||||||
|
func NewUploader(ctx context.Context, opts UploaderOptions) *Uploader {
|
||||||
|
if opts.Workers < 1 {
|
||||||
|
opts.Workers = 1
|
||||||
|
}
|
||||||
|
if opts.Retry < 1 {
|
||||||
|
opts.Retry = 1
|
||||||
|
}
|
||||||
|
if opts.RetryDelay <= 0 {
|
||||||
|
opts.RetryDelay = 3 * time.Second
|
||||||
|
}
|
||||||
|
if opts.QueueSize < opts.Workers {
|
||||||
|
opts.QueueSize = opts.Workers * 64
|
||||||
|
}
|
||||||
|
|
||||||
|
u := &Uploader{
|
||||||
|
stor: opts.Storage,
|
||||||
|
destDir: opts.DestDir,
|
||||||
|
overwrite: opts.Overwrite,
|
||||||
|
retry: opts.Retry,
|
||||||
|
retryDelay: opts.RetryDelay,
|
||||||
|
logger: log.FromContext(ctx).WithPrefix("uploader"),
|
||||||
|
jobs: make(chan uploadJob, opts.QueueSize),
|
||||||
|
inflight: make(map[string]bool),
|
||||||
|
}
|
||||||
|
|
||||||
|
for i := 0; i < opts.Workers; i++ {
|
||||||
|
u.wg.Add(1)
|
||||||
|
go u.worker(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
return u
|
||||||
|
}
|
||||||
|
|
||||||
|
// Submit enqueues an upload job. If the same file is already in flight, it is
|
||||||
|
// marked for re-upload instead of being queued again. Returns false if ctx is
|
||||||
|
// cancelled before the job can be enqueued.
|
||||||
|
func (u *Uploader) Submit(ctx context.Context, job uploadJob) bool {
|
||||||
|
u.mu.Lock()
|
||||||
|
if _, ok := u.inflight[job.localPath]; ok {
|
||||||
|
u.inflight[job.localPath] = true
|
||||||
|
u.mu.Unlock()
|
||||||
|
u.logger.Debugf("file %s already in flight, marked for re-upload", job.localPath)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
u.inflight[job.localPath] = false
|
||||||
|
u.mu.Unlock()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case u.jobs <- job:
|
||||||
|
return true
|
||||||
|
case <-ctx.Done():
|
||||||
|
u.mu.Lock()
|
||||||
|
delete(u.inflight, job.localPath)
|
||||||
|
u.mu.Unlock()
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *Uploader) worker(ctx context.Context) {
|
||||||
|
defer u.wg.Done()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case job, ok := <-u.jobs:
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
u.process(ctx, job)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *Uploader) process(ctx context.Context, job uploadJob) {
|
||||||
|
if err := u.uploadWithRetry(ctx, job); err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
u.clearInflight(job.localPath)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
u.logger.Errorf("failed to upload %s after %d attempt(s): %v", job.localPath, u.retry, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-queue if the file changed again while it was being uploaded.
|
||||||
|
u.mu.Lock()
|
||||||
|
needReupload := u.inflight[job.localPath]
|
||||||
|
if needReupload {
|
||||||
|
u.inflight[job.localPath] = false
|
||||||
|
} else {
|
||||||
|
delete(u.inflight, job.localPath)
|
||||||
|
}
|
||||||
|
u.mu.Unlock()
|
||||||
|
|
||||||
|
if needReupload {
|
||||||
|
select {
|
||||||
|
case u.jobs <- job:
|
||||||
|
u.logger.Debugf("re-queued %s due to changes during upload", job.localPath)
|
||||||
|
case <-ctx.Done():
|
||||||
|
u.clearInflight(job.localPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *Uploader) clearInflight(localPath string) {
|
||||||
|
u.mu.Lock()
|
||||||
|
delete(u.inflight, localPath)
|
||||||
|
u.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *Uploader) uploadWithRetry(ctx context.Context, job uploadJob) error {
|
||||||
|
var lastErr error
|
||||||
|
for attempt := 1; attempt <= u.retry; attempt++ {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
err := u.upload(ctx, job)
|
||||||
|
if err == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
lastErr = err
|
||||||
|
u.logger.Warnf("upload %s failed (attempt %d/%d): %v", job.localPath, attempt, u.retry, err)
|
||||||
|
if attempt < u.retry {
|
||||||
|
select {
|
||||||
|
case <-time.After(u.retryDelay):
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return lastErr
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *Uploader) upload(ctx context.Context, job uploadJob) error {
|
||||||
|
file, err := os.Open(filepath.Clean(job.localPath))
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to open file: %w", err)
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
|
||||||
|
info, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to stat file: %w", err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return fmt.Errorf("path is a directory, not a file")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Keep the relative directory structure on the storage side.
|
||||||
|
storagePath := path.Join(u.destDir, filepath.ToSlash(job.relPath))
|
||||||
|
|
||||||
|
uploadCtx := context.WithValue(ctx, ctxkey.ContentLength, info.Size())
|
||||||
|
if u.overwrite {
|
||||||
|
uploadCtx = storage.WithOverwrite(uploadCtx)
|
||||||
|
} else if u.stor.Exists(uploadCtx, storagePath) {
|
||||||
|
u.logger.Infof("skip existing file: %s", storagePath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
u.logger.Infof("uploading %s -> %s (%d bytes)", job.localPath, storagePath, info.Size())
|
||||||
|
if err := u.stor.Save(uploadCtx, file, storagePath); err != nil {
|
||||||
|
return fmt.Errorf("failed to save to storage: %w", err)
|
||||||
|
}
|
||||||
|
u.logger.Infof("uploaded %s", storagePath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops accepting jobs and waits for in-flight uploads to finish.
|
||||||
|
func (u *Uploader) Close() {
|
||||||
|
close(u.jobs)
|
||||||
|
u.wg.Wait()
|
||||||
|
}
|
||||||
269
cmd/watch/watcher.go
Normal file
269
cmd/watch/watcher.go
Normal file
@@ -0,0 +1,269 @@
|
|||||||
|
package watch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/charmbracelet/log"
|
||||||
|
"github.com/fsnotify/fsnotify"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Watcher watches a local directory and submits stable files to the Uploader.
|
||||||
|
//
|
||||||
|
// Write-completion detection: fsnotify emits Write events throughout a write.
|
||||||
|
// Watcher debounces per file and only uploads once the file size stays
|
||||||
|
// unchanged across a debounce window, avoiding uploads of partial files.
|
||||||
|
type Watcher struct {
|
||||||
|
root string
|
||||||
|
recursive bool
|
||||||
|
debounce time.Duration
|
||||||
|
uploader *Uploader
|
||||||
|
logger *log.Logger
|
||||||
|
|
||||||
|
fsw *fsnotify.Watcher
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
pending map[string]*time.Timer
|
||||||
|
// lastSize is the last observed file size, used to detect a stable write.
|
||||||
|
lastSize map[string]int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type WatcherOptions struct {
|
||||||
|
Root string
|
||||||
|
Recursive bool
|
||||||
|
Debounce time.Duration
|
||||||
|
Uploader *Uploader
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewWatcher creates a Watcher.
|
||||||
|
func NewWatcher(ctx context.Context, opts WatcherOptions) (*Watcher, error) {
|
||||||
|
if opts.Debounce <= 0 {
|
||||||
|
opts.Debounce = 2 * time.Second
|
||||||
|
}
|
||||||
|
root, err := filepath.Abs(opts.Root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to resolve root path: %w", err)
|
||||||
|
}
|
||||||
|
info, err := os.Stat(root)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to stat root path: %w", err)
|
||||||
|
}
|
||||||
|
if !info.IsDir() {
|
||||||
|
return nil, fmt.Errorf("watch path must be a directory: %s", root)
|
||||||
|
}
|
||||||
|
|
||||||
|
fsw, err := fsnotify.NewWatcher()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to create fsnotify watcher: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
w := &Watcher{
|
||||||
|
root: root,
|
||||||
|
recursive: opts.Recursive,
|
||||||
|
debounce: opts.Debounce,
|
||||||
|
uploader: opts.Uploader,
|
||||||
|
logger: log.FromContext(ctx).WithPrefix("watcher"),
|
||||||
|
fsw: fsw,
|
||||||
|
pending: make(map[string]*time.Timer),
|
||||||
|
lastSize: make(map[string]int64),
|
||||||
|
}
|
||||||
|
return w, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run starts watching and blocks until ctx is cancelled.
|
||||||
|
func (w *Watcher) Run(ctx context.Context) error {
|
||||||
|
if err := w.addDir(w.root); err != nil {
|
||||||
|
w.fsw.Close()
|
||||||
|
return fmt.Errorf("failed to watch root: %w", err)
|
||||||
|
}
|
||||||
|
w.logger.Infof("watching %s (recursive=%v, debounce=%s)", w.root, w.recursive, w.debounce)
|
||||||
|
|
||||||
|
defer w.cleanup()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
w.logger.Info("stopping watcher")
|
||||||
|
return nil
|
||||||
|
case event, ok := <-w.fsw.Events:
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
w.handleEvent(ctx, event)
|
||||||
|
case err, ok := <-w.fsw.Errors:
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
w.logger.Errorf("watch error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *Watcher) handleEvent(ctx context.Context, event fsnotify.Event) {
|
||||||
|
// Remove/Rename: cancel any pending upload for this path.
|
||||||
|
if event.Has(fsnotify.Remove) || event.Has(fsnotify.Rename) {
|
||||||
|
w.cancelPending(event.Name)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if !event.Has(fsnotify.Create) && !event.Has(fsnotify.Write) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Stat(event.Name)
|
||||||
|
if err != nil {
|
||||||
|
// File may have been removed or moved; ignore.
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if info.IsDir() {
|
||||||
|
// New directory: watch it recursively and scan files already inside.
|
||||||
|
if event.Has(fsnotify.Create) && w.recursive {
|
||||||
|
if err := w.addDir(event.Name); err != nil {
|
||||||
|
w.logger.Errorf("failed to watch new dir %s: %v", event.Name, err)
|
||||||
|
}
|
||||||
|
w.scanExisting(ctx, event.Name)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
w.scheduleUpload(ctx, event.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// scheduleUpload schedules a debounced upload for a file.
|
||||||
|
func (w *Watcher) scheduleUpload(ctx context.Context, file string) {
|
||||||
|
w.mu.Lock()
|
||||||
|
defer w.mu.Unlock()
|
||||||
|
|
||||||
|
if t, ok := w.pending[file]; ok {
|
||||||
|
t.Stop()
|
||||||
|
}
|
||||||
|
w.pending[file] = time.AfterFunc(w.debounce, func() {
|
||||||
|
w.maybeUpload(ctx, file)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeUpload submits the upload once the debounce window passes and the file
|
||||||
|
// size is stable; otherwise it waits another window.
|
||||||
|
func (w *Watcher) maybeUpload(ctx context.Context, file string) {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Stat(file)
|
||||||
|
if err != nil {
|
||||||
|
w.cancelPending(file)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
w.cancelPending(file)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
w.mu.Lock()
|
||||||
|
prevSize, seen := w.lastSize[file]
|
||||||
|
curSize := info.Size()
|
||||||
|
if !seen || prevSize != curSize {
|
||||||
|
// Size still changing: likely still being written, wait another window.
|
||||||
|
w.lastSize[file] = curSize
|
||||||
|
w.pending[file] = time.AfterFunc(w.debounce, func() {
|
||||||
|
w.maybeUpload(ctx, file)
|
||||||
|
})
|
||||||
|
w.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Size stable: treat write as complete.
|
||||||
|
delete(w.pending, file)
|
||||||
|
delete(w.lastSize, file)
|
||||||
|
w.mu.Unlock()
|
||||||
|
|
||||||
|
relPath, err := filepath.Rel(w.root, file)
|
||||||
|
if err != nil {
|
||||||
|
w.logger.Errorf("failed to compute relative path for %s: %v", file, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
w.uploader.Submit(ctx, uploadJob{localPath: file, relPath: relPath})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *Watcher) cancelPending(file string) {
|
||||||
|
w.mu.Lock()
|
||||||
|
defer w.mu.Unlock()
|
||||||
|
if t, ok := w.pending[file]; ok {
|
||||||
|
t.Stop()
|
||||||
|
delete(w.pending, file)
|
||||||
|
}
|
||||||
|
delete(w.lastSize, file)
|
||||||
|
}
|
||||||
|
|
||||||
|
// addDir adds a directory to the watch list, recursively when enabled.
|
||||||
|
func (w *Watcher) addDir(dir string) error {
|
||||||
|
if !w.recursive {
|
||||||
|
return w.fsw.Add(dir)
|
||||||
|
}
|
||||||
|
return filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
w.logger.Warnf("skip path %s: %v", p, err)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
if addErr := w.fsw.Add(p); addErr != nil {
|
||||||
|
w.logger.Warnf("failed to watch dir %s: %v", p, addErr)
|
||||||
|
} else {
|
||||||
|
w.logger.Debugf("watching dir %s", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanExisting submits files already present under dir (initial sync and new-dir backfill).
|
||||||
|
func (w *Watcher) scanExisting(ctx context.Context, dir string) {
|
||||||
|
walkFn := func(p string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
w.logger.Warnf("skip path %s: %v", p, err)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if d.IsDir() {
|
||||||
|
if !w.recursive && p != dir {
|
||||||
|
return fs.SkipDir
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
relPath, relErr := filepath.Rel(w.root, p)
|
||||||
|
if relErr != nil {
|
||||||
|
w.logger.Errorf("failed to compute relative path for %s: %v", p, relErr)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
w.uploader.Submit(ctx, uploadJob{localPath: p, relPath: relPath})
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := filepath.WalkDir(dir, walkFn); err != nil && ctx.Err() == nil {
|
||||||
|
w.logger.Errorf("failed to scan dir %s: %v", dir, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ScanExisting triggers a one-time scan and upload of existing files under the watch root.
|
||||||
|
func (w *Watcher) ScanExisting(ctx context.Context) {
|
||||||
|
w.logger.Info("scanning existing files for initial sync")
|
||||||
|
w.scanExisting(ctx, w.root)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *Watcher) cleanup() {
|
||||||
|
w.mu.Lock()
|
||||||
|
for _, t := range w.pending {
|
||||||
|
t.Stop()
|
||||||
|
}
|
||||||
|
w.pending = make(map[string]*time.Timer)
|
||||||
|
w.lastSize = make(map[string]int64)
|
||||||
|
w.mu.Unlock()
|
||||||
|
w.fsw.Close()
|
||||||
|
}
|
||||||
33
common/cache/ristretto.go
vendored
33
common/cache/ristretto.go
vendored
@@ -2,6 +2,7 @@ package cache
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
@@ -9,24 +10,26 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
var cache *ristretto.Cache[string, any]
|
var (
|
||||||
|
cache *ristretto.Cache[string, any]
|
||||||
|
initOnce sync.Once
|
||||||
|
)
|
||||||
|
|
||||||
func Init() {
|
func Init() {
|
||||||
if cache != nil {
|
initOnce.Do(func() {
|
||||||
panic("cache already initialized")
|
c, err := ristretto.NewCache(&ristretto.Config[string, any]{
|
||||||
}
|
NumCounters: config.C().Cache.NumCounters,
|
||||||
c, err := ristretto.NewCache(&ristretto.Config[string, any]{
|
MaxCost: config.C().Cache.MaxCost,
|
||||||
NumCounters: config.C().Cache.NumCounters,
|
BufferItems: 64,
|
||||||
MaxCost: config.C().Cache.MaxCost,
|
OnReject: func(item *ristretto.Item[any]) {
|
||||||
BufferItems: 64,
|
log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
|
||||||
OnReject: func(item *ristretto.Item[any]) {
|
},
|
||||||
log.Warnf("Cache item rejected: key=%d, value=%v", item.Key, item.Value)
|
})
|
||||||
},
|
if err != nil {
|
||||||
|
log.Fatalf("failed to create ristretto cache: %v", err)
|
||||||
|
}
|
||||||
|
cache = c
|
||||||
})
|
})
|
||||||
if err != nil {
|
|
||||||
log.Fatalf("failed to create ristretto cache: %v", err)
|
|
||||||
}
|
|
||||||
cache = c
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func Set(key string, value any) error {
|
func Set(key string, value any) error {
|
||||||
|
|||||||
@@ -84,8 +84,8 @@ const (
|
|||||||
BotMsgCommonPromptSelectDefaultDir Key = "bot.msg.common.prompt_select_default_dir"
|
BotMsgCommonPromptSelectDefaultDir Key = "bot.msg.common.prompt_select_default_dir"
|
||||||
BotMsgCommonPromptSelectDefaultStorage Key = "bot.msg.common.prompt_select_default_storage"
|
BotMsgCommonPromptSelectDefaultStorage Key = "bot.msg.common.prompt_select_default_storage"
|
||||||
BotMsgCommonPromptSelectDir Key = "bot.msg.common.prompt_select_dir"
|
BotMsgCommonPromptSelectDir Key = "bot.msg.common.prompt_select_dir"
|
||||||
BotMsgConfigButtonFilenameStrategy Key = "bot.msg.config.button_filename_strategy"
|
|
||||||
BotMsgConfigButtonConflictStrategy Key = "bot.msg.config.button_conflict_strategy"
|
BotMsgConfigButtonConflictStrategy Key = "bot.msg.config.button_conflict_strategy"
|
||||||
|
BotMsgConfigButtonFilenameStrategy Key = "bot.msg.config.button_filename_strategy"
|
||||||
BotMsgConfigConflictStrategyAsk Key = "bot.msg.config.conflict_strategy_ask"
|
BotMsgConfigConflictStrategyAsk Key = "bot.msg.config.conflict_strategy_ask"
|
||||||
BotMsgConfigConflictStrategyOverwrite Key = "bot.msg.config.conflict_strategy_overwrite"
|
BotMsgConfigConflictStrategyOverwrite Key = "bot.msg.config.conflict_strategy_overwrite"
|
||||||
BotMsgConfigConflictStrategyRename Key = "bot.msg.config.conflict_strategy_rename"
|
BotMsgConfigConflictStrategyRename Key = "bot.msg.config.conflict_strategy_rename"
|
||||||
@@ -93,8 +93,8 @@ const (
|
|||||||
BotMsgConfigErrorInvalidCallbackData Key = "bot.msg.config.error_invalid_callback_data"
|
BotMsgConfigErrorInvalidCallbackData Key = "bot.msg.config.error_invalid_callback_data"
|
||||||
BotMsgConfigErrorInvalidTemplate Key = "bot.msg.config.error_invalid_template"
|
BotMsgConfigErrorInvalidTemplate Key = "bot.msg.config.error_invalid_template"
|
||||||
BotMsgConfigFnametmplHelp Key = "bot.msg.config.fnametmpl_help"
|
BotMsgConfigFnametmplHelp Key = "bot.msg.config.fnametmpl_help"
|
||||||
BotMsgConfigInfoCurrentTemplatePrefix Key = "bot.msg.config.info_current_template_prefix"
|
|
||||||
BotMsgConfigInfoConflictStrategySet Key = "bot.msg.config.info_conflict_strategy_set"
|
BotMsgConfigInfoConflictStrategySet Key = "bot.msg.config.info_conflict_strategy_set"
|
||||||
|
BotMsgConfigInfoCurrentTemplatePrefix Key = "bot.msg.config.info_current_template_prefix"
|
||||||
BotMsgConfigInfoFilenameStrategySet Key = "bot.msg.config.info_filename_strategy_set"
|
BotMsgConfigInfoFilenameStrategySet Key = "bot.msg.config.info_filename_strategy_set"
|
||||||
BotMsgConfigInfoTemplateUpdated Key = "bot.msg.config.info_template_updated"
|
BotMsgConfigInfoTemplateUpdated Key = "bot.msg.config.info_template_updated"
|
||||||
BotMsgConfigPromptSelectConflictStrategy Key = "bot.msg.config.prompt_select_conflict_strategy"
|
BotMsgConfigPromptSelectConflictStrategy Key = "bot.msg.config.prompt_select_conflict_strategy"
|
||||||
@@ -150,28 +150,52 @@ const (
|
|||||||
BotMsgProgressAria2Downloading Key = "bot.msg.progress.aria2_downloading"
|
BotMsgProgressAria2Downloading Key = "bot.msg.progress.aria2_downloading"
|
||||||
BotMsgProgressAria2Start Key = "bot.msg.progress.aria2_start"
|
BotMsgProgressAria2Start Key = "bot.msg.progress.aria2_start"
|
||||||
BotMsgProgressAvgSpeedPrefix Key = "bot.msg.progress.avg_speed_prefix"
|
BotMsgProgressAvgSpeedPrefix Key = "bot.msg.progress.avg_speed_prefix"
|
||||||
BotMsgProgressBatchDonePrefix Key = "bot.msg.progress.batch_done_prefix"
|
BotMsgProgressBatchCanceled Key = "bot.msg.progress.batch_canceled"
|
||||||
BotMsgProgressBatchProcessingPrefix Key = "bot.msg.progress.batch_processing_prefix"
|
BotMsgProgressBatchDone Key = "bot.msg.progress.batch_done"
|
||||||
BotMsgProgressBatchStartPrefix Key = "bot.msg.progress.batch_start_prefix"
|
BotMsgProgressBatchDoneWithSkipped Key = "bot.msg.progress.batch_done_with_skipped"
|
||||||
|
BotMsgProgressBatchFailedGroup Key = "bot.msg.progress.batch_failed_group"
|
||||||
|
BotMsgProgressBatchFailedItem Key = "bot.msg.progress.batch_failed_item"
|
||||||
|
BotMsgProgressBatchFailedTask Key = "bot.msg.progress.batch_failed_task"
|
||||||
|
BotMsgProgressBatchFailureStageBatchUpload Key = "bot.msg.progress.batch_failure_stage_batch_upload"
|
||||||
|
BotMsgProgressBatchFailureStageCache Key = "bot.msg.progress.batch_failure_stage_cache"
|
||||||
|
BotMsgProgressBatchFailureStageConfirm Key = "bot.msg.progress.batch_failure_stage_confirm"
|
||||||
|
BotMsgProgressBatchFailureStageDownload Key = "bot.msg.progress.batch_failure_stage_download"
|
||||||
|
BotMsgProgressBatchFailureStageInternal Key = "bot.msg.progress.batch_failure_stage_internal"
|
||||||
|
BotMsgProgressBatchFailureStageUpload Key = "bot.msg.progress.batch_failure_stage_upload"
|
||||||
|
BotMsgProgressBatchItemConfirming Key = "bot.msg.progress.batch_item_confirming"
|
||||||
|
BotMsgProgressBatchItemDownloading Key = "bot.msg.progress.batch_item_downloading"
|
||||||
|
BotMsgProgressBatchItemDownloadingUnknown Key = "bot.msg.progress.batch_item_downloading_unknown"
|
||||||
|
BotMsgProgressBatchItemRetrying Key = "bot.msg.progress.batch_item_retrying"
|
||||||
|
BotMsgProgressBatchItemTransferring Key = "bot.msg.progress.batch_item_transferring"
|
||||||
|
BotMsgProgressBatchItemTransferringUnknown Key = "bot.msg.progress.batch_item_transferring_unknown"
|
||||||
|
BotMsgProgressBatchItemUploading Key = "bot.msg.progress.batch_item_uploading"
|
||||||
|
BotMsgProgressBatchStatusHeader Key = "bot.msg.progress.batch_status_header"
|
||||||
|
BotMsgProgressBatchSummaryConfirming Key = "bot.msg.progress.batch_summary_confirming"
|
||||||
|
BotMsgProgressBatchSummaryFailed Key = "bot.msg.progress.batch_summary_failed"
|
||||||
|
BotMsgProgressBatchSummaryHiddenActive Key = "bot.msg.progress.batch_summary_hidden_active"
|
||||||
|
BotMsgProgressBatchSummarySkipped Key = "bot.msg.progress.batch_summary_skipped"
|
||||||
BotMsgProgressCurrentProgressPrefix Key = "bot.msg.progress.current_progress_prefix"
|
BotMsgProgressCurrentProgressPrefix Key = "bot.msg.progress.current_progress_prefix"
|
||||||
BotMsgProgressCurrentSpeedPrefix Key = "bot.msg.progress.current_speed_prefix"
|
BotMsgProgressCurrentSpeedPrefix Key = "bot.msg.progress.current_speed_prefix"
|
||||||
BotMsgProgressDirectDonePrefix Key = "bot.msg.progress.direct_done_prefix"
|
BotMsgProgressDirectDonePrefix Key = "bot.msg.progress.direct_done_prefix"
|
||||||
BotMsgProgressDirectStart Key = "bot.msg.progress.direct_start"
|
BotMsgProgressDirectStart Key = "bot.msg.progress.direct_start"
|
||||||
BotMsgProgressDownloadDonePrefix Key = "bot.msg.progress.download_done_prefix"
|
|
||||||
BotMsgProgressDownloadFailedPrefix Key = "bot.msg.progress.download_failed_prefix"
|
|
||||||
BotMsgProgressDownloadedPrefix Key = "bot.msg.progress.downloaded_prefix"
|
BotMsgProgressDownloadedPrefix Key = "bot.msg.progress.downloaded_prefix"
|
||||||
BotMsgProgressDownloadingPrefix Key = "bot.msg.progress.downloading_prefix"
|
BotMsgProgressDownloadingPrefix Key = "bot.msg.progress.downloading_prefix"
|
||||||
BotMsgProgressErrorPrefix Key = "bot.msg.progress.error_prefix"
|
BotMsgProgressErrorPrefix Key = "bot.msg.progress.error_prefix"
|
||||||
BotMsgProgressFileNamePrefix Key = "bot.msg.progress.file_name_prefix"
|
|
||||||
BotMsgProgressFileProcessingPrefix Key = "bot.msg.progress.file_processing_prefix"
|
|
||||||
BotMsgProgressFileSizePrefix Key = "bot.msg.progress.file_size_prefix"
|
|
||||||
BotMsgProgressFileStartPrefix Key = "bot.msg.progress.file_start_prefix"
|
|
||||||
BotMsgProgressParsedDonePrefix Key = "bot.msg.progress.parsed_done_prefix"
|
BotMsgProgressParsedDonePrefix Key = "bot.msg.progress.parsed_done_prefix"
|
||||||
BotMsgProgressParsedStartPrefix Key = "bot.msg.progress.parsed_start_prefix"
|
BotMsgProgressParsedStartPrefix Key = "bot.msg.progress.parsed_start_prefix"
|
||||||
BotMsgProgressProcessingListPrefix Key = "bot.msg.progress.processing_list_prefix"
|
BotMsgProgressProcessingListPrefix Key = "bot.msg.progress.processing_list_prefix"
|
||||||
BotMsgProgressProcessingNone Key = "bot.msg.progress.processing_none"
|
BotMsgProgressProcessingNone Key = "bot.msg.progress.processing_none"
|
||||||
BotMsgProgressSavePathPrefix Key = "bot.msg.progress.save_path_prefix"
|
BotMsgProgressSavePathPrefix Key = "bot.msg.progress.save_path_prefix"
|
||||||
BotMsgProgressTaskCanceled Key = "bot.msg.progress.task_canceled"
|
BotMsgProgressSingleCanceled Key = "bot.msg.progress.single_canceled"
|
||||||
|
BotMsgProgressSingleDone Key = "bot.msg.progress.single_done"
|
||||||
|
BotMsgProgressSingleDownloading Key = "bot.msg.progress.single_downloading"
|
||||||
|
BotMsgProgressSingleDownloadingUnknown Key = "bot.msg.progress.single_downloading_unknown"
|
||||||
|
BotMsgProgressSingleFailed Key = "bot.msg.progress.single_failed"
|
||||||
|
BotMsgProgressSingleStatusHeader Key = "bot.msg.progress.single_status_header"
|
||||||
|
BotMsgProgressSingleUploadRetrying Key = "bot.msg.progress.single_upload_retrying"
|
||||||
|
BotMsgProgressSingleUploading Key = "bot.msg.progress.single_uploading"
|
||||||
|
BotMsgProgressSizeWithFiles Key = "bot.msg.progress.size_with_files"
|
||||||
|
BotMsgProgressSizeWithResources Key = "bot.msg.progress.size_with_resources"
|
||||||
BotMsgProgressTaskCanceledWithId Key = "bot.msg.progress.task_canceled_with_id"
|
BotMsgProgressTaskCanceledWithId Key = "bot.msg.progress.task_canceled_with_id"
|
||||||
BotMsgProgressTaskFailedWithError Key = "bot.msg.progress.task_failed_with_error"
|
BotMsgProgressTaskFailedWithError Key = "bot.msg.progress.task_failed_with_error"
|
||||||
BotMsgProgressTelegraphDonePrefix Key = "bot.msg.progress.telegraph_done_prefix"
|
BotMsgProgressTelegraphDonePrefix Key = "bot.msg.progress.telegraph_done_prefix"
|
||||||
@@ -200,6 +224,7 @@ const (
|
|||||||
BotMsgRuleErrorGetUserRulesFailed Key = "bot.msg.rule.error_get_user_rules_failed"
|
BotMsgRuleErrorGetUserRulesFailed Key = "bot.msg.rule.error_get_user_rules_failed"
|
||||||
BotMsgRuleErrorInvalidRuleId Key = "bot.msg.rule.error_invalid_rule_id"
|
BotMsgRuleErrorInvalidRuleId Key = "bot.msg.rule.error_invalid_rule_id"
|
||||||
BotMsgRuleErrorInvalidRuleType Key = "bot.msg.rule.error_invalid_rule_type"
|
BotMsgRuleErrorInvalidRuleType Key = "bot.msg.rule.error_invalid_rule_type"
|
||||||
|
BotMsgRuleErrorStorageNotFound Key = "bot.msg.rule.error_storage_not_found"
|
||||||
BotMsgRuleErrorUpdateUserFailed Key = "bot.msg.rule.error_update_user_failed"
|
BotMsgRuleErrorUpdateUserFailed Key = "bot.msg.rule.error_update_user_failed"
|
||||||
BotMsgRuleHelpAddSuffix Key = "bot.msg.rule.help_add_suffix"
|
BotMsgRuleHelpAddSuffix Key = "bot.msg.rule.help_add_suffix"
|
||||||
BotMsgRuleHelpAvailableOps Key = "bot.msg.rule.help_available_ops"
|
BotMsgRuleHelpAvailableOps Key = "bot.msg.rule.help_available_ops"
|
||||||
@@ -207,18 +232,20 @@ const (
|
|||||||
BotMsgRuleHelpCurrentModeEnabled Key = "bot.msg.rule.help_current_mode_enabled"
|
BotMsgRuleHelpCurrentModeEnabled Key = "bot.msg.rule.help_current_mode_enabled"
|
||||||
BotMsgRuleHelpDelSuffix Key = "bot.msg.rule.help_del_suffix"
|
BotMsgRuleHelpDelSuffix Key = "bot.msg.rule.help_del_suffix"
|
||||||
BotMsgRuleHelpExistingRulesPrefix Key = "bot.msg.rule.help_existing_rules_prefix"
|
BotMsgRuleHelpExistingRulesPrefix Key = "bot.msg.rule.help_existing_rules_prefix"
|
||||||
|
BotMsgRuleHelpPresetSuffix Key = "bot.msg.rule.help_preset_suffix"
|
||||||
BotMsgRuleHelpSwitchSuffix Key = "bot.msg.rule.help_switch_suffix"
|
BotMsgRuleHelpSwitchSuffix Key = "bot.msg.rule.help_switch_suffix"
|
||||||
BotMsgRuleHelpUsage Key = "bot.msg.rule.help_usage"
|
BotMsgRuleHelpUsage Key = "bot.msg.rule.help_usage"
|
||||||
BotMsgRuleInfoCreateRuleSuccess Key = "bot.msg.rule.info_create_rule_success"
|
BotMsgRuleInfoCreateRuleSuccess Key = "bot.msg.rule.info_create_rule_success"
|
||||||
BotMsgRuleInfoDeleteRuleSuccess Key = "bot.msg.rule.info_delete_rule_success"
|
BotMsgRuleInfoDeleteRuleSuccess Key = "bot.msg.rule.info_delete_rule_success"
|
||||||
|
BotMsgRuleInfoPresetImported Key = "bot.msg.rule.info_preset_imported"
|
||||||
BotMsgRuleInfoRuleModeDisabled Key = "bot.msg.rule.info_rule_mode_disabled"
|
BotMsgRuleInfoRuleModeDisabled Key = "bot.msg.rule.info_rule_mode_disabled"
|
||||||
BotMsgRuleInfoRuleModeEnabled Key = "bot.msg.rule.info_rule_mode_enabled"
|
BotMsgRuleInfoRuleModeEnabled Key = "bot.msg.rule.info_rule_mode_enabled"
|
||||||
BotMsgRulePromptProvideRuleId Key = "bot.msg.rule.prompt_provide_rule_id"
|
BotMsgRulePromptProvideRuleId Key = "bot.msg.rule.prompt_provide_rule_id"
|
||||||
|
BotMsgRulePromptProvideStorageName Key = "bot.msg.rule.prompt_provide_storage_name"
|
||||||
BotMsgSaveErrorInvalidIdOrUsername Key = "bot.msg.save.error_invalid_id_or_username"
|
BotMsgSaveErrorInvalidIdOrUsername Key = "bot.msg.save.error_invalid_id_or_username"
|
||||||
BotMsgSaveHelpText Key = "bot.msg.save_help_text"
|
BotMsgSaveHelpText Key = "bot.msg.save_help_text"
|
||||||
BotMsgStorageInfoFilenamePrefix Key = "bot.msg.storage.info_filename_prefix"
|
BotMsgStorageInfoFilenamePrefix Key = "bot.msg.storage.info_filename_prefix"
|
||||||
BotMsgStorageInfoPromptSelectStorage Key = "bot.msg.storage.info_prompt_select_storage"
|
BotMsgStorageInfoPromptSelectStorage Key = "bot.msg.storage.info_prompt_select_storage"
|
||||||
BotMsgSyncpeersDone Key = "bot.msg.syncpeers.done"
|
|
||||||
BotMsgSyncpeersFailed Key = "bot.msg.syncpeers.failed"
|
BotMsgSyncpeersFailed Key = "bot.msg.syncpeers.failed"
|
||||||
BotMsgSyncpeersStart Key = "bot.msg.syncpeers.start"
|
BotMsgSyncpeersStart Key = "bot.msg.syncpeers.start"
|
||||||
BotMsgSyncpeersSuccess Key = "bot.msg.syncpeers.success"
|
BotMsgSyncpeersSuccess Key = "bot.msg.syncpeers.success"
|
||||||
|
|||||||
@@ -196,7 +196,11 @@ bot:
|
|||||||
help_switch_suffix: " - Toggle rule mode\n"
|
help_switch_suffix: " - Toggle rule mode\n"
|
||||||
help_add_suffix: " <type> <data> <storage_name> <path> - Add rule\n"
|
help_add_suffix: " <type> <data> <storage_name> <path> - Add rule\n"
|
||||||
help_del_suffix: " <rule_id> - Delete rule\n"
|
help_del_suffix: " <rule_id> - Delete rule\n"
|
||||||
|
help_preset_suffix: " <storage_name> [base_path] - Import built-in filetype rules (video/image/audio/document/archive)\n"
|
||||||
help_existing_rules_prefix: "\nCurrent rules:\n"
|
help_existing_rules_prefix: "\nCurrent rules:\n"
|
||||||
|
prompt_provide_storage_name: "Please provide a storage name"
|
||||||
|
error_storage_not_found: "Storage not found: {{.Storage}}"
|
||||||
|
info_preset_imported: "Imported {{.Count}} built-in classification rules into storage {{.Storage}}"
|
||||||
dir:
|
dir:
|
||||||
error_get_user_dirs_failed: "Failed to get user directories"
|
error_get_user_dirs_failed: "Failed to get user directories"
|
||||||
error_get_user_failed: "Failed to get user"
|
error_get_user_failed: "Failed to get user"
|
||||||
@@ -347,32 +351,56 @@ bot:
|
|||||||
info_filename_prefix: "Filename: "
|
info_filename_prefix: "Filename: "
|
||||||
info_prompt_select_storage: "\nPlease select storage"
|
info_prompt_select_storage: "\nPlease select storage"
|
||||||
progress:
|
progress:
|
||||||
batch_start_prefix: "Starting batch download task\nTotal size: "
|
batch_status_header: "<b>📦 Processing</b>\n\nFiles: <code>{{.Total}}</code>\nStatus: ✅ <code>{{.Completed}}</code> | 📥 <code>{{.Downloaded}}</code> | ⏳ <code>{{.Waiting}}</code>\nTotal speed: ⬇️ <code>{{.DownloadSpeed}}</code> | ⬆️ <code>{{.UploadSpeed}}</code>"
|
||||||
batch_processing_prefix: "Processing batch download task\nTotal size: "
|
batch_item_downloading: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} Downloading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_downloading_unknown: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} Downloading</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown</blockquote>"
|
||||||
|
batch_item_transferring: "<blockquote><b>↕️ {{.Index}}/{{.Total}} Transferring</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_transferring_unknown: "<blockquote><b>↕️ {{.Index}}/{{.Total}} Transferring</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown</blockquote>"
|
||||||
|
batch_item_uploading: "<blockquote><b>⬆️ {{.Index}}/{{.Total}} Uploading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_retrying: "<blockquote><b>🔁 {{.Index}}/{{.Total}} Retrying upload</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nRetry: <code>{{.Attempt}}/{{.Limit}}</code>\nSpeed before failure: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nReason: <code>{{.Reason}}</code></blockquote>"
|
||||||
|
batch_item_confirming: "<blockquote><b>⏳ {{.Index}}/{{.Total}} Waiting</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n<i>Uploaded, awaiting remote confirmation</i></blockquote>"
|
||||||
|
batch_summary_hidden_active: "🔄 <code>{{.Count}}</code> more files are active"
|
||||||
|
batch_summary_confirming: "☁️ Uploaded, awaiting group send: <code>{{.Count}}</code>"
|
||||||
|
batch_summary_failed: "❌ Failed: <code>{{.Count}}</code>"
|
||||||
|
batch_summary_skipped: "⏭️ Skipped: <code>{{.Count}}</code>"
|
||||||
|
batch_done: "<b>✅ Completed</b>\n\nFiles: <code>{{.Count}}</code>\nTotal size: <code>{{.Size}}</code>"
|
||||||
|
batch_done_with_skipped: "<b>⚠️ Completed</b>\n\nSucceeded: <code>{{.Success}}</code>\nSkipped: <code>{{.Skipped}}</code>\nTotal size: <code>{{.Size}}</code>"
|
||||||
|
batch_canceled: "<b>🚫 Task canceled</b>\n\nFiles: <code>{{.Total}}</code>\nCompleted: <code>{{.Completed}}</code>\nIncomplete: <code>{{.Incomplete}}</code>\nSkipped: <code>{{.Skipped}}</code>"
|
||||||
|
batch_failed_item: "<b>❌ Processing failed</b>\n\nFailed file: <code>{{.Index}}. {{.Name}}</code>\nStage: <code>{{.Stage}}</code>\nProgress: <code>{{.Progress}}</code>\nSpeed before failure: <code>{{.Speed}}</code>\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n❌ Failed: <code>{{.Failed}}</code>\n⏹️ Incomplete: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failed_group: "<b>❌ Batch upload failed</b>\n\nAffected files: <code>{{.Affected}}</code>\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n❌ Batch failed: <code>{{.Failed}}</code>\n⏹️ Incomplete: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failed_task: "<b>❌ Processing failed</b>\n\nReason: <code>{{.Reason}}</code>\n\n✅ Completed: <code>{{.Completed}}</code>\n⏹️ Incomplete: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failure_stage_download: "download"
|
||||||
|
batch_failure_stage_cache: "local cache"
|
||||||
|
batch_failure_stage_upload: "upload"
|
||||||
|
batch_failure_stage_confirm: "remote confirmation"
|
||||||
|
batch_failure_stage_batch_upload: "batch upload"
|
||||||
|
batch_failure_stage_internal: "internal task"
|
||||||
|
single_status_header: "<b>📦 Processing</b>"
|
||||||
|
single_downloading: "<blockquote><b>⬇️ Downloading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_downloading_unknown: "<blockquote><b>⬇️ Downloading</b>\n<code>{{.Name}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / unknown\nSave to: <code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_uploading: "<blockquote><b>⬆️ Uploading</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_upload_retrying: "<blockquote><b>🔁 Retrying upload</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\nAttempt: <code>{{.Attempt}}</code>\nSpeed: <code>{{.Speed}}</code>\nSize: <code>{{.Current}}</code> / <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_done: "<b>✅ Completed</b>\n\nFilename: <code>{{.Name}}</code>\nTotal size: <code>{{.Size}}</code>\nSave to: <code>{{.Destination}}</code>"
|
||||||
|
single_canceled: "<b>🚫 Task canceled</b>\n\nFilename: <code>{{.Name}}</code>"
|
||||||
|
single_failed: "<b>❌ Processing failed</b>\n\nFilename: <code>{{.Name}}</code>\nReason: <code>{{.Reason}}</code>"
|
||||||
downloading_prefix: "Downloading\nTotal size: "
|
downloading_prefix: "Downloading\nTotal size: "
|
||||||
|
size_with_files: "{{.Size}} ({{.Count}} files)"
|
||||||
|
size_with_resources: "{{.Size}} ({{.Count}} resources)"
|
||||||
processing_list_prefix: "\nProcessing:\n"
|
processing_list_prefix: "\nProcessing:\n"
|
||||||
processing_none: " - None"
|
processing_none: " - None"
|
||||||
avg_speed_prefix: "\nAverage speed: "
|
avg_speed_prefix: "\nAverage speed: "
|
||||||
current_progress_prefix: "\nCurrent progress: "
|
current_progress_prefix: "\nCurrent progress: "
|
||||||
task_canceled: "Task canceled"
|
|
||||||
task_canceled_with_id: "Processing canceled: {{.TaskID}}"
|
task_canceled_with_id: "Processing canceled: {{.TaskID}}"
|
||||||
task_failed_with_error: "Processing failed: {{.Error}}"
|
task_failed_with_error: "Processing failed: {{.Error}}"
|
||||||
batch_done_prefix: "Completed\nFile count: "
|
|
||||||
direct_done_prefix: "Completed, file count: "
|
direct_done_prefix: "Completed, file count: "
|
||||||
parsed_start_prefix: "Starting download from {{.Site}}\nTotal size: "
|
parsed_start_prefix: "Starting download from {{.Site}}\nTotal size: "
|
||||||
parsed_done_prefix: "Completed, resource count: "
|
parsed_done_prefix: "Completed, resource count: "
|
||||||
telegraph_start_prefix: "Starting Telegraph download\nImage count: "
|
telegraph_start_prefix: "Starting Telegraph download\nImage count: "
|
||||||
telegraph_progress_prefix: "Downloading\nCurrent progress: "
|
telegraph_progress_prefix: "Downloading\nCurrent progress: "
|
||||||
telegraph_done_prefix: "Completed\nImage count: "
|
telegraph_done_prefix: "Completed\nImage count: "
|
||||||
file_start_prefix: "Starting download\nFilename: "
|
|
||||||
file_processing_prefix: "Processing download task\nFilename: "
|
|
||||||
download_failed_prefix: "Download failed\nFilename: "
|
|
||||||
download_done_prefix: "Download completed\nFilename: "
|
|
||||||
file_size_prefix: "\nFile size: "
|
|
||||||
save_path_prefix: "\nSave path: "
|
save_path_prefix: "\nSave path: "
|
||||||
total_size_prefix: "\nTotal size: "
|
total_size_prefix: "\nTotal size: "
|
||||||
direct_start: "Starting download, total size: {{.SizeMB}} MB ({{.Count}} files)"
|
direct_start: "Starting download, total size: {{.SizeMB}} MB ({{.Count}} files)"
|
||||||
file_name_prefix: "Filename: "
|
|
||||||
error_prefix: "\nError: "
|
error_prefix: "\nError: "
|
||||||
aria2_start: "Waiting for Aria2 to complete download (GID: {{.GID}})..."
|
aria2_start: "Waiting for Aria2 to complete download (GID: {{.GID}})..."
|
||||||
aria2_downloading: "Aria2 downloading (GID: {{.GID}})\n"
|
aria2_downloading: "Aria2 downloading (GID: {{.GID}})\n"
|
||||||
@@ -398,7 +426,7 @@ bot:
|
|||||||
transfer_failed_files_prefix: "\nFailed files: "
|
transfer_failed_files_prefix: "\nFailed files: "
|
||||||
syncpeers:
|
syncpeers:
|
||||||
start: "Starting to sync peers..."
|
start: "Starting to sync peers..."
|
||||||
done: "Peer sync completed, total {{.Count}} chats synced"
|
success: "Peer sync completed, total {{.Count}} chats synced"
|
||||||
failed: "Peer sync failed: {{.Error}}"
|
failed: "Peer sync failed: {{.Error}}"
|
||||||
aria2:
|
aria2:
|
||||||
error_aria2_not_enabled: "Aria2 feature is not enabled in the configuration"
|
error_aria2_not_enabled: "Aria2 feature is not enabled in the configuration"
|
||||||
|
|||||||
@@ -197,7 +197,11 @@ bot:
|
|||||||
help_switch_suffix: " - 开关规则模式\n"
|
help_switch_suffix: " - 开关规则模式\n"
|
||||||
help_add_suffix: " <类型> <数据> <存储名> <路径> - 添加规则\n"
|
help_add_suffix: " <类型> <数据> <存储名> <路径> - 添加规则\n"
|
||||||
help_del_suffix: " <规则ID> - 删除规则\n"
|
help_del_suffix: " <规则ID> - 删除规则\n"
|
||||||
|
help_preset_suffix: " <存储名> [基础路径] - 导入内置文件类型分类规则(视频/图片/音频/文档/压缩包)\n"
|
||||||
help_existing_rules_prefix: "\n当前已添加的规则:\n"
|
help_existing_rules_prefix: "\n当前已添加的规则:\n"
|
||||||
|
prompt_provide_storage_name: "请提供存储名称"
|
||||||
|
error_storage_not_found: "未找到存储: {{.Storage}}"
|
||||||
|
info_preset_imported: "已导入 {{.Count}} 条内置分类规则到存储 {{.Storage}}"
|
||||||
dir:
|
dir:
|
||||||
error_get_user_dirs_failed: "获取用户文件夹失败"
|
error_get_user_dirs_failed: "获取用户文件夹失败"
|
||||||
error_get_user_failed: "获取用户失败"
|
error_get_user_failed: "获取用户失败"
|
||||||
@@ -234,9 +238,9 @@ bot:
|
|||||||
info_install_plugin_success: "插件安装成功: {{.Name}}"
|
info_install_plugin_success: "插件安装成功: {{.Name}}"
|
||||||
parse:
|
parse:
|
||||||
info_parsing: "正在解析..."
|
info_parsing: "正在解析..."
|
||||||
error_parse_text_failed: "Failed to parse text: {{.Error}}"
|
error_parse_text_failed: "解析文本失败: {{.Error}}"
|
||||||
error_build_storage_select_keyboard_failed: "Failed to build storage selection keyboard: {{.Error}}"
|
error_build_storage_select_keyboard_failed: "构建存储选择键盘失败: {{.Error}}"
|
||||||
error_build_parsed_text_entity_failed: "Failed to build parsed text entity: {{.Error}}"
|
error_build_parsed_text_entity_failed: "构建解析文本实体失败: {{.Error}}"
|
||||||
info_link_prefix: "\n链接: "
|
info_link_prefix: "\n链接: "
|
||||||
info_author_prefix: "\n作者: "
|
info_author_prefix: "\n作者: "
|
||||||
info_description_prefix: "\n描述: "
|
info_description_prefix: "\n描述: "
|
||||||
@@ -348,32 +352,56 @@ bot:
|
|||||||
info_filename_prefix: "文件名: "
|
info_filename_prefix: "文件名: "
|
||||||
info_prompt_select_storage: "\n请选择存储位置"
|
info_prompt_select_storage: "\n请选择存储位置"
|
||||||
progress:
|
progress:
|
||||||
batch_start_prefix: "开始执行批量下载任务\n总大小: "
|
batch_status_header: "<b>📦 正在处理</b>\n\n文件:<code>{{.Total}}</code>\n状态:✅ <code>{{.Completed}}</code> | 📥 <code>{{.Downloaded}}</code> | ⏳ <code>{{.Waiting}}</code>\n总速度:⬇️ <code>{{.DownloadSpeed}}</code> | ⬆️ <code>{{.UploadSpeed}}</code>"
|
||||||
batch_processing_prefix: "正在处理批量下载任务\n总大小: "
|
batch_item_downloading: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} 下载中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_downloading_unknown: "<blockquote><b>⬇️ {{.Index}}/{{.Total}} 下载中</b>\n<code>{{.Name}}</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / 未知</blockquote>"
|
||||||
|
batch_item_transferring: "<blockquote><b>↕️ {{.Index}}/{{.Total}} 传输中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_transferring_unknown: "<blockquote><b>↕️ {{.Index}}/{{.Total}} 传输中</b>\n<code>{{.Name}}</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / 未知</blockquote>"
|
||||||
|
batch_item_uploading: "<blockquote><b>⬆️ {{.Index}}/{{.Total}} 上传中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code></blockquote>"
|
||||||
|
batch_item_retrying: "<blockquote><b>🔁 {{.Index}}/{{.Total}} 上传重试</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n重试次数:<code>{{.Attempt}}/{{.Limit}}</code>\n失败前速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code>\n原因:<code>{{.Reason}}</code></blockquote>"
|
||||||
|
batch_item_confirming: "<blockquote><b>⏳ {{.Index}}/{{.Total}} 等待中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n<i>文件已上传,正在等待远端确认</i></blockquote>"
|
||||||
|
batch_summary_hidden_active: "🔄 另有 <code>{{.Count}}</code> 个文件正在处理"
|
||||||
|
batch_summary_confirming: "☁️ 已上传,等待整组发送:<code>{{.Count}}</code>"
|
||||||
|
batch_summary_failed: "❌ 失败:<code>{{.Count}}</code>"
|
||||||
|
batch_summary_skipped: "⏭️ 已跳过:<code>{{.Count}}</code>"
|
||||||
|
batch_done: "<b>✅ 处理完成</b>\n\n文件数: <code>{{.Count}}</code>\n总大小: <code>{{.Size}}</code>"
|
||||||
|
batch_done_with_skipped: "<b>⚠️ 处理完成</b>\n\n成功: <code>{{.Success}}</code>\n已跳过: <code>{{.Skipped}}</code>\n总大小: <code>{{.Size}}</code>"
|
||||||
|
batch_canceled: "<b>🚫 任务已取消</b>\n\n文件数: <code>{{.Total}}</code>\n已完成: <code>{{.Completed}}</code>\n未完成: <code>{{.Incomplete}}</code>\n已跳过: <code>{{.Skipped}}</code>"
|
||||||
|
batch_failed_item: "<b>❌ 处理失败</b>\n\n失败文件: <code>{{.Index}}. {{.Name}}</code>\n失败阶段: <code>{{.Stage}}</code>\n失败进度: <code>{{.Progress}}</code>\n失败前速度: <code>{{.Speed}}</code>\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n❌ 失败: <code>{{.Failed}}</code>\n⏹️ 未完成: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failed_group: "<b>❌ 批量上传失败</b>\n\n受影响文件: <code>{{.Affected}} 个</code>\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n❌ 批次失败: <code>{{.Failed}}</code>\n⏹️ 未完成: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failed_task: "<b>❌ 处理失败</b>\n\n原因: <code>{{.Reason}}</code>\n\n✅ 已完成: <code>{{.Completed}}</code>\n⏹️ 未完成: <code>{{.Incomplete}}</code>"
|
||||||
|
batch_failure_stage_download: "下载"
|
||||||
|
batch_failure_stage_cache: "本地缓存"
|
||||||
|
batch_failure_stage_upload: "上传"
|
||||||
|
batch_failure_stage_confirm: "云端确认"
|
||||||
|
batch_failure_stage_batch_upload: "批量上传"
|
||||||
|
batch_failure_stage_internal: "任务内部"
|
||||||
|
single_status_header: "<b>📦 正在处理</b>"
|
||||||
|
single_downloading: "<blockquote><b>⬇️ 下载中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至:<code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_downloading_unknown: "<blockquote><b>⬇️ 下载中</b>\n<code>{{.Name}}</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / 未知\n保存至:<code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_uploading: "<blockquote><b>⬆️ 上传中</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至:<code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_upload_retrying: "<blockquote><b>🔁 上传重试</b>\n<code>{{.Name}}</code>\n{{.Bar}} <code>{{.Progress}}%</code>\n尝试次数:<code>{{.Attempt}}</code>\n速度:<code>{{.Speed}}</code>\n大小:<code>{{.Current}}</code> / <code>{{.Size}}</code>\n保存至:<code>{{.Destination}}</code></blockquote>"
|
||||||
|
single_done: "<b>✅ 处理完成</b>\n\n文件名:<code>{{.Name}}</code>\n总大小:<code>{{.Size}}</code>\n保存至:<code>{{.Destination}}</code>"
|
||||||
|
single_canceled: "<b>🚫 任务已取消</b>\n\n文件名:<code>{{.Name}}</code>"
|
||||||
|
single_failed: "<b>❌ 处理失败</b>\n\n文件名:<code>{{.Name}}</code>\n原因:<code>{{.Reason}}</code>"
|
||||||
downloading_prefix: "正在下载\n总大小: "
|
downloading_prefix: "正在下载\n总大小: "
|
||||||
|
size_with_files: "{{.Size}} ({{.Count}} 个文件)"
|
||||||
|
size_with_resources: "{{.Size}} ({{.Count}} 个资源)"
|
||||||
processing_list_prefix: "\n正在处理:\n"
|
processing_list_prefix: "\n正在处理:\n"
|
||||||
processing_none: " - 无"
|
processing_none: " - 无"
|
||||||
avg_speed_prefix: "\n平均速度: "
|
avg_speed_prefix: "\n平均速度: "
|
||||||
current_progress_prefix: "\n当前进度: "
|
current_progress_prefix: "\n当前进度: "
|
||||||
task_canceled: "任务已取消"
|
|
||||||
task_canceled_with_id: "处理已取消: {{.TaskID}}"
|
task_canceled_with_id: "处理已取消: {{.TaskID}}"
|
||||||
task_failed_with_error: "处理失败: {{.Error}}"
|
task_failed_with_error: "处理失败: {{.Error}}"
|
||||||
batch_done_prefix: "处理完成\n文件数: "
|
|
||||||
direct_done_prefix: "处理完成, 文件数量: "
|
direct_done_prefix: "处理完成, 文件数量: "
|
||||||
parsed_start_prefix: "开始下载 {{.Site}} 的资源\n总大小: "
|
parsed_start_prefix: "开始下载 {{.Site}} 的资源\n总大小: "
|
||||||
parsed_done_prefix: "处理完成, 资源数量: "
|
parsed_done_prefix: "处理完成, 资源数量: "
|
||||||
telegraph_start_prefix: "开始下载Telegraph\n图片数量: "
|
telegraph_start_prefix: "开始下载Telegraph\n图片数量: "
|
||||||
telegraph_progress_prefix: "正在下载\n当前进度: "
|
telegraph_progress_prefix: "正在下载\n当前进度: "
|
||||||
telegraph_done_prefix: "处理完成\n图片数量: "
|
telegraph_done_prefix: "处理完成\n图片数量: "
|
||||||
file_start_prefix: "开始下载\n文件名: "
|
|
||||||
file_processing_prefix: "正在处理下载任务\n文件名: "
|
|
||||||
download_failed_prefix: "下载失败\n文件名: "
|
|
||||||
download_done_prefix: "下载完成\n文件名: "
|
|
||||||
file_size_prefix: "\n文件大小: "
|
|
||||||
save_path_prefix: "\n保存路径: "
|
save_path_prefix: "\n保存路径: "
|
||||||
total_size_prefix: "\n总大小: "
|
total_size_prefix: "\n总大小: "
|
||||||
direct_start: "开始下载, 总大小: {{.SizeMB}} MB ({{.Count}} 个文件)"
|
direct_start: "开始下载, 总大小: {{.SizeMB}} MB ({{.Count}} 个文件)"
|
||||||
file_name_prefix: "文件名: "
|
|
||||||
error_prefix: "\n错误: "
|
error_prefix: "\n错误: "
|
||||||
aria2_start: "等待 Aria2 下载完成 (GID: {{.GID}})..."
|
aria2_start: "等待 Aria2 下载完成 (GID: {{.GID}})..."
|
||||||
aria2_downloading: "Aria2 正在下载 (GID: {{.GID}})\n"
|
aria2_downloading: "Aria2 正在下载 (GID: {{.GID}})\n"
|
||||||
|
|||||||
42
common/utils/fsutil/file_test.go
Normal file
42
common/utils/fsutil/file_test.go
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
package fsutil_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCloseAndRemove(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
preClose bool
|
||||||
|
}{
|
||||||
|
{name: "open file"},
|
||||||
|
{name: "already closed file", preClose: true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
filePath := filepath.Join(t.TempDir(), "cache-file")
|
||||||
|
file, err := fsutil.CreateFile(filePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CreateFile() failed: %v", err)
|
||||||
|
}
|
||||||
|
if tt.preClose {
|
||||||
|
if err := file.Close(); err != nil {
|
||||||
|
t.Fatalf("Close() failed: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := file.CloseAndRemove(); err != nil {
|
||||||
|
t.Fatalf("CloseAndRemove() failed: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filePath); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatalf("cache file still exists after CloseAndRemove(): %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package fsutil
|
package fsutil
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"errors"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -41,10 +42,11 @@ func (f *File) Remove() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (f *File) CloseAndRemove() error {
|
func (f *File) CloseAndRemove() error {
|
||||||
if err := f.Close(); err != nil {
|
closeErr := f.Close()
|
||||||
return err
|
if errors.Is(closeErr, os.ErrClosed) {
|
||||||
|
closeErr = nil
|
||||||
}
|
}
|
||||||
return f.Remove()
|
return errors.Join(closeErr, f.Remove())
|
||||||
}
|
}
|
||||||
|
|
||||||
func CreateFile(fp string) (*File, error) {
|
func CreateFile(fp string) (*File, error) {
|
||||||
|
|||||||
27
common/utils/fsutil/unique.go
Normal file
27
common/utils/fsutil/unique.go
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
package fsutil
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"path"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/rs/xid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UniquePath returns a non-taken path under basePath: the name itself, then
|
||||||
|
// numbered variants, then a random suffix.
|
||||||
|
func UniquePath(basePath, name string, exists func(candidate string) bool, maxAttempts int) string {
|
||||||
|
candidate := path.Join(basePath, name)
|
||||||
|
if !exists(candidate) {
|
||||||
|
return candidate
|
||||||
|
}
|
||||||
|
ext := path.Ext(name)
|
||||||
|
stem := strings.TrimSuffix(name, ext)
|
||||||
|
for i := 1; i <= maxAttempts; i++ {
|
||||||
|
candidate = path.Join(basePath, fmt.Sprintf("%s_%d%s", stem, i, ext))
|
||||||
|
if !exists(candidate) {
|
||||||
|
return candidate
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return path.Join(basePath, fmt.Sprintf("%s_%s%s", stem, xid.New().String(), ext))
|
||||||
|
}
|
||||||
@@ -17,7 +17,11 @@ type ProgressReadSeeker struct {
|
|||||||
|
|
||||||
// Seek implements io.ReadSeeker.
|
// Seek implements io.ReadSeeker.
|
||||||
func (pr *ProgressReadSeeker) Seek(offset int64, whence int) (int64, error) {
|
func (pr *ProgressReadSeeker) Seek(offset int64, whence int) (int64, error) {
|
||||||
return pr.reader.Seek(offset, whence)
|
position, err := pr.reader.Seek(offset, whence)
|
||||||
|
if err == nil {
|
||||||
|
pr.read.Store(position)
|
||||||
|
}
|
||||||
|
return position, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewProgressReader creates a new ProgressReader
|
// NewProgressReader creates a new ProgressReader
|
||||||
@@ -54,7 +58,7 @@ func (pr *ProgressReadSeeker) Progress() float64 {
|
|||||||
return float64(pr.read.Load()) / float64(pr.total.Load())
|
return float64(pr.read.Load()) / float64(pr.total.Load())
|
||||||
}
|
}
|
||||||
|
|
||||||
// Read returns the number of bytes read so far
|
// BytesRead returns the current tracked reader position.
|
||||||
func (pr *ProgressReadSeeker) BytesRead() int64 {
|
func (pr *ProgressReadSeeker) BytesRead() int64 {
|
||||||
return pr.read.Load()
|
return pr.read.Load()
|
||||||
}
|
}
|
||||||
|
|||||||
50
common/utils/ioutil/progress_reader_test.go
Normal file
50
common/utils/ioutil/progress_reader_test.go
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
package ioutil
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"io"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestProgressReadSeekerTracksReads(t *testing.T) {
|
||||||
|
var gotRead, gotTotal int64
|
||||||
|
reader := NewProgressReader(bytes.NewReader([]byte("abcdef")), 6, func(read, total int64) {
|
||||||
|
gotRead = read
|
||||||
|
gotTotal = total
|
||||||
|
})
|
||||||
|
|
||||||
|
buffer := make([]byte, 4)
|
||||||
|
if _, err := io.ReadFull(reader, buffer); err != nil {
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if gotRead != 4 || gotTotal != 6 {
|
||||||
|
t.Fatalf("progress = %d/%d, want 4/6", gotRead, gotTotal)
|
||||||
|
}
|
||||||
|
if reader.BytesRead() != 4 {
|
||||||
|
t.Fatalf("BytesRead() = %d, want 4", reader.BytesRead())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProgressReadSeekerResetsPositionOnSeek(t *testing.T) {
|
||||||
|
reader := NewProgressReader(bytes.NewReader([]byte("abcdef")), 6, nil)
|
||||||
|
buffer := make([]byte, 4)
|
||||||
|
if _, err := io.ReadFull(reader, buffer); err != nil {
|
||||||
|
t.Fatalf("read failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
position, err := reader.Seek(0, io.SeekStart)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("seek failed: %v", err)
|
||||||
|
}
|
||||||
|
if position != 0 || reader.BytesRead() != 0 {
|
||||||
|
t.Fatalf("position after seek = %d (tracked %d), want 0", position, reader.BytesRead())
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := io.ReadFull(reader, buffer[:2]); err != nil {
|
||||||
|
t.Fatalf("read after seek failed: %v", err)
|
||||||
|
}
|
||||||
|
if reader.BytesRead() != 2 {
|
||||||
|
t.Fatalf("BytesRead() after seek and read = %d, want 2", reader.BytesRead())
|
||||||
|
}
|
||||||
|
}
|
||||||
50
common/utils/progressutil/progress.go
Normal file
50
common/utils/progressutil/progress.go
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
// Package progressutil provides shared progress-update throttling for task
|
||||||
|
// progress trackers.
|
||||||
|
package progressutil
|
||||||
|
|
||||||
|
// updatesLevels picks the percent step by file size.
|
||||||
|
var updatesLevels = []struct {
|
||||||
|
size int64 // file size threshold
|
||||||
|
stepPercent int // minimum percent step between updates
|
||||||
|
}{
|
||||||
|
{10 << 20, 100},
|
||||||
|
{50 << 20, 20},
|
||||||
|
{200 << 20, 10},
|
||||||
|
{500 << 20, 5},
|
||||||
|
}
|
||||||
|
|
||||||
|
// ShouldUpdate reports whether a byte-based progress update should be shown.
|
||||||
|
func ShouldUpdate(total, downloaded int64, lastUpdatePercent int) bool {
|
||||||
|
if total <= 0 || downloaded <= 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
percent := int((downloaded * 100) / total)
|
||||||
|
if percent <= lastUpdatePercent {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
step := updatesLevels[len(updatesLevels)-1].stepPercent
|
||||||
|
for _, lvl := range updatesLevels {
|
||||||
|
if total < lvl.size {
|
||||||
|
step = lvl.stepPercent
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return percent >= lastUpdatePercent+step
|
||||||
|
}
|
||||||
|
|
||||||
|
// ShouldUpdateCount reports whether a count-based progress update (e.g. files
|
||||||
|
// downloaded so far) should be shown: every 10 units, or when finished.
|
||||||
|
func ShouldUpdateCount(downloaded, total int64) bool {
|
||||||
|
if total <= 0 || downloaded <= 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
const step = int64(10)
|
||||||
|
if downloaded < step {
|
||||||
|
return downloaded == total
|
||||||
|
}
|
||||||
|
return downloaded%step == 0 || downloaded == total
|
||||||
|
}
|
||||||
35
common/utils/tgutil/html.go
Normal file
35
common/utils/tgutil/html.go
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
package tgutil
|
||||||
|
|
||||||
|
import (
|
||||||
|
stdhtml "html"
|
||||||
|
|
||||||
|
"github.com/gotd/td/telegram/message/entity"
|
||||||
|
messagehtml "github.com/gotd/td/telegram/message/html"
|
||||||
|
"github.com/gotd/td/telegram/message/styling"
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
)
|
||||||
|
|
||||||
|
// EscapeHTMLTemplateData returns a copy of data with string values escaped for
|
||||||
|
// interpolation into Telegram HTML templates.
|
||||||
|
func EscapeHTMLTemplateData(data map[string]any) map[string]any {
|
||||||
|
escaped := make(map[string]any, len(data))
|
||||||
|
for key, value := range data {
|
||||||
|
if text, ok := value.(string); ok {
|
||||||
|
escaped[key] = stdhtml.EscapeString(text)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
escaped[key] = value
|
||||||
|
}
|
||||||
|
return escaped
|
||||||
|
}
|
||||||
|
|
||||||
|
// RenderHTML renders Telegram-compatible HTML into plain text and message
|
||||||
|
// entities.
|
||||||
|
func RenderHTML(markup string) (string, []tg.MessageEntityClass, error) {
|
||||||
|
var builder entity.Builder
|
||||||
|
if err := styling.Perform(&builder, messagehtml.String(nil, markup)); err != nil {
|
||||||
|
return "", nil, err
|
||||||
|
}
|
||||||
|
text, entities := builder.Complete()
|
||||||
|
return text, entities, nil
|
||||||
|
}
|
||||||
54
common/utils/tgutil/html_test.go
Normal file
54
common/utils/tgutil/html_test.go
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
package tgutil
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestEscapeHTMLTemplateDataDoesNotMutateInput(t *testing.T) {
|
||||||
|
input := map[string]any{
|
||||||
|
"Text": `<b>A&B</b>`,
|
||||||
|
"Count": 2,
|
||||||
|
}
|
||||||
|
escaped := EscapeHTMLTemplateData(input)
|
||||||
|
|
||||||
|
if got, want := escaped["Text"], "<b>A&B</b>"; got != want {
|
||||||
|
t.Fatalf("escaped text = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
if got := input["Text"]; got != `<b>A&B</b>` {
|
||||||
|
t.Fatalf("input was mutated: %q", got)
|
||||||
|
}
|
||||||
|
if got := escaped["Count"]; got != 2 {
|
||||||
|
t.Fatalf("non-string value = %v, want 2", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderHTMLUsesTemplateStylesAndDecodesValues(t *testing.T) {
|
||||||
|
data := EscapeHTMLTemplateData(map[string]any{"Name": `<b>A&B</b>.bin`})
|
||||||
|
markup := `<blockquote><b>Uploading</b>
|
||||||
|
<code>` + data["Name"].(string) + `</code></blockquote>`
|
||||||
|
|
||||||
|
text, entities, err := RenderHTML(markup)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("RenderHTML() failed: %v", err)
|
||||||
|
}
|
||||||
|
if want := "Uploading\n<b>A&B</b>.bin"; text != want {
|
||||||
|
t.Fatalf("rendered text = %q, want %q", text, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
var bold, code, blockquote int
|
||||||
|
for _, messageEntity := range entities {
|
||||||
|
switch messageEntity.(type) {
|
||||||
|
case *tg.MessageEntityBold:
|
||||||
|
bold++
|
||||||
|
case *tg.MessageEntityCode:
|
||||||
|
code++
|
||||||
|
case *tg.MessageEntityBlockquote:
|
||||||
|
blockquote++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if bold != 1 || code != 1 || blockquote != 1 {
|
||||||
|
t.Fatalf("entity counts = bold:%d code:%d blockquote:%d", bold, code, blockquote)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ package tgutil
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"unicode"
|
"unicode"
|
||||||
@@ -193,97 +194,6 @@ func getMessagesRange(ctx *ext.Context, chatID int64, minId, maxId int) ([]*tg.M
|
|||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// [TODO]
|
|
||||||
// type MessageItem struct {
|
|
||||||
// Message *tg.Message
|
|
||||||
// Error error
|
|
||||||
// }
|
|
||||||
|
|
||||||
// func IterMessages(ctx *ext.Context, chatID int64, minId, maxId int) (<-chan MessageItem, error) {
|
|
||||||
// total := maxId - minId + 1
|
|
||||||
// ch := make(chan MessageItem, 100)
|
|
||||||
|
|
||||||
// go func() {
|
|
||||||
// defer close(ch)
|
|
||||||
// if !ctx.Self.Bot {
|
|
||||||
// perr := ctx.PeerStorage.GetInputPeerById(chatID)
|
|
||||||
// if perr == nil || perr.(*tg.InputPeerEmpty) != nil {
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Error: fmt.Errorf("peer not found: %d", chatID),
|
|
||||||
// }
|
|
||||||
// return
|
|
||||||
// }
|
|
||||||
|
|
||||||
// for i := 0; i < total; i += 100 {
|
|
||||||
// start := minId + i
|
|
||||||
// end := min(start+100, maxId)
|
|
||||||
// msgs, err := ctx.Raw.MessagesGetHistory(ctx, &tg.MessagesGetHistoryRequest{
|
|
||||||
// Peer: perr,
|
|
||||||
// OffsetID: start,
|
|
||||||
// AddOffset: start - end,
|
|
||||||
// Limit: 100,
|
|
||||||
// })
|
|
||||||
// if err != nil {
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Error: fmt.Errorf("failed to get messages: %w", err),
|
|
||||||
// }
|
|
||||||
// return
|
|
||||||
// }
|
|
||||||
// var msgClass []tg.MessageClass
|
|
||||||
// switch msgsv := msgs.(type) {
|
|
||||||
// case *tg.MessagesMessages:
|
|
||||||
// msgClass = msgsv.GetMessages()
|
|
||||||
// case *tg.MessagesMessagesSlice:
|
|
||||||
// msgClass = msgsv.GetMessages()
|
|
||||||
// case *tg.MessagesChannelMessages:
|
|
||||||
// msgClass = msgsv.GetMessages()
|
|
||||||
// default:
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Error: fmt.Errorf("unsupported message type: %T", msgsv),
|
|
||||||
// }
|
|
||||||
// continue
|
|
||||||
// }
|
|
||||||
// for _, msg := range msgClass {
|
|
||||||
// msg, ok := msg.AsNotEmpty()
|
|
||||||
// if !ok {
|
|
||||||
// continue
|
|
||||||
// }
|
|
||||||
// switch msg := msg.(type) {
|
|
||||||
// case *tg.Message:
|
|
||||||
// key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msg.GetID())
|
|
||||||
// cache.Set(key, msg)
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Message: msg,
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// } else {
|
|
||||||
// for i := 0; i < total; i += 100 {
|
|
||||||
// start := minId + i
|
|
||||||
// end := min(start+100, maxId)
|
|
||||||
// msgs, err := GetMessagesRange(ctx, chatID, start, end)
|
|
||||||
// if err != nil {
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Error: fmt.Errorf("failed to get messages: %w", err),
|
|
||||||
// }
|
|
||||||
// return
|
|
||||||
// }
|
|
||||||
// for _, msg := range msgs {
|
|
||||||
// if msg == nil {
|
|
||||||
// continue
|
|
||||||
// }
|
|
||||||
// ch <- MessageItem{
|
|
||||||
// Message: msg,
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
// }()
|
|
||||||
|
|
||||||
// return ch, nil
|
|
||||||
// }
|
|
||||||
|
|
||||||
func getMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, error) {
|
func getMessageByID(ctx *ext.Context, chatID int64, msgID int) (*tg.Message, error) {
|
||||||
key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msgID)
|
key := fmt.Sprintf("tgmsg:%d:%d:%d", ctx.Self.ID, chatID, msgID)
|
||||||
if msg, ok := cache.Get[*tg.Message](key); ok {
|
if msg, ok := cache.Get[*tg.Message](key); ok {
|
||||||
@@ -359,9 +269,16 @@ func GetGroupedMessages(ctx *ext.Context, chatID int64, msg *tg.Message) ([]*tg.
|
|||||||
groupedMessages = append(groupedMessages, m)
|
groupedMessages = append(groupedMessages, m)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
sortMessagesByID(groupedMessages)
|
||||||
return groupedMessages, nil
|
return groupedMessages, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sortMessagesByID(messages []*tg.Message) {
|
||||||
|
sort.Slice(messages, func(i, j int) bool {
|
||||||
|
return messages[i].GetID() < messages[j].GetID()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func ExtractMessageEntityUrls(msg *tg.Message) []string {
|
func ExtractMessageEntityUrls(msg *tg.Message) []string {
|
||||||
if len(msg.Entities) == 0 {
|
if len(msg.Entities) == 0 {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
18
common/utils/tgutil/message_test.go
Normal file
18
common/utils/tgutil/message_test.go
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
package tgutil
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSortMessagesByID(t *testing.T) {
|
||||||
|
messages := []*tg.Message{{ID: 9}, {ID: 3}, {ID: 7}}
|
||||||
|
sortMessagesByID(messages)
|
||||||
|
want := []int{3, 7, 9}
|
||||||
|
for i := range messages {
|
||||||
|
if messages[i].GetID() != want[i] {
|
||||||
|
t.Fatalf("message %d has ID %d, want %d", i, messages[i].GetID(), want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -33,6 +33,36 @@ secret = ""
|
|||||||
# 转存完成后删除 Aria2 下载的本地文件
|
# 转存完成后删除 Aria2 下载的本地文件
|
||||||
remove_after_transfer = true
|
remove_after_transfer = true
|
||||||
|
|
||||||
|
# yt-dlp 视频下载配置
|
||||||
|
[ytdlp]
|
||||||
|
# 默认下载的最高视频清晰度 (按高度限制), 如 1080, 720, 480; 0 表示不限制 (下载最佳画质)
|
||||||
|
# 仅在使用 /ytdlp 命令且未手动指定任何参数时生效
|
||||||
|
max_height = 1080
|
||||||
|
# 直接指定 yt-dlp format 选择表达式, 留空则使用 max_height
|
||||||
|
# 设置后优先级高于 max_height, 例如: "bv*[height<=720]+ba/b"
|
||||||
|
format = ""
|
||||||
|
# 下载后转封装的视频容器格式, 留空则不转封装. 默认 mp4
|
||||||
|
recode = "mp4"
|
||||||
|
|
||||||
|
# 解析器配置
|
||||||
|
[parser]
|
||||||
|
# 启用 JS 解析器插件 (Go 内置解析器默认启用)
|
||||||
|
plugin_enable = false
|
||||||
|
# 插件目录, 可以是多个目录
|
||||||
|
plugin_dirs = ["./plugins"]
|
||||||
|
# 解析器默认代理
|
||||||
|
proxy = ""
|
||||||
|
|
||||||
|
# Twitter/X 解析器配置
|
||||||
|
[parser.twitter]
|
||||||
|
# 自定义 API 域名
|
||||||
|
api_domain = "api.fxtwitter.com"
|
||||||
|
# 单独为此解析器指定代理 (留空则使用 [parser] 中的 proxy)
|
||||||
|
# proxy = "http://127.0.0.1:7890"
|
||||||
|
|
||||||
|
# Kemono 解析器配置 (暂无可配置项, 留空即可)
|
||||||
|
[parser.kemono]
|
||||||
|
|
||||||
# HTTP API 配置
|
# HTTP API 配置
|
||||||
[api]
|
[api]
|
||||||
# 启用 HTTP API
|
# 启用 HTTP API
|
||||||
|
|||||||
@@ -10,13 +10,4 @@ type hookExecConfig struct {
|
|||||||
TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
|
TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
|
||||||
TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
|
TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
|
||||||
TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
|
TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
|
||||||
|
|
||||||
// TaskTypes map[string]hookExecOnTypeConfig `toml:"task_types" mapstructure:"task_types" json:"task_types"` // [TODO]
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// type hookExecOnTypeConfig struct {
|
|
||||||
// TaskBeforeStart string `toml:"task_before_start" mapstructure:"task_before_start" json:"task_before_start"`
|
|
||||||
// TaskSuccess string `toml:"task_success" mapstructure:"task_success" json:"task_success"`
|
|
||||||
// TaskFail string `toml:"task_fail" mapstructure:"task_fail" json:"task_fail"`
|
|
||||||
// TaskCancel string `toml:"task_cancel" mapstructure:"task_cancel" json:"task_cancel"`
|
|
||||||
// }
|
|
||||||
|
|||||||
@@ -8,14 +8,16 @@ import (
|
|||||||
|
|
||||||
type TelegramStorageConfig struct {
|
type TelegramStorageConfig struct {
|
||||||
BaseConfig
|
BaseConfig
|
||||||
ChatID int64 `toml:"chat_id" mapstructure:"chat_id" json:"chat_id"`
|
ChatID int64 `toml:"chat_id" mapstructure:"chat_id" json:"chat_id"`
|
||||||
ForceFile bool `toml:"force_file" mapstructure:"force_file" json:"force_file"`
|
ForceFile bool `toml:"force_file" mapstructure:"force_file" json:"force_file"`
|
||||||
RateLimit int `toml:"rate_limit" mapstructure:"rate_limit" json:"rate_limit"`
|
RateLimit int `toml:"rate_limit" mapstructure:"rate_limit" json:"rate_limit"`
|
||||||
RateBurst int `toml:"rate_burst" mapstructure:"rate_burst" json:"rate_burst"`
|
RateBurst int `toml:"rate_burst" mapstructure:"rate_burst" json:"rate_burst"`
|
||||||
SkipLarge bool `toml:"skip_large" mapstructure:"skip_large" json:"skip_large"` // skip files larger than Telegram limit(2GB)
|
SkipLarge bool `toml:"skip_large" mapstructure:"skip_large" json:"skip_large"` // skip files larger than Telegram limit(2GB)
|
||||||
// split files larger than Telegram limit(2GB) into parts of specified size, in MB, leave 0 to set default(2000MB)
|
SplitLargeVideo bool `toml:"split_large_video" mapstructure:"split_large_video" json:"split_large_video"`
|
||||||
|
// split files larger than the uploader account limit into parts of specified size, in MB
|
||||||
|
// leave 0 to use the account limit (2000MB for bots/regular users, 4000MB for Premium users)
|
||||||
// only effective when SkipLarge is false
|
// only effective when SkipLarge is false
|
||||||
// use zip when splitting
|
// use zip when splitting non-video files or when lossless video splitting is disabled/unavailable
|
||||||
SplitSizeMB int64 `toml:"split_size_mb" mapstructure:"split_size_mb" json:"split_size_mb"`
|
SplitSizeMB int64 `toml:"split_size_mb" mapstructure:"split_size_mb" json:"split_size_mb"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/charmbracelet/log"
|
||||||
"github.com/duke-git/lancet/v2/slice"
|
"github.com/duke-git/lancet/v2/slice"
|
||||||
"github.com/krau/SaveAny-Bot/config/storage"
|
"github.com/krau/SaveAny-Bot/config/storage"
|
||||||
"github.com/spf13/viper"
|
"github.com/spf13/viper"
|
||||||
@@ -35,6 +36,7 @@ type Config struct {
|
|||||||
Storages []storage.StorageConfig `toml:"-" mapstructure:"-" json:"storages"`
|
Storages []storage.StorageConfig `toml:"-" mapstructure:"-" json:"storages"`
|
||||||
Parser parserConfig `toml:"parser" mapstructure:"parser" json:"parser"`
|
Parser parserConfig `toml:"parser" mapstructure:"parser" json:"parser"`
|
||||||
Hook hookConfig `toml:"hook" mapstructure:"hook" json:"hook"`
|
Hook hookConfig `toml:"hook" mapstructure:"hook" json:"hook"`
|
||||||
|
Ytdlp YtdlpConfig `toml:"ytdlp" mapstructure:"ytdlp" json:"ytdlp"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type aria2Config struct {
|
type aria2Config struct {
|
||||||
@@ -67,6 +69,13 @@ func (c Config) GetStorageByName(name string) storage.StorageConfig {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func Init(ctx context.Context, configFile ...string) error {
|
func Init(ctx context.Context, configFile ...string) error {
|
||||||
|
logger := log.FromContext(ctx)
|
||||||
|
|
||||||
|
// Reset side tables for re-init.
|
||||||
|
storages = nil
|
||||||
|
userIDs = nil
|
||||||
|
userStorages = make(map[int64][]string)
|
||||||
|
|
||||||
viper.SetConfigType("toml")
|
viper.SetConfigType("toml")
|
||||||
viper.SetEnvPrefix("SAVEANY")
|
viper.SetEnvPrefix("SAVEANY")
|
||||||
viper.AutomaticEnv()
|
viper.AutomaticEnv()
|
||||||
@@ -75,11 +84,13 @@ func Init(ctx context.Context, configFile ...string) error {
|
|||||||
|
|
||||||
// 如果指定了配置文件路径,则使用指定的配置文件
|
// 如果指定了配置文件路径,则使用指定的配置文件
|
||||||
// 配置文件支持传入一个 http(s) URL 地址
|
// 配置文件支持传入一个 http(s) URL 地址
|
||||||
|
loadedFromURL := false
|
||||||
if len(configFile) > 0 && configFile[0] != "" {
|
if len(configFile) > 0 && configFile[0] != "" {
|
||||||
cfg := configFile[0]
|
cfg := configFile[0]
|
||||||
if strings.HasPrefix(cfg, "http://") || strings.HasPrefix(cfg, "https://") {
|
if strings.HasPrefix(cfg, "http://") || strings.HasPrefix(cfg, "https://") {
|
||||||
// 使用远程配置文件
|
// 使用远程配置文件
|
||||||
resp, err := http.Get(cfg)
|
client := &http.Client{Timeout: 30 * time.Second}
|
||||||
|
resp, err := client.Get(cfg)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to fetch remote config file: %w", err)
|
return fmt.Errorf("failed to fetch remote config file: %w", err)
|
||||||
}
|
}
|
||||||
@@ -90,6 +101,7 @@ func Init(ctx context.Context, configFile ...string) error {
|
|||||||
if err := viper.ReadConfig(resp.Body); err != nil {
|
if err := viper.ReadConfig(resp.Body); err != nil {
|
||||||
return fmt.Errorf("failed to read remote config file: %w", err)
|
return fmt.Errorf("failed to read remote config file: %w", err)
|
||||||
}
|
}
|
||||||
|
loadedFromURL = true
|
||||||
} else {
|
} else {
|
||||||
viper.SetConfigFile(cfg)
|
viper.SetConfigFile(cfg)
|
||||||
}
|
}
|
||||||
@@ -131,19 +143,24 @@ func Init(ctx context.Context, configFile ...string) error {
|
|||||||
"api.host": "0.0.0.0",
|
"api.host": "0.0.0.0",
|
||||||
"api.port": 8080,
|
"api.port": 8080,
|
||||||
"api.token": "",
|
"api.token": "",
|
||||||
|
|
||||||
|
// yt-dlp
|
||||||
|
"ytdlp.recode": "mp4",
|
||||||
}
|
}
|
||||||
|
|
||||||
for key, value := range defaultConfigs {
|
for key, value := range defaultConfigs {
|
||||||
viper.SetDefault(key, value)
|
viper.SetDefault(key, value)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := viper.ReadInConfig(); err != nil {
|
if !loadedFromURL {
|
||||||
fmt.Println("Error reading config file, ", err)
|
if err := viper.ReadInConfig(); err != nil {
|
||||||
return err
|
logger.Errorf("Error reading config file: %v", err)
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := viper.Unmarshal(cfg); err != nil {
|
if err := viper.Unmarshal(cfg); err != nil {
|
||||||
fmt.Println("Error unmarshalling config file, ", err)
|
logger.Errorf("Error unmarshalling config file: %v", err)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
13
config/ytdlp.go
Normal file
13
config/ytdlp.go
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
type YtdlpConfig struct {
|
||||||
|
// MaxHeight limits the video resolution by height in pixels (e.g. 1080, 720).
|
||||||
|
// 0 means no limit (best available). Ignored when Format is set.
|
||||||
|
MaxHeight int `toml:"max_height" mapstructure:"max_height" json:"max_height"`
|
||||||
|
// Format is a raw yt-dlp format selector (-f). When set, it takes precedence
|
||||||
|
// over MaxHeight and gives the user full control.
|
||||||
|
Format string `toml:"format" mapstructure:"format" json:"format"`
|
||||||
|
// Recode is the target video container yt-dlp recodes into (e.g. mp4).
|
||||||
|
// Empty disables recoding.
|
||||||
|
Recode string `toml:"recode" mapstructure:"recode" json:"recode"`
|
||||||
|
}
|
||||||
38
core/core.go
38
core/core.go
@@ -3,14 +3,27 @@ package core
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"sync"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/queue"
|
"github.com/krau/SaveAny-Bot/pkg/queue"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
var queueInstance *queue.TaskQueue[Executable]
|
var (
|
||||||
|
queueOnce sync.Once
|
||||||
|
queueInstance *queue.TaskQueue[Executable]
|
||||||
|
)
|
||||||
|
|
||||||
|
// initQueue lazily creates the shared task queue.
|
||||||
|
func initQueue() *queue.TaskQueue[Executable] {
|
||||||
|
queueOnce.Do(func() {
|
||||||
|
queueInstance = queue.NewTaskQueue[Executable]()
|
||||||
|
})
|
||||||
|
return queueInstance
|
||||||
|
}
|
||||||
|
|
||||||
type Executable interface {
|
type Executable interface {
|
||||||
Type() tasktype.TaskType
|
Type() tasktype.TaskType
|
||||||
@@ -30,11 +43,14 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
|
|||||||
break // queue closed and empty
|
break // queue closed and empty
|
||||||
}
|
}
|
||||||
exe := qtask.Data
|
exe := qtask.Data
|
||||||
|
taskCtx := qtask.Context()
|
||||||
logger.Infof("Processing task: %s", exe.TaskID())
|
logger.Infof("Processing task: %s", exe.TaskID())
|
||||||
if err := ExecCommandString(qtask.Context(), execHooks.TaskBeforeStart); err != nil {
|
taskevent.Emit(taskCtx, taskevent.Event{TaskID: exe.TaskID(), Phase: taskevent.PhaseStart})
|
||||||
|
if err := ExecCommandString(taskCtx, execHooks.TaskBeforeStart); err != nil {
|
||||||
logger.Errorf("Failed to execute before start hook for task %s: %v", exe.TaskID(), err)
|
logger.Errorf("Failed to execute before start hook for task %s: %v", exe.TaskID(), err)
|
||||||
}
|
}
|
||||||
if err := exe.Execute(qtask.Context()); err != nil {
|
err = exe.Execute(taskCtx)
|
||||||
|
if err != nil {
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
logger.Infof("Task %s was canceled", exe.TaskID())
|
logger.Infof("Task %s was canceled", exe.TaskID())
|
||||||
if err := ExecCommandString(ctx, execHooks.TaskCancel); err != nil {
|
if err := ExecCommandString(ctx, execHooks.TaskCancel); err != nil {
|
||||||
@@ -52,6 +68,7 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
|
|||||||
logger.Errorf("Failed to execute success hook for task %s: %v", exe.TaskID(), err)
|
logger.Errorf("Failed to execute success hook for task %s: %v", exe.TaskID(), err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(taskCtx, taskevent.Event{TaskID: exe.TaskID(), Phase: taskevent.PhaseDone, Err: err})
|
||||||
qe.Done(qtask.ID)
|
qe.Done(qtask.ID)
|
||||||
<-semaphore
|
<-semaphore
|
||||||
}
|
}
|
||||||
@@ -60,17 +77,22 @@ func worker(ctx context.Context, qe *queue.TaskQueue[Executable], semaphore chan
|
|||||||
func Run(ctx context.Context) {
|
func Run(ctx context.Context) {
|
||||||
log.FromContext(ctx).Info("Start processing tasks...")
|
log.FromContext(ctx).Info("Start processing tasks...")
|
||||||
semaphore := make(chan struct{}, config.C().Workers)
|
semaphore := make(chan struct{}, config.C().Workers)
|
||||||
if queueInstance == nil {
|
q := initQueue()
|
||||||
queueInstance = queue.NewTaskQueue[Executable]()
|
|
||||||
}
|
|
||||||
for range config.C().Workers {
|
for range config.C().Workers {
|
||||||
go worker(ctx, queueInstance, semaphore)
|
go worker(ctx, q, semaphore)
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Close stops the queue and unblocks workers in Get.
|
||||||
|
func Close() {
|
||||||
|
if q := initQueue(); q != nil {
|
||||||
|
q.Close()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func AddTask(ctx context.Context, task Executable) error {
|
func AddTask(ctx context.Context, task Executable) error {
|
||||||
return queueInstance.Add(queue.NewTask(ctx, task.TaskID(), task.Title(), task))
|
return initQueue().Add(queue.NewTask(ctx, task.TaskID(), task.Title(), task))
|
||||||
}
|
}
|
||||||
|
|
||||||
func CancelTask(ctx context.Context, id string) error {
|
func CancelTask(ctx context.Context, id string) error {
|
||||||
|
|||||||
@@ -6,12 +6,14 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/aria2"
|
"github.com/krau/SaveAny-Bot/pkg/aria2"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Execute implements core.Executable.
|
// Execute implements core.Executable.
|
||||||
@@ -77,6 +79,12 @@ func (t *Task) waitForDownload(ctx context.Context) error {
|
|||||||
if t.Progress != nil {
|
if t.Progress != nil {
|
||||||
t.Progress.OnProgress(ctx, t, status)
|
t.Progress.OnProgress(ctx, t, status)
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: parseInt64(status.TotalLength),
|
||||||
|
DownloadedBytes: parseInt64(status.CompletedLength),
|
||||||
|
})
|
||||||
|
|
||||||
// Check if download is complete
|
// Check if download is complete
|
||||||
if status.IsDownloadComplete() {
|
if status.IsDownloadComplete() {
|
||||||
@@ -248,3 +256,16 @@ func (t *Task) cancelAria2Download() {
|
|||||||
logger.Debugf("Failed to remove download result for %s: %v", t.GID, err)
|
logger.Debugf("Failed to remove download result for %s: %v", t.GID, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseInt64 parses an aria2 status string (decimal bytes) into int64,
|
||||||
|
// returning 0 on failure so it can be used directly in progress events.
|
||||||
|
func parseInt64(s string) int64 {
|
||||||
|
if s == "" {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
n, err := strconv.ParseInt(s, 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,10 +2,13 @@ package batchtfile
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/duke-git/lancet/v2/retry"
|
"github.com/duke-git/lancet/v2/retry"
|
||||||
@@ -14,44 +17,318 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
|
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type executionGroup struct {
|
||||||
|
elems []*TaskElement
|
||||||
|
batchSaver storage.StorageBatchSaver
|
||||||
|
}
|
||||||
|
|
||||||
|
func (g executionGroup) usesBatchSaver() bool {
|
||||||
|
return g.batchSaver != nil
|
||||||
|
}
|
||||||
|
|
||||||
func (t *Task) Execute(ctx context.Context) error {
|
func (t *Task) Execute(ctx context.Context) error {
|
||||||
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("batch_file[%s]", t.ID))
|
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("batch_file[%s]", t.ID))
|
||||||
logger.Info("Starting batch file task")
|
logger.Info("Starting batch file task")
|
||||||
t.Progress.OnStart(ctx, t)
|
if t.Progress != nil {
|
||||||
workers := config.C().Workers
|
t.Progress.OnStart(ctx, t)
|
||||||
eg, gctx := errgroup.WithContext(ctx)
|
}
|
||||||
eg.SetLimit(workers)
|
groups := t.executionGroups()
|
||||||
for _, elem := range t.elems {
|
var err error
|
||||||
eg.Go(func() error {
|
for i := 0; i < len(groups); {
|
||||||
t.processingMu.RLock()
|
if groups[i].usesBatchSaver() {
|
||||||
if t.processing[elem.ID] != nil {
|
err = t.processBatch(ctx, groups[i])
|
||||||
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
|
i++
|
||||||
}
|
} else {
|
||||||
t.processingMu.RUnlock()
|
end := i + 1
|
||||||
t.processingMu.Lock()
|
for end < len(groups) && !groups[end].usesBatchSaver() {
|
||||||
t.processing[elem.ID] = &elem
|
end++
|
||||||
t.processingMu.Unlock()
|
}
|
||||||
defer func() {
|
elems := make([]*TaskElement, 0, end-i)
|
||||||
t.processingMu.Lock()
|
for _, group := range groups[i:end] {
|
||||||
delete(t.processing, elem.ID)
|
elems = append(elems, group.elems...)
|
||||||
t.processingMu.Unlock()
|
}
|
||||||
}()
|
err = t.processElements(ctx, elems)
|
||||||
return t.processElement(gctx, elem)
|
i = end
|
||||||
})
|
}
|
||||||
|
if err != nil {
|
||||||
|
if !t.IgnoreErrors || errors.Is(err, context.Canceled) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
logger.Warnf("Group processing failed (ignored): %v", err)
|
||||||
|
err = nil
|
||||||
|
}
|
||||||
}
|
}
|
||||||
err := eg.Wait()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logger.Errorf("Error during batch file processing: %v", err)
|
logger.Errorf("Error during batch file processing: %v", err)
|
||||||
} else {
|
} else {
|
||||||
logger.Info("Batch file task completed successfully")
|
logger.Info("Batch file task completed successfully")
|
||||||
}
|
}
|
||||||
t.Progress.OnDone(ctx, t, err)
|
t.finishItems(err)
|
||||||
|
if t.Progress != nil {
|
||||||
|
t.Progress.OnDone(ctx, t, err)
|
||||||
|
}
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// notifyProgress reports a progress update to the optional tracker.
|
||||||
|
func (t *Task) notifyProgress(ctx context.Context) {
|
||||||
|
if t.Progress != nil {
|
||||||
|
t.Progress.OnProgress(ctx, t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) executionGroups() []executionGroup {
|
||||||
|
groups := make([]executionGroup, 0, len(t.elems))
|
||||||
|
for i := 0; i < len(t.elems); {
|
||||||
|
elem := &t.elems[i]
|
||||||
|
batchSaver, batchCapable := elem.Storage.(storage.StorageBatchSaver)
|
||||||
|
if !batchCapable || elem.sourceGroupKey == "" {
|
||||||
|
groups = append(groups, executionGroup{elems: []*TaskElement{elem}})
|
||||||
|
i++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
end := i + 1
|
||||||
|
for end < len(t.elems) {
|
||||||
|
next := &t.elems[end]
|
||||||
|
if next.Storage != elem.Storage || next.sourceGroupKey != elem.sourceGroupKey {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
end++
|
||||||
|
}
|
||||||
|
elems := make([]*TaskElement, 0, end-i)
|
||||||
|
for j := i; j < end; j++ {
|
||||||
|
elems = append(elems, &t.elems[j])
|
||||||
|
}
|
||||||
|
groups = append(groups, executionGroup{elems: elems, batchSaver: batchSaver})
|
||||||
|
i = end
|
||||||
|
}
|
||||||
|
return groups
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) processElements(ctx context.Context, elems []*TaskElement) error {
|
||||||
|
eg, gctx := errgroup.WithContext(ctx)
|
||||||
|
eg.SetLimit(config.C().Workers)
|
||||||
|
for _, elem := range elems {
|
||||||
|
eg.Go(func() error {
|
||||||
|
if err := t.markProcessing(ctx, elem); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer t.unmarkProcessing(elem.ID)
|
||||||
|
err := t.processElement(gctx, *elem)
|
||||||
|
if err != nil && t.IgnoreErrors && !errors.Is(err, context.Canceled) {
|
||||||
|
// Per-item failure: keep siblings running.
|
||||||
|
log.FromContext(ctx).Warnf("Element %s failed (ignored): %v", elem.ID, err)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return eg.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) processBatch(ctx context.Context, group executionGroup) error {
|
||||||
|
defer func() {
|
||||||
|
for _, elem := range group.elems {
|
||||||
|
if err := os.Remove(elem.localPath); err != nil && !os.IsNotExist(err) {
|
||||||
|
log.FromContext(ctx).Warnf("Failed to cleanup batch cache file %s: %v", elem.localPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
type downloadResult struct {
|
||||||
|
elem *TaskElement
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
results := make([]downloadResult, len(group.elems))
|
||||||
|
var resultsMu sync.Mutex
|
||||||
|
|
||||||
|
eg, gctx := errgroup.WithContext(ctx)
|
||||||
|
eg.SetLimit(config.C().Workers)
|
||||||
|
for i, elem := range group.elems {
|
||||||
|
eg.Go(func() error {
|
||||||
|
if err := t.markProcessing(ctx, elem); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer t.unmarkProcessing(elem.ID)
|
||||||
|
err := t.downloadElement(gctx, elem)
|
||||||
|
// Store by original index.
|
||||||
|
resultsMu.Lock()
|
||||||
|
results[i] = downloadResult{elem: elem, err: err}
|
||||||
|
resultsMu.Unlock()
|
||||||
|
if err != nil && t.IgnoreErrors && !errors.Is(err, context.Canceled) {
|
||||||
|
// Per-item failure: keep siblings running.
|
||||||
|
log.FromContext(ctx).Warnf("Element %s failed (ignored): %v", elem.ID, err)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if err := eg.Wait(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upload only successfully downloaded elements.
|
||||||
|
successElems := make([]*TaskElement, 0, len(group.elems))
|
||||||
|
for _, r := range results {
|
||||||
|
if r.err == nil {
|
||||||
|
successElems = append(successElems, r.elem)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(successElems) == 0 {
|
||||||
|
return fmt.Errorf("all elements failed to download")
|
||||||
|
}
|
||||||
|
|
||||||
|
items := make([]storagetypes.BatchItem, 0, len(successElems))
|
||||||
|
openFiles := make([]*os.File, 0, len(successElems))
|
||||||
|
defer func() {
|
||||||
|
for _, file := range openFiles {
|
||||||
|
if err := file.Close(); err != nil {
|
||||||
|
log.FromContext(ctx).Warnf("Failed to close batch cache file %s: %v", file.Name(), err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
for _, elem := range successElems {
|
||||||
|
file, err := os.Open(elem.localPath)
|
||||||
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, err)
|
||||||
|
return fmt.Errorf("failed to open cache file: %w", err)
|
||||||
|
}
|
||||||
|
stat, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
file.Close()
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, err)
|
||||||
|
return fmt.Errorf("failed to get cache file stat: %w", err)
|
||||||
|
}
|
||||||
|
openFiles = append(openFiles, file)
|
||||||
|
items = append(items, storagetypes.BatchItem{
|
||||||
|
Reader: file,
|
||||||
|
StoragePath: elem.Path,
|
||||||
|
Size: stat.Size(),
|
||||||
|
SourceGroupKey: elem.sourceGroupKey,
|
||||||
|
Caption: elem.sourceCaption,
|
||||||
|
PreserveCaption: elem.preserveCaption,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for index, item := range items {
|
||||||
|
t.recordDownloadComplete(successElems[index].ID, item.Size)
|
||||||
|
}
|
||||||
|
return t.saveBatchItems(ctx, successElems, items)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) saveBatchItems(ctx context.Context, successElems []*TaskElement, items []storagetypes.BatchItem) error {
|
||||||
|
t.startUpload(ctx)
|
||||||
|
if progressSaver, ok := successElems[0].Storage.(storage.StorageBatchProgressSaver); ok {
|
||||||
|
err := progressSaver.SaveBatchWithProgress(ctx, items, func(index int, uploaded, total int64) {
|
||||||
|
if index < 0 || index >= len(successElems) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.uploadCallback(ctx, successElems[index].ID)(uploaded, total)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
for _, elem := range successElems {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageBatchUpload, err)
|
||||||
|
}
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return fmt.Errorf("failed to save batch: %w", err)
|
||||||
|
}
|
||||||
|
for index, elem := range successElems {
|
||||||
|
t.uploadCallback(ctx, elem.ID)(items[index].Size, items[index].Size)
|
||||||
|
t.markItemCompleted(elem.ID)
|
||||||
|
}
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for i := range items {
|
||||||
|
items[i].Reader = ioutil.NewProgressReader(
|
||||||
|
items[i].Reader,
|
||||||
|
items[i].Size,
|
||||||
|
t.uploadCallback(ctx, successElems[i].ID),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if err := successElems[0].Storage.(storage.StorageBatchSaver).SaveBatch(ctx, items); err != nil {
|
||||||
|
for _, elem := range successElems {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageBatchUpload, err)
|
||||||
|
}
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return fmt.Errorf("failed to save batch: %w", err)
|
||||||
|
}
|
||||||
|
for index, elem := range successElems {
|
||||||
|
t.uploadCallback(ctx, elem.ID)(items[index].Size, items[index].Size)
|
||||||
|
t.markItemCompleted(elem.ID)
|
||||||
|
}
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markProcessing(ctx context.Context, elem *TaskElement) error {
|
||||||
|
t.processingMu.Lock()
|
||||||
|
if t.processing[elem.ID] != nil {
|
||||||
|
t.processingMu.Unlock()
|
||||||
|
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
|
||||||
|
}
|
||||||
|
t.processing[elem.ID] = elem
|
||||||
|
t.processingMu.Unlock()
|
||||||
|
t.markItemActive(elem.ID, elem.stream, time.Now())
|
||||||
|
t.notifyProgress(ctx)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) unmarkProcessing(id string) {
|
||||||
|
t.processingMu.Lock()
|
||||||
|
delete(t.processing, id)
|
||||||
|
t.processingMu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) downloadElement(ctx context.Context, elem *TaskElement) error {
|
||||||
|
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", elem.File.Name()))
|
||||||
|
logger.Info("Starting file download")
|
||||||
|
localFile, err := fsutil.CreateFile(elem.localPath)
|
||||||
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return fmt.Errorf("failed to create local file: %w", err)
|
||||||
|
}
|
||||||
|
wrAt := ioutil.NewProgressWriterAt(localFile, func(n int) {
|
||||||
|
t.recordItemDownload(elem.ID, int64(n), time.Now())
|
||||||
|
downloaded := t.downloaded.Add(int64(n))
|
||||||
|
t.notifyProgress(ctx)
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalSize,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
_, downloadErr := tdler.NewDownloader(elem.File).Parallel(ctx, wrAt)
|
||||||
|
closeErr := localFile.Close()
|
||||||
|
if downloadErr != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageDownload, downloadErr)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return fmt.Errorf("failed to download file: %w", downloadErr)
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, closeErr)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
return fmt.Errorf("failed to close cache file: %w", closeErr)
|
||||||
|
}
|
||||||
|
logger.Info("File downloaded successfully")
|
||||||
|
if path.Ext(elem.FileName()) == "" {
|
||||||
|
if ext := fsutil.DetectFileExt(elem.localPath); ext != "" {
|
||||||
|
elem.Path += ext
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.markItemDownloaded(elem.ID)
|
||||||
|
t.notifyProgress(ctx)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
||||||
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", elem.File.Name()))
|
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("file[%s]", elem.File.Name()))
|
||||||
if elem.stream {
|
if elem.stream {
|
||||||
@@ -59,11 +336,23 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
defer pr.Close()
|
defer pr.Close()
|
||||||
errg, uploadCtx := errgroup.WithContext(ctx)
|
errg, uploadCtx := errgroup.WithContext(ctx)
|
||||||
errg.Go(func() error {
|
errg.Go(func() error {
|
||||||
return elem.Storage.Save(uploadCtx, pr, elem.Path)
|
err := elem.Storage.Save(uploadCtx, pr, elem.Path)
|
||||||
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageUpload, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
}
|
||||||
|
return err
|
||||||
})
|
})
|
||||||
wr := ioutil.NewProgressWriter(pw, func(n int) {
|
wr := ioutil.NewProgressWriter(pw, func(n int) {
|
||||||
t.downloaded.Add(int64(n))
|
t.recordItemDownload(elem.ID, int64(n), time.Now())
|
||||||
t.Progress.OnProgress(ctx, t)
|
downloaded := t.downloaded.Add(int64(n))
|
||||||
|
t.notifyProgress(ctx)
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalSize,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
})
|
})
|
||||||
errg.Go(func() error {
|
errg.Go(func() error {
|
||||||
defer pw.Close()
|
defer pw.Close()
|
||||||
@@ -71,6 +360,8 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
_, err := tdler.NewDownloader(elem.File).Stream(uploadCtx, wr)
|
_, err := tdler.NewDownloader(elem.File).Stream(uploadCtx, wr)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
logger.Errorf("Failed to download file: %v", err)
|
logger.Errorf("Failed to download file: %v", err)
|
||||||
|
t.markItemFailed(elem.ID, FailureStageDownload, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
pw.CloseWithError(err)
|
pw.CloseWithError(err)
|
||||||
}
|
}
|
||||||
return err
|
return err
|
||||||
@@ -78,12 +369,22 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
if err := errg.Wait(); err != nil {
|
if err := errg.Wait(); err != nil {
|
||||||
return fmt.Errorf("failed to download file in stream mode: %w", err)
|
return fmt.Errorf("failed to download file in stream mode: %w", err)
|
||||||
}
|
}
|
||||||
|
// Streamed bytes are the uploaded bytes.
|
||||||
|
var streamedBytes int64
|
||||||
|
t.updateItem(elem.ID, func(item *itemProgressState) {
|
||||||
|
streamedBytes = item.downloaded
|
||||||
|
})
|
||||||
|
t.recordDownloadComplete(elem.ID, streamedBytes)
|
||||||
|
t.markItemCompleted(elem.ID)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
logger.Info("File downloaded successfully in stream mode")
|
logger.Info("File downloaded successfully in stream mode")
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
logger.Info("Starting file download")
|
logger.Info("Starting file download")
|
||||||
localFile, err := fsutil.CreateFile(elem.localPath)
|
localFile, err := fsutil.CreateFile(elem.localPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
return fmt.Errorf("failed to create local file: %w", err)
|
return fmt.Errorf("failed to create local file: %w", err)
|
||||||
}
|
}
|
||||||
defer func() {
|
defer func() {
|
||||||
@@ -92,11 +393,20 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
wrAt := ioutil.NewProgressWriterAt(localFile, func(n int) {
|
wrAt := ioutil.NewProgressWriterAt(localFile, func(n int) {
|
||||||
t.downloaded.Add(int64(n))
|
t.recordItemDownload(elem.ID, int64(n), time.Now())
|
||||||
t.Progress.OnProgress(ctx, t)
|
downloaded := t.downloaded.Add(int64(n))
|
||||||
|
t.notifyProgress(ctx)
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalSize,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
})
|
})
|
||||||
_, err = tdler.NewDownloader(elem.File).Parallel(ctx, wrAt)
|
_, err = tdler.NewDownloader(elem.File).Parallel(ctx, wrAt)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageDownload, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
return fmt.Errorf("failed to download file: %w", err)
|
return fmt.Errorf("failed to download file: %w", err)
|
||||||
}
|
}
|
||||||
logger.Info("File downloaded successfully")
|
logger.Info("File downloaded successfully")
|
||||||
@@ -109,21 +419,50 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
var fileStat os.FileInfo
|
var fileStat os.FileInfo
|
||||||
fileStat, err = os.Stat(elem.localPath)
|
fileStat, err = os.Stat(elem.localPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
t.markItemFailed(elem.ID, FailureStageCache, err)
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
return fmt.Errorf("failed to get file stat: %w", err)
|
return fmt.Errorf("failed to get file stat: %w", err)
|
||||||
}
|
}
|
||||||
|
t.recordDownloadComplete(elem.ID, fileStat.Size())
|
||||||
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
|
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
|
||||||
|
t.startUpload(vctx)
|
||||||
|
onProgress := t.uploadCallback(vctx, elem.ID)
|
||||||
|
attempt := 0
|
||||||
|
retryLimit := int(config.C().Retry)
|
||||||
|
lastFailureStage := FailureStageUpload
|
||||||
err = retry.Retry(func() error {
|
err = retry.Retry(func() error {
|
||||||
|
attempt++
|
||||||
var file *os.File
|
var file *os.File
|
||||||
file, err = os.Open(elem.localPath)
|
file, err = os.Open(elem.localPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
lastFailureStage = FailureStageCache
|
||||||
|
t.markItemRetry(elem.ID, lastFailureStage, attempt, retryLimit, err)
|
||||||
|
t.notifyStateChange(vctx)
|
||||||
return fmt.Errorf("failed to open cache file: %w", err)
|
return fmt.Errorf("failed to open cache file: %w", err)
|
||||||
}
|
}
|
||||||
defer file.Close()
|
defer file.Close()
|
||||||
if err = elem.Storage.Save(vctx, file, elem.Path); err != nil {
|
onProgress(0, fileStat.Size())
|
||||||
|
if progressSaver, ok := elem.Storage.(storage.StorageProgressSaver); ok {
|
||||||
|
err = progressSaver.SaveWithProgress(vctx, file, elem.Path, onProgress)
|
||||||
|
} else {
|
||||||
|
err = elem.Storage.Save(vctx, ioutil.NewProgressReader(file, fileStat.Size(), onProgress), elem.Path)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
logger.Errorf("Failed to save file: %s, retrying...", err)
|
logger.Errorf("Failed to save file: %s, retrying...", err)
|
||||||
|
lastFailureStage = t.itemFailureStage(elem.ID)
|
||||||
|
t.markItemRetry(elem.ID, lastFailureStage, attempt, retryLimit, err)
|
||||||
|
t.notifyStateChange(vctx)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}, retry.Context(vctx), retry.RetryTimes(uint(config.C().Retry)))
|
}, retry.Context(vctx), retry.RetryTimes(uint(config.C().Retry)))
|
||||||
|
if err == nil {
|
||||||
|
onProgress(fileStat.Size(), fileStat.Size())
|
||||||
|
t.markItemCompleted(elem.ID)
|
||||||
|
t.notifyStateChange(vctx)
|
||||||
|
} else {
|
||||||
|
t.markItemFailed(elem.ID, lastFailureStage, err)
|
||||||
|
t.notifyStateChange(vctx)
|
||||||
|
}
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
57
core/tasks/batchtfile/execute_group_test.go
Normal file
57
core/tasks/batchtfile/execute_group_test.go
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
package batchtfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/tfile"
|
||||||
|
tgstorage "github.com/krau/SaveAny-Bot/storage/telegram"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExecutionGroupsPreserveSourceAlbums(t *testing.T) {
|
||||||
|
stor := new(tgstorage.Telegram)
|
||||||
|
otherStor := new(tgstorage.Telegram)
|
||||||
|
task := Task{elems: []TaskElement{
|
||||||
|
{Storage: stor, sourceGroupKey: "album-1"},
|
||||||
|
{Storage: stor, sourceGroupKey: "album-1"},
|
||||||
|
{Storage: stor},
|
||||||
|
{Storage: stor, sourceGroupKey: "album-2"},
|
||||||
|
{Storage: stor, sourceGroupKey: "album-2"},
|
||||||
|
{Storage: otherStor, sourceGroupKey: "album-2"},
|
||||||
|
}}
|
||||||
|
|
||||||
|
groups := task.executionGroups()
|
||||||
|
wantSizes := []int{2, 1, 2, 1}
|
||||||
|
wantBatch := []bool{true, false, true, true}
|
||||||
|
if len(groups) != len(wantSizes) {
|
||||||
|
t.Fatalf("got %d groups, want %d", len(groups), len(wantSizes))
|
||||||
|
}
|
||||||
|
for i := range groups {
|
||||||
|
if got := len(groups[i].elems); got != wantSizes[i] {
|
||||||
|
t.Errorf("group %d has %d elements, want %d", i, got, wantSizes[i])
|
||||||
|
}
|
||||||
|
if got := groups[i].usesBatchSaver(); got != wantBatch[i] {
|
||||||
|
t.Errorf("group %d batch=%v, want %v", i, got, wantBatch[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSourceMetadataPreservesAlbumIdentityAndCaption(t *testing.T) {
|
||||||
|
msg := &tg.Message{
|
||||||
|
PeerID: &tg.PeerChannel{ChannelID: 77},
|
||||||
|
Message: "original caption",
|
||||||
|
}
|
||||||
|
msg.SetGroupedID(42)
|
||||||
|
file := tfile.NewTGFile(nil, nil, 0, "photo.jpg", tfile.WithMessage(msg))
|
||||||
|
|
||||||
|
groupKey, caption, preserveCaption := sourceMetadata(file)
|
||||||
|
if groupKey != "*tg.PeerChannel:77:42" {
|
||||||
|
t.Fatalf("group key = %q, want %q", groupKey, "*tg.PeerChannel:77:42")
|
||||||
|
}
|
||||||
|
if caption != "original caption" {
|
||||||
|
t.Fatalf("caption = %q, want original caption", caption)
|
||||||
|
}
|
||||||
|
if !preserveCaption {
|
||||||
|
t.Fatal("preserveCaption = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
360
core/tasks/batchtfile/item_progress.go
Normal file
360
core/tasks/batchtfile/item_progress.go
Normal file
@@ -0,0 +1,360 @@
|
|||||||
|
package batchtfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
transferSpeedWindow = 5 * time.Second
|
||||||
|
transferSamplePeriod = 250 * time.Millisecond
|
||||||
|
)
|
||||||
|
|
||||||
|
// ItemPhase describes the current lifecycle stage of one batch item.
|
||||||
|
type ItemPhase uint8
|
||||||
|
|
||||||
|
const (
|
||||||
|
ItemPhaseWaiting ItemPhase = iota
|
||||||
|
ItemPhaseDownloading
|
||||||
|
ItemPhaseTransferring
|
||||||
|
ItemPhaseDownloaded
|
||||||
|
ItemPhaseUploading
|
||||||
|
ItemPhaseRetrying
|
||||||
|
ItemPhaseConfirming
|
||||||
|
ItemPhaseCompleted
|
||||||
|
ItemPhaseFailed
|
||||||
|
ItemPhaseStopped
|
||||||
|
)
|
||||||
|
|
||||||
|
// FailureStage identifies the operation that failed for one batch item.
|
||||||
|
type FailureStage uint8
|
||||||
|
|
||||||
|
const (
|
||||||
|
FailureStageNone FailureStage = iota
|
||||||
|
FailureStageDownload
|
||||||
|
FailureStageCache
|
||||||
|
FailureStageUpload
|
||||||
|
FailureStageConfirm
|
||||||
|
FailureStageBatchUpload
|
||||||
|
FailureStageInternal
|
||||||
|
)
|
||||||
|
|
||||||
|
// TaskItemProgress is an immutable progress snapshot for one batch item.
|
||||||
|
type TaskItemProgress struct {
|
||||||
|
Index int
|
||||||
|
ID string
|
||||||
|
Name string
|
||||||
|
Size int64
|
||||||
|
Downloaded int64
|
||||||
|
Uploaded int64
|
||||||
|
DownloadSpeed float64
|
||||||
|
UploadSpeed float64
|
||||||
|
Phase ItemPhase
|
||||||
|
FailureStage FailureStage
|
||||||
|
RetryAttempt int
|
||||||
|
RetryLimit int
|
||||||
|
Error string
|
||||||
|
}
|
||||||
|
|
||||||
|
type transferSample struct {
|
||||||
|
at time.Time
|
||||||
|
bytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type transferMeter struct {
|
||||||
|
samples []transferSample
|
||||||
|
latest transferSample
|
||||||
|
hasData bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *transferMeter) record(now time.Time, transferred int64) {
|
||||||
|
if m.hasData && transferred < m.latest.bytes {
|
||||||
|
m.reset()
|
||||||
|
}
|
||||||
|
if m.hasData && now.Before(m.latest.at) {
|
||||||
|
now = m.latest.at
|
||||||
|
}
|
||||||
|
sample := transferSample{at: now, bytes: transferred}
|
||||||
|
m.latest = sample
|
||||||
|
m.hasData = true
|
||||||
|
if len(m.samples) == 0 {
|
||||||
|
m.samples = append(m.samples, sample)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if now.Sub(m.samples[len(m.samples)-1].at) >= transferSamplePeriod {
|
||||||
|
m.samples = append(m.samples, sample)
|
||||||
|
}
|
||||||
|
cutoff := now.Add(-transferSpeedWindow)
|
||||||
|
for len(m.samples) > 1 && m.samples[0].at.Before(cutoff) {
|
||||||
|
m.samples = m.samples[1:]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *transferMeter) speed() float64 {
|
||||||
|
if !m.hasData || len(m.samples) == 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
first := m.samples[0]
|
||||||
|
last := m.latest
|
||||||
|
elapsed := last.at.Sub(first.at).Seconds()
|
||||||
|
if elapsed <= 0 || last.bytes <= first.bytes {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return float64(last.bytes-first.bytes) / elapsed
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *transferMeter) reset() {
|
||||||
|
m.samples = m.samples[:0]
|
||||||
|
m.latest = transferSample{}
|
||||||
|
m.hasData = false
|
||||||
|
}
|
||||||
|
|
||||||
|
type itemProgressState struct {
|
||||||
|
index int
|
||||||
|
id string
|
||||||
|
name string
|
||||||
|
expectedSize int64
|
||||||
|
actualSize int64
|
||||||
|
downloaded int64
|
||||||
|
uploaded int64
|
||||||
|
phase ItemPhase
|
||||||
|
failureStage FailureStage
|
||||||
|
retryAttempt int
|
||||||
|
retryLimit int
|
||||||
|
err string
|
||||||
|
downloadMeter transferMeter
|
||||||
|
uploadMeter transferMeter
|
||||||
|
}
|
||||||
|
|
||||||
|
func newItemProgressStates(elems []TaskElement) ([]itemProgressState, map[string]int) {
|
||||||
|
states := make([]itemProgressState, 0, len(elems))
|
||||||
|
index := make(map[string]int, len(elems))
|
||||||
|
for i, elem := range elems {
|
||||||
|
name := ""
|
||||||
|
size := int64(0)
|
||||||
|
if elem.File != nil {
|
||||||
|
name = elem.File.Name()
|
||||||
|
size = elem.File.Size()
|
||||||
|
}
|
||||||
|
states = append(states, itemProgressState{
|
||||||
|
index: i + 1,
|
||||||
|
id: elem.ID,
|
||||||
|
name: name,
|
||||||
|
expectedSize: size,
|
||||||
|
phase: ItemPhaseWaiting,
|
||||||
|
})
|
||||||
|
index[elem.ID] = i
|
||||||
|
}
|
||||||
|
return states, index
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) updateItem(id string, update func(*itemProgressState)) bool {
|
||||||
|
t.itemMu.Lock()
|
||||||
|
defer t.itemMu.Unlock()
|
||||||
|
index, ok := t.itemIndex[id]
|
||||||
|
if !ok || index < 0 || index >= len(t.itemStates) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
update(&t.itemStates[index])
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markItemActive(id string, stream bool, now time.Time) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
if stream {
|
||||||
|
item.phase = ItemPhaseTransferring
|
||||||
|
} else {
|
||||||
|
item.phase = ItemPhaseDownloading
|
||||||
|
}
|
||||||
|
item.failureStage = FailureStageNone
|
||||||
|
item.err = ""
|
||||||
|
item.downloadMeter.record(now, item.downloaded)
|
||||||
|
if stream {
|
||||||
|
item.uploadMeter.record(now, item.uploaded)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) recordItemDownload(id string, n int64, now time.Time) {
|
||||||
|
if n <= 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
item.downloaded += n
|
||||||
|
item.downloadMeter.record(now, item.downloaded)
|
||||||
|
if item.phase == ItemPhaseTransferring {
|
||||||
|
item.uploaded += n
|
||||||
|
item.uploadMeter.record(now, item.uploaded)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markItemDownloaded(id string) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
item.phase = ItemPhaseDownloaded
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) recordItemDownloaded(id string, actualSize int64) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
if actualSize > 0 {
|
||||||
|
item.actualSize = actualSize
|
||||||
|
}
|
||||||
|
if item.actualSize == 0 {
|
||||||
|
item.actualSize = item.downloaded
|
||||||
|
}
|
||||||
|
if item.phase != ItemPhaseTransferring {
|
||||||
|
item.phase = ItemPhaseDownloaded
|
||||||
|
item.uploadMeter.reset()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) recordItemUpload(id string, uploaded, total int64, now time.Time) bool {
|
||||||
|
becameConfirming := false
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
if uploaded < item.uploaded {
|
||||||
|
if item.phase != ItemPhaseRetrying {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
item.uploadMeter.reset()
|
||||||
|
}
|
||||||
|
if total > 0 {
|
||||||
|
item.actualSize = total
|
||||||
|
}
|
||||||
|
item.uploaded = uploaded
|
||||||
|
item.uploadMeter.record(now, uploaded)
|
||||||
|
if total > 0 && uploaded >= total {
|
||||||
|
becameConfirming = item.phase != ItemPhaseConfirming
|
||||||
|
item.phase = ItemPhaseConfirming
|
||||||
|
return
|
||||||
|
}
|
||||||
|
item.phase = ItemPhaseUploading
|
||||||
|
item.failureStage = FailureStageNone
|
||||||
|
item.err = ""
|
||||||
|
})
|
||||||
|
return becameConfirming
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markItemRetry(id string, stage FailureStage, attempt, limit int, err error) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
item.phase = ItemPhaseRetrying
|
||||||
|
item.failureStage = stage
|
||||||
|
item.retryAttempt = attempt
|
||||||
|
item.retryLimit = limit
|
||||||
|
item.err = compactError(err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markItemFailed(id string, stage FailureStage, err error) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
if item.phase == ItemPhaseFailed || item.phase == ItemPhaseCompleted {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if errors.Is(err, context.Canceled) {
|
||||||
|
item.phase = ItemPhaseStopped
|
||||||
|
return
|
||||||
|
}
|
||||||
|
item.phase = ItemPhaseFailed
|
||||||
|
item.failureStage = stage
|
||||||
|
item.err = compactError(err)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) markItemCompleted(id string) {
|
||||||
|
t.updateItem(id, func(item *itemProgressState) {
|
||||||
|
item.phase = ItemPhaseCompleted
|
||||||
|
item.failureStage = FailureStageNone
|
||||||
|
item.err = ""
|
||||||
|
item.retryAttempt = 0
|
||||||
|
item.retryLimit = 0
|
||||||
|
if item.actualSize == 0 {
|
||||||
|
item.actualSize = max(item.downloaded, item.uploaded)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) finishItems(err error) {
|
||||||
|
if err == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.itemMu.Lock()
|
||||||
|
defer t.itemMu.Unlock()
|
||||||
|
for i := range t.itemStates {
|
||||||
|
item := &t.itemStates[i]
|
||||||
|
if item.phase == ItemPhaseCompleted || item.phase == ItemPhaseFailed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
item.phase = ItemPhaseStopped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) itemFailureStage(id string) FailureStage {
|
||||||
|
t.itemMu.RLock()
|
||||||
|
defer t.itemMu.RUnlock()
|
||||||
|
index, ok := t.itemIndex[id]
|
||||||
|
if !ok || index < 0 || index >= len(t.itemStates) {
|
||||||
|
return FailureStageUpload
|
||||||
|
}
|
||||||
|
if t.itemStates[index].phase == ItemPhaseConfirming {
|
||||||
|
return FailureStageConfirm
|
||||||
|
}
|
||||||
|
return FailureStageUpload
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) Items() []TaskItemProgress {
|
||||||
|
t.itemMu.RLock()
|
||||||
|
defer t.itemMu.RUnlock()
|
||||||
|
items := make([]TaskItemProgress, 0, len(t.itemStates))
|
||||||
|
for i := range t.itemStates {
|
||||||
|
item := &t.itemStates[i]
|
||||||
|
size := item.actualSize
|
||||||
|
if size == 0 {
|
||||||
|
size = item.expectedSize
|
||||||
|
}
|
||||||
|
items = append(items, TaskItemProgress{
|
||||||
|
Index: item.index,
|
||||||
|
ID: item.id,
|
||||||
|
Name: item.name,
|
||||||
|
Size: size,
|
||||||
|
Downloaded: item.downloaded,
|
||||||
|
Uploaded: item.uploaded,
|
||||||
|
DownloadSpeed: item.downloadMeter.speed(),
|
||||||
|
UploadSpeed: item.uploadMeter.speed(),
|
||||||
|
Phase: item.phase,
|
||||||
|
FailureStage: item.failureStage,
|
||||||
|
RetryAttempt: item.retryAttempt,
|
||||||
|
RetryLimit: item.retryLimit,
|
||||||
|
Error: item.err,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return items
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) ActualTotalSize() int64 {
|
||||||
|
items := t.Items()
|
||||||
|
var total int64
|
||||||
|
for _, item := range items {
|
||||||
|
total += item.Size
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
}
|
||||||
|
|
||||||
|
type stateProgressTracker interface {
|
||||||
|
OnStateChange(ctx context.Context, info TaskInfo)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) notifyStateChange(ctx context.Context) {
|
||||||
|
if tracker, ok := t.Progress.(stateProgressTracker); ok {
|
||||||
|
tracker.OnStateChange(ctx, t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func compactError(err error) string {
|
||||||
|
if err == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.Join(strings.Fields(err.Error()), " ")
|
||||||
|
}
|
||||||
35
core/tasks/batchtfile/notify_test.go
Normal file
35
core/tasks/batchtfile/notify_test.go
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
package batchtfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
type recordingTracker struct {
|
||||||
|
calls atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recordingTracker) OnStart(context.Context, TaskInfo) {}
|
||||||
|
func (r *recordingTracker) OnDone(context.Context, TaskInfo, error) {
|
||||||
|
}
|
||||||
|
func (r *recordingTracker) OnProgress(context.Context, TaskInfo) {
|
||||||
|
r.calls.Add(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Regression: notifyProgress must call the tracker, not itself. The previous
|
||||||
|
// self-call recursed until stack overflow on any batch task with a tracker.
|
||||||
|
func TestNotifyProgressCallsTracker(t *testing.T) {
|
||||||
|
tracker := &recordingTracker{}
|
||||||
|
task := &Task{Progress: tracker}
|
||||||
|
task.notifyProgress(t.Context())
|
||||||
|
if tracker.calls.Load() != 1 {
|
||||||
|
t.Fatalf("expected 1 tracker call, got %d", tracker.calls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The nil tracker path must stay a no-op.
|
||||||
|
func TestNotifyProgressNilTracker(t *testing.T) {
|
||||||
|
task := &Task{}
|
||||||
|
task.notifyProgress(t.Context()) // must not panic
|
||||||
|
}
|
||||||
@@ -4,20 +4,19 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"strconv"
|
"path"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/duke-git/lancet/v2/slice"
|
|
||||||
"github.com/gotd/td/telegram/message/entity"
|
|
||||||
"github.com/gotd/td/telegram/message/styling"
|
|
||||||
"github.com/gotd/td/tg"
|
"github.com/gotd/td/tg"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ProgressTracker interface {
|
type ProgressTracker interface {
|
||||||
@@ -27,159 +26,474 @@ type ProgressTracker interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type Progress struct {
|
type Progress struct {
|
||||||
MessageID int
|
MessageID int
|
||||||
ChatID int64
|
ChatID int64
|
||||||
start time.Time
|
updateMu sync.Mutex
|
||||||
lastUpdatePercent atomic.Int32
|
lastUpdateAt time.Time
|
||||||
skippedFiles []string
|
lastText string
|
||||||
|
done bool
|
||||||
|
skippedFiles []string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type renderedBatchMessage struct {
|
||||||
|
Text string
|
||||||
|
Entities []tg.MessageEntityClass
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
progressRenderInterval = time.Second
|
||||||
|
maxVisibleActiveItems = 5
|
||||||
|
progressBarWidth = 10
|
||||||
|
maxDisplayNameRunes = 36
|
||||||
|
maxDisplayErrorRunes = 240
|
||||||
|
)
|
||||||
|
|
||||||
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
||||||
p.start = time.Now()
|
p.render(ctx, info, true)
|
||||||
p.lastUpdatePercent.Store(0)
|
|
||||||
log.FromContext(ctx).Debugf("Batch task progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
|
|
||||||
entityBuilder := entity.Builder{}
|
|
||||||
var entities []tg.MessageEntityClass
|
|
||||||
if err := styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchStartPrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalSize())/(1024*1024), info.Count())),
|
|
||||||
); err != nil {
|
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
text, entities := entityBuilder.Complete()
|
|
||||||
req := &tg.MessagesEditMessageRequest{
|
|
||||||
ID: p.MessageID,
|
|
||||||
}
|
|
||||||
req.SetMessage(text)
|
|
||||||
req.SetEntities(entities)
|
|
||||||
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
|
|
||||||
Rows: []tg.KeyboardButtonRow{
|
|
||||||
{
|
|
||||||
Buttons: []tg.KeyboardButtonClass{
|
|
||||||
tgutil.BuildCancelButton(info.TaskID()),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}},
|
|
||||||
)
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
|
||||||
if ext != nil {
|
|
||||||
ext.EditMessage(p.ChatID, req)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
||||||
if !shouldUpdateProgress(info.TotalSize(), info.Downloaded(), int(p.lastUpdatePercent.Load())) {
|
p.render(ctx, info, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) OnStateChange(ctx context.Context, info TaskInfo) {
|
||||||
|
p.render(ctx, info, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) OnUploadStart(ctx context.Context, info TaskInfo, _ int64) {
|
||||||
|
p.render(ctx, info, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) OnUploadProgress(ctx context.Context, info TaskInfo, _, _ int64) {
|
||||||
|
p.render(ctx, info, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) render(ctx context.Context, info TaskInfo, priority bool) {
|
||||||
|
p.updateMu.Lock()
|
||||||
|
defer p.updateMu.Unlock()
|
||||||
|
if p.done {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
percent := int((info.Downloaded() * 100) / info.TotalSize())
|
now := time.Now()
|
||||||
if p.lastUpdatePercent.Load() == int32(percent) {
|
if !priority && !p.lastUpdateAt.IsZero() && now.Sub(p.lastUpdateAt) < progressRenderInterval {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
p.lastUpdatePercent.Store(int32(percent))
|
message := buildBatchProgressMessage(info, p.skippedFiles, visibleActiveItems())
|
||||||
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.Downloaded(), info.TotalSize())
|
if message.Err != nil {
|
||||||
entityBuilder := entity.Builder{}
|
log.FromContext(ctx).Errorf("Failed to render batch progress message: %v", message.Err)
|
||||||
var entities []tg.MessageEntityClass
|
|
||||||
if err := styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchProcessingPrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalSize())/(1024*1024), info.Count())),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
|
|
||||||
func() styling.StyledTextOption {
|
|
||||||
var lines []string
|
|
||||||
for _, elem := range info.Processing() {
|
|
||||||
lines = append(lines, fmt.Sprintf(" - %s (%.2f MB)", elem.FileName(), float64(elem.FileSize())/(1024*1024)))
|
|
||||||
}
|
|
||||||
if len(lines) == 0 {
|
|
||||||
lines = append(lines, i18n.T(i18nk.BotMsgProgressProcessingNone, nil))
|
|
||||||
}
|
|
||||||
return styling.Plain(slice.Join(lines, "\n"))
|
|
||||||
}(),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressAvgSpeedPrefix, nil)),
|
|
||||||
styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(info.Downloaded(), p.start)/(1024*1024))),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressCurrentProgressPrefix, nil)),
|
|
||||||
styling.Bold(fmt.Sprintf("%.2f%%", float64(info.Downloaded())/float64(info.TotalSize())*100)),
|
|
||||||
); err != nil {
|
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
text, entities := entityBuilder.Complete()
|
if message.Text == p.lastText {
|
||||||
req := &tg.MessagesEditMessageRequest{
|
|
||||||
ID: p.MessageID,
|
|
||||||
}
|
|
||||||
req.SetMessage(text)
|
|
||||||
req.SetEntities(entities)
|
|
||||||
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
|
|
||||||
Rows: []tg.KeyboardButtonRow{
|
|
||||||
{
|
|
||||||
Buttons: []tg.KeyboardButtonClass{
|
|
||||||
tgutil.BuildCancelButton(info.TaskID()),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}},
|
|
||||||
)
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
|
||||||
if ext != nil {
|
|
||||||
ext.EditMessage(p.ChatID, req)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
p.lastText = message.Text
|
||||||
|
p.lastUpdateAt = now
|
||||||
|
p.editMessage(ctx, info.TaskID(), message, true)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
||||||
if err != nil {
|
p.updateMu.Lock()
|
||||||
log.FromContext(ctx).Errorf("Batch task %s failed: %s", info.TaskID(), err)
|
defer p.updateMu.Unlock()
|
||||||
} else {
|
if p.done {
|
||||||
log.FromContext(ctx).Debugf("Batch task %s completed successfully", info.TaskID())
|
|
||||||
}
|
|
||||||
entityBuilder := entity.Builder{}
|
|
||||||
var stylingErr error
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
if errors.Is(err, context.Canceled) {
|
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskCanceled, nil)),
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskFailedWithError, map[string]any{
|
|
||||||
"Error": "",
|
|
||||||
})),
|
|
||||||
styling.Code(err.Error()),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressBatchDonePrefix, nil)),
|
|
||||||
styling.Code(strconv.Itoa(info.Count())),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressTotalSizePrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("%.2f MB", float64(info.TotalSize())/(1024*1024))),
|
|
||||||
func() styling.StyledTextOption {
|
|
||||||
if len(p.skippedFiles) == 0 {
|
|
||||||
return styling.Plain("")
|
|
||||||
}
|
|
||||||
return styling.Plain("\n\n" + i18n.T(i18nk.BotMsgCommonInfoConflictFilesSkipped, map[string]any{
|
|
||||||
"Skipped": strings.Join(p.skippedFiles, "\n"),
|
|
||||||
}))
|
|
||||||
}(),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
if stylingErr != nil {
|
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", stylingErr)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
p.done = true
|
||||||
text, entities := entityBuilder.Complete()
|
message := buildBatchDoneMessage(info, p.skippedFiles, err)
|
||||||
req := &tg.MessagesEditMessageRequest{
|
if message.Err != nil {
|
||||||
ID: p.MessageID,
|
log.FromContext(ctx).Errorf("Failed to render final batch progress message: %v", message.Err)
|
||||||
|
return
|
||||||
}
|
}
|
||||||
req.SetMessage(text)
|
p.lastText = message.Text
|
||||||
req.SetEntities(entities)
|
p.editMessage(ctx, info.TaskID(), message, false)
|
||||||
|
}
|
||||||
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
func (p *Progress) editMessage(ctx context.Context, taskID string, message renderedBatchMessage, cancellable bool) {
|
||||||
if ext != nil {
|
if message.Err != nil {
|
||||||
ext.EditMessage(p.ChatID, req)
|
log.FromContext(ctx).Errorf("Failed to render batch progress message: %v", message.Err)
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
req := buildBatchEditMessageRequest(p.MessageID, taskID, message, cancellable)
|
||||||
|
if ext := tgutil.ExtFromContext(ctx); ext != nil {
|
||||||
|
if _, err := ext.EditMessage(p.ChatID, req); err != nil {
|
||||||
|
log.FromContext(ctx).Errorf("Failed to edit batch progress message: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildBatchEditMessageRequest(messageID int, taskID string, message renderedBatchMessage, cancellable bool) *tg.MessagesEditMessageRequest {
|
||||||
|
req := &tg.MessagesEditMessageRequest{ID: messageID}
|
||||||
|
req.SetMessage(message.Text)
|
||||||
|
if len(message.Entities) > 0 {
|
||||||
|
req.SetEntities(message.Entities)
|
||||||
|
}
|
||||||
|
if cancellable {
|
||||||
|
req.SetReplyMarkup(&tg.ReplyInlineMarkup{Rows: []tg.KeyboardButtonRow{{
|
||||||
|
Buttons: []tg.KeyboardButtonClass{tgutil.BuildCancelButton(taskID)},
|
||||||
|
}}})
|
||||||
|
}
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildBatchProgressText(info TaskInfo, skipped []string, activeLimit int) string {
|
||||||
|
return buildBatchProgressMessage(info, skipped, activeLimit).Text
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildBatchProgressMessage(info TaskInfo, skipped []string, activeLimit int) renderedBatchMessage {
|
||||||
|
items := info.Items()
|
||||||
|
completed, waiting, downloaded, failed := itemCounts(items)
|
||||||
|
downloadSpeed, uploadSpeed := aggregateSpeeds(items)
|
||||||
|
if activeLimit < 1 {
|
||||||
|
activeLimit = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
total := len(items) + len(skipped)
|
||||||
|
downloadSpeedText := formatSpeed(downloadSpeed)
|
||||||
|
uploadSpeedText := formatSpeed(uploadSpeed)
|
||||||
|
header := localizedProgressMarkup(i18nk.BotMsgProgressBatchStatusHeader, map[string]any{
|
||||||
|
"Total": total,
|
||||||
|
"Completed": completed,
|
||||||
|
"Downloaded": downloaded,
|
||||||
|
"Waiting": waiting,
|
||||||
|
"DownloadSpeed": downloadSpeedText,
|
||||||
|
"UploadSpeed": uploadSpeedText,
|
||||||
|
})
|
||||||
|
|
||||||
|
var markup strings.Builder
|
||||||
|
markup.WriteString(header)
|
||||||
|
|
||||||
|
visibleItems, hiddenTransfers, summarizedConfirming := visibleBatchItems(items, activeLimit)
|
||||||
|
for _, item := range visibleItems {
|
||||||
|
markup.WriteString("\n\n")
|
||||||
|
markup.WriteString(formatActiveItemMarkup(item, len(items)))
|
||||||
|
}
|
||||||
|
|
||||||
|
if hiddenTransfers > 0 {
|
||||||
|
markup.WriteString("\n\n")
|
||||||
|
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryHiddenActive, map[string]any{"Count": hiddenTransfers}))
|
||||||
|
}
|
||||||
|
if summarizedConfirming > 0 {
|
||||||
|
markup.WriteString("\n\n")
|
||||||
|
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryConfirming, map[string]any{"Count": summarizedConfirming}))
|
||||||
|
}
|
||||||
|
if failed > 0 {
|
||||||
|
markup.WriteString("\n")
|
||||||
|
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummaryFailed, map[string]any{"Count": failed}))
|
||||||
|
}
|
||||||
|
if len(skipped) > 0 {
|
||||||
|
markup.WriteString("\n")
|
||||||
|
markup.WriteString(localizedProgressMarkup(i18nk.BotMsgProgressBatchSummarySkipped, map[string]any{"Count": len(skipped)}))
|
||||||
|
}
|
||||||
|
return completeBatchMessage(markup.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildBatchDoneMarkup(info TaskInfo, skipped []string, err error) string {
|
||||||
|
items := info.Items()
|
||||||
|
totalSize := info.ActualTotalSize()
|
||||||
|
if totalSize == 0 {
|
||||||
|
totalSize = info.TotalSize()
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
completed, _, _, failed := itemCounts(items)
|
||||||
|
// Report per-element failures instead of full completion.
|
||||||
|
totalSkipped := len(skipped) + failed
|
||||||
|
if totalSkipped > 0 {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchDoneWithSkipped, map[string]any{
|
||||||
|
"Success": completed,
|
||||||
|
"Skipped": totalSkipped,
|
||||||
|
"Size": dlutil.FormatSize(totalSize),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchDone, map[string]any{
|
||||||
|
"Count": len(items),
|
||||||
|
"Size": dlutil.FormatSize(totalSize),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
completed, _, _, failed := itemCounts(items)
|
||||||
|
incomplete := max(len(items)-completed-failed, 0)
|
||||||
|
if errors.Is(err, context.Canceled) {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchCanceled, map[string]any{
|
||||||
|
"Total": len(items) + len(skipped),
|
||||||
|
"Completed": completed,
|
||||||
|
"Incomplete": incomplete,
|
||||||
|
"Skipped": len(skipped),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
failedItems := make([]TaskItemProgress, 0, failed)
|
||||||
|
for _, item := range items {
|
||||||
|
if item.Phase == ItemPhaseFailed {
|
||||||
|
failedItems = append(failedItems, item)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(failedItems) > 1 && failedItems[0].FailureStage == FailureStageBatchUpload {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedGroup, map[string]any{
|
||||||
|
"Affected": len(failedItems),
|
||||||
|
"Reason": displayError(firstError(failedItems), err),
|
||||||
|
"Completed": completed,
|
||||||
|
"Failed": len(failedItems),
|
||||||
|
"Incomplete": incomplete,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if len(failedItems) == 0 {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedTask, map[string]any{
|
||||||
|
"Reason": displayError("", err),
|
||||||
|
"Completed": completed,
|
||||||
|
"Incomplete": incomplete,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
item := failedItems[0]
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchFailedItem, map[string]any{
|
||||||
|
"Index": item.Index,
|
||||||
|
"Name": truncateFilename(item.Name, maxDisplayNameRunes),
|
||||||
|
"Stage": failureStageLabel(item.FailureStage),
|
||||||
|
"Progress": failureProgress(item),
|
||||||
|
"Speed": failureSpeed(item),
|
||||||
|
"Reason": displayError(item.Error, err),
|
||||||
|
"Completed": completed,
|
||||||
|
"Failed": failed,
|
||||||
|
"Incomplete": incomplete,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildBatchDoneMessage(info TaskInfo, skipped []string, err error) renderedBatchMessage {
|
||||||
|
return completeBatchMessage(buildBatchDoneMarkup(info, skipped, err))
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatActiveItemMarkup(item TaskItemProgress, total int) string {
|
||||||
|
data := map[string]any{
|
||||||
|
"Index": item.Index,
|
||||||
|
"Total": total,
|
||||||
|
"Name": truncateFilename(item.Name, maxDisplayNameRunes),
|
||||||
|
"Speed": formatSpeed(itemSpeed(item)),
|
||||||
|
"Progress": itemPercent(item),
|
||||||
|
"Bar": textProgressBar(itemPercent(item)),
|
||||||
|
"Current": dlutil.FormatSize(itemBytes(item)),
|
||||||
|
"Size": dlutil.FormatSize(item.Size),
|
||||||
|
"Attempt": min(max(item.RetryAttempt, 1), max(item.RetryLimit, 1)),
|
||||||
|
"Limit": max(item.RetryLimit, 1),
|
||||||
|
"Reason": truncateRunes(item.Error, maxDisplayErrorRunes),
|
||||||
|
}
|
||||||
|
switch item.Phase {
|
||||||
|
case ItemPhaseDownloading:
|
||||||
|
if item.Size <= 0 {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemDownloadingUnknown, data)
|
||||||
|
}
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemDownloading, data)
|
||||||
|
case ItemPhaseTransferring:
|
||||||
|
if item.Size <= 0 {
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemTransferringUnknown, data)
|
||||||
|
}
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemTransferring, data)
|
||||||
|
case ItemPhaseUploading:
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemUploading, data)
|
||||||
|
case ItemPhaseRetrying:
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemRetrying, data)
|
||||||
|
case ItemPhaseConfirming:
|
||||||
|
return localizedProgressMarkup(i18nk.BotMsgProgressBatchItemConfirming, data)
|
||||||
|
default:
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func localizedProgressMarkup(key i18nk.Key, data map[string]any) string {
|
||||||
|
return i18n.T(key, tgutil.EscapeHTMLTemplateData(data))
|
||||||
|
}
|
||||||
|
|
||||||
|
func visibleBatchItems(items []TaskItemProgress, limit int) (visible []TaskItemProgress, hiddenTransfers, summarizedConfirming int) {
|
||||||
|
visible = make([]TaskItemProgress, 0, limit)
|
||||||
|
transferCount := 0
|
||||||
|
confirmingCount := 0
|
||||||
|
for _, item := range items {
|
||||||
|
switch {
|
||||||
|
case isTransferPhase(item.Phase):
|
||||||
|
transferCount++
|
||||||
|
if len(visible) < limit {
|
||||||
|
visible = append(visible, item)
|
||||||
|
}
|
||||||
|
case item.Phase == ItemPhaseConfirming:
|
||||||
|
confirmingCount++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hiddenTransfers = transferCount - len(visible)
|
||||||
|
if confirmingCount == 1 && len(visible) < limit {
|
||||||
|
for _, item := range items {
|
||||||
|
if item.Phase == ItemPhaseConfirming {
|
||||||
|
visible = append(visible, item)
|
||||||
|
return visible, hiddenTransfers, 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return visible, hiddenTransfers, confirmingCount
|
||||||
|
}
|
||||||
|
|
||||||
|
func completeBatchMessage(markup string) renderedBatchMessage {
|
||||||
|
text, entities, err := tgutil.RenderHTML(markup)
|
||||||
|
return renderedBatchMessage{Text: text, Entities: entities, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
func itemCounts(items []TaskItemProgress) (completed, waiting, downloaded, failed int) {
|
||||||
|
for _, item := range items {
|
||||||
|
switch item.Phase {
|
||||||
|
case ItemPhaseCompleted:
|
||||||
|
completed++
|
||||||
|
case ItemPhaseWaiting:
|
||||||
|
waiting++
|
||||||
|
case ItemPhaseDownloaded:
|
||||||
|
downloaded++
|
||||||
|
case ItemPhaseFailed:
|
||||||
|
failed++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func aggregateSpeeds(items []TaskItemProgress) (download, upload float64) {
|
||||||
|
for _, item := range items {
|
||||||
|
switch item.Phase {
|
||||||
|
case ItemPhaseDownloading:
|
||||||
|
download += item.DownloadSpeed
|
||||||
|
case ItemPhaseTransferring:
|
||||||
|
download += item.DownloadSpeed
|
||||||
|
upload += item.UploadSpeed
|
||||||
|
case ItemPhaseUploading:
|
||||||
|
upload += item.UploadSpeed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
func isTransferPhase(phase ItemPhase) bool {
|
||||||
|
switch phase {
|
||||||
|
case ItemPhaseDownloading, ItemPhaseTransferring, ItemPhaseUploading, ItemPhaseRetrying:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func itemBytes(item TaskItemProgress) int64 {
|
||||||
|
switch item.Phase {
|
||||||
|
case ItemPhaseDownloading, ItemPhaseTransferring:
|
||||||
|
return item.Downloaded
|
||||||
|
default:
|
||||||
|
return item.Uploaded
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func itemSpeed(item TaskItemProgress) float64 {
|
||||||
|
switch item.Phase {
|
||||||
|
case ItemPhaseDownloading, ItemPhaseTransferring:
|
||||||
|
return item.DownloadSpeed
|
||||||
|
case ItemPhaseUploading:
|
||||||
|
return item.UploadSpeed
|
||||||
|
case ItemPhaseRetrying:
|
||||||
|
return item.UploadSpeed
|
||||||
|
default:
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func itemPercent(item TaskItemProgress) int {
|
||||||
|
if item.Size <= 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return int(min(itemBytes(item), item.Size) * 100 / item.Size)
|
||||||
|
}
|
||||||
|
|
||||||
|
func textProgressBar(percent int) string {
|
||||||
|
percent = min(max(percent, 0), 100)
|
||||||
|
filled := percent * progressBarWidth / 100
|
||||||
|
return strings.Repeat("🟩", filled) + strings.Repeat("⬜️", progressBarWidth-filled)
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatSpeed(speed float64) string {
|
||||||
|
if speed <= 0 {
|
||||||
|
return "0 B/s"
|
||||||
|
}
|
||||||
|
return dlutil.FormatSize(int64(speed)) + "/s"
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncateFilename(name string, limit int) string {
|
||||||
|
if utf8.RuneCountInString(name) <= limit {
|
||||||
|
return name
|
||||||
|
}
|
||||||
|
ext := path.Ext(name)
|
||||||
|
if utf8.RuneCountInString(ext) >= limit-2 {
|
||||||
|
return truncateRunes(name, limit-1) + "…"
|
||||||
|
}
|
||||||
|
base := strings.TrimSuffix(name, ext)
|
||||||
|
baseLimit := limit - utf8.RuneCountInString(ext) - 1
|
||||||
|
return truncateRunes(base, baseLimit) + "…" + ext
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncateRunes(value string, limit int) string {
|
||||||
|
if limit <= 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
runes := []rune(value)
|
||||||
|
if len(runes) <= limit {
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
return string(runes[:limit])
|
||||||
|
}
|
||||||
|
|
||||||
|
func displayError(itemError string, fallback error) string {
|
||||||
|
if itemError == "" && fallback != nil {
|
||||||
|
itemError = compactError(fallback)
|
||||||
|
}
|
||||||
|
return truncateRunes(itemError, maxDisplayErrorRunes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstError(items []TaskItemProgress) string {
|
||||||
|
for _, item := range items {
|
||||||
|
if item.Error != "" {
|
||||||
|
return item.Error
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func failureStageLabel(stage FailureStage) string {
|
||||||
|
switch stage {
|
||||||
|
case FailureStageDownload:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageDownload, nil)
|
||||||
|
case FailureStageCache:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageCache, nil)
|
||||||
|
case FailureStageUpload:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageUpload, nil)
|
||||||
|
case FailureStageConfirm:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageConfirm, nil)
|
||||||
|
case FailureStageBatchUpload:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageBatchUpload, nil)
|
||||||
|
default:
|
||||||
|
return i18n.T(i18nk.BotMsgProgressBatchFailureStageInternal, nil)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func failureProgress(item TaskItemProgress) string {
|
||||||
|
if item.Size <= 0 {
|
||||||
|
return dlutil.FormatSize(failureBytes(item))
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%d%%", min(failureBytes(item), item.Size)*100/item.Size)
|
||||||
|
}
|
||||||
|
|
||||||
|
func failureSpeed(item TaskItemProgress) string {
|
||||||
|
if item.FailureStage == FailureStageDownload || item.FailureStage == FailureStageCache {
|
||||||
|
return formatSpeed(item.DownloadSpeed)
|
||||||
|
}
|
||||||
|
return formatSpeed(item.UploadSpeed)
|
||||||
|
}
|
||||||
|
|
||||||
|
func failureBytes(item TaskItemProgress) int64 {
|
||||||
|
if item.FailureStage == FailureStageDownload || item.FailureStage == FailureStageCache {
|
||||||
|
return item.Downloaded
|
||||||
|
}
|
||||||
|
return item.Uploaded
|
||||||
|
}
|
||||||
|
|
||||||
|
func visibleActiveItems() int {
|
||||||
|
return min(max(config.C().Workers, 1), maxVisibleActiveItems)
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewProgressTracker(messageID int, chatID int64) ProgressTracker {
|
func NewProgressTracker(messageID int, chatID int64) ProgressTracker {
|
||||||
|
|||||||
311
core/tasks/batchtfile/progress_regression_test.go
Normal file
311
core/tasks/batchtfile/progress_regression_test.go
Normal file
@@ -0,0 +1,311 @@
|
|||||||
|
package batchtfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/tfile"
|
||||||
|
)
|
||||||
|
|
||||||
|
type progressRegressionRecorder struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
startTotal int64
|
||||||
|
notifications []int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (*progressRegressionRecorder) OnStart(context.Context, TaskInfo) {}
|
||||||
|
func (*progressRegressionRecorder) OnProgress(context.Context, TaskInfo) {}
|
||||||
|
func (*progressRegressionRecorder) OnDone(context.Context, TaskInfo, error) {}
|
||||||
|
|
||||||
|
func (r *progressRegressionRecorder) OnUploadStart(_ context.Context, _ TaskInfo, total int64) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.startTotal = total
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *progressRegressionRecorder) OnUploadProgress(_ context.Context, _ TaskInfo, uploaded, _ int64) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.notifications = append(r.notifications, uploaded)
|
||||||
|
}
|
||||||
|
|
||||||
|
type orderedProgressRegressionRecorder struct {
|
||||||
|
firstEntered chan struct{}
|
||||||
|
releaseFirst chan struct{}
|
||||||
|
secondEntered chan struct{}
|
||||||
|
mu sync.Mutex
|
||||||
|
notifications []int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (*orderedProgressRegressionRecorder) OnStart(context.Context, TaskInfo) {}
|
||||||
|
func (*orderedProgressRegressionRecorder) OnProgress(context.Context, TaskInfo) {}
|
||||||
|
func (*orderedProgressRegressionRecorder) OnDone(context.Context, TaskInfo, error) {}
|
||||||
|
func (*orderedProgressRegressionRecorder) OnUploadStart(context.Context, TaskInfo, int64) {
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *orderedProgressRegressionRecorder) OnUploadProgress(_ context.Context, _ TaskInfo, uploaded, _ int64) {
|
||||||
|
if uploaded == 100 {
|
||||||
|
close(r.firstEntered)
|
||||||
|
<-r.releaseFirst
|
||||||
|
}
|
||||||
|
if uploaded == 200 {
|
||||||
|
close(r.secondEntered)
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
r.notifications = append(r.notifications, uploaded)
|
||||||
|
r.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchProgressShowsTransferSpeedAndSize(t *testing.T) {
|
||||||
|
useProgressRegressionLocale(t)
|
||||||
|
task := newProgressRegressionTask(nil,
|
||||||
|
progressRegressionFile{"downloading", 1000},
|
||||||
|
progressRegressionFile{"uploading", 1000},
|
||||||
|
progressRegressionFile{"waiting", 1000},
|
||||||
|
)
|
||||||
|
started := time.Unix(100, 0)
|
||||||
|
task.markItemActive("downloading", false, started)
|
||||||
|
task.recordItemDownload("downloading", 500, started.Add(time.Second))
|
||||||
|
task.recordItemDownloaded("uploading", 1000)
|
||||||
|
task.recordItemUpload("uploading", 0, 1000, started.Add(time.Second))
|
||||||
|
task.recordItemUpload("uploading", 250, 1000, started.Add(2*time.Second))
|
||||||
|
|
||||||
|
message := buildBatchProgressMessage(task, nil, 2)
|
||||||
|
if message.Err != nil {
|
||||||
|
t.Fatalf("buildBatchProgressMessage() failed: %v", message.Err)
|
||||||
|
}
|
||||||
|
assertProgressRegressionContains(t, message.Text,
|
||||||
|
"状态:✅ 0 | 📥 0 | ⏳ 1",
|
||||||
|
"⬇️ 1/3 下载中",
|
||||||
|
"速度:500 B/s",
|
||||||
|
"大小:500 B / 1000 B",
|
||||||
|
"⬆️ 2/3 上传中",
|
||||||
|
"速度:250 B/s",
|
||||||
|
"大小:250 B / 1000 B",
|
||||||
|
)
|
||||||
|
bold, _, blockquote, _ := batchEntityCounts(message.Entities)
|
||||||
|
if bold != 3 || blockquote != 2 {
|
||||||
|
t.Fatalf("entity counts = bold:%d blockquote:%d, want bold:3 blockquote:2", bold, blockquote)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchProgressLimitsRowsWithoutHidingActiveUpload(t *testing.T) {
|
||||||
|
useProgressRegressionLocale(t)
|
||||||
|
task := newProgressRegressionTask(nil,
|
||||||
|
progressRegressionFile{"confirm-01", 100},
|
||||||
|
progressRegressionFile{"confirm-02", 100},
|
||||||
|
progressRegressionFile{"uploading", 100},
|
||||||
|
progressRegressionFile{"downloading", 100},
|
||||||
|
)
|
||||||
|
started := time.Unix(100, 0)
|
||||||
|
task.recordItemUpload("confirm-01", 100, 100, started)
|
||||||
|
task.recordItemUpload("confirm-02", 100, 100, started)
|
||||||
|
task.recordItemUpload("uploading", 40, 100, started.Add(time.Second))
|
||||||
|
task.markItemActive("downloading", false, started)
|
||||||
|
|
||||||
|
message := buildBatchProgressText(task, nil, 2)
|
||||||
|
assertProgressRegressionContains(t, message,
|
||||||
|
"uploading.bin",
|
||||||
|
"downloading.bin",
|
||||||
|
"☁️ 已上传,等待整组发送:2",
|
||||||
|
)
|
||||||
|
if strings.Contains(message, "confirm-01.bin") || strings.Contains(message, "confirm-02.bin") {
|
||||||
|
t.Fatalf("confirmation rows displaced active transfers:\n%s", message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchProgressTemplateOwnsStylesAndEscapesValues(t *testing.T) {
|
||||||
|
useProgressRegressionLocale(t)
|
||||||
|
fileID := `<b>A&B</b>`
|
||||||
|
task := newProgressRegressionTask(nil, progressRegressionFile{fileID, 100})
|
||||||
|
task.markItemRetry(fileID, FailureStageUpload, 1, 3, errors.New(`<i>remote & failed</i>`))
|
||||||
|
|
||||||
|
message := buildBatchProgressMessage(task, nil, 1)
|
||||||
|
if message.Err != nil {
|
||||||
|
t.Fatalf("buildBatchProgressMessage() failed: %v", message.Err)
|
||||||
|
}
|
||||||
|
assertProgressRegressionContains(t, message.Text,
|
||||||
|
`<b>A&B</b>.bin`,
|
||||||
|
`<i>remote & failed</i>`,
|
||||||
|
)
|
||||||
|
bold, _, blockquote, italic := batchEntityCounts(message.Entities)
|
||||||
|
if bold != 2 || blockquote != 1 || italic != 0 {
|
||||||
|
t.Fatalf("entity counts = bold:%d blockquote:%d italic:%d", bold, blockquote, italic)
|
||||||
|
}
|
||||||
|
|
||||||
|
i18n.Init("en")
|
||||||
|
english := buildBatchProgressMessage(task, nil, 1)
|
||||||
|
if english.Err != nil {
|
||||||
|
t.Fatalf("English batch template failed: %v", english.Err)
|
||||||
|
}
|
||||||
|
assertProgressRegressionContains(t, english.Text, "📦 Processing", "Retrying upload", `<b>A&B</b>.bin`)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadProgressContinuesAfterUploadStarts(t *testing.T) {
|
||||||
|
useProgressRegressionLocale(t)
|
||||||
|
progress := new(Progress)
|
||||||
|
task := newProgressRegressionTask(progress,
|
||||||
|
progressRegressionFile{"uploading", 100},
|
||||||
|
progressRegressionFile{"downloading", 100},
|
||||||
|
)
|
||||||
|
progress.OnStart(t.Context(), task)
|
||||||
|
task.recordDownloadComplete("uploading", 100)
|
||||||
|
task.uploadCallback(t.Context(), "uploading")(50, 100)
|
||||||
|
|
||||||
|
started := time.Unix(100, 0)
|
||||||
|
task.markItemActive("downloading", false, started)
|
||||||
|
task.recordItemDownload("downloading", 50, started.Add(time.Second))
|
||||||
|
progress.updateMu.Lock()
|
||||||
|
progress.lastUpdateAt = time.Now().Add(-progressRenderInterval)
|
||||||
|
progress.updateMu.Unlock()
|
||||||
|
progress.OnProgress(t.Context(), task)
|
||||||
|
|
||||||
|
progress.updateMu.Lock()
|
||||||
|
text := progress.lastText
|
||||||
|
progress.updateMu.Unlock()
|
||||||
|
assertProgressRegressionContains(t, text,
|
||||||
|
"uploading.bin",
|
||||||
|
"🟩🟩🟩🟩🟩⬜️⬜️⬜️⬜️⬜️ 50%",
|
||||||
|
"总速度:⬇️ 50 B/s | ⬆️ 0 B/s",
|
||||||
|
"🔄 另有 1 个文件正在处理",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchUploadIgnoresOutOfOrderBytesAndAllowsRetryReset(t *testing.T) {
|
||||||
|
recorder := new(progressRegressionRecorder)
|
||||||
|
task := newProgressRegressionTask(recorder, progressRegressionFile{"file", 100})
|
||||||
|
task.recordDownloadComplete("file", 100)
|
||||||
|
callback := task.uploadCallback(t.Context(), "file")
|
||||||
|
callback(80, 100)
|
||||||
|
callback(10, 100)
|
||||||
|
|
||||||
|
if got := task.Items()[0].Uploaded; got != 80 {
|
||||||
|
t.Fatalf("out-of-order callback regressed item to %d, want 80", got)
|
||||||
|
}
|
||||||
|
task.markItemRetry("file", FailureStageUpload, 1, 3, context.DeadlineExceeded)
|
||||||
|
callback(0, 100)
|
||||||
|
callback(10, 100)
|
||||||
|
if got := task.Items()[0].Uploaded; got != 10 {
|
||||||
|
t.Fatalf("retry did not reset item progress: got %d, want 10", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
recorder.mu.Lock()
|
||||||
|
defer recorder.mu.Unlock()
|
||||||
|
for index := 1; index < len(recorder.notifications); index++ {
|
||||||
|
if recorder.notifications[index] < recorder.notifications[index-1] {
|
||||||
|
t.Fatalf("aggregate progress regressed: %v", recorder.notifications)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgressNotificationsRemainOrdered(t *testing.T) {
|
||||||
|
recorder := &orderedProgressRegressionRecorder{
|
||||||
|
firstEntered: make(chan struct{}),
|
||||||
|
releaseFirst: make(chan struct{}),
|
||||||
|
secondEntered: make(chan struct{}),
|
||||||
|
}
|
||||||
|
task := newProgressRegressionTask(recorder,
|
||||||
|
progressRegressionFile{"first", 100},
|
||||||
|
progressRegressionFile{"second", 100},
|
||||||
|
)
|
||||||
|
task.recordDownloadComplete("first", 100)
|
||||||
|
task.recordDownloadComplete("second", 100)
|
||||||
|
first := task.uploadCallback(t.Context(), "first")
|
||||||
|
second := task.uploadCallback(t.Context(), "second")
|
||||||
|
|
||||||
|
var wait sync.WaitGroup
|
||||||
|
wait.Go(func() {
|
||||||
|
first(100, 100)
|
||||||
|
})
|
||||||
|
<-recorder.firstEntered
|
||||||
|
wait.Go(func() {
|
||||||
|
second(100, 100)
|
||||||
|
})
|
||||||
|
|
||||||
|
overtook := false
|
||||||
|
select {
|
||||||
|
case <-recorder.secondEntered:
|
||||||
|
overtook = true
|
||||||
|
case <-time.After(100 * time.Millisecond):
|
||||||
|
}
|
||||||
|
close(recorder.releaseFirst)
|
||||||
|
wait.Wait()
|
||||||
|
if overtook {
|
||||||
|
t.Fatal("later aggregate notification overtook the first callback")
|
||||||
|
}
|
||||||
|
|
||||||
|
recorder.mu.Lock()
|
||||||
|
defer recorder.mu.Unlock()
|
||||||
|
if got := recorder.notifications; len(got) != 2 || got[0] != 100 || got[1] != 200 {
|
||||||
|
t.Fatalf("upload notifications = %v, want [100 200]", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchUploadUsesActualSizeWhenMetadataIsUnknown(t *testing.T) {
|
||||||
|
recorder := new(progressRegressionRecorder)
|
||||||
|
task := newProgressRegressionTask(recorder, progressRegressionFile{"photo", 0})
|
||||||
|
task.recordDownloadComplete("photo", 25)
|
||||||
|
task.uploadCallback(t.Context(), "photo")(25, 25)
|
||||||
|
|
||||||
|
recorder.mu.Lock()
|
||||||
|
defer recorder.mu.Unlock()
|
||||||
|
if recorder.startTotal != 25 {
|
||||||
|
t.Fatalf("upload start total = %d, want actual size 25", recorder.startTotal)
|
||||||
|
}
|
||||||
|
if got := task.ActualTotalSize(); got != 25 {
|
||||||
|
t.Fatalf("actual total size = %d, want 25", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type progressRegressionFile struct {
|
||||||
|
id string
|
||||||
|
size int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func newProgressRegressionTask(progress ProgressTracker, files ...progressRegressionFile) *Task {
|
||||||
|
elems := make([]TaskElement, 0, len(files))
|
||||||
|
for _, file := range files {
|
||||||
|
elems = append(elems, TaskElement{
|
||||||
|
ID: file.id,
|
||||||
|
File: tfile.NewTGFile(nil, nil, file.size, file.id+".bin"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return NewBatchTGFileTask("progress-regression", context.Background(), elems, progress, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func useProgressRegressionLocale(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
i18n.Init("zh-Hans")
|
||||||
|
t.Cleanup(func() { i18n.Init("zh-Hans") })
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertProgressRegressionContains(t *testing.T, value string, wants ...string) {
|
||||||
|
t.Helper()
|
||||||
|
for _, want := range wants {
|
||||||
|
if !strings.Contains(value, want) {
|
||||||
|
t.Fatalf("text does not contain %q:\n%s", want, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func batchEntityCounts(entities []tg.MessageEntityClass) (bold, code, blockquote, italic int) {
|
||||||
|
for _, messageEntity := range entities {
|
||||||
|
switch messageEntity.(type) {
|
||||||
|
case *tg.MessageEntityBold:
|
||||||
|
bold++
|
||||||
|
case *tg.MessageEntityCode:
|
||||||
|
code++
|
||||||
|
case *tg.MessageEntityBlockquote:
|
||||||
|
blockquote++
|
||||||
|
case *tg.MessageEntityItalic:
|
||||||
|
italic++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/core"
|
"github.com/krau/SaveAny-Bot/core"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
|
||||||
@@ -18,25 +19,34 @@ import (
|
|||||||
var _ core.Executable = (*Task)(nil)
|
var _ core.Executable = (*Task)(nil)
|
||||||
|
|
||||||
type TaskElement struct {
|
type TaskElement struct {
|
||||||
ID string
|
ID string
|
||||||
Storage storage.Storage
|
Storage storage.Storage
|
||||||
Path string
|
Path string
|
||||||
File tfile.TGFile
|
File tfile.TGFile
|
||||||
localPath string
|
localPath string
|
||||||
stream bool
|
stream bool
|
||||||
|
sourceGroupKey string
|
||||||
|
sourceCaption string
|
||||||
|
preserveCaption bool
|
||||||
}
|
}
|
||||||
|
|
||||||
type Task struct {
|
type Task struct {
|
||||||
ID string
|
ID string
|
||||||
ctx context.Context
|
ctx context.Context
|
||||||
elems []TaskElement
|
elems []TaskElement
|
||||||
Progress ProgressTracker
|
Progress ProgressTracker
|
||||||
IgnoreErrors bool // if true, errors during processing will be ignored
|
IgnoreErrors bool // if true, errors during processing will be ignored
|
||||||
downloaded atomic.Int64
|
downloaded atomic.Int64
|
||||||
totalSize int64
|
totalSize int64
|
||||||
processing map[string]TaskElementInfo
|
uploadTotalSize atomic.Int64
|
||||||
processingMu sync.RWMutex
|
processing map[string]TaskElementInfo
|
||||||
failed map[string]error // [TODO] errors for each element
|
processingMu sync.RWMutex
|
||||||
|
itemStates []itemProgressState
|
||||||
|
itemIndex map[string]int
|
||||||
|
itemMu sync.RWMutex
|
||||||
|
uploadOnce sync.Once
|
||||||
|
uploadMu sync.Mutex
|
||||||
|
uploaded map[string]int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// Title implements core.Exectable.
|
// Title implements core.Exectable.
|
||||||
@@ -54,6 +64,7 @@ func NewTaskElement(
|
|||||||
file tfile.TGFile,
|
file tfile.TGFile,
|
||||||
) (*TaskElement, error) {
|
) (*TaskElement, error) {
|
||||||
id := xid.New().String()
|
id := xid.New().String()
|
||||||
|
groupKey, caption, preserveCaption := sourceMetadata(file)
|
||||||
_, ok := stor.(storage.StorageCannotStream)
|
_, ok := stor.(storage.StorageCannotStream)
|
||||||
if !config.C().Stream || ok {
|
if !config.C().Stream || ok {
|
||||||
cachePath, err := filepath.Abs(filepath.Join(config.C().Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
|
cachePath, err := filepath.Abs(filepath.Join(config.C().Temp.BasePath, fmt.Sprintf("%s_%s", id, file.Name())))
|
||||||
@@ -61,22 +72,42 @@ func NewTaskElement(
|
|||||||
return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
|
return nil, fmt.Errorf("failed to get absolute path for cache: %w", err)
|
||||||
}
|
}
|
||||||
return &TaskElement{
|
return &TaskElement{
|
||||||
ID: id,
|
ID: id,
|
||||||
Storage: stor,
|
Storage: stor,
|
||||||
Path: path,
|
Path: path,
|
||||||
File: file,
|
File: file,
|
||||||
localPath: cachePath,
|
localPath: cachePath,
|
||||||
|
sourceGroupKey: groupKey,
|
||||||
|
sourceCaption: caption,
|
||||||
|
preserveCaption: preserveCaption,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
return &TaskElement{
|
return &TaskElement{
|
||||||
ID: id,
|
ID: id,
|
||||||
Storage: stor,
|
Storage: stor,
|
||||||
Path: path,
|
Path: path,
|
||||||
File: file,
|
File: file,
|
||||||
stream: true,
|
stream: true,
|
||||||
|
sourceGroupKey: groupKey,
|
||||||
|
sourceCaption: caption,
|
||||||
|
preserveCaption: preserveCaption,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sourceMetadata(file tfile.TGFile) (groupKey, caption string, preserveCaption bool) {
|
||||||
|
messageFile, ok := file.(tfile.TGFileMessage)
|
||||||
|
if !ok || messageFile.Message() == nil {
|
||||||
|
return "", "", false
|
||||||
|
}
|
||||||
|
msg := messageFile.Message()
|
||||||
|
groupID, grouped := msg.GetGroupedID()
|
||||||
|
if !grouped || groupID == 0 {
|
||||||
|
return "", "", false
|
||||||
|
}
|
||||||
|
chatID := tgutil.ChatIdFromPeer(msg.GetPeerID())
|
||||||
|
return fmt.Sprintf("%T:%d:%d", msg.GetPeerID(), chatID, groupID), msg.GetMessage(), true
|
||||||
|
}
|
||||||
|
|
||||||
func NewBatchTGFileTask(
|
func NewBatchTGFileTask(
|
||||||
id string,
|
id string,
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
@@ -84,6 +115,7 @@ func NewBatchTGFileTask(
|
|||||||
progress ProgressTracker,
|
progress ProgressTracker,
|
||||||
ignoreErrors bool,
|
ignoreErrors bool,
|
||||||
) *Task {
|
) *Task {
|
||||||
|
itemStates, itemIndex := newItemProgressStates(files)
|
||||||
task := &Task{
|
task := &Task{
|
||||||
ID: id,
|
ID: id,
|
||||||
ctx: ctx,
|
ctx: ctx,
|
||||||
@@ -98,9 +130,11 @@ func NewBatchTGFileTask(
|
|||||||
return total
|
return total
|
||||||
}(),
|
}(),
|
||||||
processing: make(map[string]TaskElementInfo),
|
processing: make(map[string]TaskElementInfo),
|
||||||
|
itemStates: itemStates,
|
||||||
|
itemIndex: itemIndex,
|
||||||
|
uploaded: make(map[string]int64),
|
||||||
IgnoreErrors: ignoreErrors,
|
IgnoreErrors: ignoreErrors,
|
||||||
processingMu: sync.RWMutex{},
|
processingMu: sync.RWMutex{},
|
||||||
failed: make(map[string]error),
|
|
||||||
}
|
}
|
||||||
return task
|
return task
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,8 +27,10 @@ type TaskInfo interface {
|
|||||||
TaskID() string
|
TaskID() string
|
||||||
TotalSize() int64
|
TotalSize() int64
|
||||||
Downloaded() int64
|
Downloaded() int64
|
||||||
|
ActualTotalSize() int64
|
||||||
Count() int
|
Count() int
|
||||||
Processing() []TaskElementInfo
|
Processing() []TaskElementInfo
|
||||||
|
Items() []TaskItemProgress
|
||||||
}
|
}
|
||||||
|
|
||||||
func (t *Task) TaskID() string {
|
func (t *Task) TaskID() string {
|
||||||
|
|||||||
70
core/tasks/batchtfile/upload_progress.go
Normal file
70
core/tasks/batchtfile/upload_progress.go
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
package batchtfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UploadProgressTracker optionally extends a batch progress tracker with a
|
||||||
|
// distinct aggregate upload phase.
|
||||||
|
type UploadProgressTracker interface {
|
||||||
|
OnUploadStart(ctx context.Context, info TaskInfo, total int64)
|
||||||
|
OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) startUpload(ctx context.Context) {
|
||||||
|
tracker, ok := t.Progress.(UploadProgressTracker)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.uploadMu.Lock()
|
||||||
|
defer t.uploadMu.Unlock()
|
||||||
|
t.uploadOnce.Do(func() {
|
||||||
|
tracker.OnUploadStart(ctx, t, t.uploadTotalSize.Load())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) uploadCallback(ctx context.Context, id string) func(uploaded, total int64) {
|
||||||
|
return func(uploaded, total int64) {
|
||||||
|
tracker, ok := t.Progress.(UploadProgressTracker)
|
||||||
|
if !ok || uploaded < 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.startUpload(ctx)
|
||||||
|
if total > 0 && uploaded > total {
|
||||||
|
uploaded = total
|
||||||
|
}
|
||||||
|
|
||||||
|
t.uploadMu.Lock()
|
||||||
|
defer t.uploadMu.Unlock()
|
||||||
|
becameConfirming := t.recordItemUpload(id, uploaded, total, time.Now())
|
||||||
|
if t.uploaded == nil {
|
||||||
|
t.uploaded = make(map[string]int64)
|
||||||
|
}
|
||||||
|
previous, tracked := t.uploaded[id]
|
||||||
|
if !tracked || uploaded > previous {
|
||||||
|
t.uploaded[id] = uploaded
|
||||||
|
}
|
||||||
|
var aggregate int64
|
||||||
|
for _, current := range t.uploaded {
|
||||||
|
aggregate += current
|
||||||
|
}
|
||||||
|
uploadTotal := t.uploadTotalSize.Load()
|
||||||
|
if aggregate > uploadTotal {
|
||||||
|
aggregate = uploadTotal
|
||||||
|
}
|
||||||
|
tracker.OnUploadProgress(ctx, t, aggregate, uploadTotal)
|
||||||
|
if becameConfirming {
|
||||||
|
t.notifyStateChange(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Task) recordDownloadComplete(id string, uploadSize int64) {
|
||||||
|
t.uploadMu.Lock()
|
||||||
|
defer t.uploadMu.Unlock()
|
||||||
|
if uploadSize > 0 {
|
||||||
|
t.uploadTotalSize.Add(uploadSize)
|
||||||
|
}
|
||||||
|
t.recordItemDownloaded(id, uploadSize)
|
||||||
|
}
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
package batchtfile
|
|
||||||
|
|
||||||
var progressUpdatesLevels = []struct {
|
|
||||||
size int64 // 文件大小阈值
|
|
||||||
stepPercent int // 每多少 % 更新一次
|
|
||||||
}{
|
|
||||||
{10 << 20, 100},
|
|
||||||
{50 << 20, 20},
|
|
||||||
{200 << 20, 10},
|
|
||||||
{500 << 20, 5},
|
|
||||||
}
|
|
||||||
|
|
||||||
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
|
|
||||||
if total <= 0 || downloaded <= 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
percent := int((downloaded * 100) / total)
|
|
||||||
if percent <= lastUpdatePercent {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
|
|
||||||
for _, lvl := range progressUpdatesLevels {
|
|
||||||
if total < lvl.size {
|
|
||||||
step = lvl.stepPercent
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return percent >= lastUpdatePercent+step
|
|
||||||
}
|
|
||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
|
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -75,12 +76,11 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
eg.SetLimit(config.C().Workers)
|
eg.SetLimit(config.C().Workers)
|
||||||
for _, file := range t.files {
|
for _, file := range t.files {
|
||||||
eg.Go(func() error {
|
eg.Go(func() error {
|
||||||
t.processingMu.RLock()
|
t.processingMu.Lock()
|
||||||
if _, ok := t.processing[file.URL]; ok {
|
if _, ok := t.processing[file.URL]; ok {
|
||||||
|
t.processingMu.Unlock()
|
||||||
return fmt.Errorf("file %s is already being processed", file.URL)
|
return fmt.Errorf("file %s is already being processed", file.URL)
|
||||||
}
|
}
|
||||||
t.processingMu.RUnlock()
|
|
||||||
t.processingMu.Lock()
|
|
||||||
t.processing[file.URL] = file
|
t.processing[file.URL] = file
|
||||||
t.processingMu.Unlock()
|
t.processingMu.Unlock()
|
||||||
defer func() {
|
defer func() {
|
||||||
@@ -89,7 +89,6 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
t.processingMu.Unlock()
|
t.processingMu.Unlock()
|
||||||
}()
|
}()
|
||||||
err := t.processLink(gctx, file)
|
err := t.processLink(gctx, file)
|
||||||
t.downloaded.Add(1)
|
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
logger.Debug("Link processing canceled")
|
logger.Debug("Link processing canceled")
|
||||||
return err
|
return err
|
||||||
@@ -98,6 +97,7 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
logger.Errorf("Error processing link %s: %v", file.URL, err)
|
logger.Errorf("Error processing link %s: %v", file.URL, err)
|
||||||
return fmt.Errorf("failed to process link %s: %w", file.URL, err)
|
return fmt.Errorf("failed to process link %s: %w", file.URL, err)
|
||||||
}
|
}
|
||||||
|
t.downloaded.Add(1)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -143,10 +143,16 @@ func (t *Task) processLink(ctx context.Context, file *File) error {
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
|
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
|
||||||
t.downloadedBytes.Add(int64(n))
|
downloaded := t.downloadedBytes.Add(int64(n))
|
||||||
if t.Progress != nil {
|
if t.Progress != nil {
|
||||||
t.Progress.OnProgress(ctx, t)
|
t.Progress.OnProgress(ctx, t)
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalBytes,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
copyResultCh := make(chan error, 1)
|
copyResultCh := make(chan error, 1)
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -102,7 +103,7 @@ func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
|||||||
|
|
||||||
// OnProgress implements ProgressTracker.
|
// OnProgress implements ProgressTracker.
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
||||||
if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
|
if !progressutil.ShouldUpdate(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
|
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
|
||||||
@@ -115,7 +116,10 @@ func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
|||||||
var entities []tg.MessageEntityClass
|
var entities []tg.MessageEntityClass
|
||||||
if err := styling.Perform(&entityBuilder,
|
if err := styling.Perform(&entityBuilder,
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
|
||||||
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalFiles())),
|
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithFiles, map[string]any{
|
||||||
|
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
|
||||||
|
"Count": info.TotalFiles(),
|
||||||
|
})),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
|
||||||
func() styling.StyledTextOption {
|
func() styling.StyledTextOption {
|
||||||
var lines []string
|
var lines []string
|
||||||
|
|||||||
@@ -45,7 +45,6 @@ type Task struct {
|
|||||||
downloaded atomic.Int64 // downloaded files count
|
downloaded atomic.Int64 // downloaded files count
|
||||||
processing map[string]*File // {"url": File}
|
processing map[string]*File // {"url": File}
|
||||||
processingMu sync.RWMutex
|
processingMu sync.RWMutex
|
||||||
failed map[string]error // [TODO] errors for each file
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Title implements core.Exectable.
|
// Title implements core.Exectable.
|
||||||
@@ -127,7 +126,6 @@ func NewTask(
|
|||||||
client: http.DefaultClient,
|
client: http.DefaultClient,
|
||||||
processing: make(map[string]*File),
|
processing: make(map[string]*File),
|
||||||
processingMu: sync.RWMutex{},
|
processingMu: sync.RWMutex{},
|
||||||
failed: make(map[string]error),
|
|
||||||
totalFiles: int64(len(files)),
|
totalFiles: int64(len(files)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -207,34 +207,3 @@ func parseFilenameFallback(cd string) string {
|
|||||||
|
|
||||||
return decodeFilenameParam(value)
|
return decodeFilenameParam(value)
|
||||||
}
|
}
|
||||||
|
|
||||||
var progressUpdatesLevels = []struct {
|
|
||||||
size int64 // 文件大小阈值
|
|
||||||
stepPercent int // 每多少 % 更新一次
|
|
||||||
}{
|
|
||||||
{10 << 20, 100},
|
|
||||||
{50 << 20, 50},
|
|
||||||
{200 << 20, 20},
|
|
||||||
{500 << 20, 10},
|
|
||||||
}
|
|
||||||
|
|
||||||
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
|
|
||||||
if total <= 0 || downloaded <= 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
percent := int((downloaded * 100) / total)
|
|
||||||
if percent <= lastUpdatePercent {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
|
|
||||||
for _, lvl := range progressUpdatesLevels {
|
|
||||||
if total < lvl.size {
|
|
||||||
step = lvl.stepPercent
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return percent >= lastUpdatePercent+step
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/parser"
|
"github.com/krau/SaveAny-Bot/pkg/parser"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,21 +30,20 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
eg.SetLimit(config.C().Workers)
|
eg.SetLimit(config.C().Workers)
|
||||||
for _, resource := range t.item.Resources {
|
for _, resource := range t.item.Resources {
|
||||||
eg.Go(func() error {
|
eg.Go(func() error {
|
||||||
t.processingMu.RLock()
|
resourceID := resource.ID()
|
||||||
if t.processing[resource.ID()] != nil {
|
|
||||||
return fmt.Errorf("resource %s is already being processed", resource.ID())
|
|
||||||
}
|
|
||||||
t.processingMu.RUnlock()
|
|
||||||
t.processingMu.Lock()
|
t.processingMu.Lock()
|
||||||
t.processing[resource.ID()] = &resource
|
if t.processing[resourceID] != nil {
|
||||||
|
t.processingMu.Unlock()
|
||||||
|
return fmt.Errorf("resource %s is already being processed", resourceID)
|
||||||
|
}
|
||||||
|
t.processing[resourceID] = &resource
|
||||||
t.processingMu.Unlock()
|
t.processingMu.Unlock()
|
||||||
defer func() {
|
defer func() {
|
||||||
t.processingMu.Lock()
|
t.processingMu.Lock()
|
||||||
delete(t.processing, resource.URL)
|
delete(t.processing, resourceID)
|
||||||
t.processingMu.Unlock()
|
t.processingMu.Unlock()
|
||||||
}()
|
}()
|
||||||
err := t.processResource(gctx, resource)
|
err := t.processResource(gctx, resource)
|
||||||
t.downloaded.Add(1)
|
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
logger.Debug("Resource processing canceled")
|
logger.Debug("Resource processing canceled")
|
||||||
return err
|
return err
|
||||||
@@ -52,6 +52,7 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
logger.Errorf("Error processing resource %s: %v", resource.URL, err)
|
logger.Errorf("Error processing resource %s: %v", resource.URL, err)
|
||||||
return fmt.Errorf("failed to process resource %s: %w", resource.URL, err)
|
return fmt.Errorf("failed to process resource %s: %w", resource.URL, err)
|
||||||
}
|
}
|
||||||
|
t.downloaded.Add(1)
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -107,10 +108,16 @@ func (t *Task) processResource(ctx context.Context, resource parser.Resource) er
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
|
wr := ioutil.NewProgressWriter(cacheFile, func(n int) {
|
||||||
t.downloadedBytes.Add(int64(n))
|
downloaded := t.downloadedBytes.Add(int64(n))
|
||||||
if t.progress != nil {
|
if t.progress != nil {
|
||||||
t.progress.OnProgress(ctx, t)
|
t.progress.OnProgress(ctx, t)
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalBytes,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
copyResultCh := make(chan error, 1)
|
copyResultCh := make(chan error, 1)
|
||||||
|
|||||||
@@ -15,40 +15,10 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
var progressUpdatesLevels = []struct {
|
|
||||||
size int64 // 文件大小阈值
|
|
||||||
stepPercent int // 每多少 % 更新一次
|
|
||||||
}{
|
|
||||||
{10 << 20, 100},
|
|
||||||
{50 << 20, 50},
|
|
||||||
{200 << 20, 20},
|
|
||||||
{500 << 20, 10},
|
|
||||||
}
|
|
||||||
|
|
||||||
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
|
|
||||||
if total <= 0 || downloaded <= 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
percent := int((downloaded * 100) / total)
|
|
||||||
if percent <= lastUpdatePercent {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
|
|
||||||
for _, lvl := range progressUpdatesLevels {
|
|
||||||
if total < lvl.size {
|
|
||||||
step = lvl.stepPercent
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return percent >= lastUpdatePercent+step
|
|
||||||
}
|
|
||||||
|
|
||||||
type ProgressTracker interface {
|
type ProgressTracker interface {
|
||||||
OnStart(ctx context.Context, info TaskInfo)
|
OnStart(ctx context.Context, info TaskInfo)
|
||||||
OnProgress(ctx context.Context, info TaskInfo)
|
OnProgress(ctx context.Context, info TaskInfo)
|
||||||
@@ -73,7 +43,10 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
|||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressParsedStartPrefix, map[string]any{
|
styling.Plain(i18n.T(i18nk.BotMsgProgressParsedStartPrefix, map[string]any{
|
||||||
"Site": info.Site(),
|
"Site": info.Site(),
|
||||||
})),
|
})),
|
||||||
styling.Code(fmt.Sprintf("%.2f MB (%d个资源)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
|
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithResources, map[string]any{
|
||||||
|
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
|
||||||
|
"Count": info.TotalResources(),
|
||||||
|
})),
|
||||||
); err != nil {
|
); err != nil {
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
||||||
return
|
return
|
||||||
@@ -101,7 +74,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
||||||
if !shouldUpdateProgress(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
|
if !progressutil.ShouldUpdate(info.TotalBytes(), info.DownloadedBytes(), int(p.lastUpdatePercent.Load())) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
|
percent := int((info.DownloadedBytes() * 100) / info.TotalBytes())
|
||||||
@@ -114,7 +87,10 @@ func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
|||||||
var entities []tg.MessageEntityClass
|
var entities []tg.MessageEntityClass
|
||||||
if err := styling.Perform(&entityBuilder,
|
if err := styling.Perform(&entityBuilder,
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadingPrefix, nil)),
|
||||||
styling.Code(fmt.Sprintf("%.2f MB (%d个文件)", float64(info.TotalBytes())/(1024*1024), info.TotalResources())),
|
styling.Code(i18n.T(i18nk.BotMsgProgressSizeWithResources, map[string]any{
|
||||||
|
"Size": fmt.Sprintf("%.2f MB", float64(info.TotalBytes())/(1024*1024)),
|
||||||
|
"Count": info.TotalResources(),
|
||||||
|
})),
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
|
styling.Plain(i18n.T(i18nk.BotMsgProgressProcessingListPrefix, nil)),
|
||||||
func() styling.StyledTextOption {
|
func() styling.StyledTextOption {
|
||||||
var lines []string
|
var lines []string
|
||||||
|
|||||||
@@ -33,7 +33,6 @@ type Task struct {
|
|||||||
downloadedBytes atomic.Int64 // downloaded bytes count
|
downloadedBytes atomic.Int64 // downloaded bytes count
|
||||||
processing map[string]ResourceInfo
|
processing map[string]ResourceInfo
|
||||||
processingMu sync.RWMutex
|
processingMu sync.RWMutex
|
||||||
failed map[string]error // [TODO] errors for each resource
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Title implements core.Exectable.
|
// Title implements core.Exectable.
|
||||||
@@ -84,6 +83,5 @@ func NewTask(
|
|||||||
progress: progressTracker,
|
progress: progressTracker,
|
||||||
processing: make(map[string]ResourceInfo),
|
processing: make(map[string]ResourceInfo),
|
||||||
processingMu: sync.RWMutex{},
|
processingMu: sync.RWMutex{},
|
||||||
failed: make(map[string]error),
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,13 +11,16 @@ import (
|
|||||||
"github.com/duke-git/lancet/v2/retry"
|
"github.com/duke-git/lancet/v2/retry"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
|
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
)
|
)
|
||||||
|
|
||||||
func (t *Task) Execute(ctx context.Context) error {
|
func (t *Task) Execute(ctx context.Context) error {
|
||||||
logger := log.FromContext(ctx)
|
logger := log.FromContext(ctx)
|
||||||
logger.Infof("Starting Telegraph task %s", t.PhPath)
|
logger.Infof("Starting Telegraph task %s", t.PhPath)
|
||||||
t.progress.OnStart(ctx, t)
|
if t.progress != nil {
|
||||||
|
t.progress.OnStart(ctx, t)
|
||||||
|
}
|
||||||
eg, gctx := errgroup.WithContext(ctx)
|
eg, gctx := errgroup.WithContext(ctx)
|
||||||
eg.SetLimit(config.C().Workers)
|
eg.SetLimit(config.C().Workers)
|
||||||
for i, pic := range t.Pics {
|
for i, pic := range t.Pics {
|
||||||
@@ -27,8 +30,16 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
logger.Errorf("Error processing picture %s: %v", pic, err)
|
logger.Errorf("Error processing picture %s: %v", pic, err)
|
||||||
return fmt.Errorf("failed to process picture %s: %w", pic, err)
|
return fmt.Errorf("failed to process picture %s: %w", pic, err)
|
||||||
}
|
}
|
||||||
t.downloaded.Add(1)
|
downloaded := t.downloaded.Add(1)
|
||||||
t.progress.OnProgress(gctx, t)
|
if t.progress != nil {
|
||||||
|
t.progress.OnProgress(gctx, t)
|
||||||
|
}
|
||||||
|
taskevent.Emit(gctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalFiles: t.totalpics,
|
||||||
|
DownloadedFiles: int(downloaded),
|
||||||
|
})
|
||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -38,7 +49,9 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
} else {
|
} else {
|
||||||
logger.Infof("Telegraph task %s completed successfully", t.PhPath)
|
logger.Infof("Telegraph task %s completed successfully", t.PhPath)
|
||||||
}
|
}
|
||||||
t.progress.OnDone(ctx, t, err)
|
if t.progress != nil {
|
||||||
|
t.progress.OnDone(ctx, t, err)
|
||||||
|
}
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
31
core/tasks/telegraph/nil_tracker_test.go
Normal file
31
core/tasks/telegraph/nil_tracker_test.go
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
package telegraph_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
storconfig "github.com/krau/SaveAny-Bot/config/storage"
|
||||||
|
"github.com/krau/SaveAny-Bot/core/tasks/telegraph"
|
||||||
|
storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type mockStorage struct{}
|
||||||
|
|
||||||
|
func (mockStorage) Init(context.Context, storconfig.StorageConfig) error { return nil }
|
||||||
|
func (mockStorage) Type() storenum.StorageType { return storenum.Local }
|
||||||
|
func (mockStorage) Name() string { return "mock" }
|
||||||
|
func (mockStorage) Save(context.Context, io.Reader, string) error { return nil }
|
||||||
|
func (mockStorage) Exists(context.Context, string) bool { return false }
|
||||||
|
|
||||||
|
var _ storage.Storage = mockStorage{}
|
||||||
|
|
||||||
|
// Regression: API-created tasks run with a nil ProgressTracker (progress goes
|
||||||
|
// through taskevent); Execute must not panic on the tracker callbacks.
|
||||||
|
func TestExecuteWithNilTracker(t *testing.T) {
|
||||||
|
task := telegraph.NewTask("id", t.Context(), "/page", nil, mockStorage{}, "/out", nil, nil)
|
||||||
|
if err := task.Execute(t.Context()); err != nil {
|
||||||
|
t.Fatalf("Execute returned error: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"github.com/gotd/td/tg"
|
"github.com/gotd/td/tg"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -60,7 +61,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
||||||
if !shouldUpdateProgress(info.Downloaded(), int64(info.TotalPics())) {
|
if !progressutil.ShouldUpdateCount(info.Downloaded(), int64(info.TotalPics())) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.Downloaded(), info.TotalPics())
|
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.TaskID(), info.Downloaded(), info.TotalPics())
|
||||||
|
|||||||
@@ -1,13 +0,0 @@
|
|||||||
package telegraph
|
|
||||||
|
|
||||||
func shouldUpdateProgress(downloaded int64, total int64) bool {
|
|
||||||
if total <= 0 || downloaded <= 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
step := int64(10)
|
|
||||||
if downloaded < step {
|
|
||||||
return downloaded == total
|
|
||||||
}
|
|
||||||
return downloaded%step == 0 || downloaded == total
|
|
||||||
}
|
|
||||||
42
core/tasks/tfile/caption_test.go
Normal file
42
core/tasks/tfile/caption_test.go
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
package tfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
tfilepkg "github.com/krau/SaveAny-Bot/pkg/tfile"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSourceCaption(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
file tfilepkg.TGFile
|
||||||
|
want string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "original caption",
|
||||||
|
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov", tfilepkg.WithMessage(&tg.Message{Message: "original caption"})),
|
||||||
|
want: "original caption",
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty caption suppresses storage fallback",
|
||||||
|
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov", tfilepkg.WithMessage(&tg.Message{})),
|
||||||
|
ok: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "file without source message",
|
||||||
|
file: tfilepkg.NewTGFile(nil, nil, 0, "video.mov"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, ok := sourceCaption(tt.file)
|
||||||
|
if ok != tt.ok || got != tt.want {
|
||||||
|
t.Fatalf("sourceCaption() = (%q, %v), want (%q, %v)", got, ok, tt.want, tt.ok)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package tfile
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
|
|
||||||
@@ -10,8 +11,12 @@ import (
|
|||||||
"github.com/duke-git/lancet/v2/retry"
|
"github.com/duke-git/lancet/v2/retry"
|
||||||
"github.com/krau/SaveAny-Bot/common/tdler"
|
"github.com/krau/SaveAny-Bot/common/tdler"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
|
"github.com/krau/SaveAny-Bot/common/utils/fsutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/ioutil"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
|
||||||
|
tfilepkg "github.com/krau/SaveAny-Bot/pkg/tfile"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
)
|
)
|
||||||
|
|
||||||
func (t *Task) Execute(ctx context.Context) error {
|
func (t *Task) Execute(ctx context.Context) error {
|
||||||
@@ -57,13 +62,34 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
return fmt.Errorf("failed to get file stat: %w", err)
|
return fmt.Errorf("failed to get file stat: %w", err)
|
||||||
}
|
}
|
||||||
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
|
vctx := context.WithValue(ctx, ctxkey.ContentLength, fileStat.Size())
|
||||||
|
if caption, ok := sourceCaption(t.File); ok {
|
||||||
|
vctx = storagetypes.WithSourceCaption(vctx, caption)
|
||||||
|
}
|
||||||
err = retry.Retry(func() error {
|
err = retry.Retry(func() error {
|
||||||
file, err := os.Open(t.localPath)
|
file, err := os.Open(t.localPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("failed to open cache file: %w", err)
|
return fmt.Errorf("failed to open cache file: %w", err)
|
||||||
}
|
}
|
||||||
defer file.Close()
|
defer file.Close()
|
||||||
if err = t.Storage.Save(vctx, file, t.Path); err != nil {
|
uploadProgress, tracksUpload := t.Progress.(UploadProgressTracker)
|
||||||
|
if !tracksUpload {
|
||||||
|
if err = t.Storage.Save(vctx, file, t.Path); err != nil {
|
||||||
|
return fmt.Errorf("failed to save file: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
uploadProgress.OnUploadStart(vctx, t, fileStat.Size())
|
||||||
|
onProgress := func(uploaded, total int64) {
|
||||||
|
uploadProgress.OnUploadProgress(vctx, t, uploaded, total)
|
||||||
|
}
|
||||||
|
if progressSaver, ok := t.Storage.(storage.StorageProgressSaver); ok {
|
||||||
|
err = progressSaver.SaveWithProgress(vctx, file, t.Path, onProgress)
|
||||||
|
} else {
|
||||||
|
var reader io.Reader = ioutil.NewProgressReader(file, fileStat.Size(), onProgress)
|
||||||
|
err = t.Storage.Save(vctx, reader, t.Path)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
return fmt.Errorf("failed to save file: %w", err)
|
return fmt.Errorf("failed to save file: %w", err)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -73,3 +99,11 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sourceCaption(file tfilepkg.TGFile) (string, bool) {
|
||||||
|
messageFile, ok := file.(tfilepkg.TGFileMessage)
|
||||||
|
if !ok || messageFile.Message() == nil {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return messageFile.Message().GetMessage(), true
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,16 +4,17 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/gotd/td/telegram/message/entity"
|
|
||||||
"github.com/gotd/td/telegram/message/styling"
|
|
||||||
"github.com/gotd/td/tg"
|
"github.com/gotd/td/tg"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -23,153 +24,319 @@ type ProgressTracker interface {
|
|||||||
OnDone(ctx context.Context, info TaskInfo, err error)
|
OnDone(ctx context.Context, info TaskInfo, err error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UploadProgressTracker optionally extends a task progress tracker with a
|
||||||
|
// distinct upload phase. Keeping it separate preserves compatibility with
|
||||||
|
// custom download-only trackers.
|
||||||
|
type UploadProgressTracker interface {
|
||||||
|
OnUploadStart(ctx context.Context, info TaskInfo, total int64)
|
||||||
|
OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64)
|
||||||
|
}
|
||||||
|
|
||||||
type Progress struct {
|
type Progress struct {
|
||||||
MessageID int
|
MessageID int
|
||||||
ChatID int64
|
ChatID int64
|
||||||
start time.Time
|
start time.Time
|
||||||
lastUpdatePercent atomic.Int32
|
lastUpdatePercent atomic.Int32
|
||||||
|
lastUpdateAt atomic.Int64
|
||||||
|
updateMu sync.Mutex
|
||||||
|
uploadAttempt int
|
||||||
|
uploadedBytes int64
|
||||||
|
actualSize int64
|
||||||
|
hasActualSize bool
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
uploadProgressMinInterval = time.Second
|
||||||
|
uploadProgressMaxInterval = 3 * time.Second
|
||||||
|
singleProgressBarWidth = 10
|
||||||
|
maxSingleErrorRunes = 240
|
||||||
|
)
|
||||||
|
|
||||||
|
type singleProgressPhase int
|
||||||
|
|
||||||
|
const (
|
||||||
|
singlePhaseDownloading singleProgressPhase = iota
|
||||||
|
singlePhaseUploading
|
||||||
|
singlePhaseRetrying
|
||||||
|
)
|
||||||
|
|
||||||
|
type renderedSingleMessage struct {
|
||||||
|
Text string
|
||||||
|
Entities []tg.MessageEntityClass
|
||||||
|
Err error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
||||||
|
p.updateMu.Lock()
|
||||||
|
defer p.updateMu.Unlock()
|
||||||
p.start = time.Now()
|
p.start = time.Now()
|
||||||
p.lastUpdatePercent.Store(0)
|
p.lastUpdatePercent.Store(0)
|
||||||
|
p.lastUpdateAt.Store(0)
|
||||||
|
p.uploadAttempt = 0
|
||||||
|
p.uploadedBytes = 0
|
||||||
|
p.actualSize = 0
|
||||||
|
p.hasActualSize = false
|
||||||
log.FromContext(ctx).Debugf("Progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
|
log.FromContext(ctx).Debugf("Progress tracking started for message %d in chat %d", p.MessageID, p.ChatID)
|
||||||
entityBuilder := entity.Builder{}
|
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(info, singlePhaseDownloading, 0, info.FileSize(), 0, 0), true)
|
||||||
var entities []tg.MessageEntityClass
|
|
||||||
if err := styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressFileStartPrefix, nil)),
|
|
||||||
styling.Code(info.FileName()),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressFileSizePrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("%.2f MB", float64(info.FileSize())/(1024*1024))),
|
|
||||||
); err != nil {
|
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
text, entities := entityBuilder.Complete()
|
|
||||||
req := &tg.MessagesEditMessageRequest{
|
|
||||||
ID: p.MessageID,
|
|
||||||
}
|
|
||||||
req.SetMessage(text)
|
|
||||||
req.SetEntities(entities)
|
|
||||||
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
|
|
||||||
Rows: []tg.KeyboardButtonRow{
|
|
||||||
{
|
|
||||||
Buttons: []tg.KeyboardButtonClass{
|
|
||||||
tgutil.BuildCancelButton(info.TaskID()),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}},
|
|
||||||
)
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
|
||||||
if ext != nil {
|
|
||||||
ext.EditMessage(p.ChatID, req)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo, downloaded, total int64) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo, downloaded, total int64) {
|
||||||
if !shouldUpdateProgress(total, downloaded, int(p.lastUpdatePercent.Load())) {
|
p.updateMu.Lock()
|
||||||
|
defer p.updateMu.Unlock()
|
||||||
|
now := time.Now()
|
||||||
|
elapsed := uploadProgressMaxInterval
|
||||||
|
if lastUpdateAt := p.lastUpdateAt.Load(); lastUpdateAt > 0 {
|
||||||
|
elapsed = now.Sub(time.Unix(0, lastUpdateAt))
|
||||||
|
}
|
||||||
|
if !shouldUpdateSingleDownloadProgress(total, downloaded, int(p.lastUpdatePercent.Load()), elapsed) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
percent := int32((downloaded * 100) / total)
|
if total > 0 {
|
||||||
if p.lastUpdatePercent.Load() == percent {
|
percent := int32((downloaded * 100) / total)
|
||||||
return
|
if p.lastUpdatePercent.Load() == percent {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.lastUpdatePercent.Store(percent)
|
||||||
}
|
}
|
||||||
p.lastUpdatePercent.Store(percent)
|
p.lastUpdateAt.Store(now.UnixNano())
|
||||||
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.FileName(), downloaded, total)
|
log.FromContext(ctx).Debugf("Progress update: %s, %d/%d", info.FileName(), downloaded, total)
|
||||||
entityBuilder := entity.Builder{}
|
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(
|
||||||
var entities []tg.MessageEntityClass
|
info,
|
||||||
if err := styling.Perform(&entityBuilder,
|
singlePhaseDownloading,
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressFileProcessingPrefix, nil)),
|
downloaded,
|
||||||
styling.Code(info.FileName()),
|
total,
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
|
dlutil.GetSpeed(downloaded, p.start),
|
||||||
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
|
0,
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressFileSizePrefix, nil)),
|
), true)
|
||||||
styling.Code(fmt.Sprintf("%.2f MB", float64(total)/(1024*1024))),
|
}
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressAvgSpeedPrefix, nil)),
|
|
||||||
styling.Bold(fmt.Sprintf("%.2f MB/s", dlutil.GetSpeed(downloaded, p.start)/(1024*1024))),
|
func shouldUpdateSingleDownloadProgress(total, downloaded int64, lastPercent int, elapsed time.Duration) bool {
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressCurrentProgressPrefix, nil)),
|
if total > 0 {
|
||||||
styling.Bold(fmt.Sprintf("%.2f%%", float64(downloaded)/float64(total)*100)),
|
return progressutil.ShouldUpdate(total, downloaded, lastPercent)
|
||||||
); err != nil {
|
}
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", err)
|
return downloaded > 0 && elapsed >= uploadProgressMaxInterval
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) OnUploadStart(ctx context.Context, info TaskInfo, total int64) {
|
||||||
|
p.updateMu.Lock()
|
||||||
|
defer p.updateMu.Unlock()
|
||||||
|
p.start = time.Now()
|
||||||
|
p.lastUpdatePercent.Store(0)
|
||||||
|
p.lastUpdateAt.Store(p.start.UnixNano())
|
||||||
|
p.uploadAttempt++
|
||||||
|
p.uploadedBytes = 0
|
||||||
|
p.actualSize = max(total, 0)
|
||||||
|
p.hasActualSize = true
|
||||||
|
log.FromContext(ctx).Debugf("Upload progress tracking started: %s", info.FileName())
|
||||||
|
phase := singleUploadPhase(p.uploadAttempt)
|
||||||
|
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(info, phase, 0, total, 0, p.uploadAttempt), true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Progress) OnUploadProgress(ctx context.Context, info TaskInfo, uploaded, total int64) {
|
||||||
|
if total <= 0 || uploaded <= 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
text, entities := entityBuilder.Complete()
|
p.updateMu.Lock()
|
||||||
req := &tg.MessagesEditMessageRequest{
|
defer p.updateMu.Unlock()
|
||||||
ID: p.MessageID,
|
if uploaded > total {
|
||||||
|
uploaded = total
|
||||||
}
|
}
|
||||||
req.SetMessage(text)
|
if uploaded < p.uploadedBytes {
|
||||||
req.SetEntities(entities)
|
return
|
||||||
req.SetReplyMarkup(&tg.ReplyInlineMarkup{
|
}
|
||||||
Rows: []tg.KeyboardButtonRow{
|
p.uploadedBytes = uploaded
|
||||||
{
|
|
||||||
Buttons: []tg.KeyboardButtonClass{
|
now := time.Now()
|
||||||
tgutil.BuildCancelButton(info.TaskID()),
|
lastUpdateAt := time.Unix(0, p.lastUpdateAt.Load())
|
||||||
},
|
lastPercent := int(p.lastUpdatePercent.Load())
|
||||||
},
|
if !shouldUpdateUploadProgress(total, uploaded, lastPercent, now.Sub(lastUpdateAt)) {
|
||||||
}},
|
|
||||||
)
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
|
||||||
if ext != nil {
|
|
||||||
ext.EditMessage(p.ChatID, req)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
percent := int32((uploaded * 100) / total)
|
||||||
|
p.lastUpdatePercent.Store(percent)
|
||||||
|
p.lastUpdateAt.Store(now.UnixNano())
|
||||||
|
log.FromContext(ctx).Debugf("Upload progress update: %s, %d/%d", info.FileName(), uploaded, total)
|
||||||
|
p.editMessage(ctx, info.TaskID(), buildSingleProgressMessage(
|
||||||
|
info,
|
||||||
|
singleUploadPhase(p.uploadAttempt),
|
||||||
|
uploaded,
|
||||||
|
total,
|
||||||
|
dlutil.GetSpeed(uploaded, p.start),
|
||||||
|
p.uploadAttempt,
|
||||||
|
), true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func shouldUpdateUploadProgress(total, uploaded int64, lastPercent int, elapsed time.Duration) bool {
|
||||||
|
if total <= 0 || uploaded <= 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if uploaded >= total {
|
||||||
|
return lastPercent < 100 && elapsed >= uploadProgressMinInterval
|
||||||
|
}
|
||||||
|
percent := int((uploaded * 100) / total)
|
||||||
|
if percent < lastPercent {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if elapsed < uploadProgressMinInterval {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if percent == lastPercent {
|
||||||
|
return elapsed >= uploadProgressMaxInterval
|
||||||
|
}
|
||||||
|
return progressutil.ShouldUpdate(total, uploaded, lastPercent) || elapsed >= uploadProgressMaxInterval
|
||||||
|
}
|
||||||
|
|
||||||
|
func singleUploadPhase(attempt int) singleProgressPhase {
|
||||||
|
if attempt > 1 {
|
||||||
|
return singlePhaseRetrying
|
||||||
|
}
|
||||||
|
return singlePhaseUploading
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
||||||
|
p.updateMu.Lock()
|
||||||
|
defer p.updateMu.Unlock()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.FromContext(ctx).Errorf("Progress error for file [%s]: %v", info.FileName(), err)
|
log.FromContext(ctx).Errorf("Progress error for file [%s]: %v", info.FileName(), err)
|
||||||
} else {
|
} else {
|
||||||
log.FromContext(ctx).Debugf("Progress done for file [%s]", info.FileName())
|
log.FromContext(ctx).Debugf("Progress done for file [%s]", info.FileName())
|
||||||
}
|
}
|
||||||
|
|
||||||
entityBuilder := entity.Builder{}
|
p.editMessage(ctx, info.TaskID(), buildSingleDoneMessage(info, p.doneSize(info), err), false)
|
||||||
var stylingErr error
|
}
|
||||||
|
|
||||||
if err != nil {
|
func (p *Progress) doneSize(info TaskInfo) int64 {
|
||||||
if errors.Is(err, context.Canceled) {
|
if p.hasActualSize {
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
return p.actualSize
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressTaskCanceled, nil)),
|
|
||||||
styling.Plain("\n"),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressFileNamePrefix, nil)),
|
|
||||||
styling.Code(info.FileName()),
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadFailedPrefix, nil)),
|
|
||||||
styling.Code(info.FileName()),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressErrorPrefix, nil)),
|
|
||||||
styling.Bold(err.Error()),
|
|
||||||
)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
stylingErr = styling.Perform(&entityBuilder,
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressDownloadDonePrefix, nil)),
|
|
||||||
styling.Code(info.FileName()),
|
|
||||||
styling.Plain(i18n.T(i18nk.BotMsgProgressSavePathPrefix, nil)),
|
|
||||||
styling.Code(fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())),
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
return max(info.FileSize(), 0)
|
||||||
|
}
|
||||||
|
|
||||||
if stylingErr != nil {
|
func (p *Progress) editMessage(ctx context.Context, taskID string, message renderedSingleMessage, cancellable bool) {
|
||||||
log.FromContext(ctx).Errorf("Failed to build entities: %s", stylingErr)
|
if message.Err != nil {
|
||||||
|
log.FromContext(ctx).Errorf("Failed to render file progress message: %v", message.Err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
req := buildSingleEditMessageRequest(p.MessageID, taskID, message, cancellable)
|
||||||
text, entities := entityBuilder.Complete()
|
if ext := tgutil.ExtFromContext(ctx); ext != nil {
|
||||||
req := &tg.MessagesEditMessageRequest{
|
if _, err := ext.EditMessage(p.ChatID, req); err != nil {
|
||||||
ID: p.MessageID,
|
log.FromContext(ctx).Errorf("Failed to edit file progress message: %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
req.SetMessage(text)
|
}
|
||||||
req.SetEntities(entities)
|
|
||||||
|
|
||||||
ext := tgutil.ExtFromContext(ctx)
|
func buildSingleEditMessageRequest(messageID int, taskID string, message renderedSingleMessage, cancellable bool) *tg.MessagesEditMessageRequest {
|
||||||
if ext != nil {
|
req := &tg.MessagesEditMessageRequest{ID: messageID}
|
||||||
ext.EditMessage(p.ChatID, req)
|
req.SetMessage(message.Text)
|
||||||
|
if len(message.Entities) > 0 {
|
||||||
|
req.SetEntities(message.Entities)
|
||||||
}
|
}
|
||||||
|
if cancellable {
|
||||||
|
req.SetReplyMarkup(&tg.ReplyInlineMarkup{Rows: []tg.KeyboardButtonRow{{
|
||||||
|
Buttons: []tg.KeyboardButtonClass{tgutil.BuildCancelButton(taskID)},
|
||||||
|
}}})
|
||||||
|
}
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSingleProgressMessage(
|
||||||
|
info TaskInfo,
|
||||||
|
phase singleProgressPhase,
|
||||||
|
current int64,
|
||||||
|
total int64,
|
||||||
|
speed float64,
|
||||||
|
attempt int,
|
||||||
|
) renderedSingleMessage {
|
||||||
|
if current < 0 {
|
||||||
|
current = 0
|
||||||
|
}
|
||||||
|
if total > 0 && current > total {
|
||||||
|
current = total
|
||||||
|
}
|
||||||
|
percent := singleProgressPercent(current, total)
|
||||||
|
destination := fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath())
|
||||||
|
data := map[string]any{
|
||||||
|
"Name": info.FileName(),
|
||||||
|
"Bar": singleProgressBar(percent),
|
||||||
|
"Progress": percent,
|
||||||
|
"Speed": singleProgressSpeed(speed),
|
||||||
|
"Current": dlutil.FormatSize(current),
|
||||||
|
"Size": dlutil.FormatSize(total),
|
||||||
|
"Destination": destination,
|
||||||
|
"Attempt": max(attempt, 1),
|
||||||
|
}
|
||||||
|
|
||||||
|
var key i18nk.Key
|
||||||
|
switch phase {
|
||||||
|
case singlePhaseUploading:
|
||||||
|
key = i18nk.BotMsgProgressSingleUploading
|
||||||
|
case singlePhaseRetrying:
|
||||||
|
key = i18nk.BotMsgProgressSingleUploadRetrying
|
||||||
|
default:
|
||||||
|
key = i18nk.BotMsgProgressSingleDownloading
|
||||||
|
if total <= 0 {
|
||||||
|
key = i18nk.BotMsgProgressSingleDownloadingUnknown
|
||||||
|
}
|
||||||
|
}
|
||||||
|
markup := i18n.T(i18nk.BotMsgProgressSingleStatusHeader, nil) + "\n\n" + localizedProgressMarkup(key, data)
|
||||||
|
return completeSingleMessage(markup)
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSingleDoneMessage(info TaskInfo, size int64, err error) renderedSingleMessage {
|
||||||
|
data := map[string]any{
|
||||||
|
"Name": info.FileName(),
|
||||||
|
"Size": dlutil.FormatSize(max(size, 0)),
|
||||||
|
"Destination": fmt.Sprintf("[%s]:%s", info.StorageName(), info.StoragePath()),
|
||||||
|
}
|
||||||
|
var key i18nk.Key
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
key = i18nk.BotMsgProgressSingleDone
|
||||||
|
case errors.Is(err, context.Canceled):
|
||||||
|
key = i18nk.BotMsgProgressSingleCanceled
|
||||||
|
default:
|
||||||
|
data["Reason"] = truncateSingleError(err.Error())
|
||||||
|
key = i18nk.BotMsgProgressSingleFailed
|
||||||
|
}
|
||||||
|
return completeSingleMessage(localizedProgressMarkup(key, data))
|
||||||
|
}
|
||||||
|
|
||||||
|
func localizedProgressMarkup(key i18nk.Key, data map[string]any) string {
|
||||||
|
return i18n.T(key, tgutil.EscapeHTMLTemplateData(data))
|
||||||
|
}
|
||||||
|
|
||||||
|
func completeSingleMessage(markup string) renderedSingleMessage {
|
||||||
|
text, entities, err := tgutil.RenderHTML(markup)
|
||||||
|
return renderedSingleMessage{Text: text, Entities: entities, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
func singleProgressPercent(current, total int64) int {
|
||||||
|
if total <= 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return int(min(max(current, 0), total) * 100 / total)
|
||||||
|
}
|
||||||
|
|
||||||
|
func singleProgressBar(percent int) string {
|
||||||
|
percent = min(max(percent, 0), 100)
|
||||||
|
filled := percent * singleProgressBarWidth / 100
|
||||||
|
return strings.Repeat("🟩", filled) + strings.Repeat("⬜️", singleProgressBarWidth-filled)
|
||||||
|
}
|
||||||
|
|
||||||
|
func singleProgressSpeed(speed float64) string {
|
||||||
|
if speed <= 0 {
|
||||||
|
return "0 B/s"
|
||||||
|
}
|
||||||
|
return dlutil.FormatSize(int64(speed)) + "/s"
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncateSingleError(value string) string {
|
||||||
|
runes := []rune(value)
|
||||||
|
if len(runes) <= maxSingleErrorRunes {
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
return string(runes[:maxSingleErrorRunes])
|
||||||
}
|
}
|
||||||
|
|
||||||
type ProgressOption func(*Progress)
|
type ProgressOption func(*Progress)
|
||||||
|
|||||||
178
core/tasks/tfile/progress_test.go
Normal file
178
core/tasks/tfile/progress_test.go
Normal file
@@ -0,0 +1,178 @@
|
|||||||
|
package tfile
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gotd/td/tg"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
|
)
|
||||||
|
|
||||||
|
type progressTestTaskInfo struct{}
|
||||||
|
|
||||||
|
func (progressTestTaskInfo) TaskID() string { return "task" }
|
||||||
|
func (progressTestTaskInfo) FileName() string { return "file.bin" }
|
||||||
|
func (progressTestTaskInfo) FileSize() int64 { return 100 << 20 }
|
||||||
|
func (progressTestTaskInfo) StoragePath() string { return "file.bin" }
|
||||||
|
func (progressTestTaskInfo) StorageName() string { return "test" }
|
||||||
|
|
||||||
|
func TestShouldUpdateUploadProgress(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
total int64
|
||||||
|
uploaded int64
|
||||||
|
lastPercent int
|
||||||
|
elapsed time.Duration
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{name: "invalid total", total: 0, uploaded: 1, want: false},
|
||||||
|
{name: "no uploaded bytes", total: 100, uploaded: 0, want: false},
|
||||||
|
{name: "percentage threshold", total: 100 << 20, uploaded: 10 << 20, elapsed: uploadProgressMinInterval, want: true},
|
||||||
|
{name: "percentage threshold rate limited", total: 100 << 20, uploaded: 10 << 20, elapsed: uploadProgressMinInterval - time.Millisecond, want: false},
|
||||||
|
{name: "maximum time threshold", total: 100 << 20, uploaded: 1 << 20, elapsed: uploadProgressMaxInterval, want: true},
|
||||||
|
{name: "below thresholds", total: 100 << 20, uploaded: 1 << 20, elapsed: uploadProgressMaxInterval - time.Millisecond, want: false},
|
||||||
|
{name: "completion", total: 100, uploaded: 100, lastPercent: 99, elapsed: uploadProgressMinInterval, want: true},
|
||||||
|
{name: "completion rate limited", total: 100, uploaded: 100, lastPercent: 99, elapsed: uploadProgressMinInterval - time.Millisecond, want: false},
|
||||||
|
{name: "completion already reported", total: 100, uploaded: 100, lastPercent: 100, elapsed: uploadProgressMinInterval, want: false},
|
||||||
|
{name: "out of order callback", total: 100, uploaded: 40, lastPercent: 60, elapsed: uploadProgressMaxInterval, want: false},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got := shouldUpdateUploadProgress(tt.total, tt.uploaded, tt.lastPercent, tt.elapsed)
|
||||||
|
if got != tt.want {
|
||||||
|
t.Fatalf("shouldUpdateUploadProgress() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgressConcurrentCallbacks(t *testing.T) {
|
||||||
|
progress := new(Progress)
|
||||||
|
ctx := context.Background()
|
||||||
|
info := progressTestTaskInfo{}
|
||||||
|
const total = int64(100 << 20)
|
||||||
|
|
||||||
|
progress.OnUploadStart(ctx, info, total)
|
||||||
|
progress.lastUpdateAt.Store(time.Now().Add(-uploadProgressMaxInterval).UnixNano())
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for uploaded := int64(1 << 20); uploaded <= total; uploaded += 1 << 20 {
|
||||||
|
uploaded := uploaded
|
||||||
|
wg.Go(func() {
|
||||||
|
progress.OnUploadProgress(ctx, info, uploaded, total)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
percent := progress.lastUpdatePercent.Load()
|
||||||
|
if percent <= 0 || percent > 100 {
|
||||||
|
t.Fatalf("last upload percentage = %d, want a value in (0, 100]", percent)
|
||||||
|
}
|
||||||
|
if progress.uploadedBytes != total {
|
||||||
|
t.Fatalf("maximum uploaded bytes = %d, want %d", progress.uploadedBytes, total)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingleUploadRetryKeepsRichProgressLayout(t *testing.T) {
|
||||||
|
i18n.Init("zh-Hans")
|
||||||
|
t.Cleanup(func() { i18n.Init("zh-Hans") })
|
||||||
|
|
||||||
|
message := buildSingleProgressMessage(
|
||||||
|
progressTestTaskInfo{},
|
||||||
|
singleUploadPhase(2),
|
||||||
|
25<<20,
|
||||||
|
100<<20,
|
||||||
|
5<<20,
|
||||||
|
2,
|
||||||
|
)
|
||||||
|
if message.Err != nil {
|
||||||
|
t.Fatalf("buildSingleProgressMessage() failed: %v", message.Err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"🔁 上传重试",
|
||||||
|
"🟩🟩⬜️⬜️⬜️⬜️⬜️⬜️⬜️⬜️ 25%",
|
||||||
|
"尝试次数:2",
|
||||||
|
"速度:5.00 MB/s",
|
||||||
|
"大小:25.00 MB / 100.00 MB",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(message.Text, want) {
|
||||||
|
t.Fatalf("retry progress does not contain %q:\n%s", want, message.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingleProgressTemplateOwnsStylesAndEscapesValues(t *testing.T) {
|
||||||
|
i18n.Init("en")
|
||||||
|
t.Cleanup(func() { i18n.Init("zh-Hans") })
|
||||||
|
info := htmlProgressTestTaskInfo{}
|
||||||
|
|
||||||
|
message := buildSingleProgressMessage(info, singlePhaseDownloading, 50, 100, 25, 0)
|
||||||
|
if message.Err != nil {
|
||||||
|
t.Fatalf("buildSingleProgressMessage() failed: %v", message.Err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
`<b>A&B</b>.bin`,
|
||||||
|
`[store<&>]:dir/<i>x</i>&`,
|
||||||
|
"Speed: 25 B/s",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(message.Text, want) {
|
||||||
|
t.Fatalf("progress text does not contain %q:\n%s", want, message.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
bold, code, blockquote, italic := singleEntityCounts(message.Entities)
|
||||||
|
if bold != 2 || code != 6 || blockquote != 1 || italic != 0 {
|
||||||
|
t.Fatalf("progress entity counts = bold:%d code:%d blockquote:%d italic:%d", bold, code, blockquote, italic)
|
||||||
|
}
|
||||||
|
|
||||||
|
failure := buildSingleDoneMessage(info, 100, errors.New(`<i>remote & failed</i>`))
|
||||||
|
if failure.Err != nil {
|
||||||
|
t.Fatalf("buildSingleDoneMessage() failed: %v", failure.Err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(failure.Text, `<i>remote & failed</i>`) {
|
||||||
|
t.Fatalf("failure reason was not preserved literally:\n%s", failure.Text)
|
||||||
|
}
|
||||||
|
bold, code, blockquote, italic = singleEntityCounts(failure.Entities)
|
||||||
|
if bold != 1 || code != 2 || blockquote != 0 || italic != 0 {
|
||||||
|
t.Fatalf("failure entity counts = bold:%d code:%d blockquote:%d italic:%d", bold, code, blockquote, italic)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingleDoneSizeUsesActualUploadSize(t *testing.T) {
|
||||||
|
progress := new(Progress)
|
||||||
|
info := progressTestTaskInfo{}
|
||||||
|
progress.OnStart(context.Background(), info)
|
||||||
|
progress.OnUploadStart(context.Background(), info, 2048)
|
||||||
|
|
||||||
|
if got := progress.doneSize(info); got != 2048 {
|
||||||
|
t.Fatalf("done size = %d, want actual upload size 2048", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type htmlProgressTestTaskInfo struct{}
|
||||||
|
|
||||||
|
func (htmlProgressTestTaskInfo) TaskID() string { return "html-task" }
|
||||||
|
func (htmlProgressTestTaskInfo) FileName() string { return `<b>A&B</b>.bin` }
|
||||||
|
func (htmlProgressTestTaskInfo) FileSize() int64 { return 100 }
|
||||||
|
func (htmlProgressTestTaskInfo) StoragePath() string { return `dir/<i>x</i>&` }
|
||||||
|
func (htmlProgressTestTaskInfo) StorageName() string { return `store<&>` }
|
||||||
|
|
||||||
|
func singleEntityCounts(entities []tg.MessageEntityClass) (bold, code, blockquote, italic int) {
|
||||||
|
for _, messageEntity := range entities {
|
||||||
|
switch messageEntity.(type) {
|
||||||
|
case *tg.MessageEntityBold:
|
||||||
|
bold++
|
||||||
|
case *tg.MessageEntityCode:
|
||||||
|
code++
|
||||||
|
case *tg.MessageEntityBlockquote:
|
||||||
|
blockquote++
|
||||||
|
case *tg.MessageEntityItalic:
|
||||||
|
italic++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
package tfile
|
|
||||||
|
|
||||||
var progressUpdatesLevels = []struct {
|
|
||||||
size int64 // 文件大小阈值
|
|
||||||
stepPercent int // 每多少 % 更新一次
|
|
||||||
}{
|
|
||||||
{10 << 20, 100},
|
|
||||||
{50 << 20, 20},
|
|
||||||
{200 << 20, 10},
|
|
||||||
{500 << 20, 5},
|
|
||||||
}
|
|
||||||
|
|
||||||
func shouldUpdateProgress(total, downloaded int64, lastUpdatePercent int) bool {
|
|
||||||
if total <= 0 || downloaded <= 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
percent := int((downloaded * 100) / total)
|
|
||||||
if percent <= lastUpdatePercent {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
step := progressUpdatesLevels[len(progressUpdatesLevels)-1].stepPercent
|
|
||||||
for _, lvl := range progressUpdatesLevels {
|
|
||||||
if total < lvl.size {
|
|
||||||
step = lvl.stepPercent
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return percent >= lastUpdatePercent+step
|
|
||||||
}
|
|
||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"io"
|
"io"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ProgressWriterAt struct {
|
type ProgressWriterAt struct {
|
||||||
@@ -20,9 +22,16 @@ func (w *ProgressWriterAt) WriteAt(p []byte, off int64) (int, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
downloaded := w.downloaded.Add(int64(at))
|
||||||
if w.progress != nil {
|
if w.progress != nil {
|
||||||
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
|
w.progress.OnProgress(w.ctx, w.info, downloaded, w.total)
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(w.ctx, taskevent.Event{
|
||||||
|
TaskID: w.info.TaskID(),
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: w.total,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
return at, nil
|
return at, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -56,9 +65,16 @@ func (w *ProgressWriter) Write(p []byte) (int, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
downloaded := w.downloaded.Add(int64(at))
|
||||||
if w.progress != nil {
|
if w.progress != nil {
|
||||||
w.progress.OnProgress(w.ctx, w.info, w.downloaded.Add(int64(at)), w.total)
|
w.progress.OnProgress(w.ctx, w.info, downloaded, w.total)
|
||||||
}
|
}
|
||||||
|
taskevent.Emit(w.ctx, taskevent.Event{
|
||||||
|
TaskID: w.info.TaskID(),
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: w.total,
|
||||||
|
DownloadedBytes: downloaded,
|
||||||
|
})
|
||||||
return at, nil
|
return at, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
75
core/tasks/transfer/cancel_test.go
Normal file
75
core/tasks/transfer/cancel_test.go
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
package transfer_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
|
storconfig "github.com/krau/SaveAny-Bot/config/storage"
|
||||||
|
"github.com/krau/SaveAny-Bot/core/tasks/transfer"
|
||||||
|
storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/storagetypes"
|
||||||
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
// initConfig seeds the global config so task execution reads a sane Workers
|
||||||
|
// value (the zero default would deadlock errgroup.SetLimit(0)).
|
||||||
|
func initConfig(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), "config.toml")
|
||||||
|
if err := os.WriteFile(path, []byte("workers = 2\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := config.Init(t.Context(), path); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type cancelSource struct{}
|
||||||
|
|
||||||
|
func (cancelSource) Init(context.Context, storconfig.StorageConfig) error { return nil }
|
||||||
|
func (cancelSource) Type() storenum.StorageType { return storenum.Local }
|
||||||
|
func (cancelSource) Name() string { return "cancel-source" }
|
||||||
|
func (cancelSource) Save(context.Context, io.Reader, string) error { return nil }
|
||||||
|
func (cancelSource) Exists(context.Context, string) bool { return false }
|
||||||
|
func (cancelSource) ListFiles(context.Context, string) ([]storagetypes.FileInfo, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// OpenFile reports cancellation, as the task context would be cancelled.
|
||||||
|
func (cancelSource) OpenFile(ctx context.Context, path string) (io.ReadCloser, int64, error) {
|
||||||
|
return nil, 0, ctx.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
type voidTarget struct{}
|
||||||
|
|
||||||
|
func (voidTarget) Init(context.Context, storconfig.StorageConfig) error { return nil }
|
||||||
|
func (voidTarget) Type() storenum.StorageType { return storenum.Local }
|
||||||
|
func (voidTarget) Name() string { return "void-target" }
|
||||||
|
func (voidTarget) Save(context.Context, io.Reader, string) error { return nil }
|
||||||
|
func (voidTarget) Exists(context.Context, string) bool { return false }
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ storage.StorageReadable = cancelSource{}
|
||||||
|
_ storage.Storage = voidTarget{}
|
||||||
|
)
|
||||||
|
|
||||||
|
// Regression: IgnoreErrors must swallow element failures but never a task
|
||||||
|
// cancellation; Execute must surface context.Canceled.
|
||||||
|
func TestIgnoreErrorsPropagatesCancel(t *testing.T) {
|
||||||
|
initConfig(t)
|
||||||
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
elem := transfer.NewTaskElement(cancelSource{}, storagetypes.FileInfo{Path: "/a", Name: "a", Size: 1}, voidTarget{}, "/out")
|
||||||
|
task := transfer.NewTransferTask("id", ctx, []transfer.TaskElement{*elem}, nil, true)
|
||||||
|
|
||||||
|
err := task.Execute(ctx)
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("expected context.Canceled, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ package transfer
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -11,6 +12,7 @@ import (
|
|||||||
"github.com/charmbracelet/log"
|
"github.com/charmbracelet/log"
|
||||||
"github.com/krau/SaveAny-Bot/config"
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
|
||||||
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
||||||
"github.com/krau/SaveAny-Bot/storage"
|
"github.com/krau/SaveAny-Bot/storage"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
)
|
)
|
||||||
@@ -19,7 +21,9 @@ import (
|
|||||||
func (t *Task) Execute(ctx context.Context) error {
|
func (t *Task) Execute(ctx context.Context) error {
|
||||||
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("transfer[%s]", t.ID))
|
logger := log.FromContext(ctx).WithPrefix(fmt.Sprintf("transfer[%s]", t.ID))
|
||||||
logger.Info("Starting transfer task")
|
logger.Info("Starting transfer task")
|
||||||
t.Progress.OnStart(ctx, t)
|
if t.Progress != nil {
|
||||||
|
t.Progress.OnStart(ctx, t)
|
||||||
|
}
|
||||||
|
|
||||||
workers := config.C().Workers
|
workers := config.C().Workers
|
||||||
eg, gctx := errgroup.WithContext(ctx)
|
eg, gctx := errgroup.WithContext(ctx)
|
||||||
@@ -27,14 +31,11 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
|
|
||||||
for _, elem := range t.elems {
|
for _, elem := range t.elems {
|
||||||
eg.Go(func() error {
|
eg.Go(func() error {
|
||||||
t.processingMu.RLock()
|
t.processingMu.Lock()
|
||||||
if t.processing[elem.ID] != nil {
|
if t.processing[elem.ID] != nil {
|
||||||
t.processingMu.RUnlock()
|
t.processingMu.Unlock()
|
||||||
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
|
return fmt.Errorf("element with ID %s is already being processed", elem.ID)
|
||||||
}
|
}
|
||||||
t.processingMu.RUnlock()
|
|
||||||
|
|
||||||
t.processingMu.Lock()
|
|
||||||
t.processing[elem.ID] = &elem
|
t.processing[elem.ID] = &elem
|
||||||
t.processingMu.Unlock()
|
t.processingMu.Unlock()
|
||||||
|
|
||||||
@@ -45,7 +46,7 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
}()
|
}()
|
||||||
|
|
||||||
err := t.processElement(gctx, elem)
|
err := t.processElement(gctx, elem)
|
||||||
if err != nil && !t.IgnoreErrors {
|
if err != nil && (!t.IgnoreErrors || errors.Is(err, context.Canceled)) {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -65,7 +66,9 @@ func (t *Task) Execute(ctx context.Context) error {
|
|||||||
logger.Info("Transfer task completed successfully")
|
logger.Info("Transfer task completed successfully")
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Progress.OnDone(ctx, t, err)
|
if t.Progress != nil {
|
||||||
|
t.Progress.OnDone(ctx, t, err)
|
||||||
|
}
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -115,7 +118,15 @@ func (t *Task) processElement(ctx context.Context, elem TaskElement) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
t.uploaded.Add(size)
|
t.uploaded.Add(size)
|
||||||
t.Progress.OnProgress(ctx, t)
|
if t.Progress != nil {
|
||||||
|
t.Progress.OnProgress(ctx, t)
|
||||||
|
}
|
||||||
|
taskevent.Emit(ctx, taskevent.Event{
|
||||||
|
TaskID: t.ID,
|
||||||
|
Phase: taskevent.PhaseProgress,
|
||||||
|
TotalBytes: t.totalSize,
|
||||||
|
DownloadedBytes: t.uploaded.Load(),
|
||||||
|
})
|
||||||
|
|
||||||
logger.Info("File uploaded successfully")
|
logger.Info("File uploaded successfully")
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"github.com/krau/SaveAny-Bot/common/i18n"
|
"github.com/krau/SaveAny-Bot/common/i18n"
|
||||||
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
"github.com/krau/SaveAny-Bot/common/i18n/i18nk"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
"github.com/krau/SaveAny-Bot/common/utils/dlutil"
|
||||||
|
"github.com/krau/SaveAny-Bot/common/utils/progressutil"
|
||||||
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
"github.com/krau/SaveAny-Bot/common/utils/tgutil"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -83,7 +84,7 @@ func (p *Progress) OnStart(ctx context.Context, info TaskInfo) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
func (p *Progress) OnProgress(ctx context.Context, info TaskInfo) {
|
||||||
if !shouldUpdateProgress(info.TotalSize(), info.Uploaded(), int(p.lastUpdatePercent.Load())) {
|
if !progressutil.ShouldUpdate(info.TotalSize(), info.Uploaded(), int(p.lastUpdatePercent.Load())) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
percent := int((info.Uploaded() * 100) / info.TotalSize())
|
percent := int((info.Uploaded() * 100) / info.TotalSize())
|
||||||
@@ -221,14 +222,6 @@ func (p *Progress) OnDone(ctx context.Context, info TaskInfo, err error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func shouldUpdateProgress(total, current int64, lastPercent int) bool {
|
|
||||||
if total == 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
currentPercent := int((current * 100) / total)
|
|
||||||
return currentPercent > lastPercent && currentPercent%5 == 0
|
|
||||||
}
|
|
||||||
|
|
||||||
func formatDuration(d time.Duration) string {
|
func formatDuration(d time.Duration) string {
|
||||||
d = d.Round(time.Second)
|
d = d.Round(time.Second)
|
||||||
h := d / time.Hour
|
h := d / time.Hour
|
||||||
|
|||||||
@@ -85,12 +85,10 @@ func (t *Task) downloadFiles(ctx context.Context, tempDir string) ([]string, err
|
|||||||
cmd := ytdlp.New().
|
cmd := ytdlp.New().
|
||||||
Output(filepath.Join(tempDir, "%(title)s.%(ext)s"))
|
Output(filepath.Join(tempDir, "%(title)s.%(ext)s"))
|
||||||
|
|
||||||
// If no custom flags are provided, use default behavior
|
// Apply config-based format/quality defaults only when the user passes no
|
||||||
|
// custom flags. Any user flag means they take full control of yt-dlp.
|
||||||
if len(t.Flags) == 0 {
|
if len(t.Flags) == 0 {
|
||||||
cmd = cmd.
|
cmd = applyFormatConfig(cmd, config.C().Ytdlp)
|
||||||
FormatSort("res,ext:mp4:m4a").
|
|
||||||
RecodeVideo("mp4").
|
|
||||||
RestrictFilenames()
|
|
||||||
}
|
}
|
||||||
// Note: If custom flags are provided, users have full control over format/quality
|
// Note: If custom flags are provided, users have full control over format/quality
|
||||||
// The output path is always set above to ensure downloads go to the correct directory
|
// The output path is always set above to ensure downloads go to the correct directory
|
||||||
|
|||||||
40
core/tasks/ytdlp/format.go
Normal file
40
core/tasks/ytdlp/format.go
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
package ytdlp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
ytdlp "github.com/lrstanley/go-ytdlp"
|
||||||
|
|
||||||
|
"github.com/krau/SaveAny-Bot/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// buildFormatSelector translates a max height into a yt-dlp format selector.
|
||||||
|
// It prefers merging the best video+audio within the height limit, then falls
|
||||||
|
// back to a single muxed stream. An empty result means "no explicit selector".
|
||||||
|
func buildFormatSelector(maxHeight int) string {
|
||||||
|
if maxHeight <= 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
h := strconv.Itoa(maxHeight)
|
||||||
|
return "bv*[height<=" + h + "]+ba/b[height<=" + h + "]/b"
|
||||||
|
}
|
||||||
|
|
||||||
|
// applyFormatConfig configures format/quality on the yt-dlp command according to
|
||||||
|
// the ytdlp config. It is only meant to be called when the user did not supply
|
||||||
|
// any custom flags, so config-driven defaults never conflict with user input.
|
||||||
|
func applyFormatConfig(cmd *ytdlp.Command, cfg config.YtdlpConfig) *ytdlp.Command {
|
||||||
|
switch {
|
||||||
|
case cfg.Format != "":
|
||||||
|
cmd = cmd.Format(cfg.Format)
|
||||||
|
case cfg.MaxHeight > 0:
|
||||||
|
cmd = cmd.Format(buildFormatSelector(cfg.MaxHeight))
|
||||||
|
default:
|
||||||
|
// Preserve the original default: prefer highest resolution mp4/m4a.
|
||||||
|
cmd = cmd.FormatSort("res,ext:mp4:m4a")
|
||||||
|
}
|
||||||
|
if cfg.Recode != "" {
|
||||||
|
cmd = cmd.RecodeVideo(cfg.Recode)
|
||||||
|
}
|
||||||
|
cmd = cmd.RestrictFilenames()
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
23
core/tasks/ytdlp/format_test.go
Normal file
23
core/tasks/ytdlp/format_test.go
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
package ytdlp
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestBuildFormatSelector(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
maxHeight int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"no limit", 0, ""},
|
||||||
|
{"negative", -1, ""},
|
||||||
|
{"1080p", 1080, "bv*[height<=1080]+ba/b[height<=1080]/b"},
|
||||||
|
{"720p", 720, "bv*[height<=720]+ba/b[height<=720]/b"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
if got := buildFormatSelector(tt.maxHeight); got != tt.want {
|
||||||
|
t.Errorf("buildFormatSelector(%d) = %q, want %q", tt.maxHeight, got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -113,6 +113,51 @@ secret = "your-rpc-secret"
|
|||||||
remove_after_transfer = true
|
remove_after_transfer = true
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### yt-dlp Configuration
|
||||||
|
|
||||||
|
Configures the behavior of the `/ytdlp` command and the `ytdlp` HTTP-API task type when no custom flags are passed.
|
||||||
|
|
||||||
|
- `max_height`: Default maximum video resolution by height in pixels (e.g. `1080`, `720`). `0` means no limit (best available). Ignored when `format` is set.
|
||||||
|
- `format`: A raw yt-dlp format selector (`-f`). When set, it takes precedence over `max_height` and gives you full control, e.g. `bv*[height<=720]+ba/b`.
|
||||||
|
- `recode`: The target video container yt-dlp recodes into after download (e.g. `mp4`). Leave empty to disable recoding.
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
These defaults only apply when using the `/ytdlp` command without passing any custom flags. Passing custom flags on the command (or `flags` in the API) overrides them.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[ytdlp]
|
||||||
|
max_height = 1080
|
||||||
|
format = "" # e.g. "bv*[height<=720]+ba/b"
|
||||||
|
recode = "mp4" # empty disables recoding
|
||||||
|
```
|
||||||
|
|
||||||
|
### HTTP API Configuration
|
||||||
|
|
||||||
|
When enabled, SaveAny-Bot exposes an HTTP API for creating/querying/canceling tasks programmatically. See [HTTP API](../../usage/api) for the full endpoint reference.
|
||||||
|
|
||||||
|
- `enable`: Whether to enable the HTTP API server, default is `false`.
|
||||||
|
- `host`: Bind address, default `0.0.0.0`.
|
||||||
|
- `port`: Listen port, default `8080`.
|
||||||
|
- `token`: Authentication token. **Strongly recommended** — if empty, the API is exposed without any authentication.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[api]
|
||||||
|
enable = false
|
||||||
|
host = "0.0.0.0"
|
||||||
|
port = 8080
|
||||||
|
token = "your-token"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Log Configuration
|
||||||
|
|
||||||
|
- `level`: Log level. One of `debug`, `info`, `warn`, `error`, `fatal`. Default is `info`.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[log]
|
||||||
|
level = "info"
|
||||||
|
```
|
||||||
|
|
||||||
### Storage Endpoints List
|
### Storage Endpoints List
|
||||||
|
|
||||||
The storage endpoints list is used to define the storage locations supported by the Bot. Each storage endpoint needs to specify a name, type, and related configuration, using the double bracket syntax `[[storages]]`.
|
The storage endpoints list is used to define the storage locations supported by the Bot. Each storage endpoint needs to specify a name, type, and related configuration, using the double bracket syntax `[[storages]]`.
|
||||||
|
|||||||
@@ -79,7 +79,8 @@ Stream mode is not supported.
|
|||||||
chat_id = "123456789" # Telegram chat ID, the bot will send files to this chat
|
chat_id = "123456789" # Telegram chat ID, the bot will send files to this chat
|
||||||
force_file = false # Force sending as file, default is false
|
force_file = false # Force sending as file, default is false
|
||||||
skip_large = false # Skip large files, default is false. If enabled, files exceeding Telegram's limit will not be uploaded.
|
skip_large = false # Skip large files, default is false. If enabled, files exceeding Telegram's limit will not be uploaded.
|
||||||
spilt_size_mb = 2000 # Split size in MB, default is 2000 MB (2 GB). Files larger than this will be split into multiple parts (zip format). Ignored when skip_large is true.
|
split_large_video = false # Losslessly split oversized videos into one album of up to 10 independently playable parts. Falls back to ZIP parts on failure or when more than 10 parts are required.
|
||||||
|
split_size_mb = 0 # Split size in MB. 0 uses the uploader account limit: 2000 MB for bots/regular users and 4000 MB for Premium users. Oversized non-video files use ZIP parts. Ignored when skip_large is true.
|
||||||
```
|
```
|
||||||
|
|
||||||
## Rclone
|
## Rclone
|
||||||
@@ -136,4 +137,4 @@ remote = "myremote"
|
|||||||
base_path = "/backup"
|
base_path = "/backup"
|
||||||
config_path = "/path/to/rclone.conf"
|
config_path = "/path/to/rclone.conf"
|
||||||
flags = ["--progress"]
|
flags = ["--progress"]
|
||||||
```
|
```
|
||||||
|
|||||||
90
docs/content/en/usage/cli.md
Normal file
90
docs/content/en/usage/cli.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
---
|
||||||
|
title: "CLI Subcommands"
|
||||||
|
weight: 21
|
||||||
|
---
|
||||||
|
|
||||||
|
# CLI Subcommands
|
||||||
|
|
||||||
|
Besides running the Telegram bot with `./saveany-bot` (no subcommand), the binary exposes two helper subcommands for moving local files into a storage backend: `upload` (one-shot) and `watch` (continuous).
|
||||||
|
|
||||||
|
These subcommands load the same `config.toml` as the bot, initialize the database and caches, then perform their task. They do **not** start the Telegram bot itself, although storages of type `telegram` will spin up the bot client just for the upload.
|
||||||
|
|
||||||
|
## `upload` — Upload a Single File
|
||||||
|
|
||||||
|
```
|
||||||
|
saveany-bot upload -f <file> -s <storage> [-d <dir>] [--no-progress]
|
||||||
|
```
|
||||||
|
|
||||||
|
Flags:
|
||||||
|
|
||||||
|
| Flag | Required | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `-f, --file` | Yes | Path to the local file to upload |
|
||||||
|
| `-s, --storage` | Yes | Target storage name (must exist in `config.toml`) |
|
||||||
|
| `-d, --dir` | No | Destination directory within the storage. Defaults to the storage's `base_path` |
|
||||||
|
| `--no-progress` | No | Disable the terminal progress bar |
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Upload a file to the default dir of storage "MyAlist"
|
||||||
|
./saveany-bot upload -f ./movie.mp4 -s MyAlist
|
||||||
|
|
||||||
|
# Upload into a specific subdirectory
|
||||||
|
./saveany-bot upload -f ./movie.mp4 -s MyAlist -d movies/2026
|
||||||
|
|
||||||
|
# Upload via Telegram storage without a progress bar
|
||||||
|
./saveany-bot upload -f ./photo.jpg -s MyChannel --no-progress
|
||||||
|
```
|
||||||
|
|
||||||
|
## `watch` — Watch a Directory and Auto-Upload
|
||||||
|
|
||||||
|
The `watch` subcommand continuously monitors a local directory and uploads created or modified files to a storage backend, preserving the relative directory structure from the watch root.
|
||||||
|
|
||||||
|
```
|
||||||
|
saveany-bot watch -p <path> -s <storage> [-d <dir>] [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
Flags:
|
||||||
|
|
||||||
|
| Flag | Default | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `-p, --path` | *(required)* | Local directory to watch |
|
||||||
|
| `-s, --storage` | *(required)* | Target storage name |
|
||||||
|
| `-d, --dir` | storage's `base_path` | Destination directory within the storage |
|
||||||
|
| `-r, --recursive` | `false` | Watch subdirectories recursively |
|
||||||
|
| `--overwrite` | `false` | Overwrite existing files on the storage instead of skipping them |
|
||||||
|
| `--initial-scan` | `false` | Upload files already present in the directory on startup |
|
||||||
|
| `--debounce` | `2s` | How long to wait after the last write before uploading a file |
|
||||||
|
| `--upload-workers` | `config.workers` | Number of concurrent uploads |
|
||||||
|
| `--retry-delay` | `3s` | Delay between upload retries |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
Write-completion detection: the watcher debounces per file and only uploads once the file size stays unchanged across the debounce window, so partial/write-in-progress files are not uploaded.
|
||||||
|
<br />
|
||||||
|
If a file changes while being uploaded, it is re-uploaded once after the current upload finishes (instead of being queued multiple times).
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Watch ./inbox and upload new files to "MyAlist" recursively
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist -r
|
||||||
|
|
||||||
|
# Watch with a custom destination dir and overwrite
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist -d backup --overwrite
|
||||||
|
|
||||||
|
# On startup, also upload everything already in ./inbox
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist --initial-scan
|
||||||
|
```
|
||||||
|
|
||||||
|
### Behavior notes
|
||||||
|
|
||||||
|
- Relative directory structure is preserved under the destination directory. A file written to `./inbox/sub/file.txt` with `--path ./inbox` is uploaded to `<dest_dir>/sub/file.txt`.
|
||||||
|
- `watch` runs until interrupted (e.g. `Ctrl-C` / `SIGINT`); in-flight uploads are drained before exit.
|
||||||
|
- Retries follow the global `retry` value from `config.toml`, with `--retry-delay` between attempts.
|
||||||
|
- Telegram-type storages will start the bot client automatically to perform uploads.
|
||||||
|
|
||||||
|
{{< hint warning >}}
|
||||||
|
`watch` is unrelated to the in-bot `/watch` command (which watches Telegram chats). This subcommand watches a **local filesystem directory** and uploads to a storage backend, independent of Telegram.
|
||||||
|
{{< /hint >}}
|
||||||
75
docs/content/en/usage/config.md
Normal file
75
docs/content/en/usage/config.md
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
---
|
||||||
|
title: "File Naming & Conflict Strategies"
|
||||||
|
weight: 11
|
||||||
|
---
|
||||||
|
|
||||||
|
# File Naming & Conflict Strategies
|
||||||
|
|
||||||
|
SaveAny-Bot lets you customize how saved files are named and how collisions with existing files are resolved, directly in Telegram via the `/config` and `/fnametmpl` commands.
|
||||||
|
|
||||||
|
## `/config` — User Configuration
|
||||||
|
|
||||||
|
The `/config` command opens an inline menu where you can change two per-user settings:
|
||||||
|
|
||||||
|
- **Filename strategy** — how the saved file is named
|
||||||
|
- **Duplicate file strategy** — what happens when a file with the same name already exists in the target storage
|
||||||
|
|
||||||
|
Settings are stored per user and apply to all of that user's subsequent save/transfer tasks.
|
||||||
|
|
||||||
|
### Filename strategy
|
||||||
|
|
||||||
|
| Option | Behavior |
|
||||||
|
|---|---|
|
||||||
|
| `Default` | Use the original media filename, or a generated name when no original filename is available |
|
||||||
|
| `Gen From Msg First` | Generate the filename from the message content (e.g. caption, text) and prefer that over the original filename |
|
||||||
|
| `Template` | Render the filename from a custom template you define with `/fnametmpl` |
|
||||||
|
|
||||||
|
### Duplicate file strategy
|
||||||
|
|
||||||
|
| Option | Behavior |
|
||||||
|
|---|---|
|
||||||
|
| `Always rename` (default) | Keep the existing file and save the new one with an alternate name |
|
||||||
|
| `Ask every time` | Prompt you with inline buttons each time a collision occurs |
|
||||||
|
| `Always overwrite` | Replace the existing file with the new one |
|
||||||
|
| `Always skip` | Do nothing for conflicting files |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
The conflict strategy only kicks in for storage backends that can detect the existence of a file. Backends that do not support existence checks will fall back to overwriting.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
## `/fnametmpl` — Custom Filename Template
|
||||||
|
|
||||||
|
When the filename strategy is set to `Template`, SaveAny-Bot renders each saved file's name using the template configured via `/fnametmpl`.
|
||||||
|
|
||||||
|
```
|
||||||
|
/fnametmpl [template]
|
||||||
|
```
|
||||||
|
|
||||||
|
- Running `/fnametmpl` without arguments shows your current template and the help text.
|
||||||
|
- Running it with a template string sets that template as your filename template.
|
||||||
|
|
||||||
|
The template uses Go [`text/template`](https://pkg.go.dev/text/template) syntax. The available variables are:
|
||||||
|
|
||||||
|
| Variable | Description |
|
||||||
|
|---|---|
|
||||||
|
| `{{.msgid}}` | Telegram message ID |
|
||||||
|
| `{{.msgtags}}` | Hashtags found in the message, joined with `_` |
|
||||||
|
| `{{.msggen}}` | Filename generated from the message |
|
||||||
|
| `{{.msgdate}}` | Message date, formatted `YYYY-MM-DD_HH-MM-SS` |
|
||||||
|
| `{{.msgraw}}` | Raw, unprocessed message text |
|
||||||
|
| `{{.origname}}` | The media's original filename (if any) |
|
||||||
|
| `{{.chatid}}` | Chat ID of the message |
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Fixed prefix + message id + date
|
||||||
|
/fnametmpl Image_{{.msgid}}_{{.msgdate}}.jpg
|
||||||
|
|
||||||
|
# Use original name if available, otherwise a generated name
|
||||||
|
/fnametmpl {{.origname}}
|
||||||
|
```
|
||||||
|
|
||||||
|
{{< hint warning >}}
|
||||||
|
The template only takes effect when the filename strategy is set to `Template`. If template parsing fails, SaveAny-Bot falls back to the default filename naming logic.
|
||||||
|
{{< /hint >}}
|
||||||
@@ -27,6 +27,45 @@ Pay attention to spaces; the bot can only parse correctly formatted syntax. Belo
|
|||||||
|
|
||||||
In addition, if `CHOSEN` is used as the storage name in the rule, it means files will be stored under the path of the storage you selected by clicking the inline button.
|
In addition, if `CHOSEN` is used as the storage name in the rule, it means files will be stored under the path of the storage you selected by clicking the inline button.
|
||||||
|
|
||||||
|
You can also toggle whether rules are applied with `/rule switch`. When rule mode is off, all files go to the default storage.
|
||||||
|
|
||||||
|
## Preset Rules
|
||||||
|
|
||||||
|
Manually writing regex rules for common file types is tedious, so the bot ships a built-in set of preset categories (video, image, audio, document, archive) that you can import in one command:
|
||||||
|
|
||||||
|
```
|
||||||
|
/rule preset <storage> [base_path]
|
||||||
|
```
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
|
||||||
|
- `storage`: Target storage name (must exist and be accessible to you)
|
||||||
|
- `base_path`: Optional. Each preset category's subdirectory is created under this path. If omitted, the default category directory names are used directly.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```
|
||||||
|
# Import preset rules into "MyAlist" with the default directory layout
|
||||||
|
/rule preset MyAlist
|
||||||
|
|
||||||
|
# Import preset rules with a custom base path "downloads/sorted"
|
||||||
|
/rule preset MyAlist downloads/sorted
|
||||||
|
```
|
||||||
|
|
||||||
|
This will create `FILENAME-REGEX` rules for each category, routing matched files to the corresponding subdirectory under `base_path`:
|
||||||
|
|
||||||
|
| Category | Matched extensions | Default directory |
|
||||||
|
|---|---|---|
|
||||||
|
| video | mp4, mkv, ts, avi, flv, mov, webm, wmv, rmvb, m2ts | `视频` |
|
||||||
|
| image | jpg, jpeg, png, gif, webp, bmp | `图片` |
|
||||||
|
| audio | mp3, flac, wav, aac, m4a, ogg | `音频` |
|
||||||
|
| document | pdf, doc, docx, xls, xlsx, ppt, pptx, txt, md, csv, epub, mobi, azw3, chm | `文档` |
|
||||||
|
| archive | zip, rar, 7z, tar, gz, bz2, xz, ... | `压缩包` |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
Preset rules are regular `FILENAME-REGEX` rules once imported. You can view, edit, or delete them individually with `/rule` and `/rule del <id>` like any other rule.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
Rule types:
|
Rule types:
|
||||||
|
|
||||||
## FILENAME-REGEX
|
## FILENAME-REGEX
|
||||||
|
|||||||
@@ -30,6 +30,7 @@ base_path = "./downloads"
|
|||||||
|
|
||||||
### 全局配置
|
### 全局配置
|
||||||
|
|
||||||
|
- `lang`: Bot 使用的语言, 默认为 `zh-CN` (简体中文), 设为 `en` 则使用英语.
|
||||||
- `stream`: 是否启用 Stream 模式, 默认为 `false`. 启用后 Bot 将直接将文件流式传输到存储端(若存储端支持), 不需要下载到本地
|
- `stream`: 是否启用 Stream 模式, 默认为 `false`. 启用后 Bot 将直接将文件流式传输到存储端(若存储端支持), 不需要下载到本地
|
||||||
{{< hint warning >}}
|
{{< hint warning >}}
|
||||||
Stream 模式对于磁盘空间有限的部署环境十分有用, 但也有一些弊端:
|
Stream 模式对于磁盘空间有限的部署环境十分有用, 但也有一些弊端:
|
||||||
@@ -47,6 +48,7 @@ Stream 模式对于磁盘空间有限的部署环境十分有用, 但也有一
|
|||||||
- `proxy`: 全局代理配置, 配置后程序内一切网络连接将会尝试使用该代理, 可选.
|
- `proxy`: 全局代理配置, 配置后程序内一切网络连接将会尝试使用该代理, 可选.
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
|
lang = "zh-CN"
|
||||||
stream = false
|
stream = false
|
||||||
workers = 3
|
workers = 3
|
||||||
threads = 4
|
threads = 4
|
||||||
@@ -111,6 +113,51 @@ secret = "your-rpc-secret"
|
|||||||
remove_after_transfer = true
|
remove_after_transfer = true
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### yt-dlp 配置
|
||||||
|
|
||||||
|
用于配置 `/ytdlp` 命令以及 HTTP API 中 `ytdlp` 任务类型在未传自定义参数时的默认行为.
|
||||||
|
|
||||||
|
- `max_height`: 默认下载的最高视频清晰度 (按高度限制), 如 `1080`, `720`, `480`; `0` 表示不限制 (下载最佳画质). 当设置了 `format` 时此项被忽略.
|
||||||
|
- `format`: 直接指定 yt-dlp format 选择表达式, 设置后优先级高于 `max_height`, 例如 `bv*[height<=720]+ba/b`.
|
||||||
|
- `recode`: 下载后转封装的视频容器格式 (如 `mp4`), 留空则不转封装.
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
这些默认值仅在使用 `/ytdlp` 命令且未传任何自定义参数时生效. 在命令上传递自定义参数 (或在 API 中传 `flags`) 会覆盖这些默认值.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[ytdlp]
|
||||||
|
max_height = 1080
|
||||||
|
format = "" # 例如 "bv*[height<=720]+ba/b"
|
||||||
|
recode = "mp4" # 留空则不转封装
|
||||||
|
```
|
||||||
|
|
||||||
|
### HTTP API 配置
|
||||||
|
|
||||||
|
启用后, SaveAny-Bot 会暴露一套 HTTP API, 用于以编程方式创建/查询/取消任务. 完整的接口说明见 [HTTP API](../../usage/api).
|
||||||
|
|
||||||
|
- `enable`: 是否启用 HTTP API 服务, 默认为 `false`.
|
||||||
|
- `host`: 监听地址, 默认 `0.0.0.0`.
|
||||||
|
- `port`: 监听端口, 默认 `8080`.
|
||||||
|
- `token`: 鉴权 Token, **强烈建议设置** — 若为空, API 将在无任何鉴权的情况下暴露.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[api]
|
||||||
|
enable = false
|
||||||
|
host = "0.0.0.0"
|
||||||
|
port = 8080
|
||||||
|
token = "your-token"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 日志配置
|
||||||
|
|
||||||
|
- `level`: 日志级别, 可选 `debug`, `info`, `warn`, `error`, `fatal`. 默认为 `info`.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[log]
|
||||||
|
level = "info"
|
||||||
|
```
|
||||||
|
|
||||||
### 存储端列表
|
### 存储端列表
|
||||||
|
|
||||||
存储端列表用于定义 Bot 支持的存储位置, 每个存储端需要指定名称、类型和相关配置, 使用双中括号语法 `[[storages]]` 定义.
|
存储端列表用于定义 Bot 支持的存储位置, 每个存储端需要指定名称、类型和相关配置, 使用双中括号语法 `[[storages]]` 定义.
|
||||||
|
|||||||
@@ -82,10 +82,12 @@ chat_id = "123456789"
|
|||||||
force_file = false
|
force_file = false
|
||||||
# 是否跳过大文件, 默认为 false. 如果启用, 超过 Telegram 限制的文件将不会上传.
|
# 是否跳过大文件, 默认为 false. 如果启用, 超过 Telegram 限制的文件将不会上传.
|
||||||
skip_large = false
|
skip_large = false
|
||||||
# 分卷大小, 单位 MB, 默认为 2000 MB (2 GB).
|
# 超限视频是否使用 FFmpeg 无损分割成一个媒体组(最多 10 个)内可独立播放的小视频;失败或需要超过 10 段时回退到 ZIP 分卷.
|
||||||
# 超过该大小的文件将被分割成多个部分上传.(使用 zip 格式)
|
split_large_video = false
|
||||||
|
# 分卷大小, 单位 MB. 设为 0 时使用实际上传账号的限制:Bot/普通用户为 2000 MB,Premium 用户为 4000 MB.
|
||||||
|
# 超过该大小的文件将被分割成多个部分上传;非视频使用 ZIP 格式.
|
||||||
# 当 skip_large 启用时, 该选项无效.
|
# 当 skip_large 启用时, 该选项无效.
|
||||||
spilt_size_mb = 2000
|
split_size_mb = 0
|
||||||
```
|
```
|
||||||
|
|
||||||
## Rclone
|
## Rclone
|
||||||
@@ -142,4 +144,4 @@ remote = "myremote"
|
|||||||
base_path = "/backup"
|
base_path = "/backup"
|
||||||
config_path = "/path/to/rclone.conf"
|
config_path = "/path/to/rclone.conf"
|
||||||
flags = ["--progress"]
|
flags = ["--progress"]
|
||||||
```
|
```
|
||||||
|
|||||||
90
docs/content/zh/usage/cli.md
Normal file
90
docs/content/zh/usage/cli.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
---
|
||||||
|
title: "命令行子命令"
|
||||||
|
weight: 21
|
||||||
|
---
|
||||||
|
|
||||||
|
# 命令行子命令
|
||||||
|
|
||||||
|
除了直接运行 `./saveany-bot` (不带子命令) 启动 Telegram Bot 外, 这个二进制文件还提供两个把本地文件上传到存储后端的辅助子命令: `upload` (一次性) 和 `watch` (持续监听).
|
||||||
|
|
||||||
|
这些子命令会读取与 Bot 相同的 `config.toml`, 初始化数据库和缓存, 然后执行任务. 它们**不会**启动 Telegram Bot 本身, 但 `telegram` 类型的存储会在需要上传时临时启动 Bot 客户端来执行上传.
|
||||||
|
|
||||||
|
## `upload` — 上传单个文件
|
||||||
|
|
||||||
|
```
|
||||||
|
saveany-bot upload -f <文件> -s <存储名> [-d <目录>] [--no-progress]
|
||||||
|
```
|
||||||
|
|
||||||
|
参数:
|
||||||
|
|
||||||
|
| 参数 | 必填 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `-f, --file` | 是 | 待上传的本地文件路径 |
|
||||||
|
| `-s, --storage` | 是 | 目标存储名 (必须存在于 `config.toml`) |
|
||||||
|
| `-d, --dir` | 否 | 存储中的目标目录, 默认使用存储的 `base_path` |
|
||||||
|
| `--no-progress` | 否 | 关闭终端进度条 |
|
||||||
|
|
||||||
|
示例:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 上传文件到 "MyAlist" 的默认目录
|
||||||
|
./saveany-bot upload -f ./movie.mp4 -s MyAlist
|
||||||
|
|
||||||
|
# 上传到指定子目录
|
||||||
|
./saveany-bot upload -f ./movie.mp4 -s MyAlist -d movies/2026
|
||||||
|
|
||||||
|
# 通过 Telegram 存储上传并关闭进度条
|
||||||
|
./saveany-bot upload -f ./photo.jpg -s MyChannel --no-progress
|
||||||
|
```
|
||||||
|
|
||||||
|
## `watch` — 监听目录并自动上传
|
||||||
|
|
||||||
|
`watch` 子命令持续监听一个本地目录, 将新建或修改的文件上传到存储后端, 并保留相对监听根目录的子目录结构.
|
||||||
|
|
||||||
|
```
|
||||||
|
saveany-bot watch -p <路径> -s <存储名> [-d <目录>] [选项]
|
||||||
|
```
|
||||||
|
|
||||||
|
参数:
|
||||||
|
|
||||||
|
| 参数 | 默认值 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `-p, --path` | *(必填)* | 要监听的本地目录 |
|
||||||
|
| `-s, --storage` | *(必填)* | 目标存储名 |
|
||||||
|
| `-d, --dir` | 存储的 `base_path` | 存储中的目标目录 |
|
||||||
|
| `-r, --recursive` | `false` | 是否递归监听子目录 |
|
||||||
|
| `--overwrite` | `false` | 覆盖存储上已有的文件, 而非跳过 |
|
||||||
|
| `--initial-scan` | `false` | 启动时将目录中已存在的文件也上传 |
|
||||||
|
| `--debounce` | `2s` | 文件最后一次写入后, 等待多久再上传 |
|
||||||
|
| `--upload-workers` | `config.workers` | 并发上传数 |
|
||||||
|
| `--retry-delay` | `3s` | 上传重试之间的延迟 |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
写入完成检测: 监听器会按文件做防抖处理, 仅当文件大小在一个 debounce 窗口内保持不变时才上传, 因此不会上传未写完的半成品文件.
|
||||||
|
<br />
|
||||||
|
若某文件在上传过程中又被修改, 它会在当前上传完成后再上传一次, 而不是被重复排队.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
示例:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 递归监听 ./inbox 并且把新文件上传到 "MyAlist"
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist -r
|
||||||
|
|
||||||
|
# 自定义目标目录并覆盖已有文件
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist -d backup --overwrite
|
||||||
|
|
||||||
|
# 启动时把 ./inbox 中已有的内容也一并上传
|
||||||
|
./saveany-bot watch -p ./inbox -s MyAlist --initial-scan
|
||||||
|
```
|
||||||
|
|
||||||
|
### 行为说明
|
||||||
|
|
||||||
|
- 相对子目录结构会被保留: 以 `--path ./inbox` 为例, 写入 `./inbox/sub/file.txt` 的文件会被上传到 `<目标目录>/sub/file.txt`.
|
||||||
|
- `watch` 会一直运行直到被中断 (如 `Ctrl-C` / `SIGINT`), 退出前会等待所有进行中的上传完成.
|
||||||
|
- 重试次数遵循 `config.toml` 中的全局 `retry` 值, 各次重试之间间隔 `--retry-delay`.
|
||||||
|
- `telegram` 类型的存储会自动启动 Bot 客户端来执行上传.
|
||||||
|
|
||||||
|
{{< hint warning >}}
|
||||||
|
`watch` 子命令与 Bot 内的 `/watch` 命令 (监听 Telegram 聊天) 无关. 本子命令监听的是**本地文件系统目录**, 不依赖 Telegram.
|
||||||
|
{{< /hint >}}
|
||||||
75
docs/content/zh/usage/config.md
Normal file
75
docs/content/zh/usage/config.md
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
---
|
||||||
|
title: "文件命名与重名策略"
|
||||||
|
weight: 11
|
||||||
|
---
|
||||||
|
|
||||||
|
# 文件命名与重名策略
|
||||||
|
|
||||||
|
SaveAny-Bot 支持在 Telegram 中通过 `/config` 和 `/fnametmpl` 命令自定义保存文件的命名方式, 以及处理与已存在文件重名时的冲突策略.
|
||||||
|
|
||||||
|
## `/config` — 用户配置
|
||||||
|
|
||||||
|
`/config` 命令会弹出一个内联菜单, 你可以在其中修改以下两项用户级设置:
|
||||||
|
|
||||||
|
- **文件名策略** — 保存文件的命名方式
|
||||||
|
- **重名文件保存策略** — 目标存储中已存在同名文件时的处理方式
|
||||||
|
|
||||||
|
设置按用户分别保存, 对该用户后续所有的保存/转存任务生效.
|
||||||
|
|
||||||
|
### 文件名策略
|
||||||
|
|
||||||
|
| 选项 | 行为 |
|
||||||
|
|---|---|
|
||||||
|
| `默认` | 使用媒体原始文件名, 没有原始文件名时使用生成名 |
|
||||||
|
| `优先从消息生成` | 优先根据消息内容 (如 caption、文本) 生成文件名, 而非原始文件名 |
|
||||||
|
| `自定义模板` | 使用 `/fnametmpl` 设置的自定义模板渲染文件名 |
|
||||||
|
|
||||||
|
### 重名文件保存策略
|
||||||
|
|
||||||
|
| 选项 | 行为 |
|
||||||
|
|---|---|
|
||||||
|
| `始终重命名` (默认) | 保留已有文件, 将新文件以另一个名字保存 |
|
||||||
|
| `每次询问` | 每次遇到重名时通过内联按钮提示你选择 |
|
||||||
|
| `始终覆盖` | 用新文件替换已有文件 |
|
||||||
|
| `始终跳过` | 对重名文件不做处理 |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
重名策略仅在能够检测文件是否已存在的存储后端生效. 不支持检测文件是否存在的存储后端会退化为覆盖行为.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
|
## `/fnametmpl` — 自定义文件名模板
|
||||||
|
|
||||||
|
当文件名策略设置为 `自定义模板` 时, SaveAny-Bot 会用 `/fnametmpl` 配置的模板来渲染所保存文件的文件名.
|
||||||
|
|
||||||
|
```
|
||||||
|
/fnametmpl [模板]
|
||||||
|
```
|
||||||
|
|
||||||
|
- 不带参数运行 `/fnametmpl` 会显示当前模板以及帮助说明
|
||||||
|
- 带模板字符串运行则会把它设为你的文件名模板
|
||||||
|
|
||||||
|
模板使用 Go [`text/template`](https://pkg.go.dev/text/template) 语法. 可用变量如下:
|
||||||
|
|
||||||
|
| 变量 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| `{{.msgid}}` | Telegram 消息 ID |
|
||||||
|
| `{{.msgtags}}` | 消息中的标签, 以 `_` 连接输出 |
|
||||||
|
| `{{.msggen}}` | 根据消息生成的文件名 |
|
||||||
|
| `{{.msgdate}}` | 消息日期, 格式 `YYYY-MM-DD_HH-MM-SS` |
|
||||||
|
| `{{.msgraw}}` | 消息的原始文本内容 (不做处理) |
|
||||||
|
| `{{.origname}}` | 媒体的原始文件名 (如有) |
|
||||||
|
| `{{.chatid}}` | 消息所在聊天的 ID |
|
||||||
|
|
||||||
|
示例:
|
||||||
|
|
||||||
|
```
|
||||||
|
# 固定前缀 + 消息 ID + 日期
|
||||||
|
/fnametmpl 图片_{{.msgid}}_{{.msgdate}}.jpg
|
||||||
|
|
||||||
|
# 优先使用原始文件名, 没有则用生成名
|
||||||
|
/fnametmpl {{.origname}}
|
||||||
|
```
|
||||||
|
|
||||||
|
{{< hint warning >}}
|
||||||
|
模板仅在文件名策略设置为 `自定义模板` 时生效. 如果模板解析失败, SaveAny-Bot 会回退到默认的文件名生成逻辑.
|
||||||
|
{{< /hint >}}
|
||||||
@@ -27,6 +27,45 @@ weight: 3
|
|||||||
|
|
||||||
此外, 规则中的存储名若使用 "CHOSEN" , 则表示存储到点击按钮选择的存储端的路径下
|
此外, 规则中的存储名若使用 "CHOSEN" , 则表示存储到点击按钮选择的存储端的路径下
|
||||||
|
|
||||||
|
你也可以使用 `/rule switch` 来开关规则模式. 关闭规则模式时, 所有文件都将保存到默认存储.
|
||||||
|
|
||||||
|
## 预设规则
|
||||||
|
|
||||||
|
为常见文件类型手动编写正则规则比较繁琐, 因此 Bot 内置了一组预设分类 (视频、图片、音频、文档、压缩包), 可以通过一条命令批量导入:
|
||||||
|
|
||||||
|
```
|
||||||
|
/rule preset <存储名> [基础路径]
|
||||||
|
```
|
||||||
|
|
||||||
|
参数:
|
||||||
|
|
||||||
|
- `存储名`: 目标存储名 (必须存在且你有权访问)
|
||||||
|
- `基础路径`: 可选. 各预设分类的子目录会创建在此路径下; 若不填则直接使用默认分类目录名
|
||||||
|
|
||||||
|
示例:
|
||||||
|
|
||||||
|
```
|
||||||
|
# 导入预设规则到 "MyAlist", 使用默认目录布局
|
||||||
|
/rule preset MyAlist
|
||||||
|
|
||||||
|
# 在自定义基础路径 "downloads/sorted" 下导入预设规则
|
||||||
|
/rule preset MyAlist downloads/sorted
|
||||||
|
```
|
||||||
|
|
||||||
|
此命令会为每个分类创建 `FILENAME-REGEX` 规则, 将匹配的文件路由到 `基础路径` 下对应的子目录:
|
||||||
|
|
||||||
|
| 分类 | 匹配的扩展名 | 默认目录 |
|
||||||
|
|---|---|---|
|
||||||
|
| 视频 | mp4, mkv, ts, avi, flv, mov, webm, wmv, rmvb, m2ts | `视频` |
|
||||||
|
| 图片 | jpg, jpeg, png, gif, webp, bmp | `图片` |
|
||||||
|
| 音频 | mp3, flac, wav, aac, m4a, ogg | `音频` |
|
||||||
|
| 文档 | pdf, doc, docx, xls, xlsx, ppt, pptx, txt, md, csv, epub, mobi, azw3, chm | `文档` |
|
||||||
|
| 压缩包 | zip, rar, 7z, tar, gz, bz2, xz, ... | `压缩包` |
|
||||||
|
|
||||||
|
{{< hint info >}}
|
||||||
|
导入后的预设规则就是普通的 `FILENAME-REGEX` 规则. 你可以像其他规则一样通过 `/rule` 查看或用 `/rule del <id>` 单独删除/编辑它们.
|
||||||
|
{{< /hint >}}
|
||||||
|
|
||||||
规则类型:
|
规则类型:
|
||||||
|
|
||||||
## FILENAME-REGEX
|
## FILENAME-REGEX
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user