From 6fd95bbe1b608845ed5aec3a6bc940a0054d5f30 Mon Sep 17 00:00:00 2001 From: krau <71133316+krau@users.noreply.github.com> Date: Tue, 11 Aug 2026 09:04:15 +0800 Subject: [PATCH] chore: update agents instruction --- AGENTS.md | 366 ++++++++++++++---------------------------------------- 1 file changed, 90 insertions(+), 276 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 93ca0f6..1efb0dc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,301 +1,115 @@ -# SaveAny-Bot Agent Guidelines - -This document provides essential information for AI coding agents working on the SaveAny-Bot project. +# Repository Guidelines ## Project Overview -SaveAny-Bot is a Telegram bot written in Go that saves files/messages from Telegram and various websites to multiple storage backends (local, S3, MinIO, WebDAV, AList, Telegram). It features a plugin system for parsing web content and extensible storage backends. +SaveAny-Bot is a Telegram bot written in Go that saves files and messages from Telegram and websites to multiple storage backends (Local, S3, MinIO, WebDAV, AList, Rclone, Telegram). It supports single-file saves, batch/album saves, streaming, multi-user access, storage rules, cross-storage transfers, yt-dlp and Aria2 downloads, and a Goja-based JavaScript parser plugin system with optional Playwright browser automation. -**Tech Stack**: Go 1.24.2, gotd/td (Telegram MTProto), Cobra (CLI), Viper (config), GORM (ORM), SQLite, Goja (JS runtime), Playwright (browser automation) +**Tech stack**: Go 1.25, gotgproto + gotd/td v0.149.0 (MTProto), Cobra (CLI), Viper (config), GORM + SQLite, Goja (JS runtime), Playwright, charmbracelet/log. License: AGPL-3.0. -## Build & Test Commands +**Note on gotd versions**: `gotd/td` must stay on `v0.149.0` — v0.150+ breaks `gotgproto` (v1.0.0-beta22) compilation (`AsInputDocumentFileLocation` signature, `gotd/log` Logger interface). Do not bump beyond v0.149. + +## Architecture & Data Flow + +``` +Telegram update → client/bot/handlers → core.AddTask(Executable) → pkg/queue (serial workers) + → core/tasks/* (download via common/tdler) → storage.Storage → progress feedback +``` + +- **Startup sequence** (`cmd/run.go::initAll`, keep this order): Config → Cache → i18n → Database → Storage → Parser plugins → Userbot → API → Bot. `bot.Init` returns the exit channel; a `SAVEANTBOT-RESTART` error restarts the process (external supervisor). +- **Task pipeline**: handlers build a task (via `core.AddTask`), the queue executes `Executable{Type, Title, TaskID, Execute(ctx)}` with `config.C().Workers` workers. Lifecycle hooks (`TaskBeforeStart/Success/Fail/Cancel`) run around `Execute`. Cancellation = canceling the task's context; tasks must check `ctx.Err()`. +- **Dual progress channels**: (1) `pkg/taskevent` context bus (consumed by `api/` for HTTP/Webhook consumers — `taskevent.WithSink` must be injected for API tasks), (2) Telegram message edits via `ProgressTracker` + `tgutil.ExtFromContext(ctx)` (bot tasks). Upload progress currently reaches the Telegram channel only. +- **Capability interfaces + type assertion fallback** is the core extensibility pattern: `StorageBatchSaver`, `StorageProgressSaver` (upload progress), `StorageListable`, `StorageReadable` are optional; consumers assert and fall back (e.g. wrap reader with `ioutil.NewProgressReader`). New backends only need to implement the interface and register. +- **Config layering**: CLI flag > env `SAVEANY_*` (dots → underscores, e.g. `SAVEANY_TELEGRAM_TOKEN`) > TOML file (path or http(s) URL). `config.C()` returns a **copy** — never mutate it. +- **Storage registration (3 places)**: `pkg/enums/storage` ENUM comment (go-enum), `config/storage/factory.go::storageFactories` (config struct with `Validate()`), `storage/storage.go::storageConstructors` (implementation). + +## Key Directories + +| Path | Purpose | +|---|---| +| `cmd/` | CLI: `run` (main bot), `upload`, `watch` (standalone subcommands that do NOT run initAll), `geni18n` (i18n key generator) | +| `core/` | `Executable` interface, queue worker loop, hooks; `core/tasks/{tfile,batchtfile,directlinks,parsed,telegraph,transfer,ytdlp,aria2dl}` | +| `client/bot/` | gotgproto client, `handlers/` (all commands + message/callback handlers), `middleware/`, `client/user/` (userbot) | +| `storage/` | 8 backends + `storage.go` (interfaces/registry) + `load.go` (per-user storage resolution) | +| `parsers/` | `parsers.go` (registry), `js/` (Goja plugins, ghttp/playwright injection, build-tagged), `parsers/` (native: twitter, kemono) | +| `config/` | Viper setup, defaults, `storage/` per-type config structs | +| `database/` | GORM models (User/Dir/Rule/WatchChat), AutoMigrate, `syncUsers` | +| `pkg/` | `queue`, `taskevent`, `tcbdata` (callback data), `rule`, `enums/{tasktype,storage,ctxkey,fnamest}`, `storagetypes`, `tfile`, `parser` | +| `common/` | `tdler` (unified downloader), `utils/{tgutil,dlutil,ioutil,fsutil,strutil,tphutil,netutil}`, `i18n` (embedded locales), `cache` (ristretto) | +| `api/` | HTTP API + webhook (task factory with sink injection) | +| `docs/` | Hugo site (hugo-book theme, zh+en mirrored), separate go.mod | +| `plugins/` | JS parser examples + `README.md` (plugin author contract) | + +## Development Commands -### Build ```bash -# Standard build -go build -o saveany-bot . - -# Run directly +# Build (standard; CGO_ENABLED=0 for static) +CGO_ENABLED=0 go build -trimpath -o saveany-bot . go run ./cmd -# Docker build (multi-stage, Alpine-based) -docker build -t saveany-bot . -docker compose up -d -``` - -### Test -```bash -# Run all tests +# Test — known failures: storage/telegram TestCreateSplitZip/TestExtractThumbFrame/TestGetVideoMetadata +# (need gitignored fixtures tests/testfile.dat, tests/testvideo; ffmpeg/ffprobe) go test ./... - -# Run tests in specific package -go test ./pkg/queue -go test ./storage/telegram - -# Run tests with verbose output -go test -v ./... - -# Run a single test +go test -race ./core/tasks/... ./storage/... ./pkg/queue/... ./common/... go test -run TestQueueBasic ./pkg/queue -# Run with coverage -go test -cover ./... -``` +# Codegen — run after editing locale YAML or enum comments +go generate ./... # geni18n (i18nk keys) + go-enum (pkg/enums/*) +# go-enum is NOT in go.mod; install externally. geni18n runs via go run. -### Lint & Format -```bash -# Format code (standard Go formatting) -go fmt ./... - -# Vet code for common issues +# Verify go vet ./... - -# Generate code (i18n keys) -go generate ./... +go fmt ./... ``` -### Other Commands -```bash -# Update dependencies -go mod tidy +**Build variants** (Dockerfile.default/micro/pico): `-tags=no_jsparser,no_playwright,no_minio,no_bubbletea,sqlite_glebarez` — each has a `*_stub.go`/`*_glebarez.go` pairing; keep stubs in sync. -# View documentation -cd docs && hugo server -D -``` +Docker: `docker build -t saveany-bot .`, `docker compose up -d` (host network, mounts `./data ./config.toml ./downloads ./cache`). CI (`.github/workflows/`) runs **no tests/lint** — only tag-triggered release/docker builds and docs deployment; run `go test ./...` manually before pushing. -## Code Style Guidelines +## Code Conventions & Common Patterns -### Imports -- Standard library first, then third-party, then project-internal -- Group imports with blank lines between groups -- Use explicit import aliases for clarity when needed (e.g., `storconfig`, `storenum`) +- **Imports**: stdlib → third-party → project-internal, blank-line separated. Aliases for clarity (`storconfig`, `storenum`). +- **Naming**: PascalCase exported, camelCase unexported, files `snake_case.go`; **not** ALL_CAPS constants. +- **Errors**: always wrap with `fmt.Errorf("context: %w", err)`; check with `errors.Is/As`; never ignore. +- **Logging**: `log.FromContext(ctx)` with prefixes (`logger.WithPrefix("component")`); never global logger when ctx is available. +- **Context values** (read from the passed ctx, never globals): `log.FromContext`, `tgutil.ExtFromContext` (Telegram ext — **required for message edits; if nil, edits are silently dropped**), `storage.FromContext`, `storagetypes.WithSourceCaption`, `ctxkey.ContentLength` / `ctxkey.OverwriteExisting`. +- **Progress rendering** (#228 convention): i18n templates declare styles with Telegram HTML (`//
/`); dynamic data MUST go through `i18n.T(key, tgutil.EscapeHTMLTemplateData(data))` before `tgutil.RenderHTML`. Never interpolate user data raw, never render-then-substring-search. +- **Progress tracking**: each task package defines its own small `ProgressTracker` interface; optional `UploadProgressTracker` is probed via type assertion (skip if absent). Serialize state + message edits with a mutex; throttle edits (≥1s); aggregate per-item progress monotonically. +- **i18n**: only edit `common/i18n/locale/{zh-Hans,en}.yaml` → `go generate ./...` → use `i18nk.` constants. No raw strings in user-facing messages. zh-Hans and en must stay in sync. +- **Registration points** (never forget): new bot command → `client/bot/handlers/register.go::CommandHandlers` (auto-publishes /help menu); new task type → `pkg/enums/tasktype` + `core/tasks//` + `api/factory.go::CreateTask`; new storage → 3 places above + `docs/content/{en,zh}/deployment/configuration/storages.md`; new enum value → ENUM comment + `go generate`. +- **Concurrency**: `errgroup.WithContext` + `SetLimit(config.C().Workers)`, `atomic.Int64` counters, `sync.Once` for single-shot events, mutex around render state. No lock-in-callback (callbacks fire after unlock). +- **Cancellation**: queue tasks carry a `WithCancel`-derived ctx; check `ctx.Err()` in loops; classify with `errors.Is(err, context.Canceled)`. +- **JS plugins**: `registerParser({metadata, canHandle, parse})`, `version >= 1.0.0`; per-plugin goja VM is single-goroutine (reqCh buffer 10). Changing `pkg/parser.Item/Resource` JSON fields requires updating `plugins/README.md` and example plugins. +- **Message edits**: `ext.EditMessage(chatID, &tg.MessagesEditMessageRequest{...})`; cancel buttons via `tgutil.BuildCancelButton(taskID)`; callback payloads via `pkg/tcbdata` + `common/cache`. -```go -import ( - "context" - "fmt" - - "github.com/charmbracelet/log" - - "github.com/krau/SaveAny-Bot/config" - "github.com/krau/SaveAny-Bot/pkg/enums/storage" -) -``` +## Important Files -### Formatting -- Line length: reasonable (no hard limit, but be sensible) -- Organize code with blank lines between logical sections -- Follow standard Go conventions for braces, spacing, etc. +- `main.go` — `//go:generate` for i18n keys +- `cmd/run.go` — startup sequence `Run/initAll/cleanCache` (cache cleanup on exit, `NoCleanCache` opt-out) +- `core/core.go` — worker loop, hooks, AddTask/CancelTask +- `pkg/queue/queue.go` — generic serial queue (cond/list; duplicate TaskID rejected) +- `storage/storage.go` — interfaces + registry + compile-time capability assertions +- `config/viper.go`, `config.example.toml` — config schema (authoritative field docs) +- `database/db.go` — GORM init, `GetDialect` (build-tag selectable SQLite driver) +- `client/bot/handlers/register.go` — handler dispatch order and CommandHandlers +- `common/tdler/dler.go` — unified download entry +- `core/tasks/batchtfile/item_progress.go` — per-item phase state machine (Downloading/Transferring/Uploading/Retrying/Confirming, FailureStage) +- `parsers/js/plugin.go` — Goja plugin runtime +- `.github/workflows/` — release/docker/docs (no test gate) -### Types & Interfaces -- Use clear, descriptive type names (PascalCase for exported, camelCase for unexported) -- Define interfaces where abstraction is needed (e.g., `Executable`, `StorageConfig`) -- Embed context in method signatures, not structs: `func (s *Service) Do(ctx context.Context) error` -- Prefer composition over inheritance +## Runtime/Tooling Preferences -```go -// Interfaces define behavior -type Executable interface { - Type() tasktype.TaskType - Title() string - TaskID() string - Execute(ctx context.Context) error -} +- **Go 1.25+**: `t.Context()`, `sync.WaitGroup.Go`, `for range n` are available. +- **Runtime binaries**: ffmpeg/ffprobe (media processing/video split), yt-dlp (ytdlp tasks), aria2 optional; Playwright browsers install on demand to `./playwright` (`playwright.Install(chromium, ...)` at first `pw.get()`); Docker images: default has ffmpeg+yt-dlp, micro only curl, pico is scratch static. +- **No Makefile, no golangci.yml, no test/lint CI** — verification is manual (`go vet`, `go test`). +- **go-enum** required externally for enum generation; **geni18n** is in-repo. +- **Docs**: Hugo site in `docs/` (separate go.mod, hugo-book); edit `docs/content/{zh,en}/` — keep both languages mirrored. `docs/public/` is gitignored build output. +- **gitignored fixtures**: `storage/telegram/tests/` (missing — 3 tests fail locally), `data/`, `config.toml`, `playwright/`, `testplugins/`. -// Structs compose behavior -type Local struct { - config config.LocalStorageConfig - logger *log.Logger -} -``` +## Testing & QA -### Naming Conventions -- **Packages**: lowercase, single word when possible (avoid underscores) -- **Files**: lowercase with underscores for multiword (e.g., `auth_terminal.go`, `progress_reader.go`) -- **Variables**: camelCase for unexported, PascalCase for exported -- **Constants**: PascalCase for exported, camelCase for unexported (not ALL_CAPS) -- **Functions/Methods**: PascalCase for exported, camelCase for unexported -- **Test files**: `*_test.go` pattern - -### Error Handling -- Always handle errors explicitly; never ignore them -- Wrap errors with context using `fmt.Errorf("context: %w", err)` -- Use `errors.Is()` and `errors.As()` for error checking -- Log errors with appropriate level (Error, Warn, Info) -- Return errors from functions rather than panicking (except for truly unrecoverable situations) - -```go -// Good error handling -if err := db.Save(user).Error; err != nil { - return fmt.Errorf("failed to save user %d: %w", user.ChatID, err) -} - -// Check specific errors -if errors.Is(err, context.Canceled) { - logger.Info("Operation was canceled") - return nil -} -``` - -### Logging -- Use `github.com/charmbracelet/log` package -- Get logger from context: `log.FromContext(ctx)` -- Create prefixed loggers for components: `logger.WithPrefix("component")` -- Use appropriate levels: Debug, Info, Warn, Error -- Include context in log messages (e.g., task IDs, file names) - -```go -logger := log.FromContext(ctx) -logger.Infof("Processing task: %s", task.ID) -logger.Errorf("Failed to save file %s: %v", filename, err) -``` - -### Concurrency -- Use channels for communication between goroutines -- Protect shared state with `sync.Mutex` or `sync.RWMutex` -- Use `sync.WaitGroup` for coordinating goroutine completion -- Always pass `context.Context` for cancellation support -- Use `context.WithCancel/WithTimeout` for managing goroutine lifetimes - -```go -// Example from queue implementation -func (tq *TaskQueue[T]) Add(task *Task[T]) error { - tq.mu.Lock() - defer tq.mu.Unlock() - // ... critical section - tq.cond.Signal() - return nil -} -``` - -### Comments -- Document exported types, functions, and packages with doc comments -- Start doc comments with the name being documented -- Use `//` for single-line comments -- Explain *why*, not *what* (code should be self-explanatory for "what") -- Add `[NOTE]`, `[WARN]`, `[IMPORTANT]` tags for important clarifications - -```go -// GetUserByChatID retrieves a user by their Telegram chat ID. -// Returns an error if the user is not found. -func GetUserByChatID(ctx context.Context, chatID int64) (*User, error) { -``` - -## Architecture & Conventions - -### Application Structure -- **Entry point**: `main.go` → `cmd.Execute(ctx)` -- **CLI root**: `cmd/root.go` (Cobra), implementation in `cmd/run.go` -- **Startup sequence**: Config → Cache → i18n → Database → Storage → Parsers → Userbot → Bot → Queue -- Follow this order when adding new initialization steps in `cmd/run.go::initAll` - -### Configuration (Viper) -- Config defined in `config/viper.go::Config` -- Read from `config.toml` (see `config.example.toml`) -- Environment variables: `SAVEANY_*` prefix (e.g., `SAVEANY_TELEGRAM_TOKEN`) -- Access via `config.C()` (returns a copy, don't modify the return value) -- Storage configs validated via `config/storage/factory.go::LoadStorageConfigs` - -### Telegram Client -- **Bot client**: `client/bot/bot.go::Init` (uses gotgproto) -- **Handlers**: Centralized in `client/bot/handlers/` directory -- **Registration**: All handlers registered in `handlers.Register` -- **Commands**: Add to `CommandHandlers` slice for automatic `/help` and bot command list updates -- **Middleware**: Common middleware in `client/middleware/` (floodwait, retry, etc.) - -### Tasks & Queue -- **Task interface**: `core/core.go::Executable` (Type, Title, TaskID, Execute methods) -- **Queue**: `pkg/queue.TaskQueue[Executable]` (generic, thread-safe) -- **Workers**: Count from `config.C().Workers` -- **Task types**: Implementations in `core/tasks/**` (tfile, parsed, telegraph, directlinks, batchtfile) -- **Lifecycle hooks**: `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` (defined in config) -- **Adding tasks**: Use `core.AddTask(ctx, task)` - -### Database (GORM + SQLite) -- **Init**: `database.Init` using `config.C().DB.Path` -- **Models**: User, Dir, Rule, WatchChat (in `database/*.go`) -- **Migrations**: Automatic via `db.AutoMigrate` -- **User sync**: `database.syncUsers` syncs DB with `config.C().Users` (don't manually create/delete users) -- **Context**: Always use `db.WithContext(ctx)` for operations - -### Storage Backends -- **Interface**: Defined in `config/storage/types.go` and `storage/` -- **Implementations**: local, alist, s3/minio, webdav, telegram (each in subdirectory) -- **Adding new storage**: - 1. Add enum to `pkg/enums/storage` - 2. Create config struct in `config/storage/` with `Validate()` method - 3. Implement storage in `storage//` - 4. Register in `storageFactories` mapping - 5. Update `config.example.toml` with example - -### Parser Plugins (JavaScript) -- **Runtime**: Goja (JS runtime) + Playwright (browser automation) -- **Plugin API**: `registerParser({ metadata, canHandle, parse })` in JS -- **Integration**: Defined in `parsers/` directory -- **Documentation**: See `plugins/README.md` -- Plugin `parse` returns `Item`/`Resource` which becomes download/transfer task - -### Internationalization (i18n) -- **Usage**: `i18n.T(i18nk.SomeKey, map[string]any{"Name": value})` -- **Locale files**: `common/i18n/locale/*.yaml` -- **Key generation**: Run `go generate ./...` to generate `common/i18n/i18nk/keys.go` -- **Adding new strings**: Add to YAML → run `go generate` → use in code -- All user-facing strings should be internationalized - -### Context Usage -- Always pass `context.Context` as first parameter -- Use `log.FromContext(ctx)` to get contextual logger -- Respect context cancellation in long-running operations -- Store request-scoped data in context (e.g., `ctxkey.ContentLength`) - -## Special Rules from .github/copilot-instructions.md - -1. **Never modify `config.C()` return values** - it returns a copy. Modify config in `config.Init` or via Viper. -2. **Handlers must update `CommandHandlers` slice** - ensures `/help` and bot commands stay in sync. -3. **Task execution must preserve hooks** - don't remove `TaskBeforeStart`, `TaskSuccess`, `TaskFail`, `TaskCancel` hook calls. -4. **User sync is automatic** - don't manually create/delete users in DB; use config-based sync. -5. **Prefer context logger** - use `log.FromContext(ctx)` over global logger when context is available. -6. **Storage factory pattern** - new storage types must register in `storageFactories` mapping. -7. **Plugin API compatibility** - changes to `Item`/`Resource` structures require updating `plugins/README.md`. - -## Common Patterns - -### Adding a New Command -1. Create handler function in `client/bot/handlers/.go` -2. Add to `CommandHandlers` slice in `register.go` -3. Add i18n key to `common/i18n/locale/*.yaml` -4. Run `go generate ./...` -5. Test with Telegram bot - -### Adding a New Task Type -1. Create struct implementing `core.Executable` in `core/tasks//` -2. Implement `Type()`, `Title()`, `TaskID()`, `Execute(ctx)` methods -3. Add task type enum to `pkg/enums/tasktype` -4. Use `core.AddTask(ctx, task)` to enqueue - -### Adding a New Storage Backend -1. Define config struct in `config/storage/.go` with `Validate()` method -2. Implement storage interface in `storage//.go` -3. Add storage type enum to `pkg/enums/storage` -4. Register factory in `config/storage/factory.go::storageFactories` -5. Update `config.example.toml` with configuration example - -## File References - -When referencing code locations, use `path/to/file.go:line` format (e.g., `core/core.go:23` for the worker function). - -## Testing Guidelines - -- Write tests for new functionality (place in `*_test.go` files) -- Test files should be in same package as code being tested -- Use table-driven tests for multiple test cases -- Mock external dependencies (databases, network calls) -- Aim for meaningful tests, not just coverage numbers - -## Notes - -- Binary size matters: use `CGO_ENABLED=0` for static binaries -- FFmpeg is included in Docker images for media processing -- Build process supports cross-compilation (amd64/arm64, Linux/macOS/Windows) -- Documentation site uses Hugo; edit files in `docs/` directory -- Session data stored in SQLite; delete `data/session.db` if changing bot token +- Pure stdlib `testing` (no testify); table-driven (`[]struct{name...}` + `t.Run`) with `t.Fatalf` got/want assertions. Mock via hand-written interface impls or package-variable replacement (`runMediaTool` in `video_split_test.go`, restored with `t.Cleanup`); in-process services for HTTP (`httptest`), S3 (`gofakes3+s3mem`), WebDAV (`x/net/webdav`). +- **Locale-dependent tests**: pin with `i18n.Init("zh-Hans")` + `t.Cleanup(...)`. +- **Progress/HTML tests**: assert rendered text with `strings.Contains` AND entity counts (`tg.MessageEntityBold/Code/Blockquote/Italic`) — verify style injection stays escaped (`A&B` input must render as literal text). +- **Known failures**: `storage/telegram` `TestCreateSplitZip`, `TestExtractThumbFrame`, `TestGetVideoMetadata` need gitignored fixtures + real ffmpeg — skip with `-skip 'Test(CreateSplitZip|ExtractThumbFrame|GetVideoMetadata)$'`; `api/handlers_test.go` has one `t.Skip` (needs initialized core). +- **Coverage expectations**: pure logic gets table tests (parsers, URL/path utils, progress throttling, grouping); regressions get bug-scenario-named tests (`progress_regression_test.go`). Network/Telegram/Playwright must never be touched by tests. +- When a permanent feature/API change ships: update `config.example.toml` if config, `docs/` if user-facing, `plugins/README.md` if plugin contract, and i18n YAML + `go generate` for new strings.