Files
SaveAny-Bot/api/factory.go
T
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00

383 lines
11 KiB
Go

package api
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/krau/SaveAny-Bot/config"
"github.com/krau/SaveAny-Bot/core"
"github.com/krau/SaveAny-Bot/core/tasks/aria2dl"
"github.com/krau/SaveAny-Bot/core/tasks/batchtfile"
"github.com/krau/SaveAny-Bot/core/tasks/directlinks"
"github.com/krau/SaveAny-Bot/core/tasks/parsed"
tphtask "github.com/krau/SaveAny-Bot/core/tasks/telegraph"
"github.com/krau/SaveAny-Bot/core/tasks/tfile"
"github.com/krau/SaveAny-Bot/core/tasks/transfer"
"github.com/krau/SaveAny-Bot/core/tasks/ytdlp"
"github.com/krau/SaveAny-Bot/parsers/parsers"
"github.com/krau/SaveAny-Bot/pkg/aria2"
"github.com/krau/SaveAny-Bot/pkg/enums/tasktype"
"github.com/krau/SaveAny-Bot/pkg/parser"
"github.com/krau/SaveAny-Bot/pkg/taskevent"
"github.com/krau/SaveAny-Bot/pkg/telegraph"
"github.com/krau/SaveAny-Bot/storage"
"github.com/rs/xid"
)
// TaskFactory 任务工厂
type TaskFactory struct {
ctx context.Context
}
// NewTaskFactory 创建任务工厂
func NewTaskFactory(ctx context.Context) *TaskFactory {
return &TaskFactory{ctx: ctx}
}
// CreateTask 创建任务
func (f *TaskFactory) CreateTask(req *CreateTaskRequest) (*CreateTaskResponse, error) {
// 验证存储
stor, ok := storage.GetStorage(req.Storage)
if !ok {
return nil, fmt.Errorf("storage not found: %s", req.Storage)
}
taskID := xid.New().String()
createdAt := time.Now()
switch req.Type {
case tasktype.TaskTypeDirectlinks:
return f.createDirectLinksTask(taskID, createdAt, req, stor)
case tasktype.TaskTypeYtdlp:
return f.createYTDLPTask(taskID, createdAt, req, stor)
case tasktype.TaskTypeAria2:
return f.createAria2Task(taskID, createdAt, req, stor)
case tasktype.TaskTypeParseditem:
return f.createParsedTask(taskID, createdAt, req, stor)
case tasktype.TaskTypeTgfiles:
return f.createTGFilesTask(taskID, createdAt, req, stor)
case tasktype.TaskTypeTphpics:
return f.createTPHPicsTask(taskID, createdAt, req, stor)
case tasktype.TaskTypeTransfer:
return f.createTransferTask(taskID, createdAt, req)
default:
return nil, fmt.Errorf("unsupported task type: %s", req.Type)
}
}
func (f *TaskFactory) registerAndEnqueueTask(task core.Executable, taskType tasktype.TaskType, storageName, path, webhook string) error {
taskID := task.TaskID()
info := RegisterTask(taskID, string(taskType), storageName, path, task.Title(), webhook)
// Inject the progress sink into the context so the task's Emit calls update
// the API store (and fire the webhook on terminal states) without the task
// knowing about the API.
taskCtx := taskevent.WithSink(f.ctx, info)
err := core.AddTask(taskCtx, task)
if err != nil {
DeleteTask(taskID)
return fmt.Errorf("failed to add task: %w", err)
}
return nil
}
// createDirectLinksTask 创建直链下载任务
func (f *TaskFactory) createDirectLinksTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params DirectLinksParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if len(params.URLs) == 0 {
return nil, fmt.Errorf("no URLs provided")
}
task := directlinks.NewTask(taskID, f.ctx, params.URLs, stor, req.Path, nil)
err := f.registerAndEnqueueTask(task, tasktype.TaskTypeDirectlinks, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeDirectlinks,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createYTDLPTask 创建 yt-dlp 任务
func (f *TaskFactory) createYTDLPTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params YTDLPParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if len(params.URLs) == 0 {
return nil, fmt.Errorf("no URLs provided")
}
task := ytdlp.NewTask(taskID, f.ctx, params.URLs, params.Flags, stor, req.Path, nil)
err := f.registerAndEnqueueTask(task, tasktype.TaskTypeYtdlp, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeYtdlp,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createAria2Task 创建 Aria2 任务
func (f *TaskFactory) createAria2Task(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params Aria2Params
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if len(params.URLs) == 0 {
return nil, fmt.Errorf("no URLs provided")
}
// 检查 Aria2 是否启用
cfg := config.C().Aria2
if !cfg.Enable {
return nil, fmt.Errorf("aria2 is not enabled")
}
aria2Client, err := aria2.NewClient(cfg.Url, cfg.Secret)
if err != nil {
return nil, fmt.Errorf("failed to create aria2 client: %w", err)
}
// 添加下载任务到 Aria2
gid, err := aria2Client.AddURI(f.ctx, params.URLs, nil)
if err != nil {
return nil, fmt.Errorf("failed to add aria2 task: %w", err)
}
task := aria2dl.NewTask(taskID, f.ctx, gid, params.URLs, aria2Client, stor, req.Path, nil)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeAria2, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeAria2,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createParsedTask 创建解析任务
func (f *TaskFactory) createParsedTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params ParsedParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if params.URL == "" {
return nil, fmt.Errorf("no URL provided")
}
// 查找合适的解析器
var p parser.Parser
for _, parserItem := range parsers.Get() {
if parserItem.CanHandle(params.URL) {
p = parserItem
break
}
}
if p == nil {
return nil, fmt.Errorf("no parser found for URL: %s", params.URL)
}
// 解析 URL
item, err := p.Parse(f.ctx, params.URL)
if err != nil {
return nil, fmt.Errorf("failed to parse URL: %w", err)
}
task := parsed.NewTask(taskID, f.ctx, stor, req.Path, item, nil)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeParseditem, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeParseditem,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createTGFilesTask 创建 Telegram 文件下载任务
func (f *TaskFactory) createTGFilesTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params TGFilesParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if len(params.MessageLinks) == 0 {
return nil, fmt.Errorf("no message links provided")
}
// 提取文件
files, err := ExtractFilesFromLinks(f.ctx, params.MessageLinks)
if err != nil {
return nil, fmt.Errorf("failed to extract files: %w", err)
}
if len(files) == 0 {
return nil, fmt.Errorf("no files found in provided links")
}
var task core.Executable
if len(files) == 1 {
// 单个文件任务
tfileTask, err := tfile.NewTGFileTask(taskID, f.ctx, files[0], stor, req.Path, nil)
if err != nil {
return nil, fmt.Errorf("failed to create tfile task: %w", err)
}
task = tfileTask
} else {
// 批量文件任务
elems := make([]batchtfile.TaskElement, 0, len(files))
for _, file := range files {
elem, err := batchtfile.NewTaskElement(stor, req.Path, file)
if err != nil {
return nil, fmt.Errorf("failed to create task element: %w", err)
}
elems = append(elems, *elem)
}
task = batchtfile.NewBatchTGFileTask(taskID, f.ctx, elems, nil, true)
}
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTgfiles, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeTgfiles,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createTPHPicsTask 创建 Telegraph 图片下载任务
func (f *TaskFactory) createTPHPicsTask(taskID string, createdAt time.Time, req *CreateTaskRequest, stor storage.Storage) (*CreateTaskResponse, error) {
var params TPHPicsParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
if params.TelegraphURL == "" {
return nil, fmt.Errorf("no telegraph URL provided")
}
// 提取图片
pics, phPath, err := ExtractTelegraphImages(f.ctx, params.TelegraphURL)
if err != nil {
return nil, fmt.Errorf("failed to extract telegraph images: %w", err)
}
if len(pics) == 0 {
return nil, fmt.Errorf("no images found in telegraph page")
}
client := telegraph.NewClient()
task := tphtask.NewTask(taskID, f.ctx, phPath, pics, stor, req.Path, client, nil)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTphpics, req.Storage, req.Path, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeTphpics,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}
// createTransferTask 创建存储间传输任务
func (f *TaskFactory) createTransferTask(taskID string, createdAt time.Time, req *CreateTaskRequest) (*CreateTaskResponse, error) {
var params TransferParams
if err := json.Unmarshal(req.Params, &params); err != nil {
return nil, fmt.Errorf("invalid params: %w", err)
}
// 验证源存储和目标存储
sourceStor, ok := storage.GetStorage(params.SourceStorage)
if !ok {
return nil, fmt.Errorf("source storage not found: %s", params.SourceStorage)
}
targetStor, ok := storage.GetStorage(params.TargetStorage)
if !ok {
return nil, fmt.Errorf("target storage not found: %s", params.TargetStorage)
}
// 检查源存储是否可读
sourceReadable, ok := sourceStor.(storage.StorageReadable)
if !ok {
return nil, fmt.Errorf("source storage does not support reading: %s", params.SourceStorage)
}
// 检查源存储是否可列
sourceListable, ok := sourceStor.(storage.StorageListable)
if !ok {
return nil, fmt.Errorf("source storage does not support listing: %s", params.SourceStorage)
}
// 列出源文件
files, err := sourceListable.ListFiles(f.ctx, params.SourcePath)
if err != nil {
return nil, fmt.Errorf("failed to list source files: %w", err)
}
if len(files) == 0 {
return nil, fmt.Errorf("no files found at source path: %s", params.SourcePath)
}
// 创建传输元素
elems := make([]transfer.TaskElement, 0, len(files))
for _, file := range files {
elem := transfer.NewTaskElement(sourceReadable, file, targetStor, params.TargetPath)
elems = append(elems, *elem)
}
task := transfer.NewTransferTask(taskID, f.ctx, elems, nil, true)
err = f.registerAndEnqueueTask(task, tasktype.TaskTypeTransfer, params.TargetStorage, params.TargetPath, req.Webhook)
if err != nil {
return nil, err
}
return &CreateTaskResponse{
TaskID: taskID,
Type: tasktype.TaskTypeTransfer,
Status: TaskStatusQueued,
CreatedAt: createdAt,
}, nil
}