mirror of
https://github.com/krau/SaveAny-Bot.git
synced 2026-08-30 20:56:42 +08:00
* fix: prevent queue deadlock after cancelling tasks Get no longer recurses while holding the mutex. Cancelled queued tasks leave the map, making their IDs reusable. Closed empty queues return ErrQueueClosed. * fix: init task queue lazily and close on shutdown AddTask is safe before Run by initializing the queue once. Close unblocks workers waiting in Get. * fix: make resource fingerprints deterministic Sort map keys before hashing so Resource.ID is stable. * fix: harden per-item processing tracking in tasks Use the resource fingerprint as the dedup key on insert and delete. Check and set processing entries atomically instead of TOCTOU. Count only successful downloads. * fix: honor IgnoreErrors in batch tasks Element failures no longer cancel sibling elements or stop later groups. * fix: report streamed upload bytes in batch tasks Stream downloads report their byte count as the upload total. * fix: validate i18n key parity across locales geni18n fails when a language file misses any key. Align the syncpeers completion key between en and zh-Hans. Translate three untranslated parse keys in zh-Hans. * refactor: share progress throttling helpers Move size-tiered and count-based throttling into progressutil. Localize hardcoded Chinese progress strings. Drop five duplicated implementations and dead local copies. * refactor: share unique filename logic Storage backends use fsutil.UniquePath instead of local loops. * fix: sanitize local storage paths Reject absolute paths and dot-dot escapes in Save. Check close errors and wrap creation failures. * fix: preserve webdav error causes Wrap mkdir and write failures with %w and drop dead error values. * fix: kill rclone subprocess on reader close Prevent cat processes from hanging after the pipe is closed. * fix: fail alist init instead of exiting Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot. Cancel token refresh with the init context and re-login on 401. * fix: enforce telegram album limits and track saved paths Use tglimit.MaxAlbumItems for album batching and video splitting. Exists now reports previously saved paths instead of always false. * fix: guard storage registry maps Protect Storages and UserStorages with mutexes and expose read accessors. * fix: harden JS parser plugin runtime Validate semver instead of panicking and require canHandle. Recover plugin worker panics and time out CanHandle calls. Return a copy from the registry and sanitize install filenames. * fix: guard kemono parser against nil fields Skip sparse preview and attachment entries instead of panicking. * refactor: remove commented-out dead code Drop unused kemono legacy types and commented response structs. * test: cover invalid plugin version rejection * fix: show all queued tasks in /task list Render up to ten tasks and append the truncation note once. * fix: guard callback data parsing Reject malformed callback payloads before indexing split parts. * fix: isolate media groups per user Key pending groups by chat, user and group id. * fix: require permission for callback handlers * fix: initialize userbot context once Replace the racy lazy init with sync.OnceValue. * fix: avoid leaking raw errors in /dir reply * fix: notify users on invalid update version * fix: fail fast when API listen fails Bind synchronously and surface errors instead of logging them. * fix: add timeouts and backoff to webhook delivery * refactor: remove dead code from api and bot Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call. * fix: load remote config without local lookup Skip the local file search after reading a config URL and add a timeout. * refactor: drop unused hook config * docs: document parser plugin config * ci: fix BuildTime formatting and align checkout Actions format does not format dates; pass the raw timestamp. * chore: ignore cache directory * fix: make cache init idempotent * fix: upload only downloaded batch elements Failed elements keep partial cache files and never reach the backend. Successfully downloaded siblings still upload when one element fails. * fix: record telegram saved paths only after upload Skip-large returns a sentinel so skipped files are not marked as saved. * fix: deduplicate alist token refreshes Guard token access with a mutex and merge concurrent logins. Reuse a recent refresh to avoid login storms. * test: cover concurrent alist 401 retry Ten parallel uploads share a single re-login under -race. * fix: count parsed resources in progress text * fix: localize storage lookup errors in /dir Use the shared i18n key and escape the dynamic error. * fix: deduplicate concurrent storage initialization singleflight merges first-time inits so side effects are not duplicated. * fix: guard nil progress trackers in api-created tasks Batch, telegraph and transfer tasks run without a Telegram tracker when created through the API; their callbacks must not panic. * fix: keep album order when filtering failed batch items Download results are stored by original index so surviving elements keep their source order. * test: cover nil-tracker task execution * fix: report partial failure in batch done message IgnoreErrors runs with failed elements show success and failed counts instead of claiming every file completed. * fix: skip login refresh for token-only alist storage 401 responses surface the auth error instead of sending a credential-less login; the refresh window never exceeds TokenExp. * test: cover alist refresh semantics Concurrent refresh uses username/password; token-only storage never attempts a login on 401. * fix: call tracker from notifyProgress instead of recursing The helper called itself, overflowing the stack on any batch task with a progress tracker. * fix: propagate cancellation past IgnoreErrors Cancelled tasks must not be reported as successful: only ordinary element failures are ignored. * test: cover notifyProgress tracker call * test: cover cancellation with IgnoreErrors * fix: refresh alist token after startup 401s The init login no longer satisfies the dedup window, so an early 401 triggers a real refresh; later 401s reuse it. Clarify that streaming uploads cannot replay their body. * test: exercise the alist 401 refresh path Concurrent uploads now reject the init token, share one refresh and retry with the new token; token-only stays inert. * style: trim verbose comments Drop process-style explanations; keep one-line behavior notes. * style: gofmt test files * fix: drop unbounded saved-path cache in telegram storage Telegram cannot reliably query remote file existence, so the cache answered a question it could not answer and grew without bound. Exists returns false again, as before. * style: format codes
201 lines
5.1 KiB
Go
201 lines
5.1 KiB
Go
package api
|
|
|
|
import (
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/krau/SaveAny-Bot/pkg/taskevent"
|
|
)
|
|
|
|
// TaskProgressInfo stores the progress of an API-submitted task. All fields are
|
|
// guarded by mu. It implements taskevent.Sink so the task layer can update it
|
|
// without knowing about the API.
|
|
type TaskProgressInfo struct {
|
|
mu sync.Mutex
|
|
TaskID string
|
|
Type string
|
|
Status TaskStatus
|
|
Title string
|
|
TotalBytes int64
|
|
DownloadedBytes int64
|
|
TotalFiles int
|
|
DownloadedFiles int
|
|
Storage string
|
|
Path string
|
|
Error string
|
|
CreatedAt time.Time
|
|
UpdatedAt time.Time
|
|
StartedAt time.Time
|
|
Webhook string
|
|
webhookNotified bool
|
|
}
|
|
|
|
// progressStore holds all API tasks. Entries are removed a fixed duration after
|
|
// they reach a terminal state to bound memory usage.
|
|
type progressStore struct {
|
|
mu sync.RWMutex
|
|
tasks map[string]*TaskProgressInfo
|
|
retention time.Duration
|
|
}
|
|
|
|
var store = &progressStore{
|
|
tasks: make(map[string]*TaskProgressInfo),
|
|
retention: 24 * time.Hour,
|
|
}
|
|
|
|
// RegisterTask registers a new API task and returns its progress info.
|
|
func RegisterTask(taskID, taskType, storage, path, title, webhook string) *TaskProgressInfo {
|
|
info := &TaskProgressInfo{
|
|
TaskID: taskID,
|
|
Type: taskType,
|
|
Status: TaskStatusQueued,
|
|
Title: title,
|
|
Storage: storage,
|
|
Path: path,
|
|
CreatedAt: time.Now(),
|
|
UpdatedAt: time.Now(),
|
|
Webhook: webhook,
|
|
}
|
|
|
|
store.mu.Lock()
|
|
store.tasks[taskID] = info
|
|
store.mu.Unlock()
|
|
|
|
return info
|
|
}
|
|
|
|
// GetTask returns the progress info for a task.
|
|
func GetTask(taskID string) (*TaskProgressInfo, bool) {
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
info, ok := store.tasks[taskID]
|
|
return info, ok
|
|
}
|
|
|
|
// GetAllTasks returns all tracked tasks.
|
|
func GetAllTasks() []*TaskProgressInfo {
|
|
store.mu.RLock()
|
|
defer store.mu.RUnlock()
|
|
|
|
tasks := make([]*TaskProgressInfo, 0, len(store.tasks))
|
|
for _, info := range store.tasks {
|
|
tasks = append(tasks, info)
|
|
}
|
|
return tasks
|
|
}
|
|
|
|
// DeleteTask removes a task record.
|
|
func DeleteTask(taskID string) {
|
|
store.mu.Lock()
|
|
defer store.mu.Unlock()
|
|
delete(store.tasks, taskID)
|
|
}
|
|
|
|
// CleanupExpired removes tasks that reached a terminal state more than the
|
|
// store's retention duration ago. It is safe to call periodically.
|
|
func CleanupExpired() {
|
|
now := time.Now()
|
|
store.mu.Lock()
|
|
defer store.mu.Unlock()
|
|
for id, info := range store.tasks {
|
|
info.mu.Lock()
|
|
terminal := info.Status == TaskStatusCompleted || info.Status == TaskStatusFailed || info.Status == TaskStatusCancelled
|
|
stale := terminal && now.Sub(info.UpdatedAt) > store.retention
|
|
info.mu.Unlock()
|
|
if stale {
|
|
delete(store.tasks, id)
|
|
}
|
|
}
|
|
}
|
|
|
|
// StartCleanupLoop runs CleanupExpired on a fixed interval until ctx is done.
|
|
// It should be started once during API server initialization.
|
|
func StartCleanupLoop(ctx interface{ Done() <-chan struct{} }) {
|
|
go func() {
|
|
ticker := time.NewTicker(10 * time.Minute)
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return
|
|
case <-ticker.C:
|
|
CleanupExpired()
|
|
}
|
|
}
|
|
}()
|
|
}
|
|
|
|
// UpdateStatus sets the task status.
|
|
func (t *TaskProgressInfo) UpdateStatus(status TaskStatus) {
|
|
t.mu.Lock()
|
|
t.Status = status
|
|
t.UpdatedAt = time.Now()
|
|
if status == TaskStatusRunning && t.StartedAt.IsZero() {
|
|
t.StartedAt = t.UpdatedAt
|
|
}
|
|
t.mu.Unlock()
|
|
}
|
|
|
|
// SetError marks the task failed with an error message.
|
|
func (t *TaskProgressInfo) SetError(err string) {
|
|
t.mu.Lock()
|
|
t.Error = err
|
|
t.Status = TaskStatusFailed
|
|
t.UpdatedAt = time.Now()
|
|
t.mu.Unlock()
|
|
}
|
|
|
|
// snapshot returns a point-in-time copy of the fields needed to render a
|
|
// response, so callers never touch the mutex directly.
|
|
func (t *TaskProgressInfo) snapshot() (status TaskStatus, total, downloaded int64, totalFiles, downloadedFiles int, startedAt time.Time, err string, updatedAt time.Time) {
|
|
t.mu.Lock()
|
|
defer t.mu.Unlock()
|
|
return t.Status, t.TotalBytes, t.DownloadedBytes, t.TotalFiles, t.DownloadedFiles, t.StartedAt, t.Error, t.UpdatedAt
|
|
}
|
|
|
|
// Emit implements taskevent.Sink. It translates task lifecycle events into
|
|
// status/progress updates and fires the webhook on terminal transitions.
|
|
func (t *TaskProgressInfo) Emit(e taskevent.Event) {
|
|
t.mu.Lock()
|
|
switch e.Phase {
|
|
case taskevent.PhaseStart:
|
|
t.Status = TaskStatusRunning
|
|
if t.StartedAt.IsZero() {
|
|
t.StartedAt = time.Now()
|
|
}
|
|
if e.TotalBytes > 0 {
|
|
t.TotalBytes = e.TotalBytes
|
|
}
|
|
case taskevent.PhaseProgress:
|
|
t.Status = TaskStatusRunning
|
|
if e.TotalBytes > 0 {
|
|
t.TotalBytes = e.TotalBytes
|
|
}
|
|
t.DownloadedBytes = e.DownloadedBytes
|
|
if e.TotalFiles > 0 {
|
|
t.TotalFiles = e.TotalFiles
|
|
}
|
|
if e.DownloadedFiles > 0 {
|
|
t.DownloadedFiles = e.DownloadedFiles
|
|
}
|
|
case taskevent.PhaseDone:
|
|
if e.Err != nil {
|
|
t.Status = TaskStatusFailed
|
|
t.Error = e.Err.Error()
|
|
} else {
|
|
t.Status = TaskStatusCompleted
|
|
}
|
|
}
|
|
t.UpdatedAt = time.Now()
|
|
notify := t.Webhook != "" && !t.webhookNotified && (t.Status == TaskStatusCompleted || t.Status == TaskStatusFailed)
|
|
if notify {
|
|
t.webhookNotified = true
|
|
}
|
|
t.mu.Unlock()
|
|
|
|
if notify {
|
|
payload := CreateWebhookPayload(t.TaskID, t.Type, t.Status, t.Storage, t.Path, e.Err)
|
|
SendWebhook(nil, payload)
|
|
}
|
|
}
|