Files
SaveAny-Bot/parsers/js/api.go
T
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00

182 lines
5.2 KiB
Go

//go:build !no_jsparser
package js
import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"github.com/blang/semver"
"github.com/charmbracelet/log"
"github.com/dop251/goja"
"github.com/krau/SaveAny-Bot/common/utils/netutil"
"github.com/krau/SaveAny-Bot/parsers/parsers"
)
func jsRegisterParser(vm *goja.Runtime) func(call goja.FunctionCall) goja.Value {
return func(call goja.FunctionCall) goja.Value {
jsObj := call.Argument(0)
if jsObj == nil || goja.IsUndefined(jsObj) || goja.IsNull(jsObj) {
return vm.NewGoError(errors.New("registerParser expects an object { canHandle, parse }"))
}
obj := jsObj.ToObject(vm)
if obj == nil {
return vm.NewGoError(errors.New("registerParser expects an object { canHandle, parse }"))
}
metaValue := obj.Get("metadata")
if metaValue == nil || goja.IsUndefined(metaValue) {
return vm.NewGoError(errors.New("parser must provide metadata"))
}
var metadata PluginMeta
if exported := metaValue.Export(); exported != nil {
data, err := json.Marshal(exported)
if err != nil {
return vm.NewGoError(fmt.Errorf("failed to marshal metadata to JSON: %w", err))
}
if err := json.Unmarshal(data, &metadata); err != nil {
return vm.NewGoError(fmt.Errorf("failed to unmarshal JSON to PluginMeta: %w", err))
}
} else {
return vm.NewGoError(errors.New("metadata cannot be null or undefined"))
}
pluginV, err := semver.Parse(metadata.Version)
if err != nil {
return vm.NewGoError(fmt.Errorf("invalid parser version %q: %w", metadata.Version, err))
}
if pluginV.LT(MinimumParserVersion) {
return vm.NewGoError(fmt.Errorf("parser version %s is not supported, must be at least %s", metadata.Version, MinimumParserVersion))
}
if pluginV.Major > LatestParserVersion.Major {
log.Printf("warning: parser major version %d is newer than latest supported major version %d", pluginV.Major, LatestParserVersion.Major)
}
handleFn := obj.Get("canHandle")
parseFn := obj.Get("parse")
if parseFn == nil || goja.IsUndefined(parseFn) {
return vm.NewGoError(errors.New("parser must provide a parse function"))
}
if handleFn == nil || goja.IsUndefined(handleFn) {
return vm.NewGoError(errors.New("parser must provide a canHandle function"))
}
parsers.Add(newJSParser(vm, handleFn, parseFn, metadata))
return goja.Undefined()
}
}
var jsConsole = func(logger *log.Logger) map[string]any {
return map[string]any{
"log": func(args ...any) {
if len(args) == 0 {
return
}
if len(args) > 1 {
logger.Info(args[0], args[1:]...)
} else {
logger.Info(args[0])
}
},
"error": func(args ...any) {
if len(args) == 0 {
return
}
if len(args) > 1 {
logger.Error(fmt.Sprint(args[0]), args[1:]...)
} else {
logger.Error(fmt.Sprint(args[0]))
}
},
}
}
/*
jsGhttp provides a http helper for js plugins
It provides the following functions:
- get(url): performs a GET request and returns the response body as string
- getJSON(url): performs a GET request and returns the response body parsed as JSON
- head(url): performs a HEAD request and returns the response headers and status code
*/
var jsGhttp = func(vm *goja.Runtime) *goja.Object {
ghttp := vm.NewObject()
client := netutil.DefaultParserHTTPClient()
ghttp.Set("get", func(call goja.FunctionCall) goja.Value {
url := call.Argument(0).String()
resp, err := client.Get(url)
if err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Sprintf("failed to fetch %s: %v", url, err),
})
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return vm.ToValue(map[string]any{
"error": fmt.Sprintf("failed to fetch %s: %s", url, resp.Status),
"status": resp.StatusCode,
})
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Errorf("failed to read response body: %w", err).Error(),
})
}
return vm.ToValue(string(body))
})
ghttp.Set("getJSON", func(call goja.FunctionCall) goja.Value {
url := call.Argument(0).String()
resp, err := client.Get(url)
if err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Sprintf("failed to fetch %s: %v", url, err),
})
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return vm.ToValue(map[string]any{
"error": fmt.Sprintf("failed to fetch %s: %s", url, resp.Status),
"status": resp.StatusCode,
})
}
body, err := io.ReadAll(resp.Body)
if err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Errorf("failed to read response body: %w", err).Error(),
})
}
var jsonData map[string]any
if err := json.Unmarshal(body, &jsonData); err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Errorf("failed to unmarshal JSON: %w", err).Error(),
})
}
return vm.ToValue(map[string]any{
"data": jsonData,
})
})
ghttp.Set("head", func(call goja.FunctionCall) goja.Value {
url := call.Argument(0).String()
resp, err := client.Head(url)
if err != nil {
return vm.ToValue(map[string]any{
"error": fmt.Sprintf("failed to fetch %s: %v", url, err),
})
}
defer resp.Body.Close()
headers := make(map[string]string)
for k, v := range resp.Header {
headers[k] = v[0]
}
return vm.ToValue(map[string]any{
"status": resp.StatusCode,
"headers": headers,
})
})
return ghttp
}