refactor: quality overhaul (#234)

* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes

storage/alist/alist.go

@@ -9,23 +9,35 @@ import (
 	"net/http"
 	"net/url"
 	"path"
-	"strings"
+	"sync"
 	"time"
 
 	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	config "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
 	"github.com/krau/SaveAny-Bot/pkg/storagetypes"
+	"golang.org/x/sync/singleflight"
 )
 
 type Alist struct {
-	client    *http.Client
-	token     string
-	baseURL   string
-	loginInfo *loginRequest
-	config    config.AlistStorageConfig
-	logger    *log.Logger
+	client      *http.Client
+	tokenMu     sync.RWMutex
+	token       string
+	lastLoginAt time.Time
+	tokenFlight singleflight.Group
+	baseURL     string
+	loginInfo   *loginRequest
+	config      config.AlistStorageConfig
+	logger      *log.Logger
+}
+
+// authHeader returns the current token for use in API requests.
+func (a *Alist) authHeader() string {
+	a.tokenMu.RLock()
+	defer a.tokenMu.RUnlock()
+	return a.token
 }
 
 func (a *Alist) Init(ctx context.Context, cfg config.StorageConfig) error {
@@ -42,39 +54,35 @@ func (a *Alist) Init(ctx context.Context, cfg config.StorageConfig) error {
 	a.logger = log.FromContext(ctx).WithPrefix(fmt.Sprintf("alist[%s]", alistConfig.Name))
 
 	if alistConfig.Token != "" {
+		a.tokenMu.Lock()
 		a.token = alistConfig.Token
-		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Minute)
+		a.tokenMu.Unlock()
+		tokenCtx, cancel := context.WithTimeout(ctx, 1*time.Minute)
 		defer cancel()
-		req, err := http.NewRequestWithContext(ctx, http.MethodGet, a.baseURL+"/api/me", nil)
+		req, err := http.NewRequestWithContext(tokenCtx, http.MethodGet, a.baseURL+"/api/me", nil)
 		if err != nil {
-			a.logger.Fatalf("Failed to create request: %v", err)
-			return err
+			return fmt.Errorf("failed to create request: %w", err)
 		}
-		req.Header.Set("Authorization", a.token)
+		req.Header.Set("Authorization", a.authHeader())
 
 		resp, err := a.client.Do(req)
 		if err != nil {
-			a.logger.Fatalf("Failed to send request: %v", err)
-			return err
+			return fmt.Errorf("failed to send request: %w", err)
 		}
 		defer resp.Body.Close()
 		if resp.StatusCode != http.StatusOK {
-			a.logger.Fatalf("Failed to get alist user info: %s", resp.Status)
-			return err
+			return fmt.Errorf("failed to get alist user info: %s", resp.Status)
 		}
 		body, err := io.ReadAll(resp.Body)
 		if err != nil {
-			a.logger.Fatalf("Failed to read response body: %v", err)
-			return err
+			return fmt.Errorf("failed to read response body: %w", err)
 		}
 		var meResp meResponse
 		if err := json.Unmarshal(body, &meResp); err != nil {
-			a.logger.Fatalf("Failed to unmarshal me response: %v", err)
-			return err
+			return fmt.Errorf("failed to unmarshal me response: %w", err)
 		}
 		if meResp.Code != http.StatusOK {
-			a.logger.Fatalf("Failed to get alist user info: %s", meResp.Message)
-			return err
+			return fmt.Errorf("failed to get alist user info: %s", meResp.Message)
 		}
 		a.logger.Debugf("Logged in Alist as %s", meResp.Data.Username)
 		return nil
@@ -85,12 +93,15 @@ func (a *Alist) Init(ctx context.Context, cfg config.StorageConfig) error {
 	}
 
 	if err := a.getToken(ctx); err != nil {
-		a.logger.Fatalf("Failed to login to Alist: %v", err)
-		return err
+		return fmt.Errorf("failed to login to Alist: %w", err)
 	}
+	// The init login must not satisfy the refresh dedup window.
+	a.tokenMu.Lock()
+	a.lastLoginAt = time.Time{}
+	a.tokenMu.Unlock()
 	a.logger.Debug("Logged in to Alist")
 
-	go a.refreshToken(*alistConfig)
+	go a.refreshToken(ctx, *alistConfig)
 	return nil
 }
 
@@ -104,33 +115,40 @@ func (a *Alist) Name() string {
 
 func (a *Alist) Save(ctx context.Context, reader io.Reader, storagePath string) error {
 	a.logger.Infof("Saving file to %s", storagePath)
-	storagePath = a.JoinStoragePath(storagePath)
-	ext := path.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
-	candidate := storagePath
+	candidate := a.JoinStoragePath(storagePath)
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
-		for i := 1; a.existsPath(ctx, candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-		}
+		candidate = fsutil.UniquePath(a.config.BasePath, storagePath, func(c string) bool {
+			return a.existsPath(ctx, c)
+		}, 1000)
 	}
 
-	req, err := http.NewRequestWithContext(ctx, http.MethodPut, a.baseURL+"/api/fs/put", reader)
+	resp, err := a.putFile(ctx, reader, candidate)
 	if err != nil {
-		return fmt.Errorf("failed to create request: %w", err)
+		return err
 	}
-	req.Header.Set("Authorization", a.token)
-	req.Header.Set("File-Path", url.PathEscape(candidate))
-	req.Header.Set("Content-Type", "application/octet-stream")
-	if length := ctx.Value(ctxkey.ContentLength); length != nil {
-		length, ok := length.(int64)
-		if ok {
-			req.ContentLength = length
+	if resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden {
+		status := resp.Status
+		resp.Body.Close()
+		// Token-only storage cannot refresh: surface the auth error.
+		if a.loginInfo == nil {
+			return fmt.Errorf("failed to save file to Alist: %s", status)
+		}
+		if err := a.getToken(ctx); err != nil {
+			return fmt.Errorf("failed to refresh alist token: %w", err)
+		}
+		rs, seekable := reader.(io.ReadSeeker)
+		if !seekable {
+			a.logger.Warnf("Upload rejected with %s; reader is not seekable, cannot retry", status)
+			return fmt.Errorf("failed to save file to Alist: %s (streaming reader cannot be replayed for retry)", status)
+		}
+		if _, err := rs.Seek(0, io.SeekStart); err != nil {
+			return fmt.Errorf("failed to rewind reader before retry: %w", err)
+		}
+		a.logger.Info("Retrying upload with refreshed token")
+		resp, err = a.putFile(ctx, reader, candidate)
+		if err != nil {
+			return err
 		}
-	}
-
-	resp, err := a.client.Do(req)
-	if err != nil {
-		return fmt.Errorf("failed to send request: %w", err)
 	}
 	defer resp.Body.Close()
 
@@ -155,6 +173,29 @@ func (a *Alist) Save(ctx context.Context, reader io.Reader, storagePath string)
 	return nil
 }
 
+// putFile performs a single PUT upload with the given token and returns the response.
+func (a *Alist) putFile(ctx context.Context, reader io.Reader, storagePath string) (*http.Response, error) {
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut, a.baseURL+"/api/fs/put", reader)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create request: %w", err)
+	}
+	req.Header.Set("Authorization", a.authHeader())
+	req.Header.Set("File-Path", url.PathEscape(storagePath))
+	req.Header.Set("Content-Type", "application/octet-stream")
+	if length := ctx.Value(ctxkey.ContentLength); length != nil {
+		length, ok := length.(int64)
+		if ok {
+			req.ContentLength = length
+		}
+	}
+
+	resp, err := a.client.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("failed to send request: %w", err)
+	}
+	return resp, nil
+}
+
 func (a *Alist) JoinStoragePath(p string) string {
 	return path.Join(a.config.BasePath, p)
 }
@@ -189,7 +230,7 @@ func (a *Alist) existsPath(ctx context.Context, storagePath string) bool {
 		a.logger.Errorf("Failed to create request: %v", err)
 		return false
 	}
-	req.Header.Set("Authorization", a.token)
+	req.Header.Set("Authorization", a.authHeader())
 	req.Header.Set("Content-Type", "application/json")
 	resp, err := a.client.Do(req)
 	if err != nil {
@@ -244,7 +285,7 @@ func (a *Alist) ListFiles(ctx context.Context, dirPath string) ([]storagetypes.F
 	if err != nil {
 		return nil, fmt.Errorf("failed to create request: %w", err)
 	}
-	req.Header.Set("Authorization", a.token)
+	req.Header.Set("Authorization", a.authHeader())
 	req.Header.Set("Content-Type", "application/json")
 
 	resp, err := a.client.Do(req)
@@ -319,7 +360,7 @@ func (a *Alist) OpenFile(ctx context.Context, filePath string) (io.ReadCloser, i
 	if err != nil {
 		return nil, 0, fmt.Errorf("failed to create request: %w", err)
 	}
-	req.Header.Set("Authorization", a.token)
+	req.Header.Set("Authorization", a.authHeader())
 	req.Header.Set("Content-Type", "application/json")
 
 	resp, err := a.client.Do(req)

storage/alist/alist_test.go

@@ -0,0 +1,152 @@
+package alist_test
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"sync"
+	"testing"
+
+	storconfig "github.com/krau/SaveAny-Bot/config/storage"
+	"github.com/krau/SaveAny-Bot/storage/alist"
+)
+
+// newAlistServer starts a fake alist whose login endpoint issues sequential
+// tokens and whose PUT endpoint rejects the given token (simulating an expired
+// credential) while accepting refreshed ones.
+func newAlistServer(t *testing.T, rejectedToken string) (*httptest.Server, *sync.Mutex, *int, *[]putRecord) {
+	t.Helper()
+	var mu sync.Mutex
+	loginCount := 0
+	tokenSeq := 0
+	var puts []putRecord
+
+	mux := http.NewServeMux()
+	mux.HandleFunc("/api/auth/login", func(w http.ResponseWriter, r *http.Request) {
+		mu.Lock()
+		loginCount++
+		tokenSeq++
+		token := fmt.Sprintf("token-%d", tokenSeq)
+		mu.Unlock()
+		w.Header().Set("Content-Type", "application/json")
+		json.NewEncoder(w).Encode(map[string]any{"code": 200, "message": "ok", "data": map[string]any{"token": token}})
+	})
+	mux.HandleFunc("/api/me", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		json.NewEncoder(w).Encode(map[string]any{"code": 200, "message": "ok", "data": map[string]any{"username": "probe"}})
+	})
+	mux.HandleFunc("/api/fs/put", func(w http.ResponseWriter, r *http.Request) {
+		mu.Lock()
+		rejected := r.Header.Get("Authorization") == rejectedToken
+		puts = append(puts, putRecord{auth: r.Header.Get("Authorization"), rejected: rejected})
+		mu.Unlock()
+		w.Header().Set("Content-Type", "application/json")
+		if rejected {
+			w.WriteHeader(http.StatusUnauthorized)
+			json.NewEncoder(w).Encode(map[string]any{"code": 401, "message": "unauthorized"})
+			return
+		}
+		json.NewEncoder(w).Encode(map[string]any{"code": 200, "message": "ok"})
+	})
+	srv := httptest.NewServer(mux)
+	t.Cleanup(srv.Close)
+	return srv, &mu, &loginCount, &puts
+}
+
+type putRecord struct {
+	auth     string
+	rejected bool
+}
+
+// Regression: concurrent uploads hitting 401 must share a single re-login
+// (singleflight) and retry with the refreshed token. Init performs login #1
+// (token-1); the server rejects it, so the concurrent uploads must trigger a
+// second, merged login (token-2).
+func TestConcurrent401RetrySingleLogin(t *testing.T) {
+	srv, mu, loginCount, putAuths := newAlistServer(t, "token-1")
+
+	cfg := &storconfig.AlistStorageConfig{}
+	cfg.Name = "probe"
+	cfg.URL = srv.URL
+	cfg.Username = "user"
+	cfg.Password = "pass"
+	cfg.BasePath = "/probe"
+
+	stor := &alist.Alist{}
+	if err := stor.Init(t.Context(), cfg); err != nil {
+		t.Fatalf("Init failed: %v", err)
+	}
+
+	const workers = 10
+	var wg sync.WaitGroup
+	errs := make(chan error, workers)
+	for range workers {
+		wg.Go(func() {
+			errs <- stor.Save(t.Context(), bytes.NewReader([]byte("data")), "dir/file.txt")
+		})
+	}
+	wg.Wait()
+	close(errs)
+	for err := range errs {
+		if err != nil {
+			t.Fatalf("Save failed: %v", err)
+		}
+	}
+
+	mu.Lock()
+	defer mu.Unlock()
+	// One login for init, one merged login for the 401 storm.
+	if *loginCount != 2 {
+		t.Fatalf("expected 2 logins (init + merged retry), got %d", *loginCount)
+	}
+	accepted := 0
+	for _, put := range *putAuths {
+		if put.rejected {
+			if put.auth != "token-1" {
+				t.Fatalf("expected rejected uploads to use the expired token-1, got %q", put.auth)
+			}
+			continue
+		}
+		accepted++
+		if put.auth != "token-2" {
+			t.Fatalf("expected accepted uploads to use the refreshed token-2, got %q", put.auth)
+		}
+	}
+	if accepted != workers {
+		t.Fatalf("expected %d accepted uploads, got %d", workers, accepted)
+	}
+}
+
+// A token-only storage receives 401 and must return the auth error without
+// attempting a login (it has no credentials to refresh with).
+func TestTokenOnlyNoLoginOn401(t *testing.T) {
+	srv, mu, loginCount, _ := newAlistServer(t, "token-0")
+
+	cfg := &storconfig.AlistStorageConfig{}
+	cfg.Name = "probe"
+	cfg.URL = srv.URL
+	cfg.Token = "token-0"
+	cfg.BasePath = "/probe"
+
+	stor := &alist.Alist{}
+	if err := stor.Init(t.Context(), cfg); err != nil {
+		t.Fatalf("Init failed: %v", err)
+	}
+
+	err := stor.Save(t.Context(), bytes.NewReader([]byte("data")), "dir/file.txt")
+	if err == nil {
+		t.Fatal("expected auth error from token-only storage, got nil")
+	}
+	if !strings.Contains(err.Error(), "401") {
+		t.Fatalf("expected 401 auth error, got: %v", err)
+	}
+
+	mu.Lock()
+	defer mu.Unlock()
+	if *loginCount != 0 {
+		t.Fatalf("expected no login attempts for token-only storage, got %d", *loginCount)
+	}
+}

storage/alist/token.go

@@ -12,7 +12,42 @@ import (
 	config "github.com/krau/SaveAny-Bot/config/storage"
 )
 
+// minTokenRefreshInterval deduplicates login storms; it is capped by the
+// configured token expiry.
+const minTokenRefreshInterval = 30 * time.Second
+
+func (a *Alist) tokenRefreshWindow() time.Duration {
+	window := minTokenRefreshInterval
+	if exp := time.Duration(a.config.TokenExp) * time.Second; exp > 0 && exp < window {
+		window = exp
+	}
+	return window
+}
+
+// getToken refreshes the JWT, merging concurrent calls.
 func (a *Alist) getToken(ctx context.Context) error {
+	a.tokenMu.RLock()
+	fresh := !a.lastLoginAt.IsZero() && time.Since(a.lastLoginAt) < a.tokenRefreshWindow()
+	a.tokenMu.RUnlock()
+	if fresh {
+		return nil
+	}
+	_, err, _ := a.tokenFlight.Do("token", func() (any, error) {
+		a.tokenMu.RLock()
+		fresh := !a.lastLoginAt.IsZero() && time.Since(a.lastLoginAt) < a.tokenRefreshWindow()
+		a.tokenMu.RUnlock()
+		if fresh {
+			return nil, nil
+		}
+		return nil, a.fetchToken(ctx)
+	})
+	return err
+}
+
+func (a *Alist) fetchToken(ctx context.Context) error {
+	if a.loginInfo == nil {
+		return fmt.Errorf("token-only alist storage cannot refresh credentials")
+	}
 	loginBody, err := json.Marshal(a.loginInfo)
 	if err != nil {
 		return fmt.Errorf("failed to marshal login request: %w", err)
@@ -44,22 +79,31 @@ func (a *Alist) getToken(ctx context.Context) error {
 		return fmt.Errorf("%w: %s", ErrAlistLoginFailed, loginResp.Message)
 	}
 
+	a.tokenMu.Lock()
 	a.token = loginResp.Data.Token
+	a.lastLoginAt = time.Now()
+	a.tokenMu.Unlock()
 	return nil
 }
 
-func (a *Alist) refreshToken(cfg config.AlistStorageConfig) {
+func (a *Alist) refreshToken(ctx context.Context, cfg config.AlistStorageConfig) {
 	tokenExp := cfg.TokenExp
 	if tokenExp <= 0 {
 		a.logger.Warn("Invalid token expiration time, using default value")
 		tokenExp = 3600
 	}
+	ticker := time.NewTicker(time.Duration(tokenExp) * time.Second)
+	defer ticker.Stop()
 	for {
-		time.Sleep(time.Duration(tokenExp) * time.Second)
-		if err := a.getToken(context.Background()); err != nil {
-			a.logger.Errorf("Failed to refresh jwt token: %v", err)
-			continue
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			if err := a.getToken(ctx); err != nil {
+				a.logger.Errorf("Failed to refresh jwt token: %v", err)
+				continue
+			}
+			a.logger.Info("Refreshed Alist jwt token")
 		}
-		a.logger.Info("Refreshed Alist jwt token")
 	}
 }

storage/load.go

@@ -3,13 +3,44 @@ package storage
 import (
 	"context"
 	"fmt"
+	"maps"
+	"sync"
 
 	"github.com/charmbracelet/log"
 	"github.com/krau/SaveAny-Bot/config"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
+	"golang.org/x/sync/singleflight"
 )
 
-var UserStorages = make(map[int64][]Storage)
+var (
+	storageMu sync.RWMutex
+	// Storages maps storage names to initialized storage instances.
+	Storages = make(map[string]Storage)
+
+	userStoragesMu sync.RWMutex
+	// UserStorages maps user IDs to their available storage instances.
+	UserStorages = make(map[int64][]Storage)
+
+	initFlight singleflight.Group
+)
+
+// GetStorage returns the initialized storage instance for name, without
+// creating one on demand.
+func GetStorage(name string) (Storage, bool) {
+	storageMu.RLock()
+	defer storageMu.RUnlock()
+	s, ok := Storages[name]
+	return s, ok
+}
+
+// AllStorages returns a snapshot copy of all initialized storages.
+func AllStorages() map[string]Storage {
+	storageMu.RLock()
+	defer storageMu.RUnlock()
+	out := make(map[string]Storage, len(Storages))
+	maps.Copy(out, Storages)
+	return out
+}
 
 // GetStorageByName returns storage by name from cache or creates new one
 // It should NOT be used to get storage for user, use GetStorageByUserIDAndName instead
@@ -18,21 +49,41 @@ func GetStorageByName(ctx context.Context, name string) (Storage, error) {
 		return nil, ErrStorageNameEmpty
 	}
 
+	storageMu.RLock()
 	storage, ok := Storages[name]
+	storageMu.RUnlock()
 	if ok {
 		return storage, nil
 	}
 	cfg := config.C().GetStorageByName(name)
 	if cfg == nil {
-		return nil, fmt.Errorf("未找到存储 %s", name)
+		return nil, fmt.Errorf("storage %s not found", name)
 	}
 
-	storage, err := NewStorage(ctx, cfg)
+	// Merge concurrent first-time initializations.
+	v, err, _ := initFlight.Do("storage:"+name, func() (any, error) {
+		storageMu.RLock()
+		if existing, ok := Storages[name]; ok {
+			storageMu.RUnlock()
+			return existing, nil
+		}
+		storageMu.RUnlock()
+		storage, err := NewStorage(ctx, cfg)
+		if err != nil {
+			return nil, err
+		}
+		storageMu.Lock()
+		defer storageMu.Unlock()
+		if existing, ok := Storages[name]; ok {
+			return existing, nil
+		}
+		Storages[name] = storage
+		return storage, nil
+	})
 	if err != nil {
 		return nil, err
 	}
-	Storages[name] = storage
-	return storage, nil
+	return v.(Storage), nil
 }
 
 // 检查 user 是否可用指定的 storage, 若不可用则返回未找到错误
@@ -52,8 +103,11 @@ func GetUserStorages(ctx context.Context, chatID int64) []Storage {
 	if chatID <= 0 {
 		return nil
 	}
-	if storages, ok := UserStorages[chatID]; ok {
-		return storages
+	userStoragesMu.RLock()
+	cached, ok := UserStorages[chatID]
+	userStoragesMu.RUnlock()
+	if ok {
+		return cached
 	}
 	var storages []Storage
 	for _, name := range config.C().GetStorageNamesByUserID(chatID) {
@@ -75,9 +129,16 @@ func LoadStorages(ctx context.Context) {
 			logger.Errorf("failed to load storage %s: %v", storage.GetName(), err)
 		}
 	}
-	logger.Infof("successfully loaded %d storages", len(Storages))
+	storageMu.RLock()
+	loaded := len(Storages)
+	storageMu.RUnlock()
+	logger.Infof("successfully loaded %d storages", loaded)
 	for user := range config.C().GetUsersID() {
-		UserStorages[int64(user)] = GetUserStorages(ctx, int64(user))
+		uid := int64(user)
+		storages := GetUserStorages(ctx, uid)
+		userStoragesMu.Lock()
+		UserStorages[uid] = storages
+		userStoragesMu.Unlock()
 	}
 }
 

storage/local/local.go

@@ -10,6 +10,7 @@ import (
 
 	"github.com/charmbracelet/log"
 	"github.com/duke-git/lancet/v2/fileutil"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	config "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
@@ -52,15 +53,17 @@ func (l *Local) JoinStoragePath(path string) string {
 
 func (l *Local) Save(ctx context.Context, r io.Reader, storagePath string) error {
 	l.logger.Infof("Saving file to %s", storagePath)
-	storagePath = l.JoinStoragePath(storagePath)
+	storagePath = filepath.Clean(storagePath)
+	if filepath.IsAbs(storagePath) {
+		return fmt.Errorf("local: storage path must be relative: %s", storagePath)
+	}
+	if storagePath == ".." || strings.HasPrefix(storagePath, ".."+string(filepath.Separator)) {
+		return fmt.Errorf("local: storage path escapes base directory: %s", storagePath)
+	}
 
-	ext := filepath.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
-	candidate := storagePath
+	candidate := l.JoinStoragePath(storagePath)
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
-		for i := 1; l.existsPath(candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-		}
+		candidate = fsutil.UniquePath(l.config.BasePath, storagePath, l.existsPath, 1000)
 	}
 
 	absPath, err := filepath.Abs(candidate)
@@ -68,13 +71,17 @@ func (l *Local) Save(ctx context.Context, r io.Reader, storagePath string) error
 		return err
 	}
 	if err := fileutil.CreateDir(filepath.Dir(absPath)); err != nil {
-		return err
+		return fmt.Errorf("failed to create directory: %w", err)
 	}
 	file, err := os.Create(absPath)
 	if err != nil {
-		return err
+		return fmt.Errorf("failed to create file: %w", err)
 	}
-	defer file.Close()
+	defer func() {
+		if err := file.Close(); err != nil {
+			l.logger.Errorf("Failed to close file %s: %v", absPath, err)
+		}
+	}()
 	_, err = io.Copy(file, r)
 	return err
 }

storage/minio/client.go

@@ -11,12 +11,12 @@ import (
 	"sync"
 
 	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	config "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
 	"github.com/minio/minio-go/v7"
 	"github.com/minio/minio-go/v7/pkg/credentials"
-	"github.com/rs/xid"
 )
 
 var (
@@ -77,19 +77,11 @@ func (m *Minio) JoinStoragePath(p string) string {
 
 func (m *Minio) Save(ctx context.Context, r io.Reader, storagePath string) error {
 	m.logger.Infof("Saving file from reader to %s", storagePath)
-	storagePath = m.JoinStoragePath(storagePath)
-	ext := path.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
-	candidate := storagePath
+	candidate := m.JoinStoragePath(storagePath)
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
-		for i := 1; m.existsObject(ctx, candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-			if i > 10 {
-				m.logger.Errorf("Too many attempts to find a unique filename for %s", storagePath)
-				candidate = fmt.Sprintf("%s_%s%s", base, xid.New().String(), ext)
-				break
-			}
-		}
+		candidate = fsutil.UniquePath(strings.TrimPrefix(m.config.BasePath, "/"), storagePath, func(c string) bool {
+			return m.existsObject(ctx, c)
+		}, 10)
 	}
 	size := int64(-1)
 	if length := ctx.Value(ctxkey.ContentLength); length != nil {

storage/rclone/errs.go

@@ -8,7 +8,4 @@ var (
 	ErrFailedToSaveFile  = errors.New("rclone: failed to save file")
 	ErrFailedToListFiles = errors.New("rclone: failed to list files")
 	ErrFailedToOpenFile  = errors.New("rclone: failed to open file")
-	ErrFailedToCheckFile = errors.New("rclone: failed to check file exists")
-	ErrFailedToCreateDir = errors.New("rclone: failed to create directory")
-	ErrCommandFailed     = errors.New("rclone: command execution failed")
 )

storage/rclone/rclone.go

@@ -13,11 +13,11 @@ import (
 	"time"
 
 	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	config "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
 	"github.com/krau/SaveAny-Bot/pkg/storagetypes"
-	"github.com/rs/xid"
 )
 
 type Rclone struct {
@@ -51,9 +51,6 @@ func (r *Rclone) Init(ctx context.Context, cfg config.StorageConfig) error {
 	}
 
 	remoteName := strings.TrimSuffix(r.config.Remote, ":")
-	if !strings.HasSuffix(r.config.Remote, ":") {
-		remoteName = r.config.Remote
-	}
 
 	found := false
 	scanner := bufio.NewScanner(bytes.NewReader(output))
@@ -105,18 +102,11 @@ func (r *Rclone) getRemotePath(storagePath string) string {
 func (r *Rclone) Save(ctx context.Context, reader io.Reader, storagePath string) error {
 	r.logger.Infof("Saving file to %s", storagePath)
 
-	ext := path.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
 	candidate := storagePath
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
-		for i := 1; r.Exists(ctx, candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-			if i > 100 {
-				r.logger.Errorf("Too many attempts to find a unique filename for %s", storagePath)
-				candidate = fmt.Sprintf("%s_%s%s", base, xid.New().String(), ext)
-				break
-			}
-		}
+		candidate = fsutil.UniquePath("", storagePath, func(c string) bool {
+			return r.Exists(ctx, c)
+		}, 100)
 	}
 
 	remotePath := r.getRemotePath(candidate)
@@ -285,6 +275,12 @@ func (r *rcloneCatReader) Close() error {
 	if err := r.reader.Close(); err != nil {
 		r.logger.Warnf("Failed to close reader: %v", err)
 	}
+	// Kill the cat process so Wait cannot block on a hung pipe.
+	if r.cmd.Process != nil {
+		if err := r.cmd.Process.Kill(); err != nil {
+			r.logger.Warnf("Failed to kill rclone cat process: %v", err)
+		}
+	}
 	if err := r.cmd.Wait(); err != nil {
 		r.logger.Warnf("rclone cat process exited with error: %v", err)
 	}

storage/s3/s3.go

@@ -8,11 +8,11 @@ import (
 	"strings"
 
 	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	storconfig "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
 	"github.com/krau/SaveAny-Bot/pkg/s3"
-	"github.com/rs/xid"
 )
 
 type S3 struct {
@@ -65,21 +65,13 @@ func (m *S3) JoinStoragePath(p string) string {
 
 func (m *S3) Save(ctx context.Context, r io.Reader, storagePath string) error {
 	m.logger.Infof("Saving file from reader to %s", storagePath)
-	storagePath = m.JoinStoragePath(storagePath)
-	ext := path.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
-	candidate := storagePath
+	candidate := m.JoinStoragePath(storagePath)
 
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
 		// Unique filename
-		for i := 1; m.existsKey(ctx, candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-			if i > 10 {
-				m.logger.Errorf("Too many attempts for unique filename: %s", storagePath)
-				candidate = fmt.Sprintf("%s_%s%s", base, xid.New().String(), ext)
-				break
-			}
-		}
+		candidate = fsutil.UniquePath(strings.TrimPrefix(m.config.BasePath, "/"), storagePath, func(c string) bool {
+			return m.existsKey(ctx, c)
+		}, 10)
 	}
 
 	// Determine content length

storage/storage.go

@@ -75,11 +75,18 @@ type StorageReadable interface {
 	OpenFile(ctx context.Context, filePath string) (io.ReadCloser, int64, error)
 }
 
-var Storages = make(map[string]Storage)
-
 var _ StorageProgressSaver = (*telegram.Telegram)(nil)
 var _ StorageBatchProgressSaver = (*telegram.Telegram)(nil)
 
+var _ StorageListable = (*alist.Alist)(nil)
+var _ StorageReadable = (*alist.Alist)(nil)
+var _ StorageListable = (*local.Local)(nil)
+var _ StorageReadable = (*local.Local)(nil)
+var _ StorageListable = (*rclone.Rclone)(nil)
+var _ StorageReadable = (*rclone.Rclone)(nil)
+var _ StorageListable = (*webdav.Webdav)(nil)
+var _ StorageReadable = (*webdav.Webdav)(nil)
+
 type StorageConstructor func() Storage
 
 var storageConstructors = map[storenum.StorageType]StorageConstructor{

storage/telegram/telegram.go

@@ -83,6 +83,9 @@ func (t *Telegram) Name() string {
 	return t.config.Name
 }
 
+// Exists always reports false: Telegram offers no reliable way to query
+// whether a file already exists in a chat, so conflict policies do not apply
+// to this backend.
 func (t *Telegram) Exists(ctx context.Context, storagePath string) bool {
 	return false
 }
@@ -447,7 +450,7 @@ func planMediaGroups(items []batchMediaItem) [][]batchMediaItem {
 			continue
 		}
 		end := i + 1
-		for end < len(items) && end-i < 10 {
+		for end < len(items) && end-i < tglimit.MaxAlbumItems {
 			next := items[end]
 			if next.useSingleSave || !next.albumEligible || next.chatID != item.chatID || next.item.SourceGroupKey != item.item.SourceGroupKey {
 				break
@@ -623,16 +626,16 @@ func (t *Telegram) splitUpload(
 
 	sender := ctx.Sender
 
-	if len(multiMedia) <= 10 {
+	if len(multiMedia) <= tglimit.MaxAlbumItems {
 		_, err = sender.WithUploader(upler).
 			To(peer).
 			Album(ctx, multiMedia[0], multiMedia[1:]...)
 		return err
 	}
 
-	// more than 10 parts, send in batches, each batch up to 10 parts
-	for i := 0; i < len(multiMedia); i += 10 {
-		end := min(i+10, len(multiMedia))
+	// more than MaxAlbumItems parts, send in batches, each batch up to MaxAlbumItems parts
+	for i := 0; i < len(multiMedia); i += tglimit.MaxAlbumItems {
+		end := min(i+tglimit.MaxAlbumItems, len(multiMedia))
 		batch := multiMedia[i:end]
 		_, err = sender.WithUploader(upler).
 			To(peer).

storage/telegram/video_split.go

@@ -19,13 +19,14 @@ import (
 	"github.com/rs/xid"
 
 	"github.com/krau/SaveAny-Bot/config"
+	"github.com/krau/SaveAny-Bot/pkg/consts/tglimit"
 )
 
 const (
 	videoPartTargetRatio  = 0.95
 	videoSplitAttempts    = 4
 	minSegmentDuration    = 1.0
-	maxLosslessVideoParts = 10
+	maxLosslessVideoParts = tglimit.MaxAlbumItems
 )
 
 type losslessVideoPart struct {

storage/webdav/errs.go

@@ -1,9 +0,0 @@
-package webdav
-
-import "errors"
-
-var (
-	ErrFailedToCreateDirectory = errors.New("webdav: failed to create directory")
-	ErrFailedToWriteFile       = errors.New("webdav: failed to write file")
-	ErrFailedToCheckFileExists = errors.New("webdav: failed to check if file exists")
-)

storage/webdav/webdav.go

@@ -11,11 +11,11 @@ import (
 	"time"
 
 	"github.com/charmbracelet/log"
+	"github.com/krau/SaveAny-Bot/common/utils/fsutil"
 	config "github.com/krau/SaveAny-Bot/config/storage"
 	"github.com/krau/SaveAny-Bot/pkg/enums/ctxkey"
 	storenum "github.com/krau/SaveAny-Bot/pkg/enums/storage"
 	"github.com/krau/SaveAny-Bot/pkg/storagetypes"
-	"github.com/rs/xid"
 )
 
 type Webdav struct {
@@ -54,28 +54,18 @@ func (w *Webdav) JoinStoragePath(p string) string {
 
 func (w *Webdav) Save(ctx context.Context, r io.Reader, storagePath string) error {
 	w.logger.Infof("Saving file to %s", storagePath)
-	storagePath = w.JoinStoragePath(storagePath)
-	ext := path.Ext(storagePath)
-	base := strings.TrimSuffix(storagePath, ext)
-	candidate := storagePath
+	candidate := w.JoinStoragePath(storagePath)
 	if overwrite, _ := ctx.Value(ctxkey.OverwriteExisting).(bool); !overwrite {
-		for i := 1; w.existsPath(ctx, candidate); i++ {
-			candidate = fmt.Sprintf("%s_%d%s", base, i, ext)
-			if i > 1000 {
-				w.logger.Errorf("Too many attempts to find a unique filename for %s", storagePath)
-				candidate = fmt.Sprintf("%s_%s%s", base, xid.New().String(), ext)
-				break
-			}
-		}
+		candidate = fsutil.UniquePath(w.config.BasePath, storagePath, func(c string) bool {
+			return w.existsPath(ctx, c)
+		}, 1000)
 	}
 
 	if err := w.client.MkDir(ctx, path.Dir(candidate)); err != nil {
-		w.logger.Errorf("Failed to create directory %s: %v", path.Dir(candidate), err)
-		return ErrFailedToCreateDirectory
+		return fmt.Errorf("failed to create directory: %w", err)
 	}
 	if err := w.client.WriteFile(ctx, candidate, r); err != nil {
-		w.logger.Errorf("Failed to write file %s: %v", candidate, err)
-		return ErrFailedToWriteFile
+		return fmt.Errorf("failed to write file: %w", err)
 	}
 	return nil
 }
@@ -136,9 +126,6 @@ func (w *Webdav) ListFiles(ctx context.Context, dirPath string) ([]storagetypes.
 
 		isDir := resp.Propstat.Prop.ResourceType.IsCollection()
 
-		filePath := strings.TrimPrefix(decodedHref, path.Join("/", strings.Trim(path.Dir(fullPath), "/")))
-		filePath = strings.TrimPrefix(filePath, "/")
-
 		fileInfo := storagetypes.FileInfo{
 			Name:    name,
 			Path:    path.Join(dirPath, name),