Commit Graph
28 Commits
Author SHA1 Message Date
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00
Haopeng Huo 0e6ed66ef5 feat: show upload progress for batch file tasks (#228)
* feat: show upload progress for file tasks

* feat: show upload progress for batch file tasks

* style: distinguish download and upload phases

* style: mark successful task completion

* fix: preserve storage save error context

* fix: serialize single-file progress updates

* fix: stabilize batch upload progress reporting

Correct batch upload totals, completion state, actual file sizes, and synchronized progress snapshots. Add regression coverage for concurrent updates and phase transitions.

* feat: show per-file transfer progress

Format single-file and batch download and upload states with Telegram entities, blockquotes, speeds, transferred sizes, progress bars, concise counters, and accurate confirmation handling. Cover upload retries plus final, error, and cancellation messages with regression tests.

* chore: remove transfer progress tests

* test: restore critical transfer progress coverage

* test: cover interleaved batch transfers

* fix: declare progress styles in locale templates
2026-08-11 08:40:29 +08:00
Haopeng Huo 1d794a7b9c feat(telegram): preserve source media groups (#226)
* feat(telegram): preserve source media groups

* fix(telegram): rewind batch readers before inspection
2026-08-04 16:46:29 +08:00
Krau 8b86330f5c feat: add rclone storage backend (#191)
* fix: update StoragePath method to return specific path for single file

* feat: add Rclone storage support with configuration and file operations

* docs: add Rclone support to documentation for configuration and usage
2026-01-30 13:34:29 +08:00
krau 9b52a3e0ce refactor: simplify storage path handling across various tasks and storage implementations 2026-01-19 21:27:53 +08:00
KrauandCopilot f92c43b9c8 feat: support import files from storages to telegram (#183)
* feat: add import command and batch import functionality

- Implemented the `/import` command to allow users to import files from storage to Telegram.
- Added support for listing files in storage and filtering based on regex patterns.
- Created a batch import task to handle multiple file uploads concurrently.
- Introduced progress tracking for batch imports, providing real-time updates to users.
- Enhanced storage interfaces to support file listing and reading capabilities.
- Updated localization files for the new import command and its usage instructions.
- Added utility functions for file size formatting and speed calculation.
- Refactored Telegram storage handling to support reading from non-seekable streams.

* feat: add  i18n for import command

* feat: implement ListFiles and OpenFile methods for WebDAV and Alist storage

* Update common/i18n/locale/zh-Hans.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/progress.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/alist/alist.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update pkg/storagetypes/fileinfo.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/telegram/telegram.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: missing progress stats i18n

* refactor: use strutil to parse args

* chore: update generated code files for consistency

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-19 17:39:47 +08:00
krau 4ccbf8177e fix(i18n): Update log messages to English and improve error handling in bot initialization and storage loading 2025-12-19 20:29:48 +08:00
krau 5f7b936270 feat: refactor storage retrieval functions and improve documentation 2025-12-19 13:49:20 +08:00
krau 91814a83c7 fix: deprecate minio and introduce s3 storage backend 2025-12-04 22:59:23 +08:00
Krau 900823cdb9 refactor: refactor task logic for better scalability (#76)
* refactor: a big refactor. wip

* refactor: port handle file

* refactor: place all handlers

* fix: task info nil pointer

* feat: enhance task progress tracking and context management

* feat: cancel task

* feat: stream mode

* feat: silent mode

* feat: dir cmd

* refactor: remove unused old file

* feat: rule cmd

* feat: handle silent mode

* feat: batch task

* fix: batch task progress and temp file cleanup

* refactor: update file creation and cleanup methods for better resource management

* feat: add save command with silent mode handling

* feat: message link

* feat: update message prompts to include file count in storage selection

* feat: slient save links

* refactor: reduce dup code

* feat: rule type

* feat: chose dir

* feat: refactor file handling and storage rules, improve error handling and logging

* feat: rule mode

* feat: telegraph pics

* fix: tphpics nil pointer and inaccurate dirpath

* feat: silent save telegraph

* feat: add suffix to avoid file overwrite

* feat: new storage telegram

* chore: tidy go mod
2025-06-15 23:57:49 +08:00
krau e85d3c9441 feat: rename file only when storagePath exists 2025-06-11 09:54:08 +08:00
krau 491ba55f1e feat: add support for handling unsupported stream storage in download process 2025-03-26 10:35:40 +08:00
krau 2d2becccf6 refactor: update storage interface to use io.Reader for Save method and remove stream implementations 2025-03-21 23:05:09 +08:00
krau ed0837a89b refactor: replace logger usage with common.Log for consistent logging 2025-03-21 21:07:53 +08:00
krau 65fee89e14 feat: refactor storage configuration to use dedicated storage package and add new storage types
BREAKING CHANGE: remove deprecated config
2025-03-21 20:52:41 +08:00
ysicing 495ad3ea5c feat: add Minio storage support
Signed-off-by: ysicing <i@ysicing.me>
2025-03-11 21:29:35 +08:00
krau 8e2dd37155 feat: add stream upload support and related configurations 2025-02-28 11:09:24 +08:00
krau 27dca2e343 perf: add UserStorages map and implement GetUserStorages function for user-specific storage retrieval 2025-02-20 22:57:45 +08:00
krau 3a1b8f34ea chore: translate some import log to cn 2025-02-19 12:36:48 +08:00
krau c4eb824457 feat: set default storage by inline keyboard 2025-02-19 12:23:12 +08:00
krau 692e970772 feat!: (WIP) switched back to using config files config storages because the conversation handling is shit 2025-02-19 11:05:30 +08:00
krauandAHCorn 80696c9661 feat: (WIP) add storage
Co-authored-by: AHCorn <42889600+AHCorn@users.noreply.github.com>
2025-02-18 22:53:07 +08:00
krau dfde65c28e feat: (WIP) migrate storage configuration to user-specific models and remove deprecated storage loading 2025-02-18 19:45:06 +08:00
krau 968547b005 feat!: (WIP) decouple storage, users, and configuration files to support multiple users 2025-02-18 17:17:02 +08:00
krau 3a4effab33 feat: refactor file processing and storage handling with improved path management 2025-02-15 15:06:06 +08:00
krau fbdfc04ad8 refactor: migrate to gotd (wip) 2024-11-08 23:00:57 +08:00
krau b3c59fde72 feat: webdav storage 2024-10-11 00:07:11 +08:00
krau 1cbcebc148 feat: init commit 2024-10-10 23:31:01 +08:00