Commit Graph
45 Commits
Author SHA1 Message Date
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00
krau 7e3e26fcf4 style: format codebase 2026-08-11 08:47:35 +08:00
Haopeng Huo e4144e73e6 feat(telegram): split oversized videos into playable parts (#229)
* feat(telegram): split oversized videos losslessly

* fix(telegram): respect Premium upload limits

* fix(telegram): keep split videos in one album
2026-08-05 16:57:46 +08:00
Haopeng Huo 1d794a7b9c feat(telegram): preserve source media groups (#226)
* feat(telegram): preserve source media groups

* fix(telegram): rewind batch readers before inspection
2026-08-04 16:46:29 +08:00
krau 2bc460c609 feat: add preset rule import functionality and update related messages 2026-06-26 15:39:14 +08:00
Krau f02860ff3f feat: implement task event system for progress tracking and reporting (#220) 2026-06-25 21:37:36 +08:00
HLD bfab4c85c8 fix: correct S3 signature and path handling (#213)
* fix: correct S3 signature and path handling

* fix: preserve existing overwrite behavior
2026-05-22 09:22:09 +08:00
Ryan 62e4a08e28 Merge pull request #204 from Rain-kl/feat/save-strategy
新增功能: 重名文件时选择保存策略 - 重命名,覆盖,跳过
2026-05-20 16:11:00 +08:00
krau 602fc251d8 fix: upgrade deps and medernize codes 2026-03-05 19:40:25 +08:00
krau f377ee3ca4 chore: format code for consistency and readability 2026-03-05 17:20:45 +08:00
Krau 8b86330f5c feat: add rclone storage backend (#191)
* fix: update StoragePath method to return specific path for single file

* feat: add Rclone storage support with configuration and file operations

* docs: add Rclone support to documentation for configuration and usage
2026-01-30 13:34:29 +08:00
krau 6990543c9f feat: update transfer command to remove target path requirement and adjust usage instructions 2026-01-19 21:20:27 +08:00
krau dd0dea8cb5 feat!: Refactor batch import task to transfer task
- Updated command usage in English and Simplified Chinese localization files to reflect changes in transfer command syntax.
- Removed batch import task implementation, replacing it with a new transfer task implementation.
- Introduced new task structure and progress tracking for file transfers.
- Updated task type enumeration to replace batch import with transfer.
- Added new fields in data structures to support transfer operations.
- Implemented file handling and progress reporting for the transfer task.
2026-01-19 21:14:01 +08:00
KrauandCopilot f92c43b9c8 feat: support import files from storages to telegram (#183)
* feat: add import command and batch import functionality

- Implemented the `/import` command to allow users to import files from storage to Telegram.
- Added support for listing files in storage and filtering based on regex patterns.
- Created a batch import task to handle multiple file uploads concurrently.
- Introduced progress tracking for batch imports, providing real-time updates to users.
- Enhanced storage interfaces to support file listing and reading capabilities.
- Updated localization files for the new import command and its usage instructions.
- Added utility functions for file size formatting and speed calculation.
- Refactored Telegram storage handling to support reading from non-seekable streams.

* feat: add  i18n for import command

* feat: implement ListFiles and OpenFile methods for WebDAV and Alist storage

* Update common/i18n/locale/zh-Hans.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/progress.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/alist/alist.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update pkg/storagetypes/fileinfo.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/telegram/telegram.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: missing progress stats i18n

* refactor: use strutil to parse args

* chore: update generated code files for consistency

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-19 17:39:47 +08:00
Copilotkraucopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
3e20dc2c5f feat: add custom parameter support to /ytdlp command (#185), close #184
* Initial plan

* Implement parameter support for /ytdlp command

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Add comprehensive tests for ytdlp parameter parsing

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Improve flag parsing logic and clarify argument order

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Preserve critical defaults and improve comments

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: krau <71133316+krau@users.noreply.github.com>
2026-01-19 13:10:21 +08:00
krau 3ce00884a0 feat: add yt-dlp support for downloading video/audio and enhance related commands 2026-01-17 17:42:11 +08:00
krau b05d86509c feat: basic aria2 integration 2026-01-17 14:57:03 +08:00
krau 63aeabb39b feat: add msgraw var to filename template and update i18n for fnamest display 2026-01-06 09:19:32 +08:00
krau 75e5fd10ea feat(aria2): add Aria2 download command and client integration 2026-01-03 17:40:55 +08:00
krau 8dd6265d55 chore: modenrize it 2025-12-27 16:01:41 +08:00
krau 0af049a507 feat: migrate S3 client implementation to a new package structure 2025-12-18 16:42:58 +08:00
krau 0ca3d97711 feat: add task command to client and Title method to Task for tasks queue managing, #157 2025-12-15 11:29:55 +08:00
krau 51198a1e3d fix: remove redundant cancellation in Done method 2025-12-15 11:15:17 +08:00
krau 651835c467 feat: refactor queue to remove unused methods and add comments 2025-12-15 10:49:40 +08:00
krau c21ff7e499 feat: add direct links download functionality
- Implemented a new task type for handling direct links downloads.
- Added command handler for downloading multiple links via /dl command.
- Introduced progress tracking for direct link downloads.
- Enhanced filename parsing to support various encoding scenarios.
- Updated enums to include direct links as a task type.
- Refactored existing task structures to accommodate new functionality.
- Improved error handling and logging throughout the download process.
2025-12-08 17:10:41 +08:00
krau 91814a83c7 fix: deprecate minio and introduce s3 storage backend 2025-12-04 22:59:23 +08:00
krau 685047e463 fix: compatibility between tdlib and bot api style chatID 2025-12-04 22:43:22 +08:00
krau acd16a91a3 chore: re gen enum code using new version go-enum 2025-11-28 10:27:40 +08:00
krau 3f40acff55 feat: add directory management functionality and update handlers to utilize default directory 2025-11-16 21:09:55 +08:00
krau f80ecae3cc feat: add Playwright support for browser automation in plugins
- Updated .dockerignore and .gitignore to exclude Playwright-related files.
- Added Playwright-Go dependency in go.mod and updated go.sum.
- Implemented jsPlaywright function in js_api.go for browser-based requests.
- Enhanced README.md to document the new Playwright functionality for plugins.
2025-11-07 11:07:47 +08:00
krau f947ee6fc7 fix: implement filename strategy in userbot listen mode 2025-09-28 16:49:32 +08:00
Krau eef051de3b feat: custom filename template (#110) 2025-09-13 10:25:45 +08:00
krau 46c21b77e9 feat: enhance Save method to validate chat ID and adjust ForceFile logic for image uploads 2025-08-24 12:02:27 +08:00
krau 68e5a51300 feat: file name staregy 2025-08-23 17:16:51 +08:00
krau 7300e54c40 refactor: rule package 2025-08-23 16:14:12 +08:00
krau 94f796d0e8 refactor: move version to config package 2025-08-23 16:10:02 +08:00
krau e5d1e143e0 feat: configurable parser and refactor config 2025-08-23 14:29:32 +08:00
krau 03eb4f8a18 feat: update parser interface to include context in Parse method 2025-08-23 14:01:00 +08:00
Krau 302db2fe75 feat: parse url with js plugins support (#96)
* feat: WIP. add parser functionality and text message handling

* fix: use json to marshal js result

* feat: add metadata handling and version validation for jsParser

* refactor: rename parser package to parsers and restructure parser handling

* refactor: core code struct and impl parse task handle

* feat: impl parsed download

* fix: seek cache file when processing tph picture

* feat: implement parsed task handling and progress tracking

* feat: enhance task processing with concurrency control and progress tracking

* feat: add resource ID generation and improve resource processing handling

* feat: improve message formatting in parsed text and progress completion

* feat: add example js plugin

* feat: implement Twitter parser

* fix: twitter parse video json decode error

* feat: impl stream mode for parse task
2025-08-21 23:48:17 +08:00
krau b9d14f79c8 refactor: simplify dler client interface 2025-08-02 10:01:59 +08:00
krau 6423fb25a7 fix: update max upload part size to use uploader's maximum value and adjust document upload options 2025-07-28 14:58:01 +08:00
krau d703f11ea0 feat: implement album handling rules and refactor related logic 2025-07-04 16:01:29 +08:00
krau f693bd6103 refactor: update file handling to use new downloader interface; remove unused tdler package 2025-06-29 23:00:40 +08:00
krau 4df2c5a06d refactor: replace key package with ctxkey for context keys 2025-06-17 22:19:06 +08:00
Krau 900823cdb9 refactor: refactor task logic for better scalability (#76)
* refactor: a big refactor. wip

* refactor: port handle file

* refactor: place all handlers

* fix: task info nil pointer

* feat: enhance task progress tracking and context management

* feat: cancel task

* feat: stream mode

* feat: silent mode

* feat: dir cmd

* refactor: remove unused old file

* feat: rule cmd

* feat: handle silent mode

* feat: batch task

* fix: batch task progress and temp file cleanup

* refactor: update file creation and cleanup methods for better resource management

* feat: add save command with silent mode handling

* feat: message link

* feat: update message prompts to include file count in storage selection

* feat: slient save links

* refactor: reduce dup code

* feat: rule type

* feat: chose dir

* feat: refactor file handling and storage rules, improve error handling and logging

* feat: rule mode

* feat: telegraph pics

* fix: tphpics nil pointer and inaccurate dirpath

* feat: silent save telegraph

* feat: add suffix to avoid file overwrite

* feat: new storage telegram

* chore: tidy go mod
2025-06-15 23:57:49 +08:00