Commit Graph
89 Commits
Author SHA1 Message Date
krau 8bdafd115c fix(tdler): validate resume state against the part file 2026-08-25 15:22:59 +08:00
krau 991f454096 feat(core): persist tasks and recover them after restart 2026-08-25 15:05:55 +08:00
krau 54dc4caafe feat(tdler): resumable chunked download with block bitmap 2026-08-25 15:05:49 +08:00
krau b0850fb5e5 fix(batch): show total size in progress header again
The #228 progress redesign dropped the total size that the old
batch progress message displayed. Add a TotalSize data field
(actual known sizes, formatted via dlutil.FormatSize) to the
batch status header template in both locales.
2026-08-22 17:39:55 +08:00
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00
krau 7e3e26fcf4 style: format codebase 2026-08-11 08:47:35 +08:00
Haopeng Huo 0e6ed66ef5 feat: show upload progress for batch file tasks (#228)
* feat: show upload progress for file tasks

* feat: show upload progress for batch file tasks

* style: distinguish download and upload phases

* style: mark successful task completion

* fix: preserve storage save error context

* fix: serialize single-file progress updates

* fix: stabilize batch upload progress reporting

Correct batch upload totals, completion state, actual file sizes, and synchronized progress snapshots. Add regression coverage for concurrent updates and phase transitions.

* feat: show per-file transfer progress

Format single-file and batch download and upload states with Telegram entities, blockquotes, speeds, transferred sizes, progress bars, concise counters, and accurate confirmation handling. Cover upload retries plus final, error, and cancellation messages with regression tests.

* chore: remove transfer progress tests

* test: restore critical transfer progress coverage

* test: cover interleaved batch transfers

* fix: declare progress styles in locale templates
2026-08-11 08:40:29 +08:00
Haopeng Huo e4144e73e6 feat(telegram): split oversized videos into playable parts (#229)
* feat(telegram): split oversized videos losslessly

* fix(telegram): respect Premium upload limits

* fix(telegram): keep split videos in one album
2026-08-05 16:57:46 +08:00
Haopeng Huo 1d794a7b9c feat(telegram): preserve source media groups (#226)
* feat(telegram): preserve source media groups

* fix(telegram): rewind batch readers before inspection
2026-08-04 16:46:29 +08:00
Krau f02860ff3f feat: implement task event system for progress tracking and reporting (#220) 2026-06-25 21:37:36 +08:00
krau 3d6cd45909 feat: add configuration options for video download settings 2026-06-25 16:17:03 +08:00
Ryan 62e4a08e28 Merge pull request #204 from Rain-kl/feat/save-strategy
新增功能: 重名文件时选择保存策略 - 重命名,覆盖,跳过
2026-05-20 16:11:00 +08:00
krau b431fa08e2 fix: update StoragePath method to return specific path for single file 2026-01-30 13:09:21 +08:00
krau 4d2c345003 fix: enhance filename extraction logic for downloads and add unit tests 2026-01-29 17:11:25 +08:00
krau 82e1efb518 refactor: update logging messages for transfer task execution and progress tracking 2026-01-19 21:34:57 +08:00
krau 9b52a3e0ce refactor: simplify storage path handling across various tasks and storage implementations 2026-01-19 21:27:53 +08:00
krau dd0dea8cb5 feat!: Refactor batch import task to transfer task
- Updated command usage in English and Simplified Chinese localization files to reflect changes in transfer command syntax.
- Removed batch import task implementation, replacing it with a new transfer task implementation.
- Introduced new task structure and progress tracking for file transfers.
- Updated task type enumeration to replace batch import with transfer.
- Added new fields in data structures to support transfer operations.
- Implemented file handling and progress reporting for the transfer task.
2026-01-19 21:14:01 +08:00
krau 3d20fbd0fe feat: implement transfer command for file transfers between storages 2026-01-19 21:01:50 +08:00
KrauandCopilot f92c43b9c8 feat: support import files from storages to telegram (#183)
* feat: add import command and batch import functionality

- Implemented the `/import` command to allow users to import files from storage to Telegram.
- Added support for listing files in storage and filtering based on regex patterns.
- Created a batch import task to handle multiple file uploads concurrently.
- Introduced progress tracking for batch imports, providing real-time updates to users.
- Enhanced storage interfaces to support file listing and reading capabilities.
- Updated localization files for the new import command and its usage instructions.
- Added utility functions for file size formatting and speed calculation.
- Refactored Telegram storage handling to support reading from non-seekable streams.

* feat: add  i18n for import command

* feat: implement ListFiles and OpenFile methods for WebDAV and Alist storage

* Update common/i18n/locale/zh-Hans.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/progress.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/alist/alist.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update pkg/storagetypes/fileinfo.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update common/i18n/locale/en.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/telegram/telegram.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update core/tasks/batchimport/execute.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update storage/webdav/webdav.go

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix: missing progress stats i18n

* refactor: use strutil to parse args

* chore: update generated code files for consistency

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-19 17:39:47 +08:00
Copilotkraucopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
3e20dc2c5f feat: add custom parameter support to /ytdlp command (#185), close #184
* Initial plan

* Implement parameter support for /ytdlp command

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Add comprehensive tests for ytdlp parameter parsing

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Improve flag parsing logic and clarify argument order

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

* Preserve critical defaults and improve comments

Co-authored-by: krau <71133316+krau@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: krau <71133316+krau@users.noreply.github.com>
2026-01-19 13:10:21 +08:00
krau 3ce00884a0 feat: add yt-dlp support for downloading video/audio and enhance related commands 2026-01-17 17:42:11 +08:00
krau cd7cf4964d fix: handle context cancellation during aria2 download and improve cleanup logic 2026-01-17 15:22:41 +08:00
krau bc3c841d1d fix: update aria2 configuration to include KeepFile option and improve download handling 2026-01-17 15:19:49 +08:00
krau 743c15f1a5 fix: enhance aria2 download handling and logging for follow-up downloads 2026-01-17 15:10:37 +08:00
krau b05d86509c feat: basic aria2 integration 2026-01-17 14:57:03 +08:00
krau 6db5633ca6 feat(i18n): Add progress tracking messages for various tasks and enhance error handling 2025-12-19 20:40:14 +08:00
krau 0ca3d97711 feat: add task command to client and Title method to Task for tasks queue managing, #157 2025-12-15 11:29:55 +08:00
krau c21ff7e499 feat: add direct links download functionality
- Implemented a new task type for handling direct links downloads.
- Added command handler for downloading multiple links via /dl command.
- Introduced progress tracking for direct link downloads.
- Enhanced filename parsing to support various encoding scenarios.
- Updated enums to include direct links as a task type.
- Refactored existing task structures to accommodate new functionality.
- Improved error handling and logging throughout the download process.
2025-12-08 17:10:41 +08:00
krau 685047e463 fix: compatibility between tdlib and bot api style chatID 2025-12-04 22:43:22 +08:00
krau 9b3f955e48 feat: use default HTTP client from netutil for task creation 2025-08-24 09:37:52 +08:00
krau e5d1e143e0 feat: configurable parser and refactor config 2025-08-23 14:29:32 +08:00
Krau 302db2fe75 feat: parse url with js plugins support (#96)
* feat: WIP. add parser functionality and text message handling

* fix: use json to marshal js result

* feat: add metadata handling and version validation for jsParser

* refactor: rename parser package to parsers and restructure parser handling

* refactor: core code struct and impl parse task handle

* feat: impl parsed download

* fix: seek cache file when processing tph picture

* feat: implement parsed task handling and progress tracking

* feat: enhance task processing with concurrency control and progress tracking

* feat: add resource ID generation and improve resource processing handling

* feat: improve message formatting in parsed text and progress completion

* feat: add example js plugin

* feat: implement Twitter parser

* fix: twitter parse video json decode error

* feat: impl stream mode for parse task
2025-08-21 23:48:17 +08:00
krau 133453b5d4 feat: implement watch for monitoring chat messages
- Added a new command handler for /watch that allows users to listen to messages from a specified chat and save them according to storage rules.
- Introduced filtering options for messages using regular expressions.
- Implemented functionality to start and stop watching chats, including error handling for invalid inputs and user settings.
- Created a new utility package for message element handling related to the watch feature.
- Updated the user model to manage watched chats, including methods to add, remove, and check if a chat is being watched.
2025-08-03 16:55:56 +08:00
krau 03907f2d32 fix: improve error handling during file download in stream mode 2025-07-28 14:37:08 +08:00
krau f693bd6103 refactor: update file handling to use new downloader interface; remove unused tdler package 2025-06-29 23:00:40 +08:00
krau c82c2462bf feat: exec command hook , close #79 2025-06-20 21:30:50 +08:00
krau 4df2c5a06d refactor: replace key package with ctxkey for context keys 2025-06-17 22:19:06 +08:00
Krau 900823cdb9 refactor: refactor task logic for better scalability (#76)
* refactor: a big refactor. wip

* refactor: port handle file

* refactor: place all handlers

* fix: task info nil pointer

* feat: enhance task progress tracking and context management

* feat: cancel task

* feat: stream mode

* feat: silent mode

* feat: dir cmd

* refactor: remove unused old file

* feat: rule cmd

* feat: handle silent mode

* feat: batch task

* fix: batch task progress and temp file cleanup

* refactor: update file creation and cleanup methods for better resource management

* feat: add save command with silent mode handling

* feat: message link

* feat: update message prompts to include file count in storage selection

* feat: slient save links

* refactor: reduce dup code

* feat: rule type

* feat: chose dir

* feat: refactor file handling and storage rules, improve error handling and logging

* feat: rule mode

* feat: telegraph pics

* fix: tphpics nil pointer and inaccurate dirpath

* feat: silent save telegraph

* feat: add suffix to avoid file overwrite

* feat: new storage telegram

* chore: tidy go mod
2025-06-15 23:57:49 +08:00
krau e85d3c9441 feat: rename file only when storagePath exists 2025-06-11 09:54:08 +08:00
krau 56ea1d6f36 feat: parse message link via userbot, close #70 2025-06-09 14:33:40 +08:00
krau c7c458f147 feat: add user client 2025-06-08 15:36:14 +08:00
krau ed99a37831 fix: add unique id to task struct to avoid duplicate file name overwrite, close #59 2025-05-09 08:58:30 +08:00
krau c8c348a182 feat: batch save files 2025-04-12 16:27:23 +08:00
krau 166c27c70f feat: automatic file organization based on rules, close #28 2025-04-12 14:27:13 +08:00
krau 491ba55f1e feat: add support for handling unsupported stream storage in download process 2025-03-26 10:35:40 +08:00
krau 7ffd9891a0 fix: not pass content length when uploading in non stream mode 2025-03-26 10:22:38 +08:00
krau 347a60f1f7 fix: implement image extraction from Telegraph nodes 2025-03-24 22:04:55 +08:00
krau f14c4367f8 feat: cancel download telegraph task 2025-03-22 12:08:19 +08:00
krau 3e3a320672 feat: download telegraph images , close #5 2025-03-22 11:52:43 +08:00
krau 635f00ac71 fix: reorganize cache destination path handling in processPendingTask function 2025-03-21 23:28:14 +08:00