Commit Graph
57 Commits
Author SHA1 Message Date
Krau c2f8ab3c01 refactor: quality overhaul (#234)
* fix: prevent queue deadlock after cancelling tasks

Get no longer recurses while holding the mutex.
Cancelled queued tasks leave the map, making their IDs reusable.
Closed empty queues return ErrQueueClosed.

* fix: init task queue lazily and close on shutdown

AddTask is safe before Run by initializing the queue once.
Close unblocks workers waiting in Get.

* fix: make resource fingerprints deterministic

Sort map keys before hashing so Resource.ID is stable.

* fix: harden per-item processing tracking in tasks

Use the resource fingerprint as the dedup key on insert and delete.
Check and set processing entries atomically instead of TOCTOU.
Count only successful downloads.

* fix: honor IgnoreErrors in batch tasks

Element failures no longer cancel sibling elements or stop later groups.

* fix: report streamed upload bytes in batch tasks

Stream downloads report their byte count as the upload total.

* fix: validate i18n key parity across locales

geni18n fails when a language file misses any key.
Align the syncpeers completion key between en and zh-Hans.
Translate three untranslated parse keys in zh-Hans.

* refactor: share progress throttling helpers

Move size-tiered and count-based throttling into progressutil.
Localize hardcoded Chinese progress strings.
Drop five duplicated implementations and dead local copies.

* refactor: share unique filename logic

Storage backends use fsutil.UniquePath instead of local loops.

* fix: sanitize local storage paths

Reject absolute paths and dot-dot escapes in Save.
Check close errors and wrap creation failures.

* fix: preserve webdav error causes

Wrap mkdir and write failures with %w and drop dead error values.

* fix: kill rclone subprocess on reader close

Prevent cat processes from hanging after the pipe is closed.

* fix: fail alist init instead of exiting

Replace log.Fatalf with wrapped errors so a bad alist cannot kill the bot.
Cancel token refresh with the init context and re-login on 401.

* fix: enforce telegram album limits and track saved paths

Use tglimit.MaxAlbumItems for album batching and video splitting.
Exists now reports previously saved paths instead of always false.

* fix: guard storage registry maps

Protect Storages and UserStorages with mutexes and expose read accessors.

* fix: harden JS parser plugin runtime

Validate semver instead of panicking and require canHandle.
Recover plugin worker panics and time out CanHandle calls.
Return a copy from the registry and sanitize install filenames.

* fix: guard kemono parser against nil fields

Skip sparse preview and attachment entries instead of panicking.

* refactor: remove commented-out dead code

Drop unused kemono legacy types and commented response structs.

* test: cover invalid plugin version rejection

* fix: show all queued tasks in /task list

Render up to ten tasks and append the truncation note once.

* fix: guard callback data parsing

Reject malformed callback payloads before indexing split parts.

* fix: isolate media groups per user

Key pending groups by chat, user and group id.

* fix: require permission for callback handlers

* fix: initialize userbot context once

Replace the racy lazy init with sync.OnceValue.

* fix: avoid leaking raw errors in /dir reply

* fix: notify users on invalid update version

* fix: fail fast when API listen fails

Bind synchronously and surface errors instead of logging them.

* fix: add timeouts and backoff to webhook delivery

* refactor: remove dead code from api and bot

Drop the empty ProgressTracker shim, unused token context key and a redundant SetBotCommands call.

* fix: load remote config without local lookup

Skip the local file search after reading a config URL and add a timeout.

* refactor: drop unused hook config

* docs: document parser plugin config

* ci: fix BuildTime formatting and align checkout

Actions format does not format dates; pass the raw timestamp.

* chore: ignore cache directory

* fix: make cache init idempotent

* fix: upload only downloaded batch elements

Failed elements keep partial cache files and never reach the backend.
Successfully downloaded siblings still upload when one element fails.

* fix: record telegram saved paths only after upload

Skip-large returns a sentinel so skipped files are not marked as saved.

* fix: deduplicate alist token refreshes

Guard token access with a mutex and merge concurrent logins.
Reuse a recent refresh to avoid login storms.

* test: cover concurrent alist 401 retry

Ten parallel uploads share a single re-login under -race.

* fix: count parsed resources in progress text

* fix: localize storage lookup errors in /dir

Use the shared i18n key and escape the dynamic error.

* fix: deduplicate concurrent storage initialization

singleflight merges first-time inits so side effects are not duplicated.

* fix: guard nil progress trackers in api-created tasks

Batch, telegraph and transfer tasks run without a Telegram tracker
when created through the API; their callbacks must not panic.

* fix: keep album order when filtering failed batch items

Download results are stored by original index so surviving
elements keep their source order.

* test: cover nil-tracker task execution

* fix: report partial failure in batch done message

IgnoreErrors runs with failed elements show success and failed
counts instead of claiming every file completed.

* fix: skip login refresh for token-only alist storage

401 responses surface the auth error instead of sending a
credential-less login; the refresh window never exceeds TokenExp.

* test: cover alist refresh semantics

Concurrent refresh uses username/password; token-only storage
never attempts a login on 401.

* fix: call tracker from notifyProgress instead of recursing

The helper called itself, overflowing the stack on any batch
task with a progress tracker.

* fix: propagate cancellation past IgnoreErrors

Cancelled tasks must not be reported as successful: only
ordinary element failures are ignored.

* test: cover notifyProgress tracker call

* test: cover cancellation with IgnoreErrors

* fix: refresh alist token after startup 401s

The init login no longer satisfies the dedup window, so an
early 401 triggers a real refresh; later 401s reuse it.
Clarify that streaming uploads cannot replay their body.

* test: exercise the alist 401 refresh path

Concurrent uploads now reject the init token, share one
refresh and retry with the new token; token-only stays inert.

* style: trim verbose comments

Drop process-style explanations; keep one-line behavior notes.

* style: gofmt test files

* fix: drop unbounded saved-path cache in telegram storage

Telegram cannot reliably query remote file existence, so the
cache answered a question it could not answer and grew without
bound. Exists returns false again, as before.

* style: format codes
2026-08-17 19:10:03 +08:00
krau 3d6cd45909 feat: add configuration options for video download settings 2026-06-25 16:17:03 +08:00
krau 8f67f778a3 feat(config): add log level configuration and update logging initialization, close #202 2026-04-14 09:41:02 +08:00
krau 159dba6224 fix(config): comment out default config saving logic 2026-03-30 18:33:03 +08:00
krau 3eb3b6e3c8 feat(api): implement task management API with handlers for creating, listing, retrieving, and canceling tasks
- Added Handlers struct and methods for task operations
- Implemented task progress tracking and storage
- Created server setup with middleware for logging and recovery
- Added support for Telegram file extraction and Telegraph image extraction
- Introduced webhook functionality for task status updates
- Defined request and response types for API interactions
2026-03-05 19:11:30 +08:00
krau bc3c841d1d fix: update aria2 configuration to include KeepFile option and improve download handling 2026-01-17 15:19:49 +08:00
krau b05d86509c feat: basic aria2 integration 2026-01-17 14:57:03 +08:00
krau 75e5fd10ea feat(aria2): add Aria2 download command and client integration 2026-01-03 17:40:55 +08:00
krau 9ed2c425be feat(i18n): Enhance internationalization support and error handling messages
- Added i18n keys for various error and info messages related to task management, user handling, and storage operations.
- Updated error messages in the watch and task handling commands to use i18n for better localization.
- Refactored error handling in the database initialization and user synchronization processes to provide clearer messages.
- Improved logging messages for better clarity and consistency across the application.
- Added comprehensive documentation for AI collaboration and project structure in the repository.
2025-12-19 16:01:36 +08:00
krau df64ec3069 feat: add configuration flags and enhance config initialization 2025-12-19 13:49:13 +08:00
krau a6f194aedd feat: add global proxy config 2025-12-02 10:11:58 +08:00
krau 215e082028 feat: implement internationalization support and update help commands 2025-08-27 11:09:38 +08:00
krau 764be2a083 fix: improve error handling in config initialization 2025-08-24 14:16:23 +08:00
krau e5d1e143e0 feat: configurable parser and refactor config 2025-08-23 14:29:32 +08:00
Krau 302db2fe75 feat: parse url with js plugins support (#96)
* feat: WIP. add parser functionality and text message handling

* fix: use json to marshal js result

* feat: add metadata handling and version validation for jsParser

* refactor: rename parser package to parsers and restructure parser handling

* refactor: core code struct and impl parse task handle

* feat: impl parsed download

* fix: seek cache file when processing tph picture

* feat: implement parsed task handling and progress tracking

* feat: enhance task processing with concurrency control and progress tracking

* feat: add resource ID generation and improve resource processing handling

* feat: improve message formatting in parsed text and progress completion

* feat: add example js plugin

* feat: implement Twitter parser

* fix: twitter parse video json decode error

* feat: impl stream mode for parse task
2025-08-21 23:48:17 +08:00
krau c82c2462bf feat: exec command hook , close #79 2025-06-20 21:30:50 +08:00
krau 758564d436 feat(config): add cache configuration options for TTL, num counters, and max cost 2025-06-18 10:51:03 +08:00
krau 473a5b9413 chore: update help text 2025-06-16 17:31:34 +08:00
krau 6c2abe3025 chore: remove deprecated config 2025-06-16 17:11:26 +08:00
Krau 900823cdb9 refactor: refactor task logic for better scalability (#76)
* refactor: a big refactor. wip

* refactor: port handle file

* refactor: place all handlers

* fix: task info nil pointer

* feat: enhance task progress tracking and context management

* feat: cancel task

* feat: stream mode

* feat: silent mode

* feat: dir cmd

* refactor: remove unused old file

* feat: rule cmd

* feat: handle silent mode

* feat: batch task

* fix: batch task progress and temp file cleanup

* refactor: update file creation and cleanup methods for better resource management

* feat: add save command with silent mode handling

* feat: message link

* feat: update message prompts to include file count in storage selection

* feat: slient save links

* refactor: reduce dup code

* feat: rule type

* feat: chose dir

* feat: refactor file handling and storage rules, improve error handling and logging

* feat: rule mode

* feat: telegraph pics

* fix: tphpics nil pointer and inaccurate dirpath

* feat: silent save telegraph

* feat: add suffix to avoid file overwrite

* feat: new storage telegram

* chore: tidy go mod
2025-06-15 23:57:49 +08:00
krau 693e20b066 Deprecated public copy media feat 2025-06-09 15:50:53 +08:00
krau c7c458f147 feat: add user client 2025-06-08 15:36:14 +08:00
krau 481427683e chore: translate config package 2025-06-08 13:37:51 +08:00
krau c798c7ae99 feat: i18n with default lang zh-Hans (translating) 2025-06-08 11:01:33 +08:00
krau 6ecfbd8385 feat: add public media copy 2025-05-28 16:43:11 +08:00
krau 66454b082a fix: improve logger initialization and reduce cache TTL 2025-05-08 21:03:48 +08:00
krau 239d5ad562 feat: update database session configuration and retry settings 2025-05-07 08:59:15 +08:00
krau 609289c16a perf: optimize user storage retrieval and remove unused rate limiting middleware 2025-04-15 21:04:48 +08:00
krau 3bdef20e85 feat: add expiration handling for database entries and enhance user model with rules 2025-04-12 11:14:13 +08:00
krau 50fba3f910 feat: add configurable timeout for Telegram client initialization 2025-04-07 10:23:50 +08:00
krau 65fee89e14 feat: refactor storage configuration to use dedicated storage package and add new storage types
BREAKING CHANGE: remove deprecated config
2025-03-21 20:52:41 +08:00
krau 8e2dd37155 feat: add stream upload support and related configurations 2025-02-28 11:09:24 +08:00
krau d7cd2ede01 feat: add configurable thread count for file processing 2025-02-21 13:51:30 +08:00
krau c4eb824457 feat: set default storage by inline keyboard 2025-02-19 12:23:12 +08:00
krau 692e970772 feat!: (WIP) switched back to using config files config storages because the conversation handling is shit 2025-02-19 11:05:30 +08:00
krau 18cd480264 fix: add json tag for config 2025-02-18 19:53:01 +08:00
krau dfde65c28e feat: (WIP) migrate storage configuration to user-specific models and remove deprecated storage loading 2025-02-18 19:45:06 +08:00
krau 968547b005 feat!: (WIP) decouple storage, users, and configuration files to support multiple users 2025-02-18 17:17:02 +08:00
krau f80c4d7d55 feat: login in alist via token 2025-02-17 16:39:27 +08:00
krau ccfde34666 fix: failed to run when config file exist 2025-02-17 16:37:46 +08:00
krau 2b23446123 fix: write default config to file if not exist, close #16 2025-02-17 16:23:10 +08:00
krau 5f3b38c788 feat: add /path command to change file save path and improve configuration handling 2025-02-15 16:25:16 +08:00
krau d26a8df15f feat: add default telegram api id and hash 2025-02-12 13:40:49 +08:00
krau a746cc0fc7 feat: add cache cleaning functionality and configuration option 2025-02-12 12:24:15 +08:00
krau a778b0fa8c feat: support env vars configuration 2025-01-20 12:06:47 +08:00
krau 79fba918cf feat: add retry configuration for file upload and validate config values 2025-01-19 14:25:56 +08:00
krau 8d0851f37a feat: add retry configuration for failed save task 2025-01-07 10:30:33 +08:00
krau 4b5cabc76e feat: add proxy config in telegram dialer 2025-01-07 10:26:09 +08:00
krau 3e5cd60cf7 refactor: update config and readme 2024-11-09 09:43:56 +08:00
krau 75f6b444fc fix: correct default chunk size 2024-10-12 17:16:51 +08:00