Compare commits

..

6 Commits

Author SHA1 Message Date
Awuqing
1e386e1205 fix: route sms otp through webhook notifier 2026-04-25 22:09:26 +08:00
Awuqing
6e7a884c64 fix: annotate validated sms webhook request 2026-04-25 21:57:52 +08:00
Awuqing
0b2263086f fix: harden sms webhook target validation 2026-04-25 21:50:20 +08:00
Awuqing
2f494818cf fix: store trusted device token in httponly cookie 2026-04-25 21:36:08 +08:00
Awuqing
7dfd12254b feat: add complete MFA support 2026-04-25 21:14:39 +08:00
Awuqing
2997e971a6 docs: add community and sponsors pages with dynamic GitHub contributor integration 2026-04-25 18:50:54 +08:00
171 changed files with 1610 additions and 13323 deletions

View File

@@ -6,10 +6,6 @@ on:
pull_request:
branches: [main, master]
# 最小权限:构建/测试仅需读取仓库内容,显式声明以收敛默认的可写令牌。
permissions:
contents: read
jobs:
backend:
name: Go Build & Test

View File

@@ -116,15 +116,12 @@ jobs:
fi
cp deploy/nginx.conf "${ARCHIVE_NAME}/nginx.conf" 2>/dev/null || true
tar czf "${ARCHIVE_NAME}.tar.gz" "${ARCHIVE_NAME}"
cp "${ARCHIVE_NAME}.tar.gz" "backupx-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz"
- name: Upload to GitHub Release
uses: softprops/action-gh-release@v2
with:
tag_name: ${{ env.VERSION }}
files: |
backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
backupx-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
files: backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
generate_release_notes: true
# ─── Job 3: Docker 多架构 → Docker Hub ───

View File

@@ -1,106 +0,0 @@
# Contributing to BackupX
感谢你对 BackupX 的关注!本指南介绍如何搭建开发环境并提交贡献。
Thanks for your interest in contributing to BackupX! This guide covers how to set up your environment and submit changes.
## 开发环境 / Development Setup
### 依赖 / Prerequisites
- **Go** 1.25+(见 `server/go.mod`
- **Node.js** 20+CI 与 Docker 使用 Node 20
- **npm** 9+
### 快速开始 / Quick Start
分别在两个终端启动前后端(后端 :8340前端 Vite HMR
```bash
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
# 终端 1 —— 后端(默认 http://localhost:8340
make dev-server
# 终端 2 —— 前端Vite 热更新,/api 代理到 8340
make dev-web
```
### 构建 / Building
```bash
make build # 同时构建后端与前端
make build-server # 仅后端 → server/bin/backupx
make build-web # 仅前端 → web/dist
make docker # 构建 Docker 镜像
make docker-cn # 国内镜像源加速构建
```
> 后端会自动托管 `web/dist`(或 `server.web_root` 指定目录),因此本地裸机部署无需额外的反向代理即可访问控制台。
## 测试 / Testing
提交前请确保测试通过:
```bash
make test # 后端 + 前端全部测试
make test-server # 仅后端cd server && go test ./...
make test-web # 仅前端cd web && npm run testvitest
```
新增功能或修复缺陷时,请尽量补充对应测试。
## 提交信息规范 / Commit Messages
本项目采用 **Conventional Commits**,正文用中文撰写:
```
<type>(<scope>): <subject>
<body>
```
| type | 说明 |
|------|------|
| `feat` | 新功能 |
| `fix` | 缺陷修复 |
| `docs` | 文档变更 |
| `style` | 不影响逻辑的格式调整 |
| `refactor` | 重构 |
| `perf` | 性能优化 |
| `test` | 测试相关 |
| `chore` | 构建/依赖/工具链 |
示例:
```
feat(storage): 新增 Wasabi S3 后端支持
fix(cluster): 修复跨节点恢复的终态处理
docs: 补充 CONTRIBUTING 指南
```
## Pull Request 流程
1. **Fork** 仓库并从最新的 `main` 切出特性分支;
2. **开发**功能或修复,必要时补充测试;
3. **自测**:确保 `make test` 通过;
4. **提交**:使用上述 Conventional Commits中文
5. **推送**并对着 `main` 发起 PR。
### PR 描述建议
- 清晰说明本 PR 做了什么;
- 对新功能/修复,补充动机与背景;
- 关联相关 Issue`Closes #62`
- 纯文档 PR 可不附测试。
> 请保持分支基于较新的 `main`:基线过旧的分支容易产生大范围冲突,难以评审与合入。
## 编码规范 / Coding Conventions
- **Go**:所有错误必须处理(禁止 `_ = err`),日志使用项目已有库(`zap`),禁止 `fmt.Println`;提交前执行 `gofmt`
- **前端**:遵循项目 ESLint/Prettier/tsconfig 配置,不擅自引入新的 CSS 框架或 UI 库。
- **包管理**`web/` 使用 npm请提交对应的 `package-lock.json`
## License
向 BackupX 贡献即表示你同意你的贡献以 [Apache License 2.0](LICENSE) 授权。

View File

@@ -60,15 +60,9 @@ docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/back
# Or prebuilt archive
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
tar xzf backupx-*.tar.gz && cd backupx-* && sudo ./install.sh
# Or build and install on bare metal without Docker
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build && sudo ./deploy/install.sh
```
For ARM64 hosts, use `backupx-linux-arm64.tar.gz`. The archive contains `backupx`, `web/`, `config.example.yaml`, and `install.sh`; run `install.sh` from the extracted directory.
Open `http://your-server:8340`, choose English or Chinese on the setup screen, create the first administrator account, then follow the [5-minute Quick Start](https://awuqing.github.io/BackupX/docs/getting-started/quick-start).
Open `http://your-server:8340`, create the admin account, then follow the [5-minute Quick Start](https://awuqing.github.io/BackupX/docs/getting-started/quick-start).
## Documentation
@@ -98,16 +92,6 @@ See the [development guide](https://awuqing.github.io/BackupX/docs/development/s
Issues and pull requests welcome. Please read the [contributing guide](https://awuqing.github.io/BackupX/docs/development/contributing) before opening a PR — commit messages and PRs on this project are written in Chinese.
## Star History
<a href="https://www.star-history.com/?repos=Awuqing%2FBackupX&type=date&legend=top-left">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&theme=dark&legend=top-left" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&legend=top-left" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&legend=top-left" />
</picture>
</a>
## License
[Apache License 2.0](LICENSE)

View File

@@ -60,15 +60,9 @@ docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/back
# 或使用预编译包
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
tar xzf backupx-*.tar.gz && cd backupx-* && sudo ./install.sh
# 或从源码构建并裸机安装(无需 Docker
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build && sudo ./deploy/install.sh
```
ARM64 主机请下载 `backupx-linux-arm64.tar.gz`。预编译包内包含 `backupx``web/``config.example.yaml``install.sh`,请在解压后的目录内执行 `install.sh`
打开 `http://your-server:8340`,在初始化页选择中文或 English 并创建首个管理员账户,按 [5 分钟快速开始](https://awuqing.github.io/BackupX/zh-Hans/docs/getting-started/quick-start) 完成首次备份。
打开 `http://your-server:8340`,创建管理员账户,按 [5 分钟快速开始](https://awuqing.github.io/BackupX/zh-Hans/docs/getting-started/quick-start) 完成首次备份
## 文档

View File

@@ -1,10 +1,6 @@
#!/bin/sh
set -e
if [ "${1:-}" = "agent" ]; then
exec /app/bin/backupx "$@"
fi
# Backend listens on internal port 8341, Nginx exposes 8340
export BACKUPX_SERVER_PORT="${BACKUPX_SERVER_PORT_INTERNAL:-8341}"

View File

@@ -1,31 +1,17 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
PROJECT_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
PROJECT_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
PREFIX="${PREFIX:-/opt/backupx}"
ETC_DIR="${ETC_DIR:-/etc/backupx}"
SERVICE_NAME="backupx"
APP_USER="backupx"
APP_GROUP="backupx"
if [ -f "$SCRIPT_DIR/backupx" ] && [ -d "$SCRIPT_DIR/web" ]; then
BIN_SOURCE="${BIN_SOURCE:-$SCRIPT_DIR/backupx}"
WEB_SOURCE="${WEB_SOURCE:-$SCRIPT_DIR/web}"
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$SCRIPT_DIR/config.example.yaml}"
NGINX_SOURCE="${NGINX_SOURCE:-$SCRIPT_DIR/nginx.conf}"
else
SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/bin/backupx"
# Keep compatibility with contributors who built the historical path by
# hand, while matching the canonical `make build` output first.
if [ ! -f "$SOURCE_BIN_DEFAULT" ] && [ -f "$PROJECT_ROOT/server/backupx" ]; then
SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/backupx"
fi
BIN_SOURCE="${BIN_SOURCE:-$SOURCE_BIN_DEFAULT}"
WEB_SOURCE="${WEB_SOURCE:-$PROJECT_ROOT/web/dist}"
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$PROJECT_ROOT/server/config.example.yaml}"
NGINX_SOURCE="${NGINX_SOURCE:-$PROJECT_ROOT/deploy/nginx.conf}"
fi
BIN_SOURCE="${BIN_SOURCE:-$PROJECT_ROOT/server/backupx}"
WEB_SOURCE="${WEB_SOURCE:-$PROJECT_ROOT/web/dist}"
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$PROJECT_ROOT/server/config.example.yaml}"
SERVICE_SOURCE="${SERVICE_SOURCE:-$PROJECT_ROOT/deploy/backupx.service}"
NGINX_SOURCE="${NGINX_SOURCE:-$PROJECT_ROOT/deploy/nginx.conf}"
if [ "$(id -u)" -ne 0 ]; then
echo "请使用 root 或 sudo 执行安装脚本。" >&2
@@ -33,22 +19,14 @@ if [ "$(id -u)" -ne 0 ]; then
fi
if [ ! -f "$BIN_SOURCE" ]; then
echo "Backend binary not found / 未找到后端二进制:$BIN_SOURCE" >&2
echo "源码树安装请先在仓库根目录执行 make build产物server/bin/backupx)。" >&2
echo "For a source install, run 'make build' in the repository root first." >&2
echo "发布包安装请确认当前目录包含 ./backupx、./web 和 ./install.sh。" >&2
echo "未找到后端二进制:$BIN_SOURCE" >&2
echo "请先执行cd \"$PROJECT_ROOT/server\" && go build -o backupx ./cmd/backupx" >&2
exit 1
fi
if [ ! -d "$WEB_SOURCE" ]; then
echo "未找到前端构建产物:$WEB_SOURCE" >&2
echo "源码树安装请先执行cd \"$PROJECT_ROOT/web\" && npm run build" >&2
echo "发布包安装请确认当前目录包含 ./web。" >&2
exit 1
fi
if [ ! -f "$CONFIG_TEMPLATE" ]; then
echo "未找到配置模板:$CONFIG_TEMPLATE" >&2
echo "请先执行cd \"$PROJECT_ROOT/web\" && npm run build" >&2
exit 1
fi
@@ -66,75 +44,14 @@ cp -R "$WEB_SOURCE/." "$PREFIX/web/"
chown -R "$APP_USER:$APP_GROUP" "$PREFIX"
if [ ! -f "$ETC_DIR/config.yaml" ]; then
install -o "$APP_USER" -g "$APP_GROUP" -m 0640 "$CONFIG_TEMPLATE" "$ETC_DIR/config.yaml"
install -m 0640 "$CONFIG_TEMPLATE" "$ETC_DIR/config.yaml"
fi
# 确保服务账户能读取配置:历史版本曾以 root:root 0640 安装配置,
# 导致以 backupx 身份运行的服务因无权读取配置而启动失败exit 1
chown "$APP_USER:$APP_GROUP" "$ETC_DIR/config.yaml"
if [ -f "$SERVICE_SOURCE" ]; then
install -m 0644 "$SERVICE_SOURCE" "/etc/systemd/system/$SERVICE_NAME.service"
else
cat > "/etc/systemd/system/$SERVICE_NAME.service" <<UNIT
[Unit]
Description=BackupX API Service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=$APP_USER
Group=$APP_GROUP
WorkingDirectory=$PREFIX
ExecStart=$PREFIX/bin/backupx -config $ETC_DIR/config.yaml
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target
UNIT
fi
install -m 0644 "$SERVICE_SOURCE" "/etc/systemd/system/$SERVICE_NAME.service"
systemctl daemon-reload
systemctl enable --now "$SERVICE_NAME"
# systemctl may return before the process has opened its HTTP listener. Verify
# the same unauthenticated endpoint used by the first-administrator screen so a
# broken bare-metal install cannot print a false success message.
HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8340/api/auth/setup/status}"
READY=0
ATTEMPT=1
while [ "$ATTEMPT" -le 30 ]; do
if systemctl is-active --quiet "$SERVICE_NAME"; then
if command -v curl >/dev/null 2>&1; then
if curl -fsS --max-time 2 "$HEALTH_URL" >/dev/null 2>&1; then
READY=1
break
fi
elif command -v wget >/dev/null 2>&1; then
if wget -q -T 2 -O /dev/null "$HEALTH_URL"; then
READY=1
break
fi
else
echo "Warning / 警告:未找到 curl 或 wget仅验证 systemd 服务状态。" >&2
READY=1
break
fi
fi
ATTEMPT=$((ATTEMPT + 1))
sleep 1
done
if [ "$READY" -ne 1 ]; then
echo "BackupX did not become ready at $HEALTH_URL / 服务未通过就绪检查。" >&2
systemctl status "$SERVICE_NAME" --no-pager >&2 || true
journalctl -u "$SERVICE_NAME" -n 50 --no-pager >&2 || true
exit 1
fi
if [ -d "/etc/nginx/conf.d" ] && [ -f "$NGINX_SOURCE" ]; then
if [ -d "/etc/nginx/conf.d" ]; then
install -m 0644 "$NGINX_SOURCE" "/etc/nginx/conf.d/$SERVICE_NAME.conf"
if command -v nginx >/dev/null 2>&1; then
nginx -t
@@ -150,19 +67,6 @@ cat <<MESSAGE
- 配置文件:$ETC_DIR/config.yaml
- systemd 服务:/etc/systemd/system/$SERVICE_NAME.service
Web 控制台已由后端直接托管,无需额外的 nginx 反向代理即可访问:
http://<本机IP>:8340
首次访问 / First sign-in:
1. 打开上面的地址,并可在登录页右上角选择 中文 或 English。
2. 页面显示“系统初始化 / System setup”时创建首个管理员用户名和密码。
3. 如果未显示初始化表单,请先检查:$HEALTH_URL
(如已安装 nginx脚本会自动写入反向代理配置可继续用 80 端口访问。)
排查:若服务未监听端口,请查看日志:
journalctl -u "$SERVICE_NAME" -n 50 --no-pager
如需修改监听地址、数据库路径或日志级别,请编辑 "$ETC_DIR/config.yaml" 后执行:
systemctl restart "$SERVICE_NAME"
MESSAGE

View File

@@ -22,8 +22,6 @@ services:
# - /home/user/data:/mnt/data:ro
environment:
- TZ=Asia/Shanghai
# 远程 Agent 需要通过公网或可路由地址连接 Master 时,取消注释并改成真实 URL
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
# 通过 BACKUPX_ 前缀环境变量覆盖配置:
# - BACKUPX_LOG_LEVEL=debug
# - BACKUPX_BACKUP_MAX_CONCURRENT=4

View File

@@ -10,34 +10,20 @@ description: systemd + Nginx deployment from the prebuilt release tarball or sou
```bash
# Download the matching tarball
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-v1.6.0-linux-amd64.tar.gz
# Extract and install
tar xzf backupx-linux-amd64.tar.gz && cd backupx-*-linux-amd64
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh
```
The installer performs these steps automatically:
1. Creates a system user `backupx`
2. Copies the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
3. Installs the default configuration at `/etc/backupx/config.yaml`
2. Copies the binary to `/opt/backupx/`
3. Generates a default `config.yaml` with safe JWT/encryption secrets
4. Installs `backupx.service` (systemd), enabled at boot
5. (Optional) installs an Nginx site file — see [Nginx Reverse Proxy](./nginx)
6. Verifies the first-setup API before reporting success
For multi-node clusters, edit `/etc/backupx/config.yaml` after installation and set the Master URL that remote Agents can reach:
```yaml
server:
external_url: "https://backup.example.com"
```
Restart BackupX after changing it:
```bash
sudo systemctl restart backupx
```
## From source
@@ -64,13 +50,11 @@ After=network.target
[Service]
Type=simple
User=backupx
Group=backupx
WorkingDirectory=/opt/backupx
ExecStart=/opt/backupx/bin/backupx -config /etc/backupx/config.yaml
ExecStart=/opt/backupx/backupx --config /opt/backupx/config.yaml
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
LimitNOFILE=65535
RestartSec=5s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
@@ -82,20 +66,17 @@ Typical operations:
sudo systemctl status backupx
sudo journalctl -u backupx -f # live logs
sudo systemctl restart backupx
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
```
Open `http://your-server:8340`, switch to English if desired, and create the first administrator on the **System setup** screen. For a custom listen port, run the installer with a matching `HEALTH_URL`.
## Password reset
If the admin password is lost:
```bash
/opt/backupx/bin/backupx reset-password \
/opt/backupx/backupx reset-password \
--username admin \
--password 'newpass123' \
--config /etc/backupx/config.yaml
--config /opt/backupx/config.yaml
```
Docker equivalent:

View File

@@ -15,14 +15,13 @@ server:
host: "0.0.0.0" # BACKUPX_SERVER_HOST
port: 8340 # BACKUPX_SERVER_PORT
mode: "release" # release | debug
external_url: "" # BACKUPX_SERVER_EXTERNAL_URL — public Master URL for Agent install scripts
database:
path: "./data/backupx.db" # BACKUPX_DATABASE_PATH — embedded SQLite
security:
jwt_secret: "" # BACKUPX_SECURITY_JWT_SECRET — auto-generated if empty
jwt_expire: "24h" # BACKUPX_SECURITY_JWT_EXPIRE
jwt_expires_in: "24h"
encryption_key: "" # AES-256-GCM key for storage config encryption
backup:
@@ -47,20 +46,7 @@ The environment wins when both file and env are set. All dot-paths become unders
| Config key | Env variable |
|------------|--------------|
| `server.port` | `BACKUPX_SERVER_PORT` |
| `server.external_url` | `BACKUPX_SERVER_EXTERNAL_URL` |
| `security.jwt_expire` | `BACKUPX_SECURITY_JWT_EXPIRE` |
| `log.level` | `BACKUPX_LOG_LEVEL` |
| `backup.max_concurrent` | `BACKUPX_BACKUP_MAX_CONCURRENT` |
| `backup.temp_dir` | `BACKUPX_BACKUP_TEMP_DIR` |
| `backup.bandwidth_limit` | `BACKUPX_BACKUP_BANDWIDTH_LIMIT` |
## Master external URL
Set `server.external_url` when BackupX is behind Docker, Nginx, a load balancer, or any reverse proxy whose internal Host is not reachable by remote Agents:
```yaml
server:
external_url: "https://backup.example.com"
```
This value is used when BackupX renders one-click Agent install scripts and docker-compose snippets. It must be reachable from every Agent host. Leave it empty only when `X-Forwarded-Proto` / `X-Forwarded-Host` are reliable and point to the same URL that Agents can access.

View File

@@ -25,8 +25,6 @@ services:
- /etc/nginx:/mnt/nginx-conf:ro
environment:
- TZ=Asia/Shanghai
# Required when remote Agents must connect through a public or routed URL:
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
- BACKUPX_LOG_LEVEL=info
- BACKUPX_BACKUP_MAX_CONCURRENT=2
@@ -44,17 +42,6 @@ docker compose up -d
To back up files from the host, mount them into the container. When creating a file-type task in the web UI, point the source path at the mount location (e.g. `/mnt/www`). Make sure the directory is visible inside the container.
## Multi-node clusters
When deploying Agents on other machines, set `BACKUPX_SERVER_EXTERNAL_URL` on the Master container to the URL that those Agents can reach:
```yaml
environment:
- BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
```
Use an HTTPS URL if Agents cross untrusted networks. The generated one-click install scripts and docker-compose snippets use this value as `BACKUPX_AGENT_MASTER`.
## Environment variables
All configuration keys can be overridden with the `BACKUPX_` prefix:

View File

@@ -8,26 +8,14 @@ description: File, MySQL, PostgreSQL, SQLite and SAP HANA — what they back up
BackupX supports five built-in backup types. Type determines which runner executes the job.
When a task is routed to a remote Agent, the source tools and paths are resolved on that Agent host. Multi-target uploads are still tracked per storage target; if at least one target succeeds, the backup record is marked successful and the per-target result table shows partial failures.
## File / Directory
File tasks offer three backup modes:
- **Full archive** — writes a self-contained tar artifact on every run
- **Differential archive** — writes only changes since the current full baseline and periodically refreshes that baseline
- **CDC repository** — splits content with stable 512 KiB / 1 MiB / 4 MiB boundaries, stores new chunks in immutable 32 MiB packs, and writes a small snapshot manifest for each run
The CDC repository deduplicates identical content across files and snapshots. Restore, selective restore, verification, download-as-tar, retention, and garbage collection all resolve data through the repository index. Compression and encryption are applied per chunk; encrypted repositories use keyed chunk IDs so plaintext hashes are not exposed.
Repository mode currently uses a single-writer index and therefore runs on the Master only. To keep repository copies on multiple backends, select multiple primary storage targets on the task. Object-level replication is intentionally disabled because a snapshot manifest without its shared packs and indexes is not a complete backup.
Common file-task options:
Tars (and optionally gzips) one or more filesystem paths.
- **Source** accepts multiple paths — one per line in the UI
- **Exclude patterns** accept gitignore-style globs
- Supports following symlinks, preserving permissions
- Full and differential modes output `.tar`, `.tar.gz`, or `.tar.zst` artifacts
- Output is a single `.tar` or `.tar.gz` artifact
## MySQL

View File

@@ -26,42 +26,8 @@ BackupX supports Master-Agent mode: backup tasks can be routed to specific nodes
- **Execution** — Agent reuses the same BackupRunner (file / mysql / postgresql / sqlite / saphana) and uploads directly to storage
- **Security** — Each node has its own token; the Agent never holds the Master's JWT secret or AES-256 key
## Centralize backups from servers B/C/D into storage M
Use the Master as the control plane and register every source server as an Agent. A task's **Source server** determines where paths and database tools are resolved; its **Storage targets** determine where the resulting artifact is retained.
BackupX chooses the data path per target:
| Destination | Data path |
| --- | --- |
| S3, WebDAV, FTP, cloud drive, or another network backend | Agent streams directly to the destination |
| `local_disk` with **Relay remote backups through Master** enabled (for example storage server M mounted through NFS) | Agent streams through the authenticated Master API; Master writes to its configured local path |
The relay is streaming: the Master does not create a second temporary copy of the entire artifact. The reverse path is used when restoring a Master-local artifact back to its source Agent. Use HTTPS whenever Agent traffic crosses an untrusted network.
To configure the common `A → {B,C,D} → M` topology:
1. Run BackupX Master on A and mount M on A if M is exposed as NFS or another filesystem.
2. Create a `local_disk` target for that mount and keep **Relay remote backups through Master** enabled, or create an S3/WebDAV target exposed by M. Existing local-disk targets keep their prior Agent-local behavior until this switch is enabled.
3. Install one Agent on B, C, and D from **Node Management**.
4. Create a backup task for each source, choose B/C/D under **Source server**, browse that server's paths, and select M as the storage target. A source-server pool label can route identical tasks dynamically.
5. Verify the per-target result in the backup record. For a Master-local target, the record reports transfer mode `master_relay`; network backends remain `direct`.
## Walkthrough
### 0. Set the Master URL for production clusters
Before generating Agent install commands, make sure the Master URL shown to Agents is stable and reachable from every target host.
If BackupX runs behind Docker, Nginx, a load balancer, or an outer reverse proxy, configure `server.external_url` or `BACKUPX_SERVER_EXTERNAL_URL` on the Master:
```yaml title="config.yaml"
server:
external_url: "https://backup.example.com"
```
This URL is baked into systemd units, foreground commands, and docker-compose snippets. If it is wrong, Agents will install successfully but stay offline because they keep polling an internal or browser-only address.
### 1. Open the install wizard
In the Web Console → **Node Management****Add Node**. You'll see a three-step wizard.
@@ -83,8 +49,6 @@ The script runs automatically and:
5. Runs `systemctl enable --now backupx-agent`
6. Polls `/api/v1/agent/self` until the master confirms `status: online` (up to 30 s)
Docker mode uses the same `BACKUPX_AGENT_MASTER`, `BACKUPX_AGENT_TOKEN`, and `BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp` environment contract. After starting the container, the installer also probes `/api/v1/agent/self`; if the node does not come online, it prints `docker ps` and `docker logs --tail=100 backupx-agent` diagnostics before exiting non-zero.
If you choose the URL-based fallback command and `curl` prints HTML or the shell reports `Syntax error: newline unexpected`, the install URL is being served by the web console instead of the backend. Ensure either `/api/install/` or `/install/` is forwarded to the BackupX backend, or use the embedded command generated by the console.
Reruns are idempotent — to upgrade or re-provision, simply generate a new install command and run it again. The one-time install link expires after its TTL or after first consumption, whichever is sooner.
@@ -99,20 +63,14 @@ In Step 1 choose "Batch" and paste node names (one per line, max 50). Step 3 sho
### 5. Route a task to the node
In the **Backup Tasks** page, pick the source server when creating the task. When the task runs:
In the **Backup Tasks** page, pick the target node when creating the task. When the task runs:
- Local (`nodeId=0`) → Master executes in-process
- Remote node → Master enqueues the command → Agent claims → Agent runs locally → uploads → reports back
The node table shows the Agent health and command queue state: pending/dispatched depth, running long commands, timeouts, oldest active command age, and the latest Agent-side error. The same queue depth, running-command, and timeout snapshots are exported as Prometheus metrics:
- `backupx_agent_command_queue_depth`
- `backupx_agent_command_running`
- `backupx_agent_command_timeout_total`
## Known limitations
- **Encrypted backups are Master-only** — the Agent doesn't hold Master's AES-256 key. Creating or updating a task with `encrypt: true` and a remote node or node pool is rejected up front
- **Encrypted backups don't work via Agent** — the Agent doesn't hold Master's AES-256 key. Tasks with `encrypt: true` will fail if routed to an Agent
- **Directory browser timeout** — remote dir listing is a synchronous RPC through the queue (15s default)
- **Dispatched command timeout** — claimed-but-unfinished commands are marked `timeout` after 10 minutes

View File

@@ -19,9 +19,7 @@ BackupX aims to accept any place you'd want to drop a backup file.
| **Google Drive** | Client ID/Secret + OAuth authorization |
| **WebDAV** | URL + username/password |
| **FTP / FTPS** | Host + port + username/password |
| **Local disk** | Target directory (absolute path) + optional Master relay for remote Agents |
New local-disk targets enable **Relay remote backups through Master** by default. This makes the configured path belong to the Master, so a storage server mounted there can collect backups from many source Agents. Turn the switch off when the path intentionally belongs to each Agent. Existing targets retain their previous Agent-local behavior until explicitly changed.
| **Local disk** | Target directory (absolute path) |
## Rclone backends

View File

@@ -55,11 +55,10 @@ sudo ./install.sh # creates system user, installs to /opt/backupx, sets u
The installer:
1. Creates a `backupx` system user
2. Installs the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
3. Creates `/etc/backupx/config.yaml` with safe defaults
2. Installs binary to `/opt/backupx/backupx`
3. Creates `/opt/backupx/config.yaml` with safe defaults
4. Installs and enables the `backupx.service` systemd unit
5. (Optional) Configures an Nginx reverse proxy
6. Waits for `/api/auth/setup/status`; if startup fails, prints systemd diagnostics and exits non-zero
## From source
@@ -68,17 +67,16 @@ Requires Go ≥ 1.25 and Node.js ≥ 20.
```bash
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build
sudo ./deploy/install.sh
# or, for builds behind the great firewall
make docker-cn
```
After `make build`, the binary is at `server/bin/backupx` and the built web UI is at `web/dist/`.
The installer consumes those exact paths, so no Docker runtime is required. If an existing configuration uses a non-default port, set `HEALTH_URL` for the readiness check, for example `sudo HEALTH_URL=http://127.0.0.1:9000/api/auth/setup/status ./deploy/install.sh`.
## Verify the install
```bash
/opt/backupx/bin/backupx --version
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
backupx --version # e.g. v1.6.0
```
Then open `http://your-server:8340`. Choose **English** or **中文** in the upper-right corner. A fresh database shows **System setup**, where you create the first administrator username and password. If that form does not appear, retry the status request above before attempting to sign in.
Then open `http://your-server:8340` to see the initial admin setup screen.

View File

@@ -42,8 +42,6 @@ Go to **Backup Tasks → New**. Three steps:
2. **Source** — paths for file backup (multi-source supported), or connection info for databases
3. **Storage & policy** — pick target(s), compression, retention days, encryption on/off
For Agent-routed tasks, encryption must stay off because the Agent never receives the Master's encryption key. BackupX rejects remote-node or node-pool tasks with encryption enabled during create/update.
Save, then click **Run Now** to trigger a test. Live logs stream on the **Backup Records** page.
:::note

View File

@@ -10,34 +10,20 @@ description: 从预编译包或源码部署 BackupXsystemd + Nginx
```bash
# 下载对应平台的压缩包
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-v1.6.0-linux-amd64.tar.gz
# 解压并安装
tar xzf backupx-linux-amd64.tar.gz && cd backupx-*-linux-amd64
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh
```
安装脚本自动完成以下步骤:
1. 创建系统用户 `backupx`
2. 复制二进制到 `/opt/backupx/bin/backupx`,并把 Web 控制台复制到 `/opt/backupx/web`
3. 把默认配置安装到 `/etc/backupx/config.yaml`
2. 复制二进制到 `/opt/backupx/`
3. 生成默认 `config.yaml`(含安全的 JWT/加密密钥)
4. 安装并启用 `backupx.service` systemd 单元
5. (可选)生成 Nginx 站点配置 — 参见 [Nginx 反向代理](./nginx)
6. 验证首次初始化接口就绪后才报告安装成功
如果要部署多节点集群,安装后请编辑 `/etc/backupx/config.yaml`,设置远程 Agent 可访问到的 Master URL
```yaml
server:
external_url: "https://backup.example.com"
```
修改后重启 BackupX
```bash
sudo systemctl restart backupx
```
## 从源码构建
@@ -64,13 +50,11 @@ After=network.target
[Service]
Type=simple
User=backupx
Group=backupx
WorkingDirectory=/opt/backupx
ExecStart=/opt/backupx/bin/backupx -config /etc/backupx/config.yaml
ExecStart=/opt/backupx/backupx --config /opt/backupx/config.yaml
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
LimitNOFILE=65535
RestartSec=5s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
@@ -82,20 +66,17 @@ WantedBy=multi-user.target
sudo systemctl status backupx
sudo journalctl -u backupx -f # 实时日志
sudo systemctl restart backupx
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
```
访问 `http://your-server:8340`,可按需切换到 English然后在“系统初始化 / System setup”页面创建首个管理员。若监听端口不是默认值请为安装脚本传入对应的 `HEALTH_URL`。
## 密码重置
忘记管理员密码时:
```bash
/opt/backupx/bin/backupx reset-password \
/opt/backupx/backupx reset-password \
--username admin \
--password 'newpass123' \
--config /etc/backupx/config.yaml
--config /opt/backupx/config.yaml
```
Docker 等效命令:

View File

@@ -15,14 +15,13 @@ server:
host: "0.0.0.0" # BACKUPX_SERVER_HOST
port: 8340 # BACKUPX_SERVER_PORT
mode: "release" # release | debug
external_url: "" # BACKUPX_SERVER_EXTERNAL_URL — Agent 安装脚本使用的 Master 对外 URL
database:
path: "./data/backupx.db" # BACKUPX_DATABASE_PATH — 内嵌 SQLite
security:
jwt_secret: "" # BACKUPX_SECURITY_JWT_SECRET — 留空自动生成
jwt_expire: "24h" # BACKUPX_SECURITY_JWT_EXPIRE
jwt_expires_in: "24h"
encryption_key: "" # 用于加密存储配置的 AES-256-GCM 密钥
backup:
@@ -47,20 +46,7 @@ log:
| 配置项 | 环境变量 |
|--------|----------|
| `server.port` | `BACKUPX_SERVER_PORT` |
| `server.external_url` | `BACKUPX_SERVER_EXTERNAL_URL` |
| `security.jwt_expire` | `BACKUPX_SECURITY_JWT_EXPIRE` |
| `log.level` | `BACKUPX_LOG_LEVEL` |
| `backup.max_concurrent` | `BACKUPX_BACKUP_MAX_CONCURRENT` |
| `backup.temp_dir` | `BACKUPX_BACKUP_TEMP_DIR` |
| `backup.bandwidth_limit` | `BACKUPX_BACKUP_BANDWIDTH_LIMIT` |
## Master 对外 URL
当 BackupX 部署在 Docker、Nginx、负载均衡或多层反向代理后面且后端收到的内部 Host 不是远程 Agent 可访问地址时,请配置 `server.external_url`
```yaml
server:
external_url: "https://backup.example.com"
```
BackupX 会用这个地址渲染一键 Agent 安装脚本和 docker-compose 片段。该地址必须能被所有 Agent 主机访问。只有在 `X-Forwarded-Proto` / `X-Forwarded-Host` 可靠且正好指向 Agent 可访问地址时,才建议留空。

View File

@@ -25,8 +25,6 @@ services:
- /etc/nginx:/mnt/nginx-conf:ro
environment:
- TZ=Asia/Shanghai
# 远程 Agent 需要通过公网或可路由地址连接 Master 时必须配置:
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
- BACKUPX_LOG_LEVEL=info
- BACKUPX_BACKUP_MAX_CONCURRENT=2
@@ -44,17 +42,6 @@ docker compose up -d
想备份宿主机上的文件,需要将对应路径挂载进容器。在 Web UI 创建文件类型任务时,把源路径指向挂载后的容器内路径(如 `/mnt/www`)。
## 多节点集群
如果要在其他机器部署 Agent请在 Master 容器上设置 `BACKUPX_SERVER_EXTERNAL_URL`,值为所有 Agent 都能访问到的 URL
```yaml
environment:
- BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
```
Agent 跨不可信网络访问时建议使用 HTTPS。控制台生成的一键安装脚本和 docker-compose 片段会把这个值写成 `BACKUPX_AGENT_MASTER`。
## 环境变量
所有配置项都可以通过 `BACKUPX_` 前缀环境变量覆盖:

View File

@@ -8,26 +8,14 @@ description: 文件、MySQL、PostgreSQL、SQLite 和 SAP HANA — 各自的能
BackupX 支持五种内置备份类型,类型决定了用哪个 runner 执行。
当任务路由到远程 Agent 时,源路径和外部工具都会在该 Agent 主机上解析。多存储目标上传仍会逐目标记录结果;只要至少一个目标上传成功,备份记录即为成功,详情中的目标结果表会展示部分失败。
## 文件 / 目录
文件任务提供三种备份模式:
- **全量归档** — 每次生成一份可独立使用的 tar 产物
- **差异归档** — 只保存相对当前全量基线的变化,并按周期刷新全量基线
- **CDC 去重仓库** — 按稳定的 512 KiB / 1 MiB / 4 MiB 内容边界切块,将新块合并到不可变的 32 MiB pack每次运行只新增一份小型快照清单
CDC 仓库会在不同文件、不同快照之间复用相同内容。完整恢复、选择性恢复、完整性校验、下载为 tar、保留策略和垃圾回收都通过仓库索引定位分块。压缩与加密按块执行启用加密时使用带密钥的块 ID不暴露明文哈希。
当前仓库索引采用单写者模型,因此 CDC 模式仅在 Master 本机执行。如需保存多份完整仓库,请在任务中直接多选主存储目标。对象级副本复制会被禁用,因为只有快照清单、没有共享 pack 与索引并不是完整备份。
文件任务的通用选项:
打包(可选 gzip一个或多个文件系统路径。
- **源路径** 支持多个UI 中每行一个)
- **排除模式** 支持 gitignore 风格的通配符
- 可选跟随符号链接、保留权限
- 全量与差异模式输出 `.tar``.tar.gz``.tar.zst`
- 输出单个 `.tar``.tar.gz`
## MySQL

View File

@@ -26,42 +26,8 @@ BackupX 支持 Master-Agent 模式:备份任务可以指定在哪个节点执
- **执行** — Agent 复用 BackupRunnerfile / mysql / postgresql / sqlite / saphana并直接上传到存储
- **安全** — 每个节点独立 TokenAgent 不持有 Master 的 JWT 密钥或 AES-256 加密密钥
## 把 B/C/D 服务器集中备份到 M
Master 作为控制面,每台源服务器安装一个 Agent。任务里的 **源服务器** 决定源路径和数据库工具在哪台机器解析,**存储目标** 决定备份产物最终保留在哪里。
BackupX 会根据目标类型选择数据路径:
| 目标 | 数据路径 |
| --- | --- |
| S3、WebDAV、FTP、云盘或其他网络后端 | Agent 直接流式上传到目标 |
| 启用 **远程备份经 Master 中转**`local_disk`(例如通过 NFS 挂载的存储服务器 M | Agent 通过认证后的 Master API 流式中转,由 Master 写入配置目录 |
中转过程不会在 Master 上额外落一份完整临时文件。把 Master 本地磁盘中的备份恢复到源 Agent 时会走反向流式通道。Agent 与 Master 之间跨越不可信网络时必须配置 HTTPS。
典型的 `A → {B,C,D} → M` 拓扑按以下步骤配置:
1. 在 A 运行 BackupX Master如果 M 以 NFS 等文件系统提供存储,先把 M 挂载到 A。
2. 为该挂载点创建 `local_disk` 目标并保持 **远程备份经 Master 中转** 开启;如果 M 提供 S3/WebDAV也可直接创建对应网络目标。升级前已有的本地磁盘目标会继续沿用 Agent 本机落盘,手动开启该选项后才切换到中央目录。
3.**节点管理** 分别在 B、C、D 安装 Agent。
4. 为每台源服务器创建任务,在 **源服务器** 选择 B/C/D浏览该服务器的路径再把 M 选为存储目标。相同任务也可用源服务器池标签动态调度。
5. 在备份记录中检查逐目标结果。Master 本地磁盘目标会记录 `master_relay` 中转模式,网络后端仍为 `direct` 直传。
## 一键部署步骤
### 0. 为生产集群设置 Master 对外 URL
生成 Agent 安装命令前,请先确认 Master URL 对所有目标主机稳定可达。
如果 BackupX 部署在 Docker、Nginx、负载均衡或外层反向代理后面请在 Master 配置 `server.external_url` 或环境变量 `BACKUPX_SERVER_EXTERNAL_URL`
```yaml title="config.yaml"
server:
external_url: "https://backup.example.com"
```
该 URL 会写入 systemd 单元、前台运行命令和 docker-compose 片段。如果地址不正确Agent 可能安装成功但始终离线,因为它会持续轮询一个内网地址或仅浏览器可访问的地址。
### 1. 打开安装向导
Web 控制台 → **节点管理****添加节点**,打开三步向导:
@@ -83,8 +49,6 @@ Web 控制台 → **节点管理** → **添加节点**,打开三步向导:
5. 执行 `systemctl enable --now backupx-agent`
6. 轮询 `/api/v1/agent/self`,直到 Master 确认 `status: online`(最多 30 秒)
Docker 模式使用同一组环境变量约定:`BACKUPX_AGENT_MASTER`、`BACKUPX_AGENT_TOKEN` 和 `BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp`。容器启动后,安装脚本同样会探测 `/api/v1/agent/self`;如果节点没有上线,会输出 `docker ps` 与 `docker logs --tail=100 backupx-agent` 排查命令,并以非零状态退出。
如果使用 URL 备用命令时 `curl` 输出 HTML或 shell 报 `Syntax error: newline unexpected`,说明安装 URL 被 Web 控制台接管而不是转发到后端。需要确保 `/api/install/``/install/` 至少一个路径能转发到 BackupX 后端,或改用控制台生成的嵌入式命令。
脚本是幂等的:升级或重装只需重新生成一条安装命令再跑一次。一次性安装链接在 TTL 到期或被首次消费后立即作废。
@@ -99,20 +63,14 @@ Docker 模式使用同一组环境变量约定:`BACKUPX_AGENT_MASTER`、`BACKU
### 5. 把任务路由到该节点
在 **备份任务** 页面新建任务时选择对应源服务器。任务触发时:
**备份任务** 页面新建任务时选择对应节点。任务触发时:
- 本机 / 未指定(`nodeId=0`Master 进程内直接执行
- 远程节点Master 写入命令队列 → Agent 拉取 → Agent 本地执行 → 上传 → 回报
节点列表会展示 Agent 健康与命令队列状态pending/dispatched 深度、运行中的长任务、超时数、最旧活跃命令年龄和最近 Agent 错误。同样的队列深度、运行中命令数和超时快照会导出为 Prometheus 指标:
- `backupx_agent_command_queue_depth`
- `backupx_agent_command_running`
- `backupx_agent_command_timeout_total`
## 已知限制
- **加密备份仅支持 Master 本机执行**Agent 不持有 Master 的 AES-256 密钥。创建或更新任务时,如果 `encrypt: true` 且选择了远程节点或节点池,会在入口直接拒绝
- **Agent 不支持加密备份**Agent 不持有 Master 的 AES-256 密钥。`encrypt: true` 的任务路由到 Agent 时会直接上报失败
- **目录浏览超时**:远程目录浏览通过命令队列做同步 RPC默认 15s 超时
- **派发命令超时**Agent 领取但未完成的命令超过 10 分钟会被置 `timeout`

View File

@@ -19,9 +19,7 @@ BackupX 的目标是接入任何你想放置备份文件的地方。
| **Google Drive** | Client ID/Secret + OAuth 授权 |
| **WebDAV** | 地址 + 用户名/密码 |
| **FTP / FTPS** | 主机 + 端口 + 用户名/密码 |
| **本地磁盘** | 目标目录(绝对路径)+ 可选的远程 Agent 经 Master 中转 |
新建本地磁盘目标默认开启 **远程备份经 Master 中转**。开启时,配置目录属于 Master挂载到 Master 的存储服务器可集中接收多台源 Agent 的备份;如果该路径本就属于各 Agent请关闭此选项。升级前已有目标保持原来的 Agent 本机落盘行为,只有显式开启后才会切换。
| **本地磁盘** | 目标目录(绝对路径) |
## Rclone 后端

View File

@@ -55,11 +55,10 @@ sudo ./install.sh # 创建系统用户、安装到 /opt/backupx、配置
安装脚本会自动:
1. 创建 `backupx` 系统用户
2. 安装二进制到 `/opt/backupx/bin/backupx`,并把 Web 控制台安装到 `/opt/backupx/web`
3. 生成 `/etc/backupx/config.yaml`(含安全默认值)
2. 安装二进制到 `/opt/backupx/backupx`
3. 生成 `/opt/backupx/config.yaml`(含安全默认值)
4. 注册并启用 `backupx.service` systemd 单元
5. (可选)配置 Nginx 反向代理
6. 等待 `/api/auth/setup/status` 就绪;启动失败时输出 systemd 诊断并返回非零状态
## 从源码构建
@@ -68,17 +67,16 @@ sudo ./install.sh # 创建系统用户、安装到 /opt/backupx、配置
```bash
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build
sudo ./deploy/install.sh
# 或使用国内镜像加速构建 Docker
make docker-cn
```
`make build` 完成后,二进制位于 `server/bin/backupx`,构建好的 Web UI 位于 `web/dist/`。
安装脚本会直接使用这两个路径,不需要 Docker 运行时。如果已有配置修改了默认端口,可覆盖就绪检查地址,例如:`sudo HEALTH_URL=http://127.0.0.1:9000/api/auth/setup/status ./deploy/install.sh`。
## 验证安装
```bash
/opt/backupx/bin/backupx --version
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
backupx --version # 输出如 v1.6.0
```
打开浏览器访问 `http://your-server:8340`可在右上角选择 **中文****English**。全新数据库会显示“系统初始化 / System setup”在这里创建首个管理员用户名和密码。如果没有出现初始化表单请先重试上面的状态接口不要直接尝试登录
打开浏览器访问 `http://your-server:8340`会进入初始化管理员账户页面

View File

@@ -42,8 +42,6 @@ description: 部署 BackupX、添加存储目标、创建第一个备份任务
2. **源配置** — 文件备份选择源路径(支持多个),数据库备份填写连接信息
3. **存储与策略** — 选择存储目标(支持多个)、压缩策略、保留天数、是否加密
对于路由到 Agent 的任务,加密必须关闭,因为 Agent 不会拿到 Master 的加密密钥。BackupX 会在创建/更新阶段拒绝开启加密的远程节点或节点池任务。
保存后可点击 **立即执行** 测试,**备份记录** 页面实时查看执行日志。
:::note

View File

@@ -1,12 +1,12 @@
{
"name": "docs-site-tmp",
"version": "1.0.0",
"version": "0.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "docs-site-tmp",
"version": "1.0.0",
"version": "0.0.0",
"dependencies": {
"@docusaurus/core": "3.10.0",
"@docusaurus/faster": "3.10.0",
@@ -164,6 +164,7 @@
"resolved": "https://registry.npmmirror.com/@algolia/client-search/-/client-search-5.50.2.tgz",
"integrity": "sha512-ypSboUJ3XJoQz5DeDo82hCnrRuwq3q9ZdFhVKAik9TnZh1DvLqoQsrbBjXg7C7zQOtV/Qbge/HmyoV6V5L7MhQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"@algolia/client-common": "5.50.2",
"@algolia/requester-browser-xhr": "5.50.2",
@@ -262,12 +263,12 @@
}
},
"node_modules/@babel/code-frame": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz",
"integrity": "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw==",
"version": "7.29.0",
"resolved": "https://registry.npmmirror.com/@babel/code-frame/-/code-frame-7.29.0.tgz",
"integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
"license": "MIT",
"dependencies": {
"@babel/helper-validator-identifier": "^7.29.7",
"@babel/helper-validator-identifier": "^7.28.5",
"js-tokens": "^4.0.0",
"picocolors": "^1.1.1"
},
@@ -289,6 +290,7 @@
"resolved": "https://registry.npmmirror.com/@babel/core/-/core-7.29.0.tgz",
"integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/code-frame": "^7.29.0",
"@babel/generator": "^7.29.0",
@@ -324,13 +326,13 @@
}
},
"node_modules/@babel/generator": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.7.tgz",
"integrity": "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ==",
"version": "7.29.1",
"resolved": "https://registry.npmmirror.com/@babel/generator/-/generator-7.29.1.tgz",
"integrity": "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw==",
"license": "MIT",
"dependencies": {
"@babel/parser": "^7.29.7",
"@babel/types": "^7.29.7",
"@babel/parser": "^7.29.0",
"@babel/types": "^7.29.0",
"@jridgewell/gen-mapping": "^0.3.12",
"@jridgewell/trace-mapping": "^0.3.28",
"jsesc": "^3.0.2"
@@ -449,9 +451,9 @@
}
},
"node_modules/@babel/helper-globals": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.29.7.tgz",
"integrity": "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA==",
"version": "7.28.0",
"resolved": "https://registry.npmmirror.com/@babel/helper-globals/-/helper-globals-7.28.0.tgz",
"integrity": "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw==",
"license": "MIT",
"engines": {
"node": ">=6.9.0"
@@ -471,27 +473,27 @@
}
},
"node_modules/@babel/helper-module-imports": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.29.7.tgz",
"integrity": "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g==",
"version": "7.28.6",
"resolved": "https://registry.npmmirror.com/@babel/helper-module-imports/-/helper-module-imports-7.28.6.tgz",
"integrity": "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw==",
"license": "MIT",
"dependencies": {
"@babel/traverse": "^7.29.7",
"@babel/types": "^7.29.7"
"@babel/traverse": "^7.28.6",
"@babel/types": "^7.28.6"
},
"engines": {
"node": ">=6.9.0"
}
},
"node_modules/@babel/helper-module-transforms": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.29.7.tgz",
"integrity": "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg==",
"version": "7.28.6",
"resolved": "https://registry.npmmirror.com/@babel/helper-module-transforms/-/helper-module-transforms-7.28.6.tgz",
"integrity": "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA==",
"license": "MIT",
"dependencies": {
"@babel/helper-module-imports": "^7.29.7",
"@babel/helper-validator-identifier": "^7.29.7",
"@babel/traverse": "^7.29.7"
"@babel/helper-module-imports": "^7.28.6",
"@babel/helper-validator-identifier": "^7.28.5",
"@babel/traverse": "^7.28.6"
},
"engines": {
"node": ">=6.9.0"
@@ -513,9 +515,9 @@
}
},
"node_modules/@babel/helper-plugin-utils": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-plugin-utils/-/helper-plugin-utils-7.29.7.tgz",
"integrity": "sha512-G7sHYigPY17oO5SYWnfD/0MTBwVR781S/JI643e/JhUYgVgWE/61SoW3NH9KWUKyKq5LVh3npif99Wkt6j86Jw==",
"version": "7.28.6",
"resolved": "https://registry.npmmirror.com/@babel/helper-plugin-utils/-/helper-plugin-utils-7.28.6.tgz",
"integrity": "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug==",
"license": "MIT",
"engines": {
"node": ">=6.9.0"
@@ -569,18 +571,18 @@
}
},
"node_modules/@babel/helper-string-parser": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
"integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
"version": "7.27.1",
"resolved": "https://registry.npmmirror.com/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
"integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==",
"license": "MIT",
"engines": {
"node": ">=6.9.0"
}
},
"node_modules/@babel/helper-validator-identifier": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz",
"integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==",
"version": "7.28.5",
"resolved": "https://registry.npmmirror.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz",
"integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
"license": "MIT",
"engines": {
"node": ">=6.9.0"
@@ -623,12 +625,12 @@
}
},
"node_modules/@babel/parser": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
"integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
"version": "7.29.2",
"resolved": "https://registry.npmmirror.com/@babel/parser/-/parser-7.29.2.tgz",
"integrity": "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA==",
"license": "MIT",
"dependencies": {
"@babel/types": "^7.29.7"
"@babel/types": "^7.29.0"
},
"bin": {
"parser": "bin/babel-parser.js"
@@ -1213,15 +1215,15 @@
}
},
"node_modules/@babel/plugin-transform-modules-systemjs": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.7.tgz",
"integrity": "sha512-TM2ZcQLoG2/y4HODiStCo10DibYhWhGWAwVv+EQKmG/7GFl0N+AAmUiXOMKM+aiJ9XBJ9AHVZBvTzMnJ2sM3cQ==",
"version": "7.29.0",
"resolved": "https://registry.npmmirror.com/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.0.tgz",
"integrity": "sha512-PrujnVFbOdUpw4UHiVwKvKRLMMic8+eC0CuNlxjsyZUiBjhFdPsewdXCkveh2KqBA9/waD0W1b4hXSOBQJezpQ==",
"license": "MIT",
"dependencies": {
"@babel/helper-module-transforms": "^7.29.7",
"@babel/helper-plugin-utils": "^7.29.7",
"@babel/helper-validator-identifier": "^7.29.7",
"@babel/traverse": "^7.29.7"
"@babel/helper-module-transforms": "^7.28.6",
"@babel/helper-plugin-utils": "^7.28.6",
"@babel/helper-validator-identifier": "^7.28.5",
"@babel/traverse": "^7.29.0"
},
"engines": {
"node": ">=6.9.0"
@@ -1918,31 +1920,31 @@
}
},
"node_modules/@babel/template": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/template/-/template-7.29.7.tgz",
"integrity": "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==",
"version": "7.28.6",
"resolved": "https://registry.npmmirror.com/@babel/template/-/template-7.28.6.tgz",
"integrity": "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ==",
"license": "MIT",
"dependencies": {
"@babel/code-frame": "^7.29.7",
"@babel/parser": "^7.29.7",
"@babel/types": "^7.29.7"
"@babel/code-frame": "^7.28.6",
"@babel/parser": "^7.28.6",
"@babel/types": "^7.28.6"
},
"engines": {
"node": ">=6.9.0"
}
},
"node_modules/@babel/traverse": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.7.tgz",
"integrity": "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw==",
"version": "7.29.0",
"resolved": "https://registry.npmmirror.com/@babel/traverse/-/traverse-7.29.0.tgz",
"integrity": "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA==",
"license": "MIT",
"dependencies": {
"@babel/code-frame": "^7.29.7",
"@babel/generator": "^7.29.7",
"@babel/helper-globals": "^7.29.7",
"@babel/parser": "^7.29.7",
"@babel/template": "^7.29.7",
"@babel/types": "^7.29.7",
"@babel/code-frame": "^7.29.0",
"@babel/generator": "^7.29.0",
"@babel/helper-globals": "^7.28.0",
"@babel/parser": "^7.29.0",
"@babel/template": "^7.28.6",
"@babel/types": "^7.29.0",
"debug": "^4.3.1"
},
"engines": {
@@ -1950,13 +1952,13 @@
}
},
"node_modules/@babel/types": {
"version": "7.29.7",
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
"integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
"version": "7.29.0",
"resolved": "https://registry.npmmirror.com/@babel/types/-/types-7.29.0.tgz",
"integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==",
"license": "MIT",
"dependencies": {
"@babel/helper-string-parser": "^7.29.7",
"@babel/helper-validator-identifier": "^7.29.7"
"@babel/helper-string-parser": "^7.27.1",
"@babel/helper-validator-identifier": "^7.28.5"
},
"engines": {
"node": ">=6.9.0"
@@ -2079,6 +2081,7 @@
}
],
"license": "MIT",
"peer": true,
"engines": {
"node": ">=18"
},
@@ -2101,6 +2104,7 @@
}
],
"license": "MIT",
"peer": true,
"engines": {
"node": ">=18"
}
@@ -2210,6 +2214,7 @@
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@@ -2631,6 +2636,7 @@
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@@ -3543,6 +3549,7 @@
"resolved": "https://registry.npmmirror.com/@docusaurus/faster/-/faster-3.10.0.tgz",
"integrity": "sha512-GNPtVH14ISjHfSwnHu3KiFGf86ICmJSQDeSv/QaanpBgiZGOtgZaslnC5q8WiguxM1EVkwcGxPuD8BXF4eggKw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@docusaurus/types": "3.10.0",
"@rspack/core": "^1.7.10",
@@ -3673,6 +3680,7 @@
"resolved": "https://registry.npmmirror.com/@docusaurus/plugin-content-docs/-/plugin-content-docs-3.10.0.tgz",
"integrity": "sha512-9BjHhf15ct8Z7TThTC0xRndKDVvMKmVsAGAN7W9FpNRzfMdScOGcXtLmcCWtJGvAezjOJIm6CxOYCy3Io5+RnQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"@docusaurus/core": "3.10.0",
"@docusaurus/logger": "3.10.0",
@@ -4704,6 +4712,7 @@
"resolved": "https://registry.npmmirror.com/@mdx-js/react/-/react-3.1.1.tgz",
"integrity": "sha512-f++rKLQgUVYDAtECQ6fn/is15GkEH9+nZPM3MS0RcxVqoTfawHvDlSCH7JbMhAM6uJ32v3eXLvLmLvjGu7PTQw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/mdx": "^2.0.0"
},
@@ -5408,6 +5417,7 @@
"resolved": "https://registry.npmmirror.com/@svgr/core/-/core-8.1.0.tgz",
"integrity": "sha512-8QqtOQT5ACVlmsvKOJNEaWmRPmcojMOzCz4Hs2BGG/toAp/K38LcsMRyLp349glq5AzJbCEeimEoxaX6v/fLrA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/core": "^7.21.3",
"@svgr/babel-preset": "8.1.0",
@@ -5512,6 +5522,7 @@
"integrity": "sha512-tglZGyx8N5PC+x1Nd/JrZxqpqlcZoSuG9gTDKO6AuFToFiVB3uS8HvbKFuO7g3lJzvFf9riAb94xs9HU2UhAHQ==",
"hasInstallScript": true,
"license": "Apache-2.0",
"peer": true,
"dependencies": {
"@swc/counter": "^0.1.3",
"@swc/types": "^0.1.26"
@@ -6236,6 +6247,7 @@
"resolved": "https://registry.npmmirror.com/@types/react/-/react-19.2.14.tgz",
"integrity": "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w==",
"license": "MIT",
"peer": true,
"dependencies": {
"csstype": "^3.2.2"
}
@@ -6577,6 +6589,7 @@
"resolved": "https://registry.npmmirror.com/acorn/-/acorn-8.16.0.tgz",
"integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==",
"license": "MIT",
"peer": true,
"bin": {
"acorn": "bin/acorn"
},
@@ -6644,6 +6657,7 @@
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-8.18.0.tgz",
"integrity": "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.3",
"fast-uri": "^3.0.1",
@@ -6689,6 +6703,7 @@
"resolved": "https://registry.npmmirror.com/algoliasearch/-/algoliasearch-5.50.2.tgz",
"integrity": "sha512-Tfp26yoNWurUjfgK4GOrVJQhSNXu9tJtHfFFNosgT2YClG+vPyUjX/gbC8rG39qLncnZg8Fj34iarQWpMkqefw==",
"license": "MIT",
"peer": true,
"dependencies": {
"@algolia/abtesting": "1.16.2",
"@algolia/client-abtesting": "5.50.2",
@@ -7042,9 +7057,9 @@
}
},
"node_modules/body-parser": {
"version": "1.20.5",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.5.tgz",
"integrity": "sha512-3grm+/2tUOvu2cjJkvsIxrv/wVpfXQW4PsQHYm7yk4vfpu7Ekl6nEsYBoJUL6qDwZUx8wUhQ8tR2qz+ad9c9OA==",
"version": "1.20.4",
"resolved": "https://registry.npmmirror.com/body-parser/-/body-parser-1.20.4.tgz",
"integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
"license": "MIT",
"dependencies": {
"bytes": "~3.1.2",
@@ -7055,7 +7070,7 @@
"http-errors": "~2.0.1",
"iconv-lite": "~0.4.24",
"on-finished": "~2.4.1",
"qs": "~6.15.1",
"qs": "~6.14.0",
"raw-body": "~2.5.3",
"type-is": "~1.6.18",
"unpipe": "~1.0.0"
@@ -7067,7 +7082,7 @@
},
"node_modules/body-parser/node_modules/bytes": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
"resolved": "https://registry.npmmirror.com/bytes/-/bytes-3.1.2.tgz",
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
"license": "MIT",
"engines": {
@@ -7076,7 +7091,7 @@
},
"node_modules/body-parser/node_modules/debug": {
"version": "2.6.9",
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
"resolved": "https://registry.npmmirror.com/debug/-/debug-2.6.9.tgz",
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
"license": "MIT",
"dependencies": {
@@ -7085,7 +7100,7 @@
},
"node_modules/body-parser/node_modules/ms": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
"resolved": "https://registry.npmmirror.com/ms/-/ms-2.0.0.tgz",
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
"license": "MIT"
},
@@ -7168,6 +7183,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"baseline-browser-mapping": "^2.10.12",
"caniuse-lite": "^1.0.30001782",
@@ -7826,7 +7842,7 @@
},
"node_modules/content-type": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
"resolved": "https://registry.npmmirror.com/content-type/-/content-type-1.0.5.tgz",
"integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
"license": "MIT",
"engines": {
@@ -8134,6 +8150,7 @@
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@@ -9263,14 +9280,14 @@
}
},
"node_modules/express": {
"version": "4.22.2",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
"integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
"version": "4.22.1",
"resolved": "https://registry.npmmirror.com/express/-/express-4.22.1.tgz",
"integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
"license": "MIT",
"dependencies": {
"accepts": "~1.3.8",
"array-flatten": "1.1.1",
"body-parser": "~1.20.5",
"body-parser": "~1.20.3",
"content-disposition": "~0.5.4",
"content-type": "~1.0.4",
"cookie": "~0.7.1",
@@ -9289,7 +9306,7 @@
"parseurl": "~1.3.3",
"path-to-regexp": "~0.1.12",
"proxy-addr": "~2.0.7",
"qs": "~6.15.1",
"qs": "~6.14.0",
"range-parser": "~1.2.1",
"safe-buffer": "5.2.1",
"send": "~0.19.0",
@@ -9397,9 +9414,9 @@
"license": "MIT"
},
"node_modules/fast-uri": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz",
"integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==",
"version": "3.1.0",
"resolved": "https://registry.npmmirror.com/fast-uri/-/fast-uri-3.1.0.tgz",
"integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==",
"funding": [
{
"type": "github",
@@ -9507,6 +9524,7 @@
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@@ -10553,7 +10571,7 @@
},
"node_modules/iconv-lite": {
"version": "0.4.24",
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
"resolved": "https://registry.npmmirror.com/iconv-lite/-/iconv-lite-0.4.24.tgz",
"integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
"license": "MIT",
"dependencies": {
@@ -12051,7 +12069,7 @@
},
"node_modules/media-typer": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
"resolved": "https://registry.npmmirror.com/media-typer/-/media-typer-0.3.0.tgz",
"integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
"license": "MIT",
"engines": {
@@ -14186,6 +14204,7 @@
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@@ -14241,7 +14260,7 @@
},
"node_modules/object-inspect": {
"version": "1.13.4",
"resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
"resolved": "https://registry.npmmirror.com/object-inspect/-/object-inspect-1.13.4.tgz",
"integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
"license": "MIT",
"engines": {
@@ -14700,9 +14719,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.14",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
"version": "8.5.10",
"resolved": "https://registry.npmmirror.com/postcss/-/postcss-8.5.10.tgz",
"integrity": "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==",
"funding": [
{
"type": "opencollective",
@@ -14718,6 +14737,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"nanoid": "^3.3.11",
"picocolors": "^1.1.1",
@@ -15621,6 +15641,7 @@
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
"license": "MIT",
"peer": true,
"dependencies": {
"cssesc": "^3.0.0",
"util-deprecate": "^1.0.2"
@@ -16314,9 +16335,9 @@
}
},
"node_modules/qs": {
"version": "6.15.2",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.15.2.tgz",
"integrity": "sha512-Rzq0KEyX/w/tEybncDgdkZrJgVUsUMk3xjh3t5bv3S1HTAtg+uOYt72+ZfwiQwKdysThkTBdL/rTi6HDmX9Ddw==",
"version": "6.14.2",
"resolved": "https://registry.npmmirror.com/qs/-/qs-6.14.2.tgz",
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
"license": "BSD-3-Clause",
"dependencies": {
"side-channel": "^1.1.0"
@@ -16380,7 +16401,7 @@
},
"node_modules/raw-body": {
"version": "2.5.3",
"resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz",
"resolved": "https://registry.npmmirror.com/raw-body/-/raw-body-2.5.3.tgz",
"integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==",
"license": "MIT",
"dependencies": {
@@ -16395,7 +16416,7 @@
},
"node_modules/raw-body/node_modules/bytes": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
"resolved": "https://registry.npmmirror.com/bytes/-/bytes-3.1.2.tgz",
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
"license": "MIT",
"engines": {
@@ -16437,6 +16458,7 @@
"resolved": "https://registry.npmmirror.com/react/-/react-19.2.5.tgz",
"integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==",
"license": "MIT",
"peer": true,
"engines": {
"node": ">=0.10.0"
}
@@ -16446,6 +16468,7 @@
"resolved": "https://registry.npmmirror.com/react-dom/-/react-dom-19.2.5.tgz",
"integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==",
"license": "MIT",
"peer": true,
"dependencies": {
"scheduler": "^0.27.0"
},
@@ -16501,6 +16524,7 @@
"resolved": "https://registry.npmmirror.com/@docusaurus/react-loadable/-/react-loadable-6.0.0.tgz",
"integrity": "sha512-YMMxTUQV/QFSnbgrP3tjDzLHRg7vsbMn8e9HAa8o/1iXoiomo48b7sk/kkmWEuWNDPJVlKSJRB6Y2fHqdJk+SQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/react": "*"
},
@@ -16529,6 +16553,7 @@
"resolved": "https://registry.npmmirror.com/react-router/-/react-router-5.3.4.tgz",
"integrity": "sha512-Ys9K+ppnJah3QuaRiLxk+jDWOR1MekYQrlytiXxC1RyfbdsZkS5pvKAzCCr031xHixZwpnsYNT5xysdFHQaYsA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/runtime": "^7.12.13",
"history": "^4.9.0",
@@ -17204,7 +17229,7 @@
},
"node_modules/safer-buffer": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
"resolved": "https://registry.npmmirror.com/safer-buffer/-/safer-buffer-2.1.2.tgz",
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
"license": "MIT"
},
@@ -17575,7 +17600,7 @@
},
"node_modules/side-channel": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
"resolved": "https://registry.npmmirror.com/side-channel/-/side-channel-1.1.0.tgz",
"integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
"license": "MIT",
"dependencies": {
@@ -17594,7 +17619,7 @@
},
"node_modules/side-channel-list": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
"resolved": "https://registry.npmmirror.com/side-channel-list/-/side-channel-list-1.0.1.tgz",
"integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
"license": "MIT",
"dependencies": {
@@ -17610,7 +17635,7 @@
},
"node_modules/side-channel-map": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
"resolved": "https://registry.npmmirror.com/side-channel-map/-/side-channel-map-1.0.1.tgz",
"integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
"license": "MIT",
"dependencies": {
@@ -17628,7 +17653,7 @@
},
"node_modules/side-channel-weakmap": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
"resolved": "https://registry.npmmirror.com/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
"integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
"license": "MIT",
"dependencies": {
@@ -18303,7 +18328,8 @@
"version": "2.8.1",
"resolved": "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz",
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
"license": "0BSD"
"license": "0BSD",
"peer": true
},
"node_modules/tsyringe": {
"version": "4.10.0",
@@ -18337,7 +18363,7 @@
},
"node_modules/type-is": {
"version": "1.6.18",
"resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
"resolved": "https://registry.npmmirror.com/type-is/-/type-is-1.6.18.tgz",
"integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
"license": "MIT",
"dependencies": {
@@ -18350,7 +18376,7 @@
},
"node_modules/type-is/node_modules/mime-db": {
"version": "1.52.0",
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
"resolved": "https://registry.npmmirror.com/mime-db/-/mime-db-1.52.0.tgz",
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
"license": "MIT",
"engines": {
@@ -18359,7 +18385,7 @@
},
"node_modules/type-is/node_modules/mime-types": {
"version": "2.1.35",
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
"resolved": "https://registry.npmmirror.com/mime-types/-/mime-types-2.1.35.tgz",
"integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
"license": "MIT",
"dependencies": {
@@ -18384,6 +18410,7 @@
"integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
"devOptional": true,
"license": "Apache-2.0",
"peer": true,
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
@@ -18725,6 +18752,7 @@
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"license": "MIT",
"peer": true,
"dependencies": {
"fast-deep-equal": "^3.1.1",
"fast-json-stable-stringify": "^2.0.0",
@@ -18923,6 +18951,7 @@
"resolved": "https://registry.npmmirror.com/webpack/-/webpack-5.106.2.tgz",
"integrity": "sha512-wGN3qcrBQIFmQ/c0AiOAQBvrZ5lmY8vbbMv4Mxfgzqd/B6+9pXtLo73WuS1dSGXM5QYY3hZnIbvx+K1xxe6FyA==",
"license": "MIT",
"peer": true,
"dependencies": {
"@types/eslint-scope": "^3.7.7",
"@types/estree": "^1.0.8",
@@ -19064,9 +19093,9 @@
}
},
"node_modules/webpack-dev-server": {
"version": "5.2.4",
"resolved": "https://registry.npmjs.org/webpack-dev-server/-/webpack-dev-server-5.2.4.tgz",
"integrity": "sha512-GqDPGZN9bRqKBTkp4aWkobDDHMsrXKoGSdOH56smIri8qR0JG8gfL8/v/f/OZR3/OKXjG8uwJbFVhKm/FNU/UA==",
"version": "5.2.3",
"resolved": "https://registry.npmmirror.com/webpack-dev-server/-/webpack-dev-server-5.2.3.tgz",
"integrity": "sha512-9Gyu2F7+bg4Vv+pjbovuYDhHX+mqdqITykfzdM9UyKqKHlsE5aAjRhR+oOEfXW5vBeu8tarzlJFIZva4ZjAdrQ==",
"license": "MIT",
"dependencies": {
"@types/bonjour": "^3.5.13",

View File

@@ -3,9 +3,6 @@ server:
host: "0.0.0.0"
port: 8340
mode: "release" # debug | release
external_url: "" # 可选Master 对 Agent 可达的 URL例如 https://backup.example.com
web_root: "" # 前端静态目录;留空自动探测(./web、/opt/backupx/web 等)。
# 命中后后端直接托管 Web 控制台,无需额外 nginx 反向代理。
database:
path: "./data/backupx.db" # SQLite 数据库路径

View File

@@ -6,15 +6,14 @@ require (
github.com/gin-gonic/gin v1.10.1
github.com/glebarez/sqlite v1.11.0
github.com/golang-jwt/jwt/v5 v5.3.0
github.com/klauspost/compress v1.18.1
github.com/natefinch/lumberjack v2.0.0+incompatible
github.com/pquerna/otp v1.5.0
github.com/prometheus/client_golang v1.23.2
github.com/rclone/rclone v1.73.5
github.com/pquerna/otp v1.5.0
github.com/rclone/rclone v1.73.3
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/viper v1.20.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.50.0
golang.org/x/crypto v0.48.0
golang.org/x/oauth2 v0.34.0
google.golang.org/api v0.255.0
gopkg.in/yaml.v3 v3.0.1
@@ -30,10 +29,10 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 // indirect
github.com/Azure/go-ntlmssp v0.1.1 // indirect
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
github.com/BurntSushi/toml v1.6.0 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.38 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
github.com/Files-com/files-sdk-go/v3 v3.2.264 // indirect
github.com/IBM/go-sdk-core/v5 v5.18.5 // indirect
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
@@ -50,26 +49,25 @@ require (
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
github.com/aws/aws-sdk-go-v2/config v1.32.8 // indirect
github.com/aws/aws-sdk-go-v2 v1.41.3 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 // indirect
github.com/aws/aws-sdk-go-v2/config v1.31.17 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.11 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 // indirect
github.com/aws/smithy-go v1.25.1 // indirect
github.com/aws/smithy-go v1.24.2 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
@@ -111,7 +109,7 @@ require (
github.com/glebarez/go-sqlite v1.21.2 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
@@ -150,6 +148,7 @@ require (
github.com/json-iterator/go v1.1.12 // indirect
github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
github.com/klauspost/compress v1.18.1 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect
@@ -225,22 +224,22 @@ require (
go.mongodb.org/mongo-driver v1.17.6 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
go.opentelemetry.io/otel v1.41.0 // indirect
go.opentelemetry.io/otel/metric v1.41.0 // indirect
go.opentelemetry.io/otel/trace v1.41.0 // indirect
go.opentelemetry.io/otel v1.39.0 // indirect
go.opentelemetry.io/otel/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.uber.org/multierr v1.10.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
golang.org/x/arch v0.14.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
golang.org/x/image v0.41.0 // indirect
golang.org/x/mod v0.35.0 // indirect
golang.org/x/net v0.53.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.43.0 // indirect
golang.org/x/term v0.42.0 // indirect
golang.org/x/text v0.37.0 // indirect
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/image v0.32.0 // indirect
golang.org/x/mod v0.32.0 // indirect
golang.org/x/net v0.51.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.44.0 // indirect
golang.org/x/tools v0.41.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/grpc v1.79.3 // indirect
google.golang.org/protobuf v1.36.10 // indirect

View File

@@ -51,8 +51,8 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 h1:ZJJNFaQ86GVKQ9ehw
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3/go.mod h1:URuDvhmATVKqHBH9/0nOiNKk0+YcwfQ3WkK5PqHKxc8=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 h1:sxgSqOB9CDToiaVFpxuvb5wGgGqWa3lCShcm5o0n3bE=
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3/go.mod h1:XdED8i399lEVblYHTZM8eXaP07gv4Z58IL6ueMlVlrg=
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26 h1:gy/jrlpp8EfSyA73a51fofoSfhp5rPNQAUvDr4Dm91c=
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM=
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE=
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
@@ -61,8 +61,8 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/FilenCloudDienste/filen-sdk-go v0.0.38 h1:YCyHs3wUXEe2BEWn40vcoyaQ2ruHNmNwkasfo3Th16A=
github.com/FilenCloudDienste/filen-sdk-go v0.0.38/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 h1:W8S9TrAyZ4//3PXsU6+Bi+fe/6uIL986GyS7PVzIDL4=
github.com/FilenCloudDienste/filen-sdk-go v0.0.37/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
github.com/Files-com/files-sdk-go/v3 v3.2.264 h1:lMHTplAYI9FtmCo/QOcpRxmPA5REVAct1r2riQmDQKw=
github.com/Files-com/files-sdk-go/v3 v3.2.264/go.mod h1:wGqkOzRu/ClJibvDgcfuJNAqI2nLhe8g91tPlDKRCdE=
github.com/IBM/go-sdk-core/v5 v5.18.5 h1:g0JRl3sYXJczB/yuDlrN6x22LJ6jIxhp0Sa4ARNW60c=
@@ -102,46 +102,44 @@ github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kk
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc h1:LoL75er+LKDHDUfU5tRvFwxH0LjPpZN8OoG8Ll+liGU=
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc/go.mod h1:w648aMHEgFYS6xb0KVMMtZ2uMeemhiKCuD2vj6gY52A=
github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
github.com/aws/aws-sdk-go-v2/config v1.32.8 h1:iu+64gwDKEoKnyTQskSku72dAwggKI5sV6rNvgSMpMs=
github.com/aws/aws-sdk-go-v2/config v1.32.8/go.mod h1:MI2XvA+qDi3i9AJxX1E2fu730syEBzp/jnXrjxuHwgI=
github.com/aws/aws-sdk-go-v2 v1.41.3 h1:4kQ/fa22KjDt13QCy1+bYADvdgcxpfH18f0zP542kZA=
github.com/aws/aws-sdk-go-v2 v1.41.3/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 h1:N4lRUXZpZ1KVEUn6hxtco/1d2lgYhNn1fHkkl8WhlyQ=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
github.com/aws/aws-sdk-go-v2/config v1.31.17 h1:QFl8lL6RgakNK86vusim14P2k8BFSxjvUkcWLDjgz9Y=
github.com/aws/aws-sdk-go-v2/config v1.31.17/go.mod h1:V8P7ILjp/Uef/aX8TjGk6OHZN6IKPM5YW6S78QnRD5c=
github.com/aws/aws-sdk-go-v2/credentials v1.19.11 h1:NdV8cwCcAXrCWyxArt58BrvZJ9pZ9Fhf9w6Uh5W3Uyc=
github.com/aws/aws-sdk-go-v2/credentials v1.19.11/go.mod h1:30yY2zqkMPdrvxBqzI9xQCM+WrlrZKSOpSJEsylVU+8=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 h1:INUvJxmhdEbVulJYHI061k4TVuS3jzzthNvjqvVvTKM=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19/go.mod h1:FpZN2QISLdEBWkayloda+sZjVJL+e9Gl0k1SyTgcswU=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1 h1:IbWiN670htmBioc+Zj32vSpJgQ2+OYSlvTvfQ1nCORQ=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1/go.mod h1:tw/B596EUhBWDFGdDGuLC21fVU4A3s4/5Efy8S39W18=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 h1:2fjfz3/G9BRvIKuNZ655GwzpklC2kEH0cowZQGO7uBg=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4/go.mod h1:Ymws824lvMypLFPwyyUXM52SXuGgxpu0+DISLfKvB+c=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 h1:/sECfyq2JTifMI2JPyZ4bdRN77zJmr6SrS1eL3augIA=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19/go.mod h1:dMf8A5oAqr9/oxOfLkC/c2LU/uMcALP0Rgn2BD5LWn0=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 h1:AWeJMk33GTBf6J20XJe6qZoRSJo0WfUhsMdUKhoODXE=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19/go.mod h1:+GWrYoaAsV7/4pNHpwh1kiNLXkKaSoppxQq9lbH8Ejw=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 h1:Y2cAXlClHsXkkOvWZFXATr34b0hxxloeQu/pAZz2row=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7/go.mod h1:idzZ7gmDeqeNrSPkdbtMp9qWMgcBwykA7P7Rzh5DXVU=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 h1:qi3e/dmpdONhj1RyIZdi6DKKpDXS5Lb8ftr3p7cyHJc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20/go.mod h1:V1K+TeJVD5JOk3D9e5tsX2KUdL7BlB+FV6cBhdobN8c=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 h1:XAq62tBTJP/85lFD5oqOOe7YYgWxY9LvWq8plyDvDVg=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 h1:BYf7XNsJMzl4mObARUBUib+j2tf0U//JAAtTnYqvqCw=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11/go.mod h1:aEUS4WrNk/+FxkBZZa7tVgp4pGH+kFGW40Y8rCPqt5g=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 h1:X1Tow7suZk9UCJHE1Iw9GMZJJl0dAnKXXP1NaSDHwmw=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19/go.mod h1:/rARO8psX+4sfjUQXp5LLifjUt8DuATZ31WptNJTyQA=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 h1:JnQeStZvPHFHeyky/7LbMlyQjUa+jIBj36OlWm0pzIk=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19/go.mod h1:HGyasyHvYdFQeJhvDHfH7HXkHh57htcJGKDZ+7z+I24=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4 h1:4ExZyubQ6LQQVuF2Qp9OsfEvsTdAWh5Gfwf6PgIdLdk=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4/go.mod h1:NF3JcMGOiARAss1ld3WGORCw71+4ExDD2cbbdKS5PpA=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 h1:iSsvB9EtQ09YrsmIc44Heqlx5ByGErqhPK1ZQLppias=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12/go.mod h1:fEWYKTRGoZNl8tZ77i61/ccwOMJdGxwOhWCkp6TXAr0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 h1:EnUdUqRP1CNzt2DkV67tJx6XDN4xlfBFm+bzeNOQVb0=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16/go.mod h1:Jic/xv0Rq/pFNCh3WwpH4BEqdbSAl+IyHro8LbibHD8=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 h1:XQTQTF75vnug2TXS8m7CVJfC2nniYPZnO1D4Np761Oo=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8/go.mod h1:Xgx+PR1NUOjNmQY+tRMnouRp83JRM8pRMw/vCaVhPkI=
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng=
github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -258,8 +256,8 @@ github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA=
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UNbRM=
github.com/go-git/go-billy/v5 v5.6.2/go.mod h1:rcFC2rAsp/erv7CMz9GczHcuD0D32fWzH+MJAU+jaUU=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -532,8 +530,8 @@ github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 h1:4MI2
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11/go.mod h1:3HLX7dwZgvB7nt+Yl/xdzVPcargQ1yBmJEUg3n+jMKM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 h1:Lc+d3ISfQaMJKWZOE7z4ZSY4RVmdzbn1B0IM8xN18qM=
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc=
github.com/rclone/rclone v1.73.5 h1:r8a9JHYIWUqk7hNRJuMJ3cROkKfB2zmfkADg8ZLYh6I=
github.com/rclone/rclone v1.73.5/go.mod h1:WVv8gvA/lEl/Y37e8I8yosm7ZY+Szq7ujXbJS8Ol63o=
github.com/rclone/rclone v1.73.3 h1:XKlobcnXxxzxnB6UBSVtRB+UeZmYDV9B4QExVSSGoAY=
github.com/rclone/rclone v1.73.3/go.mod h1:QJDWatpAY9sKGXfpKZUXbThvtHoeo78DcFP2+/cbkvc=
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
github.com/relvacode/iso8601 v1.7.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
@@ -653,16 +651,16 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 h1:RbKq8BG0FI8OiXhBfcRtqqHcZcka+gU3cskNuf05R18=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0/go.mod h1:h06DGIukJOevXaj/xrNjhi/2098RZzcLTbc0jDAUbsg=
go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c=
go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE=
go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ=
go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps=
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0=
go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis=
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/mock v0.5.0 h1:KAMbZvZPyBPWgD14IrIQ38QCyjwpvVVV6K/bHl1IwQU=
@@ -690,8 +688,8 @@ golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliY
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -702,12 +700,12 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.41.0 h1:8wS72eGJMJaBxK6okTzd4WaXumUlTVlb753MlsSvTCo=
golang.org/x/image v0.41.0/go.mod h1:uIc348UZMSvS5Z65CVZ7iDPaNobNFEPeJ4kbqTOszmA=
golang.org/x/image v0.32.0 h1:6lZQWq75h7L5IWNk0r+SCpUJ6tUVd3v4ZHnbRKLkUDQ=
golang.org/x/image v0.32.0/go.mod h1:/R37rrQmKXtO6tYXAjtDLwQgFLHmhW+V6ayXlxzP2Pc=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -732,8 +730,8 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c=
golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -773,8 +771,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA=
golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -798,8 +796,8 @@ golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -849,8 +847,8 @@ golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -862,8 +860,8 @@ golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.42.0 h1:UiKe+zDFmJobeJ5ggPwOshJIVt6/Ft0rcfrXZDLWAWY=
golang.org/x/term v0.42.0/go.mod h1:Dq/D+snpsbazcBG5+F9Q1n2rXV8Ma+71xEjTRufARgY=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -879,8 +877,8 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -933,8 +931,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

View File

@@ -5,7 +5,6 @@ import (
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -126,11 +125,10 @@ type TaskSpec struct {
// StorageTargetConfig 与 service.AgentStorageTargetConfig 对齐
type StorageTargetConfig struct {
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
TransferMode string `json:"transferMode"`
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
}
// GetTaskSpec 拉取任务规格
@@ -145,26 +143,13 @@ func (c *MasterClient) GetTaskSpec(ctx context.Context, taskID uint) (*TaskSpec,
// RecordUpdate 与 service.AgentRecordUpdate 对齐
type RecordUpdate struct {
Status string `json:"status,omitempty"`
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
StorageTargetID uint `json:"storageTargetId,omitempty"`
StorageTransferMode string `json:"storageTransferMode,omitempty"`
StorageUploadResults []StorageResultItem `json:"storageUploadResults,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"`
}
type StorageResultItem struct {
StorageTargetID uint `json:"storageTargetId"`
StorageTargetName string `json:"storageTargetName"`
Status string `json:"status"`
StoragePath string `json:"storagePath,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
TransferMode string `json:"transferMode,omitempty"`
Error string `json:"error,omitempty"`
Status string `json:"status,omitempty"`
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"`
}
// UpdateRecord 上报备份记录的状态/日志
@@ -173,39 +158,6 @@ func (c *MasterClient) UpdateRecord(ctx context.Context, recordID uint, update R
return c.do(ctx, http.MethodPost, path, update, nil)
}
// UploadArtifact streams an artifact through the Master for storage targets
// that are not directly reachable from the Agent.
func (c *MasterClient) UploadArtifact(ctx context.Context, recordID, targetID uint, objectKey string, size int64, checksum string, reader io.Reader) error {
path := fmt.Sprintf("/api/agent/records/%d/artifacts/%d", recordID, targetID)
req, err := http.NewRequestWithContext(ctx, http.MethodPut, c.baseURL+path, reader)
if err != nil {
return err
}
// The executor owns and closes the artifact file. Prevent net/http from
// closing that underlying reader when it finishes the request body.
req.Body = io.NopCloser(reader)
req.ContentLength = size
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("X-Agent-Token", c.token)
req.Header.Set("X-BackupX-Object-Key", objectKey)
req.Header.Set("X-BackupX-SHA256", checksum)
client := *c.httpClient
client.Timeout = 0
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("relay artifact to Master: %w", err)
}
data, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
closeErr := resp.Body.Close()
if readErr != nil || closeErr != nil {
return errors.Join(readErr, closeErr)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("relay artifact to Master: http %d: %s", resp.StatusCode, string(data))
}
return nil
}
// RestoreSpec 与 service.AgentRestoreSpec 对齐
type RestoreSpec struct {
RestoreRecordID uint `json:"restoreRecordId"`
@@ -227,7 +179,6 @@ type RestoreSpec struct {
Storage StorageTargetConfig `json:"storage"`
StoragePath string `json:"storagePath"`
FileName string `json:"fileName"`
Checksum string `json:"checksum,omitempty"`
}
// RestoreUpdate 与 service.AgentRestoreUpdate 对齐
@@ -247,27 +198,6 @@ func (c *MasterClient) GetRestoreSpec(ctx context.Context, restoreRecordID uint)
return &spec, nil
}
func (c *MasterClient) DownloadRestoreArtifact(ctx context.Context, restoreRecordID uint) (io.ReadCloser, error) {
path := fmt.Sprintf("/api/agent/restores/%d/artifact", restoreRecordID)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("X-Agent-Token", c.token)
client := *c.httpClient
client.Timeout = 0
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("download relayed artifact from Master: %w", err)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return resp.Body, nil
}
data, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
closeErr := resp.Body.Close()
return nil, errors.Join(fmt.Errorf("download relayed artifact from Master: http %d: %s", resp.StatusCode, string(data)), readErr, closeErr)
}
// UpdateRestore 上报恢复记录的状态/日志
func (c *MasterClient) UpdateRestore(ctx context.Context, restoreRecordID uint, update RestoreUpdate) error {
path := fmt.Sprintf("/api/agent/restores/%d", restoreRecordID)

View File

@@ -26,7 +26,7 @@ type Config struct {
HeartbeatInterval string `yaml:"heartbeatInterval"`
// PollInterval 命令轮询间隔,默认 5s
PollInterval string `yaml:"pollInterval"`
// TempDir 备份临时目录,默认 /var/lib/backupx-agent/tmp
// TempDir 备份临时目录,默认 /tmp/backupx-agent
TempDir string `yaml:"tempDir"`
// InsecureSkipTLSVerify 测试环境允许跳过 TLS 证书校验
InsecureSkipTLSVerify bool `yaml:"insecureSkipTlsVerify"`
@@ -98,7 +98,7 @@ func applyConfigDefaults(cfg *Config) (*Config, error) {
cfg.PollInterval = "5s"
}
if cfg.TempDir == "" {
cfg.TempDir = "/var/lib/backupx-agent/tmp"
cfg.TempDir = "/tmp/backupx-agent"
}
cfg.Master = strings.TrimRight(strings.TrimSpace(cfg.Master), "/")
return cfg, nil

View File

@@ -50,7 +50,7 @@ func TestLoadConfigDefaults(t *testing.T) {
if cfg.HeartbeatInterval != "15s" || cfg.PollInterval != "5s" {
t.Errorf("default intervals not applied: %+v", cfg)
}
if cfg.TempDir != "/var/lib/backupx-agent/tmp" {
if cfg.TempDir != "/tmp/backupx-agent" {
t.Errorf("default tempdir: %q", cfg.TempDir)
}
}

View File

@@ -4,8 +4,6 @@ import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
@@ -21,10 +19,10 @@ import (
// Executor 负责在 Agent 本地执行命令。
type Executor struct {
client *MasterClient
tempDir string
backupRegistry *backup.Registry
storageRegistry *storage.Registry
client *MasterClient
tempDir string
backupRegistry *backup.Registry
storageRegistry *storage.Registry
}
// NewExecutor 构造执行器。预先初始化 backup runner 与 storage registry。
@@ -35,7 +33,6 @@ func NewExecutor(client *MasterClient, tempDir string) *Executor {
backup.NewMySQLRunner(nil),
backup.NewPostgreSQLRunner(nil),
backup.NewSAPHANARunner(nil),
backup.NewMongoDBRunner(nil),
)
storageRegistry := storage.NewRegistry(
storageRclone.NewLocalDiskFactory(),
@@ -62,11 +59,6 @@ func NewExecutor(client *MasterClient, tempDir string) *Executor {
// 注意Agent 当前不支持 Encrypt=true加密密钥不下发到 Agent避免密钥扩散
// 遇到启用加密的任务会向 Master 上报失败并返回错误。
func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) error {
if err := e.ensureTempDir(); err != nil {
e.reportRecordFailure(ctx, recordID, err.Error())
return err
}
// 1) 拉取任务规格
spec, err := e.client.GetTaskSpec(ctx, taskID)
if err != nil {
@@ -82,6 +74,10 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
// 2) 构造 backup.TaskSpec 并找对应 runner
startedAt := time.Now().UTC()
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("创建临时目录失败: %v", err))
return err
}
backupSpec := buildBackupTaskSpec(spec, startedAt, e.tempDir)
runner, err := e.backupRegistry.Runner(backupSpec.Type)
if err != nil {
@@ -108,14 +104,6 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
return compressErr
}
finalPath = compressedPath
} else if strings.EqualFold(spec.Compression, "zstd") && !strings.HasSuffix(strings.ToLower(finalPath), ".zst") {
e.appendLog(ctx, recordID, "[agent] 开始压缩备份文件zstd\n")
compressedPath, compressErr := compress.ZstdFile(finalPath)
if compressErr != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("压缩失败: %v", compressErr))
return compressErr
}
finalPath = compressedPath
}
info, err := os.Stat(finalPath)
if err != nil {
@@ -124,7 +112,7 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
}
fileName := filepath.Base(finalPath)
fileSize := info.Size()
storagePath := backup.BuildRecordStorageKey(spec.Type, startedAt, recordID, fileName)
storagePath := backup.BuildStorageKey(spec.Type, startedAt, fileName)
// 5) 计算 checksum一次读一次并上传到所有目标
checksum, err := computeFileSHA256(finalPath)
@@ -136,88 +124,48 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
e.reportRecordFailure(ctx, recordID, "没有关联的存储目标")
return fmt.Errorf("no storage targets")
}
uploadResults := make([]StorageResultItem, 0, len(spec.StorageTargets))
selectedStorageTargetID := uint(0)
selectedStorageTransferMode := ""
var uploadErrors []string
for _, target := range spec.StorageTargets {
if err := e.uploadToTarget(ctx, recordID, target, finalPath, storagePath, fileSize, checksum, spec.TaskID); err != nil {
uploadResults = append(uploadResults, StorageResultItem{
StorageTargetID: target.ID,
StorageTargetName: target.Name,
Status: "failed",
TransferMode: target.TransferMode,
Error: err.Error(),
})
uploadErrors = append(uploadErrors, fmt.Sprintf("%s: %v", target.Name, err))
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 上传到存储目标 %s 失败: %v\n", target.Name, err))
continue
if err := e.uploadToTarget(ctx, recordID, target, finalPath, storagePath, fileSize, spec.TaskID); err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("上传到 %s 失败: %v", target.Name, err))
return err
}
if selectedStorageTargetID == 0 {
selectedStorageTargetID = target.ID
selectedStorageTransferMode = target.TransferMode
}
uploadResults = append(uploadResults, StorageResultItem{
StorageTargetID: target.ID,
StorageTargetName: target.Name,
Status: "success",
StoragePath: storagePath,
FileSize: fileSize,
TransferMode: target.TransferMode,
})
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 已上传到存储目标 %s\n", target.Name))
}
if selectedStorageTargetID == 0 {
msg := strings.Join(uploadErrors, "; ")
if msg == "" {
msg = "所有存储目标上传均失败"
}
e.reportRecordFailureWithUploadResults(ctx, recordID, msg, uploadResults)
return fmt.Errorf("%s", msg)
}
// 6) 上报最终成功
return e.client.UpdateRecord(ctx, recordID, RecordUpdate{
Status: "success",
FileName: fileName,
FileSize: fileSize,
Checksum: checksum,
StoragePath: storagePath,
StorageTargetID: selectedStorageTargetID,
StorageTransferMode: selectedStorageTransferMode,
StorageUploadResults: uploadResults,
LogAppend: fmt.Sprintf("[agent] 任务完成,总计 %d 字节\n", fileSize),
Status: "success",
FileName: fileName,
FileSize: fileSize,
Checksum: checksum,
StoragePath: storagePath,
LogAppend: fmt.Sprintf("[agent] 任务完成,总计 %d 字节\n", fileSize),
})
}
// uploadToTarget 上传单个目标。为保持简化不做上传级重试rclone 本身已有 low-level 重试)。
func (e *Executor) uploadToTarget(ctx context.Context, recordID uint, target StorageTargetConfig, filePath, objectKey string, fileSize int64, checksum string, taskID uint) error {
f, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("open artifact: %w", err)
}
if target.TransferMode == storage.TransferModeMasterRelay {
uploadErr := e.client.UploadArtifact(ctx, recordID, target.ID, objectKey, fileSize, checksum, f)
return errors.Join(uploadErr, f.Close())
}
func (e *Executor) uploadToTarget(ctx context.Context, recordID uint, target StorageTargetConfig, filePath, objectKey string, fileSize int64, taskID uint) error {
var rawConfig map[string]any
if len(target.Config) > 0 {
// DecodeRawConfig 通过 json 解析
if err := jsonUnmarshalMap(target.Config, &rawConfig); err != nil {
return errors.Join(fmt.Errorf("parse storage config: %w", err), f.Close())
return fmt.Errorf("parse storage config: %w", err)
}
}
provider, err := e.storageRegistry.Create(ctx, target.Type, rawConfig)
if err != nil {
closeErr := f.Close()
return errors.Join(fmt.Errorf("create provider: %w", err), closeErr)
return fmt.Errorf("create provider: %w", err)
}
f, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("open artifact: %w", err)
}
defer f.Close()
meta := map[string]string{
"taskId": fmt.Sprintf("%d", taskID),
"recordId": fmt.Sprintf("%d", recordID),
}
uploadErr := provider.Upload(ctx, objectKey, f, fileSize, meta)
return errors.Join(uploadErr, f.Close())
return provider.Upload(ctx, objectKey, f, fileSize, meta)
}
// appendLog 追加日志到 Master 记录(尽力而为,失败不中断主流程)
@@ -227,22 +175,31 @@ func (e *Executor) appendLog(ctx context.Context, recordID uint, line string) {
// reportRecordFailure 上报失败状态
func (e *Executor) reportRecordFailure(ctx context.Context, recordID uint, msg string) {
e.reportRecordFailureWithUploadResults(ctx, recordID, msg, nil)
}
func (e *Executor) reportRecordFailureWithUploadResults(ctx context.Context, recordID uint, msg string, uploadResults []StorageResultItem) {
_ = e.client.UpdateRecord(ctx, recordID, RecordUpdate{
Status: "failed",
ErrorMessage: msg,
StorageUploadResults: uploadResults,
LogAppend: fmt.Sprintf("[agent] 错误: %s\n", msg),
Status: "failed",
ErrorMessage: msg,
LogAppend: fmt.Sprintf("[agent] 错误: %s\n", msg),
})
}
// buildBackupTaskSpec 把 AgentTaskSpec 转换为 backup.TaskSpec。
func buildBackupTaskSpec(spec *TaskSpec, startedAt time.Time, tempDir string) backup.TaskSpec {
sourcePaths := parseStringListField(spec.SourcePaths)
excludes := parseStringListField(spec.ExcludePatterns)
var sourcePaths []string
if strings.TrimSpace(spec.SourcePaths) != "" {
for _, p := range strings.Split(spec.SourcePaths, "\n") {
if p = strings.TrimSpace(p); p != "" {
sourcePaths = append(sourcePaths, p)
}
}
}
var excludes []string
if strings.TrimSpace(spec.ExcludePatterns) != "" {
for _, p := range strings.Split(spec.ExcludePatterns, "\n") {
if p = strings.TrimSpace(p); p != "" {
excludes = append(excludes, p)
}
}
}
return backup.TaskSpec{
ID: spec.TaskID,
Name: spec.Name,
@@ -265,37 +222,6 @@ func buildBackupTaskSpec(spec *TaskSpec, startedAt time.Time, tempDir string) ba
}
}
func (e *Executor) ensureTempDir() error {
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
return fmt.Errorf("create agent temp dir: %w", err)
}
return nil
}
func parseStringListField(value string) []string {
trimmed := strings.TrimSpace(value)
if trimmed == "" || trimmed == "[]" {
return nil
}
var jsonItems []string
if err := json.Unmarshal([]byte(trimmed), &jsonItems); err == nil {
return compactStringList(jsonItems)
}
return compactStringList(strings.FieldsFunc(trimmed, func(r rune) bool {
return r == '\n' || r == '\r'
}))
}
func compactStringList(items []string) []string {
result := make([]string, 0, len(items))
for _, item := range items {
if trimmed := strings.TrimSpace(item); trimmed != "" {
result = append(result, trimmed)
}
}
return result
}
// recordLogger 把 runner 日志回传到 Master 记录。
// 实现 backup.LogWriter每条日志追加到 record.log_content。
type recordLogger struct {
@@ -314,8 +240,8 @@ func (l *recordLogger) WriteLine(message string) {
// restoreLogger 把 runner 日志回传到 Master 恢复记录。
type restoreLogger struct {
ctx context.Context
client *MasterClient
ctx context.Context
client *MasterClient
restoreID uint
}
@@ -339,16 +265,11 @@ func (e *Executor) DeleteStorageObject(ctx context.Context, targetType string, t
// ExecuteRestore 处理 restore_record 命令:拉规格 → 下载 → 解压 → 执行 runner.Restore → 上报结果。
//
// 与 ExecuteRunTask 对称,但方向相反:
// - 下载:直连共享存储,或通过 Master 中转其本地磁盘对象
// - 下载:通过 spec.Storage 创建 provider → Download(spec.StoragePath)
// - 解密:当前 Agent 不支持加密恢复密钥未下发spec.Encrypt=true 会直接失败
// - 执行backup.Registry.Runner(spec.Type).Restore
// - 上报:通过 UpdateRestorestatus/logAppend
func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) error {
if err := e.ensureTempDir(); err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, err.Error())
return err
}
spec, err := e.client.GetRestoreSpec(ctx, restoreRecordID)
if err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("拉取恢复规格失败: %v", err))
@@ -361,6 +282,10 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
}
e.appendRestoreLog(ctx, restoreRecordID, fmt.Sprintf("[agent] 开始恢复 %s (type=%s)\n", spec.TaskName, spec.Type))
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建临时目录失败: %v", err))
return err
}
tmpDir, err := os.MkdirTemp(e.tempDir, "restore-*")
if err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建恢复临时目录失败: %v", err))
@@ -368,31 +293,28 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
}
defer os.RemoveAll(tmpDir)
// 1) 下载
// 1) 创建 storage provider
var rawConfig map[string]any
if len(spec.Storage.Config) > 0 {
if err := jsonUnmarshalMap(spec.Storage.Config, &rawConfig); err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解析存储配置失败: %v", err))
return err
}
}
provider, err := e.storageRegistry.Create(ctx, spec.Storage.Type, rawConfig)
if err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建存储客户端失败: %v", err))
return err
}
// 2) 下载
fileName := spec.FileName
if strings.TrimSpace(fileName) == "" {
fileName = filepath.Base(spec.StoragePath)
}
artifactPath := filepath.Join(tmpDir, filepath.Base(fileName))
e.appendRestoreLog(ctx, restoreRecordID, fmt.Sprintf("[agent] 下载备份文件 %s\n", spec.StoragePath))
var reader io.ReadCloser
if spec.Storage.TransferMode == storage.TransferModeMasterRelay {
reader, err = e.client.DownloadRestoreArtifact(ctx, restoreRecordID)
} else {
var rawConfig map[string]any
if len(spec.Storage.Config) > 0 {
if err := jsonUnmarshalMap(spec.Storage.Config, &rawConfig); err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解析存储配置失败: %v", err))
return err
}
}
provider, providerErr := e.storageRegistry.Create(ctx, spec.Storage.Type, rawConfig)
if providerErr != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建存储客户端失败: %v", providerErr))
return providerErr
}
reader, err = provider.Download(ctx, spec.StoragePath)
}
reader, err := provider.Download(ctx, spec.StoragePath)
if err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("下载备份失败: %v", err))
return err
@@ -402,24 +324,6 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
return err
}
// 2.5) 完整性校验:还原前比对下载对象的 SHA-256与 Master 本地恢复路径一致)。
// 拒绝还原损坏或被篡改的备份;早期无 checksum 的备份跳过(向后兼容)。
if strings.TrimSpace(spec.Checksum) != "" {
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 校验备份完整性SHA-256\n")
actual, sumErr := computeFileSHA256(artifactPath)
if sumErr != nil {
msg := fmt.Sprintf("计算校验和失败: %v", sumErr)
e.reportRestoreFailure(ctx, restoreRecordID, msg)
return fmt.Errorf("%s", msg)
}
if !strings.EqualFold(actual, spec.Checksum) {
msg := "备份文件完整性校验失败SHA-256 不匹配,文件可能已损坏或被篡改"
e.reportRestoreFailure(ctx, restoreRecordID, msg)
return fmt.Errorf("%s期望 %s实际 %s", msg, spec.Checksum, actual)
}
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 完整性校验通过\n")
}
// 3) 解压Agent 不支持加密,遇到 .enc 会直接失败)
preparedPath := artifactPath
if strings.HasSuffix(strings.ToLower(preparedPath), ".enc") {
@@ -436,15 +340,6 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
}
preparedPath = decompressed
}
if strings.HasSuffix(strings.ToLower(preparedPath), ".zst") {
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 解压 zstd 压缩\n")
decompressed, err := compress.UnzstdFile(preparedPath)
if err != nil {
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解压失败: %v", err))
return err
}
preparedPath = decompressed
}
// 4) 运行 runner.Restore
taskSpec := buildRestoreBackupTaskSpec(spec, time.Now().UTC(), tmpDir)
@@ -503,10 +398,8 @@ func buildRestoreBackupTaskSpec(spec *RestoreSpec, startedAt time.Time, tempDir
}
// writeReaderToLocal 把 reader 写到本地文件Agent 侧工具函数)。
func writeReaderToLocal(targetPath string, reader io.ReadCloser) (err error) {
defer func() {
err = errors.Join(err, reader.Close())
}()
func writeReaderToLocal(targetPath string, reader io.ReadCloser) error {
defer reader.Close()
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return err
}
@@ -514,8 +407,9 @@ func writeReaderToLocal(targetPath string, reader io.ReadCloser) (err error) {
if err != nil {
return err
}
_, copyErr := io.Copy(file, reader)
return errors.Join(copyErr, file.Close())
defer file.Close()
_, err = io.Copy(file, reader)
return err
}
// 辅助函数

View File

@@ -1,380 +0,0 @@
package agent
import (
"archive/tar"
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
"backupx/server/internal/storage"
)
func TestBuildBackupTaskSpecParsesJSONSourcePaths(t *testing.T) {
spec := &TaskSpec{
TaskID: 7,
Name: "root-files",
Type: "file",
SourcePaths: `["/root","/etc"]`,
ExcludePatterns: `["*.log","tmp"]`,
}
got := buildBackupTaskSpec(spec, time.Unix(0, 0), "/var/lib/backupx-agent/tmp")
if !reflect.DeepEqual(got.SourcePaths, []string{"/root", "/etc"}) {
t.Fatalf("source paths = %#v", got.SourcePaths)
}
if !reflect.DeepEqual(got.ExcludePatterns, []string{"*.log", "tmp"}) {
t.Fatalf("exclude patterns = %#v", got.ExcludePatterns)
}
}
func TestParseStringListFieldKeepsLegacyLineFormat(t *testing.T) {
got := parseStringListField("/root\n /etc \n")
want := []string{"/root", "/etc"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("paths = %#v, want %#v", got, want)
}
}
func TestExecuteRunTaskRecordsPerTargetUploadResults(t *testing.T) {
sourceDir := t.TempDir()
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("hello"), 0o644); err != nil {
t.Fatalf("WriteFile returned error: %v", err)
}
var finalUpdate RecordUpdate
var updates []RecordUpdate
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
writeAgentEnvelope(t, w, TaskSpec{
TaskID: 1,
Name: "site",
Type: "file",
SourcePath: sourceDir,
Compression: "gzip",
StorageTargets: []StorageTargetConfig{
{ID: 11, Name: "broken", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken"}`)},
{ID: 12, Name: "ok", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"ok"}`)},
},
})
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
var update RecordUpdate
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
t.Fatalf("Decode update returned error: %v", err)
}
updates = append(updates, update)
if update.Status != "" {
finalUpdate = update
}
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
default:
http.NotFound(w, r)
}
}))
defer server.Close()
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
executor.storageRegistry = storage.NewRegistry(&agentTestStorageFactory{
providers: map[string]*agentTestStorageProvider{
"broken": {name: "broken", failUpload: true},
"ok": {name: "ok", objects: map[string][]byte{}},
},
})
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err != nil {
t.Fatalf("ExecuteRunTask returned error: %v", err)
}
if len(updates) == 0 || finalUpdate.Status != "success" {
t.Fatalf("expected final success update, got updates=%#v final=%#v", updates, finalUpdate)
}
if finalUpdate.StorageTargetID != 12 {
t.Fatalf("expected first successful target 12, got %d", finalUpdate.StorageTargetID)
}
if len(finalUpdate.StorageUploadResults) != 2 {
t.Fatalf("expected two upload results, got %#v", finalUpdate.StorageUploadResults)
}
if finalUpdate.StorageUploadResults[0].Status != "failed" || finalUpdate.StorageUploadResults[1].Status != "success" {
t.Fatalf("unexpected upload results: %#v", finalUpdate.StorageUploadResults)
}
if finalUpdate.StoragePath == "" || finalUpdate.FileSize <= 0 || finalUpdate.Checksum == "" {
t.Fatalf("expected artifact metadata in final update, got %#v", finalUpdate)
}
}
func TestExecuteRunTaskRelaysMasterLocalDiskTarget(t *testing.T) {
sourceDir := t.TempDir()
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("centralize me"), 0o644); err != nil {
t.Fatalf("WriteFile returned error: %v", err)
}
var relayed []byte
var finalUpdate RecordUpdate
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
writeAgentEnvelope(t, w, TaskSpec{
TaskID: 1,
Name: "remote-source",
Type: "file",
SourcePath: sourceDir,
Compression: "gzip",
StorageTargets: []StorageTargetConfig{{
ID: 11, Name: "master-disk", Type: storage.TypeLocalDisk, TransferMode: storage.TransferModeMasterRelay,
}},
})
case r.Method == http.MethodPut && r.URL.Path == "/api/agent/records/99/artifacts/11":
body, err := io.ReadAll(r.Body)
if err != nil {
t.Fatalf("ReadAll relayed body: %v", err)
}
digest := sha256.Sum256(body)
if got := r.Header.Get("X-BackupX-SHA256"); got != fmt.Sprintf("%x", digest[:]) {
t.Fatalf("relay checksum header = %q", got)
}
objectKey := r.Header.Get("X-BackupX-Object-Key")
if !strings.Contains(objectKey, "/records/99/") || r.ContentLength != int64(len(body)) {
t.Fatalf("invalid relay metadata: key=%q length=%d body=%d", objectKey, r.ContentLength, len(body))
}
relayed = append([]byte(nil), body...)
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
var update RecordUpdate
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
t.Fatalf("Decode update returned error: %v", err)
}
if update.Status != "" {
finalUpdate = update
}
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
default:
http.NotFound(w, r)
}
}))
defer server.Close()
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err != nil {
t.Fatalf("ExecuteRunTask returned error: %v", err)
}
if len(relayed) == 0 {
t.Fatal("expected artifact bytes to be streamed through Master")
}
if finalUpdate.Status != "success" || finalUpdate.StorageTransferMode != storage.TransferModeMasterRelay {
t.Fatalf("unexpected final relay update: %#v", finalUpdate)
}
if len(finalUpdate.StorageUploadResults) != 1 || finalUpdate.StorageUploadResults[0].TransferMode != storage.TransferModeMasterRelay {
t.Fatalf("unexpected relay target result: %#v", finalUpdate.StorageUploadResults)
}
}
func TestExecuteRestoreDownloadsMasterRelayedArtifact(t *testing.T) {
var archive bytes.Buffer
tarWriter := tar.NewWriter(&archive)
content := []byte("restored through Master")
header := &tar.Header{Name: "site/index.html", Mode: 0o644, Size: int64(len(content)), Typeflag: tar.TypeReg}
if err := tarWriter.WriteHeader(header); err != nil {
t.Fatalf("WriteHeader returned error: %v", err)
}
if _, err := tarWriter.Write(content); err != nil {
t.Fatalf("Write returned error: %v", err)
}
if err := tarWriter.Close(); err != nil {
t.Fatalf("Close returned error: %v", err)
}
artifact := append([]byte(nil), archive.Bytes()...)
digest := sha256.Sum256(artifact)
restoreRoot := t.TempDir()
restoreSource := filepath.Join(restoreRoot, "site")
artifactRequests := 0
var finalUpdate RestoreUpdate
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/restores/77/spec":
writeAgentEnvelope(t, w, RestoreSpec{
RestoreRecordID: 77,
BackupRecordID: 99,
TaskID: 1,
TaskName: "remote-source",
Type: "file",
SourcePath: restoreSource,
Storage: StorageTargetConfig{
ID: 11, Name: "master-disk", Type: storage.TypeLocalDisk, TransferMode: storage.TransferModeMasterRelay,
},
StoragePath: "BackupX/file/site.tar",
FileName: "site.tar",
Checksum: fmt.Sprintf("%x", digest[:]),
})
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/restores/77/artifact":
artifactRequests++
if r.Header.Get("X-Agent-Token") != "token" {
t.Fatalf("missing Agent token on relay download")
}
w.Header().Set("Content-Length", fmt.Sprintf("%d", len(artifact)))
_, _ = w.Write(artifact)
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/restores/77":
var update RestoreUpdate
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
t.Fatalf("Decode update returned error: %v", err)
}
if update.Status != "" {
finalUpdate = update
}
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
default:
http.NotFound(w, r)
}
}))
defer server.Close()
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
if err := executor.ExecuteRestore(context.Background(), 77); err != nil {
t.Fatalf("ExecuteRestore returned error: %v", err)
}
if artifactRequests != 1 {
t.Fatalf("expected one relay artifact request, got %d", artifactRequests)
}
restored, err := os.ReadFile(filepath.Join(restoreRoot, "site", "index.html"))
if err != nil {
t.Fatalf("ReadFile returned error: %v", err)
}
if string(restored) != string(content) {
t.Fatalf("restored content = %q, want %q", restored, content)
}
if finalUpdate.Status != "success" {
t.Fatalf("unexpected final restore update: %#v", finalUpdate)
}
}
func TestExecuteRunTaskReportsPerTargetUploadResultsWhenAllTargetsFail(t *testing.T) {
sourceDir := t.TempDir()
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("hello"), 0o644); err != nil {
t.Fatalf("WriteFile returned error: %v", err)
}
var finalUpdate RecordUpdate
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
writeAgentEnvelope(t, w, TaskSpec{
TaskID: 1,
Name: "site",
Type: "file",
SourcePath: sourceDir,
Compression: "gzip",
StorageTargets: []StorageTargetConfig{
{ID: 11, Name: "broken-a", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken-a"}`)},
{ID: 12, Name: "broken-b", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken-b"}`)},
},
})
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
var update RecordUpdate
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
t.Fatalf("Decode update returned error: %v", err)
}
if update.Status != "" {
finalUpdate = update
}
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
default:
http.NotFound(w, r)
}
}))
defer server.Close()
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
executor.storageRegistry = storage.NewRegistry(&agentTestStorageFactory{
providers: map[string]*agentTestStorageProvider{
"broken-a": {name: "broken-a", failUpload: true},
"broken-b": {name: "broken-b", failUpload: true},
},
})
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err == nil {
t.Fatal("expected ExecuteRunTask to return upload failure")
}
if finalUpdate.Status != "failed" {
t.Fatalf("expected final failed update, got %#v", finalUpdate)
}
if len(finalUpdate.StorageUploadResults) != 2 {
t.Fatalf("expected failed update to keep per-target results, got %#v", finalUpdate.StorageUploadResults)
}
for _, item := range finalUpdate.StorageUploadResults {
if item.Status != "failed" || item.Error == "" {
t.Fatalf("unexpected upload result: %#v", item)
}
}
}
type agentTestStorageFactory struct {
providers map[string]*agentTestStorageProvider
}
func (f *agentTestStorageFactory) Type() storage.ProviderType {
return "agent_test_storage"
}
func (f *agentTestStorageFactory) New(_ context.Context, config map[string]any) (storage.StorageProvider, error) {
name, _ := config["name"].(string)
provider := f.providers[name]
if provider == nil {
return nil, fmt.Errorf("unknown provider %q", name)
}
return provider, nil
}
type agentTestStorageProvider struct {
name string
failUpload bool
objects map[string][]byte
}
func (p *agentTestStorageProvider) Type() storage.ProviderType { return "agent_test_storage" }
func (p *agentTestStorageProvider) TestConnection(context.Context) error {
return nil
}
func (p *agentTestStorageProvider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
if p.failUpload {
return fmt.Errorf("upload failed for %s", p.name)
}
data, err := io.ReadAll(reader)
if err != nil {
return err
}
if p.objects == nil {
p.objects = map[string][]byte{}
}
p.objects[objectKey] = data
return nil
}
func (p *agentTestStorageProvider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
data, ok := p.objects[objectKey]
if !ok {
return nil, fmt.Errorf("object %s not found", objectKey)
}
return io.NopCloser(strings.NewReader(string(data))), nil
}
func (p *agentTestStorageProvider) Delete(_ context.Context, objectKey string) error {
delete(p.objects, objectKey)
return nil
}
func (p *agentTestStorageProvider) List(context.Context, string) ([]storage.ObjectInfo, error) {
return nil, nil
}
func writeAgentEnvelope(t *testing.T, w http.ResponseWriter, data any) {
t.Helper()
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(map[string]any{"code": "OK", "data": data}); err != nil {
t.Fatalf("Encode response returned error: %v", err)
}
}

View File

@@ -5,7 +5,6 @@ import (
"os"
"path/filepath"
"sort"
"strings"
)
// DirEntry Agent 返回给 Master 的目录项。
@@ -18,8 +17,8 @@ type DirEntry struct {
// listLocalDir 列出 Agent 所在机器的指定路径。
func listLocalDir(path string) ([]DirEntry, error) {
cleaned := filepath.Clean(strings.TrimSpace(path))
if strings.TrimSpace(path) == "" || cleaned == "." {
cleaned := filepath.Clean(path)
if cleaned == "" {
cleaned = "/"
}
entries, err := os.ReadDir(cleaned)

View File

@@ -36,21 +36,6 @@ func TestListLocalDir(t *testing.T) {
}
}
func TestListLocalDirEmptyPathUsesRoot(t *testing.T) {
entries, err := listLocalDir("")
if err != nil {
t.Fatalf("list root: %v", err)
}
if len(entries) == 0 {
t.Fatalf("expected root entries")
}
for _, entry := range entries {
if !filepath.IsAbs(entry.Path) {
t.Fatalf("entry path should be absolute: %+v", entry)
}
}
}
func TestSplitCommaOrNewline(t *testing.T) {
cases := []struct {
in string

View File

@@ -82,9 +82,9 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
backupTaskService := service.NewBackupTaskService(backupTaskRepo, storageTargetRepo, configCipher)
backupTaskService.SetRecordsAndStorage(backupRecordRepo, storageRegistry)
// nodeRepo 在下方 Cluster 节点管理区块才实例化,这里延后注入
backupRunnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil), backup.NewSAPHANARunner(nil), backup.NewMongoDBRunner(nil))
backupRunnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil), backup.NewSAPHANARunner(nil))
logHub := backup.NewLogHub()
retentionService := backupretention.NewService(backupRecordRepo, configCipher.Key())
retentionService := backupretention.NewService(backupRecordRepo)
notifyRegistry := notify.NewRegistry(notify.NewEmailNotifier(), notify.NewWebhookNotifier(), notify.NewTelegramNotifier())
notificationService := service.NewNotificationService(notificationRepo, notifyRegistry, configCipher)
authService.SetNotificationService(notificationService)
@@ -104,7 +104,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
restoreRecordRepo := repository.NewRestoreRecordRepository(db)
restoreLogHub := backup.NewLogHub()
dashboardService := service.NewDashboardService(backupTaskRepo, backupRecordRepo, storageTargetRepo)
reportService := service.NewReportService(backupTaskRepo, backupRecordRepo)
settingsService := service.NewSettingsService(systemConfigRepo)
// Audit
@@ -114,8 +113,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
schedulerService.SetAuditRecorder(auditService)
// 审计日志外输:启动时用当前 settings 初始化 webhook后续前端修改立即生效
settingsService.SetAuditWebhookConfigurer(ctx, auditService)
// 审计日志保留期清理:每 6h 读取 audit_retention_days 设置并清理超期日志0/缺省=永久保留)
auditService.StartRetentionMonitor(ctx, systemConfigRepo, 6*time.Hour)
// Database discovery集群依赖在 agentService 创建后注入)
databaseDiscoveryService := service.NewDatabaseDiscoveryService(backup.NewOSCommandExecutor())
@@ -134,8 +131,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
// Agent 协议服务:命令队列 + 任务下发 + 记录上报
agentCmdRepo := repository.NewAgentCommandRepository(db)
nodeService.SetAgentCommandRepository(agentCmdRepo)
agentService := service.NewAgentService(nodeRepo, backupTaskRepo, backupRecordRepo, storageTargetRepo, agentCmdRepo, configCipher, storageRegistry)
agentService := service.NewAgentService(nodeRepo, backupTaskRepo, backupRecordRepo, storageTargetRepo, agentCmdRepo, configCipher)
agentService.SetRestoreRepository(restoreRecordRepo)
agentService.StartCommandTimeoutMonitor(ctx, 30*time.Second, 10*time.Minute)
@@ -244,7 +240,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
replicationService.SetMetrics(appMetrics)
metricsCollector := metrics.NewCollector(
appMetrics,
metrics.NewRepoSource(storageTargetRepo, backupRecordRepo, nodeRepo, backupTaskRepo, agentCmdRepo),
metrics.NewRepoSource(storageTargetRepo, backupRecordRepo, nodeRepo, backupTaskRepo),
30*time.Second,
)
metricsCollector.Start(ctx)
@@ -271,7 +267,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
ApiKeyService: apiKeyService,
NotificationService: notificationService,
DashboardService: dashboardService,
ReportService: reportService,
SettingsService: settingsService,
NodeService: nodeService,
AgentService: agentService,
@@ -281,7 +276,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
UserRepository: userRepo,
SystemConfigRepo: systemConfigRepo,
InstallTokenService: installTokenService,
MasterExternalURL: cfg.Server.ExternalURL,
MasterExternalURL: "", // 如需覆盖 URL可扩展 cfg.Server 增字段;目前留空依赖 X-Forwarded-* / Request.Host
DB: db,
Metrics: appMetrics,
})

View File

@@ -3,7 +3,6 @@ package backup
import (
"archive/tar"
"context"
"encoding/json"
"fmt"
"io"
"os"
@@ -53,20 +52,6 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
defer tw.Close()
excludes := normalizeExcludePatterns(task.ExcludePatterns)
// 差异备份:基于上次全量清单仅打包新增/变更条目并记录删除;
// 全量备份记录完整清单manifest供后续差异比对。
differential := task.Differential && len(task.BaseManifest.Entries) > 0
baseIndex := map[string]ManifestEntry{}
seen := map[string]struct{}{}
var manifest *Manifest
if differential {
baseIndex = task.BaseManifest.index()
writer.WriteLine(fmt.Sprintf("差异备份模式:基线含 %d 个条目", len(baseIndex)))
} else {
manifest = &Manifest{Entries: make([]ManifestEntry, 0)}
}
totalFileCount := 0
totalDirCount := 0
@@ -103,16 +88,6 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
return nil
}
entry := entryFromInfo(archiveName, currentInfo)
if differential {
seen[entry.Path] = struct{}{}
if !changedSince(baseIndex, entry) {
return nil // 自全量以来未变更,跳过
}
} else {
manifest.Entries = append(manifest.Entries, entry)
}
if currentInfo.IsDir() {
dirCount++
writer.WriteLine(fmt.Sprintf("📁 进入目录 %s", archiveName))
@@ -128,19 +103,13 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
}
if currentInfo.Mode().IsRegular() {
// 每个文件在独立作用域内打开并关闭,避免在大目录树中累积打开的文件句柄。
if copyErr := func() error {
file, err := os.Open(currentPath)
if err != nil {
return err
}
defer file.Close()
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
return err
}
return nil
}(); copyErr != nil {
return copyErr
file, err := os.Open(currentPath)
if err != nil {
return err
}
defer file.Close()
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
return err
}
fileCount++
if fileCount%100 == 0 {
@@ -161,16 +130,10 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
totalDirCount += dirCount
}
if differential {
deletions := deletedPaths(baseIndex, seen)
if err := writeDeletionsEntry(tw, deletions); err != nil {
return nil, err
}
writer.WriteLine(fmt.Sprintf("差异备份完成(%d 个目录、%d 个文件变更,删除 %d 项)", totalDirCount, totalFileCount, len(deletions)))
} else if len(sourcePaths) > 1 {
if len(sourcePaths) > 1 {
writer.WriteLine(fmt.Sprintf("全部源路径打包完成(共 %d 个目录,%d 个文件)", totalDirCount, totalFileCount))
}
return &RunResult{ArtifactPath: artifactPath, FileName: filepath.Base(artifactPath), TempDir: tempDir, Manifest: manifest}, nil
return &RunResult{ArtifactPath: artifactPath, FileName: filepath.Base(artifactPath), TempDir: tempDir}, nil
}
func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath string, writer LogWriter) error {
@@ -185,15 +148,9 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
restoreSource = task.SourcePaths[0]
}
targetParent := filepath.Dir(filepath.Clean(strings.TrimSpace(restoreSource)))
// 恢复到指定位置:非空时归档解压到用户指定目录,而非原始源父目录。
if override := strings.TrimSpace(task.RestoreTargetPath); override != "" {
targetParent = filepath.Clean(override)
writer.WriteLine(fmt.Sprintf("恢复到指定目录:%s", targetParent))
}
if err := os.MkdirAll(targetParent, 0o755); err != nil {
return fmt.Errorf("create restore parent: %w", err)
}
var pendingDeletions []string
tr := tar.NewReader(artifactFile)
for {
header, err := tr.Next()
@@ -203,27 +160,13 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
if err != nil {
return fmt.Errorf("read tar entry: %w", err)
}
// 差异归档的删除清单不落地,留待提取完成后统一应用(避免被同批新增条目误删)。
if header.Name == deletionsEntryName {
data, readErr := io.ReadAll(tr)
if readErr != nil {
return fmt.Errorf("read deletions entry: %w", readErr)
}
if jsonErr := json.Unmarshal(data, &pendingDeletions); jsonErr != nil {
return fmt.Errorf("parse deletions entry: %w", jsonErr)
}
continue
}
cleanName := path.Clean(strings.TrimSpace(header.Name))
if cleanName == "." || cleanName == "" {
continue
}
// 选择性恢复:仅提取被选中的文件/目录(及其子项)。
if len(task.SelectedPaths) > 0 && !pathSelected(cleanName, task.SelectedPaths) {
continue
}
targetPath, ok := resolveWithinParent(targetParent, cleanName)
if !ok {
targetPath := filepath.Clean(filepath.Join(targetParent, filepath.FromSlash(cleanName)))
parentWithSep := filepath.Clean(targetParent) + string(filepath.Separator)
if targetPath != filepath.Clean(targetParent) && !strings.HasPrefix(targetPath, parentWithSep) {
return fmt.Errorf("tar entry escapes restore path")
}
switch header.Typeflag {
@@ -248,94 +191,10 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
}
}
}
// 选择性恢复时仅对选中范围应用删除,避免误删未选中的文件。
if len(task.SelectedPaths) > 0 {
pendingDeletions = filterSelectedPaths(pendingDeletions, task.SelectedPaths)
}
if err := applyDeletions(targetParent, pendingDeletions, writer); err != nil {
return err
}
writer.WriteLine("文件恢复完成")
return nil
}
// pathSelected 判断归档条目名是否落在选中集合内(精确匹配或位于选中目录之下)。
func pathSelected(name string, selected []string) bool {
for _, sel := range selected {
clean := path.Clean(strings.TrimSpace(sel))
if clean == "" || clean == "." {
continue
}
if name == clean || strings.HasPrefix(name, clean+"/") {
return true
}
}
return false
}
// filterSelectedPaths 仅保留落在选中集合内的路径。
func filterSelectedPaths(paths []string, selected []string) []string {
filtered := make([]string, 0, len(paths))
for _, p := range paths {
if pathSelected(path.Clean(strings.TrimSpace(p)), selected) {
filtered = append(filtered, p)
}
}
return filtered
}
// resolveWithinParent 将归档相对名安全解析为 targetParent 下的绝对路径;
// 越界(路径穿越)时返回 ok=false。提取与删除共用此校验杜绝逃逸。
func resolveWithinParent(targetParent, name string) (string, bool) {
targetPath := filepath.Clean(filepath.Join(targetParent, filepath.FromSlash(name)))
cleanParent := filepath.Clean(targetParent)
if targetPath == cleanParent {
return targetPath, true
}
if !strings.HasPrefix(targetPath, cleanParent+string(filepath.Separator)) {
return "", false
}
return targetPath, true
}
// writeDeletionsEntry 将差异备份的删除路径列表写入归档特殊条目。
func writeDeletionsEntry(tw *tar.Writer, deletions []string) error {
payload, err := json.Marshal(deletions)
if err != nil {
return fmt.Errorf("marshal deletions: %w", err)
}
header := &tar.Header{Name: deletionsEntryName, Mode: 0o600, Size: int64(len(payload)), Typeflag: tar.TypeReg}
if err := tw.WriteHeader(header); err != nil {
return fmt.Errorf("write deletions header: %w", err)
}
if _, err := tw.Write(payload); err != nil {
return fmt.Errorf("write deletions body: %w", err)
}
return nil
}
// applyDeletions 在基线恢复之上删除差异归档记录的路径(仅差异备份恢复时存在)。
// 每个路径经 resolveWithinParent 校验,越界即报错;目标不存在视为已删除。
func applyDeletions(targetParent string, deletions []string, writer LogWriter) error {
for _, name := range deletions {
clean := path.Clean(strings.TrimSpace(name))
if clean == "." || clean == "" {
continue
}
targetPath, ok := resolveWithinParent(targetParent, clean)
if !ok {
return fmt.Errorf("deletion entry escapes restore path")
}
if err := os.RemoveAll(targetPath); err != nil {
return fmt.Errorf("apply deletion %s: %w", clean, err)
}
}
if len(deletions) > 0 {
writer.WriteLine(fmt.Sprintf("已应用差异删除 %d 项", len(deletions)))
}
return nil
}
func normalizeExcludePatterns(items []string) []string {
result := make([]string, 0, len(items))
for _, item := range items {

View File

@@ -1,182 +0,0 @@
package backup
import (
"archive/tar"
"context"
"io"
"os"
"path/filepath"
"testing"
)
func diffWrite(t *testing.T, p, content string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", filepath.Dir(p), err)
}
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatalf("write %s: %v", p, err)
}
}
func diffAssertContent(t *testing.T, p, want string) {
t.Helper()
got, err := os.ReadFile(p)
if err != nil {
t.Fatalf("read %s: %v", p, err)
}
if string(got) != want {
t.Fatalf("%s content = %q, want %q", p, string(got), want)
}
}
func diffAssertAbsent(t *testing.T, p string) {
t.Helper()
if _, err := os.Stat(p); !os.IsNotExist(err) {
t.Fatalf("expected %s to be absent, stat err=%v", p, err)
}
}
func diffArchiveNames(t *testing.T, artifactPath string) map[string]bool {
t.Helper()
f, err := os.Open(artifactPath)
if err != nil {
t.Fatalf("open artifact: %v", err)
}
defer f.Close()
names := map[string]bool{}
tr := tar.NewReader(f)
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("read tar: %v", err)
}
names[h.Name] = true
}
return names
}
// TestFileRunnerDifferentialRoundTrip 验证差异备份的端到端正确性:
// 全量 → 修改源(变更/删除/新增)→ 差异 → 链式恢复(全量+差异)→ 结果与修改后源一致。
func TestFileRunnerDifferentialRoundTrip(t *testing.T) {
work := t.TempDir()
src := filepath.Join(work, "src")
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
diffWrite(t, filepath.Join(src, "b.txt"), "bravo")
diffWrite(t, filepath.Join(src, "sub", "c.txt"), "charlie")
runner := NewFileRunner()
full, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir()}, NopLogWriter{})
if err != nil {
t.Fatalf("full Run: %v", err)
}
if full.Manifest == nil || len(full.Manifest.Entries) == 0 {
t.Fatalf("full backup must produce a manifest, got %#v", full.Manifest)
}
// 变更 a.txt内容变长 → size 差异必被检出)、删除 b.txt、新增 d.txtsub/c.txt 不变
diffWrite(t, filepath.Join(src, "a.txt"), "ALPHA-modified-and-longer")
if err := os.Remove(filepath.Join(src, "b.txt")); err != nil {
t.Fatalf("remove b.txt: %v", err)
}
diffWrite(t, filepath.Join(src, "d.txt"), "delta")
diff, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir(), Differential: true, BaseManifest: *full.Manifest}, NopLogWriter{})
if err != nil {
t.Fatalf("differential Run: %v", err)
}
if diff.Manifest != nil {
t.Fatalf("differential backup must not produce a manifest")
}
// 差异归档应包含变更/新增条目与删除清单,但不含未变更的 sub/c.txt
names := diffArchiveNames(t, diff.ArtifactPath)
if !names["src/a.txt"] || !names["src/d.txt"] {
t.Fatalf("differential archive missing changed/new entries: %v", names)
}
if names["src/sub/c.txt"] {
t.Fatalf("differential archive should not contain unchanged file sub/c.txt")
}
if !names[deletionsEntryName] {
t.Fatalf("differential archive missing deletions entry: %v", names)
}
// 链式恢复到全新目标
restoreRoot := t.TempDir()
restoreSrc := filepath.Join(restoreRoot, "src")
restoreTask := TaskSpec{Name: "diff", Type: "file", SourcePath: restoreSrc}
if err := runner.Restore(context.Background(), restoreTask, full.ArtifactPath, NopLogWriter{}); err != nil {
t.Fatalf("restore full: %v", err)
}
if err := runner.Restore(context.Background(), restoreTask, diff.ArtifactPath, NopLogWriter{}); err != nil {
t.Fatalf("restore differential: %v", err)
}
diffAssertContent(t, filepath.Join(restoreSrc, "a.txt"), "ALPHA-modified-and-longer")
diffAssertContent(t, filepath.Join(restoreSrc, "sub", "c.txt"), "charlie")
diffAssertContent(t, filepath.Join(restoreSrc, "d.txt"), "delta")
diffAssertAbsent(t, filepath.Join(restoreSrc, "b.txt"))
}
func TestPathSelected(t *testing.T) {
sel := []string{"src/a.txt", "src/sub"}
cases := map[string]bool{
"src/a.txt": true,
"src/sub": true,
"src/sub/c.txt": true, // 选中目录下的子项
"src/b.txt": false, // 未选中文件
"src/subother": false, // 前缀相近但非子项,不应误判
}
for name, want := range cases {
if got := pathSelected(name, sel); got != want {
t.Errorf("pathSelected(%q) = %v, want %v", name, got, want)
}
}
}
// TestFileRunnerSelectiveRestore 验证按需恢复:仅选中的文件与目录被还原,未选中的文件不出现。
func TestFileRunnerSelectiveRestore(t *testing.T) {
work := t.TempDir()
src := filepath.Join(work, "src")
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
diffWrite(t, filepath.Join(src, "b.txt"), "bravo")
diffWrite(t, filepath.Join(src, "sub", "c.txt"), "charlie")
runner := NewFileRunner()
full, err := runner.Run(context.Background(), TaskSpec{Name: "sel", Type: "file", SourcePath: src, TempDir: t.TempDir()}, NopLogWriter{})
if err != nil {
t.Fatalf("full Run: %v", err)
}
restoreRoot := t.TempDir()
restoreSrc := filepath.Join(restoreRoot, "src")
task := TaskSpec{Name: "sel", Type: "file", SourcePath: restoreSrc, SelectedPaths: []string{"src/a.txt", "src/sub"}}
if err := runner.Restore(context.Background(), task, full.ArtifactPath, NopLogWriter{}); err != nil {
t.Fatalf("selective Restore: %v", err)
}
diffAssertContent(t, filepath.Join(restoreSrc, "a.txt"), "alpha")
diffAssertContent(t, filepath.Join(restoreSrc, "sub", "c.txt"), "charlie") // 选中目录 → 子项一并恢复
diffAssertAbsent(t, filepath.Join(restoreSrc, "b.txt")) // 未选中 → 不恢复
}
// TestFileRunnerDifferentialWithoutBaseIsFull 验证无基线时差异请求回退为全量(产出清单、含全部文件)。
func TestFileRunnerDifferentialWithoutBaseIsFull(t *testing.T) {
src := filepath.Join(t.TempDir(), "src")
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
runner := NewFileRunner()
res, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir(), Differential: true}, NopLogWriter{})
if err != nil {
t.Fatalf("Run: %v", err)
}
if res.Manifest == nil {
t.Fatalf("differential without base must fall back to full and produce a manifest")
}
if names := diffArchiveNames(t, res.ArtifactPath); !names["src/a.txt"] || names[deletionsEntryName] {
t.Fatalf("fallback-full archive unexpected: %v", names)
}
}

View File

@@ -1,92 +0,0 @@
package backup
import (
"encoding/json"
"os"
"path/filepath"
"sort"
)
// deletionsEntryName 是差异备份归档中记录「自全量以来被删除路径」的特殊条目名。
// 恢复时该条目不落地为文件,而是用于在基线之上删除对应路径。
const deletionsEntryName = ".backupx/deletions.json"
// ManifestEntry 记录一次全量备份中单个归档条目(文件或目录)的指纹,
// 供差异备份比对「自全量以来的变化」。Path 为归档内相对名slash 分隔,
// 与 tar header.Name 一致)。字段使用短键以压缩清单体积。
type ManifestEntry struct {
Path string `json:"p"`
Size int64 `json:"s"`
ModTimeNs int64 `json:"m"`
Mode uint32 `json:"o"`
IsDir bool `json:"d,omitempty"`
}
// Manifest 是一次全量备份的完整条目清单(文件与目录)。
type Manifest struct {
Entries []ManifestEntry `json:"entries"`
}
// EncodeManifest 将清单序列化为紧凑 JSON。
func EncodeManifest(m Manifest) ([]byte, error) {
return json.Marshal(m)
}
// DecodeManifest 反序列化清单;空输入返回空清单(视为「无基线」)。
func DecodeManifest(data []byte) (Manifest, error) {
m := Manifest{}
if len(data) == 0 {
return m, nil
}
if err := json.Unmarshal(data, &m); err != nil {
return Manifest{}, err
}
return m, nil
}
// index 构建 path -> entry 映射,便于差异比对 O(1) 查找。
func (m Manifest) index() map[string]ManifestEntry {
idx := make(map[string]ManifestEntry, len(m.Entries))
for _, e := range m.Entries {
idx[e.Path] = e
}
return idx
}
// entryFromInfo 由归档名与文件信息构造指纹条目。
func entryFromInfo(archiveName string, info os.FileInfo) ManifestEntry {
return ManifestEntry{
Path: filepath.ToSlash(archiveName),
Size: info.Size(),
ModTimeNs: info.ModTime().UnixNano(),
Mode: uint32(info.Mode().Perm()),
IsDir: info.IsDir(),
}
}
// changedSince 判断当前条目相对基线是否为「新增或变更」(即应纳入差异归档)。
// - 不在基线中 → 新增,纳入;
// - 已存在的目录 → 不携带数据,跳过(其下变更文件会各自判定);
// - 文件大小或 mtime 变化 → 变更纳入rsync 风格启发式)。
func changedSince(base map[string]ManifestEntry, cur ManifestEntry) bool {
prev, ok := base[cur.Path]
if !ok {
return true
}
if cur.IsDir {
return false
}
return prev.Size != cur.Size || prev.ModTimeNs != cur.ModTimeNs
}
// deletedPaths 返回基线中存在、但本次遍历未出现的路径(被删除的条目),按路径升序。
func deletedPaths(base map[string]ManifestEntry, seen map[string]struct{}) []string {
deleted := make([]string, 0)
for p := range base {
if _, ok := seen[p]; !ok {
deleted = append(deleted, p)
}
}
sort.Strings(deleted)
return deleted
}

View File

@@ -1,79 +0,0 @@
package backup
import (
"reflect"
"testing"
)
func TestEncodeDecodeManifestRoundTrip(t *testing.T) {
m := Manifest{Entries: []ManifestEntry{
{Path: "src/a.txt", Size: 10, ModTimeNs: 100, Mode: 0o644},
{Path: "src", Size: 0, ModTimeNs: 50, Mode: 0o755, IsDir: true},
}}
data, err := EncodeManifest(m)
if err != nil {
t.Fatalf("EncodeManifest: %v", err)
}
got, err := DecodeManifest(data)
if err != nil {
t.Fatalf("DecodeManifest: %v", err)
}
if !reflect.DeepEqual(got, m) {
t.Fatalf("roundtrip mismatch:\n got %#v\nwant %#v", got, m)
}
}
func TestDecodeManifestEmpty(t *testing.T) {
got, err := DecodeManifest(nil)
if err != nil {
t.Fatalf("DecodeManifest(nil): %v", err)
}
if len(got.Entries) != 0 {
t.Fatalf("expected empty manifest, got %#v", got)
}
}
func TestChangedSince(t *testing.T) {
base := Manifest{Entries: []ManifestEntry{
{Path: "a.txt", Size: 10, ModTimeNs: 100},
{Path: "dir", IsDir: true, ModTimeNs: 100},
}}.index()
cases := []struct {
name string
cur ManifestEntry
want bool
}{
{"unchanged file", ManifestEntry{Path: "a.txt", Size: 10, ModTimeNs: 100}, false},
{"size changed", ManifestEntry{Path: "a.txt", Size: 11, ModTimeNs: 100}, true},
{"mtime changed", ManifestEntry{Path: "a.txt", Size: 10, ModTimeNs: 200}, true},
{"new file", ManifestEntry{Path: "b.txt", Size: 1, ModTimeNs: 1}, true},
{"existing dir skipped", ManifestEntry{Path: "dir", IsDir: true, ModTimeNs: 999}, false},
{"new dir included", ManifestEntry{Path: "newdir", IsDir: true, ModTimeNs: 1}, true},
}
for _, tc := range cases {
if got := changedSince(base, tc.cur); got != tc.want {
t.Errorf("%s: changedSince=%v want %v", tc.name, got, tc.want)
}
}
}
func TestDeletedPaths(t *testing.T) {
base := Manifest{Entries: []ManifestEntry{
{Path: "a"}, {Path: "b"}, {Path: "c"},
}}.index()
seen := map[string]struct{}{"a": {}, "c": {}}
got := deletedPaths(base, seen)
want := []string{"b"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("deletedPaths=%v want %v", got, want)
}
}
func TestDeletedPathsNoneWhenAllSeen(t *testing.T) {
base := Manifest{Entries: []ManifestEntry{{Path: "a"}, {Path: "b"}}}.index()
seen := map[string]struct{}{"a": {}, "b": {}}
if got := deletedPaths(base, seen); len(got) != 0 {
t.Fatalf("expected no deletions, got %v", got)
}
}

View File

@@ -1,119 +0,0 @@
package backup
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// MongoDBRunner 通过 mongodump/mongorestore 备份与恢复 MongoDB 数据库。
// 采用 --archive 流式模式dump 写 stdout、restore 读 stdin与 MySQLRunner
// 的 mysqldump/mysql 管线保持一致;产物为未压缩的 mongo archive由备份管线统一压缩/加密。
type MongoDBRunner struct {
executor CommandExecutor
}
func NewMongoDBRunner(executor CommandExecutor) *MongoDBRunner {
if executor == nil {
executor = NewOSCommandExecutor()
}
return &MongoDBRunner{executor: executor}
}
func (r *MongoDBRunner) Type() string {
return "mongodb"
}
func (r *MongoDBRunner) Run(ctx context.Context, task TaskSpec, writer LogWriter) (*RunResult, error) {
if _, err := r.executor.LookPath("mongodump"); err != nil {
return nil, fmt.Errorf("未找到 mongodump 命令 (请确保服务器已安装 mongodb-database-tools)")
}
startedAt := task.StartedAt
if startedAt.IsZero() {
startedAt = time.Now().UTC()
}
tempDir, err := CreateTaskTempDir(task.Name, startedAt)
if err != nil {
return nil, err
}
fileName := BuildArtifactName(task.Name, startedAt, "archive")
artifactPath := filepath.Join(tempDir, fileName)
file, err := os.Create(artifactPath)
if err != nil {
return nil, fmt.Errorf("create mongodump archive file: %w", err)
}
defer file.Close()
args := mongoConnArgs(task.Database)
dbNames := normalizeDatabaseNames(task.Database.Names)
if len(dbNames) == 1 {
args = append(args, "--db", dbNames[0])
writer.WriteLine(fmt.Sprintf("备份数据库: %s", dbNames[0]))
} else {
writer.WriteLine("备份全部数据库")
}
args = append(args, "--archive") // 归档流式写入 stdout
writer.WriteLine(fmt.Sprintf("连接到 MongoDB: %s:%d", task.Database.Host, task.Database.Port))
stderrWriter := newLogLineWriter(writer, "mongodump")
writer.WriteLine("开始执行 mongodump")
if err := r.executor.Run(ctx, "mongodump", args, CommandOptions{Stdout: file, Stderr: stderrWriter}); err != nil {
return nil, fmt.Errorf("run mongodump: %w: %s", err, stderrWriter.collected())
}
info, err := file.Stat()
if err != nil {
return nil, fmt.Errorf("stat mongodump archive: %w", err)
}
if info.Size() == 0 {
return nil, fmt.Errorf("mongodump 产物为空,请检查数据库连接与权限")
}
writer.WriteLine(fmt.Sprintf("MongoDB 导出完成(文件大小: %s", formatFileSize(info.Size())))
return &RunResult{ArtifactPath: artifactPath, FileName: fileName, TempDir: tempDir, Size: info.Size(), StorageKey: BuildStorageKey("mongodb", startedAt, fileName)}, nil
}
func (r *MongoDBRunner) Restore(ctx context.Context, task TaskSpec, artifactPath string, writer LogWriter) error {
if _, err := r.executor.LookPath("mongorestore"); err != nil {
return fmt.Errorf("未找到 mongorestore 命令 (请确保服务器已安装 mongodb-database-tools)")
}
input, err := os.Open(filepath.Clean(artifactPath))
if err != nil {
return fmt.Errorf("open mongodb restore archive: %w", err)
}
defer input.Close()
args := mongoConnArgs(task.Database)
// --drop恢复前删除同名集合保证恢复后与归档一致与 mysql 恢复的整库覆盖语义对齐)。
args = append(args, "--drop", "--archive")
stderr := &bytes.Buffer{}
writer.WriteLine("开始执行 mongorestore")
if err := r.executor.Run(ctx, "mongorestore", args, CommandOptions{Stdin: input, Stderr: stderr}); err != nil {
return fmt.Errorf("run mongorestore: %w: %s", err, strings.TrimSpace(stderr.String()))
}
writer.WriteLine("MongoDB 恢复完成")
return nil
}
// mongoConnArgs 构造 mongodump/mongorestore 的连接与认证参数。
// 注意mongodb-database-tools 无类似 MYSQL_PWD 的密码环境变量,密码只能经 --password 传入;
// 认证库默认 admin绝大多数部署的管理账号所在库
func mongoConnArgs(db DatabaseSpec) []string {
args := make([]string, 0, 8)
if strings.TrimSpace(db.Host) != "" {
args = append(args, "--host", db.Host)
}
if db.Port > 0 {
args = append(args, "--port", strconv.Itoa(db.Port))
}
if strings.TrimSpace(db.User) != "" {
args = append(args, "--username", db.User, "--authenticationDatabase", "admin")
if strings.TrimSpace(db.Password) != "" {
args = append(args, "--password", db.Password)
}
}
return args
}

View File

@@ -1,102 +0,0 @@
package backup
import (
"context"
"errors"
"io"
"os"
"strings"
"testing"
)
func argIndex(args []string, target string) int {
for i, a := range args {
if a == target {
return i
}
}
return -1
}
func TestMongoDBRunnerRunUsesMongodump(t *testing.T) {
executor := &fakeCommandExecutor{runFunc: func(name string, args []string, options CommandOptions) error {
if options.Stdout != nil {
_, _ = io.WriteString(options.Stdout, "mongo archive bytes")
}
return nil
}}
runner := NewMongoDBRunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, User: "admin", Password: "secret", Names: []string{"app"}}}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
if executor.lastName != "mongodump" {
t.Fatalf("expected mongodump, got %s", executor.lastName)
}
args := executor.lastArgs
if argIndex(args, "--archive") < 0 {
t.Fatalf("expected --archive flag, got %#v", args)
}
if i := argIndex(args, "--db"); i < 0 || i+1 >= len(args) || args[i+1] != "app" {
t.Fatalf("expected --db app, got %#v", args)
}
if i := argIndex(args, "--username"); i < 0 || args[i+1] != "admin" {
t.Fatalf("expected --username admin, got %#v", args)
}
if argIndex(args, "--authenticationDatabase") < 0 || argIndex(args, "--password") < 0 {
t.Fatalf("expected auth args, got %#v", args)
}
if _, err := os.Stat(result.ArtifactPath); err != nil {
t.Fatalf("artifact file missing: %v", err)
}
if result.StorageKey == "" || !strings.HasSuffix(result.FileName, ".archive") {
t.Fatalf("unexpected result metadata: %#v", result)
}
}
func TestMongoDBRunnerRunBackupsAllWhenNoDatabase(t *testing.T) {
executor := &fakeCommandExecutor{runFunc: func(name string, args []string, options CommandOptions) error {
_, _ = io.WriteString(options.Stdout, "all dbs")
return nil
}}
runner := NewMongoDBRunner(executor)
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017}}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
if argIndex(executor.lastArgs, "--db") >= 0 {
t.Fatalf("expected no --db when backing up all databases, got %#v", executor.lastArgs)
}
}
func TestMongoDBRunnerRunRejectsEmptyOutput(t *testing.T) {
executor := &fakeCommandExecutor{} // runFunc nil → writes nothing
runner := NewMongoDBRunner(executor)
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, Names: []string{"app"}}}, NopLogWriter{})
if err == nil {
t.Fatal("expected error for empty mongodump output")
}
}
func TestMongoDBRunnerRestoreUsesMongorestore(t *testing.T) {
executor := &fakeCommandExecutor{}
runner := NewMongoDBRunner(executor)
artifact := filepathJoinTempFile(t, "dump.archive", "mongo archive bytes")
if err := runner.Restore(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, User: "admin", Password: "secret"}}, artifact, NopLogWriter{}); err != nil {
t.Fatalf("Restore returned error: %v", err)
}
if executor.lastName != "mongorestore" {
t.Fatalf("expected mongorestore, got %s", executor.lastName)
}
if argIndex(executor.lastArgs, "--drop") < 0 || argIndex(executor.lastArgs, "--archive") < 0 {
t.Fatalf("expected --drop --archive, got %#v", executor.lastArgs)
}
}
func TestMongoDBRunnerRunReturnsLookupError(t *testing.T) {
runner := NewMongoDBRunner(&fakeCommandExecutor{lookupErr: errors.New("missing")})
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, Names: []string{"app"}}}, NopLogWriter{})
if err == nil {
t.Fatal("expected error when mongodump is missing")
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,131 +0,0 @@
package backup
import (
"context"
"fmt"
"io"
)
const (
repositoryChunkMin = 512 << 10
repositoryChunkAvg = 1 << 20
repositoryChunkMax = 4 << 20
)
// contentDefinedChunker implements the normalized FastCDC cut-point strategy.
// The rolling Gear hash only retains the latest 64 bytes through uint64
// overflow, so chunk boundaries re-synchronize after insertions or deletions.
type contentDefinedChunker struct {
minSize int
avgSize int
maxSize int
smallMask uint64
largeMask uint64
gear [256]uint64
}
func newContentDefinedChunker() *contentDefinedChunker {
chunker := &contentDefinedChunker{
minSize: repositoryChunkMin,
avgSize: repositoryChunkAvg,
maxSize: repositoryChunkMax,
smallMask: (1 << 21) - 1,
largeMask: (1 << 19) - 1,
}
// SplitMix64 produces a stable, well-distributed Gear table. The seed and
// generation algorithm are part of repository format v1 and must not change.
seed := uint64(0x6a09e667f3bcc909)
for i := range chunker.gear {
seed += 0x9e3779b97f4a7c15
value := seed
value = (value ^ (value >> 30)) * 0xbf58476d1ce4e5b9
value = (value ^ (value >> 27)) * 0x94d049bb133111eb
chunker.gear[i] = value ^ (value >> 31)
}
return chunker
}
func (c *contentDefinedChunker) Split(ctx context.Context, reader io.Reader, emit func([]byte) error) error {
if reader == nil || emit == nil {
return fmt.Errorf("chunk reader and emitter are required")
}
pending := make([]byte, 0, c.maxSize+(256<<10))
readBuffer := make([]byte, 256<<10)
eof := false
for {
if !eof {
readCount, readErr := reader.Read(readBuffer)
if readCount > 0 {
pending = append(pending, readBuffer[:readCount]...)
}
switch readErr {
case nil:
case io.EOF:
eof = true
default:
return fmt.Errorf("read source for chunking: %w", readErr)
}
if readCount == 0 && readErr == nil {
continue
}
}
for len(pending) > 0 {
if err := ctx.Err(); err != nil {
return err
}
cut := c.findCutPoint(pending, eof)
if cut == 0 {
break
}
chunk := make([]byte, cut)
copy(chunk, pending[:cut])
if err := emit(chunk); err != nil {
return err
}
copy(pending, pending[cut:])
pending = pending[:len(pending)-cut]
}
if eof {
if len(pending) != 0 {
return fmt.Errorf("chunker stopped with %d buffered bytes", len(pending))
}
return nil
}
}
}
func (c *contentDefinedChunker) findCutPoint(data []byte, eof bool) int {
if len(data) < c.minSize {
if eof {
return len(data)
}
return 0
}
limit := len(data)
if limit > c.maxSize {
limit = c.maxSize
}
var hash uint64
for index := c.minSize; index < limit; index++ {
hash = (hash << 1) + c.gear[data[index]]
mask := c.largeMask
if index < c.avgSize {
mask = c.smallMask
}
if hash&mask == 0 {
return index + 1
}
}
if len(data) >= c.maxSize {
return c.maxSize
}
if eof {
return len(data)
}
return 0
}

View File

@@ -1,414 +0,0 @@
package backup
import (
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"math/rand"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"testing"
"time"
"backupx/server/internal/storage"
)
func TestContentDefinedChunkerResynchronizesAfterInsertion(t *testing.T) {
source := make([]byte, 8<<20)
if _, err := rand.New(rand.NewSource(42)).Read(source); err != nil {
t.Fatalf("generate source: %v", err)
}
modified := make([]byte, 0, len(source)+4096)
modified = append(modified, source[:2<<20]...)
modified = append(modified, bytes.Repeat([]byte("inserted"), 512)...)
modified = append(modified, source[2<<20:]...)
chunker := newContentDefinedChunker()
collect := func(data []byte) map[string]struct{} {
t.Helper()
ids := make(map[string]struct{})
err := chunker.Split(context.Background(), bytes.NewReader(data), func(chunk []byte) error {
digest := sha256.Sum256(chunk)
ids[fmt.Sprintf("%x", digest[:])] = struct{}{}
if len(chunk) > repositoryChunkMax {
return fmt.Errorf("chunk exceeds maximum: %d", len(chunk))
}
return nil
})
if err != nil {
t.Fatalf("split chunks: %v", err)
}
return ids
}
originalChunks := collect(source)
modifiedChunks := collect(modified)
shared := 0
for chunkID := range originalChunks {
if _, ok := modifiedChunks[chunkID]; ok {
shared++
}
}
if shared < len(originalChunks)/2 {
t.Fatalf("content-defined boundaries did not resynchronize: shared=%d original=%d", shared, len(originalChunks))
}
}
func TestRepositoryRoundTripDedupAndPrune(t *testing.T) {
ctx := context.Background()
tempDir := t.TempDir()
sourceDir := filepath.Join(tempDir, "dataset")
if err := os.MkdirAll(filepath.Join(sourceDir, "empty"), 0o755); err != nil {
t.Fatalf("create source: %v", err)
}
original := make([]byte, 6<<20)
if _, err := rand.New(rand.NewSource(7)).Read(original); err != nil {
t.Fatalf("generate fixture: %v", err)
}
primaryPath := filepath.Join(sourceDir, "primary.bin")
duplicatePath := filepath.Join(sourceDir, "duplicate.bin")
if err := os.WriteFile(primaryPath, original, 0o640); err != nil {
t.Fatalf("write primary: %v", err)
}
if err := os.WriteFile(duplicatePath, original, 0o640); err != nil {
t.Fatalf("write duplicate: %v", err)
}
key := sha256.Sum256([]byte("repository-test-key"))
store := NewRepositoryStore(key[:])
provider := newMemoryRepositoryProvider()
task := TaskSpec{
ID: 12,
Name: "repository-test",
Type: "file",
SourcePaths: []string{sourceDir},
Compression: "zstd",
Encrypt: true,
StartedAt: time.Date(2026, 8, 6, 1, 2, 3, 0, time.UTC),
TempDir: tempDir,
}
firstPlan, err := store.BuildPlan(ctx, task, NopLogWriter{})
if err != nil {
t.Fatalf("build first plan: %v", err)
}
firstKey := store.SnapshotKey(task.ID, 1, task.StartedAt)
firstResult, err := store.Upload(ctx, provider, firstPlan, firstKey)
if closeErr := firstPlan.Close(); closeErr != nil {
t.Fatalf("close first plan: %v", closeErr)
}
if err != nil {
t.Fatalf("upload first snapshot: %v", err)
}
if firstResult.NewChunks == 0 || firstResult.UniqueChunks == 0 {
t.Fatalf("first upload did not create chunks: %+v", firstResult)
}
if firstPlan.UniqueSize >= firstPlan.LogicalSize {
t.Fatalf("duplicate file was not deduplicated within plan: unique=%d logical=%d", firstPlan.UniqueSize, firstPlan.LogicalSize)
}
modified := make([]byte, 0, len(original)+4096)
modified = append(modified, original[:2<<20]...)
modified = append(modified, bytes.Repeat([]byte("changed!"), 512)...)
modified = append(modified, original[2<<20:]...)
if err := os.WriteFile(primaryPath, modified, 0o640); err != nil {
t.Fatalf("modify primary: %v", err)
}
task.StartedAt = task.StartedAt.Add(time.Hour)
secondPlan, err := store.BuildPlan(ctx, task, NopLogWriter{})
if err != nil {
t.Fatalf("build second plan: %v", err)
}
secondKey := store.SnapshotKey(task.ID, 2, task.StartedAt)
secondResult, err := store.Upload(ctx, provider, secondPlan, secondKey)
if closeErr := secondPlan.Close(); closeErr != nil {
t.Fatalf("close second plan: %v", closeErr)
}
if err != nil {
t.Fatalf("upload second snapshot: %v", err)
}
if secondResult.ReusedBytes <= secondResult.LogicalSize/3 {
t.Fatalf("second snapshot reused too little data: %+v", secondResult)
}
if secondResult.UploadedBytes >= secondResult.LogicalSize {
t.Fatalf("incremental upload was not smaller than logical data: %+v", secondResult)
}
verify, err := store.Verify(ctx, provider, secondKey, secondResult.Checksum)
if err != nil {
t.Fatalf("verify repository: %v", err)
}
if verify.Chunks == 0 || verify.Bytes == 0 {
t.Fatalf("empty verification result: %+v", verify)
}
restoreRoot := filepath.Join(tempDir, "restore")
restoreTask := task
restoreTask.RestoreTargetPath = restoreRoot
if err := store.Restore(ctx, provider, secondKey, strings.Repeat("0", sha256.Size*2), restoreTask, NopLogWriter{}); err == nil {
t.Fatal("restore accepted a mismatched snapshot checksum")
}
if err := store.Restore(ctx, provider, secondKey, "", restoreTask, NopLogWriter{}); err == nil {
t.Fatal("restore accepted a missing snapshot checksum")
}
if err := store.Restore(ctx, provider, secondKey, secondResult.Checksum, restoreTask, NopLogWriter{}); err != nil {
t.Fatalf("restore snapshot: %v", err)
}
restored, err := os.ReadFile(filepath.Join(restoreRoot, filepath.Base(sourceDir), "primary.bin"))
if err != nil {
t.Fatalf("read restored primary: %v", err)
}
if !bytes.Equal(restored, modified) {
t.Fatalf("restored primary differs from source")
}
if info, err := os.Stat(filepath.Join(restoreRoot, filepath.Base(sourceDir), "empty")); err != nil || !info.IsDir() {
t.Fatalf("empty directory was not restored: info=%v err=%v", info, err)
}
if err := provider.Delete(ctx, firstKey); err != nil {
t.Fatalf("delete first snapshot: %v", err)
}
if _, err := store.Prune(ctx, provider); err != nil {
t.Fatalf("prune with live snapshot: %v", err)
}
if err := provider.Delete(ctx, secondKey); err != nil {
t.Fatalf("delete second snapshot: %v", err)
}
pruned, err := store.Prune(ctx, provider)
if err != nil {
t.Fatalf("prune empty repository: %v", err)
}
if pruned.DeletedPacks == 0 || pruned.DeletedIndexes == 0 {
t.Fatalf("prune did not reclaim repository data: %+v", pruned)
}
if objects, err := provider.List(ctx, repositoryPackPrefix); err != nil || len(objects) != 0 {
t.Fatalf("packs remain after prune: objects=%v err=%v", objects, err)
}
}
func TestRepositoryRestoreRejectsUnsafeSnapshotMetadata(t *testing.T) {
cases := []struct {
name string
entries []repositoryEntry
}{
{
name: "path traversal",
entries: []repositoryEntry{{Path: "../escape", Kind: "directory", Mode: 0o755}},
},
{
name: "entry below symlink",
entries: []repositoryEntry{
{Path: "link", Kind: "symlink", Mode: 0o777, LinkTarget: "inside"},
{Path: "link/payload", Kind: "file", Mode: 0o600, Size: 1, Chunks: []string{"p-" + strings.Repeat("0", sha256.Size*2)}},
},
},
{
name: "escaping symlink target",
entries: []repositoryEntry{{Path: "escape", Kind: "symlink", Mode: 0o777, LinkTarget: "../outside"}},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ctx := context.Background()
store := NewRepositoryStore(nil)
provider := newMemoryRepositoryProvider()
snapshot := repositorySnapshot{
Version: repositoryFormatVersion,
TaskID: 1,
CreatedAt: time.Now().UTC(),
Compression: "none",
Entries: tc.entries,
}
data, err := store.encodeSnapshot(snapshot)
if err != nil {
t.Fatalf("encodeSnapshot returned error: %v", err)
}
key := store.SnapshotKey(1, 1, snapshot.CreatedAt)
if err := provider.Upload(ctx, key, bytes.NewReader(data), int64(len(data)), nil); err != nil {
t.Fatalf("Upload snapshot returned error: %v", err)
}
digest := sha256.Sum256(data)
task := TaskSpec{SourcePath: filepath.Join(t.TempDir(), "source"), RestoreTargetPath: filepath.Join(t.TempDir(), "restore")}
if err := store.Restore(ctx, provider, key, fmt.Sprintf("%x", digest[:]), task, NopLogWriter{}); err == nil {
t.Fatal("restore accepted unsafe snapshot metadata")
}
})
}
}
func TestRepositoryRestorePreservesDirectoryWhenSnapshotContainsSymlink(t *testing.T) {
ctx := context.Background()
store := NewRepositoryStore(nil)
provider := newMemoryRepositoryProvider()
snapshot := repositorySnapshot{
Version: repositoryFormatVersion,
TaskID: 1,
CreatedAt: time.Now().UTC(),
Compression: "none",
Entries: []repositoryEntry{{Path: "link", Kind: "symlink", Mode: 0o777, LinkTarget: "inside"}},
}
data, err := store.encodeSnapshot(snapshot)
if err != nil {
t.Fatalf("encodeSnapshot returned error: %v", err)
}
key := store.SnapshotKey(1, 1, snapshot.CreatedAt)
if err := provider.Upload(ctx, key, bytes.NewReader(data), int64(len(data)), nil); err != nil {
t.Fatalf("Upload snapshot returned error: %v", err)
}
digest := sha256.Sum256(data)
restoreRoot := filepath.Join(t.TempDir(), "restore")
markerPath := filepath.Join(restoreRoot, "link", "keep.txt")
if err := os.MkdirAll(filepath.Dir(markerPath), 0o755); err != nil {
t.Fatalf("MkdirAll marker parent: %v", err)
}
if err := os.WriteFile(markerPath, []byte("keep"), 0o600); err != nil {
t.Fatalf("WriteFile marker: %v", err)
}
task := TaskSpec{SourcePath: filepath.Join(t.TempDir(), "source"), RestoreTargetPath: restoreRoot}
if err := store.Restore(ctx, provider, key, fmt.Sprintf("%x", digest[:]), task, NopLogWriter{}); err == nil {
t.Fatal("restore replaced an existing directory with a symlink")
}
if data, err := os.ReadFile(markerPath); err != nil || string(data) != "keep" {
t.Fatalf("existing directory content changed: data=%q err=%v", data, err)
}
}
func TestRepositoryRejectsOversizedChunkLocation(t *testing.T) {
ctx := context.Background()
store := NewRepositoryStore(nil)
provider := newMemoryRepositoryProvider()
chunkID := "p-" + strings.Repeat("0", sha256.Size*2)
packID := strings.Repeat("a", sha256.Size*2)
segment := repositoryIndexSegment{
Version: repositoryFormatVersion,
Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, packID[:2], packID),
Chunks: map[string]repositoryChunkLocation{
chunkID: {Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, packID[:2], packID), Offset: 0, Length: repositoryMaxEncoded + 1, PlainSize: 1, Compression: "none"},
},
}
data, err := json.Marshal(segment)
if err != nil {
t.Fatalf("Marshal index returned error: %v", err)
}
indexKey := fmt.Sprintf("%s/%s.json", repositoryIndexPrefix, packID)
if err := provider.Upload(ctx, indexKey, bytes.NewReader(data), int64(len(data)), nil); err != nil {
t.Fatalf("Upload index returned error: %v", err)
}
if _, err := store.loadIndex(ctx, provider); err == nil {
t.Fatal("loadIndex accepted an oversized encoded chunk")
}
}
func TestRepositoryRejectsIndexPackMismatch(t *testing.T) {
ctx := context.Background()
store := NewRepositoryStore(nil)
provider := newMemoryRepositoryProvider()
chunkID := "p-" + strings.Repeat("0", sha256.Size*2)
indexID := strings.Repeat("a", sha256.Size*2)
otherPackID := strings.Repeat("b", sha256.Size*2)
expectedPack := fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, indexID[:2], indexID)
segment := repositoryIndexSegment{
Version: repositoryFormatVersion,
Pack: expectedPack,
Chunks: map[string]repositoryChunkLocation{
chunkID: {
Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, otherPackID[:2], otherPackID),
Offset: 0,
Length: 1,
PlainSize: 1,
Compression: "none",
},
},
}
data, err := json.Marshal(segment)
if err != nil {
t.Fatalf("Marshal index returned error: %v", err)
}
indexKey := fmt.Sprintf("%s/%s.json", repositoryIndexPrefix, indexID)
if err := provider.Upload(ctx, indexKey, bytes.NewReader(data), int64(len(data)), nil); err != nil {
t.Fatalf("Upload index returned error: %v", err)
}
if _, err := store.loadIndex(ctx, provider); err == nil {
t.Fatal("loadIndex accepted a chunk location pointing to a different pack")
}
}
type memoryRepositoryProvider struct {
mu sync.RWMutex
objects map[string][]byte
times map[string]time.Time
}
func newMemoryRepositoryProvider() *memoryRepositoryProvider {
return &memoryRepositoryProvider{objects: make(map[string][]byte), times: make(map[string]time.Time)}
}
func (p *memoryRepositoryProvider) Type() storage.ProviderType { return "memory" }
func (p *memoryRepositoryProvider) TestConnection(context.Context) error { return nil }
func (p *memoryRepositoryProvider) Upload(_ context.Context, key string, reader io.Reader, size int64, _ map[string]string) error {
data, err := io.ReadAll(reader)
if err != nil {
return err
}
if int64(len(data)) != size {
return fmt.Errorf("size mismatch for %s: %d != %d", key, len(data), size)
}
p.mu.Lock()
defer p.mu.Unlock()
p.objects[key] = append([]byte(nil), data...)
p.times[key] = time.Now().UTC()
return nil
}
func (p *memoryRepositoryProvider) Download(_ context.Context, key string) (io.ReadCloser, error) {
p.mu.RLock()
defer p.mu.RUnlock()
data, ok := p.objects[key]
if !ok {
return nil, fmt.Errorf("object %s not found", key)
}
return io.NopCloser(bytes.NewReader(append([]byte(nil), data...))), nil
}
func (p *memoryRepositoryProvider) DownloadRange(_ context.Context, key string, offset, length int64) (io.ReadCloser, error) {
p.mu.RLock()
defer p.mu.RUnlock()
data, ok := p.objects[key]
if !ok {
return nil, fmt.Errorf("object %s not found", key)
}
if offset < 0 || length <= 0 || offset+length > int64(len(data)) {
return nil, fmt.Errorf("invalid range %d:%d for %s", offset, length, key)
}
return io.NopCloser(bytes.NewReader(append([]byte(nil), data[offset:offset+length]...))), nil
}
func (p *memoryRepositoryProvider) Delete(_ context.Context, key string) error {
p.mu.Lock()
defer p.mu.Unlock()
if _, ok := p.objects[key]; !ok {
return fmt.Errorf("object %s not found", key)
}
delete(p.objects, key)
delete(p.times, key)
return nil
}
func (p *memoryRepositoryProvider) List(_ context.Context, prefix string) ([]storage.ObjectInfo, error) {
p.mu.RLock()
defer p.mu.RUnlock()
result := make([]storage.ObjectInfo, 0)
for key, data := range p.objects {
if strings.HasPrefix(key, prefix) {
result = append(result, storage.ObjectInfo{Key: key, Size: int64(len(data)), UpdatedAt: p.times[key]})
}
}
sort.Slice(result, func(i, j int) bool { return result[i].Key < result[j].Key })
return result, nil
}

View File

@@ -1,55 +0,0 @@
package retention
import (
"testing"
"backupx/server/internal/model"
)
func retentionRecIDs(records []model.BackupRecord) []uint {
ids := make([]uint, 0, len(records))
for _, r := range records {
ids = append(ids, r.ID)
}
return ids
}
// 基线全量仍被「不在删除集合中的差异」依赖 → 必须保留,否则差异无法恢复。
func TestProtectDifferentialBasesKeepsBaseWithSurvivingDiff(t *testing.T) {
all := []model.BackupRecord{
{ID: 1, BackupKind: model.BackupKindFull},
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
}
candidates := []model.BackupRecord{{ID: 1, BackupKind: model.BackupKindFull}}
if got := protectDifferentialBases(all, candidates); len(got) != 0 {
t.Fatalf("base with surviving diff must be protected, got %v", retentionRecIDs(got))
}
}
// 基线全量与其全部差异都在删除集合中 → 可一并删除(无残留差异失去基线)。
func TestProtectDifferentialBasesDeletesBaseWhenDiffAlsoDeleted(t *testing.T) {
all := []model.BackupRecord{
{ID: 1, BackupKind: model.BackupKindFull},
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
}
candidates := []model.BackupRecord{
{ID: 1, BackupKind: model.BackupKindFull},
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
}
if got := protectDifferentialBases(all, candidates); len(got) != 2 {
t.Fatalf("base+diff both expired should both be deleted, got %v", retentionRecIDs(got))
}
}
// 无差异备份时原样透传(不影响既有全量保留逻辑)。
func TestProtectDifferentialBasesNoDiffsPassThrough(t *testing.T) {
all := []model.BackupRecord{
{ID: 1, BackupKind: model.BackupKindFull},
{ID: 2, BackupKind: model.BackupKindFull},
}
candidates := []model.BackupRecord{{ID: 1, BackupKind: model.BackupKindFull}}
got := protectDifferentialBases(all, candidates)
if len(got) != 1 || got[0].ID != 1 {
t.Fatalf("no diffs should pass through unchanged, got %v", retentionRecIDs(got))
}
}

View File

@@ -2,13 +2,10 @@ package retention
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"backupx/server/internal/backup"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
@@ -42,48 +39,16 @@ type CleanupResult struct {
Warnings []string
}
type cleanupObject struct {
targetID uint
path string
}
type storedUploadResult struct {
StorageTargetID uint `json:"storageTargetId"`
Status string `json:"status"`
StoragePath string `json:"storagePath"`
}
type Service struct {
records repository.BackupRecordRepository
repositoryKey []byte
now func() time.Time
records repository.BackupRecordRepository
now func() time.Time
}
func NewService(records repository.BackupRecordRepository, repositoryKey ...[]byte) *Service {
var key []byte
if len(repositoryKey) > 0 {
key = append([]byte(nil), repositoryKey[0]...)
}
return &Service{records: records, repositoryKey: key, now: func() time.Time { return time.Now().UTC() }}
func NewService(records repository.BackupRecordRepository) *Service {
return &Service{records: records, now: func() time.Time { return time.Now().UTC() }}
}
func (s *Service) Cleanup(ctx context.Context, task *model.BackupTask, provider storage.StorageProvider) (*CleanupResult, error) {
return s.cleanup(ctx, task, func(uint) (storage.StorageProvider, bool) {
return provider, provider != nil
})
}
// CleanupProviders applies one retention decision to every successful copy of
// a record before deleting its database row. This prevents multi-target tasks
// from leaving stale objects after the first target removes the shared record.
func (s *Service) CleanupProviders(ctx context.Context, task *model.BackupTask, providers map[uint]storage.StorageProvider) (*CleanupResult, error) {
return s.cleanup(ctx, task, func(targetID uint) (storage.StorageProvider, bool) {
provider, ok := providers[targetID]
return provider, ok && provider != nil
})
}
func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolveProvider func(uint) (storage.StorageProvider, bool)) (*CleanupResult, error) {
if task == nil {
return nil, fmt.Errorf("backup task is required")
}
@@ -91,45 +56,19 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
if err != nil {
return nil, fmt.Errorf("list successful records: %w", err)
}
var candidates []model.BackupRecord
if gfsEnabled(task) {
// GFS 策略:按天/周/月/年分层保留代表性备份,取代简单的天数/数量策略。
candidates = selectGFSToDelete(records, task.KeepDaily, task.KeepWeekly, task.KeepMonthly, task.KeepYearly)
} else {
candidates = selectRecordsToDelete(records, task.RetentionDays, task.MaxBackups, s.now())
}
// 差异链保护:保留仍被存活差异依赖的全量,避免删除基线后差异无法恢复。
candidates = protectDifferentialBases(records, candidates)
candidates := selectRecordsToDelete(records, task.RetentionDays, task.MaxBackups, s.now())
result := &CleanupResult{}
repositoryProviders := make(map[uint]storage.StorageProvider)
touchedProviders := make(map[uint]storage.StorageProvider)
for _, record := range candidates {
objects, objectErr := cleanupObjectsForRecord(record)
if objectErr != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("decode storage copies for record %d failed: %v", record.ID, objectErr))
continue
}
allObjectsDeleted := true
for _, object := range objects {
provider, ok := resolveProvider(object.targetID)
if !ok {
result.Warnings = append(result.Warnings, fmt.Sprintf("record %d missing storage provider %d for cleanup", record.ID, object.targetID))
allObjectsDeleted = false
if strings.TrimSpace(record.StoragePath) != "" {
if provider == nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("record %d missing storage provider for cleanup", record.ID))
continue
}
if err := provider.Delete(ctx, object.path); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("delete storage object %s from target %d failed: %v", object.path, object.targetID, err))
allObjectsDeleted = false
if err := provider.Delete(ctx, record.StoragePath); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("delete storage object %s failed: %v", record.StoragePath, err))
continue
}
result.DeletedObjects++
touchedProviders[object.targetID] = provider
if record.BackupKind == model.BackupKindRepository {
repositoryProviders[object.targetID] = provider
}
}
if !allObjectsDeleted {
continue
}
if err := s.records.Delete(ctx, record.ID); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("delete backup record %d failed: %v", record.ID, err))
@@ -137,25 +76,13 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
}
result.DeletedRecords++
}
for targetID, provider := range repositoryProviders {
pruned, pruneErr := backup.NewRepositoryStore(s.repositoryKey).Prune(ctx, provider)
if pruneErr != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("prune CDC repository on target %d failed: %v", targetID, pruneErr))
} else {
result.DeletedObjects += pruned.DeletedPacks + pruned.DeletedIndexes
}
}
// 清理空目录:收集被删除文件的父目录,尝试移除空目录
for targetID, provider := range touchedProviders {
dirCleaner, ok := provider.(storage.StorageDirCleaner)
if !ok {
continue
}
if dirCleaner, ok := provider.(storage.StorageDirCleaner); ok && result.DeletedObjects > 0 {
prefixes := collectDirPrefixes(candidates)
for _, prefix := range prefixes {
if err := dirCleaner.RemoveEmptyDirs(ctx, prefix); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("cleanup empty dirs for %s on target %d: %v", prefix, targetID, err))
result.Warnings = append(result.Warnings, fmt.Sprintf("cleanup empty dirs for %s: %v", prefix, err))
}
}
}
@@ -163,87 +90,7 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
return result, nil
}
func cleanupObjectsForRecord(record model.BackupRecord) ([]cleanupObject, error) {
defaultPath := strings.TrimSpace(record.StoragePath)
if strings.TrimSpace(record.StorageUploadResults) == "" {
if defaultPath == "" {
return nil, nil
}
return []cleanupObject{{targetID: record.StorageTargetID, path: defaultPath}}, nil
}
var results []storedUploadResult
if err := json.Unmarshal([]byte(record.StorageUploadResults), &results); err != nil {
return nil, err
}
objects := make([]cleanupObject, 0, len(results))
seen := make(map[uint]struct{}, len(results))
for _, result := range results {
if !strings.EqualFold(strings.TrimSpace(result.Status), model.BackupRecordStatusSuccess) {
continue
}
objectPath := strings.TrimSpace(result.StoragePath)
if objectPath == "" {
objectPath = defaultPath
}
if objectPath == "" {
continue
}
if _, exists := seen[result.StorageTargetID]; exists {
continue
}
seen[result.StorageTargetID] = struct{}{}
objects = append(objects, cleanupObject{targetID: result.StorageTargetID, path: objectPath})
}
if len(objects) == 0 && defaultPath != "" {
return nil, fmt.Errorf("successful record has no successful storage copy")
}
return objects, nil
}
// protectDifferentialBases 从删除候选中剔除「仍被存活差异依赖的全量」,
// 避免删除基线后其差异备份失去依据、无法恢复。全量仅当其全部差异都已过期/删除时才会被清理。
func protectDifferentialBases(all []model.BackupRecord, candidates []model.BackupRecord) []model.BackupRecord {
deleting := make(map[uint]struct{}, len(candidates))
for _, r := range candidates {
deleting[r.ID] = struct{}{}
}
protected := make(map[uint]struct{})
for _, r := range all {
if r.BackupKind != model.BackupKindDifferential || r.BaseRecordID == 0 {
continue
}
if _, beingDeleted := deleting[r.ID]; beingDeleted {
continue // 该差异本身也将被删除,无需保护其基线
}
protected[r.BaseRecordID] = struct{}{}
}
if len(protected) == 0 {
return candidates
}
filtered := make([]model.BackupRecord, 0, len(candidates))
for _, r := range candidates {
if r.BackupKind == model.BackupKindFull {
if _, keep := protected[r.ID]; keep {
continue
}
}
filtered = append(filtered, r)
}
return filtered
}
func selectRecordsToDelete(records []model.BackupRecord, retentionDays int, maxBackups int, now time.Time) []model.BackupRecord {
// 保留锁定(法律保留)的记录永不参与清理:先从候选集中剔除,
// 锁定备份既不被删除,也不占用 maxBackups 轮转名额。
if hasLocked(records) {
unlocked := make([]model.BackupRecord, 0, len(records))
for _, r := range records {
if !r.Locked {
unlocked = append(unlocked, r)
}
}
records = unlocked
}
selected := make(map[uint]model.BackupRecord)
if maxBackups > 0 && len(records) > maxBackups {
for _, record := range records[maxBackups:] {
@@ -266,81 +113,3 @@ func selectRecordsToDelete(records []model.BackupRecord, retentionDays int, maxB
}
return result
}
func hasLocked(records []model.BackupRecord) bool {
for i := range records {
if records[i].Locked {
return true
}
}
return false
}
// gfsEnabled 判定任务是否启用 GFS 分层保留(任一层级 > 0
func gfsEnabled(task *model.BackupTask) bool {
return task.KeepDaily > 0 || task.KeepWeekly > 0 || task.KeepMonthly > 0 || task.KeepYearly > 0
}
func recordTime(r *model.BackupRecord) time.Time {
if r.CompletedAt != nil {
return *r.CompletedAt
}
return r.StartedAt
}
func isoWeekKey(t time.Time) string {
y, w := t.ISOWeek()
return fmt.Sprintf("%d-W%02d", y, w)
}
// selectGFSToDelete 按 GFS祖父-父-子)策略选出应删除的记录。
//
// 规则:对每个层级(天/周/月/年),在按时间降序排列后,保留最近 keep 个不同周期中
// 每个周期最新的一份备份;各层级保留集合取并集即「保留集」,其余删除。
// 锁定(法律保留)的记录始终排除在删除候选之外。
func selectGFSToDelete(records []model.BackupRecord, daily, weekly, monthly, yearly int) []model.BackupRecord {
active := make([]model.BackupRecord, 0, len(records))
for i := range records {
if !records[i].Locked {
active = append(active, records[i])
}
}
sort.SliceStable(active, func(i, j int) bool {
return recordTime(&active[i]).After(recordTime(&active[j]))
})
keep := make(map[uint]bool, len(active))
keepTier := func(count int, key func(time.Time) string) {
if count <= 0 {
return
}
periods := 0
lastPeriod := ""
havePrev := false
for i := range active {
p := key(recordTime(&active[i]))
if havePrev && p == lastPeriod {
continue // 同周期已保留代表(最新一份)
}
if periods >= count {
break // 该层级已保留足够多的周期
}
keep[active[i].ID] = true
lastPeriod = p
havePrev = true
periods++
}
}
keepTier(daily, func(t time.Time) string { return t.Format("2006-01-02") })
keepTier(weekly, isoWeekKey)
keepTier(monthly, func(t time.Time) string { return t.Format("2006-01") })
keepTier(yearly, func(t time.Time) string { return t.Format("2006") })
del := make([]model.BackupRecord, 0)
for i := range active {
if !keep[active[i].ID] {
del = append(del, active[i])
}
}
return del
}

View File

@@ -24,9 +24,6 @@ func (r *fakeRecordRepository) List(context.Context, repository.BackupRecordList
func (r *fakeRecordRepository) FindByID(context.Context, uint) (*model.BackupRecord, error) {
return nil, nil
}
func (r *fakeRecordRepository) FindRunningByTaskAndNode(context.Context, uint, uint) (*model.BackupRecord, error) {
return nil, nil
}
func (r *fakeRecordRepository) Create(context.Context, *model.BackupRecord) error { return nil }
func (r *fakeRecordRepository) Update(context.Context, *model.BackupRecord) error { return nil }
func (r *fakeRecordRepository) Delete(_ context.Context, id uint) error {
@@ -45,9 +42,6 @@ func (r *fakeRecordRepository) ListByTask(_ context.Context, _ uint) ([]model.Ba
func (r *fakeRecordRepository) ListSuccessfulByTask(_ context.Context, _ uint) ([]model.BackupRecord, error) {
return r.records, nil
}
func (r *fakeRecordRepository) CountDependentDifferentials(context.Context, uint) (int64, error) {
return 0, nil
}
func (r *fakeRecordRepository) Count(context.Context) (int64, error) { return 0, nil }
func (r *fakeRecordRepository) CountSince(context.Context, time.Time) (int64, error) { return 0, nil }
func (r *fakeRecordRepository) CountSuccessSince(context.Context, time.Time) (int64, error) {
@@ -96,105 +90,6 @@ func TestSelectRecordsToDelete(t *testing.T) {
}
}
func gfsRecord(id uint, ts time.Time, locked bool) model.BackupRecord {
completed := ts
return model.BackupRecord{ID: id, StartedAt: ts, CompletedAt: &completed, Locked: locked}
}
func gfsDay(y, m, d, h int) time.Time {
return time.Date(y, time.Month(m), d, h, 0, 0, 0, time.UTC)
}
func deletedIDSet(records []model.BackupRecord) map[uint]bool {
out := make(map[uint]bool, len(records))
for i := range records {
out[records[i].ID] = true
}
return out
}
func assertDeleted(t *testing.T, del []model.BackupRecord, want ...uint) {
t.Helper()
got := deletedIDSet(del)
if len(got) != len(want) {
t.Fatalf("deleted set size = %d %v, want %d %v", len(got), got, len(want), want)
}
for _, id := range want {
if !got[id] {
t.Fatalf("expected id %d to be deleted; got %v", id, got)
}
}
}
// TestSelectGFSToDelete_DailyTier 验证按天分层:每天仅保留最新一份,且只保留最近 N 天。
func TestSelectGFSToDelete_DailyTier(t *testing.T) {
records := []model.BackupRecord{
gfsRecord(5, gfsDay(2026, 3, 7, 12), false), // 今天,最新 → 保留
gfsRecord(4, gfsDay(2026, 3, 7, 6), false), // 今天,较早 → 删除(非当天代表)
gfsRecord(3, gfsDay(2026, 3, 6, 12), false), // 昨天 → 保留
gfsRecord(2, gfsDay(2026, 3, 5, 12), false), // 前天 → 超出 daily=2 → 删除
gfsRecord(1, gfsDay(2026, 3, 4, 12), false), // 更早 → 删除
}
del := selectGFSToDelete(records, 2, 0, 0, 0)
assertDeleted(t, del, 4, 2, 1)
}
// TestSelectGFSToDelete_TierUnion 验证多层级取并集:月度层级保留日度层级会删除的旧备份。
func TestSelectGFSToDelete_TierUnion(t *testing.T) {
records := []model.BackupRecord{
gfsRecord(3, gfsDay(2026, 3, 7, 12), false), // 3 月(最新)
gfsRecord(2, gfsDay(2026, 2, 15, 12), false), // 2 月
gfsRecord(1, gfsDay(2026, 1, 15, 12), false), // 1 月
}
// daily=1 只留 ID3monthly=2 留最近两个月3 月=ID3、2 月=ID2。并集={3,2},删除 ID1。
del := selectGFSToDelete(records, 1, 0, 2, 0)
assertDeleted(t, del, 1)
}
// TestSelectGFSToDelete_SkipsLocked 验证锁定记录即使超出所有层级也永不删除。
func TestSelectGFSToDelete_SkipsLocked(t *testing.T) {
records := []model.BackupRecord{
gfsRecord(3, gfsDay(2026, 3, 7, 12), false),
gfsRecord(2, gfsDay(2026, 3, 6, 12), false),
gfsRecord(1, gfsDay(2020, 1, 1, 12), true), // 远超 daily=1 但已锁定 → 不删
}
del := selectGFSToDelete(records, 1, 0, 0, 0)
assertDeleted(t, del, 2) // 仅 ID2 被删ID1 锁定豁免ID3 为当日代表
}
func TestGFSEnabled(t *testing.T) {
if gfsEnabled(&model.BackupTask{}) {
t.Fatal("empty GFS config should be disabled")
}
if !gfsEnabled(&model.BackupTask{KeepWeekly: 4}) {
t.Fatal("KeepWeekly>0 should enable GFS")
}
}
// TestSelectRecordsToDelete_SkipsLocked 验证保留锁定(法律保留)的记录永不被选中删除,
// 即使它既超过保留期、又超过 maxBackups 名额。
func TestSelectRecordsToDelete_SkipsLocked(t *testing.T) {
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
completedNew := now.Add(-24 * time.Hour)
completedOld := now.Add(-15 * 24 * time.Hour)
records := []model.BackupRecord{
{ID: 3, CompletedAt: &completedNew},
{ID: 2, CompletedAt: &completedNew},
{ID: 1, CompletedAt: &completedOld, Locked: true}, // 超期但锁定 → 不应删除
}
selected := selectRecordsToDelete(records, 7, 2, now)
for _, r := range selected {
if r.ID == 1 {
t.Fatalf("locked record #1 must never be selected for deletion: %#v", selected)
}
}
// 锁定记录不占 maxBackups 名额:未锁定仅 2 条maxBackups=2 → 无超额删除,
// 且无未锁定记录超期 → 选中集为空。
if len(selected) != 0 {
t.Fatalf("expected no deletions (locked excluded), got %#v", selected)
}
}
func TestCleanupDeletesExpiredRecords(t *testing.T) {
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
completedNew := now.Add(-24 * time.Hour)
@@ -221,66 +116,3 @@ func TestCleanupDeletesExpiredRecords(t *testing.T) {
t.Fatalf("unexpected deleted objects: %#v", provider.deleted)
}
}
func TestCleanupProvidersDeletesEverySuccessfulCopyBeforeRecord(t *testing.T) {
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
completedNew := now.Add(-time.Hour)
completedOld := now.Add(-24 * time.Hour)
repo := &fakeRecordRepository{records: []model.BackupRecord{
{ID: 2, TaskID: 1, StoragePath: "records/2", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedNew},
{
ID: 1, TaskID: 1, StoragePath: "records/1", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedOld,
StorageUploadResults: `[{"storageTargetId":11,"status":"success","storagePath":"first/1"},{"storageTargetId":12,"status":"success","storagePath":"second/1"},{"storageTargetId":13,"status":"failed"}]`,
},
}}
first := &fakeProvider{}
second := &fakeProvider{}
service := NewService(repo)
service.now = func() time.Time { return now }
result, err := service.CleanupProviders(context.Background(), &model.BackupTask{ID: 1, MaxBackups: 1}, map[uint]storage.StorageProvider{
11: first,
12: second,
})
if err != nil {
t.Fatalf("CleanupProviders returned error: %v", err)
}
if result.DeletedRecords != 1 || result.DeletedObjects != 2 || len(result.Warnings) != 0 {
t.Fatalf("unexpected cleanup result: %#v", result)
}
if len(repo.deleted) != 1 || repo.deleted[0] != 1 {
t.Fatalf("unexpected deleted records: %#v", repo.deleted)
}
if len(first.deleted) != 1 || first.deleted[0] != "first/1" {
t.Fatalf("unexpected first-target deletes: %#v", first.deleted)
}
if len(second.deleted) != 1 || second.deleted[0] != "second/1" {
t.Fatalf("unexpected second-target deletes: %#v", second.deleted)
}
}
func TestCleanupProvidersKeepsRecordWhenCopyProviderIsUnavailable(t *testing.T) {
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
completedNew := now.Add(-time.Hour)
completedOld := now.Add(-24 * time.Hour)
repo := &fakeRecordRepository{records: []model.BackupRecord{
{ID: 2, TaskID: 1, StoragePath: "records/2", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedNew},
{
ID: 1, TaskID: 1, StoragePath: "records/1", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedOld,
StorageUploadResults: `[{"storageTargetId":11,"status":"success","storagePath":"first/1"},{"storageTargetId":12,"status":"success","storagePath":"second/1"}]`,
},
}}
first := &fakeProvider{}
service := NewService(repo)
result, err := service.CleanupProviders(context.Background(), &model.BackupTask{ID: 1, MaxBackups: 1}, map[uint]storage.StorageProvider{11: first})
if err != nil {
t.Fatalf("CleanupProviders returned error: %v", err)
}
if result.DeletedRecords != 0 || result.DeletedObjects != 1 || len(result.Warnings) != 1 {
t.Fatalf("unexpected safe partial-cleanup result: %#v", result)
}
if len(repo.deleted) != 0 {
t.Fatalf("record must remain until all copies are deleted: %#v", repo.deleted)
}
}

View File

@@ -55,17 +55,6 @@ func BuildStorageKey(backupType string, startedAt time.Time, fileName string) st
return filepath.ToSlash(filepath.Join("BackupX", typeName, startedAt.UTC().Format("060102"), fileName))
}
// BuildRecordStorageKey gives remote-Agent artifacts a record-owned namespace.
// The Master validates this namespace before accepting a relayed upload, so one
// Agent cannot overwrite another record's object on centrally mounted storage.
func BuildRecordStorageKey(backupType string, startedAt time.Time, recordID uint, fileName string) string {
typeName := strings.TrimSpace(strings.ToLower(backupType))
if typeName == "" {
typeName = "file"
}
return filepath.ToSlash(filepath.Join("BackupX", typeName, startedAt.UTC().Format("060102"), "records", fmt.Sprintf("%d", recordID), fileName))
}
func sanitizeTaskName(value string) string {
trimmed := strings.TrimSpace(strings.ToLower(value))
trimmed = strings.ReplaceAll(trimmed, " ", "-")

View File

@@ -36,15 +36,6 @@ type TaskSpec struct {
MaxBackups int
StartedAt time.Time
TempDir string
// Differential 为 true 时执行差异备份:仅打包自 BaseManifest 以来新增/变更的条目,
// 并记录被删除的路径。仅文件类型任务支持BaseManifest 为空时回退为全量。
Differential bool
BaseManifest Manifest
// SelectedPaths 非空时仅恢复这些归档相对路径(及其子项),用于按需(选择性)恢复;仅文件类型生效。
SelectedPaths []string
// RestoreTargetPath 仅用于恢复:非空时,文件类型恢复将归档解压到该目录,
// 而非默认的原始源路径父目录。用于「恢复到指定位置」(迁移/测试/并排恢复)。
RestoreTargetPath string
}
type RunResult struct {
@@ -53,8 +44,6 @@ type RunResult struct {
TempDir string
Size int64
StorageKey string
// Manifest 为全量备份产出的条目清单,供后续差异备份比对;差异备份运行时为 nil。
Manifest *Manifest
}
type LogEvent struct {
@@ -73,7 +62,7 @@ type ProgressInfo struct {
BytesSent int64 `json:"bytesSent"`
TotalBytes int64 `json:"totalBytes"`
Percent float64 `json:"percent"`
SpeedBps float64 `json:"speedBps"` // bytes/sec
SpeedBps float64 `json:"speedBps"` // bytes/sec
TargetName string `json:"targetName"`
}

View File

@@ -17,14 +17,9 @@ type Config struct {
}
type ServerConfig struct {
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
Mode string `mapstructure:"mode"`
ExternalURL string `mapstructure:"external_url"`
// WebRoot 指向前端构建产物目录。留空时后端会按部署惯例自动探测
// ./web、./web/dist、/opt/backupx/web 等)。探测命中后后端直接托管
// 前端 SPA无需额外的 nginx 反向代理即可访问 Web 控制台。
WebRoot string `mapstructure:"web_root"`
Host string `mapstructure:"host"`
Port int `mapstructure:"port"`
Mode string `mapstructure:"mode"`
}
type DatabaseConfig struct {
@@ -141,8 +136,6 @@ func applyDefaults(v *viper.Viper) {
v.SetDefault("server.host", "0.0.0.0")
v.SetDefault("server.port", 8340)
v.SetDefault("server.mode", "release")
v.SetDefault("server.external_url", "")
v.SetDefault("server.web_root", "")
v.SetDefault("database.path", "./data/backupx.db")
v.SetDefault("security.jwt_expire", "24h")
v.SetDefault("backup.temp_dir", "/tmp/backupx")

View File

@@ -1,10 +1,6 @@
package config
import (
"os"
"path/filepath"
"testing"
)
import "testing"
func TestLoadUsesDefaultsWithoutConfigFile(t *testing.T) {
cfg, err := Load("")
@@ -22,33 +18,3 @@ func TestLoadUsesDefaultsWithoutConfigFile(t *testing.T) {
t.Fatalf("expected default database path, got %s", cfg.Database.Path)
}
}
func TestLoadReadsServerExternalURLFromFile(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "config.yaml")
content := []byte("server:\n external_url: \"https://backup.example.com\"\n")
if err := os.WriteFile(configPath, content, 0o600); err != nil {
t.Fatalf("write config: %v", err)
}
cfg, err := Load(configPath)
if err != nil {
t.Fatalf("Load returned error: %v", err)
}
if cfg.Server.ExternalURL != "https://backup.example.com" {
t.Fatalf("expected external URL from config, got %q", cfg.Server.ExternalURL)
}
}
func TestLoadReadsServerExternalURLFromEnv(t *testing.T) {
t.Setenv("BACKUPX_SERVER_EXTERNAL_URL", "https://env-backup.example.com")
cfg, err := Load("")
if err != nil {
t.Fatalf("Load returned error: %v", err)
}
if cfg.Server.ExternalURL != "https://env-backup.example.com" {
t.Fatalf("expected external URL from env, got %q", cfg.Server.ExternalURL)
}
}

View File

@@ -156,44 +156,6 @@ func (h *AgentHandler) UpdateRecord(c *gin.Context) {
response.Success(c, gin.H{"status": "ok"})
}
// UploadArtifact streams a remote source artifact into storage mounted only on
// the Master. The request body is never buffered as a whole in memory or disk.
func (h *AgentHandler) UploadArtifact(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
recordID, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
targetID, err := strconv.ParseUint(c.Param("targetId"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
if c.Request.ContentLength < 0 {
c.JSON(stdhttp.StatusLengthRequired, gin.H{"code": "CONTENT_LENGTH_REQUIRED", "message": "artifact content length is required"})
return
}
if err := h.agentService.UploadArtifact(
c.Request.Context(),
node,
uint(recordID),
uint(targetID),
c.GetHeader("X-BackupX-Object-Key"),
c.Request.ContentLength,
c.GetHeader("X-BackupX-SHA256"),
c.Request.Body,
); err != nil {
response.Error(c, err)
return
}
response.Success(c, gin.H{"status": "ok"})
}
// GetRestoreSpec Agent 拉取恢复规格。
func (h *AgentHandler) GetRestoreSpec(c *gin.Context) {
if h.restoreService == nil {
@@ -246,34 +208,6 @@ func (h *AgentHandler) UpdateRestore(c *gin.Context) {
response.Success(c, gin.H{"status": "ok"})
}
// DownloadRestoreArtifact streams a Master-local backup back to its source
// Agent for restore without exposing the local storage configuration.
func (h *AgentHandler) DownloadRestoreArtifact(c *gin.Context) {
if h.restoreService == nil {
c.JSON(stdhttp.StatusServiceUnavailable, gin.H{"code": "RESTORE_SERVICE_DISABLED", "message": "restore service is not enabled"})
return
}
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
restoreID, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
artifact, err := h.restoreService.DownloadAgentArtifact(c.Request.Context(), node, uint(restoreID))
if err != nil {
response.Error(c, err)
return
}
c.DataFromReader(stdhttp.StatusOK, artifact.Size, "application/octet-stream", artifact.Reader, nil)
if err := artifact.Reader.Close(); err != nil {
_ = c.Error(err)
}
}
// Self 返回当前 Agent token 所属节点的状态,供安装脚本末尾探活。
func (h *AgentHandler) Self(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))

View File

@@ -52,20 +52,6 @@ func (h *BackupRecordHandler) Get(c *gin.Context) {
response.Success(c, item)
}
// Contents 返回备份记录的文件清单(内容浏览,只读)。
func (h *BackupRecordHandler) Contents(c *gin.Context) {
id, ok := parseUintParam(c, "id")
if !ok {
return
}
contents, err := h.service.ListContents(c.Request.Context(), id)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, contents)
}
func (h *BackupRecordHandler) StreamLogs(c *gin.Context) {
id, ok := parseUintParam(c, "id")
if !ok {
@@ -151,14 +137,7 @@ func (h *BackupRecordHandler) Restore(c *gin.Context) {
if subject, exists := c.Get(contextUserSubjectKey); exists {
triggeredBy = strings.TrimSpace(fmt.Sprintf("%v", subject))
}
// 可选请求体selectedPaths 按需选择性恢复targetPath 恢复到指定目录(仅文件类型本机恢复)。
// 无 body 时为整体恢复到原始路径。
var body struct {
SelectedPaths []string `json:"selectedPaths"`
TargetPath string `json:"targetPath"`
}
_ = c.ShouldBindJSON(&body)
detail, err := h.restoreService.StartSelective(c.Request.Context(), id, body.SelectedPaths, strings.TrimSpace(body.TargetPath), triggeredBy)
detail, err := h.restoreService.Start(c.Request.Context(), id, triggeredBy)
if err != nil {
response.Error(c, err)
return
@@ -182,32 +161,6 @@ func (h *BackupRecordHandler) Delete(c *gin.Context) {
response.Success(c, gin.H{"deleted": true})
}
// SetLock 设置/解除备份记录的保留锁定(法律保留)。
func (h *BackupRecordHandler) SetLock(c *gin.Context) {
id, ok := parseUintParam(c, "id")
if !ok {
return
}
var input struct {
Locked bool `json:"locked"`
}
if err := c.ShouldBindJSON(&input); err != nil {
response.Error(c, apperror.BadRequest("BACKUP_RECORD_LOCK_INVALID", "锁定参数不合法", err))
return
}
detail, err := h.service.SetLock(c.Request.Context(), id, input.Locked)
if err != nil {
response.Error(c, err)
return
}
action, desc := "unlock", fmt.Sprintf("解除备份记录保留锁定 (ID: %d)", id)
if input.Locked {
action, desc = "lock", fmt.Sprintf("设置备份记录保留锁定 (ID: %d)", id)
}
recordAudit(c, h.auditService, "backup_record", action, "backup_record", fmt.Sprintf("%d", id), "", desc)
response.Success(c, detail)
}
func (h *BackupRecordHandler) BatchDelete(c *gin.Context) {
var input struct {
IDs []uint `json:"ids" binding:"required,min=1"`
@@ -263,7 +216,7 @@ func writeSSEEvent(writer io.Writer, event backup.LogEvent) error {
}
func parseUintString(value string) (uint, bool) {
parsed, err := strconv.ParseUint(strings.TrimSpace(value), 10, 0)
parsed, err := strconv.ParseUint(strings.TrimSpace(value), 10, 64)
if err != nil {
return 0, false
}

View File

@@ -25,14 +25,10 @@ import (
// setupInstallFlowRouter 构造一个 Node + Agent + InstallToken 全量依赖的 router
// 并返回已登录管理员 JWT。
func setupInstallFlowRouter(t *testing.T) (http.Handler, string) {
return setupInstallFlowRouterWithExternalURL(t, "")
}
func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (http.Handler, string) {
t.Helper()
tempDir := t.TempDir()
cfg := config.Config{
Server: config.ServerConfig{Host: "127.0.0.1", Port: 8340, Mode: "test", ExternalURL: externalURL},
Server: config.ServerConfig{Host: "127.0.0.1", Port: 8340, Mode: "test"},
Database: config.DatabaseConfig{Path: filepath.Join(tempDir, "backupx.db")},
Security: config.SecurityConfig{JWTExpire: "24h"},
Log: config.LogConfig{Level: "error"},
@@ -72,6 +68,9 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
installTokenRepo := repository.NewAgentInstallTokenRepository(db)
installTokenSvc := service.NewInstallTokenService(installTokenRepo, nodeRepo)
auditLogRepo := repository.NewAuditLogRepository(db)
auditSvc := service.NewAuditService(auditLogRepo)
// 用 cancelable ctx测试结束时停掉 handler 启动的后台 GC 协程,
// 避免 goroutine 持有 map 导致 tempdir 清理失败。
ctx, cancel := context.WithCancel(context.Background())
@@ -86,7 +85,7 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
SystemService: systemSvc,
NodeService: nodeSvc,
InstallTokenService: installTokenSvc,
MasterExternalURL: cfg.Server.ExternalURL,
AuditService: auditSvc,
JWTManager: jwtMgr,
UserRepository: userRepo,
SystemConfigRepo: systemConfigRepo,
@@ -115,73 +114,6 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
return router, setupResp.Data.Token
}
func TestInstallTokenUsesConfiguredExternalURL(t *testing.T) {
const externalURL = "https://public.example.com/base"
router, jwt := setupInstallFlowRouterWithExternalURL(t, externalURL)
batchBody, _ := json.Marshal(map[string][]string{"names": {"external-url-node"}})
batchReq := httptest.NewRequest(http.MethodPost, "/api/nodes/batch", bytes.NewBuffer(batchBody))
batchReq.Header.Set("Content-Type", "application/json")
batchReq.Header.Set("Authorization", "Bearer "+jwt)
batchRec := httptest.NewRecorder()
router.ServeHTTP(batchRec, batchReq)
if batchRec.Code != 200 {
t.Fatalf("batch create failed: %d %s", batchRec.Code, batchRec.Body.String())
}
var batchResp struct {
Data []struct {
ID uint `json:"id"`
} `json:"data"`
}
if err := json.Unmarshal(batchRec.Body.Bytes(), &batchResp); err != nil {
t.Fatalf("unmarshal batch: %v", err)
}
if len(batchResp.Data) != 1 {
t.Fatalf("expected 1 node, got %d", len(batchResp.Data))
}
genBody, _ := json.Marshal(map[string]any{
"mode": "systemd",
"arch": "auto",
"agentVersion": "v1.7.0",
"downloadSrc": "github",
"ttlSeconds": 900,
})
genReq := httptest.NewRequest(http.MethodPost,
"/api/nodes/"+formatUint(batchResp.Data[0].ID)+"/install-tokens", bytes.NewBuffer(genBody))
genReq.Header.Set("Content-Type", "application/json")
genReq.Header.Set("Authorization", "Bearer "+jwt)
genRec := httptest.NewRecorder()
router.ServeHTTP(genRec, genReq)
if genRec.Code != 200 {
t.Fatalf("install-tokens failed: %d %s", genRec.Code, genRec.Body.String())
}
var genResp struct {
Data struct {
InstallToken string `json:"installToken"`
URL string `json:"url"`
FallbackURL string `json:"fallbackUrl"`
ScriptBase64 string `json:"scriptBase64"`
} `json:"data"`
}
if err := json.Unmarshal(genRec.Body.Bytes(), &genResp); err != nil {
t.Fatalf("unmarshal gen: %v", err)
}
if genResp.Data.URL != externalURL+"/api/install/"+genResp.Data.InstallToken {
t.Fatalf("url should use external URL, got %q", genResp.Data.URL)
}
if genResp.Data.FallbackURL != externalURL+"/install/"+genResp.Data.InstallToken {
t.Fatalf("fallbackUrl should use external URL, got %q", genResp.Data.FallbackURL)
}
decodedScript, err := base64.StdEncoding.DecodeString(genResp.Data.ScriptBase64)
if err != nil {
t.Fatalf("scriptBase64 should be valid base64: %v", err)
}
if !strings.Contains(string(decodedScript), `MASTER_URL="`+externalURL+`"`) {
t.Fatalf("script should use external MASTER_URL:\n%s", string(decodedScript))
}
}
func TestOneClickInstallFlow(t *testing.T) {
router, jwt := setupInstallFlowRouter(t)
@@ -496,76 +428,6 @@ func TestInstallFlowComposeModeMismatch(t *testing.T) {
}
}
func TestInstallFlowComposeSuccessConsumesToken(t *testing.T) {
router, jwt := setupInstallFlowRouter(t)
batchBody, _ := json.Marshal(map[string][]string{"names": {"compose-ok"}})
batchReq := httptest.NewRequest(http.MethodPost, "/api/nodes/batch", bytes.NewBuffer(batchBody))
batchReq.Header.Set("Content-Type", "application/json")
batchReq.Header.Set("Authorization", "Bearer "+jwt)
batchRec := httptest.NewRecorder()
router.ServeHTTP(batchRec, batchReq)
if batchRec.Code != 200 {
t.Fatalf("batch create failed: %d %s", batchRec.Code, batchRec.Body.String())
}
var batchResp struct {
Data []struct {
ID uint `json:"id"`
} `json:"data"`
}
if err := json.Unmarshal(batchRec.Body.Bytes(), &batchResp); err != nil {
t.Fatalf("unmarshal batch: %v", err)
}
if len(batchResp.Data) != 1 {
t.Fatalf("expected 1 node, got %d", len(batchResp.Data))
}
genBody, _ := json.Marshal(map[string]any{
"mode": "docker",
"arch": "auto",
"agentVersion": "v1.7.0",
"downloadSrc": "github",
"ttlSeconds": 900,
})
genReq := httptest.NewRequest(http.MethodPost,
"/api/nodes/"+formatUint(batchResp.Data[0].ID)+"/install-tokens", bytes.NewBuffer(genBody))
genReq.Header.Set("Content-Type", "application/json")
genReq.Header.Set("Authorization", "Bearer "+jwt)
genRec := httptest.NewRecorder()
router.ServeHTTP(genRec, genReq)
if genRec.Code != 200 {
t.Fatalf("install-tokens failed: %d %s", genRec.Code, genRec.Body.String())
}
var genResp struct {
Data struct {
InstallToken string `json:"installToken"`
} `json:"data"`
}
if err := json.Unmarshal(genRec.Body.Bytes(), &genResp); err != nil {
t.Fatalf("unmarshal gen: %v", err)
}
if genResp.Data.InstallToken == "" {
t.Fatalf("missing installToken")
}
composeReq := httptest.NewRequest(http.MethodGet, "/api/install/"+genResp.Data.InstallToken+"/compose.yml", nil)
composeRec := httptest.NewRecorder()
router.ServeHTTP(composeRec, composeReq)
if composeRec.Code != 200 {
t.Fatalf("compose fetch failed: %d %s", composeRec.Code, composeRec.Body.String())
}
if !strings.Contains(composeRec.Body.String(), "BACKUPX_AGENT_TOKEN") {
t.Fatalf("compose missing token env:\n%s", composeRec.Body.String())
}
scriptReq := httptest.NewRequest(http.MethodGet, "/api/install/"+genResp.Data.InstallToken, nil)
scriptRec := httptest.NewRecorder()
router.ServeHTTP(scriptRec, scriptReq)
if scriptRec.Code != http.StatusGone {
t.Fatalf("script after compose should be 410, got %d: %s", scriptRec.Code, scriptRec.Body.String())
}
}
// formatUint 小工具uint → 十进制字符串(无需引入 strconv
func formatUint(u uint) string {
if u == 0 {

View File

@@ -1,6 +1,7 @@
package http
import (
"encoding/base64"
"fmt"
stdhttp "net/http"
"strconv"
@@ -244,17 +245,14 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
input.TTLSeconds = 900
}
out, err := h.installTokenSvc.CreateCommand(c.Request.Context(), service.InstallCommandInput{
InstallTokenInput: service.InstallTokenInput{
NodeID: uint(id),
Mode: input.Mode,
Arch: input.Arch,
AgentVersion: input.AgentVersion,
DownloadSrc: input.DownloadSrc,
TTLSeconds: input.TTLSeconds,
CreatedByID: h.resolveCurrentUserID(c),
},
MasterURL: resolveMasterURL(c, h.externalURL),
out, err := h.installTokenSvc.Create(c.Request.Context(), service.InstallTokenInput{
NodeID: uint(id),
Mode: input.Mode,
Arch: input.Arch,
AgentVersion: input.AgentVersion,
DownloadSrc: input.DownloadSrc,
TTLSeconds: input.TTLSeconds,
CreatedByID: h.resolveCurrentUserID(c),
})
if err != nil {
response.Error(c, err)
@@ -264,6 +262,12 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
fmt.Sprintf("%d", id), out.Node.Name,
fmt.Sprintf("生成 %s/%s install token TTL=%ds", input.Mode, input.Arch, input.TTLSeconds))
masterURL := resolveMasterURL(c, h.externalURL)
script, err := renderInstallScript(masterURL, out.Node, out.Record)
if err != nil {
response.Error(c, err)
return
}
// 使用 /api/install/... 而非 /install/... —— 让反向代理的 /api/ 转发规则
// 自动接管,避免 SPA fallback 把请求当成前端路由返回 index.htmlissue #46
// 同时返回 /install/... 备用地址,兼容会剥离 /api 前缀的外层反向代理。
@@ -272,11 +276,15 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
body := gin.H{
"installToken": out.Token,
"expiresAt": out.ExpiresAt,
"url": out.URL,
"fallbackUrl": out.FallbackURL,
"scriptBase64": out.ScriptBase64,
"composeUrl": out.ComposeURL,
"fallbackComposeUrl": out.FallbackComposeURL,
"url": masterURL + "/api/install/" + out.Token,
"fallbackUrl": masterURL + "/install/" + out.Token,
"scriptBase64": base64.StdEncoding.EncodeToString([]byte(script)),
"composeUrl": "",
"fallbackComposeUrl": "",
}
if input.Mode == "docker" {
body["composeUrl"] = masterURL + "/api/install/" + out.Token + "/compose.yml"
body["fallbackComposeUrl"] = masterURL + "/install/" + out.Token + "/compose.yml"
}
response.Success(c, body)
}

View File

@@ -1,95 +0,0 @@
package http
import (
"encoding/csv"
"fmt"
"strconv"
"strings"
"time"
"backupx/server/internal/service"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
type ReportHandler struct {
service *service.ReportService
}
func NewReportHandler(reportService *service.ReportService) *ReportHandler {
return &ReportHandler{service: reportService}
}
func reportDays(c *gin.Context) int {
days := 30
if v := strings.TrimSpace(c.Query("days")); v != "" {
if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
days = parsed
}
}
return days
}
// Compliance 返回 JSON 合规报表(按任务的备份合规证据 + 汇总)。
func (h *ReportHandler) Compliance(c *gin.Context) {
payload, err := h.service.ComplianceReport(c.Request.Context(), reportDays(c))
if err != nil {
response.Error(c, err)
return
}
response.Success(c, payload)
}
// ComplianceCSV 把合规报表导出为 CSV供审计归档。带 UTF-8 BOM 以便 Excel 正确识别中文。
func (h *ReportHandler) ComplianceCSV(c *gin.Context) {
report, err := h.service.ComplianceReport(c.Request.Context(), reportDays(c))
if err != nil {
response.Error(c, err)
return
}
filename := fmt.Sprintf("backupx-compliance-%s.csv", report.GeneratedAt.Format("20060102-150405"))
c.Header("Content-Type", "text/csv; charset=utf-8")
c.Header("Content-Disposition", "attachment; filename="+filename)
_, _ = c.Writer.WriteString("\ufeff") // UTF-8 BOM
w := csv.NewWriter(c.Writer)
_ = w.Write([]string{
"任务ID", "任务名", "类型", "启用", "节点", "加密", "保留天数", "SLA(RPO小时)",
"周期内运行", "成功", "失败", "成功率", "最近状态", "最近运行(UTC)", "最近成功(UTC)", "保护字节数", "合规判定",
})
for _, row := range report.Tasks {
_ = w.Write([]string{
strconv.FormatUint(uint64(row.TaskID), 10),
row.TaskName,
row.Type,
boolCN(row.Enabled),
row.NodeName,
boolCN(row.Encrypted),
strconv.Itoa(row.RetentionDays),
strconv.Itoa(row.SLAHoursRPO),
strconv.Itoa(row.TotalRuns),
strconv.Itoa(row.Successes),
strconv.Itoa(row.Failures),
fmt.Sprintf("%.2f%%", row.SuccessRate*100),
row.LastStatus,
fmtTimePtr(row.LastRunAt),
fmtTimePtr(row.LastSuccessAt),
strconv.FormatInt(row.ProtectedBytes, 10),
row.Risk,
})
}
w.Flush()
}
func boolCN(b bool) string {
if b {
return "是"
}
return "否"
}
func fmtTimePtr(t *time.Time) string {
if t == nil {
return ""
}
return t.UTC().Format("2006-01-02 15:04:05")
}

View File

@@ -41,7 +41,6 @@ type RouterDependencies struct {
ApiKeyService *service.ApiKeyService
NotificationService *service.NotificationService
DashboardService *service.DashboardService
ReportService *service.ReportService
SettingsService *service.SettingsService
NodeService *service.NodeService
AgentService *service.AgentService
@@ -167,11 +166,9 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
backupRecords.GET("/:id", backupRecordHandler.Get)
backupRecords.GET("/:id/logs/stream", backupRecordHandler.StreamLogs)
backupRecords.GET("/:id/download", backupRecordHandler.Download)
backupRecords.GET("/:id/contents", backupRecordHandler.Contents)
backupRecords.POST("/:id/restore", RequireNotViewer(), backupRecordHandler.Restore)
backupRecords.POST("/batch-delete", RequireNotViewer(), backupRecordHandler.BatchDelete)
backupRecords.DELETE("/:id", RequireNotViewer(), backupRecordHandler.Delete)
backupRecords.PUT("/:id/lock", RequireNotViewer(), backupRecordHandler.SetLock)
// 恢复记录独立命名空间:列表/详情/SSE 日志流。
// 创建恢复仍然走 POST /backup/records/:id/restore以源备份记录为触发点
@@ -214,15 +211,6 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
// 基于备份记录的验证入口:与 restore 对称
backupRecords.POST("/:id/verify", RequireNotViewer(), verificationHandler.TriggerByRecord)
}
// 企业合规报表:按任务的可导出备份合规证据(区别于 Dashboard 的实时聚合视图)。
if deps.ReportService != nil {
reportHandler := NewReportHandler(deps.ReportService)
reports := api.Group("/reports")
reports.Use(AuthMiddleware(deps.JWTManager, apiKeyAuth))
reports.GET("/compliance", reportHandler.Compliance)
reports.GET("/compliance/export", reportHandler.ComplianceCSV)
}
dashboard := api.Group("/dashboard")
dashboard.Use(AuthMiddleware(deps.JWTManager, apiKeyAuth))
dashboard.GET("/stats", dashboardHandler.Stats)
@@ -304,10 +292,7 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
nodes.POST("", RequireRole("admin"), nodeHandler.Create)
nodes.PUT("/:id", RequireRole("admin"), nodeHandler.Update)
nodes.DELETE("/:id", RequireRole("admin"), nodeHandler.Delete)
// 文件浏览会枚举节点文件系统目录(含 /etc、/root 等),属敏感读操作:
// 限制为非 vieweradmin/operator与"创建备份任务需选源路径"的权限对齐,
// 避免只读 viewer 借此探查服务器目录结构。
nodes.GET("/:id/fs/list", RequireNotViewer(), nodeHandler.ListDirectory)
nodes.GET("/:id/fs/list", nodeHandler.ListDirectory)
nodes.POST("/batch", RequireRole("admin"), nodeHandler.BatchCreate)
nodes.POST("/:id/install-tokens", RequireRole("admin"), nodeHandler.CreateInstallToken)
nodes.POST("/:id/rotate-token", RequireRole("admin"), nodeHandler.RotateToken)
@@ -322,9 +307,7 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
agent.POST("/commands/:id/result", agentHandler.SubmitCommandResult)
agent.GET("/tasks/:id", agentHandler.GetTaskSpec)
agent.POST("/records/:id", agentHandler.UpdateRecord)
agent.PUT("/records/:id/artifacts/:targetId", agentHandler.UploadArtifact)
agent.GET("/restores/:id/spec", agentHandler.GetRestoreSpec)
agent.GET("/restores/:id/artifact", agentHandler.DownloadRestoreArtifact)
agent.POST("/restores/:id", agentHandler.UpdateRestore)
// Agent v1安装脚本探活用仅 Self 端点
@@ -370,25 +353,9 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
engine.GET("/api/install/:token/compose.yml", installHandler.Compose)
}
// 未匹配路由处理:
// - 找到前端产物目录时,托管 SPA静态文件 + index.html 回退),
// 使后端在无 nginx 反向代理时也能直接提供 Web 控制台issue #62
// - 未找到前端目录时退化为纯 API 服务,统一返回结构化 JSON 404。
apiNotFound := func(c *gin.Context) {
engine.NoRoute(func(c *gin.Context) {
response.Error(c, apperror.New(stdhttp.StatusNotFound, "NOT_FOUND", "接口不存在", errors.New("route not found")))
}
if webRoot := resolveWebRoot(deps.Config.Server.WebRoot); webRoot != "" {
if deps.Logger != nil {
deps.Logger.Info("serving web frontend", zap.String("web_root", webRoot))
}
engine.NoRoute(spaFileServer(webRoot, apiNotFound))
} else {
if deps.Logger != nil {
deps.Logger.Warn("web frontend directory not found; serving API only",
zap.String("hint", "set server.web_root in config, or place the built frontend at ./web"))
}
engine.NoRoute(apiNotFound)
}
})
return engine
}

View File

@@ -1,86 +0,0 @@
package http
import (
stdhttp "net/http"
"os"
"path/filepath"
"strings"
"github.com/gin-gonic/gin"
)
// resolveWebRoot 返回前端静态资源目录。优先使用显式配置的路径,
// 否则按部署惯例依次探测常见位置,返回首个包含 index.html 的目录。
// 返回空字符串表示未找到前端产物,此时后端退化为纯 API 服务。
func resolveWebRoot(configured string) string {
candidates := []string{
configured,
"./web/dist", // 源码树根目录构建产物(优先于 ./web避免命中前端源码模板
"./web", // systemdWorkingDirectory=/opt/backupx → /opt/backupx/web容器 WORKDIR=/app → /app/web
"../web/dist", // 从 server/ 目录运行make dev-server
"/opt/backupx/web",
"/app/web",
}
for _, dir := range candidates {
if strings.TrimSpace(dir) == "" {
continue
}
if hasIndexHTML(dir) {
if abs, err := filepath.Abs(dir); err == nil {
return abs
}
return dir
}
}
return ""
}
func hasIndexHTML(dir string) bool {
info, err := os.Stat(filepath.Join(dir, "index.html"))
return err == nil && !info.IsDir()
}
// isReservedBackendPath 判断请求是否命中后端保留前缀API、探针、安装脚本
// 这些路径即使未匹配到具体路由,也应返回结构化 JSON 404而不是回退到
// 前端 index.html —— 否则反向代理/安装脚本会把 HTML 当成接口响应(参考 issue #46
func isReservedBackendPath(p string) bool {
switch p {
case "/health", "/ready", "/metrics", "/api", "/install":
return true
}
return strings.HasPrefix(p, "/api/") || strings.HasPrefix(p, "/install/")
}
// spaFileServer 构造 SPA 静态资源处理器,用作 gin 的 NoRoute 回退:
// - 后端保留前缀返回 apiNotFoundJSON 404
// - 其余 GET/HEAD 请求若在 webRoot 内命中真实文件则直接返回该文件;
// - 未命中文件的路径回退到 index.html交由前端路由处理history 模式刷新)。
func spaFileServer(webRoot string, apiNotFound gin.HandlerFunc) gin.HandlerFunc {
indexPath := filepath.Join(webRoot, "index.html")
return func(c *gin.Context) {
reqPath := c.Request.URL.Path
if isReservedBackendPath(reqPath) {
apiNotFound(c)
return
}
if c.Request.Method != stdhttp.MethodGet && c.Request.Method != stdhttp.MethodHead {
apiNotFound(c)
return
}
// 防目录穿越:以 webRoot 为根清理路径,确保最终目标仍位于 webRoot 内。
clean := filepath.Clean("/" + strings.TrimPrefix(reqPath, "/"))
target := filepath.Join(webRoot, clean)
if rel, err := filepath.Rel(webRoot, target); err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
apiNotFound(c)
return
}
if info, err := os.Stat(target); err == nil && !info.IsDir() {
c.File(target)
return
}
// 前端 SPA 路由(/dashboard、/tasks 等)回退到 index.html。
c.File(indexPath)
}
}

View File

@@ -1,175 +0,0 @@
package http
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/gin-gonic/gin"
)
func TestResolveWebRoot(t *testing.T) {
gin.SetMode(gin.TestMode)
t.Run("explicit configured dir with index.html", func(t *testing.T) {
dir := t.TempDir()
writeIndex(t, dir)
got := resolveWebRoot(dir)
abs, _ := filepath.Abs(dir)
if got != abs {
t.Fatalf("resolveWebRoot(%q) = %q, want %q", dir, got, abs)
}
})
t.Run("configured dir without index.html falls through to none", func(t *testing.T) {
dir := t.TempDir() // no index.html, and no conventional ./web in CWD during test
if got := resolveWebRoot(dir); got != "" {
// 允许 CWD 恰好存在约定目录的环境,但临时目录本身不应被选中。
abs, _ := filepath.Abs(dir)
if got == abs {
t.Fatalf("expected dir without index.html to be skipped, got %q", got)
}
}
})
t.Run("empty configured uses auto-detect order", func(t *testing.T) {
// 切到一个仅含 ./web/dist/index.html 的临时工作目录,验证自动探测。
root := t.TempDir()
distDir := filepath.Join(root, "web", "dist")
if err := os.MkdirAll(distDir, 0o755); err != nil {
t.Fatal(err)
}
writeIndex(t, distDir)
restore := chdir(t, root)
defer restore()
// 以 chdir 之后的实际工作目录为基准计算期望值,避免 macOS 上
// /var → /private/var 符号链接导致字符串不一致。
wd, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
want := filepath.Join(wd, "web", "dist")
got := resolveWebRoot("")
if got != want {
t.Fatalf("auto-detect = %q, want %q", got, want)
}
})
}
func TestIsReservedBackendPath(t *testing.T) {
reserved := []string{"/health", "/ready", "/metrics", "/api", "/install", "/api/", "/api/system/info", "/install/abc", "/install/abc/compose.yml"}
for _, p := range reserved {
if !isReservedBackendPath(p) {
t.Errorf("isReservedBackendPath(%q) = false, want true", p)
}
}
notReserved := []string{"/", "/dashboard", "/assets/app.js", "/installer", "/apidocs", "/favicon.ico"}
for _, p := range notReserved {
if isReservedBackendPath(p) {
t.Errorf("isReservedBackendPath(%q) = true, want false", p)
}
}
}
func TestSpaFileServer(t *testing.T) {
gin.SetMode(gin.TestMode)
webRoot := t.TempDir()
writeIndexContent(t, webRoot, "<!doctype html><title>BackupX</title>")
assetsDir := filepath.Join(webRoot, "assets")
if err := os.MkdirAll(assetsDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(assetsDir, "app.js"), []byte("console.log(1)"), 0o644); err != nil {
t.Fatal(err)
}
apiNotFoundHit := false
apiNotFound := func(c *gin.Context) {
apiNotFoundHit = true
c.JSON(http.StatusNotFound, gin.H{"code": "NOT_FOUND"})
}
engine := gin.New()
engine.NoRoute(spaFileServer(webRoot, apiNotFound))
cases := []struct {
name string
method string
path string
wantStatus int
wantBody string // 子串;为空表示不校验
wantAPI404 bool
}{
{name: "root serves index", method: http.MethodGet, path: "/", wantStatus: 200, wantBody: "BackupX"},
{name: "spa route falls back to index", method: http.MethodGet, path: "/dashboard", wantStatus: 200, wantBody: "BackupX"},
{name: "real asset served", method: http.MethodGet, path: "/assets/app.js", wantStatus: 200, wantBody: "console.log"},
{name: "api path returns json 404", method: http.MethodGet, path: "/api/garbage", wantStatus: 404, wantAPI404: true},
{name: "health returns json 404 via reserved", method: http.MethodGet, path: "/health", wantStatus: 404, wantAPI404: true},
{name: "non-GET on spa path is api 404", method: http.MethodPost, path: "/dashboard", wantStatus: 404, wantAPI404: true},
{name: "directory falls back to index", method: http.MethodGet, path: "/assets/", wantStatus: 200, wantBody: "BackupX"},
// 含 ".." 的请求路径被 net/http 在文件服务层直接拒绝400 invalid URL path
// 绝不会泄露 webRoot 之外的文件;这是在 filepath.Rel 校验之上的纵深防御。
{name: "traversal rejected, never serves passwd", method: http.MethodGet, path: "/../../etc/passwd", wantStatus: 400, wantBody: "invalid URL path"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
apiNotFoundHit = false
req := httptest.NewRequest(tc.method, tc.path, nil)
rec := httptest.NewRecorder()
engine.ServeHTTP(rec, req)
if rec.Code != tc.wantStatus {
t.Fatalf("status = %d, want %d (body=%q)", rec.Code, tc.wantStatus, rec.Body.String())
}
if tc.wantBody != "" && !contains(rec.Body.String(), tc.wantBody) {
t.Fatalf("body %q does not contain %q", rec.Body.String(), tc.wantBody)
}
if tc.wantAPI404 != apiNotFoundHit {
t.Fatalf("apiNotFoundHit = %v, want %v", apiNotFoundHit, tc.wantAPI404)
}
})
}
}
func writeIndex(t *testing.T, dir string) {
t.Helper()
writeIndexContent(t, dir, "<!doctype html><title>BackupX</title>")
}
func writeIndexContent(t *testing.T, dir, content string) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, "index.html"), []byte(content), 0o644); err != nil {
t.Fatal(err)
}
}
func chdir(t *testing.T, dir string) func() {
t.Helper()
orig, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
if err := os.Chdir(dir); err != nil {
t.Fatal(err)
}
return func() { _ = os.Chdir(orig) }
}
func contains(s, sub string) bool {
return len(sub) == 0 || (len(s) >= len(sub) && indexOf(s, sub) >= 0)
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}

View File

@@ -199,7 +199,7 @@ func (h *StorageTargetHandler) GoogleDriveProfile(c *gin.Context) {
func parseUintParam(c *gin.Context, key string) (uint, bool) {
value := strings.TrimSpace(c.Param(key))
parsed, err := strconv.ParseUint(value, 10, 0)
parsed, err := strconv.ParseUint(value, 10, 64)
if err != nil {
response.Error(c, apperror.BadRequest("INVALID_ID", fmt.Sprintf("参数 %s 不合法", key), err))
return 0, false

View File

@@ -1,61 +0,0 @@
package installscript
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestDeployInstallScriptSyntax(t *testing.T) {
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
cmd := exec.Command("sh", "-n", scriptPath)
output, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("install.sh syntax invalid: %v\n%s", err, output)
}
}
func TestDeployInstallScriptSupportsReleasePackageLayout(t *testing.T) {
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
data, err := os.ReadFile(scriptPath)
if err != nil {
t.Fatal(err)
}
script := string(data)
for _, want := range []string{
`SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)`,
`if [ -f "$SCRIPT_DIR/backupx" ] && [ -d "$SCRIPT_DIR/web" ]; then`,
`BIN_SOURCE="${BIN_SOURCE:-$SCRIPT_DIR/backupx}"`,
`WEB_SOURCE="${WEB_SOURCE:-$SCRIPT_DIR/web}"`,
`CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$SCRIPT_DIR/config.example.yaml}"`,
`发布包安装请确认当前目录包含 ./backupx、./web 和 ./install.sh。`,
`cat > "/etc/systemd/system/$SERVICE_NAME.service" <<UNIT`,
`if [ -d "/etc/nginx/conf.d" ] && [ -f "$NGINX_SOURCE" ]; then`,
} {
if !strings.Contains(script, want) {
t.Fatalf("install.sh missing %q", want)
}
}
}
func TestDeployInstallScriptSupportsSourceBuildAndVerifiesFirstSetup(t *testing.T) {
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
data, err := os.ReadFile(scriptPath)
if err != nil {
t.Fatal(err)
}
script := string(data)
for _, want := range []string{
`SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/bin/backupx"`,
`For a source install, run 'make build' in the repository root first.`,
`HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8340/api/auth/setup/status}"`,
`systemctl is-active --quiet "$SERVICE_NAME"`,
`System setup`,
} {
if !strings.Contains(script, want) {
t.Fatalf("install.sh missing %q", want)
}
}
}

View File

@@ -37,22 +37,19 @@ func TestRenderScriptBashBootstrap(t *testing.T) {
}
}
func TestRenderScriptUsesRootForBareMetalBackups(t *testing.T) {
func TestRenderScriptCreatesBackupXUserAndGroup(t *testing.T) {
got, err := RenderScript(testCtx)
if err != nil {
t.Fatalf("render err: %v", err)
}
for _, want := range []string{
"/var/lib/backupx-agent/tmp",
"install -d -m 0700 /var/lib/backupx-agent /var/lib/backupx-agent/tmp",
"getent group backupx",
"groupadd --system backupx",
"useradd --system --gid backupx",
"Group=backupx",
} {
if !strings.Contains(got, want) {
t.Errorf("script missing %q:\n%s", want, got)
}
}
for _, forbidden := range []string{"User=backupx", "Group=backupx", "NoNewPrivileges=true"} {
if strings.Contains(got, forbidden) {
t.Errorf("script should not contain %q for bare-metal backups:\n%s", forbidden, got)
}
}
}

View File

@@ -1,8 +1,6 @@
package installscript
import (
"os"
"path/filepath"
"strings"
"testing"
@@ -29,10 +27,8 @@ func TestRenderScriptSystemd(t *testing.T) {
mustContain := []string{
"BACKUPX_AGENT_MASTER=${MASTER_URL}",
`Environment="BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}"`,
"/var/lib/backupx-agent/tmp",
"systemctl daemon-reload",
"systemctl enable --now backupx-agent",
"systemctl status backupx-agent",
"X-Agent-Token: ${AGENT_TOKEN}",
"MASTER_URL=\"https://master.example.com\"",
"AGENT_TOKEN=\"deadbeefcafebabe0123456789abcdef0123456789abcdef0123456789abcdef\"",
@@ -60,9 +56,6 @@ func TestRenderScriptForeground(t *testing.T) {
if !strings.Contains(got, `exec "${INSTALL_PREFIX}/backupx" agent`) {
t.Errorf("foreground script missing exec line:\n%s", got)
}
if !strings.Contains(got, "/var/lib/backupx-agent/tmp") {
t.Errorf("foreground script missing dedicated temp dir:\n%s", got)
}
if strings.Contains(got, "systemctl daemon-reload") {
t.Errorf("foreground script should not reference systemctl:\n%s", got)
}
@@ -81,44 +74,14 @@ func TestRenderScriptDocker(t *testing.T) {
if !strings.Contains(got, "docker run") {
t.Errorf("docker script missing `docker run`:\n%s", got)
}
if !strings.Contains(got, "/var/lib/backupx-agent:/var/lib/backupx-agent") {
t.Errorf("docker script missing agent data volume:\n%s", got)
}
if !strings.Contains(got, "awuqing/backupx:${AGENT_VERSION}") {
t.Errorf("docker script missing image tag reference:\n%s", got)
}
if !strings.Contains(got, `"awuqing/backupx:${AGENT_VERSION}" agent`) {
t.Errorf("docker script must start image in agent mode:\n%s", got)
}
if !strings.Contains(got, `-e "BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp"`) {
t.Errorf("docker script missing temp dir env:\n%s", got)
}
if !strings.Contains(got, `docker logs --tail=100 backupx-agent`) {
t.Errorf("docker script missing diagnostic log command:\n%s", got)
}
if !strings.Contains(got, `grep -q '"status":"online"'`) {
t.Errorf("docker script missing online probe:\n%s", got)
}
if strings.Contains(got, "systemctl daemon-reload") {
t.Errorf("docker script should not reference systemctl:\n%s", got)
}
}
func TestDockerEntrypointForwardsAgentSubcommand(t *testing.T) {
entrypointPath := filepath.Join("..", "..", "..", "deploy", "docker", "entrypoint.sh")
got, err := os.ReadFile(entrypointPath)
if err != nil {
t.Fatalf("read docker entrypoint: %v", err)
}
script := string(got)
if !strings.Contains(script, `"${1:-}" = "agent"`) {
t.Fatalf("entrypoint must detect the agent subcommand before starting server:\n%s", script)
}
if !strings.Contains(script, `exec /app/bin/backupx "$@"`) {
t.Fatalf("entrypoint must exec backupx with forwarded args:\n%s", script)
}
}
func TestRenderComposeYaml(t *testing.T) {
ctx := testCtx
ctx.Mode = model.InstallModeDocker
@@ -129,26 +92,17 @@ func TestRenderComposeYaml(t *testing.T) {
if !strings.Contains(got, "image: awuqing/backupx:v1.7.0") {
t.Errorf("compose missing image:\n%s", got)
}
if !strings.Contains(got, `command: ["agent"]`) {
t.Errorf("compose must start image in agent mode:\n%s", got)
}
if !strings.Contains(got, `BACKUPX_AGENT_TOKEN: "deadbeefcafebabe0123456789abcdef0123456789abcdef0123456789abcdef"`) {
t.Errorf("compose missing token env:\n%s", got)
}
if !strings.Contains(got, `BACKUPX_AGENT_TEMP_DIR: "/var/lib/backupx-agent/tmp"`) {
t.Errorf("compose missing temp dir env:\n%s", got)
}
if !strings.Contains(got, "/var/lib/backupx-agent:/var/lib/backupx-agent") {
t.Errorf("compose missing agent data volume:\n%s", got)
}
}
func TestRenderScriptRejectsInjectedMasterURL(t *testing.T) {
bad := []string{
"https://example.com\" other: inject", // 含引号和空格
"javascript:alert(1)", // scheme 非法
"https://example.com\n- privileged", // 含换行YAML 注入经典 payload
"", // 空
"javascript:alert(1)", // scheme 非法
"https://example.com\n- privileged", // 含换行YAML 注入经典 payload
"", // 空
}
for _, u := range bad {
ctx := testCtx
@@ -207,8 +161,8 @@ func TestDownloadBaseMapping(t *testing.T) {
func TestRenderScriptDefaultsApplied(t *testing.T) {
ctx := testCtx
ctx.InstallPrefix = "" // 应被默认为 /opt/backupx-agent
ctx.DownloadBase = "" // 应被默认为 github
ctx.InstallPrefix = "" // 应被默认为 /opt/backupx-agent
ctx.DownloadBase = "" // 应被默认为 github
got, err := RenderScript(ctx)
if err != nil {
t.Fatalf("render err: %v", err)

View File

@@ -9,6 +9,5 @@ services:
environment:
BACKUPX_AGENT_MASTER: "{{.MasterURL}}"
BACKUPX_AGENT_TOKEN: "{{.AgentToken}}"
BACKUPX_AGENT_TEMP_DIR: "/var/lib/backupx-agent/tmp"
volumes:
- /var/lib/backupx-agent:/var/lib/backupx-agent
- /var/lib/backupx-agent:/tmp/backupx-agent

View File

@@ -47,10 +47,30 @@ else
fi
tar xzf "$TMPDIR/pkg.tar.gz" -C "$TMPDIR"
# 4. 安装二进制 + 数据目录
# 4. 安装二进制 + 用户
echo "[2/4] 安装到 ${INSTALL_PREFIX}"
install -d -m 0755 "$INSTALL_PREFIX"
install -d -m 0700 /var/lib/backupx-agent /var/lib/backupx-agent/tmp
if ! getent group backupx >/dev/null 2>&1; then
if command -v groupadd >/dev/null 2>&1; then
groupadd --system backupx
elif command -v addgroup >/dev/null 2>&1; then
addgroup --system backupx
else
echo "需要 groupadd 或 addgroup 来创建 backupx 组" >&2
exit 1
fi
fi
if ! id backupx >/dev/null 2>&1; then
if command -v useradd >/dev/null 2>&1; then
useradd --system --gid backupx --home-dir "$INSTALL_PREFIX" --shell /usr/sbin/nologin backupx
elif command -v adduser >/dev/null 2>&1; then
adduser --system --ingroup backupx --home "$INSTALL_PREFIX" --shell /usr/sbin/nologin backupx
else
echo "需要 useradd 或 adduser 来创建 backupx 用户" >&2
exit 1
fi
fi
id backupx >/dev/null 2>&1 || { echo "backupx 用户创建失败" >&2; exit 1; }
install -d -o backupx -g backupx "$INSTALL_PREFIX" /var/lib/backupx-agent
install -m 0755 "$TMPDIR/backupx-${AGENT_VERSION}-linux-${ARCH}/backupx" "$INSTALL_PREFIX/backupx"
{{end}}
@@ -65,13 +85,14 @@ Wants=network-online.target
[Service]
Type=simple
User=backupx
Group=backupx
Environment="BACKUPX_AGENT_MASTER=${MASTER_URL}"
Environment="BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}"
ExecStart=${INSTALL_PREFIX}/backupx agent --temp-dir /var/lib/backupx-agent/tmp
ExecStart=${INSTALL_PREFIX}/backupx agent --temp-dir /var/lib/backupx-agent
Restart=on-failure
RestartSec=10s
# Agent 需以 root 运行以读取任意源数据;与单机服务端保持一致的资源/句柄上限。
LimitNOFILE=65535
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
@@ -90,7 +111,6 @@ for i in $(seq 1 15); do
fi
done
echo "⚠ 30s 内未收到上线心跳,请检查防火墙或 journalctl -u backupx-agent"
echo "提示systemd 服务名是 backupx-agent可执行 systemctl status backupx-agent 查看状态。"
exit 2
{{end}}
@@ -99,7 +119,7 @@ exit 2
echo "[3/3] 前台启动 agentCtrl+C 退出)"
export BACKUPX_AGENT_MASTER="${MASTER_URL}"
export BACKUPX_AGENT_TOKEN="${AGENT_TOKEN}"
exec "${INSTALL_PREFIX}/backupx" agent --temp-dir /var/lib/backupx-agent/tmp
exec "${INSTALL_PREFIX}/backupx" agent --temp-dir /var/lib/backupx-agent
{{end}}
{{if eq .Mode "docker"}}
@@ -111,20 +131,7 @@ docker rm -f backupx-agent >/dev/null 2>&1 || true
docker run -d --name backupx-agent --restart=unless-stopped \
-e "BACKUPX_AGENT_MASTER=${MASTER_URL}" \
-e "BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}" \
-e "BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp" \
-v /var/lib/backupx-agent:/var/lib/backupx-agent \
-v /var/lib/backupx-agent:/tmp/backupx-agent \
"awuqing/backupx:${AGENT_VERSION}" agent
echo "✓ 容器已启动,等待节点上线"
for i in $(seq 1 15); do
sleep 2
if curl -fsSL -H "X-Agent-Token: ${AGENT_TOKEN}" "${MASTER_URL}/api/v1/agent/self" 2>/dev/null \
| grep -q '"status":"online"'; then
echo "✓ 节点已上线"
exit 0
fi
done
echo "⚠ 30s 内未收到上线心跳,请检查容器状态、网络与 Master URL。"
echo "排查命令docker ps -a --filter name=backupx-agent"
echo "排查命令docker logs --tail=100 backupx-agent"
exit 2
echo "✓ 容器已启动"
{{end}}

View File

@@ -13,18 +13,16 @@ type SampleSource interface {
ListStorageTargets(ctx context.Context) ([]model.StorageTarget, error)
StorageUsage(ctx context.Context) ([]repository.BackupStorageUsageItem, error)
ListNodes(ctx context.Context) ([]model.Node, error)
AgentQueueSummaries(ctx context.Context) (map[uint]repository.AgentCommandQueueSummary, error)
CountSLABreach(ctx context.Context) (int, error)
}
// repoSource 把 repository 适配到 SampleSource。
type repoSource struct {
targets repository.StorageTargetRepository
records repository.BackupRecordRepository
nodes repository.NodeRepository
tasks repository.BackupTaskRepository
commands repository.AgentCommandRepository
now func() time.Time
targets repository.StorageTargetRepository
records repository.BackupRecordRepository
nodes repository.NodeRepository
tasks repository.BackupTaskRepository
now func() time.Time
}
// NewRepoSource 用仓储实例构造 SampleSource。
@@ -33,15 +31,13 @@ func NewRepoSource(
records repository.BackupRecordRepository,
nodes repository.NodeRepository,
tasks repository.BackupTaskRepository,
commands repository.AgentCommandRepository,
) SampleSource {
return &repoSource{
targets: targets,
records: records,
nodes: nodes,
tasks: tasks,
commands: commands,
now: func() time.Time { return time.Now().UTC() },
targets: targets,
records: records,
nodes: nodes,
tasks: tasks,
now: func() time.Time { return time.Now().UTC() },
}
}
@@ -57,13 +53,6 @@ func (s *repoSource) ListNodes(ctx context.Context) ([]model.Node, error) {
return s.nodes.List(ctx)
}
func (s *repoSource) AgentQueueSummaries(ctx context.Context) (map[uint]repository.AgentCommandQueueSummary, error) {
if s.commands == nil {
return nil, nil
}
return s.commands.NodeQueueSummaries(ctx)
}
// CountSLABreach 统计当前违反 RPO 的任务:
// - 任务启用且配置了 SLAHoursRPO > 0
// - 最近一次成功备份距今超出 SLA 时间窗,或从未成功过
@@ -147,9 +136,7 @@ func (c *Collector) collect(ctx context.Context) {
}
// 节点在线状态role 约定为 master / agent
if nodes, err := c.source.ListNodes(ctx); err == nil {
queueByNode, _ := c.source.AgentQueueSummaries(ctx)
c.metrics.ResetNodeOnline()
c.metrics.ResetAgentQueue()
for i := range nodes {
n := &nodes[i]
role := "agent"
@@ -157,8 +144,6 @@ func (c *Collector) collect(ctx context.Context) {
role = "master"
}
c.metrics.SetNodeOnline(n.Name, role, n.Status == model.NodeStatusOnline)
queue := queueByNode[n.ID]
c.metrics.SetAgentQueue(n.Name, role, queue.Depth, queue.Running, queue.Timeouts)
}
}
if breach, err := c.source.CountSLABreach(ctx); err == nil {

View File

@@ -31,12 +31,6 @@ type Metrics struct {
StorageUsedBytes *prometheus.GaugeVec
// 节点在线状态labels: node_name, rolevalue: 0/1
NodeOnline *prometheus.GaugeVec
// Agent 命令队列深度labels: node_name, role
AgentCommandQueueDepth *prometheus.GaugeVec
// Agent 正在执行的长命令数labels: node_name, role
AgentCommandRunning *prometheus.GaugeVec
// Agent 命令超时累计数快照labels: node_name, role
AgentCommandTimeoutTotal *prometheus.GaugeVec
// 验证演练结果labels: status
VerifyRunTotal *prometheus.CounterVec
// 恢复操作结果labels: status
@@ -84,18 +78,6 @@ func New(version string) *Metrics {
Name: "backupx_node_online",
Help: "集群节点在线状态1 在线 / 0 离线)",
}, []string{"node_name", "role"}),
AgentCommandQueueDepth: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "backupx_agent_command_queue_depth",
Help: "Agent 当前 pending/dispatched 命令总数",
}, []string{"node_name", "role"}),
AgentCommandRunning: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "backupx_agent_command_running",
Help: "Agent 当前正在执行的长命令数",
}, []string{"node_name", "role"}),
AgentCommandTimeoutTotal: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "backupx_agent_command_timeout_total",
Help: "Agent 已超时命令数快照",
}, []string{"node_name", "role"}),
VerifyRunTotal: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "backupx_verify_run_total",
Help: "备份验证演练执行总数",
@@ -124,9 +106,6 @@ func New(version string) *Metrics {
m.TaskRunningGauge,
m.StorageUsedBytes,
m.NodeOnline,
m.AgentCommandQueueDepth,
m.AgentCommandRunning,
m.AgentCommandTimeoutTotal,
m.VerifyRunTotal,
m.RestoreRunTotal,
m.ReplicationRunTotal,
@@ -229,24 +208,6 @@ func (m *Metrics) ResetNodeOnline() {
m.NodeOnline.Reset()
}
func (m *Metrics) SetAgentQueue(name, role string, depth, running, timeoutCount int) {
if m == nil {
return
}
m.AgentCommandQueueDepth.WithLabelValues(name, role).Set(float64(depth))
m.AgentCommandRunning.WithLabelValues(name, role).Set(float64(running))
m.AgentCommandTimeoutTotal.WithLabelValues(name, role).Set(float64(timeoutCount))
}
func (m *Metrics) ResetAgentQueue() {
if m == nil {
return
}
m.AgentCommandQueueDepth.Reset()
m.AgentCommandRunning.Reset()
m.AgentCommandTimeoutTotal.Reset()
}
// ResetStorageUsed 清空存储目标 gauge。
func (m *Metrics) ResetStorageUsed() {
if m == nil {

View File

@@ -41,11 +41,9 @@ func TestObserveTaskRun_NilReceiverIsSafe(t *testing.T) {
m.DecTaskRunning()
m.SetStorageUsed("a", "s3", 1)
m.SetNodeOnline("n1", "master", true)
m.SetAgentQueue("n1", "agent", 2, 1, 3)
m.SetSLABreach(3)
m.ResetNodeOnline()
m.ResetStorageUsed()
m.ResetAgentQueue()
// no panic -> pass
}
@@ -53,7 +51,6 @@ func TestHandler_ExposesBackupxMetrics(t *testing.T) {
m := New("0.0.0-test")
m.ObserveTaskRun("file", "success", 1.0, 2048)
m.SetNodeOnline("n1", "master", true)
m.SetAgentQueue("edge-a", "agent", 3, 1, 2)
m.SetSLABreach(1)
recorder := httptest.NewRecorder()
@@ -69,9 +66,6 @@ func TestHandler_ExposesBackupxMetrics(t *testing.T) {
"backupx_task_run_total",
"backupx_task_run_duration_seconds",
"backupx_node_online",
"backupx_agent_command_queue_depth",
"backupx_agent_command_running",
"backupx_agent_command_timeout_total",
"backupx_sla_breach_tasks",
"backupx_app_info",
} {

View File

@@ -8,47 +8,28 @@ const (
BackupRecordStatusFailed = "failed"
)
const (
// BackupKindFull 全量归档BackupKindDifferential 差异归档;
// BackupKindRepository 为可独立恢复的 CDC 内容寻址快照。
BackupKindFull = "full"
BackupKindDifferential = "differential"
BackupKindRepository = "repository"
)
type BackupRecord struct {
ID uint `gorm:"primaryKey" json:"id"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
// NodeID 执行该次备份的节点0 = 本机 MasterStorageTransferMode 进一步
// 区分远程 Agent 直写与 Master 中转,避免在错误节点访问 local_disk
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Status string `gorm:"size:20;index;not null" json:"status"`
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
Checksum string `gorm:"column:checksum;size:64" json:"checksum"`
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
// 空值表示旧版 Agent 直写direct / master_relay 记录新协议的实际数据路径。
StorageTransferMode string `gorm:"column:storage_transfer_mode;size:20" json:"storageTransferMode,omitempty"`
StorageUploadResults string `gorm:"column:storage_upload_results;type:text" json:"-"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
// Locked 保留锁定(法律保留):为 true 时该备份不参与保留期/数量自动清理,
// 且禁止手动删除,直到显式解锁。用于保护合规快照、迁移前基线等关键备份。
Locked bool `gorm:"column:locked;not null;default:false;index" json:"locked"`
// BackupKind 备份类型full全量/ differential差异/ repositoryCDC 快照)。
BackupKind string `gorm:"column:backup_kind;size:16;not null;default:'full';index" json:"backupKind"`
// BaseRecordID 差异备份所基于的全量备份记录 ID全量记录为 0
BaseRecordID uint `gorm:"column:base_record_id;index;not null;default:0" json:"baseRecordId"`
// Manifest 全量备份的条目清单JSON供后续差异备份比对差异记录为空。
Manifest string `gorm:"column:manifest;type:text" json:"-"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ID uint `gorm:"primaryKey" json:"id"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
// NodeID 执行该次备份的节点0 = 本机 Master用于集群中识别 local_disk 类型
// 存储的归属节点,避免 Master 端试图跨节点访问远程 Agent 的本地存储
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Status string `gorm:"size:20;index;not null" json:"status"`
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
Checksum string `gorm:"column:checksum;size:64" json:"checksum"`
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
StorageUploadResults string `gorm:"column:storage_upload_results;type:text" json:"-"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (BackupRecord) TableName() string {

View File

@@ -8,15 +8,6 @@ const (
BackupTaskTypeSQLite = "sqlite"
BackupTaskTypePostgreSQL = "postgresql"
BackupTaskTypeSAPHANA = "saphana"
BackupTaskTypeMongoDB = "mongodb"
)
const (
// BackupModeFull 全量模式默认BackupModeDifferential 差异归档;
// BackupModeRepository 为 CDC 内容寻址仓库模式(仅文件类型本机任务)。
BackupModeFull = "full"
BackupModeDifferential = "differential"
BackupModeRepository = "repository"
)
const (
@@ -27,54 +18,42 @@ const (
)
type BackupTask struct {
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
Type string `gorm:"size:20;index;not null" json:"type"`
Enabled bool `gorm:"not null;default:true" json:"enabled"`
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
SourcePaths string `gorm:"column:source_paths;type:text" json:"sourcePaths"`
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
DBPort int `gorm:"column:db_port" json:"dbPort"`
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
Type string `gorm:"size:20;index;not null" json:"type"`
Enabled bool `gorm:"not null;default:true" json:"enabled"`
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
SourcePaths string `gorm:"column:source_paths;type:text" json:"sourcePaths"`
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
DBPort int `gorm:"column:db_port" json:"dbPort"`
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
// ExtraConfig 类型特有的扩展配置JSON如 SAP HANA 的 backupLevel / backupChannels 等
ExtraConfig string `gorm:"column:extra_config;type:text" json:"extraConfig"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"` // deprecated: 保留兼容
StorageTarget StorageTarget `json:"storageTarget,omitempty"` // deprecated: 保留兼容
StorageTargets []StorageTarget `gorm:"many2many:backup_task_storage_targets" json:"storageTargets,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Node Node `json:"node,omitempty"`
ExtraConfig string `gorm:"column:extra_config;type:text" json:"extraConfig"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"` // deprecated: 保留兼容
StorageTarget StorageTarget `json:"storageTarget,omitempty"` // deprecated: 保留兼容
StorageTargets []StorageTarget `gorm:"many2many:backup_task_storage_targets" json:"storageTargets,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Node Node `json:"node,omitempty"`
// NodePoolTag 节点池标签(可选)。非空且 NodeID=0 时,调度器会从 Node.Labels 包含该 tag
// 的在线节点中动态挑选一台执行(按运行中任务数最少原则),失败会 best-effort 切换到下一个候选。
// 典型场景NodePoolTag="db" 让 MySQL 备份任务在任意标有 "db" 的数据库节点执行。
NodePoolTag string `gorm:"column:node_pool_tag;size:64;index" json:"nodePoolTag"`
Tags string `gorm:"column:tags;size:500" json:"tags"`
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
// BackupMode 备份模式full全量默认/ differential差异归档/
// repositoryFastCDC 分块、全局去重快照)。后两者仅支持本机文件任务。
BackupMode string `gorm:"column:backup_mode;size:16;not null;default:'full'" json:"backupMode"`
// DiffFullIntervalDays 差异模式下强制全量的间隔(天):最近全量超过该天数则本次自动改为全量,
// 限制差异链跨度与单个差异体积。默认 7。
DiffFullIntervalDays int `gorm:"column:diff_full_interval_days;not null;default:7" json:"diffFullIntervalDays"`
// GFS祖父-父-子)保留:分别保留最近 N 天 / M 周 / K 月 / Y 年的代表性备份(每周期保留最新一份)。
// 任一 > 0 即启用 GFS取代 RetentionDays/MaxBackups 简单策略;全为 0 时维持简单策略(向后兼容)。
KeepDaily int `gorm:"column:keep_daily;not null;default:0" json:"keepDaily"`
KeepWeekly int `gorm:"column:keep_weekly;not null;default:0" json:"keepWeekly"`
KeepMonthly int `gorm:"column:keep_monthly;not null;default:0" json:"keepMonthly"`
KeepYearly int `gorm:"column:keep_yearly;not null;default:0" json:"keepYearly"`
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
NodePoolTag string `gorm:"column:node_pool_tag;size:64;index" json:"nodePoolTag"`
Tags string `gorm:"column:tags;size:500" json:"tags"`
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
// 验证(恢复演练)配置 — 定期自动校验备份可恢复性
VerifyEnabled bool `gorm:"column:verify_enabled;not null;default:false" json:"verifyEnabled"`
VerifyCronExpr string `gorm:"column:verify_cron_expr;size:64" json:"verifyCronExpr"`
VerifyMode string `gorm:"column:verify_mode;size:20;not null;default:'quick'" json:"verifyMode"`
VerifyEnabled bool `gorm:"column:verify_enabled;not null;default:false" json:"verifyEnabled"`
VerifyCronExpr string `gorm:"column:verify_cron_expr;size:64" json:"verifyCronExpr"`
VerifyMode string `gorm:"column:verify_mode;size:20;not null;default:'quick'" json:"verifyMode"`
// SLA 配置 — RPO期望最长未备份间隔与告警阈值
SLAHoursRPO int `gorm:"column:sla_hours_rpo;not null;default:0" json:"slaHoursRpo"`
AlertOnConsecutiveFails int `gorm:"column:alert_on_consecutive_fails;not null;default:1" json:"alertOnConsecutiveFails"`
@@ -89,9 +68,9 @@ type BackupTask struct {
// 语义:上游任务成功后自动触发本任务,形成工作流(如 DB 备份完成 → 归档压缩)。
// 调度器继续按本任务自己的 cron 触发,仅"自动触发"路径响应依赖完成事件。
// 循环依赖检查在 service 层完成,避免配置阶段即出错。
DependsOnTaskIDs string `gorm:"column:depends_on_task_ids;size:500" json:"dependsOnTaskIds"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
DependsOnTaskIDs string `gorm:"column:depends_on_task_ids;size:500" json:"dependsOnTaskIds"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (BackupTask) TableName() string {

View File

@@ -10,46 +10,22 @@ const (
NodeStatusOffline = "offline"
)
// OfflineGracePeriod 节点心跳超时判定阈值:超过该时长未心跳的远程节点视为离线。
// Agent 默认 15s 心跳一次,预留 3 次重试空间。
const OfflineGracePeriod = 45 * time.Second
// EffectiveStatus 返回节点的「实时」在线状态。
//
// 存储字段 Status 由心跳置 online、由后台离线监控置 offline二者之间存在最长一个
// 监控周期的滞后窗口;期间 List/Get/调度器可能读到过期的 "online",进而把任务下发
// 给一台刚刚失联的节点。本方法直接以 LastSeen 推导:远程节点若超过 OfflineGracePeriod
// 未心跳即视为 offline消除该滞后导致的误判。本机节点恒以存储状态为准它就是 Master
// 自身,不依赖心跳)。
func (n *Node) EffectiveStatus(now time.Time) string {
if n == nil {
return NodeStatusOffline
}
if n.IsLocal {
return n.Status
}
if now.Sub(n.LastSeen) > OfflineGracePeriod {
return NodeStatusOffline
}
return n.Status
}
// Node represents a managed server node in the cluster.
// The default "local" node is auto-created for single-machine backward compatibility.
type Node struct {
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:128;uniqueIndex;not null" json:"name"`
Hostname string `gorm:"size:255" json:"hostname"`
IPAddress string `gorm:"column:ip_address;size:64" json:"ipAddress"`
Token string `gorm:"size:128;uniqueIndex;not null" json:"-"`
Status string `gorm:"size:20;not null;default:'offline'" json:"status"`
IsLocal bool `gorm:"not null;default:false" json:"isLocal"`
OS string `gorm:"size:64" json:"os"`
Arch string `gorm:"size:32" json:"arch"`
AgentVer string `gorm:"column:agent_version;size:32" json:"agentVersion"`
LastSeen time.Time `gorm:"column:last_seen" json:"lastSeen"`
PrevToken string `gorm:"size:128;index" json:"-"`
PrevTokenExpires *time.Time `gorm:"column:prev_token_expires" json:"-"`
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:128;uniqueIndex;not null" json:"name"`
Hostname string `gorm:"size:255" json:"hostname"`
IPAddress string `gorm:"column:ip_address;size:64" json:"ipAddress"`
Token string `gorm:"size:128;uniqueIndex;not null" json:"-"`
Status string `gorm:"size:20;not null;default:'offline'" json:"status"`
IsLocal bool `gorm:"not null;default:false" json:"isLocal"`
OS string `gorm:"size:64" json:"os"`
Arch string `gorm:"size:32" json:"arch"`
AgentVer string `gorm:"column:agent_version;size:32" json:"agentVersion"`
LastSeen time.Time `gorm:"column:last_seen" json:"lastSeen"`
PrevToken string `gorm:"size:128;index" json:"-"`
PrevTokenExpires *time.Time `gorm:"column:prev_token_expires" json:"-"`
// MaxConcurrent 该节点允许的最大并发任务数0=不限制,沿用全局 cfg.Backup.MaxConcurrent
// 用于大集群中限制单节点资源占用:例如小内存 Agent 节点可配 1避免多个大备份同时跑挤爆。
MaxConcurrent int `gorm:"column:max_concurrent;not null;default:0" json:"maxConcurrent"`

View File

@@ -1,58 +0,0 @@
package model
import (
"testing"
"time"
)
func TestNodeEffectiveStatus(t *testing.T) {
now := time.Date(2026, 5, 26, 12, 0, 0, 0, time.UTC)
cases := []struct {
name string
node *Node
want string
}{
{
name: "remote fresh heartbeat → stored online",
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-10 * time.Second)},
want: NodeStatusOnline,
},
{
name: "remote stale heartbeat but stored online → derived offline",
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-90 * time.Second)},
want: NodeStatusOffline,
},
{
name: "remote just past grace period → offline",
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-(OfflineGracePeriod + time.Second))},
want: NodeStatusOffline,
},
{
name: "remote within grace period → online",
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-(OfflineGracePeriod - time.Second))},
want: NodeStatusOnline,
},
{
name: "local node ignores LastSeen → stored online",
node: &Node{IsLocal: true, Status: NodeStatusOnline, LastSeen: now.Add(-24 * time.Hour)},
want: NodeStatusOnline,
},
{
name: "remote stored offline stays offline",
node: &Node{IsLocal: false, Status: NodeStatusOffline, LastSeen: now.Add(-5 * time.Second)},
want: NodeStatusOffline,
},
{
name: "nil node → offline",
node: nil,
want: NodeStatusOffline,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := tc.node.EffectiveStatus(now); got != tc.want {
t.Fatalf("EffectiveStatus = %q, want %q", got, tc.want)
}
})
}
}

View File

@@ -11,23 +11,21 @@ const (
)
type RestoreRecord struct {
ID uint `gorm:"primaryKey" json:"id"`
BackupRecordID uint `gorm:"column:backup_record_id;index;not null" json:"backupRecordId"`
BackupRecord BackupRecord `json:"backupRecord,omitempty"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
// TargetPath 恢复到指定目录(仅文件类型本机恢复);空 = 恢复到原始源路径。
TargetPath string `gorm:"column:target_path;size:500" json:"targetPath"`
Status string `gorm:"size:20;index;not null" json:"status"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
TriggeredBy string `gorm:"column:triggered_by;size:100" json:"triggeredBy"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ID uint `gorm:"primaryKey" json:"id"`
BackupRecordID uint `gorm:"column:backup_record_id;index;not null" json:"backupRecordId"`
BackupRecord BackupRecord `json:"backupRecord,omitempty"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Status string `gorm:"size:20;index;not null" json:"status"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
TriggeredBy string `gorm:"column:triggered_by;size:100" json:"triggeredBy"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (RestoreRecord) TableName() string {

View File

@@ -37,12 +37,13 @@ func (n *EmailNotifier) Send(_ context.Context, config map[string]any, message M
from := strings.TrimSpace(asString(config["from"]))
toList := splitCommaValues(asString(config["to"]))
address := host + ":" + strconv.Itoa(port)
headers := []string{"From: " + from, "To: " + strings.Join(toList, ", "), "Subject: " + message.Title, "MIME-Version: 1.0", "Content-Type: text/plain; charset=UTF-8", "", message.Body}
var auth smtp.Auth
if username != "" {
auth = smtp.PlainAuth("", username, password, host)
}
rawMessage := buildRawMessage(from, toList, message)
rawMessage := []byte(strings.Join(headers, "\r\n"))
if port == 465 {
tlsConfig := &tls.Config{ServerName: host}
@@ -85,31 +86,3 @@ func (n *EmailNotifier) Send(_ context.Context, config map[string]any, message M
return smtp.SendMail(address, auth, from, toList, rawMessage)
}
// buildRawMessage 构造 RFC 5322 邮件原文。所有头部值都会剔除 CR/LF
// 防止 SMTP 头注入:备份任务名等用户可控内容会进入 Subject若包含
// 换行符可被用来注入额外头部(如 Bcc或伪造正文。正文本身不做处理
// 允许包含换行。
func buildRawMessage(from string, toList []string, message Message) []byte {
sanitizedTo := make([]string, 0, len(toList))
for _, addr := range toList {
if s := sanitizeHeaderValue(addr); s != "" {
sanitizedTo = append(sanitizedTo, s)
}
}
headers := []string{
"From: " + sanitizeHeaderValue(from),
"To: " + strings.Join(sanitizedTo, ", "),
"Subject: " + sanitizeHeaderValue(message.Title),
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=UTF-8",
"",
message.Body,
}
return []byte(strings.Join(headers, "\r\n"))
}
// sanitizeHeaderValue 移除头部值中的 CR 与 LF消除头注入向量。
func sanitizeHeaderValue(value string) string {
return strings.NewReplacer("\r", "", "\n", "").Replace(strings.TrimSpace(value))
}

View File

@@ -1,55 +0,0 @@
package notify
import (
"strings"
"testing"
)
// TestBuildRawMessageStripsHeaderInjection 验证用户可控内容(如备份任务名
// 进入 Subject中的 CR/LF 被剔除,无法注入额外头部或伪造正文。
func TestBuildRawMessageStripsHeaderInjection(t *testing.T) {
msg := Message{
Title: "备份失败\r\nBcc: attacker@evil.com\r\n\r\n伪造正文",
Body: "正文第一行\n正文第二行",
}
raw := string(buildRawMessage("sender@example.com", []string{"ops@example.com"}, msg))
parts := strings.SplitN(raw, "\r\n\r\n", 2)
if len(parts) != 2 {
t.Fatalf("缺少头部/正文分隔符,原文=%q", raw)
}
headerBlock, body := parts[0], parts[1]
// 头部区不得出现独立的注入头行。
for _, line := range strings.Split(headerBlock, "\r\n") {
if strings.HasPrefix(line, "Bcc:") {
t.Fatalf("检测到头注入:出现独立 Bcc 头行 %q", line)
}
}
// 头部区应恰好是固定的 5 行From/To/Subject/MIME-Version/Content-Type
if got := len(strings.Split(headerBlock, "\r\n")); got != 5 {
t.Fatalf("头部行数=%d期望 5headerBlock=%q", got, headerBlock)
}
// 正文必须保持原样(正文中的 \n 合法,不应被处理)。
if body != "正文第一行\n正文第二行" {
t.Fatalf("正文被篡改:%q", body)
}
// Subject 行必须包含原始标题文本CRLF 被移除后拼接在同一行)。
if !strings.Contains(headerBlock, "Subject: 备份失败Bcc: attacker@evil.com伪造正文") {
t.Fatalf("Subject 行不符合预期:%q", headerBlock)
}
}
func TestSanitizeHeaderValue(t *testing.T) {
cases := map[string]string{
" normal ": "normal",
"a\r\nb": "ab",
"x\ny\rz": "xyz",
"no-control-chars": "no-control-chars",
}
for in, want := range cases {
if got := sanitizeHeaderValue(in); got != want {
t.Errorf("sanitizeHeaderValue(%q) = %q, want %q", in, got, want)
}
}
}

View File

@@ -6,7 +6,6 @@ import (
"encoding/json"
"fmt"
"net/http"
"net/url"
"strings"
"time"
)
@@ -22,18 +21,9 @@ func (n *WebhookNotifier) Type() string { return "webhook" }
func (n *WebhookNotifier) SensitiveFields() []string { return []string{"secret"} }
func (n *WebhookNotifier) Validate(config map[string]any) error {
raw := strings.TrimSpace(asString(config["url"]))
if raw == "" {
if strings.TrimSpace(asString(config["url"])) == "" {
return fmt.Errorf("webhook url is required")
}
parsed, err := url.Parse(raw)
if err != nil {
return fmt.Errorf("webhook url is invalid: %w", err)
}
// 仅允许 http/https杜绝 file://、gopher:// 等可被用于 SSRF 的协议。
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("webhook url must use http or https scheme")
}
return nil
}

View File

@@ -17,46 +17,15 @@ type AgentCommandRepository interface {
// 并返回领取到的命令。无命令时返回 (nil, nil)。
ClaimPending(ctx context.Context, nodeID uint) (*model.AgentCommand, error)
Update(ctx context.Context, cmd *model.AgentCommand) error
// CompleteDispatched 只在命令仍处于 dispatched 时写入终态。
// 返回 false 表示命令已被超时监控或其它流程终结,调用方不应覆盖。
CompleteDispatched(ctx context.Context, cmd *model.AgentCommand) (bool, error)
// MarkStaleTimeout 把 dispatched 状态但超时未完成的命令标记为 timeout。
// 返回被标记的行数。不返回具体命令(供背景监控简单调用)。
MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error)
// TimeoutActive 只在命令仍处于 pending/dispatched 时写入 timeout。
// 返回 false 表示命令已被 Agent 回写为终态,调用方不应覆盖。
TimeoutActive(ctx context.Context, cmd *model.AgentCommand) (bool, error)
// ListStaleDispatched 列出 dispatched 但已超时、尚未被标记的命令。
// 调用方需要把它们逐一标记 timeout 并联动关联记录状态。
ListStaleDispatched(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error)
// ListStaleActive 列出 pending/dispatched 但已超时、尚未完成的命令。
// pending 使用 created_at 判定dispatched 使用 dispatched_at 判定。
ListStaleActive(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error)
// ListPendingByNode 列出某节点下的所有 pending/dispatched 命令。
// 用于删除节点或节点离线时的清理。
ListPendingByNode(ctx context.Context, nodeID uint) ([]model.AgentCommand, error)
NodeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error)
}
type AgentCommandQueueSummary struct {
NodeID uint `json:"nodeId"`
Pending int `json:"pending"`
Dispatched int `json:"dispatched"`
Running int `json:"running"`
Depth int `json:"depth"`
Timeouts int `json:"timeouts"`
LastError string `json:"lastError,omitempty"`
OldestActiveAt *time.Time `json:"oldestActiveAt,omitempty"`
}
type agentCommandTimeoutCount struct {
NodeID uint
Count int
}
type agentCommandLastError struct {
NodeID uint
ErrorMessage string
}
type GormAgentCommandRepository struct {
@@ -125,21 +94,6 @@ func (r *GormAgentCommandRepository) Update(ctx context.Context, cmd *model.Agen
return r.db.WithContext(ctx).Save(cmd).Error
}
func (r *GormAgentCommandRepository) CompleteDispatched(ctx context.Context, cmd *model.AgentCommand) (bool, error) {
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
Where("id = ? AND node_id = ? AND status = ?", cmd.ID, cmd.NodeID, model.AgentCommandStatusDispatched).
Updates(map[string]any{
"status": cmd.Status,
"error_message": cmd.ErrorMessage,
"result": cmd.Result,
"completed_at": cmd.CompletedAt,
})
if result.Error != nil {
return false, result.Error
}
return result.RowsAffected > 0, nil
}
func (r *GormAgentCommandRepository) MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error) {
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
Where("status = ? AND dispatched_at < ?", model.AgentCommandStatusDispatched, threshold).
@@ -153,20 +107,6 @@ func (r *GormAgentCommandRepository) MarkStaleTimeout(ctx context.Context, thres
return result.RowsAffected, nil
}
func (r *GormAgentCommandRepository) TimeoutActive(ctx context.Context, cmd *model.AgentCommand) (bool, error) {
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
Where("id = ? AND status IN ?", cmd.ID, []string{model.AgentCommandStatusPending, model.AgentCommandStatusDispatched}).
Updates(map[string]any{
"status": model.AgentCommandStatusTimeout,
"error_message": cmd.ErrorMessage,
"completed_at": cmd.CompletedAt,
})
if result.Error != nil {
return false, result.Error
}
return result.RowsAffected > 0, nil
}
// ListStaleDispatched 列出 dispatched 但 dispatched_at 早于 threshold 的命令。
func (r *GormAgentCommandRepository) ListStaleDispatched(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error) {
var items []model.AgentCommand
@@ -179,21 +119,6 @@ func (r *GormAgentCommandRepository) ListStaleDispatched(ctx context.Context, th
return items, nil
}
func (r *GormAgentCommandRepository) ListStaleActive(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error) {
var items []model.AgentCommand
if err := r.db.WithContext(ctx).
Where(
"(status = ? AND created_at < ?) OR (status = ? AND dispatched_at < ?)",
model.AgentCommandStatusPending, threshold,
model.AgentCommandStatusDispatched, threshold,
).
Order("id asc").
Find(&items).Error; err != nil {
return nil, err
}
return items, nil
}
// ListPendingByNode 列出某节点下所有待执行pending 或 dispatched命令。
func (r *GormAgentCommandRepository) ListPendingByNode(ctx context.Context, nodeID uint) ([]model.AgentCommand, error) {
var items []model.AgentCommand
@@ -208,114 +133,3 @@ func (r *GormAgentCommandRepository) ListPendingByNode(ctx context.Context, node
}
return items, nil
}
func (r *GormAgentCommandRepository) NodeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error) {
summaries, err := r.activeQueueSummaries(ctx)
if err != nil {
return nil, err
}
if err := r.applyTerminalQueueStats(ctx, summaries); err != nil {
return nil, err
}
return summaries, nil
}
func (r *GormAgentCommandRepository) activeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error) {
var items []model.AgentCommand
if err := r.db.WithContext(ctx).
Where("status IN ?", []string{
model.AgentCommandStatusPending,
model.AgentCommandStatusDispatched,
}).
Order("node_id asc, id asc").
Find(&items).Error; err != nil {
return nil, err
}
summaries := make(map[uint]AgentCommandQueueSummary)
for i := range items {
cmd := &items[i]
summary := summaries[cmd.NodeID]
summary.NodeID = cmd.NodeID
switch cmd.Status {
case model.AgentCommandStatusPending:
summary.Pending++
summary.Depth++
summary.OldestActiveAt = oldestTime(summary.OldestActiveAt, &cmd.CreatedAt)
case model.AgentCommandStatusDispatched:
summary.Dispatched++
summary.Depth++
if isLongRunningAgentCommand(cmd.Type) {
summary.Running++
}
summary.OldestActiveAt = oldestTime(summary.OldestActiveAt, cmd.DispatchedAt)
}
summaries[cmd.NodeID] = summary
}
return summaries, nil
}
func (r *GormAgentCommandRepository) applyTerminalQueueStats(ctx context.Context, summaries map[uint]AgentCommandQueueSummary) error {
var timeoutCounts []agentCommandTimeoutCount
if err := r.db.WithContext(ctx).
Model(&model.AgentCommand{}).
Select("node_id, COUNT(*) AS count").
Where("status = ?", model.AgentCommandStatusTimeout).
Group("node_id").
Scan(&timeoutCounts).Error; err != nil {
return err
}
for _, item := range timeoutCounts {
summary := summaries[item.NodeID]
summary.NodeID = item.NodeID
summary.Timeouts = item.Count
summaries[item.NodeID] = summary
}
terminalStatuses := []string{
model.AgentCommandStatusFailed,
model.AgentCommandStatusTimeout,
}
latestByNode := r.db.WithContext(ctx).
Model(&model.AgentCommand{}).
Select("node_id, MAX(COALESCE(completed_at, updated_at, created_at)) AS last_error_at").
Where("status IN ? AND error_message <> ''", terminalStatuses).
Group("node_id")
var lastErrors []agentCommandLastError
if err := r.db.WithContext(ctx).
Table("agent_commands AS cmd").
Select("cmd.node_id, cmd.error_message").
Joins("JOIN (?) latest ON latest.node_id = cmd.node_id AND latest.last_error_at = COALESCE(cmd.completed_at, cmd.updated_at, cmd.created_at)", latestByNode).
Where("cmd.status IN ? AND cmd.error_message <> ''", terminalStatuses).
Order("cmd.node_id asc, cmd.id desc").
Scan(&lastErrors).Error; err != nil {
return err
}
seenLastError := make(map[uint]struct{}, len(lastErrors))
for _, item := range lastErrors {
if _, ok := seenLastError[item.NodeID]; ok {
continue
}
summary := summaries[item.NodeID]
summary.NodeID = item.NodeID
summary.LastError = item.ErrorMessage
summaries[item.NodeID] = summary
seenLastError[item.NodeID] = struct{}{}
}
return nil
}
func oldestTime(current *time.Time, candidate *time.Time) *time.Time {
if candidate == nil {
return current
}
if current == nil || candidate.Before(*current) {
value := *candidate
return &value
}
return current
}
func isLongRunningAgentCommand(commandType string) bool {
return commandType == model.AgentCommandTypeRunTask || commandType == model.AgentCommandTypeRestoreRecord
}

View File

@@ -90,78 +90,6 @@ func TestAgentCommandRepository_Update(t *testing.T) {
}
}
func TestAgentCommandRepository_CompleteDispatchedOnlyUpdatesDispatchedCommand(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
dispatched := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusDispatched}
timeout := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusTimeout, ErrorMessage: "timeout"}
if err := repo.Create(ctx, dispatched); err != nil {
t.Fatalf("Create dispatched returned error: %v", err)
}
if err := repo.Create(ctx, timeout); err != nil {
t.Fatalf("Create timeout returned error: %v", err)
}
now := time.Now().UTC()
dispatched.Status = model.AgentCommandStatusSucceeded
dispatched.Result = `{"ok":true}`
dispatched.CompletedAt = &now
updated, err := repo.CompleteDispatched(ctx, dispatched)
if err != nil {
t.Fatalf("CompleteDispatched returned error: %v", err)
}
if !updated {
t.Fatal("expected dispatched command to be updated")
}
timeout.Status = model.AgentCommandStatusSucceeded
timeout.Result = `{"late":true}`
timeout.CompletedAt = &now
updated, err = repo.CompleteDispatched(ctx, timeout)
if err != nil {
t.Fatalf("CompleteDispatched terminal returned error: %v", err)
}
if updated {
t.Fatal("expected terminal command not to be updated")
}
gotTimeout, err := repo.FindByID(ctx, timeout.ID)
if err != nil {
t.Fatalf("FindByID timeout returned error: %v", err)
}
if gotTimeout.Status != model.AgentCommandStatusTimeout || gotTimeout.Result != "" {
t.Fatalf("expected timeout command unchanged, got %#v", gotTimeout)
}
}
func TestAgentCommandRepository_TimeoutActiveDoesNotOverwriteTerminalCommand(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
succeeded := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusSucceeded, Result: `{"ok":true}`}
if err := repo.Create(ctx, succeeded); err != nil {
t.Fatalf("Create succeeded returned error: %v", err)
}
now := time.Now().UTC()
succeeded.ErrorMessage = "timeout"
succeeded.CompletedAt = &now
updated, err := repo.TimeoutActive(ctx, succeeded)
if err != nil {
t.Fatalf("TimeoutActive returned error: %v", err)
}
if updated {
t.Fatal("expected terminal command not to be timed out")
}
got, err := repo.FindByID(ctx, succeeded.ID)
if err != nil {
t.Fatalf("FindByID returned error: %v", err)
}
if got.Status != model.AgentCommandStatusSucceeded || got.ErrorMessage != "" || got.Result != `{"ok":true}` {
t.Fatalf("expected succeeded command unchanged, got %#v", got)
}
}
func TestAgentCommandRepository_MarkStaleTimeout(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
@@ -190,72 +118,3 @@ func TestAgentCommandRepository_MarkStaleTimeout(t *testing.T) {
t.Errorf("new should stay dispatched: %+v", newGot)
}
}
func TestAgentCommandRepository_ListStaleActiveIncludesPendingAndDispatched(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
old := time.Now().Add(-time.Hour)
recent := time.Now()
oldPending := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusPending, CreatedAt: old}
oldDispatched := &model.AgentCommand{NodeID: 1, Type: "restore_record", Status: model.AgentCommandStatusDispatched, DispatchedAt: &old}
recentPending := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusPending, CreatedAt: recent}
succeeded := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusSucceeded, CreatedAt: old}
for _, cmd := range []*model.AgentCommand{oldPending, oldDispatched, recentPending, succeeded} {
if err := repo.Create(ctx, cmd); err != nil {
t.Fatalf("Create returned error: %v", err)
}
}
items, err := repo.ListStaleActive(ctx, time.Now().Add(-30*time.Minute))
if err != nil {
t.Fatalf("ListStaleActive returned error: %v", err)
}
if len(items) != 2 {
t.Fatalf("expected 2 stale active commands, got %#v", items)
}
if items[0].ID != oldPending.ID || items[1].ID != oldDispatched.ID {
t.Fatalf("unexpected stale active order/items: %#v", items)
}
}
func TestAgentCommandRepository_NodeQueueSummaries(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
old := time.Now().UTC().Add(-20 * time.Minute)
recent := time.Now().UTC().Add(-2 * time.Minute)
dispatchedAt := time.Now().UTC().Add(-5 * time.Minute)
completedAt := time.Now().UTC().Add(-1 * time.Minute)
commands := []*model.AgentCommand{
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusPending, CreatedAt: old},
{NodeID: 1, Type: model.AgentCommandTypeRestoreRecord, Status: model.AgentCommandStatusPending, CreatedAt: recent},
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusDispatched, DispatchedAt: &dispatchedAt},
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusFailed, ErrorMessage: "boom", CompletedAt: &completedAt},
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusTimeout, ErrorMessage: "late", CompletedAt: &recent},
{NodeID: 2, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusPending, CreatedAt: old},
}
for _, cmd := range commands {
if err := repo.Create(ctx, cmd); err != nil {
t.Fatalf("Create returned error: %v", err)
}
}
summaries, err := repo.NodeQueueSummaries(ctx)
if err != nil {
t.Fatalf("NodeQueueSummaries returned error: %v", err)
}
nodeOne := summaries[1]
if nodeOne.Pending != 2 || nodeOne.Dispatched != 1 || nodeOne.Running != 1 || nodeOne.Depth != 3 {
t.Fatalf("unexpected node 1 summary: %#v", nodeOne)
}
if nodeOne.Timeouts != 1 || nodeOne.LastError != "boom" {
t.Fatalf("expected terminal timeout and latest error in summary, got %#v", nodeOne)
}
if nodeOne.OldestActiveAt == nil || !nodeOne.OldestActiveAt.Equal(old) {
t.Fatalf("expected oldest active at %s, got %#v", old, nodeOne.OldestActiveAt)
}
if nodeTwo := summaries[2]; nodeTwo.Pending != 1 || nodeTwo.Depth != 1 || nodeTwo.Timeouts != 0 || nodeTwo.LastError != "" {
t.Fatalf("unexpected node 2 summary: %#v", nodeTwo)
}
}

View File

@@ -3,7 +3,6 @@ package repository
import (
"context"
"path/filepath"
"sync"
"testing"
"time"
@@ -84,59 +83,6 @@ func TestInstallTokenConsumeExpired(t *testing.T) {
}
}
func TestInstallTokenConsumeConcurrentOnlyOneWins(t *testing.T) {
db := openTestInstallTokenDB(t)
repo := NewAgentInstallTokenRepository(db)
ctx := context.Background()
tok := &model.AgentInstallToken{
Token: "concurrent", NodeID: 1, Mode: model.InstallModeSystemd,
Arch: model.InstallArchAuto, AgentVer: "v1.7.0",
DownloadSrc: model.InstallSourceGitHub,
ExpiresAt: time.Now().UTC().Add(15 * time.Minute),
CreatedByID: 1,
}
if err := repo.Create(ctx, tok); err != nil {
t.Fatalf("create: %v", err)
}
const workers = 8
var wg sync.WaitGroup
start := make(chan struct{})
results := make(chan *model.AgentInstallToken, workers)
errs := make(chan error, workers)
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
<-start
got, err := repo.ConsumeByToken(ctx, "concurrent")
if err != nil {
errs <- err
return
}
results <- got
}()
}
close(start)
wg.Wait()
close(results)
close(errs)
for err := range errs {
t.Fatalf("consume err: %v", err)
}
success := 0
for got := range results {
if got != nil {
success++
}
}
if success != 1 {
t.Fatalf("expected exactly one successful consume, got %d", success)
}
}
func TestInstallTokenGC(t *testing.T) {
db := openTestInstallTokenDB(t)
repo := NewAgentInstallTokenRepository(db)

View File

@@ -29,8 +29,6 @@ type AuditLogRepository interface {
Create(ctx context.Context, log *model.AuditLog) error
List(ctx context.Context, opts AuditLogListOptions) (*AuditLogListResult, error)
ListAll(ctx context.Context, opts AuditLogListOptions) ([]model.AuditLog, error)
// DeleteBefore 删除 created_at 早于 cutoff 的审计日志,返回删除行数。用于保留期清理。
DeleteBefore(ctx context.Context, cutoff time.Time) (int64, error)
}
type gormAuditLogRepository struct {
@@ -45,11 +43,6 @@ func (r *gormAuditLogRepository) Create(_ context.Context, log *model.AuditLog)
return r.db.Create(log).Error
}
func (r *gormAuditLogRepository) DeleteBefore(ctx context.Context, cutoff time.Time) (int64, error) {
result := r.db.WithContext(ctx).Where("created_at < ?", cutoff).Delete(&model.AuditLog{})
return result.RowsAffected, result.Error
}
func (r *gormAuditLogRepository) List(_ context.Context, opts AuditLogListOptions) (*AuditLogListResult, error) {
query := r.buildQuery(opts)
var total int64

View File

@@ -1,68 +0,0 @@
package repository
import (
"context"
"testing"
"time"
"backupx/server/internal/model"
)
// 列表查询应省略 Manifest 列(避免拖出大 JSON而 FindByID 仍保留(内容浏览/差异基线需要)。
func TestListSuccessfulByTaskOmitsManifest(t *testing.T) {
ctx := context.Background()
repo := newBackupRecordTestRepository(t)
rec := &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: "success", BackupKind: model.BackupKindFull, Manifest: `{"entries":[{"p":"x"}]}`, StartedAt: time.Now().UTC()}
if err := repo.Create(ctx, rec); err != nil {
t.Fatalf("create: %v", err)
}
items, err := repo.ListSuccessfulByTask(ctx, 1)
if err != nil {
t.Fatalf("ListSuccessfulByTask: %v", err)
}
if len(items) == 0 {
t.Fatal("expected at least one record")
}
for _, it := range items {
if it.Manifest != "" {
t.Fatalf("ListSuccessfulByTask must omit Manifest, got %q", it.Manifest)
}
}
full, err := repo.FindByID(ctx, rec.ID)
if err != nil || full == nil {
t.Fatalf("FindByID: %v / %v", full, err)
}
if full.Manifest == "" {
t.Fatal("FindByID must retain Manifest (browse/diff depend on it)")
}
}
// 仅统计「成功且依赖给定全量」的差异备份:失败的或依赖其他全量的不计入。
func TestCountDependentDifferentials(t *testing.T) {
ctx := context.Background()
repo := newBackupRecordTestRepository(t)
now := time.Now().UTC()
base := &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: "success", BackupKind: model.BackupKindFull, StartedAt: now}
if err := repo.Create(ctx, base); err != nil {
t.Fatalf("create base: %v", err)
}
mk := func(status, kind string, baseID uint) {
if err := repo.Create(ctx, &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: status, BackupKind: kind, BaseRecordID: baseID, StartedAt: now}); err != nil {
t.Fatalf("create dependent: %v", err)
}
}
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, base.ID) // 计入
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, base.ID) // 计入
mk(model.BackupRecordStatusFailed, model.BackupKindDifferential, base.ID) // 失败 → 不计
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, 99999) // 依赖其他全量 → 不计
n, err := repo.CountDependentDifferentials(ctx, base.ID)
if err != nil {
t.Fatalf("CountDependentDifferentials: %v", err)
}
if n != 2 {
t.Fatalf("want 2 dependent successful differentials, got %d", n)
}
}

View File

@@ -33,14 +33,12 @@ type BackupStorageUsageItem struct {
type BackupRecordRepository interface {
List(context.Context, BackupRecordListOptions) ([]model.BackupRecord, error)
FindByID(context.Context, uint) (*model.BackupRecord, error)
FindRunningByTaskAndNode(context.Context, uint, uint) (*model.BackupRecord, error)
Create(context.Context, *model.BackupRecord) error
Update(context.Context, *model.BackupRecord) error
Delete(context.Context, uint) error
ListRecent(context.Context, int) ([]model.BackupRecord, error)
ListByTask(context.Context, uint) ([]model.BackupRecord, error)
ListSuccessfulByTask(context.Context, uint) ([]model.BackupRecord, error)
CountDependentDifferentials(context.Context, uint) (int64, error)
Count(context.Context) (int64, error)
CountSince(context.Context, time.Time) (int64, error)
CountSuccessSince(context.Context, time.Time) (int64, error)
@@ -58,8 +56,7 @@ func NewBackupRecordRepository(db *gorm.DB) *GormBackupRecordRepository {
}
func (r *GormBackupRecordRepository) List(ctx context.Context, options BackupRecordListOptions) ([]model.BackupRecord, error) {
// Omit("Manifest"):列表不需要可能很大的清单 JSON避免每行拖出该 TEXT 列。
query := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Omit("Manifest").Preload("Task").Preload("Task.StorageTarget").Order("started_at desc")
query := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Preload("Task").Preload("Task.StorageTarget").Order("started_at desc")
if options.TaskID != nil {
query = query.Where("task_id = ?", *options.TaskID)
}
@@ -96,20 +93,6 @@ func (r *GormBackupRecordRepository) FindByID(ctx context.Context, id uint) (*mo
return &item, nil
}
func (r *GormBackupRecordRepository) FindRunningByTaskAndNode(ctx context.Context, taskID uint, nodeID uint) (*model.BackupRecord, error) {
var item model.BackupRecord
if err := r.db.WithContext(ctx).
Where("task_id = ? AND node_id = ? AND status = ?", taskID, nodeID, model.BackupRecordStatusRunning).
Order("id desc").
First(&item).Error; err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil, nil
}
return nil, err
}
return &item, nil
}
func (r *GormBackupRecordRepository) Create(ctx context.Context, item *model.BackupRecord) error {
return r.db.WithContext(ctx).Create(item).Error
}
@@ -127,7 +110,7 @@ func (r *GormBackupRecordRepository) ListRecent(ctx context.Context, limit int)
limit = 10
}
var items []model.BackupRecord
if err := r.db.WithContext(ctx).Omit("Manifest").Preload("Task").Preload("Task.StorageTarget").Order("started_at desc").Limit(limit).Find(&items).Error; err != nil {
if err := r.db.WithContext(ctx).Preload("Task").Preload("Task.StorageTarget").Order("started_at desc").Limit(limit).Find(&items).Error; err != nil {
return nil, err
}
return items, nil
@@ -135,7 +118,7 @@ func (r *GormBackupRecordRepository) ListRecent(ctx context.Context, limit int)
func (r *GormBackupRecordRepository) ListByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
var items []model.BackupRecord
if err := r.db.WithContext(ctx).Omit("Manifest").Where("task_id = ?", taskID).Order("id desc").Find(&items).Error; err != nil {
if err := r.db.WithContext(ctx).Where("task_id = ?", taskID).Order("id desc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil
@@ -143,21 +126,12 @@ func (r *GormBackupRecordRepository) ListByTask(ctx context.Context, taskID uint
func (r *GormBackupRecordRepository) ListSuccessfulByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
var items []model.BackupRecord
if err := r.db.WithContext(ctx).Omit("Manifest").Where("task_id = ? AND status = ?", taskID, "success").Order("completed_at desc, id desc").Find(&items).Error; err != nil {
if err := r.db.WithContext(ctx).Where("task_id = ? AND status = ?", taskID, "success").Order("completed_at desc, id desc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil
}
// CountDependentDifferentials 统计依赖某全量记录(作为基线)的成功差异备份数量。
func (r *GormBackupRecordRepository) CountDependentDifferentials(ctx context.Context, baseID uint) (int64, error) {
var count int64
err := r.db.WithContext(ctx).Model(&model.BackupRecord{}).
Where("base_record_id = ? AND backup_kind = ? AND status = ?", baseID, model.BackupKindDifferential, model.BackupRecordStatusSuccess).
Count(&count).Error
return count, err
}
func (r *GormBackupRecordRepository) Count(ctx context.Context) (int64, error) {
var count int64
if err := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Count(&count).Error; err != nil {

View File

@@ -226,7 +226,7 @@ func (r *GormBackupTaskRepository) Create(ctx context.Context, item *model.Backu
}
func (r *GormBackupTaskRepository) Update(ctx context.Context, item *model.BackupTask) error {
if err := r.db.WithContext(ctx).Omit("StorageTarget", "StorageTargets", "Node").Save(item).Error; err != nil {
if err := r.db.WithContext(ctx).Save(item).Error; err != nil {
return err
}
if len(item.StorageTargets) > 0 {

View File

@@ -92,49 +92,3 @@ func TestBackupTaskRepositoryCRUD(t *testing.T) {
t.Fatalf("expected task deleted, got %#v", deleted)
}
}
func TestBackupTaskRepositoryUpdateCanClearNodeIDAfterPreload(t *testing.T) {
ctx := context.Background()
repo := newBackupTaskTestRepository(t)
remoteNode := &model.Node{Name: "edge-1", Token: "edge-token", Status: model.NodeStatusOnline, IsLocal: false}
if err := repo.db.WithContext(ctx).Create(remoteNode).Error; err != nil {
t.Fatalf("create node: %v", err)
}
task := &model.BackupTask{
Name: "pooled-source",
Type: "file",
Enabled: true,
SourcePath: "/srv/www/site",
StorageTargetID: 1,
NodeID: remoteNode.ID,
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
LastStatus: "idle",
}
if err := repo.Create(ctx, task); err != nil {
t.Fatalf("Create returned error: %v", err)
}
loaded, err := repo.FindByID(ctx, task.ID)
if err != nil {
t.Fatalf("FindByID returned error: %v", err)
}
if loaded == nil || loaded.Node.ID != remoteNode.ID {
t.Fatalf("expected preloaded node %d, got %#v", remoteNode.ID, loaded)
}
loaded.NodeID = 0
loaded.NodePoolTag = "db"
if err := repo.Update(ctx, loaded); err != nil {
t.Fatalf("Update returned error: %v", err)
}
stored, err := repo.FindByID(ctx, task.ID)
if err != nil {
t.Fatalf("FindByID after update returned error: %v", err)
}
if stored.NodeID != 0 {
t.Fatalf("expected NodeID to be cleared, got %d", stored.NodeID)
}
if stored.NodePoolTag != "db" {
t.Fatalf("expected NodePoolTag db, got %q", stored.NodePoolTag)
}
}

View File

@@ -45,7 +45,7 @@ type Service struct {
func NewService(tasks repository.BackupTaskRepository, runner TaskRunner, logger *zap.Logger) *Service {
parser := cron.NewParser(cron.SecondOptional | cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor)
return &Service{
cron: cron.New(cron.WithParser(parser), cron.WithLocation(time.Local)),
cron: cron.New(cron.WithParser(parser), cron.WithLocation(time.UTC)),
tasks: tasks,
runner: runner,
logger: logger,
@@ -167,8 +167,7 @@ func (s *Service) syncTaskLocked(task *model.BackupTask) error {
// 集群感知:若任务绑定了离线的远程节点,跳过本轮触发避免堆积 failed 记录
if taskNodeID > 0 && s.nodes != nil {
node, err := s.nodes.FindByID(context.Background(), taskNodeID)
// 用实时推导的状态判定,避免后台监控刷新前把任务下发给刚失联的节点。
if err == nil && node != nil && !node.IsLocal && node.EffectiveStatus(time.Now().UTC()) != model.NodeStatusOnline {
if err == nil && node != nil && !node.IsLocal && node.Status != model.NodeStatusOnline {
if s.logger != nil {
s.logger.Warn("skip scheduled run: target node offline",
zap.Uint("task_id", taskID), zap.String("task_name", taskName),

View File

@@ -68,37 +68,3 @@ func TestServiceSyncTaskAndTrigger(t *testing.T) {
t.Fatalf("expected scheduled runner to be triggered")
}
}
func TestServiceSchedulesTasksInLocalTimezone(t *testing.T) {
location, err := time.LoadLocation("Asia/Shanghai")
if err != nil {
t.Fatalf("LoadLocation returned error: %v", err)
}
originalLocal := time.Local
time.Local = location
t.Cleanup(func() {
time.Local = originalLocal
})
service := NewService(&fakeTaskRepository{}, &fakeRunner{}, nil)
if got := service.cron.Location(); got != location {
t.Fatalf("cron location = %v, want %v", got, location)
}
task := &model.BackupTask{ID: 1, Enabled: true, CronExpr: "0 5 * * *"}
if err := service.SyncTask(context.Background(), task); err != nil {
t.Fatalf("SyncTask returned error: %v", err)
}
entryID, ok := service.entries[task.ID]
if !ok {
t.Fatalf("expected cron entry for task %d", task.ID)
}
entry := service.cron.Entry(entryID)
now := time.Date(2026, 4, 30, 4, 0, 0, 0, location)
got := entry.Schedule.Next(now)
want := time.Date(2026, 4, 30, 5, 0, 0, 0, location)
if !got.Equal(want) {
t.Fatalf("next run = %s, want %s", got, want)
}
}

View File

@@ -2,22 +2,15 @@ package service
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"math"
"path"
"strings"
"time"
"backupx/server/internal/apperror"
"backupx/server/internal/backup"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
)
@@ -30,7 +23,6 @@ type AgentService struct {
storageRepo repository.StorageTargetRepository
cmdRepo repository.AgentCommandRepository
restoreRepo repository.RestoreRecordRepository
registry *storage.Registry
cipher *codec.ConfigCipher
}
@@ -41,7 +33,6 @@ func NewAgentService(
storageRepo repository.StorageTargetRepository,
cmdRepo repository.AgentCommandRepository,
cipher *codec.ConfigCipher,
registry *storage.Registry,
) *AgentService {
return &AgentService{
nodeRepo: nodeRepo,
@@ -49,7 +40,6 @@ func NewAgentService(
recordRepo: recordRepo,
storageRepo: storageRepo,
cmdRepo: cmdRepo,
registry: registry,
cipher: cipher,
}
}
@@ -128,8 +118,7 @@ func (s *AgentService) SubmitCommandResult(ctx context.Context, node *model.Node
cmd.Result = string(result.Result)
}
cmd.CompletedAt = &now
_, err = s.cmdRepo.CompleteDispatched(ctx, cmd)
return err
return s.cmdRepo.Update(ctx, cmd)
}
// AgentTaskSpec 给 Agent 返回的任务规格,包含解密后的存储配置,供 Agent 直接执行。
@@ -155,11 +144,10 @@ type AgentTaskSpec struct {
// AgentStorageTargetConfig 存储目标配置(已解密)
type AgentStorageTargetConfig struct {
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
TransferMode string `json:"transferMode"`
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
}
// GetTaskSpec 返回 Agent 执行任务所需的完整规格。
@@ -171,8 +159,8 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
if task == nil {
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "任务不存在", nil)
}
if err := s.ensureTaskSpecAccess(ctx, node, task); err != nil {
return nil, err
if task.NodeID != node.ID {
return nil, apperror.Unauthorized("BACKUP_TASK_FORBIDDEN", "任务不属于当前节点", nil)
}
// 解密数据库密码(若有)
dbPassword := ""
@@ -198,22 +186,11 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
if err != nil {
return nil, fmt.Errorf("decrypt storage config: %w", err)
}
transferMode := storage.TransferModeDirect
if strings.EqualFold(target.Type, storage.TypeLocalDisk) {
var localConfig storage.LocalDiskConfig
if err := json.Unmarshal(configRaw, &localConfig); err != nil {
return nil, fmt.Errorf("decode local disk config: %w", err)
}
if localConfig.MasterRelay {
transferMode = storage.TransferModeMasterRelay
}
}
storageTargets = append(storageTargets, AgentStorageTargetConfig{
ID: target.ID,
Type: target.Type,
Name: target.Name,
Config: json.RawMessage(configRaw),
TransferMode: transferMode,
ID: target.ID,
Type: target.Type,
Name: target.Name,
Config: json.RawMessage(configRaw),
})
}
return &AgentTaskSpec{
@@ -236,128 +213,15 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
}, nil
}
// UploadArtifact receives a remote Agent artifact as a stream and writes it
// with a provider created on the Master. The first supported use is local_disk,
// whose configured path belongs to the Master rather than the source Agent.
func (s *AgentService) UploadArtifact(ctx context.Context, node *model.Node, recordID, targetID uint, objectKey string, size int64, checksum string, reader io.Reader) error {
if node == nil || reader == nil || s.registry == nil {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "中转上传参数不完整", nil)
}
record, err := s.recordRepo.FindByID(ctx, recordID)
if err != nil {
return err
}
if record == nil {
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "记录不存在", nil)
}
task, err := s.taskRepo.FindByID(ctx, record.TaskID)
if err != nil {
return err
}
if task == nil || !recordBelongsToNode(record, task, node.ID) {
return apperror.Unauthorized("BACKUP_RECORD_FORBIDDEN", "记录不属于当前节点", nil)
}
if isBackupRecordTerminal(record.Status) {
return apperror.BadRequest("BACKUP_RECORD_TERMINAL", "备份记录已结束,不能继续上传产物", nil)
}
allowedTarget := false
for _, configuredTargetID := range collectTargetIDs(task) {
if configuredTargetID == targetID {
allowedTarget = true
break
}
}
if !allowedTarget {
return apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
}
target, err := s.storageRepo.FindByID(ctx, targetID)
if err != nil {
return err
}
if target == nil || !strings.EqualFold(target.Type, storage.TypeLocalDisk) {
return apperror.BadRequest("AGENT_ARTIFACT_RELAY_UNSUPPORTED", "仅 Master 本地磁盘目标需要中转上传", nil)
}
configMap := map[string]any{}
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
return fmt.Errorf("decrypt storage config: %w", err)
}
masterRelay, _ := configMap["masterRelay"].(bool)
if !masterRelay {
return apperror.BadRequest("AGENT_ARTIFACT_RELAY_UNSUPPORTED", "该本地磁盘目标配置为 Agent 直接写入", nil)
}
cleanKey, keyErr := s.validateArtifactKey(record, task, objectKey, true)
if keyErr != nil {
return keyErr
}
checksum = strings.TrimSpace(checksum)
checksumBytes, checksumErr := hex.DecodeString(checksum)
if size < 0 || size == math.MaxInt64 || checksumErr != nil || len(checksumBytes) != sha256.Size {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "中转上传需要有效的大小和 SHA-256", checksumErr)
}
if target.QuotaBytes > 0 {
usage, usageErr := s.recordRepo.StorageUsage(ctx)
if usageErr != nil {
return fmt.Errorf("read storage usage: %w", usageErr)
}
currentUsed := int64(0)
for _, item := range usage {
if item.StorageTargetID == targetID {
currentUsed = item.TotalSize
break
}
}
if currentUsed >= target.QuotaBytes || size > target.QuotaBytes-currentUsed {
return apperror.BadRequest("BACKUP_STORAGE_QUOTA_EXCEEDED", fmt.Sprintf("超出存储目标配额(当前 %d新增 %d配额 %d", currentUsed, size, target.QuotaBytes), nil)
}
}
provider, err := s.registry.Create(ctx, target.Type, configMap)
if err != nil {
return fmt.Errorf("create master relay provider: %w", err)
}
limited := io.LimitReader(reader, size+1)
hashed := newHashingReader(limited)
metadata := map[string]string{
"taskId": fmt.Sprintf("%d", task.ID),
"recordId": fmt.Sprintf("%d", record.ID),
"sourceNodeId": fmt.Sprintf("%d", node.ID),
"transferMode": storage.TransferModeMasterRelay,
}
if err := provider.Upload(ctx, cleanKey, hashed, size, metadata); err != nil {
return errors.Join(fmt.Errorf("relay artifact to master storage: %w", err), provider.Delete(ctx, cleanKey))
}
if hashed.n != size || !strings.EqualFold(hashed.Sum(), checksum) {
deleteErr := provider.Delete(ctx, cleanKey)
return errors.Join(fmt.Errorf("relayed artifact integrity mismatch: received %d of %d bytes", hashed.n, size), deleteErr)
}
return nil
}
func (s *AgentService) ensureTaskSpecAccess(ctx context.Context, node *model.Node, task *model.BackupTask) error {
if task.NodeID == node.ID {
return nil
}
record, err := s.recordRepo.FindRunningByTaskAndNode(ctx, task.ID, node.ID)
if err != nil {
return err
}
if record == nil {
return apperror.Unauthorized("BACKUP_TASK_FORBIDDEN", "任务不属于当前节点", nil)
}
return nil
}
// AgentRecordUpdate Agent 上报备份记录的最终状态。
type AgentRecordUpdate struct {
Status string `json:"status"` // running | success | failed
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
StorageTargetID uint `json:"storageTargetId,omitempty"`
StorageTransferMode string `json:"storageTransferMode,omitempty"`
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"` // 增量日志,追加到 record.log_content
Status string `json:"status"` // running | success | failed
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"` // 增量日志,追加到 record.log_content
}
// UpdateRecord 更新备份记录的状态/日志。Agent 在执行过程中可多次调用。
@@ -369,109 +233,14 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
if record == nil {
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "记录不存在", nil)
}
// 通过 task.NodeID 判断是否属于当前 agent
task, err := s.taskRepo.FindByID(ctx, record.TaskID)
if err != nil {
return err
}
if task == nil || !recordBelongsToNode(record, task, node.ID) {
if task == nil || task.NodeID != node.ID {
return apperror.Unauthorized("BACKUP_RECORD_FORBIDDEN", "记录不属于当前节点", nil)
}
if isBackupRecordTerminal(record.Status) {
return nil
}
allowedTargets := make(map[uint]struct{})
for _, targetID := range collectTargetIDs(task) {
allowedTargets[targetID] = struct{}{}
}
targetCache := make(map[uint]*model.StorageTarget)
validateTransferMode := func(targetID uint, transferMode string) (string, error) {
if _, ok := allowedTargets[targetID]; !ok {
return "", apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
}
target := targetCache[targetID]
if target == nil {
var findErr error
target, findErr = s.storageRepo.FindByID(ctx, targetID)
if findErr != nil {
return "", findErr
}
if target == nil {
return "", apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "存储目标不存在", nil)
}
targetCache[targetID] = target
}
expectedMode := storage.TransferModeDirect
if strings.EqualFold(target.Type, storage.TypeLocalDisk) {
var localConfig storage.LocalDiskConfig
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &localConfig); err != nil {
return "", fmt.Errorf("decrypt storage config: %w", err)
}
if localConfig.MasterRelay {
expectedMode = storage.TransferModeMasterRelay
}
}
if transferMode != "" && transferMode != expectedMode {
return "", apperror.BadRequest("AGENT_STORAGE_TRANSFER_MODE_INVALID", "Agent 上报的存储传输模式与目标配置不一致", nil)
}
return expectedMode, nil
}
selectedTransferMode := ""
if update.StorageTargetID > 0 {
if _, ok := allowedTargets[update.StorageTargetID]; !ok {
return apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
}
var modeErr error
selectedTransferMode, modeErr = validateTransferMode(update.StorageTargetID, update.StorageTransferMode)
if modeErr != nil {
return modeErr
}
} else if update.StorageTransferMode != "" {
return apperror.BadRequest("AGENT_STORAGE_TRANSFER_MODE_INVALID", "传输模式缺少对应的存储目标", nil)
}
for index := range update.StorageUploadResults {
result := &update.StorageUploadResults[index]
expectedMode, err := validateTransferMode(result.StorageTargetID, result.TransferMode)
if err != nil {
return err
}
if result.FileSize < 0 || (result.Status != "" && result.Status != "success" && result.Status != "failed") {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的存储结果无效", nil)
}
if result.StoragePath != "" {
normalizedPath, err := s.validateArtifactKey(record, task, result.StoragePath, expectedMode == storage.TransferModeMasterRelay)
if err != nil {
return err
}
result.StoragePath = normalizedPath
}
result.TransferMode = expectedMode
}
if update.StoragePath != "" {
if update.StorageTargetID == 0 {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "存储路径缺少对应的存储目标", nil)
}
cleanStoragePath, err := s.validateArtifactKey(record, task, update.StoragePath, selectedTransferMode == storage.TransferModeMasterRelay)
if err != nil {
return err
}
if update.FileName != "" && path.Base(cleanStoragePath) != update.FileName {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "Agent 上报的文件名与存储路径不一致", nil)
}
update.StoragePath = cleanStoragePath
}
if update.Status != "" && update.Status != model.BackupRecordStatusRunning && update.Status != model.BackupRecordStatusSuccess && update.Status != model.BackupRecordStatusFailed {
return apperror.BadRequest("BACKUP_RECORD_STATUS_INVALID", "Agent 上报的备份状态无效", nil)
}
if update.FileSize < 0 || (update.FileName != "" && (path.Base(update.FileName) != update.FileName || strings.Contains(update.FileName, "\\"))) {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的备份文件信息无效", nil)
}
if update.Checksum != "" {
checksumBytes, checksumErr := hex.DecodeString(strings.TrimSpace(update.Checksum))
if checksumErr != nil || len(checksumBytes) != sha256.Size {
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的 SHA-256 无效", checksumErr)
}
update.Checksum = strings.ToLower(strings.TrimSpace(update.Checksum))
}
if update.Status != "" {
record.Status = update.Status
}
@@ -487,15 +256,6 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
if update.StoragePath != "" {
record.StoragePath = update.StoragePath
}
if update.StorageTargetID > 0 {
record.StorageTargetID = update.StorageTargetID
record.StorageTransferMode = selectedTransferMode
}
if len(update.StorageUploadResults) > 0 {
if resultsJSON, marshalErr := json.Marshal(update.StorageUploadResults); marshalErr == nil {
record.StorageUploadResults = string(resultsJSON)
}
}
if update.ErrorMessage != "" {
record.ErrorMessage = update.ErrorMessage
}
@@ -517,46 +277,11 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
// 同步更新任务的 last_status
if update.Status == model.BackupRecordStatusSuccess || update.Status == model.BackupRecordStatusFailed {
task.LastStatus = update.Status
task.LastRunAt = &record.StartedAt
if err := s.taskRepo.Update(ctx, task); err != nil {
return fmt.Errorf("update backup task summary: %w", err)
}
_ = s.taskRepo.Update(ctx, task)
}
return nil
}
func (s *AgentService) validateArtifactKey(record *model.BackupRecord, task *model.BackupTask, objectKey string, requireRecordNamespace bool) (string, error) {
if record == nil || task == nil {
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "无法确认中转对象归属", nil)
}
rawKey := objectKey
cleanKey := path.Clean(rawKey)
fileName := path.Base(cleanKey)
if rawKey == "" || strings.TrimSpace(rawKey) != rawKey || cleanKey == "." || path.IsAbs(cleanKey) || strings.HasPrefix(cleanKey, "../") || cleanKey != rawKey || strings.Contains(rawKey, "\\") || fileName == "." || fileName == "/" {
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "中转上传对象路径不安全", nil)
}
expectedKey := backup.BuildRecordStorageKey(task.Type, record.StartedAt, record.ID, fileName)
if requireRecordNamespace {
legacyKey := backup.BuildStorageKey(task.Type, record.StartedAt, fileName)
if cleanKey != expectedKey && cleanKey != legacyKey {
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "中转上传对象不属于当前备份记录", nil)
}
return expectedKey, nil
}
return cleanKey, nil
}
func recordBelongsToNode(record *model.BackupRecord, task *model.BackupTask, nodeID uint) bool {
if record.NodeID != 0 {
return record.NodeID == nodeID
}
return task.NodeID == nodeID
}
func isBackupRecordTerminal(status string) bool {
return status == model.BackupRecordStatusSuccess || status == model.BackupRecordStatusFailed
}
// EnqueueCommand Master 端调用:给指定节点插入一条待执行命令。
// 返回命令 ID。
func (s *AgentService) EnqueueCommand(ctx context.Context, nodeID uint, cmdType string, payload any) (uint, error) {
@@ -631,84 +356,25 @@ func (s *AgentService) StartCommandTimeoutMonitor(ctx context.Context, interval
}()
}
// processStaleCommands 扫描已超时的 pending/dispatched 命令并联动关联记录。
// 流程:先取超时候选 → 条件式把命令置为 timeout → 对抢到的命令联动 backup/restore 记录。
// processStaleCommands 扫描已超时的 dispatched 命令并联动关联记录。
// 流程:先取超时候选 → 对每条联动 backup/restore 记录 → 把命令置为 timeout
// 单条失败不影响后续处理。
func (s *AgentService) processStaleCommands(ctx context.Context, threshold time.Time) {
commands, err := s.cmdRepo.ListStaleActive(ctx, threshold)
commands, err := s.cmdRepo.ListStaleDispatched(ctx, threshold)
if err != nil || len(commands) == 0 {
return
}
for i := range commands {
cmd := commands[i]
if s.commandStillActive(ctx, &cmd, threshold) {
continue
}
s.failLinkedRecord(ctx, &cmd)
now := time.Now().UTC()
cmd.Status = model.AgentCommandStatusTimeout
cmd.ErrorMessage = "agent did not report result before timeout"
cmd.CompletedAt = &now
timedOut, err := s.cmdRepo.TimeoutActive(ctx, &cmd)
if err != nil || !timedOut {
continue
}
s.failLinkedRecord(ctx, &cmd)
_ = s.cmdRepo.Update(ctx, &cmd)
}
}
// commandStillActive 用关联记录状态、记录更新时间和节点心跳作为长任务续租信号。
// 仅 run_task / restore_record 允许续租,避免短 RPC 命令被在线节点长期保留。
func (s *AgentService) commandStillActive(ctx context.Context, cmd *model.AgentCommand, threshold time.Time) bool {
if cmd.Status != model.AgentCommandStatusDispatched {
return false
}
switch cmd.Type {
case model.AgentCommandTypeRunTask:
var payload struct {
RecordID uint `json:"recordId"`
}
if err := json.Unmarshal([]byte(cmd.Payload), &payload); err != nil || payload.RecordID == 0 {
return false
}
record, err := s.recordRepo.FindByID(ctx, payload.RecordID)
if err != nil || record == nil || record.Status != model.BackupRecordStatusRunning {
return false
}
if s.nodeRecentlySeen(ctx, cmd.NodeID, threshold) {
return true
}
return record.UpdatedAt.After(threshold)
case model.AgentCommandTypeRestoreRecord:
if s.restoreRepo == nil {
return false
}
var payload struct {
RestoreRecordID uint `json:"restoreRecordId"`
}
if err := json.Unmarshal([]byte(cmd.Payload), &payload); err != nil || payload.RestoreRecordID == 0 {
return false
}
restore, err := s.restoreRepo.FindByID(ctx, payload.RestoreRecordID)
if err != nil || restore == nil || restore.Status != model.RestoreRecordStatusRunning {
return false
}
if s.nodeRecentlySeen(ctx, cmd.NodeID, threshold) {
return true
}
return restore.UpdatedAt.After(threshold)
default:
return false
}
}
func (s *AgentService) nodeRecentlySeen(ctx context.Context, nodeID uint, threshold time.Time) bool {
node, err := s.nodeRepo.FindByID(ctx, nodeID)
if err != nil || node == nil {
return false
}
return node.Status == model.NodeStatusOnline && node.LastSeen.After(threshold)
}
// failLinkedRecord 根据命令类型把关联记录标记为 failed。
// 只对仍然处于 running 状态的记录生效,避免覆盖已完成的结果。
func (s *AgentService) failLinkedRecord(ctx context.Context, cmd *model.AgentCommand) {

View File

@@ -1,824 +0,0 @@
package service
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"backupx/server/internal/backup"
"backupx/server/internal/config"
"backupx/server/internal/database"
"backupx/server/internal/logger"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
storageRclone "backupx/server/internal/storage/rclone"
"gorm.io/gorm"
)
func newAgentServicePoolTestHarness(t *testing.T) (*AgentService, *gorm.DB, repository.BackupRecordRepository, repository.AgentCommandRepository, *model.Node, *model.Node) {
t.Helper()
log, err := logger.New(config.LogConfig{Level: "error"})
if err != nil {
t.Fatalf("logger.New returned error: %v", err)
}
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(t.TempDir(), "backupx.db")}, log)
if err != nil {
t.Fatalf("database.Open returned error: %v", err)
}
cipher := codec.NewConfigCipher("agent-service-secret")
nodeRepo := repository.NewNodeRepository(db)
taskRepo := repository.NewBackupTaskRepository(db)
recordRepo := repository.NewBackupRecordRepository(db)
storageRepo := repository.NewStorageTargetRepository(db)
cmdRepo := repository.NewAgentCommandRepository(db)
owner := &model.Node{Name: "edge-owner", Token: "owner-token", Status: model.NodeStatusOnline, IsLocal: false, LastSeen: time.Now().UTC()}
other := &model.Node{Name: "edge-other", Token: "other-token", Status: model.NodeStatusOnline, IsLocal: false, LastSeen: time.Now().UTC()}
if err := nodeRepo.Create(context.Background(), owner); err != nil {
t.Fatalf("create owner node: %v", err)
}
if err := nodeRepo.Create(context.Background(), other); err != nil {
t.Fatalf("create other node: %v", err)
}
targetConfig, err := cipher.EncryptJSON(map[string]any{"basePath": t.TempDir(), "masterRelay": true})
if err != nil {
t.Fatalf("EncryptJSON returned error: %v", err)
}
target := &model.StorageTarget{Name: "local", Type: "local_disk", Enabled: true, ConfigCiphertext: targetConfig, ConfigVersion: 1, LastTestStatus: "unknown"}
if err := storageRepo.Create(context.Background(), target); err != nil {
t.Fatalf("create storage target: %v", err)
}
task := &model.BackupTask{
Name: "pooled-task",
Type: "file",
Enabled: true,
SourcePath: "/srv/data",
StorageTargetID: target.ID,
NodeID: 0,
NodePoolTag: "db",
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
LastStatus: "running",
}
if err := taskRepo.Create(context.Background(), task); err != nil {
t.Fatalf("create task: %v", err)
}
record := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: target.ID,
NodeID: owner.ID,
Status: model.BackupRecordStatusRunning,
StartedAt: time.Now().UTC(),
}
if err := recordRepo.Create(context.Background(), record); err != nil {
t.Fatalf("create record: %v", err)
}
storageRegistry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
return NewAgentService(nodeRepo, taskRepo, recordRepo, storageRepo, cmdRepo, cipher, storageRegistry), db, recordRepo, cmdRepo, owner, other
}
func TestAgentServicePooledTaskUsesRecordNodeForSpecAndRecordUpdates(t *testing.T) {
svc, _, records, _, owner, other := newAgentServicePoolTestHarness(t)
ctx := context.Background()
spec, err := svc.GetTaskSpec(ctx, owner, 1)
if err != nil {
t.Fatalf("owner GetTaskSpec returned error: %v", err)
}
if spec.TaskID != 1 || len(spec.StorageTargets) != 1 {
t.Fatalf("unexpected spec: %#v", spec)
}
if spec.StorageTargets[0].TransferMode != storage.TransferModeMasterRelay {
t.Fatalf("expected local disk to use Master relay, got %#v", spec.StorageTargets[0])
}
if _, err := svc.GetTaskSpec(ctx, other, 1); err == nil {
t.Fatal("expected non-owner node to be forbidden from pooled task spec")
}
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
storagePath := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "backup.tar.gz")
if err := svc.UpdateRecord(ctx, owner, 1, AgentRecordUpdate{
Status: model.BackupRecordStatusSuccess,
FileName: "backup.tar.gz",
FileSize: 123,
StoragePath: storagePath,
StorageTargetID: 1,
StorageTransferMode: storage.TransferModeMasterRelay,
StorageUploadResults: []StorageUploadResultItem{
{StorageTargetID: 1, StorageTargetName: "local", Status: "success", StoragePath: storagePath, FileSize: 123, TransferMode: storage.TransferModeMasterRelay},
},
}); err != nil {
t.Fatalf("owner UpdateRecord returned error: %v", err)
}
updated, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID returned error: %v", err)
}
if updated.Status != model.BackupRecordStatusSuccess || updated.NodeID != owner.ID {
t.Fatalf("unexpected updated record: %#v", updated)
}
if updated.StorageTargetID != 1 {
t.Fatalf("expected successful storage target id 1, got %d", updated.StorageTargetID)
}
if updated.StorageTransferMode != storage.TransferModeMasterRelay {
t.Fatalf("expected Master relay transfer mode, got %q", updated.StorageTransferMode)
}
if !strings.Contains(updated.StorageUploadResults, `"storageTargetName":"local"`) {
t.Fatalf("expected upload results to be persisted, got %q", updated.StorageUploadResults)
}
if err := svc.UpdateRecord(ctx, other, 1, AgentRecordUpdate{LogAppend: "bad"}); err == nil {
t.Fatal("expected non-owner node to be forbidden from record update")
}
}
func TestAgentServiceRelaysRemoteArtifactToMasterLocalDisk(t *testing.T) {
svc, _, records, _, owner, other := newAgentServicePoolTestHarness(t)
ctx := context.Background()
payload := []byte("artifact from remote source server")
digest := sha256.Sum256(payload)
checksum := fmt.Sprintf("%x", digest[:])
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
objectKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "remote-source.tar")
if err := svc.UploadArtifact(ctx, owner, 1, 1, objectKey, int64(len(payload)), checksum, bytes.NewReader(payload)); err != nil {
t.Fatalf("UploadArtifact returned error: %v", err)
}
target, err := svc.storageRepo.FindByID(ctx, 1)
if err != nil || target == nil {
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
}
config := map[string]any{}
if err := svc.cipher.DecryptJSON(target.ConfigCiphertext, &config); err != nil {
t.Fatalf("DecryptJSON target config: %v", err)
}
basePath, _ := config["basePath"].(string)
stored, err := os.ReadFile(filepath.Join(basePath, filepath.FromSlash(objectKey)))
if err != nil {
t.Fatalf("read relayed artifact: %v", err)
}
if !bytes.Equal(stored, payload) {
t.Fatalf("relayed artifact differs: got %q", stored)
}
if err := svc.UploadArtifact(ctx, other, 1, 1, objectKey, int64(len(payload)), checksum, bytes.NewReader(payload)); err == nil {
t.Fatal("expected a different node to be forbidden from relaying the artifact")
}
legacyPayload := []byte("artifact from an older Agent")
legacyDigest := sha256.Sum256(legacyPayload)
legacyKey := backup.BuildStorageKey("file", record.StartedAt, "legacy-agent.tar")
canonicalKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "legacy-agent.tar")
if err := svc.UploadArtifact(ctx, owner, record.ID, target.ID, legacyKey, int64(len(legacyPayload)), fmt.Sprintf("%x", legacyDigest[:]), bytes.NewReader(legacyPayload)); err != nil {
t.Fatalf("UploadArtifact legacy key returned error: %v", err)
}
stored, err = os.ReadFile(filepath.Join(basePath, filepath.FromSlash(canonicalKey)))
if err != nil || !bytes.Equal(stored, legacyPayload) {
t.Fatalf("legacy Agent artifact was not normalized: data=%q err=%v", stored, err)
}
if _, err := os.Stat(filepath.Join(basePath, filepath.FromSlash(legacyKey))); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("legacy object key should not be written directly: %v", err)
}
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{
Status: model.BackupRecordStatusSuccess,
FileName: "legacy-agent.tar",
FileSize: int64(len(legacyPayload)),
Checksum: fmt.Sprintf("%x", legacyDigest[:]),
StoragePath: legacyKey,
StorageTargetID: target.ID,
StorageUploadResults: []StorageUploadResultItem{{
StorageTargetID: target.ID,
Status: "success",
StoragePath: legacyKey,
FileSize: int64(len(legacyPayload)),
}},
}); err != nil {
t.Fatalf("UpdateRecord legacy key returned error: %v", err)
}
updated, err := records.FindByID(ctx, record.ID)
if err != nil {
t.Fatalf("FindByID updated record returned error: %v", err)
}
if updated.StoragePath != canonicalKey || updated.StorageTransferMode != storage.TransferModeMasterRelay || !strings.Contains(updated.StorageUploadResults, canonicalKey) {
t.Fatalf("legacy Agent record was not normalized: %#v", updated)
}
}
func TestAgentServiceRejectsArtifactOutsideRecordNamespace(t *testing.T) {
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
target, err := svc.storageRepo.FindByID(ctx, 1)
if err != nil || target == nil {
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
}
config := map[string]any{}
if err := svc.cipher.DecryptJSON(target.ConfigCiphertext, &config); err != nil {
t.Fatalf("DecryptJSON target config: %v", err)
}
basePath, _ := config["basePath"].(string)
victimKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID+1, "victim.tar")
victimPath := filepath.Join(basePath, filepath.FromSlash(victimKey))
if err := os.MkdirAll(filepath.Dir(victimPath), 0o755); err != nil {
t.Fatalf("MkdirAll victim parent: %v", err)
}
if err := os.WriteFile(victimPath, []byte("keep me"), 0o600); err != nil {
t.Fatalf("WriteFile victim: %v", err)
}
payload := []byte("overwrite")
digest := sha256.Sum256(payload)
if err := svc.UploadArtifact(ctx, owner, record.ID, target.ID, victimKey, int64(len(payload)), fmt.Sprintf("%x", digest[:]), bytes.NewReader(payload)); err == nil {
t.Fatal("expected another record namespace to be rejected")
}
stored, err := os.ReadFile(victimPath)
if err != nil {
t.Fatalf("ReadFile victim: %v", err)
}
if string(stored) != "keep me" {
t.Fatalf("victim object changed: %q", stored)
}
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{StoragePath: victimKey, StorageTargetID: target.ID, StorageTransferMode: storage.TransferModeMasterRelay}); err == nil {
t.Fatal("expected another record namespace in status update to be rejected")
}
}
func TestAgentServiceKeepsExistingLocalDiskTargetsAgentLocal(t *testing.T) {
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
target, err := svc.storageRepo.FindByID(ctx, 1)
if err != nil || target == nil {
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
}
legacyConfig, err := svc.cipher.EncryptJSON(map[string]any{"basePath": t.TempDir()})
if err != nil {
t.Fatalf("EncryptJSON legacy target: %v", err)
}
target.ConfigCiphertext = legacyConfig
if err := svc.storageRepo.Update(ctx, target); err != nil {
t.Fatalf("Update legacy target: %v", err)
}
spec, err := svc.GetTaskSpec(ctx, owner, 1)
if err != nil {
t.Fatalf("GetTaskSpec returned error: %v", err)
}
if len(spec.StorageTargets) != 1 || spec.StorageTargets[0].TransferMode != storage.TransferModeDirect {
t.Fatalf("expected legacy local disk to stay Agent-local, got %#v", spec.StorageTargets)
}
payload := []byte("must not be relayed")
digest := sha256.Sum256(payload)
record, findErr := records.FindByID(ctx, 1)
if findErr != nil {
t.Fatalf("FindByID record returned error: %v", findErr)
}
objectKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "legacy.tar")
err = svc.UploadArtifact(ctx, owner, 1, 1, objectKey, int64(len(payload)), fmt.Sprintf("%x", digest[:]), bytes.NewReader(payload))
if err == nil {
t.Fatal("expected relay upload to be rejected for an Agent-local target")
}
}
func TestAgentServiceUpdateRecordRefreshesTaskSummaryOnTerminalStatus(t *testing.T) {
for _, status := range []string{model.BackupRecordStatusSuccess, model.BackupRecordStatusFailed} {
t.Run(status, func(t *testing.T) {
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{Status: status}); err != nil {
t.Fatalf("UpdateRecord returned error: %v", err)
}
task, err := svc.taskRepo.FindByID(ctx, record.TaskID)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
if task.LastStatus != status {
t.Fatalf("expected task LastStatus %q, got %q", status, task.LastStatus)
}
if task.LastRunAt == nil || !task.LastRunAt.Equal(record.StartedAt) {
t.Fatalf("expected task LastRunAt to match record startedAt %s, got %#v", record.StartedAt, task.LastRunAt)
}
})
}
}
func TestAgentServiceUpdateRecordReturnsTaskSummaryUpdateError(t *testing.T) {
svc, _, _, _, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
expectedErr := errors.New("task update failed")
svc.taskRepo = &failingUpdateTaskRepo{
BackupTaskRepository: svc.taskRepo,
err: expectedErr,
}
err := svc.UpdateRecord(ctx, owner, 1, AgentRecordUpdate{Status: model.BackupRecordStatusSuccess})
if !errors.Is(err, expectedErr) {
t.Fatalf("expected task update error %v, got %v", expectedErr, err)
}
}
func TestAgentServiceProcessStaleCommandsFailsPendingRunTaskRecord(t *testing.T) {
svc, _, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusPending,
Payload: `{"recordId":1}`,
CreatedAt: time.Now().UTC().Add(-time.Hour),
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected command timeout, got %#v", updatedCommand)
}
updatedRecord, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if updatedRecord.Status != model.BackupRecordStatusFailed {
t.Fatalf("expected record failed, got %#v", updatedRecord)
}
if updatedRecord.CompletedAt == nil {
t.Fatal("expected failed record completedAt to be set")
}
}
func TestAgentServiceProcessStaleCommandsFailsPendingRestoreRecord(t *testing.T) {
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
restoreRepo := repository.NewRestoreRecordRepository(db)
restore := &model.RestoreRecord{
BackupRecordID: 1,
TaskID: 1,
NodeID: owner.ID,
Status: model.RestoreRecordStatusRunning,
StartedAt: time.Now().UTC().Add(-time.Hour),
}
if err := restoreRepo.Create(ctx, restore); err != nil {
t.Fatalf("Create restore returned error: %v", err)
}
svc.SetRestoreRepository(restoreRepo)
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRestoreRecord,
Status: model.AgentCommandStatusPending,
Payload: `{"restoreRecordId":1}`,
CreatedAt: time.Now().UTC().Add(-time.Hour),
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected command timeout, got %#v", updatedCommand)
}
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
if err != nil {
t.Fatalf("FindByID restore returned error: %v", err)
}
if updatedRestore.Status != model.RestoreRecordStatusFailed {
t.Fatalf("expected restore failed, got %#v", updatedRestore)
}
if updatedRestore.CompletedAt == nil {
t.Fatal("expected failed restore completedAt to be set")
}
}
func TestAgentServiceProcessStaleCommandsKeepsActiveDispatchedRunTaskRecord(t *testing.T) {
svc, _, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
dispatchedAt := time.Now().UTC().Add(-time.Hour)
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusDispatched,
Payload: `{"recordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusDispatched {
t.Fatalf("expected active command to remain dispatched, got %#v", updatedCommand)
}
updatedRecord, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if updatedRecord.Status != model.BackupRecordStatusRunning {
t.Fatalf("expected active record to remain running, got %#v", updatedRecord)
}
}
func TestAgentServiceProcessStaleCommandsKeepsDispatchedRunTaskWhenNodeHeartbeatIsFresh(t *testing.T) {
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := setBackupRecordUpdatedAt(db, 1, dispatchedAt); err != nil {
t.Fatalf("set backup record updated_at: %v", err)
}
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusDispatched,
Payload: `{"recordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusDispatched {
t.Fatalf("expected command to remain dispatched while node heartbeat is fresh, got %#v", updatedCommand)
}
updatedRecord, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if updatedRecord.Status != model.BackupRecordStatusRunning {
t.Fatalf("expected record to remain running while node heartbeat is fresh, got %#v", updatedRecord)
}
}
func TestAgentServiceProcessStaleCommandsTimesOutShortCommandEvenWhenNodeHeartbeatIsFresh(t *testing.T) {
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeListDir,
Status: model.AgentCommandStatusDispatched,
Payload: `{"path":"/srv"}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected stale short command timeout, got %#v", updatedCommand)
}
}
func TestAgentServiceProcessStaleCommandsTimesOutDispatchedRunTaskWhenRecordIsTerminalEvenWithFreshHeartbeat(t *testing.T) {
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
completedAt := time.Now().UTC().Add(-time.Minute)
record.Status = model.BackupRecordStatusFailed
record.CompletedAt = &completedAt
if err := records.Update(ctx, record); err != nil {
t.Fatalf("Update terminal record returned error: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusDispatched,
Payload: `{"recordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected command timeout when linked record is terminal, got %#v", updatedCommand)
}
}
func TestAgentServiceProcessStaleCommandsTimesOutInactiveDispatchedRunTaskRecord(t *testing.T) {
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := setBackupRecordUpdatedAt(db, 1, dispatchedAt); err != nil {
t.Fatalf("set backup record updated_at: %v", err)
}
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", dispatchedAt).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusDispatched,
Payload: `{"recordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected inactive command timeout, got %#v", updatedCommand)
}
updatedRecord, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if updatedRecord.Status != model.BackupRecordStatusFailed {
t.Fatalf("expected inactive record failed, got %#v", updatedRecord)
}
}
func TestAgentServiceProcessStaleCommandsKeepsActiveDispatchedRestoreRecord(t *testing.T) {
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
restoreRepo := repository.NewRestoreRecordRepository(db)
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
svc.SetRestoreRepository(restoreRepo)
dispatchedAt := time.Now().UTC().Add(-time.Hour)
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRestoreRecord,
Status: model.AgentCommandStatusDispatched,
Payload: `{"restoreRecordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusDispatched {
t.Fatalf("expected active restore command to remain dispatched, got %#v", updatedCommand)
}
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
if err != nil {
t.Fatalf("FindByID restore returned error: %v", err)
}
if updatedRestore.Status != model.RestoreRecordStatusRunning {
t.Fatalf("expected active restore to remain running, got %#v", updatedRestore)
}
}
func TestAgentServiceProcessStaleCommandsKeepsDispatchedRestoreWhenNodeHeartbeatIsFresh(t *testing.T) {
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
restoreRepo := repository.NewRestoreRecordRepository(db)
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
svc.SetRestoreRepository(restoreRepo)
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := setRestoreRecordUpdatedAt(db, restore.ID, dispatchedAt); err != nil {
t.Fatalf("set restore record updated_at: %v", err)
}
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRestoreRecord,
Status: model.AgentCommandStatusDispatched,
Payload: `{"restoreRecordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusDispatched {
t.Fatalf("expected restore command to remain dispatched while node heartbeat is fresh, got %#v", updatedCommand)
}
}
func TestAgentServiceProcessStaleCommandsTimesOutInactiveDispatchedRestoreRecord(t *testing.T) {
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
restoreRepo := repository.NewRestoreRecordRepository(db)
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
svc.SetRestoreRepository(restoreRepo)
dispatchedAt := time.Now().UTC().Add(-time.Hour)
if err := setRestoreRecordUpdatedAt(db, restore.ID, dispatchedAt); err != nil {
t.Fatalf("set restore record updated_at: %v", err)
}
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", dispatchedAt).Error; err != nil {
t.Fatalf("set owner last_seen: %v", err)
}
oldCommand := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRestoreRecord,
Status: model.AgentCommandStatusDispatched,
Payload: `{"restoreRecordId":1}`,
CreatedAt: dispatchedAt,
DispatchedAt: &dispatchedAt,
}
if err := commands.Create(ctx, oldCommand); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected inactive restore command timeout, got %#v", updatedCommand)
}
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
if err != nil {
t.Fatalf("FindByID restore returned error: %v", err)
}
if updatedRestore.Status != model.RestoreRecordStatusFailed {
t.Fatalf("expected inactive restore failed, got %#v", updatedRestore)
}
}
func TestAgentServiceSubmitCommandResultDoesNotOverwriteTerminalCommand(t *testing.T) {
svc, _, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
completedAt := time.Now().UTC().Add(-time.Minute)
command := &model.AgentCommand{
NodeID: owner.ID,
Type: model.AgentCommandTypeRunTask,
Status: model.AgentCommandStatusTimeout,
Payload: `{"recordId":1}`,
ErrorMessage: "timeout",
CompletedAt: &completedAt,
}
if err := commands.Create(ctx, command); err != nil {
t.Fatalf("Create command returned error: %v", err)
}
if err := svc.SubmitCommandResult(ctx, owner, command.ID, AgentCommandResult{Success: true, Result: []byte(`{"ok":true}`)}); err != nil {
t.Fatalf("SubmitCommandResult returned error: %v", err)
}
updatedCommand, err := commands.FindByID(ctx, command.ID)
if err != nil {
t.Fatalf("FindByID command returned error: %v", err)
}
if updatedCommand.Status != model.AgentCommandStatusTimeout {
t.Fatalf("expected terminal command status to remain timeout, got %#v", updatedCommand)
}
if updatedCommand.Result != "" {
t.Fatalf("expected terminal command result to remain empty, got %q", updatedCommand.Result)
}
}
func TestAgentServiceUpdateRecordDoesNotOverwriteTerminalRecord(t *testing.T) {
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
ctx := context.Background()
record, err := records.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
completedAt := time.Now().UTC().Add(-time.Minute)
record.Status = model.BackupRecordStatusFailed
record.ErrorMessage = "timeout"
record.CompletedAt = &completedAt
if err := records.Update(ctx, record); err != nil {
t.Fatalf("Update record returned error: %v", err)
}
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{
Status: model.BackupRecordStatusSuccess,
FileName: "late.tar.gz",
FileSize: 42,
Checksum: "late",
StoragePath: "late/path",
ErrorMessage: "late success",
LogAppend: "late log\n",
}); err != nil {
t.Fatalf("UpdateRecord returned error: %v", err)
}
updatedRecord, err := records.FindByID(ctx, record.ID)
if err != nil {
t.Fatalf("FindByID updated record returned error: %v", err)
}
if updatedRecord.Status != model.BackupRecordStatusFailed {
t.Fatalf("expected terminal record status to remain failed, got %#v", updatedRecord)
}
if updatedRecord.FileName != "" || updatedRecord.StoragePath != "" || updatedRecord.ErrorMessage != "timeout" {
t.Fatalf("expected terminal record fields to remain unchanged, got %#v", updatedRecord)
}
}
func createAgentServiceRestoreRecord(t *testing.T, repo repository.RestoreRecordRepository, nodeID uint) *model.RestoreRecord {
t.Helper()
restore := &model.RestoreRecord{
BackupRecordID: 1,
TaskID: 1,
NodeID: nodeID,
Status: model.RestoreRecordStatusRunning,
StartedAt: time.Now().UTC().Add(-time.Hour),
}
if err := repo.Create(context.Background(), restore); err != nil {
t.Fatalf("Create restore returned error: %v", err)
}
return restore
}
func setBackupRecordUpdatedAt(db *gorm.DB, id uint, updatedAt time.Time) error {
return db.Model(&model.BackupRecord{}).Where("id = ?", id).UpdateColumn("updated_at", updatedAt).Error
}
func setRestoreRecordUpdatedAt(db *gorm.DB, id uint, updatedAt time.Time) error {
return db.Model(&model.RestoreRecord{}).Where("id = ?", id).UpdateColumn("updated_at", updatedAt).Error
}
type failingUpdateTaskRepo struct {
repository.BackupTaskRepository
err error
}
func (r *failingUpdateTaskRepo) Update(context.Context, *model.BackupTask) error {
return r.err
}

View File

@@ -1,86 +0,0 @@
package service
import (
"context"
"path/filepath"
"testing"
"time"
"backupx/server/internal/config"
"backupx/server/internal/database"
"backupx/server/internal/logger"
"backupx/server/internal/model"
"backupx/server/internal/repository"
)
func TestAuditRetention(t *testing.T) {
baseDir := t.TempDir()
log, err := logger.New(config.LogConfig{Level: "error"})
if err != nil {
t.Fatal(err)
}
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(baseDir, "a.db")}, log)
if err != nil {
t.Fatal(err)
}
auditRepo := repository.NewAuditLogRepository(db)
configRepo := repository.NewSystemConfigRepository(db)
svc := NewAuditService(auditRepo)
ctx := context.Background()
// seed 写一条审计日志并把 created_at 强制改到 daysAgo 天前。
seed := func(daysAgo int) {
rec := &model.AuditLog{Username: "t", Category: "test", Action: "seed"}
if err := auditRepo.Create(ctx, rec); err != nil {
t.Fatalf("create: %v", err)
}
ts := time.Now().UTC().AddDate(0, 0, -daysAgo)
if err := db.Model(&model.AuditLog{}).Where("id = ?", rec.ID).Update("created_at", ts).Error; err != nil {
t.Fatalf("backdate: %v", err)
}
}
count := func() int64 {
var n int64
db.Model(&model.AuditLog{}).Count(&n)
return n
}
seed(100)
seed(40)
seed(0)
if count() != 3 {
t.Fatalf("expected 3 seeded, got %d", count())
}
// days<=0 不清理。
if n, _ := svc.PurgeOlderThan(ctx, 0); n != 0 || count() != 3 {
t.Fatalf("days<=0 must not purge (n=%d count=%d)", n, count())
}
// 保留 50 天 → 删除 100 天前那条。
n, err := svc.PurgeOlderThan(ctx, 50)
if err != nil {
t.Fatalf("purge: %v", err)
}
if n != 1 || count() != 2 {
t.Fatalf("expected 1 purged / 2 remaining, got n=%d count=%d", n, count())
}
// 设置驱动retention=10 天 → 再删 40 天前那条。
if err := configRepo.Upsert(ctx, &model.SystemConfig{Key: SettingKeyAuditRetentionDays, Value: "10"}); err != nil {
t.Fatalf("upsert setting: %v", err)
}
svc.runRetentionOnce(ctx, configRepo)
if count() != 1 {
t.Fatalf("expected 1 remaining after retention=10, got %d", count())
}
// retention=0 → 永久保留,不再删除。
if err := configRepo.Upsert(ctx, &model.SystemConfig{Key: SettingKeyAuditRetentionDays, Value: "0"}); err != nil {
t.Fatalf("upsert setting: %v", err)
}
svc.runRetentionOnce(ctx, configRepo)
if count() != 1 {
t.Fatalf("retention=0 must keep all, got %d", count())
}
}

View File

@@ -10,7 +10,6 @@ import (
"fmt"
"log"
"net/http"
"strconv"
"strings"
"sync"
"time"
@@ -52,59 +51,6 @@ func NewAuditService(repo repository.AuditLogRepository) *AuditService {
}
}
// PurgeOlderThan 删除早于 days 天前的审计日志返回删除条数。days<=0 时不清理。
func (s *AuditService) PurgeOlderThan(ctx context.Context, days int) (int64, error) {
if days <= 0 {
return 0, nil
}
cutoff := time.Now().UTC().AddDate(0, 0, -days)
return s.repo.DeleteBefore(ctx, cutoff)
}
// StartRetentionMonitor 启动后台审计保留期清理:按 interval 周期读取
// audit_retention_days 设置,>0 时删除超期审计日志。缺省/0 表示永久保留
// 向后兼容默认不删任何历史。ctx 取消后退出。
func (s *AuditService) StartRetentionMonitor(ctx context.Context, configs repository.SystemConfigRepository, interval time.Duration) {
if s == nil || configs == nil {
return
}
if interval <= 0 {
interval = 6 * time.Hour
}
go func() {
ticker := time.NewTicker(interval)
defer ticker.Stop()
s.runRetentionOnce(ctx, configs) // 启动后立即跑一次
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.runRetentionOnce(ctx, configs)
}
}
}()
}
func (s *AuditService) runRetentionOnce(ctx context.Context, configs repository.SystemConfigRepository) {
cfg, err := configs.GetByKey(ctx, SettingKeyAuditRetentionDays)
if err != nil || cfg == nil {
return
}
days, err := strconv.Atoi(strings.TrimSpace(cfg.Value))
if err != nil || days <= 0 {
return
}
deleted, err := s.PurgeOlderThan(ctx, days)
if err != nil {
log.Printf("[audit] retention purge failed: %v", err)
return
}
if deleted > 0 {
log.Printf("[audit] retention purge: deleted %d logs older than %d days", deleted, days)
}
}
// SetWebhook 动态配置审计事件转发 URL 与签名密钥。
// - url 为空字符串时禁用转发
// - secret 非空时对 payload 计算 HMAC-SHA256作为 X-BackupX-Signature header

View File

@@ -46,10 +46,6 @@ func (r *fakeAuditRepo) ListAll(context.Context, repository.AuditLogListOptions)
return nil, nil
}
func (r *fakeAuditRepo) DeleteBefore(context.Context, time.Time) (int64, error) {
return 0, nil
}
func TestAuditService_WebhookDeliversSignedPayload(t *testing.T) {
var hits atomic.Int32
var got struct {

View File

@@ -293,7 +293,7 @@ func (s *AuthService) verifyLoginMFA(ctx context.Context, user *model.User, inpu
}
func (s *AuthService) userBySubject(ctx context.Context, subject string) (*model.User, error) {
userID, err := strconv.ParseUint(subject, 10, 0)
userID, err := strconv.ParseUint(subject, 10, 64)
if err != nil {
return nil, apperror.Unauthorized("AUTH_INVALID_SUBJECT", "无效用户身份", err)
}

View File

@@ -5,13 +5,11 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"hash"
"io"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
@@ -51,31 +49,14 @@ type StorageUploadResultItem struct {
Status string `json:"status"`
StoragePath string `json:"storagePath,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
TransferMode string `json:"transferMode,omitempty"`
Error string `json:"error,omitempty"`
}
const (
uploadMaxAttempts = 3
uploadRetryBackoff = 10 * time.Second
)
type DownloadedArtifact struct {
FileName string
Reader io.ReadCloser
}
type temporaryArtifactReader struct {
*os.File
directory string
}
func (r *temporaryArtifactReader) Close() error {
closeErr := r.File.Close()
removeErr := os.RemoveAll(r.directory)
return errors.Join(closeErr, removeErr)
}
// collectTargetIDs 获取任务关联的所有存储目标 ID
func collectTargetIDs(task *model.BackupTask) []uint {
if len(task.StorageTargets) > 0 {
@@ -92,34 +73,29 @@ func collectTargetIDs(task *model.BackupTask) []uint {
}
type BackupExecutionService struct {
tasks repository.BackupTaskRepository
records repository.BackupRecordRepository
targets repository.StorageTargetRepository
nodeRepo repository.NodeRepository
storageRegistry *storage.Registry
runnerRegistry *backup.Registry
logHub *backup.LogHub
retention *backupretention.Service
cipher *codec.ConfigCipher
tasks repository.BackupTaskRepository
records repository.BackupRecordRepository
targets repository.StorageTargetRepository
nodeRepo repository.NodeRepository
storageRegistry *storage.Registry
runnerRegistry *backup.Registry
logHub *backup.LogHub
retention *backupretention.Service
cipher *codec.ConfigCipher
notifier BackupResultNotifier
agentDispatcher AgentDispatcher
replicationHook ReplicationTrigger
dependentsResolver DependentsResolver
async func(func())
now func() time.Time
tempDir string
semaphore chan struct{}
async func(func())
now func() time.Time
tempDir string
semaphore chan struct{}
// nodeSemaphores 节点级并发限制(按 NodeID 映射)。
// 没命中的 NodeID 走全局 semaphore节点配置 MaxConcurrent>0 时按该节点独立排队。
nodeSemaphores sync.Map
retries int // rclone 底层重试次数
bandwidthLimit string // rclone 带宽限制(全局默认,节点配置可覆盖)
retries int // rclone 底层重试次数
bandwidthLimit string // rclone 带宽限制(全局默认,节点配置可覆盖)
metrics *metrics.Metrics
taskLocks sync.Map
// repositoryLocks serializes immutable index updates per storage target.
// Repository mode is intentionally single-writer in v1 to avoid orphaned
// duplicate packs when two local tasks discover the same missing chunk.
repositoryLocks sync.Map
}
// SetMetrics 注入 Prometheus 采集器。nil 时所有埋点退化为 no-op。
@@ -229,25 +205,6 @@ func (s *BackupExecutionService) DownloadRecord(ctx context.Context, recordID ui
if err != nil {
return nil, err
}
if record.BackupKind == model.BackupKindRepository {
tempDir, err := os.MkdirTemp(s.tempDir, "repository-download-*")
if err != nil {
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法创建 CDC 导出目录", err)
}
exportName := fmt.Sprintf("backupx-record-%d.tar", record.ID)
exportPath := filepath.Join(tempDir, exportName)
store := backup.NewRepositoryStore(s.cipher.Key())
if err := store.ExportTar(ctx, provider, record.StoragePath, record.Checksum, exportPath); err != nil {
cleanupErr := os.RemoveAll(tempDir)
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法从 CDC 仓库导出归档", errors.Join(err, cleanupErr))
}
file, err := os.Open(exportPath)
if err != nil {
cleanupErr := os.RemoveAll(tempDir)
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法打开 CDC 导出归档", errors.Join(err, cleanupErr))
}
return &DownloadedArtifact{FileName: exportName, Reader: &temporaryArtifactReader{File: file, directory: tempDir}}, nil
}
reader, err := provider.Download(ctx, record.StoragePath)
if err != nil {
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法下载备份文件", err)
@@ -271,16 +228,6 @@ func (s *BackupExecutionService) RestoreRecord(ctx context.Context, recordID uin
if task == nil {
return apperror.New(404, "BACKUP_TASK_NOT_FOUND", "关联的备份任务不存在,无法执行恢复", fmt.Errorf("backup task %d not found", record.TaskID))
}
if record.BackupKind == model.BackupKindRepository {
spec, specErr := s.buildTaskSpec(task, record.StartedAt)
if specErr != nil {
return specErr
}
if err := backup.NewRepositoryStore(s.cipher.Key()).Restore(ctx, provider, record.StoragePath, record.Checksum, spec, backup.NopLogWriter{}); err != nil {
return apperror.Internal("BACKUP_RECORD_RESTORE_FAILED", "从 CDC 仓库恢复备份失败", err)
}
return nil
}
tempDir, err := os.MkdirTemp("", "backupx-restore-*")
if err != nil {
return apperror.Internal("BACKUP_RECORD_RESTORE_FAILED", "无法创建恢复目录", err)
@@ -323,76 +270,11 @@ func (s *BackupExecutionService) DeleteRecord(ctx context.Context, recordID uint
if record == nil {
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", fmt.Errorf("backup record %d not found", recordID))
}
if record.Locked {
return apperror.BadRequest("BACKUP_RECORD_LOCKED",
"该备份已保留锁定(法律保留),请先解锁再删除", nil)
}
// 差异链保护:禁止删除仍被差异备份依赖的全量,否则这些差异将无法恢复(与保留清理的保护一致)。
if record.BackupKind == model.BackupKindFull {
deps, depErr := s.records.CountDependentDifferentials(ctx, record.ID)
if depErr != nil {
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法检查差异备份依赖", depErr)
}
if deps > 0 {
return apperror.BadRequest("BACKUP_RECORD_HAS_DEPENDENTS",
fmt.Sprintf("该全量备份仍有 %d 个差异备份依赖它,删除会导致这些差异无法恢复。请先删除相关差异备份或等待其过期。", deps), nil)
}
}
if record.BackupKind == model.BackupKindRepository {
copies := []StorageUploadResultItem{{
StorageTargetID: record.StorageTargetID,
Status: model.BackupRecordStatusSuccess,
StoragePath: record.StoragePath,
}}
if strings.TrimSpace(record.StorageUploadResults) != "" {
if err := json.Unmarshal([]byte(record.StorageUploadResults), &copies); err != nil {
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法解析 CDC 仓库副本信息,已停止删除以避免遗留数据", err)
}
}
copyPaths := make(map[uint]string, len(copies))
for _, copy := range copies {
if strings.EqualFold(copy.Status, model.BackupRecordStatusSuccess) && strings.TrimSpace(copy.StoragePath) != "" {
copyPaths[copy.StorageTargetID] = copy.StoragePath
}
}
targetIDs := make([]uint, 0, len(copyPaths))
for targetID := range copyPaths {
targetIDs = append(targetIDs, targetID)
}
sort.Slice(targetIDs, func(i, j int) bool { return targetIDs[i] < targetIDs[j] })
unlocks := make([]func(), 0, len(targetIDs))
for _, targetID := range targetIDs {
unlocks = append(unlocks, s.acquireRepositoryLock(targetID))
}
defer func() {
for index := len(unlocks) - 1; index >= 0; index-- {
unlocks[index]()
}
}()
providers := make(map[uint]storage.StorageProvider, len(targetIDs))
for _, targetID := range targetIDs {
provider, resolveErr := s.resolveProvider(ctx, targetID)
if resolveErr != nil {
return resolveErr
}
if deleteErr := provider.Delete(ctx, copyPaths[targetID]); deleteErr != nil {
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法删除 CDC 仓库快照", deleteErr)
}
providers[targetID] = provider
}
for targetID, provider := range providers {
if _, pruneErr := backup.NewRepositoryStore(s.cipher.Key()).Prune(ctx, provider); pruneErr != nil {
return apperror.Internal("BACKUP_REPOSITORY_PRUNE_FAILED", fmt.Sprintf("无法清理存储目标 %d 的 CDC 仓库;记录暂时保留以便重试", targetID), pruneErr)
}
}
if err := s.records.Delete(ctx, recordID); err != nil {
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法删除备份记录", err)
}
return nil
}
if remote, err := s.deleteRemoteLocalDiskObject(ctx, record); err != nil {
// 集群场景保护:跨节点 local_disk 文件 Master 无法远程删除,拒绝操作以避免存储泄漏的错觉
if err := s.validateClusterAccessible(ctx, record); err != nil {
return err
} else if !remote && strings.TrimSpace(record.StoragePath) != "" {
}
if strings.TrimSpace(record.StoragePath) != "" {
provider, err := s.resolveProvider(ctx, record.StorageTargetID)
if err != nil {
return err
@@ -407,50 +289,33 @@ func (s *BackupExecutionService) DeleteRecord(ctx context.Context, recordID uint
return nil
}
func (s *BackupExecutionService) deleteRemoteLocalDiskObject(ctx context.Context, record *model.BackupRecord) (bool, error) {
if strings.TrimSpace(record.StoragePath) == "" || s.nodeRepo == nil {
return false, nil
}
if record.StorageTransferMode == storage.TransferModeMasterRelay {
return false, nil
}
node, err := s.nodeRepo.FindByID(ctx, record.NodeID)
if err != nil || node == nil || node.IsLocal {
return false, nil
}
target, err := s.targets.FindByID(ctx, record.StorageTargetID)
if err != nil {
return false, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
}
if target == nil || !strings.EqualFold(target.Type, "local_disk") {
return false, nil
}
if s.agentDispatcher == nil {
return true, apperror.BadRequest("BACKUP_RECORD_CROSS_NODE_LOCAL_DISK",
fmt.Sprintf("该备份位于节点 %s 的本地磁盘local_diskMaster 无法跨节点删除。请确保 Agent 在线后再操作。", node.Name),
nil)
}
configMap := map[string]any{}
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
return true, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
}
if _, err := s.agentDispatcher.EnqueueCommand(ctx, record.NodeID, model.AgentCommandTypeDeleteStorageObject, map[string]any{
"targetType": target.Type,
"targetConfig": configMap,
"storagePath": record.StoragePath,
}); err != nil {
return true, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "无法下发远程备份文件删除命令", err)
}
return true, nil
}
// validateClusterAccessible 在跨节点 + local_disk 场景下拒绝 Master 端直接访问。
// 场景说明:远程 Agent 把备份写到其本机磁盘local_disk basePathMaster 的
// provider 指向的是 Master 本机的同名路径,访问会静默取错文件或 404。明确拒绝
// 让用户知情,避免假成功。
func (s *BackupExecutionService) validateClusterAccessible(ctx context.Context, record *model.BackupRecord) error {
return validateCrossNodeLocalDisk(ctx, s.nodeRepo, s.targets, record,
"BACKUP_RECORD_CROSS_NODE_LOCAL_DISK", "访问。请登录该节点或改用云存储后再操作")
if record == nil || record.NodeID == 0 {
return nil
}
// 检查是否为远程节点
if s.nodeRepo == nil {
return nil
}
node, err := s.nodeRepo.FindByID(ctx, record.NodeID)
if err != nil || node == nil || node.IsLocal {
return nil
}
// 检查存储类型是否为 local_disk跨节点不可达
target, err := s.targets.FindByID(ctx, record.StorageTargetID)
if err != nil || target == nil {
return nil
}
if strings.EqualFold(target.Type, "local_disk") {
return apperror.BadRequest("BACKUP_RECORD_CROSS_NODE_LOCAL_DISK",
fmt.Sprintf("该备份位于节点 %s 的本地磁盘local_diskMaster 无法跨节点访问。请登录该节点或改用云存储后再操作。", node.Name),
nil)
}
return nil
}
func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async bool) (*BackupRecordDetail, error) {
@@ -461,11 +326,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
if task == nil {
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "备份任务不存在", fmt.Errorf("backup task %d not found", id))
}
unlock := s.acquireTaskStartLock(task.ID)
defer unlock()
if err := s.ensureTaskNotRunning(ctx, task); err != nil {
return nil, err
}
// 维护窗口校验:手动执行同样尊重窗口,避免业务高峰期误触发。
if strings.TrimSpace(task.MaintenanceWindows) != "" {
windows := backup.ParseMaintenanceWindows(task.MaintenanceWindows)
@@ -486,9 +346,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
return nil, perr
}
}
if strings.EqualFold(task.BackupMode, model.BackupModeRepository) && s.resolveRemoteNode(ctx, resolvedNodeID) != nil {
return nil, apperror.BadRequest("BACKUP_TASK_REPOSITORY_REMOTE_UNSUPPORTED", "CDC 仓库模式当前仅支持 Master 本机单写者执行", nil)
}
startedAt := s.now()
// 取第一个存储目标 ID 做兼容
primaryTargetID := task.StorageTargetID
@@ -499,8 +356,8 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
if err := s.records.Create(ctx, record); err != nil {
return nil, apperror.Internal("BACKUP_RECORD_CREATE_FAILED", "无法创建备份记录", err)
}
runTask := *task
runTask.NodeID = resolvedNodeID
// 用池选出的节点 ID 复写 task 副本,使后续路由/执行沿用
task.NodeID = resolvedNodeID
task.LastRunAt = &startedAt
task.LastStatus = "running"
if err := s.tasks.Update(ctx, task); err != nil {
@@ -508,27 +365,27 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
}
// 多节点路由task.NodeID 指向远程节点时,把执行任务入队给 Agent
// NodeID=0 或本机节点时由 Master 直接执行。
if remoteNode := s.resolveRemoteNode(ctx, resolvedNodeID); remoteNode != nil {
if remoteNode := s.resolveRemoteNode(ctx, task.NodeID); remoteNode != nil {
// 节点离线 → 立即把刚创建的 running 记录标记 failed返回明确错误
if remoteNode.Status != model.NodeStatusOnline {
offlineMsg := fmt.Sprintf("节点 %s 当前离线,无法执行备份任务", remoteNode.Name)
_ = s.finalizeRecord(ctx, &runTask, record.ID, startedAt, model.BackupRecordStatusFailed,
offlineMsg, "", "", 0, "", "", primaryTargetID)
_ = s.finalizeRecord(ctx, task, record.ID, startedAt, model.BackupRecordStatusFailed,
offlineMsg, "", "", 0, "", "")
return nil, apperror.BadRequest("NODE_OFFLINE", offlineMsg, nil)
}
if _, enqueueErr := s.agentDispatcher.EnqueueCommand(ctx, resolvedNodeID, model.AgentCommandTypeRunTask, map[string]any{
if _, enqueueErr := s.agentDispatcher.EnqueueCommand(ctx, task.NodeID, model.AgentCommandTypeRunTask, map[string]any{
"taskId": task.ID,
"recordId": record.ID,
}); enqueueErr != nil {
// 入队失败 → 在记录中标记失败,继续返回详情
_ = s.finalizeRecord(ctx, &runTask, record.ID, startedAt, model.BackupRecordStatusFailed,
"无法下发任务到远程节点: "+enqueueErr.Error(), "", "", 0, "", "", primaryTargetID)
_ = s.finalizeRecord(ctx, task, record.ID, startedAt, model.BackupRecordStatusFailed,
"无法下发任务到远程节点: "+enqueueErr.Error(), "", "", 0, "", "")
return nil, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "无法下发任务到远程节点", enqueueErr)
}
return s.getRecordDetail(ctx, record.ID)
}
run := func() {
s.executeTask(context.Background(), &runTask, record.ID, startedAt)
s.executeTask(context.Background(), task, record.ID, startedAt)
}
if async {
s.async(run)
@@ -538,27 +395,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
return s.getRecordDetail(ctx, record.ID)
}
func (s *BackupExecutionService) acquireTaskStartLock(taskID uint) func() {
value, _ := s.taskLocks.LoadOrStore(taskID, &sync.Mutex{})
mu := value.(*sync.Mutex)
mu.Lock()
return mu.Unlock
}
func (s *BackupExecutionService) ensureTaskNotRunning(ctx context.Context, task *model.BackupTask) error {
taskID := task.ID
items, err := s.records.List(ctx, repository.BackupRecordListOptions{TaskID: &taskID, Status: model.BackupRecordStatusRunning})
if err != nil {
return apperror.Internal("BACKUP_RECORD_LIST_FAILED", "无法检查任务运行状态", err)
}
if len(items) == 0 {
return nil
}
return apperror.BadRequest("BACKUP_TASK_ALREADY_RUNNING",
fmt.Sprintf("任务「%s」正在运行记录 #%d请等待完成后再触发。", task.Name, items[0].ID),
nil)
}
// shouldNotify 按任务的告警策略决定是否发送本次通知。
// 成功结果:始终发送(方便用户确认备份状态)。
// 失败结果:仅当"最近 N 条记录(含本次)均为 failed"时发送N = AlertOnConsecutiveFails。
@@ -693,181 +529,14 @@ func (s *BackupExecutionService) isRemoteNode(ctx context.Context, nodeID uint)
// resolveRemoteNode 返回 NodeID 对应的远程节点指针,或 nil 表示本机执行。
// 相比 isRemoteNode它让调用方能读取节点状态在线/离线)做进一步判断。
func (s *BackupExecutionService) resolveRemoteNode(ctx context.Context, nodeID uint) *model.Node {
return resolveRemoteExecutionNode(ctx, s.nodeRepo, s.agentDispatcher != nil, nodeID)
}
// resolveDifferentialBase 为差异备份解析基线全量仅本机NodeID=0文件任务且 BackupMode=differential 时生效。
// 返回最近一次「成功、含清单、未超过 DiffFullIntervalDays」的全量记录 ID 及其清单;
// 无合适基线(首次备份 / 最近全量已过期 / 清单缺失)时 ok=false调用方回退为全量。
func (s *BackupExecutionService) resolveDifferentialBase(ctx context.Context, task *model.BackupTask) (uint, backup.Manifest, bool) {
if task.Type != model.BackupTaskTypeFile || task.NodeID != 0 || !strings.EqualFold(task.BackupMode, model.BackupModeDifferential) {
return 0, backup.Manifest{}, false
if s.nodeRepo == nil || s.agentDispatcher == nil || nodeID == 0 {
return nil
}
records, err := s.records.ListSuccessfulByTask(ctx, task.ID)
if err != nil {
return 0, backup.Manifest{}, false
node, err := s.nodeRepo.FindByID(ctx, nodeID)
if err != nil || node == nil || node.IsLocal {
return nil
}
intervalDays := task.DiffFullIntervalDays
if intervalDays <= 0 {
intervalDays = 7
}
cutoff := time.Now().Add(-time.Duration(intervalDays) * 24 * time.Hour)
for i := range records {
rec := records[i]
if rec.BackupKind != model.BackupKindFull {
continue
}
// 最近的全量已超过强制全量间隔 → 触发新全量,限制差异链跨度与单个差异体积。
if rec.StartedAt.Before(cutoff) {
return 0, backup.Manifest{}, false
}
// 列表查询已省略 Manifest 列这里按需单独加载最近全量的清单FindByID 含 Manifest
full, ferr := s.records.FindByID(ctx, rec.ID)
if ferr != nil || full == nil || strings.TrimSpace(full.Manifest) == "" {
return 0, backup.Manifest{}, false
}
manifest, decErr := backup.DecodeManifest([]byte(full.Manifest))
if decErr != nil || len(manifest.Entries) == 0 {
return 0, backup.Manifest{}, false
}
return rec.ID, manifest, true
}
return 0, backup.Manifest{}, false
}
type repositoryTaskResult struct {
fileName string
logicalSize int64
checksum string
storagePath string
storageTargetID uint
manifestJSON string
uploadResults []StorageUploadResultItem
providers map[uint]storage.StorageProvider
}
func (s *BackupExecutionService) executeRepositoryTask(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, spec backup.TaskSpec, logger *backup.ExecutionLogger) (*repositoryTaskResult, error) {
store := backup.NewRepositoryStore(s.cipher.Key())
plan, err := store.BuildPlan(ctx, spec, logger)
if err != nil {
return nil, err
}
defer func() {
if closeErr := plan.Close(); closeErr != nil {
logger.Warnf("清理 CDC 临时计划失败:%v", closeErr)
}
}()
manifestBytes, err := backup.EncodeManifest(plan.Manifest)
if err != nil {
return nil, fmt.Errorf("encode repository manifest: %w", err)
}
targetIDs := collectTargetIDs(task)
if len(targetIDs) == 0 {
return nil, fmt.Errorf("没有关联的存储目标")
}
storageUsage, usageErr := s.storageUsageSnapshot(ctx)
if usageErr != nil {
logger.Warnf("读取存储目标用量失败,跳过本次软配额校验:%v", usageErr)
storageUsage = map[uint]int64{}
}
snapshotKey := store.SnapshotKey(task.ID, recordID, startedAt)
result := &repositoryTaskResult{
fileName: filepath.Base(snapshotKey),
logicalSize: plan.LogicalSize,
storagePath: snapshotKey,
manifestJSON: string(manifestBytes),
uploadResults: make([]StorageUploadResultItem, 0, len(targetIDs)),
providers: make(map[uint]storage.StorageProvider),
}
var failures []string
for _, targetID := range targetIDs {
target, findErr := s.targets.FindByID(ctx, targetID)
targetName := fmt.Sprintf("target-%d", targetID)
if findErr == nil && target != nil {
targetName = target.Name
}
if findErr != nil || target == nil {
message := "存储目标不存在"
if findErr != nil {
message = findErr.Error()
}
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: message})
failures = append(failures, fmt.Sprintf("%s: %s", targetName, message))
continue
}
provider, resolveErr := s.resolveProviderForNode(ctx, targetID, task.NodeID)
if resolveErr != nil {
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: resolveErr.Error()})
failures = append(failures, fmt.Sprintf("%s: %v", targetName, resolveErr))
continue
}
logger.Infof("同步 CDC 仓库到存储目标:%s", targetName)
unlock := s.acquireRepositoryLock(targetID)
estimatedSize, estimateErr := store.EstimateUploadSize(ctx, provider, plan)
if estimateErr != nil {
unlock()
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: estimateErr.Error()})
failures = append(failures, fmt.Sprintf("%s: %v", targetName, estimateErr))
continue
}
if target.QuotaBytes > 0 && storageUsage[targetID]+estimatedSize > target.QuotaBytes {
unlock()
message := fmt.Sprintf("超出存储目标配额(%d + 预计 %d > %d", storageUsage[targetID], estimatedSize, target.QuotaBytes)
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: message})
failures = append(failures, fmt.Sprintf("%s: %s", targetName, message))
continue
}
upload, uploadErr := store.Upload(ctx, provider, plan, snapshotKey)
unlock()
if uploadErr != nil {
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: uploadErr.Error()})
failures = append(failures, fmt.Sprintf("%s: %v", targetName, uploadErr))
logger.Warnf("存储目标 %s CDC 仓库同步失败:%v", targetName, uploadErr)
continue
}
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{
StorageTargetID: targetID, StorageTargetName: targetName, Status: "success",
StoragePath: upload.SnapshotKey, FileSize: upload.UploadedBytes,
})
result.providers[targetID] = provider
if result.storageTargetID == 0 {
result.storageTargetID = targetID
result.checksum = upload.Checksum
}
logger.Infof("存储目标 %s CDC 同步完成:新块 %d/%d复用 %d bytes实际上传 %d bytes", targetName, upload.NewChunks, upload.UniqueChunks, upload.ReusedBytes, upload.UploadedBytes)
}
if result.storageTargetID == 0 {
return nil, fmt.Errorf("所有存储目标 CDC 仓库同步均失败:%s", strings.Join(failures, "; "))
}
if len(failures) > 0 {
logger.Warnf("部分存储目标 CDC 仓库同步失败:%s", strings.Join(failures, "; "))
}
if s.dependentsResolver != nil {
go func(upstreamID uint, upstreamName string) {
dependents, resolveErr := s.dependentsResolver.TriggerDependents(context.Background(), upstreamID)
if resolveErr != nil {
logger.Warnf("解析任务 %s 的下游依赖失败:%v", upstreamName, resolveErr)
return
}
for _, dependentID := range dependents {
if _, runErr := s.RunTaskByID(context.Background(), dependentID); runErr != nil {
logger.Warnf("触发下游任务 #%d 失败(上游: %s%v", dependentID, upstreamName, runErr)
} else {
logger.Infof("已触发下游任务 #%d上游: %s", dependentID, upstreamName)
}
}
}(task.ID, task.Name)
}
return result, nil
}
func (s *BackupExecutionService) acquireRepositoryLock(targetID uint) func() {
created := &sync.Mutex{}
actual, _ := s.repositoryLocks.LoadOrStore(targetID, created)
lock := actual.(*sync.Mutex)
lock.Lock()
return lock.Unlock
return node
}
func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time) {
@@ -892,80 +561,19 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
var fileSize int64
var checksum string
var storagePath string
selectedStorageTargetID := task.StorageTargetID
var uploadResults []StorageUploadResultItem
// 差异备份链信息:实际类型(全量/差异)、基线全量 ID、全量清单 JSON。
backupKind := model.BackupKindFull
var baseRecordID uint
var manifestJSON string
var repositoryProviders map[uint]storage.StorageProvider
completeRecord := func() {
readyForRepositoryRetention := status == model.BackupRecordStatusSuccess
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, checksum, storagePath, selectedStorageTargetID); finalizeErr != nil {
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, checksum, storagePath); finalizeErr != nil {
logger.Errorf("写回备份记录失败:%v", finalizeErr)
readyForRepositoryRetention = false
}
// 采集任务执行结果到 Prometheus耗时 + 产出字节 + 状态计数)
s.metrics.ObserveTaskRun(task.Type, status, time.Since(startedAt).Seconds(), fileSize)
// 写入多目标上传结果
if len(uploadResults) > 0 {
resultsJSON, marshalErr := json.Marshal(uploadResults)
if marshalErr != nil {
logger.Warnf("序列化多目标上传结果失败:%v", marshalErr)
readyForRepositoryRetention = false
} else if record, findErr := s.records.FindByID(ctx, recordID); findErr != nil || record == nil {
if findErr != nil {
logger.Warnf("读取备份记录以写回多目标结果失败:%v", findErr)
} else {
logger.Warnf("备份记录 #%d 不存在,无法写回多目标结果", recordID)
}
readyForRepositoryRetention = false
} else {
record.StorageUploadResults = string(resultsJSON)
if updateErr := s.records.Update(ctx, record); updateErr != nil {
logger.Warnf("写回多目标上传结果失败:%v", updateErr)
readyForRepositoryRetention = false
}
}
}
// 持久化差异链信息:全量记录其清单(供后续差异比对),差异记录其基线全量 ID。
if status == model.BackupRecordStatusSuccess && (backupKind != model.BackupKindFull || baseRecordID != 0 || manifestJSON != "") {
if record, findErr := s.records.FindByID(ctx, recordID); findErr == nil && record != nil {
record.BackupKind = backupKind
record.BaseRecordID = baseRecordID
record.Manifest = manifestJSON
if updErr := s.records.Update(ctx, record); updErr != nil {
logger.Warnf("写回差异链信息失败:%v", updErr)
readyForRepositoryRetention = false
}
} else {
if findErr != nil {
logger.Warnf("读取备份记录以写回备份类型失败:%v", findErr)
} else {
logger.Warnf("备份记录 #%d 不存在,无法写回备份类型", recordID)
}
readyForRepositoryRetention = false
}
}
if readyForRepositoryRetention && backupKind == model.BackupKindRepository && s.retention != nil && len(repositoryProviders) > 0 {
targetIDs := make([]uint, 0, len(repositoryProviders))
for targetID := range repositoryProviders {
targetIDs = append(targetIDs, targetID)
}
sort.Slice(targetIDs, func(i, j int) bool { return targetIDs[i] < targetIDs[j] })
unlocks := make([]func(), 0, len(targetIDs))
for _, targetID := range targetIDs {
unlocks = append(unlocks, s.acquireRepositoryLock(targetID))
}
cleanupResult, cleanupErr := s.retention.CleanupProviders(ctx, task, repositoryProviders)
for index := len(unlocks) - 1; index >= 0; index-- {
unlocks[index]()
}
if cleanupErr != nil {
logger.Warnf("执行 CDC 仓库保留策略失败:%v", cleanupErr)
} else {
for _, warning := range cleanupResult.Warnings {
logger.Warnf("CDC 仓库保留策略警告:%s", warning)
if resultsJSON, marshalErr := json.Marshal(uploadResults); marshalErr == nil {
if record, findErr := s.records.FindByID(ctx, recordID); findErr == nil && record != nil {
record.StorageUploadResults = string(resultsJSON)
_ = s.records.Update(ctx, record)
}
}
}
@@ -986,33 +594,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
logger.Errorf("构建任务运行时配置失败:%v", err)
return
}
if task.Type == model.BackupTaskTypeFile && strings.EqualFold(task.BackupMode, model.BackupModeRepository) {
backupKind = model.BackupKindRepository
repositoryResult, repositoryErr := s.executeRepositoryTask(ctx, task, recordID, startedAt, spec, logger)
if repositoryErr != nil {
errMessage = repositoryErr.Error()
logger.Errorf("执行 CDC 仓库备份失败:%v", repositoryErr)
return
}
fileName = repositoryResult.fileName
fileSize = repositoryResult.logicalSize
checksum = repositoryResult.checksum
storagePath = repositoryResult.storagePath
selectedStorageTargetID = repositoryResult.storageTargetID
uploadResults = repositoryResult.uploadResults
repositoryProviders = repositoryResult.providers
manifestJSON = repositoryResult.manifestJSON
status = model.BackupRecordStatusSuccess
logger.Infof("CDC 仓库备份执行完成")
return
}
// 差异备份:解析基线全量,命中则切换为差异模式(仅本机文件任务)。
if baseID, baseManifest, ok := s.resolveDifferentialBase(ctx, task); ok {
spec.Differential = true
spec.BaseManifest = baseManifest
baseRecordID = baseID
logger.Infof("差异备份模式:基于全量备份 #%d 仅打包变更", baseID)
}
runner, err := s.runnerRegistry.Runner(spec.Type)
if err != nil {
errMessage = err.Error()
@@ -1026,17 +607,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
return
}
defer os.RemoveAll(result.TempDir)
// 依据运行器产出判定实际类型:产出清单 → 全量(记录清单供后续差异比对);否则为差异。
if result.Manifest != nil {
backupKind = model.BackupKindFull
if data, encErr := backup.EncodeManifest(*result.Manifest); encErr == nil {
manifestJSON = string(data)
} else {
logger.Warnf("备份清单序列化失败(不影响本次备份,但将禁用后续差异):%v", encErr)
}
} else {
backupKind = model.BackupKindDifferential
}
finalPath := result.ArtifactPath
if strings.EqualFold(task.Compression, "gzip") && !strings.HasSuffix(strings.ToLower(finalPath), ".gz") {
logger.Infof("开始压缩备份文件")
@@ -1047,15 +617,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
return
}
finalPath = compressedPath
} else if strings.EqualFold(task.Compression, "zstd") && !strings.HasSuffix(strings.ToLower(finalPath), ".zst") {
logger.Infof("开始压缩备份文件zstd")
compressedPath, compressErr := compress.ZstdFile(finalPath)
if compressErr != nil {
errMessage = compressErr.Error()
logger.Errorf("压缩备份文件失败:%v", compressErr)
return
}
finalPath = compressedPath
}
if task.Encrypt {
logger.Infof("开始加密备份文件")
@@ -1084,11 +645,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
logger.Errorf("没有关联的存储目标")
return
}
storageUsage, err := s.storageUsageSnapshot(ctx)
if err != nil {
logger.Warnf("读取存储目标用量失败,跳过本次软配额校验:%v", err)
storageUsage = map[uint]int64{}
}
// 并行上传到所有目标
uploadResults = make([]StorageUploadResultItem, len(targetIDs))
@@ -1112,7 +668,15 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
}
// 软限额校验QuotaBytes > 0 时,已累计 + 本次 > 配额 → 拒绝上传
if target != nil && target.QuotaBytes > 0 {
currentUsed := storageUsage[targetID]
currentUsed := int64(0)
if items, err := s.records.StorageUsage(ctx); err == nil {
for _, it := range items {
if it.StorageTargetID == targetID {
currentUsed = it.TotalSize
break
}
}
}
if currentUsed+fileSize > target.QuotaBytes {
quotaMsg := fmt.Sprintf("超出存储目标 %s 的配额(%d + %d > %d", targetName, currentUsed, fileSize, target.QuotaBytes)
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: quotaMsg}
@@ -1121,18 +685,15 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
}
}
logger.Infof("开始上传备份到存储目标:%s", targetName)
// 上传级重试:最多 3 次,等待时间随 context 取消及时退出。
// 上传级重试:最多 3 次,指数退避10s, 30s, 90s
maxAttempts := 3
var lastUploadErr error
var hr *hashingReader
for attempt := 1; attempt <= uploadMaxAttempts; attempt++ {
for attempt := 1; attempt <= maxAttempts; attempt++ {
if attempt > 1 {
backoff := time.Duration(attempt-1) * uploadRetryBackoff
backoff := time.Duration(attempt*attempt) * 10 * time.Second
logger.Warnf("存储目标 %s 第 %d 次重试(等待 %v%v", targetName, attempt, backoff, lastUploadErr)
if waitErr := waitForUploadRetry(ctx, backoff); waitErr != nil {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: waitErr.Error()}
logger.Warnf("存储目标 %s 上传重试已取消:%v", targetName, waitErr)
return
}
time.Sleep(backoff)
}
artifact, openErr := os.Open(finalPath)
if openErr != nil {
@@ -1162,7 +723,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
}
if lastUploadErr != nil {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: lastUploadErr.Error()}
logger.Warnf("存储目标 %s 上传失败(已重试 %d 次):%v", targetName, uploadMaxAttempts, lastUploadErr)
logger.Warnf("存储目标 %s 上传失败(已重试 %d 次):%v", targetName, maxAttempts, lastUploadErr)
return
}
// 完整性校验:对比实际传输字节数
@@ -1198,9 +759,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
for _, r := range uploadResults {
if r.Status == "success" {
anySuccess = true
if selectedStorageTargetID == task.StorageTargetID {
selectedStorageTargetID = r.StorageTargetID
}
} else if r.Error != "" {
failedMessages = append(failedMessages, fmt.Sprintf("%s: %s", r.StorageTargetName, r.Error))
}
@@ -1233,7 +791,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
record := &model.BackupRecord{
ID: recordID,
TaskID: task.ID,
StorageTargetID: selectedStorageTargetID,
StorageTargetID: task.StorageTargetID,
NodeID: task.NodeID,
Status: "success",
FileName: fileName,
@@ -1258,7 +816,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
}
}
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, checksum string, storagePath string, storageTargetID uint) error {
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, checksum string, storagePath string) error {
record, err := s.records.FindByID(ctx, recordID)
if err != nil {
return err
@@ -1268,9 +826,6 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
}
completedAt := s.now()
record.Status = status
if storageTargetID > 0 {
record.StorageTargetID = storageTargetID
}
record.FileName = fileName
record.FileSize = fileSize
record.Checksum = checksum
@@ -1287,32 +842,6 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
return s.tasks.Update(ctx, task)
}
func (s *BackupExecutionService) storageUsageSnapshot(ctx context.Context) (map[uint]int64, error) {
items, err := s.records.StorageUsage(ctx)
if err != nil {
return nil, fmt.Errorf("storage usage snapshot: %w", err)
}
usage := make(map[uint]int64, len(items))
for _, item := range items {
usage[item.StorageTargetID] = item.TotalSize
}
return usage, nil
}
func waitForUploadRetry(ctx context.Context, delay time.Duration) error {
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func (s *BackupExecutionService) resolveProvider(ctx context.Context, targetID uint) (storage.StorageProvider, error) {
return s.resolveProviderForNode(ctx, targetID, 0)
}
@@ -1326,11 +855,78 @@ func (s *BackupExecutionService) resolveProviderForNode(ctx context.Context, tar
LowLevelRetries: s.retries,
BandwidthLimit: s.effectiveBandwidth(ctx, nodeID),
})
return resolveStorageProvider(ctx, s.targets, s.storageRegistry, s.cipher, targetID)
target, err := s.targets.FindByID(ctx, targetID)
if err != nil {
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
}
if target == nil {
return nil, apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "关联的存储目标不存在", nil)
}
configMap := map[string]any{}
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
}
provider, err := s.storageRegistry.Create(ctx, target.Type, configMap)
if err != nil {
return nil, err
}
return provider, nil
}
func (s *BackupExecutionService) buildTaskSpec(task *model.BackupTask, startedAt time.Time) (backup.TaskSpec, error) {
return buildBackupTaskSpec(s.cipher, task, startedAt, s.tempDir)
excludePatterns := []string{}
if strings.TrimSpace(task.ExcludePatterns) != "" {
if err := json.Unmarshal([]byte(task.ExcludePatterns), &excludePatterns); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析排除规则", err)
}
}
password := ""
if strings.TrimSpace(task.DBPasswordCiphertext) != "" {
plain, err := s.cipher.Decrypt(task.DBPasswordCiphertext)
if err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECRYPT_FAILED", "无法解密数据库密码", err)
}
password = string(plain)
}
sourcePaths := []string{}
if strings.TrimSpace(task.SourcePaths) != "" {
if err := json.Unmarshal([]byte(task.SourcePaths), &sourcePaths); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
}
}
dbSpec := backup.DatabaseSpec{
Host: task.DBHost,
Port: task.DBPort,
User: task.DBUser,
Password: password,
Names: []string{task.DBName},
Path: task.DBPath,
}
// 解析 ExtraConfig 填充类型特有字段(目前主要用于 SAP HANA
if strings.TrimSpace(task.ExtraConfig) != "" {
extra := map[string]any{}
if err := json.Unmarshal([]byte(task.ExtraConfig), &extra); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
}
applyHANAExtraConfig(&dbSpec, extra)
}
return backup.TaskSpec{
ID: task.ID,
Name: task.Name,
Type: task.Type,
SourcePath: task.SourcePath,
SourcePaths: sourcePaths,
ExcludePatterns: excludePatterns,
StorageTargetID: task.StorageTargetID,
StorageTargetType: "",
Compression: task.Compression,
Encrypt: task.Encrypt,
RetentionDays: task.RetentionDays,
MaxBackups: task.MaxBackups,
StartedAt: startedAt,
TempDir: s.tempDir,
Database: dbSpec,
}, nil
}
// applyHANAExtraConfig 从 ExtraConfig map 中提取 SAP HANA 字段填入 DatabaseSpec。
@@ -1361,9 +957,6 @@ func (s *BackupExecutionService) loadRecordProvider(ctx context.Context, recordI
if record == nil {
return nil, nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", fmt.Errorf("backup record %d not found", recordID))
}
if err := s.validateClusterAccessible(ctx, record); err != nil {
return nil, nil, err
}
provider, err := s.resolveProvider(ctx, record.StorageTargetID)
if err != nil {
return nil, nil, err
@@ -1372,7 +965,22 @@ func (s *BackupExecutionService) loadRecordProvider(ctx context.Context, recordI
}
func (s *BackupExecutionService) prepareArtifactForRestore(artifactPath string) (string, error) {
return prepareBackupArtifact(s.cipher, artifactPath, nil)
currentPath := artifactPath
if strings.HasSuffix(strings.ToLower(currentPath), ".enc") {
decryptedPath, err := backupcrypto.DecryptFile(s.cipher.Key(), currentPath)
if err != nil {
return "", err
}
currentPath = decryptedPath
}
if strings.HasSuffix(strings.ToLower(currentPath), ".gz") {
decompressedPath, err := compress.GunzipFile(currentPath)
if err != nil {
return "", err
}
currentPath = decompressedPath
}
return currentPath, nil
}
func (s *BackupExecutionService) getRecordDetail(ctx context.Context, recordID uint) (*BackupRecordDetail, error) {

View File

@@ -1,17 +1,10 @@
package service
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"backupx/server/internal/backup"
backupretention "backupx/server/internal/backup/retention"
@@ -25,70 +18,6 @@ import (
storageRclone "backupx/server/internal/storage/rclone"
)
type testStorageFactory struct {
providers map[string]*testStorageProvider
}
func (f *testStorageFactory) Type() storage.ProviderType {
return "test_storage"
}
func (f *testStorageFactory) New(_ context.Context, config map[string]any) (storage.StorageProvider, error) {
name, _ := config["name"].(string)
provider := f.providers[name]
if provider == nil {
return nil, fmt.Errorf("unknown provider %q", name)
}
return provider, nil
}
type testStorageProvider struct {
name string
failUpload bool
blockUpload <-chan struct{}
onUpload func()
objects map[string][]byte
}
func (p *testStorageProvider) Type() storage.ProviderType { return "test_storage" }
func (p *testStorageProvider) TestConnection(context.Context) error {
return nil
}
func (p *testStorageProvider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
if p.blockUpload != nil {
<-p.blockUpload
}
if p.onUpload != nil {
p.onUpload()
}
if p.failUpload {
return fmt.Errorf("upload failed for %s", p.name)
}
data, err := io.ReadAll(reader)
if err != nil {
return err
}
if p.objects == nil {
p.objects = map[string][]byte{}
}
p.objects[objectKey] = data
return nil
}
func (p *testStorageProvider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
data, ok := p.objects[objectKey]
if !ok {
return nil, fmt.Errorf("object %s not found", objectKey)
}
return io.NopCloser(strings.NewReader(string(data))), nil
}
func (p *testStorageProvider) Delete(_ context.Context, objectKey string) error {
delete(p.objects, objectKey)
return nil
}
func (p *testStorageProvider) List(context.Context, string) ([]storage.ObjectInfo, error) {
return nil, nil
}
func newExecutionTestServices(t *testing.T) (*BackupExecutionService, *BackupRecordService, repository.BackupTaskRepository, repository.StorageTargetRepository, repository.BackupRecordRepository, string, string) {
t.Helper()
baseDir := t.TempDir()
@@ -156,529 +85,6 @@ func TestBackupExecutionServiceRunTaskByIDSync(t *testing.T) {
}
}
func TestBackupExecutionServiceRepositoryModeRoundTrip(t *testing.T) {
executionService, recordService, tasks, _, records, sourceDir, storageDir := newExecutionTestServices(t)
ctx := context.Background()
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
task.BackupMode = model.BackupModeRepository
task.Compression = "zstd"
if err := tasks.Update(ctx, task); err != nil {
t.Fatalf("Update repository task returned error: %v", err)
}
large := make([]byte, 4<<20)
for index := range large {
large[index] = byte((index * 31) % 251)
}
largePath := filepath.Join(sourceDir, "large.bin")
if err := os.WriteFile(largePath, large, 0o640); err != nil {
t.Fatalf("write large fixture: %v", err)
}
first, err := executionService.RunTaskByIDSync(ctx, task.ID)
if err != nil {
t.Fatalf("first repository backup returned error: %v", err)
}
if first.Status != model.BackupRecordStatusSuccess || first.BackupKind != model.BackupKindRepository {
t.Fatalf("unexpected first repository record: %#v", first)
}
if !strings.HasPrefix(first.StoragePath, ".backupx/repository/v1/snapshots/") {
t.Fatalf("unexpected repository snapshot path: %s", first.StoragePath)
}
large[2<<20] ^= 0xff
if err := os.WriteFile(largePath, large, 0o640); err != nil {
t.Fatalf("modify large fixture: %v", err)
}
second, err := executionService.RunTaskByIDSync(ctx, task.ID)
if err != nil {
t.Fatalf("second repository backup returned error: %v", err)
}
stored, err := records.FindByID(ctx, second.ID)
if err != nil {
t.Fatalf("FindByID repository record returned error: %v", err)
}
if stored == nil || stored.BackupKind != model.BackupKindRepository || stored.Manifest == "" {
t.Fatalf("repository metadata was not persisted: %#v", stored)
}
download, err := recordService.Download(ctx, second.ID)
if err != nil {
t.Fatalf("export repository snapshot returned error: %v", err)
}
exported, readErr := io.ReadAll(download.Reader)
closeErr := download.Reader.Close()
if readErr != nil || closeErr != nil {
t.Fatalf("read repository export: read=%v close=%v", readErr, closeErr)
}
if len(exported) == 0 || !strings.HasSuffix(download.FileName, ".tar") {
t.Fatalf("unexpected repository export: name=%s size=%d", download.FileName, len(exported))
}
if err := os.WriteFile(largePath, bytes.Repeat([]byte{0}, len(large)), 0o640); err != nil {
t.Fatalf("damage source before restore: %v", err)
}
if err := executionService.RestoreRecord(ctx, second.ID); err != nil {
t.Fatalf("restore repository record returned error: %v", err)
}
restored, err := os.ReadFile(largePath)
if err != nil {
t.Fatalf("read restored source: %v", err)
}
if !bytes.Equal(restored, large) {
t.Fatalf("repository restore did not reproduce the source")
}
if err := recordService.Delete(ctx, first.ID); err != nil {
t.Fatalf("delete first repository record: %v", err)
}
if err := recordService.Delete(ctx, second.ID); err != nil {
t.Fatalf("delete second repository record: %v", err)
}
packRoot := filepath.Join(storageDir, filepath.FromSlash(".backupx/repository/v1/packs"))
remainingPacks := 0
if err := filepath.Walk(packRoot, func(_ string, info os.FileInfo, walkErr error) error {
if walkErr != nil {
if os.IsNotExist(walkErr) {
return nil
}
return walkErr
}
if info != nil && !info.IsDir() {
remainingPacks++
}
return nil
}); err != nil {
t.Fatalf("inspect repository packs: %v", err)
}
if remainingPacks != 0 {
t.Fatalf("repository prune left %d packs after deleting all snapshots", remainingPacks)
}
}
func TestBackupExecutionServiceNodePoolSelectionDoesNotPersistTaskNodeID(t *testing.T) {
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
nodeRepo := &nodeRepoStub{nodes: []model.Node{
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline, Labels: "prod,db"},
{ID: 11, Name: "edge-b", Token: "edge-b-token", Status: model.NodeStatusOnline, Labels: "prod,db"},
}}
dispatcher := &fakeDispatcher{}
executionService.SetClusterDependencies(nodeRepo, dispatcher)
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID returned error: %v", err)
}
task.NodeID = 0
task.NodePoolTag = "db"
if err := tasks.Update(ctx, task); err != nil {
t.Fatalf("Update task returned error: %v", err)
}
detail, err := executionService.RunTaskByID(ctx, 1)
if err != nil {
t.Fatalf("RunTaskByID returned error: %v", err)
}
storedTask, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID after run returned error: %v", err)
}
if storedTask.NodeID != 0 {
t.Fatalf("expected pooled task NodeID to remain 0, got %d", storedTask.NodeID)
}
if storedTask.NodePoolTag != "db" {
t.Fatalf("expected pooled task tag to remain db, got %q", storedTask.NodePoolTag)
}
storedRecord, err := records.FindByID(ctx, detail.ID)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if storedRecord == nil || storedRecord.NodeID != 10 {
t.Fatalf("expected record to keep selected node 10, got %#v", storedRecord)
}
calls := dispatcher.snapshot()
if len(calls) != 1 || calls[0].NodeID != 10 || calls[0].CmdType != model.AgentCommandTypeRunTask {
t.Fatalf("unexpected dispatcher calls: %#v", calls)
}
}
func TestBackupExecutionServiceRejectsDuplicateRunningTask(t *testing.T) {
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
startedAt := time.Now().UTC()
running := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: task.StorageTargetID,
NodeID: 0,
Status: model.BackupRecordStatusRunning,
StartedAt: startedAt,
}
if err := records.Create(ctx, running); err != nil {
t.Fatalf("Create running record returned error: %v", err)
}
_, err = executionService.RunTaskByIDSync(ctx, task.ID)
if err == nil || !strings.Contains(err.Error(), "正在运行") {
t.Fatalf("expected duplicate running task to be rejected, got %v", err)
}
items, err := records.List(ctx, repository.BackupRecordListOptions{Status: model.BackupRecordStatusRunning})
if err != nil {
t.Fatalf("List running records returned error: %v", err)
}
if len(items) != 1 || items[0].ID != running.ID {
t.Fatalf("expected only the original running record, got %#v", items)
}
}
func TestBackupExecutionServiceDeleteRecordDispatchesRemoteLocalDiskCleanup(t *testing.T) {
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
nodeRepo := &nodeRepoStub{nodes: []model.Node{
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
}}
dispatcher := &fakeDispatcher{}
executionService.SetClusterDependencies(nodeRepo, dispatcher)
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
completedAt := time.Now().UTC()
record := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: task.StorageTargetID,
NodeID: 10,
Status: model.BackupRecordStatusSuccess,
FileName: "remote.tar.gz",
StoragePath: "file/2026/05/09/remote.tar.gz",
StartedAt: completedAt.Add(-time.Second),
CompletedAt: &completedAt,
}
if err := records.Create(ctx, record); err != nil {
t.Fatalf("Create record returned error: %v", err)
}
if err := executionService.DeleteRecord(ctx, record.ID); err != nil {
t.Fatalf("DeleteRecord returned error: %v", err)
}
deleted, err := records.FindByID(ctx, record.ID)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if deleted != nil {
t.Fatalf("expected record deleted, got %#v", deleted)
}
calls := dispatcher.snapshot()
if len(calls) != 1 {
t.Fatalf("expected one dispatcher call, got %#v", calls)
}
if calls[0].NodeID != 10 || calls[0].CmdType != model.AgentCommandTypeDeleteStorageObject {
t.Fatalf("unexpected dispatcher call: %#v", calls[0])
}
if calls[0].Payload["storagePath"] != record.StoragePath {
t.Fatalf("expected storagePath %q, got %#v", record.StoragePath, calls[0].Payload)
}
if calls[0].Payload["targetType"] != string(storage.ProviderTypeLocalDisk) {
t.Fatalf("expected local_disk targetType, got %#v", calls[0].Payload)
}
if _, ok := calls[0].Payload["targetConfig"].(map[string]any); !ok {
t.Fatalf("expected targetConfig map, got %#v", calls[0].Payload["targetConfig"])
}
}
func TestBackupExecutionServiceRestoreRecordRejectsRemoteLocalDisk(t *testing.T) {
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
executionService.SetClusterDependencies(&nodeRepoStub{nodes: []model.Node{
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
}}, &fakeDispatcher{})
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
completedAt := time.Now().UTC()
record := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: task.StorageTargetID,
NodeID: 10,
Status: model.BackupRecordStatusSuccess,
FileName: "remote.tar.gz",
StoragePath: "file/2026/05/09/remote.tar.gz",
StartedAt: completedAt.Add(-time.Second),
CompletedAt: &completedAt,
}
if err := records.Create(ctx, record); err != nil {
t.Fatalf("Create record returned error: %v", err)
}
err = executionService.RestoreRecord(ctx, record.ID)
if err == nil {
t.Fatal("expected remote local_disk restore to be rejected")
}
if !strings.Contains(err.Error(), "Master 无法跨节点访问") {
t.Fatalf("expected cross-node local_disk error, got %v", err)
}
}
func TestBackupExecutionServiceDownloadsMasterRelayedLocalDiskRecord(t *testing.T) {
executionService, _, tasks, _, records, _, storageDir := newExecutionTestServices(t)
ctx := context.Background()
executionService.SetClusterDependencies(&nodeRepoStub{nodes: []model.Node{
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
}}, &fakeDispatcher{})
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
storagePath := "file/2026/05/09/relayed.tar"
artifactPath := filepath.Join(storageDir, filepath.FromSlash(storagePath))
if err := os.MkdirAll(filepath.Dir(artifactPath), 0o755); err != nil {
t.Fatalf("MkdirAll artifact parent returned error: %v", err)
}
content := []byte("stored on Master")
if err := os.WriteFile(artifactPath, content, 0o600); err != nil {
t.Fatalf("WriteFile artifact returned error: %v", err)
}
completedAt := time.Now().UTC()
record := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: task.StorageTargetID,
NodeID: 10,
Status: model.BackupRecordStatusSuccess,
FileName: "relayed.tar",
FileSize: int64(len(content)),
StoragePath: storagePath,
StorageTransferMode: storage.TransferModeMasterRelay,
StartedAt: completedAt.Add(-time.Second),
CompletedAt: &completedAt,
}
if err := records.Create(ctx, record); err != nil {
t.Fatalf("Create record returned error: %v", err)
}
download, err := executionService.DownloadRecord(ctx, record.ID)
if err != nil {
t.Fatalf("DownloadRecord returned error: %v", err)
}
got, readErr := io.ReadAll(download.Reader)
closeErr := download.Reader.Close()
if readErr != nil || closeErr != nil {
t.Fatalf("read relayed artifact: read=%v close=%v", readErr, closeErr)
}
if !bytes.Equal(got, content) {
t.Fatalf("downloaded content = %q, want %q", got, content)
}
}
func TestBackupExecutionServiceRecordsFirstSuccessfulStorageTarget(t *testing.T) {
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
second := &testStorageProvider{name: "second", objects: map[string][]byte{}}
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
"second": second,
}})
cipher := codec.NewConfigCipher("execution-secret")
firstConfig, err := cipher.EncryptJSON(map[string]any{"name": "missing"})
if err != nil {
t.Fatalf("EncryptJSON first returned error: %v", err)
}
secondConfig, err := cipher.EncryptJSON(map[string]any{"name": "second"})
if err != nil {
t.Fatalf("EncryptJSON second returned error: %v", err)
}
if err := targets.Create(ctx, &model.StorageTarget{Name: "first", Type: "test_storage", Enabled: true, ConfigCiphertext: firstConfig, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
t.Fatalf("Create first target returned error: %v", err)
}
if err := targets.Create(ctx, &model.StorageTarget{Name: "second", Type: "test_storage", Enabled: true, ConfigCiphertext: secondConfig, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
t.Fatalf("Create second target returned error: %v", err)
}
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
task.StorageTargetID = 2
task.StorageTargets = []model.StorageTarget{{ID: 2}, {ID: 3}}
if err := tasks.Update(ctx, task); err != nil {
t.Fatalf("Update task returned error: %v", err)
}
detail, err := executionService.RunTaskByIDSync(ctx, 1)
if err != nil {
t.Fatalf("RunTaskByIDSync returned error: %v", err)
}
if detail.Status != model.BackupRecordStatusSuccess {
t.Fatalf("expected success, got %#v", detail)
}
storedRecord, err := records.FindByID(ctx, detail.ID)
if err != nil {
t.Fatalf("FindByID record returned error: %v", err)
}
if storedRecord.StorageTargetID != 3 {
t.Fatalf("expected record StorageTargetID to point at successful target 3, got %d", storedRecord.StorageTargetID)
}
if _, ok := second.objects[storedRecord.StoragePath]; !ok {
t.Fatalf("expected object in successful provider at %q", storedRecord.StoragePath)
}
}
func TestBackupExecutionServiceUploadRetryStopsWhenContextCancelled(t *testing.T) {
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
ctx, cancel := context.WithCancel(context.Background())
var cancelOnce sync.Once
failing := &testStorageProvider{
name: "failing",
failUpload: true,
onUpload: func() {
cancelOnce.Do(cancel)
},
}
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
"failing": failing,
}})
cipher := codec.NewConfigCipher("execution-secret")
failingConfig, err := cipher.EncryptJSON(map[string]any{"name": "failing"})
if err != nil {
t.Fatalf("EncryptJSON returned error: %v", err)
}
if err := targets.Update(ctx, &model.StorageTarget{
ID: 1,
Name: "local",
Type: "test_storage",
Enabled: true,
ConfigCiphertext: failingConfig,
ConfigVersion: 1,
LastTestStatus: "unknown",
}); err != nil {
t.Fatalf("Update target returned error: %v", err)
}
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
startedAt := time.Now().UTC()
record := &model.BackupRecord{
TaskID: task.ID,
StorageTargetID: task.StorageTargetID,
Status: model.BackupRecordStatusRunning,
StartedAt: startedAt,
}
if err := records.Create(ctx, record); err != nil {
t.Fatalf("Create record returned error: %v", err)
}
done := make(chan struct{})
go func() {
executionService.executeTask(ctx, task, record.ID, startedAt)
close(done)
}()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("expected cancelled upload retry to stop without waiting for backoff sleep")
}
}
func TestBackupExecutionServiceReadsStorageUsageOnceForMultiTargetQuotaChecks(t *testing.T) {
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
first := &testStorageProvider{name: "first", objects: map[string][]byte{}}
second := &testStorageProvider{name: "second", objects: map[string][]byte{}}
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
"first": first,
"second": second,
}})
cipher := codec.NewConfigCipher("execution-secret")
firstConfig, err := cipher.EncryptJSON(map[string]any{"name": "first"})
if err != nil {
t.Fatalf("EncryptJSON first returned error: %v", err)
}
secondConfig, err := cipher.EncryptJSON(map[string]any{"name": "second"})
if err != nil {
t.Fatalf("EncryptJSON second returned error: %v", err)
}
if err := targets.Update(ctx, &model.StorageTarget{ID: 1, Name: "local", Type: "test_storage", Enabled: true, ConfigCiphertext: firstConfig, ConfigVersion: 1, LastTestStatus: "unknown", QuotaBytes: 1 << 30}); err != nil {
t.Fatalf("Update first target returned error: %v", err)
}
if err := targets.Create(ctx, &model.StorageTarget{Name: "second", Type: "test_storage", Enabled: true, ConfigCiphertext: secondConfig, ConfigVersion: 1, LastTestStatus: "unknown", QuotaBytes: 1 << 30}); err != nil {
t.Fatalf("Create second target returned error: %v", err)
}
task, err := tasks.FindByID(ctx, 1)
if err != nil {
t.Fatalf("FindByID task returned error: %v", err)
}
task.StorageTargets = []model.StorageTarget{{ID: 1}, {ID: 2}}
if err := tasks.Update(ctx, task); err != nil {
t.Fatalf("Update task returned error: %v", err)
}
executionService.records = &storageUsageCountingRecordRepo{BackupRecordRepository: records}
detail, err := executionService.RunTaskByIDSync(ctx, task.ID)
if err != nil {
t.Fatalf("RunTaskByIDSync returned error: %v", err)
}
if detail.Status != model.BackupRecordStatusSuccess {
t.Fatalf("expected success, got %#v", detail)
}
countingRepo := executionService.records.(*storageUsageCountingRecordRepo)
if countingRepo.usageCalls != 1 {
t.Fatalf("expected StorageUsage to be called once for quota snapshot, got %d", countingRepo.usageCalls)
}
if len(first.objects) != 1 || len(second.objects) != 1 {
t.Fatalf("expected both targets to receive upload, got first=%d second=%d", len(first.objects), len(second.objects))
}
}
func TestBackupExecutionServiceContinuesWhenStorageUsageSnapshotFails(t *testing.T) {
executionService, _, _, targets, records, _, _ := newExecutionTestServices(t)
ctx := context.Background()
provider := &testStorageProvider{name: "primary", objects: map[string][]byte{}}
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
"primary": provider,
}})
cipher := codec.NewConfigCipher("execution-secret")
configCiphertext, err := cipher.EncryptJSON(map[string]any{"name": "primary"})
if err != nil {
t.Fatalf("EncryptJSON returned error: %v", err)
}
if err := targets.Update(ctx, &model.StorageTarget{
ID: 1,
Name: "local",
Type: "test_storage",
Enabled: true,
ConfigCiphertext: configCiphertext,
ConfigVersion: 1,
LastTestStatus: "unknown",
QuotaBytes: 1 << 30,
}); err != nil {
t.Fatalf("Update target returned error: %v", err)
}
executionService.records = &storageUsageFailingRecordRepo{
BackupRecordRepository: records,
err: errStorageUsageFailed,
}
detail, err := executionService.RunTaskByIDSync(ctx, 1)
if err != nil {
t.Fatalf("RunTaskByIDSync returned error: %v", err)
}
if detail.Status != model.BackupRecordStatusSuccess {
t.Fatalf("expected success despite soft quota usage snapshot error, got %#v", detail)
}
if len(provider.objects) != 1 {
t.Fatalf("expected upload to proceed, got %d uploaded objects", len(provider.objects))
}
}
func TestBackupRecordServiceRestore(t *testing.T) {
executionService, recordService, _, _, _, sourceDir, _ := newExecutionTestServices(t)
detail, err := executionService.RunTaskByIDSync(context.Background(), 1)
@@ -699,27 +105,3 @@ func TestBackupRecordServiceRestore(t *testing.T) {
t.Fatalf("unexpected restored content: %s", string(content))
}
}
type storageUsageCountingRecordRepo struct {
repository.BackupRecordRepository
mu sync.Mutex
usageCalls int
}
func (r *storageUsageCountingRecordRepo) StorageUsage(ctx context.Context) ([]repository.BackupStorageUsageItem, error) {
r.mu.Lock()
r.usageCalls++
r.mu.Unlock()
return r.BackupRecordRepository.StorageUsage(ctx)
}
type storageUsageFailingRecordRepo struct {
repository.BackupRecordRepository
err error
}
func (r *storageUsageFailingRecordRepo) StorageUsage(context.Context) ([]repository.BackupStorageUsageItem, error) {
return nil, r.err
}
var errStorageUsageFailed = errors.New("storage usage failed")

View File

@@ -1,104 +0,0 @@
package service
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"backupx/server/internal/backup"
"backupx/server/internal/config"
"backupx/server/internal/database"
"backupx/server/internal/logger"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
storageRclone "backupx/server/internal/storage/rclone"
)
func newLockTestHarness(t *testing.T) (*BackupRecordService, *BackupExecutionService) {
t.Helper()
baseDir := t.TempDir()
sourceDir := filepath.Join(baseDir, "data")
storeDir := filepath.Join(baseDir, "store")
if err := os.MkdirAll(sourceDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(sourceDir, "f.txt"), []byte("lock-data"), 0o644); err != nil {
t.Fatal(err)
}
log, err := logger.New(config.LogConfig{Level: "error"})
if err != nil {
t.Fatal(err)
}
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(baseDir, "backupx.db")}, log)
if err != nil {
t.Fatal(err)
}
cipher := codec.NewConfigCipher("lock-secret")
targets := repository.NewStorageTargetRepository(db)
tasks := repository.NewBackupTaskRepository(db)
records := repository.NewBackupRecordRepository(db)
cfg, err := cipher.EncryptJSON(map[string]any{"basePath": storeDir})
if err != nil {
t.Fatal(err)
}
if err := targets.Create(context.Background(), &model.StorageTarget{Name: "s", Type: string(storage.ProviderTypeLocalDisk), Enabled: true, ConfigCiphertext: cfg, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
t.Fatal(err)
}
task := &model.BackupTask{Name: "lock-task", Type: "file", Enabled: true, SourcePath: sourceDir, StorageTargetID: 1, NodeID: 0, RetentionDays: 30, Compression: "gzip", MaxBackups: 10, LastStatus: "idle"}
if err := tasks.Create(context.Background(), task); err != nil {
t.Fatal(err)
}
logHub := backup.NewLogHub()
runnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil))
storageRegistry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
execution := NewBackupExecutionService(tasks, records, targets, storageRegistry, runnerRegistry, logHub, nil, cipher, nil, baseDir, 2, 10, "")
recordService := NewBackupRecordService(records, execution, logHub)
return recordService, execution
}
// TestBackupRecordLock_BlocksDeletion 验证保留锁定后手动删除被拒绝,解锁后可删除。
func TestBackupRecordLock_BlocksDeletion(t *testing.T) {
recordService, execution := newLockTestHarness(t)
ctx := context.Background()
bd, err := execution.RunTaskByIDSync(ctx, 1)
if err != nil {
t.Fatalf("RunTaskByIDSync: %v", err)
}
if bd.Status != "success" {
t.Fatalf("backup not success: %s", bd.Status)
}
// 锁定。
detail, err := recordService.SetLock(ctx, bd.ID, true)
if err != nil {
t.Fatalf("SetLock(true): %v", err)
}
if !detail.Locked {
t.Fatal("expected detail.Locked = true")
}
// 锁定状态下删除应被拒绝。
if err := execution.DeleteRecord(ctx, bd.ID); err == nil {
t.Fatal("expected delete of locked record to be rejected")
} else if !strings.Contains(err.Error(), "保留锁定") {
t.Fatalf("unexpected delete error: %v", err)
}
// 记录仍然存在。
if got, _ := recordService.Get(ctx, bd.ID); got == nil {
t.Fatal("locked record must still exist after rejected delete")
}
// 解锁后可删除。
if _, err := recordService.SetLock(ctx, bd.ID, false); err != nil {
t.Fatalf("SetLock(false): %v", err)
}
if err := execution.DeleteRecord(ctx, bd.ID); err != nil {
t.Fatalf("delete after unlock should succeed: %v", err)
}
}

View File

@@ -3,7 +3,6 @@ package service
import (
"context"
"encoding/json"
"sort"
"strings"
"time"
@@ -23,30 +22,27 @@ type BackupRecordListInput struct {
}
type BackupRecordSummary struct {
ID uint `json:"id"`
TaskID uint `json:"taskId"`
TaskName string `json:"taskName"`
StorageTargetID uint `json:"storageTargetId"`
StorageTargetName string `json:"storageTargetName"`
Status string `json:"status"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
Checksum string `json:"checksum"`
StoragePath string `json:"storagePath"`
StorageTransferMode string `json:"storageTransferMode,omitempty"`
DurationSeconds int `json:"durationSeconds"`
ErrorMessage string `json:"errorMessage"`
StartedAt time.Time `json:"startedAt"`
CompletedAt *time.Time `json:"completedAt,omitempty"`
Locked bool `json:"locked"`
BackupKind string `json:"backupKind"`
ID uint `json:"id"`
TaskID uint `json:"taskId"`
TaskName string `json:"taskName"`
StorageTargetID uint `json:"storageTargetId"`
StorageTargetName string `json:"storageTargetName"`
Status string `json:"status"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
Checksum string `json:"checksum"`
StoragePath string `json:"storagePath"`
DurationSeconds int `json:"durationSeconds"`
ErrorMessage string `json:"errorMessage"`
StartedAt time.Time `json:"startedAt"`
CompletedAt *time.Time `json:"completedAt,omitempty"`
}
type BackupRecordDetail struct {
BackupRecordSummary
LogContent string `json:"logContent"`
LogEvents []backup.LogEvent `json:"logEvents,omitempty"`
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
}
type BackupRecordService struct {
@@ -82,64 +78,6 @@ func (s *BackupRecordService) Get(ctx context.Context, id uint) (*BackupRecordDe
return toBackupRecordDetail(item, s.logHub), nil
}
// BackupContentEntry 描述备份内单个条目(文件或目录),用于内容浏览。
type BackupContentEntry struct {
Path string `json:"path"`
Size int64 `json:"size"`
IsDir bool `json:"isDir"`
}
// BackupRecordContents 是一次备份的内容清单视图。
type BackupRecordContents struct {
RecordID uint `json:"recordId"`
Total int `json:"total"`
Truncated bool `json:"truncated"`
BasedOnFull uint `json:"basedOnFull,omitempty"` // 差异记录时,清单取自该基线全量
Entries []BackupContentEntry `json:"entries"`
}
const backupContentsMaxEntries = 10000
// ListContents 返回某备份记录的文件清单(仅文件类型的新全量备份会记录清单)。
// 差异记录回退到其基线全量的清单,近似展示恢复后的目录结构。无清单时返回明确错误。
func (s *BackupRecordService) ListContents(ctx context.Context, id uint) (*BackupRecordContents, error) {
item, err := s.records.FindByID(ctx, id)
if err != nil {
return nil, apperror.Internal("BACKUP_RECORD_GET_FAILED", "无法获取备份记录", err)
}
if item == nil {
return nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", nil)
}
manifestJSON := item.Manifest
basedOnFull := uint(0)
if strings.TrimSpace(manifestJSON) == "" && item.BaseRecordID != 0 {
if base, baseErr := s.records.FindByID(ctx, item.BaseRecordID); baseErr == nil && base != nil {
manifestJSON = base.Manifest
basedOnFull = base.ID
}
}
if strings.TrimSpace(manifestJSON) == "" {
return nil, apperror.New(422, "BACKUP_CONTENTS_UNAVAILABLE", "该备份未记录文件清单(仅文件类型的新全量备份支持内容浏览),请重新执行一次全量备份后再试。", nil)
}
manifest, decErr := backup.DecodeManifest([]byte(manifestJSON))
if decErr != nil {
return nil, apperror.Internal("BACKUP_CONTENTS_DECODE_FAILED", "解析备份清单失败", decErr)
}
entries := manifest.Entries
sort.Slice(entries, func(i, j int) bool { return entries[i].Path < entries[j].Path })
total := len(entries)
truncated := false
if total > backupContentsMaxEntries {
entries = entries[:backupContentsMaxEntries]
truncated = true
}
result := &BackupRecordContents{RecordID: item.ID, Total: total, Truncated: truncated, BasedOnFull: basedOnFull, Entries: make([]BackupContentEntry, 0, len(entries))}
for _, e := range entries {
result.Entries = append(result.Entries, BackupContentEntry{Path: e.Path, Size: e.Size, IsDir: e.IsDir})
}
return result, nil
}
func (s *BackupRecordService) SubscribeLogs(ctx context.Context, id uint, buffer int) (<-chan backup.LogEvent, func(), error) {
item, err := s.records.FindByID(ctx, id)
if err != nil {
@@ -164,44 +102,22 @@ func (s *BackupRecordService) Delete(ctx context.Context, id uint) error {
return s.execution.DeleteRecord(ctx, id)
}
// SetLock 设置或解除备份记录的保留锁定(法律保留)。
// 锁定后该记录免于保留期/数量自动清理,且禁止手动删除,直至显式解锁。
func (s *BackupRecordService) SetLock(ctx context.Context, id uint, locked bool) (*BackupRecordDetail, error) {
item, err := s.records.FindByID(ctx, id)
if err != nil {
return nil, apperror.Internal("BACKUP_RECORD_GET_FAILED", "无法获取备份记录详情", err)
}
if item == nil {
return nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", nil)
}
if item.Locked != locked {
item.Locked = locked
if err := s.records.Update(ctx, item); err != nil {
return nil, apperror.Internal("BACKUP_RECORD_LOCK_FAILED", "无法更新备份锁定状态", err)
}
}
return toBackupRecordDetail(item, s.logHub), nil
}
func toBackupRecordSummary(item *model.BackupRecord) BackupRecordSummary {
return BackupRecordSummary{
ID: item.ID,
TaskID: item.TaskID,
TaskName: item.Task.Name,
StorageTargetID: item.StorageTargetID,
StorageTargetName: item.StorageTarget.Name,
Status: item.Status,
FileName: item.FileName,
FileSize: item.FileSize,
Checksum: item.Checksum,
StoragePath: item.StoragePath,
StorageTransferMode: item.StorageTransferMode,
DurationSeconds: item.DurationSeconds,
ErrorMessage: item.ErrorMessage,
StartedAt: item.StartedAt,
CompletedAt: item.CompletedAt,
Locked: item.Locked,
BackupKind: item.BackupKind,
ID: item.ID,
TaskID: item.TaskID,
TaskName: item.Task.Name,
StorageTargetID: item.StorageTargetID,
StorageTargetName: item.StorageTarget.Name,
Status: item.Status,
FileName: item.FileName,
FileSize: item.FileSize,
Checksum: item.Checksum,
StoragePath: item.StoragePath,
DurationSeconds: item.DurationSeconds,
ErrorMessage: item.ErrorMessage,
StartedAt: item.StartedAt,
CompletedAt: item.CompletedAt,
}
}

View File

@@ -21,7 +21,7 @@ const backupTaskMaskedValue = "********"
type BackupTaskUpsertInput struct {
Name string `json:"name" binding:"required,min=1,max=100"`
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql saphana mongodb"`
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql saphana"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr" binding:"max=64"`
SourcePath string `json:"sourcePath" binding:"max=500"`
@@ -33,16 +33,16 @@ type BackupTaskUpsertInput struct {
DBPassword string `json:"dbPassword" binding:"max=255"`
DBName string `json:"dbName" binding:"max=255"`
DBPath string `json:"dbPath" binding:"max=500"`
StorageTargetID uint `json:"storageTargetId"` // deprecated: 向后兼容
StorageTargetIDs []uint `json:"storageTargetIds"` // 新增:多存储目标
NodeID uint `json:"nodeId"` // 执行节点0 = 本机 Master 或节点池)
StorageTargetID uint `json:"storageTargetId"` // deprecated: 向后兼容
StorageTargetIDs []uint `json:"storageTargetIds"` // 新增:多存储目标
NodeID uint `json:"nodeId"` // 执行节点0 = 本机 Master 或节点池)
// NodePoolTag 节点池标签。NodeID=0 且本字段非空时,调度器动态从 Labels 命中的在线节点中选负载最低者。
NodePoolTag string `json:"nodePoolTag" binding:"max=64"`
Tags string `json:"tags" binding:"max=500"` // 逗号分隔标签
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression" binding:"omitempty,oneof=gzip zstd none"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
NodePoolTag string `json:"nodePoolTag" binding:"max=64"`
Tags string `json:"tags" binding:"max=500"` // 逗号分隔标签
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression" binding:"omitempty,oneof=gzip none"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
// ExtraConfig 类型特有扩展配置(如 SAP HANA 的 backupLevel/backupChannels
ExtraConfig map[string]any `json:"extraConfig"`
// 验证(恢复演练)配置
@@ -52,14 +52,6 @@ type BackupTaskUpsertInput struct {
// SLA 配置
SLAHoursRPO int `json:"slaHoursRpo"`
AlertOnConsecutiveFails int `json:"alertOnConsecutiveFails"`
// GFS 分层保留(任一 > 0 启用,取代 RetentionDays/MaxBackups
KeepDaily int `json:"keepDaily"`
KeepWeekly int `json:"keepWeekly"`
KeepMonthly int `json:"keepMonthly"`
KeepYearly int `json:"keepYearly"`
// BackupMode 备份模式full默认/ differential差异归档/ repositoryCDC 去重仓库)
BackupMode string `json:"backupMode" binding:"omitempty,oneof=full differential repository"`
DiffFullIntervalDays int `json:"diffFullIntervalDays"`
// 备份复制目标存储 ID 列表3-2-1 规则)
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
// 维护窗口CSV详见 backup/window.go
@@ -78,8 +70,8 @@ type BackupTaskSummary struct {
Type string `json:"type"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr"`
StorageTargetID uint `json:"storageTargetId"` // deprecated: 取第一个
StorageTargetName string `json:"storageTargetName"` // deprecated: 取第一个
StorageTargetID uint `json:"storageTargetId"` // deprecated: 取第一个
StorageTargetName string `json:"storageTargetName"` // deprecated: 取第一个
StorageTargetIDs []uint `json:"storageTargetIds"`
StorageTargetNames []string `json:"storageTargetNames"`
NodeID uint `json:"nodeId"`
@@ -98,17 +90,11 @@ type BackupTaskSummary struct {
VerifyMode string `json:"verifyMode"`
SLAHoursRPO int `json:"slaHoursRpo"`
AlertOnConsecutiveFails int `json:"alertOnConsecutiveFails"`
KeepDaily int `json:"keepDaily"`
KeepWeekly int `json:"keepWeekly"`
KeepMonthly int `json:"keepMonthly"`
KeepYearly int `json:"keepYearly"`
BackupMode string `json:"backupMode"`
DiffFullIntervalDays int `json:"diffFullIntervalDays"`
// 备份复制目标3-2-1
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
MaintenanceWindows string `json:"maintenanceWindows"`
DependsOnTaskIDs []uint `json:"dependsOnTaskIds"`
UpdatedAt time.Time `json:"updatedAt"`
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
MaintenanceWindows string `json:"maintenanceWindows"`
DependsOnTaskIDs []uint `json:"dependsOnTaskIds"`
UpdatedAt time.Time `json:"updatedAt"`
}
type BackupTaskDetail struct {
@@ -414,50 +400,21 @@ func (s *BackupTaskService) cleanupRemoteFiles(ctx context.Context, taskID uint)
recordCount = len(records)
// 缓存 provider 避免同一存储目标重复创建连接
providerCache := make(map[uint]storage.StorageProvider)
repositoryProviders := make(map[uint]storage.StorageProvider)
for _, record := range records {
copies := []StorageUploadResultItem{{
StorageTargetID: record.StorageTargetID,
Status: model.BackupRecordStatusSuccess,
StoragePath: record.StoragePath,
}}
if strings.TrimSpace(record.StorageUploadResults) != "" {
var storedCopies []StorageUploadResultItem
if unmarshalErr := json.Unmarshal([]byte(record.StorageUploadResults), &storedCopies); unmarshalErr == nil {
copies = storedCopies
}
}
seenTargets := make(map[uint]struct{}, len(copies))
for _, copy := range copies {
if !strings.EqualFold(copy.Status, model.BackupRecordStatusSuccess) || strings.TrimSpace(copy.StoragePath) == "" {
continue
}
if _, seen := seenTargets[copy.StorageTargetID]; seen {
continue
}
seenTargets[copy.StorageTargetID] = struct{}{}
provider, ok := providerCache[copy.StorageTargetID]
if !ok {
provider, err = s.resolveStorageProvider(ctx, copy.StorageTargetID)
if err != nil {
continue
}
providerCache[copy.StorageTargetID] = provider
}
if err := provider.Delete(ctx, copy.StoragePath); err == nil {
cleanedFiles++
if record.BackupKind == model.BackupKindRepository {
repositoryProviders[copy.StorageTargetID] = provider
}
}
}
}
for _, provider := range repositoryProviders {
pruned, pruneErr := backup.NewRepositoryStore(s.cipher.Key()).Prune(ctx, provider)
if pruneErr != nil {
if strings.TrimSpace(record.StoragePath) == "" {
continue
}
cleanedFiles += pruned.DeletedIndexes + pruned.DeletedPacks
provider, ok := providerCache[record.StorageTargetID]
if !ok {
provider, err = s.resolveStorageProvider(ctx, record.StorageTargetID)
if err != nil {
continue
}
providerCache[record.StorageTargetID] = provider
}
if err := provider.Delete(ctx, record.StoragePath); err == nil {
cleanedFiles++
}
}
return recordCount, cleanedFiles
}
@@ -531,7 +488,6 @@ func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.B
return apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", fmt.Sprintf("关联的存储目标 %d 不存在", tid), nil)
}
}
var fixedNode *model.Node
if input.NodeID > 0 && s.nodes != nil {
node, err := s.nodes.FindByID(ctx, input.NodeID)
if err != nil {
@@ -540,36 +496,12 @@ func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.B
if node == nil {
return apperror.BadRequest("BACKUP_TASK_INVALID", "所选执行节点不存在", nil)
}
fixedNode = node
}
// 节点池与固定节点互斥:固定节点已确定执行位置,不再动态调度
if input.NodeID > 0 && strings.TrimSpace(input.NodePoolTag) != "" {
return apperror.BadRequest("BACKUP_TASK_INVALID",
"固定执行节点与节点池标签只能选其一", nil)
}
if input.Encrypt && (strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal)) {
return apperror.BadRequest("BACKUP_TASK_REMOTE_ENCRYPT_UNSUPPORTED",
"远程节点暂不支持加密备份。请关闭加密,或将任务固定在 Master 本机执行。", nil)
}
if strings.EqualFold(strings.TrimSpace(input.BackupMode), model.BackupModeDifferential) {
if input.Type != model.BackupTaskTypeFile {
return apperror.BadRequest("BACKUP_TASK_DIFF_UNSUPPORTED", "差异备份仅支持文件目录类型任务", nil)
}
if strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal) {
return apperror.BadRequest("BACKUP_TASK_DIFF_REMOTE_UNSUPPORTED", "差异备份当前仅支持本机 Master 执行,请将任务固定在本机或改用全量备份。", nil)
}
}
if strings.EqualFold(strings.TrimSpace(input.BackupMode), model.BackupModeRepository) {
if input.Type != model.BackupTaskTypeFile {
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_UNSUPPORTED", "CDC 仓库模式仅支持文件目录类型任务", nil)
}
if strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal) {
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_REMOTE_UNSUPPORTED", "CDC 仓库模式当前采用单写者索引,仅支持 Master 本机执行。远程服务器备份请暂用全量模式。", nil)
}
if len(input.ReplicationTargetIDs) > 0 {
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_REPLICATION_UNSUPPORTED", "CDC 仓库快照不能使用对象级复制;请直接为任务选择多个存储目标以生成完整仓库副本。", nil)
}
}
if input.RetentionDays < 0 {
return apperror.BadRequest("BACKUP_TASK_INVALID", "保留天数不能小于 0", nil)
}
@@ -631,7 +563,7 @@ func validateTaskTypeSpecificFields(input BackupTaskUpsertInput, passwordRequire
if !hasSourcePaths {
return apperror.BadRequest("BACKUP_TASK_INVALID", "文件备份必须填写源路径", nil)
}
case "mysql", "postgresql", "saphana", "mongodb":
case "mysql", "postgresql", "saphana":
if strings.TrimSpace(input.DBHost) == "" {
return apperror.BadRequest("BACKUP_TASK_INVALID", "数据库主机不能为空", nil)
}
@@ -707,44 +639,38 @@ func (s *BackupTaskService) buildTask(existing *model.BackupTask, input BackupTa
return nil, apperror.BadRequest("BACKUP_TASK_INVALID", "扩展配置格式不合法", err)
}
item := &model.BackupTask{
Name: strings.TrimSpace(input.Name),
Type: normalizeBackupTaskType(input.Type),
Enabled: input.Enabled,
CronExpr: strings.TrimSpace(input.CronExpr),
SourcePath: primarySourcePath,
SourcePaths: sourcePathsJSON,
ExcludePatterns: excludePatterns,
DBHost: strings.TrimSpace(input.DBHost),
DBPort: input.DBPort,
DBUser: strings.TrimSpace(input.DBUser),
DBPasswordCiphertext: passwordCiphertext,
DBName: strings.TrimSpace(input.DBName),
DBPath: strings.TrimSpace(input.DBPath),
ExtraConfig: extraConfigJSON,
StorageTargetID: primaryTargetID,
StorageTargets: storageTargets,
NodeID: input.NodeID,
NodePoolTag: strings.TrimSpace(input.NodePoolTag),
Tags: strings.TrimSpace(input.Tags),
RetentionDays: input.RetentionDays,
Compression: compression,
Encrypt: input.Encrypt,
MaxBackups: maxBackups,
LastStatus: "idle",
VerifyEnabled: input.VerifyEnabled,
VerifyCronExpr: strings.TrimSpace(input.VerifyCronExpr),
VerifyMode: normalizeVerifyMode(input.VerifyMode),
SLAHoursRPO: maxInt(0, input.SLAHoursRPO),
Name: strings.TrimSpace(input.Name),
Type: normalizeBackupTaskType(input.Type),
Enabled: input.Enabled,
CronExpr: strings.TrimSpace(input.CronExpr),
SourcePath: primarySourcePath,
SourcePaths: sourcePathsJSON,
ExcludePatterns: excludePatterns,
DBHost: strings.TrimSpace(input.DBHost),
DBPort: input.DBPort,
DBUser: strings.TrimSpace(input.DBUser),
DBPasswordCiphertext: passwordCiphertext,
DBName: strings.TrimSpace(input.DBName),
DBPath: strings.TrimSpace(input.DBPath),
ExtraConfig: extraConfigJSON,
StorageTargetID: primaryTargetID,
StorageTargets: storageTargets,
NodeID: input.NodeID,
NodePoolTag: strings.TrimSpace(input.NodePoolTag),
Tags: strings.TrimSpace(input.Tags),
RetentionDays: input.RetentionDays,
Compression: compression,
Encrypt: input.Encrypt,
MaxBackups: maxBackups,
LastStatus: "idle",
VerifyEnabled: input.VerifyEnabled,
VerifyCronExpr: strings.TrimSpace(input.VerifyCronExpr),
VerifyMode: normalizeVerifyMode(input.VerifyMode),
SLAHoursRPO: maxInt(0, input.SLAHoursRPO),
AlertOnConsecutiveFails: alertThreshold(input.AlertOnConsecutiveFails),
KeepDaily: maxInt(0, input.KeepDaily),
KeepWeekly: maxInt(0, input.KeepWeekly),
KeepMonthly: maxInt(0, input.KeepMonthly),
KeepYearly: maxInt(0, input.KeepYearly),
BackupMode: normalizeBackupMode(input.BackupMode, input.Type),
DiffFullIntervalDays: diffFullInterval(input.DiffFullIntervalDays),
ReplicationTargetIDs: encodeUintCSV(input.ReplicationTargetIDs),
MaintenanceWindows: strings.TrimSpace(input.MaintenanceWindows),
DependsOnTaskIDs: encodeUintCSV(input.DependsOnTaskIDs),
ReplicationTargetIDs: encodeUintCSV(input.ReplicationTargetIDs),
MaintenanceWindows: strings.TrimSpace(input.MaintenanceWindows),
DependsOnTaskIDs: encodeUintCSV(input.DependsOnTaskIDs),
}
if existing != nil {
item.LastRunAt = existing.LastRunAt
@@ -810,40 +736,34 @@ func toBackupTaskSummary(item *model.BackupTask) BackupTaskSummary {
primaryName = targetNames[0]
}
return BackupTaskSummary{
ID: item.ID,
Name: item.Name,
Type: normalizeBackupTaskType(item.Type),
Enabled: item.Enabled,
CronExpr: item.CronExpr,
StorageTargetID: primaryID,
StorageTargetName: primaryName,
StorageTargetIDs: targetIDs,
StorageTargetNames: targetNames,
NodeID: item.NodeID,
NodeName: item.Node.Name,
NodePoolTag: item.NodePoolTag,
Tags: item.Tags,
RetentionDays: item.RetentionDays,
Compression: item.Compression,
Encrypt: item.Encrypt,
MaxBackups: item.MaxBackups,
LastRunAt: item.LastRunAt,
LastStatus: item.LastStatus,
ID: item.ID,
Name: item.Name,
Type: normalizeBackupTaskType(item.Type),
Enabled: item.Enabled,
CronExpr: item.CronExpr,
StorageTargetID: primaryID,
StorageTargetName: primaryName,
StorageTargetIDs: targetIDs,
StorageTargetNames: targetNames,
NodeID: item.NodeID,
NodeName: item.Node.Name,
NodePoolTag: item.NodePoolTag,
Tags: item.Tags,
RetentionDays: item.RetentionDays,
Compression: item.Compression,
Encrypt: item.Encrypt,
MaxBackups: item.MaxBackups,
LastRunAt: item.LastRunAt,
LastStatus: item.LastStatus,
VerifyEnabled: item.VerifyEnabled,
VerifyCronExpr: item.VerifyCronExpr,
VerifyMode: item.VerifyMode,
SLAHoursRPO: item.SLAHoursRPO,
AlertOnConsecutiveFails: item.AlertOnConsecutiveFails,
KeepDaily: item.KeepDaily,
KeepWeekly: item.KeepWeekly,
KeepMonthly: item.KeepMonthly,
KeepYearly: item.KeepYearly,
BackupMode: item.BackupMode,
DiffFullIntervalDays: item.DiffFullIntervalDays,
ReplicationTargetIDs: parseUintCSV(item.ReplicationTargetIDs),
MaintenanceWindows: item.MaintenanceWindows,
DependsOnTaskIDs: parseUintCSV(item.DependsOnTaskIDs),
UpdatedAt: item.UpdatedAt,
UpdatedAt: item.UpdatedAt,
}
}
@@ -975,29 +895,6 @@ func decodeExtraConfig(value string) (map[string]any, error) {
return result, nil
}
// normalizeBackupMode 归一化备份模式:仅文件类型可启用差异或 CDC 仓库,
// 其余一律全量(双保险,防绕过校验)。
func normalizeBackupMode(mode, taskType string) string {
if normalizeBackupTaskType(taskType) != model.BackupTaskTypeFile {
return model.BackupModeFull
}
switch strings.ToLower(strings.TrimSpace(mode)) {
case model.BackupModeDifferential:
return model.BackupModeDifferential
case model.BackupModeRepository:
return model.BackupModeRepository
}
return model.BackupModeFull
}
// diffFullInterval 归一化差异模式下的强制全量间隔(天),非正值回退默认 7。
func diffFullInterval(days int) int {
if days <= 0 {
return 7
}
return days
}
func normalizeBackupTaskType(value string) string {
normalized := strings.TrimSpace(strings.ToLower(value))
if normalized == "pgsql" {

View File

@@ -3,7 +3,6 @@ package service
import (
"context"
"path/filepath"
"strings"
"testing"
"backupx/server/internal/config"
@@ -30,82 +29,6 @@ func newBackupTaskServiceForTest(t *testing.T) (*BackupTaskService, repository.S
return service, targets, tasks
}
func TestBackupTaskServiceRejectsEncryptedRemoteTasks(t *testing.T) {
ctx := context.Background()
service, targets, _ := newBackupTaskServiceForTest(t)
service.SetNodeRepository(&nodeRepoStub{nodes: []model.Node{
{ID: 41, Name: "master", Token: "master-token", Status: model.NodeStatusOnline, IsLocal: true},
{ID: 42, Name: "edge", Token: "edge-token", Status: model.NodeStatusOnline, IsLocal: false},
}})
if err := targets.Create(ctx, &model.StorageTarget{Name: "local", Type: "local_disk", Enabled: true, ConfigCiphertext: "ciphertext", ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
t.Fatalf("seed storage target error: %v", err)
}
_, err := service.Create(ctx, BackupTaskUpsertInput{
Name: "encrypted-node-pool",
Type: "file",
Enabled: true,
SourcePath: "/srv/site",
StorageTargetID: 1,
NodePoolTag: "db",
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
Encrypt: true,
})
if err == nil || !strings.Contains(err.Error(), "远程节点暂不支持加密备份") {
t.Fatalf("expected encrypted node-pool task to be rejected, got %v", err)
}
created, err := service.Create(ctx, BackupTaskUpsertInput{
Name: "local-encrypted",
Type: "file",
Enabled: true,
SourcePath: "/srv/site",
StorageTargetID: 1,
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
Encrypt: true,
})
if err != nil {
t.Fatalf("Create local encrypted task returned error: %v", err)
}
localNodeTask, err := service.Create(ctx, BackupTaskUpsertInput{
Name: "local-node-encrypted",
Type: "file",
Enabled: true,
SourcePath: "/srv/site",
StorageTargetID: 1,
NodeID: 41,
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
Encrypt: true,
})
if err != nil {
t.Fatalf("Create encrypted task pinned to local node returned error: %v", err)
}
if localNodeTask.NodeID != 41 || !localNodeTask.Encrypt {
t.Fatalf("expected encrypted task to keep local node, got %#v", localNodeTask)
}
_, err = service.Update(ctx, created.ID, BackupTaskUpsertInput{
Name: created.Name,
Type: created.Type,
Enabled: true,
SourcePath: "/srv/site",
StorageTargetID: 1,
NodeID: 42,
RetentionDays: 30,
Compression: "gzip",
MaxBackups: 10,
Encrypt: true,
})
if err == nil || !strings.Contains(err.Error(), "远程节点暂不支持加密备份") {
t.Fatalf("expected encrypted fixed-node update to be rejected, got %v", err)
}
}
func TestBackupTaskServiceCreateAndGet(t *testing.T) {
ctx := context.Background()
service, targets, _ := newBackupTaskServiceForTest(t)

View File

@@ -1,219 +0,0 @@
package service
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"time"
"backupx/server/internal/apperror"
"backupx/server/internal/backup"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
"backupx/server/pkg/compress"
backupcrypto "backupx/server/pkg/crypto"
)
// 本文件集中放置「备份执行 / 恢复 / 验证 / 复制」四个执行服务共享的执行期辅助逻辑。
//
// 历史上这些函数(解密存储配置创建 provider、按后缀解密解压归档、判定远程节点、
// 跨节点 local_disk 保护、构建任务执行规格)在四个服务里各复制了一份,差异仅在
// 字段名与少量错误码/日志文案。重复实现既增加维护成本,也容易出现"改了一处忘了
// 另一处"的不一致缺陷。这里抽取为单一实现,各服务通过薄封装方法委托调用,调用方
// 无需改动。
// fileSHA256 计算文件内容的 SHA-256小写 hex与备份上传时记录到
// BackupRecord.Checksum 的格式一致,用于恢复/复制前的完整性校验。
func fileSHA256(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// verifyArtifactChecksum 校验下载到本地的备份对象与记录的 SHA-256 是否一致。
// expected 为空时跳过(兼容早期未记录 checksum 的备份);不一致返回结构化错误,
// 调用方应据此中止恢复,避免还原已损坏或被篡改的数据。
func verifyArtifactChecksum(path, expected string) error {
expected = strings.TrimSpace(expected)
if expected == "" {
return nil
}
actual, err := fileSHA256(path)
if err != nil {
return apperror.Internal("BACKUP_CHECKSUM_READ_FAILED", "无法读取备份文件计算校验和", err)
}
if !strings.EqualFold(actual, expected) {
// 包装错误同样使用中文并附上期望/实际哈希apperror.Error() 会优先返回包装错误,
// 而恢复记录的 ErrorMessage 取自 err.Error(),需保证对用户可读。
return apperror.BadRequest("BACKUP_CHECKSUM_MISMATCH",
"备份文件完整性校验失败SHA-256 不匹配,文件可能已损坏或被篡改",
fmt.Errorf("备份文件完整性校验失败SHA-256 不匹配(期望 %s实际 %s文件可能已损坏或被篡改", expected, actual))
}
return nil
}
// resolveStorageProvider 查询存储目标、解密其配置并创建 provider。
func resolveStorageProvider(ctx context.Context, targets repository.StorageTargetRepository, registry *storage.Registry, cipher *codec.ConfigCipher, targetID uint) (storage.StorageProvider, error) {
target, err := targets.FindByID(ctx, targetID)
if err != nil {
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
}
if target == nil {
return nil, apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "关联的存储目标不存在", nil)
}
configMap := map[string]any{}
if err := cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
}
return registry.Create(ctx, target.Type, configMap)
}
// prepareBackupArtifact 按文件后缀依次解密(.enc)与解压(.gz),返回最终可读路径。
// logger 可为 nil此时静默执行
func prepareBackupArtifact(cipher *codec.ConfigCipher, artifactPath string, logger *backup.ExecutionLogger) (string, error) {
current := artifactPath
if strings.HasSuffix(strings.ToLower(current), ".enc") {
if logger != nil {
logger.Infof("检测到加密后缀,开始解密")
}
decrypted, err := backupcrypto.DecryptFile(cipher.Key(), current)
if err != nil {
return "", err
}
current = decrypted
}
if strings.HasSuffix(strings.ToLower(current), ".gz") {
if logger != nil {
logger.Infof("检测到 gzip 压缩,开始解压")
}
decompressed, err := compress.GunzipFile(current)
if err != nil {
return "", err
}
current = decompressed
}
if strings.HasSuffix(strings.ToLower(current), ".zst") {
if logger != nil {
logger.Infof("检测到 zstd 压缩,开始解压")
}
decompressed, err := compress.UnzstdFile(current)
if err != nil {
return "", err
}
current = decompressed
}
return current, nil
}
// resolveRemoteExecutionNode 返回远程(非本机)节点指针,用于判定任务应下发给
// Agent 还是在 Master 本地执行。clusterEnabled 通常为「该服务是否注入了 Agent
// 下发能力」。本机 / 未启用集群 / nodeID=0 / 未找到时返回 nil走本地执行
func resolveRemoteExecutionNode(ctx context.Context, nodeRepo repository.NodeRepository, clusterEnabled bool, nodeID uint) *model.Node {
if nodeRepo == nil || !clusterEnabled || nodeID == 0 {
return nil
}
node, err := nodeRepo.FindByID(ctx, nodeID)
if err != nil || node == nil || node.IsLocal {
return nil
}
return node
}
// validateCrossNodeLocalDisk 跨节点 local_disk 保护:若备份记录归属某远程节点,
// 且其存储目标是 local_disk数据位于该节点本地磁盘Master 无法跨节点访问,
// 直接返回错误。errCode/opName 由各服务定制,以给出贴合场景的提示文案。
func validateCrossNodeLocalDisk(ctx context.Context, nodeRepo repository.NodeRepository, targets repository.StorageTargetRepository, record *model.BackupRecord, errCode, opName string) error {
if record == nil || record.NodeID == 0 || nodeRepo == nil {
return nil
}
// 中转模式的对象实际落在 Master 配置的本地磁盘Master 可以安全访问。
// 空值和 direct 均按旧版 Agent 本地落盘处理,保持升级兼容。
if record.StorageTransferMode == storage.TransferModeMasterRelay {
return nil
}
node, err := nodeRepo.FindByID(ctx, record.NodeID)
if err != nil || node == nil || node.IsLocal {
return nil
}
target, err := targets.FindByID(ctx, record.StorageTargetID)
if err != nil || target == nil {
return nil
}
if strings.EqualFold(target.Type, "local_disk") {
return apperror.BadRequest(errCode,
fmt.Sprintf("备份位于节点 %s 的本地磁盘local_diskMaster 无法跨节点%s。", node.Name, opName),
nil)
}
return nil
}
// buildBackupTaskSpec 由备份任务构建执行规格:解析排除规则/源路径、解密 DB 密码、
// 套用 ExtraConfigSAP HANA 等类型特有字段)。被备份执行与恢复服务共享。
func buildBackupTaskSpec(cipher *codec.ConfigCipher, task *model.BackupTask, startedAt time.Time, tempDir string) (backup.TaskSpec, error) {
excludePatterns := []string{}
if strings.TrimSpace(task.ExcludePatterns) != "" {
if err := json.Unmarshal([]byte(task.ExcludePatterns), &excludePatterns); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析排除规则", err)
}
}
password := ""
if strings.TrimSpace(task.DBPasswordCiphertext) != "" {
plain, err := cipher.Decrypt(task.DBPasswordCiphertext)
if err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECRYPT_FAILED", "无法解密数据库密码", err)
}
password = string(plain)
}
sourcePaths := []string{}
if strings.TrimSpace(task.SourcePaths) != "" {
if err := json.Unmarshal([]byte(task.SourcePaths), &sourcePaths); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
}
}
dbSpec := backup.DatabaseSpec{
Host: task.DBHost,
Port: task.DBPort,
User: task.DBUser,
Password: password,
Names: []string{task.DBName},
Path: task.DBPath,
}
// 解析 ExtraConfig 填充类型特有字段(目前主要用于 SAP HANA
if strings.TrimSpace(task.ExtraConfig) != "" {
extra := map[string]any{}
if err := json.Unmarshal([]byte(task.ExtraConfig), &extra); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
}
applyHANAExtraConfig(&dbSpec, extra)
}
return backup.TaskSpec{
ID: task.ID,
Name: task.Name,
Type: task.Type,
SourcePath: task.SourcePath,
SourcePaths: sourcePaths,
ExcludePatterns: excludePatterns,
StorageTargetID: task.StorageTargetID,
Compression: task.Compression,
Encrypt: task.Encrypt,
RetentionDays: task.RetentionDays,
MaxBackups: task.MaxBackups,
StartedAt: startedAt,
TempDir: tempDir,
Database: dbSpec,
}, nil
}

Some files were not shown because too many files have changed in this diff Show More