mirror of
https://github.com/Awuqing/BackupX.git
synced 2026-08-26 10:41:19 +08:00
Compare commits
6 Commits
codex/reso
...
feat/compl
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1e386e1205 | ||
|
|
6e7a884c64 | ||
|
|
0b2263086f | ||
|
|
2f494818cf | ||
|
|
7dfd12254b | ||
|
|
2997e971a6 |
4
.github/workflows/ci.yml
vendored
4
.github/workflows/ci.yml
vendored
@@ -6,10 +6,6 @@ on:
|
||||
pull_request:
|
||||
branches: [main, master]
|
||||
|
||||
# 最小权限:构建/测试仅需读取仓库内容,显式声明以收敛默认的可写令牌。
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend:
|
||||
name: Go Build & Test
|
||||
|
||||
5
.github/workflows/release.yml
vendored
5
.github/workflows/release.yml
vendored
@@ -116,15 +116,12 @@ jobs:
|
||||
fi
|
||||
cp deploy/nginx.conf "${ARCHIVE_NAME}/nginx.conf" 2>/dev/null || true
|
||||
tar czf "${ARCHIVE_NAME}.tar.gz" "${ARCHIVE_NAME}"
|
||||
cp "${ARCHIVE_NAME}.tar.gz" "backupx-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz"
|
||||
|
||||
- name: Upload to GitHub Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
tag_name: ${{ env.VERSION }}
|
||||
files: |
|
||||
backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
|
||||
backupx-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
|
||||
files: backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
|
||||
generate_release_notes: true
|
||||
|
||||
# ─── Job 3: Docker 多架构 → Docker Hub ───
|
||||
|
||||
106
CONTRIBUTING.md
106
CONTRIBUTING.md
@@ -1,106 +0,0 @@
|
||||
# Contributing to BackupX
|
||||
|
||||
感谢你对 BackupX 的关注!本指南介绍如何搭建开发环境并提交贡献。
|
||||
Thanks for your interest in contributing to BackupX! This guide covers how to set up your environment and submit changes.
|
||||
|
||||
## 开发环境 / Development Setup
|
||||
|
||||
### 依赖 / Prerequisites
|
||||
|
||||
- **Go** 1.25+(见 `server/go.mod`)
|
||||
- **Node.js** 20+(CI 与 Docker 使用 Node 20)
|
||||
- **npm** 9+
|
||||
|
||||
### 快速开始 / Quick Start
|
||||
|
||||
分别在两个终端启动前后端(后端 :8340,前端 Vite HMR):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
|
||||
# 终端 1 —— 后端(默认 http://localhost:8340)
|
||||
make dev-server
|
||||
|
||||
# 终端 2 —— 前端(Vite 热更新,/api 代理到 8340)
|
||||
make dev-web
|
||||
```
|
||||
|
||||
### 构建 / Building
|
||||
|
||||
```bash
|
||||
make build # 同时构建后端与前端
|
||||
make build-server # 仅后端 → server/bin/backupx
|
||||
make build-web # 仅前端 → web/dist
|
||||
make docker # 构建 Docker 镜像
|
||||
make docker-cn # 国内镜像源加速构建
|
||||
```
|
||||
|
||||
> 后端会自动托管 `web/dist`(或 `server.web_root` 指定目录),因此本地裸机部署无需额外的反向代理即可访问控制台。
|
||||
|
||||
## 测试 / Testing
|
||||
|
||||
提交前请确保测试通过:
|
||||
|
||||
```bash
|
||||
make test # 后端 + 前端全部测试
|
||||
make test-server # 仅后端:cd server && go test ./...
|
||||
make test-web # 仅前端:cd web && npm run test(vitest)
|
||||
```
|
||||
|
||||
新增功能或修复缺陷时,请尽量补充对应测试。
|
||||
|
||||
## 提交信息规范 / Commit Messages
|
||||
|
||||
本项目采用 **Conventional Commits**,正文用中文撰写:
|
||||
|
||||
```
|
||||
<type>(<scope>): <subject>
|
||||
|
||||
<body>
|
||||
```
|
||||
|
||||
| type | 说明 |
|
||||
|------|------|
|
||||
| `feat` | 新功能 |
|
||||
| `fix` | 缺陷修复 |
|
||||
| `docs` | 文档变更 |
|
||||
| `style` | 不影响逻辑的格式调整 |
|
||||
| `refactor` | 重构 |
|
||||
| `perf` | 性能优化 |
|
||||
| `test` | 测试相关 |
|
||||
| `chore` | 构建/依赖/工具链 |
|
||||
|
||||
示例:
|
||||
|
||||
```
|
||||
feat(storage): 新增 Wasabi S3 后端支持
|
||||
fix(cluster): 修复跨节点恢复的终态处理
|
||||
docs: 补充 CONTRIBUTING 指南
|
||||
```
|
||||
|
||||
## Pull Request 流程
|
||||
|
||||
1. **Fork** 仓库并从最新的 `main` 切出特性分支;
|
||||
2. **开发**功能或修复,必要时补充测试;
|
||||
3. **自测**:确保 `make test` 通过;
|
||||
4. **提交**:使用上述 Conventional Commits(中文);
|
||||
5. **推送**并对着 `main` 发起 PR。
|
||||
|
||||
### PR 描述建议
|
||||
|
||||
- 清晰说明本 PR 做了什么;
|
||||
- 对新功能/修复,补充动机与背景;
|
||||
- 关联相关 Issue(如 `Closes #62`);
|
||||
- 纯文档 PR 可不附测试。
|
||||
|
||||
> 请保持分支基于较新的 `main`:基线过旧的分支容易产生大范围冲突,难以评审与合入。
|
||||
|
||||
## 编码规范 / Coding Conventions
|
||||
|
||||
- **Go**:所有错误必须处理(禁止 `_ = err`),日志使用项目已有库(`zap`),禁止 `fmt.Println`;提交前执行 `gofmt`。
|
||||
- **前端**:遵循项目 ESLint/Prettier/tsconfig 配置,不擅自引入新的 CSS 框架或 UI 库。
|
||||
- **包管理**:`web/` 使用 npm,请提交对应的 `package-lock.json`。
|
||||
|
||||
## License
|
||||
|
||||
向 BackupX 贡献即表示你同意你的贡献以 [Apache License 2.0](LICENSE) 授权。
|
||||
18
README.md
18
README.md
@@ -60,15 +60,9 @@ docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/back
|
||||
# Or prebuilt archive
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
|
||||
tar xzf backupx-*.tar.gz && cd backupx-* && sudo ./install.sh
|
||||
|
||||
# Or build and install on bare metal without Docker
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
make build && sudo ./deploy/install.sh
|
||||
```
|
||||
|
||||
For ARM64 hosts, use `backupx-linux-arm64.tar.gz`. The archive contains `backupx`, `web/`, `config.example.yaml`, and `install.sh`; run `install.sh` from the extracted directory.
|
||||
|
||||
Open `http://your-server:8340`, choose English or Chinese on the setup screen, create the first administrator account, then follow the [5-minute Quick Start](https://awuqing.github.io/BackupX/docs/getting-started/quick-start).
|
||||
Open `http://your-server:8340`, create the admin account, then follow the [5-minute Quick Start](https://awuqing.github.io/BackupX/docs/getting-started/quick-start).
|
||||
|
||||
## Documentation
|
||||
|
||||
@@ -98,16 +92,6 @@ See the [development guide](https://awuqing.github.io/BackupX/docs/development/s
|
||||
|
||||
Issues and pull requests welcome. Please read the [contributing guide](https://awuqing.github.io/BackupX/docs/development/contributing) before opening a PR — commit messages and PRs on this project are written in Chinese.
|
||||
|
||||
## Star History
|
||||
|
||||
<a href="https://www.star-history.com/?repos=Awuqing%2FBackupX&type=date&legend=top-left">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&theme=dark&legend=top-left" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&legend=top-left" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=Awuqing/BackupX&type=date&legend=top-left" />
|
||||
</picture>
|
||||
</a>
|
||||
|
||||
## License
|
||||
|
||||
[Apache License 2.0](LICENSE)
|
||||
|
||||
@@ -60,15 +60,9 @@ docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/back
|
||||
# 或使用预编译包
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
|
||||
tar xzf backupx-*.tar.gz && cd backupx-* && sudo ./install.sh
|
||||
|
||||
# 或从源码构建并裸机安装(无需 Docker)
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
make build && sudo ./deploy/install.sh
|
||||
```
|
||||
|
||||
ARM64 主机请下载 `backupx-linux-arm64.tar.gz`。预编译包内包含 `backupx`、`web/`、`config.example.yaml` 和 `install.sh`,请在解压后的目录内执行 `install.sh`。
|
||||
|
||||
打开 `http://your-server:8340`,在初始化页选择中文或 English 并创建首个管理员账户,按 [5 分钟快速开始](https://awuqing.github.io/BackupX/zh-Hans/docs/getting-started/quick-start) 完成首次备份。
|
||||
打开 `http://your-server:8340`,创建管理员账户,按 [5 分钟快速开始](https://awuqing.github.io/BackupX/zh-Hans/docs/getting-started/quick-start) 完成首次备份。
|
||||
|
||||
## 文档
|
||||
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
if [ "${1:-}" = "agent" ]; then
|
||||
exec /app/bin/backupx "$@"
|
||||
fi
|
||||
|
||||
# Backend listens on internal port 8341, Nginx exposes 8340
|
||||
export BACKUPX_SERVER_PORT="${BACKUPX_SERVER_PORT_INTERNAL:-8341}"
|
||||
|
||||
|
||||
@@ -1,31 +1,17 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
PROJECT_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd)
|
||||
PROJECT_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
||||
PREFIX="${PREFIX:-/opt/backupx}"
|
||||
ETC_DIR="${ETC_DIR:-/etc/backupx}"
|
||||
SERVICE_NAME="backupx"
|
||||
APP_USER="backupx"
|
||||
APP_GROUP="backupx"
|
||||
if [ -f "$SCRIPT_DIR/backupx" ] && [ -d "$SCRIPT_DIR/web" ]; then
|
||||
BIN_SOURCE="${BIN_SOURCE:-$SCRIPT_DIR/backupx}"
|
||||
WEB_SOURCE="${WEB_SOURCE:-$SCRIPT_DIR/web}"
|
||||
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$SCRIPT_DIR/config.example.yaml}"
|
||||
NGINX_SOURCE="${NGINX_SOURCE:-$SCRIPT_DIR/nginx.conf}"
|
||||
else
|
||||
SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/bin/backupx"
|
||||
# Keep compatibility with contributors who built the historical path by
|
||||
# hand, while matching the canonical `make build` output first.
|
||||
if [ ! -f "$SOURCE_BIN_DEFAULT" ] && [ -f "$PROJECT_ROOT/server/backupx" ]; then
|
||||
SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/backupx"
|
||||
fi
|
||||
BIN_SOURCE="${BIN_SOURCE:-$SOURCE_BIN_DEFAULT}"
|
||||
WEB_SOURCE="${WEB_SOURCE:-$PROJECT_ROOT/web/dist}"
|
||||
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$PROJECT_ROOT/server/config.example.yaml}"
|
||||
NGINX_SOURCE="${NGINX_SOURCE:-$PROJECT_ROOT/deploy/nginx.conf}"
|
||||
fi
|
||||
BIN_SOURCE="${BIN_SOURCE:-$PROJECT_ROOT/server/backupx}"
|
||||
WEB_SOURCE="${WEB_SOURCE:-$PROJECT_ROOT/web/dist}"
|
||||
CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$PROJECT_ROOT/server/config.example.yaml}"
|
||||
SERVICE_SOURCE="${SERVICE_SOURCE:-$PROJECT_ROOT/deploy/backupx.service}"
|
||||
NGINX_SOURCE="${NGINX_SOURCE:-$PROJECT_ROOT/deploy/nginx.conf}"
|
||||
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
echo "请使用 root 或 sudo 执行安装脚本。" >&2
|
||||
@@ -33,22 +19,14 @@ if [ "$(id -u)" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
if [ ! -f "$BIN_SOURCE" ]; then
|
||||
echo "Backend binary not found / 未找到后端二进制:$BIN_SOURCE" >&2
|
||||
echo "源码树安装请先在仓库根目录执行 make build(产物:server/bin/backupx)。" >&2
|
||||
echo "For a source install, run 'make build' in the repository root first." >&2
|
||||
echo "发布包安装请确认当前目录包含 ./backupx、./web 和 ./install.sh。" >&2
|
||||
echo "未找到后端二进制:$BIN_SOURCE" >&2
|
||||
echo "请先执行:cd \"$PROJECT_ROOT/server\" && go build -o backupx ./cmd/backupx" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -d "$WEB_SOURCE" ]; then
|
||||
echo "未找到前端构建产物:$WEB_SOURCE" >&2
|
||||
echo "源码树安装请先执行:cd \"$PROJECT_ROOT/web\" && npm run build" >&2
|
||||
echo "发布包安装请确认当前目录包含 ./web。" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$CONFIG_TEMPLATE" ]; then
|
||||
echo "未找到配置模板:$CONFIG_TEMPLATE" >&2
|
||||
echo "请先执行:cd \"$PROJECT_ROOT/web\" && npm run build" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -66,75 +44,14 @@ cp -R "$WEB_SOURCE/." "$PREFIX/web/"
|
||||
chown -R "$APP_USER:$APP_GROUP" "$PREFIX"
|
||||
|
||||
if [ ! -f "$ETC_DIR/config.yaml" ]; then
|
||||
install -o "$APP_USER" -g "$APP_GROUP" -m 0640 "$CONFIG_TEMPLATE" "$ETC_DIR/config.yaml"
|
||||
install -m 0640 "$CONFIG_TEMPLATE" "$ETC_DIR/config.yaml"
|
||||
fi
|
||||
# 确保服务账户能读取配置:历史版本曾以 root:root 0640 安装配置,
|
||||
# 导致以 backupx 身份运行的服务因无权读取配置而启动失败(exit 1)。
|
||||
chown "$APP_USER:$APP_GROUP" "$ETC_DIR/config.yaml"
|
||||
|
||||
if [ -f "$SERVICE_SOURCE" ]; then
|
||||
install -m 0644 "$SERVICE_SOURCE" "/etc/systemd/system/$SERVICE_NAME.service"
|
||||
else
|
||||
cat > "/etc/systemd/system/$SERVICE_NAME.service" <<UNIT
|
||||
[Unit]
|
||||
Description=BackupX API Service
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=$APP_USER
|
||||
Group=$APP_GROUP
|
||||
WorkingDirectory=$PREFIX
|
||||
ExecStart=$PREFIX/bin/backupx -config $ETC_DIR/config.yaml
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
LimitNOFILE=65535
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
UNIT
|
||||
fi
|
||||
install -m 0644 "$SERVICE_SOURCE" "/etc/systemd/system/$SERVICE_NAME.service"
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now "$SERVICE_NAME"
|
||||
|
||||
# systemctl may return before the process has opened its HTTP listener. Verify
|
||||
# the same unauthenticated endpoint used by the first-administrator screen so a
|
||||
# broken bare-metal install cannot print a false success message.
|
||||
HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8340/api/auth/setup/status}"
|
||||
READY=0
|
||||
ATTEMPT=1
|
||||
while [ "$ATTEMPT" -le 30 ]; do
|
||||
if systemctl is-active --quiet "$SERVICE_NAME"; then
|
||||
if command -v curl >/dev/null 2>&1; then
|
||||
if curl -fsS --max-time 2 "$HEALTH_URL" >/dev/null 2>&1; then
|
||||
READY=1
|
||||
break
|
||||
fi
|
||||
elif command -v wget >/dev/null 2>&1; then
|
||||
if wget -q -T 2 -O /dev/null "$HEALTH_URL"; then
|
||||
READY=1
|
||||
break
|
||||
fi
|
||||
else
|
||||
echo "Warning / 警告:未找到 curl 或 wget,仅验证 systemd 服务状态。" >&2
|
||||
READY=1
|
||||
break
|
||||
fi
|
||||
fi
|
||||
ATTEMPT=$((ATTEMPT + 1))
|
||||
sleep 1
|
||||
done
|
||||
|
||||
if [ "$READY" -ne 1 ]; then
|
||||
echo "BackupX did not become ready at $HEALTH_URL / 服务未通过就绪检查。" >&2
|
||||
systemctl status "$SERVICE_NAME" --no-pager >&2 || true
|
||||
journalctl -u "$SERVICE_NAME" -n 50 --no-pager >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ -d "/etc/nginx/conf.d" ] && [ -f "$NGINX_SOURCE" ]; then
|
||||
if [ -d "/etc/nginx/conf.d" ]; then
|
||||
install -m 0644 "$NGINX_SOURCE" "/etc/nginx/conf.d/$SERVICE_NAME.conf"
|
||||
if command -v nginx >/dev/null 2>&1; then
|
||||
nginx -t
|
||||
@@ -150,19 +67,6 @@ cat <<MESSAGE
|
||||
- 配置文件:$ETC_DIR/config.yaml
|
||||
- systemd 服务:/etc/systemd/system/$SERVICE_NAME.service
|
||||
|
||||
Web 控制台已由后端直接托管,无需额外的 nginx 反向代理即可访问:
|
||||
http://<本机IP>:8340
|
||||
|
||||
首次访问 / First sign-in:
|
||||
1. 打开上面的地址,并可在登录页右上角选择 中文 或 English。
|
||||
2. 页面显示“系统初始化 / System setup”时,创建首个管理员用户名和密码。
|
||||
3. 如果未显示初始化表单,请先检查:$HEALTH_URL
|
||||
|
||||
(如已安装 nginx,脚本会自动写入反向代理配置,可继续用 80 端口访问。)
|
||||
|
||||
排查:若服务未监听端口,请查看日志:
|
||||
journalctl -u "$SERVICE_NAME" -n 50 --no-pager
|
||||
|
||||
如需修改监听地址、数据库路径或日志级别,请编辑 "$ETC_DIR/config.yaml" 后执行:
|
||||
systemctl restart "$SERVICE_NAME"
|
||||
MESSAGE
|
||||
|
||||
@@ -22,8 +22,6 @@ services:
|
||||
# - /home/user/data:/mnt/data:ro
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
# 远程 Agent 需要通过公网或可路由地址连接 Master 时,取消注释并改成真实 URL:
|
||||
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
|
||||
# 通过 BACKUPX_ 前缀环境变量覆盖配置:
|
||||
# - BACKUPX_LOG_LEVEL=debug
|
||||
# - BACKUPX_BACKUP_MAX_CONCURRENT=4
|
||||
|
||||
@@ -10,34 +10,20 @@ description: systemd + Nginx deployment from the prebuilt release tarball or sou
|
||||
|
||||
```bash
|
||||
# Download the matching tarball
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-v1.6.0-linux-amd64.tar.gz
|
||||
|
||||
# Extract and install
|
||||
tar xzf backupx-linux-amd64.tar.gz && cd backupx-*-linux-amd64
|
||||
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
|
||||
sudo ./install.sh
|
||||
```
|
||||
|
||||
The installer performs these steps automatically:
|
||||
|
||||
1. Creates a system user `backupx`
|
||||
2. Copies the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
|
||||
3. Installs the default configuration at `/etc/backupx/config.yaml`
|
||||
2. Copies the binary to `/opt/backupx/`
|
||||
3. Generates a default `config.yaml` with safe JWT/encryption secrets
|
||||
4. Installs `backupx.service` (systemd), enabled at boot
|
||||
5. (Optional) installs an Nginx site file — see [Nginx Reverse Proxy](./nginx)
|
||||
6. Verifies the first-setup API before reporting success
|
||||
|
||||
For multi-node clusters, edit `/etc/backupx/config.yaml` after installation and set the Master URL that remote Agents can reach:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
Restart BackupX after changing it:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart backupx
|
||||
```
|
||||
|
||||
## From source
|
||||
|
||||
@@ -64,13 +50,11 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=backupx
|
||||
Group=backupx
|
||||
WorkingDirectory=/opt/backupx
|
||||
ExecStart=/opt/backupx/bin/backupx -config /etc/backupx/config.yaml
|
||||
ExecStart=/opt/backupx/backupx --config /opt/backupx/config.yaml
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
LimitNOFILE=65535
|
||||
RestartSec=5s
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -82,20 +66,17 @@ Typical operations:
|
||||
sudo systemctl status backupx
|
||||
sudo journalctl -u backupx -f # live logs
|
||||
sudo systemctl restart backupx
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
```
|
||||
|
||||
Open `http://your-server:8340`, switch to English if desired, and create the first administrator on the **System setup** screen. For a custom listen port, run the installer with a matching `HEALTH_URL`.
|
||||
|
||||
## Password reset
|
||||
|
||||
If the admin password is lost:
|
||||
|
||||
```bash
|
||||
/opt/backupx/bin/backupx reset-password \
|
||||
/opt/backupx/backupx reset-password \
|
||||
--username admin \
|
||||
--password 'newpass123' \
|
||||
--config /etc/backupx/config.yaml
|
||||
--config /opt/backupx/config.yaml
|
||||
```
|
||||
|
||||
Docker equivalent:
|
||||
|
||||
@@ -15,14 +15,13 @@ server:
|
||||
host: "0.0.0.0" # BACKUPX_SERVER_HOST
|
||||
port: 8340 # BACKUPX_SERVER_PORT
|
||||
mode: "release" # release | debug
|
||||
external_url: "" # BACKUPX_SERVER_EXTERNAL_URL — public Master URL for Agent install scripts
|
||||
|
||||
database:
|
||||
path: "./data/backupx.db" # BACKUPX_DATABASE_PATH — embedded SQLite
|
||||
|
||||
security:
|
||||
jwt_secret: "" # BACKUPX_SECURITY_JWT_SECRET — auto-generated if empty
|
||||
jwt_expire: "24h" # BACKUPX_SECURITY_JWT_EXPIRE
|
||||
jwt_expires_in: "24h"
|
||||
encryption_key: "" # AES-256-GCM key for storage config encryption
|
||||
|
||||
backup:
|
||||
@@ -47,20 +46,7 @@ The environment wins when both file and env are set. All dot-paths become unders
|
||||
| Config key | Env variable |
|
||||
|------------|--------------|
|
||||
| `server.port` | `BACKUPX_SERVER_PORT` |
|
||||
| `server.external_url` | `BACKUPX_SERVER_EXTERNAL_URL` |
|
||||
| `security.jwt_expire` | `BACKUPX_SECURITY_JWT_EXPIRE` |
|
||||
| `log.level` | `BACKUPX_LOG_LEVEL` |
|
||||
| `backup.max_concurrent` | `BACKUPX_BACKUP_MAX_CONCURRENT` |
|
||||
| `backup.temp_dir` | `BACKUPX_BACKUP_TEMP_DIR` |
|
||||
| `backup.bandwidth_limit` | `BACKUPX_BACKUP_BANDWIDTH_LIMIT` |
|
||||
|
||||
## Master external URL
|
||||
|
||||
Set `server.external_url` when BackupX is behind Docker, Nginx, a load balancer, or any reverse proxy whose internal Host is not reachable by remote Agents:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
This value is used when BackupX renders one-click Agent install scripts and docker-compose snippets. It must be reachable from every Agent host. Leave it empty only when `X-Forwarded-Proto` / `X-Forwarded-Host` are reliable and point to the same URL that Agents can access.
|
||||
|
||||
@@ -25,8 +25,6 @@ services:
|
||||
- /etc/nginx:/mnt/nginx-conf:ro
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
# Required when remote Agents must connect through a public or routed URL:
|
||||
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
|
||||
- BACKUPX_LOG_LEVEL=info
|
||||
- BACKUPX_BACKUP_MAX_CONCURRENT=2
|
||||
|
||||
@@ -44,17 +42,6 @@ docker compose up -d
|
||||
|
||||
To back up files from the host, mount them into the container. When creating a file-type task in the web UI, point the source path at the mount location (e.g. `/mnt/www`). Make sure the directory is visible inside the container.
|
||||
|
||||
## Multi-node clusters
|
||||
|
||||
When deploying Agents on other machines, set `BACKUPX_SERVER_EXTERNAL_URL` on the Master container to the URL that those Agents can reach:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
|
||||
```
|
||||
|
||||
Use an HTTPS URL if Agents cross untrusted networks. The generated one-click install scripts and docker-compose snippets use this value as `BACKUPX_AGENT_MASTER`.
|
||||
|
||||
## Environment variables
|
||||
|
||||
All configuration keys can be overridden with the `BACKUPX_` prefix:
|
||||
|
||||
@@ -8,26 +8,14 @@ description: File, MySQL, PostgreSQL, SQLite and SAP HANA — what they back up
|
||||
|
||||
BackupX supports five built-in backup types. Type determines which runner executes the job.
|
||||
|
||||
When a task is routed to a remote Agent, the source tools and paths are resolved on that Agent host. Multi-target uploads are still tracked per storage target; if at least one target succeeds, the backup record is marked successful and the per-target result table shows partial failures.
|
||||
|
||||
## File / Directory
|
||||
|
||||
File tasks offer three backup modes:
|
||||
|
||||
- **Full archive** — writes a self-contained tar artifact on every run
|
||||
- **Differential archive** — writes only changes since the current full baseline and periodically refreshes that baseline
|
||||
- **CDC repository** — splits content with stable 512 KiB / 1 MiB / 4 MiB boundaries, stores new chunks in immutable 32 MiB packs, and writes a small snapshot manifest for each run
|
||||
|
||||
The CDC repository deduplicates identical content across files and snapshots. Restore, selective restore, verification, download-as-tar, retention, and garbage collection all resolve data through the repository index. Compression and encryption are applied per chunk; encrypted repositories use keyed chunk IDs so plaintext hashes are not exposed.
|
||||
|
||||
Repository mode currently uses a single-writer index and therefore runs on the Master only. To keep repository copies on multiple backends, select multiple primary storage targets on the task. Object-level replication is intentionally disabled because a snapshot manifest without its shared packs and indexes is not a complete backup.
|
||||
|
||||
Common file-task options:
|
||||
Tars (and optionally gzips) one or more filesystem paths.
|
||||
|
||||
- **Source** accepts multiple paths — one per line in the UI
|
||||
- **Exclude patterns** accept gitignore-style globs
|
||||
- Supports following symlinks, preserving permissions
|
||||
- Full and differential modes output `.tar`, `.tar.gz`, or `.tar.zst` artifacts
|
||||
- Output is a single `.tar` or `.tar.gz` artifact
|
||||
|
||||
## MySQL
|
||||
|
||||
|
||||
@@ -26,42 +26,8 @@ BackupX supports Master-Agent mode: backup tasks can be routed to specific nodes
|
||||
- **Execution** — Agent reuses the same BackupRunner (file / mysql / postgresql / sqlite / saphana) and uploads directly to storage
|
||||
- **Security** — Each node has its own token; the Agent never holds the Master's JWT secret or AES-256 key
|
||||
|
||||
## Centralize backups from servers B/C/D into storage M
|
||||
|
||||
Use the Master as the control plane and register every source server as an Agent. A task's **Source server** determines where paths and database tools are resolved; its **Storage targets** determine where the resulting artifact is retained.
|
||||
|
||||
BackupX chooses the data path per target:
|
||||
|
||||
| Destination | Data path |
|
||||
| --- | --- |
|
||||
| S3, WebDAV, FTP, cloud drive, or another network backend | Agent streams directly to the destination |
|
||||
| `local_disk` with **Relay remote backups through Master** enabled (for example storage server M mounted through NFS) | Agent streams through the authenticated Master API; Master writes to its configured local path |
|
||||
|
||||
The relay is streaming: the Master does not create a second temporary copy of the entire artifact. The reverse path is used when restoring a Master-local artifact back to its source Agent. Use HTTPS whenever Agent traffic crosses an untrusted network.
|
||||
|
||||
To configure the common `A → {B,C,D} → M` topology:
|
||||
|
||||
1. Run BackupX Master on A and mount M on A if M is exposed as NFS or another filesystem.
|
||||
2. Create a `local_disk` target for that mount and keep **Relay remote backups through Master** enabled, or create an S3/WebDAV target exposed by M. Existing local-disk targets keep their prior Agent-local behavior until this switch is enabled.
|
||||
3. Install one Agent on B, C, and D from **Node Management**.
|
||||
4. Create a backup task for each source, choose B/C/D under **Source server**, browse that server's paths, and select M as the storage target. A source-server pool label can route identical tasks dynamically.
|
||||
5. Verify the per-target result in the backup record. For a Master-local target, the record reports transfer mode `master_relay`; network backends remain `direct`.
|
||||
|
||||
## Walkthrough
|
||||
|
||||
### 0. Set the Master URL for production clusters
|
||||
|
||||
Before generating Agent install commands, make sure the Master URL shown to Agents is stable and reachable from every target host.
|
||||
|
||||
If BackupX runs behind Docker, Nginx, a load balancer, or an outer reverse proxy, configure `server.external_url` or `BACKUPX_SERVER_EXTERNAL_URL` on the Master:
|
||||
|
||||
```yaml title="config.yaml"
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
This URL is baked into systemd units, foreground commands, and docker-compose snippets. If it is wrong, Agents will install successfully but stay offline because they keep polling an internal or browser-only address.
|
||||
|
||||
### 1. Open the install wizard
|
||||
|
||||
In the Web Console → **Node Management** → **Add Node**. You'll see a three-step wizard.
|
||||
@@ -83,8 +49,6 @@ The script runs automatically and:
|
||||
5. Runs `systemctl enable --now backupx-agent`
|
||||
6. Polls `/api/v1/agent/self` until the master confirms `status: online` (up to 30 s)
|
||||
|
||||
Docker mode uses the same `BACKUPX_AGENT_MASTER`, `BACKUPX_AGENT_TOKEN`, and `BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp` environment contract. After starting the container, the installer also probes `/api/v1/agent/self`; if the node does not come online, it prints `docker ps` and `docker logs --tail=100 backupx-agent` diagnostics before exiting non-zero.
|
||||
|
||||
If you choose the URL-based fallback command and `curl` prints HTML or the shell reports `Syntax error: newline unexpected`, the install URL is being served by the web console instead of the backend. Ensure either `/api/install/` or `/install/` is forwarded to the BackupX backend, or use the embedded command generated by the console.
|
||||
|
||||
Reruns are idempotent — to upgrade or re-provision, simply generate a new install command and run it again. The one-time install link expires after its TTL or after first consumption, whichever is sooner.
|
||||
@@ -99,20 +63,14 @@ In Step 1 choose "Batch" and paste node names (one per line, max 50). Step 3 sho
|
||||
|
||||
### 5. Route a task to the node
|
||||
|
||||
In the **Backup Tasks** page, pick the source server when creating the task. When the task runs:
|
||||
In the **Backup Tasks** page, pick the target node when creating the task. When the task runs:
|
||||
|
||||
- Local (`nodeId=0`) → Master executes in-process
|
||||
- Remote node → Master enqueues the command → Agent claims → Agent runs locally → uploads → reports back
|
||||
|
||||
The node table shows the Agent health and command queue state: pending/dispatched depth, running long commands, timeouts, oldest active command age, and the latest Agent-side error. The same queue depth, running-command, and timeout snapshots are exported as Prometheus metrics:
|
||||
|
||||
- `backupx_agent_command_queue_depth`
|
||||
- `backupx_agent_command_running`
|
||||
- `backupx_agent_command_timeout_total`
|
||||
|
||||
## Known limitations
|
||||
|
||||
- **Encrypted backups are Master-only** — the Agent doesn't hold Master's AES-256 key. Creating or updating a task with `encrypt: true` and a remote node or node pool is rejected up front
|
||||
- **Encrypted backups don't work via Agent** — the Agent doesn't hold Master's AES-256 key. Tasks with `encrypt: true` will fail if routed to an Agent
|
||||
- **Directory browser timeout** — remote dir listing is a synchronous RPC through the queue (15s default)
|
||||
- **Dispatched command timeout** — claimed-but-unfinished commands are marked `timeout` after 10 minutes
|
||||
|
||||
|
||||
@@ -19,9 +19,7 @@ BackupX aims to accept any place you'd want to drop a backup file.
|
||||
| **Google Drive** | Client ID/Secret + OAuth authorization |
|
||||
| **WebDAV** | URL + username/password |
|
||||
| **FTP / FTPS** | Host + port + username/password |
|
||||
| **Local disk** | Target directory (absolute path) + optional Master relay for remote Agents |
|
||||
|
||||
New local-disk targets enable **Relay remote backups through Master** by default. This makes the configured path belong to the Master, so a storage server mounted there can collect backups from many source Agents. Turn the switch off when the path intentionally belongs to each Agent. Existing targets retain their previous Agent-local behavior until explicitly changed.
|
||||
| **Local disk** | Target directory (absolute path) |
|
||||
|
||||
## Rclone backends
|
||||
|
||||
|
||||
@@ -55,11 +55,10 @@ sudo ./install.sh # creates system user, installs to /opt/backupx, sets u
|
||||
The installer:
|
||||
|
||||
1. Creates a `backupx` system user
|
||||
2. Installs the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
|
||||
3. Creates `/etc/backupx/config.yaml` with safe defaults
|
||||
2. Installs binary to `/opt/backupx/backupx`
|
||||
3. Creates `/opt/backupx/config.yaml` with safe defaults
|
||||
4. Installs and enables the `backupx.service` systemd unit
|
||||
5. (Optional) Configures an Nginx reverse proxy
|
||||
6. Waits for `/api/auth/setup/status`; if startup fails, prints systemd diagnostics and exits non-zero
|
||||
|
||||
## From source
|
||||
|
||||
@@ -68,17 +67,16 @@ Requires Go ≥ 1.25 and Node.js ≥ 20.
|
||||
```bash
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
make build
|
||||
sudo ./deploy/install.sh
|
||||
# or, for builds behind the great firewall
|
||||
make docker-cn
|
||||
```
|
||||
|
||||
After `make build`, the binary is at `server/bin/backupx` and the built web UI is at `web/dist/`.
|
||||
The installer consumes those exact paths, so no Docker runtime is required. If an existing configuration uses a non-default port, set `HEALTH_URL` for the readiness check, for example `sudo HEALTH_URL=http://127.0.0.1:9000/api/auth/setup/status ./deploy/install.sh`.
|
||||
|
||||
## Verify the install
|
||||
|
||||
```bash
|
||||
/opt/backupx/bin/backupx --version
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
backupx --version # e.g. v1.6.0
|
||||
```
|
||||
|
||||
Then open `http://your-server:8340`. Choose **English** or **中文** in the upper-right corner. A fresh database shows **System setup**, where you create the first administrator username and password. If that form does not appear, retry the status request above before attempting to sign in.
|
||||
Then open `http://your-server:8340` to see the initial admin setup screen.
|
||||
|
||||
@@ -42,8 +42,6 @@ Go to **Backup Tasks → New**. Three steps:
|
||||
2. **Source** — paths for file backup (multi-source supported), or connection info for databases
|
||||
3. **Storage & policy** — pick target(s), compression, retention days, encryption on/off
|
||||
|
||||
For Agent-routed tasks, encryption must stay off because the Agent never receives the Master's encryption key. BackupX rejects remote-node or node-pool tasks with encryption enabled during create/update.
|
||||
|
||||
Save, then click **Run Now** to trigger a test. Live logs stream on the **Backup Records** page.
|
||||
|
||||
:::note
|
||||
|
||||
@@ -10,34 +10,20 @@ description: 从预编译包或源码部署 BackupX(systemd + Nginx)。
|
||||
|
||||
```bash
|
||||
# 下载对应平台的压缩包
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-v1.6.0-linux-amd64.tar.gz
|
||||
|
||||
# 解压并安装
|
||||
tar xzf backupx-linux-amd64.tar.gz && cd backupx-*-linux-amd64
|
||||
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
|
||||
sudo ./install.sh
|
||||
```
|
||||
|
||||
安装脚本自动完成以下步骤:
|
||||
|
||||
1. 创建系统用户 `backupx`
|
||||
2. 复制二进制到 `/opt/backupx/bin/backupx`,并把 Web 控制台复制到 `/opt/backupx/web`
|
||||
3. 把默认配置安装到 `/etc/backupx/config.yaml`
|
||||
2. 复制二进制到 `/opt/backupx/`
|
||||
3. 生成默认 `config.yaml`(含安全的 JWT/加密密钥)
|
||||
4. 安装并启用 `backupx.service` systemd 单元
|
||||
5. (可选)生成 Nginx 站点配置 — 参见 [Nginx 反向代理](./nginx)
|
||||
6. 验证首次初始化接口就绪后才报告安装成功
|
||||
|
||||
如果要部署多节点集群,安装后请编辑 `/etc/backupx/config.yaml`,设置远程 Agent 可访问到的 Master URL:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
修改后重启 BackupX:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart backupx
|
||||
```
|
||||
|
||||
## 从源码构建
|
||||
|
||||
@@ -64,13 +50,11 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=backupx
|
||||
Group=backupx
|
||||
WorkingDirectory=/opt/backupx
|
||||
ExecStart=/opt/backupx/bin/backupx -config /etc/backupx/config.yaml
|
||||
ExecStart=/opt/backupx/backupx --config /opt/backupx/config.yaml
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
LimitNOFILE=65535
|
||||
RestartSec=5s
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -82,20 +66,17 @@ WantedBy=multi-user.target
|
||||
sudo systemctl status backupx
|
||||
sudo journalctl -u backupx -f # 实时日志
|
||||
sudo systemctl restart backupx
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
```
|
||||
|
||||
访问 `http://your-server:8340`,可按需切换到 English,然后在“系统初始化 / System setup”页面创建首个管理员。若监听端口不是默认值,请为安装脚本传入对应的 `HEALTH_URL`。
|
||||
|
||||
## 密码重置
|
||||
|
||||
忘记管理员密码时:
|
||||
|
||||
```bash
|
||||
/opt/backupx/bin/backupx reset-password \
|
||||
/opt/backupx/backupx reset-password \
|
||||
--username admin \
|
||||
--password 'newpass123' \
|
||||
--config /etc/backupx/config.yaml
|
||||
--config /opt/backupx/config.yaml
|
||||
```
|
||||
|
||||
Docker 等效命令:
|
||||
|
||||
@@ -15,14 +15,13 @@ server:
|
||||
host: "0.0.0.0" # BACKUPX_SERVER_HOST
|
||||
port: 8340 # BACKUPX_SERVER_PORT
|
||||
mode: "release" # release | debug
|
||||
external_url: "" # BACKUPX_SERVER_EXTERNAL_URL — Agent 安装脚本使用的 Master 对外 URL
|
||||
|
||||
database:
|
||||
path: "./data/backupx.db" # BACKUPX_DATABASE_PATH — 内嵌 SQLite
|
||||
|
||||
security:
|
||||
jwt_secret: "" # BACKUPX_SECURITY_JWT_SECRET — 留空自动生成
|
||||
jwt_expire: "24h" # BACKUPX_SECURITY_JWT_EXPIRE
|
||||
jwt_expires_in: "24h"
|
||||
encryption_key: "" # 用于加密存储配置的 AES-256-GCM 密钥
|
||||
|
||||
backup:
|
||||
@@ -47,20 +46,7 @@ log:
|
||||
| 配置项 | 环境变量 |
|
||||
|--------|----------|
|
||||
| `server.port` | `BACKUPX_SERVER_PORT` |
|
||||
| `server.external_url` | `BACKUPX_SERVER_EXTERNAL_URL` |
|
||||
| `security.jwt_expire` | `BACKUPX_SECURITY_JWT_EXPIRE` |
|
||||
| `log.level` | `BACKUPX_LOG_LEVEL` |
|
||||
| `backup.max_concurrent` | `BACKUPX_BACKUP_MAX_CONCURRENT` |
|
||||
| `backup.temp_dir` | `BACKUPX_BACKUP_TEMP_DIR` |
|
||||
| `backup.bandwidth_limit` | `BACKUPX_BACKUP_BANDWIDTH_LIMIT` |
|
||||
|
||||
## Master 对外 URL
|
||||
|
||||
当 BackupX 部署在 Docker、Nginx、负载均衡或多层反向代理后面,且后端收到的内部 Host 不是远程 Agent 可访问地址时,请配置 `server.external_url`:
|
||||
|
||||
```yaml
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
BackupX 会用这个地址渲染一键 Agent 安装脚本和 docker-compose 片段。该地址必须能被所有 Agent 主机访问。只有在 `X-Forwarded-Proto` / `X-Forwarded-Host` 可靠且正好指向 Agent 可访问地址时,才建议留空。
|
||||
|
||||
@@ -25,8 +25,6 @@ services:
|
||||
- /etc/nginx:/mnt/nginx-conf:ro
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
# 远程 Agent 需要通过公网或可路由地址连接 Master 时必须配置:
|
||||
# - BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
|
||||
- BACKUPX_LOG_LEVEL=info
|
||||
- BACKUPX_BACKUP_MAX_CONCURRENT=2
|
||||
|
||||
@@ -44,17 +42,6 @@ docker compose up -d
|
||||
|
||||
想备份宿主机上的文件,需要将对应路径挂载进容器。在 Web UI 创建文件类型任务时,把源路径指向挂载后的容器内路径(如 `/mnt/www`)。
|
||||
|
||||
## 多节点集群
|
||||
|
||||
如果要在其他机器部署 Agent,请在 Master 容器上设置 `BACKUPX_SERVER_EXTERNAL_URL`,值为所有 Agent 都能访问到的 URL:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- BACKUPX_SERVER_EXTERNAL_URL=https://backup.example.com
|
||||
```
|
||||
|
||||
Agent 跨不可信网络访问时建议使用 HTTPS。控制台生成的一键安装脚本和 docker-compose 片段会把这个值写成 `BACKUPX_AGENT_MASTER`。
|
||||
|
||||
## 环境变量
|
||||
|
||||
所有配置项都可以通过 `BACKUPX_` 前缀环境变量覆盖:
|
||||
|
||||
@@ -8,26 +8,14 @@ description: 文件、MySQL、PostgreSQL、SQLite 和 SAP HANA — 各自的能
|
||||
|
||||
BackupX 支持五种内置备份类型,类型决定了用哪个 runner 执行。
|
||||
|
||||
当任务路由到远程 Agent 时,源路径和外部工具都会在该 Agent 主机上解析。多存储目标上传仍会逐目标记录结果;只要至少一个目标上传成功,备份记录即为成功,详情中的目标结果表会展示部分失败。
|
||||
|
||||
## 文件 / 目录
|
||||
|
||||
文件任务提供三种备份模式:
|
||||
|
||||
- **全量归档** — 每次生成一份可独立使用的 tar 产物
|
||||
- **差异归档** — 只保存相对当前全量基线的变化,并按周期刷新全量基线
|
||||
- **CDC 去重仓库** — 按稳定的 512 KiB / 1 MiB / 4 MiB 内容边界切块,将新块合并到不可变的 32 MiB pack,每次运行只新增一份小型快照清单
|
||||
|
||||
CDC 仓库会在不同文件、不同快照之间复用相同内容。完整恢复、选择性恢复、完整性校验、下载为 tar、保留策略和垃圾回收都通过仓库索引定位分块。压缩与加密按块执行;启用加密时使用带密钥的块 ID,不暴露明文哈希。
|
||||
|
||||
当前仓库索引采用单写者模型,因此 CDC 模式仅在 Master 本机执行。如需保存多份完整仓库,请在任务中直接多选主存储目标。对象级副本复制会被禁用,因为只有快照清单、没有共享 pack 与索引并不是完整备份。
|
||||
|
||||
文件任务的通用选项:
|
||||
打包(可选 gzip)一个或多个文件系统路径。
|
||||
|
||||
- **源路径** 支持多个(UI 中每行一个)
|
||||
- **排除模式** 支持 gitignore 风格的通配符
|
||||
- 可选跟随符号链接、保留权限
|
||||
- 全量与差异模式输出 `.tar`、`.tar.gz` 或 `.tar.zst`
|
||||
- 输出单个 `.tar` 或 `.tar.gz`
|
||||
|
||||
## MySQL
|
||||
|
||||
|
||||
@@ -26,42 +26,8 @@ BackupX 支持 Master-Agent 模式:备份任务可以指定在哪个节点执
|
||||
- **执行** — Agent 复用 BackupRunner(file / mysql / postgresql / sqlite / saphana)并直接上传到存储
|
||||
- **安全** — 每个节点独立 Token;Agent 不持有 Master 的 JWT 密钥或 AES-256 加密密钥
|
||||
|
||||
## 把 B/C/D 服务器集中备份到 M
|
||||
|
||||
Master 作为控制面,每台源服务器安装一个 Agent。任务里的 **源服务器** 决定源路径和数据库工具在哪台机器解析,**存储目标** 决定备份产物最终保留在哪里。
|
||||
|
||||
BackupX 会根据目标类型选择数据路径:
|
||||
|
||||
| 目标 | 数据路径 |
|
||||
| --- | --- |
|
||||
| S3、WebDAV、FTP、云盘或其他网络后端 | Agent 直接流式上传到目标 |
|
||||
| 启用 **远程备份经 Master 中转** 的 `local_disk`(例如通过 NFS 挂载的存储服务器 M) | Agent 通过认证后的 Master API 流式中转,由 Master 写入配置目录 |
|
||||
|
||||
中转过程不会在 Master 上额外落一份完整临时文件。把 Master 本地磁盘中的备份恢复到源 Agent 时会走反向流式通道。Agent 与 Master 之间跨越不可信网络时必须配置 HTTPS。
|
||||
|
||||
典型的 `A → {B,C,D} → M` 拓扑按以下步骤配置:
|
||||
|
||||
1. 在 A 运行 BackupX Master;如果 M 以 NFS 等文件系统提供存储,先把 M 挂载到 A。
|
||||
2. 为该挂载点创建 `local_disk` 目标并保持 **远程备份经 Master 中转** 开启;如果 M 提供 S3/WebDAV,也可直接创建对应网络目标。升级前已有的本地磁盘目标会继续沿用 Agent 本机落盘,手动开启该选项后才切换到中央目录。
|
||||
3. 从 **节点管理** 分别在 B、C、D 安装 Agent。
|
||||
4. 为每台源服务器创建任务,在 **源服务器** 选择 B/C/D,浏览该服务器的路径,再把 M 选为存储目标。相同任务也可用源服务器池标签动态调度。
|
||||
5. 在备份记录中检查逐目标结果。Master 本地磁盘目标会记录 `master_relay` 中转模式,网络后端仍为 `direct` 直传。
|
||||
|
||||
## 一键部署步骤
|
||||
|
||||
### 0. 为生产集群设置 Master 对外 URL
|
||||
|
||||
生成 Agent 安装命令前,请先确认 Master URL 对所有目标主机稳定可达。
|
||||
|
||||
如果 BackupX 部署在 Docker、Nginx、负载均衡或外层反向代理后面,请在 Master 配置 `server.external_url` 或环境变量 `BACKUPX_SERVER_EXTERNAL_URL`:
|
||||
|
||||
```yaml title="config.yaml"
|
||||
server:
|
||||
external_url: "https://backup.example.com"
|
||||
```
|
||||
|
||||
该 URL 会写入 systemd 单元、前台运行命令和 docker-compose 片段。如果地址不正确,Agent 可能安装成功但始终离线,因为它会持续轮询一个内网地址或仅浏览器可访问的地址。
|
||||
|
||||
### 1. 打开安装向导
|
||||
|
||||
Web 控制台 → **节点管理** → **添加节点**,打开三步向导:
|
||||
@@ -83,8 +49,6 @@ Web 控制台 → **节点管理** → **添加节点**,打开三步向导:
|
||||
5. 执行 `systemctl enable --now backupx-agent`
|
||||
6. 轮询 `/api/v1/agent/self`,直到 Master 确认 `status: online`(最多 30 秒)
|
||||
|
||||
Docker 模式使用同一组环境变量约定:`BACKUPX_AGENT_MASTER`、`BACKUPX_AGENT_TOKEN` 和 `BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp`。容器启动后,安装脚本同样会探测 `/api/v1/agent/self`;如果节点没有上线,会输出 `docker ps` 与 `docker logs --tail=100 backupx-agent` 排查命令,并以非零状态退出。
|
||||
|
||||
如果使用 URL 备用命令时 `curl` 输出 HTML,或 shell 报 `Syntax error: newline unexpected`,说明安装 URL 被 Web 控制台接管而不是转发到后端。需要确保 `/api/install/` 或 `/install/` 至少一个路径能转发到 BackupX 后端,或改用控制台生成的嵌入式命令。
|
||||
|
||||
脚本是幂等的:升级或重装只需重新生成一条安装命令再跑一次。一次性安装链接在 TTL 到期或被首次消费后立即作废。
|
||||
@@ -99,20 +63,14 @@ Docker 模式使用同一组环境变量约定:`BACKUPX_AGENT_MASTER`、`BACKU
|
||||
|
||||
### 5. 把任务路由到该节点
|
||||
|
||||
在 **备份任务** 页面新建任务时选择对应源服务器。任务触发时:
|
||||
在 **备份任务** 页面新建任务时选择对应节点。任务触发时:
|
||||
|
||||
- 本机 / 未指定(`nodeId=0`):Master 进程内直接执行
|
||||
- 远程节点:Master 写入命令队列 → Agent 拉取 → Agent 本地执行 → 上传 → 回报
|
||||
|
||||
节点列表会展示 Agent 健康与命令队列状态:pending/dispatched 深度、运行中的长任务、超时数、最旧活跃命令年龄和最近 Agent 错误。同样的队列深度、运行中命令数和超时快照会导出为 Prometheus 指标:
|
||||
|
||||
- `backupx_agent_command_queue_depth`
|
||||
- `backupx_agent_command_running`
|
||||
- `backupx_agent_command_timeout_total`
|
||||
|
||||
## 已知限制
|
||||
|
||||
- **加密备份仅支持 Master 本机执行**:Agent 不持有 Master 的 AES-256 密钥。创建或更新任务时,如果 `encrypt: true` 且选择了远程节点或节点池,会在入口直接拒绝
|
||||
- **Agent 不支持加密备份**:Agent 不持有 Master 的 AES-256 密钥。`encrypt: true` 的任务路由到 Agent 时会直接上报失败
|
||||
- **目录浏览超时**:远程目录浏览通过命令队列做同步 RPC,默认 15s 超时
|
||||
- **派发命令超时**:Agent 领取但未完成的命令超过 10 分钟会被置 `timeout`
|
||||
|
||||
|
||||
@@ -19,9 +19,7 @@ BackupX 的目标是接入任何你想放置备份文件的地方。
|
||||
| **Google Drive** | Client ID/Secret + OAuth 授权 |
|
||||
| **WebDAV** | 地址 + 用户名/密码 |
|
||||
| **FTP / FTPS** | 主机 + 端口 + 用户名/密码 |
|
||||
| **本地磁盘** | 目标目录(绝对路径)+ 可选的远程 Agent 经 Master 中转 |
|
||||
|
||||
新建本地磁盘目标默认开启 **远程备份经 Master 中转**。开启时,配置目录属于 Master,挂载到 Master 的存储服务器可集中接收多台源 Agent 的备份;如果该路径本就属于各 Agent,请关闭此选项。升级前已有目标保持原来的 Agent 本机落盘行为,只有显式开启后才会切换。
|
||||
| **本地磁盘** | 目标目录(绝对路径) |
|
||||
|
||||
## Rclone 后端
|
||||
|
||||
|
||||
@@ -55,11 +55,10 @@ sudo ./install.sh # 创建系统用户、安装到 /opt/backupx、配置
|
||||
安装脚本会自动:
|
||||
|
||||
1. 创建 `backupx` 系统用户
|
||||
2. 安装二进制到 `/opt/backupx/bin/backupx`,并把 Web 控制台安装到 `/opt/backupx/web`
|
||||
3. 生成 `/etc/backupx/config.yaml`(含安全默认值)
|
||||
2. 安装二进制到 `/opt/backupx/backupx`
|
||||
3. 生成 `/opt/backupx/config.yaml`(含安全默认值)
|
||||
4. 注册并启用 `backupx.service` systemd 单元
|
||||
5. (可选)配置 Nginx 反向代理
|
||||
6. 等待 `/api/auth/setup/status` 就绪;启动失败时输出 systemd 诊断并返回非零状态
|
||||
|
||||
## 从源码构建
|
||||
|
||||
@@ -68,17 +67,16 @@ sudo ./install.sh # 创建系统用户、安装到 /opt/backupx、配置
|
||||
```bash
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
make build
|
||||
sudo ./deploy/install.sh
|
||||
# 或使用国内镜像加速构建 Docker
|
||||
make docker-cn
|
||||
```
|
||||
|
||||
`make build` 完成后,二进制位于 `server/bin/backupx`,构建好的 Web UI 位于 `web/dist/`。
|
||||
安装脚本会直接使用这两个路径,不需要 Docker 运行时。如果已有配置修改了默认端口,可覆盖就绪检查地址,例如:`sudo HEALTH_URL=http://127.0.0.1:9000/api/auth/setup/status ./deploy/install.sh`。
|
||||
|
||||
## 验证安装
|
||||
|
||||
```bash
|
||||
/opt/backupx/bin/backupx --version
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
backupx --version # 输出如 v1.6.0
|
||||
```
|
||||
|
||||
打开浏览器访问 `http://your-server:8340`,可在右上角选择 **中文** 或 **English**。全新数据库会显示“系统初始化 / System setup”,在这里创建首个管理员用户名和密码。如果没有出现初始化表单,请先重试上面的状态接口,不要直接尝试登录。
|
||||
打开浏览器访问 `http://your-server:8340`,会进入初始化管理员账户页面。
|
||||
|
||||
@@ -42,8 +42,6 @@ description: 部署 BackupX、添加存储目标、创建第一个备份任务
|
||||
2. **源配置** — 文件备份选择源路径(支持多个),数据库备份填写连接信息
|
||||
3. **存储与策略** — 选择存储目标(支持多个)、压缩策略、保留天数、是否加密
|
||||
|
||||
对于路由到 Agent 的任务,加密必须关闭,因为 Agent 不会拿到 Master 的加密密钥。BackupX 会在创建/更新阶段拒绝开启加密的远程节点或节点池任务。
|
||||
|
||||
保存后可点击 **立即执行** 测试,**备份记录** 页面实时查看执行日志。
|
||||
|
||||
:::note
|
||||
|
||||
237
docs-site/package-lock.json
generated
237
docs-site/package-lock.json
generated
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "docs-site-tmp",
|
||||
"version": "1.0.0",
|
||||
"version": "0.0.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "docs-site-tmp",
|
||||
"version": "1.0.0",
|
||||
"version": "0.0.0",
|
||||
"dependencies": {
|
||||
"@docusaurus/core": "3.10.0",
|
||||
"@docusaurus/faster": "3.10.0",
|
||||
@@ -164,6 +164,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@algolia/client-search/-/client-search-5.50.2.tgz",
|
||||
"integrity": "sha512-ypSboUJ3XJoQz5DeDo82hCnrRuwq3q9ZdFhVKAik9TnZh1DvLqoQsrbBjXg7C7zQOtV/Qbge/HmyoV6V5L7MhQ==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@algolia/client-common": "5.50.2",
|
||||
"@algolia/requester-browser-xhr": "5.50.2",
|
||||
@@ -262,12 +263,12 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/code-frame": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.7.tgz",
|
||||
"integrity": "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw==",
|
||||
"version": "7.29.0",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/code-frame/-/code-frame-7.29.0.tgz",
|
||||
"integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/helper-validator-identifier": "^7.29.7",
|
||||
"@babel/helper-validator-identifier": "^7.28.5",
|
||||
"js-tokens": "^4.0.0",
|
||||
"picocolors": "^1.1.1"
|
||||
},
|
||||
@@ -289,6 +290,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@babel/core/-/core-7.29.0.tgz",
|
||||
"integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/code-frame": "^7.29.0",
|
||||
"@babel/generator": "^7.29.0",
|
||||
@@ -324,13 +326,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/generator": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.7.tgz",
|
||||
"integrity": "sha512-DkXD5OJQaAQIdZ1bt3UZdEnHAn9Imd3IVBdX03UFe+ony9Ojw5pzr9YVKGDY1jt+Gcn/FnGkNf8r+Vj5NOJWtQ==",
|
||||
"version": "7.29.1",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/generator/-/generator-7.29.1.tgz",
|
||||
"integrity": "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/parser": "^7.29.7",
|
||||
"@babel/types": "^7.29.7",
|
||||
"@babel/parser": "^7.29.0",
|
||||
"@babel/types": "^7.29.0",
|
||||
"@jridgewell/gen-mapping": "^0.3.12",
|
||||
"@jridgewell/trace-mapping": "^0.3.28",
|
||||
"jsesc": "^3.0.2"
|
||||
@@ -449,9 +451,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-globals": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.29.7.tgz",
|
||||
"integrity": "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA==",
|
||||
"version": "7.28.0",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-globals/-/helper-globals-7.28.0.tgz",
|
||||
"integrity": "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -471,27 +473,27 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-module-imports": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.29.7.tgz",
|
||||
"integrity": "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g==",
|
||||
"version": "7.28.6",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-module-imports/-/helper-module-imports-7.28.6.tgz",
|
||||
"integrity": "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/traverse": "^7.29.7",
|
||||
"@babel/types": "^7.29.7"
|
||||
"@babel/traverse": "^7.28.6",
|
||||
"@babel/types": "^7.28.6"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-module-transforms": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.29.7.tgz",
|
||||
"integrity": "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg==",
|
||||
"version": "7.28.6",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-module-transforms/-/helper-module-transforms-7.28.6.tgz",
|
||||
"integrity": "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/helper-module-imports": "^7.29.7",
|
||||
"@babel/helper-validator-identifier": "^7.29.7",
|
||||
"@babel/traverse": "^7.29.7"
|
||||
"@babel/helper-module-imports": "^7.28.6",
|
||||
"@babel/helper-validator-identifier": "^7.28.5",
|
||||
"@babel/traverse": "^7.28.6"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -513,9 +515,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-plugin-utils": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-plugin-utils/-/helper-plugin-utils-7.29.7.tgz",
|
||||
"integrity": "sha512-G7sHYigPY17oO5SYWnfD/0MTBwVR781S/JI643e/JhUYgVgWE/61SoW3NH9KWUKyKq5LVh3npif99Wkt6j86Jw==",
|
||||
"version": "7.28.6",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-plugin-utils/-/helper-plugin-utils-7.28.6.tgz",
|
||||
"integrity": "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -569,18 +571,18 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-string-parser": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz",
|
||||
"integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==",
|
||||
"version": "7.27.1",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
|
||||
"integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/helper-validator-identifier": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz",
|
||||
"integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==",
|
||||
"version": "7.28.5",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz",
|
||||
"integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -623,12 +625,12 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/parser": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz",
|
||||
"integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==",
|
||||
"version": "7.29.2",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/parser/-/parser-7.29.2.tgz",
|
||||
"integrity": "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/types": "^7.29.7"
|
||||
"@babel/types": "^7.29.0"
|
||||
},
|
||||
"bin": {
|
||||
"parser": "bin/babel-parser.js"
|
||||
@@ -1213,15 +1215,15 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/plugin-transform-modules-systemjs": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.7.tgz",
|
||||
"integrity": "sha512-TM2ZcQLoG2/y4HODiStCo10DibYhWhGWAwVv+EQKmG/7GFl0N+AAmUiXOMKM+aiJ9XBJ9AHVZBvTzMnJ2sM3cQ==",
|
||||
"version": "7.29.0",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/plugin-transform-modules-systemjs/-/plugin-transform-modules-systemjs-7.29.0.tgz",
|
||||
"integrity": "sha512-PrujnVFbOdUpw4UHiVwKvKRLMMic8+eC0CuNlxjsyZUiBjhFdPsewdXCkveh2KqBA9/waD0W1b4hXSOBQJezpQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/helper-module-transforms": "^7.29.7",
|
||||
"@babel/helper-plugin-utils": "^7.29.7",
|
||||
"@babel/helper-validator-identifier": "^7.29.7",
|
||||
"@babel/traverse": "^7.29.7"
|
||||
"@babel/helper-module-transforms": "^7.28.6",
|
||||
"@babel/helper-plugin-utils": "^7.28.6",
|
||||
"@babel/helper-validator-identifier": "^7.28.5",
|
||||
"@babel/traverse": "^7.29.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -1918,31 +1920,31 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/template": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/template/-/template-7.29.7.tgz",
|
||||
"integrity": "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==",
|
||||
"version": "7.28.6",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/template/-/template-7.28.6.tgz",
|
||||
"integrity": "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/code-frame": "^7.29.7",
|
||||
"@babel/parser": "^7.29.7",
|
||||
"@babel/types": "^7.29.7"
|
||||
"@babel/code-frame": "^7.28.6",
|
||||
"@babel/parser": "^7.28.6",
|
||||
"@babel/types": "^7.28.6"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/traverse": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.7.tgz",
|
||||
"integrity": "sha512-EhlfNQtZ+NK22w5BM61ciuiq1m58ed33Wr1Xan//ZRTy6hgjnwyCffRYwzsGXdASJSUJ1guZILsErh1eQcl+zw==",
|
||||
"version": "7.29.0",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/traverse/-/traverse-7.29.0.tgz",
|
||||
"integrity": "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/code-frame": "^7.29.7",
|
||||
"@babel/generator": "^7.29.7",
|
||||
"@babel/helper-globals": "^7.29.7",
|
||||
"@babel/parser": "^7.29.7",
|
||||
"@babel/template": "^7.29.7",
|
||||
"@babel/types": "^7.29.7",
|
||||
"@babel/code-frame": "^7.29.0",
|
||||
"@babel/generator": "^7.29.0",
|
||||
"@babel/helper-globals": "^7.28.0",
|
||||
"@babel/parser": "^7.29.0",
|
||||
"@babel/template": "^7.28.6",
|
||||
"@babel/types": "^7.29.0",
|
||||
"debug": "^4.3.1"
|
||||
},
|
||||
"engines": {
|
||||
@@ -1950,13 +1952,13 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@babel/types": {
|
||||
"version": "7.29.7",
|
||||
"resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz",
|
||||
"integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==",
|
||||
"version": "7.29.0",
|
||||
"resolved": "https://registry.npmmirror.com/@babel/types/-/types-7.29.0.tgz",
|
||||
"integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@babel/helper-string-parser": "^7.29.7",
|
||||
"@babel/helper-validator-identifier": "^7.29.7"
|
||||
"@babel/helper-string-parser": "^7.27.1",
|
||||
"@babel/helper-validator-identifier": "^7.28.5"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
@@ -2079,6 +2081,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
@@ -2101,6 +2104,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
@@ -2210,6 +2214,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
|
||||
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"cssesc": "^3.0.0",
|
||||
"util-deprecate": "^1.0.2"
|
||||
@@ -2631,6 +2636,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
|
||||
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"cssesc": "^3.0.0",
|
||||
"util-deprecate": "^1.0.2"
|
||||
@@ -3543,6 +3549,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@docusaurus/faster/-/faster-3.10.0.tgz",
|
||||
"integrity": "sha512-GNPtVH14ISjHfSwnHu3KiFGf86ICmJSQDeSv/QaanpBgiZGOtgZaslnC5q8WiguxM1EVkwcGxPuD8BXF4eggKw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@docusaurus/types": "3.10.0",
|
||||
"@rspack/core": "^1.7.10",
|
||||
@@ -3673,6 +3680,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@docusaurus/plugin-content-docs/-/plugin-content-docs-3.10.0.tgz",
|
||||
"integrity": "sha512-9BjHhf15ct8Z7TThTC0xRndKDVvMKmVsAGAN7W9FpNRzfMdScOGcXtLmcCWtJGvAezjOJIm6CxOYCy3Io5+RnQ==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@docusaurus/core": "3.10.0",
|
||||
"@docusaurus/logger": "3.10.0",
|
||||
@@ -4704,6 +4712,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@mdx-js/react/-/react-3.1.1.tgz",
|
||||
"integrity": "sha512-f++rKLQgUVYDAtECQ6fn/is15GkEH9+nZPM3MS0RcxVqoTfawHvDlSCH7JbMhAM6uJ32v3eXLvLmLvjGu7PTQw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/mdx": "^2.0.0"
|
||||
},
|
||||
@@ -5408,6 +5417,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@svgr/core/-/core-8.1.0.tgz",
|
||||
"integrity": "sha512-8QqtOQT5ACVlmsvKOJNEaWmRPmcojMOzCz4Hs2BGG/toAp/K38LcsMRyLp349glq5AzJbCEeimEoxaX6v/fLrA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/core": "^7.21.3",
|
||||
"@svgr/babel-preset": "8.1.0",
|
||||
@@ -5512,6 +5522,7 @@
|
||||
"integrity": "sha512-tglZGyx8N5PC+x1Nd/JrZxqpqlcZoSuG9gTDKO6AuFToFiVB3uS8HvbKFuO7g3lJzvFf9riAb94xs9HU2UhAHQ==",
|
||||
"hasInstallScript": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@swc/counter": "^0.1.3",
|
||||
"@swc/types": "^0.1.26"
|
||||
@@ -6236,6 +6247,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@types/react/-/react-19.2.14.tgz",
|
||||
"integrity": "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"csstype": "^3.2.2"
|
||||
}
|
||||
@@ -6577,6 +6589,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/acorn/-/acorn-8.16.0.tgz",
|
||||
"integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"acorn": "bin/acorn"
|
||||
},
|
||||
@@ -6644,6 +6657,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-8.18.0.tgz",
|
||||
"integrity": "sha512-PlXPeEWMXMZ7sPYOHqmDyCJzcfNrUr3fGNKtezX14ykXOEIvyK81d+qydx89KY5O71FKMPaQ2vBfBFI5NHR63A==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.3",
|
||||
"fast-uri": "^3.0.1",
|
||||
@@ -6689,6 +6703,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/algoliasearch/-/algoliasearch-5.50.2.tgz",
|
||||
"integrity": "sha512-Tfp26yoNWurUjfgK4GOrVJQhSNXu9tJtHfFFNosgT2YClG+vPyUjX/gbC8rG39qLncnZg8Fj34iarQWpMkqefw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@algolia/abtesting": "1.16.2",
|
||||
"@algolia/client-abtesting": "5.50.2",
|
||||
@@ -7042,9 +7057,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/body-parser": {
|
||||
"version": "1.20.5",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.5.tgz",
|
||||
"integrity": "sha512-3grm+/2tUOvu2cjJkvsIxrv/wVpfXQW4PsQHYm7yk4vfpu7Ekl6nEsYBoJUL6qDwZUx8wUhQ8tR2qz+ad9c9OA==",
|
||||
"version": "1.20.4",
|
||||
"resolved": "https://registry.npmmirror.com/body-parser/-/body-parser-1.20.4.tgz",
|
||||
"integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"bytes": "~3.1.2",
|
||||
@@ -7055,7 +7070,7 @@
|
||||
"http-errors": "~2.0.1",
|
||||
"iconv-lite": "~0.4.24",
|
||||
"on-finished": "~2.4.1",
|
||||
"qs": "~6.15.1",
|
||||
"qs": "~6.14.0",
|
||||
"raw-body": "~2.5.3",
|
||||
"type-is": "~1.6.18",
|
||||
"unpipe": "~1.0.0"
|
||||
@@ -7067,7 +7082,7 @@
|
||||
},
|
||||
"node_modules/body-parser/node_modules/bytes": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/bytes/-/bytes-3.1.2.tgz",
|
||||
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -7076,7 +7091,7 @@
|
||||
},
|
||||
"node_modules/body-parser/node_modules/debug": {
|
||||
"version": "2.6.9",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/debug/-/debug-2.6.9.tgz",
|
||||
"integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -7085,7 +7100,7 @@
|
||||
},
|
||||
"node_modules/body-parser/node_modules/ms": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/ms/-/ms-2.0.0.tgz",
|
||||
"integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==",
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -7168,6 +7183,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"baseline-browser-mapping": "^2.10.12",
|
||||
"caniuse-lite": "^1.0.30001782",
|
||||
@@ -7826,7 +7842,7 @@
|
||||
},
|
||||
"node_modules/content-type": {
|
||||
"version": "1.0.5",
|
||||
"resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/content-type/-/content-type-1.0.5.tgz",
|
||||
"integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -8134,6 +8150,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
|
||||
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"cssesc": "^3.0.0",
|
||||
"util-deprecate": "^1.0.2"
|
||||
@@ -9263,14 +9280,14 @@
|
||||
}
|
||||
},
|
||||
"node_modules/express": {
|
||||
"version": "4.22.2",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
|
||||
"integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
|
||||
"version": "4.22.1",
|
||||
"resolved": "https://registry.npmmirror.com/express/-/express-4.22.1.tgz",
|
||||
"integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"accepts": "~1.3.8",
|
||||
"array-flatten": "1.1.1",
|
||||
"body-parser": "~1.20.5",
|
||||
"body-parser": "~1.20.3",
|
||||
"content-disposition": "~0.5.4",
|
||||
"content-type": "~1.0.4",
|
||||
"cookie": "~0.7.1",
|
||||
@@ -9289,7 +9306,7 @@
|
||||
"parseurl": "~1.3.3",
|
||||
"path-to-regexp": "~0.1.12",
|
||||
"proxy-addr": "~2.0.7",
|
||||
"qs": "~6.15.1",
|
||||
"qs": "~6.14.0",
|
||||
"range-parser": "~1.2.1",
|
||||
"safe-buffer": "5.2.1",
|
||||
"send": "~0.19.0",
|
||||
@@ -9397,9 +9414,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/fast-uri": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.2.tgz",
|
||||
"integrity": "sha512-rVjf7ArG3LTk+FS6Yw81V1DLuZl1bRbNrev6Tmd/9RaroeeRRJhAt7jg/6YFxbvAQXUCavSoZhPPj6oOx+5KjQ==",
|
||||
"version": "3.1.0",
|
||||
"resolved": "https://registry.npmmirror.com/fast-uri/-/fast-uri-3.1.0.tgz",
|
||||
"integrity": "sha512-iPeeDKJSWf4IEOasVVrknXpaBV0IApz/gp7S2bb7Z4Lljbl2MGJRqInZiUrQwV16cpzw/D3S5j5Julj/gT52AA==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "github",
|
||||
@@ -9507,6 +9524,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
|
||||
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.1",
|
||||
"fast-json-stable-stringify": "^2.0.0",
|
||||
@@ -10553,7 +10571,7 @@
|
||||
},
|
||||
"node_modules/iconv-lite": {
|
||||
"version": "0.4.24",
|
||||
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/iconv-lite/-/iconv-lite-0.4.24.tgz",
|
||||
"integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -12051,7 +12069,7 @@
|
||||
},
|
||||
"node_modules/media-typer": {
|
||||
"version": "0.3.0",
|
||||
"resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/media-typer/-/media-typer-0.3.0.tgz",
|
||||
"integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -14186,6 +14204,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
|
||||
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.1",
|
||||
"fast-json-stable-stringify": "^2.0.0",
|
||||
@@ -14241,7 +14260,7 @@
|
||||
},
|
||||
"node_modules/object-inspect": {
|
||||
"version": "1.13.4",
|
||||
"resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/object-inspect/-/object-inspect-1.13.4.tgz",
|
||||
"integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -14700,9 +14719,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/postcss": {
|
||||
"version": "8.5.14",
|
||||
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz",
|
||||
"integrity": "sha512-SoSL4+OSEtR99LHFZQiJLkT59C5B1amGO1NzTwj7TT1qCUgUO6hxOvzkOYxD+vMrXBM3XJIKzokoERdqQq/Zmg==",
|
||||
"version": "8.5.10",
|
||||
"resolved": "https://registry.npmmirror.com/postcss/-/postcss-8.5.10.tgz",
|
||||
"integrity": "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==",
|
||||
"funding": [
|
||||
{
|
||||
"type": "opencollective",
|
||||
@@ -14718,6 +14737,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"nanoid": "^3.3.11",
|
||||
"picocolors": "^1.1.1",
|
||||
@@ -15621,6 +15641,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/postcss-selector-parser/-/postcss-selector-parser-7.1.1.tgz",
|
||||
"integrity": "sha512-orRsuYpJVw8LdAwqqLykBj9ecS5/cRHlI5+nvTo8LcCKmzDmqVORXtOIYEEQuL9D4BxtA1lm5isAqzQZCoQ6Eg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"cssesc": "^3.0.0",
|
||||
"util-deprecate": "^1.0.2"
|
||||
@@ -16314,9 +16335,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/qs": {
|
||||
"version": "6.15.2",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.15.2.tgz",
|
||||
"integrity": "sha512-Rzq0KEyX/w/tEybncDgdkZrJgVUsUMk3xjh3t5bv3S1HTAtg+uOYt72+ZfwiQwKdysThkTBdL/rTi6HDmX9Ddw==",
|
||||
"version": "6.14.2",
|
||||
"resolved": "https://registry.npmmirror.com/qs/-/qs-6.14.2.tgz",
|
||||
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"side-channel": "^1.1.0"
|
||||
@@ -16380,7 +16401,7 @@
|
||||
},
|
||||
"node_modules/raw-body": {
|
||||
"version": "2.5.3",
|
||||
"resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/raw-body/-/raw-body-2.5.3.tgz",
|
||||
"integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -16395,7 +16416,7 @@
|
||||
},
|
||||
"node_modules/raw-body/node_modules/bytes": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/bytes/-/bytes-3.1.2.tgz",
|
||||
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -16437,6 +16458,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/react/-/react-19.2.5.tgz",
|
||||
"integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -16446,6 +16468,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/react-dom/-/react-dom-19.2.5.tgz",
|
||||
"integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"scheduler": "^0.27.0"
|
||||
},
|
||||
@@ -16501,6 +16524,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/@docusaurus/react-loadable/-/react-loadable-6.0.0.tgz",
|
||||
"integrity": "sha512-YMMxTUQV/QFSnbgrP3tjDzLHRg7vsbMn8e9HAa8o/1iXoiomo48b7sk/kkmWEuWNDPJVlKSJRB6Y2fHqdJk+SQ==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/react": "*"
|
||||
},
|
||||
@@ -16529,6 +16553,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/react-router/-/react-router-5.3.4.tgz",
|
||||
"integrity": "sha512-Ys9K+ppnJah3QuaRiLxk+jDWOR1MekYQrlytiXxC1RyfbdsZkS5pvKAzCCr031xHixZwpnsYNT5xysdFHQaYsA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/runtime": "^7.12.13",
|
||||
"history": "^4.9.0",
|
||||
@@ -17204,7 +17229,7 @@
|
||||
},
|
||||
"node_modules/safer-buffer": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/safer-buffer/-/safer-buffer-2.1.2.tgz",
|
||||
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -17575,7 +17600,7 @@
|
||||
},
|
||||
"node_modules/side-channel": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/side-channel/-/side-channel-1.1.0.tgz",
|
||||
"integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -17594,7 +17619,7 @@
|
||||
},
|
||||
"node_modules/side-channel-list": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/side-channel-list/-/side-channel-list-1.0.1.tgz",
|
||||
"integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -17610,7 +17635,7 @@
|
||||
},
|
||||
"node_modules/side-channel-map": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/side-channel-map/-/side-channel-map-1.0.1.tgz",
|
||||
"integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -17628,7 +17653,7 @@
|
||||
},
|
||||
"node_modules/side-channel-weakmap": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
|
||||
"integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -18303,7 +18328,8 @@
|
||||
"version": "2.8.1",
|
||||
"resolved": "https://registry.npmmirror.com/tslib/-/tslib-2.8.1.tgz",
|
||||
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
|
||||
"license": "0BSD"
|
||||
"license": "0BSD",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/tsyringe": {
|
||||
"version": "4.10.0",
|
||||
@@ -18337,7 +18363,7 @@
|
||||
},
|
||||
"node_modules/type-is": {
|
||||
"version": "1.6.18",
|
||||
"resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/type-is/-/type-is-1.6.18.tgz",
|
||||
"integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -18350,7 +18376,7 @@
|
||||
},
|
||||
"node_modules/type-is/node_modules/mime-db": {
|
||||
"version": "1.52.0",
|
||||
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/mime-db/-/mime-db-1.52.0.tgz",
|
||||
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -18359,7 +18385,7 @@
|
||||
},
|
||||
"node_modules/type-is/node_modules/mime-types": {
|
||||
"version": "2.1.35",
|
||||
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
|
||||
"resolved": "https://registry.npmmirror.com/mime-types/-/mime-types-2.1.35.tgz",
|
||||
"integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -18384,6 +18410,7 @@
|
||||
"integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==",
|
||||
"devOptional": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
@@ -18725,6 +18752,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/ajv/-/ajv-6.14.0.tgz",
|
||||
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.1",
|
||||
"fast-json-stable-stringify": "^2.0.0",
|
||||
@@ -18923,6 +18951,7 @@
|
||||
"resolved": "https://registry.npmmirror.com/webpack/-/webpack-5.106.2.tgz",
|
||||
"integrity": "sha512-wGN3qcrBQIFmQ/c0AiOAQBvrZ5lmY8vbbMv4Mxfgzqd/B6+9pXtLo73WuS1dSGXM5QYY3hZnIbvx+K1xxe6FyA==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/eslint-scope": "^3.7.7",
|
||||
"@types/estree": "^1.0.8",
|
||||
@@ -19064,9 +19093,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/webpack-dev-server": {
|
||||
"version": "5.2.4",
|
||||
"resolved": "https://registry.npmjs.org/webpack-dev-server/-/webpack-dev-server-5.2.4.tgz",
|
||||
"integrity": "sha512-GqDPGZN9bRqKBTkp4aWkobDDHMsrXKoGSdOH56smIri8qR0JG8gfL8/v/f/OZR3/OKXjG8uwJbFVhKm/FNU/UA==",
|
||||
"version": "5.2.3",
|
||||
"resolved": "https://registry.npmmirror.com/webpack-dev-server/-/webpack-dev-server-5.2.3.tgz",
|
||||
"integrity": "sha512-9Gyu2F7+bg4Vv+pjbovuYDhHX+mqdqITykfzdM9UyKqKHlsE5aAjRhR+oOEfXW5vBeu8tarzlJFIZva4ZjAdrQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/bonjour": "^3.5.13",
|
||||
|
||||
@@ -3,9 +3,6 @@ server:
|
||||
host: "0.0.0.0"
|
||||
port: 8340
|
||||
mode: "release" # debug | release
|
||||
external_url: "" # 可选:Master 对 Agent 可达的 URL,例如 https://backup.example.com
|
||||
web_root: "" # 前端静态目录;留空自动探测(./web、/opt/backupx/web 等)。
|
||||
# 命中后后端直接托管 Web 控制台,无需额外 nginx 反向代理。
|
||||
|
||||
database:
|
||||
path: "./data/backupx.db" # SQLite 数据库路径
|
||||
|
||||
@@ -6,15 +6,14 @@ require (
|
||||
github.com/gin-gonic/gin v1.10.1
|
||||
github.com/glebarez/sqlite v1.11.0
|
||||
github.com/golang-jwt/jwt/v5 v5.3.0
|
||||
github.com/klauspost/compress v1.18.1
|
||||
github.com/natefinch/lumberjack v2.0.0+incompatible
|
||||
github.com/pquerna/otp v1.5.0
|
||||
github.com/prometheus/client_golang v1.23.2
|
||||
github.com/rclone/rclone v1.73.5
|
||||
github.com/pquerna/otp v1.5.0
|
||||
github.com/rclone/rclone v1.73.3
|
||||
github.com/robfig/cron/v3 v3.0.1
|
||||
github.com/spf13/viper v1.20.0
|
||||
go.uber.org/zap v1.27.0
|
||||
golang.org/x/crypto v0.50.0
|
||||
golang.org/x/crypto v0.48.0
|
||||
golang.org/x/oauth2 v0.34.0
|
||||
google.golang.org/api v0.255.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
@@ -30,10 +29,10 @@ require (
|
||||
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 // indirect
|
||||
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 // indirect
|
||||
github.com/Azure/go-ntlmssp v0.1.1 // indirect
|
||||
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26 // indirect
|
||||
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
|
||||
github.com/BurntSushi/toml v1.6.0 // indirect
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.38 // indirect
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
|
||||
github.com/Files-com/files-sdk-go/v3 v3.2.264 // indirect
|
||||
github.com/IBM/go-sdk-core/v5 v5.18.5 // indirect
|
||||
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
|
||||
@@ -50,26 +49,25 @@ require (
|
||||
github.com/anchore/go-lzo v0.1.0 // indirect
|
||||
github.com/andybalholm/cascadia v1.3.3 // indirect
|
||||
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.3 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/config v1.31.17 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.11 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 // indirect
|
||||
github.com/aws/smithy-go v1.25.1 // indirect
|
||||
github.com/aws/smithy-go v1.24.2 // indirect
|
||||
github.com/bahlo/generic-list-go v0.2.0 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/boombuler/barcode v1.1.0 // indirect
|
||||
@@ -111,7 +109,7 @@ require (
|
||||
github.com/glebarez/go-sqlite v1.21.2 // indirect
|
||||
github.com/go-chi/chi/v5 v5.2.5 // indirect
|
||||
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
|
||||
github.com/go-git/go-billy/v5 v5.9.0 // indirect
|
||||
github.com/go-git/go-billy/v5 v5.6.2 // indirect
|
||||
github.com/go-logr/logr v1.4.3 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-ole/go-ole v1.3.0 // indirect
|
||||
@@ -150,6 +148,7 @@ require (
|
||||
github.com/json-iterator/go v1.1.12 // indirect
|
||||
github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
|
||||
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
|
||||
github.com/klauspost/compress v1.18.1 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
|
||||
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect
|
||||
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect
|
||||
@@ -225,22 +224,22 @@ require (
|
||||
go.mongodb.org/mongo-driver v1.17.6 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
|
||||
go.opentelemetry.io/otel v1.41.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.41.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.41.0 // indirect
|
||||
go.opentelemetry.io/otel v1.39.0 // indirect
|
||||
go.opentelemetry.io/otel/metric v1.39.0 // indirect
|
||||
go.opentelemetry.io/otel/trace v1.39.0 // indirect
|
||||
go.uber.org/multierr v1.10.0 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.3 // indirect
|
||||
golang.org/x/arch v0.14.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
|
||||
golang.org/x/image v0.41.0 // indirect
|
||||
golang.org/x/mod v0.35.0 // indirect
|
||||
golang.org/x/net v0.53.0 // indirect
|
||||
golang.org/x/sync v0.20.0 // indirect
|
||||
golang.org/x/sys v0.43.0 // indirect
|
||||
golang.org/x/term v0.42.0 // indirect
|
||||
golang.org/x/text v0.37.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
|
||||
golang.org/x/image v0.32.0 // indirect
|
||||
golang.org/x/mod v0.32.0 // indirect
|
||||
golang.org/x/net v0.51.0 // indirect
|
||||
golang.org/x/sync v0.19.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/term v0.40.0 // indirect
|
||||
golang.org/x/text v0.34.0 // indirect
|
||||
golang.org/x/time v0.14.0 // indirect
|
||||
golang.org/x/tools v0.44.0 // indirect
|
||||
golang.org/x/tools v0.41.0 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
|
||||
google.golang.org/grpc v1.79.3 // indirect
|
||||
google.golang.org/protobuf v1.36.10 // indirect
|
||||
|
||||
122
server/go.sum
122
server/go.sum
@@ -51,8 +51,8 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 h1:ZJJNFaQ86GVKQ9ehw
|
||||
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3/go.mod h1:URuDvhmATVKqHBH9/0nOiNKk0+YcwfQ3WkK5PqHKxc8=
|
||||
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 h1:sxgSqOB9CDToiaVFpxuvb5wGgGqWa3lCShcm5o0n3bE=
|
||||
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3/go.mod h1:XdED8i399lEVblYHTZM8eXaP07gv4Z58IL6ueMlVlrg=
|
||||
github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw=
|
||||
github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26 h1:gy/jrlpp8EfSyA73a51fofoSfhp5rPNQAUvDr4Dm91c=
|
||||
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk=
|
||||
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM=
|
||||
github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE=
|
||||
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs=
|
||||
@@ -61,8 +61,8 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
|
||||
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
|
||||
github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
|
||||
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.38 h1:YCyHs3wUXEe2BEWn40vcoyaQ2ruHNmNwkasfo3Th16A=
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.38/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 h1:W8S9TrAyZ4//3PXsU6+Bi+fe/6uIL986GyS7PVzIDL4=
|
||||
github.com/FilenCloudDienste/filen-sdk-go v0.0.37/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0=
|
||||
github.com/Files-com/files-sdk-go/v3 v3.2.264 h1:lMHTplAYI9FtmCo/QOcpRxmPA5REVAct1r2riQmDQKw=
|
||||
github.com/Files-com/files-sdk-go/v3 v3.2.264/go.mod h1:wGqkOzRu/ClJibvDgcfuJNAqI2nLhe8g91tPlDKRCdE=
|
||||
github.com/IBM/go-sdk-core/v5 v5.18.5 h1:g0JRl3sYXJczB/yuDlrN6x22LJ6jIxhp0Sa4ARNW60c=
|
||||
@@ -102,46 +102,44 @@ github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kk
|
||||
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
|
||||
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc h1:LoL75er+LKDHDUfU5tRvFwxH0LjPpZN8OoG8Ll+liGU=
|
||||
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc/go.mod h1:w648aMHEgFYS6xb0KVMMtZ2uMeemhiKCuD2vj6gY52A=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.8 h1:iu+64gwDKEoKnyTQskSku72dAwggKI5sV6rNvgSMpMs=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.8/go.mod h1:MI2XvA+qDi3i9AJxX1E2fu730syEBzp/jnXrjxuHwgI=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.3 h1:4kQ/fa22KjDt13QCy1+bYADvdgcxpfH18f0zP542kZA=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.3/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 h1:N4lRUXZpZ1KVEUn6hxtco/1d2lgYhNn1fHkkl8WhlyQ=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.31.17 h1:QFl8lL6RgakNK86vusim14P2k8BFSxjvUkcWLDjgz9Y=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.31.17/go.mod h1:V8P7ILjp/Uef/aX8TjGk6OHZN6IKPM5YW6S78QnRD5c=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.11 h1:NdV8cwCcAXrCWyxArt58BrvZJ9pZ9Fhf9w6Uh5W3Uyc=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.11/go.mod h1:30yY2zqkMPdrvxBqzI9xQCM+WrlrZKSOpSJEsylVU+8=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 h1:INUvJxmhdEbVulJYHI061k4TVuS3jzzthNvjqvVvTKM=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19/go.mod h1:FpZN2QISLdEBWkayloda+sZjVJL+e9Gl0k1SyTgcswU=
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1 h1:IbWiN670htmBioc+Zj32vSpJgQ2+OYSlvTvfQ1nCORQ=
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.1/go.mod h1:tw/B596EUhBWDFGdDGuLC21fVU4A3s4/5Efy8S39W18=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23/go.mod h1:xYWD6BS9ywC5bS3sz9Xh04whO/hzK2plt2Zkyrp4JuA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueOw02f/duEPTMK59Q4QMAoTTtTo=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk=
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 h1:2fjfz3/G9BRvIKuNZ655GwzpklC2kEH0cowZQGO7uBg=
|
||||
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4/go.mod h1:Ymws824lvMypLFPwyyUXM52SXuGgxpu0+DISLfKvB+c=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 h1:/sECfyq2JTifMI2JPyZ4bdRN77zJmr6SrS1eL3augIA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19/go.mod h1:dMf8A5oAqr9/oxOfLkC/c2LU/uMcALP0Rgn2BD5LWn0=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 h1:AWeJMk33GTBf6J20XJe6qZoRSJo0WfUhsMdUKhoODXE=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19/go.mod h1:+GWrYoaAsV7/4pNHpwh1kiNLXkKaSoppxQq9lbH8Ejw=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
|
||||
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15 h1:ieLCO1JxUWuxTZ1cRd0GAaeX7O6cIxnwk7tc1LsQhC4=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.15/go.mod h1:e3IzZvQ3kAWNykvE0Tr0RDZCMFInMvhku3qNpcIQXhM=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23/go.mod h1:/CMNUqoj46HpS3MNRDEDIwcgEnrtZlKRaHNaHxIFpNA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23 h1:03xatSQO4+AM1lTAbnRg5OK528EUg744nW7F73U8DKw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.23/go.mod h1:M8l3mwgx5ToK7wot2sBBce/ojzgnPzZXUV445gTSyE8=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0 h1:etqBTKY581iwLL/H/S2sVgk3C9lAsTJFeXWFDsDcWOU=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0/go.mod h1:L2dcoOgS2VSgbPLvpak2NyUPsO1TBN7M45Z4H7DlRc4=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7 h1:Y2cAXlClHsXkkOvWZFXATr34b0hxxloeQu/pAZz2row=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.0.7/go.mod h1:idzZ7gmDeqeNrSPkdbtMp9qWMgcBwykA7P7Rzh5DXVU=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 h1:qi3e/dmpdONhj1RyIZdi6DKKpDXS5Lb8ftr3p7cyHJc=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20/go.mod h1:V1K+TeJVD5JOk3D9e5tsX2KUdL7BlB+FV6cBhdobN8c=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 h1:XAq62tBTJP/85lFD5oqOOe7YYgWxY9LvWq8plyDvDVg=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 h1:BYf7XNsJMzl4mObARUBUib+j2tf0U//JAAtTnYqvqCw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11/go.mod h1:aEUS4WrNk/+FxkBZZa7tVgp4pGH+kFGW40Y8rCPqt5g=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 h1:X1Tow7suZk9UCJHE1Iw9GMZJJl0dAnKXXP1NaSDHwmw=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19/go.mod h1:/rARO8psX+4sfjUQXp5LLifjUt8DuATZ31WptNJTyQA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 h1:JnQeStZvPHFHeyky/7LbMlyQjUa+jIBj36OlWm0pzIk=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19/go.mod h1:HGyasyHvYdFQeJhvDHfH7HXkHh57htcJGKDZ+7z+I24=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4 h1:4ExZyubQ6LQQVuF2Qp9OsfEvsTdAWh5Gfwf6PgIdLdk=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4/go.mod h1:NF3JcMGOiARAss1ld3WGORCw71+4ExDD2cbbdKS5PpA=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 h1:iSsvB9EtQ09YrsmIc44Heqlx5ByGErqhPK1ZQLppias=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12/go.mod h1:fEWYKTRGoZNl8tZ77i61/ccwOMJdGxwOhWCkp6TXAr0=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 h1:EnUdUqRP1CNzt2DkV67tJx6XDN4xlfBFm+bzeNOQVb0=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16/go.mod h1:Jic/xv0Rq/pFNCh3WwpH4BEqdbSAl+IyHro8LbibHD8=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 h1:XQTQTF75vnug2TXS8m7CVJfC2nniYPZnO1D4Np761Oo=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8/go.mod h1:Xgx+PR1NUOjNmQY+tRMnouRp83JRM8pRMw/vCaVhPkI=
|
||||
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
||||
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
github.com/aws/smithy-go v1.24.2 h1:FzA3bu/nt/vDvmnkg+R8Xl46gmzEDam6mZ1hzmwXFng=
|
||||
github.com/aws/smithy-go v1.24.2/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
|
||||
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
|
||||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
@@ -258,8 +256,8 @@ github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
|
||||
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
|
||||
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
|
||||
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
|
||||
github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA=
|
||||
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
|
||||
github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UNbRM=
|
||||
github.com/go-git/go-billy/v5 v5.6.2/go.mod h1:rcFC2rAsp/erv7CMz9GczHcuD0D32fWzH+MJAU+jaUU=
|
||||
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
@@ -532,8 +530,8 @@ github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 h1:4MI2
|
||||
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11/go.mod h1:3HLX7dwZgvB7nt+Yl/xdzVPcargQ1yBmJEUg3n+jMKM=
|
||||
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 h1:Lc+d3ISfQaMJKWZOE7z4ZSY4RVmdzbn1B0IM8xN18qM=
|
||||
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc=
|
||||
github.com/rclone/rclone v1.73.5 h1:r8a9JHYIWUqk7hNRJuMJ3cROkKfB2zmfkADg8ZLYh6I=
|
||||
github.com/rclone/rclone v1.73.5/go.mod h1:WVv8gvA/lEl/Y37e8I8yosm7ZY+Szq7ujXbJS8Ol63o=
|
||||
github.com/rclone/rclone v1.73.3 h1:XKlobcnXxxzxnB6UBSVtRB+UeZmYDV9B4QExVSSGoAY=
|
||||
github.com/rclone/rclone v1.73.3/go.mod h1:QJDWatpAY9sKGXfpKZUXbThvtHoeo78DcFP2+/cbkvc=
|
||||
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
|
||||
github.com/relvacode/iso8601 v1.7.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
@@ -653,16 +651,16 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 h1:RbKq8BG0FI8OiXhBfcRtqqHcZcka+gU3cskNuf05R18=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0/go.mod h1:h06DGIukJOevXaj/xrNjhi/2098RZzcLTbc0jDAUbsg=
|
||||
go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c=
|
||||
go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE=
|
||||
go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ=
|
||||
go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps=
|
||||
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
|
||||
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
|
||||
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
|
||||
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
|
||||
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
|
||||
go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0=
|
||||
go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis=
|
||||
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
|
||||
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/mock v0.5.0 h1:KAMbZvZPyBPWgD14IrIQ38QCyjwpvVVV6K/bHl1IwQU=
|
||||
@@ -690,8 +688,8 @@ golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliY
|
||||
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
|
||||
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
|
||||
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
|
||||
golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI=
|
||||
golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q=
|
||||
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
|
||||
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
|
||||
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
|
||||
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
|
||||
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
|
||||
@@ -702,12 +700,12 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
|
||||
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
|
||||
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
|
||||
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
|
||||
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
|
||||
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
|
||||
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70=
|
||||
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
|
||||
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
|
||||
golang.org/x/image v0.41.0 h1:8wS72eGJMJaBxK6okTzd4WaXumUlTVlb753MlsSvTCo=
|
||||
golang.org/x/image v0.41.0/go.mod h1:uIc348UZMSvS5Z65CVZ7iDPaNobNFEPeJ4kbqTOszmA=
|
||||
golang.org/x/image v0.32.0 h1:6lZQWq75h7L5IWNk0r+SCpUJ6tUVd3v4ZHnbRKLkUDQ=
|
||||
golang.org/x/image v0.32.0/go.mod h1:/R37rrQmKXtO6tYXAjtDLwQgFLHmhW+V6ayXlxzP2Pc=
|
||||
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
|
||||
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
@@ -732,8 +730,8 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM=
|
||||
golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU=
|
||||
golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c=
|
||||
golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
|
||||
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
@@ -773,8 +771,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
|
||||
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
|
||||
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
|
||||
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
|
||||
golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA=
|
||||
golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs=
|
||||
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
|
||||
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
@@ -798,8 +796,8 @@ golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
|
||||
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
|
||||
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
@@ -849,8 +847,8 @@ golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
|
||||
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
@@ -862,8 +860,8 @@ golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
|
||||
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
|
||||
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
|
||||
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
|
||||
golang.org/x/term v0.42.0 h1:UiKe+zDFmJobeJ5ggPwOshJIVt6/Ft0rcfrXZDLWAWY=
|
||||
golang.org/x/term v0.42.0/go.mod h1:Dq/D+snpsbazcBG5+F9Q1n2rXV8Ma+71xEjTRufARgY=
|
||||
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
|
||||
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
|
||||
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
@@ -879,8 +877,8 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
|
||||
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
|
||||
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
|
||||
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
@@ -933,8 +931,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
|
||||
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
|
||||
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
|
||||
golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c=
|
||||
golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI=
|
||||
golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
|
||||
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
|
||||
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -126,11 +125,10 @@ type TaskSpec struct {
|
||||
|
||||
// StorageTargetConfig 与 service.AgentStorageTargetConfig 对齐
|
||||
type StorageTargetConfig struct {
|
||||
ID uint `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Config json.RawMessage `json:"config"`
|
||||
TransferMode string `json:"transferMode"`
|
||||
ID uint `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Config json.RawMessage `json:"config"`
|
||||
}
|
||||
|
||||
// GetTaskSpec 拉取任务规格
|
||||
@@ -145,26 +143,13 @@ func (c *MasterClient) GetTaskSpec(ctx context.Context, taskID uint) (*TaskSpec,
|
||||
|
||||
// RecordUpdate 与 service.AgentRecordUpdate 对齐
|
||||
type RecordUpdate struct {
|
||||
Status string `json:"status,omitempty"`
|
||||
FileName string `json:"fileName,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
StorageTargetID uint `json:"storageTargetId,omitempty"`
|
||||
StorageTransferMode string `json:"storageTransferMode,omitempty"`
|
||||
StorageUploadResults []StorageResultItem `json:"storageUploadResults,omitempty"`
|
||||
ErrorMessage string `json:"errorMessage,omitempty"`
|
||||
LogAppend string `json:"logAppend,omitempty"`
|
||||
}
|
||||
|
||||
type StorageResultItem struct {
|
||||
StorageTargetID uint `json:"storageTargetId"`
|
||||
StorageTargetName string `json:"storageTargetName"`
|
||||
Status string `json:"status"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
TransferMode string `json:"transferMode,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Status string `json:"status,omitempty"`
|
||||
FileName string `json:"fileName,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
ErrorMessage string `json:"errorMessage,omitempty"`
|
||||
LogAppend string `json:"logAppend,omitempty"`
|
||||
}
|
||||
|
||||
// UpdateRecord 上报备份记录的状态/日志
|
||||
@@ -173,39 +158,6 @@ func (c *MasterClient) UpdateRecord(ctx context.Context, recordID uint, update R
|
||||
return c.do(ctx, http.MethodPost, path, update, nil)
|
||||
}
|
||||
|
||||
// UploadArtifact streams an artifact through the Master for storage targets
|
||||
// that are not directly reachable from the Agent.
|
||||
func (c *MasterClient) UploadArtifact(ctx context.Context, recordID, targetID uint, objectKey string, size int64, checksum string, reader io.Reader) error {
|
||||
path := fmt.Sprintf("/api/agent/records/%d/artifacts/%d", recordID, targetID)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPut, c.baseURL+path, reader)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// The executor owns and closes the artifact file. Prevent net/http from
|
||||
// closing that underlying reader when it finishes the request body.
|
||||
req.Body = io.NopCloser(reader)
|
||||
req.ContentLength = size
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
req.Header.Set("X-Agent-Token", c.token)
|
||||
req.Header.Set("X-BackupX-Object-Key", objectKey)
|
||||
req.Header.Set("X-BackupX-SHA256", checksum)
|
||||
client := *c.httpClient
|
||||
client.Timeout = 0
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relay artifact to Master: %w", err)
|
||||
}
|
||||
data, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
closeErr := resp.Body.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
return errors.Join(readErr, closeErr)
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("relay artifact to Master: http %d: %s", resp.StatusCode, string(data))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RestoreSpec 与 service.AgentRestoreSpec 对齐
|
||||
type RestoreSpec struct {
|
||||
RestoreRecordID uint `json:"restoreRecordId"`
|
||||
@@ -227,7 +179,6 @@ type RestoreSpec struct {
|
||||
Storage StorageTargetConfig `json:"storage"`
|
||||
StoragePath string `json:"storagePath"`
|
||||
FileName string `json:"fileName"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
}
|
||||
|
||||
// RestoreUpdate 与 service.AgentRestoreUpdate 对齐
|
||||
@@ -247,27 +198,6 @@ func (c *MasterClient) GetRestoreSpec(ctx context.Context, restoreRecordID uint)
|
||||
return &spec, nil
|
||||
}
|
||||
|
||||
func (c *MasterClient) DownloadRestoreArtifact(ctx context.Context, restoreRecordID uint) (io.ReadCloser, error) {
|
||||
path := fmt.Sprintf("/api/agent/restores/%d/artifact", restoreRecordID)
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+path, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("X-Agent-Token", c.token)
|
||||
client := *c.httpClient
|
||||
client.Timeout = 0
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("download relayed artifact from Master: %w", err)
|
||||
}
|
||||
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
|
||||
return resp.Body, nil
|
||||
}
|
||||
data, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
closeErr := resp.Body.Close()
|
||||
return nil, errors.Join(fmt.Errorf("download relayed artifact from Master: http %d: %s", resp.StatusCode, string(data)), readErr, closeErr)
|
||||
}
|
||||
|
||||
// UpdateRestore 上报恢复记录的状态/日志
|
||||
func (c *MasterClient) UpdateRestore(ctx context.Context, restoreRecordID uint, update RestoreUpdate) error {
|
||||
path := fmt.Sprintf("/api/agent/restores/%d", restoreRecordID)
|
||||
|
||||
@@ -26,7 +26,7 @@ type Config struct {
|
||||
HeartbeatInterval string `yaml:"heartbeatInterval"`
|
||||
// PollInterval 命令轮询间隔,默认 5s
|
||||
PollInterval string `yaml:"pollInterval"`
|
||||
// TempDir 备份临时目录,默认 /var/lib/backupx-agent/tmp
|
||||
// TempDir 备份临时目录,默认 /tmp/backupx-agent
|
||||
TempDir string `yaml:"tempDir"`
|
||||
// InsecureSkipTLSVerify 测试环境允许跳过 TLS 证书校验
|
||||
InsecureSkipTLSVerify bool `yaml:"insecureSkipTlsVerify"`
|
||||
@@ -98,7 +98,7 @@ func applyConfigDefaults(cfg *Config) (*Config, error) {
|
||||
cfg.PollInterval = "5s"
|
||||
}
|
||||
if cfg.TempDir == "" {
|
||||
cfg.TempDir = "/var/lib/backupx-agent/tmp"
|
||||
cfg.TempDir = "/tmp/backupx-agent"
|
||||
}
|
||||
cfg.Master = strings.TrimRight(strings.TrimSpace(cfg.Master), "/")
|
||||
return cfg, nil
|
||||
|
||||
@@ -50,7 +50,7 @@ func TestLoadConfigDefaults(t *testing.T) {
|
||||
if cfg.HeartbeatInterval != "15s" || cfg.PollInterval != "5s" {
|
||||
t.Errorf("default intervals not applied: %+v", cfg)
|
||||
}
|
||||
if cfg.TempDir != "/var/lib/backupx-agent/tmp" {
|
||||
if cfg.TempDir != "/tmp/backupx-agent" {
|
||||
t.Errorf("default tempdir: %q", cfg.TempDir)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,8 +4,6 @@ import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
@@ -21,10 +19,10 @@ import (
|
||||
|
||||
// Executor 负责在 Agent 本地执行命令。
|
||||
type Executor struct {
|
||||
client *MasterClient
|
||||
tempDir string
|
||||
backupRegistry *backup.Registry
|
||||
storageRegistry *storage.Registry
|
||||
client *MasterClient
|
||||
tempDir string
|
||||
backupRegistry *backup.Registry
|
||||
storageRegistry *storage.Registry
|
||||
}
|
||||
|
||||
// NewExecutor 构造执行器。预先初始化 backup runner 与 storage registry。
|
||||
@@ -35,7 +33,6 @@ func NewExecutor(client *MasterClient, tempDir string) *Executor {
|
||||
backup.NewMySQLRunner(nil),
|
||||
backup.NewPostgreSQLRunner(nil),
|
||||
backup.NewSAPHANARunner(nil),
|
||||
backup.NewMongoDBRunner(nil),
|
||||
)
|
||||
storageRegistry := storage.NewRegistry(
|
||||
storageRclone.NewLocalDiskFactory(),
|
||||
@@ -62,11 +59,6 @@ func NewExecutor(client *MasterClient, tempDir string) *Executor {
|
||||
// 注意:Agent 当前不支持 Encrypt=true(加密密钥不下发到 Agent,避免密钥扩散)。
|
||||
// 遇到启用加密的任务会向 Master 上报失败并返回错误。
|
||||
func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) error {
|
||||
if err := e.ensureTempDir(); err != nil {
|
||||
e.reportRecordFailure(ctx, recordID, err.Error())
|
||||
return err
|
||||
}
|
||||
|
||||
// 1) 拉取任务规格
|
||||
spec, err := e.client.GetTaskSpec(ctx, taskID)
|
||||
if err != nil {
|
||||
@@ -82,6 +74,10 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
|
||||
|
||||
// 2) 构造 backup.TaskSpec 并找对应 runner
|
||||
startedAt := time.Now().UTC()
|
||||
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
|
||||
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("创建临时目录失败: %v", err))
|
||||
return err
|
||||
}
|
||||
backupSpec := buildBackupTaskSpec(spec, startedAt, e.tempDir)
|
||||
runner, err := e.backupRegistry.Runner(backupSpec.Type)
|
||||
if err != nil {
|
||||
@@ -108,14 +104,6 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
|
||||
return compressErr
|
||||
}
|
||||
finalPath = compressedPath
|
||||
} else if strings.EqualFold(spec.Compression, "zstd") && !strings.HasSuffix(strings.ToLower(finalPath), ".zst") {
|
||||
e.appendLog(ctx, recordID, "[agent] 开始压缩备份文件(zstd)\n")
|
||||
compressedPath, compressErr := compress.ZstdFile(finalPath)
|
||||
if compressErr != nil {
|
||||
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("压缩失败: %v", compressErr))
|
||||
return compressErr
|
||||
}
|
||||
finalPath = compressedPath
|
||||
}
|
||||
info, err := os.Stat(finalPath)
|
||||
if err != nil {
|
||||
@@ -124,7 +112,7 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
|
||||
}
|
||||
fileName := filepath.Base(finalPath)
|
||||
fileSize := info.Size()
|
||||
storagePath := backup.BuildRecordStorageKey(spec.Type, startedAt, recordID, fileName)
|
||||
storagePath := backup.BuildStorageKey(spec.Type, startedAt, fileName)
|
||||
|
||||
// 5) 计算 checksum(一次读一次)并上传到所有目标
|
||||
checksum, err := computeFileSHA256(finalPath)
|
||||
@@ -136,88 +124,48 @@ func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) er
|
||||
e.reportRecordFailure(ctx, recordID, "没有关联的存储目标")
|
||||
return fmt.Errorf("no storage targets")
|
||||
}
|
||||
uploadResults := make([]StorageResultItem, 0, len(spec.StorageTargets))
|
||||
selectedStorageTargetID := uint(0)
|
||||
selectedStorageTransferMode := ""
|
||||
var uploadErrors []string
|
||||
for _, target := range spec.StorageTargets {
|
||||
if err := e.uploadToTarget(ctx, recordID, target, finalPath, storagePath, fileSize, checksum, spec.TaskID); err != nil {
|
||||
uploadResults = append(uploadResults, StorageResultItem{
|
||||
StorageTargetID: target.ID,
|
||||
StorageTargetName: target.Name,
|
||||
Status: "failed",
|
||||
TransferMode: target.TransferMode,
|
||||
Error: err.Error(),
|
||||
})
|
||||
uploadErrors = append(uploadErrors, fmt.Sprintf("%s: %v", target.Name, err))
|
||||
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 上传到存储目标 %s 失败: %v\n", target.Name, err))
|
||||
continue
|
||||
if err := e.uploadToTarget(ctx, recordID, target, finalPath, storagePath, fileSize, spec.TaskID); err != nil {
|
||||
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("上传到 %s 失败: %v", target.Name, err))
|
||||
return err
|
||||
}
|
||||
if selectedStorageTargetID == 0 {
|
||||
selectedStorageTargetID = target.ID
|
||||
selectedStorageTransferMode = target.TransferMode
|
||||
}
|
||||
uploadResults = append(uploadResults, StorageResultItem{
|
||||
StorageTargetID: target.ID,
|
||||
StorageTargetName: target.Name,
|
||||
Status: "success",
|
||||
StoragePath: storagePath,
|
||||
FileSize: fileSize,
|
||||
TransferMode: target.TransferMode,
|
||||
})
|
||||
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 已上传到存储目标 %s\n", target.Name))
|
||||
}
|
||||
if selectedStorageTargetID == 0 {
|
||||
msg := strings.Join(uploadErrors, "; ")
|
||||
if msg == "" {
|
||||
msg = "所有存储目标上传均失败"
|
||||
}
|
||||
e.reportRecordFailureWithUploadResults(ctx, recordID, msg, uploadResults)
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
|
||||
// 6) 上报最终成功
|
||||
return e.client.UpdateRecord(ctx, recordID, RecordUpdate{
|
||||
Status: "success",
|
||||
FileName: fileName,
|
||||
FileSize: fileSize,
|
||||
Checksum: checksum,
|
||||
StoragePath: storagePath,
|
||||
StorageTargetID: selectedStorageTargetID,
|
||||
StorageTransferMode: selectedStorageTransferMode,
|
||||
StorageUploadResults: uploadResults,
|
||||
LogAppend: fmt.Sprintf("[agent] 任务完成,总计 %d 字节\n", fileSize),
|
||||
Status: "success",
|
||||
FileName: fileName,
|
||||
FileSize: fileSize,
|
||||
Checksum: checksum,
|
||||
StoragePath: storagePath,
|
||||
LogAppend: fmt.Sprintf("[agent] 任务完成,总计 %d 字节\n", fileSize),
|
||||
})
|
||||
}
|
||||
|
||||
// uploadToTarget 上传单个目标。为保持简化不做上传级重试(rclone 本身已有 low-level 重试)。
|
||||
func (e *Executor) uploadToTarget(ctx context.Context, recordID uint, target StorageTargetConfig, filePath, objectKey string, fileSize int64, checksum string, taskID uint) error {
|
||||
f, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open artifact: %w", err)
|
||||
}
|
||||
if target.TransferMode == storage.TransferModeMasterRelay {
|
||||
uploadErr := e.client.UploadArtifact(ctx, recordID, target.ID, objectKey, fileSize, checksum, f)
|
||||
return errors.Join(uploadErr, f.Close())
|
||||
}
|
||||
func (e *Executor) uploadToTarget(ctx context.Context, recordID uint, target StorageTargetConfig, filePath, objectKey string, fileSize int64, taskID uint) error {
|
||||
var rawConfig map[string]any
|
||||
if len(target.Config) > 0 {
|
||||
// DecodeRawConfig 通过 json 解析
|
||||
if err := jsonUnmarshalMap(target.Config, &rawConfig); err != nil {
|
||||
return errors.Join(fmt.Errorf("parse storage config: %w", err), f.Close())
|
||||
return fmt.Errorf("parse storage config: %w", err)
|
||||
}
|
||||
}
|
||||
provider, err := e.storageRegistry.Create(ctx, target.Type, rawConfig)
|
||||
if err != nil {
|
||||
closeErr := f.Close()
|
||||
return errors.Join(fmt.Errorf("create provider: %w", err), closeErr)
|
||||
return fmt.Errorf("create provider: %w", err)
|
||||
}
|
||||
f, err := os.Open(filePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open artifact: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
meta := map[string]string{
|
||||
"taskId": fmt.Sprintf("%d", taskID),
|
||||
"recordId": fmt.Sprintf("%d", recordID),
|
||||
}
|
||||
uploadErr := provider.Upload(ctx, objectKey, f, fileSize, meta)
|
||||
return errors.Join(uploadErr, f.Close())
|
||||
return provider.Upload(ctx, objectKey, f, fileSize, meta)
|
||||
}
|
||||
|
||||
// appendLog 追加日志到 Master 记录(尽力而为,失败不中断主流程)
|
||||
@@ -227,22 +175,31 @@ func (e *Executor) appendLog(ctx context.Context, recordID uint, line string) {
|
||||
|
||||
// reportRecordFailure 上报失败状态
|
||||
func (e *Executor) reportRecordFailure(ctx context.Context, recordID uint, msg string) {
|
||||
e.reportRecordFailureWithUploadResults(ctx, recordID, msg, nil)
|
||||
}
|
||||
|
||||
func (e *Executor) reportRecordFailureWithUploadResults(ctx context.Context, recordID uint, msg string, uploadResults []StorageResultItem) {
|
||||
_ = e.client.UpdateRecord(ctx, recordID, RecordUpdate{
|
||||
Status: "failed",
|
||||
ErrorMessage: msg,
|
||||
StorageUploadResults: uploadResults,
|
||||
LogAppend: fmt.Sprintf("[agent] 错误: %s\n", msg),
|
||||
Status: "failed",
|
||||
ErrorMessage: msg,
|
||||
LogAppend: fmt.Sprintf("[agent] 错误: %s\n", msg),
|
||||
})
|
||||
}
|
||||
|
||||
// buildBackupTaskSpec 把 AgentTaskSpec 转换为 backup.TaskSpec。
|
||||
func buildBackupTaskSpec(spec *TaskSpec, startedAt time.Time, tempDir string) backup.TaskSpec {
|
||||
sourcePaths := parseStringListField(spec.SourcePaths)
|
||||
excludes := parseStringListField(spec.ExcludePatterns)
|
||||
var sourcePaths []string
|
||||
if strings.TrimSpace(spec.SourcePaths) != "" {
|
||||
for _, p := range strings.Split(spec.SourcePaths, "\n") {
|
||||
if p = strings.TrimSpace(p); p != "" {
|
||||
sourcePaths = append(sourcePaths, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
var excludes []string
|
||||
if strings.TrimSpace(spec.ExcludePatterns) != "" {
|
||||
for _, p := range strings.Split(spec.ExcludePatterns, "\n") {
|
||||
if p = strings.TrimSpace(p); p != "" {
|
||||
excludes = append(excludes, p)
|
||||
}
|
||||
}
|
||||
}
|
||||
return backup.TaskSpec{
|
||||
ID: spec.TaskID,
|
||||
Name: spec.Name,
|
||||
@@ -265,37 +222,6 @@ func buildBackupTaskSpec(spec *TaskSpec, startedAt time.Time, tempDir string) ba
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Executor) ensureTempDir() error {
|
||||
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
|
||||
return fmt.Errorf("create agent temp dir: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseStringListField(value string) []string {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" || trimmed == "[]" {
|
||||
return nil
|
||||
}
|
||||
var jsonItems []string
|
||||
if err := json.Unmarshal([]byte(trimmed), &jsonItems); err == nil {
|
||||
return compactStringList(jsonItems)
|
||||
}
|
||||
return compactStringList(strings.FieldsFunc(trimmed, func(r rune) bool {
|
||||
return r == '\n' || r == '\r'
|
||||
}))
|
||||
}
|
||||
|
||||
func compactStringList(items []string) []string {
|
||||
result := make([]string, 0, len(items))
|
||||
for _, item := range items {
|
||||
if trimmed := strings.TrimSpace(item); trimmed != "" {
|
||||
result = append(result, trimmed)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// recordLogger 把 runner 日志回传到 Master 记录。
|
||||
// 实现 backup.LogWriter,每条日志追加到 record.log_content。
|
||||
type recordLogger struct {
|
||||
@@ -314,8 +240,8 @@ func (l *recordLogger) WriteLine(message string) {
|
||||
|
||||
// restoreLogger 把 runner 日志回传到 Master 恢复记录。
|
||||
type restoreLogger struct {
|
||||
ctx context.Context
|
||||
client *MasterClient
|
||||
ctx context.Context
|
||||
client *MasterClient
|
||||
restoreID uint
|
||||
}
|
||||
|
||||
@@ -339,16 +265,11 @@ func (e *Executor) DeleteStorageObject(ctx context.Context, targetType string, t
|
||||
// ExecuteRestore 处理 restore_record 命令:拉规格 → 下载 → 解压 → 执行 runner.Restore → 上报结果。
|
||||
//
|
||||
// 与 ExecuteRunTask 对称,但方向相反:
|
||||
// - 下载:直连共享存储,或通过 Master 中转其本地磁盘对象
|
||||
// - 下载:通过 spec.Storage 创建 provider → Download(spec.StoragePath)
|
||||
// - 解密:当前 Agent 不支持加密恢复(密钥未下发),spec.Encrypt=true 会直接失败
|
||||
// - 执行:backup.Registry.Runner(spec.Type).Restore
|
||||
// - 上报:通过 UpdateRestore(status/logAppend)
|
||||
func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) error {
|
||||
if err := e.ensureTempDir(); err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, err.Error())
|
||||
return err
|
||||
}
|
||||
|
||||
spec, err := e.client.GetRestoreSpec(ctx, restoreRecordID)
|
||||
if err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("拉取恢复规格失败: %v", err))
|
||||
@@ -361,6 +282,10 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
|
||||
}
|
||||
e.appendRestoreLog(ctx, restoreRecordID, fmt.Sprintf("[agent] 开始恢复 %s (type=%s)\n", spec.TaskName, spec.Type))
|
||||
|
||||
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建临时目录失败: %v", err))
|
||||
return err
|
||||
}
|
||||
tmpDir, err := os.MkdirTemp(e.tempDir, "restore-*")
|
||||
if err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建恢复临时目录失败: %v", err))
|
||||
@@ -368,31 +293,28 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
// 1) 下载
|
||||
// 1) 创建 storage provider
|
||||
var rawConfig map[string]any
|
||||
if len(spec.Storage.Config) > 0 {
|
||||
if err := jsonUnmarshalMap(spec.Storage.Config, &rawConfig); err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解析存储配置失败: %v", err))
|
||||
return err
|
||||
}
|
||||
}
|
||||
provider, err := e.storageRegistry.Create(ctx, spec.Storage.Type, rawConfig)
|
||||
if err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建存储客户端失败: %v", err))
|
||||
return err
|
||||
}
|
||||
|
||||
// 2) 下载
|
||||
fileName := spec.FileName
|
||||
if strings.TrimSpace(fileName) == "" {
|
||||
fileName = filepath.Base(spec.StoragePath)
|
||||
}
|
||||
artifactPath := filepath.Join(tmpDir, filepath.Base(fileName))
|
||||
e.appendRestoreLog(ctx, restoreRecordID, fmt.Sprintf("[agent] 下载备份文件 %s\n", spec.StoragePath))
|
||||
var reader io.ReadCloser
|
||||
if spec.Storage.TransferMode == storage.TransferModeMasterRelay {
|
||||
reader, err = e.client.DownloadRestoreArtifact(ctx, restoreRecordID)
|
||||
} else {
|
||||
var rawConfig map[string]any
|
||||
if len(spec.Storage.Config) > 0 {
|
||||
if err := jsonUnmarshalMap(spec.Storage.Config, &rawConfig); err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解析存储配置失败: %v", err))
|
||||
return err
|
||||
}
|
||||
}
|
||||
provider, providerErr := e.storageRegistry.Create(ctx, spec.Storage.Type, rawConfig)
|
||||
if providerErr != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("创建存储客户端失败: %v", providerErr))
|
||||
return providerErr
|
||||
}
|
||||
reader, err = provider.Download(ctx, spec.StoragePath)
|
||||
}
|
||||
reader, err := provider.Download(ctx, spec.StoragePath)
|
||||
if err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("下载备份失败: %v", err))
|
||||
return err
|
||||
@@ -402,24 +324,6 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
|
||||
return err
|
||||
}
|
||||
|
||||
// 2.5) 完整性校验:还原前比对下载对象的 SHA-256(与 Master 本地恢复路径一致)。
|
||||
// 拒绝还原损坏或被篡改的备份;早期无 checksum 的备份跳过(向后兼容)。
|
||||
if strings.TrimSpace(spec.Checksum) != "" {
|
||||
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 校验备份完整性(SHA-256)\n")
|
||||
actual, sumErr := computeFileSHA256(artifactPath)
|
||||
if sumErr != nil {
|
||||
msg := fmt.Sprintf("计算校验和失败: %v", sumErr)
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, msg)
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
if !strings.EqualFold(actual, spec.Checksum) {
|
||||
msg := "备份文件完整性校验失败:SHA-256 不匹配,文件可能已损坏或被篡改"
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, msg)
|
||||
return fmt.Errorf("%s(期望 %s,实际 %s)", msg, spec.Checksum, actual)
|
||||
}
|
||||
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 完整性校验通过\n")
|
||||
}
|
||||
|
||||
// 3) 解压(Agent 不支持加密,遇到 .enc 会直接失败)
|
||||
preparedPath := artifactPath
|
||||
if strings.HasSuffix(strings.ToLower(preparedPath), ".enc") {
|
||||
@@ -436,15 +340,6 @@ func (e *Executor) ExecuteRestore(ctx context.Context, restoreRecordID uint) err
|
||||
}
|
||||
preparedPath = decompressed
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(preparedPath), ".zst") {
|
||||
e.appendRestoreLog(ctx, restoreRecordID, "[agent] 解压 zstd 压缩\n")
|
||||
decompressed, err := compress.UnzstdFile(preparedPath)
|
||||
if err != nil {
|
||||
e.reportRestoreFailure(ctx, restoreRecordID, fmt.Sprintf("解压失败: %v", err))
|
||||
return err
|
||||
}
|
||||
preparedPath = decompressed
|
||||
}
|
||||
|
||||
// 4) 运行 runner.Restore
|
||||
taskSpec := buildRestoreBackupTaskSpec(spec, time.Now().UTC(), tmpDir)
|
||||
@@ -503,10 +398,8 @@ func buildRestoreBackupTaskSpec(spec *RestoreSpec, startedAt time.Time, tempDir
|
||||
}
|
||||
|
||||
// writeReaderToLocal 把 reader 写到本地文件(Agent 侧工具函数)。
|
||||
func writeReaderToLocal(targetPath string, reader io.ReadCloser) (err error) {
|
||||
defer func() {
|
||||
err = errors.Join(err, reader.Close())
|
||||
}()
|
||||
func writeReaderToLocal(targetPath string, reader io.ReadCloser) error {
|
||||
defer reader.Close()
|
||||
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -514,8 +407,9 @@ func writeReaderToLocal(targetPath string, reader io.ReadCloser) (err error) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, copyErr := io.Copy(file, reader)
|
||||
return errors.Join(copyErr, file.Close())
|
||||
defer file.Close()
|
||||
_, err = io.Copy(file, reader)
|
||||
return err
|
||||
}
|
||||
|
||||
// 辅助函数
|
||||
|
||||
@@ -1,380 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/storage"
|
||||
)
|
||||
|
||||
func TestBuildBackupTaskSpecParsesJSONSourcePaths(t *testing.T) {
|
||||
spec := &TaskSpec{
|
||||
TaskID: 7,
|
||||
Name: "root-files",
|
||||
Type: "file",
|
||||
SourcePaths: `["/root","/etc"]`,
|
||||
ExcludePatterns: `["*.log","tmp"]`,
|
||||
}
|
||||
|
||||
got := buildBackupTaskSpec(spec, time.Unix(0, 0), "/var/lib/backupx-agent/tmp")
|
||||
|
||||
if !reflect.DeepEqual(got.SourcePaths, []string{"/root", "/etc"}) {
|
||||
t.Fatalf("source paths = %#v", got.SourcePaths)
|
||||
}
|
||||
if !reflect.DeepEqual(got.ExcludePatterns, []string{"*.log", "tmp"}) {
|
||||
t.Fatalf("exclude patterns = %#v", got.ExcludePatterns)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseStringListFieldKeepsLegacyLineFormat(t *testing.T) {
|
||||
got := parseStringListField("/root\n /etc \n")
|
||||
want := []string{"/root", "/etc"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunTaskRecordsPerTargetUploadResults(t *testing.T) {
|
||||
sourceDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("hello"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile returned error: %v", err)
|
||||
}
|
||||
var finalUpdate RecordUpdate
|
||||
var updates []RecordUpdate
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
|
||||
writeAgentEnvelope(t, w, TaskSpec{
|
||||
TaskID: 1,
|
||||
Name: "site",
|
||||
Type: "file",
|
||||
SourcePath: sourceDir,
|
||||
Compression: "gzip",
|
||||
StorageTargets: []StorageTargetConfig{
|
||||
{ID: 11, Name: "broken", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken"}`)},
|
||||
{ID: 12, Name: "ok", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"ok"}`)},
|
||||
},
|
||||
})
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
|
||||
var update RecordUpdate
|
||||
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
|
||||
t.Fatalf("Decode update returned error: %v", err)
|
||||
}
|
||||
updates = append(updates, update)
|
||||
if update.Status != "" {
|
||||
finalUpdate = update
|
||||
}
|
||||
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
|
||||
executor.storageRegistry = storage.NewRegistry(&agentTestStorageFactory{
|
||||
providers: map[string]*agentTestStorageProvider{
|
||||
"broken": {name: "broken", failUpload: true},
|
||||
"ok": {name: "ok", objects: map[string][]byte{}},
|
||||
},
|
||||
})
|
||||
|
||||
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err != nil {
|
||||
t.Fatalf("ExecuteRunTask returned error: %v", err)
|
||||
}
|
||||
if len(updates) == 0 || finalUpdate.Status != "success" {
|
||||
t.Fatalf("expected final success update, got updates=%#v final=%#v", updates, finalUpdate)
|
||||
}
|
||||
if finalUpdate.StorageTargetID != 12 {
|
||||
t.Fatalf("expected first successful target 12, got %d", finalUpdate.StorageTargetID)
|
||||
}
|
||||
if len(finalUpdate.StorageUploadResults) != 2 {
|
||||
t.Fatalf("expected two upload results, got %#v", finalUpdate.StorageUploadResults)
|
||||
}
|
||||
if finalUpdate.StorageUploadResults[0].Status != "failed" || finalUpdate.StorageUploadResults[1].Status != "success" {
|
||||
t.Fatalf("unexpected upload results: %#v", finalUpdate.StorageUploadResults)
|
||||
}
|
||||
if finalUpdate.StoragePath == "" || finalUpdate.FileSize <= 0 || finalUpdate.Checksum == "" {
|
||||
t.Fatalf("expected artifact metadata in final update, got %#v", finalUpdate)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunTaskRelaysMasterLocalDiskTarget(t *testing.T) {
|
||||
sourceDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("centralize me"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile returned error: %v", err)
|
||||
}
|
||||
var relayed []byte
|
||||
var finalUpdate RecordUpdate
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
|
||||
writeAgentEnvelope(t, w, TaskSpec{
|
||||
TaskID: 1,
|
||||
Name: "remote-source",
|
||||
Type: "file",
|
||||
SourcePath: sourceDir,
|
||||
Compression: "gzip",
|
||||
StorageTargets: []StorageTargetConfig{{
|
||||
ID: 11, Name: "master-disk", Type: storage.TypeLocalDisk, TransferMode: storage.TransferModeMasterRelay,
|
||||
}},
|
||||
})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/agent/records/99/artifacts/11":
|
||||
body, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadAll relayed body: %v", err)
|
||||
}
|
||||
digest := sha256.Sum256(body)
|
||||
if got := r.Header.Get("X-BackupX-SHA256"); got != fmt.Sprintf("%x", digest[:]) {
|
||||
t.Fatalf("relay checksum header = %q", got)
|
||||
}
|
||||
objectKey := r.Header.Get("X-BackupX-Object-Key")
|
||||
if !strings.Contains(objectKey, "/records/99/") || r.ContentLength != int64(len(body)) {
|
||||
t.Fatalf("invalid relay metadata: key=%q length=%d body=%d", objectKey, r.ContentLength, len(body))
|
||||
}
|
||||
relayed = append([]byte(nil), body...)
|
||||
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
|
||||
var update RecordUpdate
|
||||
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
|
||||
t.Fatalf("Decode update returned error: %v", err)
|
||||
}
|
||||
if update.Status != "" {
|
||||
finalUpdate = update
|
||||
}
|
||||
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
|
||||
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err != nil {
|
||||
t.Fatalf("ExecuteRunTask returned error: %v", err)
|
||||
}
|
||||
if len(relayed) == 0 {
|
||||
t.Fatal("expected artifact bytes to be streamed through Master")
|
||||
}
|
||||
if finalUpdate.Status != "success" || finalUpdate.StorageTransferMode != storage.TransferModeMasterRelay {
|
||||
t.Fatalf("unexpected final relay update: %#v", finalUpdate)
|
||||
}
|
||||
if len(finalUpdate.StorageUploadResults) != 1 || finalUpdate.StorageUploadResults[0].TransferMode != storage.TransferModeMasterRelay {
|
||||
t.Fatalf("unexpected relay target result: %#v", finalUpdate.StorageUploadResults)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreDownloadsMasterRelayedArtifact(t *testing.T) {
|
||||
var archive bytes.Buffer
|
||||
tarWriter := tar.NewWriter(&archive)
|
||||
content := []byte("restored through Master")
|
||||
header := &tar.Header{Name: "site/index.html", Mode: 0o644, Size: int64(len(content)), Typeflag: tar.TypeReg}
|
||||
if err := tarWriter.WriteHeader(header); err != nil {
|
||||
t.Fatalf("WriteHeader returned error: %v", err)
|
||||
}
|
||||
if _, err := tarWriter.Write(content); err != nil {
|
||||
t.Fatalf("Write returned error: %v", err)
|
||||
}
|
||||
if err := tarWriter.Close(); err != nil {
|
||||
t.Fatalf("Close returned error: %v", err)
|
||||
}
|
||||
artifact := append([]byte(nil), archive.Bytes()...)
|
||||
digest := sha256.Sum256(artifact)
|
||||
restoreRoot := t.TempDir()
|
||||
restoreSource := filepath.Join(restoreRoot, "site")
|
||||
artifactRequests := 0
|
||||
var finalUpdate RestoreUpdate
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/restores/77/spec":
|
||||
writeAgentEnvelope(t, w, RestoreSpec{
|
||||
RestoreRecordID: 77,
|
||||
BackupRecordID: 99,
|
||||
TaskID: 1,
|
||||
TaskName: "remote-source",
|
||||
Type: "file",
|
||||
SourcePath: restoreSource,
|
||||
Storage: StorageTargetConfig{
|
||||
ID: 11, Name: "master-disk", Type: storage.TypeLocalDisk, TransferMode: storage.TransferModeMasterRelay,
|
||||
},
|
||||
StoragePath: "BackupX/file/site.tar",
|
||||
FileName: "site.tar",
|
||||
Checksum: fmt.Sprintf("%x", digest[:]),
|
||||
})
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/restores/77/artifact":
|
||||
artifactRequests++
|
||||
if r.Header.Get("X-Agent-Token") != "token" {
|
||||
t.Fatalf("missing Agent token on relay download")
|
||||
}
|
||||
w.Header().Set("Content-Length", fmt.Sprintf("%d", len(artifact)))
|
||||
_, _ = w.Write(artifact)
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/restores/77":
|
||||
var update RestoreUpdate
|
||||
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
|
||||
t.Fatalf("Decode update returned error: %v", err)
|
||||
}
|
||||
if update.Status != "" {
|
||||
finalUpdate = update
|
||||
}
|
||||
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
|
||||
if err := executor.ExecuteRestore(context.Background(), 77); err != nil {
|
||||
t.Fatalf("ExecuteRestore returned error: %v", err)
|
||||
}
|
||||
if artifactRequests != 1 {
|
||||
t.Fatalf("expected one relay artifact request, got %d", artifactRequests)
|
||||
}
|
||||
restored, err := os.ReadFile(filepath.Join(restoreRoot, "site", "index.html"))
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile returned error: %v", err)
|
||||
}
|
||||
if string(restored) != string(content) {
|
||||
t.Fatalf("restored content = %q, want %q", restored, content)
|
||||
}
|
||||
if finalUpdate.Status != "success" {
|
||||
t.Fatalf("unexpected final restore update: %#v", finalUpdate)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunTaskReportsPerTargetUploadResultsWhenAllTargetsFail(t *testing.T) {
|
||||
sourceDir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(sourceDir, "index.html"), []byte("hello"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile returned error: %v", err)
|
||||
}
|
||||
var finalUpdate RecordUpdate
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/agent/tasks/1":
|
||||
writeAgentEnvelope(t, w, TaskSpec{
|
||||
TaskID: 1,
|
||||
Name: "site",
|
||||
Type: "file",
|
||||
SourcePath: sourceDir,
|
||||
Compression: "gzip",
|
||||
StorageTargets: []StorageTargetConfig{
|
||||
{ID: 11, Name: "broken-a", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken-a"}`)},
|
||||
{ID: 12, Name: "broken-b", Type: "agent_test_storage", Config: json.RawMessage(`{"name":"broken-b"}`)},
|
||||
},
|
||||
})
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/agent/records/99":
|
||||
var update RecordUpdate
|
||||
if err := json.NewDecoder(r.Body).Decode(&update); err != nil {
|
||||
t.Fatalf("Decode update returned error: %v", err)
|
||||
}
|
||||
if update.Status != "" {
|
||||
finalUpdate = update
|
||||
}
|
||||
writeAgentEnvelope(t, w, map[string]string{"status": "ok"})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
executor := NewExecutor(NewMasterClient(server.URL, "token", false), filepath.Join(t.TempDir(), "tmp"))
|
||||
executor.storageRegistry = storage.NewRegistry(&agentTestStorageFactory{
|
||||
providers: map[string]*agentTestStorageProvider{
|
||||
"broken-a": {name: "broken-a", failUpload: true},
|
||||
"broken-b": {name: "broken-b", failUpload: true},
|
||||
},
|
||||
})
|
||||
|
||||
if err := executor.ExecuteRunTask(context.Background(), 1, 99); err == nil {
|
||||
t.Fatal("expected ExecuteRunTask to return upload failure")
|
||||
}
|
||||
if finalUpdate.Status != "failed" {
|
||||
t.Fatalf("expected final failed update, got %#v", finalUpdate)
|
||||
}
|
||||
if len(finalUpdate.StorageUploadResults) != 2 {
|
||||
t.Fatalf("expected failed update to keep per-target results, got %#v", finalUpdate.StorageUploadResults)
|
||||
}
|
||||
for _, item := range finalUpdate.StorageUploadResults {
|
||||
if item.Status != "failed" || item.Error == "" {
|
||||
t.Fatalf("unexpected upload result: %#v", item)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type agentTestStorageFactory struct {
|
||||
providers map[string]*agentTestStorageProvider
|
||||
}
|
||||
|
||||
func (f *agentTestStorageFactory) Type() storage.ProviderType {
|
||||
return "agent_test_storage"
|
||||
}
|
||||
|
||||
func (f *agentTestStorageFactory) New(_ context.Context, config map[string]any) (storage.StorageProvider, error) {
|
||||
name, _ := config["name"].(string)
|
||||
provider := f.providers[name]
|
||||
if provider == nil {
|
||||
return nil, fmt.Errorf("unknown provider %q", name)
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
type agentTestStorageProvider struct {
|
||||
name string
|
||||
failUpload bool
|
||||
objects map[string][]byte
|
||||
}
|
||||
|
||||
func (p *agentTestStorageProvider) Type() storage.ProviderType { return "agent_test_storage" }
|
||||
func (p *agentTestStorageProvider) TestConnection(context.Context) error {
|
||||
return nil
|
||||
}
|
||||
func (p *agentTestStorageProvider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
|
||||
if p.failUpload {
|
||||
return fmt.Errorf("upload failed for %s", p.name)
|
||||
}
|
||||
data, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if p.objects == nil {
|
||||
p.objects = map[string][]byte{}
|
||||
}
|
||||
p.objects[objectKey] = data
|
||||
return nil
|
||||
}
|
||||
func (p *agentTestStorageProvider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
|
||||
data, ok := p.objects[objectKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("object %s not found", objectKey)
|
||||
}
|
||||
return io.NopCloser(strings.NewReader(string(data))), nil
|
||||
}
|
||||
func (p *agentTestStorageProvider) Delete(_ context.Context, objectKey string) error {
|
||||
delete(p.objects, objectKey)
|
||||
return nil
|
||||
}
|
||||
func (p *agentTestStorageProvider) List(context.Context, string) ([]storage.ObjectInfo, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func writeAgentEnvelope(t *testing.T, w http.ResponseWriter, data any) {
|
||||
t.Helper()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(map[string]any{"code": "OK", "data": data}); err != nil {
|
||||
t.Fatalf("Encode response returned error: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// DirEntry Agent 返回给 Master 的目录项。
|
||||
@@ -18,8 +17,8 @@ type DirEntry struct {
|
||||
|
||||
// listLocalDir 列出 Agent 所在机器的指定路径。
|
||||
func listLocalDir(path string) ([]DirEntry, error) {
|
||||
cleaned := filepath.Clean(strings.TrimSpace(path))
|
||||
if strings.TrimSpace(path) == "" || cleaned == "." {
|
||||
cleaned := filepath.Clean(path)
|
||||
if cleaned == "" {
|
||||
cleaned = "/"
|
||||
}
|
||||
entries, err := os.ReadDir(cleaned)
|
||||
|
||||
@@ -36,21 +36,6 @@ func TestListLocalDir(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestListLocalDirEmptyPathUsesRoot(t *testing.T) {
|
||||
entries, err := listLocalDir("")
|
||||
if err != nil {
|
||||
t.Fatalf("list root: %v", err)
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
t.Fatalf("expected root entries")
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if !filepath.IsAbs(entry.Path) {
|
||||
t.Fatalf("entry path should be absolute: %+v", entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitCommaOrNewline(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
|
||||
@@ -82,9 +82,9 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
backupTaskService := service.NewBackupTaskService(backupTaskRepo, storageTargetRepo, configCipher)
|
||||
backupTaskService.SetRecordsAndStorage(backupRecordRepo, storageRegistry)
|
||||
// nodeRepo 在下方 Cluster 节点管理区块才实例化,这里延后注入
|
||||
backupRunnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil), backup.NewSAPHANARunner(nil), backup.NewMongoDBRunner(nil))
|
||||
backupRunnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil), backup.NewSAPHANARunner(nil))
|
||||
logHub := backup.NewLogHub()
|
||||
retentionService := backupretention.NewService(backupRecordRepo, configCipher.Key())
|
||||
retentionService := backupretention.NewService(backupRecordRepo)
|
||||
notifyRegistry := notify.NewRegistry(notify.NewEmailNotifier(), notify.NewWebhookNotifier(), notify.NewTelegramNotifier())
|
||||
notificationService := service.NewNotificationService(notificationRepo, notifyRegistry, configCipher)
|
||||
authService.SetNotificationService(notificationService)
|
||||
@@ -104,7 +104,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
restoreRecordRepo := repository.NewRestoreRecordRepository(db)
|
||||
restoreLogHub := backup.NewLogHub()
|
||||
dashboardService := service.NewDashboardService(backupTaskRepo, backupRecordRepo, storageTargetRepo)
|
||||
reportService := service.NewReportService(backupTaskRepo, backupRecordRepo)
|
||||
settingsService := service.NewSettingsService(systemConfigRepo)
|
||||
|
||||
// Audit
|
||||
@@ -114,8 +113,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
schedulerService.SetAuditRecorder(auditService)
|
||||
// 审计日志外输:启动时用当前 settings 初始化 webhook,后续前端修改立即生效
|
||||
settingsService.SetAuditWebhookConfigurer(ctx, auditService)
|
||||
// 审计日志保留期清理:每 6h 读取 audit_retention_days 设置并清理超期日志(0/缺省=永久保留)
|
||||
auditService.StartRetentionMonitor(ctx, systemConfigRepo, 6*time.Hour)
|
||||
|
||||
// Database discovery(集群依赖在 agentService 创建后注入)
|
||||
databaseDiscoveryService := service.NewDatabaseDiscoveryService(backup.NewOSCommandExecutor())
|
||||
@@ -134,8 +131,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
|
||||
// Agent 协议服务:命令队列 + 任务下发 + 记录上报
|
||||
agentCmdRepo := repository.NewAgentCommandRepository(db)
|
||||
nodeService.SetAgentCommandRepository(agentCmdRepo)
|
||||
agentService := service.NewAgentService(nodeRepo, backupTaskRepo, backupRecordRepo, storageTargetRepo, agentCmdRepo, configCipher, storageRegistry)
|
||||
agentService := service.NewAgentService(nodeRepo, backupTaskRepo, backupRecordRepo, storageTargetRepo, agentCmdRepo, configCipher)
|
||||
agentService.SetRestoreRepository(restoreRecordRepo)
|
||||
agentService.StartCommandTimeoutMonitor(ctx, 30*time.Second, 10*time.Minute)
|
||||
|
||||
@@ -244,7 +240,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
replicationService.SetMetrics(appMetrics)
|
||||
metricsCollector := metrics.NewCollector(
|
||||
appMetrics,
|
||||
metrics.NewRepoSource(storageTargetRepo, backupRecordRepo, nodeRepo, backupTaskRepo, agentCmdRepo),
|
||||
metrics.NewRepoSource(storageTargetRepo, backupRecordRepo, nodeRepo, backupTaskRepo),
|
||||
30*time.Second,
|
||||
)
|
||||
metricsCollector.Start(ctx)
|
||||
@@ -271,7 +267,6 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
ApiKeyService: apiKeyService,
|
||||
NotificationService: notificationService,
|
||||
DashboardService: dashboardService,
|
||||
ReportService: reportService,
|
||||
SettingsService: settingsService,
|
||||
NodeService: nodeService,
|
||||
AgentService: agentService,
|
||||
@@ -281,7 +276,7 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
|
||||
UserRepository: userRepo,
|
||||
SystemConfigRepo: systemConfigRepo,
|
||||
InstallTokenService: installTokenService,
|
||||
MasterExternalURL: cfg.Server.ExternalURL,
|
||||
MasterExternalURL: "", // 如需覆盖 URL,可扩展 cfg.Server 增字段;目前留空依赖 X-Forwarded-* / Request.Host
|
||||
DB: db,
|
||||
Metrics: appMetrics,
|
||||
})
|
||||
|
||||
@@ -3,7 +3,6 @@ package backup
|
||||
import (
|
||||
"archive/tar"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
@@ -53,20 +52,6 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
|
||||
defer tw.Close()
|
||||
|
||||
excludes := normalizeExcludePatterns(task.ExcludePatterns)
|
||||
|
||||
// 差异备份:基于上次全量清单仅打包新增/变更条目并记录删除;
|
||||
// 全量备份:记录完整清单(manifest)供后续差异比对。
|
||||
differential := task.Differential && len(task.BaseManifest.Entries) > 0
|
||||
baseIndex := map[string]ManifestEntry{}
|
||||
seen := map[string]struct{}{}
|
||||
var manifest *Manifest
|
||||
if differential {
|
||||
baseIndex = task.BaseManifest.index()
|
||||
writer.WriteLine(fmt.Sprintf("差异备份模式:基线含 %d 个条目", len(baseIndex)))
|
||||
} else {
|
||||
manifest = &Manifest{Entries: make([]ManifestEntry, 0)}
|
||||
}
|
||||
|
||||
totalFileCount := 0
|
||||
totalDirCount := 0
|
||||
|
||||
@@ -103,16 +88,6 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
|
||||
return nil
|
||||
}
|
||||
|
||||
entry := entryFromInfo(archiveName, currentInfo)
|
||||
if differential {
|
||||
seen[entry.Path] = struct{}{}
|
||||
if !changedSince(baseIndex, entry) {
|
||||
return nil // 自全量以来未变更,跳过
|
||||
}
|
||||
} else {
|
||||
manifest.Entries = append(manifest.Entries, entry)
|
||||
}
|
||||
|
||||
if currentInfo.IsDir() {
|
||||
dirCount++
|
||||
writer.WriteLine(fmt.Sprintf("📁 进入目录 %s", archiveName))
|
||||
@@ -128,19 +103,13 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
|
||||
}
|
||||
|
||||
if currentInfo.Mode().IsRegular() {
|
||||
// 每个文件在独立作用域内打开并关闭,避免在大目录树中累积打开的文件句柄。
|
||||
if copyErr := func() error {
|
||||
file, err := os.Open(currentPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer file.Close()
|
||||
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}(); copyErr != nil {
|
||||
return copyErr
|
||||
file, err := os.Open(currentPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer file.Close()
|
||||
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
|
||||
return err
|
||||
}
|
||||
fileCount++
|
||||
if fileCount%100 == 0 {
|
||||
@@ -161,16 +130,10 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
|
||||
totalDirCount += dirCount
|
||||
}
|
||||
|
||||
if differential {
|
||||
deletions := deletedPaths(baseIndex, seen)
|
||||
if err := writeDeletionsEntry(tw, deletions); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
writer.WriteLine(fmt.Sprintf("差异备份完成(%d 个目录、%d 个文件变更,删除 %d 项)", totalDirCount, totalFileCount, len(deletions)))
|
||||
} else if len(sourcePaths) > 1 {
|
||||
if len(sourcePaths) > 1 {
|
||||
writer.WriteLine(fmt.Sprintf("全部源路径打包完成(共 %d 个目录,%d 个文件)", totalDirCount, totalFileCount))
|
||||
}
|
||||
return &RunResult{ArtifactPath: artifactPath, FileName: filepath.Base(artifactPath), TempDir: tempDir, Manifest: manifest}, nil
|
||||
return &RunResult{ArtifactPath: artifactPath, FileName: filepath.Base(artifactPath), TempDir: tempDir}, nil
|
||||
}
|
||||
|
||||
func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath string, writer LogWriter) error {
|
||||
@@ -185,15 +148,9 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
|
||||
restoreSource = task.SourcePaths[0]
|
||||
}
|
||||
targetParent := filepath.Dir(filepath.Clean(strings.TrimSpace(restoreSource)))
|
||||
// 恢复到指定位置:非空时归档解压到用户指定目录,而非原始源父目录。
|
||||
if override := strings.TrimSpace(task.RestoreTargetPath); override != "" {
|
||||
targetParent = filepath.Clean(override)
|
||||
writer.WriteLine(fmt.Sprintf("恢复到指定目录:%s", targetParent))
|
||||
}
|
||||
if err := os.MkdirAll(targetParent, 0o755); err != nil {
|
||||
return fmt.Errorf("create restore parent: %w", err)
|
||||
}
|
||||
var pendingDeletions []string
|
||||
tr := tar.NewReader(artifactFile)
|
||||
for {
|
||||
header, err := tr.Next()
|
||||
@@ -203,27 +160,13 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
|
||||
if err != nil {
|
||||
return fmt.Errorf("read tar entry: %w", err)
|
||||
}
|
||||
// 差异归档的删除清单不落地,留待提取完成后统一应用(避免被同批新增条目误删)。
|
||||
if header.Name == deletionsEntryName {
|
||||
data, readErr := io.ReadAll(tr)
|
||||
if readErr != nil {
|
||||
return fmt.Errorf("read deletions entry: %w", readErr)
|
||||
}
|
||||
if jsonErr := json.Unmarshal(data, &pendingDeletions); jsonErr != nil {
|
||||
return fmt.Errorf("parse deletions entry: %w", jsonErr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
cleanName := path.Clean(strings.TrimSpace(header.Name))
|
||||
if cleanName == "." || cleanName == "" {
|
||||
continue
|
||||
}
|
||||
// 选择性恢复:仅提取被选中的文件/目录(及其子项)。
|
||||
if len(task.SelectedPaths) > 0 && !pathSelected(cleanName, task.SelectedPaths) {
|
||||
continue
|
||||
}
|
||||
targetPath, ok := resolveWithinParent(targetParent, cleanName)
|
||||
if !ok {
|
||||
targetPath := filepath.Clean(filepath.Join(targetParent, filepath.FromSlash(cleanName)))
|
||||
parentWithSep := filepath.Clean(targetParent) + string(filepath.Separator)
|
||||
if targetPath != filepath.Clean(targetParent) && !strings.HasPrefix(targetPath, parentWithSep) {
|
||||
return fmt.Errorf("tar entry escapes restore path")
|
||||
}
|
||||
switch header.Typeflag {
|
||||
@@ -248,94 +191,10 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
|
||||
}
|
||||
}
|
||||
}
|
||||
// 选择性恢复时仅对选中范围应用删除,避免误删未选中的文件。
|
||||
if len(task.SelectedPaths) > 0 {
|
||||
pendingDeletions = filterSelectedPaths(pendingDeletions, task.SelectedPaths)
|
||||
}
|
||||
if err := applyDeletions(targetParent, pendingDeletions, writer); err != nil {
|
||||
return err
|
||||
}
|
||||
writer.WriteLine("文件恢复完成")
|
||||
return nil
|
||||
}
|
||||
|
||||
// pathSelected 判断归档条目名是否落在选中集合内(精确匹配或位于选中目录之下)。
|
||||
func pathSelected(name string, selected []string) bool {
|
||||
for _, sel := range selected {
|
||||
clean := path.Clean(strings.TrimSpace(sel))
|
||||
if clean == "" || clean == "." {
|
||||
continue
|
||||
}
|
||||
if name == clean || strings.HasPrefix(name, clean+"/") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// filterSelectedPaths 仅保留落在选中集合内的路径。
|
||||
func filterSelectedPaths(paths []string, selected []string) []string {
|
||||
filtered := make([]string, 0, len(paths))
|
||||
for _, p := range paths {
|
||||
if pathSelected(path.Clean(strings.TrimSpace(p)), selected) {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
return filtered
|
||||
}
|
||||
|
||||
// resolveWithinParent 将归档相对名安全解析为 targetParent 下的绝对路径;
|
||||
// 越界(路径穿越)时返回 ok=false。提取与删除共用此校验,杜绝逃逸。
|
||||
func resolveWithinParent(targetParent, name string) (string, bool) {
|
||||
targetPath := filepath.Clean(filepath.Join(targetParent, filepath.FromSlash(name)))
|
||||
cleanParent := filepath.Clean(targetParent)
|
||||
if targetPath == cleanParent {
|
||||
return targetPath, true
|
||||
}
|
||||
if !strings.HasPrefix(targetPath, cleanParent+string(filepath.Separator)) {
|
||||
return "", false
|
||||
}
|
||||
return targetPath, true
|
||||
}
|
||||
|
||||
// writeDeletionsEntry 将差异备份的删除路径列表写入归档特殊条目。
|
||||
func writeDeletionsEntry(tw *tar.Writer, deletions []string) error {
|
||||
payload, err := json.Marshal(deletions)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal deletions: %w", err)
|
||||
}
|
||||
header := &tar.Header{Name: deletionsEntryName, Mode: 0o600, Size: int64(len(payload)), Typeflag: tar.TypeReg}
|
||||
if err := tw.WriteHeader(header); err != nil {
|
||||
return fmt.Errorf("write deletions header: %w", err)
|
||||
}
|
||||
if _, err := tw.Write(payload); err != nil {
|
||||
return fmt.Errorf("write deletions body: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyDeletions 在基线恢复之上删除差异归档记录的路径(仅差异备份恢复时存在)。
|
||||
// 每个路径经 resolveWithinParent 校验,越界即报错;目标不存在视为已删除。
|
||||
func applyDeletions(targetParent string, deletions []string, writer LogWriter) error {
|
||||
for _, name := range deletions {
|
||||
clean := path.Clean(strings.TrimSpace(name))
|
||||
if clean == "." || clean == "" {
|
||||
continue
|
||||
}
|
||||
targetPath, ok := resolveWithinParent(targetParent, clean)
|
||||
if !ok {
|
||||
return fmt.Errorf("deletion entry escapes restore path")
|
||||
}
|
||||
if err := os.RemoveAll(targetPath); err != nil {
|
||||
return fmt.Errorf("apply deletion %s: %w", clean, err)
|
||||
}
|
||||
}
|
||||
if len(deletions) > 0 {
|
||||
writer.WriteLine(fmt.Sprintf("已应用差异删除 %d 项", len(deletions)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeExcludePatterns(items []string) []string {
|
||||
result := make([]string, 0, len(items))
|
||||
for _, item := range items {
|
||||
|
||||
@@ -1,182 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func diffWrite(t *testing.T, p, content string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatalf("mkdir %s: %v", filepath.Dir(p), err)
|
||||
}
|
||||
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
|
||||
t.Fatalf("write %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
func diffAssertContent(t *testing.T, p, want string) {
|
||||
t.Helper()
|
||||
got, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", p, err)
|
||||
}
|
||||
if string(got) != want {
|
||||
t.Fatalf("%s content = %q, want %q", p, string(got), want)
|
||||
}
|
||||
}
|
||||
|
||||
func diffAssertAbsent(t *testing.T, p string) {
|
||||
t.Helper()
|
||||
if _, err := os.Stat(p); !os.IsNotExist(err) {
|
||||
t.Fatalf("expected %s to be absent, stat err=%v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
func diffArchiveNames(t *testing.T, artifactPath string) map[string]bool {
|
||||
t.Helper()
|
||||
f, err := os.Open(artifactPath)
|
||||
if err != nil {
|
||||
t.Fatalf("open artifact: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
names := map[string]bool{}
|
||||
tr := tar.NewReader(f)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("read tar: %v", err)
|
||||
}
|
||||
names[h.Name] = true
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
// TestFileRunnerDifferentialRoundTrip 验证差异备份的端到端正确性:
|
||||
// 全量 → 修改源(变更/删除/新增)→ 差异 → 链式恢复(全量+差异)→ 结果与修改后源一致。
|
||||
func TestFileRunnerDifferentialRoundTrip(t *testing.T) {
|
||||
work := t.TempDir()
|
||||
src := filepath.Join(work, "src")
|
||||
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
|
||||
diffWrite(t, filepath.Join(src, "b.txt"), "bravo")
|
||||
diffWrite(t, filepath.Join(src, "sub", "c.txt"), "charlie")
|
||||
|
||||
runner := NewFileRunner()
|
||||
|
||||
full, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir()}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("full Run: %v", err)
|
||||
}
|
||||
if full.Manifest == nil || len(full.Manifest.Entries) == 0 {
|
||||
t.Fatalf("full backup must produce a manifest, got %#v", full.Manifest)
|
||||
}
|
||||
|
||||
// 变更 a.txt(内容变长 → size 差异必被检出)、删除 b.txt、新增 d.txt;sub/c.txt 不变
|
||||
diffWrite(t, filepath.Join(src, "a.txt"), "ALPHA-modified-and-longer")
|
||||
if err := os.Remove(filepath.Join(src, "b.txt")); err != nil {
|
||||
t.Fatalf("remove b.txt: %v", err)
|
||||
}
|
||||
diffWrite(t, filepath.Join(src, "d.txt"), "delta")
|
||||
|
||||
diff, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir(), Differential: true, BaseManifest: *full.Manifest}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("differential Run: %v", err)
|
||||
}
|
||||
if diff.Manifest != nil {
|
||||
t.Fatalf("differential backup must not produce a manifest")
|
||||
}
|
||||
|
||||
// 差异归档应包含变更/新增条目与删除清单,但不含未变更的 sub/c.txt
|
||||
names := diffArchiveNames(t, diff.ArtifactPath)
|
||||
if !names["src/a.txt"] || !names["src/d.txt"] {
|
||||
t.Fatalf("differential archive missing changed/new entries: %v", names)
|
||||
}
|
||||
if names["src/sub/c.txt"] {
|
||||
t.Fatalf("differential archive should not contain unchanged file sub/c.txt")
|
||||
}
|
||||
if !names[deletionsEntryName] {
|
||||
t.Fatalf("differential archive missing deletions entry: %v", names)
|
||||
}
|
||||
|
||||
// 链式恢复到全新目标
|
||||
restoreRoot := t.TempDir()
|
||||
restoreSrc := filepath.Join(restoreRoot, "src")
|
||||
restoreTask := TaskSpec{Name: "diff", Type: "file", SourcePath: restoreSrc}
|
||||
if err := runner.Restore(context.Background(), restoreTask, full.ArtifactPath, NopLogWriter{}); err != nil {
|
||||
t.Fatalf("restore full: %v", err)
|
||||
}
|
||||
if err := runner.Restore(context.Background(), restoreTask, diff.ArtifactPath, NopLogWriter{}); err != nil {
|
||||
t.Fatalf("restore differential: %v", err)
|
||||
}
|
||||
|
||||
diffAssertContent(t, filepath.Join(restoreSrc, "a.txt"), "ALPHA-modified-and-longer")
|
||||
diffAssertContent(t, filepath.Join(restoreSrc, "sub", "c.txt"), "charlie")
|
||||
diffAssertContent(t, filepath.Join(restoreSrc, "d.txt"), "delta")
|
||||
diffAssertAbsent(t, filepath.Join(restoreSrc, "b.txt"))
|
||||
}
|
||||
|
||||
func TestPathSelected(t *testing.T) {
|
||||
sel := []string{"src/a.txt", "src/sub"}
|
||||
cases := map[string]bool{
|
||||
"src/a.txt": true,
|
||||
"src/sub": true,
|
||||
"src/sub/c.txt": true, // 选中目录下的子项
|
||||
"src/b.txt": false, // 未选中文件
|
||||
"src/subother": false, // 前缀相近但非子项,不应误判
|
||||
}
|
||||
for name, want := range cases {
|
||||
if got := pathSelected(name, sel); got != want {
|
||||
t.Errorf("pathSelected(%q) = %v, want %v", name, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFileRunnerSelectiveRestore 验证按需恢复:仅选中的文件与目录被还原,未选中的文件不出现。
|
||||
func TestFileRunnerSelectiveRestore(t *testing.T) {
|
||||
work := t.TempDir()
|
||||
src := filepath.Join(work, "src")
|
||||
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
|
||||
diffWrite(t, filepath.Join(src, "b.txt"), "bravo")
|
||||
diffWrite(t, filepath.Join(src, "sub", "c.txt"), "charlie")
|
||||
|
||||
runner := NewFileRunner()
|
||||
full, err := runner.Run(context.Background(), TaskSpec{Name: "sel", Type: "file", SourcePath: src, TempDir: t.TempDir()}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("full Run: %v", err)
|
||||
}
|
||||
|
||||
restoreRoot := t.TempDir()
|
||||
restoreSrc := filepath.Join(restoreRoot, "src")
|
||||
task := TaskSpec{Name: "sel", Type: "file", SourcePath: restoreSrc, SelectedPaths: []string{"src/a.txt", "src/sub"}}
|
||||
if err := runner.Restore(context.Background(), task, full.ArtifactPath, NopLogWriter{}); err != nil {
|
||||
t.Fatalf("selective Restore: %v", err)
|
||||
}
|
||||
diffAssertContent(t, filepath.Join(restoreSrc, "a.txt"), "alpha")
|
||||
diffAssertContent(t, filepath.Join(restoreSrc, "sub", "c.txt"), "charlie") // 选中目录 → 子项一并恢复
|
||||
diffAssertAbsent(t, filepath.Join(restoreSrc, "b.txt")) // 未选中 → 不恢复
|
||||
}
|
||||
|
||||
// TestFileRunnerDifferentialWithoutBaseIsFull 验证无基线时差异请求回退为全量(产出清单、含全部文件)。
|
||||
func TestFileRunnerDifferentialWithoutBaseIsFull(t *testing.T) {
|
||||
src := filepath.Join(t.TempDir(), "src")
|
||||
diffWrite(t, filepath.Join(src, "a.txt"), "alpha")
|
||||
|
||||
runner := NewFileRunner()
|
||||
res, err := runner.Run(context.Background(), TaskSpec{Name: "diff", Type: "file", SourcePath: src, TempDir: t.TempDir(), Differential: true}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if res.Manifest == nil {
|
||||
t.Fatalf("differential without base must fall back to full and produce a manifest")
|
||||
}
|
||||
if names := diffArchiveNames(t, res.ArtifactPath); !names["src/a.txt"] || names[deletionsEntryName] {
|
||||
t.Fatalf("fallback-full archive unexpected: %v", names)
|
||||
}
|
||||
}
|
||||
@@ -1,92 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// deletionsEntryName 是差异备份归档中记录「自全量以来被删除路径」的特殊条目名。
|
||||
// 恢复时该条目不落地为文件,而是用于在基线之上删除对应路径。
|
||||
const deletionsEntryName = ".backupx/deletions.json"
|
||||
|
||||
// ManifestEntry 记录一次全量备份中单个归档条目(文件或目录)的指纹,
|
||||
// 供差异备份比对「自全量以来的变化」。Path 为归档内相对名(slash 分隔,
|
||||
// 与 tar header.Name 一致)。字段使用短键以压缩清单体积。
|
||||
type ManifestEntry struct {
|
||||
Path string `json:"p"`
|
||||
Size int64 `json:"s"`
|
||||
ModTimeNs int64 `json:"m"`
|
||||
Mode uint32 `json:"o"`
|
||||
IsDir bool `json:"d,omitempty"`
|
||||
}
|
||||
|
||||
// Manifest 是一次全量备份的完整条目清单(文件与目录)。
|
||||
type Manifest struct {
|
||||
Entries []ManifestEntry `json:"entries"`
|
||||
}
|
||||
|
||||
// EncodeManifest 将清单序列化为紧凑 JSON。
|
||||
func EncodeManifest(m Manifest) ([]byte, error) {
|
||||
return json.Marshal(m)
|
||||
}
|
||||
|
||||
// DecodeManifest 反序列化清单;空输入返回空清单(视为「无基线」)。
|
||||
func DecodeManifest(data []byte) (Manifest, error) {
|
||||
m := Manifest{}
|
||||
if len(data) == 0 {
|
||||
return m, nil
|
||||
}
|
||||
if err := json.Unmarshal(data, &m); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// index 构建 path -> entry 映射,便于差异比对 O(1) 查找。
|
||||
func (m Manifest) index() map[string]ManifestEntry {
|
||||
idx := make(map[string]ManifestEntry, len(m.Entries))
|
||||
for _, e := range m.Entries {
|
||||
idx[e.Path] = e
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// entryFromInfo 由归档名与文件信息构造指纹条目。
|
||||
func entryFromInfo(archiveName string, info os.FileInfo) ManifestEntry {
|
||||
return ManifestEntry{
|
||||
Path: filepath.ToSlash(archiveName),
|
||||
Size: info.Size(),
|
||||
ModTimeNs: info.ModTime().UnixNano(),
|
||||
Mode: uint32(info.Mode().Perm()),
|
||||
IsDir: info.IsDir(),
|
||||
}
|
||||
}
|
||||
|
||||
// changedSince 判断当前条目相对基线是否为「新增或变更」(即应纳入差异归档)。
|
||||
// - 不在基线中 → 新增,纳入;
|
||||
// - 已存在的目录 → 不携带数据,跳过(其下变更文件会各自判定);
|
||||
// - 文件大小或 mtime 变化 → 变更,纳入(rsync 风格启发式)。
|
||||
func changedSince(base map[string]ManifestEntry, cur ManifestEntry) bool {
|
||||
prev, ok := base[cur.Path]
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if cur.IsDir {
|
||||
return false
|
||||
}
|
||||
return prev.Size != cur.Size || prev.ModTimeNs != cur.ModTimeNs
|
||||
}
|
||||
|
||||
// deletedPaths 返回基线中存在、但本次遍历未出现的路径(被删除的条目),按路径升序。
|
||||
func deletedPaths(base map[string]ManifestEntry, seen map[string]struct{}) []string {
|
||||
deleted := make([]string, 0)
|
||||
for p := range base {
|
||||
if _, ok := seen[p]; !ok {
|
||||
deleted = append(deleted, p)
|
||||
}
|
||||
}
|
||||
sort.Strings(deleted)
|
||||
return deleted
|
||||
}
|
||||
@@ -1,79 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEncodeDecodeManifestRoundTrip(t *testing.T) {
|
||||
m := Manifest{Entries: []ManifestEntry{
|
||||
{Path: "src/a.txt", Size: 10, ModTimeNs: 100, Mode: 0o644},
|
||||
{Path: "src", Size: 0, ModTimeNs: 50, Mode: 0o755, IsDir: true},
|
||||
}}
|
||||
data, err := EncodeManifest(m)
|
||||
if err != nil {
|
||||
t.Fatalf("EncodeManifest: %v", err)
|
||||
}
|
||||
got, err := DecodeManifest(data)
|
||||
if err != nil {
|
||||
t.Fatalf("DecodeManifest: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, m) {
|
||||
t.Fatalf("roundtrip mismatch:\n got %#v\nwant %#v", got, m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecodeManifestEmpty(t *testing.T) {
|
||||
got, err := DecodeManifest(nil)
|
||||
if err != nil {
|
||||
t.Fatalf("DecodeManifest(nil): %v", err)
|
||||
}
|
||||
if len(got.Entries) != 0 {
|
||||
t.Fatalf("expected empty manifest, got %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChangedSince(t *testing.T) {
|
||||
base := Manifest{Entries: []ManifestEntry{
|
||||
{Path: "a.txt", Size: 10, ModTimeNs: 100},
|
||||
{Path: "dir", IsDir: true, ModTimeNs: 100},
|
||||
}}.index()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cur ManifestEntry
|
||||
want bool
|
||||
}{
|
||||
{"unchanged file", ManifestEntry{Path: "a.txt", Size: 10, ModTimeNs: 100}, false},
|
||||
{"size changed", ManifestEntry{Path: "a.txt", Size: 11, ModTimeNs: 100}, true},
|
||||
{"mtime changed", ManifestEntry{Path: "a.txt", Size: 10, ModTimeNs: 200}, true},
|
||||
{"new file", ManifestEntry{Path: "b.txt", Size: 1, ModTimeNs: 1}, true},
|
||||
{"existing dir skipped", ManifestEntry{Path: "dir", IsDir: true, ModTimeNs: 999}, false},
|
||||
{"new dir included", ManifestEntry{Path: "newdir", IsDir: true, ModTimeNs: 1}, true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := changedSince(base, tc.cur); got != tc.want {
|
||||
t.Errorf("%s: changedSince=%v want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeletedPaths(t *testing.T) {
|
||||
base := Manifest{Entries: []ManifestEntry{
|
||||
{Path: "a"}, {Path: "b"}, {Path: "c"},
|
||||
}}.index()
|
||||
seen := map[string]struct{}{"a": {}, "c": {}}
|
||||
got := deletedPaths(base, seen)
|
||||
want := []string{"b"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("deletedPaths=%v want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeletedPathsNoneWhenAllSeen(t *testing.T) {
|
||||
base := Manifest{Entries: []ManifestEntry{{Path: "a"}, {Path: "b"}}}.index()
|
||||
seen := map[string]struct{}{"a": {}, "b": {}}
|
||||
if got := deletedPaths(base, seen); len(got) != 0 {
|
||||
t.Fatalf("expected no deletions, got %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,119 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MongoDBRunner 通过 mongodump/mongorestore 备份与恢复 MongoDB 数据库。
|
||||
// 采用 --archive 流式模式(dump 写 stdout、restore 读 stdin),与 MySQLRunner
|
||||
// 的 mysqldump/mysql 管线保持一致;产物为未压缩的 mongo archive,由备份管线统一压缩/加密。
|
||||
type MongoDBRunner struct {
|
||||
executor CommandExecutor
|
||||
}
|
||||
|
||||
func NewMongoDBRunner(executor CommandExecutor) *MongoDBRunner {
|
||||
if executor == nil {
|
||||
executor = NewOSCommandExecutor()
|
||||
}
|
||||
return &MongoDBRunner{executor: executor}
|
||||
}
|
||||
|
||||
func (r *MongoDBRunner) Type() string {
|
||||
return "mongodb"
|
||||
}
|
||||
|
||||
func (r *MongoDBRunner) Run(ctx context.Context, task TaskSpec, writer LogWriter) (*RunResult, error) {
|
||||
if _, err := r.executor.LookPath("mongodump"); err != nil {
|
||||
return nil, fmt.Errorf("未找到 mongodump 命令 (请确保服务器已安装 mongodb-database-tools)")
|
||||
}
|
||||
startedAt := task.StartedAt
|
||||
if startedAt.IsZero() {
|
||||
startedAt = time.Now().UTC()
|
||||
}
|
||||
tempDir, err := CreateTaskTempDir(task.Name, startedAt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fileName := BuildArtifactName(task.Name, startedAt, "archive")
|
||||
artifactPath := filepath.Join(tempDir, fileName)
|
||||
file, err := os.Create(artifactPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create mongodump archive file: %w", err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
args := mongoConnArgs(task.Database)
|
||||
dbNames := normalizeDatabaseNames(task.Database.Names)
|
||||
if len(dbNames) == 1 {
|
||||
args = append(args, "--db", dbNames[0])
|
||||
writer.WriteLine(fmt.Sprintf("备份数据库: %s", dbNames[0]))
|
||||
} else {
|
||||
writer.WriteLine("备份全部数据库")
|
||||
}
|
||||
args = append(args, "--archive") // 归档流式写入 stdout
|
||||
|
||||
writer.WriteLine(fmt.Sprintf("连接到 MongoDB: %s:%d", task.Database.Host, task.Database.Port))
|
||||
stderrWriter := newLogLineWriter(writer, "mongodump")
|
||||
writer.WriteLine("开始执行 mongodump")
|
||||
if err := r.executor.Run(ctx, "mongodump", args, CommandOptions{Stdout: file, Stderr: stderrWriter}); err != nil {
|
||||
return nil, fmt.Errorf("run mongodump: %w: %s", err, stderrWriter.collected())
|
||||
}
|
||||
info, err := file.Stat()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stat mongodump archive: %w", err)
|
||||
}
|
||||
if info.Size() == 0 {
|
||||
return nil, fmt.Errorf("mongodump 产物为空,请检查数据库连接与权限")
|
||||
}
|
||||
writer.WriteLine(fmt.Sprintf("MongoDB 导出完成(文件大小: %s)", formatFileSize(info.Size())))
|
||||
return &RunResult{ArtifactPath: artifactPath, FileName: fileName, TempDir: tempDir, Size: info.Size(), StorageKey: BuildStorageKey("mongodb", startedAt, fileName)}, nil
|
||||
}
|
||||
|
||||
func (r *MongoDBRunner) Restore(ctx context.Context, task TaskSpec, artifactPath string, writer LogWriter) error {
|
||||
if _, err := r.executor.LookPath("mongorestore"); err != nil {
|
||||
return fmt.Errorf("未找到 mongorestore 命令 (请确保服务器已安装 mongodb-database-tools)")
|
||||
}
|
||||
input, err := os.Open(filepath.Clean(artifactPath))
|
||||
if err != nil {
|
||||
return fmt.Errorf("open mongodb restore archive: %w", err)
|
||||
}
|
||||
defer input.Close()
|
||||
|
||||
args := mongoConnArgs(task.Database)
|
||||
// --drop:恢复前删除同名集合,保证恢复后与归档一致(与 mysql 恢复的整库覆盖语义对齐)。
|
||||
args = append(args, "--drop", "--archive")
|
||||
stderr := &bytes.Buffer{}
|
||||
writer.WriteLine("开始执行 mongorestore")
|
||||
if err := r.executor.Run(ctx, "mongorestore", args, CommandOptions{Stdin: input, Stderr: stderr}); err != nil {
|
||||
return fmt.Errorf("run mongorestore: %w: %s", err, strings.TrimSpace(stderr.String()))
|
||||
}
|
||||
writer.WriteLine("MongoDB 恢复完成")
|
||||
return nil
|
||||
}
|
||||
|
||||
// mongoConnArgs 构造 mongodump/mongorestore 的连接与认证参数。
|
||||
// 注意:mongodb-database-tools 无类似 MYSQL_PWD 的密码环境变量,密码只能经 --password 传入;
|
||||
// 认证库默认 admin(绝大多数部署的管理账号所在库)。
|
||||
func mongoConnArgs(db DatabaseSpec) []string {
|
||||
args := make([]string, 0, 8)
|
||||
if strings.TrimSpace(db.Host) != "" {
|
||||
args = append(args, "--host", db.Host)
|
||||
}
|
||||
if db.Port > 0 {
|
||||
args = append(args, "--port", strconv.Itoa(db.Port))
|
||||
}
|
||||
if strings.TrimSpace(db.User) != "" {
|
||||
args = append(args, "--username", db.User, "--authenticationDatabase", "admin")
|
||||
if strings.TrimSpace(db.Password) != "" {
|
||||
args = append(args, "--password", db.Password)
|
||||
}
|
||||
}
|
||||
return args
|
||||
}
|
||||
@@ -1,102 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func argIndex(args []string, target string) int {
|
||||
for i, a := range args {
|
||||
if a == target {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
func TestMongoDBRunnerRunUsesMongodump(t *testing.T) {
|
||||
executor := &fakeCommandExecutor{runFunc: func(name string, args []string, options CommandOptions) error {
|
||||
if options.Stdout != nil {
|
||||
_, _ = io.WriteString(options.Stdout, "mongo archive bytes")
|
||||
}
|
||||
return nil
|
||||
}}
|
||||
runner := NewMongoDBRunner(executor)
|
||||
result, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, User: "admin", Password: "secret", Names: []string{"app"}}}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("Run returned error: %v", err)
|
||||
}
|
||||
if executor.lastName != "mongodump" {
|
||||
t.Fatalf("expected mongodump, got %s", executor.lastName)
|
||||
}
|
||||
args := executor.lastArgs
|
||||
if argIndex(args, "--archive") < 0 {
|
||||
t.Fatalf("expected --archive flag, got %#v", args)
|
||||
}
|
||||
if i := argIndex(args, "--db"); i < 0 || i+1 >= len(args) || args[i+1] != "app" {
|
||||
t.Fatalf("expected --db app, got %#v", args)
|
||||
}
|
||||
if i := argIndex(args, "--username"); i < 0 || args[i+1] != "admin" {
|
||||
t.Fatalf("expected --username admin, got %#v", args)
|
||||
}
|
||||
if argIndex(args, "--authenticationDatabase") < 0 || argIndex(args, "--password") < 0 {
|
||||
t.Fatalf("expected auth args, got %#v", args)
|
||||
}
|
||||
if _, err := os.Stat(result.ArtifactPath); err != nil {
|
||||
t.Fatalf("artifact file missing: %v", err)
|
||||
}
|
||||
if result.StorageKey == "" || !strings.HasSuffix(result.FileName, ".archive") {
|
||||
t.Fatalf("unexpected result metadata: %#v", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMongoDBRunnerRunBackupsAllWhenNoDatabase(t *testing.T) {
|
||||
executor := &fakeCommandExecutor{runFunc: func(name string, args []string, options CommandOptions) error {
|
||||
_, _ = io.WriteString(options.Stdout, "all dbs")
|
||||
return nil
|
||||
}}
|
||||
runner := NewMongoDBRunner(executor)
|
||||
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017}}, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("Run returned error: %v", err)
|
||||
}
|
||||
if argIndex(executor.lastArgs, "--db") >= 0 {
|
||||
t.Fatalf("expected no --db when backing up all databases, got %#v", executor.lastArgs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMongoDBRunnerRunRejectsEmptyOutput(t *testing.T) {
|
||||
executor := &fakeCommandExecutor{} // runFunc nil → writes nothing
|
||||
runner := NewMongoDBRunner(executor)
|
||||
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, Names: []string{"app"}}}, NopLogWriter{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error for empty mongodump output")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMongoDBRunnerRestoreUsesMongorestore(t *testing.T) {
|
||||
executor := &fakeCommandExecutor{}
|
||||
runner := NewMongoDBRunner(executor)
|
||||
artifact := filepathJoinTempFile(t, "dump.archive", "mongo archive bytes")
|
||||
if err := runner.Restore(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, User: "admin", Password: "secret"}}, artifact, NopLogWriter{}); err != nil {
|
||||
t.Fatalf("Restore returned error: %v", err)
|
||||
}
|
||||
if executor.lastName != "mongorestore" {
|
||||
t.Fatalf("expected mongorestore, got %s", executor.lastName)
|
||||
}
|
||||
if argIndex(executor.lastArgs, "--drop") < 0 || argIndex(executor.lastArgs, "--archive") < 0 {
|
||||
t.Fatalf("expected --drop --archive, got %#v", executor.lastArgs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMongoDBRunnerRunReturnsLookupError(t *testing.T) {
|
||||
runner := NewMongoDBRunner(&fakeCommandExecutor{lookupErr: errors.New("missing")})
|
||||
_, err := runner.Run(context.Background(), TaskSpec{Name: "mongo", Type: "mongodb", Database: DatabaseSpec{Host: "127.0.0.1", Port: 27017, Names: []string{"app"}}}, NopLogWriter{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error when mongodump is missing")
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,131 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
)
|
||||
|
||||
const (
|
||||
repositoryChunkMin = 512 << 10
|
||||
repositoryChunkAvg = 1 << 20
|
||||
repositoryChunkMax = 4 << 20
|
||||
)
|
||||
|
||||
// contentDefinedChunker implements the normalized FastCDC cut-point strategy.
|
||||
// The rolling Gear hash only retains the latest 64 bytes through uint64
|
||||
// overflow, so chunk boundaries re-synchronize after insertions or deletions.
|
||||
type contentDefinedChunker struct {
|
||||
minSize int
|
||||
avgSize int
|
||||
maxSize int
|
||||
smallMask uint64
|
||||
largeMask uint64
|
||||
gear [256]uint64
|
||||
}
|
||||
|
||||
func newContentDefinedChunker() *contentDefinedChunker {
|
||||
chunker := &contentDefinedChunker{
|
||||
minSize: repositoryChunkMin,
|
||||
avgSize: repositoryChunkAvg,
|
||||
maxSize: repositoryChunkMax,
|
||||
smallMask: (1 << 21) - 1,
|
||||
largeMask: (1 << 19) - 1,
|
||||
}
|
||||
|
||||
// SplitMix64 produces a stable, well-distributed Gear table. The seed and
|
||||
// generation algorithm are part of repository format v1 and must not change.
|
||||
seed := uint64(0x6a09e667f3bcc909)
|
||||
for i := range chunker.gear {
|
||||
seed += 0x9e3779b97f4a7c15
|
||||
value := seed
|
||||
value = (value ^ (value >> 30)) * 0xbf58476d1ce4e5b9
|
||||
value = (value ^ (value >> 27)) * 0x94d049bb133111eb
|
||||
chunker.gear[i] = value ^ (value >> 31)
|
||||
}
|
||||
return chunker
|
||||
}
|
||||
|
||||
func (c *contentDefinedChunker) Split(ctx context.Context, reader io.Reader, emit func([]byte) error) error {
|
||||
if reader == nil || emit == nil {
|
||||
return fmt.Errorf("chunk reader and emitter are required")
|
||||
}
|
||||
|
||||
pending := make([]byte, 0, c.maxSize+(256<<10))
|
||||
readBuffer := make([]byte, 256<<10)
|
||||
eof := false
|
||||
for {
|
||||
if !eof {
|
||||
readCount, readErr := reader.Read(readBuffer)
|
||||
if readCount > 0 {
|
||||
pending = append(pending, readBuffer[:readCount]...)
|
||||
}
|
||||
switch readErr {
|
||||
case nil:
|
||||
case io.EOF:
|
||||
eof = true
|
||||
default:
|
||||
return fmt.Errorf("read source for chunking: %w", readErr)
|
||||
}
|
||||
if readCount == 0 && readErr == nil {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
for len(pending) > 0 {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
cut := c.findCutPoint(pending, eof)
|
||||
if cut == 0 {
|
||||
break
|
||||
}
|
||||
chunk := make([]byte, cut)
|
||||
copy(chunk, pending[:cut])
|
||||
if err := emit(chunk); err != nil {
|
||||
return err
|
||||
}
|
||||
copy(pending, pending[cut:])
|
||||
pending = pending[:len(pending)-cut]
|
||||
}
|
||||
|
||||
if eof {
|
||||
if len(pending) != 0 {
|
||||
return fmt.Errorf("chunker stopped with %d buffered bytes", len(pending))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *contentDefinedChunker) findCutPoint(data []byte, eof bool) int {
|
||||
if len(data) < c.minSize {
|
||||
if eof {
|
||||
return len(data)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
limit := len(data)
|
||||
if limit > c.maxSize {
|
||||
limit = c.maxSize
|
||||
}
|
||||
var hash uint64
|
||||
for index := c.minSize; index < limit; index++ {
|
||||
hash = (hash << 1) + c.gear[data[index]]
|
||||
mask := c.largeMask
|
||||
if index < c.avgSize {
|
||||
mask = c.smallMask
|
||||
}
|
||||
if hash&mask == 0 {
|
||||
return index + 1
|
||||
}
|
||||
}
|
||||
if len(data) >= c.maxSize {
|
||||
return c.maxSize
|
||||
}
|
||||
if eof {
|
||||
return len(data)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -1,414 +0,0 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/rand"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/storage"
|
||||
)
|
||||
|
||||
func TestContentDefinedChunkerResynchronizesAfterInsertion(t *testing.T) {
|
||||
source := make([]byte, 8<<20)
|
||||
if _, err := rand.New(rand.NewSource(42)).Read(source); err != nil {
|
||||
t.Fatalf("generate source: %v", err)
|
||||
}
|
||||
modified := make([]byte, 0, len(source)+4096)
|
||||
modified = append(modified, source[:2<<20]...)
|
||||
modified = append(modified, bytes.Repeat([]byte("inserted"), 512)...)
|
||||
modified = append(modified, source[2<<20:]...)
|
||||
|
||||
chunker := newContentDefinedChunker()
|
||||
collect := func(data []byte) map[string]struct{} {
|
||||
t.Helper()
|
||||
ids := make(map[string]struct{})
|
||||
err := chunker.Split(context.Background(), bytes.NewReader(data), func(chunk []byte) error {
|
||||
digest := sha256.Sum256(chunk)
|
||||
ids[fmt.Sprintf("%x", digest[:])] = struct{}{}
|
||||
if len(chunk) > repositoryChunkMax {
|
||||
return fmt.Errorf("chunk exceeds maximum: %d", len(chunk))
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("split chunks: %v", err)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
originalChunks := collect(source)
|
||||
modifiedChunks := collect(modified)
|
||||
shared := 0
|
||||
for chunkID := range originalChunks {
|
||||
if _, ok := modifiedChunks[chunkID]; ok {
|
||||
shared++
|
||||
}
|
||||
}
|
||||
if shared < len(originalChunks)/2 {
|
||||
t.Fatalf("content-defined boundaries did not resynchronize: shared=%d original=%d", shared, len(originalChunks))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepositoryRoundTripDedupAndPrune(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
tempDir := t.TempDir()
|
||||
sourceDir := filepath.Join(tempDir, "dataset")
|
||||
if err := os.MkdirAll(filepath.Join(sourceDir, "empty"), 0o755); err != nil {
|
||||
t.Fatalf("create source: %v", err)
|
||||
}
|
||||
original := make([]byte, 6<<20)
|
||||
if _, err := rand.New(rand.NewSource(7)).Read(original); err != nil {
|
||||
t.Fatalf("generate fixture: %v", err)
|
||||
}
|
||||
primaryPath := filepath.Join(sourceDir, "primary.bin")
|
||||
duplicatePath := filepath.Join(sourceDir, "duplicate.bin")
|
||||
if err := os.WriteFile(primaryPath, original, 0o640); err != nil {
|
||||
t.Fatalf("write primary: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(duplicatePath, original, 0o640); err != nil {
|
||||
t.Fatalf("write duplicate: %v", err)
|
||||
}
|
||||
|
||||
key := sha256.Sum256([]byte("repository-test-key"))
|
||||
store := NewRepositoryStore(key[:])
|
||||
provider := newMemoryRepositoryProvider()
|
||||
task := TaskSpec{
|
||||
ID: 12,
|
||||
Name: "repository-test",
|
||||
Type: "file",
|
||||
SourcePaths: []string{sourceDir},
|
||||
Compression: "zstd",
|
||||
Encrypt: true,
|
||||
StartedAt: time.Date(2026, 8, 6, 1, 2, 3, 0, time.UTC),
|
||||
TempDir: tempDir,
|
||||
}
|
||||
|
||||
firstPlan, err := store.BuildPlan(ctx, task, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("build first plan: %v", err)
|
||||
}
|
||||
firstKey := store.SnapshotKey(task.ID, 1, task.StartedAt)
|
||||
firstResult, err := store.Upload(ctx, provider, firstPlan, firstKey)
|
||||
if closeErr := firstPlan.Close(); closeErr != nil {
|
||||
t.Fatalf("close first plan: %v", closeErr)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("upload first snapshot: %v", err)
|
||||
}
|
||||
if firstResult.NewChunks == 0 || firstResult.UniqueChunks == 0 {
|
||||
t.Fatalf("first upload did not create chunks: %+v", firstResult)
|
||||
}
|
||||
if firstPlan.UniqueSize >= firstPlan.LogicalSize {
|
||||
t.Fatalf("duplicate file was not deduplicated within plan: unique=%d logical=%d", firstPlan.UniqueSize, firstPlan.LogicalSize)
|
||||
}
|
||||
|
||||
modified := make([]byte, 0, len(original)+4096)
|
||||
modified = append(modified, original[:2<<20]...)
|
||||
modified = append(modified, bytes.Repeat([]byte("changed!"), 512)...)
|
||||
modified = append(modified, original[2<<20:]...)
|
||||
if err := os.WriteFile(primaryPath, modified, 0o640); err != nil {
|
||||
t.Fatalf("modify primary: %v", err)
|
||||
}
|
||||
task.StartedAt = task.StartedAt.Add(time.Hour)
|
||||
secondPlan, err := store.BuildPlan(ctx, task, NopLogWriter{})
|
||||
if err != nil {
|
||||
t.Fatalf("build second plan: %v", err)
|
||||
}
|
||||
secondKey := store.SnapshotKey(task.ID, 2, task.StartedAt)
|
||||
secondResult, err := store.Upload(ctx, provider, secondPlan, secondKey)
|
||||
if closeErr := secondPlan.Close(); closeErr != nil {
|
||||
t.Fatalf("close second plan: %v", closeErr)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("upload second snapshot: %v", err)
|
||||
}
|
||||
if secondResult.ReusedBytes <= secondResult.LogicalSize/3 {
|
||||
t.Fatalf("second snapshot reused too little data: %+v", secondResult)
|
||||
}
|
||||
if secondResult.UploadedBytes >= secondResult.LogicalSize {
|
||||
t.Fatalf("incremental upload was not smaller than logical data: %+v", secondResult)
|
||||
}
|
||||
|
||||
verify, err := store.Verify(ctx, provider, secondKey, secondResult.Checksum)
|
||||
if err != nil {
|
||||
t.Fatalf("verify repository: %v", err)
|
||||
}
|
||||
if verify.Chunks == 0 || verify.Bytes == 0 {
|
||||
t.Fatalf("empty verification result: %+v", verify)
|
||||
}
|
||||
|
||||
restoreRoot := filepath.Join(tempDir, "restore")
|
||||
restoreTask := task
|
||||
restoreTask.RestoreTargetPath = restoreRoot
|
||||
if err := store.Restore(ctx, provider, secondKey, strings.Repeat("0", sha256.Size*2), restoreTask, NopLogWriter{}); err == nil {
|
||||
t.Fatal("restore accepted a mismatched snapshot checksum")
|
||||
}
|
||||
if err := store.Restore(ctx, provider, secondKey, "", restoreTask, NopLogWriter{}); err == nil {
|
||||
t.Fatal("restore accepted a missing snapshot checksum")
|
||||
}
|
||||
if err := store.Restore(ctx, provider, secondKey, secondResult.Checksum, restoreTask, NopLogWriter{}); err != nil {
|
||||
t.Fatalf("restore snapshot: %v", err)
|
||||
}
|
||||
restored, err := os.ReadFile(filepath.Join(restoreRoot, filepath.Base(sourceDir), "primary.bin"))
|
||||
if err != nil {
|
||||
t.Fatalf("read restored primary: %v", err)
|
||||
}
|
||||
if !bytes.Equal(restored, modified) {
|
||||
t.Fatalf("restored primary differs from source")
|
||||
}
|
||||
if info, err := os.Stat(filepath.Join(restoreRoot, filepath.Base(sourceDir), "empty")); err != nil || !info.IsDir() {
|
||||
t.Fatalf("empty directory was not restored: info=%v err=%v", info, err)
|
||||
}
|
||||
|
||||
if err := provider.Delete(ctx, firstKey); err != nil {
|
||||
t.Fatalf("delete first snapshot: %v", err)
|
||||
}
|
||||
if _, err := store.Prune(ctx, provider); err != nil {
|
||||
t.Fatalf("prune with live snapshot: %v", err)
|
||||
}
|
||||
if err := provider.Delete(ctx, secondKey); err != nil {
|
||||
t.Fatalf("delete second snapshot: %v", err)
|
||||
}
|
||||
pruned, err := store.Prune(ctx, provider)
|
||||
if err != nil {
|
||||
t.Fatalf("prune empty repository: %v", err)
|
||||
}
|
||||
if pruned.DeletedPacks == 0 || pruned.DeletedIndexes == 0 {
|
||||
t.Fatalf("prune did not reclaim repository data: %+v", pruned)
|
||||
}
|
||||
if objects, err := provider.List(ctx, repositoryPackPrefix); err != nil || len(objects) != 0 {
|
||||
t.Fatalf("packs remain after prune: objects=%v err=%v", objects, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepositoryRestoreRejectsUnsafeSnapshotMetadata(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
entries []repositoryEntry
|
||||
}{
|
||||
{
|
||||
name: "path traversal",
|
||||
entries: []repositoryEntry{{Path: "../escape", Kind: "directory", Mode: 0o755}},
|
||||
},
|
||||
{
|
||||
name: "entry below symlink",
|
||||
entries: []repositoryEntry{
|
||||
{Path: "link", Kind: "symlink", Mode: 0o777, LinkTarget: "inside"},
|
||||
{Path: "link/payload", Kind: "file", Mode: 0o600, Size: 1, Chunks: []string{"p-" + strings.Repeat("0", sha256.Size*2)}},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "escaping symlink target",
|
||||
entries: []repositoryEntry{{Path: "escape", Kind: "symlink", Mode: 0o777, LinkTarget: "../outside"}},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := NewRepositoryStore(nil)
|
||||
provider := newMemoryRepositoryProvider()
|
||||
snapshot := repositorySnapshot{
|
||||
Version: repositoryFormatVersion,
|
||||
TaskID: 1,
|
||||
CreatedAt: time.Now().UTC(),
|
||||
Compression: "none",
|
||||
Entries: tc.entries,
|
||||
}
|
||||
data, err := store.encodeSnapshot(snapshot)
|
||||
if err != nil {
|
||||
t.Fatalf("encodeSnapshot returned error: %v", err)
|
||||
}
|
||||
key := store.SnapshotKey(1, 1, snapshot.CreatedAt)
|
||||
if err := provider.Upload(ctx, key, bytes.NewReader(data), int64(len(data)), nil); err != nil {
|
||||
t.Fatalf("Upload snapshot returned error: %v", err)
|
||||
}
|
||||
digest := sha256.Sum256(data)
|
||||
task := TaskSpec{SourcePath: filepath.Join(t.TempDir(), "source"), RestoreTargetPath: filepath.Join(t.TempDir(), "restore")}
|
||||
if err := store.Restore(ctx, provider, key, fmt.Sprintf("%x", digest[:]), task, NopLogWriter{}); err == nil {
|
||||
t.Fatal("restore accepted unsafe snapshot metadata")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepositoryRestorePreservesDirectoryWhenSnapshotContainsSymlink(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := NewRepositoryStore(nil)
|
||||
provider := newMemoryRepositoryProvider()
|
||||
snapshot := repositorySnapshot{
|
||||
Version: repositoryFormatVersion,
|
||||
TaskID: 1,
|
||||
CreatedAt: time.Now().UTC(),
|
||||
Compression: "none",
|
||||
Entries: []repositoryEntry{{Path: "link", Kind: "symlink", Mode: 0o777, LinkTarget: "inside"}},
|
||||
}
|
||||
data, err := store.encodeSnapshot(snapshot)
|
||||
if err != nil {
|
||||
t.Fatalf("encodeSnapshot returned error: %v", err)
|
||||
}
|
||||
key := store.SnapshotKey(1, 1, snapshot.CreatedAt)
|
||||
if err := provider.Upload(ctx, key, bytes.NewReader(data), int64(len(data)), nil); err != nil {
|
||||
t.Fatalf("Upload snapshot returned error: %v", err)
|
||||
}
|
||||
digest := sha256.Sum256(data)
|
||||
restoreRoot := filepath.Join(t.TempDir(), "restore")
|
||||
markerPath := filepath.Join(restoreRoot, "link", "keep.txt")
|
||||
if err := os.MkdirAll(filepath.Dir(markerPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll marker parent: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(markerPath, []byte("keep"), 0o600); err != nil {
|
||||
t.Fatalf("WriteFile marker: %v", err)
|
||||
}
|
||||
task := TaskSpec{SourcePath: filepath.Join(t.TempDir(), "source"), RestoreTargetPath: restoreRoot}
|
||||
if err := store.Restore(ctx, provider, key, fmt.Sprintf("%x", digest[:]), task, NopLogWriter{}); err == nil {
|
||||
t.Fatal("restore replaced an existing directory with a symlink")
|
||||
}
|
||||
if data, err := os.ReadFile(markerPath); err != nil || string(data) != "keep" {
|
||||
t.Fatalf("existing directory content changed: data=%q err=%v", data, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepositoryRejectsOversizedChunkLocation(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := NewRepositoryStore(nil)
|
||||
provider := newMemoryRepositoryProvider()
|
||||
chunkID := "p-" + strings.Repeat("0", sha256.Size*2)
|
||||
packID := strings.Repeat("a", sha256.Size*2)
|
||||
segment := repositoryIndexSegment{
|
||||
Version: repositoryFormatVersion,
|
||||
Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, packID[:2], packID),
|
||||
Chunks: map[string]repositoryChunkLocation{
|
||||
chunkID: {Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, packID[:2], packID), Offset: 0, Length: repositoryMaxEncoded + 1, PlainSize: 1, Compression: "none"},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(segment)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal index returned error: %v", err)
|
||||
}
|
||||
indexKey := fmt.Sprintf("%s/%s.json", repositoryIndexPrefix, packID)
|
||||
if err := provider.Upload(ctx, indexKey, bytes.NewReader(data), int64(len(data)), nil); err != nil {
|
||||
t.Fatalf("Upload index returned error: %v", err)
|
||||
}
|
||||
if _, err := store.loadIndex(ctx, provider); err == nil {
|
||||
t.Fatal("loadIndex accepted an oversized encoded chunk")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepositoryRejectsIndexPackMismatch(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := NewRepositoryStore(nil)
|
||||
provider := newMemoryRepositoryProvider()
|
||||
chunkID := "p-" + strings.Repeat("0", sha256.Size*2)
|
||||
indexID := strings.Repeat("a", sha256.Size*2)
|
||||
otherPackID := strings.Repeat("b", sha256.Size*2)
|
||||
expectedPack := fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, indexID[:2], indexID)
|
||||
segment := repositoryIndexSegment{
|
||||
Version: repositoryFormatVersion,
|
||||
Pack: expectedPack,
|
||||
Chunks: map[string]repositoryChunkLocation{
|
||||
chunkID: {
|
||||
Pack: fmt.Sprintf("%s/%s/%s.pack", repositoryPackPrefix, otherPackID[:2], otherPackID),
|
||||
Offset: 0,
|
||||
Length: 1,
|
||||
PlainSize: 1,
|
||||
Compression: "none",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(segment)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal index returned error: %v", err)
|
||||
}
|
||||
indexKey := fmt.Sprintf("%s/%s.json", repositoryIndexPrefix, indexID)
|
||||
if err := provider.Upload(ctx, indexKey, bytes.NewReader(data), int64(len(data)), nil); err != nil {
|
||||
t.Fatalf("Upload index returned error: %v", err)
|
||||
}
|
||||
if _, err := store.loadIndex(ctx, provider); err == nil {
|
||||
t.Fatal("loadIndex accepted a chunk location pointing to a different pack")
|
||||
}
|
||||
}
|
||||
|
||||
type memoryRepositoryProvider struct {
|
||||
mu sync.RWMutex
|
||||
objects map[string][]byte
|
||||
times map[string]time.Time
|
||||
}
|
||||
|
||||
func newMemoryRepositoryProvider() *memoryRepositoryProvider {
|
||||
return &memoryRepositoryProvider{objects: make(map[string][]byte), times: make(map[string]time.Time)}
|
||||
}
|
||||
|
||||
func (p *memoryRepositoryProvider) Type() storage.ProviderType { return "memory" }
|
||||
func (p *memoryRepositoryProvider) TestConnection(context.Context) error { return nil }
|
||||
|
||||
func (p *memoryRepositoryProvider) Upload(_ context.Context, key string, reader io.Reader, size int64, _ map[string]string) error {
|
||||
data, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if int64(len(data)) != size {
|
||||
return fmt.Errorf("size mismatch for %s: %d != %d", key, len(data), size)
|
||||
}
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.objects[key] = append([]byte(nil), data...)
|
||||
p.times[key] = time.Now().UTC()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *memoryRepositoryProvider) Download(_ context.Context, key string) (io.ReadCloser, error) {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
data, ok := p.objects[key]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("object %s not found", key)
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(append([]byte(nil), data...))), nil
|
||||
}
|
||||
|
||||
func (p *memoryRepositoryProvider) DownloadRange(_ context.Context, key string, offset, length int64) (io.ReadCloser, error) {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
data, ok := p.objects[key]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("object %s not found", key)
|
||||
}
|
||||
if offset < 0 || length <= 0 || offset+length > int64(len(data)) {
|
||||
return nil, fmt.Errorf("invalid range %d:%d for %s", offset, length, key)
|
||||
}
|
||||
return io.NopCloser(bytes.NewReader(append([]byte(nil), data[offset:offset+length]...))), nil
|
||||
}
|
||||
|
||||
func (p *memoryRepositoryProvider) Delete(_ context.Context, key string) error {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if _, ok := p.objects[key]; !ok {
|
||||
return fmt.Errorf("object %s not found", key)
|
||||
}
|
||||
delete(p.objects, key)
|
||||
delete(p.times, key)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *memoryRepositoryProvider) List(_ context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
p.mu.RLock()
|
||||
defer p.mu.RUnlock()
|
||||
result := make([]storage.ObjectInfo, 0)
|
||||
for key, data := range p.objects {
|
||||
if strings.HasPrefix(key, prefix) {
|
||||
result = append(result, storage.ObjectInfo{Key: key, Size: int64(len(data)), UpdatedAt: p.times[key]})
|
||||
}
|
||||
}
|
||||
sort.Slice(result, func(i, j int) bool { return result[i].Key < result[j].Key })
|
||||
return result, nil
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
package retention
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"backupx/server/internal/model"
|
||||
)
|
||||
|
||||
func retentionRecIDs(records []model.BackupRecord) []uint {
|
||||
ids := make([]uint, 0, len(records))
|
||||
for _, r := range records {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// 基线全量仍被「不在删除集合中的差异」依赖 → 必须保留,否则差异无法恢复。
|
||||
func TestProtectDifferentialBasesKeepsBaseWithSurvivingDiff(t *testing.T) {
|
||||
all := []model.BackupRecord{
|
||||
{ID: 1, BackupKind: model.BackupKindFull},
|
||||
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
|
||||
}
|
||||
candidates := []model.BackupRecord{{ID: 1, BackupKind: model.BackupKindFull}}
|
||||
if got := protectDifferentialBases(all, candidates); len(got) != 0 {
|
||||
t.Fatalf("base with surviving diff must be protected, got %v", retentionRecIDs(got))
|
||||
}
|
||||
}
|
||||
|
||||
// 基线全量与其全部差异都在删除集合中 → 可一并删除(无残留差异失去基线)。
|
||||
func TestProtectDifferentialBasesDeletesBaseWhenDiffAlsoDeleted(t *testing.T) {
|
||||
all := []model.BackupRecord{
|
||||
{ID: 1, BackupKind: model.BackupKindFull},
|
||||
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
|
||||
}
|
||||
candidates := []model.BackupRecord{
|
||||
{ID: 1, BackupKind: model.BackupKindFull},
|
||||
{ID: 2, BackupKind: model.BackupKindDifferential, BaseRecordID: 1},
|
||||
}
|
||||
if got := protectDifferentialBases(all, candidates); len(got) != 2 {
|
||||
t.Fatalf("base+diff both expired should both be deleted, got %v", retentionRecIDs(got))
|
||||
}
|
||||
}
|
||||
|
||||
// 无差异备份时原样透传(不影响既有全量保留逻辑)。
|
||||
func TestProtectDifferentialBasesNoDiffsPassThrough(t *testing.T) {
|
||||
all := []model.BackupRecord{
|
||||
{ID: 1, BackupKind: model.BackupKindFull},
|
||||
{ID: 2, BackupKind: model.BackupKindFull},
|
||||
}
|
||||
candidates := []model.BackupRecord{{ID: 1, BackupKind: model.BackupKindFull}}
|
||||
got := protectDifferentialBases(all, candidates)
|
||||
if len(got) != 1 || got[0].ID != 1 {
|
||||
t.Fatalf("no diffs should pass through unchanged, got %v", retentionRecIDs(got))
|
||||
}
|
||||
}
|
||||
@@ -2,13 +2,10 @@ package retention
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/backup"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
"backupx/server/internal/storage"
|
||||
@@ -42,48 +39,16 @@ type CleanupResult struct {
|
||||
Warnings []string
|
||||
}
|
||||
|
||||
type cleanupObject struct {
|
||||
targetID uint
|
||||
path string
|
||||
}
|
||||
|
||||
type storedUploadResult struct {
|
||||
StorageTargetID uint `json:"storageTargetId"`
|
||||
Status string `json:"status"`
|
||||
StoragePath string `json:"storagePath"`
|
||||
}
|
||||
|
||||
type Service struct {
|
||||
records repository.BackupRecordRepository
|
||||
repositoryKey []byte
|
||||
now func() time.Time
|
||||
records repository.BackupRecordRepository
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
func NewService(records repository.BackupRecordRepository, repositoryKey ...[]byte) *Service {
|
||||
var key []byte
|
||||
if len(repositoryKey) > 0 {
|
||||
key = append([]byte(nil), repositoryKey[0]...)
|
||||
}
|
||||
return &Service{records: records, repositoryKey: key, now: func() time.Time { return time.Now().UTC() }}
|
||||
func NewService(records repository.BackupRecordRepository) *Service {
|
||||
return &Service{records: records, now: func() time.Time { return time.Now().UTC() }}
|
||||
}
|
||||
|
||||
func (s *Service) Cleanup(ctx context.Context, task *model.BackupTask, provider storage.StorageProvider) (*CleanupResult, error) {
|
||||
return s.cleanup(ctx, task, func(uint) (storage.StorageProvider, bool) {
|
||||
return provider, provider != nil
|
||||
})
|
||||
}
|
||||
|
||||
// CleanupProviders applies one retention decision to every successful copy of
|
||||
// a record before deleting its database row. This prevents multi-target tasks
|
||||
// from leaving stale objects after the first target removes the shared record.
|
||||
func (s *Service) CleanupProviders(ctx context.Context, task *model.BackupTask, providers map[uint]storage.StorageProvider) (*CleanupResult, error) {
|
||||
return s.cleanup(ctx, task, func(targetID uint) (storage.StorageProvider, bool) {
|
||||
provider, ok := providers[targetID]
|
||||
return provider, ok && provider != nil
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolveProvider func(uint) (storage.StorageProvider, bool)) (*CleanupResult, error) {
|
||||
if task == nil {
|
||||
return nil, fmt.Errorf("backup task is required")
|
||||
}
|
||||
@@ -91,45 +56,19 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list successful records: %w", err)
|
||||
}
|
||||
var candidates []model.BackupRecord
|
||||
if gfsEnabled(task) {
|
||||
// GFS 策略:按天/周/月/年分层保留代表性备份,取代简单的天数/数量策略。
|
||||
candidates = selectGFSToDelete(records, task.KeepDaily, task.KeepWeekly, task.KeepMonthly, task.KeepYearly)
|
||||
} else {
|
||||
candidates = selectRecordsToDelete(records, task.RetentionDays, task.MaxBackups, s.now())
|
||||
}
|
||||
// 差异链保护:保留仍被存活差异依赖的全量,避免删除基线后差异无法恢复。
|
||||
candidates = protectDifferentialBases(records, candidates)
|
||||
candidates := selectRecordsToDelete(records, task.RetentionDays, task.MaxBackups, s.now())
|
||||
result := &CleanupResult{}
|
||||
repositoryProviders := make(map[uint]storage.StorageProvider)
|
||||
touchedProviders := make(map[uint]storage.StorageProvider)
|
||||
for _, record := range candidates {
|
||||
objects, objectErr := cleanupObjectsForRecord(record)
|
||||
if objectErr != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("decode storage copies for record %d failed: %v", record.ID, objectErr))
|
||||
continue
|
||||
}
|
||||
allObjectsDeleted := true
|
||||
for _, object := range objects {
|
||||
provider, ok := resolveProvider(object.targetID)
|
||||
if !ok {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("record %d missing storage provider %d for cleanup", record.ID, object.targetID))
|
||||
allObjectsDeleted = false
|
||||
if strings.TrimSpace(record.StoragePath) != "" {
|
||||
if provider == nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("record %d missing storage provider for cleanup", record.ID))
|
||||
continue
|
||||
}
|
||||
if err := provider.Delete(ctx, object.path); err != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("delete storage object %s from target %d failed: %v", object.path, object.targetID, err))
|
||||
allObjectsDeleted = false
|
||||
if err := provider.Delete(ctx, record.StoragePath); err != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("delete storage object %s failed: %v", record.StoragePath, err))
|
||||
continue
|
||||
}
|
||||
result.DeletedObjects++
|
||||
touchedProviders[object.targetID] = provider
|
||||
if record.BackupKind == model.BackupKindRepository {
|
||||
repositoryProviders[object.targetID] = provider
|
||||
}
|
||||
}
|
||||
if !allObjectsDeleted {
|
||||
continue
|
||||
}
|
||||
if err := s.records.Delete(ctx, record.ID); err != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("delete backup record %d failed: %v", record.ID, err))
|
||||
@@ -137,25 +76,13 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
|
||||
}
|
||||
result.DeletedRecords++
|
||||
}
|
||||
for targetID, provider := range repositoryProviders {
|
||||
pruned, pruneErr := backup.NewRepositoryStore(s.repositoryKey).Prune(ctx, provider)
|
||||
if pruneErr != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("prune CDC repository on target %d failed: %v", targetID, pruneErr))
|
||||
} else {
|
||||
result.DeletedObjects += pruned.DeletedPacks + pruned.DeletedIndexes
|
||||
}
|
||||
}
|
||||
|
||||
// 清理空目录:收集被删除文件的父目录,尝试移除空目录
|
||||
for targetID, provider := range touchedProviders {
|
||||
dirCleaner, ok := provider.(storage.StorageDirCleaner)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if dirCleaner, ok := provider.(storage.StorageDirCleaner); ok && result.DeletedObjects > 0 {
|
||||
prefixes := collectDirPrefixes(candidates)
|
||||
for _, prefix := range prefixes {
|
||||
if err := dirCleaner.RemoveEmptyDirs(ctx, prefix); err != nil {
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("cleanup empty dirs for %s on target %d: %v", prefix, targetID, err))
|
||||
result.Warnings = append(result.Warnings, fmt.Sprintf("cleanup empty dirs for %s: %v", prefix, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -163,87 +90,7 @@ func (s *Service) cleanup(ctx context.Context, task *model.BackupTask, resolvePr
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func cleanupObjectsForRecord(record model.BackupRecord) ([]cleanupObject, error) {
|
||||
defaultPath := strings.TrimSpace(record.StoragePath)
|
||||
if strings.TrimSpace(record.StorageUploadResults) == "" {
|
||||
if defaultPath == "" {
|
||||
return nil, nil
|
||||
}
|
||||
return []cleanupObject{{targetID: record.StorageTargetID, path: defaultPath}}, nil
|
||||
}
|
||||
var results []storedUploadResult
|
||||
if err := json.Unmarshal([]byte(record.StorageUploadResults), &results); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
objects := make([]cleanupObject, 0, len(results))
|
||||
seen := make(map[uint]struct{}, len(results))
|
||||
for _, result := range results {
|
||||
if !strings.EqualFold(strings.TrimSpace(result.Status), model.BackupRecordStatusSuccess) {
|
||||
continue
|
||||
}
|
||||
objectPath := strings.TrimSpace(result.StoragePath)
|
||||
if objectPath == "" {
|
||||
objectPath = defaultPath
|
||||
}
|
||||
if objectPath == "" {
|
||||
continue
|
||||
}
|
||||
if _, exists := seen[result.StorageTargetID]; exists {
|
||||
continue
|
||||
}
|
||||
seen[result.StorageTargetID] = struct{}{}
|
||||
objects = append(objects, cleanupObject{targetID: result.StorageTargetID, path: objectPath})
|
||||
}
|
||||
if len(objects) == 0 && defaultPath != "" {
|
||||
return nil, fmt.Errorf("successful record has no successful storage copy")
|
||||
}
|
||||
return objects, nil
|
||||
}
|
||||
|
||||
// protectDifferentialBases 从删除候选中剔除「仍被存活差异依赖的全量」,
|
||||
// 避免删除基线后其差异备份失去依据、无法恢复。全量仅当其全部差异都已过期/删除时才会被清理。
|
||||
func protectDifferentialBases(all []model.BackupRecord, candidates []model.BackupRecord) []model.BackupRecord {
|
||||
deleting := make(map[uint]struct{}, len(candidates))
|
||||
for _, r := range candidates {
|
||||
deleting[r.ID] = struct{}{}
|
||||
}
|
||||
protected := make(map[uint]struct{})
|
||||
for _, r := range all {
|
||||
if r.BackupKind != model.BackupKindDifferential || r.BaseRecordID == 0 {
|
||||
continue
|
||||
}
|
||||
if _, beingDeleted := deleting[r.ID]; beingDeleted {
|
||||
continue // 该差异本身也将被删除,无需保护其基线
|
||||
}
|
||||
protected[r.BaseRecordID] = struct{}{}
|
||||
}
|
||||
if len(protected) == 0 {
|
||||
return candidates
|
||||
}
|
||||
filtered := make([]model.BackupRecord, 0, len(candidates))
|
||||
for _, r := range candidates {
|
||||
if r.BackupKind == model.BackupKindFull {
|
||||
if _, keep := protected[r.ID]; keep {
|
||||
continue
|
||||
}
|
||||
}
|
||||
filtered = append(filtered, r)
|
||||
}
|
||||
return filtered
|
||||
}
|
||||
|
||||
func selectRecordsToDelete(records []model.BackupRecord, retentionDays int, maxBackups int, now time.Time) []model.BackupRecord {
|
||||
// 保留锁定(法律保留)的记录永不参与清理:先从候选集中剔除,
|
||||
// 锁定备份既不被删除,也不占用 maxBackups 轮转名额。
|
||||
if hasLocked(records) {
|
||||
unlocked := make([]model.BackupRecord, 0, len(records))
|
||||
for _, r := range records {
|
||||
if !r.Locked {
|
||||
unlocked = append(unlocked, r)
|
||||
}
|
||||
}
|
||||
records = unlocked
|
||||
}
|
||||
selected := make(map[uint]model.BackupRecord)
|
||||
if maxBackups > 0 && len(records) > maxBackups {
|
||||
for _, record := range records[maxBackups:] {
|
||||
@@ -266,81 +113,3 @@ func selectRecordsToDelete(records []model.BackupRecord, retentionDays int, maxB
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func hasLocked(records []model.BackupRecord) bool {
|
||||
for i := range records {
|
||||
if records[i].Locked {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// gfsEnabled 判定任务是否启用 GFS 分层保留(任一层级 > 0)。
|
||||
func gfsEnabled(task *model.BackupTask) bool {
|
||||
return task.KeepDaily > 0 || task.KeepWeekly > 0 || task.KeepMonthly > 0 || task.KeepYearly > 0
|
||||
}
|
||||
|
||||
func recordTime(r *model.BackupRecord) time.Time {
|
||||
if r.CompletedAt != nil {
|
||||
return *r.CompletedAt
|
||||
}
|
||||
return r.StartedAt
|
||||
}
|
||||
|
||||
func isoWeekKey(t time.Time) string {
|
||||
y, w := t.ISOWeek()
|
||||
return fmt.Sprintf("%d-W%02d", y, w)
|
||||
}
|
||||
|
||||
// selectGFSToDelete 按 GFS(祖父-父-子)策略选出应删除的记录。
|
||||
//
|
||||
// 规则:对每个层级(天/周/月/年),在按时间降序排列后,保留最近 keep 个不同周期中
|
||||
// 每个周期最新的一份备份;各层级保留集合取并集即「保留集」,其余删除。
|
||||
// 锁定(法律保留)的记录始终排除在删除候选之外。
|
||||
func selectGFSToDelete(records []model.BackupRecord, daily, weekly, monthly, yearly int) []model.BackupRecord {
|
||||
active := make([]model.BackupRecord, 0, len(records))
|
||||
for i := range records {
|
||||
if !records[i].Locked {
|
||||
active = append(active, records[i])
|
||||
}
|
||||
}
|
||||
sort.SliceStable(active, func(i, j int) bool {
|
||||
return recordTime(&active[i]).After(recordTime(&active[j]))
|
||||
})
|
||||
|
||||
keep := make(map[uint]bool, len(active))
|
||||
keepTier := func(count int, key func(time.Time) string) {
|
||||
if count <= 0 {
|
||||
return
|
||||
}
|
||||
periods := 0
|
||||
lastPeriod := ""
|
||||
havePrev := false
|
||||
for i := range active {
|
||||
p := key(recordTime(&active[i]))
|
||||
if havePrev && p == lastPeriod {
|
||||
continue // 同周期已保留代表(最新一份)
|
||||
}
|
||||
if periods >= count {
|
||||
break // 该层级已保留足够多的周期
|
||||
}
|
||||
keep[active[i].ID] = true
|
||||
lastPeriod = p
|
||||
havePrev = true
|
||||
periods++
|
||||
}
|
||||
}
|
||||
keepTier(daily, func(t time.Time) string { return t.Format("2006-01-02") })
|
||||
keepTier(weekly, isoWeekKey)
|
||||
keepTier(monthly, func(t time.Time) string { return t.Format("2006-01") })
|
||||
keepTier(yearly, func(t time.Time) string { return t.Format("2006") })
|
||||
|
||||
del := make([]model.BackupRecord, 0)
|
||||
for i := range active {
|
||||
if !keep[active[i].ID] {
|
||||
del = append(del, active[i])
|
||||
}
|
||||
}
|
||||
return del
|
||||
}
|
||||
|
||||
@@ -24,9 +24,6 @@ func (r *fakeRecordRepository) List(context.Context, repository.BackupRecordList
|
||||
func (r *fakeRecordRepository) FindByID(context.Context, uint) (*model.BackupRecord, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (r *fakeRecordRepository) FindRunningByTaskAndNode(context.Context, uint, uint) (*model.BackupRecord, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (r *fakeRecordRepository) Create(context.Context, *model.BackupRecord) error { return nil }
|
||||
func (r *fakeRecordRepository) Update(context.Context, *model.BackupRecord) error { return nil }
|
||||
func (r *fakeRecordRepository) Delete(_ context.Context, id uint) error {
|
||||
@@ -45,9 +42,6 @@ func (r *fakeRecordRepository) ListByTask(_ context.Context, _ uint) ([]model.Ba
|
||||
func (r *fakeRecordRepository) ListSuccessfulByTask(_ context.Context, _ uint) ([]model.BackupRecord, error) {
|
||||
return r.records, nil
|
||||
}
|
||||
func (r *fakeRecordRepository) CountDependentDifferentials(context.Context, uint) (int64, error) {
|
||||
return 0, nil
|
||||
}
|
||||
func (r *fakeRecordRepository) Count(context.Context) (int64, error) { return 0, nil }
|
||||
func (r *fakeRecordRepository) CountSince(context.Context, time.Time) (int64, error) { return 0, nil }
|
||||
func (r *fakeRecordRepository) CountSuccessSince(context.Context, time.Time) (int64, error) {
|
||||
@@ -96,105 +90,6 @@ func TestSelectRecordsToDelete(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func gfsRecord(id uint, ts time.Time, locked bool) model.BackupRecord {
|
||||
completed := ts
|
||||
return model.BackupRecord{ID: id, StartedAt: ts, CompletedAt: &completed, Locked: locked}
|
||||
}
|
||||
|
||||
func gfsDay(y, m, d, h int) time.Time {
|
||||
return time.Date(y, time.Month(m), d, h, 0, 0, 0, time.UTC)
|
||||
}
|
||||
|
||||
func deletedIDSet(records []model.BackupRecord) map[uint]bool {
|
||||
out := make(map[uint]bool, len(records))
|
||||
for i := range records {
|
||||
out[records[i].ID] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func assertDeleted(t *testing.T, del []model.BackupRecord, want ...uint) {
|
||||
t.Helper()
|
||||
got := deletedIDSet(del)
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("deleted set size = %d %v, want %d %v", len(got), got, len(want), want)
|
||||
}
|
||||
for _, id := range want {
|
||||
if !got[id] {
|
||||
t.Fatalf("expected id %d to be deleted; got %v", id, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSelectGFSToDelete_DailyTier 验证按天分层:每天仅保留最新一份,且只保留最近 N 天。
|
||||
func TestSelectGFSToDelete_DailyTier(t *testing.T) {
|
||||
records := []model.BackupRecord{
|
||||
gfsRecord(5, gfsDay(2026, 3, 7, 12), false), // 今天,最新 → 保留
|
||||
gfsRecord(4, gfsDay(2026, 3, 7, 6), false), // 今天,较早 → 删除(非当天代表)
|
||||
gfsRecord(3, gfsDay(2026, 3, 6, 12), false), // 昨天 → 保留
|
||||
gfsRecord(2, gfsDay(2026, 3, 5, 12), false), // 前天 → 超出 daily=2 → 删除
|
||||
gfsRecord(1, gfsDay(2026, 3, 4, 12), false), // 更早 → 删除
|
||||
}
|
||||
del := selectGFSToDelete(records, 2, 0, 0, 0)
|
||||
assertDeleted(t, del, 4, 2, 1)
|
||||
}
|
||||
|
||||
// TestSelectGFSToDelete_TierUnion 验证多层级取并集:月度层级保留日度层级会删除的旧备份。
|
||||
func TestSelectGFSToDelete_TierUnion(t *testing.T) {
|
||||
records := []model.BackupRecord{
|
||||
gfsRecord(3, gfsDay(2026, 3, 7, 12), false), // 3 月(最新)
|
||||
gfsRecord(2, gfsDay(2026, 2, 15, 12), false), // 2 月
|
||||
gfsRecord(1, gfsDay(2026, 1, 15, 12), false), // 1 月
|
||||
}
|
||||
// daily=1 只留 ID3;monthly=2 留最近两个月(3 月=ID3、2 月=ID2)。并集={3,2},删除 ID1。
|
||||
del := selectGFSToDelete(records, 1, 0, 2, 0)
|
||||
assertDeleted(t, del, 1)
|
||||
}
|
||||
|
||||
// TestSelectGFSToDelete_SkipsLocked 验证锁定记录即使超出所有层级也永不删除。
|
||||
func TestSelectGFSToDelete_SkipsLocked(t *testing.T) {
|
||||
records := []model.BackupRecord{
|
||||
gfsRecord(3, gfsDay(2026, 3, 7, 12), false),
|
||||
gfsRecord(2, gfsDay(2026, 3, 6, 12), false),
|
||||
gfsRecord(1, gfsDay(2020, 1, 1, 12), true), // 远超 daily=1 但已锁定 → 不删
|
||||
}
|
||||
del := selectGFSToDelete(records, 1, 0, 0, 0)
|
||||
assertDeleted(t, del, 2) // 仅 ID2 被删;ID1 锁定豁免,ID3 为当日代表
|
||||
}
|
||||
|
||||
func TestGFSEnabled(t *testing.T) {
|
||||
if gfsEnabled(&model.BackupTask{}) {
|
||||
t.Fatal("empty GFS config should be disabled")
|
||||
}
|
||||
if !gfsEnabled(&model.BackupTask{KeepWeekly: 4}) {
|
||||
t.Fatal("KeepWeekly>0 should enable GFS")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSelectRecordsToDelete_SkipsLocked 验证保留锁定(法律保留)的记录永不被选中删除,
|
||||
// 即使它既超过保留期、又超过 maxBackups 名额。
|
||||
func TestSelectRecordsToDelete_SkipsLocked(t *testing.T) {
|
||||
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
|
||||
completedNew := now.Add(-24 * time.Hour)
|
||||
completedOld := now.Add(-15 * 24 * time.Hour)
|
||||
records := []model.BackupRecord{
|
||||
{ID: 3, CompletedAt: &completedNew},
|
||||
{ID: 2, CompletedAt: &completedNew},
|
||||
{ID: 1, CompletedAt: &completedOld, Locked: true}, // 超期但锁定 → 不应删除
|
||||
}
|
||||
selected := selectRecordsToDelete(records, 7, 2, now)
|
||||
for _, r := range selected {
|
||||
if r.ID == 1 {
|
||||
t.Fatalf("locked record #1 must never be selected for deletion: %#v", selected)
|
||||
}
|
||||
}
|
||||
// 锁定记录不占 maxBackups 名额:未锁定仅 2 条,maxBackups=2 → 无超额删除,
|
||||
// 且无未锁定记录超期 → 选中集为空。
|
||||
if len(selected) != 0 {
|
||||
t.Fatalf("expected no deletions (locked excluded), got %#v", selected)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupDeletesExpiredRecords(t *testing.T) {
|
||||
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
|
||||
completedNew := now.Add(-24 * time.Hour)
|
||||
@@ -221,66 +116,3 @@ func TestCleanupDeletesExpiredRecords(t *testing.T) {
|
||||
t.Fatalf("unexpected deleted objects: %#v", provider.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupProvidersDeletesEverySuccessfulCopyBeforeRecord(t *testing.T) {
|
||||
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
|
||||
completedNew := now.Add(-time.Hour)
|
||||
completedOld := now.Add(-24 * time.Hour)
|
||||
repo := &fakeRecordRepository{records: []model.BackupRecord{
|
||||
{ID: 2, TaskID: 1, StoragePath: "records/2", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedNew},
|
||||
{
|
||||
ID: 1, TaskID: 1, StoragePath: "records/1", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedOld,
|
||||
StorageUploadResults: `[{"storageTargetId":11,"status":"success","storagePath":"first/1"},{"storageTargetId":12,"status":"success","storagePath":"second/1"},{"storageTargetId":13,"status":"failed"}]`,
|
||||
},
|
||||
}}
|
||||
first := &fakeProvider{}
|
||||
second := &fakeProvider{}
|
||||
service := NewService(repo)
|
||||
service.now = func() time.Time { return now }
|
||||
|
||||
result, err := service.CleanupProviders(context.Background(), &model.BackupTask{ID: 1, MaxBackups: 1}, map[uint]storage.StorageProvider{
|
||||
11: first,
|
||||
12: second,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("CleanupProviders returned error: %v", err)
|
||||
}
|
||||
if result.DeletedRecords != 1 || result.DeletedObjects != 2 || len(result.Warnings) != 0 {
|
||||
t.Fatalf("unexpected cleanup result: %#v", result)
|
||||
}
|
||||
if len(repo.deleted) != 1 || repo.deleted[0] != 1 {
|
||||
t.Fatalf("unexpected deleted records: %#v", repo.deleted)
|
||||
}
|
||||
if len(first.deleted) != 1 || first.deleted[0] != "first/1" {
|
||||
t.Fatalf("unexpected first-target deletes: %#v", first.deleted)
|
||||
}
|
||||
if len(second.deleted) != 1 || second.deleted[0] != "second/1" {
|
||||
t.Fatalf("unexpected second-target deletes: %#v", second.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupProvidersKeepsRecordWhenCopyProviderIsUnavailable(t *testing.T) {
|
||||
now := time.Date(2026, 3, 7, 16, 0, 0, 0, time.UTC)
|
||||
completedNew := now.Add(-time.Hour)
|
||||
completedOld := now.Add(-24 * time.Hour)
|
||||
repo := &fakeRecordRepository{records: []model.BackupRecord{
|
||||
{ID: 2, TaskID: 1, StoragePath: "records/2", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedNew},
|
||||
{
|
||||
ID: 1, TaskID: 1, StoragePath: "records/1", Status: model.BackupRecordStatusSuccess, CompletedAt: &completedOld,
|
||||
StorageUploadResults: `[{"storageTargetId":11,"status":"success","storagePath":"first/1"},{"storageTargetId":12,"status":"success","storagePath":"second/1"}]`,
|
||||
},
|
||||
}}
|
||||
first := &fakeProvider{}
|
||||
service := NewService(repo)
|
||||
|
||||
result, err := service.CleanupProviders(context.Background(), &model.BackupTask{ID: 1, MaxBackups: 1}, map[uint]storage.StorageProvider{11: first})
|
||||
if err != nil {
|
||||
t.Fatalf("CleanupProviders returned error: %v", err)
|
||||
}
|
||||
if result.DeletedRecords != 0 || result.DeletedObjects != 1 || len(result.Warnings) != 1 {
|
||||
t.Fatalf("unexpected safe partial-cleanup result: %#v", result)
|
||||
}
|
||||
if len(repo.deleted) != 0 {
|
||||
t.Fatalf("record must remain until all copies are deleted: %#v", repo.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,17 +55,6 @@ func BuildStorageKey(backupType string, startedAt time.Time, fileName string) st
|
||||
return filepath.ToSlash(filepath.Join("BackupX", typeName, startedAt.UTC().Format("060102"), fileName))
|
||||
}
|
||||
|
||||
// BuildRecordStorageKey gives remote-Agent artifacts a record-owned namespace.
|
||||
// The Master validates this namespace before accepting a relayed upload, so one
|
||||
// Agent cannot overwrite another record's object on centrally mounted storage.
|
||||
func BuildRecordStorageKey(backupType string, startedAt time.Time, recordID uint, fileName string) string {
|
||||
typeName := strings.TrimSpace(strings.ToLower(backupType))
|
||||
if typeName == "" {
|
||||
typeName = "file"
|
||||
}
|
||||
return filepath.ToSlash(filepath.Join("BackupX", typeName, startedAt.UTC().Format("060102"), "records", fmt.Sprintf("%d", recordID), fileName))
|
||||
}
|
||||
|
||||
func sanitizeTaskName(value string) string {
|
||||
trimmed := strings.TrimSpace(strings.ToLower(value))
|
||||
trimmed = strings.ReplaceAll(trimmed, " ", "-")
|
||||
|
||||
@@ -36,15 +36,6 @@ type TaskSpec struct {
|
||||
MaxBackups int
|
||||
StartedAt time.Time
|
||||
TempDir string
|
||||
// Differential 为 true 时执行差异备份:仅打包自 BaseManifest 以来新增/变更的条目,
|
||||
// 并记录被删除的路径。仅文件类型任务支持;BaseManifest 为空时回退为全量。
|
||||
Differential bool
|
||||
BaseManifest Manifest
|
||||
// SelectedPaths 非空时仅恢复这些归档相对路径(及其子项),用于按需(选择性)恢复;仅文件类型生效。
|
||||
SelectedPaths []string
|
||||
// RestoreTargetPath 仅用于恢复:非空时,文件类型恢复将归档解压到该目录,
|
||||
// 而非默认的原始源路径父目录。用于「恢复到指定位置」(迁移/测试/并排恢复)。
|
||||
RestoreTargetPath string
|
||||
}
|
||||
|
||||
type RunResult struct {
|
||||
@@ -53,8 +44,6 @@ type RunResult struct {
|
||||
TempDir string
|
||||
Size int64
|
||||
StorageKey string
|
||||
// Manifest 为全量备份产出的条目清单,供后续差异备份比对;差异备份运行时为 nil。
|
||||
Manifest *Manifest
|
||||
}
|
||||
|
||||
type LogEvent struct {
|
||||
@@ -73,7 +62,7 @@ type ProgressInfo struct {
|
||||
BytesSent int64 `json:"bytesSent"`
|
||||
TotalBytes int64 `json:"totalBytes"`
|
||||
Percent float64 `json:"percent"`
|
||||
SpeedBps float64 `json:"speedBps"` // bytes/sec
|
||||
SpeedBps float64 `json:"speedBps"` // bytes/sec
|
||||
TargetName string `json:"targetName"`
|
||||
}
|
||||
|
||||
|
||||
@@ -17,14 +17,9 @@ type Config struct {
|
||||
}
|
||||
|
||||
type ServerConfig struct {
|
||||
Host string `mapstructure:"host"`
|
||||
Port int `mapstructure:"port"`
|
||||
Mode string `mapstructure:"mode"`
|
||||
ExternalURL string `mapstructure:"external_url"`
|
||||
// WebRoot 指向前端构建产物目录。留空时后端会按部署惯例自动探测
|
||||
// (./web、./web/dist、/opt/backupx/web 等)。探测命中后后端直接托管
|
||||
// 前端 SPA,无需额外的 nginx 反向代理即可访问 Web 控制台。
|
||||
WebRoot string `mapstructure:"web_root"`
|
||||
Host string `mapstructure:"host"`
|
||||
Port int `mapstructure:"port"`
|
||||
Mode string `mapstructure:"mode"`
|
||||
}
|
||||
|
||||
type DatabaseConfig struct {
|
||||
@@ -141,8 +136,6 @@ func applyDefaults(v *viper.Viper) {
|
||||
v.SetDefault("server.host", "0.0.0.0")
|
||||
v.SetDefault("server.port", 8340)
|
||||
v.SetDefault("server.mode", "release")
|
||||
v.SetDefault("server.external_url", "")
|
||||
v.SetDefault("server.web_root", "")
|
||||
v.SetDefault("database.path", "./data/backupx.db")
|
||||
v.SetDefault("security.jwt_expire", "24h")
|
||||
v.SetDefault("backup.temp_dir", "/tmp/backupx")
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
import "testing"
|
||||
|
||||
func TestLoadUsesDefaultsWithoutConfigFile(t *testing.T) {
|
||||
cfg, err := Load("")
|
||||
@@ -22,33 +18,3 @@ func TestLoadUsesDefaultsWithoutConfigFile(t *testing.T) {
|
||||
t.Fatalf("expected default database path, got %s", cfg.Database.Path)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadReadsServerExternalURLFromFile(t *testing.T) {
|
||||
configPath := filepath.Join(t.TempDir(), "config.yaml")
|
||||
content := []byte("server:\n external_url: \"https://backup.example.com\"\n")
|
||||
if err := os.WriteFile(configPath, content, 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
cfg, err := Load(configPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load returned error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.Server.ExternalURL != "https://backup.example.com" {
|
||||
t.Fatalf("expected external URL from config, got %q", cfg.Server.ExternalURL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadReadsServerExternalURLFromEnv(t *testing.T) {
|
||||
t.Setenv("BACKUPX_SERVER_EXTERNAL_URL", "https://env-backup.example.com")
|
||||
|
||||
cfg, err := Load("")
|
||||
if err != nil {
|
||||
t.Fatalf("Load returned error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.Server.ExternalURL != "https://env-backup.example.com" {
|
||||
t.Fatalf("expected external URL from env, got %q", cfg.Server.ExternalURL)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -156,44 +156,6 @@ func (h *AgentHandler) UpdateRecord(c *gin.Context) {
|
||||
response.Success(c, gin.H{"status": "ok"})
|
||||
}
|
||||
|
||||
// UploadArtifact streams a remote source artifact into storage mounted only on
|
||||
// the Master. The request body is never buffered as a whole in memory or disk.
|
||||
func (h *AgentHandler) UploadArtifact(c *gin.Context) {
|
||||
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
recordID, err := strconv.ParseUint(c.Param("id"), 10, 32)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
targetID, err := strconv.ParseUint(c.Param("targetId"), 10, 32)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
if c.Request.ContentLength < 0 {
|
||||
c.JSON(stdhttp.StatusLengthRequired, gin.H{"code": "CONTENT_LENGTH_REQUIRED", "message": "artifact content length is required"})
|
||||
return
|
||||
}
|
||||
if err := h.agentService.UploadArtifact(
|
||||
c.Request.Context(),
|
||||
node,
|
||||
uint(recordID),
|
||||
uint(targetID),
|
||||
c.GetHeader("X-BackupX-Object-Key"),
|
||||
c.Request.ContentLength,
|
||||
c.GetHeader("X-BackupX-SHA256"),
|
||||
c.Request.Body,
|
||||
); err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
response.Success(c, gin.H{"status": "ok"})
|
||||
}
|
||||
|
||||
// GetRestoreSpec Agent 拉取恢复规格。
|
||||
func (h *AgentHandler) GetRestoreSpec(c *gin.Context) {
|
||||
if h.restoreService == nil {
|
||||
@@ -246,34 +208,6 @@ func (h *AgentHandler) UpdateRestore(c *gin.Context) {
|
||||
response.Success(c, gin.H{"status": "ok"})
|
||||
}
|
||||
|
||||
// DownloadRestoreArtifact streams a Master-local backup back to its source
|
||||
// Agent for restore without exposing the local storage configuration.
|
||||
func (h *AgentHandler) DownloadRestoreArtifact(c *gin.Context) {
|
||||
if h.restoreService == nil {
|
||||
c.JSON(stdhttp.StatusServiceUnavailable, gin.H{"code": "RESTORE_SERVICE_DISABLED", "message": "restore service is not enabled"})
|
||||
return
|
||||
}
|
||||
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
restoreID, err := strconv.ParseUint(c.Param("id"), 10, 32)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
artifact, err := h.restoreService.DownloadAgentArtifact(c.Request.Context(), node, uint(restoreID))
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
c.DataFromReader(stdhttp.StatusOK, artifact.Size, "application/octet-stream", artifact.Reader, nil)
|
||||
if err := artifact.Reader.Close(); err != nil {
|
||||
_ = c.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// Self 返回当前 Agent token 所属节点的状态,供安装脚本末尾探活。
|
||||
func (h *AgentHandler) Self(c *gin.Context) {
|
||||
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
|
||||
|
||||
@@ -52,20 +52,6 @@ func (h *BackupRecordHandler) Get(c *gin.Context) {
|
||||
response.Success(c, item)
|
||||
}
|
||||
|
||||
// Contents 返回备份记录的文件清单(内容浏览,只读)。
|
||||
func (h *BackupRecordHandler) Contents(c *gin.Context) {
|
||||
id, ok := parseUintParam(c, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
contents, err := h.service.ListContents(c.Request.Context(), id)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
response.Success(c, contents)
|
||||
}
|
||||
|
||||
func (h *BackupRecordHandler) StreamLogs(c *gin.Context) {
|
||||
id, ok := parseUintParam(c, "id")
|
||||
if !ok {
|
||||
@@ -151,14 +137,7 @@ func (h *BackupRecordHandler) Restore(c *gin.Context) {
|
||||
if subject, exists := c.Get(contextUserSubjectKey); exists {
|
||||
triggeredBy = strings.TrimSpace(fmt.Sprintf("%v", subject))
|
||||
}
|
||||
// 可选请求体:selectedPaths 按需(选择性)恢复;targetPath 恢复到指定目录(仅文件类型本机恢复)。
|
||||
// 无 body 时为整体恢复到原始路径。
|
||||
var body struct {
|
||||
SelectedPaths []string `json:"selectedPaths"`
|
||||
TargetPath string `json:"targetPath"`
|
||||
}
|
||||
_ = c.ShouldBindJSON(&body)
|
||||
detail, err := h.restoreService.StartSelective(c.Request.Context(), id, body.SelectedPaths, strings.TrimSpace(body.TargetPath), triggeredBy)
|
||||
detail, err := h.restoreService.Start(c.Request.Context(), id, triggeredBy)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
@@ -182,32 +161,6 @@ func (h *BackupRecordHandler) Delete(c *gin.Context) {
|
||||
response.Success(c, gin.H{"deleted": true})
|
||||
}
|
||||
|
||||
// SetLock 设置/解除备份记录的保留锁定(法律保留)。
|
||||
func (h *BackupRecordHandler) SetLock(c *gin.Context) {
|
||||
id, ok := parseUintParam(c, "id")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var input struct {
|
||||
Locked bool `json:"locked"`
|
||||
}
|
||||
if err := c.ShouldBindJSON(&input); err != nil {
|
||||
response.Error(c, apperror.BadRequest("BACKUP_RECORD_LOCK_INVALID", "锁定参数不合法", err))
|
||||
return
|
||||
}
|
||||
detail, err := h.service.SetLock(c.Request.Context(), id, input.Locked)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
action, desc := "unlock", fmt.Sprintf("解除备份记录保留锁定 (ID: %d)", id)
|
||||
if input.Locked {
|
||||
action, desc = "lock", fmt.Sprintf("设置备份记录保留锁定 (ID: %d)", id)
|
||||
}
|
||||
recordAudit(c, h.auditService, "backup_record", action, "backup_record", fmt.Sprintf("%d", id), "", desc)
|
||||
response.Success(c, detail)
|
||||
}
|
||||
|
||||
func (h *BackupRecordHandler) BatchDelete(c *gin.Context) {
|
||||
var input struct {
|
||||
IDs []uint `json:"ids" binding:"required,min=1"`
|
||||
@@ -263,7 +216,7 @@ func writeSSEEvent(writer io.Writer, event backup.LogEvent) error {
|
||||
}
|
||||
|
||||
func parseUintString(value string) (uint, bool) {
|
||||
parsed, err := strconv.ParseUint(strings.TrimSpace(value), 10, 0)
|
||||
parsed, err := strconv.ParseUint(strings.TrimSpace(value), 10, 64)
|
||||
if err != nil {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
@@ -25,14 +25,10 @@ import (
|
||||
// setupInstallFlowRouter 构造一个 Node + Agent + InstallToken 全量依赖的 router,
|
||||
// 并返回已登录管理员 JWT。
|
||||
func setupInstallFlowRouter(t *testing.T) (http.Handler, string) {
|
||||
return setupInstallFlowRouterWithExternalURL(t, "")
|
||||
}
|
||||
|
||||
func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (http.Handler, string) {
|
||||
t.Helper()
|
||||
tempDir := t.TempDir()
|
||||
cfg := config.Config{
|
||||
Server: config.ServerConfig{Host: "127.0.0.1", Port: 8340, Mode: "test", ExternalURL: externalURL},
|
||||
Server: config.ServerConfig{Host: "127.0.0.1", Port: 8340, Mode: "test"},
|
||||
Database: config.DatabaseConfig{Path: filepath.Join(tempDir, "backupx.db")},
|
||||
Security: config.SecurityConfig{JWTExpire: "24h"},
|
||||
Log: config.LogConfig{Level: "error"},
|
||||
@@ -72,6 +68,9 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
|
||||
installTokenRepo := repository.NewAgentInstallTokenRepository(db)
|
||||
installTokenSvc := service.NewInstallTokenService(installTokenRepo, nodeRepo)
|
||||
|
||||
auditLogRepo := repository.NewAuditLogRepository(db)
|
||||
auditSvc := service.NewAuditService(auditLogRepo)
|
||||
|
||||
// 用 cancelable ctx,测试结束时停掉 handler 启动的后台 GC 协程,
|
||||
// 避免 goroutine 持有 map 导致 tempdir 清理失败。
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
@@ -86,7 +85,7 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
|
||||
SystemService: systemSvc,
|
||||
NodeService: nodeSvc,
|
||||
InstallTokenService: installTokenSvc,
|
||||
MasterExternalURL: cfg.Server.ExternalURL,
|
||||
AuditService: auditSvc,
|
||||
JWTManager: jwtMgr,
|
||||
UserRepository: userRepo,
|
||||
SystemConfigRepo: systemConfigRepo,
|
||||
@@ -115,73 +114,6 @@ func setupInstallFlowRouterWithExternalURL(t *testing.T, externalURL string) (ht
|
||||
return router, setupResp.Data.Token
|
||||
}
|
||||
|
||||
func TestInstallTokenUsesConfiguredExternalURL(t *testing.T) {
|
||||
const externalURL = "https://public.example.com/base"
|
||||
router, jwt := setupInstallFlowRouterWithExternalURL(t, externalURL)
|
||||
|
||||
batchBody, _ := json.Marshal(map[string][]string{"names": {"external-url-node"}})
|
||||
batchReq := httptest.NewRequest(http.MethodPost, "/api/nodes/batch", bytes.NewBuffer(batchBody))
|
||||
batchReq.Header.Set("Content-Type", "application/json")
|
||||
batchReq.Header.Set("Authorization", "Bearer "+jwt)
|
||||
batchRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(batchRec, batchReq)
|
||||
if batchRec.Code != 200 {
|
||||
t.Fatalf("batch create failed: %d %s", batchRec.Code, batchRec.Body.String())
|
||||
}
|
||||
var batchResp struct {
|
||||
Data []struct {
|
||||
ID uint `json:"id"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(batchRec.Body.Bytes(), &batchResp); err != nil {
|
||||
t.Fatalf("unmarshal batch: %v", err)
|
||||
}
|
||||
if len(batchResp.Data) != 1 {
|
||||
t.Fatalf("expected 1 node, got %d", len(batchResp.Data))
|
||||
}
|
||||
|
||||
genBody, _ := json.Marshal(map[string]any{
|
||||
"mode": "systemd",
|
||||
"arch": "auto",
|
||||
"agentVersion": "v1.7.0",
|
||||
"downloadSrc": "github",
|
||||
"ttlSeconds": 900,
|
||||
})
|
||||
genReq := httptest.NewRequest(http.MethodPost,
|
||||
"/api/nodes/"+formatUint(batchResp.Data[0].ID)+"/install-tokens", bytes.NewBuffer(genBody))
|
||||
genReq.Header.Set("Content-Type", "application/json")
|
||||
genReq.Header.Set("Authorization", "Bearer "+jwt)
|
||||
genRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(genRec, genReq)
|
||||
if genRec.Code != 200 {
|
||||
t.Fatalf("install-tokens failed: %d %s", genRec.Code, genRec.Body.String())
|
||||
}
|
||||
var genResp struct {
|
||||
Data struct {
|
||||
InstallToken string `json:"installToken"`
|
||||
URL string `json:"url"`
|
||||
FallbackURL string `json:"fallbackUrl"`
|
||||
ScriptBase64 string `json:"scriptBase64"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(genRec.Body.Bytes(), &genResp); err != nil {
|
||||
t.Fatalf("unmarshal gen: %v", err)
|
||||
}
|
||||
if genResp.Data.URL != externalURL+"/api/install/"+genResp.Data.InstallToken {
|
||||
t.Fatalf("url should use external URL, got %q", genResp.Data.URL)
|
||||
}
|
||||
if genResp.Data.FallbackURL != externalURL+"/install/"+genResp.Data.InstallToken {
|
||||
t.Fatalf("fallbackUrl should use external URL, got %q", genResp.Data.FallbackURL)
|
||||
}
|
||||
decodedScript, err := base64.StdEncoding.DecodeString(genResp.Data.ScriptBase64)
|
||||
if err != nil {
|
||||
t.Fatalf("scriptBase64 should be valid base64: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(decodedScript), `MASTER_URL="`+externalURL+`"`) {
|
||||
t.Fatalf("script should use external MASTER_URL:\n%s", string(decodedScript))
|
||||
}
|
||||
}
|
||||
|
||||
func TestOneClickInstallFlow(t *testing.T) {
|
||||
router, jwt := setupInstallFlowRouter(t)
|
||||
|
||||
@@ -496,76 +428,6 @@ func TestInstallFlowComposeModeMismatch(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstallFlowComposeSuccessConsumesToken(t *testing.T) {
|
||||
router, jwt := setupInstallFlowRouter(t)
|
||||
|
||||
batchBody, _ := json.Marshal(map[string][]string{"names": {"compose-ok"}})
|
||||
batchReq := httptest.NewRequest(http.MethodPost, "/api/nodes/batch", bytes.NewBuffer(batchBody))
|
||||
batchReq.Header.Set("Content-Type", "application/json")
|
||||
batchReq.Header.Set("Authorization", "Bearer "+jwt)
|
||||
batchRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(batchRec, batchReq)
|
||||
if batchRec.Code != 200 {
|
||||
t.Fatalf("batch create failed: %d %s", batchRec.Code, batchRec.Body.String())
|
||||
}
|
||||
var batchResp struct {
|
||||
Data []struct {
|
||||
ID uint `json:"id"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(batchRec.Body.Bytes(), &batchResp); err != nil {
|
||||
t.Fatalf("unmarshal batch: %v", err)
|
||||
}
|
||||
if len(batchResp.Data) != 1 {
|
||||
t.Fatalf("expected 1 node, got %d", len(batchResp.Data))
|
||||
}
|
||||
|
||||
genBody, _ := json.Marshal(map[string]any{
|
||||
"mode": "docker",
|
||||
"arch": "auto",
|
||||
"agentVersion": "v1.7.0",
|
||||
"downloadSrc": "github",
|
||||
"ttlSeconds": 900,
|
||||
})
|
||||
genReq := httptest.NewRequest(http.MethodPost,
|
||||
"/api/nodes/"+formatUint(batchResp.Data[0].ID)+"/install-tokens", bytes.NewBuffer(genBody))
|
||||
genReq.Header.Set("Content-Type", "application/json")
|
||||
genReq.Header.Set("Authorization", "Bearer "+jwt)
|
||||
genRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(genRec, genReq)
|
||||
if genRec.Code != 200 {
|
||||
t.Fatalf("install-tokens failed: %d %s", genRec.Code, genRec.Body.String())
|
||||
}
|
||||
var genResp struct {
|
||||
Data struct {
|
||||
InstallToken string `json:"installToken"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(genRec.Body.Bytes(), &genResp); err != nil {
|
||||
t.Fatalf("unmarshal gen: %v", err)
|
||||
}
|
||||
if genResp.Data.InstallToken == "" {
|
||||
t.Fatalf("missing installToken")
|
||||
}
|
||||
|
||||
composeReq := httptest.NewRequest(http.MethodGet, "/api/install/"+genResp.Data.InstallToken+"/compose.yml", nil)
|
||||
composeRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(composeRec, composeReq)
|
||||
if composeRec.Code != 200 {
|
||||
t.Fatalf("compose fetch failed: %d %s", composeRec.Code, composeRec.Body.String())
|
||||
}
|
||||
if !strings.Contains(composeRec.Body.String(), "BACKUPX_AGENT_TOKEN") {
|
||||
t.Fatalf("compose missing token env:\n%s", composeRec.Body.String())
|
||||
}
|
||||
|
||||
scriptReq := httptest.NewRequest(http.MethodGet, "/api/install/"+genResp.Data.InstallToken, nil)
|
||||
scriptRec := httptest.NewRecorder()
|
||||
router.ServeHTTP(scriptRec, scriptReq)
|
||||
if scriptRec.Code != http.StatusGone {
|
||||
t.Fatalf("script after compose should be 410, got %d: %s", scriptRec.Code, scriptRec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
// formatUint 小工具:uint → 十进制字符串(无需引入 strconv)。
|
||||
func formatUint(u uint) string {
|
||||
if u == 0 {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
stdhttp "net/http"
|
||||
"strconv"
|
||||
@@ -244,17 +245,14 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
|
||||
input.TTLSeconds = 900
|
||||
}
|
||||
|
||||
out, err := h.installTokenSvc.CreateCommand(c.Request.Context(), service.InstallCommandInput{
|
||||
InstallTokenInput: service.InstallTokenInput{
|
||||
NodeID: uint(id),
|
||||
Mode: input.Mode,
|
||||
Arch: input.Arch,
|
||||
AgentVersion: input.AgentVersion,
|
||||
DownloadSrc: input.DownloadSrc,
|
||||
TTLSeconds: input.TTLSeconds,
|
||||
CreatedByID: h.resolveCurrentUserID(c),
|
||||
},
|
||||
MasterURL: resolveMasterURL(c, h.externalURL),
|
||||
out, err := h.installTokenSvc.Create(c.Request.Context(), service.InstallTokenInput{
|
||||
NodeID: uint(id),
|
||||
Mode: input.Mode,
|
||||
Arch: input.Arch,
|
||||
AgentVersion: input.AgentVersion,
|
||||
DownloadSrc: input.DownloadSrc,
|
||||
TTLSeconds: input.TTLSeconds,
|
||||
CreatedByID: h.resolveCurrentUserID(c),
|
||||
})
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
@@ -264,6 +262,12 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
|
||||
fmt.Sprintf("%d", id), out.Node.Name,
|
||||
fmt.Sprintf("生成 %s/%s install token TTL=%ds", input.Mode, input.Arch, input.TTLSeconds))
|
||||
|
||||
masterURL := resolveMasterURL(c, h.externalURL)
|
||||
script, err := renderInstallScript(masterURL, out.Node, out.Record)
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
// 使用 /api/install/... 而非 /install/... —— 让反向代理的 /api/ 转发规则
|
||||
// 自动接管,避免 SPA fallback 把请求当成前端路由返回 index.html(issue #46)。
|
||||
// 同时返回 /install/... 备用地址,兼容会剥离 /api 前缀的外层反向代理。
|
||||
@@ -272,11 +276,15 @@ func (h *NodeHandler) CreateInstallToken(c *gin.Context) {
|
||||
body := gin.H{
|
||||
"installToken": out.Token,
|
||||
"expiresAt": out.ExpiresAt,
|
||||
"url": out.URL,
|
||||
"fallbackUrl": out.FallbackURL,
|
||||
"scriptBase64": out.ScriptBase64,
|
||||
"composeUrl": out.ComposeURL,
|
||||
"fallbackComposeUrl": out.FallbackComposeURL,
|
||||
"url": masterURL + "/api/install/" + out.Token,
|
||||
"fallbackUrl": masterURL + "/install/" + out.Token,
|
||||
"scriptBase64": base64.StdEncoding.EncodeToString([]byte(script)),
|
||||
"composeUrl": "",
|
||||
"fallbackComposeUrl": "",
|
||||
}
|
||||
if input.Mode == "docker" {
|
||||
body["composeUrl"] = masterURL + "/api/install/" + out.Token + "/compose.yml"
|
||||
body["fallbackComposeUrl"] = masterURL + "/install/" + out.Token + "/compose.yml"
|
||||
}
|
||||
response.Success(c, body)
|
||||
}
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"encoding/csv"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/service"
|
||||
"backupx/server/pkg/response"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
type ReportHandler struct {
|
||||
service *service.ReportService
|
||||
}
|
||||
|
||||
func NewReportHandler(reportService *service.ReportService) *ReportHandler {
|
||||
return &ReportHandler{service: reportService}
|
||||
}
|
||||
|
||||
func reportDays(c *gin.Context) int {
|
||||
days := 30
|
||||
if v := strings.TrimSpace(c.Query("days")); v != "" {
|
||||
if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
|
||||
days = parsed
|
||||
}
|
||||
}
|
||||
return days
|
||||
}
|
||||
|
||||
// Compliance 返回 JSON 合规报表(按任务的备份合规证据 + 汇总)。
|
||||
func (h *ReportHandler) Compliance(c *gin.Context) {
|
||||
payload, err := h.service.ComplianceReport(c.Request.Context(), reportDays(c))
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
response.Success(c, payload)
|
||||
}
|
||||
|
||||
// ComplianceCSV 把合规报表导出为 CSV(供审计归档)。带 UTF-8 BOM 以便 Excel 正确识别中文。
|
||||
func (h *ReportHandler) ComplianceCSV(c *gin.Context) {
|
||||
report, err := h.service.ComplianceReport(c.Request.Context(), reportDays(c))
|
||||
if err != nil {
|
||||
response.Error(c, err)
|
||||
return
|
||||
}
|
||||
filename := fmt.Sprintf("backupx-compliance-%s.csv", report.GeneratedAt.Format("20060102-150405"))
|
||||
c.Header("Content-Type", "text/csv; charset=utf-8")
|
||||
c.Header("Content-Disposition", "attachment; filename="+filename)
|
||||
_, _ = c.Writer.WriteString("\ufeff") // UTF-8 BOM
|
||||
w := csv.NewWriter(c.Writer)
|
||||
_ = w.Write([]string{
|
||||
"任务ID", "任务名", "类型", "启用", "节点", "加密", "保留天数", "SLA(RPO小时)",
|
||||
"周期内运行", "成功", "失败", "成功率", "最近状态", "最近运行(UTC)", "最近成功(UTC)", "保护字节数", "合规判定",
|
||||
})
|
||||
for _, row := range report.Tasks {
|
||||
_ = w.Write([]string{
|
||||
strconv.FormatUint(uint64(row.TaskID), 10),
|
||||
row.TaskName,
|
||||
row.Type,
|
||||
boolCN(row.Enabled),
|
||||
row.NodeName,
|
||||
boolCN(row.Encrypted),
|
||||
strconv.Itoa(row.RetentionDays),
|
||||
strconv.Itoa(row.SLAHoursRPO),
|
||||
strconv.Itoa(row.TotalRuns),
|
||||
strconv.Itoa(row.Successes),
|
||||
strconv.Itoa(row.Failures),
|
||||
fmt.Sprintf("%.2f%%", row.SuccessRate*100),
|
||||
row.LastStatus,
|
||||
fmtTimePtr(row.LastRunAt),
|
||||
fmtTimePtr(row.LastSuccessAt),
|
||||
strconv.FormatInt(row.ProtectedBytes, 10),
|
||||
row.Risk,
|
||||
})
|
||||
}
|
||||
w.Flush()
|
||||
}
|
||||
|
||||
func boolCN(b bool) string {
|
||||
if b {
|
||||
return "是"
|
||||
}
|
||||
return "否"
|
||||
}
|
||||
|
||||
func fmtTimePtr(t *time.Time) string {
|
||||
if t == nil {
|
||||
return ""
|
||||
}
|
||||
return t.UTC().Format("2006-01-02 15:04:05")
|
||||
}
|
||||
@@ -41,7 +41,6 @@ type RouterDependencies struct {
|
||||
ApiKeyService *service.ApiKeyService
|
||||
NotificationService *service.NotificationService
|
||||
DashboardService *service.DashboardService
|
||||
ReportService *service.ReportService
|
||||
SettingsService *service.SettingsService
|
||||
NodeService *service.NodeService
|
||||
AgentService *service.AgentService
|
||||
@@ -167,11 +166,9 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
|
||||
backupRecords.GET("/:id", backupRecordHandler.Get)
|
||||
backupRecords.GET("/:id/logs/stream", backupRecordHandler.StreamLogs)
|
||||
backupRecords.GET("/:id/download", backupRecordHandler.Download)
|
||||
backupRecords.GET("/:id/contents", backupRecordHandler.Contents)
|
||||
backupRecords.POST("/:id/restore", RequireNotViewer(), backupRecordHandler.Restore)
|
||||
backupRecords.POST("/batch-delete", RequireNotViewer(), backupRecordHandler.BatchDelete)
|
||||
backupRecords.DELETE("/:id", RequireNotViewer(), backupRecordHandler.Delete)
|
||||
backupRecords.PUT("/:id/lock", RequireNotViewer(), backupRecordHandler.SetLock)
|
||||
|
||||
// 恢复记录独立命名空间:列表/详情/SSE 日志流。
|
||||
// 创建恢复仍然走 POST /backup/records/:id/restore(以源备份记录为触发点)。
|
||||
@@ -214,15 +211,6 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
|
||||
// 基于备份记录的验证入口:与 restore 对称
|
||||
backupRecords.POST("/:id/verify", RequireNotViewer(), verificationHandler.TriggerByRecord)
|
||||
}
|
||||
// 企业合规报表:按任务的可导出备份合规证据(区别于 Dashboard 的实时聚合视图)。
|
||||
if deps.ReportService != nil {
|
||||
reportHandler := NewReportHandler(deps.ReportService)
|
||||
reports := api.Group("/reports")
|
||||
reports.Use(AuthMiddleware(deps.JWTManager, apiKeyAuth))
|
||||
reports.GET("/compliance", reportHandler.Compliance)
|
||||
reports.GET("/compliance/export", reportHandler.ComplianceCSV)
|
||||
}
|
||||
|
||||
dashboard := api.Group("/dashboard")
|
||||
dashboard.Use(AuthMiddleware(deps.JWTManager, apiKeyAuth))
|
||||
dashboard.GET("/stats", dashboardHandler.Stats)
|
||||
@@ -304,10 +292,7 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
|
||||
nodes.POST("", RequireRole("admin"), nodeHandler.Create)
|
||||
nodes.PUT("/:id", RequireRole("admin"), nodeHandler.Update)
|
||||
nodes.DELETE("/:id", RequireRole("admin"), nodeHandler.Delete)
|
||||
// 文件浏览会枚举节点文件系统目录(含 /etc、/root 等),属敏感读操作:
|
||||
// 限制为非 viewer(admin/operator),与"创建备份任务需选源路径"的权限对齐,
|
||||
// 避免只读 viewer 借此探查服务器目录结构。
|
||||
nodes.GET("/:id/fs/list", RequireNotViewer(), nodeHandler.ListDirectory)
|
||||
nodes.GET("/:id/fs/list", nodeHandler.ListDirectory)
|
||||
nodes.POST("/batch", RequireRole("admin"), nodeHandler.BatchCreate)
|
||||
nodes.POST("/:id/install-tokens", RequireRole("admin"), nodeHandler.CreateInstallToken)
|
||||
nodes.POST("/:id/rotate-token", RequireRole("admin"), nodeHandler.RotateToken)
|
||||
@@ -322,9 +307,7 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
|
||||
agent.POST("/commands/:id/result", agentHandler.SubmitCommandResult)
|
||||
agent.GET("/tasks/:id", agentHandler.GetTaskSpec)
|
||||
agent.POST("/records/:id", agentHandler.UpdateRecord)
|
||||
agent.PUT("/records/:id/artifacts/:targetId", agentHandler.UploadArtifact)
|
||||
agent.GET("/restores/:id/spec", agentHandler.GetRestoreSpec)
|
||||
agent.GET("/restores/:id/artifact", agentHandler.DownloadRestoreArtifact)
|
||||
agent.POST("/restores/:id", agentHandler.UpdateRestore)
|
||||
|
||||
// Agent v1(安装脚本探活用),仅 Self 端点
|
||||
@@ -370,25 +353,9 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
|
||||
engine.GET("/api/install/:token/compose.yml", installHandler.Compose)
|
||||
}
|
||||
|
||||
// 未匹配路由处理:
|
||||
// - 找到前端产物目录时,托管 SPA(静态文件 + index.html 回退),
|
||||
// 使后端在无 nginx 反向代理时也能直接提供 Web 控制台(issue #62);
|
||||
// - 未找到前端目录时退化为纯 API 服务,统一返回结构化 JSON 404。
|
||||
apiNotFound := func(c *gin.Context) {
|
||||
engine.NoRoute(func(c *gin.Context) {
|
||||
response.Error(c, apperror.New(stdhttp.StatusNotFound, "NOT_FOUND", "接口不存在", errors.New("route not found")))
|
||||
}
|
||||
if webRoot := resolveWebRoot(deps.Config.Server.WebRoot); webRoot != "" {
|
||||
if deps.Logger != nil {
|
||||
deps.Logger.Info("serving web frontend", zap.String("web_root", webRoot))
|
||||
}
|
||||
engine.NoRoute(spaFileServer(webRoot, apiNotFound))
|
||||
} else {
|
||||
if deps.Logger != nil {
|
||||
deps.Logger.Warn("web frontend directory not found; serving API only",
|
||||
zap.String("hint", "set server.web_root in config, or place the built frontend at ./web"))
|
||||
}
|
||||
engine.NoRoute(apiNotFound)
|
||||
}
|
||||
})
|
||||
|
||||
return engine
|
||||
}
|
||||
|
||||
@@ -1,86 +0,0 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
stdhttp "net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// resolveWebRoot 返回前端静态资源目录。优先使用显式配置的路径,
|
||||
// 否则按部署惯例依次探测常见位置,返回首个包含 index.html 的目录。
|
||||
// 返回空字符串表示未找到前端产物,此时后端退化为纯 API 服务。
|
||||
func resolveWebRoot(configured string) string {
|
||||
candidates := []string{
|
||||
configured,
|
||||
"./web/dist", // 源码树根目录构建产物(优先于 ./web,避免命中前端源码模板)
|
||||
"./web", // systemd:WorkingDirectory=/opt/backupx → /opt/backupx/web;容器 WORKDIR=/app → /app/web
|
||||
"../web/dist", // 从 server/ 目录运行(make dev-server)
|
||||
"/opt/backupx/web",
|
||||
"/app/web",
|
||||
}
|
||||
for _, dir := range candidates {
|
||||
if strings.TrimSpace(dir) == "" {
|
||||
continue
|
||||
}
|
||||
if hasIndexHTML(dir) {
|
||||
if abs, err := filepath.Abs(dir); err == nil {
|
||||
return abs
|
||||
}
|
||||
return dir
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func hasIndexHTML(dir string) bool {
|
||||
info, err := os.Stat(filepath.Join(dir, "index.html"))
|
||||
return err == nil && !info.IsDir()
|
||||
}
|
||||
|
||||
// isReservedBackendPath 判断请求是否命中后端保留前缀(API、探针、安装脚本)。
|
||||
// 这些路径即使未匹配到具体路由,也应返回结构化 JSON 404,而不是回退到
|
||||
// 前端 index.html —— 否则反向代理/安装脚本会把 HTML 当成接口响应(参考 issue #46)。
|
||||
func isReservedBackendPath(p string) bool {
|
||||
switch p {
|
||||
case "/health", "/ready", "/metrics", "/api", "/install":
|
||||
return true
|
||||
}
|
||||
return strings.HasPrefix(p, "/api/") || strings.HasPrefix(p, "/install/")
|
||||
}
|
||||
|
||||
// spaFileServer 构造 SPA 静态资源处理器,用作 gin 的 NoRoute 回退:
|
||||
// - 后端保留前缀返回 apiNotFound(JSON 404);
|
||||
// - 其余 GET/HEAD 请求若在 webRoot 内命中真实文件则直接返回该文件;
|
||||
// - 未命中文件的路径回退到 index.html,交由前端路由处理(history 模式刷新)。
|
||||
func spaFileServer(webRoot string, apiNotFound gin.HandlerFunc) gin.HandlerFunc {
|
||||
indexPath := filepath.Join(webRoot, "index.html")
|
||||
return func(c *gin.Context) {
|
||||
reqPath := c.Request.URL.Path
|
||||
if isReservedBackendPath(reqPath) {
|
||||
apiNotFound(c)
|
||||
return
|
||||
}
|
||||
if c.Request.Method != stdhttp.MethodGet && c.Request.Method != stdhttp.MethodHead {
|
||||
apiNotFound(c)
|
||||
return
|
||||
}
|
||||
|
||||
// 防目录穿越:以 webRoot 为根清理路径,确保最终目标仍位于 webRoot 内。
|
||||
clean := filepath.Clean("/" + strings.TrimPrefix(reqPath, "/"))
|
||||
target := filepath.Join(webRoot, clean)
|
||||
if rel, err := filepath.Rel(webRoot, target); err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
apiNotFound(c)
|
||||
return
|
||||
}
|
||||
|
||||
if info, err := os.Stat(target); err == nil && !info.IsDir() {
|
||||
c.File(target)
|
||||
return
|
||||
}
|
||||
// 前端 SPA 路由(/dashboard、/tasks 等)回退到 index.html。
|
||||
c.File(indexPath)
|
||||
}
|
||||
}
|
||||
@@ -1,175 +0,0 @@
|
||||
package http
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
func TestResolveWebRoot(t *testing.T) {
|
||||
gin.SetMode(gin.TestMode)
|
||||
|
||||
t.Run("explicit configured dir with index.html", func(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeIndex(t, dir)
|
||||
got := resolveWebRoot(dir)
|
||||
abs, _ := filepath.Abs(dir)
|
||||
if got != abs {
|
||||
t.Fatalf("resolveWebRoot(%q) = %q, want %q", dir, got, abs)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("configured dir without index.html falls through to none", func(t *testing.T) {
|
||||
dir := t.TempDir() // no index.html, and no conventional ./web in CWD during test
|
||||
if got := resolveWebRoot(dir); got != "" {
|
||||
// 允许 CWD 恰好存在约定目录的环境,但临时目录本身不应被选中。
|
||||
abs, _ := filepath.Abs(dir)
|
||||
if got == abs {
|
||||
t.Fatalf("expected dir without index.html to be skipped, got %q", got)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("empty configured uses auto-detect order", func(t *testing.T) {
|
||||
// 切到一个仅含 ./web/dist/index.html 的临时工作目录,验证自动探测。
|
||||
root := t.TempDir()
|
||||
distDir := filepath.Join(root, "web", "dist")
|
||||
if err := os.MkdirAll(distDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeIndex(t, distDir)
|
||||
|
||||
restore := chdir(t, root)
|
||||
defer restore()
|
||||
|
||||
// 以 chdir 之后的实际工作目录为基准计算期望值,避免 macOS 上
|
||||
// /var → /private/var 符号链接导致字符串不一致。
|
||||
wd, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := filepath.Join(wd, "web", "dist")
|
||||
|
||||
got := resolveWebRoot("")
|
||||
if got != want {
|
||||
t.Fatalf("auto-detect = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestIsReservedBackendPath(t *testing.T) {
|
||||
reserved := []string{"/health", "/ready", "/metrics", "/api", "/install", "/api/", "/api/system/info", "/install/abc", "/install/abc/compose.yml"}
|
||||
for _, p := range reserved {
|
||||
if !isReservedBackendPath(p) {
|
||||
t.Errorf("isReservedBackendPath(%q) = false, want true", p)
|
||||
}
|
||||
}
|
||||
notReserved := []string{"/", "/dashboard", "/assets/app.js", "/installer", "/apidocs", "/favicon.ico"}
|
||||
for _, p := range notReserved {
|
||||
if isReservedBackendPath(p) {
|
||||
t.Errorf("isReservedBackendPath(%q) = true, want false", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSpaFileServer(t *testing.T) {
|
||||
gin.SetMode(gin.TestMode)
|
||||
webRoot := t.TempDir()
|
||||
writeIndexContent(t, webRoot, "<!doctype html><title>BackupX</title>")
|
||||
assetsDir := filepath.Join(webRoot, "assets")
|
||||
if err := os.MkdirAll(assetsDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(assetsDir, "app.js"), []byte("console.log(1)"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
apiNotFoundHit := false
|
||||
apiNotFound := func(c *gin.Context) {
|
||||
apiNotFoundHit = true
|
||||
c.JSON(http.StatusNotFound, gin.H{"code": "NOT_FOUND"})
|
||||
}
|
||||
|
||||
engine := gin.New()
|
||||
engine.NoRoute(spaFileServer(webRoot, apiNotFound))
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
method string
|
||||
path string
|
||||
wantStatus int
|
||||
wantBody string // 子串;为空表示不校验
|
||||
wantAPI404 bool
|
||||
}{
|
||||
{name: "root serves index", method: http.MethodGet, path: "/", wantStatus: 200, wantBody: "BackupX"},
|
||||
{name: "spa route falls back to index", method: http.MethodGet, path: "/dashboard", wantStatus: 200, wantBody: "BackupX"},
|
||||
{name: "real asset served", method: http.MethodGet, path: "/assets/app.js", wantStatus: 200, wantBody: "console.log"},
|
||||
{name: "api path returns json 404", method: http.MethodGet, path: "/api/garbage", wantStatus: 404, wantAPI404: true},
|
||||
{name: "health returns json 404 via reserved", method: http.MethodGet, path: "/health", wantStatus: 404, wantAPI404: true},
|
||||
{name: "non-GET on spa path is api 404", method: http.MethodPost, path: "/dashboard", wantStatus: 404, wantAPI404: true},
|
||||
{name: "directory falls back to index", method: http.MethodGet, path: "/assets/", wantStatus: 200, wantBody: "BackupX"},
|
||||
// 含 ".." 的请求路径被 net/http 在文件服务层直接拒绝(400 invalid URL path),
|
||||
// 绝不会泄露 webRoot 之外的文件;这是在 filepath.Rel 校验之上的纵深防御。
|
||||
{name: "traversal rejected, never serves passwd", method: http.MethodGet, path: "/../../etc/passwd", wantStatus: 400, wantBody: "invalid URL path"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
apiNotFoundHit = false
|
||||
req := httptest.NewRequest(tc.method, tc.path, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
engine.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != tc.wantStatus {
|
||||
t.Fatalf("status = %d, want %d (body=%q)", rec.Code, tc.wantStatus, rec.Body.String())
|
||||
}
|
||||
if tc.wantBody != "" && !contains(rec.Body.String(), tc.wantBody) {
|
||||
t.Fatalf("body %q does not contain %q", rec.Body.String(), tc.wantBody)
|
||||
}
|
||||
if tc.wantAPI404 != apiNotFoundHit {
|
||||
t.Fatalf("apiNotFoundHit = %v, want %v", apiNotFoundHit, tc.wantAPI404)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func writeIndex(t *testing.T, dir string) {
|
||||
t.Helper()
|
||||
writeIndexContent(t, dir, "<!doctype html><title>BackupX</title>")
|
||||
}
|
||||
|
||||
func writeIndexContent(t *testing.T, dir, content string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, "index.html"), []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func chdir(t *testing.T, dir string) func() {
|
||||
t.Helper()
|
||||
orig, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Chdir(dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return func() { _ = os.Chdir(orig) }
|
||||
}
|
||||
|
||||
func contains(s, sub string) bool {
|
||||
return len(sub) == 0 || (len(s) >= len(sub) && indexOf(s, sub) >= 0)
|
||||
}
|
||||
|
||||
func indexOf(s, sub string) int {
|
||||
for i := 0; i+len(sub) <= len(s); i++ {
|
||||
if s[i:i+len(sub)] == sub {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
@@ -199,7 +199,7 @@ func (h *StorageTargetHandler) GoogleDriveProfile(c *gin.Context) {
|
||||
|
||||
func parseUintParam(c *gin.Context, key string) (uint, bool) {
|
||||
value := strings.TrimSpace(c.Param(key))
|
||||
parsed, err := strconv.ParseUint(value, 10, 0)
|
||||
parsed, err := strconv.ParseUint(value, 10, 64)
|
||||
if err != nil {
|
||||
response.Error(c, apperror.BadRequest("INVALID_ID", fmt.Sprintf("参数 %s 不合法", key), err))
|
||||
return 0, false
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
package installscript
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDeployInstallScriptSyntax(t *testing.T) {
|
||||
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
|
||||
cmd := exec.Command("sh", "-n", scriptPath)
|
||||
output, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("install.sh syntax invalid: %v\n%s", err, output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeployInstallScriptSupportsReleasePackageLayout(t *testing.T) {
|
||||
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
|
||||
data, err := os.ReadFile(scriptPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
script := string(data)
|
||||
for _, want := range []string{
|
||||
`SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)`,
|
||||
`if [ -f "$SCRIPT_DIR/backupx" ] && [ -d "$SCRIPT_DIR/web" ]; then`,
|
||||
`BIN_SOURCE="${BIN_SOURCE:-$SCRIPT_DIR/backupx}"`,
|
||||
`WEB_SOURCE="${WEB_SOURCE:-$SCRIPT_DIR/web}"`,
|
||||
`CONFIG_TEMPLATE="${CONFIG_TEMPLATE:-$SCRIPT_DIR/config.example.yaml}"`,
|
||||
`发布包安装请确认当前目录包含 ./backupx、./web 和 ./install.sh。`,
|
||||
`cat > "/etc/systemd/system/$SERVICE_NAME.service" <<UNIT`,
|
||||
`if [ -d "/etc/nginx/conf.d" ] && [ -f "$NGINX_SOURCE" ]; then`,
|
||||
} {
|
||||
if !strings.Contains(script, want) {
|
||||
t.Fatalf("install.sh missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeployInstallScriptSupportsSourceBuildAndVerifiesFirstSetup(t *testing.T) {
|
||||
scriptPath := filepath.Join("..", "..", "..", "deploy", "install.sh")
|
||||
data, err := os.ReadFile(scriptPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
script := string(data)
|
||||
for _, want := range []string{
|
||||
`SOURCE_BIN_DEFAULT="$PROJECT_ROOT/server/bin/backupx"`,
|
||||
`For a source install, run 'make build' in the repository root first.`,
|
||||
`HEALTH_URL="${HEALTH_URL:-http://127.0.0.1:8340/api/auth/setup/status}"`,
|
||||
`systemctl is-active --quiet "$SERVICE_NAME"`,
|
||||
`System setup`,
|
||||
} {
|
||||
if !strings.Contains(script, want) {
|
||||
t.Fatalf("install.sh missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -37,22 +37,19 @@ func TestRenderScriptBashBootstrap(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderScriptUsesRootForBareMetalBackups(t *testing.T) {
|
||||
func TestRenderScriptCreatesBackupXUserAndGroup(t *testing.T) {
|
||||
got, err := RenderScript(testCtx)
|
||||
if err != nil {
|
||||
t.Fatalf("render err: %v", err)
|
||||
}
|
||||
for _, want := range []string{
|
||||
"/var/lib/backupx-agent/tmp",
|
||||
"install -d -m 0700 /var/lib/backupx-agent /var/lib/backupx-agent/tmp",
|
||||
"getent group backupx",
|
||||
"groupadd --system backupx",
|
||||
"useradd --system --gid backupx",
|
||||
"Group=backupx",
|
||||
} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("script missing %q:\n%s", want, got)
|
||||
}
|
||||
}
|
||||
for _, forbidden := range []string{"User=backupx", "Group=backupx", "NoNewPrivileges=true"} {
|
||||
if strings.Contains(got, forbidden) {
|
||||
t.Errorf("script should not contain %q for bare-metal backups:\n%s", forbidden, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
package installscript
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -29,10 +27,8 @@ func TestRenderScriptSystemd(t *testing.T) {
|
||||
mustContain := []string{
|
||||
"BACKUPX_AGENT_MASTER=${MASTER_URL}",
|
||||
`Environment="BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}"`,
|
||||
"/var/lib/backupx-agent/tmp",
|
||||
"systemctl daemon-reload",
|
||||
"systemctl enable --now backupx-agent",
|
||||
"systemctl status backupx-agent",
|
||||
"X-Agent-Token: ${AGENT_TOKEN}",
|
||||
"MASTER_URL=\"https://master.example.com\"",
|
||||
"AGENT_TOKEN=\"deadbeefcafebabe0123456789abcdef0123456789abcdef0123456789abcdef\"",
|
||||
@@ -60,9 +56,6 @@ func TestRenderScriptForeground(t *testing.T) {
|
||||
if !strings.Contains(got, `exec "${INSTALL_PREFIX}/backupx" agent`) {
|
||||
t.Errorf("foreground script missing exec line:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "/var/lib/backupx-agent/tmp") {
|
||||
t.Errorf("foreground script missing dedicated temp dir:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "systemctl daemon-reload") {
|
||||
t.Errorf("foreground script should not reference systemctl:\n%s", got)
|
||||
}
|
||||
@@ -81,44 +74,14 @@ func TestRenderScriptDocker(t *testing.T) {
|
||||
if !strings.Contains(got, "docker run") {
|
||||
t.Errorf("docker script missing `docker run`:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "/var/lib/backupx-agent:/var/lib/backupx-agent") {
|
||||
t.Errorf("docker script missing agent data volume:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "awuqing/backupx:${AGENT_VERSION}") {
|
||||
t.Errorf("docker script missing image tag reference:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `"awuqing/backupx:${AGENT_VERSION}" agent`) {
|
||||
t.Errorf("docker script must start image in agent mode:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `-e "BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp"`) {
|
||||
t.Errorf("docker script missing temp dir env:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `docker logs --tail=100 backupx-agent`) {
|
||||
t.Errorf("docker script missing diagnostic log command:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `grep -q '"status":"online"'`) {
|
||||
t.Errorf("docker script missing online probe:\n%s", got)
|
||||
}
|
||||
if strings.Contains(got, "systemctl daemon-reload") {
|
||||
t.Errorf("docker script should not reference systemctl:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDockerEntrypointForwardsAgentSubcommand(t *testing.T) {
|
||||
entrypointPath := filepath.Join("..", "..", "..", "deploy", "docker", "entrypoint.sh")
|
||||
got, err := os.ReadFile(entrypointPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read docker entrypoint: %v", err)
|
||||
}
|
||||
script := string(got)
|
||||
if !strings.Contains(script, `"${1:-}" = "agent"`) {
|
||||
t.Fatalf("entrypoint must detect the agent subcommand before starting server:\n%s", script)
|
||||
}
|
||||
if !strings.Contains(script, `exec /app/bin/backupx "$@"`) {
|
||||
t.Fatalf("entrypoint must exec backupx with forwarded args:\n%s", script)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderComposeYaml(t *testing.T) {
|
||||
ctx := testCtx
|
||||
ctx.Mode = model.InstallModeDocker
|
||||
@@ -129,26 +92,17 @@ func TestRenderComposeYaml(t *testing.T) {
|
||||
if !strings.Contains(got, "image: awuqing/backupx:v1.7.0") {
|
||||
t.Errorf("compose missing image:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `command: ["agent"]`) {
|
||||
t.Errorf("compose must start image in agent mode:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `BACKUPX_AGENT_TOKEN: "deadbeefcafebabe0123456789abcdef0123456789abcdef0123456789abcdef"`) {
|
||||
t.Errorf("compose missing token env:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, `BACKUPX_AGENT_TEMP_DIR: "/var/lib/backupx-agent/tmp"`) {
|
||||
t.Errorf("compose missing temp dir env:\n%s", got)
|
||||
}
|
||||
if !strings.Contains(got, "/var/lib/backupx-agent:/var/lib/backupx-agent") {
|
||||
t.Errorf("compose missing agent data volume:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderScriptRejectsInjectedMasterURL(t *testing.T) {
|
||||
bad := []string{
|
||||
"https://example.com\" other: inject", // 含引号和空格
|
||||
"javascript:alert(1)", // scheme 非法
|
||||
"https://example.com\n- privileged", // 含换行,YAML 注入经典 payload
|
||||
"", // 空
|
||||
"javascript:alert(1)", // scheme 非法
|
||||
"https://example.com\n- privileged", // 含换行,YAML 注入经典 payload
|
||||
"", // 空
|
||||
}
|
||||
for _, u := range bad {
|
||||
ctx := testCtx
|
||||
@@ -207,8 +161,8 @@ func TestDownloadBaseMapping(t *testing.T) {
|
||||
|
||||
func TestRenderScriptDefaultsApplied(t *testing.T) {
|
||||
ctx := testCtx
|
||||
ctx.InstallPrefix = "" // 应被默认为 /opt/backupx-agent
|
||||
ctx.DownloadBase = "" // 应被默认为 github
|
||||
ctx.InstallPrefix = "" // 应被默认为 /opt/backupx-agent
|
||||
ctx.DownloadBase = "" // 应被默认为 github
|
||||
got, err := RenderScript(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("render err: %v", err)
|
||||
|
||||
@@ -9,6 +9,5 @@ services:
|
||||
environment:
|
||||
BACKUPX_AGENT_MASTER: "{{.MasterURL}}"
|
||||
BACKUPX_AGENT_TOKEN: "{{.AgentToken}}"
|
||||
BACKUPX_AGENT_TEMP_DIR: "/var/lib/backupx-agent/tmp"
|
||||
volumes:
|
||||
- /var/lib/backupx-agent:/var/lib/backupx-agent
|
||||
- /var/lib/backupx-agent:/tmp/backupx-agent
|
||||
|
||||
@@ -47,10 +47,30 @@ else
|
||||
fi
|
||||
tar xzf "$TMPDIR/pkg.tar.gz" -C "$TMPDIR"
|
||||
|
||||
# 4. 安装二进制 + 数据目录
|
||||
# 4. 安装二进制 + 用户
|
||||
echo "[2/4] 安装到 ${INSTALL_PREFIX}"
|
||||
install -d -m 0755 "$INSTALL_PREFIX"
|
||||
install -d -m 0700 /var/lib/backupx-agent /var/lib/backupx-agent/tmp
|
||||
if ! getent group backupx >/dev/null 2>&1; then
|
||||
if command -v groupadd >/dev/null 2>&1; then
|
||||
groupadd --system backupx
|
||||
elif command -v addgroup >/dev/null 2>&1; then
|
||||
addgroup --system backupx
|
||||
else
|
||||
echo "需要 groupadd 或 addgroup 来创建 backupx 组" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
if ! id backupx >/dev/null 2>&1; then
|
||||
if command -v useradd >/dev/null 2>&1; then
|
||||
useradd --system --gid backupx --home-dir "$INSTALL_PREFIX" --shell /usr/sbin/nologin backupx
|
||||
elif command -v adduser >/dev/null 2>&1; then
|
||||
adduser --system --ingroup backupx --home "$INSTALL_PREFIX" --shell /usr/sbin/nologin backupx
|
||||
else
|
||||
echo "需要 useradd 或 adduser 来创建 backupx 用户" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
id backupx >/dev/null 2>&1 || { echo "backupx 用户创建失败" >&2; exit 1; }
|
||||
install -d -o backupx -g backupx "$INSTALL_PREFIX" /var/lib/backupx-agent
|
||||
install -m 0755 "$TMPDIR/backupx-${AGENT_VERSION}-linux-${ARCH}/backupx" "$INSTALL_PREFIX/backupx"
|
||||
{{end}}
|
||||
|
||||
@@ -65,13 +85,14 @@ Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=backupx
|
||||
Group=backupx
|
||||
Environment="BACKUPX_AGENT_MASTER=${MASTER_URL}"
|
||||
Environment="BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}"
|
||||
ExecStart=${INSTALL_PREFIX}/backupx agent --temp-dir /var/lib/backupx-agent/tmp
|
||||
ExecStart=${INSTALL_PREFIX}/backupx agent --temp-dir /var/lib/backupx-agent
|
||||
Restart=on-failure
|
||||
RestartSec=10s
|
||||
# Agent 需以 root 运行以读取任意源数据;与单机服务端保持一致的资源/句柄上限。
|
||||
LimitNOFILE=65535
|
||||
NoNewPrivileges=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -90,7 +111,6 @@ for i in $(seq 1 15); do
|
||||
fi
|
||||
done
|
||||
echo "⚠ 30s 内未收到上线心跳,请检查防火墙或 journalctl -u backupx-agent"
|
||||
echo "提示:systemd 服务名是 backupx-agent,可执行 systemctl status backupx-agent 查看状态。"
|
||||
exit 2
|
||||
{{end}}
|
||||
|
||||
@@ -99,7 +119,7 @@ exit 2
|
||||
echo "[3/3] 前台启动 agent(Ctrl+C 退出)"
|
||||
export BACKUPX_AGENT_MASTER="${MASTER_URL}"
|
||||
export BACKUPX_AGENT_TOKEN="${AGENT_TOKEN}"
|
||||
exec "${INSTALL_PREFIX}/backupx" agent --temp-dir /var/lib/backupx-agent/tmp
|
||||
exec "${INSTALL_PREFIX}/backupx" agent --temp-dir /var/lib/backupx-agent
|
||||
{{end}}
|
||||
|
||||
{{if eq .Mode "docker"}}
|
||||
@@ -111,20 +131,7 @@ docker rm -f backupx-agent >/dev/null 2>&1 || true
|
||||
docker run -d --name backupx-agent --restart=unless-stopped \
|
||||
-e "BACKUPX_AGENT_MASTER=${MASTER_URL}" \
|
||||
-e "BACKUPX_AGENT_TOKEN=${AGENT_TOKEN}" \
|
||||
-e "BACKUPX_AGENT_TEMP_DIR=/var/lib/backupx-agent/tmp" \
|
||||
-v /var/lib/backupx-agent:/var/lib/backupx-agent \
|
||||
-v /var/lib/backupx-agent:/tmp/backupx-agent \
|
||||
"awuqing/backupx:${AGENT_VERSION}" agent
|
||||
echo "✓ 容器已启动,等待节点上线"
|
||||
for i in $(seq 1 15); do
|
||||
sleep 2
|
||||
if curl -fsSL -H "X-Agent-Token: ${AGENT_TOKEN}" "${MASTER_URL}/api/v1/agent/self" 2>/dev/null \
|
||||
| grep -q '"status":"online"'; then
|
||||
echo "✓ 节点已上线"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
echo "⚠ 30s 内未收到上线心跳,请检查容器状态、网络与 Master URL。"
|
||||
echo "排查命令:docker ps -a --filter name=backupx-agent"
|
||||
echo "排查命令:docker logs --tail=100 backupx-agent"
|
||||
exit 2
|
||||
echo "✓ 容器已启动"
|
||||
{{end}}
|
||||
|
||||
@@ -13,18 +13,16 @@ type SampleSource interface {
|
||||
ListStorageTargets(ctx context.Context) ([]model.StorageTarget, error)
|
||||
StorageUsage(ctx context.Context) ([]repository.BackupStorageUsageItem, error)
|
||||
ListNodes(ctx context.Context) ([]model.Node, error)
|
||||
AgentQueueSummaries(ctx context.Context) (map[uint]repository.AgentCommandQueueSummary, error)
|
||||
CountSLABreach(ctx context.Context) (int, error)
|
||||
}
|
||||
|
||||
// repoSource 把 repository 适配到 SampleSource。
|
||||
type repoSource struct {
|
||||
targets repository.StorageTargetRepository
|
||||
records repository.BackupRecordRepository
|
||||
nodes repository.NodeRepository
|
||||
tasks repository.BackupTaskRepository
|
||||
commands repository.AgentCommandRepository
|
||||
now func() time.Time
|
||||
targets repository.StorageTargetRepository
|
||||
records repository.BackupRecordRepository
|
||||
nodes repository.NodeRepository
|
||||
tasks repository.BackupTaskRepository
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// NewRepoSource 用仓储实例构造 SampleSource。
|
||||
@@ -33,15 +31,13 @@ func NewRepoSource(
|
||||
records repository.BackupRecordRepository,
|
||||
nodes repository.NodeRepository,
|
||||
tasks repository.BackupTaskRepository,
|
||||
commands repository.AgentCommandRepository,
|
||||
) SampleSource {
|
||||
return &repoSource{
|
||||
targets: targets,
|
||||
records: records,
|
||||
nodes: nodes,
|
||||
tasks: tasks,
|
||||
commands: commands,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
targets: targets,
|
||||
records: records,
|
||||
nodes: nodes,
|
||||
tasks: tasks,
|
||||
now: func() time.Time { return time.Now().UTC() },
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,13 +53,6 @@ func (s *repoSource) ListNodes(ctx context.Context) ([]model.Node, error) {
|
||||
return s.nodes.List(ctx)
|
||||
}
|
||||
|
||||
func (s *repoSource) AgentQueueSummaries(ctx context.Context) (map[uint]repository.AgentCommandQueueSummary, error) {
|
||||
if s.commands == nil {
|
||||
return nil, nil
|
||||
}
|
||||
return s.commands.NodeQueueSummaries(ctx)
|
||||
}
|
||||
|
||||
// CountSLABreach 统计当前违反 RPO 的任务:
|
||||
// - 任务启用且配置了 SLAHoursRPO > 0
|
||||
// - 最近一次成功备份距今超出 SLA 时间窗,或从未成功过
|
||||
@@ -147,9 +136,7 @@ func (c *Collector) collect(ctx context.Context) {
|
||||
}
|
||||
// 节点在线状态:role 约定为 master / agent
|
||||
if nodes, err := c.source.ListNodes(ctx); err == nil {
|
||||
queueByNode, _ := c.source.AgentQueueSummaries(ctx)
|
||||
c.metrics.ResetNodeOnline()
|
||||
c.metrics.ResetAgentQueue()
|
||||
for i := range nodes {
|
||||
n := &nodes[i]
|
||||
role := "agent"
|
||||
@@ -157,8 +144,6 @@ func (c *Collector) collect(ctx context.Context) {
|
||||
role = "master"
|
||||
}
|
||||
c.metrics.SetNodeOnline(n.Name, role, n.Status == model.NodeStatusOnline)
|
||||
queue := queueByNode[n.ID]
|
||||
c.metrics.SetAgentQueue(n.Name, role, queue.Depth, queue.Running, queue.Timeouts)
|
||||
}
|
||||
}
|
||||
if breach, err := c.source.CountSLABreach(ctx); err == nil {
|
||||
|
||||
@@ -31,12 +31,6 @@ type Metrics struct {
|
||||
StorageUsedBytes *prometheus.GaugeVec
|
||||
// 节点在线状态(labels: node_name, role;value: 0/1)
|
||||
NodeOnline *prometheus.GaugeVec
|
||||
// Agent 命令队列深度(labels: node_name, role)
|
||||
AgentCommandQueueDepth *prometheus.GaugeVec
|
||||
// Agent 正在执行的长命令数(labels: node_name, role)
|
||||
AgentCommandRunning *prometheus.GaugeVec
|
||||
// Agent 命令超时累计数快照(labels: node_name, role)
|
||||
AgentCommandTimeoutTotal *prometheus.GaugeVec
|
||||
// 验证演练结果(labels: status)
|
||||
VerifyRunTotal *prometheus.CounterVec
|
||||
// 恢复操作结果(labels: status)
|
||||
@@ -84,18 +78,6 @@ func New(version string) *Metrics {
|
||||
Name: "backupx_node_online",
|
||||
Help: "集群节点在线状态(1 在线 / 0 离线)",
|
||||
}, []string{"node_name", "role"}),
|
||||
AgentCommandQueueDepth: prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Name: "backupx_agent_command_queue_depth",
|
||||
Help: "Agent 当前 pending/dispatched 命令总数",
|
||||
}, []string{"node_name", "role"}),
|
||||
AgentCommandRunning: prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Name: "backupx_agent_command_running",
|
||||
Help: "Agent 当前正在执行的长命令数",
|
||||
}, []string{"node_name", "role"}),
|
||||
AgentCommandTimeoutTotal: prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Name: "backupx_agent_command_timeout_total",
|
||||
Help: "Agent 已超时命令数快照",
|
||||
}, []string{"node_name", "role"}),
|
||||
VerifyRunTotal: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: "backupx_verify_run_total",
|
||||
Help: "备份验证演练执行总数",
|
||||
@@ -124,9 +106,6 @@ func New(version string) *Metrics {
|
||||
m.TaskRunningGauge,
|
||||
m.StorageUsedBytes,
|
||||
m.NodeOnline,
|
||||
m.AgentCommandQueueDepth,
|
||||
m.AgentCommandRunning,
|
||||
m.AgentCommandTimeoutTotal,
|
||||
m.VerifyRunTotal,
|
||||
m.RestoreRunTotal,
|
||||
m.ReplicationRunTotal,
|
||||
@@ -229,24 +208,6 @@ func (m *Metrics) ResetNodeOnline() {
|
||||
m.NodeOnline.Reset()
|
||||
}
|
||||
|
||||
func (m *Metrics) SetAgentQueue(name, role string, depth, running, timeoutCount int) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
m.AgentCommandQueueDepth.WithLabelValues(name, role).Set(float64(depth))
|
||||
m.AgentCommandRunning.WithLabelValues(name, role).Set(float64(running))
|
||||
m.AgentCommandTimeoutTotal.WithLabelValues(name, role).Set(float64(timeoutCount))
|
||||
}
|
||||
|
||||
func (m *Metrics) ResetAgentQueue() {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
m.AgentCommandQueueDepth.Reset()
|
||||
m.AgentCommandRunning.Reset()
|
||||
m.AgentCommandTimeoutTotal.Reset()
|
||||
}
|
||||
|
||||
// ResetStorageUsed 清空存储目标 gauge。
|
||||
func (m *Metrics) ResetStorageUsed() {
|
||||
if m == nil {
|
||||
|
||||
@@ -41,11 +41,9 @@ func TestObserveTaskRun_NilReceiverIsSafe(t *testing.T) {
|
||||
m.DecTaskRunning()
|
||||
m.SetStorageUsed("a", "s3", 1)
|
||||
m.SetNodeOnline("n1", "master", true)
|
||||
m.SetAgentQueue("n1", "agent", 2, 1, 3)
|
||||
m.SetSLABreach(3)
|
||||
m.ResetNodeOnline()
|
||||
m.ResetStorageUsed()
|
||||
m.ResetAgentQueue()
|
||||
// no panic -> pass
|
||||
}
|
||||
|
||||
@@ -53,7 +51,6 @@ func TestHandler_ExposesBackupxMetrics(t *testing.T) {
|
||||
m := New("0.0.0-test")
|
||||
m.ObserveTaskRun("file", "success", 1.0, 2048)
|
||||
m.SetNodeOnline("n1", "master", true)
|
||||
m.SetAgentQueue("edge-a", "agent", 3, 1, 2)
|
||||
m.SetSLABreach(1)
|
||||
|
||||
recorder := httptest.NewRecorder()
|
||||
@@ -69,9 +66,6 @@ func TestHandler_ExposesBackupxMetrics(t *testing.T) {
|
||||
"backupx_task_run_total",
|
||||
"backupx_task_run_duration_seconds",
|
||||
"backupx_node_online",
|
||||
"backupx_agent_command_queue_depth",
|
||||
"backupx_agent_command_running",
|
||||
"backupx_agent_command_timeout_total",
|
||||
"backupx_sla_breach_tasks",
|
||||
"backupx_app_info",
|
||||
} {
|
||||
|
||||
@@ -8,47 +8,28 @@ const (
|
||||
BackupRecordStatusFailed = "failed"
|
||||
)
|
||||
|
||||
const (
|
||||
// BackupKindFull 全量归档;BackupKindDifferential 差异归档;
|
||||
// BackupKindRepository 为可独立恢复的 CDC 内容寻址快照。
|
||||
BackupKindFull = "full"
|
||||
BackupKindDifferential = "differential"
|
||||
BackupKindRepository = "repository"
|
||||
)
|
||||
|
||||
type BackupRecord struct {
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
|
||||
Task BackupTask `json:"task,omitempty"`
|
||||
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
|
||||
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
|
||||
// NodeID 执行该次备份的节点(0 = 本机 Master)。StorageTransferMode 进一步
|
||||
// 区分远程 Agent 直写与 Master 中转,避免在错误节点访问 local_disk。
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
Status string `gorm:"size:20;index;not null" json:"status"`
|
||||
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
|
||||
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
|
||||
Checksum string `gorm:"column:checksum;size:64" json:"checksum"`
|
||||
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
|
||||
// 空值表示旧版 Agent 直写;direct / master_relay 记录新协议的实际数据路径。
|
||||
StorageTransferMode string `gorm:"column:storage_transfer_mode;size:20" json:"storageTransferMode,omitempty"`
|
||||
StorageUploadResults string `gorm:"column:storage_upload_results;type:text" json:"-"`
|
||||
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
|
||||
// Locked 保留锁定(法律保留):为 true 时该备份不参与保留期/数量自动清理,
|
||||
// 且禁止手动删除,直到显式解锁。用于保护合规快照、迁移前基线等关键备份。
|
||||
Locked bool `gorm:"column:locked;not null;default:false;index" json:"locked"`
|
||||
// BackupKind 备份类型:full(全量)/ differential(差异)/ repository(CDC 快照)。
|
||||
BackupKind string `gorm:"column:backup_kind;size:16;not null;default:'full';index" json:"backupKind"`
|
||||
// BaseRecordID 差异备份所基于的全量备份记录 ID(全量记录为 0)。
|
||||
BaseRecordID uint `gorm:"column:base_record_id;index;not null;default:0" json:"baseRecordId"`
|
||||
// Manifest 全量备份的条目清单(JSON),供后续差异备份比对;差异记录为空。
|
||||
Manifest string `gorm:"column:manifest;type:text" json:"-"`
|
||||
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
|
||||
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
|
||||
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
|
||||
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
|
||||
Task BackupTask `json:"task,omitempty"`
|
||||
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
|
||||
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
|
||||
// NodeID 执行该次备份的节点(0 = 本机 Master)。用于集群中识别 local_disk 类型
|
||||
// 存储的归属节点,避免 Master 端试图跨节点访问远程 Agent 的本地存储。
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
Status string `gorm:"size:20;index;not null" json:"status"`
|
||||
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
|
||||
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
|
||||
Checksum string `gorm:"column:checksum;size:64" json:"checksum"`
|
||||
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
|
||||
StorageUploadResults string `gorm:"column:storage_upload_results;type:text" json:"-"`
|
||||
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
|
||||
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
|
||||
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
|
||||
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
|
||||
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
func (BackupRecord) TableName() string {
|
||||
|
||||
@@ -8,15 +8,6 @@ const (
|
||||
BackupTaskTypeSQLite = "sqlite"
|
||||
BackupTaskTypePostgreSQL = "postgresql"
|
||||
BackupTaskTypeSAPHANA = "saphana"
|
||||
BackupTaskTypeMongoDB = "mongodb"
|
||||
)
|
||||
|
||||
const (
|
||||
// BackupModeFull 全量模式(默认);BackupModeDifferential 差异归档;
|
||||
// BackupModeRepository 为 CDC 内容寻址仓库模式(仅文件类型本机任务)。
|
||||
BackupModeFull = "full"
|
||||
BackupModeDifferential = "differential"
|
||||
BackupModeRepository = "repository"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -27,54 +18,42 @@ const (
|
||||
)
|
||||
|
||||
type BackupTask struct {
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
|
||||
Type string `gorm:"size:20;index;not null" json:"type"`
|
||||
Enabled bool `gorm:"not null;default:true" json:"enabled"`
|
||||
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
|
||||
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
|
||||
SourcePaths string `gorm:"column:source_paths;type:text" json:"sourcePaths"`
|
||||
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
|
||||
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
|
||||
DBPort int `gorm:"column:db_port" json:"dbPort"`
|
||||
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
|
||||
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
|
||||
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
|
||||
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
|
||||
Type string `gorm:"size:20;index;not null" json:"type"`
|
||||
Enabled bool `gorm:"not null;default:true" json:"enabled"`
|
||||
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
|
||||
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
|
||||
SourcePaths string `gorm:"column:source_paths;type:text" json:"sourcePaths"`
|
||||
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
|
||||
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
|
||||
DBPort int `gorm:"column:db_port" json:"dbPort"`
|
||||
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
|
||||
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
|
||||
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
|
||||
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
|
||||
// ExtraConfig 类型特有的扩展配置(JSON),如 SAP HANA 的 backupLevel / backupChannels 等
|
||||
ExtraConfig string `gorm:"column:extra_config;type:text" json:"extraConfig"`
|
||||
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"` // deprecated: 保留兼容
|
||||
StorageTarget StorageTarget `json:"storageTarget,omitempty"` // deprecated: 保留兼容
|
||||
StorageTargets []StorageTarget `gorm:"many2many:backup_task_storage_targets" json:"storageTargets,omitempty"`
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
Node Node `json:"node,omitempty"`
|
||||
ExtraConfig string `gorm:"column:extra_config;type:text" json:"extraConfig"`
|
||||
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"` // deprecated: 保留兼容
|
||||
StorageTarget StorageTarget `json:"storageTarget,omitempty"` // deprecated: 保留兼容
|
||||
StorageTargets []StorageTarget `gorm:"many2many:backup_task_storage_targets" json:"storageTargets,omitempty"`
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
Node Node `json:"node,omitempty"`
|
||||
// NodePoolTag 节点池标签(可选)。非空且 NodeID=0 时,调度器会从 Node.Labels 包含该 tag
|
||||
// 的在线节点中动态挑选一台执行(按运行中任务数最少原则),失败会 best-effort 切换到下一个候选。
|
||||
// 典型场景:NodePoolTag="db" 让 MySQL 备份任务在任意标有 "db" 的数据库节点执行。
|
||||
NodePoolTag string `gorm:"column:node_pool_tag;size:64;index" json:"nodePoolTag"`
|
||||
Tags string `gorm:"column:tags;size:500" json:"tags"`
|
||||
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
|
||||
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
|
||||
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
|
||||
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
|
||||
// BackupMode 备份模式:full(全量,默认)/ differential(差异归档)/
|
||||
// repository(FastCDC 分块、全局去重快照)。后两者仅支持本机文件任务。
|
||||
BackupMode string `gorm:"column:backup_mode;size:16;not null;default:'full'" json:"backupMode"`
|
||||
// DiffFullIntervalDays 差异模式下强制全量的间隔(天):最近全量超过该天数则本次自动改为全量,
|
||||
// 限制差异链跨度与单个差异体积。默认 7。
|
||||
DiffFullIntervalDays int `gorm:"column:diff_full_interval_days;not null;default:7" json:"diffFullIntervalDays"`
|
||||
// GFS(祖父-父-子)保留:分别保留最近 N 天 / M 周 / K 月 / Y 年的代表性备份(每周期保留最新一份)。
|
||||
// 任一 > 0 即启用 GFS,取代 RetentionDays/MaxBackups 简单策略;全为 0 时维持简单策略(向后兼容)。
|
||||
KeepDaily int `gorm:"column:keep_daily;not null;default:0" json:"keepDaily"`
|
||||
KeepWeekly int `gorm:"column:keep_weekly;not null;default:0" json:"keepWeekly"`
|
||||
KeepMonthly int `gorm:"column:keep_monthly;not null;default:0" json:"keepMonthly"`
|
||||
KeepYearly int `gorm:"column:keep_yearly;not null;default:0" json:"keepYearly"`
|
||||
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
|
||||
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
|
||||
NodePoolTag string `gorm:"column:node_pool_tag;size:64;index" json:"nodePoolTag"`
|
||||
Tags string `gorm:"column:tags;size:500" json:"tags"`
|
||||
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
|
||||
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
|
||||
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
|
||||
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
|
||||
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
|
||||
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
|
||||
// 验证(恢复演练)配置 — 定期自动校验备份可恢复性
|
||||
VerifyEnabled bool `gorm:"column:verify_enabled;not null;default:false" json:"verifyEnabled"`
|
||||
VerifyCronExpr string `gorm:"column:verify_cron_expr;size:64" json:"verifyCronExpr"`
|
||||
VerifyMode string `gorm:"column:verify_mode;size:20;not null;default:'quick'" json:"verifyMode"`
|
||||
VerifyEnabled bool `gorm:"column:verify_enabled;not null;default:false" json:"verifyEnabled"`
|
||||
VerifyCronExpr string `gorm:"column:verify_cron_expr;size:64" json:"verifyCronExpr"`
|
||||
VerifyMode string `gorm:"column:verify_mode;size:20;not null;default:'quick'" json:"verifyMode"`
|
||||
// SLA 配置 — RPO(期望最长未备份间隔)与告警阈值
|
||||
SLAHoursRPO int `gorm:"column:sla_hours_rpo;not null;default:0" json:"slaHoursRpo"`
|
||||
AlertOnConsecutiveFails int `gorm:"column:alert_on_consecutive_fails;not null;default:1" json:"alertOnConsecutiveFails"`
|
||||
@@ -89,9 +68,9 @@ type BackupTask struct {
|
||||
// 语义:上游任务成功后自动触发本任务,形成工作流(如 DB 备份完成 → 归档压缩)。
|
||||
// 调度器继续按本任务自己的 cron 触发,仅"自动触发"路径响应依赖完成事件。
|
||||
// 循环依赖检查在 service 层完成,避免配置阶段即出错。
|
||||
DependsOnTaskIDs string `gorm:"column:depends_on_task_ids;size:500" json:"dependsOnTaskIds"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
DependsOnTaskIDs string `gorm:"column:depends_on_task_ids;size:500" json:"dependsOnTaskIds"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
func (BackupTask) TableName() string {
|
||||
|
||||
@@ -10,46 +10,22 @@ const (
|
||||
NodeStatusOffline = "offline"
|
||||
)
|
||||
|
||||
// OfflineGracePeriod 节点心跳超时判定阈值:超过该时长未心跳的远程节点视为离线。
|
||||
// Agent 默认 15s 心跳一次,预留 3 次重试空间。
|
||||
const OfflineGracePeriod = 45 * time.Second
|
||||
|
||||
// EffectiveStatus 返回节点的「实时」在线状态。
|
||||
//
|
||||
// 存储字段 Status 由心跳置 online、由后台离线监控置 offline,二者之间存在最长一个
|
||||
// 监控周期的滞后窗口;期间 List/Get/调度器可能读到过期的 "online",进而把任务下发
|
||||
// 给一台刚刚失联的节点。本方法直接以 LastSeen 推导:远程节点若超过 OfflineGracePeriod
|
||||
// 未心跳即视为 offline,消除该滞后导致的误判。本机节点恒以存储状态为准(它就是 Master
|
||||
// 自身,不依赖心跳)。
|
||||
func (n *Node) EffectiveStatus(now time.Time) string {
|
||||
if n == nil {
|
||||
return NodeStatusOffline
|
||||
}
|
||||
if n.IsLocal {
|
||||
return n.Status
|
||||
}
|
||||
if now.Sub(n.LastSeen) > OfflineGracePeriod {
|
||||
return NodeStatusOffline
|
||||
}
|
||||
return n.Status
|
||||
}
|
||||
|
||||
// Node represents a managed server node in the cluster.
|
||||
// The default "local" node is auto-created for single-machine backward compatibility.
|
||||
type Node struct {
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
Name string `gorm:"size:128;uniqueIndex;not null" json:"name"`
|
||||
Hostname string `gorm:"size:255" json:"hostname"`
|
||||
IPAddress string `gorm:"column:ip_address;size:64" json:"ipAddress"`
|
||||
Token string `gorm:"size:128;uniqueIndex;not null" json:"-"`
|
||||
Status string `gorm:"size:20;not null;default:'offline'" json:"status"`
|
||||
IsLocal bool `gorm:"not null;default:false" json:"isLocal"`
|
||||
OS string `gorm:"size:64" json:"os"`
|
||||
Arch string `gorm:"size:32" json:"arch"`
|
||||
AgentVer string `gorm:"column:agent_version;size:32" json:"agentVersion"`
|
||||
LastSeen time.Time `gorm:"column:last_seen" json:"lastSeen"`
|
||||
PrevToken string `gorm:"size:128;index" json:"-"`
|
||||
PrevTokenExpires *time.Time `gorm:"column:prev_token_expires" json:"-"`
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
Name string `gorm:"size:128;uniqueIndex;not null" json:"name"`
|
||||
Hostname string `gorm:"size:255" json:"hostname"`
|
||||
IPAddress string `gorm:"column:ip_address;size:64" json:"ipAddress"`
|
||||
Token string `gorm:"size:128;uniqueIndex;not null" json:"-"`
|
||||
Status string `gorm:"size:20;not null;default:'offline'" json:"status"`
|
||||
IsLocal bool `gorm:"not null;default:false" json:"isLocal"`
|
||||
OS string `gorm:"size:64" json:"os"`
|
||||
Arch string `gorm:"size:32" json:"arch"`
|
||||
AgentVer string `gorm:"column:agent_version;size:32" json:"agentVersion"`
|
||||
LastSeen time.Time `gorm:"column:last_seen" json:"lastSeen"`
|
||||
PrevToken string `gorm:"size:128;index" json:"-"`
|
||||
PrevTokenExpires *time.Time `gorm:"column:prev_token_expires" json:"-"`
|
||||
// MaxConcurrent 该节点允许的最大并发任务数(0=不限制,沿用全局 cfg.Backup.MaxConcurrent)。
|
||||
// 用于大集群中限制单节点资源占用:例如小内存 Agent 节点可配 1,避免多个大备份同时跑挤爆。
|
||||
MaxConcurrent int `gorm:"column:max_concurrent;not null;default:0" json:"maxConcurrent"`
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNodeEffectiveStatus(t *testing.T) {
|
||||
now := time.Date(2026, 5, 26, 12, 0, 0, 0, time.UTC)
|
||||
cases := []struct {
|
||||
name string
|
||||
node *Node
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "remote fresh heartbeat → stored online",
|
||||
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-10 * time.Second)},
|
||||
want: NodeStatusOnline,
|
||||
},
|
||||
{
|
||||
name: "remote stale heartbeat but stored online → derived offline",
|
||||
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-90 * time.Second)},
|
||||
want: NodeStatusOffline,
|
||||
},
|
||||
{
|
||||
name: "remote just past grace period → offline",
|
||||
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-(OfflineGracePeriod + time.Second))},
|
||||
want: NodeStatusOffline,
|
||||
},
|
||||
{
|
||||
name: "remote within grace period → online",
|
||||
node: &Node{IsLocal: false, Status: NodeStatusOnline, LastSeen: now.Add(-(OfflineGracePeriod - time.Second))},
|
||||
want: NodeStatusOnline,
|
||||
},
|
||||
{
|
||||
name: "local node ignores LastSeen → stored online",
|
||||
node: &Node{IsLocal: true, Status: NodeStatusOnline, LastSeen: now.Add(-24 * time.Hour)},
|
||||
want: NodeStatusOnline,
|
||||
},
|
||||
{
|
||||
name: "remote stored offline stays offline",
|
||||
node: &Node{IsLocal: false, Status: NodeStatusOffline, LastSeen: now.Add(-5 * time.Second)},
|
||||
want: NodeStatusOffline,
|
||||
},
|
||||
{
|
||||
name: "nil node → offline",
|
||||
node: nil,
|
||||
want: NodeStatusOffline,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := tc.node.EffectiveStatus(now); got != tc.want {
|
||||
t.Fatalf("EffectiveStatus = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -11,23 +11,21 @@ const (
|
||||
)
|
||||
|
||||
type RestoreRecord struct {
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
BackupRecordID uint `gorm:"column:backup_record_id;index;not null" json:"backupRecordId"`
|
||||
BackupRecord BackupRecord `json:"backupRecord,omitempty"`
|
||||
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
|
||||
Task BackupTask `json:"task,omitempty"`
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
// TargetPath 恢复到指定目录(仅文件类型本机恢复);空 = 恢复到原始源路径。
|
||||
TargetPath string `gorm:"column:target_path;size:500" json:"targetPath"`
|
||||
Status string `gorm:"size:20;index;not null" json:"status"`
|
||||
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
|
||||
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
|
||||
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
|
||||
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
|
||||
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
|
||||
TriggeredBy string `gorm:"column:triggered_by;size:100" json:"triggeredBy"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
ID uint `gorm:"primaryKey" json:"id"`
|
||||
BackupRecordID uint `gorm:"column:backup_record_id;index;not null" json:"backupRecordId"`
|
||||
BackupRecord BackupRecord `json:"backupRecord,omitempty"`
|
||||
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
|
||||
Task BackupTask `json:"task,omitempty"`
|
||||
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
|
||||
Status string `gorm:"size:20;index;not null" json:"status"`
|
||||
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
|
||||
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
|
||||
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
|
||||
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
|
||||
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
|
||||
TriggeredBy string `gorm:"column:triggered_by;size:100" json:"triggeredBy"`
|
||||
CreatedAt time.Time `json:"createdAt"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
func (RestoreRecord) TableName() string {
|
||||
|
||||
@@ -37,12 +37,13 @@ func (n *EmailNotifier) Send(_ context.Context, config map[string]any, message M
|
||||
from := strings.TrimSpace(asString(config["from"]))
|
||||
toList := splitCommaValues(asString(config["to"]))
|
||||
address := host + ":" + strconv.Itoa(port)
|
||||
headers := []string{"From: " + from, "To: " + strings.Join(toList, ", "), "Subject: " + message.Title, "MIME-Version: 1.0", "Content-Type: text/plain; charset=UTF-8", "", message.Body}
|
||||
var auth smtp.Auth
|
||||
if username != "" {
|
||||
auth = smtp.PlainAuth("", username, password, host)
|
||||
}
|
||||
|
||||
rawMessage := buildRawMessage(from, toList, message)
|
||||
rawMessage := []byte(strings.Join(headers, "\r\n"))
|
||||
|
||||
if port == 465 {
|
||||
tlsConfig := &tls.Config{ServerName: host}
|
||||
@@ -85,31 +86,3 @@ func (n *EmailNotifier) Send(_ context.Context, config map[string]any, message M
|
||||
|
||||
return smtp.SendMail(address, auth, from, toList, rawMessage)
|
||||
}
|
||||
|
||||
// buildRawMessage 构造 RFC 5322 邮件原文。所有头部值都会剔除 CR/LF,
|
||||
// 防止 SMTP 头注入:备份任务名等用户可控内容会进入 Subject,若包含
|
||||
// 换行符可被用来注入额外头部(如 Bcc)或伪造正文。正文本身不做处理,
|
||||
// 允许包含换行。
|
||||
func buildRawMessage(from string, toList []string, message Message) []byte {
|
||||
sanitizedTo := make([]string, 0, len(toList))
|
||||
for _, addr := range toList {
|
||||
if s := sanitizeHeaderValue(addr); s != "" {
|
||||
sanitizedTo = append(sanitizedTo, s)
|
||||
}
|
||||
}
|
||||
headers := []string{
|
||||
"From: " + sanitizeHeaderValue(from),
|
||||
"To: " + strings.Join(sanitizedTo, ", "),
|
||||
"Subject: " + sanitizeHeaderValue(message.Title),
|
||||
"MIME-Version: 1.0",
|
||||
"Content-Type: text/plain; charset=UTF-8",
|
||||
"",
|
||||
message.Body,
|
||||
}
|
||||
return []byte(strings.Join(headers, "\r\n"))
|
||||
}
|
||||
|
||||
// sanitizeHeaderValue 移除头部值中的 CR 与 LF,消除头注入向量。
|
||||
func sanitizeHeaderValue(value string) string {
|
||||
return strings.NewReplacer("\r", "", "\n", "").Replace(strings.TrimSpace(value))
|
||||
}
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestBuildRawMessageStripsHeaderInjection 验证用户可控内容(如备份任务名
|
||||
// 进入 Subject)中的 CR/LF 被剔除,无法注入额外头部或伪造正文。
|
||||
func TestBuildRawMessageStripsHeaderInjection(t *testing.T) {
|
||||
msg := Message{
|
||||
Title: "备份失败\r\nBcc: attacker@evil.com\r\n\r\n伪造正文",
|
||||
Body: "正文第一行\n正文第二行",
|
||||
}
|
||||
raw := string(buildRawMessage("sender@example.com", []string{"ops@example.com"}, msg))
|
||||
|
||||
parts := strings.SplitN(raw, "\r\n\r\n", 2)
|
||||
if len(parts) != 2 {
|
||||
t.Fatalf("缺少头部/正文分隔符,原文=%q", raw)
|
||||
}
|
||||
headerBlock, body := parts[0], parts[1]
|
||||
|
||||
// 头部区不得出现独立的注入头行。
|
||||
for _, line := range strings.Split(headerBlock, "\r\n") {
|
||||
if strings.HasPrefix(line, "Bcc:") {
|
||||
t.Fatalf("检测到头注入:出现独立 Bcc 头行 %q", line)
|
||||
}
|
||||
}
|
||||
// 头部区应恰好是固定的 5 行(From/To/Subject/MIME-Version/Content-Type)。
|
||||
if got := len(strings.Split(headerBlock, "\r\n")); got != 5 {
|
||||
t.Fatalf("头部行数=%d,期望 5;headerBlock=%q", got, headerBlock)
|
||||
}
|
||||
// 正文必须保持原样(正文中的 \n 合法,不应被处理)。
|
||||
if body != "正文第一行\n正文第二行" {
|
||||
t.Fatalf("正文被篡改:%q", body)
|
||||
}
|
||||
// Subject 行必须包含原始标题文本(CRLF 被移除后拼接在同一行)。
|
||||
if !strings.Contains(headerBlock, "Subject: 备份失败Bcc: attacker@evil.com伪造正文") {
|
||||
t.Fatalf("Subject 行不符合预期:%q", headerBlock)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeHeaderValue(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
" normal ": "normal",
|
||||
"a\r\nb": "ab",
|
||||
"x\ny\rz": "xyz",
|
||||
"no-control-chars": "no-control-chars",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := sanitizeHeaderValue(in); got != want {
|
||||
t.Errorf("sanitizeHeaderValue(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -22,18 +21,9 @@ func (n *WebhookNotifier) Type() string { return "webhook" }
|
||||
func (n *WebhookNotifier) SensitiveFields() []string { return []string{"secret"} }
|
||||
|
||||
func (n *WebhookNotifier) Validate(config map[string]any) error {
|
||||
raw := strings.TrimSpace(asString(config["url"]))
|
||||
if raw == "" {
|
||||
if strings.TrimSpace(asString(config["url"])) == "" {
|
||||
return fmt.Errorf("webhook url is required")
|
||||
}
|
||||
parsed, err := url.Parse(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("webhook url is invalid: %w", err)
|
||||
}
|
||||
// 仅允许 http/https,杜绝 file://、gopher:// 等可被用于 SSRF 的协议。
|
||||
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||
return fmt.Errorf("webhook url must use http or https scheme")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -17,46 +17,15 @@ type AgentCommandRepository interface {
|
||||
// 并返回领取到的命令。无命令时返回 (nil, nil)。
|
||||
ClaimPending(ctx context.Context, nodeID uint) (*model.AgentCommand, error)
|
||||
Update(ctx context.Context, cmd *model.AgentCommand) error
|
||||
// CompleteDispatched 只在命令仍处于 dispatched 时写入终态。
|
||||
// 返回 false 表示命令已被超时监控或其它流程终结,调用方不应覆盖。
|
||||
CompleteDispatched(ctx context.Context, cmd *model.AgentCommand) (bool, error)
|
||||
// MarkStaleTimeout 把 dispatched 状态但超时未完成的命令标记为 timeout。
|
||||
// 返回被标记的行数。不返回具体命令(供背景监控简单调用)。
|
||||
MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error)
|
||||
// TimeoutActive 只在命令仍处于 pending/dispatched 时写入 timeout。
|
||||
// 返回 false 表示命令已被 Agent 回写为终态,调用方不应覆盖。
|
||||
TimeoutActive(ctx context.Context, cmd *model.AgentCommand) (bool, error)
|
||||
// ListStaleDispatched 列出 dispatched 但已超时、尚未被标记的命令。
|
||||
// 调用方需要把它们逐一标记 timeout 并联动关联记录状态。
|
||||
ListStaleDispatched(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error)
|
||||
// ListStaleActive 列出 pending/dispatched 但已超时、尚未完成的命令。
|
||||
// pending 使用 created_at 判定,dispatched 使用 dispatched_at 判定。
|
||||
ListStaleActive(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error)
|
||||
// ListPendingByNode 列出某节点下的所有 pending/dispatched 命令。
|
||||
// 用于删除节点或节点离线时的清理。
|
||||
ListPendingByNode(ctx context.Context, nodeID uint) ([]model.AgentCommand, error)
|
||||
NodeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error)
|
||||
}
|
||||
|
||||
type AgentCommandQueueSummary struct {
|
||||
NodeID uint `json:"nodeId"`
|
||||
Pending int `json:"pending"`
|
||||
Dispatched int `json:"dispatched"`
|
||||
Running int `json:"running"`
|
||||
Depth int `json:"depth"`
|
||||
Timeouts int `json:"timeouts"`
|
||||
LastError string `json:"lastError,omitempty"`
|
||||
OldestActiveAt *time.Time `json:"oldestActiveAt,omitempty"`
|
||||
}
|
||||
|
||||
type agentCommandTimeoutCount struct {
|
||||
NodeID uint
|
||||
Count int
|
||||
}
|
||||
|
||||
type agentCommandLastError struct {
|
||||
NodeID uint
|
||||
ErrorMessage string
|
||||
}
|
||||
|
||||
type GormAgentCommandRepository struct {
|
||||
@@ -125,21 +94,6 @@ func (r *GormAgentCommandRepository) Update(ctx context.Context, cmd *model.Agen
|
||||
return r.db.WithContext(ctx).Save(cmd).Error
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) CompleteDispatched(ctx context.Context, cmd *model.AgentCommand) (bool, error) {
|
||||
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
|
||||
Where("id = ? AND node_id = ? AND status = ?", cmd.ID, cmd.NodeID, model.AgentCommandStatusDispatched).
|
||||
Updates(map[string]any{
|
||||
"status": cmd.Status,
|
||||
"error_message": cmd.ErrorMessage,
|
||||
"result": cmd.Result,
|
||||
"completed_at": cmd.CompletedAt,
|
||||
})
|
||||
if result.Error != nil {
|
||||
return false, result.Error
|
||||
}
|
||||
return result.RowsAffected > 0, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error) {
|
||||
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
|
||||
Where("status = ? AND dispatched_at < ?", model.AgentCommandStatusDispatched, threshold).
|
||||
@@ -153,20 +107,6 @@ func (r *GormAgentCommandRepository) MarkStaleTimeout(ctx context.Context, thres
|
||||
return result.RowsAffected, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) TimeoutActive(ctx context.Context, cmd *model.AgentCommand) (bool, error) {
|
||||
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
|
||||
Where("id = ? AND status IN ?", cmd.ID, []string{model.AgentCommandStatusPending, model.AgentCommandStatusDispatched}).
|
||||
Updates(map[string]any{
|
||||
"status": model.AgentCommandStatusTimeout,
|
||||
"error_message": cmd.ErrorMessage,
|
||||
"completed_at": cmd.CompletedAt,
|
||||
})
|
||||
if result.Error != nil {
|
||||
return false, result.Error
|
||||
}
|
||||
return result.RowsAffected > 0, nil
|
||||
}
|
||||
|
||||
// ListStaleDispatched 列出 dispatched 但 dispatched_at 早于 threshold 的命令。
|
||||
func (r *GormAgentCommandRepository) ListStaleDispatched(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error) {
|
||||
var items []model.AgentCommand
|
||||
@@ -179,21 +119,6 @@ func (r *GormAgentCommandRepository) ListStaleDispatched(ctx context.Context, th
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) ListStaleActive(ctx context.Context, threshold time.Time) ([]model.AgentCommand, error) {
|
||||
var items []model.AgentCommand
|
||||
if err := r.db.WithContext(ctx).
|
||||
Where(
|
||||
"(status = ? AND created_at < ?) OR (status = ? AND dispatched_at < ?)",
|
||||
model.AgentCommandStatusPending, threshold,
|
||||
model.AgentCommandStatusDispatched, threshold,
|
||||
).
|
||||
Order("id asc").
|
||||
Find(&items).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// ListPendingByNode 列出某节点下所有待执行(pending 或 dispatched)命令。
|
||||
func (r *GormAgentCommandRepository) ListPendingByNode(ctx context.Context, nodeID uint) ([]model.AgentCommand, error) {
|
||||
var items []model.AgentCommand
|
||||
@@ -208,114 +133,3 @@ func (r *GormAgentCommandRepository) ListPendingByNode(ctx context.Context, node
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) NodeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error) {
|
||||
summaries, err := r.activeQueueSummaries(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := r.applyTerminalQueueStats(ctx, summaries); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return summaries, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) activeQueueSummaries(ctx context.Context) (map[uint]AgentCommandQueueSummary, error) {
|
||||
var items []model.AgentCommand
|
||||
if err := r.db.WithContext(ctx).
|
||||
Where("status IN ?", []string{
|
||||
model.AgentCommandStatusPending,
|
||||
model.AgentCommandStatusDispatched,
|
||||
}).
|
||||
Order("node_id asc, id asc").
|
||||
Find(&items).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
summaries := make(map[uint]AgentCommandQueueSummary)
|
||||
for i := range items {
|
||||
cmd := &items[i]
|
||||
summary := summaries[cmd.NodeID]
|
||||
summary.NodeID = cmd.NodeID
|
||||
switch cmd.Status {
|
||||
case model.AgentCommandStatusPending:
|
||||
summary.Pending++
|
||||
summary.Depth++
|
||||
summary.OldestActiveAt = oldestTime(summary.OldestActiveAt, &cmd.CreatedAt)
|
||||
case model.AgentCommandStatusDispatched:
|
||||
summary.Dispatched++
|
||||
summary.Depth++
|
||||
if isLongRunningAgentCommand(cmd.Type) {
|
||||
summary.Running++
|
||||
}
|
||||
summary.OldestActiveAt = oldestTime(summary.OldestActiveAt, cmd.DispatchedAt)
|
||||
}
|
||||
summaries[cmd.NodeID] = summary
|
||||
}
|
||||
return summaries, nil
|
||||
}
|
||||
|
||||
func (r *GormAgentCommandRepository) applyTerminalQueueStats(ctx context.Context, summaries map[uint]AgentCommandQueueSummary) error {
|
||||
var timeoutCounts []agentCommandTimeoutCount
|
||||
if err := r.db.WithContext(ctx).
|
||||
Model(&model.AgentCommand{}).
|
||||
Select("node_id, COUNT(*) AS count").
|
||||
Where("status = ?", model.AgentCommandStatusTimeout).
|
||||
Group("node_id").
|
||||
Scan(&timeoutCounts).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
for _, item := range timeoutCounts {
|
||||
summary := summaries[item.NodeID]
|
||||
summary.NodeID = item.NodeID
|
||||
summary.Timeouts = item.Count
|
||||
summaries[item.NodeID] = summary
|
||||
}
|
||||
|
||||
terminalStatuses := []string{
|
||||
model.AgentCommandStatusFailed,
|
||||
model.AgentCommandStatusTimeout,
|
||||
}
|
||||
latestByNode := r.db.WithContext(ctx).
|
||||
Model(&model.AgentCommand{}).
|
||||
Select("node_id, MAX(COALESCE(completed_at, updated_at, created_at)) AS last_error_at").
|
||||
Where("status IN ? AND error_message <> ''", terminalStatuses).
|
||||
Group("node_id")
|
||||
|
||||
var lastErrors []agentCommandLastError
|
||||
if err := r.db.WithContext(ctx).
|
||||
Table("agent_commands AS cmd").
|
||||
Select("cmd.node_id, cmd.error_message").
|
||||
Joins("JOIN (?) latest ON latest.node_id = cmd.node_id AND latest.last_error_at = COALESCE(cmd.completed_at, cmd.updated_at, cmd.created_at)", latestByNode).
|
||||
Where("cmd.status IN ? AND cmd.error_message <> ''", terminalStatuses).
|
||||
Order("cmd.node_id asc, cmd.id desc").
|
||||
Scan(&lastErrors).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
seenLastError := make(map[uint]struct{}, len(lastErrors))
|
||||
for _, item := range lastErrors {
|
||||
if _, ok := seenLastError[item.NodeID]; ok {
|
||||
continue
|
||||
}
|
||||
summary := summaries[item.NodeID]
|
||||
summary.NodeID = item.NodeID
|
||||
summary.LastError = item.ErrorMessage
|
||||
summaries[item.NodeID] = summary
|
||||
seenLastError[item.NodeID] = struct{}{}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func oldestTime(current *time.Time, candidate *time.Time) *time.Time {
|
||||
if candidate == nil {
|
||||
return current
|
||||
}
|
||||
if current == nil || candidate.Before(*current) {
|
||||
value := *candidate
|
||||
return &value
|
||||
}
|
||||
return current
|
||||
}
|
||||
|
||||
func isLongRunningAgentCommand(commandType string) bool {
|
||||
return commandType == model.AgentCommandTypeRunTask || commandType == model.AgentCommandTypeRestoreRecord
|
||||
}
|
||||
|
||||
@@ -90,78 +90,6 @@ func TestAgentCommandRepository_Update(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentCommandRepository_CompleteDispatchedOnlyUpdatesDispatchedCommand(t *testing.T) {
|
||||
db := newTestDB(t)
|
||||
repo := NewAgentCommandRepository(db)
|
||||
ctx := context.Background()
|
||||
dispatched := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusDispatched}
|
||||
timeout := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusTimeout, ErrorMessage: "timeout"}
|
||||
if err := repo.Create(ctx, dispatched); err != nil {
|
||||
t.Fatalf("Create dispatched returned error: %v", err)
|
||||
}
|
||||
if err := repo.Create(ctx, timeout); err != nil {
|
||||
t.Fatalf("Create timeout returned error: %v", err)
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
dispatched.Status = model.AgentCommandStatusSucceeded
|
||||
dispatched.Result = `{"ok":true}`
|
||||
dispatched.CompletedAt = &now
|
||||
updated, err := repo.CompleteDispatched(ctx, dispatched)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteDispatched returned error: %v", err)
|
||||
}
|
||||
if !updated {
|
||||
t.Fatal("expected dispatched command to be updated")
|
||||
}
|
||||
|
||||
timeout.Status = model.AgentCommandStatusSucceeded
|
||||
timeout.Result = `{"late":true}`
|
||||
timeout.CompletedAt = &now
|
||||
updated, err = repo.CompleteDispatched(ctx, timeout)
|
||||
if err != nil {
|
||||
t.Fatalf("CompleteDispatched terminal returned error: %v", err)
|
||||
}
|
||||
if updated {
|
||||
t.Fatal("expected terminal command not to be updated")
|
||||
}
|
||||
gotTimeout, err := repo.FindByID(ctx, timeout.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID timeout returned error: %v", err)
|
||||
}
|
||||
if gotTimeout.Status != model.AgentCommandStatusTimeout || gotTimeout.Result != "" {
|
||||
t.Fatalf("expected timeout command unchanged, got %#v", gotTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentCommandRepository_TimeoutActiveDoesNotOverwriteTerminalCommand(t *testing.T) {
|
||||
db := newTestDB(t)
|
||||
repo := NewAgentCommandRepository(db)
|
||||
ctx := context.Background()
|
||||
succeeded := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusSucceeded, Result: `{"ok":true}`}
|
||||
if err := repo.Create(ctx, succeeded); err != nil {
|
||||
t.Fatalf("Create succeeded returned error: %v", err)
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
succeeded.ErrorMessage = "timeout"
|
||||
succeeded.CompletedAt = &now
|
||||
updated, err := repo.TimeoutActive(ctx, succeeded)
|
||||
if err != nil {
|
||||
t.Fatalf("TimeoutActive returned error: %v", err)
|
||||
}
|
||||
if updated {
|
||||
t.Fatal("expected terminal command not to be timed out")
|
||||
}
|
||||
got, err := repo.FindByID(ctx, succeeded.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID returned error: %v", err)
|
||||
}
|
||||
if got.Status != model.AgentCommandStatusSucceeded || got.ErrorMessage != "" || got.Result != `{"ok":true}` {
|
||||
t.Fatalf("expected succeeded command unchanged, got %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentCommandRepository_MarkStaleTimeout(t *testing.T) {
|
||||
db := newTestDB(t)
|
||||
repo := NewAgentCommandRepository(db)
|
||||
@@ -190,72 +118,3 @@ func TestAgentCommandRepository_MarkStaleTimeout(t *testing.T) {
|
||||
t.Errorf("new should stay dispatched: %+v", newGot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentCommandRepository_ListStaleActiveIncludesPendingAndDispatched(t *testing.T) {
|
||||
db := newTestDB(t)
|
||||
repo := NewAgentCommandRepository(db)
|
||||
ctx := context.Background()
|
||||
old := time.Now().Add(-time.Hour)
|
||||
recent := time.Now()
|
||||
oldPending := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusPending, CreatedAt: old}
|
||||
oldDispatched := &model.AgentCommand{NodeID: 1, Type: "restore_record", Status: model.AgentCommandStatusDispatched, DispatchedAt: &old}
|
||||
recentPending := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusPending, CreatedAt: recent}
|
||||
succeeded := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusSucceeded, CreatedAt: old}
|
||||
for _, cmd := range []*model.AgentCommand{oldPending, oldDispatched, recentPending, succeeded} {
|
||||
if err := repo.Create(ctx, cmd); err != nil {
|
||||
t.Fatalf("Create returned error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
items, err := repo.ListStaleActive(ctx, time.Now().Add(-30*time.Minute))
|
||||
if err != nil {
|
||||
t.Fatalf("ListStaleActive returned error: %v", err)
|
||||
}
|
||||
if len(items) != 2 {
|
||||
t.Fatalf("expected 2 stale active commands, got %#v", items)
|
||||
}
|
||||
if items[0].ID != oldPending.ID || items[1].ID != oldDispatched.ID {
|
||||
t.Fatalf("unexpected stale active order/items: %#v", items)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentCommandRepository_NodeQueueSummaries(t *testing.T) {
|
||||
db := newTestDB(t)
|
||||
repo := NewAgentCommandRepository(db)
|
||||
ctx := context.Background()
|
||||
old := time.Now().UTC().Add(-20 * time.Minute)
|
||||
recent := time.Now().UTC().Add(-2 * time.Minute)
|
||||
dispatchedAt := time.Now().UTC().Add(-5 * time.Minute)
|
||||
completedAt := time.Now().UTC().Add(-1 * time.Minute)
|
||||
commands := []*model.AgentCommand{
|
||||
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusPending, CreatedAt: old},
|
||||
{NodeID: 1, Type: model.AgentCommandTypeRestoreRecord, Status: model.AgentCommandStatusPending, CreatedAt: recent},
|
||||
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusDispatched, DispatchedAt: &dispatchedAt},
|
||||
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusFailed, ErrorMessage: "boom", CompletedAt: &completedAt},
|
||||
{NodeID: 1, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusTimeout, ErrorMessage: "late", CompletedAt: &recent},
|
||||
{NodeID: 2, Type: model.AgentCommandTypeRunTask, Status: model.AgentCommandStatusPending, CreatedAt: old},
|
||||
}
|
||||
for _, cmd := range commands {
|
||||
if err := repo.Create(ctx, cmd); err != nil {
|
||||
t.Fatalf("Create returned error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
summaries, err := repo.NodeQueueSummaries(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("NodeQueueSummaries returned error: %v", err)
|
||||
}
|
||||
nodeOne := summaries[1]
|
||||
if nodeOne.Pending != 2 || nodeOne.Dispatched != 1 || nodeOne.Running != 1 || nodeOne.Depth != 3 {
|
||||
t.Fatalf("unexpected node 1 summary: %#v", nodeOne)
|
||||
}
|
||||
if nodeOne.Timeouts != 1 || nodeOne.LastError != "boom" {
|
||||
t.Fatalf("expected terminal timeout and latest error in summary, got %#v", nodeOne)
|
||||
}
|
||||
if nodeOne.OldestActiveAt == nil || !nodeOne.OldestActiveAt.Equal(old) {
|
||||
t.Fatalf("expected oldest active at %s, got %#v", old, nodeOne.OldestActiveAt)
|
||||
}
|
||||
if nodeTwo := summaries[2]; nodeTwo.Pending != 1 || nodeTwo.Depth != 1 || nodeTwo.Timeouts != 0 || nodeTwo.LastError != "" {
|
||||
t.Fatalf("unexpected node 2 summary: %#v", nodeTwo)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,7 +3,6 @@ package repository
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -84,59 +83,6 @@ func TestInstallTokenConsumeExpired(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstallTokenConsumeConcurrentOnlyOneWins(t *testing.T) {
|
||||
db := openTestInstallTokenDB(t)
|
||||
repo := NewAgentInstallTokenRepository(db)
|
||||
ctx := context.Background()
|
||||
|
||||
tok := &model.AgentInstallToken{
|
||||
Token: "concurrent", NodeID: 1, Mode: model.InstallModeSystemd,
|
||||
Arch: model.InstallArchAuto, AgentVer: "v1.7.0",
|
||||
DownloadSrc: model.InstallSourceGitHub,
|
||||
ExpiresAt: time.Now().UTC().Add(15 * time.Minute),
|
||||
CreatedByID: 1,
|
||||
}
|
||||
if err := repo.Create(ctx, tok); err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
|
||||
const workers = 8
|
||||
var wg sync.WaitGroup
|
||||
start := make(chan struct{})
|
||||
results := make(chan *model.AgentInstallToken, workers)
|
||||
errs := make(chan error, workers)
|
||||
for i := 0; i < workers; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-start
|
||||
got, err := repo.ConsumeByToken(ctx, "concurrent")
|
||||
if err != nil {
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
results <- got
|
||||
}()
|
||||
}
|
||||
close(start)
|
||||
wg.Wait()
|
||||
close(results)
|
||||
close(errs)
|
||||
|
||||
for err := range errs {
|
||||
t.Fatalf("consume err: %v", err)
|
||||
}
|
||||
success := 0
|
||||
for got := range results {
|
||||
if got != nil {
|
||||
success++
|
||||
}
|
||||
}
|
||||
if success != 1 {
|
||||
t.Fatalf("expected exactly one successful consume, got %d", success)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstallTokenGC(t *testing.T) {
|
||||
db := openTestInstallTokenDB(t)
|
||||
repo := NewAgentInstallTokenRepository(db)
|
||||
|
||||
@@ -29,8 +29,6 @@ type AuditLogRepository interface {
|
||||
Create(ctx context.Context, log *model.AuditLog) error
|
||||
List(ctx context.Context, opts AuditLogListOptions) (*AuditLogListResult, error)
|
||||
ListAll(ctx context.Context, opts AuditLogListOptions) ([]model.AuditLog, error)
|
||||
// DeleteBefore 删除 created_at 早于 cutoff 的审计日志,返回删除行数。用于保留期清理。
|
||||
DeleteBefore(ctx context.Context, cutoff time.Time) (int64, error)
|
||||
}
|
||||
|
||||
type gormAuditLogRepository struct {
|
||||
@@ -45,11 +43,6 @@ func (r *gormAuditLogRepository) Create(_ context.Context, log *model.AuditLog)
|
||||
return r.db.Create(log).Error
|
||||
}
|
||||
|
||||
func (r *gormAuditLogRepository) DeleteBefore(ctx context.Context, cutoff time.Time) (int64, error) {
|
||||
result := r.db.WithContext(ctx).Where("created_at < ?", cutoff).Delete(&model.AuditLog{})
|
||||
return result.RowsAffected, result.Error
|
||||
}
|
||||
|
||||
func (r *gormAuditLogRepository) List(_ context.Context, opts AuditLogListOptions) (*AuditLogListResult, error) {
|
||||
query := r.buildQuery(opts)
|
||||
var total int64
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/model"
|
||||
)
|
||||
|
||||
// 列表查询应省略 Manifest 列(避免拖出大 JSON),而 FindByID 仍保留(内容浏览/差异基线需要)。
|
||||
func TestListSuccessfulByTaskOmitsManifest(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
repo := newBackupRecordTestRepository(t)
|
||||
rec := &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: "success", BackupKind: model.BackupKindFull, Manifest: `{"entries":[{"p":"x"}]}`, StartedAt: time.Now().UTC()}
|
||||
if err := repo.Create(ctx, rec); err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
|
||||
items, err := repo.ListSuccessfulByTask(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("ListSuccessfulByTask: %v", err)
|
||||
}
|
||||
if len(items) == 0 {
|
||||
t.Fatal("expected at least one record")
|
||||
}
|
||||
for _, it := range items {
|
||||
if it.Manifest != "" {
|
||||
t.Fatalf("ListSuccessfulByTask must omit Manifest, got %q", it.Manifest)
|
||||
}
|
||||
}
|
||||
|
||||
full, err := repo.FindByID(ctx, rec.ID)
|
||||
if err != nil || full == nil {
|
||||
t.Fatalf("FindByID: %v / %v", full, err)
|
||||
}
|
||||
if full.Manifest == "" {
|
||||
t.Fatal("FindByID must retain Manifest (browse/diff depend on it)")
|
||||
}
|
||||
}
|
||||
|
||||
// 仅统计「成功且依赖给定全量」的差异备份:失败的或依赖其他全量的不计入。
|
||||
func TestCountDependentDifferentials(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
repo := newBackupRecordTestRepository(t)
|
||||
now := time.Now().UTC()
|
||||
base := &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: "success", BackupKind: model.BackupKindFull, StartedAt: now}
|
||||
if err := repo.Create(ctx, base); err != nil {
|
||||
t.Fatalf("create base: %v", err)
|
||||
}
|
||||
mk := func(status, kind string, baseID uint) {
|
||||
if err := repo.Create(ctx, &model.BackupRecord{TaskID: 1, StorageTargetID: 1, Status: status, BackupKind: kind, BaseRecordID: baseID, StartedAt: now}); err != nil {
|
||||
t.Fatalf("create dependent: %v", err)
|
||||
}
|
||||
}
|
||||
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, base.ID) // 计入
|
||||
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, base.ID) // 计入
|
||||
mk(model.BackupRecordStatusFailed, model.BackupKindDifferential, base.ID) // 失败 → 不计
|
||||
mk(model.BackupRecordStatusSuccess, model.BackupKindDifferential, 99999) // 依赖其他全量 → 不计
|
||||
|
||||
n, err := repo.CountDependentDifferentials(ctx, base.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("CountDependentDifferentials: %v", err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Fatalf("want 2 dependent successful differentials, got %d", n)
|
||||
}
|
||||
}
|
||||
@@ -33,14 +33,12 @@ type BackupStorageUsageItem struct {
|
||||
type BackupRecordRepository interface {
|
||||
List(context.Context, BackupRecordListOptions) ([]model.BackupRecord, error)
|
||||
FindByID(context.Context, uint) (*model.BackupRecord, error)
|
||||
FindRunningByTaskAndNode(context.Context, uint, uint) (*model.BackupRecord, error)
|
||||
Create(context.Context, *model.BackupRecord) error
|
||||
Update(context.Context, *model.BackupRecord) error
|
||||
Delete(context.Context, uint) error
|
||||
ListRecent(context.Context, int) ([]model.BackupRecord, error)
|
||||
ListByTask(context.Context, uint) ([]model.BackupRecord, error)
|
||||
ListSuccessfulByTask(context.Context, uint) ([]model.BackupRecord, error)
|
||||
CountDependentDifferentials(context.Context, uint) (int64, error)
|
||||
Count(context.Context) (int64, error)
|
||||
CountSince(context.Context, time.Time) (int64, error)
|
||||
CountSuccessSince(context.Context, time.Time) (int64, error)
|
||||
@@ -58,8 +56,7 @@ func NewBackupRecordRepository(db *gorm.DB) *GormBackupRecordRepository {
|
||||
}
|
||||
|
||||
func (r *GormBackupRecordRepository) List(ctx context.Context, options BackupRecordListOptions) ([]model.BackupRecord, error) {
|
||||
// Omit("Manifest"):列表不需要可能很大的清单 JSON,避免每行拖出该 TEXT 列。
|
||||
query := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Omit("Manifest").Preload("Task").Preload("Task.StorageTarget").Order("started_at desc")
|
||||
query := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Preload("Task").Preload("Task.StorageTarget").Order("started_at desc")
|
||||
if options.TaskID != nil {
|
||||
query = query.Where("task_id = ?", *options.TaskID)
|
||||
}
|
||||
@@ -96,20 +93,6 @@ func (r *GormBackupRecordRepository) FindByID(ctx context.Context, id uint) (*mo
|
||||
return &item, nil
|
||||
}
|
||||
|
||||
func (r *GormBackupRecordRepository) FindRunningByTaskAndNode(ctx context.Context, taskID uint, nodeID uint) (*model.BackupRecord, error) {
|
||||
var item model.BackupRecord
|
||||
if err := r.db.WithContext(ctx).
|
||||
Where("task_id = ? AND node_id = ? AND status = ?", taskID, nodeID, model.BackupRecordStatusRunning).
|
||||
Order("id desc").
|
||||
First(&item).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return &item, nil
|
||||
}
|
||||
|
||||
func (r *GormBackupRecordRepository) Create(ctx context.Context, item *model.BackupRecord) error {
|
||||
return r.db.WithContext(ctx).Create(item).Error
|
||||
}
|
||||
@@ -127,7 +110,7 @@ func (r *GormBackupRecordRepository) ListRecent(ctx context.Context, limit int)
|
||||
limit = 10
|
||||
}
|
||||
var items []model.BackupRecord
|
||||
if err := r.db.WithContext(ctx).Omit("Manifest").Preload("Task").Preload("Task.StorageTarget").Order("started_at desc").Limit(limit).Find(&items).Error; err != nil {
|
||||
if err := r.db.WithContext(ctx).Preload("Task").Preload("Task.StorageTarget").Order("started_at desc").Limit(limit).Find(&items).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
@@ -135,7 +118,7 @@ func (r *GormBackupRecordRepository) ListRecent(ctx context.Context, limit int)
|
||||
|
||||
func (r *GormBackupRecordRepository) ListByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
|
||||
var items []model.BackupRecord
|
||||
if err := r.db.WithContext(ctx).Omit("Manifest").Where("task_id = ?", taskID).Order("id desc").Find(&items).Error; err != nil {
|
||||
if err := r.db.WithContext(ctx).Where("task_id = ?", taskID).Order("id desc").Find(&items).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
@@ -143,21 +126,12 @@ func (r *GormBackupRecordRepository) ListByTask(ctx context.Context, taskID uint
|
||||
|
||||
func (r *GormBackupRecordRepository) ListSuccessfulByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
|
||||
var items []model.BackupRecord
|
||||
if err := r.db.WithContext(ctx).Omit("Manifest").Where("task_id = ? AND status = ?", taskID, "success").Order("completed_at desc, id desc").Find(&items).Error; err != nil {
|
||||
if err := r.db.WithContext(ctx).Where("task_id = ? AND status = ?", taskID, "success").Order("completed_at desc, id desc").Find(&items).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// CountDependentDifferentials 统计依赖某全量记录(作为基线)的成功差异备份数量。
|
||||
func (r *GormBackupRecordRepository) CountDependentDifferentials(ctx context.Context, baseID uint) (int64, error) {
|
||||
var count int64
|
||||
err := r.db.WithContext(ctx).Model(&model.BackupRecord{}).
|
||||
Where("base_record_id = ? AND backup_kind = ? AND status = ?", baseID, model.BackupKindDifferential, model.BackupRecordStatusSuccess).
|
||||
Count(&count).Error
|
||||
return count, err
|
||||
}
|
||||
|
||||
func (r *GormBackupRecordRepository) Count(ctx context.Context) (int64, error) {
|
||||
var count int64
|
||||
if err := r.db.WithContext(ctx).Model(&model.BackupRecord{}).Count(&count).Error; err != nil {
|
||||
|
||||
@@ -226,7 +226,7 @@ func (r *GormBackupTaskRepository) Create(ctx context.Context, item *model.Backu
|
||||
}
|
||||
|
||||
func (r *GormBackupTaskRepository) Update(ctx context.Context, item *model.BackupTask) error {
|
||||
if err := r.db.WithContext(ctx).Omit("StorageTarget", "StorageTargets", "Node").Save(item).Error; err != nil {
|
||||
if err := r.db.WithContext(ctx).Save(item).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if len(item.StorageTargets) > 0 {
|
||||
|
||||
@@ -92,49 +92,3 @@ func TestBackupTaskRepositoryCRUD(t *testing.T) {
|
||||
t.Fatalf("expected task deleted, got %#v", deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupTaskRepositoryUpdateCanClearNodeIDAfterPreload(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
repo := newBackupTaskTestRepository(t)
|
||||
remoteNode := &model.Node{Name: "edge-1", Token: "edge-token", Status: model.NodeStatusOnline, IsLocal: false}
|
||||
if err := repo.db.WithContext(ctx).Create(remoteNode).Error; err != nil {
|
||||
t.Fatalf("create node: %v", err)
|
||||
}
|
||||
task := &model.BackupTask{
|
||||
Name: "pooled-source",
|
||||
Type: "file",
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/www/site",
|
||||
StorageTargetID: 1,
|
||||
NodeID: remoteNode.ID,
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
LastStatus: "idle",
|
||||
}
|
||||
if err := repo.Create(ctx, task); err != nil {
|
||||
t.Fatalf("Create returned error: %v", err)
|
||||
}
|
||||
loaded, err := repo.FindByID(ctx, task.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID returned error: %v", err)
|
||||
}
|
||||
if loaded == nil || loaded.Node.ID != remoteNode.ID {
|
||||
t.Fatalf("expected preloaded node %d, got %#v", remoteNode.ID, loaded)
|
||||
}
|
||||
loaded.NodeID = 0
|
||||
loaded.NodePoolTag = "db"
|
||||
if err := repo.Update(ctx, loaded); err != nil {
|
||||
t.Fatalf("Update returned error: %v", err)
|
||||
}
|
||||
stored, err := repo.FindByID(ctx, task.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID after update returned error: %v", err)
|
||||
}
|
||||
if stored.NodeID != 0 {
|
||||
t.Fatalf("expected NodeID to be cleared, got %d", stored.NodeID)
|
||||
}
|
||||
if stored.NodePoolTag != "db" {
|
||||
t.Fatalf("expected NodePoolTag db, got %q", stored.NodePoolTag)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -45,7 +45,7 @@ type Service struct {
|
||||
func NewService(tasks repository.BackupTaskRepository, runner TaskRunner, logger *zap.Logger) *Service {
|
||||
parser := cron.NewParser(cron.SecondOptional | cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor)
|
||||
return &Service{
|
||||
cron: cron.New(cron.WithParser(parser), cron.WithLocation(time.Local)),
|
||||
cron: cron.New(cron.WithParser(parser), cron.WithLocation(time.UTC)),
|
||||
tasks: tasks,
|
||||
runner: runner,
|
||||
logger: logger,
|
||||
@@ -167,8 +167,7 @@ func (s *Service) syncTaskLocked(task *model.BackupTask) error {
|
||||
// 集群感知:若任务绑定了离线的远程节点,跳过本轮触发避免堆积 failed 记录
|
||||
if taskNodeID > 0 && s.nodes != nil {
|
||||
node, err := s.nodes.FindByID(context.Background(), taskNodeID)
|
||||
// 用实时推导的状态判定,避免后台监控刷新前把任务下发给刚失联的节点。
|
||||
if err == nil && node != nil && !node.IsLocal && node.EffectiveStatus(time.Now().UTC()) != model.NodeStatusOnline {
|
||||
if err == nil && node != nil && !node.IsLocal && node.Status != model.NodeStatusOnline {
|
||||
if s.logger != nil {
|
||||
s.logger.Warn("skip scheduled run: target node offline",
|
||||
zap.Uint("task_id", taskID), zap.String("task_name", taskName),
|
||||
|
||||
@@ -68,37 +68,3 @@ func TestServiceSyncTaskAndTrigger(t *testing.T) {
|
||||
t.Fatalf("expected scheduled runner to be triggered")
|
||||
}
|
||||
}
|
||||
|
||||
func TestServiceSchedulesTasksInLocalTimezone(t *testing.T) {
|
||||
location, err := time.LoadLocation("Asia/Shanghai")
|
||||
if err != nil {
|
||||
t.Fatalf("LoadLocation returned error: %v", err)
|
||||
}
|
||||
originalLocal := time.Local
|
||||
time.Local = location
|
||||
t.Cleanup(func() {
|
||||
time.Local = originalLocal
|
||||
})
|
||||
|
||||
service := NewService(&fakeTaskRepository{}, &fakeRunner{}, nil)
|
||||
if got := service.cron.Location(); got != location {
|
||||
t.Fatalf("cron location = %v, want %v", got, location)
|
||||
}
|
||||
|
||||
task := &model.BackupTask{ID: 1, Enabled: true, CronExpr: "0 5 * * *"}
|
||||
if err := service.SyncTask(context.Background(), task); err != nil {
|
||||
t.Fatalf("SyncTask returned error: %v", err)
|
||||
}
|
||||
entryID, ok := service.entries[task.ID]
|
||||
if !ok {
|
||||
t.Fatalf("expected cron entry for task %d", task.ID)
|
||||
}
|
||||
|
||||
entry := service.cron.Entry(entryID)
|
||||
now := time.Date(2026, 4, 30, 4, 0, 0, 0, location)
|
||||
got := entry.Schedule.Next(now)
|
||||
want := time.Date(2026, 4, 30, 5, 0, 0, 0, location)
|
||||
if !got.Equal(want) {
|
||||
t.Fatalf("next run = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,22 +2,15 @@ package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/apperror"
|
||||
"backupx/server/internal/backup"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
"backupx/server/internal/storage"
|
||||
"backupx/server/internal/storage/codec"
|
||||
)
|
||||
|
||||
@@ -30,7 +23,6 @@ type AgentService struct {
|
||||
storageRepo repository.StorageTargetRepository
|
||||
cmdRepo repository.AgentCommandRepository
|
||||
restoreRepo repository.RestoreRecordRepository
|
||||
registry *storage.Registry
|
||||
cipher *codec.ConfigCipher
|
||||
}
|
||||
|
||||
@@ -41,7 +33,6 @@ func NewAgentService(
|
||||
storageRepo repository.StorageTargetRepository,
|
||||
cmdRepo repository.AgentCommandRepository,
|
||||
cipher *codec.ConfigCipher,
|
||||
registry *storage.Registry,
|
||||
) *AgentService {
|
||||
return &AgentService{
|
||||
nodeRepo: nodeRepo,
|
||||
@@ -49,7 +40,6 @@ func NewAgentService(
|
||||
recordRepo: recordRepo,
|
||||
storageRepo: storageRepo,
|
||||
cmdRepo: cmdRepo,
|
||||
registry: registry,
|
||||
cipher: cipher,
|
||||
}
|
||||
}
|
||||
@@ -128,8 +118,7 @@ func (s *AgentService) SubmitCommandResult(ctx context.Context, node *model.Node
|
||||
cmd.Result = string(result.Result)
|
||||
}
|
||||
cmd.CompletedAt = &now
|
||||
_, err = s.cmdRepo.CompleteDispatched(ctx, cmd)
|
||||
return err
|
||||
return s.cmdRepo.Update(ctx, cmd)
|
||||
}
|
||||
|
||||
// AgentTaskSpec 给 Agent 返回的任务规格,包含解密后的存储配置,供 Agent 直接执行。
|
||||
@@ -155,11 +144,10 @@ type AgentTaskSpec struct {
|
||||
|
||||
// AgentStorageTargetConfig 存储目标配置(已解密)
|
||||
type AgentStorageTargetConfig struct {
|
||||
ID uint `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Config json.RawMessage `json:"config"`
|
||||
TransferMode string `json:"transferMode"`
|
||||
ID uint `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Config json.RawMessage `json:"config"`
|
||||
}
|
||||
|
||||
// GetTaskSpec 返回 Agent 执行任务所需的完整规格。
|
||||
@@ -171,8 +159,8 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
|
||||
if task == nil {
|
||||
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "任务不存在", nil)
|
||||
}
|
||||
if err := s.ensureTaskSpecAccess(ctx, node, task); err != nil {
|
||||
return nil, err
|
||||
if task.NodeID != node.ID {
|
||||
return nil, apperror.Unauthorized("BACKUP_TASK_FORBIDDEN", "任务不属于当前节点", nil)
|
||||
}
|
||||
// 解密数据库密码(若有)
|
||||
dbPassword := ""
|
||||
@@ -198,22 +186,11 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decrypt storage config: %w", err)
|
||||
}
|
||||
transferMode := storage.TransferModeDirect
|
||||
if strings.EqualFold(target.Type, storage.TypeLocalDisk) {
|
||||
var localConfig storage.LocalDiskConfig
|
||||
if err := json.Unmarshal(configRaw, &localConfig); err != nil {
|
||||
return nil, fmt.Errorf("decode local disk config: %w", err)
|
||||
}
|
||||
if localConfig.MasterRelay {
|
||||
transferMode = storage.TransferModeMasterRelay
|
||||
}
|
||||
}
|
||||
storageTargets = append(storageTargets, AgentStorageTargetConfig{
|
||||
ID: target.ID,
|
||||
Type: target.Type,
|
||||
Name: target.Name,
|
||||
Config: json.RawMessage(configRaw),
|
||||
TransferMode: transferMode,
|
||||
ID: target.ID,
|
||||
Type: target.Type,
|
||||
Name: target.Name,
|
||||
Config: json.RawMessage(configRaw),
|
||||
})
|
||||
}
|
||||
return &AgentTaskSpec{
|
||||
@@ -236,128 +213,15 @@ func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID
|
||||
}, nil
|
||||
}
|
||||
|
||||
// UploadArtifact receives a remote Agent artifact as a stream and writes it
|
||||
// with a provider created on the Master. The first supported use is local_disk,
|
||||
// whose configured path belongs to the Master rather than the source Agent.
|
||||
func (s *AgentService) UploadArtifact(ctx context.Context, node *model.Node, recordID, targetID uint, objectKey string, size int64, checksum string, reader io.Reader) error {
|
||||
if node == nil || reader == nil || s.registry == nil {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "中转上传参数不完整", nil)
|
||||
}
|
||||
record, err := s.recordRepo.FindByID(ctx, recordID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if record == nil {
|
||||
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "记录不存在", nil)
|
||||
}
|
||||
task, err := s.taskRepo.FindByID(ctx, record.TaskID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if task == nil || !recordBelongsToNode(record, task, node.ID) {
|
||||
return apperror.Unauthorized("BACKUP_RECORD_FORBIDDEN", "记录不属于当前节点", nil)
|
||||
}
|
||||
if isBackupRecordTerminal(record.Status) {
|
||||
return apperror.BadRequest("BACKUP_RECORD_TERMINAL", "备份记录已结束,不能继续上传产物", nil)
|
||||
}
|
||||
allowedTarget := false
|
||||
for _, configuredTargetID := range collectTargetIDs(task) {
|
||||
if configuredTargetID == targetID {
|
||||
allowedTarget = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !allowedTarget {
|
||||
return apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
|
||||
}
|
||||
target, err := s.storageRepo.FindByID(ctx, targetID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if target == nil || !strings.EqualFold(target.Type, storage.TypeLocalDisk) {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_RELAY_UNSUPPORTED", "仅 Master 本地磁盘目标需要中转上传", nil)
|
||||
}
|
||||
configMap := map[string]any{}
|
||||
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
|
||||
return fmt.Errorf("decrypt storage config: %w", err)
|
||||
}
|
||||
masterRelay, _ := configMap["masterRelay"].(bool)
|
||||
if !masterRelay {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_RELAY_UNSUPPORTED", "该本地磁盘目标配置为 Agent 直接写入", nil)
|
||||
}
|
||||
cleanKey, keyErr := s.validateArtifactKey(record, task, objectKey, true)
|
||||
if keyErr != nil {
|
||||
return keyErr
|
||||
}
|
||||
checksum = strings.TrimSpace(checksum)
|
||||
checksumBytes, checksumErr := hex.DecodeString(checksum)
|
||||
if size < 0 || size == math.MaxInt64 || checksumErr != nil || len(checksumBytes) != sha256.Size {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "中转上传需要有效的大小和 SHA-256", checksumErr)
|
||||
}
|
||||
if target.QuotaBytes > 0 {
|
||||
usage, usageErr := s.recordRepo.StorageUsage(ctx)
|
||||
if usageErr != nil {
|
||||
return fmt.Errorf("read storage usage: %w", usageErr)
|
||||
}
|
||||
currentUsed := int64(0)
|
||||
for _, item := range usage {
|
||||
if item.StorageTargetID == targetID {
|
||||
currentUsed = item.TotalSize
|
||||
break
|
||||
}
|
||||
}
|
||||
if currentUsed >= target.QuotaBytes || size > target.QuotaBytes-currentUsed {
|
||||
return apperror.BadRequest("BACKUP_STORAGE_QUOTA_EXCEEDED", fmt.Sprintf("超出存储目标配额(当前 %d,新增 %d,配额 %d)", currentUsed, size, target.QuotaBytes), nil)
|
||||
}
|
||||
}
|
||||
provider, err := s.registry.Create(ctx, target.Type, configMap)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create master relay provider: %w", err)
|
||||
}
|
||||
limited := io.LimitReader(reader, size+1)
|
||||
hashed := newHashingReader(limited)
|
||||
metadata := map[string]string{
|
||||
"taskId": fmt.Sprintf("%d", task.ID),
|
||||
"recordId": fmt.Sprintf("%d", record.ID),
|
||||
"sourceNodeId": fmt.Sprintf("%d", node.ID),
|
||||
"transferMode": storage.TransferModeMasterRelay,
|
||||
}
|
||||
if err := provider.Upload(ctx, cleanKey, hashed, size, metadata); err != nil {
|
||||
return errors.Join(fmt.Errorf("relay artifact to master storage: %w", err), provider.Delete(ctx, cleanKey))
|
||||
}
|
||||
if hashed.n != size || !strings.EqualFold(hashed.Sum(), checksum) {
|
||||
deleteErr := provider.Delete(ctx, cleanKey)
|
||||
return errors.Join(fmt.Errorf("relayed artifact integrity mismatch: received %d of %d bytes", hashed.n, size), deleteErr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *AgentService) ensureTaskSpecAccess(ctx context.Context, node *model.Node, task *model.BackupTask) error {
|
||||
if task.NodeID == node.ID {
|
||||
return nil
|
||||
}
|
||||
record, err := s.recordRepo.FindRunningByTaskAndNode(ctx, task.ID, node.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if record == nil {
|
||||
return apperror.Unauthorized("BACKUP_TASK_FORBIDDEN", "任务不属于当前节点", nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AgentRecordUpdate Agent 上报备份记录的最终状态。
|
||||
type AgentRecordUpdate struct {
|
||||
Status string `json:"status"` // running | success | failed
|
||||
FileName string `json:"fileName,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
StorageTargetID uint `json:"storageTargetId,omitempty"`
|
||||
StorageTransferMode string `json:"storageTransferMode,omitempty"`
|
||||
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
|
||||
ErrorMessage string `json:"errorMessage,omitempty"`
|
||||
LogAppend string `json:"logAppend,omitempty"` // 增量日志,追加到 record.log_content
|
||||
Status string `json:"status"` // running | success | failed
|
||||
FileName string `json:"fileName,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
ErrorMessage string `json:"errorMessage,omitempty"`
|
||||
LogAppend string `json:"logAppend,omitempty"` // 增量日志,追加到 record.log_content
|
||||
}
|
||||
|
||||
// UpdateRecord 更新备份记录的状态/日志。Agent 在执行过程中可多次调用。
|
||||
@@ -369,109 +233,14 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
|
||||
if record == nil {
|
||||
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "记录不存在", nil)
|
||||
}
|
||||
// 通过 task.NodeID 判断是否属于当前 agent
|
||||
task, err := s.taskRepo.FindByID(ctx, record.TaskID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if task == nil || !recordBelongsToNode(record, task, node.ID) {
|
||||
if task == nil || task.NodeID != node.ID {
|
||||
return apperror.Unauthorized("BACKUP_RECORD_FORBIDDEN", "记录不属于当前节点", nil)
|
||||
}
|
||||
if isBackupRecordTerminal(record.Status) {
|
||||
return nil
|
||||
}
|
||||
allowedTargets := make(map[uint]struct{})
|
||||
for _, targetID := range collectTargetIDs(task) {
|
||||
allowedTargets[targetID] = struct{}{}
|
||||
}
|
||||
targetCache := make(map[uint]*model.StorageTarget)
|
||||
validateTransferMode := func(targetID uint, transferMode string) (string, error) {
|
||||
if _, ok := allowedTargets[targetID]; !ok {
|
||||
return "", apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
|
||||
}
|
||||
target := targetCache[targetID]
|
||||
if target == nil {
|
||||
var findErr error
|
||||
target, findErr = s.storageRepo.FindByID(ctx, targetID)
|
||||
if findErr != nil {
|
||||
return "", findErr
|
||||
}
|
||||
if target == nil {
|
||||
return "", apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "存储目标不存在", nil)
|
||||
}
|
||||
targetCache[targetID] = target
|
||||
}
|
||||
expectedMode := storage.TransferModeDirect
|
||||
if strings.EqualFold(target.Type, storage.TypeLocalDisk) {
|
||||
var localConfig storage.LocalDiskConfig
|
||||
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &localConfig); err != nil {
|
||||
return "", fmt.Errorf("decrypt storage config: %w", err)
|
||||
}
|
||||
if localConfig.MasterRelay {
|
||||
expectedMode = storage.TransferModeMasterRelay
|
||||
}
|
||||
}
|
||||
if transferMode != "" && transferMode != expectedMode {
|
||||
return "", apperror.BadRequest("AGENT_STORAGE_TRANSFER_MODE_INVALID", "Agent 上报的存储传输模式与目标配置不一致", nil)
|
||||
}
|
||||
return expectedMode, nil
|
||||
}
|
||||
selectedTransferMode := ""
|
||||
if update.StorageTargetID > 0 {
|
||||
if _, ok := allowedTargets[update.StorageTargetID]; !ok {
|
||||
return apperror.Unauthorized("BACKUP_STORAGE_TARGET_FORBIDDEN", "存储目标不属于该任务", nil)
|
||||
}
|
||||
var modeErr error
|
||||
selectedTransferMode, modeErr = validateTransferMode(update.StorageTargetID, update.StorageTransferMode)
|
||||
if modeErr != nil {
|
||||
return modeErr
|
||||
}
|
||||
} else if update.StorageTransferMode != "" {
|
||||
return apperror.BadRequest("AGENT_STORAGE_TRANSFER_MODE_INVALID", "传输模式缺少对应的存储目标", nil)
|
||||
}
|
||||
for index := range update.StorageUploadResults {
|
||||
result := &update.StorageUploadResults[index]
|
||||
expectedMode, err := validateTransferMode(result.StorageTargetID, result.TransferMode)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if result.FileSize < 0 || (result.Status != "" && result.Status != "success" && result.Status != "failed") {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的存储结果无效", nil)
|
||||
}
|
||||
if result.StoragePath != "" {
|
||||
normalizedPath, err := s.validateArtifactKey(record, task, result.StoragePath, expectedMode == storage.TransferModeMasterRelay)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result.StoragePath = normalizedPath
|
||||
}
|
||||
result.TransferMode = expectedMode
|
||||
}
|
||||
if update.StoragePath != "" {
|
||||
if update.StorageTargetID == 0 {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "存储路径缺少对应的存储目标", nil)
|
||||
}
|
||||
cleanStoragePath, err := s.validateArtifactKey(record, task, update.StoragePath, selectedTransferMode == storage.TransferModeMasterRelay)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if update.FileName != "" && path.Base(cleanStoragePath) != update.FileName {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "Agent 上报的文件名与存储路径不一致", nil)
|
||||
}
|
||||
update.StoragePath = cleanStoragePath
|
||||
}
|
||||
if update.Status != "" && update.Status != model.BackupRecordStatusRunning && update.Status != model.BackupRecordStatusSuccess && update.Status != model.BackupRecordStatusFailed {
|
||||
return apperror.BadRequest("BACKUP_RECORD_STATUS_INVALID", "Agent 上报的备份状态无效", nil)
|
||||
}
|
||||
if update.FileSize < 0 || (update.FileName != "" && (path.Base(update.FileName) != update.FileName || strings.Contains(update.FileName, "\\"))) {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的备份文件信息无效", nil)
|
||||
}
|
||||
if update.Checksum != "" {
|
||||
checksumBytes, checksumErr := hex.DecodeString(strings.TrimSpace(update.Checksum))
|
||||
if checksumErr != nil || len(checksumBytes) != sha256.Size {
|
||||
return apperror.BadRequest("AGENT_ARTIFACT_INVALID", "Agent 上报的 SHA-256 无效", checksumErr)
|
||||
}
|
||||
update.Checksum = strings.ToLower(strings.TrimSpace(update.Checksum))
|
||||
}
|
||||
if update.Status != "" {
|
||||
record.Status = update.Status
|
||||
}
|
||||
@@ -487,15 +256,6 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
|
||||
if update.StoragePath != "" {
|
||||
record.StoragePath = update.StoragePath
|
||||
}
|
||||
if update.StorageTargetID > 0 {
|
||||
record.StorageTargetID = update.StorageTargetID
|
||||
record.StorageTransferMode = selectedTransferMode
|
||||
}
|
||||
if len(update.StorageUploadResults) > 0 {
|
||||
if resultsJSON, marshalErr := json.Marshal(update.StorageUploadResults); marshalErr == nil {
|
||||
record.StorageUploadResults = string(resultsJSON)
|
||||
}
|
||||
}
|
||||
if update.ErrorMessage != "" {
|
||||
record.ErrorMessage = update.ErrorMessage
|
||||
}
|
||||
@@ -517,46 +277,11 @@ func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recor
|
||||
// 同步更新任务的 last_status
|
||||
if update.Status == model.BackupRecordStatusSuccess || update.Status == model.BackupRecordStatusFailed {
|
||||
task.LastStatus = update.Status
|
||||
task.LastRunAt = &record.StartedAt
|
||||
if err := s.taskRepo.Update(ctx, task); err != nil {
|
||||
return fmt.Errorf("update backup task summary: %w", err)
|
||||
}
|
||||
_ = s.taskRepo.Update(ctx, task)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *AgentService) validateArtifactKey(record *model.BackupRecord, task *model.BackupTask, objectKey string, requireRecordNamespace bool) (string, error) {
|
||||
if record == nil || task == nil {
|
||||
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "无法确认中转对象归属", nil)
|
||||
}
|
||||
rawKey := objectKey
|
||||
cleanKey := path.Clean(rawKey)
|
||||
fileName := path.Base(cleanKey)
|
||||
if rawKey == "" || strings.TrimSpace(rawKey) != rawKey || cleanKey == "." || path.IsAbs(cleanKey) || strings.HasPrefix(cleanKey, "../") || cleanKey != rawKey || strings.Contains(rawKey, "\\") || fileName == "." || fileName == "/" {
|
||||
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "中转上传对象路径不安全", nil)
|
||||
}
|
||||
expectedKey := backup.BuildRecordStorageKey(task.Type, record.StartedAt, record.ID, fileName)
|
||||
if requireRecordNamespace {
|
||||
legacyKey := backup.BuildStorageKey(task.Type, record.StartedAt, fileName)
|
||||
if cleanKey != expectedKey && cleanKey != legacyKey {
|
||||
return "", apperror.BadRequest("AGENT_ARTIFACT_INVALID_PATH", "中转上传对象不属于当前备份记录", nil)
|
||||
}
|
||||
return expectedKey, nil
|
||||
}
|
||||
return cleanKey, nil
|
||||
}
|
||||
|
||||
func recordBelongsToNode(record *model.BackupRecord, task *model.BackupTask, nodeID uint) bool {
|
||||
if record.NodeID != 0 {
|
||||
return record.NodeID == nodeID
|
||||
}
|
||||
return task.NodeID == nodeID
|
||||
}
|
||||
|
||||
func isBackupRecordTerminal(status string) bool {
|
||||
return status == model.BackupRecordStatusSuccess || status == model.BackupRecordStatusFailed
|
||||
}
|
||||
|
||||
// EnqueueCommand Master 端调用:给指定节点插入一条待执行命令。
|
||||
// 返回命令 ID。
|
||||
func (s *AgentService) EnqueueCommand(ctx context.Context, nodeID uint, cmdType string, payload any) (uint, error) {
|
||||
@@ -631,84 +356,25 @@ func (s *AgentService) StartCommandTimeoutMonitor(ctx context.Context, interval
|
||||
}()
|
||||
}
|
||||
|
||||
// processStaleCommands 扫描已超时的 pending/dispatched 命令并联动关联记录。
|
||||
// 流程:先取超时候选 → 条件式把命令置为 timeout → 对抢到的命令联动 backup/restore 记录。
|
||||
// processStaleCommands 扫描已超时的 dispatched 命令并联动关联记录。
|
||||
// 流程:先取超时候选 → 对每条联动 backup/restore 记录 → 把命令置为 timeout。
|
||||
// 单条失败不影响后续处理。
|
||||
func (s *AgentService) processStaleCommands(ctx context.Context, threshold time.Time) {
|
||||
commands, err := s.cmdRepo.ListStaleActive(ctx, threshold)
|
||||
commands, err := s.cmdRepo.ListStaleDispatched(ctx, threshold)
|
||||
if err != nil || len(commands) == 0 {
|
||||
return
|
||||
}
|
||||
for i := range commands {
|
||||
cmd := commands[i]
|
||||
if s.commandStillActive(ctx, &cmd, threshold) {
|
||||
continue
|
||||
}
|
||||
s.failLinkedRecord(ctx, &cmd)
|
||||
now := time.Now().UTC()
|
||||
cmd.Status = model.AgentCommandStatusTimeout
|
||||
cmd.ErrorMessage = "agent did not report result before timeout"
|
||||
cmd.CompletedAt = &now
|
||||
timedOut, err := s.cmdRepo.TimeoutActive(ctx, &cmd)
|
||||
if err != nil || !timedOut {
|
||||
continue
|
||||
}
|
||||
s.failLinkedRecord(ctx, &cmd)
|
||||
_ = s.cmdRepo.Update(ctx, &cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// commandStillActive 用关联记录状态、记录更新时间和节点心跳作为长任务续租信号。
|
||||
// 仅 run_task / restore_record 允许续租,避免短 RPC 命令被在线节点长期保留。
|
||||
func (s *AgentService) commandStillActive(ctx context.Context, cmd *model.AgentCommand, threshold time.Time) bool {
|
||||
if cmd.Status != model.AgentCommandStatusDispatched {
|
||||
return false
|
||||
}
|
||||
switch cmd.Type {
|
||||
case model.AgentCommandTypeRunTask:
|
||||
var payload struct {
|
||||
RecordID uint `json:"recordId"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(cmd.Payload), &payload); err != nil || payload.RecordID == 0 {
|
||||
return false
|
||||
}
|
||||
record, err := s.recordRepo.FindByID(ctx, payload.RecordID)
|
||||
if err != nil || record == nil || record.Status != model.BackupRecordStatusRunning {
|
||||
return false
|
||||
}
|
||||
if s.nodeRecentlySeen(ctx, cmd.NodeID, threshold) {
|
||||
return true
|
||||
}
|
||||
return record.UpdatedAt.After(threshold)
|
||||
case model.AgentCommandTypeRestoreRecord:
|
||||
if s.restoreRepo == nil {
|
||||
return false
|
||||
}
|
||||
var payload struct {
|
||||
RestoreRecordID uint `json:"restoreRecordId"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(cmd.Payload), &payload); err != nil || payload.RestoreRecordID == 0 {
|
||||
return false
|
||||
}
|
||||
restore, err := s.restoreRepo.FindByID(ctx, payload.RestoreRecordID)
|
||||
if err != nil || restore == nil || restore.Status != model.RestoreRecordStatusRunning {
|
||||
return false
|
||||
}
|
||||
if s.nodeRecentlySeen(ctx, cmd.NodeID, threshold) {
|
||||
return true
|
||||
}
|
||||
return restore.UpdatedAt.After(threshold)
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func (s *AgentService) nodeRecentlySeen(ctx context.Context, nodeID uint, threshold time.Time) bool {
|
||||
node, err := s.nodeRepo.FindByID(ctx, nodeID)
|
||||
if err != nil || node == nil {
|
||||
return false
|
||||
}
|
||||
return node.Status == model.NodeStatusOnline && node.LastSeen.After(threshold)
|
||||
}
|
||||
|
||||
// failLinkedRecord 根据命令类型把关联记录标记为 failed。
|
||||
// 只对仍然处于 running 状态的记录生效,避免覆盖已完成的结果。
|
||||
func (s *AgentService) failLinkedRecord(ctx context.Context, cmd *model.AgentCommand) {
|
||||
|
||||
@@ -1,824 +0,0 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/backup"
|
||||
"backupx/server/internal/config"
|
||||
"backupx/server/internal/database"
|
||||
"backupx/server/internal/logger"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
"backupx/server/internal/storage"
|
||||
"backupx/server/internal/storage/codec"
|
||||
storageRclone "backupx/server/internal/storage/rclone"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
func newAgentServicePoolTestHarness(t *testing.T) (*AgentService, *gorm.DB, repository.BackupRecordRepository, repository.AgentCommandRepository, *model.Node, *model.Node) {
|
||||
t.Helper()
|
||||
log, err := logger.New(config.LogConfig{Level: "error"})
|
||||
if err != nil {
|
||||
t.Fatalf("logger.New returned error: %v", err)
|
||||
}
|
||||
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(t.TempDir(), "backupx.db")}, log)
|
||||
if err != nil {
|
||||
t.Fatalf("database.Open returned error: %v", err)
|
||||
}
|
||||
cipher := codec.NewConfigCipher("agent-service-secret")
|
||||
nodeRepo := repository.NewNodeRepository(db)
|
||||
taskRepo := repository.NewBackupTaskRepository(db)
|
||||
recordRepo := repository.NewBackupRecordRepository(db)
|
||||
storageRepo := repository.NewStorageTargetRepository(db)
|
||||
cmdRepo := repository.NewAgentCommandRepository(db)
|
||||
|
||||
owner := &model.Node{Name: "edge-owner", Token: "owner-token", Status: model.NodeStatusOnline, IsLocal: false, LastSeen: time.Now().UTC()}
|
||||
other := &model.Node{Name: "edge-other", Token: "other-token", Status: model.NodeStatusOnline, IsLocal: false, LastSeen: time.Now().UTC()}
|
||||
if err := nodeRepo.Create(context.Background(), owner); err != nil {
|
||||
t.Fatalf("create owner node: %v", err)
|
||||
}
|
||||
if err := nodeRepo.Create(context.Background(), other); err != nil {
|
||||
t.Fatalf("create other node: %v", err)
|
||||
}
|
||||
targetConfig, err := cipher.EncryptJSON(map[string]any{"basePath": t.TempDir(), "masterRelay": true})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON returned error: %v", err)
|
||||
}
|
||||
target := &model.StorageTarget{Name: "local", Type: "local_disk", Enabled: true, ConfigCiphertext: targetConfig, ConfigVersion: 1, LastTestStatus: "unknown"}
|
||||
if err := storageRepo.Create(context.Background(), target); err != nil {
|
||||
t.Fatalf("create storage target: %v", err)
|
||||
}
|
||||
task := &model.BackupTask{
|
||||
Name: "pooled-task",
|
||||
Type: "file",
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/data",
|
||||
StorageTargetID: target.ID,
|
||||
NodeID: 0,
|
||||
NodePoolTag: "db",
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
LastStatus: "running",
|
||||
}
|
||||
if err := taskRepo.Create(context.Background(), task); err != nil {
|
||||
t.Fatalf("create task: %v", err)
|
||||
}
|
||||
record := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: target.ID,
|
||||
NodeID: owner.ID,
|
||||
Status: model.BackupRecordStatusRunning,
|
||||
StartedAt: time.Now().UTC(),
|
||||
}
|
||||
if err := recordRepo.Create(context.Background(), record); err != nil {
|
||||
t.Fatalf("create record: %v", err)
|
||||
}
|
||||
storageRegistry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
|
||||
return NewAgentService(nodeRepo, taskRepo, recordRepo, storageRepo, cmdRepo, cipher, storageRegistry), db, recordRepo, cmdRepo, owner, other
|
||||
}
|
||||
|
||||
func TestAgentServicePooledTaskUsesRecordNodeForSpecAndRecordUpdates(t *testing.T) {
|
||||
svc, _, records, _, owner, other := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
|
||||
spec, err := svc.GetTaskSpec(ctx, owner, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("owner GetTaskSpec returned error: %v", err)
|
||||
}
|
||||
if spec.TaskID != 1 || len(spec.StorageTargets) != 1 {
|
||||
t.Fatalf("unexpected spec: %#v", spec)
|
||||
}
|
||||
if spec.StorageTargets[0].TransferMode != storage.TransferModeMasterRelay {
|
||||
t.Fatalf("expected local disk to use Master relay, got %#v", spec.StorageTargets[0])
|
||||
}
|
||||
if _, err := svc.GetTaskSpec(ctx, other, 1); err == nil {
|
||||
t.Fatal("expected non-owner node to be forbidden from pooled task spec")
|
||||
}
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
storagePath := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "backup.tar.gz")
|
||||
|
||||
if err := svc.UpdateRecord(ctx, owner, 1, AgentRecordUpdate{
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "backup.tar.gz",
|
||||
FileSize: 123,
|
||||
StoragePath: storagePath,
|
||||
StorageTargetID: 1,
|
||||
StorageTransferMode: storage.TransferModeMasterRelay,
|
||||
StorageUploadResults: []StorageUploadResultItem{
|
||||
{StorageTargetID: 1, StorageTargetName: "local", Status: "success", StoragePath: storagePath, FileSize: 123, TransferMode: storage.TransferModeMasterRelay},
|
||||
},
|
||||
}); err != nil {
|
||||
t.Fatalf("owner UpdateRecord returned error: %v", err)
|
||||
}
|
||||
updated, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID returned error: %v", err)
|
||||
}
|
||||
if updated.Status != model.BackupRecordStatusSuccess || updated.NodeID != owner.ID {
|
||||
t.Fatalf("unexpected updated record: %#v", updated)
|
||||
}
|
||||
if updated.StorageTargetID != 1 {
|
||||
t.Fatalf("expected successful storage target id 1, got %d", updated.StorageTargetID)
|
||||
}
|
||||
if updated.StorageTransferMode != storage.TransferModeMasterRelay {
|
||||
t.Fatalf("expected Master relay transfer mode, got %q", updated.StorageTransferMode)
|
||||
}
|
||||
if !strings.Contains(updated.StorageUploadResults, `"storageTargetName":"local"`) {
|
||||
t.Fatalf("expected upload results to be persisted, got %q", updated.StorageUploadResults)
|
||||
}
|
||||
if err := svc.UpdateRecord(ctx, other, 1, AgentRecordUpdate{LogAppend: "bad"}); err == nil {
|
||||
t.Fatal("expected non-owner node to be forbidden from record update")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceRelaysRemoteArtifactToMasterLocalDisk(t *testing.T) {
|
||||
svc, _, records, _, owner, other := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
payload := []byte("artifact from remote source server")
|
||||
digest := sha256.Sum256(payload)
|
||||
checksum := fmt.Sprintf("%x", digest[:])
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
objectKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "remote-source.tar")
|
||||
|
||||
if err := svc.UploadArtifact(ctx, owner, 1, 1, objectKey, int64(len(payload)), checksum, bytes.NewReader(payload)); err != nil {
|
||||
t.Fatalf("UploadArtifact returned error: %v", err)
|
||||
}
|
||||
target, err := svc.storageRepo.FindByID(ctx, 1)
|
||||
if err != nil || target == nil {
|
||||
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
|
||||
}
|
||||
config := map[string]any{}
|
||||
if err := svc.cipher.DecryptJSON(target.ConfigCiphertext, &config); err != nil {
|
||||
t.Fatalf("DecryptJSON target config: %v", err)
|
||||
}
|
||||
basePath, _ := config["basePath"].(string)
|
||||
stored, err := os.ReadFile(filepath.Join(basePath, filepath.FromSlash(objectKey)))
|
||||
if err != nil {
|
||||
t.Fatalf("read relayed artifact: %v", err)
|
||||
}
|
||||
if !bytes.Equal(stored, payload) {
|
||||
t.Fatalf("relayed artifact differs: got %q", stored)
|
||||
}
|
||||
if err := svc.UploadArtifact(ctx, other, 1, 1, objectKey, int64(len(payload)), checksum, bytes.NewReader(payload)); err == nil {
|
||||
t.Fatal("expected a different node to be forbidden from relaying the artifact")
|
||||
}
|
||||
|
||||
legacyPayload := []byte("artifact from an older Agent")
|
||||
legacyDigest := sha256.Sum256(legacyPayload)
|
||||
legacyKey := backup.BuildStorageKey("file", record.StartedAt, "legacy-agent.tar")
|
||||
canonicalKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "legacy-agent.tar")
|
||||
if err := svc.UploadArtifact(ctx, owner, record.ID, target.ID, legacyKey, int64(len(legacyPayload)), fmt.Sprintf("%x", legacyDigest[:]), bytes.NewReader(legacyPayload)); err != nil {
|
||||
t.Fatalf("UploadArtifact legacy key returned error: %v", err)
|
||||
}
|
||||
stored, err = os.ReadFile(filepath.Join(basePath, filepath.FromSlash(canonicalKey)))
|
||||
if err != nil || !bytes.Equal(stored, legacyPayload) {
|
||||
t.Fatalf("legacy Agent artifact was not normalized: data=%q err=%v", stored, err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(basePath, filepath.FromSlash(legacyKey))); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatalf("legacy object key should not be written directly: %v", err)
|
||||
}
|
||||
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "legacy-agent.tar",
|
||||
FileSize: int64(len(legacyPayload)),
|
||||
Checksum: fmt.Sprintf("%x", legacyDigest[:]),
|
||||
StoragePath: legacyKey,
|
||||
StorageTargetID: target.ID,
|
||||
StorageUploadResults: []StorageUploadResultItem{{
|
||||
StorageTargetID: target.ID,
|
||||
Status: "success",
|
||||
StoragePath: legacyKey,
|
||||
FileSize: int64(len(legacyPayload)),
|
||||
}},
|
||||
}); err != nil {
|
||||
t.Fatalf("UpdateRecord legacy key returned error: %v", err)
|
||||
}
|
||||
updated, err := records.FindByID(ctx, record.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID updated record returned error: %v", err)
|
||||
}
|
||||
if updated.StoragePath != canonicalKey || updated.StorageTransferMode != storage.TransferModeMasterRelay || !strings.Contains(updated.StorageUploadResults, canonicalKey) {
|
||||
t.Fatalf("legacy Agent record was not normalized: %#v", updated)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceRejectsArtifactOutsideRecordNamespace(t *testing.T) {
|
||||
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
target, err := svc.storageRepo.FindByID(ctx, 1)
|
||||
if err != nil || target == nil {
|
||||
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
|
||||
}
|
||||
config := map[string]any{}
|
||||
if err := svc.cipher.DecryptJSON(target.ConfigCiphertext, &config); err != nil {
|
||||
t.Fatalf("DecryptJSON target config: %v", err)
|
||||
}
|
||||
basePath, _ := config["basePath"].(string)
|
||||
victimKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID+1, "victim.tar")
|
||||
victimPath := filepath.Join(basePath, filepath.FromSlash(victimKey))
|
||||
if err := os.MkdirAll(filepath.Dir(victimPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll victim parent: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(victimPath, []byte("keep me"), 0o600); err != nil {
|
||||
t.Fatalf("WriteFile victim: %v", err)
|
||||
}
|
||||
payload := []byte("overwrite")
|
||||
digest := sha256.Sum256(payload)
|
||||
if err := svc.UploadArtifact(ctx, owner, record.ID, target.ID, victimKey, int64(len(payload)), fmt.Sprintf("%x", digest[:]), bytes.NewReader(payload)); err == nil {
|
||||
t.Fatal("expected another record namespace to be rejected")
|
||||
}
|
||||
stored, err := os.ReadFile(victimPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile victim: %v", err)
|
||||
}
|
||||
if string(stored) != "keep me" {
|
||||
t.Fatalf("victim object changed: %q", stored)
|
||||
}
|
||||
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{StoragePath: victimKey, StorageTargetID: target.ID, StorageTransferMode: storage.TransferModeMasterRelay}); err == nil {
|
||||
t.Fatal("expected another record namespace in status update to be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceKeepsExistingLocalDiskTargetsAgentLocal(t *testing.T) {
|
||||
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
target, err := svc.storageRepo.FindByID(ctx, 1)
|
||||
if err != nil || target == nil {
|
||||
t.Fatalf("FindByID target: target=%#v err=%v", target, err)
|
||||
}
|
||||
legacyConfig, err := svc.cipher.EncryptJSON(map[string]any{"basePath": t.TempDir()})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON legacy target: %v", err)
|
||||
}
|
||||
target.ConfigCiphertext = legacyConfig
|
||||
if err := svc.storageRepo.Update(ctx, target); err != nil {
|
||||
t.Fatalf("Update legacy target: %v", err)
|
||||
}
|
||||
|
||||
spec, err := svc.GetTaskSpec(ctx, owner, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("GetTaskSpec returned error: %v", err)
|
||||
}
|
||||
if len(spec.StorageTargets) != 1 || spec.StorageTargets[0].TransferMode != storage.TransferModeDirect {
|
||||
t.Fatalf("expected legacy local disk to stay Agent-local, got %#v", spec.StorageTargets)
|
||||
}
|
||||
payload := []byte("must not be relayed")
|
||||
digest := sha256.Sum256(payload)
|
||||
record, findErr := records.FindByID(ctx, 1)
|
||||
if findErr != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", findErr)
|
||||
}
|
||||
objectKey := backup.BuildRecordStorageKey("file", record.StartedAt, record.ID, "legacy.tar")
|
||||
err = svc.UploadArtifact(ctx, owner, 1, 1, objectKey, int64(len(payload)), fmt.Sprintf("%x", digest[:]), bytes.NewReader(payload))
|
||||
if err == nil {
|
||||
t.Fatal("expected relay upload to be rejected for an Agent-local target")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceUpdateRecordRefreshesTaskSummaryOnTerminalStatus(t *testing.T) {
|
||||
for _, status := range []string{model.BackupRecordStatusSuccess, model.BackupRecordStatusFailed} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
|
||||
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{Status: status}); err != nil {
|
||||
t.Fatalf("UpdateRecord returned error: %v", err)
|
||||
}
|
||||
|
||||
task, err := svc.taskRepo.FindByID(ctx, record.TaskID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
if task.LastStatus != status {
|
||||
t.Fatalf("expected task LastStatus %q, got %q", status, task.LastStatus)
|
||||
}
|
||||
if task.LastRunAt == nil || !task.LastRunAt.Equal(record.StartedAt) {
|
||||
t.Fatalf("expected task LastRunAt to match record startedAt %s, got %#v", record.StartedAt, task.LastRunAt)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceUpdateRecordReturnsTaskSummaryUpdateError(t *testing.T) {
|
||||
svc, _, _, _, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
expectedErr := errors.New("task update failed")
|
||||
svc.taskRepo = &failingUpdateTaskRepo{
|
||||
BackupTaskRepository: svc.taskRepo,
|
||||
err: expectedErr,
|
||||
}
|
||||
|
||||
err := svc.UpdateRecord(ctx, owner, 1, AgentRecordUpdate{Status: model.BackupRecordStatusSuccess})
|
||||
if !errors.Is(err, expectedErr) {
|
||||
t.Fatalf("expected task update error %v, got %v", expectedErr, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsFailsPendingRunTaskRecord(t *testing.T) {
|
||||
svc, _, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusPending,
|
||||
Payload: `{"recordId":1}`,
|
||||
CreatedAt: time.Now().UTC().Add(-time.Hour),
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected command timeout, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRecord, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if updatedRecord.Status != model.BackupRecordStatusFailed {
|
||||
t.Fatalf("expected record failed, got %#v", updatedRecord)
|
||||
}
|
||||
if updatedRecord.CompletedAt == nil {
|
||||
t.Fatal("expected failed record completedAt to be set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsFailsPendingRestoreRecord(t *testing.T) {
|
||||
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
restoreRepo := repository.NewRestoreRecordRepository(db)
|
||||
restore := &model.RestoreRecord{
|
||||
BackupRecordID: 1,
|
||||
TaskID: 1,
|
||||
NodeID: owner.ID,
|
||||
Status: model.RestoreRecordStatusRunning,
|
||||
StartedAt: time.Now().UTC().Add(-time.Hour),
|
||||
}
|
||||
if err := restoreRepo.Create(ctx, restore); err != nil {
|
||||
t.Fatalf("Create restore returned error: %v", err)
|
||||
}
|
||||
svc.SetRestoreRepository(restoreRepo)
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRestoreRecord,
|
||||
Status: model.AgentCommandStatusPending,
|
||||
Payload: `{"restoreRecordId":1}`,
|
||||
CreatedAt: time.Now().UTC().Add(-time.Hour),
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected command timeout, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID restore returned error: %v", err)
|
||||
}
|
||||
if updatedRestore.Status != model.RestoreRecordStatusFailed {
|
||||
t.Fatalf("expected restore failed, got %#v", updatedRestore)
|
||||
}
|
||||
if updatedRestore.CompletedAt == nil {
|
||||
t.Fatal("expected failed restore completedAt to be set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsKeepsActiveDispatchedRunTaskRecord(t *testing.T) {
|
||||
svc, _, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"recordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusDispatched {
|
||||
t.Fatalf("expected active command to remain dispatched, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRecord, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if updatedRecord.Status != model.BackupRecordStatusRunning {
|
||||
t.Fatalf("expected active record to remain running, got %#v", updatedRecord)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsKeepsDispatchedRunTaskWhenNodeHeartbeatIsFresh(t *testing.T) {
|
||||
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := setBackupRecordUpdatedAt(db, 1, dispatchedAt); err != nil {
|
||||
t.Fatalf("set backup record updated_at: %v", err)
|
||||
}
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"recordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusDispatched {
|
||||
t.Fatalf("expected command to remain dispatched while node heartbeat is fresh, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRecord, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if updatedRecord.Status != model.BackupRecordStatusRunning {
|
||||
t.Fatalf("expected record to remain running while node heartbeat is fresh, got %#v", updatedRecord)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsTimesOutShortCommandEvenWhenNodeHeartbeatIsFresh(t *testing.T) {
|
||||
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeListDir,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"path":"/srv"}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected stale short command timeout, got %#v", updatedCommand)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsTimesOutDispatchedRunTaskWhenRecordIsTerminalEvenWithFreshHeartbeat(t *testing.T) {
|
||||
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
completedAt := time.Now().UTC().Add(-time.Minute)
|
||||
record.Status = model.BackupRecordStatusFailed
|
||||
record.CompletedAt = &completedAt
|
||||
if err := records.Update(ctx, record); err != nil {
|
||||
t.Fatalf("Update terminal record returned error: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"recordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected command timeout when linked record is terminal, got %#v", updatedCommand)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsTimesOutInactiveDispatchedRunTaskRecord(t *testing.T) {
|
||||
svc, db, records, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := setBackupRecordUpdatedAt(db, 1, dispatchedAt); err != nil {
|
||||
t.Fatalf("set backup record updated_at: %v", err)
|
||||
}
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", dispatchedAt).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"recordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected inactive command timeout, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRecord, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if updatedRecord.Status != model.BackupRecordStatusFailed {
|
||||
t.Fatalf("expected inactive record failed, got %#v", updatedRecord)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsKeepsActiveDispatchedRestoreRecord(t *testing.T) {
|
||||
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
restoreRepo := repository.NewRestoreRecordRepository(db)
|
||||
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
|
||||
svc.SetRestoreRepository(restoreRepo)
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRestoreRecord,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"restoreRecordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusDispatched {
|
||||
t.Fatalf("expected active restore command to remain dispatched, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID restore returned error: %v", err)
|
||||
}
|
||||
if updatedRestore.Status != model.RestoreRecordStatusRunning {
|
||||
t.Fatalf("expected active restore to remain running, got %#v", updatedRestore)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsKeepsDispatchedRestoreWhenNodeHeartbeatIsFresh(t *testing.T) {
|
||||
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
restoreRepo := repository.NewRestoreRecordRepository(db)
|
||||
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
|
||||
svc.SetRestoreRepository(restoreRepo)
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := setRestoreRecordUpdatedAt(db, restore.ID, dispatchedAt); err != nil {
|
||||
t.Fatalf("set restore record updated_at: %v", err)
|
||||
}
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", time.Now().UTC()).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRestoreRecord,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"restoreRecordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusDispatched {
|
||||
t.Fatalf("expected restore command to remain dispatched while node heartbeat is fresh, got %#v", updatedCommand)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceProcessStaleCommandsTimesOutInactiveDispatchedRestoreRecord(t *testing.T) {
|
||||
svc, db, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
restoreRepo := repository.NewRestoreRecordRepository(db)
|
||||
restore := createAgentServiceRestoreRecord(t, restoreRepo, owner.ID)
|
||||
svc.SetRestoreRepository(restoreRepo)
|
||||
dispatchedAt := time.Now().UTC().Add(-time.Hour)
|
||||
if err := setRestoreRecordUpdatedAt(db, restore.ID, dispatchedAt); err != nil {
|
||||
t.Fatalf("set restore record updated_at: %v", err)
|
||||
}
|
||||
if err := db.Model(&model.Node{}).Where("id = ?", owner.ID).UpdateColumn("last_seen", dispatchedAt).Error; err != nil {
|
||||
t.Fatalf("set owner last_seen: %v", err)
|
||||
}
|
||||
oldCommand := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRestoreRecord,
|
||||
Status: model.AgentCommandStatusDispatched,
|
||||
Payload: `{"restoreRecordId":1}`,
|
||||
CreatedAt: dispatchedAt,
|
||||
DispatchedAt: &dispatchedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, oldCommand); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
svc.processStaleCommands(ctx, time.Now().UTC().Add(-30*time.Minute))
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, oldCommand.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected inactive restore command timeout, got %#v", updatedCommand)
|
||||
}
|
||||
updatedRestore, err := restoreRepo.FindByID(ctx, restore.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID restore returned error: %v", err)
|
||||
}
|
||||
if updatedRestore.Status != model.RestoreRecordStatusFailed {
|
||||
t.Fatalf("expected inactive restore failed, got %#v", updatedRestore)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceSubmitCommandResultDoesNotOverwriteTerminalCommand(t *testing.T) {
|
||||
svc, _, _, commands, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
completedAt := time.Now().UTC().Add(-time.Minute)
|
||||
command := &model.AgentCommand{
|
||||
NodeID: owner.ID,
|
||||
Type: model.AgentCommandTypeRunTask,
|
||||
Status: model.AgentCommandStatusTimeout,
|
||||
Payload: `{"recordId":1}`,
|
||||
ErrorMessage: "timeout",
|
||||
CompletedAt: &completedAt,
|
||||
}
|
||||
if err := commands.Create(ctx, command); err != nil {
|
||||
t.Fatalf("Create command returned error: %v", err)
|
||||
}
|
||||
|
||||
if err := svc.SubmitCommandResult(ctx, owner, command.ID, AgentCommandResult{Success: true, Result: []byte(`{"ok":true}`)}); err != nil {
|
||||
t.Fatalf("SubmitCommandResult returned error: %v", err)
|
||||
}
|
||||
|
||||
updatedCommand, err := commands.FindByID(ctx, command.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID command returned error: %v", err)
|
||||
}
|
||||
if updatedCommand.Status != model.AgentCommandStatusTimeout {
|
||||
t.Fatalf("expected terminal command status to remain timeout, got %#v", updatedCommand)
|
||||
}
|
||||
if updatedCommand.Result != "" {
|
||||
t.Fatalf("expected terminal command result to remain empty, got %q", updatedCommand.Result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAgentServiceUpdateRecordDoesNotOverwriteTerminalRecord(t *testing.T) {
|
||||
svc, _, records, _, owner, _ := newAgentServicePoolTestHarness(t)
|
||||
ctx := context.Background()
|
||||
record, err := records.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
completedAt := time.Now().UTC().Add(-time.Minute)
|
||||
record.Status = model.BackupRecordStatusFailed
|
||||
record.ErrorMessage = "timeout"
|
||||
record.CompletedAt = &completedAt
|
||||
if err := records.Update(ctx, record); err != nil {
|
||||
t.Fatalf("Update record returned error: %v", err)
|
||||
}
|
||||
|
||||
if err := svc.UpdateRecord(ctx, owner, record.ID, AgentRecordUpdate{
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "late.tar.gz",
|
||||
FileSize: 42,
|
||||
Checksum: "late",
|
||||
StoragePath: "late/path",
|
||||
ErrorMessage: "late success",
|
||||
LogAppend: "late log\n",
|
||||
}); err != nil {
|
||||
t.Fatalf("UpdateRecord returned error: %v", err)
|
||||
}
|
||||
|
||||
updatedRecord, err := records.FindByID(ctx, record.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID updated record returned error: %v", err)
|
||||
}
|
||||
if updatedRecord.Status != model.BackupRecordStatusFailed {
|
||||
t.Fatalf("expected terminal record status to remain failed, got %#v", updatedRecord)
|
||||
}
|
||||
if updatedRecord.FileName != "" || updatedRecord.StoragePath != "" || updatedRecord.ErrorMessage != "timeout" {
|
||||
t.Fatalf("expected terminal record fields to remain unchanged, got %#v", updatedRecord)
|
||||
}
|
||||
}
|
||||
|
||||
func createAgentServiceRestoreRecord(t *testing.T, repo repository.RestoreRecordRepository, nodeID uint) *model.RestoreRecord {
|
||||
t.Helper()
|
||||
restore := &model.RestoreRecord{
|
||||
BackupRecordID: 1,
|
||||
TaskID: 1,
|
||||
NodeID: nodeID,
|
||||
Status: model.RestoreRecordStatusRunning,
|
||||
StartedAt: time.Now().UTC().Add(-time.Hour),
|
||||
}
|
||||
if err := repo.Create(context.Background(), restore); err != nil {
|
||||
t.Fatalf("Create restore returned error: %v", err)
|
||||
}
|
||||
return restore
|
||||
}
|
||||
|
||||
func setBackupRecordUpdatedAt(db *gorm.DB, id uint, updatedAt time.Time) error {
|
||||
return db.Model(&model.BackupRecord{}).Where("id = ?", id).UpdateColumn("updated_at", updatedAt).Error
|
||||
}
|
||||
|
||||
func setRestoreRecordUpdatedAt(db *gorm.DB, id uint, updatedAt time.Time) error {
|
||||
return db.Model(&model.RestoreRecord{}).Where("id = ?", id).UpdateColumn("updated_at", updatedAt).Error
|
||||
}
|
||||
|
||||
type failingUpdateTaskRepo struct {
|
||||
repository.BackupTaskRepository
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *failingUpdateTaskRepo) Update(context.Context, *model.BackupTask) error {
|
||||
return r.err
|
||||
}
|
||||
@@ -1,86 +0,0 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/config"
|
||||
"backupx/server/internal/database"
|
||||
"backupx/server/internal/logger"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
)
|
||||
|
||||
func TestAuditRetention(t *testing.T) {
|
||||
baseDir := t.TempDir()
|
||||
log, err := logger.New(config.LogConfig{Level: "error"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(baseDir, "a.db")}, log)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
auditRepo := repository.NewAuditLogRepository(db)
|
||||
configRepo := repository.NewSystemConfigRepository(db)
|
||||
svc := NewAuditService(auditRepo)
|
||||
ctx := context.Background()
|
||||
|
||||
// seed 写一条审计日志并把 created_at 强制改到 daysAgo 天前。
|
||||
seed := func(daysAgo int) {
|
||||
rec := &model.AuditLog{Username: "t", Category: "test", Action: "seed"}
|
||||
if err := auditRepo.Create(ctx, rec); err != nil {
|
||||
t.Fatalf("create: %v", err)
|
||||
}
|
||||
ts := time.Now().UTC().AddDate(0, 0, -daysAgo)
|
||||
if err := db.Model(&model.AuditLog{}).Where("id = ?", rec.ID).Update("created_at", ts).Error; err != nil {
|
||||
t.Fatalf("backdate: %v", err)
|
||||
}
|
||||
}
|
||||
count := func() int64 {
|
||||
var n int64
|
||||
db.Model(&model.AuditLog{}).Count(&n)
|
||||
return n
|
||||
}
|
||||
|
||||
seed(100)
|
||||
seed(40)
|
||||
seed(0)
|
||||
if count() != 3 {
|
||||
t.Fatalf("expected 3 seeded, got %d", count())
|
||||
}
|
||||
|
||||
// days<=0 不清理。
|
||||
if n, _ := svc.PurgeOlderThan(ctx, 0); n != 0 || count() != 3 {
|
||||
t.Fatalf("days<=0 must not purge (n=%d count=%d)", n, count())
|
||||
}
|
||||
|
||||
// 保留 50 天 → 删除 100 天前那条。
|
||||
n, err := svc.PurgeOlderThan(ctx, 50)
|
||||
if err != nil {
|
||||
t.Fatalf("purge: %v", err)
|
||||
}
|
||||
if n != 1 || count() != 2 {
|
||||
t.Fatalf("expected 1 purged / 2 remaining, got n=%d count=%d", n, count())
|
||||
}
|
||||
|
||||
// 设置驱动:retention=10 天 → 再删 40 天前那条。
|
||||
if err := configRepo.Upsert(ctx, &model.SystemConfig{Key: SettingKeyAuditRetentionDays, Value: "10"}); err != nil {
|
||||
t.Fatalf("upsert setting: %v", err)
|
||||
}
|
||||
svc.runRetentionOnce(ctx, configRepo)
|
||||
if count() != 1 {
|
||||
t.Fatalf("expected 1 remaining after retention=10, got %d", count())
|
||||
}
|
||||
|
||||
// retention=0 → 永久保留,不再删除。
|
||||
if err := configRepo.Upsert(ctx, &model.SystemConfig{Key: SettingKeyAuditRetentionDays, Value: "0"}); err != nil {
|
||||
t.Fatalf("upsert setting: %v", err)
|
||||
}
|
||||
svc.runRetentionOnce(ctx, configRepo)
|
||||
if count() != 1 {
|
||||
t.Fatalf("retention=0 must keep all, got %d", count())
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,6 @@ import (
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -52,59 +51,6 @@ func NewAuditService(repo repository.AuditLogRepository) *AuditService {
|
||||
}
|
||||
}
|
||||
|
||||
// PurgeOlderThan 删除早于 days 天前的审计日志,返回删除条数。days<=0 时不清理。
|
||||
func (s *AuditService) PurgeOlderThan(ctx context.Context, days int) (int64, error) {
|
||||
if days <= 0 {
|
||||
return 0, nil
|
||||
}
|
||||
cutoff := time.Now().UTC().AddDate(0, 0, -days)
|
||||
return s.repo.DeleteBefore(ctx, cutoff)
|
||||
}
|
||||
|
||||
// StartRetentionMonitor 启动后台审计保留期清理:按 interval 周期读取
|
||||
// audit_retention_days 设置,>0 时删除超期审计日志。缺省/0 表示永久保留
|
||||
// (向后兼容,默认不删任何历史)。ctx 取消后退出。
|
||||
func (s *AuditService) StartRetentionMonitor(ctx context.Context, configs repository.SystemConfigRepository, interval time.Duration) {
|
||||
if s == nil || configs == nil {
|
||||
return
|
||||
}
|
||||
if interval <= 0 {
|
||||
interval = 6 * time.Hour
|
||||
}
|
||||
go func() {
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
s.runRetentionOnce(ctx, configs) // 启动后立即跑一次
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.runRetentionOnce(ctx, configs)
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
func (s *AuditService) runRetentionOnce(ctx context.Context, configs repository.SystemConfigRepository) {
|
||||
cfg, err := configs.GetByKey(ctx, SettingKeyAuditRetentionDays)
|
||||
if err != nil || cfg == nil {
|
||||
return
|
||||
}
|
||||
days, err := strconv.Atoi(strings.TrimSpace(cfg.Value))
|
||||
if err != nil || days <= 0 {
|
||||
return
|
||||
}
|
||||
deleted, err := s.PurgeOlderThan(ctx, days)
|
||||
if err != nil {
|
||||
log.Printf("[audit] retention purge failed: %v", err)
|
||||
return
|
||||
}
|
||||
if deleted > 0 {
|
||||
log.Printf("[audit] retention purge: deleted %d logs older than %d days", deleted, days)
|
||||
}
|
||||
}
|
||||
|
||||
// SetWebhook 动态配置审计事件转发 URL 与签名密钥。
|
||||
// - url 为空字符串时禁用转发
|
||||
// - secret 非空时对 payload 计算 HMAC-SHA256,作为 X-BackupX-Signature header
|
||||
|
||||
@@ -46,10 +46,6 @@ func (r *fakeAuditRepo) ListAll(context.Context, repository.AuditLogListOptions)
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *fakeAuditRepo) DeleteBefore(context.Context, time.Time) (int64, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func TestAuditService_WebhookDeliversSignedPayload(t *testing.T) {
|
||||
var hits atomic.Int32
|
||||
var got struct {
|
||||
|
||||
@@ -293,7 +293,7 @@ func (s *AuthService) verifyLoginMFA(ctx context.Context, user *model.User, inpu
|
||||
}
|
||||
|
||||
func (s *AuthService) userBySubject(ctx context.Context, subject string) (*model.User, error) {
|
||||
userID, err := strconv.ParseUint(subject, 10, 0)
|
||||
userID, err := strconv.ParseUint(subject, 10, 64)
|
||||
if err != nil {
|
||||
return nil, apperror.Unauthorized("AUTH_INVALID_SUBJECT", "无效用户身份", err)
|
||||
}
|
||||
|
||||
@@ -5,13 +5,11 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -51,31 +49,14 @@ type StorageUploadResultItem struct {
|
||||
Status string `json:"status"`
|
||||
StoragePath string `json:"storagePath,omitempty"`
|
||||
FileSize int64 `json:"fileSize,omitempty"`
|
||||
TransferMode string `json:"transferMode,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
const (
|
||||
uploadMaxAttempts = 3
|
||||
uploadRetryBackoff = 10 * time.Second
|
||||
)
|
||||
|
||||
type DownloadedArtifact struct {
|
||||
FileName string
|
||||
Reader io.ReadCloser
|
||||
}
|
||||
|
||||
type temporaryArtifactReader struct {
|
||||
*os.File
|
||||
directory string
|
||||
}
|
||||
|
||||
func (r *temporaryArtifactReader) Close() error {
|
||||
closeErr := r.File.Close()
|
||||
removeErr := os.RemoveAll(r.directory)
|
||||
return errors.Join(closeErr, removeErr)
|
||||
}
|
||||
|
||||
// collectTargetIDs 获取任务关联的所有存储目标 ID
|
||||
func collectTargetIDs(task *model.BackupTask) []uint {
|
||||
if len(task.StorageTargets) > 0 {
|
||||
@@ -92,34 +73,29 @@ func collectTargetIDs(task *model.BackupTask) []uint {
|
||||
}
|
||||
|
||||
type BackupExecutionService struct {
|
||||
tasks repository.BackupTaskRepository
|
||||
records repository.BackupRecordRepository
|
||||
targets repository.StorageTargetRepository
|
||||
nodeRepo repository.NodeRepository
|
||||
storageRegistry *storage.Registry
|
||||
runnerRegistry *backup.Registry
|
||||
logHub *backup.LogHub
|
||||
retention *backupretention.Service
|
||||
cipher *codec.ConfigCipher
|
||||
tasks repository.BackupTaskRepository
|
||||
records repository.BackupRecordRepository
|
||||
targets repository.StorageTargetRepository
|
||||
nodeRepo repository.NodeRepository
|
||||
storageRegistry *storage.Registry
|
||||
runnerRegistry *backup.Registry
|
||||
logHub *backup.LogHub
|
||||
retention *backupretention.Service
|
||||
cipher *codec.ConfigCipher
|
||||
notifier BackupResultNotifier
|
||||
agentDispatcher AgentDispatcher
|
||||
replicationHook ReplicationTrigger
|
||||
dependentsResolver DependentsResolver
|
||||
async func(func())
|
||||
now func() time.Time
|
||||
tempDir string
|
||||
semaphore chan struct{}
|
||||
async func(func())
|
||||
now func() time.Time
|
||||
tempDir string
|
||||
semaphore chan struct{}
|
||||
// nodeSemaphores 节点级并发限制(按 NodeID 映射)。
|
||||
// 没命中的 NodeID 走全局 semaphore,节点配置 MaxConcurrent>0 时按该节点独立排队。
|
||||
nodeSemaphores sync.Map
|
||||
retries int // rclone 底层重试次数
|
||||
bandwidthLimit string // rclone 带宽限制(全局默认,节点配置可覆盖)
|
||||
retries int // rclone 底层重试次数
|
||||
bandwidthLimit string // rclone 带宽限制(全局默认,节点配置可覆盖)
|
||||
metrics *metrics.Metrics
|
||||
taskLocks sync.Map
|
||||
// repositoryLocks serializes immutable index updates per storage target.
|
||||
// Repository mode is intentionally single-writer in v1 to avoid orphaned
|
||||
// duplicate packs when two local tasks discover the same missing chunk.
|
||||
repositoryLocks sync.Map
|
||||
}
|
||||
|
||||
// SetMetrics 注入 Prometheus 采集器。nil 时所有埋点退化为 no-op。
|
||||
@@ -229,25 +205,6 @@ func (s *BackupExecutionService) DownloadRecord(ctx context.Context, recordID ui
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if record.BackupKind == model.BackupKindRepository {
|
||||
tempDir, err := os.MkdirTemp(s.tempDir, "repository-download-*")
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法创建 CDC 导出目录", err)
|
||||
}
|
||||
exportName := fmt.Sprintf("backupx-record-%d.tar", record.ID)
|
||||
exportPath := filepath.Join(tempDir, exportName)
|
||||
store := backup.NewRepositoryStore(s.cipher.Key())
|
||||
if err := store.ExportTar(ctx, provider, record.StoragePath, record.Checksum, exportPath); err != nil {
|
||||
cleanupErr := os.RemoveAll(tempDir)
|
||||
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法从 CDC 仓库导出归档", errors.Join(err, cleanupErr))
|
||||
}
|
||||
file, err := os.Open(exportPath)
|
||||
if err != nil {
|
||||
cleanupErr := os.RemoveAll(tempDir)
|
||||
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法打开 CDC 导出归档", errors.Join(err, cleanupErr))
|
||||
}
|
||||
return &DownloadedArtifact{FileName: exportName, Reader: &temporaryArtifactReader{File: file, directory: tempDir}}, nil
|
||||
}
|
||||
reader, err := provider.Download(ctx, record.StoragePath)
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_DOWNLOAD_FAILED", "无法下载备份文件", err)
|
||||
@@ -271,16 +228,6 @@ func (s *BackupExecutionService) RestoreRecord(ctx context.Context, recordID uin
|
||||
if task == nil {
|
||||
return apperror.New(404, "BACKUP_TASK_NOT_FOUND", "关联的备份任务不存在,无法执行恢复", fmt.Errorf("backup task %d not found", record.TaskID))
|
||||
}
|
||||
if record.BackupKind == model.BackupKindRepository {
|
||||
spec, specErr := s.buildTaskSpec(task, record.StartedAt)
|
||||
if specErr != nil {
|
||||
return specErr
|
||||
}
|
||||
if err := backup.NewRepositoryStore(s.cipher.Key()).Restore(ctx, provider, record.StoragePath, record.Checksum, spec, backup.NopLogWriter{}); err != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_RESTORE_FAILED", "从 CDC 仓库恢复备份失败", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
tempDir, err := os.MkdirTemp("", "backupx-restore-*")
|
||||
if err != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_RESTORE_FAILED", "无法创建恢复目录", err)
|
||||
@@ -323,76 +270,11 @@ func (s *BackupExecutionService) DeleteRecord(ctx context.Context, recordID uint
|
||||
if record == nil {
|
||||
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", fmt.Errorf("backup record %d not found", recordID))
|
||||
}
|
||||
if record.Locked {
|
||||
return apperror.BadRequest("BACKUP_RECORD_LOCKED",
|
||||
"该备份已保留锁定(法律保留),请先解锁再删除", nil)
|
||||
}
|
||||
// 差异链保护:禁止删除仍被差异备份依赖的全量,否则这些差异将无法恢复(与保留清理的保护一致)。
|
||||
if record.BackupKind == model.BackupKindFull {
|
||||
deps, depErr := s.records.CountDependentDifferentials(ctx, record.ID)
|
||||
if depErr != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法检查差异备份依赖", depErr)
|
||||
}
|
||||
if deps > 0 {
|
||||
return apperror.BadRequest("BACKUP_RECORD_HAS_DEPENDENTS",
|
||||
fmt.Sprintf("该全量备份仍有 %d 个差异备份依赖它,删除会导致这些差异无法恢复。请先删除相关差异备份或等待其过期。", deps), nil)
|
||||
}
|
||||
}
|
||||
if record.BackupKind == model.BackupKindRepository {
|
||||
copies := []StorageUploadResultItem{{
|
||||
StorageTargetID: record.StorageTargetID,
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
StoragePath: record.StoragePath,
|
||||
}}
|
||||
if strings.TrimSpace(record.StorageUploadResults) != "" {
|
||||
if err := json.Unmarshal([]byte(record.StorageUploadResults), &copies); err != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法解析 CDC 仓库副本信息,已停止删除以避免遗留数据", err)
|
||||
}
|
||||
}
|
||||
copyPaths := make(map[uint]string, len(copies))
|
||||
for _, copy := range copies {
|
||||
if strings.EqualFold(copy.Status, model.BackupRecordStatusSuccess) && strings.TrimSpace(copy.StoragePath) != "" {
|
||||
copyPaths[copy.StorageTargetID] = copy.StoragePath
|
||||
}
|
||||
}
|
||||
targetIDs := make([]uint, 0, len(copyPaths))
|
||||
for targetID := range copyPaths {
|
||||
targetIDs = append(targetIDs, targetID)
|
||||
}
|
||||
sort.Slice(targetIDs, func(i, j int) bool { return targetIDs[i] < targetIDs[j] })
|
||||
unlocks := make([]func(), 0, len(targetIDs))
|
||||
for _, targetID := range targetIDs {
|
||||
unlocks = append(unlocks, s.acquireRepositoryLock(targetID))
|
||||
}
|
||||
defer func() {
|
||||
for index := len(unlocks) - 1; index >= 0; index-- {
|
||||
unlocks[index]()
|
||||
}
|
||||
}()
|
||||
providers := make(map[uint]storage.StorageProvider, len(targetIDs))
|
||||
for _, targetID := range targetIDs {
|
||||
provider, resolveErr := s.resolveProvider(ctx, targetID)
|
||||
if resolveErr != nil {
|
||||
return resolveErr
|
||||
}
|
||||
if deleteErr := provider.Delete(ctx, copyPaths[targetID]); deleteErr != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法删除 CDC 仓库快照", deleteErr)
|
||||
}
|
||||
providers[targetID] = provider
|
||||
}
|
||||
for targetID, provider := range providers {
|
||||
if _, pruneErr := backup.NewRepositoryStore(s.cipher.Key()).Prune(ctx, provider); pruneErr != nil {
|
||||
return apperror.Internal("BACKUP_REPOSITORY_PRUNE_FAILED", fmt.Sprintf("无法清理存储目标 %d 的 CDC 仓库;记录暂时保留以便重试", targetID), pruneErr)
|
||||
}
|
||||
}
|
||||
if err := s.records.Delete(ctx, recordID); err != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_DELETE_FAILED", "无法删除备份记录", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if remote, err := s.deleteRemoteLocalDiskObject(ctx, record); err != nil {
|
||||
// 集群场景保护:跨节点 local_disk 文件 Master 无法远程删除,拒绝操作以避免存储泄漏的错觉
|
||||
if err := s.validateClusterAccessible(ctx, record); err != nil {
|
||||
return err
|
||||
} else if !remote && strings.TrimSpace(record.StoragePath) != "" {
|
||||
}
|
||||
if strings.TrimSpace(record.StoragePath) != "" {
|
||||
provider, err := s.resolveProvider(ctx, record.StorageTargetID)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -407,50 +289,33 @@ func (s *BackupExecutionService) DeleteRecord(ctx context.Context, recordID uint
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) deleteRemoteLocalDiskObject(ctx context.Context, record *model.BackupRecord) (bool, error) {
|
||||
if strings.TrimSpace(record.StoragePath) == "" || s.nodeRepo == nil {
|
||||
return false, nil
|
||||
}
|
||||
if record.StorageTransferMode == storage.TransferModeMasterRelay {
|
||||
return false, nil
|
||||
}
|
||||
node, err := s.nodeRepo.FindByID(ctx, record.NodeID)
|
||||
if err != nil || node == nil || node.IsLocal {
|
||||
return false, nil
|
||||
}
|
||||
target, err := s.targets.FindByID(ctx, record.StorageTargetID)
|
||||
if err != nil {
|
||||
return false, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
|
||||
}
|
||||
if target == nil || !strings.EqualFold(target.Type, "local_disk") {
|
||||
return false, nil
|
||||
}
|
||||
if s.agentDispatcher == nil {
|
||||
return true, apperror.BadRequest("BACKUP_RECORD_CROSS_NODE_LOCAL_DISK",
|
||||
fmt.Sprintf("该备份位于节点 %s 的本地磁盘(local_disk),Master 无法跨节点删除。请确保 Agent 在线后再操作。", node.Name),
|
||||
nil)
|
||||
}
|
||||
configMap := map[string]any{}
|
||||
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
|
||||
return true, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
|
||||
}
|
||||
if _, err := s.agentDispatcher.EnqueueCommand(ctx, record.NodeID, model.AgentCommandTypeDeleteStorageObject, map[string]any{
|
||||
"targetType": target.Type,
|
||||
"targetConfig": configMap,
|
||||
"storagePath": record.StoragePath,
|
||||
}); err != nil {
|
||||
return true, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "无法下发远程备份文件删除命令", err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// validateClusterAccessible 在跨节点 + local_disk 场景下拒绝 Master 端直接访问。
|
||||
// 场景说明:远程 Agent 把备份写到其本机磁盘(local_disk basePath)时,Master 的
|
||||
// provider 指向的是 Master 本机的同名路径,访问会静默取错文件或 404。明确拒绝
|
||||
// 让用户知情,避免假成功。
|
||||
func (s *BackupExecutionService) validateClusterAccessible(ctx context.Context, record *model.BackupRecord) error {
|
||||
return validateCrossNodeLocalDisk(ctx, s.nodeRepo, s.targets, record,
|
||||
"BACKUP_RECORD_CROSS_NODE_LOCAL_DISK", "访问。请登录该节点或改用云存储后再操作")
|
||||
if record == nil || record.NodeID == 0 {
|
||||
return nil
|
||||
}
|
||||
// 检查是否为远程节点
|
||||
if s.nodeRepo == nil {
|
||||
return nil
|
||||
}
|
||||
node, err := s.nodeRepo.FindByID(ctx, record.NodeID)
|
||||
if err != nil || node == nil || node.IsLocal {
|
||||
return nil
|
||||
}
|
||||
// 检查存储类型是否为 local_disk(跨节点不可达)
|
||||
target, err := s.targets.FindByID(ctx, record.StorageTargetID)
|
||||
if err != nil || target == nil {
|
||||
return nil
|
||||
}
|
||||
if strings.EqualFold(target.Type, "local_disk") {
|
||||
return apperror.BadRequest("BACKUP_RECORD_CROSS_NODE_LOCAL_DISK",
|
||||
fmt.Sprintf("该备份位于节点 %s 的本地磁盘(local_disk),Master 无法跨节点访问。请登录该节点或改用云存储后再操作。", node.Name),
|
||||
nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async bool) (*BackupRecordDetail, error) {
|
||||
@@ -461,11 +326,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
|
||||
if task == nil {
|
||||
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "备份任务不存在", fmt.Errorf("backup task %d not found", id))
|
||||
}
|
||||
unlock := s.acquireTaskStartLock(task.ID)
|
||||
defer unlock()
|
||||
if err := s.ensureTaskNotRunning(ctx, task); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// 维护窗口校验:手动执行同样尊重窗口,避免业务高峰期误触发。
|
||||
if strings.TrimSpace(task.MaintenanceWindows) != "" {
|
||||
windows := backup.ParseMaintenanceWindows(task.MaintenanceWindows)
|
||||
@@ -486,9 +346,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
|
||||
return nil, perr
|
||||
}
|
||||
}
|
||||
if strings.EqualFold(task.BackupMode, model.BackupModeRepository) && s.resolveRemoteNode(ctx, resolvedNodeID) != nil {
|
||||
return nil, apperror.BadRequest("BACKUP_TASK_REPOSITORY_REMOTE_UNSUPPORTED", "CDC 仓库模式当前仅支持 Master 本机单写者执行", nil)
|
||||
}
|
||||
startedAt := s.now()
|
||||
// 取第一个存储目标 ID 做兼容
|
||||
primaryTargetID := task.StorageTargetID
|
||||
@@ -499,8 +356,8 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
|
||||
if err := s.records.Create(ctx, record); err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_CREATE_FAILED", "无法创建备份记录", err)
|
||||
}
|
||||
runTask := *task
|
||||
runTask.NodeID = resolvedNodeID
|
||||
// 用池选出的节点 ID 复写 task 副本,使后续路由/执行沿用
|
||||
task.NodeID = resolvedNodeID
|
||||
task.LastRunAt = &startedAt
|
||||
task.LastStatus = "running"
|
||||
if err := s.tasks.Update(ctx, task); err != nil {
|
||||
@@ -508,27 +365,27 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
|
||||
}
|
||||
// 多节点路由:task.NodeID 指向远程节点时,把执行任务入队给 Agent;
|
||||
// NodeID=0 或本机节点时由 Master 直接执行。
|
||||
if remoteNode := s.resolveRemoteNode(ctx, resolvedNodeID); remoteNode != nil {
|
||||
if remoteNode := s.resolveRemoteNode(ctx, task.NodeID); remoteNode != nil {
|
||||
// 节点离线 → 立即把刚创建的 running 记录标记 failed,返回明确错误
|
||||
if remoteNode.Status != model.NodeStatusOnline {
|
||||
offlineMsg := fmt.Sprintf("节点 %s 当前离线,无法执行备份任务", remoteNode.Name)
|
||||
_ = s.finalizeRecord(ctx, &runTask, record.ID, startedAt, model.BackupRecordStatusFailed,
|
||||
offlineMsg, "", "", 0, "", "", primaryTargetID)
|
||||
_ = s.finalizeRecord(ctx, task, record.ID, startedAt, model.BackupRecordStatusFailed,
|
||||
offlineMsg, "", "", 0, "", "")
|
||||
return nil, apperror.BadRequest("NODE_OFFLINE", offlineMsg, nil)
|
||||
}
|
||||
if _, enqueueErr := s.agentDispatcher.EnqueueCommand(ctx, resolvedNodeID, model.AgentCommandTypeRunTask, map[string]any{
|
||||
if _, enqueueErr := s.agentDispatcher.EnqueueCommand(ctx, task.NodeID, model.AgentCommandTypeRunTask, map[string]any{
|
||||
"taskId": task.ID,
|
||||
"recordId": record.ID,
|
||||
}); enqueueErr != nil {
|
||||
// 入队失败 → 在记录中标记失败,继续返回详情
|
||||
_ = s.finalizeRecord(ctx, &runTask, record.ID, startedAt, model.BackupRecordStatusFailed,
|
||||
"无法下发任务到远程节点: "+enqueueErr.Error(), "", "", 0, "", "", primaryTargetID)
|
||||
_ = s.finalizeRecord(ctx, task, record.ID, startedAt, model.BackupRecordStatusFailed,
|
||||
"无法下发任务到远程节点: "+enqueueErr.Error(), "", "", 0, "", "")
|
||||
return nil, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "无法下发任务到远程节点", enqueueErr)
|
||||
}
|
||||
return s.getRecordDetail(ctx, record.ID)
|
||||
}
|
||||
run := func() {
|
||||
s.executeTask(context.Background(), &runTask, record.ID, startedAt)
|
||||
s.executeTask(context.Background(), task, record.ID, startedAt)
|
||||
}
|
||||
if async {
|
||||
s.async(run)
|
||||
@@ -538,27 +395,6 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
|
||||
return s.getRecordDetail(ctx, record.ID)
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) acquireTaskStartLock(taskID uint) func() {
|
||||
value, _ := s.taskLocks.LoadOrStore(taskID, &sync.Mutex{})
|
||||
mu := value.(*sync.Mutex)
|
||||
mu.Lock()
|
||||
return mu.Unlock
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) ensureTaskNotRunning(ctx context.Context, task *model.BackupTask) error {
|
||||
taskID := task.ID
|
||||
items, err := s.records.List(ctx, repository.BackupRecordListOptions{TaskID: &taskID, Status: model.BackupRecordStatusRunning})
|
||||
if err != nil {
|
||||
return apperror.Internal("BACKUP_RECORD_LIST_FAILED", "无法检查任务运行状态", err)
|
||||
}
|
||||
if len(items) == 0 {
|
||||
return nil
|
||||
}
|
||||
return apperror.BadRequest("BACKUP_TASK_ALREADY_RUNNING",
|
||||
fmt.Sprintf("任务「%s」正在运行(记录 #%d),请等待完成后再触发。", task.Name, items[0].ID),
|
||||
nil)
|
||||
}
|
||||
|
||||
// shouldNotify 按任务的告警策略决定是否发送本次通知。
|
||||
// 成功结果:始终发送(方便用户确认备份状态)。
|
||||
// 失败结果:仅当"最近 N 条记录(含本次)均为 failed"时发送,N = AlertOnConsecutiveFails。
|
||||
@@ -693,181 +529,14 @@ func (s *BackupExecutionService) isRemoteNode(ctx context.Context, nodeID uint)
|
||||
// resolveRemoteNode 返回 NodeID 对应的远程节点指针,或 nil 表示本机执行。
|
||||
// 相比 isRemoteNode,它让调用方能读取节点状态(在线/离线)做进一步判断。
|
||||
func (s *BackupExecutionService) resolveRemoteNode(ctx context.Context, nodeID uint) *model.Node {
|
||||
return resolveRemoteExecutionNode(ctx, s.nodeRepo, s.agentDispatcher != nil, nodeID)
|
||||
}
|
||||
|
||||
// resolveDifferentialBase 为差异备份解析基线全量:仅本机(NodeID=0)文件任务且 BackupMode=differential 时生效。
|
||||
// 返回最近一次「成功、含清单、未超过 DiffFullIntervalDays」的全量记录 ID 及其清单;
|
||||
// 无合适基线(首次备份 / 最近全量已过期 / 清单缺失)时 ok=false,调用方回退为全量。
|
||||
func (s *BackupExecutionService) resolveDifferentialBase(ctx context.Context, task *model.BackupTask) (uint, backup.Manifest, bool) {
|
||||
if task.Type != model.BackupTaskTypeFile || task.NodeID != 0 || !strings.EqualFold(task.BackupMode, model.BackupModeDifferential) {
|
||||
return 0, backup.Manifest{}, false
|
||||
if s.nodeRepo == nil || s.agentDispatcher == nil || nodeID == 0 {
|
||||
return nil
|
||||
}
|
||||
records, err := s.records.ListSuccessfulByTask(ctx, task.ID)
|
||||
if err != nil {
|
||||
return 0, backup.Manifest{}, false
|
||||
node, err := s.nodeRepo.FindByID(ctx, nodeID)
|
||||
if err != nil || node == nil || node.IsLocal {
|
||||
return nil
|
||||
}
|
||||
intervalDays := task.DiffFullIntervalDays
|
||||
if intervalDays <= 0 {
|
||||
intervalDays = 7
|
||||
}
|
||||
cutoff := time.Now().Add(-time.Duration(intervalDays) * 24 * time.Hour)
|
||||
for i := range records {
|
||||
rec := records[i]
|
||||
if rec.BackupKind != model.BackupKindFull {
|
||||
continue
|
||||
}
|
||||
// 最近的全量已超过强制全量间隔 → 触发新全量,限制差异链跨度与单个差异体积。
|
||||
if rec.StartedAt.Before(cutoff) {
|
||||
return 0, backup.Manifest{}, false
|
||||
}
|
||||
// 列表查询已省略 Manifest 列,这里按需单独加载最近全量的清单(FindByID 含 Manifest)。
|
||||
full, ferr := s.records.FindByID(ctx, rec.ID)
|
||||
if ferr != nil || full == nil || strings.TrimSpace(full.Manifest) == "" {
|
||||
return 0, backup.Manifest{}, false
|
||||
}
|
||||
manifest, decErr := backup.DecodeManifest([]byte(full.Manifest))
|
||||
if decErr != nil || len(manifest.Entries) == 0 {
|
||||
return 0, backup.Manifest{}, false
|
||||
}
|
||||
return rec.ID, manifest, true
|
||||
}
|
||||
return 0, backup.Manifest{}, false
|
||||
}
|
||||
|
||||
type repositoryTaskResult struct {
|
||||
fileName string
|
||||
logicalSize int64
|
||||
checksum string
|
||||
storagePath string
|
||||
storageTargetID uint
|
||||
manifestJSON string
|
||||
uploadResults []StorageUploadResultItem
|
||||
providers map[uint]storage.StorageProvider
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) executeRepositoryTask(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, spec backup.TaskSpec, logger *backup.ExecutionLogger) (*repositoryTaskResult, error) {
|
||||
store := backup.NewRepositoryStore(s.cipher.Key())
|
||||
plan, err := store.BuildPlan(ctx, spec, logger)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() {
|
||||
if closeErr := plan.Close(); closeErr != nil {
|
||||
logger.Warnf("清理 CDC 临时计划失败:%v", closeErr)
|
||||
}
|
||||
}()
|
||||
manifestBytes, err := backup.EncodeManifest(plan.Manifest)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encode repository manifest: %w", err)
|
||||
}
|
||||
targetIDs := collectTargetIDs(task)
|
||||
if len(targetIDs) == 0 {
|
||||
return nil, fmt.Errorf("没有关联的存储目标")
|
||||
}
|
||||
storageUsage, usageErr := s.storageUsageSnapshot(ctx)
|
||||
if usageErr != nil {
|
||||
logger.Warnf("读取存储目标用量失败,跳过本次软配额校验:%v", usageErr)
|
||||
storageUsage = map[uint]int64{}
|
||||
}
|
||||
|
||||
snapshotKey := store.SnapshotKey(task.ID, recordID, startedAt)
|
||||
result := &repositoryTaskResult{
|
||||
fileName: filepath.Base(snapshotKey),
|
||||
logicalSize: plan.LogicalSize,
|
||||
storagePath: snapshotKey,
|
||||
manifestJSON: string(manifestBytes),
|
||||
uploadResults: make([]StorageUploadResultItem, 0, len(targetIDs)),
|
||||
providers: make(map[uint]storage.StorageProvider),
|
||||
}
|
||||
var failures []string
|
||||
for _, targetID := range targetIDs {
|
||||
target, findErr := s.targets.FindByID(ctx, targetID)
|
||||
targetName := fmt.Sprintf("target-%d", targetID)
|
||||
if findErr == nil && target != nil {
|
||||
targetName = target.Name
|
||||
}
|
||||
if findErr != nil || target == nil {
|
||||
message := "存储目标不存在"
|
||||
if findErr != nil {
|
||||
message = findErr.Error()
|
||||
}
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: message})
|
||||
failures = append(failures, fmt.Sprintf("%s: %s", targetName, message))
|
||||
continue
|
||||
}
|
||||
provider, resolveErr := s.resolveProviderForNode(ctx, targetID, task.NodeID)
|
||||
if resolveErr != nil {
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: resolveErr.Error()})
|
||||
failures = append(failures, fmt.Sprintf("%s: %v", targetName, resolveErr))
|
||||
continue
|
||||
}
|
||||
logger.Infof("同步 CDC 仓库到存储目标:%s", targetName)
|
||||
unlock := s.acquireRepositoryLock(targetID)
|
||||
estimatedSize, estimateErr := store.EstimateUploadSize(ctx, provider, plan)
|
||||
if estimateErr != nil {
|
||||
unlock()
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: estimateErr.Error()})
|
||||
failures = append(failures, fmt.Sprintf("%s: %v", targetName, estimateErr))
|
||||
continue
|
||||
}
|
||||
if target.QuotaBytes > 0 && storageUsage[targetID]+estimatedSize > target.QuotaBytes {
|
||||
unlock()
|
||||
message := fmt.Sprintf("超出存储目标配额(%d + 预计 %d > %d)", storageUsage[targetID], estimatedSize, target.QuotaBytes)
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: message})
|
||||
failures = append(failures, fmt.Sprintf("%s: %s", targetName, message))
|
||||
continue
|
||||
}
|
||||
upload, uploadErr := store.Upload(ctx, provider, plan, snapshotKey)
|
||||
unlock()
|
||||
if uploadErr != nil {
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: uploadErr.Error()})
|
||||
failures = append(failures, fmt.Sprintf("%s: %v", targetName, uploadErr))
|
||||
logger.Warnf("存储目标 %s CDC 仓库同步失败:%v", targetName, uploadErr)
|
||||
continue
|
||||
}
|
||||
result.uploadResults = append(result.uploadResults, StorageUploadResultItem{
|
||||
StorageTargetID: targetID, StorageTargetName: targetName, Status: "success",
|
||||
StoragePath: upload.SnapshotKey, FileSize: upload.UploadedBytes,
|
||||
})
|
||||
result.providers[targetID] = provider
|
||||
if result.storageTargetID == 0 {
|
||||
result.storageTargetID = targetID
|
||||
result.checksum = upload.Checksum
|
||||
}
|
||||
logger.Infof("存储目标 %s CDC 同步完成:新块 %d/%d,复用 %d bytes,实际上传 %d bytes", targetName, upload.NewChunks, upload.UniqueChunks, upload.ReusedBytes, upload.UploadedBytes)
|
||||
}
|
||||
if result.storageTargetID == 0 {
|
||||
return nil, fmt.Errorf("所有存储目标 CDC 仓库同步均失败:%s", strings.Join(failures, "; "))
|
||||
}
|
||||
if len(failures) > 0 {
|
||||
logger.Warnf("部分存储目标 CDC 仓库同步失败:%s", strings.Join(failures, "; "))
|
||||
}
|
||||
if s.dependentsResolver != nil {
|
||||
go func(upstreamID uint, upstreamName string) {
|
||||
dependents, resolveErr := s.dependentsResolver.TriggerDependents(context.Background(), upstreamID)
|
||||
if resolveErr != nil {
|
||||
logger.Warnf("解析任务 %s 的下游依赖失败:%v", upstreamName, resolveErr)
|
||||
return
|
||||
}
|
||||
for _, dependentID := range dependents {
|
||||
if _, runErr := s.RunTaskByID(context.Background(), dependentID); runErr != nil {
|
||||
logger.Warnf("触发下游任务 #%d 失败(上游: %s):%v", dependentID, upstreamName, runErr)
|
||||
} else {
|
||||
logger.Infof("已触发下游任务 #%d(上游: %s)", dependentID, upstreamName)
|
||||
}
|
||||
}
|
||||
}(task.ID, task.Name)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) acquireRepositoryLock(targetID uint) func() {
|
||||
created := &sync.Mutex{}
|
||||
actual, _ := s.repositoryLocks.LoadOrStore(targetID, created)
|
||||
lock := actual.(*sync.Mutex)
|
||||
lock.Lock()
|
||||
return lock.Unlock
|
||||
return node
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time) {
|
||||
@@ -892,80 +561,19 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
var fileSize int64
|
||||
var checksum string
|
||||
var storagePath string
|
||||
selectedStorageTargetID := task.StorageTargetID
|
||||
var uploadResults []StorageUploadResultItem
|
||||
// 差异备份链信息:实际类型(全量/差异)、基线全量 ID、全量清单 JSON。
|
||||
backupKind := model.BackupKindFull
|
||||
var baseRecordID uint
|
||||
var manifestJSON string
|
||||
var repositoryProviders map[uint]storage.StorageProvider
|
||||
completeRecord := func() {
|
||||
readyForRepositoryRetention := status == model.BackupRecordStatusSuccess
|
||||
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, checksum, storagePath, selectedStorageTargetID); finalizeErr != nil {
|
||||
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, checksum, storagePath); finalizeErr != nil {
|
||||
logger.Errorf("写回备份记录失败:%v", finalizeErr)
|
||||
readyForRepositoryRetention = false
|
||||
}
|
||||
// 采集任务执行结果到 Prometheus(耗时 + 产出字节 + 状态计数)
|
||||
s.metrics.ObserveTaskRun(task.Type, status, time.Since(startedAt).Seconds(), fileSize)
|
||||
// 写入多目标上传结果
|
||||
if len(uploadResults) > 0 {
|
||||
resultsJSON, marshalErr := json.Marshal(uploadResults)
|
||||
if marshalErr != nil {
|
||||
logger.Warnf("序列化多目标上传结果失败:%v", marshalErr)
|
||||
readyForRepositoryRetention = false
|
||||
} else if record, findErr := s.records.FindByID(ctx, recordID); findErr != nil || record == nil {
|
||||
if findErr != nil {
|
||||
logger.Warnf("读取备份记录以写回多目标结果失败:%v", findErr)
|
||||
} else {
|
||||
logger.Warnf("备份记录 #%d 不存在,无法写回多目标结果", recordID)
|
||||
}
|
||||
readyForRepositoryRetention = false
|
||||
} else {
|
||||
record.StorageUploadResults = string(resultsJSON)
|
||||
if updateErr := s.records.Update(ctx, record); updateErr != nil {
|
||||
logger.Warnf("写回多目标上传结果失败:%v", updateErr)
|
||||
readyForRepositoryRetention = false
|
||||
}
|
||||
}
|
||||
}
|
||||
// 持久化差异链信息:全量记录其清单(供后续差异比对),差异记录其基线全量 ID。
|
||||
if status == model.BackupRecordStatusSuccess && (backupKind != model.BackupKindFull || baseRecordID != 0 || manifestJSON != "") {
|
||||
if record, findErr := s.records.FindByID(ctx, recordID); findErr == nil && record != nil {
|
||||
record.BackupKind = backupKind
|
||||
record.BaseRecordID = baseRecordID
|
||||
record.Manifest = manifestJSON
|
||||
if updErr := s.records.Update(ctx, record); updErr != nil {
|
||||
logger.Warnf("写回差异链信息失败:%v", updErr)
|
||||
readyForRepositoryRetention = false
|
||||
}
|
||||
} else {
|
||||
if findErr != nil {
|
||||
logger.Warnf("读取备份记录以写回备份类型失败:%v", findErr)
|
||||
} else {
|
||||
logger.Warnf("备份记录 #%d 不存在,无法写回备份类型", recordID)
|
||||
}
|
||||
readyForRepositoryRetention = false
|
||||
}
|
||||
}
|
||||
if readyForRepositoryRetention && backupKind == model.BackupKindRepository && s.retention != nil && len(repositoryProviders) > 0 {
|
||||
targetIDs := make([]uint, 0, len(repositoryProviders))
|
||||
for targetID := range repositoryProviders {
|
||||
targetIDs = append(targetIDs, targetID)
|
||||
}
|
||||
sort.Slice(targetIDs, func(i, j int) bool { return targetIDs[i] < targetIDs[j] })
|
||||
unlocks := make([]func(), 0, len(targetIDs))
|
||||
for _, targetID := range targetIDs {
|
||||
unlocks = append(unlocks, s.acquireRepositoryLock(targetID))
|
||||
}
|
||||
cleanupResult, cleanupErr := s.retention.CleanupProviders(ctx, task, repositoryProviders)
|
||||
for index := len(unlocks) - 1; index >= 0; index-- {
|
||||
unlocks[index]()
|
||||
}
|
||||
if cleanupErr != nil {
|
||||
logger.Warnf("执行 CDC 仓库保留策略失败:%v", cleanupErr)
|
||||
} else {
|
||||
for _, warning := range cleanupResult.Warnings {
|
||||
logger.Warnf("CDC 仓库保留策略警告:%s", warning)
|
||||
if resultsJSON, marshalErr := json.Marshal(uploadResults); marshalErr == nil {
|
||||
if record, findErr := s.records.FindByID(ctx, recordID); findErr == nil && record != nil {
|
||||
record.StorageUploadResults = string(resultsJSON)
|
||||
_ = s.records.Update(ctx, record)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -986,33 +594,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
logger.Errorf("构建任务运行时配置失败:%v", err)
|
||||
return
|
||||
}
|
||||
if task.Type == model.BackupTaskTypeFile && strings.EqualFold(task.BackupMode, model.BackupModeRepository) {
|
||||
backupKind = model.BackupKindRepository
|
||||
repositoryResult, repositoryErr := s.executeRepositoryTask(ctx, task, recordID, startedAt, spec, logger)
|
||||
if repositoryErr != nil {
|
||||
errMessage = repositoryErr.Error()
|
||||
logger.Errorf("执行 CDC 仓库备份失败:%v", repositoryErr)
|
||||
return
|
||||
}
|
||||
fileName = repositoryResult.fileName
|
||||
fileSize = repositoryResult.logicalSize
|
||||
checksum = repositoryResult.checksum
|
||||
storagePath = repositoryResult.storagePath
|
||||
selectedStorageTargetID = repositoryResult.storageTargetID
|
||||
uploadResults = repositoryResult.uploadResults
|
||||
repositoryProviders = repositoryResult.providers
|
||||
manifestJSON = repositoryResult.manifestJSON
|
||||
status = model.BackupRecordStatusSuccess
|
||||
logger.Infof("CDC 仓库备份执行完成")
|
||||
return
|
||||
}
|
||||
// 差异备份:解析基线全量,命中则切换为差异模式(仅本机文件任务)。
|
||||
if baseID, baseManifest, ok := s.resolveDifferentialBase(ctx, task); ok {
|
||||
spec.Differential = true
|
||||
spec.BaseManifest = baseManifest
|
||||
baseRecordID = baseID
|
||||
logger.Infof("差异备份模式:基于全量备份 #%d 仅打包变更", baseID)
|
||||
}
|
||||
runner, err := s.runnerRegistry.Runner(spec.Type)
|
||||
if err != nil {
|
||||
errMessage = err.Error()
|
||||
@@ -1026,17 +607,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
return
|
||||
}
|
||||
defer os.RemoveAll(result.TempDir)
|
||||
// 依据运行器产出判定实际类型:产出清单 → 全量(记录清单供后续差异比对);否则为差异。
|
||||
if result.Manifest != nil {
|
||||
backupKind = model.BackupKindFull
|
||||
if data, encErr := backup.EncodeManifest(*result.Manifest); encErr == nil {
|
||||
manifestJSON = string(data)
|
||||
} else {
|
||||
logger.Warnf("备份清单序列化失败(不影响本次备份,但将禁用后续差异):%v", encErr)
|
||||
}
|
||||
} else {
|
||||
backupKind = model.BackupKindDifferential
|
||||
}
|
||||
finalPath := result.ArtifactPath
|
||||
if strings.EqualFold(task.Compression, "gzip") && !strings.HasSuffix(strings.ToLower(finalPath), ".gz") {
|
||||
logger.Infof("开始压缩备份文件")
|
||||
@@ -1047,15 +617,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
return
|
||||
}
|
||||
finalPath = compressedPath
|
||||
} else if strings.EqualFold(task.Compression, "zstd") && !strings.HasSuffix(strings.ToLower(finalPath), ".zst") {
|
||||
logger.Infof("开始压缩备份文件(zstd)")
|
||||
compressedPath, compressErr := compress.ZstdFile(finalPath)
|
||||
if compressErr != nil {
|
||||
errMessage = compressErr.Error()
|
||||
logger.Errorf("压缩备份文件失败:%v", compressErr)
|
||||
return
|
||||
}
|
||||
finalPath = compressedPath
|
||||
}
|
||||
if task.Encrypt {
|
||||
logger.Infof("开始加密备份文件")
|
||||
@@ -1084,11 +645,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
logger.Errorf("没有关联的存储目标")
|
||||
return
|
||||
}
|
||||
storageUsage, err := s.storageUsageSnapshot(ctx)
|
||||
if err != nil {
|
||||
logger.Warnf("读取存储目标用量失败,跳过本次软配额校验:%v", err)
|
||||
storageUsage = map[uint]int64{}
|
||||
}
|
||||
|
||||
// 并行上传到所有目标
|
||||
uploadResults = make([]StorageUploadResultItem, len(targetIDs))
|
||||
@@ -1112,7 +668,15 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
}
|
||||
// 软限额校验:QuotaBytes > 0 时,已累计 + 本次 > 配额 → 拒绝上传
|
||||
if target != nil && target.QuotaBytes > 0 {
|
||||
currentUsed := storageUsage[targetID]
|
||||
currentUsed := int64(0)
|
||||
if items, err := s.records.StorageUsage(ctx); err == nil {
|
||||
for _, it := range items {
|
||||
if it.StorageTargetID == targetID {
|
||||
currentUsed = it.TotalSize
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if currentUsed+fileSize > target.QuotaBytes {
|
||||
quotaMsg := fmt.Sprintf("超出存储目标 %s 的配额(%d + %d > %d)", targetName, currentUsed, fileSize, target.QuotaBytes)
|
||||
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: quotaMsg}
|
||||
@@ -1121,18 +685,15 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
}
|
||||
}
|
||||
logger.Infof("开始上传备份到存储目标:%s", targetName)
|
||||
// 上传级重试:最多 3 次,等待时间随 context 取消及时退出。
|
||||
// 上传级重试:最多 3 次,指数退避(10s, 30s, 90s)
|
||||
maxAttempts := 3
|
||||
var lastUploadErr error
|
||||
var hr *hashingReader
|
||||
for attempt := 1; attempt <= uploadMaxAttempts; attempt++ {
|
||||
for attempt := 1; attempt <= maxAttempts; attempt++ {
|
||||
if attempt > 1 {
|
||||
backoff := time.Duration(attempt-1) * uploadRetryBackoff
|
||||
backoff := time.Duration(attempt*attempt) * 10 * time.Second
|
||||
logger.Warnf("存储目标 %s 第 %d 次重试(等待 %v):%v", targetName, attempt, backoff, lastUploadErr)
|
||||
if waitErr := waitForUploadRetry(ctx, backoff); waitErr != nil {
|
||||
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: waitErr.Error()}
|
||||
logger.Warnf("存储目标 %s 上传重试已取消:%v", targetName, waitErr)
|
||||
return
|
||||
}
|
||||
time.Sleep(backoff)
|
||||
}
|
||||
artifact, openErr := os.Open(finalPath)
|
||||
if openErr != nil {
|
||||
@@ -1162,7 +723,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
}
|
||||
if lastUploadErr != nil {
|
||||
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: lastUploadErr.Error()}
|
||||
logger.Warnf("存储目标 %s 上传失败(已重试 %d 次):%v", targetName, uploadMaxAttempts, lastUploadErr)
|
||||
logger.Warnf("存储目标 %s 上传失败(已重试 %d 次):%v", targetName, maxAttempts, lastUploadErr)
|
||||
return
|
||||
}
|
||||
// 完整性校验:对比实际传输字节数
|
||||
@@ -1198,9 +759,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
for _, r := range uploadResults {
|
||||
if r.Status == "success" {
|
||||
anySuccess = true
|
||||
if selectedStorageTargetID == task.StorageTargetID {
|
||||
selectedStorageTargetID = r.StorageTargetID
|
||||
}
|
||||
} else if r.Error != "" {
|
||||
failedMessages = append(failedMessages, fmt.Sprintf("%s: %s", r.StorageTargetName, r.Error))
|
||||
}
|
||||
@@ -1233,7 +791,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
record := &model.BackupRecord{
|
||||
ID: recordID,
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: selectedStorageTargetID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
NodeID: task.NodeID,
|
||||
Status: "success",
|
||||
FileName: fileName,
|
||||
@@ -1258,7 +816,7 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
|
||||
}
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, checksum string, storagePath string, storageTargetID uint) error {
|
||||
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, checksum string, storagePath string) error {
|
||||
record, err := s.records.FindByID(ctx, recordID)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -1268,9 +826,6 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
|
||||
}
|
||||
completedAt := s.now()
|
||||
record.Status = status
|
||||
if storageTargetID > 0 {
|
||||
record.StorageTargetID = storageTargetID
|
||||
}
|
||||
record.FileName = fileName
|
||||
record.FileSize = fileSize
|
||||
record.Checksum = checksum
|
||||
@@ -1287,32 +842,6 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
|
||||
return s.tasks.Update(ctx, task)
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) storageUsageSnapshot(ctx context.Context) (map[uint]int64, error) {
|
||||
items, err := s.records.StorageUsage(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("storage usage snapshot: %w", err)
|
||||
}
|
||||
usage := make(map[uint]int64, len(items))
|
||||
for _, item := range items {
|
||||
usage[item.StorageTargetID] = item.TotalSize
|
||||
}
|
||||
return usage, nil
|
||||
}
|
||||
|
||||
func waitForUploadRetry(ctx context.Context, delay time.Duration) error {
|
||||
if delay <= 0 {
|
||||
return nil
|
||||
}
|
||||
timer := time.NewTimer(delay)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-timer.C:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) resolveProvider(ctx context.Context, targetID uint) (storage.StorageProvider, error) {
|
||||
return s.resolveProviderForNode(ctx, targetID, 0)
|
||||
}
|
||||
@@ -1326,11 +855,78 @@ func (s *BackupExecutionService) resolveProviderForNode(ctx context.Context, tar
|
||||
LowLevelRetries: s.retries,
|
||||
BandwidthLimit: s.effectiveBandwidth(ctx, nodeID),
|
||||
})
|
||||
return resolveStorageProvider(ctx, s.targets, s.storageRegistry, s.cipher, targetID)
|
||||
target, err := s.targets.FindByID(ctx, targetID)
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
|
||||
}
|
||||
if target == nil {
|
||||
return nil, apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "关联的存储目标不存在", nil)
|
||||
}
|
||||
configMap := map[string]any{}
|
||||
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
|
||||
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
|
||||
}
|
||||
provider, err := s.storageRegistry.Create(ctx, target.Type, configMap)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) buildTaskSpec(task *model.BackupTask, startedAt time.Time) (backup.TaskSpec, error) {
|
||||
return buildBackupTaskSpec(s.cipher, task, startedAt, s.tempDir)
|
||||
excludePatterns := []string{}
|
||||
if strings.TrimSpace(task.ExcludePatterns) != "" {
|
||||
if err := json.Unmarshal([]byte(task.ExcludePatterns), &excludePatterns); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析排除规则", err)
|
||||
}
|
||||
}
|
||||
password := ""
|
||||
if strings.TrimSpace(task.DBPasswordCiphertext) != "" {
|
||||
plain, err := s.cipher.Decrypt(task.DBPasswordCiphertext)
|
||||
if err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECRYPT_FAILED", "无法解密数据库密码", err)
|
||||
}
|
||||
password = string(plain)
|
||||
}
|
||||
sourcePaths := []string{}
|
||||
if strings.TrimSpace(task.SourcePaths) != "" {
|
||||
if err := json.Unmarshal([]byte(task.SourcePaths), &sourcePaths); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
|
||||
}
|
||||
}
|
||||
dbSpec := backup.DatabaseSpec{
|
||||
Host: task.DBHost,
|
||||
Port: task.DBPort,
|
||||
User: task.DBUser,
|
||||
Password: password,
|
||||
Names: []string{task.DBName},
|
||||
Path: task.DBPath,
|
||||
}
|
||||
// 解析 ExtraConfig 填充类型特有字段(目前主要用于 SAP HANA)
|
||||
if strings.TrimSpace(task.ExtraConfig) != "" {
|
||||
extra := map[string]any{}
|
||||
if err := json.Unmarshal([]byte(task.ExtraConfig), &extra); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
|
||||
}
|
||||
applyHANAExtraConfig(&dbSpec, extra)
|
||||
}
|
||||
return backup.TaskSpec{
|
||||
ID: task.ID,
|
||||
Name: task.Name,
|
||||
Type: task.Type,
|
||||
SourcePath: task.SourcePath,
|
||||
SourcePaths: sourcePaths,
|
||||
ExcludePatterns: excludePatterns,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
StorageTargetType: "",
|
||||
Compression: task.Compression,
|
||||
Encrypt: task.Encrypt,
|
||||
RetentionDays: task.RetentionDays,
|
||||
MaxBackups: task.MaxBackups,
|
||||
StartedAt: startedAt,
|
||||
TempDir: s.tempDir,
|
||||
Database: dbSpec,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// applyHANAExtraConfig 从 ExtraConfig map 中提取 SAP HANA 字段填入 DatabaseSpec。
|
||||
@@ -1361,9 +957,6 @@ func (s *BackupExecutionService) loadRecordProvider(ctx context.Context, recordI
|
||||
if record == nil {
|
||||
return nil, nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", fmt.Errorf("backup record %d not found", recordID))
|
||||
}
|
||||
if err := s.validateClusterAccessible(ctx, record); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
provider, err := s.resolveProvider(ctx, record.StorageTargetID)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
@@ -1372,7 +965,22 @@ func (s *BackupExecutionService) loadRecordProvider(ctx context.Context, recordI
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) prepareArtifactForRestore(artifactPath string) (string, error) {
|
||||
return prepareBackupArtifact(s.cipher, artifactPath, nil)
|
||||
currentPath := artifactPath
|
||||
if strings.HasSuffix(strings.ToLower(currentPath), ".enc") {
|
||||
decryptedPath, err := backupcrypto.DecryptFile(s.cipher.Key(), currentPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
currentPath = decryptedPath
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(currentPath), ".gz") {
|
||||
decompressedPath, err := compress.GunzipFile(currentPath)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
currentPath = decompressedPath
|
||||
}
|
||||
return currentPath, nil
|
||||
}
|
||||
|
||||
func (s *BackupExecutionService) getRecordDetail(ctx context.Context, recordID uint) (*BackupRecordDetail, error) {
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/backup"
|
||||
backupretention "backupx/server/internal/backup/retention"
|
||||
@@ -25,70 +18,6 @@ import (
|
||||
storageRclone "backupx/server/internal/storage/rclone"
|
||||
)
|
||||
|
||||
type testStorageFactory struct {
|
||||
providers map[string]*testStorageProvider
|
||||
}
|
||||
|
||||
func (f *testStorageFactory) Type() storage.ProviderType {
|
||||
return "test_storage"
|
||||
}
|
||||
|
||||
func (f *testStorageFactory) New(_ context.Context, config map[string]any) (storage.StorageProvider, error) {
|
||||
name, _ := config["name"].(string)
|
||||
provider := f.providers[name]
|
||||
if provider == nil {
|
||||
return nil, fmt.Errorf("unknown provider %q", name)
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
type testStorageProvider struct {
|
||||
name string
|
||||
failUpload bool
|
||||
blockUpload <-chan struct{}
|
||||
onUpload func()
|
||||
objects map[string][]byte
|
||||
}
|
||||
|
||||
func (p *testStorageProvider) Type() storage.ProviderType { return "test_storage" }
|
||||
func (p *testStorageProvider) TestConnection(context.Context) error {
|
||||
return nil
|
||||
}
|
||||
func (p *testStorageProvider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
|
||||
if p.blockUpload != nil {
|
||||
<-p.blockUpload
|
||||
}
|
||||
if p.onUpload != nil {
|
||||
p.onUpload()
|
||||
}
|
||||
if p.failUpload {
|
||||
return fmt.Errorf("upload failed for %s", p.name)
|
||||
}
|
||||
data, err := io.ReadAll(reader)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if p.objects == nil {
|
||||
p.objects = map[string][]byte{}
|
||||
}
|
||||
p.objects[objectKey] = data
|
||||
return nil
|
||||
}
|
||||
func (p *testStorageProvider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
|
||||
data, ok := p.objects[objectKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("object %s not found", objectKey)
|
||||
}
|
||||
return io.NopCloser(strings.NewReader(string(data))), nil
|
||||
}
|
||||
func (p *testStorageProvider) Delete(_ context.Context, objectKey string) error {
|
||||
delete(p.objects, objectKey)
|
||||
return nil
|
||||
}
|
||||
func (p *testStorageProvider) List(context.Context, string) ([]storage.ObjectInfo, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func newExecutionTestServices(t *testing.T) (*BackupExecutionService, *BackupRecordService, repository.BackupTaskRepository, repository.StorageTargetRepository, repository.BackupRecordRepository, string, string) {
|
||||
t.Helper()
|
||||
baseDir := t.TempDir()
|
||||
@@ -156,529 +85,6 @@ func TestBackupExecutionServiceRunTaskByIDSync(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceRepositoryModeRoundTrip(t *testing.T) {
|
||||
executionService, recordService, tasks, _, records, sourceDir, storageDir := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
task.BackupMode = model.BackupModeRepository
|
||||
task.Compression = "zstd"
|
||||
if err := tasks.Update(ctx, task); err != nil {
|
||||
t.Fatalf("Update repository task returned error: %v", err)
|
||||
}
|
||||
large := make([]byte, 4<<20)
|
||||
for index := range large {
|
||||
large[index] = byte((index * 31) % 251)
|
||||
}
|
||||
largePath := filepath.Join(sourceDir, "large.bin")
|
||||
if err := os.WriteFile(largePath, large, 0o640); err != nil {
|
||||
t.Fatalf("write large fixture: %v", err)
|
||||
}
|
||||
|
||||
first, err := executionService.RunTaskByIDSync(ctx, task.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("first repository backup returned error: %v", err)
|
||||
}
|
||||
if first.Status != model.BackupRecordStatusSuccess || first.BackupKind != model.BackupKindRepository {
|
||||
t.Fatalf("unexpected first repository record: %#v", first)
|
||||
}
|
||||
if !strings.HasPrefix(first.StoragePath, ".backupx/repository/v1/snapshots/") {
|
||||
t.Fatalf("unexpected repository snapshot path: %s", first.StoragePath)
|
||||
}
|
||||
|
||||
large[2<<20] ^= 0xff
|
||||
if err := os.WriteFile(largePath, large, 0o640); err != nil {
|
||||
t.Fatalf("modify large fixture: %v", err)
|
||||
}
|
||||
second, err := executionService.RunTaskByIDSync(ctx, task.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("second repository backup returned error: %v", err)
|
||||
}
|
||||
stored, err := records.FindByID(ctx, second.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID repository record returned error: %v", err)
|
||||
}
|
||||
if stored == nil || stored.BackupKind != model.BackupKindRepository || stored.Manifest == "" {
|
||||
t.Fatalf("repository metadata was not persisted: %#v", stored)
|
||||
}
|
||||
|
||||
download, err := recordService.Download(ctx, second.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("export repository snapshot returned error: %v", err)
|
||||
}
|
||||
exported, readErr := io.ReadAll(download.Reader)
|
||||
closeErr := download.Reader.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
t.Fatalf("read repository export: read=%v close=%v", readErr, closeErr)
|
||||
}
|
||||
if len(exported) == 0 || !strings.HasSuffix(download.FileName, ".tar") {
|
||||
t.Fatalf("unexpected repository export: name=%s size=%d", download.FileName, len(exported))
|
||||
}
|
||||
|
||||
if err := os.WriteFile(largePath, bytes.Repeat([]byte{0}, len(large)), 0o640); err != nil {
|
||||
t.Fatalf("damage source before restore: %v", err)
|
||||
}
|
||||
if err := executionService.RestoreRecord(ctx, second.ID); err != nil {
|
||||
t.Fatalf("restore repository record returned error: %v", err)
|
||||
}
|
||||
restored, err := os.ReadFile(largePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read restored source: %v", err)
|
||||
}
|
||||
if !bytes.Equal(restored, large) {
|
||||
t.Fatalf("repository restore did not reproduce the source")
|
||||
}
|
||||
|
||||
if err := recordService.Delete(ctx, first.ID); err != nil {
|
||||
t.Fatalf("delete first repository record: %v", err)
|
||||
}
|
||||
if err := recordService.Delete(ctx, second.ID); err != nil {
|
||||
t.Fatalf("delete second repository record: %v", err)
|
||||
}
|
||||
packRoot := filepath.Join(storageDir, filepath.FromSlash(".backupx/repository/v1/packs"))
|
||||
remainingPacks := 0
|
||||
if err := filepath.Walk(packRoot, func(_ string, info os.FileInfo, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
if os.IsNotExist(walkErr) {
|
||||
return nil
|
||||
}
|
||||
return walkErr
|
||||
}
|
||||
if info != nil && !info.IsDir() {
|
||||
remainingPacks++
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("inspect repository packs: %v", err)
|
||||
}
|
||||
if remainingPacks != 0 {
|
||||
t.Fatalf("repository prune left %d packs after deleting all snapshots", remainingPacks)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceNodePoolSelectionDoesNotPersistTaskNodeID(t *testing.T) {
|
||||
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
|
||||
nodeRepo := &nodeRepoStub{nodes: []model.Node{
|
||||
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline, Labels: "prod,db"},
|
||||
{ID: 11, Name: "edge-b", Token: "edge-b-token", Status: model.NodeStatusOnline, Labels: "prod,db"},
|
||||
}}
|
||||
dispatcher := &fakeDispatcher{}
|
||||
executionService.SetClusterDependencies(nodeRepo, dispatcher)
|
||||
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID returned error: %v", err)
|
||||
}
|
||||
task.NodeID = 0
|
||||
task.NodePoolTag = "db"
|
||||
if err := tasks.Update(ctx, task); err != nil {
|
||||
t.Fatalf("Update task returned error: %v", err)
|
||||
}
|
||||
|
||||
detail, err := executionService.RunTaskByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("RunTaskByID returned error: %v", err)
|
||||
}
|
||||
storedTask, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID after run returned error: %v", err)
|
||||
}
|
||||
if storedTask.NodeID != 0 {
|
||||
t.Fatalf("expected pooled task NodeID to remain 0, got %d", storedTask.NodeID)
|
||||
}
|
||||
if storedTask.NodePoolTag != "db" {
|
||||
t.Fatalf("expected pooled task tag to remain db, got %q", storedTask.NodePoolTag)
|
||||
}
|
||||
storedRecord, err := records.FindByID(ctx, detail.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if storedRecord == nil || storedRecord.NodeID != 10 {
|
||||
t.Fatalf("expected record to keep selected node 10, got %#v", storedRecord)
|
||||
}
|
||||
calls := dispatcher.snapshot()
|
||||
if len(calls) != 1 || calls[0].NodeID != 10 || calls[0].CmdType != model.AgentCommandTypeRunTask {
|
||||
t.Fatalf("unexpected dispatcher calls: %#v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceRejectsDuplicateRunningTask(t *testing.T) {
|
||||
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
startedAt := time.Now().UTC()
|
||||
running := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
NodeID: 0,
|
||||
Status: model.BackupRecordStatusRunning,
|
||||
StartedAt: startedAt,
|
||||
}
|
||||
if err := records.Create(ctx, running); err != nil {
|
||||
t.Fatalf("Create running record returned error: %v", err)
|
||||
}
|
||||
|
||||
_, err = executionService.RunTaskByIDSync(ctx, task.ID)
|
||||
if err == nil || !strings.Contains(err.Error(), "正在运行") {
|
||||
t.Fatalf("expected duplicate running task to be rejected, got %v", err)
|
||||
}
|
||||
items, err := records.List(ctx, repository.BackupRecordListOptions{Status: model.BackupRecordStatusRunning})
|
||||
if err != nil {
|
||||
t.Fatalf("List running records returned error: %v", err)
|
||||
}
|
||||
if len(items) != 1 || items[0].ID != running.ID {
|
||||
t.Fatalf("expected only the original running record, got %#v", items)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceDeleteRecordDispatchesRemoteLocalDiskCleanup(t *testing.T) {
|
||||
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
nodeRepo := &nodeRepoStub{nodes: []model.Node{
|
||||
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
|
||||
}}
|
||||
dispatcher := &fakeDispatcher{}
|
||||
executionService.SetClusterDependencies(nodeRepo, dispatcher)
|
||||
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
completedAt := time.Now().UTC()
|
||||
record := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
NodeID: 10,
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "remote.tar.gz",
|
||||
StoragePath: "file/2026/05/09/remote.tar.gz",
|
||||
StartedAt: completedAt.Add(-time.Second),
|
||||
CompletedAt: &completedAt,
|
||||
}
|
||||
if err := records.Create(ctx, record); err != nil {
|
||||
t.Fatalf("Create record returned error: %v", err)
|
||||
}
|
||||
|
||||
if err := executionService.DeleteRecord(ctx, record.ID); err != nil {
|
||||
t.Fatalf("DeleteRecord returned error: %v", err)
|
||||
}
|
||||
deleted, err := records.FindByID(ctx, record.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if deleted != nil {
|
||||
t.Fatalf("expected record deleted, got %#v", deleted)
|
||||
}
|
||||
calls := dispatcher.snapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected one dispatcher call, got %#v", calls)
|
||||
}
|
||||
if calls[0].NodeID != 10 || calls[0].CmdType != model.AgentCommandTypeDeleteStorageObject {
|
||||
t.Fatalf("unexpected dispatcher call: %#v", calls[0])
|
||||
}
|
||||
if calls[0].Payload["storagePath"] != record.StoragePath {
|
||||
t.Fatalf("expected storagePath %q, got %#v", record.StoragePath, calls[0].Payload)
|
||||
}
|
||||
if calls[0].Payload["targetType"] != string(storage.ProviderTypeLocalDisk) {
|
||||
t.Fatalf("expected local_disk targetType, got %#v", calls[0].Payload)
|
||||
}
|
||||
if _, ok := calls[0].Payload["targetConfig"].(map[string]any); !ok {
|
||||
t.Fatalf("expected targetConfig map, got %#v", calls[0].Payload["targetConfig"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceRestoreRecordRejectsRemoteLocalDisk(t *testing.T) {
|
||||
executionService, _, tasks, _, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
executionService.SetClusterDependencies(&nodeRepoStub{nodes: []model.Node{
|
||||
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
|
||||
}}, &fakeDispatcher{})
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
completedAt := time.Now().UTC()
|
||||
record := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
NodeID: 10,
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "remote.tar.gz",
|
||||
StoragePath: "file/2026/05/09/remote.tar.gz",
|
||||
StartedAt: completedAt.Add(-time.Second),
|
||||
CompletedAt: &completedAt,
|
||||
}
|
||||
if err := records.Create(ctx, record); err != nil {
|
||||
t.Fatalf("Create record returned error: %v", err)
|
||||
}
|
||||
|
||||
err = executionService.RestoreRecord(ctx, record.ID)
|
||||
if err == nil {
|
||||
t.Fatal("expected remote local_disk restore to be rejected")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "Master 无法跨节点访问") {
|
||||
t.Fatalf("expected cross-node local_disk error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceDownloadsMasterRelayedLocalDiskRecord(t *testing.T) {
|
||||
executionService, _, tasks, _, records, _, storageDir := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
executionService.SetClusterDependencies(&nodeRepoStub{nodes: []model.Node{
|
||||
{ID: 10, Name: "edge-a", Token: "edge-a-token", Status: model.NodeStatusOnline},
|
||||
}}, &fakeDispatcher{})
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
storagePath := "file/2026/05/09/relayed.tar"
|
||||
artifactPath := filepath.Join(storageDir, filepath.FromSlash(storagePath))
|
||||
if err := os.MkdirAll(filepath.Dir(artifactPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll artifact parent returned error: %v", err)
|
||||
}
|
||||
content := []byte("stored on Master")
|
||||
if err := os.WriteFile(artifactPath, content, 0o600); err != nil {
|
||||
t.Fatalf("WriteFile artifact returned error: %v", err)
|
||||
}
|
||||
completedAt := time.Now().UTC()
|
||||
record := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
NodeID: 10,
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
FileName: "relayed.tar",
|
||||
FileSize: int64(len(content)),
|
||||
StoragePath: storagePath,
|
||||
StorageTransferMode: storage.TransferModeMasterRelay,
|
||||
StartedAt: completedAt.Add(-time.Second),
|
||||
CompletedAt: &completedAt,
|
||||
}
|
||||
if err := records.Create(ctx, record); err != nil {
|
||||
t.Fatalf("Create record returned error: %v", err)
|
||||
}
|
||||
|
||||
download, err := executionService.DownloadRecord(ctx, record.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("DownloadRecord returned error: %v", err)
|
||||
}
|
||||
got, readErr := io.ReadAll(download.Reader)
|
||||
closeErr := download.Reader.Close()
|
||||
if readErr != nil || closeErr != nil {
|
||||
t.Fatalf("read relayed artifact: read=%v close=%v", readErr, closeErr)
|
||||
}
|
||||
if !bytes.Equal(got, content) {
|
||||
t.Fatalf("downloaded content = %q, want %q", got, content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceRecordsFirstSuccessfulStorageTarget(t *testing.T) {
|
||||
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
second := &testStorageProvider{name: "second", objects: map[string][]byte{}}
|
||||
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
|
||||
"second": second,
|
||||
}})
|
||||
cipher := codec.NewConfigCipher("execution-secret")
|
||||
firstConfig, err := cipher.EncryptJSON(map[string]any{"name": "missing"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON first returned error: %v", err)
|
||||
}
|
||||
secondConfig, err := cipher.EncryptJSON(map[string]any{"name": "second"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON second returned error: %v", err)
|
||||
}
|
||||
if err := targets.Create(ctx, &model.StorageTarget{Name: "first", Type: "test_storage", Enabled: true, ConfigCiphertext: firstConfig, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
|
||||
t.Fatalf("Create first target returned error: %v", err)
|
||||
}
|
||||
if err := targets.Create(ctx, &model.StorageTarget{Name: "second", Type: "test_storage", Enabled: true, ConfigCiphertext: secondConfig, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
|
||||
t.Fatalf("Create second target returned error: %v", err)
|
||||
}
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
task.StorageTargetID = 2
|
||||
task.StorageTargets = []model.StorageTarget{{ID: 2}, {ID: 3}}
|
||||
if err := tasks.Update(ctx, task); err != nil {
|
||||
t.Fatalf("Update task returned error: %v", err)
|
||||
}
|
||||
|
||||
detail, err := executionService.RunTaskByIDSync(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("RunTaskByIDSync returned error: %v", err)
|
||||
}
|
||||
if detail.Status != model.BackupRecordStatusSuccess {
|
||||
t.Fatalf("expected success, got %#v", detail)
|
||||
}
|
||||
storedRecord, err := records.FindByID(ctx, detail.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID record returned error: %v", err)
|
||||
}
|
||||
if storedRecord.StorageTargetID != 3 {
|
||||
t.Fatalf("expected record StorageTargetID to point at successful target 3, got %d", storedRecord.StorageTargetID)
|
||||
}
|
||||
if _, ok := second.objects[storedRecord.StoragePath]; !ok {
|
||||
t.Fatalf("expected object in successful provider at %q", storedRecord.StoragePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceUploadRetryStopsWhenContextCancelled(t *testing.T) {
|
||||
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
var cancelOnce sync.Once
|
||||
failing := &testStorageProvider{
|
||||
name: "failing",
|
||||
failUpload: true,
|
||||
onUpload: func() {
|
||||
cancelOnce.Do(cancel)
|
||||
},
|
||||
}
|
||||
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
|
||||
"failing": failing,
|
||||
}})
|
||||
cipher := codec.NewConfigCipher("execution-secret")
|
||||
failingConfig, err := cipher.EncryptJSON(map[string]any{"name": "failing"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON returned error: %v", err)
|
||||
}
|
||||
if err := targets.Update(ctx, &model.StorageTarget{
|
||||
ID: 1,
|
||||
Name: "local",
|
||||
Type: "test_storage",
|
||||
Enabled: true,
|
||||
ConfigCiphertext: failingConfig,
|
||||
ConfigVersion: 1,
|
||||
LastTestStatus: "unknown",
|
||||
}); err != nil {
|
||||
t.Fatalf("Update target returned error: %v", err)
|
||||
}
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
startedAt := time.Now().UTC()
|
||||
record := &model.BackupRecord{
|
||||
TaskID: task.ID,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
Status: model.BackupRecordStatusRunning,
|
||||
StartedAt: startedAt,
|
||||
}
|
||||
if err := records.Create(ctx, record); err != nil {
|
||||
t.Fatalf("Create record returned error: %v", err)
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
executionService.executeTask(ctx, task, record.ID, startedAt)
|
||||
close(done)
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("expected cancelled upload retry to stop without waiting for backoff sleep")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceReadsStorageUsageOnceForMultiTargetQuotaChecks(t *testing.T) {
|
||||
executionService, _, tasks, targets, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
first := &testStorageProvider{name: "first", objects: map[string][]byte{}}
|
||||
second := &testStorageProvider{name: "second", objects: map[string][]byte{}}
|
||||
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
|
||||
"first": first,
|
||||
"second": second,
|
||||
}})
|
||||
cipher := codec.NewConfigCipher("execution-secret")
|
||||
firstConfig, err := cipher.EncryptJSON(map[string]any{"name": "first"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON first returned error: %v", err)
|
||||
}
|
||||
secondConfig, err := cipher.EncryptJSON(map[string]any{"name": "second"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON second returned error: %v", err)
|
||||
}
|
||||
if err := targets.Update(ctx, &model.StorageTarget{ID: 1, Name: "local", Type: "test_storage", Enabled: true, ConfigCiphertext: firstConfig, ConfigVersion: 1, LastTestStatus: "unknown", QuotaBytes: 1 << 30}); err != nil {
|
||||
t.Fatalf("Update first target returned error: %v", err)
|
||||
}
|
||||
if err := targets.Create(ctx, &model.StorageTarget{Name: "second", Type: "test_storage", Enabled: true, ConfigCiphertext: secondConfig, ConfigVersion: 1, LastTestStatus: "unknown", QuotaBytes: 1 << 30}); err != nil {
|
||||
t.Fatalf("Create second target returned error: %v", err)
|
||||
}
|
||||
task, err := tasks.FindByID(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("FindByID task returned error: %v", err)
|
||||
}
|
||||
task.StorageTargets = []model.StorageTarget{{ID: 1}, {ID: 2}}
|
||||
if err := tasks.Update(ctx, task); err != nil {
|
||||
t.Fatalf("Update task returned error: %v", err)
|
||||
}
|
||||
executionService.records = &storageUsageCountingRecordRepo{BackupRecordRepository: records}
|
||||
|
||||
detail, err := executionService.RunTaskByIDSync(ctx, task.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("RunTaskByIDSync returned error: %v", err)
|
||||
}
|
||||
if detail.Status != model.BackupRecordStatusSuccess {
|
||||
t.Fatalf("expected success, got %#v", detail)
|
||||
}
|
||||
countingRepo := executionService.records.(*storageUsageCountingRecordRepo)
|
||||
if countingRepo.usageCalls != 1 {
|
||||
t.Fatalf("expected StorageUsage to be called once for quota snapshot, got %d", countingRepo.usageCalls)
|
||||
}
|
||||
if len(first.objects) != 1 || len(second.objects) != 1 {
|
||||
t.Fatalf("expected both targets to receive upload, got first=%d second=%d", len(first.objects), len(second.objects))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExecutionServiceContinuesWhenStorageUsageSnapshotFails(t *testing.T) {
|
||||
executionService, _, _, targets, records, _, _ := newExecutionTestServices(t)
|
||||
ctx := context.Background()
|
||||
provider := &testStorageProvider{name: "primary", objects: map[string][]byte{}}
|
||||
executionService.storageRegistry = storage.NewRegistry(&testStorageFactory{providers: map[string]*testStorageProvider{
|
||||
"primary": provider,
|
||||
}})
|
||||
cipher := codec.NewConfigCipher("execution-secret")
|
||||
configCiphertext, err := cipher.EncryptJSON(map[string]any{"name": "primary"})
|
||||
if err != nil {
|
||||
t.Fatalf("EncryptJSON returned error: %v", err)
|
||||
}
|
||||
if err := targets.Update(ctx, &model.StorageTarget{
|
||||
ID: 1,
|
||||
Name: "local",
|
||||
Type: "test_storage",
|
||||
Enabled: true,
|
||||
ConfigCiphertext: configCiphertext,
|
||||
ConfigVersion: 1,
|
||||
LastTestStatus: "unknown",
|
||||
QuotaBytes: 1 << 30,
|
||||
}); err != nil {
|
||||
t.Fatalf("Update target returned error: %v", err)
|
||||
}
|
||||
executionService.records = &storageUsageFailingRecordRepo{
|
||||
BackupRecordRepository: records,
|
||||
err: errStorageUsageFailed,
|
||||
}
|
||||
|
||||
detail, err := executionService.RunTaskByIDSync(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("RunTaskByIDSync returned error: %v", err)
|
||||
}
|
||||
if detail.Status != model.BackupRecordStatusSuccess {
|
||||
t.Fatalf("expected success despite soft quota usage snapshot error, got %#v", detail)
|
||||
}
|
||||
if len(provider.objects) != 1 {
|
||||
t.Fatalf("expected upload to proceed, got %d uploaded objects", len(provider.objects))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRecordServiceRestore(t *testing.T) {
|
||||
executionService, recordService, _, _, _, sourceDir, _ := newExecutionTestServices(t)
|
||||
detail, err := executionService.RunTaskByIDSync(context.Background(), 1)
|
||||
@@ -699,27 +105,3 @@ func TestBackupRecordServiceRestore(t *testing.T) {
|
||||
t.Fatalf("unexpected restored content: %s", string(content))
|
||||
}
|
||||
}
|
||||
|
||||
type storageUsageCountingRecordRepo struct {
|
||||
repository.BackupRecordRepository
|
||||
mu sync.Mutex
|
||||
usageCalls int
|
||||
}
|
||||
|
||||
func (r *storageUsageCountingRecordRepo) StorageUsage(ctx context.Context) ([]repository.BackupStorageUsageItem, error) {
|
||||
r.mu.Lock()
|
||||
r.usageCalls++
|
||||
r.mu.Unlock()
|
||||
return r.BackupRecordRepository.StorageUsage(ctx)
|
||||
}
|
||||
|
||||
type storageUsageFailingRecordRepo struct {
|
||||
repository.BackupRecordRepository
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *storageUsageFailingRecordRepo) StorageUsage(context.Context) ([]repository.BackupStorageUsageItem, error) {
|
||||
return nil, r.err
|
||||
}
|
||||
|
||||
var errStorageUsageFailed = errors.New("storage usage failed")
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"backupx/server/internal/backup"
|
||||
"backupx/server/internal/config"
|
||||
"backupx/server/internal/database"
|
||||
"backupx/server/internal/logger"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
"backupx/server/internal/storage"
|
||||
"backupx/server/internal/storage/codec"
|
||||
storageRclone "backupx/server/internal/storage/rclone"
|
||||
)
|
||||
|
||||
func newLockTestHarness(t *testing.T) (*BackupRecordService, *BackupExecutionService) {
|
||||
t.Helper()
|
||||
baseDir := t.TempDir()
|
||||
sourceDir := filepath.Join(baseDir, "data")
|
||||
storeDir := filepath.Join(baseDir, "store")
|
||||
if err := os.MkdirAll(sourceDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(sourceDir, "f.txt"), []byte("lock-data"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
log, err := logger.New(config.LogConfig{Level: "error"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := database.Open(config.DatabaseConfig{Path: filepath.Join(baseDir, "backupx.db")}, log)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cipher := codec.NewConfigCipher("lock-secret")
|
||||
targets := repository.NewStorageTargetRepository(db)
|
||||
tasks := repository.NewBackupTaskRepository(db)
|
||||
records := repository.NewBackupRecordRepository(db)
|
||||
cfg, err := cipher.EncryptJSON(map[string]any{"basePath": storeDir})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := targets.Create(context.Background(), &model.StorageTarget{Name: "s", Type: string(storage.ProviderTypeLocalDisk), Enabled: true, ConfigCiphertext: cfg, ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := &model.BackupTask{Name: "lock-task", Type: "file", Enabled: true, SourcePath: sourceDir, StorageTargetID: 1, NodeID: 0, RetentionDays: 30, Compression: "gzip", MaxBackups: 10, LastStatus: "idle"}
|
||||
if err := tasks.Create(context.Background(), task); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
logHub := backup.NewLogHub()
|
||||
runnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil))
|
||||
storageRegistry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
|
||||
execution := NewBackupExecutionService(tasks, records, targets, storageRegistry, runnerRegistry, logHub, nil, cipher, nil, baseDir, 2, 10, "")
|
||||
recordService := NewBackupRecordService(records, execution, logHub)
|
||||
return recordService, execution
|
||||
}
|
||||
|
||||
// TestBackupRecordLock_BlocksDeletion 验证保留锁定后手动删除被拒绝,解锁后可删除。
|
||||
func TestBackupRecordLock_BlocksDeletion(t *testing.T) {
|
||||
recordService, execution := newLockTestHarness(t)
|
||||
ctx := context.Background()
|
||||
|
||||
bd, err := execution.RunTaskByIDSync(ctx, 1)
|
||||
if err != nil {
|
||||
t.Fatalf("RunTaskByIDSync: %v", err)
|
||||
}
|
||||
if bd.Status != "success" {
|
||||
t.Fatalf("backup not success: %s", bd.Status)
|
||||
}
|
||||
|
||||
// 锁定。
|
||||
detail, err := recordService.SetLock(ctx, bd.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("SetLock(true): %v", err)
|
||||
}
|
||||
if !detail.Locked {
|
||||
t.Fatal("expected detail.Locked = true")
|
||||
}
|
||||
|
||||
// 锁定状态下删除应被拒绝。
|
||||
if err := execution.DeleteRecord(ctx, bd.ID); err == nil {
|
||||
t.Fatal("expected delete of locked record to be rejected")
|
||||
} else if !strings.Contains(err.Error(), "保留锁定") {
|
||||
t.Fatalf("unexpected delete error: %v", err)
|
||||
}
|
||||
|
||||
// 记录仍然存在。
|
||||
if got, _ := recordService.Get(ctx, bd.ID); got == nil {
|
||||
t.Fatal("locked record must still exist after rejected delete")
|
||||
}
|
||||
|
||||
// 解锁后可删除。
|
||||
if _, err := recordService.SetLock(ctx, bd.ID, false); err != nil {
|
||||
t.Fatalf("SetLock(false): %v", err)
|
||||
}
|
||||
if err := execution.DeleteRecord(ctx, bd.ID); err != nil {
|
||||
t.Fatalf("delete after unlock should succeed: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -3,7 +3,6 @@ package service
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -23,30 +22,27 @@ type BackupRecordListInput struct {
|
||||
}
|
||||
|
||||
type BackupRecordSummary struct {
|
||||
ID uint `json:"id"`
|
||||
TaskID uint `json:"taskId"`
|
||||
TaskName string `json:"taskName"`
|
||||
StorageTargetID uint `json:"storageTargetId"`
|
||||
StorageTargetName string `json:"storageTargetName"`
|
||||
Status string `json:"status"`
|
||||
FileName string `json:"fileName"`
|
||||
FileSize int64 `json:"fileSize"`
|
||||
Checksum string `json:"checksum"`
|
||||
StoragePath string `json:"storagePath"`
|
||||
StorageTransferMode string `json:"storageTransferMode,omitempty"`
|
||||
DurationSeconds int `json:"durationSeconds"`
|
||||
ErrorMessage string `json:"errorMessage"`
|
||||
StartedAt time.Time `json:"startedAt"`
|
||||
CompletedAt *time.Time `json:"completedAt,omitempty"`
|
||||
Locked bool `json:"locked"`
|
||||
BackupKind string `json:"backupKind"`
|
||||
ID uint `json:"id"`
|
||||
TaskID uint `json:"taskId"`
|
||||
TaskName string `json:"taskName"`
|
||||
StorageTargetID uint `json:"storageTargetId"`
|
||||
StorageTargetName string `json:"storageTargetName"`
|
||||
Status string `json:"status"`
|
||||
FileName string `json:"fileName"`
|
||||
FileSize int64 `json:"fileSize"`
|
||||
Checksum string `json:"checksum"`
|
||||
StoragePath string `json:"storagePath"`
|
||||
DurationSeconds int `json:"durationSeconds"`
|
||||
ErrorMessage string `json:"errorMessage"`
|
||||
StartedAt time.Time `json:"startedAt"`
|
||||
CompletedAt *time.Time `json:"completedAt,omitempty"`
|
||||
}
|
||||
|
||||
type BackupRecordDetail struct {
|
||||
BackupRecordSummary
|
||||
LogContent string `json:"logContent"`
|
||||
LogEvents []backup.LogEvent `json:"logEvents,omitempty"`
|
||||
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
|
||||
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
|
||||
}
|
||||
|
||||
type BackupRecordService struct {
|
||||
@@ -82,64 +78,6 @@ func (s *BackupRecordService) Get(ctx context.Context, id uint) (*BackupRecordDe
|
||||
return toBackupRecordDetail(item, s.logHub), nil
|
||||
}
|
||||
|
||||
// BackupContentEntry 描述备份内单个条目(文件或目录),用于内容浏览。
|
||||
type BackupContentEntry struct {
|
||||
Path string `json:"path"`
|
||||
Size int64 `json:"size"`
|
||||
IsDir bool `json:"isDir"`
|
||||
}
|
||||
|
||||
// BackupRecordContents 是一次备份的内容清单视图。
|
||||
type BackupRecordContents struct {
|
||||
RecordID uint `json:"recordId"`
|
||||
Total int `json:"total"`
|
||||
Truncated bool `json:"truncated"`
|
||||
BasedOnFull uint `json:"basedOnFull,omitempty"` // 差异记录时,清单取自该基线全量
|
||||
Entries []BackupContentEntry `json:"entries"`
|
||||
}
|
||||
|
||||
const backupContentsMaxEntries = 10000
|
||||
|
||||
// ListContents 返回某备份记录的文件清单(仅文件类型的新全量备份会记录清单)。
|
||||
// 差异记录回退到其基线全量的清单,近似展示恢复后的目录结构。无清单时返回明确错误。
|
||||
func (s *BackupRecordService) ListContents(ctx context.Context, id uint) (*BackupRecordContents, error) {
|
||||
item, err := s.records.FindByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_GET_FAILED", "无法获取备份记录", err)
|
||||
}
|
||||
if item == nil {
|
||||
return nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", nil)
|
||||
}
|
||||
manifestJSON := item.Manifest
|
||||
basedOnFull := uint(0)
|
||||
if strings.TrimSpace(manifestJSON) == "" && item.BaseRecordID != 0 {
|
||||
if base, baseErr := s.records.FindByID(ctx, item.BaseRecordID); baseErr == nil && base != nil {
|
||||
manifestJSON = base.Manifest
|
||||
basedOnFull = base.ID
|
||||
}
|
||||
}
|
||||
if strings.TrimSpace(manifestJSON) == "" {
|
||||
return nil, apperror.New(422, "BACKUP_CONTENTS_UNAVAILABLE", "该备份未记录文件清单(仅文件类型的新全量备份支持内容浏览),请重新执行一次全量备份后再试。", nil)
|
||||
}
|
||||
manifest, decErr := backup.DecodeManifest([]byte(manifestJSON))
|
||||
if decErr != nil {
|
||||
return nil, apperror.Internal("BACKUP_CONTENTS_DECODE_FAILED", "解析备份清单失败", decErr)
|
||||
}
|
||||
entries := manifest.Entries
|
||||
sort.Slice(entries, func(i, j int) bool { return entries[i].Path < entries[j].Path })
|
||||
total := len(entries)
|
||||
truncated := false
|
||||
if total > backupContentsMaxEntries {
|
||||
entries = entries[:backupContentsMaxEntries]
|
||||
truncated = true
|
||||
}
|
||||
result := &BackupRecordContents{RecordID: item.ID, Total: total, Truncated: truncated, BasedOnFull: basedOnFull, Entries: make([]BackupContentEntry, 0, len(entries))}
|
||||
for _, e := range entries {
|
||||
result.Entries = append(result.Entries, BackupContentEntry{Path: e.Path, Size: e.Size, IsDir: e.IsDir})
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (s *BackupRecordService) SubscribeLogs(ctx context.Context, id uint, buffer int) (<-chan backup.LogEvent, func(), error) {
|
||||
item, err := s.records.FindByID(ctx, id)
|
||||
if err != nil {
|
||||
@@ -164,44 +102,22 @@ func (s *BackupRecordService) Delete(ctx context.Context, id uint) error {
|
||||
return s.execution.DeleteRecord(ctx, id)
|
||||
}
|
||||
|
||||
// SetLock 设置或解除备份记录的保留锁定(法律保留)。
|
||||
// 锁定后该记录免于保留期/数量自动清理,且禁止手动删除,直至显式解锁。
|
||||
func (s *BackupRecordService) SetLock(ctx context.Context, id uint, locked bool) (*BackupRecordDetail, error) {
|
||||
item, err := s.records.FindByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_GET_FAILED", "无法获取备份记录详情", err)
|
||||
}
|
||||
if item == nil {
|
||||
return nil, apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "备份记录不存在", nil)
|
||||
}
|
||||
if item.Locked != locked {
|
||||
item.Locked = locked
|
||||
if err := s.records.Update(ctx, item); err != nil {
|
||||
return nil, apperror.Internal("BACKUP_RECORD_LOCK_FAILED", "无法更新备份锁定状态", err)
|
||||
}
|
||||
}
|
||||
return toBackupRecordDetail(item, s.logHub), nil
|
||||
}
|
||||
|
||||
func toBackupRecordSummary(item *model.BackupRecord) BackupRecordSummary {
|
||||
return BackupRecordSummary{
|
||||
ID: item.ID,
|
||||
TaskID: item.TaskID,
|
||||
TaskName: item.Task.Name,
|
||||
StorageTargetID: item.StorageTargetID,
|
||||
StorageTargetName: item.StorageTarget.Name,
|
||||
Status: item.Status,
|
||||
FileName: item.FileName,
|
||||
FileSize: item.FileSize,
|
||||
Checksum: item.Checksum,
|
||||
StoragePath: item.StoragePath,
|
||||
StorageTransferMode: item.StorageTransferMode,
|
||||
DurationSeconds: item.DurationSeconds,
|
||||
ErrorMessage: item.ErrorMessage,
|
||||
StartedAt: item.StartedAt,
|
||||
CompletedAt: item.CompletedAt,
|
||||
Locked: item.Locked,
|
||||
BackupKind: item.BackupKind,
|
||||
ID: item.ID,
|
||||
TaskID: item.TaskID,
|
||||
TaskName: item.Task.Name,
|
||||
StorageTargetID: item.StorageTargetID,
|
||||
StorageTargetName: item.StorageTarget.Name,
|
||||
Status: item.Status,
|
||||
FileName: item.FileName,
|
||||
FileSize: item.FileSize,
|
||||
Checksum: item.Checksum,
|
||||
StoragePath: item.StoragePath,
|
||||
DurationSeconds: item.DurationSeconds,
|
||||
ErrorMessage: item.ErrorMessage,
|
||||
StartedAt: item.StartedAt,
|
||||
CompletedAt: item.CompletedAt,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ const backupTaskMaskedValue = "********"
|
||||
|
||||
type BackupTaskUpsertInput struct {
|
||||
Name string `json:"name" binding:"required,min=1,max=100"`
|
||||
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql saphana mongodb"`
|
||||
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql saphana"`
|
||||
Enabled bool `json:"enabled"`
|
||||
CronExpr string `json:"cronExpr" binding:"max=64"`
|
||||
SourcePath string `json:"sourcePath" binding:"max=500"`
|
||||
@@ -33,16 +33,16 @@ type BackupTaskUpsertInput struct {
|
||||
DBPassword string `json:"dbPassword" binding:"max=255"`
|
||||
DBName string `json:"dbName" binding:"max=255"`
|
||||
DBPath string `json:"dbPath" binding:"max=500"`
|
||||
StorageTargetID uint `json:"storageTargetId"` // deprecated: 向后兼容
|
||||
StorageTargetIDs []uint `json:"storageTargetIds"` // 新增:多存储目标
|
||||
NodeID uint `json:"nodeId"` // 执行节点(0 = 本机 Master 或节点池)
|
||||
StorageTargetID uint `json:"storageTargetId"` // deprecated: 向后兼容
|
||||
StorageTargetIDs []uint `json:"storageTargetIds"` // 新增:多存储目标
|
||||
NodeID uint `json:"nodeId"` // 执行节点(0 = 本机 Master 或节点池)
|
||||
// NodePoolTag 节点池标签。NodeID=0 且本字段非空时,调度器动态从 Labels 命中的在线节点中选负载最低者。
|
||||
NodePoolTag string `json:"nodePoolTag" binding:"max=64"`
|
||||
Tags string `json:"tags" binding:"max=500"` // 逗号分隔标签
|
||||
RetentionDays int `json:"retentionDays"`
|
||||
Compression string `json:"compression" binding:"omitempty,oneof=gzip zstd none"`
|
||||
Encrypt bool `json:"encrypt"`
|
||||
MaxBackups int `json:"maxBackups"`
|
||||
NodePoolTag string `json:"nodePoolTag" binding:"max=64"`
|
||||
Tags string `json:"tags" binding:"max=500"` // 逗号分隔标签
|
||||
RetentionDays int `json:"retentionDays"`
|
||||
Compression string `json:"compression" binding:"omitempty,oneof=gzip none"`
|
||||
Encrypt bool `json:"encrypt"`
|
||||
MaxBackups int `json:"maxBackups"`
|
||||
// ExtraConfig 类型特有扩展配置(如 SAP HANA 的 backupLevel/backupChannels)
|
||||
ExtraConfig map[string]any `json:"extraConfig"`
|
||||
// 验证(恢复演练)配置
|
||||
@@ -52,14 +52,6 @@ type BackupTaskUpsertInput struct {
|
||||
// SLA 配置
|
||||
SLAHoursRPO int `json:"slaHoursRpo"`
|
||||
AlertOnConsecutiveFails int `json:"alertOnConsecutiveFails"`
|
||||
// GFS 分层保留(任一 > 0 启用,取代 RetentionDays/MaxBackups)
|
||||
KeepDaily int `json:"keepDaily"`
|
||||
KeepWeekly int `json:"keepWeekly"`
|
||||
KeepMonthly int `json:"keepMonthly"`
|
||||
KeepYearly int `json:"keepYearly"`
|
||||
// BackupMode 备份模式:full(默认)/ differential(差异归档)/ repository(CDC 去重仓库)
|
||||
BackupMode string `json:"backupMode" binding:"omitempty,oneof=full differential repository"`
|
||||
DiffFullIntervalDays int `json:"diffFullIntervalDays"`
|
||||
// 备份复制目标存储 ID 列表(3-2-1 规则)
|
||||
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
|
||||
// 维护窗口(CSV,详见 backup/window.go)
|
||||
@@ -78,8 +70,8 @@ type BackupTaskSummary struct {
|
||||
Type string `json:"type"`
|
||||
Enabled bool `json:"enabled"`
|
||||
CronExpr string `json:"cronExpr"`
|
||||
StorageTargetID uint `json:"storageTargetId"` // deprecated: 取第一个
|
||||
StorageTargetName string `json:"storageTargetName"` // deprecated: 取第一个
|
||||
StorageTargetID uint `json:"storageTargetId"` // deprecated: 取第一个
|
||||
StorageTargetName string `json:"storageTargetName"` // deprecated: 取第一个
|
||||
StorageTargetIDs []uint `json:"storageTargetIds"`
|
||||
StorageTargetNames []string `json:"storageTargetNames"`
|
||||
NodeID uint `json:"nodeId"`
|
||||
@@ -98,17 +90,11 @@ type BackupTaskSummary struct {
|
||||
VerifyMode string `json:"verifyMode"`
|
||||
SLAHoursRPO int `json:"slaHoursRpo"`
|
||||
AlertOnConsecutiveFails int `json:"alertOnConsecutiveFails"`
|
||||
KeepDaily int `json:"keepDaily"`
|
||||
KeepWeekly int `json:"keepWeekly"`
|
||||
KeepMonthly int `json:"keepMonthly"`
|
||||
KeepYearly int `json:"keepYearly"`
|
||||
BackupMode string `json:"backupMode"`
|
||||
DiffFullIntervalDays int `json:"diffFullIntervalDays"`
|
||||
// 备份复制目标(3-2-1)
|
||||
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
|
||||
MaintenanceWindows string `json:"maintenanceWindows"`
|
||||
DependsOnTaskIDs []uint `json:"dependsOnTaskIds"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
ReplicationTargetIDs []uint `json:"replicationTargetIds"`
|
||||
MaintenanceWindows string `json:"maintenanceWindows"`
|
||||
DependsOnTaskIDs []uint `json:"dependsOnTaskIds"`
|
||||
UpdatedAt time.Time `json:"updatedAt"`
|
||||
}
|
||||
|
||||
type BackupTaskDetail struct {
|
||||
@@ -414,50 +400,21 @@ func (s *BackupTaskService) cleanupRemoteFiles(ctx context.Context, taskID uint)
|
||||
recordCount = len(records)
|
||||
// 缓存 provider 避免同一存储目标重复创建连接
|
||||
providerCache := make(map[uint]storage.StorageProvider)
|
||||
repositoryProviders := make(map[uint]storage.StorageProvider)
|
||||
for _, record := range records {
|
||||
copies := []StorageUploadResultItem{{
|
||||
StorageTargetID: record.StorageTargetID,
|
||||
Status: model.BackupRecordStatusSuccess,
|
||||
StoragePath: record.StoragePath,
|
||||
}}
|
||||
if strings.TrimSpace(record.StorageUploadResults) != "" {
|
||||
var storedCopies []StorageUploadResultItem
|
||||
if unmarshalErr := json.Unmarshal([]byte(record.StorageUploadResults), &storedCopies); unmarshalErr == nil {
|
||||
copies = storedCopies
|
||||
}
|
||||
}
|
||||
seenTargets := make(map[uint]struct{}, len(copies))
|
||||
for _, copy := range copies {
|
||||
if !strings.EqualFold(copy.Status, model.BackupRecordStatusSuccess) || strings.TrimSpace(copy.StoragePath) == "" {
|
||||
continue
|
||||
}
|
||||
if _, seen := seenTargets[copy.StorageTargetID]; seen {
|
||||
continue
|
||||
}
|
||||
seenTargets[copy.StorageTargetID] = struct{}{}
|
||||
provider, ok := providerCache[copy.StorageTargetID]
|
||||
if !ok {
|
||||
provider, err = s.resolveStorageProvider(ctx, copy.StorageTargetID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
providerCache[copy.StorageTargetID] = provider
|
||||
}
|
||||
if err := provider.Delete(ctx, copy.StoragePath); err == nil {
|
||||
cleanedFiles++
|
||||
if record.BackupKind == model.BackupKindRepository {
|
||||
repositoryProviders[copy.StorageTargetID] = provider
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, provider := range repositoryProviders {
|
||||
pruned, pruneErr := backup.NewRepositoryStore(s.cipher.Key()).Prune(ctx, provider)
|
||||
if pruneErr != nil {
|
||||
if strings.TrimSpace(record.StoragePath) == "" {
|
||||
continue
|
||||
}
|
||||
cleanedFiles += pruned.DeletedIndexes + pruned.DeletedPacks
|
||||
provider, ok := providerCache[record.StorageTargetID]
|
||||
if !ok {
|
||||
provider, err = s.resolveStorageProvider(ctx, record.StorageTargetID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
providerCache[record.StorageTargetID] = provider
|
||||
}
|
||||
if err := provider.Delete(ctx, record.StoragePath); err == nil {
|
||||
cleanedFiles++
|
||||
}
|
||||
}
|
||||
return recordCount, cleanedFiles
|
||||
}
|
||||
@@ -531,7 +488,6 @@ func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.B
|
||||
return apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", fmt.Sprintf("关联的存储目标 %d 不存在", tid), nil)
|
||||
}
|
||||
}
|
||||
var fixedNode *model.Node
|
||||
if input.NodeID > 0 && s.nodes != nil {
|
||||
node, err := s.nodes.FindByID(ctx, input.NodeID)
|
||||
if err != nil {
|
||||
@@ -540,36 +496,12 @@ func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.B
|
||||
if node == nil {
|
||||
return apperror.BadRequest("BACKUP_TASK_INVALID", "所选执行节点不存在", nil)
|
||||
}
|
||||
fixedNode = node
|
||||
}
|
||||
// 节点池与固定节点互斥:固定节点已确定执行位置,不再动态调度
|
||||
if input.NodeID > 0 && strings.TrimSpace(input.NodePoolTag) != "" {
|
||||
return apperror.BadRequest("BACKUP_TASK_INVALID",
|
||||
"固定执行节点与节点池标签只能选其一", nil)
|
||||
}
|
||||
if input.Encrypt && (strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal)) {
|
||||
return apperror.BadRequest("BACKUP_TASK_REMOTE_ENCRYPT_UNSUPPORTED",
|
||||
"远程节点暂不支持加密备份。请关闭加密,或将任务固定在 Master 本机执行。", nil)
|
||||
}
|
||||
if strings.EqualFold(strings.TrimSpace(input.BackupMode), model.BackupModeDifferential) {
|
||||
if input.Type != model.BackupTaskTypeFile {
|
||||
return apperror.BadRequest("BACKUP_TASK_DIFF_UNSUPPORTED", "差异备份仅支持文件目录类型任务", nil)
|
||||
}
|
||||
if strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal) {
|
||||
return apperror.BadRequest("BACKUP_TASK_DIFF_REMOTE_UNSUPPORTED", "差异备份当前仅支持本机 Master 执行,请将任务固定在本机或改用全量备份。", nil)
|
||||
}
|
||||
}
|
||||
if strings.EqualFold(strings.TrimSpace(input.BackupMode), model.BackupModeRepository) {
|
||||
if input.Type != model.BackupTaskTypeFile {
|
||||
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_UNSUPPORTED", "CDC 仓库模式仅支持文件目录类型任务", nil)
|
||||
}
|
||||
if strings.TrimSpace(input.NodePoolTag) != "" || (fixedNode != nil && !fixedNode.IsLocal) {
|
||||
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_REMOTE_UNSUPPORTED", "CDC 仓库模式当前采用单写者索引,仅支持 Master 本机执行。远程服务器备份请暂用全量模式。", nil)
|
||||
}
|
||||
if len(input.ReplicationTargetIDs) > 0 {
|
||||
return apperror.BadRequest("BACKUP_TASK_REPOSITORY_REPLICATION_UNSUPPORTED", "CDC 仓库快照不能使用对象级复制;请直接为任务选择多个存储目标以生成完整仓库副本。", nil)
|
||||
}
|
||||
}
|
||||
if input.RetentionDays < 0 {
|
||||
return apperror.BadRequest("BACKUP_TASK_INVALID", "保留天数不能小于 0", nil)
|
||||
}
|
||||
@@ -631,7 +563,7 @@ func validateTaskTypeSpecificFields(input BackupTaskUpsertInput, passwordRequire
|
||||
if !hasSourcePaths {
|
||||
return apperror.BadRequest("BACKUP_TASK_INVALID", "文件备份必须填写源路径", nil)
|
||||
}
|
||||
case "mysql", "postgresql", "saphana", "mongodb":
|
||||
case "mysql", "postgresql", "saphana":
|
||||
if strings.TrimSpace(input.DBHost) == "" {
|
||||
return apperror.BadRequest("BACKUP_TASK_INVALID", "数据库主机不能为空", nil)
|
||||
}
|
||||
@@ -707,44 +639,38 @@ func (s *BackupTaskService) buildTask(existing *model.BackupTask, input BackupTa
|
||||
return nil, apperror.BadRequest("BACKUP_TASK_INVALID", "扩展配置格式不合法", err)
|
||||
}
|
||||
item := &model.BackupTask{
|
||||
Name: strings.TrimSpace(input.Name),
|
||||
Type: normalizeBackupTaskType(input.Type),
|
||||
Enabled: input.Enabled,
|
||||
CronExpr: strings.TrimSpace(input.CronExpr),
|
||||
SourcePath: primarySourcePath,
|
||||
SourcePaths: sourcePathsJSON,
|
||||
ExcludePatterns: excludePatterns,
|
||||
DBHost: strings.TrimSpace(input.DBHost),
|
||||
DBPort: input.DBPort,
|
||||
DBUser: strings.TrimSpace(input.DBUser),
|
||||
DBPasswordCiphertext: passwordCiphertext,
|
||||
DBName: strings.TrimSpace(input.DBName),
|
||||
DBPath: strings.TrimSpace(input.DBPath),
|
||||
ExtraConfig: extraConfigJSON,
|
||||
StorageTargetID: primaryTargetID,
|
||||
StorageTargets: storageTargets,
|
||||
NodeID: input.NodeID,
|
||||
NodePoolTag: strings.TrimSpace(input.NodePoolTag),
|
||||
Tags: strings.TrimSpace(input.Tags),
|
||||
RetentionDays: input.RetentionDays,
|
||||
Compression: compression,
|
||||
Encrypt: input.Encrypt,
|
||||
MaxBackups: maxBackups,
|
||||
LastStatus: "idle",
|
||||
VerifyEnabled: input.VerifyEnabled,
|
||||
VerifyCronExpr: strings.TrimSpace(input.VerifyCronExpr),
|
||||
VerifyMode: normalizeVerifyMode(input.VerifyMode),
|
||||
SLAHoursRPO: maxInt(0, input.SLAHoursRPO),
|
||||
Name: strings.TrimSpace(input.Name),
|
||||
Type: normalizeBackupTaskType(input.Type),
|
||||
Enabled: input.Enabled,
|
||||
CronExpr: strings.TrimSpace(input.CronExpr),
|
||||
SourcePath: primarySourcePath,
|
||||
SourcePaths: sourcePathsJSON,
|
||||
ExcludePatterns: excludePatterns,
|
||||
DBHost: strings.TrimSpace(input.DBHost),
|
||||
DBPort: input.DBPort,
|
||||
DBUser: strings.TrimSpace(input.DBUser),
|
||||
DBPasswordCiphertext: passwordCiphertext,
|
||||
DBName: strings.TrimSpace(input.DBName),
|
||||
DBPath: strings.TrimSpace(input.DBPath),
|
||||
ExtraConfig: extraConfigJSON,
|
||||
StorageTargetID: primaryTargetID,
|
||||
StorageTargets: storageTargets,
|
||||
NodeID: input.NodeID,
|
||||
NodePoolTag: strings.TrimSpace(input.NodePoolTag),
|
||||
Tags: strings.TrimSpace(input.Tags),
|
||||
RetentionDays: input.RetentionDays,
|
||||
Compression: compression,
|
||||
Encrypt: input.Encrypt,
|
||||
MaxBackups: maxBackups,
|
||||
LastStatus: "idle",
|
||||
VerifyEnabled: input.VerifyEnabled,
|
||||
VerifyCronExpr: strings.TrimSpace(input.VerifyCronExpr),
|
||||
VerifyMode: normalizeVerifyMode(input.VerifyMode),
|
||||
SLAHoursRPO: maxInt(0, input.SLAHoursRPO),
|
||||
AlertOnConsecutiveFails: alertThreshold(input.AlertOnConsecutiveFails),
|
||||
KeepDaily: maxInt(0, input.KeepDaily),
|
||||
KeepWeekly: maxInt(0, input.KeepWeekly),
|
||||
KeepMonthly: maxInt(0, input.KeepMonthly),
|
||||
KeepYearly: maxInt(0, input.KeepYearly),
|
||||
BackupMode: normalizeBackupMode(input.BackupMode, input.Type),
|
||||
DiffFullIntervalDays: diffFullInterval(input.DiffFullIntervalDays),
|
||||
ReplicationTargetIDs: encodeUintCSV(input.ReplicationTargetIDs),
|
||||
MaintenanceWindows: strings.TrimSpace(input.MaintenanceWindows),
|
||||
DependsOnTaskIDs: encodeUintCSV(input.DependsOnTaskIDs),
|
||||
ReplicationTargetIDs: encodeUintCSV(input.ReplicationTargetIDs),
|
||||
MaintenanceWindows: strings.TrimSpace(input.MaintenanceWindows),
|
||||
DependsOnTaskIDs: encodeUintCSV(input.DependsOnTaskIDs),
|
||||
}
|
||||
if existing != nil {
|
||||
item.LastRunAt = existing.LastRunAt
|
||||
@@ -810,40 +736,34 @@ func toBackupTaskSummary(item *model.BackupTask) BackupTaskSummary {
|
||||
primaryName = targetNames[0]
|
||||
}
|
||||
return BackupTaskSummary{
|
||||
ID: item.ID,
|
||||
Name: item.Name,
|
||||
Type: normalizeBackupTaskType(item.Type),
|
||||
Enabled: item.Enabled,
|
||||
CronExpr: item.CronExpr,
|
||||
StorageTargetID: primaryID,
|
||||
StorageTargetName: primaryName,
|
||||
StorageTargetIDs: targetIDs,
|
||||
StorageTargetNames: targetNames,
|
||||
NodeID: item.NodeID,
|
||||
NodeName: item.Node.Name,
|
||||
NodePoolTag: item.NodePoolTag,
|
||||
Tags: item.Tags,
|
||||
RetentionDays: item.RetentionDays,
|
||||
Compression: item.Compression,
|
||||
Encrypt: item.Encrypt,
|
||||
MaxBackups: item.MaxBackups,
|
||||
LastRunAt: item.LastRunAt,
|
||||
LastStatus: item.LastStatus,
|
||||
ID: item.ID,
|
||||
Name: item.Name,
|
||||
Type: normalizeBackupTaskType(item.Type),
|
||||
Enabled: item.Enabled,
|
||||
CronExpr: item.CronExpr,
|
||||
StorageTargetID: primaryID,
|
||||
StorageTargetName: primaryName,
|
||||
StorageTargetIDs: targetIDs,
|
||||
StorageTargetNames: targetNames,
|
||||
NodeID: item.NodeID,
|
||||
NodeName: item.Node.Name,
|
||||
NodePoolTag: item.NodePoolTag,
|
||||
Tags: item.Tags,
|
||||
RetentionDays: item.RetentionDays,
|
||||
Compression: item.Compression,
|
||||
Encrypt: item.Encrypt,
|
||||
MaxBackups: item.MaxBackups,
|
||||
LastRunAt: item.LastRunAt,
|
||||
LastStatus: item.LastStatus,
|
||||
VerifyEnabled: item.VerifyEnabled,
|
||||
VerifyCronExpr: item.VerifyCronExpr,
|
||||
VerifyMode: item.VerifyMode,
|
||||
SLAHoursRPO: item.SLAHoursRPO,
|
||||
AlertOnConsecutiveFails: item.AlertOnConsecutiveFails,
|
||||
KeepDaily: item.KeepDaily,
|
||||
KeepWeekly: item.KeepWeekly,
|
||||
KeepMonthly: item.KeepMonthly,
|
||||
KeepYearly: item.KeepYearly,
|
||||
BackupMode: item.BackupMode,
|
||||
DiffFullIntervalDays: item.DiffFullIntervalDays,
|
||||
ReplicationTargetIDs: parseUintCSV(item.ReplicationTargetIDs),
|
||||
MaintenanceWindows: item.MaintenanceWindows,
|
||||
DependsOnTaskIDs: parseUintCSV(item.DependsOnTaskIDs),
|
||||
UpdatedAt: item.UpdatedAt,
|
||||
UpdatedAt: item.UpdatedAt,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -975,29 +895,6 @@ func decodeExtraConfig(value string) (map[string]any, error) {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// normalizeBackupMode 归一化备份模式:仅文件类型可启用差异或 CDC 仓库,
|
||||
// 其余一律全量(双保险,防绕过校验)。
|
||||
func normalizeBackupMode(mode, taskType string) string {
|
||||
if normalizeBackupTaskType(taskType) != model.BackupTaskTypeFile {
|
||||
return model.BackupModeFull
|
||||
}
|
||||
switch strings.ToLower(strings.TrimSpace(mode)) {
|
||||
case model.BackupModeDifferential:
|
||||
return model.BackupModeDifferential
|
||||
case model.BackupModeRepository:
|
||||
return model.BackupModeRepository
|
||||
}
|
||||
return model.BackupModeFull
|
||||
}
|
||||
|
||||
// diffFullInterval 归一化差异模式下的强制全量间隔(天),非正值回退默认 7。
|
||||
func diffFullInterval(days int) int {
|
||||
if days <= 0 {
|
||||
return 7
|
||||
}
|
||||
return days
|
||||
}
|
||||
|
||||
func normalizeBackupTaskType(value string) string {
|
||||
normalized := strings.TrimSpace(strings.ToLower(value))
|
||||
if normalized == "pgsql" {
|
||||
|
||||
@@ -3,7 +3,6 @@ package service
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"backupx/server/internal/config"
|
||||
@@ -30,82 +29,6 @@ func newBackupTaskServiceForTest(t *testing.T) (*BackupTaskService, repository.S
|
||||
return service, targets, tasks
|
||||
}
|
||||
|
||||
func TestBackupTaskServiceRejectsEncryptedRemoteTasks(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
service, targets, _ := newBackupTaskServiceForTest(t)
|
||||
service.SetNodeRepository(&nodeRepoStub{nodes: []model.Node{
|
||||
{ID: 41, Name: "master", Token: "master-token", Status: model.NodeStatusOnline, IsLocal: true},
|
||||
{ID: 42, Name: "edge", Token: "edge-token", Status: model.NodeStatusOnline, IsLocal: false},
|
||||
}})
|
||||
if err := targets.Create(ctx, &model.StorageTarget{Name: "local", Type: "local_disk", Enabled: true, ConfigCiphertext: "ciphertext", ConfigVersion: 1, LastTestStatus: "unknown"}); err != nil {
|
||||
t.Fatalf("seed storage target error: %v", err)
|
||||
}
|
||||
|
||||
_, err := service.Create(ctx, BackupTaskUpsertInput{
|
||||
Name: "encrypted-node-pool",
|
||||
Type: "file",
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/site",
|
||||
StorageTargetID: 1,
|
||||
NodePoolTag: "db",
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
Encrypt: true,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "远程节点暂不支持加密备份") {
|
||||
t.Fatalf("expected encrypted node-pool task to be rejected, got %v", err)
|
||||
}
|
||||
|
||||
created, err := service.Create(ctx, BackupTaskUpsertInput{
|
||||
Name: "local-encrypted",
|
||||
Type: "file",
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/site",
|
||||
StorageTargetID: 1,
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
Encrypt: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Create local encrypted task returned error: %v", err)
|
||||
}
|
||||
localNodeTask, err := service.Create(ctx, BackupTaskUpsertInput{
|
||||
Name: "local-node-encrypted",
|
||||
Type: "file",
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/site",
|
||||
StorageTargetID: 1,
|
||||
NodeID: 41,
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
Encrypt: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Create encrypted task pinned to local node returned error: %v", err)
|
||||
}
|
||||
if localNodeTask.NodeID != 41 || !localNodeTask.Encrypt {
|
||||
t.Fatalf("expected encrypted task to keep local node, got %#v", localNodeTask)
|
||||
}
|
||||
_, err = service.Update(ctx, created.ID, BackupTaskUpsertInput{
|
||||
Name: created.Name,
|
||||
Type: created.Type,
|
||||
Enabled: true,
|
||||
SourcePath: "/srv/site",
|
||||
StorageTargetID: 1,
|
||||
NodeID: 42,
|
||||
RetentionDays: 30,
|
||||
Compression: "gzip",
|
||||
MaxBackups: 10,
|
||||
Encrypt: true,
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "远程节点暂不支持加密备份") {
|
||||
t.Fatalf("expected encrypted fixed-node update to be rejected, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupTaskServiceCreateAndGet(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
service, targets, _ := newBackupTaskServiceForTest(t)
|
||||
|
||||
@@ -1,219 +0,0 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"backupx/server/internal/apperror"
|
||||
"backupx/server/internal/backup"
|
||||
"backupx/server/internal/model"
|
||||
"backupx/server/internal/repository"
|
||||
"backupx/server/internal/storage"
|
||||
"backupx/server/internal/storage/codec"
|
||||
"backupx/server/pkg/compress"
|
||||
backupcrypto "backupx/server/pkg/crypto"
|
||||
)
|
||||
|
||||
// 本文件集中放置「备份执行 / 恢复 / 验证 / 复制」四个执行服务共享的执行期辅助逻辑。
|
||||
//
|
||||
// 历史上这些函数(解密存储配置创建 provider、按后缀解密解压归档、判定远程节点、
|
||||
// 跨节点 local_disk 保护、构建任务执行规格)在四个服务里各复制了一份,差异仅在
|
||||
// 字段名与少量错误码/日志文案。重复实现既增加维护成本,也容易出现"改了一处忘了
|
||||
// 另一处"的不一致缺陷。这里抽取为单一实现,各服务通过薄封装方法委托调用,调用方
|
||||
// 无需改动。
|
||||
|
||||
// fileSHA256 计算文件内容的 SHA-256(小写 hex),与备份上传时记录到
|
||||
// BackupRecord.Checksum 的格式一致,用于恢复/复制前的完整性校验。
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
|
||||
// verifyArtifactChecksum 校验下载到本地的备份对象与记录的 SHA-256 是否一致。
|
||||
// expected 为空时跳过(兼容早期未记录 checksum 的备份);不一致返回结构化错误,
|
||||
// 调用方应据此中止恢复,避免还原已损坏或被篡改的数据。
|
||||
func verifyArtifactChecksum(path, expected string) error {
|
||||
expected = strings.TrimSpace(expected)
|
||||
if expected == "" {
|
||||
return nil
|
||||
}
|
||||
actual, err := fileSHA256(path)
|
||||
if err != nil {
|
||||
return apperror.Internal("BACKUP_CHECKSUM_READ_FAILED", "无法读取备份文件计算校验和", err)
|
||||
}
|
||||
if !strings.EqualFold(actual, expected) {
|
||||
// 包装错误同样使用中文并附上期望/实际哈希:apperror.Error() 会优先返回包装错误,
|
||||
// 而恢复记录的 ErrorMessage 取自 err.Error(),需保证对用户可读。
|
||||
return apperror.BadRequest("BACKUP_CHECKSUM_MISMATCH",
|
||||
"备份文件完整性校验失败:SHA-256 不匹配,文件可能已损坏或被篡改",
|
||||
fmt.Errorf("备份文件完整性校验失败:SHA-256 不匹配(期望 %s,实际 %s),文件可能已损坏或被篡改", expected, actual))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveStorageProvider 查询存储目标、解密其配置并创建 provider。
|
||||
func resolveStorageProvider(ctx context.Context, targets repository.StorageTargetRepository, registry *storage.Registry, cipher *codec.ConfigCipher, targetID uint) (storage.StorageProvider, error) {
|
||||
target, err := targets.FindByID(ctx, targetID)
|
||||
if err != nil {
|
||||
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
|
||||
}
|
||||
if target == nil {
|
||||
return nil, apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "关联的存储目标不存在", nil)
|
||||
}
|
||||
configMap := map[string]any{}
|
||||
if err := cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
|
||||
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_DECRYPT_FAILED", "无法解密存储目标配置", err)
|
||||
}
|
||||
return registry.Create(ctx, target.Type, configMap)
|
||||
}
|
||||
|
||||
// prepareBackupArtifact 按文件后缀依次解密(.enc)与解压(.gz),返回最终可读路径。
|
||||
// logger 可为 nil(此时静默执行)。
|
||||
func prepareBackupArtifact(cipher *codec.ConfigCipher, artifactPath string, logger *backup.ExecutionLogger) (string, error) {
|
||||
current := artifactPath
|
||||
if strings.HasSuffix(strings.ToLower(current), ".enc") {
|
||||
if logger != nil {
|
||||
logger.Infof("检测到加密后缀,开始解密")
|
||||
}
|
||||
decrypted, err := backupcrypto.DecryptFile(cipher.Key(), current)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
current = decrypted
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(current), ".gz") {
|
||||
if logger != nil {
|
||||
logger.Infof("检测到 gzip 压缩,开始解压")
|
||||
}
|
||||
decompressed, err := compress.GunzipFile(current)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
current = decompressed
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(current), ".zst") {
|
||||
if logger != nil {
|
||||
logger.Infof("检测到 zstd 压缩,开始解压")
|
||||
}
|
||||
decompressed, err := compress.UnzstdFile(current)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
current = decompressed
|
||||
}
|
||||
return current, nil
|
||||
}
|
||||
|
||||
// resolveRemoteExecutionNode 返回远程(非本机)节点指针,用于判定任务应下发给
|
||||
// Agent 还是在 Master 本地执行。clusterEnabled 通常为「该服务是否注入了 Agent
|
||||
// 下发能力」。本机 / 未启用集群 / nodeID=0 / 未找到时返回 nil(走本地执行)。
|
||||
func resolveRemoteExecutionNode(ctx context.Context, nodeRepo repository.NodeRepository, clusterEnabled bool, nodeID uint) *model.Node {
|
||||
if nodeRepo == nil || !clusterEnabled || nodeID == 0 {
|
||||
return nil
|
||||
}
|
||||
node, err := nodeRepo.FindByID(ctx, nodeID)
|
||||
if err != nil || node == nil || node.IsLocal {
|
||||
return nil
|
||||
}
|
||||
return node
|
||||
}
|
||||
|
||||
// validateCrossNodeLocalDisk 跨节点 local_disk 保护:若备份记录归属某远程节点,
|
||||
// 且其存储目标是 local_disk(数据位于该节点本地磁盘),Master 无法跨节点访问,
|
||||
// 直接返回错误。errCode/opName 由各服务定制,以给出贴合场景的提示文案。
|
||||
func validateCrossNodeLocalDisk(ctx context.Context, nodeRepo repository.NodeRepository, targets repository.StorageTargetRepository, record *model.BackupRecord, errCode, opName string) error {
|
||||
if record == nil || record.NodeID == 0 || nodeRepo == nil {
|
||||
return nil
|
||||
}
|
||||
// 中转模式的对象实际落在 Master 配置的本地磁盘,Master 可以安全访问。
|
||||
// 空值和 direct 均按旧版 Agent 本地落盘处理,保持升级兼容。
|
||||
if record.StorageTransferMode == storage.TransferModeMasterRelay {
|
||||
return nil
|
||||
}
|
||||
node, err := nodeRepo.FindByID(ctx, record.NodeID)
|
||||
if err != nil || node == nil || node.IsLocal {
|
||||
return nil
|
||||
}
|
||||
target, err := targets.FindByID(ctx, record.StorageTargetID)
|
||||
if err != nil || target == nil {
|
||||
return nil
|
||||
}
|
||||
if strings.EqualFold(target.Type, "local_disk") {
|
||||
return apperror.BadRequest(errCode,
|
||||
fmt.Sprintf("备份位于节点 %s 的本地磁盘(local_disk),Master 无法跨节点%s。", node.Name, opName),
|
||||
nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// buildBackupTaskSpec 由备份任务构建执行规格:解析排除规则/源路径、解密 DB 密码、
|
||||
// 套用 ExtraConfig(SAP HANA 等类型特有字段)。被备份执行与恢复服务共享。
|
||||
func buildBackupTaskSpec(cipher *codec.ConfigCipher, task *model.BackupTask, startedAt time.Time, tempDir string) (backup.TaskSpec, error) {
|
||||
excludePatterns := []string{}
|
||||
if strings.TrimSpace(task.ExcludePatterns) != "" {
|
||||
if err := json.Unmarshal([]byte(task.ExcludePatterns), &excludePatterns); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析排除规则", err)
|
||||
}
|
||||
}
|
||||
password := ""
|
||||
if strings.TrimSpace(task.DBPasswordCiphertext) != "" {
|
||||
plain, err := cipher.Decrypt(task.DBPasswordCiphertext)
|
||||
if err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECRYPT_FAILED", "无法解密数据库密码", err)
|
||||
}
|
||||
password = string(plain)
|
||||
}
|
||||
sourcePaths := []string{}
|
||||
if strings.TrimSpace(task.SourcePaths) != "" {
|
||||
if err := json.Unmarshal([]byte(task.SourcePaths), &sourcePaths); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
|
||||
}
|
||||
}
|
||||
dbSpec := backup.DatabaseSpec{
|
||||
Host: task.DBHost,
|
||||
Port: task.DBPort,
|
||||
User: task.DBUser,
|
||||
Password: password,
|
||||
Names: []string{task.DBName},
|
||||
Path: task.DBPath,
|
||||
}
|
||||
// 解析 ExtraConfig 填充类型特有字段(目前主要用于 SAP HANA)
|
||||
if strings.TrimSpace(task.ExtraConfig) != "" {
|
||||
extra := map[string]any{}
|
||||
if err := json.Unmarshal([]byte(task.ExtraConfig), &extra); err != nil {
|
||||
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
|
||||
}
|
||||
applyHANAExtraConfig(&dbSpec, extra)
|
||||
}
|
||||
return backup.TaskSpec{
|
||||
ID: task.ID,
|
||||
Name: task.Name,
|
||||
Type: task.Type,
|
||||
SourcePath: task.SourcePath,
|
||||
SourcePaths: sourcePaths,
|
||||
ExcludePatterns: excludePatterns,
|
||||
StorageTargetID: task.StorageTargetID,
|
||||
Compression: task.Compression,
|
||||
Encrypt: task.Encrypt,
|
||||
RetentionDays: task.RetentionDays,
|
||||
MaxBackups: task.MaxBackups,
|
||||
StartedAt: startedAt,
|
||||
TempDir: tempDir,
|
||||
Database: dbSpec,
|
||||
}, nil
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user