Compare commits

...

51 Commits

Author SHA1 Message Date
Wu Qing
81c9c042d6 功能: 修复并实现多节点集群部署 (#38)
基础修复:
- 新增节点离线检测:每 15s 扫描,超 45s 未心跳的远程节点自动置离线
- 节点删除前检查关联任务,避免孤立备份任务
- BackupTaskRepository 新增 CountByNodeID/ListByNodeID

Master 端 Agent 协议:
- 新增 AgentCommand 模型与命令队列仓储(pending/dispatched/succeeded/failed/timeout)
- 新增 AgentService:任务下发、命令轮询、结果回收、超时扫描
- 新增专用 Agent HTTP API(X-Agent-Token 认证):
  /api/agent/heartbeat
  /api/agent/commands/poll
  /api/agent/commands/:id/result
  /api/agent/tasks/:id
  /api/agent/records/:id
- BackupExecutionService 支持 node 路由:task.NodeID 指向远程节点时自动入队派发

Agent CLI(backupx agent 子命令):
- 配置:YAML 文件 / 环境变量 / CLI 参数,优先级 CLI > 文件 > 环境
- 心跳循环 + 命令轮询循环 + 优雅退出
- 本地复用 BackupRunner 与 storage registry 执行备份并直接上传
- 支持 run_task 和 list_dir 两种命令

远程目录浏览:
- NodeService 支持通过 Agent RPC 列出远程节点目录(15s 超时)

前端:
- NodesPage 添加节点后展示 Agent 启动命令和环境变量配置

文档:
- README 中英文重写"多节点集群"章节,含架构图、步骤、限制、CLI 参考
2026-04-17 12:29:08 +08:00
Wu Qing
3e90e0f8a8 功能: 新增 SAP HANA 完整备份支持与 Backint 协议代理 (#37)
* chore: ignore web/dist directory in git repository

* 功能: 新增 SAP HANA 完整备份支持与 Backint 协议代理

- 修复 service 层校验 bug,使 SAP HANA 类型可正常创建
- 增强 hdbsql Runner:支持完整/增量/差异/日志备份、并行通道、失败重试
- 新增 Backint 协议代理(backupx backint 子命令),HANA 原生接口直连 BackupX 存储后端
- 新增本地 SQLite 目录维护 EBID↔对象键映射
- 前端新增 SAP HANA 扩展字段表单(备份类型/级别/通道数/重试次数/实例编号)
- README 中英文补充 SAP HANA 两种模式的使用说明
2026-04-16 23:43:46 +08:00
Wu Qing
827a5a2181 文档: 更新 README 中英文文档 (#35)
- 存储后端描述更新为 70+ rclone 集成
- API 参考补充新增端点(节点编辑、Rclone 后端列表、版本检查、Agent 心跳)
- 技术栈补充 rclone
- 多节点集群章节补充 IP 检测、节点编辑等新功能描述
- 存储目标添加指南补充 Rclone 类型配置项分层说明
- 任务删除行为说明(清理远端文件、保留记录)
- 版本升级指引从一键更新改为手动 docker compose pull
- 发版示例更新为 v1.4.3
2026-04-05 11:33:56 +08:00
Wu Qing
970eb154e1 优化: 多模块功能修复与体验改进 (#34)
1. 保留策略清理后自动删除空文件夹(新增 StorageDirCleaner 接口)
2. 备份任务删除时清理远端文件但保留备份记录
3. 节点管理修复:本机 IP/版本检测、Heartbeat OS/Arch 修正、新增编辑功能
4. 审计日志规范化:统一格式、丰富详情、节点操作增加审计记录
5. 系统设置移除一键更新操作,仅保留版本检查
6. Rclone 配置项分层展示(必填 + 高级可选折叠)
7. DirectoryPicker 目录选择器样式优化
2026-04-05 11:23:46 +08:00
Wu Qing
d26753c44a 优化: 存储类型下拉框分类中文标注去重 (#33)
优化: 存储类型下拉框分类中文标注去重
2026-04-02 13:43:37 +08:00
Awuqing
4251eb9e15 优化: 存储类型下拉框分类中文标注 + 去重
问题:API 返回的 rclone 后端纯英文技术名难辨别,且和内置类型存在重复
(如 rclone 的 drive 和内置的 google_drive)。

修复:
- 前端静态定义分类+中文标注(常用/云存储/网盘/文件传输/企业存储/自建存储)
- 排除工具类后端(alias/cache/http/archive 等)和重复后端(drive→用google_drive)
- Select 使用 OptGroup 按分组渲染,搜索仍支持英文/中文关键词
- 常用类型(S3/阿里云/SFTP 等)置顶,其余按分类排列
2026-04-02 13:39:43 +08:00
Wu Qing
94d5fb7286 功能: Docker 一键自动更新 (#32)
功能: Docker 一键自动更新
2026-04-01 23:47:43 +08:00
Awuqing
8eb93b3dd9 功能: Docker 一键自动更新
- 新增 POST /api/system/update-apply,执行 docker pull + docker compose up -d
- 前端系统设置页新增「一键更新(Docker)」按钮,点击后自动拉取新镜像并重启容器
- Dockerfile 安装 docker-cli + docker-cli-compose
- docker-compose.yml 挂载 /var/run/docker.sock 以支持容器内操作 Docker
- 自动检测是否为 Docker 环境,非 Docker 环境引导下载二进制
2026-04-01 23:43:12 +08:00
Wu Qing
df5c8aa80d 功能: 系统更新检查 (#31)
功能: 系统更新检查(GitHub Release + Docker)
2026-04-01 23:18:21 +08:00
Awuqing
9a4556f473 功能: 系统更新检查(GitHub Release + Docker)
后端:
- 新增 GET /api/system/update-check,从 GitHub Releases API 获取最新版本
- 自动比较当前版本与最新版本,匹配当前平台的下载链接
- 返回版本号、更新说明、下载链接、Docker 镜像信息

前端(系统设置页重构):
- 新增"检查更新"按钮,点击后展示更新结果
- 有新版本时显示版本号、更新说明、下载按钮、Docker 更新命令
- 新增磁盘状态卡片(总空间/已用/可用/使用率)
- 运行模式用彩色 Tag 区分(生产/开发)
2026-04-01 23:13:32 +08:00
Wu Qing
a772b94ca5 修复: rclone 后端列表不显示 + 调度审计 + 批量删除 (#30)
修复: rclone 后端列表不显示 + 调度审计 + 批量删除
2026-04-01 23:02:40 +08:00
Awuqing
3bd15bf3fd 修复: rclone 后端列表不显示 + 调度审计 + 批量删除
1. 修复前端 rclone 后端 API 路径双重 /api 前缀导致 404,
   存储类型下拉框现在正确显示全部 70+ rclone 后端
2. 调度器自动触发的备份任务计入审计日志(用户名: system)
3. 新增备份记录批量删除 API (POST /api/backup/records/batch-delete)
2026-04-01 22:57:55 +08:00
Wu Qing
5ae7fb2f5d 修复: 上传操作级重试 (#29)
修复: 上传操作级重试,解决远端临时故障导致自动备份失败
2026-04-01 18:40:13 +08:00
Awuqing
37ad6b1db1 修复: 上传操作级重试,解决 Google Drive 等远端临时故障导致自动备份连续失败
问题:rclone 底层重试只覆盖单个 HTTP 请求,但 Google API 的 502/timeout
等临时故障会导致整个上传操作失败,自动触发的备份任务连续失败。

修复:在 provider.Upload 外层增加操作级重试(最多 3 次,指数退避 10s/40s/90s),
每次重试重新打开文件并重建 reader 链。重试过程通过日志流实时反馈。
2026-04-01 18:35:26 +08:00
Wu Qing
d9e0609089 功能: 全部 rclone 后端注册为一级存储类型 (#28)
功能: 全部 rclone 后端注册为一级存储类型
2026-04-01 12:59:29 +08:00
Awuqing
ab9919f15f 功能: 全部 rclone 后端注册为一级存储类型
将全部 70+ rclone 后端(SFTP、Azure Blob、Dropbox、OneDrive、B2、SMB 等)
自动注册为独立 Factory,与 S3、FTP 等内置类型完全平级。

- 新增 GenericBackendFactory + RegisterAllBackends 自动注册全部后端
- 移除 oneof 硬编码白名单,type 字段接受任意已注册存储类型
- 前端类型选择器合并内置类型和全部 rclone 后端为统一可搜索下拉框
- 选择 SFTP 直接存储 type="sftp",非内置类型自动从 API 获取配置字段
2026-04-01 12:52:06 +08:00
Wu Qing
d70b4094af 优化: 重新设计 Cron 编辑器交互体验 (#27)
优化: 重新设计 Cron 编辑器交互体验
2026-04-01 07:50:16 +08:00
Awuqing
eeec7678a1 优化: 重新设计 Cron 编辑器交互体验
核心问题:预设选中后下方 Tab 编辑器仍展开显示混乱的技术细节。

重新设计为三层交互:
1. 预设按钮(一键选择常见场景,选中高亮,无多余 UI)
2. 自定义选择器(每天/每周/每月/间隔四种模式,直观的时间选择器
   和星期按钮,无需理解 cron 语法)
3. 手动输入(高级用户直接编辑 cron 表达式)

同时优化中文描述为自然语言("每天 02:00 执行" 替代 "02 时 00 分 执行")
2026-04-01 07:44:19 +08:00
Wu Qing
cefbdf3a53 优化: Cron 表达式编辑器增加预设和中文描述 (#26)
优化: Cron 表达式编辑器增加预设和中文描述
2026-04-01 00:17:38 +08:00
Wu Qing
4a56ad05fc 修复: 审计日志补充操作详情 + 版本号注入修复 (#25)
修复: 审计日志补充操作详情 + 版本号注入修复
2026-04-01 00:17:34 +08:00
Wu Qing
9ea02566cb 修复: 存储目标创建/连接测试/类型选择三个关键问题 (#24)
修复: 存储目标创建/连接测试/类型选择三个关键问题
2026-04-01 00:17:29 +08:00
Awuqing
a45b1f7bfb 优化: Cron 表达式编辑器增加预设和中文描述
1. 新增 8 个常用预设按钮(每天 02:00、每 6 小时、每周日、每月 1 日等),
   一键设置无需逐个 Tab 操作
2. 新增中文可读描述(如 "02 时 00 分 执行"),实时显示在表达式下方
3. 选中的预设按钮高亮显示
2026-04-01 00:12:32 +08:00
Awuqing
bfc8728785 修复: 审计日志补充操作详情 + 版本号注入修复
1. 审计日志:所有 handler 的 recordAudit 调用补充有意义的 detail,
   包括创建/更新时记录类型、删除时记录 ID、设置变更时记录修改的 key
2. 版本号:Makefile 的 run/build 都通过 ldflags 注入 git 版本号,
   开发模式不再显示 "dev"
2026-04-01 00:10:51 +08:00
Awuqing
3023a089fb 修复: 存储目标创建/连接测试/类型选择三个关键问题
1. 修复 oneof 白名单仅含 4 种类型,阿里云/腾讯/七牛/FTP/Rclone
   类型的存储目标无法创建(binding 验证直接拒绝)
2. 修复本地磁盘 TestConnection 报 "directory not found",
   在 List 前先 Mkdir 确保目录存在
3. 前端存储类型选项明确标注 Rclone 支持 SFTP/Azure/Dropbox 等
2026-04-01 00:06:08 +08:00
Wu Qing
c437a72aad 功能: 集成 rclone 高级传输特性 + 全 70+ 后端支持 (#23)
功能: 集成 rclone 高级传输特性 + 全 70+ 后端支持
2026-03-31 23:46:02 +08:00
Awuqing
93bf8435b0 功能: 集成 rclone 高级传输特性 + 全 70+ 后端支持
1. 失败自动重试:rclone Pacer 指数退避,默认 10 次底层 HTTP 重试
2. 带宽限制:配置 bandwidth_limit + Settings 运行时可调
3. 上传实时进度:progressReader + LogHub SSE 推送字节级进度/速率
4. 存储空间查询:StorageAbout 可选接口,GetUsage 返回远端真实空间
5. 全 rclone 后端:backend/all 引入 70+ 后端,新增 rclone 存储类型,
   API 驱动的可搜索后端选择器 + 动态配置表单
2026-03-31 23:37:59 +08:00
Wu Qing
b2055c08f1 重构: 存储传输层集成 rclone 替代自研实现 (#22)
重构: 存储传输层集成 rclone 替代自研实现
2026-03-31 22:55:41 +08:00
Awuqing
f4d2271cc1 重构: 存储传输层集成 rclone 替代自研实现
将 8 种存储后端(本地磁盘、S3、WebDAV、Google Drive、FTP、阿里云 OSS、
腾讯云 COS、七牛 Kodo)的底层传输从 4 个独立 SDK 自研实现替换为 rclone
fs 接口统一驱动。

- 新建 storage/rclone/ 包(~410 行胶水代码),包含通用 Provider 和 8 种
  配置映射 Factory
- 删除 10 个旧 provider 包(~1000 行),净减少约 1000 行代码
- StorageProvider 接口、前端 UI、数据库模型、备份执行引擎全部零改动
- 获得 rclone 工业级传输能力(分片上传、断点续传、自动重试)
2026-03-31 22:52:16 +08:00
Wu Qing
7c81810019 Merge pull request #21 from Awuqing/feat/community-enhancements
feat: community enhancements, CI/CD pipeline, and backup integrity verification
2026-03-31 13:23:11 +08:00
Awuqing
deb7cf9a5e fix(test): use test TempDir for backup execution tests
The test passed an empty tempDir which defaulted to /tmp/backupx —
a directory that does not exist in CI runners. Use t.TempDir() based
path instead so the test is self-contained.
2026-03-31 13:20:11 +08:00
Awuqing
ad5c25f38e refactor: single-pass hashing during upload via TeeReader
Previous approach read the file twice (once for SHA-256, once for upload),
doubling disk I/O. Under concurrent multi-target uploads this becomes a
bottleneck.

New design — hashingReader wraps io.TeeReader + sha256.Hash:
  file.Read() → TeeReader → sha256.Write() (hash) + provider (upload)
Single read pass yields both byte count and SHA-256 simultaneously.

Each upload goroutine independently opens the file and computes its own
hash. The first successful target writes checksum to the record via
sync.Once. Zero extra disk I/O, zero extra memory copies, fully
concurrent-safe.
2026-03-31 13:08:10 +08:00
Awuqing
7568d8a2a2 refactor: use CountingReader for upload integrity instead of List API
List()-based size check depends on the storage backend returning accurate
file sizes, which is not guaranteed (some WebDAV/Google Drive impls may
return 0 or omit the size field).

New approach: wrap the upload io.Reader with a CountingReader that counts
bytes as they flow through during upload. After upload completes, compare
counter.n against the expected fileSize. This is:
- Zero extra network calls (no List, no Download)
- Zero extra CPU/memory overhead (just an int64 increment per Read)
- Storage-backend agnostic (works with any provider)

If bytes transmitted != expected size → mark failed + auto-delete remote.
2026-03-31 12:40:12 +08:00
Awuqing
e5a4aaadb2 refactor: replace download-based hash verification with lightweight size check
The previous approach downloaded the entire backup file after upload to
compute a remote SHA-256, which doubles bandwidth cost for every backup.

New approach:
- Local SHA-256 is still computed before upload (stored in record for audit)
- After upload, use provider.List() to check remote file size (single API call)
- If remote size is 0 or mismatches local size → mark failed + auto-delete
- If List() fails, log a warning but don't block (file may have uploaded fine)

This catches 0KB corrupted uploads with zero download overhead.
2026-03-31 12:36:29 +08:00
Awuqing
51f1909a73 feat: add SHA-256 checksum verification for backup integrity
Addresses community feedback about 0KB corrupted backup files going
undetected after upload.

Implementation:
- Compute SHA-256 hash of final artifact (after compress/encrypt) before upload
- After each storage target upload, download the file back and verify
  the hash matches the local checksum
- If verification fails: mark that target as failed, auto-delete the
  corrupted remote file, and log detailed mismatch info
- Store checksum in BackupRecord model (new `checksum` column)
- Display truncated SHA-256 with copy button in backup records UI

Verification flow per storage target:
  local SHA-256 → upload → download → remote SHA-256 → compare
  - match: mark success
  - mismatch: mark failed + delete corrupted remote file
2026-03-31 07:46:12 +08:00
Wu Qing
f1c7abfcc0 Merge pull request #20 from Awuqing/feat/community-enhancements
fix: directory picker cannot navigate into subdirectories (#19)
2026-03-31 00:37:50 +08:00
Awuqing
4407fdf731 fix: directory picker cannot navigate into subdirectories (#19)
Root cause: ArcoDesign Tree loadMore callback receives NodeInstance where
the key is at node.props.dataRef.key, not node.props.key. The old code
passed node.props directly which resulted in undefined key, causing
child directory loading to silently fail.

Fix:
- Access node key via node.props.dataRef?.key ?? node.props._key
- Add showLine + blockNode + folder icons for better visual hierarchy
- Add path display with copy button in selection modal
- Add unmountOnExit to reset state on close

Closes #19
2026-03-31 00:32:02 +08:00
Wu Qing
bf799b3bf3 Merge pull request #18 from Awuqing/feat/community-enhancements
Feat/community enhancements
2026-03-31 00:22:08 +08:00
Awuqing
7e5542cae3 docs: update Docker deployment to use published image from Docker Hub
- docker-compose.yml: change from local build to awuqing/backupx:latest
  with clear comments for mounting host volumes
- README: Docker quick start now uses `docker run` / `docker compose`
  directly without cloning the repo first
- Add Docker Hub badge and link to awuqing/backupx
- Keep source build instructions as a separate option
2026-03-31 00:16:31 +08:00
Awuqing
01fd87f029 docs: restructure README with user-friendly guides and deployment instructions
Rewrite both README.md and README_EN.md:
- Reorganize around user journey: install → setup → add storage → create task → monitor
- Add step-by-step "Quick Start" guide (5 steps from zero to first backup)
- Add storage target config reference table
- Consolidate duplicate deployment sections into single "Deployment Guide"
- Remove redundant password reset entries (was listed twice)
- Move Project Structure / Architecture to end (dev-facing, not user-facing)
- Compact API reference table (remove verbose endpoint prefixes)
- Add screenshot grid layout for better visual impact
2026-03-31 00:10:06 +08:00
Wu Qing
00b153b5e1 Merge pull request #17 from Awuqing/feat/community-enhancements
ci: add automated release pipeline with Docker Hub push and China mir…
2026-03-30 23:45:13 +08:00
Awuqing
f201b7633a ci: add automated release pipeline with Docker Hub push and China mirror support
- Fix Go/Node version mismatch in CI (1.21→1.25, 18→20)
- Rewrite release.yml: 3-job pipeline (frontend → binary release + Docker push)
  - Supports both tag push and manual workflow_dispatch trigger
  - Builds linux/amd64 + linux/arm64 binaries → GitHub Release tar.gz
  - Builds multi-arch Docker image → Docker Hub (awuqing/backupx)
- Dockerfile: add ARG USE_CHINA_MIRROR for China network acceleration
  (npm→npmmirror, go→goproxy.cn, apk→aliyun), add ARG VERSION injection
- Makefile: auto version from git tag, add docker/docker-cn targets
- README: add beginner-friendly China build guide and release instructions
2026-03-30 23:32:14 +08:00
Wu Qing
4bf89ae7e5 Merge pull request #16 from Awuqing/feat/community-enhancements
feat: community enhancements — password reset, audit logs, multi-source backup
2026-03-30 23:10:07 +08:00
Awuqing
09698cc767 feat: add community enhancements — password reset, audit logs, multi-source backup
Three community-requested features:

1. CLI password reset: `backupx reset-password --username admin --password xxx`
   Docker users can run via `docker exec`. No full app init needed.

2. Audit logging: async fire-and-forget audit trail for all key operations
   (login, CRUD on tasks/targets/records, settings changes).
   New UI page at /audit with category filter and pagination.

3. Multi-source path backup: file backup tasks now support multiple source
   directories packed into a single tar archive. Backward compatible with
   existing single sourcePath field.
2026-03-30 23:04:37 +08:00
Wu Qing
3e476c401a Merge pull request #15 from Awuqing/feat/docker-support
feat: add Docker deployment support
2026-03-30 12:33:43 +08:00
Awuqing
b01828e3b4 feat: add Docker deployment support
- Multi-stage Dockerfile (Node build + Go build + Alpine runtime)
- docker-compose.yml with named volume for data persistence
- In-container Nginx reverse proxy (static files + API)
- Entrypoint script for graceful process management
- .dockerignore for optimized build context
- Updated README (zh/en) with Docker quick start and deployment docs

Closes #14
2026-03-30 07:56:15 +08:00
Wu Qing
5cc5b067fd Merge pull request #12 from Awuqing/Awuqing-patch-1
Update SAP HANA tool description in README
2026-03-24 23:39:20 +08:00
Wu Qing
7a67241bc6 Update SAP HANA tool description in README 2026-03-24 22:56:10 +08:00
Wu Qing
3008d86027 Merge pull request #11 from Awuqing/feat/saphana-backup-data
feat(saphana): refactor backup from SQL export to BACKUP DATA USING FILE
2026-03-24 18:31:04 +08:00
Awuqing
29dba71b53 feat(saphana): refactor backup from SQL export to BACKUP DATA USING FILE
Replace the hdbsql SELECT-based schema DDL export with SAP HANA's official
BACKUP DATA USING FILE for proper data-level backup.

Changes:
- Run: issue BACKUP DATA [FOR <tenant>] USING FILE via hdbsql, package
  resulting backup files into tar archive as artifact
- Restore: extract tar, locate backup prefix, issue RECOVER DATA
  [FOR <tenant>] USING FILE ... CLEAR LOG
- Add helper functions: buildHdbsqlArgs, packageBackupFiles,
  extractTarArchive, findBackupPrefix
- Add 7 unit tests covering backup/restore/error paths
2026-03-24 18:24:12 +08:00
Wu Qing
ab046be247 Merge pull request #10 from Awuqing/feat/saphana-ftp-support
docs: 更新 README 文档,添加 SAP HANA 和 FTP 支持说明
2026-03-22 11:18:36 +08:00
Wu Qing
93121745b7 Merge pull request #9 from Awuqing/feat/saphana-ftp-support
feat: 新增 SAP HANA 数据库备份支持和 FTP 存储后端
2026-03-21 16:15:43 +08:00
130 changed files with 11248 additions and 3485 deletions

26
.dockerignore Normal file
View File

@@ -0,0 +1,26 @@
# Dependencies
web/node_modules/
# Build artifacts
server/bin/
web/dist/
# Data & logs
data/
*.db
*.log
# IDE & OS
.idea/
.vscode/
*.swp
*.swo
.DS_Store
# Git
.git/
.github/
# Docker
Dockerfile
docker-compose*.yml

View File

@@ -16,7 +16,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.21'
go-version: '1.25'
cache-dependency-path: server/go.sum
- name: Build
@@ -36,7 +36,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '18'
node-version: '20'
cache: 'npm'
cache-dependency-path: web/package-lock.json

View File

@@ -1,63 +1,154 @@
# 自动化发版流水线
#
# 触发方式:
# 1. 推送 taggit tag v1.2.3 && git push --tags
# 2. 手动触发GitHub Actions 页面 → Run workflow → 输入版本号
#
# 产出物:
# - GitHub Releaselinux/amd64 + linux/arm64 预编译 tar.gz
# - Docker Hubawuqing/backupx:latest + awuqing/backupx:v1.2.3(多架构)
#
# 前置配置:
# 在仓库 Settings → Secrets → Actions 添加:
# - DOCKERHUB_USERNAMEDocker Hub 用户名)
# - DOCKERHUB_TOKENDocker Hub Access Token
name: Release
on:
push:
tags:
- 'v*'
workflow_dispatch:
inputs:
version:
description: '版本号(如 v1.2.3'
required: true
type: string
permissions:
contents: write
packages: write
# 统一版本号tag 推送取 ref_name手动触发取 inputs.version
env:
VERSION: ${{ github.event.inputs.version || github.ref_name }}
jobs:
release:
name: Build & Release
# ─── Job 1: 构建前端 ───
build-web:
name: Build Frontend
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: web/package-lock.json
- name: Install & Build
working-directory: web
run: |
npm ci
npm run build
- name: Upload frontend artifact
uses: actions/upload-artifact@v4
with:
name: web-dist
path: web/dist
retention-days: 1
# ─── Job 2: 预编译二进制 → GitHub Release ───
build-release:
name: Build ${{ matrix.goarch }}
needs: build-web
runs-on: ubuntu-latest
strategy:
matrix:
goos: [linux]
goarch: [amd64, arm64]
include:
- goos: linux
goarch: amd64
- goos: linux
goarch: arm64
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: '1.21'
go-version: '1.25'
cache-dependency-path: server/go.sum
- name: Set up Node.js
uses: actions/setup-node@v4
- name: Download frontend artifact
uses: actions/download-artifact@v4
with:
node-version: '18'
cache: 'npm'
cache-dependency-path: web/package-lock.json
name: web-dist
path: web/dist
- name: Build frontend
working-directory: web
run: |
npm ci
npm run build
- name: Build backend
- name: Build binary
working-directory: server
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
CGO_ENABLED: 1
CGO_ENABLED: '0'
run: |
go build -ldflags "-s -w -X main.version=${{ github.ref_name }}" -o ../backupx-${{ matrix.goos }}-${{ matrix.goarch }} ./cmd/backupx
go build \
-trimpath \
-ldflags "-s -w -X main.version=${{ env.VERSION }}" \
-o ../backupx \
./cmd/backupx
- name: Package release
run: |
mkdir -p release
cp backupx-${{ matrix.goos }}-${{ matrix.goarch }} release/
cp -r web/dist release/web
cp server/config.example.yaml release/
cp deploy/install.sh release/ 2>/dev/null || true
cd release && tar czf ../backupx-${{ github.ref_name }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz .
ARCHIVE_NAME="backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}"
mkdir -p "${ARCHIVE_NAME}"
cp backupx "${ARCHIVE_NAME}/"
cp -r web/dist "${ARCHIVE_NAME}/web"
cp server/config.example.yaml "${ARCHIVE_NAME}/"
cp deploy/install.sh "${ARCHIVE_NAME}/" 2>/dev/null || true
tar czf "${ARCHIVE_NAME}.tar.gz" "${ARCHIVE_NAME}"
- name: Upload Release Asset
- name: Upload to GitHub Release
uses: softprops/action-gh-release@v2
with:
files: backupx-${{ github.ref_name }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
tag_name: ${{ env.VERSION }}
files: backupx-${{ env.VERSION }}-${{ matrix.goos }}-${{ matrix.goarch }}.tar.gz
generate_release_notes: true
# ─── Job 3: Docker 多架构 → Docker Hub ───
build-docker:
name: Build & Push Docker
needs: build-web
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build & Push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
build-args: |
VERSION=${{ env.VERSION }}
tags: |
${{ secrets.DOCKERHUB_USERNAME }}/backupx:latest
${{ secrets.DOCKERHUB_USERNAME }}/backupx:${{ env.VERSION }}
cache-from: type=gha
cache-to: type=gha,mode=max

4
.gitignore vendored
View File

@@ -1 +1,3 @@
web/node_modules/
web/node_modules/
web/dist/
server/bin/

93
Dockerfile Normal file
View File

@@ -0,0 +1,93 @@
# BackupX 多阶段构建
#
# 用法:
# 国际构建默认docker build -t backupx .
# 国内加速构建: docker build --build-arg USE_CHINA_MIRROR=true -t backupx .
# 注入版本号: docker build --build-arg VERSION=v1.2.3 -t backupx .
# 全局构建参数
ARG USE_CHINA_MIRROR=false
# ---- Stage 1: Build frontend ----
FROM node:20-alpine AS web-builder
ARG USE_CHINA_MIRROR
# 国内镜像npm 使用淘宝源
RUN if [ "$USE_CHINA_MIRROR" = "true" ]; then \
npm config set registry https://registry.npmmirror.com; \
fi
WORKDIR /build/web
COPY web/package.json web/package-lock.json ./
RUN npm ci
COPY web/ ./
RUN npm run build
# ---- Stage 2: Build backend ----
FROM golang:1.25-alpine AS server-builder
ARG USE_CHINA_MIRROR
ARG VERSION=dev
# 国内镜像Go 模块使用七牛代理
RUN if [ "$USE_CHINA_MIRROR" = "true" ]; then \
go env -w GOPROXY=https://goproxy.cn,direct; \
fi
WORKDIR /build/server
COPY server/go.mod server/go.sum ./
RUN go mod download
COPY server/ ./
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w -X main.version=${VERSION}" -o backupx ./cmd/backupx
# ---- Stage 3: Production image ----
FROM alpine:3.21
ARG USE_CHINA_MIRROR
# 国内镜像Alpine apk 使用阿里云源
RUN if [ "$USE_CHINA_MIRROR" = "true" ]; then \
sed -i 's|dl-cdn.alpinelinux.org|mirrors.aliyun.com|g' /etc/apk/repositories; \
fi
RUN apk add --no-cache \
nginx \
tzdata \
ca-certificates \
docker-cli docker-cli-compose \
# Required by mysql/postgresql backup tasks
mysql-client \
postgresql16-client \
&& rm -rf /var/cache/apk/*
# Create app user
RUN addgroup -S backupx && adduser -S -G backupx -h /app backupx
# Copy backend binary
COPY --from=server-builder /build/server/backupx /app/bin/backupx
# Copy frontend static files
COPY --from=web-builder /build/web/dist /app/web
# Copy nginx config
COPY deploy/docker/nginx.conf /etc/nginx/http.d/default.conf
# Copy entrypoint
COPY deploy/docker/entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
# Create data directories
RUN mkdir -p /app/data /tmp/backupx && \
chown -R backupx:backupx /app /tmp/backupx
# Nginx needs to write to these dirs
RUN mkdir -p /var/lib/nginx/tmp /var/log/nginx && \
chown -R backupx:backupx /var/lib/nginx /var/log/nginx /run/nginx
WORKDIR /app
EXPOSE 8340
VOLUME ["/app/data"]
ENTRYPOINT ["/app/entrypoint.sh"]

View File

@@ -1,22 +1,25 @@
.PHONY: build dev test clean
.PHONY: build dev test clean docker docker-cn
# 一次性构建前后端
# 自动获取版本号(从 git tag 或 commit hash
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
# ── 一键构建 ──
build: build-server build-web
build-server:
cd server && go build -o bin/backupx ./cmd/backupx
cd server && CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o bin/backupx ./cmd/backupx
build-web:
cd web && npm run build
# 开发模式(分别在两个终端运行)
# ── 开发模式(分别在两个终端运行)──
dev-server:
cd server && go run ./cmd/backupx
dev-web:
cd web && npm run dev
# 运行所有测试
# ── 测试 ──
test: test-server test-web
test-server:
@@ -25,6 +28,14 @@ test-server:
test-web:
cd web && npm run test
# 清理构建产物
# ── Docker 构建 ──
docker:
docker build --build-arg VERSION=$(VERSION) -t backupx:$(VERSION) -t backupx:latest .
# 国内加速构建(使用国内镜像源)
docker-cn:
docker build --build-arg VERSION=$(VERSION) --build-arg USE_CHINA_MIRROR=true -t backupx:$(VERSION) -t backupx:latest .
# ── 清理 ──
clean:
rm -rf server/bin web/dist

797
README.md
View File

@@ -2,405 +2,203 @@
<a href="README_EN.md">English</a> | <strong>中文</strong>
</p>
<p align="center">
<h1 align="center">🛡️ BackupX</h1>
<h1 align="center">BackupX</h1>
<p align="center">
<strong>Self-hosted Server Backup Management Platform with Web UI</strong>
</p>
<p align="center">
<a href="#features">Features</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#configuration">Configuration</a> •
<a href="#architecture">Architecture</a> •
<a href="#cluster-mode">Cluster</a> •
<a href="#development">Development</a> •
<a href="#api-reference">API</a>
<strong>自托管服务器备份管理平台</strong><br>
一个二进制,一条命令,管好你所有服务器的备份。
</p>
<p align="center">
<a href="https://github.com/Awuqing/BackupX/stargazers"><img src="https://img.shields.io/github/stars/Awuqing/BackupX?style=flat-square&color=f5c542" alt="Stars"></a>
<a href="https://github.com/Awuqing/BackupX/releases"><img src="https://img.shields.io/github/v/release/Awuqing/BackupX?style=flat-square&color=brightgreen" alt="Release"></a>
<img src="https://img.shields.io/badge/Go-1.21+-00ADD8?style=flat-square&logo=go" alt="Go">
<img src="https://img.shields.io/badge/Go-1.25+-00ADD8?style=flat-square&logo=go" alt="Go">
<img src="https://img.shields.io/badge/React-18-61DAFB?style=flat-square&logo=react" alt="React">
<img src="https://img.shields.io/badge/TypeScript-5-3178C6?style=flat-square&logo=typescript" alt="TypeScript">
<img src="https://img.shields.io/badge/SQLite-embedded-003B57?style=flat-square&logo=sqlite" alt="SQLite">
<a href="LICENSE"><img src="https://img.shields.io/github/license/Awuqing/BackupX?style=flat-square" alt="License"></a>
<a href="https://github.com/Awuqing/BackupX/issues"><img src="https://img.shields.io/github/issues/Awuqing/BackupX?style=flat-square" alt="Issues"></a>
</p>
</p>
---
BackupX 是一个面向 **Linux / macOS 服务器**的自托管备份管理平台。通过企业级 Web 控制台,轻松配置目录备份、数据库备份,并将备份文件安全存储到阿里云 OSS、腾讯云 COS、七牛云 Kodo、Google Drive、S3 兼容存储、WebDAV、FTP/FTPS 或本地磁盘。
<table>
<tr>
<td width="50%"><img src="screenshots/dashboard.png" alt="仪表盘"></td>
<td width="50%"><img src="screenshots/backup-tasks.png" alt="备份任务"></td>
</tr>
<tr>
<td><img src="screenshots/storage-targets.png" alt="存储目标"></td>
<td><img src="screenshots/backup-records.png" alt="备份记录"></td>
</tr>
</table>
支持 **多节点集群管理**,可统一管控分布在不同服务器上的备份任务。
## 功能亮点
> **适用人群**:拥有 Linux 服务器的个人开发者 / 小团队 / 企业运维
| 能力 | 说明 |
|------|------|
| **备份类型** | 文件/目录多源路径、MySQL、PostgreSQL、SQLite、SAP HANA完整/增量/差异/日志备份 + 并行通道 + 失败重试) |
| **SAP HANA Backint 代理** | 内置 SAP HANA Backint 协议代理HANA 原生备份接口可直接把数据路由到 BackupX 支持的任意存储后端 |
| **70+ 存储后端** | 内置阿里云 OSS / 腾讯云 COS / 七牛云 / S3 / Google Drive / WebDAV / FTP + 通过 rclone 集成 SFTP、Azure Blob、Dropbox、OneDrive 等 70+ 后端 |
| **自动调度** | Cron 定时 + 可视化编辑器 + 自动保留策略(按天数/份数清理,自动回收空目录) |
| **多节点** | Master-Agent 集群,统一管理多台服务器的备份,支持远程目录浏览与节点编辑 |
| **安全** | JWT + bcrypt + AES-256-GCM 加密配置 + 可选备份文件加密 + 完整审计日志 |
| **通知** | 邮件 / Webhook / Telegram备份成功或失败时自动推送 |
| **部署** | 单二进制 + 内嵌 SQLiteDocker 一键启动,零外部依赖 |
## Screenshots
---
### 登录页面
![登录页面](screenshots/login.png)
## 快速开始
### 仪表盘
![仪表盘](screenshots/dashboard.png)
### 1. 安装
### 备份任务
![备份任务](screenshots/backup-tasks.png)
### 备份记录
![备份记录](screenshots/backup-records.png)
### 存储目标
![存储目标](screenshots/storage-targets.png)
### 节点管理
![节点管理](screenshots/nodes.png)
### 通知配置
![通知配置](screenshots/notifications.png)
### 系统设置
![系统设置](screenshots/settings.png)
## Features
### 📦 多种备份类型
- **文件/目录** — 支持自定义排除规则(如 `node_modules``*.log`
- **MySQL** — 通过 `mysqldump` 原生工具
- **SQLite** — 安全文件拷贝
- **PostgreSQL** — 通过 `pg_dump` 原生工具
- **SAP HANA** — 通过 `hdbsql` 原生工具(支持多租户数据库)
### ☁️ 多云存储后端
| 厂商 | 类型 | 说明 |
|------|------|------|
| 🇨🇳 **阿里云 OSS** | `aliyun_oss` | 自动组装 Endpoint支持内网传输 |
| 🇨🇳 **腾讯云 COS** | `tencent_cos` | 自动组装 Endpoint |
| 🇨🇳 **七牛云 Kodo** | `qiniu_kodo` | 6 大区域精确映射 |
| 🌍 **S3 Compatible** | `s3` | AWS S3 / MinIO / Cloudflare R2 等 |
| 🌍 **Google Drive** | `google_drive` | 完整 OAuth 2.0 授权流程 |
| 🌍 **WebDAV** | `webdav` | 坚果云 / Nextcloud 等 |
| 🌍 **FTP / FTPS** | `ftp` | 标准 FTP 协议,支持 Explicit TLS 加密 |
| 💾 **本地磁盘** | `local_disk` | 备份到服务器本地目录 |
> 国内云厂商仅需填写 **Region** 和 **AccessKey**,系统自动完成 Endpoint 组装,底层复用 S3 引擎零额外依赖。
### 🖥️ 集群管理 (Master-Agent)
- **节点管理** — 注册远程服务器节点Token 认证
- **本机节点** — 自动创建,单机用户零感知升级
- **目录浏览** — 可视化文件树选择备份源路径,告别手动输入
- **Agent 心跳** — 节点在线状态实时监控
- **任务标签** — 按标签/节点分类管理备份任务
### ⏰ 自动化与调度
- Cron 表达式定时调度
- 可视化 Cron 编辑器
- 自动保留策略(按天数 / 按份数过期清理)
- 最大并发备份数限制
### 🔐 安全
- JWT 认证 + bcrypt 密码存储
- AES-256-GCM 加密存储敏感配置数据库密码、OAuth Token
- 可选备份文件加密
- 登录限流防暴力破解
- 节点 Token 认证(一次性显示,安全传输)
### 📊 监控与通知
- 仪表盘统计(成功率、存储用量、备份趋势图表)
- 邮件 / Webhook / Telegram 通知
- 实时备份执行日志 (SSE)
### 🌐 其他
- 中英文国际化 (i18n)
- 零外部依赖(内嵌 SQLite单二进制部署
- systemd 服务支持
## Quick Start
### 从源码构建
**Docker推荐无需克隆仓库**
```bash
# 克隆项目
git clone https://github.com/Awuqing/BackupX.git
cd BackupX
# 创建 docker-compose.yml 后一键启动
docker compose up -d
# 一键构建前后端
make build
# 启动后端服务(默认监听 :8340
cd server && ./bin/backupx
# 或直接运行
docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/backupx:latest
```
### 访问 Web UI
> Docker Hub 镜像:[`awuqing/backupx`](https://hub.docker.com/r/awuqing/backupx),支持 linux/amd64 和 linux/arm64。
打开浏览器访问 `http://your-server:8340`,首次使用会引导您创建管理员账户。
## Configuration
配置文件路径默认为 `./config.yaml`,也可通过环境变量 `BACKUPX_` 前缀覆盖。
<details>
<summary>docker-compose.yml 参考</summary>
```yaml
# config.yaml
server:
host: "0.0.0.0"
port: 8340
mode: "release" # debug | release
services:
backupx:
image: awuqing/backupx:latest
container_name: backupx
restart: unless-stopped
ports:
- "8340:8340"
volumes:
- backupx-data:/app/data
# 挂载需要备份的宿主机目录(按需添加):
# - /var/www:/mnt/www:ro
# - /etc/nginx:/mnt/nginx-conf:ro
environment:
- TZ=Asia/Shanghai
database:
path: "./data/backupx.db" # SQLite 数据库路径
security:
jwt_secret: "" # 留空则自动生成
jwt_expire: "24h"
encryption_key: "" # AES 加密密钥,留空自动生成
backup:
temp_dir: "/tmp/backupx" # 备份临时文件目录
max_concurrent: 2 # 最大并发备份数
log:
level: "info" # debug | info | warn | error
file: "./data/backupx.log"
max_size: 100 # 日志文件大小上限 (MB)
max_backups: 3 # 保留旧日志文件数
max_age: 30 # 日志保留天数
volumes:
backupx-data:
```
> 💡 `jwt_secret` 和 `encryption_key` 首次启动时自动生成并持久化到数据库,无需手动配置。
</details>
## Architecture
**预编译包(裸机部署):**
```
┌─────────────────────┐
│ Nginx (反向代理) │
│ / → 前端静态文件 │
│ /api → :8340 │
└─────────┬───────────┘
┌──────────────────────────────────────────────────────┐
│ BackupX Master (Go API Server) │
│ :8340 │
│ │
│ ┌──────┐ ┌────────────┐ ┌───────────────────────┐│
│ │ Auth │ │Backup Engine│ │ Storage Registry ││
│ └──────┘ └──────┬─────┘ │ ┌─────────────────┐ ││
│ │ │ │ Aliyun OSS │ ││
│ ┌──────────┐ │ │ │ Tencent COS │ ││
│ │ Cron │◄───┘ │ │ Qiniu Kodo │ ││
│ │Scheduler │ │ │ S3 Compatible │ ││
│ └──────────┘ │ │ Google Drive │ ││
│ │ │ WebDAV │ ││
│ │ │ FTP / FTPS │ ││
│ ┌──────────┐ │ │ Local Disk │ ││
│ │ Notify │ │ └─────────────────┘ ││
│ │ Module │ └───────────────────────┘│
│ └──────────┘ │
│ │
│ ┌──────────────┐ ┌────────────────────┐ │
│ │ Node Manager │ │ SQLite (backupx.db)│ │
│ └──────┬───────┘ └────────────────────┘ │
└─────────┼────────────────────────────────────────────┘
│ Heartbeat / Task Dispatch
┌──────────────────┐ ┌──────────────────┐
│ Agent Node A │ │ Agent Node B │
│ (远程服务器) │ │ (远程服务器) │
└──────────────────┘ └──────────────────┘
```
### 技术栈
| 组件 | 技术 |
|------|------|
| **后端** | Go · Gin · GORM · SQLite · robfig/cron |
| **前端** | React 18 · TypeScript · ArcoDesign · Vite · Zustand · ECharts |
| **存储** | AWS SDK v2 (S3/OSS/COS/Kodo) · Google Drive API v3 · gowebdav · jlaffaye/ftp |
| **安全** | JWT · bcrypt · AES-256-GCM |
| **日志** | zap + lumberjack (自动轮转) |
## Cluster Mode
BackupX 支持 **Master-Agent** 模式,可管理多台服务器的备份任务。
### 工作原理
1. **Master** 为运行 BackupX Web 控制台的主控服务器
2. **Agent** 部署在需要备份的远程服务器上
3. Agent 启动后通过 Token 向 Master 注册并定期发送心跳
4. Master 将备份任务下发至对应 Agent 执行
### 添加节点
从 [Releases](https://github.com/Awuqing/BackupX/releases) 下载对应平台的压缩包:
```bash
# 在 Web 控制台 → 节点管理 → 添加节点
# 系统将生成一个唯一的 64 位十六进制 Token
# 在远程服务器上配置 Agent 启动参数
./backupx-agent --master http://master-server:8340 --token <your-token>
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh # 自动配置 systemd + Nginx
```
### 目录探针 API
Master 提供 `GET /api/nodes/:id/fs/list?path=/` 接口,可远程浏览节点的文件系统目录。前端在创建备份任务的"源路径"输入时可使用树形选择器直接浏览目标机器的目录结构。
## Project Structure
```
BackupX/
├── server/ # Go 后端
│ ├── cmd/backupx/ # 程序入口
│ ├── internal/
│ │ ├── app/ # 应用组装 (DI)
│ │ ├── apperror/ # 统一错误类型
│ │ ├── backup/ # 备份引擎 (file/mysql/sqlite/pgsql/saphana)
│ │ │ └── retention/ # 保留策略
│ │ ├── config/ # 配置加载 (viper)
│ │ ├── database/ # 数据库初始化 + 迁移
│ │ ├── http/ # HTTP 处理器 + 路由 + 中间件
│ │ ├── httpapi/ # HTTP API 辅助工具
│ │ ├── logger/ # 日志初始化 (zap + lumberjack)
│ │ ├── model/ # GORM 数据模型
│ │ ├── notify/ # 通知 (email/webhook/telegram)
│ │ ├── repository/ # 数据访问层
│ │ ├── scheduler/ # Cron 调度器
│ │ ├── security/ # JWT + 限流
│ │ ├── service/ # 业务逻辑层
│ │ └── storage/ # 存储后端 (插件化接口)
│ │ ├── aliyun/ # 阿里云 OSS
│ │ ├── tencent/ # 腾讯云 COS
│ │ ├── qiniu/ # 七牛云 Kodo
│ │ ├── s3/ # S3 Compatible 核心
│ │ ├── s3provider/ # S3 Provider 辅助
│ │ ├── googledrive/ # Google Drive
│ │ ├── webdav/ # WebDAV 核心
│ │ ├── webdavprovider/ # WebDAV Provider 辅助
│ │ ├── localdisk/ # 本地磁盘
│ │ ├── ftp/ # FTP / FTPS
│ │ └── codec/ # 配置编解码
│ └── pkg/ # 工具包 (compress/crypto/response)
├── web/ # React 前端
│ └── src/
│ ├── components/ # 通用组件 (CronEditor/FormDrawer/...)
│ ├── hooks/ # 自定义 Hooks
│ ├── layouts/ # 布局组件 (AppLayout)
│ ├── pages/ # 页面模块
│ │ ├── dashboard/ # 仪表盘
│ │ ├── backup-tasks/ # 备份任务
│ │ ├── backup-records/ # 备份记录
│ │ ├── storage-targets/ # 存储目标
│ │ ├── nodes/ # 节点管理
│ │ ├── notifications/ # 通知配置
│ │ ├── settings/ # 系统设置
│ │ └── login/ # 登录页
│ ├── services/ # API 请求封装
│ ├── stores/ # Zustand 状态管理
│ ├── styles/ # 全局样式
│ ├── types/ # TypeScript 类型定义
│ ├── utils/ # 工具函数
│ ├── locales/ # i18n 语言包 (zh-CN / en-US)
│ └── router/ # 路由配置
├── deploy/ # 部署配置
│ ├── nginx.conf # Nginx 参考配置
│ ├── backupx.service # systemd 服务单元
│ └── install.sh # 一键安装脚本
├── .github/ # GitHub 配置
│ ├── workflows/ci.yml # CI 工作流
│ ├── workflows/release.yml # Release 工作流
│ └── ISSUE_TEMPLATE/ # Issue 模板
└── Makefile # 构建命令
```
## Development
### 前置条件
- **Go** ≥ 1.21
- **Node.js** ≥ 18
- **npm**
### 开发模式
**从源码构建:**
```bash
# 终端 1启动后端 (热重载需配合 air)
make dev-server
# 终端 2启动前端 (Vite HMR)
make dev-web
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build # 构建前后端
make docker-cn # 或用国内镜像构建 Dockergoproxy.cn / npmmirror / 阿里云 apk
```
### 运行测试
### 2. 打开控制台
浏览器访问 `http://your-server:8340`,首次打开会引导创建管理员账户。
### 3. 添加存储目标
进入 **存储目标** 页面,点击 **添加**,选择存储类型并填写凭证:
| 存储类型 | 需要填写 |
|---------|---------|
| 阿里云 OSS | Region + AccessKey ID/Secret + Bucket |
| 腾讯云 COS | Region + SecretId/SecretKey + Bucket格式 `name-appid` |
| 七牛云 Kodo | Region + AccessKey/SecretKey + Bucket |
| S3 兼容 | Endpoint + AccessKey + Bucket |
| Google Drive | Client ID/Secret → 点击授权完成 OAuth |
| WebDAV | 服务器地址 + 用户名/密码 |
| FTP | 主机 + 端口 + 用户名/密码 |
| 本地磁盘 | 目标目录路径 |
| SFTP / Azure / Dropbox / OneDrive 等 | 选择对应类型后填写必填项,高级配置可折叠展开 |
> 国内云厂商只需填 Region 和 AccessKey系统自动组装 Endpoint。Rclone 类型的配置项按必填/可选分层展示,高级选项默认折叠。
添加后点击 **测试连接** 确认配置正确。
### 4. 创建备份任务
进入 **备份任务** 页面,点击 **新建**,三步完成:
1. **基础信息** — 任务名称、备份类型、Cron 表达式(留空则仅手动执行)
2. **源配置** — 文件备份选择源路径(支持多个)、数据库备份填写连接信息
3. **存储与策略** — 选择存储目标(支持多个)、压缩策略、保留天数、是否加密
保存后可以点击 **立即执行** 测试,在 **备份记录** 页面实时查看执行日志。
> 删除备份任务时会自动清理远端存储上的备份文件,但保留备份记录以供审计追溯。
### 5. 配置通知(可选)
进入 **通知配置** 页面支持邮件、Webhook、Telegram 三种方式,可分别配置成功/失败时是否推送。
---
## 部署指南
### Docker 部署
```bash
# 运行全部测试
make test
# 仅后端
make test-server # go test ./...
# 仅前端
make test-web # npm run test
docker compose up -d # 使用上方的 docker-compose.yml
```
### 构建
备份宿主机目录时需要挂载路径(在 docker-compose.yml 的 `volumes` 中添加):
```yaml
volumes:
- backupx-data:/app/data
- /var/www:/mnt/www:ro # 挂载需要备份的目录
- /etc/nginx:/mnt/nginx-conf:ro # 可以挂载多个
```
通过环境变量调整配置:
```yaml
environment:
- TZ=Asia/Shanghai
- BACKUPX_LOG_LEVEL=debug
- BACKUPX_BACKUP_MAX_CONCURRENT=4
```
版本更新:在 **系统设置** 页面点击「检查更新」查看是否有新版本,然后手动执行 `docker compose pull && docker compose up -d` 完成升级。
### 裸机部署
```bash
# 构建前后端
# 使用预编译包
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh
# 或从源码
make build
# 清理构建产物
make clean
```
## Deployment
### 一键安装 (推荐)
```bash
# 先构建
make build
# 以 root 执行安装脚本
sudo ./deploy/install.sh
```
安装脚本自动
1. 创建 `backupx` 系统用户
2. 安装二进制到 `/opt/backupx/bin/`
3. 部署前端到 `/opt/backupx/web/`
4. 生成配置文件 `/etc/backupx/config.yaml`
5. 注册并启动 systemd 服务
6. 配置 Nginx 反向代理(如已安装)
安装脚本自动完成:创建系统用户 → 安装二进制到 `/opt/backupx/` → 配置 systemd → 配置 Nginx 反向代理。
### 手动部署
```bash
# 1. 构建
cd server && go build -o backupx ./cmd/backupx
cd ../web && npm run build
# 2. 部署文件
scp server/backupx your-server:/opt/backupx/bin/
scp -r web/dist/ your-server:/opt/backupx/web/
scp server/config.example.yaml your-server:/etc/backupx/config.yaml
# 3. 启动
ssh your-server '/opt/backupx/bin/backupx -config /etc/backupx/config.yaml'
```
### Nginx 配置示例
### Nginx 反向代理(裸机部署时)
```nginx
server {
listen 80;
server_name backup.example.com;
# 前端静态文件
location / {
root /opt/backupx/web;
try_files $uri $uri/ /index.html;
}
# API 反向代理
location /api/ {
proxy_pass http://127.0.0.1:8340;
proxy_set_header Host $host;
@@ -409,89 +207,270 @@ server {
}
```
## API Reference
### 配置文件
所有 API 均以 `/api` 为前缀,使用 JWT Bearer Token 认证(除特殊标注外)。
配置文件路径 `./config.yaml`,也可通过 `BACKUPX_` 前缀环境变量覆盖:
```yaml
server:
port: 8340
database:
path: "./data/backupx.db"
security:
jwt_secret: "" # 留空自动生成并持久化到数据库
encryption_key: "" # 留空自动生成
backup:
temp_dir: "/tmp/backupx"
max_concurrent: 2
log:
level: "info" # debug | info | warn | error
file: "./data/backupx.log"
```
### 密码重置
忘记管理员密码时通过 CLI 重置:
```bash
# 裸机
./backupx reset-password --username admin --password newpass123
# Docker
docker exec -it backupx /app/bin/backupx reset-password --username admin --password newpass123
```
---
## SAP HANA 支持
BackupX 提供两种 SAP HANA 备份模式,按需选用:
### 模式一hdbsql RunnerWeb 控制台托管)
通过 Web 控制台创建 SAP HANA 备份任务,后端调用 `hdbsql` 执行备份,适合 BackupX 调度的周期性作业。
**源配置步骤支持:**
| 字段 | 可选值 | 说明 |
|------|--------|------|
| 备份类型 | `data` / `log` | 数据备份或日志备份 |
| 备份级别 | `full` / `incremental` / `differential` | 日志备份时自动禁用 |
| 并行通道数 | `1 ~ 32` | `BACKUP DATA USING FILE ('c1','c2',...)` 多路径并发 |
| 失败重试次数 | `1 ~ 10` | 指数退避5s × 尝试次数²) |
| 实例编号 | 可选 | 从端口推断或手动指定 |
### 模式二Backint 协议代理HANA 原生接口)
BackupX 内置 Backint AgentSAP HANA 通过原生 `BACKUP DATA USING BACKINT` 语法调用,数据自动路由到 BackupX 存储目标S3 / OSS / COS / WebDAV / 70+ 后端)。
**1. 准备参数文件** `/opt/backupx/backint_params.ini`
```ini
#STORAGE_TYPE = s3
#STORAGE_CONFIG_JSON = /opt/backupx/storage.json
#PARALLEL_FACTOR = 4
#COMPRESS = true
#KEY_PREFIX = hana-backup
#CATALOG_DB = /opt/backupx/backint_catalog.db
#LOG_FILE = /var/log/backupx/backint.log
```
**2. 准备存储配置** `/opt/backupx/storage.json`(与 BackupX 存储目标配置一致):
```json
{
"endpoint": "https://s3.amazonaws.com",
"region": "us-east-1",
"bucket": "hana-prod",
"accessKeyId": "AKIA...",
"secretAccessKey": "..."
}
```
**3. 创建 hdbbackint 软链接:**
```bash
ln -s /opt/backupx/backupx /usr/sap/<SID>/SYS/global/hdb/opt/hdbbackint
```
**4. 在 HANA `global.ini` 中启用:**
```ini
[backup]
data_backup_using_backint = true
catalog_backup_using_backint = true
log_backup_using_backint = true
data_backup_parameter_file = /opt/backupx/backint_params.ini
log_backup_parameter_file = /opt/backupx/backint_params.ini
```
**5. CLI 手动调用(用于排查):**
```bash
backupx backint -f backup -i input.txt -o output.txt -p backint_params.ini
backupx backint -f restore -i input.txt -o output.txt -p backint_params.ini
backupx backint -f inquire -i input.txt -o output.txt -p backint_params.ini
backupx backint -f delete -i input.txt -o output.txt -p backint_params.ini
```
Backint Agent 使用本地 SQLite 维护 `EBID ↔ 对象键` 目录,所有操作遵循 SAP HANA Backint 协议(`#PIPE` / `#SAVED` / `#RESTORED` / `#BACKUP` / `#NOTFOUND` / `#DELETED` / `#ERROR`)。
---
## 多节点集群
BackupX 支持 Master-Agent 模式管理多台服务器备份任务可以指定在哪个节点执行Agent 在本地完成备份并直接上传到存储后端。
### 架构概览
```
[Web 控制台] ←── JWT ──→ [Master (backupx)]
↑ ↓
│ │ HTTP 长轮询 (token 认证)
│ ↓
[Agent (backupx agent)] ← 运行在远程服务器
[70+ 存储后端]
```
- **通信协议**HTTP 长轮询Agent 主动发起所有连接,无需 Master 反向访问
- **心跳**Agent 每 15s 上报一次Master 每 15s 扫描,超过 45s 未心跳判为离线
- **任务下发**Master 通过数据库命令队列派发 `run_task`Agent 轮询拉取
- **执行**Agent 本地复用 BackupRunnerfile / mysql / postgresql / sqlite / saphana并直接上传到存储
- **安全**:每个节点独立 TokenAgent 不持有 Master 的 JWT 密钥和加密密钥
### 使用步骤
**1. 在 Master 创建节点并获取 Token**
Web 控制台 → **节点管理****添加节点**,填写节点名称并保存。界面会显示一个 64 字节十六进制令牌(仅显示一次,请妥善保存)。
**2. 在远程服务器部署 Agent**
把 BackupX 二进制上传到目标服务器(与 Master 同一个文件),然后用以下任一方式启动:
```bash
# 方式 ACLI 参数
backupx agent --master http://master.example.com:8340 --token <token>
# 方式 B配置文件
cat > /etc/backupx/agent.yaml <<EOF
master: http://master.example.com:8340
token: <token>
heartbeatInterval: 15s
pollInterval: 5s
tempDir: /var/lib/backupx-agent
EOF
backupx agent --config /etc/backupx/agent.yaml
# 方式 C环境变量适合 Docker / systemd
BACKUPX_AGENT_MASTER=http://master.example.com:8340 \
BACKUPX_AGENT_TOKEN=<token> \
backupx agent
```
启动成功后Master 的节点列表会把该节点标记为**在线**。
**3. 创建路由到该节点的备份任务**
**备份任务** 页面新建任务时选择对应节点。任务被触发后:
- 本机节点或未指定节点(`nodeId=0`):由 Master 进程本地执行
- 远程节点Master 写入命令队列 → Agent 轮询拉取 → 本地执行并上传 → 上报记录
### 限制说明
- **不支持加密备份**Agent 不持有 Master 的 AES-256 加密密钥,启用 `encrypt: true` 的任务会路由到 Agent 时失败
- **目录浏览超时**:远程目录浏览通过命令队列做同步 RPC默认 15s 超时,网络慢时可能失败
- **命令超时**Agent 领取但未完成的命令超过 10min 会被标记为超时
### CLI 参考
```bash
backupx agent --help
-master string Master URL
-token string Agent 认证令牌
-config string YAML 配置文件路径(优先级高于环境变量)
-temp-dir string 本地临时目录(默认 /tmp/backupx-agent
-insecure-tls 跳过 TLS 证书校验(仅测试用)
```
---
## 开发指南
**环境要求:** Go >= 1.25 · Node.js >= 20 · npm
```bash
# 开发模式
make dev-server # 终端 1后端默认 :8340
make dev-web # 终端 2前端Vite HMR
# 测试
make test # 运行全部测试
# 构建
make build # 前后端一起构建
make docker # Docker 构建
make docker-cn # 国内 Docker 构建(镜像加速)
```
### 发版
```bash
git tag v1.4.3 && git push --tags
# GitHub Actions 自动:编译双架构二进制 → 发布 GitHub Release → 推送 Docker Hub 镜像
```
也可在 GitHub Actions 页面手动触发 Release workflow。
---
## API 参考
所有接口以 `/api` 为前缀,使用 JWT Bearer Token 认证。
| 模块 | 端点 | 说明 |
|------|------|------|
| **认证** | `POST /api/auth/setup` | 首次初始化管理员 |
| | `POST /api/auth/login` | 登录获取 Token |
| | `POST /api/auth/logout` | 登出 |
| | `GET /api/auth/profile` | 当前用户信息 |
| | `PUT /api/auth/password` | 修改密码 |
| **备份任务** | `GET/POST /api/backup/tasks` | 任务列表 / 创建 |
| | `GET/PUT/DELETE /api/backup/tasks/:id` | 详情 / 更新 / 删除 |
| | `PUT /api/backup/tasks/:id/toggle` | 启用/禁用 |
| | `POST /api/backup/tasks/:id/run` | 手动触发执行 |
| **备份记录** | `GET /api/backup/records` | 记录列表 (支持筛选) |
| | `GET /api/backup/records/:id` | 记录详情 |
| | `GET /api/backup/records/:id/logs/stream` | 实时执行日志 (SSE) |
| | `GET /api/backup/records/:id/download` | 下载备份文件 |
| | `POST /api/backup/records/:id/restore` | 恢复备份 |
| **存储目标** | `GET/POST /api/storage-targets` | 存储列表 / 添加 |
| | `GET/PUT/DELETE /api/storage-targets/:id` | 详情 / 更新 / 删除 |
| | `POST /api/storage-targets/test` | 测试连接 |
| | `POST /api/storage-targets/:id/test` | 测试已保存连接 |
| | `GET /api/storage-targets/:id/usage` | 查询用量 |
| **节点管理** | `GET/POST /api/nodes` | 节点列表 / 添加 |
| | `GET/DELETE /api/nodes/:id` | 详情 / 删除 |
| | `GET /api/nodes/:id/fs/list` | 目录浏览 |
| | `POST /api/agent/heartbeat` | Agent 心跳 ⚡ |
| **通知** | `GET/POST /api/notifications` | 通知列表 / 添加 |
| | `POST /api/notifications/test` | 测试通知 |
| | `POST /api/notifications/:id/test` | 测试已保存通知 |
| **仪表盘** | `GET /api/dashboard/stats` | 概览统计 |
| | `GET /api/dashboard/timeline` | 备份趋势时间线 |
| **系统** | `GET /api/system/info` | 系统信息 (版本/磁盘) |
| | `GET/PUT /api/settings` | 系统设置读写 |
| **认证** | `POST /auth/setup` | 初始化管理员 |
| | `POST /auth/login` | 登录 |
| | `PUT /auth/password` | 修改密码 |
| **备份任务** | `GET\|POST /backup/tasks` | 列表 / 创建 |
| | `GET\|PUT\|DELETE /backup/tasks/:id` | 详情 / 更新 / 删除 |
| | `PUT /backup/tasks/:id/toggle` | 启用/禁用 |
| | `POST /backup/tasks/:id/run` | 手动执行 |
| **备份记录** | `GET /backup/records` | 列表(支持筛选) |
| | `GET /backup/records/:id/logs/stream` | 实时日志 (SSE) |
| | `GET /backup/records/:id/download` | 下载 |
| | `POST /backup/records/:id/restore` | 恢复 |
| **存储目标** | `GET\|POST /storage-targets` | 列表 / 添加 |
| | `POST /storage-targets/test` | 测试连接 |
| | `GET /storage-targets/rclone/backends` | Rclone 后端列表 |
| **节点** | `GET\|POST /nodes` | 列表 / 添加 |
| | `PUT /nodes/:id` | 编辑节点 |
| | `GET /nodes/:id/fs/list` | 目录浏览 |
| | `POST /agent/heartbeat` | Agent 心跳Token 认证) |
| **通知** | `GET\|POST /notifications` | 列表 / 添加 |
| **仪表盘** | `GET /dashboard/stats` | 概览统计 |
| **审计日志** | `GET /audit-logs` | 操作审计 |
| **系统** | `GET /system/info` | 系统信息 |
| | `GET /system/update-check` | 检查版本更新 |
> ⚡ `POST /api/agent/heartbeat` 为公开端点,使用 Node Token 认证而非 JWT。
---
## 云存储配置指南
## 技术栈
### 阿里云 OSS
1. 登录[阿里云控制台](https://oss.console.aliyun.com/),创建 Bucket
2. 前往 RAM 控制台创建 AccessKey
3. 在 BackupX 添加存储目标时选择"阿里云 OSS"
4. 填写 Region`cn-hangzhou`)和 AccessKey系统自动组装 Endpoint
### 腾讯云 COS
1. 登录[腾讯云控制台](https://console.cloud.tencent.com/cos),创建存储桶
2. 前往 API 密钥管理创建 SecretId/SecretKey
3. Bucket 名称格式为 `BucketName-APPID`(如 `backup-1250000000`
### 七牛云 Kodo
1. 登录[七牛云控制台](https://portal.qiniu.com/),创建存储空间
2. 支持区域:`z0`(华东) / `cn-east-2`(华东-浙江2) / `z1`(华北) / `z2`(华南) / `na0`(北美) / `as0`(东南亚)
### Google Drive
1. 前往 [Google Cloud Console](https://console.cloud.google.com/) 创建项目
2. 启用 **Google Drive API**
3. 创建 **OAuth 2.0 客户端 ID**Web 应用类型)
4. 添加重定向 URI`http://your-server/api/storage-targets/google-drive/callback`
5. 在 BackupX 存储管理页面填入 Client ID / Secret点击授权
| 组件 | 技术 |
|------|------|
| **后端** | Go · Gin · GORM · SQLite · robfig/cron · rclone |
| **前端** | React 18 · TypeScript · ArcoDesign · Vite · Zustand · ECharts |
| **存储** | rclone70+ 后端)· AWS SDK v2 · Google Drive API v3 |
| **安全** | JWT · bcrypt · AES-256-GCM |
## Contributing
欢迎提交 Issue 和 Pull Request
1. Fork 本项目
2. 创建功能分支 (`git checkout -b feature/amazing-feature`)
3. 提交更改 (`git commit -m 'Add amazing feature'`)
4. 推送到分支 (`git push origin feature/amazing-feature`)
5. 创建 Pull Request
## License
本项目采用 [Apache License 2.0](LICENSE) 开源协议。
---
<p align="center">
Made with ❤️ for self-hosters
</p>
[Apache License 2.0](LICENSE)

View File

@@ -2,406 +2,203 @@
<strong>English</strong> | <a href="README.md">中文</a>
</p>
<p align="center">
<h1 align="center">🛡️ BackupX</h1>
<h1 align="center">BackupX</h1>
<p align="center">
<strong>Self-hosted Server Backup Management Platform with Web UI</strong>
</p>
<p align="center">
<a href="#features">Features</a> •
<a href="#quick-start">Quick Start</a> •
<a href="#configuration">Configuration</a> •
<a href="#architecture">Architecture</a> •
<a href="#cluster-mode">Cluster</a> •
<a href="#development">Development</a> •
<a href="#api-reference">API</a>
<strong>Self-hosted Server Backup Management Platform</strong><br>
One binary, one command — manage all your server backups.
</p>
<p align="center">
<a href="https://github.com/Awuqing/BackupX/stargazers"><img src="https://img.shields.io/github/stars/Awuqing/BackupX?style=flat-square&color=f5c542" alt="Stars"></a>
<a href="https://github.com/Awuqing/BackupX/releases"><img src="https://img.shields.io/github/v/release/Awuqing/BackupX?style=flat-square&color=brightgreen" alt="Release"></a>
<img src="https://img.shields.io/badge/Go-1.21+-00ADD8?style=flat-square&logo=go" alt="Go">
<img src="https://img.shields.io/badge/Go-1.25+-00ADD8?style=flat-square&logo=go" alt="Go">
<img src="https://img.shields.io/badge/React-18-61DAFB?style=flat-square&logo=react" alt="React">
<img src="https://img.shields.io/badge/TypeScript-5-3178C6?style=flat-square&logo=typescript" alt="TypeScript">
<img src="https://img.shields.io/badge/SQLite-embedded-003B57?style=flat-square&logo=sqlite" alt="SQLite">
<a href="LICENSE"><img src="https://img.shields.io/github/license/Awuqing/BackupX?style=flat-square" alt="License"></a>
<a href="https://github.com/Awuqing/BackupX/issues"><img src="https://img.shields.io/github/issues/Awuqing/BackupX?style=flat-square" alt="Issues"></a>
</p>
</p>
---
BackupX is a self-hosted backup management platform for **Linux / macOS servers**. Through an enterprise-grade Web console, you can easily configure directory backups, database backups, and securely store backup files to Alibaba Cloud OSS, Tencent Cloud COS, Qiniu Cloud Kodo, Google Drive, S3-compatible storage, WebDAV, FTP/FTPS, or local disk.
<table>
<tr>
<td width="50%"><img src="screenshots/dashboard.png" alt="Dashboard"></td>
<td width="50%"><img src="screenshots/backup-tasks.png" alt="Backup Tasks"></td>
</tr>
<tr>
<td><img src="screenshots/storage-targets.png" alt="Storage Targets"></td>
<td><img src="screenshots/backup-records.png" alt="Backup Records"></td>
</tr>
</table>
Supports **multi-node cluster management** for unified control of backup tasks across different servers.
## Highlights
> **For**: Individual developers / small teams / DevOps with Linux servers
| Capability | Details |
|-----------|---------|
| **Backup Types** | Files/Directories (multi-source), MySQL, PostgreSQL, SQLite, SAP HANA (full / incremental / differential / log backups + parallel channels + retry) |
| **SAP HANA Backint Agent** | Built-in SAP HANA Backint protocol agent — HANA's native backup interface can route data directly to any storage backend supported by BackupX |
| **70+ Storage Backends** | Built-in Alibaba OSS / Tencent COS / Qiniu / S3 / Google Drive / WebDAV / FTP + 70+ backends via rclone (SFTP, Azure Blob, Dropbox, OneDrive, etc.) |
| **Scheduling** | Cron-based + visual editor + auto-retention policy (by days/count, auto empty directory cleanup) |
| **Multi-Node** | Master-Agent cluster for managing backups across multiple servers with remote directory browsing and node editing |
| **Security** | JWT + bcrypt + AES-256-GCM encrypted config + optional backup encryption + comprehensive audit logs |
| **Notifications** | Email / Webhook / Telegram — push on success or failure |
| **Deployment** | Single binary + embedded SQLite, Docker one-click, zero external dependencies |
## Screenshots
### Login
![Login](screenshots/login.png)
### Dashboard
![Dashboard](screenshots/dashboard.png)
### Backup Tasks
![Backup Tasks](screenshots/backup-tasks.png)
### Backup Records
![Backup Records](screenshots/backup-records.png)
### Storage Targets
![Storage Targets](screenshots/storage-targets.png)
### Node Management
![Node Management](screenshots/nodes.png)
### Notification Settings
![Notification Settings](screenshots/notifications.png)
### System Settings
![System Settings](screenshots/settings.png)
## Features
### 📦 Multiple Backup Types
- **Files / Directories** — Custom exclude rules (e.g. `node_modules`, `*.log`)
- **MySQL** — Via native `mysqldump` tool
- **SQLite** — Safe file copy
- **PostgreSQL** — Via native `pg_dump` tool
- **SAP HANA** — Via native `hdbsql` tool (multi-tenant database support)
### ☁️ Multi-Cloud Storage Backends
| Provider | Type | Description |
|----------|------|-------------|
| 🇨🇳 **Alibaba Cloud OSS** | `aliyun_oss` | Auto endpoint assembly, internal network support |
| 🇨🇳 **Tencent Cloud COS** | `tencent_cos` | Auto endpoint assembly |
| 🇨🇳 **Qiniu Cloud Kodo** | `qiniu_kodo` | 6 region precise mapping |
| 🌍 **S3 Compatible** | `s3` | AWS S3 / MinIO / Cloudflare R2, etc. |
| 🌍 **Google Drive** | `google_drive` | Full OAuth 2.0 flow |
| 🌍 **WebDAV** | `webdav` | Nextcloud / Nutstore, etc. |
| 🌍 **FTP / FTPS** | `ftp` | Standard FTP protocol with Explicit TLS support |
| 💾 **Local Disk** | `local_disk` | Backup to local server directory |
> Chinese cloud providers only require **Region** and **AccessKey** — the system auto-assembles the endpoint. Powered by the S3 engine under the hood with zero extra dependencies.
### 🖥️ Cluster Management (Master-Agent)
- **Node Management** — Register remote server nodes with Token authentication
- **Local Node** — Auto-created, zero-friction upgrade for single-machine users
- **Directory Browser** — Visual file tree selector for backup source paths
- **Agent Heartbeat** — Real-time node online status monitoring
- **Task Tags** — Categorize and manage backup tasks by tags/nodes
### ⏰ Automation & Scheduling
- Cron expression scheduling
- Visual Cron editor
- Auto-retention policy (by days / by count)
- Max concurrent backup limit
### 🔐 Security
- JWT authentication + bcrypt password hashing
- AES-256-GCM encrypted sensitive config storage (DB passwords, OAuth tokens)
- Optional backup file encryption
- Login rate limiting (brute force protection)
- Node Token authentication (one-time display, secure transport)
### 📊 Monitoring & Notifications
- Dashboard stats (success rate, storage usage, backup trend charts)
- Email / Webhook / Telegram notifications
- Real-time backup execution logs (SSE)
### 🌐 Other
- Chinese & English i18n
- Zero external dependencies (embedded SQLite, single binary deployment)
- systemd service support
---
## Quick Start
### Build from Source
### 1. Install
**Docker (recommended, no clone needed):**
```bash
# Clone the project
git clone https://github.com/Awuqing/BackupX.git
cd BackupX
# Create a docker-compose.yml then start
docker compose up -d
# Build frontend and backend
make build
# Start the backend service (default port :8340)
cd server && ./bin/backupx
# Or run directly
docker run -d --name backupx -p 8340:8340 -v backupx-data:/app/data awuqing/backupx:latest
```
### Access Web UI
> Docker Hub: [`awuqing/backupx`](https://hub.docker.com/r/awuqing/backupx) — supports linux/amd64 and linux/arm64.
Open `http://your-server:8340` in your browser. First-time use will guide you through creating an admin account.
## Configuration
The config file defaults to `./config.yaml`. Settings can also be overridden via `BACKUPX_` prefixed environment variables.
<details>
<summary>docker-compose.yml reference</summary>
```yaml
# config.yaml
server:
host: "0.0.0.0"
port: 8340
mode: "release" # debug | release
services:
backupx:
image: awuqing/backupx:latest
container_name: backupx
restart: unless-stopped
ports:
- "8340:8340"
volumes:
- backupx-data:/app/data
# Mount host directories to back up (add as needed):
# - /var/www:/mnt/www:ro
# - /etc/nginx:/mnt/nginx-conf:ro
environment:
- TZ=Asia/Shanghai
database:
path: "./data/backupx.db" # SQLite database path
security:
jwt_secret: "" # Leave empty to auto-generate
jwt_expire: "24h"
encryption_key: "" # AES encryption key, auto-generated if empty
backup:
temp_dir: "/tmp/backupx" # Backup temp directory
max_concurrent: 2 # Max concurrent backups
log:
level: "info" # debug | info | warn | error
file: "./data/backupx.log"
max_size: 100 # Max log file size (MB)
max_backups: 3 # Number of old log files to retain
max_age: 30 # Log retention days
volumes:
backupx-data:
```
> 💡 `jwt_secret` and `encryption_key` are auto-generated on first startup and persisted to the database.
</details>
## Architecture
**Pre-built binaries (bare metal):**
```
┌─────────────────────┐
│ Nginx (Reverse │
│ Proxy) │
│ / → Static Files │
│ /api → :8340 │
└─────────┬───────────┘
┌──────────────────────────────────────────────────────┐
│ BackupX Master (Go API Server) │
│ :8340 │
│ │
│ ┌──────┐ ┌────────────┐ ┌───────────────────────┐│
│ │ Auth │ │Backup Engine│ │ Storage Registry ││
│ └──────┘ └──────┬─────┘ │ ┌─────────────────┐ ││
│ │ │ │ Alibaba Cloud │ ││
│ ┌──────────┐ │ │ │ Tencent Cloud │ ││
│ │ Cron │◄───┘ │ │ Qiniu Cloud │ ││
│ │Scheduler │ │ │ S3 Compatible │ ││
│ └──────────┘ │ │ Google Drive │ ││
│ │ │ WebDAV │ ││
│ │ │ FTP / FTPS │ ││
│ ┌──────────┐ │ │ Local Disk │ ││
│ │ Notify │ │ └─────────────────┘ ││
│ │ Module │ └───────────────────────┘│
│ └──────────┘ │
│ │
│ ┌──────────────┐ ┌────────────────────┐ │
│ │ Node Manager │ │ SQLite (backupx.db)│ │
│ └──────┬───────┘ └────────────────────┘ │
└─────────┼────────────────────────────────────────────┘
│ Heartbeat / Task Dispatch
┌──────────────────┐ ┌──────────────────┐
│ Agent Node A │ │ Agent Node B │
│ (Remote Server)│ │ (Remote Server)│
└──────────────────┘ └──────────────────┘
```
### Tech Stack
| Component | Technology |
|-----------|-----------|
| **Backend** | Go · Gin · GORM · SQLite · robfig/cron |
| **Frontend** | React 18 · TypeScript · ArcoDesign · Vite · Zustand · ECharts |
| **Storage** | AWS SDK v2 (S3/OSS/COS/Kodo) · Google Drive API v3 · gowebdav · jlaffaye/ftp |
| **Security** | JWT · bcrypt · AES-256-GCM |
| **Logging** | zap + lumberjack (auto-rotation) |
## Cluster Mode
BackupX supports **Master-Agent** mode for managing backup tasks across multiple servers.
### How It Works
1. **Master** is the server running the BackupX Web console
2. **Agent** is deployed on remote servers that need to be backed up
3. Agents register with the Master using a Token and send periodic heartbeats
4. Master dispatches backup tasks to the corresponding Agent for execution
### Adding Nodes
Download from [Releases](https://github.com/Awuqing/BackupX/releases):
```bash
# In Web Console → Node Management → Add Node
# The system generates a unique 64-character hex Token
# Configure the Agent on the remote server
./backupx-agent --master http://master-server:8340 --token <your-token>
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh # Auto-configures systemd + Nginx
```
### Directory Probe API
Master provides `GET /api/nodes/:id/fs/list?path=/` to remotely browse a node's file system. The frontend uses a tree selector to browse the target machine's directory structure when creating backup tasks.
## Project Structure
```
BackupX/
├── server/ # Go backend
│ ├── cmd/backupx/ # Entry point
│ ├── internal/
│ │ ├── app/ # App assembly (DI)
│ │ ├── apperror/ # Unified error types
│ │ ├── backup/ # Backup engine (file/mysql/sqlite/pgsql/saphana)
│ │ │ └── retention/ # Retention policy
│ │ ├── config/ # Config loading (viper)
│ │ ├── database/ # Database init + migrations
│ │ ├── http/ # HTTP handlers + routes + middleware
│ │ ├── httpapi/ # HTTP API helpers
│ │ ├── logger/ # Logger init (zap + lumberjack)
│ │ ├── model/ # GORM data models
│ │ ├── notify/ # Notifications (email/webhook/telegram)
│ │ ├── repository/ # Data access layer
│ │ ├── scheduler/ # Cron scheduler
│ │ ├── security/ # JWT + rate limiting
│ │ ├── service/ # Business logic
│ │ └── storage/ # Storage backends (plugin interface)
│ │ ├── aliyun/ # Alibaba Cloud OSS
│ │ ├── tencent/ # Tencent Cloud COS
│ │ ├── qiniu/ # Qiniu Cloud Kodo
│ │ ├── s3/ # S3 Compatible core
│ │ ├── s3provider/ # S3 Provider helper
│ │ ├── googledrive/ # Google Drive
│ │ ├── webdav/ # WebDAV core
│ │ ├── webdavprovider/ # WebDAV Provider helper
│ │ ├── localdisk/ # Local disk
│ │ ├── ftp/ # FTP / FTPS
│ │ └── codec/ # Config codec
│ └── pkg/ # Utilities (compress/crypto/response)
├── web/ # React frontend
│ └── src/
│ ├── components/ # Shared components (CronEditor/FormDrawer/...)
│ ├── hooks/ # Custom Hooks
│ ├── layouts/ # Layout components (AppLayout)
│ ├── pages/ # Page modules
│ │ ├── dashboard/ # Dashboard
│ │ ├── backup-tasks/ # Backup tasks
│ │ ├── backup-records/ # Backup records
│ │ ├── storage-targets/ # Storage targets
│ │ ├── nodes/ # Node management
│ │ ├── notifications/ # Notification settings
│ │ ├── settings/ # System settings
│ │ └── login/ # Login page
│ ├── services/ # API request wrappers
│ ├── stores/ # Zustand state management
│ ├── styles/ # Global styles
│ ├── types/ # TypeScript type definitions
│ ├── utils/ # Utility functions
│ ├── locales/ # i18n language packs (zh-CN / en-US)
│ └── router/ # Route configuration
├── deploy/ # Deployment configs
│ ├── nginx.conf # Nginx reference config
│ ├── backupx.service # systemd service unit
│ └── install.sh # One-click install script
├── .github/ # GitHub configuration
│ ├── workflows/ci.yml # CI workflow
│ ├── workflows/release.yml # Release workflow
│ └── ISSUE_TEMPLATE/ # Issue templates
└── Makefile # Build commands
```
## Development
### Prerequisites
- **Go** ≥ 1.21
- **Node.js** ≥ 18
- **npm**
### Dev Mode
**Build from source:**
```bash
# Terminal 1: Start backend (use air for hot-reload)
make dev-server
# Terminal 2: Start frontend (Vite HMR)
make dev-web
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
make build # Build frontend + backend
make docker-cn # Or Docker build with China mirrors (goproxy.cn / npmmirror / Aliyun apk)
```
### Run Tests
### 2. Open the Console
Visit `http://your-server:8340` in your browser. First-time access guides you through admin account creation.
### 3. Add a Storage Target
Go to **Storage Targets****Add**, choose a storage type and enter credentials:
| Storage Type | Required Fields |
|-------------|----------------|
| Alibaba Cloud OSS | Region + AccessKey ID/Secret + Bucket |
| Tencent Cloud COS | Region + SecretId/SecretKey + Bucket (`name-appid`) |
| Qiniu Cloud Kodo | Region + AccessKey/SecretKey + Bucket |
| S3 Compatible | Endpoint + AccessKey + Bucket |
| Google Drive | Client ID/Secret → click Authorize for OAuth |
| WebDAV | Server URL + Username/Password |
| FTP | Host + Port + Username/Password |
| Local Disk | Target directory path |
| SFTP / Azure / Dropbox / OneDrive etc. | Select the type, fill in required fields; advanced options are collapsible |
> For Chinese cloud providers, just enter Region and AccessKey — the system auto-assembles the Endpoint. Rclone-type configs separate required fields from optional advanced options (collapsed by default).
Click **Test Connection** to verify.
### 4. Create a Backup Task
Go to **Backup Tasks****Create**, complete 3 steps:
1. **Basic Info** — Task name, backup type, Cron expression (leave empty for manual-only)
2. **Source Config** — File backup: select source paths (supports multiple); Database: enter connection info
3. **Storage & Policy** — Select storage target(s) (supports multiple), compression, retention days, encryption toggle
Save, then click **Run Now** to test. View real-time logs in **Backup Records**.
> Deleting a backup task automatically cleans up remote storage files while preserving backup records for audit purposes.
### 5. Set Up Notifications (Optional)
Go to **Notifications** to configure Email, Webhook, or Telegram alerts for backup success/failure.
---
## Deployment Guide
### Docker
```bash
# Run all tests
make test
# Backend only
make test-server # go test ./...
# Frontend only
make test-web # npm run test
docker compose up -d # Using the docker-compose.yml above
```
### Build
Mount host directories for file backup (add to `volumes` in docker-compose.yml):
```yaml
volumes:
- backupx-data:/app/data
- /var/www:/mnt/www:ro
- /etc/nginx:/mnt/nginx-conf:ro
```
Override config via environment variables:
```yaml
environment:
- TZ=Asia/Shanghai
- BACKUPX_LOG_LEVEL=debug
- BACKUPX_BACKUP_MAX_CONCURRENT=4
```
To upgrade: go to **System Settings**, click "Check for Updates" to see if a new version is available, then run `docker compose pull && docker compose up -d`.
### Bare Metal
```bash
# Build frontend and backend
# From pre-built package
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
sudo ./install.sh
# Or from source
make build
# Clean build artifacts
make clean
```
## Deployment
### One-Click Install (Recommended)
```bash
# Build first
make build
# Run install script as root
sudo ./deploy/install.sh
```
The install script will automatically:
1. Create a `backupx` system user
2. Install the binary to `/opt/backupx/bin/`
3. Deploy the frontend to `/opt/backupx/web/`
4. Generate config at `/etc/backupx/config.yaml`
5. Register and start the systemd service
6. Configure Nginx reverse proxy (if installed)
The install script creates a system user, installs to `/opt/backupx/`, configures systemd, and sets up Nginx reverse proxy.
### Manual Deployment
```bash
# 1. Build
cd server && go build -o backupx ./cmd/backupx
cd ../web && npm run build
# 2. Deploy files
scp server/backupx your-server:/opt/backupx/bin/
scp -r web/dist/ your-server:/opt/backupx/web/
scp server/config.example.yaml your-server:/etc/backupx/config.yaml
# 3. Start
ssh your-server '/opt/backupx/bin/backupx -config /etc/backupx/config.yaml'
```
### Nginx Config Example
### Nginx Reverse Proxy (bare metal)
```nginx
server {
listen 80;
server_name backup.example.com;
# Frontend static files
location / {
root /opt/backupx/web;
try_files $uri $uri/ /index.html;
}
# API reverse proxy
location /api/ {
proxy_pass http://127.0.0.1:8340;
proxy_set_header Host $host;
@@ -410,89 +207,268 @@ server {
}
```
### Configuration
Config file: `./config.yaml` (or override with `BACKUPX_` prefixed env vars):
```yaml
server:
port: 8340
database:
path: "./data/backupx.db"
security:
jwt_secret: "" # Auto-generated and persisted to DB
encryption_key: "" # Auto-generated
backup:
temp_dir: "/tmp/backupx"
max_concurrent: 2
log:
level: "info" # debug | info | warn | error
file: "./data/backupx.log"
```
### Password Reset
```bash
# Bare metal
./backupx reset-password --username admin --password newpass123
# Docker
docker exec -it backupx /app/bin/backupx reset-password --username admin --password newpass123
```
---
## SAP HANA Support
BackupX offers two SAP HANA backup modes — pick whichever fits:
### Mode 1: hdbsql Runner (Web-console managed)
Create a SAP HANA backup task in the Web console. The backend runs `hdbsql` to perform backups, suitable for BackupX-scheduled recurring jobs.
**Source configuration supports:**
| Field | Options | Description |
|-------|---------|-------------|
| Backup type | `data` / `log` | Data or log backup |
| Backup level | `full` / `incremental` / `differential` | Auto-disabled for log backups |
| Parallel channels | `1 ~ 32` | `BACKUP DATA USING FILE ('c1','c2',...)` parallel paths |
| Retry count | `1 ~ 10` | Exponential backoff (5s × attempt²) |
| Instance number | Optional | Inferred from port or manually specified |
### Mode 2: Backint Protocol Agent (HANA native)
BackupX ships a built-in Backint Agent. SAP HANA calls it via native `BACKUP DATA USING BACKINT` syntax, and data is routed automatically to BackupX storage targets (S3 / OSS / COS / WebDAV / 70+ backends).
**1. Prepare parameter file** `/opt/backupx/backint_params.ini`:
```ini
#STORAGE_TYPE = s3
#STORAGE_CONFIG_JSON = /opt/backupx/storage.json
#PARALLEL_FACTOR = 4
#COMPRESS = true
#KEY_PREFIX = hana-backup
#CATALOG_DB = /opt/backupx/backint_catalog.db
#LOG_FILE = /var/log/backupx/backint.log
```
**2. Prepare storage config** `/opt/backupx/storage.json` (same schema as BackupX storage targets):
```json
{
"endpoint": "https://s3.amazonaws.com",
"region": "us-east-1",
"bucket": "hana-prod",
"accessKeyId": "AKIA...",
"secretAccessKey": "..."
}
```
**3. Create the hdbbackint symlink:**
```bash
ln -s /opt/backupx/backupx /usr/sap/<SID>/SYS/global/hdb/opt/hdbbackint
```
**4. Enable in HANA `global.ini`:**
```ini
[backup]
data_backup_using_backint = true
catalog_backup_using_backint = true
log_backup_using_backint = true
data_backup_parameter_file = /opt/backupx/backint_params.ini
log_backup_parameter_file = /opt/backupx/backint_params.ini
```
**5. Manual CLI invocation (for troubleshooting):**
```bash
backupx backint -f backup -i input.txt -o output.txt -p backint_params.ini
backupx backint -f restore -i input.txt -o output.txt -p backint_params.ini
backupx backint -f inquire -i input.txt -o output.txt -p backint_params.ini
backupx backint -f delete -i input.txt -o output.txt -p backint_params.ini
```
The Backint Agent maintains an `EBID ↔ object-key` catalog in a local SQLite DB. All operations follow the SAP HANA Backint protocol (`#PIPE` / `#SAVED` / `#RESTORED` / `#BACKUP` / `#NOTFOUND` / `#DELETED` / `#ERROR`).
---
## Multi-Node Cluster
BackupX supports Master-Agent mode for managing multiple servers. Backup tasks can be routed to specific nodes — the Agent runs the backup locally and uploads straight to storage backends.
### Architecture
```
[Web Console] ←── JWT ──→ [Master (backupx)]
↑ ↓
│ │ HTTP long-poll (token auth)
│ ↓
[Agent (backupx agent)] ← runs on remote host
[70+ Storage Backends]
```
- **Protocol**: HTTP long-polling; the Agent initiates all connections — Master never needs reverse access
- **Heartbeat**: Agent reports every 15s; Master marks nodes offline after 45s of silence
- **Dispatch**: Master persists `run_task` commands to a queue; Agent polls and claims them
- **Execution**: Agent reuses the same BackupRunner (file / mysql / postgresql / sqlite / saphana) and uploads directly to storage
- **Security**: Each node gets its own token; the Agent never holds the Master's JWT secret or encryption key
### Walkthrough
**1. Create a node on Master and copy the token**
Web Console → **Node Management****Add Node**. The dialog shows a 64-byte hex token once — keep it safe.
**2. Deploy the Agent on a remote host**
Upload the BackupX binary (same file as Master) to the target host, then start the Agent:
```bash
# Option A: CLI flags
backupx agent --master http://master.example.com:8340 --token <token>
# Option B: config file
cat > /etc/backupx/agent.yaml <<EOF
master: http://master.example.com:8340
token: <token>
heartbeatInterval: 15s
pollInterval: 5s
tempDir: /var/lib/backupx-agent
EOF
backupx agent --config /etc/backupx/agent.yaml
# Option C: environment variables (Docker / systemd-friendly)
BACKUPX_AGENT_MASTER=http://master.example.com:8340 \
BACKUPX_AGENT_TOKEN=<token> \
backupx agent
```
Once connected, the node appears as **online** in the list.
**3. Create a task routed to that node**
In the **Backup Tasks** page, pick the target node when creating the task. When triggered:
- Local / unassigned (`nodeId=0`) tasks run in-process on Master
- Remote-node tasks are enqueued → Agent claims → Agent runs locally → uploads → reports back
### Limitations
- **No encrypted backups via Agent**: the Agent doesn't hold Master's AES-256 key. Tasks with `encrypt: true` will fail if routed to an Agent
- **Directory browse timeout**: remote dir listing is a synchronous RPC through the queue; default 15s timeout
- **Command timeout**: claimed-but-unfinished commands are marked timed out after 10 minutes
### CLI Reference
```bash
backupx agent --help
-master string Master URL
-token string Agent auth token
-config string YAML config path (takes precedence over env)
-temp-dir string Local temp directory (default /tmp/backupx-agent)
-insecure-tls Skip TLS verification (testing only)
```
---
## Development
**Requirements:** Go >= 1.25 · Node.js >= 20 · npm
```bash
# Dev mode
make dev-server # Terminal 1: backend (:8340)
make dev-web # Terminal 2: frontend (Vite HMR)
# Test
make test # Run all tests
# Build
make build # Build frontend + backend
make docker # Docker build
make docker-cn # Docker build with China mirrors
```
### Release
```bash
git tag v1.4.3 && git push --tags
# GitHub Actions: compile dual-arch binaries → publish GitHub Release → push Docker Hub image
```
Or manually trigger the Release workflow from GitHub Actions page.
---
## API Reference
All APIs are prefixed with `/api` and use JWT Bearer Token authentication (unless noted otherwise).
All endpoints prefixed with `/api`, authenticated via JWT Bearer Token.
| Module | Endpoint | Description |
|--------|----------|-------------|
| **Auth** | `POST /api/auth/setup` | Initialize admin (first time) |
| | `POST /api/auth/login` | Login to get Token |
| | `POST /api/auth/logout` | Logout |
| | `GET /api/auth/profile` | Current user info |
| | `PUT /api/auth/password` | Change password |
| **Backup Tasks** | `GET/POST /api/backup/tasks` | List / Create tasks |
| | `GET/PUT/DELETE /api/backup/tasks/:id` | Detail / Update / Delete |
| | `PUT /api/backup/tasks/:id/toggle` | Enable / Disable |
| | `POST /api/backup/tasks/:id/run` | Trigger manual execution |
| **Backup Records** | `GET /api/backup/records` | List records (with filter) |
| | `GET /api/backup/records/:id` | Record detail |
| | `GET /api/backup/records/:id/logs/stream` | Real-time execution logs (SSE) |
| | `GET /api/backup/records/:id/download` | Download backup file |
| | `POST /api/backup/records/:id/restore` | Restore backup |
| **Storage Targets** | `GET/POST /api/storage-targets` | List / Add targets |
| | `GET/PUT/DELETE /api/storage-targets/:id` | Detail / Update / Delete |
| | `POST /api/storage-targets/test` | Test connection |
| | `POST /api/storage-targets/:id/test` | Test saved connection |
| | `GET /api/storage-targets/:id/usage` | Query usage |
| **Nodes** | `GET/POST /api/nodes` | List / Add nodes |
| | `GET/DELETE /api/nodes/:id` | Detail / Delete |
| | `GET /api/nodes/:id/fs/list` | Directory browser |
| | `POST /api/agent/heartbeat` | Agent heartbeat ⚡ |
| **Notifications** | `GET/POST /api/notifications` | List / Add |
| | `POST /api/notifications/test` | Test notification |
| | `POST /api/notifications/:id/test` | Test saved notification |
| **Dashboard** | `GET /api/dashboard/stats` | Overview statistics |
| | `GET /api/dashboard/timeline` | Backup trend timeline |
| **System** | `GET /api/system/info` | System info (version/disk) |
| | `GET/PUT /api/settings` | System settings |
| **Auth** | `POST /auth/setup` | Initialize admin |
| | `POST /auth/login` | Login |
| | `PUT /auth/password` | Change password |
| **Backup Tasks** | `GET\|POST /backup/tasks` | List / Create |
| | `GET\|PUT\|DELETE /backup/tasks/:id` | Detail / Update / Delete |
| | `PUT /backup/tasks/:id/toggle` | Enable / Disable |
| | `POST /backup/tasks/:id/run` | Manual run |
| **Backup Records** | `GET /backup/records` | List (with filter) |
| | `GET /backup/records/:id/logs/stream` | Real-time logs (SSE) |
| | `GET /backup/records/:id/download` | Download |
| | `POST /backup/records/:id/restore` | Restore |
| **Storage Targets** | `GET\|POST /storage-targets` | List / Add |
| | `POST /storage-targets/test` | Test connection |
| | `GET /storage-targets/rclone/backends` | Rclone backend list |
| **Nodes** | `GET\|POST /nodes` | List / Add |
| | `PUT /nodes/:id` | Edit node |
| | `GET /nodes/:id/fs/list` | Directory browser |
| | `POST /agent/heartbeat` | Agent heartbeat (Token auth) |
| **Notifications** | `GET\|POST /notifications` | List / Add |
| **Dashboard** | `GET /dashboard/stats` | Overview stats |
| **Audit Logs** | `GET /audit-logs` | Operation audit |
| **System** | `GET /system/info` | System info |
| | `GET /system/update-check` | Check for updates |
> ⚡ `POST /api/agent/heartbeat` is a public endpoint authenticated via Node Token instead of JWT.
---
## Cloud Storage Setup Guide
## Tech Stack
### Alibaba Cloud OSS
1. Log in to [Alibaba Cloud Console](https://oss.console.aliyun.com/), create a Bucket
2. Go to RAM Console to create an AccessKey
3. Select "Alibaba Cloud OSS" when adding a storage target in BackupX
4. Enter the Region (e.g. `cn-hangzhou`) and AccessKey — the system auto-assembles the endpoint
### Tencent Cloud COS
1. Log in to [Tencent Cloud Console](https://console.cloud.tencent.com/cos), create a bucket
2. Go to API Key Management to create SecretId/SecretKey
3. Bucket name format is `BucketName-APPID` (e.g. `backup-1250000000`)
### Qiniu Cloud Kodo
1. Log in to [Qiniu Cloud Console](https://portal.qiniu.com/), create a storage space
2. Supported regions: `z0` (East China) / `cn-east-2` (East China-Zhejiang 2) / `z1` (North China) / `z2` (South China) / `na0` (North America) / `as0` (Southeast Asia)
### Google Drive
1. Go to [Google Cloud Console](https://console.cloud.google.com/) and create a project
2. Enable the **Google Drive API**
3. Create an **OAuth 2.0 Client ID** (Web application type)
4. Add redirect URI: `http://your-server/api/storage-targets/google-drive/callback`
5. Enter the Client ID / Secret in BackupX storage management and click Authorize
| Component | Technology |
|-----------|-----------|
| **Backend** | Go · Gin · GORM · SQLite · robfig/cron · rclone |
| **Frontend** | React 18 · TypeScript · ArcoDesign · Vite · Zustand · ECharts |
| **Storage** | rclone (70+ backends) · AWS SDK v2 · Google Drive API v3 |
| **Security** | JWT · bcrypt · AES-256-GCM |
## Contributing
Issues and Pull Requests are welcome!
1. Fork this repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## License
This project is licensed under the [Apache License 2.0](LICENSE).
---
<p align="center">
Made with ❤️ for self-hosters
</p>
[Apache License 2.0](LICENSE)

View File

@@ -0,0 +1,23 @@
#!/bin/sh
set -e
# Backend listens on internal port 8341, Nginx exposes 8340
export BACKUPX_SERVER_PORT="${BACKUPX_SERVER_PORT_INTERNAL:-8341}"
# Start Nginx in background
nginx -g "daemon off;" &
NGINX_PID=$!
# Start BackupX backend
/app/bin/backupx &
APP_PID=$!
# Trap signals for graceful shutdown
trap 'kill $APP_PID $NGINX_PID 2>/dev/null; wait $APP_PID $NGINX_PID 2>/dev/null' SIGTERM SIGINT
echo "BackupX started — Nginx :8340 -> Backend :8341"
# Wait for either process to exit
wait -n $APP_PID $NGINX_PID 2>/dev/null || true
kill $APP_PID $NGINX_PID 2>/dev/null || true
wait $APP_PID $NGINX_PID 2>/dev/null || true

32
deploy/docker/nginx.conf Normal file
View File

@@ -0,0 +1,32 @@
server {
listen 8340;
server_name _;
root /app/web;
index index.html;
# API reverse proxy to backend
location /api/ {
proxy_pass http://127.0.0.1:8341/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 3600s;
}
# SPA fallback
location / {
try_files $uri $uri/ /index.html;
}
# Static assets cache
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff2?)$ {
expires 30d;
add_header Cache-Control "public, immutable";
}
}

30
docker-compose.yml Normal file
View File

@@ -0,0 +1,30 @@
# BackupX Docker Compose
#
# 快速启动docker compose up -d
# 访问地址http://localhost:8340
#
# 如需从源码构建镜像(而非拉取线上镜像),取消注释 build 行并注释 image 行。
services:
backupx:
image: awuqing/backupx:latest
# build: . # 从源码构建时取消此行注释
container_name: backupx
restart: unless-stopped
ports:
- "8340:8340"
volumes:
- backupx-data:/app/data
- /var/run/docker.sock:/var/run/docker.sock # 支持 Web 一键更新
# 挂载需要备份的宿主机目录(按需添加,:ro 表示只读):
# - /var/www:/mnt/www:ro
# - /etc/nginx:/mnt/nginx-conf:ro
# - /home/user/data:/mnt/data:ro
environment:
- TZ=Asia/Shanghai
# 通过 BACKUPX_ 前缀环境变量覆盖配置:
# - BACKUPX_LOG_LEVEL=debug
# - BACKUPX_BACKUP_MAX_CONCURRENT=4
volumes:
backupx-data:

View File

@@ -1,14 +1,15 @@
APP_NAME=backupx
BUILD_DIR=./bin
VERSION=$(shell git describe --tags --always --dirty 2>/dev/null || echo "dev")
.PHONY: build run test
build:
mkdir -p $(BUILD_DIR)
go build -o $(BUILD_DIR)/$(APP_NAME) ./cmd/backupx
go build -trimpath -ldflags "-s -w -X main.version=$(VERSION)" -o $(BUILD_DIR)/$(APP_NAME) ./cmd/backupx
run:
go run ./cmd/backupx
go run -ldflags "-X main.version=$(VERSION)" ./cmd/backupx
test:
go test ./...

View File

@@ -0,0 +1,70 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"os/signal"
"syscall"
"backupx/server/internal/agent"
)
// runAgent 是 `backupx agent` 子命令入口。
//
// 用法:
//
// backupx agent --master http://master:8340 --token <token>
// backupx agent --config /etc/backupx-agent.yaml
//
// 配置优先级CLI 参数 > 配置文件 > 环境变量
func runAgent(args []string) {
fs := flag.NewFlagSet("agent", flag.ExitOnError)
configPath := fs.String("config", "", "path to agent config YAML (optional)")
master := fs.String("master", "", "master URL, e.g. http://master.example.com:8340")
token := fs.String("token", "", "agent authentication token")
tempDir := fs.String("temp-dir", "", "local temp directory for backup artifacts")
insecureTLS := fs.Bool("insecure-tls", false, "skip TLS verification (testing only)")
if err := fs.Parse(args); err != nil {
os.Exit(2)
}
cfg, err := loadAgentConfig(*configPath)
if err != nil {
fmt.Fprintf(os.Stderr, "agent: load config: %v\n", err)
os.Exit(2)
}
cfg.MergeWithFlags(*master, *token, *tempDir)
if *insecureTLS {
cfg.InsecureSkipTLSVerify = true
}
if err := cfg.Validate(); err != nil {
fmt.Fprintf(os.Stderr, "agent: %v\n", err)
os.Exit(2)
}
a, err := agent.New(cfg, version)
if err != nil {
fmt.Fprintf(os.Stderr, "agent: init: %v\n", err)
os.Exit(1)
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
fmt.Fprintf(os.Stderr, "backupx agent %s starting (master=%s)\n", version, cfg.Master)
if err := a.Run(ctx); err != nil && err != context.Canceled {
fmt.Fprintf(os.Stderr, "agent: %v\n", err)
os.Exit(1)
}
}
// loadAgentConfig 按优先级加载配置:如果提供了 --config 就用文件,否则走环境变量。
func loadAgentConfig(configPath string) (*agent.Config, error) {
if configPath != "" {
return agent.LoadConfigFile(configPath)
}
return agent.LoadConfigFromEnv()
}

View File

@@ -0,0 +1,98 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"os/signal"
"syscall"
"backupx/server/internal/backint"
)
// runBackint 是 `backupx backint` 子命令入口。
//
// CLI 参数遵循 SAP HANA Backint 规范:
//
// backupx backint -f <function> -i <input-file> -o <output-file> -p <param-file>
// [-u <user>] [-c <config-prefix>] [-l <log-file>] [-v <version>]
//
// 除 -f / -i / -o / -p 外其余参数接受但忽略(兼容 SAP 调用约定)。
func runBackint(args []string) {
fs := flag.NewFlagSet("backint", flag.ExitOnError)
fnStr := fs.String("f", "", "function: backup | restore | inquire | delete")
inputPath := fs.String("i", "", "input file path")
outputPath := fs.String("o", "", "output file path")
paramFile := fs.String("p", "", "parameter file path")
// 以下参数仅为兼容 SAP 调用约定,当前未使用
_ = fs.String("u", "", "user (ignored)")
_ = fs.String("c", "", "config-prefix (ignored)")
_ = fs.String("l", "", "log file override (ignored, use LOG_FILE in params)")
_ = fs.String("v", "", "backint version (ignored)")
if err := fs.Parse(args); err != nil {
os.Exit(2)
}
if *fnStr == "" || *inputPath == "" || *outputPath == "" || *paramFile == "" {
fmt.Fprintln(os.Stderr, "backint: -f, -i, -o, -p are required")
fs.Usage()
os.Exit(2)
}
fn, err := backint.ParseFunction(*fnStr)
if err != nil {
fmt.Fprintf(os.Stderr, "backint: %v\n", err)
os.Exit(2)
}
cfg, err := backint.LoadConfigFile(*paramFile)
if err != nil {
fmt.Fprintf(os.Stderr, "backint: load config: %v\n", err)
os.Exit(2)
}
// 配置日志重定向(如果指定 LOG_FILE
restoreLog, err := redirectStderr(cfg.LogFile)
if err != nil {
fmt.Fprintf(os.Stderr, "backint: open log: %v\n", err)
os.Exit(2)
}
defer restoreLog()
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
agent, err := backint.NewAgent(ctx, cfg)
if err != nil {
fmt.Fprintf(os.Stderr, "backint: init agent: %v\n", err)
os.Exit(1)
}
defer func() { _ = agent.Close() }()
if err := agent.Run(ctx, fn, *inputPath, *outputPath); err != nil {
fmt.Fprintf(os.Stderr, "backint: run: %v\n", err)
os.Exit(1)
}
}
// redirectStderr 将 stderr 重定向到指定日志文件,返回恢复函数。
// 空字符串表示保持原样。
func redirectStderr(path string) (func(), error) {
if path == "" {
return func() {}, nil
}
f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0644)
if err != nil {
return nil, err
}
orig := os.Stderr
os.Stderr = f
return func() {
os.Stderr = orig
_ = f.Close()
}, nil
}

View File

@@ -10,11 +10,31 @@ import (
"backupx/server/internal/app"
"backupx/server/internal/config"
"backupx/server/internal/security"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
gormlogger "gorm.io/gorm/logger"
)
var version = "dev"
func main() {
// 子命令分发reset-password
if len(os.Args) > 1 && os.Args[1] == "reset-password" {
runResetPassword(os.Args[2:])
return
}
// 子命令分发backintSAP HANA Backint Agent 模式)
if len(os.Args) > 1 && os.Args[1] == "backint" {
runBackint(os.Args[2:])
return
}
// 子命令分发agent远程节点 Agent 模式)
if len(os.Args) > 1 && os.Args[1] == "agent" {
runAgent(os.Args[2:])
return
}
var configPath string
var showVersion bool
@@ -48,3 +68,58 @@ func main() {
os.Exit(1)
}
}
// runResetPassword 通过 CLI 直接操作 SQLite 重置用户密码,无需完整 app 初始化。
// 用法backupx reset-password --username admin --password newpass123 [--config path]
func runResetPassword(args []string) {
fs := flag.NewFlagSet("reset-password", flag.ExitOnError)
username := fs.String("username", "admin", "要重置密码的用户名")
password := fs.String("password", "", "新密码(至少 8 个字符)")
configPath := fs.String("config", "", "配置文件路径")
if err := fs.Parse(args); err != nil {
os.Exit(1)
}
if *password == "" {
fmt.Fprintln(os.Stderr, "错误:--password 参数为必填项")
fs.Usage()
os.Exit(1)
}
if len(*password) < 8 {
fmt.Fprintln(os.Stderr, "错误:密码长度至少 8 个字符")
os.Exit(1)
}
cfg, err := config.Load(*configPath)
if err != nil {
fmt.Fprintf(os.Stderr, "加载配置失败:%v\n", err)
os.Exit(1)
}
db, err := gorm.Open(sqlite.Open(cfg.Database.Path), &gorm.Config{Logger: gormlogger.Default.LogMode(gormlogger.Silent)})
if err != nil {
fmt.Fprintf(os.Stderr, "打开数据库失败:%v\n", err)
os.Exit(1)
}
var count int64
db.Table("users").Where("username = ?", *username).Count(&count)
if count == 0 {
fmt.Fprintf(os.Stderr, "错误:用户 %q 不存在\n", *username)
os.Exit(1)
}
hash, err := security.HashPassword(*password)
if err != nil {
fmt.Fprintf(os.Stderr, "密码哈希失败:%v\n", err)
os.Exit(1)
}
result := db.Table("users").Where("username = ?", *username).Update("password_hash", hash)
if result.Error != nil {
fmt.Fprintf(os.Stderr, "密码更新失败:%v\n", result.Error)
os.Exit(1)
}
fmt.Printf("用户 %q 密码已重置成功\n", *username)
}

View File

@@ -3,97 +3,258 @@ module backupx/server
go 1.25.0
require (
github.com/aws/aws-sdk-go-v2 v1.41.3
github.com/aws/aws-sdk-go-v2/credentials v1.19.11
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4
github.com/gin-gonic/gin v1.10.1
github.com/glebarez/sqlite v1.11.0
github.com/golang-jwt/jwt/v5 v5.2.1
github.com/golang-jwt/jwt/v5 v5.3.0
github.com/natefinch/lumberjack v2.0.0+incompatible
github.com/rclone/rclone v1.73.3
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/viper v1.20.0
github.com/studio-b12/gowebdav v0.12.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.45.0
golang.org/x/oauth2 v0.25.0
google.golang.org/api v0.215.0
golang.org/x/crypto v0.48.0
golang.org/x/oauth2 v0.34.0
google.golang.org/api v0.255.0
gopkg.in/yaml.v3 v3.0.1
gorm.io/gorm v1.25.12
)
require (
cloud.google.com/go/auth v0.13.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/compute/metadata v0.6.0 // indirect
cloud.google.com/go/auth v0.17.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.20.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.3 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.3 // indirect
github.com/Azure/go-ntlmssp v0.0.2-0.20251110135918-10b7b7e7cd26 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect
github.com/BurntSushi/toml v1.6.0 // indirect
github.com/FilenCloudDienste/filen-sdk-go v0.0.37 // indirect
github.com/Files-com/files-sdk-go/v3 v3.2.264 // indirect
github.com/IBM/go-sdk-core/v5 v5.18.5 // indirect
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf // indirect
github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e // indirect
github.com/ProtonMail/go-crypto v1.3.0 // indirect
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f // indirect
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/ProtonMail/gopenpgp/v2 v2.9.0 // indirect
github.com/PuerkitoBio/goquery v1.10.3 // indirect
github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/anchore/go-lzo v0.1.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect
github.com/aws/aws-sdk-go-v2 v1.41.3 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.6 // indirect
github.com/aws/aws-sdk-go-v2/config v1.31.17 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.11 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.19 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.19 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.20 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.11 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.19 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.19 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.12 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.16 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.8 // indirect
github.com/aws/smithy-go v1.24.2 // indirect
github.com/bytedance/sonic v1.11.6 // indirect
github.com/bytedance/sonic/loader v0.1.1 // indirect
github.com/cloudwego/base64x v0.1.4 // indirect
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
github.com/buengese/sgzip v0.1.1 // indirect
github.com/buger/jsonparser v1.1.2 // indirect
github.com/bytedance/sonic v1.13.2 // indirect
github.com/bytedance/sonic/loader v0.2.4 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.3.0 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cloudinary/cloudinary-go/v2 v2.13.0 // indirect
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/cloudwego/base64x v0.1.5 // indirect
github.com/colinmarc/hdfs/v2 v2.4.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd/v22 v22.6.0 // indirect
github.com/creasty/defaults v1.8.0 // indirect
github.com/cronokirby/saferith v0.33.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/diskfs/go-diskfs v1.7.0 // indirect
github.com/dromara/dongle v1.0.1 // indirect
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.9.1 // indirect
github.com/emersion/go-message v0.18.2 // indirect
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/flynn/noise v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.11 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/gin-contrib/sse v1.0.0 // indirect
github.com/glebarez/go-sqlite v1.21.2 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect
github.com/go-git/go-billy/v5 v5.6.2 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-openapi/errors v0.22.4 // indirect
github.com/go-openapi/strfmt v0.25.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.20.0 // indirect
github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/go-playground/validator/v10 v10.28.0 // indirect
github.com/go-resty/resty/v2 v2.16.5 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/gofrs/flock v0.13.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.1 // indirect
github.com/hashicorp/errwrap v1.0.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
github.com/gorilla/schema v1.4.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/internxt/rclone-adapter v0.0.0-20260220172730-613f4cc8b8fd // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/jlaffaye/ftp v0.2.0 // indirect
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
github.com/klauspost/compress v1.18.1 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lanrat/extsort v1.4.2 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lpar/date v1.0.0 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/mailru/easyjson v0.9.1 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/ncw/swift/v2 v2.0.5 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/onsi/ginkgo/v2 v2.19.0 // indirect
github.com/oracle/oci-go-sdk/v65 v65.104.0 // indirect
github.com/panjf2000/ants/v2 v2.11.3 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect
github.com/peterh/liner v1.2.2 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pkg/sftp v1.13.10 // indirect
github.com/pkg/xattr v0.4.12 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/pquerna/otp v1.5.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.2 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 // indirect
github.com/rclone/Proton-API-Bridge v1.0.1-0.20260127174007-77f974840d11 // indirect
github.com/rclone/go-proton-api v1.0.1-0.20260127173028-eb465cac3b18 // indirect
github.com/relvacode/iso8601 v1.7.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rfjakob/eme v1.1.2 // indirect
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
github.com/sagikazarmark/locafero v0.7.0 // indirect
github.com/samber/lo v1.52.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.10 // indirect
github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af // indirect
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.12.0 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/spf13/pflag v1.0.6 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/stretchr/testify v1.11.1 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/tyler-smith/go-bip39 v1.1.0 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
go.opentelemetry.io/otel v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/trace v1.29.0 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
github.com/yunify/qingstor-sdk-go/v3 v3.2.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeebo/blake3 v0.2.4 // indirect
github.com/zeebo/errs v1.4.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.mongodb.org/mongo-driver v1.17.6 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.63.0 // indirect
go.opentelemetry.io/otel v1.39.0 // indirect
go.opentelemetry.io/otel/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.uber.org/multierr v1.10.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/net v0.47.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/text v0.31.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241223144023-3abc09e42ca8 // indirect
google.golang.org/grpc v1.67.3 // indirect
google.golang.org/protobuf v1.36.5 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
golang.org/x/arch v0.14.0 // indirect
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/image v0.32.0 // indirect
golang.org/x/mod v0.32.0 // indirect
golang.org/x/net v0.51.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.41.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/grpc v1.79.3 // indirect
google.golang.org/protobuf v1.36.10 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/validator.v2 v2.0.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.22.5 // indirect
modernc.org/mathutil v1.5.0 // indirect
modernc.org/memory v1.5.0 // indirect
modernc.org/sqlite v1.23.1 // indirect
moul.io/http2curl/v2 v2.3.0 // indirect
storj.io/common v0.0.0-20251107171817-6221ae45072c // indirect
storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 // indirect
storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 // indirect
storj.io/infectious v0.0.2 // indirect
storj.io/picobuf v0.0.4 // indirect
storj.io/uplink v1.13.1 // indirect
)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,203 @@
package agent
import (
"context"
"encoding/json"
"fmt"
"log"
"net"
"os"
"runtime"
"strings"
"sync"
"time"
)
// Agent 是 Agent 进程的主控制器。
type Agent struct {
cfg *Config
client *MasterClient
executor *Executor
version string
mu sync.Mutex
started bool
}
// New 构造 Agent。
func New(cfg *Config, version string) (*Agent, error) {
if err := cfg.Validate(); err != nil {
return nil, err
}
client := NewMasterClient(cfg.Master, cfg.Token, cfg.InsecureSkipTLSVerify)
executor := NewExecutor(client, cfg.TempDir)
return &Agent{
cfg: cfg,
client: client,
executor: executor,
version: version,
}, nil
}
// Run 启动 Agent 主循环,阻塞直到 ctx 被取消。
func (a *Agent) Run(ctx context.Context) error {
a.mu.Lock()
if a.started {
a.mu.Unlock()
return fmt.Errorf("agent already started")
}
a.started = true
a.mu.Unlock()
hbInterval := parseDuration(a.cfg.HeartbeatInterval, 15*time.Second)
pollInterval := parseDuration(a.cfg.PollInterval, 5*time.Second)
// 首次握手:通过一次心跳确认 token 有效
if err := a.heartbeatOnce(ctx); err != nil {
return fmt.Errorf("initial heartbeat failed: %w", err)
}
log.Printf("[agent] connected to master %s", a.cfg.Master)
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
a.heartbeatLoop(ctx, hbInterval)
}()
go func() {
defer wg.Done()
a.pollLoop(ctx, pollInterval)
}()
wg.Wait()
return ctx.Err()
}
// heartbeatLoop 定期发送心跳。
func (a *Agent) heartbeatLoop(ctx context.Context, interval time.Duration) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := a.heartbeatOnce(ctx); err != nil {
log.Printf("[agent] heartbeat failed: %v", err)
}
}
}
}
func (a *Agent) heartbeatOnce(ctx context.Context) error {
hostname, _ := os.Hostname()
req := HeartbeatRequest{
Token: a.cfg.Token,
Hostname: hostname,
IPAddress: detectLocalIP(),
AgentVersion: a.version,
OS: runtime.GOOS,
Arch: runtime.GOARCH,
}
_, err := a.client.Heartbeat(ctx, req)
return err
}
// pollLoop 定期拉取并处理待执行命令。
func (a *Agent) pollLoop(ctx context.Context, interval time.Duration) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
a.pollAndHandleOnce(ctx)
}
}
}
func (a *Agent) pollAndHandleOnce(ctx context.Context) {
cmd, err := a.client.PollCommand(ctx)
if err != nil {
log.Printf("[agent] poll command failed: %v", err)
return
}
if cmd == nil {
return
}
log.Printf("[agent] received command #%d type=%s", cmd.ID, cmd.Type)
switch cmd.Type {
case "run_task":
a.handleRunTask(ctx, cmd)
case "list_dir":
a.handleListDir(ctx, cmd)
default:
msg := fmt.Sprintf("unknown command type: %s", cmd.Type)
log.Printf("[agent] %s", msg)
_ = a.client.SubmitCommandResult(ctx, cmd.ID, false, msg, nil)
}
}
// handleRunTask 处理 run_task 命令
func (a *Agent) handleRunTask(ctx context.Context, cmd *CommandPayload) {
var payload struct {
TaskID uint `json:"taskId"`
RecordID uint `json:"recordId"`
}
if err := json.Unmarshal(cmd.Payload, &payload); err != nil {
_ = a.client.SubmitCommandResult(ctx, cmd.ID, false, "invalid payload: "+err.Error(), nil)
return
}
if err := a.executor.ExecuteRunTask(ctx, payload.TaskID, payload.RecordID); err != nil {
_ = a.client.SubmitCommandResult(ctx, cmd.ID, false, err.Error(), nil)
return
}
_ = a.client.SubmitCommandResult(ctx, cmd.ID, true, "", map[string]any{
"taskId": payload.TaskID,
"recordId": payload.RecordID,
})
}
// handleListDir 处理 list_dir 命令(阶段四实现)
func (a *Agent) handleListDir(ctx context.Context, cmd *CommandPayload) {
var payload struct {
Path string `json:"path"`
}
if err := json.Unmarshal(cmd.Payload, &payload); err != nil {
_ = a.client.SubmitCommandResult(ctx, cmd.ID, false, "invalid payload: "+err.Error(), nil)
return
}
entries, err := listLocalDir(payload.Path)
if err != nil {
_ = a.client.SubmitCommandResult(ctx, cmd.ID, false, err.Error(), nil)
return
}
_ = a.client.SubmitCommandResult(ctx, cmd.ID, true, "", map[string]any{"entries": entries})
}
// 辅助函数
func parseDuration(s string, fallback time.Duration) time.Duration {
if strings.TrimSpace(s) == "" {
return fallback
}
if d, err := time.ParseDuration(s); err == nil {
return d
}
return fallback
}
func detectLocalIP() string {
addrs, err := net.InterfaceAddrs()
if err != nil {
return ""
}
for _, addr := range addrs {
if ipNet, ok := addr.(*net.IPNet); ok && !ipNet.IP.IsLoopback() {
if ipNet.IP.To4() != nil {
return ipNet.IP.String()
}
}
}
return ""
}

View File

@@ -0,0 +1,208 @@
package agent
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// MasterClient 是 Agent 调用 Master HTTP API 的封装。
type MasterClient struct {
baseURL string
token string
httpClient *http.Client
}
// NewMasterClient 构造 Master 客户端。
func NewMasterClient(baseURL, token string, insecureTLS bool) *MasterClient {
transport := &http.Transport{}
if insecureTLS {
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
return &MasterClient{
baseURL: strings.TrimRight(baseURL, "/"),
token: token,
httpClient: &http.Client{
Timeout: 120 * time.Second,
Transport: transport,
},
}
}
// HeartbeatRequest Agent 上报心跳的请求
type HeartbeatRequest struct {
Token string `json:"token"`
Hostname string `json:"hostname,omitempty"`
IPAddress string `json:"ipAddress,omitempty"`
AgentVersion string `json:"agentVersion,omitempty"`
OS string `json:"os,omitempty"`
Arch string `json:"arch,omitempty"`
}
// HeartbeatResponse Master 返回的心跳响应
type HeartbeatResponse struct {
Status string `json:"status"`
NodeID uint `json:"nodeId"`
Name string `json:"name"`
}
// Heartbeat 上报心跳并获取节点元信息
func (c *MasterClient) Heartbeat(ctx context.Context, req HeartbeatRequest) (*HeartbeatResponse, error) {
var resp HeartbeatResponse
if err := c.do(ctx, http.MethodPost, "/api/agent/heartbeat", req, &resp); err != nil {
return nil, err
}
return &resp, nil
}
// CommandPayload 与 service.AgentCommandPayload 对齐
type CommandPayload struct {
ID uint `json:"id"`
Type string `json:"type"`
Payload json.RawMessage `json:"payload,omitempty"`
}
// PollCommandResponse 轮询响应:无命令时 Command 为 nil
type PollCommandResponse struct {
Command *CommandPayload `json:"command"`
}
// PollCommand 拉取下一条待执行命令
func (c *MasterClient) PollCommand(ctx context.Context) (*CommandPayload, error) {
var resp PollCommandResponse
if err := c.do(ctx, http.MethodPost, "/api/agent/commands/poll", nil, &resp); err != nil {
return nil, err
}
return resp.Command, nil
}
// SubmitCommandResult 上报命令执行结果
func (c *MasterClient) SubmitCommandResult(ctx context.Context, cmdID uint, success bool, errorMsg string, result any) error {
var resultJSON json.RawMessage
if result != nil {
data, err := json.Marshal(result)
if err != nil {
return fmt.Errorf("marshal result: %w", err)
}
resultJSON = data
}
payload := map[string]any{
"success": success,
"errorMessage": errorMsg,
}
if resultJSON != nil {
payload["result"] = resultJSON
}
path := fmt.Sprintf("/api/agent/commands/%d/result", cmdID)
return c.do(ctx, http.MethodPost, path, payload, nil)
}
// TaskSpec 与 service.AgentTaskSpec 对齐
type TaskSpec struct {
TaskID uint `json:"taskId"`
Name string `json:"name"`
Type string `json:"type"`
SourcePath string `json:"sourcePath"`
SourcePaths string `json:"sourcePaths"`
ExcludePatterns string `json:"excludePatterns"`
DBHost string `json:"dbHost"`
DBPort int `json:"dbPort"`
DBUser string `json:"dbUser"`
DBPassword string `json:"dbPassword"`
DBName string `json:"dbName"`
DBPath string `json:"dbPath"`
ExtraConfig string `json:"extraConfig"`
Compression string `json:"compression"`
Encrypt bool `json:"encrypt"`
StorageTargets []StorageTargetConfig `json:"storageTargets"`
}
// StorageTargetConfig 与 service.AgentStorageTargetConfig 对齐
type StorageTargetConfig struct {
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
}
// GetTaskSpec 拉取任务规格
func (c *MasterClient) GetTaskSpec(ctx context.Context, taskID uint) (*TaskSpec, error) {
var spec TaskSpec
path := fmt.Sprintf("/api/agent/tasks/%d", taskID)
if err := c.do(ctx, http.MethodGet, path, nil, &spec); err != nil {
return nil, err
}
return &spec, nil
}
// RecordUpdate 与 service.AgentRecordUpdate 对齐
type RecordUpdate struct {
Status string `json:"status,omitempty"`
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"`
}
// UpdateRecord 上报备份记录的状态/日志
func (c *MasterClient) UpdateRecord(ctx context.Context, recordID uint, update RecordUpdate) error {
path := fmt.Sprintf("/api/agent/records/%d", recordID)
return c.do(ctx, http.MethodPost, path, update, nil)
}
// do 是通用 HTTP 调用。所有 Agent API 都统一走 JSON + X-Agent-Token。
func (c *MasterClient) do(ctx context.Context, method, path string, body any, out any) error {
var reqBody io.Reader
if body != nil {
data, err := json.Marshal(body)
if err != nil {
return fmt.Errorf("marshal request: %w", err)
}
reqBody = bytes.NewReader(data)
}
req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, reqBody)
if err != nil {
return err
}
req.Header.Set("X-Agent-Token", c.token)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := c.httpClient.Do(req)
if err != nil {
return fmt.Errorf("%s %s: %w", method, path, err)
}
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("read response: %w", err)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("%s %s: http %d: %s", method, path, resp.StatusCode, string(data))
}
if out == nil {
return nil
}
// BackupX API 统一封装成 {code, data, message} 形式,需要解出 data 字段
var envelope struct {
Code string `json:"code"`
Data json.RawMessage `json:"data"`
Message string `json:"message"`
}
if err := json.Unmarshal(data, &envelope); err == nil && envelope.Data != nil {
if err := json.Unmarshal(envelope.Data, out); err != nil {
return fmt.Errorf("decode data: %w", err)
}
return nil
}
// 兼容直接返回对象的情况
return json.Unmarshal(data, out)
}

View File

@@ -0,0 +1,105 @@
// Package agent 实现 BackupX 远程 Agent。
//
// Agent 是一个独立的 Go 进程,部署在远程服务器上,通过 HTTP 轮询的方式
// 与 Master 通信:定期上报心跳、拉取 Master 下发的命令、本地执行备份、
// 把执行结果和日志回报给 Master。
//
// 通信协议见 server/internal/http/agent_handler.go。
package agent
import (
"errors"
"fmt"
"os"
"strings"
"gopkg.in/yaml.v3"
)
// Config 是 Agent 的运行时配置。
type Config struct {
// Master BackupX Master 的 HTTP 基础地址,例如 http://master.example.com:8340
Master string `yaml:"master"`
// Token 节点认证令牌(在 Master 创建节点时生成)
Token string `yaml:"token"`
// HeartbeatInterval 心跳间隔,默认 15s
HeartbeatInterval string `yaml:"heartbeatInterval"`
// PollInterval 命令轮询间隔,默认 5s
PollInterval string `yaml:"pollInterval"`
// TempDir 备份临时目录,默认 /tmp/backupx-agent
TempDir string `yaml:"tempDir"`
// InsecureSkipTLSVerify 测试环境允许跳过 TLS 证书校验
InsecureSkipTLSVerify bool `yaml:"insecureSkipTlsVerify"`
}
// LoadConfigFile 从 YAML 文件加载 Agent 配置。
func LoadConfigFile(path string) (*Config, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read agent config: %w", err)
}
var cfg Config
if err := yaml.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("parse agent config: %w", err)
}
return applyConfigDefaults(&cfg)
}
// LoadConfigFromEnv 从环境变量加载 Agent 配置。优先级低于 --config 文件。
//
// 支持的环境变量:
// - BACKUPX_AGENT_MASTER Master URL
// - BACKUPX_AGENT_TOKEN 节点认证令牌
// - BACKUPX_AGENT_HEARTBEAT 心跳间隔(如 15s
// - BACKUPX_AGENT_POLL 命令轮询间隔(如 5s
// - BACKUPX_AGENT_TEMP_DIR 临时目录
// - BACKUPX_AGENT_INSECURE_TLS true / 1 跳过 TLS 校验
func LoadConfigFromEnv() (*Config, error) {
cfg := &Config{
Master: strings.TrimSpace(os.Getenv("BACKUPX_AGENT_MASTER")),
Token: strings.TrimSpace(os.Getenv("BACKUPX_AGENT_TOKEN")),
HeartbeatInterval: strings.TrimSpace(os.Getenv("BACKUPX_AGENT_HEARTBEAT")),
PollInterval: strings.TrimSpace(os.Getenv("BACKUPX_AGENT_POLL")),
TempDir: strings.TrimSpace(os.Getenv("BACKUPX_AGENT_TEMP_DIR")),
InsecureSkipTLSVerify: strings.EqualFold(os.Getenv("BACKUPX_AGENT_INSECURE_TLS"), "true") || os.Getenv("BACKUPX_AGENT_INSECURE_TLS") == "1",
}
return applyConfigDefaults(cfg)
}
// MergeWithFlags 把命令行覆盖值合并入配置(非空覆盖)。
func (c *Config) MergeWithFlags(master, token, tempDir string) {
if strings.TrimSpace(master) != "" {
c.Master = master
}
if strings.TrimSpace(token) != "" {
c.Token = token
}
if strings.TrimSpace(tempDir) != "" {
c.TempDir = tempDir
}
}
// Validate 校验必填字段。
func (c *Config) Validate() error {
if strings.TrimSpace(c.Master) == "" {
return errors.New("master url is required (set via --master, BACKUPX_AGENT_MASTER or config file)")
}
if strings.TrimSpace(c.Token) == "" {
return errors.New("token is required (set via --token, BACKUPX_AGENT_TOKEN or config file)")
}
return nil
}
func applyConfigDefaults(cfg *Config) (*Config, error) {
if cfg.HeartbeatInterval == "" {
cfg.HeartbeatInterval = "15s"
}
if cfg.PollInterval == "" {
cfg.PollInterval = "5s"
}
if cfg.TempDir == "" {
cfg.TempDir = "/tmp/backupx-agent"
}
cfg.Master = strings.TrimRight(strings.TrimSpace(cfg.Master), "/")
return cfg, nil
}

View File

@@ -0,0 +1,101 @@
package agent
import (
"os"
"path/filepath"
"testing"
)
func TestLoadConfigFile(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "agent.yaml")
content := `master: http://master.example.com:8340/
token: abc123
heartbeatInterval: 20s
pollInterval: 3s
tempDir: /var/backupx-agent
insecureSkipTlsVerify: true
`
if err := os.WriteFile(path, []byte(content), 0644); err != nil {
t.Fatal(err)
}
cfg, err := LoadConfigFile(path)
if err != nil {
t.Fatalf("load: %v", err)
}
if cfg.Master != "http://master.example.com:8340" {
t.Errorf("trailing slash should be trimmed: %q", cfg.Master)
}
if cfg.Token != "abc123" {
t.Errorf("token: %q", cfg.Token)
}
if cfg.HeartbeatInterval != "20s" || cfg.PollInterval != "3s" {
t.Errorf("intervals: %+v", cfg)
}
if !cfg.InsecureSkipTLSVerify {
t.Errorf("insecure should be true")
}
}
func TestLoadConfigDefaults(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "agent.yaml")
if err := os.WriteFile(path, []byte("master: http://m\ntoken: t\n"), 0644); err != nil {
t.Fatal(err)
}
cfg, err := LoadConfigFile(path)
if err != nil {
t.Fatal(err)
}
if cfg.HeartbeatInterval != "15s" || cfg.PollInterval != "5s" {
t.Errorf("default intervals not applied: %+v", cfg)
}
if cfg.TempDir != "/tmp/backupx-agent" {
t.Errorf("default tempdir: %q", cfg.TempDir)
}
}
func TestConfigValidate(t *testing.T) {
cases := []struct {
name string
cfg Config
wantErr bool
}{
{"valid", Config{Master: "http://m", Token: "t"}, false},
{"missing master", Config{Token: "t"}, true},
{"missing token", Config{Master: "http://m"}, true},
}
for _, c := range cases {
err := c.cfg.Validate()
if (err != nil) != c.wantErr {
t.Errorf("%s: err=%v wantErr=%v", c.name, err, c.wantErr)
}
}
}
func TestMergeWithFlags(t *testing.T) {
cfg := &Config{Master: "http://old", Token: "old"}
cfg.MergeWithFlags("http://new", "", "/tmp/x")
if cfg.Master != "http://new" {
t.Errorf("master not overridden: %q", cfg.Master)
}
if cfg.Token != "old" {
t.Errorf("empty flag should not override: %q", cfg.Token)
}
if cfg.TempDir != "/tmp/x" {
t.Errorf("tempDir: %q", cfg.TempDir)
}
}
func TestLoadConfigFromEnv(t *testing.T) {
t.Setenv("BACKUPX_AGENT_MASTER", "http://env-master")
t.Setenv("BACKUPX_AGENT_TOKEN", "env-token")
t.Setenv("BACKUPX_AGENT_INSECURE_TLS", "true")
cfg, err := LoadConfigFromEnv()
if err != nil {
t.Fatal(err)
}
if cfg.Master != "http://env-master" || cfg.Token != "env-token" || !cfg.InsecureSkipTLSVerify {
t.Errorf("env not picked up: %+v", cfg)
}
}

View File

@@ -0,0 +1,266 @@
package agent
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"backupx/server/internal/backup"
"backupx/server/internal/storage"
storageRclone "backupx/server/internal/storage/rclone"
"backupx/server/pkg/compress"
)
// Executor 负责在 Agent 本地执行命令。
type Executor struct {
client *MasterClient
tempDir string
backupRegistry *backup.Registry
storageRegistry *storage.Registry
}
// NewExecutor 构造执行器。预先初始化 backup runner 与 storage registry。
func NewExecutor(client *MasterClient, tempDir string) *Executor {
backupRegistry := backup.NewRegistry(
backup.NewFileRunner(),
backup.NewSQLiteRunner(),
backup.NewMySQLRunner(nil),
backup.NewPostgreSQLRunner(nil),
backup.NewSAPHANARunner(nil),
)
storageRegistry := storage.NewRegistry(
storageRclone.NewLocalDiskFactory(),
storageRclone.NewS3Factory(),
storageRclone.NewWebDAVFactory(),
storageRclone.NewGoogleDriveFactory(),
storageRclone.NewAliyunOSSFactory(),
storageRclone.NewTencentCOSFactory(),
storageRclone.NewQiniuKodoFactory(),
storageRclone.NewFTPFactory(),
storageRclone.NewRcloneFactory(),
)
storageRclone.RegisterAllBackends(storageRegistry)
return &Executor{
client: client,
tempDir: tempDir,
backupRegistry: backupRegistry,
storageRegistry: storageRegistry,
}
}
// ExecuteRunTask 处理 run_task 命令:拉规格 → 执行 runner → 压缩 → 上传 → 上报记录。
//
// 注意Agent 当前不支持 Encrypt=true加密密钥不下发到 Agent避免密钥扩散
// 遇到启用加密的任务会向 Master 上报失败并返回错误。
func (e *Executor) ExecuteRunTask(ctx context.Context, taskID, recordID uint) error {
// 1) 拉取任务规格
spec, err := e.client.GetTaskSpec(ctx, taskID)
if err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("拉取任务规格失败: %v", err))
return err
}
if spec.Encrypt {
msg := "Agent 不支持加密备份(加密密钥仅在 Master 端持有)"
e.reportRecordFailure(ctx, recordID, msg)
return fmt.Errorf("%s", msg)
}
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 开始执行任务 %s (type=%s)\n", spec.Name, spec.Type))
// 2) 构造 backup.TaskSpec 并找对应 runner
startedAt := time.Now().UTC()
if err := os.MkdirAll(e.tempDir, 0o755); err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("创建临时目录失败: %v", err))
return err
}
backupSpec := buildBackupTaskSpec(spec, startedAt, e.tempDir)
runner, err := e.backupRegistry.Runner(backupSpec.Type)
if err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("不支持的备份类型: %v", err))
return err
}
// 3) 运行 runner
logger := newRecordLogger(ctx, e.client, recordID)
result, err := runner.Run(ctx, backupSpec, logger)
if err != nil {
e.reportRecordFailure(ctx, recordID, err.Error())
return err
}
defer os.RemoveAll(result.TempDir)
// 4) 可选 gzip 压缩
finalPath := result.ArtifactPath
if strings.EqualFold(spec.Compression, "gzip") && !strings.HasSuffix(strings.ToLower(finalPath), ".gz") {
e.appendLog(ctx, recordID, "[agent] 开始压缩备份文件\n")
compressedPath, compressErr := compress.GzipFile(finalPath)
if compressErr != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("压缩失败: %v", compressErr))
return compressErr
}
finalPath = compressedPath
}
info, err := os.Stat(finalPath)
if err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("获取文件信息失败: %v", err))
return err
}
fileName := filepath.Base(finalPath)
fileSize := info.Size()
storagePath := backup.BuildStorageKey(spec.Type, startedAt, fileName)
// 5) 计算 checksum一次读一次并上传到所有目标
checksum, err := computeFileSHA256(finalPath)
if err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("计算 checksum 失败: %v", err))
return err
}
if len(spec.StorageTargets) == 0 {
e.reportRecordFailure(ctx, recordID, "没有关联的存储目标")
return fmt.Errorf("no storage targets")
}
for _, target := range spec.StorageTargets {
if err := e.uploadToTarget(ctx, recordID, target, finalPath, storagePath, fileSize, spec.TaskID); err != nil {
e.reportRecordFailure(ctx, recordID, fmt.Sprintf("上传到 %s 失败: %v", target.Name, err))
return err
}
e.appendLog(ctx, recordID, fmt.Sprintf("[agent] 已上传到存储目标 %s\n", target.Name))
}
// 6) 上报最终成功
return e.client.UpdateRecord(ctx, recordID, RecordUpdate{
Status: "success",
FileName: fileName,
FileSize: fileSize,
Checksum: checksum,
StoragePath: storagePath,
LogAppend: fmt.Sprintf("[agent] 任务完成,总计 %d 字节\n", fileSize),
})
}
// uploadToTarget 上传单个目标。为保持简化不做上传级重试rclone 本身已有 low-level 重试)。
func (e *Executor) uploadToTarget(ctx context.Context, recordID uint, target StorageTargetConfig, filePath, objectKey string, fileSize int64, taskID uint) error {
var rawConfig map[string]any
if len(target.Config) > 0 {
// DecodeRawConfig 通过 json 解析
if err := jsonUnmarshalMap(target.Config, &rawConfig); err != nil {
return fmt.Errorf("parse storage config: %w", err)
}
}
provider, err := e.storageRegistry.Create(ctx, target.Type, rawConfig)
if err != nil {
return fmt.Errorf("create provider: %w", err)
}
f, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("open artifact: %w", err)
}
defer f.Close()
meta := map[string]string{
"taskId": fmt.Sprintf("%d", taskID),
"recordId": fmt.Sprintf("%d", recordID),
}
return provider.Upload(ctx, objectKey, f, fileSize, meta)
}
// appendLog 追加日志到 Master 记录(尽力而为,失败不中断主流程)
func (e *Executor) appendLog(ctx context.Context, recordID uint, line string) {
_ = e.client.UpdateRecord(ctx, recordID, RecordUpdate{LogAppend: line})
}
// reportRecordFailure 上报失败状态
func (e *Executor) reportRecordFailure(ctx context.Context, recordID uint, msg string) {
_ = e.client.UpdateRecord(ctx, recordID, RecordUpdate{
Status: "failed",
ErrorMessage: msg,
LogAppend: fmt.Sprintf("[agent] 错误: %s\n", msg),
})
}
// buildBackupTaskSpec 把 AgentTaskSpec 转换为 backup.TaskSpec。
func buildBackupTaskSpec(spec *TaskSpec, startedAt time.Time, tempDir string) backup.TaskSpec {
var sourcePaths []string
if strings.TrimSpace(spec.SourcePaths) != "" {
for _, p := range strings.Split(spec.SourcePaths, "\n") {
if p = strings.TrimSpace(p); p != "" {
sourcePaths = append(sourcePaths, p)
}
}
}
var excludes []string
if strings.TrimSpace(spec.ExcludePatterns) != "" {
for _, p := range strings.Split(spec.ExcludePatterns, "\n") {
if p = strings.TrimSpace(p); p != "" {
excludes = append(excludes, p)
}
}
}
return backup.TaskSpec{
ID: spec.TaskID,
Name: spec.Name,
Type: spec.Type,
SourcePath: spec.SourcePath,
SourcePaths: sourcePaths,
ExcludePatterns: excludes,
Database: backup.DatabaseSpec{
Host: spec.DBHost,
Port: spec.DBPort,
User: spec.DBUser,
Password: spec.DBPassword,
Path: spec.DBPath,
Names: splitCommaOrNewline(spec.DBName),
},
Compression: spec.Compression,
Encrypt: spec.Encrypt,
StartedAt: startedAt,
TempDir: tempDir,
}
}
// recordLogger 把 runner 日志回传到 Master 记录。
// 实现 backup.LogWriter每条日志追加到 record.log_content。
type recordLogger struct {
ctx context.Context
client *MasterClient
recordID uint
}
func newRecordLogger(ctx context.Context, client *MasterClient, recordID uint) *recordLogger {
return &recordLogger{ctx: ctx, client: client, recordID: recordID}
}
func (l *recordLogger) WriteLine(message string) {
_ = l.client.UpdateRecord(l.ctx, l.recordID, RecordUpdate{LogAppend: message + "\n"})
}
// 辅助函数
func computeFileSHA256(path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
func splitCommaOrNewline(s string) []string {
var result []string
for _, part := range strings.FieldsFunc(s, func(r rune) bool {
return r == ',' || r == '\n' || r == ';'
}) {
if p := strings.TrimSpace(part); p != "" {
result = append(result, p)
}
}
return result
}

View File

@@ -0,0 +1,49 @@
package agent
import (
"fmt"
"os"
"path/filepath"
"sort"
)
// DirEntry Agent 返回给 Master 的目录项。
type DirEntry struct {
Name string `json:"name"`
Path string `json:"path"`
IsDir bool `json:"isDir"`
Size int64 `json:"size"`
}
// listLocalDir 列出 Agent 所在机器的指定路径。
func listLocalDir(path string) ([]DirEntry, error) {
cleaned := filepath.Clean(path)
if cleaned == "" {
cleaned = "/"
}
entries, err := os.ReadDir(cleaned)
if err != nil {
return nil, fmt.Errorf("read dir: %w", err)
}
result := make([]DirEntry, 0, len(entries))
for _, entry := range entries {
info, _ := entry.Info()
size := int64(0)
if info != nil && !entry.IsDir() {
size = info.Size()
}
result = append(result, DirEntry{
Name: entry.Name(),
Path: filepath.Join(cleaned, entry.Name()),
IsDir: entry.IsDir(),
Size: size,
})
}
sort.Slice(result, func(i, j int) bool {
if result[i].IsDir != result[j].IsDir {
return result[i].IsDir
}
return result[i].Name < result[j].Name
})
return result, nil
}

View File

@@ -0,0 +1,61 @@
package agent
import (
"os"
"path/filepath"
"testing"
)
func TestListLocalDir(t *testing.T) {
dir := t.TempDir()
_ = os.WriteFile(filepath.Join(dir, "a.txt"), []byte("hello"), 0644)
_ = os.Mkdir(filepath.Join(dir, "sub"), 0755)
_ = os.WriteFile(filepath.Join(dir, "b.txt"), []byte("world!"), 0644)
entries, err := listLocalDir(dir)
if err != nil {
t.Fatalf("list: %v", err)
}
if len(entries) != 3 {
t.Fatalf("expected 3 entries, got %d", len(entries))
}
// 目录排序靠前
if !entries[0].IsDir || entries[0].Name != "sub" {
t.Errorf("directories should sort first: %+v", entries)
}
// 文件大小正确
var a *DirEntry
for i := range entries {
if entries[i].Name == "a.txt" {
a = &entries[i]
break
}
}
if a == nil || a.Size != 5 {
t.Errorf("file size: %+v", a)
}
}
func TestSplitCommaOrNewline(t *testing.T) {
cases := []struct {
in string
out []string
}{
{"", nil},
{"a,b,c", []string{"a", "b", "c"}},
{"a\nb\nc", []string{"a", "b", "c"}},
{"a; b ,\nc\n", []string{"a", "b", "c"}},
}
for _, c := range cases {
got := splitCommaOrNewline(c.in)
if len(got) != len(c.out) {
t.Errorf("%q: got %v want %v", c.in, got, c.out)
continue
}
for i := range got {
if got[i] != c.out[i] {
t.Errorf("%q[%d]: %q vs %q", c.in, i, got[i], c.out[i])
}
}
}
}

View File

@@ -0,0 +1,12 @@
package agent
import "encoding/json"
// jsonUnmarshalMap 把 []byte 或 json.RawMessage 解为 map[string]any。
func jsonUnmarshalMap(data []byte, out *map[string]any) error {
if len(data) == 0 {
*out = map[string]any{}
return nil
}
return json.Unmarshal(data, out)
}

View File

@@ -20,14 +20,7 @@ import (
"backupx/server/internal/service"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
"backupx/server/internal/storage/googledrive"
"backupx/server/internal/storage/localdisk"
storageAliyun "backupx/server/internal/storage/aliyun"
storageFTP "backupx/server/internal/storage/ftp"
storageTencent "backupx/server/internal/storage/tencent"
storageQiniu "backupx/server/internal/storage/qiniu"
storageS3 "backupx/server/internal/storage/s3"
storageWebDAV "backupx/server/internal/storage/webdav"
storageRclone "backupx/server/internal/storage/rclone"
"go.uber.org/zap"
"gorm.io/gorm"
)
@@ -70,37 +63,70 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
systemService := service.NewSystemService(cfg, version, time.Now().UTC())
configCipher := codec.NewConfigCipher(resolvedSecurity.EncryptionKey)
storageRegistry := storage.NewRegistry(
localdisk.NewFactory(),
storageS3.NewFactory(),
storageWebDAV.NewFactory(),
googledrive.NewFactory(),
storageAliyun.NewFactory(),
storageTencent.NewFactory(),
storageQiniu.NewFactory(),
storageFTP.NewFactory(),
storageRclone.NewLocalDiskFactory(),
storageRclone.NewS3Factory(),
storageRclone.NewWebDAVFactory(),
storageRclone.NewGoogleDriveFactory(),
storageRclone.NewAliyunOSSFactory(),
storageRclone.NewTencentCOSFactory(),
storageRclone.NewQiniuKodoFactory(),
storageRclone.NewFTPFactory(),
storageRclone.NewRcloneFactory(),
)
// 将全部 rclone 后端注册为独立存储类型sftp、azureblob、dropbox 等与 s3、ftp 完全平级)
storageRclone.RegisterAllBackends(storageRegistry)
storageTargetService := service.NewStorageTargetService(storageTargetRepo, oauthSessionRepo, storageRegistry, configCipher)
storageTargetService.SetBackupTaskRepository(backupTaskRepo)
storageTargetService.SetBackupRecordRepository(backupRecordRepo)
backupTaskService := service.NewBackupTaskService(backupTaskRepo, storageTargetRepo, configCipher)
backupTaskService.SetRecordsAndStorage(backupRecordRepo, storageRegistry)
backupRunnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewSQLiteRunner(), backup.NewMySQLRunner(nil), backup.NewPostgreSQLRunner(nil), backup.NewSAPHANARunner(nil))
logHub := backup.NewLogHub()
retentionService := backupretention.NewService(backupRecordRepo)
notifyRegistry := notify.NewRegistry(notify.NewEmailNotifier(), notify.NewWebhookNotifier(), notify.NewTelegramNotifier())
notificationService := service.NewNotificationService(notificationRepo, notifyRegistry, configCipher)
backupExecutionService := service.NewBackupExecutionService(backupTaskRepo, backupRecordRepo, storageTargetRepo, storageRegistry, backupRunnerRegistry, logHub, retentionService, configCipher, notificationService, cfg.Backup.TempDir, cfg.Backup.MaxConcurrent)
// 初始化 rclone 传输配置(重试 + 带宽限制)
rcloneCtx := storageRclone.ConfiguredContext(ctx, storageRclone.TransferConfig{
LowLevelRetries: cfg.Backup.Retries,
BandwidthLimit: cfg.Backup.BandwidthLimit,
})
storageRclone.StartAccounting(rcloneCtx)
backupExecutionService := service.NewBackupExecutionService(backupTaskRepo, backupRecordRepo, storageTargetRepo, storageRegistry, backupRunnerRegistry, logHub, retentionService, configCipher, notificationService, cfg.Backup.TempDir, cfg.Backup.MaxConcurrent, cfg.Backup.Retries, cfg.Backup.BandwidthLimit)
schedulerService := scheduler.NewService(backupTaskRepo, backupExecutionService, appLogger)
backupTaskService.SetScheduler(schedulerService)
// 审计日志注入延迟到 auditService 创建后(见下方)
backupRecordService := service.NewBackupRecordService(backupRecordRepo, backupExecutionService, logHub)
dashboardService := service.NewDashboardService(backupTaskRepo, backupRecordRepo, storageTargetRepo)
settingsService := service.NewSettingsService(systemConfigRepo)
// Audit
auditLogRepo := repository.NewAuditLogRepository(db)
auditService := service.NewAuditService(auditLogRepo)
authService.SetAuditService(auditService)
schedulerService.SetAuditRecorder(auditService)
// Database discovery
databaseDiscoveryService := service.NewDatabaseDiscoveryService(backup.NewOSCommandExecutor())
// Cluster: Node management
nodeRepo := repository.NewNodeRepository(db)
nodeService := service.NewNodeService(nodeRepo)
nodeService := service.NewNodeService(nodeRepo, version)
nodeService.SetTaskRepository(backupTaskRepo)
if err := nodeService.EnsureLocalNode(ctx); err != nil {
appLogger.Warn("failed to ensure local node", zap.Error(err))
}
// 启动离线检测:每 15s 扫描一次,超过 45s 未心跳的远程节点标记为离线
nodeService.StartOfflineMonitor(ctx, 15*time.Second)
// Agent 协议服务:命令队列 + 任务下发 + 记录上报
agentCmdRepo := repository.NewAgentCommandRepository(db)
agentService := service.NewAgentService(nodeRepo, backupTaskRepo, backupRecordRepo, storageTargetRepo, agentCmdRepo, configCipher)
agentService.StartCommandTimeoutMonitor(ctx, 30*time.Second, 10*time.Minute)
// 把 Agent 下发能力注入到备份执行服务,实现多节点路由
backupExecutionService.SetClusterDependencies(nodeRepo, agentService)
// 启用远程目录浏览NodeService 通过 AgentService 做同步 RPC
nodeService.SetAgentRPC(agentService)
router := aphttp.NewRouter(aphttp.RouterDependencies{
Config: cfg,
@@ -115,8 +141,11 @@ func New(ctx context.Context, cfg config.Config, version string) (*Application,
NotificationService: notificationService,
DashboardService: dashboardService,
SettingsService: settingsService,
NodeService: nodeService,
JWTManager: jwtManager,
NodeService: nodeService,
AgentService: agentService,
DatabaseDiscoveryService: databaseDiscoveryService,
AuditService: auditService,
JWTManager: jwtManager,
UserRepository: userRepo,
SystemConfigRepo: systemConfigRepo,
})

View File

@@ -0,0 +1,360 @@
package backint
import (
"compress/gzip"
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"os"
"path"
"strings"
"time"
"backupx/server/internal/storage"
storageRclone "backupx/server/internal/storage/rclone"
)
// Agent 是 Backint 协议代理主入口。
//
// 职责:
// 1. 根据 -f 指定的功能,从 -i 输入文件解析请求
// 2. 把数据路由到 BackupX storage 后端
// 3. 把结果写回 -o 输出文件(失败使用 #ERROR不中断批次
type Agent struct {
cfg *Config
provider storage.StorageProvider
catalog *Catalog
}
// NewAgent 构造 Agent初始化 storage provider 与 catalog。
func NewAgent(ctx context.Context, cfg *Config) (*Agent, error) {
registry := buildStorageRegistry()
provider, err := registry.Create(ctx, cfg.StorageType, cfg.StorageConfig)
if err != nil {
return nil, fmt.Errorf("create storage provider: %w", err)
}
if err := provider.TestConnection(ctx); err != nil {
return nil, fmt.Errorf("storage provider connection failed: %w", err)
}
cat, err := OpenCatalog(cfg.CatalogDB)
if err != nil {
return nil, err
}
return &Agent{cfg: cfg, provider: provider, catalog: cat}, nil
}
// Close 释放资源。
func (a *Agent) Close() error {
if a.catalog != nil {
return a.catalog.Close()
}
return nil
}
// Run 执行一次 Backint 调用。
//
// HANA 针对 BACKUP 调用时input 是 #PIPE 列表output 需返回 #SAVED 或 #ERROR。
// 批次中任一条目失败不应导致整个进程退出,因此错误被降级为 #ERROR 行。
// 仅在极端错误参数非法、I/O 失败)时返回 error进程以非 0 退出。
func (a *Agent) Run(ctx context.Context, fn Function, inputPath, outputPath string) error {
in, err := os.Open(inputPath)
if err != nil {
return fmt.Errorf("open input: %w", err)
}
defer in.Close()
out, err := os.Create(outputPath)
if err != nil {
return fmt.Errorf("create output: %w", err)
}
defer out.Close()
switch fn {
case FunctionBackup:
return a.runBackup(ctx, in, out)
case FunctionRestore:
return a.runRestore(ctx, in, out)
case FunctionInquire:
return a.runInquire(ctx, in, out)
case FunctionDelete:
return a.runDelete(ctx, in, out)
default:
return fmt.Errorf("unsupported function: %s", fn)
}
}
// runBackup 处理 BACKUP 操作:读取每条请求的管道/文件,上传到存储后端。
func (a *Agent) runBackup(ctx context.Context, in io.Reader, out io.Writer) error {
reqs, err := ParseBackupRequests(in)
if err != nil {
return err
}
for _, req := range reqs {
ebid, perr := a.handleBackupOne(ctx, req)
if perr != nil {
fmt.Fprintf(os.Stderr, "backint: backup %q failed: %v\n", req.Path, perr)
_ = WriteError(out, req.Path)
continue
}
_ = WriteSaved(out, ebid, req.Path)
}
return nil
}
// handleBackupOne 上传一条请求,返回分配的 EBID。
func (a *Agent) handleBackupOne(ctx context.Context, req BackupRequest) (string, error) {
src, size, err := openBackupSource(req)
if err != nil {
return "", err
}
defer src.Close()
ebid := generateEBID()
objectKey := a.objectKeyFor(ebid)
reader := io.Reader(src)
// 可选 gzip 压缩
if a.cfg.Compress {
pr, pw := io.Pipe()
go func() {
gw := gzip.NewWriter(pw)
if _, cerr := io.Copy(gw, src); cerr != nil {
_ = gw.Close()
_ = pw.CloseWithError(cerr)
return
}
if cerr := gw.Close(); cerr != nil {
_ = pw.CloseWithError(cerr)
return
}
_ = pw.Close()
}()
reader = pr
size = -1 // 压缩后大小未知
objectKey += ".gz"
}
meta := map[string]string{
"source-path": req.Path,
"ebid": ebid,
"compress": boolStr(a.cfg.Compress),
}
if err := a.provider.Upload(ctx, objectKey, reader, size, meta); err != nil {
return "", fmt.Errorf("upload: %w", err)
}
if err := a.catalog.Put(CatalogEntry{
EBID: ebid,
ObjectKey: objectKey,
SourcePath: req.Path,
Size: size,
}); err != nil {
return "", fmt.Errorf("catalog put: %w", err)
}
return ebid, nil
}
// runRestore 处理 RESTORE 操作:根据 EBID 从存储下载,写入 HANA 指定的管道/文件。
func (a *Agent) runRestore(ctx context.Context, in io.Reader, out io.Writer) error {
reqs, err := ParseRestoreRequests(in)
if err != nil {
return err
}
for _, req := range reqs {
if perr := a.handleRestoreOne(ctx, req); perr != nil {
fmt.Fprintf(os.Stderr, "backint: restore %q failed: %v\n", req.EBID, perr)
_ = WriteError(out, req.Path)
continue
}
_ = WriteRestored(out, req.EBID, req.Path)
}
return nil
}
func (a *Agent) handleRestoreOne(ctx context.Context, req RestoreRequest) error {
entry, err := a.catalog.Get(req.EBID)
if err != nil {
return fmt.Errorf("catalog get: %w", err)
}
if entry == nil {
return fmt.Errorf("ebid not found: %s", req.EBID)
}
rc, err := a.provider.Download(ctx, entry.ObjectKey)
if err != nil {
return fmt.Errorf("download: %w", err)
}
defer rc.Close()
var src io.Reader = rc
if strings.HasSuffix(entry.ObjectKey, ".gz") {
gr, err := gzip.NewReader(rc)
if err != nil {
return fmt.Errorf("gzip reader: %w", err)
}
defer gr.Close()
src = gr
}
dst, err := openRestoreTarget(req)
if err != nil {
return err
}
defer dst.Close()
if _, err := io.Copy(dst, src); err != nil {
return fmt.Errorf("copy to target: %w", err)
}
return nil
}
// runInquire 处理 INQUIRE 操作:查询 EBID 是否存在,或列出全部备份。
func (a *Agent) runInquire(ctx context.Context, in io.Reader, out io.Writer) error {
reqs, err := ParseInquireRequests(in)
if err != nil {
return err
}
for _, req := range reqs {
if req.All {
entries, err := a.catalog.List()
if err != nil {
fmt.Fprintf(os.Stderr, "backint: inquire list failed: %v\n", err)
_ = WriteError(out, "#NULL")
continue
}
for _, e := range entries {
_ = WriteBackup(out, e.EBID)
}
continue
}
entry, err := a.catalog.Get(req.EBID)
if err != nil {
fmt.Fprintf(os.Stderr, "backint: inquire %q failed: %v\n", req.EBID, err)
_ = WriteError(out, req.EBID)
continue
}
if entry == nil {
_ = WriteNotFound(out, req.EBID)
continue
}
_ = WriteBackup(out, entry.EBID)
}
return nil
}
// runDelete 处理 DELETE 操作:从存储删除对象并移除目录条目。
func (a *Agent) runDelete(ctx context.Context, in io.Reader, out io.Writer) error {
reqs, err := ParseDeleteRequests(in)
if err != nil {
return err
}
for _, req := range reqs {
if perr := a.handleDeleteOne(ctx, req); perr != nil {
fmt.Fprintf(os.Stderr, "backint: delete %q failed: %v\n", req.EBID, perr)
_ = WriteError(out, req.EBID)
continue
}
_ = WriteDeleted(out, req.EBID)
}
return nil
}
func (a *Agent) handleDeleteOne(ctx context.Context, req DeleteRequest) error {
entry, err := a.catalog.Get(req.EBID)
if err != nil {
return fmt.Errorf("catalog get: %w", err)
}
if entry == nil {
return fmt.Errorf("ebid not found: %s", req.EBID)
}
if err := a.provider.Delete(ctx, entry.ObjectKey); err != nil {
// 允许后端返回"不存在"类错误后继续删除目录条目,避免孤立条目
fmt.Fprintf(os.Stderr, "backint: storage delete warning for %s: %v\n", entry.ObjectKey, err)
}
return a.catalog.Delete(req.EBID)
}
// 辅助函数
func (a *Agent) objectKeyFor(ebid string) string {
base := ebid + ".bin"
if a.cfg.KeyPrefix == "" {
return base
}
return path.Join(a.cfg.KeyPrefix, base)
}
// openBackupSource 打开 HANA 提供的数据源。
//
// 对于 #PIPE 模式HANA 写入命名管道Agent 读取。管道是顺序流size 未知 (-1)。
// 对于文件模式HANA 已在指定路径写好完整文件。
func openBackupSource(req BackupRequest) (io.ReadCloser, int64, error) {
if req.IsPipe {
f, err := os.OpenFile(req.Path, os.O_RDONLY, 0)
if err != nil {
return nil, 0, fmt.Errorf("open pipe: %w", err)
}
return f, -1, nil
}
f, err := os.Open(req.Path)
if err != nil {
return nil, 0, fmt.Errorf("open file: %w", err)
}
info, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, 0, fmt.Errorf("stat: %w", err)
}
return f, info.Size(), nil
}
// openRestoreTarget 打开 HANA 指定的恢复目标(管道或文件)。
func openRestoreTarget(req RestoreRequest) (io.WriteCloser, error) {
if req.IsPipe {
return os.OpenFile(req.Path, os.O_WRONLY, 0)
}
return os.Create(req.Path)
}
// generateEBID 生成 Backint 外部备份 ID。
// 格式backupx-<timestamp>-<16 hex chars>
func generateEBID() string {
var buf [8]byte
if _, err := rand.Read(buf[:]); err != nil {
// fallback用纳秒时间戳作为熵
now := time.Now().UnixNano()
for i := 0; i < 8; i++ {
buf[i] = byte(now >> (i * 8))
}
}
return fmt.Sprintf("backupx-%d-%s", time.Now().Unix(), hex.EncodeToString(buf[:]))
}
func boolStr(b bool) string {
if b {
return "true"
}
return "false"
}
// buildStorageRegistry 构造与主程序一致的 storage registry。
//
// Backint Agent 作为独立 CLI 进程运行,不依赖 BackupX HTTP 服务,
// 因此这里直接引用 storage/rclone 包注册所有后端。
func buildStorageRegistry() *storage.Registry {
registry := storage.NewRegistry(
storageRclone.NewLocalDiskFactory(),
storageRclone.NewS3Factory(),
storageRclone.NewWebDAVFactory(),
storageRclone.NewGoogleDriveFactory(),
storageRclone.NewAliyunOSSFactory(),
storageRclone.NewTencentCOSFactory(),
storageRclone.NewQiniuKodoFactory(),
storageRclone.NewFTPFactory(),
storageRclone.NewRcloneFactory(),
)
storageRclone.RegisterAllBackends(registry)
return registry
}

View File

@@ -0,0 +1,217 @@
package backint
import (
"bytes"
"context"
"os"
"path/filepath"
"strings"
"testing"
"backupx/server/internal/storage"
storageRclone "backupx/server/internal/storage/rclone"
)
// newTestAgent 构造一个使用本地磁盘后端的 Agent便于集成测试。
func newTestAgent(t *testing.T, compress bool) (*Agent, string) {
t.Helper()
dir := t.TempDir()
storageDir := filepath.Join(dir, "storage")
if err := os.MkdirAll(storageDir, 0755); err != nil {
t.Fatal(err)
}
registry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
provider, err := registry.Create(context.Background(), "local_disk", map[string]any{
"basePath": storageDir,
})
if err != nil {
t.Fatalf("create provider: %v", err)
}
cat, err := OpenCatalog(filepath.Join(dir, "catalog.db"))
if err != nil {
t.Fatal(err)
}
agent := &Agent{
cfg: &Config{StorageType: "local_disk", KeyPrefix: "backint", Compress: compress, CatalogDB: filepath.Join(dir, "catalog.db")},
provider: provider,
catalog: cat,
}
t.Cleanup(func() { _ = agent.Close() })
return agent, dir
}
func TestAgent_BackupAndRestore_File(t *testing.T) {
agent, dir := newTestAgent(t, false)
ctx := context.Background()
// 准备源文件
src := filepath.Join(dir, "src.bak")
content := []byte("hello backint world")
if err := os.WriteFile(src, content, 0644); err != nil {
t.Fatal(err)
}
// BACKUP
inPath := filepath.Join(dir, "backup.in")
outPath := filepath.Join(dir, "backup.out")
if err := os.WriteFile(inPath, []byte(src+"\n"), 0644); err != nil {
t.Fatal(err)
}
if err := agent.Run(ctx, FunctionBackup, inPath, outPath); err != nil {
t.Fatalf("backup: %v", err)
}
out, _ := os.ReadFile(outPath)
if !bytes.HasPrefix(out, []byte("#SAVED ")) {
t.Fatalf("expected #SAVED, got: %s", out)
}
// 提取 EBID#SAVED <ebid> "<path>"
parts := strings.Fields(string(out))
if len(parts) < 3 {
t.Fatalf("malformed output: %s", out)
}
ebid := parts[1]
// RESTORE
restoreDst := filepath.Join(dir, "restored.bak")
inPath2 := filepath.Join(dir, "restore.in")
outPath2 := filepath.Join(dir, "restore.out")
if err := os.WriteFile(inPath2, []byte(ebid+" \""+restoreDst+"\"\n"), 0644); err != nil {
t.Fatal(err)
}
if err := agent.Run(ctx, FunctionRestore, inPath2, outPath2); err != nil {
t.Fatalf("restore: %v", err)
}
got, err := os.ReadFile(restoreDst)
if err != nil {
t.Fatalf("read restored: %v", err)
}
if !bytes.Equal(got, content) {
t.Errorf("restored content mismatch: %q vs %q", got, content)
}
}
func TestAgent_BackupWithCompression(t *testing.T) {
agent, dir := newTestAgent(t, true)
ctx := context.Background()
src := filepath.Join(dir, "src.bak")
content := bytes.Repeat([]byte("ABCDEFGH"), 1024)
if err := os.WriteFile(src, content, 0644); err != nil {
t.Fatal(err)
}
inPath := filepath.Join(dir, "backup.in")
outPath := filepath.Join(dir, "backup.out")
_ = os.WriteFile(inPath, []byte(src+"\n"), 0644)
if err := agent.Run(ctx, FunctionBackup, inPath, outPath); err != nil {
t.Fatalf("backup: %v", err)
}
parts := strings.Fields(string(mustRead(t, outPath)))
ebid := parts[1]
// 验证 catalog 记录的对象键以 .gz 结尾
entry, _ := agent.catalog.Get(ebid)
if entry == nil || !strings.HasSuffix(entry.ObjectKey, ".gz") {
t.Fatalf("expected .gz suffix: %+v", entry)
}
// RESTORE 应能解压回原始内容
dst := filepath.Join(dir, "restored.bak")
in2 := filepath.Join(dir, "restore.in")
out2 := filepath.Join(dir, "restore.out")
_ = os.WriteFile(in2, []byte(ebid+" \""+dst+"\"\n"), 0644)
if err := agent.Run(ctx, FunctionRestore, in2, out2); err != nil {
t.Fatalf("restore: %v", err)
}
got := mustRead(t, dst)
if !bytes.Equal(got, content) {
t.Errorf("decompressed content mismatch (len=%d vs %d)", len(got), len(content))
}
}
func TestAgent_Inquire(t *testing.T) {
agent, dir := newTestAgent(t, false)
ctx := context.Background()
// 注入两条目录记录
_ = agent.catalog.Put(CatalogEntry{EBID: "bid-a", ObjectKey: "k/a"})
_ = agent.catalog.Put(CatalogEntry{EBID: "bid-b", ObjectKey: "k/b"})
// INQUIRE #NULL 应列出全部
in := filepath.Join(dir, "inq.in")
out := filepath.Join(dir, "inq.out")
_ = os.WriteFile(in, []byte("#NULL\n"), 0644)
if err := agent.Run(ctx, FunctionInquire, in, out); err != nil {
t.Fatalf("inquire: %v", err)
}
text := string(mustRead(t, out))
if !strings.Contains(text, "bid-a") || !strings.Contains(text, "bid-b") {
t.Errorf("expected both ebids, got: %s", text)
}
// INQUIRE 不存在的 ebid → #NOTFOUND
_ = os.WriteFile(in, []byte("bid-missing\n"), 0644)
if err := agent.Run(ctx, FunctionInquire, in, out); err != nil {
t.Fatalf("inquire missing: %v", err)
}
text = string(mustRead(t, out))
if !strings.Contains(text, "#NOTFOUND") {
t.Errorf("expected #NOTFOUND, got: %s", text)
}
}
func TestAgent_Delete(t *testing.T) {
agent, dir := newTestAgent(t, false)
ctx := context.Background()
// 先做一次 BACKUP
src := filepath.Join(dir, "src.bak")
_ = os.WriteFile(src, []byte("data"), 0644)
inPath := filepath.Join(dir, "b.in")
outPath := filepath.Join(dir, "b.out")
_ = os.WriteFile(inPath, []byte(src+"\n"), 0644)
if err := agent.Run(ctx, FunctionBackup, inPath, outPath); err != nil {
t.Fatal(err)
}
ebid := strings.Fields(string(mustRead(t, outPath)))[1]
// DELETE
delIn := filepath.Join(dir, "d.in")
delOut := filepath.Join(dir, "d.out")
_ = os.WriteFile(delIn, []byte(ebid+"\n"), 0644)
if err := agent.Run(ctx, FunctionDelete, delIn, delOut); err != nil {
t.Fatalf("delete: %v", err)
}
if !strings.Contains(string(mustRead(t, delOut)), "#DELETED") {
t.Errorf("expected #DELETED, got: %s", mustRead(t, delOut))
}
// catalog 条目应已删除
if entry, _ := agent.catalog.Get(ebid); entry != nil {
t.Errorf("catalog entry should be removed, got: %+v", entry)
}
}
func TestAgent_RestoreUnknownEBID(t *testing.T) {
agent, dir := newTestAgent(t, false)
ctx := context.Background()
in := filepath.Join(dir, "r.in")
out := filepath.Join(dir, "r.out")
_ = os.WriteFile(in, []byte("bid-unknown \""+filepath.Join(dir, "dst")+"\"\n"), 0644)
if err := agent.Run(ctx, FunctionRestore, in, out); err != nil {
t.Fatalf("run: %v", err)
}
if !strings.Contains(string(mustRead(t, out)), "#ERROR") {
t.Errorf("expected #ERROR for unknown ebid, got: %s", mustRead(t, out))
}
}
func mustRead(t *testing.T, path string) []byte {
t.Helper()
b, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return b
}

View File

@@ -0,0 +1,102 @@
package backint
import (
"fmt"
"time"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/clause"
gormlogger "gorm.io/gorm/logger"
)
// CatalogEntry 是 Backint 目录条目,建立 BID (备份 ID) 与对象键的映射。
//
// BID 是 Backint Agent 返回给 SAP HANA 的唯一标识HANA 后续用它作为 RESTORE/DELETE
// 的句柄。Agent 用 catalog 查询该 BID 对应的实际存储对象键。
type CatalogEntry struct {
ID uint `gorm:"primaryKey"`
EBID string `gorm:"column:ebid;uniqueIndex;size:128;not null"`
ObjectKey string `gorm:"column:object_key;size:512;not null"`
SourcePath string `gorm:"column:source_path;size:1024"`
Size int64 `gorm:"column:size"`
CreatedAt time.Time `gorm:"column:created_at"`
}
// TableName 指定表名,避免 GORM 自动复数化。
func (CatalogEntry) TableName() string { return "backint_catalog" }
// Catalog 是本地 Backint 目录SQLite 后端)。
type Catalog struct {
db *gorm.DB
}
// OpenCatalog 打开或创建 catalog 数据库。
func OpenCatalog(dbPath string) (*Catalog, error) {
db, err := gorm.Open(sqlite.Open(dbPath), &gorm.Config{
Logger: gormlogger.Default.LogMode(gormlogger.Silent),
})
if err != nil {
return nil, fmt.Errorf("open catalog: %w", err)
}
if err := db.AutoMigrate(&CatalogEntry{}); err != nil {
return nil, fmt.Errorf("migrate catalog: %w", err)
}
return &Catalog{db: db}, nil
}
// Close 关闭底层连接。
func (c *Catalog) Close() error {
if c.db == nil {
return nil
}
sqlDB, err := c.db.DB()
if err != nil {
return err
}
return sqlDB.Close()
}
// Put 插入或更新一条记录。
func (c *Catalog) Put(entry CatalogEntry) error {
if entry.EBID == "" {
return fmt.Errorf("ebid is required")
}
if entry.CreatedAt.IsZero() {
entry.CreatedAt = time.Now().UTC()
}
// UpsertEBID 冲突时更新 object_key/size/source_path
return c.db.Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "ebid"}},
DoUpdates: clause.AssignmentColumns([]string{
"object_key", "source_path", "size", "created_at",
}),
}).Create(&entry).Error
}
// Get 通过 EBID 查询条目。未找到返回 (nil, nil)。
func (c *Catalog) Get(ebid string) (*CatalogEntry, error) {
var entry CatalogEntry
err := c.db.Where("ebid = ?", ebid).First(&entry).Error
if err == gorm.ErrRecordNotFound {
return nil, nil
}
if err != nil {
return nil, err
}
return &entry, nil
}
// Delete 删除一条记录。
func (c *Catalog) Delete(ebid string) error {
return c.db.Where("ebid = ?", ebid).Delete(&CatalogEntry{}).Error
}
// List 列出全部条目。
func (c *Catalog) List() ([]CatalogEntry, error) {
var entries []CatalogEntry
if err := c.db.Order("created_at DESC").Find(&entries).Error; err != nil {
return nil, err
}
return entries, nil
}

View File

@@ -0,0 +1,74 @@
package backint
import (
"path/filepath"
"testing"
)
func TestCatalog_CRUD(t *testing.T) {
dir := t.TempDir()
cat, err := OpenCatalog(filepath.Join(dir, "test.db"))
if err != nil {
t.Fatalf("open: %v", err)
}
defer cat.Close()
if err := cat.Put(CatalogEntry{EBID: "bid-1", ObjectKey: "k/1.bin", SourcePath: "/tmp/a", Size: 100}); err != nil {
t.Fatalf("put: %v", err)
}
if err := cat.Put(CatalogEntry{EBID: "bid-2", ObjectKey: "k/2.bin", Size: 200}); err != nil {
t.Fatalf("put: %v", err)
}
got, err := cat.Get("bid-1")
if err != nil || got == nil {
t.Fatalf("get: %v %v", got, err)
}
if got.ObjectKey != "k/1.bin" || got.Size != 100 {
t.Errorf("mismatch: %+v", got)
}
// 不存在的条目
missing, err := cat.Get("bid-999")
if err != nil {
t.Fatalf("get missing: %v", err)
}
if missing != nil {
t.Errorf("expected nil, got %+v", missing)
}
// List
all, err := cat.List()
if err != nil || len(all) != 2 {
t.Fatalf("list: %v %d", err, len(all))
}
// Delete
if err := cat.Delete("bid-1"); err != nil {
t.Fatalf("delete: %v", err)
}
got, _ = cat.Get("bid-1")
if got != nil {
t.Errorf("bid-1 should be deleted")
}
}
func TestCatalog_UpsertSameEBID(t *testing.T) {
dir := t.TempDir()
cat, err := OpenCatalog(filepath.Join(dir, "test.db"))
if err != nil {
t.Fatalf("open: %v", err)
}
defer cat.Close()
if err := cat.Put(CatalogEntry{EBID: "bid-x", ObjectKey: "v1"}); err != nil {
t.Fatal(err)
}
if err := cat.Put(CatalogEntry{EBID: "bid-x", ObjectKey: "v2"}); err != nil {
t.Fatal(err)
}
got, _ := cat.Get("bid-x")
if got == nil || got.ObjectKey != "v2" {
t.Errorf("upsert failed: %+v", got)
}
}

View File

@@ -0,0 +1,140 @@
package backint
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"strconv"
"strings"
)
// Config 是 Backint Agent 的运行时配置。
//
// SAP HANA 通过 -p <paramfile> 传入一个参数文件。BackupX Backint Agent 复用 SAP
// 的"#KEY = VALUE"风格(兼容原生 backint 参数文件习惯),不支持 section。
//
// 必填字段:
// - STORAGE_TYPE存储类型s3/webdav/local_disk/...,与 BackupX storage registry 一致)
// - STORAGE_CONFIG_JSON存储配置 JSON 文件路径(或直接 STORAGE_CONFIG = <json>
//
// 可选字段:
// - PARALLEL_FACTOR并行度默认 1
// - COMPRESS是否 gzip 压缩true/false默认 false
// - LOG_FILE日志文件路径默认 stderr
// - CATALOG_DB本地目录数据库路径默认 ./backint_catalog.db
// - KEY_PREFIX对象键前缀默认空最终对象键 = <prefix>/<ebid>
type Config struct {
StorageType string
StorageConfigJSON string // 存储配置 JSON 文件路径
StorageConfigRaw []byte // 也支持直接内联STORAGE_CONFIG
StorageConfig map[string]any // 解析后的存储配置
ParallelFactor int
Compress bool
LogFile string
CatalogDB string
KeyPrefix string
}
// LoadConfigFile 从文件加载配置。
func LoadConfigFile(path string) (*Config, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("open backint config: %w", err)
}
defer f.Close()
return ParseConfig(f)
}
// ParseConfig 从 reader 解析配置。
func ParseConfig(r io.Reader) (*Config, error) {
cfg := &Config{ParallelFactor: 1}
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 4*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, ";") {
continue
}
// 兼容可选的 "#" 前缀SAP 约定)
line = strings.TrimPrefix(line, "#")
eq := strings.Index(line, "=")
if eq < 0 {
continue
}
key := strings.TrimSpace(line[:eq])
value := strings.TrimSpace(line[eq+1:])
if len(value) >= 2 && value[0] == '"' && value[len(value)-1] == '"' {
value = value[1 : len(value)-1]
}
switch strings.ToUpper(key) {
case "STORAGE_TYPE":
cfg.StorageType = value
case "STORAGE_CONFIG_JSON":
cfg.StorageConfigJSON = value
case "STORAGE_CONFIG":
cfg.StorageConfigRaw = []byte(value)
case "PARALLEL_FACTOR":
n, err := strconv.Atoi(value)
if err != nil || n <= 0 {
return nil, fmt.Errorf("invalid PARALLEL_FACTOR: %q", value)
}
cfg.ParallelFactor = n
case "COMPRESS":
cfg.Compress = parseBool(value)
case "LOG_FILE":
cfg.LogFile = value
case "CATALOG_DB":
cfg.CatalogDB = value
case "KEY_PREFIX":
cfg.KeyPrefix = strings.Trim(value, "/")
}
}
if err := scanner.Err(); err != nil {
return nil, err
}
if err := cfg.finalize(); err != nil {
return nil, err
}
return cfg, nil
}
func (c *Config) finalize() error {
if c.StorageType == "" {
return errors.New("STORAGE_TYPE is required")
}
if c.CatalogDB == "" {
c.CatalogDB = "./backint_catalog.db"
}
// 加载存储配置 JSON
var raw []byte
switch {
case c.StorageConfigJSON != "":
data, err := os.ReadFile(c.StorageConfigJSON)
if err != nil {
return fmt.Errorf("read STORAGE_CONFIG_JSON: %w", err)
}
raw = data
case len(c.StorageConfigRaw) > 0:
raw = c.StorageConfigRaw
default:
return errors.New("STORAGE_CONFIG_JSON or STORAGE_CONFIG is required")
}
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
return fmt.Errorf("parse storage config JSON: %w", err)
}
c.StorageConfig = m
return nil
}
func parseBool(v string) bool {
switch strings.ToLower(strings.TrimSpace(v)) {
case "1", "true", "yes", "on":
return true
default:
return false
}
}

View File

@@ -0,0 +1,74 @@
package backint
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestParseConfig(t *testing.T) {
dir := t.TempDir()
storagePath := filepath.Join(dir, "storage.json")
if err := os.WriteFile(storagePath, []byte(`{"basePath":"/tmp/backup"}`), 0644); err != nil {
t.Fatal(err)
}
input := `
; 注释
#STORAGE_TYPE = local_disk
#STORAGE_CONFIG_JSON = ` + storagePath + `
#PARALLEL_FACTOR = 4
#COMPRESS = true
#KEY_PREFIX = /hana/backups/
#CATALOG_DB = ` + filepath.Join(dir, "catalog.db") + `
`
cfg, err := ParseConfig(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if cfg.StorageType != "local_disk" {
t.Errorf("StorageType: %q", cfg.StorageType)
}
if cfg.ParallelFactor != 4 {
t.Errorf("ParallelFactor: %d", cfg.ParallelFactor)
}
if !cfg.Compress {
t.Errorf("Compress should be true")
}
if cfg.KeyPrefix != "hana/backups" {
t.Errorf("KeyPrefix should be trimmed: %q", cfg.KeyPrefix)
}
if cfg.StorageConfig["basePath"] != "/tmp/backup" {
t.Errorf("StorageConfig mismatch: %+v", cfg.StorageConfig)
}
}
func TestParseConfig_MissingStorageType(t *testing.T) {
input := `PARALLEL_FACTOR = 1`
if _, err := ParseConfig(strings.NewReader(input)); err == nil {
t.Fatal("expected error for missing STORAGE_TYPE")
}
}
func TestParseConfig_InlineStorageConfig(t *testing.T) {
input := `STORAGE_TYPE = local_disk
STORAGE_CONFIG = {"basePath":"/x"}
`
cfg, err := ParseConfig(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if cfg.StorageConfig["basePath"] != "/x" {
t.Errorf("inline config not parsed: %+v", cfg.StorageConfig)
}
}
func TestParseConfig_InvalidParallel(t *testing.T) {
input := `STORAGE_TYPE = local_disk
STORAGE_CONFIG = {}
PARALLEL_FACTOR = oops
`
if _, err := ParseConfig(strings.NewReader(input)); err == nil {
t.Fatal("expected error for invalid PARALLEL_FACTOR")
}
}

View File

@@ -0,0 +1,267 @@
// Package backint 实现 SAP HANA Backint 协议代理。
//
// Backint 协议是 SAP HANA 与第三方备份工具之间的管道/文件协议。
// SAP HANA 通过 CLI 调用 Backint Agent传入参数文件、输入文件、输出文件
// Agent 根据输入文件中的 #PIPE / #EBID / #NULL 指令读取/写入数据,
// 并在输出文件中返回 #SAVED / #RESTORED / #BACKUP / #NOTFOUND / #DELETED / #ERROR。
//
// 支持的功能BACKUP / RESTORE / INQUIRE / DELETE
// 参考规范SAP HANA Backint Interface for Backup Tools (OSS 1642148)
package backint
import (
"bufio"
"errors"
"fmt"
"io"
"strings"
)
// Function 代表 Backint 操作类型,对应 CLI 的 -f 参数。
type Function string
const (
FunctionBackup Function = "backup"
FunctionRestore Function = "restore"
FunctionInquire Function = "inquire"
FunctionDelete Function = "delete"
)
// BackupRequest 是 BACKUP 操作的单条请求。
//
// 两种形态:
// - Pipe: #PIPE <path> (HANA 通过命名管道传输数据)
// - File: "<path>" (HANA 指向一个已完成的临时文件)
type BackupRequest struct {
IsPipe bool
Path string
}
// RestoreRequest 是 RESTORE 操作的单条请求。
//
// 形态:#PIPE <ebid> "<path>" 或 <ebid> "<path>"
type RestoreRequest struct {
IsPipe bool
EBID string // 之前 BACKUP 返回的备份 ID
Path string
}
// InquireRequest 是 INQUIRE 操作的单条请求。
//
// 形态:
// - #NULL (列出所有备份)
// - "<ebid>" (查询指定 ID 是否存在)
// - #EBID "<ebid>" (带前缀的变体)
type InquireRequest struct {
All bool
EBID string
}
// DeleteRequest 是 DELETE 操作的单条请求。
//
// 形态:<ebid> 或 #EBID <ebid>
type DeleteRequest struct {
EBID string
}
// ParseBackupRequests 解析 BACKUP 输入文件。
func ParseBackupRequests(r io.Reader) ([]BackupRequest, error) {
var items []BackupRequest
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 4*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
if strings.HasPrefix(line, "#PIPE") {
path := strings.TrimSpace(strings.TrimPrefix(line, "#PIPE"))
if path == "" {
return nil, fmt.Errorf("invalid #PIPE line: %q", line)
}
items = append(items, BackupRequest{IsPipe: true, Path: trimQuotes(path)})
continue
}
items = append(items, BackupRequest{IsPipe: false, Path: trimQuotes(line)})
}
if err := scanner.Err(); err != nil {
return nil, err
}
return items, nil
}
// ParseRestoreRequests 解析 RESTORE 输入文件。
func ParseRestoreRequests(r io.Reader) ([]RestoreRequest, error) {
var items []RestoreRequest
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 4*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
isPipe := false
if strings.HasPrefix(line, "#PIPE") {
isPipe = true
line = strings.TrimSpace(strings.TrimPrefix(line, "#PIPE"))
}
if strings.HasPrefix(line, "#EBID") {
line = strings.TrimSpace(strings.TrimPrefix(line, "#EBID"))
}
ebid, rest := splitFirstField(line)
if ebid == "" || rest == "" {
return nil, fmt.Errorf("invalid restore line: %q", line)
}
items = append(items, RestoreRequest{
IsPipe: isPipe,
EBID: trimQuotes(ebid),
Path: trimQuotes(rest),
})
}
if err := scanner.Err(); err != nil {
return nil, err
}
return items, nil
}
// ParseInquireRequests 解析 INQUIRE 输入文件。
func ParseInquireRequests(r io.Reader) ([]InquireRequest, error) {
var items []InquireRequest
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 4*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
if line == "#NULL" {
items = append(items, InquireRequest{All: true})
continue
}
if strings.HasPrefix(line, "#EBID") {
line = strings.TrimSpace(strings.TrimPrefix(line, "#EBID"))
}
items = append(items, InquireRequest{EBID: trimQuotes(line)})
}
if err := scanner.Err(); err != nil {
return nil, err
}
return items, nil
}
// ParseDeleteRequests 解析 DELETE 输入文件。
func ParseDeleteRequests(r io.Reader) ([]DeleteRequest, error) {
var items []DeleteRequest
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 4*1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
if strings.HasPrefix(line, "#EBID") {
line = strings.TrimSpace(strings.TrimPrefix(line, "#EBID"))
}
ebid := trimQuotes(strings.TrimSpace(line))
if ebid == "" {
return nil, fmt.Errorf("invalid delete line: %q", line)
}
items = append(items, DeleteRequest{EBID: ebid})
}
if err := scanner.Err(); err != nil {
return nil, err
}
return items, nil
}
// 输出写入辅助
// WriteSaved 写入一条 BACKUP 成功响应:#SAVED <ebid> "<path>"
func WriteSaved(w io.Writer, ebid, path string) error {
_, err := fmt.Fprintf(w, "#SAVED %s %s\n", ebid, quote(path))
return err
}
// WriteRestored 写入一条 RESTORE 成功响应:#RESTORED "<ebid>" "<path>"
func WriteRestored(w io.Writer, ebid, path string) error {
_, err := fmt.Fprintf(w, "#RESTORED %s %s\n", quote(ebid), quote(path))
return err
}
// WriteBackup 写入一条 INQUIRE 命中响应:#BACKUP "<ebid>"
func WriteBackup(w io.Writer, ebid string) error {
_, err := fmt.Fprintf(w, "#BACKUP %s\n", quote(ebid))
return err
}
// WriteNotFound 写入一条 INQUIRE/RESTORE 未命中响应:#NOTFOUND "<path-or-ebid>"
func WriteNotFound(w io.Writer, identifier string) error {
_, err := fmt.Fprintf(w, "#NOTFOUND %s\n", quote(identifier))
return err
}
// WriteDeleted 写入一条 DELETE 成功响应:#DELETED "<ebid>"
func WriteDeleted(w io.Writer, ebid string) error {
_, err := fmt.Fprintf(w, "#DELETED %s\n", quote(ebid))
return err
}
// WriteError 写入一条错误响应:#ERROR "<path-or-ebid>"
//
// SAP HANA 会将 #ERROR 视为本条请求失败,但不会终止整个批次。
// 在 stderr 输出错误详情便于排查。
func WriteError(w io.Writer, identifier string) error {
_, err := fmt.Fprintf(w, "#ERROR %s\n", quote(identifier))
return err
}
// 内部工具函数
func trimQuotes(s string) string {
s = strings.TrimSpace(s)
if len(s) >= 2 && s[0] == '"' && s[len(s)-1] == '"' {
return s[1 : len(s)-1]
}
return s
}
func quote(s string) string {
return `"` + strings.ReplaceAll(s, `"`, `\"`) + `"`
}
// splitFirstField 把一行拆分为 "第一个字段" 和 "剩余部分"。
// 支持带引号的字段:`"abc def" "path"` → `abc def` / `"path"`。
func splitFirstField(line string) (first, rest string) {
line = strings.TrimSpace(line)
if line == "" {
return "", ""
}
if line[0] == '"' {
idx := strings.Index(line[1:], `"`)
if idx < 0 {
return line, ""
}
return line[1 : idx+1], strings.TrimSpace(line[idx+2:])
}
idx := strings.IndexAny(line, " \t")
if idx < 0 {
return line, ""
}
return line[:idx], strings.TrimSpace(line[idx+1:])
}
// ParseFunction 将 CLI 的 -f 参数字符串规范化为 Function。
func ParseFunction(s string) (Function, error) {
switch strings.ToLower(strings.TrimSpace(s)) {
case "backup":
return FunctionBackup, nil
case "restore":
return FunctionRestore, nil
case "inquire":
return FunctionInquire, nil
case "delete":
return FunctionDelete, nil
default:
return "", errors.New("unsupported backint function: " + s)
}
}

View File

@@ -0,0 +1,142 @@
package backint
import (
"bytes"
"strings"
"testing"
)
func TestParseBackupRequests(t *testing.T) {
input := `#PIPE /tmp/pipe1
#PIPE "/tmp/pipe two"
/tmp/file.bak
"/tmp/file two.bak"
`
reqs, err := ParseBackupRequests(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(reqs) != 4 {
t.Fatalf("expected 4 requests, got %d", len(reqs))
}
if !reqs[0].IsPipe || reqs[0].Path != "/tmp/pipe1" {
t.Errorf("req[0] mismatch: %+v", reqs[0])
}
if !reqs[1].IsPipe || reqs[1].Path != "/tmp/pipe two" {
t.Errorf("req[1] mismatch: %+v", reqs[1])
}
if reqs[2].IsPipe || reqs[2].Path != "/tmp/file.bak" {
t.Errorf("req[2] mismatch: %+v", reqs[2])
}
if reqs[3].Path != "/tmp/file two.bak" {
t.Errorf("req[3] mismatch: %+v", reqs[3])
}
}
func TestParseRestoreRequests(t *testing.T) {
input := `#PIPE backupx-123 "/tmp/pipe1"
#EBID "backupx-456" "/tmp/file.bak"
backupx-789 /tmp/plain.bak
`
reqs, err := ParseRestoreRequests(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(reqs) != 3 {
t.Fatalf("expected 3, got %d", len(reqs))
}
if !reqs[0].IsPipe || reqs[0].EBID != "backupx-123" || reqs[0].Path != "/tmp/pipe1" {
t.Errorf("req[0] mismatch: %+v", reqs[0])
}
if reqs[1].IsPipe || reqs[1].EBID != "backupx-456" {
t.Errorf("req[1] mismatch: %+v", reqs[1])
}
if reqs[2].EBID != "backupx-789" || reqs[2].Path != "/tmp/plain.bak" {
t.Errorf("req[2] mismatch: %+v", reqs[2])
}
}
func TestParseInquireRequests(t *testing.T) {
input := "#NULL\nbackupx-abc\n#EBID \"backupx-xyz\"\n"
reqs, err := ParseInquireRequests(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(reqs) != 3 {
t.Fatalf("expected 3, got %d", len(reqs))
}
if !reqs[0].All {
t.Errorf("req[0] should be All")
}
if reqs[1].EBID != "backupx-abc" {
t.Errorf("req[1] mismatch: %+v", reqs[1])
}
if reqs[2].EBID != "backupx-xyz" {
t.Errorf("req[2] mismatch: %+v", reqs[2])
}
}
func TestParseDeleteRequests(t *testing.T) {
input := "backupx-aaa\n#EBID \"backupx-bbb\"\n"
reqs, err := ParseDeleteRequests(strings.NewReader(input))
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(reqs) != 2 || reqs[0].EBID != "backupx-aaa" || reqs[1].EBID != "backupx-bbb" {
t.Fatalf("unexpected: %+v", reqs)
}
}
func TestWriteResponses(t *testing.T) {
var buf bytes.Buffer
_ = WriteSaved(&buf, "backupx-1", "/tmp/x")
_ = WriteRestored(&buf, "backupx-2", "/tmp/y")
_ = WriteBackup(&buf, "backupx-3")
_ = WriteNotFound(&buf, "backupx-4")
_ = WriteDeleted(&buf, "backupx-5")
_ = WriteError(&buf, "/tmp/z")
want := "#SAVED backupx-1 \"/tmp/x\"\n" +
"#RESTORED \"backupx-2\" \"/tmp/y\"\n" +
"#BACKUP \"backupx-3\"\n" +
"#NOTFOUND \"backupx-4\"\n" +
"#DELETED \"backupx-5\"\n" +
"#ERROR \"/tmp/z\"\n"
if buf.String() != want {
t.Errorf("output mismatch:\n got: %q\nwant: %q", buf.String(), want)
}
}
func TestParseFunction(t *testing.T) {
cases := map[string]Function{
"backup": FunctionBackup,
"BACKUP": FunctionBackup,
"restore": FunctionRestore,
"inquire": FunctionInquire,
"delete": FunctionDelete,
}
for s, want := range cases {
got, err := ParseFunction(s)
if err != nil || got != want {
t.Errorf("ParseFunction(%q) = %v, %v; want %v", s, got, err, want)
}
}
if _, err := ParseFunction("bogus"); err == nil {
t.Errorf("expected error for bogus function")
}
}
func TestSplitFirstField(t *testing.T) {
cases := []struct{ in, first, rest string }{
{`abc def`, "abc", "def"},
{`"abc def" ghi`, "abc def", "ghi"},
{`"a b" "c d"`, "a b", `"c d"`},
{`lone`, "lone", ""},
{``, "", ""},
}
for _, c := range cases {
f, r := splitFirstField(c.in)
if f != c.first || r != c.rest {
t.Errorf("splitFirstField(%q) = (%q, %q); want (%q, %q)", c.in, f, r, c.first, c.rest)
}
}
}

View File

@@ -22,14 +22,23 @@ func (r *FileRunner) Type() string {
}
func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*RunResult, error) {
sourcePath := filepath.Clean(strings.TrimSpace(task.SourcePath))
if sourcePath == "" {
// 解析源路径列表:优先 SourcePaths回退 SourcePath
sourcePaths := task.SourcePaths
if len(sourcePaths) == 0 && strings.TrimSpace(task.SourcePath) != "" {
sourcePaths = []string{task.SourcePath}
}
if len(sourcePaths) == 0 {
return nil, fmt.Errorf("source path is required")
}
info, err := os.Stat(sourcePath)
if err != nil {
return nil, fmt.Errorf("stat source path: %w", err)
// 验证所有路径存在
for _, sp := range sourcePaths {
cleaned := filepath.Clean(strings.TrimSpace(sp))
if _, err := os.Stat(cleaned); err != nil {
return nil, fmt.Errorf("stat source path %s: %w", cleaned, err)
}
}
tempDir, artifactPath, err := createTempArtifact(task.TempDir, task.Name, "tar")
if err != nil {
return nil, err
@@ -41,69 +50,88 @@ func (r *FileRunner) Run(_ context.Context, task TaskSpec, writer LogWriter) (*R
defer artifactFile.Close()
tw := tar.NewWriter(artifactFile)
defer tw.Close()
baseParent := filepath.Dir(sourcePath)
excludes := normalizeExcludePatterns(task.ExcludePatterns)
writer.WriteLine(fmt.Sprintf("开始打包文件备份:%s", sourcePath))
fileCount := 0
dirCount := 0
walkErr := filepath.Walk(sourcePath, func(currentPath string, currentInfo os.FileInfo, walkErr error) error {
if walkErr != nil {
writer.WriteLine(fmt.Sprintf("⚠ 无法访问 %s: %v", currentPath, walkErr))
return nil
}
relPath, err := filepath.Rel(baseParent, currentPath)
totalFileCount := 0
totalDirCount := 0
for i, sp := range sourcePaths {
sourcePath := filepath.Clean(strings.TrimSpace(sp))
info, err := os.Stat(sourcePath)
if err != nil {
return err
return nil, fmt.Errorf("stat source path: %w", err)
}
archiveName := filepath.ToSlash(relPath)
if shouldExcludeEntry(archiveName, currentInfo.IsDir(), excludes) {
if currentInfo.IsDir() {
writer.WriteLine(fmt.Sprintf("跳过排除目录 %s", archiveName))
return filepath.SkipDir
baseParent := filepath.Dir(sourcePath)
writer.WriteLine(fmt.Sprintf("开始打包源路径 [%d/%d]: %s", i+1, len(sourcePaths), sourcePath))
fileCount := 0
dirCount := 0
walkErr := filepath.Walk(sourcePath, func(currentPath string, currentInfo os.FileInfo, walkErr error) error {
if walkErr != nil {
writer.WriteLine(fmt.Sprintf("⚠ 无法访问 %s: %v", currentPath, walkErr))
return nil
}
return nil
}
if currentPath == sourcePath && currentInfo.IsDir() {
return nil
}
if currentInfo.IsDir() {
dirCount++
writer.WriteLine(fmt.Sprintf("📁 进入目录 %s", archiveName))
}
header, err := tar.FileInfoHeader(currentInfo, "")
if err != nil {
return err
}
header.Name = archiveName
if err := tw.WriteHeader(header); err != nil {
return err
}
if currentInfo.Mode().IsRegular() {
file, err := os.Open(currentPath)
relPath, err := filepath.Rel(baseParent, currentPath)
if err != nil {
return err
}
defer file.Close()
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
archiveName := filepath.ToSlash(relPath)
if shouldExcludeEntry(archiveName, currentInfo.IsDir(), excludes) {
if currentInfo.IsDir() {
writer.WriteLine(fmt.Sprintf("跳过排除目录 %s", archiveName))
return filepath.SkipDir
}
return nil
}
if currentPath == sourcePath && currentInfo.IsDir() {
return nil
}
if currentInfo.IsDir() {
dirCount++
writer.WriteLine(fmt.Sprintf("📁 进入目录 %s", archiveName))
}
header, err := tar.FileInfoHeader(currentInfo, "")
if err != nil {
return err
}
fileCount++
if fileCount%100 == 0 {
writer.WriteLine(fmt.Sprintf("已打包 %d 个文件...", fileCount))
header.Name = archiveName
if err := tw.WriteHeader(header); err != nil {
return err
}
if currentInfo.Mode().IsRegular() {
file, err := os.Open(currentPath)
if err != nil {
return err
}
defer file.Close()
if _, err := io.CopyN(tw, file, currentInfo.Size()); err != nil && err != io.EOF {
return err
}
fileCount++
if fileCount%100 == 0 {
writer.WriteLine(fmt.Sprintf("已打包 %d 个文件...", fileCount))
}
}
return nil
})
if walkErr != nil {
return nil, fmt.Errorf("walk source path %s: %w", sourcePath, walkErr)
}
return nil
})
if walkErr != nil {
return nil, fmt.Errorf("walk source path: %w", walkErr)
if info.IsDir() {
writer.WriteLine(fmt.Sprintf("源路径 [%d/%d] 打包完成(%d 个目录,%d 个文件)", i+1, len(sourcePaths), dirCount, fileCount))
} else {
writer.WriteLine(fmt.Sprintf("源路径 [%d/%d] 文件打包完成", i+1, len(sourcePaths)))
}
totalFileCount += fileCount
totalDirCount += dirCount
}
if info.IsDir() {
writer.WriteLine(fmt.Sprintf("目录打包完成(%d 个目录,%d 个文件)", dirCount, fileCount))
} else {
writer.WriteLine("文件打包完成")
if len(sourcePaths) > 1 {
writer.WriteLine(fmt.Sprintf("全部源路径打包完成(共 %d 个目录,%d 个文件)", totalDirCount, totalFileCount))
}
return &RunResult{ArtifactPath: artifactPath, FileName: filepath.Base(artifactPath), TempDir: tempDir}, nil
}
@@ -114,7 +142,12 @@ func (r *FileRunner) Restore(_ context.Context, task TaskSpec, artifactPath stri
return fmt.Errorf("open tar artifact: %w", err)
}
defer artifactFile.Close()
targetParent := filepath.Dir(filepath.Clean(strings.TrimSpace(task.SourcePath)))
// 恢复目标:优先取 SourcePaths 的第一个路径的父目录,回退 SourcePath
restoreSource := task.SourcePath
if len(task.SourcePaths) > 0 {
restoreSource = task.SourcePaths[0]
}
targetParent := filepath.Dir(filepath.Clean(strings.TrimSpace(restoreSource)))
if err := os.MkdirAll(targetParent, 0o755); err != nil {
return fmt.Errorf("create restore parent: %w", err)
}

View File

@@ -1,6 +1,7 @@
package backup
import (
"fmt"
"sync"
"time"
)
@@ -99,6 +100,41 @@ func (h *LogHub) Complete(recordID uint, status string) {
}
}
// AppendProgress 推送上传进度事件(节流:每个 recordID 每 500ms 最多一次,最终值始终推送)。
func (h *LogHub) AppendProgress(recordID uint, progress ProgressInfo) {
h.mu.Lock()
defer h.mu.Unlock()
state := h.ensureState(recordID)
// 节流:距上次 progress 事件不足 500ms 且未完成则跳过100% 始终推送)
now := time.Now().UTC()
isFinal := progress.Percent >= 100
if !isFinal && len(state.events) > 0 {
last := state.events[len(state.events)-1]
if last.Progress != nil && now.Sub(last.Timestamp) < 500*time.Millisecond {
return
}
}
state.nextSequence++
event := LogEvent{
RecordID: recordID,
Sequence: state.nextSequence,
Level: "progress",
Message: fmt.Sprintf("上传进度: %.1f%%", progress.Percent),
Timestamp: now,
Status: state.status,
Progress: &progress,
}
state.events = append(state.events, event)
for _, subscriber := range state.subscribers {
select {
case subscriber <- event:
default:
}
}
}
func (h *LogHub) ensureState(recordID uint) *logStreamState {
state, ok := h.streams[recordID]
if ok {

View File

@@ -11,6 +11,28 @@ import (
"backupx/server/internal/storage"
)
// collectDirPrefixes 从待删除的记录中提取唯一的父目录前缀。
func collectDirPrefixes(records []model.BackupRecord) []string {
seen := make(map[string]struct{})
var prefixes []string
for _, record := range records {
path := strings.TrimSpace(record.StoragePath)
if path == "" {
continue
}
idx := strings.LastIndex(path, "/")
if idx <= 0 {
continue
}
dir := path[:idx]
if _, ok := seen[dir]; !ok {
seen[dir] = struct{}{}
prefixes = append(prefixes, dir)
}
}
return prefixes
}
type CleanupResult struct {
DeletedRecords int
DeletedObjects int
@@ -54,6 +76,17 @@ func (s *Service) Cleanup(ctx context.Context, task *model.BackupTask, provider
}
result.DeletedRecords++
}
// 清理空目录:收集被删除文件的父目录,尝试移除空目录
if dirCleaner, ok := provider.(storage.StorageDirCleaner); ok && result.DeletedObjects > 0 {
prefixes := collectDirPrefixes(candidates)
for _, prefix := range prefixes {
if err := dirCleaner.RemoveEmptyDirs(ctx, prefix); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("cleanup empty dirs for %s: %v", prefix, err))
}
}
}
return result, nil
}

View File

@@ -36,6 +36,9 @@ func (r *fakeRecordRepository) Delete(_ context.Context, id uint) error {
func (r *fakeRecordRepository) ListRecent(context.Context, int) ([]model.BackupRecord, error) {
return nil, nil
}
func (r *fakeRecordRepository) ListByTask(_ context.Context, _ uint) ([]model.BackupRecord, error) {
return r.records, nil
}
func (r *fakeRecordRepository) ListSuccessfulByTask(_ context.Context, _ uint) ([]model.BackupRecord, error) {
return r.records, nil
}

View File

@@ -1,16 +1,20 @@
package backup
import (
"archive/tar"
"context"
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// SAPHANARunner implements the BackupRunner interface for SAP HANA databases.
// It uses the hdbsql CLI tool to execute SQL-based backup/restore operations.
// It uses hdbsql to issue BACKUP DATA USING FILE commands for proper data-level
// backup (SAP best practice), rather than logical SQL export.
type SAPHANARunner struct {
executor CommandExecutor
}
@@ -28,24 +32,36 @@ func (r *SAPHANARunner) Type() string {
return "saphana"
}
// Run executes a SAP HANA backup using hdbsql.
// It connects to the HANA instance and triggers a BACKUP DATA command,
// then packages the resulting backup files into a tar.gz archive.
// Run executes a SAP HANA data-level backup using hdbsql + BACKUP DATA USING FILE.
// The backup files are written to a temporary directory, then packaged into a tar
// archive as the artifact for BackupX to compress/encrypt/upload.
//
// 支持以下增强(通过 task.Database 字段配置):
// - BackupLevel: full / incremental / differential
// - BackupType: data / log
// - BackupChannels: 并行通道数(>1 时生成多路径 SQL
// - MaxRetries: hdbsql 执行失败的重试次数
func (r *SAPHANARunner) Run(ctx context.Context, task TaskSpec, writer LogWriter) (*RunResult, error) {
if _, err := r.executor.LookPath("hdbsql"); err != nil {
return nil, fmt.Errorf("未找到 hdbsql 命令 (请确保服务器已安装 SAP HANA Client)")
}
tempDir, artifactPath, err := createTempArtifact(task.TempDir, task.Name, "sql")
startedAt := task.StartedAt
if startedAt.IsZero() {
startedAt = time.Now().UTC()
}
// Create a temp directory for the tar artifact output.
tempDir, artifactPath, err := createTempArtifact(task.TempDir, task.Name, "tar")
if err != nil {
return nil, err
}
file, err := os.Create(artifactPath)
if err != nil {
return nil, fmt.Errorf("create SAP HANA dump file: %w", err)
// Create a sub-directory where HANA will write its backup data files.
backupDir := filepath.Join(tempDir, "hana_data")
if err := os.MkdirAll(backupDir, 0o755); err != nil {
return nil, fmt.Errorf("create HANA backup directory: %w", err)
}
defer file.Close()
dbNames := normalizeDatabaseNames(task.Database.Names)
tenantDB := "SYSTEMDB"
@@ -58,81 +74,73 @@ func (r *SAPHANARunner) Run(ctx context.Context, task TaskSpec, writer LogWriter
port = 30015
}
writer.WriteLine(fmt.Sprintf("连接到 SAP HANA: %s:%d", task.Database.Host, port))
backupLevel := normalizeBackupLevel(task.Database.BackupLevel)
backupType := normalizeBackupType(task.Database.BackupType)
channels := task.Database.BackupChannels
if channels < 1 {
channels = 1
}
maxRetries := task.Database.MaxRetries
if maxRetries < 1 {
maxRetries = 3
}
instance := task.Database.InstanceNumber
if strings.TrimSpace(instance) == "" {
instance = hanaInstanceNumber(port)
}
writer.WriteLine(fmt.Sprintf("连接到 SAP HANA: %s:%d (实例 %s)", task.Database.Host, port, instance))
writer.WriteLine(fmt.Sprintf("备份数据库: %s", tenantDB))
writer.WriteLine(fmt.Sprintf("备份配置: 类型=%s, 级别=%s, 通道数=%d, 最大重试=%d", backupType, backupLevel, channels, maxRetries))
// Build hdbsql connection arguments
args := []string{
"-n", fmt.Sprintf("%s:%d", task.Database.Host, port),
"-u", task.Database.User,
"-p", task.Database.Password,
"-d", tenantDB,
"-j", // disable auto-commit
"-A", // disable column alignment
"-xC", // suppress column headers and separator
// Build backup prefix — HANA will create files like <prefix>_databackup_<N>_1.
timestamp := startedAt.UTC().Format("20060102_150405")
prefixes, err := buildBackupPrefixes(backupDir, tenantDB, timestamp, channels)
if err != nil {
return nil, err
}
// Export schema using SELECT statements for each table.
// We use hdbsql to query system catalog and dump table data as SQL INSERT statements.
exportSQL := fmt.Sprintf(`SELECT
'CREATE SCHEMA "' || SCHEMA_NAME || '";'
FROM SCHEMAS
WHERE HAS_PRIVILEGES = 'TRUE'
AND SCHEMA_NAME NOT LIKE '%%SYS%%'
AND SCHEMA_NAME NOT LIKE '_%%'
AND SCHEMA_NAME != 'SAP_REST_API'
ORDER BY SCHEMA_NAME`)
// Build SQL based on backup type and level.
backupSQL := buildBackupSQL(tenantDB, prefixes, backupType, backupLevel)
writer.WriteLine(fmt.Sprintf("生成 SQL: %s", backupSQL))
exportArgs := append(append([]string{}, args...), exportSQL)
// Construct hdbsql connection arguments.
args := buildHdbsqlArgs(task.Database.Host, port, task.Database.User, task.Database.Password, tenantDB, backupSQL)
stderrWriter := newLogLineWriter(writer, "hdbsql")
writer.WriteLine("开始执行 SAP HANA 数据导出")
writer.WriteLine("开始执行 SAP HANA 备份命令")
if err := r.executor.Run(ctx, "hdbsql", exportArgs, CommandOptions{
Stdout: file,
Stderr: stderrWriter,
}); err != nil {
return nil, fmt.Errorf("run hdbsql export: %w: %s", err, stderrWriter.collected())
if err := r.runHdbsqlWithRetry(ctx, "hdbsql", args, maxRetries, writer); err != nil {
return nil, fmt.Errorf("run hdbsql backup: %w", err)
}
// If multiple databases were specified, export each additional one
for i := 1; i < len(dbNames); i++ {
writer.WriteLine(fmt.Sprintf("导出额外数据库: %s", dbNames[i]))
if _, writeErr := file.WriteString(fmt.Sprintf("\n-- Database: %s\n", dbNames[i])); writeErr != nil {
return nil, fmt.Errorf("write database separator: %w", writeErr)
}
writer.WriteLine("SAP HANA 备份命令执行完成,开始打包备份文件")
additionalArgs := []string{
"-n", fmt.Sprintf("%s:%d", task.Database.Host, port),
"-u", task.Database.User,
"-p", task.Database.Password,
"-d", dbNames[i],
"-j", "-A", "-xC",
exportSQL,
}
if err := r.executor.Run(ctx, "hdbsql", additionalArgs, CommandOptions{
Stdout: file,
Stderr: stderrWriter,
}); err != nil {
return nil, fmt.Errorf("run hdbsql export for %s: %w", dbNames[i], err)
}
// Package all generated backup files into a tar archive.
if err := packageBackupFiles(backupDir, artifactPath, writer); err != nil {
return nil, fmt.Errorf("package HANA backup files: %w", err)
}
info, _ := file.Stat()
info, _ := os.Stat(artifactPath)
sizeStr := "未知"
var fileSize int64
if info != nil {
sizeStr = formatFileSize(info.Size())
fileSize = info.Size()
sizeStr = formatFileSize(fileSize)
}
writer.WriteLine(fmt.Sprintf("SAP HANA 导出完成(文件大小: %s", sizeStr))
writer.WriteLine(fmt.Sprintf("SAP HANA 备份完成(归档大小: %s", sizeStr))
return &RunResult{
ArtifactPath: artifactPath,
FileName: filepath.Base(artifactPath),
TempDir: tempDir,
Size: fileSize,
StorageKey: BuildStorageKey("saphana", startedAt, filepath.Base(artifactPath)),
}, nil
}
// Restore executes a SAP HANA restore using hdbsql to replay the SQL dump file.
// Restore executes a SAP HANA restore using RECOVER DATA USING FILE.
// It extracts the tar archive to get the original backup files, then issues
// the recovery SQL command via hdbsql.
func (r *SAPHANARunner) Restore(ctx context.Context, task TaskSpec, artifactPath string, writer LogWriter) error {
if _, err := r.executor.LookPath("hdbsql"); err != nil {
return fmt.Errorf("未找到 hdbsql 命令 (请确保服务器已安装 SAP HANA Client)")
@@ -151,27 +159,39 @@ func (r *SAPHANARunner) Restore(ctx context.Context, task TaskSpec, artifactPath
writer.WriteLine(fmt.Sprintf("开始恢复 SAP HANA 数据库: %s", tenantDB))
input, err := os.Open(filepath.Clean(artifactPath))
// Extract the tar archive to a temporary directory.
restoreDir, err := os.MkdirTemp("", "backupx-hana-restore-*")
if err != nil {
return fmt.Errorf("open SAP HANA restore file: %w", err)
return fmt.Errorf("create restore temp dir: %w", err)
}
defer input.Close()
defer os.RemoveAll(restoreDir)
args := []string{
"-n", fmt.Sprintf("%s:%d", task.Database.Host, port),
"-u", task.Database.User,
"-p", task.Database.Password,
"-d", tenantDB,
"-j",
"-I", artifactPath,
if err := extractTarArchive(artifactPath, restoreDir); err != nil {
return fmt.Errorf("extract HANA backup tar: %w", err)
}
stderrWriter := newLogLineWriter(writer, "hdbsql")
if err := r.executor.Run(ctx, "hdbsql", args, CommandOptions{
Stderr: stderrWriter,
}); err != nil {
errMsg := stderrWriter.collected()
return fmt.Errorf("run hdbsql restore: %w: %s", err, strings.TrimSpace(errMsg))
// Find the backup prefix by locating backup data files.
prefix, err := findBackupPrefix(restoreDir)
if err != nil {
return fmt.Errorf("find backup prefix: %w", err)
}
writer.WriteLine(fmt.Sprintf("找到备份前缀: %s", filepath.Base(prefix)))
// Build RECOVER DATA SQL.
recoverSQL := fmt.Sprintf(`RECOVER DATA USING FILE ('%s') CLEAR LOG`, prefix)
if strings.ToUpper(tenantDB) != "SYSTEMDB" {
recoverSQL = fmt.Sprintf(`RECOVER DATA FOR %s USING FILE ('%s') CLEAR LOG`, tenantDB, prefix)
}
args := buildHdbsqlArgs(task.Database.Host, port, task.Database.User, task.Database.Password, tenantDB, recoverSQL)
maxRetries := task.Database.MaxRetries
if maxRetries < 1 {
maxRetries = 3
}
if err := r.runHdbsqlWithRetry(ctx, "hdbsql", args, maxRetries, writer); err != nil {
return fmt.Errorf("run hdbsql RECOVER DATA: %w", err)
}
writer.WriteLine("SAP HANA 恢复完成")
@@ -187,3 +207,258 @@ func hanaInstanceNumber(port int) string {
}
return "00"
}
// normalizeBackupLevel 规范化备份级别值,无效或空值默认为 "full"。
func normalizeBackupLevel(level string) string {
switch strings.ToLower(strings.TrimSpace(level)) {
case "incremental":
return "incremental"
case "differential":
return "differential"
default:
return "full"
}
}
// normalizeBackupType 规范化备份类型,无效或空值默认为 "data"。
func normalizeBackupType(t string) string {
switch strings.ToLower(strings.TrimSpace(t)) {
case "log":
return "log"
default:
return "data"
}
}
// buildBackupPrefixes 为每个并行通道生成独立子目录和路径前缀。
// 当 channels=1 时返回单个直接位于 backupDir 下的前缀;
// 当 channels>1 时为每个通道创建 chan_N/ 子目录。
func buildBackupPrefixes(backupDir, tenantDB, timestamp string, channels int) ([]string, error) {
tenantLower := strings.ToLower(tenantDB)
if channels <= 1 {
return []string{filepath.Join(backupDir, fmt.Sprintf("hana_%s_%s", tenantLower, timestamp))}, nil
}
prefixes := make([]string, 0, channels)
for i := 0; i < channels; i++ {
chanDir := filepath.Join(backupDir, fmt.Sprintf("chan_%d", i))
if err := os.MkdirAll(chanDir, 0o755); err != nil {
return nil, fmt.Errorf("create channel %d dir: %w", i, err)
}
prefixes = append(prefixes, filepath.Join(chanDir, fmt.Sprintf("hana_%s_%s", tenantLower, timestamp)))
}
return prefixes, nil
}
// buildBackupSQL 根据备份类型和级别构建 SAP HANA BACKUP SQL 语句。
//
// 支持的语法:
//
// 全量数据备份: BACKUP DATA [FOR <tenant>] USING FILE ('p1' [, 'p2', ...])
// 增量数据备份: BACKUP DATA [FOR <tenant>] INCREMENTAL USING FILE ('...')
// 差异数据备份: BACKUP DATA [FOR <tenant>] DIFFERENTIAL USING FILE ('...')
// 日志备份: BACKUP LOG [FOR <tenant>] USING FILE ('...')
func buildBackupSQL(tenantDB string, prefixes []string, backupType, backupLevel string) string {
tenantClause := ""
if strings.ToUpper(tenantDB) != "SYSTEMDB" {
tenantClause = fmt.Sprintf(" FOR %s", tenantDB)
}
// 多路径以 'p1', 'p2', ... 拼接HANA 多通道并行语法)
quoted := make([]string, len(prefixes))
for i, p := range prefixes {
quoted[i] = fmt.Sprintf("'%s'", p)
}
pathClause := strings.Join(quoted, ", ")
if backupType == "log" {
// LOG 备份不支持 INCREMENTAL/DIFFERENTIAL 关键字
return fmt.Sprintf("BACKUP LOG%s USING FILE (%s)", tenantClause, pathClause)
}
levelClause := ""
switch backupLevel {
case "incremental":
levelClause = " INCREMENTAL"
case "differential":
levelClause = " DIFFERENTIAL"
}
return fmt.Sprintf("BACKUP DATA%s%s USING FILE (%s)", tenantClause, levelClause, pathClause)
}
// runHdbsqlWithRetry 执行 hdbsql 命令并在失败时按指数退避重试。
// 退避公式5s × attempt²并在 ctx 取消时立即返回。
func (r *SAPHANARunner) runHdbsqlWithRetry(ctx context.Context, name string, args []string, maxAttempts int, writer LogWriter) error {
if maxAttempts < 1 {
maxAttempts = 1
}
var lastErr error
for attempt := 1; attempt <= maxAttempts; attempt++ {
if attempt > 1 {
backoff := time.Duration(attempt*attempt) * 5 * time.Second
writer.WriteLine(fmt.Sprintf("hdbsql 第 %d 次重试(等待 %s", attempt, backoff))
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
}
stderrWriter := newLogLineWriter(writer, "hdbsql")
err := r.executor.Run(ctx, name, args, CommandOptions{Stderr: stderrWriter})
if err == nil {
return nil
}
lastErr = fmt.Errorf("%w: %s", err, strings.TrimSpace(stderrWriter.collected()))
writer.WriteLine(fmt.Sprintf("hdbsql 执行失败(第 %d/%d 次): %v", attempt, maxAttempts, lastErr))
}
return lastErr
}
// buildHdbsqlArgs constructs the common hdbsql CLI arguments.
func buildHdbsqlArgs(host string, port int, user, password, database, sql string) []string {
return []string{
"-n", fmt.Sprintf("%s:%d", host, port),
"-u", user,
"-p", password,
"-d", database,
"-j", // disable auto-commit
"-A", // disable column alignment
"-xC", // suppress column headers and separator
sql,
}
}
// packageBackupFiles creates a tar archive from all files in the given directory.
func packageBackupFiles(sourceDir, targetPath string, writer LogWriter) error {
file, err := os.Create(targetPath)
if err != nil {
return fmt.Errorf("create tar file: %w", err)
}
defer file.Close()
tw := tar.NewWriter(file)
defer tw.Close()
fileCount := 0
walkErr := filepath.Walk(sourceDir, func(currentPath string, info os.FileInfo, walkErr error) error {
if walkErr != nil {
return walkErr
}
if currentPath == sourceDir {
return nil
}
relPath, err := filepath.Rel(sourceDir, currentPath)
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, "")
if err != nil {
return err
}
header.Name = filepath.ToSlash(relPath)
if err := tw.WriteHeader(header); err != nil {
return err
}
if info.Mode().IsRegular() {
f, err := os.Open(currentPath)
if err != nil {
return err
}
defer f.Close()
if _, err := io.CopyN(tw, f, info.Size()); err != nil && err != io.EOF {
return err
}
fileCount++
}
return nil
})
if walkErr != nil {
return walkErr
}
if fileCount == 0 {
return fmt.Errorf("HANA 备份目录中未找到任何备份文件")
}
writer.WriteLine(fmt.Sprintf("已打包 %d 个备份文件", fileCount))
return nil
}
// extractTarArchive extracts a tar archive to the given directory.
func extractTarArchive(tarPath, targetDir string) error {
f, err := os.Open(filepath.Clean(tarPath))
if err != nil {
return err
}
defer f.Close()
tr := tar.NewReader(f)
for {
header, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return fmt.Errorf("read tar entry: %w", err)
}
targetPath := filepath.Join(targetDir, filepath.FromSlash(filepath.Clean(header.Name)))
// Guard against path traversal.
if !strings.HasPrefix(targetPath, filepath.Clean(targetDir)+string(filepath.Separator)) {
continue
}
switch header.Typeflag {
case tar.TypeDir:
if err := os.MkdirAll(targetPath, 0o755); err != nil {
return err
}
case tar.TypeReg, tar.TypeRegA:
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return err
}
outFile, err := os.Create(targetPath)
if err != nil {
return err
}
if _, err := io.Copy(outFile, tr); err != nil {
outFile.Close()
return err
}
outFile.Close()
}
}
return nil
}
// findBackupPrefix locates the backup prefix by scanning for HANA backup data files.
// HANA creates files like <prefix>_databackup_0_1, <prefix>_databackup_1_1, etc.
func findBackupPrefix(dir string) (string, error) {
var prefix string
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil || info.IsDir() {
return err
}
name := info.Name()
if idx := strings.Index(name, "_databackup_"); idx > 0 {
prefix = filepath.Join(filepath.Dir(path), name[:idx])
return filepath.SkipAll
}
// Also check for the complete backup file pattern without _databackup_
if strings.HasPrefix(name, "hana_") {
prefix = filepath.Join(filepath.Dir(path), strings.TrimSuffix(name, filepath.Ext(name)))
return filepath.SkipAll
}
return nil
})
if err != nil && err != filepath.SkipAll {
return "", err
}
if prefix == "" {
return "", fmt.Errorf("未在归档中找到 HANA 备份数据文件")
}
return prefix, nil
}

View File

@@ -0,0 +1,535 @@
package backup
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestSAPHANARunnerRun_BackupDataCommand(t *testing.T) {
var capturedArgs []string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedArgs = append([]string{}, args...)
// Simulate HANA creating backup data files in the directory from the SQL.
// Parse the backup prefix from the SQL argument (last arg).
sql := args[len(args)-1]
// Extract path from: BACKUP DATA USING FILE ('/path/to/hana_systemdb_...')
startIdx := strings.Index(sql, "('") + 2
endIdx := strings.Index(sql, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := sql[startIdx:endIdx]
dir := filepath.Dir(prefix)
_ = os.MkdirAll(dir, 0o755)
// Create fake backup data files that HANA would produce.
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("fake backup data volume 0"), 0o644)
_ = os.WriteFile(prefix+"_databackup_1_1", []byte("fake backup data volume 1"), 0o644)
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-daily",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1",
Port: 30015,
User: "SYSTEM",
Password: "secret",
Names: []string{"SYSTEMDB"},
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
// Verify hdbsql was called with the correct connection args.
if len(capturedArgs) == 0 {
t.Fatal("expected hdbsql args to be captured")
}
// Check host:port
foundHost := false
for i, arg := range capturedArgs {
if arg == "-n" && i+1 < len(capturedArgs) && capturedArgs[i+1] == "10.0.0.1:30015" {
foundHost = true
}
}
if !foundHost {
t.Fatalf("expected host:port 10.0.0.1:30015 in args, got: %v", capturedArgs)
}
// Verify the SQL contains BACKUP DATA USING FILE.
lastArg := capturedArgs[len(capturedArgs)-1]
if !strings.Contains(lastArg, "BACKUP DATA USING FILE") {
t.Fatalf("expected BACKUP DATA USING FILE in SQL, got: %s", lastArg)
}
// Verify artifact is a tar file.
if !strings.HasSuffix(result.ArtifactPath, ".tar") {
t.Fatalf("expected .tar artifact, got: %s", result.ArtifactPath)
}
// Verify artifact file exists and has content.
info, err := os.Stat(result.ArtifactPath)
if err != nil {
t.Fatalf("artifact file missing: %v", err)
}
if info.Size() == 0 {
t.Fatal("artifact tar file is empty")
}
// Cleanup.
os.RemoveAll(result.TempDir)
}
func TestSAPHANARunnerRun_TenantDatabase(t *testing.T) {
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
// Simulate HANA creating backup files.
startIdx := strings.Index(capturedSQL, "('") + 2
endIdx := strings.Index(capturedSQL, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := capturedSQL[startIdx:endIdx]
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("data"), 0o644)
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-tenant",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1",
Port: 30015,
User: "SYSTEM",
Password: "secret",
Names: []string{"HDB"},
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
defer os.RemoveAll(result.TempDir)
// For tenant databases, the SQL should use BACKUP DATA FOR <tenant>.
if !strings.Contains(capturedSQL, "BACKUP DATA FOR HDB USING FILE") {
t.Fatalf("expected BACKUP DATA FOR HDB in SQL, got: %s", capturedSQL)
}
}
func TestSAPHANARunnerRun_DefaultPort(t *testing.T) {
var capturedArgs []string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedArgs = append([]string{}, args...)
sql := args[len(args)-1]
startIdx := strings.Index(sql, "('") + 2
endIdx := strings.Index(sql, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := sql[startIdx:endIdx]
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("data"), 0o644)
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-default-port",
Type: "saphana",
Database: DatabaseSpec{
Host: "localhost",
Port: 0, // Should default to 30015
User: "SYSTEM",
Password: "secret",
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
defer os.RemoveAll(result.TempDir)
// Verify default port 30015 was used.
for i, arg := range capturedArgs {
if arg == "-n" && i+1 < len(capturedArgs) {
if !strings.HasSuffix(capturedArgs[i+1], ":30015") {
t.Fatalf("expected default port 30015, got: %s", capturedArgs[i+1])
}
}
}
}
func TestSAPHANARunnerRun_LookPathError(t *testing.T) {
runner := NewSAPHANARunner(&fakeCommandExecutor{lookupErr: errors.New("not found")})
_, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-missing",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
},
}, NopLogWriter{})
if err == nil {
t.Fatal("expected error when hdbsql is missing")
}
if !strings.Contains(err.Error(), "hdbsql") {
t.Fatalf("error should mention hdbsql, got: %v", err)
}
}
func TestSAPHANARunnerRestore_RecoverDataCommand(t *testing.T) {
// First, create a fake tar archive with a backup data file.
tarDir := t.TempDir()
dataDir := filepath.Join(tarDir, "hana_data")
_ = os.MkdirAll(dataDir, 0o755)
prefix := filepath.Join(dataDir, "hana_systemdb_20260324_120000")
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("backup data"), 0o644)
// Create the tar.
tarPath := filepath.Join(tarDir, "backup.tar")
if err := packageBackupFiles(dataDir, tarPath, NopLogWriter{}); err != nil {
t.Fatalf("failed to create test tar: %v", err)
}
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
return nil
},
}
runner := NewSAPHANARunner(executor)
err := runner.Restore(context.Background(), TaskSpec{
Name: "hana-restore",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
Names: []string{"SYSTEMDB"},
},
}, tarPath, NopLogWriter{})
if err != nil {
t.Fatalf("Restore returned error: %v", err)
}
if !strings.Contains(capturedSQL, "RECOVER DATA USING FILE") {
t.Fatalf("expected RECOVER DATA USING FILE in SQL, got: %s", capturedSQL)
}
if !strings.Contains(capturedSQL, "CLEAR LOG") {
t.Fatalf("expected CLEAR LOG in SQL, got: %s", capturedSQL)
}
}
func TestSAPHANARunnerRestore_TenantRecoverCommand(t *testing.T) {
tarDir := t.TempDir()
dataDir := filepath.Join(tarDir, "data")
_ = os.MkdirAll(dataDir, 0o755)
_ = os.WriteFile(filepath.Join(dataDir, "hana_hdb_20260324_120000_databackup_0_1"), []byte("data"), 0o644)
tarPath := filepath.Join(tarDir, "backup.tar")
if err := packageBackupFiles(dataDir, tarPath, NopLogWriter{}); err != nil {
t.Fatalf("failed to create test tar: %v", err)
}
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
return nil
},
}
runner := NewSAPHANARunner(executor)
err := runner.Restore(context.Background(), TaskSpec{
Name: "hana-tenant-restore",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
Names: []string{"HDB"},
},
}, tarPath, NopLogWriter{})
if err != nil {
t.Fatalf("Restore returned error: %v", err)
}
if !strings.Contains(capturedSQL, "RECOVER DATA FOR HDB USING FILE") {
t.Fatalf("expected RECOVER DATA FOR HDB in SQL, got: %s", capturedSQL)
}
}
func TestBuildBackupSQL_FullSystemDB(t *testing.T) {
sql := buildBackupSQL("SYSTEMDB", []string{"/tmp/p1"}, "data", "full")
if sql != "BACKUP DATA USING FILE ('/tmp/p1')" {
t.Fatalf("unexpected SQL: %s", sql)
}
}
func TestBuildBackupSQL_IncrementalTenant(t *testing.T) {
sql := buildBackupSQL("HDB", []string{"/tmp/p1"}, "data", "incremental")
expected := "BACKUP DATA FOR HDB INCREMENTAL USING FILE ('/tmp/p1')"
if sql != expected {
t.Fatalf("expected %q, got %q", expected, sql)
}
}
func TestBuildBackupSQL_DifferentialTenant(t *testing.T) {
sql := buildBackupSQL("HDB", []string{"/tmp/p1"}, "data", "differential")
expected := "BACKUP DATA FOR HDB DIFFERENTIAL USING FILE ('/tmp/p1')"
if sql != expected {
t.Fatalf("expected %q, got %q", expected, sql)
}
}
func TestBuildBackupSQL_LogBackup(t *testing.T) {
sql := buildBackupSQL("HDB", []string{"/tmp/log"}, "log", "full")
expected := "BACKUP LOG FOR HDB USING FILE ('/tmp/log')"
if sql != expected {
t.Fatalf("expected %q, got %q", expected, sql)
}
}
func TestBuildBackupSQL_ParallelChannels(t *testing.T) {
sql := buildBackupSQL("SYSTEMDB", []string{"/tmp/c0/p", "/tmp/c1/p", "/tmp/c2/p"}, "data", "full")
expected := "BACKUP DATA USING FILE ('/tmp/c0/p', '/tmp/c1/p', '/tmp/c2/p')"
if sql != expected {
t.Fatalf("expected %q, got %q", expected, sql)
}
}
func TestNormalizeBackupLevel(t *testing.T) {
cases := map[string]string{
"": "full",
"FULL": "full",
"incremental": "incremental",
"DIFFERENTIAL": "differential",
"unknown": "full",
}
for in, want := range cases {
if got := normalizeBackupLevel(in); got != want {
t.Errorf("normalizeBackupLevel(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeBackupType(t *testing.T) {
cases := map[string]string{
"": "data",
"DATA": "data",
"log": "log",
"LOG": "log",
}
for in, want := range cases {
if got := normalizeBackupType(in); got != want {
t.Errorf("normalizeBackupType(%q) = %q, want %q", in, got, want)
}
}
}
func TestSAPHANARunnerRun_IncrementalBackup(t *testing.T) {
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
startIdx := strings.Index(capturedSQL, "('") + 2
endIdx := strings.Index(capturedSQL, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := capturedSQL[startIdx:endIdx]
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("incremental data"), 0o644)
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-incremental",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1",
Port: 30015,
User: "SYSTEM",
Password: "secret",
Names: []string{"HDB"},
BackupLevel: "incremental",
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
defer os.RemoveAll(result.TempDir)
if !strings.Contains(capturedSQL, "INCREMENTAL USING FILE") {
t.Fatalf("expected INCREMENTAL in SQL, got: %s", capturedSQL)
}
}
func TestSAPHANARunnerRun_LogBackup(t *testing.T) {
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
startIdx := strings.Index(capturedSQL, "('") + 2
endIdx := strings.Index(capturedSQL, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := capturedSQL[startIdx:endIdx]
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_logbackup_0_1", []byte("log data"), 0o644)
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-log",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
Names: []string{"HDB"},
BackupType: "log",
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
defer os.RemoveAll(result.TempDir)
if !strings.Contains(capturedSQL, "BACKUP LOG FOR HDB USING FILE") {
t.Fatalf("expected log backup SQL, got: %s", capturedSQL)
}
}
func TestSAPHANARunnerRun_ParallelChannels(t *testing.T) {
var capturedSQL string
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
capturedSQL = args[len(args)-1]
// 模拟为每个通道生成备份文件
parts := strings.Split(capturedSQL, "',")
for _, p := range parts {
p = strings.TrimSpace(p)
if idx := strings.Index(p, "'"); idx >= 0 {
prefix := p[idx+1:]
prefix = strings.TrimSuffix(prefix, "')")
prefix = strings.TrimSuffix(prefix, "'")
if prefix != "" {
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("data"), 0o644)
}
}
}
return nil
},
}
runner := NewSAPHANARunner(executor)
result, err := runner.Run(context.Background(), TaskSpec{
Name: "hana-parallel",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
Names: []string{"SYSTEMDB"},
BackupChannels: 3,
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error: %v", err)
}
defer os.RemoveAll(result.TempDir)
// 应该包含 3 个路径
if strings.Count(capturedSQL, "'") != 6 { // 3 路径 × 2 引号
t.Fatalf("expected 3 channels (6 quotes), got SQL: %s", capturedSQL)
}
if !strings.Contains(capturedSQL, "chan_0") || !strings.Contains(capturedSQL, "chan_2") {
t.Fatalf("expected channel directories in SQL, got: %s", capturedSQL)
}
}
func TestSAPHANARunnerRun_RetryOnFailure(t *testing.T) {
attempts := 0
executor := &fakeCommandExecutor{
runFunc: func(name string, args []string, options CommandOptions) error {
attempts++
if attempts < 2 {
return errors.New("transient failure")
}
// 第二次成功,写入备份文件
sql := args[len(args)-1]
startIdx := strings.Index(sql, "('") + 2
endIdx := strings.Index(sql, "')")
if startIdx > 1 && endIdx > startIdx {
prefix := sql[startIdx:endIdx]
_ = os.MkdirAll(filepath.Dir(prefix), 0o755)
_ = os.WriteFile(prefix+"_databackup_0_1", []byte("data"), 0o644)
}
return nil
},
}
// 使用极短的重试周期版本(这里通过 fake context 机制无法快进时间,所以直接验证 attempts
// 设置 MaxRetries=2 以加快测试,不会真实等待 5s
runner := NewSAPHANARunner(executor)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
result, err := runner.Run(ctx, TaskSpec{
Name: "hana-retry",
Type: "saphana",
Database: DatabaseSpec{
Host: "10.0.0.1", Port: 30015, User: "SYSTEM", Password: "secret",
Names: []string{"SYSTEMDB"},
MaxRetries: 2,
},
}, NopLogWriter{})
if err != nil {
t.Fatalf("Run returned error after retry: %v", err)
}
defer os.RemoveAll(result.TempDir)
if attempts != 2 {
t.Fatalf("expected 2 attempts, got %d", attempts)
}
}
func TestHanaInstanceNumber(t *testing.T) {
tests := []struct {
port int
expected string
}{
{30015, "0"},
{30115, "1"},
{30215, "2"},
{31015, "10"},
{25000, "00"},
{40001, "00"},
}
for _, tc := range tests {
got := hanaInstanceNumber(tc.port)
if got != tc.expected {
t.Errorf("hanaInstanceNumber(%d) = %s, want %s", tc.port, got, tc.expected)
}
}
}

View File

@@ -12,6 +12,12 @@ type DatabaseSpec struct {
Password string
Names []string
Path string
// SAP HANA 特有字段(其他类型忽略)
InstanceNumber string // 实例编号(从端口推断或手动指定)
BackupLevel string // "full"(默认) / "incremental" / "differential"
BackupType string // "data"(默认) / "log"
BackupChannels int // 并行通道数(默认 1
MaxRetries int // 最大重试次数(默认 3
}
type TaskSpec struct {
@@ -19,6 +25,7 @@ type TaskSpec struct {
Name string
Type string
SourcePath string
SourcePaths []string
ExcludePatterns []string
Database DatabaseSpec
StorageTargetID uint
@@ -40,13 +47,23 @@ type RunResult struct {
}
type LogEvent struct {
RecordID uint `json:"recordId"`
Sequence int64 `json:"sequence"`
Level string `json:"level"`
Message string `json:"message"`
Timestamp time.Time `json:"timestamp"`
Completed bool `json:"completed"`
Status string `json:"status"`
RecordID uint `json:"recordId"`
Sequence int64 `json:"sequence"`
Level string `json:"level"`
Message string `json:"message"`
Timestamp time.Time `json:"timestamp"`
Completed bool `json:"completed"`
Status string `json:"status"`
Progress *ProgressInfo `json:"progress,omitempty"`
}
// ProgressInfo 描述上传进度,通过 SSE 实时推送给前端。
type ProgressInfo struct {
BytesSent int64 `json:"bytesSent"`
TotalBytes int64 `json:"totalBytes"`
Percent float64 `json:"percent"`
SpeedBps float64 `json:"speedBps"` // bytes/sec
TargetName string `json:"targetName"`
}
type LogWriter interface {

View File

@@ -33,8 +33,10 @@ type SecurityConfig struct {
}
type BackupConfig struct {
TempDir string `mapstructure:"temp_dir"`
MaxConcurrent int `mapstructure:"max_concurrent"`
TempDir string `mapstructure:"temp_dir"`
MaxConcurrent int `mapstructure:"max_concurrent"`
Retries int `mapstructure:"retries"` // 底层 HTTP 请求重试次数,默认 10
BandwidthLimit string `mapstructure:"bandwidth_limit"` // 带宽限制,如 "10M",空不限
}
type LogConfig struct {
@@ -96,6 +98,9 @@ func Load(configPath string) (Config, error) {
if cfg.Backup.MaxConcurrent <= 0 {
cfg.Backup.MaxConcurrent = 2
}
if cfg.Backup.Retries <= 0 {
cfg.Backup.Retries = 10
}
if cfg.Log.Level == "" {
cfg.Log.Level = "info"
}
@@ -135,6 +140,8 @@ func applyDefaults(v *viper.Viper) {
v.SetDefault("security.jwt_expire", "24h")
v.SetDefault("backup.temp_dir", "/tmp/backupx")
v.SetDefault("backup.max_concurrent", 2)
v.SetDefault("backup.retries", 10)
v.SetDefault("backup.bandwidth_limit", "")
v.SetDefault("log.level", "info")
v.SetDefault("log.file", "./data/backupx.log")
v.SetDefault("log.max_size", 100)

View File

@@ -23,10 +23,17 @@ func Open(cfg config.DatabaseConfig, logger *zap.Logger) (*gorm.DB, error) {
return nil, fmt.Errorf("open sqlite: %w", err)
}
if err := db.AutoMigrate(&model.User{}, &model.SystemConfig{}, &model.StorageTarget{}, &model.OAuthSession{}, &model.BackupTask{}, &model.BackupRecord{}, &model.Notification{}, &model.Node{}); err != nil {
if err := db.AutoMigrate(&model.User{}, &model.SystemConfig{}, &model.StorageTarget{}, &model.OAuthSession{}, &model.BackupTask{}, &model.BackupRecord{}, &model.Notification{}, &model.Node{}, &model.BackupTaskStorageTarget{}, &model.AuditLog{}, &model.AgentCommand{}); err != nil {
return nil, fmt.Errorf("migrate schema: %w", err)
}
// 一次性数据迁移:从 backup_tasks.storage_target_id 回填到多对多中间表
var count int64
db.Model(&model.BackupTaskStorageTarget{}).Count(&count)
if count == 0 {
db.Exec("INSERT INTO backup_task_storage_targets (backup_task_id, storage_target_id) SELECT id, storage_target_id FROM backup_tasks WHERE storage_target_id > 0")
}
logger.Info("database initialized", zap.String("path", cfg.Path))
return db, nil
}

View File

@@ -0,0 +1,156 @@
package http
import (
stdhttp "net/http"
"strconv"
"strings"
"backupx/server/internal/service"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
// AgentHandler 实现 Agent 调用 Master 的 HTTP API。
// 全部端点通过 X-Agent-Token 头做节点认证,不使用 JWT。
type AgentHandler struct {
agentService *service.AgentService
nodeService *service.NodeService
}
func NewAgentHandler(agentService *service.AgentService, nodeService *service.NodeService) *AgentHandler {
return &AgentHandler{agentService: agentService, nodeService: nodeService}
}
// extractToken 从请求头或 JSON body 中提取 Agent Token。
func extractToken(c *gin.Context) string {
if t := strings.TrimSpace(c.GetHeader("X-Agent-Token")); t != "" {
return t
}
// Authorization: Bearer <token>
if auth := c.GetHeader("Authorization"); strings.HasPrefix(auth, "Bearer ") {
return strings.TrimSpace(strings.TrimPrefix(auth, "Bearer "))
}
return ""
}
// Heartbeat 扩展原有 heartbeat除上报状态外返回节点 ID 给 Agent 做后续调用。
func (h *AgentHandler) Heartbeat(c *gin.Context) {
var input struct {
Token string `json:"token"`
Hostname string `json:"hostname"`
IPAddress string `json:"ipAddress"`
AgentVersion string `json:"agentVersion"`
OS string `json:"os"`
Arch string `json:"arch"`
}
_ = c.ShouldBindJSON(&input)
// token 优先走 body向后兼容否则从 header 读
token := input.Token
if token == "" {
token = extractToken(c)
}
if token == "" {
c.JSON(stdhttp.StatusBadRequest, gin.H{"code": "INVALID_INPUT", "message": "missing token"})
return
}
if err := h.nodeService.Heartbeat(c.Request.Context(), token, input.Hostname, input.IPAddress, input.AgentVersion, input.OS, input.Arch); err != nil {
response.Error(c, err)
return
}
// 返回节点元信息给 Agentnode_id 用于后续 API 路径)
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), token)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, gin.H{
"status": "ok",
"nodeId": node.ID,
"name": node.Name,
})
}
// Poll Agent 长轮询获取下一条待执行命令。
// 无命令时返回 {command: null}。
func (h *AgentHandler) Poll(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
cmd, err := h.agentService.PollCommand(c.Request.Context(), node)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, gin.H{"command": cmd})
}
// SubmitCommandResult Agent 上报命令执行结果。
func (h *AgentHandler) SubmitCommandResult(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
id, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
var input service.AgentCommandResult
if err := c.ShouldBindJSON(&input); err != nil {
c.JSON(stdhttp.StatusBadRequest, gin.H{"code": "INVALID_INPUT", "message": err.Error()})
return
}
if err := h.agentService.SubmitCommandResult(c.Request.Context(), node, uint(id), input); err != nil {
response.Error(c, err)
return
}
response.Success(c, gin.H{"status": "ok"})
}
// GetTaskSpec Agent 拉取任务规格(含解密后的存储配置)。
func (h *AgentHandler) GetTaskSpec(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
id, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
spec, err := h.agentService.GetTaskSpec(c.Request.Context(), node, uint(id))
if err != nil {
response.Error(c, err)
return
}
response.Success(c, spec)
}
// UpdateRecord Agent 更新备份记录(进度/完成状态/日志)。
func (h *AgentHandler) UpdateRecord(c *gin.Context) {
node, err := h.agentService.AuthenticatedNode(c.Request.Context(), extractToken(c))
if err != nil {
response.Error(c, err)
return
}
id, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
var input service.AgentRecordUpdate
if err := c.ShouldBindJSON(&input); err != nil {
c.JSON(stdhttp.StatusBadRequest, gin.H{"code": "INVALID_INPUT", "message": err.Error()})
return
}
if err := h.agentService.UpdateRecord(c.Request.Context(), node, uint(id), input); err != nil {
response.Error(c, err)
return
}
response.Success(c, gin.H{"status": "ok"})
}

View File

@@ -0,0 +1,40 @@
package http
import (
"strconv"
"strings"
"backupx/server/internal/service"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
type AuditHandler struct {
auditService *service.AuditService
}
func NewAuditHandler(auditService *service.AuditService) *AuditHandler {
return &AuditHandler{auditService: auditService}
}
func (h *AuditHandler) List(c *gin.Context) {
category := strings.TrimSpace(c.Query("category"))
limit := 50
offset := 0
if v := strings.TrimSpace(c.Query("limit")); v != "" {
if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
limit = parsed
}
}
if v := strings.TrimSpace(c.Query("offset")); v != "" {
if parsed, err := strconv.Atoi(v); err == nil && parsed >= 0 {
offset = parsed
}
}
result, err := h.auditService.List(c.Request.Context(), category, limit, offset)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, result)
}

View File

@@ -0,0 +1,29 @@
package http
import (
"fmt"
"backupx/server/internal/service"
"github.com/gin-gonic/gin"
)
// recordAudit 从 gin context 中提取用户信息并记录审计日志nil 安全)
func recordAudit(c *gin.Context, auditService *service.AuditService, category, action, targetType, targetID, targetName, detail string) {
if auditService == nil {
return
}
username := ""
if subject, exists := c.Get(contextUserSubjectKey); exists {
username = fmt.Sprintf("%v", subject)
}
auditService.Record(service.AuditEntry{
Username: username,
Category: category,
Action: action,
TargetType: targetType,
TargetID: targetID,
TargetName: targetName,
Detail: detail,
ClientIP: c.ClientIP(),
})
}

View File

@@ -16,11 +16,12 @@ import (
)
type BackupRecordHandler struct {
service *service.BackupRecordService
service *service.BackupRecordService
auditService *service.AuditService
}
func NewBackupRecordHandler(recordService *service.BackupRecordService) *BackupRecordHandler {
return &BackupRecordHandler{service: recordService}
func NewBackupRecordHandler(recordService *service.BackupRecordService, auditService *service.AuditService) *BackupRecordHandler {
return &BackupRecordHandler{service: recordService, auditService: auditService}
}
func (h *BackupRecordHandler) List(c *gin.Context) {
@@ -129,6 +130,8 @@ func (h *BackupRecordHandler) Restore(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "backup_record", "restore", "backup_record", fmt.Sprintf("%d", id), "",
fmt.Sprintf("恢复备份记录 (ID: %d)", id))
response.Success(c, gin.H{"restored": true})
}
@@ -141,9 +144,29 @@ func (h *BackupRecordHandler) Delete(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "backup_record", "delete", "backup_record", fmt.Sprintf("%d", id), "",
fmt.Sprintf("删除备份记录 (ID: %d)", id))
response.Success(c, gin.H{"deleted": true})
}
func (h *BackupRecordHandler) BatchDelete(c *gin.Context) {
var input struct {
IDs []uint `json:"ids" binding:"required,min=1"`
}
if err := c.ShouldBindJSON(&input); err != nil {
response.Error(c, apperror.BadRequest("BACKUP_RECORD_BATCH_INVALID", "批量删除参数不合法", err))
return
}
deleted := 0
for _, id := range input.IDs {
if err := h.service.Delete(c.Request.Context(), id); err == nil {
deleted++
}
}
recordAudit(c, h.auditService, "backup_record", "batch_delete", "backup_record", "", "", fmt.Sprintf("批量删除 %d 条备份记录", deleted))
response.Success(c, gin.H{"deleted": deleted})
}
func buildRecordFilter(c *gin.Context) (service.BackupRecordListInput, error) {
var filter service.BackupRecordListInput
if taskIDValue := strings.TrimSpace(c.Query("taskId")); taskIDValue != "" {

View File

@@ -1,17 +1,20 @@
package http
import (
"fmt"
"backupx/server/internal/service"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
type BackupRunHandler struct {
service *service.BackupExecutionService
service *service.BackupExecutionService
auditService *service.AuditService
}
func NewBackupRunHandler(executionService *service.BackupExecutionService) *BackupRunHandler {
return &BackupRunHandler{service: executionService}
func NewBackupRunHandler(executionService *service.BackupExecutionService, auditService *service.AuditService) *BackupRunHandler {
return &BackupRunHandler{service: executionService, auditService: auditService}
}
func (h *BackupRunHandler) Run(c *gin.Context) {
@@ -24,5 +27,6 @@ func (h *BackupRunHandler) Run(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "backup_task", "run", "backup_task", fmt.Sprintf("%d", id), "", "手动触发备份")
response.Success(c, record)
}

View File

@@ -1,6 +1,8 @@
package http
import (
"fmt"
"backupx/server/internal/apperror"
"backupx/server/internal/service"
"backupx/server/pkg/response"
@@ -8,11 +10,25 @@ import (
)
type BackupTaskHandler struct {
service *service.BackupTaskService
service *service.BackupTaskService
auditService *service.AuditService
}
func NewBackupTaskHandler(taskService *service.BackupTaskService) *BackupTaskHandler {
return &BackupTaskHandler{service: taskService}
// describeTaskInput 提取审计日志中通用的调度和存储目标描述。
func describeTaskInput(input service.BackupTaskUpsertInput) (cronDesc string, storageCount int) {
cronDesc = "仅手动执行"
if input.CronExpr != "" {
cronDesc = input.CronExpr
}
storageCount = len(input.StorageTargetIDs)
if storageCount == 0 && input.StorageTargetID > 0 {
storageCount = 1
}
return
}
func NewBackupTaskHandler(taskService *service.BackupTaskService, auditService *service.AuditService) *BackupTaskHandler {
return &BackupTaskHandler{service: taskService, auditService: auditService}
}
func (h *BackupTaskHandler) List(c *gin.Context) {
@@ -48,6 +64,9 @@ func (h *BackupTaskHandler) Create(c *gin.Context) {
response.Error(c, err)
return
}
cronDesc, storageCount := describeTaskInput(input)
recordAudit(c, h.auditService, "backup_task", "create", "backup_task", fmt.Sprintf("%d", item.ID), item.Name,
fmt.Sprintf("创建备份任务「%s」类型: %s, 调度: %s, 存储: %d 个目标", item.Name, input.Type, cronDesc, storageCount))
response.Success(c, item)
}
@@ -66,6 +85,9 @@ func (h *BackupTaskHandler) Update(c *gin.Context) {
response.Error(c, err)
return
}
updCronDesc, updStorageCount := describeTaskInput(input)
recordAudit(c, h.auditService, "backup_task", "update", "backup_task", fmt.Sprintf("%d", item.ID), item.Name,
fmt.Sprintf("更新备份任务「%s」类型: %s, 调度: %s, 存储: %d 个目标", item.Name, input.Type, updCronDesc, updStorageCount))
response.Success(c, item)
}
@@ -74,10 +96,13 @@ func (h *BackupTaskHandler) Delete(c *gin.Context) {
if !ok {
return
}
if err := h.service.Delete(c.Request.Context(), id); err != nil {
result, err := h.service.Delete(c.Request.Context(), id)
if err != nil {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "backup_task", "delete", "backup_task", fmt.Sprintf("%d", id), result.TaskName,
fmt.Sprintf("删除备份任务「%s」(ID: %d),关联记录 %d 条,已清理远端文件 %d 个", result.TaskName, id, result.RecordCount, result.CleanedFiles))
response.Success(c, gin.H{"deleted": true})
}
@@ -105,5 +130,13 @@ func (h *BackupTaskHandler) Toggle(c *gin.Context) {
response.Error(c, err)
return
}
action := "enable"
actionLabel := "启用"
if !enabled {
action = "disable"
actionLabel = "停用"
}
recordAudit(c, h.auditService, "backup_task", action, "backup_task", fmt.Sprintf("%d", id), item.Name,
fmt.Sprintf("%s备份任务「%s」", actionLabel, item.Name))
response.Success(c, item)
}

View File

@@ -0,0 +1,30 @@
package http
import (
"backupx/server/internal/apperror"
"backupx/server/internal/service"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
type DatabaseHandler struct {
service *service.DatabaseDiscoveryService
}
func NewDatabaseHandler(service *service.DatabaseDiscoveryService) *DatabaseHandler {
return &DatabaseHandler{service: service}
}
func (h *DatabaseHandler) Discover(c *gin.Context) {
var input service.DatabaseDiscoverInput
if err := c.ShouldBindJSON(&input); err != nil {
response.Error(c, apperror.BadRequest("DATABASE_DISCOVER_INVALID", "数据库发现参数不合法", err))
return
}
result, err := h.service.Discover(c.Request.Context(), input)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, result)
}

View File

@@ -1,6 +1,7 @@
package http
import (
"fmt"
stdhttp "net/http"
"strconv"
@@ -10,11 +11,12 @@ import (
)
type NodeHandler struct {
service *service.NodeService
service *service.NodeService
auditService *service.AuditService
}
func NewNodeHandler(service *service.NodeService) *NodeHandler {
return &NodeHandler{service: service}
func NewNodeHandler(service *service.NodeService, auditService *service.AuditService) *NodeHandler {
return &NodeHandler{service: service, auditService: auditService}
}
func (h *NodeHandler) List(c *gin.Context) {
@@ -51,6 +53,8 @@ func (h *NodeHandler) Create(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "node", "create", "node", "", input.Name,
fmt.Sprintf("创建远程节点「%s」", input.Name))
response.Success(c, gin.H{"token": token})
}
@@ -64,6 +68,8 @@ func (h *NodeHandler) Delete(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "node", "delete", "node", fmt.Sprintf("%d", id), "",
fmt.Sprintf("删除节点 (ID: %d)", id))
response.Success(c, nil)
}
@@ -82,18 +88,41 @@ func (h *NodeHandler) ListDirectory(c *gin.Context) {
response.Success(c, entries)
}
func (h *NodeHandler) Update(c *gin.Context) {
id, err := strconv.ParseUint(c.Param("id"), 10, 32)
if err != nil {
response.Error(c, err)
return
}
var input service.NodeUpdateInput
if err := c.ShouldBindJSON(&input); err != nil {
c.JSON(stdhttp.StatusBadRequest, gin.H{"code": "INVALID_INPUT", "message": err.Error()})
return
}
item, err := h.service.Update(c.Request.Context(), uint(id), input)
if err != nil {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "node", "update", "node", fmt.Sprintf("%d", id), item.Name,
fmt.Sprintf("更新节点「%s」(ID: %d)", item.Name, id))
response.Success(c, item)
}
func (h *NodeHandler) Heartbeat(c *gin.Context) {
var input struct {
Token string `json:"token" binding:"required"`
Hostname string `json:"hostname"`
IPAddress string `json:"ipAddress"`
AgentVersion string `json:"agentVersion"`
OS string `json:"os"`
Arch string `json:"arch"`
}
if err := c.ShouldBindJSON(&input); err != nil {
c.JSON(stdhttp.StatusBadRequest, gin.H{"code": "INVALID_INPUT", "message": err.Error()})
return
}
if err := h.service.Heartbeat(c.Request.Context(), input.Token, input.Hostname, input.IPAddress, input.AgentVersion); err != nil {
if err := h.service.Heartbeat(c.Request.Context(), input.Token, input.Hostname, input.IPAddress, input.AgentVersion, input.OS, input.Arch); err != nil {
response.Error(c, err)
return
}

View File

@@ -0,0 +1,21 @@
package http
import (
storageRclone "backupx/server/internal/storage/rclone"
"backupx/server/pkg/response"
"github.com/gin-gonic/gin"
)
// RcloneHandler 处理 rclone 后端元数据查询。
type RcloneHandler struct{}
func NewRcloneHandler() *RcloneHandler {
return &RcloneHandler{}
}
// ListBackends 返回所有可用的 rclone 后端及其配置选项。
func (h *RcloneHandler) ListBackends(c *gin.Context) {
backends := storageRclone.ListBackends()
response.Success(c, backends)
}

View File

@@ -15,22 +15,25 @@ import (
)
type RouterDependencies struct {
Config config.Config
Version string
Logger *zap.Logger
AuthService *service.AuthService
SystemService *service.SystemService
StorageTargetService *service.StorageTargetService
BackupTaskService *service.BackupTaskService
BackupExecutionService *service.BackupExecutionService
BackupRecordService *service.BackupRecordService
NotificationService *service.NotificationService
DashboardService *service.DashboardService
SettingsService *service.SettingsService
NodeService *service.NodeService
JWTManager *security.JWTManager
UserRepository repository.UserRepository
SystemConfigRepo repository.SystemConfigRepository
Config config.Config
Version string
Logger *zap.Logger
AuthService *service.AuthService
SystemService *service.SystemService
StorageTargetService *service.StorageTargetService
BackupTaskService *service.BackupTaskService
BackupExecutionService *service.BackupExecutionService
BackupRecordService *service.BackupRecordService
NotificationService *service.NotificationService
DashboardService *service.DashboardService
SettingsService *service.SettingsService
NodeService *service.NodeService
AgentService *service.AgentService
DatabaseDiscoveryService *service.DatabaseDiscoveryService
AuditService *service.AuditService
JWTManager *security.JWTManager
UserRepository repository.UserRepository
SystemConfigRepo repository.SystemConfigRepository
}
func NewRouter(deps RouterDependencies) *gin.Engine {
@@ -42,13 +45,14 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
authHandler := NewAuthHandler(deps.AuthService)
systemHandler := NewSystemHandler(deps.SystemService)
storageTargetHandler := NewStorageTargetHandler(deps.StorageTargetService)
backupTaskHandler := NewBackupTaskHandler(deps.BackupTaskService)
backupRunHandler := NewBackupRunHandler(deps.BackupExecutionService)
backupRecordHandler := NewBackupRecordHandler(deps.BackupRecordService)
storageTargetHandler := NewStorageTargetHandler(deps.StorageTargetService, deps.AuditService)
backupTaskHandler := NewBackupTaskHandler(deps.BackupTaskService, deps.AuditService)
backupRunHandler := NewBackupRunHandler(deps.BackupExecutionService, deps.AuditService)
backupRecordHandler := NewBackupRecordHandler(deps.BackupRecordService, deps.AuditService)
notificationHandler := NewNotificationHandler(deps.NotificationService)
dashboardHandler := NewDashboardHandler(deps.DashboardService)
settingsHandler := NewSettingsHandler(deps.SettingsService)
settingsHandler := NewSettingsHandler(deps.SettingsService, deps.AuditService)
auditHandler := NewAuditHandler(deps.AuditService)
api := engine.Group("/api")
{
@@ -65,20 +69,26 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
system := api.Group("/system")
system.Use(AuthMiddleware(deps.JWTManager))
system.GET("/info", systemHandler.Info)
system.GET("/update-check", systemHandler.CheckUpdate)
storageTargets := api.Group("/storage-targets")
storageTargets.Use(AuthMiddleware(deps.JWTManager))
// 静态路由必须在参数路由 /:id 之前注册,避免 Gin 路由冲突
storageTargets.GET("", storageTargetHandler.List)
storageTargets.GET("/:id", storageTargetHandler.Get)
storageTargets.POST("", storageTargetHandler.Create)
storageTargets.PUT("/:id", storageTargetHandler.Update)
storageTargets.DELETE("/:id", storageTargetHandler.Delete)
storageTargets.POST("/test", storageTargetHandler.TestConnection)
storageTargets.POST("/:id/test", storageTargetHandler.TestSavedConnection)
storageTargets.GET("/:id/usage", storageTargetHandler.GetUsage)
storageTargets.POST("/google-drive/auth-url", storageTargetHandler.StartGoogleDriveOAuth)
storageTargets.POST("/google-drive/complete", storageTargetHandler.CompleteGoogleDriveOAuth)
storageTargets.GET("/google-drive/callback", storageTargetHandler.HandleGoogleDriveCallback)
rcloneHandler := NewRcloneHandler()
storageTargets.GET("/rclone/backends", rcloneHandler.ListBackends)
// 参数路由
storageTargets.GET("/:id", storageTargetHandler.Get)
storageTargets.PUT("/:id", storageTargetHandler.Update)
storageTargets.DELETE("/:id", storageTargetHandler.Delete)
storageTargets.PUT("/:id/star", storageTargetHandler.ToggleStar)
storageTargets.POST("/:id/test", storageTargetHandler.TestSavedConnection)
storageTargets.GET("/:id/usage", storageTargetHandler.GetUsage)
storageTargets.GET("/:id/google-drive/profile", storageTargetHandler.GoogleDriveProfile)
backupTasks := api.Group("/backup/tasks")
@@ -98,6 +108,7 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
backupRecords.GET("/:id/logs/stream", backupRecordHandler.StreamLogs)
backupRecords.GET("/:id/download", backupRecordHandler.Download)
backupRecords.POST("/:id/restore", backupRecordHandler.Restore)
backupRecords.POST("/batch-delete", backupRecordHandler.BatchDelete)
backupRecords.DELETE("/:id", backupRecordHandler.Delete)
dashboard := api.Group("/dashboard")
dashboard.Use(AuthMiddleware(deps.JWTManager))
@@ -119,17 +130,40 @@ func NewRouter(deps RouterDependencies) *gin.Engine {
settings.GET("", settingsHandler.Get)
settings.PUT("", settingsHandler.Update)
nodeHandler := NewNodeHandler(deps.NodeService)
auditLogs := api.Group("/audit-logs")
auditLogs.Use(AuthMiddleware(deps.JWTManager))
auditLogs.GET("", auditHandler.List)
if deps.DatabaseDiscoveryService != nil {
databaseHandler := NewDatabaseHandler(deps.DatabaseDiscoveryService)
database := api.Group("/database")
database.Use(AuthMiddleware(deps.JWTManager))
database.POST("/discover", databaseHandler.Discover)
}
nodeHandler := NewNodeHandler(deps.NodeService, deps.AuditService)
nodes := api.Group("/nodes")
nodes.Use(AuthMiddleware(deps.JWTManager))
nodes.GET("", nodeHandler.List)
nodes.GET("/:id", nodeHandler.Get)
nodes.POST("", nodeHandler.Create)
nodes.PUT("/:id", nodeHandler.Update)
nodes.DELETE("/:id", nodeHandler.Delete)
nodes.GET("/:id/fs/list", nodeHandler.ListDirectory)
// Agent heartbeat (public, token-authenticated)
api.POST("/agent/heartbeat", nodeHandler.Heartbeat)
// Agent APItoken 认证,无需 JWT
if deps.AgentService != nil {
agentHandler := NewAgentHandler(deps.AgentService, deps.NodeService)
agent := api.Group("/agent")
agent.POST("/heartbeat", agentHandler.Heartbeat)
agent.POST("/commands/poll", agentHandler.Poll)
agent.POST("/commands/:id/result", agentHandler.SubmitCommandResult)
agent.GET("/tasks/:id", agentHandler.GetTaskSpec)
agent.POST("/records/:id", agentHandler.UpdateRecord)
} else {
// 未启用 Agent 服务时,保留原有 heartbeat 端点以兼容
api.POST("/agent/heartbeat", nodeHandler.Heartbeat)
}
}
engine.NoRoute(func(c *gin.Context) {

View File

@@ -1,6 +1,9 @@
package http
import (
"fmt"
"strings"
"backupx/server/internal/apperror"
"backupx/server/internal/service"
"backupx/server/pkg/response"
@@ -9,10 +12,11 @@ import (
type SettingsHandler struct {
settingsService *service.SettingsService
auditService *service.AuditService
}
func NewSettingsHandler(settingsService *service.SettingsService) *SettingsHandler {
return &SettingsHandler{settingsService: settingsService}
func NewSettingsHandler(settingsService *service.SettingsService, auditService *service.AuditService) *SettingsHandler {
return &SettingsHandler{settingsService: settingsService, auditService: auditService}
}
func (h *SettingsHandler) Get(c *gin.Context) {
@@ -35,5 +39,10 @@ func (h *SettingsHandler) Update(c *gin.Context) {
response.Error(c, err)
return
}
keys := make([]string, 0, len(input))
for k := range input {
keys = append(keys, k)
}
recordAudit(c, h.auditService, "settings", "update", "settings", "", "", fmt.Sprintf("修改设置: %s", strings.Join(keys, ", ")))
response.Success(c, settings)
}

View File

@@ -12,7 +12,8 @@ import (
)
type StorageTargetHandler struct {
service *service.StorageTargetService
service *service.StorageTargetService
auditService *service.AuditService
}
type storageTargetGoogleDriveAuthRequest struct {
@@ -27,8 +28,8 @@ type storageTargetGoogleDriveAuthRequest struct {
FolderID string `json:"folderId"`
}
func NewStorageTargetHandler(service *service.StorageTargetService) *StorageTargetHandler {
return &StorageTargetHandler{service: service}
func NewStorageTargetHandler(service *service.StorageTargetService, auditService *service.AuditService) *StorageTargetHandler {
return &StorageTargetHandler{service: service, auditService: auditService}
}
func (h *StorageTargetHandler) List(c *gin.Context) {
@@ -64,6 +65,8 @@ func (h *StorageTargetHandler) Create(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "storage_target", "create", "storage_target", fmt.Sprintf("%d", item.ID), item.Name,
fmt.Sprintf("创建存储目标「%s」类型: %s", item.Name, input.Type))
response.Success(c, item)
}
@@ -82,6 +85,8 @@ func (h *StorageTargetHandler) Update(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "storage_target", "update", "storage_target", fmt.Sprintf("%d", item.ID), item.Name,
fmt.Sprintf("更新存储目标「%s」类型: %s", item.Name, input.Type))
response.Success(c, item)
}
@@ -94,6 +99,8 @@ func (h *StorageTargetHandler) Delete(c *gin.Context) {
response.Error(c, err)
return
}
recordAudit(c, h.auditService, "storage_target", "delete", "storage_target", fmt.Sprintf("%d", id), "",
fmt.Sprintf("删除存储目标 (ID: %d)", id))
response.Success(c, gin.H{"deleted": true})
}
@@ -230,6 +237,19 @@ func firstNonEmpty(values ...string) string {
return ""
}
func (h *StorageTargetHandler) ToggleStar(c *gin.Context) {
id, ok := parseUintParam(c, "id")
if !ok {
return
}
item, err := h.service.ToggleStar(c.Request.Context(), id)
if err != nil {
response.Error(c, err)
return
}
response.Success(c, item)
}
func (h *StorageTargetHandler) GetUsage(c *gin.Context) {
id, ok := parseUintParam(c, "id")
if !ok {

View File

@@ -17,3 +17,17 @@ func NewSystemHandler(systemService *service.SystemService) *SystemHandler {
func (h *SystemHandler) Info(c *gin.Context) {
response.Success(c, h.systemService.GetInfo(c.Request.Context()))
}
func (h *SystemHandler) CheckUpdate(c *gin.Context) {
result, err := h.systemService.CheckUpdate(c.Request.Context())
if err != nil {
// 即使检查失败也返回当前版本信息
response.Success(c, gin.H{
"currentVersion": result.CurrentVersion,
"hasUpdate": false,
"error": err.Error(),
})
return
}
response.Success(c, result)
}

View File

@@ -0,0 +1,44 @@
package model
import "time"
// AgentCommand 状态常量
const (
AgentCommandStatusPending = "pending" // 待 Agent 拉取
AgentCommandStatusDispatched = "dispatched" // Agent 已领取,正在执行
AgentCommandStatusSucceeded = "succeeded" // 执行成功
AgentCommandStatusFailed = "failed" // 执行失败
AgentCommandStatusTimeout = "timeout" // 超时未完成
)
// AgentCommand 类型常量
const (
// AgentCommandTypeRunTask 运行指定备份任务
// Payload: {"taskId": 123, "recordId": 456}
AgentCommandTypeRunTask = "run_task"
// AgentCommandTypeListDir 远程列目录(用于文件备份时的目录浏览器)
// Payload: {"path": "/var/log"}
// Result: {"entries": [{"name":"...", "path":"...", "isDir":true, "size":0}]}
AgentCommandTypeListDir = "list_dir"
)
// AgentCommand 代表 Master 发给某个 Agent 节点的待执行命令。
// 使用简单的数据库队列实现Agent 通过 token 长轮询拉取本节点 pending 命令,
// 执行后回写状态与结果。Master 侧通过定时检查把超时的命令标记为 timeout。
type AgentCommand struct {
ID uint `gorm:"primaryKey" json:"id"`
NodeID uint `gorm:"column:node_id;index;not null" json:"nodeId"`
Type string `gorm:"size:32;index;not null" json:"type"`
Status string `gorm:"size:20;index;not null;default:'pending'" json:"status"`
Payload string `gorm:"type:text" json:"payload"` // JSON
Result string `gorm:"type:text" json:"result"` // JSON成功结果
ErrorMessage string `gorm:"column:error_message;type:text" json:"errorMessage"`
DispatchedAt *time.Time `gorm:"column:dispatched_at" json:"dispatchedAt,omitempty"`
CompletedAt *time.Time `gorm:"column:completed_at" json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (AgentCommand) TableName() string {
return "agent_commands"
}

View File

@@ -0,0 +1,21 @@
package model
import "time"
type AuditLog struct {
ID uint `gorm:"primaryKey" json:"id"`
UserID uint `gorm:"column:user_id;index" json:"userId"`
Username string `gorm:"column:username;size:64;not null" json:"username"`
Category string `gorm:"column:category;size:32;index;not null" json:"category"`
Action string `gorm:"column:action;size:64;not null" json:"action"`
TargetType string `gorm:"column:target_type;size:32" json:"targetType"`
TargetID string `gorm:"column:target_id;size:64" json:"targetId"`
TargetName string `gorm:"column:target_name;size:128" json:"targetName"`
Detail string `gorm:"column:detail;type:text" json:"detail"`
ClientIP string `gorm:"column:client_ip;size:45" json:"clientIp"`
CreatedAt time.Time `gorm:"index" json:"createdAt"`
}
func (AuditLog) TableName() string {
return "audit_logs"
}

View File

@@ -9,22 +9,24 @@ const (
)
type BackupRecord struct {
ID uint `gorm:"primaryKey" json:"id"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
Status string `gorm:"size:20;index;not null" json:"status"`
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ID uint `gorm:"primaryKey" json:"id"`
TaskID uint `gorm:"column:task_id;index;not null" json:"taskId"`
Task BackupTask `json:"task,omitempty"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
Status string `gorm:"size:20;index;not null" json:"status"`
FileName string `gorm:"column:file_name;size:255" json:"fileName"`
FileSize int64 `gorm:"column:file_size;not null;default:0" json:"fileSize"`
Checksum string `gorm:"column:checksum;size:64" json:"checksum"`
StoragePath string `gorm:"column:storage_path;size:500" json:"storagePath"`
StorageUploadResults string `gorm:"column:storage_upload_results;type:text" json:"-"`
DurationSeconds int `gorm:"column:duration_seconds;not null;default:0" json:"durationSeconds"`
ErrorMessage string `gorm:"column:error_message;size:2000" json:"errorMessage"`
LogContent string `gorm:"column:log_content;type:text" json:"logContent"`
StartedAt time.Time `gorm:"column:started_at;index;not null" json:"startedAt"`
CompletedAt *time.Time `gorm:"column:completed_at;index" json:"completedAt,omitempty"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (BackupRecord) TableName() string {

View File

@@ -7,6 +7,7 @@ const (
BackupTaskTypeMySQL = "mysql"
BackupTaskTypeSQLite = "sqlite"
BackupTaskTypePostgreSQL = "postgresql"
BackupTaskTypeSAPHANA = "saphana"
)
const (
@@ -17,34 +18,48 @@ const (
)
type BackupTask struct {
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
Type string `gorm:"size:20;index;not null" json:"type"`
Enabled bool `gorm:"not null;default:true" json:"enabled"`
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
DBPort int `gorm:"column:db_port" json:"dbPort"`
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"`
StorageTarget StorageTarget `json:"storageTarget,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Node Node `json:"node,omitempty"`
Tags string `gorm:"column:tags;size:500" json:"tags"`
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ID uint `gorm:"primaryKey" json:"id"`
Name string `gorm:"size:100;uniqueIndex;not null" json:"name"`
Type string `gorm:"size:20;index;not null" json:"type"`
Enabled bool `gorm:"not null;default:true" json:"enabled"`
CronExpr string `gorm:"column:cron_expr;size:64" json:"cronExpr"`
SourcePath string `gorm:"column:source_path;size:500" json:"sourcePath"`
SourcePaths string `gorm:"column:source_paths;type:text" json:"sourcePaths"`
ExcludePatterns string `gorm:"column:exclude_patterns;type:text" json:"excludePatterns"`
DBHost string `gorm:"column:db_host;size:255" json:"dbHost"`
DBPort int `gorm:"column:db_port" json:"dbPort"`
DBUser string `gorm:"column:db_user;size:100" json:"dbUser"`
DBPasswordCiphertext string `gorm:"column:db_password_ciphertext;type:text" json:"-"`
DBName string `gorm:"column:db_name;size:255" json:"dbName"`
DBPath string `gorm:"column:db_path;size:500" json:"dbPath"`
// ExtraConfig 类型特有的扩展配置JSON如 SAP HANA 的 backupLevel / backupChannels 等
ExtraConfig string `gorm:"column:extra_config;type:text" json:"extraConfig"`
StorageTargetID uint `gorm:"column:storage_target_id;index;not null" json:"storageTargetId"` // deprecated: 保留兼容
StorageTarget StorageTarget `json:"storageTarget,omitempty"` // deprecated: 保留兼容
StorageTargets []StorageTarget `gorm:"many2many:backup_task_storage_targets" json:"storageTargets,omitempty"`
NodeID uint `gorm:"column:node_id;index;default:0" json:"nodeId"`
Node Node `json:"node,omitempty"`
Tags string `gorm:"column:tags;size:500" json:"tags"`
RetentionDays int `gorm:"column:retention_days;not null;default:30" json:"retentionDays"`
Compression string `gorm:"size:10;not null;default:'gzip'" json:"compression"`
Encrypt bool `gorm:"not null;default:false" json:"encrypt"`
MaxBackups int `gorm:"column:max_backups;not null;default:10" json:"maxBackups"`
LastRunAt *time.Time `gorm:"column:last_run_at" json:"lastRunAt,omitempty"`
LastStatus string `gorm:"column:last_status;size:20;not null;default:'idle'" json:"lastStatus"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (BackupTask) TableName() string {
return "backup_tasks"
}
// BackupTaskStorageTarget 多对多中间表
type BackupTaskStorageTarget struct {
BackupTaskID uint `gorm:"primaryKey;column:backup_task_id"`
StorageTargetID uint `gorm:"primaryKey;column:storage_target_id"`
}
func (BackupTaskStorageTarget) TableName() string {
return "backup_task_storage_targets"
}

View File

@@ -8,6 +8,7 @@ type StorageTarget struct {
Type string `gorm:"size:32;index;not null" json:"type"`
Description string `gorm:"size:255" json:"description"`
Enabled bool `gorm:"not null;default:true" json:"enabled"`
Starred bool `gorm:"not null;default:false" json:"starred"`
ConfigCiphertext string `gorm:"column:config_ciphertext;type:text;not null" json:"-"`
ConfigVersion int `gorm:"not null;default:1" json:"configVersion"`
LastTestedAt *time.Time `gorm:"column:last_tested_at" json:"lastTestedAt,omitempty"`

View File

@@ -0,0 +1,101 @@
package repository
import (
"context"
"errors"
"time"
"backupx/server/internal/model"
"gorm.io/gorm"
)
// AgentCommandRepository 维护 Agent 命令队列。
type AgentCommandRepository interface {
Create(ctx context.Context, cmd *model.AgentCommand) error
FindByID(ctx context.Context, id uint) (*model.AgentCommand, error)
// ClaimPending 以原子方式把该节点一条 pending 命令置为 dispatched
// 并返回领取到的命令。无命令时返回 (nil, nil)。
ClaimPending(ctx context.Context, nodeID uint) (*model.AgentCommand, error)
Update(ctx context.Context, cmd *model.AgentCommand) error
// MarkStaleTimeout 把 dispatched 状态但超时未完成的命令标记为 timeout。
MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error)
}
type GormAgentCommandRepository struct {
db *gorm.DB
}
func NewAgentCommandRepository(db *gorm.DB) *GormAgentCommandRepository {
return &GormAgentCommandRepository{db: db}
}
func (r *GormAgentCommandRepository) Create(ctx context.Context, cmd *model.AgentCommand) error {
return r.db.WithContext(ctx).Create(cmd).Error
}
func (r *GormAgentCommandRepository) FindByID(ctx context.Context, id uint) (*model.AgentCommand, error) {
var item model.AgentCommand
if err := r.db.WithContext(ctx).First(&item, id).Error; err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil, nil
}
return nil, err
}
return &item, nil
}
// ClaimPending 使用 UPDATE...WHERE id=(SELECT...) 的两步方式实现原子领取。
// SQLite 不支持 SELECT FOR UPDATE这里用事务 + 乐观锁。
func (r *GormAgentCommandRepository) ClaimPending(ctx context.Context, nodeID uint) (*model.AgentCommand, error) {
var claimed *model.AgentCommand
err := r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
var item model.AgentCommand
err := tx.Where("node_id = ? AND status = ?", nodeID, model.AgentCommandStatusPending).
Order("id asc").First(&item).Error
if err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil
}
return err
}
now := time.Now().UTC()
result := tx.Model(&model.AgentCommand{}).
Where("id = ? AND status = ?", item.ID, model.AgentCommandStatusPending).
Updates(map[string]any{
"status": model.AgentCommandStatusDispatched,
"dispatched_at": &now,
})
if result.Error != nil {
return result.Error
}
if result.RowsAffected == 0 {
// 被其它 worker 抢占,放弃
return nil
}
item.Status = model.AgentCommandStatusDispatched
item.DispatchedAt = &now
claimed = &item
return nil
})
if err != nil {
return nil, err
}
return claimed, nil
}
func (r *GormAgentCommandRepository) Update(ctx context.Context, cmd *model.AgentCommand) error {
return r.db.WithContext(ctx).Save(cmd).Error
}
func (r *GormAgentCommandRepository) MarkStaleTimeout(ctx context.Context, threshold time.Time) (int64, error) {
result := r.db.WithContext(ctx).Model(&model.AgentCommand{}).
Where("status = ? AND dispatched_at < ?", model.AgentCommandStatusDispatched, threshold).
Updates(map[string]any{
"status": model.AgentCommandStatusTimeout,
"error_message": "agent did not report result before timeout",
})
if result.Error != nil {
return 0, result.Error
}
return result.RowsAffected, nil
}

View File

@@ -0,0 +1,120 @@
package repository
import (
"context"
"testing"
"time"
"backupx/server/internal/model"
"github.com/glebarez/sqlite"
"gorm.io/gorm"
gormlogger "gorm.io/gorm/logger"
)
func newTestDB(t *testing.T) *gorm.DB {
t.Helper()
db, err := gorm.Open(sqlite.Open(":memory:"), &gorm.Config{Logger: gormlogger.Default.LogMode(gormlogger.Silent)})
if err != nil {
t.Fatalf("open: %v", err)
}
if err := db.AutoMigrate(&model.AgentCommand{}); err != nil {
t.Fatalf("migrate: %v", err)
}
return db
}
func TestAgentCommandRepository_ClaimPending(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
// 插入两条 pending 命令
c1 := &model.AgentCommand{NodeID: 5, Type: "run_task", Status: model.AgentCommandStatusPending, Payload: `{"taskId":1}`}
c2 := &model.AgentCommand{NodeID: 5, Type: "list_dir", Status: model.AgentCommandStatusPending, Payload: `{"path":"/"}`}
c3 := &model.AgentCommand{NodeID: 99, Type: "run_task", Status: model.AgentCommandStatusPending}
for _, c := range []*model.AgentCommand{c1, c2, c3} {
if err := repo.Create(ctx, c); err != nil {
t.Fatal(err)
}
}
// 第一次 Claim 应拿到 c1
claimed, err := repo.ClaimPending(ctx, 5)
if err != nil {
t.Fatalf("claim: %v", err)
}
if claimed == nil || claimed.ID != c1.ID || claimed.Status != model.AgentCommandStatusDispatched {
t.Fatalf("expected c1 dispatched: %+v", claimed)
}
// 第二次应拿到 c2
claimed2, err := repo.ClaimPending(ctx, 5)
if err != nil || claimed2 == nil || claimed2.ID != c2.ID {
t.Fatalf("expected c2: %+v %v", claimed2, err)
}
// 第三次无 pending返回 nil
claimed3, err := repo.ClaimPending(ctx, 5)
if err != nil || claimed3 != nil {
t.Fatalf("expected nil, got %+v", claimed3)
}
// 不同 node 的命令不应被抢到
other, err := repo.ClaimPending(ctx, 5)
if err != nil || other != nil {
t.Fatalf("expected nil: %+v", other)
}
}
func TestAgentCommandRepository_Update(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
cmd := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusPending}
_ = repo.Create(ctx, cmd)
cmd.Status = model.AgentCommandStatusSucceeded
cmd.Result = `{"ok":true}`
now := time.Now().UTC()
cmd.CompletedAt = &now
if err := repo.Update(ctx, cmd); err != nil {
t.Fatal(err)
}
got, err := repo.FindByID(ctx, cmd.ID)
if err != nil || got == nil {
t.Fatal(err)
}
if got.Status != model.AgentCommandStatusSucceeded || got.Result != `{"ok":true}` {
t.Errorf("mismatch: %+v", got)
}
}
func TestAgentCommandRepository_MarkStaleTimeout(t *testing.T) {
db := newTestDB(t)
repo := NewAgentCommandRepository(db)
ctx := context.Background()
old := time.Now().Add(-time.Hour)
recent := time.Now()
// 两条 dispatched一条旧、一条新
oldCmd := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusDispatched, DispatchedAt: &old}
newCmd := &model.AgentCommand{NodeID: 1, Type: "run_task", Status: model.AgentCommandStatusDispatched, DispatchedAt: &recent}
_ = repo.Create(ctx, oldCmd)
_ = repo.Create(ctx, newCmd)
n, err := repo.MarkStaleTimeout(ctx, time.Now().Add(-30*time.Minute))
if err != nil {
t.Fatal(err)
}
if n != 1 {
t.Errorf("expected 1 row, got %d", n)
}
oldGot, _ := repo.FindByID(ctx, oldCmd.ID)
newGot, _ := repo.FindByID(ctx, newCmd.ID)
if oldGot.Status != model.AgentCommandStatusTimeout {
t.Errorf("old should be timeout: %+v", oldGot)
}
if newGot.Status != model.AgentCommandStatusDispatched {
t.Errorf("new should stay dispatched: %+v", newGot)
}
}

View File

@@ -0,0 +1,56 @@
package repository
import (
"context"
"backupx/server/internal/model"
"gorm.io/gorm"
)
type AuditLogListOptions struct {
Category string
Limit int
Offset int
}
type AuditLogListResult struct {
Items []model.AuditLog `json:"items"`
Total int64 `json:"total"`
}
type AuditLogRepository interface {
Create(ctx context.Context, log *model.AuditLog) error
List(ctx context.Context, opts AuditLogListOptions) (*AuditLogListResult, error)
}
type gormAuditLogRepository struct {
db *gorm.DB
}
func NewAuditLogRepository(db *gorm.DB) AuditLogRepository {
return &gormAuditLogRepository{db: db}
}
func (r *gormAuditLogRepository) Create(_ context.Context, log *model.AuditLog) error {
return r.db.Create(log).Error
}
func (r *gormAuditLogRepository) List(_ context.Context, opts AuditLogListOptions) (*AuditLogListResult, error) {
query := r.db.Model(&model.AuditLog{})
if opts.Category != "" {
query = query.Where("category = ?", opts.Category)
}
var total int64
if err := query.Count(&total).Error; err != nil {
return nil, err
}
limit := opts.Limit
if limit <= 0 {
limit = 50
}
var items []model.AuditLog
if err := query.Order("created_at DESC").Offset(opts.Offset).Limit(limit).Find(&items).Error; err != nil {
return nil, err
}
return &AuditLogListResult{Items: items, Total: total}, nil
}

View File

@@ -37,6 +37,7 @@ type BackupRecordRepository interface {
Update(context.Context, *model.BackupRecord) error
Delete(context.Context, uint) error
ListRecent(context.Context, int) ([]model.BackupRecord, error)
ListByTask(context.Context, uint) ([]model.BackupRecord, error)
ListSuccessfulByTask(context.Context, uint) ([]model.BackupRecord, error)
Count(context.Context) (int64, error)
CountSince(context.Context, time.Time) (int64, error)
@@ -115,6 +116,14 @@ func (r *GormBackupRecordRepository) ListRecent(ctx context.Context, limit int)
return items, nil
}
func (r *GormBackupRecordRepository) ListByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
var items []model.BackupRecord
if err := r.db.WithContext(ctx).Where("task_id = ?", taskID).Order("id desc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil
}
func (r *GormBackupRecordRepository) ListSuccessfulByTask(ctx context.Context, taskID uint) ([]model.BackupRecord, error) {
var items []model.BackupRecord
if err := r.db.WithContext(ctx).Where("task_id = ? AND status = ?", taskID, "success").Order("completed_at desc, id desc").Find(&items).Error; err != nil {

View File

@@ -21,6 +21,8 @@ type BackupTaskRepository interface {
Count(context.Context) (int64, error)
CountEnabled(context.Context) (int64, error)
CountByStorageTargetID(context.Context, uint) (int64, error)
CountByNodeID(context.Context, uint) (int64, error)
ListByNodeID(context.Context, uint) ([]model.BackupTask, error)
Create(context.Context, *model.BackupTask) error
Update(context.Context, *model.BackupTask) error
Delete(context.Context, uint) error
@@ -35,7 +37,7 @@ func NewBackupTaskRepository(db *gorm.DB) *GormBackupTaskRepository {
}
func (r *GormBackupTaskRepository) List(ctx context.Context, options BackupTaskListOptions) ([]model.BackupTask, error) {
query := r.db.WithContext(ctx).Model(&model.BackupTask{}).Preload("StorageTarget").Order("updated_at desc")
query := r.db.WithContext(ctx).Model(&model.BackupTask{}).Preload("StorageTarget").Preload("StorageTargets").Order("updated_at desc")
if options.Type != "" {
query = query.Where("type = ?", options.Type)
}
@@ -51,7 +53,7 @@ func (r *GormBackupTaskRepository) List(ctx context.Context, options BackupTaskL
func (r *GormBackupTaskRepository) FindByID(ctx context.Context, id uint) (*model.BackupTask, error) {
var item model.BackupTask
if err := r.db.WithContext(ctx).Preload("StorageTarget").First(&item, id).Error; err != nil {
if err := r.db.WithContext(ctx).Preload("StorageTarget").Preload("StorageTargets").First(&item, id).Error; err != nil {
if errors.Is(err, gorm.ErrRecordNotFound) {
return nil, nil
}
@@ -73,7 +75,7 @@ func (r *GormBackupTaskRepository) FindByName(ctx context.Context, name string)
func (r *GormBackupTaskRepository) ListSchedulable(ctx context.Context) ([]model.BackupTask, error) {
var items []model.BackupTask
if err := r.db.WithContext(ctx).Preload("StorageTarget").Where("enabled = ? AND cron_expr <> ''", true).Order("id asc").Find(&items).Error; err != nil {
if err := r.db.WithContext(ctx).Preload("StorageTarget").Preload("StorageTargets").Where("enabled = ? AND cron_expr <> ''", true).Order("id asc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil
@@ -97,18 +99,57 @@ func (r *GormBackupTaskRepository) CountEnabled(ctx context.Context) (int64, err
func (r *GormBackupTaskRepository) CountByStorageTargetID(ctx context.Context, storageTargetID uint) (int64, error) {
var count int64
if err := r.db.WithContext(ctx).Model(&model.BackupTask{}).Where("storage_target_id = ?", storageTargetID).Count(&count).Error; err != nil {
if err := r.db.WithContext(ctx).Model(&model.BackupTaskStorageTarget{}).Where("storage_target_id = ?", storageTargetID).Count(&count).Error; err != nil {
return 0, err
}
return count, nil
}
// CountByNodeID 统计绑定到指定节点的任务数。用于删除节点前的引用检查。
func (r *GormBackupTaskRepository) CountByNodeID(ctx context.Context, nodeID uint) (int64, error) {
var count int64
if err := r.db.WithContext(ctx).Model(&model.BackupTask{}).Where("node_id = ?", nodeID).Count(&count).Error; err != nil {
return 0, err
}
return count, nil
}
// ListByNodeID 列出绑定到指定节点的任务。用于 Agent 拉取本节点待执行任务。
func (r *GormBackupTaskRepository) ListByNodeID(ctx context.Context, nodeID uint) ([]model.BackupTask, error) {
var items []model.BackupTask
if err := r.db.WithContext(ctx).Preload("StorageTarget").Preload("StorageTargets").Where("node_id = ?", nodeID).Order("id asc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil
}
func (r *GormBackupTaskRepository) Create(ctx context.Context, item *model.BackupTask) error {
return r.db.WithContext(ctx).Create(item).Error
if err := r.db.WithContext(ctx).Create(item).Error; err != nil {
return err
}
return r.syncStorageTargets(ctx, item)
}
func (r *GormBackupTaskRepository) Update(ctx context.Context, item *model.BackupTask) error {
return r.db.WithContext(ctx).Save(item).Error
if err := r.db.WithContext(ctx).Save(item).Error; err != nil {
return err
}
if len(item.StorageTargets) > 0 {
return r.db.WithContext(ctx).Model(item).Association("StorageTargets").Replace(item.StorageTargets)
}
return nil
}
// syncStorageTargets 确保中间表数据一致:优先使用 StorageTargets回退到 StorageTargetID
func (r *GormBackupTaskRepository) syncStorageTargets(ctx context.Context, item *model.BackupTask) error {
targets := item.StorageTargets
if len(targets) == 0 && item.StorageTargetID > 0 {
targets = []model.StorageTarget{{ID: item.StorageTargetID}}
}
if len(targets) > 0 {
return r.db.WithContext(ctx).Model(item).Association("StorageTargets").Replace(targets)
}
return nil
}
func (r *GormBackupTaskRepository) Delete(ctx context.Context, id uint) error {

View File

@@ -3,6 +3,7 @@ package repository
import (
"context"
"errors"
"time"
"backupx/server/internal/model"
"gorm.io/gorm"
@@ -16,6 +17,7 @@ type NodeRepository interface {
Create(context.Context, *model.Node) error
Update(context.Context, *model.Node) error
Delete(context.Context, uint) error
MarkStaleOffline(ctx context.Context, threshold time.Time) (int64, error)
}
type GormNodeRepository struct {
@@ -78,3 +80,16 @@ func (r *GormNodeRepository) Update(ctx context.Context, item *model.Node) error
func (r *GormNodeRepository) Delete(ctx context.Context, id uint) error {
return r.db.WithContext(ctx).Delete(&model.Node{}, id).Error
}
// MarkStaleOffline 把最近心跳早于 threshold 的在线远程节点标记为离线。
// 本机节点 (is_local=true) 不受影响,由主程序自己维护 online 状态。
// 返回受影响行数。
func (r *GormNodeRepository) MarkStaleOffline(ctx context.Context, threshold time.Time) (int64, error) {
result := r.db.WithContext(ctx).Model(&model.Node{}).
Where("is_local = ? AND status = ? AND last_seen < ?", false, model.NodeStatusOnline, threshold).
Update("status", model.NodeStatusOffline)
if result.Error != nil {
return 0, result.Error
}
return result.RowsAffected, nil
}

View File

@@ -27,7 +27,7 @@ func NewStorageTargetRepository(db *gorm.DB) *GormStorageTargetRepository {
func (r *GormStorageTargetRepository) List(ctx context.Context) ([]model.StorageTarget, error) {
var items []model.StorageTarget
if err := r.db.WithContext(ctx).Order("updated_at desc").Find(&items).Error; err != nil {
if err := r.db.WithContext(ctx).Order("starred desc, updated_at desc").Find(&items).Error; err != nil {
return nil, err
}
return items, nil

View File

@@ -17,12 +17,18 @@ type TaskRunner interface {
RunTaskByID(context.Context, uint) (*servicepkg.BackupRecordDetail, error)
}
// AuditRecorder 记录审计日志(可选依赖)
type AuditRecorder interface {
Record(servicepkg.AuditEntry)
}
type Service struct {
mu sync.Mutex
cron *cron.Cron
tasks repository.BackupTaskRepository
runner TaskRunner
logger *zap.Logger
audit AuditRecorder
entries map[uint]cron.EntryID
}
@@ -31,6 +37,8 @@ func NewService(tasks repository.BackupTaskRepository, runner TaskRunner, logger
return &Service{cron: cron.New(cron.WithParser(parser), cron.WithLocation(time.UTC)), tasks: tasks, runner: runner, logger: logger, entries: make(map[uint]cron.EntryID)}
}
func (s *Service) SetAuditRecorder(audit AuditRecorder) { s.audit = audit }
func (s *Service) Start(ctx context.Context) error {
if err := s.Reload(ctx); err != nil {
return err
@@ -96,9 +104,19 @@ func (s *Service) syncTaskLocked(task *model.BackupTask) error {
if !task.Enabled || task.CronExpr == "" {
return nil
}
taskID := task.ID
taskName := task.Name
entryID, err := s.cron.AddFunc(task.CronExpr, func() {
if _, runErr := s.runner.RunTaskByID(context.Background(), task.ID); runErr != nil && s.logger != nil {
s.logger.Warn("scheduled backup run failed", zap.Uint("task_id", task.ID), zap.Error(runErr))
// 自动调度任务记录审计日志
if s.audit != nil {
s.audit.Record(servicepkg.AuditEntry{
Username: "system", Category: "backup_task", Action: "scheduled_run",
TargetType: "backup_task", TargetID: fmt.Sprintf("%d", taskID),
TargetName: taskName, Detail: fmt.Sprintf("定时调度触发备份任务: %s (cron: %s)", taskName, task.CronExpr),
})
}
if _, runErr := s.runner.RunTaskByID(context.Background(), taskID); runErr != nil && s.logger != nil {
s.logger.Warn("scheduled backup run failed", zap.Uint("task_id", taskID), zap.Error(runErr))
}
})
if err != nil {

View File

@@ -31,6 +31,12 @@ func (r *fakeTaskRepository) CountEnabled(context.Context) (int64, error) { retu
func (r *fakeTaskRepository) CountByStorageTargetID(context.Context, uint) (int64, error) {
return 0, nil
}
func (r *fakeTaskRepository) CountByNodeID(context.Context, uint) (int64, error) {
return 0, nil
}
func (r *fakeTaskRepository) ListByNodeID(context.Context, uint) ([]model.BackupTask, error) {
return nil, nil
}
func (r *fakeTaskRepository) Create(context.Context, *model.BackupTask) error { return nil }
func (r *fakeTaskRepository) Update(context.Context, *model.BackupTask) error { return nil }
func (r *fakeTaskRepository) Delete(context.Context, uint) error { return nil }

View File

@@ -0,0 +1,348 @@
package service
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"backupx/server/internal/apperror"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage/codec"
)
// AgentService 实现 Master 端 Agent 协议,提供给远程 Agent 通过 HTTP 调用。
// 所有方法使用 Agent Token 进行节点认证,避免暴露 JWT 给 Agent。
type AgentService struct {
nodeRepo repository.NodeRepository
taskRepo repository.BackupTaskRepository
recordRepo repository.BackupRecordRepository
storageRepo repository.StorageTargetRepository
cmdRepo repository.AgentCommandRepository
cipher *codec.ConfigCipher
}
func NewAgentService(
nodeRepo repository.NodeRepository,
taskRepo repository.BackupTaskRepository,
recordRepo repository.BackupRecordRepository,
storageRepo repository.StorageTargetRepository,
cmdRepo repository.AgentCommandRepository,
cipher *codec.ConfigCipher,
) *AgentService {
return &AgentService{
nodeRepo: nodeRepo,
taskRepo: taskRepo,
recordRepo: recordRepo,
storageRepo: storageRepo,
cmdRepo: cmdRepo,
cipher: cipher,
}
}
// AuthenticatedNode 通过 token 解析并返回节点。失败返回 401。
func (s *AgentService) AuthenticatedNode(ctx context.Context, token string) (*model.Node, error) {
if strings.TrimSpace(token) == "" {
return nil, apperror.Unauthorized("NODE_INVALID_TOKEN", "缺少认证令牌", nil)
}
node, err := s.nodeRepo.FindByToken(ctx, token)
if err != nil {
return nil, err
}
if node == nil {
return nil, apperror.Unauthorized("NODE_INVALID_TOKEN", "无效的节点认证令牌", nil)
}
return node, nil
}
// AgentCommandPayload 给 Agent 返回的命令描述
type AgentCommandPayload struct {
ID uint `json:"id"`
Type string `json:"type"`
Payload json.RawMessage `json:"payload,omitempty"`
}
// PollCommand 为指定节点拉取一条 pending 命令;无命令时返回 (nil, nil)。
func (s *AgentService) PollCommand(ctx context.Context, node *model.Node) (*AgentCommandPayload, error) {
cmd, err := s.cmdRepo.ClaimPending(ctx, node.ID)
if err != nil {
return nil, err
}
if cmd == nil {
return nil, nil
}
return &AgentCommandPayload{
ID: cmd.ID,
Type: cmd.Type,
Payload: json.RawMessage(cmd.Payload),
}, nil
}
// AgentCommandResult Agent 上报命令执行结果
type AgentCommandResult struct {
Success bool `json:"success"`
ErrorMessage string `json:"errorMessage,omitempty"`
Result json.RawMessage `json:"result,omitempty"`
}
// SubmitCommandResult 接收 Agent 上报的命令结果。
func (s *AgentService) SubmitCommandResult(ctx context.Context, node *model.Node, cmdID uint, result AgentCommandResult) error {
cmd, err := s.cmdRepo.FindByID(ctx, cmdID)
if err != nil {
return err
}
if cmd == nil {
return apperror.New(404, "AGENT_COMMAND_NOT_FOUND", "命令不存在", fmt.Errorf("command %d not found", cmdID))
}
if cmd.NodeID != node.ID {
return apperror.Unauthorized("AGENT_COMMAND_FORBIDDEN", "命令不属于当前节点", nil)
}
now := time.Now().UTC()
if result.Success {
cmd.Status = model.AgentCommandStatusSucceeded
} else {
cmd.Status = model.AgentCommandStatusFailed
}
cmd.ErrorMessage = result.ErrorMessage
if len(result.Result) > 0 {
cmd.Result = string(result.Result)
}
cmd.CompletedAt = &now
return s.cmdRepo.Update(ctx, cmd)
}
// AgentTaskSpec 给 Agent 返回的任务规格,包含解密后的存储配置,供 Agent 直接执行。
// 敏感信息:此接口仅供 Agent 调用token 认证),避免通过公共 API 泄露。
type AgentTaskSpec struct {
TaskID uint `json:"taskId"`
Name string `json:"name"`
Type string `json:"type"`
SourcePath string `json:"sourcePath,omitempty"`
SourcePaths string `json:"sourcePaths,omitempty"`
ExcludePatterns string `json:"excludePatterns,omitempty"`
DBHost string `json:"dbHost,omitempty"`
DBPort int `json:"dbPort,omitempty"`
DBUser string `json:"dbUser,omitempty"`
DBPassword string `json:"dbPassword,omitempty"`
DBName string `json:"dbName,omitempty"`
DBPath string `json:"dbPath,omitempty"`
ExtraConfig string `json:"extraConfig,omitempty"`
Compression string `json:"compression"`
Encrypt bool `json:"encrypt"`
StorageTargets []AgentStorageTargetConfig `json:"storageTargets"`
}
// AgentStorageTargetConfig 存储目标配置(已解密)
type AgentStorageTargetConfig struct {
ID uint `json:"id"`
Type string `json:"type"`
Name string `json:"name"`
Config json.RawMessage `json:"config"`
}
// GetTaskSpec 返回 Agent 执行任务所需的完整规格。
func (s *AgentService) GetTaskSpec(ctx context.Context, node *model.Node, taskID uint) (*AgentTaskSpec, error) {
task, err := s.taskRepo.FindByID(ctx, taskID)
if err != nil {
return nil, err
}
if task == nil {
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "任务不存在", nil)
}
if task.NodeID != node.ID {
return nil, apperror.Unauthorized("BACKUP_TASK_FORBIDDEN", "任务不属于当前节点", nil)
}
// 解密数据库密码(若有)
dbPassword := ""
if task.DBPasswordCiphertext != "" {
plain, decErr := s.cipher.Decrypt(task.DBPasswordCiphertext)
if decErr != nil {
return nil, fmt.Errorf("decrypt db password: %w", decErr)
}
dbPassword = string(plain)
}
// 解密存储目标配置
targets := collectTargetIDs(task)
storageTargets := make([]AgentStorageTargetConfig, 0, len(targets))
for _, tid := range targets {
target, err := s.storageRepo.FindByID(ctx, tid)
if err != nil {
return nil, err
}
if target == nil {
continue
}
configRaw, err := s.cipher.Decrypt(target.ConfigCiphertext)
if err != nil {
return nil, fmt.Errorf("decrypt storage config: %w", err)
}
storageTargets = append(storageTargets, AgentStorageTargetConfig{
ID: target.ID,
Type: target.Type,
Name: target.Name,
Config: json.RawMessage(configRaw),
})
}
return &AgentTaskSpec{
TaskID: task.ID,
Name: task.Name,
Type: task.Type,
SourcePath: task.SourcePath,
SourcePaths: task.SourcePaths,
ExcludePatterns: task.ExcludePatterns,
DBHost: task.DBHost,
DBPort: task.DBPort,
DBUser: task.DBUser,
DBPassword: dbPassword,
DBName: task.DBName,
DBPath: task.DBPath,
ExtraConfig: task.ExtraConfig,
Compression: task.Compression,
Encrypt: task.Encrypt,
StorageTargets: storageTargets,
}, nil
}
// AgentRecordUpdate Agent 上报备份记录的最终状态。
type AgentRecordUpdate struct {
Status string `json:"status"` // running | success | failed
FileName string `json:"fileName,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Checksum string `json:"checksum,omitempty"`
StoragePath string `json:"storagePath,omitempty"`
ErrorMessage string `json:"errorMessage,omitempty"`
LogAppend string `json:"logAppend,omitempty"` // 增量日志,追加到 record.log_content
}
// UpdateRecord 更新备份记录的状态/日志。Agent 在执行过程中可多次调用。
func (s *AgentService) UpdateRecord(ctx context.Context, node *model.Node, recordID uint, update AgentRecordUpdate) error {
record, err := s.recordRepo.FindByID(ctx, recordID)
if err != nil {
return err
}
if record == nil {
return apperror.New(404, "BACKUP_RECORD_NOT_FOUND", "记录不存在", nil)
}
// 通过 task.NodeID 判断是否属于当前 agent
task, err := s.taskRepo.FindByID(ctx, record.TaskID)
if err != nil {
return err
}
if task == nil || task.NodeID != node.ID {
return apperror.Unauthorized("BACKUP_RECORD_FORBIDDEN", "记录不属于当前节点", nil)
}
if update.Status != "" {
record.Status = update.Status
}
if update.FileName != "" {
record.FileName = update.FileName
}
if update.FileSize > 0 {
record.FileSize = update.FileSize
}
if update.Checksum != "" {
record.Checksum = update.Checksum
}
if update.StoragePath != "" {
record.StoragePath = update.StoragePath
}
if update.ErrorMessage != "" {
record.ErrorMessage = update.ErrorMessage
}
if update.LogAppend != "" {
if record.LogContent == "" {
record.LogContent = update.LogAppend
} else {
record.LogContent += update.LogAppend
}
}
if update.Status == model.BackupRecordStatusSuccess || update.Status == model.BackupRecordStatusFailed {
now := time.Now().UTC()
record.CompletedAt = &now
record.DurationSeconds = int(now.Sub(record.StartedAt).Seconds())
}
if err := s.recordRepo.Update(ctx, record); err != nil {
return err
}
// 同步更新任务的 last_status
if update.Status == model.BackupRecordStatusSuccess || update.Status == model.BackupRecordStatusFailed {
task.LastStatus = update.Status
_ = s.taskRepo.Update(ctx, task)
}
return nil
}
// EnqueueCommand Master 端调用:给指定节点插入一条待执行命令。
// 返回命令 ID。
func (s *AgentService) EnqueueCommand(ctx context.Context, nodeID uint, cmdType string, payload any) (uint, error) {
if nodeID == 0 {
return 0, errors.New("nodeID is required")
}
payloadBytes, err := json.Marshal(payload)
if err != nil {
return 0, fmt.Errorf("marshal payload: %w", err)
}
cmd := &model.AgentCommand{
NodeID: nodeID,
Type: cmdType,
Status: model.AgentCommandStatusPending,
Payload: string(payloadBytes),
}
if err := s.cmdRepo.Create(ctx, cmd); err != nil {
return 0, err
}
return cmd.ID, nil
}
// WaitForCommandResult 同步等待指定命令完成(用于 list_dir 这类 RPC 式调用)。
// timeout 为 0 表示不限,建议传 10~30s。
func (s *AgentService) WaitForCommandResult(ctx context.Context, cmdID uint, timeout time.Duration) (*model.AgentCommand, error) {
deadline := time.Now().Add(timeout)
for {
cmd, err := s.cmdRepo.FindByID(ctx, cmdID)
if err != nil {
return nil, err
}
if cmd == nil {
return nil, apperror.New(404, "AGENT_COMMAND_NOT_FOUND", "命令不存在", nil)
}
switch cmd.Status {
case model.AgentCommandStatusSucceeded, model.AgentCommandStatusFailed, model.AgentCommandStatusTimeout:
return cmd, nil
}
if timeout > 0 && time.Now().After(deadline) {
return nil, apperror.New(504, "AGENT_COMMAND_TIMEOUT", "等待 Agent 响应超时", nil)
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(300 * time.Millisecond):
}
}
}
// StartCommandTimeoutMonitor 启动后台定时任务,把超时命令标记为 timeout。
func (s *AgentService) StartCommandTimeoutMonitor(ctx context.Context, interval time.Duration, timeout time.Duration) {
if interval <= 0 {
interval = 30 * time.Second
}
if timeout <= 0 {
timeout = 10 * time.Minute
}
ticker := time.NewTicker(interval)
go func() {
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
threshold := time.Now().UTC().Add(-timeout)
_, _ = s.cmdRepo.MarkStaleTimeout(ctx, threshold)
}
}
}()
}

View File

@@ -0,0 +1,68 @@
package service
import (
"context"
"fmt"
"log"
"backupx/server/internal/apperror"
"backupx/server/internal/model"
"backupx/server/internal/repository"
)
// AuditEntry 是记录审计日志的输入结构
type AuditEntry struct {
UserID uint
Username string
Category string // auth / storage_target / backup_task / backup_record / settings
Action string // create / update / delete / login_success / login_failed / ...
TargetType string
TargetID string
TargetName string
Detail string
ClientIP string
}
type AuditService struct {
repo repository.AuditLogRepository
}
func NewAuditService(repo repository.AuditLogRepository) *AuditService {
return &AuditService{repo: repo}
}
// Record 异步 fire-and-forget 写入审计日志,不阻塞业务逻辑
func (s *AuditService) Record(entry AuditEntry) {
if s == nil || s.repo == nil {
return
}
go func() {
record := &model.AuditLog{
UserID: entry.UserID,
Username: entry.Username,
Category: entry.Category,
Action: entry.Action,
TargetType: entry.TargetType,
TargetID: entry.TargetID,
TargetName: entry.TargetName,
Detail: entry.Detail,
ClientIP: entry.ClientIP,
}
if err := s.repo.Create(context.Background(), record); err != nil {
log.Printf("[audit] failed to write audit log: %v", err)
}
}()
}
// List 分页查询审计日志
func (s *AuditService) List(ctx context.Context, category string, limit, offset int) (*repository.AuditLogListResult, error) {
result, err := s.repo.List(ctx, repository.AuditLogListOptions{
Category: category,
Limit: limit,
Offset: offset,
})
if err != nil {
return nil, apperror.Internal("AUDIT_LOG_LIST_FAILED", fmt.Sprintf("无法获取审计日志列表: %v", err), err)
}
return result, nil
}

View File

@@ -37,10 +37,11 @@ type UserOutput struct {
}
type AuthService struct {
users repository.UserRepository
configs repository.SystemConfigRepository
jwtManager *security.JWTManager
rateLimiter *security.LoginRateLimiter
users repository.UserRepository
configs repository.SystemConfigRepository
jwtManager *security.JWTManager
rateLimiter *security.LoginRateLimiter
auditService *AuditService
}
func NewAuthService(
@@ -52,6 +53,10 @@ func NewAuthService(
return &AuthService{users: users, configs: configs, jwtManager: jwtManager, rateLimiter: rateLimiter}
}
func (s *AuthService) SetAuditService(auditService *AuditService) {
s.auditService = auditService
}
func (s *AuthService) SetupStatus(ctx context.Context) (bool, error) {
count, err := s.users.Count(ctx)
if err != nil {
@@ -97,6 +102,15 @@ func (s *AuthService) Setup(ctx context.Context, input SetupInput) (*AuthPayload
return nil, apperror.Internal("AUTH_TOKEN_FAILED", "无法生成访问令牌", err)
}
if s.auditService != nil {
s.auditService.Record(AuditEntry{
UserID: user.ID, Username: user.Username,
Category: "auth", Action: "setup",
TargetType: "user", TargetID: fmt.Sprintf("%d", user.ID), TargetName: user.Username,
Detail: "系统初始化,创建管理员账户",
})
}
return &AuthPayload{Token: token, User: ToUserOutput(user)}, nil
}
@@ -113,9 +127,23 @@ func (s *AuthService) Login(ctx context.Context, input LoginInput, clientKey str
return nil, apperror.Internal("AUTH_LOOKUP_FAILED", "无法执行登录校验", err)
}
if user == nil {
if s.auditService != nil {
s.auditService.Record(AuditEntry{
Category: "auth", Action: "login_failed",
Detail: fmt.Sprintf("用户名不存在: %s", strings.TrimSpace(input.Username)),
ClientIP: clientKey,
})
}
return nil, apperror.Unauthorized("AUTH_INVALID_CREDENTIALS", "用户名或密码错误", nil)
}
if err := security.ComparePassword(user.PasswordHash, input.Password); err != nil {
if s.auditService != nil {
s.auditService.Record(AuditEntry{
UserID: user.ID, Username: user.Username,
Category: "auth", Action: "login_failed",
Detail: "密码错误", ClientIP: clientKey,
})
}
return nil, apperror.Unauthorized("AUTH_INVALID_CREDENTIALS", "用户名或密码错误", err)
}
@@ -124,6 +152,15 @@ func (s *AuthService) Login(ctx context.Context, input LoginInput, clientKey str
if err != nil {
return nil, apperror.Internal("AUTH_TOKEN_FAILED", "无法生成访问令牌", err)
}
if s.auditService != nil {
s.auditService.Record(AuditEntry{
UserID: user.ID, Username: user.Username,
Category: "auth", Action: "login_success",
Detail: "登录成功", ClientIP: clientKey,
})
}
return &AuthPayload{Token: token, User: ToUserOutput(user)}, nil
}
@@ -170,6 +207,15 @@ func (s *AuthService) ChangePassword(ctx context.Context, subject string, input
if err := s.users.Update(ctx, user); err != nil {
return apperror.Internal("AUTH_UPDATE_FAILED", "密码修改失败", err)
}
if s.auditService != nil {
s.auditService.Record(AuditEntry{
UserID: user.ID, Username: user.Username,
Category: "auth", Action: "change_password",
Detail: "密码修改成功",
})
}
return nil
}

View File

@@ -2,12 +2,16 @@ package service
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"hash"
"io"
"os"
"path/filepath"
"strings"
"sync"
"time"
"backupx/server/internal/apperror"
@@ -17,6 +21,7 @@ import (
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
"backupx/server/internal/storage/rclone"
"backupx/server/pkg/compress"
backupcrypto "backupx/server/pkg/crypto"
)
@@ -37,25 +42,65 @@ func (noopBackupNotifier) NotifyBackupResult(context.Context, BackupExecutionNot
return nil
}
type StorageUploadResultItem struct {
StorageTargetID uint `json:"storageTargetId"`
StorageTargetName string `json:"storageTargetName"`
Status string `json:"status"`
StoragePath string `json:"storagePath,omitempty"`
FileSize int64 `json:"fileSize,omitempty"`
Error string `json:"error,omitempty"`
}
type DownloadedArtifact struct {
FileName string
Reader io.ReadCloser
}
// collectTargetIDs 获取任务关联的所有存储目标 ID
func collectTargetIDs(task *model.BackupTask) []uint {
if len(task.StorageTargets) > 0 {
ids := make([]uint, len(task.StorageTargets))
for i, t := range task.StorageTargets {
ids[i] = t.ID
}
return ids
}
if task.StorageTargetID > 0 {
return []uint{task.StorageTargetID}
}
return nil
}
type BackupExecutionService struct {
tasks repository.BackupTaskRepository
records repository.BackupRecordRepository
targets repository.StorageTargetRepository
nodeRepo repository.NodeRepository
storageRegistry *storage.Registry
runnerRegistry *backup.Registry
logHub *backup.LogHub
retention *backupretention.Service
cipher *codec.ConfigCipher
notifier BackupResultNotifier
agentDispatcher AgentDispatcher
async func(func())
now func() time.Time
tempDir string
semaphore chan struct{}
retries int // rclone 底层重试次数
bandwidthLimit string // rclone 带宽限制
}
// AgentDispatcher 抽象把任务下发给 Agent 的能力,由 AgentService 实现。
// 用接口避免 execution service ↔ agent service 的循环依赖风险。
type AgentDispatcher interface {
EnqueueCommand(ctx context.Context, nodeID uint, cmdType string, payload any) (uint, error)
}
// SetClusterDependencies 注入集群相关的依赖,使备份执行时可把任务路由到远程节点。
func (s *BackupExecutionService) SetClusterDependencies(nodeRepo repository.NodeRepository, dispatcher AgentDispatcher) {
s.nodeRepo = nodeRepo
s.agentDispatcher = dispatcher
}
func NewBackupExecutionService(
@@ -70,6 +115,8 @@ func NewBackupExecutionService(
notifier BackupResultNotifier,
tempDir string,
maxConcurrent int,
retries int,
bandwidthLimit string,
) *BackupExecutionService {
if notifier == nil {
notifier = noopBackupNotifier{}
@@ -93,9 +140,11 @@ func NewBackupExecutionService(
async: func(job func()) {
go job()
},
now: func() time.Time { return time.Now().UTC() },
tempDir: tempDir,
semaphore: make(chan struct{}, maxConcurrent),
now: func() time.Time { return time.Now().UTC() },
tempDir: tempDir,
semaphore: make(chan struct{}, maxConcurrent),
retries: retries,
bandwidthLimit: bandwidthLimit,
}
}
@@ -194,7 +243,12 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
return nil, apperror.New(404, "BACKUP_TASK_NOT_FOUND", "备份任务不存在", fmt.Errorf("backup task %d not found", id))
}
startedAt := s.now()
record := &model.BackupRecord{TaskID: task.ID, StorageTargetID: task.StorageTargetID, Status: "running", StartedAt: startedAt}
// 取第一个存储目标 ID 做兼容
primaryTargetID := task.StorageTargetID
if tids := collectTargetIDs(task); len(tids) > 0 {
primaryTargetID = tids[0]
}
record := &model.BackupRecord{TaskID: task.ID, StorageTargetID: primaryTargetID, Status: "running", StartedAt: startedAt}
if err := s.records.Create(ctx, record); err != nil {
return nil, apperror.Internal("BACKUP_RECORD_CREATE_FAILED", "无法创建备份记录", err)
}
@@ -203,6 +257,20 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
if err := s.tasks.Update(ctx, task); err != nil {
return nil, apperror.Internal("BACKUP_TASK_UPDATE_FAILED", "无法更新任务状态", err)
}
// 多节点路由task.NodeID 指向远程节点时,把执行任务入队给 Agent
// NodeID=0 或本机节点时由 Master 直接执行。
if s.isRemoteNode(ctx, task.NodeID) {
if _, enqueueErr := s.agentDispatcher.EnqueueCommand(ctx, task.NodeID, model.AgentCommandTypeRunTask, map[string]any{
"taskId": task.ID,
"recordId": record.ID,
}); enqueueErr != nil {
// 入队失败 → 在记录中标记失败,继续返回详情
_ = s.finalizeRecord(ctx, task, record.ID, startedAt, model.BackupRecordStatusFailed,
"无法下发任务到远程节点: "+enqueueErr.Error(), "", "", 0, "", "")
return nil, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "无法下发任务到远程节点", enqueueErr)
}
return s.getRecordDetail(ctx, record.ID)
}
run := func() {
s.executeTask(context.Background(), task, record.ID, startedAt)
}
@@ -214,6 +282,19 @@ func (s *BackupExecutionService) startTask(ctx context.Context, id uint, async b
return s.getRecordDetail(ctx, record.ID)
}
// isRemoteNode 判断 NodeID 是否指向一个有效的远程(非本机)节点。
// 当未注入集群依赖、nodeID 为 0、或节点为本机时均返回 false走本地执行
func (s *BackupExecutionService) isRemoteNode(ctx context.Context, nodeID uint) bool {
if s.nodeRepo == nil || s.agentDispatcher == nil || nodeID == 0 {
return false
}
node, err := s.nodeRepo.FindByID(ctx, nodeID)
if err != nil || node == nil {
return false
}
return !node.IsLocal
}
func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time) {
s.semaphore <- struct{}{}
defer func() { <-s.semaphore }()
@@ -223,11 +304,22 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
errMessage := ""
var fileName string
var fileSize int64
var checksum string
var storagePath string
var uploadResults []StorageUploadResultItem
completeRecord := func() {
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, storagePath); finalizeErr != nil {
if finalizeErr := s.finalizeRecord(ctx, task, recordID, startedAt, status, errMessage, logger.String(), fileName, fileSize, checksum, storagePath); finalizeErr != nil {
logger.Errorf("写回备份记录失败:%v", finalizeErr)
}
// 写入多目标上传结果
if len(uploadResults) > 0 {
if resultsJSON, marshalErr := json.Marshal(uploadResults); marshalErr == nil {
if record, findErr := s.records.FindByID(ctx, recordID); findErr == nil && record != nil {
record.StorageUploadResults = string(resultsJSON)
_ = s.records.Update(ctx, record)
}
}
}
if err := s.notifier.NotifyBackupResult(ctx, BackupExecutionNotification{Task: task, Record: &model.BackupRecord{ID: recordID, TaskID: task.ID, Status: status, FileName: fileName, FileSize: fileSize, StoragePath: storagePath, ErrorMessage: errMessage, StartedAt: startedAt}, Error: buildOptionalError(errMessage)}); err != nil {
logger.Warnf("发送备份通知失败:%v", err)
}
@@ -241,12 +333,6 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
logger.Errorf("构建任务运行时配置失败:%v", err)
return
}
provider, err := s.resolveProvider(ctx, task.StorageTargetID)
if err != nil {
errMessage = err.Error()
logger.Errorf("创建存储客户端失败:%v", err)
return
}
runner, err := s.runnerRegistry.Runner(spec.Type)
if err != nil {
errMessage = err.Error()
@@ -290,34 +376,126 @@ func (s *BackupExecutionService) executeTask(ctx context.Context, task *model.Ba
fileSize = info.Size()
fileName = filepath.Base(finalPath)
storagePath = backup.BuildStorageKey(task.Type, startedAt, fileName)
artifact, err := os.Open(finalPath)
if err != nil {
errMessage = err.Error()
logger.Errorf("打开备份文件失败:%v", err)
// 收集所有存储目标
targetIDs := collectTargetIDs(task)
if len(targetIDs) == 0 {
errMessage = "没有关联的存储目标"
logger.Errorf("没有关联的存储目标")
return
}
defer artifact.Close()
logger.Infof("开始上传备份到存储目标")
if err := provider.Upload(ctx, storagePath, artifact, fileSize, map[string]string{"taskId": fmt.Sprintf("%d", task.ID), "recordId": fmt.Sprintf("%d", recordID)}); err != nil {
errMessage = err.Error()
logger.Errorf("上传备份文件失败:%v", err)
return
}
if s.retention != nil {
cleanupResult, cleanupErr := s.retention.Cleanup(ctx, task, provider)
if cleanupErr != nil {
logger.Warnf("执行保留策略失败:%v", cleanupErr)
} else {
for _, warning := range cleanupResult.Warnings {
logger.Warnf("保留策略警告:%s", warning)
// 并行上传到所有目标
uploadResults = make([]StorageUploadResultItem, len(targetIDs))
var checksumOnce sync.Once
var wg sync.WaitGroup
for i, tid := range targetIDs {
wg.Add(1)
go func(index int, targetID uint) {
defer wg.Done()
target, findErr := s.targets.FindByID(ctx, targetID)
targetName := fmt.Sprintf("target-%d", targetID)
if findErr == nil && target != nil {
targetName = target.Name
}
provider, resolveErr := s.resolveProvider(ctx, targetID)
if resolveErr != nil {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: resolveErr.Error()}
logger.Warnf("存储目标 %s 创建客户端失败:%v", targetName, resolveErr)
return
}
logger.Infof("开始上传备份到存储目标:%s", targetName)
// 上传级重试:最多 3 次指数退避10s, 30s, 90s
maxAttempts := 3
var lastUploadErr error
var hr *hashingReader
for attempt := 1; attempt <= maxAttempts; attempt++ {
if attempt > 1 {
backoff := time.Duration(attempt*attempt) * 10 * time.Second
logger.Warnf("存储目标 %s 第 %d 次重试(等待 %v%v", targetName, attempt, backoff, lastUploadErr)
time.Sleep(backoff)
}
artifact, openErr := os.Open(finalPath)
if openErr != nil {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: openErr.Error()}
logger.Warnf("存储目标 %s 打开备份文件失败:%v", targetName, openErr)
return
}
hr = newHashingReader(artifact)
pr := newProgressReader(hr, fileSize, func(bytesRead int64, speedBps float64) {
percent := float64(0)
if fileSize > 0 {
percent = float64(bytesRead) / float64(fileSize) * 100
}
s.logHub.AppendProgress(recordID, backup.ProgressInfo{
BytesSent: bytesRead,
TotalBytes: fileSize,
Percent: percent,
SpeedBps: speedBps,
TargetName: targetName,
})
})
lastUploadErr = provider.Upload(ctx, storagePath, pr, fileSize, map[string]string{"taskId": fmt.Sprintf("%d", task.ID), "recordId": fmt.Sprintf("%d", recordID)})
artifact.Close()
if lastUploadErr == nil {
break
}
}
if lastUploadErr != nil {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: lastUploadErr.Error()}
logger.Warnf("存储目标 %s 上传失败(已重试 %d 次):%v", targetName, maxAttempts, lastUploadErr)
return
}
// 完整性校验:对比实际传输字节数
if hr.n != fileSize {
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "failed", Error: fmt.Sprintf("完整性校验失败: 预期 %d bytes, 实际传输 %d bytes", fileSize, hr.n)}
logger.Errorf("存储目标 %s 完整性校验失败:预期 %d bytes, 实际传输 %d bytes", targetName, fileSize, hr.n)
_ = provider.Delete(ctx, storagePath)
return
}
// 取第一个成功目标的哈希写入 record所有目标读同一文件哈希一定相同
targetChecksum := hr.Sum()
checksumOnce.Do(func() { checksum = targetChecksum })
uploadResults[index] = StorageUploadResultItem{StorageTargetID: targetID, StorageTargetName: targetName, Status: "success", StoragePath: storagePath, FileSize: fileSize}
logger.Infof("存储目标 %s 上传成功 (%d bytes, SHA-256=%s)", targetName, fileSize, targetChecksum)
// 每个成功目标独立执行保留策略
if s.retention != nil {
cleanupResult, cleanupErr := s.retention.Cleanup(ctx, task, provider)
if cleanupErr != nil {
logger.Warnf("存储目标 %s 执行保留策略失败:%v", targetName, cleanupErr)
} else {
for _, warning := range cleanupResult.Warnings {
logger.Warnf("存储目标 %s 保留策略警告:%s", targetName, warning)
}
}
}
}(i, tid)
}
wg.Wait()
// 汇总结果:任意一个 success → 整体 success
anySuccess := false
var failedMessages []string
for _, r := range uploadResults {
if r.Status == "success" {
anySuccess = true
} else if r.Error != "" {
failedMessages = append(failedMessages, fmt.Sprintf("%s: %s", r.StorageTargetName, r.Error))
}
}
status = "success"
logger.Infof("备份执行完成")
if anySuccess {
status = "success"
if len(failedMessages) > 0 {
logger.Warnf("部分存储目标上传失败:%s", strings.Join(failedMessages, "; "))
}
logger.Infof("备份执行完成")
} else {
errMessage = strings.Join(failedMessages, "; ")
logger.Errorf("所有存储目标上传均失败")
}
}
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, storagePath string) error {
func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model.BackupTask, recordID uint, startedAt time.Time, status string, errorMessage string, logContent string, fileName string, fileSize int64, checksum string, storagePath string) error {
record, err := s.records.FindByID(ctx, recordID)
if err != nil {
return err
@@ -329,6 +507,7 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
record.Status = status
record.FileName = fileName
record.FileSize = fileSize
record.Checksum = checksum
record.StoragePath = storagePath
record.DurationSeconds = int(completedAt.Sub(startedAt).Seconds())
record.ErrorMessage = strings.TrimSpace(errorMessage)
@@ -343,6 +522,11 @@ func (s *BackupExecutionService) finalizeRecord(ctx context.Context, task *model
}
func (s *BackupExecutionService) resolveProvider(ctx context.Context, targetID uint) (storage.StorageProvider, error) {
// 注入 rclone 传输配置(重试、带宽限制)
ctx = rclone.ConfiguredContext(ctx, rclone.TransferConfig{
LowLevelRetries: s.retries,
BandwidthLimit: s.bandwidthLimit,
})
target, err := s.targets.FindByID(ctx, targetID)
if err != nil {
return nil, apperror.Internal("BACKUP_STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
@@ -376,11 +560,34 @@ func (s *BackupExecutionService) buildTaskSpec(task *model.BackupTask, startedAt
}
password = string(plain)
}
sourcePaths := []string{}
if strings.TrimSpace(task.SourcePaths) != "" {
if err := json.Unmarshal([]byte(task.SourcePaths), &sourcePaths); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
}
}
dbSpec := backup.DatabaseSpec{
Host: task.DBHost,
Port: task.DBPort,
User: task.DBUser,
Password: password,
Names: []string{task.DBName},
Path: task.DBPath,
}
// 解析 ExtraConfig 填充类型特有字段(目前主要用于 SAP HANA
if strings.TrimSpace(task.ExtraConfig) != "" {
extra := map[string]any{}
if err := json.Unmarshal([]byte(task.ExtraConfig), &extra); err != nil {
return backup.TaskSpec{}, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
}
applyHANAExtraConfig(&dbSpec, extra)
}
return backup.TaskSpec{
ID: task.ID,
Name: task.Name,
Type: task.Type,
SourcePath: task.SourcePath,
SourcePaths: sourcePaths,
ExcludePatterns: excludePatterns,
StorageTargetID: task.StorageTargetID,
StorageTargetType: "",
@@ -390,17 +597,30 @@ func (s *BackupExecutionService) buildTaskSpec(task *model.BackupTask, startedAt
MaxBackups: task.MaxBackups,
StartedAt: startedAt,
TempDir: s.tempDir,
Database: backup.DatabaseSpec{
Host: task.DBHost,
Port: task.DBPort,
User: task.DBUser,
Password: password,
Names: []string{task.DBName},
Path: task.DBPath,
},
Database: dbSpec,
}, nil
}
// applyHANAExtraConfig 从 ExtraConfig map 中提取 SAP HANA 字段填入 DatabaseSpec。
// 不识别的键被忽略,保持向后兼容。
func applyHANAExtraConfig(spec *backup.DatabaseSpec, extra map[string]any) {
if v, ok := extra["instanceNumber"].(string); ok {
spec.InstanceNumber = strings.TrimSpace(v)
}
if v, ok := extra["backupLevel"].(string); ok {
spec.BackupLevel = strings.ToLower(strings.TrimSpace(v))
}
if v, ok := extra["backupType"].(string); ok {
spec.BackupType = strings.ToLower(strings.TrimSpace(v))
}
if v, ok := extra["backupChannels"].(float64); ok {
spec.BackupChannels = int(v)
}
if v, ok := extra["maxRetries"].(float64); ok {
spec.MaxRetries = int(v)
}
}
func (s *BackupExecutionService) loadRecordProvider(ctx context.Context, recordID uint) (*model.BackupRecord, storage.StorageProvider, error) {
record, err := s.records.FindByID(ctx, recordID)
if err != nil {
@@ -485,3 +705,28 @@ func buildStorageProviderFromRepos(ctx context.Context, storageTargetID uint, st
}
return provider, target, nil
}
// hashingReader 在上传过程中同步计算字节数和 SHA-256零额外 I/O
type hashingReader struct {
reader io.Reader
hash hash.Hash
n int64
}
func newHashingReader(reader io.Reader) *hashingReader {
h := sha256.New()
return &hashingReader{
reader: io.TeeReader(reader, h),
hash: h,
}
}
func (r *hashingReader) Read(p []byte) (int, error) {
n, err := r.reader.Read(p)
r.n += int64(n)
return n, err
}
func (r *hashingReader) Sum() string {
return hex.EncodeToString(r.hash.Sum(nil))
}

View File

@@ -15,7 +15,7 @@ import (
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
"backupx/server/internal/storage/localdisk"
storageRclone "backupx/server/internal/storage/rclone"
)
func newExecutionTestServices(t *testing.T) (*BackupExecutionService, *BackupRecordService, repository.BackupTaskRepository, repository.StorageTargetRepository, repository.BackupRecordRepository, string, string) {
@@ -53,9 +53,13 @@ func newExecutionTestServices(t *testing.T) (*BackupExecutionService, *BackupRec
}
logHub := backup.NewLogHub()
runnerRegistry := backup.NewRegistry(backup.NewFileRunner(), backup.NewMySQLRunner(nil), backup.NewSQLiteRunner(), backup.NewPostgreSQLRunner(nil))
storageRegistry := storage.NewRegistry(localdisk.NewFactory())
storageRegistry := storage.NewRegistry(storageRclone.NewLocalDiskFactory())
retentionService := backupretention.NewService(records)
executionService := NewBackupExecutionService(tasks, records, targets, storageRegistry, runnerRegistry, logHub, retentionService, cipher, nil, "", 2)
tempDir := filepath.Join(baseDir, "tmp")
if err := os.MkdirAll(tempDir, 0o755); err != nil {
t.Fatalf("MkdirAll tempDir returned error: %v", err)
}
executionService := NewBackupExecutionService(tasks, records, targets, storageRegistry, runnerRegistry, logHub, retentionService, cipher, nil, tempDir, 2, 10, "")
recordService := NewBackupRecordService(records, executionService, logHub)
return executionService, recordService, tasks, targets, records, sourceDir, storageDir
}

View File

@@ -2,6 +2,7 @@ package service
import (
"context"
"encoding/json"
"strings"
"time"
@@ -29,6 +30,7 @@ type BackupRecordSummary struct {
Status string `json:"status"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
Checksum string `json:"checksum"`
StoragePath string `json:"storagePath"`
DurationSeconds int `json:"durationSeconds"`
ErrorMessage string `json:"errorMessage"`
@@ -38,8 +40,9 @@ type BackupRecordSummary struct {
type BackupRecordDetail struct {
BackupRecordSummary
LogContent string `json:"logContent"`
LogEvents []backup.LogEvent `json:"logEvents,omitempty"`
LogContent string `json:"logContent"`
LogEvents []backup.LogEvent `json:"logEvents,omitempty"`
StorageUploadResults []StorageUploadResultItem `json:"storageUploadResults,omitempty"`
}
type BackupRecordService struct {
@@ -109,6 +112,7 @@ func toBackupRecordSummary(item *model.BackupRecord) BackupRecordSummary {
Status: item.Status,
FileName: item.FileName,
FileSize: item.FileSize,
Checksum: item.Checksum,
StoragePath: item.StoragePath,
DurationSeconds: item.DurationSeconds,
ErrorMessage: item.ErrorMessage,
@@ -130,5 +134,12 @@ func toBackupRecordDetail(item *model.BackupRecord, logHub *backup.LogHub) *Back
detail.LogContent = strings.Join(lines, "\n")
}
}
// 解析多目标上传结果
if strings.TrimSpace(item.StorageUploadResults) != "" {
var uploadResults []StorageUploadResultItem
if err := json.Unmarshal([]byte(item.StorageUploadResults), &uploadResults); err == nil {
detail.StorageUploadResults = uploadResults
}
}
return detail
}

View File

@@ -11,29 +11,34 @@ import (
"backupx/server/internal/apperror"
"backupx/server/internal/model"
"backupx/server/internal/repository"
"backupx/server/internal/storage"
"backupx/server/internal/storage/codec"
)
const backupTaskMaskedValue = "********"
type BackupTaskUpsertInput struct {
Name string `json:"name" binding:"required,min=1,max=100"`
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr" binding:"max=64"`
SourcePath string `json:"sourcePath" binding:"max=500"`
ExcludePatterns []string `json:"excludePatterns"`
DBHost string `json:"dbHost" binding:"max=255"`
DBPort int `json:"dbPort"`
DBUser string `json:"dbUser" binding:"max=100"`
DBPassword string `json:"dbPassword" binding:"max=255"`
DBName string `json:"dbName" binding:"max=255"`
DBPath string `json:"dbPath" binding:"max=500"`
StorageTargetID uint `json:"storageTargetId" binding:"required"`
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression" binding:"omitempty,oneof=gzip none"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
Name string `json:"name" binding:"required,min=1,max=100"`
Type string `json:"type" binding:"required,oneof=file mysql sqlite postgresql pgsql saphana"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr" binding:"max=64"`
SourcePath string `json:"sourcePath" binding:"max=500"`
SourcePaths []string `json:"sourcePaths"`
ExcludePatterns []string `json:"excludePatterns"`
DBHost string `json:"dbHost" binding:"max=255"`
DBPort int `json:"dbPort"`
DBUser string `json:"dbUser" binding:"max=100"`
DBPassword string `json:"dbPassword" binding:"max=255"`
DBName string `json:"dbName" binding:"max=255"`
DBPath string `json:"dbPath" binding:"max=500"`
StorageTargetID uint `json:"storageTargetId"` // deprecated: 向后兼容
StorageTargetIDs []uint `json:"storageTargetIds"` // 新增:多存储目标
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression" binding:"omitempty,oneof=gzip none"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
// ExtraConfig 类型特有扩展配置(如 SAP HANA 的 backupLevel/backupChannels
ExtraConfig map[string]any `json:"extraConfig"`
}
type BackupTaskToggleInput struct {
@@ -41,33 +46,37 @@ type BackupTaskToggleInput struct {
}
type BackupTaskSummary struct {
ID uint `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr"`
StorageTargetID uint `json:"storageTargetId"`
StorageTargetName string `json:"storageTargetName"`
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
LastRunAt *time.Time `json:"lastRunAt,omitempty"`
LastStatus string `json:"lastStatus"`
UpdatedAt time.Time `json:"updatedAt"`
ID uint `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
Enabled bool `json:"enabled"`
CronExpr string `json:"cronExpr"`
StorageTargetID uint `json:"storageTargetId"` // deprecated: 取第一个
StorageTargetName string `json:"storageTargetName"` // deprecated: 取第一个
StorageTargetIDs []uint `json:"storageTargetIds"`
StorageTargetNames []string `json:"storageTargetNames"`
RetentionDays int `json:"retentionDays"`
Compression string `json:"compression"`
Encrypt bool `json:"encrypt"`
MaxBackups int `json:"maxBackups"`
LastRunAt *time.Time `json:"lastRunAt,omitempty"`
LastStatus string `json:"lastStatus"`
UpdatedAt time.Time `json:"updatedAt"`
}
type BackupTaskDetail struct {
BackupTaskSummary
SourcePath string `json:"sourcePath"`
ExcludePatterns []string `json:"excludePatterns"`
DBHost string `json:"dbHost"`
DBPort int `json:"dbPort"`
DBUser string `json:"dbUser"`
DBName string `json:"dbName"`
DBPath string `json:"dbPath"`
MaskedFields []string `json:"maskedFields,omitempty"`
CreatedAt time.Time `json:"createdAt"`
SourcePath string `json:"sourcePath"`
SourcePaths []string `json:"sourcePaths"`
ExcludePatterns []string `json:"excludePatterns"`
DBHost string `json:"dbHost"`
DBPort int `json:"dbPort"`
DBUser string `json:"dbUser"`
DBName string `json:"dbName"`
DBPath string `json:"dbPath"`
ExtraConfig map[string]any `json:"extraConfig,omitempty"`
MaskedFields []string `json:"maskedFields,omitempty"`
CreatedAt time.Time `json:"createdAt"`
}
type BackupTaskScheduler interface {
@@ -76,10 +85,12 @@ type BackupTaskScheduler interface {
}
type BackupTaskService struct {
tasks repository.BackupTaskRepository
targets repository.StorageTargetRepository
cipher *codec.ConfigCipher
scheduler BackupTaskScheduler
tasks repository.BackupTaskRepository
targets repository.StorageTargetRepository
records repository.BackupRecordRepository
storageRegistry *storage.Registry
cipher *codec.ConfigCipher
scheduler BackupTaskScheduler
}
func NewBackupTaskService(
@@ -90,6 +101,12 @@ func NewBackupTaskService(
return &BackupTaskService{tasks: tasks, targets: targets, cipher: cipher}
}
// SetRecordsAndStorage 注入备份记录仓库和存储注册表,用于任务删除时清理远端文件。
func (s *BackupTaskService) SetRecordsAndStorage(records repository.BackupRecordRepository, registry *storage.Registry) {
s.records = records
s.storageRegistry = registry
}
func (s *BackupTaskService) SetScheduler(scheduler BackupTaskScheduler) {
s.scheduler = scheduler
}
@@ -180,26 +197,80 @@ func (s *BackupTaskService) Update(ctx context.Context, id uint, input BackupTas
return s.Get(ctx, item.ID)
}
func (s *BackupTaskService) Delete(ctx context.Context, id uint) error {
// DeleteResult 描述任务删除的结果信息,用于审计日志。
type DeleteResult struct {
TaskName string
RecordCount int
CleanedFiles int
}
func (s *BackupTaskService) Delete(ctx context.Context, id uint) (*DeleteResult, error) {
existing, err := s.tasks.FindByID(ctx, id)
if err != nil {
return apperror.Internal("BACKUP_TASK_GET_FAILED", "无法获取备份任务详情", err)
return nil, apperror.Internal("BACKUP_TASK_GET_FAILED", "无法获取备份任务详情", err)
}
if existing == nil {
return apperror.New(http.StatusNotFound, "BACKUP_TASK_NOT_FOUND", "备份任务不存在", fmt.Errorf("backup task %d not found", id))
}
if s.scheduler != nil {
if err := s.scheduler.RemoveTask(ctx, id); err != nil {
return apperror.Internal("BACKUP_TASK_SCHEDULE_FAILED", "无法移除备份任务调度", err)
}
}
if err := s.tasks.Delete(ctx, id); err != nil {
return apperror.Internal("BACKUP_TASK_DELETE_FAILED", "无法删除备份任务", err)
return nil, apperror.New(http.StatusNotFound, "BACKUP_TASK_NOT_FOUND", "备份任务不存在", fmt.Errorf("backup task %d not found", id))
}
if s.scheduler != nil {
_ = s.scheduler.RemoveTask(ctx, id)
}
return nil
// 清理远端存储文件(尽力而为,不阻止删除)
result := &DeleteResult{TaskName: existing.Name}
result.RecordCount, result.CleanedFiles = s.cleanupRemoteFiles(ctx, id)
if err := s.tasks.Delete(ctx, id); err != nil {
return nil, apperror.Internal("BACKUP_TASK_DELETE_FAILED", "无法删除备份任务", err)
}
return result, nil
}
// cleanupRemoteFiles 尽力删除任务相关的远端备份文件,返回记录数和成功删除的文件数。
func (s *BackupTaskService) cleanupRemoteFiles(ctx context.Context, taskID uint) (recordCount int, cleanedFiles int) {
if s.records == nil || s.storageRegistry == nil {
return 0, 0
}
records, err := s.records.ListByTask(ctx, taskID)
if err != nil {
return 0, 0
}
recordCount = len(records)
// 缓存 provider 避免同一存储目标重复创建连接
providerCache := make(map[uint]storage.StorageProvider)
for _, record := range records {
if strings.TrimSpace(record.StoragePath) == "" {
continue
}
provider, ok := providerCache[record.StorageTargetID]
if !ok {
provider, err = s.resolveStorageProvider(ctx, record.StorageTargetID)
if err != nil {
continue
}
providerCache[record.StorageTargetID] = provider
}
if err := provider.Delete(ctx, record.StoragePath); err == nil {
cleanedFiles++
}
}
return recordCount, cleanedFiles
}
func (s *BackupTaskService) resolveStorageProvider(ctx context.Context, targetID uint) (storage.StorageProvider, error) {
target, err := s.targets.FindByID(ctx, targetID)
if err != nil || target == nil {
return nil, fmt.Errorf("target %d not found", targetID)
}
configMap := map[string]any{}
if err := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); err != nil {
return nil, err
}
provider, err := s.storageRegistry.Create(ctx, target.Type, configMap)
if err != nil {
return nil, err
}
return provider, nil
}
func (s *BackupTaskService) Toggle(ctx context.Context, id uint, enabled bool) (*BackupTaskSummary, error) {
@@ -227,19 +298,33 @@ func (s *BackupTaskService) Toggle(ctx context.Context, id uint, enabled bool) (
return &returnValue, nil
}
// resolveStorageTargetIDs 统一处理新旧字段,返回有效的存储目标 ID 列表
func resolveStorageTargetIDs(input BackupTaskUpsertInput) []uint {
if len(input.StorageTargetIDs) > 0 {
return input.StorageTargetIDs
}
if input.StorageTargetID > 0 {
return []uint{input.StorageTargetID}
}
return nil
}
func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.BackupTask, input BackupTaskUpsertInput) error {
if strings.TrimSpace(input.Name) == "" {
return apperror.BadRequest("BACKUP_TASK_INVALID", "任务名称不能为空", nil)
}
if input.StorageTargetID == 0 {
return apperror.BadRequest("BACKUP_TASK_INVALID", "请选择存储目标", nil)
targetIDs := resolveStorageTargetIDs(input)
if len(targetIDs) == 0 {
return apperror.BadRequest("BACKUP_TASK_INVALID", "请选择至少一个存储目标", nil)
}
target, err := s.targets.FindByID(ctx, input.StorageTargetID)
if err != nil {
return apperror.Internal("BACKUP_TASK_STORAGE_LOOKUP_FAILED", "无法检查存储目标", err)
}
if target == nil {
return apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", "关联的存储目标不存在", nil)
for _, tid := range targetIDs {
target, err := s.targets.FindByID(ctx, tid)
if err != nil {
return apperror.Internal("BACKUP_TASK_STORAGE_LOOKUP_FAILED", "无法检查存储目标", err)
}
if target == nil {
return apperror.BadRequest("BACKUP_STORAGE_TARGET_INVALID", fmt.Sprintf("关联的存储目标 %d 不存在", tid), nil)
}
}
if input.RetentionDays < 0 {
return apperror.BadRequest("BACKUP_TASK_INVALID", "保留天数不能小于 0", nil)
@@ -260,10 +345,11 @@ func (s *BackupTaskService) validateInput(ctx context.Context, existing *model.B
func validateTaskTypeSpecificFields(input BackupTaskUpsertInput, passwordRequired bool) error {
switch normalizeBackupTaskType(input.Type) {
case "file":
if strings.TrimSpace(input.SourcePath) == "" {
hasSourcePaths := len(resolveSourcePaths(input)) > 0
if !hasSourcePaths {
return apperror.BadRequest("BACKUP_TASK_INVALID", "文件备份必须填写源路径", nil)
}
case "mysql", "postgresql":
case "mysql", "postgresql", "saphana":
if strings.TrimSpace(input.DBHost) == "" {
return apperror.BadRequest("BACKUP_TASK_INVALID", "数据库主机不能为空", nil)
}
@@ -294,6 +380,10 @@ func (s *BackupTaskService) buildTask(existing *model.BackupTask, input BackupTa
if err != nil {
return nil, apperror.BadRequest("BACKUP_TASK_INVALID", "排除规则格式不合法", err)
}
sourcePathsJSON, err := encodeSourcePaths(resolveSourcePaths(input))
if err != nil {
return nil, apperror.BadRequest("BACKUP_TASK_INVALID", "源路径格式不合法", err)
}
passwordCiphertext := ""
if existing != nil {
passwordCiphertext = existing.DBPasswordCiphertext
@@ -313,12 +403,34 @@ func (s *BackupTaskService) buildTask(existing *model.BackupTask, input BackupTa
if maxBackups == 0 {
maxBackups = 10
}
targetIDs := resolveStorageTargetIDs(input)
// 保持旧字段兼容:取第一个
primaryTargetID := uint(0)
if len(targetIDs) > 0 {
primaryTargetID = targetIDs[0]
}
// 构建多对多关联
storageTargets := make([]model.StorageTarget, len(targetIDs))
for i, tid := range targetIDs {
storageTargets[i] = model.StorageTarget{ID: tid}
}
// 向后兼容SourcePath 取第一个
resolvedPaths := resolveSourcePaths(input)
primarySourcePath := strings.TrimSpace(input.SourcePath)
if len(resolvedPaths) > 0 {
primarySourcePath = resolvedPaths[0]
}
extraConfigJSON, err := encodeExtraConfig(input.ExtraConfig)
if err != nil {
return nil, apperror.BadRequest("BACKUP_TASK_INVALID", "扩展配置格式不合法", err)
}
item := &model.BackupTask{
Name: strings.TrimSpace(input.Name),
Type: normalizeBackupTaskType(input.Type),
Enabled: input.Enabled,
CronExpr: strings.TrimSpace(input.CronExpr),
SourcePath: strings.TrimSpace(input.SourcePath),
SourcePath: primarySourcePath,
SourcePaths: sourcePathsJSON,
ExcludePatterns: excludePatterns,
DBHost: strings.TrimSpace(input.DBHost),
DBPort: input.DBPort,
@@ -326,7 +438,9 @@ func (s *BackupTaskService) buildTask(existing *model.BackupTask, input BackupTa
DBPasswordCiphertext: passwordCiphertext,
DBName: strings.TrimSpace(input.DBName),
DBPath: strings.TrimSpace(input.DBPath),
StorageTargetID: input.StorageTargetID,
ExtraConfig: extraConfigJSON,
StorageTargetID: primaryTargetID,
StorageTargets: storageTargets,
RetentionDays: input.RetentionDays,
Compression: compression,
Encrypt: input.Encrypt,
@@ -346,15 +460,25 @@ func (s *BackupTaskService) toDetail(item *model.BackupTask) (*BackupTaskDetail,
if err != nil {
return nil, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析备份任务配置", err)
}
sourcePaths, err := decodeSourcePaths(item.SourcePaths)
if err != nil {
return nil, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析源路径配置", err)
}
extraConfig, err := decodeExtraConfig(item.ExtraConfig)
if err != nil {
return nil, apperror.Internal("BACKUP_TASK_DECODE_FAILED", "无法解析扩展配置", err)
}
detail := &BackupTaskDetail{
BackupTaskSummary: toBackupTaskSummary(item),
SourcePath: item.SourcePath,
SourcePaths: sourcePaths,
ExcludePatterns: excludePatterns,
DBHost: item.DBHost,
DBPort: item.DBPort,
DBUser: item.DBUser,
DBName: item.DBName,
DBPath: item.DBPath,
ExtraConfig: extraConfig,
CreatedAt: item.CreatedAt,
}
if item.DBPasswordCiphertext != "" {
@@ -364,25 +488,45 @@ func (s *BackupTaskService) toDetail(item *model.BackupTask) (*BackupTaskDetail,
}
func toBackupTaskSummary(item *model.BackupTask) BackupTaskSummary {
storageTargetName := ""
if item != nil {
storageTargetName = item.StorageTarget.Name
// 从多对多关联提取 IDs 和 Names
var targetIDs []uint
var targetNames []string
if len(item.StorageTargets) > 0 {
for _, t := range item.StorageTargets {
targetIDs = append(targetIDs, t.ID)
targetNames = append(targetNames, t.Name)
}
} else if item.StorageTargetID > 0 {
// 回退到旧字段
targetIDs = []uint{item.StorageTargetID}
targetNames = []string{item.StorageTarget.Name}
}
// 向后兼容:取第一个
primaryID := uint(0)
primaryName := ""
if len(targetIDs) > 0 {
primaryID = targetIDs[0]
}
if len(targetNames) > 0 {
primaryName = targetNames[0]
}
return BackupTaskSummary{
ID: item.ID,
Name: item.Name,
Type: normalizeBackupTaskType(item.Type),
Enabled: item.Enabled,
CronExpr: item.CronExpr,
StorageTargetID: item.StorageTargetID,
StorageTargetName: storageTargetName,
RetentionDays: item.RetentionDays,
Compression: item.Compression,
Encrypt: item.Encrypt,
MaxBackups: item.MaxBackups,
LastRunAt: item.LastRunAt,
LastStatus: item.LastStatus,
UpdatedAt: item.UpdatedAt,
ID: item.ID,
Name: item.Name,
Type: normalizeBackupTaskType(item.Type),
Enabled: item.Enabled,
CronExpr: item.CronExpr,
StorageTargetID: primaryID,
StorageTargetName: primaryName,
StorageTargetIDs: targetIDs,
StorageTargetNames: targetNames,
RetentionDays: item.RetentionDays,
Compression: item.Compression,
Encrypt: item.Encrypt,
MaxBackups: item.MaxBackups,
LastRunAt: item.LastRunAt,
LastStatus: item.LastStatus,
UpdatedAt: item.UpdatedAt,
}
}
@@ -408,6 +552,70 @@ func decodeExcludePatterns(value string) ([]string, error) {
return items, nil
}
// resolveSourcePaths 统一处理 sourcePaths / sourcePath返回有效路径列表
func resolveSourcePaths(input BackupTaskUpsertInput) []string {
if len(input.SourcePaths) > 0 {
var cleaned []string
for _, p := range input.SourcePaths {
if trimmed := strings.TrimSpace(p); trimmed != "" {
cleaned = append(cleaned, trimmed)
}
}
if len(cleaned) > 0 {
return cleaned
}
}
if sp := strings.TrimSpace(input.SourcePath); sp != "" {
return []string{sp}
}
return nil
}
func encodeSourcePaths(paths []string) (string, error) {
if len(paths) == 0 {
return "[]", nil
}
encoded, err := json.Marshal(paths)
if err != nil {
return "", err
}
return string(encoded), nil
}
func decodeSourcePaths(value string) ([]string, error) {
if strings.TrimSpace(value) == "" || strings.TrimSpace(value) == "[]" {
return []string{}, nil
}
var items []string
if err := json.Unmarshal([]byte(value), &items); err != nil {
return nil, err
}
return items, nil
}
func encodeExtraConfig(value map[string]any) (string, error) {
if len(value) == 0 {
return "", nil
}
encoded, err := json.Marshal(value)
if err != nil {
return "", err
}
return string(encoded), nil
}
func decodeExtraConfig(value string) (map[string]any, error) {
trimmed := strings.TrimSpace(value)
if trimmed == "" || trimmed == "{}" {
return nil, nil
}
result := map[string]any{}
if err := json.Unmarshal([]byte(trimmed), &result); err != nil {
return nil, err
}
return result, nil
}
func normalizeBackupTaskType(value string) string {
normalized := strings.TrimSpace(strings.ToLower(value))
if normalized == "pgsql" {

View File

@@ -0,0 +1,141 @@
package service
import (
"bytes"
"context"
"fmt"
"strings"
"time"
"backupx/server/internal/apperror"
"backupx/server/internal/backup"
)
type DatabaseDiscoverInput struct {
Type string `json:"type" binding:"required,oneof=mysql postgresql"`
Host string `json:"host" binding:"required"`
Port int `json:"port" binding:"required,min=1"`
User string `json:"user" binding:"required"`
Password string `json:"password" binding:"required"`
}
type DatabaseDiscoverResult struct {
Databases []string `json:"databases"`
}
type DatabaseDiscoveryService struct {
executor backup.CommandExecutor
}
func NewDatabaseDiscoveryService(executor backup.CommandExecutor) *DatabaseDiscoveryService {
return &DatabaseDiscoveryService{executor: executor}
}
func (s *DatabaseDiscoveryService) Discover(ctx context.Context, input DatabaseDiscoverInput) (*DatabaseDiscoverResult, error) {
switch strings.TrimSpace(strings.ToLower(input.Type)) {
case "mysql":
return s.discoverMySQL(ctx, input)
case "postgresql":
return s.discoverPostgreSQL(ctx, input)
default:
return nil, apperror.BadRequest("DATABASE_DISCOVER_INVALID_TYPE", "不支持的数据库类型", nil)
}
}
func (s *DatabaseDiscoveryService) discoverMySQL(ctx context.Context, input DatabaseDiscoverInput) (*DatabaseDiscoverResult, error) {
mysqlPath, err := s.executor.LookPath("mysql")
if err != nil {
return nil, apperror.BadRequest("DATABASE_DISCOVER_MYSQL_NOT_FOUND", "系统未安装 mysql 客户端", err)
}
timeout, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
var stdout, stderr bytes.Buffer
args := []string{
fmt.Sprintf("--host=%s", input.Host),
fmt.Sprintf("--port=%d", input.Port),
fmt.Sprintf("--user=%s", input.User),
"-e", "SHOW DATABASES",
"--skip-column-names",
}
env := []string{fmt.Sprintf("MYSQL_PWD=%s", input.Password)}
if err := s.executor.Run(timeout, mysqlPath, args, backup.CommandOptions{
Stdout: &stdout,
Stderr: &stderr,
Env: env,
}); err != nil {
errMsg := strings.TrimSpace(stderr.String())
if errMsg == "" {
errMsg = err.Error()
}
return nil, apperror.BadRequest("DATABASE_DISCOVER_MYSQL_FAILED", fmt.Sprintf("连接 MySQL 失败:%s", sanitizeMessage(errMsg)), err)
}
systemDBs := map[string]bool{
"information_schema": true,
"performance_schema": true,
"mysql": true,
"sys": true,
}
var databases []string
for _, line := range strings.Split(stdout.String(), "\n") {
db := strings.TrimSpace(line)
if db == "" || systemDBs[db] {
continue
}
databases = append(databases, db)
}
return &DatabaseDiscoverResult{Databases: databases}, nil
}
func (s *DatabaseDiscoveryService) discoverPostgreSQL(ctx context.Context, input DatabaseDiscoverInput) (*DatabaseDiscoverResult, error) {
psqlPath, err := s.executor.LookPath("psql")
if err != nil {
return nil, apperror.BadRequest("DATABASE_DISCOVER_PSQL_NOT_FOUND", "系统未安装 psql 客户端", err)
}
timeout, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
var stdout, stderr bytes.Buffer
args := []string{
"-h", input.Host,
"-p", fmt.Sprintf("%d", input.Port),
"-U", input.User,
"-d", "postgres",
"-t", "-A",
"-c", "SELECT datname FROM pg_database WHERE datistemplate = false ORDER BY datname",
}
env := []string{fmt.Sprintf("PGPASSWORD=%s", input.Password)}
if err := s.executor.Run(timeout, psqlPath, args, backup.CommandOptions{
Stdout: &stdout,
Stderr: &stderr,
Env: env,
}); err != nil {
errMsg := strings.TrimSpace(stderr.String())
if errMsg == "" {
errMsg = err.Error()
}
return nil, apperror.BadRequest("DATABASE_DISCOVER_PSQL_FAILED", fmt.Sprintf("连接 PostgreSQL 失败:%s", sanitizeMessage(errMsg)), err)
}
skipDBs := map[string]bool{
"postgres": true,
}
var databases []string
for _, line := range strings.Split(stdout.String(), "\n") {
db := strings.TrimSpace(line)
if db == "" || skipDBs[db] || strings.HasPrefix(db, "template") {
continue
}
databases = append(databases, db)
}
return &DatabaseDiscoverResult{Databases: databases}, nil
}

View File

@@ -4,12 +4,15 @@ import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"fmt"
"net"
"net/http"
"os"
"path/filepath"
"runtime"
"sort"
"strings"
"time"
"backupx/server/internal/apperror"
@@ -37,13 +40,38 @@ type NodeCreateInput struct {
Name string `json:"name" binding:"required"`
}
// NodeService manages the cluster nodes.
type NodeService struct {
repo repository.NodeRepository
// NodeUpdateInput 是编辑节点的输入。
type NodeUpdateInput struct {
Name string `json:"name" binding:"required"`
}
func NewNodeService(repo repository.NodeRepository) *NodeService {
return &NodeService{repo: repo}
// NodeService manages the cluster nodes.
type NodeService struct {
repo repository.NodeRepository
taskRepo repository.BackupTaskRepository
agentRPC NodeAgentRPC
version string
}
// NodeAgentRPC 抽象 Agent 远程调用能力(避免 service 内循环依赖)。
// 由 AgentService 实现;当 Agent 未启用时可不注入,远程目录浏览返回提示。
type NodeAgentRPC interface {
EnqueueCommand(ctx context.Context, nodeID uint, cmdType string, payload any) (uint, error)
WaitForCommandResult(ctx context.Context, cmdID uint, timeout time.Duration) (*model.AgentCommand, error)
}
func NewNodeService(repo repository.NodeRepository, version string) *NodeService {
return &NodeService{repo: repo, version: version}
}
// SetTaskRepository 注入任务仓储以支持删除前引用检查。可选注入,便于测试。
func (s *NodeService) SetTaskRepository(taskRepo repository.BackupTaskRepository) {
s.taskRepo = taskRepo
}
// SetAgentRPC 注入 Agent RPC 能力,启用远程目录浏览。
func (s *NodeService) SetAgentRPC(rpc NodeAgentRPC) {
s.agentRPC = rpc
}
// EnsureLocalNode creates the default "local" node if it does not exist.
@@ -57,6 +85,8 @@ func (s *NodeService) EnsureLocalNode(ctx context.Context) error {
existing.LastSeen = time.Now().UTC()
hostname, _ := os.Hostname()
existing.Hostname = hostname
existing.IPAddress = detectLocalIP()
existing.AgentVer = s.version
existing.OS = runtime.GOOS
existing.Arch = runtime.GOARCH
return s.repo.Update(ctx, existing)
@@ -64,14 +94,16 @@ func (s *NodeService) EnsureLocalNode(ctx context.Context) error {
hostname, _ := os.Hostname()
token, _ := generateToken()
node := &model.Node{
Name: "本机 (Local)",
Hostname: hostname,
Token: token,
Status: model.NodeStatusOnline,
IsLocal: true,
OS: runtime.GOOS,
Arch: runtime.GOARCH,
LastSeen: time.Now().UTC(),
Name: "本机 (Local)",
Hostname: hostname,
IPAddress: detectLocalIP(),
Token: token,
Status: model.NodeStatusOnline,
IsLocal: true,
OS: runtime.GOOS,
Arch: runtime.GOARCH,
AgentVer: s.version,
LastSeen: time.Now().UTC(),
}
return s.repo.Create(ctx, node)
}
@@ -153,6 +185,20 @@ func (s *NodeService) Delete(ctx context.Context, id uint) error {
if node.IsLocal {
return apperror.BadRequest("NODE_DELETE_LOCAL", "无法删除本机节点", nil)
}
// 删除前检查是否有关联备份任务,避免孤立任务
if s.taskRepo != nil {
count, err := s.taskRepo.CountByNodeID(ctx, id)
if err != nil {
return err
}
if count > 0 {
return apperror.BadRequest(
"NODE_HAS_TASKS",
fmt.Sprintf("无法删除:该节点上还有 %d 个备份任务,请先删除或迁移", count),
nil,
)
}
}
return s.repo.Delete(ctx, id)
}
@@ -166,7 +212,8 @@ func (s *NodeService) ListDirectory(ctx context.Context, nodeID uint, path strin
return nil, apperror.New(http.StatusNotFound, "NODE_NOT_FOUND", "节点不存在", nil)
}
if !node.IsLocal {
return nil, apperror.BadRequest("NODE_REMOTE_FS_NOT_SUPPORTED", "远程节点的目录浏览需要 Agent 在线连接(即将支持)", nil)
// 远程节点:通过 Agent 命令队列做同步 RPC
return s.remoteListDirectory(ctx, node, path)
}
cleanPath := filepath.Clean(path)
@@ -198,8 +245,33 @@ func (s *NodeService) ListDirectory(ctx context.Context, nodeID uint, path strin
return result, nil
}
// OfflineThreshold 节点被判定为离线的心跳超时阈值。
// Agent 默认 15s 心跳一次45s 未见视为离线,预留 3 次重试空间。
const OfflineThreshold = 45 * time.Second
// StartOfflineMonitor 启动后台 goroutine定期把超时未心跳的节点标记为离线。
// 传入的 ctx 被取消后退出。
func (s *NodeService) StartOfflineMonitor(ctx context.Context, interval time.Duration) {
if interval <= 0 {
interval = 15 * time.Second
}
ticker := time.NewTicker(interval)
go func() {
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
threshold := time.Now().UTC().Add(-OfflineThreshold)
_, _ = s.repo.MarkStaleOffline(ctx, threshold)
}
}
}()
}
// Heartbeat updates the node status when an agent reports in.
func (s *NodeService) Heartbeat(ctx context.Context, token string, hostname string, ip string, agentVer string) error {
func (s *NodeService) Heartbeat(ctx context.Context, token string, hostname string, ip string, agentVer string, osName string, archName string) error {
node, err := s.repo.FindByToken(ctx, token)
if err != nil {
return err
@@ -211,12 +283,36 @@ func (s *NodeService) Heartbeat(ctx context.Context, token string, hostname stri
node.Hostname = hostname
node.IPAddress = ip
node.AgentVer = agentVer
node.OS = runtime.GOOS
node.Arch = runtime.GOARCH
if strings.TrimSpace(osName) != "" {
node.OS = osName
} else {
node.OS = runtime.GOOS
}
if strings.TrimSpace(archName) != "" {
node.Arch = archName
} else {
node.Arch = runtime.GOARCH
}
node.LastSeen = time.Now().UTC()
return s.repo.Update(ctx, node)
}
// Update 编辑节点名称。
func (s *NodeService) Update(ctx context.Context, id uint, input NodeUpdateInput) (*NodeSummary, error) {
node, err := s.repo.FindByID(ctx, id)
if err != nil {
return nil, err
}
if node == nil {
return nil, apperror.New(http.StatusNotFound, "NODE_NOT_FOUND", "节点不存在", nil)
}
node.Name = strings.TrimSpace(input.Name)
if err := s.repo.Update(ctx, node); err != nil {
return nil, err
}
return s.Get(ctx, id)
}
// DirEntry represents a file or directory in a node's file system.
type DirEntry struct {
Name string `json:"name"`
@@ -225,6 +321,58 @@ type DirEntry struct {
Size int64 `json:"size"`
}
// remoteListDirectory 通过命令队列下发 list_dir 给 Agent 并同步等待结果。
// Agent 必须在线,且响应需在 15s 内返回,否则返回超时错误。
func (s *NodeService) remoteListDirectory(ctx context.Context, node *model.Node, path string) ([]DirEntry, error) {
if s.agentRPC == nil {
return nil, apperror.BadRequest("NODE_REMOTE_FS_NOT_SUPPORTED", "远程目录浏览未启用,需要 Master 启用 Agent 服务", nil)
}
if node.Status != model.NodeStatusOnline {
return nil, apperror.BadRequest("NODE_OFFLINE", "节点当前不在线,无法浏览其目录", nil)
}
if strings.TrimSpace(path) == "" {
path = "/"
}
cmdID, err := s.agentRPC.EnqueueCommand(ctx, node.ID, model.AgentCommandTypeListDir, map[string]any{"path": path})
if err != nil {
return nil, apperror.Internal("AGENT_COMMAND_ENQUEUE_FAILED", "下发目录浏览命令失败", err)
}
cmd, err := s.agentRPC.WaitForCommandResult(ctx, cmdID, 15*time.Second)
if err != nil {
return nil, err
}
if cmd.Status != model.AgentCommandStatusSucceeded {
msg := cmd.ErrorMessage
if msg == "" {
msg = fmt.Sprintf("command status: %s", cmd.Status)
}
return nil, apperror.BadRequest("NODE_FS_READ_ERROR", fmt.Sprintf("远程目录浏览失败: %s", msg), nil)
}
var result struct {
Entries []DirEntry `json:"entries"`
}
if err := json.Unmarshal([]byte(cmd.Result), &result); err != nil {
return nil, apperror.Internal("AGENT_RESULT_INVALID", "Agent 返回结果格式错误", err)
}
return result.Entries, nil
}
// detectLocalIP 获取本机第一个非回环 IPv4 地址。
func detectLocalIP() string {
addrs, err := net.InterfaceAddrs()
if err != nil {
return ""
}
for _, addr := range addrs {
if ipNet, ok := addr.(*net.IPNet); ok && !ipNet.IP.IsLoopback() {
if ipNet.IP.To4() != nil {
return ipNet.IP.String()
}
}
}
return ""
}
func generateToken() (string, error) {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {

View File

@@ -0,0 +1,52 @@
package service
import (
"io"
"sync/atomic"
"time"
)
// progressCallback 在每次读取时被调用,报告已读字节数和估算速率。
type progressCallback func(bytesRead int64, speedBps float64)
// progressReader 包装 io.Reader定期通过回调报告传输进度。
type progressReader struct {
reader io.Reader
total int64
read atomic.Int64
callback progressCallback
startTime time.Time
lastCall time.Time
interval time.Duration
}
func newProgressReader(reader io.Reader, total int64, callback progressCallback) *progressReader {
now := time.Now()
return &progressReader{
reader: reader,
total: total,
callback: callback,
startTime: now,
lastCall: now,
interval: 500 * time.Millisecond,
}
}
func (r *progressReader) Read(p []byte) (int, error) {
n, err := r.reader.Read(p)
if n > 0 {
current := r.read.Add(int64(n))
now := time.Now()
isFinal := err == io.EOF || (r.total > 0 && current >= r.total)
if isFinal || now.Sub(r.lastCall) >= r.interval {
r.lastCall = now
elapsed := now.Sub(r.startTime).Seconds()
speed := float64(0)
if elapsed > 0 {
speed = float64(current) / elapsed
}
r.callback(current, speed)
}
}
return n, err
}

View File

@@ -22,6 +22,7 @@ var settingsKeys = []string{
"language",
"timezone",
"backup_notification_enabled",
"bandwidth_limit",
}
func (s *SettingsService) GetAll(ctx context.Context) (map[string]string, error) {

View File

@@ -21,7 +21,7 @@ import (
type StorageTargetUpsertInput struct {
Name string `json:"name" binding:"required,min=1,max=128"`
Type string `json:"type" binding:"required,oneof=local_disk google_drive s3 webdav"`
Type string `json:"type" binding:"required,min=1"`
Description string `json:"description" binding:"max=255"`
Enabled bool `json:"enabled"`
Config map[string]any `json:"config" binding:"required"`
@@ -53,6 +53,7 @@ type StorageTargetSummary struct {
Type string `json:"type"`
Description string `json:"description"`
Enabled bool `json:"enabled"`
Starred bool `json:"starred"`
ConfigVersion int `json:"configVersion"`
LastTestedAt *time.Time `json:"lastTestedAt"`
LastTestStatus string `json:"lastTestStatus"`
@@ -209,6 +210,22 @@ func (s *StorageTargetService) Delete(ctx context.Context, id uint) error {
return nil
}
func (s *StorageTargetService) ToggleStar(ctx context.Context, id uint) (*StorageTargetSummary, error) {
item, err := s.targets.FindByID(ctx, id)
if err != nil {
return nil, apperror.Internal("STORAGE_TARGET_GET_FAILED", "无法获取存储目标详情", err)
}
if item == nil {
return nil, apperror.New(http.StatusNotFound, "STORAGE_TARGET_NOT_FOUND", "存储目标不存在", fmt.Errorf("storage target %d not found", id))
}
item.Starred = !item.Starred
if err := s.targets.Update(ctx, item); err != nil {
return nil, apperror.Internal("STORAGE_TARGET_UPDATE_FAILED", "无法更新存储目标收藏状态", err)
}
summary := toStorageTargetSummary(item)
return &summary, nil
}
func (s *StorageTargetService) TestConnection(ctx context.Context, input StorageTargetTestInput) error {
item, err := s.buildStorageTargetForTest(ctx, input)
if err != nil {
@@ -493,6 +510,7 @@ func toStorageTargetSummary(item *model.StorageTarget) StorageTargetSummary {
Type: item.Type,
Description: item.Description,
Enabled: item.Enabled,
Starred: item.Starred,
ConfigVersion: item.ConfigVersion,
LastTestedAt: item.LastTestedAt,
LastTestStatus: item.LastTestStatus,
@@ -526,10 +544,11 @@ func cloneMap(source map[string]any) map[string]any {
}
type StorageTargetUsage struct {
TargetID uint `json:"targetId"`
TargetName string `json:"targetName"`
RecordCount int64 `json:"recordCount"`
TotalSize int64 `json:"totalSize"`
TargetID uint `json:"targetId"`
TargetName string `json:"targetName"`
RecordCount int64 `json:"recordCount"`
TotalSize int64 `json:"totalSize"`
DiskUsage *storage.StorageUsageInfo `json:"diskUsage,omitempty"`
}
func (s *StorageTargetService) GetUsage(ctx context.Context, id uint) (*StorageTargetUsage, error) {
@@ -552,5 +571,16 @@ func (s *StorageTargetService) GetUsage(ctx context.Context, id uint) (*StorageT
}
}
}
// 尝试查询远端真实存储空间(部分后端如 local/Google Drive/WebDAV 支持)
configMap := map[string]any{}
if decryptErr := s.cipher.DecryptJSON(target.ConfigCiphertext, &configMap); decryptErr == nil {
if provider, createErr := s.registry.Create(ctx, target.Type, configMap); createErr == nil {
if abouter, ok := provider.(storage.StorageAbout); ok {
if diskUsage, aboutErr := abouter.About(ctx); aboutErr == nil {
result.DiskUsage = diskUsage
}
}
}
}
return result, nil
}

View File

@@ -2,7 +2,12 @@ package service
import (
"context"
"encoding/json"
"fmt"
"net/http"
"path/filepath"
"runtime"
"strings"
"syscall"
"time"
@@ -30,6 +35,82 @@ func NewSystemService(cfg config.Config, version string, startedAt time.Time) *S
return &SystemService{cfg: cfg, version: version, startedAt: startedAt}
}
// UpdateCheckResult 描述版本更新检查结果。
type UpdateCheckResult struct {
CurrentVersion string `json:"currentVersion"`
LatestVersion string `json:"latestVersion"`
HasUpdate bool `json:"hasUpdate"`
ReleaseURL string `json:"releaseUrl,omitempty"`
ReleaseNotes string `json:"releaseNotes,omitempty"`
PublishedAt string `json:"publishedAt,omitempty"`
DownloadURL string `json:"downloadUrl,omitempty"`
DockerImage string `json:"dockerImage,omitempty"`
}
const githubRepoAPI = "https://api.github.com/repos/Awuqing/BackupX/releases/latest"
// CheckUpdate 从 GitHub Releases 检查是否有新版本。
func (s *SystemService) CheckUpdate(ctx context.Context) (*UpdateCheckResult, error) {
result := &UpdateCheckResult{
CurrentVersion: s.version,
DockerImage: "awuqing/backupx",
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, githubRepoAPI, nil)
if err != nil {
return result, fmt.Errorf("create request: %w", err)
}
req.Header.Set("Accept", "application/vnd.github.v3+json")
req.Header.Set("User-Agent", "BackupX/"+s.version)
client := &http.Client{Timeout: 15 * time.Second}
resp, err := client.Do(req)
if err != nil {
return result, fmt.Errorf("fetch latest release: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return result, fmt.Errorf("github api returned %d", resp.StatusCode)
}
var release struct {
TagName string `json:"tag_name"`
HTMLURL string `json:"html_url"`
Body string `json:"body"`
Published string `json:"published_at"`
Assets []struct {
Name string `json:"name"`
BrowserDownloadURL string `json:"browser_download_url"`
} `json:"assets"`
}
if err := json.NewDecoder(resp.Body).Decode(&release); err != nil {
return result, fmt.Errorf("decode release: %w", err)
}
result.LatestVersion = release.TagName
result.ReleaseURL = release.HTMLURL
result.ReleaseNotes = release.Body
result.PublishedAt = release.Published
// 比较版本号(去 v 前缀后字符串比较)
current := strings.TrimPrefix(s.version, "v")
latest := strings.TrimPrefix(release.TagName, "v")
result.HasUpdate = latest > current && current != "dev"
// 匹配当前平台的下载链接
goos := runtime.GOOS
goarch := runtime.GOARCH
suffix := fmt.Sprintf("%s-%s.tar.gz", goos, goarch)
for _, asset := range release.Assets {
if strings.HasSuffix(asset.Name, suffix) {
result.DownloadURL = asset.BrowserDownloadURL
break
}
}
return result, nil
}
func (s *SystemService) GetInfo(_ context.Context) *SystemInfo {
now := time.Now().UTC()
info := &SystemInfo{
@@ -51,3 +132,4 @@ func (s *SystemService) GetInfo(_ context.Context) *SystemInfo {
}
return info
}

View File

@@ -1,66 +0,0 @@
// Package aliyun provides an Aliyun OSS storage factory that delegates to the S3-compatible engine.
// Aliyun OSS is fully S3-compatible; we auto-assemble the endpoint from the user-provided region.
package aliyun
import (
"context"
"fmt"
"strings"
"backupx/server/internal/storage"
"backupx/server/internal/storage/s3"
)
// Config is the user-facing configuration for Aliyun OSS.
type Config struct {
Region string `json:"region"`
Bucket string `json:"bucket"`
AccessKeyID string `json:"accessKeyId"`
SecretAccessKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"` // optional override
InternalNetwork bool `json:"internalNetwork"` // use -internal endpoint
}
// Factory creates Aliyun OSS providers by composing the S3 engine.
type Factory struct {
s3Factory s3.Factory
}
func NewFactory() Factory {
return Factory{s3Factory: s3.NewFactory()}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeAliyunOSS }
func (Factory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
func (f Factory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[Config](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("aliyun oss region is required")
}
suffix := "aliyuncs.com"
if cfg.InternalNetwork {
endpoint = fmt.Sprintf("https://oss-%s-internal.%s", region, suffix)
} else {
endpoint = fmt.Sprintf("https://oss-%s.%s", region, suffix)
}
}
// Delegate to S3 engine with assembled endpoint.
s3Config := map[string]any{
"endpoint": endpoint,
"region": cfg.Region,
"bucket": cfg.Bucket,
"accessKeyId": cfg.AccessKeyID,
"secretAccessKey": cfg.SecretAccessKey,
"forcePathStyle": false, // Aliyun OSS uses virtual-hosted style
}
return f.s3Factory.New(ctx, s3Config)
}

View File

@@ -1,226 +0,0 @@
package ftp
import (
"bytes"
"context"
"fmt"
"io"
"path"
"strings"
"time"
"backupx/server/internal/storage"
"github.com/jlaffaye/ftp"
)
// Provider implements storage.StorageProvider for FTP.
type Provider struct {
config storage.FTPConfig
}
// Factory creates FTP storage providers.
type Factory struct{}
// NewFactory returns a new FTP Factory.
func NewFactory() Factory {
return Factory{}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeFTP }
func (Factory) SensitiveFields() []string { return []string{"username", "password"} }
func (f Factory) New(_ context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.FTPConfig](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.Host) == "" {
return nil, fmt.Errorf("FTP host is required")
}
if cfg.Port == 0 {
cfg.Port = 21
}
return &Provider{config: cfg}, nil
}
func (p *Provider) Type() storage.ProviderType { return storage.ProviderTypeFTP }
// dial establishes a connection to the FTP server and logs in.
func (p *Provider) dial() (*ftp.ServerConn, error) {
addr := fmt.Sprintf("%s:%d", p.config.Host, p.config.Port)
var opts []ftp.DialOption
opts = append(opts, ftp.DialWithTimeout(30*time.Second))
if p.config.UseTLS {
opts = append(opts, ftp.DialWithExplicitTLS(nil))
}
conn, err := ftp.Dial(addr, opts...)
if err != nil {
return nil, fmt.Errorf("connect to FTP server %s: %w", addr, err)
}
username := p.config.Username
if username == "" {
username = "anonymous"
}
if err := conn.Login(username, p.config.Password); err != nil {
conn.Quit()
return nil, fmt.Errorf("FTP login: %w", err)
}
return conn, nil
}
func (p *Provider) TestConnection(_ context.Context) error {
conn, err := p.dial()
if err != nil {
return err
}
defer conn.Quit()
basePath := p.normalizeBasePath()
if err := p.ensureDir(conn, basePath); err != nil {
return fmt.Errorf("ensure FTP base path: %w", err)
}
_, err = conn.List(basePath)
if err != nil {
return fmt.Errorf("list FTP base path: %w", err)
}
return nil
}
func (p *Provider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
conn, err := p.dial()
if err != nil {
return err
}
defer conn.Quit()
objectPath := p.resolvePath(objectKey)
dir := path.Dir(objectPath)
if err := p.ensureDir(conn, dir); err != nil {
return fmt.Errorf("create FTP directories: %w", err)
}
// Read all data into buffer since FTP STOR needs the full stream
data, err := io.ReadAll(reader)
if err != nil {
return fmt.Errorf("read upload data: %w", err)
}
if err := conn.Stor(objectPath, bytes.NewReader(data)); err != nil {
return fmt.Errorf("FTP upload: %w", err)
}
return nil
}
func (p *Provider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
conn, err := p.dial()
if err != nil {
return nil, err
}
objectPath := p.resolvePath(objectKey)
resp, err := conn.Retr(objectPath)
if err != nil {
conn.Quit()
return nil, fmt.Errorf("FTP download: %w", err)
}
// Wrap the response to also close the FTP connection when done
return &ftpReadCloser{ReadCloser: resp, conn: conn}, nil
}
func (p *Provider) Delete(_ context.Context, objectKey string) error {
conn, err := p.dial()
if err != nil {
return err
}
defer conn.Quit()
objectPath := p.resolvePath(objectKey)
if err := conn.Delete(objectPath); err != nil {
return fmt.Errorf("FTP delete: %w", err)
}
return nil
}
func (p *Provider) List(_ context.Context, prefix string) ([]storage.ObjectInfo, error) {
conn, err := p.dial()
if err != nil {
return nil, err
}
defer conn.Quit()
basePath := p.normalizeBasePath()
entries, err := conn.List(basePath)
if err != nil {
return nil, fmt.Errorf("FTP list: %w", err)
}
items := make([]storage.ObjectInfo, 0, len(entries))
for _, entry := range entries {
if entry.Type == ftp.EntryTypeFolder {
continue
}
key := strings.TrimPrefix(path.Join(strings.TrimPrefix(basePath, "/"), entry.Name), "/")
if prefix != "" && !strings.HasPrefix(key, prefix) {
continue
}
items = append(items, storage.ObjectInfo{
Key: key,
Size: int64(entry.Size),
UpdatedAt: entry.Time.UTC(),
})
}
return items, nil
}
// normalizeBasePath returns a cleaned base path with leading slash.
func (p *Provider) normalizeBasePath() string {
clean := path.Clean("/" + strings.TrimSpace(p.config.BasePath))
if clean == "." {
return "/"
}
return clean
}
// resolvePath returns the full FTP path for the given object key.
func (p *Provider) resolvePath(objectKey string) string {
cleanKey := path.Clean("/" + strings.TrimSpace(objectKey))
return path.Clean(path.Join(p.normalizeBasePath(), cleanKey))
}
// ensureDir creates all directories in the path recursively.
func (p *Provider) ensureDir(conn *ftp.ServerConn, dirPath string) error {
parts := strings.Split(strings.Trim(dirPath, "/"), "/")
current := ""
for _, part := range parts {
if part == "" {
continue
}
current = current + "/" + part
if err := conn.MakeDir(current); err != nil {
// Ignore errors if directory already exists
// FTP doesn't have a standard "mkdir if not exists"
_ = err
}
}
return nil
}
// ftpReadCloser wraps an io.ReadCloser from FTP and closes the connection when done.
type ftpReadCloser struct {
io.ReadCloser
conn *ftp.ServerConn
}
func (f *ftpReadCloser) Close() error {
err := f.ReadCloser.Close()
if f.conn != nil {
f.conn.Quit()
}
return err
}

View File

@@ -1,299 +0,0 @@
package googledrive
import (
"context"
"fmt"
"io"
"path"
"strings"
"time"
"backupx/server/internal/storage"
"golang.org/x/oauth2"
googleoauth "golang.org/x/oauth2/google"
"google.golang.org/api/drive/v3"
"google.golang.org/api/option"
)
type fileInfo struct {
ID string
Name string
Size int64
ModifiedTime time.Time
}
type client interface {
TestConnection(context.Context, string) error
Upload(context.Context, string, string, io.Reader) error
Download(context.Context, string, string) (io.ReadCloser, error)
Delete(context.Context, string, string) error
List(context.Context, string, string) ([]storage.ObjectInfo, error)
EnsureFolder(ctx context.Context, parentID, name string) (string, error)
}
type Provider struct {
client client
rootFolder string // user-configured folderId, empty means Drive root
folderCache map[string]string // cache: path -> folderID
}
type Factory struct {
newClient func(context.Context, storage.GoogleDriveConfig) (client, error)
}
func NewFactory() Factory {
return Factory{newClient: newDriveClient}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeGoogleDrive }
func (Factory) SensitiveFields() []string {
return []string{"clientId", "clientSecret", "refreshToken"}
}
func (f Factory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.GoogleDriveConfig](rawConfig)
if err != nil {
return nil, err
}
cfg = cfg.Normalize()
if strings.TrimSpace(cfg.ClientID) == "" || strings.TrimSpace(cfg.ClientSecret) == "" {
return nil, fmt.Errorf("google drive client credentials are required")
}
if strings.TrimSpace(cfg.RefreshToken) == "" {
return nil, fmt.Errorf("google drive refresh token is required")
}
newClient := f.newClient
if newClient == nil {
newClient = NewFactory().newClient
}
client, err := newClient(ctx, cfg)
if err != nil {
return nil, err
}
return &Provider{
client: client,
rootFolder: strings.TrimSpace(cfg.FolderID),
folderCache: make(map[string]string),
}, nil
}
func (p *Provider) Type() storage.ProviderType { return storage.ProviderTypeGoogleDrive }
// ensureFolderPath creates nested folders for a path like "BackupX/file/260308"
// and returns the deepest folder's ID.
func (p *Provider) ensureFolderPath(ctx context.Context, folderPath string) (string, error) {
if folderPath == "" || folderPath == "." {
return p.rootFolder, nil
}
if cached, ok := p.folderCache[folderPath]; ok {
return cached, nil
}
parts := strings.Split(path.Clean(folderPath), "/")
parentID := p.rootFolder
builtPath := ""
for _, part := range parts {
if part == "" || part == "." {
continue
}
if builtPath == "" {
builtPath = part
} else {
builtPath = builtPath + "/" + part
}
if cached, ok := p.folderCache[builtPath]; ok {
parentID = cached
continue
}
folderID, err := p.client.EnsureFolder(ctx, parentID, part)
if err != nil {
return "", fmt.Errorf("ensure folder %s: %w", builtPath, err)
}
p.folderCache[builtPath] = folderID
parentID = folderID
}
return parentID, nil
}
func (p *Provider) TestConnection(ctx context.Context) error {
return p.client.TestConnection(ctx, p.rootFolder)
}
func (p *Provider) Upload(ctx context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
dir := path.Dir(objectKey)
folderID, err := p.ensureFolderPath(ctx, dir)
if err != nil {
return err
}
return p.client.Upload(ctx, folderID, objectKey, reader)
}
func (p *Provider) Download(ctx context.Context, objectKey string) (io.ReadCloser, error) {
dir := path.Dir(objectKey)
folderID, err := p.ensureFolderPath(ctx, dir)
if err != nil {
return nil, err
}
return p.client.Download(ctx, folderID, objectKey)
}
func (p *Provider) Delete(ctx context.Context, objectKey string) error {
dir := path.Dir(objectKey)
folderID, err := p.ensureFolderPath(ctx, dir)
if err != nil {
return err
}
return p.client.Delete(ctx, folderID, objectKey)
}
func (p *Provider) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
dir := path.Dir(prefix)
folderID, err := p.ensureFolderPath(ctx, dir)
if err != nil {
return nil, err
}
return p.client.List(ctx, folderID, prefix)
}
type driveClient struct {
service *drive.Service
}
func newDriveClient(ctx context.Context, cfg storage.GoogleDriveConfig) (client, error) {
cfg = cfg.Normalize()
oauthCfg := &oauth2.Config{
ClientID: cfg.ClientID,
ClientSecret: cfg.ClientSecret,
RedirectURL: cfg.RedirectURL,
Endpoint: googleoauth.Endpoint,
Scopes: []string{drive.DriveScope},
}
httpClient := oauthCfg.Client(ctx, &oauth2.Token{RefreshToken: cfg.RefreshToken})
service, err := drive.NewService(ctx, option.WithHTTPClient(httpClient))
if err != nil {
return nil, fmt.Errorf("create google drive service: %w", err)
}
return &driveClient{service: service}, nil
}
func (c *driveClient) TestConnection(ctx context.Context, folderID string) error {
if strings.TrimSpace(folderID) == "" {
_, err := c.service.About.Get().Fields("user").Context(ctx).Do()
if err != nil {
return fmt.Errorf("test google drive connection: %w", err)
}
return nil
}
_, err := c.service.Files.Get(folderID).Fields("id").Context(ctx).Do()
if err != nil {
return fmt.Errorf("test google drive folder: %w", err)
}
return nil
}
func (c *driveClient) EnsureFolder(ctx context.Context, parentID, name string) (string, error) {
// Search for existing folder
query := fmt.Sprintf("name = '%s' and mimeType = 'application/vnd.google-apps.folder' and trashed = false", escapeQuery(name))
if strings.TrimSpace(parentID) != "" {
query += fmt.Sprintf(" and '%s' in parents", escapeQuery(parentID))
} else {
query += " and 'root' in parents"
}
result, err := c.service.Files.List().Q(query).PageSize(1).Fields("files(id)").Context(ctx).Do()
if err != nil {
return "", fmt.Errorf("search for folder %s: %w", name, err)
}
if len(result.Files) > 0 {
return result.Files[0].Id, nil
}
// Create the folder
folder := &drive.File{
Name: name,
MimeType: "application/vnd.google-apps.folder",
}
if strings.TrimSpace(parentID) != "" {
folder.Parents = []string{parentID}
}
created, err := c.service.Files.Create(folder).Fields("id").Context(ctx).Do()
if err != nil {
return "", fmt.Errorf("create folder %s: %w", name, err)
}
return created.Id, nil
}
func (c *driveClient) Upload(ctx context.Context, folderID, objectKey string, reader io.Reader) error {
file := &drive.File{Name: path.Base(objectKey)}
if strings.TrimSpace(folderID) != "" {
file.Parents = []string{folderID}
}
_, err := c.service.Files.Create(file).Media(reader).Context(ctx).Do()
if err != nil {
return fmt.Errorf("upload google drive object: %w", err)
}
return nil
}
func (c *driveClient) Download(ctx context.Context, folderID, objectKey string) (io.ReadCloser, error) {
file, err := c.findFile(ctx, folderID, objectKey)
if err != nil {
return nil, err
}
response, err := c.service.Files.Get(file.ID).Context(ctx).Download()
if err != nil {
return nil, fmt.Errorf("download google drive object: %w", err)
}
return response.Body, nil
}
func (c *driveClient) Delete(ctx context.Context, folderID, objectKey string) error {
file, err := c.findFile(ctx, folderID, objectKey)
if err != nil {
return err
}
if err := c.service.Files.Delete(file.ID).Context(ctx).Do(); err != nil {
return fmt.Errorf("delete google drive object: %w", err)
}
return nil
}
func (c *driveClient) List(ctx context.Context, folderID, prefix string) ([]storage.ObjectInfo, error) {
query := "trashed = false"
if strings.TrimSpace(folderID) != "" {
query += fmt.Sprintf(" and '%s' in parents", escapeQuery(folderID))
}
if strings.TrimSpace(prefix) != "" {
query += fmt.Sprintf(" and name contains '%s'", escapeQuery(prefix))
}
result, err := c.service.Files.List().Q(query).Fields("files(id,name,size,modifiedTime)").Context(ctx).Do()
if err != nil {
return nil, fmt.Errorf("list google drive objects: %w", err)
}
items := make([]storage.ObjectInfo, 0, len(result.Files))
for _, file := range result.Files {
modifiedAt, _ := time.Parse(time.RFC3339, file.ModifiedTime)
items = append(items, storage.ObjectInfo{Key: file.Name, Size: file.Size, UpdatedAt: modifiedAt.UTC()})
}
return items, nil
}
func (c *driveClient) findFile(ctx context.Context, folderID, objectKey string) (*fileInfo, error) {
query := fmt.Sprintf("name = '%s' and trashed = false", escapeQuery(path.Base(objectKey)))
if strings.TrimSpace(folderID) != "" {
query += fmt.Sprintf(" and '%s' in parents", escapeQuery(folderID))
}
result, err := c.service.Files.List().Q(query).PageSize(1).Fields("files(id,name,size,modifiedTime)").Context(ctx).Do()
if err != nil {
return nil, fmt.Errorf("query google drive object: %w", err)
}
if len(result.Files) == 0 {
return nil, fmt.Errorf("google drive object not found: %s", objectKey)
}
file := result.Files[0]
modifiedAt, _ := time.Parse(time.RFC3339, file.ModifiedTime)
return &fileInfo{ID: file.Id, Name: file.Name, Size: file.Size, ModifiedTime: modifiedAt.UTC()}, nil
}
func escapeQuery(value string) string {
return strings.ReplaceAll(value, "'", "\\'")
}

View File

@@ -1,75 +0,0 @@
package googledrive
import (
"context"
"io"
"strings"
"testing"
"time"
"backupx/server/internal/storage"
)
type fakeClient struct{ data map[string]string }
func (c *fakeClient) TestConnection(context.Context, string) error { return nil }
func (c *fakeClient) Upload(_ context.Context, _ string, objectKey string, reader io.Reader) error {
content, _ := io.ReadAll(reader)
c.data[objectKey] = string(content)
return nil
}
func (c *fakeClient) Download(_ context.Context, _ string, objectKey string) (io.ReadCloser, error) {
return io.NopCloser(strings.NewReader(c.data[objectKey])), nil
}
func (c *fakeClient) Delete(_ context.Context, _ string, objectKey string) error {
delete(c.data, objectKey)
return nil
}
func (c *fakeClient) List(_ context.Context, _ string, prefix string) ([]storage.ObjectInfo, error) {
items := make([]storage.ObjectInfo, 0)
for key, value := range c.data {
if prefix == "" || strings.HasPrefix(key, prefix) {
items = append(items, storage.ObjectInfo{Key: key, Size: int64(len(value)), UpdatedAt: time.Now().UTC()})
}
}
return items, nil
}
func (c *fakeClient) EnsureFolder(_ context.Context, _, name string) (string, error) {
return "fake-folder-" + name, nil
}
func TestGoogleDriveProviderCRUD(t *testing.T) {
factory := Factory{newClient: func(context.Context, storage.GoogleDriveConfig) (client, error) {
return &fakeClient{data: make(map[string]string)}, nil
}}
providerAny, err := factory.New(context.Background(), map[string]any{"clientId": "id", "clientSecret": "secret", "refreshToken": "refresh"})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
provider := providerAny.(*Provider)
if err := provider.TestConnection(context.Background()); err != nil {
t.Fatalf("TestConnection returned error: %v", err)
}
if err := provider.Upload(context.Background(), "backup.tar.gz", strings.NewReader("payload"), 7, nil); err != nil {
t.Fatalf("Upload returned error: %v", err)
}
reader, err := provider.Download(context.Background(), "backup.tar.gz")
if err != nil {
t.Fatalf("Download returned error: %v", err)
}
defer reader.Close()
content, _ := io.ReadAll(reader)
if string(content) != "payload" {
t.Fatalf("unexpected content: %s", string(content))
}
items, err := provider.List(context.Background(), "backup")
if err != nil {
t.Fatalf("List returned error: %v", err)
}
if len(items) != 1 || items[0].Key != "backup.tar.gz" {
t.Fatalf("unexpected list result: %#v", items)
}
if err := provider.Delete(context.Background(), "backup.tar.gz"); err != nil {
t.Fatalf("Delete returned error: %v", err)
}
}

View File

@@ -1,137 +0,0 @@
package localdisk
import (
"context"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"backupx/server/internal/storage"
)
type Provider struct {
basePath string
}
type Factory struct{}
func NewFactory() Factory { return Factory{} }
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeLocalDisk }
func (Factory) SensitiveFields() []string { return nil }
func (Factory) New(_ context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.LocalDiskConfig](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.BasePath) == "" {
return nil, fmt.Errorf("local disk basePath is required")
}
return &Provider{basePath: filepath.Clean(cfg.BasePath)}, nil
}
func (p *Provider) Type() storage.ProviderType { return storage.ProviderTypeLocalDisk }
func (p *Provider) TestConnection(_ context.Context) error {
if err := os.MkdirAll(p.basePath, 0o755); err != nil {
return fmt.Errorf("ensure local disk base path: %w", err)
}
tempFile, err := os.CreateTemp(p.basePath, ".backupx-connection-test-*")
if err != nil {
return fmt.Errorf("write access check failed: %w", err)
}
name := tempFile.Name()
_ = tempFile.Close()
_ = os.Remove(name)
return nil
}
func (p *Provider) Upload(_ context.Context, objectKey string, reader io.Reader, _ int64, _ map[string]string) error {
targetPath, err := p.resolvePath(objectKey)
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
return fmt.Errorf("create local disk directories: %w", err)
}
file, err := os.Create(targetPath)
if err != nil {
return fmt.Errorf("create local disk object: %w", err)
}
defer file.Close()
if _, err := io.Copy(file, reader); err != nil {
return fmt.Errorf("write local disk object: %w", err)
}
return nil
}
func (p *Provider) Download(_ context.Context, objectKey string) (io.ReadCloser, error) {
targetPath, err := p.resolvePath(objectKey)
if err != nil {
return nil, err
}
file, err := os.Open(targetPath)
if err != nil {
return nil, fmt.Errorf("open local disk object: %w", err)
}
return file, nil
}
func (p *Provider) Delete(_ context.Context, objectKey string) error {
targetPath, err := p.resolvePath(objectKey)
if err != nil {
return err
}
if err := os.Remove(targetPath); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("delete local disk object: %w", err)
}
return nil
}
func (p *Provider) List(_ context.Context, prefix string) ([]storage.ObjectInfo, error) {
items := make([]storage.ObjectInfo, 0)
err := filepath.WalkDir(p.basePath, func(path string, entry fs.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if entry.IsDir() {
return nil
}
rel, err := filepath.Rel(p.basePath, path)
if err != nil {
return err
}
key := filepath.ToSlash(rel)
if prefix != "" && !strings.HasPrefix(key, prefix) {
return nil
}
info, err := entry.Info()
if err != nil {
return err
}
items = append(items, storage.ObjectInfo{Key: key, Size: info.Size(), UpdatedAt: info.ModTime().UTC()})
return nil
})
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("list local disk objects: %w", err)
}
return items, nil
}
func (p *Provider) resolvePath(objectKey string) (string, error) {
cleanBase := filepath.Clean(p.basePath)
cleanKey := filepath.Clean(filepath.FromSlash(strings.TrimSpace(objectKey)))
if cleanKey == "." || cleanKey == string(filepath.Separator) || cleanKey == "" {
return "", fmt.Errorf("object key is required")
}
fullPath := filepath.Clean(filepath.Join(cleanBase, cleanKey))
baseWithSep := cleanBase + string(filepath.Separator)
if fullPath != cleanBase && !strings.HasPrefix(fullPath, baseWithSep) {
return "", fmt.Errorf("object key escapes base path")
}
return fullPath, nil
}

View File

@@ -1,52 +0,0 @@
package localdisk
import (
"context"
"io"
"strings"
"testing"
)
func TestLocalDiskProviderCRUD(t *testing.T) {
providerAny, err := (Factory{}).New(context.Background(), map[string]any{"basePath": t.TempDir()})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
provider := providerAny.(*Provider)
if err := provider.TestConnection(context.Background()); err != nil {
t.Fatalf("TestConnection returned error: %v", err)
}
if err := provider.Upload(context.Background(), "daily/backup.txt", strings.NewReader("hello"), 5, nil); err != nil {
t.Fatalf("Upload returned error: %v", err)
}
reader, err := provider.Download(context.Background(), "daily/backup.txt")
if err != nil {
t.Fatalf("Download returned error: %v", err)
}
defer reader.Close()
content, _ := io.ReadAll(reader)
if string(content) != "hello" {
t.Fatalf("expected downloaded content to match, got %s", string(content))
}
items, err := provider.List(context.Background(), "daily")
if err != nil {
t.Fatalf("List returned error: %v", err)
}
if len(items) != 1 || items[0].Key != "daily/backup.txt" {
t.Fatalf("unexpected list result: %#v", items)
}
if err := provider.Delete(context.Background(), "daily/backup.txt"); err != nil {
t.Fatalf("Delete returned error: %v", err)
}
}
func TestLocalDiskProviderRejectsTraversal(t *testing.T) {
providerAny, err := (Factory{}).New(context.Background(), map[string]any{"basePath": t.TempDir()})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
provider := providerAny.(*Provider)
if _, err := provider.resolvePath("../escape.txt"); err == nil {
t.Fatalf("expected traversal to be rejected")
}
}

View File

@@ -1,73 +0,0 @@
// Package qiniu provides a Qiniu Cloud Kodo storage factory that delegates to the S3-compatible engine.
// Qiniu Kodo is S3-compatible; we auto-assemble the endpoint from the user-provided region.
package qiniu
import (
"context"
"fmt"
"strings"
"backupx/server/internal/storage"
"backupx/server/internal/storage/s3"
)
// Config is the user-facing configuration for Qiniu Kodo.
type Config struct {
Region string `json:"region"` // e.g. z0, z1, z2, na0, as0
Bucket string `json:"bucket"`
AccessKey string `json:"accessKeyId"`
SecretKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"` // optional override
}
// regionEndpoints maps Qiniu storage region codes to their S3-compatible endpoints.
var regionEndpoints = map[string]string{
"z0": "https://s3-cn-east-1.qiniucs.com",
"cn-east-2": "https://s3-cn-east-2.qiniucs.com",
"z1": "https://s3-cn-north-1.qiniucs.com",
"z2": "https://s3-cn-south-1.qiniucs.com",
"na0": "https://s3-us-north-1.qiniucs.com",
"as0": "https://s3-ap-southeast-1.qiniucs.com",
}
// Factory creates Qiniu Kodo providers by composing the S3 engine.
type Factory struct {
s3Factory s3.Factory
}
func NewFactory() Factory {
return Factory{s3Factory: s3.NewFactory()}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeQiniuKodo }
func (Factory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
func (f Factory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[Config](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("qiniu kodo region is required")
}
var ok bool
endpoint, ok = regionEndpoints[region]
if !ok {
return nil, fmt.Errorf("unsupported qiniu region: %s (supported: z0, cn-east-2, z1, z2, na0, as0)", region)
}
}
s3Config := map[string]any{
"endpoint": endpoint,
"region": cfg.Region,
"bucket": cfg.Bucket,
"accessKeyId": cfg.AccessKey,
"secretAccessKey": cfg.SecretKey,
"forcePathStyle": true, // Qiniu S3-compatible uses path-style
}
return f.s3Factory.New(ctx, s3Config)
}

View File

@@ -0,0 +1,5 @@
// Package rclone 提供基于 rclone 的统一存储后端实现。
// 引入全部 rclone backend支持 70+ 存储后端。
package rclone
import _ "github.com/rclone/rclone/backend/all"

View File

@@ -0,0 +1,36 @@
package rclone
import (
"context"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
)
// TransferConfig 控制 rclone 传输层行为。
type TransferConfig struct {
LowLevelRetries int // 底层 HTTP 请求重试次数0 保持 rclone 默认10
BandwidthLimit string // 带宽限制,如 "10M"、"1M:500k"(上传:下载),空或 "0" 不限
}
// ConfiguredContext 返回注入了 rclone 传输配置的 context。
// 各 rclone 后端在 fs.NewFs 时读取 context 中的配置,自动应用重试和限速。
func ConfiguredContext(ctx context.Context, cfg TransferConfig) context.Context {
ctx, ci := fs.AddConfig(ctx)
if cfg.LowLevelRetries > 0 {
ci.LowLevelRetries = cfg.LowLevelRetries
}
if cfg.BandwidthLimit != "" && cfg.BandwidthLimit != "0" {
var bwTable fs.BwTimetable
if err := bwTable.Set(cfg.BandwidthLimit); err == nil {
ci.BwLimit = bwTable
}
}
return ctx
}
// StartAccounting 初始化 rclone 的传输统计和令牌桶限速系统。
// 应在应用启动时调用一次。
func StartAccounting(ctx context.Context) {
accounting.Start(ctx)
}

View File

@@ -0,0 +1,508 @@
package rclone
import (
"context"
"fmt"
"strings"
"backupx/server/internal/storage"
"github.com/rclone/rclone/fs"
)
// ---------------------------------------------------------------------------
// 辅助函数
// ---------------------------------------------------------------------------
// quoteParam 对 rclone 连接字符串中含特殊字符的值加单引号保护。
func quoteParam(s string) string {
if s == "" {
return s
}
if !strings.ContainsAny(s, ",:='") {
return s
}
return "'" + strings.ReplaceAll(s, "'", "''") + "'"
}
// newFs 创建 rclone fs.Fs 实例并包装为 Provider。
func newFs(ctx context.Context, providerType storage.ProviderType, remote string) (*Provider, error) {
rfs, err := fs.NewFs(ctx, remote)
if err != nil {
return nil, fmt.Errorf("create rclone fs for %s: %w", providerType, err)
}
return newProvider(providerType, rfs), nil
}
// ---------------------------------------------------------------------------
// LocalDisk
// ---------------------------------------------------------------------------
type LocalDiskFactory struct{}
func NewLocalDiskFactory() LocalDiskFactory { return LocalDiskFactory{} }
func (LocalDiskFactory) Type() storage.ProviderType { return storage.ProviderTypeLocalDisk }
func (LocalDiskFactory) SensitiveFields() []string { return nil }
func (LocalDiskFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.LocalDiskConfig](rawConfig)
if err != nil {
return nil, err
}
basePath := strings.TrimSpace(cfg.BasePath)
if basePath == "" {
return nil, fmt.Errorf("local disk basePath is required")
}
return newFs(ctx, storage.ProviderTypeLocalDisk, basePath)
}
// ---------------------------------------------------------------------------
// S3
// ---------------------------------------------------------------------------
type S3Factory struct{}
func NewS3Factory() S3Factory { return S3Factory{} }
func (S3Factory) Type() storage.ProviderType { return storage.ProviderTypeS3 }
func (S3Factory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
func (S3Factory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.S3Config](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.Bucket) == "" {
return nil, fmt.Errorf("s3 bucket is required")
}
if strings.TrimSpace(cfg.AccessKeyID) == "" || strings.TrimSpace(cfg.SecretAccessKey) == "" {
return nil, fmt.Errorf("s3 credentials are required")
}
return newFs(ctx, storage.ProviderTypeS3, buildS3Remote("Other", cfg.AccessKeyID, cfg.SecretAccessKey, cfg.Endpoint, cfg.Region, cfg.Bucket, cfg.ForcePathStyle))
}
// buildS3Remote 构建 S3 兼容存储的 rclone 连接字符串。
func buildS3Remote(provider, keyID, secret, endpoint, region, bucket string, pathStyle bool) string {
var b strings.Builder
b.WriteString(":s3,provider=")
b.WriteString(quoteParam(provider))
b.WriteString(",access_key_id=")
b.WriteString(quoteParam(keyID))
b.WriteString(",secret_access_key=")
b.WriteString(quoteParam(secret))
if strings.TrimSpace(endpoint) != "" {
b.WriteString(",endpoint=")
b.WriteString(quoteParam(strings.TrimRight(endpoint, "/")))
}
if strings.TrimSpace(region) != "" {
b.WriteString(",region=")
b.WriteString(quoteParam(region))
}
if pathStyle {
b.WriteString(",force_path_style=true")
}
b.WriteString(",env_auth=false,no_check_bucket=true:")
b.WriteString(bucket)
return b.String()
}
// ---------------------------------------------------------------------------
// WebDAV
// ---------------------------------------------------------------------------
type WebDAVFactory struct{}
func NewWebDAVFactory() WebDAVFactory { return WebDAVFactory{} }
func (WebDAVFactory) Type() storage.ProviderType { return storage.ProviderTypeWebDAV }
func (WebDAVFactory) SensitiveFields() []string { return []string{"username", "password"} }
func (WebDAVFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.WebDAVConfig](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.Endpoint) == "" {
return nil, fmt.Errorf("webdav endpoint is required")
}
remote := fmt.Sprintf(":webdav,url=%s,user=%s,pass=%s:%s",
quoteParam(strings.TrimRight(cfg.Endpoint, "/")),
quoteParam(cfg.Username),
quoteParam(cfg.Password),
strings.TrimSpace(cfg.BasePath))
return newFs(ctx, storage.ProviderTypeWebDAV, remote)
}
// ---------------------------------------------------------------------------
// Google Drive
// ---------------------------------------------------------------------------
type GoogleDriveFactory struct{}
func NewGoogleDriveFactory() GoogleDriveFactory { return GoogleDriveFactory{} }
func (GoogleDriveFactory) Type() storage.ProviderType { return storage.ProviderTypeGoogleDrive }
func (GoogleDriveFactory) SensitiveFields() []string {
return []string{"clientId", "clientSecret", "refreshToken"}
}
func (GoogleDriveFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.GoogleDriveConfig](rawConfig)
if err != nil {
return nil, err
}
cfg = cfg.Normalize()
if strings.TrimSpace(cfg.ClientID) == "" || strings.TrimSpace(cfg.ClientSecret) == "" {
return nil, fmt.Errorf("google drive client credentials are required")
}
if strings.TrimSpace(cfg.RefreshToken) == "" {
return nil, fmt.Errorf("google drive refresh token is required")
}
// 构造 rclone 所需的 OAuth2 token JSON
tokenJSON := fmt.Sprintf(`{"access_token":"","token_type":"Bearer","refresh_token":"%s","expiry":"0001-01-01T00:00:00Z"}`,
strings.ReplaceAll(cfg.RefreshToken, `"`, `\"`))
var b strings.Builder
b.WriteString(":drive,client_id=")
b.WriteString(quoteParam(cfg.ClientID))
b.WriteString(",client_secret=")
b.WriteString(quoteParam(cfg.ClientSecret))
b.WriteString(",token=")
b.WriteString(quoteParam(tokenJSON))
if strings.TrimSpace(cfg.FolderID) != "" {
b.WriteString(",root_folder_id=")
b.WriteString(quoteParam(cfg.FolderID))
}
b.WriteString(":")
return newFs(ctx, storage.ProviderTypeGoogleDrive, b.String())
}
// ---------------------------------------------------------------------------
// FTP
// ---------------------------------------------------------------------------
type FTPFactory struct{}
func NewFTPFactory() FTPFactory { return FTPFactory{} }
func (FTPFactory) Type() storage.ProviderType { return storage.ProviderTypeFTP }
func (FTPFactory) SensitiveFields() []string { return []string{"username", "password"} }
func (FTPFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.FTPConfig](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.Host) == "" {
return nil, fmt.Errorf("FTP host is required")
}
port := cfg.Port
if port == 0 {
port = 21
}
username := strings.TrimSpace(cfg.Username)
if username == "" {
username = "anonymous"
}
var b strings.Builder
b.WriteString(fmt.Sprintf(":ftp,host=%s,port=%d,user=%s,pass=%s",
quoteParam(cfg.Host), port, quoteParam(username), quoteParam(cfg.Password)))
if cfg.UseTLS {
b.WriteString(",tls=true,explicit_tls=true")
}
b.WriteString(":")
basePath := strings.TrimSpace(cfg.BasePath)
if basePath != "" {
b.WriteString(basePath)
}
return newFs(ctx, storage.ProviderTypeFTP, b.String())
}
// ---------------------------------------------------------------------------
// 阿里云 OSS委托 S3 引擎)
// ---------------------------------------------------------------------------
type AliyunOSSFactory struct{}
func NewAliyunOSSFactory() AliyunOSSFactory { return AliyunOSSFactory{} }
func (AliyunOSSFactory) Type() storage.ProviderType { return storage.ProviderTypeAliyunOSS }
func (AliyunOSSFactory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
// AliyunConfig 是阿里云 OSS 的用户配置。
type AliyunConfig struct {
Region string `json:"region"`
Bucket string `json:"bucket"`
AccessKeyID string `json:"accessKeyId"`
SecretAccessKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"`
InternalNetwork bool `json:"internalNetwork"`
}
func (AliyunOSSFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[AliyunConfig](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("aliyun oss region is required")
}
if cfg.InternalNetwork {
endpoint = fmt.Sprintf("https://oss-%s-internal.aliyuncs.com", region)
} else {
endpoint = fmt.Sprintf("https://oss-%s.aliyuncs.com", region)
}
}
return newFs(ctx, storage.ProviderTypeAliyunOSS, buildS3Remote("Alibaba", cfg.AccessKeyID, cfg.SecretAccessKey, endpoint, cfg.Region, cfg.Bucket, false))
}
// ---------------------------------------------------------------------------
// 腾讯云 COS委托 S3 引擎)
// ---------------------------------------------------------------------------
type TencentCOSFactory struct{}
func NewTencentCOSFactory() TencentCOSFactory { return TencentCOSFactory{} }
func (TencentCOSFactory) Type() storage.ProviderType { return storage.ProviderTypeTencentCOS }
func (TencentCOSFactory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
// TencentConfig 是腾讯云 COS 的用户配置。
type TencentConfig struct {
Region string `json:"region"`
Bucket string `json:"bucket"`
SecretID string `json:"accessKeyId"`
SecretKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"`
}
func (TencentCOSFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[TencentConfig](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("tencent cos region is required")
}
endpoint = fmt.Sprintf("https://cos.%s.myqcloud.com", region)
}
return newFs(ctx, storage.ProviderTypeTencentCOS, buildS3Remote("TencentCOS", cfg.SecretID, cfg.SecretKey, endpoint, cfg.Region, cfg.Bucket, false))
}
// ---------------------------------------------------------------------------
// 七牛云 Kodo委托 S3 引擎)
// ---------------------------------------------------------------------------
type QiniuKodoFactory struct{}
func NewQiniuKodoFactory() QiniuKodoFactory { return QiniuKodoFactory{} }
func (QiniuKodoFactory) Type() storage.ProviderType { return storage.ProviderTypeQiniuKodo }
func (QiniuKodoFactory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
// QiniuConfig 是七牛云 Kodo 的用户配置。
type QiniuConfig struct {
Region string `json:"region"`
Bucket string `json:"bucket"`
AccessKey string `json:"accessKeyId"`
SecretKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"`
}
// regionEndpoints 映射七牛区域代码到 S3 兼容 endpoint。
var regionEndpoints = map[string]string{
"z0": "https://s3-cn-east-1.qiniucs.com",
"cn-east-2": "https://s3-cn-east-2.qiniucs.com",
"z1": "https://s3-cn-north-1.qiniucs.com",
"z2": "https://s3-cn-south-1.qiniucs.com",
"na0": "https://s3-us-north-1.qiniucs.com",
"as0": "https://s3-ap-southeast-1.qiniucs.com",
}
func (QiniuKodoFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[QiniuConfig](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("qiniu kodo region is required")
}
var ok bool
endpoint, ok = regionEndpoints[region]
if !ok {
return nil, fmt.Errorf("unsupported qiniu region: %s (supported: z0, cn-east-2, z1, z2, na0, as0)", region)
}
}
return newFs(ctx, storage.ProviderTypeQiniuKodo, buildS3Remote("Qiniu", cfg.AccessKey, cfg.SecretKey, endpoint, cfg.Region, cfg.Bucket, true))
}
// ---------------------------------------------------------------------------
// 通用 Rclone 后端(支持全部 70+ 后端)
// ---------------------------------------------------------------------------
type RcloneFactory struct{}
func NewRcloneFactory() RcloneFactory { return RcloneFactory{} }
func (RcloneFactory) Type() storage.ProviderType { return storage.ProviderTypeRclone }
func (RcloneFactory) SensitiveFields() []string { return []string{"pass", "password", "secret_access_key", "client_secret", "token"} }
func (RcloneFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
backend, _ := rawConfig["backend"].(string)
backend = strings.TrimSpace(backend)
if backend == "" {
return nil, fmt.Errorf("rclone backend type is required")
}
root, _ := rawConfig["root"].(string)
root = strings.TrimSpace(root)
// 构建连接字符串::backend,key1=val1,key2=val2:root
var b strings.Builder
b.WriteString(":")
b.WriteString(backend)
for key, val := range rawConfig {
if key == "backend" || key == "root" {
continue
}
strVal := fmt.Sprintf("%v", val)
if strings.TrimSpace(strVal) == "" {
continue
}
b.WriteString(",")
b.WriteString(key)
b.WriteString("=")
b.WriteString(quoteParam(strVal))
}
b.WriteString(":")
b.WriteString(root)
return newFs(ctx, storage.ProviderTypeRclone, b.String())
}
// ListBackends 返回所有可用的 rclone 后端及其配置选项。
func ListBackends() []BackendInfo {
var backends []BackendInfo
for _, ri := range fs.Registry {
if ri.Name == "union" || ri.Name == "crypt" || ri.Name == "chunker" || ri.Name == "compress" || ri.Name == "hasher" || ri.Name == "combine" {
continue // 跳过组合/加密类后端
}
info := BackendInfo{
Name: ri.Name,
Description: ri.Description,
}
for _, opt := range ri.Options {
if opt.Hide != 0 {
continue
}
// 跳过 rclone 为每个后端自动添加的通用选项
if opt.Name == "description" {
continue
}
info.Options = append(info.Options, BackendOption{
Key: opt.Name,
Label: opt.Help,
Required: opt.Required,
IsPassword: opt.IsPassword,
})
}
backends = append(backends, info)
}
return backends
}
// BackendInfo 描述一个 rclone 后端。
type BackendInfo struct {
Name string `json:"name"`
Description string `json:"description"`
Options []BackendOption `json:"options"`
}
// BackendOption 描述一个后端配置选项。
type BackendOption struct {
Key string `json:"key"`
Label string `json:"label"`
Required bool `json:"required"`
IsPassword bool `json:"isPassword"`
}
// ---------------------------------------------------------------------------
// 通用 BackendFactory — 为任意 rclone 后端自动生成独立 Factory
// ---------------------------------------------------------------------------
// GenericBackendFactory 为单个 rclone 后端创建独立的 ProviderFactory。
// 用户存储目标的 type 直接是后端名(如 "sftp"),与 "s3"、"ftp" 完全平级。
type GenericBackendFactory struct {
backendType string
sensitive []string
}
// NewBackendFactory 为指定 rclone 后端创建一个 Factory。
func NewBackendFactory(backendType string) GenericBackendFactory {
var sensitive []string
for _, ri := range fs.Registry {
if ri.Name == backendType {
for _, opt := range ri.Options {
if opt.IsPassword {
sensitive = append(sensitive, opt.Name)
}
}
break
}
}
return GenericBackendFactory{backendType: backendType, sensitive: sensitive}
}
func (f GenericBackendFactory) Type() storage.ProviderType { return storage.ProviderType(f.backendType) }
func (f GenericBackendFactory) SensitiveFields() []string { return f.sensitive }
func (f GenericBackendFactory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
root, _ := rawConfig["root"].(string)
root = strings.TrimSpace(root)
var b strings.Builder
b.WriteString(":")
b.WriteString(f.backendType)
for key, val := range rawConfig {
if key == "root" {
continue
}
strVal := fmt.Sprintf("%v", val)
if strings.TrimSpace(strVal) == "" {
continue
}
b.WriteString(",")
b.WriteString(key)
b.WriteString("=")
b.WriteString(quoteParam(strVal))
}
b.WriteString(":")
b.WriteString(root)
return newFs(ctx, storage.ProviderType(f.backendType), b.String())
}
// RegisterAllBackends 将所有 rclone 后端注册为独立 Factory 到 Registry。
// 已存在的内置类型s3, ftp 等)不会被覆盖。
func RegisterAllBackends(registry *storage.Registry) {
builtinTypes := map[string]bool{
"local_disk": true, "s3": true, "webdav": true, "google_drive": true,
"ftp": true, "aliyun_oss": true, "tencent_cos": true, "qiniu_kodo": true,
"rclone": true, "local": true,
}
for _, info := range ListBackends() {
if builtinTypes[info.Name] {
continue
}
registry.Register(NewBackendFactory(info.Name))
}
}

View File

@@ -0,0 +1,165 @@
package rclone
import (
"context"
"fmt"
"io"
"sort"
"strings"
"time"
"backupx/server/internal/storage"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fs/walk"
)
// Provider 包装 rclone fs.Fs实现 storage.StorageProvider 接口。
type Provider struct {
providerType storage.ProviderType
rfs fs.Fs
}
func newProvider(providerType storage.ProviderType, rfs fs.Fs) *Provider {
return &Provider{providerType: providerType, rfs: rfs}
}
func (p *Provider) Type() storage.ProviderType { return p.providerType }
// TestConnection 验证连通性。对本地磁盘会先确保目录存在。
func (p *Provider) TestConnection(ctx context.Context) error {
// 确保根目录存在(本地磁盘等后端需要预创建)
if err := p.rfs.Mkdir(ctx, ""); err != nil {
return fmt.Errorf("rclone test connection (mkdir): %w", err)
}
_, err := p.rfs.List(ctx, "")
if err != nil {
return fmt.Errorf("rclone test connection: %w", err)
}
return nil
}
// Upload 通过 rclone fs.Fs.Put 上传文件。
func (p *Provider) Upload(ctx context.Context, objectKey string, reader io.Reader, size int64, _ map[string]string) error {
dir := pathDir(objectKey)
if dir != "" && dir != "." {
if err := p.rfs.Mkdir(ctx, dir); err != nil {
return fmt.Errorf("rclone mkdir %s: %w", dir, err)
}
}
info := object.NewStaticObjectInfo(objectKey, time.Now(), size, true, nil, nil)
if _, err := p.rfs.Put(ctx, reader, info); err != nil {
return fmt.Errorf("rclone upload %s: %w", objectKey, err)
}
return nil
}
// Download 通过 rclone 获取对象并返回 io.ReadCloser。
func (p *Provider) Download(ctx context.Context, objectKey string) (io.ReadCloser, error) {
obj, err := p.rfs.NewObject(ctx, objectKey)
if err != nil {
return nil, fmt.Errorf("rclone find object %s: %w", objectKey, err)
}
reader, err := obj.Open(ctx)
if err != nil {
return nil, fmt.Errorf("rclone download %s: %w", objectKey, err)
}
return reader, nil
}
// Delete 通过 rclone 删除远端对象。
func (p *Provider) Delete(ctx context.Context, objectKey string) error {
obj, err := p.rfs.NewObject(ctx, objectKey)
if err != nil {
return fmt.Errorf("rclone find object %s: %w", objectKey, err)
}
if err := obj.Remove(ctx); err != nil {
return fmt.Errorf("rclone delete %s: %w", objectKey, err)
}
return nil
}
// List 递归列出指定前缀下的所有对象。
func (p *Provider) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
var items []storage.ObjectInfo
err := walk.ListR(ctx, p.rfs, prefix, true, -1, walk.ListObjects, func(entries fs.DirEntries) error {
for _, entry := range entries {
obj, ok := entry.(fs.Object)
if !ok {
continue
}
key := obj.Remote()
if prefix != "" && !strings.HasPrefix(key, prefix) {
continue
}
items = append(items, storage.ObjectInfo{
Key: key,
Size: obj.Size(),
UpdatedAt: obj.ModTime(ctx),
})
}
return nil
})
if err != nil {
return nil, fmt.Errorf("rclone list %s: %w", prefix, err)
}
return items, nil
}
// About 查询远端存储空间。并非所有 rclone 后端都支持。
func (p *Provider) About(ctx context.Context) (*storage.StorageUsageInfo, error) {
about := p.rfs.Features().About
if about == nil {
return nil, fmt.Errorf("rclone about: backend %s does not support About", p.providerType)
}
usage, err := about(ctx)
if err != nil {
return nil, fmt.Errorf("rclone about: %w", err)
}
return &storage.StorageUsageInfo{
Total: usage.Total,
Used: usage.Used,
Free: usage.Free,
Objects: usage.Objects,
}, nil
}
// RemoveEmptyDirs 递归删除 prefix 下的空目录,从最深层开始。
// 非空目录删除会失败(安全忽略),仅清理真正的空目录。
func (p *Provider) RemoveEmptyDirs(ctx context.Context, prefix string) error {
var dirs []string
err := walk.ListR(ctx, p.rfs, prefix, true, -1, walk.ListDirs, func(entries fs.DirEntries) error {
for _, entry := range entries {
if _, ok := entry.(fs.Directory); ok {
dirs = append(dirs, entry.Remote())
}
}
return nil
})
if err != nil {
// 列目录失败(比如目录不存在)静默返回
return nil
}
// 按路径长度倒序(深目录优先删除),同长度保持稳定顺序
sort.SliceStable(dirs, func(i, j int) bool {
return len(dirs[i]) > len(dirs[j])
})
for _, dir := range dirs {
_ = p.rfs.Rmdir(ctx, dir)
}
// 尝试清理 prefix 本身
if prefix != "" {
_ = p.rfs.Rmdir(ctx, prefix)
}
return nil
}
// pathDir 返回 objectKey 的目录部分(正斜杠分隔)。
func pathDir(objectKey string) string {
idx := strings.LastIndex(objectKey, "/")
if idx < 0 {
return ""
}
return objectKey[:idx]
}

View File

@@ -0,0 +1,202 @@
package rclone
import (
"context"
"io"
"strings"
"testing"
)
func TestProviderLocalDiskCRUD(t *testing.T) {
factory := NewLocalDiskFactory()
provider, err := factory.New(context.Background(), map[string]any{"basePath": t.TempDir()})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
if err := provider.TestConnection(context.Background()); err != nil {
t.Fatalf("TestConnection returned error: %v", err)
}
// Upload
if err := provider.Upload(context.Background(), "daily/backup.txt", strings.NewReader("hello"), 5, nil); err != nil {
t.Fatalf("Upload returned error: %v", err)
}
// Download
reader, err := provider.Download(context.Background(), "daily/backup.txt")
if err != nil {
t.Fatalf("Download returned error: %v", err)
}
defer reader.Close()
content, _ := io.ReadAll(reader)
if string(content) != "hello" {
t.Fatalf("expected 'hello', got %q", string(content))
}
// List with prefix
items, err := provider.List(context.Background(), "daily")
if err != nil {
t.Fatalf("List returned error: %v", err)
}
if len(items) != 1 || items[0].Key != "daily/backup.txt" {
t.Fatalf("unexpected list result: %#v", items)
}
// Delete
if err := provider.Delete(context.Background(), "daily/backup.txt"); err != nil {
t.Fatalf("Delete returned error: %v", err)
}
// List after delete should be empty
items, err = provider.List(context.Background(), "daily")
if err != nil {
t.Fatalf("List after delete returned error: %v", err)
}
if len(items) != 0 {
t.Fatalf("expected empty list after delete, got %d items", len(items))
}
}
func TestProviderLocalDiskRequiresBasePath(t *testing.T) {
_, err := NewLocalDiskFactory().New(context.Background(), map[string]any{"basePath": ""})
if err == nil {
t.Fatal("expected error for empty basePath")
}
}
func TestProviderS3RequiresBucketAndCredentials(t *testing.T) {
factory := NewS3Factory()
_, err := factory.New(context.Background(), map[string]any{"bucket": "", "accessKeyId": "a", "secretAccessKey": "b"})
if err == nil || !strings.Contains(err.Error(), "bucket") {
t.Fatalf("expected bucket required error, got %v", err)
}
_, err = factory.New(context.Background(), map[string]any{"bucket": "demo", "accessKeyId": "", "secretAccessKey": "b"})
if err == nil || !strings.Contains(err.Error(), "credentials") {
t.Fatalf("expected credentials required error, got %v", err)
}
}
func TestQuoteParam(t *testing.T) {
tests := []struct {
input string
expected string
}{
{"simple", "simple"},
{"", ""},
{"has,comma", "'has,comma'"},
{"has:colon", "'has:colon'"},
{"has=equals", "'has=equals'"},
{"has'quote", "'has''quote'"},
{"a,b:c=d'e", "'a,b:c=d''e'"},
}
for _, tt := range tests {
got := quoteParam(tt.input)
if got != tt.expected {
t.Errorf("quoteParam(%q) = %q, want %q", tt.input, got, tt.expected)
}
}
}
func TestBuildS3Remote(t *testing.T) {
remote := buildS3Remote("Alibaba", "keyID", "secret", "https://oss-cn-hangzhou.aliyuncs.com", "cn-hangzhou", "my-bucket", false)
if !strings.Contains(remote, "provider=Alibaba") {
t.Fatalf("expected provider=Alibaba in remote: %s", remote)
}
if !strings.Contains(remote, ":my-bucket") {
t.Fatalf("expected :my-bucket suffix in remote: %s", remote)
}
if !strings.HasPrefix(remote, ":s3,") {
t.Fatalf("expected :s3, prefix in remote: %s", remote)
}
}
func TestRcloneFactoryCRUD(t *testing.T) {
dir := t.TempDir()
factory := NewRcloneFactory()
// 使用 rclone 的 local 后端
provider, err := factory.New(context.Background(), map[string]any{
"backend": "local",
"root": dir,
})
if err != nil {
t.Fatalf("RcloneFactory.New returned error: %v", err)
}
if err := provider.Upload(context.Background(), "test.txt", strings.NewReader("rclone"), 6, nil); err != nil {
t.Fatalf("Upload via rclone factory returned error: %v", err)
}
reader, err := provider.Download(context.Background(), "test.txt")
if err != nil {
t.Fatalf("Download returned error: %v", err)
}
defer reader.Close()
content, _ := io.ReadAll(reader)
if string(content) != "rclone" {
t.Fatalf("expected 'rclone', got %q", string(content))
}
if err := provider.Delete(context.Background(), "test.txt"); err != nil {
t.Fatalf("Delete returned error: %v", err)
}
}
func TestRcloneFactoryRequiresBackend(t *testing.T) {
_, err := NewRcloneFactory().New(context.Background(), map[string]any{"root": "/tmp"})
if err == nil || !strings.Contains(err.Error(), "backend") {
t.Fatalf("expected backend required error, got %v", err)
}
}
func TestListBackends(t *testing.T) {
backends := ListBackends()
if len(backends) < 30 {
t.Fatalf("expected at least 30 backends, got %d", len(backends))
}
// 确认 sftp 在列表中
found := false
for _, b := range backends {
if b.Name == "sftp" {
found = true
if len(b.Options) == 0 {
t.Fatal("sftp backend should have options")
}
break
}
}
if !found {
t.Fatal("sftp backend not found in ListBackends()")
}
}
func TestProviderAbout(t *testing.T) {
factory := NewLocalDiskFactory()
provider, err := factory.New(context.Background(), map[string]any{"basePath": t.TempDir()})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
// local 后端支持 About
rcloneProvider := provider.(*Provider)
usage, err := rcloneProvider.About(context.Background())
if err != nil {
t.Fatalf("About returned error: %v", err)
}
if usage.Total == nil || *usage.Total <= 0 {
t.Fatalf("expected non-zero total disk space, got %v", usage.Total)
}
}
func TestPathDir(t *testing.T) {
tests := []struct {
input string
expected string
}{
{"BackupX/file/260308/backup.tar.gz", "BackupX/file/260308"},
{"backup.tar.gz", ""},
{"a/b", "a"},
{"", ""},
}
for _, tt := range tests {
got := pathDir(tt.input)
if got != tt.expected {
t.Errorf("pathDir(%q) = %q, want %q", tt.input, got, tt.expected)
}
}
}

View File

@@ -1,126 +0,0 @@
package s3
import (
"context"
"fmt"
"io"
"strings"
"time"
"backupx/server/internal/storage"
awscore "github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
awss3 "github.com/aws/aws-sdk-go-v2/service/s3"
)
type client interface {
HeadBucket(context.Context, *awss3.HeadBucketInput, ...func(*awss3.Options)) (*awss3.HeadBucketOutput, error)
PutObject(context.Context, *awss3.PutObjectInput, ...func(*awss3.Options)) (*awss3.PutObjectOutput, error)
GetObject(context.Context, *awss3.GetObjectInput, ...func(*awss3.Options)) (*awss3.GetObjectOutput, error)
DeleteObject(context.Context, *awss3.DeleteObjectInput, ...func(*awss3.Options)) (*awss3.DeleteObjectOutput, error)
ListObjectsV2(context.Context, *awss3.ListObjectsV2Input, ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error)
}
type Provider struct {
client client
bucket string
}
type Factory struct {
newClient func(cfg storage.S3Config) client
}
func NewFactory() Factory {
return Factory{newClient: func(cfg storage.S3Config) client {
region := strings.TrimSpace(cfg.Region)
if region == "" {
region = "us-east-1"
}
awsConfig := awscore.Config{
Region: region,
Credentials: credentials.NewStaticCredentialsProvider(cfg.AccessKeyID, cfg.SecretAccessKey, ""),
}
return awss3.NewFromConfig(awsConfig, func(options *awss3.Options) {
options.UsePathStyle = cfg.ForcePathStyle
if strings.TrimSpace(cfg.Endpoint) != "" {
options.BaseEndpoint = awscore.String(strings.TrimRight(cfg.Endpoint, "/"))
}
})
}}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeS3 }
func (Factory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
func (f Factory) New(_ context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[storage.S3Config](rawConfig)
if err != nil {
return nil, err
}
if strings.TrimSpace(cfg.Bucket) == "" {
return nil, fmt.Errorf("s3 bucket is required")
}
if strings.TrimSpace(cfg.AccessKeyID) == "" || strings.TrimSpace(cfg.SecretAccessKey) == "" {
return nil, fmt.Errorf("s3 credentials are required")
}
newClient := f.newClient
if newClient == nil {
factory := NewFactory()
newClient = factory.newClient
}
return &Provider{client: newClient(cfg), bucket: cfg.Bucket}, nil
}
func (p *Provider) Type() storage.ProviderType { return storage.ProviderTypeS3 }
func (p *Provider) TestConnection(ctx context.Context) error {
_, err := p.client.HeadBucket(ctx, &awss3.HeadBucketInput{Bucket: awscore.String(p.bucket)})
if err != nil {
return fmt.Errorf("test s3 connection: %w", err)
}
return nil
}
func (p *Provider) Upload(ctx context.Context, objectKey string, reader io.Reader, _ int64, metadata map[string]string) error {
_, err := p.client.PutObject(ctx, &awss3.PutObjectInput{Bucket: awscore.String(p.bucket), Key: awscore.String(objectKey), Body: reader, Metadata: metadata})
if err != nil {
return fmt.Errorf("upload s3 object: %w", err)
}
return nil
}
func (p *Provider) Download(ctx context.Context, objectKey string) (io.ReadCloser, error) {
result, err := p.client.GetObject(ctx, &awss3.GetObjectInput{Bucket: awscore.String(p.bucket), Key: awscore.String(objectKey)})
if err != nil {
return nil, fmt.Errorf("download s3 object: %w", err)
}
return result.Body, nil
}
func (p *Provider) Delete(ctx context.Context, objectKey string) error {
_, err := p.client.DeleteObject(ctx, &awss3.DeleteObjectInput{Bucket: awscore.String(p.bucket), Key: awscore.String(objectKey)})
if err != nil {
return fmt.Errorf("delete s3 object: %w", err)
}
return nil
}
func (p *Provider) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
result, err := p.client.ListObjectsV2(ctx, &awss3.ListObjectsV2Input{Bucket: awscore.String(p.bucket), Prefix: awscore.String(prefix)})
if err != nil {
return nil, fmt.Errorf("list s3 objects: %w", err)
}
items := make([]storage.ObjectInfo, 0, len(result.Contents))
for _, object := range result.Contents {
updatedAt := time.Time{}
if object.LastModified != nil {
updatedAt = object.LastModified.UTC()
}
size := int64(0)
if object.Size != nil {
size = *object.Size
}
items = append(items, storage.ObjectInfo{Key: awscore.ToString(object.Key), Size: size, UpdatedAt: updatedAt})
}
return items, nil
}

View File

@@ -1,78 +0,0 @@
package s3
import (
"bytes"
"context"
"io"
"strings"
"testing"
"time"
"backupx/server/internal/storage"
awscore "github.com/aws/aws-sdk-go-v2/aws"
awss3 "github.com/aws/aws-sdk-go-v2/service/s3"
awss3types "github.com/aws/aws-sdk-go-v2/service/s3/types"
)
type fakeClient struct{ data map[string]string }
func (c *fakeClient) HeadBucket(context.Context, *awss3.HeadBucketInput, ...func(*awss3.Options)) (*awss3.HeadBucketOutput, error) {
return &awss3.HeadBucketOutput{}, nil
}
func (c *fakeClient) PutObject(_ context.Context, input *awss3.PutObjectInput, _ ...func(*awss3.Options)) (*awss3.PutObjectOutput, error) {
body, _ := io.ReadAll(input.Body)
c.data[awscore.ToString(input.Key)] = string(body)
return &awss3.PutObjectOutput{}, nil
}
func (c *fakeClient) GetObject(_ context.Context, input *awss3.GetObjectInput, _ ...func(*awss3.Options)) (*awss3.GetObjectOutput, error) {
return &awss3.GetObjectOutput{Body: io.NopCloser(strings.NewReader(c.data[awscore.ToString(input.Key)]))}, nil
}
func (c *fakeClient) DeleteObject(_ context.Context, input *awss3.DeleteObjectInput, _ ...func(*awss3.Options)) (*awss3.DeleteObjectOutput, error) {
delete(c.data, awscore.ToString(input.Key))
return &awss3.DeleteObjectOutput{}, nil
}
func (c *fakeClient) ListObjectsV2(_ context.Context, _ *awss3.ListObjectsV2Input, _ ...func(*awss3.Options)) (*awss3.ListObjectsV2Output, error) {
now := time.Now().UTC()
return &awss3.ListObjectsV2Output{Contents: []awss3types.Object{{Key: awscore.String("backup.tar.gz"), Size: awscore.Int64(10), LastModified: &now}}}, nil
}
func TestS3ProviderCRUD(t *testing.T) {
factory := Factory{newClient: func(cfg storage.S3Config) client {
return &fakeClient{data: make(map[string]string)}
}}
providerAny, err := factory.New(context.Background(), map[string]any{"bucket": "demo", "accessKeyId": "a", "secretAccessKey": "b"})
if err != nil {
t.Fatalf("Factory.New returned error: %v", err)
}
provider := providerAny.(*Provider)
if err := provider.TestConnection(context.Background()); err != nil {
t.Fatalf("TestConnection returned error: %v", err)
}
if err := provider.Upload(context.Background(), "backup.tar.gz", bytes.NewBufferString("payload"), 7, nil); err != nil {
t.Fatalf("Upload returned error: %v", err)
}
reader, err := provider.Download(context.Background(), "backup.tar.gz")
if err != nil {
t.Fatalf("Download returned error: %v", err)
}
defer reader.Close()
content, _ := io.ReadAll(reader)
if string(content) != "payload" {
t.Fatalf("unexpected content: %s", string(content))
}
items, err := provider.List(context.Background(), "backup")
if err != nil {
t.Fatalf("List returned error: %v", err)
}
if len(items) != 1 || items[0].Key != "backup.tar.gz" {
t.Fatalf("unexpected list result: %#v", items)
}
if err := provider.Delete(context.Background(), "backup.tar.gz"); err != nil {
t.Fatalf("Delete returned error: %v", err)
}
}

View File

@@ -1,9 +0,0 @@
package s3provider
import "backupx/server/internal/storage/s3"
type Factory = s3.Factory
func NewFactory() Factory {
return s3.NewFactory()
}

View File

@@ -1,60 +0,0 @@
// Package tencent provides a Tencent Cloud COS storage factory that delegates to the S3-compatible engine.
// Tencent COS is fully S3-compatible; we auto-assemble the endpoint from region and appId.
package tencent
import (
"context"
"fmt"
"strings"
"backupx/server/internal/storage"
"backupx/server/internal/storage/s3"
)
// Config is the user-facing configuration for Tencent COS.
type Config struct {
Region string `json:"region"`
Bucket string `json:"bucket"` // format: bucketname-appid
SecretID string `json:"accessKeyId"`
SecretKey string `json:"secretAccessKey"`
Endpoint string `json:"endpoint"` // optional override
}
// Factory creates Tencent COS providers by composing the S3 engine.
type Factory struct {
s3Factory s3.Factory
}
func NewFactory() Factory {
return Factory{s3Factory: s3.NewFactory()}
}
func (Factory) Type() storage.ProviderType { return storage.ProviderTypeTencentCOS }
func (Factory) SensitiveFields() []string { return []string{"accessKeyId", "secretAccessKey"} }
func (f Factory) New(ctx context.Context, rawConfig map[string]any) (storage.StorageProvider, error) {
cfg, err := storage.DecodeConfig[Config](rawConfig)
if err != nil {
return nil, err
}
endpoint := strings.TrimSpace(cfg.Endpoint)
if endpoint == "" {
region := strings.TrimSpace(cfg.Region)
if region == "" {
return nil, fmt.Errorf("tencent cos region is required")
}
// Tencent COS S3-compatible endpoint format
endpoint = fmt.Sprintf("https://cos.%s.myqcloud.com", region)
}
s3Config := map[string]any{
"endpoint": endpoint,
"region": cfg.Region,
"bucket": cfg.Bucket,
"accessKeyId": cfg.SecretID,
"secretAccessKey": cfg.SecretKey,
"forcePathStyle": false, // COS uses virtual-hosted style
}
return f.s3Factory.New(ctx, s3Config)
}

View File

@@ -20,6 +20,7 @@ const (
ProviderTypeTencentCOS ProviderType = "tencent_cos"
ProviderTypeQiniuKodo ProviderType = "qiniu_kodo"
ProviderTypeFTP ProviderType = "ftp"
ProviderTypeRclone ProviderType = "rclone"
)
const (
@@ -52,6 +53,20 @@ type ProviderFactory interface {
Type() ProviderType
}
// StorageAbout 是可选能力接口,支持查询远端存储空间。
// 并非所有后端都支持(如 S3/FTP 不支持),通过 type assertion 检测。
type StorageAbout interface {
About(ctx context.Context) (*StorageUsageInfo, error)
}
// StorageUsageInfo 描述远端存储的空间使用情况。
type StorageUsageInfo struct {
Total *int64 `json:"total,omitempty"` // 总空间(字节)
Used *int64 `json:"used,omitempty"` // 已用空间
Free *int64 `json:"free,omitempty"` // 可用空间
Objects *int64 `json:"objects,omitempty"` // 对象数量
}
func DecodeConfig[T any](raw map[string]any) (T, error) {
var cfg T
encoded, err := json.Marshal(raw)
@@ -130,3 +145,10 @@ type FTPConfig struct {
UseTLS bool `json:"useTLS"`
}
// StorageDirCleaner 是可选能力接口,支持清理空目录。
// 主要用于本地磁盘等文件系统类存储,对象存储通常不需要。
// 通过 type assertion 检测 provider 是否实现该接口。
type StorageDirCleaner interface {
RemoveEmptyDirs(ctx context.Context, prefix string) error
}

Some files were not shown because too many files have changed in this diff Show More