mirror of
https://github.com/Awuqing/BackupX.git
synced 2026-08-21 16:23:04 +08:00
feat: 解决 CDC 去重、集中备份与首次初始化问题 (#105)
实现 CDC 内容寻址仓库、远程 Agent 中央中转备份与首次初始化体验，并补充安全校验、测试及双语文档。 Closes #94 Closes #101 Closes #104
This commit is contained in:
Wu Qing
@@ -10,20 +10,21 @@ description: systemd + Nginx deployment from the prebuilt release tarball or sou
|
||||
|
||||
```bash
|
||||
# Download the matching tarball
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-v1.6.0-linux-amd64.tar.gz
|
||||
curl -LO https://github.com/Awuqing/BackupX/releases/latest/download/backupx-linux-amd64.tar.gz
|
||||
|
||||
# Extract and install
|
||||
tar xzf backupx-v*-linux-amd64.tar.gz && cd backupx-*
|
||||
tar xzf backupx-linux-amd64.tar.gz && cd backupx-*-linux-amd64
|
||||
sudo ./install.sh
|
||||
```
|
||||
|
||||
The installer performs these steps automatically:
|
||||
|
||||
1. Creates a system user `backupx`
|
||||
2. Copies the binary to `/opt/backupx/`
|
||||
3. Generates a default `config.yaml` with safe JWT/encryption secrets
|
||||
2. Copies the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
|
||||
3. Installs the default configuration at `/etc/backupx/config.yaml`
|
||||
4. Installs `backupx.service` (systemd), enabled at boot
|
||||
5. (Optional) installs an Nginx site file — see [Nginx Reverse Proxy](./nginx)
|
||||
6. Verifies the first-setup API before reporting success
|
||||
|
||||
For multi-node clusters, edit `/etc/backupx/config.yaml` after installation and set the Master URL that remote Agents can reach:
|
||||
|
||||
@@ -63,11 +64,13 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=backupx
|
||||
Group=backupx
|
||||
WorkingDirectory=/opt/backupx
|
||||
ExecStart=/opt/backupx/backupx --config /opt/backupx/config.yaml
|
||||
ExecStart=/opt/backupx/bin/backupx -config /etc/backupx/config.yaml
|
||||
Restart=on-failure
|
||||
RestartSec=5s
|
||||
LimitNOFILE=65536
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
LimitNOFILE=65535
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -79,17 +82,20 @@ Typical operations:
|
||||
sudo systemctl status backupx
|
||||
sudo journalctl -u backupx -f # live logs
|
||||
sudo systemctl restart backupx
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
```
|
||||
|
||||
Open `http://your-server:8340`, switch to English if desired, and create the first administrator on the **System setup** screen. For a custom listen port, run the installer with a matching `HEALTH_URL`.
|
||||
|
||||
## Password reset
|
||||
|
||||
If the admin password is lost:
|
||||
|
||||
```bash
|
||||
/opt/backupx/backupx reset-password \
|
||||
/opt/backupx/bin/backupx reset-password \
|
||||
--username admin \
|
||||
--password 'newpass123' \
|
||||
--config /opt/backupx/config.yaml
|
||||
--config /etc/backupx/config.yaml
|
||||
```
|
||||
|
||||
Docker equivalent:
|
||||
|
||||
@@ -12,12 +12,22 @@ When a task is routed to a remote Agent, the source tools and paths are resolved
|
||||
|
||||
## File / Directory
|
||||
|
||||
Tars (and optionally gzips) one or more filesystem paths.
|
||||
File tasks offer three backup modes:
|
||||
|
||||
- **Full archive** — writes a self-contained tar artifact on every run
|
||||
- **Differential archive** — writes only changes since the current full baseline and periodically refreshes that baseline
|
||||
- **CDC repository** — splits content with stable 512 KiB / 1 MiB / 4 MiB boundaries, stores new chunks in immutable 32 MiB packs, and writes a small snapshot manifest for each run
|
||||
|
||||
The CDC repository deduplicates identical content across files and snapshots. Restore, selective restore, verification, download-as-tar, retention, and garbage collection all resolve data through the repository index. Compression and encryption are applied per chunk; encrypted repositories use keyed chunk IDs so plaintext hashes are not exposed.
|
||||
|
||||
Repository mode currently uses a single-writer index and therefore runs on the Master only. To keep repository copies on multiple backends, select multiple primary storage targets on the task. Object-level replication is intentionally disabled because a snapshot manifest without its shared packs and indexes is not a complete backup.
|
||||
|
||||
Common file-task options:
|
||||
|
||||
- **Source** accepts multiple paths — one per line in the UI
|
||||
- **Exclude patterns** accept gitignore-style globs
|
||||
- Supports following symlinks, preserving permissions
|
||||
- Output is a single `.tar` or `.tar.gz` artifact
|
||||
- Full and differential modes output `.tar`, `.tar.gz`, or `.tar.zst` artifacts
|
||||
|
||||
## MySQL
|
||||
|
||||
|
||||
@@ -26,6 +26,27 @@ BackupX supports Master-Agent mode: backup tasks can be routed to specific nodes
|
||||
- **Execution** — Agent reuses the same BackupRunner (file / mysql / postgresql / sqlite / saphana) and uploads directly to storage
|
||||
- **Security** — Each node has its own token; the Agent never holds the Master's JWT secret or AES-256 key
|
||||
|
||||
## Centralize backups from servers B/C/D into storage M
|
||||
|
||||
Use the Master as the control plane and register every source server as an Agent. A task's **Source server** determines where paths and database tools are resolved; its **Storage targets** determine where the resulting artifact is retained.
|
||||
|
||||
BackupX chooses the data path per target:
|
||||
|
||||
| Destination | Data path |
|
||||
| --- | --- |
|
||||
| S3, WebDAV, FTP, cloud drive, or another network backend | Agent streams directly to the destination |
|
||||
| `local_disk` with **Relay remote backups through Master** enabled (for example storage server M mounted through NFS) | Agent streams through the authenticated Master API; Master writes to its configured local path |
|
||||
|
||||
The relay is streaming: the Master does not create a second temporary copy of the entire artifact. The reverse path is used when restoring a Master-local artifact back to its source Agent. Use HTTPS whenever Agent traffic crosses an untrusted network.
|
||||
|
||||
To configure the common `A → {B,C,D} → M` topology:
|
||||
|
||||
1. Run BackupX Master on A and mount M on A if M is exposed as NFS or another filesystem.
|
||||
2. Create a `local_disk` target for that mount and keep **Relay remote backups through Master** enabled, or create an S3/WebDAV target exposed by M. Existing local-disk targets keep their prior Agent-local behavior until this switch is enabled.
|
||||
3. Install one Agent on B, C, and D from **Node Management**.
|
||||
4. Create a backup task for each source, choose B/C/D under **Source server**, browse that server's paths, and select M as the storage target. A source-server pool label can route identical tasks dynamically.
|
||||
5. Verify the per-target result in the backup record. For a Master-local target, the record reports transfer mode `master_relay`; network backends remain `direct`.
|
||||
|
||||
## Walkthrough
|
||||
|
||||
### 0. Set the Master URL for production clusters
|
||||
@@ -78,7 +99,7 @@ In Step 1 choose "Batch" and paste node names (one per line, max 50). Step 3 sho
|
||||
|
||||
### 5. Route a task to the node
|
||||
|
||||
In the **Backup Tasks** page, pick the target node when creating the task. When the task runs:
|
||||
In the **Backup Tasks** page, pick the source server when creating the task. When the task runs:
|
||||
|
||||
- Local (`nodeId=0`) → Master executes in-process
|
||||
- Remote node → Master enqueues the command → Agent claims → Agent runs locally → uploads → reports back
|
||||
|
||||
@@ -19,7 +19,9 @@ BackupX aims to accept any place you'd want to drop a backup file.
|
||||
| **Google Drive** | Client ID/Secret + OAuth authorization |
|
||||
| **WebDAV** | URL + username/password |
|
||||
| **FTP / FTPS** | Host + port + username/password |
|
||||
| **Local disk** | Target directory (absolute path) |
|
||||
| **Local disk** | Target directory (absolute path) + optional Master relay for remote Agents |
|
||||
|
||||
New local-disk targets enable **Relay remote backups through Master** by default. This makes the configured path belong to the Master, so a storage server mounted there can collect backups from many source Agents. Turn the switch off when the path intentionally belongs to each Agent. Existing targets retain their previous Agent-local behavior until explicitly changed.
|
||||
|
||||
## Rclone backends
|
||||
|
||||
|
||||
@@ -55,10 +55,11 @@ sudo ./install.sh # creates system user, installs to /opt/backupx, sets u
|
||||
The installer:
|
||||
|
||||
1. Creates a `backupx` system user
|
||||
2. Installs binary to `/opt/backupx/backupx`
|
||||
3. Creates `/opt/backupx/config.yaml` with safe defaults
|
||||
2. Installs the binary to `/opt/backupx/bin/backupx` and the web console to `/opt/backupx/web`
|
||||
3. Creates `/etc/backupx/config.yaml` with safe defaults
|
||||
4. Installs and enables the `backupx.service` systemd unit
|
||||
5. (Optional) Configures an Nginx reverse proxy
|
||||
6. Waits for `/api/auth/setup/status`; if startup fails, prints systemd diagnostics and exits non-zero
|
||||
|
||||
## From source
|
||||
|
||||
@@ -67,16 +68,17 @@ Requires Go ≥ 1.25 and Node.js ≥ 20.
|
||||
```bash
|
||||
git clone https://github.com/Awuqing/BackupX.git && cd BackupX
|
||||
make build
|
||||
# or, for builds behind the great firewall
|
||||
make docker-cn
|
||||
sudo ./deploy/install.sh
|
||||
```
|
||||
|
||||
After `make build`, the binary is at `server/bin/backupx` and the built web UI is at `web/dist/`.
|
||||
The installer consumes those exact paths, so no Docker runtime is required. If an existing configuration uses a non-default port, set `HEALTH_URL` for the readiness check, for example `sudo HEALTH_URL=http://127.0.0.1:9000/api/auth/setup/status ./deploy/install.sh`.
|
||||
|
||||
## Verify the install
|
||||
|
||||
```bash
|
||||
backupx --version # e.g. v1.6.0
|
||||
/opt/backupx/bin/backupx --version
|
||||
curl -fsS http://127.0.0.1:8340/api/auth/setup/status
|
||||
```
|
||||
|
||||
Then open `http://your-server:8340` to see the initial admin setup screen.
|
||||
Then open `http://your-server:8340`. Choose **English** or **中文** in the upper-right corner. A fresh database shows **System setup**, where you create the first administrator username and password. If that form does not appear, retry the status request above before attempting to sign in.
|
||||
|
||||
Reference in New Issue
Block a user