docs: 完善部署与运维文档 (#107)

新增中英文升级恢复、安全加固、监控告警与故障排查手册，校正安装部署、CLI 与 API 参考，并修复安全密钥环境变量注入及其回归测试。
2026-08-12 07:54:14 +08:00 · 2026-08-09 13:51:38 +08:00
parent 5827074334
commit bdd16dafa8
35 changed files with 1844 additions and 317 deletions
--- a/docs-site/docs/deployment/bare-metal.md
+++ b/docs-site/docs/deployment/bare-metal.md
@@ -68,8 +68,9 @@ The installed unit:

 ```ini title="/etc/systemd/system/backupx.service"
 [Unit]
-Description=BackupX backup management service
-After=network.target
+Description=BackupX API Service
+After=network-online.target
+Wants=network-online.target

 [Service]
 Type=simple
@@ -100,6 +101,8 @@ Open `http://your-server:8340`, switch to English if desired, and create the fir

 For production, expose BackupX through HTTPS or restrict port `8340` at the firewall. The installer does not make firewall changes.

+Before replacing a release, snapshot `/etc/backupx`, `/opt/backupx/data`, the installed binary, and web assets while the service is stopped. Follow the versioned procedure in [Upgrade and Recovery](../operations/upgrade-recovery); running an older binary against a database already migrated by a newer release is not a safe rollback.
+
 ## Password reset

 If the admin password is lost:
--- a/docs-site/docs/deployment/configuration.md
+++ b/docs-site/docs/deployment/configuration.md
@@ -1,7 +1,7 @@
 ---
 sidebar_position: 4
 title: Configuration Reference
-description: All server.yaml configuration keys with defaults and matching environment variables.
+description: All config.yaml server keys with defaults and matching environment variables.
 ---

 # Configuration Reference
@@ -32,12 +32,15 @@ security:
 backup:
  temp_dir: "/tmp/backupx"    # BACKUPX_BACKUP_TEMP_DIR
  max_concurrent: 2           # BACKUPX_BACKUP_MAX_CONCURRENT
-  retries: 3                  # Per-upload rclone low-level retries
+  retries: 10                 # Per-upload rclone low-level retries
  bandwidth_limit: ""         # e.g. "10M" to cap transfers at 10 MB/s

 log:
  level: "info"               # debug | info | warn | error
  file: "./data/backupx.log"
+  max_size: 100               # MB per log file
+  max_backups: 3              # rotated files retained
+  max_age: 30                 # retention in days
 ```

 ## Secret generation
@@ -53,11 +56,17 @@ The environment wins when both file and env are set. All dot-paths become unders
 | `server.port` | `BACKUPX_SERVER_PORT` |
 | `server.external_url` | `BACKUPX_SERVER_EXTERNAL_URL` |
 | `server.trusted_proxies` | `BACKUPX_SERVER_TRUSTED_PROXIES` (comma-separated for env) |
+| `security.jwt_secret` | `BACKUPX_SECURITY_JWT_SECRET` |
 | `security.jwt_expire` | `BACKUPX_SECURITY_JWT_EXPIRE` |
+| `security.encryption_key` | `BACKUPX_SECURITY_ENCRYPTION_KEY` |
 | `log.level` | `BACKUPX_LOG_LEVEL` |
 | `backup.max_concurrent` | `BACKUPX_BACKUP_MAX_CONCURRENT` |
 | `backup.temp_dir` | `BACKUPX_BACKUP_TEMP_DIR` |
+| `backup.retries` | `BACKUPX_BACKUP_RETRIES` |
 | `backup.bandwidth_limit` | `BACKUPX_BACKUP_BANDWIDTH_LIMIT` |
+| `log.max_size` | `BACKUPX_LOG_MAX_SIZE` |
+| `log.max_backups` | `BACKUPX_LOG_MAX_BACKUPS` |
+| `log.max_age` | `BACKUPX_LOG_MAX_AGE` |

 ## Master external URL

@@ -70,7 +79,7 @@ server:

 This value is used when BackupX renders one-click Agent install scripts and docker-compose snippets. It must be reachable from every Agent host. Leave it empty only when `X-Forwarded-Proto` / `X-Forwarded-Host` are reliable and point to the same URL that Agents can access.

-The install wizard can set an Agent-specific runtime URL for a proxy or SSH-bastion node. The public install link continues to use `server.external_url`, while the generated Agent config uses that override.
+The install wizard can set an Agent-specific URL for a proxy or SSH-bastion node. That override is used by both the target-side one-time install URL and the generated Agent runtime configuration, while the browser continues to use the normal public address.

 ## Trusted reverse proxies

@@ -84,3 +93,5 @@ server:
 ```

 Do not configure `0.0.0.0/0`: client addresses feed authentication throttling, install-token throttling, and audit records. Set an empty list when BackupX is exposed directly and should trust no forwarded headers.
+
+Back up the complete data directory and configuration before changing security keys or database paths. See [Upgrade and Recovery](../operations/upgrade-recovery) for a tested snapshot and rollback sequence.
--- a/docs-site/docs/deployment/docker.md
+++ b/docs-site/docs/deployment/docker.md
@@ -85,7 +85,7 @@ environment:

 The image's internal port is fixed at `8340`; change only the published host port with `BACKUPX_PORT`.

-## Upgrade and rollback preparation
+## Upgrade prerequisites

 ```bash
 docker compose pull
@@ -94,3 +94,5 @@ docker compose ps
 ```

 Wait for `healthy` before switching traffic or removing an old deployment. Before upgrades, stop the Master for a file-level copy or take an atomic snapshot of the entire `backupx-data` volume. Keep exactly one active Master for a data volume; SQLite does not support multiple Master containers sharing `/app/data`.
+
+Use a release tag or digest instead of `latest`, and keep the matching pre-upgrade data snapshot. The complete upgrade, rollback, and disaster-recovery procedure is in [Upgrade and Recovery](../operations/upgrade-recovery).
--- a/docs-site/docs/deployment/nginx.md
+++ b/docs-site/docs/deployment/nginx.md
@@ -29,6 +29,7 @@ server {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Port $server_port;
+        proxy_set_header Connection "";

        # Large uploads (restore flow)
        client_max_body_size 0;
@@ -36,9 +37,28 @@ server {

        # Live log stream uses SSE — buffering must be off
        proxy_buffering off;
+        proxy_cache off;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
+
+    # Compatibility route for installers generated by older releases.
+    # Current installers use /api/install/ through the API block above.
+    location /install/ {
+        proxy_pass http://127.0.0.1:8340/install/;
+        proxy_http_version 1.1;
+        proxy_set_header Host $http_host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $http_host;
+        proxy_set_header X-Forwarded-Port $server_port;
+    }
+
+    # Keep probes and metrics out of the SPA fallback.
+    location = /health { proxy_pass http://127.0.0.1:8340/health; }
+    location = /ready  { proxy_pass http://127.0.0.1:8340/ready; }
+    location = /metrics { proxy_pass http://127.0.0.1:8340/metrics; }
 }
 ```

@@ -46,6 +66,8 @@ server {

 If Nginx runs on another host or in another container, add only that proxy IP or subnet to `server.trusted_proxies`. Do not use `0.0.0.0/0`; BackupX uses the trusted client address for login throttling, install-token throttling, and audit records.

+`/health`, `/ready`, and `/metrics` do not require BackupX authentication. Allow probe and Prometheus source networks explicitly, or keep these locations on an internal listener instead of exposing them to the Internet.
+
 ## HTTPS with certbot

 ```bash
@@ -56,5 +78,5 @@ sudo certbot --nginx -d backup.example.com
 Certbot rewrites the config to listen on 443 with auto-renewal.

 :::caution Agent needs a stable URL
-If Master is behind HTTPS, remote Agent deployments must use the public HTTPS URL for `--master`. Self-signed certs require `--insecure-tls` (testing only).
+If Master is behind HTTPS, remote Agent deployments must use the final HTTPS URL for `--master`; redirects are not followed. For a private CA, pre-provision its PEM certificate and use `--ca-cert /path/to/ca.pem`. Reserve `--insecure-tls` for short-lived testing.
 :::
--- a/docs-site/docs/getting-started/installation.md
+++ b/docs-site/docs/getting-started/installation.md
@@ -10,38 +10,25 @@ BackupX ships as a single static binary. Three ways to install, pick the one tha

 ## Docker (recommended)

-No cloning required.
+Download the canonical hardened Compose file and start the service:

 ```bash
-docker run -d --name backupx \
-  -p 8340:8340 \
-  -v backupx-data:/app/data \
-  awuqing/backupx:latest
+curl -fLO https://raw.githubusercontent.com/Awuqing/BackupX/main/docker-compose.yml
+docker compose up -d
+docker compose ps
 ```

-Or use `docker compose`:
+The Compose definition enables init and graceful shutdown, persists `/app/data`, runs the application as an unprivileged user, drops unnecessary capabilities, and checks `/ready`. Images at [`awuqing/backupx`](https://hub.docker.com/r/awuqing/backupx) support `linux/amd64` and `linux/arm64`.

-```yaml title="docker-compose.yml"
-services:
-  backupx:
-    image: awuqing/backupx:latest
-    container_name: backupx
-    restart: unless-stopped
-    ports:
-      - "8340:8340"
-    volumes:
-      - backupx-data:/app/data
-      # Mount host directories to back up (as needed):
-      # - /var/www:/mnt/www:ro
-      # - /etc/nginx:/mnt/nginx-conf:ro
-    environment:
-      - TZ=Asia/Shanghai
+For production, create a protected `.env` and pin a release instead of relying on `latest`:

-volumes:
-  backupx-data:
+```dotenv
+BACKUPX_IMAGE=awuqing/backupx:vX.Y.Z
+BACKUPX_BIND_ADDRESS=127.0.0.1
+TZ=Asia/Shanghai
 ```

-Images: [`awuqing/backupx`](https://hub.docker.com/r/awuqing/backupx) — supports `linux/amd64` and `linux/arm64`.
+Use the loopback binding when a reverse proxy runs on the same host. For direct access, choose the intended interface and enforce a firewall. Mount host backup sources read-only or deploy an Agent on the source host. See [Docker Deployment](../deployment/docker) for the full configuration.

 ## Prebuilt archive (bare metal)

--- a/docs-site/docs/getting-started/quick-start.md
+++ b/docs-site/docs/getting-started/quick-start.md
@@ -57,5 +57,6 @@ Deleting a task also removes remote backup files to prevent orphans, but records
 ## Next up

 - Explore [backup types](/docs/features/backup-types) and [storage backends](/docs/features/storage-backends)
+- Before production, review [Security Hardening](/docs/operations/security), [Monitoring and Alerts](/docs/operations/monitoring), and [Upgrade and Recovery](/docs/operations/upgrade-recovery)
 - Running SAP HANA? See [SAP HANA Support](/docs/features/sap-hana)
 - Managing many servers? See [Multi-Node Cluster](/docs/features/multi-node)
--- a/docs-site/docs/intro.md
+++ b/docs-site/docs/intro.md
@@ -35,6 +35,8 @@ Tasks routed to the local Master run in-process; tasks assigned to remote nodes

 - **New to BackupX?** Read the [Quick Start](/docs/getting-started/quick-start) first.
 - **Deploying to production?** See the [Deployment Guide](/docs/deployment/docker).
+- **Planning upgrades or recovery?** Follow [Upgrade and Recovery](/docs/operations/upgrade-recovery).
+- **Operating production?** Start with [Security Hardening](/docs/operations/security) and [Monitoring and Alerts](/docs/operations/monitoring).
 - **SAP HANA operator?** Both `hdbsql` Runner and native Backint are supported — see [SAP HANA](/docs/features/sap-hana).
 - **Managing multiple servers?** See [Multi-Node Cluster](/docs/features/multi-node).
 - **Integrating programmatically?** See the [API Reference](/docs/reference/api).
--- a/docs-site/docs/operations/monitoring.md
+++ b/docs-site/docs/operations/monitoring.md
@@ -0,0 +1,149 @@
+---
+sidebar_position: 3
+title: Monitoring and Alerts
+description: Health probes, Prometheus metrics, initial alert rules, and operational validation.
+---
+
+# Monitoring and Alerts
+
+BackupX exposes low-cost health endpoints and a dedicated Prometheus registry. Monitor both the control plane and the outcome of backup, restore, verification, and replication work.
+
+## Probes
+
+| Endpoint | Meaning | Expected response |
+| --- | --- | --- |
+| `/health` | Liveness: the HTTP process can respond | HTTP 200 with `status: live` |
+| `/ready` | Readiness: the process can reach SQLite | HTTP 200 with `status: ready`; HTTP 503 on database failure |
+| `/api/health` | API-prefixed alias for liveness | Same as `/health` |
+| `/api/ready` | API-prefixed alias for readiness | Same as `/ready` |
+| `/metrics` | Prometheus exposition | HTTP 200 when metrics are enabled |
+
+Use `/health` for a liveness probe and `/ready` for readiness or load-balancer traffic decisions. Do not restart a process only because an external storage provider is unavailable; storage health belongs in task and target alerts.
+
+~~~bash
+curl -fsS http://127.0.0.1:8340/health
+curl -fsS http://127.0.0.1:8340/ready
+curl -fsS http://127.0.0.1:8340/metrics | head
+~~~
+
+These endpoints are unauthenticated. Restrict them to orchestrator and monitoring networks.
+
+## Prometheus scrape
+
+~~~yaml
+scrape_configs:
+  - job_name: backupx
+    scheme: https
+    metrics_path: /metrics
+    static_configs:
+      - targets: [backup.example.com]
+~~~
+
+When Nginx terminates TLS, allow the Prometheus source address to reach `/metrics` and deny other public clients. The internal collector refreshes storage, node, command-queue, and SLA gauges every 30 seconds.
+
+## BackupX metrics
+
+| Metric | Type | Labels | Purpose |
+| --- | --- | --- | --- |
+| `backupx_app_info` | gauge | `version` | Running release metadata |
+| `backupx_task_run_total` | counter | `status`, `task_type` | Backup outcomes |
+| `backupx_task_run_duration_seconds` | histogram | `task_type` | Backup duration distribution |
+| `backupx_task_bytes_total` | counter | `task_type` | Produced backup bytes |
+| `backupx_task_running` | gauge | none | Current backup concurrency |
+| `backupx_storage_used_bytes` | gauge | `target_name`, `target_type` | Recorded usage per target |
+| `backupx_node_online` | gauge | `node_name`, `role` | Node online state, 1 or 0 |
+| `backupx_agent_command_queue_depth` | gauge | `node_name`, `role` | Pending and dispatched commands |
+| `backupx_agent_command_running` | gauge | `node_name`, `role` | Long-running Agent commands |
+| `backupx_agent_command_timeout_total` | gauge | `node_name`, `role` | Snapshot of timed-out commands |
+| `backupx_verify_run_total` | counter | `status` | Verification outcomes |
+| `backupx_restore_run_total` | counter | `status` | Restore outcomes |
+| `backupx_replication_run_total` | counter | `status` | Replication outcomes |
+| `backupx_sla_breach_tasks` | gauge | none | Enabled tasks outside their configured RPO |
+
+Standard Go runtime and process collectors are registered in the same endpoint.
+
+## Initial alert rules
+
+Tune windows and thresholds to the schedules and RPOs of each environment:
+
+~~~yaml
+groups:
+  - name: backupx
+    rules:
+      - alert: BackupXTargetDown
+        expr: up{job="backupx"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: BackupX metrics endpoint is unreachable
+
+      - alert: BackupXNotReady
+        expr: probe_success{job="backupx-ready"} == 0
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: BackupX readiness check is failing
+
+      - alert: BackupXBackupFailure
+        expr: sum(increase(backupx_task_run_total{status="failed"}[15m])) > 0
+        labels:
+          severity: warning
+        annotations:
+          summary: A BackupX backup failed
+
+      - alert: BackupXSLABreach
+        expr: backupx_sla_breach_tasks > 0
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: One or more backup tasks are outside RPO
+
+      - alert: BackupXAgentOffline
+        expr: backupx_node_online{role="agent"} == 0
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: BackupX Agent is offline
+
+      - alert: BackupXAgentQueueBacklog
+        expr: backupx_agent_command_queue_depth > 20
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: BackupX Agent command queue is growing
+~~~
+
+The `BackupXNotReady` example assumes a blackbox probe job named `backupx-ready`. If no blackbox exporter is used, alert from the load balancer or orchestrator readiness signal instead.
+
+## Operational dashboard
+
+Track these views together:
+
+- Success and failure rate by task type.
+- P50, P95, and maximum run duration relative to the backup window.
+- Bytes produced compared with the expected data-change rate.
+- Current running tasks versus `backup.max_concurrent`.
+- Offline Agents, queue depth, running commands, and timeout-count changes.
+- Storage growth, free capacity from the storage provider, and retention cleanup.
+- SLA breach count and age of the most recent successful backup for critical tasks.
+- Verification, restore, and replication success rates.
+
+Prometheus storage usage is based on BackupX record metadata, not necessarily the provider's billable capacity. Monitor provider quota and filesystem free space separately.
+
+## Post-deployment validation
+
+After installation, upgrade, proxy changes, or recovery:
+
+1. Check liveness and readiness locally and through the public proxy.
+2. Confirm Prometheus sees one active Master and the expected version label.
+3. Verify every expected Agent reports `backupx_node_online == 1`.
+4. Run a small backup and confirm the success counter increases.
+5. Run a verification or isolated restore and confirm its counter increases.
+6. Trigger a test notification and verify the alert delivery path.
+
+Continue with [Troubleshooting](./troubleshooting) when a probe or metric is abnormal.
--- a/docs-site/docs/operations/security.md
+++ b/docs-site/docs/operations/security.md
@@ -0,0 +1,102 @@
+---
+sidebar_position: 2
+title: Security Hardening
+description: Production controls for network exposure, roles, secrets, Agents, containers, and public endpoints.
+---
+
+# Security Hardening
+
+BackupX coordinates access to source files, database credentials, storage credentials, and restore destinations. Deploy the Master as a security-sensitive control plane, not as a general public web application.
+
+## Recommended exposure model
+
+| Component | Inbound access | Outbound access |
+| --- | --- | --- |
+| Master | HTTPS from administrators and Agents; metrics only from monitoring networks | Storage providers, notification endpoints, release checks |
+| Agent | No inbound port required | Master HTTPS endpoint and assigned storage targets |
+| SQLite data | Local or block-backed filesystem only | None |
+
+Bind Docker to `127.0.0.1` when a reverse proxy runs on the same host:
+
+~~~dotenv
+BACKUPX_BIND_ADDRESS=127.0.0.1
+~~~
+
+For bare metal, set `server.host` to loopback when only a local proxy should reach BackupX. Otherwise restrict TCP 8340 with the host or network firewall.
+
+## TLS and reverse proxies
+
+- Use HTTPS across every untrusted network segment.
+- Set `server.external_url` to the stable URL that Agents can reach.
+- Add only the exact proxy IP or subnet to `server.trusted_proxies`. Never trust `0.0.0.0/0`.
+- Send the final HTTPS URL to Agents; the Agent does not follow redirects.
+- For private PKI, install a PEM CA on the Agent and configure `caCertFile` or `--ca-cert`.
+- Use `--insecure-tls` only for temporary testing.
+- Keep Nginx request and response buffering disabled for relay uploads and SSE logs.
+
+When an SSH bastion is required, bind tunnels to loopback, verify host keys, use a dedicated account and key, and make the Agent service depend on the tunnel. See [Multi-Node Cluster](../features/multi-node).
+
+## Roles and API keys
+
+| Role | Intended access |
+| --- | --- |
+| `viewer` | Read dashboards, tasks, records, reports, and audit data; cannot browse node filesystems or mutate resources |
+| `operator` | Viewer access plus task, storage, notification, backup, restore, verification, and file-browse operations |
+| `admin` | Operator access plus users, API keys, settings, node lifecycle, install tokens, and token rotation |
+
+Create separate named users instead of sharing the initial administrator. Enable two-factor authentication or passkeys for privileged accounts. Review trusted devices and recovery codes periodically.
+
+User JWTs are stateless. Logout removes the client copy but does not revoke a token that was already copied elsewhere. Set `security.jwt_expire` to the shortest practical lifetime, protect Bearer tokens, and rotate the JWT secret when all active sessions must be invalidated.
+
+API keys use the same role checks as interactive users. Their plaintext is shown only once; the database stores a keyed hash. Give automation the lowest role it needs, set an expiry, keep the key in a secret manager, and revoke unused keys. Avoid administrator API keys for monitoring.
+
+## Protect control-plane secrets
+
+- Restrict `/etc/backupx/config.yaml` to `root:backupx` mode `0640` and the data directory to the service account.
+- If `jwt_secret` and `encryption_key` are empty, generated values are persisted in the SQLite database. Back up the complete data directory.
+- Losing or replacing the encryption key makes saved storage credentials unreadable.
+- The database includes password hashes, configuration secrets, Agent tokens, API-key hashes, trusted-device state, and audit data. Encrypt snapshots and control their retention.
+- Do not put tokens in shell history, issue text, screenshots, or support bundles.
+
+Each node has an independent long-lived Agent token. The systemd installer stores it in `/etc/backupx-agent/agent.token` with mode `0600`. Rotate a token after personnel changes, host compromise, or accidental disclosure, update the token file during the overlap window, then restart the Agent.
+
+One-time install URLs are valid for 5 minutes to 24 hours and are consumed after use. Treat the URL and the embedded fallback command as secrets: the generated installation material provisions the long-lived node token.
+
+## Container and host permissions
+
+The canonical Compose deployment drops all capabilities and adds back only those needed to repair legacy volume ownership and switch to the unprivileged `backupx` user. Keep `no-new-privileges` enabled and do not mount the Docker socket.
+
+Mount backup sources read-only. Add a separate, narrowly scoped writable mount only when a restore destination requires it. Prefer a host Agent over running the Master container as root for privileged filesystem access.
+
+The systemd Master runs as `backupx`. The Agent normally runs as root because it may back up or restore files belonging to arbitrary system users. Limit who can create tasks and protect the root-owned Agent configuration.
+
+## Public endpoints
+
+The following endpoints intentionally do not use BackupX JWT or API-key authentication:
+
+- `/health` and `/api/health`
+- `/ready` and `/api/ready`
+- `/metrics`
+- one-time `/install/:token` and `/api/install/:token` routes
+
+Health responses expose status, version, uptime, timestamp, and readiness checks; a failed readiness check can include database error detail. `/metrics` also includes node and storage-target labels. Restrict metrics and probes to monitoring networks at the firewall or reverse proxy. Do not cache or log full install-token URLs.
+
+## Backup encryption boundary
+
+Encrypted backup tasks run on the Master because remote Agents never receive the Master's encryption key. Do not work around this boundary by copying the Master key to Agents. For Agent-routed tasks, rely on transport encryption and the destination provider's server-side encryption when required.
+
+Test restores for encrypted backups after every key-management change. A backup whose key is unavailable is not recoverable.
+
+## Audit and incident response
+
+BackupX records privileged actions in the audit log and can forward signed audit events to an external webhook. Send high-value audit records to a separately administered SIEM or append-only store so a compromised Master cannot erase the only copy.
+
+After suspected compromise:
+
+1. Isolate the Master without deleting evidence.
+2. Revoke exposed API keys and rotate affected Agent tokens and storage credentials.
+3. Replace JWT and encryption keys only with a planned migration; changing the encryption key invalidates saved encrypted configuration.
+4. Review user, trusted-device, API-key, node, settings, restore, and deletion events.
+5. Recover from a known-good control-plane snapshot when integrity cannot be established.
+
+Use [Upgrade and Recovery](./upgrade-recovery) for the paired application-and-database recovery procedure.
--- a/docs-site/docs/operations/troubleshooting.md
+++ b/docs-site/docs/operations/troubleshooting.md
@@ -0,0 +1,160 @@
+---
+sidebar_position: 4
+title: Troubleshooting
+description: A safe diagnostic sequence for the Master, reverse proxy, Agents, backup tools, and SQLite.
+---
+
+# Troubleshooting
+
+Start with the first failing boundary and preserve evidence. Avoid deleting the database, recreating volumes, rotating every token, or reinstalling until the failure is understood.
+
+## Fast triage
+
+| Symptom | First check | Likely boundary |
+| --- | --- | --- |
+| Web console unavailable | Local `/health`, then proxy `/health` | Process, listener, firewall, proxy, or static assets |
+| `/health` works but `/ready` is 503 | Service logs, database path, disk space, ownership | SQLite or data filesystem |
+| Login loops or client IP is wrong | Forwarded headers and `trusted_proxies` | Reverse-proxy trust |
+| Live logs stop updating | Nginx response buffering and timeout | SSE proxy path |
+| Relay upload stalls or proxy disk fills | Request buffering and body-size limit | Reverse proxy |
+| Agent offline | Agent service logs, final Master URL, proxy, DNS, CA | Agent-to-Master path |
+| Backup starts but fails | Record log, source path, native database tool | Task runner or permissions |
+| Restore fails | Record log, destination mount and write access | Storage read or destination permissions |
+
+## Collect status without secrets
+
+Docker Master:
+
+~~~bash
+docker compose ps
+docker compose logs --tail=200 backupx
+curl -i http://127.0.0.1:8340/health
+curl -i http://127.0.0.1:8340/ready
+~~~
+
+Bare-metal Master:
+
+~~~bash
+sudo systemctl status backupx --no-pager
+sudo journalctl -u backupx -n 200 --no-pager
+sudo ss -lntp | grep 8340
+curl -i http://127.0.0.1:8340/health
+curl -i http://127.0.0.1:8340/ready
+~~~
+
+Systemd Agent:
+
+~~~bash
+sudo systemctl status backupx-agent --no-pager
+sudo journalctl -u backupx-agent -n 200 --no-pager
+sudo systemctl status backupx-agent-tunnel --no-pager
+~~~
+
+The tunnel command is relevant only to bastion deployments. Before sharing output, remove Authorization headers, API keys, Agent tokens, install URLs, database passwords, storage credentials, proxy credentials, and private paths that reveal sensitive topology.
+
+## Web console or first setup
+
+Check the unauthenticated setup endpoint:
+
+~~~bash
+curl -fsS http://127.0.0.1:8340/api/auth/setup/status
+~~~
+
+If the API works but the browser receives a blank page or JSON:
+
+- Confirm the release contains web assets.
+- Bare metal: verify `/opt/backupx/web` is readable and `server.web_root` is correct when explicitly set.
+- Docker: confirm the official image is running and no custom mount hides the packaged web directory.
+- Nginx static mode: confirm `root /opt/backupx/web` and SPA fallback are present.
+- Clear an old service-worker or browser cache after a release change.
+
+For authentication failures, verify system time before diagnosing TOTP or passkeys. Confirm the browser origin matches the final HTTPS host, and inspect the audit log for throttling, disabled users, or revoked trusted devices.
+
+## Reverse proxy
+
+Validate and reload Nginx:
+
+~~~bash
+sudo nginx -t
+sudo systemctl reload nginx
+curl -i https://backup.example.com/health
+curl -i https://backup.example.com/ready
+~~~
+
+Common corrections:
+
+- HTTP 413: set `client_max_body_size 0` for the API route.
+- Relay uploads fill proxy temporary storage: set `proxy_request_buffering off`.
+- SSE logs arrive in bursts or disconnect: set `proxy_buffering off`, disable proxy cache, and increase read timeout.
+- One-click installer returns HTML: proxy `/api/` and retain the legacy `/install/` route.
+- Agent receives a redirect: configure the final HTTPS Master URL instead of an HTTP URL.
+- Audit shows the proxy address for every user: add only the real proxy IP or subnet to `server.trusted_proxies`.
+
+Use the complete [Nginx configuration](../deployment/nginx) as the comparison baseline.
+
+## Agent offline
+
+An Agent normally heartbeats every 15 seconds and is marked offline after 45 seconds.
+
+1. Confirm the Agent and optional tunnel services are active.
+2. Verify the configured Master URL has no trailing redirect and resolves from the Agent host.
+3. Check the explicit `proxyUrl`. Use `socks5h://` when DNS must resolve through an SSH dynamic tunnel.
+4. Confirm the private CA path exists and is readable. Do not switch permanently to insecure TLS.
+5. Check outbound firewall access to the Master and assigned storage backends.
+6. Verify `/etc/backupx-agent/agent.token` exists with mode `0600`.
+7. If a token was rotated, install the new value during the overlap window and restart the Agent.
+
+Do not paste the token into a diagnostic command that will be saved in shell history. A 401 in Agent logs usually indicates a missing, expired-overlap, or mismatched node token; repeated connection errors indicate URL, DNS, proxy, tunnel, firewall, or CA problems.
+
+## Backup task failures
+
+Open the backup record and inspect its complete log before changing the task.
+
+- File tasks resolve paths on the selected Master or Agent. Confirm the path exists in that host's namespace.
+- Docker sees only mounted paths. Backup mounts should normally be read-only.
+- MySQL requires `mysqldump` on the execution host's `PATH`.
+- PostgreSQL requires `pg_dump` on the execution host's `PATH`.
+- SAP HANA runner mode requires its configured client tools and environment.
+- Confirm the service account can read sources and write the temporary directory.
+- Test the selected storage target from the console.
+- Check DNS, egress policy, provider quota, clock skew, and proxy settings for remote storage.
+
+If multiple targets are configured, inspect the per-target result instead of assuming every copy failed. Preserve successful remote artifacts while correcting the failing target.
+
+## Restore, download, or verification failures
+
+- Confirm the remote artifact still exists and the storage credentials can read it.
+- Check that the destination is mounted on the host that performs the restore.
+- Use a separate writable restore path; do not make every backup-source mount writable.
+- Check free space in the destination and Agent temporary directory.
+- For encrypted backups, confirm the original Master encryption key is available.
+- For CDC repositories, keep manifests, indexes, and shared packs together; a manifest alone is not a complete backup.
+
+Prefer an isolated restore destination during diagnosis. Do not repeatedly restore over the production source.
+
+## SQLite and readiness failures
+
+When `/health` is 200 but `/ready` is 503:
+
+1. Read the exact database error from service logs.
+2. Check free disk space, inode availability, path ownership, and mount state.
+3. Confirm only one Master process or container uses the data directory.
+4. Keep SQLite on a local or block-backed filesystem, not a shared multi-writer or unreliable network filesystem.
+5. Check whether an external backup or antivirus process is holding files for long periods.
+
+BackupX uses a five-second SQLite busy timeout, but that does not make SQLite a clustered database. Do not fix lock errors by starting another Master. For a file-level copy, stop the service and copy the whole data directory.
+
+## Escalation package
+
+When opening an issue, include:
+
+- BackupX version, installation method, operating system, and architecture.
+- Whether the failure affects the Master, Agent, proxy, storage target, or one task.
+- Redacted service logs covering the first failure.
+- HTTP status and response body from `/health` and `/ready`.
+- A minimal reproduction and whether it began after an upgrade or configuration change.
+- Relevant proxy configuration with hostnames, credentials, and private addresses redacted.
+
+Never attach `backupx.db`, `.env`, full configuration files, Agent token files, API keys, install commands, or storage credentials to a public issue.
+
+If integrity or rollback is involved, stop making destructive changes and follow [Upgrade and Recovery](./upgrade-recovery).
--- a/docs-site/docs/operations/upgrade-recovery.md
+++ b/docs-site/docs/operations/upgrade-recovery.md
@@ -0,0 +1,153 @@
+---
+sidebar_position: 1
+title: Upgrade and Recovery
+description: Back up the control plane, upgrade safely, roll back as a unit, and recover a failed Master.
+---
+
+# Upgrade and Recovery
+
+Backup artifacts and the BackupX control plane are different recovery domains. Object storage may still contain every archive while a lost Master database removes users, encrypted storage credentials, schedules, records, node tokens, and audit history. Protect both.
+
+## Non-negotiable rules
+
+1. Run exactly one active Master against a data directory or SQLite database.
+2. Snapshot the complete data directory and configuration while the Master is stopped, or use a storage-level atomic snapshot.
+3. Keep the old application version and its pre-upgrade data snapshot together. Schema migration happens at startup, so switching only the binary or image back is not a safe rollback.
+4. Store control-plane snapshots outside the Master host and test restoring them.
+5. Let active backup and restore jobs finish before stopping the Master.
+
+| Deployment | Persistent control-plane data | Configuration and release state |
+| --- | --- | --- |
+| Docker | `/app/data` in the `backupx-data` volume | Compose file, protected `.env`, pinned image tag or digest |
+| Bare metal | `/opt/backupx/data` | `/etc/backupx`, `/opt/backupx/bin`, `/opt/backupx/web`, systemd unit |
+
+The SQLite database contains generated JWT and encryption keys when they are not supplied in configuration. Treat every control-plane snapshot as a secret.
+
+## Change checklist
+
+Before an upgrade, host migration, or security-key change:
+
+- Record the current BackupX version and the exact image digest or release checksum.
+- Confirm `/ready` returns HTTP 200 and review recent failures.
+- Wait for running backup, restore, verification, and replication work to finish.
+- Test at least one storage target and confirm Agents are online.
+- Create a full control-plane snapshot and copy it off-host.
+- Optionally export task definitions for human review. Task export excludes database passwords and storage credentials, so it is not a replacement for the database snapshot.
+- Define the rollback decision and maintenance-window deadline before starting.
+
+## Snapshot a Docker deployment
+
+This example creates a consistent file-level copy without requiring access to Docker's volume directory:
+
+~~~bash
+snapshot="backupx-control-plane-$(date -u +%Y%m%dT%H%M%SZ)"
+install -d -m 0700 "$snapshot"
+
+docker compose stop backupx
+docker cp backupx:/app/data "$snapshot/data"
+cp docker-compose.yml "$snapshot/"
+if [ -f .env ]; then cp .env "$snapshot/"; fi
+docker compose start backupx
+
+tar -czf "$snapshot.tar.gz" "$snapshot"
+sha256sum "$snapshot.tar.gz" > "$snapshot.tar.gz.sha256"
+curl -fsS http://127.0.0.1:8340/ready
+~~~
+
+If copying fails, start the stopped service before investigating. Protect the archive because `.env` and the database can contain credentials. A block-volume or storage-provider snapshot is also valid when it is atomic across the whole volume.
+
+## Snapshot a bare-metal deployment
+
+~~~bash
+snapshot="/var/backups/backupx/backupx-control-plane-$(date -u +%Y%m%dT%H%M%SZ).tar.gz"
+sudo install -d -m 0700 /var/backups/backupx
+
+sudo systemctl stop backupx
+sudo tar --acls --xattrs -C / -czf "$snapshot" \
+  etc/backupx \
+  etc/systemd/system/backupx.service \
+  opt/backupx/bin \
+  opt/backupx/web \
+  opt/backupx/data
+sudo systemctl start backupx
+
+sudo sha256sum "$snapshot" | sudo tee "$snapshot.sha256"
+curl -fsS http://127.0.0.1:8340/ready
+~~~
+
+Copy the archive and checksum to protected off-host storage. Do not copy only `backupx.db` while the service is running.
+
+## Upgrade Docker
+
+1. Put a release tag or immutable digest in `BACKUPX_IMAGE`. Do not use `latest` for a controlled production upgrade.
+2. Create and verify the pre-upgrade snapshot.
+3. Pull and recreate the service:
+
+~~~bash
+docker compose pull backupx
+docker compose up -d backupx
+docker compose ps
+docker compose logs --tail=100 backupx
+curl -fsS http://127.0.0.1:8340/ready
+~~~
+
+4. Sign in, test a storage target, confirm Agent heartbeats, and run one small backup plus a restore or verification drill.
+5. Keep the old image reference and snapshot until the observation window ends.
+
+Upgrade Agents after the Master, in small batches. Keep the node-specific proxy, private-CA, token-file, and bastion configuration unchanged unless that configuration is the purpose of the change.
+
+## Upgrade bare metal
+
+Download the target release and checksum, verify them, then extract the archive. The installer preserves an existing `/etc/backupx/config.yaml`, replaces the binary, web assets, and systemd unit, and restarts the service.
+
+~~~bash
+sha256sum -c backupx-vX.Y.Z-linux-amd64.tar.gz.sha256
+tar xzf backupx-vX.Y.Z-linux-amd64.tar.gz
+cd backupx-vX.Y.Z-linux-amd64
+sudo ./install.sh
+
+sudo systemctl status backupx --no-pager
+curl -fsS http://127.0.0.1:8340/ready
+~~~
+
+Create the stopped-service snapshot before running the installer. Use the same post-upgrade application checks as Docker.
+
+## Roll back
+
+Rollback is a paired operation: restore both the previous application release and the snapshot created immediately before the upgrade.
+
+For Docker, preserve the failed volume for analysis and restore the snapshot into a new empty volume. Point Compose at that volume and the previous image tag, then start exactly one Master. For bare metal, stop the service, preserve the failed state, restore the old configuration, binary, web assets, data directory, and unit from the same archive, reload systemd, and start the service.
+
+After rollback:
+
+~~~bash
+curl -fsS http://127.0.0.1:8340/health
+curl -fsS http://127.0.0.1:8340/ready
+~~~
+
+Then verify login, storage access, schedules, Agent heartbeats, a backup, and a non-destructive restore drill. Do not delete the failed state until the incident is understood.
+
+## Recover a lost Master
+
+1. Provision a replacement host with the same architecture and the exact application version recorded with the snapshot.
+2. Keep the replacement isolated from production traffic and ensure the old Master cannot start.
+3. Restore configuration and the complete data directory with their original permissions.
+4. Start one Master and check `/ready` locally.
+5. Move the stable DNS name or virtual IP only after local validation.
+6. Confirm users, storage targets, tasks, records, notifications, and audit history.
+7. Existing Agents reconnect automatically when the restored database contains their matching tokens. Investigate and rotate tokens that may have been exposed.
+8. Run a small backup and a restore or verification drill before ending the incident.
+
+External backup artifacts are not recreated by restoring the control plane; they remain on their configured storage targets. Conversely, task JSON export is useful for rebuilding schedules but omits secrets, storage definitions, and some node bindings. Use it only as an additional recovery aid.
+
+## Test the recovery plan
+
+At least quarterly, restore a recent snapshot into an isolated network, start the recorded BackupX version, and verify:
+
+- `/ready` becomes healthy without contacting the production Master.
+- An administrator can sign in and encrypted storage configurations can be read.
+- Task, node, record, and audit counts are plausible.
+- A storage target can be tested without writing production data.
+- A selected backup can be verified or restored to an isolated destination.
+
+Record restore duration and the newest recoverable snapshot time. Those measured values are the real control-plane RTO and RPO.
--- a/docs-site/docs/reference/api.md
+++ b/docs-site/docs/reference/api.md
@@ -1,135 +1,268 @@
 ---
 sidebar_position: 1
 title: API Reference
-description: REST API endpoints — all under /api with JWT Bearer authentication.
+description: BackupX REST endpoints, authentication methods, role boundaries, streaming responses, and public probes.
 ---

 # API Reference

-All endpoints are prefixed with `/api` and authenticated with a JWT Bearer token, obtained via `POST /api/auth/login`. Agent endpoints use `X-Agent-Token` instead.
+The interactive API is rooted at `/api`. Most endpoints accept either a user JWT or an API key; Agent protocol endpoints use a node-specific token. Public probes and one-time installers are listed separately.

 ## Authentication

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/auth/setup/status` | Check whether admin initialization is needed |
-| `POST` | `/api/auth/setup` | Initialize the first admin (only when no user exists) |
-| `POST` | `/api/auth/login` | Log in and receive a JWT |
-| `POST` | `/api/auth/logout` | Log out (invalidate current token) |
-| `GET`  | `/api/auth/profile` | Current user profile |
-| `PUT`  | `/api/auth/password` | Change password |
+### User JWT

-## Backup Tasks
+Obtain a JWT through `POST /api/auth/login` and send it as a Bearer token:

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/backup/tasks` | List tasks |
-| `POST` | `/api/backup/tasks` | Create |
-| `GET` | `/api/backup/tasks/:id` | Detail |
-| `PUT` | `/api/backup/tasks/:id` | Update |
-| `DELETE` | `/api/backup/tasks/:id` | Delete |
-| `PUT` | `/api/backup/tasks/:id/toggle` | Enable / disable |
-| `POST` | `/api/backup/tasks/:id/run` | Trigger a manual run |
+~~~bash
+curl -H "Authorization: Bearer $BACKUPX_TOKEN" \
+  https://backup.example.com/api/backup/tasks
+~~~

-## Backup Records
+The login flow may require OTP, TOTP, recovery code, a trusted-device token, or WebAuthn depending on account and system settings.

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/backup/records` | List records with filters |
-| `GET` | `/api/backup/records/:id` | Record detail |
-| `GET` | `/api/backup/records/:id/logs/stream` | Live logs (SSE) |
-| `GET` | `/api/backup/records/:id/download` | Download the artifact |
-| `POST` | `/api/backup/records/:id/restore` | Restore to the original source |
-| `DELETE` | `/api/backup/records/:id` | Delete a record |
-| `POST` | `/api/backup/records/batch-delete` | Bulk delete |
+### API key

-## Storage Targets
+An administrator creates API keys in the console or through `POST /api/api-keys`. The plaintext `bax_...` value is returned only once.

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/storage-targets` | List |
-| `POST` | `/api/storage-targets` | Create |
-| `GET` | `/api/storage-targets/:id` | Detail |
-| `PUT` | `/api/storage-targets/:id` | Update |
-| `DELETE` | `/api/storage-targets/:id` | Delete |
-| `POST` | `/api/storage-targets/test` | Test connection with pending config |
-| `POST` | `/api/storage-targets/:id/test` | Re-test a saved target |
-| `PUT` | `/api/storage-targets/:id/star` | Toggle favourite |
-| `GET` | `/api/storage-targets/:id/usage` | Query remote usage (where supported) |
-| `GET` | `/api/storage-targets/rclone/backends` | List all available rclone backends |
-| `POST` | `/api/storage-targets/google-drive/auth-url` | Start Google Drive OAuth |
-| `POST` | `/api/storage-targets/google-drive/complete` | Complete OAuth flow |
+~~~bash
+curl -H "X-Api-Key: $BACKUPX_API_KEY" \
+  https://backup.example.com/api/dashboard/stats
+~~~

-## Nodes (Cluster)
+`Authorization: Bearer bax_...` is also accepted. API keys carry an `admin`, `operator`, or `viewer` role and can be disabled or given an expiry.

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/nodes` | List nodes |
-| `POST` | `/api/nodes` | Create a node and return its token |
-| `GET` | `/api/nodes/:id` | Node detail |
-| `PUT` | `/api/nodes/:id` | Rename |
-| `DELETE` | `/api/nodes/:id` | Delete (rejected if tasks are still attached) |
-| `GET` | `/api/nodes/:id/fs/list` | Browse a directory (remote nodes use an async RPC via Agent) |
+### Agent token

-## Agent Protocol (X-Agent-Token)
+Agent protocol handlers authenticate the node token supplied in `X-Agent-Token`. This token is not a user credential and must not be used with the interactive resource API.

-Dedicated endpoints for the Agent CLI. Authenticated via the `X-Agent-Token` header instead of JWT.
+### Access labels

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `POST` | `/api/agent/heartbeat` | Report liveness; returns the node ID |
-| `POST` | `/api/agent/commands/poll` | Claim one pending command |
-| `POST` | `/api/agent/commands/:id/result` | Report command result |
-| `GET` | `/api/agent/tasks/:id` | Fetch task spec with decrypted storage configs |
-| `POST` | `/api/agent/records/:id` | Append logs / update record status |
+The tables use these labels:

-## Notifications
+| Label | Required access |
+| --- | --- |
+| Public | No JWT or API key; an install route still requires its one-time token |
+| Auth | Any authenticated `viewer`, `operator`, or `admin` |
+| Operator | `operator` or `admin` |
+| Admin | `admin` only |
+| Agent | Valid node-specific Agent token |

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/notifications` | List |
-| `POST` | `/api/notifications` | Create |
-| `GET` | `/api/notifications/:id` | Detail |
-| `PUT` | `/api/notifications/:id` | Update |
-| `DELETE` | `/api/notifications/:id` | Delete |
-| `POST` | `/api/notifications/test` | Test with pending config |
-| `POST` | `/api/notifications/:id/test` | Re-test a saved notifier |
+Viewers can use read endpoints except node filesystem browsing. Operators can run and mutate backup resources. Administrators additionally manage users, API keys, settings, nodes, install tokens, and node-token rotation. A rejected role returns HTTP 403.

-## Dashboard
+## Authentication and account security

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/dashboard/stats` | Overview statistics |
-| `GET` | `/api/dashboard/timeline` | Recent activity timeline |
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/auth/setup/status` | Public | Check whether first-admin setup is required |
+| `POST` | `/api/auth/setup` | Public | Create the first administrator when no user exists |
+| `POST` | `/api/auth/login` | Public | Complete password or MFA login and obtain a JWT |
+| `POST` | `/api/auth/otp/send` | Public | Send a configured login OTP |
+| `POST` | `/api/auth/webauthn/login/options` | Public | Begin passkey login |
+| `POST` | `/api/auth/logout` | Auth | Acknowledge logout; the client must discard its stateless JWT |
+| `GET` | `/api/auth/profile` | Auth | Read the current account |
+| `PUT` | `/api/auth/password` | Auth | Change the current account password |
+| `POST` | `/api/auth/2fa/setup` | Auth | Prepare TOTP enrollment |
+| `POST` | `/api/auth/2fa/enable` | Auth | Enable TOTP after verification |
+| `POST` | `/api/auth/2fa/recovery-codes` | Auth | Regenerate recovery codes |
+| `DELETE` | `/api/auth/2fa` | Auth | Disable TOTP |
+| `PUT` | `/api/auth/otp/config` | Auth | Update OTP login configuration |
+| `POST` | `/api/auth/webauthn/register/options` | Auth | Begin passkey registration |
+| `POST` | `/api/auth/webauthn/register/finish` | Auth | Finish passkey registration |
+| `GET` | `/api/auth/webauthn/credentials` | Auth | List passkeys |
+| `DELETE` | `/api/auth/webauthn/credentials/:id` | Auth | Delete a passkey |
+| `GET` | `/api/auth/trusted-devices` | Auth | List trusted devices |
+| `DELETE` | `/api/auth/trusted-devices/:id` | Auth | Revoke a trusted device |

-## Audit / System / Settings
+Use an interactive JWT, not an automation API key, for account-security endpoints.

-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/api/audit-logs` | Audit log list |
-| `GET` | `/api/system/info` | System information |
-| `GET` | `/api/system/update-check` | Check for a newer release |
-| `GET` | `/api/settings` | System-level settings |
-| `PUT` | `/api/settings` | Update system settings |
+## System and storage targets

-## Response Envelope
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/system/info` | Auth | Version and system information |
+| `GET` | `/api/system/update-check` | Auth | Check available releases |
+| `GET` | `/api/storage-targets` | Auth | List storage targets |
+| `POST` | `/api/storage-targets` | Operator | Create a target |
+| `POST` | `/api/storage-targets/test` | Operator | Test an unsaved configuration |
+| `GET` | `/api/storage-targets/rclone/backends` | Auth | List available rclone backends |
+| `POST` | `/api/storage-targets/google-drive/auth-url` | Operator | Start Google Drive authorization |
+| `POST` | `/api/storage-targets/google-drive/complete` | Operator | Complete Google Drive authorization |
+| `GET` | `/api/storage-targets/google-drive/callback` | Auth | Handle the OAuth callback |
+| `GET` | `/api/storage-targets/:id` | Auth | Read a target |
+| `PUT` | `/api/storage-targets/:id` | Operator | Update a target |
+| `DELETE` | `/api/storage-targets/:id` | Operator | Delete a target |
+| `PUT` | `/api/storage-targets/:id/star` | Operator | Toggle favorite state |
+| `POST` | `/api/storage-targets/:id/test` | Operator | Test a saved target |
+| `GET` | `/api/storage-targets/:id/usage` | Auth | Read recorded usage |
+| `GET` | `/api/storage-targets/:id/google-drive/profile` | Auth | Read the connected Google Drive profile |

-All successful responses follow the shape:
+## Backup tasks

-```json
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/backup/tasks` | Auth | List tasks |
+| `GET` | `/api/backup/tasks/tags` | Auth | List task tags |
+| `GET` | `/api/backup/tasks/export` | Auth | Download all task definitions, or select them with `?ids=1,2` |
+| `POST` | `/api/backup/tasks/import` | Operator | Import task definitions, up to 1 MiB |
+| `POST` | `/api/backup/tasks/batch/toggle` | Operator | Enable or disable tasks in bulk |
+| `POST` | `/api/backup/tasks/batch/delete` | Operator | Delete tasks in bulk |
+| `POST` | `/api/backup/tasks/batch/run` | Operator | Run tasks in bulk |
+| `GET` | `/api/backup/tasks/:id` | Auth | Read a task |
+| `POST` | `/api/backup/tasks` | Operator | Create a task |
+| `PUT` | `/api/backup/tasks/:id` | Operator | Update a task |
+| `DELETE` | `/api/backup/tasks/:id` | Operator | Delete a task |
+| `PUT` | `/api/backup/tasks/:id/toggle` | Operator | Enable or disable a task |
+| `POST` | `/api/backup/tasks/:id/run` | Operator | Trigger a backup |
+| `POST` | `/api/backup/tasks/:id/verify` | Operator | Trigger verification from a task |
+
+Task export intentionally excludes database passwords and storage credentials. It is useful for migration and review, not a complete control-plane backup.
+
+## Backup and restore records
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/backup/records` | Auth | List and filter backup records |
+| `POST` | `/api/backup/records/batch-delete` | Operator | Delete records in bulk |
+| `GET` | `/api/backup/records/:id` | Auth | Read a backup record |
+| `GET` | `/api/backup/records/:id/logs/stream` | Auth | Stream logs with server-sent events |
+| `GET` | `/api/backup/records/:id/download` | Auth | Download an artifact |
+| `GET` | `/api/backup/records/:id/contents` | Auth | Browse artifact contents where supported |
+| `POST` | `/api/backup/records/:id/restore` | Operator | Start a restore |
+| `POST` | `/api/backup/records/:id/replicate` | Operator | Replicate an existing artifact |
+| `POST` | `/api/backup/records/:id/verify` | Operator | Verify an existing artifact |
+| `PUT` | `/api/backup/records/:id/lock` | Operator | Set retention lock state |
+| `DELETE` | `/api/backup/records/:id` | Operator | Delete a record and its managed artifact |
+| `GET` | `/api/restore/records` | Auth | List restore records |
+| `GET` | `/api/restore/records/:id` | Auth | Read a restore record |
+| `GET` | `/api/restore/records/:id/logs/stream` | Auth | Stream restore logs |
+| `GET` | `/api/replication/records` | Auth | List replication records |
+| `GET` | `/api/replication/records/:id` | Auth | Read a replication record |
+| `GET` | `/api/verify/records` | Auth | List verification records |
+| `GET` | `/api/verify/records/:id` | Auth | Read a verification record |
+| `GET` | `/api/verify/records/:id/logs/stream` | Auth | Stream verification logs |
+
+## Templates, reports, and dashboard
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/task-templates` | Auth | List task templates |
+| `GET` | `/api/task-templates/:id` | Auth | Read a task template |
+| `POST` | `/api/task-templates` | Operator | Create a template |
+| `PUT` | `/api/task-templates/:id` | Operator | Update a template |
+| `DELETE` | `/api/task-templates/:id` | Operator | Delete a template |
+| `POST` | `/api/task-templates/:id/apply` | Operator | Create tasks from a template |
+| `GET` | `/api/reports/compliance` | Auth | Read compliance evidence |
+| `GET` | `/api/reports/compliance/export` | Auth | Export compliance evidence as CSV |
+| `GET` | `/api/dashboard/stats` | Auth | Summary statistics |
+| `GET` | `/api/dashboard/timeline` | Auth | Recent activity |
+| `GET` | `/api/dashboard/sla` | Auth | RPO and SLA status |
+| `GET` | `/api/dashboard/cluster` | Auth | Cluster summary |
+| `GET` | `/api/dashboard/breakdown` | Auth | Task and record breakdown |
+| `GET` | `/api/dashboard/node-performance` | Auth | Per-node performance |
+
+## Notifications, settings, and administration
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/notifications` | Auth | List notification channels |
+| `GET` | `/api/notifications/:id` | Auth | Read a channel |
+| `POST` | `/api/notifications` | Operator | Create a channel |
+| `PUT` | `/api/notifications/:id` | Operator | Update a channel |
+| `DELETE` | `/api/notifications/:id` | Operator | Delete a channel |
+| `POST` | `/api/notifications/test` | Operator | Test an unsaved configuration |
+| `POST` | `/api/notifications/:id/test` | Operator | Test a saved channel |
+| `GET` | `/api/settings` | Auth | Read system settings |
+| `PUT` | `/api/settings` | Admin | Update system settings |
+| `GET` | `/api/users` | Admin | List users |
+| `POST` | `/api/users` | Admin | Create a user |
+| `PUT` | `/api/users/:id` | Admin | Update a user |
+| `POST` | `/api/users/:id/2fa/reset` | Admin | Reset a user's second factor |
+| `DELETE` | `/api/users/:id` | Admin | Delete a user |
+| `GET` | `/api/api-keys` | Admin | List API keys without plaintext values |
+| `POST` | `/api/api-keys` | Admin | Create an API key and return its plaintext once |
+| `PUT` | `/api/api-keys/:id/toggle` | Admin | Enable or disable an API key |
+| `DELETE` | `/api/api-keys/:id` | Admin | Revoke an API key |
+
+## Audit, events, search, and discovery
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/audit-logs` | Auth | List and filter audit records |
+| `GET` | `/api/audit-logs/export` | Auth | Export audit records |
+| `GET` | `/api/events/stream` | Auth | Stream real-time application events with SSE |
+| `GET` | `/api/search` | Auth | Search supported resources |
+| `POST` | `/api/database/discover` | Auth | Discover databases from supplied connection details |
+
+## Nodes
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/api/nodes` | Auth | List nodes |
+| `GET` | `/api/nodes/:id` | Auth | Read a node |
+| `GET` | `/api/nodes/:id/fs/list` | Operator | Browse the selected node filesystem |
+| `POST` | `/api/nodes` | Admin | Create a node |
+| `POST` | `/api/nodes/batch` | Admin | Create up to 50 nodes |
+| `PUT` | `/api/nodes/:id` | Admin | Update a node |
+| `DELETE` | `/api/nodes/:id` | Admin | Delete an unreferenced node |
+| `POST` | `/api/nodes/:id/install-tokens` | Admin | Create a one-time installer |
+| `GET` | `/api/nodes/:id/install-script-preview` | Admin | Preview generated install material |
+| `POST` | `/api/nodes/:id/rotate-token` | Admin | Rotate the long-lived node token |
+
+## Agent protocol
+
+These routes are for the `backupx agent` process and authenticate inside the handler with the node token.
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `POST` | `/api/agent/heartbeat` | Agent | Report liveness and node state |
+| `POST` | `/api/agent/commands/poll` | Agent | Claim a pending command |
+| `POST` | `/api/agent/commands/:id/result` | Agent | Report a command result |
+| `GET` | `/api/agent/tasks/:id` | Agent | Fetch a runnable task specification |
+| `POST` | `/api/agent/records/:id` | Agent | Append logs or update backup state |
+| `PUT` | `/api/agent/records/:id/artifacts/:targetId` | Agent | Stream a relayed artifact to the Master |
+| `GET` | `/api/agent/restores/:id/spec` | Agent | Fetch restore instructions |
+| `GET` | `/api/agent/restores/:id/artifact` | Agent | Stream a restore artifact |
+| `POST` | `/api/agent/restores/:id` | Agent | Update restore state |
+| `GET` | `/api/v1/agent/self` | Agent | Validate node identity during installation |
+
+## Public operational and install routes
+
+| Method | Endpoint | Access | Description |
+| --- | --- | --- | --- |
+| `GET` | `/health` | Public | Liveness |
+| `GET` | `/api/health` | Public | API-prefixed liveness alias |
+| `GET` | `/ready` | Public | SQLite readiness |
+| `GET` | `/api/ready` | Public | API-prefixed readiness alias |
+| `GET` | `/metrics` | Public | Prometheus metrics |
+| `GET` | `/install/:token` | Public | Consume a one-time Agent installer token |
+| `GET` | `/api/install/:token` | Public | API-prefixed installer route |
+| `GET` | `/install/:token/compose.yml` | Public | Render a Docker Agent Compose file |
+| `GET` | `/api/install/:token/compose.yml` | Public | API-prefixed Docker Compose route |
+
+Restrict probes and metrics to monitoring networks. Install tokens are single-use, time-limited secrets and must not be written to public logs.
+
+## Response formats
+
+Most JSON successes use:
+
+~~~json
 {
  "code": "OK",
-  "message": "",
-  "data": { /* actual payload */ }
+  "message": "success",
+  "data": {}
 }
-```
+~~~

-Errors return an HTTP 4xx/5xx plus:
+Errors use an HTTP 4xx or 5xx status plus a stable application code:

-```json
+~~~json
 {
  "code": "BACKUP_TASK_NOT_FOUND",
-  "message": "备份任务不存在",
-  "data": null
+  "message": "备份任务不存在"
 }
-```
+~~~
+
+Clients should branch on the HTTP status and `code`, not the localized `message`.
+
+Artifact downloads, task JSON export, audit or compliance exports, installer responses, and `/metrics` return their native content types instead of the JSON envelope. Log and event streams use `text/event-stream`; reverse proxies must keep response buffering disabled.
--- a/docs-site/docs/reference/cli.md
+++ b/docs-site/docs/reference/cli.md
@@ -17,15 +17,17 @@ backupx --version

 | Flag | Description |
 |------|-------------|
-| `--config <path>` | Path to config YAML (default: `./config.yaml`) |
+| `--config <path>` | Explicit config YAML path; omitted uses the search paths below |
 | `--version` | Print version and exit |

+When `--config` is omitted, the server searches `./config.yaml`, `./server/config.yaml`, and `/etc/backupx/config.yaml`. `BACKUPX_*` environment variables override matching server configuration keys. See [Configuration Reference](../deployment/configuration).
+
 ## `backupx agent`

 Run in Agent mode, connecting to a Master. See [Multi-Node Cluster](../features/multi-node).

 ```bash
-backupx agent --master http://master:8340 --token <token>
+backupx agent --master https://backup.example.com --token-file /etc/backupx-agent/agent.token
 ```

 | Flag | Description |
@@ -33,13 +35,15 @@ backupx agent --master http://master:8340 --token <token>
 | `--master <url>` | Master URL |
 | `--token <token>` | Agent auth token |
 | `--token-file <path>` | Read the Agent Token from a file; preferred for services and containers |
-| `--config <path>` | YAML config (takes precedence over env) |
-| `--temp-dir <path>` | Local temp directory (default `/tmp/backupx-agent`) |
+| `--config <path>` | Load Agent YAML; when present, environment-based Agent config is not loaded |
+| `--temp-dir <path>` | Local temp directory (default `/var/lib/backupx-agent/tmp`) |
 | `--proxy-url <url>` | Explicit HTTP(S) or SOCKS5(H) proxy |
 | `--ca-cert <path>` | PEM CA certificate used to verify the Master |
 | `--insecure-tls` | Skip TLS verification (testing only) |

-Environment variables: `BACKUPX_AGENT_MASTER`, `BACKUPX_AGENT_TOKEN`, `BACKUPX_AGENT_TOKEN_FILE`, `BACKUPX_AGENT_HEARTBEAT`, `BACKUPX_AGENT_POLL`, `BACKUPX_AGENT_TEMP_DIR`, `BACKUPX_AGENT_PROXY_URL`, `BACKUPX_AGENT_CA_CERT_FILE`, `BACKUPX_AGENT_INSECURE_TLS`. When no explicit proxy URL is set, the Agent also honors `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`.
+Agent precedence is explicit CLI flags over a YAML file. If `--config` is not supplied, Agent settings are loaded from `BACKUPX_AGENT_MASTER`, `BACKUPX_AGENT_TOKEN`, `BACKUPX_AGENT_TOKEN_FILE`, `BACKUPX_AGENT_HEARTBEAT`, `BACKUPX_AGENT_POLL`, `BACKUPX_AGENT_TEMP_DIR`, `BACKUPX_AGENT_PROXY_URL`, `BACKUPX_AGENT_CA_CERT_FILE`, and `BACKUPX_AGENT_INSECURE_TLS`. When no explicit proxy URL is set, the Agent also honors `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`.
+
+`--token` overrides `--token-file`. Keep long-lived tokens in a root-readable file rather than command history. A private CA and `--insecure-tls` cannot be enabled together.

 ## `backupx backint`

@@ -57,6 +61,8 @@ backupx backint -f <function> -i <input> -o <output> -p <params>
 | `-p <path>` | Parameter file |
 | `-u / -c / -l / -v` | Accepted and ignored for SAP compatibility |

+The `-p` file must define `STORAGE_TYPE` and either `STORAGE_CONFIG_JSON` or `STORAGE_CONFIG`. Optional keys include `PARALLEL_FACTOR`, `COMPRESS`, `LOG_FILE`, `CATALOG_DB`, and `KEY_PREFIX`.
+
 ## `backupx reset-password`

 Reset an admin password directly in the SQLite database. No server restart needed.
@@ -70,3 +76,5 @@ backupx reset-password --username admin --password 'newpass123' [--config /path/
 | `--username` | Target username (default: `admin`) |
 | `--password` | New password (min 8 chars, required) |
 | `--config` | Config path (used to locate the database file) |
+
+Run this command on the Master host with access to the configured SQLite path. Avoid placing the new password directly in retained shell history.