From f33f93e74d15d62acea7416ac32d206a0fd90908 Mon Sep 17 00:00:00 2001 From: DullJZ <79080562+DullJZ@users.noreply.github.com> Date: Thu, 2 Oct 2025 21:02:57 +0800 Subject: [PATCH] English doc --- README.md | 2 + README_EN.md | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 README_EN.md diff --git a/README.md b/README.md index ee478c5..f21fc6e 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # S3 Balance +中文 | [English](./README_EN.md) + S3 Balance 是一个用 Go 编写的 S3 兼容负载均衡器,可在多套对象存储之间自动选择最佳目标桶,向上提供完整 S3 API。项目支持多种后端(AWS S3、MinIO、OSS 等)、可插拔负载策略、健康检查与统计采集、代理/重定向两种传输模式,以及简易认证和虚拟主机访问。 ## 关键特性 diff --git a/README_EN.md b/README_EN.md new file mode 100644 index 0000000..74dd29f --- /dev/null +++ b/README_EN.md @@ -0,0 +1,109 @@ +# S3 Balance + +S3 Balance is an S3-compatible load balancer written in Go. It automatically selects the best target bucket among multiple object storage backends and provides a complete S3 API. The project supports various backends (AWS S3, MinIO, OSS, etc.), pluggable balancing strategies, health checks and metrics collection, proxy/redirect transfer modes, simple authentication, and virtual host access. + +## Key Features + +- **Standard S3 API**: Bucket and object operations, multipart uploads, and pre-signed URLs are fully compatible with native S3. +- **Multi-bucket Scheduling**: Supports strategies like round-robin, least-space, weighted, and consistent hashing, with hot-swapping capabilities. +- **Health Monitoring**: Periodic health checks and capacity statistics, with Prometheus metrics exposed at `/metrics`. +- **Virtual Bucket Mapping**: Only virtual bucket names are exposed externally, while real buckets are transparently scheduled in the backend. +- **Proxy or Redirect Mode**: Choose between service-forwarded data or pre-signed URLs for direct client connections. +- **SigV4 Authentication**: When `s3api.auth_required` is enabled, AWS Signature Version 4 requests are validated using `github.com/DullJZ/s3-validate`. + +## Quick Start + +### Docker Compose + +#### S3 Balance Service Only + +```yaml +services: + s3-balance: + image: dulljz/s3-balance:latest + container_name: s3-balance + ports: + - "8080:8080" + volumes: + - ./config.yaml:/app/config/config.yaml + - ./data:/app/data + environment: + - TZ=Asia/Shanghai + restart: unless-stopped + command: ["/app/s3-balance", "-config", "/app/config/config.yaml"] +``` + +#### With Prometheus & Grafana Monitoring + +```bash +# Clone the repository +git clone https://github.com/DullJZ/s3-balance.git --depth=1 +cd s3-balance/deploy/docker + +# Modify the configuration +vim config.yaml + +# Start the service +docker compose up -d +``` + +### Docker +```bash +# Pull the image +docker pull dulljz/s3-balance:latest +# Prepare configuration and data directories +mkdir -p ~/s3-balance/config ~/s3-balance/data +wget https://raw.githubusercontent.com/DullJZ/s3-balance/refs/heads/main/config/config.example.yaml -O ~/s3-balance/config/config.yaml +vim ~/s3-balance/config/config.yaml # Configure backend buckets, database, and S3 API options +# Run the service +docker run -d --name s3-balance -p 8080:8080 -v ~/s3-balance/config:/app/config -v ~/s3-balance/data:/app/data dulljz/s3-balance:latest +``` + +### Local Build and Run + +```bash +# Clone the repository +git clone https://github.com/DullJZ/s3-balance.git --depth=1 +cd s3-balance + +# Install dependencies +go mod tidy + +# Prepare configuration +cp config/config.example.yaml config/config.yaml +vim config/config.yaml # Configure backend buckets, database, and S3 API options + +# Run the service +go run cmd/s3-balance/main.go -config config/config.yaml +# Or build the binary: +go build -o s3-balance cmd/s3-balance/main.go +./s3-balance -config config/config.yaml +``` + +## Configuration Highlights + +- `server`: Listening address and timeout settings. +- `database`: GORM supports sqlite/mysql/postgres and stores object metadata, multipart sessions, etc. +- `buckets`: Lists real and virtual buckets. Entries with `virtual: true` are exposed externally, while real buckets are marked as `virtual: false`. Supports `path_style` and `max_size` settings. +- `balancer`: Strategy (`round-robin`|`least-space`|`weighted`), health check intervals, retry counts, and delays. +- `metrics`: Whether to enable Prometheus metrics and their path. +- `s3api`: Access/Secret Key, `proxy_mode` (true=proxy, false=redirect), `auth_required` (SigV4 validation), `virtual_host` (host-style routing). + +## API & Testing + +- Default listening at `http://localhost:8080`, supports `GET /health` for health checks and `GET /metrics` for metrics. +- Compatibility can be verified using AWS CLI, s3cmd, MinIO Client, or `python3 test_virtual_bucket_s3.py`. Modify the endpoint and credentials in the script before running. + +## Project Structure + +``` +cmd/s3-balance/ # Service entrypoint +internal/api/ # S3 routing, object/multipart handling +internal/bucket/ # Bucket client & health monitoring +internal/balancer/ # Strategy implementations & metrics +internal/middleware/ # Middleware for SigV4, virtual hosts, etc. +internal/storage/ # GORM models and services +pkg/presigner/ # Pre-signed URL utilities +config/ # Example configurations and deployment manifests +deploy/ # Docker/Kubernetes/Helm manifests +``` \ No newline at end of file