Replicating to S3-Compatible Services

This guide covers how to use Litestream with S3-compatible storage providers beyond Amazon S3. Many cloud providers and self-hosted solutions implement the S3 API, allowing you to use Litestream’s S3 replica type with minimal configuration changes.

Overview#

S3-compatible services implement Amazon’s S3 API, enabling tools built for AWS S3 to work with alternative storage providers. Litestream supports these services through the endpoint configuration option, which redirects API requests to your chosen provider.

Key benefits of S3-compatible storage:

• Cost savings: Many providers offer lower prices than AWS S3
• Data sovereignty: Choose where your data is stored
• No vendor lock-in: Switch providers without changing your workflow
• Self-hosted options: Run your own object storage with MinIO

Configuration Methods#

You can configure S3-compatible replicas using either URL parameters or explicit configuration fields.

URL Format with Endpoint Parameter#

The simplest approach uses the endpoint query parameter in the replica URL:

dbs:
  - path: /path/to/db
    replica:
      url: s3://mybucket/db?endpoint=s3.us-west-1.wasabisys.com
      access-key-id: ${ACCESS_KEY}
      secret-access-key: ${SECRET_KEY}

Explicit Configuration Fields#

For more control, specify each field separately:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.us-west-1.wasabisys.com
      region: us-west-1
      access-key-id: ${ACCESS_KEY}
      secret-access-key: ${SECRET_KEY}

Default HTTPS Behavior#

Litestream uses HTTPS by default when connecting to endpoints. If your endpoint URL doesn’t include a scheme, HTTPS is assumed:

# These are equivalent:
endpoint: s3.provider.com
endpoint: https://s3.provider.com

For local development with HTTP (not recommended for production):

endpoint: http://localhost:9000

Provider-Specific Configuration#

Each S3-compatible provider has unique requirements. The sections below provide configuration examples and notes for popular providers.

MinIO#

MinIO is a high-performance, self-hosted object storage server. It’s ideal for development, testing, or running your own S3-compatible infrastructure.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: https://minio.example.com:9000
      region: us-east-1
      access-key-id: ${MINIO_ACCESS_KEY}
      secret-access-key: ${MINIO_SECRET_KEY}

Notes:

• Create access keys in the MinIO Console before configuring Litestream
• The region field is required but MinIO ignores the value; use us-east-1
• For self-signed certificates, add skip-verify: true (not for production)
• Default credentials for new installations are minioadmin/minioadmin

See the Configuration Reference for detailed MinIO setup instructions.

Backblaze B2#

Backblaze B2 offers affordable cloud storage with S3-compatible API access. B2 charges only for storage and downloads, with free uploads.

Configuration:

access-key-id: ${B2_KEY_ID}
secret-access-key: ${B2_APPLICATION_KEY}

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: my-bucket
      path: db
      endpoint: s3.us-west-004.backblazeb2.com
      region: us-west-004
      force-path-style: true

Notes:

• Endpoint format: s3.<region>.backblazeb2.com
• The region must match your bucket’s region exactly
• Always set force-path-style: true for B2
• Create application keys with access limited to specific buckets for security
• B2 requires minimum 5 MB part sizes for multipart uploads (Litestream default)

See the Backblaze B2 Guide for detailed setup instructions.

Wasabi#

Wasabi provides S3-compatible storage with no egress fees and predictable pricing.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.us-east-1.wasabisys.com
      region: us-east-1
      access-key-id: ${WASABI_ACCESS_KEY}
      secret-access-key: ${WASABI_SECRET_KEY}

Regional Endpoints:

Notes:

• Use the endpoint matching your bucket’s region
• Wasabi is 100% S3 API compatible
• No egress fees make it cost-effective for frequent restores

Cloudflare R2#

Cloudflare R2 provides S3-compatible object storage with zero egress fees and global distribution.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: ${CF_ACCOUNT_ID}.r2.cloudflarestorage.com
      region: auto
      access-key-id: ${R2_ACCESS_KEY}
      secret-access-key: ${R2_SECRET_KEY}

Notes:

• Find your Account ID in the Cloudflare dashboard under R2
• Region should be set to auto (or us-east-1 for compatibility)
• Create API tokens in the R2 section with “Object Read & Write” permissions
• R2 supports both path-style and virtual-hosted style addressing
• Jurisdiction-specific endpoints available for EU data residency: ${ACCOUNT_ID}.eu.r2.cloudflarestorage.com

DigitalOcean Spaces#

DigitalOcean Spaces provides simple, scalable object storage with a CDN included.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: my-space
      path: db
      endpoint: nyc3.digitaloceanspaces.com
      region: nyc3
      access-key-id: ${DO_SPACES_KEY}
      secret-access-key: ${DO_SPACES_SECRET}

Regional Endpoints:

Notes:

• Generate Spaces access keys in the API section of the DigitalOcean console
• Connection resets may appear in logs; Litestream handles these automatically

See the DigitalOcean Spaces Guide for detailed setup.

Linode Object Storage#

Linode Object Storage provides S3-compatible storage across multiple regions.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: us-east-1.linodeobjects.com
      region: us-east-1
      access-key-id: ${LINODE_ACCESS_KEY}
      secret-access-key: ${LINODE_SECRET_KEY}

Regional Endpoints:

See the Linode Guide for detailed setup instructions.

OCI Object Storage#

Oracle Cloud Infrastructure Object Storage provides S3-compatible API access to OCI buckets.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: ${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com
      region: us-east-1
      access-key-id: ${OCI_ACCESS_KEY}
      secret-access-key: ${OCI_SECRET_KEY}

Notes:

• Find your namespace in the OCI Console under Tenancy Details
• Create Customer Secret Keys for S3 compatibility (not standard API keys)
• Set region to us-east-1 or leave blank if your client doesn’t support OCI region identifiers
• Endpoint format includes your namespace: <namespace>.compat.objectstorage.<region>.oraclecloud.com

Example Endpoints:

• US Phoenix: <namespace>.compat.objectstorage.us-phoenix-1.oraclecloud.com
• US Ashburn: <namespace>.compat.objectstorage.us-ashburn-1.oraclecloud.com
• EU Frankfurt: <namespace>.compat.objectstorage.eu-frankfurt-1.oraclecloud.com

Vultr Object Storage#

Vultr Object Storage provides S3-compatible storage with simple, predictable pricing (no API request charges).

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: ewr1.vultrobjects.com
      region: ewr1
      access-key-id: ${VULTR_ACCESS_KEY}
      secret-access-key: ${VULTR_SECRET_KEY}

Regional Endpoints:

Notes:

• No charges for API requests, only storage and egress
• HTTPS required for all connections
• Virtual-hosted style addressing preferred

Scaleway Object Storage#

Scaleway Object Storage provides S3-compatible storage in European data centers.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.fr-par.scw.cloud
      region: fr-par
      access-key-id: ${SCW_ACCESS_KEY}
      secret-access-key: ${SCW_SECRET_KEY}

Regional Endpoints:

See the Scaleway Guide for detailed setup instructions.

Filebase#

Filebase provides S3-compatible access to decentralized storage networks including IPFS and Sia.

Configuration:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.filebase.com
      region: us-east-1
      access-key-id: ${FILEBASE_ACCESS_KEY}
      secret-access-key: ${FILEBASE_SECRET_KEY}

Notes:

• Single global endpoint: s3.filebase.com
• Region must be set to us-east-1
• Only supports AWS Signature Version 4
• Rate limit of 100 requests per second
• Provides decentralized storage with IPFS pinning

Provider Compatibility Matrix#

The following table summarizes configuration requirements for each provider:

Advanced Configuration Options#

These options help with compatibility across different S3 implementations:

force-path-style#

Some providers require path-style URLs instead of virtual-hosted style. This is automatically enabled when you set an endpoint, but can be explicitly controlled:

replica:
  type: s3
  bucket: mybucket
  endpoint: s3.provider.com
  force-path-style: true

skip-verify#

Disable TLS certificate verification for self-signed certificates (development only):

replica:
  type: s3
  bucket: mybucket
  endpoint: https://minio.local:9000
  skip-verify: true

Warning: Never use skip-verify: true in production environments.

Performance Tuning#

For large databases or high-throughput scenarios, tune multipart upload settings:

replica:
  type: s3
  bucket: mybucket
  endpoint: s3.provider.com
  part-size: 10MB
  concurrency: 10

See the Configuration Reference for all available S3 replica options.

Troubleshooting#

Authentication Errors#

Symptom: AccessDenied or SignatureDoesNotMatch errors

Solutions:

1. Verify access key and secret key are correct
2. Check that credentials have appropriate bucket permissions
3. Ensure your system clock is synchronized (signatures are time-sensitive)
4. Confirm the region matches your bucket’s actual region

Connection Errors#

Symptom: connection refused or timeout errors

Solutions:

1. Verify the endpoint URL is correct
2. Check that HTTPS is supported (or use http:// for local development)
3. Ensure firewall rules allow outbound connections on port 443
4. For self-hosted services, confirm the service is running

Invalid Endpoint#

Symptom: InvalidEndpoint or hostname resolution errors

Solutions:

1. Remove the scheme from the endpoint if specifying separately
2. Include the port number if using a non-standard port
3. Verify DNS resolution for the endpoint hostname

Region Mismatch#

Symptom: PermanentRedirect or AuthorizationHeaderMalformed errors

Solutions:

1. Ensure the region field matches the bucket’s actual region
2. For providers that ignore region, try us-east-1
3. Check provider documentation for region identifiers

Connection Resets#

Symptom: connection reset by peer errors in logs

Solutions:

Some providers (notably DigitalOcean Spaces) may reset connections occasionally. Litestream handles these automatically through retries. If you see these errors but replication continues, no action is needed.

Environment Variables#

Use environment variables to keep credentials out of configuration files:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: ${BUCKET_NAME}
      path: db
      endpoint: ${S3_ENDPOINT}
      region: ${S3_REGION}
      access-key-id: ${S3_ACCESS_KEY}
      secret-access-key: ${S3_SECRET_KEY}

Standard AWS environment variables are also supported:

• AWS_ACCESS_KEY_ID or LITESTREAM_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY or LITESTREAM_SECRET_ACCESS_KEY

Litestream-specific variables take precedence over AWS variables.