S3 Advanced Configuration

This guide covers advanced S3 configuration options, including multipart upload tuning and storage classes. These settings control how Litestream uploads data to S3 and S3-compatible storage services.

v0.5.3 Part size and concurrency configuration options were added in Litestream v0.5.3.

Overview

When Litestream uploads files to S3, it uses multipart uploads for efficient data transfer. Two key settings control this behavior:

Part Size: The size of each chunk uploaded to S3
Concurrency: The number of parts uploaded simultaneously

Understanding these settings helps you optimize performance for your specific network conditions, storage provider, and resource constraints.

Configuration Options

Part Size

The part-size field controls the size of each part in multipart uploads. Litestream accepts human-readable size strings.

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      part-size: 10MB

Supported formats:

Bytes: 5242880 (numeric value in bytes)
Kilobytes: 5120KB or 5120K
Megabytes: 5MB or 5M
Gigabytes: 1GB or 1G
Decimal values: 1.5MB

Defaults and limits:

Setting	Value
Default	5 MiB (AWS SDK default)
Minimum	5 MiB (S3 requirement)
Maximum	5 GiB

The final part of an upload may be smaller than the minimum size.

Concurrency

The concurrency field controls how many parts are uploaded in parallel.

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      concurrency: 10

Defaults:

Setting	Value
Default	5 concurrent uploads

Provider Requirements

Different S3-compatible providers have specific requirements for multipart uploads.

AWS S3

AWS S3 uses the standard multipart upload limits:

Minimum part size: 5 MiB
Maximum part size: 5 GiB
Maximum parts per upload: 10,000

The default settings work well for most AWS S3 use cases.

Backblaze B2

Backblaze B2’s S3-compatible API requires a minimum part size of 5 MB. The defaults work correctly, but if you customize the part size, ensure it meets this minimum.

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.us-west-000.backblazeb2.com
      force-path-style: true
      part-size: 10MB  # Must be >= 5MB for B2

Backblaze recommends 100 MB part sizes for optimal performance with large files.

MinIO

MinIO follows standard S3 multipart upload limits. The default settings work without modification.

Other S3-Compatible Services

Check your provider’s documentation for specific multipart upload requirements. Most services follow the standard S3 limits (5 MiB minimum part size).

Performance Tuning

When to Increase Part Size

Larger part sizes may improve performance when:

You have high bandwidth and low latency connections
You’re uploading large database snapshots
You want to reduce the total number of API requests (cost savings)

part-size: 50MB

When to Decrease Part Size

Smaller part sizes may be beneficial when:

You have unreliable network connections (smaller retries on failure)
Memory is constrained
You want faster feedback on upload progress

Note: You cannot go below 5 MiB due to S3’s minimum requirement.

When to Increase Concurrency

Higher concurrency improves throughput when:

You have high bandwidth connections
Network latency is high (parallel uploads hide latency)
You’re not CPU or memory constrained

concurrency: 10

When to Decrease Concurrency

Lower concurrency may be appropriate when:

Memory is limited (each concurrent part uses memory)
CPU resources are constrained
You want to limit network bandwidth usage

concurrency: 2

Configuration Examples

High-Performance Configuration

For servers with fast network connections and ample resources:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      part-size: 50MB
      concurrency: 10

Resource-Constrained Configuration

For embedded devices or containers with limited memory:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      part-size: 5MB
      concurrency: 2

Backblaze B2 Optimized

Following Backblaze’s recommendations for optimal performance:

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      endpoint: s3.us-west-000.backblazeb2.com
      force-path-style: true
      part-size: 100MB
      concurrency: 5

Cost Considerations

Part size affects the number of API requests made to your storage provider:

Larger parts = fewer PUT requests = lower request costs
Smaller parts = more PUT requests = higher request costs

For high-volume databases with frequent syncs, consider larger part sizes to reduce the total number of API calls.

Storage Classes

v0.5.14 The storage-class field sets the S3 storage class for objects that Litestream uploads. When it is not set, objects use the provider’s default storage class (STANDARD on AWS S3).

dbs:
  - path: /path/to/db
    replica:
      type: s3
      bucket: mybucket
      path: db
      storage-class: STANDARD_IA

The storage class can also be set as a URL query parameter:

dbs:
  - path: /path/to/db
    replica:
      url: s3://mybucket/db?storage-class=STANDARD_IA

The value is passed to S3 as-is, so any storage class supported by your provider works. Common AWS S3 choices:

Storage Class	Best For
`STANDARD`	Default; frequent access, no retrieval fees
`STANDARD_IA`	Long-retention replicas that are rarely restored
`ONEZONE_IA`	Same as `STANDARD_IA` but single-AZ, lower cost
`GLACIER_IR`	Archival storage with instant retrieval
`INTELLIGENT_TIERING`	Automatic tiering based on access patterns

Retention Caveats

Infrequent-access and archival classes charge for a minimum storage duration (30 days for STANDARD_IA and ONEZONE_IA, 90 days for GLACIER_IR) and per-GB retrieval fees. Litestream continuously uploads new LTX files and deletes old ones during retention enforcement, so:

Objects deleted before the minimum storage duration still incur the full minimum-duration charge. Use these classes only when your retention period is longer than the class’s minimum storage duration.
Restores read every LTX file needed to rebuild the database, so retrieval fees apply to the full restore size.

Avoid asynchronous archive classes such as GLACIER (Flexible Retrieval) and DEEP_ARCHIVE. Objects in these classes must be restored to S3 before they can be read, so litestream restore fails against them.

S3-compatible providers support different storage class values — check your provider’s documentation before setting this option.

Troubleshooting

Upload Failures with Small Part Size

If you see errors about part size being too small, ensure your part-size is at least 5 MiB:

part-size: 5MB  # Minimum allowed

Memory Issues

If Litestream uses too much memory during uploads, reduce concurrency:

concurrency: 2

Each concurrent upload buffers a part in memory before sending.

Slow Uploads

For slow uploads, try:

Increasing concurrency to better utilize bandwidth:
```
concurrency: 10
```
Adjusting part size based on your network conditions

Provider-Specific Errors

If you receive errors from your storage provider about invalid part sizes, verify you meet their minimum requirements. Backblaze B2 and some other providers have a strict 5 MB minimum.

Distributed Leasing

v0.5.8 Litestream supports distributed leasing for safe multi-instance deployments using S3 conditional writes. This ensures only one Litestream instance actively replicates a database at a time, enabling high-availability patterns where a standby instance can take over if the primary fails.

How It Works

Litestream uses a lock.json file stored alongside your replica data. The leasing algorithm uses S3’s If-Match and If-None-Match conditional headers for atomic leader election:

Acquire: Read existing lock.json, check if the lease has expired, then conditionally write a new lease using ETag matching
Renew: The active leader periodically refreshes the lease before expiry
Release: On graceful shutdown, the leader releases the lease

If another instance attempts to acquire the lease while it is held, S3 returns a 412 Precondition Failed response and the instance waits.

Lock File Format

{"generation": 1, "expires_at": "2025-02-21T15:04:33.206179Z", "owner": "hostname:pid"}

generation — Monotonically increasing counter for lease generations
expires_at — RFC 3339 timestamp when the lease expires
owner — Identifier of the instance holding the lease

Failover Behavior

When the active instance stops (crash, shutdown, or network partition), the lease expires and a standby instance acquires it automatically. The failover time depends on the lease duration.

Requirements

Distributed leasing requires an S3 provider that supports conditional writes. AWS S3 supports this natively. Check your provider’s documentation for If-Match/If-None-Match header support.

This feature is currently available through the Go library API. See the Go library guide for programmatic usage.

S3 Advanced Configuration

Overview#

Configuration Options#

Part Size#

Concurrency#

Provider Requirements#

AWS S3#

Backblaze B2#

MinIO#

Other S3-Compatible Services#

Performance Tuning#

When to Increase Part Size#

When to Decrease Part Size#

When to Increase Concurrency#

When to Decrease Concurrency#

Configuration Examples#

High-Performance Configuration#

Resource-Constrained Configuration#

Backblaze B2 Optimized#

Cost Considerations#

Storage Classes#

Retention Caveats#

Troubleshooting#

Upload Failures with Small Part Size#

Memory Issues#

Slow Uploads#

Provider-Specific Errors#

Distributed Leasing#

How It Works#

Lock File Format#

Failover Behavior#

Requirements#

See Also#