Replicating to Amazon S3

This guide will show you how to use Amazon S3 as a database replica path for Litestream. You will need an Amazon AWS account to complete this guide.

Setup#

Create an IAM user#

You will need to set up a user with programmatic access to work with Amazon S3. From the AWS Console, go to the IAM service. Next, click Users from the left-hand navigation and then click the Add User button.

Enter a name for your user and make sure to enable Programmatic Access. Then click the Next button.

On the permissions screen, click on “Attach existing policies directly”, then search for “S3” and choose AmazonS3FullAccess. You can also specify a more restrictive policy as described later in this guide.

Then click the Next button twice and then click the Create user button. This will create the user and display credentials for programmatic access. You will need to save the “Access key ID” and “Secret access key” for later use in this guide.

Create a bucket#

Once you have a user created, go to the S3 service in the AWS Console. Click the “Create bucket” button.

You’ll need to choose a globally unique bucket name and choose a region to store the bucket data.

Then click the “Create bucket” button at the bottom of the screen. Your bucket has now been created.

Usage#

Command line usage#

You can replicate to S3 from the command line by setting environment variables with the credentials you obtained after creating your IAM user:

export AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

Then you can specify your bucket as a replica URL on the command line. For example, you can replicate a database to your bucket with the following command.

Be sure to replace /path/to/db with the local file path of your database, replace BUCKETNAME with the name of your bucket, and replace PATHNAME with the path you want to store your replica within your bucket.

litestream replicate /path/to/db s3://BUCKETNAME/PATHNAME

You can later restore your database from S3 to a local my.db path with the following command.

litestream restore -o my.db s3://BUCKETNAME/PATHNAME

Configuration file usage#

Litestream is typically run as a background service which uses a configuration file. You can configure a replica for your database using the url format:

access-key-id: AKIAxxxxxxxxxxxxxxxx
secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

dbs:
  - path: /path/to/local/db
    replicas:
      - url: s3://BUCKETNAME/PATHNAME

Or you can expand your configuration into multiple fields:

dbs:
  - path: /path/to/local/db
    replicas:
      - type: s3
        bucket: BUCKETNAME
        path:   PATHNAME
        region: us-east-1   # optional, set to your region

You may also specify your AWS credentials on a per-replica basis:

dbs:
  - path: /path/to/local/db
    replicas:
      - url: s3://BUCKETNAME/PATHNAME
        access-key-id: AKIAxxxxxxxxxxxxxxxx
        secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

Restrictive IAM Policy#

While specifying AmazonS3FullAccess is an easy way to get up and running, you may want to specify a more restrictive policy in order to limit abuse if your credentials are compromised.

The following is the minimum policy to use with Litestream. Please replace the <BUCKET> with the name of your bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<BUCKET>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET>/*",
                "arn:aws:s3:::<BUCKET>"
            ]
        }
    ]
}

Thanks to Martin for contributing this policy.