Replicating to Amazon S3

This guide will show you how to use Amazon S3 as a database replica path for Litestream. You will need an Amazon AWS account to complete this guide.

Setup

Create an IAM user

You will need to set up a user with programmatic access to work with Amazon S3. From the AWS Console, go to the IAM service. Next, click Users from the left-hand navigation and then click the Add User button.

Enter a name for your user and make sure to enable Programmatic Access. Then click the Next button.

Screenshot of creating an IAM user

On the permissions screen, click on “Attach existing policies directly”, then search for “S3” and choose AmazonS3FullAccess. You can also specify a more restrictive policy as described later in this guide.

Screenshot of attaching policy to IAM user

Then click the Next button twice and then click the Create user button. This will create the user and display credentials for programmatic access. You will need to to save the “Access key ID” and “Secret access key” for later use in this guide.

Screenshot of AWS credentials for created user

Create a bucket

Once you have a user created, go to the S3 service in the AWS Console. Click the “Create bucket” button.

You’ll need to choose a globally unique bucket name and choose a region to store the bucket data.

Screenshot of AWS S3 create bucket UI

Then click the “Create bucket” button at the bottom of the screen. Your bucket has now been created.

Usage

Command line usage

You can replicate to S3 from the command line by setting environment variables with the credentials you obtained after creating your IAM user:

export AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

Then you can specify your bucket as a replica URL on the command line. For example, you can replicate a database to your bucket with the following command.

Be sure to replace /path/to/db with the local file path of your database, replace BUCKETNAME with the name of your bucket, and replace PATHNAME with the path you want to store your replica within your bucket.

litestream replicate /path/to/db s3://BUCKETNAME/PATHNAME

You can later restore your database from S3 to a local my.db path with the following command.

litestream restore -o my.db s3://BUCKETNAME/PATHNAME

Configuration file usage

Litestream is typically run as a background service which uses a configuration file. You can configure a replica for your database using the url format:

access-key-id: AKIAxxxxxxxxxxxxxxxx
secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

dbs:
  - path: /path/to/local/db
    replicas:
      - url: s3://BUCKETNAME/PATHNAME

Or you can expand your configuration into multiple fields:

dbs:
  - path: /path/to/local/db
    replicas:
      - type: s3
        bucket: BUCKETNAME
        path:   PATHNAME
        region: us-east-1   # optional, set to your region

You may also specify your AWS credentials on a per-replica basis:

dbs:
  - path: /path/to/local/db
    replicas:
      - url: s3://BUCKETNAME/PATHNAME
        access-key-id: AKIAxxxxxxxxxxxxxxxx
        secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

Restrictive IAM Policy

While specifying AmazonS3FullAccess is an easy way to get up and running, you may want to specify a more restrictive policy in order to limit abuse if your credentials are compromised.

The following is the minimum policy to use with Litestream. Please replace the <BUCKET> with the name of your bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::<BUCKET>"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET>/*",
                "arn:aws:s3:::<BUCKET>"
            ]
        }
    ]
}

Thanks to Martin for contributing this policy.