VFS Extension

The Litestream VFS provides read-only access to a replicated SQLite database directly from object storage. It registers a VFS named litestream, fetches pages on-demand, and continuously polls the replica for new LTX files. The VFS is built separately from the main litestream binary and requires CGO. Pre-built binaries are available in GitHub releases.

Installation via package managers#

The VFS extension is distributed as litestream-vfs through PyPI, npm, and RubyGems. These packages bundle the pre-built shared library for supported platforms, eliminating the need for manual downloads or CGO builds.

Supported platforms:

• x86_64-linux (Linux Intel/AMD 64-bit)
• aarch64-linux (Linux ARM 64-bit)
• x86_64-darwin (macOS Intel)
• arm64-darwin (macOS Apple Silicon)

Python#

Install from PyPI:

pip install litestream-vfs

API:

• litestream_vfs.load(conn) — Load the extension into a SQLite connection
• litestream_vfs.loadable_path() — Return the path to the shared library

Example:

Set the replica URL in the shell before starting Python:

export LITESTREAM_REPLICA_URL=s3://mybucket/db

import sqlite3
import litestream_vfs

conn = sqlite3.connect(":memory:")
litestream_vfs.load(conn)

# Open a replica database
conn.execute("ATTACH DATABASE 'file:replica.db?vfs=litestream' AS replica")

Node.js#

Install from npm:

npm install litestream-vfs

The package uses optionalDependencies to install only the binary for your platform automatically.

API:

• getLoadablePath() — Return the path to the shared library

Example:

Set the replica URL in the shell before starting Node.js:

export LITESTREAM_REPLICA_URL=s3://mybucket/db

const Database = require('better-sqlite3');
const { getLoadablePath } = require('litestream-vfs');

const db = new Database(':memory:');
db.loadExtension(getLoadablePath());

// Open a replica database
db.exec("ATTACH DATABASE 'file:replica.db?vfs=litestream' AS replica");

Ruby#

Install from RubyGems:

gem install litestream-vfs

Platform-specific gems are published for each supported platform.

API:

• LitestreamVfs.load(db) — Load the extension into a SQLite connection
• LitestreamVfs.loadable_path — Return the path to the shared library

Example:

Set the replica URL in the shell before starting Ruby:

export LITESTREAM_REPLICA_URL=s3://mybucket/db

require 'sqlite3'
require 'litestream_vfs'

db = SQLite3::Database.new(':memory:')
LitestreamVfs.load(db)

# Open a replica database
db.execute("ATTACH DATABASE 'file:replica.db?vfs=litestream' AS replica")

Configuration#

Configure the replica location using environment variables as described in the Configuration section below. The LITESTREAM_REPLICA_URL environment variable is required and must be set in the process environment before your application starts — the extension fails to initialize without it. Setting it from inside the process (e.g. os.environ, process.env, or ENV) is not reliably visible to the extension’s embedded runtime, so export it in the shell or set it in your process manager instead.

Build requirements#

• Go 1.25+ with a working CGO toolchain (gcc/clang).
• Build with the VFS & extension tags using the repository Makefile (handles platform flags):

make vfs

• Manual Linux build:

CGO_ENABLED=1 go build -tags "vfs,SQLITE3VFS_LOADABLE_EXT" -buildmode=c-archive -o dist/litestream-vfs.a ./cmd/litestream-vfs
cp dist/litestream-vfs.h src/litestream-vfs.h
gcc -DSQLITE3VFS_LOADABLE_EXT -fPIC -shared -o dist/litestream-vfs.so src/litestream-vfs.c dist/litestream-vfs.a -lpthread -ldl -lm

• macOS requires the extra frameworks in the Makefile; prefer make vfs.
• Use .dylib (macOS) or .dll (Windows) if your platform expects it.
• SQLite entrypoint symbol: sqlite3_litestreamvfs_init.
• The shared library registers the litestream VFS when loaded into SQLite.
• Go applications can link directly by importing github.com/benbjohnson/litestream with -tags vfs and registering the VFS via sqlite3vfs.RegisterVFS.

Supported storage backends#

The loadable extension supports all Litestream replica backends via the LITESTREAM_REPLICA_URL environment variable:

• S3 — AWS S3 and S3-compatible storage (MinIO, R2, Tigris, etc.)
• GCS — Google Cloud Storage
• ABS — Azure Blob Storage
• SFTP — SSH File Transfer Protocol
• File — Local filesystem
• NATS — NATS JetStream
• WebDAV — WebDAV and WebDAVS servers
• Alibaba OSS — Alibaba Cloud Object Storage Service

Go applications can use any ReplicaClient when building with -tags vfs. SQLite clients must support loadable extensions.

VFS registration & usage#

• VFS name: litestream
• Open URIs with vfs=litestream, for example:

sqlite3
sqlite> .load ./dist/litestream-vfs sqlite3_litestreamvfs_init
sqlite> .open 'file:replica.db?vfs=litestream'

• The VFS forces connections to be read-only and rewrites the header so SQLite reports DELETE journal mode, matching SQLite expectations for external VFSes.
• Temporary files are stored in a private directory created by the VFS and cleaned up automatically.

Configuration (environment variables)#

Replica URL#

Set LITESTREAM_REPLICA_URL to specify the replica location using a URL format. This variable is required—the loadable extension fails to initialize without it.

For S3-compatible storage, append query parameters:

# MinIO example
LITESTREAM_REPLICA_URL="s3://mybucket/db?endpoint=minio.example.com"

# Cloudflare R2 example
LITESTREAM_REPLICA_URL="s3://mybucket/db?endpoint=<account>.r2.cloudflarestorage.com"

Other configuration#

• LITESTREAM_LOG_LEVEL — DEBUG or INFO (default).
• LITESTREAM_LOG_FILE — append log output to a file instead of stdout. v0.5.7
• LITESTREAM_S3_ENDPOINT — custom endpoint for S3-compatible storage (alternative to the endpoint query parameter in the replica URL).
• Standard cloud provider credentials (AWS, GCP, Azure) are honored by their respective SDKs. For example: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_PROFILE, GOOGLE_APPLICATION_CREDENTIALS, AZURE_STORAGE_ACCOUNT.

Runtime tuning#

VFS runtime tuning is set in code (Go) by adjusting VFS.PollInterval (default 1s) and VFS.CacheSize (default 10MB).

Write mode configuration#

Write mode enables the VFS to sync writes back to object storage instead of being read-only. See the VFS Write Mode Guide for usage details.

Write mode assumes a single writer. Multiple concurrent writers trigger conflict detection.

Hydration configuration#

Hydration restores the full database to a local file in the background while the VFS continues serving reads. See the VFS Hydration Guide for usage details.

Once hydration completes, all reads are served from the local file instead of remote storage.

v0.5.9 When LITESTREAM_HYDRATION_PATH is set to an explicit path, the hydration file persists across connection restarts. A companion .meta file (e.g. hydrated.db.meta) is written alongside the hydration file to track the current transaction ID (TXID). On the next open, if both files exist, hydration resumes from the saved TXID instead of performing a full restore. See the VFS Hydration Guide for details.

PRAGMAs & SQL functions#

The VFS extension registers custom PRAGMAs and SQL functions for observability and time travel queries.

PRAGMA litestream_txid#

Returns the current transaction ID as a 16-character hexadecimal string.

PRAGMA litestream_txid;
-- Returns: 0000000000000042

PRAGMA litestream_lag#

Returns the number of seconds since the last successful poll for new LTX files. Useful for monitoring replica freshness and alerting on stale replicas.

PRAGMA litestream_lag;
-- Returns: 2

A value of -1 indicates the VFS has not completed its initial poll.

PRAGMA litestream_time#

Gets or sets the point-in-time view for time travel queries.

Get current time:

PRAGMA litestream_time;
-- Returns: 2024-01-15T10:30:00.123456789Z (RFC3339Nano format)

Set to specific timestamp:

PRAGMA litestream_time = '2024-01-15T10:30:00Z';

Set to relative time:

PRAGMA litestream_time = '5 minutes ago';
PRAGMA litestream_time = 'yesterday';
PRAGMA litestream_time = '2 hours ago';

Reset to latest:

PRAGMA litestream_time = 'latest';

Note: Time travel requires l0-retention to be configured on the primary so historical LTX files remain available.

SQL function equivalents#

The same functionality is available via SQL functions:

Example:

SELECT litestream_txid(), litestream_lag(), litestream_time();
SELECT litestream_set_time('10 minutes ago');

PRAGMA litestream_hydration_progress#

v0.5.7 Returns hydration progress as a percentage (0–100, one decimal place). Returns 0 when hydration is not enabled. Read-only.

PRAGMA litestream_hydration_progress;
-- Returns: 45.0

PRAGMA litestream_hydration_file#

v0.5.7 Returns the local file path of the hydrated database. Read-only.

PRAGMA litestream_hydration_file;
-- Returns: /var/lib/litestream/hydrated.db

PRAGMA litestream_write_enabled#

v0.5.9 Gets or sets write mode at runtime.

Get current state:

PRAGMA litestream_write_enabled;
-- Returns: 0 (disabled) or 1 (enabled)

Enable or disable:

PRAGMA litestream_write_enabled = 1;   -- also accepts true, on
PRAGMA litestream_write_enabled = 0;   -- also accepts false, off

Performance characteristics#

• On-demand page fetch with LRU caching sized by CacheSize (default 10MB).
• Polling cadence controlled by PollInterval (default 1s) to discover new LTX files.
• Page fetch retries: up to 6 attempts with backoff for transient network/read errors.
• Supports SQLite page sizes from 512 to 65536 bytes (auto-detected from LTX headers).
• Two-index isolation: incoming LTX pages are staged while readers hold locks, then swapped in to keep long-running reads consistent. See How it works: VFS for memory considerations with long-held transactions.

Limitations & constraints#

• Default read-only: Write attempts fail with attempt to write a readonly database unless write mode is enabled with LITESTREAM_WRITE_ENABLED=true.
• Write mode constraints: When enabled, write mode assumes a single writer. Multiple concurrent writers trigger conflict detection (not prevention). Applications must handle conflicts. See conflict handling.
• CGO-only: Requires a CGO-enabled SQLite driver (github.com/mattn/go-sqlite3) and -tags vfs.
• Contiguous LTX coverage: Missing L0 files (e.g., aggressive retention) will error until new files appear.
• Initial snapshot required: The VFS waits for available backup files before serving reads.
• Network latency: First-page reads incur network latency; place consumers close to the replica endpoint.
• Hydration disk space: When hydration is enabled, requires local disk space for the full database.

Architecture overview#

The VFS derives a restore plan from the replica, detects the database page size, builds an in-memory page index from LTX files, and serves pages directly from remote storage. A polling loop keeps the index fresh by reading new L0/L1 files, and cached pages are invalidated when newer frames arrive. See How it works: VFS for a deeper explanation.