Importing SQLite Database Snapshots

Overview

A new gateway builds its index of the Arweave network block by block. That can take days or weeks, depending on your hardware and on how much data you index. You can skip most of it in one of two ways:

• Import a SQLite snapshot. The 2025-04-23 snapshot holds transactions and the data items uploaded through ArDrive products, including Turbo, up to block 1645229.
• Bootstrap from L1 bands (Release 85 or later). Your gateway follows the chain tip and, while it runs, fills in the Arweave base layer beneath it (blocks, transactions, tags and owners) from signed bands that another gateway publishes. Data items are not included.

Import a SQLite Snapshot

SQLite snapshots are large and hard to update in steps, so ar.io distributes them over BitTorrent.

# Ubuntu/Debian
sudo apt-get install transmission-cli

# CentOS/RHEL
sudo yum install transmission-cli

# macOS
brew install transmission-cli

transmission-cli "magnet:?xt=urn:btih:62ca6e05248e6df59fac9e38252e9c71951294ed&dn=2025-04-23-sqlite.tar.gz&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=https%3A%2F%2Ftracker.bt4g.com%3A443%2Fannounce"

tar -xzf 2025-04-23-sqlite.tar.gz
ls -la 2025-04-23-sqlite/

docker compose down

mkdir sqlite-backup
mv data/sqlite/* sqlite-backup/
mv 2025-04-23-sqlite/* data/sqlite/

docker compose up -d

docker compose logs -f core

Bootstrap from L1 Bands

An L1 band (parquet-l1) holds the Arweave base layer for a range of block heights, as Parquet files: blocks, transactions, tags and owners. Gateways that publish them sign every file, and your gateway checks each one before it installs the band. Your gateway then fills its core.db from the installed bands, instead of indexing the chain from block 0.

L1 bands carry no data items. Your gateway still unbundles the bundles your filters select, and it does that for imported history only if you also set BACKFILL_BUNDLE_RECORDS.

There are two ways to do it:

• Fill the history while the gateway runs, as below. This is the usual way. It works on a new gateway, on one that began mid-chain, and on one with a hole in what it holds.
• Import with the gateway stopped. Faster, but only for a new core.db that the gateway has never run on.

With L1_BAND_IMPORT_ENABLED=true, core runs ar-io-node index-l1-import --fill every 10 minutes (L1_BAND_IMPORT_INTERVAL_SECONDS, counted from the end of the last run), as a child process, so a crash in the import can't take the gateway down. A fill writes only the heights core.db lacks beneath what the block importer writes itself, so nothing your gateway indexed is rewritten. It writes in short transactions with a pause after each, so the gateway keeps indexing and serving the whole time.

Verified Before It Is Written

From Release 86, before a band's first row reaches core.db, the import checks it, with the gateway running or stopped:

1. The band, whole, with the same code as index-l1-verify --bands-dir: its name is its identity, its files and columns are the layout's, its tables digest to what band.json says, and every block, transaction, link, tag and wallet row passes.
2. Its edges, against core.db. The first height written must build on the block core.db holds below it, and the block core.db holds above the last height written must build on the band.
3. A binding to the chain. The block above, when core.db holds one the gateway synced from its own nodes, or one an earlier import wrote bound to the chain without accepting a gap. From the 2.0 fork, the band's hash_list_merkle recurrence then binds every block hash, weave size and tx_root beneath it. Otherwise, an anchor fetched from the sources (--anchor-from), as index-l1-verify does. An l1-4 band below the 2.0 fork that ends at a pinned height is anchored there by its witnessed folds, with no header fetched.
4. A sample. What the binding does not reach, such as transaction ids, their order and every transaction field, is checked by the same keyed sample of headers and signatures as index-l1-verify, from the same sources. Without a sample the import is refused unless you accept that (--allow-unsampled). A sample that could not be read is an outage, not evidence: the band is refused as unreadable and a later run tries again.
5. What nothing binds is refused unless accepted. An l1-4 band's witness files bind its blocks below the 2.0 fork and its blocks holding a format-1 transaction with data, so a whole chain of l1-4 bands imports with no --allow-* flag, and the importer writes the witnessed data roots and tx_roots over whatever core.db kept. What is left is blocks whose transactions or owners' keys the band lacks (--allow-incomplete). A band without witnesses (l1-3) has those first two kinds unbound, and is refused below height 2,000,000 unless they are accepted (--allow-unbound, --allow-unverified-format1).

A refused band removes nothing that was there before. While the gateway runs, ranges are filled from the top down, so each is bound by the block the gateway holds above it: its own, or one the range above it just wrote and verified. The run stops at the first range refused.

The gateway's own fill takes these settings from .env:

A refused fill is not re-verified every interval: the worker waits, twice as long each time (up to a day), until the bands directory changes. One refused only because its sources could not be read is tried again after an interval, twice as long each time it repeats (up to two hours), and at once when the bands directory changes.

On a new gateway, set this up before the first start and leave START_HEIGHT unset: run the setup script below without --restart (on a gateway that isn't running, --restart would start core and the sidecar alone), then start the gateway as usual (the Index Sharing sidecar starts with it). With an empty core.db, the block importer then starts at the chain tip instead of block 0, and the history beneath it arrives as the bands do.

Prerequisites

• A gateway on Release 85 or later, and Release 87 to take l1-4 bands
• Room on disk: about 15 GB for the bands, and about 120 GB for a core.db that covers the whole chain. The setup script sets the Index Sharing disk budget to 100 GiB if it is unset, and warns if yours is lower

./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --l1 --torrent --dry-run
./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --l1 --torrent --restart

./tools/index-swarm-status

Uncovered heights are normal for a while: bands still downloading, and, on a new gateway, the heights between the newest band and where it began, which the publisher's next tip band holds, usually within a day. A run with nothing to fill writes nothing. Only one import or fill runs against a core.db at a time.

Import L1 Bands with the Gateway Stopped

The fastest way to load the whole chain into an empty database, at the cost of the gateway being down while it runs. It is only for a new core.db that the gateway has not run on yet. The import refuses a core.db that another process is writing to, and one that holds unstable blocks (the new_* tables). A gateway that has run always holds some, because it moves blocks to the stable tables only once they are 18 deep. For a gateway that already runs, fill while it runs instead.

Prerequisites

• A gateway on Release 85 or later, and a migrated core.db (yarn db:migrate up; the core service also runs it when it starts). The import refuses one without its ledger table
• Room on disk: about 15 GB for the bands, and about 120 GB for a core.db that covers the whole chain. The import refuses to start without the space it needs
• Time: the whole chain takes about 15 hours on 12 cores. The import logs its progress once a minute

INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":["root-tx-index","parquet-l1"]}]'

docker compose stop core

./tools/ar-io-node index-l1-import \
  --bands-dir data/indexes/installed/parquet-l1 \
  --core-db data/sqlite/core.db \
  --cache-mib 8192 \
  --anchor-from https://<gateway one>,https://<gateway two>

docker compose up -d --no-deps core

To check a set of bands, or your gateway's own core.db, against the chain, use ./tools/ar-io-node index-l1-verify (see The ar-io-node CLI). Every option, and the JSON each command prints, is in the ar-io-node CLI reference.