Index Sharing

To serve a data item, your gateway first has to find which Arweave transaction holds it, and where inside that transaction it sits. CDB64 indexes answer that locally. Index Sharing keeps those indexes fresh by letting gateways share them: one gateway publishes its indexes, others subscribe, and every file is signed and checked before it is used. Bands move over HTTP, and, with the optional torrent engine, peer to peer over BitTorrent.

For how it works and why it can be trusted, see Index Sharing in Learn.

Index Sharing runs in an optional sidecar container, index-swarm, beside your gateway; its scripts, compose profiles and settings all use that name. Two scripts in the gateway repository do the work, run from the gateway's directory (where .env and docker-compose.yaml are). They need only Docker, since they run in the core image:

• ./tools/index-swarm-setup edits .env and, with --restart, restarts what the changes need.
• ./tools/index-swarm-status checks that everything works and says how to fix what doesn't.

HTTP or BitTorrent?

Decide this first; you can change it later by running the setup script again.

BitTorrent is the better default: your first pull is faster and free, and each gateway that seeds makes the next one's faster. It works behind NAT or a closed port too, because your gateway connects out to the publisher and to other peers; an open port adds the peers that can only be reached, not reach you. Choose HTTP only if you don't want to upload. Network and Disk has the details.

What runs on your gateway once it is set up. The torrent engine, dashed, is there only with --torrent; the tracker only answers when you publish:

Subscribe to a Publisher

Subscribing is what most operators want: your gateway downloads a publisher's index bands, verifies them, and answers lookups from them.

Prerequisites

• A running ar.io gateway on Release 84 or later, with Docker, and a checkout of the gateway repository at that release (the scripts live in tools/)
• 50 GiB of free disk on the filesystem that holds data/indexes, the budget the setup script sets. The bands turbo-gateway.com publishes take about 21 GB once installed; the rest is room for a band to be on disk twice while it arrives and for a new band beside the one it replaces. This is on top of what your gateway already uses, and it doesn't change the gateway's minimum requirements. See Network and Disk
• For BitTorrent: port 6881, TCP and UDP, open to the internet is recommended, not required

34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5

./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --dry-run
./tools/index-swarm-setup --subscribe 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5 --torrent --restart

./tools/index-swarm-status

Bands arrive newest heights first, because most lookups are for recent data. A first pull of a full index takes minutes to hours. After that, new bands from the publisher install, and replace the ones they supersede, by themselves; the gateway loads each within 30 seconds, without a restart.

What to Expect

The status script's CDB64 lookups ... found X of Y line shows how many root transaction lookups the installed bands answered. Don't expect every lookup to hit. A publisher's index covers the bundles it indexed, densest for recent data, and the tip band is rebuilt periodically, so the newest items aren't in it yet. A miss costs nothing: your gateway falls back to its usual lookup order. Following more than one publisher widens coverage.

Publish Your Indexes

Any registered gateway can publish. Other gateways then find it in the registry and subscribe by its wallet.

Publishing is for gateways that already build root transaction indexes. The sidecar publishes the bands you put in place; it doesn't build them, and there is no automatic export yet. The turbo-gateway.com gateway, for example, builds its bands from its own database and rebuilds the tip band daily.

Prerequisites

• A gateway registered in the gateway registry, reachable at its registered URL, on Release 84 or later. The routes that serve bands live in the gateway, not the sidecar
• Its registered observer key, which signs what it publishes
• Index bands to offer: partitioned CDB64 indexes
• For BitTorrent: ports 6881 (TCP and UDP) and 6969 (TCP) open to the internet, and this node's public IP

data/indexes/published/root-tx-index/<band>/manifest.json

./tools/export-sqlite-to-cdb64 --partitioned \
  --output-dir data/indexes/published/root-tx-index/band-tip.tmp

m=data/indexes/published/root-tx-index/band-tip.tmp/manifest.json
jq '.metadata = ((.metadata // {}) + {heightRange: [1950000, null]})' "$m" > "$m.new" && mv "$m.new" "$m"
mv data/indexes/published/root-tx-index/band-tip.tmp data/indexes/published/root-tx-index/band-tip

AR_IO_WALLET=<your gateway wallet>

OBSERVER_PRIVATE_KEY=<base58 observer private key>

# Host path of the observer keypair file; only this file is mounted
INDEX_SWARM_OBSERVER_KEYPAIR_FILE=/path/to/observer-keypair.json

./tools/index-swarm-setup --publish --torrent --public-host <this node's public IP> --restart

./tools/index-swarm-status
curl -s https://<your-gateway>/ar-io/indexes | jq '{sequence, publisher, bands: [.indexes[].bands[].id]}'
curl -s https://<your-gateway>/ar-io/info | jq .indexes

A gateway can do both: pass --subscribe and --publish together.

To replace a band, publish the new one under a fresh id with "supersedes": "<old band id>" in its manifest's metadata. The old band is withdrawn and removed after a grace period. Subscribers keep serving the old band until the new one has installed, so they never lose a lookup.

If you run more than one node behind a load balancer, read Publishing from a fleet before anyone subscribes.

The Setup and Status Scripts

index-swarm-setup

The script edits .env and nothing else, unless given --restart.

It is idempotent: a second run changes only what is missing, so it is also how to add a publisher or turn on torrents later. It never replaces a value it cannot parse or a password it did not write; it stops and says what to fix. Before writing, it copies .env to .env.bak-index-swarm-<time>, readable only by its owner, since it holds secrets. It warns when an explicit ROOT_TX_LOOKUP_ORDER keeps hyperbeam, which fails unless you run the hb profile, but does not remove it.

INDEX_SWARM_ENGINE_AUTH alone turns the torrent engine on: with a password set, INDEX_SWARM_ENGINE_URL defaults to the engine in the compose file. Set the URL only for an engine you run some other way.

index-swarm-status

The status script is read-only. It runs inside the sidecar, so it sees exactly what the sidecar sees, and checks:

• that the sidecar is up and the gateway's release is new enough
• per publisher: the sequence accepted and its age, any signature_failed, replayed or verify_failed, and failed downloads
• installed bands and their size; that the gateway reads the installed directory, has every band loaded, and sends lookups to them
• publishing: the document served, its expiry, and how many bands seed
• the torrent engine: that it answers, whether any peer has connected in (so a closed port shows), and the day's upload against the budget

It exits 1 when a check fails, so it can run from cron or a health script.

Doing It by Hand

What the setup script writes, if you would rather edit .env yourself:

INDEX_SWARM_SUBSCRIBE='[{"publisher":"34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5","name":"root-tx-index"}]'
INDEX_SWARM_MAX_DISK_BYTES=53687091200   # 50 GiB; about twice what the publisher offers
CDB64_ROOT_TX_INDEX_SOURCES=data/indexes/installed/root-tx-index,resources/cdb64-root-tx-index-non-ao-non-redstone-with-content-type-to-height-1820000,resources/cdb64-root-tx-index-non-ao-non-redstone-without-content-type-to-height-1820000,resources/cdb64-root-tx-index-ao-to-height-1820000
ROOT_TX_LOOKUP_ORDER=db,cdb,gateways,graphql
INDEX_SWARM_ENGINE_AUTH=swarm:<generated>   # only for BitTorrent: at least 16 characters, e.g. openssl rand -hex 24

The installed bands go first in CDB64_ROOT_TX_INDEX_SOURCES, so a fresh band answers before the older shipped indexes. The three resources/ entries are the shipped default; keep them to go on searching them, or leave them off, since each lookup against them fetches from Arweave and a subscribed set that covers the whole chain makes them redundant.

ROOT_TX_LOOKUP_ORDER matters most. By default, cdb is asked last, after every network source, so your gateway would barely use the bands it downloaded. Drop hyperbeam unless you run the hb profile.

Then restart by service name, with the same -f files your gateway was started with:

docker compose up -d --no-deps core
docker compose --profile index-swarm --profile index-swarm-torrent \
  up -d --no-deps index-swarm-engine-init index-swarm-engine index-swarm

Without BitTorrent, leave out the index-swarm-torrent profile and the two engine services. Name the services as shown: a bare docker compose up also starts or recreates every default service, including your observer.

Network and Disk

Ports

Outbound, the sidecar needs HTTPS to each publisher's gateway, and the torrent engine needs to reach peers anywhere on the internet on any port. If your host filters outbound traffic, allow both.

Both 6881 and 6969 are published on the host whenever the sidecar runs, HTTP only included, so the sidecar won't start if another program already holds either. Move them with --engine-port and INDEX_SWARM_TRACKER_PORT. Keep the engine port below 32768: Linux uses 32768–60999 for outgoing connections, and one of those can hold the port when the engine restarts.

Some hosting providers restrict BitTorrent traffic in their terms of service. Check yours before turning the engine on.

Disk

• The budget is a ceiling, not a reservation. INDEX_SWARM_MAX_DISK_BYTES stops the sidecar taking more than that, but nothing sets the space aside, so keep at least the budget free on the filesystem. It counts installed bands, downloads in progress and every torrent download twice (the download and the copy that installs), so with that much free, the budget is reached before the disk fills. A band that would go over is skipped, and index-swarm-status says so.
• One filesystem. Everything under data/indexes must be on one filesystem: an install is a rename and a publisher seeds from hard links. To put it on another disk, move the whole directory and set INDEX_SWARM_DATA_PATH; the gateway mounts the same path.
• SSD or spinning disk. An installed band answers a lookup in about 2 ms on SSD and tens of milliseconds on a busy spinning disk, still far faster than asking the network. Before installing, the sidecar reads each band once to check it, about a minute for a 7 GB band on a spinning disk. If the same disk also serves your gateway's data, cap the download rate with INDEX_SWARM_DOWNLOAD_RATE_LIMIT_BYTES_PER_SEC.

Bandwidth

A first pull of the index turbo-gateway.com publishes is about 21 GB. After that, the tip band is rebuilt daily and the next band every few days. Upload is bounded by the upload limits: at the defaults, up to 100 GB a day, about 3 TB a month. On a host that charges for bandwidth, lower them.

Firewalls and Docker Ports

The torrent engine publishes only its peer port, INDEX_SWARM_ENGINE_PORT (6881, TCP and UDP). The sidecar also publishes the tracker port, INDEX_SWARM_TRACKER_PORT (6969, TCP), on every node, but only a node that publishes torrents listens on it; a subscriber needn't open it. The engine's Web API is never published.

The engine is kept away from the rest of the gateway, because the peers and trackers it talks to are chosen by other gateways. It runs on its own Docker network, shared only with the sidecar, so it can't reach the gateway, the observer, ClickHouse or anything else on the node's network. Its IP filter refuses private, loopback, link-local and carrier-grade NAT addresses for peers, trackers and WebSeeds. It can write only its own download and configuration directories; the bands your gateway serves are mounted read-only.

Ports that Docker publishes are forwarded before the host's INPUT chain sees them, so a host firewall such as ufw neither blocks nor protects them. Open them wherever traffic reaches the host (a cloud security group or router, for example). To restrict them on the host, filter where Docker forwards: with Docker's default iptables backend, in the DOCKER-USER chain; with its nftables backend, which has no DOCKER-USER, in a chain of your own table on the forward hook, at a priority before Docker's.

Bounding Upload

Seeding is free to peers but not to you: every byte is your upload, and a peer can fetch the bands again and again. Every node that runs the engine seeds, subscribers included, so two limits apply to every node, with defaults:

A large publisher should raise both. The engine's memory also grows with the bytes it seeds: it maps the files, and mapped pages count against a container memory limit.

Tuning the Torrent Engine

The defaults suit most gateways. These are the settings worth changing, all in .env, applied by restarting the sidecar and engine:

Sharing Between Your Own Gateways

If you run several gateways on one private network, they can seed bands to each other over it instead of each pulling from the internet. By default the engine refuses private addresses, so that other gateways can't use it to reach your network. For a private swarm, set INDEX_SWARM_ENGINE_BLOCK_PRIVATE=false on each node and list your own tracker's announce URL in INDEX_SWARM_ALLOWED_TRACKERS. Do this only on a network you control.

Publishing from a Fleet

A large gateway is often several nodes behind an HTTP load balancer, and only one of them holds the observer key and signs.

For BitTorrent:

1. One node publishes and seeds. The signing node, the one /ar-io/indexes is sent to, runs the engine and the tracker. The other nodes need neither.
2. The peer port reaches that node directly. BitTorrent is not HTTP, so the load balancer can't carry it. Publish INDEX_SWARM_ENGINE_PORT (TCP and UDP) on the node's own public address, make sure the internet reaches it there (Docker-published ports bypass the host firewall; see Firewalls and Docker ports to restrict them), and set INDEX_SWARM_ENGINE_PUBLIC_HOST (--public-host) to that address. Otherwise the tracker lists the engine under the host of its tracker URL, which for a fleet is the load balancer.
3. The tracker, one of two ways:
   • Directly: publish INDEX_SWARM_TRACKER_PORT on the same address and announce to http://<that address>:6969/announce in INDEX_SWARM_TRACKERS.
   • Through the load balancer: route /announce to the signing node's tracker port, uncached, with proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;, and list the proxies' addresses in INDEX_SWARM_TRACKER_TRUSTED_PROXIES. Otherwise every peer appears at the proxy's address, the tracker's per-address limits throttle them together, and it hands out an address nobody can connect to.
4. The .torrent and WebSeed routes are under /ar-io/indexes, so the NGINX block for that prefix already covers them.

Rate Limiting and x402 in a Fleet

Nothing extra to configure. The rate limiter and x402 apply to the byte routes (files by name, by digest, and the WebSeed) the same way they apply to data. The publication and the .torrent files are free. Peer-to-peer transfer and tracker announces never reach the gateway at all; the upload budget bounds them instead. Because /ar-io/indexes goes to one node, all metering happens there, so per-address limits stay consistent even if your nodes don't share a rate limiter.

Giving the Other Nodes the Index

If your other nodes answer lookups from their own disk, they need the bands installed too. Subscribe each one over HTTP to your own publication, pointed at the publishing node directly:

INDEX_SWARM_SUBSCRIBE='[{"publisher":"<your gateway wallet>","name":"root-tx-index","url":"http://<publishing node>:4000"}]'

url only changes where the publication and files are fetched from. The signature is still checked against your registered observer key, so an internal address is safe. Then add those nodes' addresses to RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST on the publishing node, so they aren't rate-limited or asked to pay (the allowlist applies when the rate limiter is on). They don't need a torrent engine: between nodes on one network, HTTP is simpler.

Subscribers behind NAT still work: they reach the publisher's engine, and a reachable subscriber can be reached back.

Behind NGINX

Most gateways run NGINX in front, and many cache. The index routes are built to be correct through a cache without special configuration: errors are never cacheable, and when your gateway meters, the files it serves are marked private. Two things are still worth setting:

• Forward the client IP with X-Forwarded-For, as the quick start configuration does, so each subscriber gets its own rate limit.
• If you run more than one node, send /ar-io/indexes to the node that holds the observer key. Only that node signs.

The torrent routes, /ar-io/indexes/torrents/ and /ar-io/indexes/webseed/, sit under the same prefix, so the same rules cover them. See Advanced NGINX Caching for a complete location block.

Monitoring

./tools/index-swarm-status covers the day-to-day checks. The sidecar also serves Prometheus metrics on port 9101 inside its container. The ones worth watching:

On the gateway, indexes_requests_total{route,status} shows who is downloading from you, by route (publication, file, blob, torrent, webseed), including 402 and 429 from your rate limits.

Troubleshooting

Run ./tools/index-swarm-status first: it names most problems and their fix.

Nothing installs, and the logs show 402 or 429. The publisher is rate-limiting you. It is expected on a first download over HTTP: each poll picks up where the last stopped. Turn on BitTorrent, ask the publisher to raise your limits, or wait.

Bands installed, but lookups still go to the network. Check ROOT_TX_LOOKUP_ORDER puts cdb right after db, and that CDB64_ROOT_TX_INDEX_SOURCES starts with data/indexes/installed/root-tx-index. Both need a gateway restart; ./tools/index-swarm-setup --restart does it.

No peer has connected in. Port 6881 is closed somewhere between the internet and the engine. Bands still arrive, from peers and over HTTP, and the peers you connect to can download from you, but peers that can't reach you can't. Open or forward the port if you can; for a subscriber it is optional.

Bind for 0.0.0.0:6969 failed: port is already allocated (or 6881). Another program on the host holds the port. Move the engine with --engine-port, or the tracker with INDEX_SWARM_TRACKER_PORT, and restart.

The engine refuses the sidecar. Check INDEX_SWARM_ENGINE_AUTH. The sidecar does not retry a failed login, because the engine bans an address after five.

Publication was signed by an unregistered key. The publication you fetched was not signed by the observer key registered for that wallet. Check the publisher's registry record; if it just changed keys, it can take up to an hour to reach your gateway's registry view.

Set OBSERVER_KEYPAIR_PATH or OBSERVER_PRIVATE_KEY, not both. On a publisher, OBSERVER_PRIVATE_KEY and INDEX_SWARM_OBSERVER_KEYPAIR_FILE are both set (inside the sidecar, INDEX_SWARM_OBSERVER_KEYPAIR_FILE becomes OBSERVER_KEYPAIR_PATH). Keep one.

Gateway is too old to load installed index bands. Upgrade the gateway to Release 84 or later. The sidecar notices the upgrade by itself.

Turning It Off

Stopping the sidecar changes nothing your gateway serves: installed bands stay loaded, and published bands stay served from the last publication written.

docker compose --profile index-swarm stop index-swarm
docker compose --profile index-swarm rm -f index-swarm

With the torrent engine, also stop and remove index-swarm-engine and index-swarm-engine-init (profile index-swarm-torrent), then delete data/indexes/swarm/, data/indexes/torrents/ and data/index-swarm-engine/. Remove INDEX_SWARM_ENGINE_AUTH from .env too: while it is set, the sidecar expects the engine and warns that it is not answering.

• On a subscriber, restore your previous CDB64_ROOT_TX_INDEX_SOURCES and restart the gateway, then delete data/indexes/installed/. In that order, so the gateway is no longer holding the files open.
• On a publisher, delete data/indexes/published/publication.json. The gateway stops serving the routes on its next request, without a restart. The band directories can then go too.

Related