Reading Shared Indexes

Gateways that take part in Index Sharing publish signed indexes that locate data items inside their bundles. Gateways use them automatically, but nothing about them is gateway-only: any client can fetch a publication, verify it, and look up an ID itself. This page shows how. To have your own gateway subscribe or publish, see Index Sharing.

The Publication

A publishing gateway serves one JSON publication at /ar-io/indexes:

{
  "version": 1,
  "publisher": "34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5",
  "sequence": 12,
  "issuedAt": "2026-10-01T12:00:00.000Z",
  "expiresAt": "2026-10-02T12:00:00.000Z",
  "indexes": [
    {
      "name": "root-tx-index",
      "kind": "cdb64-root-tx",
      "bands": [
        {
          "id": "b1-h1950000-tip-20260917",
          "heightRange": [1950000, null],
          "files": [
            { "name": "manifest.json", "size": 41233, "sha256": "…" },
            { "name": "00.cdb", "size": 7012345, "sha256": "…" }
          ]
        }
      ]
    }
  ],
  "signature": { "alg": "ed25519", "keyId": "34LYv…", "sig": "…" }
}

publisher is the gateway's wallet, signature.keyId its observer address, and signature.sig a base64 Ed25519 signature. The publication may carry fields not shown here, and later versions may add more. Keep them: the signature covers them.

Verify It

import { canonicalize } from "json-canonicalize";

const B58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
function base58Decode(s) {
  let n = 0n;
  for (const c of s) n = n * 58n + BigInt(B58.indexOf(c));
  const bytes = [];
  while (n > 0n) {
    bytes.unshift(Number(n % 256n));
    n /= 256n;
  }
  for (const c of s) {
    if (c !== "1") break;
    bytes.unshift(0);
  }
  return new Uint8Array(bytes);
}

async function verifyPublication(doc, observerAddress) {
  const { signature, ...unsigned } = doc;
  if (signature?.alg !== "ed25519") throw new Error("unknown algorithm");
  if (signature.keyId !== observerAddress) throw new Error("not the registered key");

  const key = await crypto.subtle.importKey(
    "raw",
    base58Decode(signature.keyId),
    { name: "Ed25519" },
    false,
    ["verify"],
  );
  const message = new TextEncoder().encode(
    "ar-io-index-publication/v1\n" + canonicalize(unsigned),
  );
  const sig = Uint8Array.from(atob(signature.sig), (c) => c.charCodeAt(0));
  return crypto.subtle.verify("Ed25519", key, sig, message);
}

const doc = await fetch("https://turbo-gateway.com/ar-io/indexes").then((r) => r.json());
console.log(await verifyPublication(doc, "<observerAddress from the registry>"));

Fetch a Band File

Fetch files by their SHA-256. The address can't change meaning, so it is safe to cache and can come from any server that has the file:

async function fetchFile(gateway, file) {
  const res = await fetch(`${gateway}/ar-io/indexes/blob/${file.sha256}`);
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const bytes = new Uint8Array(await res.arrayBuffer());
  const digest = await crypto.subtle.digest("SHA-256", bytes);
  const hex = [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, "0")).join("");
  if (bytes.length !== file.size || hex !== file.sha256) throw new Error("file does not match");
  return bytes;
}

Range requests are supported, which is how a large file resumes. Peers seeding a band over BitTorrent are not metered; see Bands as torrents. A 503 with Retry-After means the publisher is part way through replacing a band: fetch the publication again after the delay.

Bands as Torrents

A publisher that runs a torrent engine adds an optional torrent entry to each band it seeds:

"torrent": {
  "infohashV1": "<40 hex characters>",
  "infohashV2": "<64 hex characters>",
  "magnet": "magnet:?xt=urn:btih:…",
  "torrentUrl": "/ar-io/indexes/torrents/<infohashV1>.torrent"
}

The entry is absent when the publisher runs no engine; the HTTP routes always work. The torrent route can answer 404 until the publisher has built a band's torrent, which is normal.

The torrent name is derived from content, not the band id: the first 16 hex characters of the SHA-256 over one line per file, <name>\0<size>\0<sha256 hex>\n, with files in bytewise name order. So publishers of the same bytes share one infohash and one swarm. It is also the <torrent name> in the WebSeed route, /ar-io/indexes/webseed/<torrent name>/<file>, which serves band files to torrent clients and is metered like the blob route.

Only the info dictionary is signed. The infohashes cover the torrent's info dictionary (file names, sizes and piece hashes) and nothing else. Trackers and WebSeeds in a .torrent file are outside it and unsigned. The gateway's own subscriber checks a .torrent against the signed infohashes, checks that its file list is exactly the band's signed files and sizes (plus BEP 47 pad files), drops every WebSeed it names, and hashes every downloaded file against its signed SHA-256, as over HTTP. A client of its own should do the same.

Pull a Whole Index with Any BitTorrent Client

To mirror a publisher's full index, for a pipeline or an agent, use the magnet link or the .torrent file with any BitTorrent client. Peers aren't metered, so this is the cheapest way to take everything. Before using a .torrent, check its infohash matches the signed infohashV1, since the file itself is unsigned. After the download, check every file's size and SHA-256 against the signed publication, exactly as for an HTTP fetch: the infohash protects the pieces, but only the publication says these are the right files. Seed it afterwards if you can; that is what keeps the swarm fast. Downloading Indexes with BitTorrent walks through it with qBittorrent and aria2.

Look Up One ID

A cdb64-root-tx band is a partitioned CDB64 index. To find the root transaction of one data item:

1. Fetch and check the band's manifest.json.
2. Pick the partition for the ID's first byte: the manifest lists partitions by two-character hex prefix.
3. Fetch and check that one partition, up to about 30 MB.
4. Look the 32-byte ID up in it. The value is MessagePack, holding the root transaction ID and, when known, the item's byte offsets.

Bands may overlap. A publisher should never have two bands disagree about an item, since an item has one location, so search them newest first and stop at the first match.

The reference client does all of this in about 130 lines of Python, with the CDB64 reader and MessagePack decoder written out.

Query a Parquet Dataset

A cdb64-root-tx band answers one question quickly: where is this ID. A parquet-l1 band answers a different kind: how many, how large, which, over the whole chain. It holds Arweave's base layer as Apache Parquet, partitioned into bands by block height, so any engine that reads Parquet can query it. No band files to download, no import into a database, and no API to learn. The examples use DuckDB, which installs its own httpfs and json extensions on first use.

See What a Gateway Publishes

Not every gateway publishes every dataset. parquet-l1 is opt-in, so check first:

curl -s https://turbo-gateway.com/ar-io/indexes \
  | jq '.indexes[] | {name, kind, bands: (.bands | length)}'

{ "name": "root-tx-index", "kind": "cdb64-root-tx", "bands": 5 }
{ "name": "parquet-l1", "kind": "parquet-l1", "bands": 24 }

name is the dataset you want; kind tells you how to read it. They are not the same thing, and for root-tx-index they differ.

The Tables

Each band directory holds one Parquet file per table, plus a band.json with the band's heights, row counts and a schema version:

IDs, addresses, hashes and tag names and values are stored as raw bytes (BLOB), not text. Base64url is a presentation format, so convert when you want to read one.

Query It in Place, Downloading Nothing

The byte routes serve HTTP range requests, and Parquet is designed to be read that way: the reader fetches the footer, decides which row groups it needs, and fetches only those. So a query against a dataset on a gateway moves a small fraction of it.

This is a complete script. DuckDB is the only dependency, and it reads the publication itself to find the bands:

INSTALL httpfs; LOAD httpfs; INSTALL json; LOAD json;

SET VARIABLE gateway = 'https://turbo-gateway.com';

SET VARIABLE txs = (
  SELECT list(getvariable('gateway') || '/ar-io/indexes/parquet-l1/' || b.id || '/transactions.parquet')
  FROM (SELECT unnest(ix.bands) AS b
        FROM (SELECT unnest(indexes) AS ix
              FROM read_json(getvariable('gateway') || '/ar-io/indexes'))
        WHERE ix.name = 'parquet-l1')
);

SELECT count(*) AS transactions
FROM read_parquet(getvariable('txs'))
WHERE height BETWEEN 1500000 AND 1500099;

Swap the last statement for whatever you want to ask. The largest transactions ever posted to the base layer, with their IDs converted back to base64url:

SELECT
  rtrim(replace(replace(to_base64(id), '+', '-'), '/', '_'), '=') AS tx_id,
  height,
  round(data_size / power(2, 30), 1) AS gib
FROM read_parquet(getvariable('txs'))
ORDER BY data_size DESC
LIMIT 3;

┌─────────────────────────────────────────────┬─────────┬───────┐
│                    tx_id                    │ height  │  gib  │
├─────────────────────────────────────────────┼─────────┼───────┤
│ oWRzBr3KHhULAL-s5ULeXac1mb_WQOX5uFBRea16iRI │ 1992471 │ 621.9 │
│ 7vg2832WFsisEcBr1oBQ8ldc4EGOkjQdwW46hDvJsOs │ 1989512 │ 159.6 │
│ SFUs16XzaWFhWvWOR55hb-5X081r2LeWWAHq4HWUt00 │ 1372669 │  33.7 │
└─────────────────────────────────────────────┴─────────┴───────┘

Tags and content types work the same way. Both columns are bytes, so cast them:

SELECT CAST(tag_name AS VARCHAR) AS tag, count(*) AS n
FROM read_parquet(getvariable('gateway') || '/ar-io/indexes/parquet-l1/<band-id>/tags.parquet')
GROUP BY 1 ORDER BY n DESC LIMIT 5;

What Actually Crosses the Network

Worth understanding before you point a pipeline at a gateway, because the cost depends on the query rather than on the dataset. Measured against a 24-band dataset covering heights 0 to 2,016,168, whose transactions.parquet files total 5.51 GB on the server:

The first two cost the same because a selective filter still reads 24 footers and then one row group: that is predicate pushdown working. The third is larger because an aggregate over a whole column has to read that column. Filtering on height is the cheapest filter, since bands are partitioned by it and row groups are ordered within a band.

Verify Before You Trust It

Querying over HTTP means trusting the server for the bytes it streams you. If the answer matters, verify the publication first, then fetch the band files you need by digest and query them locally. Two things make this cheaper than it sounds:

• Every band file has a signed SHA-256 in the publication, so a local copy is checkable.
• Repr-Digest is co-signable, so even a 206 carries a signature over the digest of the whole file it came from. A client can confirm which file it is reading bytes out of without fetching all of it.

A parquet-l1 dataset is also checkable against Arweave itself, independently of the publisher: block linkage, hash_list_merkle, tx_root and weave accounting can all be recomputed from the rows. ar-io-node index-l1-verify --bands-dir does exactly that over a local copy.

Metering

The publication and .torrent files are free. The byte routes are metered like data, so an analytical client is a paying or allowlisted client, with a free allowance first. An accidental full-file download can spend that allowance in one request and the next range request returns 402; that is metering, not a fault. For sustained or heavy work, take the whole dataset over BitTorrent, which is not metered, and query it locally:

SELECT count(*) FROM read_parquet('parquet-l1/*/transactions.parquet')
WHERE height BETWEEN 1000000 AND 1000100;

This or GraphQL?

Both read a gateway's index, and they are good at opposite things.

GraphQL cannot aggregate: there is no count or sum, and a large result has to be paged. Parquet has no index on id, so a point lookup means a scan, and a published band set lags the tip by its publishing cadence. Reach for GraphQL to find things and parquet-l1 to measure them.