Reading Shared Indexes

Gateways that take part in Index Sharing publish signed indexes that locate data items inside their bundles. Gateways use them automatically, but nothing about them is gateway-only: any client can fetch a publication, verify it, and look up an ID itself. This page shows how. To have your own gateway subscribe or publish, see Index Sharing.

The Publication

A publishing gateway serves one JSON publication at /ar-io/indexes:

{
  "version": 1,
  "publisher": "34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5",
  "sequence": 12,
  "issuedAt": "2026-10-01T12:00:00.000Z",
  "expiresAt": "2026-10-02T12:00:00.000Z",
  "indexes": [
    {
      "name": "root-tx-index",
      "kind": "cdb64-root-tx",
      "bands": [
        {
          "id": "b1-h1950000-tip-20260917",
          "heightRange": [1950000, null],
          "files": [
            { "name": "manifest.json", "size": 41233, "sha256": "…" },
            { "name": "00.cdb", "size": 7012345, "sha256": "…" }
          ]
        }
      ]
    }
  ],
  "signature": { "alg": "ed25519", "keyId": "34LYv…", "sig": "…" }
}

publisher is the gateway's wallet, signature.keyId its observer address, and signature.sig a base64 Ed25519 signature. The publication may carry fields not shown here, and later versions may add more. Keep them: the signature covers them.

Verify It

import { canonicalize } from "json-canonicalize";

const B58 = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz";
function base58Decode(s) {
  let n = 0n;
  for (const c of s) n = n * 58n + BigInt(B58.indexOf(c));
  const bytes = [];
  while (n > 0n) {
    bytes.unshift(Number(n % 256n));
    n /= 256n;
  }
  for (const c of s) {
    if (c !== "1") break;
    bytes.unshift(0);
  }
  return new Uint8Array(bytes);
}

async function verifyPublication(doc, observerAddress) {
  const { signature, ...unsigned } = doc;
  if (signature?.alg !== "ed25519") throw new Error("unknown algorithm");
  if (signature.keyId !== observerAddress) throw new Error("not the registered key");

  const key = await crypto.subtle.importKey(
    "raw",
    base58Decode(signature.keyId),
    { name: "Ed25519" },
    false,
    ["verify"],
  );
  const message = new TextEncoder().encode(
    "ar-io-index-publication/v1\n" + canonicalize(unsigned),
  );
  const sig = Uint8Array.from(atob(signature.sig), (c) => c.charCodeAt(0));
  return crypto.subtle.verify("Ed25519", key, sig, message);
}

const doc = await fetch("https://turbo-gateway.com/ar-io/indexes").then((r) => r.json());
console.log(await verifyPublication(doc, "<observerAddress from the registry>"));

Fetch a Band File

Fetch files by their SHA-256. The address can't change meaning, so it is safe to cache and can come from any server that has the file:

async function fetchFile(gateway, file) {
  const res = await fetch(`${gateway}/ar-io/indexes/blob/${file.sha256}`);
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const bytes = new Uint8Array(await res.arrayBuffer());
  const digest = await crypto.subtle.digest("SHA-256", bytes);
  const hex = [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, "0")).join("");
  if (bytes.length !== file.size || hex !== file.sha256) throw new Error("file does not match");
  return bytes;
}

Range requests are supported, which is how a large file resumes. Peers seeding a band over BitTorrent are not metered; see Bands as torrents. A 503 with Retry-After means the publisher is part way through replacing a band: fetch the publication again after the delay.

Bands as Torrents

A publisher that runs a torrent engine adds an optional torrent entry to each band it seeds:

"torrent": {
  "infohashV1": "<40 hex characters>",
  "infohashV2": "<64 hex characters>",
  "magnet": "magnet:?xt=urn:btih:…",
  "torrentUrl": "/ar-io/indexes/torrents/<infohashV1>.torrent"
}

The entry is absent when the publisher runs no engine; the HTTP routes always work. The torrent route can answer 404 until the publisher has built a band's torrent, which is normal.

The torrent name is derived from content, not the band id: the first 16 hex characters of the SHA-256 over one line per file, <name>\0<size>\0<sha256 hex>\n, with files in bytewise name order. So publishers of the same bytes share one infohash and one swarm. It is also the <torrent name> in the WebSeed route, /ar-io/indexes/webseed/<torrent name>/<file>, which serves band files to torrent clients and is metered like the blob route.

Only the info dictionary is signed. The infohashes cover the torrent's info dictionary (file names, sizes and piece hashes) and nothing else. Trackers and WebSeeds in a .torrent file are outside it and unsigned. The gateway's own subscriber checks a .torrent against the signed infohashes, checks that its file list is exactly the band's signed files and sizes (plus BEP 47 pad files), drops every WebSeed it names, and hashes every downloaded file against its signed SHA-256, as over HTTP. A client of its own should do the same.

Pull a Whole Index with Any BitTorrent Client

To mirror a publisher's full index, for a pipeline or an agent, use the magnet link or the .torrent file with any BitTorrent client. Peers aren't metered, so this is the cheapest way to take everything. Before using a .torrent, check its infohash matches the signed infohashV1, since the file itself is unsigned. After the download, check every file's size and SHA-256 against the signed publication, exactly as for an HTTP fetch: the infohash protects the pieces, but only the publication says these are the right files. Seed it afterwards if you can; that is what keeps the swarm fast. Downloading Indexes with BitTorrent walks through it with qBittorrent and aria2.

Look Up One ID

A cdb64-root-tx band is a partitioned CDB64 index. To find the root transaction of one data item:

1. Fetch and check the band's manifest.json.
2. Pick the partition for the ID's first byte: the manifest lists partitions by two-character hex prefix.
3. Fetch and check that one partition, up to about 30 MB.
4. Look the 32-byte ID up in it. The value is MessagePack, holding the root transaction ID and, when known, the item's byte offsets.

Bands may overlap. A publisher should never have two bands disagree about an item, since an item has one location, so search them newest first and stop at the first match.

The reference client does all of this in about 130 lines of Python, with the CDB64 reader and MessagePack decoder written out.