Verifying Shared Indexes

A signed publication proves who published an index and that the files you hold are the ones they signed. It does not prove the index is right: that every row is where it says, that every location points at the item it names, that the lookups match the tables. The ar-io-node CLI has one read-only check per index kind that proves those things from the bands alone. Anyone holding a set of bands can run it: a subscriber after installing them, a publisher before publishing, or an agent asked whether an index is sound.

Which Check, for Which Index

Every command follows the CLI contract: the result as JSON on stdout and exit 0; on failure the full result on stderr and exit 1. Every option is in the ar-io-node CLI reference.

Run It

IMAGE=ghcr.io/ar-io/ar-io-core:latest
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/ardrive-l2:/bands:ro" \
  -w /app --entrypoint /nodejs/bin/node "$IMAGE" \
  /app/dist/cli/cli.js index-l2-verify --bands-dir /bands \
  --publisher 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5

./tools/ar-io-node index-l2-verify --bands-dir data/indexes/installed/ardrive-l2 \
  --publisher 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5

{
  "ok": true,
  "bands": 23,
  "items": 127734401,
  "checks": {
    "identity": { "ok": true, "failures": [] },
    "selection": { "ok": true, "failures": [] },
    "files": { "ok": true, "failures": [] },
    "tables": { "ok": true, "failures": [] },
    "locations": { "ok": true, "failures": [] },
    "parents": { "ok": true, "failures": [] },
    "lookups": { "ok": true, "failures": [] },
    "onchain": {
      "ok": true,
      "checked": 800, "wrong": 0, "unreadable": 12, "errors": 0,
      "seed": 1934802271,
      "failures": []
    }
  },
  "superseded": [],
  "unlocatedBands": []
}

What index-l2-verify Checks

The set is refused outright, with no checks run, if its bands overlap or leave a hole: that is not one index. --from and --to check only the bands inside those heights. A band that its own publisher's newer band supersedes is left out, as a subscriber leaves it out, and listed under superseded.

Every row is read in file order, so memory stays flat. The lookup check sorts and joins, and spills to the system temp directory: --spill-max caps it at 128 GB unless set. Measured on a 2-core, 10 GB host, the whole ardrive-l2 index of 23 bands and 127.7 million items, with 50 items per band read from Arweave nodes, took 1 hour 51 minutes; its largest band (69 million items) takes 66 minutes alone and spills about 65 GB. The on-chain sample is seconds per band.

Reading from Arweave Nodes

index-l2-verify, and index-band-verify --arweave-nodes, read root transactions from Arweave nodes and prove every byte, so the check trusts no gateway, the publisher's included:

1. The root's id is bound to its data. Its header's signature must hash to its id, and the signature must verify, since a transaction signs its data_root and data_size. For the few valid transactions arweave-js can't verify, at least two nodes must serve the same data_root and data_size instead; the result says which (bySignature, byAgreement).
2. Its place in the weave comes from /tx/{id}/offset, whose size must be the signed data_size.
3. Every chunk is proven. Its data_path must validate against the signed data_root under the consensus rules for its weave offset, the chunk must hash to the path's leaf, and it is placed by the bounds the proof gives. A node whose chunk fails is passed over for the next. A wrong offset can't pass wrong bytes: the chunks it leads to would not prove against the root's data_root.

Each sampled header comes out as one of:

A 404 counts only when every node that answered said 404. One busy node among them, or one whose chunk failed its proof, makes it an error instead: that node may hold the data, and an unprovable chunk is the node's fault, not an absence.

The default pool is the nodes a gateway reads chunks from. Most of them hold a partition of the weave and answer 404 for most offsets; that is normal. Name your own with --arweave-nodes http://a:1984,http://b:1984. With --gateway-url instead, roots are read through that gateway's /raw: faster, but that gateway is trusted.

When a Check Fails

A band's id is a digest of its heights, selection and rows, so a publisher, or anyone with the same source data, can rebuild a band and compare ids: equal ids are equal rows. A source that has changed since (items backfilled, offsets repaired) gives a new id, which is what re-settling publishes; compare against the source as of the same time, or diff the rows.

For Agents

Run the command, read stdout (or stderr on exit 1) as JSON, and act on ok and each check's failures. unreadable is never a fault of the index, and errors are never a verdict on it either way, so rerun before acting on them. Never treat a passing sample as proof of every row: the row-by-row checks are what cover every row.