Verifying Shared Indexes

A signed publication proves who published an index and that the files you hold are the ones they signed. It does not prove the index is right: that every row is where it says, that every location points at the item it names, that the lookups match the tables. The ar-io-node CLI has one read-only check per index kind that proves those things from the bands alone. Anyone holding a set of bands can run it: a subscriber after installing them, a publisher before publishing, or an agent asked whether an index is sound.

Which Check, for Which Index

Every command follows the CLI contract: the result as JSON on stdout and exit 0; on failure the full result on stderr and exit 1. Every option is in the ar-io-node CLI reference.

Run It

IMAGE=ghcr.io/ar-io/ar-io-core:latest
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/ardrive-l2:/bands:ro" \
  -w /app --entrypoint /nodejs/bin/node "$IMAGE" \
  /app/dist/cli/cli.js index-l2-verify --bands-dir /bands \
  --publisher 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5

./tools/ar-io-node index-l2-verify --bands-dir data/indexes/installed/ardrive-l2 \
  --publisher 34LYvMptiDvBP5sqfh1oAd6Q4qFsy4PWaZ1HTFmML7h5

IMAGE=ghcr.io/ar-io/ar-io-core:latest   # Release 87 or later
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD/parquet-l1:/bands:ro" \
  -w /app --entrypoint /nodejs/bin/node "$IMAGE" \
  /app/dist/cli/cli.js index-l1-verify --bands-dir /bands \
  --anchor-from https://<gateway one>,https://<gateway two> \
  --publication https://<publisher gateway>/ar-io/indexes \
  --publisher-key <the publisher's observer address>

./tools/ar-io-node index-l1-verify --bands-dir data/indexes/installed/parquet-l1 \
  --anchor-from https://<gateway one>,https://<gateway two>

./tools/ar-io-node index-l1-verify --core-db data/sqlite/core.db \
  --anchor-from https://<gateway one>,https://<gateway two> \
  --allow-unbound --allow-unverified-format1

{
  "ok": true,
  "bands": 23,
  "items": 127734401,
  "checks": {
    "identity": { "ok": true, "failures": [] },
    "selection": { "ok": true, "failures": [] },
    "files": { "ok": true, "failures": [] },
    "tables": { "ok": true, "failures": [] },
    "locations": { "ok": true, "failures": [] },
    "parents": { "ok": true, "failures": [] },
    "lookups": { "ok": true, "failures": [] },
    "onchain": {
      "ok": true,
      "checked": 800, "wrong": 0, "unreadable": 12, "errors": 0,
      "seed": 1934802271,
      "failures": []
    }
  },
  "superseded": [],
  "unlocatedBands": []
}

What index-l2-verify Checks

The set is refused outright, with no checks run, if its bands overlap or leave a hole: that is not one index. --from and --to check only the bands inside those heights. A band that its own publisher's newer band supersedes is left out, as a subscriber leaves it out, and listed under superseded.

Every row is read in file order, so memory stays flat. The lookup check sorts and joins, and spills to the system temp directory: --spill-max caps it at 128 GB unless set. Measured on a 2-core, 10 GB host, the whole ardrive-l2 index of 23 bands and 127.7 million items, with 50 items per band read from Arweave nodes, took 1 hour 51 minutes; its largest band (69 million items) takes 66 minutes alone and spills about 65 GB. The on-chain sample is seconds per band.

Reading from Arweave Nodes

index-l2-verify, and index-band-verify --arweave-nodes, read root transactions from Arweave nodes and prove every byte, so the check trusts no gateway, the publisher's included:

1. The root's id is bound to its data. Its header's signature must hash to its id, and the signature must verify, since a transaction signs its data_root and data_size. For the few valid transactions arweave-js can't verify, at least two nodes must serve the same data_root and data_size instead; the result says which (bySignature, byAgreement).
2. Its place in the weave comes from /tx/{id}/offset, whose size must be the signed data_size.
3. Every chunk is proven. Its data_path must validate against the signed data_root under the consensus rules for its weave offset, the chunk must hash to the path's leaf, and it is placed by the bounds the proof gives. A node whose chunk fails is passed over for the next. A wrong offset can't pass wrong bytes: the chunks it leads to would not prove against the root's data_root.

Each sampled header comes out as one of:

A 404 counts only when every node that answered said 404. One busy node among them, or one whose chunk failed its proof, makes it an error instead: that node may hold the data, and an unprovable chunk is the node's fault, not an absence.

The default pool is the nodes a gateway reads chunks from. Most of them hold a partition of the weave and answer 404 for most offsets; that is normal. Name your own with --arweave-nodes http://a:1984,http://b:1984. With --gateway-url instead, roots are read through that gateway's /raw: faster, but that gateway is trusted.

What index-l1-verify Checks

index-l1-verify has two views of the same index. --core-db is the operator's: does this gateway's own database hold the chain? --bands-dir is the consumer's: a directory of bands, checked with no gateway and no database, so anyone deciding whether to install someone else's bands can check them before importing a byte. Given both, it checks core.db. It is read-only, so the gateway can stay up. Both views run the same judge over the same chain rules, and index-l1-import runs that judge too before it writes.

What Binds What

An index agreeing with itself proves nothing: a fabricated chain built by Arweave's rules passes every self-check. Three things tie it to the real chain:

• The anchor (--anchor-from). The block at the top of the range is fetched (/block/height/N) from every source given. At least --anchor-min distinct machines (default 2, and never fewer) must serve it, and names that resolve to a shared address count as one machine. Each header's identity hash is recomputed from its own fields, for every era from the 2.0 fork; every source that answered must give the same hash; and the index's row must equal the header field by field, with its transaction ids in block order. A header the command cannot recompute exactly is a failure, never a pass. Below the 2.0 fork no header recomputes, so a range ending at 422,249 is anchored at the fork block, whose previous_block binds it, and an l1-4 range ending at a pinned height (99,999, 199,999, 299,999 or 399,999) is anchored by its witnessed folds reaching the pins there (anchor.pinned: true, no header fetched). So pre-2.0 bands verify and import with no --allow-unanchored.
• The hash_list_merkle recurrence. From the 2.0 fork, each block's hash_list_merkle folds the hash, weave size and tx_root of every block below it, so an anchored top binds those three for every post-2.0 block in the range. The links then bind each previous_block, the weave each block_size, and a recomputed tx_root each block's transactions' data sizes and data roots. Not their ids: tx_root drops them.
• The sample. Nothing above binds transaction ids, their order, any other transaction field, or any other block field. Checking every one would mean fetching every header or every signature, so the command draws a keyed uniform sample from the --anchor-from sources: the headers of --sample-blocks blocks (default 1000), recomputed and compared field by field from the 2.0 fork, and below it attested: the header --anchor-min distinct machines serve alike, the sources' word rather than a proof (counted as blocksAttested, and what binds a pre-2.0 block's transaction order); --sample-txs transactions per band (default 1000), pre-2.0 included, each fetched and checked against its signature over the fields the index holds; and --sample-format1 format-1 transactions with data per band (default 200) drawn from the 2.0 fork, which catches a format-2 transaction relabelled format 1, witnessed or not. --sample-blocks 0 with sources, or no format-1 draw over such transactions (--sample-format1 0, or no --anchor-from), fails the run unless --allow-unsampled. sample.detects says what a pass means: the share of forged rows that would have been caught with 95% probability. A forgery confined to fewer rows may pass. --seed repeats a run.

RSA signatures are verified exactly as Arweave verifies them, not by OpenSSL, which is stricter and rejects some real mined transactions. A recomputed header or a verifying signature decides on one source's word. A refutation on a source's word needs --anchor-min distinct machines saying so: a single lying source makes a draw unreadable, never a failure. More than 20% of any one kind of draw (blocks, transactions, format-1) unreadable in any one band fails the sample as unreadable.

What two agreeing sources mean depends on who they are. Two ar.io gateways both route /block/ to their operators' chosen Arweave nodes, so their agreement is those choices agreeing, not two independent nodes. A source that redirects is not followed. The result names each source and the address it was counted as.

Witness Files (l1-4)

Two kinds of block can't be bound from a band's tables alone, because the chain committed to something no header carries:

• A block holding a format-1 transaction with data. Its tx_root leaf is the root of that transaction's data, and a format-1 header carries no data root.
• A block below the 2.0 fork (height 422,250). Its identity hash covers the full wallet list, which no header carries, so it can't be recomputed. Arweave's 2.0 fork bound these blocks another way, through the 2.0 seed of hash_list_merkle.

From layout l1-4 a band carries what is missing, as two witness files:

A witness is an untrusted value: whoever wrote it needs no trust, because the verifier recomputes everything it can from the band and checks the rest against hashes the chain fixes. A v1 data root is put into its block's recomputed tx_root, which must then equal the header's (below the 2.0 fork, the witnessed transition tx_root). Each transition tx_root is recomputed from the block's transactions. Each fold is recomputed from the one below and must reach block 422,250's hash_list_merkle, or a fold pinned in ar-io-node at heights 99,999, 199,999, 299,999, 399,999 and 422,249, recomputed by a test from Arweave's own 2.0 checkpoint. That binds each pre-2.0 block's hash, weave size and tx_root to the chain. Each block's 2.0 hash is recomputed from the band's own fields and transaction ids, and their fold must reach the pinned folds of Arweave's genesis_data/hash_list_1_0, the list Arweave checks every pre-2.0 header it syncs against. That binds the rest of a pre-2.0 header and its transaction ids to Arweave's checkpoint, not to a hash the chain's own blocks reach, and the result says so. A pre-2.0 block is bound only when both folds reach their pins.

A wrong witness fails the check (tx_root, hash_list_merkle, witnesses or hash_list_1_0); it never falls back to unbound. An l1-4 band holds every witness, and a block of it left unbound fails witnesses, whatever --allow-* flag is given. A band whose witnesses its publisher could not have or confirm in full is published as l1-3 instead.

Witnesses are not part of a band's id. A publisher adds them to a published l1-3 band in place: only band.json and the new files change, the band keeps its id, and a subscriber that holds it fetches only those. Every witness is checked against the chain before the band is published, by the same code every verifier runs, and again by every index-l1-verify and import that reads it. The full format is in the publication protocol.

What It Cannot Bind (unbound)

It fails closed. What it has no way to bind is reported in unbound, by class, and fails the run unless you accept that class with its flag. A whole set of l1-4 bands needs none of them:

--allow-unbound and --allow-unverified-format1 accept such blocks of a core.db or of bands without witnesses, never an l1-4 band's. Without --anchor-from, nothing outside the index vouches for it: the result says anchored: false, and the run fails unless --allow-unanchored. Every flag that accepted something is echoed in accepted, so a pass that rests on one says so; read unbound and assurance and accept a class only if a pass on those terms is enough for your use.

The Checks

--from and --to set the heights to check (default: everything held). With --bands-dir, the directory is listed explicitly, never globbed: a folder without a readable band.json (a retired copy, or a stray directory) is ignored and listed in ignored, and two installed generations of one band with the same tables are read once. Bands only partly inside --from/--to are not checked and fail coverage. A set that overlaps or leaves a hole, or a --bands-dir that is missing or cannot be read, is refused outright. --publication, with --publisher-key or --registry-url, binds the bands to the publication their publisher signed; without it the result says publisherBound: false.

The lookup check spills to the system temp directory (through tools/ar-io-node, data/tmp/index-verify on the node's data disk). The spill is capped at --spill-max (default 128GB) and at 80% of the space free there, and never goes into the last --spill-reserve-gb (default 100) of it. With no room under that, the run is refused before it starts, unless --skip-lookups.

Every option and the full result are in the CLI reference.

When a Check Fails

A sample mismatch on reward_pool between heights 585,633 and 633,719 is a known fault of indexes built before Release 86, not of the chain: before the 2.4 fork, headers write reward_pool as a bare JSON number, and from 585,633 it passes 2^53, so the block importer, parsing it as a JavaScript number, stored an odd value rounded to an even neighbour. Gateways that indexed those heights themselves, or imported L1 bands built from them, hold the rounded values. From Release 86 block headers are parsed with long integers kept exact.

A band's id is a digest of its heights, selection and rows, so a publisher, or anyone with the same source data, can rebuild a band and compare ids: equal ids are equal rows. A source that has changed since (items backfilled, offsets repaired) gives a new id, which is what re-settling publishes; compare against the source as of the same time, or diff the rows.

For Agents

Run the command, read stdout (or stderr on exit 1) as JSON, and act on ok and each check's failures. unreadable is never a fault of the index, and errors are never a verdict on it either way, so rerun before acting on them. Never treat a passing sample as proof of every row: the row-by-row checks are what cover every row. For index-l1-verify, also read anchored, unbound, accepted and assurance: a pass that rests on an accepted gap lists it in accepted.