Versions API

Push Protocol (Negotiate → Records → Commit)

All pushes use the negotiate protocol. You send a manifest of record hashes; the server tells you which ones it needs; you send only those records; then you commit. For collections where most records are unchanged between versions, only a few records are transferred.

Step 1: POST /api/collections/:owner/:slug/versions/negotiate

Auth: write scope

Start a negotiate session. Send your full manifest of record hashes plus schemas. The server checks which record and file hashes it already has and returns what it still needs.

Request

{
  "base_version": "v1.0.0",
  "schemas": {
    "Publication": {
      "type": "object",
      "properties": {
        "title": {"type": "string"}
      }
    }
  },
  "manifest": [
    {"id": "pub-001", "type": "Publication", "hash": "abc123..."},
    {"id": "pub-002", "type": "Publication", "hash": "def456..."},
    {"id": "pub-003", "type": "Publication", "hash": "789abc..."}
  ],
  "files": ["7a8b9c..."],
  "message": "Add new publications",
  "metadata": {
    "description": "PubPub archive"
  }
}

Fields

Response 200

{
  "session_id": "uuid",
  "needed_records": ["def456...", "789abc..."],
  "needed_files": [],
  "total_records": 3,
  "total_files": 1,
  "already_have_records": 1,
  "already_have_files": 1
}

Large collections: uploading the manifest in chunks

The manifest above is a single JSON body, which is fine up to 500,000 entries (beyond that the endpoint returns 413). At a few million records it would be hundreds of megabytes, parsed whole. Instead, omit manifest and declare how many records you will send:

{
  "base_version": null,
  "schemas": { "Preprint": { "type": "object", "properties": {...} } },
  "manifest_expected": 3110000,
  "message": "arXiv metadata"
}

The server opens the session without asking for any records yet — nothing in this response is proportional to the collection:

{
  "session_id": "uuid",
  "manifest_expected": 3110000,
  "manifest_received": 0,
  "needed_files": [],
  "total_files": 0,
  "already_have_files": 0,
  "next": "POST .../versions/negotiate/uuid/manifest"
}

Then POST .../versions/negotiate/:sessionId/manifest with up to 50,000 JSONL entries per request (Content-Type: application/x-ndjson), each line a {"id", "type", "hash", "private"?} object. Each response tells you which records from that chunk the server still needs, so you can start sending bodies before the whole manifest is uploaded:

{
  "received": 50000,
  "needed_records": ["def456...", "789abc..."],
  "manifest_received": 150000,
  "manifest_expected": 3110000
}

Chunks are idempotent: entries are keyed by hash, so re-sending a chunk after a timeout is safe and manifest_received will not move. Commit refuses to build a version until manifest_received equals manifest_expected, so a client that dies partway through the upload cannot silently produce a version that dropped records.

The session's 10-minute expiry is an idle timeout — every manifest chunk and record batch pushes it back — so a push that legitimately runs for an hour will not expire underneath you.

Step 2: POST .../negotiate/:sessionId/records

Auth: write scope

Send needed records as a JSONL body (Content-Type: application/x-ndjson). Each line is one JSON record. Only send records whose hashes appear in needed_records from the negotiate response.

Call this endpoint multiple times to send records in batches (up to 10,000 per request). The server tracks which records have been received. If needed_records was empty, skip this step and go directly to commit.

Request

{"id":"pub-002","type":"Publication","data":{"title":"New Paper"}}
{"id":"pub-003","type":"Publication","data":{"title":"Another Paper"}}

Response 200

{
  "received": 2,
  "remaining": 0,
  "total_needed": 2
}

When remaining reaches 0, all needed records have been received and you can commit.

Step 3: POST .../negotiate/:sessionId/commit

Auth: write scope

Finalize the push. The server validates all records against schemas, computes version hashes, and creates the new immutable version. No request body is needed.

Response 201

{
  "semver": "v1.1.0",
  "hash": "private:a1b2c3d4...",
  "recordCount": 3,
  "fileCount": 1
}

Large pushes: async finalize

Commit work is proportional to the size of the collection, so on a very large one it can run for minutes — longer than a proxy or client will hold a request open. Pass ?async=true (or {"async": true} in the body) and the server answers 202 immediately and builds the version in the background:

{
  "session_id": "uuid",
  "status": "committing",
  "message": "Commit accepted. Poll GET .../versions/negotiate/uuid until status is \"committed\" or \"failed\"."
}

Then poll GET .../versions/negotiate/:sessionId until status is committed or failed. On success result holds exactly what the synchronous 201 would have returned; on failure error holds the rejection body it would have returned instead, so the two paths are interchangeable apart from timing.

{
  "session_id": "uuid",
  "status": "committed",
  "total_records": 3110000,
  "needed_records": 0,
  "finalize_started_at": "2026-07-31T12:00:00.000Z",
  "result": {
    "semver": "v1.1.0",
    "hash": "private:a1b2c3d4...",
    "recordCount": 3110000,
    "fileCount": 0
  },
  "error": null
}

The version is not visible to readers until the finalize completes — it is created in a creating state and only published at the end, so there is no window where a half-built version can be read. A finalize whose process dies is swept and marked failed, and its partial version removed.

Privacy

You can add "private": true at two levels in the schema, and at a third on the manifest entry:

• Type-level: Add "private": true to a type definition to hide all records of that type from public readers.
• Field-level: Add "private": true to a field definition to strip that field from records returned to public readers.
• Record-level: Add "private": true to a manifest entry (not the record body) to hide that one record. Unlike type and field privacy — which live in the schema — this is declared per record per push and stored on the version’s reference to the record, so it must be re-declared on every push; omitting it means public. The manifest endpoint echoes private back so a round-trip is lossless.

Redaction is per-version and forward-only: marking a record private in v2 hides it in v2 only — v1 is immutable and still serves it at /versions/v1.0.0/records. File access resolves across every ready version, so a file referenced publicly in v1 stays downloadable after the referencing record is redacted in v2.

"schemas": {
  "Article": {
    "type": "object",
    "properties": {
      "title": {"type": "string"},
      "body": {"type": "string"},
      "internalScore": {"type": "number", "private": true}
    }
  },
  "InternalNote": {
    "type": "object",
    "private": true,
    "properties": {
      "note": {"type": "string"}
    }
  }
}

Session management

Errors

GET /api/collections/:owner/:slug/versions

No auth for public collections

List versions, newest first.

Query parameters

Response 200

[
  {
    "semver": "v1.1.0",
    "hash": "private:a1b2c3d4...",
    "message": "Add new publications",
    "appId": "pubpub-sync",
    "actorId": "user-42",
    "recordCount": 150,
    "fileCount": 12,
    "totalBytes": 52428800,
    "createdAt": "2026-04-01T00:00:00.000Z"
  }
]

GET /api/collections/:owner/:slug/versions/latest

No auth for public collections

Get the most recent version. Returns the full version object.

GET /api/collections/:owner/:slug/versions/:n

No auth for public collections

Get a specific version by semver (e.g. v1.1.0). Returns the full version object including schemas.

GET /api/collections/:owner/:slug/versions/:n/records

No auth for public collections

Get records for a specific version. Supports cursor-based pagination for efficient traversal of large collections.

Query parameters

Walking a whole collection is bounded by request count, not bytes: 60 requests/minute anonymous, 5,000 authenticated. Ask for the largest page you can handle — a 3-million-record collection is 6,200 requests at 500/page and 1,550 at 2,000/page.

Response 200

{
  "records": [
    {
      "id": "pub-001",
      "type": "Publication",
      "data": {
        "title": "Example Paper",
        "doi": "10.1234/example"
      }
    }
  ],
  "pagination": {
    "limit": 100,
    "hasMore": true,
    "nextCursor": "pub-002",
    "total": 150
  }
}

Use pagination.nextCursor as the after parameter in the next request. When hasMore is false, you've reached the end. For large collections, always paginate with after rather than offset.

pagination.total respects the type filter and excludes private types. On collections that mark individual records private it is an upper bound for anonymous callers, since those records are hidden but still counted — use hasMore if you need an exact end-of-set signal.

GET /api/collections/:owner/:slug/versions/:n/records.ndjson

No auth for public collections

Every record in the version, streamed as newline-delimited JSON in a single response. This is the bulk read path. Paging /records costs a round trip per page purely to re-establish a cursor the server just had — 1,556 requests for a 3.1-million-record collection, against one here. The server reads through a database cursor and writes as it goes, so memory stays constant on both ends and you can process the first line before the last is sent.

Query parameters

Response 200

HTTP/1.1 200 OK
Content-Type: application/x-ndjson
X-Underlay-Record-Count: 3113504

{"id":"pub-001","type":"Publication","data":{"title":"..."},"hash":"sha256:..."}
{"id":"pub-002","type":"Publication","data":{"title":"..."},"hash":"sha256:..."}
{"id":"pub-003","type":"Publication","data":{"title":"..."},"hash":"sha256:..."}

hash is the same content-address /records serves: the full record hash for owners, the public hash for everyone else. Privacy filtering is identical too — private types and private records are absent, private fields stripped.

Check completeness yourself. A stream that fails partway cannot report it: the 200 and headers were sent before anything went wrong. X-Underlay-Record-Count tells you how many lines to expect — the count for this request, privacy-filtered for your access level and scoped to ?type= if you passed one, so the comparison is exact. (Don’t compare against the version’s recordCount: that is the full total and counts private records you may not be receiving.) If you receive fewer, resume with ?after= set to the id of the last complete line you parsed — don't start over.

A record id is not guaranteed unique within a version — the same id can appear under more than one hash. Because after resumes strictly past the id, a stream that broke between two lines sharing an id will skip the second on resume. This matches /records paging, and the line-count check above is what catches it.

Responses are compressed when you send Accept-Encoding: gzip, which most HTTP clients do automatically — roughly 3× on record data, and it applies to this stream as well.

GET /api/collections/:owner/:slug/versions/:n/manifest

No auth for public collections

Get the manifest: every record's id, type and content hash, without the bodies. This is the cheapest way to learn what a version contains — at roughly 120 bytes per entry, a million records is one order of magnitude smaller than fetching them.

Query parameters

Response 200

{
  "semver": "v1.1.0",
  "hash": "private:a1b2c3d4...",
  "schemas": {"Publication": "abc123..."},
  "records": [
    {"id": "pub-001", "type": "Publication", "hash": "def456..."},
    {"id": "pub-002", "type": "Publication", "hash": "789abc..."}
  ],
  "files": ["a1b2c3...", "d4e5f6..."],
  "pagination": {
    "limit": 10000,
    "hasMore": true,
    "nextCursor": "eyJhZGRlZCI6WyJwdWItMDAyIiwiZGVmNDU2Il0..."
  }
}

Response with ?since= 200

{
  "semver": "v1.1.0",
  "hash": "private:a1b2c3d4...",
  "since": "v1.0.0",
  "schemas": {"Publication": "abc123..."},
  "delta": {
    "added":   [{"id": "pub-003", "type": "Publication", "hash": "def456..."}],
    "updated": [{"id": "pub-001", "type": "Publication", "hash": "def456...",
                 "previousHash": "abc123..."}],
    "removed": [{"id": "pub-old", "type": "Publication", "hash": "def456..."}]
  },
  "files": ["a1b2c3..."],
  "pagination": {
    "limit": 10000,
    "hasMore": false,
    "nextCursor": null
  },
  "truncated": false
}

A delta of any size can be walked to completion: keep re-requesting with cursor=pagination.nextCursor until hasMore is false. The three lists drain independently and the cursor tracks each one, so a page late in the walk may contain only updated entries.

truncated is retained for older clients, which treated a capped delta as "give up and rebuild from the full manifest". It now simply mirrors pagination.hasMore. Clients that understand the cursor should page instead of rebuilding.

GET /api/collections/:owner/:slug/versions/:n/diff

No auth for public collections

Diff two versions. By default compares version :n against the previous version.

Query parameters

Response 200

{
  "from": "v1.0.0",
  "to": "v1.1.0",
  "added": [
    {"id": "pub-003", "type": "Publication", "data": {...}}
  ],
  "updated": [
    {"id": "pub-001", "type": "Publication", "data": {...}}
  ],
  "removed": ["pub-old"],
  "pagination": {
    "limit": 500,
    "hasMore": false,
    "nextCursor": null
  },
  "meta": {
    "schemaChanged": false,
    "metadataChanged": false,
    "filesAdded": 0,
    "filesRemoved": 0
  }
}