Versions API

Versions are the core of Underlay. Each version is an immutable snapshot of a collection: schemas, records and file references. You publish one with a delta push: open a session against the version you started from, upload the records you add or change and the ids you delete, and commit.

Version ids are ulv3:<sha256>, the SHA-256 of the version’s commit, and are the same for every reader. Versions published before Underlay moved to protocol v3, on November 13, 2026, also answer to the ulv2: ids they had then, anywhere a version id is accepted; responses always give the ulv3: id. Wherever a version is named in a path (:n), it can be a semver (v1.1.0), a version id, or latest. Record, schema and file hashes are bare hex with no prefix.

A version never changes, so responses under .../versions/:n are cached when :n is a semver or hash: public, max-age=600, stale-while-revalidate=60 for anonymous readers, private, max-age=3600 for signed-in ones. latest moves, so it isn’t cached. Only 200s are cached.


Delta push (open → upload → commit)

A push costs work in proportion to what changed, not to the size of the collection. Only the records you send are validated and hashed; unchanged records are never re-sent.

POST /api/collections/:owner/:slug/push

Auth: write scope

Open a session. Every field is optional.

Request

{
  "base": "v1.0.0",
  "schemas": {
    "Publication": {
      "type": "object",
      "properties": {
        "title": {"type": "string"},
        "pdf": {"type": "object"}
      }
    }
  },
  "metadata_patch": {"description": "PubPub archive"},
  "files": {"add": ["7a8b9c..."]},
  "message": "Add new publications"
}

Fields

baseThe semver you started from. If it isn’t the collection’s head, the answer is 409 with currentVersion. null or absent applies the changes to whatever the head is (use null for the first version).
schemasThe full type set, {"TypeName": schema}. It replaces the base’s: a type left out is removed with its records. Absent keeps the base’s types. Every schema of the resulting type set, kept ones included, is checked in full when the session opens (the draft-07 meta-schema, patterns, $refs that resolve, and the schema rules); one that fails is 422. So is a type name that begins with $, a space, !, " or #, or is __proto__, and, in a public collection, a schema marked "private": true (see Privacy).
metadata / metadata_patchmetadata replaces the version metadata (description, readme, license, …) and must be an object or null; metadata_patch, an object, merges its top-level members into the base’s. Anything else is 400. With neither, the metadata is kept.
files{"add": [hash, …], "remove": [hash, …]}: files to declare or drop beyond those your records reference. Each hash is bare 64-character lowercase hex. Any other string, including one with a sha256: prefix, is ignored without an error.
message, app_id, actor_idStrings recorded with the version. message is part of its commit; all three appear in the version list (actorId to members only).
strip_unknown_fieldsIf true, top-level fields a record’s schema doesn’t list in properties are dropped before the record is hashed, instead of refusing the record.

Response 200

{
  "session_id": "uuid",
  "base": "v1.0.0",
  "needed_files": ["7a8b9c..."],
  "expires_at": "2026-10-05T00:00:00.000Z",
  "limits": {
    "open_bytes": 8388608,
    "batch_bytes": 8388608,
    "batch_lines": 10000,
    "session_idle_seconds": 43200,
    "open_sessions": 20,
    "file_bytes": 33554432
  }
}

needed_files are the declared files this collection doesn’t hold yet (files already uploaded and verified for it are held): upload them before you commit. limits are this node’s; size your batches by them. The session expires after session_idle_seconds without an upload. Each records or deletes batch pushes expires_at back; file uploads don’t.

POST .../push/:sid/records

Auth: write scope

Upserts as NDJSON (Content-Type: application/x-ndjson), one record per line: {"id", "type", "data"}. Call it as many times as you need, up to batch_lines lines and batch_bytes bytes each.

Request

{"id":"pub-002","type":"Publication","data":{"title":"New Paper"}}
{"id":"pub-003","type":"Publication","data":{"title":"Draft"}}

Response 200

{ "received": 2 }

Each line passes the input rules and its type’s schema, and its type must be in the session’s type set. If any line fails, the answer is 422 and nothing from the batch is stored. validationErrors lists the first 100 failing lines, each with its 1-based line number; totalErrors is the full count. In a public collection, a batch with any line marked "private": true is also 422 (see Privacy).

{
  "error": "Invalid records",
  "validationErrors": [
    {"line": 2, "recordId": "pub-003", "type": "Publication",
     "errors": ["..."]}
  ],
  "totalErrors": 1,
  "statusCode": 422
}

A batch with no lines is 400. A session that is no longer open is 409 ("Session is not open").

POST .../push/:sid/deletes

Auth: write scope

Deletes as NDJSON, one {"type", "id"} per line; the answer is { "received": n }. Deleting an id the base doesn’t hold is not an error. Within a session the later upload of a (type, id) wins, whether it is a record or a delete.

{"type":"Publication","id":"pub-old"}

Each line passes the input rules, as a record line does, and its id and type the same checks. A line that fails them, isn’t {"type", "id"}, or names a type not in the session’s type set fails the batch: 422, with validationErrors (the first 100 failing lines) and totalErrors as for records, and nothing from the batch is stored. An empty batch is 400; a session that is no longer open is 409.

PUT .../files/:hash

Auth: write scope

Upload a file’s bytes under its SHA-256 hash: 201, or 400 if the bytes don’t match the hash. Files over file_bytes are a 413; upload those through the Files API. Every file a new record references must be uploaded before the commit.

POST .../push/:sid/commit

Auth: write scope

Build the version. No body is needed. Two fields are read: "async": true (below), and "rebase": true, which commits on the head as it is now instead of the session’s base: the session’s records and deletes apply on top of whatever landed since, and its schemas and metadata replace the head’s. Use it only when your changes don’t depend on the base, such as a load of upserts.

Response 201

{
  "semver": "v1.1.0",
  "hash": "ulv3:a1b2c3d4...",
  "recordCount": 3,
  "fileCount": 1,
  "changes": {"added": 2, "removed": 1, "updated": 0}
}

recordCount and fileCount count the whole version, not only what this push changed; changes counts the records this push added, removed and updated.

The semver follows from what changed against the base: major when a type was added or removed or a schema changed, otherwise minor when any record was added, removed or changed, otherwise patch (metadata or files only). A push that changes nothing makes no version and answers 409.

Committing a session that is already committed answers its 201 result again. A session whose commit failed on the server’s side (a 500 or 503) can be committed again; one refused for its content is 409, with the refusal as cause: push again in a new session. A session in any other state that isn’t open is 409 ("Session is <status>").

With ?async=true, ?async=1 or a body of {"async": true}, the answer is 202 and the version is built in the background. The commit runs this way without being asked when the session uploaded more than 100,000 records, or when a schema change means more than 100,000 of the base’s records must be revalidated.

{
  "session_id": "uuid",
  "status": "committing",
  "poll": "GET /api/collections/:owner/:slug/push/uuid"
}

A synchronous commit can also answer 202, with {"session_id", "status": "committing"} and no poll, when the server finishes a large commit in background jobs. Treat both the same way: poll GET .../push/:sid until status is committed (result is the 201 body) or failed (error is the rejection), or back to open with error after a "Missing files" refusal. The version isn’t visible to readers until it is committed.

A commit can be refused:

  • 409 {"error": "Version conflict", "currentVersion"}: the head moved after the session opened; currentVersion is the head now. Read it and push again.
  • 409 "No changes detected", with the head’s hash.
  • 422 "Missing files": filesNeeded lists up to 100 files not uploaded, as bare hex. The session stays open: upload them and commit the same session again.
  • 422 "Schema validation failed", with validationErrors and totalErrors: a schema change makes records of the base invalid.

Clients that keep no copy

A client that exports its whole dataset each time, rather than tracking changes, reads the head’s manifest, diffs against it, and pushes only the differences. The upload is the size of the changes, whatever the size of the collection.

# 1. What the head holds: page the manifest
GET .../versions/latest/manifest?limit=25000           # then ?cursor=<nextCursor>
# 2. Compare by (type, id): hash your current records (canonical form, SHA-256)
#    new, or changed hash                 -> upsert
#    in the manifest but not in your data -> delete
# 3. Push only those, against the manifest's semver
POST .../push  {"base": "<manifest semver>"}
POST .../push/:sid/records   ...
POST .../push/:sid/deletes   ...
POST .../push/:sid/commit

Privacy

A collection is public or private as a whole. There are no private records or private types: to keep data from the public, push it to a private collection, which only members of the owning organization can read.

  • Public collection: a session whose schemas mark a type "private": true is refused when it opens, and a records batch with a line marked "private": true is refused, both with 422: "This collection is public, and keeps no private records. Make the collection private, or push them to a private collection." (for a type, the message names it).
  • Private collection: the mark is accepted and changes nothing, since every record is already private.

"private": true on a property inside a schema (field-level privacy) is refused in either kind of collection.

Errors

400A body that isn’t a JSON object, metadata that isn’t an object or null, metadata_patch that isn’t an object, rebase that isn’t a boolean, an empty records or deletes batch, or file bytes that don’t match their hash.
401You aren’t signed in.
403You can read the collection but not write to it, or the session is another user’s.
404Collection or session not found, or not visible to you.
409base isn’t the head, or the head moved before the commit (both with currentVersion, the head now); the session isn’t open (or failed in a way another commit can’t fix); or the push changes nothing ("No changes detected").
413A body over open_bytes or batch_bytes, a batch over batch_lines, or a file over file_bytes.
422Records or deletes that fail (validationErrors), a line whose type isn’t in the session’s type set, a schema or type name that is refused (at open), a type or record marked private in a public collection, a commit with files not uploaded (filesNeeded), or a schema change that base records fail ("Schema validation failed").
429You have open_sessions sessions in progress (commit or abandon one), or a rate limit (Retry-After).
503Storage cleanup held up a file upload; retry after Retry-After.

GET /api/collections/:owner/:slug/push/:sid

Auth: write scope; your own sessions only

A push session’s status. Poll it after a commit answers 202.

Response 200

{
  "session_id": "uuid",
  "status": "committed",
  "records_received": 3110000,
  "expires_at": "2026-10-05T00:19:00.000Z",
  "created_at": "2026-10-04T12:00:00.000Z",
  "finalize_started_at": "2026-10-04T12:20:00.000Z",
  "result": {
    "semver": "v1.1.0",
    "hash": "ulv3:a1b2c3d4...",
    "recordCount": 3110000,
    "fileCount": 0,
    "changes": {"added": 3110000, "removed": 0, "updated": 0}
  },
  "error": null
}
statusopen (taking uploads; error holds a "Missing files" refusal, if the last commit had one), committing, committed, failed, or expired (abandoned, or idle too long).
records_receivedRecord lines accepted so far, across batches.
finalize_started_atWhen the commit began, or null.
resultOnce committed, the commit’s 201 body; otherwise null.
errorOnce failed, the rejection body (for example {"error": "Version conflict", "statusCode": 409}); otherwise null.
progressOnly while open or committing: the uploaded runs and, once the commit has begun, its phase.

DELETE /api/collections/:owner/:slug/push/:sid

Auth: write scope; your own sessions only

Abandon an open session: it becomes expired and stops counting toward open_sessions. The answer is {"ok": true}. A session that isn’t open is left as it is.


GET /api/collections/:owner/:slug/versions

No auth for public collections

List versions, newest first.

Query parameters

limitMax results (default 50, max 100)
offsetPagination offset

Response 200

[
  {
    "semver": "v1.1.0",
    "major": 1,
    "minor": 1,
    "patch": 0,
    "hash": "ulv3:a1b2c3d4...",
    "baseSemver": "v1.0.0",
    "message": "Add new publications",
    "appId": "pubpub-sync",
    "pushedBy": "user-42",
    "pushedByName": "Ada Lovelace",
    "actorId": "user-42",
    "recordCount": 150,
    "fileCount": 12,
    "totalBytes": 52428800,
    "typeCounts": {"Publication": 150},
    "createdAt": "2026-04-01T00:00:00.000Z",
    "ark": "https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0.v1.1.0"
  }
]

recordCount, fileCount, totalBytes and typeCounts count the whole version. baseSemver is the version before it (null for the first). pushedBy, pushedByName and actorId are for the collection’s members only. ark is null when the collection’s ARK is off.


GET /api/collections/:owner/:slug/versions/latest

No auth for public collections

Get the most recent version. Returns the same object as .../versions/:n. Not cached.


GET /api/collections/:owner/:slug/versions/:n

No auth for public collections

Get a specific version by semver (e.g. v1.1.0) or version id. Returns the version object as listed above, plus metadata and the schemas of its types.

With ?records=<type> (empty for the first type), it also returns recordsPage, {"type", "records", "total"}: a page of that type’s records (offset, limit as on /records) and the type’s total. schemas then holds only that type’s schema; typeCounts still lists every type. The records page uses this to need only one call. Where /records would answer 503 for a deep offset (see below), recordsPage is empty and has "preparing": true.


GET /api/collections/:owner/:slug/versions/:n/records

No auth for public collections

Get records for a specific version, in (type, id) order: types in slug order (UTF-8 byte order), then ids within a type.

Query parameters

typeFilter by record type
limitMax results (default 100, max 2000)
afterOpaque keyset cursor from pagination.nextCursor; it names the (type, id) of the last record returned. Ids are unique within a type, so nothing is skipped at a page boundary. cursor is an alias. A bare record id is accepted only with ?type=; without it, the id is ignored and paging starts at the beginning.
offsetSkip this many records. Ignored when after is given. With type, a deep offset starts from an index of the type’s positions, which the server builds the first time one is asked for. An offset past 250,000 records whose index isn’t ready yet is 503 with Retry-After: 60. Paging by after never waits.

Walking a whole collection is bounded by request count, not bytes: 60 rate units a minute anonymous, 5,000 signed in, and a page costs 1. Ask for the largest page you can handle: a 3-million-record collection is 6,000 requests at 500 per page and 1,500 at 2,000 per page. A page also ends before 4 MiB of records, so large records make smaller pages. records.ndjson reads it in one request for 10 units.

Response 200

{
  "records": [
    {
      "id": "pub-001",
      "type": "Publication",
      "data": {
        "title": "Example Paper",
        "doi": "10.1234/example"
      },
      "hash": "def456..."
    },
    {
      "id": "pub-002",
      "type": "Publication",
      "data": {"title": "Draft"},
      "hash": "789abc..."
    }
  ],
  "pagination": {
    "limit": 2,
    "hasMore": true,
    "nextCursor": "eyJ0IjoiUHVibGljYXRpb24iLCJrIjoicHViLTAwMiJ9",
    "total": 150
  }
}

Each record carries its hash.

Use pagination.nextCursor as the after parameter in the next request, unchanged: treat it as opaque. When hasMore is false, you’ve reached the end.

pagination.total is the exact count of records in the version, within type if given. The one exception: records withheld from serving are counted but skipped.


GET /api/collections/:owner/:slug/versions/:n/records/:type/:id

No auth for public collections

One record at a version. A record that doesn’t exist, or is withheld from serving, is 404.

Response 200

{
  "id": "pub-001",
  "type": "Publication",
  "data": {"title": "Example Paper"},
  "hash": "def456...",
  "semver": "v1.1.0"
}

GET /api/collections/:owner/:slug/records/:type/:id/history

No auth for public collections

A record’s history: each version where it was added, updated or removed, oldest first. hash is the record’s hash in that version, null when removed. It covers every version of the collection, so truncated is always false. A record never seen is 404. It costs 5 rate units.

Response 200

{
  "type": "Publication",
  "id": "pub-001",
  "changes": [
    {"seq": 1, "semver": "v1.0.0", "createdAt": "2026-03-01T00:00:00.000Z",
     "change": "added", "hash": "abc123..."},
    {"seq": 2, "semver": "v1.1.0", "createdAt": "2026-04-01T00:00:00.000Z",
     "change": "updated", "hash": "def456..."}
  ],
  "truncated": false
}

GET /api/collections/:owner/:slug/versions/:n/records.ndjson

No auth for public collections

Every record in the version, streamed as newline-delimited JSON in a single response. This is the bulk read path: one request, costing 10 rate units, against 1,500 pages for a 3-million-record collection. The server streams records as it reads them, so memory stays constant on both ends and you can process the first line before the last is sent.

Lines come in the same order as /records: types in slug order, then ids.

Query parameters

typeRestrict to a single record type
afterResume after the record with this id. With after_type, the stream continues after (after_type, after) through the later types; with type, it stays within that type. With neither, it is 400.
after_typeThe type of the after record. Without after it is 400.

Response 200

HTTP/1.1 200 OK
Content-Type: application/x-ndjson
X-Underlay-Record-Count: 3110000

{"id":"pub-001","type":"Publication","data":{"title":"..."},"hash":"def456..."}
{"id":"pub-002","type":"Publication","data":{"title":"..."},"hash":"789abc..."}
{"id":"pub-003","type":"Publication","data":{"title":"..."},"hash":"a1b2c3..."}

hash is the same content address /records serves.

Check completeness yourself. A stream that fails partway cannot report it: the 200 and headers were sent before anything went wrong. X-Underlay-Record-Count tells you how many lines to expect: the count for this request, within ?type= if you passed one, and after the resume point if you gave one. It can exceed the lines sent only by records withheld from serving. (Don’t compare against the version’s recordCount: it covers every type.)

If you receive fewer, resume rather than start over: take the type and id of the last complete line you parsed and request ?after_type=<type>&after=<id>. The stream picks up after that record and runs to the end of the version.


GET /api/collections/:owner/:slug/versions/:n/records.ndjson.gz

No auth for public collections

The version’s records as a gzip file (Content-Type: application/gzip, sent as an attachment). With ?type=, one type’s; without it, every type’s in slug order. The file is several gzip members concatenated, one about every megabyte of records. Most gzip readers handle that; some, such as browsers’ DecompressionStream, stop after the first member.

Lines are canonical records, {"id", "type", "data"}, without hash (it is the SHA-256 of the line). A type with no records is 404. It costs 10 rate units.


GET /api/collections/:owner/:slug/versions/:n/manifest

No auth for public collections

Get the manifest: every record’s id, type and content hash, without the bodies, in (type, id) order. This is the cheapest way to learn what a version contains (at roughly 120 bytes per entry, a million records is an order of magnitude smaller than fetching them), and what a client that keeps no copy diffs against before it pushes.

Query parameters

limitEntries per page (default 10000, max 25000). The first page also lists the version’s files, at most 25,000, with filesTruncated: true past that; later pages have files: [].
cursorOpaque keyset cursor from pagination.nextCursor. Do not construct or parse it: pass back exactly what you were given.
sinceA semver or version id, at or before :n (a later one is 400). Return a delta against that version instead of the full manifest: which records were added, updated and removed between the two.

Response 200

{
  "semver": "v1.1.0",
  "hash": "ulv3:a1b2c3d4...",
  "schemas": {"Publication": "abc123..."},
  "records": [
    {"id": "pub-001", "type": "Publication", "hash": "def456..."},
    {"id": "pub-002", "type": "Publication", "hash": "789abc..."}
  ],
  "files": ["a1b2c3...", "d4e5f6..."],
  "pagination": {
    "limit": 2,
    "hasMore": true,
    "nextCursor": "eyJ0IjoiUHVibGljYXRpb24iLCJrIjoicHViLTAwMiJ9"
  }
}

Response with ?since= 200

{
  "semver": "v1.1.0",
  "hash": "ulv3:a1b2c3d4...",
  "since": "v1.0.0",
  "schemas": {"Publication": "abc123..."},
  "delta": {
    "added":   [{"id": "pub-003", "type": "Publication", "hash": "def456..."}],
    "updated": [{"id": "pub-001", "type": "Publication", "hash": "def456...",
                 "previousHash": "abc123..."}],
    "removed": [{"id": "pub-old", "type": "Publication", "hash": "def456..."}]
  },
  "files": ["a1b2c3..."],
  "pagination": {
    "limit": 10000,
    "hasMore": false,
    "nextCursor": null
  }
}

A delta is one walk over both versions in (type, id) order, with a single cursor; limit counts entries across the three lists. Keep re-requesting with cursor=pagination.nextCursor until hasMore is false.

files in a delta is the full file list of :n (first page only), not a file delta.


GET /api/collections/:owner/:slug/versions/:n/diff

No auth for public collections

Diff two versions, with full record bodies. Without from, the diff is against the version before :n, which the response’s from names. The first version has none before it, so its diff is against an empty version: every record is added, from is null and schemaChanged is true. It costs 5 rate units.

Query parameters

fromSemver or version id to diff from (e.g. v1.0.0), at or before :n (a later one is 400). Default: the version before :n.
limitEntries per page, across the three lists (default 500, max 5000)
cursorOpaque keyset cursor from pagination.nextCursor, as on the manifest endpoint. Diff returns full record bodies, so pages are much larger than manifest pages: prefer manifest?since= when you only need the hashes.

Response 200

{
  "from": "v1.0.0",
  "to": "v1.1.0",
  "added": [
    {"id": "pub-003", "type": "Publication", "data": {...}}
  ],
  "updated": [
    {"id": "pub-001", "type": "Publication", "data": {...}}
  ],
  "removed": [{"id": "pub-old", "type": "Publication"}],
  "pagination": {
    "limit": 500,
    "hasMore": false,
    "nextCursor": null
  },
  "meta": {
    "schemaChanged": false,
    "metadataChanged": false,
    "filesAdded": 0,
    "filesRemoved": 0
  }
}

removed entries are {"id", "type"}, without bodies. A page also ends before 4 MiB of record bodies, so it can hold fewer entries than limit. meta describes the whole diff on every page.


GET /api/collections/:owner/:slug/versions/:n/files

No auth for public collections

The version’s files, sorted by hash, at most 10,000. Which records reference a file isn’t indexed, so referenceCount is always 0 and references always empty. createdAt is when the file was first uploaded, or null.

Response 200

[
  {
    "hash": "a1b2c3...",
    "size": 1048576,
    "mimeType": "application/pdf",
    "createdAt": "2026-04-01T00:00:00.000Z",
    "referenceCount": 0,
    "references": []
  }
]