Protocol v3 · underlay.org today

Push and pull

Delta push

On underlay.org you make a commit through a push session: open it against the head, upload the records you add or change and the keys you remove, and commit. The server turns the session into rows, builds the sorted-record-files, and makes an unsigned change whose base is the head’s version id. It then lands the change as the next commit, signs the commit, and assigns the semver (semver rules). A push session is the only way to commit today: uploads of files a client builds itself are not available yet.

This page describes the reference server, not the protocol. Paths are relative to the collection URL, https://www.underlay.org/api/collections/<owner>/<slug>. The Versions API and Files API give the full request and response shapes.

POST   <collection>/push                  open a session
POST   <collection>/push/<sid>/records    upload records (NDJSON)
POST   <collection>/push/<sid>/deletes    upload deletes (NDJSON)
PUT    <collection>/files/<sha256>        upload a file
POST   <collection>/push/<sid>/commit     commit (?async=true for the background)
GET    <collection>/push/<sid>            session status
DELETE <collection>/push/<sid>            abandon the session

Opening a session

The body is a JSON object; each member is optional.

MemberMeaning
baseThe semver of the version the changes are against. If it is not the head, the server answers 409 with currentVersion. null or absent: the head at opening.
schemasThe complete type set, type → schema. Replaces the head’s; an omitted type is removed with its records. Absent: the head’s type set.
metadataReplaces the head’s metadata. An object or null.
metadata_patchAn object whose top-level members are merged into the head’s metadata. Ignored if metadata is present.
files{"add": [sha256, …], "remove": [sha256, …]}: files declared or removed beyond those records refer to.
messageThe change’s message.
app_id, actor_idStrings the server records with the version, outside the commit.
strip_unknown_fieldsBoolean; see Records and deletes.

Each type MUST be a valid v3 type (see Reserved types), and each schema MUST pass the schema rules. A public collection refuses a schema with a root "private": true.

The response is 200 with session_id; base (the semver the session is against, or null); needed_files, the declared files the collection does not yet hold, which the client MUST upload before committing; expires_at, extended by each records or deletes request; and limits, by which a client sizes its requests: open_bytes, batch_bytes, batch_lines, session_idle_seconds, open_sessions (per user, open or committing), and file_bytes.

Records and deletes

  • …/records takes NDJSON record lines, {"id", "type", "data"}, and answers {"received": n}. Each line MUST satisfy the input rules and its type’s schema, and its type MUST be in the session’s type set.
  • The envelope is v2’s: other members are ignored, and a private member, if present, MUST be a boolean. In a public collection a line with "private": true is refused (422): make the collection private, or push to a private collection. In a private collection the mark changes nothing.
  • A record whose data has top-level members absent from its schema’s root properties is refused, unless the session set strip_unknown_fields, in which case they are removed before hashing. A schema without root properties admits any members.
  • If any line fails, the server answers 422 with validationErrors, one per failing line with its 1-based line, and stores nothing from the request. It may list only the first failures, with totalErrors giving their number.
  • …/deletes takes NDJSON {"type", "id"} lines and answers {"received": n}. The type MUST be in the session’s type set. Deleting a key the head does not hold is not an error. A request with no lines is a 400.
  • Within a session, the later upload of a key supersedes an earlier one, record or delete.

Files

PUT …/files/<sha256> stores a file for the collection: 201, or 400 if the bytes do not hash to <sha256>. Larger files go through the upload routes of the Files API. A file counts as held only if the head names it in a $asset row or it was uploaded to this collection. A file held for another collection is not held, and each upload is hashed, even of a file the server already stores.

What a commit holds

At commit the server builds these rows, in key order:

  • a $schema row for each type whose schema the session adds or changes, and a removal of the $schema row and of each record for each type it drops;
  • a $meta row if the metadata differs from the head’s (a removal if it is now null);
  • a $asset row, {"size", "mediaType"}, for each file the session declares or its records refer to that the head does not name, and a removal for each file in files.remove that the head names and the session’s records do not refer to;
  • the session’s records and removals, the last upload of each key winning.

Each row is compared with the head, and a row that changes nothing (the same record, or the removal of one that is not there) is left out. If none is left, the commit is refused with 409 and the head’s version id. Otherwise the semver is major if a schema changed, minor if a record changed, and patch for metadata or assets alone.

Commit

  • …/commit answers 201 with the version, or 202 when it runs in the background. A client MAY request the background with ?async=true; the server chooses it for large commits. The client then polls GET …/push/<sid> until status is committed (result holds the 201 body) or failed (error holds the failure).
  • The commit is made against the head the session opened on. If another commit landed since, the commit answers 409 with currentVersion, and the client pushes again. A commit body of {"rebase": true} commits on the head as it is now instead: the session’s records and deletes apply on top of whatever landed since, and its schemas and metadata replace the head’s.
  • A commit refused because files are not held (422 with filesNeeded) leaves the session open: the client uploads the files and commits the same session again. A polling client sees status open, with the refusal in error.
  • Committing a session that has already committed answers 201 with the same body.
  • hash in the response is the new version’s ulv3: id.
POST <collection>/push
{"base": "v1.2.0", "schemas": {"Publication": {...}}, "metadata_patch": {"readme": "..."},
 "files": {"add": ["9f86d0..."]}, "message": "Weekly update"}
→ 200 {"session_id": "...", "base": "v1.2.0", "needed_files": ["9f86d0..."],
       "expires_at": "...", "limits": {"open_bytes": ..., "batch_bytes": ..., "batch_lines": ...,
       "session_idle_seconds": ..., "open_sessions": ..., "file_bytes": ...}}

PUT <collection>/files/9f86d0...                       (file bytes)
→ 201

POST <collection>/push/<sid>/records
{"id":"pub-004","type":"Publication","data":{"title":"Notes","pdf":{"$file":"sha256:9f86d0..."}}}
{"id":"pub-005","type":"Publication","data":{"title":"Draft"}}
→ 200 {"received": 2}

POST <collection>/push/<sid>/deletes
{"type":"Publication","id":"pub-003"}
→ 200 {"received": 1}

POST <collection>/push/<sid>/commit
→ 201 {"semver": "v1.3.0", "hash": "ulv3:...", "recordCount": 4, "fileCount": 1,
       "changes": {"added": 2, "removed": 1, "updated": 0}}
→ 202 {"session_id": "...", "status": "committing"}         (background commit)

Clients without a copy

Informative. A client that keeps no copy of the collection reads the head’s manifest, GET …/versions/latest/manifest, which lists each record’s id, type, and hash in key order. It computes the hash of each of its own records (the SHA-256 of the record line), uploads the records that are new or changed, and deletes the keys it no longer has. The upload is then proportional to the changes. If another commit lands first, the session or commit answers 409 and the client repeats the comparison against the new head.

Pull

Copying a collection by its log and packs is not available yet: GET …/log and GET …/versions/<v>/pack answer 501 with code not_on_v3. To copy a version, read GET …/versions/<v>/records.ndjson.gz. To follow changes, read the manifest with ?since=<version>, which lists the records added, updated, and removed after that version, or the diff between two versions (Versions API).

Errors

Errors carry {"error": <message>}. Content the caller may not read is 404, never 403.

  • 403: the caller may read the collection but not push to it, or the session belongs to another user.
  • 400: a malformed body, a metadata_patch that is not an object, a metadata that is neither an object nor null, or a request with no lines.
  • 409: base is not the head (with currentVersion); the head changed before the commit; the session is not open; or the push changes nothing (with the head’s hash).
  • 413: a body over open_bytes or batch_bytes, a request over batch_lines, or a file over file_bytes.
  • 422: records or deletes that fail; a refused schema or type; a private mark in a public collection; records carried over from the head that fail a changed schema (validationErrors); or a commit lacking files (filesNeeded).
  • 429: open_sessions sessions already in progress, or a rate limit (with Retry-After).
  • 501: a route not available for v3 collections yet (code not_on_v3).

Security considerations

  • A server MUST NOT accept a change that names a file or asset unless its bytes came with the upload or the collection already holds it (Naming a file). On underlay.org records are always uploaded in full, and a file counts only if it is held for the collection.
  • Who may read is decided per collection. A private collection answers 404 to a caller who may not read it, as if it did not exist.
  • A version id does not prove that a server shows all readers the same versions. Commits chain through previous and the server signs each one, but the log cannot be read over HTTP yet.
  • Open issue: how a reader obtains and trusts the server’s signing key is not specified.
  • File bytes are served from an origin separate from the server’s pages, with Content-Disposition: attachment, so that an uploaded HTML or SVG file cannot run in the server’s origin.