Protocol v3 · Draft

The Underlay protocol

Version 3 of the Underlay protocol defines four things: the sorted-record-file, which holds records in key order; the commit, which names the files one upload added; the layout file, which lists the files that hold a collection after a rebuild; and the collection that holds them. A collection is a log of numbered commits, and a version is the log up to one commit.

It does not define how a server keeps these objects, when it rebuilds, whether it stores identical files once, who may read a collection, or how it bills. Those are choices of each server. Records, canonical JSON, the input rules, key order, and schemas are those of version 2 unless version 3 changes them.

These pages restate the specification, docs/protocol-v3.md, section by section. It is a draft: values and formats may still change, and a change to the file format changes file ids. Where these pages and the specification differ, the specification is authoritative.

Collections on underlay.org moved from protocol v2 to v3 on 2026-11-13. Their versions kept their semvers, and their old ulv2: version ids still resolve wherever the API takes a version id. Responses give the ulv3: id.

Conformance

The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as described in BCP 14 (RFC 2119, RFC 8174) when they appear in all capitals. Notes, examples, and sections marked informative are not normative.

A conforming writer produces, from the same lines and entries, the same footer content and so the same file id, whatever compressor, compression level, or write buffering it uses. A conforming reader accepts and refuses the same files, commits, and layouts as the specification. Version ids rest on commits and on the unlabeled files they name. A change to a line, an unlabeled file, or a commit means a second format beside the first.

Terminology

  • Record: an id, a type, and data (any JSON value). Its key is (type, id).
  • Key order: by type, then by id, each compared as UTF-8 bytes.
  • Sorted-record-file: an immutable file of record lines in key order with a table of contents, named by its file id.
  • Commit: a numbered object naming the sorted-record-files one upload added.
  • Version: the commits of a collection up to and including one commit. Its version id is ulv3: followed by the SHA-256 of that commit’s bytes.
  • Rebuild: writing new sorted-record-files that hold the rows of older ones.
  • Layout file: a numbered object, written by a rebuild, listing the sorted-record-files that hold everything through a commit.
  • Asset: a byte string named by its SHA-256.
  • Collection: a description, its commits, its layouts, and the files and assets those name.
  • Server: an HTTP service that keeps collections, serves them, and accepts uploads. It creates commits and layouts.

Properties

Informative. The rules have these consequences:

  • A sorted-record-file depends only on its rows, so the same rows give the same file id on any machine.
  • Creating a commit reads only the commit before it, so it costs the same whatever the collection holds. Checking an upload reads the schema rows it touches, and the records of a type whose schema it tightens.
  • A version id rests on its commit, and through previous on the commits before it and the files they name. Layout files and rebuilt files carry no version id, so a server can rebuild without changing any version.
  • A reader checks what it reads against the id that names it: a footer against the file id, a key block against the footer, a line against its entry.
  • Who may read a collection is not part of the protocol. On underlay.org a collection is public or private as a whole: there are no private records and no private types.

Version 3 has no single hash over a version’s records (a content root) and no proof that a record is in a version. Neither is available yet.

Contents

  • Records and schemas (§2; v2 §§2–5, 7): canonical JSON, input rules, record and removal lines, reserved types, key order, schemas, validation, and assets.
  • Files and versions (§§3, 4, 6): the sorted-record-file, commits, version ids, base, signatures, naming a file, semver, and reading a version.
  • Layouts and collections (§§5, 7–9): layout files, rebuilds, collections and forks, and the reference server’s storage.
  • Push and pull: how a commit is made on underlay.org through the HTTP API today, how to read a version back, errors, and security considerations.

Not available yet

  • Copying a collection by its log and packs: GET …/log and GET …/versions/:v/pack answer 501 with code not_on_v3.
  • Uploads of sorted-record-files built by a client. Today the server builds the files from a push session.
  • A command-line client.
  • Content roots and inclusion proofs.

Reference

Specification: docs/protocol-v3.md in the Underlay repository. Reference implementation: packages/protocol/src/v3/ in the same repository, which runs in Node and Cloudflare Workers. It is not published to npm. Test vectors: packages/protocol/test/vectors/v3-files.json and v3-commits.json. The protocol is stewarded by Knowledge Futures.