Protocol v3 · §2; v2 §§2–5, 7

Records and schemas

Canonical JSON

Each hashed JSON document MUST be serialized as RFC 8785 (JCS): no insignificant whitespace; strings escaped as by JSON.stringify; numbers serialized as by ECMAScript Number.prototype.toString, with -0 as 0; object members ordered by the UTF-16 code units of their keys.

Note: ECMAScript objects enumerate integer-like keys ("9", "10") first regardless of insertion order. Sorting keys into a new object and calling JSON.stringify does not produce JCS; an implementation MUST emit object members as strings itself.

Records and lines

A record is an id (string), a type, and data (any JSON value). Its key is (type, id), and within a version a key names at most one record. A sorted-record-file holds two kinds of line, each UTF-8 and ending with \n:

record line:   '{"id":' + JCS(id) + ',"type":' + JCS(type) + ',"data":' + JCS(data) + '}'
removal line:  '{"id":' + JCS(id) + ',"removed":true,"type":' + JCS(type) + '}'

size of a line = its length in bytes, without the "\n"
record hash    = SHA-256(record line)     64 lowercase hex characters
a removal line has no hash
  • A record line is the record’s canonical form: a fixed envelope, id, type, data in that order, with only data canonicalized. The record hash is v2’s.
  • A removal line says the record with that key is removed.
  • An upload’s input lines are a record, {"id", "type", "data"}, or a removal, {"id", "type", "removed": true}, with exactly those members. Unlike v2, a line with any other member is refused: there is no private.

Lines of one file, in key order:

{"id":"collection","type":"$meta","data":{"readme":"Weekly snapshot"}}
{"id":"Publication","type":"$schema","data":{"properties":{"title":{"type":"string"}},"type":"object"}}
{"id":"pub-001","type":"Publication","data":{"title":"Notes"}}
{"id":"pub-002","removed":true,"type":"Publication"}

Strings are not Unicode-normalized: é as U+00E9 and as U+0065 U+0301 are distinct ids.

Reserved types

Types that begin with $ are reserved. A collection’s schemas, assets, and metadata are rows of these types, in the same files as its records.

TypeIdData
$schemaa type namethat type’s JSON Schema
$assetan asset’s SHA-256, hex{"size": n, "mediaType": string or null}
$metacollectionthe collection’s metadata object

A publisher’s own types MUST NOT begin with $, and no other $ type is valid. A publisher’s type also MUST NOT begin with a character that sorts before $ (U+0020 to U+0023), so that the reserved rows sort before all records, and MUST NOT be __proto__. A type or an id with a lone surrogate is not valid.

Key order

Keys are ordered by type, then by id. Each is compared lexicographically by its UTF-8 encoding, octet by octet, a proper prefix first: Unicode code point order. Implementations MUST NOT compare UTF-16 code units (ECMAScript’s default < and sort()), which differ when one string has a character at or above U+10000 where the other has one in U+E000–U+FFFF.

Input rules

A writer MUST apply these rules to each line it uploads, and a server to each line it receives. They apply to the whole line and are evaluated on the source text, since a parsed value no longer carries the information they require. Schemas are subject to the duplicate key, unsafe integer, and lone surrogate rules.

RuleRejected ifCode
SyntaxNot JSON as defined by RFC 8259, including an unterminated string or invalid escapesyntax
Duplicate keysAn object with two members whose keys are equal after unescapingduplicate_key
Unsafe integersAn integer literal (no fraction, no exponent) of magnitude above 2⁵³ − 1. Literals with a fraction or exponent are accepted as IEEE 754 binary64unsafe_integer
Lone surrogatesA UTF-16 surrogate code unit, literal or \u-escaped, not part of a pair, in a string or keylone_surrogate
DepthThe line nesting more than 65 levels: each object or array is a level and the envelope is level 1, so data may nest 64too_deep
EnvelopeNot an object; a record whose members are not exactly id, type, and data; or a removal whose members are not exactly id, type, and "removed": truebad_envelope
Record idAbsent, not a string, empty, or over 1,024 UTF-8 bytesbad_id
TypeAbsent, not a string, empty, over 128 UTF-8 bytes, beginning with ".", or containing /, \, U+0000–U+001F or U+007F; beginning with "$" other than a reserved type; beginning with a character that sorts before "$" (U+0020–U+0023); or "__proto__"bad_type
Record sizeA record line longer than 8,388,608 bytesrecord_too_large

A line that breaks several rules is reported with the code of the first in this order: the first violation in text order of an unterminated string or invalid escape (syntax), duplicate_key, unsafe_integer, lone_surrogate or too_deep, whether or not the text is otherwise JSON; syntax for any other text that is not JSON; bad_envelope for a non-object; bad_id or bad_type; bad_envelope for the wrong members; record_too_large. The codes are part of the protocol.

The push API on underlay.org takes record lines under v2’s envelope instead (see Push and pull), and writes them into files as the lines above.

Schemas

A type’s schema is a JSON Schema draft-07 object, held as the data of the type’s $schema row. Schema hash = SHA-256 of JCS(schema). A schema MUST be rejected if:

  • its canonical form exceeds 262,144 bytes;
  • any pattern or patternProperties key exceeds 256 UTF-16 code units. A pattern inside const, enum, default or examples is data and is not limited;
  • the type it is given for is not a valid type;
  • its root private is present and not a boolean, or a schema that is the value of a member of a properties object, at any depth, has "private": true.

A root private has no meaning in v3. On underlay.org a public collection refuses a schema with "private": true: make the collection private instead.

Within a version:

  • A record of a publisher’s type T MUST satisfy the $schema row for T, and a type with records MUST have a $schema row.
  • When a commit changes a schema, each record of that type the version carries over MUST satisfy the new schema, and the commit MUST be refused if any does not. A change that only relaxes the schema (any document valid before is valid after) needs no check of the records carried over.
  • A record whose line a server withholds is not checked.

Validation

A schema MUST be rejected unless:

  • its root $schema, if present, is http://json-schema.org/draft-07/schema, with or without a trailing #;
  • it is valid against the draft-07 meta-schema;
  • each pattern compiles as an ECMAScript regular expression with the u flag;
  • each $ref resolves within the schema or to the draft-07 meta-schema, against the base URI https://schema.underlay.invalid/ unless a $id sets another.

Records are validated under draft-07 with these refinements:

  • keywords adjacent to $ref apply, as in draft 2019-09;
  • keywords draft-07 does not define are ignored (unevaluatedProperties, prefixItems, draft-04 id, …); $defs and $anchor are honored;
  • multipleOf m accepts x when r = x mod m satisfies |r| < 1.1920929 × 10⁻⁷ or |m − r| < 1.1920929 × 10⁻⁷;
  • string length is measured in Unicode code points;
  • format constrains strings only, for the formats ajv-formats 3.0 defines in full mode (date-time and time require a time zone); other format names are ignored.

Only the verdict is normative; error messages are not.

Assets

An asset is a byte string named by its SHA-256. A collection holds an asset when a $asset row names it; the row’s size MUST be the length of the bytes. A $asset row that adds an asset to a collection is accepted only if the asset’s bytes were sent to that collection, or one of its versions already names it. That the server stores the bytes for another collection is never enough.

Informative. On underlay.org a record refers to an asset by an object, at any depth of data, whose $file member is sha256: followed by 64 lowercase hex characters. A push adds a $asset row for each asset its records refer to, or it declares, that the version before did not name.