Sync and integrations API
Endpoints for copying collections elsewhere and connecting them to other systems: whole-version exports, webhooks, ARK identifiers, and a couple of service endpoints. Paths below that start …/:owner/:slug are under /api/collections. Where a version is asked for, it is a semver (v1.2.0), a version id (ulv3:…) or latest.
Tree sync
Reading a collection’s log and fetching versions as packs, for mirrors and clients that keep a full copy, is not available yet: GET …/:owner/:slug/log and GET …/:owner/:slug/versions/:n/pack answer 501 with "code": "not_on_v3". To copy a version, use the export below, or records.ndjson.
Export
GET …/:owner/:slug/export?version=&format=
No auth for public collections
A whole version as one archive. version defaults to latest; format is tar.gz (the default) or tar. The archive is named <owner>-<slug>-<semver>.tar.gz in Content-Disposition. This request counts as 20 requests against your rate limit.
Entries, in this order:
manifest.json | Always first: the collection, the version, its schemas, and file notes |
README.md | The version’s metadata.readme, when it has one |
records/<Type>.ndjson | One file per type that has records, one {"id", "type", "data", "hash"} per line |
files/<hash> | The bytes of each file, named by its SHA-256 hash |
curl -o archive.tar.gz \
"https://www.underlay.org/api/collections/kf/archive/export?version=v3.2.0"
tar -tzf archive.tar.gz
# manifest.json
# README.md
# records/Author.ndjson
# records/Publication.ndjson
# files/a1b2c3d4e5f6...manifest.json: files_missing lists files the version references whose bytes the platform doesn’t hold, and files_withheld files that have been blocked; neither is in files/. Blocked records are left out of the NDJSON. totalBytes is the records’ canonical size plus the files’ size.
{
"collection": {
"id": "uuid",
"owner": "kf",
"slug": "archive",
"name": "PubPub Archive",
"description": "Full archive of PubPub publications"
},
"version": {
"semver": "v3.2.0",
"hash": "ulv3:e5f6...",
"message": "April sync",
"recordCount": 4521,
"fileCount": 892,
"totalBytes": 1073741824,
"createdAt": "2026-04-01T00:00:00.000Z"
},
"schemas": { "Author": { "type": "object" }, "Publication": { "type": "object" } },
"files_missing": [],
"files_withheld": []
}The archive streams as it is built, with no limit on the number of records; for very large collections, format=tar is quicker to produce. Entries are stamped with the time the version was created, so exporting the same version again gives a byte-identical archive. If something fails partway, the response is cut off rather than completed, so treat an archive that doesn’t end cleanly as failed.
Webhooks
A webhook sends a signed POST to your URL each time a version of the collection is published. Managing webhooks takes an owner or admin of the owning organization, signed in or with an admin API key; write and read keys, and keys confined to specific collections whatever their scope, get 403.
GET …/:owner/:slug/webhooks
Auth: org owner or admin
{"webhooks": [{id, url, bumpFilter, enabled, createdAt, lastDeliveryAt}]}, newest first. The secret is never listed.
POST …/:owner/:slug/webhooks
Auth: org owner or admin
Body: {"url", "bumpFilter"?, "enabled"?}. url must be https and may not name a local or private address (422). bumpFilter is a non-empty list of major, minor, patch, the kinds of version that trigger a delivery (default: all three). enabled defaults to true. The 201 response is the only time the signing secret is shown; store it.
curl -X POST https://www.underlay.org/api/collections/kf/archive/webhooks \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.org/hooks/underlay", "bumpFilter": ["major", "minor"]}'{
"id": "uuid",
"url": "https://example.org/hooks/underlay",
"bumpFilter": ["major", "minor"],
"enabled": true,
"createdAt": "2026-04-01T00:00:00.000Z",
"lastDeliveryAt": null,
"secret": "ulwhsec_3f9a..."
}PATCH …/:owner/:slug/webhooks/:id
Auth: org owner or admin
Change any of url, bumpFilter and enabled, checked as on create. Returns the updated webhook (without the secret).
DELETE …/:owner/:slug/webhooks/:id
Auth: org owner or admin
Returns {"ok": true}.
POST …/:owner/:slug/webhooks/:id/test
Auth: org owner or admin
Queue a signed test delivery, returning {"ok": true, "deliveryId"}. Its event is ping, its version is null and it carries "test": true. It is logged with the other deliveries.
GET …/:owner/:slug/webhooks/:id/deliveries?limit=
Auth: org owner or admin
Recent deliveries, newest first (limit default 50, max 200). status is pending, success or failed. Deliveries are kept for 30 days.
{
"deliveries": [
{
"id": "uuid",
"event": "version.created",
"semver": "v3.2.0",
"bumpType": "minor",
"status": "success",
"attempts": 1,
"responseCode": 200,
"error": null,
"durationMs": 184,
"createdAt": "2026-04-01T00:00:01.000Z",
"deliveredAt": "2026-04-01T00:00:02.000Z"
}
]
}POST …/:owner/:slug/webhooks/:id/deliveries/:deliveryId/retry
Auth: org owner or admin
Send a delivery again, with its attempts reset. Returns {"ok": true, "status": "pending"}.
Webhook deliveries
Each delivery is a JSON POST to the webhook’s URL. Any 2xx response within 10 seconds is a success; redirects are not followed. A failed delivery is retried up to 5 attempts in all, waiting 1 minute and doubling each time (at most 6 hours between attempts). Deliveries to a disabled webhook fail without being sent.
POST /hooks/underlay HTTP/1.1
Content-Type: application/json
User-Agent: Underlay-Webhook/2.0
X-Underlay-Event: version.created
X-Underlay-Delivery: 0b6f...
X-Underlay-Signature: sha256=5d41402abc4b2a76b9719d911017c592...
{
"event": "version.created",
"collection": { "owner": "kf", "slug": "archive" },
"version": {
"semver": "v3.2.0",
"hash": "ulv3:e5f6...",
"major": 3,
"minor": 2,
"patch": 0,
"recordCount": 4521,
"fileCount": 892
},
"bumpType": "minor",
"delivery": { "id": "0b6f...", "timestamp": "2026-04-01T00:00:01.000Z" }
}X-Underlay-Delivery is the same for every attempt of one delivery, so you can use it to ignore repeats. X-Underlay-Signature is sha256= followed by the hex HMAC-SHA256 of the raw request body, keyed with the webhook’s secret. Check it against the body bytes as received, before parsing:
import { createHmac, timingSafeEqual } from 'node:crypto'
// rawBody: the request body exactly as received (a Buffer or string), before JSON.parse.
function verifyUnderlaySignature(secret, rawBody, header) {
const expected = 'sha256=' + createHmac('sha256', secret).update(rawBody).digest('hex')
const a = Buffer.from(expected)
const b = Buffer.from(header ?? '')
return a.length === b.length && timingSafeEqual(a, b)
}
// e.g. in an Express handler with express.raw({ type: 'application/json' }):
// if (!verifyUnderlaySignature(SECRET, req.body, req.get('x-underlay-signature')))
// return res.sendStatus(401)ARK identifiers
A collection can have an ARK, a persistent identifier that redirects to the collection, to one of its versions, or to a URL a record names. Resolution sees what the caller can read, as everywhere else.
https://underlay.org/ark:<NAAN>/<shoulder><id><check>[.vX.Y.Z][/<Type>/<id>]
https://underlay.org/ark:12345/ulb9bq4n5gmv3k0 the collection
https://underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0 a version
https://underlay.org/ark:12345/ulb9bq4n5gmv3k0/Publication/pub-1 a recordGET /ark:<NAAN>/<name>
No auth required
302 to the target: the collection or version page, or the collection’s custom URL when it has one. A record ARK redirects to the URL in the record field set for its type (below), and is a 404 when none is set, the record isn’t readable, or the field isn’t an http(s) URL. Append ?info or ?? for an Electronic Resource Citation (ERC) as plain text, or ?json for the metadata as JSON. /ark:<NAAN>/ alone returns the NAAN’s policy statement.
GET /api/ark/resolve?path=
No auth required
Resolve an ARK without following it. path is anything containing ark:<NAAN>/… (400 otherwise). Returns {"type": "redirect", "url", "metadata"}, or 404 with {"type": "not_found"}. A relative url is a page on this site.
{
"type": "redirect",
"url": "/kf/archive/v/3.2.0",
"metadata": {
"type": "version",
"who": "Knowledge Futures",
"what": "PubPub Archive v3.2.0",
"when": "20260401",
"where": "https://underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0",
"naan": "12345",
"collectionName": "PubPub Archive",
"ownerName": "Knowledge Futures",
"semver": "v3.2.0",
"message": "April sync",
"appId": null,
"createdAt": "2026-04-01T00:00:00.000Z",
"arkUrl": "https://underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0"
}
}GET | PATCH …/:owner/:slug/ark
Auth: org member (PATCH: org owner or admin, signed in or with an admin key)
GET returns {"enabled", "customUrl", "arkUrl", "shoulder", "arkId"}. PATCH takes {"enabled"?, "customUrl"?}: customUrl is an http(s) URL to redirect the collection and version ARKs to, or null to use the Underlay pages. A collection that has no ARK yet gets one on its first PATCH. Returns {"ok": true}.
GET | PUT | PATCH | DELETE …/:owner/:slug/ark/record-types
Auth: org member (changes: org owner or admin, signed in or with an admin key)
Which record field a type’s record ARKs redirect to. GET returns [{"recordType", "redirectUrlField"}]. PUT or PATCH {"recordType", "redirectUrlField"} sets one; PATCH with "redirectUrlField": null, or DELETE …/record-types/:recordType, removes it. Changes return {"ok": true}.
PATCH /api/accounts/:slug/ark
Auth: org owner or admin, signed in or with an admin key
Set the NAAN the organization’s ARKs are minted under: {"naan": "…"} (digits, at most 16), or null for the default. A NAAN another organization already uses is a 409. Read and write keys, and keys confined to specific collections, get 403.
Storage locations and mirrors
An organization can add its own S3-compatible buckets as locations and mirror collections to them: Underlay keeps a copy of each collection there, its signed commits, current layout, record files and files, at the keys the protocol uses (see Layouts and collections), so anyone with access to the bucket can read every version without Underlay. A new version is copied after it is published. The credentials must be able to both write and read the bucket. Managing locations and mirrors takes an owner or admin of the organization, signed in or with an admin API key not confined to specific collections. Removing a location or mirror never deletes what was copied to the bucket.
GET | POST /api/orgs/:org/locations
Auth: org owner or admin
GET returns {"locations": [...]} (credentials are never returned). POST adds one:
{
"name": "Library mirror",
"endpoint": "https://s3.us-east-1.amazonaws.com",
"bucket": "kf-underlay-mirror",
"prefix": "underlay",
"accessKeyId": "AKIA...",
"secretAccessKey": "..."
}endpoint is an https address with no path; prefix is optional. Before keeping the location, Underlay writes a check object, reads it back, tests whether the bucket serves objects without credentials, and looks for lifecycle rules that would delete mirrored objects. If the check fails the location is not kept and the response is 422; otherwise 201 {"location", "check"}.
POST /api/orgs/:org/locations/:id/check
Auth: org owner or admin
Run the check again and update the location’s status. Returns {"location", "check": {"ok", "publicRead", "readBack", "error", "warnings"}}.
DELETE /api/orgs/:org/locations/:id
Auth: org owner or admin
Remove a location and every mirror to it. Returns 204.
GET | POST | DELETE /api/orgs/:org/placements[/:id]
Auth: org owner or admin
Organization defaults: every collection of the organization, existing and new, is mirrored to each default location. POST {"locationId", "sets"}, where sets is public (the default: public collections only) or public+private (private collections too). public+private needs an https endpoint and a bucket that doesn’t serve objects without credentials (422). A public default never copies a private collection: its mirror of one shows error and says why. Deleting a default also removes the organization’s mirrors to that location.
GET …/:owner/:slug/placements
Auth: org member
Where the collection is stored: the primary and each mirror, with its state (active, backfilling, lagging, error, paused), how many versions it is behind (lag), and whether it comes from an organization default (inherited).
{
"headSeq": 12,
"placements": [
{
"id": "uuid",
"role": "mirror",
"sets": "public",
"inherited": false,
"state": "active",
"syncedSeq": 12,
"lag": 0,
"lastError": null,
"updatedAt": "2026-04-01T00:00:00.000Z",
"location": {
"id": "uuid",
"name": "Library mirror",
"kind": "s3",
"bucket": "kf-underlay-mirror",
"prefix": "underlay",
"status": "active"
}
}
]
}POST …/:owner/:slug/placements
Auth: org owner or admin
Mirror this collection to a location: {"locationId", "sets"}, as for organization defaults. The mirror starts in backfilling. Returns 201, or 422 for a public mirror of a private collection. A mirror of a collection made private later stops, in error.
POST …/:owner/:slug/placements/:id/sync
Auth: org owner or admin
Start a mirror catching up again, for example after fixing a broken location. Returns 202.
DELETE …/:owner/:slug/placements/:id
Auth: org owner or admin
Stop mirroring the collection there. What was copied stays in the bucket. Returns 204, or 409 for a mirror that comes from an organization default (remove the default instead).
Health
GET /api/health
No auth required
Whether the service is up. deployment names the deployment answering: prod or staging.
{
"ok": true,
"version": 2,
"deployment": "prod",
"time": "2026-04-01T00:00:00.000Z"
}Abuse reports
POST /api/abuse-reports
No auth required
Report content that shouldn’t be served. Body: {"hash"?, "url"?, "reason", "contact"?}. Name the content with a file or record hash (64 hex characters, sha256: prefix optional), a url, or both. reason is required (up to 4,000 characters); contact is how to reach you (up to 320). Returns 201 {"ok": true, "id"}.
{
"hash": "sha256:a1b2c3d4e5f6...",
"reason": "This file republishes copyrighted material.",
"contact": "rights@example.org"
}