Integration Guide

Everything a developer or LLM needs to push data to the registry. No SDK required. HTTPS and JSON. For a machine-readable version, see ai.txt.

What is Underlay?

Underlay is a versioned registry for structured knowledge. Apps push snapshots of their data; Underlay preserves them, deduplicates files, and serves them via a stable API. Think npm for data, or Docker Hub for structured content.

Core Concepts

Collection — A named, versioned body of structured data. Identified by :owner/:slug.
Version — An immutable snapshot: JSON Schema + records + file references + metadata. Numbered sequentially.
Record — A flat JSON object with an id, a type, and a data payload conforming to the schema.
File — A binary blob (PDF, image, etc.) stored by SHA-256 hash. Referenced in records via {"$file": "sha256:<hex>"}.

Authentication

Create an API key at /settings/keys or via the API. Pass it as:

Authorization: Bearer ul_your_key_here

Keys are scoped: read, write, or admin. Use write for pushing data.

The Push Flow

Get the current latest version number
Upload any new binary files by hash
Push a version with base_version, schema (if changed), and record changes
On 409 Conflict, re-fetch latest and retry

# 1. Get current state
curl https://underlay.org/api/collections/:owner/:slug/versions/latest

# 2. Upload any new files
HASH=$(shasum -a 256 paper.pdf | cut -d' ' -f1)
curl -X PUT "https://underlay.org/api/collections/:owner/:slug/files/sha256:$HASH" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/pdf" \
  --data-binary @paper.pdf

# 3. Push changes (only what changed since base_version)
curl -X POST https://underlay.org/api/collections/:owner/:slug/versions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "base_version": 42,
    "message": "Daily sync",
    "app_id": "my-app",
    "changes": {
      "added": [...],
      "updated": [...],
      "removed": ["old-record-id"]
    }
  }'

Record Format

Every record has three fields: id (stable string), type (matches schema), and data (the payload).

Relationships are plain ID strings (e.g. "authorId": "author-1")
Files are referenced as {"$file": "sha256:<hex>"}
No joins, no nesting — keep records flat

First Push Example

To push the first version of a collection (creates the initial snapshot):

{
  "base_version": null,
  "message": "Initial import",
  "app_id": "my-app",
  "schema": {
    "type": "object",
    "properties": {
      "Article": {
        "type": "object",
        "properties": {
          "title": {"type": "string"},
          "body": {"type": "string"},
          "authorId": {"type": "string"},
          "publishedAt": {"type": "string", "format": "date-time"}
        }
      },
      "Author": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "email": {"type": "string"}
        }
      }
    }
  },
  "changes": {
    "added": [
      {"id": "author-1", "type": "Author", "data": {"name": "Jane Doe", "email": "[email protected]"}},
      {"id": "article-1", "type": "Article", "data": {"title": "Hello World", "body": "...", "authorId": "author-1", "publishedAt": "2026-04-01T00:00:00Z"}}
    ]
  }
}

Mapping a SQL Database

Most apps store data in SQL. Here's how to map it to Underlay records:

-- For each table, generate a JSON Schema type:
-- table name → type name
-- column name → property name  
-- column type → JSON Schema type (text→string, integer→integer, etc.)
-- foreign keys → note as ID references in the schema description

-- Example: a "publications" table with columns (id, title, doi, author_id)
-- becomes a "Publication" type with properties {title: string, doi: string, authorId: string}
-- The record id is the primary key value.

General rules:

Each table becomes a record type
Each row becomes a record (primary key → record id)
Foreign keys become string ID references
Binary columns (BLOBs) → upload as files, replace with $file references
Generate a JSON Schema from your column types

Versioning

Versions are numbered sequentially and also carry a semver tag derived automatically:

Schema changes → major bump
Record changes → minor bump
Metadata-only changes → patch bump

Version 1 is always v1.0.0.

Privacy

You can control what's publicly visible at three levels:

Private types: Add "private": true to a type in the schema. All records of that type are hidden from public readers.
Private fields: Add "private": true to a field in the schema. That field is stripped from public responses.
Private records: Add "private": true to individual records when pushing. Those records are hidden from public queries.

Private content is stored in the same version — the owner always sees everything. Public readers see only the filtered view. The public content hash excludes private data, so verifiers can confirm integrity of the public subset.

API Reference

Full API docs are at /docs. The key endpoints:

`POST .../versions`	Push a new version (up to 100MB)
`POST .../versions/upload`	Start chunked upload (for large pushes)
`GET .../versions/latest`	Get latest version
`GET .../versions/:n/records`	Get records
`PUT .../files/:hash`	Upload a file
`GET /api/collections`	Browse public collections

Large Pushes (Chunked Upload)

For pushes exceeding 100MB or containing hundreds of thousands of records, use the chunked upload protocol:

Start session: POST .../versions/upload with metadata (base_version, schemas, message). Returns a sessionId.
Append batches: PUT .../versions/upload/:sessionId with up to 10,000 records per batch. Repeat as needed.
Finalize: POST .../versions/upload/:sessionId/finalize to validate and create the version.

Sessions expire after 1 hour. If the same record ID appears in multiple batches, last write wins. See the Versions API docs for full details.

Error Handling

409 Conflict — Another version was pushed since your base_version. Re-fetch and retry.
422 Unprocessable — Records reference files that haven't been uploaded. Upload them first.
400 Bad Request — Schema validation failed or hash mismatch on file upload.

Source Code

Underlay is open source: github.com/knowledgefutures/underlay

Built by Knowledge Futures, a 501(c)(3) nonprofit. Contact: [email protected]