Quickstart

Push your first version in 5 minutes. All you need is curl and a running Underlay instance.

Fastest path: use an AI agent

Point your coding agent at llms.txt and tell it what data you want to push. It has everything it needs to create a collection, write the push script, and handle hashing and negotiation for you. The steps below explain the same flow manually.

1. Sign in and create an API key

Sign in at underlay.org/login via KF Auth SSO. Your account is created automatically on first sign-in. Then go to Settings → API Keys and create a write-scoped key.

# Sign in via KF Auth SSO at https://underlay.org/login
# Your account is created automatically on first sign-in.
# Then create an API key at https://underlay.org/settings/keys

Save the key value. It's shown only once.

2. Create a collection

export KEY="ul_abc123..."

curl -X POST https://underlay.org/api/accounts/yourname/collections \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "slug": "my-dataset",
    "name": "My Dataset",
    "public": true
  }'

3. Push a version

Pushes use a three-step negotiate protocol: send a manifest of record hashes, upload only the records the server needs, then commit.

3a. Negotiate

Hash each record and send the manifest. The server tells you which records it already has.

# Step 1: Hash your records and negotiate with the server
# Record hash = SHA-256 of canonical JSON: {"id","type","data"} with keys sorted recursively
# For this example we'll use pre-computed hashes.

curl -X POST https://underlay.org/api/collections/yourname/my-dataset/versions/negotiate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "base_version": null,
    "message": "Initial import",
    "app_id": "my-app",
    "metadata": {
      "description": "A curated book list",
      "readme": "# My Dataset\nA collection of notable books."
    },
    "schemas": {
      "Book": {
        "type": "object",
        "properties": {
          "title": {"type": "string"},
          "author": {"type": "string"},
          "year": {"type": "integer"}
        }
      }
    },
    "manifest": [
      {"id": "book-1", "type": "Book", "hash": "a1b2c3..."},
      {"id": "book-2", "type": "Book", "hash": "d4e5f6..."}
    ]
  }'
# → {"session_id":"abc123","needed_records":["a1b2c3...","d4e5f6..."],...}

3b. Send records

Send the needed records as JSONL (one JSON object per line). For large datasets, send in batches of up to 10,000 records per request. Skip this step if needed_records is empty.

# Step 2: Send the records the server needs (as JSONL)
curl -X POST https://underlay.org/api/collections/yourname/my-dataset/versions/negotiate/SESSION_ID/records \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: Bearer $KEY" \
  --data-binary @- << 'EOF'
{"id":"book-1","type":"Book","data":{"author":"Douglas Hofstadter","title":"Gödel, Escher, Bach","year":1979}}
{"id":"book-2","type":"Book","data":{"author":"Thomas Kuhn","title":"The Structure of Scientific Revolutions","year":1962}}
EOF
# → {"received":2,"remaining":0}

3c. Commit

# Step 3: Commit the version
curl -X POST https://underlay.org/api/collections/yourname/my-dataset/versions/negotiate/SESSION_ID/commit \
  -H "Authorization: Bearer $KEY"
# → {"semver":"v1.0.0","hash":"...","recordCount":2,"fileCount":0}

4. Read it back

# Get collection info
curl https://underlay.org/api/collections/yourname/my-dataset

# Get latest version records
curl https://underlay.org/api/collections/yourname/my-dataset/versions/v1.0.0/records

# Get the manifest (list of record hashes)
curl https://underlay.org/api/collections/yourname/my-dataset/versions/v1.0.0/manifest

5. Push an update

On subsequent pushes, set base_version to the current latest. The server deduplicates, so only new or changed records need to be sent.

# To push an update, negotiate again with base_version set.
# The server already has book-1 and book-2, so needed_records
# will only include the new/changed records.

curl -X POST https://underlay.org/api/collections/yourname/my-dataset/versions/negotiate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "base_version": "v1.0.0",
    "message": "Add third book",
    "manifest": [
      {"id": "book-1", "type": "Book", "hash": "a1b2c3..."},
      {"id": "book-2", "type": "Book", "hash": "d4e5f6..."},
      {"id": "book-3", "type": "Book", "hash": "g7h8i9..."}
    ]
  }'
# → {"session_id":"xyz789","needed_records":["g7h8i9..."],...}

# Send only the new record, then commit
curl -X POST .../negotiate/SESSION_ID/records \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: Bearer $KEY" \
  --data-binary '{"id":"book-3","type":"Book","data":{"author":"Ludwig Wittgenstein","title":"Philosophical Investigations","year":1953}}'

curl -X POST .../negotiate/SESSION_ID/commit -H "Authorization: Bearer $KEY"
# → {"semver":"v1.1.0","hash":"...","recordCount":3,"fileCount":0}

6. Diff versions

curl https://underlay.org/api/collections/yourname/my-dataset/versions/v1.1.0/diff?from=v1.0.0
# → {"from":"v1.0.0","to":"v1.1.0","added":[...],"updated":[...],"removed":[]}

Record hashing

Each record must be hashed client-side before negotiating. The hash is the SHA-256 of JSON.stringify({id, type, data}) with all object keys sorted recursively. This ensures any client produces the same hash for the same data regardless of key insertion order.

# Record hashing: SHA-256 of canonical JSON
# 1. Build object: {id, type, data}
# 2. Sort all keys recursively (including nested objects in data)
# 3. JSON.stringify the sorted object
# 4. SHA-256 hex digest

# Example in Node.js:
import { createHash } from 'node:crypto'

function canonicalize(value) {
  if (value === null || typeof value !== 'object') return value
  if (Array.isArray(value)) return value.map(canonicalize)
  const sorted = {}
  for (const key of Object.keys(value).sort()) {
    sorted[key] = canonicalize(value[key])
  }
  return sorted
}

function hashRecord(record) {
  const obj = { id: record.id, type: record.type, data: canonicalize(record.data) }
  const json = JSON.stringify(obj)
  return createHash('sha256').update(json).digest('hex')
}

Working with files

To attach files (PDFs, images, etc.) to records, upload them first by hash:

# Compute hash
HASH=$(shasum -a 256 paper.pdf | cut -d' ' -f1)

# Upload
curl -X PUT "https://underlay.org/api/collections/yourname/my-dataset/files/sha256:$HASH" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/pdf" \
  --data-binary @paper.pdf

# Reference in a record
# {"id": "book-1", "type": "Book", "data": {"title": "...", "pdf": {"$file": "sha256:..."}}}

Next steps