Build and version a dataset

Owner and boundary

A dataset is a mutable definition. A dataset version is the materialized, immutable training input. You can populate it from captured traffic, a connector pull, or a JSONL upload, but a run always references the resulting dsv_… version ID.

Pilot may define filters, pull an existing connector, create a version, and monitor state. Credentials, local file bytes, multipart URLs, downloads, and raw captured bodies stay outside model context and require a human or trusted server.

Choose one ingest path

Do not merge paths merely for convenience. Preserve each row’s origin, then create separate versions or combine data only after you can explain how each row was selected.

Build a dataset from captured traffic

Use source IDs rather than display names. The filter below keeps support traffic served by the teacher or fallback, removes exact duplicates, scrubs common PII, and reserves a stable 20% holdout. sample_pct is deterministic selection, not the source's capture sampling rate.

curl -sS -X POST "https://lizzy.albinilabs.com/v1/distill/datasets" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: dataset-support-v1" \
  --data '{"loop":"dlp_REPLACE_ME","name":"Support teacher corpus","filter":{"sources":["dsc_REPLACE_ME"],"tags_any":["support"],"served_by":["upstream","upstream_fallback"],"dedupe":"exact","pii_scrub":true,"sample_pct":100},"split_config":{"holdout_pct":20,"seed":42}}'

Pull from Hugging Face or Braintrust

1. 1

   Create the target dataset

   Create an empty dataset first to establish the destination and split policy.

2. 2

   Create the connector with a human credential handoff

   Hugging Face uses config.repo, with optional split,revision, config, file, and field mapping. Braintrust uses config.project. The credential is write-only.

3. 3

   Start one connector pull

   Persist the returned pull ID. Do not infer completion from connector status.

4. 4

   Poll the pull, then use its version

   A non-empty succeeded pull returns a ready dataset-version ID. A valid zero-row pull returns dataset_version: null; decide whether the filters or source data need to change.

curl --fail-with-body -sS -X POST "https://lizzy.albinilabs.com/v1/distill/connectors" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: connector-support-hf-v1" \
  --data '{"loop":"dlp_REPLACE_ME","kind":"huggingface","name":"support-logs","config":{"repo":"acme/support-logs","split":"train","revision":"main"},"credential":"<optional-private-repo-token>","dataset":"ds_REPLACE_ME"}'

curl --fail-with-body -sS -X POST "https://lizzy.albinilabs.com/v1/distill/connectors/conn_REPLACE_ME/pull" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: pull-support-20260815" \
  --data '{"dataset":"ds_REPLACE_ME","max_rows":50000}'

{
  "id": "cpl_01K...",
  "object": "distill_connector_pull",
  "connector": "conn_01K...",
  "dataset": "ds_01K...",
  "status": "running",
  "rows_pulled": 0,
  "dataset_version": null,
  "error": null
}

1. 1
   running

   Rows are being fetched, normalized, and written to the dataset.

2. 2
   succeeded

   Read rows_pulled and dataset_version; null is valid when zero rows matched.

3. 3
   failed

   Read the error code. Credential failures require human action.

Upload JSONL without exposing raw bytes

Declare the exact byte length and one supported format: messages_jsonl oropenai_chat_jsonl. The maximum object size is 5 GiB. The create response contains signed part URLs; send bytes directly to those URLs, retain each returned ETag, and complete with the ordered part list.

{"messages":[{"role":"system","content":"Answer support questions accurately."},{"role":"user","content":"Can I change my address?"},{"role":"assistant","content":"Yes. Open Order details before dispatch."}],"metadata":{"source":"reviewed-support","ticket_id":"ticket-123"}}

BYTES=$(wc -c < support.jsonl | tr -d ' ')
UPLOAD_BODY=$(jq -nc   --argjson bytes "$BYTES"   --arg dataset "ds_REPLACE_ME"   '{filename:"support.jsonl",bytes:$bytes,format:"messages_jsonl",dataset:$dataset,tags:["reviewed"]}')
curl -sS -X POST "https://lizzy.albinilabs.com/v1/distill/uploads" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: upload-support-v1" \
  --data "$UPLOAD_BODY"

1. 1
   awaiting_parts

   Signed URLs exist; upload every part and retain each ETag.

2. 2
   processing

   Completion was accepted and rows are being imported.

3. 3
   completed

   Import ended. Locate and inspect the resulting dataset version separately.

4. 4
   failed or aborted

   Read the import error, or confirm the deliberate abort. Do not reuse expired part URLs.

Freeze and watch an immutable version

Creating a version is asynchronous and does not currently reserve an idempotency record. After an unknown network outcome, list existing versions before posting again. A returned version starts at building; persist its ID and poll that object only.

curl -sS -X POST "https://lizzy.albinilabs.com/v1/distill/datasets/ds_REPLACE_ME/versions" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"note":"reviewed baseline","split_config":{"holdout_pct":20,"seed":42}}'

{
  "id": "dsv_01K...",
  "object": "distill_dataset_version",
  "dataset": "ds_01K...",
  "status": "building",
  "note": "reviewed baseline",
  "split_config": {"holdout_pct": 20, "seed": 42},
  "row_count": null,
  "error": null
}

Diagnose data by resource ID