How Distill resources fit together

How resources connect

A Distill workflow is a chain of resources, not one long request. Each resource records one stage and links to the inputs needed by the next stage. Store those IDs instead of trying to reconstruct the chain from display names.

1. 1
   Loop · dlp_

   Names one optimization objective and groups the pipeline for people and navigation.

2. 2
   Source or import

   A dsc_ source captures calls, while a conn_ connector or dup_ upload brings existing examples.

3. 3
   Dataset · ds_

   A mutable definition selects, filters, deduplicates, and splits examples.

4. 4
   Dataset version · dsv_

   An immutable materialization freezes the exact train and holdout inputs.

5. 5
   Reward and recipe

   rwd_ resources define measurable outcomes; a proposed recipe describes stages but starts no compute.

6. 6
   Run · run_

   The execution record owns validation, stages, budgets, logs, metrics, and terminal outcome.

7. 7
   Report, model, deployment

   rpt_ proves the result, sm_ identifies the student artifact, and dep_ controls serving.

1. 1

   Store the returned object, not just its display name

   Names and slugs help humans. IDs are the foreign keys. Persist at least the ID, object type, status, mode, creation time, and the upstream operation ID that produced each resource.

2. 2

   Make lineage visible in your own system

   A useful experiment record links loop → source or import → dataset → version → rewards → dry run → live run → report/model → deployment. This is also the shortest diagnostic path when a result looks wrong.

3. 3

   Change the resource that owns the setting

   Edit a dataset definition before creating another version. Create a new run for a changed recipe or budget. Patch a deployment for rollout changes. Never pretend an immutable version or completed run changed in place.

Test and live resources are separate

An API token is created in exactly one environment. A token beginning withlz_test_ resolves test resources; a token beginning withlz_live_ resolves live resources. The backend, not a request parameter, applies that boundary. A resource that exists in live mode normally looks absent to a test token and vice versa.

Handle asynchronous work by resource ID

A create request often returns before the work finishes. The response ID is how you find that work again. Restarting a process, closing a browser, or ending an agent turn does not cancel the server-side resource. Create once, then inspect by ID.

Dataset version   building → ready | failed
Upload            awaiting_parts → processing → completed | failed
                  awaiting_parts → aborted
Connector pull    queued → running → succeeded | failed
Training run      queued → validating → running → evaluating
                  → succeeded | failed | canceled
Deployment        provisioning → live | degraded → retired
Pilot workflow    scope → data → rewards → plan → dry_run → approval
                  → training → results → deployment → complete
                  (may also become blocked or abandoned)

degraded is not terminal: a deployment can recover. Run terminal states aresucceeded, failed, and canceled. Dataset versions stop at ready or failed. Model your poller from the resource’s documented state family instead of treating every non-success string as equivalent.

1. 1

   Create once

   Supply an Idempotency-Key when the endpoint requires or supports it. Persist the key before sending and the resource ID as soon as it arrives.

2. 2

   Poll with a deadline

   Use the resource GET endpoint, exponential backoff with jitter, and an application deadline. A local deadline should return “still in progress” to the caller, not mutate the server state.

3. 3

   Resume by ID

   On the next job attempt or agent turn, read the saved ID first. Only create again when you can prove the original create never produced a resource, or when the operation explicitly requires a new attempt.

4. 4

   Explain terminal failure

   Retain the resource’s error object, its stage or phase, the latestX-Request-Id, and non-secret input references. Fix the named prerequisite before selecting a retry path.

REST, proxy, and Pilot permissions differ

These surfaces share resource contracts but they do different jobs. Confusing them is a common cause of unsafe automation and copilot loops.

The proxy has another important boundary. On an upstream provider error it relays that provider’s HTTP status and body rather than wrapping it in Lizzy’s control-plane error envelope. Preserve X-Lizzy-Call-Id and X-Lizzy-Served-By to decide whether a failed or degraded response came from capture, an upstream teacher, a student, or teacher fallback.

Identify resources and requests

IDs are time-ordered opaque strings with a type prefix. The prefix helps logs and diagnostics, but it is not authorization and should not replace a resource lookup. Lizzy always scopes reads to the authenticated workspace and mode.

ws_     workspace             tok_    API token
dlp_    Distill loop          dsc_    capture source
call_   captured call         ds_     mutable dataset
dsv_    dataset version       dup_    dataset upload
conn_   connector             cpl_    connector pull
rwd_    reward                run_    training run
stg_    run stage             sm_     student model
rpt_    evaluation report     dep_    deployment
sbx_    Distill sandbox       pls_    Pilot session
plw_    Pilot workflow        plh_    Pilot handoff
plp_    Pilot proposal        plwth_  Pilot watch