Evaluate a training run

Three resources answer three different questions

Start with the run, follow its report ID for evaluation evidence, and follow its student_model ID for the artifact that can be deployed. Do not infer one from a display name or from the latest resource in a list.

Test and live results are isolated by credential mode. A test credential cannot see a live run with the same-looking ID, and a dry-run report is stub evidence that cannot satisfy the deployment gate.

Read the complete run before its outputs

1. 1
   queued

   The run and its five-stage timeline have been created.

2. 2
   validating

   Frozen data and the executable plan are being prepared.

3. 3
   running

   Training or publication is active.

4. 4
   evaluating

   Holdout candidates are being compared.

5. 5
   succeeded, failed, or canceled

   The terminal run contains final totals, stage state, and any output IDs.

curl --fail-with-body -sS \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  "https://lizzy.albinilabs.com/v1/distill/runs/run_..."

{
  "id": "run_...",
  "object": "distill_run",
  "livemode": true,
  "name": "Support student live v1",
  "status": "succeeded",
  "dry_run": false,
  "dataset_version": "dsv_...",
  "plan_hash": "9b4d...",
  "validated_dry_run": "run_dry...",
  "student_model_name": "support-student-v1",
  "progress": { "percent": 100, "message": "Done." },
  "totals": { "gpu_seconds": 2184, "judge_tokens": 42000, "cost_usd": 18.73 },
  "budget": { "max_cost_usd": 25 },
  "report": "rpt_...",
  "student_model": "sm_...",
  "error": null,
  "current_stage": { "seq": 5, "kind": "publish", "status": "succeeded" },
  "stages": [
    { "seq": 1, "kind": "materialize", "status": "succeeded", "attempt": 0, "error": null },
    { "seq": 2, "kind": "sft", "status": "succeeded", "attempt": 0, "error": null },
    { "seq": 3, "kind": "grpo", "status": "skipped", "attempt": 0, "error": null },
    { "seq": 4, "kind": "eval", "status": "succeeded", "attempt": 0, "error": null },
    { "seq": 5, "kind": "publish", "status": "succeeded", "attempt": 0, "error": null }
  ]
}

Stage states are pending, running, succeeded,failed, skipped, or canceled. Every run includes all five canonical stages. A recipe that omits GRPO or publish shows that stage as skipped, which lets monitoring use one stable timeline.

Use curves to explain the final stage summary

The run detail includes a short summary for each stage. The metrics endpoint returns time series as [step, value] pairs. SFT can expose loss, learning rate, eval loss, and holdout accuracy. GRPO can expose reward mean, reward standard deviation, KL, and holdout accuracy.

curl --fail-with-body -sS \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  "https://lizzy.albinilabs.com/v1/distill/runs/run_.../metrics"

{
  "object": "distill_run_metrics",
  "stages": [
    {
      "stage": "sft",
      "series": {
        "loss": [[0, 2.31], [100, 1.42], [200, 1.08]],
        "eval_loss": [[100, 1.57], [200, 1.22]],
        "holdout_accuracy": [[0, 0.71], [200, 0.82]]
      }
    }
  ]
}

The service can downsample long series while retaining ordered endpoints. An empty stage list can be valid before a trainer reports points or when the recipe had no SFT or GRPO series. Re-read the run and its stage status before diagnosing missing metrics as a failure.

Inspect the report and drill into examples

{
  "id": "rpt_...",
  "object": "distill_report",
  "livemode": true,
  "run": "run_...",
  "student_model": null,
  "dataset_version": "dsv_...",
  "status": "ready",
  "verdict": "pass_with_regressions",
  "sample_size": 500,
  "metrics": {
    "quality_retention": 0.964,
    "judge": { "win": 118, "loss": 18, "tie": 364 },
    "checks": [
      { "name": "support-exact-answer", "kind": "exact_match", "pass_rate": 0.942, "scored": 500 }
    ],
    "latency": {
      "student_p50_ms": 190,
      "student_p95_ms": 420,
      "teacher_p50_ms": 610,
      "teacher_p95_ms": 1240
    },
    "cost": {
      "student_per_call_micro_usd": 92,
      "teacher_per_call_micro_usd": 680,
      "reduction_pct": 86.5
    }
  },
  "examples_available": true,
  "examples_expire_at": "2026-09-14T12:00:00Z",
  "share": { "enabled": false, "url": null, "expires_at": null, "include_examples": false }
}

A report can be created during eval before the later publish stage creates a student, soreport.student_model may be null. Resolve the published model fromrun.student_model, or list/distill/models?run={run_id}. The student resource points back to its report.

Current verdicts are pass, pass_with_regressions, andfail. They summarize the frozen evaluation; they are not the serving runtime gate. Serving requires a ready student with a complete real report, even when the verdict is fail, so your product policy must decide whether the evidence is acceptable.

{
  "id": "sm_...",
  "object": "distill_student_model",
  "livemode": true,
  "name": "support-student-v1",
  "base_model": "qwen3-8b",
  "format": "lora",
  "size_params": "8B",
  "run": "run_...",
  "dataset_version": "dsv_...",
  "report": "rpt_...",
  "status": "ready",
  "deployment_count": 0,
  "drift": { "windows": [] }
}

Decide whether to iterate or deploy

• The run is live-mode, terminal, and tied to the expected dry run and plan hash.
• The holdout version and sample size represent the production traffic you will route.
• Quality retention and each selected reward meet the product threshold.
• Loss examples and tool slices contain no severe regression hidden by the average.
• Student latency and per-call cost support the original objective.
• The student is ready, uses LoRA format, and points to this run and report.
• A known-good fallback source and teacher model still exist in the same live mode.

If the decision is to proceed, create a shadow deployment using the serving guide.

Recover from missing or incomplete results

Persist run_id, report_id, student_model_id,dataset_version, plan_hash, the terminal run body, and the report snapshot used for the decision. These references let a later operator reproduce the evidence even after detailed example bodies expire.