Define and test rewards

Owner and boundary

A reward turns product success into a score that evaluation or reinforcement stages can consume. Lizzy stores the reward definition separately from a recipe so you can test and revise the signal before spending compute. Reward IDs are mode-isolated.

Pilot can create and test non-secret rewards. Webhook secrets are write-only and should be supplied by a human or trusted server. A judge test may call a real model and consume judge tokens; a verifier runs locally and is the clearest diagnostic starting point.

Choose the narrowest signal that matches the job

Create an explainable verifier

This example requires valid JSON with answer and confidence. Persist the returned reward ID, not only its display name. The create call honors an idempotency key.

curl -sS -X POST "https://lizzy.albinilabs.com/v1/distill/rewards" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: reward-valid-ticket-json-v1" \
  --data '{"name":"valid-ticket-json","kind":"verifier","config":{"check":"json_schema_valid","schema":{"type":"object","required":["answer","confidence"],"properties":{"answer":{"type":"string"},"confidence":{"type":"number","minimum":0,"maximum":1}}}}}'

{
  "id": "rwd_01K...",
  "object": "distill_reward",
  "name": "valid-ticket-json",
  "kind": "verifier",
  "status": "active",
  "config": {
    "check": "json_schema_valid",
    "schema": {"type": "object", "required": ["answer", "confidence"]}
  },
  "livemode": true
}

Test passing, failing, and malformed candidates

A test accepts exactly one mode: an inline candidate, one capturedcall, or a recent sample count from 1 to 25. Inline mode may also include prompt and reference. Testing a judge can incur token cost; testing a webhook exposes scorer availability errors instead of silently skipping them.

TEST_BODY=$(jq -nc   --arg candidate '{"answer":"Open Order details before dispatch.","confidence":0.92}'   '{candidate:$candidate}')
curl --fail-with-body -sS -X POST "https://lizzy.albinilabs.com/v1/distill/rewards/rwd_REPLACE_ME/test" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data "$TEST_BODY"

curl --fail-with-body -sS -X POST "https://lizzy.albinilabs.com/v1/distill/rewards/rwd_REPLACE_ME/test" \
  -H "Authorization: Bearer $LIZZY_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"candidate":"not json"}'

{
  "object": "distill_reward_test",
  "reward": "rwd_01K...",
  "results": [
    {
      "call": null,
      "status": "scored",
      "score": 0,
      "explanation": null,
      "detail": {
        "errors": [{"path": "$", "message": "output is not valid JSON"}]
      }
    }
  ],
  "mean_score": 0,
  "duration_ms": 1
}

1. 1

   Prove the positive case

   A valid answer should score as expected for the intended reason.

2. 2

   Prove the negative case

   A plausible but wrong answer must fail. A reward that accepts everything cannot guide training.

3. 3

   Probe malformed and edge cases

   Exercise empty text, unexpected tool calls, schema extras, boundary values, and missing references.

4. 4

   Inspect score distribution

   If almost every test case gets the same score, revise the signal or the dataset before training.

List reward IDs in every proposal

The three request shapes below have intentionally different meanings:

{
  "dataset_version": "dsv_01K...",
  "objective": "quality",
  "max_cost_usd": 25,
  "rewards": ["rwd_01K..."],
  "outcome": "deployable_student"
}

deployable_student asks the proposal to include a publish stage. Areport outcome evaluates but does not guarantee that a student model will be registered for serving.

Diagnose the signal before blaming training