Skip to content

AI engineering workshop · 7 chapters · 70 minutes

Evals, not vibes.

You changed a prompt and the answers look better. Are they? Build a tiny eval harness for one LLM feature in plain Python, then compare prompts and models with numbers instead of opinions.

  • No dependencies: Python standard library only.
  • No API key needed to start: an offline baseline runs right away.
  • Any OpenAI-compatible model when you are ready: Groq, OpenAI or Ollama.
  • A local Bring your provider page: check a provider and run the eval, no code.
Start the workshopView the code
uv run python -m triage runoffline baseline
PASS double-charge billing, normal
FAIL invoice-copy billing, normal
FAIL card-declined-blocked other, normal
PASS api-500 bug, high
PASS forgot-password account, high
FAIL dark-mode feature_request, normal
...
category 69%
urgency 38%
urgent_never_missed 94%
overall 67%

An eval has four parts

Casesevals/data/cases.jsonlReal inputs with the answer you expect.
Featureevals/triage/feature.pyThe prompt and code under test.
Gradersevals/triage/graders.pyFunctions that score one answer from 0 to 1.
Runnerevals/triage/run.pyRuns every case, averages, saves, compares.

7 short chapters

Each one is a few minutes of reading, one thing to run, and an exercise. By the end you can put an eval in front of any AI feature you ship.

  1. 01Why evals5 min
  2. 02Your first run10 min
  3. 03Write good cases10 min
  4. 04Graders10 min
  5. 05Plug in a real model10 min
  6. 06Compare runs15 min
  7. 07Make it a gate10 min

Run it in a minute

You need uv and git. uv installs Python for you.

git clone https://github.com/jars-demo/evals-demo.git
cd evals-demo/evals
uv sync
uv run python -m triage run