Skip to content

7 chapters · 70 minutes

The workshop

From your first scorecard to an eval that guards every change. Work through them in order; each ends with a way to check it worked.

  1. 01Why evalsTrying a prompt five times and liking the answers is not a test. An eval is.5 min
  2. 02Your first runInstall, run the eval against an offline baseline, and read the scorecard.10 min
  3. 03Write good casesThe dataset is the eval. Real, varied cases with clear expected answers.10 min
  4. 04GradersSmall functions that score one answer. Measure what actually matters.10 min
  5. 05Plug in a real modelPoint the harness at any OpenAI-compatible model and get its score.10 min
  6. 06Compare runsSave a run, change one thing, run again, and see exactly which cases moved.15 min
  7. 07Make it a gateFail the build when the score drops below a line you choose.10 min