How to build good evals without getting lost or in trouble.

How to build good evals without getting lost or in trouble.

You should see evals as saving past work and replaying it with new models or prompts/skills to see if it improved.

This helps you do 3 things:

1. see where it fails so you can tell your boss/users

2. optimize it until it improves

3. make it work on a cheaper / open source model

Measuring "if it improved" is 90% of the work and 10% is infra.

For infra just tell your agent to store tasks in Cloudflare R2 (a cheap/free way to store files in the cloud) in Harbor task format.

This means saving the instructions for the task and how the folder looked before you started the task (does the agent have all the files you had access to, in the state they were in?).

When you run a task your agent should pull down the files, run a docker container (a sandbox on your computer so your agent can't peek at the answer), and send the instructions. Ideally you copy across your auth credentials so you can use your codex/claude sub (otherwise it has to use expensive API tokens).

That's probably all the sophistication you need but one more cool thing you can do is mock any tools your agent might use i.e. if your agent usually reads slack channels or searches google or searches your documents you just need a way to fake it and send back something without touching the real services (otherwise it'll get something new every time you run it).

Ok onto the good stuff: measuring performance.

When you have the output of what the different models or prompts/skills did on your task, you need a way to grade how good or bad that output is or you can't improve it.

There are three types of evals:

1. Human

2. Model

3. Code

Humans are the most slow and expensive but usually* the most accurate if you're working with domain experts.

Do everything you can to bring the AI output into where they already live and work. For an editor put the AI edits in Google docs as comments. Engineers should be reviewing actual PRs. Marketers should be reviewing copy in notion or wherever they normally do it.

This decreases cognitive load which is important because their time is valuable and data labeling sucks. Be grateful for anything you can get from them and use their time extremely wisely.

Jump on a call with them and get them to talk out loud about what they like or don't like. AI can always turn those transcripts into structured data afterwards.

The good news is when you've got some feedback out of them you can train an AI judge by rewriting it's prompt until it agrees 90% with human judgements. Once you have that you can run a/b tests and optimize the prompt/skill for your original task and have the model grade it until you find something that performs better/cheaper.

Simple yes/no questions work best, assign 1 or 0 for each specific thing that could go right or wrong, so it's impossible to misinterpret the grade. For example did this article have a specific AI tell (one judgement for each type of tell), did the powerpoint use our brand colors, or is it following our coding conventions?

Models are cheaper and faster at grading than humans, so you can run many more tests and automatically optimize your agents when you have an AI judge you can trust. If you get here you're in the top 5% of AI users today and life will continue to improve for you. You'll have proof of what works or doesn't and actually be able to see the frontier and when it shifts.

Look at the data when something rests badly and you'll learn something new about your domain.

This is when you'll realize that the human is not the ceiling. Quite often the human domain experts get things wrong and explore methods for superhuman results.

For more sophistication you want to invest in building something closer to an RL environment with deterministic code tests that run faster and cheaper at scale, more on that another time. Good luck building evals - this is one of the things AI still does badly and there's a great career to be had!

Key Takeaways