Design, run, evaluate — rigor for AI decisions, not guesswork.
An agentic system for designing, executing, and scoring real AI experiments, measured on cost, latency, tokens, and a blind 5-criterion quality rubric. The execution playbook is shaped by real failures and learnings, not hypothetical ones.
Pick two versions of something, test them under the same real conditions, and measure which one wins. Same idea as a science experiment in school, just applied to AI.
Cost, latency, token usage, and quality — graded blind against five criteria: correctness, completeness, groundedness, audience fit, and conciseness. If your question maps to one of these, this can test it: does a shorter prompt hurt correctness or completeness? Does one model actually run cheaper or faster for this task? Does a format cut token cost without losing groundedness?
This scores the output itself, not what happens after someone reads it — but the two are connected. A slow step in an AI process isn't just a technical number, it's a moment where a user might assume something's broken and walk away. Reading that connection — between a metric and what it actually means for the person using the product — is how you build products people actually want to keep using.
Built for AI systems specifically — though the underlying principles stretch further than that, if you want them to.
A real experiment, not a hypothetical
Flight 1 tested two ways of formatting a knowledge base — and cut its footprint by 47% without losing answer quality.
What the design skill produced before a single call was made: the question, the three arms, the test-case set, the hypotheses, and the limits it named up front rather than after the numbers came back.
All 42 outputs graded blind against the 5-criterion rubric, then read back against the hypotheses — including the one the run did not support and the ceiling effect that limits what the rest can claim.
The shape of the result at a glance. The filterable, row-level dashboard — and the raw 42-row dataset behind it — comes with the full system.
Try it free. Buy the full system when you're ready.
Design a rigorous experiment and grade the results — same rubric, same discipline, no cost. Works with whatever system you're already running, not just this one.
Install as Claude Skills. They trigger when your task matches, or ask for one by name.
The same two as portable prompts. Paste one at the start of a task, then add your own material underneath.
Everything you need to run this yourself, repeatedly — no subscription, no platform to maintain.
No — design and execution are separate on purpose. Design it, then run it yourself or hand it off to your team.
No. The paid agent handles execution for you. And the free design and scoring skills aren't limited to this system — use them to design a rigorous experiment for whatever you're already running.
No. One-time purchase, yours to keep and reuse.
It's built specifically for evaluating AI outputs — models, prompts, agents. The underlying principles apply more broadly if you want to stretch them, but it's not built to be domain-agnostic.
No refunds — it's a digital download, delivered instantly.
Yes — it's yours once you buy it, no seat limits.