What running an LLM-as-judge eval suite costs per run.
LLM-as-judge evals pay to generate and to grade, so tokens stack up fast in CI. A cheap judge and a tight golden set keep it affordable. Size the underlying calls on the LLM token cost calculator.
The LLM Evaluation Cost Calculator estimates the token spend of a single evaluation run, where a set of test cases is passed through one or more models and scored by a judge model. It multiplies the number of test cases by the models compared, then by the tokens per case — covering both the generation tokens the tested models produce and the judge tokens spent grading each response — and applies your $ per 1 million tokens rate to convert that total into a dollar figure. The main drivers are therefore the size of your test set, how many models you evaluate in parallel, and how many tokens each generation-plus-judge cycle consumes.
The key trade-off to watch is that cost scales multiplicatively: adding models or expanding the test set doesn't just add tokens, it compounds them, and the judge pass can quietly double per-case usage. Before running a large sweep, use the calculator to size a smaller representative sample and confirm the per-run cost is acceptable, since a heavy judge prompt or a long test suite can make repeated evaluation runs far more expensive than a single generation would suggest. Checking the estimate first helps you decide whether to trim cases, reduce the models compared, or use a cheaper judge.
Cost is test cases times models times tokens per case, including the judge model reading each answer. An LLM-as-judge roughly doubles tokens because every generated answer is also scored. Running the suite on every commit multiplies that across the month.
Use a smaller, cheaper model as the judge, keep a focused golden set rather than thousands of cases, and run the full suite nightly with a quick smoke set on each commit. You do not need to eval every model on every push.