THE AI EVALUATION BUDGET PLANNER

Know the cost.
Before the run.

Turn test cases, model calls and AI judges into a clear budget. Compare three scenarios before your next evaluation.

No sign-upRuns in your browserExport your estimate

YOUR WORKSPACE

Build your evaluation budget

All preset prices are illustrative USD rates, not live provider prices. Replace them with the rates for your exact model and billing tier.

Define the workload

One shared workload for all three scenarios.

Planned evaluation results3,000

Each scenario assumes the same token use and prices across its candidate models. Scenarios are alternatives, not costs to add together.

Include an AI judge

Shared judge assumptions. Set calls to zero to disable.

This model applies the extra-call allowance to candidate and judge calls. Adjust it to match your actual evaluation pipeline.

Compare your scenarios

Edit token assumptions and prices. Totals update as you type.

USD / one evaluation run

Costs compare your inputs only. They do not measure model quality. Display values are rounded; exports retain calculation precision.

Your inputs stay in this browser. Saving locally is optional; clearing browser data removes saved scenarios. No accounts, API keys or analytics.

NO BLACK BOX

Every assumption.
Out in the open.

A small, transparent planning tool. Use it to set a budget, then validate against a real pilot run.

How the estimate works

Planned results = test cases × repetitions × candidate models. Billed candidate calls = planned results × (1 + extra-call allowance). Judge calls = billed candidate calls × judges per result.

Call cost = (input tokens × input rate + output tokens × output rate) ÷ 1,000,000Budget = (candidate cost + judge cost) × (1 + reserve percentage)

Cost per result divides the total budget by planned results, excluding extra calls from the denominator. Fractional call totals are planning averages.

What this estimate leaves out

This linear model assumes homogeneous calls. It excludes prompt caching, batch discounts, tools, search, image/audio/video tokens, hosting, human review, taxes and currency conversion. Failed-call billing differs by provider. The extra-call allowance is a direct billed-call uplift, not a failure probability; the reserve is your planning buffer, not a vendor charge.

Check the source prices

External reference pages below are starting points. Rates are not fetched or verified by this tool. Check the exact model, region, service tier and effective date before relying on an estimate.

Anthropic Google Gemini DeepSeek