Built for the brief "run evals, generate new benchmarks such as agent-bot evals, and maintain the existing ones". It has three parts: a case format with graded checks, a scorer that runs live in your browser, and a regression diff between two runs so a model or prompt change shows up as named cases, not as one average. Agent transcripts here are generated from fixed mutations of the expected calls. They are not real model output and no Composio account was touched. Tool names are illustrative.
One JSON object per case. Checks are graded, so a failure says which part broke: tool choice, arguments, call order, extra calls, or a forbidden call.
Regressions are listed first. A candidate with a higher average and one new failure in a destructive-action case should not ship, and the diff makes that visible.
Edit an argument, swap two calls, add a delete call, then press Score. The same function scores the whole suite in the other tabs.
| Step | What happens | Output |
|---|---|---|
| 1. Inventory | Read the existing evals, record what each case checks, which ones are flaky, which have a stale expected value. | Case list with owner, last pass rate, flake rate |
| 2. New benchmark | Add agent-bot cases by category (single call, multi-step, ambiguous request, missing permission, destructive action). Each case has an expected trace and a forbidden-call list. | Suite file in the format above, plus a short rationale per category |
| 3. Run | Run each case several times against the models you name, store full traces. | Per-case table and per-category score |
| 4. Compare | Diff against the previous run, split regressions from fixes, mark cases that flip between trials. | Diff report like tab 3 |
| 5. Maintain | When a tool schema or app version changes, update expected values and re-baseline; retire cases that no longer test anything. | Changelog per suite version |
Which tools and apps the first benchmark must cover, which models are compared, where the existing evals live and how they are run today, and whether the work is a fixed deliverable or a recurring weekly run. Those decide the real scope and price.