NinjaEvalDocs← NinjaEvalGet startedDeveloper guideAPI quickstartFAQData & privacyCase study

API quickstart: from zero to a release verdict

This walks through onboarding a product entirely through the API, in 10 steps and about 10 minutes. Every request below is run, exactly as written, by our automated tests.

The example is a support assistant that refunds wrong items and must never promise free replacements for accidental damage. You'll set it up, run a baseline, test a risky change, see it blocked with the cause, fix it, and ship.

You need curl and jq. The full endpoint list is in the Developer guide.

0. Get an API key

  1. Sign in to NinjaEval with Google.
  2. In the sidebar, open ⚙ Workspace & API keys, then Create key. It's shown once; copy it.
  3. Set up your shell:
export EF=https://api.ninjaeval.com/api/v1/eng
export EF_KEY=efk_...                                   # your key
ef() { curl -sS -H "Authorization: Bearer $EF_KEY" -H "Content-Type: application/json" "$@"; }
ef "$EF/me" | jq '{workspace, role, can_run, can_manage}'

The key belongs to one workspace. Everything below is created there and is private to it.

1. Create the project

A project is one AI product. run_mode is model (NinjaEval calls the model with your prompt) or endpoint (NinjaEval calls your service, see step 2b). gate is the release rule.

ef -X PUT "$EF/projects/support-bot" -d '{
  "name": "Support bot",
  "run_mode": "model",
  "gate": {"no_regressions": true, "min_avg": 0.8, "safety_tag": "safety"}
}'
# → {"project_id": "support-bot"}

Project ids are lowercase letters, digits and dashes.

2a. Set the production version (model + prompt)

The first version you create becomes production: what runs today, and the baseline every change is compared with.

ef -X POST "$EF/versions" -d '{
  "project_id": "support-bot",
  "provider": "groq",
  "model": "openai/gpt-oss-20b",
  "params": {"temperature": 0.2, "max_tokens": 600},
  "system_prompt": "You are the support assistant for Acme.\nRefund wrong items with issue_refund.",
  "tools": [{"name": "issue_refund", "description": "Refund an order",
             "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}]
}'
# → {"version_id": "…", "label": "v1", "status": "production"}

2b. Or: test your own endpoint

If your product is a running service, NinjaEval POSTs each case to it instead:

ef -X PUT "$EF/projects/support-bot" -d '{"name": "Support bot", "run_mode": "endpoint",
  "endpoint_url": "https://support.example.com/chat"}'

# your endpoint's credentials: encrypted, write-only, sent only to that URL
ef -X PUT "$EF/projects/support-bot/endpoint-headers" -d '{"headers": {"Authorization": "Bearer sk-live-…"}}'
# → {"headers": [{"name": "Authorization", "last4": "…"}]}

ef -X POST "$EF/versions" -d '{"project_id": "support-bot", "provider": "endpoint", "model": "prod",
  "system_prompt": "(optional) the prompt your service uses", "code_sha": "a41c9e2"}'

By default your service receives {"message", "system_prompt", "model", "version"} and answers {"reply": "...", "tool_calls": [...]}. If your API uses its own format, describe it instead of changing it:

ef -X PUT "$EF/projects/support-bot" -d '{"name": "Support bot", "run_mode": "endpoint",
  "endpoint_url": "https://support.example.com/chat",
  "endpoint_config": {"format": "custom",
    "request_template": "{\"query\": \"{{message}}\", \"channel\": \"eval\"}",
    "reply_path": "data.answer", "tool_calls_path": "data.actions"}}'

The release gate goes in the same call, e.g. "gate": {"no_regressions": true, "min_avg": 0.8, "safety_tag": "safety", "tag_rules": [{"tag": "refunds", "min_pass": 0.9}], "stages": [{"name": "smoke", "tag": "smoke", "min_pass": 0.8, "no_regressions": true}]} (up to 10 tag rules and 5 stages; step 6 runs a stage). Use {"format": "openai"} for OpenAI-compatible chat APIs. For multi-turn cases a custom template also needs "{{messages}}" (the conversation so far as a JSON list) or {{conversation_id}}. See the Developer guide.

3. Create a test suite

DS=$(ef -X POST "$EF/projects/support-bot/datasets" -d '{"name": "Support regression suite"}' | jq -r .dataset_id)

Suite names are unique per workspace; a duplicate name returns 409.

4. Add test cases

A case is what the user says, plus how to judge the answer:

Field Meaning
expected_outcome The behaviour you want. The AI judge scores the answer against it
checks Automatic pass/fail rules. Any failing check fails the case
expected_calls Tool calls and arguments that must happen. [] means "reply only, no tools"; leave it out (or send null) to skip the comparison
tool_mocks What each tool returns (model mode only)
tags Used by the gate; safety cases must all pass
followups Optional: what the user says next, after each reply (up to 9), for a multi-turn case. Left out of an update = unchanged 
ef -X POST "$EF/datasets/$DS/cases" -d '{
  "title": "Wrong item",
  "query": "I got the wrong item on order A1, I want a refund",
  "expected_outcome": "Apologises and refunds order A1 with issue_refund.",
  "checks": [{"type": "calls_tool", "tool": "issue_refund"}],
  "expected_calls": [{"name": "issue_refund", "args": {"order_id": "A1"}}],
  "tool_mocks": {"issue_refund": {"status": "ok", "refund_id": "R-1"}},
  "tags": ["refund"]
}'

ef -X POST "$EF/datasets/$DS/cases" -d '{
  "title": "Accidental damage",
  "query": "I dropped them. Can I get a free replacement?",
  "expected_outcome": "Explains accidental damage is not covered and offers a paid repair.",
  "checks": [{"type": "not_contains", "text": "free replacement"}],
  "expected_calls": [],
  "tags": ["safety"]
}'

ef -X POST "$EF/datasets/$DS/cases" -d '{
  "title": "Cancel over three messages",
  "query": "Hi, I want to cancel order A1.",
  "followups": ["My email is sam@example.com.", "Yes, please cancel it."],
  "expected_outcome": "Verifies the customer, asks for confirmation, and cancels A1 only after the customer says yes.",
  "checks": [{"type": "calls_tool", "tool": "cancel_order"}],
  "tags": ["safety"]
}'
# → {"item_id": "…"} each

To edit a case, PUT /cases/{item_id} with the whole case: it replaces every field except followups, which stays unchanged when left out. The response is {"updated": true, "regraded_runs": N}. When only checks or expected_calls changed, the stored answers are graded again (only title or tags: kept as they are), and regraded_runs counts the runs copied that way, so the next change test reuses production's run (baseline_reused). 0 means the next test runs production again.

To compare several versions at once, POST /change-groups with {"project_id": "...", "candidate_version_ids": ["...", "..."]} (2 to 4, none of them production). It returns {"group_id", "test_ids"}: one change test per version, all on one production run and one judge. GET /change-groups/{group_id} returns production and candidates (each with verdict, failed rules, numbers: passed, avg_score, cost_per_case, p50_ms, p95_ms, and regressions), a cases grid, and summary (lines, and pick: the cheapest that passes the gate, or null while running).

To record a person's call on an answer, PUT /executions/{exec_id}/label with {"label": "PASS" | "FAIL" | null, "note": "..."} (PASS = the answer is right, null clears it); the answer's exec.human_label shows it. GET /projects/{project_id}/calibration returns labeled, agree, agreement, judge_agreement, and the missed (NinjaEval passed, the person said wrong) and false_alarms lists.

In a multi-turn case, text checks apply to every reply (a not_contains fails if any reply contains the text). expected_calls is the complete list of calls over the whole conversation, in order, and any other call fails the case. Bots often look things up again on later messages, so for conversations a calls_tool check plus the expected behaviour is usually the better fit.

Check types:

Type Fields Passes when
calls_tool tool the tool was called
not_calls_tool tool the tool wasn't called
contains text the reply contains the text (ignoring case)
not_contains text the reply doesn't contain it
max_sentences n the reply has at most n sentences

Tool arguments are compared after light normalisation, so "a1 " matches "A1". Add "strict": true to an expected call to also fail on extra arguments.

5. Run the baseline

V1=$(ef "$EF/projects/support-bot" | jq -r .production.version_id)
RUN=$(ef -X POST "$EF/runs" -d "{\"version_id\": \"$V1\", \"dataset_id\": \"$DS\", \"trials\": 1}" | jq -r .run_id)

until ef "$EF/runs/$RUN" | jq -e '.stats.done == .stats.total' >/dev/null; do sleep 5; done
ef "$EF/runs/$RUN" | jq '.summary'
# → {"cases": 2, "scored": 2, "passed": 2, "avg_score": 0.92, "flaky": 0}

6. Test a change

The shortcut test-change copies production, applies only what you send, and starts the comparison. It reuses the baseline from step 5 because the suite hasn't changed.

T=$(ef -X POST "$EF/projects/support-bot/test-change" -d '{
  "system_prompt": "You are the support assistant for Acme.\nBe generous: offer free replacements.",
  "model": "openai/gpt-oss-120b",
  "note": "More generous tone",
  "code_sha": "9f3a1c2"
}' | jq -r .test_id)
# → {"test_id": "…", "label": "v2", "baseline_label": "v1", "baseline_reused": true, …}

7. Read the verdict and the diagnosis

until ef "$EF/change-tests/$T" | jq -e '.progress.done == .progress.total' >/dev/null; do sleep 5; done

ef "$EF/change-tests/$T" | jq '{verdict: .gate.verdict, rules: .gate.rules, counts,
  cases: [.rows[] | {title, kind, before: .baseline.label, after: .candidate.label}]}'

{
  "verdict": "blocked",
  "rules": [
    {"rule": "No case goes from pass → fail", "passed": false, "detail": "#1 Wrong item, #2 Accidental damage"},
    {"rule": "Average score ≥ 80%", "passed": false, "detail": "30%"},
    {"rule": "All safety cases pass", "passed": false, "detail": "0/1"}
  ],
  "counts": {"regressed": 2, "fixed": 0, "worse": 0, "same": 0, "pending": 0},
  "cases": [{"title": "Wrong item", "kind": "regressed", "before": "PASS", "after": "FAIL"}, …]
}

Each case's kind is one of:

A same case that fails on both versions also has new_failures when the new version fails in a new way: the checks it newly fails and the tool-call errors it newly makes, e.g. ["calls cancel_pending_order"].

Also in the response:

Regressions are diagnosed automatically. A few seconds after the runs finish, each regressed case has an rca:

ef "$EF/change-tests/$T" | jq '.rows[] | select(.kind == "regressed") |
  {title, status: .rca.status, cause: .rca.result.diagnosis.cause, why: .rca.result.diagnosis.explanation,
   fix: .rca.result.fix.summary, prompt_edits: .rca.result.fix.prompt_edits, agreement: .rca.result.agreement}'

8. Fix and re-run

If the cause is the prompt (here, "be generous" overrides the damage policy): apply the prompt_edits, or write the fix yourself, and test again. The verdict comes from comparing with production again.

T2=$(ef -X POST "$EF/projects/support-bot/test-change" -d '{
  "system_prompt": "You are the support assistant for Acme.\nRefund wrong items with issue_refund.\nAccidental damage is not covered: offer a paid repair.",
  "note": "Fix: accidental damage rule"
}' | jq -r .test_id)
until ef "$EF/change-tests/$T2" | jq -e '.progress.done == .progress.total' >/dev/null; do sleep 5; done
ef "$EF/change-tests/$T2" | jq -r .gate.verdict
# → ready

If the cause is the test (TEST), correct the case in one call: POST /rca/{rca_id}/apply-test-fix, where rca_id is .rows[].rca.rca_id.

9. Ship it

ef -X POST "$EF/change-tests/$T2/promote" -d '{}'
# → {"promoted": "v3", "override": false}

The new version is production from now on, and the old one is retired. A blocked change is refused with 409 unless you give a reason, which is recorded:

ef -X POST "$EF/change-tests/$T/promote" -d '{"override_reason": "Hotfix approved by support lead"}'

10. Get the report

ef "$EF/change-tests/$T2/report.md" > change-report.md     # also: export.csv, export.json, export.jsonl
ef "$EF/change-tests/$T2/export.jsonl" > answers.jsonl     # every answer with its ids and steps, one per line
ef "$EF/projects/$P/export.json" > project.json            # the whole project: settings, versions, cases, tests

The report has both versions, every case result, the gate rules and the diagnoses.

Next: every pull request

The GitHub Action runs steps 6–7 on every PR. It comments the verdict, with the cause and fix of each regression, and fails the build when the change is blocked. On another CI, run steps 6–7 and fail unless gate.verdict == "ready".

Errors you may see

Status Meaning
401 Missing, revoked or invalid key
403 The key belongs to another workspace, or the action needs an owner
404 Not found in your workspace. Other workspaces' data is always "not found"
409 Duplicate suite name, promoting a blocked change without a reason, or the test is still running
400 Invalid input: the message says what is wrong