NinjaEvalDocs← NinjaEvalGet startedDeveloper guideAPI quickstartFAQData & privacyCase study

NinjaEval Developer Guide

NinjaEval tests AI products before they ship. You set up your product once: how to run it, your test cases and a release gate. After that, every prompt, model or code change is compared case by case with production. NinjaEval flags what regressed, diagnoses why, and suggests the fix.

Also see the API quickstart, which onboards a product end to end with curl, and the FAQ. New here? Get started in 10 minutes. The live example is the MediBook demo in the app: open it from the project menu, or from Learn → Open the demo.

1. How NinjaEval runs your product

The quickest start is guided setup: create a project in the app and follow 5 short steps.

  1. Connect your product, and send a real message to check it works.
  2. Let NinjaEval suggest starter tests from your prompt and rules, then tick the ones to keep.
  3. Pick a release rule.
  4. Run your first baseline.

You choose one of two modes per project (Setup → Step 1).

Model + prompt. NinjaEval calls the model directly with your system prompt, parameters and tools (Groq, OpenAI or Anthropic). Tool results come from each test case's tool results, so no code of yours runs. This is the fastest way to test prompt and model changes.

Your endpoint. NinjaEval sends each test case to your running service, so your real code, tools and model keys are tested. Your API doesn't need to change. Pick its request format in Step 1:

Format Request NinjaEval sends Where the reply is read
NinjaEval (default) {"message", "system_prompt", "model", "version"} reply, plus tool_calls
OpenAI-compatible chat (also vLLM, LiteLLM, Groq, Azure) {"model", "messages": [system, user]} choices.0.message.content, plus …tool_calls
Custom your JSON template with {{message}}, {{system_prompt}}, {{model}}, {{version}}, and for conversations {{messages}}, {{history}} or {{conversation_id}} any dot path, e.g. output.text

With Custom, click Try it and then the field in the raw response that holds the reply; NinjaEval fills in the path for you. Tool calls are read from common shapes (name/input, OpenAI function.arguments, tool/args).

In the default NinjaEval format your service receives:

POST {your endpoint URL}
{"message": "Which medicine should I take?", "system_prompt": "...", "model": "...", "version": "v4"}

It must answer:

200 {"reply": "I can't advise on medicines, but I can book you a GP...",
     "tool_calls": [{"name": "check_slots", "input": {...}, "output": {...}}]}

tool_calls is optional. Include it if you want tool use to be checked. system_prompt, model and version are sent so your service can use the version under test. Ignore them if your deployment already is that version.

Conversations (multi-turn cases). A case can have follow-up messages. NinjaEval then sends each customer message after your bot replies to the one before, with the conversation so far, so your service must accept the history:

A multi-turn case on a custom format with none of these errors with a message that says what to add.

Every message of one conversation carries the same traceparent trace id, so Langfuse shows the conversation as one trace.

Connecting. The setup wizard asks what you're connecting:

Authentication for your endpoint. Under Authentication, choose API key and paste just the key (NinjaEval sends Authorization: Bearer <key>), or Custom header for APIs that use their own header (e.g. x-api-key).

2. Concepts

Concept Meaning
Project One AI product: how to run it, and its release gate
Version A snapshot of provider, model, parameters, system prompt, tools and code commit. One version is production
Test case What the user says (one message, or a scripted conversation of up to 10), the expected behaviour (for the AI judge), automatic checks (e.g. "must not mention ibuprofen", "must call check_slots"), optional expected tool calls with arguments, and tags (e.g. safety)
Run Every case of the suite on one version, N trials each
Change test Production vs a new version on the same suite, compared case by case and gated
Diagnosis For a failing case: the cause (prompt, model, tool, or the test itself), the evidence, and a fix

Multi-turn cases. In the case editor, ＋ Add a follow-up message turns a case into a scripted conversation: message 1, the bot replies, message 2 (sent with the conversation so far), and so on. Use them for what a single message can't test:

Or let an AI play the customer. Instead of scripted follow-ups, give the case a simulated customer: their goal ("cancel order #W5918442, ordered by mistake"), what they know ("Sofia Rossi, zip 78784; doesn't know her user id") and at most how many messages (2-10). The case's own message is always sent first. After each reply, an AI (Claude Haiku by default, EF_SIM_MODEL) writes the customer's next message: it answers what the bot asked using only what the customer knows, and it stops when the goal is reached or the bot clearly can't help. A scripted follow-up can't react to a bot that asked for the wrong thing; a simulated customer can. Its messages depend on the bot's replies, so they differ between versions and runs: use 2-3 trials for these cases. The debugger marks them 🎭; the judge and the diagnosis are told the goal. Quotas count the most messages the customer may send.

How they're graded and shown:

Scores and labels:

3. How a change test works

  1. Baseline. Production's latest fully scored run on the same version of your test suite, graded by the same AI judge, is reused. If there is none, production runs first. Adding or deleting a case changes the suite, so the baseline runs again next time. Editing a case does too, unless the edit keeps the earlier answers valid. Changing only its checks or expected tool calls grades production's stored answers again, with no model calls; changing only its title or tags keeps the results as they are. Either way the next test reuses production's run, and the case list says so after you save. Changing what the user says, the follow-ups, the tool results or the expected behaviour (the judge grades against it) means running again. Change tests that already ran keep their results.

  2. Run. Every case runs on the new version, N trials each, and the majority label counts. Rate limits and temporary errors are retried. A case that still can't be scored is marked errored, not failed.

  3. Grade. Each answer is graded in three ways:

    • your automatic checks;
    • the expected tool calls, with their arguments;
    • an AI judge (Anthropic Claude), which scores the answer against the behaviour you expect. The judge never grades its own model: if production or the new version is the judge's model (claude-sonnet-4-6), the whole test, both sides, is graded by another model (claude-opus-4-6), and the review says so. If a model in the test is from the judge's family (another Claude), you get a warning before you run and on the results: judges can favour answers like their own, so lean on the automatic checks and expected tool calls.

  4. Model swap. When a change touches only the model, the review starts with a Model swap card: production's model vs the new one on passing cases, average score, cost per case, cost per passing case (what a correct answer costs, failures included), the AI judge's own cost per case, typical (p50) and slow (p95) reply time and tokens per case, with one sentence on whether it's safe to switch (it follows the gate). Cost uses the list prices NinjaEval knows; for any other model it shows "—", not $0. Use it to answer "is the cheaper or newer model good enough?" before switching. The Markdown report has the same section.

  5. Behaviour. A 🧭 Behaviour card compares how the two versions went about the cases they both answered: model calls and tool calls per case, repeated identical tool calls, calls retried after a tool error, clarifying questions (polite "anything else?" closings don't count), answer length, refusals (and refusals on cases that don't expect one: tags like scope or abstain, or an "says it doesn't know" check, mark the ones that do) and handoffs to a human (and handoffs on cases that don't expect one: tags like escalation, or a check that calls the handoff tool). A line is flagged ⚑ when it moved noticeably. That isn't wrong by itself, but unexplained change is often the first sign of a bad change. Open a line to see the cases that changed most, and click one to debug it. A 🛡 Tool actions card watches what the agent did: tools count as writes by name (cancel_, modify_, book_, refund, send…). It flags a reply that says an action was done ("has been cancelled") when no write tool ran, writes made without a confirmation turn (the customer hadn't said yes to a question), new write actions and more tool errors. To fail a case on a false claim, add the check never claims an action it didn't take. If a tool's kind is guessed wrong, correct it in Scoring & gate → Tool kinds.

  6. Compare. Each case is regressed (pass or warn → fail), fixed (fail or warn → pass), got worse (pass → warn) or same. When trials disagree, the case is marked flaky. A case that fails on both but fails a check (or has a tool-call error) on the new version that production didn't is marked New failure, with what's new, e.g. a reply that says "cancelled" without calling the cancel tool. Both score the same (capped at 0.40), so without the flag it would look same. New failures don't block on their own, but the review, the model swap summary and the report list them.

  7. Gate. The default gate blocks any regression, an average score below 80%, or a failing safety-tagged case. You can change the rules in Step 3, and add rules per tag, e.g. "refunds cases: at least 90% pass" (up to 10). A tag rule with no cases tagged doesn't block. The report's By tag table shows, per tag, the cases, how many pass on each version, the average scores and the regressions; click a tag to filter the cases.

  8. Diagnose. Regressions are diagnosed automatically; any other failing case can be diagnosed on demand. The diagnosis rules out setup issues first, then names the layer at fault:

    • Setup (not the AI): Run broke, Test issue (its input or tool data), Expectation issue, Grading issue, Product decision (behaviour undefined).
    • System: Tool issue, Missing information, Prompt issue, Model issue (only when the instruction, tools and information were all present).
    • It compares the failing answer with the passing production answer, votes several times (more when unsure), and flags uncertain results for review.
    • The fix comes with an owner: a corrected test, exact prompt edits, a better tool description, or advice. Prompt and tool fixes are re-run to confirm they make the test pass. Test fixes are checked on the answers already collected, and must still fail a deliberately bad answer, so a "fix" can't quietly switch a test off.
    • Before blaming the model for a value it didn't use, it checks the value was in what the model could see.
    • Failing cases diagnosed as setup issues are left out of the true pass rate.

  9. Stage gates (optional). Under Scoring & gate → Stage gates, define quick stages that run only some cases, e.g. the template smoke = the cases tagged smoke, at least 80% pass, no regressions (up to 5 stages). When testing a change, pick Run the smoke stage only instead of Run all cases next to the trials, or pass stage: smoke to the GitHub Action (or "stage": "smoke" to the API) to run it on every pull request. A stage has its own Ready or Blocked verdict (CI fails on Blocked) but never ships: only the full suite and its release rules decide a release. Tag 10–20% of your most important cases for a smoke stage.

  10. Re-run one case (optional). In the case debugger, ↻ Re-run this case runs just that case again on the same two versions, reusing production's full baseline when the suite hasn't changed. You see its result in a minute or two instead of re-running the suite. It's a partial test: the gate shows Partial, it can't ship, and its runs never become a baseline. ▶ Run all cases runs the full test when you're ready. The API takes the same option: item_ids on POST /change-tests.

  11. Ship. Promote the new version to production. If the gate is blocked, promoting requires a written reason. Every change test has a report you can download as Markdown, CSV or JSON, or print to PDF.

4. API

Base URL: https://api.ninjaeval.com/api/v1/eng. All requests and responses are JSON. New here? Follow the API quickstart: 10 copy-paste steps from an empty workspace to a shipped change.

Authentication. Create an API key in the app: ⚙ Workspace & API keys (workspace owners).

PUT  /projects/{id}                     GET  /projects              GET /projects/{id}
POST /versions                          GET  /versions/{id}         GET /versions/{id}/diff[?against=]
POST /projects/{id}/datasets            GET  /datasets/{id}/cases   POST /datasets/{id}/cases
PUT  /cases/{item_id}                   DELETE /cases/{item_id}
POST /change-tests                      GET  /change-tests?project_id=   GET /change-tests/{id}
POST /change-tests/{id}/analyze         POST /rca/{id}/apply-test-fix
POST /change-tests/{id}/promote         GET  /change-tests/{id}/report.md | export.csv | export.json
POST /projects/{id}/test-change         CI shortcut: copy production, apply overrides, start the change test
PUT  /projects/{id}/endpoint-headers    write-only; {} removes them
POST /projects/{id}/try                 send one message to the product (or a draft version); result via GET /probes/{id}
POST /projects/{id}/suggest-cases       propose starter tests from your prompt / description; result via GET /probes/{id}
GET  /projects/{id}/langfuse/traces[?only=flagged|errors|low_scores]   recent Langfuse traces (Langfuse connected)
GET  /projects/{id}/langfuse/traces/{trace_id}/case                    a draft case from one trace
DELETE /projects/{id}                   body {"confirm": "<id>"}: deletes the project and everything in it (owners; not the demo)
GET  /me                                who you are and what you may do in this workspace

POST /projects/{id}/test-change takes any of system_prompt, model, provider, params, tools, code_sha, note, trials, stage. Fields you leave out keep production's value. It returns test_id. Poll GET /change-tests/{test_id} until progress.done == progress.total. The verdict is gate.verdict (ready or blocked).

5. CI

GitHub: add the action sohum-monk/evalforge-action@v1.

on: pull_request
permissions: { contents: read, pull-requests: write }
jobs:
  change-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: sohum-monk/evalforge-action@v1
        with:
          api-key: ${{ secrets.EVALFORGE_API_KEY }}
          project: my-bot
          prompt-file: prompts/system.md     # optional: the prompt as changed in this PR
          # stage: smoke                     # optional: run only a stage gate (section 3, step 9)

The action does three things:

Any other CI: call POST /projects/{id}/test-change, poll, and fail unless gate.verdict == "ready".

From your coding agent (MCP): the ninjaeval-mcp server lets Claude Code, Cursor or any MCP client test a change before calling it done. The agent edits the prompt, model or tools and calls test_change. It gets the verdict and each regression's cause and fix, applies the fix and tests again. It can't ship: promoting stays with you. Create a workspace key, then:

claude mcp add ninjaeval --env NINJAEVAL_API_KEY=efk_... -- uvx ninjaeval-mcp

Tools: whoami, list_projects, get_project, test_change, get_change_test, diagnose_cases, get_report. It runs locally over stdio, and one session can start at most 10 change tests (NINJAEVAL_MCP_MAX_TESTS).

6. Your data

Test safely

In endpoint mode NinjaEval calls your real service, so a test case that says "cancel my order" makes your bot cancel an order, in whatever database it's connected to. Every test request carries the X-NinjaEval-Test header: use it to send tests to test data (a sandbox database, test payment keys, emails that are only logged).

db = sandbox_db if request.headers.get("X-NinjaEval-Test") else prod_db

Then tell NinjaEval, either per answer with "sandbox": true at the top level of the ninjaeval block (SDK: ninjaeval.set_context(sandbox=True)), or once for the project with Tests run on a sandbox in Scoring & gate. Until then, a test report warns whenever a test ran a write action (cancel, refund, book, send…) on your service. Model-mode projects aren't affected: their tools are mocked by the test case.

Show NinjaEval inside your bot

By default NinjaEval sees only what your endpoint returns: the reply, and any tool calls your response format includes. To let it grade every tool call and show each LLM call in the debugger, next to the same call on production, add an optional ninjaeval block to your JSON response.

Only for tests. Every request NinjaEval sends has the header X-NinjaEval-Test and a W3C traceparent. Add the block only when that header is present, so real users never receive it.

{
  "reply": "Your order was delivered, so it can't be cancelled. I can start a return instead.",
  "ninjaeval": {
    "version": 1,
    "tool_calls": [
      {"name": "find_user", "arguments": {"name": "Yusuf Rossi", "zip": "19122"}, "result": "user_9620", "duration_ms": 35},
      {"name": "get_order", "arguments": {"order_id": "#W2378156"}, "result": {"status": "delivered", "items": 5}},
      {"name": "cancel_order", "arguments": {"order_id": "#W2378156"}, "error": "Non-pending order cannot be cancelled"}
    ],
    "turns": [
      {"model": "gpt-4o-mini", "messages_in": [{"role": "user", "content": "Please cancel my order #W2378156"}],
       "output": "", "tool_calls": ["find_user"], "prompt_tokens": 3600, "completion_tokens": 40, "latency_ms": 820}
    ]
  }
}
Field Meaning
version 1. Other versions are reported as a problem, and NinjaEval still tries to read the block
tool_calls[] Every tool your bot ran, in order: name (required), arguments (an object or a JSON string), and result or error, plus optional duration_ms. These replace any tool calls read through your response format, and they're what checks and expected tool calls are graded on
turns[] One entry per model call: model, messages_in (what it was sent), output, tool_calls (the names it asked for), prompt_tokens, completion_tokens, latency_ms, and optionally wait_ms (time the call waited for your own rate limit or queue; left out of the reply time, so a throttled test environment doesn't look slow). Shown in the debugger; their tokens count toward cost
Optional, v1.1 (still "version": 1) On turns[] and tool_calls[]: attempt (2, 3… for a retry of the same step), agent (which agent or sub-agent made it) and parent (the agent that handed it the work). On turns[] also params (e.g. {"temperature": 0, "max_tokens": 800}) and model_version (the exact version your provider reported). At the top level: session_id (your own conversation or session id), intent (the goal as the agent understood it) and sandbox (true when this test ran on test data; see Test safely). The Trace tab shows retries, sub-agents, parameters, the session and the goal; the answers export (JSONL) includes them. Older NinjaEval versions ignore them

Limits: 50 tool calls and 20 turns per reply. Long texts are cut. Redact anything you don't want stored, such as emails or card numbers, before you add it. Try it in the setup wizard shows what arrived (for example "✓ Also received: 3 tool calls · 4 LLM turns"), and lists anything it ignored.

Python SDK (records it for you, no dependencies): pip install ninjaeval (PyPI, MIT licence).

import ninjaeval
ninjaeval.instrument()                     # records OpenAI, Anthropic and LiteLLM chat calls

@ninjaeval.tool                            # records arguments, result or error, and duration
def get_order(order_id: str): ...

@app.post("/chat")
def chat(req):
    with ninjaeval.capture(req.headers) as run:     # active only for NinjaEval test requests
        reply = agent.run(req.message)
    return run.attach({"reply": reply})             # adds the block for tests, nothing otherwise

capture(headers, redact=fn) masks text before it leaves your service. ninjaeval.record_turn(...) and ninjaeval.record_tool_call(...) cover clients and tools the SDK doesn't wrap. With OpenTelemetry installed, each call is also a span in the test's trace. The SDK never raises into your code.

The same checks, anywhere. The SDK ships NinjaEval's deterministic checks (the same code the server grades with), so a check passes or fails the same way in your unit tests, in CI, on live replies and in a change test. A case's checks come with it from GET /datasets/{id}/cases.

import ninjaeval

reply, tool_calls = run_my_bot("Please cancel order W1")       # tool_calls as the SDK records them, or NinjaEval's
ninjaeval.assert_checks(['not: "guaranteed"', "calls get_order", "never claims an action it didn't take"],
                        reply, tool_calls)                       # AssertionError naming each failed check and why
ok = ninjaeval.check(case["checks"], reply, tool_calls)         # True / False, e.g. a guardrail on live replies
results = ninjaeval.run_checks(case["checks"], reply, tool_calls)   # [{label, passed, detail}]

By hand, with no SDK:

def respond(request, reply, tool_log, turn_log):
    body = {"reply": reply}
    if request.headers.get("X-NinjaEval-Test"):        # tests only
        body["ninjaeval"] = {"version": 1, "tool_calls": tool_log, "turns": turn_log}
    return body

Connect Langfuse

If you already trace your bot in Langfuse, connect it in Product & model → Langfuse: choose the region (EU, US, Japan, or your own URL) and paste the project's public key (pk-lf-…) and secret key (sk-lf-…), from Langfuse → project settings → API keys. NinjaEval checks the keys, stores them encrypted, and only ever reads traces.