Docs← NinjaEvalGet startedDeveloper guideAPI quickstartFAQData & privacyCase studyWhat's the fastest way to get started? Sign in and create a project. Guided setup walks you through 5 steps, about 5 minutes:
Get suggested starter tests, written from your prompt's rules, including attempts to break them. Or paste tests from a spreadsheet.
Pick a release rule (Strict, Standard or Lenient).
A project that isn't fully set up shows ✨ Continue guided setup on its home page.
I don't have an endpoint ready. Can I still try endpoint mode? Yes. In guided setup (Connect & try), pick an example. The project is created in your own workspace, and you can delete it any time.
How do I delete a project? Open Product & model (Setup step 1) and click Delete project…, then type the project id to confirm. This removes its versions, tests, runs, reports and saved headers, and can't be undone. The shared demo can't be deleted.
Are the suggested tests good enough? They are a strong start: one per important rule, the main happy paths, safety attacks and an edge case, each with automatic checks. Review them before adding, and make them stricter over time in Test cases.
What's the difference between "Model + prompt" and "Your endpoint"? With Model + prompt, NinjaEval calls the model itself, using your system prompt and tools. Tool results come from each case's tool results (mocks). This is the fastest way to test prompt or model changes. With Your endpoint, NinjaEval POSTs each case to your running service, so your real code and tools are tested. MediBook uses endpoint mode.
Can the AI judge favour its own model? It could, so NinjaEval prevents the worst case: when a model in the test is the judge's own model, the whole test is graded by a different model, on both versions (so production runs again rather than reusing answers the usual judge graded). When a model is from the same family (another Claude), you see a warning before you run and on the results; the automatic checks and expected tool calls aren't affected by it.
How far can I trust the AI judge? Check it against yourself. In a test's debugger, mark answers ✓ right or ✗ wrong ("Your call", next to each answer). Scoring & gate → Calibration then shows how often NinjaEval's result matches your calls, and the AI judge's own grade before checks. It also lists the answers it passed that you called wrong (add a check or tighten the expected behaviour) and the ones it failed that you called right (the test may be too strict). About 20 calls give a useful number. Your calls never change a result. Review answers (in the sidebar) lists the answers most worth a call: regressions, checks that overrode the AI judge, borderline and flaky results, and a few random passes (only a person checking passes can find wrong answers NinjaEval passed). A few minutes after each test keeps the number honest.
What makes a case pass? The judge's score must be at least 0.75, every automatic check must pass, and the expected tool calls must match (when set). A failed check or a tool mismatch is a hard fail.
Can a test be a whole conversation, not one message?
Yes. In Test cases, open a case and click ＋ Add a follow-up message (up to 10 messages). NinjaEval sends each
message after your bot replies to the one before, with the conversation so far, and grades the whole conversation:
use it for memory ("the order number from message 1"), rules under pressure, and "confirm before you act". Your
endpoint has to accept the history: the NinjaEval and OpenAI formats do, and custom formats add {{messages}} or
{{conversation_id}}. Each message counts as one execution.
What do Regressed / Fixed / Got worse mean?
The default gate blocks any regression, an average below 80%, or a failing safety-tagged case.
You can change it in Step 3.
Why does a test fail on both versions? Usually either the product never handled it, or the test itself is wrong. Click Diagnose. If the cause is Test issue, apply the suggested test fix. Not every failure is the product's fault.
How does the diagnosis decide what's wrong? It works in layers, ruling out "not the AI's fault" first:
It reconstructs what each version relied on (prompt lines, tool descriptions, tool results), comparing the failing answer with the passing production answer.
Setup or system? Is the test's input, expectation or grading wrong, or is the right behaviour undefined? These are setup issues: fix the test, not the product.
Otherwise, which layer: a tool, missing information, the prompt, or the model. It's only called a model issue when the instruction, tools and information were all there.
Where it can, it re-runs the test with the suggested fix (or a stronger model) and shows whether that passes. A suggested test fix is re-graded on the answers you already have, and must still catch a deliberately bad answer; if it wouldn't, it's marked Not recommended.
My API (or its model provider) has a low rate limit. Can NinjaEval go slower? Yes. In Product & model → Rate limits (or My API has a low rate limit in the setup wizard), choose One at a time, and allow a longer wait per reply (up to 10 minutes). Each finished test starts the next, so a run takes longer but doesn't fail on rate limits. Free model tiers usually need this.
A case says "Errored". What does that mean? The new version didn't answer at all: your endpoint returned an error, or the model produced an invalid tool call, on every trial. It counts as finished, and it blocks the release ("Every case gets an answer"), because a version that crashes on a test can't ship. It's diagnosed automatically: a broken run (rate limit, timeout, 5xx) is flagged as Run broke (re-run it), and an invalid tool call as a Model issue.
How do I test that my bot can't be talked out of its rules?
In Test cases, click 🛡 Safety pack. It suggests attack cases fitted to your bot's tools and system prompt:
prompt injection, extracting its instructions, role-play jailbreaks, harmful requests, other people's data, a fake
developer, a planted link, rules worn down over several messages, acting without the usual checks, and (in model mode)
instructions hidden in a tool result. They're grouped by the OWASP Top 10 for LLM applications and tagged safety,
so the release gate blocks a change that fails one. Review each case's expected behaviour after adding it.
Which checks can a test have?
Tools: calls a tool, doesn't call a tool, calls tools in order (others may come in between), at most N tool calls, and valid tool arguments (each call matches the tool's parameters: required ones, types, allowed values).
Actions: never claims an action it didn't take.
Text is matched as plain, case-insensitive text (not a pattern). Rules that depend on meaning, such as "doesn't promise 20-minute delivery", belong in the expected behaviour, which the AI judge reads.
How reliable is the diagnosis? Each step is decided by several independent votes; if they disagree, it asks more times and flags the result Needs a human look when it's still unsure. A fix marked ✓ Confirmed by re-run has made the test pass on two re-runs in a row, and kept two other passing cases passing. Treat the rest as a strong hint, then re-run, and the numbers will tell you.
What is the true pass rate? Failing cases diagnosed as setup issues (a broken run, a wrong test, expectation or check, or undefined product behaviour) say nothing about the AI, so the true pass rate leaves them out. It's shown on the review screen and in the report whenever there are any.
Why did the baseline re-run when I only changed a test? Results only compare on the same suite. Editing, adding or deleting a case bumps the suite version, so the next change test runs the baseline again. Except when an edit keeps the earlier answers valid: changing only a case's checks or expected tool calls grades the stored answers again (no model calls), and changing only its title or tags keeps them as they are, so the next test reuses production's run. Changing what the user says, the follow-ups, the tool results or the expected behaviour (the AI judge grades against it) means running again. Change tests that ran before you added a case stay as they were: the new case is listed in a note at the top ("added after this test ran"), not shown as running, and their verdict doesn't change.
Can production failures become tests? Yes, with Langfuse connected: Test cases → Import from Langfuse → ⚠ Went wrong lists recent traces where something errored or that users rated badly (a 👎 recorded as a score). One click turns each into a test, conversations included.
Which of several models should we use? Use Compare models in the sidebar: enter 2 to 4 models. Each runs your suite against one production run and is graded by the same judge, so they're compared on the same answers. You get one table (each model's verdict, passing cases, score, cost and reply time, and its regressions), a case-by-case grid, and a line naming the cheapest model that passes your gate. Each model is also an ordinary change test you can open, debug and ship.
Is a cheaper or newer model good enough? Test a change that switches only the model. The review then opens with a Model swap card: passing cases, score, cost per case, reply time (typical and slow) and tokens, side by side, and whether it's safe to switch on your suite. Cost uses list prices NinjaEval knows; for other models it shows "—" rather than a guess.
Can I run a quick check on every pull request and the full suite before release?
Yes: set up a stage gate (Scoring & gate → Stage gates, e.g. smoke: the cases tagged smoke, at least 80%
pass). Run it from Test a change (pick Run the smoke stage only next to the trials), or with stage: smoke in
the GitHub Action. It has its own
Ready/Blocked verdict, and only the full suite can ship.
Can I re-run just one case? Yes: open the case and click ↻ Re-run this case. It runs only that case on both versions (production's baseline is reused when the suite hasn't changed), so you see the result quickly and spend less of your model quota. It's a partial test: it can't ship and never becomes a baseline. To ship, run every case with ▶ Run all cases.
What are trials? Each case runs N times and the majority label counts. When trials disagree, the case is marked flaky. Use 3 for important releases, and 1 for free-tier providers.
Can I compare with something other than production?
Yes, through the API: pass baseline_version_id to POST /change-tests.
How do I use it in CI?
On GitHub, add the action uses: sohum-monk/evalforge-action@v1 (github.com/sohum-monk/evalforge-action). It tests the PR against production,
comments the verdict with the cause and fix of each regression, and fails the job when the gate says blocked.
Any other CI makes one call with an API key: POST /projects/{id}/test-change with what changed (system_prompt, model,
code_sha). It copies production and starts the test. Then poll GET /change-tests/{id} until progress.done == progress.total,
and fail unless gate.verdict == "ready".
Can my coding agent (Claude Code, Cursor) use it?
Yes, through the NinjaEval MCP server. Create a workspace API key, then in Claude Code:
claude mcp add ninjaeval --env NINJAEVAL_API_KEY=efk_... -- uvx ninjaeval-mcp (other clients: "command": "uvx",
"args": ["ninjaeval-mcp"] with the same env). The agent calls test_change after editing the prompt, model or tools,
reads the verdict and each regression's cause and fix, and tests again. It can't promote a change, and one session
starts at most 10 tests. Details: Developer guide.
Does my endpoint have to use NinjaEval's request format?
No. In Step 1 (or the guided setup), pick OpenAI-compatible chat or Custom. Custom means your own JSON
body with {{message}}, plus where the reply is. Use Try it and click the field that holds the reply.
My endpoint needs an API key. How do I give it to NinjaEval?
Under Authentication, choose API key and paste just the key; NinjaEval adds Bearer itself. Use Custom header
if your API wants its own header, e.g. x-api-key.
How much does it cost? NinjaEval is free during the beta. To keep it available to everyone, each workspace can start up to 30 change tests and 1,500 test executions a day (one execution is one test case run once). Your current usage is on the ⚙ Workspace & API keys page. Need more? Use the Feedback button.
What happens to my data? See the privacy policy and terms. In short, your workspace is private. Test inputs and answers are sent to the AI providers that run and grade them, and you can delete a project at any time, or ask us to delete everything.
How do I send feedback? Use the 💬 Feedback button (bottom left), or rate any diagnosis with 👍 / 👎. We read everything.
"N runs errored": is my product bad? No. Errored means it couldn't be scored. Read the case's error text:
429 / rate limit: the model provider is throttling. Wait and re-run, or use 1 trial.404 from the provider: the model was renamed or retired. Pick another model in your change.endpoint 401/403: your endpoint rejected the key. Compare the saved key's last 4 characters with your real key, or paste it again under Authentication.endpoint 5xx or a timeout: your service failed or took longer than 90 seconds.A test-suite name already exists (409)? Suite names are unique per workspace. Pick another name.
"Not scoped to a workspace" from Anthropic?
Use a workspace API key from platform.claude.com → Settings → API Keys, not a user-level sk-ant-usr- key.
Can I find every answer that said something, or called a tool?
Yes: Search answers in the sidebar searches the project's last 90 days of answers by text (in the reply, the
conversation or the question), a tool it called, its result (pass, warn, fail, error), the diagnosis cause, a tag
or a version. Filters combine; click a hit to open it in its test. The API: GET /projects/{id}/search?q=…&tool=…&label=…&cause=…&tag=…&version=….
Can I get my data out, and join it to our own traces?
Yes. Each change test's report has CSV and JSON exports, with one row per case. Each row carries the ids
(test, run, answer, trace) and the raw numbers: label, score, the AI judge's own grade, failed checks, tool calls,
tokens, cost, reply time, trials and your call. Answers (JSONL) has every answer with every step (model calls,
tool calls with arguments and results, the conversation), one per line: ready for a warehouse or to replay. Its
trace_id is the W3C trace id NinjaEval sent your endpoint, so it matches your own tracing. Export project
(Product & model, next to Delete) downloads settings, versions, test cases, change tests with their verdicts, and
your calls. It never includes saved headers or keys.
Who can see my prompts and results? Only you and the people in your workspace.
The server works out your workspace from your sign-in and checks it on every request. Anything outside your workspace is "not found".
CI uses per-workspace API keys, which are stored only as hashes.
What can I do in the MediBook demo? The demo is the one shared workspace.
Signed in, you can run the challenges, test your own changes and add cases: up to 5 change tests, 10 cases and 15 diagnoses a day.
Only admins can edit its setup or promote.
Visitor changes are removed every night, so the example stays the same for everyone. Your own workspace has none of these limits.
How do I get an API key for CI?
Workspace owners: in the sidebar, open ⚙ Workspace & API keys, then create a key. It's shown once.
Send it as Authorization: Bearer efk_….
Where is my data stored? In an encrypted database on Google Cloud (us-central1, United States). Test inputs and answers are also sent to the AI judge (Anthropic Claude) for grading and, in model mode, to the model provider you choose. The full list of what's stored and who receives it: Data & privacy.
What does NinjaEval keep, and for how long? Your setup (prompts, tools, test cases) and every answer from your tests: the reply, the conversation, tool calls with their arguments and results, and the model turns. Langfuse traces are read live, not stored. Answers lose their text after 90 days by default (the workspace owner can set 7-365 days in ⚙ Workspace & API keys); verdicts, scores and reports are kept. Delete a project in Product & model; to delete your whole workspace, write to evalforgeai@gmail.com.
Can I keep personal data out?
Yes. Use made-up data in test cases, mask fields with the SDK's redact= hook, or have your service answer
differently when it sees the X-NinjaEval-Test header. You can also turn on Mask personal data in answers (⚙
Workspace & API keys, off by default): emails, phone numbers and card numbers are masked before answers are stored or
graded.
Is NinjaEval certified (SOC 2, ISO, HIPAA)? Not yet. Data & privacy lists the security measures that are in place.
What does one run do? For each case and trial: one conversation with your product (your endpoint or your chosen model), plus the AI judge (about 2–4 calls). Each diagnosis is 3 judge calls. Regressions are diagnosed automatically; everything else is on demand. NinjaEval is free during the beta.
My Mac's Python says CERTIFICATE_VERIFY_FAILED when calling the API.
python.org Python on macOS ships without certificates. Run Install Certificates.command, or set
SSL_CERT_FILE=$(python -c 'import certifi;print(certifi.where())').