Get started in 10 minutes

From sign-in to your first ready / blocked verdict. You need a Google account, plus one of:

• your bot's HTTP chat endpoint (any JSON format), or
• an OpenAI-compatible model key (Groq, OpenAI, OpenRouter…) and your bot's instructions, or
• just your bot's instructions (system prompt): NinjaEval runs the model for you.

Step	Time
1. Sign in and name your project	1 min
2. Connect your bot and send one message	2 min
3. Add 5–10 tests	3 min
4. Pick a release rule and run the baseline	2 min
5. Test your first change	2 min (+ run time)

1. Sign in and name your project (1 min)

Open ninjaeval.com and Sign in with Google. You get your own private workspace, and setup opens straight away.

• Start from: your own product, or an example (customer support, booking).
• Product name, and what it does in a sentence or two. NinjaEval uses it to suggest tests.
• How should NinjaEval reach it? 🔌 Your endpoint (your running service gets each test as an HTTP request: it tests your real code and tools) or 🧠 Model + prompt (no code: NinjaEval calls the model with your instructions).

2. Connect your bot and send one message (2 min)

Your endpoint asks What are you connecting?

• 🤖 An AI model API: pick Groq, OpenAI, OpenRouter or Other, paste just the key, pick a model, paste your bot's instructions.

• 🔌 Your own app: your URL, how it authenticates (API key, custom header or none), and its request format (NinjaEval, OpenAI-compatible, or Custom, where you click the field that holds the reply).

• 🧪 An example: our demo bots, nothing to set up.

Model + prompt asks for your bot's instructions, the model, and optionally its tools.

Then type a real message in 💬 Try it and Send. You should see the reply and ✓ Connected. If your service reports its internals, Try it also says ✓ Also received: N tool calls · N LLM turns (how).

Rate-limited model or API (for example a free tier)? Tick My API has a low rate limit: tests then run one at a time and wait longer for each reply.

3. Add 5–10 tests (3 min)

A test is what a user says plus what a good answer does. Three ways to add them:

• ✨ Suggest tests for me. NinjaEval reads your instructions and proposes happy paths, every important rule, and attempts to break it. Keep the good ones.

• ✍️ Write one. Start with the rules that matter most (refunds, safety, what it must never promise).

• 📋 Paste from a spreadsheet: title, user says, good answer, tags.

The tests you've added are listed underneath. Later, in Test cases, you can add automatic checks (says / doesn't say / calls a tool), expected tool calls and follow-up messages (to test a whole conversation), or import real conversations from Langfuse, starting with the ones that went wrong in production. Tag safety rules safety: they must always pass.

4. Pick a release rule and run the baseline (2 min)

• Release rule: Standard blocks any regression, an average under 80%, or a failing safety test. Strict and Lenient are stricter and looser. Later, in Scoring & gate, you can add rules per tag (e.g. "refunds cases: at least 90% pass") and a quick smoke stage for pull requests (how).

• First run: NinjaEval runs every test on what's in production today. This is the baseline every change is compared with. It takes about a minute (longer with one test at a time).

5. Test your first change (2 min, plus run time)

Click ＋ Test a change, change something (a prompt line, the model, a tool), and ▶ Run tests on this change. Each test runs on the new version and is compared with production, case by case.

You get a verdict: Ready, or Blocked with the rule that failed. Changed only the model? The review starts with a Model swap card: passing cases, score, cost, speed and tokens side by side. A case that fails on both versions but in a new way is flagged New failure. Then click any case:

Tab	Shows
Overview	Both answers side by side (for a multi-turn case, both conversations), the checks that failed, the judge's note
Tool calls	Each tool call production and the new version made, marked same or different, and against the expected calls
Trace	A timeline: every step side by side, turn by turn, with the first difference named; click a step for the LLM calls behind it. Map shows the same as a mind map. The case list shrinks so the trace gets the width
Root cause	Why it failed. Regressions are diagnosed automatically; for any other failure click 🔎 Diagnose

The diagnosis rules out setup issues first (a broken run, a wrong test, check or expectation) before blaming the prompt, a tool, missing information or the model. It proposes a fix, such as an exact prompt edit or a corrected test, and re-runs the test to confirm it. Use ✎ Apply to the prompt & re-run or Apply to the test case, then rate the diagnosis 👍 / 👎.

Fixed a test, or want to check one case again? ↻ Re-run this case in the case view runs only that case on both versions, in a minute or two. It's a partial test and can't ship: use ▶ Run all cases before you ship.

When the verdict is Ready, Continue to ship makes the new version production. It's also the new baseline.

Next

• Every pull request: create a key in Workspace & API keys and add the GitHub Action, which comments the verdict and fails the build when a change is blocked. Add stage: smoke for a quick check on each pull request, and run the full suite before release.

• Show NinjaEval inside your bot: add the ninjaeval block to your response, or pip install ninjaeval and three lines of the Python SDK, to grade every tool call and see every LLM call.

• Already on Langfuse? Connect it: each test's trace appears in the debugger, the AI diagnosis reads it, and real conversations become tests.

• Questions: the FAQ, the 💬 Feedback button, or evalforgeai@gmail.com.