Docs← NinjaEvalGet startedDeveloper guideAPI quickstartFAQData & privacyCase studyWhat NinjaEval stores, where it lives, who can see it, which other companies receive it, and how you control and delete it. NinjaEval is in beta: where something isn't built yet, this page says coming.
Short version: NinjaEval stores your test setup and every answer your product gives in a test, shows them only to your workspace, sends test inputs and answers to Anthropic so the AI judge can grade them, and deletes them when you ask. Don't put real customer data in test cases unless you're comfortable with that.
| What | What it holds |
|---|---|
| Your account | Your Google account id, email address and name, when you first and last signed in. Nothing else from Google |
| Projects | Name, description, mode (model, endpoint or SDK), your endpoint URL and request format, release-gate rules, rate limits |
| Versions | Label, provider and model, settings (temperature, max tokens), the system prompt, the tool definitions, a commit sha and a note |
| Test cases | The customer message(s), expected behaviour, checks, tags, the tool results the test provides (mocks) and the expected tool calls |
| Answers (every execution) | The reply, the whole conversation for multi-turn cases, every tool call with its arguments and results, the model turns (the messages sent to the model and its reply, tokens, timings), the bot's session id and reading of the goal (if it reports them), cost, errors |
| Grades | Check results, the tool-call comparison, the AI judge's score and its reasoning for each step |
| Runs, change tests, reports | Which version ran which suite, summaries, the Ready / Blocked verdict, when it shipped |
| Diagnoses | The reasoning, the cause, quotes from your prompt, tools and answers, the suggested fix and the result of re-running it |
| Setup helpers | The message and reply of a "Try a message", and suggested starter tests |
| Your calls on answers | Right / wrong and a note, used for calibration |
| Feedback | What you send with the feedback button and 👍 / 👎 on diagnoses, with your email |
| Usage events | Which steps you used (for example "ran a change test"), with your workspace and account id. Used only to improve NinjaEval |
| API keys | Only a hash of each key and its first characters. The key itself is shown once and never stored |
| Endpoint headers | Encrypted (see below) |
| Langfuse keys | Encrypted, like endpoint headers |
Public totals. The home page shows counts added up across all workspaces: answers graded, changes tested, regressions caught and root causes found. They are numbers only, rounded down above 100, and never include a name, a workspace, a prompt, a test or an answer. Cases whose diagnosis says the test itself was at fault aren't counted.
Langfuse traces are not stored. When you open a test's trace, or a diagnosis looks inside your bot, NinjaEval reads the trace from your Langfuse at that moment. What is kept: a production trace you choose to turn into a test case (it becomes a normal test case), and diagnoses, which may quote what they read.
Google Cloud, region us-central1 (United States). The app and API run on Cloud Run; data is in a Cloud SQL PostgreSQL database.
In transit: HTTPS to the app and the API; the database accepts encrypted connections only and has no open network access.
At rest: encrypted by Google Cloud by default. Database backups are kept for 7 days.
Your workspace. Each person who signs in gets a private workspace. The server works out your workspace from your sign-in or API key and checks it on every request; anything outside your workspace is "not found".
Roles. The workspace owner can do everything, including creating and revoking API keys. API keys (for CI) can read, set up and run tests in their own workspace, but can't manage keys, delete a project, save endpoint headers or connect Langfuse: those need a person signed in.
The MediBook demo is the one shared workspace. Anyone can read it without signing in, so cases or changes you add there are visible to everyone until the nightly reset deletes them.
The NinjaEval team can access the database to run the service, debug problems and help when you ask. We don't look at your workspace's content otherwise.
| Company | When | What it receives |
|---|---|---|
| Google Cloud | Always | Hosting and the database: everything above |
| Anthropic (Claude) | Every test and diagnosis | The AI judge and the diagnosis: the test case (messages, expected behaviour, checks, tool data), your product's answers, tool calls with arguments and results, your system prompt and tool descriptions. A diagnosis also sends what it read inside your bot (your Langfuse trace or the model turns your service reported). Suggested starter tests send your prompt and tools. Also runs the model when a version uses an Anthropic model in model mode |
| Groq | Only when a version uses a Groq model in model mode | The test messages, your system prompt and tool definitions, and the test's tool data, to run the model |
| Google Sign-In | When you sign in | Google confirms who you are; NinjaEval receives your account id, email and name |
| Langfuse | Only if you connect it | Nothing is sent to it: NinjaEval reads your traces with your keys |
| GitHub | Only if you use the NinjaEval GitHub Action | It runs in your repository: it sends your prompt file to NinjaEval and posts the verdict, a summary and the change report as a PR comment |
In endpoint mode, NinjaEval sends your test messages to the URL you configure; that is your own service.
NinjaEval never needs your model keys: in endpoint mode it calls your service, and your service calls its model. Headers you save for your endpoint (for example an API key) are encrypted before they're stored, used only when calling your endpoint, and never shown again: the app shows the header name and the last 4 characters. Exports never include them. The same goes for Langfuse keys.
Today:
Delete a case: removes it from the suite, with every answer recorded for it, their grades, diagnoses and your calls.
Delete a project (Product & model → Delete project): removes the project with its versions, test cases, runs, answers, grades, change tests, diagnoses, setup-helper results, saved endpoint headers, the Langfuse connection and your right / wrong calls on answers. It can't be undone.
Export first (Product & model → Export project): settings, versions, test cases, change tests with verdicts and your calls, never headers or keys. Each change test's report also has CSV, JSON and an Answers (JSONL) export with every answer and step.
Delete your workspace or account: on request, write to evalforgeai@gmail.com. We delete it within 30 days.
Answers are kept 90 days by default. Every night, stored answers older than your workspace's retention lose their text: replies, conversations, tool calls with their arguments and results, model turns, and the AI judge's notes about them. Verdicts, scores, checks, summaries, reports and diagnoses stay, so your history and trends keep working. The workspace owner can set 7 to 365 days in ⚙ Workspace & API keys → Data retention. A test whose answers were removed says so in the debugger; run it again to see answers. Database backups keep removed data for up to 7 more days.
Mask data before it reaches NinjaEval. The Python SDK's redact= hook runs on everything it records:
ninjaeval.capture(req.headers, redact=mask_emails). See the SDK README.
Answer tests differently. Every request NinjaEval sends carries an X-NinjaEval-Test header. Your service can
use it to answer from test data, skip writing to your own systems, or leave out fields you don't want stored.
Use test data. Cases and tool data are yours to write: use made-up names, orders and numbers.
[email], [phone] and [card] before an
answer is stored or sent to the AI judge, including tool arguments and results, model calls, diagnosis re-runs, "Try
a message" replies and Langfuse internals read for a diagnosis. Order ids, zip codes, amounts and dates stay. Checks
still run on the real answer, so a check for an email still works. It applies to new answers, not stored ones.
Masking is pattern-based. It doesn't catch names or addresses, so use test data for those.See also the privacy policy. Questions: evalforgeai@gmail.com.