Privacy policy

Short version: we store what you put into NinjaEval so we can test your AI product, we show it only to your workspace, we send test inputs and answers to AI providers to run and grade them, and we delete it when you ask.

What we collect

• Your Google account basics when you sign in: name, email address and a Google account id. Nothing else from Google.
• What you put into NinjaEval: project settings, system prompts, tool definitions, test cases, and the endpoint URL and request format of your product.
• Results: the answers your product gave to each test, tool calls, scores, diagnoses and reports.
• Endpoint credentials (request headers such as an API key) are stored encrypted, are never shown again after you save them, and are sent only to your endpoint.
• Usage events and feedback: which steps you used (for example "ran a change test") and anything you send through the feedback button or 👍 / 👎 ratings. We use these only to improve NinjaEval.

Who can see it

• Only you and members of your workspace. Every request is checked on our server.
• The MediBook demo is shared: test cases or changes you add there are visible to other demo visitors and are deleted every night.
• The NinjaEval team can access data to operate the service and help when you ask us to.
• Public totals: our home page shows counts added up across all workspaces (answers graded, changes tested, regressions caught, root causes found). They are numbers only, rounded, and never include a name, a workspace, your prompts, your tests or your answers.

Who processes it

• Google Cloud (us-central1, United States): hosting, database, encryption at rest.
• Anthropic: grades answers (the AI judge), writes diagnoses and suggests tests. Test inputs, answers, your prompt and tool definitions are sent to it.
• Groq: runs the model in "Model + prompt" mode. Test inputs, your prompt and tools are sent to it.
• Your own endpoint, in "Your endpoint" mode: we send your test inputs to the URL you configure.

Don't put personal or confidential data in test cases unless you're comfortable with these providers processing it.

Security

HTTPS everywhere; the database is encrypted at rest, only accepts encrypted connections and is on a private network; model keys are held in a secrets manager; API keys are stored only as hashes. NinjaEval is in beta and does not yet hold certifications such as SOC 2.

Keeping and deleting

• We keep your setup and results while your account exists. The text of test answers (replies, conversations, tool calls, model calls) is deleted after 90 days by default; your workspace owner can choose 7 to 365 days. Verdicts, scores and reports stay. Your workspace owner can also turn on masking of emails, phone numbers and card numbers in answers. Delete a project at any time in Product & model → Delete project.
• To delete your account and everything in your workspace, write to evalforgeai@gmail.com; we'll do it within 30 days.
• Database backups are kept for up to 7 days.

Contact

Questions or requests: evalforgeai@gmail.com.