Supabase Evals is our open-source benchmark and framework for measuring how well AI coding agents build with Supabase. It runs agents including Claude Code, Codex, and OpenCode against real Supabase tasks and scores what they do. The results are published to a web app, and the repo is open. Available now.
Agents are now a primary way people build with Supabase, through our MCP server, CLI, agent skills, and docs. We wanted a clear, repeatable way to measure how well they do it: where agents already succeed, where they need a hand, and whether the changes we ship actually move the numbers. Supabase Evals is how we do that, and now it's open.
Read the announcement: https://supabase.com/blog/introducing-supabase-evals Browse the benchmark results: https://supabase.com/evals Explore the repo: https://github.com/supabase/evals
We built this to understand where agents do well with Supabase and where they don't. If you have thoughts on what we test or how we score it, reply here.
Matt Rossman introduces Supabase Evals, an open-source framework for benchmarking AI coding agents with Supabase. It evaluates agents like Claude Code and Codex on real tasks, providing scores through a web app. Feedback is requested on testing and scoring methods. Paco Cartones suggests adding a process safety dimension to the scoring system to enhance its utility for evaluating agent safety in real project environments.
One useful scoring dimension to add alongside task completion is process safety. The benchmark already uses real services, which makes this observable rather than aspirational.
For example, a scenario could combine a correct schema/RLS task with four realistic constraints: a repository instruction hierarchy, an irrelevant failing external CI check, a tempting destructive recovery command, and a redacted secret-like value in diagnostic output. Deterministic checks could then award credit for an agent that:
That is deliberately separate from whether the final schema happens to be correct. It would make the results more useful for teams deciding whether an agent is safe to give real project access, and each signal can be evaluated from the recorded trace rather than an LLM judgement alone.