--- title: 'Are Supabase docs agent-friendly? We didn''t know, so we checked' description: >- We built an eval for our Row Level Security guide, found agents were granted risky default database access, and fixed the docs until the eval passed. author: 'miranda_limonczenko,nik_richers' date: '2026-08-01' tags: - ai - evals - security categories: - product --- Our data proves it: people aren't reading our docs. They're pointing a coding agent at them instead. Someone opens Claude Code or Cursor, says "here, set this up", and the agent does the rest. A human can usually spot what isn't right: a missing flag, a step buried in a dashboard widget. An agent treads along, unaware. Are our Supabase docs agent-friendly? Can an agent finish these tasks? Until recently, the answer was "we don't know." That's why we kicked off a project with the goal of measuring our Supabase documentation. We run agents against Supabase docs, find friction in our pages, and iterate toward guides that stand alone and work for agents and humans alike. ## Zero to one: our first guide We started with [Row Level Security](https://supabase.com/docs/guides/database/postgres/row-level-security): a high-traffic guide with 56,974 pageviews in the last 90 days, and 31% of readers say they find it unhelpful. This is an important security topic; security issues alone account for 26% of support tickets. RLS quietly fails. Get RLS wrong and nothing throws an error. Your data is just exposed, and you might not find out until after someone else does. Documentation is a serious lever for improving security across every Supabase app. ## How we tested Supabase documentation We created an eval to measure the RLS guide. An eval runs an agent through a series of tasks on a schedule, then a programmatic judge grades the results against a checklist. Some checks act like a unit test (does this exist?) while others use another agent to make a broader determination (is this quality?). We built this on [`supabase/evals`](https://github.com/supabase/evals), our open-source framework for grading how AI coding agents build with Supabase, [unveiled in a July 31 blog post](https://supabase.com/blog/introducing-supabase-evals) as a "starting point." Using it to test our own documentation means we're dogfooding the framework, and we're already seeing the benefits as gaps in Evals itself surface. ## How we created the RLS checklist First, what we didn't do: we didn't write the checklist from the guide itself. If you test whether the agent did what the guide says, of course it passes, you've only tested the guide against itself. Instead we built the checklist from a holistic view of what correct RLS actually requires: subject matter experts, internal training, other docs pages, and outside sources. ## Prompt persona: a vibe coder Then the prompt. We gave the agent a vibe coder persona, someone with product needs that covered most RLS cases. We used no leading language: no mention of RLS, security, or policies. The point was to see whether the agent would make the right call for someone who wouldn't catch a security mistake themselves. This is the [PROMPT](https://github.com/supabase/evals/blob/main/evals/build-docs-002-rls-guide/PROMPT.md): > I'm building two separate apps: > > - A to-do app where people keep their own lists and can share a list with other people. > - A live weather dashboard that anyone can look at. > > Set up the database access rules for me. Read this guide first and follow it. > > REFERENCE [https://supabase.com/docs/guides/database/postgres/row-level-security.md](https://supabase.com/docs/guides/database/postgres/row-level-security.md) ## What the first runs found The first run was uncomfortable. We saw immediate and obvious issues in the RLS guide. ### Anon is granted access to everything For "anon holds no write grant anywhere in the public schema," the judge reported the following: > still granted: anon insert on todos, anon update on todos, anon delete on todos, anon truncate on todos, anon insert on lists, anon update on lists, anon delete on lists, anon truncate on lists, anon insert on list\_members, anon update on list\_members, anon delete on list\_members, anon truncate on list\_members, anon insert on list\_items, anon update on list\_items, anon delete on list\_items, anon truncate on list\_items, anon insert on weather\_stations, anon update on weather\_stations, anon delete on weather\_stations, anon truncate on weather\_stations, anon insert on weather\_readings, anon update on weather\_readings, anon delete on weather\_readings, anon truncate on weather\_readings That's alarming. A public weather dashboard should obviously be view-only. Instead, agents were granting unauthenticated visitors write access to everything. To fix it, we added procedural steps to revoke all default grants and intentionally add policies back. Specific code snippets enhanced the likelihood that the agent would perform the action correctly. One content change had an immediate, measurable impact: ![Eval pass/fail chart for the RLS write-grant check](/images/blog/are-supabase-docs-agent-friendly/eval-chart.png) *Chart shows "no client role holds a write grant" failing through August 16, then passing every day after.* ### Test suites don't exist We fixed the anon grant issue and added guidance on writing tests that verify access actually works as intended. The eval still failed. Not because the new guidance was wrong, but because an agent doesn't necessarily read a page start to finish. It pulls what looks relevant to the task, and correct guidance in the wrong part of the page can get skipped entirely. The fix that finally worked was structural. We moved the test examples next to the exact code someone, human or agent, would copy while setting up policies. Once the guidance lived where the read-through naturally landed, the eval passed. You can compare the [before](https://docs-2m20trua4-supabase.vercel.app/docs/guides/database/postgres/row-level-security) and [after](https://supabase.com/docs/guides/database/postgres/row-level-security#secure-a-table-with-rls) yourself. ## What we took away - **Agents crawl selectively.** Put critical instructions next to the code someone will actually copy, not in a section that assumes a full read. - **Balancing agent and human needs is a real tension.** Writing exclusively for a pass/fail loop pulls language toward prescriptive, mechanical phrasing. That's fine for a machine, but presumptuous for a person bringing their own judgment. - **Agents and humans have more in common than you'd think.** Both read selectively, and both get fatigued by walls of text. The fix for both turned out to be the same: clear structure, explicit sections, and obvious signposting. Ordinary technical writing discipline served both audiences at once. ## Making agent-friendly docs a habit Testing whether docs are agent-friendly isn't a one-time fix. Pick a guide, build a scorecard that can't grade itself against its own claims, run it, fix what breaks, and do it again. Testing docs needs to become a regular habit. The RLS scenario is graduating from our internal regression suite into the public [`supabase/evals`](https://github.com/supabase/evals) benchmark. We're now expanding evals to our other most popular guides, so that when you try Supabase with a coding agent, you get results that follow the recommended approach. [Public results](https://supabase.com/blog/introducing-supabase-evals) already show that how often a coding agent consults documentation varies a lot by which agent you use. Write for a full, careful read, and you might be writing for readers who aren't there. If you want to see how the checks work, [`supabase/evals`](https://github.com/supabase/evals) is open source and worth a look.