The RAG production checklist

Garbage retrieval produces hallucinated answers regardless of model quality. Fix retrieval first.

• Chunk at semantic boundaries, not arbitrary character counts critical Splitting mid-sentence or mid-paragraph breaks the context a chunk needs to be useful. Use paragraph breaks, section headers, or sentence boundaries as split points. A 512-token chunk that crosses a section boundary is worse than a 300-token chunk that doesn't.
• Run an offline retrieval eval before shipping Pick 20-30 representative questions. For each, confirm the right document is in the top-k results. Calculate recall@3. If it's below ~70% for your domain, fix chunking and embedding before touching the generation side.
• Set ivfflat lists proportional to your document count The default lists=100 is wrong for most datasets. Use lists = sqrt(row_count) as a starting point. Rebuild the index after significant ingestion. An undersized index degrades recall silently.
• Add hybrid search (BM25 + vector) for keyword-heavy queries recommended Pure vector search misses exact model names, version numbers, and error codes. BM25 catches these. Reciprocal Rank Fusion merges both result sets without a learned combiner. Use Supabase's fts column or a separate tsvector.
• Store chunk metadata: source URL, section title, document date Metadata lets you filter stale documents (date), show attribution (source URL), and group by section. Without it, you can't surface citations or filter by recency. Add a metadata jsonb column from day one.

Citations are not a UI polish feature. They are the honesty contract between the system and the user.

• Show citations before the answer, not after ux Showing "Based on: [source]" before the answer text lets the user calibrate trust before reading. Showing sources after conditions them to believe the answer first. The order is not cosmetic — it changes how users interpret confidence.
• Surface the retrieved text verbatim, not a summary of it If a citation shows a model-generated summary of the source chunk, you've added another layer of possible error. Expandable verbatim excerpts let the user verify. The CiteKit <details> citation component does this by default.
• Never cite chunks that scored below your retrieval threshold Low-scoring chunks are padding. Citing them signals false confidence. Set a minimum cosine similarity threshold (0.70 is a reasonable starting point) and omit chunks below it from both the context window and the citation list.
• Explicitly say "I don't know" when retrieval returns nothing above threshold An LLM prompted with no relevant context will hallucinate one from training data. The prompt must contain an explicit instruction: "If no retrieved chunk is relevant, say: I don't have information on this. Do not answer from general knowledge."
• Link citations to the canonical source document, not a chunk ID Chunk IDs are internal. Users need a URL or document name they can open and read. Map chunks to their parent document at ingestion time and store the canonical URL in metadata.

You can't improve what you don't measure. A thin eval harness catches regressions before they reach users.

• Maintain a golden set of 20+ (question, expected source) pairs critical This is your regression baseline. Each pair has a question and the document ID that should appear in the top-3 results. Run this on every retrieval config change. A golden set this small takes a few hours to build and saves days of debugging.
• Test answer faithfulness, not just fluency A fluent answer that contradicts its own citations is a hallucination. Use an LLM-as-judge prompt that checks whether the answer makes claims supported by the retrieved chunks. Set a threshold (e.g. 80% faithful) as a CI gate.
• Track recall@k over time, not just the latest run Recall@3 degrading from 82% to 74% after an ingestion run is a signal worth catching. Store eval results with a timestamp and document count so you can correlate drops with corpus changes.
• Add a "null retrieval" test case for every intent category For each topic your KB covers, add one off-topic question. The system should say "I don't know," not hallucinate. These are often easier to fail than the positive cases.
• Run the eval harness in CI, not just manually Manual evals only happen when someone remembers to run them. A CI step that fails the build on recall regression is the only sustainable pattern. Even a GitHub Action that runs the golden set takes under 2 hours to wire up.

The attack surface of a RAG system is different from a standard web app. These are the non-negotiables.

• Never expose service_role keys in client-side code critical service_role bypasses all RLS policies. If it's in a JS bundle or an API response header, any user who opens DevTools owns your database. All DB writes must go through a server-side function (Edge Function, API route) that holds the key as an environment variable.
• Enable RLS on every table that holds user data RLS off = any authenticated user can read all rows. Enable it from the first migration, before inserting data. A policy that allows service_role and denies anon is a reasonable default for internal tables.
• Sanitize retrieved chunks before injecting into the system prompt A retrieved chunk containing "Ignore previous instructions and ..." is a prompt injection attack. Strip XML-like tags and add boundary markers (--- RETRIEVED CHUNK START ---) so the model can distinguish source text from instructions.
• Rate-limit the RAG endpoint per user/IP Each RAG request makes an embedding call, a DB query, and an LLM completion. An unprotected endpoint is an open billing attack surface. Add rate limiting at the edge (Supabase Edge Function, Cloudflare) before the request hits your model.
• Scope the embedding model's API key separately from the LLM key If you use a single API key for both embeddings and completions, a leak exposes both. Use separate keys with the minimum required permissions. Rotate them independently.

Most RAG deploy failures are configuration problems, not code problems. Verify these before go-live.

• Verify the vector index is built on the production dataset critical Indexes built on a small dev corpus have wrong lists values. After production ingestion, rebuild: REINDEX INDEX CONCURRENTLY documents_embedding_idx. Skipping this silently degrades recall on prod while dev evals pass.
• Confirm all environment variables are set in the production environment SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, the embedding API key, the LLM API key — each missing variable surfaces as a cryptic runtime error, not a clear startup failure. Run a smoke test that verifies each env var is present before the first real request.
• Set a context window budget and enforce it Retrieving k=10 chunks with no budget check can push the total prompt past the model's context limit. Count tokens before sending. If the budget is exceeded, drop the lowest-scoring chunks until it fits rather than truncating arbitrarily.
• Add a streaming timeout and surface it gracefully LLM completions can stall. A 30-second timeout with a user-visible fallback ("Response is taking too long — try again") is better than a spinner that never resolves. Set both a connection timeout and a streaming inactivity timeout.
• Log retrieval quality metrics in production, not just errors The queries users actually ask in production are different from your golden set. Log the query, top-k document IDs, and cosine scores for a sample of production traffic. Review weekly. This is how you find blind spots in your KB that no eval caught.