How to add RAG to an existing product without a rewrite

Start with a vertical slice, not a platform

The most reliable way to waste a quarter on RAG is to begin by indexing everything. You build ingestion for six content types, spend two weeks arguing about vector databases, and end up with a search box nobody trusts. Start from the other end: take one question your users already ask support, and answer only that question, for one class of user, in one place in the interface.

A good first slice has four properties. The corpus is bounded and already lives in your database. A wrong answer is embarrassing rather than dangerous. There is an existing screen to put the feature on. And you can tell whether an answer was right simply by reading it. Documentation search, "explain this record", and support deflection all qualify. Anything touching money, medical or legal advice does not — not first.

Decide what actually goes in the index

Retrieval quality is decided before a single embedding is computed. Two questions matter: which content answers the question, and which content merely contains the words in it.

• Prefer canonical sources over derived ones. Index the help centre article, not the ten chat messages arguing about it.
• Exclude anything you cannot date. If a chunk has no updated_at, you cannot expire it, and your assistant will confidently quote a policy that changed a year ago.
• Keep the structure. Titles, headings, product names and identifiers are retrieval signals — stripping them throws away the cheapest features you have.
• Store the metadata you will filter on later: tenant, locale, product area, visibility, source ID. Adding a filter column after a million rows have been embedded is annoying; re-embedding everything because you did not keep the source ID is worse.

Chunking: structure first, size second

Chunking is where most RAG systems quietly lose. A fixed 1,000-character window over documents that have headings will cut definitions in half and merge two unrelated sections into one vector. Split on structure first — headings, list items, table rows, function boundaries — then pack those units up to a target size, rather than slicing blindly and hoping the boundaries land well.

Sensible defaults to start from, then tune against your eval set: roughly 300 to 800 tokens per chunk, 10 to 15 percent overlap, and a hard rule that a chunk never spans two documents. Overlap is insurance against a boundary landing mid-sentence; beyond about 20 percent, it mostly buys duplicate results and a larger bill.

Prepend context to every chunk before embedding it. A chunk that reads "It must be renewed every 12 months" is useless alone. The same text embedded as "Billing › Subscriptions › Renewal — It must be renewed every 12 months" retrieves correctly and also reads correctly once it is sitting in a prompt.

type Chunk = { text: string; heading: string; sourceId: string; ord: number };

export function chunkSections(
  sections: { heading: string; body: string }[],
  sourceId: string,
  maxChars = 2400,
  overlapChars = 300
): Chunk[] {
  const chunks: Chunk[] = [];
  for (const section of sections) {
    const prefix = section.heading + '\n\n';
    let start = 0;
    while (start < section.body.length) {
      const end = Math.min(start + maxChars, section.body.length);
      // prefer a paragraph break near the end of the window
      const cut = section.body.lastIndexOf('\n\n', end);
      const stop = cut > start + maxChars / 2 ? cut : end;
      chunks.push({
        text: prefix + section.body.slice(start, stop).trim(),
        heading: section.heading,
        sourceId,
        ord: chunks.length,
      });
      if (stop >= section.body.length) break;
      start = Math.max(stop - overlapChars, start + 1);
    }
  }
  return chunks;
}

Embeddings and where the vectors live

For a first slice inside an existing product, pgvector in the Postgres you already run is almost always the right answer. Chunks live next to the rows they came from, tenant filters are ordinary SQL, transactions keep the index consistent with the source data, and you add no new operational surface. For up to a few million chunks with an HNSW index, that is not a compromise.

Reach for Pinecone or Weaviate when one of these is genuinely true: you need to scale the index independently of your primary database, you want managed sharding and replication you are not going to build yourself, or your team cannot take on another Postgres extension. Those are real reasons. "It is the vector database everyone uses" is not.

create extension if not exists vector;

create table doc_chunk (
  id         bigserial primary key,
  tenant_id  uuid        not null,
  source_id  text        not null,
  locale     text        not null,
  heading    text,
  content    text        not null,
  embedding  vector(1536) not null,
  model      text        not null,   -- which model produced this vector
  updated_at timestamptz not null default now()
);

-- ANN index: build it after the initial bulk load, not before
create index doc_chunk_embedding_idx
  on doc_chunk using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 64);

-- the filters you retrieve with must be indexed too
create index doc_chunk_scope_idx on doc_chunk (tenant_id, locale);

-- lexical half of hybrid search
alter table doc_chunk add column tsv tsvector
  generated always as (
    to_tsvector('simple', coalesce(heading, '') || ' ' || content)
  ) stored;
create index doc_chunk_tsv_idx on doc_chunk using gin (tsv);

Hybrid search, then re-ranking

Pure vector search fails on exactly the queries your users type: error codes, SKUs, invoice numbers, function names, rare proper nouns. Lexical search fails on paraphrase. Run both and fuse the result lists — reciprocal rank fusion takes about twelve lines, needs no tuning, and does not care that the two scores are on different scales.

with semantic as (
  select id, row_number() over (order by embedding <=> $1) as rank
  from doc_chunk
  where tenant_id = $2 and locale = $3
  order by embedding <=> $1
  limit 50
),
lexical as (
  select id, row_number() over (order by ts_rank_cd(tsv, q) desc) as rank
  from doc_chunk, websearch_to_tsquery('simple', $4) q
  where tenant_id = $2 and locale = $3 and tsv @@ q
  limit 50
)
select coalesce(s.id, l.id) as id,
       coalesce(1.0 / (60 + s.rank), 0)
     + coalesce(1.0 / (60 + l.rank), 0) as score
from semantic s
full outer join lexical l on l.id = s.id
order by score desc
limit 20;

Then re-rank the fused top twenty with a cross-encoder or a small language model, and keep the best four to six. Re-ranking is the highest-leverage single step in most RAG systems: embeddings compress a chunk into one vector long before it ever meets your query, while a re-ranker reads the query and the chunk together. Budget for it in your latency plan, reduce k afterwards, and you often come out faster overall because the generation prompt is much shorter.

Evaluate retrieval before you touch the prompt

You cannot improve what you do not measure, and "it feels better" is not a measurement. Build a golden set before tuning anything: 30 to 50 real questions, each labelled with the chunk IDs that genuinely answer it. Two engineers and a support lead can produce that in an afternoon, and it will outlive three prompt rewrites.

• Recall@k — does a correct chunk appear anywhere in the top k you retrieve? If recall@20 is poor, no prompt will save you: fix chunking, filters or the query.
• Precision@k after re-ranking — of the chunks you actually put in the prompt, how many are relevant? This is what drives both cost and hallucination.
• Groundedness — is every claim in the answer supported by a retrieved chunk? Score it with a model against the retrieved context, and spot-check the judge by hand until you trust it.
• Refusal rate on out-of-scope questions. An assistant that never says "I do not know" is not aligned with your corpus; it is guessing.

Run the evaluation in CI on every change to chunking, embedding model, retrieval parameters or prompt. It takes minutes, and it is the only thing standing between you and a silent regression shipped on a Friday.

Prompt assembly and citations

Keep this part boring. Use a system message that states the scope and the refusal rule, followed by the retrieved chunks with stable identifiers and their source titles, then the question. Ask for citations by identifier, and validate them after generation: if the model cites an ID that was not in the context, that answer is a bug, not a quirk.

function buildMessages(question: string, chunks: Chunk[]) {
  const context = chunks
    .map((c, i) => '[' + (i + 1) + '] ' + c.heading + '\n' + c.text)
    .join('\n\n');

  return [
    {
      role: 'system' as const,
      content:
        'Answer only from the CONTEXT. If the context does not contain the ' +
        'answer, say you do not know and point to support. ' +
        'Cite the sources you used as [n] after each claim.',
    },
    {
      role: 'user' as const,
      content: 'CONTEXT:\n' + context + '\n\nQUESTION: ' + question,
    },
  ];
}

Render those citations in the interface as links back to the source document. It changes user behaviour: people verify, they report a bad chunk instead of a bad "AI", and you get a free feedback channel that feeds your eval set.

Permissions and tenant filtering

This is the part that turns a pleasant feature into an incident. The rule is simple: filter at query time, inside the same query, using the caller's identity from your session — never post-filter results the model has already read, and never trust a tenant ID that arrived from the client.

• Pass tenant and visibility as bound parameters into the retrieval query, and make those columns non-nullable in the schema so a missing filter fails loudly.
• Mirror your application ACLs onto the chunk rows, and re-index a document when its permissions change, not only when its text changes.
• Write the test: user A asks a question whose only good answer lives in user B's data, and the assistant must say it does not know.

If your permission model is too complex to mirror cleanly, restrict the first slice to content that is public within the tenant. Ship that, learn from it, then extend.

Latency and cost, on purpose

A RAG request comprises embedding, retrieval, re-ranking and generation, and the user feels all four stages. A few things help consistently.

• Cache embeddings keyed by a hash of the normalised query. Repeated and templated questions are far more common than people expect.
• Stream the answer. Time to first token is the number users perceive; total time is the one you optimise second.
• Cap the context aggressively. Six good chunks beat twenty mediocre ones on quality and on cost at the same time.
• Use a cheap model for query rewriting and re-ranking, and keep the expensive one for the final answer.
• Set a timeout per stage with a defined fallback, and degrade to plain lexical search rather than returning an error.

Track cost per answered question rather than cost per token. Token counts are an implementation detail; the cost of one useful answer is a business metric you can communicate clearly.

Observability

Log the whole retrieval trace: the raw query, the rewritten query, the filters applied, retrieved IDs with scores, the re-ranked order, prompt token count, model, latency per stage, and the final answer with its citations. Without that, every "the AI gave a wrong answer" report is unfalsifiable. With it, most reports are diagnosed in a minute — and the diagnosis is usually that the right chunk was never retrieved at all.

Add a thumbs-down option with a free-text reason and route those responses straight into the eval backlog. Real failures from real users are worth more than any synthetic question set you can write.

Roll out behind a flag

Ship it dark first: run retrieval against real traffic, log everything, generate nothing. You will find your filter bugs and your empty-corpus queries for free before anyone sees an answer. Then enable generation for your own team, then for a small percentage of users, with a kill switch that does not require a deploy.

Keep the pre-AI path intact throughout. The assistant is an addition to the search box, not a replacement for it, until your numbers say otherwise.

Checklist

• One question, one surface, one corpus — a slice you can evaluate by reading.
• Chunks split on structure, headings prepended, source metadata stored.
• Vectors next to your data in pgvector, unless you have a stated reason not to.
• Hybrid retrieval, fused, then re-ranked down to a handful of chunks.
• A golden set of 30 to 50 items running in CI, with recall@k and groundedness.
• Tenant and permission filters bound into the retrieval query, with a test that proves it.
• Citations rendered and validated; an unknown citation ID treated as a bug.
• Full retrieval traces logged, thumbs-down feeding the eval set.
• A flag, a kill switch, and the existing search flow still working.

If one item on this list looks expensive for your team, that is useful information about the slice you picked. It usually means the corpus or the permission model is the real project — and it is much cheaper to learn that in week one than in month three.