Lumen answers questions over an internal corpus — runbooks, policies, and architecture notes — with citations back to the source paragraph. Used daily by support and on-call engineers.
Tip
The interesting engineering here isn't the model call. It's the retrieval quality and the evaluation harness that keeps a prompt change from silently making answers worse.
Problem
The first version was a thin wrapper: embed the question, fetch the top ten chunks, stuff them into a prompt. It demoed well and failed in production because:
- Chunking ignored structure. Fixed 800-token windows split tables and procedures down the middle, so retrieved context was often half a runbook step.
- No citations meant no trust. Users couldn't verify an answer, so they re-checked the source anyway — the tool saved nothing.
- Changes were unmeasurable. Every prompt tweak was evaluated by whoever made it, on whatever question they remembered.
Architecture
Two pipelines: an offline indexer and an online query path, with an evaluation suite gating changes to either.
Indexer (nightly) Query path (per request)
──────────────── ────────────────────────
docs ─► structural split ─► embed question ─► embed ─► vector search (k=30)
(heading-aware) │ │
▼ ▼ rerank (cross-encoder)
chunk + parent id pgvector │
top 6 ─► prompt
│
answer + citations- Heading-aware chunking. Documents split on heading boundaries first, then by size within a section, with the parent heading path prepended to each chunk. A chunk always carries the context of where it came from.
- Retrieve wide, rerank narrow. Vector search returns 30 candidates; a cross-encoder reranks and only the top 6 reach the prompt. This was the single biggest quality jump — bigger than any prompt change.
- Citations are structural, not generated. Each chunk keeps its document id and heading anchor, and the response renders those as links. The model is never asked to produce a URL, so it can't invent one.
- Refusal path. If the top reranked score is below a threshold, the assistant says it doesn't know instead of answering from parametric memory.
Evaluation in CI
A fixed set of 120 question/expected-source pairs runs on every change to prompts, chunking, or retrieval parameters:
pnpm eval --suite retrieval # recall@6 on known-answer questions
pnpm eval --suite answers # graded answers, flags regressions vs baselineThe retrieval suite is the useful one: it's deterministic, cheap, and catches the majority of regressions before any answer grading is needed.
Warning
Answer grading uses a model as judge, which drifts as the judge model changes. Pinning the judge version is not optional — an unpinned judge turns your baseline into noise.
Stack
One database for chunks, metadata, and embeddings — no separate vector store to keep in sync.
Answer synthesis with a strict citation format and a refusal path.
Reranks 30 candidates down to 6; the largest single quality gain.
120 graded pairs, run in CI on every retrieval or prompt change.
Results
| Metric | First version | Current |
|---|---|---|
| recall@6 on eval set | 0.61 | 0.89 |
| Answers with a working citation | ~40% | 98% |
| Unsupported-answer rate (graded) | 17% | 3% |
| Median response time | 3.1s | 2.4s |
What I'd do differently
I built the evaluation suite after the second quality complaint instead of before the first release. Everything before that point was opinion, and two of the "improvements" we shipped in that window turned out to be regressions once there was a baseline to measure against.
I'd also keep a permanent record of retrieved chunk ids per answer from day one. We added it during a debugging session, and it immediately became the fastest way to tell a retrieval problem from a generation problem.