Git Guide

The problem

Docs chatbots fail in one specific way: they answer. Ask about a feature that was removed two releases ago and a naive RAG pipeline finds something adjacent, and the model stitches it into fluent prose. The failure is silent — plausible, well-formatted, wrong, and indistinguishable from a good answer unless you already knew the answer, in which case you did not need to ask.

So the design target was never “answer more questions”. It was “make the failure loud”: ground every claim, cite every source, and drop weak retrieval before the writer ever sees it.

How it works

Ingestion is a one-time phase — sparse-checkout the GitLab docs, chunk them header-aware, embed locally, persist to ChromaDB. Runtime is two CrewAI agents behind a router: a retriever with four search tools, and a synthesiser whose only job is to write from what it was given and cite it.

The router is the cheap trick that pays for itself. Short or ambiguous queries go through the full pipeline; everything else skips straight to retrieval.

User query
   │
   ▼
Smart Router
   ├─ under ~4 words / ambiguous ──► FULL pipeline
   │                                   intent classify ─┐ asyncio
   │                                   query rewrite  ──┘ .gather()
   └─ clear, 5+ words ──────────────► straight to retrieval
   │
   ▼
Retriever Agent  (CrewAI)
   semantic_search · filtered_search · multi_query_search
   ChromaDB cosine similarity · RRF fusion
   │
   │  heuristic validation: drop every chunk below 0.40 relevance
   │  (nothing clears it → no answer, which is the correct outcome)
   ▼
Synthesiser Agent  (CrewAI)
   writes only from surviving chunks · cites every claim
   flags deprecated features · never extrapolates
   │
   ▼
Cited answer + source URLs + route pill + response time

Decisions worth defending

1. 01

   A relevance floor instead of a validator agent.

   The obvious design adds an LLM validator to judge whether the retrieved chunks support an answer. That costs five to eight seconds and adds another call that can itself hallucinate. Instead, chunks below 0.40 cosine relevance are dropped before the synthesiser ever sees them — a threshold, not an opinion. Answer quality held; the latency did not come back.

2. 02

   Route around the pipeline when it is not needed.

   A clear, well-formed question does not need intent classification and query rewriting; running them anyway is latency spent to reformulate something that was already fine. The router sends short or ambiguous queries through the full pipeline and everything else straight to retrieval, taking 30–50% off average latency.

3. 03

   The two pre-processing steps do not depend on each other.

   Intent classification and query rewriting are independent, so they run under asyncio.gather() instead of one after the other. Pre-processing went from roughly eight seconds to four. Cheap, and only visible if you look at what the steps actually need from one another.

4. 04

   Asymmetric embeddings, used asymmetrically.

   BAAI/bge-small-en-v1.5 is trained with a query prefix — Represent this sentence for searching relevant passages: — that goes on the query and *not* on the documents. Getting that backwards costs retrieval accuracy and produces no error, no warning and no obvious symptom. It is exactly the kind of defect that only shows up if you read the model card instead of the quickstart.

5. 05

   Local embeddings and a free-tier LLM, so it stays up.

   Embeddings run on CPU and cache to disk, ChromaDB is a local persistent client, and the LLM is OpenRouter's free tier behind LiteLLM with three retries. Total operating cost is $0.00 — which is the reason this is a project you can still run today rather than a demo I took down when the bill arrived.

Proving it works

Grounding is not a claim in the README here; it is visible in the interface on every answer.

• Source chips on every answer, linking back to the GitLab doc the claim came from. A claim without a citation is a defect you can see.
• The relevance floor is the assertion. If nothing clears 0.40, there is nothing to write from, and refusing is the correct behaviour rather than a bug to route around.
• The synthesiser is instructed to flag deprecated features and to never extrapolate past the retrieved chunks — the two failure modes that make a docs bot actively harmful.
• Route and response time are surfaced per answer, so a mis-routed or slow query is observable instead of something you infer from feel.