Knowledge Base Agent

Problem

Internal knowledge is scattered across Notion, Confluence, Google Drive, and Slack. Employees spend 20% of their week searching for information, and answers are inconsistent because no one knows which document is the current source of truth.

Solution

Build a RAG agent that indexes your documentation into a vector store, enforces citation requirements on every answer, grades response quality with hallucination detection, and refreshes the index on a schedule.

Implementation Steps

1. Ingest and chunk documentation

   Pull content from Notion, Confluence, Google Drive, and other sources. Chunk into ~500 token segments with metadata for source, date, and category.

   Tip: Use overlapping chunks (50-100 token overlap) to preserve context at chunk boundaries. This improves retrieval accuracy by 10-15%.

2. Build retrieval pipeline

   Implement hybrid search (keyword BM25 + vector similarity) for the best retrieval accuracy. Use metadata filters for freshness and category.

   Tip: Define a measurable success metric and review weekly to improve quality and cost.

   # Hybrid retrieval with Supabase
   results = await supabase.rpc('hybrid_search', {
       query_text: user_question,
       query_embedding: embed(user_question),
       match_count: 5,
       full_text_weight: 0.3,
       semantic_weight: 0.7
   })

3. Enforce citation requirements

   Instruct the LLM to only answer based on retrieved sources and include inline citations. Reject answers that reference information not in the retrieved context.

4. Add hallucination detection

   Run a separate verification step that checks each claim in the answer against the retrieved sources. Flag unsupported claims for review.

5. Schedule index refresh

   Set up periodic re-indexing (daily or weekly) with stale content detection. Alert content owners when source documents haven't been updated in 90+ days.

Recommended combos

FAQs