Grounded Answers: Citations, Span Highlighting, and Confidence/Abstain

Summary
Ship a grounding layer that enforces source-anchored answers. The model must (1) cite retrieved chunks, (2) highlight exact supporting spans, (3) compute a simple confidence score, and (4) abstain when evidence is insufficient. This reduces hallucinations, improves trust, and makes the system evaluation-friendly.

---

Goals

• Inline citations: Attach per-sentence citations like [1], [2] mapping to retrieved chunks/URLs.
• Span highlighting: Return start–end offsets (or text snippets) from supporting chunks for UI hover/highlight.
• Answerability + abstain: If top-k evidence doesn’t meet a threshold, return a graceful “cannot answer from sources” with best related snippets.
• Confidence score: Heuristic 0–1 score from retrieval signals (BM25/semantic similarity, agreement across chunks) and model self-rating.
• Consistent schema: Structured JSON payload alongside the natural language answer for API/UI consumption.

---

Non-Goals

• Changing core retrieval stack (that’s #4).
• Production API refactors (that’s #3), beyond adding one response schema.

---

Design Overview

1) Answer schema (API contract)

Augment /chat response with:

{
  "text": "Final answer with [1][2] ...",
  "citations": [
    { "id": "doc-abc", "source": "file://... or https://...", "score": 0.78,
      "spans": [{ "chunk_id":"c1", "start": 128, "end": 214, "preview":"..."}] }
  ],
  "confidence": 0.72,
  "abstained": false,
  "reason": null
}

2) Generator wrapper

• Build a grounded prompt that requires citing chunk ids and marking span quotes.
• Post-process LLM output to:
  • Validate each claim has at least one citation.
  • Map [n] → citations[n-1].
  • If mapping fails or evidence weak → trigger abstain path.

3) Span extraction

• During retrieval, keep chunk text and offset mapping.
• When LLM returns quoted support, fuzzy-match into chunk to produce (start,end).
  Fallback: top-N sentences from the cited chunk via sentence-level similarity.

4) Confidence & abstain heuristic

• Combine:
  • max/mean similarity of supporting chunks,
  • number of distinct sources supporting the claim,
  • coverage ratio (citations per sentence),
  • model self-rating (1–5 converted to 0–1).
• If < threshold (e.g., 0.5) or citations missing → abstained=true and provide top snippets.

5) UI touch (optional)

• In Colab/Flask demo, render superscript citations; hover shows highlighted span.

---

Acceptance Criteria

• Responses include citations[] with valid source ids/urls and at least one span per citation.
• Every sentence in text with a factual claim contains ≥1 citation token [n].
• confidence present and bounded [0,1]; below threshold yields abstained=true and neutral fallback message.
• Unit tests: citation mapping, span matching, abstain path, malformed output repair.
• Demo route in Flask shows hover highlight for cited spans.

---

Tasks

• Define pydantic schema for grounded responses.
• Update prompt template to force citations + span quotes (with an explicit JSON block).
• Implement parser/validator and a light auto-repair pass if JSON invalid.
• Add span-matcher (exact → fuzzy) against chunk text; expose offsets.
• Compute confidence metrics; configurable thresholds via env.
• Extend Flask /chat to return both text and structured citations.
• Add tests (happy path, no-evidence, partial-evidence, broken JSON).
• Mini UI tweak in Colab/Flask to display superscripts and hover previews.
• Docs: README section “Grounded Answers & Citations” with examples.

---

Risks & Mitigations

• LLM format drift → strict JSON schema + repair pass.
• Span mismatch → fuzzy matcher with char-window previews; log misses for tuning.
• Over-abstention → start conservative (threshold ~0.4) and expose env knob.