Feature: Add Hybrid Search (FAISS + Semantic + Keyword) for Better Retrieval Accuracy

Summary

Enhance the current FAISS vector store retrieval by integrating hybrid search (semantic + keyword-based retrieval). This ensures queries that rely on exact terms (like company names, tickers, or sector codes) are retrieved as reliably as semantically similar queries.

Problem / Motivation

• The current FAISS + embeddings pipeline works well for general semantic queries, but struggles when:

  • Users search with rare proper nouns (e.g., “Vizzo Capital”, “Calendly”).
  • Queries involve numerical or structured data (dates, metrics, IDs).
  • Users expect exact keyword matches (like “investment in AI sector” where "AI" is too short to embed effectively).

• Without hybrid retrieval, important but lexically exact information may be missed.

Proposed Solution

• Implement hybrid retrieval combining:

  1. Dense Vector Search (current FAISS + all-MiniLM-L6-v2 embeddings).
  2. Sparse Retrieval (BM25 / keyword-based retriever).

• Merge results using a re-ranking strategy (e.g., weighted scoring or reciprocal rank fusion).

• Provide an option to configure weights between semantic and keyword matches.

Technical Details

Backend (Python / LangChain)

• Use langchain.retrievers.BM25Retriever or ElasticSearchRetriever.

• Wrap both retrievers in a MultiRetriever:

  from langchain.retrievers import EnsembleRetriever
  
  retriever = EnsembleRetriever(
      retrievers=[faiss_retriever, bm25_retriever],
      weights=[0.7, 0.3]  # adjustable
  )
  

• Add configuration in Flask API (config.py) to toggle hybrid mode and adjust weights.

• Extend /chat endpoint: add a query param retrieval_mode=[semantic|keyword|hybrid].

Frontend / Colab Notebook

• Add a CLI flag:

  retrieval_mode = "hybrid"
  

• Show debug logs with top results from both retrievers.

Acceptance Criteria

• Users can specify retrieval_mode=hybrid.
• Queries with proper nouns (e.g., “Calendly”) return correct matches.
• Queries with short tokens (e.g., “AI”, “IoT”) retrieve relevant docs.
• Default weights (0.7 semantic, 0.3 keyword) can be overridden.
• Unit tests show hybrid > semantic-only in precision/recall.

Benefits

• Improves accuracy of responses, especially for edge cases with proper nouns or structured terms.
• Keeps compatibility with current FAISS setup while extending functionality.
• Makes system more robust and production-ready for real portfolio support use cases.

Next Steps

1. Prototype with BM25Retriever (in-memory) for proof-of-concept.
2. Benchmark hybrid vs FAISS-only on sample queries.
3. Extend to ElasticSearch (optional) for scalable deployments.

---