Feature: Source Connectors and Incremental Sync Pipeline for Continuous Ingestion

Summary

Build a connector-based ingestion pipeline that continuously syncs knowledge from external sources (Google Drive, Notion export, S3/local folders), performs incremental diffing, and updates indexes without full re-ingestion.

Why this matters

Manual bulk ingestion quickly goes stale. Portfolio intelligence needs fresh source data with predictable sync behavior and observability.

Scope

• Connector interface for pull-based sources.
• Incremental sync with change detection (new/updated/deleted docs).
• Document fingerprinting/versioning and idempotent upserts.
• Background scheduler + job queue for sync runs.
• Sync run status API and UI panel (last run, delta counts, failures).

Non-goals

• Real-time webhook ingestion for every source in this issue.
• Complex OCR pipelines (keep text/PDF first).

Proposed implementation

1. Define connector contract (list, fetch, checkpoint).
2. Add normalized ingestion metadata (source_id, external_doc_id, hash, version, last_seen).
3. Implement incremental index update path in RAG service.
4. Add scheduler worker and retry policy with dead-letter handling.
5. Expose /sync/jobs API and frontend status components.

Acceptance criteria

• At least two connectors implemented (local folder + one remote source).
• Re-running sync with unchanged docs causes zero duplicate chunks.
• Updated source docs replace/merge old chunks deterministically.
• Deletions are handled according to policy (soft-delete or remove from retrieval).
• Sync failures are visible via API/UI with actionable error details.
• Integration tests validate idempotency and update/delete behavior.

Relationship to existing issues

• Complements issue #3 (modularization + ingestion endpoint) by adding continuous, connector-driven refresh.

Labels

enhancement, data-ingestion, backend, ops