Thotqen

Overview

Most "chat with your PDF" demos are a thin wrapper around a single LLM call. Thotqen — the AI Knowledge Copilot behind this project — isn't that. Upload a document and every question against it runs through an edge gateway, a cost-aware routing layer, a seven-provider LLM fallback chain, and a stateful retrieval graph that grades its own results before answering. It's a production-grade RAG system built to solve the trade-offs that actually show up once a prototype has to serve real traffic — latency, cost, and retrieval accuracy pulling in different directions at once. The system is split across two deployments: bbygrl, a Next.js frontend with Supabase auth, and thotqen, a FastAPI backend running on Hugging Face Spaces — with a Cloudflare Worker sitting in front of both as an edge gateway.

Stack

• Next.js (React 19), Supabase Auth — frontend (bbygrl)
• Cloudflare Workers — edge gateway: CORS, semantic cache, R2 document serving, reverse proxy
• FastAPI, LangGraph, Celery + CloudAMQP — backend (thotqen), hosted on Hugging Face Spaces via Docker
• Supabase PostgreSQL with pgvector + HNSW indexing, plus a custom hybrid_search() SQL function
• Local embeddings via sentence-transformers (all-MiniLM-L6-v2, 384-dim) and a local BGE cross-encoder reranker
• Multi-provider LLM fallback: NVIDIA, Groq (2 accounts), Gemini (2 accounts), OpenRouter, self-hosted vLLM
• Tavily web search, with DuckDuckGo as a fallback, for corrective retrieval
• LangSmith for tracing, RAGAS for offline evaluation

Three Layers Before a Model Ever Sees the Question

Every request follows the same path before it reaches a model. The Next.js client sends a REST or SSE request to a Cloudflare Worker edge gateway. That gateway enforces CORS against an explicit origin allowlist, checks a semantic cache in Cloudflare KV, and serves stored documents directly from R2 at /edge/docs/* — bypassing the origin entirely for static file requests. Only a cache miss reaches the FastAPI backend. The backend classifies the query's intent, routes it through the LLM fallback chain, and hands it to a LangGraph pipeline that retrieves from Postgres/pgvector and R2. Every node in that pipeline is traced through LangSmith, so a single request is debuggable end to end instead of disappearing into a stack of opaque API calls.

Not Every Question Needs the Same Firepower

A fast, cheap classifier looks at each query first and labels it cheap or complex before it reaches a real answering model — a one-word greeting and a multi-document synthesis question shouldn't cost the same. Complex queries route through NVIDIA-hosted Llama 3.1 70B, then Groq's second account running Llama 3.3 70B, then Gemini's second account, then OpenRouter's free auto-router (tried twice — a different free model each draw), then Groq's first account, then Gemini's first account, then a self-hosted vLLM instance as the final no-API-key fallback. Cheap queries run the same chain minus the NVIDIA hop. Two accounts each for Groq and Gemini isn't redundancy for its own sake — the first account on each hit a real account-level restriction, an org restriction on one and quota exhaustion on the other. Rather than block on fixing that immediately, the second account was moved to the front of the chain while the first stays further down in case it recovers on its own. Every provider gets a hard 18-second cutoff enforced externally, through a ThreadPoolExecutor, because not every provider's SDK actually honors its own timeout parameter — ChatNVIDIA in particular exposes none at all. And each fallback is a full provider switch, not a retry: .with_fallbacks() moves to the next provider on any failure. A same-provider retry loop used to sit on top of this, up to four attempts, until it got measured live and turned out to just re-hit an already-exhausted chain on account-level failures — multiplying worst-case latency without helping. It was cut to a single pass.

A Retrieval Graph That Grades Its Own Work

Once a provider is picked, the actual answering happens inside a stateful LangGraph, not a single prompt-and-response call. An input guardrail checks for prompt injection or unsafe content first — flagged requests get blocked before touching retrieval at all. What comes next is HyDE: rather than embedding the user's literal question, the system generates a hypothetical answer to it first and embeds that instead, which closes a lot of the semantic gap between how people ask questions and how the source document actually phrases things. Retrieval itself is hybrid — dense HNSW vector search and sparse BM25 keyword search, fused with Reciprocal Rank Fusion so exact terms like names, codes, and IDs and semantic similarity both count, without needing to normalize two different scoring scales against each other. A local BGE cross-encoder then reranks the results down to the top 5, which keeps the LLM's context window tight and avoids the "lost in the middle" problem, where a model quietly ignores relevant context buried past the first few results. Before generating anything, one batched LLM call grades every retrieved document's relevance at once, not one call per document. If fewer than half come back relevant, the system either falls back to a live web search or rewrites the query and retries, capped at two rounds total. Only once that bar is cleared does it generate an answer from the filtered context and chat history.

Chunking and Ingestion Happen in the Background

Uploading a document doesn't block on any of this. POST /upload stores the raw PDF straight to R2, creates a document_jobs row, and returns a job ID instantly — the actual work happens in the background. A Celery task picked up via CloudAMQP handles parsing (PyMuPDF) and chunking, while the client connects to a Server-Sent Events stream to watch progress move from 0% to 100% in real time instead of polling. Chunking itself is hierarchical: large ~1500-token parent chunks get stored for context, small ~300-token child chunks get embedded and indexed for precision. A child chunk is what actually gets matched during search, but its parent is what gets handed to the LLM — tight chunks for accurate retrieval, wide chunks for a model that actually has enough surrounding text to answer well. A Postgres trigger auto-populates a tsvector column on write, so the BM25 half of hybrid search stays in sync without a separate indexing job.

The Grading Step I Deleted

An earlier version of the pipeline added a second grading pass after generation — a Self-RAG-style check asking whether the answer itself was grounded in the retrieved context and actually useful, auto-rewriting on failure. It got cut. Measured live, the same question against the same document needed zero extra rounds on one run and three on another. "Is this answer grounded" turned out to be a much fuzzier judgment call than CRAG's per-document relevance grading, and it was the dominant source of latency variance in the whole system — pushing some runs out to roughly 145 seconds — without a proportionate improvement in correctness. CRAG's cheaper, more predictable retrieval-quality loop stayed; the fuzzier post-generation check didn't. Good engineering sometimes means deleting a feature once the data says it isn't earning its latency cost.

Traced End to End, Evaluated Offline

Every node in the graph — guardrail, HyDE, retrieve, rerank, grade, web search, generate — carries a named, traceable span in LangSmith, so a single request renders as one readable trace tree instead of a scattered pile of individual LLM calls. GET /api/admin/metrics pulls real run counts, success rate, latency, and token usage straight from the LangSmith API into a live dashboard — numbers pulled from actual traffic, not hardcoded claims. Quality gets checked offline, not on every request: a RAGAS harness generates a synthetic test set from real ingested documents and scores faithfulness, answer relevancy, and context precision/recall against it. It runs on demand, after a retrieval, reranking, or prompt change, as a regression gate — the goal is catching a quality drop before it ships, not manufacturing an improvement number on every commit.

Built So a Broken Piece Fails Loud, Not Silent

The edge gateway does more than cache. Every request it proxies to the FastAPI origin carries an internal verification header that the backend checks before doing any work at all — requests without it get rejected immediately, so the origin is never directly exposed to the public internet. On failure, the reverse proxy returns real 502/504s instead of an opaque CORS error, which makes the difference between "the origin is down" and "something is silently broken" actually visible to the client. The same instinct runs through the whole system: every LLM call already has a cross-provider fallback chain and a hard per-provider timeout, so one stuck or rate-limited provider can't stall a request. And when retrieval genuinely can't find relevant context, the pipeline is built to say so — an honest "I don't know" instead of a confident hallucination.