Folio

Overview

Folio started as a general-purpose "chat with your PDF" RAG system, then got rebuilt around a much harder constraint: a legal contract analysis tool where the answer isn't allowed to just sound right, it has to be checkable. Upload a contract and ask a question, and every factual claim the model makes comes back wrapped in a quote — and before that quote is ever shown, my own code re-locates it in the actual extracted document text, independent of anything the model claims about where it came from. If the quote isn't genuinely in the document, it's flagged as unverified rather than presented as real. That one constraint — never trust the model's citation, verify it yourself — is the thing the rest of the system is built around. It's split across two deployments: bbygrl, a Next.js frontend with Supabase auth, and a FastAPI backend (internally still named thotqen) running on Hugging Face Spaces — with a Cloudflare Worker sitting in front of both as an edge gateway.

Quotes You Can Actually Check

The model is instructed to wrap every verbatim quote it uses in a sentinel marker. Once it finishes generating, a verification step re-locates each quote in the document's raw extracted text itself — never trusting any page number or offset the model reports, because models hallucinate those just as readily as they hallucinate content. Matching is whitespace-tolerant but not fuzzy: PDF extraction adds and strips line breaks and spaces without changing the actual words (a word hyphenated across a line wrap is a common real example), so an exact-string match would wrongly reject quotes that are genuinely present. The matcher tokenizes the quote and requires every word to appear, in order, verbatim, with only whitespace allowed to vary — a paraphrase doesn't verify, only a real quote does. A quote that can't be located gets flagged unverified in the UI rather than silently removed, so the failure mode is visible instead of just quietly worse.

Click a Quote, See It in the Document

A verified quote isn't just a checkmark — clicking it opens the source document and scrolls to, and highlights, the exact passage, for both PDF and DOCX. PDFs render through react-pdf's real text layer; DOCX converts to HTML client-side via mammoth. Both end up as ordinary DOM text nodes, so one shared utility walks either rendering, builds a whitespace-tolerant pattern from the quote (the same matching rule the backend verification uses), and wraps the matched span in a highlight — regardless of how many separate text-layer elements or table cells the match happens to be split across.

Stack

• Next.js (React 19), Supabase Auth, react-pdf, mammoth — frontend (bbygrl)
• Cloudflare Workers — edge gateway: CORS, semantic cache, R2 document serving, reverse proxy
• FastAPI, LangGraph, Celery + CloudAMQP — backend (internally named thotqen), hosted on Hugging Face Spaces via Docker
• Supabase PostgreSQL with pgvector + HNSW indexing, plus a custom hybrid_search() SQL function scoped per-document
• PyMuPDF + python-docx for extraction (with per-page character offsets for citation lookup); local embeddings via sentence-transformers and a local BGE cross-encoder reranker
• Multi-provider LLM fallback: NVIDIA, Groq (2 accounts), Gemini (2 accounts), OpenRouter, self-hosted vLLM
• Tavily web search, with DuckDuckGo as a fallback, for corrective retrieval
• LangSmith for tracing, RAGAS for offline evaluation

Three Layers Before a Model Ever Sees the Question

Every request follows the same path before it reaches a model. The Next.js client sends a REST or SSE request to a Cloudflare Worker edge gateway. That gateway enforces CORS against an explicit origin allowlist, checks a semantic cache in Cloudflare KV, and serves stored documents directly from R2 at /edge/docs/* — bypassing the origin entirely for static file requests. Only a cache miss reaches the FastAPI backend. The backend classifies the query's intent, routes it through the LLM fallback chain, and hands it to a LangGraph pipeline that retrieves from Postgres/pgvector and R2 — scoped to exactly the document(s) attached to the current chat, not pooled across every document the user has ever uploaded. Every node in that pipeline is traced through LangSmith, so a single request is debuggable end to end instead of disappearing into a stack of opaque API calls.

Not Every Question Needs the Same Firepower

A fast, cheap classifier looks at each query first and labels it cheap or complex before it reaches a real answering model — a one-word greeting and a multi-document synthesis question shouldn't cost the same. Complex queries route through NVIDIA-hosted Llama 3.1 70B, then Groq's second account running Llama 3.3 70B, then Gemini's second account, then OpenRouter's free auto-router (tried twice — a different free model each draw), then Groq's first account, then Gemini's first account, then a self-hosted vLLM instance as the final no-API-key fallback. Cheap queries run the same chain minus the NVIDIA hop. Every provider gets a hard 18-second cutoff enforced externally, through a ThreadPoolExecutor, because not every provider's SDK actually honors its own timeout parameter. And each fallback is a full provider switch, not a retry: .with_fallbacks() moves to the next provider on any failure, rather than hammering an already-failing one.

A Retrieval Graph That Grades Its Own Work

The actual answering happens inside a stateful LangGraph, not a single prompt-and-response call. An input guardrail checks for prompt injection or unsafe content first. Then HyDE: rather than embedding the user's literal question, the system generates a hypothetical answer to it first and embeds that instead, closing a lot of the semantic gap between how people ask questions and how a contract actually phrases things. Retrieval is hybrid — dense HNSW vector search and sparse BM25 keyword search, fused with Reciprocal Rank Fusion so exact terms like names, dollar figures, and clause numbers and semantic similarity both count. A local BGE cross-encoder reranks down to the top 5. One batched LLM call grades every retrieved document's relevance at once; if fewer than half come back relevant, the system falls back to a live web search or rewrites the query, capped at two rounds. The model is also explicitly told what it's looking at is the most relevant excerpts, not the whole document — so it says "not found in what was retrieved" instead of confidently declaring a clause absent from a document it only partially read, which matters once you're past a page or two and genuinely can't fit the whole thing in one prompt.

Asking Across Documents, and Diffing Two Versions

A question can target several documents at once, selected from a library view — the answer compares across them rather than listing separate answers per document, and every quote is still verified against its own specific source, not the pooled set, so a citation never gets attributed to the wrong contract. Comparing two versions of a contract works differently from a straight diff on purpose: paragraphs are matched with a sequence alignment (not a character-by-character diff, which fragments on a single reworded word), and each changed pair goes to the model for a plain-language significance rating — a reworded sentence is "minor"; a liability cap or payment figure changing is "major," graded with an explicit instruction that a rewrite isn't the same thing as a substantive change even when every word in the paragraph is different.

An Agent That Has to Go Look Things Up

There's a second mode for a single document: instead of pre-fetching context and stuffing it into one prompt, the model gets three tools — search_document, get_section, list_clauses — and has to decide for itself what to look up before it's allowed to answer, across a bounded number of rounds, with each step shown live as it happens. The tool-calling itself is a hand-parsed TOOL_CALL/FINAL_ANSWER protocol rather than any provider's native function-calling — the backend already runs a five-provider fallback chain specifically so one dead API key doesn't take the feature down, and function-calling support is inconsistent enough across that chain (a self-hosted vLLM instance in particular needs specific server flags most default setups don't have) that relying on it would reintroduce the single point of failure the fallback chain exists to remove. The upside of parsing it by hand: a malformed or invented tool call — wrong name, missing arguments, broken JSON — degrades to an error string fed back to the model instead of crashing the request, because the parser has to handle that case by construction rather than being able to assume a provider already validated it.

Ingestion Happens in the Background, for Either Format

Uploading a document doesn't block on any of this. POST /upload rejects anything that isn't a PDF or DOCX with a clear message, stores the raw file straight to R2, creates a document_jobs row, and returns a job ID instantly — the actual work happens in the background via Celery, picked up through CloudAMQP. A scanned PDF with no extractable text is caught and rejected explicitly, rather than silently saved as an empty, unusable document and reported as a success. Chunking is hierarchical: large ~1500-character parent chunks get stored for context, small ~300-character child chunks get embedded and indexed for precision — a child chunk is what gets matched during search, but its parent is what gets handed to the model, so retrieval stays precise while generation still has enough surrounding text to actually answer well.

The Grading Step I Deleted

An earlier version of the pipeline added a second grading pass after generation — a Self-RAG-style check asking whether the answer itself was grounded in the retrieved context, auto-rewriting on failure. It got cut. Measured live, the same question against the same document needed zero extra rounds on one run and three on another. "Is this answer grounded" turned out to be a much fuzzier judgment call than CRAG's per-document relevance grading, and it was the dominant source of latency variance in the whole system — pushing some runs out past two minutes — without a proportionate improvement in correctness. CRAG's cheaper, more predictable retrieval-quality loop stayed; the fuzzier post-generation check didn't. Good engineering sometimes means deleting a feature once the data says it isn't earning its latency cost.

Traced End to End, Evaluated Offline

Every node in the graph — guardrail, HyDE, retrieve, rerank, grade, web search, generate — carries a named, traceable span in LangSmith, so a single request renders as one readable trace tree instead of a scattered pile of individual LLM calls. GET /api/admin/metrics pulls real run counts, success rate, latency, and token usage straight from the LangSmith API into a live dashboard. Quality gets checked offline: a RAGAS harness generates a synthetic test set from real ingested documents and scores faithfulness, answer relevancy, and context precision/recall against it, run on demand after a retrieval or prompt change as a regression gate rather than a number manufactured on every commit.

Built So a Broken Piece Fails Loud, Not Silent

The edge gateway proxies every request to the FastAPI origin with an internal verification header the backend checks before doing any work — requests without it get rejected immediately, so the origin is never directly exposed to the public internet. On failure, the reverse proxy returns real 502/504s instead of an opaque CORS error. The same instinct runs through the whole system: every LLM call has a cross-provider fallback chain and a hard per-provider timeout, so one stuck or rate-limited provider can't stall a request. When retrieval genuinely can't find relevant context, the pipeline says so instead of guessing. And the verified-quotes layer is really the same philosophy applied one level higher — not just "the system should fail loud when something breaks," but "the system should fail loud when the model itself might be wrong," which is the harder and more important version of the same problem.