Chat with your documents, privately, on your own GPU

Retrieval-augmented generation, with no server

Everything a hosted “chat with your PDF” service does on its machines happens here in the tab. pdf.js reads the text of each PDF page by page; Word files are unzipped by the browser itself. The text is cut into passages of about 900 characters that end at paragraph and sentence breaks and remember their file, page and heading.

Each passage is turned into a vector by a small embedding model (bge-small-en-v1.5, or multilingual-e5-small for about a hundred languages) running with transformers.js on your GPU, or on the CPU where WebGPU is missing. A question is searched two ways at once, by meaning against those vectors and by keyword with BM25, and the two rankings are merged by reciprocal rank fusion. The best passages that fit the model’s 4,096-token window are numbered and handed to Qwen3.5, run by WebLLM on your graphics card, with one instruction: answer only from these, cite each claim, and say so when the answer is not there.

What you get for each question

• A short answer in which every citation is a button that brings its passage into view.
• The numbered passages the model was given, each with its file, page and section, and why it was chosen: its rank by meaning and its rank by keyword.
• A check of the reply: how many sentences cite nothing, and any citation of a passage the model was never given.
• The documents that were searched, and whether each was indexed now or already saved in this browser.
• The answer and its sources as a Markdown file you can keep.

How heavy it is

Two models do the work. The search model is small: 34 MB when it runs on the CPU, 67 or 133 MB on the GPU depending on whether your card supports 16-bit shaders. It downloads the first time you ask. The answer model is the large one: Qwen3.5 2B is a 1.1 GB download and needs about 2.5 GB of GPU memory, and Qwen3.5 4B is 2.4 GB and needs about 4.5 GB. It downloads only after you allow it, and until then you still get the passages.

Both are fetched once from Hugging Face, along with WebLLM’s compiled GPU program from GitHub, and kept in the browser’s own storage, so later questions start without a download and work offline. Indexed documents are saved too, under a fingerprint of their contents, so asking about the same file tomorrow needs no indexing. The settings form lists what is saved and removes any of it.

What a small model is good at, and what it is not

Given the right passages, a 2-billion-parameter model reads and quotes them well: figures, dates, names and clauses come back as the document wrote them. It is weaker at reasoning across many passages, and it can fasten on a passage that is close to the question but not quite about it. That is why every claim has to carry a citation, why the citations are checked, and why the passages are always on screen beside the answer.

Two things are out of reach for now. Scanned PDFs have no text to read until they are run through OCR, which this tool does not do, and a summary of a whole long document is more than a handful of passages can carry.

From one question to a research library

A tab can answer questions about a handful of files. The research workspace FlexContext is being built for keeps hundreds of sources in one project, keeps every answer tied to the passage it came from, and lets you return to that passage months later.