Token counter
Exact GPT-4o token counts, per-model cost, and how much of each context window a prompt actually fills.
/tokens
Free · no account · nothing uploaded
RAG text chunker and split visualizer
Chunking decides what your retriever can find. This tool splits your text the way a pipeline would, and draws every resulting chunk in proportion so you can see where the boundaries actually landed.
Text chunker
Say it the way you would say it to a person. Your words are matched to one of 18 tools by readable rules, and the tool runs right here in your browser. No upload, no account.
This tool runs in your browser and needs JavaScript turned on.
esc
How to use the text chunker
- Add your document: paste it into the message box, or drop a .txt or .md file into “Your files”.
- Say how to split it — “chunk size 400 with 50 overlap” or “split by markdown headings” — or open the settings form.
- Switch between Markdown sections, sentence boundaries and fixed character windows to compare them.
- Inspect every chunk drawn to scale with its overlap, then download the chunks as JSON.
Things you can type: split it into chunks chunk size 400 with 50 overlap
Retrieval quality is mostly a chunking problem
Most bad answers from a retrieval system are not model failures. They are chunking failures: the passage that held the answer was cut in half, so neither half scored well enough to be retrieved, and the model answered from whatever came back instead.
The parameters are simple, and their effects are not. Chunk size trades recall against precision. Overlap protects boundaries at the cost of index size and duplicate hits. The strategy — fixed windows, sentence boundaries, Markdown structure — decides whether a chunk is a coherent thought or an arbitrary slice.
Three strategies, compared on your own text
Paste a document and switch between them:
- Fixed — a hard character window. Predictable, cheap, and perfectly willing to cut a sentence in half.
- Sentence — windows that close at sentence boundaries, so a chunk ends where a thought ends.
- Markdown — splits on document structure, keeping headings with the section they introduce.
Lossless by construction
The chunker tracks byte ranges into the original text rather than rebuilding strings. That sounds like an implementation detail until you meet the alternative: an earlier version concatenated substrings and silently dropped blank lines, so 27 of 30 characters survived and the output still looked entirely plausible.
The test suite now asserts that zero-overlap chunks reproduce the input exactly, for all three strategies. What you see is what your pipeline would index.
Read the visualization, not just the numbers
Each chunk is drawn at its true relative size, with overlap regions marked. A ragged bar chart tells you a strategy is fighting your document — usually a sign that the structure you are splitting on is not the structure the document actually has.
Once the chunks look right, take the same text to the retrieval playground and confirm the ranking finds the passage you expect.
Questions
Text chunker: common questions
What chunk size should I use?
There is no universal answer, which is the reason this tool exists. Prose usually settles between 500 and 1,500 characters with 10–20% overlap; dense reference material and code want smaller chunks. Try your own document and look at where the boundaries fall.
Does overlap actually help?
It helps when answers straddle boundaries, which is common in narrative and legal text. It costs index size and produces near-duplicate hits. Set it to zero here and watch what gets cut — that is the fastest way to decide.
Is my document uploaded anywhere?
No. The chunker runs in your browser tab. Nothing is sent to a server, and there is no account to create.