learn
Tokens
What Tokens means, how it works, and which tools and models on this site relate to it.
Definition
Tokens are the pieces of text a model reads and writes — often subwords rather than whole words.
How it works
A tokenizer splits input into token IDs. Context windows, output limits, and API bills are usually measured in tokens, not characters.