A developer is told the model processes text as tokens rather than whole words. What is a token most accurately?

Why this is the answer

A token is a chunk of text — often a whole common word, often a fragment of a longer or rarer one — and it is the unit the model reads and emits. Whitespace splitting is close enough to mislead but wrong on long words, punctuation and code; an embedding is what a token is converted into rather than what it is; and byte-level modelling is a real design that is not how these tokenizers work.