xavier collantes

What are Tokens in LLMs?

• Full words ("hello")
• Parts of words ("un" + "likely")
• Characters ("a", "!")
• Spaces and punctuation

Why Tokens Matter

• Cost calculation: Most API-based LLM services charge based on token usage.
• Context window limits: Every model has a maximum number of tokens it can process at once (its "context window").
• Performance impact: More tokens generally mean more processing time and higher computational costs in terms of memory and time.

The Ambiguity Problem of Tokens

• Different tokenization algorithms: GPT models use tiktoken, Claude uses its own tokenizer, Llama uses SentencePiece, etc.
• i18n differences: Some models tokenize certain languages more efficiently than others.
• Special tokens: Models handle special tokens (like those for code, formatting, or system instructions) differently.

• 4 tokens in GPT-4
• 5 tokens in Claude
• 6 tokens in a different model

• Cost comparisons become difficult
• Context window utilization varies by model
• Performance benchmarks can be misleading if not accounting for tokenization differences

Measuring Tokens in Practice

Using tiktoken to Count Tokens

1import tiktoken
2
3def count_tokens(text, model="gpt-4"):
4    """Count the number of tokens in a text string."""
5    encoder = tiktoken.encoding_for_model(model)
6    tokens = encoder.encode(text)
7    return len(tokens)
8
9# Example usage.
10sample_text = "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration."
11
12print(f"GPT-4: {count_tokens(sample_text, 'gpt-4')} tokens")
13print(f"GPT-3.5: {count_tokens(sample_text, 'gpt-3.5-turbo')} tokens")
14print(f"davinci: {count_tokens(sample_text, 'text-davinci-003')} tokens")
15

Practical Tips for Working with Tokens

• Always measure before sending: Count tokens before sending requests to avoid errors or unexpected costs.
• Be aware of hidden tokens: System prompts, formatting, and special characters all count toward your token limits.
• Consider token efficiency: Some prompts can be rewritten to use fewer tokens while conveying the same information.
• Different models, different strategies: Adapt your prompt strategy based on the specific tokenization of your chosen model.
• Monitor token usage: Keep track of token consumption to optimize costs and performance. These are usually available in the API response or the provider's dashboard.

Further Reading