Tag

#tokenizer

2 articles

Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face tokenizers Crate

Perplexity AI open-sources a new Unigram tokenizer that reduces p50 latency by 5x and cuts CPU utilization by 5-6x compared to Hugging Face tokenizers.

May 2825

First token counts reveal Opus 4.7 costs significantly more than 4.6 despite Anthropic's flat pricing

Learn how tokenizers work in AI models and why changes to text processing can dramatically affect costs, even when prices per token stay the same.

Apr 1941