Token theory: a formal account of tokens as identity-similarity conversion interfaces, and its implications for LLM architecture
research information-theory transformer attention-mechanism tokenization machine-learning-theory vector-quantization llm token-theory
-
Updated
Sep 22, 2026