[kalebbroo]

Stacks on #797 (feat/kimi-k31-text-route). Base this PR on that branch, not on main; merge #797 first.

What changed

The Moonshot repo ships tiktoken.model and no tokenizer.json or chat_template.jinja, so #797 refused the real checkpoint. The Text path now loads the rank file when tokenizer.json is absent.

• TiktokenFileTokenizer (ModelAssets, shared): loads a tiktoken rank file with a pre-tokenizer regex and a special-token table. It reuses the existing TiktokenConverter and HfTokenizerJson, so no second BPE is added. Also CountRanks and WithReservedRange (<|reserved_token_N|> fill, as in tokenization_kimi.py).
• KimiK31Tokenizer (LLM/Kimi): the tokenization_kimi.py pat_str rewritten for .NET (see below), plus the special table from tokenizer_config.json and the reserved range.
• KimiK31ChatTemplate (LLM/Kimi): a text-only port of build_chat_segments from encoding_k3.py. Each segment is encoded separately, as the reference does. Tools, tool turns, tool calls, images, response formats and prefilled turns are refused by name.
• Hook in KimiK31TextModel.Load and HfTextDirectoryLoader.RequireKimiK31Runnable. tokenizer.json wins when present and keeps its old path. Without it, tiktoken.model is used, and the template is chat_template.jinja if present, else the built-in template. A directory with neither tokenizer file is still refused. KimiK31Model.cs and KimiK31WeightLoader.cs are not touched.
• CHANGELOG: updated the existing Unreleased Kimi sentence (no longer "refused until supplied"). No version bump.

Real-repo facts

• File list: https://huggingface.co/api/models/moonshotai/Kimi-K3.1-Pro/tree/main. 96 safetensors shards plus config.json, configuration_kimi_k3.py, encoding_k3.py, generation_config.json, kimi_k3_processor.py, kimi_k3_vision_processing.py, media_utils.py, preprocessor_config.json, tiktoken.model, tokenization_kimi.py, tokenizer_config.json, README.md, LICENSE. No tokenizer.json. resolve/main/tokenizer.json and resolve/main/chat_template.jinja both return 404.
• tiktoken.model (https://huggingface.co/moonshotai/Kimi-K3.1-Pro/resolve/main/tiktoken.model): 2,795,286 bytes, 163,584 lines of base64 rank, ranks dense from 0. All 256 single bytes present. No token shaped like <|...|>, so the GgufTokenizer template-shape heuristic does not misclassify base tokens.
• tokenizer_config.json: tokenizer_class TikTokenTokenizer; 16 named added tokens at ids 163584 to 163839 (non-contiguous); eos_token is [EOS] = 163585.
• config.json and generation_config.json: bos_token_id 163584, eos_token_id 163586 (<|end_of_msg|>), pad_token_id 163839, vocab_size 163840. The loader uses the config ids, so EOS is 163586. The tokenizer_config.json eos_token disagrees and is not used.
• tokenization_kimi.py: pat_str (Han, cased letters, digits 1 to 3, punctuation, whitespace), num_reserved_special_tokens = 256, TikTokenTokenizer.
• encoding_k3.py: the XTML chat format. apply_chat_template defaults thinking_effort to max and adds a thinking-effort system turn when thinking is on.

Tests

• ModelAssets CPU lane: 28 passed, 0 failed (10 new in TiktokenFileTokenizerTests).
• LLM CPU lane: 202 passed, 0 failed (24 new: pre-tokenizer rows and the tokenizer and template facts).
• Engine CPU lane: 317 passed, 1 skipped (an ffmpeg fixture, unrelated). 3 new Kimi route tests (tiktoken-only load, jinja precedence, refusal by name).
• Real vocabulary (KimiK31RealVocabularyTests, Integration, skips unless HARTSY_KIMI_TOKENIZER_DIR names a directory with tiktoken.model): 2 passed locally against the downloaded tiktoken.model. Checks six text encodings and the 93-id default chat prompt, plus decode round-trip. No GPU, no checkpoint weights.

Not verified

• Real vocabulary and the tiktoken library. The expected ids come from a Python re-implementation of tiktoken's BPE, run with the regex module on the published pattern. The tiktoken package is not installed, so the ids are not checked against tiktoken itself.
• Chat template. encoding_k3.py was not executed, because it is downloaded code. The expected prompt text was derived by reading the source and mirroring its segmentation. The Python ports are not an oracle for the template; running the real encoding_k3.py on the same messages would be.
• Pattern translation. .NET has no \p{Han} or && set intersection. Han is written as its BMP blocks plus surrogate-pair ranges for planes 2 and 3. .NET letter classes are UTF-16 based, so astral letters outside Han can split differently from Rust regex.
• Decode drops special tokens, as GgufTokenizer does everywhere. tiktoken would print the literal text of control tokens.
• Stop ids are the config EOS only. [EOT] (163593) and other end markers are not verified as stops.
• Output. The model's <think> channel is not parsed. The engine's numeric reasoning effort is ignored; the reference fixes max.
• Generation has not been run on the real checkpoint.

Notes

• Changelog: the existing Unreleased line in this series was edited; it was not given a new entry.
• Author: kalebbroo only. No co-author or generated-by lines.