Issues

• High Token Inflation and Lack of Representation for Bangladeshi Bengali

  openai/tiktoken#593 · Filter by author NayemIslam12r opened on Aug 16, 2026

• Feature Request: SuperCompress integration example — token counting before/after compression

  openai/tiktoken#573 · Filter by author arjunkshah opened on Jun 30, 2026

• [Security] Reachable Rust panic (denial of service) when tokenizing with an untrusted encoder missing single-byte tokens

  openai/tiktoken#568 · Filter by author geo-chen opened on Jun 10, 2026
  1

• CoreBPE::new has an unconstrained NSE type parameter, making normal downstream calls fail type inference

  openai/tiktoken#557 · Filter by author fallintoplace opened on May 24, 2026

• Use Trusted Publishing to upload to PyPI

  openai/tiktoken#547 · Filter by author nsoranzo opened on May 19, 2026

• encode_with_unstable does not handle surrogate pairs like encode

  openai/tiktoken#541 · Filter by author shreejaykurhade opened on May 13, 2026
  1

• encode_ordinary_batch — reproducible multi-second tail stalls on 32-core box (o200k_base, num_threads=8)

  openai/tiktoken#530 · Filter by author daechoi opened on Apr 21, 2026
  1

• tiktoken 0.12 cannot handle qwen3.5 in huggingface

  openai/tiktoken#521 · Filter by author jslhcl opened on Apr 8, 2026
  1

• [Bug] tiktoken 0.9.0 ignores TIKTOKEN_OFFLINE and TIKTOKEN_CACHE_DIR, still attempts network download

  openai/tiktoken#513 · Filter by author gavin9508 opened on Mar 24, 2026
  1

• [RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)

  openai/tiktoken#511 · Filter by author nicezic opened on Mar 22, 2026
  1

• Add riscv64 (linux_riscv64) wheel to PyPI releases

  openai/tiktoken#502 · Filter by author gounthar opened on Mar 11, 2026

• Security: Cache Poisoning via Shared Temp Directory with Optional Hash Verification

  openai/tiktoken#500 · Filter by author spartan8806 opened on Mar 8, 2026

• Token count discrepancy between tiktoken and API response when messages contain tool calls

  openai/tiktoken#474 · Filter by author yangchou19 opened on Jan 6, 2026
  5

• Add support for tiktoken CLI

  openai/tiktoken#473 · Filter by author remyleone opened on Dec 22, 2025
  1

• Community Resource: HuggingFace to TikToken Converter

  openai/tiktoken#471 · Filter by author shakedzy opened on Dec 16, 2025

• Add Hooks for PyInstaller

  openai/tiktoken#469 · Filter by author MaidScientistIzutsumiMarin opened on Nov 24, 2025

• Feature Request: Allow tiktoken to accept a custom HTTP client

  openai/tiktoken#462 · Filter by author GabrielVGS opened on Nov 12, 2025

• bug: o200k_harmony duplicates special token id 200018

  openai/tiktoken#457 · Filter by author jinzhuer opened on Oct 13, 2025
  1

• 0.12.0 no longer provides manylinux2014 wheels and breaks builds in AWS SAM Python 3.11 images(requires Rust compiler)

  openai/tiktoken#456 · Filter by author jotasixto opened on Oct 10, 2025
  3

• Interest in porting any of the implementation from TokenDagger? (With Possible PR)

  openai/tiktoken#441 · Filter by author Jeffrharr opened on Aug 26, 2025

• Unable to Release Memory Used by Tiktoken Encoding After Setting to None and GC Collect

  openai/tiktoken#414 · Filter by author kimhyun5u opened on Jul 14, 2025

• dead lock

  openai/tiktoken#413 · Filter by author tiandalu opened on Jul 11, 2025

• Bug in buffer protocol implementation

  openai/tiktoken#405 · Filter by author mariosasko opened on Jun 1, 2025

• Fail to Install on win-arm64

  openai/tiktoken#403 · Filter by author MrHinsh opened on May 31, 2025
  1

• RuntimeError(StackOverflow) with tiktoken.encoding_for_model('gpt-4').encode(txt, disallowed_special=())

  openai/tiktoken#400 · Filter by author sumitKpandey opened on May 6, 2025
  2