Skip to content

Add UTF-8 fast-path to PyUnicode_DecodeLocale() - #21

Draft
vstinner wants to merge 1 commit into
mainfrom
WIP_current_locale_utf8
Draft

vstinner wants to merge 1 commit into
mainfrom
WIP_current_locale_utf8

Conversation

@vstinner

@vstinner vstinner commented Nov 7, 2026

Copy link
Copy Markdown
Owner
  • Add also UTF-8 fast-path to _Py_DecodeLocale() and _Py_EncodeLocale().
  • Add _PyRuntime.current_locale_is_utf8 to remember if the LC_CTYPE encoding is UTF-8.
  • Add _PyRuntime.cached_locale_encoding to make sure that setlocale(LC_CTYPE) or setlocale(LC_ALL) was not called outside Python.
  • Rename _Py_ERROR_UNKNOWN to _Py_ERROR_UNSET.
  • Rename _Py_ERROR_OTHER to _Py_ERROR_UNKNOWN.

HACK for the micro-benchmark:

  • Add bench_loc.py.
  • Add "y#" fast-path to Argument Clinic.
  • Move unicode_decodelocale from _testlimitedcapi to _testcapi to use faster Argument Clinic code.
  • Need to define _Py_IDENTIFIER() API if Py_BUILD_CORE macro is defined, so that _testcapi can test this API. _testcapi has to define Py_BUILD_CORE for Argument Clinic.

@vstinner

vstinner commented Nov 7, 2026

Copy link
Copy Markdown
Owner Author

Results of bench_loc.py:

Benchmark ref cache
decode python 73.0 ns 1.02x faster
decode locale 233 ns 3.20x faster
strerror 283 ns 1.98x faster
Geometric mean (ref) 1.86x faster

The UTF-8 cache makes os.strerror() 2.0x faster: 233 ns +- 4 ns => 143 ns +- 4 ns.

Raw numbers:

$ python -m pyperf show ref.json -q
decode python: Mean +- std dev: 73.0 ns +- 1.3 ns
decode locale: Mean +- std dev: 233 ns +- 4 ns
strerror: Mean +- std dev: 283 ns +- 8 ns

$ python -m pyperf show cache.json -q
decode python: Mean +- std dev: 71.5 ns +- 1.6 ns
decode locale: Mean +- std dev: 73.0 ns +- 1.4 ns
strerror: Mean +- std dev: 143 ns +- 4 ns

With the UTF-8 cache, PyUnicode_DecodeLocale() is basically as fast as bytes.decode("utf-8"), even if it has to call nl_langinfo() + strcmp() to check that the cache is still valid.

FIXME: _Py_UpdateLocale() is not called at Python startup, locale.setlocale() must be called to test the optimization.

@vstinner

Copy link
Copy Markdown
Owner Author

Results of python#161455 benchmark:

Benchmark ref decode_locale_cache
stat_missing() 1.06 us 814 ns: 1.31x faster
open_missing() 2.09 us 1.48 us: 1.41x faster
raise_oserror() 260 ns 241 ns: 1.08x faster
os.strerror(2) 288 ns 147 ns: 1.97x faster
Geometric mean (ref) 1.41x faster

I updated the branch so the cache is used without having to call locale.setlocale().

@vstinner
vstinner force-pushed the WIP_current_locale_utf8 branch from bfb9e28 to 225058b Compare November 17, 2026 00:09
* Add also UTF-8 fast-path to _Py_DecodeLocale() and
  _Py_EncodeLocale().
* Add _PyRuntime.current_locale_is_utf8 to remember if the LC_CTYPE
  encoding is UTF-8.
* Add _PyRuntime.cached_locale_encoding to make sure that
  setlocale(LC_CTYPE) or setlocale(LC_ALL) was not called outside
  Python.
* Rename _Py_ERROR_UNKNOWN to _Py_ERROR_UNSET.
* Rename _Py_ERROR_OTHER to _Py_ERROR_UNKNOWN.

HACKS for the micro-benchmark:

* Add bench_loc.py.
* Add "y#" fast-path to Argument Clinic.
* Move unicode_decodelocale from _testlimitedcapi to _testcapi
  to use faster Argument Clinic code.
* Need to define _Py_IDENTIFIER() API if Py_BUILD_CORE macro is
  defined, so that _testcapi can test this API. _testcapi has to
  define Py_BUILD_CORE for Argument Clinic.
@vstinner
vstinner force-pushed the WIP_current_locale_utf8 branch from 225058b to 791b3fe Compare November 17, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant