You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
iconv codec: encoding names that glibc strips to nothing resolve to the locale encoding #161924
Since the iconv-based codec (gh-152997, GH-153001) landed, codecs.lookup() on Linux/glibc accepts encoding names whose charset part glibc's iconv_open() reduces to nothing, and returns a codec for the current locale's encoding. Before 3.16 these names raise LookupError.
The result depends on the environment: with LC_ALL=C the same call fails with UnicodeEncodeError: '!!!' codec can't encode character '\xe9', because the codec is now ASCII.
name
3.15.2
3.16.0a2 (Linux, UTF-8 locale)
'!!!', '@', '()', 'é', '한글'
LookupError
codec for the locale encoding
'iconv:한글'
LookupError
codec for the locale encoding
',', '/', '//TRANSLIT', '//abc'
LookupError
codec for the locale encoding
'no-such-codec', '8', 'ü8'
LookupError
LookupError
Cause
iconv_search_function in Lib/encodings/__init__.py rejects only the empty name and then probes the name with _iconv_encode(name, ''). glibc's iconv_open() first removes trailing whitespace, / and , and any //-suffix such as //TRANSLIT (gconv_parse_code() in iconv/gconv_charset.c), then drops every remaining character that is not an ASCII alphanumeric, / or one of _-.,: (strip() in iconv/gconv_charset.h), and treats the resulting empty name as the charset of the current locale (__gconv_open in iconv/gconv_open.c: "If the string is empty define this to mean the charset of the currently selected locale"). So the probe succeeds for any name made only of dropped characters, and also for names whose charset part is empty once the suffix is removed, such as ',', '/', '//TRANSLIT' and '//abc'.
deficonv_search_function(encoding):
...
ifnotname:
returnNone# Test if the encoding is supported by iconv.try:
_iconv_encode(name, '')
except (LookupError, OSError):
returnNone
Possible fix
Before probing, reject a name unless the part before its first / or , contains an ASCII alphanumeric. Requiring an alphanumeric anywhere in the name is not enough: '//TRANSLIT' and '//abc' contain one and still resolve to the locale encoding.
I can reproduce the behavior described on Linux/glibc and will investigate the validation performed by iconv_search_function before _iconv_encode() is called. I plan to add regression tests covering names whose charset portion is stripped to an empty string, including cases such as !!!, @, //TRANSLIT, and //abc, and then implement the validation needed to preserve the previous LookupError behavior.
If there are no objections, I’d like to take this issue.
@Abhishekpandey123-git No objection from me, and I'm not planning a PR myself. CPython doesn't assign issues, so you don't need my sign-off. That said, since Serhiy has been cc'd, it may be worth waiting for his view on where the check should live before writing the patch.
Activity
Abhishekpandey123-git commentedon Nov 14, 2026
Hi @jseop-lim, I’d like to work on this issue.
I can reproduce the behavior described on Linux/glibc and will investigate the validation performed by
iconv_search_functionbefore_iconv_encode()is called. I plan to add regression tests covering names whose charset portion is stripped to an empty string, including cases such as!!!,@,//TRANSLIT, and//abc, and then implement the validation needed to preserve the previousLookupErrorbehavior.If there are no objections, I’d like to take this issue.
vstinner commentedon Nov 15, 2026
cc @serhiy-storchaka
jseop-lim commentedon Nov 17, 2026
@Abhishekpandey123-git No objection from me, and I'm not planning a PR myself. CPython doesn't assign issues, so you don't need my sign-off. That said, since Serhiy has been cc'd, it may be worth waiting for his view on where the check should live before writing the patch.
pythongh-161924: Expect DeprecationWarning for non-ASCII encoding nam…