You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Files with ASCII filename and UTF-8 comment lost EFS flag when writing with zipfile #152845
According to the spec, the EFS flag enforces the filename and comment of a file entry to be UTF-8.
However, the EFS flag is not set when a file with ASCII filename and UTF-8 comment is written through zipfile. This renders its comment at risk of mis-decoding.
Similarly, a file with ASCII filename loses its EFS flag when the central directory is rewritten in append mode. This causes an unexpected metadata mutation and introduces a structural inconsistency between the immutable Local File Header (LFH) and the rewritten Central Directory Entry (CDE). This is also a regression of gh-84353/gh-150091, before which the EFS flag was not altered.
Additionally, the metadata_encoding parameter cannot be specified when an archive is opened with 'a' mode, disallowing a proper decoding with a customized codec.
A closely related sub-issue occurs in append mode: for a pre-existing entry with an ASCII-only filename and comment, its comment cannot be updated to UTF-8 safely in the future, since it will likely be decoded with cp437 or another local encoding.
I think we should consider a more robust encoding promotion logic. One straightforward approach is to enforce the EFS flag for all newly added entries.
The side effects should be minimal, as enabling the EFS flag (Bit 11) does not introduce any additional data payload, and the required "Version needed to extract" is identically at least 20 (2.0).
@KasraGhodsi In your gh-152845 I notice the concern about the potential misleading on automatic UTF-8 flagging merely due to a non-ASCII comment and the added support for metadata_encoding in append mode.
However, the patch did not fix the issue where a filename encoded with a non-cp437 encoding (e.g., Big5) be mutated when the central directory is rewritten in append mode—either be re-encoded with cp437 or UTF-8. This is why my implementation in gh-152846 has introduced the _metadata_encoding attribute to trace the correct original encoding. Do you have any good idea on how to fix this?
Activity
pythongh-152845: Fix missing EFS flag for files with UTF-8 comments
pythongh-152845: Allow metadata_encoding in all ZipFile modes
pythongh-152845: Fix missing EFS flag for files with UTF-8 comments
pythongh-152845: Allow metadata_encoding in all ZipFile modes
pythongh-152845: Fix missing EFS flag for files with UTF-8 comments
pythongh-152845: Allow metadata_encoding in all ZipFile modes
danny0838 commentedon Jul 4, 2026
PR 152845 should fix most of this issue.
A closely related sub-issue occurs in append mode: for a pre-existing entry with an ASCII-only filename and comment, its comment cannot be updated to UTF-8 safely in the future, since it will likely be decoded with
cp437or another local encoding.I think we should consider a more robust encoding promotion logic. One straightforward approach is to enforce the EFS flag for all newly added entries.
The side effects should be minimal, as enabling the EFS flag (Bit 11) does not introduce any additional data payload, and the required "Version needed to extract" is identically at least 20 (2.0).
pythongh-152845: Preserve ZIP encoding flags when appending
gh-152845: Preserve ZIP encoding flags when appending (gh-161950)
[3.14] gh-152845: Preserve ZIP encoding flags when appending (gh-161950…
[3.15] gh-152845: Preserve ZIP encoding flags when appending (gh-161950…
danny0838 commentedon Nov 18, 2026
@KasraGhodsi In your gh-152845 I notice the concern about the potential misleading on automatic UTF-8 flagging merely due to a non-ASCII comment and the added support for
metadata_encodingin append mode.However, the patch did not fix the issue where a filename encoded with a non-cp437 encoding (e.g., Big5) be mutated when the central directory is rewritten in append mode—either be re-encoded with cp437 or UTF-8. This is why my implementation in gh-152846 has introduced the
_metadata_encodingattribute to trace the correct original encoding. Do you have any good idea on how to fix this?