Repository navigation
avoid duplicate byte order marks when appending text - #260
gheshm-jpg wants to merge 1 commit into
Conversation
Wkhanex
left a comment
There was a problem hiding this comment.
At f6712df, I checked repeated appends of empty text, Greek characters, a non-BMP character and mixed newline sequences for UTF-8, UTF-16BE, UTF-16LE, UTF-16, UTF-32 and UTF-8-SIG. For each encoding I used both a missing and a pre-touched file, with linesep='\n', linesep='' and linesep=None.
In all 36 combinations, the bytes matched encoding the complete expected text once, and the decoded contents matched as well (72 assertions). The base fails the UTF-16 case because intermediate BOMs remain. The existing unicode/append-BOM test selection also passes: 13 tests on Linux, CPython 3.12.14. I did not run Windows or the full suite.
Review prepared with Codex assistance; the checks described above were executed locally.
appending with
write_text(..., encoding='UTF-16', append=True)currently inserts another byte-order mark in the middle of the file. the unicode test already skips the append checks for this reason. UTF-32 and UTF-8-SIG have the same problem.this uses a text stream for append mode so the encoder handles the file position and writes the initial signature only when needed.
newline=''preserves the method's existing explicit newline normalization. overwrite mode keeps its existing encode-before-write behavior.removes the UTF-16 early return, extends unicode coverage to UTF-32 and UTF-8-SIG, and checks repeated appends starting from both a missing file and an empty file. byte expectations now encode the combined text once, so duplicate signatures are detected.
validation on macOS, Python 3.13.7:
python -m pytest -q: 168 passed, 1 skipped, and 7 known ruff failures. these match the preexisting failures documented on the unchanged base in the same environment; this is not a clean full-check pass.git diff --checkpassed.