Repository navigation
Decode ShExC as UTF-8 first; use chardet only as a fallback - #117
Merged
Merged
Conversation
load_shex_file() trusted chardet whenever it was > 0.9 confident, but
chardet samples only the start of the data: for FHIR R4's fhir.shex
(690 KB, first non-ASCII byte ~250 KB in) it reports ascii with
confidence 1.0, and decode('ascii') raises UnicodeDecodeError. ShExC is
UTF-8 by definition, so try that first (utf-8-sig, which also strips a
BOM) and only guess for files that aren't valid UTF-8.
The old code also crashed on short non-UTF-8 files: chardet's low
confidence sent them to the UTF-8 default. Now chardet's guess is used
whatever its confidence, with latin-1 as a last resort that can't fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
load_shex_file()(inpyshex/utils/schema_loader.py) decoded with whatever chardet guessed whenever its confidence was above 0.9. chardet only samples the start of the data, so a large file that is ASCII for a long stretch before its first non-ASCII character gets reported asasciiwith confidence 1.0, and loading crashes:(chardet 7.4.3 on FHIR R4's 690 KB
fhir.shex.)Fix: ShExC is UTF-8 by definition, so decode as UTF-8 first (
utf-8-sig, which also strips a BOM). Only if that fails, use chardet's guess, withlatin-1as a last resort that can't fail. This also fixes a second crash: a short non-UTF-8 file got low confidence from chardet, so the old code sent it to the UTF-8 default and failed. chardet also no longer runs over every schema, only over files that aren't valid UTF-8.Tests (in
test_schema_loader.py, usingtmp_pathso nothing tracked is written):asciierror)utf-8error)Full suite (
SKIP_EXTERNAL_URLS=true): 452 passed, 0 failed. FHIR R4'sfhir.shexnow decodes and parses (896 shapes).pyshexc's own
generate_shexj.load_shex_filehas the same chardet-first logic. PyShEx doesn't call it, but pyshexc's CLI would hit the same crash.🤖 Generated with Claude Code