Repository navigation
fix(csv-parse): keep parsing after skipped max_record_size and rtrim errors - #546
Open
LuisFigueroaG wants to merge 1 commit into
Open
LuisFigueroaG wants to merge 1 commit into
LuisFigueroaG wants to merge 1 commit into
Conversation
…errors With `skip_records_with_error`, the `CSV_MAX_RECORD_SIZE` and `CSV_NON_TRIMABLE_CHAR_AFTER_CLOSING_QUOTE` errors returned straight out of the parse loop. The record was reported as skipped, but the rest of the current chunk was dropped, and the next chunk was parsed together with the stale buffer of the previous one. Handle them like the other skippable errors and only report them once per record.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
skip_records_with_error, two errors don't behave like the others:CSV_MAX_RECORD_SIZEandCSV_NON_TRIMABLE_CHAR_AFTER_CLOSING_QUOTEare returned straight from the parse loop (return this.__error(...)). When the record is skipped,__errorreturnsundefined, soparsejust stops in the middle of the buffer without saving the rest inpreviousBuf.The result is a
skipevent followed by silent data loss:With the stream API it can also produce wrong records, because the next chunk is concatenated with the stale
previousBuf:This checks the result of
__errorlike the other skippable errors, so the record is flagged and dropped in__onRecordand parsing goes on. Both checks are skipped once the record already has an error, otherwise every following character of the record would emit anotherskipevent. Withoutskip_records_with_errornothing changes, sincerecordHasErroris only set in that mode.Tests are added to
test/option.skip_records_with_error.tsfor both errors, themax_record_sizeone with the oversized field split across writes. Both fail on master.Validation (Ubuntu, Node 24.22.0):
npm testinpackages/csv-parse: 631 passing, 2 pendingnpm run lint:check: no errors (5 existing pending-test warnings)