Skip to content

fix(csv-parse): keep parsing after skipped max_record_size and rtrim errors - #546

Open
LuisFigueroaG wants to merge 1 commit into
adaltas:masterfrom
LuisFigueroaG:fix/skip-records-resume-parsing
Open

LuisFigueroaG wants to merge 1 commit into
adaltas:masterfrom
LuisFigueroaG:fix/skip-records-resume-parsing

Conversation

@LuisFigueroaG

Copy link
Copy Markdown

With skip_records_with_error, two errors don't behave like the others: CSV_MAX_RECORD_SIZE and CSV_NON_TRIMABLE_CHAR_AFTER_CLOSING_QUOTE are returned straight from the parse loop (return this.__error(...)). When the record is skipped, __error returns undefined, so parse just stops in the middle of the buffer without saving the rest in previousBuf.

The result is a skip event followed by silent data loss:

import { parse } from "csv-parse/sync";

parse('"a" b,c\nd,e\nf,g\n', { rtrim: true, skip_records_with_error: true });
// []  (expected [["d", "e"], ["f", "g"]])

With the stream API it can also produce wrong records, because the next chunk is concatenated with the stale previousBuf:

const parser = parse({ max_record_size: 5, skip_records_with_error: true });
parser.write("x,y\nab");
parser.write("cdefghij,c\nd,e\n");
parser.write("f,g\n");
parser.end();
// [["x", "y"], ["abf", "g"]]

This checks the result of __error like the other skippable errors, so the record is flagged and dropped in __onRecord and parsing goes on. Both checks are skipped once the record already has an error, otherwise every following character of the record would emit another skip event. Without skip_records_with_error nothing changes, since recordHasError is only set in that mode.

Tests are added to test/option.skip_records_with_error.ts for both errors, the max_record_size one with the oversized field split across writes. Both fail on master.

Validation (Ubuntu, Node 24.22.0):

  • npm test in packages/csv-parse: 631 passing, 2 pending
  • npm run lint:check: no errors (5 existing pending-test warnings)
  • prettier check on the changed files

…errors

With `skip_records_with_error`, the `CSV_MAX_RECORD_SIZE` and
`CSV_NON_TRIMABLE_CHAR_AFTER_CLOSING_QUOTE` errors returned straight out of
the parse loop. The record was reported as skipped, but the rest of the
current chunk was dropped, and the next chunk was parsed together with the
stale buffer of the previous one.

Handle them like the other skippable errors and only report them once per
record.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant