Repository navigation
Fix SAS readability issue in written 32-bit sas7bdat - #421
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
32-bit SAS data files (sas7bdat) written by ReadStat are currently not correctly readable by SAS.
Currently, SAS will load the file without aborting with an error but it will start reading uncompressed row data from the wrong offset. As it starts 4 bytes too early, all strings are seemingly prefixed with a short bit of line noise and truncated if filling almost the whole width of the text column. Numbers suffer from the same problem and are in effect unrecognizable.
The Fix
The PR proposes to change the writing of the row size subheader for 32-bit SAS data files such that
Moving the row counter doesn't seem to have an actual effect with SAS but it aligns the produced file format with SAS, which is a good thing.
Switching the mystery flag/offset is the actual fix. Increasing it by 4 for 32-bit SAS data files shifts the offset for reading uncompressed rows to the right position as SAS now displays the files as expected. The value 5 is also what's produced in 32-bit files generated with SAS on Windows.