Repository navigation
Stop savers and loaders crashing on file objects in get_file_metadata - #1822
Open
dhananjay6561 wants to merge 3 commits into
Open
dhananjay6561 wants to merge 3 commits into
dhananjay6561 wants to merge 3 commits into
Conversation
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Author
|
Hi @skrawcz @jernejfrank, small one when you have a moment. It fixes the IO-type TODO from the matplotlib writer in |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Many savers and loaders accept a file object as their path (pandas types it as
str | Path | BytesIO | BufferedReader, numpy asstr | Path | IO), but they all end withutils.get_file_metadata(self.path), which passes the object tourlparseand crashes. So the data gets written or read, and then the call fails:There is already a TODO about this in the matplotlib writer ("make utils.get_file_metadata() safer for when self.path is IO type"), so this fixes it in that one place instead of in every plugin.
Changes
get_file_metadatanow accepts a file object:name, e.g. whatopen()returns) is described by that file name, so size and mtime are filled in as before;io.BytesIOgetspath="<BytesIO>",size=Noneand a note saying file metadata is unsupported for it.pathstays a string because the OpenLineage, MLflow and experiments hooks read it as one.os.PathLikeis turned into a string withos.fspath(before onlypathlib.Pathwas).matplotlib_extensions.py.BytesIOand an open file handle.How I tested this
AttributeErrorabove and pass with the change.pytest tests/io tests/plugins/test_numpy_extensions.py(62 passed).NumpyNpyWriter/NumpyNpyReader,PandasCSVWriterandPandasPickleWriterwith aBytesIO: they return metadata instead of raising.ruff format --checkandruff check(0.16.9, the pre-commit version) on the touched files.Notes
Related to #1758 / #1763, which fix the Polars JSON and NDJSON classes on their own. The two don't conflict: with this change those classes would also stop crashing even without per-class checks. I left the
timestampline alone since #1785 is changing it.Checklist