Skip to content

Stop savers and loaders crashing on file objects in get_file_metadata - #1822

Open
dhananjay6561 wants to merge 3 commits into
apache:mainfrom
dhananjay6561:fix-file-metadata-file-objects
Open

dhananjay6561 wants to merge 3 commits into
apache:mainfrom
dhananjay6561:fix-file-metadata-file-objects

Conversation

@dhananjay6561

Copy link
Copy Markdown

Many savers and loaders accept a file object as their path (pandas types it as str | Path | BytesIO | BufferedReader, numpy as str | Path | IO), but they all end with utils.get_file_metadata(self.path), which passes the object to urlparse and crashes. So the data gets written or read, and then the call fails:

import io, numpy as np, pandas as pd
from hamilton.plugins.numpy_extensions import NumpyNpyWriter
from hamilton.plugins.pandas_extensions import PandasCSVWriter

NumpyNpyWriter(path=io.BytesIO()).save_data(np.arange(3))
PandasCSVWriter(path=io.BytesIO()).save_data(pd.DataFrame({"a": [1, 2]}))
# AttributeError: '_io.BytesIO' object has no attribute 'decode'

There is already a TODO about this in the matplotlib writer ("make utils.get_file_metadata() safer for when self.path is IO type"), so this fixes it in that one place instead of in every plugin.

Changes

  • get_file_metadata now accepts a file object:
    • a handle opened on a local file (anything with a string name, e.g. what open() returns) is described by that file name, so size and mtime are filled in as before;
    • an in-memory buffer like io.BytesIO gets path="<BytesIO>", size=None and a note saying file metadata is unsupported for it. path stays a string because the OpenLineage, MLflow and experiments hooks read it as one.
  • Any os.PathLike is turned into a string with os.fspath (before only pathlib.Path was).
  • Removed the now resolved TODO in matplotlib_extensions.py.
  • Added tests for a BytesIO and an open file handle.

How I tested this

  • New tests fail on main with the AttributeError above and pass with the change.
  • pytest tests/io tests/plugins/test_numpy_extensions.py (62 passed).
  • Ran the snippet above for NumpyNpyWriter/NumpyNpyReader, PandasCSVWriter and PandasPickleWriter with a BytesIO: they return metadata instead of raising.
  • ruff format --check and ruff check (0.16.9, the pre-commit version) on the touched files.

Notes

Related to #1758 / #1763, which fix the Polars JSON and NDJSON classes on their own. The two don't conflict: with this change those classes would also stop crashing even without per-class checks. I left the timestamp line alone since #1785 is changing it.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 07:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@dhananjay6561

Copy link
Copy Markdown
Author

Hi @skrawcz @jernejfrank, small one when you have a moment. It fixes the IO-type TODO from the matplotlib writer in get_file_metadata itself, so numpy, pandas and the other savers/loaders stop crashing on a BytesIO after the data is already written. Happy to adjust the <BytesIO> path naming if you'd prefer something else.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants