Skip to content

Keep current results when evicting stale versions from the diskcache adapter - #1823

Open
dhananjay6561 wants to merge 2 commits into
apache:mainfrom
dhananjay6561:fix-diskcache-evict-kept-results
Open

dhananjay6561 wants to merge 2 commits into
apache:mainfrom
dhananjay6561:fix-diskcache-evict-kept-results

Conversation

@dhananjay6561

Copy link
Copy Markdown

h_diskcache.evict_all_except (and evict_all_except_driver) is meant to drop every cached version of a node except the current one. It has two bugs:

  1. It evicts the results it should keep when the current version was run with more than one set of inputs. DiskCacheAdapter appends the node's hash to _nodes_history on every cache miss, so after running doubled(x) with x=1 and x=2 the history is [h, h]. history.remove(h) only takes out the first one, and the loop then evicts tag doubled.h, which deletes all the current results.
  2. It raises ValueError: list.remove(x): x not in list when the current version of a node has no cached results yet, e.g. you edited the function and call eviction before running it again.

Repro for the first one:

import tempfile
from hamilton import ad_hoc_utils, driver
from hamilton.plugins import h_diskcache

def doubled(x: int) -> int:
    print("computing", x)
    return x * 2

module = ad_hoc_utils.create_temporary_module(doubled)
path = tempfile.mkdtemp()

def build():
    return driver.Builder().with_modules(module).with_adapters(
        h_diskcache.DiskCacheAdapter(cache_path=path)).build()

build().execute(["doubled"], inputs={"x": 1})
build().execute(["doubled"], inputs={"x": 2})
h_diskcache.evict_all_except_driver(build())  # nothing is stale, but eviction_counter == 1
build().execute(["doubled"], inputs={"x": 1})  # prints "computing 1" again

Changes

  • evict_all_except dedupes the hashes in a node's history before evicting, and only keeps (and skips evicting) the current hash when it is actually in the history. Otherwise the node is left out of the new history, since nothing is cached for it.
  • Added two tests: one where the current hash is repeated (checks the cached values survive and the old version is evicted), and one where only an older version is in the cache.

How I tested this

  • Both new tests fail on main (assert 2 == 1 with the current results gone, and the ValueError), and pass with the change.
  • pytest tests/plugins/test_h_diskcache.py (8 passed).
  • The repro above no longer recomputes x=1 after eviction, and eviction after editing the function no longer raises.
  • ruff format --check and ruff check (0.16.9) on the touched files.

Notes

I know DiskCacheAdapter is deprecated in favour of Builder.with_cache(), but it is still shipped and this loses cached results silently, so it seemed worth a small fix. eviction_counter now counts each stale version once rather than once per history entry.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Signed-off-by: dhananjay6561 <dhananjayaggarwal6561@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 07:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@dhananjay6561

Copy link
Copy Markdown
Author

Hi @skrawcz @jernejfrank, another small fix when you get a chance. evict_all_except was deleting the current results whenever a node had run with more than one input, and crashed right after editing a node. Two tests added that fail on main. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants