Skip to content

Add a functools.cache variant for methods to avoid keeping instances alive #102618

Description

@stevendaprano

functools.cache, when applied to a method, caches the self argument. This can also keep the instance alive long after it is no longer needed. See for example this discussion.

Following the same API as singledispatch and singledispatchmethod, we should consider adding a cachemethod decorator that either avoids caching self, or uses a weakref for it.

Linked PRs

Activity

  1. ktbarrett commented on Jun 11, 2024

    @ktbarrett

    To me the fact that lru_cache on methods passes the self parameter is obvious as a regular user of the language. The issue in my mind is that the exact caching mechanism is not mentioned and the problem where this causes a "leak" is due to an implementation detail that isn't explicitly mentioned. The documentation could hint at the potential pitfall by stating that important implementation detail explicitly.

    Something along the lines of...

    All arguments passed to the cached function are retained by the cache with strong references until its eviction. This means there may be cases where the cache is the only thing with strong references to the argument objects.

  2. nmoreaud commented on Jun 27, 2024

    @nmoreaud

    I encountered this behavior too, which results in a memory leak.
    I think that there should be a warning in the documentation, because the cache seems to work on method whereas actually it leaks memory: https://docs.python.org/3/library/functools.html#functools.lru_cache
    The best would be, of course, to have a cached_method decorator which doesn't leak.
    A debug assertion would have saved me a lot of time, but it is probably not possible to implement it because of the additional performance cost of such a test and because there is an official example here: https://docs.python.org/3/faq/programming.html#how-do-i-cache-method-calls

  3. dsal3389 commented on Aug 13, 2024

    @dsal3389

    encountered it too, I wrote a module for it until python supports it officially, PR's are more then welcome
    https://github.com/dsal3389/cachemethod

  4. nmoreaud commented on Aug 13, 2024

    @nmoreaud

    encountered it too, I wrote a module for it until python supports it officially, PR's are more then welcome https://github.com/dsal3389/cachemethod

    There are existing implementations in kids.cache and cachetools (more manual) packages

  5. ktbarrett commented on Aug 14, 2024

    @ktbarrett

    @nmoreaud kids.cache is not mantained and I don't see such a feature. And cachetools as mentioned is more manual.

    @dsal3389 Ideally something could be made to work just like cached_property, where it rewrites the attribute on the instance to be a cached method like cached_property rewrites the attribute to be the value that was computed.

    class cached_method:
    
        def __init__(self, method):
            self._method = method
            update_wrapper(self, method)
    
        def __set_name__(self, owner, name):
            self.__name__ = name
            self.__qualname__ = f"{type(owner).__qualname__}.{name}"
    
        def __get__(self, instance, objtype=None):
            if instance is None:
                return self
            
            try:
                return instance.__dict__[self.__name__]
            except AttributeError:
                cache = {}
    
                @wraps(self._method)
                def lookup(*args, **kwargs):
                    key = (args, tuple(kwargs.items()))
                    try:
                        return cache[key]
                    except KeyError:
                        res = self._method(instance, *args, **kwargs)
                        cache[key] = res
                        return res
    
                instance.__dict__[self.__name__] = lookup
                return lookup
    
        def __call__(self, instance, *args, **kwargs):
            func = getattr(instance, self.__name__)
            return func(*args, **kwargs)

    EDIT 1: Support type(obj).attr(inst) by implementing __call__.

    EDIT 2: Use __set_name__ for setting the object name to support attr = cached_method(lambda: stuff(123))

    EDIT 3:
    In __get__ first try to look up the cached function call from the object since some builtin functions do some very funny lookups.

    For example, len(a) does not call the equivalent of getattr(a, '__len__')(), it looks in the class first. And it doesn't do the equivalent of getattr(type(a), '__len__')(a) either, __get__ is always invoked with the instance object. So without this trial getattr, the implementation would recreate the cache and function on every call of len(a).

  6. ktbarrett commented on Aug 14, 2024

    @ktbarrett

    Has anyone filed a report against linting tools to warn when functools.cache and functools.lru_cache are used on methods? This is assuming that this will never be fixed in the language proper.

  7. AlexWaygood commented on Aug 14, 2024

    @AlexWaygood
    Member

    Has anyone filed a report against linting tools to warn when functools.cache and functools.lru_cache are used on methods?

    @ktbarrett, both flake8-bugbear and ruff already have this rule. The rule is B019 in both tools.

  8. patrick-kidger commented on Apr 29, 2025

    @patrick-kidger

    For those like me who landed on this issue, here's a cachemethod that solves this.

    Notably, it:

    • works on self types without a __dict__;
    • works on self types that are not hashable;
    • should be reasonably efficient as it avoids re-hashing the runtime arguments;
    • does not rely on descriptors, so it will cache even when called via SomeClass.some_cached_method(self, *args, **kwargs), not just self.some_cached_method(*args, **kwargs);
    • is static typing compatible.
    import functools as ft
    import inspect
    import weakref
    from collections.abc import Callable
    from typing import TYPE_CHECKING, Concatenate, ParamSpec, TypeVar
    
    _Self = TypeVar("_Self")
    _Params = ParamSpec("_Params")
    _Return = TypeVar("_Return")
    
    
    # Like `weakref.WeakKeyDictionary`, but uses identity-based hashing and equality.
    class _WeakIdDict:
        __slots__ = ("_dict",)
    
        def __init__(self):
            self._dict = {}
    
        def __getitem__(self, key):
            item, _ = self._dict[id(key)]
            return item
    
        def __setitem__(self, key, value):
            id_key = id(key)
            ref = weakref.ref(key, lambda _: self._dict.pop(id_key))
            self._dict[id_key] = (value, ref)
    
    
    # Used to compute an object's hash just once.
    class _CacheKey:
        __slots__ = ("hashvalue", "value")
    
        def __init__(self, value):
            self.hashvalue = hash(value)
            self.value = value
    
        def __hash__(self) -> int:
            return self.hashvalue
    
        def __eq__(self, other) -> bool:
            # Assume `type(other) is _Key`
            return self.value == other.value
    
    
    def cachemethod(fn: Callable[Concatenate[_Self, _Params], _Return]) -> Callable[Concatenate[_Self, _Params], _Return]:
        """Like `functools.cache`, except that it only holds a weak reference to its first argument.
    
        Note that `functools.cached_property` (which also uses a weak reference) can often be used for similar purposes.
        The differences are that (a) `cached_property` will be pickled whilst `cachemethod` will not, and (b) `cachemethod`
        can be used on functions with additional arguments, and (c) `cachemethod` requires brackets to call, helping to
        visually emphasise that computationl work may be being performed.
        """
        cache1 = _WeakIdDict()
        sig = inspect.signature(fn)
        parameters = list(sig.parameters.values())
        if len(parameters) == 0:
            raise ValueError("Cannot use `cachemethod` on a function without a `self` argument.")
        if parameters[0].kind not in {inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD}:
            raise ValueError("Cannot use `cachemethod` on a function without a positional argument.")
        parameters = parameters[1:]
        sig = sig.replace(parameters=parameters)
    
        @ft.wraps(fn)
        def fn_wrapped(self: _Self, *args: _Params.args, **kwargs: _Params.kwargs) -> _Return:
            # Standardise arguments to a single form to encourage cache hits.
            # Not binding `self` (and instead doing the signature-adjustment above) in order to avoid keeping a strong
            # reference to `self` via `argkey`.
            bound = sig.bind(*args, **kwargs)
            del args, kwargs
            argkey = _CacheKey((bound.args, tuple(bound.kwargs.items())))
            try:
                out = cache1[self][argkey]
            except KeyError:
                try:
                    cache2 = cache1[self]
                except KeyError:
                    cache2 = cache1[self] = {}
                out = cache2[argkey] = fn(self, *bound.args, **bound.kwargs)
            return out
    
        return fn_wrapped
  9. dsal3389 commented on Apr 29, 2025

    @dsal3389

    For those like me who landed on this issue, here's a cachemethod that solves this.

    Notably, it:

    • works on self types without a __dict__;
    • works on self types that are not hashable;
    • should be reasonably efficient as it avoids re-hashing the runtime arguments;
    • does not rely on descriptors, so it will cache even when called via SomeClass.some_cached_method(self, *args, **kwargs), not just self.some_cached_method(*args, **kwargs);
    • is static typing compatible.

    import functools as ft
    import inspect
    import weakref
    from collections.abc import Callable
    from typing import TYPE_CHECKING, Concatenate, ParamSpec, TypeVar

    _Self = TypeVar("_Self")
    _Params = ParamSpec("_Params")
    _Return = TypeVar("_Return")

    Like weakref.WeakKeyDictionary, but uses identity-based hashing and equality.

    class _WeakIdDict:
    slots = ("_dict",)

    def __init__(self):
        self._dict = {}
    
    def __getitem__(self, key):
        item, _ = self._dict[id(key)]
        return item
    
    def __setitem__(self, key, value):
        id_key = id(key)
        ref = weakref.ref(key, lambda _: self._dict.pop(id_key))
        self._dict[id_key] = (value, ref)
    

    Used to compute an object's hash just once.

    class _CacheKey:
    slots = ("hashvalue", "value")

    def __init__(self, value):
        self.hashvalue = hash(value)
        self.value = value
    
    def __hash__(self) -> int:
        return self.hashvalue
    
    def __eq__(self, other) -> bool:
        # Assume `type(other) is _Key`
        return self.value == other.value
    

    def cachemethod(fn: Callable[Concatenate[_Self, _Params], _Return]) -> Callable[Concatenate[_Self, _Params], _Return]:
    """Like functools.cache, except that it only holds a weak reference to its first argument.

    Note that `functools.cached_property` (which also uses a weak reference) can often be used for similar purposes.
    The differences are that (a) `cached_property` will be pickled whilst `cachemethod` will not, and (b) `cachemethod`
    can be used on functions with additional arguments, and (c) `cachemethod` requires brackets to call, helping to
    visually emphasise that computationl work may be being performed.
    """
    cache1 = _WeakIdDict()
    sig = inspect.signature(fn)
    parameters = list(sig.parameters.values())
    if len(parameters) == 0:
        raise ValueError("Cannot use `cachemethod` on a function without a `self` argument.")
    if parameters[0].kind not in {inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD}:
        raise ValueError("Cannot use `cachemethod` on a function without a positional argument.")
    parameters = parameters[1:]
    sig = sig.replace(parameters=parameters)
    
    @ft.wraps(fn)
    def fn_wrapped(self: _Self, *args: _Params.args, **kwargs: _Params.kwargs) -> _Return:
        # Standardise arguments to a single form to encourage cache hits.
        # Not binding `self` (and instead doing the signature-adjustment above) in order to avoid keeping a strong
        # reference to `self` via `argkey`.
        bound = sig.bind(*args, **kwargs)
        del args, kwargs
        argkey = _CacheKey((bound.args, tuple(bound.kwargs.items())))
        try:
            out = cache1[self][argkey]
        except KeyError:
            try:
                cache2 = cache1[self]
            except KeyError:
                cache2 = cache1[self] = {}
            out = cache2[argkey] = fn(self, *bound.args, **bound.kwargs)
        return out
    
    return fn_wrapped
    

    @patrick-kidger

    you might not keep references to the instance so it can be garbage collected, but you use the builtin id function which is just returning the data position in memory (if I remember correctly) and what the default __hash__ implementation also uses behind the scenes

    https://docs.python.org/3/library/functions.html#id

    Two objects with non-overlapping lifetimes may have the same id() value.

    this will cause false cache hits if 2 different objects are allocated on the same memory location

    this is also the reason the default cache methods keep reference to the data, so it won't be garbage collected thus it will be impossible for other instances to be allocated on the same memory location

    I wrote a quick an dirty POC for this

    class Foo:
        def __init__(self, name: str) -> None:
            self.name = name
    
    seen = set()
    i = 0
    
    while True:
        x = Foo("bob" + str(i))
        i += 1
    
        if id(x) in seen:
            print("duplicate", id(x))
            break
        seen.add(id(x))
        del x
        print(".")

    Image

    the only solution is to bind the cache to the running instance (as a private property or something) and make the cache not hold references to self, so when the instance is deleted the cache will also be deleted

  10. patrick-kidger commented on Apr 29, 2025

    @patrick-kidger

    Is it possible for an object to GC'd, then a new one to be reallocated, before weakref.ref(..., callback=...) runs?

    From the weakref.ref documentation, it states that the callback is evaluated when the wrapped object is about to be finalized. The wording of this suggests to me that the callback is evaluated immediately before the old object is GC'd, and therefore necessarily a new one cannot yet be allocated in the same memory location.

  11. dsal3389 commented on Apr 29, 2025

    @dsal3389

    totally missed that lambda callback while glancing over the code
    ref = weakref.ref(key, lambda _: self._dict.pop(id_key))

    this looks like a good approach, I wonder why the stdlib doesn't really use it like u suggested, maybe a good question / suggestion for the python discourse?

  12. ktbarrett commented on Jun 4, 2025

    @ktbarrett

    @dsal3389 I think caching the self object, weakref or not, is unnecessary. It's doing more than it needs to.

  13. dsal3389 commented on Jun 4, 2025

    @dsal3389

    @ktbarrett but if you don't cache the instance, how can you distinguish between cache hits that are coming from different instances? most of the time you don't want to mix them

    here is a very simple example (theory) how it can hit you (assume self is not used in the cache key)

    from functools import cache
    
    class Foo:
        def __init__(self, n: int) -> None:
            self._n = n
    
        @cache
        def foo(self) -> str:
            if self._n % 5 == 0:
                return "yes"
            else:
                return "no"
    
    x = Foo(5)
    y = Foo(7)
    
    print(x.foo())  # will print "yes"
    print(y.foo())  # will print also "yes", expected to be "no"
  14. ktbarrett commented on Jun 4, 2025

    @ktbarrett

    You place the cache on the instance rather than keeping it in the class. This is how cached_property works (well it's a "cache" of 1).

  15. patrick-kidger commented on Jun 4, 2025

    @patrick-kidger

    You place the cache on the instance rather than keeping it in the class. This is how cached_property works (well it's a "cache" of 1).

    This has the disadvantage of requiring that the instance be mutable in this way, which is the kind of sneakiness that can lead to unexpected behavior!

    For example, functools.cached_property will actually be pickled. This is unlike regular property - and is actually the reason that prompted me to write the cachemethod above.

  16. ktbarrett commented on Jun 4, 2025

    @ktbarrett

    Neither issues bothered the core devs when @cached_property was merged. stdlib isn't some random library that can change frequently and attempt to handle everyone's case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions