feat: raise BackendUnavailableError when the server cannot be reached (#260)

changelog.d/260.added.md

@@ -0,0 +1,8 @@
+**The Redis and Memcached backends raise `BackendUnavailableError` when their
+server cannot be reached.** A refused or reset connection, a timeout and a
+Memcached server out of rotation all raise it, chained to the client's own
+exception, so one `except BackendUnavailableError` handles "the cache is down"
+around `CacheManager`, `CacheLock` or a backend call whatever the backend. The
+error is also an instance of the client's exception type, so existing `except
+redis.exceptions.ConnectionError` or `except OSError` clauses keep catching it.
+`@cache` still fails open on it.

docs/BACKENDS.md

@@ -170,7 +170,8 @@ after about 0.25 s. The costs:
 - `socket_timeout` also bounds slow replies. Set both timeouts above your
   normal Redis latency, including for the largest entries you cache.
 - The settings apply to every command the backend sends, not only to `@cache`.
-  `CacheManager` and `CacheLock`, which raise backend errors instead of
+  `CacheManager` and `CacheLock`, which raise
+  [`BackendUnavailableError`](#backend-unavailable) instead of
   failing open, see those errors sooner.
 
 `RedisConfig` has no `retry` field, so pass it to the constructor.
@@ -232,8 +233,9 @@ to a worker thread, including the multi-step
 of the keys existed; before 0.3.8 it took one thread trip per key and returned
 how many keys it was given.
 
-When a server cannot be reached, every call that would go to it raises, and the
-server is tried again after one second. Before 0.3.8, calls in the second after a
+When a server cannot be reached, every call that would go to it raises
+[`BackendUnavailableError`](#backend-unavailable), and the server is tried again
+after one second. Before 0.3.8, calls in the second after a
 failure returned made-up results instead: `get()` a miss, `set()` nothing (the write
 was lost), `increment()` a fresh counter of 0. With several servers, a failed one is
 taken out of rotation at once and its keys go to the remaining servers until it
@@ -276,6 +278,39 @@ when the block ends, including when the block raises.
   `MemoryBackend` had `aclose()`, and the Redis and Memcached backends were
   closed through `backend.client`.
 
+## When the server is unavailable {#backend-unavailable}
+
+When the Redis or Memcached server cannot be reached, the backend raises
+`BackendUnavailableError`, a `CacheXError`. That covers a refused or reset
+connection, a timeout, and a Memcached server out of rotation ("All servers
+seem to be down"). The memory backend never raises it. So code that wants to
+handle "the cache is down" needs one `except`, whichever backend is set:
+
+```python
+from fastapi_cachex import AppCache, BackendUnavailableError
+
+
+@app.get("/report")
+async def report(cache: AppCache) -> dict[str, int]:
+    try:
+        return await cache.get_or_set("report", build_report, ttl=300)
+    except BackendUnavailableError:
+        return build_report()  # serve it uncached while the cache is down
+```
+
+The error is chained to the client's own exception (`__cause__`), and is also
+an instance of that exception's type: a refused Memcached connection is a
+`BackendUnavailableError` and a `ConnectionRefusedError`, a Redis one a
+`BackendUnavailableError` and a `redis.exceptions.ConnectionError`. An
+`except` clause naming the client's exception, written before 0.5.1, keeps
+catching it. Errors that are not about reaching the server are left as they
+are: an invalid argument, a value Memcached refuses as too large, a key that
+holds something else.
+
+`@cache` is not affected: it [fails open](HTTP_CACHING.md#when-the-backend-fails)
+on any backend error, this one included. `CacheManager`, `CacheLock` and direct
+backend calls let it propagate.
+
 ## Atomic backend primitives
 
 Every backend exposes atomic operations on top of `get`/`set`/`delete`, for

fastapi_cachex/__init__.py

@@ -16,6 +16,7 @@
 from .dependencies import get_app_cache as get_app_cache
 from .dependencies import get_cache_backend as get_cache_backend
 from .exceptions import BackendNotFoundError as BackendNotFoundError
+from .exceptions import BackendUnavailableError as BackendUnavailableError
 from .exceptions import CacheXError as CacheXError
 from .exceptions import InvalidTTLError as InvalidTTLError
 from .exceptions import LockTimeoutError as LockTimeoutError
@@ -55,6 +56,7 @@ def _read_version() -> str:
     "AppCache",
     "BackendNotFoundError",
     "BackendProxy",
+    "BackendUnavailableError",
     "CacheBackend",
     "CacheKey",
     "CacheKeyBuilder",

fastapi_cachex/backends/_unavailable.py

@@ -0,0 +1,100 @@
+"""Turn a network backend's transport failures into ``BackendUnavailableError``.
+
+The Redis and Memcached clients each raise their own exceptions when the
+server cannot be reached: ``redis.exceptions.ConnectionError`` and
+``TimeoutError``, socket errors, pymemcache's "All servers seem to be down".
+A caller that only wants to know "the cache is down" would have to know which
+backend is configured and import that client's exceptions (#260).
+
+The backends' methods are wrapped so that such an error is raised again as a
+``BackendUnavailableError`` chained to the original. The class raised is
+built per original type and inherits from both, so it is also the original
+type: an ``except redis.exceptions.ConnectionError`` or ``except OSError``
+written before keeps catching it, and ``except BackendUnavailableError`` now
+works on every backend.
+"""
+
+import functools
+from collections.abc import Callable
+from collections.abc import Coroutine
+from typing import Any
+from typing import ParamSpec
+from typing import TypeVar
+from typing import cast
+
+from fastapi_cachex.exceptions import BackendUnavailableError
+
+P = ParamSpec("P")
+R = TypeVar("R")
+
+_SUBCLASSES: dict[type[Exception], type[BackendUnavailableError]] = {}
+
+
+def _rebuild(error_type: type[Exception], args: tuple[Any, ...]) -> Exception:
+    """Unpickle a translated error: pickle cannot import a runtime class."""
+    return _subclass_for(error_type)(*args)
+
+
+def _reduce(self: BaseException) -> tuple[Any, ...]:
+    return _rebuild, (type(self).__bases__[1], self.args), self.__dict__ or None
+
+
+def _subclass_for(error_type: type[Exception]) -> type[BackendUnavailableError]:
+    """The ``BackendUnavailableError`` that is also an ``error_type``."""
+    subclass = _SUBCLASSES.get(error_type)
+    if subclass is None:
+        # Named like its base, so a traceback reads BackendUnavailableError;
+        # the chained original below it names the client's own exception.
+        subclass = cast(
+            "type[BackendUnavailableError]",
+            type(
+                "BackendUnavailableError",
+                (BackendUnavailableError, error_type),
+                {
+                    "__module__": BackendUnavailableError.__module__,
+                    "__qualname__": BackendUnavailableError.__qualname__,
+                    "__doc__": BackendUnavailableError.__doc__,
+                    "__reduce__": _reduce,
+                },
+            ),
+        )
+        _SUBCLASSES[error_type] = subclass
+    return subclass
+
+
+def as_unavailable(error: Exception) -> BackendUnavailableError:
+    """``error`` as a ``BackendUnavailableError`` that is also its own type."""
+    subclass = _subclass_for(type(error))
+    try:
+        translated = subclass(*error.args)
+    except Exception:  # noqa: BLE001 - a constructor that wants other arguments
+        translated = subclass(str(error))
+    translated.__dict__.update(error.__dict__)
+    return translated
+
+
+def translate_unavailable(
+    is_unavailable: Callable[[Exception], bool],
+) -> Callable[
+    [Callable[P, Coroutine[Any, Any, R]]], Callable[P, Coroutine[Any, Any, R]]
+]:
+    """A decorator raising the errors ``is_unavailable`` accepts as unavailable."""
+
+    def decorator(
+        func: Callable[P, Coroutine[Any, Any, R]],
+    ) -> Callable[P, Coroutine[Any, Any, R]]:
+        @functools.wraps(func)
+        async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
+            try:
+                return await func(*args, **kwargs)
+            except BackendUnavailableError:
+                # Already translated by a wrapped method this one called.
+                raise
+            except Exception as e:
+                if not is_unavailable(e):
+                    raise
+                raise as_unavailable(e) from e
+
+        return wrapper
+
+    return decorator

fastapi_cachex/backends/memcached.py

@@ -15,12 +15,35 @@
 from fastapi_cachex.exceptions import TTLValueError
 from fastapi_cachex.types import CacheEntry
 
+from ._unavailable import translate_unavailable
 from .base import BaseCacheBackend
 from .base import validate_delta
 from .base import validate_ttl
 
 logger = logging.getLogger(__name__)
 
+# What pymemcache's HashClient raises while every server is out of rotation.
+_ALL_SERVERS_DOWN = "All servers seem to be down"
+
+
+def _is_unavailable(error: Exception) -> bool:
+    """Whether ``error`` means no Memcached server could be reached.
+
+    A socket error (refused, reset, timed out), a connection the server
+    closed mid-reply, and pymemcache's "All servers seem to be down" while a
+    failed server is out of rotation. A ``MemcacheClientError`` (bad input,
+    an item over the size limit) is not about availability and is left as is.
+    """
+    from pymemcache.exceptions import MemcacheError
+    from pymemcache.exceptions import MemcacheUnexpectedCloseError
+
+    if isinstance(error, (OSError, MemcacheUnexpectedCloseError)):
+        return True
+    return type(error) is MemcacheError and _ALL_SERVERS_DOWN in str(error)
+
+
+_unavailable = translate_unavailable(_is_unavailable)
+
 # Default Memcached key prefix for fastapi-cachex
 DEFAULT_MEMCACHE_PREFIX = "fastapi_cachex:"
 
@@ -202,6 +225,7 @@ def _make_key(self, key: str) -> str:
         logger.debug("Memcached key hashed; key=%s digest=%s", key, digest)
         return f"{self.key_prefix}{digest}"
 
+    @_unavailable
     async def get(self, key: str) -> CacheEntry | None:
         """Get value from cache.
 
@@ -221,6 +245,7 @@ async def get(self, key: str) -> CacheEntry | None:
             logger.debug("Memcached HIT; key=%s", key)
         return value
 
+    @_unavailable
     async def set(
         self, key: str, value: CacheEntry, ttl: int | timedelta | None = None
     ) -> None:
@@ -237,6 +262,7 @@ async def set(
         )
         logger.debug("Memcached SET; key=%s ttl=%s", key, ttl)
 
+    @_unavailable
     async def get_and_delete(self, key: str) -> CacheEntry | None:
         """Atomically retrieve and remove a cached entry (see base class).
 
@@ -275,6 +301,7 @@ def _get_and_delete(self, key: str) -> CacheEntry | None:
         msg = f"Memcached get_and_delete exceeded {_CAS_MAX_RETRIES} attempts on key {key!r}"
         raise CacheXError(msg)
 
+    @_unavailable
     async def set_if_absent(
         self, key: str, value: CacheEntry, ttl: int | timedelta | None = None
     ) -> bool:
@@ -298,6 +325,7 @@ async def set_if_absent(
         )
         return bool(stored)
 
+    @_unavailable
     async def delete_if_equals(self, key: str, expected: CacheEntry) -> bool:
         """Atomically remove ``key`` while it holds ``expected`` (see base class).
 
@@ -323,6 +351,7 @@ def _delete_if_equals(self, key: str, expected: CacheEntry) -> bool:
         )
         return bool(deleted)
 
+    @_unavailable
     async def expire_if_equals(
         self, key: str, expected: CacheEntry, ttl: int | timedelta
     ) -> bool:
@@ -356,6 +385,7 @@ def _expire_if_equals(
         )
         return bool(updated)
 
+    @_unavailable
     async def set_if_equals(
         self,
         key: str,
@@ -451,6 +481,7 @@ def _increment(self, prefixed_key: str, delta: int, exptime: int) -> int | None:
             logger.debug("Memcached INCREMENT RETRY; key=%s", prefixed_key)
         return None
 
+    @_unavailable
     async def increment(
         self, key: str, delta: int = 1, ttl: int | timedelta | None = None
     ) -> int:
@@ -491,6 +522,7 @@ async def increment(
         logger.debug("Memcached INCREMENT; key=%s value=%s ttl=%s", key, value, ttl)
         return value
 
+    @_unavailable
     async def delete(self, key: str) -> bool:
         """Delete value from cache.
 
@@ -507,6 +539,7 @@ async def delete(self, key: str) -> bool:
         logger.debug("Memcached DELETE; key=%s removed=%s", key, removed)
         return removed
 
+    @_unavailable
     async def delete_many(self, keys: Iterable[str]) -> int:
         """Remove every key in ``keys`` in one worker call; returns how many existed.
 
@@ -531,6 +564,7 @@ def _delete_each(self, prefixed_keys: list[str]) -> int:
             bool(self.client.delete(key, noreply=False)) for key in prefixed_keys
         )
 
+    @_unavailable
     async def clear(self) -> None:
         """Clear all values from cache.
 
@@ -550,6 +584,7 @@ async def clear(self) -> None:
         await asyncio.to_thread(self.client.flush_all)
         logger.debug("Memcached CLEAR; flush_all issued")
 
+    @_unavailable
     async def clear_path(self, path: str, include_params: bool = False) -> int:
         """Delete the key that is exactly ``path``; warns on every call.
 
@@ -588,6 +623,7 @@ async def clear_path(self, path: str, include_params: bool = False) -> int:
         )
         return removed
 
+    @_unavailable
     async def clear_pattern(self, pattern: str) -> int:
         """Clear cached responses matching a pattern.
 
@@ -611,6 +647,7 @@ async def clear_pattern(self, pattern: str) -> int:
         logger.debug("Memcached CLEAR_PATTERN unsupported; pattern=%s", pattern)
         return 0
 
+    @_unavailable
     async def get_all_keys(self) -> list[str]:
         """Get all cache keys in the backend.
 
@@ -633,6 +670,7 @@ async def get_all_keys(self) -> list[str]:
         logger.debug("Memcached GET_ALL_KEYS unsupported; returning empty list")
         return []
 
+    @_unavailable
     async def get_cache_data(self) -> dict[str, tuple[CacheEntry, float | None]]:
         """Get all cache data with expiry information.