Skip to content

fix(aggregation): apply --deletion_timeout to Partitioned DML deletes - #794

Open
n-h-diaz wants to merge 3 commits into
datacommonsorg:masterfrom
n-h-diaz:fix-deletion-timeout
Open

n-h-diaz wants to merge 3 commits into
datacommonsorg:masterfrom
n-h-diaz:fix-deletion-timeout

Conversation

@n-h-diaz

@n-h-diaz n-h-diaz commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

#730 set out to raise the aggregation deletes' Partitioned DML deadline from 1h to 6h after the Place import delete timed out. Its first version overrode the default deadline on the Spanner client's ExecuteStreamingSql wrapper. In review that was swapped for passing timeout= to Database.execute_partitioned_dml(), but that method takes no timeout argument (it raises TypeError), and the timeout= change was never made. Since then --deletion_timeout is stored on AggregationDeleter and never used, so every delete stream still gets the generated client's 3600s default.

Changes

  • deleter.py: restore the override on the database's own ExecuteStreamingSql wrapper. google-cloud-spanner has no public setting for this deadline (checked up to the latest 3.71.0). If the library internals change, the deleter raises instead of silently going back to 1h.
  • deleter_timeout_test.py: runs the real execute_partitioned_dml() over faked gRPC stubs and checks the deadline sent (the configured value, 21600s by default), plus the failure path. It's a separate file so it doesn't conflict with feat(spanner): set PRIORITY_LOW on batch ingestion and aggregation deletes #785's changes to deleter_test.py.
  • run_test.sh: run the aggregation-helper unit tests in CI; until now they never ran on PRs.

Context: #785 moves these deletes to PRIORITY_LOW, which makes them slower on a busy instance, so the 1h cap is more likely to be affected. #786 raises the Java Partitioned DML timeout to 6h; this PR is the Python counterpart.

Testing: uv run --locked pytest in aggregation-helper: 81 passed. Merges with #785 without conflicts; combined tests pass.

datacommonsorg#730 meant to raise the Partitioned DML deadline from 1h to 6h, but the
override it added was dropped in review in favor of passing `timeout=` to
Database.execute_partitioned_dml(), which takes no such argument (it raises
TypeError). Since then --deletion_timeout has been stored but never used,
and every delete stream still gets the generated client's 3600s default.

google-cloud-spanner has no public setting for this (still true in the
latest 3.71.0), so restore the override on this database's own
ExecuteStreamingSql wrapper, and fail loudly if the library internals
change instead of silently falling back to 1h. deleter_timeout_test.py runs
the real execute_partitioned_dml() over faked gRPC stubs and checks the
deadline that is sent.
The Python CI step only ran the simple/ and ingestion-helper tests, so
aggregation-helper changes, including the deadline test above, were never
tested on PRs. e2e tests stay excluded via the pyproject addopts.
@codacy-production

codacy-production Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 6 minor

Alerts:
⚠ 6 issues (≤ 0 issues of at least minor severity)

Results:
6 new issues

Category Results
Documentation 6 minor

View in Codacy

🟢 Metrics 10 complexity

Metric Results
Complexity 10

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request configures the Spanner ExecuteStreamingSql deadline for Partitioned DML deletes by overriding the internal _timeout attribute of the google-cloud-spanner client, and adds comprehensive unit tests along with test script updates. The review feedback suggests wrapping the internal attribute traversal in a try-except block to ensure that any unexpected internal structural changes in the library fail loudly with a descriptive RuntimeError rather than raising unhandled AttributeErrors or KeyErrors.

Comment on lines +76 to +83
transport = database.spanner_api._transport
streaming_sql = transport._wrapped_methods[transport.execute_streaming_sql]
if not hasattr(streaming_sql, "_timeout"):
raise RuntimeError(
"google-cloud-spanner internals changed: cannot set the "
"ExecuteStreamingSql deadline used by Partitioned DML deletes."
)
streaming_sql._timeout = float(self.timeout)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation directly accesses internal attributes (_transport, _wrapped_methods, etc.) of the google-cloud-spanner library. If any of these intermediate attributes or keys are renamed or removed in a future library version, Python will raise an AttributeError or KeyError before reaching the hasattr check, bypassing the custom RuntimeError and producing a less descriptive error. Wrapping the entire attribute traversal in a try-except block ensures that any internal structural changes always fail loudly with the intended descriptive RuntimeError message.

Suggested change
transport = database.spanner_api._transport
streaming_sql = transport._wrapped_methods[transport.execute_streaming_sql]
if not hasattr(streaming_sql, "_timeout"):
raise RuntimeError(
"google-cloud-spanner internals changed: cannot set the "
"ExecuteStreamingSql deadline used by Partitioned DML deletes."
)
streaming_sql._timeout = float(self.timeout)
try:
transport = database.spanner_api._transport
streaming_sql = transport._wrapped_methods[transport.execute_streaming_sql]
if not hasattr(streaming_sql, "_timeout"):
raise AttributeError("Missing _timeout attribute")
streaming_sql._timeout = float(self.timeout)
except (AttributeError, KeyError) as e:
raise RuntimeError(
"google-cloud-spanner internals changed: cannot set the "
"ExecuteStreamingSql deadline used by Partitioned DML deletes."
) from e
References
  1. Do not catch and gracefully handle exceptions for critical operations (such as data deletion or cleanup) if the system/job is designed to fail loudly and halt execution upon failure. Raising a RuntimeError ensures the deletion operation fails loudly.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to configure the streaming timeout for Spanner ExecuteStreamingSql RPCs in the AggregationDeleter by overriding the internal _timeout attribute of the wrapped execute_streaming_sql method. It also adds a comprehensive suite of unit tests in deleter_timeout_test.py to verify this behavior and ensure robust error handling if the underlying library internals change. Additionally, the run_test.sh script is updated to execute these new tests. There are no review comments to address, and we have no further feedback to provide.

Review feedback: an AttributeError/KeyError from an earlier step of the
private traversal bypassed the descriptive RuntimeError. Wrap the whole
traversal, including the write (in case _timeout becomes read-only), and
chain the original error. The _timeout check is a read before the write,
since assigning to a renamed attribute would silently do nothing. Use the
public SpannerClient.transport instead of _transport, and name
google-api-core in the error since _timeout lives in its _GapicCallable.
The test now covers each internal the override relies on.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant