Skip to content

Extend PRA coverage to deletions, overwrites and non-versioned buckets - #2554

Open
delthas wants to merge 3 commits into
development/2.16from
improvement/ZENKO-5355/pra-delete-and-overwrite-tests
Open

delthas wants to merge 3 commits into
development/2.16from
improvement/ZENKO-5355/pra-delete-and-overwrite-tests

Conversation

@delthas

@delthas delthas commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Same test commits as:

I had to change the PR twice due to branch name issues.

PR body unchanged


The PRA scenario deleted nothing, and its only bucket was versioned — so an object was never overwritten in place, and never deleted without a version to remove. This adds those paths and points the solution at the builds that make them pass.

Scenarios

scenario before this PR
PRA (nominal case) — Versioned ✓ ✓, plus a version deletion
PRA (nominal case) — Non versioned — restored to the Examples table
PRA (nominal case) — Suspended — new row
PRA (null version in a versioned bucket) — new scenario
PRA (in-place overwrite) — new scenario

The Non versioned row was in the table until 137e69dd91 swapped it for Versioned, in the same commit that added the cross-site restore steps. Nothing records why — not the commit message, not the pull request, not the ticket — and the restore helper it extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state. Both rows run again.

Deletion rides the Examples matrix rather than getting scenarios of its own: the steps are identical, and what differs is the bucket they run against. A versioned bucket removes a version; an unversioned one has no version to remove, so the delete takes away the object's only document; a suspended one creates a null delete marker, and the marker itself is not replicated — it carries the bucket's location constraint, which can never be cold. What removes the object there is the other write the delete makes: cloudserver removes the null version the marker replaces, and that removal carries the cold object's own metadata. Three different metadata events, one set of steps.

Two cases cannot be parameterised, so they run as their own scenarios rather than being repeated identically in every row. The in-place overwrite needs a non-versioned bucket, which no other row provides. The null version in a versioned bucket is about ordering: an object written before versioning is enabled stays a bare master with no version id, and stays one after the bucket becomes Enabled — the state cloudserver describes as "a single null version [that] looks like a master key". Both scenarios install D/R first and write afterwards, so the write travels the change stream rather than the bootstrap dump.

Deleting an unversioned object already passes on development/2.16, but for the wrong reason — the delete flag lands on the master, the document a version-less HeadObject reads, so the object reads as absent while its document stays behind forever. That row is a regression guard, not a demonstration.

A known race, not fixed here

Bucket definitions reach the sink through the Kafka Connect entity sync; object entries reach it through the mongo-processor. Nothing orders the two. An object entry that arrives before its bucket gets NoSuchBucket, and the offset is committed regardless — so that object's metadata is dropped for good, and the scenario fails on an assertion no amount of waiting can satisfy.

We have not tried to fix it here. The entity path this races is expected to be retired.

Issue: ZENKO-5355

The examples table carried `Non versioned` alone until 137e69d swapped
it for `Versioned`, in the same commit that added the cross-site restore
steps. Nothing records why: the commit message says only "Complete PRA
logic", the pull request body names the tests it was missing, its review
discussed the zenkoversion, the locations and the scripts, and the ticket
has no description at all. Nor does the code suggest a reason -- the
restore helper that commit extracted keeps the version id optional, so it
runs against a bucket that has none.

Since the swap, nothing has restored or failed over an object whose bucket
holds no version to carry its state, which is where a restore writes
`x-amz-restore` onto a master with nothing beneath it.

Both rows rather than the swap back: the version delete this scenario
covers needs a bucket with versions to delete.

Issue: ZENKO-5355
Three paths the PRA scenario never touched: it deletes nothing, and its
only bucket is versioned, so an object is never overwritten in place and
never deleted without a version to remove. None of the deletions D/R has
never replicated, nor the metadata a non-versioned overwrite has to
carry, is visible to CI today.

Removing a version and overwriting in place are steps of the nominal
scenario, which already holds the state they need. Removing an
unversioned object gets a scenario of its own: a failing step abandons
the rest of its scenario, and this is the one whose outcome on a
released deployment is not yet known. @pra is assigned at most one
pickle at a time, so scenarios cannot fight over the two clusters.

Delete markers stay uncovered, as ZENKO-4886 records: a marker carries
no placement, so the location the source selects on cannot match it.

A suspended bucket gets a scenario of its own too. A put there writes a
null version, which the delete addresses as a version even though the
master may be the only document holding it, so what removes the object is
the teardown of the placeholder left on that master -- a write the source
makes in no other case.

Issue: ZENKO-5355
@bert-e

bert-e commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@delthas
delthas requested review from a team, DarkIsDude and benzekrimaha October 13, 2026 09:58
Not for merge. Exercises ZKOP-625 end to end: the sink's Kafka Connect
connector subscribes by name to the collections the source replicates,
rather than matching the source database's topics with a pattern, so a
bucket created after the D/R install reaches the sink without waiting out
a consumer metadata refresh.

zenko-operator a5d19711, pinned as the bare commit its image is tagged
with, since no 1.9 release carries it yet. It is the head of
scality/zenko-operator#660.

The @pra scenario over a versioned bucket fails on precisely that window:
its bucket is created minutes after the install the scenario before it
leaves running, so the objects reach the sink first and are dropped with
NoSuchBucket, and nothing re-drives them.

Issue: ZENKO-5355
Comment thread solution/deps.yaml
dashboard: zenko-operator/zenko-operator-dashboards
image: zenko-operator
tag: v1.9.0-preview.1
tag: a5d19711af75da684c487980764ac8567ee16024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zenko-operator now points to a commit SHA build instead of a released tag (it was v1.9.0-preview.1). deps.yaml should pin a published release tag. Cut a zenko-operator release (or preview) that includes this fix and pin that tag before merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants