Repository navigation
Conversation
The examples table carried `Non versioned` alone until 137e69d swapped it for `Versioned`, in the same commit that added the cross-site restore steps. Nothing records why: the commit message says only "Complete PRA logic", the pull request body names the tests it was missing, its review discussed the zenkoversion, the locations and the scripts, and the ticket has no description at all. Nor does the code suggest a reason -- the restore helper that commit extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state, which is where a restore writes `x-amz-restore` onto a master with nothing beneath it. Both rows rather than the swap back: the version delete this scenario covers needs a bucket with versions to delete. Issue: ZENKO-5355
Three paths the PRA scenario never touched: it deletes nothing, and its only bucket is versioned, so an object is never overwritten in place and never deleted without a version to remove. None of the deletions D/R has never replicated, nor the metadata a non-versioned overwrite has to carry, is visible to CI today. Removing a version and overwriting in place are steps of the nominal scenario, which already holds the state they need. Removing an unversioned object gets a scenario of its own: a failing step abandons the rest of its scenario, and this is the one whose outcome on a released deployment is not yet known. @pra is assigned at most one pickle at a time, so scenarios cannot fight over the two clusters. Delete markers stay uncovered, as ZENKO-4886 records: a marker carries no placement, so the location the source selects on cannot match it. A suspended bucket gets a scenario of its own too. A put there writes a null version, which the delete addresses as a version even though the master may be the only document holding it, so what removes the object is the teardown of the placeholder left on that master -- a write the source makes in no other case. Issue: ZENKO-5355
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
benzekrimaha
approved these changes
Oct 13, 2026
DarkIsDude
approved these changes
Oct 13, 2026
Not for merge. Exercises ZKOP-625 end to end: the sink's Kafka Connect connector subscribes by name to the collections the source replicates, rather than matching the source database's topics with a pattern, so a bucket created after the D/R install reaches the sink without waiting out a consumer metadata refresh. zenko-operator a5d19711, pinned as the bare commit its image is tagged with, since no 1.9 release carries it yet. It is the head of scality/zenko-operator#660. The @pra scenario over a versioned bucket fails on precisely that window: its bucket is created minutes after the install the scenario before it leaves running, so the objects reach the sink first and are dropped with NoSuchBucket, and nothing re-drives them. Issue: ZENKO-5355
| dashboard: zenko-operator/zenko-operator-dashboards | ||
| image: zenko-operator | ||
| tag: v1.9.0-preview.1 | ||
| tag: a5d19711af75da684c487980764ac8567ee16024 |
There was a problem hiding this comment.
zenko-operator now points to a commit SHA build instead of a released tag (it was v1.9.0-preview.1). deps.yaml should pin a published release tag. Cut a zenko-operator release (or preview) that includes this fix and pin that tag before merging.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same test commits as:
I had to change the PR twice due to branch name issues.
PR body unchanged
The PRA scenario deleted nothing, and its only bucket was versioned — so an object was never overwritten in place, and never deleted without a version to remove. This adds those paths and points the solution at the builds that make them pass.
Scenarios
PRA (nominal case)—VersionedPRA (nominal case)—Non versionedPRA (nominal case)—SuspendedPRA (null version in a versioned bucket)PRA (in-place overwrite)The
Non versionedrow was in the table until137e69dd91swapped it forVersioned, in the same commit that added the cross-site restore steps. Nothing records why — not the commit message, not the pull request, not the ticket — and the restore helper it extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state. Both rows run again.Deletion rides the Examples matrix rather than getting scenarios of its own: the steps are identical, and what differs is the bucket they run against. A versioned bucket removes a version; an unversioned one has no version to remove, so the delete takes away the object's only document; a suspended one creates a null delete marker, and the marker itself is not replicated — it carries the bucket's location constraint, which can never be cold. What removes the object there is the other write the delete makes: cloudserver removes the null version the marker replaces, and that removal carries the cold object's own metadata. Three different metadata events, one set of steps.
Two cases cannot be parameterised, so they run as their own scenarios rather than being repeated identically in every row. The in-place overwrite needs a non-versioned bucket, which no other row provides. The null version in a versioned bucket is about ordering: an object written before versioning is enabled stays a bare master with no version id, and stays one after the bucket becomes
Enabled— the state cloudserver describes as "a single null version [that] looks like a master key". Both scenarios install D/R first and write afterwards, so the write travels the change stream rather than the bootstrap dump.Deleting an unversioned object already passes on
development/2.16, but for the wrong reason — the delete flag lands on the master, the document a version-lessHeadObjectreads, so the object reads as absent while its document stays behind forever. That row is a regression guard, not a demonstration.A known race, not fixed here
Bucket definitions reach the sink through the Kafka Connect entity sync; object entries reach it through the mongo-processor. Nothing orders the two. An object entry that arrives before its bucket gets
NoSuchBucket, and the offset is committed regardless — so that object's metadata is dropped for good, and the scenario fails on an assertion no amount of waiting can satisfy.We have not tried to fix it here. The entity path this races is expected to be retired.
Issue: ZENKO-5355