Repository navigation
Conversation
Disaster recovery from S3 alone needs the backups read back into Kafka; nothing in mirror-v3 does that yet. First piece, in a new crate mirror-restore: the chain of one partition directory, from its names. An append-mode blob destination names objects by consumer position (`from` = previous `to` + 1, holes kept inside the objects), so a complete backup is a chain without gap or overlap from the offset it starts at: 0, or an explicit --chain-start when objects before it were removed on purpose (never silently: a chain that starts elsewhere is an error naming where it starts). Every name in the directory must be an object of the backup's format, as the sink's own chain validation requires. Objects may mix clear and encrypted blobs and several key ids (a key rotation, turning encryption on); every key id named must be in the keys-dir, checked before any object is read, with the missing ids listed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "is the backup complete" check for disaster recovery: every object of a validated chain is read (decrypted with the key its name carries) and checked against its name and the backup's source. An object holds the records of positions from..=to that the source still had, strictly increasing, the last one at `to` (the sink names an object after its last record); a record of another topic or partition means the directory is not the backup it is taken for. Each is an error naming the object. Positions without a record are the source's offset holes (compaction, transaction markers): counted per object, with the first one, so --offsets=preserve can refuse a backup that has any. The tests write the backup with the mirror's own blob sink, encrypted across a key rotation (k1, then k2), and read it back record for record (keys, tombstones, headers with null values, both timestamp types); also a wrong key, an object renamed to claim more than it holds, and another source's records. mirror_fs::blob::decode_blob takes the keyring instead of the destination's BlobEncryption: reading needs the keys, not the active key id that new blobs are written with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umbered The produce pass writes every record of the verified chain to a Sink, in order, with its key, value, headers and timestamp. In production the sink is the Kafka destination, so the core rule holds for restore as for the mirror: the target's high watermark is read before each produce and must equal the record's offset, the producer never retries (message.send.max.retries=0), and the broker must report that offset back. Around it, restore requires an empty target (high watermark 0) before the first record and a high watermark equal to the records produced after the last; anything else is an error. Two offset modes and no default, since the choice depends on what reads the topic: - preserve: every record at its original offset. a downstream index keys rows on the operations topic's offsets and resumes from MAX(offset), so that topic must come back with them. A Kafka topic starts at 0 and cannot be written with holes, so a backup that starts elsewhere or has a hole is refused before anything is produced, naming the first. - renumber: records at 0, 1, 2, ... in their original order, for compacted topics (user-states), where holes are expected and keys, values and order are what matter. An object that reads back with another record count than in the verify pass is an error (the backup changed between the passes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… API Restore reads a backup with one identity that lists and gets objects; it must not need, or hold, the writing identity. S3Store::read_only opens a store without one, and a write to it is an error rather than a write with the reading identity. partition_prefix (was build_prefix) is public so that restore finds a destination's partition directory where the sink writes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Disaster recovery from S3 alone: after losing the VM, a new cluster gets
the backed-up topics back from the bucket.
mirror-v3 restore --config <yaml> --mirror <name> [--destination <name>]
--offsets preserve|renumber [--chain-start N]
(--verify-only | --bootstrap-servers <servers> --topic <topic>)
The backup is named by the config the mirror runs with, so its
directory, format, keys-dir and reading identity (`credentials.read`,
which must be allowed to get objects as well as list them) are the
destination's own; filesystem destinations work the same way, which
makes restore testable without S3. A compaction-mode mirror is refused:
its objects are snapshots, not the topic's records. The S3 store is
opened read-only; restore never holds the writing identity.
Every run lists the chain, reads and checks every object, and prints a
summary to stdout (objects, offsets, records, holes). --verify-only
stops there: the DR "is the backup complete" check, which also fails
when the backup cannot be restored with the --offsets given (preserve
and a hole). Otherwise the records are produced to the target topic's
partition of the same number through the Kafka destination (high
watermark read before each produce, zero producer retries), the target
must be empty before and hold exactly what was produced after; any
failed assumption exits non-zero with the reason. The target topic must
exist: restore does not create it, since its configuration (compaction,
retention) is the operator's decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proves restore against real brokers, end to end through the mirror's own backup: - preserve: 40 records with keys, tombstones, headers (one with a null value) and timestamps, backed up as Parquet and restored into a new topic, read back identical at identical offsets; a second restore into the now non-empty topic is refused (TargetNotEmpty(40)); a target topic that does not exist is an error, and restore does not create it. - holes: a source written in four transactions has a commit marker at offsets 3, 7, 11 and 15, holes for every consumer. The backup keeps them inside its objects; preserve refuses it before producing (target high watermark still 0), renumber restores the 12 records in order at 0..11. The scenarios run on the Docker stack (kafka-native source, Redpanda target; Redpanda also sources the transactions, which a one-node broker must serve), and as #[ignore]d variants against a broker named by MIRROR_E2E_EXTERNAL_KAFKA, for hosts without Docker. Run here (no Docker) with the external variant against Apache Kafka 4.3.1 in KRaft mode, userspace: both pass. The Docker variants compile and are not run here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documents `mirror-v3 restore` for the operator: how the backup is named by the config that wrote it, the verify pass and its summary (and --verify-only as the scheduled "is the backup complete" check), the per-record high-watermark gate and what it costs (about 140 records/s against a local broker), and the two offset modes: - preserve, for offset-keyed readers (a downstream index on operations): refused for a backup that starts after 0 or has a hole; afterwards the backup mirror continues the same chain. - renumber, for compacted topics (user-states): what changes (offsets from the first hole on, so stored offsets of the old topic do not apply; the old chain ends past the new topic, so its backup needs a new chain). Also timestamps (produced as CreateTime), that restore neither creates the target topic nor resumes, and the two things DR needs besides the bucket: an identity that may get objects, not only list them, and every Parquet key the objects name, kept off the machine that is lost. TESTING.md: the e2e restore test's external-broker variant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The holes scenario asserted Apache Kafka's transaction marker offsets (one commit marker after each transaction: records at 0, 1, 2, 4, ...). On the Docker stack the transactions run on Redpanda, which also writes a control batch before each transaction (records at 1, 2, 3, 6, ..., the first hole at offset 0), so the test failed there while passing against the external Apache Kafka it was written on. The expected hole count and first hole are now derived from the offsets the source actually holds; the test still requires a marker between every transaction. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`cargo test -p mirror-e2e` stopped at the first failing test binary, so the binaries after it reported nothing: --no-fail-fast runs them all. README: a topic written in transactions on Redpanda has a hole at offset 0 (a control batch before each transaction), so its backup restores only with `renumber`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review: the sentence read as if a topic written in transactions on Apache Kafka could be restored with preserve. Every transaction marker takes an offset on either broker; Redpanda adds one at offset 0 even after a single transaction. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The disaster-recovery path had no automated test: restore was e2e tested from a filesystem backup only, and from S3 by hand. The new scenario runs on kafka-native and VersityGW: - the mirror backs a topic up to S3, encrypted with k1, and after a restart with k2 as the active key, so the chain mixes both; - a keyring without k2 is refused before any object is read; - restore reads the bucket through S3Store::read_only, with the destination's partition prefix, and the restored topic reads back identical to the source (offsets, keys, tombstones, headers, timestamps). S3MirrorSpec takes the destination's encryption. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of the S3 restore e2e: - The scenario was Docker only, so a host without Docker could not produce the restore evidence. An #[ignore]d variant runs it on the broker MIRROR_E2E_EXTERNAL_KAFKA names and the S3 endpoint MIRROR_E2E_EXTERNAL_S3 names (bucket and credentials overridable, defaulting to the Docker stack's), under a prefix of its own per run. - The wait for the backup ignored an invalid chain (shown only as "holds 0 of 40" after 60 s); it now fails at once naming the cause. - The keyring check is from the names alone; the comment says so. The external variant passes against a standalone Redpanda and VersityGW. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tested e2e with appliance disaster recovery