Repository navigation
restore modes: fast and follow - #8
Merged
Merged
Conversation
Restore produced one record at a time behind the mirror's gate: a high-watermark read before each produce and no retries. About 140 records/s, so a million records took two hours, and any transient broker error in those hours failed the restore, which then needs the topic deleted, created again and the restore run from the start. The gate protects resuming: after a restart, nothing is produced twice. Restore does not resume, so it does not need the gate: - mirror_kafka::RestoreProducer: the idempotent producer (retries without duplicating or reordering), acks=all, linger.ms=20, up to 10000 records in flight. Every delivery report must name the offset the record was sent for, so another writer on the partition, or a record lost or written twice, still ends the restore. - mirror_restore::RestoreTarget: what produce writes to, with the checks around it unchanged: holes refused before producing, an empty target before, its high watermark equal to the records after. - The verify pass hashes every object's bytes, and the produce pass requires the same bytes when it reads the object again (it compared record counts). A million records of about 100 bytes restore from VersityGW into Redpanda in about 5 s, the verify pass included (debug build), and read back identical to the source. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of the fast restore:
- The one-shot restore had produced through the Kafka destination,
which checks every key and value against the mirror's column types
(UTF-8, parseable JSON). RestoreProducer dropped that check. Parquet
validates at encode, but ndjson stores bytes, so for an ndjson backup
it was the only one. RestoreProducer::open takes the column types and
send checks them before queueing a record.
- A failed delivery is reported for the oldest record in flight, up to
10000 before the one being sent, so "producing offset {next}" named
the wrong one. The error now stands on its own offset.
- Records sent before a failure can still be stored, so the error says
to delete the topic and create it again (RestoreError::Produce),
rather than leaving the operator to look for another writer.
- enable.gapless.guarantee: a batch that fails while later ones succeed
is a fatal producer error at once.
- README: the gated two hours for a million records is extrapolated
("would take"); the redundant async-trait dev-dependency is gone.
A second writer producing into the target during a 1M-record restore
now ends it with: "target: sent for offset 115000, the broker stored it
at restored-preserve/0 offset 115001: another writer on the partition,
or a record lost or written twice. Records sent before this may still
be stored: delete the topic and create it again before restoring again".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of the fast restore: - An error after records were sent but not raised by the target (an object changed after the verify pass, a store failure re-reading one) was returned as is, without the instruction to delete and create the topic again, and with records still in flight. Any failure after the first record is sent now waits for the records in flight and is a RestoreError::Produce: "after sending N records: <cause>. Records sent before this may still be stored: delete the topic and create it again before restoring again". A failure before anything is sent leaves the target as it was and says nothing of deleting it. - mirror-restore linked librdkafka only for a forwarding impl of RestoreTarget. The trait is now in mirror-core beside Source and Sink, mirror-kafka implements it on RestoreProducer, and mirror-restore no longer depends on mirror-kafka. - README: time-based retention counts from the records' timestamps, so a target with a retention shorter than the backup's age deletes the restored records soon after the restore; set the retention first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The one-shot restore is for a backup that is all there is. A standby, or a topic moved between clusters through a bucket, needs a restore that keeps up with a backup its mirror is still writing, and that can be stopped and started again. mirror_restore::ChainSource reads a backup as a mirror_core::Source, so run_mirror into the Kafka destination brings the per-record gate, the destination-ahead check and the idle drift check: - Positions are target offsets: a record's own offset with preserve, its index in the chain with renumber. A target at high watermark h holds the chain's first h records in both modes, so resuming needs no state: seek(h). `mirror-v3 restore --follow` first requires the target's last record to be the backup's record there (key, value, headers, timestamp), so a topic holding anything else is refused. - After every object listed so far it lists after the last one, every --poll-interval-ms; new objects must continue the chain (a gap is an error naming the missing offsets). - Every object is read and checked as in the verify pass before its records are yielded; with preserve a hole ends the follower when it is reached. - An object read survives the run loop dropping poll_one on a heartbeat: a large object may take longer than one to read. - SourceError::Inconsistent: a backup the destination cannot take, which trying again cannot fix (an unreachable store stays a transport error). mirror_kafka::read_record_at reads the target's last record for the resume check. Tested with run_mirror over a filesystem backup (growing backup, empty backup waited for, resume mid-object in both modes, a hole, a target past the backup, a gap in new objects), e2e against kafka-native and Redpanda while the backup mirror writes (stopped, more records backed up, resumed at 50), and by hand through the CLI against VersityGW. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of restore --follow: - A filesystem backup directory that does not exist is refused, as by the one-shot restore: the mirror creates it when it opens, so a missing one is a wrong root or a mirror that never ran. A directory or S3 prefix with no objects yet is waited for, and every listing logs the location and how long it has waited, so a wrong prefix shows. - The resume check moved into mirror-restore (ChainSource::check_resume, same_record), with unit tests for a match, another record, a target past the backup (both offset modes) and a backup record without a timestamp; the e2e calls it instead of repeating the comparison. The README says that only the last record is compared. - SIGTERM: the follower installs the same handlers as `run`, and fails if the SIGTERM handler cannot be installed instead of falling back to SIGINT only. They are installed first, so a SIGTERM during startup stops the follower rather than killing it. - The wait for new objects keeps its deadline when run_mirror drops poll_one on a heartbeat. Before, with a poll interval at or above the heartbeat, the wait never ended, poll_one never returned None and the idle drift check never ran; a test with a 3 ms heartbeat and a 200 ms poll interval now ends with DestinationDrift when another writer adds a record (it timed out before the change). - README: with renumber, a follower reads every object once at start. - mirror_kafka::read_record_at reports read failures as KafkaError::Read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of restore --follow: - A store or broker that could not be reached ended the follower with exit 1, as a gap or a hole does, though nothing distinguished the two and nothing retried. The follower is now opened again in the process after such a failure, with run's backoff (1 s doubling to 60 s), and resumes at the target's high watermark; anything else ends it. is_transient also knows RestoreError::Store and KafkaError::Read. - Startup (listing, counting, the resume check) ignored SIGTERM until it finished. It now races the shutdown signal, and the runtime is shut down in the background on exit, so a broker query still blocking in the pool does not hold the exit for its timeout (SIGTERM during a start against an unreachable broker: exit 0 in 19 ms, was 8 s). - --poll-interval-ms 0 listed the backup and queried the broker in a tight loop: it must be at least 100. - With renumber, high_watermark and seek read and decoded every object before the first record. high_watermark now gives an upper bound from the object names (positions, holes included), and seek counts only up to the target's position, with past-the-end checked exactly there. The object a start resumes in is read once (record_at, counting and the first fill share it). - read_record_at had no auto.offset.reset=error: a target whose last record retention had removed jumped to the end and timed out after 10 s with "no record". It is now KafkaError::Gone, "no longer in the topic (retention or compaction removed it)", and not retried. - same_record compared timestamps on a target that stamps LogAppendTime, so such a target could never be resumed; timestamps are compared only when the target kept the produced CreateTime. Tested: unit tests for the read counts, a renumber target past the backup's records and LogAppendTime; through the CLI against VersityGW and Redpanda, VersityGW stopped for 8 s mid-follow (logged, opened again, resumed at 1000, caught up to 1200), a trimmed target refused with the Gone message, SIGTERM during startup, the poll interval floor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Followup on #7