[blink1073]

PYTHON-6207

Summary

Adds a Rust implementation of the BSON codec as a drop-in replacement for the C extension. On the perf host it is faster than C on all six benchmarks, from 1.02x on FullDecoding to 1.71x on encoding.

Motivation

The C extensions hold the repo's highest memory-safety and reviewability risk. This ticket is the first step of the rust-shared-core program (PYTHON-6162): prove the codec can move to Rust with byte-exact parity and no performance regression before any larger migration.

Changes

• Added a Rust implementation of the BSON codec under rust/bson, differential-tested against the C extension.
• Added a benchmark selector that switches the performance suite between the C and Rust codecs.
• Added an Evergreen task that builds both codecs on the perf host and publishes the comparison.

Testing

• Differential suite against the C extension: 124 passed.
• BSON micro-benchmarks: 6 passed per codec.
• Full comparison protocol run on the perf host: task results.

Throughput, megabytes per second, perf host (October 31 run):

Benchmark	C MB/s	Rust MB/s	Rust/C
FlatEncoding	508.3	870.7	1.71x
DeepEncoding	229.6	391.8	1.71x
FullEncoding	287.8	331.6	1.15x
DeepDecoding	188.7	267.0	1.42x
FlatDecoding	455.7	515.5	1.13x
FullDecoding	143.3	145.6	1.02x

Rust is at parity or better on every benchmark. The run also captured DeepDecoding profiles for both codecs as task artifacts.

[Copilot]

🟡 Changes recommended

Unresolved critical defects risk undefined behavior when decoding writable buffers and BSON length overflow from unchecked cumulative growth, alongside codec compatibility and CI regressions.

9 open issues

• Enforce the cumulative BSON size limit on writes · New
• Avoid zero-copy slices over mutable Python buffers · New
• Separate appended RUSTFLAGS with a space · New
• Make profiling rebuild failures best-effort · New
• Keep the shared performance host unchanged · New
• Allow None for a present DBRef $db value · New
• Encode mappings using iteration and key lookup · New
• Use concrete list and tuple lengths when encoding · New
• Include the final field-name byte in the scan bound · New

What changed in this PR

Adds an experimental Rust-backed BSON codec alongside PyMongo’s C extension, with differential tests and benchmark tooling to evaluate compatibility and throughput before broader migration.

Changes:

• Implements six BSON codec entry points using PyO3, including codec options and BSON-specific types.
• Adds differential tests, benchmark fixtures, and a BSON_IMPL selector.
• Adds Evergreen comparison runs, optional decoding profiles, and artifact publishing.

File	Description
tools/​compare_bson_perf.py	Formats C/Rust throughput comparisons.
test/​performance/​data/​extended_bson/​full_bson.json	Adds mixed-type benchmark data.
test/​performance/​data/​extended_bson/​flat_bson.json	Adds flat-document benchmark data.
test/​performance/​data/​extended_bson/​deep_bson.json	Adds nested-document benchmark data.
test/​performance/​conftest.py	Activates codec selection before benchmarks.
test/​performance/​bson_impl_select.py	Selects and rebinds codec entry points.
rust/​bson/​tests/​test_rbson.py	Adds differential compatibility tests.
rust/​bson/​src/​types.rs	Caches Python types used by the codec.
rust/​bson/​src/​options.rs	Parses codec options and type markers.
rust/​bson/​src/​lib.rs	Exposes Python entry points and buffer handling.
rust/​bson/​src/​errors.rs	Constructs BSON exceptions.
rust/​bson/​src/​encode.rs	Implements BSON encoding.
rust/​bson/​src/​decode.rs	Implements BSON decoding and field-name caching.
rust/​bson/​src/​datetime_conv.rs	Converts datetime fields and epoch milliseconds.
rust/​bson/​src/​buffer.rs	Provides encoding buffer operations.
rust/​bson/​pyproject.toml	Defines Python packaging with maturin.
rust/​bson/​Cargo.toml	Defines Rust dependencies and release settings.
rust/​bson/​Cargo.lock	Locks Rust dependencies.
rust/​bson/​build.rs	Enables Python-version compilation flags.
AGENTS.md	Documents the experimental codec workflow.
.pre-commit-config.yaml	Allows Rust terminology in spell checks.
.gitignore	Excludes build and comparison outputs.
.evergreen/​scripts/​test_generate_config.py	Tests artifact and profiling configuration.
.evergreen/​scripts/​generate_config.py	Defines comparison tasks, variants, and uploads.
.evergreen/​scripts/​generate_config_utils.py	Changes the shared performance host assignment.
.evergreen/​scripts/​bson_codec_compare.sh	Builds, benchmarks, compares, and profiles codecs.
.evergreen/​generated_configs/​variants.yml	Adds the comparison variant and updates hosts.
.evergreen/​generated_configs/​tasks.yml	Adds the generated comparison task.
.evergreen/​generated_configs/​functions.yml	Adds generated artifact-upload commands.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Raw-buffer initialization can execute Python that mutates memory borrowed as an immutable Rust slice, leaving a critical safety issue alongside unresolved codec-parity defects.

7 open issues

• Initialize decode threshold before borrowing the buffer · New
• Run the parity suite before benchmarking · New
• Detect dollar keys after decoding field names · New
• Match ObjectId’s fixed 12-byte BSON payload · New
• Match Decimal128’s fixed 16-byte BSON payload · New
• Propagate invalid error-handler conversion errors · New
• Mutate the buffer used by the decode test · New

9 resolved since last review

• Avoid zero-copy slices over mutable Python buffers
• Enforce the cumulative BSON size limit on writes
• Include the final field-name byte in the scan bound
• Use concrete list and tuple lengths when encoding
• Encode mappings using iteration and key lookup
• Allow None for a present DBRef $db value
• Keep the shared performance host unchanged
• Make profiling rebuild failures best-effort
• Separate appended RUSTFLAGS with a space

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Malformed UTF-8 field names can trigger a custom error handler that re-enters decoding while the key-cache mutex is held, deadlocking the process.

No open issues 🎉

7 previously missed issues

In code that hasn't changed since last review

Bypass UTF-8 cache for invalid input

rust/​bson/​src/​decode.rs:1028

The strict error handler can be replaced with codecs.register_error("strict", handler), and PyUnicode_DecodeUTF8 invokes that Python handler even when its errors argument is NULL. For a malformed short field name, a handler that calls BSON decoding again therefore re-enters decode_key_cached while its non-reentrant mutex is held, deadlocking the process. Bypass the cache for invalid UTF-8 before taking the lock; this also prevents caching handler-dependent replacements as though they were strict decoding results.

Mark Evergreen task as patch-only

.evergreen/​scripts/​generate_config.py:1211

Omitting batchtime does not make an Evergreen variant patch-only, and this task has no patch_only setting. The Rust build and four benchmark passes can therefore run on normal commits despite the stated patch-only intent. Set patch_only=True on this task and regenerate the Evergreen configuration.

Restrict DBRef shortcut to exact dicts

rust/​bson/​src/​decode.rs:312

The no-dollar-key shortcut is only valid for an exact dict. A supported custom document_class can add DBRef keys in its constructor or rewrite ref/id to $ref/$id in __setitem__. The C codec checks the resulting mapping and returns a DBRef, but this gate skips the hook and returns the custom mapping instead. Please restrict this shortcut to the exact-dict case and add a differential regression test for a mapping that rewrites keys.

Propagate top-level TypeDecoder exceptions

rust/​bson/​src/​decode.rs:620

A registered TypeDecoder that raises ValueError for a top-level scalar produces InvalidBSON with Rust, while C preserves ValueError, breaking callers that handle their decoder's exceptions. Match C's exception boundary: wrap parsing and construction failures before invoking the converter, and propagate its failure without another wrapper at that level.

Preserve mapping lookups for dict subclasses

rust/​bson/​src/​decode.rs:654

cast::<PyDict>() also accepts dict subclasses, so this fast path bypasses their __getitem__ overrides. The C DBRef hook uses mapping lookups even for those subclasses. For example, a custom document_class that normalizes the $ref value on lookup produces a different DBRef.collection with Rust because this code reads the unnormalized stored value. Please use the raw-storage fast path only for an exact dict and preserve mapping lookups for subclasses.

Propagate exceptions outside the Exception hierarchy

rust/​bson/​src/​decode.rs:1038

This wraps every PyErr, including KeyboardInterrupt and SystemExit, as InvalidBSON. For example, a registered TypeDecoder raising KeyboardInterrupt is turned into a data error instead of interrupting the operation. The C reference explicitly wraps only subclasses of Exception; preserve exceptions outside that hierarchy before performing the conversion.

Accept signed-byte buffers in BSON decoding

rust/​bson/​src/​lib.rs:97

PyBuffer::<u8>::get rejects signed-byte buffers such as memoryview(encoded).cast("b") and array.array("b", encoded), although the C codec accepts them, so valid BSON fails to decode. Use a shared format-independent buffer acquisition helper that preserves contiguity, item-size, ownership, release and snapshot checks. The same issue affects raw-document decoding at rust/bson/src/lib.rs:72 and raw-document encoding at rust/bson/src/encode.rs:711.

7 resolved since last review

• Initialize decode threshold before borrowing the buffer
• Mutate the buffer used by the decode test
• Propagate invalid error-handler conversion errors
• Match Decimal128’s fixed 16-byte BSON payload
• Match ObjectId’s fixed 12-byte BSON payload
• Detect dollar keys after decoding field names
• Run the parity suite before benchmarking

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The raw-byte shortcut misses decoded $ref and $id keys produced by a registered strict handler, causing Rust to return a dict where the C decoder returns a DBRef.

1 open issue

• Restrict DBRef shortcut to valid UTF-8 field names · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The callback-induced panic risk, benchmark-selection inconsistency, incomplete Rust clamping coverage, and large-document copying overhead require human assessment of their combined impact.

No open issues 🎉

1 resolved since last review

• Restrict DBRef shortcut to valid UTF-8 field names

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Rust-selected benchmarks can silently measure the C codec, while allocation failures during decoding can abort Python and dictionary mutation during encoding can trigger Rust panics.

6 open issues

• Route driver benchmarks through the selected BSON codec · New
• Preserve the decoder traceback when raising InvalidBSON · New
• Avoid dictionary mutation panics during BSON encoding · New
• Use fallible allocation for bytearray and memoryview snapshots · New
• Exercise Rust datetime clamping in the differential test · New
• Apply Rust codec selection in direct benchmark execution · New

_{Review effort: Balanced}