Skip to content

feat(xml): add eager deserialization for buffered payloads - #849

Open
stobrien89 wants to merge 2 commits into
developfrom
xml-eager-deserialization
Open

stobrien89 wants to merge 2 commits into
developfrom
xml-eager-deserialization

Conversation

@stobrien89

Copy link
Copy Markdown
Contributor

Description

Adds eager XML deserialization for buffered payloads while preserving streaming deserialization for reader inputs.

Changes

  • Adds XMLDeserializationMode with:
    • AUTO: eagerly parses bytes and incrementally parses BytesReader inputs.
    • EAGER: always materializes the complete XML tree.
    • STREAMING: preserves the existing pull-parser implementation.
  • Adds an eager tree-backed deserializer supporting structures, attributes, lists, maps, flattened collections, wrappers, timestamps, and unknown members.
  • Adds xml_deserialization_mode to generated Rest XML, AWS Query, and EC2 Query client configurations.
  • Reuses the configured XML codec when deserializing modeled AWS Query errors.
  • Preserves custom protocol codec settings unless explicitly overridden.
  • Updates generated client snapshots, documentation, and changelog metadata.

Performance

Benchmarked on an x86 m7i instance using Python 3.12.14, with 5,000 warmup and 10,000 measured iterations.

XML-decoding cases improved by 28.1–31.4% at p50.

Case Streaming p50 Eager p50 Change
restXml_CopyObjectOutput_M 70.24 µs 48.57 µs 30.9% faster
awsQuery_GetMetricDataResponse_S 150.23 µs 106.04 µs 29.4% faster
awsQuery_GetMetricDataResponse_M 431.90 µs 296.30 µs 31.4% faster
awsQuery_GetMetricDataResponse_L 4,156.95 µs 2,988.69 µs 28.1% faster

Cases that do not decode an XML document through the payload codec showed benchmark noise:

Case Streaming p50 Eager p50 Change
restXml_CopyObjectOutput_Baseline 3.11 µs 3.27 µs 5.3% slower
restXml_GetObject_S 14.55 µs 14.86 µs 2.1% slower
restXml_GetObject_M 14.39 µs 14.86 µs 3.3% slower
restXml_GetObject_L 14.65 µs 14.91 µs 1.7% slower

Compatibility

  • Existing XMLCodec construction remains supported.
  • AUTO preserves incremental parsing for reader inputs.
  • STREAMING retains the previous deserialization path.
  • Existing exception types and XML behavior are preserved.
  • No schema or generated model changes are required.

Validation

  • 2,504 Python tests passed.
  • All generated protocol test suites passed.
  • Full Java build passed.
  • Ruff, Pyright, Spotless, and git diff --check passed.

/**
* The core smithy-xml python package.
*/
public static final PythonDependency SMITHY_XML = new PythonDependency(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Service clients that rely on a query-based protocol already get a dependency on smithy_aws_core[xml]. That package has an xml dependency group which defines what smithy_xml version it is compatible with.

xml = [
"smithy-xml~=0.2.0"
]

Since you have a feature changelog entry being added, our release infra will bump smithy_xml to v0.3.0. Since its a minor bump, that gets propagated all the way through our dependency chain. So:

  • smithy_xml v0.2.0 --> v0.3.0
  • smithy_aws_core v0.12.0 --> v0.13.0
    • xml dependency group updates: smithy_xml v0.2.0 --> v0.3.0
  • aws-sdk-<query-service> v0.x.0 --> v0.x+1.0
    • smithy_aws_core[xml] v0.12.0 --> v0.13.0

In summary, we don't need this and version management is changelog driven and handled at release time

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@stobrien89
stobrien89 force-pushed the xml-eager-deserialization branch from 676fb80 to 6dc92f8 Compare October 13, 2026 21:21
from smithy_core.shapes import ShapeID
from smithy_core.types import TimestampFormat

from ._private.deserializers import XMLShapeDeserializer as _XMLShapeDeserializer

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing is using XMLShapeDeserializer or readers.py anymore right? Can we delete the streaming deserializer and move XMLValueDeserializer into deserializers.py? We might even just want to rename back to XMLShapeDeserializer for consistency.

reader = _XMLEventReader(
iterparse(source, events=("start", "end")) # noqa: S314
root = (
fromstring(source) # noqa: S314

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that parsing is eager, we're calling fromstring() on the same body two or three times per response. Could we parse once and pass the root element through? Didn't deep dive, but if that just affects the error case, not a big deal. But we definitely want the happy path not to do this multiple times if preventable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants