feat(storage): write pipeline robustness and edge-case handling for object_store backend #3166

Problem / Context

In #3165 (reviving #2257 for #2259), the foundational ObjectStoreStorage::S3 backend is established. To ensure production robustness and performance in the write pipeline, several edge cases and optimizations should be addressed as follow-up work:

Proposed Improvements

1. Zero-Copy Writes in ObjectStoreWriter:

   • FileWrite::write receives owned bytes::Bytes.
   • object_store::WriteMultipart provides a put(Bytes) method that buffers without slice copying, avoiding redundant extend_from_slice allocations.

2. S3 URL Boundary Validation:

   • Empty bucket strings (e.g. s3:///path/to/file or s3://) currently pass initial URL host parsing as "". These should explicitly return ErrorKind::DataInvalid.
   • Ensure paths without trailing slashes (s3://bucket) extract bucket and empty relative path cleanly.

3. Stream Handling & State Machine Guards:

   • Validate delete_stream on empty streams (futures::stream::empty()) and ensure immediate error propagation on invalid URLs.
   • Add unit test coverage for ObjectStoreWriter state guards (e.g., verifying close() and write() error handling after writer has been closed).

4. Backpressure & Memory Throttling:

   • For high-throughput streaming, evaluate WriteMultipart::wait_for_capacity to throttle producers and prevent unbounded 5MB chunk allocations in memory ahead of S3 network uploads.

Related

• Implement Storage using object_store #2258 (Umbrella: Implement Storage using object_store)
• Implement ObjectStoreStorage::S3 #2259 (Implement ObjectStoreStorage::S3)
• feat(storage): Implement ObjectStoreStorage::S3 (Supersedes #2257) #3165 (feat(storage): Implement ObjectStoreStorage::S3)

AI Disclosure

Drafted with AI assistance; reviewed and verified by contributor.