[sduchesneau]

On Azure, WalkFrom and WalkFromTo listed every file under the prefix and skipped the ones before the starting point. With millions of merged-blocks files, that takes one call per 5000 files and runs past the 30s lookup timeout of the tier1 hub in streamingfast/bstream#92.

They now pass the starting point as startFrom to List Blobs (azblob already supports it, the REST docs still mark it as preview). The names are still checked against both bounds, so an endpoint that ignores startFrom returns the same files. The end bound stays on our side: Azure only takes one with the Arrow response format.

Tested

• A test against a fake endpoint checks the startFrom sent and the filtering of files below it.
• storetests pass against Azurite.
• Not tried on a real Azure account yet.

[UlysseCorbeil]

I would approve it but can we run it on a real azure account ? How hard/worth is it, since if real Azure handles startFrom differently, every Azure listing breaks

[sduchesneau]

I would approve it but can we run it on a real azure account ? How hard/worth is it, since if real Azure handles startFrom differently, every Azure listing breaks

It's been tested on Azurite github.com/Azure/Azurite which is Microsoft-provided emulator for their object storage.

docker compose up -d azurite
STORETESTS_AZ_EMULATOR_STORE_URL="az://devstoreaccount1.store-tests" \
AZURE_STORAGE_ENDPOINT="http://localhost:10000" \
AZURE_STORAGE_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" \
go test ./storetests/azure/ -count=1

(the storage key is the advertised "development key" for the Azurite setup)

And the feature has been out for a while (2026-01-12 added to the SDK)