Repository navigation
Keep hub final blocks until they are merged - #92
Conversation
A merger more than keepFinalBlocks behind LIB left a range of blocks that neither the merged-blocks files nor the hub could serve, so streams reading through the files waited there for the merger.
A hub following the merged-blocks store no longer also applies a fixed depth below LIB, so there is one rule to reason about. The fixed depth remains for hubs without the store.
The lag counts from the first streamable block until the first merged-blocks file exists, and a hub booting further above the last file than it can hold searches for where merged blocks really end.
The hub used the process-wide default, so a hub reading files of another size looked for files that never exist and stopped following the merger.
Each live block the hub could not link searched the merged-blocks store again until the hub had a LIB. It now reuses the last result, and waits 5s after a failed search.
Below maxKeptFinalBlocks the search could miss files that do not start at the first streamable block, leaving the hub waiting on a file the merger never writes.
A hub waiting on missing one-block files now says which blocks it waits for. Merged-blocks lookups stop after 30s and on shutdown instead of hanging the hub.
A hub starting while the merger is behind looked up one file per call going down from LIB, and one per call going up when the merger caught up. Both are now one listing. The listing also shows a file written ahead of a missing one, which is no longer taken for the end of merged blocks.
One appended to a slice from concurrent downloads. The other started its consumer before pushing blocks, so which partial blocks got skipped depended on timing.
| } | ||
|
|
||
| ok = false | ||
| err = t.store.WalkFrom(ctx, "", mergedBlocksFilename(base), func(filename string) error { |
There was a problem hiding this comment.
On Azure, WalkFrom lists the whole store from the first file, so with millions of merged files this times out and locate never succeeds. A prefix would bound it
I checked and its not an issue for dstore local disk
There was a problem hiding this comment.
good catch: when streamingfast/dstore#63 is merged, I will bump dstore here and it will fix the issue. (azure did NOT have the walkFrom feature when this was implemented, it does now.)
There was a problem hiding this comment.
bumping dstore!
| // purgeLimit is a forkable.WithFinalBlocksPurgeLimit function. It keeps the blocks of the | ||
| // last merged-blocks file and everything above: a stream reading the files joins the hub | ||
| // on a block it received from a file, so the hub has to overlap the last one. | ||
| func (t *mergedBlocksTracker) purgeLimit(libNum, cutoff uint64) uint64 { |
There was a problem hiding this comment.
Non-blocking but if the merger is behind, a stream joining the hub gets every unmerged block at once and has 30s to get through them
So a slow consumer won't finish in time, it gets disconnected -> reconnects -> repeats
Should the blocks queued at join time not count toward the 30s?
There was a problem hiding this comment.
30s should be enough even with a few hundreds more blocks, and if the user doesn't consume fast enough, they will simply loop in the "retry" that will reevaluate the source from the cursor and keep going from there (without an actual disconnection) so:
- it is only a problem when an unusual situation (merger lags behind) happens
- it is only in the rare situation where the user will not be able to process/download those extra blocks within 30 seconds
- the impact is very limited and invisible to the user (creating a new source from last sent cursor) -- it will still eventually allow the user to catch up.
so no real problem here.
UlysseCorbeil
left a comment
UlysseCorbeil
left a comment
There was a problem hiding this comment.
Approved with minor interrogation points
A hub keeps a fixed
keepFinalBlocksbelow LIB. A merger further behind than that left a range of blocks that neither the merged-blocks files nor the hub could serve: a stream reading through the files reached the last one and waited there until the merger caught up.With
hub.WithMergedBlocksStore(store, bundleSize, maxKeptFinalBlocks), the merged-blocks store decides what the hub keeps, andkeepFinalBlocksis passed as 0:maxKeptFinalBlocksbelow LIB.FileExistscall on the next expected file, at most every 5 seconds and only once LIB passed the end of that file. When LIB is two files or more past it, as when the merger catches up, it uses one listing instead of one call per file.maxKeptFinalBlocksit drops the oldest blocks anyway, and it logs a warning every 30 seconds while merged blocks are more than three files behind.hub.WithMergedBlocksLagMetricreports the final blocks not found in a merged-blocks file, andForkableHub.MergedBlocksUpToreturns the first of them.bundleSizeis the size of the files in that store (0 meansbstream.DefaultMergedBlocksBundleSize), so a hub reading files of another size than the process default looks for the right files.Cases around missing files and failures:
maxKeptFinalBlocks: it starts at its maximum depth and does not look for files below it. The lag metric reads aboutmaxKeptFinalBlocksat first, and the hub picks up from the file at that depth once the merger writes it.A hub built without the option behaves as before (the relayer uses that).
go test -race ./hub/reports a data race inTestForkableHub_LinkLiveUsingOneBlocks_StopsAtLiveBlock, in that test's own store hook; it is there without this PR's last commit too.