Repository navigation
Conversation
…queries Repeated batch queries whose tag filter selects exactly one device rebuilt the device and timeseries-index metadata on every call. query_table now resolves the requested field columns once through a bounded LRU and runs them as prepared queries: one column through query_prepared, several aligned columns through query_prepared_multi. Queries that select a TAG column, use row-by-row mode, or whose filter matches zero or several devices keep the previous path. - Add TsFileReader::prepare_table_series(), which resolves (table, column, tag filter) to a PreparedLocator through the existing device index and parses it once. - Let the prepared query paths take a TsBlock row capacity (block_rows) instead of always allocating PREPARED_BATCH_ROWS, so cached queries keep the caller's batch_size block boundaries. The existing C entry points keep their signatures; tsfile_reader_query_prepared_ex and tsfile_reader_query_prepared_multi_ex carry the new parameter. - Expose prepare_table_series() in Python and cache single-device field queries behind TSFILE_QUERY_CACHE_SIZE (default 1024, 0 disables). Only "not cacheable" error codes fall back to the regular path; I/O, short-read, corruption and OOM errors still reach the caller. close() releases the cached handles. Measured locally (323 devices x 20,000 points, 2048-point windows, 2,000 queries, warm cache): one column 229 us -> 54 us median; two columns 287 us -> 102 us median. Full Python suite: 620 passed. Closes apache#1081. Relates to apache#1080.
ColinLeeo
force-pushed
the
perf/python-query-table-prepared-cache
branch
from
November 10, 2026 08:35
285bd62 to
4267cb9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repeated batch queries whose tag filter selects exactly one device rebuilt the device and timeseries-index metadata on every call.
query_tablenow resolves the requested field columns once through a bounded LRU and executes them as prepared queries: one column throughquery_prepared, several aligned columns throughquery_prepared_multi. Queries that select a TAG column, use row-by-row mode, or whose filter matches zero or several devices keep the previous path.TsFileReader::prepare_table_series()resolves(table, column, tag filter)to aPreparedLocatorthrough the existing device index and parses it once.block_rows) instead of always allocatingPREPARED_BATCH_ROWS, so cached queries keep the caller'sbatch_sizeblock boundaries. The existing C entry points keep their signatures;tsfile_reader_query_prepared_exandtsfile_reader_query_prepared_multi_excarry the new parameter.prepare_table_series()and caches single-device field queries behindTSFILE_QUERY_CACHE_SIZE(default 1024, 0 disables).close()releases the cached handles.Measurements
323 devices x 20,000 points, 2,000 point queries per run, warm page cache, alternating cache on/off:
The means include one miss per device; steady state is roughly 50 us (one column) and 100 us (two columns) per query.
Scope
tests/test_reader_sources.pystill verifies that every injected read failure propagates.block_rowsalso removes the hard-coded 65,536-row block that prepared queries used when no row limit was given ([Perf][C++] Point queries rebuild index metadata on every call; prepared series queries are ~7x faster but need fixes #1080).Tests
python/tests/test_query_table_cache.py: 17 cases.Closes #1081. Relates to #1080.