diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 0cba21d82..7940d57b7 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation. - [GROUPING SETS, ROLLUP, and CUBE](/docs/query/sql/grouping-sets/) - Compute detail rows, subtotals, and a grand total in a single scan, with `GROUP BY` or `SAMPLE BY`. Covers composite syntax, `GROUPING()` and `GROUPING_ID()`, `FILL` support, parallel execution, when a `UNION ALL` of plain `GROUP BY` queries is faster, and the limitations, plus the [`cairo.sql.max.grouping.sets`](/docs/configuration/cairo-engine/#cairosqlmaxgroupingsets) setting - [Reports with subtotals and grand totals](/docs/cookbook/sql/advanced/subtotals-and-grand-totals/) - Cookbook recipe for a desk volume report with per-ECN subtotals and a grand total from `ROLLUP`, labelled with `GROUPING()`, plus totals in both directions with `CUBE` and hourly totals with `SAMPLE BY` - [adbcBridge](/docs/integrations/other/adbcbridge/) - New page for the third-party ADBC drivers that read QuestDB into Apache Arrow from C#, Java, Go, TypeScript, and C: a native QWP driver for QuestDB 10.0 and newer, and an ODBC bridge over PGWire for older servers +- [ALTER TABLE FORCE REINDEX PARQUET](/docs/query/sql/alter-table-force-reindex-parquet/) - Rebuild bitmap and posting indexes on Parquet partitions while the table stays online, available from QuestDB 10.2.0 and QuestDB Enterprise 4.2.0. Use it to repair Parquet partitions left with stale column metadata by QuestDB 9.3.4 through 10.0.1. Covers column and partition selection, cancellation, skipped partitions, automatic repair by `ADD INDEX` and out-of-order writes, and running it on QuestDB Enterprise replicas ### Reference @@ -24,6 +25,7 @@ This page tracks significant updates to the QuestDB documentation. ### Updated +- [REINDEX](/docs/query/sql/reindex/) - Explains that `LOCK EXCLUSIVE` is required and takes the table's writer lock, and that `REINDEX` rebuilds native partitions only: it skips Parquet partitions and rejects a Parquet partition named in `PARTITION` - [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#limit-and-window-function-restrictions) - Breaking change in QuestDB 10.2.0 and QuestDB Enterprise 4.2.0: the part of a materialized view's query that reads its base table (this includes a subquery diff --git a/documentation/concepts/deep-dive/posting-index.md b/documentation/concepts/deep-dive/posting-index.md index 5aff08872..c7544c258 100644 --- a/documentation/concepts/deep-dive/posting-index.md +++ b/documentation/concepts/deep-dive/posting-index.md @@ -517,7 +517,10 @@ file. Decompression is transparent to the query engine. unfiltered `SAMPLE BY`, and unfiltered `GROUP BY` — fall back to a regular page-frame scan. - `REINDEX` on WAL tables requires dropping and re-adding the index - (this applies to all index types, not just posting). + (this applies to all index types, not just posting). Parquet partitions + are the exception: + [`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/) + rebuilds their indexes online. - `ALTER TABLE … ADD INDEX TYPE POSTING INCLUDE (col, …)` can fail on very large partitions when at least one `INCLUDE` column is variable-width (`STRING`, `VARCHAR`, `BINARY`) and the seal phase diff --git a/documentation/concepts/parquet.md b/documentation/concepts/parquet.md index d2e638c91..b246f7284 100644 --- a/documentation/concepts/parquet.md +++ b/documentation/concepts/parquet.md @@ -242,6 +242,14 @@ Parquet-related settings. blocked. * Some parallel queries are not yet optimized for Parquet. +### Rebuilding indexes + +To rebuild the indexes of Parquet partitions without taking the table offline, +use +[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/), +available from QuestDB 10.2.0 and QuestDB Enterprise 4.2.0. Run it after upgrading from QuestDB 9.3.4 through 10.0.1, which could leave +stale column metadata on Parquet partitions that received out-of-order rows. + ## Export Query results and whole tables can be exported as external Parquet files. Unlike diff --git a/documentation/query/sql/alter-table-force-reindex-parquet.md b/documentation/query/sql/alter-table-force-reindex-parquet.md new file mode 100644 index 000000000..a1c9fb483 --- /dev/null +++ b/documentation/query/sql/alter-table-force-reindex-parquet.md @@ -0,0 +1,156 @@ +--- +title: ALTER TABLE FORCE REINDEX PARQUET +sidebar_label: FORCE REINDEX PARQUET +description: + Rebuild bitmap and posting indexes on Parquet partitions online, without an + exclusive lock, and repair partitions affected by stale column metadata. +--- + +Rebuilds the [indexes](/docs/concepts/deep-dive/indexes/) of +[Parquet partitions](/docs/concepts/parquet/) while the table stays online. +Queries keep running against the current partition data until the rebuild +commits, so you can repair indexes without the downtime that +[`REINDEX ... LOCK EXCLUSIVE`](/docs/query/sql/reindex/) requires. + +The main use is repairing tables written by QuestDB 9.3.4 through 10.0.1. +Those versions could leave stale column-top metadata on a Parquet partition +after merging out-of-order rows into it, leaving the partition's indexes out of +step with its data. `FORCE REINDEX PARQUET` corrects that metadata and rebuilds +the indexes from the Parquet file. It also rebuilds an index that is damaged +for any other reason, such as a missing index file. + +`FORCE REINDEX PARQUET` is available from QuestDB 10.2.0 and QuestDB +Enterprise 4.2.0. + +## Syntax + +```questdb-sql +ALTER TABLE tableName FORCE REINDEX PARQUET + [COLUMN columnName] + [PARTITION LIST partitionName [, partitionName ...]]; +``` + +- `COLUMN columnName` rebuilds only the index of `columnName`, which must be an + indexed column. Without it, every index of the table is rebuilt. +- `PARTITION LIST` rebuilds only the listed partitions. The partition name + must match the name of the partition's directory, as described in + [Partitions](/docs/concepts/partitions/). Without it, every Parquet partition + of the table is rebuilt. + +Both [bitmap](/docs/concepts/deep-dive/indexes/) and +[posting](/docs/concepts/deep-dive/posting-index/) indexes are rebuilt, +including the covering columns of a posting index. + +## Examples + +```questdb-sql title="Rebuild every index on every Parquet partition" +ALTER TABLE trades FORCE REINDEX PARQUET; +``` + +```questdb-sql title="Rebuild the index of one column" +ALTER TABLE trades FORCE REINDEX PARQUET COLUMN symbol; +``` + +```questdb-sql title="Rebuild one column's index on selected partitions" +ALTER TABLE trades FORCE REINDEX PARQUET COLUMN symbol + PARTITION LIST '2026-03-01', '2026-03-02'; +``` + +To find the partitions that are in Parquet format, filter +[`table_partitions()`](/docs/query/functions/meta/#table_partitions) on its +`isParquet` column: + +```questdb-sql title="List Parquet partitions" +SELECT name FROM table_partitions('trades') WHERE isParquet; +``` + +## How it works + +Each partition is rebuilt into a new partition directory version. The new +version shares the Parquet file with the old one through hard links and gets +freshly built index files of its own. Queries that started before the rebuild +keep reading the old version until they finish, and QuestDB removes the old +version once nothing references it. Retired index files are also reclaimed by +[`VACUUM TABLE`](/docs/query/sql/vacuum-table/). + +The statement commits each partition separately. If it stops partway, the +partitions it already rebuilt stay rebuilt, and running the statement again +finishes the rest. + +The statement has no timeout. A +[`CANCEL QUERY`](/docs/query/sql/cancel-query/) or a dropped client connection +stops it once the partition in progress is committed. + +The statement runs outside the +[write-ahead log](/docs/concepts/write-ahead-log/) and changes only the node it +runs on. It is not replicated. + +### Partitions that are skipped + +- **Native partitions** are never rebuilt. On a non-WAL table the active + partition stays in native format, so it is left out. Use + [`REINDEX TABLE ... LOCK EXCLUSIVE`](/docs/query/sql/reindex/) for native + partitions. +- **Cold partitions** that live only in + [object storage](/docs/concepts/cold-storage/) have no local files to + rebuild. They are skipped, and the statement fails if every selected + partition is cold. +- **Partitions attached from a soft link** are skipped, because a new + directory version would move them off their attached volume. + +## Automatic repair + +You do not need `FORCE REINDEX PARQUET` for affected partitions that are +rewritten for another reason. QuestDB repairs stale column metadata on its own +when: + +- [`ALTER TABLE ALTER COLUMN ADD INDEX`](/docs/query/sql/alter-table-alter-column-add-index/) + runs. It repairs every affected local Parquet partition in the same + transaction, so the new index is built over correct metadata. +- An out-of-order write lands in an affected Parquet partition. The partition + is rewritten with correct metadata. + +[Deduplication](/docs/concepts/deduplication/) on Parquet partitions reads key +values from the Parquet data rather than from the stale metadata, so it removes +duplicates correctly whether or not the partition has been repaired. + +## Permissions + +In QuestDB Enterprise, the statement requires the `REINDEX` permission on the +table, or on the column when `COLUMN` is given. See +[permissions](/docs/security/rbac/#permissions). + +## Replication + +In QuestDB Enterprise, the statement also runs on +[read-only replicas](/docs/high-availability/setup/). Its changes stay on the +local node and do not advance the replica's replicated transaction sequence. +Because nothing is replicated, run it on the primary and on every replica that +holds affected partitions. + +## Errors + +| Error | Cause | +| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| `'parquet' expected` | The `PARQUET` keyword is missing after `FORCE REINDEX`. | +| `Invalid column: columnName` | The column does not exist. | +| `column is not indexed` | The `COLUMN` clause names a column without an index. | +| `partition does not exist` | A partition in `PARTITION LIST` does not exist. | +| `partition is not parquet, use REINDEX TABLE ... LOCK EXCLUSIVE for native partitions` | A partition in `PARTITION LIST` is in native format. No partition is rebuilt. | +| `cannot reindex cold partitions, they have no local data` | Every selected partition is cold. | +| `table is not partitioned, use REINDEX TABLE ... LOCK EXCLUSIVE` | The table has no partitions, so it has no Parquet partitions. | + +The statement also fails, rather than waiting, when another operation holds the +table writer, for example while WAL transactions are being applied. Retry it +once the writer is free. + +## See also + +- [Parquet](/docs/concepts/parquet/) +- [Indexes](/docs/concepts/deep-dive/indexes/) +- [Posting index](/docs/concepts/deep-dive/posting-index/) +- [REINDEX](/docs/query/sql/reindex/), for native partitions +- [ALTER TABLE ALTER COLUMN ADD INDEX](/docs/query/sql/alter-table-alter-column-add-index/) +- [ALTER TABLE ALTER COLUMN DROP INDEX](/docs/query/sql/alter-table-alter-column-drop-index/) +- [`table_partitions()`](/docs/query/functions/meta/#table_partitions) +- [VACUUM TABLE](/docs/query/sql/vacuum-table/) diff --git a/documentation/query/sql/reindex.md b/documentation/query/sql/reindex.md index 970c35f6e..fe6864de7 100644 --- a/documentation/query/sql/reindex.md +++ b/documentation/query/sql/reindex.md @@ -4,13 +4,14 @@ sidebar_label: REINDEX description: REINDEX SQL keyword reference documentation. --- -Rebuilds one or more [index](/docs/concepts/deep-dive/indexes/) columns of the given table. -This operation is intended to be used after a hardware or software crash, when -the index data are corrupted and the table cannot be opened for writes. +Rebuilds one or more [index](/docs/concepts/deep-dive/indexes/) columns of the +given table from the column data. Use it after a hardware or software crash, +when the index data are corrupted and the table cannot be opened for writes. -The operation can only be performed when there is no other reader and writer -working on the table. During the operation, the table is locked and no read and -write should be performed on the selected table. +`REINDEX` works on native partitions only. To rebuild indexes on Parquet +partitions while the table stays online, use +[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/) +instead. ## Syntax @@ -23,16 +24,43 @@ REINDEX TABLE tableName ## Options -By default, `REINDEX` rebuilds all indexes in the selected table. The following -options can be used to narrow down the scope of the operation: +By default, `REINDEX` rebuilds every index of the table in every native +partition. The following options narrow down the scope of the operation: -- `COLUMN`: When defined, `REINDEX` rebuilds the index for the selected column. -- `PARTITION`: When defined, `REINDEX` rebuilds index files in the selected - partition only. The partition name must match the name of the directory for - the given partition. The naming convention is detailed in +- `COLUMN`: rebuilds the index of the selected column only. +- `PARTITION`: rebuilds index files in the selected partition only. The + partition name must match the name of the directory for the given + partition. The naming convention is detailed in [Partitions](/docs/concepts/partitions/). -## Example +## LOCK EXCLUSIVE + +`LOCK EXCLUSIVE` is required. It is not an optional clause, and the statement +fails with `LOCK EXCLUSIVE expected` without it. The keywords state that +`REINDEX` takes the table's exclusive writer lock for the whole operation, so +nothing else can write to the table while its indexes are rebuilt. + +If another process or an open table writer already holds the lock, `REINDEX` +fails with `cannot lock table` rather than waiting. Stop ingestion into the +table before running it. Readers are not locked out, but queries should not run +against the table during the operation, because they may read an index that is +only partly rebuilt. + +## Parquet partitions + +[Parquet partitions](/docs/concepts/parquet/) have no native column files for +`REINDEX` to rebuild an index from: + +- Without `PARTITION`, `REINDEX` skips every Parquet partition and leaves its + existing index files untouched. Only the native partitions are rebuilt. +- With `PARTITION` naming a Parquet partition, `REINDEX` fails with + `cannot reindex parquet partition`. + +Use +[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/) +for Parquet partitions. It needs no exclusive lock. + +## Examples Rebuilding all the indexes in the table `trades`: @@ -52,3 +80,9 @@ Rebuilding one partition (`2021-12-17`) of the index in the column ```questdb-sql title="Rebuilding an index" REINDEX TABLE trades COLUMN instruments PARTITION '2021-12-17' LOCK EXCLUSIVE; ``` + +## See also + +- [ALTER TABLE FORCE REINDEX PARQUET](/docs/query/sql/alter-table-force-reindex-parquet/) +- [Indexes](/docs/concepts/deep-dive/indexes/) +- [Partitions](/docs/concepts/partitions/) diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 03cc71f32..73a7f929c 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -355,6 +355,7 @@ module.exports = { "query/sql/alter-table-detach-partition", // DETACH PARTITION "query/sql/alter-table-drop-column", // DROP COLUMN "query/sql/alter-table-drop-partition", // DROP PARTITION + "query/sql/alter-table-force-reindex-parquet", // FORCE REINDEX PARQUET "query/sql/alter-table-rebase-wal", // REBASE WAL "query/sql/alter-table-rename-column", // RENAME COLUMN "query/sql/alter-table-resume-wal", // RESUME WAL