Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions documentation/changelog.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ This page tracks significant updates to the QuestDB documentation.
- [GROUPING SETS, ROLLUP, and CUBE](/docs/query/sql/grouping-sets/) - Compute detail rows, subtotals, and a grand total in a single scan, with `GROUP BY` or `SAMPLE BY`. Covers composite syntax, `GROUPING()` and `GROUPING_ID()`, `FILL` support, parallel execution, when a `UNION ALL` of plain `GROUP BY` queries is faster, and the limitations, plus the [`cairo.sql.max.grouping.sets`](/docs/configuration/cairo-engine/#cairosqlmaxgroupingsets) setting
- [Reports with subtotals and grand totals](/docs/cookbook/sql/advanced/subtotals-and-grand-totals/) - Cookbook recipe for a desk volume report with per-ECN subtotals and a grand total from `ROLLUP`, labelled with `GROUPING()`, plus totals in both directions with `CUBE` and hourly totals with `SAMPLE BY`
- [adbcBridge](/docs/integrations/other/adbcbridge/) - New page for the third-party ADBC drivers that read QuestDB into Apache Arrow from C#, Java, Go, TypeScript, and C: a native QWP driver for QuestDB 10.0 and newer, and an ODBC bridge over PGWire for older servers
- [ALTER TABLE FORCE REINDEX PARQUET](/docs/query/sql/alter-table-force-reindex-parquet/) - Rebuild bitmap and posting indexes on Parquet partitions while the table stays online, available from QuestDB 10.2.0 and QuestDB Enterprise 4.2.0. Use it to repair Parquet partitions left with stale column metadata by QuestDB 9.3.4 through 10.0.1. Covers column and partition selection, cancellation, skipped partitions, automatic repair by `ADD INDEX` and out-of-order writes, and running it on QuestDB Enterprise replicas

### Reference

Expand All @@ -24,6 +25,7 @@ This page tracks significant updates to the QuestDB documentation.

### Updated

- [REINDEX](/docs/query/sql/reindex/) - Explains that `LOCK EXCLUSIVE` is required and takes the table's writer lock, and that `REINDEX` rebuilds native partitions only: it skips Parquet partitions and rejects a Parquet partition named in `PARTITION`
- [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/#limit-and-window-function-restrictions) -
Breaking change in QuestDB 10.2.0 and QuestDB Enterprise 4.2.0: the part of a
materialized view's query that reads its base table (this includes a subquery
Expand Down
5 changes: 4 additions & 1 deletion documentation/concepts/deep-dive/posting-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -517,7 +517,10 @@ file. Decompression is transparent to the query engine.
unfiltered `SAMPLE BY`, and unfiltered `GROUP BY` — fall back to a
regular page-frame scan.
- `REINDEX` on WAL tables requires dropping and re-adding the index
(this applies to all index types, not just posting).
(this applies to all index types, not just posting). Parquet partitions
are the exception:
[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/)
rebuilds their indexes online.
- `ALTER TABLE … ADD INDEX TYPE POSTING INCLUDE (col, …)` can fail on
very large partitions when at least one `INCLUDE` column is
variable-width (`STRING`, `VARCHAR`, `BINARY`) and the seal phase
Expand Down
8 changes: 8 additions & 0 deletions documentation/concepts/parquet.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,14 @@ Parquet-related settings.
blocked.
* Some parallel queries are not yet optimized for Parquet.

### Rebuilding indexes

To rebuild the indexes of Parquet partitions without taking the table offline,
use
[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/),
available from QuestDB 10.2.0 and QuestDB Enterprise 4.2.0. Run it after upgrading from QuestDB 9.3.4 through 10.0.1, which could leave
stale column metadata on Parquet partitions that received out-of-order rows.

## Export

Query results and whole tables can be exported as external Parquet files. Unlike
Expand Down
156 changes: 156 additions & 0 deletions documentation/query/sql/alter-table-force-reindex-parquet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
---
title: ALTER TABLE FORCE REINDEX PARQUET
sidebar_label: FORCE REINDEX PARQUET
description:
Rebuild bitmap and posting indexes on Parquet partitions online, without an
exclusive lock, and repair partitions affected by stale column metadata.
---

Rebuilds the [indexes](/docs/concepts/deep-dive/indexes/) of
[Parquet partitions](/docs/concepts/parquet/) while the table stays online.
Queries keep running against the current partition data until the rebuild
commits, so you can repair indexes without the downtime that
[`REINDEX ... LOCK EXCLUSIVE`](/docs/query/sql/reindex/) requires.

The main use is repairing tables written by QuestDB 9.3.4 through 10.0.1.
Those versions could leave stale column-top metadata on a Parquet partition
after merging out-of-order rows into it, leaving the partition's indexes out of
step with its data. `FORCE REINDEX PARQUET` corrects that metadata and rebuilds
the indexes from the Parquet file. It also rebuilds an index that is damaged
for any other reason, such as a missing index file.

`FORCE REINDEX PARQUET` is available from QuestDB 10.2.0 and QuestDB
Enterprise 4.2.0.

## Syntax

```questdb-sql
ALTER TABLE tableName FORCE REINDEX PARQUET
[COLUMN columnName]
[PARTITION LIST partitionName [, partitionName ...]];
```

- `COLUMN columnName` rebuilds only the index of `columnName`, which must be an
indexed column. Without it, every index of the table is rebuilt.
- `PARTITION LIST` rebuilds only the listed partitions. The partition name
must match the name of the partition's directory, as described in
[Partitions](/docs/concepts/partitions/). Without it, every Parquet partition
of the table is rebuilt.

Both [bitmap](/docs/concepts/deep-dive/indexes/) and
[posting](/docs/concepts/deep-dive/posting-index/) indexes are rebuilt,
including the covering columns of a posting index.

## Examples

```questdb-sql title="Rebuild every index on every Parquet partition"
ALTER TABLE trades FORCE REINDEX PARQUET;
```

```questdb-sql title="Rebuild the index of one column"
ALTER TABLE trades FORCE REINDEX PARQUET COLUMN symbol;
```

```questdb-sql title="Rebuild one column's index on selected partitions"
ALTER TABLE trades FORCE REINDEX PARQUET COLUMN symbol
PARTITION LIST '2026-03-01', '2026-03-02';
```

To find the partitions that are in Parquet format, filter
[`table_partitions()`](/docs/query/functions/meta/#table_partitions) on its
`isParquet` column:

```questdb-sql title="List Parquet partitions"
SELECT name FROM table_partitions('trades') WHERE isParquet;
```

## How it works

Each partition is rebuilt into a new partition directory version. The new
version shares the Parquet file with the old one through hard links and gets
freshly built index files of its own. Queries that started before the rebuild
keep reading the old version until they finish, and QuestDB removes the old
version once nothing references it. Retired index files are also reclaimed by
[`VACUUM TABLE`](/docs/query/sql/vacuum-table/).

The statement commits each partition separately. If it stops partway, the
partitions it already rebuilt stay rebuilt, and running the statement again
finishes the rest.

The statement has no timeout. A
[`CANCEL QUERY`](/docs/query/sql/cancel-query/) or a dropped client connection
stops it once the partition in progress is committed.

The statement runs outside the
[write-ahead log](/docs/concepts/write-ahead-log/) and changes only the node it
runs on. It is not replicated.

### Partitions that are skipped

- **Native partitions** are never rebuilt. On a non-WAL table the active
partition stays in native format, so it is left out. Use
[`REINDEX TABLE ... LOCK EXCLUSIVE`](/docs/query/sql/reindex/) for native
partitions.
- **Cold partitions** that live only in
[object storage](/docs/concepts/cold-storage/) have no local files to
rebuild. They are skipped, and the statement fails if every selected
partition is cold.
- **Partitions attached from a soft link** are skipped, because a new
directory version would move them off their attached volume.

## Automatic repair

You do not need `FORCE REINDEX PARQUET` for affected partitions that are
rewritten for another reason. QuestDB repairs stale column metadata on its own
when:

- [`ALTER TABLE ALTER COLUMN ADD INDEX`](/docs/query/sql/alter-table-alter-column-add-index/)
runs. It repairs every affected local Parquet partition in the same
transaction, so the new index is built over correct metadata.
- An out-of-order write lands in an affected Parquet partition. The partition
is rewritten with correct metadata.

[Deduplication](/docs/concepts/deduplication/) on Parquet partitions reads key
values from the Parquet data rather than from the stale metadata, so it removes
duplicates correctly whether or not the partition has been repaired.

## Permissions

In QuestDB Enterprise, the statement requires the `REINDEX` permission on the
table, or on the column when `COLUMN` is given. See
[permissions](/docs/security/rbac/#permissions).

## Replication

In QuestDB Enterprise, the statement also runs on
[read-only replicas](/docs/high-availability/setup/). Its changes stay on the
local node and do not advance the replica's replicated transaction sequence.
Because nothing is replicated, run it on the primary and on every replica that
holds affected partitions.

## Errors

| Error | Cause |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `'parquet' expected` | The `PARQUET` keyword is missing after `FORCE REINDEX`. |
| `Invalid column: columnName` | The column does not exist. |
| `column is not indexed` | The `COLUMN` clause names a column without an index. |
| `partition does not exist` | A partition in `PARTITION LIST` does not exist. |
| `partition is not parquet, use REINDEX TABLE ... LOCK EXCLUSIVE for native partitions` | A partition in `PARTITION LIST` is in native format. No partition is rebuilt. |
| `cannot reindex cold partitions, they have no local data` | Every selected partition is cold. |
| `table is not partitioned, use REINDEX TABLE ... LOCK EXCLUSIVE` | The table has no partitions, so it has no Parquet partitions. |

The statement also fails, rather than waiting, when another operation holds the
table writer, for example while WAL transactions are being applied. Retry it
once the writer is free.

## See also

- [Parquet](/docs/concepts/parquet/)
- [Indexes](/docs/concepts/deep-dive/indexes/)
- [Posting index](/docs/concepts/deep-dive/posting-index/)
- [REINDEX](/docs/query/sql/reindex/), for native partitions
- [ALTER TABLE ALTER COLUMN ADD INDEX](/docs/query/sql/alter-table-alter-column-add-index/)
- [ALTER TABLE ALTER COLUMN DROP INDEX](/docs/query/sql/alter-table-alter-column-drop-index/)
- [`table_partitions()`](/docs/query/functions/meta/#table_partitions)
- [VACUUM TABLE](/docs/query/sql/vacuum-table/)
60 changes: 47 additions & 13 deletions documentation/query/sql/reindex.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@ sidebar_label: REINDEX
description: REINDEX SQL keyword reference documentation.
---

Rebuilds one or more [index](/docs/concepts/deep-dive/indexes/) columns of the given table.
This operation is intended to be used after a hardware or software crash, when
the index data are corrupted and the table cannot be opened for writes.
Rebuilds one or more [index](/docs/concepts/deep-dive/indexes/) columns of the
given table from the column data. Use it after a hardware or software crash,
when the index data are corrupted and the table cannot be opened for writes.

The operation can only be performed when there is no other reader and writer
working on the table. During the operation, the table is locked and no read and
write should be performed on the selected table.
`REINDEX` works on native partitions only. To rebuild indexes on Parquet
partitions while the table stays online, use
[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/)
instead.

## Syntax

Expand All @@ -23,16 +24,43 @@ REINDEX TABLE tableName

## Options

By default, `REINDEX` rebuilds all indexes in the selected table. The following
options can be used to narrow down the scope of the operation:
By default, `REINDEX` rebuilds every index of the table in every native
partition. The following options narrow down the scope of the operation:

- `COLUMN`: When defined, `REINDEX` rebuilds the index for the selected column.
- `PARTITION`: When defined, `REINDEX` rebuilds index files in the selected
partition only. The partition name must match the name of the directory for
the given partition. The naming convention is detailed in
- `COLUMN`: rebuilds the index of the selected column only.
- `PARTITION`: rebuilds index files in the selected partition only. The
partition name must match the name of the directory for the given
partition. The naming convention is detailed in
[Partitions](/docs/concepts/partitions/).

## Example
## LOCK EXCLUSIVE

`LOCK EXCLUSIVE` is required. It is not an optional clause, and the statement
fails with `LOCK EXCLUSIVE expected` without it. The keywords state that
`REINDEX` takes the table's exclusive writer lock for the whole operation, so
nothing else can write to the table while its indexes are rebuilt.

If another process or an open table writer already holds the lock, `REINDEX`
fails with `cannot lock table` rather than waiting. Stop ingestion into the
table before running it. Readers are not locked out, but queries should not run
against the table during the operation, because they may read an index that is
only partly rebuilt.

## Parquet partitions

[Parquet partitions](/docs/concepts/parquet/) have no native column files for
`REINDEX` to rebuild an index from:

- Without `PARTITION`, `REINDEX` skips every Parquet partition and leaves its
existing index files untouched. Only the native partitions are rebuilt.
- With `PARTITION` naming a Parquet partition, `REINDEX` fails with
`cannot reindex parquet partition`.

Use
[`ALTER TABLE FORCE REINDEX PARQUET`](/docs/query/sql/alter-table-force-reindex-parquet/)
for Parquet partitions. It needs no exclusive lock.

## Examples

Rebuilding all the indexes in the table `trades`:

Expand All @@ -52,3 +80,9 @@ Rebuilding one partition (`2021-12-17`) of the index in the column
```questdb-sql title="Rebuilding an index"
REINDEX TABLE trades COLUMN instruments PARTITION '2021-12-17' LOCK EXCLUSIVE;
```

## See also

- [ALTER TABLE FORCE REINDEX PARQUET](/docs/query/sql/alter-table-force-reindex-parquet/)
- [Indexes](/docs/concepts/deep-dive/indexes/)
- [Partitions](/docs/concepts/partitions/)
1 change: 1 addition & 0 deletions documentation/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,7 @@ module.exports = {
"query/sql/alter-table-detach-partition", // DETACH PARTITION
"query/sql/alter-table-drop-column", // DROP COLUMN
"query/sql/alter-table-drop-partition", // DROP PARTITION
"query/sql/alter-table-force-reindex-parquet", // FORCE REINDEX PARQUET
"query/sql/alter-table-rebase-wal", // REBASE WAL
"query/sql/alter-table-rename-column", // RENAME COLUMN
"query/sql/alter-table-resume-wal", // RESUME WAL
Expand Down
Loading