diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 0cba21d82..9c96f5810 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -45,6 +45,9 @@ This page tracks significant updates to the QuestDB documentation. capped set. Likewise, move a window function into the query you run against the view. +- [SHOW CREATE DATABASE](/docs/query/sql/show/#show-create-database) - Documented QuestDB Enterprise 4.2.0 ACL output, principal/schema/grant ordering, the requirement for both `LIST USERS` and `USER DETAILS`, active storage policies, and replay limits. Documented the removal of `OWNED BY` from all `SHOW CREATE` output in 4.2.0, and updated the schema-copy recipe, RBAC, and governance guidance +- [Views](/docs/concepts/views/#administrative-operations-in-view-metadata) - Documented QuestDB Enterprise 4.2.0 restrictions on administrative operations in `PIVOT IN` subqueries, schema-only database dumps in materialized and live views, and [upgrade guidance](/docs/operations/upgrade/#upgrading-to-420) for existing scripts and views + ## October 2026 ### Releases diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 43e9c5702..c4f10b790 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -412,10 +412,12 @@ replaying it: `START FROM BEGINNING` to include all history still present in the base table. - If the view was created with a trailing `;`, the saved query keeps it inside `AS ( ... )` and replaying fails with `')' expected`. Remove that `;`. -- On QuestDB Enterprise, the saved `OWNED BY` names the user running - `SHOW CREATE`, and dropping the view removes its grants. Set `OWNED BY` to - the user or group that should receive the creator's permissions, and grant - the other permissions on the new view again. +- On QuestDB Enterprise, the saved DDL contains no `OWNED BY` or grants, and + dropping the view removes its grants. Save them first with + [`SHOW CREATE DATABASE INCLUDE (PERMISSIONS)`](/docs/query/sql/show/#enterprise-access-control) + or inspect them with `active_grants()`. On recreation, add `OWNED BY` if + another principal should receive the creator's permissions, and restore the + other grants on the new view. Dropping the invalid view removes its materialized rows, including rows whose source history is no longer retained by the base table. @@ -540,11 +542,12 @@ in [Base table lifecycle](#base-table-lifecycle). objects, such as `tables()`, follow the [materialized view rules](/docs/query/sql/create-mat-view/#catalogue-functions): in QuestDB Enterprise, only a creator with `SYSTEM ADMIN` can use them. -- **No administrative functions.** Enterprise functions that check the - caller's permissions, such as `permissions()` and `backup_instance_name()`, - are rejected, since a live view refreshes in the background. See - [query constraints](/docs/query/sql/create-live-view/#query-constraints) for - the full list. +- **No administrative functions or ACL dumps.** Enterprise functions that + check the caller's permissions, such as `permissions()` and + `backup_instance_name()`, and `SHOW CREATE DATABASE` with any ACL category + are rejected, since a live view refreshes in the background. Use + `INCLUDE (SCHEMA)` for schema metadata under the catalogue rules above. See + [query constraints](/docs/query/sql/create-live-view/#query-constraints). - **No TTL on the view.** Live-view disk growth is unbounded in this version. Base-table retention does not shrink the view, because `DROP PARTITION`, `TRUNCATE`, and TTL eviction on the base table leave the view's rows in place. diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index 7830b1786..731d251e9 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -836,6 +836,10 @@ Materialized view queries: refresh: `active_grants()`, `active_permissions()`, `backup_instance_name()`, `backup_status()`, `backups()`, `permissions()`, `reload_tick_calendars()`, and `reload_tls()` +- Must not use `SHOW CREATE DATABASE` with any ACL category when Enterprise + access control is enabled. The default includes ACL output; use + `INCLUDE (SCHEMA)` under the + [catalogue rules](/docs/query/sql/create-mat-view/#catalogue-functions) - Must use join conditions compatible with incremental refresh - When the base table uses [deduplication](/docs/concepts/deduplication/), non-aggregate columns must be a subset of the `DEDUP` keys diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index 0bc3d9dd0..c8da56f77 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -418,9 +418,10 @@ they can read through `views()` and `SHOW CREATE VIEW`, still names the table. The view's definer permissions are preserved even if the creator's account is later disabled or deleted. -Access control functions such as `active_grants()` and `active_permissions()` -are an exception: in a view, they check the permissions of the user who queries -the view, not those of its definer. See +Access control functions such as `active_grants()` and `active_permissions()`, +and [`SHOW CREATE DATABASE`](/docs/query/sql/show/#use-in-views) with any ACL +category, are exceptions: in a view's query body, they check the permissions of +the user who queries the view, not those of its definer. See [access control functions](/docs/query/functions/access-control/#limitations). ### Metadata functions in views (Enterprise) {#metadata-functions-in-views} @@ -464,6 +465,43 @@ table-name functions name, so it resolves them with the reader's own permissions until you run [`ALTER VIEW`](/docs/query/sql/alter-view/) or `CREATE OR REPLACE VIEW` on it. +### Administrative operations in view metadata + +A regular view can use administrative functions and administrative `SHOW` +statements in its query body, including ordinary subqueries. These operations +check the permissions of the user who queries the view, not those of its definer. + +A [`PIVOT IN` subquery](/docs/query/sql/pivot/#dynamic-in-list-from-subquery) is a +special case: its result values determine the output column names. QuestDB must +execute it at compile time to derive the view's columns, not just when a user +queries the view. This also happens during background recompilation and startup, +when there is no calling session. The subquery therefore cannot use administrative +operations that need a calling session or have side effects. + +This restriction extends to nested subqueries and other views read by the +`PIVOT IN` subquery. It does not apply to ordinary subqueries elsewhere in the +view's query body. + +With Enterprise access control enabled, `SHOW CREATE DATABASE` with any ACL +category counts as an administrative statement. Use +`SHOW CREATE DATABASE INCLUDE (SCHEMA)` if the subquery only needs schema +metadata. Catalogue queries that do not require a calling session remain +allowed. + +:::warning Existing views + +Starting with QuestDB Enterprise 4.2.0, creating, altering, or replacing a view +with a prohibited operation fails. Previously accepted views become invalid at +their next compilation after upgrading, including at startup. Inspect +`view_status` and `invalidation_reason` in +[`views()`](/docs/query/functions/meta/#views). Replace ACL dumps with +`INCLUDE (SCHEMA)` where appropriate, or remove the administrative operation +from the subquery, then update the definition with +[`ALTER VIEW`](/docs/query/sql/alter-view/) and run +[`COMPILE VIEW`](/docs/query/sql/compile-view/). + +::: + ### No column-level permissions on views Unlike tables, views do **not** support column-level permissions. You can only diff --git a/documentation/cookbook/operations/copy-schema-between-instances.md b/documentation/cookbook/operations/copy-schema-between-instances.md index bf2f63af5..a34b90895 100644 --- a/documentation/cookbook/operations/copy-schema-between-instances.md +++ b/documentation/cookbook/operations/copy-schema-between-instances.md @@ -7,7 +7,8 @@ description: --- Recreate the structure of one QuestDB instance on another: tables, views, -materialized views, and, on QuestDB Enterprise, the access control layer. +materialized views, and live views, with the option to copy access control on +QuestDB Enterprise. ## Problem @@ -15,16 +16,16 @@ You want to copy the schema from one QuestDB instance to another. ## Solution -[`SHOW CREATE DATABASE`](/docs/query/sql/show/#show-create-database) returns a -data-free dump of the whole database, with one round-trippable DDL statement per -row: +[`SHOW CREATE DATABASE INCLUDE (SCHEMA)`](/docs/query/sql/show/#show-create-database) +returns a data-free schema dump, with one replayable DDL statement per row. The +explicit category keeps this recipe schema-only on both editions: ```questdb-sql title="Dump the database schema" demo -SHOW CREATE DATABASE; +SHOW CREATE DATABASE INCLUDE (SCHEMA); ``` -Statements come back in dependency order, so replaying them top to bottom on an -empty instance recreates the database. +Statements come back in dependency order. Replay them top to bottom on an empty +instance to recreate the schema. On QuestDB Enterprise, the dump contains only the objects the caller [can see](/docs/security/rbac/#object-visibility). To copy the whole schema, @@ -38,7 +39,7 @@ result set as JSON, with the statements under `.dataset[][0]`: ```shell curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ - --data-urlencode "query=SHOW CREATE DATABASE" + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" ``` From there you can either save the statements to a file or apply them straight @@ -48,7 +49,7 @@ to the target instance. ```shell curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ - --data-urlencode "query=SHOW CREATE DATABASE" \ + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" \ | jq -r '.dataset[][0] | sub(";?$";";") + "\n"' > schema.sql ``` @@ -63,7 +64,7 @@ Send each statement to the target's `/execute` endpoint as you read it: ```shell curl -s -G "https://demo.questdb.io/api/v1/sql/execute" \ - --data-urlencode "query=SHOW CREATE DATABASE" \ + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" \ | jq --raw-output0 '.dataset[][0]' \ | xargs -0 -n1 sh -c \ 'curl -sG "http://localhost:9000/api/v1/sql/execute" \ @@ -104,14 +105,15 @@ export QDB_TOKEN="your-rest-api-token" curl -s -G "https://source:9000/api/v1/sql/execute" \ -H "Authorization: Bearer $QDB_TOKEN" \ - --data-urlencode "query=SHOW CREATE DATABASE" + --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" ``` -## Enterprise: schema without permissions +## Enterprise: schema and access control {#enterprise-schema-without-permissions} -In [QuestDB Enterprise](/enterprise/), the default dump also carries the access -control layer: users, groups, service accounts, memberships, and grants. That is -usually not what you want when seeding a development instance from production. +In [QuestDB Enterprise](/enterprise/) 4.2.0 and later, the default dump also +carries the access control layer: users, groups, service accounts, memberships, +and grants. That is usually not what you want when seeding a development +instance from production. Select what to copy with `INCLUDE`: @@ -121,19 +123,29 @@ curl -s -G "https://source:9000/api/v1/sql/execute" \ -H "Authorization: Bearer $QDB_TOKEN" \ --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (SCHEMA)" -# Permissions only, no tables or views +# Identities, memberships, and grants; no schema curl -s -G "https://source:9000/api/v1/sql/execute" \ -H "Authorization: Bearer $QDB_TOKEN" \ --data-urlencode "query=SHOW CREATE DATABASE INCLUDE (ACL)" ``` -The ACL categories each require the matching `LIST` or `USER DETAILS` -permission, while the schema categories need none, so a user with only `SELECT` -can still dump the structure of the objects they can see. - -Passwords and tokens are never dumped, so `CREATE USER` and -`CREATE SERVICE ACCOUNT` statements replay without credentials. Set those on the -target after the replay. +Every selection that includes an ACL category, including plain +`SHOW CREATE DATABASE`, requires both `LIST USERS` and `USER DETAILS`. Missing +either permission aborts the dump. `INCLUDE (SCHEMA)` requires neither, so a +user with only `SELECT` can still dump the structure of the objects they can see. +Active table storage policies are included in the schema. + +To copy access control too, use plain `SHOW CREATE DATABASE` in the commands +above. It creates principals first, schema objects next, then memberships, +service account assignments, and grants. Replay on an empty target as its +built-in admin to avoid adding another principal's creator grants. Schema-only +and per-object dumps contain neither grants nor `OWNED BY`. + +Before opening the target to users, set credentials and reapply memory limits, +resource groups, and resource group mappings: none are dumped. Only the external +alias shown by `SHOW GROUPS` is restored for each group. Avoid concurrent schema +and ACL changes during the export; the dump is best-effort, not a consistent +snapshot. See [Replay limits](/docs/query/sql/show/#replay-limits). ## Manual alternative: copy from the Web Console diff --git a/documentation/operations/upgrade.md b/documentation/operations/upgrade.md index 40b2cba64..256eb1bd7 100644 --- a/documentation/operations/upgrade.md +++ b/documentation/operations/upgrade.md @@ -193,6 +193,29 @@ check how these changes affect your users and clients: versions started with them, then rejected every OIDC login. See [Startup validation](/docs/security/oidc/#startup-validation). +### Upgrading to 4.2.0 + +QuestDB Enterprise 4.2.0 changes `SHOW CREATE` output and restricts +administrative operations in views. Check schema-export scripts and stored +views for the following before upgrading: + +- With access control enabled, plain `SHOW CREATE DATABASE` now requires both + `LIST USERS` and `USER DETAILS`. Schema-only scripts should use + `INCLUDE (SCHEMA)` instead of acquiring more permissions. See + [database dumps](/docs/query/sql/show/#enterprise-access-control). +- Per-object `SHOW CREATE` and database dumps no longer append `OWNED BY`. + Schema-only output does not restore grants. Use the ACL dump for permissions, + and review + [creator permissions on replay](/docs/query/sql/show/#enterprise-variant). +- With access control enabled, materialized and live views reject + `SHOW CREATE DATABASE` with any ACL category, including the default. Change + schema-metadata queries to `INCLUDE (SCHEMA)` under the + [catalogue rules](/docs/query/sql/create-mat-view/#catalogue-functions). +- A regular view's `PIVOT IN` subquery rejects administrative functions and + administrative `SHOW` statements. Existing definitions become invalid at + their next compilation, including startup. See + [view migration guidance](/docs/concepts/views/#administrative-operations-in-view-metadata). + ### QuestDB Enterprise BYOC For QuestDB Enterprise BYOC customers, QuestDB manages the upgrade process. The diff --git a/documentation/query/functions/access-control.md b/documentation/query/functions/access-control.md index 7be2bf1d6..451a67d3a 100644 --- a/documentation/query/functions/access-control.md +++ b/documentation/query/functions/access-control.md @@ -91,7 +91,9 @@ These apply to both `active_grants()` and `active_permissions()`: the permissions a user gains by assuming a service account. Use [`SHOW GROUPS userName`](/docs/query/sql/show/#show-groups) and [`SHOW SERVICE ACCOUNTS userName`](/docs/query/sql/show/#show-service-accounts) - to see them. + to see them. To export memberships, assignments, and direct grants as + replayable SQL, use + [`SHOW CREATE DATABASE INCLUDE (PERMISSIONS)`](/docs/query/sql/show/#enterprise-access-control). - Access through a view is not shown on the tables the view reads. A view reads them with the permissions of its [definer](/docs/concepts/views/#definer-security-model-enterprise), so a user @@ -105,8 +107,11 @@ These apply to both `active_grants()` and `active_permissions()`: refresh in the background without a session whose permissions could be checked. Creating such a view fails with an error such as `administrative function cannot be used in materialized view: active_grants`. - A regular view can call them. Unlike the tables it reads, the functions check - the permissions of the user who queries the view, not those of its definer. + A regular view can call them in its query body, but not in a `PIVOT IN` + subquery, including through nested queries or another view. Unlike the tables + it reads, the functions check the permissions of the user who queries the + view, not those of its definer. See + [administrative operations in view metadata](/docs/concepts/views/#administrative-operations-in-view-metadata). ### Examples diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index f779c2f60..6a35d2f7a 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -267,13 +267,18 @@ The view query is validated at creation time and must: `systimestamp()`, or `rnd_*()`. - Not use [catalogue functions](/docs/query/sql/create-mat-view/#catalogue-functions) - or `SHOW` statements, such as `tables()` in a `WHERE` sub-query, unless the - creator holds `SYSTEM ADMIN` (Enterprise). The same rules as for materialized - views apply. + or catalogue `SHOW` statements, such as schema-only + [`SHOW CREATE DATABASE`](/docs/query/sql/show/#use-in-views) in a `WHERE` + subquery, unless the creator holds `SYSTEM ADMIN` (Enterprise). The same rules + as for materialized views apply. - Not use Enterprise administrative functions, which check the permissions of the calling session: `active_grants()`, `active_permissions()`, `backup_instance_name()`, `backup_status()`, `backups()`, `permissions()`, `reload_tick_calendars()`, and `reload_tls()`. +- Not use `SHOW CREATE DATABASE` with any ACL category when access control is + enabled (QuestDB Enterprise 4.2.0 and later), even as an administrator. The + default statement includes ACL output; use `INCLUDE (SCHEMA)` under the + catalogue rules above. - Not read another live view. ## Examples @@ -476,6 +481,7 @@ the query. | `live view select cannot filter on the designated timestamp yet` | The `WHERE` clause filters the designated timestamp | | `non-deterministic function cannot be used in live view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | | `administrative function cannot be used in live view: ` | The query calls an Enterprise administrative function, such as `permissions()` or `reload_tls()` | +| `administrative statement cannot be used in live view: SHOW CREATE DATABASE` | The query includes an ACL dump (Enterprise); use `INCLUDE (SCHEMA)` under the [catalogue rules](/docs/query/sql/create-mat-view/#catalogue-functions) | ### Window errors diff --git a/documentation/query/sql/create-mat-view.md b/documentation/query/sql/create-mat-view.md index 36078a741..c8c87a791 100644 --- a/documentation/query/sql/create-mat-view.md +++ b/documentation/query/sql/create-mat-view.md @@ -454,7 +454,8 @@ Materialized view queries must: - Not read [catalogue functions](#catalogue-functions) through a regular view, and in QuestDB Enterprise call them only with `SYSTEM ADMIN` - Not use Enterprise administrative functions, such as `permissions()` or - `backups()` + `backups()`, or `SHOW CREATE DATABASE` with any ACL category. Use + `INCLUDE (SCHEMA)` for schema metadata, subject to the catalogue rules below See [query constraints](/docs/concepts/materialized-views/#query-constraints) for the full list. @@ -475,7 +476,16 @@ metadata: `files()`, and `glob()` - `SHOW TABLES`, `SHOW COLUMNS`, `SHOW PARTITIONS`, `SHOW CREATE TABLE`, `SHOW CREATE VIEW`, `SHOW CREATE MATERIALIZED VIEW`, `SHOW CREATE LIVE VIEW`, - and `SHOW CREATE DATABASE` + and schema-only + [`SHOW CREATE DATABASE`](/docs/query/sql/show/#use-in-views), such as + `SHOW CREATE DATABASE INCLUDE (SCHEMA)` + +In QuestDB Enterprise 4.2.0 and later, with access control enabled, a database +dump that includes any ACL category is administrative, not just a catalogue +query. Materialized and live views reject it for every principal, even one +holding `SYSTEM ADMIN` or `DATABASE ADMIN`. This includes plain +`SHOW CREATE DATABASE`, since its default is `INCLUDE ALL`. The check applies at +creation and refresh. A materialized view refreshes under a context that sees every object, and every reader of the view reads what the refresh stored. To keep a view from @@ -568,6 +578,7 @@ GRANT DROP MATERIALIZED VIEW ON trades_hourly TO user1; | `catalogue function from view cannot be used in materialized view: ` | A regular view the query reads calls a [catalogue function](#catalogue-functions). A `SHOW` statement reports `catalogue statement from view` | | `catalogue function cannot be used in materialized view without SYSTEM ADMIN: ` | The query calls a catalogue function and the creator lacks `SYSTEM ADMIN` (Enterprise). A `SHOW` statement reports `catalogue statement` | | `administrative function cannot be used in materialized view: ` | The query calls an Enterprise administrative function, such as `active_permissions()` or `backups()` | +| `administrative statement cannot be used in materialized view: SHOW CREATE DATABASE` | The query includes an ACL dump (Enterprise); use `INCLUDE (SCHEMA)` under the [catalogue rules](#catalogue-functions) | | `permission denied` | Missing required permission (Enterprise) | ## See also diff --git a/documentation/query/sql/create-view.md b/documentation/query/sql/create-view.md index 7d71619e6..a65d78aac 100644 --- a/documentation/query/sql/create-view.md +++ b/documentation/query/sql/create-view.md @@ -192,6 +192,17 @@ CREATE VIEW with_timestamp AS ( ) ``` +## Query constraints + +A `PIVOT IN` subquery in a view cannot use administrative functions or +administrative `SHOW` statements, including Enterprise `SHOW CREATE DATABASE` +with any ACL category. Use `INCLUDE (SCHEMA)` when only schema metadata is +needed. The restriction extends to nested queries and other views read by the +`PIVOT IN` subquery, not to ordinary subqueries elsewhere in the view's query +body. It is checked when creating, replacing, altering, or recompiling a view. See +[administrative operations in view metadata](/docs/concepts/views/#administrative-operations-in-view-metadata) +for existing-view migration guidance. + ## Errors | Error | Cause | @@ -200,6 +211,8 @@ CREATE VIEW with_timestamp AS ( | `table does not exist` | Referenced table doesn't exist | | `Invalid column` | Column in query doesn't exist | | `circular dependency detected` | View would create circular reference | +| `administrative function cannot be used in view PIVOT IN subquery: ` | The `PIVOT IN` subquery uses an administrative function, directly or through another query or view | +| `administrative statement cannot be used in view PIVOT IN subquery: SHOW CREATE DATABASE` | The `PIVOT IN` subquery includes an ACL dump (Enterprise); use `INCLUDE (SCHEMA)` if only schema is needed | | `variable is not overridable` | Attempted to override a non-`OVERRIDABLE` DECLARE variable | | `Access denied [AUDIT VIEW]` | `WITH AUDIT`, or a replacement of an audited view, without the `AUDIT VIEW` permission (Enterprise) | | `cannot audit an existing view, drop and re-create it WITH AUDIT` | `CREATE OR REPLACE VIEW ... WITH AUDIT` over a view that is not audited (Enterprise) | diff --git a/documentation/query/sql/pivot.md b/documentation/query/sql/pivot.md index ebbfcde83..b43d5ee09 100644 --- a/documentation/query/sql/pivot.md +++ b/documentation/query/sql/pivot.md @@ -266,6 +266,14 @@ For exploratory analysis, dynamic subqueries are convenient. For production quer use a constant list or store keys in a small dimension table for better performance. ::: +In a stored view, a `PIVOT IN` subquery cannot use administrative functions or +administrative `SHOW` statements, including Enterprise ACL dumps. This also +applies to nested queries and other views read by that subquery, not to ordinary +subqueries elsewhere in a regular view's query body. Schema-only +`SHOW CREATE DATABASE INCLUDE (SCHEMA)` remains allowed in a regular view. See +[administrative operations in view metadata](/docs/concepts/views/#administrative-operations-in-view-metadata) +for restrictions and migration guidance. + ### With CTEs ```questdb-sql demo title="Pivot with CTE" diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index bc1000717..e134ac146 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -39,8 +39,8 @@ SHOW { COLUMNS FROM tableName - `SHOW COLUMNS` returns all the columns and their metadata for the selected table. -- `SHOW CREATE DATABASE` returns DDL statements that recreate every object in - the database, one per row, ordered so dependencies come first. +- `SHOW CREATE DATABASE` returns statements to recreate the schema and, on + Enterprise, access control, one per row in replay order. - `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live view. - `SHOW CREATE MATERIALIZED VIEW` returns a DDL query that allows you to @@ -65,11 +65,13 @@ SHOW { COLUMNS FROM tableName - `SHOW USERS` lists all users with their enabled flag, memory limit and resource group (enterprise-only) -In QuestDB Enterprise, `SHOW TABLES` and `SHOW CREATE DATABASE` cover only the -objects you [can see](/docs/security/rbac/#object-visibility). The statements -that name an object, such as `SHOW COLUMNS` or `SHOW CREATE TABLE`, fail with -the missing-object error, for example `table does not exist [table=trades]`, -for an object hidden from you. +In QuestDB Enterprise, `SHOW TABLES` and the schema portion of +`SHOW CREATE DATABASE` cover only the objects you +[can see](/docs/security/rbac/#object-visibility). Dumping any access control +category also requires [administrative permissions](#enterprise-access-control). +Statements that name an object, such as `SHOW COLUMNS` or `SHOW CREATE TABLE`, +fail with the missing-object error, for example +`table does not exist [table=trades]`, for an object hidden from you. ## Examples @@ -96,10 +98,12 @@ a comma-separated string. ### SHOW CREATE DATABASE -`SHOW CREATE DATABASE` returns a logical, data-free dump of the whole database: -one round-trippable DDL statement per row for every user object, much like -`pg_dump --schema-only`. Replaying the statements from top to bottom on an empty -instance recreates the database. No table data and no credentials are included. +`SHOW CREATE DATABASE` returns a logical, data-free dump as replayable SQL: +one statement per row for tables, views, materialized views, and live views. +In QuestDB Enterprise 4.2.0 and later, the dump also includes users, groups, +service accounts, memberships, and grants by default. Use it to copy a database's +structure and access control to an empty instance. It is not a data backup; see +[Replay limits](#replay-limits). ```questdb-sql title="SHOW CREATE DATABASE syntax" SHOW CREATE DATABASE @@ -109,36 +113,29 @@ SHOW CREATE DATABASE An optional `INCLUDE` or `EXCLUDE` clause selects which object categories to dump. Each `category` is one of: -- **Schema objects**: `TABLES`, `VIEWS`, `MATERIALIZED_VIEWS`, `LIVE_VIEWS`. -- **Access control** (Enterprise only): `USERS`, `GROUPS`, `SERVICE_ACCOUNTS`, - `PERMISSIONS`. -- **Umbrellas**: `SCHEMA` (all schema objects), `ACL` (all access control - objects), `ALL` (`SCHEMA` plus `ACL`). +- **Schema objects**: `LIVE_VIEWS`, `MATERIALIZED_VIEWS`, `TABLES`, `VIEWS`. +- **Access control** (Enterprise only): `GROUPS`, `PERMISSIONS`, + `SERVICE_ACCOUNTS`, `USERS`. +- **Umbrellas**: `ACL` (all access control categories), `ALL` (`SCHEMA` plus + `ACL`), `SCHEMA` (all schema objects). `INCLUDE`/`EXCLUDE` accept either `ALL` or a parenthesised list, so `INCLUDE ALL`, `EXCLUDE (MATERIALIZED_VIEWS)`, and `INCLUDE (TABLES, VIEWS)` are -all valid. Called without a clause, the statement dumps the whole database: +all valid. With no clause, the statement defaults to `INCLUDE ALL`: -```questdb-sql title="Dump the database schema" demo +```questdb-sql title="Dump schema and access control" SHOW CREATE DATABASE; ``` +In QuestDB open source, `ALL` and `SCHEMA` produce the same output. On +Enterprise, select `SCHEMA` explicitly to omit access control: + +```questdb-sql title="Dump only the database schema" demo +SHOW CREATE DATABASE INCLUDE (SCHEMA); +``` + The result set has a single `ddl` column with one self-contained statement per -row. Run against a database holding the [demo](https://demo.questdb.io) tables -and materialized views, it returns one row per object: - -| ddl | -| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| CREATE TABLE 'market_data' ( timestamp TIMESTAMP, symbol SYMBOL, bids DOUBLE[][], asks DOUBLE[][], best_bid DOUBLE, best_ask DOUBLE ) timestamp(timestamp) PARTITION BY HOUR TTL 3 DAYS; | -| CREATE MATERIALIZED VIEW 'bbo_1s' WITH BASE 'market_data' REFRESH IMMEDIATE AS ( SELECT timestamp, symbol, last(bids[1][1]) AS bid, last(asks[1][1]) AS ask FROM market_data SAMPLE BY 1s ) PARTITION BY DAY; | -| CREATE MATERIALIZED VIEW 'bbo_1m' WITH BASE 'bbo_1s' REFRESH EVERY 1m DEFERRED START '2025-06-01T00:00:00.000000Z' AS ( SELECT timestamp, symbol, max(bid) AS bid, min(ask) AS ask FROM bbo_1s SAMPLE BY 1m ) PARTITION BY DAY; | -| CREATE MATERIALIZED VIEW 'bbo_1h' WITH BASE 'bbo_1m' REFRESH EVERY 10m DEFERRED START '2025-06-01T00:00:00.000000Z' AS ( SELECT timestamp, symbol, max(bid) AS bid, min(ask) AS ask FROM bbo_1m SAMPLE BY 1h ) PARTITION BY MONTH; | -| CREATE MATERIALIZED VIEW 'bbo_1d' WITH BASE 'bbo_1h' REFRESH EVERY 1h DEFERRED START '2025-06-01T00:00:00.000000Z' AS ( SELECT timestamp, symbol, max(bid) AS bid, min(ask) AS ask FROM bbo_1h SAMPLE BY 1d ) PARTITION BY YEAR; | -| ... | -| CREATE TABLE 'trips' ( cab_type SYMBOL, vendor_id SYMBOL, pickup_datetime TIMESTAMP, dropoff_datetime TIMESTAMP, rate_code_id SYMBOL, pickup_latitude DOUBLE, pickup_longitude DOUBLE, dropoff_latitude DOUBLE, dropoff_longitude DOUBLE, passenger_count INT, trip_distance DOUBLE, fare_amount DOUBLE, extra DOUBLE, mta_tax DOUBLE, tip_amount DOUBLE, tolls_amount DOUBLE, ehail_fee DOUBLE, improvement_surcharge DOUBLE, congestion_surcharge DOUBLE, total_amount DOUBLE, payment_type SYMBOL, trip_type SYMBOL, pickup_location_id INT, dropoff_location_id INT ) timestamp(pickup_datetime) PARTITION BY MONTH; | - -Each `ddl` value is stored with formatting characters, so pasting a row into a -text editor expands it to the indented form shown by +row. Each value preserves newlines and indentation, as in [`SHOW CREATE TABLE`](#show-create-table). For turning the result set into a `.sql` file or replaying it on another server, @@ -147,13 +144,21 @@ see #### Output order -Objects are emitted in dependency order: a materialized view or view is never -reported before the base table or base materialized view it reads from. Within -that constraint objects are ordered alphabetically. The demo chains several -materialized views, for example `market_data` then `bbo_1s`, `bbo_1m`, `bbo_1h`, -`bbo_1d`, and `fx_trades` then `fx_trades_ohlc_1m`, `fx_trades_ohlc_1d`. Each -view in a chain appears only after the object it depends on, so a top-to-bottom -replay always succeeds. +A full Enterprise dump has three parts: + +1. Users, groups, and service accounts, ordered by name across all three kinds. + A disabled user or service account has an `ALTER ... DISABLE` statement + immediately after its `CREATE` statement. +2. Schema objects in dependency order, with alphabetical ordering where + dependencies allow. A view follows the objects it reads, including other + views in a chain. +3. Group memberships, service account assignments, and direct grants, grouped + by principal name. Within each principal's grants, database scope precedes + table scope, which precedes column scope. + +Schema-only dumps contain just the second part. Preserve the returned order +when replaying: a broader grant can clear a narrower grant option, which a +later `WITH GRANT OPTION` statement restores. #### Filtering by category @@ -174,21 +179,18 @@ SHOW CREATE DATABASE INCLUDE (TABLES, MATERIALIZED_VIEWS); SHOW CREATE DATABASE EXCLUDE (MATERIALIZED_VIEWS); ``` -With no clause the statement defaults to `INCLUDE ALL`. In QuestDB open source -there is no access control layer, so `ALL` and `SCHEMA` produce the same output. -In [QuestDB Enterprise](/enterprise/) the default `ALL` also dumps the access -control block, so use `INCLUDE (SCHEMA)` when you want the structure only. - Filtering is applied per category, like `pg_dump -t`. Excluding a category that others depend on can leave dangling references, so a dump that omits a base -table does not replay cleanly for the views built on it. +table does not replay cleanly for the views built on it. Similarly, +`INCLUDE (PERMISSIONS)` omits principal creation: those users, groups, and +service accounts must already exist on the target. To select individual objects, such as the objects of a single tenant, wrap the -statement in parentheses and filter the `ddl` column with `WHERE`. Each row -starts with `CREATE`, so match the object name with a leading wildcard: +statement in parentheses and filter the `ddl` column with `WHERE`. Match names +wherever they appear in the statement: -```questdb-sql title="Objects whose DDL mentions fx_" demo -(SHOW CREATE DATABASE) WHERE ddl ILIKE '%fx_%'; +```questdb-sql title="Schema statements whose DDL mentions fx_" demo +(SHOW CREATE DATABASE INCLUDE (SCHEMA)) WHERE ddl ILIKE '%fx_%'; ``` The filtered rows keep dependency order. A pattern also matches views that read @@ -197,34 +199,112 @@ included before replaying it. #### Enterprise: access control -In [QuestDB Enterprise](/enterprise/), `SHOW CREATE DATABASE` also dumps the -access control layer after the schema objects, so a dump captures identities, -memberships, and permissions alongside the tables and views: - -- `CREATE USER`, `CREATE GROUP`, and `CREATE SERVICE ACCOUNT` for each entity. -- Memberships, as `ADD USER ... TO ...` and `ASSUME SERVICE ACCOUNT ... TO ...`. -- Grants, as `GRANT [ON ...] TO ... [WITH GRANT OPTION]`. - -QuestDB stores no owner for an object, so the `GRANT` statements are what -recreate its permissions. The `CREATE TABLE`, `CREATE VIEW`, and -`CREATE MATERIALIZED VIEW` statements in an Enterprise dump also carry an -`OWNED BY` clause, which names the principal that ran the dump, as in -[`SHOW CREATE TABLE`](#enterprise-variant). Replaying the dump gives that -principal all permissions on each of these objects, with the `GRANT` option, on -top of the dumped grants. Edit or remove the clause before replaying if that -principal should not hold them. - -Credentials are never dumped: the `CREATE USER` and `CREATE SERVICE ACCOUNT` -statements carry no password or token, so set these after replaying the dump. - -The Enterprise ACL categories are `USERS`, `GROUPS`, `SERVICE_ACCOUNTS`, and -`PERMISSIONS`, grouped by the `ACL` umbrella. Each requires the matching `LIST` -or `USER DETAILS` permission, while the schema categories need no access control -permission, so a user with only `SELECT` can still dump the structure of the -objects it can see. To dump the whole schema, run the statement as a principal -that sees every object, such as the built-in admin or a user holding -`DATABASE ADMIN`. When access control is disabled the command degrades to a -schema-only dump. +With access control enabled, selecting any ACL category requires both +`LIST USERS` and `USER DETAILS`, even `INCLUDE (USERS)` or +`INCLUDE (PERMISSIONS)` alone. Missing either permission aborts the whole dump; +it does not return a partial schema-only result. + +:::warning Breaking change + +Starting with QuestDB Enterprise 4.2.0, plain `SHOW CREATE DATABASE` includes +ACL output and requires both permissions. For callers that only need the visible +schema, change it to `SHOW CREATE DATABASE INCLUDE (SCHEMA)` rather than granting +administrative permissions. + +::: + +Schema-only categories require neither ACL permission, but still follow +[object visibility](/docs/security/rbac/#object-visibility). To dump the whole +user schema, run as the built-in admin or a principal holding `DATABASE ADMIN`. +These visibility rules do not filter the ACL output: an authorized ACL dump +lists all stored principals and their direct grants, including grants naming +objects hidden from the caller. When access control is disabled, the statement +returns only schema output, without the ACL permission checks. + +The `ACL` umbrella selects all four categories: + +| Category | Statements included | +| -------- | ------------------- | +| `GROUPS` | `CREATE GROUP`, including the external alias reported by `SHOW GROUPS` | +| `PERMISSIONS` | `ADD USER ... TO ...`, `GRANT ASSUME SERVICE ACCOUNT ... TO ...`, and database-, table-, and column-level `GRANT` statements | +| `SERVICE_ACCOUNTS` | `CREATE SERVICE ACCOUNT`, followed by `ALTER SERVICE ACCOUNT ... DISABLE` when disabled | +| `USERS` | `CREATE USER`, followed by `ALTER USER ... DISABLE` when disabled | + +Grants preserve `WITH GRANT OPTION`, including grant options held at a narrower +scope than the permission itself. Only direct grants are emitted. Permissions +and service account assignments inherited from a group are restored through +group membership, not copied into direct user grants. + +For example, on an otherwise empty instance, run as the built-in admin: + +```questdb-sql title="Create a group-based entitlement to export" +CREATE GROUP fx_readers; +CREATE USER fx_analyst; +ADD USER fx_analyst TO fx_readers; +GRANT SELECT ON ALL TABLES TO fx_readers; + +SHOW CREATE DATABASE INCLUDE (ACL); +``` + +| ddl | +| --- | +| CREATE USER 'fx_analyst'; | +| CREATE GROUP 'fx_readers'; | +| ADD USER 'fx_analyst' TO 'fx_readers'; | +| GRANT SELECT ON ALL TABLES TO 'fx_readers'; | + +The built-in admin and external (SSO/OIDC) users are not dumped. External users +continue to get access through their identity provider and the dumped groups' +external aliases. + +Database dumps also include active table +[storage policies](#storage-policy-clause). No `SHOW CREATE` statement emits +`OWNED BY`: object permissions are restored through grants, not an owner field. +See [Enterprise variant](#enterprise-variant) for creator permissions during +replay. For an ACL restore, replay on an empty target as its built-in admin to +avoid adding another principal's creator grants. + +#### Replay limits + +An Enterprise dump omits the following settings. Reapply them on the target +before opening it to users: + +- Passwords and tokens. Recreated users and service accounts have no + credentials until you set them. +- Per-user, group, and service account + [memory limits](/docs/security/rbac/#memory-limits), + [resource groups](/docs/concepts/resource-groups/), and resource group + mappings. Recreated principals have no per-principal memory override and use + the default resource group until you configure them. +- Older external aliases that may still be active for a group. Only the alias + reported by `SHOW GROUPS` is restored. Review identity-provider mappings + separately if more than one alias has been assigned to a group. + +:::caution Best-effort dump + +The dump is not a transactionally consistent snapshot across schema objects, +principals, and memberships. Each principal's enabled state and direct grants +come from the same snapshot, but concurrent changes to other entities or group +memberships may still produce a mixed result. Avoid schema and ACL changes +while exporting for replay, and verify the target's grants afterwards with +[`active_grants()`](/docs/query/functions/access-control/#active_grants). + +::: + +#### Use in views + +In QuestDB Enterprise 4.2.0 and later, when access control is enabled: + +- A regular view may read an ACL dump in its query body. The dump checks the + reader's `LIST USERS` and `USER DETAILS`, not the view definer's permissions. +- A regular view's `PIVOT IN` subquery cannot read an ACL dump, directly or + through another view. Use `INCLUDE (SCHEMA)` if only schema is needed. See + [administrative operations in view metadata](/docs/concepts/views/#administrative-operations-in-view-metadata). +- Materialized and live views reject any ACL category, including the default + `SHOW CREATE DATABASE`, even for an administrator. A schema-only form such as + `INCLUDE (SCHEMA)` is allowed under the + [catalogue statement rules](/docs/query/sql/create-mat-view/#catalogue-functions): + it must be in the view's own query, and the creator must hold `SYSTEM ADMIN`. ### SHOW CREATE LIVE VIEW @@ -238,9 +318,9 @@ SHOW CREATE LIVE VIEW trades_ma; This returns the `CREATE LIVE VIEW` statement that would recreate the [live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, -`IN MEMORY`, `PARTITION BY`, and `START FROM` clauses. On QuestDB Enterprise the -output also carries an `OWNED BY` clause naming the user running `SHOW CREATE`. -See the [Enterprise variant](#enterprise-variant) of `SHOW CREATE TABLE`. +`IN MEMORY`, `PARTITION BY`, and `START FROM` clauses. Like other `SHOW CREATE` +statements, it does not include `OWNED BY` or grants. See the +[Enterprise variant](#enterprise-variant) of `SHOW CREATE TABLE`. ### SHOW CREATE MATERIALIZED VIEW @@ -319,52 +399,44 @@ CREATE TABLE sensors ( When an active [storage policy](/docs/concepts/storage-policy/) is attached to a table (Enterprise only), the policy renders as a `STORAGE POLICY(...)` clause in -the `SHOW CREATE TABLE` output: +the `SHOW CREATE TABLE` and `SHOW CREATE DATABASE` output: -```questdb-sql -SHOW CREATE TABLE sensor_data; +```questdb-sql title="Retrieve a table's storage policy in its DDL" +SHOW CREATE TABLE fx_quotes; ``` -```text -CREATE TABLE 'sensor_data' ( - ts TIMESTAMP, - value DOUBLE -) timestamp(ts) PARTITION BY DAY +```questdb-sql title="Returned DDL" +CREATE TABLE 'fx_quotes' ( + timestamp TIMESTAMP, + bid DOUBLE +) timestamp(timestamp) PARTITION BY DAY STORAGE POLICY(TO PARQUET 3 DAYS, DROP LOCAL 1 MONTH); ``` Stages that are not configured on the policy are omitted from the clause. Only an active policy renders: after `ALTER TABLE ... DISABLE STORAGE POLICY`, the -policy is not shown in `SHOW CREATE TABLE`. See +policy is not shown in either dump. See [ALTER TABLE SET STORAGE POLICY](/docs/query/sql/alter-table-set-storage-policy/). #### Enterprise variant -In [QuestDB Enterprise](/enterprise/), `SHOW CREATE TABLE`, `SHOW CREATE VIEW`, -`SHOW CREATE MATERIALIZED VIEW`, and `SHOW CREATE LIVE VIEW` add an `OWNED BY` -clause naming the user running `SHOW CREATE`. QuestDB stores no owner for an -object, so the clause does not tell you who created it or who holds permissions -on it. To see the permissions on an object, use -[`active_grants()`](/docs/query/functions/access-control/#active_grants). - -For example, run as `admin`: - -```questdb-sql -CREATE TABLE trades ( - symbol SYMBOL CAPACITY 256 CACHE, - side SYMBOL CAPACITY 256 CACHE, - price DOUBLE, - amount DOUBLE, - timestamp TIMESTAMP -) timestamp(timestamp) PARTITION BY DAY -WITH maxUncommittedRows=500000, o3MaxLag=600000000us -OWNED BY 'admin'; -``` - -When the statement is replayed, the clause gives that user all permissions on -the new table, with the `GRANT` option. To give them to a different user, group, -or service account, change the clause before replaying it. See -[OWNED BY](/docs/query/sql/create-table/#owned-by). +In [QuestDB Enterprise](/enterprise/) 4.2.0 and later, `SHOW CREATE TABLE`, +`SHOW CREATE VIEW`, `SHOW CREATE MATERIALIZED VIEW`, and `SHOW CREATE LIVE VIEW` +return structure, not permissions. Unlike earlier releases, they do not append +an `OWNED BY` clause naming the caller. QuestDB stores no owner for an object; +`OWNED BY` at creation only decides who receives the creator's permissions. + +On replay without `OWNED BY`, the principal executing the `CREATE` receives the +normal creator permissions, with the `GRANT` option. To give those permissions +to another principal, add an explicit +[`OWNED BY`](/docs/query/sql/create-table/#owned-by) clause. External users must +still supply that clause when creating objects. + +To inspect existing permissions, use +[`active_grants()`](/docs/query/functions/access-control/#active_grants). To +export them as replayable SQL, use +[`SHOW CREATE DATABASE INCLUDE (ACL)`](#enterprise-access-control). A schema-only +dump or a per-object `SHOW CREATE` does not preserve grants. ### SHOW CREATE VIEW diff --git a/documentation/security/governance.md b/documentation/security/governance.md index f73b5b269..1b542c071 100644 --- a/documentation/security/governance.md +++ b/documentation/security/governance.md @@ -128,17 +128,20 @@ functions have [limitations](/docs/query/functions/access-control/#limitations): the built-in admin and OIDC users are not listed, for example. -For a point-in-time snapshot of the whole access control layer, -[`SHOW CREATE DATABASE`](/docs/query/sql/show/#enterprise-access-control) -dumps every user, group, service account, membership, and grant as replayable -SQL, without credentials: +For a replayable export of stored identities, memberships, and direct grants, +use [`SHOW CREATE DATABASE`](/docs/query/sql/show/#enterprise-access-control). +Every ACL category requires both `LIST USERS` and `USER DETAILS`: -```questdb-sql title="Snapshot of identities, memberships, and grants" +```questdb-sql title="Export identities, memberships, and direct grants" SHOW CREATE DATABASE INCLUDE (ACL); ``` Storing that output on a schedule gives you a history of entitlements that you -can diff between two reviews. +can diff between two reviews. It is best-effort, not a transactionally +consistent snapshot across principals and memberships. The export omits the +built-in admin, external users, credentials, memory limits, and resource groups +and their mappings. For a group's external aliases, it preserves only the one +reported by `SHOW GROUPS`. See [Replay limits](/docs/query/sql/show/#replay-limits). ## Audit data access @@ -195,8 +198,9 @@ FROM materialized_views(); ``` [`SHOW CREATE`](/docs/query/sql/show/) returns the full DDL of a single table or -view, and `SHOW CREATE DATABASE` returns every object in dependency order, so -each view appears after the objects it reads from. Running it on a schedule and +view, and `SHOW CREATE DATABASE INCLUDE (SCHEMA)` returns visible schema +objects in dependency order, so each view appears after the objects it reads +from when those dependencies are included. Running it on a schedule and keeping the output records how the schema changed over time. ## Retain and protect data diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 01b4700c0..f1ab41120 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -948,6 +948,13 @@ even to read your own rows. ::: +To export identities, group memberships, service account assignments, and direct +grants as replayable SQL, use +[`SHOW CREATE DATABASE INCLUDE (ACL)`](/docs/query/sql/show/#enterprise-access-control). +It requires both `LIST USERS` and `USER DETAILS`, as does plain +`SHOW CREATE DATABASE`. For structure only, use `INCLUDE (SCHEMA)`, which +requires neither permission and includes only visible objects. + ## Memory limits {#memory-limits} QuestDB Enterprise can limit the native memory tracked for a single query,