From 08fc1f76c3b3dc266cb25e46d846266cd9fdd0a9 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 16:33:14 +0000 Subject: [PATCH 01/10] docs: document the ACL audit log --- documentation/configuration/iam.md | 119 +++++--- documentation/high-availability/overview.md | 6 + documentation/operations/backup.md | 37 +++ documentation/operations/upgrade.md | 80 +++++ .../query/sql/acl/create-service-account.md | 51 ++-- documentation/security/acl-audit-log.md | 284 ++++++++++++++++++ documentation/security/governance.md | 40 ++- documentation/security/rbac.md | 15 + documentation/sidebars.js | 5 + 9 files changed, 576 insertions(+), 61 deletions(-) create mode 100644 documentation/security/acl-audit-log.md diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md index f3ecdec57..1741ac4fb 100644 --- a/documentation/configuration/iam.md +++ b/documentation/configuration/iam.md @@ -1,7 +1,9 @@ --- -title: Identity and Access Management (IAM) -sidebar_label: Identity and Access Management -description: Configuration settings for Identity and Access Management in QuestDB Enterprise. +title: Identity and access management (IAM) +sidebar_label: Identity and access management +description: + Configuration settings for Identity and Access Management in QuestDB + Enterprise. --- :::note @@ -13,11 +15,14 @@ Identity and Access Management is available within Identity and Access Management (IAM) controls authentication and authorization for all QuestDB interfaces. These settings cover the built-in admin user, -password hashing, and REST token behavior. +password hashing, REST token behavior, ACL audit retention, and permission +compaction. For a full explanation of IAM, see the [Identity and Access Management (IAM) documentation](/docs/security/rbac). +## Authentication and authorization + ### acl.admin.password - **Default**: `quest` @@ -44,8 +49,8 @@ Enables or disables the built-in admin user. - **Default**: `false` - **Reloadable**: no -When enabled, the browser's basic auth popup window is used instead of the -Web Console's login screen. Present for backwards compatibility only. +When enabled, the browser's basic auth popup window is used instead of the Web +Console's login screen. Present for backwards compatibility only. ### acl.enabled @@ -54,30 +59,6 @@ Web Console's login screen. Present for backwards compatibility only. Enables or disables Identity and Access Management. -### acl.permissions.compaction.row.threshold - -- **Default**: `10000` -- **Reloadable**: no - -Number of rows in `sys.acl_permissions` above which the table is compacted. -Set to `0` or below to disable compaction. - -`sys.acl_permissions` is an append-only log of grants and revokes, collapsed -at read time to give each entity its current permissions. A deployment that -re-grants on a schedule accumulates rows indefinitely, and past a certain size -every replicated `GRANT` makes a replica reload a progressively larger access -list, slowing logins. Compaction rewrites the log to keep only the rows that -still matter. - -### acl.permissions.compaction.startup.delay - -- **Default**: `300000` -- **Reloadable**: no - -Grace period in milliseconds before the backlog check runs after a boot or a -promotion. The delay keeps compaction from competing with a node that is still -starting up or taking over as primary. - ### acl.entity.name.max.length - **Default**: `255` @@ -91,8 +72,8 @@ Maximum length of user, group, and service account names. - **Reloadable**: no Number of hash iterations used in password hashing. QuestDB Enterprise never -stores passwords in plain text. Higher values are safer but slower. This -should almost never be changed. +stores passwords in plain text. Higher values are safer but slower. This should +almost never be changed. ### acl.rest.token.refresh.threshold @@ -101,12 +82,80 @@ should almost never be changed. When a REST token is created in REFRESH mode, its TTL is extended on every successful authentication, unless the last authentication was within this -threshold (in seconds). Removes unnecessary overhead of continuously -refreshing tokens that are used frequently. +threshold (in seconds). Removes unnecessary overhead of continuously refreshing +tokens that are used frequently. ### line.tcp.acl.enabled - **Default**: `true` - **Reloadable**: no -Enables or disables authentication for the ILP over TCP endpoint only. +Enables or disables authentication for the ILP over TCP endpoint only. Keep it +enabled to enforce write protections on system-managed tables, including the +[ACL audit log](/docs/security/acl-audit-log/#protection-and-administration). + +## ACL audit log + +### acl.audit.storage.policy + +- **Default**: `TO PARQUET 1d` +- **Reloadable**: no + +[Storage policy](/docs/concepts/storage-policy/) used when the server creates +`sys.acl_audit_log`. The value is a comma-separated list of stages, without +surrounding parentheses. The default converts daily partitions to Parquet one +day after each partition ends and never deletes rows. Add `DROP LOCAL` to bound +disk usage, at the cost of deleting older audit evidence: + +```ini title="server.conf: Keep one year of ACL changes" +acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 365d +``` + +- Only `TO PARQUET` and `DROP LOCAL` are supported. `TO REMOTE` and + `DROP REMOTE` are rejected. +- Set the property on every node before the table is first created. Replicas + normally receive the primary's table and policy, but a replica promoted before + receiving the table creates it using its own setting. +- Changing this setting does not change an existing table's policy, even after a + restart. Only the built-in admin can change it with + [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/). +- An empty value creates the table without a policy. If the policy is rejected, + the server logs the error and creates the table without it. Check the result + with `SHOW CREATE TABLE 'sys.acl_audit_log'`. +- On a database restored from a backup, creation of the log and its policy can + be deferred to preserve restore compatibility. See + [Restoring access control tables](/docs/operations/backup/#restoring-access-control-tables). + +See [ACL audit log retention](/docs/security/acl-audit-log/#retention) for +administration when the built-in admin is disabled. + +## Permission compaction + +### acl.permissions.compaction.row.threshold + +- **Default**: `10000` +- **Reloadable**: no + +Number of rows in `sys.acl_permissions` above which the table is compacted. Set +to `0` or below to disable compaction. + +`sys.acl_permissions` is an append-only log of grants and revokes, collapsed at +read time to give each entity its current permissions. A deployment that +re-grants on a schedule accumulates rows indefinitely, and past a certain size +every replicated `GRANT` makes a replica reload a progressively larger access +list, slowing logins. Compaction rewrites the log to keep only the rows that +still matter. + +Archived rows in `sys.acl_permissions_history` retain their `change_id`, which +links them to the +[ACL audit log](/docs/security/acl-audit-log/#link-changes-to-acl-rows), +provided the history table has the column. + +### acl.permissions.compaction.startup.delay + +- **Default**: `300000` +- **Reloadable**: no + +Grace period in milliseconds before the backlog check runs after a boot or a +promotion. The delay keeps compaction from competing with a node that is still +starting up or taking over as primary. diff --git a/documentation/high-availability/overview.md b/documentation/high-availability/overview.md index 3f44f0b44..59889660e 100644 --- a/documentation/high-availability/overview.md +++ b/documentation/high-availability/overview.md @@ -106,6 +106,12 @@ The audit trail of [audited views](/docs/security/audited-views/), but whose rows do not: each node records the reads it serves. See [Replication, backups, and read-only instances](/docs/security/audited-views/#replication-backups-and-read-only-instances). +The [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, +differs from the view audit trail: both its definition and rows replicate. +A replica promoted before receiving the table creates it using its own +[`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) +setting, so configure that property on every node before upgrading. + ## Storage policies in a replicated cluster [Storage policy](/docs/concepts/storage-policy/) definitions are stored in diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index cdd15d539..1d558a0d0 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -542,6 +542,43 @@ the version that created the backup (e.g., 8.1.0 and 8.1.1 are compatible). Restart QuestDB. If restore succeeds, `_backup_restore` is removed automatically. +#### Restoring access control tables + +A backup includes the [ACL audit log](/docs/security/acl-audit-log/), +`sys.acl_audit_log`, if it exists on the source. Its rows replicate, unlike +those of `sys.view_audit`, so a primary backup includes the replicated ACL +change history. + +When a newer server restores a backup taken before the ACL audit log was +introduced, startup leaves the restored ACL and credential tables unchanged: +it neither adds their `change_id` columns nor creates the missing audit log. +This allows further restore updates while the source still runs the older +version. The log and columns arrive with the first restored backup taken +after the source is upgraded. + +Do not run ACL-changing statements on a node that must keep accepting restore +updates. Such a statement changes the restored tables, and the next update +refuses those local changes. The first ACL change also creates the audit log +if it is missing, but without its configured storage policy, to avoid changing +the restored storage policy tables. + +For a standalone recovery that will receive no further restore updates, such +as recovery into a new cluster: + +1. As the built-in admin, add `change_id TIMESTAMP` to each restored table + that lacks it. The server logs the affected tables and their exact + `ALTER TABLE ... ADD COLUMN change_id TIMESTAMP` statements at startup. + Restarting alone does not add these columns, because restore markers stay + on the tables. +2. Once the audit log exists, check its policy with + `SHOW CREATE TABLE 'sys.acl_audit_log'`. If needed, apply the desired + [retention policy](/docs/security/acl-audit-log/#retention) as the built-in + admin. Changing `acl.audit.storage.policy` does not update an existing + table. + +Existing rows retain a `NULL` `change_id`; adding the column does not +reconstruct historical audit entries. + #### Restore failure recovery If restore fails, QuestDB creates artifacts to help diagnose and recover: diff --git a/documentation/operations/upgrade.md b/documentation/operations/upgrade.md index 40b2cba64..fe991a571 100644 --- a/documentation/operations/upgrade.md +++ b/documentation/operations/upgrade.md @@ -101,6 +101,86 @@ Others, such as accounts that triggered role switches through `SYSTEM ADMIN` on [Failover operator](/docs/security/rbac/#failover-operator) and [Database administrator](/docs/security/rbac/#database-administrator). +### ACL audit log + +When upgrading from a version without the +[ACL audit log](/docs/security/acl-audit-log/), the server creates +`sys.acl_audit_log` and adds `change_id` columns to ACL and credential tables. +Existing rows keep a `NULL` `change_id`; past changes are not reconstructed. +Review the following before upgrading: + +- **Configure retention before the first start.** Set + [`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) + on every node. Its default, `TO PARQUET 1d`, never deletes rows. Once the + log exists, only the built-in admin can change its policy, so set retention + before disabling that account. Verify it with + `SHOW CREATE TABLE 'sys.acl_audit_log'` after startup and promotion. +- **Grant auditors the required permissions.** Reading the log takes + `LIST USERS`, `USER DETAILS`, and `SELECT` on the table. A principal with + only `SELECT ON ALL TABLES` cannot read it. Other table grants do not let a + principal modify the log. +- **Check for a name collision.** An existing `sys.acl_audit_log` is reused, + with its rows, only if it is a WAL table with the expected columns and types + and `change_id` as its designated timestamp. Any other object with that + name fails startup before the ACL migrations. Rename or drop the conflicting + object using the previous release, then retry the upgrade. Do not disable + ACL to get past this check: doing so can mark migrations as complete without + applying them. +- **Check deferred migrations.** Tables with suspended WAL apply do not get + `change_id` until a later startup after they are resumed. Databases restored + from older backups have separate compatibility rules; see + [Restoring access control tables](/docs/operations/backup/#restoring-access-control-tables). + +A replica promoted before receiving the log creates it with its own retention +setting. If an older primary replicated a conflicting object with the log's +name, promotion itself does not fail, but ACL changes do. The built-in admin +must drop or rename that conflicting object; the next ACL statement or +startup creates the log. This exception applies to the conflicting object, +not to a valid audit log. + +:::warning Rollback limitations + +An older release does not enforce the audit log's special read or write +protections. It treats the log as an ordinary table, so existing broad table +grants may expose or allow changes to audit evidence. A later upgrade reuses +a structurally compatible log, including any rows written during the rollback. + +An older release also cannot compact `sys.acl_permissions` after it gains +`change_id`. Compaction reports +`unexpected acl permissions column count, refusing to compact` and is +disabled until the next restart. Plan rollback around your pre-upgrade backup +rather than assuming the migrated database behaves as it did before. + +::: + +#### Embedded Enterprise Java integrations + +Applications that directly implement Enterprise Java APIs must update and +rebuild: + +- ACL write methods on `AccessListWriter`, and credential write methods on + `PasswordStore`, `JwkStore`, and `PersistentRestTokenStore`, now take an + `AclChange` as their first argument. Production callers use these methods, + not the older overloads retained for tests. +- `EntSecurityContext` implementations must implement + `isSessionPrincipalExternal()`. It describes the authenticated identity + and must not change when the session assumes a service account. Forwarding + contexts must forward this method too. + +### Service account creation + +[`CREATE SERVICE ACCOUNT IF NOT EXISTS`](/docs/query/sql/acl/create-service-account/) +now leaves an existing account unchanged. Earlier versions assigned the +existing account to the statement's owner with grant option; now ownership is +assigned only when the statement creates the account. An invalid owner is +rejected before any changes, even when the account already exists. + +Update provisioning scripts that relied on `IF NOT EXISTS ... OWNED BY` to +assign an existing account. Use +[`GRANT ASSUME SERVICE ACCOUNT ... WITH GRANT OPTION`](/docs/query/sql/acl/grant-assume-service-account/) +explicitly, from a principal authorized to grant it. Review existing +assignments with `SHOW SERVICE ACCOUNTS` for each user or group. + ### Upgrading to 4.1.0 QuestDB Enterprise 4.1.0 hides tables and views from principals that hold no diff --git a/documentation/query/sql/acl/create-service-account.md b/documentation/query/sql/acl/create-service-account.md index 97651a2ea..53586954b 100644 --- a/documentation/query/sql/acl/create-service-account.md +++ b/documentation/query/sql/acl/create-service-account.md @@ -12,8 +12,8 @@ import { EnterpriseNote } from "@site/src/components/EnterpriseNote" RBAC provides fine-grained database permissions management. -To create a new service account in the database, the `CREATE SERVICE ACCOUNT` -keywords are used. +`CREATE SERVICE ACCOUNT` creates an identity for an application or a set of +permissions that users can assume without sharing credentials. For full documentation of the Access Control List and Role-based Access Control, see the [RBAC operations](/docs/security/rbac) page. @@ -36,43 +36,56 @@ each of the service account's queries may allocate, use after creating it. Service accounts do not inherit group limits. The chosen name must be unique across all users (including the built-in admin), -groups and service accounts. If the name has already been reserved, the command -fails and an error is raised, unless the `IF NOT EXISTS` clause is included in -the statement. +groups and service accounts. `IF NOT EXISTS` leaves an existing service account +unchanged: it does not assign it to the caller or to the principal named in +`OWNED BY`, change its credentials, or create an +[ACL audit row](/docs/security/acl-audit-log/). A name held by a user or group +still fails. Note that new service accounts can only access the database if the necessary [endpoint permissions](/docs/security/rbac/#endpoint-permissions) have been granted. -The user creating the service account automatically receives the -`ASSUME SERVICE ACCOUNT` permission with `GRANT` option, unless the `OWNED BY` -clause is present, in which case the permission is granted to the user or -group specified in the clause. +When the statement creates a new account, the creating user automatically +receives the `ASSUME SERVICE ACCOUNT` permission with `GRANT` option, unless the +`OWNED BY` clause is present, in which case the permission is granted to the +user or group specified in the clause. To assign an existing account, use +[`GRANT ASSUME SERVICE ACCOUNT`](/docs/query/sql/acl/grant-assume-service-account/) +explicitly. -The `OWNED BY` clause cannot be omitted if the service account is created by -an external user, because permissions cannot be granted to them. +The owner must be eligible to hold a service account. For example, a service +account cannot own another service account. Owner validation runs even with +`IF NOT EXISTS`, before any changes are made. + +The `OWNED BY` clause cannot be omitted if the service account is created by an +external user, because permissions cannot be granted to them. ## Examples -```questdb-sql -CREATE SERVICE ACCOUNT audit; +```questdb-sql title="Create a service account without credentials" +CREATE SERVICE ACCOUNT risk_report; -CREATE SERVICE ACCOUNT IF NOT EXISTS audit; +CREATE SERVICE ACCOUNT IF NOT EXISTS risk_report; ``` -``` +The second statement leaves the existing account unchanged. + +```questdb-sql title="Create an account owned by a group" CREATE GROUP analysts; CREATE SERVICE ACCOUNT dashboard OWNED BY analysts; +``` +```questdb-sql title="Assign an existing account explicitly" +GRANT ASSUME SERVICE ACCOUNT risk_report TO analysts WITH GRANT OPTION; ``` -It can be verified with: +Check the authentication methods: -```questdb-sql -SHOW SERVICE ACCOUNT audit; +```questdb-sql title="Inspect the new account's authentication methods" +SHOW SERVICE ACCOUNT risk_report; ``` -that yields: +The account has no credentials yet: | auth_type | enabled | | ---------- | ------- | diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md new file mode 100644 index 000000000..b8697cf38 --- /dev/null +++ b/documentation/security/acl-audit-log.md @@ -0,0 +1,284 @@ +--- +title: ACL audit log +sidebar_label: ACL audit log +description: + Track changes to users, groups, service accounts, credentials, and permissions + in QuestDB Enterprise, with principal attribution and configurable retention. +--- + +The ACL audit log records changes to access control in `sys.acl_audit_log`, so +you can investigate who granted access, changed credentials, or modified a +principal. Each row identifies the authenticated principal, the principal it +acted as, and the statement that caused the change. + +:::note + +The ACL audit log is available in [QuestDB Enterprise](/enterprise/). For read +auditing rather than access control changes, see +[Audited views](/docs/security/audited-views/). + +::: + +## Read the log + +Reading the log requires all three permissions: + +- `LIST USERS`. +- `USER DETAILS`. +- `SELECT` on `sys.acl_audit_log`, or on all tables. + +The statement texts disclose identities, memberships, grants, and external group +aliases, so `SELECT ON ALL TABLES` alone is not enough. Without both +`LIST USERS` and `USER DETAILS`, the table looks missing. Permissions may be +inherited through a group. The built-in admin and principals with +`DATABASE ADMIN` can read the log. + +For an existing user named `auditor`, grant: + +```questdb-sql title="Allow an auditor to read ACL changes" +GRANT LIST USERS, USER DETAILS TO auditor; +GRANT SELECT ON 'sys.acl_audit_log' TO auditor; +``` + +The user also needs an +[endpoint permission](/docs/security/rbac/#endpoint-permissions) such as `HTTP` +or `PGWIRE` to connect. + +```questdb-sql title="Review access control changes in the last week" +SELECT change_id, session_principal, effective_principal, + is_external, statement_kind, statement_text +FROM 'sys.acl_audit_log' +WHERE change_id IN '$now-7d..$now' +ORDER BY change_id DESC; +``` + +## Log columns + +`sys.acl_audit_log` is a WAL table partitioned by day, with `change_id` as its +[designated timestamp](/docs/concepts/designated-timestamp/). + +| Column | Type | Description | +| --------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `change_id` | `TIMESTAMP` | Microsecond timestamp identifying the change. Rows written by the same change to ACL and credential tables carry this value. | +| `session_principal` | `VARCHAR` | Principal that authenticated the session. `NULL` when no principal is available. | +| `effective_principal` | `VARCHAR` | Principal whose permissions were used, such as an assumed service account. `NULL` when no principal is available. | +| `is_external` | `BOOLEAN` | Whether the session principal is managed by an external identity provider, such as OIDC. Assuming a service account does not change this flag. | +| `origin` | `VARCHAR` | `user` for changes caused by user statements, or `migration` for changes made by an internal ACL migration. | +| `statement_kind` | `VARCHAR` | Statement category, such as `CREATE USER`, `GRANT`, or `ALTER SERVICE ACCOUNT`. | +| `statement_text` | `VARCHAR` | Statement text with passwords and REST token values replaced by `'****'`. `NULL` when the text is unavailable. | + +With [`ASSUME SERVICE ACCOUNT`](/docs/query/sql/acl/assume-service-account/), +`session_principal` remains the user who connected, while `effective_principal` +names the service account. Otherwise, the two normally match. Use `is_external` +to distinguish an OIDC identity from a local user with the same name. + +```questdb-sql title="Find changes made through an assumed service account" +SELECT change_id, session_principal, is_external, statement_text +FROM 'sys.acl_audit_log' +WHERE effective_principal = 'risk_admin' + AND session_principal != effective_principal +ORDER BY change_id DESC; +``` + +## Recorded changes + +The log covers changes to users, groups, service accounts, group membership, +external aliases, grants and revokes, passwords, JWK tokens, and persistent REST +tokens. It also records per-principal memory limit changes and resource group +assignments. Creating, altering, or dropping a resource group itself is not an +ACL change and is not recorded here. + +An audited statement produces one audit row even when it writes multiple ACL or +credential rows. For example, `CREATE USER ... WITH PASSWORD` records the +creation and password assignment under one `change_id`, and a cascading `REVOKE` +uses the same ID for all affected grants. Each statement in a batch records its +own text, not the entire batch. + +DDL can produce audit rows for its permission changes, but this is not a +complete DDL log: + +- Owner grants on new tables, views, materialized views, live views, and columns + record the creator as the principal, with no statement text. +- Grant cleanup after dropping or renaming a table or column records the + statement kind, with `NULL` principals and statement text. Its origin is still + `user`. +- DDL that writes no ACL rows leaves no audit row. + +The following also leave no audit row: + +- Authentication attempts and ordinary queries. +- Statements that fail before their first write, or write nothing, such as + `CREATE USER IF NOT EXISTS` when the user already exists. +- Transient REST token creation or removal. These tokens exist only in memory on + the node that issued them. +- REST token TTL refreshes. Refreshed credential rows retain the issuing + change's ID instead of creating another audit row. `DROP TOKEN TYPE REST` is + audited only when it removes a persistent token. + +## Link changes to ACL rows + +The following system tables carry a `change_id` column linking a row to its +audit entry: + +- `sys.acl_entities` +- `sys.acl_external_groups` +- `sys.acl_jwk_tokens` +- `sys.acl_links` +- `sys.acl_passwords` +- `sys.acl_permissions` +- `sys.acl_permissions_history` +- `sys.acl_rest_tokens` + +These are [protected tables](/docs/security/rbac/#visibility-rules): only the +built-in admin can query them, even when another principal can read the audit +log. For example, run this as the built-in admin: + +```questdb-sql title="Trace permission rows back to the statement that wrote them" +SELECT p.name, p.table_name, p.column_name, p.status, + a.change_id, a.session_principal, a.statement_text +FROM 'sys.acl_permissions' p +JOIN 'sys.acl_audit_log' a ON p.change_id = a.change_id +WHERE p.name = 'risk_analyst' +ORDER BY a.change_id DESC; +``` + +[Permission compaction](/docs/configuration/iam/#aclpermissionscompactionrowthreshold) +preserves `change_id` when archiving rows to `sys.acl_permissions_history`. +Query that table instead for archived grants and revokes. Rows written before +the column was added keep a `NULL` ID and do not join to the log. Audit +retention can also remove the corresponding log row. + +## Delivery and failure behavior + +Before an ACL statement writes its first change, QuestDB commits its audit row +to the log's WAL. If that commit fails, the ACL change does not proceed. The row +becomes queryable after WAL apply, so it may not appear immediately. An audit +row is not proof that the whole statement succeeded: a later write can fail +after the audit row was committed. + +Two post-DDL operations are best-effort exceptions: grant cleanup after a drop +or rename, and the owner grant on a newly added column. The DDL has already +committed, so these permission changes still proceed if their audit row cannot +be written. The server logs the failure, and the affected rows have no +`change_id`. + +:::warning Audit write availability + +With +[`cairo.wal.apply.suspended.write.denied=true`](/docs/configuration/wal/#cairowalapplysuspendedwritedenied), +listing `sys.acl_audit_log` in +[`cairo.wal.apply.suspended.tables`](/docs/configuration/wal/#cairowalapplysuspendedtables) +blocks ACL statements. It also blocks creating objects for owners other than the +built-in admin, including tables created by ingestion, because these operations +must grant permissions to the owner. Remove the audit table from the list to +restore those operations. + +::: + +Change IDs increase even when the system clock moves backward. Audit, ACL, and +credential timestamps can therefore be ahead of the wall clock until it catches +up. Do not assume that an ID is an exact wall-clock event time, or that a query +bounded by `$now` includes every recent change after a clock correction. + +## Protection and administration + +The log is system-managed. With access control enabled, no principal, including +the built-in admin, can insert or update its rows through SQL or authenticated +ingestion. + +Only the built-in admin, without an assumed service account, can truncate the +log, drop partitions, set TTL, or change its storage policy. Ordinary table +grants, `ALL`, `DATABASE ADMIN`, and `SYSTEM ADMIN` do not authorize these +operations. These capabilities mean the log is not an immutable archive. + +The following are refused even to the built-in admin: + +- Creating an object under the reserved name `sys.acl_audit_log`. Only the + server creates the log. +- Dropping or renaming the log, or renaming another table to its name. +- Structural changes to its columns or deduplication settings. +- `SUSPEND WAL` and `RESUME WAL FROM TXN`. Skipping transactions could discard + audit rows for changes that already took effect. `REBASE WAL` is also + unavailable because it requires a manually suspended table. + +A failed WAL apply can still suspend the table. Plain +[`RESUME WAL`](/docs/query/sql/alter-table-resume-wal/) applies waiting rows +without skipping them and remains available with the `RESUME WAL` permission. +`SYSTEM ADMIN` and `DATABASE ADMIN` holders can inspect the log's WAL state +through WAL diagnostics even without permission to read its rows. + +:::warning Keep authentication enabled + +Keep both [`acl.enabled`](/docs/configuration/iam/#aclenabled) and +[`line.tcp.acl.enabled`](/docs/configuration/iam/#linetcpaclenabled) enabled, +and restrict network access to ingestion endpoints. Unauthenticated ILP over TCP +does not enforce the system-managed table write protections. + +::: + +## Retention + +The default policy, `TO PARQUET 1d`, converts daily partitions to Parquet one +day after each partition ends and never deletes rows. Set +[`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) on +every node before the log is first created if you need automatic retention: + +```ini title="server.conf: Keep one year of ACL changes" +acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 365d +``` + +The setting applies only at table creation. To change an existing log's policy, +run this as the built-in admin on the primary: + +```questdb-sql title="Set retention on an existing ACL audit log" +ALTER TABLE 'sys.acl_audit_log' +SET STORAGE POLICY(TO PARQUET 1d, DROP LOCAL 365d); +``` + +Only `TO PARQUET` and `DROP LOCAL` stages are supported. `TO REMOTE` and +`DROP REMOTE` are rejected, even though the log replicates. Without remote +storage, `DROP LOCAL` deletes the affected partitions, so keep backups or export +the log before retention removes evidence you need. + +An empty setting creates the table without a policy. A rejected policy is logged +and the table is created without it. Verify the policy after the first startup +and after a replica's first promotion: + +```questdb-sql title="Check the ACL audit log's storage policy" +SHOW CREATE TABLE 'sys.acl_audit_log'; +``` + +A non-zero TTL blocks setting a storage policy. Clear it first with +`ALTER TABLE 'sys.acl_audit_log' SET TTL 0 HOURS`. + +If the [built-in admin](/docs/security/rbac/#built-in-admin) is disabled, nobody +can change the policy or manually remove rows. Configure retention before +disabling it. An existing policy keeps running without an enabled admin; +changing it later requires temporarily enabling the built-in admin and +restarting the server. + +## Replication, upgrades, and restores + +Unlike `sys.view_audit`, both the definition and rows of `sys.acl_audit_log` +[replicate](/docs/high-availability/overview/). The primary writes the log, and +replicas receive it along with its storage policy. A replica promoted before +receiving the log creates its own, using its local `acl.audit.storage.policy` +setting. An ACL statement that finds the log missing during promotion creates it +with the same policy. + +On an ordinary upgrade, the server creates the log and adds `change_id` to +existing ACL and credential tables. It does not reconstruct past changes. Tables +with suspended WAL apply get the column at a later startup after they are +resumed. Until the permissions history table has the column, compaction archives +rows without their change IDs. + +A backup taken before the log was introduced is handled differently: startup +neither creates the missing log nor adds `change_id` to restored tables. This +allows continued restore updates from an older source. See +[Restoring access control tables](/docs/operations/backup/#restoring-access-control-tables) +for when the log and columns appear, and what to do after the final restore. + +Before upgrading or rolling back, review the +[ACL audit log upgrade considerations](/docs/operations/upgrade/#acl-audit-log). +Older versions do not enforce the log's protections and cannot compact the +permissions table after it gains `change_id`. diff --git a/documentation/security/governance.md b/documentation/security/governance.md index f73b5b269..cc24876b6 100644 --- a/documentation/security/governance.md +++ b/documentation/security/governance.md @@ -2,9 +2,8 @@ title: Data governance and auditing sidebar_label: Governance and auditing description: - How QuestDB controls who can reach your data, records who read it, traces - the lineage of derived data, and retains and recovers it, with links to - each feature. + How QuestDB controls access, audits access control changes and data reads, + traces the lineage of derived data, and retains and recovers it. --- Data governance is the set of controls that answer, for any piece of data, who @@ -27,6 +26,7 @@ available in all editions. | What may they even see? | [Object visibility](/docs/security/rbac/#object-visibility) | QuestDB Enterprise 4.1.0+ | | How much may they consume? | [Resource groups](/docs/concepts/resource-groups/) and [per-principal memory limits](/docs/security/rbac/#memory-limits) | QuestDB Enterprise 4.1.0+ | | Who can do what, right now? | [Access control functions](/docs/query/functions/access-control/), [`SHOW CREATE DATABASE`](/docs/query/sql/show/#enterprise-access-control) | QuestDB Enterprise | +| Who changed access? | [ACL audit log](/docs/security/acl-audit-log/) | QuestDB Enterprise | | Who read this data? | [Audited views](/docs/security/audited-views/) | QuestDB Enterprise | | What queries ran recently? | [Query tracing](/docs/concepts/deep-dive/query-tracing/), [`query_activity()`](/docs/query/functions/meta/#query_activity) | All | | Where does this data come from? | [`SHOW CREATE`](/docs/query/sql/show/), [`views()`](/docs/query/functions/meta/#views), [`materialized_views()`](/docs/query/functions/meta/#materialized_views) | All | @@ -140,6 +140,29 @@ SHOW CREATE DATABASE INCLUDE (ACL); Storing that output on a schedule gives you a history of entitlements that you can diff between two reviews. +## Audit access control changes + +The [ACL audit log](/docs/security/acl-audit-log/) records changes to users, +groups, service accounts, credentials, and permissions in `sys.acl_audit_log`. +It identifies both the authenticated principal and any assumed service +account, with passwords and REST token values redacted from statement text. +Unlike the view audit trail, its rows replicate. + +```questdb-sql title="Who granted or revoked access in the last week" +SELECT change_id, session_principal, effective_principal, statement_text +FROM 'sys.acl_audit_log' +WHERE change_id IN '$now-7d..$now' + AND statement_kind IN ('GRANT', 'REVOKE') +ORDER BY change_id DESC; +``` + +Reading it requires `LIST USERS`, `USER DETAILS`, and `SELECT` on the log. +Ordinary ACL changes commit an audit row before writing the change; an audit +write failure blocks the change. This is not a complete DDL or authentication +log, and post-DDL grant cleanup has weaker guarantees. See +[Recorded changes](/docs/security/acl-audit-log/#recorded-changes) and +[Delivery and failure behavior](/docs/security/acl-audit-log/#delivery-and-failure-behavior). + ## Audit data access ### Audited views @@ -272,10 +295,13 @@ compliance control: on the underlying tables reads the same rows with no audit row, and denied reads are usually not recorded. - **There is no durable query log.** Query tracing keeps 24 hours. -- **There is no audit log of DDL, grants, or logins.** Schema changes, grants - and revokes, and authentication attempts are not recorded in a queryable - table. Snapshots from `SHOW CREATE DATABASE` and the server log are the - closest substitutes. +- **ACL auditing is not a complete DDL or login log.** The + [ACL audit log](/docs/security/acl-audit-log/) records grants, revokes, and + other persisted access control changes, but not authentication attempts, + transient REST tokens, or statements that write nothing. DDL is recorded + only when it changes permissions, and some such rows lack principals and + statement text. Retention and built-in admin maintenance can remove rows. + Keep `SHOW CREATE DATABASE` snapshots for schema history. - **There is no session listing.** `sys.metrics` counts open connections per protocol, idle ones included, but not who holds them, and `query_activity()` shows only running queries. diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 8025247e6..ff194e296 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -512,6 +512,15 @@ When a user creates a table, they automatically receive all permissions on it with the GRANT option. This ownership does not persist - if revoked, they cannot get it back without someone re-granting it. +## Audit access control changes + +The [ACL audit log](/docs/security/acl-audit-log/) records who changes users, +groups, service accounts, credentials, and permissions, including both the +session principal and any assumed service account. Query `sys.acl_audit_log` +with `LIST USERS`, `USER DETAILS`, and `SELECT` on the table; `SELECT ON ALL +TABLES` alone does not grant access. Only the built-in admin can change its +retention policy or remove rows. + ## Object visibility {#object-visibility} QuestDB Enterprise hides tables, views, materialized views, and live views from @@ -566,6 +575,12 @@ and `wait_wal_table()`, and WAL recovery statements such as [`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/) reach them for principals that hold the permission the statement requires. +The [ACL audit log](/docs/security/acl-audit-log/#read-the-log), +`sys.acl_audit_log`, is a separate exception: it requires both `LIST USERS` +and `USER DETAILS` in addition to ordinary table permissions. Without both, +it looks missing even to a principal with a grant `ON ALL TABLES`. Unlike +the protected ACL tables, its rows can be read by an authorized auditor. + Permissions count from every source the principal is authorized with: its own grants and those of its groups or, after [`ASSUME SERVICE ACCOUNT`](/docs/query/sql/acl/assume-service-account/), the diff --git a/documentation/sidebars.js b/documentation/sidebars.js index 6cef970b3..8fd390cce 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -806,6 +806,11 @@ module.exports = { type: "doc", label: "OpenID Connect (OIDC)", }, + { + id: "security/acl-audit-log", + type: "doc", + label: "ACL audit log", + }, { id: "security/audited-views", type: "doc", From f0bc945ac9ca171fe8fb0cbe046d6e9765776675 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 17:31:17 +0000 Subject: [PATCH 02/10] docs: address ACL audit log review findings --- documentation/changelog.mdx | 16 +++++ documentation/configuration/iam.md | 7 ++ documentation/high-availability/overview.md | 5 +- documentation/operations/backup.md | 71 ++++++++++++++++--- documentation/operations/upgrade.md | 25 +++++-- .../query/sql/acl/create-service-account.md | 13 +++- documentation/security/acl-audit-log.md | 19 ++--- documentation/security/governance.md | 15 +++- documentation/security/rbac.md | 7 +- 9 files changed, 146 insertions(+), 32 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 296809763..ced9f634c 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -8,6 +8,22 @@ description: Recent updates and improvements to the QuestDB documentation. This page tracks significant updates to the QuestDB documentation. +## November 2026 + +### New + +- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier + +### Reference + +- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, supported stages, and built-in-admin requirements for changing an existing log's policy + +### Updated + +- [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the upcoming change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation). This behavior is not available in Enterprise 4.1.0 or earlier +- [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the upcoming ACL audit log +- [Governance and auditing](/docs/security/governance/#audit-access-control-changes) - Added an ACL audit query covering ordinary grants and revokes as well as service-account assumption assignments and revocations + ## October 2026 ### Releases diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md index 1741ac4fb..3b8db9d87 100644 --- a/documentation/configuration/iam.md +++ b/documentation/configuration/iam.md @@ -101,6 +101,13 @@ enabled to enforce write protections on system-managed tables, including the - **Default**: `TO PARQUET 1d` - **Reloadable**: no +:::note Upcoming Enterprise setting + +This setting belongs to the upcoming ACL audit log feature. It is not available +in QuestDB Enterprise 4.1.0 or earlier. + +::: + [Storage policy](/docs/concepts/storage-policy/) used when the server creates `sys.acl_audit_log`. The value is a comma-separated list of stages, without surrounding parentheses. The default converts daily partitions to Parquet one diff --git a/documentation/high-availability/overview.md b/documentation/high-availability/overview.md index 59889660e..3642e6995 100644 --- a/documentation/high-availability/overview.md +++ b/documentation/high-availability/overview.md @@ -106,8 +106,9 @@ The audit trail of [audited views](/docs/security/audited-views/), but whose rows do not: each node records the reads it serves. See [Replication, backups, and read-only instances](/docs/security/audited-views/#replication-backups-and-read-only-instances). -The [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, -differs from the view audit trail: both its definition and rows replicate. +The upcoming [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, +is not available in Enterprise 4.1.0 or earlier. It differs from the view audit +trail: both its definition and rows replicate. A replica promoted before receiving the table creates it using its own [`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) setting, so configure that property on every node before upgrading. diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 1d558a0d0..abab441d8 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -528,12 +528,14 @@ SELECT start_ts FROM backups() WHERE status = 'backup complete'; If no backup exists at or before the specified timestamp, QuestDB fails to start with the error: `backup restore error: No backup timestamp found that is <=`. -:::warning +:::warning Restore target -Restore requires an empty database directory. If the target database already -has data (indicated by the presence of `db/.data_id`), restore fails with: -"The local database is not empty." Use a fresh installation directory for -restore operations. +Use an empty database directory for an initial restore. An existing directory +can be reused for a retry or forward update from the same backup lineage, as +described in [Retrying or updating a restore](#retrying-or-updating-a-restore). +The presence of `db/.data_id` alone does not prevent these operations: the local +data ID must match the backup's data ID. To restore a different lineage, use a +fresh installation directory. ::: @@ -542,8 +544,54 @@ the version that created the backup (e.g., 8.1.0 and 8.1.1 are compatible). Restart QuestDB. If restore succeeds, `_backup_restore` is removed automatically. +#### Retrying or updating a restore + +QuestDB Enterprise 4.1.0 and later can reuse a restored database directory for +an interrupted-restore retry or a forward update to a newer backup from the +same lineage. This does not merge independent databases or roll a database back. + +Before reusing the directory, check these restrictions: + +- If a local data ID exists, it must match the backup's data ID. Keep the + original `db/.data_id` and restore commit markers; do not remove them to + bypass this check. +- The selected backup must not be behind the target's committed table state. + A backward restore is refused. Keep the target free of ingestion, ACL changes, + and other writes to restored tables while it must accept further updates; + changes absent from the selected backup can make the update fail. +- Forward updates cannot remove a whole table. If a user table exists locally + but is absent from the selected backup, for example because it was dropped on + the source, the update is refused. Target-only tables also prevent an update. + The server preserves its node-local metrics and its own view audit trail as + exceptions. + +To retry or update: + +1. Stop the target QuestDB process. For a failed restore, first resolve the + reported error and follow [Restore failure recovery](#restore-failure-recovery) + if failure artifacts are present. +2. Keep the database directory in place. Leave or recreate `_backup_restore` + in the install root with the same `backup.object.store` and + `backup.instance.name` as the original restore. For a retry, set + `backup.restore.timestamp` to the same snapshot. For a forward update, set + it to a newer backup timestamp, or omit it to select the latest backup. +3. Start QuestDB. A successful restore removes `_backup_restore`. The + [replication catch-up warning](#restoring-an-older-backup) still applies: + a non-empty replication store can advance the node beyond that backup. + +If the target has diverged, or you need an older snapshot or an update that +removes a table, restore into a fresh installation directory instead of deleting +the existing recovery copy. + #### Restoring access control tables +:::note Upcoming Enterprise feature + +The ACL audit log is not available in QuestDB Enterprise 4.1.0 or earlier. This +section describes restore behavior on the upcoming release that introduces it. + +::: + A backup includes the [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, if it exists on the source. Its rows replicate, unlike those of `sys.view_audit`, so a primary backup includes the replicated ACL @@ -552,9 +600,9 @@ change history. When a newer server restores a backup taken before the ACL audit log was introduced, startup leaves the restored ACL and credential tables unchanged: it neither adds their `change_id` columns nor creates the missing audit log. -This allows further restore updates while the source still runs the older -version. The log and columns arrive with the first restored backup taken -after the source is upgraded. +This allows [forward restore updates](#retrying-or-updating-a-restore) while the +source still runs the older version. The log and columns arrive with the first +restored backup taken after the source is upgraded. Do not run ACL-changing statements on a node that must keep accepting restore updates. Such a statement changes the restored tables, and the next update @@ -653,8 +701,11 @@ If you encounter errors during backup or restore: - **"Failed restore directory found"**: A previous restore attempt failed. Remove the `.restore_failed/` directory and `_restore_failed` file, then restart. See [Restore failure recovery](#restore-failure-recovery). -- **"The local database is not empty"**: Restore requires an empty database - directory. Use a fresh installation or remove the existing `db/` directory. +- **"The local database belongs to a different lineage"**: The target's data ID + differs from the backup's. Restore into a fresh installation directory; do not + remove `db/.data_id` from an existing database to bypass the check. For a retry + or forward update, verify that the backup source and lineage are unchanged; + see [Retrying or updating a restore](#retrying-or-updating-a-restore). ## QuestDB OSS: manual backups with checkpoints diff --git a/documentation/operations/upgrade.md b/documentation/operations/upgrade.md index fe991a571..c28200bf7 100644 --- a/documentation/operations/upgrade.md +++ b/documentation/operations/upgrade.md @@ -103,6 +103,14 @@ Others, such as accounts that triggered role switches through `SYSTEM ADMIN` on ### ACL audit log +:::note Upcoming Enterprise feature + +This section describes an unreleased feature and its migrations. The ACL audit +log is not available in QuestDB Enterprise 4.1.0 or earlier. These instructions +apply when upgrading to a release that includes it. + +::: + When upgrading from a version without the [ACL audit log](/docs/security/acl-audit-log/), the server creates `sys.acl_audit_log` and adds `change_id` columns to ACL and credential tables. @@ -169,11 +177,20 @@ rebuild: ### Service account creation +:::note Upcoming Enterprise behavior change + +The behavior described here is unreleased. It does not apply to QuestDB +Enterprise 4.1.0 or earlier. + +::: + +In the upcoming release, [`CREATE SERVICE ACCOUNT IF NOT EXISTS`](/docs/query/sql/acl/create-service-account/) -now leaves an existing account unchanged. Earlier versions assigned the -existing account to the statement's owner with grant option; now ownership is -assigned only when the statement creates the account. An invalid owner is -rejected before any changes, even when the account already exists. +leaves an existing account unchanged. In Enterprise 4.1.0 and earlier, the +statement could assign the existing account to the statement's owner with grant +option. With the new behavior, ownership is assigned only when the statement +creates the account. An invalid owner is rejected before any changes, even when +the account already exists. Update provisioning scripts that relied on `IF NOT EXISTS ... OWNED BY` to assign an existing account. Use diff --git a/documentation/query/sql/acl/create-service-account.md b/documentation/query/sql/acl/create-service-account.md index 53586954b..1377d7a70 100644 --- a/documentation/query/sql/acl/create-service-account.md +++ b/documentation/query/sql/acl/create-service-account.md @@ -35,6 +35,16 @@ each of the service account's queries may allocate, use [`ALTER SERVICE ACCOUNT ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-service-account-set-memory-limit/) after creating it. Service accounts do not inherit group limits. +:::note Upcoming behavior change + +The `IF NOT EXISTS` and owner-validation behavior described below is unreleased. +In QuestDB Enterprise 4.1.0 and earlier, `IF NOT EXISTS` could assign an existing +account to the statement's owner rather than leave it unchanged. See the +[service account upgrade guidance](/docs/operations/upgrade/#service-account-creation) +before relying on the new behavior in provisioning scripts. + +::: + The chosen name must be unique across all users (including the built-in admin), groups and service accounts. `IF NOT EXISTS` leaves an existing service account unchanged: it does not assign it to the caller or to the principal named in @@ -68,7 +78,8 @@ CREATE SERVICE ACCOUNT risk_report; CREATE SERVICE ACCOUNT IF NOT EXISTS risk_report; ``` -The second statement leaves the existing account unchanged. +With the upcoming behavior, the second statement leaves the existing account +unchanged. ```questdb-sql title="Create an account owned by a group" CREATE GROUP analysts; diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index b8697cf38..0d2122c34 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -2,19 +2,20 @@ title: ACL audit log sidebar_label: ACL audit log description: - Track changes to users, groups, service accounts, credentials, and permissions - in QuestDB Enterprise, with principal attribution and configurable retention. + Preview the upcoming ACL audit log in QuestDB Enterprise, with principal + attribution for access control changes and configurable retention. --- -The ACL audit log records changes to access control in `sys.acl_audit_log`, so -you can investigate who granted access, changed credentials, or modified a -principal. Each row identifies the authenticated principal, the principal it -acted as, and the statement that caused the change. +The upcoming ACL audit log records changes to access control in +`sys.acl_audit_log`, so you can investigate who granted access, changed +credentials, or modified a principal. Each row identifies the authenticated +principal, the principal it acted as, and the statement that caused the change. -:::note +:::note Upcoming Enterprise feature -The ACL audit log is available in [QuestDB Enterprise](/enterprise/). For read -auditing rather than access control changes, see +This page describes an unreleased [QuestDB Enterprise](/enterprise/) feature. +The ACL audit log and `acl.audit.storage.policy` are not available in Enterprise +4.1.0 or earlier. For read auditing rather than access control changes, see [Audited views](/docs/security/audited-views/). ::: diff --git a/documentation/security/governance.md b/documentation/security/governance.md index cc24876b6..2e47b5915 100644 --- a/documentation/security/governance.md +++ b/documentation/security/governance.md @@ -26,7 +26,7 @@ available in all editions. | What may they even see? | [Object visibility](/docs/security/rbac/#object-visibility) | QuestDB Enterprise 4.1.0+ | | How much may they consume? | [Resource groups](/docs/concepts/resource-groups/) and [per-principal memory limits](/docs/security/rbac/#memory-limits) | QuestDB Enterprise 4.1.0+ | | Who can do what, right now? | [Access control functions](/docs/query/functions/access-control/), [`SHOW CREATE DATABASE`](/docs/query/sql/show/#enterprise-access-control) | QuestDB Enterprise | -| Who changed access? | [ACL audit log](/docs/security/acl-audit-log/) | QuestDB Enterprise | +| Who changed access? | [ACL audit log](/docs/security/acl-audit-log/) | QuestDB Enterprise (upcoming) | | Who read this data? | [Audited views](/docs/security/audited-views/) | QuestDB Enterprise | | What queries ran recently? | [Query tracing](/docs/concepts/deep-dive/query-tracing/), [`query_activity()`](/docs/query/functions/meta/#query_activity) | All | | Where does this data come from? | [`SHOW CREATE`](/docs/query/sql/show/), [`views()`](/docs/query/functions/meta/#views), [`materialized_views()`](/docs/query/functions/meta/#materialized_views) | All | @@ -142,7 +142,8 @@ can diff between two reviews. ## Audit access control changes -The [ACL audit log](/docs/security/acl-audit-log/) records changes to users, +The [ACL audit log](/docs/security/acl-audit-log/) is an upcoming Enterprise +feature, unavailable in Enterprise 4.1.0 or earlier. It records changes to users, groups, service accounts, credentials, and permissions in `sys.acl_audit_log`. It identifies both the authenticated principal and any assumed service account, with passwords and REST token values redacted from statement text. @@ -152,10 +153,18 @@ Unlike the view audit trail, its rows replicate. SELECT change_id, session_principal, effective_principal, statement_text FROM 'sys.acl_audit_log' WHERE change_id IN '$now-7d..$now' - AND statement_kind IN ('GRANT', 'REVOKE') + AND statement_kind IN ( + 'GRANT', 'REVOKE', + 'GRANT ASSUME SERVICE ACCOUNT', + 'REVOKE ASSUME SERVICE ACCOUNT' + ) ORDER BY change_id DESC; ``` +The query includes both ordinary permission grants and revokes and assignments +that let users or groups assume a service account. Other ways to change access, +such as group membership changes, have different statement kinds. + Reading it requires `LIST USERS`, `USER DETAILS`, and `SELECT` on the log. Ordinary ACL changes commit an audit row before writing the change; an audit write failure blocks the change. This is not a complete DDL or authentication diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index ff194e296..cbcf7e1ee 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -514,8 +514,9 @@ get it back without someone re-granting it. ## Audit access control changes -The [ACL audit log](/docs/security/acl-audit-log/) records who changes users, -groups, service accounts, credentials, and permissions, including both the +The [ACL audit log](/docs/security/acl-audit-log/) is an upcoming Enterprise +feature, unavailable in Enterprise 4.1.0 or earlier. It records who changes +users, groups, service accounts, credentials, and permissions, including both the session principal and any assumed service account. Query `sys.acl_audit_log` with `LIST USERS`, `USER DETAILS`, and `SELECT` on the table; `SELECT ON ALL TABLES` alone does not grant access. Only the built-in admin can change its @@ -575,7 +576,7 @@ and `wait_wal_table()`, and WAL recovery statements such as [`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/) reach them for principals that hold the permission the statement requires. -The [ACL audit log](/docs/security/acl-audit-log/#read-the-log), +The upcoming [ACL audit log](/docs/security/acl-audit-log/#read-the-log), `sys.acl_audit_log`, is a separate exception: it requires both `LIST USERS` and `USER DETAILS` in addition to ordinary table permissions. Without both, it looks missing even to a principal with a grant `ON ALL TABLES`. Unlike From 11a65acd52a95dcf185bd2d91470db615ae259a8 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 17:58:39 +0000 Subject: [PATCH 03/10] docs: clarify ACL audit WAL suspension directory names --- documentation/security/acl-audit-log.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index 0d2122c34..9976f6174 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -167,12 +167,17 @@ be written. The server logs the failure, and the affected rows have no With [`cairo.wal.apply.suspended.write.denied=true`](/docs/configuration/wal/#cairowalapplysuspendedwritedenied), -listing `sys.acl_audit_log` in +listing the directory name of `sys.acl_audit_log` in [`cairo.wal.apply.suspended.tables`](/docs/configuration/wal/#cairowalapplysuspendedtables) blocks ACL statements. It also blocks creating objects for owners other than the built-in admin, including tables created by ingestion, because these operations -must grant permissions to the owner. Remove the audit table from the list to -restore those operations. +must grant permissions to the owner. + +Use the `directoryName` column of [`tables()`](/docs/query/functions/meta/#tables) +to find the value, including its table-ID suffix. The SQL table name alone does +not match this setting. To restore those operations, remove that directory-name +entry and [reload the configuration](/docs/configuration/overview/#reloadable-settings) +or restart the server. ::: From ded51da36d893b99d84ba7e146fbb26205994a8f Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 18:55:27 +0000 Subject: [PATCH 04/10] docs: correct ACL audit retention timing and change ID guarantees --- documentation/changelog.mdx | 4 +-- documentation/configuration/iam.md | 14 +++++++--- documentation/security/acl-audit-log.md | 35 +++++++++++++++++++------ 3 files changed, 40 insertions(+), 13 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index ced9f634c..102e749d8 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -12,11 +12,11 @@ This page tracks significant updates to the QuestDB documentation. ### New -- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier +- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, change-ID recovery limits, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier ### Reference -- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, supported stages, and built-in-admin requirements for changing an existing log's policy +- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, partition-start-based conversion eligibility, supported stages, and built-in-admin requirements for changing an existing log's policy ### Updated diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md index 3b8db9d87..24c8287b9 100644 --- a/documentation/configuration/iam.md +++ b/documentation/configuration/iam.md @@ -110,9 +110,17 @@ in QuestDB Enterprise 4.1.0 or earlier. [Storage policy](/docs/concepts/storage-policy/) used when the server creates `sys.acl_audit_log`. The value is a comma-separated list of stages, without -surrounding parentheses. The default converts daily partitions to Parquet one -day after each partition ends and never deletes rows. Add `DROP LOCAL` to bound -disk usage, at the cost of deleting older audit evidence: +surrounding parentheses. + +The default policy, `TO PARQUET 1d`, never deletes rows. It makes inactive daily +partitions eligible for Parquet conversion when the reference time is at least +one day after the partition's start, not its end. By default, the reference time +is the earlier of the latest audit timestamp and the wall clock. Conversion runs +asynchronously, and a quiet log may wait for later changes before a partition +becomes eligible. + +Add `DROP LOCAL` to bound disk usage, at the cost of deleting older audit +evidence: ```ini title="server.conf: Keep one year of ACL changes" acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 365d diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index 9976f6174..8a753a43e 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -60,7 +60,7 @@ ORDER BY change_id DESC; | Column | Type | Description | | --------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| `change_id` | `TIMESTAMP` | Microsecond timestamp identifying the change. Rows written by the same change to ACL and credential tables carry this value. | +| `change_id` | `TIMESTAMP` | Microsecond timestamp for correlating a change with its ACL and credential rows. Subject to [recovery limits](#change-ids-and-recovery). | | `session_principal` | `VARCHAR` | Principal that authenticated the session. `NULL` when no principal is available. | | `effective_principal` | `VARCHAR` | Principal whose permissions were used, such as an assumed service account. `NULL` when no principal is available. | | `is_external` | `BOOLEAN` | Whether the session principal is managed by an external identity provider, such as OIDC. Assuming a service account does not change this flag. | @@ -143,6 +143,10 @@ WHERE p.name = 'risk_analyst' ORDER BY a.change_id DESC; ``` +Do not assume this join is one-to-one: a `change_id` can be reused in the +[recovery case described below](#change-ids-and-recovery), so a permission row +can match more than one audit entry. + [Permission compaction](/docs/configuration/iam/#aclpermissionscompactionrowthreshold) preserves `change_id` when archiving rows to `sys.acl_permissions_history`. Query that table instead for archived grants and revokes. Rows written before @@ -181,10 +185,20 @@ or restart the server. ::: -Change IDs increase even when the system clock moves backward. Audit, ACL, and -credential timestamps can therefore be ahead of the wall clock until it catches -up. Do not assume that an ID is an exact wall-clock event time, or that a query -bounded by `$now` includes every recent change after a clock correction. +### Change IDs and recovery + +While a node remains primary, newly allocated change IDs increase even when the +system clock moves backward. Audit, ACL, and credential timestamps can therefore +be ahead of the wall clock until it catches up. Do not assume that an ID is an +exact wall-clock event time, or that a query bounded by `$now` includes every +recent change after a clock correction. + +After a restart or promotion, QuestDB derives the next ID from timestamps in +applied audit, ACL, and credential rows. A statement that committed its audit +row but failed before writing any ACL or credential row can be missed if its +audit row is still unapplied during recovery. If the clock is at or below that +ID, a later change can reuse it. Do not use `change_id` alone as a globally +unique event identifier, such as a unique key in an external archive. ## Protection and administration @@ -224,9 +238,14 @@ does not enforce the system-managed table write protections. ## Retention -The default policy, `TO PARQUET 1d`, converts daily partitions to Parquet one -day after each partition ends and never deletes rows. Set -[`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) on +The default policy, `TO PARQUET 1d`, never deletes rows. It makes inactive daily +partitions eligible for Parquet conversion when the reference time is at least +one day after the partition's start, not its end. By default, the reference time +is the earlier of the latest audit timestamp and the wall clock. Conversion runs +asynchronously, and a quiet log may wait for later changes before a partition +becomes eligible. + +Set [`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) on every node before the log is first created if you need automatic retention: ```ini title="server.conf: Keep one year of ACL changes" From 495db9db07a946ca8d7c82e39f89e843ae252a79 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 19:32:26 +0000 Subject: [PATCH 05/10] docs: clarify partition-based audit retention --- documentation/changelog.mdx | 3 ++- documentation/concepts/storage-policy.md | 34 +++++++++++++++++------- documentation/configuration/iam.md | 18 ++++++++----- documentation/security/acl-audit-log.md | 24 +++++++++++------ 4 files changed, 55 insertions(+), 24 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 5c5271084..2604efc28 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -17,13 +17,14 @@ This page tracks significant updates to the QuestDB documentation. ### Reference -- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, partition-start-based conversion eligibility, supported stages, and built-in-admin requirements for changing an existing log's policy +- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, partition-start-based conversion and deletion eligibility, supported stages, and built-in-admin requirements for changing an existing log's policy ### Updated - [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the upcoming change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation). This behavior is not available in Enterprise 4.1.0 or earlier - [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the upcoming ACL audit log - [Governance and auditing](/docs/security/governance/#audit-access-control-changes) - Added an ACL audit query covering ordinary grants and revokes as well as service-account assumption assignments and revocations +- [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected stage eligibility to use partition starts rather than ends, and explained the partition allowance for minimum timestamp-based retention, including the ACL audit log examples ## October 2026 diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index 07e4c03b4..bef203988 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -90,18 +90,22 @@ late-arrival, replay, and correction window for the table. See ### TTL evaluation -Storage policy TTLs follow the same evaluation rules as -[TTL](/docs/concepts/ttl/). A partition becomes eligible for a lifecycle action -when its **entire time range** falls outside the TTL window: +Storage policy TTLs measure age from the partition's start. This differs from +[table TTL](/docs/concepts/ttl/), which measures from the end of the partition +and waits until its entire time range is outside the retention window. For +hour-, day-, and week-based durations, a partition becomes eligible when: ```text -eligible when: partition_end_time < reference_time - TTL +reference_time - partition_start_time >= TTL ``` -**This rule is applied independently for each stage's TTL.** A partition can -be eligible for `TO PARQUET` long before it is eligible for `TO REMOTE`, -`DROP LOCAL`, or `DROP REMOTE`. Each stage uses its own `TTL` in the formula -above; the stages share only the reference time and the +Month- and year-based durations compare elapsed calendar months from the same +partition-start boundary. + +Every stage uses this boundary, including `DROP LOCAL` and `DROP REMOTE`, but +each evaluates its own TTL independently. A partition can be eligible for +`TO PARQUET` long before it is eligible for `TO REMOTE`, `DROP LOCAL`, or +`DROP REMOTE`. The stages share the reference time and the [ordering constraints](#ordering-constraint). The reference time is `min(wall_clock_time, latest_timestamp)` by default — @@ -113,7 +117,19 @@ the wall-clock cap for both TTL and storage policy evaluation. See and the data-loss hazard of disabling the cap. QuestDB checks storage policies periodically (every 5 minutes by default) and -processes eligible partitions automatically. +processes eligible partitions automatically. A quiet table may retain partitions +longer because its latest timestamp, and therefore the reference time, stops +advancing. + +:::warning Minimum row retention + +A stage's TTL is not a minimum age for every row in the partition. With daily +partitions, `DROP LOCAL 365d` can delete a row near the end of a partition almost +a day before that row is 365 days old. For a minimum 365-day timestamp-based +retention window, use `DROP LOCAL 366d` to include a one-day partition allowance. +This window is based on the designated timestamp, not the time since ingestion. + +::: ## Storage policy vs TTL diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md index 24c8287b9..1619a460c 100644 --- a/documentation/configuration/iam.md +++ b/documentation/configuration/iam.md @@ -116,16 +116,22 @@ The default policy, `TO PARQUET 1d`, never deletes rows. It makes inactive daily partitions eligible for Parquet conversion when the reference time is at least one day after the partition's start, not its end. By default, the reference time is the earlier of the latest audit timestamp and the wall clock. Conversion runs -asynchronously, and a quiet log may wait for later changes before a partition -becomes eligible. +asynchronously. -Add `DROP LOCAL` to bound disk usage, at the cost of deleting older audit -evidence: +Add `DROP LOCAL` to delete older audit evidence automatically. It uses the same +[partition-start rule](/docs/concepts/storage-policy/#ttl-evaluation) as +conversion: with daily partitions, `DROP LOCAL 365d` can delete a row almost a +day before it is 365 days old. For a minimum 365-day retention window based on +`change_id`, include a one-day partition allowance: -```ini title="server.conf: Keep one year of ACL changes" -acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 365d +```ini title="server.conf: 365-day minimum timestamp-based retention" +acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 366d ``` +This is timestamp-based retention, not elapsed time since a change was recorded. +A quiet log may retain partitions longer while its latest timestamp does not +advance. + - Only `TO PARQUET` and `DROP LOCAL` are supported. `TO REMOTE` and `DROP REMOTE` are rejected. - Set the property on every node before the table is first created. Replicas diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index 8a753a43e..da2ae122b 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -242,22 +242,30 @@ The default policy, `TO PARQUET 1d`, never deletes rows. It makes inactive daily partitions eligible for Parquet conversion when the reference time is at least one day after the partition's start, not its end. By default, the reference time is the earlier of the latest audit timestamp and the wall clock. Conversion runs -asynchronously, and a quiet log may wait for later changes before a partition -becomes eligible. +asynchronously. + +`DROP LOCAL` uses the same +[partition-start rule](/docs/concepts/storage-policy/#ttl-evaluation), not each +row's age. With daily partitions, `DROP LOCAL 365d` can delete a row almost a day +before it is 365 days old. For a minimum 365-day timestamp-based retention +window, use `DROP LOCAL 366d` to include a one-day partition allowance. The +window is based on `change_id`, not elapsed time since the change was recorded; +see [Change IDs and recovery](#change-ids-and-recovery). A quiet log may retain +partitions longer while its latest timestamp does not advance. Set [`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) on -every node before the log is first created if you need automatic retention: +every node before the log is first created: -```ini title="server.conf: Keep one year of ACL changes" -acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 365d +```ini title="server.conf: 365-day minimum timestamp-based retention" +acl.audit.storage.policy=TO PARQUET 1d, DROP LOCAL 366d ``` -The setting applies only at table creation. To change an existing log's policy, -run this as the built-in admin on the primary: +The setting applies only at table creation. To apply the same retention window +to an existing log, run this as the built-in admin on the primary: ```questdb-sql title="Set retention on an existing ACL audit log" ALTER TABLE 'sys.acl_audit_log' -SET STORAGE POLICY(TO PARQUET 1d, DROP LOCAL 365d); +SET STORAGE POLICY(TO PARQUET 1d, DROP LOCAL 366d); ``` Only `TO PARQUET` and `DROP LOCAL` stages are supported. `TO REMOTE` and From 96748fa2fa36ef7c61617a9d0b654c7967d0fefa Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 21:41:10 +0000 Subject: [PATCH 06/10] docs: align storage policy retention guidance --- documentation/changelog.mdx | 2 +- documentation/configuration/audited-views.md | 10 ++++++---- .../getting-started/enterprise-quick-start.md | 17 ++++++++++------- documentation/security/governance.md | 8 +++++++- 4 files changed, 24 insertions(+), 13 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 2604efc28..6ee6fbf6d 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -24,7 +24,7 @@ This page tracks significant updates to the QuestDB documentation. - [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the upcoming change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation). This behavior is not available in Enterprise 4.1.0 or earlier - [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the upcoming ACL audit log - [Governance and auditing](/docs/security/governance/#audit-access-control-changes) - Added an ACL audit query covering ordinary grants and revokes as well as service-account assumption assignments and revocations -- [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected stage eligibility to use partition starts rather than ends, and explained the partition allowance for minimum timestamp-based retention, including the ACL audit log examples +- [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected stage eligibility to use partition starts rather than ends, aligned the audit configuration and Enterprise quick start, and clarified minimum timestamp-based retention and the governance example's partition-age thresholds ## October 2026 diff --git a/documentation/configuration/audited-views.md b/documentation/configuration/audited-views.md index 78d204000..28bd5bc25 100644 --- a/documentation/configuration/audited-views.md +++ b/documentation/configuration/audited-views.md @@ -19,8 +19,8 @@ None of these settings are reloadable: changing any of them requires a restart. The server checks the values at startup, and refuses to start, naming the property, when a numeric value is out of range. -For example, to absorb longer bursts of audited reads and keep 90 days of the -trail: +For example, to absorb longer bursts of audited reads and delete daily +partitions using a 90-day partition-age threshold: ```ini title="server.conf" view.audit.queue.capacity=16384 @@ -82,8 +82,10 @@ same log message. [Storage policy](/docs/concepts/storage-policy/) that `sys.view_audit` is created with. The value is the list of stages written inside `SET STORAGE POLICY(...)`, separated by commas, such as -`TO PARQUET 1d, DROP LOCAL 90d`. The default converts each daily partition to -Parquet one day after the partition ends, and never deletes a row. +`TO PARQUET 1d, DROP LOCAL 90d`. The default never deletes rows. It makes +inactive daily partitions eligible for Parquet conversion when the +[reference time](/docs/concepts/storage-policy/#ttl-evaluation) is at least one +day after the partition's start, not its end. Conversion runs asynchronously. - The setting applies only when the server creates the table. Replicas take the table, and its policy, from the primary. diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index aed0c7d96..74f867bb6 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -684,12 +684,15 @@ between all involved parties. **Q: I set a storage policy with `TO PARQUET`, but I'm not observing reduced storage usage. What could be the issue?** -A: A partition converts only once its entire time range is older than the -`TO PARQUET` delay, and the active partition never converts. Policies are -checked every 5 minutes by default, so allow time for the first run. Use -`SHOW PARTITIONS FROM tableName` to confirm which partitions have `isParquet` -set to `true` and compare `parquetFileSize` with `diskSize`. The compression -ratio varies with the nature of your data, and some data does not compress -well. To trade CPU for a higher ratio, switch the codec to `ZSTD`, as described +A: An inactive partition becomes eligible for conversion when the +[reference time](/docs/concepts/storage-policy/#ttl-evaluation) is at least the +`TO PARQUET` delay after the partition's start, not its end. By default, the +reference time is the earlier of the latest table timestamp and the wall clock, +so a quiet table can delay conversion. The active partition never converts. +Policies are checked every 5 minutes by default, and eligible conversions run +asynchronously. Use `SHOW PARTITIONS FROM tableName` to confirm which partitions +have `isParquet` set to `true` and compare `parquetFileSize` with `diskSize`. +The compression ratio varies with the nature of your data, and some data does +not compress well. To trade CPU for a higher ratio, switch the codec to `ZSTD`, as described in [Parquet data compression](/docs/concepts/parquet/#data-compression). If the ratio still seems out of the expected range, please contact us. diff --git a/documentation/security/governance.md b/documentation/security/governance.md index 2e47b5915..3282dea5d 100644 --- a/documentation/security/governance.md +++ b/documentation/security/governance.md @@ -239,7 +239,7 @@ its lifecycle on a schedule: conversion to Parquet, upload to the local copy, and final deletion. This makes retention a property of the table, declared once, instead of a script that has to keep running: -```questdb-sql title="Keep 90 days locally and three years in object storage" +```questdb-sql title="Tier partitions using age-based thresholds" ALTER TABLE fx_trades SET STORAGE POLICY( TO PARQUET 7d, TO REMOTE 14d, @@ -248,6 +248,12 @@ ALTER TABLE fx_trades SET STORAGE POLICY( ); ``` +The delays are +[partition-age thresholds](/docs/concepts/storage-policy/#ttl-evaluation), +measured from each partition's start, not each row's timestamp or the upload +time. For a minimum timestamp-based retention window, allow for the full +partition interval when choosing the deletion thresholds. + Once `DROP LOCAL` runs, a partition is [sealed](/docs/concepts/cold-storage/#immutability-after-drop-local): later inserts and updates that target it are not applied, and removing the policy does From d27dbd0ef69d6308e3ab390a8dc399ad2cd2d778 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 22:23:29 +0000 Subject: [PATCH 07/10] docs: fix audit recovery and retention guidance --- documentation/changelog.mdx | 4 +-- documentation/concepts/parquet.md | 13 ++++++--- .../getting-started/enterprise-quick-start.md | 27 +++++++++++-------- documentation/security/acl-audit-log.md | 12 +++++---- documentation/security/audited-views.md | 13 ++++++--- 5 files changed, 45 insertions(+), 24 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 6ee6fbf6d..818cdd302 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -12,7 +12,7 @@ This page tracks significant updates to the QuestDB documentation. ### New -- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, change-ID recovery limits, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier +- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, WAL availability troubleshooting, change-ID recovery limits, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier - [GROUPING SETS, ROLLUP, and CUBE](/docs/query/sql/grouping-sets/) - Compute detail rows, subtotals, and a grand total in a single scan, with `GROUP BY` or `SAMPLE BY`. Covers composite syntax, `GROUPING()` and `GROUPING_ID()`, `FILL` support, parallel execution, when a `UNION ALL` of plain `GROUP BY` queries is faster, and the limitations, plus the [`cairo.sql.max.grouping.sets`](/docs/configuration/cairo-engine/#cairosqlmaxgroupingsets) setting ### Reference @@ -24,7 +24,7 @@ This page tracks significant updates to the QuestDB documentation. - [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the upcoming change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation). This behavior is not available in Enterprise 4.1.0 or earlier - [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the upcoming ACL audit log - [Governance and auditing](/docs/security/governance/#audit-access-control-changes) - Added an ACL audit query covering ordinary grants and revokes as well as service-account assumption assignments and revocations -- [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected stage eligibility to use partition starts rather than ends, aligned the audit configuration and Enterprise quick start, and clarified minimum timestamp-based retention and the governance example's partition-age thresholds +- [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected partition-start eligibility across audit retention, TTL migration, Parquet conversion, and Enterprise quick-start guidance, with partition allowances for minimum timestamp-based retention and clearer governance example thresholds ## October 2026 diff --git a/documentation/concepts/parquet.md b/documentation/concepts/parquet.md index d2e638c91..58f014e35 100644 --- a/documentation/concepts/parquet.md +++ b/documentation/concepts/parquet.md @@ -78,13 +78,20 @@ A policy can be attached when creating the table with the `STORAGE POLICY` clause on [`CREATE TABLE`](/docs/query/sql/create-table/#storage-policy), or set later with [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/). -For example, convert a partition to Parquet one hour after it stops being the -active partition: +For example, make an inactive partition eligible when the +[reference time](/docs/concepts/storage-policy/#ttl-evaluation) is at least one +hour after the partition's start: -```questdb-sql title="Convert to Parquet after 1 hour (Enterprise)" +```questdb-sql title="Set a one-hour Parquet conversion threshold" ALTER TABLE market_data SET STORAGE POLICY(TO PARQUET 1h); ``` +For hourly partitions, this threshold can be met as soon as the next partition +becomes active; there is no extra one-hour wait. The active partition never +converts, and eligible conversions run asynchronously. By default, the reference +time is the earlier of the latest table timestamp and the wall clock, so a quiet +table can delay conversion. + The TTL unit cannot be finer than the table's partition size (`market_data` is partitioned by hour). See the [Storage Policy](/docs/concepts/storage-policy/) concept guide for the full partition lifecycle, including `DROP LOCAL`. diff --git a/documentation/getting-started/enterprise-quick-start.md b/documentation/getting-started/enterprise-quick-start.md index 74f867bb6..e9383231e 100644 --- a/documentation/getting-started/enterprise-quick-start.md +++ b/documentation/getting-started/enterprise-quick-start.md @@ -561,7 +561,14 @@ TTL is not supported on Enterprise tables; use a storage policy instead translates it into a `STORAGE POLICY(DROP LOCAL ...)`.) Materialized views are not affected: they continue to use `TTL` for retention in Enterprise. -To move a legacy table from `TTL` to a storage policy: +Before clearing the TTL, choose the replacement policy. Storage policies measure +age from the [partition's start](/docs/concepts/storage-policy/#ttl-evaluation), +while table TTL measures from its end. Reusing the same duration can delete data +up to one partition interval earlier. + +The following example migrates a daily-partitioned `trades` table with +`TTL 30 DAYS`. It uses `DROP LOCAL 31 DAYS` to preserve a minimum 30-day window +based on the designated timestamp: 1. **Clear the existing TTL** by setting it to `0`. This is the only `SET TTL` value Enterprise accepts on a user table, and it is required before a @@ -571,22 +578,20 @@ To move a legacy table from `TTL` to a storage policy: ALTER TABLE trades SET TTL 0 HOURS; ``` -2. **Attach a storage policy** that reproduces, and ideally extends, the - retention the TTL used to provide. A policy lets you keep data in Parquet - after you would previously have dropped it, so `DROP LOCAL` is the stage - that replaces the old TTL horizon: +2. **Attach the replacement policy.** `TO PARQUET` compresses older partitions + without deleting their rows; `DROP LOCAL` provides retention: - ```questdb-sql title="Replace a 1-month TTL with an equivalent policy" + ```questdb-sql title="Preserve 30-day retention with daily partitions" ALTER TABLE trades SET STORAGE POLICY( TO PARQUET 3 DAYS, - DROP LOCAL 1 MONTH + DROP LOCAL 31 DAYS ); ``` - If you want the policy to behave exactly like the old TTL (delete the - partition outright after the same interval, with no Parquet conversion), - use a single-stage policy, for example `STORAGE POLICY(DROP LOCAL 1 MONTH)` - to match `TTL 1 MONTH`. + For retention without Parquet conversion, use + `STORAGE POLICY(DROP LOCAL 31 DAYS)` on this daily-partitioned table. For other + partition intervals or calendar-month/year durations, calculate the threshold + from the partition-start rule rather than copying the old TTL value. Do this for every table you want to keep managed automatically. Tables without a storage policy retain their data indefinitely once their legacy TTL has been diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index da2ae122b..9eede8641 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -177,11 +177,13 @@ blocks ACL statements. It also blocks creating objects for owners other than the built-in admin, including tables created by ingestion, because these operations must grant permissions to the owner. -Use the `directoryName` column of [`tables()`](/docs/query/functions/meta/#tables) -to find the value, including its table-ID suffix. The SQL table name alone does -not match this setting. To restore those operations, remove that directory-name -entry and [reload the configuration](/docs/configuration/overview/#reloadable-settings) -or restart the server. +To restore those operations, remove the audit log's directory-name entry from +`cairo.wal.apply.suspended.tables`, then +[reload the configuration](/docs/configuration/overview/#reloadable-settings) +or restart the server. The entry includes the table-ID suffix, for example +`sys.acl_audit_log~42`; the SQL table name alone does not match. Use the existing +configured entry rather than looking it up in `tables()`, which does not list +this system table. ::: diff --git a/documentation/security/audited-views.md b/documentation/security/audited-views.md index 578786b31..47b7f776e 100644 --- a/documentation/security/audited-views.md +++ b/documentation/security/audited-views.md @@ -350,13 +350,20 @@ sets, `TO PARQUET 1d` by default, so older partitions of the trail move to Parquet. That default never deletes a row. The setting applies only when the server creates the table. After that, change the policy with [`ALTER TABLE SET STORAGE POLICY`](/docs/query/sql/alter-table-set-storage-policy/), -for example to keep a fixed window of the trail: +for example to keep a minimum 90-day timestamp-based window: -```questdb-sql title="Keep 90 days of trail on every node" +```questdb-sql title="Set a 90-day minimum timestamp-based retention window" ALTER TABLE 'sys.view_audit' - SET STORAGE POLICY(TO PARQUET 1d, DROP LOCAL 90d); + SET STORAGE POLICY(TO PARQUET 1d, DROP LOCAL 91d); ``` +The trail is partitioned by day. `DROP LOCAL 91d` includes a one-day allowance +because storage policies measure age from +[each partition's start](/docs/concepts/storage-policy/#ttl-evaluation), not each +row's timestamp. The window is based on audit timestamps, not elapsed time since +a row was recorded. With the default reference time, a quiet node can retain +partitions longer while its latest audit timestamp stops advancing. + - `SET STORAGE POLICY` replaces the whole policy, so restate every stage you want to keep. - A policy set on the primary applies on every node, replicas included. From 567d1abbd70aee9fc1f6238251650bf1c6d25f54 Mon Sep 17 00:00:00 2001 From: glasstiger Date: Thu, 5 Nov 2026 22:52:19 +0000 Subject: [PATCH 08/10] docs: preserve retention in TTL replacement example --- documentation/concepts/storage-policy.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/documentation/concepts/storage-policy.md b/documentation/concepts/storage-policy.md index bef203988..735879c88 100644 --- a/documentation/concepts/storage-policy.md +++ b/documentation/concepts/storage-policy.md @@ -146,16 +146,20 @@ you are already familiar with TTL, this comparison is the fastest way in: In QuestDB Enterprise, use storage policies instead of TTL. On a regular table, `ALTER TABLE SET TTL` with a non-zero value is rejected, while `CREATE TABLE ... TTL` is accepted only for backward compatibility and is -translated into a `STORAGE POLICY(DROP LOCAL ...)`: +translated into a `STORAGE POLICY(DROP LOCAL ...)`. -```questdb-sql --- Instead of: --- ALTER TABLE trades SET TTL 30 DAYS; +For a daily-partitioned `trades` table, preserving the minimum 30-day +timestamp-based retention window of `TTL 30 DAYS` requires a one-day +[partition allowance](#ttl-evaluation): --- Use: -ALTER TABLE trades SET STORAGE POLICY(DROP LOCAL 30d); +```questdb-sql title="Preserve 30-day retention with daily partitions" +ALTER TABLE trades SET STORAGE POLICY(DROP LOCAL 31d); ``` +For other partition intervals or calendar-month/year durations, calculate the +threshold from the [partition-start rule](#ttl-evaluation) rather than copying +the old TTL value. + :::note A table can carry a TTL if it was created in QuestDB Open Source and later From 4245953403dffd2ccd91af71d4341666b8e1bd9d Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 6 Nov 2026 14:24:44 +0000 Subject: [PATCH 09/10] docs: clarify ACL restore migration tables --- documentation/operations/backup.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index abab441d8..5bde77391 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -597,9 +597,23 @@ A backup includes the [ACL audit log](/docs/security/acl-audit-log/), those of `sys.view_audit`, so a primary backup includes the replicated ACL change history. +The following internal ACL and credential tables use a `change_id TIMESTAMP` +column to link their rows to audit entries: + +- `sys.acl_entities` +- `sys.acl_external_groups` +- `sys.acl_jwk_tokens` +- `sys.acl_links` +- `sys.acl_passwords` +- `sys.acl_permissions` +- `sys.acl_permissions_history` +- `sys.acl_rest_tokens` + +Only these system tables need this column; do not add it to user-created tables. + When a newer server restores a backup taken before the ACL audit log was -introduced, startup leaves the restored ACL and credential tables unchanged: -it neither adds their `change_id` columns nor creates the missing audit log. +introduced, startup leaves these restored system tables unchanged: it neither +adds their `change_id` columns nor creates the missing audit log. This allows [forward restore updates](#retrying-or-updating-a-restore) while the source still runs the older version. The log and columns arrive with the first restored backup taken after the source is upgraded. @@ -613,9 +627,15 @@ the restored storage policy tables. For a standalone recovery that will receive no further restore updates, such as recovery into a new cluster: -1. As the built-in admin, add `change_id TIMESTAMP` to each restored table - that lacks it. The server logs the affected tables and their exact - `ALTER TABLE ... ADD COLUMN change_id TIMESTAMP` statements at startup. +1. As the built-in admin, run the `ALTER TABLE` statements in the startup log + to add `change_id TIMESTAMP` to the system tables listed above that lack + it. The log identifies each affected table and the exact statement to run. + For example, if it identifies `sys.acl_permissions`: + + ```questdb-sql + ALTER TABLE 'sys.acl_permissions' ADD COLUMN change_id TIMESTAMP; + ``` + Restarting alone does not add these columns, because restore markers stay on the tables. 2. Once the audit log exists, check its policy with From 5f0efa3002821065ca1d594b1fc55c0aa8c2b1dc Mon Sep 17 00:00:00 2001 From: glasstiger Date: Fri, 6 Nov 2026 14:37:07 +0000 Subject: [PATCH 10/10] docs: address ACL audit log review for Enterprise 4.2.0 --- documentation/changelog.mdx | 8 +-- documentation/configuration/iam.md | 7 +-- documentation/high-availability/overview.md | 6 +-- documentation/operations/backup.md | 21 +++----- documentation/operations/upgrade.md | 50 +++++++------------ .../query/sql/acl/create-service-account.md | 14 +++--- documentation/security/acl-audit-log.md | 48 +++++++++--------- documentation/security/governance.md | 8 +-- documentation/security/rbac.md | 25 +++++----- 9 files changed, 83 insertions(+), 104 deletions(-) diff --git a/documentation/changelog.mdx b/documentation/changelog.mdx index 614534959..926188a4d 100644 --- a/documentation/changelog.mdx +++ b/documentation/changelog.mdx @@ -12,7 +12,7 @@ This page tracks significant updates to the QuestDB documentation. ### New -- [ACL audit log](/docs/security/acl-audit-log/) - Documented the upcoming Enterprise audit log for access control changes, including principal attribution, read permissions, redaction, delivery guarantees, WAL availability troubleshooting, change-ID recovery limits, replication, and retention. The feature is not available in Enterprise 4.1.0 or earlier +- [ACL audit log](/docs/security/acl-audit-log/) - Documented the access control audit log introduced in QuestDB Enterprise 4.2.0, including principal attribution, read permissions, redaction, delivery guarantees, WAL availability troubleshooting, change-ID recovery limits, replication, and retention - [Expiring rows (`EXPIRE ROWS`)](/docs/concepts/expire-rows/) and [ALTER MATERIALIZED VIEW SET EXPIRE](/docs/query/sql/alter-mat-view-set-expire/) - Keep only some rows in a materialized view. Set it when you create the view (see [CREATE MATERIALIZED VIEW](/docs/query/sql/create-mat-view/)) or add it later. Queries hide expired rows right away, and a background job frees their disk space for the policies where that is safe. - [GROUPING SETS, ROLLUP, and CUBE](/docs/query/sql/grouping-sets/) - Compute detail rows, subtotals, and a grand total in a single scan, with `GROUP BY` or `SAMPLE BY`. Covers composite syntax, `GROUPING()` and `GROUPING_ID()`, `FILL` support, parallel execution, when a `UNION ALL` of plain `GROUP BY` queries is faster, and the limitations, plus the [`cairo.sql.max.grouping.sets`](/docs/configuration/cairo-engine/#cairosqlmaxgroupingsets) setting - [Reports with subtotals and grand totals](/docs/cookbook/sql/advanced/subtotals-and-grand-totals/) - Cookbook recipe for a desk volume report with per-ECN subtotals and a grand total from `ROLLUP`, labelled with `GROUPING()`, plus totals in both directions with `CUBE` and hourly totals with `SAMPLE BY` @@ -22,7 +22,7 @@ This page tracks significant updates to the QuestDB documentation. - Added [`qwp.min.idle.ms.before.writer.release`](/docs/configuration/qwp/#qwpminidlemsbeforewriterrelease), which sets how long an idle QWP WebSocket ingestion connection holds its table writers before QuestDB releases them - [`acl.oidc.pkce.required`](/docs/configuration/oidc/#acloidcpkcerequired) - Corrected the PKCE setting name. The page listed `acl.oidc.pkce.enabled`, which the server rejects as unrecognized, leaving PKCE at its default of `true` -- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented the upcoming `acl.audit.storage.policy` setting, its creation-only behavior, partition-start-based conversion and deletion eligibility, supported stages, and built-in-admin requirements for changing an existing log's policy +- [IAM configuration](/docs/configuration/iam/#aclauditstoragepolicy) - Documented `acl.audit.storage.policy`, introduced in QuestDB Enterprise 4.2.0, including its creation-only behavior, partition-start-based conversion and deletion eligibility, supported stages, and built-in-admin requirements for changing an existing log's policy ### Updated @@ -47,8 +47,8 @@ This page tracks significant updates to the QuestDB documentation. capped set. Likewise, move a window function into the query you run against the view. -- [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the upcoming change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation). This behavior is not available in Enterprise 4.1.0 or earlier -- [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the upcoming ACL audit log +- [CREATE SERVICE ACCOUNT](/docs/query/sql/acl/create-service-account/) - Documented the QuestDB Enterprise 4.2.0 change that leaves an existing account unchanged under `IF NOT EXISTS`, with explicit assignment through `GRANT ASSUME SERVICE ACCOUNT` and [upgrade guidance](/docs/operations/upgrade/#service-account-creation) +- [Backup and restore](/docs/operations/backup/#retrying-or-updating-a-restore) - Distinguished fresh restores from same-lineage retries and forward updates, with their procedures and restrictions, and documented [access control recovery](/docs/operations/backup/#restoring-access-control-tables) from backups taken before the ACL audit log was introduced in QuestDB Enterprise 4.2.0 - [Governance and auditing](/docs/security/governance/#audit-access-control-changes) - Added an ACL audit query covering ordinary grants and revokes as well as service-account assumption assignments and revocations - [Storage policy](/docs/concepts/storage-policy/#ttl-evaluation) - Corrected partition-start eligibility across audit retention, TTL migration, Parquet conversion, and Enterprise quick-start guidance, with partition allowances for minimum timestamp-based retention and clearer governance example thresholds diff --git a/documentation/configuration/iam.md b/documentation/configuration/iam.md index 1619a460c..8b8d0faf1 100644 --- a/documentation/configuration/iam.md +++ b/documentation/configuration/iam.md @@ -101,12 +101,7 @@ enabled to enforce write protections on system-managed tables, including the - **Default**: `TO PARQUET 1d` - **Reloadable**: no -:::note Upcoming Enterprise setting - -This setting belongs to the upcoming ACL audit log feature. It is not available -in QuestDB Enterprise 4.1.0 or earlier. - -::: +Available in QuestDB Enterprise 4.2.0 and later. [Storage policy](/docs/concepts/storage-policy/) used when the server creates `sys.acl_audit_log`. The value is a comma-separated list of stages, without diff --git a/documentation/high-availability/overview.md b/documentation/high-availability/overview.md index 3642e6995..4570bcb9e 100644 --- a/documentation/high-availability/overview.md +++ b/documentation/high-availability/overview.md @@ -106,9 +106,9 @@ The audit trail of [audited views](/docs/security/audited-views/), but whose rows do not: each node records the reads it serves. See [Replication, backups, and read-only instances](/docs/security/audited-views/#replication-backups-and-read-only-instances). -The upcoming [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, -is not available in Enterprise 4.1.0 or earlier. It differs from the view audit -trail: both its definition and rows replicate. +The [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, is +available in QuestDB Enterprise 4.2.0 and later. Unlike the view audit trail, +both its definition and rows replicate. A replica promoted before receiving the table creates it using its own [`acl.audit.storage.policy`](/docs/configuration/iam/#aclauditstoragepolicy) setting, so configure that property on every node before upgrading. diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 5bde77391..29c77e6be 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -585,17 +585,10 @@ the existing recovery copy. #### Restoring access control tables -:::note Upcoming Enterprise feature - -The ACL audit log is not available in QuestDB Enterprise 4.1.0 or earlier. This -section describes restore behavior on the upcoming release that introduces it. - -::: - -A backup includes the [ACL audit log](/docs/security/acl-audit-log/), -`sys.acl_audit_log`, if it exists on the source. Its rows replicate, unlike -those of `sys.view_audit`, so a primary backup includes the replicated ACL -change history. +The [ACL audit log](/docs/security/acl-audit-log/), `sys.acl_audit_log`, is +available in QuestDB Enterprise 4.2.0 and later. A backup includes it if it +exists on the source. Its rows replicate, unlike those of `sys.view_audit`, so +a primary backup includes the replicated ACL change history. The following internal ACL and credential tables use a `change_id TIMESTAMP` column to link their rows to audit entries: @@ -611,12 +604,12 @@ column to link their rows to audit entries: Only these system tables need this column; do not add it to user-created tables. -When a newer server restores a backup taken before the ACL audit log was -introduced, startup leaves these restored system tables unchanged: it neither +When QuestDB Enterprise 4.2.0 or later restores a backup taken on a release +before 4.2.0, startup leaves these restored system tables unchanged: it neither adds their `change_id` columns nor creates the missing audit log. This allows [forward restore updates](#retrying-or-updating-a-restore) while the source still runs the older version. The log and columns arrive with the first -restored backup taken after the source is upgraded. +restored backup taken after the source is upgraded to 4.2.0 or later. Do not run ACL-changing statements on a node that must keep accepting restore updates. Such a statement changes the restored tables, and the next update diff --git a/documentation/operations/upgrade.md b/documentation/operations/upgrade.md index c28200bf7..531033549 100644 --- a/documentation/operations/upgrade.md +++ b/documentation/operations/upgrade.md @@ -101,19 +101,14 @@ Others, such as accounts that triggered role switches through `SYSTEM ADMIN` on [Failover operator](/docs/security/rbac/#failover-operator) and [Database administrator](/docs/security/rbac/#database-administrator). -### ACL audit log +### Upgrading to 4.2.0 -:::note Upcoming Enterprise feature +#### ACL audit log -This section describes an unreleased feature and its migrations. The ACL audit -log is not available in QuestDB Enterprise 4.1.0 or earlier. These instructions -apply when upgrading to a release that includes it. - -::: - -When upgrading from a version without the -[ACL audit log](/docs/security/acl-audit-log/), the server creates -`sys.acl_audit_log` and adds `change_id` columns to ACL and credential tables. +QuestDB Enterprise 4.2.0 introduces the +[ACL audit log](/docs/security/acl-audit-log/). When upgrading from an earlier +release, the server creates `sys.acl_audit_log` and adds `change_id` columns to +ACL and credential tables. Existing rows keep a `NULL` `change_id`; past changes are not reconstructed. Review the following before upgrading: @@ -148,12 +143,13 @@ not to a valid audit log. :::warning Rollback limitations -An older release does not enforce the audit log's special read or write -protections. It treats the log as an ordinary table, so existing broad table -grants may expose or allow changes to audit evidence. A later upgrade reuses -a structurally compatible log, including any rows written during the rollback. +Releases before QuestDB Enterprise 4.2.0 do not enforce the audit log's special +read or write protections. They treat the log as an ordinary table, so existing +broad table grants may expose or allow changes to audit evidence. A later +upgrade reuses a structurally compatible log, including any rows written during +the rollback. -An older release also cannot compact `sys.acl_permissions` after it gains +These releases also cannot compact `sys.acl_permissions` after it gains `change_id`. Compaction reports `unexpected acl permissions column count, refusing to compact` and is disabled until the next restart. Plan rollback around your pre-upgrade backup @@ -161,7 +157,7 @@ rather than assuming the migrated database behaves as it did before. ::: -#### Embedded Enterprise Java integrations +##### Embedded Enterprise Java integrations Applications that directly implement Enterprise Java APIs must update and rebuild: @@ -175,22 +171,14 @@ rebuild: and must not change when the session assumes a service account. Forwarding contexts must forward this method too. -### Service account creation - -:::note Upcoming Enterprise behavior change - -The behavior described here is unreleased. It does not apply to QuestDB -Enterprise 4.1.0 or earlier. - -::: +#### Service account creation -In the upcoming release, +In QuestDB Enterprise 4.2.0 and later, [`CREATE SERVICE ACCOUNT IF NOT EXISTS`](/docs/query/sql/acl/create-service-account/) -leaves an existing account unchanged. In Enterprise 4.1.0 and earlier, the -statement could assign the existing account to the statement's owner with grant -option. With the new behavior, ownership is assigned only when the statement -creates the account. An invalid owner is rejected before any changes, even when -the account already exists. +leaves an existing account unchanged. Before 4.2.0, the statement could assign +an existing account to the statement's owner with grant option. Ownership is +now assigned only when the statement creates the account. An invalid owner is +rejected before any changes, even when the account already exists. Update provisioning scripts that relied on `IF NOT EXISTS ... OWNED BY` to assign an existing account. Use diff --git a/documentation/query/sql/acl/create-service-account.md b/documentation/query/sql/acl/create-service-account.md index 1377d7a70..216467140 100644 --- a/documentation/query/sql/acl/create-service-account.md +++ b/documentation/query/sql/acl/create-service-account.md @@ -35,13 +35,13 @@ each of the service account's queries may allocate, use [`ALTER SERVICE ACCOUNT ... SET MEMORY LIMIT`](/docs/query/sql/acl/alter-service-account-set-memory-limit/) after creating it. Service accounts do not inherit group limits. -:::note Upcoming behavior change +:::note Behavior change in QuestDB Enterprise 4.2.0 -The `IF NOT EXISTS` and owner-validation behavior described below is unreleased. -In QuestDB Enterprise 4.1.0 and earlier, `IF NOT EXISTS` could assign an existing -account to the statement's owner rather than leave it unchanged. See the +The `IF NOT EXISTS` and owner-validation behavior described below applies to +QuestDB Enterprise 4.2.0 and later. Before 4.2.0, `IF NOT EXISTS` could assign an +existing account to the statement's owner rather than leave it unchanged. See the [service account upgrade guidance](/docs/operations/upgrade/#service-account-creation) -before relying on the new behavior in provisioning scripts. +before relying on this behavior in provisioning scripts. ::: @@ -78,8 +78,8 @@ CREATE SERVICE ACCOUNT risk_report; CREATE SERVICE ACCOUNT IF NOT EXISTS risk_report; ``` -With the upcoming behavior, the second statement leaves the existing account -unchanged. +In QuestDB Enterprise 4.2.0 and later, the second statement leaves the existing +account unchanged. ```questdb-sql title="Create an account owned by a group" CREATE GROUP analysts; diff --git a/documentation/security/acl-audit-log.md b/documentation/security/acl-audit-log.md index 9eede8641..4c3b740b7 100644 --- a/documentation/security/acl-audit-log.md +++ b/documentation/security/acl-audit-log.md @@ -2,24 +2,20 @@ title: ACL audit log sidebar_label: ACL audit log description: - Preview the upcoming ACL audit log in QuestDB Enterprise, with principal - attribution for access control changes and configurable retention. + Track access control changes with the ACL audit log in QuestDB Enterprise + 4.2.0 and later, including principal attribution and configurable retention. --- -The upcoming ACL audit log records changes to access control in -`sys.acl_audit_log`, so you can investigate who granted access, changed -credentials, or modified a principal. Each row identifies the authenticated -principal, the principal it acted as, and the statement that caused the change. +In [QuestDB Enterprise](/enterprise/) 4.2.0 and later, the ACL audit log records +changes to access control in `sys.acl_audit_log`, so you can investigate who +granted access, changed credentials, or modified a principal. Each row identifies +the authenticated principal, the principal it acted as, and the statement that +caused the change, when available. See [Recorded changes](#recorded-changes) for +cases where these fields are `NULL`. -:::note Upcoming Enterprise feature - -This page describes an unreleased [QuestDB Enterprise](/enterprise/) feature. -The ACL audit log and `acl.audit.storage.policy` are not available in Enterprise -4.1.0 or earlier. For read auditing rather than access control changes, see +For read auditing rather than access control changes, see [Audited views](/docs/security/audited-views/). -::: - ## Read the log Reading the log requires all three permissions: @@ -81,6 +77,12 @@ WHERE effective_principal = 'risk_admin' ORDER BY change_id DESC; ``` +This query detects assumptions only when the principal names differ. An +external user can assume a local service account with the same name; those +cases are not returned. `is_external` describes the authenticated identity, +not whether a service account was assumed, so these columns alone cannot +conclusively distinguish same-name assumptions. + ## Recorded changes The log covers changes to users, groups, service accounts, group membership, @@ -301,19 +303,19 @@ receiving the log creates its own, using its local `acl.audit.storage.policy` setting. An ACL statement that finds the log missing during promotion creates it with the same policy. -On an ordinary upgrade, the server creates the log and adds `change_id` to -existing ACL and credential tables. It does not reconstruct past changes. Tables -with suspended WAL apply get the column at a later startup after they are -resumed. Until the permissions history table has the column, compaction archives -rows without their change IDs. +When upgrading from a release before QuestDB Enterprise 4.2.0, the server +creates the log and adds `change_id` to existing ACL and credential tables. +It does not reconstruct past changes. Tables with suspended WAL apply get the +column at a later startup after they are resumed. Until the permissions history +table has the column, compaction archives rows without their change IDs. -A backup taken before the log was introduced is handled differently: startup -neither creates the missing log nor adds `change_id` to restored tables. This -allows continued restore updates from an older source. See +A backup taken on a release before QuestDB Enterprise 4.2.0 is handled +differently: startup neither creates the missing log nor adds `change_id` to +restored tables. This allows continued restore updates from an older source. See [Restoring access control tables](/docs/operations/backup/#restoring-access-control-tables) for when the log and columns appear, and what to do after the final restore. Before upgrading or rolling back, review the [ACL audit log upgrade considerations](/docs/operations/upgrade/#acl-audit-log). -Older versions do not enforce the log's protections and cannot compact the -permissions table after it gains `change_id`. +Releases before 4.2.0 do not enforce the log's protections and cannot compact +the permissions table after it gains `change_id`. diff --git a/documentation/security/governance.md b/documentation/security/governance.md index 3282dea5d..3c8d580c1 100644 --- a/documentation/security/governance.md +++ b/documentation/security/governance.md @@ -26,7 +26,7 @@ available in all editions. | What may they even see? | [Object visibility](/docs/security/rbac/#object-visibility) | QuestDB Enterprise 4.1.0+ | | How much may they consume? | [Resource groups](/docs/concepts/resource-groups/) and [per-principal memory limits](/docs/security/rbac/#memory-limits) | QuestDB Enterprise 4.1.0+ | | Who can do what, right now? | [Access control functions](/docs/query/functions/access-control/), [`SHOW CREATE DATABASE`](/docs/query/sql/show/#enterprise-access-control) | QuestDB Enterprise | -| Who changed access? | [ACL audit log](/docs/security/acl-audit-log/) | QuestDB Enterprise (upcoming) | +| Who changed access? | [ACL audit log](/docs/security/acl-audit-log/) | QuestDB Enterprise 4.2.0+ | | Who read this data? | [Audited views](/docs/security/audited-views/) | QuestDB Enterprise | | What queries ran recently? | [Query tracing](/docs/concepts/deep-dive/query-tracing/), [`query_activity()`](/docs/query/functions/meta/#query_activity) | All | | Where does this data come from? | [`SHOW CREATE`](/docs/query/sql/show/), [`views()`](/docs/query/functions/meta/#views), [`materialized_views()`](/docs/query/functions/meta/#materialized_views) | All | @@ -142,9 +142,9 @@ can diff between two reviews. ## Audit access control changes -The [ACL audit log](/docs/security/acl-audit-log/) is an upcoming Enterprise -feature, unavailable in Enterprise 4.1.0 or earlier. It records changes to users, -groups, service accounts, credentials, and permissions in `sys.acl_audit_log`. +The [ACL audit log](/docs/security/acl-audit-log/) is available in QuestDB +Enterprise 4.2.0 and later. It records changes to users, groups, service accounts, +credentials, and permissions in `sys.acl_audit_log`. It identifies both the authenticated principal and any assumed service account, with passwords and REST token values redacted from statement text. Unlike the view audit trail, its rows replicate. diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 5a1bfe942..de33810f4 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -532,13 +532,13 @@ get it back without someone re-granting it. ## Audit access control changes -The [ACL audit log](/docs/security/acl-audit-log/) is an upcoming Enterprise -feature, unavailable in Enterprise 4.1.0 or earlier. It records who changes -users, groups, service accounts, credentials, and permissions, including both the -session principal and any assumed service account. Query `sys.acl_audit_log` -with `LIST USERS`, `USER DETAILS`, and `SELECT` on the table; `SELECT ON ALL -TABLES` alone does not grant access. Only the built-in admin can change its -retention policy or remove rows. +The [ACL audit log](/docs/security/acl-audit-log/) is available in QuestDB +Enterprise 4.2.0 and later. It records who changes users, groups, service accounts, +credentials, and permissions, including both the session principal and any +assumed service account. Query `sys.acl_audit_log` with `LIST USERS`, +`USER DETAILS`, and `SELECT` on the table; `SELECT ON ALL TABLES` alone does +not grant access. Only the built-in admin can change its retention policy or +remove rows. ## Object visibility {#object-visibility} @@ -594,11 +594,12 @@ and `wait_wal_table()`, and WAL recovery statements such as [`ALTER TABLE RESUME WAL`](/docs/query/sql/alter-table-resume-wal/) reach them for principals that hold the permission the statement requires. -The upcoming [ACL audit log](/docs/security/acl-audit-log/#read-the-log), -`sys.acl_audit_log`, is a separate exception: it requires both `LIST USERS` -and `USER DETAILS` in addition to ordinary table permissions. Without both, -it looks missing even to a principal with a grant `ON ALL TABLES`. Unlike -the protected ACL tables, its rows can be read by an authorized auditor. +The [ACL audit log](/docs/security/acl-audit-log/#read-the-log), +`sys.acl_audit_log`, is available in QuestDB Enterprise 4.2.0 and later. It is a +separate exception: it requires both `LIST USERS` and `USER DETAILS` in addition +to ordinary table permissions. Without both, it looks missing even to a +principal with a grant `ON ALL TABLES`. Unlike the protected ACL tables, its +rows can be read by an authorized auditor. Permissions count from every source the principal is authorized with: its own grants and those of its groups or, after