QuestDB Enterprise 4.2.1 brings DEDUP SKIP and adaptive commit durability, along with an ACL audit log, a faster Parquet path for cold storage, and safer replication and backup index writes. It includes everything in QuestDB 10.2.0 plus the fixes merged to OSS since.
Breaking changes09
- enterprise
sys.view_audit,sys.sp_entriesandsys.sp_linksare now system-managed tables. Reading them remains an ordinarySELECTgrant, but writing their rows is refused to every principal, including the built-in admin. Other changes apart from WAL control, includingTRUNCATE,DROP TABLE,RENAME TABLE,VACUUM TABLE,REINDEX, otherALTER TABLEstatements and storage policy statements, are refused to every principal except the built-in admin; grants such asON ALL TABLES,ALL,DATABASE ADMINandSYSTEM ADMINno longer allow them. In 4.1.0 these tables took ordinary table permissions.Refused statements fail with
only the built-in admin can modify a system-managed tableorcannot write to a system-managed table, only the server writes it. WAL control keeps its usual permissions on the storage policy tables; on the audit trail,SUSPEND WALandRESUME WAL FROM TXNstay refused to every principal, as in 4.1.0.With the built-in admin disabled (
acl.admin.user.enabled=false), nobody can change these tables apart from WAL control on the storage policy tables. Such deployments set the trail's retention up front, with aDROP LOCALstage inview.audit.storage.policy, or withALTER TABLE ... SET STORAGE POLICYbefore upgrading.This change prevents table grants from being used to erase the audit trail, for example via
TRUNCATE TABLE ON ALL TABLESor by insertingDROP LOCALstorage policy rows. On a replica, the emergency path letting aSYSTEM ADMINholder force ALTERs on a suspended table no longer reaches these tables. - enterprise
SHOW CREATE DATABASEnow dumps users, groups, service accounts, memberships and grants for replay, and database dumps include table storage policies. Dumps omit credentials, memory limits and resource groups, and ACL dumps are best-effort rather than a transactionally consistent snapshot across entities.Plain
SHOW CREATE DATABASEnow requiresLIST USERSandUSER DETAILSwhen ACL is on; callers without them can dump the visible schema withINCLUDE (SCHEMA). Materialized and live views accept only schema-onlySHOW CREATE DATABASE. Views now reject administrative functions and ACL dumps inPIVOT INsubqueries when created or altered, and existing such views turn invalid at their next compile;INCLUDE (SCHEMA)repairs ACL dumps.SHOW CREATE TABLEno longer omits the TTL of tables that still carry one, and replay turns it into aDROP LOCALstorage policy.SHOW CREATEoutput no longer appends anOWNED BYclause naming the caller. A new
EXPIRE ROWSpolicy on materialized views hides expired rows from every query. Policy modes includeWHEN <predicate>,KEEP LATEST,KEEP [N] HIGHEST|LOWESTand window predicates; a background job frees disk for monotonicWHENpolicies such asts < now(). Passthrough views such asCREATE MATERIALIZED VIEW v AS (SELECT * FROM base)mirror a table.ALTER MATERIALIZED VIEW ... SET EXPIRE ROWSand... DROP EXPIREchange a view's policy,SHOW CREATEandmaterialized_views()report it, andcairo.mat.view.row.expiry.cleanup.*settings control the cleanup job.Breaking:
CREATEnow rejectsLIMITin queries that read the base table, and rejects base-table window functions nested in expressions orSAMPLE BYsubqueries. Replaying olderSHOW CREATEoutput with suchLIMITor window functions fails.This change also fixes a cached re-partitioning
COPYfailing withtable does not existwhen rerun, refresh workers spinning at full CPU while another refresh holds a view's lock, and refreshes of ranges emptied byREPLACErunning one query perSAMPLE BYbucket. Views readingSELECT * FROM <view>now go invalid whenALTER VIEWreorders its columns.Tradeoffs include:
KEEPand window policies rank the whole view on every read; only monotonicWHENpolicies free disk; passthrough views inherit baseSYMBOLindexes, slowing ingestion and adding disk; materialized and live views cannot read a view that has anEXPIRE ROWSpolicy; and downgrading, then running DDL on a policied view, silently drops its policy.CASEandcoalesce()no longer fail mid-query withUnsupportedOperationExceptionwhen a CHAR or SYMBOL branch is combined with IPv4 or UUID. Such queries now fail to compile withinconvertible typeseven when that branch is never read, as the two-branch form andINSERT ... SELECTalready do. An explicit cast such assym::IPv4makes such a query work.CHAR followed by SYMBOL now maps to STRING, like SYMBOL followed by CHAR and the other text pairs. As a result,
EXPLAINshowsch::stringinstead ofch::symbol, and with CHAR, SYMBOL, then an unconvertible type, the error names STRING, not SYMBOL.GROUP BYandmin()/max()overcoalesce(char, symbol)can now run in parallel.New or altered views now reject administrative functions and administrative
SHOWstatements inPIVOT INsubqueries. Existing views of this kind become invalid at their next compile, including at startup.Materialized and live views now reject administrative
SHOWstatements, as they already do administrative functions. Rejections from a view'sPIVOT INsubquery point at the view's name in the statement.For extensions,
SHOW CREATEcursors dropputAdditional(), andFunctionParser.addObjectDisclosingStatement()is renamed toaddStatementRequirements().QWP now rejects table blocks that carry the designated timestamp twice, once through
at(...)and once as a named column. Before this change, QuestDB accepted them and stored wrong timestamps.A sender that uses both forms on one table now gets a schema error on every later batch for that table.
An unknown
cairo.commit.modevalue now fails startup, instead of silently selectingnosync.After 4.2.1 has run on a database, downgrading will need additional steps.
After using
DEDUP SKIP, older versions treatSKIPtables asUPSERTwithout an error, and suspend WAL apply when they replay a loggedALTER TABLE ... DEDUP SKIP.
New features08
- enterprise
Adaptive commit mode makes the primary's WAL durable while applying table state lazily. QuestDB Enterprise preserves this contract through uploader startup, promotion/demotion, backup and durable acknowledgements. A live replica resolves configured ADAPTIVE to NOSYNC for applied-state and downloaded-log flush decisions, while explicit SYNC stays SYNC.
In ADAPTIVE and SYNC, uploads wait for the local durable frontier, so the uploader cannot publish a transaction number the primary may reuse after losing an unflushed sequencer tail. Partition purge after a replica promotion is now enabled: a replica promoted to primary can purge superseded partitions once its own recovery authority is durable.
The durable-ACK registry composes local and replicated tiers on an eligible adaptive primary.
The shipped default remains NOSYNC. The replication-aware default is conditioned on replication configuration and never overrides an explicit mode. No Enterprise on-disk format is added.
Going back to 4.1.x works. If adaptive commit durability (
cairo.commit.mode=adaptive) was enabled, restart once withnosyncbefore downgrading.Known limitations: on an ADAPTIVE replica, a power loss can lose the view audit table's most recent records and leave it suspended with no working manual step, and an Enterprise backup can ship records past the local durable frontier, as in NOSYNC.
- enterprise
The storage policy no longer converts partitions itself;
PartitionCompactionScanJobdoes all native-to-Parquet work.TO PARQUETasks the compaction job to convert the logical partition (plain, composite or split) into one Parquet file in a single pass with a new frame-based Parquet writer, replacing the Parquet rewrite that hard-linked stale index files.An uploaded (REMOTE) native partition keeps REMOTE when every converted folder was REMOTE.
Non-WAL tables now reach the compaction job. Upload while native asks the compaction job for a Parquet copy, built off a reader and installed beside the native files after the partition is re-checked; a split or composite partition is folded first. A partition still due for
TO PARQUETis not sealed in the same pass.A
cairo.read.onlyinstance runs no storage policy jobs, while replicas and demoted nodes still enforce the policy. This change also fixes the writer's stray-directory purge deleting an in-flight.generatingdirectory of a folder with no name txn suffix.Frame conversion writes an absent column as nulls with column top 0, while CONVERT keeps the full top. A refused swap leaves its in-flight record until the idle timeout, and a non-WAL writer whose transactions keep crossing partition boundaries can delay a swap.
- enterprise
This feature brings the
EXPIRE ROWScapability (passthrough materialized views and row-level retention) to QuestDB Enterprise. The background cleanup job runs on the primary only, and theREPLACE_RANGEreclamation it performs ships to replicas as ordinary WAL transactions, so replicas reclaim identical rows.The job checks the node's role on every sweep, so a promoted replica starts reclaiming without a restart.
OWNED BYnow ends anEXPIRE ROWSclause, soCREATE MATERIALIZED VIEW … EXPIRE ROWS … OWNED BY …parses for every policy form,SHOW CREATE MATERIALIZED VIEWoutput can be re-executed, and external users can create materialized views with a policy. The expiry policy, passthrough flag, covering-index columns and stored view query are now forwarded onCREATEinstead of being dropped.No new permission is introduced.
ALTER MATERIALIZED VIEW … SET EXPIRE ROWS …and… DROP EXPIRErequireSET TABLE PARAMon the view, andSET EXPIRE ROWSalso requiresSELECTon every column the policy reads. Reading a materialized view with a policy requiresSELECTon the columns the policy reads, so setting a policy can reduce what users with column-level grants can read.After a policy is set or replaced, existing SQL views over the materialized view deny users who hold only the view grant; access returns once the view is re-issued with
ALTER VIEW … AS (…), whileCOMPILE VIEWdoes not restore it.ALTER MATERIALIZED VIEW … DROP STORAGE POLICYis still rejected. - enterprise
This feature records each ACL change in
sys.acl_audit_log: both principals, an external-identity flag, origin, statement kind and statement text, with passwords and REST tokens shown as'****'. A newchange_idcolumn on ACL and credential tables links rows to their audit row. Reading the log requiresLIST USERSandUSER DETAILSin addition toSELECT, and nobody can write its rows.The new
acl.audit.storage.policysetting (defaultTO PARQUET 1d) sets the log's storage policy whenever a node creates it;TO REMOTEandDROP REMOTEare refused. The log replicates, and a replica promoted before receiving it creates its own. At startup, an existingsys.acl_audit_logtable with the log's columns becomes the log, while any other object of that name fails startup.On a database restored from a backup taken before this release, the server neither creates the log nor adds
change_idto restored tables; both arrive with the first backup the source takes on this release. An earlier release cannot compactsys.acl_permissionsonce it haschange_id.This change also fixes credential changes reverting on restart or replicas after a backward clock step, and
DROP TOKEN TYPE RESTnaming another user's token disabling it until restart.CREATE SERVICE ACCOUNT IF NOT EXISTSno longer assigns an existing service account to the statement's owner. In the Java API, ACL writers and credential stores take anAclChangeas the first argument, andEntSecurityContextimplementations must implementisSessionPrincipalExternal(). - enterprise
This feature brings the
SKIPdedup mode to QuestDB Enterprise. Previously,CREATE TABLE ... DEDUP SKIP KEYS(...), CTAS withDEDUP SKIP, andCREATE TABLE ... (LIKE t)of aSKIPtable created anUPSERTtable on the primary and its replicas.The dedup mode is now correctly forwarded at table creation, so all three forms create
SKIPtables.ALTER TABLE ... DEDUP ENABLE SKIPwas not affected. This feature flushes the writer's private WAL before sequencing, applies WAL to table files without a per-apply column flush, and periodically saves materialized state (
_txn,_cv,_meta) into_epoch.manifest.0/_epoch.manifest.1save points. Recovery restores a validated generation and replays retained WAL.The shipped default stays
nosync, and the mode is instance-wide, changed between restarts.For WAL tables on a standalone server or QuestDB Enterprise primary, assuming storage honours flushes: with a group window of 0, acknowledged WAL commits survive power loss; with a nonzero window (default 50 ms), only transactions covered by the local durable frontier survive. Non-WAL tables keep their legacy durability, and legacy line UDP into adaptive WAL tables is refused.
New configuration properties:
cairo.adaptive.epoch.interval,cairo.adaptive.epoch.max.rows,cairo.adaptive.commit.group.window,cairo.adaptive.epoch.flush.on.close,cairo.adaptive.epoch.column.sync.batchedandcairo.wal.commit.writeback.drain.wal_tables()gainslocalDurableSeqTxn,durableEpochSeqTxn,recoveryIncarnationandlastEpochTscolumns.X-QWP-Request-Durable-Ackacceptslocal,replicatedorlocal,replicated, withSTATUS_LOCAL_DURABLE_ACKreporting local durability.In SYNC and ADAPTIVE, a failed sync call now suspends the affected table; a standalone server halts and an embedded engine fences itself. Other changes in all modes: an unknown
cairo.commit.modenow fails startup instead of silently selecting NOSYNC,DETACHuses a temporary<partition>.<txn>.detachingdirectory renamed atomically to.detached, and the lag checksum overwrite and REBASE seed timestamp defects are fixed. Downgrade, write with an older release, then re-upgrade is unsupported.GROUP BYsupportsROLLUP,CUBEandGROUPING SETS, withGROUPINGandGROUPING_ID.SAMPLE BYsupports them too, withFILLas NULL, constants or NONE. Queries run in parallel where the plan allows (requiring sets × (query workers + 1) ≤ 256), otherwise serially; grouping sets never vectorize.An explicit empty grouping set returns a grand-total row, even when no rows match. The new
cairo.sql.max.grouping.setsproperty caps how many grouping sets one query expands to (default 4096).Limitations:
GROUPING SETS ()and trailing commas are syntax errors, each element needs parentheses, and plain keys may precedeROLLUPorCUBEbut notGROUPING SETS.SAMPLE BYgrouping sets rejectFILL(PREV),FILL(LINEAR),ALIGN TO FIRST OBSERVATION, and non-constant interval orFROMbounds.GROUPING_IDtakes at most 31 arguments, and these constructs fail withHORIZON JOIN, in materialized views, and when correlated insideLATERAL.The new
DEDUP SKIPmode keeps the first-written row and drops later duplicates.CREATE TABLE, CTAS andALTER TABLE ... DEDUPacceptSKIP, while materialized views keepUPSERT.SHOW CREATE TABLEprints the mode,tables()shows only whether a table deduplicates,CREATE TABLE ... LIKEcopies the mode, andDEDUP DISABLEresets it toUPSERT.SKIPcommits holding only duplicates leave native and Parquet partitions untouched. Commits that only resend stored rows no longer rebuild the partition's POSTING index, and resending unchanged rows intoDEDUP UPSERTParquet partitions skips the rewrite (including resent row groups in mixed commits).This change also includes many fixes:
SET TYPE WALandREBASE WALno longer drop TTL,FORMAT PARQUETor Parquet encoding;REBASE WALno longer suspends tables with non-timestamp dedup keys; dedup keys on columns added after Parquet conversion no longer suspend WAL apply; index scans andCONVERT TO NATIVEon Parquet partitions from 10.0.1 or older return correct rows; NULL handling for BOOLEAN/BYTE/SHORT columns across Parquet conversion andALTER COLUMN TYPEis corrected;VACUUM TABLEnow removes POSTING files of dropped columns and dropped or changed indexes; and a failed commit no longer makes new queries wait until the writer reopens.A downgrade turns
SKIPtables intoUPSERTwithout an error, and older versions suspend WAL apply on a loggedALTER TABLE ... DEDUP SKIP.
Improvements02
With async munmap enabled (
cairo.file.async.munmap.enabled, off by default),MmapCachenow treats the async munmap queue as eviction hints rather than unmap commands. When the last user releases a file's current cached mapping, it stays cached with a zero reference count and a hint is queued; a lookup of the same file range before the hint is consumed revives the mapping without a newmmap.Superseded mappings, async off, or a full queue retire the mapping at once, with a synchronous munmap when the queue is full. Memory accounting is recorded only on the actual native
munmap, and the uncachedmunmapmoved out of the lock.No delay, TTL or LRU is added: a mapping survives only until the consumer reaches its hint. Measured on composite-partition WAL block apply, this alone gave zero reuse; any gain comes from code paths that release and re-request the same file range shortly after.
One-row QWP messages ingest at about 3.9M rows/s on the benchmark box (one connection, 12 cores, tmpfs), up from about 210k. 1000-row messages are unchanged within noise. The wire protocol and the meaning of acks and NACKs are unchanged.
Plain single-block frames of one receive pass now commit as one WAL txn, bounded by
qwp.max.uncommitted.rows, and are acked once after it; durable-ack connections batch too. A frame that fails inside a batch is NACKed after the earlier frames commit and ack; if the batch's own commit fails, the NACK names its first frame.Acks go out every 64 messages inside a receive batch instead of every 8, and schema parsing and column resolution are reused across identical frames.
Append files are backed by range instead of from offset 0 on every extension. The sequencer txn log opens with a 64 KiB extend segment, so
_txnlogis at least 64 KiB per WAL table.This change also fixes
wal_tables().sequencerTxnlagging the log after anALTERon a suspended WAL table; the apply job still stays asleep untilRESUME.
Bug fixes48
- enterprise
On several OIDC login failure paths, the OIDC client wrote identity-provider responses into the server log, including token-endpoint responses carrying access, refresh and ID tokens, and user-info responses or ID-token payloads carrying user claims. When the provider answered with a non-200 status, or a token endpoint request failed without a response, the log could carry leftover content from the client's reused response buffer, which could hold another user's tokens or claims. Server logs from earlier releases can therefore hold credentials.
The client now logs the status code, the parser or connection error, or the names of expected fields and claims instead of the responses. JWKS responses, which hold only public keys, are still logged when they cannot be parsed.
Operators diagnosing a claim misconfiguration no longer find the provider's response in the log; they have to query the user-info endpoint or decode the ID token outside QuestDB.
- enterprise
TokenHashertruncated each UTF-16 character to its low byte, so distinct non-ASCII credentials could share an OIDC credential-cache key. This fix hashes the UTF-8 representation while preserving byte-identical hashes for ASCII credentials and persisted ASCII REST tokens. - enterprise
ALTER USER|SERVICE ACCOUNT ... DROP TOKEN TYPE REST '<token>'wrote the live REST token to the server log, because the statement was not marked as secret and its query log line (finon success,erron a parse or authorization failure) carried the full SQL text.The compiler now marks the statement as secret as soon as it reads
DROP TOKEN TYPE REST, before parsing the token or checking authorization, so those log lines show** redacted for privacy **. The drop-all form's log line shows the same placeholder. - enterprise
Unauthenticated JWTs with unknown key IDs previously triggered a synchronous JWKS download per request. Concurrent misses or a failing provider could stall workers and amplify IdP traffic.
This fix makes reloads use one non-blocking loader and a 10-second attempt interval, or the key expiry if that is shorter. Fresh immutable key maps are published, and the loader flag is always released. The interval starts when each load finishes, and a backward wall-clock adjustment permits one refresh instead of suppressing reloads.
Concurrent misses may fail authentication rather than wait, and newly rotated keys may receive 401 responses until the interval expires. After a failed load leaves the cache empty, for example at startup while the provider is unavailable, every JWT may receive 401 responses until the interval expires.
- enterprise
A database restored from backup returned NULL from covering index queries on Parquet partitions, including the designated timestamp, while the same queries with
/*+ no_covering */were correct. The Parquet index rebuild during restore filled the covering sidecars from native column files that a Parquet partition does not have.This fix also picks up fixes for the same NULLs after
RENAME,DROP INDEXorUPDATEof a covered column.An index rebuilt by an earlier restore keeps its NULL sidecars and keeps returning NULLs after the upgrade until it is rebuilt with
DROP INDEX+ADD INDEXon the primary (REINDEXskips Parquet partitions). A new restore with this fix rebuilds it correctly. - enterprise
A transient object store error (an HTTP 500, 502, 503 or 504, or a dropped connection) no longer fails a backup. A late or retried index upload can no longer roll the backup index back, and on S3, GCS and Azure a backup index changed by another writer is no longer overwritten; that backup fails instead.
A backup whose first index upload failed now stays listed in
backups(), unless the server restarts before a later backup to that destination succeeds. Abackups()query sent while a backup start fails no longer leaves two entries for that backup, which stopped every later cleanup of the destination.A backup whose final local index write fails, for example on a full disk, now fails instead of staying listed as complete while its files are deleted.
S3-compatible backup stores must support
If-None-MatchandIf-MatchonPutObject. On GCS each index write takes two requests instead of one. In a store outage, index writes andbackups()wait about 2 minutes, not 30 s, and a restart before a lost first index upload lands can fail the next backup once. - enterprise
The primary no longer halts with ER006 when its own late index upload lands, and a late index upload can no longer roll
index.msgpackback to older contents. Another primary's index now triggers ER006 at the next write, even under a fresh lease. If a start or promotion cannot reread the index, it fails without halting the process.S3-compatible stores must support or ignore
If-MatchandIf-None-MatchonPutObject; on a store that rejects them, a primary cannot start, migrate or be promoted. On GCS each index upload takes two requests instead of one. Index upload timeouts can stretch a shutdown or role switch to four timeouts, about 40 s. - enterprise
This fix allows affected Parquet indexes to be repaired online on read-only replicas without advancing the replicated transaction sequence.
ALTER TABLE ... FORCE REINDEX PARQUETis supported through the existing internal-writer andREINDEXpermission paths, rebuilding indexes in a node-local transaction outside the WAL. - enterprise
Previously,
progress_percentinbackups()counted every partition as one equal step, so a backup with many tiny partitions and one large partition could report a high percentage quickly and then stay there while the large partition compressed, with no backup log output.Partitions now take 95% of the progress, each weighted by rows times columns from the manifest, so no file is measured. Cold partitions weigh the minimum, since they carry
_pmplus any index files but no column data. The metadata steps share the remaining 5%, andbackups()keeps the same columns.A partition whose compression outlasts
backup.activity.log.interval(default 60s) now logs progress once per interval, including read bytes, total bytes, percent and elapsed time. Faster compressions log nothing. - enterprise
S3
DeleteObjectscarries keys in an XML body, and a table name holding a control character produces a key XML 1.0 can't carry. S3 rejected the whole batch withMalformedXML, so WAL pruning failed every pass and that table's WAL was never deleted.On S3, keys XML can carry still go in one
DeleteObjectsbatch, while the others are sent as single-object DELETEs with URL-encoded keys, 8 at a time. A failed single DELETE names its key. GCS and Azure are unchanged. - enterprise
COPY <table> TOnow requiresSELECTon every column it exports. Previously,SELECTon any one column was enough, so a user granted a single column could export the whole table.The check runs again when the export starts. If a grant is narrowed, or a column is added without a grant, after the
COPYwas compiled, the export fails before any file is written.Users granted only some columns must name them:
COPY (SELECT a, b FROM t) TOA public table, such as
telemetry, can now be exported withCOPY <table> TOwithout a grant, just as it can be read withSELECT. When
INSERT ... SELECThad a column list and a selected value could not be converted, the "inconvertible types" error named the table column at that position in the list instead of the column the list names. For example,INSERT INTO t (c, a) SELECT ...on a table(a INT, b INT, c CHAR)reportedto=awhile pointing atc.Only the error text changes; which inserts succeed or fail stays the same.
An out-of-order commit that failed part-way, for example on a failed mmap, could crash the server, hang, or leave its tasks queued for the next commit. Failed dedup commits could also leak native memory or unmap memory they had not mapped.
This fix corrects in-flight task counting on tables with dropped columns, prevents the writer from completing a partition twice and abandoning other partitions, and avoids hangs when path setup or cleanup fails. Cleanup after a failed commit now logs a release that throws and carries on, so the commit fails with its original error.
Resources are no longer leaked when an O3 merge fails to read dedup keys, including on every failing
RESUME WAL. A WAL dedup commit that fails to map LAG rows, for example oncevm.max_map_countruns out, no longer unmaps ranges owned by other mappings. Log messages that printed nanosecond timestamps as microseconds were also fixed.The server now marks redirect responses as empty and removes stray trailing bytes, allowing clients to reuse the connection.
SAMPLE BYwithLIMITand a copy of the timestamp in the select list now appliesLIMITafter the sort instead of taking buckets in hash order. Window queries over a column selected under several names no longer fail withInvalid columnor sort by the wrong column.Month and year buckets anchored partway through a month or year (
ALIGN TO FIRST OBSERVATION,FROM, orWITH OFFSET) now place rows correctly, and multi-month or multi-year strides count from the anchor. Rows at the end of short months with a negative offset orFROMon day 29 to 31 no longer land in the previous bucket, which previously failed materialized view incremental refresh with "timestamp out of replace range". MixedFILLlists withLINEARnow produce the same values asFILL(LINEAR)alone.A materialized view no longer stays stuck in 'refreshing' and leaks memory after a refresh with no intervals. Materialized views with a month or year stride and an offset may hold rows in wrong buckets from before the fix;
REFRESH MATERIALIZED VIEW ... FULLrewrites them. Timer refreshes withEVERY nMorEVERY nyfollow the corrected grid, and multi-month cadences count fromSTARTrather than from January.After a base table column changed type, for example from
TIMESTAMPtoSYMBOL, a full refresh of a materialized view reading that column could fill it with wrong values while still reporting the view as valid. The refresh job now generates a new row copier whenever it recompiles the view query and checks every result column against its view column.A result type that cannot be converted invalidates the view with an error naming both columns.
The conversion rule follows
INSERT INTO ... SELECT, but acceptsDOUBLEintoDATEandTIMESTAMPand rejects numeric and temporal values intoCHAR. Row copiers now also implementSYMBOLtoTIMESTAMPconversion for microsecond and nanosecond columns, in bothINSERT INTO ... SELECTand view refresh.Behaviour change: a view whose recompiled query yields a type that cannot be converted now becomes invalid on refresh instead of refreshing with wrong values. This includes a base column changed from
BOOLEANtoBYTE; such views need to be recreated.These operators held a cursor or raw address into an internal map across calls that could grow the map and move its memory, then read freed memory. Depending on the freed memory, queries returned wrong rows, failed with
InternalError: a fault occurred in an unsafe memory access operation, or crashed the JVM.This fix makes
RIGHTandFULLhash joins take the build map cursor only when the unmatched-row scan starts, makesSPLICE JOINcache row ids instead of value addresses, and makes keyedSAMPLE BY FILLtake a fresh keys cursor for every fill scan.Converting a split active WAL partition to Parquet could leave the replaced native column mappings open, causing later schema changes to fail. Native-to-Parquet conversion now identifies the active partition by logical timestamp, as it compacts all split siblings.
Queries running during out-of-order ingestion no longer return changed rows or boundary errors, and materialized view refreshes no longer turn invalid during out-of-order ingestion. Late rows before an added column's data no longer null its values in running queries.
ALTER TABLE SQUASH PARTITIONSno longer loses rows whenFORCE DROP PARTITIONleft a day's split partitions behind. Split partitions also squash sooner, as a query started just before the target partition was written no longer delays it.As a tradeoff, while a query older than a split is open, late rows at the split partition's last timestamp re-split or copy it.
Covering indexes (
INDEX TYPE POSTING INCLUDE (...)) could return NULL for every covered value of a partition, including the designated timestamp, when their sidecar files were missing or wrong. A snapshot or backup restore over Parquet partitions now rebuilds covering indexes by decoding covered columns from the Parquet file.RENAMEandDROP INDEXof a covered column now link the index sidecars under the column's new version, andUPDATEof a covered column now also rebuilds each covering index that includes it.When covered data is missing, the query now fails with
covering index data is missing, rebuild the index or use the no_covering hintinstead of returning NULLs. Partitions left without sidecars by an older version fail covering queries until a write reseals the index, the index is rebuilt, or the query uses/*+ no_covering */.An index already rebuilt by a restore over Parquet partitions keeps returning NULLs until it is rebuilt with
DROP INDEX+ADD INDEX, asREINDEXskips Parquet partitions.This fix also resolves Parquet conversion of a split last partition leaving writer columns open, which made a later
ALTER COLUMN TYPEfail withwriter invariant violated: columns[] partially open.Quiet QWP connections that send PINGs no longer block WAL cleanup. A new
qwp.min.idle.ms.before.writer.releasesetting (default 30s, 0 disables) sets the idle time after which writers are released.Connections that never send PINGs keep their WAL writers until they close. Released writers free their WAL files and handles 2-7 minutes later at defaults.
After a release, the first write per table re-acquires its writer, like a new connection. Each release logs two INFO lines per released table.
WAL apply no longer pauses when partition compaction holds the table writer. These collisions also no longer log a false CRITICAL
unsolicited table lockmessage.An O3 commit that updates a Parquet partition in place changed the row layout while symbol indexes were rebuilt in place, so cursors opened before the commit could return rows of other keys or miss matching ones; POSTING indexes returned no rows at all. Indexes of Parquet partitions now store index rows mapped through the
_pmfooter, and in-place updates append only the rows of new row groups. A footer without the map means identity, so no migration is needed.Once an update moves row groups to fresh index rows, an older binary rejects that partition's footer. Covering POSTING indexes are still fully resealed after an O3 commit, and
SELECT DISTINCTover a POSTING index gets slower after scattered late writes into a Parquet day until compaction or a rewrite of the day.This fix also stops
recordAt()from reading a recycled Parquet decode buffer, and preserves column tops in Parquet-to-native conversion, in-place O3 updates andCONVERT TO NATIVE(wheres IS NULLcould return top rows twice). Frame conversion now writes an absent column as nulls with column top 0.Storage-policy conversion moves from the writer to the partition compaction job, which now also serves non-WAL tables and builds Parquet copies of native partitions.
CONVERT PARTITION TO PARQUETconverts a non-active logical partition (plain, composite, or main folder plus splits) into one Parquet file in a single pass.For partitions affected by stale column tops from QuestDB 9.3.4 - 10.0.1, index queries now fail with
index does not cover parquet partition data, partition compaction rebuilds itinstead of returning wrong rows, and the compaction job rebuilds their indexes. Durability of compaction-job conversions was also improved, including flushing under the table's commit mode and fixing a short BITMAP index after a power loss following a SYNC conversion.When a WAL table had a backlog of small out-of-order transactions spanning a wide timestamp range, the radix sort key could overflow with
error_sort_timestamp_txn_range_overflow, andTableWriterthen applied transactions one by one and could fall behind ingestion indefinitely. Candidate blocks now stop extending before the timestamp range plus transaction count overflows the sort key, so such blocks are split into smaller blocks instead.The check uses the table's native timestamp unit, so nanosecond tables are capped at a 1000x narrower time spread. The cap does not apply to non-dedup blocks of in-order, non-overlapping transactions from a single WAL segment.
This fix also corrects WAL apply block sizing for tables with a Parquet last partition, where blocks on wide rows could be hundreds of times too many rows long. WAL apply now measures bytes per row itself until the column files have been measured.
On vectorized
WINDOW JOINplans,sum()of a LONG256 column returned0x00or random values because uninitialized memory was read.UNION,UNION ALL,INTERSECTandEXCEPTover geohash columns of the same storage width but different precision (for exampleGEOHASH(2c)andGEOHASH(3c)) kept the trailing bits instead of the leading bits, returning the wrong cell and matching the wrong rows.SAMPLE BY FILL(NULL)underALIGN TO FIRST OBSERVATION, with aFROMexpression, or in a mixed fill such asFILL(NULL, 0)no longer fails withUnsupported typefor aggregates returning DATE, STRING, VARCHAR, SYMBOL or LONG256. BOOLEAN and CHAR stay unsupported forFILL(NULL).length_bytes(),starts_with(),strpos(),nullif(),replace(),json_extract()andarg_max()no longer fail with an empty internal error on an IPv4 argument, norlength_bytes()on a SYMBOL. In UDP line protocol ingestion, an empty IPv4 value no longer appends two extra bytes to the column, which shifted every later value.This fix also lets array bind variables accept NULL text as a NULL array through the Java API, and fixes
AssertionErrorfailures with about 600 or moreGROUP BYkey columns plus a computed UUID or LONG128 key, and with very wideINSERT ... SELECTstatements involving geohash precision or timestamp unit conversions.Querying a
DECIMALcolumn with precision up to 9 over QWP egress (the WebSocket query protocol used by the Java, Ruby, Python, C and Rust clients) failed withQWP egress: unsupported column type. Narrow decimals are now mapped to the existingDECIMAL64wire type, so existing clients decode them unchanged.Clients see these columns as
DECIMAL64with the column's scale and an implied precision of 18. These columns take a per-row path that is slower than the columnar copy used by nativeDECIMAL64columns. The wire format and bind parameters are unchanged.ALTER COLUMN TYPEnow preserves indexed results when decoding Parquet partitions. Later Parquet conversions no longer suspend WAL tables after out-of-order writes.Interpolation could overflow to infinity (read as NULL) for large DOUBLE values, starting from about 2.5E298 with hourly buckets and lower for wider buckets, longer gaps or nanosecond timestamps. This fix recomputes the value with an alternative formula when the original result is not finite, so ordinary results do not change.
Buckets that a keyed
FILL(LINEAR)extrapolates before a key's first row or after its last can still read NULL for large values, as before.Checkpoints now preserve altered regular views despite WAL apply lag or a concurrent
ALTER VIEW. Previously, comparing the captured view definition with the live transaction file could skip the checkpoint transaction file.This fix builds an empty transaction file at the captured definition's sequencer transaction, so both describe the same view state.
QuestDB 9.3.4–10.0.1 could leave stale column tops after merging out-of-order Parquet rows. This fix normalizes that metadata and rebuilds indexes in replacement directories while preserving active readers.
Affected Parquet indexes can be repaired online with
ALTER TABLE ... FORCE REINDEX PARQUET, which supports column and partition selection, runs outside the WAL, and commits each partition separately.ADD INDEXincludes repairs in its own transaction, and out-of-order writes rewrite affected local Parquet partitions automatically.Repairs fail on busy writers, retain completed partitions after cancellation, and skip cold and soft-linked partitions.
Parquet deduplication now uses decoded row-group values instead of stale partition-level column tops, so it matches merged Parquet keys instead of retaining duplicates.
VACUUMreclaims dropped posting-index files after repairs, discovering retired bitmap and posting files independently of the current index type.Concurrent parallel filter queries no longer hang PostgreSQL Wire Protocol or HTTP workers at full CPU.
PageFrameSequence.next()now parks the owner when a dispatch round publishes no new frame, or yields its worker when the shared query pool's fiber dispatcher is unavailable.Parallel filters with a bind variable no longer skip rows in batched PostgreSQL Wire Protocol fetches; the server now waits for dispatched frames of open cursors before rewriting bind variables. Parallel queries no longer stall up to a second waiting for a missed wake-up, as every wait now wakes on the event it waits for.
wait_wal_table()no longer returns up to a second late after a WAL notification queue overflow, and repeated timeouts on one connection no longer return concatenated error messages.VACUUM TABLEno longer deletes another table's column files, such as symbol dictionaries, and no longer fails on tables whose only column files belong to a dropped column. It now removes a dropped column's files from tables that have no other column files.The scan previously started at column index -1, which is also the index of the first dropped column found, so the purge task could keep the table and column of a previous
VACUUM. The scan now starts at an index no column has.Queries on an attached foreign Parquet partition no longer crash the server. For a column QuestDB cannot infer, such as a
fixed_len_byte_array(8)without a logical type, it records type -1 in the generated_pm, which previously caused a panic that aborted the JVM. The decoder now converts the stored type with a checked conversion, so an invalid type becomes an ordinary query error.ATTACH PARTITIONnow rejects a Parquet file containing a table column that queries cannot read. Parquet columns that no table column maps to still attach. A failed attach removes the_pmit generated, so a retry with a corrected file attaches.On WAL tables, WAL apply skips such an attach, while the statement itself returns OK.
lpad()andrpad()with a constant VARCHAR string and a variable fill text no longer give wrong results in comparisons.Some non-digit CHAR values, such as
İ, no longer convert to a digit, because CHAR-to-number conversions now check the full character value instead of dropping its high byte. This applies to explicit casts, implicit numeric reads, bind andINSERTconversions, and constant window bounds. Queries, views and materialized views that convert such values to a number now fail.LONG256 bind parameters now accept hex text with an optional
0xor0Xprefix, asINSERT ... VALUESdoes.Replace-range writes no longer suspend the table with
pieces must ascend by tsLo. Previously, a replace-range commit whose range ended just below the start timestamp of the next piece could produce a piece starting on the same timestamp as the next one. Such a plan is now detected and the partition is written as a fresh flat version.A partition can no longer be saved with two pieces that start at the same timestamp, which could make readers read the wrong piece. This check previously ran only when the JVM had assertions enabled (
-ea) and now throws a criticalCairoException.Period materialized views no longer keep a bucket after a range replace deletes its rows, and no longer miss late rows after an
ALTERmoves their period end earlier. They also no longer miss older rows when a refresh cannot read the base table's WAL; the fallback now keeps the earlier start.Period views with a time zone no longer refresh unfinished periods around DST changes, and views in time zones behind UTC no longer lose late rows after DST ends. The refresh and period timer now take the offset in effect at the end of the period and count DELAY in UTC.
materialized_views()now reportsrefresh_period_hiafter the first refresh, even when no complete period has rows. Period views without rows in complete periods now refresh every period, adding two WAL commits, and views built on such a view add one or more WAL commits every period.After a restart,
TIMESTAMP_NStables no longer keep files of dropped or rewritten columns. A restart, including the first after an upgrade, no longer losesTIMESTAMP_NScolumn data.The column purge log saved partition timestamps in microseconds, which dropped the nanoseconds of O3 split partitions in
TIMESTAMP_NStables. A purge replayed after a restart could then look for files in a nonexistent directory, or in another partition starting in the same microsecond, and delete column files that partition still read.This fix makes the purge job save the exact timestamp in a new
partition_timestamp_rawcolumn ofsys.column_versions_purge_log, which is added to existing logs at startup. Rows written by older versions are replayed by locating partitions in a table snapshot and purging a version only whereVACUUM TABLE's liveness test finds it dead.TIMESTAMP_NSpartition layout files that older versions queued for purge stay on disk.TIMESTAMP_NStables a restart cannot read keep old queued files untilVACUUM TABLE.GROUP BY NULLno longer crashes the server whencairo.sql.small.map.page.sizeis very small.GROUP BYandDISTINCTkeys after aNULLkey no longer fail or return wrong values.INSERT ... SELECTnow storesNULL, not another column's data, in untypedNULLcolumns. ANULLdesignated timestamp fails without being read.Out-of-order writes into a restored Parquet partition no longer suspend the table.
A checkpoint restore keeps
data.parquetlonger than its committed size, and when an in-place out-of-order update wrote less than that stale tail, the file length was published as the partition's committed size. This fix publishes the size the update wrote.<=and>=on a constantNULLCHARnow return false, like aCHARcolumn. Previously, constantNULLCHARvalues such asnull::charpicked theSTRINGcomparison, which readsNULL <= NULLas true.Now
<,<=,>and>=between a constantNULLCHARand aNULLorCHARoperand use theCHARcomparison, matching the Java filter and the JIT.Indexed
SYMBOLINlists with repeated bind values no longer return duplicate rows.SYMBOLINandNOT INlists holding both$1and'$1'now return correct rows.bpchar parameters compared with indexed
SYMBOLcolumns no longer fail with an internal error. Non-text parameters such asINTnow compare with indexedSYMBOLcolumns as they do without an index, andBOOLEANparameters on indexed orLATEST ONSYMBOLcolumns now fail, as without an index.NOT BETWEENon indexed orLATEST ONSYMBOLcolumns now fails instead of returning wrong rows. JIT filters comparingSYMBOLwith constant arithmetic, such ass = 0 + 1, andSYMBOLcompared with arithmetic on aSTRINGparameter, such ass = $1 + 0, now return correct rows.The
/metricsendpoint no longer exportsnullduring startup, which made parsers reject the whole scrape.questdb_workers_job_start_micros_minandquestdb_workers_job_start_micros_maxnow exportNaNuntil workers report a sample.Checkpoint restores no longer suspend POSTING-indexed tables after writes overlap the backup copy. Restore reads the newest entry of each POSTING index; if that entry is newer than the restored
_txn, restore opens the index with that_txnand the writer's recovery drops the newer entries. Indexes with nothing newer are only read, never opened for writing.Restore rebuilds a column's POSTING index from the restored column data or the Parquet file when the copy lost the
.pkof a column that has rows in the partition, when the.pkis shorter than the data it describes, or when the newest kept entry names a missing or short value file. A marker file makes a restore that stops during a rebuild redo it on the next start.Restore now fails, naming the file, when it cannot read a POSTING index it must repair. Restore takes about 30 µs longer per POSTING index per partition.
PostgreSQL Wire Protocol logins with a malformed password message no longer log a critical internal error; the authenticator now logs the message type and closes the connection without a reply. Logins without a user now fail with "invalid username/password" instead of a critical error.
Such logins no longer raise the unhandled-error metric or fail the pessimistic health check. Without
-ea(Windows service), the server no longer reads other message types as a password.DynamicUsernamePasswordMatcherno longer throws a NullPointerException when the startup message carries no user; it now rejects a null username.A failed
ADD INDEXcould leave partial index files and, on a non-partitioned table, delete the column's.d/.idata files. Cleanup now removes only the files of the index being built and never touches column data; a failedALTER COLUMN TYPE ... INDEXalso removes the converted column and its symbol map.Retrying an interrupted
ADD INDEX TYPE POSTINGno longer trusts leftover files that may cover only part of a partition or be torn. A partition's index is reused only if it is complete; otherwise it is deleted and rebuilt, including for Parquet partitions. Opening a torn index no longer overwrites its header with a zeroed entry.Checkpoint restore now clears index files unknown to the restored metadata in every writable partition, on WAL and non-WAL tables; if it can't, the restore fails and keeps the checkpoint for the next start.
DISTINCTon a POSTING-indexed symbol no longer dropsNULLs of rows written before the column was added.On macOS 26, APFS is now identified by file system name, so HFS+ roots are no longer reported as supported.
An
ADD INDEXrerun after a checkpoint restore or failed build now rebuilds everything.ADD INDEX TYPE POSTINGand checkpoint restore now fail if leftover index files can't be removed, and checkpoint restore takes about 0.8 s more (2.2 s cold) per 10,000 partitions of 100 columns on macOS.A crafted
/explink can no longer add headers such asSet-Cookieto the response. Previously,/expcopied itsfilenameURL parameter into theContent-Dispositionheader unchecked: CR and LF in the decoded value ended the header line, and a double quote ended the quoted file name.This fix makes the endpoint reject file names that hold control characters (0x00-0x1F, 0x7F) or invalid UTF-8 with HTTP 400. It also escapes
"and\with a backslash inside the quoted file name, as RFC 9110 specifies for quoted strings, so quotes no longer cut the downloaded file name short or drop its extension.BOOLEAN, BYTE and SHORT have no NULL value, so QuestDB writes their missing rows to Parquet at definition level 0, which a type change such as BYTE to DOUBLE reads as NULL. Table reads find those rows from the definition levels anywhere in a row group, but
CONVERT TO NATIVEtook only the leading column top from the file metadata.As a result, missing rows after the first stored value, such as in the later piece of a split partition, became 0 or false.
With this fix, the decoder behind
CONVERT TO NATIVEfinds these rows from the definition levels too, for files QuestDB wrote. It now follows table reads rather than the metadata column top, even where an out-of-order merge lowered that top.COPY <table> TOnow requiresSELECTpermission on every exported column, checked at both compile and export time.Users granted only some columns must name them explicitly:
COPY (SELECT a, b FROM t) TO ...