4.1.2 Release Notes

[yiguolei]

4.1.1 release notes

New Features

Query & Execution

• Support function ARRAY_CROSS_PRODUCT ([Feature](function) Support function ARRAY_CROSS_PRODUCT #64031)
• Add exponential_moving_average aggregate function ([feature](be/fe) Add exponential_moving_average aggregate function #63499)
• Support murmur_hash3_128 function ([feature](function) Support murmur_hash3_128 function #63196)
• Add datasketches HLL sketch aggregate functions (Add datasketches HLL sketch aggregate functions #63143)
• Skip collecting stats for long string columns ([feature](statistics) Skip collecting stats for long string columns #62686)
• Add information_schema role mappings table ([feature](fe) Add information_schema role mappings table #62077)
• Add FE constant folding for cosine_similarity and standardize test patterns ([Feature] Add FE constant folding for cosine_similarity and standardize test patterns #60403)

Cloud Native

• Show compute group for MTMV refresh task ([feature](fe) Show compute group for MTMV refresh task #63206)

Security & Authentication

• Integrate OIDC authentication, MySQL login bridge, and role mapping ([feature](fe) Integrate OIDC authentication, MySQL login bridge, and role mapping #61819)

Improvement

Query & Execution

• Use real elapsed time to compute workload group metrics refresh interval ([improvement](executor) use real elapsed time to compute workload group metrics refresh interval #63537)
• Support TIMESTAMPTZ in multiple aggregate and array functions ([fix](agg,array) support TIMESTAMPTZ in multiple aggregate and array functions #62756)

Storage & Compaction

• Enable packed file and empty rowset optimization by default ([improvement](cloud) Enable packed file and empty rowset optimization by default #63475)
• Support configurable S3 credentials providers ([improvement](cloud) Support configurable S3 credentials providers #62788)
• Add enable_recycler config to skip recycler dynamically ([improvement](cloud) Add enable_recycler config to skip recycler dynamically #63286)

Bugfix

Index & Search

• Fix reading packed inverted index file on file cache miss ([fix](cloud) Fix reading packed inverted index file on file cache miss #64383)
• Fix ANN IVF/PQ recall, avoid init-time large ANN build-buffer reservation, and skip ANN index bui... ([fix](ann-index) Fix ANN IVF/PQ recall, avoid init-time large ANN build-buffer reservation, and skip ANN index build for segments with insufficient rows. #64082)
• Guard LogicalView.computeOutput() against schema drift (IndexOutOfBoundsException) ([fix](nereids) Guard LogicalView.computeOutput() against schema drift (IndexOutOfBoundsException) #64007)
• Fix ANN range search state leakage and incorrect slot index tracking. ([fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. #63666)
• Clamp variant_sparse_hash_shard_count to >=1 in SHOW CREATE output ([fix](variant) clamp variant_sparse_hash_shard_count to >=1 in SHOW CREATE output #63661)
• Bind Variant search to nested indexes ([fix](variant) Bind Variant search to nested indexes #63660)
• Reject Lucene-syntax SEARCH on columns without inverted index ([fix](search) reject Lucene-syntax SEARCH on columns without inverted index #63637)
• Preserve operative slots when deep copying logical relations ([fix](fe) Preserve operative slots when deep copying logical relations #63315)
• Preserve variant subfields in view definitions to fix select view result wrong when view select h... ([fix](analyze) Preserve variant subfields in view definitions to fix select view result wrong when view select has variant field #62907)
• Fix tokenize function incorrect result when first argument is const ([fix](function) fix tokenize function incorrect result when first argument is const #62699)
• Allow MATCH on aliased variant subcolumns ([fix](match) Allow MATCH on aliased variant subcolumns #63772)
• Improving AI function performance ([Opt](ai-func) Improving AI function performance #62494)

Query & Execution

• Reject TopN in correlated scalar subquery ([fix](fe) Reject TopN in correlated scalar subquery #64251)
• Optimize floating fmod fast path ([optimize](be) optimize floating fmod fast path #64161)
• Make role-mapping keywords RULE/CEL/MAPPING non-reserved ([fix](nereids) Make role-mapping keywords RULE/CEL/MAPPING non-reserved #64104)
• Prevent unsafe runtime filter pushdown through outer joins ([fix](fe) Prevent unsafe runtime filter pushdown through outer joins #64102)
• Fix datediff folding for zero date ([fix](fe) Fix datediff folding for zero date #64084)
• Align convert_tz folding with BE DST handling ([fix](fe) Align convert_tz folding with BE DST handling #64029)
• Merge struct_element into element_at ([enhance](struct) Merge struct_element into element_at #64027)
• Fix TopN runtime filter activation ([fix](be) Fix TopN runtime filter activation #63969)
• Preserve negative zero sign in SIGNBIT constant folding ([Fix](nereids) Preserve negative zero sign in SIGNBIT constant folding #63954)
• Fix assert row join pushdown alias handling ([fix](fe) Fix assert row join pushdown alias handling #63892)
• Fix array subscript on pruned variant subpath ([fix](variant) fix array subscript on pruned variant subpath #63891)
• Align COM_RESET_CONNECTION behavior with MySQL ([fix](connect) Align COM_RESET_CONNECTION behavior with MySQL #63884)
• Preserve NaN in numeric constant folding ([Fix](FoldConst) Preserve NaN in numeric constant folding #63870)
• Avoid potential OOM when reading large snapshot splits ([improve](streaming-job) avoid potential OOM when reading large snapshot splits #63833)
• Cast variant subcolumn as json in variant_hirachinal for stable output ([fix](test) Cast variant subcolumn as json in variant_hirachinal for stable output #63828)
• Fix changed variable output in show variables ([fix](show variables) Fix changed variable output in show variables #63734)
• Fix computeDestIdToInstanceId picking wrong ExchangeNode for multi-input fragments ([fix](coordinator) fix computeDestIdToInstanceId picking wrong ExchangeNode for multi-input fragments #63615)
• Avoid unsigned underflow in JSON modify path ([fix](be) Avoid unsigned underflow in JSON modify path #63579)
• Handle legacy DecimalV2 segments with missing precision/frac ([fix](be) Handle legacy DecimalV2 segments with missing precision/frac #63569)
• Preserve TIMESTAMPTZ values in sparse path ([fix](variant) preserve TIMESTAMPTZ values in sparse path #63522)
• Align legacy literal compareLiteral with Nereids ComparableLiteral semantics ([fix](planner) align legacy literal compareLiteral with Nereids ComparableLiteral semantics #63481)
• Reject COUNT DISTINCT on variant arguments ([fix](variant) Reject COUNT DISTINCT on variant arguments #63479)
• Compare JSON numeric values by value ([fix](be) Compare JSON numeric values by value #63396)
• Remove unsafe JsonbWriter key overload ([fix](be) Remove unsafe JsonbWriter key overload #63355)
• Prevent invalid alias rewrite in view definitions ([fix](view) Prevent invalid alias rewrite in view definitions #63353)
• Reject non-positive topn count argument ([Fix](topn) Reject non-positive topn count argument #63350)
• Fix json contains duplicate array candidates ([fix](be) Fix json contains duplicate array candidates #63301)
• Reject super wildcard path in json keys ([fix](be) Reject super wildcard path in json keys #63300)
• Report TIMESTAMPTZ as string to MySQL clients ([fix](timestamptz) Report TIMESTAMPTZ as string to MySQL clients #63292)
• Add null reject compensation for join rewrite ([fix](fe) Add null reject compensation for join rewrite #63268)
• Reject lone UTF-16 surrogates in JSONB literals (RFC 8259 §8.2) ([fix](fe) Reject lone UTF-16 surrogates in JSONB literals (RFC 8259 §8.2) #63255)
• Fix alias function with cast outermost expr and reject illegal expressions ([fix](fe) Fix alias function with cast outermost expr and reject illegal expressions #63254)
• Support TIMESTAMPTZ in TopN runtime predicate ([fix](be) Support TIMESTAMPTZ in TopN runtime predicate #63220)
• Reject JSONB and variant distribution columns ([fix](fe) Reject JSONB and variant distribution columns #63211)
• Move the pruning of predicates that are always true after partition pruning into the PlanPostProc... ([fix](partition_prune) Move the pruning of predicates that are always true after partition pruning into the PlanPostProcessor #63111)
• Fix coalesce function output null ([BUG](exec) fix coalesce function output null #63092)
• Keep first duplicate Variant JSON path ([Improve](variant) Keep first duplicate Variant JSON path #63082)
• Avoid unioning query-unused MV partitions ([fix](mv) Avoid unioning query-unused MV partitions #63081)
• Fix integer typing and prefer Variable.realExpression for argument/type resolution ([fix](user_var)fix integer typing and prefer Variable.realExpression for argument/type resolution #62524)
• Fix redundant aggregation in agg-union query plan ([fix](fe) Fix redundant aggregation in agg-union query plan #62231)
• Pass ConnectContext to canUseNereidsDistributePlanner method instead of call ConnectContext.get() ([opt](nereids)pass ConnectContext to canUseNereidsDistributePlanner method instead of call ConnectContext.get() #60529)

Storage & Compaction

• Add from-to cdc WAL-search timeout and stale-reader release ([improve](streaming-job) add from-to cdc WAL-search timeout and stale-reader release #64013)
• Validate recycle rowset key state during commit rowset ([fix](cloud) Validate recycle rowset key state during commit rowset #63985)
• Reduce sparse variant parse memory ([opt](variant) Reduce sparse variant parse memory #63970)
• Avoid retaining full segment key bounds buffers ([opt](memory) avoid retaining full segment key bounds buffers #63968)
• Release packed file writer buffer after flush ([opt](memory) release packed file writer buffer after flush #63967)
• Normalize SC rowset graph before delete bitmap capture ([fix](cloud) Normalize SC rowset graph before delete bitmap capture #63960)
• Support cdc_client JVM opts and adopt externally-managed cdc_client ([improve](streaming-job) support cdc_client JVM opts and adopt externally-managed cdc_client #63898)
• Remove the udf cache expiration_time ‌property‌ ([refact](udf) remove the udf cache expiration_time ‌property‌ #63897)
• Drain txn lazy committer workers before destruction ([fix](cloud) Drain txn lazy committer workers before destruction #63876)
• Filter nereidsPrunedTabletIds per partition in distributionPrune ([improve](nereids) filter nereidsPrunedTabletIds per partition in distributionPrune #63851)
• Keep prefetch reader alive for async tasks ([fix](be) Keep prefetch reader alive for async tasks #63796)
• Avoid retrying object storage SlowDown errors ([fix](s3) Avoid retrying object storage SlowDown errors #63776)
• Remove single replica compaction ([chore](compaction) remove single replica compaction #63771)
• Cache cluster id per query and drop redundant locks on getBackendId hot path ([opt](cloud) cache cluster id per query and drop redundant locks on getBackendId hot path #63636)
• Fix meta tool build on master (Fix meta tool build on master #63540)
• Skip stale tablet cache check for STOP_TOKEN ([fix](cloud) skip stale tablet cache check for STOP_TOKEN #63520)
• Preserve labels for histogram metrics to fix wrong metric name for prometheus ([fix](metric) Preserve labels for histogram metrics to fix wrong metric name for prometheus #63485)
• Normalize default HDFS paths in LocationPath ([fix](fe) Normalize default HDFS paths in LocationPath #63476)
• Truncate segment key bounds before storing segment stats ([opt](memory) truncate segment key bounds before storing segment stats #63469)
• Fill schema change version holes before running ([fix](cloud) Fill schema change version holes before running #63443)
• Disable dict encoding in row store columns ([opt](storage) disable dict encoding in row store columns #63438)
• Remove pure attribute from assert_cast ([fix](be) Remove pure attribute from assert_cast #63417)
• Fix clear_file_cache right after reboot causing file cache size percent overflow ([fix](filecache) fix clear_file_cache right after reboot causing file cache size percent overflow #63410)
• Verify DML and 3-replica create table when one BE is down ([test](docker) Verify DML and 3-replica create table when one BE is down #63401)
• Fix pre-aggregation context leakage across join branches ([fix](nereids) Fix pre-aggregation context leakage across join branches #63357)
• Clean empty v3 cache dirs ([fix](filecache) clean empty v3 cache dirs #63344)
• Avoid concurrent tablet stat iteration failures ([fix](fe) avoid concurrent tablet stat iteration failures #63298)
• Allow show tablet without selected database ([fix](fe) Allow show tablet without selected database #63280)
• Restore split-bound Java types when reading FE-persisted CDC offset ([fix](streaming-job) restore split-bound Java types when reading FE-persisted CDC offset #63219)
• Support storage vault for clone instance ([feat](snapshot) Support storage vault for clone instance #63217)
• Delay overwrite partition routing until incremental open ([fix](insert overwrite) delay overwrite partition routing until incremental open #63209)
• Fix simple aggregate cache after partition recycle ([fix](fe) Fix simple aggregate cache after partition recycle #63175)
• Enhance OOM error message for statistics analyze tasks ([fix](statistics) Enhance OOM error message for statistics analyze tasks #63172)
• Fix incorrect memory availability check in RowSourceBuffer during vertical compaction ([fix](compaction) Fix incorrect memory availability check in RowSourceBuffer during vertical compaction #63152)
• Async chunk splitting for cdc source job ([improve](streaming-job) async chunk splitting for cdc source job #63079)
• Replace std::mutex/std::lock_guard with annotated wrappers for thread safety analysis ([refine](exec/operator) replace std::mutex/std::lock_guard with annotated wrappers for thread safety analysis #63070)
• Avoid NPE for force-finished publish task ([fix](publish) Avoid NPE for force-finished publish task #63069)
• Add SubQueue abstraction and thread-safety annotations to DataQueue ( [refine](exec) add SubQueue abstraction and thread-safety annotations to DataQueue  #62947)
• Skip wait for async rowset warmup ([fix](cloud) Skip wait for async rowset warmup #62764)
• Cache version and get tablet stats actively for RestoreJob ([fix](fe) cache version and get tablet stats actively for RestoreJob #62704)
• Support storage vault for create/list snapshot ([feat](snapshot) Support storage vault for create/list snapshot #62523)
• Infer null-reject from INNER JoinEdge for multi-hop outer join MV rewrite ([fix](mtmv) Infer null-reject from INNER JoinEdge for multi-hop outer join MV rewrite #62492)
• Avoid crash when late holder cleanup sees removed cache cell ([fix](filecache) avoid crash when late holder cleanup sees removed cache cell #62437)
• Fix PartitionRebalancer generating invalid moves to BEs without required storage medium ([fix](balance) Fix PartitionRebalancer generating invalid moves to BEs without required storage medium #62206)
• Add system rate limit for meta-service ([feat](cloud) Add system rate limit for meta-service #61516)
• Replace Tablet references with tabletId in CloudTabletRebalancer ([refactor](cloud) Replace Tablet references with tabletId in CloudTabletRebalancer #61233)
• Add async lru update machanism and fix partial hit in cache reader ([fix](filecache) add async lru update machanism and fix partial hit in cache reader #61083)
• Avoid duplicated FileCache counter accumulation in NewOlapScanner ([fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner #61072)
• Show proc should display partition cached version ([fix](cloud) show proc should display partition cached version #60807)
• Avoid false tablet diagnosis alarms in cloud mode ([fix](cloud) avoid false tablet diagnosis alarms in cloud mode #60805)

Load

• Fix nullable date literal binding and date *_diff folding ([Fix](nereids) Fix nullable date literal binding and date *_diff folding #64127)
• Keep load row metrics monotonic for auto partition ([fix](auto partition) keep load row metrics monotonic for auto partition #64109)
• Fix postgres cdc multi-table publication data loss and binlog duplicate key ([fix](streaming-job) fix postgres cdc multi-table publication data loss and binlog duplicate key #64075)
• Match STRUCT sub-fields by name when loading JSON ([fix](serde) match STRUCT sub-fields by name when loading JSON #64011)
• Change lzo download link ([fix](deps) change lzo download link #63785)
• Fix postgres historical-date timestamp handling in cdc-client ([fix](streaming-job) fix postgres historical-date timestamp handling in cdc-client #63618)
• Add per-job routine load metrics ([feat](job) add per-job routine load metrics #63576)
• Fix int overflow in BeIdComparator causing stream load failure ([fix](fe) Fix int overflow in BeIdComparator causing stream load failure #63565)
• Add txn write amplification brpc metrics for sub txn load ([feat](txn) Add txn write amplification brpc metrics for sub txn load #63545)
• Enforce explicit compute group form for workload DDLs ([fix](workload) enforce explicit compute group form for workload DDLs #63505)
• Support user-specified mysql server_id with per-reader assignment ([improve](streaming-job) support user-specified mysql server_id with per-reader assignment #63490)
• Misc fixes for typo/log/validation/visibility ([fix](streaming-job) misc fixes for typo/log/validation/visibility #63480)
• Avoid NPE on cross-table DML during snapshot chunk read ([fix](streaming-job) Avoid NPE on cross-table DML during snapshot chunk read #63435)
• Keep isCanceled set when cancel runs on terminal task ([fix](streaming-job) keep isCanceled set when cancel runs on terminal task #63427)
• Drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient ([fix](streaming-job) drop neighbour-table rows leaked by JDBC LIKE wildcards in JdbcPostgreSQLClient #63402)
• Optimize row-store memtable flush memory in the row-store scenario ([opt](memory) optimize row-store memtable flush memory in the row-store scenario #63342)
• Fix broken pipe risk on stream load redirect with unconsumed request body ([fix](fe) Fix broken pipe risk on stream load redirect with unconsumed request body #63332)
• Pin esdk-obs-java-bundle to 3.21.11 to fix version range resolution failure ([chore](dep) pin esdk-obs-java-bundle to 3.21.11 to fix version range resolution failure #63278)
• Add per-job lag metric to streaming insert jobs ([improve](streaming-job) add per-job lag metric to streaming insert jobs #63194)
• Fix NPE in routine load Kafka meta request ([fix](job) fix NPE in routine load Kafka meta request #63180)
• Remove dead code across core types and utilities ( [refine](code) remove dead code across core types and utilities #62994)
• CancelTaskById should not be blocked by unrelated streaming jobs ([fix](job-manager) cancelTaskById should not be blocked by unrelated streaming jobs #62940)
• Recompute derived fields after replay and ALTER ([fix](streaming-job) recompute derived fields after replay and ALTER #62936)
• Persist cdc_stream TVF offset across FE checkpoint ([fix](streamingjob) Persist cdc_stream TVF offset across FE checkpoint #62902)
• Avoid mutating shared variant columns ([fix](variant) Avoid mutating shared variant columns #64092)
• Select txn insert backend from current cluster ([fix](transaction) select txn insert backend from current cluster #63634)
• Fix host not match if start fe in metadata_failure_recovery ([fix](fe) fix host not match if start fe in metadata_failure_recovery #62748)
• Deduplicate pending one-shot warm up jobs ([fix](cloud) Deduplicate pending one-shot warm up jobs #62384)

Lakehouse

• Mapping iceberg varint type to unsupported type ([bug](iceberg) mapping iceberg varint type to unsupported type #64331)
• Fix iceberg write special partition error. (branch-4.1:[fix](iceberg)fix iceberg write special partition error. #64225)
• Fix can't get migrated Iceberg tables format type ([bug](iceberg) fix can't get migrated Iceberg tables format type #64134)
• Use object store path for data location ([fix](iceberg) Use object store path for data location #64028)
• Reject iceberg COW table row-level DML ([fix](iceberg) Reject iceberg COW table row-level DML #63950)
• Fix iceberg v3 row lineage count distinct error result ([fix](iceberg)fix iceberg v3 row lineage count distinct error result #63826)
• Fix DCHECK in LocalExchangeSharedState::sub_total_mem_usage ([fix](be) Fix DCHECK in LocalExchangeSharedState::sub_total_mem_usage #63742)
• Add missing Iceberg field IDs for position delete files. ([fix](iceberg) Add missing Iceberg field IDs for position delete files. #63483)
• Fill Hive meta cache when loading row count for queries ([fix](hive table) Fill Hive meta cache when loading row count for queries #63470)
• Fix iceberg sink writer with spill report error ([bug](iceberg) fix iceberg sink writer with spill report error #62899)
• Rest & S3Table Support Iam-role ([feat](Iceberg)Rest & S3Table Support Iam-role #60498)

Security & Authentication

• Bound length in MysqlProto.readLenEncodedString ([fix](fe) Bound length in MysqlProto.readLenEncodedString #63604)
• Include HDFS connection in file handle cache key ([fix](be) Include HDFS connection in file handle cache key #63516)
• Fix arrow flight client ip auth ([fix](auth)Fix arrow flight client ip auth #63506)
• Preserve narrowing datetimev2 casts in simplify in predicate ([fix](fe) Preserve narrowing datetimev2 casts in simplify in predicate #63343)
• Fix arrow::Status inline static empty msg core ([Fix](arrow flight) Fix arrow::Status inline static empty msg core #63191)
• Support multi-root auth plugin loading and normalize OIDC access token auth ([feat](plugin) Support multi-root auth plugin loading and normalize OIDC access token auth #62159)
• Improve masking of user's password for ALTER USER and CREATE USER commands in audit logs ([enhance](nereids) improve masking of user's password for ALTER USER and CREATE USER commands in audit logs #62141)
• Improve LDAP authentication resiliency and diagnostics ( [fix](ldap) Improve LDAP authentication resiliency and diagnostics #61673)
• Integrate authentication chain and simplify fallback config ([feat](authentication)Integrate authentication chain and simplify fallback config #61362)