[Test] Add release notes for 0.242

[caithagoras]

Missing Release Notes

Ariel Weisberg

• Link to attribution guidelines in PR template #15243 Link to attribution guidelines in PR template (Merged by: Andrii Rosa)

Daniel Ohayon

• Core support for enums, take #2 #15219 Core support for enums, take Add byte[] backed tuple and block implementations #2 (Merged by: Rongrong Zhong)

Ke

• 04fbe44 Revert "Defer the creation of dictionary in SliceDictionarySelectiveReader"
• 8c60697 Revert "Remove stripeDictionaryData buffer in SliceDictionarySelectiveReader"
• b8e72e8 Revert "Introduce large dictionary mode in SliceDictionarySelectiveReader"

Mayank Garg

• Consider predicate columns for encryption #15187 Consider predicate columns for encryption (Merged by: Rebecca Schlussel)

Nikhil Collooru

• Add support for versioning metastore API #15044 Add support for versioning metastore API (Merged by: Shixuan Fan)

Rongrong Zhong

• Remote plan execution #14012 Remote plan execution (Merged by: Rongrong Zhong)

Saumitra Shahapure

• Fixing bug: IS DISTINCT FROM NULL should work for NULL column values #15128 Fixing bug: IS DISTINCT FROM NULL should work for NULL column values (Merged by: Rongrong Zhong)

Xiang Fu

• Pinot Connector: Adding limit to push down order by for broker query #15135 Pinot Connector: Adding limit to push down order by for broker query (Merged by: Zhenxiao Luo)

Yuya Ebihara

• Support smallint, tinyint, date type of Cassandra #15147 Support smallint, tinyint, date type of Cassandra (Merged by: James Sun)

asdf2014

• Bump Apache Druid to 0.19.0 #15148 Bump Apache Druid to 0.19.0 (Merged by: Zhenxiao Luo)

Extracted Release Notes

• Rename files written by PageSink #14974 (Author: Nikhil Collooru): Rename files written by PageSink
  • Add support for file renaming for Hive connector. This can be enabled with hive.file_renaming_enabled configuration property.
• Add support in parquet reader for reading TIMESTAMP_MICROS type. #15074 (Author: Sanjay Sundaresan): Add support in parquet reader for reading TIMESTAMP_MICROS type.
  • Parquet files written by parquet-avro library that uses TIMESTAMP_MICROS as the OriginalType to represent timestamp can now be queried by presto.
• Support explain verification #15101 (Author: Leiqing Cai): Support explain verification
  • Add support to run explain verification. (:pr:15101). This can be enabled by configuration property explain.
• Add property to limit output size of a query #15108 (Author: Mayank Garg): Add property to limit output size of a query
  • Add session property query_max_output_bytes and configuration property query.max-output-bytes to control how much data a query can output.
• Support fetching queries to be verified using a Presto query #15132 (Author: Leiqing Cai): Support fetching queries to be verified using a Presto query
  • Add support to fetch the list of queries to be verified by running a Presto query.
• Support fragment result caching #15155 (Author: Shixuan Fan): Support fragment result caching
  • Add support for fragment result caching. When enabled, if the same plan fragment and same connector split hit the same worker, engine would directly fetch result from cache and skip computation. Currently only partial aggregation is supported. Cache could be enabled by setting fragment-result-cache.enabled to true and tuned by other configs started with fragment-result-cache. Query could use fragment result cache by setting config experimental.fragment-result-caching-enabled or session property fragment_result_caching_enabled to true.
  • Add getSplitIdentifier to ConnectorSplit. Split identifier is used in fragment result caching to identify if splits are identical.
  • Add getIdentifier to ConnectorTableLayoutHandle. Layout identifier is used in fragment result caching to construct canonical plan.
• Fix performance regression when hive SerDe doesn't prefer Writables #15163 (Author: James Petty): Fix performance regression when hive SerDe doesn't prefer Writables
  • Fix a performance regression for String field handling in GenericHiveRecordCursor when the SerDe does not provide an efficient Writable implementation.
• Implement Spilling Strategies #15173 (Author: Saksham Sachdev): Implement Spilling Strategies
  • Add config experimental.spiller.task-spilling-strategy for choosing different spilling strategy to use.
• Fix LimitQueryDeterminismAnalyzer #15181 (Author: Leiqing Cai): Fix LimitQueryDeterminismAnalyzer
  • Fix an issue in determinism analysis would indicate failing due to data being changed while the data is not changed.
• Fix compiler error in LambdaBytecodeGenerator #15183 (Author: Rongrong Zhong): Fix compiler error in LambdaBytecodeGenerator
  • Fix compiler error in LambdaBytecodeGenerator when CSE is enabled and multiple SQL functions contain lambda expressions.
• [Verifier] Add application-name config #15191 (Author: Palash Goel): [Verifier] Add application-name config
  • Add application-name config to override source passed in ClientInfo.
• Improve plan comparison for explain verification #15198 (Author: Leiqing Cai): Improve plan comparison for explain verification
  • Improve JSON plan comparison for explain verification. (:pr:15198).
• Fix encrypt large value #15199 (Author: Rebecca Schlussel): Fix encrypt large value
  • Fix a bug where DWRF encryption would fail for large uncompressed column values.
• Upgrade Esri to 2.2.4 #15200 (Author: James Gill): Upgrade Esri to 2.2.4
  • Upgrade Esri to 2.2.4. This includes two fixes for bug (OGCGeometry.union hangs Esri/geometry-api-java#266 and Simple geometry reported as non-simple Esri/geometry-api-java#247) that were seen in production.
• Clamp BackgroundHiveSplitLoader concurrency to usable parallelism #15208 (Author: James Petty): Clamp BackgroundHiveSplitLoader concurrency to usable parallelism
  • Improve split loading efficiency by only using as many threads as are required.
• Fix order of dwrf encryption group stats #15214 (Author: Rebecca Schlussel): Fix order of dwrf encryption group stats
  • Fix a bug where non-Presto readers could not read encrypted DWRF files written by Presto if the encryption group listed columns out of order.
• Fix potential deadlock when PageBufferClient exceeds memory limit #15220 (Author: James Petty): Fix potential deadlock when PageBufferClient exceeds memory limit
  • Fix possible permanent stuck queries when memory limit exceeded during shuffle.

All Commits

• 04fbe44 Revert "Defer the creation of dictionary in SliceDictionarySelectiveReader" (Ke)
• 8c60697 Revert "Remove stripeDictionaryData buffer in SliceDictionarySelectiveReader" (Ke)
• b8e72e8 Revert "Introduce large dictionary mode in SliceDictionarySelectiveReader" (Ke)
• 4cd2141 Add integration tests for file renaming (Nikhil Collooru)
• d3ef69a Add config parameters to enable hive file renaming (Nikhil Collooru)
• a786699 POST the metadata results to worker (Nikhil Collooru)
• c2b2b0d Rename files written by HivePageSink (Nikhil Collooru)
• a74b034 Add new POST {taskId}/metadataresults endpoint on worker (Nikhil Collooru)
• f15fe1f Piggyback MetadataUpdateRequests to coordinator as part of TaskInfo (Nikhil Collooru)
• 9eaa655 Support fragment result caching (Shixuan Fan)
• 99445d9 Use static import and lambda in TestDriver (Shixuan Fan)
• 1489cac Introduce fragment result cache manager (Shixuan Fan)
• 069629b Introduce split identifier (Shixuan Fan)
• a1b9176 Introduce CanonicalPlanGenerator (Shixuan Fan)
• 53aa7f6 Reserve memory before projecting rows (Ying Su)
• 00e61e4 Do not ignore table bucketing when query uses bucket column (Vic Zhang)
• d57ce0f Remove TypeManager.resolveOperator (Rongrong Zhong)
• e6ad497 Collapse nested DEREF expressions into a single one. (Sreeni Viswanadha)
• d8be70d Add extra config and session property to toggle join spilling (Saksham Sachdev)
• 9a11908 Use newer Alluxio client (Bin Fan)
• 544b5a4 Introduce TaskSpillingStrategy and multiple spilling strategies (Saksham Sachdev)
• 30e4606 Link to attribution guidelines in PR template (Ariel Weisberg)
• cef58ff Add enum operators (Daniel Ohayon)
• 9fe32b5 Support enum literals in queries (Daniel Ohayon)
• 7e880e6 Support type bound in TypeVariableConstraint (Daniel Ohayon)
• 5e20973 Add long and varchar enum types (Daniel Ohayon)
• 2dfb7cd Remove BlockEncodingManager's dependency on TypeManager (Rongrong Zhong)
• ef5843a Make TestQueryManager#testFailQueryPrerun more reliable (Tim Meehan)
• b57f7ff Upgrade Esri to 2.2.4 (James Gill)
• 139f15a [Verifier] Add application-name config (Palash Goel)
• e2ff32e Simplify PinotQueryGenerator pushdown logic (Zhenxiao Luo)
• 2d1416e Pinot Connector: Adding config to push down order by for broker query (Xiang Fu)
• bb0d04a Add docs for exchange materialization (Wenlei Xie)
• 6b97e58 Update peak memory in TaskContext for Presto-on-Spark task (Wenlei Xie)
• c9e550b Minor tweak in Presto-on-Spark peak task user memory report (Wenlei Xie)
• ebf428a Add peak task user memory to TaskStats (Wenlei Xie)
• 46bcc63 Add comment to SerializedTaskInfo (Wenlei Xie)
• 746824c Fix potential deadlock when PageBufferClient exceeds memory limit (James Petty)
• 35f267d Disable Elasticsearch connector flaky test (Zhenxiao Luo)
• 8ae554d Enable nullifying iterator for broadcast join (Andrii Rosa)
• 77bfa04 Cassandra connector ccreate table (Yuya Ebihara)
• c857741 Fix order of dwrf encryption group stats (Rebecca Schlussel)
• d57bbfd Implement pass through mode UNION ALL on Presto-on-Spark (Vic Zhang)
• 4d91a60 Improve plan matching for explain verification (Leiqing Cai)
• fdfd9be Remove special rule to skip control queries for explain verification (Leiqing Cai)
• 1fc69dc Handle storage connection exceptions correctly (Nikhil Collooru)
• 6bb12b6 Fix error message formatting (Rebecca Schlussel)
• da0d5c2 Fix dwrf encryption for large column values (Rebecca Schlussel)
• 913be28 Clamp BackgroundHiveSplitLoader concurrency to usable parallelism (James Petty)
• 79c3a3f Improve Elasticsearch connector documentation (fornaix)
• 00fbb9b Update default Presto Pinot Connector Configs (Xiang Fu)
• 14ba79c Refactor more operators to use applicate Page method helpers (James Petty)
• 20a6c40 Add Page#extractChannel(int) to enable efficient single channel extraction (James Petty)
• fa513e2 Consider predicate columns for encryption (Mayank Garg)
• c0b38fe Disallow partial aggregation pushdown when (Vivek)
• c065b1d Do not support dynamic filtering with grouped-execution (Ke Wang)
• 244d880 Update drift library to 1.28 (Vic Zhang)
• 95bcfcd Include storage format name to temporary table name (Vic Zhang)
• 28bd920 Use PAGEFILE format for hive unsupported type (Vic Zhang)
• 2d228db Add session property to automatically use PAGEFILE (Vic Zhang)
• 20b72bd Parquet code cleanup (Zhenxiao Luo)
• 4b8021f Add unit tests for partition versioning enabled Metastore (Nikhil Collooru)
• 584fd8c Add new getPartitionNamesWithVersionByFilter metastore api (Nikhil Collooru)
• cdbd177 Add config to enable Partition versioning (Nikhil Collooru)
• 9dcc558 Add optional version parameter to metastore Partition (Nikhil Collooru)
• 9ae10f0 Skip unsupported types when pushing down in Cassandra (Yuya Ebihara)
• ad2a1c7 Add support in parquet reader for reading TIMESTAMP_MICROS type. (Sanjay Sundaresan)
• d9c8807 Remove unused Parquet classes (Vivek)
• b88a9a9 Fix compiler error in LambdaBytecodeGenerator (Rongrong Zhong)
• 30d2588 Remove unused configuration property from properties.rst (Ying Su)
• 4ea4139 Revert "Add long and varchar enum types" (Ying Su)
• 748e69e Revert "Support type bound in TypeVariableConstraint" (Ying Su)
• 93bf9be Revert "Support enum literals in queries" (Ying Su)
• 923d3ed Revert "Add enum operators" (Ying Su)
• 77c442c Revert "Fix IllegalArgumentException in RelationPlanner" (Ying Su)
• d79804d Fixing bug: IS DISTINCT FROM NULL should work for NULL column values (Saumitra Shahapure)
• ee1627e Bug fix in LimitQueryDeterminismAnalyzer (Leiqing Cai)
• 600c157 Avoid DateTimeZone.getDefault() in GenericHiveRecordCursor hot path (James Petty)
• e7e8ecc Fix performance regression when Hive SerDe doesn't prefer Writables (James Petty)
• 06d5f49 Support smallint, tinyint, date type of Cassandra (Yuya Ebihara)
• 5bdb41d Parser warnings when the query contains mixed AND/OR operator without proper parenthesis. (ankit0811)
• df8228f Fix wrong group by column definition in druid connector (Weidong Duan)
• af273e1 Fix broken link in presto-verifier README (Palash Goel)
• faebf2a Add more functions to TestHiveExternalWorkersQueries (Masha Basmanova)
• 2108955 Add waitForMinimumCoordinators to ClusterSizeMonitor (Tim Meehan)
• dd7dad9 Fix IllegalArgumentException in RelationPlanner (Rongrong Zhong)
• aaeff3f Support Remote function execution (Rongrong Zhong)
• 3825bdb Remote function execution with thrift executor (Rongrong Zhong)
• b1b5e8f Fix ProjectNode locality in PlanFragmenter (Rongrong Zhong)
• 219c7d7 Introduce large dictionary mode in SliceDictionarySelectiveReader (Ying Su)
• 0006741 Remove stripeDictionaryData buffer in SliceDictionarySelectiveReader (Ying Su)
• 7994754 Defer the creation of dictionary in SliceDictionarySelectiveReader (Ying Su)
• 93d1b04 Bump Apache Druid to 0.19.0 (asdf2014)
• 405ebe6 Adding multivalued column support in Pinot connector (Dharak Kharod)
• ec9b5cf Add tests for sum, min, max and avg queries running on native workers (Masha Basmanova)
• 926df09 TupleDomain cleanup (Zhenxiao Luo)
• a7bf06f Add property to limit output size of a query (Mayank Garg)
• 1bc26f0 Dynamic partition pruning on workers (Ke Wang)
• 710f04e Support explain verification (Leiqing Cai)
• b388a1b Extract class QueryBundle (Leiqing Cai)
• d385583 Extract interface MatchResult (Leiqing Cai)
• f2d19cf Extract determinism analysis from AbstractVerification (Leiqing Cai)
• b050fc1 Extract query rewrite from AbstractVerification (Leiqing Cai)
• aae0157 Minor fixes in Verifier (Leiqing Cai)
• f3a43a8 Add test infrastructure to run queries using native workers (Masha Basmanova)
• 2e179d3 Support fetching queries to be verified using a Presto query (Leiqing Cai)
• 6a763b2 Cleanup of partial aggregation pushdown for ORC/Parquet (Zhenxiao Luo)
• b66f102 Remove dependency on javafx (Vivek)
• baece53 Restore commented out tests (Ying Su)