You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/content/blog/december-2025.mdx
+13-4Lines changed: 13 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,9 +8,6 @@ published: true
8
8
9
9
[TODO - Adam to fill in Datafusion 52 support]
10
10
11
-
The work mentioned in the previous bulletin to revamp Array evaluation to be fully lazy was released on January 6th. This happens by converting their execution to an Operator model that evaluates into Vectors (fully decompressed, zero-copy to Arrow representation). As a reminder, this work enables many more optimizations, and also provides unified abstractions for evaluating on
12
-
different processor types (CPUs & GPUs).
13
-
14
11
## GPU Support
15
12
16
13
Speaking of, focus is now on enabling reading Vortex files into GPUs. To achieve this, the team is:
@@ -22,12 +19,24 @@ The first supported output types will be Arrow Device Arrays and cuDF. As with a
22
19
23
20
## Around the Ecosystem
24
21
25
-
- DuckDB Labs published performance benchmarks comparing Vortex (using the `duckdb-vortex` extension) versus Parquet. Spoiler alert: Vortex was ~18% faster!
22
+
- DuckDB Labs published [performance benchmarks](https://duckdb.org/2026/01/23/duckdb-vortex-extension) comparing Vortex (using the `vortex` extension) versus Parquet. Spoiler alert: Vortex was ~18% faster!
26
23
- Spice AI began a blog series about building their data accelerato Cayenne with Vortex and Datafusion
27
24
-[🌪️ Vortex: The Bet on Encoding-Efficient Columnar Storage for Hot Data](https://www.linkedin.com/posts/lukekim_datafusion-spiceai-data-activity-7417019189477126144-1TRe/)
28
25
-[🔬 The Research Behind Modern Data Compression & Vortex](https://www.linkedin.com/posts/lukekim_datafusion-developers-ai-activity-7417649503291498496-TD5_/)
29
26
-[🤖 Three Data Problems Vortex Solves for Applications and Agents in 2026](https://www.linkedin.com/posts/lukekim_vortex-efficient-columnar-storage-for-hot-activity-7419472524750798848-lZC6/)
30
27
28
+
### DataFusion
29
+
30
+
Over the past two months we've released support for DataFusion 51, and then merged support for DataFusion 52.
31
+
32
+
- Expression conversion is now extendable, allowing users to push down custom UDFs or any other DataFusion expressions.
33
+
- Support for schema evolution between files within the same table has improved dramatically, fixing many bugs and and allowing deeper pushdown over nested structs columns.
34
+
- We now support tables that have more complex arrow types (Like Dict, REE), while also doing less work exporting from Vortex to Arrow.
35
+
- Delegate some caching to to DataFusion's built in caching, allowing users to tune it using familiar configurations.
36
+
37
+
The work mentioned in the previous bulletin to revamp Array evaluation to be fully lazy was released on January 6th. This happens by converting their execution to an Operator model that evaluates into Vectors (fully decompressed, zero-copy to Arrow representation). As a reminder, this work enables many more optimizations, and also provides unified abstractions for evaluating on
38
+
different processor types (CPUs & GPUs).
39
+
31
40
## Acknowledgments
32
41
33
42
We want to thank to anyone who has tried Vortex, provided feedback, asked question and filed issues.
0 commit comments