|
17 | 17 | under the License. |
18 | 18 | --> |
19 | 19 |
|
20 | | -# Apache DataFusion |
| 20 | +> [!WARNING] |
| 21 | +> This is NOT [DataFusion](https://github.com/apache/datafusion/). |
21 | 22 |
|
22 | | -[![Crates.io][crates-badge]][crates-url] |
23 | | -[![Apache licensed][license-badge]][license-url] |
24 | | -[![Build Status][actions-badge]][actions-url] |
25 | | -![Commit Activity][commit-activity-badge] |
26 | | -[![Open Issues][open-issues-badge]][open-issues-url] |
27 | | -[![Discord chat][discord-badge]][discord-url] |
28 | | -[![Linkedin][linkedin-badge]][linkedin-url] |
29 | | -![Crates.io MSRV][msrv-badge] |
| 23 | +# DataFusion Sandbox |
30 | 24 |
|
31 | | -[crates-badge]: https://img.shields.io/crates/v/datafusion.svg |
32 | | -[crates-url]: https://crates.io/crates/datafusion |
33 | | -[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg |
34 | | -[license-url]: https://github.com/apache/datafusion/blob/main/LICENSE.txt |
35 | | -[actions-badge]: https://github.com/apache/datafusion/actions/workflows/rust.yml/badge.svg |
36 | | -[actions-url]: https://github.com/apache/datafusion/actions?query=branch%3Amain |
37 | | -[discord-badge]: https://img.shields.io/badge/Chat-Discord-purple |
38 | | -[discord-url]: https://discord.com/invite/Qw5gKqHxUM |
39 | | -[commit-activity-badge]: https://img.shields.io/github/commit-activity/m/apache/datafusion |
40 | | -[open-issues-badge]: https://img.shields.io/github/issues-raw/apache/datafusion |
41 | | -[open-issues-url]: https://github.com/apache/datafusion/issues |
42 | | -[linkedin-badge]: https://img.shields.io/badge/Follow-Linkedin-blue |
43 | | -[linkedin-url]: https://www.linkedin.com/company/apache-datafusion/ |
44 | | -[msrv-badge]: https://img.shields.io/crates/msrv/datafusion?label=Min%20Rust%20Version |
45 | | - |
46 | | -[Website](https://datafusion.apache.org/) | |
47 | | -[API Docs](https://docs.rs/datafusion/latest/datafusion/) | |
48 | | -[Chat](https://discord.com/channels/885562378132000778/885562378132000781) |
49 | | - |
50 | | -<a href="https://datafusion.apache.org/"> |
51 | | - <img src="https://github.com/apache/datafusion/raw/HEAD/docs/source/_static/images/2x_bgwhite_original.png" width="512" alt="logo"/> |
52 | | -</a> |
53 | | - |
54 | | -DataFusion is an extensible query engine written in [Rust] that |
55 | | -uses [Apache Arrow] as its in-memory format. |
56 | | - |
57 | | -This crate provides libraries and binaries for developers building fast and |
58 | | -feature rich database and analytic systems, customized to particular workloads. |
59 | | -See [use cases] for examples. The following related subprojects target end users: |
60 | | - |
61 | | -- [DataFusion Python](https://github.com/apache/datafusion-python/) offers a Python interface for SQL and DataFrame |
62 | | - queries. |
63 | | -- [DataFusion Comet](https://github.com/apache/datafusion-comet/) is an accelerator for Apache Spark based on |
64 | | - DataFusion. |
65 | | - |
66 | | -"Out of the box," |
67 | | -DataFusion offers [SQL] and [`Dataframe`] APIs, excellent [performance], |
68 | | -built-in support for CSV, Parquet, JSON, and Avro, extensive customization, and |
69 | | -a great community. |
70 | | - |
71 | | -DataFusion features a full query planner, a columnar, streaming, multi-threaded, |
72 | | -vectorized execution engine, and partitioned data sources. You can |
73 | | -customize DataFusion at almost all points including additional data sources, |
74 | | -query languages, functions, custom operators and more. |
75 | | -See the [Architecture] section for more details. |
76 | | - |
77 | | -[rust]: http://rustlang.org |
78 | | -[apache arrow]: https://arrow.apache.org |
79 | | -[use cases]: https://datafusion.apache.org/user-guide/introduction.html#use-cases |
80 | | -[python bindings]: https://github.com/apache/datafusion-python |
81 | | -[performance]: https://benchmark.clickhouse.com/ |
82 | | -[architecture]: https://datafusion.apache.org/contributor-guide/architecture.html |
83 | | - |
84 | | -Here are links to some important information |
85 | | - |
86 | | -- [Project Site](https://datafusion.apache.org/) |
87 | | -- [Installation](https://datafusion.apache.org/user-guide/cli/installation.html) |
88 | | -- [Rust Getting Started](https://datafusion.apache.org/user-guide/example-usage.html) |
89 | | -- [Rust DataFrame API](https://datafusion.apache.org/user-guide/dataframe.html) |
90 | | -- [Rust API docs](https://docs.rs/datafusion/latest/datafusion) |
91 | | -- [Rust Examples](https://github.com/apache/datafusion/tree/main/datafusion-examples) |
92 | | -- [Python DataFrame API](https://arrow.apache.org/datafusion-python/) |
93 | | -- [Architecture](https://docs.rs/datafusion/latest/datafusion/index.html#architecture) |
94 | | - |
95 | | -## What can you do with this crate? |
96 | | - |
97 | | -DataFusion is great for building projects such as domain specific query engines, new database platforms and data pipelines, query languages and more. |
98 | | -It lets you start quickly from a fully working engine, and then customize those features specific to your use. [Click Here](https://datafusion.apache.org/user-guide/introduction.html#known-users) to see a list known users. |
99 | | - |
100 | | -## Contributing to DataFusion |
101 | | - |
102 | | -Please see the [contributor guide] and [communication] pages for more information. |
103 | | - |
104 | | -[contributor guide]: https://datafusion.apache.org/contributor-guide |
105 | | -[communication]: https://datafusion.apache.org/contributor-guide/communication.html |
106 | | - |
107 | | -## Crate features |
108 | | - |
109 | | -This crate has several [features] which can be specified in your `Cargo.toml`. |
110 | | - |
111 | | -[features]: https://doc.rust-lang.org/cargo/reference/features.html |
112 | | - |
113 | | -Default features: |
114 | | - |
115 | | -- `nested_expressions`: functions for working with nested type function such as `array_to_string` |
116 | | -- `compression`: reading files compressed with `xz2`, `bzip2`, `flate2`, and `zstd` |
117 | | -- `crypto_expressions`: cryptographic functions such as `md5` and `sha256` |
118 | | -- `datetime_expressions`: date and time functions such as `to_timestamp` |
119 | | -- `encoding_expressions`: `encode` and `decode` functions |
120 | | -- `parquet`: support for reading the [Apache Parquet] format |
121 | | -- `sql`: Support for sql parsing / planning |
122 | | -- `regex_expressions`: regular expression functions, such as `regexp_match` |
123 | | -- `unicode_expressions`: Include unicode aware functions such as `character_length` |
124 | | -- `unparser`: enables support to reverse LogicalPlans back into SQL |
125 | | -- `recursive_protection`: uses [recursive](https://docs.rs/recursive/latest/recursive/) for stack overflow protection. |
126 | | - |
127 | | -Optional features: |
128 | | - |
129 | | -- `avro`: support for reading the [Apache Avro] format |
130 | | -- `backtrace`: include backtrace information in error messages |
131 | | -- `parquet_encryption`: support for using [Parquet Modular Encryption] |
132 | | -- `pyarrow`: conversions between PyArrow and DataFusion types |
133 | | -- `serde`: enable arrow-schema's `serde` feature |
134 | | - |
135 | | -[apache avro]: https://avro.apache.org/ |
136 | | -[apache parquet]: https://parquet.apache.org/ |
137 | | -[parquet modular encryption]: https://parquet.apache.org/docs/file-format/data-pages/encryption/ |
138 | | - |
139 | | -## DataFusion API Evolution and Deprecation Guidelines |
140 | | - |
141 | | -Public methods in Apache DataFusion evolve over time: while we try to maintain a |
142 | | -stable API, we also improve the API over time. As a result, we typically |
143 | | -deprecate methods before removing them, according to the [deprecation guidelines]. |
144 | | - |
145 | | -[deprecation guidelines]: https://datafusion.apache.org/contributor-guide/api-health.html |
146 | | - |
147 | | -## Dependencies and `Cargo.lock` |
148 | | - |
149 | | -Following the [guidance] on committing `Cargo.lock` files, this project commits |
150 | | -its `Cargo.lock` file. |
151 | | - |
152 | | -CI uses the committed `Cargo.lock` file, and dependencies are updated regularly |
153 | | -using [Dependabot] PRs. |
154 | | - |
155 | | -[guidance]: https://blog.rust-lang.org/2023/08/29/committing-lockfiles.html |
156 | | -[dependabot]: https://docs.github.com/en/code-security/dependabot/working-with-dependabot |
| 25 | +This repo is a sandbox for experimenting with DataFusion CI. **Do not use this repo for anything else.** |
0 commit comments