diff --git a/README.md b/README.md index 095083d6468..b4c84085ce7 100644 --- a/README.md +++ b/README.md @@ -1,371 +1,59 @@ -# InfluxDB IOx +# InfluxDB Edge -InfluxDB IOx (short for Iron Oxide, pronounced InfluxDB "eye-ox") is the core of InfluxDB, an open source time series database. -The name is in homage to Rust, the language this project is written in. -It is built using [Apache Arrow](https://arrow.apache.org/) and [DataFusion](https://arrow.apache.org/datafusion/) among other technologies. -InfluxDB IOx aims to be: - -* The core of InfluxDB; providing industry standard SQL, InfluxQL, and Flux -* An in-memory columnar store using object storage for persistence -* A fast analytic database for structured and semi-structured events (like logs and tracing data) -* A system for defining replication (synchronous, asynchronous, push and pull) and partitioning rules for InfluxDB time series data and tabular analytics data -* A system supporting real-time subscriptions -* A processor that can transform and do arbitrary computation on time series and event data as it arrives -* An analytic database built for data science, supporting Apache Arrow Flight for fast data transfer - -Persistence is through Parquet files in object storage. -It is a design goal to support integration with other big data systems through object storage and Parquet specifically. - -For more details on the motivation behind the project and some of our goals, read through the [InfluxDB IOx announcement blog post](https://www.influxdata.com/blog/announcing-influxdb-iox/). -If you prefer a video that covers a little bit of InfluxDB history and high level goals for [InfluxDB IOx you can watch Paul Dix's announcement talk from InfluxDays NA 2020](https://www.youtube.com/watch?v=pnwkAAyMp18). -For more details on the motivation behind the selection of [Apache Arrow, Flight and Parquet, read this](https://www.influxdata.com/blog/apache-arrow-parquet-flight-and-their-ecosystem-are-a-game-changer-for-olap/). - -## Platforms - -Our current goal is that the following platforms will be able to run InfluxDB IOx. - -* Linux x86 (`x86_64-unknown-linux-gnu`) -* Darwin x86 (`x86_64-apple-darwin`) -* Darwin arm (`aarch64-apple-darwin`) - - -## Project Status - -This project is in active development, which is why we're not producing builds yet. - -If you would like contact the InfluxDB IOx developers, -join the [InfluxData Community Slack](https://influxdata.com/slack) and look for the #influxdb_iox channel. - -We're also hosting monthly tech talks and community office hours on the project on the 2nd Wednesday of the month at 8:30 AM Pacific Time. - -* [Signup for upcoming IOx tech talks](https://www.influxdata.com/community-showcase/influxdb-tech-talks) -* [Watch past IOx tech talks](https://www.youtube.com/playlist?list=PLYt2jfZorkDp-PKBS05kf2Yx2NrRyPAAz) - -## Get started - -1. [Install dependencies](#install-dependencies) -1. [Clone the repository](#clone-the-repository) -1. [Configure the server](#configure-the-server) -1. [Compiling and Running](#compiling-and-running) - (You can also [build a Docker image](#build-a-docker-image-optional) to run InfluxDB IOx.) -1. [Write and read data](#write-and-read-data) -1. [Use the CLI](#use-the-cli) -1. [Use InfluxDB 2.0 API compatibility](#use-influxdb-20-api-compatibility) -1. [Run health checks](#run-health-checks) -1. [Manually call the gRPC API](#manually-call-the-grpc-api) - -### Install dependencies - -To compile and run InfluxDB IOx from source, you'll need the following: - -* [Rust](#rust) -* [Clang](#clang) -* [lld (on Linux)](#lld) -* [protoc (on Apple Silicon)](#protoc) -* [Postgres](#postgres) - -#### Rust - -The easiest way to install Rust is to use [`rustup`](https://rustup.rs/), a Rust version manager. -Follow the instructions for your operating system on the `rustup` site. - -`rustup` will check the [`rust-toolchain`](./rust-toolchain.toml) file and automatically install and use the correct Rust version for you. - -#### C/C++ Compiler - -You need some C/C++ compiler for some non-Rust dependencies like [`zstd`](https://crates.io/crates/zstd). - -#### lld - -If you are building InfluxDB IOx on Linux then you will need to ensure you have installed the `lld` LLVM linker. -Check if you have already installed it by running `lld -version`. - -```shell -lld -version -lld is a generic driver. -Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead -``` - -If `lld` is not already present, it can typically be installed with the system package manager. - -#### protoc - -Prost no longer bundles a `protoc` binary. -For instructions on how to install `protoc`, refer to the [official gRPC documentation](https://grpc.io/docs/protoc-installation/). - -IOx should then build correctly. - -#### Postgres - -The catalog is stored in Postgres (unless you're running in ephemeral mode). Postgres can be installed via Homebrew: - -```shell -brew install postgresql -``` - -then follow the instructions for starting Postgres either at system startup or on-demand. - -### Clone the repository - -Clone this repository using `git`. -If you use the `git` command line, this looks like: - -```shell -git clone git@github.com:influxdata/influxdb_iox.git -``` - -Then change into the directory containing the code: - -```shell -cd influxdb_iox -``` - -The rest of these instructions assume you are in this directory. - -### Configure the server - -InfluxDB IOx can be configured using either environment variables or a configuration file, -making it suitable for deployment in containerized environments. - -For a list of configuration options, run `influxdb_iox --help`, after installing IOx. -For configuration options for specific subcommands, run `influxdb_iox --help`. - -To use a configuration file, use a `.env` file in the working directory. -See the provided [example configuration file](docs/env.example). -To use the example configuration file, run: - -```shell -cp docs/env.example .env -``` - -### Compiling and Running - -InfluxDB IOx is built using Cargo, Rust's package manager and build tool. - -To compile for development, run: - -```shell -cargo build -``` - -To compile for release and install the `influxdb_iox` binary in your path (so you can run `influxdb_iox` directly) do: - -```shell -# from within the main `influxdb_iox` checkout -cargo install --path influxdb_iox -``` - -This creates a binary at `target/debug/influxdb_iox`. - -### Build a Docker image (optional) - -Building the Docker image requires: - -* Docker 18.09+ -* BuildKit - -To [enable BuildKit] by default, set `{ "features": { "buildkit": true } }` in the Docker engine configuration, -or run `docker build` with`DOCKER_BUILDKIT=1` - -To build the Docker image: - -```shell -DOCKER_BUILDKIT=1 docker build . -``` - -[Enable BuildKit]: https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds - -#### Local filesystem testing mode - -InfluxDB IOx supports testing backed by the local filesystem. - -> **Note** +> [!NOTE] +> On 2023-09-21 this repo changed the default branch from master to main. At the same time, we moved all InfluxDB 2.x development into the main-2.x branch. If you relied on the 2.x codebase in the former master branch, update your tooling to point to main-2.x, which is the new home for any future InfluxDB 2.x development. This branch (main) is now the default branch for this repo and is for development of InfluxDB 3.x. > -> This mode should NOT be used for production systems: it will have poor performance and limited tuning knobs are available. - -To run IOx in local testing mode, use: - -```shell -./target/debug/influxdb_iox -# shorthand for -./target/debug/influxdb_iox run all-in-one -``` - -This will start an "all-in-one" IOx server with the following configuration: -1. File backed catalog (sqlite), object store, and write ahead log (wal) stored under `/.influxdb_iox` -2. HTTP `v2` api server on port `8080`, querier gRPC server on port `8082` and several ports for other internal services. - -You can also change the configuration in limited ways, such as choosing a different data directory: - -```shell -./target/debug/influxdb_iox run all-in-one --data-dir=/tmp/iox_data -``` - - -#### Compile and run - -Rather than building and running the binary in `target`, you can also compile and run with one -command: - -```shell -cargo run -- run all-in-one -``` - -#### Release mode for performance testing - -To compile for performance testing, build in release mode then use the binary in `target/release`: - -```shell -cargo build --release -./target/release/influxdb_iox run all-in-one -``` +> For now, this means that InfluxDB 3.0 and its upstream dependencies are the focus of our open source efforts. We continue to support both the 1.x and 2.x versions of InfluxDB for our customers, but our new development efforts are now focused on 3.x. The remainder of this readme has more details on 3.0 and what you can expect. -You can also compile and run in release mode with one step: +InfluxDB is an open source time series database written in Rust, using Apache Arrow, Apache Parquet, and Apache DataFusion as its foundational building blocks. This latest version (3.x) of InfluxDB focuses on providing a real-time buffer for observational data of all kinds (metrics, events, logs, traces, etc.) that is queryable via SQL or InfluxQL, and persisted in bulk to object storage as Parquet files, which other third-party systems can then use. It is able to run either with a write ahead log or completely off object storage if the write ahead log is disabled (in this mode of operation there is a potential window of data loss for any data buffered that has not yet been persisted to object store). -```shell -cargo run --release -- run all-in-one -``` +The open source project runs as a standalone system in a single process. If you're looking for a clustered, distributed time series database with a bunch of enterprise security features, we have a commercial offering available as a managed hosted service or as on-premise software designed to run inside Kubernetes. The distributed version also includes functionality to reorganize the files in object storage for optimal query performance. In the future, we intend to have a commercial version of the single server software that adds fine-grained security, federated query capabilities, file reorganization for query optimization and deletes, and integration with other systems. -#### Running tests - -You can run tests using: - -```shell -cargo test --all -``` - -See [docs/testing.md] for more information - - -### Write and read data - -Data can be written to InfluxDB IOx by sending [line protocol] format to the `/api/v2/write` endpoint or using the CLI. - -For example, assuming you are running in local mode, this command will send data in the `test_fixtures/lineproto/metrics.lp` file to the `company_sensors` namespace. - -```shell -./target/debug/influxdb_iox -vv write company_sensors test_fixtures/lineproto/metrics.lp --host http://localhost:8080 -``` - -Note that `--host http://localhost:8080` is required as the `/v2/api` endpoint is hosted on port `8080` while the default is the querier gRPC port `8082`. - -To query the data stored in the `company_sensors` namespace: - -```shell -./target/debug/influxdb_iox query company_sensors "SELECT * FROM cpu LIMIT 10" -``` - -### Use the CLI - -InfluxDB IOx is packaged as a binary with commands to start the IOx server, -as well as a CLI interface for interacting with and configuring such servers. - -The CLI itself is documented via built-in help which you can access by running `influxdb_iox --help` - -### Use InfluxDB 2.0 API compatibility - -InfluxDB IOx allows seamless interoperability with InfluxDB 2.0. - -Where InfluxDB 2.0 stores data in organizations and buckets, -InfluxDB IOx stores data in _namespaces_. -IOx maps `organization` and `bucket` pairs to namespaces with the two parts separated by an underscore (`_`): -`organization_bucket`. - -Here's an example using [`curl`] to send data into the `company_sensors` namespace using the InfluxDB 2.0 `/api/v2/write` API: - -```shell -curl -v "http://127.0.0.1:8080/api/v2/write?org=company&bucket=sensors" --data-binary @test_fixtures/lineproto/metrics.lp -``` - -[line protocol]: https://docs.influxdata.com/influxdb/v2.6/reference/syntax/line-protocol/ -[`curl`]: https://curl.se/ - -### Run health checks - -The HTTP API exposes a healthcheck endpoint at `/health` - -```console -$ curl http://127.0.0.1:8080/health -OK -``` +## Project Status -The gRPC API implements the [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md). -This can be tested with [`grpc-health-probe`](https://github.com/grpc-ecosystem/grpc-health-probe): +Currently this project is under active prototype development without documentation or official builds. This README will be updated with getting started details and links to docs when the time comes. -```console -$ grpc_health_probe -addr 127.0.0.1:8082 -service influxdata.platform.storage.Storage -status: SERVING -``` +## Roadmap -### Manually call the gRPC API +The scope of this open source InfluxDB 3.0 is different from either InfluxDB 1.x or 2.x. This may change over time, but for now here are the basics of what we have planned: -To manually invoke one of the gRPC APIs, use a gRPC CLI client such as [grpcurl](https://github.com/fullstorydev/grpcurl). -Because the gRPC server library in IOx doesn't provide service reflection, you need to pass the IOx `.proto` files to your client -when making requests. -After you install **grpcurl**, you can use the `./scripts/grpcurl` wrapper script to make requests that use the `.proto` files for you--for example: +* InfluxDB 1.x and 2.x HTTP write API (supporting Line Protocol) +* InfluxDB 1.x HTTP query API (InfluxQL) +* Flight SQL (query API using SQL) +* InfluxQL over Flight +* Data migration tooling for InfluxDB 1.x & 2.x to 3.0 +* InfluxDB 3.0 HTTP write API (a new way to write data with a more expressive data model than 1.x or 2.x) +* InfluxDB 3.0 HTTP query API (send InfluxQL or SQL queries as an HTTP GET and get back JSON lines, CSV, or pretty print response) +* Persist event stream (subscribe to the Parquet file persist events, useful for downstream clients to pick up files from object store) +* Embedded VM (either Python, Javascript, WASM, or some combination thereof) + * Individual queries + * Triggers on write + * On persist (run whatever is being persisted through script) + * On schedule +* Bearer token authentication (all or nothing, token is set at startup through env variable, more fine-grained security is outside the scope of the open source effort) -Use the `list` command to list gRPC API services: +What this means is that InfluxDB 3.0 can be pointed to as though it is an InfluxDB 1.x server with most of the functionality present. For InfluxDB 2.x users that primarily interact with the database through the InfluxQL query capability, they will also be able to use this database in a similar way. Version 3.0 will not be implementing the rest of the 2.x API natively, although there could be separate processes that could be added on at some later date that would provide that functionality. -```console -./scripts/grpcurl -plaintext 127.0.0.1:8082 list -``` +## Flux -```console -google.longrunning.Operations -grpc.health.v1.Health -influxdata.iox.authz.v1.IoxAuthorizerService -influxdata.iox.catalog.v1.CatalogService -influxdata.iox.compactor.v1.CompactionService -influxdata.iox.delete.v1.DeleteService -influxdata.iox.ingester.v1.PartitionBufferService -influxdata.iox.ingester.v1.PersistService -influxdata.iox.ingester.v1.ReplicationService -influxdata.iox.ingester.v1.WriteInfoService -influxdata.iox.ingester.v1.WriteService -influxdata.iox.namespace.v1.NamespaceService -influxdata.iox.object_store.v1.ObjectStoreService -influxdata.iox.schema.v1.SchemaService -influxdata.platform.storage.IOxTesting -influxdata.platform.storage.Storage -``` +Flux is the custom scripting and query language we developed as part of our effort on InfluxDB 2.0. While we will continue to support Flux for our customers, it is noticeably absent from the description of InfluxDB 3.0. Written in Go, we built Flux hoping it would get broad adoption and empower users to do things with the database that were previously impossible. While we delivered a powerful new way to work with time series data, many users found Flux to be an adoption blocker for the database. -Use the `describe` command to view methods for a service: +We spent years of developer effort on Flux starting in 2018 with a small team of developers. However, the size of the effort, including creating a new language, VM, query planner, parser, optimizer and execution engine, was significant. We ultimately weren’t able to devote the kind of attention we would have liked to more language features, tooling, and overall usability and developer experience. We worked constantly on performance, but because we were building everything from scratch, all the effort was solely on the shoulders of our small team. We think this ultimately kept us from working on the kinds of usability improvements that would have helped Flux get broader adoption. -```console -./scripts/grpcurl -plaintext 127.0.0.1:8082 describe influxdata.iox.namespace.v1.NamespaceService -``` +For InfluxDB 3.0 we adopted Apache Arrow DataFusion, an existing query parser, planner, and executor as our core engine. That was in mid-2020, and over the course of the last three years, there have been significant contributions from an active and growing community. While we remain major contributors to the project, it is continuously getting feature enhancements and performance improvements from a worldwide pool of developers. Our efforts on the Flux implementation would simply not be able to keep pace with the much larger group of DataFusion developers. -```console -service NamespaceService { - ... - rpc GetNamespaces ( .influxdata.iox.namespace.v1.GetNamespacesRequest ) returns ( .influxdata.iox.namespace.v1.GetNamespacesResponse ); - ... -} -``` +With InfluxDB 3.0 being a ground-up rewrite of the database in a new language (from Go to Rust), we weren’t able to bring the Flux implementation along. For InfluxQL we were able to support it natively by writing a language parser in Rust and then converting InfluxQL queries into logical plans that our new native query engine, Apache Arrow DataFusion, can understand and process. We also had to add new capabilities to the query engine to support some of the time series queries that InfluxQL enables. This is an effort that took a little over a year and is still ongoing. This approach means that the contributions to DataFusion become improvements to InfluxQL as well given it is the underlying engine. -Invoke a method: +Initially, our plan to support Flux in 3.0 was to do so through a lower level API that the database would provide. In our Cloud2 product, Flux processes connect to the InfluxDB 1 & 2 TSM storage engine through a gRPC API. We built support for this in InfluxDB 3.0 and started testing with mirrored production workloads. We quickly found that this interface performed poorly and had unforeseen bugs, eliminating it as a viable option for Flux users to bring their scripts over to 3.0. This is due to the API being designed around the TSM storage engine’s very specific format, which the 3.0 engine is unable to serve up as quickly. -```console -./scripts/grpcurl -plaintext 127.0.0.1:8082 influxdata.iox.namespace.v1.NamespaceService.GetNamespaces -``` +We’ll continue to support Flux for our users and customers. But given Flux is a scripting language in addition to being a query language, planner, optimizer, and execution engine, a Rust-native version of it is likely out of reach. And because the surface area of the language is so large, such an effort would be unlikely to yield a version that is compatible enough to run existing Flux queries without modification or rewrites, which would eliminate the point of the effort to begin with. -```console -{ - "namespaces": [ - { - "id": "1", - "name": "company_sensors" - } - ] -} -``` +For Flux to have a path forward, we believe the best plan is to update the core engine so that it can use Flight SQL to talk to InfluxDB 3.0. This would make an architecture where independent processes that serve the InfluxDB 2.x query API (i.e. Flux) would be able to convert whatever portion of a Flux script that is a query into a SQL query that gets sent to the InfluxDB 3.0 process with the result being post-processed by the Flux engine. -## Contributing +This is likely not a small effort as the Flux engine is built around InfluxDB 2.0's TSM storage engine and the representation of all data as individual time series. InfluxDB 3.0 doesn't keep a concept of series so the SQL query would either have to do a bunch of work to return individual series, or the Flux engine would do work with the resulting query response to construct the series. For the moment, we’re focused on improvements to the core SQL and (and by extension InfluxQL) query engine and experience both in InfluxDB 3.0 and DataFusion. -We welcome community contributions from anyone! +We may come back to this effort in the future, but we don’t want to stop the community from self-organizing an effort to bring Flux forward. The Flux runtime and language exists as permissively licensed open source here. We've also created a community fork of Flux where the community can self-organize and move development forward without requiring our code review process. There are already a few community members working on this potential path forward. If you're interested in helping with this effort, please speak up on this tracked issue. -Read our [Contributing Guide](CONTRIBUTING.md) for instructions on how to run tests and how to make your first contribution. +We realize that Flux still has an enthusiastic, if small, user base and we’d like to figure out the best path forward for these users. For now, with our limited resources, we think focusing our efforts on improvements to Apache Arrow DataFusion and InfluxDB 3.0’s usage of it is the best way to serve our users that are willing to convert to either InfluxQL or SQL. In the meantime, we’ll continue to maintain Flux with security and critical fixes for our users and customers. -## Architecture and Technical Documentation -There are a variety of technical documents describing various parts of IOx in the [docs](docs) directory.