By Anais Dotis-Georgiou / Product, Use Cases
Dec 12, 2022

Note: To take advantage of InfluxDB 3.0 (formerly known as IOx) and the advances mentioned in this post, sign up here.

If you’ve been following InfluxDB, you’ve probably heard of InfluxDB IOx, the next evolution of the storage engine powering InfluxDB Cloud. However, I wanted to learn more about how the open source components of the new engine help achieve the requirements for the new InfluxDB engine and why they were chosen. This post covers that precise topic. We’ll also learn why InfluxDB chose to contribute to these open source projects and what our commitment to open source looks like today.

The new InfluxDB engine is built with Rust, Apache Arrow (and Arrow Flight), DataFusion, and Parquet. In order to understand how these technologies help build IOx we need to understand what they are, which we’ll discuss in detail in the following sections. However, it’s important to have a broad understanding of each of these components before diving in:

• Rust is a programming language that is very performant and offers fine grain memory management.

• Apache Arrow is a framework for defining in-memory columnar data.

• Parquet is a column-oriented durable file format.

• Arrow Flight is a “new general-purpose client-server framework to simplify high performance transport of large datasets over network interfaces.”

• DataFusion is an “extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.”

We also need to understand the requirements for InfluxDB IOx in order to understand how these technologies meet them. The following table outlines the requirements for InfluxDB IOx and which technologies are critical to achieving them. These requirements come directly from Paul Dix’s post Announcing InfluxDB IOx - The Future Core of InfluxDB Built with Rust and Arrow.

However, it’s worth mentioning that the decision to invest in these technologies is years in the making. I encourage you to read Paul’s early blog post about how Apache Arrow, Parquet, Flight and Their Ecosystem are a Game Changer for OLAP.

Why Rust?

Rust is “a language empowering everyone to build reliable and efficient software,” according to its documentation. Rust was chosen because of its exceptional performance and reliability. Rust is syntactically similar to C++ and has similar performance as it also compiles to native code. However, it has much better memory safety than C++. Memory safety is protection against bugs or security vulnerabilities that lead to excessive memory usage or memory leaks. Rust achieves these memory safety improvements from its innovative type system. It is also a system-level language which does not allow dangling pointers or buffer overflows by default. A dangling pointer points at invalid memory. Dangling pointers are one of the main classes of errors that lead to exploitable security vulnerabilities in languages such as C++.

Now that we know what Rust is, let’s take a look at how it meets the requirements for InfluxDB IOx.

1. No limits on cardinality. Write any kind of event data and don’t worry about what a tag or field is.

Rust helps meet this requirement because InfluxDB IOx is built on the Rust implementation of Apache Arrow (more on this in the next section). Additionally, the approach to handling unlimited cardinality requires non-trivial CPU during query processing. You are therefore very reliant on squeezing the most performance out of your CPU, which Rust is well-suited to do.

2. Best-in-class performance on analytics queries in addition to our already well-served metrics queries.

Rust indirectly meets this requirement because it provides the foundation for the implementation of Arrow, Parquet, and DataFusion (more on how these other components contribute to this goal in later sections). The benefits of Rust’s memory optimization directly impacts the storage performance and query performance of the new engine. You can’t have excellent analytics queries without a query execution framework (DataFusion). You can’t take advantage of those analytics queries without data to query, provided through durable storage (Parquet) and in-memory storage and data exchange (Arrow).

4. Operator control over memory usage. The operator should be able to define how much memory is used for each buffering, caching, and query processing.

IOx uses Rust for memory control. According to Paul Dix, “Rust gives us more fine-grained control over runtime behavior and memory management. As an added bonus it makes concurrent programming easier and eliminates data races. Its packaging system, Crates.io, is fantastic and includes everything you need out of the box. With the addition of async/await to fix race conditions”. For example, buffering crate guarantees protection against buffer overflows and bounds checking. The cached crate ensures thread-safe, async caching structures which help prevent data races. To understand more about the relationship between memory and thread safety practices with Rust programs, I recommend checking out this video. Essentially, Rust offers the fine-grained memory control that you would expect with C/C++ but improved upon many of those languages’ shortcomings, including security and multi-threading.

6. Broader ecosystem compatibility. Where possible, we should aim to use and embrace emerging standards in the data and analytics ecosystem.

The fact that InfluxDB IOx is built on the Rust implementation of Apache Arrow satisfies this requirement. More on this in the following sections.

7. Run at the edge and in the datacenter. Federated by design.

Optimizing your memory usage with Rust, means that InfluxDB IOx also provides these memory optimizations at the edge or in the datacenter.

Why Apache Arrow?

Apache Arrow is “a language-agnostic software framework for developing data analytics applications that process columnar data” according to Wikipedia.

Here’s how Apache Arrow meets the requirements for InfluxDB IOx.

1. No limits on cardinality. Write any kind of event data and don’t worry about what a tag or field is.

Apache Arrow overcomes memory challenges associated with large-cardinality use cases by providing efficient columnar data exchange. Let’s take a look at how InfluxDB IOx stores data and how the columnar data storage enables more efficient compression.

Imagine that we are writing the following lines of line protocol to InfluxDB IOx:

measurement1,tag1=tagvalue1 field1=1i timestamp1
measurement1,tag1=tagvalue2 field1=2i timestamp2
measurement1,tag2=tagvalue3 field1=3i timestamp3
measurement1,tag1=tagvalue1,tag2=tagvalue3 field1=4i,field2=true timestamp4
measurement1, field1=1i timestamp5

IOx returns the following table where tagsets and timestamps identify new rows on write.

However, IOx stores the data in a columnar format like this:

Or, in other words, the data would be stored like this:

1i, 2i, 3i, 4i, 1i; 
null, null, null, true, null; 
tagvalue1, tagvalue2, null, tagvalue1, null; 
null, null, null, tagvalue3, tagvalue3, null; 
null, null, null, tagvalue4, null; 
timestamp1, timestamp2, timestamp3, timestamp4, timestamp5.

Notice that neighboring values are the same data type and similar, if not the same. This provides an opportunity for cheap compression, which enables high cardinality use cases. This also enables faster scan rates using the SIMD instructions found in all modern CPUs. Depending on how you sort the data, you may only need to look at the first column of data to find the max value of a particular field.

Contrast this to a row-oriented storage where you need to look at every field, tag set, and timestamp to find the max field value. In other words, you have to read the first row, parse the record into columns, include the field values in your result, and repeat. IOx provides a much faster and efficient process, thanks to Apache Arrow.

2. Best-in-class performance on analytics queries in addition to our already well-served metrics queries.

Arrow offers best-in-class performance on analytics through the memory optimizations and efficient data exchange described above. You can’t have highly performant analytics without the efficient in-memory columnar storage that Arrow provides.

3. Separate compute from storage and tiered data storage. The DB should use cheaper object storage as its long-term durable store.

Arrow provides the in-memory columnar storage while Parquet provides the column-oriented data file format on disk. Parquet and Arrow are both column-oriented formats and have fantastic interoperability with read and write APIs between the two. This interoperability enables the desired storage separation. Parquet also acts as cheaper object storage as IOx’s long-term durable store (more on this in the Parquet section).

4. Operator control over memory usage. The operator should be able to define how much memory is used for each buffering, caching, and query processing.

Remember, the new storage engine is built on the Rust implementation of Apache Arrow. This is how Apache Arrow is able to gain fine-grained control over memory usage.

6. Broader ecosystem compatibility. Where possible, we should aim to use and embrace emerging standards in the data and analytics ecosystem.

Using Arrow as the underlying format allows fast and easy data interchange with the large, and growing, Arrow-based ecosystem. There are libraries written in C, C++, Java, JavaScript, Python, Ruby, and more (at least 12 in total). The broad ecosystem compatibility of Apache Arrow is part of the reason why technologies like Google’s BigLake, Apache Spark, Snowflake, Amazon Athena, and RISELab’s Ray integrated it into their respective stacks for efficient data exchange.

Why DataFusion?

DataFusion is an “extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format”.

Let’s take a look at how DataFusion meets the requirements for InfluxDB IOx.

1. No limits on cardinality. Write any kind of event data and don’t worry about what a tag or field is.

Without a query engine able to handle the high cardinality data, you can’t really do very much with it. While DataFusion doesn’t help users write high cardinality data, it does help users query, process, and transform that data.

2. Best-in-class performance on analytics queries in addition to our already well-served metrics queries.

IOx uses DataFusion to execute logical query plans, optimize query optimization, and serve an execution engine that is capable of parallelization using threads. Many different projects and products use DataFusion, and it has a thriving community of contributors who add broad SQL support and sophisticated query optimizations. This community investment makes IOx’s query engine much more mature and performant than it would be if we wrote a custom engine ourselves. Since DataFusion is also built on top of Rust, uses Arrow as its in-memory columnar format, and Parquet as its durable format, the performance benefits from those technologies carry over.

3. Separate compute from storage and tiered data storage. The DB should use cheaper object storage as its long-term durable store.

DataFusion has the native ability to read Parquet files from object store (without downloading them locally). Also, DataFusion can selectively read the parts of the Parquet files that are needed*. It accomplishes this selectivity through projection pushdowns and object store range scans. DataFusion also utilizes other advanced techniques that further reduce the bytes required to process data (such as predicate pruning, page index pushdown, row filtering, late materialization, etc). All these capabilities contribute to fast queries against data stored on cheaper object store and separate compute.

Keep an eye out for an upcoming post about this topic. We will include a link here after publication.

6. Broader ecosystem compatibility. Where possible, we should aim to use and embrace emerging standards in the data and analytics ecosystem.

DataFusion supports both a postgres compatible SQL and DataFrame API. This means the new InfluxDB engine will support a large community of users from broader ecosystems that use SQL (and eventually Pandas DataFrames).

Why Parquet?

Parquet files are a compressed columnar data format.

Let’s take a look at how Parquet meets the requirements for the new IOx engine.

2. Best-in-class performance on analytics queries in addition to our already well-served metrics queries.

Because Arrow and Parquet are both columnar data formats, all the benefits associated with columnar data already mentioned carry over and increase their performance, too. Parquet achieves fantastic compression, almost as good as, if not better than, custom file formats. Parquet also supports interoperability with several analytics and ML tools.

3. Separate compute from storage and tiered data storage. The DB should use cheaper object storage as its long-term durable store.

Parquet files require little disk space. The column-oriented design of the file format supports fast filtering and scanning. In fact, Parquet files are 16 times cheaper to store than CSV files. I recommend reading this article on Apache Arrow and Apache Parquet: Why We Needed Different Projects for Columnar Data, On Disk and In-Memory to better understand the historical context and evolution of these tools, as well as the benefits of the separation. To summarize, advancements in RAM and bottlenecks resulting from diskIO pushed the need for this separation and these tools.

5. Bulk data export and import.

Parquet files are specifically used for analytics (OLAP) use cases because they enable bulk data export and import to other tools in the ecosystem that can read and write Parquet files.

6. Broader ecosystem compatibility. Where possible, we should aim to use and embrace emerging standards in the data and analytics ecosystem.

Parquet offers interoperability with almost all modern ML and analytics tools that offer best-in-class analytics. DataFusion supports both a SQL and DataFrame API for logical query plans, query optimization, and execution engine against Parquet files. InfluxDB IOx utilizes SQL support for queries in this big data ecosystem (Arrow, Parquet, and Flight). Parquet enables interoperability because many of the most popular languages, such as C++, Python, and Java, have first-class support in the Arrow project for reading and writing Parquet files. I’m personally extremely excited about the Python support. This means that you can query Parquet files directly from InfluxDB and convert them directly into Pandas DataFrames and vice versa. You’ll also have interoperability with other tools that support Parquet, like Tableau, PowerBI, Athena, Snowflake, DataBricks, Spark, and more.

7. Run at the edge and in the datacenter. Federated by design.

Because Parquet files are so efficient, they facilitate and increase the capacity for data storage at the edge.

Contributions to upstream projects and the commitment to open source

InfluxData takes its commitment to open source seriously. Open source is at the heart of the business model and culture for InfluxData. Much of InfluxData’s code, including the influxdb_iox repository, fall under the permissive MIT or the Apache 2.0 licenses. Permissive licenses provide developers with:

1. Freedom — The ability to use the code to build their own applications and businesses on top of the open source code.

2. Evolution — The ability to evolve the software or make a derivative product.

3. Impact — When smart people have access to good tools, they have the freedom, creativity, and control to create impactful solutions.

The contributions that InfluxDB IOx engineers made to upstream projects, like DataFusion and Arrow also reflect their commitment to open source. The more code and community engagement that IOx devs contribute to the technology used to build IOx, the better IOx becomes. It’s a feed-forward, symbiotic relationship that’s worth investing in. Two Influxers are members of the Apache Arrow PMC, which helps with project governance. IOx engineers actively contribute to that community, not only with code, but with reviews, documentation, blogs, and other evangelism, too.

Some notable code contributions include:

1. Object_store library (contributed to Apache Arrow-rs).

2. Contributions to Arrow-flight SQL.

3. Contributions to Parquet

4. Contributions to the Rust ecosystem including: rskafka and pbjson.

5. Contributions to DataFusion.

The object_store library contributions

Proposed and donated the entire object_store crate to arrow-rs. An object store offers almost unlimited highly available and durable key-value storage on demand. You can read all about it in the Rush Object Store Donation blog post and apache/arrow-rs#2030. Carol Goulding (@carols10cents) primarily authored this work, which also had significant contributions from Marco Neumann (@crepererum). Raphael Taylor-Davies (@tustvold) and Andrew Lamb (@alamb) also handled 3 releases of the object_store crate to the community.

Contributions to Arrow

The InfluxDB IOx team manages the weekly release of https://crates.io/crates/arrow and https://crates.io/crates/parquet releases. They also help author and support DataFusion blog posts like: Apache Arrow DataFusion Project Update and June 2022 Rust Apache Arrow and Parquet 16.0.0 Highlights.

Other contributions to Arrow include:

• Fast, memory-efficient sorts (and a blog about it), which other implementations already picked up.
• Performance improvements at all levels.
  • apache/arrow-rs#1248
  • apache/arrow-rs#2646
• Making the Arrow crate safe by default and additional error checking
  • apache/arrow-rs#817

Contributions to Parquet

The InfluxDB IOx team added significant contributions to Parquet by effectively rewriting a large portion of the Parquet crate. Raphael Taylor-Davies (@tustvold) deserves recognition for contributing:

• Significant work optimizing:
  • Reading Parquet (e.g., apache/arrow-rs#1284 and apache/arrow-rs#1965).
  • Writing Parquet, including:
    • apache/arrow-rs#2221
    • apache/arrow-rs#2278
    • apache/arrow-r#2322
• Reading Parquet directly to Arrow
• Async reader for integration with object storage.
• Significant work to guide and help implement predicate pushdown, such as defining the API (apache/arrow-r#2335).
• Support for reading/writing arbitrarily nested data. To learn more about this contribution, read Arrow and Parquet Part 1: Primitive Types and Nullability.
• Reworked Arrow schema inference to handle all the various backwards compatibility rules.
• Significant work to add better test coverage, triage community issue reports, etc.

Rust ecosystem contributions

• Contributions to Rustlang’s Async Tokio, including optimizing thread pools to manage the execution of CPU bound and I/O work within InfluxDB IOx (tokio-rs/tokio#4105).

• Fixed soundness error in widely used flatbuffers crate.

DataFusion contributions

Contributions to DataFusion include memory optimizations to more efficiently query InfluxDB IOx. Specifically, optimizations to perform really large groupbys and joins (when one or both of the inputs don’t fit in the available RAM). Additionally, there are contributions that enable optional control over and ability to limit memory usage for functions that require large amounts of memory (like sort, groupby, and join functions). InfluxDB IOx devs also contributed additional SQL features. For example:

• Helped rewrite Parquet reader that can read thousands of files from object store

• Support for approximate percentile (median): (apache/arrow-datafusion#1539).

• Performance metrics: influxdb_iox#2025 and apache/arrow-datafusion#866.

• Constant expression evaluation: (arrow-datafusion#1153).

• Sort preserving merge: (apache/arrow-datafusion#379).

Technical writing contributions

Some technical writing contributions include:

• Arrow and Parquet Part 1: Primitive Types and Nullability

• Fast and Memory Efficient Multi-Column Sorts in Apache Arrow Rust, Part 1

DataFusion education and evangelism:

• DataFusion and Arrow: Supercharge Your Data Analytical Tool with a Rusty Query Engine - Data + AI Summit 2022 - Databricks

• UCS Database Systems Implementation Talk: Apache Arrow and its impact on the database industry.

• The Data Thread 2022: Apache Arrow and DataFusion: Changing the Game for Implementing Database Systems.

Connect with IOx developers and additional resources

To take advantage of all the advancements with InfluxDB IOx, sign up here.

If you would like to contact the InfluxDB IOx developers, join the InfluxData Community Slack and look for the #influxdb_iox channel. I encourage you to attend monthly tech talks and community office hours on the project on the 2nd Wednesday of the month at 8:30 AM Pacific Time. To learn more, please message that channel.

Check out the following content on InfluxDB IOx to learn more:

• Intro to InfluxDB IOx

• Welcome to InfluxDB IOx: InfluxData’s New Storage Engine

• The Journey of InfluxDB by Paul Dix

• InfluxData Deploys Next-Generation InfluxDB Time Series Engine with Unlimited Scale

• Announcing InfluxDB IOx - The Future Core of InfluxDB Built with Rust and Arrow

• Evolving InfluxDB into the Smart Data Platform for Time Series

• InfluxData is Building a Fast Implementation of Apache Arrow in Go Using c2goasm and SIMD

• On InfluxData’s New Storage Engine. Q&A with Andrew Lamb

• Apache Arrow, Parquet, Flight and Their Ecosystem are a Game Changer for OLAP

I hope this blog post inspires you to explore InfluxDB IOx. If you need any help, please reach out using our community site or Slack channel. I’d love to hear about what you’re trying to achieve and what features you’d like the task system in InfluxDB to have.

Finally, If you’re developing a cool IoT application on top of InfluxDB, we’d love to hear about it, so make sure to share it on social using #InfluxDB!

Feel free to reach out to me directly in our community slack channel to share your thoughts, concerns, or questions. I’d love to get your feedback and help you with any problems you run into! Or share your story and get a free InfluxDB hoodie.