FDAP Stack

The exponential growth of data necessitates specialized systems that can efficiently handle different data types. The FDAP stack offers a solution by providing reusable components that enable faster and more efficient data processing.

The FDAP stack, consisting of Flight, DataFusion, Arrow, and Parquet, is revolutionizing the way we build analytic systems. In this article, you will learn about the key components of the FDAP stack and why it has become the foundation for InfluxDB 3.0 and other data-centric applications.

FDAP stack key components

Apache Arrow

Apache Arrow is an open-source, in-memory columnar data format. It standardizes data representation in memory, eliminating the need for reimplementing complex systems techniques. Arrow’s efficiency, performance, and broad ecosystem make it an ideal choice for data processing.

Arrow Flight

Apache Arrow Flight is a protocol for fast and efficient network data transfer. It simplifies communication within a cluster and between clients and servers. Flight eliminates the need to define custom network protocols and enables easy integration with different languages.

DataFusion

DataFusion is a state-of-the-art analytic query engine written in Rust. It utilizes Apache Arrow as its memory model and provides SQL support, streaming vectorized execution, query optimizations, and resource management. DataFusion allows for quick data ingestion and querying, making it an essential component of the FDAP stack.

Parquet

Apache Parquet is an open, column-oriented data file format. It offers excellent data encoding, compression, and efficient structured types. Parquet’s compatibility with a wide range of tools and systems, along with its fast query performance, makes it an ideal choice for storing large amounts of data.

FDAP stack benefits

The FDAP stack brings several benefits to developers looking to build data focused applications:

Developer productivity

By leveraging the FDAP stack, developers can focus on building domain-specific features rather than reinventing low-level, complex components. This results in better performance and more features for data-centric applications.

Integration and Interoperability

The FDAP stack’s open standards enable seamless integration with other systems. InfluxDB 3.0, for example, can easily integrate with JDBC, Python, and other tools that support Arrow. Parquet’s interoperability allows for querying by systems like Snowflake or Presto without complex ETL pipelines.

Amplified Investment

The FDAP stack benefits from a large community of developers working on Flight, Arrow, DataFusion, and Parquet. This collective effort leads to continuous improvement and innovation, providing features that would be challenging to achieve individually.

Open Standards

Working within the Apache Software Foundation ensures well-understood decision-making processes and avoids uncertainties that can arise in other open-source projects. The FDAP stack benefits from the stability and predictability of these open standards.