InfluxDB 2.0 Alpha Release and the Road Ahead

By Paul Dix / Company, Product
Jan 23, 2019

Today we’re releasing the first alpha build of InfluxDB 2.0. Our vision for 2.0 is to collapse the TICK Stack into one cohesive and consistent whole which combines the time series database, the UI and dashboarding tool, and the background processing and monitoring agent behind a single API. The move from the 1.x line to 2.0 represents the biggest evolution of our product since we started InfluxDB in 2013. In this post I’ll cover what we’re trying to accomplish with the 2.x line of InfluxDB and Flux, why we’re building it out the way we are, and what the development process through alpha, then beta, and then general availability of the 2.0 release will be.

If you want to skip the history and higher level goal and just get to playing around, have a look at our getting started guide for installation and setup instructions in addition to documentation on Flux. We’d love to hear your feedback on our community site under the influxdb2 category. Or if you’re looking for more of a product walkthrough have a look at Russ Savage’s blog post on the release.

Flux, a new language

One of the biggest parts of InfluxDB 2.0 is Flux, our new data scripting and query language. We decided to build a new language after reviewing years of feature requests, community questions, and issues with InfluxQL, our current query language, and TICKscript, our language for working with Kapacitor. From a design goals perspective, Flux should:

• Support language services that drive graphical user experiences so that users can accomplish more without learning the actual language. Flux's design should make this easier than we could achieve with other language choices.
• Designed to integrate across disparate data sources. These could be other databases, third-party APIs, filesystems, or anywhere data lives. Integration with other systems is a core language design feature.
• Able to cross-compile. Working with time series is possible in many different languages like InfluxQL, PromQL, Flux and others. We wanted this to work on on a single optimizer that could plan against many disparate sources. Reasonable developers can disagree about language choices, and we want to embrace a broader ecosystem for working with data.
• In addition to supporting multiple syntaxes for querying, Flux must integrate seamlessly with other analytics tools and environments. This includes things like integration with Jupyter and using Apache Arrow as an underlying data interchange format so that it can integrate with other big data analytics systems.
• Flux is a fourth generation programming language designed for data scripting, ETL, monitoring and alerting. It's more than a query language, but it's not just a programming language. It includes a planner and optimizer. It should be seamless between querying and programming, combining them into a single consistent whole.

All of this leads to a few things. First, Flux will be Turing complete and useful for querying and working with not only time series data, but data in general. In the coming months we’ll detail how to create these connectors and pair them with the query planner and optimizer to convert Flux queries to use each data store’s unique capabilities. In the meantime you can hop over to the Fluxlang section of our community site to tell us which connectors you’d like to see first.

For this release we’ve focused on the query abilities against the embedded InfluxDB 2.0 data store. Long standing feature requests like math across measurements, top N, time shift, group by field, and sorting by values are all possible in this initial alpha release. Creating a new language and a new query engine are a daunting and significant task, but that effort is starting to pay dividends in the amount of new functionality we’re able to add. In fact, adding new functions to the language couldn’t be easier. For example, you can see how count is implemented. A primary design goal of the Flux codebase is to make it easy for external contributors to add new functions, just like we do with input functions in Telegraf.

We’ve kept Flux as a separate repository from InfluxDB because the language will have a life of its own as an open source project that is useful whether you’re using InfluxDB or not. To that end we’ve picked the liberal MIT license and we’ll accept pull requests to integrate Flux with all sorts of third party systems, even if they’re competitive with InfluxDB. This mirrors our philosophy around contributions to Telegraf, which is widely in use not only by InfluxDB users, but also by customers of other companies like Microsoft, DataDog, SignalFX, Wavefront, and many others. The reason for accepting connectors which invite competition is pretty simple: we recognize that the value of Telegraf is driven by its input plugins and if everyone can use it as their collector then it will attract a larger community contributing plugins which also benefits InfluxDB users. We’ve watched this play out over the last 3 and 1/2 years with Telegraf now having over 200 plugins mostly contributed by community (including competitors) and a level of community contribution and use that eclipses InfluxDB itself. It’s our most popular piece of open source software based on usage and community contribution.

TICK is combining into one cohesive whole

The TICK stack is what we call the set of components that make up the InfluxData platform. In order it represents four components for solving problems with time series data: Telegraf, our data collector, InfluxDB, our time series database, Chronograf our UI for visualization, and Kapacitor, our service for processing and monitoring. The current incarnations of those components are an artifact of how the company was built over time.

I hatched the idea for the TICK components in 2014 while raising the series A round for Influx. The pitch was pretty simple: time series is a base level abstraction that is useful for solving many problems and we’re building a platform for working with time series data. Said another way: many data problems are actually time series data problems and we’re going to help people solve those. We had a prototype database and now we needed to hire developers and designers to finish that work and build out the other three components. Over the course of 2015 and 2016 we hired the teams to start building these components with the initial Telegraf release in June of 2015, Kapacitor in November of 2015, InfluxDB 1.0 in September of 2016, and a rebooted Chronograf in November of 2016.

Over that time we’ve learned more about how people are using the platform and what types of problems they frequently encounter. With InfluxQL, there were a large number of feature requests that we simply couldn’t figure out an elegant way to add the language. Kapacitor is currently used by about 13% of InfluxDB users. We’ve frequently heard users mention that TICKscript was difficult to learn and difficult to debug, but for the users that made the journey, they frequently think of it as a killer app. Chronograf is used by about 20% of InfluxDB users. In the early days of InfluxDB we had the UI accessible as part of the database. The move to put all UI into Chronograf was contentious at the time and we now realize a better user experience could be presented if we simply made that UI available as part of the database.

Many of these things led to the creation of Flux, a new language to deliver the longstanding query feature requests we couldn’t address in InfluxQL. Flux as a language looks much more familiar than TICKscript, which is a deliberate language design choice. Having Flux as the language for both interactive ad-hoc queries and background processing will make it easier to develop tooling to help users debug and see what’s going on in their monitoring, alerting, and ETL tasks.

The other big push was to create an API that is unified for the entire platform. It has concepts built in that are designed to make InfluxDB a multi-tenanted, time series service. Because of all these components, InfluxDB is actually much more than just a database. It’s a monitoring system, it a dashboarding engine, it’s an analytics service, it’s a event and metrics processor. As the capabilities of Flux expand, so will InfluxDB’s capabilities and scope.

Telegraf continues to have a life of its own outside of InfluxDB, but we will be adding APIs in InfluxDB 2.0 for Telegraf to integrate with it. One example which is available in the alpha is that we now have a user interface for creating Telegraf configurations, which Telegraf can pull directly from InfluxDB.

Push and pull in one place

Another big part of InfluxDB 2.0 is our first class support for the Prometheus exposition format. InfluxDB 2.0 can act as a metrics scraper and the initial onboarding experience has an option to set up scraping of the local InfluxDB server’s metrics. With InfluxDB 2.0 we support both push and pull models out of the box. Eventually we’ll also support querying via PromQL out of the box.

Future work

We wanted to get this early release out for users in the community to play with to give us feedback. There are some core features missing that need to get done before we’re ready to move into the beta phase. We still have to add compatibility (being able to query InfluxDB 2.0 via InfluxQL and the 1.x API), and being able write to InfluxDB 2.0 as though it were a 1.x server. We also need to add features for backup & restore, bulk data import and export, and data deletes.

Most of these features represent lower level API work. As we’re getting that done, the UI team will be able to iterate quickly on user feedback to add more user facing features that can be built on top of the core 2.0 API. We plan to cut weekly versioned alpha builds starting in the first week of February. Once we get to the minimum set of features we’ll move into the beta phase of the project. It is during the beta period that we’ll be doing additional bug fix, performance testing and other operational readiness activities. The alpha builds are not meant for testing performance, but more for giving feedback on the UI and API.

The entire InfluxDB 2.0 API is documented and implemented based on this Swagger file. Once we’ve moved into the beta phase of the project, the API should be stable with only additive changes. We still need to add all the capabilities of TICKscript into Flux, which will make Tasks useful for monitoring and alerting workloads. We also have to create a TICKscript to Flux converter to help Kapacitor users migration to InfluxDB 2.0.

This is definitely an early release and there’s much more we’re planning on doing over the course of the 2.x line of InfluxDB releases. This will set us up for years of continued innovation with Flux as a core building block.

We’re also building out our new InfluxData Cloud offering, which will be a fully managed and hosted version of the InfluxDB 2.x API. It will have usage based pricing similar to what you see in other fully hosted multi-tenant services. We’ll have more news on this soon.

We need your feedback!

This is only the beginning of a journey of continued development and iteration out in the open. Please have a look at our getting started guide for installation and setup instructions in addition to documentation on Flux. We’d love to hear your feedback on our community site under the influxdb2 category. For more of a product walkthrough have a look at Russ Savage’s blog post on the release.

I’d love to hear from you so don’t hesitate to get at me on Twitter or submit to HN and I’ll comment there.