InfluxDB 1.0 GA Released: A Retrospective and What's Next

By Paul Dix / Product, Use Cases
Sep 08, 2016

Today we’re excited to announce the release of the open source time series database, InfluxDB 1.0 and our commercial offering, InfluxDB Enterprise, which supports high availability deployments and scale out clustering for increased throughput. This makes today the biggest day in our company’s history. This release has been almost 3 years in the making and on this occasion I’d like to take a look back at the project’s history, let users know what compatibility guarantees the 1.x line of releases will have and talk about what’s next for InfluxDB.

As we announce the release today, there are tens of thousands of organizations around the globe using InfluxDB. They’re using it to monitor their network infrastructure, security, container infrastructure, solar panels, agriculture, scientific experiments, user analytics, business intelligence, home automation, and countless other specific use cases. To learn more about how companies big and small are using InfluxDB to manage their time-series data, checkout our testimonials page which currently has over 100 companies listed. What’s your story?

Why a time-series database?

In September of 2013 Todd Persen, John Shahid, and I were working on Errplane, a SaaS application for doing real-time metrics and monitoring. Todd and I had started the company in 2012 and despite getting a leg up from taking it through the Winter 2013 batch of Y Combinator, it wasn’t working as we’d hoped. We had raised a modest seed round of funding so we weren’t in danger of imminent demise, but we weren’t having success gaining real customer traction.

We had been grappling with what we should do for most of August 2013 and had another idea that I planned to debut at Monitorama in Berlin in late September. The conference was all about monitoring and I thought it would be a good place to find a receptive audience for a new monitoring product. What I found instead, was not validation of this new product, but validation of another idea we had been kicking around for a long time: an open source time series database.

The roots for this idea pre-date Errplane. In 2011, John and I met while working at a FinTech startup that had plans to track hundreds of thousands of financial instruments in real time. At the beginning we were using a commercial time series database called OneTick. However, it wasn’t designed to track hundreds of thousands of unique time series, so we started looking at alternatives.

In the open source world we found OpenTSDB, Graphite, RRD and not much else. These projects all had limitations that didn’t work for our use case. One specific need we had was the ability to store irregular time series. That is, series created by specific events, like trades in a market. So we set out to build a “time series database” as a set of web services written in Scala that built on top of Cassandra while using Redis for some real-time indexing.

Later, when I built the initial API for Errplane in 2012, it was using the same technologies. I had solved the same problem twice, but for two totally different use cases. On both occasions I had to write a significant amount of backend code to make time series “just work.”

It was a formative experience in how difficult it was to work with time series data on existing technologies. So in the fall of 2012 when Todd and I applied to Y Combinator, we listed “open source time series database” as our secondary idea of what to work on if Errplane didn’t pan out.

Back at Monitorama in September of 2013, I talked to other monitoring and server analytics companies along with DevOps engineers in very large organizations. The thing I kept hearing again and again was that they were all looking to solve the problem of how to store and query their time series data at scale. They were all in the process of reinventing the same wheel.

It was enough to give me conviction that we should give the open source time series database idea a shot and InfluxDB was born. It quickly gained traction so we moved to focus completely on it. In late 2014 we raised a $8.1M series A from Mayfield and Trinity to accelerate things.

Over the course of the last three years we’ve learned many hard lessons about how to scale storage of time series data, building distributed systems for the time series workload, and how to build and create a sustainable open source company that continues to innovate.

We’ve written our own open source storage engine for time series data called the TSM Tree. We’ve created a commercial product called InfluxDB Enterprise that adds high availability and clustering to InfluxDB, giving us the viability as a business to continue to develop open source software. We’ve also made our scalable InfluxDB clustering available on AWS as a managed service via InfluxDB Cloud.

In the last year we created two new open source projects. Telegraf, a data collector for time series data and metrics, and Kapacitor, an engine for time series data processing, monitoring, and alerting. These projects make InfluxData an open source platform for working with time series data, going beyond just the storage and queryability that InfluxDB provides.

Finally, we plan to expand our open source footprint later this year with a re-envisioned and fully open source version of Chronograf. It won’t be about dashboards, it’ll be about an out of the box user experience for monitoring containers, Kubernetes, and Docker Swarm. If you’re interested and would like to participate in early feature iteration, please get in touch.

1.x API compatibility and stability

One of the more important aspects of the 1.0 release is that this marks the stabilization of our API and storage format. Over the course of the last three years we’ve iterated aggressively, often breaking the API in the process. With the release of 1.0 and for the entire 1.x line of releases we’re committing to the following:

No breaking HTTP API changes: When it comes to the HTTP API, if a command works in 1.0 it will work unchanged in all 1.x releases…with one caveat. We will be adding keywords to the query language. New keywords won’t break your queries if you  wrap all identifiers in double quotes and all string literals in single quotes. This is generally considered best practice so it should be followed anyway. For users following that guideline, the query and ingestion APIs will have no breaking changes for all 1.x releases. Note that this does not include the Go code in the project. The underlying Go API in InfluxDB can and will change over the course of 1.x development. Users should be accessing InfluxDB through the HTTP API.

Storage engine stability: The TSM storage engine file format is now at version 1. While we may introduce new versions of the format in the 1.x releases, these new versions will run side-by-side with previous versions. What this means for users is there will be no lengthy migrations when upgrading from one 1.x release to another.

Additive changes: The query engine will have additive changes over the course of the new releases. We’ll introduce new query functions and new functionality into the language without breaking backwards compatibility. We may introduce new protocol endpoints (like a binary format) and versions of the line protocol and query API to improve performance and/or functionality, but they will have to run in parallel with the existing versions. Existing versions will be supported for the entirety of the 1.x release line.

Ongoing support: We’ll continue to fix bugs on the 1.x versions of the line protocol, query API, and TSM storage format. Users should expect to upgrade to the latest 1.x.x release for bug fixes, but those releases will all be compatible with the 1.0 API and won’t require data migrations. For instance, if a user is running 1.1 and there are bug fixes released in 1.2, they should upgrade to the 1.2 release. Until 1.3 is released, patch fixes will go into 1.2.x. Because all future 1.x releases are drop in replacements for previous 1.x releases, users should upgrade to the latest in the 1.x line to get all bug fixes.

What's next?

We have many improvements coming for the 1.x line of releases and we’ve already started on some of these. Generally, you can expect continued work on the open source and commercial versions. As a project we’re known for our speed of iteration and we intend to keep up the pace. The only difference is that now we’re committed to maintaining API contracts as you’d expect with Semantic Versioning.

We’re working on solving the “high cardinality” limitations of the database, giving users the ability to index hundreds of millions of series in tags without consuming unlimited amounts of memory. We’re also working on intelligent rules for automatic rollups and the querying of lower precision data, making visualizations that zoom from short time frames to months or years performant and snappy.

We’ll also be adding in query language functionality like performing calculations across multiple measurements, combining series together through different calculations, sub-queries, ranking queries and much more.

As for the release cycles of InfluxDB, we’ll be iterating on the best strategy as we go. Users can expect patch releases to 1.0 for any serious bug fixes as they arise. We’re targeting the 1.1 release 2 to 3 months from now. Users should watch the 1.1 milestone of what has been closed and merged to see what will be included. We’re not promising all open features in the milestone because we’d rather have a predictable release timeframe, but you can get a sense for what’s in the pipeline.

After 3 years, InfluxDB 1.0 is the biggest and longest project I’ve ever been involved in from start to finish. I couldn’t be prouder of what we’ve accomplished and am looking forward to all the great features and development we have on the horizon. It’s an exciting time for the time series category and it feels like it’s only getting started!

Contributor Shout-Outs

I’d like to wrap things up by highlighting the contributions of a few community members who were instrumental in helping us get to 1.0.

• John Shahid: He started it all with Todd and I and worked tirelessly for the first year of the project.
• Philip O'Toole: He was with us for the entire 0.9 rewrite/redesign and helped build the initial versions of TSM.
• Paulo Pires: multiple bug fixes and features over the last year.
• Kun Oiooj: Fixes in 0.10, multiple 0.9 versions and a new feature coming in 1.1
• Adarsha Mvadu: So many fixes to make InfluxDB run in Windows.
• Jon Seymour: Fixes to TSM and others.

There are many other people we should thank, but we’ll have to wrap it up somewhere. We’ve had over 200 contributors over the last three years. Thanks to everyone who’s ever fixed a bug, updated docs, written a feature, or helped us get to where we are now.

What's next

• Download: 1.0 GA downloads for the TICK-stack are live on our "downloads" page
• Deploy on the Cloud: Get started with a FREE trial of InfluxDB Cloud featuring fully-managed clusters, Kapacitor and Grafana.
• Deploy on Your Servers: Want to run InfluxDB clusters on your servers? Try a FREE 14-day trial of InfluxDB Enterprise featuring an intuitive UI for deploying, monitoring and rebalancing clusters, plus managing backups and restores.
• Tell Your Story: Over 100 companies have shared their story on how InfluxDB is helping them succeed. Submit your testimonial and get a limited edition InfluxDB hoodie as a thank you.