InfluxDB Now Supports Prometheus Remote Read & Write Natively

By Paul Dix / Developer, Product
Sep 14, 2017

Last month I attended PromCon, the annual Prometheus conference, in Munich. The conference had a mix of talks from users to implementers with topics ranging from how-to’s and case studies to details on the implementation of the Prometheus 2.0 storage engine. I gave a talk on Integrating InfluxDB and Prometheus. After my talk I had multiple people come up to me and express interest in more integration between Prometheus and InfluxDB. I was inspired to have our team put more effort into bringing native Prometheus support into InfluxDB. I am excited to announce that the first phase of this is now complete: the nightly builds of InfluxDB 1.4 now support the Prometheus remote read and write API natively. I’ll cover the details later in this post.

Why Support Prometheus, Aren't They Doing Similar Things?

I get this question a fair amount, so let me address this. The Prometheus server is focused squarely on metrics data and is meant to be an ephemeral pull-based store and monitoring system. Meanwhile, InfluxDB is focused on time series (metrics and events) and is meant to be used either as an ephemeral data store or as a store of record for data that lives forever. It’s a distinction that matters and has large implications for use cases appropriate to each project. For cases where you want to keep your data around or push individual events to the database, InfluxDB is a more appropriate choice. For ephemeral pull-based data, either server will suffice.

But Prometheus is much more than just the server. I see Prometheus as a set of standards and projects, with the server being just one part of a much greater whole. When viewed in that light, I think there are many ways for us to work with the overall Prometheus ecosystem and make meaningful contributions.

Prometheus as a Standard Metrics Format

Prometheus has developed a standard exposition format with excellent client libraries, and over 100 exporters that expose metrics in the Prometheus format. The service discovery code in Prometheus is also used in other projects, including our own Kapacitor. In addition to that, Prometheus has their Alert Manager project. We think this is quality work, and we intend to support the Prometheus exposition format as a first-class citizen in the InfluxData ecosystem.

Adding Prometheus Remote APIs to InfluxDB

Prometheus has support for a remote read and write API, which lets it store scraped data in other data stores. Writes get forwarded onto the remote store. For reads, the functionality is fairly basic in that it pushes down the start and stop time of the Prometheus query and the label matches (tags in Influx) that the query wants. All that data is streamed over the network, and the Prometheus server does the computation to return the result back to the user.

Previously, to integrate Prometheus with InfluxDB you had to use the example remote storage adapter. The drawback to this was that it is marked as an experimental example, not for production use. It also meant that you’d have another piece of infrastructure to operate and manage. Finally, it introduces unnecessary marshaling overhead.

Over the Labor Day weekend, I decided to bring native support for the Prometheus read and write API into InfluxDB. The implementation differs a bit from the example Prometheus implementation. All labels are converted to tags and the metric name is converted into a tag name like it is in the underlying Prometheus implementation. Everything is written to a measurement _ with a field name of f64. Basically, we’re ignoring the measurement and field and instead putting everything into tags. This mirrors the direction we’re going in the new data model and query language.

In the near term this model might be clunky to work with in InfluxQL, but it’s more ideal for pulling back data on the Prometheus remote read side of things. And with the new “IFQL” language, it will be in a structure that works perfectly with the language and the engine. This functionality should be viewed as a beta for the time being.

To get going with this you’ll have to work with the nightly InfluxDB build and set the remote sections of your Prometheus configs. Here’s an example of the config:

# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
remote_write:
- url: "http://localhost:8086/api/v1/prom/write?u=paul&p=foo&db=prometheus"
# Remote read configuration (for InfluxDB only at the moment).
remote_read:
- url: "http://localhost:8086/api/v1/prom/read?u=paul&p=foo&db=prometheus"

From the example you can see that the new endpoints are /api/v1/prom/read and /api/v1/prom/write. You use query parameters to pass in the database to write data to and optional user and password. It’s not ideal to keep the password in the config file so it would be nice to update Prometheus to be able to read this from an environment variable.

Once this gets rolled into our InfluxDB Cloud offering, our customers will be able to use our cloud product as a long-term store for their on-site Prometheus servers. Or for our InfluxDB Enterprise customers, they’ll have a clustered, highly available long-term data store for their Prometheus data.

We’ll look at other ways we can contribute to the open source Prometheus community and work to add more support for Prometheus’ standards and APIs to the Influx stack. PromCon 2017 was a great conference, and I’m inspired to push our products to work well within this new ecosystem of tools.