TL;DR InfluxDB’s MQTT Native Collector

By Susannah Brodnitz / Product
Oct 18, 2022

There are a lot of ways to get data into InfluxDB, including over a dozen client libraries, hundreds of Telegraf plugins, and Native Collectors. Native Collectors let you directly ingest data from a cloud broker to InfluxDB Cloud so you don’t need to install anything else or write extra code.

This blog will focus on the MQTT Native Collector. MQTT is one of the most popular IoT message brokers because it’s lightweight and works well in environments with limited bandwidth and intermittent network connectivity. The MQTT Native Collector is the fastest way to get your data into InfluxDB so you can analyze it, transform it, create visualizations, and set up alerts. That speed is very important for many IoT applications that rely on real-time analysis to keep smart devices running. To get started with the MQTT Native Collector you just need to have a message broker in place and an InfluxDB Cloud account.

When to use InfluxDB MQTT Native Collector

Because there are so many ways to get data into InfluxDB, it’s helpful to know when to use each method. A Native Collector is useful when you’re really busy and don’t want to go through the additional step of writing software using a client library to ingest data. But in that case you may also want to use Telegraf, and it includes processor and aggregator plugins to transform and downsample your data before it reaches InfluxDB.

However, Telegraf requires you to install it somewhere. If your message broker is in the cloud and you’re using InfluxDB Cloud there might not be a convenient place to install Telegraf, especially if you’re using a third party cloud service. And it can also be more efficient to keep your data pipeline in the cloud rather than needlessly downloading your data onto a server before uploading it back to InfluxDB Cloud. You can sometimes install Telegraf in a virtual machine in your cloud environment, such as if you’re using HiveMQ or Mosquito, but if you’re using a third party cloud-based message broker like HiveMQ Cloud that’s not possible.

The MQTT Native Collector is a great solution for people who want to keep their data in the cloud and get it into InfluxDB quickly. You save on time by getting rid of download or upload speed and bandwidth issues because you don’t need to download data to wherever you’re running Telegraf. You also save compute time associated with running Telegraf on a virtual machine.

Using a client library gives you the most flexibility because the code you write is specifically designed for your application, and that may be a good option if there are complex transformations you need to do on your data. But if you’re trying to ingest raw data directly from IoT devices to InfluxDB and speed matters most, the MQTT Native Collector is the simplest and quickest method.

Configuring & parsing MQTT data with InfluxDB’s Native Collector

It’s simple to configure native collection and get started. First you need to configure the IP address or URL of your MQTT message broker and any authentication parameters. Then you need to set the topic(s) you want to subscribe to. If there are multiple topics you want to subscribe to, you can repeat this process for each topic or you can use wildcards.

Topics generally come in the format my/fav/set/topics separated by tree levels. At any tree level you can use a + to indicate that you want to subscribe to anything from that tree level, such as my/+/set/topics or a # to indicate you want to subscribe to anything from that tree level down, such as my/fav/#.

Finally, you need to set parsing rules. This tells InfluxDB how to read raw data from the MQTT broker and where to put it. Data in an MQTT broker can be in any format, and InfluxDB needs to know the measurement, timestamp, fields, and tags. You don’t need parsing rules if you’re subscribing to a topic where the data is already in line protocol format.

One of the most common data formats is JSON, so InfluxDB has a specific parser already set up for it. If your data is in another format you can use string parsing to let InfluxDB know what variables in your data should be read in as measurements, fields, timestamps, and tags. It’s that simple and can be set up in minutes.

To learn more about InfluxDB’s MQTT Native Collector you can watch our series of videos on this topic.

Part one is an introduction to Native Collectors.

Part two explains when to use Native Collectors.

And part three gives examples for how to set up the MQTT Native Collector.