Telegraf Deployment Strategies with Docker Compose

By Community / Product
Aug 21, 2023

Navigate to:

This article, written by Shan Desai, was originally published on his blog and is reposted here with permission. Shan is a Software engineer currently employed at Emerson Discrete Automation and is an Open-Source Contributor / DIY Tech Enthusiast currently working with Industrial IoT.

Telegraf is widely used as a metric aggregation tool thanks to the diverse number of plugins it provides that interface with a multitude of systems without having to write complex software logic. Telegraf is a front-runner in the Low-Code/No-Code paradigm in system operations.

However, Low-Code/No-Code tools can make it complicated to maintain a lot of additional – yet necessary – information like credentials in other subsystems. Although standard practice dictates the use of Environment Variables within Telegraf, with version 1.27 and beyond Secret Stores come in handy for passing credentials into Telegraf plugins without having to pass environment variables explicitly, especially when using a containerization tool like Docker.

This post goes through some strategies for deploying a Telegraf Docker Container using the Docker Compose v2 tool that may help intermediate to advanced users decide on how to organize their stack configurations appropriately.

Using Docker Secrets

Docker Compose v2 specifications provide a useful Secrets feature that users can use for standalone Compose Application Stacks and not just in Docker Swarm mode. Docker Secrets mount the environment variables that contain credentials for other subsystems into the Telegraf Container as files. The Docker Secret Store plugin reads these secret files and passes them to the respective plugins in a safe manner that avoids environment variables that could potentially leak secrets. The secrets are now hidden behind runtime secret-files only visible inside the targeted container.

Standard method with environment variables

As an example, it is possible to pass the credentials to a plugin via the environment variable placeholder in a Telegraf configuration file where the credentials for a plugin exist in a .env file (e.g. MQTT input plugin)

MQTT_USERNAME=telegraf
MQTT_PASSWORD=superSecurePasswordMQTT

A telegraf configuration file for the MQTT Input plugin can be as follows:

[[inputs.mqtt_consumer]]

  servers = [ "tcp://<broker_ip>:1883" ]
  topics = [ "test1/#", "test2/+/topic" ]
  username = "${MQTT_USERNAME}"
  password = "${MQTT_PASSWORD}"

A Docker Compose File for the example can be as follows:

services:
  telegraf:
    image: docker.io/telegraf:latest
   container_name: telegraf
   environment:
     - MQTT_USERNAME=${MQTT_USERNAME}
     - MQTT_PASSWORD=${MQTT_PASSWORD}
   volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro

When the container is brought up, Telegraf interpolates the values of the environment variables to connect to the MQTT Broker. This assumes that the .env file and the docker-compose.yml file are in the same directory. This approach works well; however, a simple inspection of the running container may yield the credential values via the environment variable by a command such as:

docker compose exec telegraf env

Or

docker inspect -f “{{ .Config.Env }}” < telegraf_container_name / telegraf_container_id>

With Docker Secrets and Telegraf Docker Secret Store Plugin

With Docker Secrets we can change the Compose file to pick the environment variables up from the .env file and let Docker Compose mount these values as files under the /run/secrets directory in the Telegraf container as follows:

services: 
  image: docker.io/telegraf:latest
  container_name: telegraf
  secrets:
    - source: mqtt_username
      mode: 0444
   - source: mqtt_password
     mode: 0444
  volumes: 
./telegraf.conf:/etc/telegraf/telegraf.conf:ro

secrets: 
  mqtt_username: 
    environment: MQTT_USERNAME
  mqtt_password: 
    environment: MQTT_PASSWORD

Docker Compose mounts the values of MQTT_USERNAME and MQTT_PASSWORD into the /run/secrets/mqtt_username and /run/secrets/mqtt_password file in the container at runtime. We also set the file permissions to world-readable (0444) so that all users within the container can read the values.

Similarly, we adapt the Telegraf configuration file with the Docker Secret Store plugin and the MQTT credentials as follows:

[[ inputs.mqtt_consumer ]]
  servers = [ “tcp://<broker_ip>:1883” ]
  topics = [ "test1/#", "test2/+/topic" ]
  username = “@{custom_secretstore:mqtt_username}”
  password = “@{custom_secretstore:mqtt_password}”

[[ secretstores.docker ]]
  id = “custom_secretstore”

Bringing the container up with docker compose up lets Telegraf connect to the MQTT Broker and subscribe to the required topics.

A benefit with this method is that the previously visible environment variables within the Docker container are now safely mounted into the container at runtime and no longer visible upon inspection via

docker compose exec telegraf env

Or

docker inspect -f “{{ .Config.Env }}” <telegraf_container_name/ telegraf_container_id>

This approach provides a safer usage of credentials.

Splitting Telegraf configuration files

There may be situations where a Telegraf Configuration file starts to grow in size and the number of plugin configurations and splitting it into individual files makes it easier to maintain. Telegraf is able to consume multiple configuration plugin files and to process them one-by-one to run the Telegraf agent as if these files were merged together. (It is not a simple merge, however, so the source code has to undertake some additional effort.)

It is possible to mount a directory with split configuration files to the Telegraf Compose service as follows:

telegraf/
| - inputs.<plugin_name#1>.conf
| - inputs.<plugin_name#2>.conf
| - outputs.<plugin_name#1>.conf
| - outputs.<plugin_name#2>.conf
| - processors.<plugin_name#1>.conf
| - processors.<plugin_name#2>.conf
| - secretstores.<plugin_name>.conf

The respective Compose file is as follows:

services:
  telegraf:
    image: telegraf:latest
    container_name: telegraf
    volumes:
      - ./telegraf:/etc/telegraf/telegraf.d/

We notify Telegraf that the configuration is to be picked up from a directory i.e. /etc/telegraf/telegraf.d/ and not a standalone configuration file here.

NOTE: It is recommended to use the order parameter in processor plugins to let Telegraf know in which order the data coming in from input plugins needs to be manipulated.

Of course, the split configuration files can be used together with Docker Secrets.

Docker config for Telegraf configuration file

There is also a possibility where a Telegraf configuration file can be mounted via the Docker Configs feature instead of volume mounts. According to the Compose specification document for Docker Configs:

Configs allow services to adapt their behavior without the need to rebuild a Docker image.

Mounting Telegraf Configurations files via Docker Config is seldom used in deployments for standalone Compose applications. However they may be beneficial when working with custom Docker images for Telegraf.

A common Compose file for Telegraf:

services:
  image: telegraf:latest
  container_name: telegraf
  Volumes:
./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro

Can be transformed to let Compose mount the Telegraf configuration file via Docker Config as follows:

services:
  image: telegraf:latest
  container_name: telegraf
  command: --config /telegraf_conf
  configs:
    - telegraf_conf

configs:
  telegraf_conf:
    file: ./telegraf/telegraf.conf

By default, the configuration is mounted in the root of the container with the name of the docker config mapped to the file of the configuration file. When using such a configuration it is essential to mention within the command parameter where Telegraf should load the configuration file from i.e. command: –config /telegraf_conf.

Inference

This guide provides a baseline for how engineers can design their Telegraf Configuration file(s) with Docker Compose v2 as a deployment tool. This is not an exhaustive guide by any means, but it explores some new features in Telegraf (i.e., Docker Secret Store) and some less familiar Docker Compose v2 specifications that may not be often used in deployments.