MQTT and Graylog Integration

Input and output integration overview

The MQTT Telegraf plugin is designed to read from specified MQTT topics and create metrics, enabling users to leverage MQTT for real-time data collection and monitoring.

The Graylog plugin allows you to send Telegraf metrics to a Graylog server, utilizing the GELF format for structured logging.

Integration details

MQTT

The MQTT plugin allows for reading metrics from specified MQTT topics, creating metrics using supported input data formats. This plugin operates as a service input, which listens for incoming metrics or events rather than gathering them at set intervals like normal plugins. The flexibility of the plugin is enhanced with support for various broker URLs, topics, and connection features, including Quality of Service (QoS) levels and persistent sessions. Its configuration options incorporate global settings to modify metrics and handle startup errors effectively. It also supports secret-store configurations for securing username and password options, ensuring secure connections to MQTT servers.

Graylog

The Graylog plugin is designed for sending metrics to a Graylog instance using the GELF (Graylog Extended Log Format) format. GELF helps standardize the logging data, making it easier for systems to send and analyze logs. The plugin adheres to the GELF specification, which lays out requirements for specific fields within the payload. Notably, the timestamp must be in UNIX format, and if present, the plugin sends the timestamp as-is to Graylog without alterations. If omitted, it automatically generates a timestamp. Additionally, any extra fields not explicitly defined by the spec will be prefixed with an underscore, helping to keep the data organized and compliant with GELF’s requirements. This capability is particularly valuable for users monitoring applications and infrastructure in real-time, as it allows for seamless integration and improved visibility across multiple systems.

Configuration

MQTT

[[inputs.mqtt_consumer]]
  servers = ["tcp://127.0.0.1:1883"]
  topics = [
    "telegraf/host01/cpu",
    "telegraf/+/mem",
    "sensors/#",
  ]
  # topic_tag = "topic"
  # qos = 0
  # connection_timeout = "30s"
  # keepalive = "60s"
  # ping_timeout = "10s"
  # max_undelivered_messages = 1000
  # persistent_session = false
  # client_id = ""
  # username = "telegraf"
  # password = "metricsmetricsmetricsmetrics"
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  # insecure_skip_verify = false
  # client_trace = false
  data_format = "influx"
  # [[inputs.mqtt_consumer.topic_parsing]]
  #   topic = ""
  #   measurement = ""
  #   tags = ""
  #   fields = ""
  #   [inputs.mqtt_consumer.topic_parsing.types]
  #      key = type

Graylog

[[outputs.graylog]]
  ## Endpoints for your graylog instances.
  servers = ["udp://127.0.0.1:12201"]

  ## Connection timeout.
  # timeout = "5s"

  ## The field to use as the GELF short_message, if unset the static string
  ## "telegraf" will be used.
  ##   example: short_message_field = "message"
  # short_message_field = ""

  ## According to GELF payload specification, additional fields names must be prefixed
  ## with an underscore. Previous versions did not prefix custom field 'name' with underscore.
  ## Set to true for backward compatibility.
  # name_field_no_prefix = false

  ## Connection retry options
  ## Attempt to connect to the endpoints if the initial connection fails.
  ## If 'false', Telegraf will give up after 3 connection attempt and will
  ## exit with an error. If set to 'true', the plugin will retry to connect
  ## to the unconnected endpoints infinitely.
  # connection_retry = false
  ## Time to wait between connection retry attempts.
  # connection_retry_wait_time = "15s"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Input and output integration examples

MQTT

1. Smart Home Monitoring: Use the MQTT Consumer plugin to monitor various sensors in a smart home setup. In this scenario, the plugin can be configured to subscribe to topics for different devices, such as temperature, humidity, and energy consumption. By aggregating this data, homeowners can visualize trends and receive alerts for unusual patterns, enhancing the overall quality and efficiency of home automation systems.

2. IoT Environmental Sensing: Deploy the MQTT Consumer to gather environmental data from sensors distributed across different locations. For instance, this can include readings from air quality sensors, temperature sensors, and noise level meters. The plugin can be configured to extract relevant tags and fields from the MQTT topics which allows for detailed analyses and reporting on environmental conditions at scale, supporting better decision making for urban planning or environmental initiatives.

3. Real-Time Vehicle Tracking and Telemetry: Integrate the MQTT Consumer plugin within a vehicle telemetry system that collects data from various sensors in real-time. With the plugin, metrics related to vehicle performance, location, and fuel consumption can be sent to a centralized monitoring dashboard. This real-time telemetry data enables fleet managers to optimize routes, reduce fuel costs, and improve vehicle maintenance schedules through proactive data analysis.

4. Agricultural Monitoring System: Leverage this plugin to collect data from agricultural sensors that monitor soil moisture, crop health, and weather conditions. The MQTT Consumer can subscribe to multiple topics associated with farming equipment and environmental sensors, allowing farmers to make data-driven decisions to improve crop yields while also conserving resources, enhancing sustainability in agriculture.

Graylog

1. Enhanced Log Management for Cloud Applications: Use the Graylog Telegraf plugin to aggregate logs from cloud-deployed applications across multiple servers. By integrating this plugin, teams can centralize logging data, making it easier to troubleshoot issues, monitor application performance, and maintain compliance with logging standards.

2. Real-Time Security Monitoring: Leverage the Graylog plugin to collect and send security-related metrics and logs to a Graylog server for real-time analysis. This allows security teams to quickly identify anomalies, track potential breaches, and respond to incidents promptly by correlating logs from various sources within the infrastructure.

3. Dynamic Alerting and Notification System: Implement the Graylog plugin to enhance alerting mechanisms in your infrastructure. By sending metrics to Graylog, teams can set up dynamic alerts based on log patterns or unexpected behavior, enabling proactive monitoring and rapid incident response strategies.

4. Cross-Platform Log Consolidation: Use the Graylog plugin to facilitate cross-platform log consolidation across diverse environments such as on-premises, hybrid, and cloud. By standardizing logging in the GELF format, organizations can ensure consistent monitoring and troubleshooting practices, regardless of where their services are hosted.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.