NATS and Graylog Integration

Input and output integration overview

The NATS Consumer Input Plugin enables real-time data consumption from NATS messaging subjects, integrating seamlessly into the Telegraf data pipeline for monitoring and metrics gathering.

The Graylog plugin allows you to send Telegraf metrics to a Graylog server, utilizing the GELF format for structured logging.

Integration details

NATS

The NATS Consumer Plugin allows Telegraf to read metrics from specified NATS subjects and create metrics based on supported input data formats. Utilizing a Queue Group allows multiple instances of Telegraf to read from a NATS cluster in parallel, enhancing throughput and reliability. This plugin also supports various authentication methods, including username/password, NATS credentials files, and nkey seed files, ensuring secure communication with the NATS servers. It is particularly useful in environments where data persistence and message reliability are critical, thanks to features such as JetStream that facilitate the consumption of historical messages. Additionally, the ability to configure various operational parameters makes this plugin suitable for high-throughput scenarios while maintaining performance integrity.

Graylog

The Graylog plugin is designed for sending metrics to a Graylog instance using the GELF (Graylog Extended Log Format) format. GELF helps standardize the logging data, making it easier for systems to send and analyze logs. The plugin adheres to the GELF specification, which lays out requirements for specific fields within the payload. Notably, the timestamp must be in UNIX format, and if present, the plugin sends the timestamp as-is to Graylog without alterations. If omitted, it automatically generates a timestamp. Additionally, any extra fields not explicitly defined by the spec will be prefixed with an underscore, helping to keep the data organized and compliant with GELF’s requirements. This capability is particularly valuable for users monitoring applications and infrastructure in real-time, as it allows for seamless integration and improved visibility across multiple systems.

Configuration

NATS

[[inputs.nats_consumer]]
  ## urls of NATS servers
  servers = ["nats://localhost:4222"]

  ## subject(s) to consume
  ## If you use jetstream you need to set the subjects
  ## in jetstream_subjects
  subjects = ["telegraf"]

  ## jetstream subjects
  ## jetstream is a streaming technology inside of nats.
  ## With jetstream the nats-server persists messages and
  ## a consumer can consume historical messages. This is
  ## useful when telegraf needs to restart it don't miss a
  ## message. You need to configure the nats-server.
  ## https://docs.nats.io/nats-concepts/jetstream.
  jetstream_subjects = ["js_telegraf"]

  ## name a queue group
  queue_group = "telegraf_consumers"

  ## Optional authentication with username and password credentials
  # username = ""
  # password = ""

  ## Optional authentication with NATS credentials file (NATS 2.0)
  # credentials = "/etc/telegraf/nats.creds"

  ## Optional authentication with nkey seed file (NATS 2.0)
  # nkey_seed = "/etc/telegraf/seed.txt"

  ## Use Transport Layer Security
  # secure = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Sets the limits for pending msgs and bytes for each subscription
  ## These shouldn't need to be adjusted except in very high throughput scenarios
  # pending_message_limit = 65536
  # pending_bytes_limit = 67108864

  ## Max undelivered messages
  ## This plugin uses tracking metrics, which ensure messages are read to
  ## outputs before acknowledging them to the original broker to ensure data
  ## is not lost. This option sets the maximum messages to read from the
  ## broker that have not been written by an output.
  ##
  ## This value needs to be picked with awareness of the agent's
  ## metric_batch_size value as well. Setting max undelivered messages too high
  ## can result in a constant stream of data batches to the output. While
  ## setting it too low may never flush the broker's messages.
  # max_undelivered_messages = 1000

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

Graylog

[[outputs.graylog]]
  ## Endpoints for your graylog instances.
  servers = ["udp://127.0.0.1:12201"]

  ## Connection timeout.
  # timeout = "5s"

  ## The field to use as the GELF short_message, if unset the static string
  ## "telegraf" will be used.
  ##   example: short_message_field = "message"
  # short_message_field = ""

  ## According to GELF payload specification, additional fields names must be prefixed
  ## with an underscore. Previous versions did not prefix custom field 'name' with underscore.
  ## Set to true for backward compatibility.
  # name_field_no_prefix = false

  ## Connection retry options
  ## Attempt to connect to the endpoints if the initial connection fails.
  ## If 'false', Telegraf will give up after 3 connection attempt and will
  ## exit with an error. If set to 'true', the plugin will retry to connect
  ## to the unconnected endpoints infinitely.
  # connection_retry = false
  ## Time to wait between connection retry attempts.
  # connection_retry_wait_time = "15s"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Input and output integration examples

NATS

1. Real-Time Analytics Dashboard: Utilize the NATS plugin to gather metrics from various NATS subjects in real time and feed them into a centralized analytics dashboard. This setup allows for immediate visibility into live application performance, enabling teams to react swiftly to operational issues or performance degradation.

2. Distributed System Monitoring: Deploy multiple instances of Telegraf configured with the NATS plugin across a distributed architecture. This approach allows teams to aggregate metrics from various microservices efficiently, providing a holistic view of system health and performance while ensuring no messages are dropped during transmission.

3. Historical Message Recovery: Leverage the capabilities of NATS JetStream along with this plugin to recover and process historical messages after Telegraf has been restarted. This feature is particularly beneficial for applications that require high reliability, ensuring that no critical metrics are lost even in case of service disruptions.

4. Dynamic Load Balancing: Implement a dynamic load balancing scenario where Telegraf instances consume messages from a NATS cluster based on load. Adjust the queue group settings to control the number of active consumers, allowing for better resource utilization and performance scaling as demand fluctuations occur.

Graylog

1. Enhanced Log Management for Cloud Applications: Use the Graylog Telegraf plugin to aggregate logs from cloud-deployed applications across multiple servers. By integrating this plugin, teams can centralize logging data, making it easier to troubleshoot issues, monitor application performance, and maintain compliance with logging standards.

2. Real-Time Security Monitoring: Leverage the Graylog plugin to collect and send security-related metrics and logs to a Graylog server for real-time analysis. This allows security teams to quickly identify anomalies, track potential breaches, and respond to incidents promptly by correlating logs from various sources within the infrastructure.

3. Dynamic Alerting and Notification System: Implement the Graylog plugin to enhance alerting mechanisms in your infrastructure. By sending metrics to Graylog, teams can set up dynamic alerts based on log patterns or unexpected behavior, enabling proactive monitoring and rapid incident response strategies.

4. Cross-Platform Log Consolidation: Use the Graylog plugin to facilitate cross-platform log consolidation across diverse environments such as on-premises, hybrid, and cloud. By standardizing logging in the GELF format, organizations can ensure consistent monitoring and troubleshooting practices, regardless of where their services are hosted.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.