NATS and Redis Integration

Input and output integration overview

The NATS Consumer Input Plugin enables real-time data consumption from NATS messaging subjects, integrating seamlessly into the Telegraf data pipeline for monitoring and metrics gathering.

The Redis plugin enables users to send metrics collected by Telegraf directly to Redis. This integration is ideal for applications that require robust time series data storage and analysis.

Integration details

NATS

The NATS Consumer Plugin allows Telegraf to read metrics from specified NATS subjects and create metrics based on supported input data formats. Utilizing a Queue Group allows multiple instances of Telegraf to read from a NATS cluster in parallel, enhancing throughput and reliability. This plugin also supports various authentication methods, including username/password, NATS credentials files, and nkey seed files, ensuring secure communication with the NATS servers. It is particularly useful in environments where data persistence and message reliability are critical, thanks to features such as JetStream that facilitate the consumption of historical messages. Additionally, the ability to configure various operational parameters makes this plugin suitable for high-throughput scenarios while maintaining performance integrity.

Redis

The Redis Telegraf plugin is designed for writing metrics to RedisTimeSeries, a specialized Redis database module for time series data. This plugin facilitates the integration of Telegraf with RedisTimeSeries, allowing for the efficient storage and retrieval of timestamped data. With RedisTimeSeries, users can take advantage of enhanced capabilities for managing time series data, including aggregated views and range queries. The plugin offers various configuration options to enable the flexibility needed to connect securely to your Redis database, including support for Authentication, Timeouts, data type conversions, and TLS configurations. The underlying technology leverages Redis’ efficiency and scalability, making it an excellent choice for high-volume metric environments, where real-time processing is essential.

Configuration

NATS

[[inputs.nats_consumer]]
  ## urls of NATS servers
  servers = ["nats://localhost:4222"]

  ## subject(s) to consume
  ## If you use jetstream you need to set the subjects
  ## in jetstream_subjects
  subjects = ["telegraf"]

  ## jetstream subjects
  ## jetstream is a streaming technology inside of nats.
  ## With jetstream the nats-server persists messages and
  ## a consumer can consume historical messages. This is
  ## useful when telegraf needs to restart it don't miss a
  ## message. You need to configure the nats-server.
  ## https://docs.nats.io/nats-concepts/jetstream.
  jetstream_subjects = ["js_telegraf"]

  ## name a queue group
  queue_group = "telegraf_consumers"

  ## Optional authentication with username and password credentials
  # username = ""
  # password = ""

  ## Optional authentication with NATS credentials file (NATS 2.0)
  # credentials = "/etc/telegraf/nats.creds"

  ## Optional authentication with nkey seed file (NATS 2.0)
  # nkey_seed = "/etc/telegraf/seed.txt"

  ## Use Transport Layer Security
  # secure = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Sets the limits for pending msgs and bytes for each subscription
  ## These shouldn't need to be adjusted except in very high throughput scenarios
  # pending_message_limit = 65536
  # pending_bytes_limit = 67108864

  ## Max undelivered messages
  ## This plugin uses tracking metrics, which ensure messages are read to
  ## outputs before acknowledging them to the original broker to ensure data
  ## is not lost. This option sets the maximum messages to read from the
  ## broker that have not been written by an output.
  ##
  ## This value needs to be picked with awareness of the agent's
  ## metric_batch_size value as well. Setting max undelivered messages too high
  ## can result in a constant stream of data batches to the output. While
  ## setting it too low may never flush the broker's messages.
  # max_undelivered_messages = 1000

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

Redis

[[outputs.redistimeseries]]
  ## The address of the RedisTimeSeries server.
  address = "127.0.0.1:6379"

  ## Redis ACL credentials
  # username = ""
  # password = ""
  # database = 0

  ## Timeout for operations such as ping or sending metrics
  # timeout = "10s"

  ## Enable attempt to convert string fields to numeric values
  ## If "false" or in case the string value cannot be converted the string
  ## field will be dropped.
  # convert_string_fields = true

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  # insecure_skip_verify = false

Input and output integration examples

NATS

1. Real-Time Analytics Dashboard: Utilize the NATS plugin to gather metrics from various NATS subjects in real time and feed them into a centralized analytics dashboard. This setup allows for immediate visibility into live application performance, enabling teams to react swiftly to operational issues or performance degradation.

2. Distributed System Monitoring: Deploy multiple instances of Telegraf configured with the NATS plugin across a distributed architecture. This approach allows teams to aggregate metrics from various microservices efficiently, providing a holistic view of system health and performance while ensuring no messages are dropped during transmission.

3. Historical Message Recovery: Leverage the capabilities of NATS JetStream along with this plugin to recover and process historical messages after Telegraf has been restarted. This feature is particularly beneficial for applications that require high reliability, ensuring that no critical metrics are lost even in case of service disruptions.

4. Dynamic Load Balancing: Implement a dynamic load balancing scenario where Telegraf instances consume messages from a NATS cluster based on load. Adjust the queue group settings to control the number of active consumers, allowing for better resource utilization and performance scaling as demand fluctuations occur.

Redis

1. Monitoring IoT Sensor Data: Utilize the Redis Telegraf plugin to collect and store data from IoT sensors in real-time. By connecting the plugin to a RedisTimeSeries database, users can analyze trends in temperature, humidity, or other environmental factors. The ability to query historical sensor data efficiently will aid in predictive maintenance and help in resource management.

2. Financial Market Data Aggregation: Employ this plugin to track and store time-sensitive financial data from various sources. By sending metrics to Redis, financial institutions can aggregate and analyze market trends or price changes over time, providing them with actionable insights derived from reliable time series analytics.

3. Application Performance Monitoring (APM): Implement the Redis plugin for gathering application performance metrics such as response times and CPU usage. Users can visualize their application’s performance over time with RedisTimeSeries, allowing them to identify bottlenecks and optimize resource allocation swiftly.

4. Energy Consumption Tracking: Leverage this plugin to monitor energy usage in buildings over time. By integrating with smart meters and sending data to RedisTimeSeries, municipalities or enterprises can analyze energy consumption patterns, helping to implement energy-saving measures and sustainability practices.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.