Fluentd and Redis Integration

Input and output integration overview

The Fluentd Input Plugin gathers metrics from Fluentd’s in_monitor plugin endpoint. It provides insights into various plugin metrics while allowing for custom configurations to reduce series cardinality.

The Redis Time Series output plugin is designed to publish metrics to a Redis efficiently.

Integration details

Fluentd

This plugin gathers metrics from the Fluentd plugin endpoint provided by the in_monitor plugin. It reads data from the /api/plugin.json resource and allows exclusion of specific plugins based on their type.

Redis

The Redis output plugin writes metrics to the Redis server.

Configuration

Fluentd

[[inputs.fluentd]]
  ## This plugin reads information exposed by fluentd (using /api/plugins.json endpoint).
  ##
  ## Endpoint:
  ## - only one URI is allowed
  ## - https is not supported
  endpoint = "http://localhost:24220/api/plugins.json"

  ## Define which plugins have to be excluded (based on "type" field - e.g. monitor_agent)
  exclude = [
    "monitor_agent",
    "dummy",
  ]

Redis

[[outputs.redistimeseries]]
  ## The address of the RedisTimeSeries server.
  address = "127.0.0.1:6379"

  ## Redis ACL credentials
  # username = ""
  # password = ""
  # database = 0

  ## Timeout for operations such as ping or sending metrics
  # timeout = "10s"

  ## Enable attempt to convert string fields to numeric values
  ## If "false" or in case the string value cannot be converted the string
  ## field will be dropped.
  # convert_string_fields = true

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  # insecure_skip_verify = false

Input and output integration examples

Fluentd

1. Basic Configuration: Set up the Fluentd Input Plugin to gather metrics from your Fluentd instance’s monitoring endpoint, ensuring you are able to track performance and usage statistics.
2. Excluding Plugins: Use the exclude option to ignore specific plugins’ metrics that are not necessary for your monitoring needs, streamlining data collection and focusing on what matters.
3. Custom Plugin ID: Implement the @id parameter in your Fluentd configuration to maintain a consistent plugin_id, which helps avoid issues with high series cardinality during frequent restarts.

Redis

1. Metrics Storage: Utilize the Redis output plugin to store time-series metrics collected from various sources directly into a Redis database for quick retrieval and analysis.
2. Dynamic Configuration: Adjust the address and other settings dynamically to publish metrics to different Redis instances based on the deployment environment.
3. String Field Conversion: Leverage the convert_string_fields option to automatically convert string metrics to numeric formats, ensuring that data is stored in the desired type for analytics.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.