Apache Zookeeper and MariaDB Integration

Input and output integration overview

The Zookeeper Telegraf plugin collects and reports metrics from Zookeeper servers, facilitating monitoring and performance analysis. It utilizes the ‘mntr’ command output to gather essential statistics critical for maintaining Zookeeper’s operational health.

This plugin writes metrics from Telegraf directly into MariaDB using parameterized SQL INSERT statements, offering a flexible way to store metrics in structured, relational tables.

Integration details

Apache Zookeeper

The Zookeeper plugin for Telegraf is designed to collect vital statistics from Zookeeper servers by executing the ‘mntr’ command. This plugin serves as a monitoring tool that captures important metrics related to Zookeeper’s performance, including connection details, latency, and various operational statistics, facilitating the assessment of the health and efficiency of Zookeeper deployments. In contrast to the Prometheus input plugin, which is recommended when the Prometheus metrics provider is enabled, the Zookeeper plugin accesses raw output from the ‘mntr’ command, rendering it tailored for configurations that do not adopt Prometheus for metrics reporting. This unique approach allows administrators to gather Java Properties formatted metrics directly from Zookeeper, ensuring comprehensive visibility into Zookeeper’s operational state and enabling timely responses to performance anomalies. It specifically excels in environments where Zookeeper operates as a centralized service for maintaining configuration information and names for distributed systems, thus providing immeasurable insights essential for troubleshooting and capacity planning.

MariaDB

The SQL output plugin in Telegraf enables direct writing of metrics into SQL-compatible databases like MariaDB by executing parameterized SQL statements. With support for the MySQL driver, the plugin seamlessly integrates with MariaDB for reliable, structured metric storage. This setup is ideal for users who prefer SQL-based analytics or want to store metrics alongside business data for unified querying. MariaDB is a community-developed, enterprise-grade fork of MySQL that emphasizes performance, security, and openness. The plugin supports inserting time series metrics into custom schemas, enabling flexible analytics and integrations with BI tools like Metabase or Grafana using SQL connectors.

Configuration

Apache Zookeeper

[[inputs.zookeeper]]
  ## An array of address to gather stats about. Specify an ip or hostname
  ## with port. ie localhost:2181, 10.0.0.1:2181, etc.

  ## If no servers are specified, then localhost is used as the host.
  ## If no port is specified, 2181 is used
  servers = [":2181"]

  ## Timeout for metric collections from all servers. Minimum timeout is "1s".
  # timeout = "5s"

  ## Float Parsing - the initial implementation forced any value unable to be
  ## parsed as an int to be a string. Setting this to "float" will attempt to
  ## parse float values as floats and not strings. This would break existing
  ## metrics and may cause issues if a value switches between a float and int.
  # parse_floats = "string"

  ## Optional TLS Config
  # enable_tls = false
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## If false, skip chain & host verification
  # insecure_skip_verify = true

MariaDB

[[outputs.sql]]
  ## Database driver
  ## Valid options: mssql (Microsoft SQL Server), mysql (MySQL), pgx (Postgres),
  ##  sqlite (SQLite3), snowflake (snowflake.com) clickhouse (ClickHouse)
  driver = "mysql"

  ## Data source name
  ## The format of the data source name is different for each database driver.
  ## See the plugin readme for details.
  data_source_name = "username:password@tcp(host:port)/dbname"

  ## Timestamp column name
  timestamp_column = "timestamp"

  ## Table creation template
  ## Available template variables:
  ##  {TABLE} - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS} - column definitions (list of quoted identifiers and types)
  table_template = "CREATE TABLE {TABLE}({COLUMNS})"

  ## SQL INSERT statement with placeholders. Telegraf will substitute values at runtime.
  ## table_template = "INSERT INTO metrics (timestamp, name, value, tags) VALUES (?, ?, ?, ?)"

  ## Table existence check template
  ## Available template variables:
  ##  {TABLE} - tablename as a quoted identifier
  table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  ## Initialization SQL
  init_sql = "SET sql_mode='ANSI_QUOTES';"

  ## Maximum amount of time a connection may be idle. "0s" means connections are
  ## never closed due to idle time.
  connection_max_idle_time = "0s"

  ## Maximum amount of time a connection may be reused. "0s" means connections
  ## are never closed due to age.
  connection_max_lifetime = "0s"

  ## Maximum number of connections in the idle connection pool. 0 means unlimited.
  connection_max_idle = 2

  ## Maximum number of open connections to the database. 0 means unlimited.
  connection_max_open = 0

  ## NOTE: Due to the way TOML is parsed, tables must be at the END of the
  ## plugin definition, otherwise additional config options are read as part of the
  ## table

  ## Metric type to SQL type conversion
  ## The values on the left are the data types Telegraf has and the values on
  ## the right are the data types Telegraf will use when sending to a database.
  ##
  ## The database values used must be data types the destination database
  ## understands. It is up to the user to ensure that the selected data type is
  ## available in the database they are using. Refer to your database
  ## documentation for what data types are available and supported.
  #[outputs.sql.convert]
  #  integer              = "INT"
  #  real                 = "DOUBLE"
  #  text                 = "TEXT"
  #  timestamp            = "TIMESTAMP"
  #  defaultvalue         = "TEXT"
  #  unsigned             = "UNSIGNED"
  #  bool                 = "BOOL"
  #  ## This setting controls the behavior of the unsigned value. By default the
  #  ## setting will take the integer value and append the unsigned value to it. The other
  #  ## option is "literal", which will use the actual value the user provides to
  #  ## the unsigned option. This is useful for a database like ClickHouse where
  #  ## the unsigned value should use a value like "uint64".
  #  # conversion_style = "unsigned_suffix"

Input and output integration examples

Apache Zookeeper

1. Cluster Health Monitoring: Integrate the Zookeeper plugin to monitor the health and performance of a distributed application relying on Zookeeper for configuration management and service discovery. By tracking metrics such as session count, latency, and data size, DevOps teams can identify potential issues before they escalate, ensuring high availability and reliability across applications.

2. Performance Benchmarks: Utilize the plugin to benchmark Zookeeper performance in varying workload scenarios. This not only helps in understanding how Zookeeper behaves under load but also assists in tuning configurations to optimize throughput and reduce latency during peak operations.

3. Alerting for Anomalies: Combine this plugin with alerting tools to create a proactive monitoring system that notifies engineers if specific Zookeeper metrics exceed threshold limits, such as open file descriptor counts or high latency values. This enables teams to respond promptly to issues that could impact service reliability.

4. Historical Data Analysis: Store the metrics collected by the Zookeeper plugin in a time-series database to analyze historical performance trends. This allows teams to evaluate the impact of changes over time, assess the effectiveness of scaling actions, and plan for future capacity needs.

MariaDB

1. Business Intelligence Integration: Store application performance metrics directly into MariaDB and connect it to BI tools like Metabase or Apache Superset. This setup allows blending of operational data with business KPIs for unified dashboards, enhancing visibility across departments.

2. Compliance Reporting with Historical Metrics: Use this plugin to log metrics into MariaDB for audit and compliance use cases. The relational model enables precise querying of past performance indicators with timestamped entries, supporting regulatory documentation.

3. Custom Alerting Based on SQL Logic: Insert metrics into MariaDB and use custom SQL queries to define alert thresholds or conditions. Combined with cron jobs or scheduled scripts, this enables advanced alerting workflows not possible with traditional metric platforms.

4. IoT Sensor Metrics Storage: Collect sensor data from IoT devices via Telegraf and store it in MariaDB using a normalized schema. This approach is cost-effective and integrates well with existing SQL-based systems for real-time or historical analysis.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.