Apache Zookeeper and SQLite Integration

Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.

info

This is not the recommended configuration for real-time query at scale. For query and compression optimization, high-speed ingest, and high availability, you may want to consider Apache Zookeeper and InfluxDB.

5B+

Telegraf downloads

#1

Time series database
Source: DB Engines

1B+

Downloads of InfluxDB

2,800+

Contributors

Table of Contents

Powerful Performance, Limitless Scale

Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.

See Ways to Get Started

Input and output integration overview

The Zookeeper Telegraf plugin collects and reports metrics from Zookeeper servers, facilitating monitoring and performance analysis. It utilizes the ‘mntr’ command output to gather essential statistics critical for maintaining Zookeeper’s operational health.

Telegraf’s SQL output plugin stores metrics in an SQL database by creating tables dynamically for each metric type. When configured for SQLite, it utilizes a file-based DSN and a minimal SQL schema tailored for lightweight, embedded database usage.

Integration details

Apache Zookeeper

The Zookeeper plugin for Telegraf is designed to collect vital statistics from Zookeeper servers by executing the ‘mntr’ command. This plugin serves as a monitoring tool that captures important metrics related to Zookeeper’s performance, including connection details, latency, and various operational statistics, facilitating the assessment of the health and efficiency of Zookeeper deployments. In contrast to the Prometheus input plugin, which is recommended when the Prometheus metrics provider is enabled, the Zookeeper plugin accesses raw output from the ‘mntr’ command, rendering it tailored for configurations that do not adopt Prometheus for metrics reporting. This unique approach allows administrators to gather Java Properties formatted metrics directly from Zookeeper, ensuring comprehensive visibility into Zookeeper’s operational state and enabling timely responses to performance anomalies. It specifically excels in environments where Zookeeper operates as a centralized service for maintaining configuration information and names for distributed systems, thus providing immeasurable insights essential for troubleshooting and capacity planning.

SQLite

The SQL output plugin writes Telegraf metrics to an SQL database using a dynamic schema where each metric type corresponds to a table. For SQLite, the plugin uses the modernc.org/sqlite driver and requires a DSN in the format of a file URI (e.g., ‘file:/path/to/telegraf.db?cache=shared’). This configuration leverages standard ANSI SQL for table creation and data insertion, ensuring compatibility with SQLite’s capabilities.

Configuration

Apache Zookeeper

[[inputs.zookeeper]]
  ## An array of address to gather stats about. Specify an ip or hostname
  ## with port. ie localhost:2181, 10.0.0.1:2181, etc.

  ## If no servers are specified, then localhost is used as the host.
  ## If no port is specified, 2181 is used
  servers = [":2181"]

  ## Timeout for metric collections from all servers. Minimum timeout is "1s".
  # timeout = "5s"

  ## Float Parsing - the initial implementation forced any value unable to be
  ## parsed as an int to be a string. Setting this to "float" will attempt to
  ## parse float values as floats and not strings. This would break existing
  ## metrics and may cause issues if a value switches between a float and int.
  # parse_floats = "string"

  ## Optional TLS Config
  # enable_tls = false
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## If false, skip chain & host verification
  # insecure_skip_verify = true

SQLite

[[outputs.sql]]
  ## Database driver
  ## Valid options: mssql (Microsoft SQL Server), mysql (MySQL), pgx (Postgres),
  ## sqlite (SQLite3), snowflake (snowflake.com), clickhouse (ClickHouse)
  driver = "sqlite"

  ## Data source name
  ## For SQLite, the DSN is a filename or URL with the scheme "file:".
  ## Example: "file:/path/to/telegraf.db?cache=shared"
  data_source_name = "file:/path/to/telegraf.db?cache=shared"

  ## Timestamp column name
  timestamp_column = "timestamp"

  ## Table creation template
  ## Available template variables:
  ##  {TABLE}        - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS}      - column definitions (list of quoted identifiers and types)
  table_template = "CREATE TABLE {TABLE} ({COLUMNS})"

  ## Table existence check template
  ## Available template variables:
  ##  {TABLE} - table name as a quoted identifier
  table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  ## Initialization SQL (optional)
  init_sql = ""

  ## Maximum amount of time a connection may be idle. "0s" means connections are never closed due to idle time.
  connection_max_idle_time = "0s"

  ## Maximum amount of time a connection may be reused. "0s" means connections are never closed due to age.
  connection_max_lifetime = "0s"

  ## Maximum number of connections in the idle connection pool. 0 means unlimited.
  connection_max_idle = 2

  ## Maximum number of open connections to the database. 0 means unlimited.
  connection_max_open = 0

  ## Metric type to SQL type conversion
  ## The values on the left are the data types Telegraf has and the values on the right are the SQL types used when writing to SQLite.
  #[outputs.sql.convert]
  #  integer       = "INT"
  #  real          = "DOUBLE"
  #  text          = "TEXT"
  #  timestamp     = "TIMESTAMP"
  #  defaultvalue  = "TEXT"
  #  unsigned      = "UNSIGNED"
  #  bool          = "BOOL"

Input and output integration examples

Apache Zookeeper

  1. Cluster Health Monitoring: Integrate the Zookeeper plugin to monitor the health and performance of a distributed application relying on Zookeeper for configuration management and service discovery. By tracking metrics such as session count, latency, and data size, DevOps teams can identify potential issues before they escalate, ensuring high availability and reliability across applications.

  2. Performance Benchmarks: Utilize the plugin to benchmark Zookeeper performance in varying workload scenarios. This not only helps in understanding how Zookeeper behaves under load but also assists in tuning configurations to optimize throughput and reduce latency during peak operations.

  3. Alerting for Anomalies: Combine this plugin with alerting tools to create a proactive monitoring system that notifies engineers if specific Zookeeper metrics exceed threshold limits, such as open file descriptor counts or high latency values. This enables teams to respond promptly to issues that could impact service reliability.

  4. Historical Data Analysis: Store the metrics collected by the Zookeeper plugin in a time-series database to analyze historical performance trends. This allows teams to evaluate the impact of changes over time, assess the effectiveness of scaling actions, and plan for future capacity needs.

SQLite

  1. Local Monitoring Storage: Configure the plugin to write metrics to a local SQLite database file. This is ideal for lightweight deployments where setting up a full-scale database server is not required.
  2. Embedded Applications: Use SQLite as the backend for applications embedded in edge devices, benefiting from its file-based architecture and minimal resource requirements.
  3. Quick Setup for Testing: Leverage SQLite’s ease of use to quickly set up a testing environment for Telegraf metrics collection without the need for external database services.
  4. Custom Schema Management: Adjust the table creation templates to predefine your schema if you require specific column types or indexes, ensuring compatibility with your application’s needs.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.

Powerful Performance, Limitless Scale

Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.

See Ways to Get Started

Related Integrations

HTTP and InfluxDB Integration

The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.

View Integration

Kafka and InfluxDB Integration

This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.

View Integration

Kinesis and InfluxDB Integration

The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.

View Integration