Mesos and Clickhouse Integration

Input and output integration overview

This input plugin gathers metrics from Mesos.

Telegraf’s SQL plugin sends collected metrics to an SQL database using a straightforward table schema and dynamic column generation. When configured for ClickHouse, it adjusts DSN formatting and type conversion settings to ensure seamless data integration.

Integration details

Mesos

The Mesos plugin for Telegraf is designed to collect and report metrics from Apache Mesos clusters, which is essential for monitoring and observability in container orchestration and resource management. Mesos, known for its scalability and ability to manage diverse workloads, generates various metrics about resource usage, tasks, frameworks, and overall system performance. By utilizing this plugin, users can track the health and efficiency of their Mesos clusters, gather insights into resource distribution, and ensure that applications receive the necessary resources in a timely manner. The configuration allows users to specify the relevant Mesos master’s details, along with the desired metric groups to collect, making it adaptable to different deployments and monitoring needs. Overall, this plugin integrates seamlessly within the Telegraf collection pipeline, supporting detailed observability for cloud-native environments.

Clickhouse

Telegraf’s SQL plugin is engineered to write metric data into an SQL database by dynamically creating tables and columns based on incoming metrics. When configured for ClickHouse, it utilizes the clickhouse-go v1.5.4 driver, which employs a unique DSN format and a set of specialized type conversion rules to map Telegraf’s data types directly to ClickHouse’s native types. This approach ensures optimal storage and retrieval performance in high-throughput environments, making it well-suited for real-time analytics and large-scale data warehousing. The dynamic schema creation and precise type mapping enable detailed time-series data logging, crucial for monitoring modern, distributed systems.

Configuration

Mesos

[[inputs.mesos]]
  ## Timeout, in ms.
  timeout = 100

  ## A list of Mesos masters.
  masters = ["http://localhost:5050"]

  ## Master metrics groups to be collected, by default, all enabled.
  master_collections = [
    "resources",
    "master",
    "system",
    "agents",
    "frameworks",
    "framework_offers",
    "tasks",
    "messages",
    "evqueue",
    "registrar",
    "allocator",
  ]

  ## A list of Mesos slaves, default is []
  # slaves = []

  ## Slave metrics groups to be collected, by default, all enabled.
  # slave_collections = [
  #   "resources",
  #   "agent",
  #   "system",
  #   "executors",
  #   "tasks",
  #   "messages",
  # ]

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Clickhouse

[[outputs.sql]]
  ## Database driver
  ## Valid options include mssql, mysql, pgx, sqlite, snowflake, clickhouse
  driver = "clickhouse"

  ## Data source name
  ## For ClickHouse, the DSN follows the clickhouse-go v1.5.4 format.
  ## Example DSN: "tcp://localhost:9000?debug=true"
  data_source_name = "tcp://localhost:9000?debug=true"

  ## Timestamp column name
  timestamp_column = "timestamp"

  ## Table creation template
  ## Available template variables:
  ##  {TABLE}        - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS}      - column definitions (list of quoted identifiers and types)
  table_template = "CREATE TABLE {TABLE} ({COLUMNS})"

  ## Table existence check template
  ## Available template variables:
  ##  {TABLE} - table name as a quoted identifier
  table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  ## Initialization SQL (optional)
  init_sql = ""

  ## Maximum amount of time a connection may be idle. "0s" means connections are never closed due to idle time.
  connection_max_idle_time = "0s"

  ## Maximum amount of time a connection may be reused. "0s" means connections are never closed due to age.
  connection_max_lifetime = "0s"

  ## Maximum number of connections in the idle connection pool. 0 means unlimited.
  connection_max_idle = 2

  ## Maximum number of open connections to the database. 0 means unlimited.
  connection_max_open = 0

  ## Metric type to SQL type conversion for ClickHouse.
  ## The conversion maps Telegraf metric types to ClickHouse native data types.
  [outputs.sql.convert]
    conversion_style = "literal"
    integer          = "Int64"
    text             = "String"
    timestamp        = "DateTime"
    defaultvalue     = "String"
    unsigned         = "UInt64"
    bool             = "UInt8"
    real             = "Float64"

Input and output integration examples

Mesos

1. Resource Utilization Monitoring: Use the Mesos plugin to continually monitor CPU, memory, and disk usage across your Mesos cluster. For a rapidly scaling application, tracking these metrics helps ensure that resources are dynamically allocated according to workloads, preventing bottlenecks and optimizing performance.

2. Framework Performance Analysis: Integrate this plugin to measure the performance of different frameworks running on Mesos. By comparing active frameworks and their task success rates, you can identify which frameworks provide the best resource efficiency or may require optimization.

3. Alerts for System Health: Set up alerts based on metrics collected by the Mesos plugin to notify engineering teams when resource utilization exceeds key thresholds or when specific tasks fail. This allows for proactive intervention and maintenance before critical failures occur.

4. Capacity Planning: Utilize gathered metrics to analyze historical resource usage patterns to assist in capacity planning. By understanding peak loads and resource utilization trends, teams can make informed decisions on scaling infrastructure and deploying additional resources as needed.

Clickhouse

1. Real-Time Analytics for High-Volume Data: Use the plugin to feed streaming metrics from large-scale systems into ClickHouse. This setup supports ultra-fast query performance and near real-time analytics, ideal for monitoring high-traffic applications.

2. Time-Series Data Warehousing: Integrate the plugin with ClickHouse to create a robust time-series data warehouse. This use case allows organizations to store detailed historical metrics and perform complex queries for trend analysis and capacity planning.

3. Scalable Monitoring in Distributed Environments: Leverage the plugin to dynamically create tables per metric type in ClickHouse, making it easier to manage and query data from a multitude of distributed systems without prior schema definitions.

4. Optimized Storage for IoT Deployments: Deploy the plugin to ingest data from IoT sensors into ClickHouse. Its efficient schema creation and native type mapping facilitate the handling of massive volumes of data, enabling real-time monitoring and predictive maintenance.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.