Amazon CloudWatch and Clickhouse Integration

Input and output integration overview

This plugin will pull Metric Statistics from Amazon CloudWatch, streamlining the process of monitoring and analyzing AWS resources.

Telegraf’s SQL output plugin sends collected metrics to an SQL database using a straightforward table schema and dynamic column generation. When configured for ClickHouse, it adjusts DSN formatting and type conversion settings to ensure seamless data integration.

Integration details

Amazon CloudWatch

The Amazon CloudWatch Plugin allows users to pull detailed metric statistics from Amazon’s CloudWatch service. As a monitoring solution, CloudWatch enables users to track various metrics related to AWS resources and applications, facilitating improved operational and performance insights. The plugin uses a structured authentication method that prioritizes security and flexibility through a combination of STS (Security Token Service), shared credentials, environment variables, and EC2 instance profiles, ensuring robust access control to AWS resources. Key features include the ability to define specific metric namespaces, aggregated periods for metrics, and optional inclusion of linked accounts for cross-account monitoring. A significant aspect of this plugin is its capacity to handle both sparse and dense metric formats, allowing for varied output structures depending on user preference. Thus, it supports versatile use cases in cloud monitoring and analytics by providing comprehensive, timely data directly from CloudWatch.

Clickhouse

The SQL output plugin is designed to store Telegraf metrics in an SQL database using a simple, hard-coded schema. Each metric type gets its own table, and columns are generated for every tag and field, with an optional timestamp column. For ClickHouse, the plugin leverages a specialized DSN format as defined by clickhouse-go v1.5.4 and customizes metric type conversion to align with ClickHouse data types. This ensures that integers, texts, timestamps, booleans, and real numbers are mapped to ClickHouse’s native types such as Int64, String, DateTime, UInt8, and Float64 respectively.

Configuration

Amazon CloudWatch

[[inputs.cloudwatch]]
  region = "us-east-1"
  # access_key = ""
  # secret_key = ""
  # token = ""
  # role_arn = ""
  # web_identity_token_file = ""
  # role_session_name = ""
  # profile = ""
  # shared_credential_file = ""
  # include_linked_accounts = false
  # endpoint_url = ""
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"
  period = "5m"
  delay = "5m"
  interval = "5m"
  #recently_active = "PT3H"
  # cache_ttl = "1h"
  namespaces = ["AWS/ELB"]
  # metric_format = "sparse"
  # ratelimit = 25
  # timeout = "5s"
  # batch_size = 500
  # statistic_include = ["average", "sum", "minimum", "maximum", sample_count]
  # statistic_exclude = []
  # [[inputs.cloudwatch.metrics]]
  #  names = ["Latency", "RequestCount"]
  #  [[inputs.cloudwatch.metrics.dimensions]]
  #    name = "LoadBalancerName"
  #    value = "p-example"

Clickhouse

[[outputs.sql]]
  ## Database driver
  ## Valid options include mssql, mysql, pgx, sqlite, snowflake, clickhouse
  driver = "clickhouse"

  ## Data source name
  ## For ClickHouse, the DSN follows the clickhouse-go v1.5.4 format.
  ## Example DSN: "tcp://localhost:9000?debug=true"
  data_source_name = "tcp://localhost:9000?debug=true"

  ## Timestamp column name
  timestamp_column = "timestamp"

  ## Table creation template
  ## Available template variables:
  ##  {TABLE}        - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS}      - column definitions (list of quoted identifiers and types)
  table_template = "CREATE TABLE {TABLE} ({COLUMNS})"

  ## Table existence check template
  ## Available template variables:
  ##  {TABLE} - table name as a quoted identifier
  table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  ## Initialization SQL (optional)
  init_sql = ""

  ## Maximum amount of time a connection may be idle. "0s" means connections are never closed due to idle time.
  connection_max_idle_time = "0s"

  ## Maximum amount of time a connection may be reused. "0s" means connections are never closed due to age.
  connection_max_lifetime = "0s"

  ## Maximum number of connections in the idle connection pool. 0 means unlimited.
  connection_max_idle = 2

  ## Maximum number of open connections to the database. 0 means unlimited.
  connection_max_open = 0

  ## Metric type to SQL type conversion for ClickHouse.
  ## The conversion maps Telegraf metric types to ClickHouse native data types.
  [outputs.sql.convert]
    conversion_style = "literal"
    integer          = "Int64"
    text             = "String"
    timestamp        = "DateTime"
    defaultvalue     = "String"
    unsigned         = "UInt64"
    bool             = "UInt8"
    real             = "Float64"

Input and output integration examples

Amazon CloudWatch

1. Cross-Account Monitoring: Utilize this plugin to monitor resources across multiple AWS accounts by enabling the include_linked_accounts option. This scenario allows companies managing multiple AWS accounts to aggregate metrics into a central monitoring dashboard, providing a unified view of all metrics while ensuring secure data access and compliance through proper role management.

2. Dynamic Alerting System: Integrate this plugin with alerting tools to create an automated system that triggers alerts based on defined thresholds for CloudWatch metrics. For instance, if latency metrics exceed specified limits, alerts can be sent to relevant teams, enabling proactive responses to performance issues and reducing downtime.

3. Cost Management Dashboard: Use the metrics gathered from the plugin to build a cost management dashboard that visualizes AWS service usage metrics over time. By correlating these metrics with billing data, organizations can identify high-cost services and take informed actions to optimize their resource usage and spending.

4. Performance Benchmarking for Applications: Leverage the metrics collected from applications running on AWS to perform performance benchmarks. For example, by tracking latency and request count metrics for an ELB, developers can assess the impact of application changes on its performance, making data-driven decisions for optimization.

Clickhouse

1. Basic ClickHouse Integration: Configure the plugin by setting the driver to ‘clickhouse’ and providing the appropriate DSN format as required by clickhouse-go v1.5.4. This ensures that Telegraf can connect and write metrics to your ClickHouse database.

2. Customized Table Schemas: Leverage the table creation and existence check templates to tailor the database schema. This allows you to predefine column types and even disable automatic table creation if you prefer manual schema management.

3. Advanced Type Conversion: Utilize the ClickHouse-specific conversion settings to map Telegraf metric types directly to ClickHouse data types (e.g., mapping integers to Int64 and timestamps to DateTime). This ensures data is stored with the correct precision and format.

4. Initialization and Connection Tuning: Use the init_sql setting to run custom SQL commands upon connection, and adjust connection pool settings (like connection_max_idle_time and connection_max_open) to optimize performance for high-throughput environments.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.