SNMP and Clickhouse Integration

Input and output integration overview

The SNMP plugin allows you to collect a variety of metrics from SNMP (Simple Network Management Protocol) agents. It provides flexibility in how data is retrieved, whether collecting single metrics or entire tables.

Telegraf’s SQL output plugin sends collected metrics to an SQL database using a straightforward table schema and dynamic column generation. When configured for ClickHouse, it adjusts DSN formatting and type conversion settings to ensure seamless data integration.

Integration details

SNMP

This plugin uses polling to gather metrics from SNMP agents, supporting retrieval of individual OIDs and complete SNMP tables. It can be configured to handle multiple SNMP versions, authentication, and other features.

Clickhouse

The SQL output plugin is designed to store Telegraf metrics in an SQL database using a simple, hard-coded schema. Each metric type gets its own table, and columns are generated for every tag and field, with an optional timestamp column. For ClickHouse, the plugin leverages a specialized DSN format as defined by clickhouse-go v1.5.4 and customizes metric type conversion to align with ClickHouse data types. This ensures that integers, texts, timestamps, booleans, and real numbers are mapped to ClickHouse’s native types such as Int64, String, DateTime, UInt8, and Float64 respectively.

Configuration

SNMP

[[inputs.snmp]]
  agents = ["udp://127.0.0.1:161"]

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysUpTime.0"
    name = "sysUptime"
    conversion = "float(2)"

  [[inputs.snmp.field]]
    oid = "RFC1213-MIB::sysName.0"
    name = "sysName"
    is_tag = true

  [[inputs.snmp.table]]
    oid = "IF-MIB::ifTable"
    name = "interface"
    inherit_tags = ["sysName"]

    [[inputs.snmp.table.field]]
      oid = "IF-MIB::ifDescr"
      name = "ifDescr"
      is_tag = true

Clickhouse

[[outputs.sql]]
  ## Database driver
  ## Valid options include mssql, mysql, pgx, sqlite, snowflake, clickhouse
  driver = "clickhouse"

  ## Data source name
  ## For ClickHouse, the DSN follows the clickhouse-go v1.5.4 format.
  ## Example DSN: "tcp://localhost:9000?debug=true"
  data_source_name = "tcp://localhost:9000?debug=true"

  ## Timestamp column name
  timestamp_column = "timestamp"

  ## Table creation template
  ## Available template variables:
  ##  {TABLE}        - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS}      - column definitions (list of quoted identifiers and types)
  table_template = "CREATE TABLE {TABLE} ({COLUMNS})"

  ## Table existence check template
  ## Available template variables:
  ##  {TABLE} - table name as a quoted identifier
  table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  ## Initialization SQL (optional)
  init_sql = ""

  ## Maximum amount of time a connection may be idle. "0s" means connections are never closed due to idle time.
  connection_max_idle_time = "0s"

  ## Maximum amount of time a connection may be reused. "0s" means connections are never closed due to age.
  connection_max_lifetime = "0s"

  ## Maximum number of connections in the idle connection pool. 0 means unlimited.
  connection_max_idle = 2

  ## Maximum number of open connections to the database. 0 means unlimited.
  connection_max_open = 0

  ## Metric type to SQL type conversion for ClickHouse.
  ## The conversion maps Telegraf metric types to ClickHouse native data types.
  [outputs.sql.convert]
    conversion_style = "literal"
    integer          = "Int64"
    text             = "String"
    timestamp        = "DateTime"
    defaultvalue     = "String"
    unsigned         = "UInt64"
    bool             = "UInt8"
    real             = "Float64"

Input and output integration examples

SNMP

1. Basic SNMP Configuration: Collect metrics from a local SNMP agent using typical SNMP community string settings. This setup is ideal for local monitoring of device performance.
2. Advanced SNMPv3 Setup: Securely collect metrics using SNMPv3 with authentication and encryption to enhance security. This configuration is recommended for production environments.
3. Collect Interface Metrics: Configure the plugin to collect interface metrics from the device’s SNMP table. Utilize fields to capture specific data points for traffic analysis.
4. Join Two SNMP Tables: By using translation fields, join data from two SNMP tables for a comprehensive view of correlated performance metrics.

Clickhouse

1. Basic ClickHouse Integration: Configure the plugin by setting the driver to ‘clickhouse’ and providing the appropriate DSN format as required by clickhouse-go v1.5.4. This ensures that Telegraf can connect and write metrics to your ClickHouse database.

2. Customized Table Schemas: Leverage the table creation and existence check templates to tailor the database schema. This allows you to predefine column types and even disable automatic table creation if you prefer manual schema management.

3. Advanced Type Conversion: Utilize the ClickHouse-specific conversion settings to map Telegraf metric types directly to ClickHouse data types (e.g., mapping integers to Int64 and timestamps to DateTime). This ensures data is stored with the correct precision and format.

4. Initialization and Connection Tuning: Use the init_sql setting to run custom SQL commands upon connection, and adjust connection pool settings (like connection_max_idle_time and connection_max_open) to optimize performance for high-throughput environments.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.