Kubernetes and Thanos Integration

Input and output integration overview

This plugin captures metrics for Kubernetes pods and containers by communicating with the Kubelet API.

This plugin sends metrics from Telegraf to Thanos using the Prometheus remote write protocol over HTTP, allowing efficient and scalable ingestion into Thanos Receive components.

Integration details

Kubernetes

The Kubernetes input plugin interfaces with the Kubelet API to gather metrics for running pods and containers on a single host, ideally as part of a daemonset in a Kubernetes installation. By operating on each node within the cluster, it collects metrics from the locally running kubelet, ensuring that the data reflects the real-time state of the environment. Being a rapidly evolving project, Kubernetes sees frequent updates, and this plugin adheres to the major cloud providers’ supported versions, maintaining compatibility across multiple releases within a limited time span. Significant consideration is given to the potential high series cardinality, which can burden the database; thus, users are advised to implement filtering techniques and retention policies to manage this load effectively. Configuration options provide flexible customization of the plugin’s behavior to integrate seamlessly into different setups, enhancing its utility in monitoring Kubernetes environments.

Thanos

Telegraf’s HTTP plugin can send metrics directly to Thanos via its Remote Write-compatible Receive component. By setting the data format to prometheusremotewrite, Telegraf can serialize metrics into the same protobuf-based format used by native Prometheus clients. This setup enables high-throughput, low-latency metric ingestion into Thanos, facilitating centralized observability at scale. It is particularly useful in hybrid environments where Telegraf is collecting metrics from systems outside Prometheus’ native reach, such as SNMP devices, Windows hosts, or custom apps, and streams them directly to Thanos for long-term storage and global querying.

Configuration

Kubernetes

[[inputs.kubernetes]]
  ## URL for the kubelet, if empty read metrics from all nodes in the cluster
  url = "http://127.0.0.1:10255"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /var/run/secrets/kubernetes.io/serviceaccount/token
  ##
  ## To re-read the token at each interval, please use a file with the
  ## bearer_token option. If given a string, Telegraf will always use that
  ## token.
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"

  ## Kubernetes Node Metric Name
  ## The default Kubernetes node metric name (i.e. kubernetes_node) is the same
  ## for the kubernetes and kube_inventory plugins. To avoid conflicts, set this
  ## option to a different value.
  # node_metric_name = "kubernetes_node"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Thanos

[[outputs.http]]
  ## Thanos Receive endpoint for remote write
  url = "http://thanos-receive.example.com/api/v1/receive"

  ## HTTP method
  method = "POST"

  ## Data format set to Prometheus remote write
  data_format = "prometheusremotewrite"

  ## Optional headers (authorization, etc.)
  # [outputs.http.headers]
  #   Authorization = "Bearer YOUR_TOKEN"

  ## Optional TLS configuration
  # tls_ca = "/path/to/ca.pem"
  # tls_cert = "/path/to/cert.pem"
  # tls_key = "/path/to/key.pem"
  # insecure_skip_verify = false

  ## Request timeout
  timeout = "10s"

Input and output integration examples

Kubernetes

1. Dynamic Resource Allocation Monitoring: By utilizing the Kubernetes plugin, teams can set up alerts for resource usage patterns across various pods and containers. This proactive monitoring approach enables automatic scaling of resources in response to specific thresholds—helping to optimize performance while minimizing costs during peak usage.

2. Multi-tenancy Resource Isolation Analysis: Organizations using Kubernetes can leverage this plugin to track resource consumption per namespace. In a multi-tenant scenario, understanding the resource allocations and usages across different teams becomes critical for ensuring fair access and performance guarantees, leading to better resource management strategies.

3. Real-time Health Dashboards: Integrate the data captured by the Kubernetes plugin into visualization tools like Grafana to create real-time dashboards. These dashboards provide insights into the overall health and performance of the Kubernetes environment, allowing teams to quickly identify and rectify issues across clusters, pods, and containers.

4. Automated Incident Response Workflows: By combining the Kubernetes plugin with alert management systems, teams can automate incident response procedures based on real-time metrics. If a pod’s resource usage exceeds predefined limits, an automated workflow can trigger remediation actions, such as restarting the pod or reallocating resources—all of which can help improve system resilience.

Thanos

1. Agentless Cloud Monitoring: Deploy Telegraf agents across cloud VMs to collect system and application metrics, then stream them directly into Thanos using Remote Write. This provides centralized observability without requiring Prometheus nodes at each location.

2. Scalable Windows Host Monitoring: Use Telegraf on Windows machines to collect OS-level metrics and send them via Remote Write to Thanos Receive. This enables observability across heterogeneous environments with native Prometheus support only on Linux.

3. Cross-Region Metrics Federation: Telegraf agents in multiple geographic regions can push data to region-local Thanos Receivers using this plugin. From there, Thanos can deduplicate and query metrics globally, reducing latency and network egress costs.

4. Integrating Third-Party Data into Thanos: Collect metrics from custom telemetry sources such as REST APIs or proprietary logs using Telegraf inputs and forward them to Thanos via Remote Write. This brings non-native data into a Prometheus-compatible, long-term analytics pipeline.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.