Kubernetes and Prometheus Integration

Input and output integration overview

This plugin captures metrics for Kubernetes pods and containers by communicating with the Kubelet API.

The Prometheus Output Plugin enables Telegraf to expose metrics at an HTTP endpoint for scraping by a Prometheus server. This integration allows users to collect and aggregate metrics from various sources in a format that Prometheus can process efficiently.

Integration details

Kubernetes

The Kubernetes input plugin interfaces with the Kubelet API to gather metrics for running pods and containers on a single host, ideally as part of a daemonset in a Kubernetes installation. By operating on each node within the cluster, it collects metrics from the locally running kubelet, ensuring that the data reflects the real-time state of the environment. Being a rapidly evolving project, Kubernetes sees frequent updates, and this plugin adheres to the major cloud providers’ supported versions, maintaining compatibility across multiple releases within a limited time span. Significant consideration is given to the potential high series cardinality, which can burden the database; thus, users are advised to implement filtering techniques and retention policies to manage this load effectively. Configuration options provide flexible customization of the plugin’s behavior to integrate seamlessly into different setups, enhancing its utility in monitoring Kubernetes environments.

Prometheus

This plugin for facilitates the integration with Prometheus, a well-known open-source monitoring and alerting toolkit designed for reliability and efficiency in large-scale environments. By working as a Prometheus client, it allows users to expose a defined set of metrics via an HTTP server that Prometheus can scrape at specified intervals. This plugin plays a crucial role in monitoring diverse systems by allowing them to publish performance metrics in a standardized format, enabling extensive visibility into system health and behavior. Key features include support for configuring various endpoints, enabling TLS for secure communication, and options for HTTP basic authentication. The plugin also integrates seamlessly with global Telegraf configuration settings, supporting extensive customization to fit specific monitoring needs. This promotes interoperability in environments where different systems must communicate performance data effectively. Leveraging Prometheus’s metric format, it allows for flexible metric management through advanced configurations such as metric expiration and collectors control, offering a sophisticated solution for monitoring and alerting workflows.

Configuration

Kubernetes

[[inputs.kubernetes]]
  ## URL for the kubelet, if empty read metrics from all nodes in the cluster
  url = "http://127.0.0.1:10255"

  ## Use bearer token for authorization. ('bearer_token' takes priority)
  ## If both of these are empty, we'll use the default serviceaccount:
  ## at: /var/run/secrets/kubernetes.io/serviceaccount/token
  ##
  ## To re-read the token at each interval, please use a file with the
  ## bearer_token option. If given a string, Telegraf will always use that
  ## token.
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  ## OR
  # bearer_token_string = "abc_123"

  ## Kubernetes Node Metric Name
  ## The default Kubernetes node metric name (i.e. kubernetes_node) is the same
  ## for the kubernetes and kube_inventory plugins. To avoid conflicts, set this
  ## option to a different value.
  # node_metric_name = "kubernetes_node"

  ## Pod labels to be added as tags.  An empty array for both include and
  ## exclude will include all labels.
  # label_include = []
  # label_exclude = ["*"]

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## Optional TLS Config
  # tls_ca = /path/to/cafile
  # tls_cert = /path/to/certfile
  # tls_key = /path/to/keyfile
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Prometheus

[[outputs.prometheus_client]]
  ## Address to listen on.
  ##   ex:
  ##     listen = ":9273"
  ##     listen = "vsock://:9273"
  listen = ":9273"

  ## Maximum duration before timing out read of the request
  # read_timeout = "10s"
  ## Maximum duration before timing out write of the response
  # write_timeout = "10s"

  ## Metric version controls the mapping from Prometheus metrics into Telegraf metrics.
  ## See "Metric Format Configuration" in plugins/inputs/prometheus/README.md for details.
  ## Valid options: 1, 2
  # metric_version = 1

  ## Use HTTP Basic Authentication.
  # basic_username = "Foo"
  # basic_password = "Bar"

  ## If set, the IP Ranges which are allowed to access metrics.
  ##   ex: ip_range = ["192.168.0.0/24", "192.168.1.0/30"]
  # ip_range = []

  ## Path to publish the metrics on.
  # path = "/metrics"

  ## Expiration interval for each metric. 0 == no expiration
  # expiration_interval = "60s"

  ## Collectors to enable, valid entries are "gocollector" and "process".
  ## If unset, both are enabled.
  # collectors_exclude = ["gocollector", "process"]

  ## Send string metrics as Prometheus labels.
  ## Unless set to false all string metrics will be sent as labels.
  # string_as_label = true

  ## If set, enable TLS with the given certificate.
  # tls_cert = "/etc/ssl/telegraf.crt"
  # tls_key = "/etc/ssl/telegraf.key"

  ## Set one or more allowed client CA certificate file names to
  ## enable mutually authenticated TLS connections
  # tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]

  ## Export metric collection time.
  # export_timestamp = false

  ## Specify the metric type explicitly.
  ## This overrides the metric-type of the Telegraf metric. Globbing is allowed.
  # [outputs.prometheus_client.metric_types]
  #   counter = []
  #   gauge = []

Input and output integration examples

Kubernetes

1. Dynamic Resource Allocation Monitoring: By utilizing the Kubernetes plugin, teams can set up alerts for resource usage patterns across various pods and containers. This proactive monitoring approach enables automatic scaling of resources in response to specific thresholds—helping to optimize performance while minimizing costs during peak usage.

2. Multi-tenancy Resource Isolation Analysis: Organizations using Kubernetes can leverage this plugin to track resource consumption per namespace. In a multi-tenant scenario, understanding the resource allocations and usages across different teams becomes critical for ensuring fair access and performance guarantees, leading to better resource management strategies.

3. Real-time Health Dashboards: Integrate the data captured by the Kubernetes plugin into visualization tools like Grafana to create real-time dashboards. These dashboards provide insights into the overall health and performance of the Kubernetes environment, allowing teams to quickly identify and rectify issues across clusters, pods, and containers.

4. Automated Incident Response Workflows: By combining the Kubernetes plugin with alert management systems, teams can automate incident response procedures based on real-time metrics. If a pod’s resource usage exceeds predefined limits, an automated workflow can trigger remediation actions, such as restarting the pod or reallocating resources—all of which can help improve system resilience.

Prometheus

1. Monitoring Multi-cloud Deployments: Utilize the Prometheus plugin to collect metrics from applications running across multiple cloud providers. This scenario allows teams to centralize monitoring through a single Prometheus instance that scrapes metrics from different environments, providing a unified view of performance metrics across hybrid infrastructures. It streamlines reporting and alerting, enhancing operational efficiency without needing complex integrations.

2. Enhancing Microservices Visibility: Implement the plugin to expose metrics from various microservices within a Kubernetes cluster. Using Prometheus, teams can visualize service metrics in real time, identify bottlenecks, and maintain system health checks. This setup supports adaptive scaling and resource utilization optimization based on insights generated from the collected metrics. It enhances the ability to troubleshoot service interactions, significantly improving the resilience of the microservice architecture.

3. Real-time Anomaly Detection in E-commerce: By leveraging this plugin alongside Prometheus, an e-commerce platform can monitor key performance indicators such as response times and error rates. Integrating anomaly detection algorithms with scraped metrics allows the identification of unexpected patterns indicating potential issues, such as sudden traffic spikes or backend service failure. This proactive monitoring empowers business continuity and operational efficiency, minimizing potential downtimes while ensuring service reliability.

4. Performance Metrics Reporting for APIs: Utilize the Prometheus Output Plugin to gather and report API performance metrics, which can then be visualized in Grafana dashboards. This use case enables detailed analysis of API response times, throughput, and error rates, promoting continuous improvement of API services. By closely monitoring these metrics, teams can quickly react to degradation, ensuring optimal API performance and maintaining a high level of service availability.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.