Supervisor and Loki Integration

Input and output integration overview

This plugin gathers information about processes running under Supervisor using the XML-RPC API.

The Loki plugin allows users to send logs to Loki for aggregation and querying, leveraging Loki’s efficient storage capabilities.

Integration details

Supervisor

The Supervisor plugin for Telegraf is designed to collect metrics about processes managed by the Supervisor process control system using its XML-RPC API. The plugin is able to track various metrics, including process states and uptime, and provides options for configuring which metrics to collect through include or exclude lists. This integration is particularly useful for monitoring applications running under Supervisor, providing insights into their operational status and performance metrics. A minimum tested Supervisor version is 3.3.2, and it is recommended to secure the HTTP server with basic authentication for better security.

Loki

This Loki plugin integrates with Grafana Loki, a powerful log aggregation system. By sending logs in a format compatible with Loki, this plugin allows for efficient storage and querying of logs. Each log entry is structured in a key-value format where keys represent the field names and values represent the corresponding log information. The sorting of logs by timestamp ensures that the log streams maintain chronological order when queried through Loki. This plugin’s support for secrets makes it easier to manage authentication parameters securely, while options for HTTP headers, gzip encoding, and TLS configuration enhance the adaptability and security of log transmission, fitting various deployment needs.

Configuration

Supervisor

[[inputs.supervisor]]
  ## Url of supervisor's XML-RPC endpoint if basic auth enabled in supervisor http server,
  ## than you have to add credentials to url (ex. http://login:pass@localhost:9001/RPC2)
  # url="http://localhost:9001/RPC2"
  ## With settings below you can manage gathering additional information about processes
  ## If both of them empty, then all additional information will be collected.
  ## Currently supported supported additional metrics are: pid, rc
  # metrics_include = []
  # metrics_exclude = ["pid", "rc"]

Loki

[[outputs.loki]]
  ## The domain of Loki
  domain = "https://loki.domain.tld"

  ## Endpoint to write api
  # endpoint = "/loki/api/v1/push"

  ## Connection timeout, defaults to "5s" if not set.
  # timeout = "5s"

  ## Basic auth credential
  # username = "loki"
  # password = "pass"

  ## Additional HTTP headers
  # http_headers = {"X-Scope-OrgID" = "1"}

  ## If the request must be gzip encoded
  # gzip_request = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"

  ## Sanitize Tag Names
  ## If true, all tag names will have invalid characters replaced with
  ## underscores that do not match the regex: ^[a-zA-Z_:][a-zA-Z0-9_:]*.
  # sanitize_label_names = false

  ## Metric Name Label
  ## Label to use for the metric name to when sending metrics. If set to an
  ## empty string, this will not add the label. This is NOT suggested as there
  ## is no way to differentiate between multiple metrics.
  # metric_name_label = "__name"

Input and output integration examples

Supervisor

1. Centralized Monitoring Dashboard: Implement this plugin to feed Supervisor metrics directly into a centralized monitoring dashboard, allowing teams to visualize the health and performance of their applications in real-time. This integration enables quick identification of issues, helps track service performance over time, and aids in capacity planning based on observed trends.

2. Alerting for Process Failures: Utilize the metrics gathered by the Supervisor plugin to create an alerting mechanism that notifies engineers when critical processes go down or enter a fatal state. By setting thresholds in your monitoring system, teams can respond proactively to potential problems, minimizing downtime and ensuring system reliability.

3. Historical Analysis of Process States: Store the metrics collected over time to analyze process state changes and patterns. By examining historical data, teams can identify recurring issues, track the impact of deployment changes, and optimize resource allocation based on process trends, leading to improved overall system performance.

4. Integration with Incident Management Systems: Configure the Supervisor plugin to automatically send alerts to incident management systems like PagerDuty or OpsGenie when a process reaches a critical state. This integration streamlines the incident response process, ensuring that the right team members are notified promptly and can take action without delay.

Loki

1. Centralized Logging for Microservices: Utilize the Loki plugin to gather logs from multiple microservices running in a Kubernetes cluster. By directing logs to a centralized Loki instance, developers can monitor, search, and analyze logs from all services in one place, facilitating easier troubleshooting and performance monitoring. This setup streamlines operations and supports rapid response to issues across distributed applications.

2. Real-Time Log Anomaly Detection: Combine Loki with monitoring tools to analyze log outputs in real-time for unusual patterns that could indicate system errors or security threats. Implementing anomaly detection on log streams enables teams to proactively identify and respond to incidents, thereby improving system reliability and enhancing security postures.

3. Enhanced Log Processing with Gzip Compression: Configure the Loki plugin to utilize gzip compression for log transmission. This approach can reduce bandwidth usage and improve transmission speeds, especially beneficial in environments where network bandwidth may be a constraint. It’s particularly useful for high-volume logging applications where every byte counts and performance is critical.

4. Multi-Tenancy Support with Custom Headers: Leverage the ability to add custom HTTP headers to segregate logs from different tenants in a multi-tenant application environment. By using the Loki plugin to send different headers for each tenant, operators can ensure proper log management and compliance with data isolation requirements, making it a versatile solution for SaaS applications.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.