Tail and Thanos Integration

Input and output integration overview

The Tail Telegraf plugin collects metrics by tailing specified log files, capturing new log entries in real-time for further analysis.

This plugin sends metrics from Telegraf to Thanos using the Prometheus remote write protocol over HTTP, allowing efficient and scalable ingestion into Thanos Receive components.

Integration details

Tail

The tail plugin is designed to continuously monitor and parse log files, making it ideal for real-time log analysis and monitoring. It mimics the functionality of the Unix tail command, allowing users to specify a file or pattern and begin reading new lines as they are added. Key features include the ability to follow log-rotated files, start reading from the end of a file, and support various parsing formats for the log messages. Users can customize the plugin through various configuration options, such as specifying file encoding, the method for watching file updates, and filter settings for processing log data. This plugin is particularly valuable in environments where log data is critical for monitoring application performance and diagnosing issues.

Thanos

Telegraf’s HTTP plugin can send metrics directly to Thanos via its Remote Write-compatible Receive component. By setting the data format to prometheusremotewrite, Telegraf can serialize metrics into the same protobuf-based format used by native Prometheus clients. This setup enables high-throughput, low-latency metric ingestion into Thanos, facilitating centralized observability at scale. It is particularly useful in hybrid environments where Telegraf is collecting metrics from systems outside Prometheus’ native reach, such as SNMP devices, Windows hosts, or custom apps, and streams them directly to Thanos for long-term storage and global querying.

Configuration

Tail

[[inputs.tail]]
  ## File names or a pattern to tail.
  ## These accept standard unix glob matching rules, but with the addition of
  ## ** as a "super asterisk". ie:
  ##   "/var/log/**.log"  -> recursively find all .log files in /var/log
  ##   "/var/log/*/*.log" -> find all .log files with a parent dir in /var/log
  ##   "/var/log/apache.log" -> just tail the apache log file
  ##   "/var/log/log[!1-2]*  -> tail files without 1-2
  ##   "/var/log/log[^1-2]*  -> identical behavior as above
  ## See https://github.com/gobwas/glob for more examples
  ##
  files = ["/var/mymetrics.out"]

  ## Read file from beginning.
  # from_beginning = false

  ## Whether file is a named pipe
  # pipe = false

  ## Method used to watch for file updates.  Can be either "inotify" or "poll".
  ## inotify is supported on linux, *bsd, and macOS, while Windows requires
  ## using poll. Poll checks for changes every 250ms.
  # watch_method = "inotify"

  ## Maximum lines of the file to process that have not yet be written by the
  ## output.  For best throughput set based on the number of metrics on each
  ## line and the size of the output's metric_batch_size.
  # max_undelivered_lines = 1000

  ## Character encoding to use when interpreting the file contents.  Invalid
  ## characters are replaced using the unicode replacement character.  When set
  ## to the empty string the data is not decoded to text.
  ##   ex: character_encoding = "utf-8"
  ##       character_encoding = "utf-16le"
  ##       character_encoding = "utf-16be"
  ##       character_encoding = ""
  # character_encoding = ""

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

  ## Set the tag that will contain the path of the tailed file. If you don't want this tag, set it to an empty string.
  # path_tag = "path"

  ## Filters to apply to files before generating metrics
  ## "ansi_color" removes ANSI colors
  # filters = []

  ## multiline parser/codec
  ## https://www.elastic.co/guide/en/logstash/2.4/plugins-filters-multiline.html
  #[inputs.tail.multiline]
    ## The pattern should be a regexp which matches what you believe to be an indicator that the field is part of an event consisting of multiple lines of log data.
    #pattern = "^\s"

    ## The field's value must be previous or next and indicates the relation to the
    ## multi-line event.
    #match_which_line = "previous"

    ## The invert_match can be true or false (defaults to false).
    ## If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)
    #invert_match = false

    ## The handling method for quoted text (defaults to 'ignore').
    ## The following methods are available:
    ##   ignore  -- do not consider quotation (default)
    ##   single-quotes -- consider text quoted by single quotes (')
    ##   double-quotes -- consider text quoted by double quotes (")
    ##   backticks     -- consider text quoted by backticks (`)
    ## When handling quotes, escaped quotes (e.g. \") are handled correctly.
    #quotation = "ignore"

    ## The preserve_newline option can be true or false (defaults to false).
    ## If true, the newline character is preserved for multiline elements,
    ## this is useful to preserve message-structure e.g. for logging outputs.
    #preserve_newline = false

    #After the specified timeout, this plugin sends the multiline event even if no new pattern is found to start a new event. The default is 5s.
    #timeout = 5s

Thanos

[[outputs.http]]
  ## Thanos Receive endpoint for remote write
  url = "http://thanos-receive.example.com/api/v1/receive"

  ## HTTP method
  method = "POST"

  ## Data format set to Prometheus remote write
  data_format = "prometheusremotewrite"

  ## Optional headers (authorization, etc.)
  # [outputs.http.headers]
  #   Authorization = "Bearer YOUR_TOKEN"

  ## Optional TLS configuration
  # tls_ca = "/path/to/ca.pem"
  # tls_cert = "/path/to/cert.pem"
  # tls_key = "/path/to/key.pem"
  # insecure_skip_verify = false

  ## Request timeout
  timeout = "10s"

Input and output integration examples

Tail

1. Real-Time Server Health Monitoring: Implement the Tail plugin to parse web server access logs in real-time, providing immediate visibility into user activity, error rates, and performance metrics. By visualizing this log data, operations teams can quickly identify and respond to spikes in traffic or errors, enhancing system reliability and user experience.

2. Centralized Log Management: Utilize the Tail plugin to aggregate logs from multiple sources across a distributed system. By configuring each service to send its logs to a centralized location via the Tail plugin, teams can simplify log analysis and ensure that all relevant data is accessible from a single interface, streamlining troubleshooting processes.

3. Security Incident Detection: Use this plugin to monitor authentication logs for unauthorized access attempts or suspicious activity. By setting up alerts on certain log messages, teams can leverage this plugin to enhance security postures and respond promptly to potential security threats, reducing the risk of breaches and increasing overall system integrity.

4. Dynamic Application Performance Insights: Integrate with analytics tools to create real-time dashboards that display application performance metrics based on log data. This setup not only helps developers diagnose bottlenecks and inefficiencies but also allows for proactive performance tuning and resource allocation, optimizing application behavior under varying loads.

Thanos

1. Agentless Cloud Monitoring: Deploy Telegraf agents across cloud VMs to collect system and application metrics, then stream them directly into Thanos using Remote Write. This provides centralized observability without requiring Prometheus nodes at each location.

2. Scalable Windows Host Monitoring: Use Telegraf on Windows machines to collect OS-level metrics and send them via Remote Write to Thanos Receive. This enables observability across heterogeneous environments with native Prometheus support only on Linux.

3. Cross-Region Metrics Federation: Telegraf agents in multiple geographic regions can push data to region-local Thanos Receivers using this plugin. From there, Thanos can deduplicate and query metrics globally, reducing latency and network egress costs.

4. Integrating Third-Party Data into Thanos: Collect metrics from custom telemetry sources such as REST APIs or proprietary logs using Telegraf inputs and forward them to Thanos via Remote Write. This brings non-native data into a Prometheus-compatible, long-term analytics pipeline.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.