Google Cloud Storage and Thanos Integration

Input and output integration overview

The Google Cloud Storage plugin collects metrics from specified Google Cloud Storage buckets, providing insight into storage usage and performance.

This plugin sends metrics from Telegraf to Thanos using the Prometheus remote write protocol over HTTP, allowing efficient and scalable ingestion into Thanos Receive components.

Integration details

Google Cloud Storage

The Google Cloud Storage Telegraf plugin enables the collection of metrics from specified Google Cloud Storage buckets. As organizations increasingly rely on cloud storage solutions for their data management, the ability to monitor the performance and utilization of these resources becomes essential. This plugin is particularly useful for tracking how storage is used, understanding data patterns, and ensuring operational efficiency. By integrating with Google Cloud Storage APIs, it allows users to gather insights from their cloud environments, feeding metrics directly into monitoring systems for further analysis. The plugin supports various configuration options, enabling users to customize the data collection process based on their specific needs.

Thanos

Telegraf’s HTTP plugin can send metrics directly to Thanos via its Remote Write-compatible Receive component. By setting the data format to prometheusremotewrite, Telegraf can serialize metrics into the same protobuf-based format used by native Prometheus clients. This setup enables high-throughput, low-latency metric ingestion into Thanos, facilitating centralized observability at scale. It is particularly useful in hybrid environments where Telegraf is collecting metrics from systems outside Prometheus’ native reach, such as SNMP devices, Windows hosts, or custom apps, and streams them directly to Thanos for long-term storage and global querying.

Configuration

Google Cloud Storage

[[inputs.google_cloud_storage]]
  bucket = "my-bucket"
  # key_prefix = "my-bucket"
  offset_key = "offset_key"
  objects_per_iteration = 10
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"

Thanos

[[outputs.http]]
  ## Thanos Receive endpoint for remote write
  url = "http://thanos-receive.example.com/api/v1/receive"

  ## HTTP method
  method = "POST"

  ## Data format set to Prometheus remote write
  data_format = "prometheusremotewrite"

  ## Optional headers (authorization, etc.)
  # [outputs.http.headers]
  #   Authorization = "Bearer YOUR_TOKEN"

  ## Optional TLS configuration
  # tls_ca = "/path/to/ca.pem"
  # tls_cert = "/path/to/cert.pem"
  # tls_key = "/path/to/key.pem"
  # insecure_skip_verify = false

  ## Request timeout
  timeout = "10s"

Input and output integration examples

Google Cloud Storage

1. Automated Backup Monitoring: Utilize the Google Cloud Storage plugin to regularly monitor the status of backup files stored in a Cloud Storage bucket. By configuring the plugin to track file metrics, organizations can automate alerts if backup sizes deviate from expected patterns, ensuring that data protection processes are functioning properly and any anomalies are promptly addressed.

2. Cost Optimization Insights: Integrate this plugin into a cost management tool to analyze the usage patterns of Cloud Storage. By collecting metrics on file sizes and access frequencies, teams can optimize their storage solutions and make informed decisions about data retention policies, potentially reducing unnecessary storage costs and improving resource allocation.

3. Compliance and Auditing: Use the plugin to generate metrics that aid in compliance verification for data stored in Google Cloud Storage. By providing detailed insights into data access and storage usage, organizations can ensure adherence to regulatory requirements, helping in audits and aligning with best practices for data governance.

4. Performance Benchmarking: Deploy the plugin to benchmark the performance of data retrieval and storage operations in Google Cloud Storage. By analyzing metrics over time, teams can identify performance bottlenecks or inefficiencies, allowing them to optimize their applications and infrastructure that depend on cloud storage services.

Thanos

1. Agentless Cloud Monitoring: Deploy Telegraf agents across cloud VMs to collect system and application metrics, then stream them directly into Thanos using Remote Write. This provides centralized observability without requiring Prometheus nodes at each location.

2. Scalable Windows Host Monitoring: Use Telegraf on Windows machines to collect OS-level metrics and send them via Remote Write to Thanos Receive. This enables observability across heterogeneous environments with native Prometheus support only on Linux.

3. Cross-Region Metrics Federation: Telegraf agents in multiple geographic regions can push data to region-local Thanos Receivers using this plugin. From there, Thanos can deduplicate and query metrics globally, reducing latency and network egress costs.

4. Integrating Third-Party Data into Thanos: Collect metrics from custom telemetry sources such as REST APIs or proprietary logs using Telegraf inputs and forward them to Thanos via Remote Write. This brings non-native data into a Prometheus-compatible, long-term analytics pipeline.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.