Google Cloud Storage and Cortex Integration

Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.

info

This is not the recommended configuration for real-time query at scale. For query and compression optimization, high-speed ingest, and high availability, you may want to consider Google Cloud Storage and InfluxDB.

5B+

Telegraf downloads

#1

Time series database
Source: DB Engines

1B+

Downloads of InfluxDB

2,800+

Contributors

Table of Contents

Powerful Performance, Limitless Scale

Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.

See Ways to Get Started

Input and output integration overview

The Google Cloud Storage plugin collects metrics from specified Google Cloud Storage buckets, providing insight into storage usage and performance.

This plugin enables Telegraf to send metrics to Cortex using the Prometheus remote write protocol, allowing seamless ingestion into Cortex’s scalable, multi-tenant time series storage.

Integration details

Google Cloud Storage

The Google Cloud Storage Telegraf plugin enables the collection of metrics from specified Google Cloud Storage buckets. As organizations increasingly rely on cloud storage solutions for their data management, the ability to monitor the performance and utilization of these resources becomes essential. This plugin is particularly useful for tracking how storage is used, understanding data patterns, and ensuring operational efficiency. By integrating with Google Cloud Storage APIs, it allows users to gather insights from their cloud environments, feeding metrics directly into monitoring systems for further analysis. The plugin supports various configuration options, enabling users to customize the data collection process based on their specific needs.

Cortex

With Telegraf’s HTTP output plugin and the prometheusremotewrite data format you can send metrics directly to Cortex, a horizontally scalable, long-term storage backend for Prometheus. Cortex supports multi-tenancy and accepts remote write requests using the Prometheus protobuf format. By using Telegraf as the collection agent and Remote Write as the transport mechanism, organizations can extend observability into sources not natively supported by Prometheus—such as Windows hosts, SNMP-enabled devices, or custom application metrics—while leveraging Cortex’s high-availability and long-retention capabilities.

Configuration

Google Cloud Storage

[[inputs.google_cloud_storage]]
  bucket = "my-bucket"
  # key_prefix = "my-bucket"
  offset_key = "offset_key"
  objects_per_iteration = 10
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"

Cortex

[[outputs.http]]
  ## Cortex Remote Write endpoint
  url = "http://cortex.example.com/api/v1/push"

  ## Use POST to send data
  method = "POST"

  ## Send metrics using Prometheus remote write format
  data_format = "prometheusremotewrite"

  ## Optional HTTP headers for authentication
  # [outputs.http.headers]
  #   X-Scope-OrgID = "your-tenant-id"
  #   Authorization = "Bearer YOUR_API_TOKEN"

  ## Optional TLS configuration
  # tls_ca = "/path/to/ca.pem"
  # tls_cert = "/path/to/cert.pem"
  # tls_key = "/path/to/key.pem"
  # insecure_skip_verify = false

  ## Request timeout
  timeout = "10s"

Input and output integration examples

Google Cloud Storage

  1. Automated Backup Monitoring: Utilize the Google Cloud Storage plugin to regularly monitor the status of backup files stored in a Cloud Storage bucket. By configuring the plugin to track file metrics, organizations can automate alerts if backup sizes deviate from expected patterns, ensuring that data protection processes are functioning properly and any anomalies are promptly addressed.

  2. Cost Optimization Insights: Integrate this plugin into a cost management tool to analyze the usage patterns of Cloud Storage. By collecting metrics on file sizes and access frequencies, teams can optimize their storage solutions and make informed decisions about data retention policies, potentially reducing unnecessary storage costs and improving resource allocation.

  3. Compliance and Auditing: Use the plugin to generate metrics that aid in compliance verification for data stored in Google Cloud Storage. By providing detailed insights into data access and storage usage, organizations can ensure adherence to regulatory requirements, helping in audits and aligning with best practices for data governance.

  4. Performance Benchmarking: Deploy the plugin to benchmark the performance of data retrieval and storage operations in Google Cloud Storage. By analyzing metrics over time, teams can identify performance bottlenecks or inefficiencies, allowing them to optimize their applications and infrastructure that depend on cloud storage services.

Cortex

  1. Unified Multi-Tenant Monitoring: Use Telegraf to collect metrics from different teams or environments and push them to Cortex with separate X-Scope-OrgID headers. This enables isolated data ingestion and querying per tenant, ideal for managed services and platform teams.

  2. Extending Prometheus Coverage to Edge Devices: Deploy Telegraf on edge or IoT devices to collect system metrics and send them to a centralized Cortex cluster. This approach ensures consistent observability even for environments without local Prometheus scrapers.

  3. Global Service Observability with Federated Tenants: Aggregate metrics from global infrastructure by configuring Telegraf agents to push data into regional Cortex clusters, each tagged with tenant identifiers. Cortex handles deduplication and centralized access across regions.

  4. Custom App Telemetry Pipeline: Collect app-specific telemetry via Telegraf’s exec or http input plugins and forward it to Cortex. This allows DevOps teams to monitor app-specific KPIs in a scalable, query-efficient format while keeping metrics logically grouped by tenant or service.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.

Powerful Performance, Limitless Scale

Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.

See Ways to Get Started

Related Integrations

HTTP and InfluxDB Integration

The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.

View Integration

Kafka and InfluxDB Integration

This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.

View Integration

Kinesis and InfluxDB Integration

The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.

View Integration