Google Cloud PubSub and Redis Integration

Input and output integration overview

This plugin ingests metrics from Google Cloud PubSub, allowing for real-time data processing and integration into monitoring setups.

The Redis plugin enables users to send metrics collected by Telegraf directly to Redis. This integration is ideal for applications that require robust time series data storage and analysis.

Integration details

Google Cloud PubSub

The Google Cloud PubSub input plugin is designed to ingest metrics from Google Cloud PubSub, a messaging service that facilitates real-time communication between different systems. It allows users to create and process metrics by pulling messages from a specified subscription in a Google Cloud Project. One of the critical features of this plugin is its ability to operate as a service input, actively listening for incoming messages rather than merely polling for metrics at set intervals. Through various configuration options, users can customize the behavior of message ingestion, such as handling credentials, managing message sizes, and tuning the acknowledgment settings to ensure that messages are only acknowledged after successful processing. By leveraging the strengths of Google PubSub, this plugin integrates seamlessly with cloud-native architectures, enabling users to build robust and scalable applications that can react to events in real-time.

Redis

The Redis Telegraf plugin is designed for writing metrics to RedisTimeSeries, a specialized Redis database module for time series data. This plugin facilitates the integration of Telegraf with RedisTimeSeries, allowing for the efficient storage and retrieval of timestamped data. With RedisTimeSeries, users can take advantage of enhanced capabilities for managing time series data, including aggregated views and range queries. The plugin offers various configuration options to enable the flexibility needed to connect securely to your Redis database, including support for Authentication, Timeouts, data type conversions, and TLS configurations. The underlying technology leverages Redis’ efficiency and scalability, making it an excellent choice for high-volume metric environments, where real-time processing is essential.

Configuration

Google Cloud PubSub

[[inputs.cloud_pubsub]]
  project = "my-project"
  subscription = "my-subscription"
  data_format = "influx"
  # credentials_file = "path/to/my/creds.json"
  # retry_delay_seconds = 5
  # max_message_len = 1000000
  # max_undelivered_messages = 1000
  # max_extension = 0
  # max_outstanding_messages = 0
  # max_outstanding_bytes = 0
  # max_receiver_go_routines = 0
  # base64_data = false
  # content_encoding = "identity"
  # max_decompression_size = "500MB"

Redis

[[outputs.redistimeseries]]
  ## The address of the RedisTimeSeries server.
  address = "127.0.0.1:6379"

  ## Redis ACL credentials
  # username = ""
  # password = ""
  # database = 0

  ## Timeout for operations such as ping or sending metrics
  # timeout = "10s"

  ## Enable attempt to convert string fields to numeric values
  ## If "false" or in case the string value cannot be converted the string
  ## field will be dropped.
  # convert_string_fields = true

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  # insecure_skip_verify = false

Input and output integration examples

Google Cloud PubSub

1. Real-Time Analytics for IoT Devices: Utilize the Google Cloud PubSub plugin to aggregate metrics from IoT devices scattered across various locations. By streaming data from devices to Google PubSub and using this plugin to ingest metrics, organizations can create a centralized dashboard for real-time monitoring and alerting. This setup allows for immediate insights into device performance, facilitating proactive maintenance and operational efficiency.

2. Dynamic Log Processing and Monitoring: Ingest logs from numerous sources via Google Cloud PubSub into a Telegraf pipeline, utilizing the plugin to parse and analyze log messages. This can help teams quickly identify anomalies or patterns in logs and streamline the process of troubleshooting issues across distributed systems. By consolidating log data, organizations can enhance their observability and response capabilities.

3. Event-Driven Workflow Integrations: Use the Google Cloud PubSub plugin to connect various cloud functions or services. Each time a new message is pushed to a subscription, actions can be triggered in other parts of the cloud architecture, such as starting data processing jobs, notifications, or even updates to reports. This event-driven approach allows for a more reactive system architecture that can adapt to changing business needs.

Redis

1. Monitoring IoT Sensor Data: Utilize the Redis Telegraf plugin to collect and store data from IoT sensors in real-time. By connecting the plugin to a RedisTimeSeries database, users can analyze trends in temperature, humidity, or other environmental factors. The ability to query historical sensor data efficiently will aid in predictive maintenance and help in resource management.

2. Financial Market Data Aggregation: Employ this plugin to track and store time-sensitive financial data from various sources. By sending metrics to Redis, financial institutions can aggregate and analyze market trends or price changes over time, providing them with actionable insights derived from reliable time series analytics.

3. Application Performance Monitoring (APM): Implement the Redis plugin for gathering application performance metrics such as response times and CPU usage. Users can visualize their application’s performance over time with RedisTimeSeries, allowing them to identify bottlenecks and optimize resource allocation swiftly.

4. Energy Consumption Tracking: Leverage this plugin to monitor energy usage in buildings over time. By integrating with smart meters and sending data to RedisTimeSeries, municipalities or enterprises can analyze energy consumption patterns, helping to implement energy-saving measures and sustainability practices.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.