Google Cloud Stackdriver and MongoDB Integration

Input and output integration overview

This plugin enables the collection of monitoring data from Google Cloud services through the Stackdriver Monitoring API. It is designed to help users monitor their cloud infrastructure’s performance and health by gathering relevant metrics.

The MongoDB Telegraf Plugin enables users to send metrics to a MongoDB database, automatically managing time series collections.

Integration details

Google Cloud Stackdriver

The Stackdriver Telegraf plugin allows users to query timeseries data from Google Cloud Monitoring using the Cloud Monitoring API v3. With this plugin, users can easily integrate Google Cloud monitoring metrics into their monitoring stacks. This API provides a wealth of insights about resources and applications running in Google Cloud, including performance, uptime, and operational metrics. The plugin supports various configuration options to filter and refine the data retrieved, enabling users to customize their monitoring setup according to their specific needs. This integration facilitates a smoother experience in maintaining the health and performance of cloud resources and assists teams in making data-driven decisions based on historical and current performance statistics.

MongoDB

This plugin sends metrics to MongoDB and seamlessly integrates with its time series functionality, allowing for automatic creation of collections as time series when they don’t already exist. It requires MongoDB version 5.0 or higher to utilize the time series collections feature, which is vital for efficiently storing and querying time-based data. This plugin enhances the monitoring capabilities by ensuring that all relevant metrics are stored and organized correctly within MongoDB, providing users the ability to leverage MongoDB’s powerful querying and aggregation features for time series analysis.

Configuration

Google Cloud Stackdriver

[[inputs.stackdriver]]
  ## GCP Project
  project = "erudite-bloom-151019"

  ## Include timeseries that start with the given metric type.
  metric_type_prefix_include = [
    "compute.googleapis.com/",
  ]

  ## Exclude timeseries that start with the given metric type.
  # metric_type_prefix_exclude = []

  ## Most metrics are updated no more than once per minute; it is recommended
  ## to override the agent level interval with a value of 1m or greater.
  interval = "1m"

  ## Maximum number of API calls to make per second.  The quota for accounts
  ## varies, it can be viewed on the API dashboard:
  ##   https://cloud.google.com/monitoring/quotas#quotas_and_limits
  # rate_limit = 14

  ## The delay and window options control the number of points selected on
  ## each gather.  When set, metrics are gathered between:
  ##   start: now() - delay - window
  ##   end:   now() - delay
  #
  ## Collection delay; if set too low metrics may not yet be available.
  # delay = "5m"
  #
  ## If unset, the window will start at 1m and be updated dynamically to span
  ## the time between calls (approximately the length of the plugin interval).
  # window = "1m"

  ## TTL for cached list of metric types.  This is the maximum amount of time
  ## it may take to discover new metrics.
  # cache_ttl = "1h"

  ## If true, raw bucket counts are collected for distribution value types.
  ## For a more lightweight collection, you may wish to disable and use
  ## distribution_aggregation_aligners instead.
  # gather_raw_distribution_buckets = true

  ## Aggregate functions to be used for metrics whose value type is
  ## distribution.  These aggregate values are recorded in in addition to raw
  ## bucket counts; if they are enabled.
  ##
  ## For a list of aligner strings see:
  ##   https://cloud.google.com/monitoring/api/ref_v3/rpc/google.monitoring.v3#aligner
  # distribution_aggregation_aligners = [
  #  "ALIGN_PERCENTILE_99",
  #  "ALIGN_PERCENTILE_95",
  #  "ALIGN_PERCENTILE_50",
  # ]

  ## Filters can be added to reduce the number of time series matched.  All
  ## functions are supported: starts_with, ends_with, has_substring, and
  ## one_of.  Only the '=' operator is supported.
  ##
  ## The logical operators when combining filters are defined statically using
  ## the following values:
  ##   filter ::=  {AND  AND  AND }
  ##   resource_labels ::=  {OR }
  ##   metric_labels ::=  {OR }
  ##   user_labels ::=  {OR }
  ##   system_labels ::=  {OR }
  ##
  ## For more details, see https://cloud.google.com/monitoring/api/v3/filters
  #
  ## Resource labels refine the time series selection with the following expression:
  ##   resource.labels. = 
  # [[inputs.stackdriver.filter.resource_labels]]
  #   key = "instance_name"
  #   value = 'starts_with("localhost")'
  #
  ## Metric labels refine the time series selection with the following expression:
  ##   metric.labels. = 
  #  [[inputs.stackdriver.filter.metric_labels]]
  #    key = "device_name"
  #    value = 'one_of("sda", "sdb")'
  #
  ## User labels refine the time series selection with the following expression:
  ##   metadata.user_labels."" = 
  #  [[inputs.stackdriver.filter.user_labels]]
  #    key = "environment"
  #    value = 'one_of("prod", "staging")'
  #
  ## System labels refine the time series selection with the following expression:
  ##   metadata.system_labels."" = 
  #  [[inputs.stackdriver.filter.system_labels]]
  #    key = "machine_type"
  #    value = 'starts_with("e2-")'
</code></pre>

MongoDB

[[outputs.mongodb]]
              # connection string examples for mongodb
              dsn = "mongodb://localhost:27017"
              # dsn = "mongodb://mongod1:27017,mongod2:27017,mongod3:27017/admin&replicaSet=myReplSet&w=1"

              # overrides serverSelectionTimeoutMS in dsn if set
              # timeout = "30s"

              # default authentication, optional
              # authentication = "NONE"

              # for SCRAM-SHA-256 authentication
              # authentication = "SCRAM"
              # username = "root"
              # password = "***"

              # for x509 certificate authentication
              # authentication = "X509"
              # tls_ca = "ca.pem"
              # tls_key = "client.pem"
              # # tls_key_pwd = "changeme" # required for encrypted tls_key
              # insecure_skip_verify = false

              # database to store measurements and time series collections
              # database = "telegraf"

              # granularity can be seconds, minutes, or hours.
              # configuring this value will be based on your input collection frequency.
              # see https://docs.mongodb.com/manual/core/timeseries-collections/#create-a-time-series-collection
              # granularity = "seconds"

              # optionally set a TTL to automatically expire documents from the measurement collections.
              # ttl = "360h"

Input and output integration examples

Google Cloud Stackdriver

1. Integrating Cloud Metrics into Custom Dashboards: With this plugin, teams can funnel metrics from Google Cloud into personalized dashboards, allowing for real-time monitoring of application performance and resource utilization. By customizing the visual representation of cloud metrics, operations teams can easily identify trends and anomalies, enabling proactive management before issues escalate.

2. Automated Alerts and Analysis: Users can set up automated alerting mechanisms leveraging the plugin’s metrics to track resource thresholds. This capability allows teams to act swiftly in response to performance degradation or outages by providing immediate notifications, thus reducing the mean time to recovery and ensuring continued operational efficiency.

3. Cross-Platform Resource Comparison: The plugin can be used to draw metrics from various Google Cloud services and compare them with on-premise resources. This cross-platform visibility helps organizations make informed decisions about resource allocation and scaling strategies, as well as optimize cloud spending versus on-premise infrastructure.

4. Historical Data Analysis for Capacity Planning: By collecting historical metrics over time, the plugin empowers teams to conduct thorough capacity planning. Understanding past performance trends facilitates accurate forecasting for resource needs, leading to better budgeting and investment strategies.

MongoDB

1. Dynamic Logging to MongoDB for IoT Devices: Utilize this plugin to collect and store metrics from a fleet of IoT devices in real-time. By sending device logs directly to MongoDB, you can create a centralized database that allows for easy access and querying of health metrics and performance data, enabling proactive maintenance and troubleshooting based on historical trends.

2. Time Series Analysis of Web Traffic: Use the MongoDB Telegraf Plugin to gather and analyze web traffic metrics over time. This application can help you understand peak usage times, user interactions, and behavior patterns, which can guide marketing strategies and infrastructure scaling decisions for improved user experience.

3. Automated Monitoring and Alerting System: Integrate the MongoDB plugin into an automated monitoring system that tracks application performance metrics. With time series collections, you can set up alerts based on specific thresholds, allowing your team to respond to potential issues before they affect users. This proactive management can enhance service reliability and overall performance.

4. Data Retention and TTL Management in Metrics Storage: Leverage the TTL feature for documents within MongoDB collections to auto-expire outdated metrics. This is particularly useful for environments where only recent performance data is relevant, preventing your MongoDB database from becoming cluttered with old metrics and ensuring efficient data management.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.