Amazon CloudWatch and MongoDB Integration
Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.
5B+
Telegraf downloads
#1
Time series database
Source: DB Engines
1B+
Downloads of InfluxDB
2,800+
Contributors
Table of Contents
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Input and output integration overview
This plugin will pull Metric Statistics from Amazon CloudWatch, streamlining the process of monitoring and analyzing AWS resources.
The MongoDB Telegraf Plugin enables users to send metrics to a MongoDB database, automatically managing time series collections.
Integration details
Amazon CloudWatch
The Amazon CloudWatch Plugin allows users to pull detailed metric statistics from Amazon’s CloudWatch service. As a monitoring solution, CloudWatch enables users to track various metrics related to AWS resources and applications, facilitating improved operational and performance insights. The plugin uses a structured authentication method that prioritizes security and flexibility through a combination of STS (Security Token Service), shared credentials, environment variables, and EC2 instance profiles, ensuring robust access control to AWS resources. Key features include the ability to define specific metric namespaces, aggregated periods for metrics, and optional inclusion of linked accounts for cross-account monitoring. A significant aspect of this plugin is its capacity to handle both sparse and dense metric formats, allowing for varied output structures depending on user preference. Thus, it supports versatile use cases in cloud monitoring and analytics by providing comprehensive, timely data directly from CloudWatch.
MongoDB
This plugin sends metrics to MongoDB and seamlessly integrates with its time series functionality, allowing for automatic creation of collections as time series when they don’t already exist. It requires MongoDB version 5.0 or higher to utilize the time series collections feature, which is vital for efficiently storing and querying time-based data. This plugin enhances the monitoring capabilities by ensuring that all relevant metrics are stored and organized correctly within MongoDB, providing users the ability to leverage MongoDB’s powerful querying and aggregation features for time series analysis.
Configuration
Amazon CloudWatch
[[inputs.cloudwatch]]
region = "us-east-1"
# access_key = ""
# secret_key = ""
# token = ""
# role_arn = ""
# web_identity_token_file = ""
# role_session_name = ""
# profile = ""
# shared_credential_file = ""
# include_linked_accounts = false
# endpoint_url = ""
# use_system_proxy = false
# http_proxy_url = "http://localhost:8888"
period = "5m"
delay = "5m"
interval = "5m"
#recently_active = "PT3H"
# cache_ttl = "1h"
namespaces = ["AWS/ELB"]
# metric_format = "sparse"
# ratelimit = 25
# timeout = "5s"
# batch_size = 500
# statistic_include = ["average", "sum", "minimum", "maximum", sample_count]
# statistic_exclude = []
# [[inputs.cloudwatch.metrics]]
# names = ["Latency", "RequestCount"]
# [[inputs.cloudwatch.metrics.dimensions]]
# name = "LoadBalancerName"
# value = "p-example"
MongoDB
[[outputs.mongodb]]
# connection string examples for mongodb
dsn = "mongodb://localhost:27017"
# dsn = "mongodb://mongod1:27017,mongod2:27017,mongod3:27017/admin&replicaSet=myReplSet&w=1"
# overrides serverSelectionTimeoutMS in dsn if set
# timeout = "30s"
# default authentication, optional
# authentication = "NONE"
# for SCRAM-SHA-256 authentication
# authentication = "SCRAM"
# username = "root"
# password = "***"
# for x509 certificate authentication
# authentication = "X509"
# tls_ca = "ca.pem"
# tls_key = "client.pem"
# # tls_key_pwd = "changeme" # required for encrypted tls_key
# insecure_skip_verify = false
# database to store measurements and time series collections
# database = "telegraf"
# granularity can be seconds, minutes, or hours.
# configuring this value will be based on your input collection frequency.
# see https://docs.mongodb.com/manual/core/timeseries-collections/#create-a-time-series-collection
# granularity = "seconds"
# optionally set a TTL to automatically expire documents from the measurement collections.
# ttl = "360h"
Input and output integration examples
Amazon CloudWatch
-
Cross-Account Monitoring: Utilize this plugin to monitor resources across multiple AWS accounts by enabling the
include_linked_accounts
option. This scenario allows companies managing multiple AWS accounts to aggregate metrics into a central monitoring dashboard, providing a unified view of all metrics while ensuring secure data access and compliance through proper role management. -
Dynamic Alerting System: Integrate this plugin with alerting tools to create an automated system that triggers alerts based on defined thresholds for CloudWatch metrics. For instance, if latency metrics exceed specified limits, alerts can be sent to relevant teams, enabling proactive responses to performance issues and reducing downtime.
-
Cost Management Dashboard: Use the metrics gathered from the plugin to build a cost management dashboard that visualizes AWS service usage metrics over time. By correlating these metrics with billing data, organizations can identify high-cost services and take informed actions to optimize their resource usage and spending.
-
Performance Benchmarking for Applications: Leverage the metrics collected from applications running on AWS to perform performance benchmarks. For example, by tracking latency and request count metrics for an ELB, developers can assess the impact of application changes on its performance, making data-driven decisions for optimization.
MongoDB
-
Dynamic Logging to MongoDB for IoT Devices: Utilize this plugin to collect and store metrics from a fleet of IoT devices in real-time. By sending device logs directly to MongoDB, you can create a centralized database that allows for easy access and querying of health metrics and performance data, enabling proactive maintenance and troubleshooting based on historical trends.
-
Time Series Analysis of Web Traffic: Use the MongoDB Telegraf Plugin to gather and analyze web traffic metrics over time. This application can help you understand peak usage times, user interactions, and behavior patterns, which can guide marketing strategies and infrastructure scaling decisions for improved user experience.
-
Automated Monitoring and Alerting System: Integrate the MongoDB plugin into an automated monitoring system that tracks application performance metrics. With time series collections, you can set up alerts based on specific thresholds, allowing your team to respond to potential issues before they affect users. This proactive management can enhance service reliability and overall performance.
-
Data Retention and TTL Management in Metrics Storage: Leverage the TTL feature for documents within MongoDB collections to auto-expire outdated metrics. This is particularly useful for environments where only recent performance data is relevant, preventing your MongoDB database from becoming cluttered with old metrics and ensuring efficient data management.
Feedback
Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Related Integrations
Related Integrations
HTTP and InfluxDB Integration
The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.
View IntegrationKafka and InfluxDB Integration
This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.
View IntegrationKinesis and InfluxDB Integration
The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.
View Integration