Amazon CloudWatch and Datadog Integration

Input and output integration overview

This plugin will pull Metric Statistics from Amazon CloudWatch, streamlining the process of monitoring and analyzing AWS resources.

The Datadog Telegraf Plugin enables the submission of metrics to the Datadog Metrics API, facilitating efficient monitoring and data analysis through a reliable metric ingestion process.

Integration details

Amazon CloudWatch

The Amazon CloudWatch Plugin allows users to pull detailed metric statistics from Amazon’s CloudWatch service. As a monitoring solution, CloudWatch enables users to track various metrics related to AWS resources and applications, facilitating improved operational and performance insights. The plugin uses a structured authentication method that prioritizes security and flexibility through a combination of STS (Security Token Service), shared credentials, environment variables, and EC2 instance profiles, ensuring robust access control to AWS resources. Key features include the ability to define specific metric namespaces, aggregated periods for metrics, and optional inclusion of linked accounts for cross-account monitoring. A significant aspect of this plugin is its capacity to handle both sparse and dense metric formats, allowing for varied output structures depending on user preference. Thus, it supports versatile use cases in cloud monitoring and analytics by providing comprehensive, timely data directly from CloudWatch.

Datadog

This plugin writes to the Datadog Metrics API, enabling users to send metrics for monitoring and performance analysis. By utilizing the Datadog API key, users can configure the plugin to establish a connection with Datadog’s v1 API. The plugin supports various configuration options including connection timeouts, HTTP proxy settings, and data compression methods, ensuring adaptability to different deployment environments. The ability to transform count metrics into rates enhances the integration of Telegraf with Datadog agents, particularly beneficial for applications that rely on real-time performance metrics.

Configuration

Amazon CloudWatch

[[inputs.cloudwatch]]
  region = "us-east-1"
  # access_key = ""
  # secret_key = ""
  # token = ""
  # role_arn = ""
  # web_identity_token_file = ""
  # role_session_name = ""
  # profile = ""
  # shared_credential_file = ""
  # include_linked_accounts = false
  # endpoint_url = ""
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"
  period = "5m"
  delay = "5m"
  interval = "5m"
  #recently_active = "PT3H"
  # cache_ttl = "1h"
  namespaces = ["AWS/ELB"]
  # metric_format = "sparse"
  # ratelimit = 25
  # timeout = "5s"
  # batch_size = 500
  # statistic_include = ["average", "sum", "minimum", "maximum", sample_count]
  # statistic_exclude = []
  # [[inputs.cloudwatch.metrics]]
  #  names = ["Latency", "RequestCount"]
  #  [[inputs.cloudwatch.metrics.dimensions]]
  #    name = "LoadBalancerName"
  #    value = "p-example"

Datadog

[[outputs.datadog]]
  ## Datadog API key
  apikey = "my-secret-key"

  ## Connection timeout.
  # timeout = "5s"

  ## Write URL override; useful for debugging.
  ## This plugin only supports the v1 API currently due to the authentication
  ## method used.
  # url = "https://app.datadoghq.com/api/v1/series"

  ## Set http_proxy
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"

  ## Override the default (none) compression used to send data.
  ## Supports: "zlib", "none"
  # compression = "none"

  ## When non-zero, converts count metrics submitted by inputs.statsd
  ## into rate, while dividing the metric value by this number.
  ## Note that in order for metrics to be submitted simultaenously alongside
  ## a Datadog agent, rate_interval has to match the interval used by the
  ## agent - which defaults to 10s
  # rate_interval = 0s

Input and output integration examples

Amazon CloudWatch

1. Cross-Account Monitoring: Utilize this plugin to monitor resources across multiple AWS accounts by enabling the include_linked_accounts option. This scenario allows companies managing multiple AWS accounts to aggregate metrics into a central monitoring dashboard, providing a unified view of all metrics while ensuring secure data access and compliance through proper role management.

2. Dynamic Alerting System: Integrate this plugin with alerting tools to create an automated system that triggers alerts based on defined thresholds for CloudWatch metrics. For instance, if latency metrics exceed specified limits, alerts can be sent to relevant teams, enabling proactive responses to performance issues and reducing downtime.

3. Cost Management Dashboard: Use the metrics gathered from the plugin to build a cost management dashboard that visualizes AWS service usage metrics over time. By correlating these metrics with billing data, organizations can identify high-cost services and take informed actions to optimize their resource usage and spending.

4. Performance Benchmarking for Applications: Leverage the metrics collected from applications running on AWS to perform performance benchmarks. For example, by tracking latency and request count metrics for an ELB, developers can assess the impact of application changes on its performance, making data-driven decisions for optimization.

Datadog

1. Real-Time Infrastructure Monitoring: Use the Datadog plugin to monitor server metrics in real-time by sending CPU usage and memory statistics directly to Datadog. This integration allows IT teams to visualize and analyze system performance metrics in a centralized dashboard, enabling proactive response to any emerging issues, such as resource bottlenecks or server overloads.

2. Application Performance Tracking: Leverage this plugin to submit application-specific metrics, such as request counts and error rates, to Datadog. By integrating with application monitoring tools, teams can correlate infrastructure metrics with application performance, providing insights that enable them to optimize code performance and improve user experience.

3. Anomaly Detection in Metrics: Configure the Datadog plugin to send metrics that can trigger alerts and notifications based on unusual patterns detected by Datadog’s machine learning features. This proactive monitoring helps teams swiftly react to potential outages or performance degradation before customers are impacted.

4. Integrating with Cloud Services: By utilizing the Datadog plugin to send metrics from cloud resources, IT teams can gain visibility into cloud application performance. Monitoring metrics like latency and error rates helps with ensuring service-level agreements (SLAs) are met and also assists in optimizing resource allocation across cloud environments.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.