AWS Data Firehose and ServiceNow Integration
Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.
5B+
Telegraf downloads
#1
Time series database
Source: DB Engines
1B+
Downloads of InfluxDB
2,800+
Contributors
Table of Contents
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Input and output integration overview
This plugin listens for metrics sent via HTTP from AWS Data Firehose in supported data formats, providing real-time data ingestion capabilities.
This output plugin streams metrics from Telegraf directly to a ServiceNow MID Server via HTTP, leveraging the nowmetric
serializer for efficient integration with ServiceNow’s Operational Intelligence and Event Management.
Integration details
AWS Data Firehose
The AWS Data Firehose Telegraf plugin is designed to receive metrics from AWS Data Firehose via HTTP. This plugin listens for incoming data in various formats and processes it according to the request-response schema outlined in the official AWS documentation. Unlike standard input plugins that operate on a fixed interval, this service plugin initializes a listener that remains active, waiting for incoming metrics. This allows for real-time data ingestion from AWS Data Firehose, making it suitable for scenarios where immediate data processing is required. Key features include the ability to specify service addresses, paths, and support for TLS connections for secure data transmission. Additionally, the plugin accommodates optional authentication keys and custom tags, enhancing its flexibility in various use cases involving data streaming and processing.
ServiceNow
Telegraf can be used to send metric data directly to a ServiceNow MID Server REST endpoint. Metrics are formatted either using ServiceNow’s Operational Intelligence (OI) format or JSONv2 format, enabling seamless integration with ServiceNow’s Event Management and Operational Intelligence platforms. The serializer batches metrics efficiently, reducing network overhead by minimizing the number of HTTP POST requests. This integration allows users to quickly leverage metrics in ServiceNow for enhanced observability, proactive incident management, and performance monitoring, with ServiceNow’s operational intelligence capabilities.
Configuration
AWS Data Firehose
[[inputs.firehose]]
## Address and port to host HTTP listener on
service_address = ":8080"
## Paths to listen to.
# paths = ["/telegraf"]
## maximum duration before timing out read of the request
# read_timeout = "5s"
## maximum duration before timing out write of the response
# write_timeout = "5s"
## Set one or more allowed client CA certificate file names to
## enable mutually authenticated TLS connections
# tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
## Add service certificate and key
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Minimal TLS version accepted by the server
# tls_min_version = "TLS12"
## Optional access key to accept for authentication.
## AWS Data Firehose uses "x-amz-firehose-access-key" header to set the access key.
## If no access_key is provided (default), authentication is completely disabled and
## this plugin will accept all request ignoring the provided access-key in the request!
# access_key = "foobar"
## Optional setting to add parameters as tags
## If the http header "x-amz-firehose-common-attributes" is not present on the
## request, no corresponding tag will be added. The header value should be a
## json and should follow the schema as describe in the official documentation:
## https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#requestformat
# parameter_tags = ["env"]
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
# data_format = "influx"
ServiceNow
[[outputs.http]]
## ServiceNow MID Server metrics endpoint
url = "http://mid-server.example.com:9082/api/mid/sa/metrics"
## HTTP request method
method = "POST"
## Basic Authentication credentials
username = "evt.integration"
password = "P@$$w0rd!"
## Data serialization format for ServiceNow
data_format = "nowmetric"
## Metric format type: "oi" (default) or "jsonv2"
nowmetric_format = "oi"
## HTTP Headers
[outputs.http.headers]
Content-Type = "application/json"
Accept = "application/json"
## Optional timeout
# timeout = "5s"
## TLS configuration options
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
# insecure_skip_verify = false
Input and output integration examples
AWS Data Firehose
-
Real-Time Data Analytics: Using the AWS Data Firehose plugin, organizations can stream data in real-time from various sources, such as application logs or IoT devices, directly into analytics platforms. This allows data teams to analyze incoming data as it is generated, enabling rapid insights and operational adjustments based on fresh metrics.
-
Profile Access Patterns for Optimization: By collecting data about how clients interact with applications through AWS Data Firehose, businesses can gain valuable insights into user behavior. This can drive content personalization strategies or optimize server architecture for better performance based on traffic patterns.
-
Automated Alerting Mechanism: Integrating AWS Data Firehose with alerting systems via this plugin allows teams to set up automated alerts based on specific metrics collected. For example, if a particular threshold is reached in the input data, alerts can trigger operations teams to investigate potential issues before they escalate.
ServiceNow
-
Proactive Incident Management: Utilize the Telegraf and ServiceNow integration to stream infrastructure and application metrics in real-time to ServiceNow Event Management. Automatically trigger incidents or remediation workflows based on thresholds, significantly reducing incident detection and response times.
-
End-to-End Application Monitoring: Deploy Telegraf agents across multiple layers of an application stack, sending performance metrics directly into ServiceNow. Leveraging ServiceNow’s Operational Intelligence, teams can correlate metrics across components, quickly identifying performance bottlenecks.
-
Dynamic CI Performance Tracking: Integrate Telegraf metrics with ServiceNow’s CMDB by using this plugin to push performance data, allowing automatic updates of Configuration Item (CI) health states based on live metrics. This ensures an accurate and current state of infrastructure health in ServiceNow.
-
Cloud Resource Optimization: Collect metrics from hybrid and multi-cloud infrastructures using Telegraf, streaming directly to ServiceNow. Leverage these metrics for real-time analytics, predictive capacity planning, and resource optimization, enabling proactive management and reduced operational costs.
Feedback
Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Related Integrations
Related Integrations
HTTP and InfluxDB Integration
The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.
View IntegrationKafka and InfluxDB Integration
This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.
View IntegrationKinesis and InfluxDB Integration
The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.
View Integration