AMQP and MySQL Integration
Powerful performance with an easy integration, powered by Telegraf, the open source data connector built by InfluxData.
5B+
Telegraf downloads
#1
Time series database
Source: DB Engines
1B+
Downloads of InfluxDB
2,800+
Contributors
Table of Contents
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Input and output integration overview
The AMQP Consumer Input Plugin allows you to ingest data from an AMQP 0-9-1 compliant message broker, such as RabbitMQ, enabling seamless data collection for monitoring and analytics purposes.
The Telegraf SQL plugin allows you to store metrics from Telegraf directly into a MySQL database, making it easier to analyze and visualize the collected metrics.
Integration details
AMQP
This plugin provides a consumer for use with AMQP 0-9-1, a prominent implementation of which is RabbitMQ. AMQP, or Advanced Message Queuing Protocol, was originally developed to enable reliable, interoperable messaging between diverse systems in a network. The plugin reads metrics from a topic exchange using a configured queue and binding key, delivering a flexible and efficient means of collecting data from AMQP-compliant messaging systems. This enables users to leverage existing RabbitMQ implementations to monitor their applications effectively by capturing detailed metrics for analysis and alerting.
MySQL
Telegraf’s SQL output plugin is designed to seamlessly write metric data to a SQL database by dynamically creating tables and columns based on the incoming metrics. When configured for MySQL, the plugin leverages the go-sql-driver/mysql, which requires enabling the ANSI_QUOTES SQL mode to ensure proper handling of quoted identifiers. This dynamic schema creation approach ensures that each metric is stored in its own table with a structure derived from its fields and tags, providing a detailed, timestamped record of system performance. The flexibility of the plugin allows it to handle high-throughput environments, making it ideal for scenarios that demand robust, granular metric logging and historical data analysis.
Configuration
AMQP
[[inputs.amqp_consumer]]
## Brokers to consume from. If multiple brokers are specified a random broker
## will be selected anytime a connection is established. This can be
## helpful for load balancing when not using a dedicated load balancer.
brokers = ["amqp://localhost:5672/influxdb"]
## Authentication credentials for the PLAIN auth_method.
# username = ""
# password = ""
## Name of the exchange to declare. If unset, no exchange will be declared.
exchange = "telegraf"
## Exchange type; common types are "direct", "fanout", "topic", "header", "x-consistent-hash".
# exchange_type = "topic"
## If true, exchange will be passively declared.
# exchange_passive = false
## Exchange durability can be either "transient" or "durable".
# exchange_durability = "durable"
## Additional exchange arguments.
# exchange_arguments = { }
# exchange_arguments = {"hash_property" = "timestamp"}
## AMQP queue name.
queue = "telegraf"
## AMQP queue durability can be "transient" or "durable".
queue_durability = "durable"
## If true, queue will be passively declared.
# queue_passive = false
## Additional arguments when consuming from Queue
# queue_consume_arguments = { }
# queue_consume_arguments = {"x-stream-offset" = "first"}
## A binding between the exchange and queue using this binding key is
## created. If unset, no binding is created.
binding_key = "#"
## Maximum number of messages server should give to the worker.
# prefetch_count = 50
## Max undelivered messages
## This plugin uses tracking metrics, which ensure messages are read to
## outputs before acknowledging them to the original broker to ensure data
## is not lost. This option sets the maximum messages to read from the
## broker that have not been written by an output.
##
## This value needs to be picked with awareness of the agent's
## metric_batch_size value as well. Setting max undelivered messages too high
## can result in a constant stream of data batches to the output. While
## setting it too low may never flush the broker's messages.
# max_undelivered_messages = 1000
## Timeout for establishing the connection to a broker
# timeout = "30s"
## Auth method. PLAIN and EXTERNAL are supported
## Using EXTERNAL requires enabling the rabbitmq_auth_mechanism_ssl plugin as
## described here: https://www.rabbitmq.com/plugins.html
# auth_method = "PLAIN"
## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Content encoding for message payloads, can be set to
## "gzip", "identity" or "auto"
## - Use "gzip" to decode gzip
## - Use "identity" to apply no encoding
## - Use "auto" determine the encoding using the ContentEncoding header
# content_encoding = "identity"
## Maximum size of decoded message.
## Acceptable units are B, KiB, KB, MiB, MB...
## Without quotes and units, interpreted as size in bytes.
# max_decompression_size = "500MB"
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "influx"
MySQL
[[outputs.sql]]
## Database driver
## Valid options: mssql (Microsoft SQL Server), mysql (MySQL), pgx (Postgres),
## sqlite (SQLite3), snowflake (snowflake.com) clickhouse (ClickHouse)
driver = "mysql"
## Data source name
## The format of the data source name is different for each database driver.
## See the plugin readme for details.
data_source_name = "username:password@tcp(host:port)/dbname"
## Timestamp column name
timestamp_column = "timestamp"
## Table creation template
## Available template variables:
## {TABLE} - table name as a quoted identifier
## {TABLELITERAL} - table name as a quoted string literal
## {COLUMNS} - column definitions (list of quoted identifiers and types)
table_template = "CREATE TABLE {TABLE}({COLUMNS})"
## Table existence check template
## Available template variables:
## {TABLE} - tablename as a quoted identifier
table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"
## Initialization SQL
init_sql = "SET sql_mode='ANSI_QUOTES';"
## Maximum amount of time a connection may be idle. "0s" means connections are
## never closed due to idle time.
connection_max_idle_time = "0s"
## Maximum amount of time a connection may be reused. "0s" means connections
## are never closed due to age.
connection_max_lifetime = "0s"
## Maximum number of connections in the idle connection pool. 0 means unlimited.
connection_max_idle = 2
## Maximum number of open connections to the database. 0 means unlimited.
connection_max_open = 0
## NOTE: Due to the way TOML is parsed, tables must be at the END of the
## plugin definition, otherwise additional config options are read as part of the
## table
## Metric type to SQL type conversion
## The values on the left are the data types Telegraf has and the values on
## the right are the data types Telegraf will use when sending to a database.
##
## The database values used must be data types the destination database
## understands. It is up to the user to ensure that the selected data type is
## available in the database they are using. Refer to your database
## documentation for what data types are available and supported.
#[outputs.sql.convert]
# integer = "INT"
# real = "DOUBLE"
# text = "TEXT"
# timestamp = "TIMESTAMP"
# defaultvalue = "TEXT"
# unsigned = "UNSIGNED"
# bool = "BOOL"
# ## This setting controls the behavior of the unsigned value. By default the
# ## setting will take the integer value and append the unsigned value to it. The other
# ## option is "literal", which will use the actual value the user provides to
# ## the unsigned option. This is useful for a database like ClickHouse where
# ## the unsigned value should use a value like "uint64".
# # conversion_style = "unsigned_suffix"
Input and output integration examples
AMQP
-
Integrating Application Metrics with AMQP: Use the AMQP Consumer plugin to gather application metrics that are published to a RabbitMQ exchange. By configuring the plugin to listen to specific queues, teams can gain insights into application performance, track request rates, error counts, and latency metrics, all in real-time. This setup not only aids in anomaly detection but also provides valuable data for capacity planning and system optimization.
-
Event-Driven Monitoring: Configure the AMQP Consumer to trigger specific monitoring events whenever certain conditions are met within an application. For instance, if a message indicating a high error rate is received, the plugin can feed this data into monitoring tools, generating alerts or scaling events. This integration can improve responsiveness to issues and automate parts of the operations workflow.
-
Cross-Platform Data Aggregation: Leverage the AMQP Consumer plugin to consolidate metrics from various applications distributed across different platforms. By utilizing RabbitMQ as a centralized message broker, organizations can unify their monitoring data, allowing for comprehensive analysis and dashboarding through Telegraf, thus maintaining visibility across heterogeneous environments.
-
Real-Time Log Processing: Extend the use of the AMQP Consumer to capture log data sent to a RabbitMQ exchange, processing logs in real time for monitoring and alerting purposes. This application ensures that operational issues are detected and addressed swiftly by analyzing log patterns, trends, and anomalies as they occur.
MySQL
-
Real-Time Web Analytics Storage: Leverage the plugin to capture website performance metrics and store them in MySQL. This setup enables teams to monitor user interactions, analyze traffic patterns, and dynamically adjust site features based on real-time data insights.
-
IoT Device Monitoring: Utilize the plugin to collect metrics from a network of IoT sensors and log them into a MySQL database. This use case supports continuous monitoring of device health and performance, allowing for predictive maintenance and immediate response to anomalies.
-
Financial Transaction Logging: Record high-frequency financial transaction data with precise timestamps. This approach supports robust audit trails, real-time fraud detection, and comprehensive historical analysis for compliance and reporting purposes.
-
Application Performance Benchmarking: Integrate the plugin with application performance monitoring systems to log metrics into MySQL. This facilitates detailed benchmarking and trend analysis over time, enabling organizations to identify performance bottlenecks and optimize resource allocation effectively.
Feedback
Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.
Powerful Performance, Limitless Scale
Collect, organize, and act on massive volumes of high-velocity data. Any data is more valuable when you think of it as time series data. with InfluxDB, the #1 time series platform built to scale with Telegraf.
See Ways to Get Started
Related Integrations
Related Integrations
HTTP and InfluxDB Integration
The HTTP plugin collects metrics from one or more HTTP(S) endpoints. It supports various authentication methods and configuration options for data formats.
View IntegrationKafka and InfluxDB Integration
This plugin reads messages from Kafka and allows the creation of metrics based on those messages. It supports various configurations including different Kafka settings and message processing options.
View IntegrationKinesis and InfluxDB Integration
The Kinesis plugin allows for reading metrics from AWS Kinesis streams. It supports multiple input data formats and offers checkpointing features with DynamoDB for reliable message processing.
View Integration