Zipkin and Datadog Integration

Input and output integration overview

The Zipkin Input Plugin allows for the collection of tracing information and timing data from microservices. This capability is essential for diagnosing latency troubles within complex service-oriented environments.

This plugin writes to the Datadog Metrics API and requires an apikey which can be obtained for the account. This plugin supports the v1 API.

Integration details

Zipkin

This plugin implements the Zipkin HTTP server to gather trace and timing data necessary for troubleshooting latency issues in microservice architectures. Zipkin is a distributed tracing system that helps gather timing data across various microservices, allowing teams to visualize the flow of requests and identify bottlenecks in performance. The plugin offers support for input traces in JSON or thrift formats based on the specified Content-Type. Additionally, it utilizes span metadata to track the timing of requests, enhancing the observability of applications that adhere to the OpenTracing standard. As an experimental feature, its configuration and schema may evolve over time to better align with user requirements and advancements in distributed tracing methodologies.

Datadog

Datadog metric names are formed by joining the Telegraf metric name and the field key with a . character.

Field values are converted to floating point numbers. Strings and floats that cannot be sent over JSON, namely NaN and Inf, are ignored.

Setting rate_interval to non-zero will convert count metrics to rate and divide its value by this interval before submitting to Datadog. This allows Telegraf to submit metrics alongside Datadog agents when their rate intervals are the same (Datadog defaults to 10s). Note that this only supports metrics ingested via inputs.statsd given the dependency on the metric_type tag it creates. There is only support for counter metrics, and count values from timing and histogram metrics.

Configuration

Zipkin

[[inputs.zipkin]]
  ## URL path for span data
  # path = "/api/v1/spans"

  ## Port on which Telegraf listens
  # port = 9411

  ## Maximum duration before timing out read of the request
  # read_timeout = "10s"
  ## Maximum duration before timing out write of the response
  # write_timeout = "10s"

Datadog

[[outputs.datadog]]
  ## Datadog API key
  apikey = "my-secret-key"

  ## Connection timeout.
  # timeout = "5s"

  ## Write URL override; useful for debugging.
  ## This plugin only supports the v1 API currently due to the authentication
  ## method used.
  # url = "https://app.datadoghq.com/api/v1/series"

  ## Set http_proxy
  # use_system_proxy = false
  # http_proxy_url = "http://localhost:8888"

  ## Override the default (none) compression used to send data.
  ## Supports: "zlib", "none"
  # compression = "none"

  ## When non-zero, converts count metrics submitted by inputs.statsd
  ## into rate, while dividing the metric value by this number.
  ## Note that in order for metrics to be submitted simultaenously alongside
  ## a Datadog agent, rate_interval has to match the interval used by the
  ## agent - which defaults to 10s
  # rate_interval = 0s

Input and output integration examples

Zipkin

1. Latency Monitoring in Microservices: Use the Zipkin Input Plugin to capture and analyze tracing data from a microservices architecture. By visualizing the request flow and pinpointing latency sources, development teams can optimize service interactions, improve response times, and ensure a smoother user experience across services.

2. Performance Optimization in Essential Services: Integrate the plugin within critical services to monitor not only the response times but also track specific annotations that could highlight performance issues. The ability to gather span data can help prioritize areas needing performance enhancements, leading to targeted improvements.

3. Dynamic Service Dependency Mapping: With the collected trace data, automatically map service dependencies and visualize them in dashboards. This helps teams understand how different services interact and the impact of failures or slowdowns, ultimately leading to better architectural decisions and faster resolutions of issues.

4. Anomaly Detection in Service Latency: Combine Zipkin data with machine learning models to detect unusual patterns in service latencies and request processing times. By automatically identifying anomalies, operations teams can respond proactively to emerging issues before they escalate into critical failures.

Datadog

1. Basic Metric Submission: Utilize the Datadog Output Plugin to transmit metrics from your Telegraf instance to Datadog. By configuring the apikey and enabling necessary metrics, you can easily monitor application performance over time.
2. Debugging Write URL: In cases where you need to troubleshoot your metric submissions, you can override the default write URL with a custom endpoint to debug the metrics being sent, ensuring that they are reaching the correct destination.
3. Proxy Configuration: If your network setup requires routing through a proxy for outgoing requests, use the http_proxy_url option to set the appropriate proxy. This allows for seamless integration in restrictive network environments.

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.