InfluxDB and the /debug/vars Endpoint
By
Mark Rushakoff /
Product, Use Cases, Developer
Apr 03, 2017
Navigate to:
Like many Go programs with an HTTP server, InfluxDB exposes some diagnostic information over the /debug/vars endpoint. Because the information we expose there is simply JSON, it should be very straightforward to expose InfluxDB’s diagnostics to other custom utilities that you might want to use to monitor your InfluxDB instance.
Go makes it trivial to add a /debug/vars to your program via the /debug/vars package:
Package expvar provides a standardized interface to public variables, such as operation counters in servers. It exposes these variables via HTTP at /debug/vars in JSON format.
expvar works great for most use cases. Even though InfluxDB started using expvar directly, there was, and still is, a missing feature that caused us to stop using the standard expvar package: it does not allow you to remove published variables. There were at least a couple of issues filed about stats being reported for entities that were deleted. A low signal-to-noise ratio is never helpful when you're trying to debug an active problem.
In InfluxDB PR 6964 (which was first present in the 1.0 release) we moved away from using expvar directly, in favor of a custom implementation with expvar-compatible output.
Kapacitor took a slightly different approach: forking the standard library expvar package to add a Delete method on the Map type.
/debug/vars
InfluxDB's expvar Format
The output of InfluxDB’s /debug/vars endpoint is one JSON object that roughly corresponds with the contents of the _internal database. The_internal database stores the values every 10 seconds by default, but the /debug/vars endpoint, like the SHOW STATS query, gives you an instant-in-time view of those stats.
First, there are two key-value pairs that match the standard expvar output:
cmdline is an array of strings representing the command line arguments used to invoke the process. If you're running influxd as a service on Linux, the value for cmdline will look something like: ["/usr/bin/influxd","-config","/etc/influxdb/influxdb.conf"].
memstats is a JSON object corresponding to the Goruntime.MemStats struct.
The rest of the /debug/vars output have keys representing the details of the object being measured, with values in "InfluxDB expvar format". The "InfluxDB expvar format" is an object with this structure:
name: a string describing what's being measured, i.e. the corresponding InfluxDB measurementtags: an object with string keys and values, corresponding to the tags to use for the fieldsvalues: an object whose keys and values correspond with fields for the measurement
Kapacitor's /kapacitor/v1/debug/vars Endpoint
Kapacitor uses a mix of “standard” expvar format with simple top-level key-value pairs, and InfluxDB-formatted expvars for received writes underneath the kapacitor key.
Telegraf's InfluxDB Input Plugin
Telegraf’s Influxdb input plugin supports ingesting InfluxDB-formatted expvars from a remote HTTP endpoint. It also handles memstats as a special case. For a single instance of InfluxDB, this will result only in information redundant with the _internal database, but this is a straightforward approach to monitoring multiple InfluxDB instances in one central location.
Unlike InfluxDB and Kapacitor, Telegraf does not (currently) expose a /debug/vars HTTP endpoint.
Emitting InfluxDB-Formatted expvars in Your Own Application
I haven’t seen any standalone projects that allow you to emit InfluxDB-formatted expvars, but it doesn’t take much code to do that yourself if you understand the format. The entire file of our first, pure-expvar implementation is less than 50 lines, and the corresponding code to serve expvars over HTTP is only about 11 lines.
Have you implemented code to emit InfluxDB-formatted expvars, in Go or in any other language? Share your solutions over on this thread on community.influxdata.com.
