Announcing Grade: A Tool to Track Your Go Benchmarks in InfluxDB

By Mark Rushakoff / Product, Use Cases, Developer
Sep 28, 2017

Navigate to:

Have you written Go benchmarks? How often do you run them?

Many Go developers will write and run a benchmark when working on critical code,and then maybe run the benchmark again when modifying that area of the code, to decide whether the change is likely to affect performance. If you follow that use pattern, benchcmp is an excellent utility to compare benchmark output, but if you want to run your benchmarks in CI and track their performance over time with InfluxDB, grade is the tool for you.

To use grade, first you need the output from a benchmark run. For example, here are the results from running go test -bench=. -run=^$ -benchmem ./models/... 2>/dev/null against the InfluxDB v1.0.2 tag:

PASS
BenchmarkMarshal-2                  	  500000	      2901 ns/op	     560 B/op	      13 allocs/op
BenchmarkParsePointNoTags-2         	 2000000	       733 ns/op	  31.36 MB/s	     208 B/op	       4 allocs/op
BenchmarkParsePointWithPrecisionN-2 	 2000000	       627 ns/op	  36.68 MB/s	     208 B/op	       4 allocs/op
BenchmarkParsePointWithPrecisionU-2 	 2000000	       636 ns/op	  36.15 MB/s	     208 B/op	       4 allocs/op
BenchmarkParsePointsTagsSorted2-2   	 2000000	       947 ns/op	  53.85 MB/s	     240 B/op	       4 allocs/op
BenchmarkParsePointsTagsSorted5-2   	 1000000	      1189 ns/op	  69.75 MB/s	     272 B/op	       4 allocs/op
BenchmarkParsePointsTagsSorted10-2  	 1000000	      1624 ns/op	  88.05 MB/s	     320 B/op	       4 allocs/op
BenchmarkParsePointsTagsUnSorted2-2 	 1000000	      1167 ns/op	  43.69 MB/s	     272 B/op	       5 allocs/op
BenchmarkParsePointsTagsUnSorted5-2 	 1000000	      1627 ns/op	  50.99 MB/s	     336 B/op	       5 allocs/op
BenchmarkParsePointsTagsUnSorted10-2	  500000	      2733 ns/op	  52.32 MB/s	     448 B/op	       5 allocs/op
BenchmarkParseKey-2                 	 1000000	      2361 ns/op	    1030 B/op	      24 allocs/op
ok  	github.com/influxdata/influxdb/models	19.809s

(One detail easy to overlook about this output is that the -2 suffix on all the names indicates the test was run with GOMAXPROCS set to 2).

I ran this on an EC2 c4.large instance, under Go 1.6.2, which is what we used to build InfluxDB at that time.If I had this output stored as models-1.0.2.txt, I could run:

grade \
  -influxurl '' \
  -goversion "$(go version | cut -d' ' -f3-)" \
  -hardwareid c4.large \
  -revision v1.0.2 \
  -timestamp "$(cd $GOPATH/src/github.com/influxdata/influxdb && git log v1.0.2 -1 --format=%ct)" \
  < models-1.0.2.txt

Line-by-line, the options are:

  • -influxurl set to an empty string so that I can print the line protocol of what would be sent to a real host
  • -goversion set to the output of go version, without the string prefix go version
  • -hardwareid set to c4.large, so that when querying the data I understand what hardware ran the benchmarks
  • -revision set to the tag of the commit being tested (but I could have just as well used the SHA of the commit)
  • -timestamp set to the Unix timestamp of the commit being tested.

The above command would produce the following line protocol:

go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=31.36,n=2000000i,ns_per_op=733,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.68,n=2000000i,ns_per_op=627,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=208i,allocs_per_op=4i,mb_per_s=36.15,n=2000000i,ns_per_op=636,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=4i,mb_per_s=53.85,n=2000000i,ns_per_op=947,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=4i,mb_per_s=69.75,n=1000000i,ns_per_op=1189,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=4i,mb_per_s=88.05,n=1000000i,ns_per_op=1624,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=272i,allocs_per_op=5i,mb_per_s=43.69,n=1000000i,ns_per_op=1167,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=5i,mb_per_s=50.99,n=1000000i,ns_per_op=1627,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=5i,mb_per_s=52.32,n=500000i,ns_per_op=2733,revision="v1.0.2" 1475695157000000000
go,goversion=go1.6.2\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1030i,allocs_per_op=24i,n=1000000i,ns_per_op=2361,revision="v1.0.2" 1475695157000000000

And if we were to repeat that process for the other tags v1.1.5, v1.2.4, and v1.3.5, we would produce line protocol like:

go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1260,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3424i,allocs_per_op=28i,n=200000i,ns_per_op=6387,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1644800i,allocs_per_op=5002i,mb_per_s=52.81,n=1000i,ns_per_op=2272374,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=36.65,n=2000000i,ns_per_op=627,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.47,n=3000000i,ns_per_op=517,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=336i,allocs_per_op=3i,mb_per_s=44.51,n=3000000i,ns_per_op=516,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=368i,allocs_per_op=3i,mb_per_s=62.39,n=2000000i,ns_per_op=817,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=80.09,n=1000000i,ns_per_op=1036,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=448i,allocs_per_op=3i,mb_per_s=98.62,n=1000000i,ns_per_op=1449,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=4i,mb_per_s=51.44,n=2000000i,ns_per_op=991,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=464i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1412,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=576i,allocs_per_op=4i,mb_per_s=59.45,n=1000000i,ns_per_op=2405,revision="v1.1.5" 1493408827000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=705,revision="v1.1.5" 1493408827000000000

go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,n=1000000i,ns_per_op=1259,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=3048i,allocs_per_op=22i,n=300000i,ns_per_op=5168,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=50.18,n=1000i,ns_per_op=2391554,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=37.14,n=2000000i,ns_per_op=619,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=45.16,n=3000000i,ns_per_op=509,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=44.57,n=3000000i,ns_per_op=516,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=63.47,n=2000000i,ns_per_op=803,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=79.19,n=1000000i,ns_per_op=1048,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.4,n=1000000i,ns_per_op=1468,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=51.63,n=2000000i,ns_per_op=987,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.78,n=1000000i,ns_per_op=1411,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=59.81,n=1000000i,ns_per_op=2390,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=700,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=67.5,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=196,revision="v1.2.4" 1494272869000000000
go,goversion=go1.7.4\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision="v1.2.4" 1494272869000000000

go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=Marshal,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=256i,allocs_per_op=2i,n=1000000i,ns_per_op=1043,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPoint,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=2888i,allocs_per_op=20i,n=300000i,ns_per_op=4945,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=NewPointFromBinary,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=240i,allocs_per_op=1i,n=3000000i,ns_per_op=456,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags5000,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=1404800i,allocs_per_op=5002i,mb_per_s=46.66,n=500i,ns_per_op=2571739,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointNoTags,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=35.96,n=2000000i,ns_per_op=639,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionN,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=43.28,n=3000000i,ns_per_op=531,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointWithPrecisionU,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=288i,allocs_per_op=3i,mb_per_s=42.4,n=3000000i,ns_per_op=542,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=320i,allocs_per_op=3i,mb_per_s=61.32,n=2000000i,ns_per_op=831,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=3i,mb_per_s=77.86,n=1000000i,ns_per_op=1066,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=400i,allocs_per_op=3i,mb_per_s=97.54,n=1000000i,ns_per_op=1466,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted2,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=352i,allocs_per_op=4i,mb_per_s=49.99,n=1000000i,ns_per_op=1020,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted5,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=416i,allocs_per_op=4i,mb_per_s=58.28,n=1000000i,ns_per_op=1424,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParsePointsTagsUnSorted10,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=528i,allocs_per_op=4i,mb_per_s=58.9,n=500000i,ns_per_op=2427,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=ParseKey,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=611i,allocs_per_op=3i,n=2000000i,ns_per_op=782,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeStringField_Plain,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=16i,allocs_per_op=1i,n=20000000i,ns_per_op=66.3,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Quotes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=48i,allocs_per_op=3i,n=10000000i,ns_per_op=169,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_Backslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=80i,allocs_per_op=3i,n=10000000i,ns_per_op=192,revision="v1.3.5" 1504050409000000000
go,goversion=go1.8.3\ linux/amd64,hwid=c4.large,name=EscapeString_QuotesAndBackslashes,pkg=github.com/influxdata/influxdb/models,procs=2 alloced_bytes_per_op=160i,allocs_per_op=5i,n=3000000i,ns_per_op=412,revision="v1.3.5" 1504050409000000000

To get output similar to the Go benchmark output, go to the influx CLI and execute: select revision, ns_per_op, mb_per_s, alloced_bytes_per_op, allocs_per_op from go group by "name".

You will see tabular data like:

name: go
tags: name=ParseKey
time                 revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op
----                 -------- --------- -------- -------------------- -------------
2016-10-05T19:19:17Z v1.0.2   2361               1030                 24
2017-04-28T19:47:07Z v1.1.5   705                611                  3
2017-05-08T19:47:49Z v1.2.4   700                611                  3
2017-08-29T23:46:49Z v1.3.5   782                611                  3

name: go
tags: name=ParsePointNoTags
time                 revision ns_per_op mb_per_s alloced_bytes_per_op allocs_per_op
----                 -------- --------- -------- -------------------- -------------
2016-10-05T19:19:17Z v1.0.2   733       31.36    208                  4
2017-04-28T19:47:07Z v1.1.5   627       36.65    336                  3
2017-05-08T19:47:49Z v1.2.4   619       37.14    288                  3
2017-08-29T23:46:49Z v1.3.5   639       35.96    288                  3

That’s how easy it is to use grade. The decisions you’ll need to make if you use grade are:

  • Am I going to run benchmarks against every commit, one commit per day or per week, only against tags, or something else?
  • What hardware is going to execute my benchmarks, and what operating system am I going to test?
  • Am I going to run with different values of GOMAXPROCS or just the default value on my benchmark runner?
  • Is a 1-second sample long enough, or will I use the -benchtime flag for a longer duration?

And the last decision you’ll need to make is how you’ll act on the data you’re collecting. In the future, we will share TICK scripts to use with Kapacitor so that you can get an alert if a new benchmark indicates a performance decrease.