Announcing Flux (formerly IFQL) v0.0.5 with 12+ New Functions

By Nathaniel Cook / Product, Developer
Feb 27, 2018

Navigate to:

This release represents a big leap forward for Flux.

In the first part of the release cycle, we focused on formalizing the way query operations are chained within Flux. Once that was complete we tried it out, and quickly found that it was easy to add new functions to Flux by composing existing functions together. We were able to add nearly a dozen functions to the language using only Flux code—no Go code was required. In addition we added a handful of new functions backed by specialized Go implementations.

The new functions in the release include:

  • top
  • bottom
  • highestMax
  • highestAverage
  • highestCurrent
  • lowestMin
  • lowestAverage
  • lowestCurrent
  • stateCount
  • stateDuration
  • distinct
  • percentile
  • median

From that list only stateCount, stateDuration, percentile, and distinct required specialized Go implementations. The rest are pure Flux functions.

This large batch of functions is in part due to how we have formalized the way functions are chained in Flux. Every query function accepts a table object and returns a table object. In order to chain them, the functions are called using the return values of the previous functions.

For example, to chain a range operation off the from operation it looks like this:

range(table:from(db:"telegraf"), start:-1m)

Nesting functions in this way has the potential to result in a very long query which is difficult to read and even harder to debug. That is why we have introduced a new syntax to the language to make it easier to describe these chains of operations. The pipe forward operator looks like this |> and allows chaining functions by passing the left hand result to the right hand function call.

// These two lines are equivalent
range(table:from(db:"telegraf"), start:-1m)
from(db:"telegraf") |> range(start: -1m)

Then to apply a filter to the above operations simply add a new |> and filter function call.

from(db:"telegraf") |> range(start: -1m) |> filter(fn: (r) => r._measurement == "mem")

By using this simple model it becomes easy for new functions to be added to the language (both built-in and by the user), that are really just a combination of existing functions. For example, the top operation is a combination of the sort and limit operations. As such the top function can be implemented using this small Flux code snippet:

top = (n, cols=["_value"], table=<-) =>
    table
        |> sort(cols:cols, desc:true)
        |> limit(n:n)

The top function is defined as having three arguments n, cols, and table.

The n argument specifies how many records the result should contain.

The cols argument specifies the list of columns on which to sort the table—it has a default of ["_value"].

The table argument is the source table on which to apply the operation. The table’s default value is the special <- syntax, meaning that it comes from the left hand result of the pipe forward |> operator.

The function body simply chains the sort and limit functions off the table with the provided arguments.

Many of the new functions in this release have been implemented using a similar strategy. If you want to learn more about these more advanced features check out the Flux README as it details what is going on here. Also, some of you may be concerned that implementing top as a sort and limit operation is expensive and slow. You are right, but do not worry as the query planner is able to find these kinds of patterns and rewrite them with a specialized implementation that is efficient.

I’d like to highlight one other new function in this release, the percentile function. It has long been a requested feature to be able to compute percentiles more efficiently using an approximation method instead of computing exact values.

The Flux percentile function by default will return a t-digest approximation unless explicitly asked to compute the exact percentile. The t-digest is an efficient and accurate method for approximating percentiles. Using Flux, the default will be to compute results efficiently by returning good approximations and returning exact results when specifically requested.

This has been a busy release with lots of great additions and development – check out the complete CHANGELOG for all the details. Thanks to all who contributed valuable input throughout the design process this week. We are excited to hear your feedback on the Github project.