What to Expect from Flux 1.0

By Nathaniel Cook / Product
Nov 03, 2022

Navigate to:

(Update: InfluxDB 3.0 moved away from Flux and a built-in task engine. Users can use external tools, like Python-based Quix, to create tasks in InfluxDB 3.0.)

This week at InfluxDays we announced that Flux 1.0 is coming soon. Version 1.0 of Flux lang is a commitment to no longer make breaking changes to the Flux language. Importantly, today’s Flux scripts will work on Flux 1.0, and no breaking changes will be introduced between now and the release of Flux 1.0. Along with version 1.0, we have some features we are also releasing soon. Here are the features we have coming and a short explanation of why you might want to leverage them:

  • Modules: Share Flux code within your organization and build useful abstractions over common queries.

  • Editions: Opt into new features of Flux without the risk of breaking changes.

  • Label Polymorphism: Write Flux code that can abstract over the schema of your data.

  • Dynamic Type: Work with JSON data that is fully dynamic without limitations.

The Dynamic Type feature is available today and you can try it out in the cloud. Now let’s dive into some of these features in more detail.

How to use modules

Flux modules are a system that allows you to publish Flux code to a registry that’s private to your organization. Then you can import that code and reuse it anywhere from tasks to dashboard cells and within your application’s Flux code.

Dynamic Type

Graphic illustrating Flux modules and sharing code

Here’s an example of how modules will work once they are released. A common use case we see is that organizations will tag their data with a value that really represents much more data than is present in the tag alone. For example, you might have an env tag on your data that represents the data center where the metrics originated. There are properties of that environment not present in other tags, for example which cloud provider hosts that environment or whether the environment is production or not. Using modules, you can encode this information into a Flux dictionary and then leverage it to annotate data with that extra information.

The first step to using modules is to define a Flux script:

// Notice we give this script a package name since it will be part of a module.
package env

import "dict"

// Map envs to their cloud provider.
providers =
	[
    	"prod01": "aws",
    	"prod02": "aws",
    	"prod03": "gcp",
    	"prod04": "azure",
    	"stag01": "aws",
    	"stag02": "gcp",
	]

// Map envs to their purpose.
// We could just regex match on the name,
// but sometimes it’s important to be explicit.
kinds =
	[
    	"prod01": "production",
    	"prod02": "production",
    	"prod03": "production",
    	"prod04": "production",
    	"stag01": "staging",
    	"stag02": "staging",
	]

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with provider: dict.get(dict: providers, key: r.env, default: "unknown"),
                	kind: dict.get(dict: kinds, key: r.env, default: "unknown"),
            	}),
    	)

Then we publish this module using the API. More details on the publish process will be coming with the release of modules. After publishing, we can import this module in a dashboard cell and make it easy to group the data by cloud provider and filter out the staging environments.

import "modules/env"

// Render median request duration grouped by provider in only production envs
from(bucket: "request")
	|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
	|> filter(fn: (r) => r._measurement == "duration")
	|> env.annotate()
	|> filter(fn: (r) => r.kind == "production")
	|> group(columns: ["provider"])
	|> aggregateWindow(every: v.windowPeriod, fn: median)

Editions and breaking changes

Editions are a key component of the Flux 1.0 release. With 1.0 we will no longer make breaking changes to Flux. However, on occasion there are valuable features that we should add to Flux that would constitute breaking changes. We need a way to safely add these features without breaking Flux scripts. This is where editions come in. An edition is a set of features that are disabled by default. You can opt-in to an edition which will enable those features, but if you do not opt-in nothing will change and your scripts will continue to work without breaking. Because editions are opt-in, you can update your scripts before updating to a new edition to ensure they continue to work as expected. An edition can be selected on a per-script basis or enabled via the API or organization-level settings, providing flexibility in upgrading scripts.

Creating new editions will be a rare event and we anticipate creating at most one new edition per year. Therefore we are naming editions after the year in which they are created. The first edition of Flux will be called 2022.1. We’ll likely cut a second edition of Flux next year and it will be called 2023.1.

Users can enable a new edition by specifying it in their scripts like this:

// Enable a specific edition with this line as the first line in the script
#edition("2023.1")

// The rest of the script follows

What to expect from label polymorphism

Label polymorphism is one of the first features that we will release behind an edition because the feature adds a new syntax to Flux that is breaking.

Label polymorphism allows users to abstract over the schema of their data. Have you ever wanted to write a Flux function that takes a column name as an argument and then uses that name dynamically within the function? This has been a long requested feature of Flux and it’s coming soon as part of the second edition of Flux.

Let’s look at the same example we used earlier showing modules. We wrote this function in the env package:

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with provider: dict.get(dict: providers, key: r.env, default: "unknown"),
                	kind: dict.get(dict: kinds, key: r.env, default: "unknown"),
            	}),
    	)

If we look closely at this function, we realize that it hard-codes a lot of information about the schema of the data. It assumes that the env tag can be accessed as r.env, and additionally it hard-codes the columns provider and kind on the output table. What if we have data that comes from inconsistent systems that sometimes use env and other times use environment? We can use labels to abstract away which columns we are using like this:

// We need to enable the edition in order to use labels
#edition("2023.1")

// annotate adds extra information about the env to the table.
//
// ## Parameters
// - env: Label for the environment column. Default `env`.
// - provider: Output label for the provider column. Default `provider`.
// - kind: Output label for the kind column. Default `kind`.
// - tables: Input data. Default is piped-forward data (`<-`).
//
annotate = (tables=<-, env=.env, provider=.provider, kind=.kind) =>
	tables
    	|> map(
        	fn: (r) =>
            	({r with [provider]: dict.get(dict: providers, key: r[env], default: "unknown"),
                	[kind]: dict.get(dict: kinds, key: r[env], default: "unknown"),
            	}),
    	)

There are three new syntax elements here. First we have .env which defines a label with the value env and the . which tells Flux it’s a label. We use a label value as the default for each of the three column parameters we added to the function. Second, we see r[env]. This is how we access the env column on the record r. Finally, in the output record we see [provider] and [kind], which adds columns on the output record based on the values of the provider and kind labels.

Here’s what it looks like to call this new function, given we are working with a legacy system that uses environment instead of env and already has a tag named kind.

// We need to enable the edition in order to use labels
#edition("2023.1")

import "modules/env"

// Render median request duration grouped by provider in all production envs
from(bucket: "legacy_request")
	|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
	|> filter(fn: (r) => r._measurement == "duration")
	// We can specify the column names when calling the annotate function
	|> env.annotate(env: .environment, kind: .env_kind)
	|> filter(fn: (r) => r.env_kind == "production")
	|> group(columns: ["provider"])
	|> aggregateWindow(every: v.windowPeriod, fn: median)

Neither labels nor edition 2023.1 have been released yet, so the above script will error on existing Flux, but keep an eye out for these features coming soon.

Dynamic type for JSON data

Dynamic type is another new type added to Flux that allows any JSON data to be accurately represented with Flux data types. Flux is a statically typed language that requires container types (i.e. arrays, records, and dictionaries) to contain data of a consistent type. For example, a list of integers is a consistent type, but a list of strings and integers is not. JSON doesn’t have these restrictions, and with the new dynamic type feature we can represent JSON within Flux correctly. Flux is still statically typed, but we’ve added a new type called “dynamic” that allows Flux to handle dynamically typed data.

For example, this JSON was not possible to represent in Flux previously:

{
  "items":  [
  	{"name": "a", "pos": {"x": 10, "y": 10}},
  	{"name": "b", "pos": {"x": 30}},
  	{"name": "c", "pos": {}},
  	{"name": "d"}
	]
}

Unlike these other features, dynamic type is available today.

You can now parse the above JSON without error using the new experimental/dynamic package.

import "experimental/dynamic"

parsed = dynamic.parseJSON(data: jsonDataAsBytes)

This is possible because the function dynamic.parseJSON returns a Flux value with type dynamic.

A dynamic value represents a value whose type is not known until runtime. Flux also can have containers (i.e. arrays etc.) of dynamic values.

Dynamic values aren’t allowed inside Flux tables, so we explicitly convert a dynamic value to a value with a statically known type. This conversion makes use of the array package helper functions and some new functions in the dynamic package.

import "array"
import "experimental/dynamic"

parsed = dynamic.parseJSON(data: jsonDataAsBytes)

// parsed is dynamic so parsed.items is also dynamic
parsed.items
	// convert the plain dynamic value to an array of dynamic values
	|> dynamic.asArray()
	// map over each dynamic value in the array and construct a record
	|> array.map(fn: (r) => ({
    	name: string(v: x.name),
    	posX: int(v: x.pos.x),
    	posY: int(v: x.pos.y),
	}))
	// convert the array of records (no longer containing any dynamic values) into a table
	|> array.from()

For the JSON rows that did not have a pos value, the table will contain nulls.

Final thoughts

There’s a lot included in Flux 1.0. Many new features are coming soon and dynamic type is available today. Keep an eye out for the Flux 1.0 release and new editions of Flux. In the meantime, you can check out my session from InfluxDays 2022.