Quick Start: Telegraf’s Starlark Processor Plugin

By Community / Product, Use Cases
Jul 27, 2022

This article was written by Dr. Phill West. Scroll down for the author’s photo and bio.

After a mortgage payment, energy costs are typically the largest household expense. In my case it was an easy decision to install solar panels, but I wanted to perform in-depth analyses with historical data. Deploying monitoring sensors was straightforward; collecting and processing the raw data became the main challenge.

Telegraf and InfluxDB are ideal choices for managing time series data. Although I had no prior experience, a Docker instance of Telegraf was onboarded in no time. With data collection working, the emphasis shifted to processing.

This article describes building a custom (Starlark Processor) plugin that transforms power meter readings into something much more usable for analysis and visualization.

What is Starlark?

Although there are a large number of Telegraf plugins, situations arise where collected data requires custom processing. The Starlark Plugin is a programming language interface with Telegraf that allows advanced math and data manipulation on the incoming data stream. Telegraf provides a processor and aggregator version giving considerable flexibility as to how and where the transformations are applied.

Starlark is strongly influenced by Python, and is almost a subset of that language. Starlark is intended for expressing configuration. Its programs are short-lived and have no external side effects.

Starlark Processor use case

In this solar energy monitoring example, power meters track total solar panel production and total site consumption along with a large amount of health/status information. Telegraf gets data from this endpoint via the [[inputs.http]] plugin which parses the production and consumption meter readings from a JSON array. Each extracted metric has the same measurement name, “site-foo”.

[[inputs.http]]
  urls = [ "energy monitoring URL" ]
  data_format = "json_v2"

  [[inputs.http.json_v2]]
    measurement_name = "site-foo"

  # Parse Production & Consumption Data
    [[inputs.http.json_v2.object]]
      path = "location in JSON array"
      included_keys = ["production", "consumption"]
      tags = ["subtype"]
      [inputs.http.json_v2.object.renames]
        subtype = "power meter"

The power meters record cumulative values in kilowatt-hours (KWh). To determine power levels at this moment in time, the difference between the current measurement and the last one must be calculated. But the prior meter readings have already passed through this plugin and are no longer available. Fortunately, the Starlark Processor Plugin easily performs this task.

[[processors.starlark]]

  ## Source of the Starlark script.
  source = '''
state = {
  "last_prod": None,
  "last_cons": None
}

def apply(metric):
  for k, v in metric.tags.items():
    if k == "power meter":
      if v == "Cumulative Production":
        # Load from the shared state the metric assigned to the key "last_prod"
        S = state["last_prod"]
        # Using deepcopy, store the current metric into the shared state and assign it to the key "last_prod"
        state["last_prod"] = deepcopy(metric)
        metric.tags["power meter"] = "production"
        # Compute and add field for delta power
        if S != None:
          metric.fields["delta_power"] = metric.fields["total power"] - S.fields["total power"]

      elif v == "Cumulative Consumption":
        S = state["last_cons"]
        state["last_cons"] = deepcopy(metric)
        metric.tags["power meter"] = "consumption"
        # Compute and add fields for delta powers
        if S != None:
          metric.fields["delta_power"] = metric.fields["total power"] - S.fields["total power"]

      else:
        # ERROR - This condition should never occur
        print ("ERROR: Unrecognized value for Power Meter")

  return metric
'''

The processor “remembers” the prior measurement by using the deepcopy function to store it. Once the next measurement is received and parsed, its next stop is the Starlark processor. The difference is calculated and applied to the current measurement as a new field named “delta power”. The original data is unaltered, but the resulting measurement now contains a new field which is the “instantaneous” power for this timestamp. Deepcopy sets the current measurement as the new prior one, and the process repeats.

Telegraf’s interface with Starlark is unique because it exposes additional functionality not available in other plugins. Since Starlark is a language, sophisticated computations, logic, and/or data manipulation are possible. Measurements can be altered with surgical precision as they flow through Telegraf’s chain of plugins.

Syntax

• Indentation is very important and can be tricky. In particular, the transition from Telegraf TOML to Starlark code is outdented — opposite to the normal indentation convention. From that point Starlark language syntax governs the indentation requirements.

• The VS Code extension for Starlark is helpful for syntax checking and debugging.

• Starlark source code is enclosed by 3 single quotes (ASCII 0x27). Although single-quoted strings are permitted within the body of Starlark code, double-quoted strings are preferred to improve readability and avoid confusion with the single quote delimiters which enclose the entire code block.

  ## Source of the Starlark script.
  source = '''
state = {

Starlark source code is initially outdented.

Conclusion

This project materialized in less time than expected due in large part to the wealth of resources available. The Starlark language specification is essential for grasping fundamental language concepts, syntax and example usage. The Starlark processor and aggregator GitHub documentation contains several helpful examples. In this video - How to Use Starlark in Telegraf, Samantha Wang demonstrates several examples taken from the GitHub repository. The Community Forum is an excellent place to get personalized help for specific problems.

About the author

Dr. Phill West is a retired aerospace manager. His dissertation focused on computer-aided design of control systems, an emergent technology in the 1980’s. With a deep interest in offgrid living, he now pursues technologies that reduce environmental impact and promote a sustainable future.