How to Use Telegraf and Its Plugin Ecosystem

Session date: Dec 03, 2020 08:00am (Pacific Time)

Telegraf is the open source server agent which is used to collect metrics from your stacks, sensors and systems. It is InfluxDB’s native data collector that supports over 250+ inputs and outputs. Learn how to send data from a variety of systems, apps, databases and services in the appropriate format to InfluxDB. Discover tips and tricks on how to write your own plugins.

Join this webinar as Jessica Ingrassellino and Samantha Wang dive into:

• Types of Telegraf plugins (i.e. input, output, aggregator and processor)
• Specific plugins including Execd input plugins and the Starlark processor plugin
• How to create your own Telegraf plugin

Additional resources:

• Getting started with Telegraf
• Telegraf plugins
• Telegraf GitHub
• External plugins guide

Click here for presentation

Watch the Webinar

Watch the webinar “How to Use Telegraf and Its Plugin Ecosystem” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “How to Use Telegraf and Its Plugin Ecosystem”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• • Caitlin Croft: Customer Marketing Manager, InfluxData
  • Jessica Ingrassellino: Engineering Manager, InfluxData
  • Samantha Wang: Product Manager, InfluxData

Caitlin Croft: 00:00:04.421 All right. I think we will get started here. Once again, hello everyone and welcome to today’s webinar. My name is Caitlin Croft. I work here at InfluxData and I’m very excited to have Jessica and Samantha here to talk about Telegraf and its entire plugin ecosystem. So before I hand it off, once again, friendly reminder, please feel free to post any questions you may have in the Q and A box in Zoom, and I will be monitoring it throughout the webinar, and we will answer all questions at the end. So without further ado, I’m going to hand things off to Samantha and Jessica.

Samantha Wang: 00:00:46.951 Cool, thank you, Caitlin. I think I’m going to get started here. So hi everyone, I’m Samantha. I’m one of the product managers here over InfluxData, at InfluxData, working on Telegraf and data acquisition as a whole. And Jess is our engineering manager over the same thing that will be speaking, will be taking over this presentation in the later half. So yeah, feel free if you have any questions for us, as Caitlin mentioned, drop it in the chat and we’ll probably answer at the end or hopefully maybe even answer some during the presentation. But yeah, so let’s get started. So what are we going to talk about today? Just a really beginner intro into Telegraf and its whole plugin ecosystem and how to extend beyond the ecosystem. But we’re going to talk about what is Telegraf - it’s a plugin ecosystem, in general - configuration basics, and how to also extend outside of that plugin ecosystem as well.

Samantha Wang: 00:02:00.318 So getting started, we’re going to talk first about InfluxDB and Telegraf too and how they’re related and the current state and progress that InfluxData has been making with InfluxDB and Telegraf. So this is InfluxData’s current platform architecture. We have kind of your core storage engine InfluxDB that is the core of InfluxData. Its purpose-built time series database that includes visualizations, querying, that launch in all of our 2.0 products. Flux, which is our purpose-built time series querying and scripting language, it’s all included in the InfluxDB platform. And you can see on the left hand side is where the data injection piece comes into play. We have what we’re going to talk a lot more about today, Telegraf, where you’re going to be having all your devices, your public cloud, private cloud, your data centers, IoT devices on the left that you can feed data in with Telegraf and send it over to InfluxDB. And once all that data is stored into InfluxDB, you can alert on it, do machine learning on it, service discovery, doing other querying language, doing that actionable work, getting those insights that you want to get from your data. So it all starts out with Telegraf. And there’s a few other methods of data ingestion that we’ll talk briefly over, but the core and what we definitely recommend, especially from a time series perspective for continuously ingesting data, Telegraf is definitely one of the best solutions even out there for you to ingest your data.

Samantha Wang: 00:04:03.005 So before we get a little deeper into Telegraf, just talking about InfluxData’s products and offering as a whole, InfluxDB Cloud is really our new and exciting product that we’ve been spending a lot of time and effort here on the engineering team at Influx, just growing the product. We’re on all three, actually, cloud platforms: GCP, Google Cloud, AWS, and Azure. We just even launched - we’re growing locations constantly. I think last Monday we launched our AWS East location. We already have locations on the west as well. And so definitely check that out InfluxDB Cloud is free to sign up - no credit card required - pay as you go, which we’ve actually seen that model fit a lot of use cases out there. So if you’re looking for a place to try out Telegraf, you want to see your data somewhere and visualize it, I would definitely recommend trying out InfluxDB Cloud and seeing how that fits your use case and potentially how your entire environment can grow by using InfluxDB Cloud.

Samantha Wang: 00:05:23.035 InfluxDB Enterprise is a really popular product as well. It’s a really powerful on-premises solution for a lot of our enterprise customers. A lot of things that are included with InfluxDB Enterprise that wouldn’t necessarily be with OSS are high availability and security that a lot of our large enterprise customers want. So if you’re looking for an on-premises, large-scale solution, InfluxDB Enterprise is something that our sales team is still working with - continues working with all the time and it is also a great solution that’s out there as well. So for the bottom two, our open-source code, Telegraf is open source, and we have our InfluxDB OSS 1.x and 2.0 that just GA’d earlier this month at our InfluxDays North America session. So definitely take a look, too, if you’re looking for an on-premises solution, something that you just want to play around with, get familiar with. If you want an OSS database to fit into your other applications you’re building and you want some open-source code, definitely check out InfluxDB, the OSS 2.0 version. It comes with a lot of the new exciting features, Flux, the great new UI that we’ve been improving a lot on from the 1.x version. And you can use Telegraf to feed data into all of these platforms as well, so.

Samantha Wang: 00:07:13.581 So I’m going to just talk briefly about how our stack changed from 1.x to 2.0. I think this is just kind of important in understanding how Telegraf fits into it. So on the left hand side, we have something, what, if you’re familiar with InfluxData, InfluxDB, you would know something called the TICK stack, which essentially is Telegraf, InfluxDB, Chronograf, and Kapacitor. And so we have Telegraf on this left hand side that we saw in the diagram before and all of the other ICK components on the right hand side. And so InfluxDB 2, all those ICK components got put in together in one place in what we call - into this InfluxDB 2.0. So we have all your stored, your querying, your visualization all in one place, in what we call InfluxDB 2. But we still have Telegraf out there that is that data collection agent on the outside that will feed into InfluxDB 2. So I wanted to just show this so you could understand how we evolved from the TICK stack with Telegraf into InfluxDB 2.0 and how Telegraf plays into that, into that game. So yeah, so Telegraf is still really similar, but then you still have all that power of InfluxDB 2.0 on the right-hand side.

Samantha Wang: 00:08:42.303 So I quickly wanted to touch base on just all our current data acquisition methods. We have broken them into four different quadrants. Our agent-based push, which is our really bread and butter, what a lot of our users love - we’ll obviously talk about a lot more. Client libraries is a popular solution as well if you’re just familiar with Python, Java and wanted to use client libraries, that’s a method as well. We have agentless pull that we’re still growing and working on, especially with Flux. We’ve been able to address a lot of Flux “From” functions that can pull from URLs, Prometheus scrapers on our OSS version. It’s a method to get data in quickly without having to download an agent. And then we have some native third-party ecosystems, JMeter, NiFi, that allows you to easily ingest from other methods. But Telegraf is really our no (low) code solution. We like to say that [inaudible] show you the coding that you need to do, which is like filling out a form. It’s a really large community Jess started one or two months ago. She’s done a lot of work in the time that she’s been here, but really the help that the community - just seeing how great our community is and being able to help out the team. It’s lightweight and powerful, and it’s customizable as well, that open source aspect, being able to do what you want with it and build on top of it.

Samantha Wang: 00:10:32.621 So yeah, let’s focus on Telegraf, solely, now and all the awesome things about it and its plugin. So what is Telegraf? It’s a plugin-driven server agent. It’s written in Go, single binary, no external dependencies. Very similar to a lot of our other InfluxDB products. It’s designed for very minimal footprint for CPU and memory. It’s definitely known as one of the most efficient data collection agents just out in the industry, and it’s really optimized for writing data into InfluxDB with formatting, retries, tag sorting. So if you’re looking for an agent to write into InfluxDB, specifically, definitely use Telegraf in suggesting just converting your data into line protocol and getting all that data easily in there. And lastly, it’s optimized for streaming data which is really important for time series data. You could set your agent to the nanosecond level if you want to look at your time series data that granularly. I don’t know if that’s a word. But yeah, Telegraf is definitely optimized for that, for getting in your real-time data into InfluxDB, which is what a lot of our people come to look to us for.

Samantha Wang: 00:12:03.156 So we’re going to talk a little more about the plugins, but the Influx plugins are essentially the source of where you’re going to tap into [inaudible] from the systems running into your database. We have Apple plugins as well. They’re really popular, too. We are keen to having - we actually have a lot of our competitors out there. We have Apple plugins for Datadog, OpenTSDB, Timestream. You can write to multiple outputs at the same time, so you can write to InfluxDB and to Datadog if you want. So if you’re moving over from Datadog to InfluxDB, you can slowly transition over by dual writing to both databases and then slowly move over to InfluxDB as a whole. Metrics routing and conversions is huge and an aggregation for InfluxDB as well. So I want to talk a little about the benefits of the Telegraf agent before we go deeper into the plugins. So we talk a lot about just how we have 200 plus plugins for Telegraf, but there’s a lot of benefits from the actual - a lot of act just key functionality from the Telegraf agent itself. The first thing is really just all the benefits of a robust scheduler. It adjusts for clock drift, so it adjusts for if you get close to the end of the interval on your computer and other data collection agents may skip to the next interval anyways, but our timer will adjust for that clock time to make sure that all the metrics that will arrive into your database.

Samantha Wang: 00:13:56.619 For polling plugins, the agent itself will go and tell the plugin that it’s time to go and collect data from their input, so will adjust for a job scheduling - just scheduling issues that may occur if there’s any problems that come up from a pulling perspective. A lot of in-memory metric buffers - it will keep things going even if one of the downstream databases is temporal or temporarily unavailable. Reliable. Reliable metric guarantees with backpressure support in certain plugins. Really popular in plugins like Kafka. Full streaming support. Metric routing is a really big one. We often get questions on, “If I even only want my metrics from - that end in 002 to go to database A, versus 003 go to database 2 or database B,” Telegraf is something that you could easily route metrics for with our tags. Flexible parsing and formatting and sterilizing. This is a huge thing as well. Taking pride into parsing all those different types of data that are out there. JSON, CSV, Prometheus. We actually, I think, will be having a Prometheus part - just got merged in yesterday, I think, actually. So, flexible parsing. And if you don’t see one out there, just what we’ve been talking about this more later, in a bit, feel free to build one yourself.

Samantha Wang: 00:15:50.547 So going into just talking about our plugin ecosystem. Yeah, I touched base on just the four - we have four type of plugins that we look at: input plugins, output plugins, processors, and aggregators. So input plugins are where you’re going to be collecting the data and output plugins are where you’re going to be writing the data. The processors are where you’re going to be transforming that data, decorating and filtering, and aggregators are where you can create aggregate metrics such as pulling the min and max of your time-interval data and get that data processed into your database. On the left hand side, this is how you can easily look at our data and our documentation. If you’re curious on what Influx - what kind of input/outputs and you want to easily search on the plugin categories as well. These are just the different plugin types that we have. We have, obviously, over 170 input plugins, so all of these different types of input methods on getting your data in. 40 output plugins and the various processors and aggregators as well. So I’m going to also quickly go over just - we’ll go over these quickly, and I think in a Q&A session, we’ll be showing these again at the end. Just these are just a huge list of our input plugins so you can just see everything that we cover. You can see just even Kinesis, Nginx - a lot of Nginx plugins out there and just adding on our strong community. Most of these plugins are community written. We’re constantly getting new plugins written by the community. So a good majority of these plugins are written by our community.

Samantha Wang: 00:17:57.943 So here’s a diagram of just how these four different plugins take place into Telegraf. I think this is a good visualization on how they act. You can use them in Telegraf itself. I think Jess will go in a little more on the - you’ll see if from a configuration perspective. But you’ll see that - you’ll set your input plugins - so let’s just talk basically your CPU memory disc metrics. And then say you wanted to transform or filter them. You would add a processor plugin. So say you wanted to change a field that you have from one of your plugins into a tag. You would do that with a processor plugin. And then next, say you wanted to keep only the mean of every minute of data coming in. You can use an aggregate plugin. In all of those transformations, what happened before it gets outputted to your Apple plugin, so. Our example Apple plugins here are InfluxDB file, Kafka, all the inputting, processing, and aggregating would happen before your data gets sent out to your Apple plugin - to your output using your Apple plugin.

Samantha Wang: 00:19:21.223 So also, can’t find the plugin you’ll need? Jess will definitely talk about this a little more. Feel free to write your own. We have a lot of good training out there on how to write your own plugin, a contribution guide, and we have a really good community as well. Caitlin mentioned our slack at the top of this meeting. We have even a Telegraf dash dev, I think, community channel out there that is for a lot of people who are writing, not even just plugins, but PRs, features, enhancements, and bug fixes. If you have any questions, feel free to drop in there. And you can get some help out from not only our engineers, but our really strong community too. And here’s a look at just some of our popular really enterprise plugins that a lot of people have used. I think just Kubernetes is a really popular one out there. SNMP, just for network monitoring, is stuff that we’ve seen a lot of growth in. We released an SNMP trap plugin, I think, earlier this - yeah, we’re in December. Earlier this year. And we have a series of IoT plugins that are included as well. We recently launched OPC-UA and I think [we released before that Modbus. But our IoT plugins are really growing in popularity recently, especially with our InfluxDB Cloud product.

Samantha Wang: 00:20:57.115 So I mentioned parsing. Here, a list of just various parsings that people used to - our JSON, CSV, and Graphite are probably the most popular parsing methods, JSON, especially. For example, MQTT will often report out into JSON, and parsing that data is something that you can use with our - that our Telegraf parser is really easy to use. And I think one of the main things, too, is we want to make our configuration a little simpler than what you would do if you were parsing with a Python script or something. So that’s my portion for now. I think, actually moving over to talking about configurations, that’s where Jess is going to pick up.

Jessica Ingrassellino: 00:21:48.145 Yeah, great. Thanks, Samantha. Hi, everyone. I’m Jessica Ingrassellino and I am the engineering manager for the data acquisition team which includes Telegraf, and I’m really excited to be sharing with you today, especially as I’m getting to learn about more and more about our community and our users. So Samantha, you can slide forward. So one of the benefits of Telegraf is that you can work with configurations in the no-code or low-code manner. Every plugin has a TOML file. It’s easily configurable. It’s as simple as filling out a form. As Samantha mentioned, you can have multiple plugins per Telegraf and that can be easily configured in a single configuration file. So, depending on your architecture, you could have either thousands of Telegraf instances with only one or two plugins running, or you might be a hobbyist who is doing some smart homework and you might have a single Telegraf that has multiple input, output, processor, and aggregator plugins running. It’s really up to you.

Jessica Ingrassellino: 00:23:05.479 So the configuration file - Telegraf requires a config file. The default file is quite long. If you, in fact, take it and convert it to PDF, it’s 111 pages and it contains the setup for everything, including the Telegraf agent. So you can specify your collection interval, your jitter, buffer and batch size. You can add global tags to make sure that all of your metrics from all the inputs have the same tagging, if you’d like. And you can also enable all the outputs, processors, aggregators, and inputs and specify their configurations. And the thing that I want to point out is that you don’t need to use the entire config file. So if we can move to the next slide, I’d like to show you how you can actually reduce the size of that file for your reading. So in order to avoid reading through that long file, you can actually generate different config examples for your use case or you can even play with different plugins. So you can open the telegraf.conf file in a text editor and you can work with it that way, or you can also use the command line to edit the file directly and write those changes to your conf file.

Jessica Ingrassellino: 00:24:36.152 So in this example, you’ll see that I’m specifying using the different command line flags in input, plugin for CPU and system, the output to InfluxDB. And I could’ve specified multiple outputs, the Starlark processor, and the value counter aggregator. And you can see that I’m writing those changes directly to my config file. Once you’ve done that, you can run your config file with a test tag via command line, and that will give you an output to let you know that your config file is formatted correctly before you actually run your Telegraf. So it’s a really helpful way to make sure that you are getting what you want, how you want it, before you get started without having to read through a massive, massive file. So I definitely recommend checking out the documentation on that. So the next aspect that I’d like to talk with you about, and something that I really find amazing about Telegraf, is that not only do we have a massive ecosystem of plugins as you saw with over 200 plugins altogether, but this is an ecosystem that can be extended. So there are a few ways that you can extend Telegraf. You can go to the next slide. You can work within our open source community, so you can chat in the community, ask questions, you can submit upgrades or enhancements for existing plugins, and you can also use Execd to write a plugin in GO or a language of your choice. And I think that Execd is really powerful aspect of Telegraf, so I’d like to talk a little bit more about what Execd offers you to extend Telegraf for your specific needs.

Jessica Ingrassellino: 00:26:32.151 So some basics about Execd, for those of you who might not know about it, Execd plugins run in their own process as a daemon. They require line protocol. They’re a great way to utilize the Telegraf agent features and avoid the need for review by the Telegraf team or including a plugin in the Telegraf repository or in the binary. So Execd plugins support the same API as an internal plugin. They can be used for Go and non-GO, and they can be used for licensed software plugins as a result of the fact that they don’t need to be in the binary to run. Also, you can make an Execd for any type of plugin, so you can do input, output, processor, or aggregator plugins through Execd. So this diagram is a visualization of how Execd works in Telegraf. And this communicates, essentially, the - Execd runs as a plugin within Telegraf, as you can see on the left, and communicates with the external binary over standard in and standard out using line protocol. So in the case of an Execd input plugin, Execd input reads line protocol metrics coming from your standard out. So to write an Execd input, you’d want to start your process, write those metrics to your standard out, and then Telegraf will take care of it. And the processor Execd plugin, Telegraf would send your metrics to your process. Your process is parsing, modifying, doing whatever you need to do, and then serializing it back to line protocol and writing to that standard out. An aggregator plugin follows nearly the exact same process as the processor plugin. And in the case of output plugins, Telegraf writes metrics using your standard in, and you’re writing them to an external service.

Jessica Ingrassellino: 00:28:33.054 Some people have asked about performance differences with Execd as opposed to running plugins within the binary, and because it is streaming, the total time for your data to go through the processor is fairly quick depending on what you’re doing, and generally, the throughput is good. It’s pretty similar to running it through a plugin that’s a part of the binary. So the Execd configuration is pretty much the same as the internal plugin configurations. They can be edited in the same way and you can add and change options the same way within the configuration file. So the great thing about Execd plugins is that if you have a shop that uses all Python or all Ruby or Java, you can write a plugin in any language. And we have two ways that we talk about these. We refer to them as shim and no shim. So the no shim plugins are written in a language other than Go, calling your external process. And the parsing and serializing of your data, in that case, would need to be written as a part of your plugin’s code. So if you’re working in other languages, this is a great way to go because you don’t need to learn GO in order to write what you need to process. Execd plugins using the shim are plugins written with Go that use our Shim, which was written by developer Steven Soroka to do the parsing and the serializing of your data. So if you’re comfortable with Go, it’s great to use the shim because not only do you get the parsing and serializing that’s built into Telegraf for free - the shim will also set up a logger for you by default. And by writing it in Go and using the shim, it’s easier if at some point you do want to submit your plugin for potential inclusion into the binary.

Jessica Ingrassellino: 00:30:46.142 So I think that is it for me.

Samantha Wang: 00:30:49.667 Yeah. We want to just end it with some quotes. I think actually Caitlin gave these to me from some of our - from some of our customers that we have, just what they’ve said about Telegraf and good summaries of how they can use it. So one of our customers mentioned that Telegraf is like a Swiss Army knife for connecting various MQTT sources or OPC-UA sources. And I think that really just plays into the aspect of the variety of plugins that you can use with it, manipulating stuff. I think the processors and aggregators as well, you can do a lot of manipulations of - we talk about routing data and cleaning up your data. Actually, the recorded talk that you can find on YouTube that I did for InfluxDays actually talks specifically about the various ways you can use Telegraf as that Swiss Army knife for transforming data. So if you’re curious more about that, check that out. And then one of our big customers, Wayfair, mentioned that “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” So yeah, those four things were really essential to Wayfair and they were able to do with Telegraf and really get this robust data topology that they’ve mentioned by using Telegraf as their streaming service. And you can imagine the type of environment that Wayfair is running and constantly growing on.

Samantha Wang: 00:32:54.088 So yeah, I wanted to just end it on those two. I think if you take a look at our - if you get a hold of these slides, here are some links to getting started with Telegraf, a list of Telegraf - the list of 200+ plus plugins, Telegraf plugins. Telegraf GitHub page has a lot of good information, along with our InfluxData documentation if you’re more on the developer side and you want to write your own plugin. And then the external plugin guide that Jess had mentioned. We’d love to see - we have a good number of these, but we’d love to see even more of them. So if you’re going to write a plugin that you think that you want to get it in quickly and using it, you want other people to start using it, definitely submit an external plugin using the external plug guide. But I think that it for the both of us, so. feel free to put your questions in question channel. I think Caitlin will get them out to us, so.

Caitlin Croft: 00:34:02.220 Perfect. Thank you, Samantha. Thank you, Jessica. That was fantastic. So that first quote, just for those on the webinar. It’s really cool. It’s a print manufacturer in Germany, and so they’re using InfluxDB to extract a lot of their data from all the different machinery within a print manufacturer, and they’ve been able to correlate the uses of paper with different types of ink and looking at how as machines get older, the degradation of the machines actually can cause that ink to become thicker and then they ended up using more ink. So Telegraf was really instrumental to them. So I just want to give that bit of context. Thank you so much, both of you. We have lots of questions. I know people are super excited. So the first question here is, what kind of testing methodologies or methods are you using to work on this stability and correctness of the product?

Jessica Ingrassellino: 00:35:00.594 Yeah, I can go ahead and address that. So of course every - you’ll notice that if you do any kind of a community contribution that one of our requirements for contributing a plugin or any code is to add unit tests, so we have robust unit testing. As you can imagine, we make builds for Mac, Windows, and Linux, so we run on all the operating systems. We have 200 - as we said, 200 plus plugins. So really, our community users give us a lot of information about the use cases that we can then test either through setting up our own testing, which we do before we release plugins. We are also working on a really exciting project to enable people to do integration testing of their specific plugins. So that’s upcoming in January. So we really do use a combination of the automated testing methodologies as well as exploratory methodologies, integration methodologies to make sure that we are addressing, to the best of our ability, our really broad ecosystem across all of the different potential use case areas.

Caitlin Croft: 00:36:29.063 Perfect. What is the minimal sampling interval that can be used with Telegraf? The template indicates values and seconds for IoT-type of applications which require a slightly shorter sampling and reporting intervals. How can this be done?

Samantha Wang: 00:36:49.595 I’m trying to - I’m also reading the question too. The minimum? I know I’ve been able to configure it to the nanosecond level and I think we can even - NASA’s one of our customers we talk about, right Caitlin? They’re one of our - they’re one of probably the example of using it at that granular of a level. So yeah, if you’re talking about just the interval that Telegraf would read, you can easily - just setting your interval to one nanosecond is something that is definitely possible. I don’t know what’s shorter than that. I mean, I know there’s something shorter than that, but I know for sure in the Telegraf plugin you can go down to the nanosecond scale. We have a lot of customers that do that.

Caitlin Croft: 00:37:47.649 What is the best way to monitor Telegraf performance itself? Are there any metric drops? What plugins are currently active, etc.? Can we send Telegraf’s performance metrics directly into InfluxDB? So I know there is the InfluxDB template that people can use to monitor Telegraf, but maybe you guys can expand upon that. And I will drop a link to that template in the chat for everyone.

Samantha Wang: 00:38:17.334 Yeah, Caitlin definitely did that big push for templates as well. But there is a Telegraf - I think Stephen actually even wrote it or the engineer that Jess had mentioned, Steven Soroka. There is a Telegraf - there’s a Telegraf template out there that you can easily upload and get those metrics that you mentioned. But the plugin itself, I believe, is the internal input plugin that will give you these Telegraf internal metrics of everything that you have mentioned, making sure that there’s metric drops, what’s running. And you can send these performance metrics directly into InfluxDB. And yeah, I definitely - try out, if you want to get into InfluxDB, into the UI, there’s a new quick and easier way of getting it InfluxDB OSS 2 and Cloud 2 with the templates if you want to - if you want to send those metrics to InfluxDB.

Caitlin Croft: 00:39:27.183 Do you mind expanding on what InfluxDB templates are? I know a lot of people - we’ve had a lot. A lot of them are made by the community, but just thought maybe it might be good to highlight what they are and how to use them.

Samantha Wang: 00:39:40.918 Yeah, so InfluxDB templates are something that we launched earlier this year. I think it came about that there’s a lot of just these really common use cases out there that people - for example, Dimitri, your example wanting to monitor your Telegraf internal metrics. There’s a lot of people who are out there who have these same scenarios that they want to - that they want to get a dashboard and they’re monitoring - just their entire monitoring environment up and running easily and quickly. So this is something that we call InfluxDB template that includes the Telegraf configuration, the dashboards. Often if there’s querying or analysis that are involved in the visualization or be included with that. So this whole package service of these common monitoring environments. So SNMP, Docker - we even have a lot of fun ones like Counter-Strike. Is that the game? Minecraft? I don’t play these games. I know we have a Minecraft Telegraf plugin and I think we have a - no, Fortnite. Sorry. God, this is embarrassing. Fortnite’s the game. I don’t think we have a Minecraft one yet, so feel free to create one. But yeah, so even fun ones like that, we have templates for. But definitely a lot of just these common popular ones, too. If you want to monitor your cloud, your CloudWatch metrics, we have things for that. So yeah, it’s an ever-growing big community push for them as well. But yeah, just a quick and easy way of getting your whole monitoring setup done quickly into InfluxDB.

Caitlin Croft: 00:41:39.205 Awesome. Thank you. Where’s the best place to find documentations on the options in the config file that I can enable?

Jessica Ingrassellino: 00:41:56.437 So I’d actually say that what was helpful for me was actually going to the raw file, the config file, and reading it because the raw config file contains all of the options that are available for every single plugin. Like I said, it’s 111 pages if you convert it to PDF, or you can search it through GitHub using your computer and find command. And if you take a look, you can see everything that you can comment out or not comment out, and then that’ll give you good basics. And then, of course, you can look in the code and see more details about each of the configuration options that you have for each of the plugins.

Caitlin Croft: 00:42:47.540 I use some input plugins like Elasticsearch and Postgres. These plugins need a username and password in plain text in the input configuration. How can I secure or hide the password?

Samantha Wang: 00:43:04.650 You can save them as environment variables. I think that may be the solution you’re looking for. Yeah, there’s stuff in our documentation around that too, but just saving your environment variables pretty normal. Yeah, we have a good section on how to do that. We recommend it for - we show it a lot on your output plugins as well for your InfluxDB stack, so.

Caitlin Croft: 00:43:33.877 Great. What are some good practices in setting up flush intervals, batch sizes, and collection intervals? How do I minimize loss of data?

Samantha Wang: 00:43:48.500 I guess, I don’t know, Jess, if you have anything to add. It varies.

Jessica Ingrassellino: 00:43:52.059 It depends, yeah.

Samantha Wang: 00:43:53.352 Depending on what you’re collecting.

Jessica Ingrassellino: 00:43:54.971 We’ve had - exactly. We’ve had some customers in the past talk to us about how they experimented to figure out what works best for them with regard to those settings, and it really does depend on your data volume, on your data needs, what kinds of processing you need to do. So it’s one of the things that you will want to experiment with before and find out what is working for your particular use case.

Caitlin Croft: 00:44:31.162 Perfect. How do I round timestamps to fixed intervals, such as minutes? I’m assuming this would be Flux with downsampling.

Samantha Wang: 00:44:42.880 I don’t know if you’re - I’m not exactly sure if - so I know if you’re collecting, if you’re intervals set at the - I guess, so I know - hold on. I’m trying to - yeah, I guess that that’s what - I was trying to think of if you’re collecting at the hour level, it would collect right at the hour level. But yeah, if you wanted to do some downsampling, that’s stuff you can do with Flux. But if you also want to do it with Telegraf, that’s stuff you can do with the aggregators as well. So it depends if you want to keep your data or get rid of that rounded data.

Caitlin Croft: 00:45:33.053 Does the Telegraf allow feedback loops in the processing of data? For example, one may want to detect some conditions dynamically, and at that point, change filtering to collect more detailed data from the source.

Jessica Ingrassellino: 00:45:55.384 I understand the question, but I’m actually not positive of the answer to that question. So that’s something that, if you - it says ‘anonymous attendee’, but if you can drop contact info to Caitlin, email us, let us know. We will get back to you about that.

Caitlin Croft: 00:46:13.739 Perfect. The sample videos and tutorials on our website provide information on how to develop the single binary type of extensions. Where can I find information about the Execd option of extending plugins?

Samantha Wang: 00:46:35.650 I think the link that I had mentioned at the top of - at the end of the presentation. I think that’s what you’re addressing. Just more information about the Execd stuff will all be in our GitHub documentation page. If you go in there, you should be able to find information. If you still can’t find it, I think, once again, find us in Slack and ask us about it. But yeah, I don’t - but that actually might be stuff we need to get more on the marketing side as well, so. Thanks for bringing that up.

Caitlin Croft: 00:47:14.611 And also the links that Samantha and Jess had the end of their slides, I will make sure that they are available on the webinar registration page, which is where you will also go to find the recording. So it’ll all be available there as well. Let’s see, what is the difference between using Telegraf and a simple script to send data to InfluxDB? Our team doesn’t want to use Telegraf on our IoT devices and we think a script will be enough.

Jessica Ingrassellino: 00:47:57.448 I mean, I guess the one thing that I would say is that if you’ve got a simple script and you really don’t have a lot of things that you want to do with your data, then you can definitely do that. What I would say is that if you want to - Telegraf offers a lot of options in terms of processing, aggregating, chaining processors and aggregators. So if you want to do some more complex interactions or actions on your data, if you want to dual write data to multiple output sources, then I would consider a Telegraf because it allows you to do all that without having to write all of that yourself. So I think that it offers a lot of extensibility and flexibility for acting on your data.

Caitlin Croft: 00:48:52.224 Perfect. Can we run multiple instances of Telegraf when all of them are writing into the same single InfluxDB instance?

Samantha Wang: 00:49:05.043 Yes. The output that that same - you would just use that same InfluxDB output plugin in all your different Telegraf agents. I think even with our large customers, that’s a very - we had, I think, Discover Financial at InfluxDays talk about how they have 10,000 Telegrafs running.

Caitlin Croft: 00:49:38.689 Just another reminder, I know a couple of people have asked, this session is being recorded and the recording and the slides will be made available sometime tonight or tomorrow. Where’s the best place to find recommendations and best practices for defining a hardware configuration that supports an on-premises OSS deployment of a centralized InfluxDB instance that supports high hundreds to low thousands of posts generating hundreds of thousands of metrics?

Samantha Wang: 00:50:21.001 I guess, if you’re looking for recommendations, I mean, definitely if you have something really specific. Yeah, feel free to drop into that Slack that we had mentioned. There’s a lot of - I think there’s a lot of talks that we have as well from our sales engineers that might address certain best practices that you’re talking about, too. And I think even the internet talks a lot about how you can use - you can use Telegraf - you can use a bunch of Telegrafs to read into [Kafka?] [inaudible] to read into another Telegraf. So if that doesn’t fulfill your use case, those talks, feel free to just reach out to us and talk [inaudible] you’re specifically working on.

Caitlin Croft: 00:51:13.063 What are the best practices for monitoring Telegraf processes to verify if it’s crashed or become a zombie process, up and running but not sending data on the client side? When it occurs it requires a Telegraf restart.

Samantha Wang: 00:51:41.738 I guess I’d be curious if this ties into the first one about monitor your own Telegraf, using that Telegraf internal plugin. But yeah, there’s a lot of - even on the Influx side, talking about the learning on the InfluxDB side, doing dead man checks to see if your data is not running and needs to be troubleshooted.

Caitlin Croft: 00:52:20.290 Can you please provide some links regarding aggregators and processor plugin usage? Official documentation explains it very briefly without many examples.

Jessica Ingrassellino: 00:52:34.626 Yeah, we can pull together some links to send out with the slides.

Caitlin Croft: 00:52:38.487 Okay, perfect. Yeah, I’ll make sure to get those out to everyone on this webinar. Let’s see. So I think this is a follow-up question to a previous one. Can we do the processing on the server side? What is the advantage of doing it on the actual devices?

Samantha Wang: 00:53:00.815 I think one of the big benefits of using Telegraf is collecting data as close to the edge as possible. So just for any latency purposes, we like for people to collect - I mean, I think a lot of our - a lot of our users often just find it the most beneficial for your collecting - trying to collect that data, the most [ideal?] time to collect it, to collect it on the device. But yeah, Telegraf can be downloaded and installed. It varies by whatever your architecture looks like and what you want.

Jessica Ingrassellino: 00:53:52.996 Yeah, and another point is - we have some customer use cases where they are not necessarily able to connect to the cloud all the time, so if they want to have their data collecting offshore, so on a ship or on an oil rig, and they don’t have reliable internet connection, that’s one instance depending on the availability of server or cloud instance where running it at the edge on the device is sometimes the only option for our customers. So that way that they can actually have their data completely collected and not worry about the availability of the internet.

Caitlin Croft: 00:54:43.282 How does the Telegraf agent of agents work?

Samantha Wang: 00:54:53.551 I guess I don’t know exactly what - I guess we talked a little bit about the Telegraf agent itself and the Telegraf agent, it has to be used along with the input plugins and output plugins. But a lot of the times, you don’t necessarily need to do anything with the agent if that’s what you’re asking about. The agent itself doesn’t require much configurations out of the box. But if you want something more specific for your use case, you can manipulate agent settings in there.

Caitlin Croft: 00:55:34.015 How do I make a lightweight version of Telegraf Exec IoT applications where space is constrained? The default Exec seems to be around 100 megabytes in size.

Samantha Wang: 00:55:53.320 I don’t know. I guess I’m wondering if you’re saying - I know Telegraf itself, like the Telegraf [inaudible] is, I think, a little over 100 megabits in size, but you can do some custom builds, which I believe is on our GitHub docs as well, where you don’t need to import all the plugins. It’s actually something that we get mentioned a few times about, about just, “I only want -“ So say, for example, you only want the Exec plugin to run an IoT program that you’re running. You can download a Telegraf binary that just includes that plugin. That should reduce the size a little bit. But yeah, I think - and that information should be somewhere on our - it should be in our GitHub. I think in the main GitHub [inaudible].

Caitlin Croft: 00:57:02.040 Once we install Telegraf, we will edit the Telegraf config file to enable InfluxDB and input plugin. Once we write a custom input plugin and generate the plugin, do we need to inject into the Telegraf config stating that a new custom plugin has been created?

Samantha Wang: 00:57:28.509 So the way that I’m understanding the question is, if once you have it actually, the plugin, included in the Telegraf, either binary or in the Exec, then, yes, you do need to add it to all the files so that way when you call it with the configuration, you can actually use it and do all your configuration settings.

Caitlin Croft: 00:58:02.447 What is the maximum file size in InfluxDB?

Samantha Wang: 00:58:14.687 I’m not exactly sure what, I guess, this question is asking. I guess maybe file size that you can import, which there is no limit of a file size that you can import. But if you want to expand on that question more, maybe that’d be something [inaudible], or reach out to us, we can chat more about it.

Caitlin Croft: 00:58:41.082 Okay I am going to - [inaudible], I am going to unmute you so you can talk if you want to expand upon your question. While we wait for that, looks like there’s a few more questions here.

Attendee: 00:59:04.872 Hi.

Caitlin Croft: 00:59:06.634 Hello.

Attendee: 00:59:08.214 Can you hear me?

Caitlin Croft: 00:59:09.892 Yes.

Attendee: 00:59:12.224 Yeah, the question is, how much data can be [supported?] in InfluxDB? So what would be the size? Maximum size?

Samantha Wang: 00:59:20.482 Yeah, it’s infinite. I guess it depends where you’re storing with the OSS version. It’s whatever is in your InfluxDB OSS, The cloud version, as well as pay as you go. So there’s more information on our website about the [inaudible] you go to for storage base. But there’s no really limitations that we have for InfluxDB.

Attendee: 01:00:03.377 Okay.

Caitlin Croft: 01:00:07.191 Does that answer your question?

Attendee: 01:00:07.746 Okay?

Samantha Wang: 01:00:08.855 Yeah, I guess, but we but we also do suggest downsampling in the past that we do with time series data, specifically. With historic data, you’ll probably end up wanting to downsample it, so the size of your database doesn’t continuously grow because with time series data, your data from three years ago isn’t as - you don’t need, necessarily, maybe that nanosecond granularity as you would currently. So there’s definitely ways to optimize it as well if you want to - if you want to minimize it.

Attendee: 01:00:46.030 Okay. Thank you for answering. [inaudible].

Caitlin Croft: 01:00:50.373 Thank you. What functionality do we have for integrating? So read and write with systems like historian systems like OSS, Pi, Wonderware, and GE Proficy?

Samantha Wang: 01:01:07.703 I think it varies on if those plugins are supported by our plugins. I know a lot of - I know a lot of OSS, Pi, a lot of customers out there are trying to transition off that. So yeah, it just depends on your use case and if we have a plugin to support that. If we don’t, then one could be written itself, too.

Caitlin Croft: 01:01:41.206 Great. Let’s see. All right, I think that it looks like we’ve covered all of the questions that were posted in the Q&A as well as chat. Thank you, everyone, for joining today’s webinar. Thank you to Jessica and Samantha for sharing your knowledge and talking about Influx - sorry, about Telegraf and the ecosystem for all the plugins. I believe you had some slides that showed all of the different plugins. Did you want to -

Samantha Wang: 01:02:15.036 Yeah, I guess I was going to - yeah, going to run through them before the questions, but then I was too focused on the questions, so. Yeah, here’s just some, going through - I can even go back. But yeah, all our alpha plugins, aggregators, processors, all this information is on our official docs. A lot of them, I think, on the marketing web in our GitHub as well, so.

Caitlin Croft: 01:02:52.500 Yeah, perfect. Thank you, Samantha. And once again, these slides will be made available. So I know there were a ton of plugins that were just quickly shown on screen, so if you were trying to figure out what they were all saying, you of course can go back and review them later. And I’ll

make sure that - I’ll work with Samantha and Jessica today to get some of those links out to you all, because I know there was lots of conversation in chat about some of the other documentation and content. So once again, thank you everyone for joining today’s webinar. Hope to see you next week at the InfluxDB IOx Tech Talks. Hope to see you again on one of these webinars. And thank you.

Samantha Wang: 01:03:38.139 Thank you.

Jessica Ingrassellino: 01:03:39.628 Thank you.

[/et_pb_toggle]