Building an IoT Monitoring App with InfluxDB and LoRa

Here is an unedited transcript of the webinar “Building an IoT Monitoring App with InfluxDB and LoRa”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• Caitlin Croft: Sr. Manager, Customer and Community Marketing, InfluxData
• Jay Clifford: Developer Advocate, InfluxData

Caitlin Croft 00:04 Hello, everyone and welcome to today’s webinar. My name is Caitlin Croft. I’m very excited to see you all here today. Very excited to have Jay, who is part of our DevRel team here talking about how to build an IoT monitoring app using InfluxDB and LoRaWAN. So this session will be about an hour long with time at the end for Q and A. It is being recorded and the recording and the slides will be made available later today. So please feel free to post any questions you may have for Jay in the Q and A, which you can find at the bottom of your Zoom screen. And I just want to remind everyone that we want to make sure that all these virtual events and in person events, once we get back to them, are safe, exciting, happy place for attendees and speakers. So please be respectful to all participants. And without further ado, I’m going to hand things off to Jay.

Jay Clifford 01:02 Fantastic. Thank you ever so much, Caitlin, as always. Welcome, all, to the workshop. So I’ve been really excited to present this workshop. So just for a little bit of context here. Before I actually started this project, my technical experience with LoRaWAN networks and LoRa was a little bit lackluster. I knew about the technology, but I actually never really got involved in using it. So it’s been really awesome to do this project and so I wanted to level straight out the back with you, I’m combining my Influx experience here with my now new Laura experience to present a project to you from start to finish.

Jay Clifford 01:39 So just a little bit about myself first. Let me see if I can jump on. So I’m Jay Clifford. I’m a developer here at InfluxData. In my past life, I was a sales engineer for Industrial IoT Solutions. I have a strong passion in like the autonomous and vision-based projects. I’m driven to make IoT accessible to all, and my belief is industrial IoT success belongs with the domain experts. So we have a lot to cover today, so what I thought I would do is rattle on. So my aim is to get through at least these three sections today. If I have enough time, I will cover these two sections. But I’ve already spoken with Caitlin and worst comes to worst, we would just do a Part 2 on this webinar. So we are going to go from start to finish, from setting up your development environment, provisioning your device, all the way through to putting a front end application on your InfluxDB instant. It’s going to be quite a lot, but we will try our best to get through it. Anything that you want more information or more of a deep dive on, just let us know in the comments and we can always do that as blog posts, more material content, webinars or anything throughout. So part one will be environmental setups. We’ll be looking at setting up VS code with the device itself the network. Part two will be connecting The Things Network to InfluxDB. So that will be using Telegraf and setting up InfluxDB. Part three will be a custom IoT application. So I’ll be showing you how to create your own custom IoT application and query and visualize data from InfluxDB. Part four will be showing data transformation capabilities in InfluxDB. And last of all, showing how to do automated alerting in InfluxDB, and potentially tying that back to your custom application.

Jay Clifford 03:30 So let me quickly jump on. But before I get to that, what I want to do is give you a bit of a background of what we’re trying to do here. So my project in this was to develop an application that was going to monitor the health of my plant. So this was originally conceived by one of our dev managers called Rick Spencer and it’s called Plant Buddy. And essentially what we’re doing is we’re monitoring taking a bunch of sensors, we’re streaming that data from our microcontroller and then we’re putting that onto a Laura Wan based network. Now what I want to do is quickly define in my opinion what the LoRaWAN network is and sort of give you that overview. So you’ve got to split it into two. First of all, you’ve got what we consider is LoRa. So LoRa is the wireless modulation technique. And you can imagine this as it almost is a bit like dolphins or bats. It’s a bit like the sonar signal. It’s a long range wireless protocol which runs over radio, and essentially what this does is it allows you to send and transmit data over long distances vastly further than like a standard wireless network. And so we can actually do this on devices, extremely low power as well. So what a lot of LoRa devices are capable of is running off external small batteries. So you can have these in quite remote locations. The cool thing as well is the actual bandwidth for the frequency of these LoRa devices run with are on basically protected networks where only LoRa based frequencies can run on these networks, which means interference from other signals doesn’t - isn’t quite regularly a problem.

Jay Clifford 05:11 So the next layer up is what we call a LoRaWAN. So you can imagine this like the Mac layer protocol, basically like a standard wireless network, but the under backbone is LoRa. And this is what a standard sort of LoRaWAN network would look like. Stolen with pride from The Things Network team, just to show you exactly what this looks like. So what this comprises of is a series of nodes. So our nodes here could be considered our microcontrollers that have LoRaWAN connectivity functionality. We then have our - sorry, I’m just making sure I’ve got the right notes on this going through. We then have our network server, which is here. But before we have our network server, we have our relays and these basically bounce the information coming from our nodes directly to the network server. So you either have them as relays or gateways. They basically just route the information directly to our network server. And the network server itself takes all the information coming from our end devices and translates that data coming from the LoRa nodes. So basically takes it from serial form or sorry, takes it from any encoded form and transforms that into what you need it to be and manages all of those devices all around the network. And then what network servers usually have as well is an application server. So the application server is a selection of services which allows you to do really cool features such as offload your data to other raw protocols. Maybe that be HTTP, MQTT to other cloud services, and also potentially allows you to manage devices as well, which we’ll see in a moment.

Jay Clifford 06:47 So what I want to say to you straight here is there’s - we keep throwing around the Things Network and The Things Stack. So The Things Network is essentially a community-driven open source LoRaWAN network that anyone connect can connect to for free and stream their data. And what the LoRaWAN collective do - sorry, what The Things Network collective do is they actually run this open source community edition of called The Things Stack. And now The Things Stack runs on their network server and that basically allows you to manage, maintain your devices and stream data from those devices to the network server to allow you to stream that to your other applications. And from there, of course, that’s where we come in as Influx Data. We’ll be taking the information from the application server and the network server, streaming that to InfluxDB cloud or InfluxDB edge using Telegraf. So without further ado, let’s get cracking.

Jay Clifford 07:48 Let’s work through. So first we’re going to talk about the environment setup. So here is our LoRaWAN microcontroller and sensors. So all I’m using here is an Arduino MKR 1310. So this is a really cheap device. You can get it for about 30 to 40 dollars. And essentially all you can do is you can basically put it on a breadboard and wire up a bunch of lightweight sensors. It comes with an integrated LoRa chip, which allows you to connect directly to The Things Network, and maybe to a LoRa network of your choice if it’s not The Things Network. And all I’ve done is I’ve combined it with a sensor pack, a really cheap sensor pack. So in this case we have the temperature and humidity, a light sensor which is based on a photo resistor, a soil and moisture sensor and a soil and temperature sensor. So I’ve got the GitHub repo for this, which also contains all of the code for the microcontroller as well. So you can actually just use that code directly if you’re buying one of these, and you can basically use that to do the demo yourself. But let me show you what this looks like in terms of code. So if I jump into Arduino code here, all you can see here is essentially I am wiring up -

Caitlin Croft 09:09 Jay?

Jay Clifford 09:09 - a bunch of - yeah, sorry. Hello?

Caitlin Croft 09:11 Is it possible to zoom in a little bit?

Jay Clifford 09:14 Oh, yeah. Let me see if I can zoom in for you. What’s too small?

Caitlin Croft 09:19 Just the text. Is it possible to - usually you can zoom in on terminal windows.

Jay Clifford 09:28 There we go. I’ve gone a little bit far now.

Caitlin Croft 09:33 Yeah, that looks better. I hope everyone -

Jay Clifford 09:37 [crosstalk].

Caitlin Croft 09:37 It looks good, yep.

Jay Clifford 09:40 Awesome. But yeah. So essentially within this Arduino code, all I’m doing is I’m pulling the readings of the sensor data and then I’m connecting directly to the Things network. So that’s all within this code base which you can see here. But we can cover that in depth if you want to another time. So moving on, let’s discuss setting up your VS code environment for dealing with Flux. So if you didn’t already know is we actually now, as Influx Data, have a Flux plugin for VS code. And this is an awesome tool which allows you to add Flux language support directly to VS code, and also provides administration features for your Influx plugins. So it means you don’t actually have to jump into Influx anymore in order to work with your InfluxDB instance, which is awesome because a lot of the time I found it so disjointed to jump from VS code and then jump out of VS code to do stuff within Flux and then jump back into VS code. So let’s have a quick look at installing that.

Jay Clifford 10:40 So all you need to do is jump into extensions. You need to search Flux with Influx here and then you would click install. So I already have the instance installed here. From there, if you jump back to your explorer and if I jump here, and then if I jump into our section. As you can see in our explorer we have a new panel called InfluxDB. From this InfluxDB panel all I would do is click plus, and this allows you to define a new connection so you can define an arbitrary name, then you would add your host, your token. I recommend using an all access token since it allows you to use all the administration features, and then the organization name that you wish to connect to and then you can test and save. So if I quickly show you this, if I edit that connection, you can see I’m connecting to my cloud instance here, my token which I won’t show you, and then my organization name that I’d be connecting to. And once you’re connected, you’ll be able to see something that looks like this. So you’ll be able to see your buckets, your scripts, and your tasks, which allows you to navigate through these all directly from within your VS code plugin itself. So we’re going to park this for now. We’re going to get this later when we discuss the custom application. But I just wanted to set up VS code plugin from the start.

Jay Clifford 11:59 So the next thing I wanted to discuss was connecting our Arduino device to The Things Network, or yeah, The Things Network or The Things Stack. So let’s jump over to that now. Hey, Caitlin, is that okay? Is this a good size or do you want me to zoom in on here too?

Caitlin Croft 12:15 That looks good.

Jay Clifford 12:16 That’s okay. Awesome. So essentially this is the control panel for the Things Stack community edition, which is the Things network. We could do an entire webinar just on this, so we won’t. But what I want to do is get you guys up and started quickly. So what you’re going to do is you’re going to jump into go to applications. From here, you would create a new application. I’ve already created an application here. It’s really simple just to find an ID and name and description. Not much to do there. Once you jump into your application, you come to your application control panel and this is where it gets interesting, where you can actually start adding your devices, formatting your data. And then you can also look at integrations to other platforms, which we’ll be doing. So what I wanted to show you is I already have two devices set up, one here in Edinburgh and then another in Newcastle. But to define a new device, what you do is you click add end device. Luckily, what The Things Stack have done is they have a library of devices, but you can define them manually as well. But essentially, since I’m using Arduino device, which is quite a standard device, it’s within the library. So I click Arduino, the model, the hardware version and the software version. Top tip here. This got me. There is a new version released from Arduino. I think it’s 1.3.0 and this is not currently a firmware version on The Things Stack yet. But don’t panic about it. It works just fine enough leaving - so you can go up to 1.3.0 as your firmware version on your microcontroller and it will still work with The Things Stack, so don’t worry about that. And then you would be defining your region. So your region is defined based upon where you are in the world. So there are currently three regions. There’s the US, Europe and Australia. And I believe there’s other defining other subset regions, but that’s not in my area of expertise to give you advice on that.

Jay Clifford 14:18 So once you do that, you can finish the registration of your device. So you’d be selecting the frequency plan. I always go with the recommended SF9. You can create an app ID, so you generate your own app ID here. You could fill it with zeroes, but that’s not exactly good practice. So just generate your own app UI here. This will also go within your Arduino code as well, which I’ve outlined within the code that you must fill in. You can generate a dev UI, which is - sorry, you can’t generate a dev UI, that’s a lie. The dev UI comes from the manufacturer itself. So when you turn your Arduino on and you look at the console, it will give you your dev UI, which is basically normally your LoRa chip that’s on there. They have like a little serial code on them. So that’s what that normally is. And then you would generate your app key. So the app key is just that security code for allowing you to connect to your LoRa instance. Once you’ve done that, you click register and your device should connect. So if I jump over to here and if I click on my Edinburgh device, you can see here that I have my live data coming through. I’m located with my Edinburgh one here and I have all of this good stuff filled out. So my dev UI, my app UI and then I obviously have my security app key as well to make sure that I am securing my connection to my device.

Jay Clifford 15:44 So if I have a quick look at the live data here and I jump in, you’re going to see something quite interesting. So the live data coming from these devices actually comes as a byte stream. And this is because within LoRA networks, payload is king. Like, the smaller the payload the better. Essentially this is due to your time and error and your transmission. So the longer airtime that you have the worse it’s going to be. This is a chance for your actual data to become corrupted, certain bytes not getting there, etc., etc. So what you want to do is you want to be able to condense your payload message into a small bitesize message that you could then format on basically the service side of the Things network. So this is what we’re going to do. So I’ve within the Arduino code, taking all my sensor data, turned it into like a byte array and I’ve sent that along my LoRa network. I’ve received it here on The Things Stack. I’m now going to convert that byte stream in back into a JSON payload. So if I jump over to payload format is here, all I’ve simply done here is used a bit of simple JavaScript. So I pulled in my bytes that I’ve received from my device. I then convert that from the Char code, and then I place that within a JSON payload. So you can see that JSON payload here. And then all I’ve done is basically past that soil moisture, air temperature, humidity, soil temperature, and light. And what you’ll be able to see from the live data is you now should be able to see that payload converted here. So you can see all my values from my sensor here.

Jay Clifford 17:26 So this is how far we’re going to go in terms of connecting up our device here before we start streaming our platforms. So for me, this is the - there’s a lot more that you can do within the Things network itself. This is me at a beginner’s level showing you a beginner’s level overview. So this is me connecting my device and streaming data directly to The Things Stack. So from there we’re going to move on to connecting our platforms. So this is where we’re going to connect The Things Stack community to InfluxDB. And what we’re going to do first is we’re actually going to set up InfluxDB to make sure that’s initialized and ready to accept the data that we want to send to it. So I’ve actually done a lot of the hard work for you. I’ve created two InfluxDB templates called Plant Buddy and The Things Network. So if you don’t know, an InfluxDB template is basically a YAML configuration file which automates the setup of dashboards, buckets, tasks, variables, lots of the assets that you require to run Influx without you needing to define them manually yourself. So we have an amazing community library of these for lots of different applications, not just these two. So yeah, definitely check them out if you are using a different platform as well. But yeah. So let’s go and install these two templates now.

Jay Clifford 18:54 So if I quickly jump over here. So this is our repository. I will send you a link at this end and we have our two templates for Influx here, which you can see here. We have InfluxDB, Plant Buddy, and the Things network. So to install these two templates you can go into buckets - sorry, so you can go into your Influx instance, you can go down to your configuration down here and you can click templates. Jump into templates and you can see I’ve already copied and pasted one here. What you need is the RAW file. So to get the RAW file, all you simply do is if you jumped into the YAML file on the Git repository, click RAW and that gives you your RAW URL, and you would copy and paste that directly into this URL here. And then you would click lookup template. And when you click lookup template it shows you the number of assets that you have ready to go. So I’m not going to install this template directly because I’ve already generated the assets in my cloud instance, so they already exist since I already exported the template for you. But you can see there’s a number of them here. So including buckets, tasks, Telegraf configuration, and dashboards. So I will give you a whistle stop tour of those in a second once we’ve configured Telegraf. So let me quickly jump back over.

Jay Clifford 20:21 So the next thing we need to do is configure Telegraf. So if you don’t know, Telegraf is an open source metrics collector that we as Influx Data manage. It’s completely open source, so everyone can contribute plugins to Telegraf, so we have over 250 to 300 plus plugins. Now some of those input plugins include OPCUA, MQTT, Rest, Modbus like AWS Cloud Watch. There’s loads of specific and also more general plugins. And then we also have output plugins to a variety of different sources as well, whether that be directly to Influx, which we’ll be doing now, maybe to MongoDB or a SQL database. Other enterprise platforms. Telegraf really has a plugin that fits everyone’s needs just about. So essentially what we’re going to do is we’re going to configure our Telegraf config. And the way I’m going to show you this is we’re going to first generate an event, like an environment file, and then we’re going to generate a The Things Network broker password. So let’s work through that together. So if I jump back into our configuration and let me bring up - this is the only thing making this bigger, there we go. Awesome. So let’s quickly work through our Telegraf config.

Jay Clifford 21:42 So if you haven’t seen the Telegraf config before, essentially Telegraf in most cases is no code, and this works with this COMP file to tell Telegraf what plugins to use and how to work with the data. So as you can see here, we have two output plugins which are pointing to different buckets within my cloud instance, one pointing to Plant Buddy and another one pointing to Things network stats. Then we also have two input plugins for the MQTT consumer and this is how we will be connecting to The Things Stack and streaming data to InfluxDB. So you probably could do this within all one MQTT consumer, but just for the purpose of simplicity, I did it as two separate input plugins. So for this first one, all I’m doing is collecting the stats from the LoRa network itself, because I also want to monitor The Things Network as well to make sure that my devices are connecting and are healthy connecting to that network. So since all I’m doing is I’m subscribing to The Things Network server, which I’ll show you where to find this URL. I’m subscribing to the top level topic within that MQTT broker. I’m defining the username and password for that broker, that Things Network broker, and then I’m defining the format. So it’s really important that the payload format is JSON. So we will be pulling that JSON payload, we will be converting that JSON payload into line protocol, and sending that to InfluxDB. But that’s all done automatically through Telegraf.

Jay Clifford 23:15 So as you can see here, I’ve also defined a tag for my devices. So that means I can filter based on my device ID, which is great. So I can keep my devices separate within my data so I can basically drill down and understand how each device is performing with regards to the network. From there, I’m looking for the uplink message and the uplink message metadata, and then I’m excluding some keys that I don’t want. And then lastly I’m doing equivalently the same for the MQTT consumer for Plant Buddy. So you can see that here, basically the same type of stuff. The only information that I want is that decoded payload which we saw earlier. I don’t care about any of the other meta information, just the payload to go within the Plant Buddy bucket. So you can see that here. So what I’m going to do now is I’m going to show you what I meant by the environment file. So the reason why I suggest using the environment file is for security. So what you could do is you could define your username and password directly within the comp itself, but that’s just not secure. So the next best thing you could do is set up environment variables to define those. So all I’ve done here is I’ve set up an environment file with export. So I’ve exported my host, my organization, where my token should go for Influx and my Things username and Things API key. And so all I would do is simply source this file before to something like this. So source. Find out where this - I’ve put this into microcontroller and the Things network, and then if I jump into dot [inaudible], yep - okay, that just because of what I’m doing here. Sorry that’s my bad because I filled these in, but essentially you would source this file and then you would run your Telegraf config and then that would basically fill those environment variable spots in your Telegraf config. So let’s find out where you need to pull these from.

Jay Clifford 25:26 So if I quickly jump over to - back to our The Things Stack here. If I jump into integrations and go to MQTT, we can actually directly -The Things Stack actually has an MQTT broker directly installed within - on the server itself or within The Things Stack that you can utilize. So here’s our public address that I want to know. You also have a TLS address if you want to use that instead to be more secure. And then you also have the username and password. So I would want the username out of there to go into my environment, and then the password I can just generate a new API key. So if I generate a new API key, copy that, and then I would paste that directly into my Things API key here. I’m not going to do that now just because I’d be exposing a password on a webinar. So once we’ve done that, all you need to do is to test that connection. Just simply run Telegraf dash, dash debug dash, dash config and start up your configuration. So note just on this, you will only ever receive data from The Things Stack via Telegraf when your device pinged homes. So when your device is pinging data to The Things Stack that’s when you’ll receive data from their broker, it’s not on an interval. So that’s how you would know. Just to make sure, so safety device is only sending data every minute. You would only receive a message every minute in Telegraf. So I’ve basically got a home server at home where I run my Telegraf configuration from, so that’s all preset up. So I’m already streaming data directly to my Influx instance, so I can quickly show you that now. So let me jump back into Influx itself.

Jay Clifford 27:17 If I go into our dashboards, I can show you our Things network dashboard here. This is the one I’ve created, and if I query this, you can see some of the information that I’m gleaning from the metadata from our The Things Stack. So the locations of where the gateways where my devices are pinging to. So obviously there’s one in Edinburgh and there’s one in Newcastle. The bandwidth from which my devices are talking on, the consumed airtime, which we said is quite important. So clearly my Edinburgh device has much better connection to the gateways than the device in Newcastle. And that makes sense because of where I place my device in Newcastle. I have meta information on my actual devices themselves, including meta information on my application that I’m hosting on The Things Stack. And then we also have some really interesting information on channel RSSI and channel SNR as well, which is fantastic. And then we also have correlations between those, and this allows you to really monitor granularly how well your devices are performing on those connections. And I’m going to have a shout out to one of our community members. So this bottom visualization that you see here comes from one of our community members using The Things Stack already, and he’s a great guy in the UK called Mike, and he basically found a way of showing the connectivity cycles between your sensors. So it’s amazing to see what the community has to offer here already and how it gives back to Influx every day show different ways you can monitor your networks.

Jay Clifford 28:58 So the next thing I wanted to show you here, lastly was Plant Buddy itself. So we have a Plant Buddy dashboard which you can see here, which actually shows the data from our sensors. So you can see our monitoring on a per device basis. We defined that tag, if you remember rightly, which shows you the soil moisture, the histogram, the air temperatures, the air humidity, soil temperature, light, and then correlations between those both as well. So that’s great. That’s great as a visualization to make sure that we actually have data coming into Influx. But why don’t we take it a step further and create our own custom front facing application that we would give to users rather than them jumping into Influx to use any of the UI there. So let me jump back over here.

Jay Clifford 29:49 So for the custom IoT application, I decided to use Plotly, Plask, and Flux and the InfluxDB client UI. So I decided to use Python because it’s what I’m most comfortable in and it allowed me to use a variety of different libraries to show you there’s a lot possible using Influx as your back end. So within this, what I want to show you is how to create a Flux file, how to parameterize a Flux file, and then send that data and transform it into a data frame before we visualize it with Plotly. So if I jump back over to our VS code here, let’s start working through this together. So let me jump into our Flux files. So, if you have a look here, this is a static Flux query which you’re probably all familiar with from the Data Explorer if you use the UI itself. So essentially I’m querying from bucket Plant Buddy from the last hour and I’m filtering under the measurement sensor data under that device UID, which is my Edinburgh device. And I want to know the data from the light sensor. So the cool thing that we have about that VS code Flux plugin that we installed is we can actually run the query directly from VS code, which is awesome. So let’s have a look at this. So I actually can see now a table of that data which is returned. So you can basically query directly and see that raw data without actually having to jump directly into InfluxDB. So I have the values here, the fields, and the times in which that they’ve come in over the last hour. So this is great, but this is a static query. So imagine if I wanted to pull data about our moisture sensor in where I would have to build out a Flux file for every single sensor, which is just not efficient. So I really wouldn’t do it that way. So what I could do is use parameterization and parameterization allows me to make my Flux files a lot more dynamic. And I basically just done this through basically simple formatting and injection in Python.

Jay Clifford 31:58 So if we have a look at our graphed Flux file here, you can see that I’ve just parameterized the parts that I want to make dynamic. So I want to parameterize bucket, the device ID, and the field, which if I then jump into my actual Python code, if I jump into my Influx helper here, you can see that I’m actually calling here. So I’ve got a wrapper function called query data, and I’m expecting bucket measurement and field. And then essentially what I’m doing is calling that Flux file and I’m injecting those bucket field and measurement variables into those parameters. And from there, I can then send that query directly to Influx and query for that information. And now this is a super awesome feature that we’ve added to our Influx client library for Python is we can pull that data back and we can convert it directly into a Pandas data frame. So at Influx, we understand that there is already like data formats and tools out there that people are using every day that are industry standards. So Pandas data frame has been here from people from start to finish with Python, especially for data scientists, people working with large quantities of data and analytics. So we have an automated process of converting whatever Influx data comes in and turning it directly into that data frame.

Jay Clifford 33:32 So I query, and I save the resulting data frame from that query in result and I return that. So where do I actually use this query data function? Well, if I jump into our application here, you can actually see I’m using Plotly Express to visualize my data. So from the code here, all you can see is I’m defining a Plotly Express instance, and I’m defining a line graph. And then I’m basically taking that data frame and taking the time, the value, and the field of title and generating those visualizations. So let’s see this in action. Let me quickly jump here and run. This is where demo woes happen and everything breaks on me. But I’m hoping to the demo gods it will not break. So I’ll quickly jump over here. Fantastic. Okay. And to anyone who is a front end expert out there, please do not shoot me for my horrific bootstrapping here. It was just to show you a demo. I am no whiz when it comes to front end design. So what you can see presented to you here is Plant Buddy. So this is my front end IoT monitoring application. And as you can see, you can jump between tabs and look at the different data that we’ve queried [inaudible] DB as our back end. So you can see soil moisture, I can jump between tabs and look at soil temperature and air temperature. And that’s all done through parameterization. We’re calling the same functions in that same Flux query over and over just with dynamic parameterization there. And then I can click and generate new graphs, which is fantastic.

Jay Clifford 35:19 So this is a really basic idea. Obviously, I expect your front end IoT application to be vastly superior and vastly more complex than this, but I just wanted to give you a brief idea of what a simple IoT front end application would look like. So I think we’re actually doing reasonably okay on time, are we, Caitlin? Have we got like ten more minutes or so?

Caitlin Croft 35:44 Yep, go for it.

Jay Clifford 35:45 Sweet. Okay, so I want to introduce you guys to some of the more interesting InfluxDB features that sort of bring this project to the next level. So what we’re going to introduce is we’re going to introduce data transformation, and data transformation can be done using a task within Influx. You can imagine a task like a CRON job and this basically allows us to perform a variety of manipulations on our data for whatever given reason. So within this scenario, we basically said, look, we’re collecting data every minute from our sensors, which is awesome. We now know how our plant’s feeling every minute of the day. But do we really need to know how our plant’s feeling every minute of the day? Some people would say yes. I would say I don’t have enough time for that. So really, all I care about is knowing that my plant is okay maybe every hour of the day and maybe I only want to know how it’s been doing in ten minute intervals of that hour. So essentially what I’ve done is I’ve defined a downsampling task which will allow me to which will run every hour and take the last hour of that data and then iterate over 10 minute intervals and take the last value of those 10 minute intervals. In this case, it would have been an average. So I can take an average over every 10 minutes. But for this use case, I’m just going to keep it simple and do the last value over those 10-minute iterations. So what I’m going to do is I’m going to quickly do this in front of you now. And the cool thing is we can do this with the VS code plugin, so we actually don’t have to jump into Influx again to be able to do this. So if I jump in, I have actually provided the task for you as part of the template. So if I quickly jump into our Plant Buddy downsampling task, I click edit task here, you should be able to see this task that I’ve generated for you. And so just to run through it again, this runs on every hour. I pull in the last data from the last successful task. So this is an awesome wee little feature. So instead of running based on an arbitrary pull, I can check where my last successful run of this task was and only pull the newest data from that last successful run.

Jay Clifford 38:13 I then filter for the measurement sensor data. I then aggregate that data into 10-minute intervals and take only the last value of those 10-minute intervals. And I then basically push that data into a new bucket called downsampled. So this means that I can set retention policies on my data, which is more efficient for storage. So maybe I might only set a one day retention policy on my raw data. I can set a 30-day retention policy on my downsample, so that would improve how much data I’m storing. So the cool thing here is I can actually run this again directly from VS code, and you can now see the time is I’m only taking the 10-minute intervals like I expected to, so that last value at every 10-minute interval, which you can see here. Awesome stuff. So that’s a downsampling task. That gives you an overview of what you can do from downsampling. What about if I wanted to add alerting to my IoT application [inaudible].

Jay Clifford 39:13 So you can actually do alerting from directly from the InfluxDB UI. But I thought I would show you what an alert looks like as Flux. So when you generate an alert from the UI, what you’re essentially doing is you’re generating a glorified task. So the task is built with basically the alerting library and the check library built in, and allows you to send that data, send those alerts to where you want to go. So you can actually code the alert yourself, which adds a lot more customization and a lot more flexibility to how you want to prepare and how you want to send that data. So let’s have a look at that now. Let me quickly check here. So let’s have a look at the create notification. So if I jump into our task here again, I have our Plant Buddy moisture test here. So if I click edit task again, what you can see here is I’ve created a check and notification. So the separation here is a check is what’s checking your data, see if it goes above a certain threshold or maybe it’s like a deadman’s check and you haven’t got data coming in for a certain reason, and then you also have a notification. So where would you like your message to end up? Like where would you like to send the alert to? So if we look at the data here, essentially all I’m doing is I’m querying my sensor data for soil moisture under the device ID that I have here in Edinburgh, and I’m checking for a trigger. So if my soil moisture is smaller than 50%, then I want to know about it because my plant needs to be fed. The messageFn here allows me to define a template for the message I would like to send to my end point. So I’m basically saying my plant’s getting thirsty. My moisture level is at this percentage. I then basically take all of these building blocks which make up our checking notification and then I put them in and I build them together within this task data here.

Jay Clifford 41:24 So schema field as column you can imagine like a pivot, we’ll be moving to pivot at some point. But the schema field [inaudible] column is just something that’s translated over from the early days of Influx V2 and it’s just constant user task. And the monitor libraries here allows us to create that check and that notification for our data. So I’m just using a simple Slack endpoint here. But I’ll tell you what you can use later, which would be more interesting. So the cool thing again is you can actually run this query again, which is even more interesting because it gives you the end result and you can see this, my checks and my moisture test the level is okay, here’s my message. So this would be what my message would have been if there had been a problem. So your plant is getting thirsty. Moisture level is at 66%, and you can see that the message should not have been sent. I wanted to check why that got sent, but it should have been when the threshold is hit. Sorry, the threshold hadn’t been hit so the data would not have been sent essentially how it should have worked. That’s probably something in my code that I’ve done wrong there, probably. You should be able to see a Slack notification from me doing that. If I quickly jump into Slack, there we go. If I jump into here, you can see my Slack notification here. So getting 30 moisture levels at 66%.

Jay Clifford 42:56 So the cool thing that you could do instead is you could replace a Slack notification with a HTTP endpoint, and I’ve done some of the groundwork for you. So this is your homework to do it yourself is you could actually expose your application onto the internet. Exposure is probably a bad word, but then connect your InfluxDB instance to that HTTP endpoint and send the notification column Flux to your user application to say to them hey, here’s a notification. Your plant is thirsty, you need to water it. And you can find that in the main application here. If I quickly jump back into that, and I cross off this, and I jump down. You can see our notify route here. So essentially what I would do is I would use our HTTP Flux endpoint and I would post that message directly to this endpoint, which would create a notification within our IoT application to say alert the user that our plant needs watering. And so you can see it’s all really basic at the moment, but these are the building blocks to building a fully-fledged IoT application from the ground up using all open source applications and platforms. So apart from buying the sensors and the little microcontroller, this cost me no money. You can use InfluxDB for cloud for free to a certain extent, and you can use the Things network for free. So you can build a monitoring application for your house, for a proof of concept for an industrial customer that you might be using all for free. All for using community driven projects. Which is why we love to partner with people such as The Things Network and The Things Stack that are working on these projects, just like we’re open source with InfluxDB itself.

Jay Clifford 44:54 So last but not least, let me quickly jump across. Next steps. So why don’t you have a go yourself? I always think it’s better to do than to listen to me rabbit on all the time. So I’ve put all of the code on Influx community. You can download the repo there, and you can actually just get up and started with the code that’s on there. It would be great to see some of you get involved, maybe commit some of your own codes, some of your own ideas. I would love to see it. It would be really nice, and I know Rick Spencer who’s worked on this recently, would love to see some contributions to this as well. It would be fantastic. It’s a cool little project and we’re hoping to see lots of little plant buddies spark up all around the world. So yeah, I’m hoping you guys all get involved and join us on the Plant Buddy revolution. And yeah. So I think that is me, Caitlin. So is there any questions?

Caitlin Croft 45:48 Of course there’s questions for you. Is it good practice to write to InfluxDB directly from IoT devices instead of collecting the data somewhere in the cloud first?

Jay Clifford 46:04 So it really depends on your use case, I think. So we obviously have an Arduino library that you can basically send data directly from your Arduino device to InfluxDB. That’s completely possible now, but it’s really about how you want to prepare your data. So if you use a middle ground solution like Telegraf or management solution like the Things network, this gives you a little bit more control over your device and allows you to monitor how well your device is performing, where InfluxDB is not designed for that. We’re there to collect and store your data and analyze your data. We’re not a device management tool. So I think if you’ve got a fleet of devices that we’re looking after, then I highly recommend looking at a device management tool like the Things network or something equivalent for your protocol that makes sense. And then if you need any data manipulation, anything like that, then your best bet is something like Telegraf to do all that automatically for you.

Caitlin Croft 47:08 Can we access to this TTN organization?

Jay Clifford 47:14 Yes. If I jump across - I think I got you there, Caitlin [inaudible]. Can I have access to The Things Network? Is it -

Caitlin Croft 47:22 Yes, yes.

Jay Clifford 47:24 Yeah. So yeah, of course. So this is The Things Stack community edition, so all you need to do is jump to this URL which I will copy and paste into the chat now. Let me do that. There we go. And you can just start - basically create an account and it’s all for free. So you can just sign up and add your own devices, create your own applications. Yep, don’t worry about it. But just the one thing to bear in mind, it’s for the people, by the people. It’s completely open. So if you wanted to be private, if you’ve got privacy implications in place that you need to an extent, then always check what you’re doing there because these are publicly run servers and relays.

Caitlin Croft 48:15 Can you show these graphs without a user login?

Jay Clifford 48:20 Yes. Yeah, yeah. I’m wondering when and where this question came in. On which graphs?

Caitlin Croft 48:32 [Dana?], if you don’t mind letting us know which graph, but I’m assuming just is there a way to share a link up to the graph about someone being logged in?

Jay Clifford 48:43 Yeah. Cool. So I guess I’m going to take this from the Influx way, so maybe these graphs that he’s talking about. So at the moment, these dashboards themselves are private to log in to Influx. But what you can do is you can create an InfluxDB notebook, which is awesome. So notebooks you can imagine a little bit like a Python notebook, which generates the code and the visualizations like this with all of the standard libraries. And what you can do is you can actually share this anonymously through the link, which allows any user that doesn’t - that has the link - you imagine like a Google Drive link, you can share it with someone and they can view the data, but without needing to log into your InfluxDB instance. They have no control over your InfluxDB instance at all. They can just visualize a notebook and they can only visualize the notebook as it appears, so they can’t edit how the data looks.

Caitlin Croft 49:43 Cool. Where is the Telegraf agent running?

Jay Clifford 49:48 Yeah, awesome. So this is always a question. So there’s a multiple places you can run this. So for me I’m running this - the Telegraf agent on my home server at home. So I just have a little Jetson Nano or a little Raspberry Pi sitting at home. And this is the only reason I do this is because I want it on all the time. So it just runs at home, collecting data from the Things Network and sending that to Influx. But if you didn’t want to do that, you could run Telegraf from a cloud instance, maybe like an EC2 instance or an Azure VN. Or you could run it even directly on your laptop. So you can just install it on your laptop, get it running, and just leave it as a background service. It’s really lightweight, so you wouldn’t even notice it in terms of performance unless you’re like running with 1,000 plugins, which I wouldn’t expect you’d be doing with your own PC.

Caitlin Croft 50:37 That’d be a lot. Could Pi Zero or something similar be used or must it be Arduino? Not sure of Pi Zero specs, just asking for information purposes.

Jay Clifford 50:52 About devices, sorry, was it? Was it?

Caitlin Croft 50:54 Yeah. So just curious if they can use Pi Zero or if they have to use Arduino.

Jay Clifford 51:00 Oh, yeah, okay. So I just showed you Arduino because it was easy to use and an easy upstart. But if I jump back into applications and devices, there is a whole host of catalogs of who they’ve chosen. So these are all manufacturers and providers. I don’t know specifically if the provider that they’re looking for is there, but you just select a brand or provider and then you have a variety of models that you can choose from which allows you to automatically provision that device for The Things Stack. Now if the device isn’t there, you can do a manual configuration. So you can just set up the LoRa solution yourself, which basically is relatively the same as setting up an actual device. So if you’re making your own custom LoRa devices or you don’t see it within the list, d

on’t worry about it. You can manually define it.

Caitlin Croft 51:57 Okay. So Donna’s asked me about this video. It is being recorded and will be made available later today or tomorrow. It will be the exact same place that you actually registered for the webinar. So it’s super easy and the slides will be there as well. Is there a way to log the notifications back into InfluxDB for analysis later?

Jay Clifford 52:23 Yeah. So there’s a few ways you can go about this. So if I jump back into Influx itself. So first of all, you have - they’re already technically actually stored, but you just have less control over how they’re stored through this method. So if I jump into tasks and I jump into runs here, you can actually see the results of past tasks being run and notifications from - [inaudible] because I got a string value. If I swap over to raw data. That was success. Sorry, I think I’m in the wrong bucket. Notifications. Here we go. Yeah. So you can see that I already have our check here, which is a Plant Buddy moisture test. And I can actually see that data directly as well as the payload of that message. So I can query directly from the monitoring bucket and so I can analyze the notifications. If you didn’t want to use that, because that’s how we define the monitoring tasks, what you could simply do is by going into task here and I go into our moisture - oh, God, where’s it gone? Moisture test. If I edit the task, what you could do instead of sending it to a Slack notification, or as well as send it to a Slack notification, you could just use that to function from Flux and send that data directly to a new bucket where you can analyze that later within a new bucket.

Caitlin Croft 54:03 So someone is asking for more information about Flux. I will find a link right after this. Actually, I’ll do it right now. So we do have an internal Flux training coming up on - next Tuesday, I believe. So it’s a really great course. Scott Anderson, who is our - he’s part of our docs team.

Jay Clifford 54:31 Legend.

Caitlin Croft 54:31 Yeah, he’s amazing. And he’s definitely one of the experts on Flux. So let me just pull up this link right now and I will throw it in the Zoom chat so you guys can find it and register for it. You’ll get to listen to me again, as I will be facilitating that training as well.

Jay Clifford 54:54 Yeah, I think between Scott and Anais, they are literally the Flux gods. I think they know all, so.

Caitlin Croft 55:01 I think so, yeah. Anais is one of our other DevRels, so you see her name in the forums or in Slack, definitely bug her with your Slack questions. Let’s see, when using the exec-D plugin with Telegraf, you need to keep your script and scriptable in a certain folder structure related to the config files?

Jay Clifford 55:24 No. No, so you basically just - wherever you’re running Telegraf from, you can just - you can imagine it like Telegraf is basically calling your script, like it’s calling it from command line. So you can just either use a relative path or a direct path directly from your script and then just append any commands that you need to execute that script on the end. So if you want to see a use case for that, if you go into the Influx community GitHub again and you go into Jetson stats, I have an example of using the exact plugin to pull out Jetson information there where I’ve just basically called my Python script from a different part of my system to start monitoring that data. So, yeah, don’t worry about where those scripts are located. Put then wherever you like, just make sure you’re using either direct or relative pathing.

Caitlin Croft 56:17 Cool. I know we’re just at the top of the hour. I know people are dropping off. We just have a couple more questions here. Jay, do you have a couple more minutes?

Jay Clifford 56:24 Yeah, of course, of course.

Caitlin Croft 56:26 Cool. How do you configure the LoRa network?

Jay Clifford 56:30 So the LoRa network itself?

Caitlin Croft 56:32 Yeah.

Jay Clifford 56:33 So good question. This is heavy, essentially with regards to the network that I’m using, which is The Things Network and Things Stack. You don’t. Essentially all you do is you set up your device, and as long as there’s a router sort of within range or a gateway within range, since it’s community based, the device will automatically connect to that network and start sending data to the Things Network. That’s how powerful it is, it’s almost - I don’t want to say plug-and-play because that’s an awful way of calling it. But you really just turn your device on. You’ve provisioned your device with the Things Network, it will start sending data directly because as long as there’s a LoRa networking range, you’re cool. You can set up private LoRa networks yourself. But yeah, that’s beyond my expertise by far. We’d have to get you in touch with someone from the Things Stack or network to help you with that.

Caitlin Croft 57:30 Very cool. Well, thank you, Jay. I think this was a fantastic session. Lots of really great questions. Thank you everyone for joining today’s webinar. I know someone was asking about case study applications. If you have a look at the Influx Data website, the customer page, you can definitely find a lot of fantastic industrial IoT use cases. So if you’re interested in learning more about how other people are using it, definitely check it out. One of my favorites is there’s this German company, print manufacturer, and they are using InfluxDB to monitor their manufacturing plant to understand the correlation between how much paper they’re using and how much ink they are using. So they’re looking at these large print manufacturing machines and understanding that natural degradation of the machine actually adds little particles to the ink, which is just a normal process. But prior to using InfluxDB, they didn’t have insights into that. And as a result of understanding that better, they realized that they were using a lot more ink than they ever realized or ever knew before.

Jay Clifford 58:40 That’s so cool.

Caitlin Croft 58:43 I am a little partial. I do focus on our customers and community, so I always love getting to hear these cool use cases.

Jay Clifford 58:51 Awesome.

Caitlin Croft 58:52 Thank you everyone once again for joining. The session has been recorded, the webinar is recording, and the slides will be made available later today or tomorrow morning. Everyone should have my email address. So if you have any questions for myself or for Jay, feel free to email me. I’m happy to put you in contact with him and definitely come back. I feel like this was a really good session, so maybe we’ll do another one and we’ll see you on another webinar or training.

Jay Clifford 59:23 Awesome, guys. Thanks very much. See you guys in the next one.

Caitlin Croft 59:26 Thank you. Bye.

Jay Clifford 59:28 Bye.