Build an Edge-to-Cloud Solution with the MING Stack

Session date: Jun 27, 2023 08:00am (Pacific Time)

FlowForge enables organizations to reliably deliver Node-RED applications in a continuous, collaborative, and secure manner. Node-RED is the popular, low-code programming solution that makes it easy to connect different services using a visual programming environment. InfluxData is the creator of InfluxDB, the purpose-built time series database run by developers at scale and in any environment in the cloud, on-premises, or at the edge.

Jump-start monitoring your industrial IoT devices and discover how to build an edge-to-cloud solution with the MING stack. The MING stack includes Mosquitto/MQTT, InfluxDB, Node-RED, and Grafana. This solution can be used to improve fleet management, enable predictive maintenance of industrial machines and power generation equipment (i.e. turbines and generators) and increase safety practices (i.e. buildings, construction sites). Join this webinar to learn best practices from industrial IoT SME’s.

In this webinar, Robert Marcer and Jay Clifford dive into:

• Best practices for monitoring sensor data collected by everyone — from the edge to the factory
• Tips and tricks for using Node-RED and InfluxDB together
• Demo — see Node-RED and InfluxDB live

Click here for presentation

Watch the Webinar

Watch the webinar “Build an Edge-to-Cloud Solution with the MING Stack” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “Build an Edge-to-Cloud Solution with the MING Stack”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• Caitlin Croft: Director of Marketing, InfluxData
• Jay Clifford: Developer Advocate, InfluxData
• Robert Marcer: Developer Educator, FlowForge

Caitlin Croft: 00:00:00.890 So without further ado, I’m going to hand things off to Jay and Rob.

Jay Clifford: 00:00:06.801 Thank you very much, Caitlin. That’s awesome. So as Caitlin said, welcome all to this presentation on the MING stack and how you can build a hybrid solution from edge to cloud. I thought it’d be nice if we introduced ourselves first. You get to know myself and Rob. So as Caitlin said, my name is Jay Clifford and I’m a developer advocate for InfluxData. In my past life, I was a sales engineer for industrial IoT solutions, so a lot in the manufacturing space dealing with Kepware, Node-RED, InfluxDB, Grafana, and wiring up lots of machines there. I’m a massive fan of home automation projects. So that’s a big passion of mine. And I’m driven and my belief is to make IoT accessible to all. And I’m really excited to introduce Rob. Me and Rob have been working on this project for quite a while now. Rob, would you like to introduce yourself?

Rob Marcer: 00:01:00.499 Yeah. Hello, everyone. Hi. I’m Rob Marcer. I’m the customer success manager at FlowForge. So similar to what Jay was talking about, so my past life. So before being at FlowForge, I was doing quite a lot of work on using Node-RED to link together various CRM systems to do with hospitality. So pulling data from one platform and pushing it into another to facilitate CRMs. I’m quite into home automation as well as Jay said he was. I think that’s a pretty common thing within the Node-RED communities, how a lot of us first discover what Node-RED can be used for. What I quite enjoy doing is helping people get the best value out of Node-RED. So helping them just overcome the next thing to learn, get that new trick or tip into their brain so they can deliver better projects. And my belief, when following the format of these slides, I really like to try to do things in a no-code way. So yeah, I always try and find a way to do something without writing a function node. Sometimes it can’t be done, but yeah, that’s a challenge that I always set myself. So hello, everybody. Nice to meet you.

Jay Clifford: 00:02:21.032 I love that. That’s the power of Node-RED right there. And I feel we’ve got quite a lot of that within the project today, so excited. Well, Rob, do you want to jump to the next slide? Let’s let everyone know what they’re in for. So what we thought we would do today, we’ve got quite a bit to cover, so what we’ve done is split it into four parts so it’s easy to digest. Rob’s going to cover a bit of what FlowForge exactly is. Then we’re going to have a look at a project together. I think the best way to learn is to always try out a solution yourselves that you can try at home. So we’ll introduce you to Plant Buddy, which we’ve deployed using the MING stack and we’ll have a bit of an introduction to what the MING stack is there as well. We’re then going to sort of dig into the architecture a little bit of how this solution’s been constructed. And Rob’s going to give you a bit of behind-the-scenes on FlowForge’s Edge to Cloud solution which will be awesome. And then finally we’re going to get our hands dirty and we’re going to walk through the demo. You’re going to see it live in action. And then we’ll move on to some next steps and you can see how you can get the source code yourselves and get home automating at home which will be really exciting. So, Rob, would you like to take it away and tell us a little about what FlowForge is?

Rob Marcer: 00:03:32.489 Yeah, I would love to. Thank you. Well, actually, before I tell you what FlowForge is, I should perhaps not assume that everybody knows what Node-RED is. Although, I suspect a lot of you do. So what’s Node-RED? So it’s a low-code programming environment. It’s primarily designed for facilitating event-driven applications. It’s open source. It was started at IBM by quite a few of my colleagues actually at FlowForge who are ex-IBM employees, and that were working on that project at the start or in its early days. What’s great about Node-RED is there’s a huge library of third-party custom nodes, and we’ll be looking at some of those today. So quite often if you want to interface with a particular service or technology, for instance, InfluxDB, there’s already a way to do that without you starting from scratch and writing your own code to interface. So that’s great. It’s part of the OpenJS Foundation. I was actually having to look up what At Large project means, but it means one of the OpenJS Foundation projects, which they consider to be used by significant amount of people and something that potentially could have an impact on the way in which people are using JavaScript applications. One last sort of sound bite.

Rob Marcer: 00:05:00.260 So it’s used by — hopefully, this demonstrates a lot of people, so it’s had over 100 million pulls from the official Docker image. So yeah, it’s a great way to quickly prototype and build applications using a visual interface that’s used by a whole load of people. So now I’ve covered Node-RED. So what is FlowForge? So FlowForge was founded by one of the co-creators of Node-RED. So my colleague Nick O’Leary who people who were in the Node-RED community are probably aware of. Well, FlowForge basically makes it really easy for teams to collaborate on building and maintaining Node-RED applications. Those applications may well consist of one through to hundreds of instances of Node-RED, either hosted in the cloud, potentially on edge devices, so IOT devices, and also potentially on-prem. So a lot of what I spend a lot of my time working around is a cloud version, which is called FlowForge Cloud, but it’s also, there’s an on-prem version. If you wish, you can host a complete copy of FlowForge at your facilities or in your own private cloud space if you wanted to do so. It really makes it easy for teams to build and maintain Node-RED applications.

Rob Marcer: 00:06:28.943 And I think this is where a lot of where FlowForge came from is people are getting so much value out of Node-RED, but how can we make it easier so that if you have multiple developers working on an application, how can you make it easier for them to collaborate if they’re not physically in the same location or if they’re needing the different parts of the application to work well together. So yeah, that’s the problem that we’re trying to solve. Some of the perhaps points I’ve covered off there a bit, but I wanted to include something a bit more detailed in the slide. So we’ve talked about collaborating. So when you’re building an application in Node-RED on FlowForge, you can have multiple developers with different access to different parts of the application. You can easily see when people have made changes, you can snapshot those changes and share them between you. You can also share sections of flows so you can save those off and then import them into other projects so other people can benefit on your team from the work you’ve been doing. You can very easily — and this is something we’ve been working a lot on recently, and I think it’s got so much value. You can easily manage remote instances of Node-RED. So using something called our device agent, you can write code on the cloud and then push it out to devices and we’ll be showing this today because this is how Plant Buddy works. So it makes it a lot easier to deal with tens if not hundreds of IoT devices running specific flows and updating those.

Rob Marcer: 00:08:01.389 Even if they are in a facility on the other side of the planet, you can still manage that from your internet connection. And yeah, the one thing that we’re not going to cover too much today but is another area we’ve been working on quite a bit is working on the DevOps pipeline. So making it easier to actually manage your applications, which will be written in Node-RED through from development staging and then on to production. So providing better tools to make that a more structured process, perhaps more in line with you have people who are used to working around things like Git, where you’ve got a little bit more control about what size will be ending up on your production hardware. Some new stuff. What haven’t I covered? A few of these things. Yep, okay. So we’ve been working quite a bit recently on high availability and scalability, so how can we make it so Node-RED in itself, one instance of Node-RED is single-threaded? It is not necessarily designed to scale up particularly well to high throughput, so we’re working on ways that will allow an application that’s written in Node-RED on FlowForge, for instance, to scale out to multiple instances of Node-RED. Also looking at things such as hot swaps, anything that would allow a mission-critical application to keep running in case of any problem either with hardware or software.

Rob Marcer: 00:09:29.417 We’re also working on Swagger for our API. So we do have an API that allows customers to interact with our platform. That is perhaps somewhat the case. It is not very well documented, if documented at all, and so we’re working on getting it so that a lot of the administrative tasks that customers are doing with FlowForge, say spinning up a new instance of Node-RED, for instance, or deploying a new build through the DevOps pipeline, can potentially be ultimated via the API. We’re working on custom instance domains, so this would mean that currently if you are using our cloud-hosted version, all of the URLs, say to access an API that you’ve deployed on there or a project, will be within FlowForge’s own domain. So we’re working on allowing customers to use their own domain, something that can be branded as your company. And then finally, for those of you who know what Node-RED dashboard is, or perhaps for those of you who don’t, it’s a very, very popular way to create a user interface for Node-RED projects. We are just starting down the road of building a new version of this, and what I should explain is the conversion of dashboard uses underlying frameworks which are no longer supported. As much as they work right now, potentially we could see security problems or compatibility problems down the road. So we’ve just started to work on what might be the next — and we would hope is an open-source project.

Rob Marcer: 00:11:01.161 Everything we do is open source, but we’d hope the next very popular way to build dashboards are Node-RED. So yeah, that’s something we’re putting quite a bit of time into now, and we’ll be continuing to do so over the next six months. I think that covers all of that. Okay. So what is MING? I think this is you Jay if that’s okay. Oh, you’re muted if you’re — I can’t hear you.

Jay Clifford: 00:11:26.458 Can you hear me now?

Rob Marcer: 00:11:27.636 I can indeed, yes.

Jay Clifford: 00:11:28.829 Yes. Fantastic. Awesome. Yeah. Honestly, Rob, that’s fantastic. I feel this is one of the major things for me was when working with Node-RED as sort of an open-source user, having a production method for Node-RED was always a problem. And the fact FlowForge is solving this and how we’ll see how it’s solving it is extremely impressive from what you guys have worked on so far. And the fact that you’ve got a new dashboarding solution comes out is music to my ears. So excited for that. So what we thought we would do next before we specifically talk about the Plant Buddy project is talk about the MING stack. It’s a bit of a buzzword that’s going around that’s kind of been thrown out by myself. It’s been thrown out by FlowForge. It’s been thrown out by the guys at MQTT as well, sort of HiveMQ. It sort of originates from a bit of a play on words from the LAMP stack. Now, if you don’t know what the LAMP stack is, it was something that we sort of did run about my university days, and it was an original way of helping you get to know some open-source software to build your very first sort of web-based applications. So LAMP kind of stood for Linux, which was your operating system, an Apache web server, MySQL, which was your database or your storage. And then depending on who you talk to, the P stood for either PHP, Perl, or Python. I think it was PHP for me. I don’t know what it was for you, Rob, but PHP was mine.

Rob Marcer: 00:12:57.444 The same as well. Yeah.

Jay Clifford: 00:12:59.350 Yeah. But yeah, don’t want to tread on anyone’s toes for anyone else using the other Ps, but. So the idea is that what we wanted to do was collaborate together and say for an industrial IoT solution, how can we bring these open source projects together as basically the founding blocks of building a no-code or a low-code solution for industrial IoT, enterprise OT, or even just home automation projects? And so MING stands for — we have M for MQTT, I for InfluxDB, and N for Node-RED, and then G for Grafana. And it’s been around for a long time. We’ve just put a name to it now, but this is kind of where a lot of people end up when they’re building their home automation solutions. If you want to jump to the next slide, Rob. And I just wanted to give a quick shout-out to Balena. So they are another solution provider that we work with who kind of sort of signposts the MING stack and gave it a name. And they’ve sort of been building our architectures for us of how they believe sort of IoT gateways and also how basically cloud-based IoT solutions would work under this architecture. The idea is Node-RED really sits there as your task or your data collection agent, so it’s things like Modbus, OPC UA, MQTT connectors, variety of HTTP connectors.

Jay Clifford: 00:14:30.470 You have Mosquito as a broker that sits on the side either for publishing or subscribing, so another form of communication. InfluxDB sits there as your time series store for all of your time series-based data coming from Node-RED or from MQTT. And then Grafana is sitting there as our observability panel looking to visualize that time series data over course period of time. We can look at alerting and so forth. And that’s something that we can deliver in Node-RED as well. We can have a look at alerting and other reporting-based features in Node-RED as well. So if you want to go to the next slide, Rob. So I just wanted to give a quick recap on InfluxDB 3.0. I feel like we’ve had a lot of versions out at the moment. We’ve gone from 1.8 to 2.6, and now we’re coming into this 3.0 world. And I just wanted to re sort of set where InfluxDB was in our journey and what you can expect from us from an open-source solution in the future going forwards as well as what we have now and what we’ll be showing today. So Rob, if you go next slide again, please. So InfluxDB 3.0 was a complete underlining storage engine rewrite. So it was all built now on free open-source ecosystem projects: Apache Arrow, Parquet, and DataFusion.

Jay Clifford: 00:15:52.106 And the idea here was to eliminate cardinality issues with data, so the ability to have millions if not more, so millions if not billions of sensors or unique series available to yourself which provided a series of solutions for IoT-based solutions but also DevOps-based solutions such as traces, monitoring, observability, stack cases as well. That’s why we’re saying we’re now looking at to be a single store for time series data but also metrics, logs, and traces as well. We’re also now providing native SQL support and further support for InfluxQL. That’s all done under Flight SQL or DataFusion as we like to call it. So that’s all based on a new columnar underlining system. So dealing with millions of rows in millisecond query times. So we’re really trying to increase the right frequency and querying as well. And then you keep all the old features of InfluxDB that you loved, that everyone was known for. Schema On Write, which means the schema is formed as you write data in. You’ll see this a little bit later with Node-RED on how we constructed the payload for our sensors and then how we wrote that to InfluxDB.

Jay Clifford: 00:17:05.711 So that’s just a little bit of a rundown to give you 3.0. At the moment, we’re running 3.0 Serverless and Cloud Dedicated to do cloud solutions, but we will have an open-source offering coming, and we will have a node edge version coming next year as well. So really exciting stuff there. If you go to the next slide, Rob. So I’m just going to gloss over this, Clyde. This is kind of the structure of the InfluxDB 3.0 platform now. Series of data sources to collect; a series of ways of collecting that data whether it be through our own systems or through Node-RED as we’re discussing today, store that within InfluxDB, and then provide your analysis. We now actually provide connectivities for Power BI, for Tablo, for Grafana, which we’ll be showing today. So lots of ways of connecting to InfluxDB and visualizing and working with your time series data. Next slide, please, Rob. So just last thing I wanted to mention just to level-set the room, just in case you’ve not used InfluxDB before. What you’ll see when Rob demos the solution shortly is the idea of the schema that we had to construct. So all databases have a schema, and this is kind of how we facilitate that within InfluxDB. So what you have is what we call a bucket or a database. This is where you store your data within InfluxDB.

Jay Clifford: 00:18:34.793 You then have a measurement, which is what you can consider to be the table, so that’s where all of your data will reside. A tag set; you can imagine this as part of your primary key. It’s a way of differentiating your series from one another. So if you say have a tag key called server, you’d have a number of tag values underneath that with different server IDs, and this is how you would differentiate your series coming from different servers. And you’ll see how we do this in the project as well, where we differentiate our devices from one another using tag sets. We then have field set. This is where your actual values are stored. So this is where you’ll be storing all of your sensor data, or all of your — sort of if you’re dealing in the DevOps world, your traces and log data as well. Timestamp; that is your primary key. It forms part of that along with your tag set. And then as we’ve been saying, series is a unique combination of measurement and tag. So with all that in mind, let’s introduce you to Plant Buddy. So if we skip ahead again. So when me and Rob were creating this webinar, what we wanted to do was have a novel way of basically showing how you can take a project that you can try at home, and then scale it indefinitely.

Jay Clifford: 00:19:57.133 So the idea of what we wanted to do here was basically take a series of different analog sensors that we could wire up to a Raspberry Pi. So say you have a Raspberry Pi here. And then we had a bunch of sensors — like light sensors, moisture sensors — like this. So really cheap sensors that you can buy in sort of home automation projects, and temperature sensors, like so. We said, “How can we collect data from these sensors using Node-RED deployed on a Raspberry Pi using FlowForge? And how can we derive, basically, our plant health? How can we observe how well our plants are doing over the course of time?” And we want this solution to be able to scale, and we also want this solution to be applicable to much more hardened solutions, whether they be deployed on a factory floor or done at scale within an industrial greenhouse. Wherever you’re deploying, this should be translatable to your own solutions. So do you want to quickly jump onward? Thank you. So the reason why we’re using the MING Stack for this is because of the types of questions that we’re trying to answer. A lot of the questions that we’re trying to answer require a time series database, not only, but they require us to answer questions that are all to do with timestamps.

Jay Clifford: 00:21:19.237 So we kind of want to be able to determine the state of which our plants are in over the course of a period of time. We want to know how long our plants were in a particular state, whether they were in a healthy or a bad state, and when we should be alerting when they’re in a bad state. And then once we kind of have that ability, we want to then look at sort of the key buzzwords here. How can we then start applying anomaly detection to a fleet of our Raspberry Pis that are monitoring hundreds of our plants? How can we identify when one plant is behaving irregularly compared to the other plants? And yes, we’re referring to this as plants because we’re dealing with Plant Buddy, but this same core feature goes along with if we’re working on machine data from a series of similar machines. If we’re looking at, say, vibration data, the similar rules apply. It’s the similar goals that we are trying to achieve. If you want to jump to the next one. So if we have a look at the architecture as we see here, this is a very bird’s eye view of what Rob will be showing you. So we have our Raspberry Pi hooked up to our plants that will have FlowForge deployed to it. And we’ll have a series of these deployed. For instance, for me now, I’ve got one deployed here. Rob’s got one deployed at his house, which is great. As you can see there, he’s done a better job than me. We’ll have these deployed on the edge devices, we’ll bring that data into FlowForge, send that data to InfluxDB, and then finally visualize that data with Grafana.

Jay Clifford: 00:22:54.221 So if you maybe want to move on again, Rob. So if we want to dive a little bit deeper into this, this is kind of how the data is going to flow. And Rob’s going to drill into each of these as he talks through the sort of FlowForge environment and each of the nodes that he created. So what you have here is you’ll have a sensor DAQ or an application on Node-RED that collects the data directly from the I2C on the Raspberry Pi, which is how we’re getting our data from the analog sensors. We’re then enriching that data before we send it to — before we send it to FlowForge in the cloud using their Edge to Cloud feature. More on that later. Rob will give you a quick overview on how that Edge to Cloud feature works. We’ll do some further enrichment upon that data. We’ll then send that data to the InfluxDB third-party connector, which you’ll get to see. We’ll collect that data into InfluxDB 3.0 and then we’ll dashboard that data. And we’ll show you how basically some of those goals that we want to achieve in that dashboard, looking at the plant state over time and how we can further analyze our IoT data coming from our plants. And so, Rob, I feel like we’re missing the “M” in our stack at the moment. We’ve got the “ING” stack. Could you give us an overview of how Edge to Cloud is working?

Rob Marcer: 00:24:20.311 Yes, I could. And it’s almost as if we planned that. Thank you. So we are using MQTT in this demonstration, but we’ve actually built MQTT directly into FlowForge. We use it to deal with some of our own administrative tasks, so managing edge devices, but we’ve built custom nodes, which you can use to very easily send information from single device to many or vice versa. So it just facilitates that. It makes it a lot easier for you to spend your time building your application rather than hooking up to an MQTT broker. I’ve put a link here. I don’t know, Jay, if you could possibly drop that into chats if people want to look at the project. It is open source as everything that we do, so always happy for people to give us feedback and suggestions about how it could be made better. So we are using MQTT. It will be somewhat hidden away, but I will explain, in the demonstration where that magic is being done. So I think then — unless there’s anything else you want to add, Jay, I think we’ll get to a demonstration of certainly the first part of how this works, the FlowForge part of this. Okay then. So I’m going to start off on FlowForge. So one second, full screen that.

Rob Marcer: 00:25:51.321 Okay. So within FlowForge, you can have applications, and applications can consist of one or more instance of Node-RED. In this particular case, we have a few instances of Node-RED. The ones we’re going to look at, we have one instance that’s running on my Raspberry Pi and Jay’s Raspberry Pi. That’s our houses. So these things, hopefully — I try to get it so people could see it. I know you can’t see detail, but this is definitely a plant. I can confirm that. And this is the sensors. So as Jay was showing, this is actually the moisture sensor, for instance, which is actually in the soil of this apparently very neglected plant according to the sensor. So let’s look first of all at the Node-RED instance, which is running on this Pi here, and also on the Pi that Jay’s got. So I’m going to go into our devices section. What devices allow us to do is to manage one or more remote instances of Node-RED. So we can see here this application has two devices. Now —

Caitlin Croft: 00:26:56.685 Rob?

Rob Marcer: 00:26:58.215 Yeah. Sorry, yeah.

Caitlin Croft: 00:26:59.106 Sorry to interrupt. We’re still seeing the let’s start slide. Did you mean for us to see that, or?

Rob Marcer: 00:27:05.095 Okay. I’m really sorry. No, no, you should be seeing a demonstration. Yeah. I don’t know why it’s not doing that. So I will try just sharing my desktop, see if that — thanks for pointing that out, Caitlin. Can you now see —

Caitlin Croft: 00:27:19.674 Now I see FlowForge.

Rob Marcer: 00:27:22.022 Awesome. Thank you for pointing that out. And sorry, excuse my — I find no matter how much time I spend doing these webinars, I always mess up some part of the Zoom interface. No criticism of Zoom, obviously. So, well, I’ll just jump back a little bit and say — just to show the things I was talking through there. An application on FlowForge has one or more instances of Node-RED, which we can see here, which we’ll be looking at these in a bit. We can then also have instances of Node-RED running on devices, which is where I’m going to start. So I’m going to click onto Devices. This is where I was when I realized that nobody could see my screen. So here we can see we have two instances of Node-RED running on Raspberry Pis. So we can see — I mean, here, it’s kind of easy to manage two instances, but potentially, you could have hundreds of these. We see the situations where there’s literally hundreds of IoT devices, edge devices which are feeding data back in. So as much as this — so if I click one here, this is going to tell me a little bit more about the —

Caitlin Croft: 00:28:29.723 I’m sorry to interrupt you again. Would it be possible to zoom in a little bit?

Rob Marcer: 00:28:35.515 Yeah, of course. Yeah. Yeah.

Jay Clifford: 00:28:37.010 So sorry about that, Rob. Cheers.

Rob Marcer: 00:28:38.470 It’s fine, it’s fine. One second. If I can hit the zoom button, I’m going to over-the-top and hopefully, that works well for people.

Caitlin Croft: 00:28:48.465 Perfect.

Rob Marcer: 00:28:49.695 Does that seem good?

Caitlin Croft: 00:28:50.535 Yeah.

Rob Marcer: 00:28:51.155 Yeah. Great. Okay. So here, I’ve gone into the information about my device specifically. I can get certain information like its current status, is it running, how long ago we checked in, so we can get an idea of what’s going on with these particular devices. We can also see things like the current version of code that’s running on this device. So I was mentioning earlier, a little bit about the DevOps pipeline. This is allowing us to have a look at, yeah, if we have different builds over, in this case, Plant Buddy, which one is currently deployed to their devices. Now, as much as this device is right here next to me, even if it was on the other side of the planet, literally somewhere with internet access, I can now interact with it. So what I’ve done is I’ve put my device into developer mode and then I can press at device editor. And it will just reload it, or [inaudible] opened and I will — I’m quite a bit zoomed in, so please guys, let me know when this loads if it’s not legible. One second. Hang on, I did already have that open, so don’t know if the way in which I’ve done that has messed something up.

Rob Marcer: 00:30:13.540 So what I’m just doing here is, there is a client that allows us to remote-control this device. And hopefully, when doing live demos. It’s not wanting to enable, so just give me a moment. I’m not panicking yet. You will know when I’m panicking. So I’m just putting it back into developer mode. Sorry. Because I had already left this open, I think the session’s remote control has disabled. Now I’m just doing really badly here. One final attempt to get this to work.

Jay Clifford: 00:30:57.118 There, regular demo.

Rob Marcer: 00:30:59.598 I’m breaking things. I’m going to try and remote-control Jay’s. I don’t know if when I’ve been gesturing towards mine, if I’ve broken it in some way. Let’s try on Jay’s.

Caitlin Croft: 00:31:10.118 It’s the joys in live demos. It’s all good.

Rob Marcer: 00:31:12.237 Yeah. Yeah. I’m struggling today. Sorry, guys. Just let me — I’m going to stop just clicking the same thing after this and do it in a different way. Which is a real shame because I really wanted to show you [inaudible], but I don’t know, I cannot. So sorry about that. I do have another version of the code which is running on here, which I can access through all the devices. So if I go into edge device here, hopefully, I will do a bit better here. Okay. Great. So this is a copy of the code which happily I have running on FlowForge Cloud, so I can actually take a look at it. It might be that, yes, you’ll have to bear with me a little bit because some of the things I was going to demo might not work quite as I expected. But what this is basically doing is when it receives a command — and the command comes via MQTT — it makes requests to the four sensors on the Pi which are monitoring the plant. It collects the data back from those sensors, it changes it into a standardized format. So yeah, normalizes any of the measurements that may be in a format that isn’t particularly friendly to be sent on to another application. And then passes it back through MQTT to a central project, a project that’s running on FlowForge Cloud, which is then going to process that data. So what I’m going to do to demonstrate that is I’m now going to go on to the central instance, which I can get to here.

Rob Marcer: 00:32:59.439 Okay. So I’m going to go on to the monitoring and collation. So yeah, sorry for everyone. That’s not as slick as I would have hoped. So what’s happening here is this is running on a central system on FlowForge Cloud. And when I press this button, what it’s going to do is make a request over MQTT. So this is what I mentioned earlier, we’ve got these custom nodes that deal with MQTT, and all I have to do is tell them that they need to talk to edge device, and they’re triggering — so the topic’s triggering that, just basically saying, “Hey, give us some readings from all of the sensors.” That will send the command off to — this is also known as devices in here, which will then read all the sensors. And yes, I’ve scrolled in a bit — I’ve zoomed in a bit now, so I can’t get it all on screen. But these are the four different sensors; moisture, light, temperature, and humidity. And once it’s read the sensors, it’s going to output here and send that again back through MQTT back to the central system. So let’s have a look at that. So if I press this, this should make a request. So we’ve got four responses there, and this is where we find out if I had in — so my device is still online. I know that because — I don’t know if everyone can see this okay. Maybe zoom a bit on this one. So these are the responses coming back from the device.

Rob Marcer: 00:34:27.178 And so I know that my device is online because we had eight responses; four of them have come in from Jay’s device, four of them are coming back from the one that’s next to me. So if we go in and have a look, so each one of these that we can see on the Node-RED debug, so which I should explain a little bit. So within Node-RED, the messages coming in through these wires flowing through here, you can have some logical things here. So for instance, I just showed that I have a version of the sensor code running in the cloud. But because that doesn’t have access to any devices, I need to make sure that I’m only going to push the relevant information that’s coming from edge devices onto InfluxDB. So that’s what I was doing there. And then this is going to be sourcing the data and tidying it up, checking which particular sensor the data’s coming from. This thing here will show if it comes from another sensor that we don’t know about. That should never happen. But the endpoint is we have these debug notes. So all the information’s coming through here, and then if I roll over these debug outputs, we can see we’ve got these, yeah, orange checkered line around there. So that’s when the messages gets into here, so did the sensor ever know, and where it’s going to write it into InfluxDB, we’re getting these outputs. So we can have a look at one of these payloads.

Rob Marcer: 00:35:49.730 So this particular one, what does it tell us? So it has a timestamp in it. This is from a humidity sensor, so the humidity reading is 46, and I believe that’s a percentage humidity. I think that’s how humidity works. So we have a unique device identifier and I think perhaps Jay in a little while may show where some of these values end up in InfluxDB. So we have a unique identifier for the device which is assigned by FlowForge. So if you’ve deployed this out to lots of devices, each one will respond with a unique key so you can understand which devices are returning that data. But what we’ve also done is we’ve mapped this really helpful DQBG, etc., etc., over to a more helpful name which is DemoPiJay. So we can without looking up, yeah, it’s pretty easy when there’s only two devices. But if there was 100 or so, that makes it a lot easier to understand where that information is coming from. What we’re also doing is, I think, in this case, I assume your Pi is a Pi 4, Jay. I’ve just guessed it is. Okay, yes, so is mine. But you could also potentially — if you have different versions of hardware, you could also start to show that there based on devices that the informations come back from.

Rob Marcer: 00:37:06.220 I’ll open up another couple of examples. So this time, again, we’ve got timestamp, temperature this time is in centigrade. This again was from Jay’s device. I’m really interested now to see if I can find some results from mine. Is it that I literally broke my device whilst I was pointing at it? I may have done that. Jay will be able to tell us this shortly because we will see if my data has stopped flowing through. I’m going to stop looking. I can’t see any from — never do a live demo, I think is the lesson to be learned there. So the data flowed through —

Jay Clifford: 00:37:43.051 And I can prove that you got —

Rob Marcer: 00:37:44.998 Say that again. Sorry.

Jay Clifford: 00:37:46.169 I can tell you that you have — your device has imploded.

Rob Marcer: 00:37:51.728 Once I let you talk, I’m going to try and fix it just for — yeah. Because it’s a little bit frustrating, and I apologize to everyone. But yeah, live demos and all. Yeah. So I’ll just talk back through that from the top level. So what we’ve demonstrated is we’re using FlowForge to manage two devices. So edge devices, which can be anywhere with an internet connection out on the planet. Those devices are reading sensor data when they’re requested to do so by the central system. So this sends out a request. I think I’ve got this to currently do this every probably 15 seconds, I guess. So this data request, the sensors are read, the readings are put into a standard format, and information is appended such as what the device is called. That then flows back in here. The data is tidied up a little bit more. I won’t go into too much detail, but tidied up a little bit more. And then it’s pushed through to InfluxDB using, as I think I’ve already mentioned and perhaps we’ll touch on a little bit later, the custom node, which makes interacting with InfluxDB really easy to push the data through. I think, Jay, that’s a sensible point to hand over to you if that’s okay, whilst I smash my Pi with a hammer if that’s okay. Oh, I think you’re muted if you’re talking.

Jay Clifford: 00:39:19.489 Can you hear me now?

Rob Marcer: 00:39:20.743 I can, yes. Hello.

Jay Clifford: 00:39:22.272 Fantastic. Rob, would you mind relinquishing control of your screen, and then I will take over if that’s okay?

Rob Marcer: 00:39:29.402 Hopefully, I’ve done that now. Yeah.

Jay Clifford: 00:39:31.301 Yep. So hopefully you can all see this screen.

Caitlin Croft: 00:39:36.871 Looks great.

Jay Clifford: 00:39:38.144 Fantastic. So welcome to InfluxDB 3.0. So this is our current serverless option. This is our free tier on serverless. So sort of the 30 days of retention on your buckets. The major difference with 3.0 now compared to TSM is you now have unlimited cardinality for your devices. But what we’re going to do is let’s have a look at where Rob is storing that data for our sensors. So what we’re going to do is we’re going to navigate to our buckets that you can see here. We can see that we have a series of buckets that I own. I’ve got FlowForge. What’s going to be a little bit more interesting is the fact that now you’ll see on this side is we actually have a SQL generator because the 3.0 of an InfluxDB interacts directly with native SQL which has then built out time series functions on top. So Rob was sending the data from our sensors over the measurement called sensor_data. So if I quickly run this, you’ll be able to see the data coming from each of our sensors. Now just to prove to you that Rob’s sensor was working originally, you can see that we do have Rob’s device name there historically that it works. So we can approve that and the data that you had here. So currently this is the data that we have.

Jay Clifford: 00:41:03.149 You can see we’re bringing in the humidity, the light, the moisture, and the temperature. Moisture, so the light sensor had some interesting readings. I definitely think if anyone has any better suggestions on how to monitor these or photoresistor light sensors, especially Node-RED, please let us know. That’d be really great, and we’ll add that to the project. Currently, we just kind of do a calculation to perform a percentage of on and off on these, but it’s definitely not the best way to measure the light from these sensors. So if you have any other suggestions, that would be great. So we have the data stored in InfluxDB. We also have those series of tag keys that Rob sent to us which was the device ID, the device name, and the device type. So the core thing here is that these will scale with use. So it used to be the case in TSM land that if you get past say a certain amount of sensors — sorry, IDs or tag keys, your InfluxDB instance would start running slow or fall over. Based on the new 3.0 system, this will not happen. You have unbound cardinality for these tag keys and devices. So this is all well and good. We got a nice table. Let’s then visualize these in Grafana using actually our new plugin.

Jay Clifford: 00:42:26.755 So based on Grafana Cloud here — so this is Grafana Cloud. You can also use this on a free plan on Grafana Cloud. I believe they said it’s free for life, so there’s plenty. The cool thing is I have not reached any limitations with their free plan based on using InfluxDB, which is awesome. So if we have a look at our data source, and we have a looked for FlowForge, we have a new plugin called Flight SQL. So this is a new form of protocol that basically allows you to send a SQL request to InfluxDB but returns a columnar data payload back, which is within what we call the Flight protocol. So this is based in Apache Flight. It’s a bleeding-edge protocol that allows for efficient data transfer. It’s heavily used by Databricks, by other data warehouse providers as well. So this is kind of where we’re trying to leverage that power of big data and millions of points per second, and this is kind of how we’re doing that. So FlowForge is using Flight SQL, but we have now also brought back compatibility for InfluxQL, for Grafana as well via the v1 API. So you’ll be able to leverage that as well, but we’ll be using SQL for this. So if I jump back and if I jump back into our dashboards, this is a dashboard that we created to monitor our plants.

Jay Clifford: 00:43:58.797 So I jump in. You can see at the moment we’re on the device here, device Jay. And we also have Rob’s device here. You can see we have our plant health, we have a device log of each of those sensor points that are coming in along with sort of our histograms of temperature — sorry, not histograms. Our sort of line graphs of our temperature, our humidity, our moisture, and our light. If I sort of bring this up, say if we bring up for 24 hours and let these load into Grafana. Just a second. There’s quite a bit of data that we’re loading into Grafana for this since we’re taking this every 15 seconds, and I haven’t aggregated. So you can see each point that we’re bringing in, so you can see how the moisture level dropped over time as well. What we’re trying to do here is trying to show you how you can basically use SQL to generate a plant health score. So if I quickly click on this node here and I quickly show you how we’re doing here, what you can do in SQL, which is a bit more flexible than what you used to be able to do in a lot of query languages, is be able to create case statements. And what we do here is be able to say, “Okay, well we know that temperature should be between 19 and 23 degrees. Our moisture should be between 60 and 100 percent. We do humidity between 45 to 55 and light 75 to 100.”

Jay Clifford: 00:45:27.399 And essentially, we build a series of case statements and basically make these weighted parameters, and then formalize a percentage on a health score. And this gives us basically a timeline for our plants to basically say how our plants — the status of our plants and how long they’re within that status for. So the great thing is in the future as well — this is the first step on this — as we expand InfluxQL v1 again, you’ll be able to have a look at anomaly detection basically through MAD, medium average deviation. You’ll be able to look at forecasting once again through Holt’s method. All of these functions are back included within InfluxQL, which is awesome. So that kind of gives us a bit of an overview of the project there, as you can see, and then I can actually show you what’s happening with Rob’s device as well. I just quickly bring this back down. This doesn’t freeze on me. There we go. I go, Rob. I actually think, Rob, we’re bringing data back in for you.

Rob Marcer: 00:46:37.538 I think, yeah. I turned it off and turned it back on again.

Jay Clifford: 00:46:41.679 There we go. The big red spartan. There we go. Fantastic. So as you can see, we now also have Rob’s device. And I’m actually generating these device variables using our InfluxQL v1 parameter on those as well. So moving on from here, I think from the demo now, if you are happy, Rob, unless you want to quickly go, do you want to try your part of the demo again or should we move on to best practices? Up to you.

Rob Marcer: 00:47:08.968 I’m mindful of time. I don’t know if — yeah, giving some time for people for questions. So I think it’s okay, yeah. Yes. If anyone’s interested, the thing I couldn’t demo is how we can remote manage devices. If anyone’s interested in seeing that, I would be more than happy to do a demo. Please contact us directly.

Jay Clifford: 00:47:30.690 Sweet.

Caitlin Croft: 00:47:32.084 Yeah. And maybe, Rob, depending on timing at the end, maybe we can still do it even if people have to drop off, then it’s still in the recording.

Rob Marcer: 00:47:40.969 Yeah. Sure. Yeah, I’d be more than happy to do that. Yeah.

Caitlin Croft: 00:47:42.491 Okay. Let’s do that. We can go through the rest of the slides, answer everyone’s Q&A, and then we can do the bit of the demo at the end.

Jay Clifford: 00:47:50.763 Sweet. Okay. Let’s get through a quick blaze pass of tips and tricks. So things that we learned whilst me and Rob were working on this project. So one of the main things that FlowForge has really helped with InfluxDB on is we’ve kind of sort of strayed a little bit from using a task engine within InfluxDB. We wanted to refocus on becoming that high-performance storage engine, so that’s left asking many people going, “Hey, well, what do I do as a task engine for InfluxDB?” And actually, FlowForge, it becomes an incredible task engine for the ability to actually query data out from InfluxDB, transform it in some way or provide some sort of enrichment to it, and then writing that back to InfluxDB. So it was one of my top tips, suggest checking out FlowForge’s hosted Node-RED just as a task engine for your time series data. The next one or last one from me here is basically a gotcha when you’re working with the time series node. It’s the difference between the payload formats of InfluxDB out and InfluxDB batch.

Jay Clifford: 00:48:57.995 So InfluxDB out, you have to format your payloads as such. You can basically see it’s a list of objects. One object contains our field values. The next object in the list contains our time values, whereas, within InfluxDB batch, you can see we actually define a list of basically a nested object, which contains our measurement name, our fields, our tags, and our timestamps. So just a gotcha for you to check out there. And last but not least, just for people that want to use the v1 API, you have to use DBRP mappings. Come talk to me in the community if you need any help with this. We can chat about it, but just to give you an idea of how this kind of works in Grafana and in Node-RED if you were to use the task engine with InfluxQL. Okay. Yeah. Rob, talk a little bit about your Node-RED tips.

Rob Marcer: 00:49:53.975 Yeah. Sure. Thank you. So one thing that I have touched on a couple of times but stir off, there are some amazing custom what I call within Node-RED is custom nodes outlet, which can speed up integration to literally thousands of different services and APIs and systems. So I would always advise look for those first. What you will quite often find is there might be more than one, so I think when I first searched for InfluxDB, I found a few different ones. You may want to read and have a look at which ones are being maintained more often. You may find ones which are official or ones which are actually being developed entirely by third parties just because they like a particular technology. Always look for the custom nodes. You can find them on the Node-RED website, or if you use the pallet manager within Node-RED, there’s a search engine in there. Another thing which I didn’t get to demonstrate, but maybe if we get some time at the end, I will show you. I think it’s really good to develop a user interface early on in the development process, definitely when you’re working with edge devices or IOT devices. So what I could’ve shown and will at the end is I’ve got basic user interface which allows me to show the four readings from any one of those edge devices locally.

Rob Marcer: 00:51:14.360 So I can point a browser at that device, it brings up an interface. They can just make it so much easier to troubleshoot problems, pick out if I have apparently broken my Pi at the most inopportune time. Yeah. It makes that so much easier. So developing a user interface using a custom node searches dashboard is a really good idea early on in the process. It makes it a lot easier to troubleshoot problems. Another thing that I think is generally great practice is to use groups. So when we were going through when I was showing the flows I had laid out — I don’t know, Jay, perhaps if you jump back a couple slides, I think you’d used groups somewhere, I think. Or maybe three or four slides, but. So it’s a way within the Node-RED interface to group together. Yeah. So the way in which they’re — Jay’s got the border around that set of functionality. And yeah, the ones that I was showing earlier, you had several of those groups. And what makes it a lot easier is, if you’re coming back to read your flow in a year’s time, or if someone else is coming back to it to make changes, it makes it a lot easier for them to understand. I mean, in this case, there’s just one group, but it makes a lot easier for us to understand what that particular group was doing because Jay had labeled it. Oh, yes. Yeah. Sorry. Thanks. Thanks, Jay.

Jay Clifford: 00:52:40.594 [crosstalk].

Rob Marcer: 00:52:41.464 Yeah. Thanks. And the thing is — I think I have a — yeah, so just one other thing using the exec mode is — so exec mode allows you on a device to interact with the underlying operating system. So whatever you can access, say through SSH or batch command, you can access through Node-RED. So exec is very powerful, where you have access to the operating system of the edge device. The last thing I’d say is, use the Node-RED forums. They are such a fantastic group of helpful people who will go out of their way to — as long as try and provide them a clear description of the problem you’re facing, they will go the extra 10 miles to help you get unstuck on a project. So they’re great. And then one last thing is, I’ve done some blog posts around Node-RED tips which go into a lot more detail than I could cover here, which is on our blog on the flowforge.com website. Thank you very much.

Jay Clifford: 00:53:43.852 Sorry, just resharing my screen. We had someone thought it was a bit zoomed in, so just was trying — oh, there we go. Awesome. Thank you so much, Rob. That’s great. So I guess what we’ll do now is — I think, Caitlin, do you just want to jump to questions? What I’m just going to leave onscreen here is just where you guys can collect the code from. So if you just use the QR code, or the address, you can then collect, basically, all of the flows that Rob was using for this project, and try it yourself in FlowForge.

Caitlin Croft: 00:54:14.772 I think we just have to do another webinar with you guys since there is clearly so much to cover.

Rob Marcer: 00:54:20.910 [inaudible].

Caitlin Croft: 00:54:22.407 So we’ll go through some of the questions. I know a few people have to drop off. Please feel free to email me if you want the recording link. So happy to share that. So, Rob, do you guys have a few minutes? There are a bunch of questions, so I just want to be conscious of the time.

Rob Marcer: 00:54:38.508 That’s absolutely fine with me. Yeah. Happy to.

Caitlin Croft: 00:54:41.045 Cool. All right. So let’s see. It sounds like someone is using a competitor of ours. I’m not sure if it’s InfluxDB or Node-RED. But their question is, to what extent should the remote edge controller — i.e. a Raspberry Pi — be easily configurable to use Zigbee2MQTT? Hence why I’m talking about MINGZ. So MING stack with a Z.

Jay Clifford: 00:55:15.697 Yeah. I mean, absolutely. So I feel like we’ve kind of sort of branded the general MING stack there as a starting ground, but the acronym can increase or decrease with every solution and how you like. The flexibility with Node-RED in a lot of cases is there are lots of connectors available to allow you to talk — basically, translate and talk different protocols, and then bring them out into a protocol that you want to utilize. We’re the same with Telegraf, our own connection, our own suite as well. Which allows you to convert protocols, write them to InfluxDB, write them to Node-RED, write them to a different protocol. So yeah, I wouldn’t say it’s a strict methodology, but it’s a nice place to start and add your own acronyms to it too.

Caitlin Croft: 00:56:06.436 And I think it just shows you the flexibility of whether you can add in what tools you’re comfortable with and stuff like that and what works best for you. So there’s a couple of questions, Jay, here about why Telegraf isn’t a part of this stack and if Telegraf is being used at all.

Jay Clifford: 00:56:25.695 Absolutely. So Telegraf could absolutely be used within the stack. It really depends on what level — the fact is, Node-RED acts as a Swiss Army knife that can do just about everything, and it’s adoptable in a way that allows people to have a no-code gooey interface for, basically, getting things up to production level ready as you go. And I’m sure Rob will tell you this. Telegraf itself is a low-code configuration-based agent, which allows you to collect data and write data to InfluxDB or to other series of platforms. So it’s great for data collection. The one thing that it doesn’t provide is — it does provide the ability to do aggregations in some form of processing, but it doesn’t provide that flexibility in terms of where FlowForge does of sort of mass deployment that you can control like Rob had that gooey interface which is approachable, and also provide a series of very well-supported functionality on top to allow you to basically write to different services or collect from different services. They’re intermingled. I’m a massive Telegraf fan. So you can use both or at your leisure.

Caitlin Croft: 00:57:47.455 Awesome. All right. So let’s see. Can the Flow — sorry. Can the FlowForge edge deployment start an edge broker and handle MQTTs?

Rob Marcer: 00:58:01.970 So pretty much anything that you can run within Node-RED, you run on one of those edge devices. So the answer is most probably yes to that specific question, although it’s not something which I’ve tried. But yeah, I mean, whereas — so when you’re running an instance of Node-RED within FlowForge, so rather than normal device, it’s provisioned in a particular way, which means you have different ways to access their file system or potentially the underlying hardware when using the devices. It’s pretty much the same as running for all in Node-RED and you can almost always find a way to access a particular system. So I’m pretty sure the answer to the question is, yes, you can do that. I’d be happy to take the feedback if I’m wrong and try to work it out how to do it.

Caitlin Croft: 00:58:58.143 Yeah. And everyone should have my email since there is only —

Rob Marcer: 00:59:01.038 Okay. Caitlin?

Caitlin Croft: 00:59:03.512 Yes?

Rob Marcer: 00:59:05.059 Sorry. Sorry. I think there was just a bit delay. Sorry. Sorry, go on. Sorry.

Caitlin Croft: 00:59:09.518 I was going to say if any of you want to be connected with Jay and Rob, feel free to email me and I’m happy to put you in contact. All right. Next question. Can FlowForge — wow, I cannot say that today apparently. Can FlowForge be deployed at the edge with InfluxDB so it can still collect data in low-connectivity environments?

Rob Marcer: 00:59:32.513 So I’ll perhaps let Jay talk about the InfluxDB part of this. I don’t see any reason why that can’t be done. There are ways to use the FlowForge device engine with less than the full internet. You do need a certain amount of internet available to get the flows out to those devices. But for instance, in scenarios where a device can’t access NPM, no package manager, there are still ways to use FlowForge on those devices. So I don’t see any reason why you could not. Jay?

Jay Clifford: 01:00:13.871 Yeah. Yeah, absolutely. I mean, one of the main stacks there is InfluxDB 2.6 or OSS, currently, or 1.8. People using collaboration on edge devices with Node-RED all the time. We’ll have a version of our 3.0 open-source solution out later, hopefully, this year, maybe next so that — yeah, and that will have compatibility directly with Node-RED as well, so you can deploy that locally on your edge devices, absolutely.

Caitlin Croft: 01:00:44.042 What safeguards have been built in to alert and deal with breakdowns in comms between edge and cloud, i.e. interruptions and internet?

Rob Marcer: 01:00:54.610 Yeah. Good question. All Pis just breaking partway through demos. Yes, indeed. So yeah, I mean, within FlowForge, you can view which devices are offline. I think my timing was just so perfectly wrong that I went and looked, and it said it had been online so many seconds. I suspect I’d just poked it. I’m not going to say Pis aren’t really reliable and aren’t perfect for lots of situations, but then again, this is perhaps not quite as production as I suspect a lot of people might actually be deploying in their businesses. So yeah. So FlowForge can show if devices are going offline. Also, potentially, you could use the data which is going through to InfluxDB to alert when a device stops responding. So that could be something, yeah, if certain devices dropped offline. There are lots of different ways to do it; either built into your own application or potentially using — well, yeah, using the interface within FlowForge. Jay, if you’ve got anything you want to add to that.

Jay Clifford: 01:02:10.631 Yeah. I guess just sort from an InfluxDB point of view, if you really have any critical issues with long periods of data outage, you can make use of InfluxDB’s own EDR feature, which basically stores the data on disc and will retry the data from a durable queue. So if you are finding you have horrific internet outages over a long period of time, there are other solutions to achieve EDR while still using FlowForge and InfluxDB. So yeah, definitely consider checking this out.

Caitlin Croft: 01:02:45.494 There are so many different methods for hosting Node-RED and InfluxDB. If using Linux, what are your favorite distributions?

Rob Marcer: 01:02:56.601 Debian, personally. But I don’t know. I’m not saying that’s a particularly educated point of view. I think it’s kind of what I grew up on coming from the sort of LAMP stack days that we were mentioning earlier. So yeah, I find Debian to be a great environment to run Node-RED.

Jay Clifford: 01:03:16.553 Yeah, same for me. Ubuntu. I mostly deploy everything really in Docker containers nowadays and do a lot of my development in Docker containers. Just it’s a self-fulfilling prophecy. If I break anything, I can just kill the container and bring something else back up, do that big-bed RED button. So yeah, I would agree, most things are containerized. If not Ubuntu, Debian’s a great place to start.

Caitlin Croft: 01:03:40.786 And if you use InfluxDB Cloud Serverless, do you have to have the queries in Flux or just SQL?

Rob Marcer: 01:03:51.129 So currently with 3.0, the two query engines that you can execute against the two query languages are InfluxQL and also SQL. The Flux is still under there, under the API. I don’t recommend it as highly. It’s flexible, but the performance isn’t there for 3.0, whereas it is with SQL and with InfluxQL. So highly advise using either SQL or InfluxQL.

Caitlin Croft: 01:04:19.925 All right. So someone says, “I’ve used Node-RED on RPIs in an industrial space reading PLC data via Modbus every five seconds in Node-RED then sending via MQTT to InfluxDB in the cloud. We have some typical network connectivity problems on the edge devices, sometimes losing them for 30 minutes or more. The native MQTT out node in Node-RED, unfortunately, drops unsent messages even on QoS 2. Seems that a new message replaces the old message on the node regardless of whether it’s sent or not every five seconds in our case. Interested to find out what others are doing to solve this problem. We created a function node that essentially buffers the MQTT messages and memory if the MQTT out node’s status is unconnected, and we would lose the data if they have any power issues.”

Rob Marcer: 01:05:24.741 Yeah. I mean, you just — yeah. I mean, I was thinking when I was listening to that thinking about solution and it pretty much ended up where you then said you were solving it. You could potentially use the file system. So I suppose it’s, yeah, you start off with storing things in memory. But if you’re wanting something to be a little bit more robust, writing something to the file system could provide you a little bit more security. I do unfortunately see memory cards die in Raspberry Pis from time to time which is still susceptible to that. But yeah, the file system is something that’s quite easy to interact with in Node-RED and would provide you slightly more robust storage. But yeah, I mean, as you said, queuing them up, I mean, yeah, 30-minute outages on network sounds quite difficult to deal with. So it’s great that you found a solution to that other than trying to fix — yeah, get the internet fixed. But yeah, the file system could definitely help there.

Caitlin Croft: 01:06:26.506 All right. Here’s a question for Jay. How do you suggest down-sampling with InfluxDB 3.0?

Jay Clifford: 01:06:36.363 Awesome question. So we are actually releasing an automated down-sampling feature for 3.0 that will be coming soon, which will give you automatic down-sampling capabilities for your data. So potentially, when your device hits a retention period, rather than just deleting the data, you’ll down-sample, move that to a bucket with a higher — or sorry, database with a higher retention. The other method that we currently suggest right now is use FlowForge. That’s a great way of down-sampling. Run an InfluxQL series, which does a form of aggregation, so basically say take the average or the max or the last, and then rewrite that to a new database with InfluxDB. So that’s a great way of doing a no-code version. There’s plenty of our client libraries out there, so if you use AWS or Azure or even just standard Docker containers, spin up a Python container, use one of the client libraries, or yeah, spin up any type of language that you want, and then down-sample there and write that back to InfluxDB. So that would be my suggestions for downsampling currently.

Caitlin Croft: 01:07:42.700 Cool. There’s been a lot of questions around versions of InfluxDB that are currently available, as well as ones asking what is coming out. Unfortunately, we can’t share too much of what’s coming out until it’s actually out, so I would say definitely keep an eye on our blogs and I have Twitter and all of that because we definitely like to promote what we’ve been working on. And whenever there’s a new version, we definitely like to get it out there. I know we’ve completely run over time. I think we’ve answered most people’s questions. Rob, did you want to try to show what you were trying to when the demo gods were not with us, or?

Rob Marcer: 01:08:25.113 Yeah. I’m almost interested to see if it can go wrong again. So absolutely, let’s give it a go.

Jay Clifford: 01:08:31.411 I stopped sharing.

Rob Marcer: 01:08:32.980 Yeah. Thank you. Okay. So I’m going to share my screen, and I’m going to share my desktop. Okay.

Caitlin Croft: 01:08:41.334 I love how many people are still staying on with us even though we’re like 12 minutes over, so really appreciate it.

Rob Marcer: 01:08:46.793 Yeah. Thanks everybody. Okay. So I’m just going to close a couple things down. Okay. So I should now, hopefully, be able to do. So I’ve come back to the device management within FlowForge here. So this is why I was clicking buttons earlier on and nothing useful was happening. So what I’ve done is — I’ve put my device here into developer mode, which allows me to remotely access it. I’m just going to try clicking this. I’m going to see if it works. It is working, yes. So as much as, yes, this is just this Pi that’s right next to me, it’s like, well, so what’s so cool about that? It could be a device anywhere else on the internet. And what’s great about doing the development in this way, where you have lots of devices with similar sensors, is I can actually be interacting with the sensor data locally as it is for that device rather than trying to think, how would that sensor respond, and perhaps coming up with fake payloads to test the flow, I can actually see the data flowing through. So within this flow, what I have is this. I mean, it’s a beautiful user interface. I’m sure everyone will agree.

Rob Marcer: 01:10:01.923 You can see that I am very much focused on UX design. But I mentioned this earlier, what this is doing is — so this is running, so this is my local network here and this is the IP address of this Pi 10.0.0.10, and then 1880 is the standard ports for Node-RED. So what I can do here, even if the central system isn’t up and running, if, yeah, say we’re doing maintenance or changing something, and I’m just like, “Oh, are my sensors working?” So I can press here and what that’s done is just read from these sensors and outputs the values onto screen. And since we’ve been overrun by time, I guess that if I take the moisture sensor and dump it into a glass of water, what do we think chances of this working? I’m a bit nervous. Yay. 100% [crosstalk] of how moist we are, so. But as I’ve mentioned in the tips, this is a really good way. And I will pretty much always do this, when you’re working with sensors, developing — and you can see if you download the flows from where Jay shared them in the presentation, developing some sort of basic user interface like this is great for when you’re developing. It’s great for if this is on a shop floor somewhere because it means the engineers, potentially, can go over to a particular device and maybe they could access this on a tablet device or a phone and just go, “Yeah, what is actually going on with that local device?”

Rob Marcer: 01:11:27.070 So yeah, it’s a much better way to interact with the devices. And then once if I make changes to these flows, I won’t show this now because I’m very over, but I can take snapshots of this and push it through to Jay’s device. So if I realize that actually we need to change the way we’re dealing with the moisture sensor, it’s very easy for me to make that change, save it to FlowForge, and then deploy it back to 2 or 2,000 devices. So thanks for sticking around to watch that, everyone. I’m sorry it didn’t work the first time.

Jay Clifford: 01:12:03.154 Love it. What a great way to end it as well, all working.

Caitlin Croft: 01:12:06.862 And on a high note. And everyone was super understanding of having connectivity issues. So, hey, it happens. Just give us one second here. All right. Cool. So I think we’ve answered most people’s questions. I think there were a lot of follow-ups, so maybe keep an eye out on our blog. I definitely will be writing some blog content on this as well. Thank you Rob and Jay for this fantastic presentation. I think just based on the amount of questions that we received, there is still lots of questions and people are just still learning lots about the MING stack. So really appreciate you guys presenting today. Thank you to everyone who stuck with us, and I hope everyone has a great day.

Jay Clifford: 01:13:05.780 Thank you for hosting us, Caitlin.

Caitlin Croft: 01:13:08.228 Talk to you later.

Rob Marcer: 01:13:08.536 Yes. Thank you.

Caitlin Croft: 01:13:09.172 Thank you. Bye.

Rob Marcer: 01:13:10.499 Bye.

Jay Clifford: 01:13:10.869 Bye.

[/et_pb_toggle]