eSoftLink and Real-Time Energy Management with InfluxDB Enterprise

Session date: Oct 29, 2019 08:00am (Pacific Time)

Smart metering is one of the first steps to address challenges surrounding energy consumption and water usage. By providing real-time data on electricity and gas usage, smart meters allow consumers to make smarter decisions about their energy usage. In this webinar, Samuel Chevrier, Engineer at eSoftThings will share how you can use the eSoftLink platform to transform a smart meter into a real-time feedback tool for consumers. He will show how they are able to collect a large volume of time series data with their implementation of InfluxDB and Telegraf that provides value-added services while keeping in compliance with legislation like GDPR. He will also show a demo of the solution.

Watch the Webinar

Watch the webinar “eSoftLink and Real-Time Energy Management with InfluxDB Enterprise” by filling out the form and clicking on the download button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “eSoftLink and Real-Time Energy Management with InfluxDB Enterprise”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• Chris Churilo: Director Product Marketing, InfluxData
• Samuel Chevrier: Technical Marketing Engineer, eSoftThings

Chris Churilo: 00:00:00.000 All right then. Let’s get started then. Thank you so much for joining us everybody and today we have a really great webinar covering an IoT use case. And we have my friend Samuel who will be doing his presentation today. As I mentioned, we are recording this session. So if you are interested in listening to it again, you will be able to once we do a real quick edit and we do the upload. And if you have any questions please feel free to post them in the Q&A in the chat panel at the bottom of your Zoom application. All right. With that, I will let you take it away.

Samuel Chevrier: 00:00:33.799 Okay. Thanks a lot. Thanks a lot for having me to do this webinar. I’m Samuel Chevrier from eSoftThings. I’m working as a technical marketing in eSoftThings and I will present to you today eSoftLink, let’s say, the IoT platform and real-time energy management with InfluxDB. So what we will explain today, the agenda for the webinar. So first I will give you a short overview of eSoftThings, the company: who we are, where we are, what we are doing; then, what is the smart metering network and what are the limitations; how to transform a smart meter into an IoT node. This is what we’ve done for a lead customer, Total Direct Énergie. Then I will explain a bit more what is eSoftLink, so the IoT platform, and what is the feature list. Small talk about GDPR which is a European regulation about data privacy and the restrictions. Then the eSoftLink architecture overview, so what are the block within eSoftLink platform. A little highlight on the volume of data that we exchange daily on the platform and the tests we’ve done to ensure that the architecture and the technology are good enough. And we will end by a demonstration.

Samuel Chevrier: 00:02:10.133 Okay. Let’s start the webinar. First, about eSoftThings. So we are a tech company that is solution-focused. We were founded in 2014, so five years. We are based on Rennes in France. This is the west part of France in Brittany. We are one hour from Paris by plane and one hour and fifteen minutes, thirty minutes by high-speed train. Our customers are located in Europe, North America, Asia as well. And we have a strong experience from telecom and semiconductors industry. So some of us are coming from STMicro, Renesas, which were located in Rennes before as well. We have a team of 50 engineers today, more or less, that are covering hardware and software activities. And for software we are covering the embedded part, either it is bare metal Linux, the mobile application as well, the application, the website and the backend software of course.

Samuel Chevrier: 00:03:22.784 We do have two business units. One is called Autonomous Vehicles, so those cars they are dealing with algorithm for the vehicle about pedestrian detection, driver monitoring system. So being sure that those algorithm are well integrated into the car working with high efficiency for CPU and power. And us, I belong to the Smart Devices business unit that has two main activities. One related to audio acoustics, so we do have a audio lab and acoustic experts and we are doing [inaudible] design. We are helping people to integrate the voice recognition, local voice or Amazon Alexa voice services. So everything that is related to audio in terms of performance and software integration as well. And the other part is IoT, so Internet of Things, where strong focus is put on the eSoftLink IoT platform that we offer to our customers when they want to recover all the data from an existing [feeder?] device or if they want to create a new IoT product. So two business units, Smart Devices for IoT and audio and Autonomous Vehicles.

Samuel Chevrier: 00:05:01.561 So first, in order for you to understand why we transform the smart meter in an IoT system, I explain before, what is a smart metering system, a little more about the network and its limitation. So first, why people are implementing, deploying smart metering. So it comes from the European directive from 2012. So the European Union says that there must be a save of 20% of the energy from 2012 to 2020. So first things that you need to do in order to save energy is first to assess how much you consume and have a good idea of the production versus the consumption along the [base?]. Okay. So they created, for example in France, what we call the Linky program. So Linky is the green smart meter that is on the left side of the picture. Linky is connected through Power Line Communication to a concentrator. Okay. So there are a bunch of smart meter connected to the concentrator and the concentrator upload the data he collect from all the meter in GPRS to the information system.

Samuel Chevrier: 00:06:27.804 Regarding the data flow, so the smart meter is uploading its consumption information only once a day because the PLC tunnel is quite small. The data rate is limited. So only once a day and with a 30 minute granularity. Then the utility offer web portal to the subscribers, to its own customer, to follow their energy consumption on a yearly, monthly, daily basis. So it’s quite convenient if you want to check how looks the progression of your energy consumption between summer and winter, for example, and month by month. But it’s really difficult for the end user to identify the devices that are highest contributors in the overall consumption. Okay. So if I tell you, “Look, you should decrease your power consumption,” you want to be able to plug some devices and check in real-time what is the biggest increase and which device is working properly, the one that might have some failure and so on.

Samuel Chevrier: 00:07:45.230 Also there was some important study that have been conducted by national institute or university like Oxford, even the American Council for an Energy-Efficient Economy and all the study are demonstrating the same thing that the most real-time is the feedback tool, the biggest percentage of energy is saved at the end of the day by the consumer. Okay. So if you have only Enhanced Billing, so it’s going to be only 3.8% that you will save at the end of the day. But if the utility is providing a tool to help you having some deep dive into your energy consumption, into the consumption curve, with a real-time plus feedback tool, then you’ll be able to save from 12 to 23 percent of energy. So the tool to put in place is really key for people to manage their own energy consumption and their own energy bill as well.

Samuel Chevrier: 00:08:55.793 Then we have smart metering system. We say that it’s important to get real-time feedback tool, so why don’t we use the smart metering existing system and transform it into a real-time tool? This is what we did for our lead customer Total Direct Énergie. So we took the smart meter and we use one available interface slot that is existing on the smart meter. We created, design an IoT module that we plug into this interface. The IoT module is connected through Wi-Fi to the internet thanks to the home set-top box and is pushing its own data every second to the IoT platform that we design, that we created, that is called eSoftLink. So we reused the measurement capability of the smart meter since it’s already certified, it’s already well-designed and we used its measurement capability and just get all the register and send it to the IoT platform, eSoftLink, every second so that the people can view and monitor their energy easily.

Samuel Chevrier: 00:10:26.622 So what is eSoftLink platform? The main idea of this platform is to offer data collection. So of course, this is the main idea, to collect all the data coming from all the IoT nodes connected to the smart meter. Offering data visualization and there is two profile here. Data visualization for people on their smart phone to monitor the energy, but also for the utility to manage the fleet of devices that are working and connected to the platform. Thanks to Influx we are doing data storage. We are running some analytics. I will come back on this after. More recently, we added smart contract as well. So more and more into the network topology you find some micro-grade network with producers and consumers. So there are producers that want to sell directly their energy to some consumer connected in the same urban zone. Okay. So we created smart contract thanks to blockchain. So it’s an application where people can come, check about the contract that are available, the offer that are available from producers and close a deal with the producer if the price is okay, if the amount of energy is okay. Of course, the platform is able to do device management, which means knowing which device is connecting, which device is active, inactive, what is the firmware running on each device, if the Wi-Fi connection, Wi-Fi RSSI is good enough, and so on.

Samuel Chevrier: 00:12:29.220 A little bit more about the eSoftLink architecture. So there is three main pillars in this architecture. First one is the IoT node. So in a generic way the IoT node can be a sensor or an actuator. It can exist already or you come to us and ask for specific design. In the middle you have eSoftLink, which is the backend platform, with some administration panels to manage data securely. The data can be hosted in customer premises on your own Linux server or it can cloud-based solution as well using Amazon, Azure, Google, OVH, whatever. We do have a DM, the end user frontend, which can be mobile application for end user or web application for the solution owner. So designed by the customer or directly by us.

Samuel Chevrier: 00:13:36.344 The key feature of the solution? So we want it scalable so it’s important that we can match any customer request. So there are customer that wants to get one message per second but other wants to get maybe one or two message a day. So it was an important point to make it scalable. We wanted to get some real-time monitoring and alerting capabilities either by email or with the application using iOS and Android free notifications. We wanted to make very flexible so you can have it, I said it already, on the edge, on the commercial cloud or private cloud. We need to manage the security, so having a connection between the IoT node and the IoT platform on the internet that is secure. So we are using HTTPS protocol and we are, of course, GDPR compliant. I will come back on what has been implemented in the next slide for GDPR. We integrated some artificial intelligence as well, so analytics. I will come back on this point at the end also and secure connection to data lake. We do device management with remote debug logs, firmware updates, auto-diagnostic that can be done by the IoT node itself and pushed to the eSoftLink platform. And we do have administration console with customizable panels and interfaces.

Samuel Chevrier: 00:15:19.325 A word now on GDPR. So maybe you don’t know, but in European, Chris told me in [inaudible] as well, and it should be the case in some other country, there are some data regulation or general data protection regulation for data privacy. So you no longer allowed to do whatever you want with data you collect from people and you need to take care about this. There are a few features that we implemented and that had an impact on the architecture, on the design within the database. So first, the right to be forgotten. So if someone is asking you must be able to delete all the data from the database that are linked to this person. So it’s very important. So you need to do it in the active database, in the backup, everywhere. The right to consult. Anyone is able to ask for download of all the data related to his own account. So it goes together with data portability.

Samuel Chevrier: 00:16:33.752 Data portability. You are no longer lock on a solution and you might want to retrieve all your personal data in any format so that you can switch from one provider to another. So we are doing with Influx with some requests in CSV to the database. Data log access. You need to monitor each access to any attribute of the database and log who access those information. So you need to manage and to regulate the data access so that not all the people can see all the data. And anonymization. You should not be able to link the personal that are stored into the databases. So the personal data should be anonymized so that you cannot link the data with the ID or with a given profile. So this was a bit specific and this needs really to be taken into account at the beginning of the design because those feature are all along the design in different pieces of the architecture and it was key for us to support this and we must support this by law.

Samuel Chevrier: 00:18:09.881 Then a little more about the blocks within eSoftLink architecture. So we have again the IoT node on the left that has two different things, so one dedicated to FOTA and the other one that is secured to the platform. So within eSoftLink platform you will see which kind of technology we implemented inside but you find that the message broker, that is homemade by us, that will get all the messages coming from all the node. We do have a device management block so that you are able to do some firmware upgrade on any of the device that is connected to the platform or even check its connection status or its firmware version and so on. We have two kind of database. We have the InfluxDB database, so time series database, to manage the high volume of data coming from the node. And we have as well the Postgres relational database, more for profiling configuration of each [inaudible]. We have analytic blocks with prediction capabilities. So if we take the power consumption use case for the prediction, so we connect the platform to weather forecast system. And so with the forecast information and the history of your consumption we are able to predict to you what will look like the power consumption tomorrow and the week after.

Samuel Chevrier: 00:20:03.371 We have non-intrusive load monitoring capabilities. So any device is having power signature and the idea is that we can, with AI, we can detect the power signature of type of devices and then classify the power consumption per device’s classes. We do also clustering for own comparison. For example, still in the power or energy use case, so that you are able to know and to understand your own power consumption in your house and compare it to similar houses with similar occupants in terms of numbers and check if you have some issue with your house or if you are in the average.

Samuel Chevrier: 00:21:03.966 And we do manage some notification as well, so if people again want to get notified for whatever threshold or whatever data this is possible. There is a dedicated alerting block. So this alerting block you can configure it on your own and you can put some threshold for any data that you collect from the node. We have an administration and web panel, web access, for either the mobile application, the end user, or the administration frontend. And we have some non-mandatory interfaces that are good to know that it exists. There are customer data lake output, so data are key today and it’s important for the solution owner to retrieve all the data and being able to compute whatever needs to be at the end of the day. And we do have also the capability to link the system with some other customer databases.

Samuel Chevrier: 00:22:18.115 So what are the technology that we used for our eSoftLink IoT solution? I told you the message broker is something that we’ve done on our own for many reasons. And the best one is that we can manage things more easily when we create it on your own. And then we tried to choose the right solution where it make sense so we keep the Apache Zookeeper, we keep the Apache Kafka we have the Telegraf plus InfluxDB for the time series databases. And we do Chart.js with PHP to build the graphics that you will see after. So it’s quite straight forward architecture and blocks that you can find in most of the solution today and it makes sense for us to use those solutions in eSoftLink.

Samuel Chevrier: 00:23:36.809 So why Influx now? We are collecting data 24 hours, 7 a week at one second rate with a huge number of devices, so it lead us to time series databases, for sure, and we wanted a very strong solution that can support this data volume. We need it to be scalable because we want to face different type of customer and so the enterprise edition was a must for this reason. We need a popular solution which will guarantee to find easily documentation, guidance from the community but also long-term support thanks to the popularity. Influx is for sure very popular. It has 17,000 stars on GitHub. And thanks to all of this we pick up the InfluxDB and then we did some tests to ensure that the solution fits well into the system.

Samuel Chevrier: 00:24:52.258 So we did some load tests that we’ve done. Yeah. We’ve done load tests to record the activity of the server in the following situations. So we had 48,000 users that push one data every second. So one data every second per user. And there were two tags per data to write in the databases. What we did is that we recorded the Telegraf and the Influx KPI’s, let’s say. So first about Telegraf. We had six instances in total for this split in two, so three per server. The average of CPU usage per server was around 63%, so it was quite flat, no big increase of the CPU. But the same activity for all the test period. Then regarding the traffic and the amount of data that we exchange, there was a mean of 460 Mbps, in and out messages, aggregated. This is very flat, again. The memory consumed by Telegraf was 63% of the server RAM, 4% of the cache and 0% swap.

Samuel Chevrier: 00:26:34.716 Now about the Influx itself, the database. We had two instances of the database on two different servers, and the mean for Influx was 43% of CPU used. There was 360 Mbps of traffic on both servers with 58% of RAMs, 33% cache. So with these load tests, we guarantee that the InfluxDB solution was the right to integrate and that the configuration that we put in place was the right one. Now I will give you overview about the two visualization that you can have or that the customer from Direct Énergie Total has today and the administration panel as well. So first about the mobile application that we did.

Samuel Chevrier: 00:27:47.437 Here is an overview of the application. So first window is kind of what we called here expert mode. So you have four tiles on the window that reminds you the main items. So the consumption information, instantaneous power, the consumption in euro and price that is active at a given time. Just after you have the main menu, let’s say, with some profile information, some tips that you can visualize, the configuration, the reference consumption and the entry point to manage your alerts, your own comparator tool and so on. Then the third window is about the graph that you can have either in euro, in kilowatt or - I don’t remember, there is one more. It’s a graph that you can have on 24 hours, 7 days, 30 days or 12 months, so that you can have a deep look on your own consumption. And you can also configure and manage your alerts in the last window. So you can get an alert if your power is going above a given threshold. If you are very near the Fuse Limit, which is very useful. So there are many more alerts that you can configure.

Samuel Chevrier: 00:29:31.559 And here is more information about the mobile app. So this is the own comparison tool which tells you the consumption in regard of similar house that you can find. Then you have your consumption in real-time. So the power, the amount of money and the pricing that is ongoing. That’s it for the mobile application. It’s very well rated on the Android market and people are very happy about this application. So next I will present the backend or let’s say the administration panel that Direct Énergie is using to visualize all the data. So I will switch from one PC to the other.

[silence]

Samuel Chevrier: 00:30:56.210 Okay.

Nicolas Guilbaud: 00:30:56.642 [inaudible].

Samuel Chevrier: 00:30:58.022 Huh?

Nicolas Guilbaud: 00:30:59.344 [inaudible].

Samuel Chevrier: 00:30:59.837 If you can just give me one second, I will switch and display the demo.

Chris Churilo: 00:31:09.819 No problem.

Samuel Chevrier: 00:31:11.490 Thank you, Chris.

[silence]

Samuel Chevrier: 00:31:26.486 [inaudible].

Nicolas Guilbaud: 00:31:28.777 [inaudible].

[silence]

Chris Churilo: 00:32:42.303 Looks like we can see the user management UI.

Samuel Chevrier: 00:32:45.715 Okay. Very good. So here is the administration panel that we offer to the solution owner. So you see first this dashboard where you can see the main information. So the number of active session that is ongoing, the connection per day, the total energy that has been accumulated the day before, the instantaneous power, the number of user accounts that has been created and then there is the unique connections per day, and the average time per session. Then we have some nice figures for you to compute some statistics. So there is the number of ATOMES that are connected, the number that are disconnected, the type of OS used by the end user. So you have the Android percentage and the iOS percentage. The evolution of connected ATOME so it’s a good provision here. And unique connection of the last 10 days. So you have two different colors, so the one that are in the eligibility process, the one that are currently installing the solution and the one for those people that are actually using the app. Okay.

Samuel Chevrier: 00:34:40.150 And then you have some more nice figures about firmware version of the IoT node. So it’s supposed to be the latest one. Then there is all the new ATOME that has been installed. The firmware version or the app version for the iOS user and the app version for the Android user. So it’s nice for you to administrate, to do some maintenance on the system to ensure that the firmware upgrade has been well done and the number of devices that need to be updated still. And then we have the user database that you can consult, and you consult any user ID. So if I click on Luna Adalyn, I see that the IoT node has been paired and connected. It’s plugged into the smart meter. It has been updated for the firmware, the 22nd of October. You have many information plus the serial number and you have some detail as well on the eligibility phase and process, how it went well and what was the issue the customer had. Then you have some more information about the phone application, what is the type of the phone model, the application version. So that’s it.

Samuel Chevrier: 00:36:32.795 Some more information about the consumption. You have here many figures and many graph that you can see. So you can see the power curve, energy, you can also select the period, you can check the amount of the spending, the evolution of the consumption, so compare to last month for example. The predictions that we are doing thanks to weather forecast connections. Data completion is more for debug and own comparator tool, which gives you - the orange line is the comparison with the cluster, the rest of the people that are similar to you. Some more information about the meter with some idea of the meter. Notifications or what type of alert and notification did the user select and configure in his application. Then notification status, okay, so which is key for [inaudible] as well. So it’s very good to know that the Bluetooth connection has been disconnected, received invalid data and so on. So if customer is pushing a ticket to you for maintenance it will be key for you to know what was the latest on the key that side. And you have some personal information as well.

Samuel Chevrier: 00:38:35.598 There is OTA management menu, so here you can see and you can post the latest firmware that you want to push on the device. So in our implementation, the device - or this implementation, the device will check on its own, once a day, if there is firmware that is more recent than the one currently running. So he will check this thanks to eSoftLink platform and then retrieve the firmware if the one is newer thanks to the firmware version. You can manage it by firmware version, by other version as well, you can enable or disable some firmware, so it’s quite a flexible tool. And you have some information about the account itself. And I think I’m done for the demo and for the webinar. I hope I was clear enough. Excuse my English if something was not clear.

Chris Churilo: 00:39:58.020 No. It was really great. No it was really, really good.

Samuel Chevrier: 00:39:59.809 Okay. If there are some question.

Chris Churilo: 00:40:01.915 Yeah. In fact I think I’m most impressed with two things. I mean, I think the consumer application as well as the provider application really make them both very powerful with all the information that you’re providing them. From the consumer perspective, I mean, being able to have that real-time information about how they’re consuming energy really allows them to know that they can make a difference by turning off the TV or turning off the computer. They could see instantaneously what kind of a change that they can impact, which is always I think the most motivating way to change behavior. And then looking at this provider UI, I mean, they must feel very powerful because they can really help to make sure they can service and support their consumers. I mean there’s so much information here, so much useful information, so I think kudos to you and the team. This is a fabulous overall service that you’ve built.

Samuel Chevrier: 00:41:00.653 Thank you, Chris.

Chris Churilo: 00:41:01.598 And I think the other thing that kind of comes to light for me is that I can see why you guys are very concerned about GDPR, because in order to provide such power you do need to have a lot of personal information. Which on one side is required and then the other side it’s a little scary, you’re like, “Hey, I definitely want to be able to hide this or remove this depending on the needs of the consumer.” So maybe you can talk a little bit about the retention policies that you have set in place. How long do you keep that time series data? Because I imagine, from both sides, there’s probably some need to have some historical data. Obviously the instantaneous data is really important. Maybe you can just talk a little about that.

Samuel Chevrier: 00:41:47.419 Okay. So let me hand over to Nicola. He’s with me in this webinar and he knows a lot more. So he’s the CTO of the company and he was managing the whole implementation on this [inaudible]. He will answer.

Nicolas Guilbaud: 00:42:04.938 Yeah. So, of course, we have applied some retention policy because you can imagine that the amount of data is very huge. We are sampling energy every one second so if we keep all the data in the database we should have very, very huge data so what we are doing today is we applying several policy for retention. So we have first one every day, so every day we are doing some 10-minute average and then after one month we are as well doing some retention as well. And anyway because of GPRD policy we can’t keep personal data for a long time, so after one year we just remove all the data and we are applying another retention policy to just anonymize the data and doing some mean with some cluster. So we are grouping together several customer and doing some mean and square mean and all this statistics stuff to keep some information but of course it’s anonymized. So this is basically retention. And we are using Influx retention tool to do that, which is very, very easy tool to apply this kind of retention. So very useful for GPRD stuff.

Chris Churilo: 00:43:35.995 That’s cool. Any other kind of forecasting tools that you might be using for both the consumer UI or the service provider UI?

Nicolas Guilbaud: 00:43:45.780 So other tools. Yeah, we are using I think probably you will see that before, we are using some Chart.js to print the different tool. Samuel as well mentioned some of the tools like Telegraf as well, we are using inside the solution.

Chris Churilo: 00:44:11.824 Very cool. If there are any other questions please feel free to put them in the chat or the Q&A button at the bottom of your Zoom application. As I mentioned this session is being recorded so you can take another listen to it. The other thing that I - what are some other things that you guys are planning for maybe the next iteration?

Samuel Chevrier: 00:44:33.514 So for the next innovation, I talked about it. So one very recent [inaudible] that we added is the blockchain with the capabilities to do smart contract. So we see this request that is coming more and more often because we have now some [inaudible] capabilities with some that are producing energy and want to sell their energy locally to people that are in the neighborhood. So by creating a smart contract application we offer people a place to meet so that they can negotiate a contract all together and making sure that the renewable energy is well used within the local area. So it’s the first thing that we worked on very recently. Another point is about [inaudible]. So I think capabilities - at the beginning of the presentation I will show some study for real-time feedback tool and the cherry on the cake for those tool will be to identify in the power consumption what are the main contributors thanks to their power signature. So it’s very nice achievement that we are doing today is trying to classify the big contributors of your bill so that you can check where is the effort to be done if you want to decrease the bill at the end of the day. So this is the two major steps that are ongoing that we are deploying and improving today.

Chris Churilo: 00:46:30.670 Cool. Awesome. Great job and thank you very much.

Samuel Chevrier: 00:46:36.718 Thank you, Chris.

[/et_pb_toggle]