Distributed Performance Analysis Using InfluxDB and the Linux eBPF Virtual Machine
Session date: Oct 30, 2018 08:00am (Pacific Time)
Since the release of the Linux kernel 4.x series, a lot of enhancements have reached mainline to the eBPF ecosystem giving us the capability to do a lot more than just network stuff. The purpose of this talk is to provide an initial understanding on what eBPF programs are and how to hook the output of eBPF programs to InfluxDB in order to answer targeted questions at the cluster level and very specific fine-grained situations happening in our programs like:
- Has that function in my program been called?
- For a given function, which arguments have been passed to it? And what it did return?
- Which TCP packets are being retransmitted?
- What are the queries running slow?
- Insights on programming language events/gc
- Has that file been opened?
Watch the Webinar
Watch the webinar “Distributed Performance Analysis Using InfluxDB and the Linux eBPF Virtual Machine” by filling out the form and clicking on the download button on the right. This will open the recording.
[et_pb_toggle _builder_version="3.17.6" title="Transcript" title_font_size="26" border_width_all="0px" border_width_bottom="1px" module_class="transcript-toggle" closed_toggle_background_color="rgba(255,255,255,0)"]
Here is an unedited transcript of the webinar “Distributed Performance Analysis Using InfluxDB and the Linux eBPF Virtual Machine”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.
Speakers:
- Chris Churilo: Director Product Marketing, InfluxData
- Lorenzo Fontana: SRE, InfluxData
Chris Churilo 00:00:00.774 All right. Good morning, everybody. Thanks for joining us today. Today we have Lorenzo who's going to share his screen, who's going to be talking about the work that he's actually been doing. Lorenzo is actually part of our SRE team for Influx Cloud, so he's going to be talking about how he's been using InfluxDB and really trying to make sure that he can keep our Influx Cloud servers performant. And I'm going to go ahead and let him take it away. Do you want to share your screen real quick?
Lorenzo Fontana 00:00:35.411 Yeah.
Chris Churilo 00:00:35.951 Excellent. Very good. All right. Let's go.
Lorenzo Fontana 00:00:37.838 Okay. Thanks everybody for joining. Today we are going to talk about how we are using-well this is not an official InfluxDB product or something like, but we had the necessity to monitor our systems for cloud in a way that go more deeper inside the other resources, and now the system where being used by our users. To do that we started by looking at different sources. We have many different kind of monitoring in place like starting from Telegraf to tracing to monitoring ingresses, monitoring [inaudible] resources, monitoring every kind of service we could monitor in user space, but we also had the need to go a bit deeper inside how the kernel works and in a way that we can see what's happening in kernel space for our processes. To do that, there are a few different alternatives. The one that we choose because it was most flexible was by using eBPF programs. Let's see what they are.
Lorenzo Fontana 00:01:58.803 eBPF stands for extended Berkeley Packet Filter. Extended because Berkeley Packet Filters are something that is already in a lot of Linux distribution. It is something that was born to allow people understanding what's happening around packets like, "Are those packets being retransmitted? Are those packets arriving?" Things like that. Extended because the API has been extended to support a lot of different things. So probably packet is not anymore the right definition but now you can do whatever you want to do.
Lorenzo Fontana 00:02:41.829 People will define BPF as a tracing framework nowadays. As the title says, it's used to access kernel trace, backend implementation tools, but it's not just like that. The way we are seeing it now is that way but with BPFs you can actually instrument how the kernel works. For example, with XDP which is a framework built on top of BPF you can do packet filtering, packet mangling, create firewalls, things like that. Now with BPF too you can actually define how a component works inside your machine. So it's not just tracing. We are using that for tracing because it lets us access specific information about how a function is being called, what it returns, etc.
Lorenzo Fontana 00:03:39.964 BPF programs can be used to access a few different kinds of kernel subsystems as written here. Those are to access kernel but can access trace backend instrumentation tools. Among these tools there are the static trace points or trace [inaudible] in the kernel that are [inaudible] there. They are defined by the kernel maintainers and you can just look at them to extract your information. For example, with tcp there's a retransmit handler that you can-well you can see if your packets are being retransmitted. That can be useful, for example, to see things like if you are being [SIMM?] flooded by someone or identifying DDOS attacks. But the one that are real interesting are the dynamic trace functionalities like uprobes and kprobes. Uprobe stands for user probes in user space and kprobes are kind of [inaudible]-with uprobes you can access the information in your programs running in user space. For example if you write a Go program and you want to see if the function is called, you can use a uprobe, while if you want to see if a kernel structure has been called, you use a kprobe.
Lorenzo Fontana 00:05:04.536 BPF programs are-well, this needs some context. The kernel allows you to load your own programs in different ways. The first way is like just write your own program in the kernel source tree, load it, and it will work. [inaudible] recompile the kernel and load the ukernel [inaudible] well [inaudible]. The second way will be by using net filter. So you just use net filter socket and you load your programs in it, but this is just for networking and you cannot really load whatever you want; you have to load what you're allowed to load by your net filter plugin. Another way to instrument the kernel can be with syscalls. So you just call a syscall, like when you open a file, you call a syscall. So you go kernel space [inaudible] open this file and it will open, but you cannot really write your own programs that way. Or you can write a kernel module and, well, you write the module. So you are actually hooking some code in the kernel and that has the same disadvantages of recompiling the kernel [inaudible] for example.
Lorenzo Fontana 00:06:19.596 In eBPF programs, there is some kind of tradeoff between writing a module and having your own program running in the kernel. The difference is that eBPF programs are inherently less powerful because they cannot be whatever they want. For example, as the master's [inaudible] says, an eBPF program can be Turing complete. This means that you cannot do things like for loops. Well you can do for loops for things that can be established at the compile time. You cannot do for loops that are dynamic. It cannot run forever, for example, because the static verifier will identify that and block your program. So the way an eBPF program-this is like the only tradeoff I see for an eBPF program. It cannot be whatever it want but it can still be very powerful.
Lorenzo Fontana 00:07:20.046 So from the right your user program-we will see an example after - creates some [inaudible] BPF bytecode that's usually compiled from some C code, and it loads it with the BPF_PROG_LOAD argument to the BPF syscall. If you're on a Linux machine you can do man BPF and see all the things I'm saying because it's actually probably in your kernel if you're on 4.8 [inaudible] 4.8 and you will see that. When you load the program, the kernel will say, "Well, let's see if this program can be loaded" and will pass the control to the static verifier that loads a program inside the BPF. Virtual machine is called like that because BPF has its own instruction set. We will see that later. And after loading that it decides if the program being loaded is one of the allowed programs. [inaudible] socket filter or XTP, and then if your program has any BPF map in it, it will give you the result back. So your program is being loaded from user space to kernel space. When it's in kernel space it executes, and while it executes it can asynchronously update your program in user space to give back the result. It's just if you think about this this is very interesting to do like [inaudible] inside that. It's also interesting to do programs that interact with the kernel via [inaudible].
Lorenzo Fontana 00:09:12.732 Since this might be a bit confusing I will give some real-world examples. So [inaudible] programs [inaudible] used for. For example, everyone knows TCPDUMP. If you are programmer or assistant administrator you probably use it in your life. TCPDUMP is the program that lets you dump specific packets while they are flowing in your machine. And to TCPDUMP you usually pass a program with the expression it tells to use to filter your packets. Like in this case, let's filter only PV4 packets and that are on TCP and that are on Port 80. So this is probably going to be [inaudible] packet. If you pass d to TCPDUMP it will dump the program it loads. This means this is not a BPF program. This is the bytecode for a BPF program. This is its instruction set. This is the instruction set that is loaded to the starting verifier to be executed by the BPF virtual machine. TCPDUMP is like this-I was going to say, has been like this for a few years. It wasn't born like this but in this way-this is very useful for people that write BPF programs because you can-if you are doing packet network stuff you can just write your TCPDUMP rule and get the BPF program out of it and understand [inaudible]-in fact, in this case this section says, "Is this an alternative PV4 packet?" Its offset 14 means Port 80 and this is the same source and the destination. There is a type of [inaudible] destination.
Lorenzo Fontana 00:11:17.725 Another real-world example is seccomp. Seccomp is if you use Docker, you probably know it. Seccomp is a Linux security module that allows you to stop specific syscalls from running in your kernel, so you can basically say, "I want this process from now on. Stop using like in this case the dup2 syscall. With Docker it's used very heavily to disallow you to get [inaudible] from inside your containers. This is one of the components that made the container [inaudible] basics. More practical examples of what you can do with it, like trace file always by file name, trace queries [inaudible] again. So database like Influx or [inaudible]. Trace [inaudible] transmissions or trace done in a batch shell, we will have this for the [inaudible]. Trace blocking your devices [inaudible] duplicate data flows, [inaudible] events, [inaudible] current time events. You can write debuggers with it or firewalls.
Lorenzo Fontana 00:12:45.906 [inaudible] are some interesting project that uses BPFs to [inaudible] the results. [inaudible] BCC is basically a framework that is based on top of the BPF to allow you loading your program in an easier way. It has a library that calls lead bcc that allows you to compile programs, load them without the need to understand how to compile them with Clank and without needing you to write your own [inaudible] to get the data back. [inaudible] means a network solution for used for containers that is aware of what's happening inside the wire by using the BPF programs, in this case XTP programs. And Gobpf is a library based on bcc that allows you to write your own BPF programs, to load your own [inaudible] programs using [inaudible], and then there are other Linux security modules other than seccomp gives you maybe just like Landlock.
Lorenzo Fontana 00:14:05.849 Just for to be more complete Iovisor bcc lets you do things like that, for example. You use the trace to on this specific [inaudible] so that it knows where it took the function, and then you set the user probe on this specific function, and this is the format it will use to dump your arguments, and these are the arguments you're interested in. So every time that function will be called you will see the arguments numbered down, and this is like the most basic use of a uprobe, but it really shows you that you can basically write something like a debugger for that. Let's imagine a program that you're deploying in production and you want to know if a function is being called or of it's died. Or let's say that you want to byte trace the request coming from a user. You can specify all the function you want to see if [inaudible] are called and see all your arguments.
Lorenzo Fontana 00:15:15.766 That was a very nice example combining-we have seen the first graph showing that your program can load the BPF program to the kernel. To get back the results we haven't seen yet how to use those. One possible use could be to send the results back to InfluxDB. So we will see this in the [inaudible] show you in a second. You have a main Go program that loads a BPF bytecode into static verifier, then there's a specific uprobe that you [inaudible] in the example I prepared that populates a BPF map with BPF [inaudible] output and get the results back with a [inaudible] local [inaudible] and sends everything to an InfluxDB. If you load that program, the part on the right inside [inaudible] set you will see that now you have that program loaded in all the kernels of your [inaudible] clusters. So you have a way to load programs inside the kernel and [inaudible] Influx without using anything else and looking from the kernel without anything else [inaudible] program.
Lorenzo Fontana 00:16:51.102 I mentioned [inaudible] probe, but this is the end results, so the idea is that we get...
Lorenzo Fontana 00:17:14.990 So this is the representation of what we have in this light. Basically what happens here is that we have these-if you have program written in C that defines the get_return_value uprobe, this uprobe is simply loading the function that we pass to it. That is a red line in this case because we want to see all the readline [inaudible] Readline. Readline is a library that is used by bash, for example, to get all the commands you run on it. So whenever you run a command in bash, it will be first processed by Readline so that it can tell bash the command you want to run. So by loading this eBPF program you will basically see all of the commands loaded to bash. And another thing we have here is to get the current process ID so that we can associate the process ID with the command that has been run. And here's the [inaudible] data structure that contains our process ID and the command and that has accompanying data structure so that we can serialize the results back. This is all boilerplate to run the program again to write InfluxDB with using the right library to load the uprobe program on it. And basically what we are using here is Gobpf-that's the library I talked to you about before - and with Gobpf we load the uprobe get_return_value that is still this one and the one defined here. And the uprobe is attaching to the binary. Binary name you can to the kprobe to a binary like [inaudible] bash in this case or another shell implementation or whatever program that's used. That's Readline in this case. Or specifically to a library. If you want to see all the code to Readline regardless of the binary you can just attach it to the library. The function is Readline. This is the uprobe that we loaded here, and this minus one means that we are attaching to every process ID because we go [inaudible] here the specific process ID and just see if they match for that specific one.
Lorenzo Fontana 00:20:26.084 Then, I don't yet talk about this one, we have these Readline event staple that is this one that we populate with this perf_submit where we send results back to this staple and that way we can populate the channel that [staple?] asynchronously and in a [inaudible] routine we just get the data back from that channel and they send them back to InfluxDB. That's just some boilerplate to create the text and the fields here that we have as text, uprobe, and process ID and hostname, and [inaudible] fields, and the process ID, and the command. Then we send the point back to Influx. We add the point. We write it, and this is the duty of the coroutine. Then when you press [inaudible] now here it will just stop all the coroutines and stop the program.
Lorenzo Fontana 00:21:40.047 This repo also deploys yml file that contains definition of Kubernetes resources. You need to deploy these in the cluster. And there's the service for InfluxDB. There's the toolset for InfluxDB, like there are InfluxDB [inaudible] way, and the system volume for those and all the things, but the important thing is this daemonset that is configured to connect on [inaudible], it uses the image of this daemon and that is loaded to this against this readline so that I'm now loading it against [inaudible] bash, but specifically against binary so that I basically can see [inaudible] [make sense of it?]. And now we can see it live?].
Lorenzo Fontana 00:23:00.498 This is the Chronograf [inaudible] up with the number file or...
[silence]
Lorenzo Fontana 00:23:27.625 Okay. Yeah. Have to import the dashboard. I got a screenshot of that. And what happens is that you receive every command inside of it and all the commands are just showing-in this case imagine someone logging in your systems and hopefully bash sending some commands to your machines. A record will be recorded by the BPF program and loaded with its [feed?]. And you will just see them in dashboard as commands. This is a very simple example but you can find some more examples on the BPF program if you're interested in the examples folder on bcc. For example here there's-like let's say that you want to [inaudible] all your queries to Mexico and send them back to Influx you can use this program that attaches to the lock query function of MySQL. In this case it's very different because this is C++ that adds a bit of overhead on your functions so you have to get the specific symbol for your function. Or if you are still interested, you can go to the Linux repo and go to the samples folders, BPF, and for example-this XTP is not right. You will note that these examples are a bit different in a way that the examples here in my repo and this one. And the ones in bcc are included in in a variable. And then they are loaded by the [inaudible] program or the other C++ program in this case while these examples are really the program you load. So if you want to take this program you just take all this, put it in this variable, and load it in your machines.
Lorenzo Fontana 00:26:25.088 They are a bit different in this way because the kernel doesn't have the structure to load the program yet. You have to compile the program with a compiler like [inaudible] or bcc and then load it. If you are one of the people that doesn't want the overhead of bcc in this case you can-well the good thing is that there's a backend for a BPF in [inaudible] so you can basically just compile your program. I'll show you an example. This is just an example [inaudible] compiling. You just compile the program with the backend BPF by specifying target BPF, and in this case for now you can only use C as front and compile them, but there are already frontends supporting other programming languages like Rust if you look for Rust BPF. So you can basically already define the program already in the language and just compile it, but you will still need another program to load it. For example, in this case this Go program. So in my example I let bcc compile and load it, but I could have just used bcc to load it and compile it myself, or I could have removed bcc completely and compile it [inaudible] myself.
Lorenzo Fontana 00:28:07.744 This is a set of references and links for more information [inaudible] there's [inaudible] wrote here [inaudible] XTP programs using XTP route. As I said, XTP programs are programs that are used to-they are specific for network configurations. We are using them in our cloud to define at a very low level how we route the traffic for different customers and so we have done a lot of work around it and a lot of discoveries. And it was very nice to know that there are already programs in user space like IP that already allows you to load the XTP programs without having to write a loading program like I did, for example. Here's a YouTube link that's a very nice look from Brendan Gregg from Netflix that explains a bunch of things about the BPF, and this is he specifically [inaudible] he works on the [inaudible]. And one of the most interesting documentation on how a BPF works is on the Cilium website. It's very important that they wrote this because that's basically documentation on this stuff from the internet. Just some slides and documents that are not usually updated. If you're interested in how your kernel supports the BPFs, you can refresh this document. This contains every single feature of BPF compared to the kernel version. And I think it's everything. Thanks everyone for joining me.
Chris Churilo 00:30:13.645 So before you go [laughter] let's chat a little bit.
Lorenzo Fontana 00:30:17.783 [inaudible].
Chris Churilo 00:30:18.788 So you're talking about in the end there how it actually helps you with cloud. Maybe we can take a little step back. So what was initially the set of problems that you were trying to solve that led you down this path?
Lorenzo Fontana 00:30:33.350 Well there were more than-well there were a lot of different kind of problems. The first problem I found for this was that at some point we wanted to be able to debug our programs in production. We started doing things like executing debuggers in production and using-we use a lot of [inaudible]. So we started using these tools like Delve. We started logging a lot inside that [inaudible]. Still, logging and debugging doesn't allow you to see a specific thing without having to interfere or to write application in a specific way for the debugging session you have to do. So in our case we wanted to be able to just see, "Well this is the binary we have and this binary contains this function. Is that function called with what arguments, what it returns," this kind of things. Then we went [inaudible] of that for between [inaudible], so XTP as I explained before. And also we use it for-now we use it a lot to [inaudible] specific functioning, know if they are being called, how many times, from which users. These kinds of things.
Chris Churilo 00:32:15.020 So just so everybody knows, so we have a cloud offering that we called InfluxDB Cloud, and it's basically a managed database as a service, I guess, of InfluxDB. So if you didn't want to host InfluxDB yourself you can go with this offering. And so some of the tricky things that Lorenzo and the team have to deal with is, of course, if you buy an instance of InfluxDB we need to make sure in the background after we process your payment that we can spin up a cluster, and then we also want to make sure that we can maintain the health of the cluster, deal with any kind of upgrades. Also ensure that if you're potentially reaching any kind of a limit of the database, that we can appropriately warn the user so that we don't have things go really poorly for any reason. And then also I think there's just a ton of constant configuration and constant monitoring just to make sure that we can really maintain the health of those clusters. And I think Lorenzo and team do a fabulous job and are very humble about the work that they are doing, although previous to them joining, I can tell you that it was a lot of work that the team had to do. And so they've done a really good job of making it seem like there's never a problem with our cloud infrastructure.
Chris Churilo 00:33:39.340 But as you can imagine, doing all of that can be really tricky, and there's probably a lot more to being able to monitor than we actually understand on the surface, and so I think that's why-that's one of the key reasons that Lorenzo and team had to come up with an approach similar to the one described today on being able to figure out what is happening in the code that they're deploying in production, what kind of an impact it's having, and then being able to make those adjustments as quickly as possible.
Lorenzo Fontana 00:34:15.956 Another important thing to know about-well because, one, while Chris was talking I had a question that I remember the question [inaudible] when I started using this stuff, because one could say, "Why should I write all this stuff and load the program when I can just put a log line there or use debugger or whatever it is?" It's, as I said before, because you don't always know what will happen when you deploy something in production, first thing. You will not always have been writing that code. For example, well we write InfluxDB but not specifically my team. So I don't really know very well the code base in a way that I could write a specific request to the storage while another thing looks for the [baseline?] not specifically the storage because it's very [inaudible] in this case. But I have the necessary knowledge to understand what I should look at in terms of [inaudible] usages or disk usage or I/O usage or network usage, this kind of things. And the thing is that when I want to see those stuff it's easier for me to just write down simple eBPF program, load it in my system and see what it gives back, instead of modifying Influx, redeploying it, and [inaudible] has. And also I normally don't know the code base of all the other programs I use in my infrastructure.
Lorenzo Fontana 00:36:04.697 Let's say for example you don't know really well what the kernel does. You can't look in that and see. You don't know very well what your proxy does. Well I use a bunch of different proxies. I don't know all of them very well. I don't know what they do but they can go to the GitHub repo, see the function names and see if they are called so that I can find my way in understanding what they are doing. This is how I use it, personally, most of the time. And when I used it I normally created a dashboard Chronograf that recreates that [inaudible] because I already wrote those programs [inaudible] know what is happening so I can just deploy them indefinitely and send metrics back. This stuff is still a bit early stages for us in the sense that there are not yet plans on having the BPF, for example, in Telegraf like Telegraf general loader, but we are thinking about that. I'm working closely with the Telegraf team to do that. And for me, for example, it would be terribly easier to have that because now I just have some common lines loading them from-I actually load them from [inaudible] so I write the programs there and I write them using an API. All the programs written there are loaded and I use them and it's just that. There's a question, please.
Chris Churilo 00:37:44.482 Yep. Go ahead and read the question and provide [inaudible].
Lorenzo Fontana 00:37:50.563 Do you happen to know if [inaudible] is supported [inaudible]. First of all, I don't use it, but I think it uses kernel [inaudible]. So probably it's not the best kernel to use it. I'm not sure. I never use Amazon Linux, so I don't know. I use Amazon a lot, but I don't use Amazon Linux. Sorry.
Chris Churilo 00:38:18.724 Sorry about that, Peter, but hopefully it's probably something that's easy to find out. If anybody has any other questions please feel free to chat or put your questions in the chat or Q&A, and I think this is a rare opportunity where you can actually talk with an InfluxDB employee who's actually not only deploying a SaaS solution but actually actively using InfluxDB as well. So I would highly recommend that if you have any questions about your use of InfluxDB or any of the components in the TICK Stack now's a great time to ask Lorenzo about his experiences or his thoughts or maybe he'll be able to answer any of your questions.
Chris Churilo 00:39:08.332 Lorenzo, so how often are you guys-so what are some of the performance achievements that you guys had with this? What are some of the things that-are you able to better see what's going on? Did you make some improvements? Maybe you can just share the successes you might have had.
Lorenzo Fontana 00:39:32.288 Well the first thing is that compared to the kind of debugging techniques which were used before BPF feels very-well it's very transparent with the system in the sense that it doesn't put-like, for example, let's make a simple example. If you trace a process with [inaudible] trace to know if it calls specific syscalls or if it opens files, things like that, you are actually modifying its execution in a way that you're actually putting specific interrupts in it to know what is happening based on what you are loading. If you use an eBPF program to know what files are open you are not doing that because you're actually in a lower level where every information is broadcaster in this case. So I can say that we are probably reducing our downtimes by using this kind of things because before we-sometimes while debugging the process we [inaudible]-we needed the process to be restarted. We still do that kind of things but only when there's no other way, because the debugger [inaudible] trace are still powerful tools, and sometimes achieving what they do is not very straightforward within eBPF, but that is now like 1% of the way we solve our needs compared to before.
Chris Churilo 00:41:20.294 Cool. And it looks like Marisol was kind enough to Peter to give him a link to where he can actually find the answer to his question about Amazon Linux 2. So, Peter, hopefully that-
Lorenzo Fontana 00:41:31.714 [inaudible].
Chris Churilo 00:41:32.776 Yeah. Hopefully that will help. Well we'll keep the lines open for just a few more minutes and what we'll do is definitely we'll post this video. We'll also build a landing page where we can put all the links on there and then I'm sure Lorenzo's going to be kind of working on this going forward. He's going to be doing a similar-probably a more up-to-date talk of what he did today at InfluxDays next week during the workshop. He's also going to be doing this updated talk at KubeCon in Seattle, so if you have questions even after this presentation, feel free to send me the email and I'll just forward it to Lorenzo, and chances are he'll work that into the presentation because there's probably other people that would normally have that question.
Chris Churilo 00:42:28.616 So, Lorenzo, any other advice or maybe advice to yourself if you were to start over this project, what you would have done differently? What were some things that kind of surprised you?
Lorenzo Fontana 00:42:42.390 Well I thought it was harder, honestly. And every time I meet some-well, it's difficult, because I'm not new to this stuff today. I was new to this stuff like one year ago. I always take a specific domain and I work on it for like six months at least so that I can understand it very well. And the most important thing I understood is that to really understand this stuff you have to understand that you are not working directly with the kernel and you're not working directly with the user-space program. You are just asking the kernel to do some instructions. It's not like when you write a program and you completely control it, so you have to keep that in mind. You're just writing some instructions, or very limited instructions, that are very powerful, but sometimes it can be frustrating because it seems that it's too magical that you cannot understand for what it does. But in the end it's very logical. So if something doesn't work it's because you-normally it's because you did not the right checks. Since the static verifier doesn't want you to kernel [inaudible] the machine you have to be very careful like in variable allocation and that you check everything, because it's not that the program will run and give you a [inaudible] or whatever it is because there's no option for you to don't allocate a variable, for example, because if you don't allocate the variable and then the program gets loaded in the kernel, at that point the machine panics, and the static verifier doesn't allow you to do that.
Lorenzo Fontana 00:44:48.049 So it's important to know that you are in this different kind of zone than you normally are when you write programs and you have to construct your mindset specifically to work in this situation instead of the normal one. That's the most important thing, I think.
Chris Churilo 00:45:13.507 Excellent. Cool. Well this was great. Thank you so much. It's always really great to hear from people that are using InfluxDB in interesting ways. And I did record the session, so after a quick edit I will post it so you can take another listen to it, and as I mentioned, we'll build a page where we'll start to make a collection of all the materials that we find around the topic and of course the conversations and the logs that Lorenzo will be putting together on this topic as well. And if you happen to have any other questions for Lorenzo, he's always happy to answer. You can either just directly chat with him on his blog that he posted on Medium or you can just send me an email and I can forward it on to him. So thank you so much, Lorenzo. This was great, and I will see you actually in a couple of days.
Lorenzo Fontana 00:46:05.441 Thank you.
Chris Churilo 00:46:06.193 Thank you. Thanks everybody, and have a wonderful rest of your day.
Lorenzo Fontana 00:46:10.799 Bye. Thanks everybody.
[/et_pb_toggle]
