How Texas Instruments Uses InfluxDB to Uphold Product Standards and to Improve Efficiencies

Session date: May 26, 2020 08:00am (Pacific Time)

Discover how Texas Instruments uses a time series database to gain better insights into their industrial operations. By collecting high level metrics and events, they are continuously improving productivity and are becoming data-driven. Learn how InfluxDB can result in cost savings and have a direct impact on performance by analyzing seasonality, trends and behaviors.

In this webinar, Michael Hinkle will cover:

• Texas Instruments' ability to innovate while creating electronic systems for multiple industries
• How an organization's machines, tools and processes can impact proficiency
• InfluxDB's impact on their production and quality assurance

Watch the Webinar

Watch the webinar “How Texas Instruments Uses InfluxDB to Uphold Product Standards and to Improve Efficiencies” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “How Texas Instruments Uses InfluxDB to Uphold Product Standards and to Improve Efficiencies”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• • Caitlin Croft: Customer Marketing Manager, InfluxData
  • Michael Hinkle: Probe Engineering and Manufacturing Supervisor, Texas Instruments

Caitlin Croft: 00:00:04.334 Hello, everyone. My name is Caitlin Croft. Super excited today to have Michael Hinkle from Texas Instruments presenting on how they are using InfluxDB to understand and improve inefficiencies in their operations. Just another quick reminder, feel free to put any questions in the Q&A box or the chat. We are monitoring both. This session is recorded. So you will be able to watch it afterwards. But without further ado, I will hand it off to Mike.

Michael Hinkle: 00:00:40.216 Okay. Well, thank you, everyone, for joining. Well, let me go ahead and get to it. So as Caitlin already outlined, this is using InfluxDB in essence to look for inefficiencies in our operations. So some of what I’ll cover today is briefly about Texas Instruments, what we do, a little about myself, some of the common terms you might hear throughout the presentation. I’ll probably use them without thinking. So I’m definitely going to discuss them and a lot of them have to do with some of the metrics that I’m actually tracking. Also discuss some of our database decisions. I mean, pretty much, in essence, all of the decisions we make on a day-to-day basis or database or at least we try to be as much as possible. And then, some of the current challenges we currently have, or at least I feel like we have, with our system or how we go about viewing our data or looking at our data. And then I’ll talk, finally, about why I chose to use InfluxDB, what my current set up is. So I’ll show you some examples of some dashboards or how I [inaudible] with the traces on, and explain the calculations behind them. And then I’ll eventually get into some of the small issues I encountered when I was installing the system RBM, and then what I plan on doing next.

Michael Hinkle: 00:02:03.962 So Texas Instruments. It was founded in 1951. Actually, before that, we’ve been open since 1930s. It was more for a size - I think it was more oil - more in the oil industry and seismic investigation. And we did some, I believe, when creating electrical devices. But currently, we have 30,000 employees worldwide approximately. We have semiconductor fabrication sites or facilities in the United States, Germany, Japan, and China. We actually have offices all over the world, but these are factories which I’m in an employee of. And so what do we do in essence? I say we make calculators in jest, because I said that on my first job interview as a joke, and I think a lot of people think that. But we actually have a pretty broad portfolio. We do stuff with automotive, we do stuff with MEMS technology, we have three, five materials, a lot of analog components. Some of you might be familiar with our MFPs or our microcontrollers. We design, build, sell, all of that stuff. So let me back up here a sec. We are a 24-hour operation, 365 days a year, and any downtime costs us money. So we’ve been open through this whole global pandemic that we’ve had recently, maybe at a limited capacity. But our tools are still up and running.

Michael Hinkle: 00:03:40.301 So a little about myself. I graduated with a degree in electrical engineering from the University of Texas at Dallas. I’m currently the probe engineering and the manufacturing supervisor at TI in DFAB. And it’s kind of a unique position because oftentimes, in manufacturing settings, manufacturers, at least in my experience, want to get the product through the line as quickly as possible, and engineers seem to err more on the side of caution. And so oftentimes, we have conflicting viewpoints on, well, what would be best. Do we err on the side of caution? Do we get it through the line as quick as possible? So the position I’m in now, which I took on or I started in October of last year - so I’m relatively new to it - I kind of get to make that decision on, well, what’s the best of both worlds? Whereas before, I was strictly in engineering. Prior to my current position, I was a process and equipment engineer in diffusion. Specifically, I did low-pressure chemical vapor deposition with furnaces. So I actually [inaudible] material onto semiconductors or I doped the semiconductors to make them electrically active. I am not, and I’d like to highlight this, I don’t have a background in computer science. This is just playing with the software and using the software just more out of academic curiosity. And it’s been a passion of mine my entire life. So that’s how I ended up getting into it. I read Hacker News and I do all that stuff and kind of nerd out on it, but that’s not my primary job.

Michael Hinkle: 00:05:25.776 So I’m going to cover some of our common terms that we use, that I want to highlight because I’ll probably use it multiple times and you’ll wonder what I’m talking about. We refer to our semiconductors that we’re - or circuits that we’re actually creating or building as wafers. So to your right, you can see a wafer map that actually shows - all the gridded sections are actually die. So each of those is an individual IC that hasn’t been packaged yet. You’re probably used to seeing them on your computer or on a microcontroller board that’s already in a package. It can be a flat pack or a [inaudible] or whatever. But prior to that, we start with a circular sharp straight. Now, this is silicon. I have an image that I grabbed of a 150-millimetre wafer, but this is what we make. And eventually, we take it and dice and send to an assembly site and constructed into something that we’re going to - a component for whatever your application is. But I wanted to put the picture to what we’re actually making.

Michael Hinkle: 00:06:33.534 Now, I also mentioned “lot” here. When we say lot, a lot’s comprised of up to 25 wafers. So when we start a lot or a device, we could have upwards of 25 wafers in it. No more, but we could have less. So we might do a lot starting with 12 wafers. So just to highlight that, this is what we’re making and this is what we do on a day-to-day basis. Now, I use “tool” a lot. I’m not talking about a Makita or a DeWalt drill. I’m talking about multi-million-dollar equipment. I illustrated to the right, you can see a diffusion stack. So these are actually four furnaces. I don’t know if you can see my cursor. But these are actually horizontal furnaces or LYDOP reactors that we’re using to dope the poly or whatever processes at that time, to make it electrically active. So some other examples of tools would be Ion Implanters or Epi Reactors, Plasma Etchers. My current group, we have testers, either for TestProbe or MultiProbe. Even a wet chemical bag, wet chemistry, we refer to as a tool. So that’s pretty common nomenclature in my world, so you’ll hear me say it a lot. And I had to remember that people aren’t used to that.

Michael Hinkle: 00:07:57.568 Also, “State”. We use state to judge what a particular state at a point in time a tool was in. Is it currently in production? Is it producing semiconductors actively? Is it in an idle state we don’t have material to put on it? Are we performing a time-based or a run-based PM? Is it [inaudible] down for an electrical or mechanical failure? So all this stuff we use to kind of [inaudible] out how our tools are performing, what they’re currently in or we’re utilizing them efficiently. We have 515 different states currently in our fab. I actually proved that the other day just to see where we’re at. It’s constantly changing. Cyclical. We’ll go through phases where people will want to prune out states because it’s just too much and then we’ll be adding states back in because, well, it’s not meeting all the edge cases. Some specific group needs this new state to monitor or something. So we use it for a lot of tracking. Now, those states, we also - we bend them out to smaller categories. It can be as if engineering state, is it a manufacturing state? And then, ultimately, it’ll go down into two categories. It’s either an up state or a down state. So I have some stuff to show later on, but if you hear me say state, that’s what I’m referring to. The state of the tool.

Michael Hinkle: 00:09:26.952 So modules. In the factories, we’re pretty much divided in modules. Now, that could be different. A module is a group. And a group owns tools or processes. So I was in Diffusion before and you have the Implant group, Epi group, Plasma group, Photo group. Now, different facilities or factories may group them differently. Somewhat like - we’re actually grouped with Surface Preparation, but that’s not consistent across all the factories. It’s pretty much what our needs are for a specific building. I was previously in diffusion and now I’m in multi-probing test group. So if you hear me saying module, that’s what I’m referring to. Now, most of the work we do and meetings we have are with our own modules trying to solve common problems across our groups. We do a lot of intermodule work. Our reporting out is often done as a module. We report out consistently usually on a monthly basis on maybe our tool stability or our process stability. And so this a lot of where the - moving towards using time series. Because we do a lot of reporting and it’s very time-consuming.

Michael Hinkle: 00:10:42.856 So some of the metrics that we do report out on are tool availability, we currently refer to it as AO. Our overall equipment utilization, which we refer to as OEU, and you’ll see some dashboards and reports on that later. We look at our Fail Events for our tools, we look at raw and normalized. So we’ll normalize it to the amount of wafers that are actually used through the tool to get it more on par with everything so that it’s not - you don’t have some crazy distribution. We look at Mean Time to Repair, Mean Time Between Failures on the tools. Our First Pass Success refers to if I have a - say I reached an angstrom count off - let me back up. My last group, I would have a PM network come up for - if I have 3,600 or 36,000 angstroms of material, then I would have to put the tools down to switch out filters on something. Anyway, we count that. And so when we do the PM, we put it into a PM state. Now, First Pass Success is referring to after I’ve changed out the filters in this case and started running the tool again. And I always have to qualify it to make sure that I didn’t introduce any failures or - you know, it’s operating as expected. So, typically, a First Pass Success is referring to after that PM, did I pass my qualification or did I fail it? So that’s one of the metrics we often look at.

Michael Hinkle: 00:12:13.868 We also look at cycle time. I wasn’t really keen on looking - I’m not keen on it. I didn’t really pay attention to it much as a process and equipment engineer. I mainly wanted to know that my tools were performing as intended and my process was centered and everything was good. But now that I’m on the manufacturing, I’m paying more attention to cycle time and throughput. How quickly can we get the actual wafers to align, to make sure we get it to the customers and in their hands on time? So this is just to name a few. This little bar charge is just for one of my static sites that I’ve created just so I can see what our current downs were or which ones are carrying the weight of most of our [inaudible] tool availability.

Michael Hinkle: 00:12:58.378 Okay. So why is this data important? Well, as I mentioned earlier, we pretty much try to make all of our decisions - make them all data-driven. So any time I’m adjusting a process, am I increasing depth time or I’m decreasing depth time? Or I’m looking at my temperatures or I’m trying to get material faster. I could be making adjustments, but all of these I’m looking at charts, I’m looking at the data behind the charts with my angstrom count, how long it was taking, what my temperature traces look like, all of that. Process, stability, Cpk is a process capability number. It’s kind of an index. It just shows you: is my process - is it centered? The K is the centering factor. Still, we assume most of our processes are [inaudible], that’s not always true. But most of them will have a normal distribution. So we often look at the distribution and the raw measurements and say, “Okay. Well, is it normal? What are my standard deviations?” And these numbers take that into account. My standard deviations, my upper spec limit, my lower spec limit, upper control limit, low control limit. And they are pretty much indicators from a high-level to say, “Hey, my process is robust. It’s doing what it’s supposed to do.” And ultimately, our goal is to try to decrease the upper and lower control limit and make it tighter. And reduce our dispersion in our data. So we look at that.

Michael Hinkle: 00:14:32.694 Our tool stability, I’ve already covered some of these. Our availability or our overall equipment utilization, mean time to repair, mean time between failure. We use data to track our maintenance. I already spoke on run-based counters and stuff for our actual angstrom counts to say, “Hey, I’ve run through 36,000 angstroms and it’s time to put it down for a PM.” We also use it actively for troubleshooting if there’s any anomaly that happens or a hard tool failure or any of that stuff. We go back and we trace and we say, “Okay. What are the tool comments? What happened? Let me look at my charts on the tool. What were my measurements post-process?” Just all that stuff. Anomaly detection and [addition?], we have systems that do that for advanced process control. We also have the ability to put logic and logic gates into traces similar to lab units. Not lab units, but it’s similar. We look at yield and we try to make sure we’re paying attention to planning. Do we have any bottlenecks right now? What type of device mix do we have coming up the line? Do we have proper tooling to account for it or is it going to become a bottleneck?” So all of these stuff is important. We have this, and this is just a small slice of it. I mean, we have thousands of the small signals that we’re expected to pay attention to.

Michael Hinkle: 00:15:59.179 My goal, typically, is to try to condense that down into a few pages where I can get a good high-level overview of what’s going on just to general-health-check on where it’s at so I can focus on whatever my priorities are for the day. But any of these things that I’ve listed, if they’re inaccurate or misinterpreted or misrepresented, then we could ultimately, I mean, we could scrap wafers, ultimately miss deadlines, and then we might lose business. So if one of our - say a diffusion furnace ended up failing and I wasn’t paying attention or my eye wasn’t on the ball, I could scrap 50 or 100 thousand dollars in a blink of an eye. It would not be hard since it’s a batch process. So being able to drill in quickly or at least see that something is actually happening is very important in our industry. And like I said, we operate 24 hours a day, 365 days a year. So it’s hard to keep your eye on the ball that consistently without some form of reporting and some form of trying to make sense of the world in an efficient fashion.

Michael Hinkle: 00:17:12.176 So some of our current challenges. Now, I said we might be a little presumptuous. It’s more me. But I think these are some of the - there’s a handful of us that are interested in data and digging into data and how to more efficiently deal with data. And I think from the general perspective, these are some of the challenges that I face at least on a given day. So I need to have immediate feedback if or when a tool process issue arises. But I’ve already spoken to that with my previous module. If a diffusion furnace, if something was problematic and I didn’t catch it, I could easily scrap $100,000 worth of material. So it’s critical to be able to know real-time or very soon if something atypical is occurring on the floor. Now, I’ll also need something that can give me email or text notifications. Text notifications are probably better because I end up with about a thousand emails a day at least. So I have to wade through that. But ultimately, the goal is to have a system that’s automated enough that if there’s a problem, it can shut down. Now, we have systems like that. Some of them are kind of duct-taped together. Some of them are more robust and a little - I don’t know. I don’t want to say better, but, yeah, ultimately, that’s our goal is to try to interdict on something when it’s occurring. Because like I said, we have 1,000 different small signals that we have to watch all day and it’s really hard. So if we could automate it to the best of our ability in a smart way, well, that’s the end goal.

Michael Hinkle: 00:18:57.084 So we also need an efficient way of extracting relevant data for reporting. I mentioned it on a few slides earlier that we report that pretty often. Some of those presentations for reporting out might take me, on and off, a week to prepare. Now, that’s a week that I could be spending on trying to improve a process or increase our throughput or our cost-saving measures, or any of these other projects that we have or actions that we have. The time that we prepare for those presentations ends up eating away into the time I could be bettering TI as a whole. So if we can do that in an efficient manner, well, that saves me time. And it saves any other engineer in the building time.

Michael Hinkle: 00:19:47.290 We also need a system that can be efficiently used by non-computer science and non-computer engineering employees. Most of the people I work with are electrical engineers, mechanical engineers, chemical engineers, material scientists, physicists. And they’re not full-stack devs. They’re not in DevOps. We are pretty embedded in - we have a lot of shared drives and a lot of Excel Spreadsheets and trying to wade through that stuff or find where a specific report is or a dataset is can be a challenge in itself. So the more we can start utilizing databases and get people sold on that idea of efficient data access, the better off, I think, we’ll be as a company. So I want to reiterate our primary job does not revolve around creating applications. So I’m not a front end developer. Now, I know some HTML and CSS and JavaScript. I’m probably not the most proficient in the world and I’ll probably have some curious people looking at my code that say it’s horrible. But that’s really not our job.

Michael Hinkle: 00:20:53.933 So we also need the ability to prototype and experiment. For me to get table space on a relational data basis requires a lot of ROI proof, more presentations, more justification on how it’s - how many use it, how much space I need, exactly what my goal is, what my plan for my project is. And some of that, you have a - at least for me, I have a broad idea on kind of what I want to do or what direction I want to go into. But that doesn’t necessarily translate to, “Hey, I can give you a [inaudible] and a day-by-day progression on how I’m going to solve the problem.” Because, well, you know, sometimes that changes on the fly. So using software like Influx, I can actually - I can prototype more. I can experiment more. But I think that some of my current challenges that I deal with on a day-to-day basis and somehow what I’m trying to answer using software.

Michael Hinkle: 00:21:57.071 So why did I choose Influx? Caitlin mentioned it earlier. I use it at home for personal project. I track my daughter’s pulse oximeter data with it. So I look at weather and various other traces. So I use it for that. And so I knew that the installation only would take a few minutes. I was shocked - I’ve installed clients for relational databases before that, no joke, took over an hour to install. Now it’s just the client. And then, in this case, well, it took me probably five minutes to download it, install it, set it up, get the configuration file, fine-tune, and start up the service. So that was very appealing. There’s a multitude of available clients. So I use primarily Orange in Python. I’ve also used the Influx client from the command line interface. But there’s many other clients for most of the programming languages that are accounted for. So that availability is there.

Michael Hinkle: 00:23:01.393 This is one of my first forays into noSQL. I’ve played with Mongo before, and also some graph databases. I like the fact that I don’t have to sit there and plan out how these things are combined together. Which table goes to where, what’s the primary key, what’s the foreign key, secondary key, whatever you want to call it. And I know for our systems, there were - I mean, for me, if I were to ask to add a column on, I mean, that’d be like asking to move a mountain. So in this case, while I can change it on the fly or I can drop a measurement, or I can add a measurement, or whatever - I mean, the sky is the limit - it’s a lot easier to prototype. And if I’m not 100% sure on what I’m doing but I kind of know the direction that I’m going in, well, I can prototype and I can play around with it. And it’s not going to - like I said, I don’t have to move a mountain to change it. I was happy there was, I say, SQL-esque. I’ve got probably over a decade of experience using SQL or relational databases. So this wasn’t a big jump using InfluxQL from what I’m used to. Now, you don’t have the joint capabilities, cross measurements, and you use double quotes and - there’s some various differences, but, by and large, it’s very similar. So the learning curve was not really great. I know about Flux. I haven’t used Flux, but it looks a little different. I know I tried Cypher and I’ve looked at Gremlin and some of the other ones. I don’t have all the time in the world to learn 50 million different languages every other day. So this was very nice that it was already in a similar language that I’m used to using.

Michael Hinkle: 00:24:49.438 So fast query response - I highlighted that because at least, the way some of our tables are set up or partitioned or indexed it may take me eight minutes for a real basic query to return. Now, based on the millions of lines that we have in our tables and how they have it set up or how I need it structured, I mean, I’ve been kicked off the server more than once and I’m pretty confident in writing SQL. So on this case, yes, it’s index by time. You can also index it by tags. But I can go back on months of data at really small chunks every second, every minute and be able to bring back the data very quickly. And Jupyter Notebook or whatever I’m doing my analysis in. So I also like the fact that since most of our metrics are temporal at least in a lot of the high-level measures, I run into a lot of issues where I have edge cases. So what I mean by that is, oftentimes, we’re looking at timestamps across the months. So I may have a state that a tool’s in across two months now. I have to usually handle that on the fly and say, “Okay, I need to split it and that way, I can allocate the time on one month and the other time on the other month and make sure that my calculations come out correct.” Or, “The way I’m currently collecting data,” which I’ll get into in a little bit, “I don’t have to handle these edge cases as much.” So it actually handles it for me with time.

Michael Hinkle: 00:26:24.347 And I often get asked by - I’ve been asked by a manufacturing superintendent - I think he gave me - if I could give them a report on our tool availability. Now, I’m used to, or at least I was as an engineer reporting on that on a monthly basis. But a lot of people - I always had to ask the question, “Well, do you want it by month, do you want it by hour, do you want it by shift?” Because oftentimes, the manufacturing person’s going to want it by shift or by day. And they’re going to want to see that my shift on the front end of the week, are they more efficient than my shift on the back end of the week? And the way that I’m collecting the data right now in this database, I can handle both cases. I can do it by a month. Or I can do it by a year. Or I can do it by an hour or by a shift. And it’s easy. So that’s a big advantage.

Michael Hinkle: 00:27:16.897 Also, I like the math functions and the forecasting functions that are built-in. I’m already using the derivative to look at the rate of change. I’ve started playing with the whole winners and looking at ARIMA. Because they don’t have ARIMA here, but the fact that you have this inbuilt math function is great for one of our use cases at least right now. Now, I write over about a million and a half points of records a day at TI. So far, I’ve been using this since, I want to say December. So I have a small development map and I’ll get into kind of how I have a set up here in a little bit. But the way it utilizes the memories is amazing. I don’t really have a benchmark, but I don’t have to worry about my memory consumption on development now. I’m writing 1.5 million points a day or greater than. Another selling point was that it plugs in easily with Grafana. I’ve experimented with Chronograf. I have it running at the house. I also have Grafana running at the house. But just the ease of syncing that up, honing the Grafana at the actual database and to start screening data, I mean, it’s great. Because I don’t have to build a whole application around it or like I said, we’re not full-stack developers. It’s also open-source, and it’s free. At least this version is. So I’m not running a distributing system. I’ve got a single instance running. But currently, it’s checking all the boxes and doing what I need it to do. So that’s great.

Michael Hinkle: 00:28:54.464 So the current installation and setup. I’m running two VMs. Linux VMs. So I have a development mount, I mentioned, which is five gigabyte. I have the - the time series merge-tree is in the Write-Ahead-Logs and everything pointing to that development mount to store. It is expandable. I haven’t run into any need yet to expand it. I am actually pulling most of the data. It’s is not coming from - it’s not RS-232 data coming straight off the [inaudible]. So most of it I’m actually pulling, forcing, sanitizing, and everything off our pre-existing relational databases which we pretty dug into. So I have a bunch of loader scripts which run on cron jobs which go out, query, get it organized how I need it, and send it back off into the Influx database for real-time viewing. So I also have a Grafana server running on one of the VM instances and they’re running Apache here. Some [inaudible] or something. But I have it set up as a reverse proxy. So I can serve my static websites as well as redirect people to the Grafana server whenever they log in.

Michael Hinkle: 00:30:07.488 So currently, I’m collecting all over our equipment utilization. I’m actually doing this by the minute, which is the smallest slice of time we could do. And then I’m looking at total availability. I’m doing some for the fab, which I’m not directly in anymore, but I’m doing it for some of the manufacturing people in the fab and some of the managers. I’m looking at the recurrences of state, which I spoke to earlier, the state of our tools. A lot of people with 515 states don’t know that a certain state might count against them. So it seems like it’s a good practice to show people the distribution and what those states count against. But I’ll get into that some in my dashboard example. I’m also starting to collect cycle time per device as one of my tags. And we run probably 1,000 different devices through. It could take - and this is primarily for the group I’m in right now. It might take an hour to test a way for it. It may take six days to test the way for it. It depends on what type of electrical test we’re running on the wafers and there’s a bunch of other variables. But I’m starting to collect that per device. So here is just a basic block diagram to show these are relational databases on point from multiple different data sources. They’re being pulled into using Python and I use the InfluxDB client in Python, and then I end up writing that to the Influx instance and then I’m reading that into - I have that plugged into Grafana. So this is a high-level overview on just what my current setup is at TI.

Michael Hinkle: 00:31:53.799 Going into the dashboard. So here is a screenshot of our overall equipment utilization dashboard for my group that I’m carrying. Now, I’m sampling it every minute whereas before, I think, the best we could get was about every hour when we have reporting for some of our industrial engineering groups just getting to it on an hourly basis. So my rationale was, well, since I have this issue with time where some people are expecting it to be aggregated over 12 hours, so some people are expecting to get aggregated by quarter, if I get it done with the smallest building block I can, well, then I can extrapolate any other number from that. So we’ve summarized our overall equipment utilization on a per-minute basis. Now, what this - it’s an index pretty much, it’s a percentage, and it’s in essence how much time - well, it’s a ratio. So the numerator is how much time I’ve spent in one minute testing with good dye versus that one minute. So typically, for a month, it’d just be the summation of my time-testing good dye over my total time for that model. So we use that kind of as a high-level determination to say, “Oh, well, the floor is looking pretty good right now.” Now, the yellow line right there is another value I’m feeding into the database, but it’s our current goal, which is I think 82% I believe. And then I was just playing around with smoothing it over an hour just so it wasn’t so noisy.

Michael Hinkle: 00:33:32.892 But anyway, this was something I was able to get up and running in probably - I don’t know. I had a few hiccups. It would have taken me probably 10 minutes. I had a few issues with the proxy that I’ll get into towards the end of the presentation, but also normally, it’d probably take me 10 minutes. It took me about 10 minutes to get it up and running at home. So I was a little more convoluted since I didn’t set up the VM’s [inaudible], but I’ll get into that in the end. I do look at this consistently throughout the day. Part of my job is - I was more of a full-time technical role before in my last position, now I kind of divide my time into growing my teams, reaching in their goal, what do they want to do, trying to push them to be their best. And then the other half I try to - I use methods like this to try to understand my new world and what we’re doing. So I dig through databases, I try to look for inefficiencies, whereas before, I was looking more my processes and my tools. I have taken over a tool set or a test platform since I’ve been - I’m out on paternity leave right now. But when I go back, I’ll be a proud owner of a test platform or a few. So I’ll be looking more at my tool stability and more of my - all my tests I care about. But currently, I do look at this on a day-to-day basis to see how we’re performing on the floor.

Michael Hinkle: 00:35:04.506 One of the directions I want to go is to start using smarter techniques to analyze our data. Our time series data. Most of our high-level metrics are time series. I haven’t really played with this too much. But I went ahead just practicing decompositions. Eventually, I want to get into decomposing my signals and moving into forecasting so I can make more methodical decisions toward things. I see a lot of people using less than sound reasoning for saying, “Hey, we’re going to have an issue next month or -“You know, trying to predict what’s going to happen. This is kind of my foray into looking in time series. I didn’t know that in my background, we didn’t discuss time series data and that there’s ways to approach it. They’re using ARIMA analysis or Holt-Winters or any of that stuff. So I’m starting to play around with actually decomposing the data so I could get a good look at do I add a seasonality to it? What’s my trend? In this case right here, my residual doesn’t look really random enough. So obviously, I’ll probably have more work to do in this. But this is kind of the direction I can go with our current setup and I was able to do this in a Jupyter Notebook on my laptop in about, I don’t know, 5 minutes with 10 or 15 lines of code using the Python stats-models library. Now, we don’t have a lot of people that are involved in that in my building, but I think there’s a real benefit to using a good sound math-based forecasting method versus my gut feeling on something or where I think I might be next month where I can give a genuine - say, “Hey, I know with a 95% confidence that I’m going to probably be within these boundaries.”

Michael Hinkle: 00:37:05.889 So this is the direction I want to go. I set the frequency on this for the period linked on this for 14 days because that’s kind of a full cycle of our shifts. But, obviously, there’s more work that needs to be performed. I haven’t done any differencing or any transformations on the original data. I aggregated it on export to per hour. That’s kind of where I’m at and just kind of the direction I’m going.

Michael Hinkle: 00:37:31.312 This is just zooming in. I’m discussing that - really, showing - I’m showing the trend in that - just doing a quick sanity check. So this shows around the time that we were issued the stay-at-home order in Dallas. So you can see that by my trend plot, the second one down. And sanity check, yes, that would be accurate. Because we are still running, but we’re running at a diminished capacity at this point. But really, ultimately, my goal in fine-tuning this type of analysis is to understand my seasonality. And I say that because I’m trying to understand some of the behaviors. When I came into the group, I was told that our two ends of the week operate on kind of two ends of the spectrum. I went into the group that likes to ride up everything that if it sounds wrong, say the tool has a weird noise, well, I’m not going to ride it up and let it sit for 12 hours until the engineering or the technicians can get in and look at it. And then the other half of the week, well, a lot of them are pretty seasoned veterans and they’ve learned how to live with problems over the years. So they can ramp it through, and a lot of times, those issues go unnoticed. And ultimately, most of the problems that we finally get handed to engineering, we’re not curing cancer or landing something on the moon. They’re usually pretty easy. The main goal is opening up - an open-up enough environment where people can bring you problems and feel fine about it. Where they don’t feel attacked or they’re doing something wrong and you’re just helping [inaudible] ideas and hoping that people bring it to you so you can fix them. So this is my attempt or my start of an attempt to try and understand behaviors across the [inaudible]. So that way I can address them at least in my end of the week real-time, and talk to the supervisors at the end of the week to be able to come to some consensus and show them some data to justify why I think something is the way it is. So this is in that spirit and just kind of the direction I want to go. Obviously, it needs more work, but this is where I’m going.

Michael Hinkle: 00:39:44.084 Now, one of the advantages I spoke to earlier, one of my challenges was being able to know real time that something’s happening so we can catch it earlier on. When I first set up this in December, I had a message sent out every time we broke the goal. Now my co-worker, Joel, was quick to say, “Well, we don’t really care if it breaks the goal for five minutes.” Which he’s right. He’s 100% right. Our men care about, at least in this instance is maybe if over a certain duration or change, if my rate of change in my equipment utilization suddenly drops off a cliff, I might have a bigger problem. We could have a power sag on the floor, we could have - in this case, we actually had a failure to write to a certain database and it was locking up tools. But I was able to utilize Influx’s derivative function to look at the rate of change on our equipment utilization and say, “Hey, if I fell off too fast, then I’ve got a bigger problem than just some tool falls or someone going to lunch or whatever the issue may be.” Something that we really need to address. And typically, if we have a power sag in the building, you’ll hear phones go off all over the place. And maybe 15 minutes after the fact, but it starts happening. Something like this might get you a lot faster if you knew kind of what you’re looking for and how much is going to follow.

Michael Hinkle: 00:41:18.867 But anyway, this is an example of how we can use it real time. And you can see right here - I don’t know if you can see my cursor, where we have a drastic drop. And it was a legit issue that we needed to address. Now, we couldn’t address that issue. Our IT is kind of segregated from our day-to-day factory operations. We do have IT people that deal with our production. But most of the group is no longer in our building. So we have to pick up a phone and reach out and have them go back there because we don’t have access to see what’s going on and why the systems were locking up. But this sent me a notification and an email and said, “Hey. You’ve got some problem because your rate of change, your OEU, fell off over this duration of time pretty quickly. So maybe you need to look at it.” So functionality like this is a big selling point for me to be able to use some quick basic function to catch something. Integrate - I didn’t mention this, but a lot - we’ll put signals up sometimes and I’ve seen people use the integrate just to get a baseline signal so they can guard-band it or put limits around it to look for anomalies or - it’s just a real quick boiler plate way to get something up and running, or even start monitoring a system or a sensor, or whatever is going on. So it’s powerful.

Michael Hinkle: 00:42:45.428 Moving on. Now, it looks kind of like cool tie dye, but really, what this is - it’s our states. If you look to the right, you can see how I have it categorized or colored by our various states. Now, I use the states as a tag. So it is indexed by that. Now, for me, I can use this to see, oh, well, our OEU is really down. So I can go up here to look at all my down states and toggle them on and off to see kind of how we trended. And then, do we have stuff that was in the down state or an “01”, which is our down state, for a long duration of time? Also, sounds kind of creepy, but do I have all - the people on my team, are they addressing our failures as they come up or are they letting them sit for hours? Do we have certain times of the day that people decide to work more? Because I know naturally that some people might be more inclined to come in and knock out as much as they can at the end of the day and try to take it easy. So this is kind of in the spirit of seeing how we’re responding to tools, how the different shifts are responding to tools. And again, it goes back to behaviors and the differences on shifts or the differences between how people respond to things. I look at this also - primarily, I look at it to see how many tools are up or down at a given moment. So I do have this open consistently throughout the day just so I can see kind of where we’re at really quickly.

Michael Hinkle: 00:44:25.204 Now, the pie charts, I talked about how our states are mapped into other categories. We call one of them our e10 distribution or our e10 groups. Now, e10 group as an engineering state is in a non-schedule state production. Its schedule down state is in standby or is in an unscheduled down state. So was it a hard tool fault, or some failure that caused it to go into that state. So that’s just a brief - so we have this just to show the distribution and bidding. So oftentimes, someone will come up with a state and you’ll see a tool in the state and go, “God, I don’t even know what that is.” But I could easily come over and say, “Ah, well, that is being [inaudible] out and that’s the last down state.” Or, “It’s counting against us or not counting against us or whatever.” It’s kind of a game and - not a game, but it’s a - you’ll find a lot of people trying to take advantage of different states. But anyway, this is in a spirit of behavior and to try and attract behavior on the floor.

Michael Hinkle: 00:45:35.575 This is a tool availability. It’s analogous to our overall equipment utilization. The factory is handling it a little differently than a worldwide probe site so the benchmarking on the metric should differ. AO is pretty is much your up time of your tool over the total time of your tool. So it could be up and idle and still be good for you AO. So you can sand-bag it if you don’t run your tool overtime. I can include my metrics by having the tool up even though, well, it’s not really doing anything. So it gets kind of muddy looking at some of those things and explaining them and going through it, but just think of it as how much time is my tool available to use? Now, I’ve split it out here at the bottom just to show some of my old toolsets in my last group. Now, again, I have tagged all of these data index to buy the different toolsets across different modules. So that way, it’s easier to zero in on it. By I can look at my overall tool group availability.

Michael Hinkle: 00:46:43.624 I also have an index by tools, but it gets really murky at that point because you got hundreds and hundreds, probably close to 1,000 tools across the board. So to show that all under one plot, it just gets confusing. But in this case, using the phone, I can always toggle it on and off. But the green trace up here is diffusions overall, our AO. Oh, actually, this is our - no. It’s diffusion’s AO. Okay. And then these are my toolsets. So again, it was taking it every minute a lot to answer the problem of people requesting different time windows or different sets of time that they’re trying to look at. I don’t have any hard limits set on this or any messages or any alerting or alarms. Still, this is relatively new in our factory. With everything that’s going on in the world right now, we haven’t had a lot of time to dump into it. Anyway, I want to go ahead and move on.

Michael Hinkle: 00:47:49.257 So some of the issues I encountered while I was installing TI. So I had my some of my runcom files, I had proxies set, my http_proxy and my https_proxy. I couldn’t write - my cron job is failing on me, and I couldn’t figure out why. And since I’m not a sysadmin, I’ve run Linux for probably two decades. But I was spinning my wheels for hours trying to figure out why on God’s green earth could I not write to the database? And it turns out these proxies were set. So I had to go in and comment - I had to go out and comment them out of my runcom that I was sourcing for my cron job.

Michael Hinkle: 00:48:32.557 Other problems that I ran into, derivative from my specific application, I don’t know why I was having issues, but I needed to use a WHERE clause for me to get the correct return. I think it was pretty much screaming at me, telling me I need to use a WHERE clause, whereas a lot of window of functions I’ve used in the past, I did not need to use a WHERE clause in a group by state. In this case, I did. And whether that has to do with the lining up where I want to start my time, my [inaudible] - do I want to start at the top of the hour or do I want to start at 13 minutes into the hour? That’s probably kind of why, but I’m not sure. Also, usually, date times will eat my lunch on most things. I initially wrote to the database using our timestamps which are zonal. So they’re actually - we are using American Chicago time zone. And I wasn’t writing UTC time zone. So I needed to convert to UTC time to actually write it to the database. But anyway, that wasn’t an issue with Influx. It’s just an issue with me and date times and it always getting kind of interesting every time I deal with them.

Michael Hinkle: 00:49:51.757 Finally, the last issue around kind of encountering is that I can’t aggregate by a month. So I can do it by 30 days or I can do it by some other chunk of time. Now, I’ve read up on this, I haven’t implemented it yet, but I hear that Flux can handle months and then also, I could add a tag to my dataset and actually include words. Month one, month two, month three. That way, I can end up splitting it out by months or grouping it by months [inaudible]. So those are really some of the issues I encountered. Really, overall, it was really easy to get up and running. And to get something up and running where it’s visible on the web for everyone in your organization to see it within 15, 20 minutes, 30 minutes, whatever it may be is really saying something. Now, we have other ways to do that, but it’s a little more difficult. It requires a little more scripting, stuff like that. But it’s possible. But this is - overall, it was easy. I did it at home. I was shocked at how easy it was. So I went in and started pushing to be able to install this at work and to be able to start working on this at work. Because I think it would have been beneficial for our whole organization.

Michael Hinkle: 00:51:07.747 So what I plan on doing next. There are some things in Grafana that - it’s pretty general. You can go in and set up panels and do what you need to do. You can do the same thing in Chronograf. There’s some chart types that I wish that were available even in the plugins. For example, box plots. I look at the box plots all the time for an easy side by side comparison on my distributions. I haven’t been able to find that. So I was looking at doing some [inaudible] or Dash apps to try to have a little more control over what my final [inaudible], UI is going to be. I could also - I’ve considered, well, maybe I can contribute and actually write a panel that handles box plot. So I don’t know. I’m mauling that over. It really depends on time constraints and whether I can or can’t do it. But I plan on making a Dash app to kind of summarize all of our tool stability and our process stability, all in or two tabs or sections. And right now, if I were to prepare a presentation from my tool stability meeting, I would probably have like 20 reports open. Excel, Spotfire, you name it. And I’ve got to jump back and forth doing this, this, this, and this. And really, just to get all the data I need to my tool’s health. I need to look through 20 different sources. I am making up a number, but it’s a lot of different sources. Whereas in this day and age, we have plenty of software out there and plenty of capability to be able to summarize this all up into one page, say, “This is how my tool’s doing from a eye level.” Of course, anything below that, I’m not scratching the surface at all, I’m going to have to drill down more. But to get a general indicator on whether my tool is doing well, my process is doing well, I could summarize it all in one page. And this is kind of what my goal is.

Michael Hinkle: 00:53:17.093 Also, I want to work on decompositions. I showed you an example of my first foray into doing the decomposition earlier. I’ve been reading more and more on this ARIMA forecasting and [Holt-Winters. What really keyed me in was when I found Holt-Winters in Influx. I’ve never analyzed time series data or really went about it so methodically. So I’ve been reading a lot in the last few weeks, last month, trying to wrap my head around stationarity, non-stationarity, seasonality, just all of the different parts or making accurate projections. And what’s an augmented Dickey-Fuller test. And all the different things that go into analyzing time series data. And since that’s a lot of what we deal with on a high level, it’d be nice to go about it more methodically. And I say that going into one of the - what’s next- is I plan on giving a presentation on how to methodically do it. And, obviously, I’ve got to read more on it, practice more on it, but that’s kind of the goal. I’ve been asked before the g

oal of SQL and how you do SQL and have a class on that. Every once in a while, one of my - one of my co-workers [inaudible] a few years ago a class on that. But I was asked more recently. But next time I get asked, I can go into maybe some more time series data analysis and this stack and how we can use it. Because ultimately, the goal is trying to free up my own time and another engineer’s more time to solve our real problems, our critical problems, instead of preparing for presentations, instead of trying to determine where do I need to put my time and effort. Because a lot of times, I don’t feel like people have enough information to make a good decision on prioritization. But I think we can move in that direction. And hopefully, that will open up the flood gates for new ideas and new applications for using this database and some of this technology. So hopefully, I didn’t stumble over it too much, but if anyone has any questions and comments, I guess I’ll be answering these as best as I can right now.

Caitlin Croft: 00:55:55.003 Thank you, Mike, that was fantastic. So, well, I’ll just give the room a minute or so to kind of think of their questions. Just wanted to remind everyone once again, we have InfluxDays coming up. It’s the first time that it’s completely free. Obviously, given the current health concerns, we’ve gone completely virtual, so the event is completely free. There’s going to be a ton of different presentations, Influx people as well as their customers. So be sure to register for it today. And the cool thing about it being free is you don’t have to worry about getting approvals for a budget or a travel or all that. And in addition to - one of the components of InfluxDays is our Flux training. So Mike did mention a couple of times playing around with Flux and learning more about it. Flux is InfluxDays’ newest - it’s a data scripting and querying language. So it’s a really great hands-on training. We’ve run it before. Everyone really enjoys it. It’s the great way to get better at InfluxDB as well learn Flux. And what’s also fun is the real example that you use during the class is actually all about pizza-making. So the trainers definitely make it fun for everyone. So we have one question here. How easy was it to build reports by Chef? I imagine you must have had some sort of tool previously to InfluxDB to collect this kind of data.

Michael Hinkle: 00:57:28.889 Oh, we have tons of tools or applications. And oftentimes, within the application, it doesn’t meet everyone’s needs. Everyone looks at data a little differently. And the more I’ve created applications or little websites or data applications, web-based, I’ve come to realize that everyone looks at the data a little differently. Now, to build it with Influx and Grafana was really easy. Now, I don’t have a tag set in the shift right not, but typically, what we’ll do is we’ll build the shift into it. So I’ll attach the shift onto the dataset. That way, it’s easy to split and group and categorize, and I can split it up by shift. But, no, in this case, ours is a little more tricky because we don’t run - it’s three days on for our shift and four days off, and the next week, it will be four days on and three days off. So we get swing days on night and on day. They’re 12-hour shifts. So I haven’t added a shift tag onto this dataset yet. If you looked at some of the dashboards, I split it up just to break it up. I used the coloring feature on the plot so it showed that, hey, we’re in the weekend. Just so visually, I can see that. No. What I planned on doing was adding a tag that said, “Hey, I’m A shift, B shift, C shift, D shift.” So that way, when I’m actually writing that data for that minute snapshot of what our equipment utilization is, I’ll also be passing on the shift too. So that way, if I want to split if off later and group it by that shift, then it would be a cinch. So it’s really not hard in this case. I don’t know if that answers the question.

Caitlin Croft: 00:59:19.613 Great.

Michael Hinkle: 00:59:21.083 Go ahead.

Caitlin Croft: 00:59:21.628 I think so. And I’m assuming you must have had some tools prior to InfluxDB. Can you give any insights into how you were looking into your efficiencies prior to using Influx?

Michael Hinkle: 00:59:34.669 Sure. Usually, you have a cross sectional view. So this is more of longitudinal data where I get real time. So typically, you’re going to - whenever you would open up a report of - I don’t know if I can throw out - I’ve been trying to shy away from company brand names, but we use Spotfire for a lot of our analysis. So I have a lot of automated reports and templates that when I open up it’s going to give me a snapshot of, say, my historical AO [inaudible] I open that tool. So we have, like I said, a lot of web-based reports. I’ve put together some - people over the last two decades have put over some Perl-CGI reports out there, and all sorts of stuff. Now, typically, it’s - I mean, for someone that doesn’t have a real programming background, it’s kind of hard to implement. And then, like I said, it’s frustrating because you spend all this time and you get the styling right or you use data tables or you’re using your - you have D3 interactive plots and they give you that snapshot of time that cross sectional view and then someone comes along it’s like, “Well, it doesn’t really help me.” And yeah- and then you have to move mountains to try to change this whole report whereas with this setup, if I train people on how to use the creative panel, what data is available and how to put it together, they - actually, 20 or 30 minutes of sitting there clicking on buttons, and the query builders, they can get an accurate snapshot of what’s happening over time. So it takes a lot of the work out of it, which is excellent because, yeah, that’s not our main focus.

Caitlin Croft: 01:01:22.715 Yeah. Totally. I can understand where you want your team to focus on the important stuff. There’s another question. Do you have a finite set of states? You mentioned that you’re collecting 500-plus states and that the number of states [inaudible] can go up and down. How do you manage all the different states? Does someone know? Is that a group decision? Did your team members want to manage them themselves? How does that work with all the different states?

Michael Hinkle: 01:01:54.877 Well, there’s some overlap. So between the modules, usually, it’s how you deal with it. So like I keep using the Diffusion because I’m familiar with it because I was in there before. We had states for a tube change and furnace or a paddle change and a furnace, and no other group or module has a paddle change or a tube change. So that was unique to that. Now, no one is a certain - it’s just a hard [inaudible]. Now, everyone uses that. But when it comes to the unique states, it’s pretty much handled in the module or in that group itself. Typically, it’s taken up the chain a little bit. So we go to our section managers and everything like that. You pretty much pitch it. Like, “Hey, I really need this state and here is why, and this is what it’s going to benefit.” And the, on the other side, it’s always a back and forth. We have so many states and it’s really hard - I’ve had seasoned engineers that have been there 30 years that - I told them that, “Hey, your AO was lacking. You got to go present in this meeting.” And they’re like, “Well, I don’t get it. I didn’t have any down states the whole month.” And then when I sat down and looked at the data, I said, “Well, really, this day, this day, and this day, all goes against you. So that’s why your percentage is down.” There’s no common coordination when they add or take a state off, to answer that question. So it’s not like an announcement is enough like, “Hey, we’re adding on state 911.” They don’t do that.

Michael Hinkle: 01:03:23.053 And since we have so many states, really, a lot of us who are used to seeing, I mean, like 01 or prod or - there’s various states that a lot of us know just inherently because they’re consistent. But I’d say that there’s a lack of knowledge on how that’s mapped out. And really, the only way that I was able - I’d ask questions when I’d started in TI and how they were mapping it out. Because I’m curious, how am I getting graded on this metrics. And I want to understand the calculations. And most of the people I asked didn’t know. They gave me some answer with some simple summation equation. Well, I know this summation equation, but what state’s mapped to it? And it was until someone keyed me on to where they actually mapping some other database that we have that most people don’t have access to, then I was able to see it. So I try to bring that into everything now so people can realize, hey, this gets mapped to an up state. This is mapped to a down state. But, no, there’s no general [inaudible]. You take it up with management, you highlight the need for - why I need to add a state or why I think it’s good to take a state away. And then, eventually, there’s a section there they’re using then we’ll talk to IT and make the change. But, yeah, usually, you have to state your case on it.

Caitlin Croft: 01:04:44.802 Wow. Yeah. It sounds like even just finding the data in the company was a challenge for you.

Michael Hinkle: 01:04:50.683 Well, we’re a really old company. And we have decades of databases. And, really, we have - our main backbone database is mapped out. We have kind of the [inaudible] and the schema and everything, now it’s joined together. Which is good because that’s where most of our data is extrapolated or [inaudible]. But we have planning databases and industrial engineering databases and tools that we - it’s spread out all over the map. Really, there’s only one that has any sort of map. And the rest, you’ll look around in the dark. You look at what table names are there. “Oh, that one looks interesting. Let me go and pull that.” And so we have a lot of good data, but yeah, it’s hard to navigate. And if you’re not really into that - like I’m passionate about computers. I like this stuff. It’s more of a hobby, but I enjoy it. So for me to go into database, well, it’s not a big deal. But you ask someone from perhaps a physics background and they’re really into device physics scenario, into the nitty-gritty on semiconductors, they don’t care about the database because they’re using some pre-canned report. Most of them aren’t aware that there’s 20 different databases that they could be hitting for just various different information. So they end up - we have a handful of people in the organization that do that, or they’re into that. But I guess my goal is to try to give that ability to everyone so that they know that the information is there and they can make more calculated decisions on what they think the priorities are. Yeah. I guess that’s it. Yeah.

Caitlin Croft: 01:06:39.615 Yeah. Jumping off of that, you mentioned that you’ve had to provide some ROI to get some table space on the relational database. Was it difficult to get an instance to install InfluxDB?

 

Michael Hinkle: 01:06:53.598 Well, I bother IT enough that I know who I can talk to from the system admins to expedite it. Now, it took a little longer than I’d wanted. Just because, A, I didn’t want to be pushy, and I think it got lost in his email. But, no. I pretty much know who I can talk to, to expedite it. Now, to get a legit table on one of our relational databases and do it through all the means that they would typically require, yeah, it’s not easy at all. And right now, we have a lot of major changes that are happening. I mean, that would give file 13 to end up on some long list of actions by IT, and it may not happen for two or three years. So no. In this case, I knew who to talk to. He gave me the go-ahead, one of the system admins, and I was able to get the upper line really quickly. So -

Caitlin Croft: 01:07:54.432 That’s awesome.

Michael Hinkle: 01:07:55.393 Yeah. It was great. Because otherwise, you’re using SQLite or you’re using [inaudible] systems to try to [inaudible] your prototype or create an application or run analysis or anything, do experiment with anything. It’s outside of the norm of what we have. So, no. It was very easy. It doesn’t have a lot of dependencies or anything. So, yeah, it’s - I mean, yeah. I totally [inaudible] a little bit.

Caitlin Croft: 01:08:21.520 Looks like there’s one more question here. What retention policy do you plan on using for this data?

Michael Hinkle: 01:08:29.383 Currently, I’m using the default, autogen. So I guess it’s infinite I guess. Really, I’d initially pitched it to the system admin to use six months. We’re still kind of - I’m still kind of - I’ve been pushing it to some of management, showing them the capabilities. Really, I was also watching my percentage of hard drive space used on the development mount. So I backed up my six months and right now, I have my month in it until we come to a - typically, most of our data is about a year and a half. Now, we have all of that backed up on tape or whatever for 10, 20 years. We had to request one of our IT sides to hold data. Typically, that would happen if, say, a customer returned [inaudible] back and, well, that data fell off the system six months ago. Then you’d have to call and request and say, “Well, I need this data from two years ago to see what happened in this duration of time.” So right now, I’m just using the default retention policy. Initially, I pitched because I thought I was going to require a lot more drive space six months. Really, I need to sit down with my team and figure out what makes sense. Should we down sample after - do I really need every minute three years down the road? No. Probably not. So I haven’t had it long enough to really sit down and have that meeting with everything going on right now and [inaudible] opportune time. So, yeah, we haven’t settled on that yet.

Caitlin Croft: 01:10:13.144 Yeah. I mean, that makes sense. We find a lot of customers find they need to have the data for a certain threshold, and then once they have that initial dataset, they can downsample and they don’t need as much data granularity.

Michael Hinkle: 01:10:25.864 Yeah. I wouldn’t think -

Caitlin Croft: 01:10:26.390 Oh, good. It looks -

Michael Hinkle: 01:10:27.837 Go ahead.

Caitlin Croft: 01:10:29.316 Oh, go ahead.

Michael Hinkle: 01:10:30.545 Oh, I was going to say, for our use cases, usually, probably a year would suffice. I know planning in - there are finance people, they’d probably want to see the last five years. So it really depends on what they’re really looking at. For something like OEU, I couldn’t see more than a year of being useful. Really. So.

Caitlin Croft: 01:10:53.392 Great. Well, thank you so much, Mike. Thank you for presenting that webinar and answering everyone’s questions. That was fantastic. So for everyone who’s still on the line, if you guys think of any other questions that you’d like to ask Mike, feel free to email me. You should have my email address in your registration emails. So feel free to shoot them over and I will connect you with Mike. Once again everyone, this session has been recorded and it will be available for replay later today. Thank you very much everyone for joining and I hope you have a good day.

Michael Hinkle: 01:11:32.107 Yeah. Thank you.

[/et_pb_toggle]