InfluxDB Cloud Product Update

Session date: Apr 28, 2021 08:00am (Pacific Time)

Learn more about InfluxData’s time series platform. InfluxDB Cloud is a fast, elastic, serverless real-time monitoring platform, dashboarding engine, analytics service and event and metrics processor. It is available on AWS, Azure and Google Cloud. Since its launch, we have been busy making updates to the product!

Join Balaji Palani, Director of Product Management, as he demonstrates the latest features of InfluxDB Cloud. This one-hour webinar will feature a product update and Q&A time.

Click here for presentation

Watch the Webinar

Watch the webinar “InfluxDB Cloud Product Update” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “InfluxDB Cloud Product Update”. This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.

Speakers:

• Caitlin Croft: Customer Marketing Manager, InfluxData
• Balaji Palani: Director, Product Management, InfluxData

Caitlin Croft: 00:00:00.490 And welcome to today’s webinar. My name is Caitlin Croft. Glad to see you all here. We are joined by Balaji, who is part of our product management team, and he’ll be providing an update on InfluxDB. If you have any questions, please feel free to post them in the Q&A. We will be taking questions at the end. So don’t worry, we will answer them after Balaji’s session. And this session is being recorded and will be made available later. Without further ado, I’m going to hand things off to Balaji.

Balaji Palani: 00:00:36.134 Thank you, Caitlin. Hello, everyone. Welcome. First of all, I’m happy to be here, very excited to be here. I want to mention one thing before I get started. The Flux training that Caitlin just mentioned, it’s just phenomenal and awesome. There’s plenty of great feedback that I personally received from customers and users, that it just was the right sort of content and to be able to go to there. So don’t miss the opportunity. All right. So why are we here? We’re here to talk about Cloud road map. I thought, when they said, “Hey, provide an update,” me being product manager, I’m just going to say, “Okay. Hey, let me show you what the vision is and what we are doing now, next, later.” So without further ado - just a little bit about me. I’ve been in the industry quite a while. Worn many different hats, development, support, customer success, and product manager in bunch of different domains, monitoring, automation, just testing, and so on. I joined InfluxData about three years ago. I’ve been focused mainly on what we call as Cloud 2.0, but it’s really Cloud. So I’m the PM for that. I also run another project, Lighthouse, which is what we call it internally. It’s basically a program where I work with a select set of customers, strategic customers, on board them onto Cloud 2.0, being new and all. At the same time, this gives me an opportunity - or us an opportunity to get some feedback and very important signals that we need to be looking for in terms of building the product road map.

Balaji Palani: 00:02:20.129 Please feel free to contact me. I’m not shy about talking to people, especially customers, users. If you have a question, my email is there. You can also DM me in the community Slack. I’m going to put another thing. So I’m always available on the - book an hour with that Calendly link. I usually run some feedback sessions, so I provide that link. And I’m going to be doing managed functions. I’m going to talk about that a little bit in my slides later on. But book an hour if you want to just talk about the road map or even want to provide some feedback, always here to talk about that. All right. Without further ado, why are we here today? The objectives for today’s meeting is really to provide a road map update. I’m going to be mainly focusing on who, what, why, and how. Typically, I don’t want to mention anything about when, but I know that folks are very interested, so I’m going to be talking in context of, hey, what we’re working on now. Now means it’s something that it’s already out or it’s immediate, imminent shortly, will be available soon. Next is something it will be available within the quarter. And later is something, hey, you have six months out or greater than six months, so. All right. So who and what are we focused on? So we are focused on developers, builders, operators. If you have an application or if you have a application that emits timestamped data or you have a set of devices, IoT, servers, sensors, or even infrastructure servers, what have you, which emits, basically, timestamped data, right, things which move or change over time, you want a really fast, scalable performance database on cloud so you don’t have to manage it, you don’t have to worry about the operations.

Balaji Palani: 00:04:26.597 And yes. You are our focus. We really are building for you. And some of the things that you will see is we really care about developer happiness, in other terms, developer productivity. What you will see themes like teams and collaboration where we really care about - InfluxDB is a platform for not just one person, but you as a team. As you build your application, as you roll your application out, how you bring them together without getting over each other. And then make it easier to build on the platform, really. I mean, we believe that the platform should do a lot of the things easily so that you can just focus on the application building. So that’s something I will be focusing on. Time to Awesome. It’s all about time. Right. Time is a cliche. But basically making data acquisition easier, including faster data movement in or out, either two queries or you are trying to run a task which exports data to a different pipeline, all of those things are focus areas for us. We also want to enable higher cardinality use cases. There are use cases where today you might think about, hey, I really need to lower my cardinality because it’s going to impact - it’s going to [inaudible] and so on and so forth. But we are working on something really cool, like the next-gen platform, which is going to enable even those high current multiuse cases and really to be fast in terms of how you want to go. And last but not least, run where you want and pay how you want. Basically, this is about distribution of your workloads. We’ll talk about where our clusters are being launched in different regions and we have some billing options and how you can control your bill, control what you pay for, and so on so forth.

Balaji Palani: 00:06:21.901 So that’s about it. So these are the themes that you would see for the road map. So I’m just going to focus on them. This is a very high-level bird’s-eye view of what we are looking at now, next, later. I’m going to be talking about each of these in detail in a couple of - in the next few slides, mainly on the now and next. So if you look at the teams collaborations, right. So we started with notebooks and dashboard annotations. That was a foundation step. It’s like an MVP, but there’s tons to do. For example, we want you to be part of multiple accounts. Or you want to be able to create multiple organizations within the account so you can focus on - you can separate development, staging, and production, enterprise SSOs, keys, single sign-on, maybe you have an Azure AD or something that you want to connect into. Just be aware that Enterprise SSO is only offered with annual deals for now, but that’s something we can work with you. And later, what we’ll come to is more role-based access control or deeper actual default roles. And in terms of making it more automation to achieve the GitOps. GitOps is basically how we see all of the tasks and resources as code that you develop. And you really want version control or you want better CI/CD pipeline integration and so on, that would be part of that. Then making it easier to build on the platform. Things like observability. We want to give you visibility into your usage, your cardinality, if there are errors. You really want the platform to do more rather than you do the stuff outside of the platform. So things like named functions, triggers, and managed function. And I’ll go over each one of these in detail on the upcoming slide.

Balaji Palani: 00:08:25.212 And what’s coming later is schema, buckets. Again, our vision is to become like a polyglot platform. Polyglot, speaking many languages. We have InfluxQL and Flux in terms of queries. We’re looking to add SQL later on in the year. And that’s about that. And then making data movement easier. It’s really Telegraf. Telegraf is like an awesome agent that you install, and we support a whole bunch of like 200+ plugins that you can either ingest in or operate on or do some aggregation and output, so it’s really an awesome tool. Open source, again. So there’s constant [inaudible] that we do as part of Telegraf 1.18, 1.19. And then we also have bulk data import and export. Then we have some Cloud Native integration. I’m going to be talking about that. And then archival of the data. IOx, I think you might have heard about IOx a lot. So IOx is a next-gen storage platform. It’s really about storing your data in memory for [inaudible] queries and then backed by S3 for [inaudible] files for later access or if you want to push it into a pipeline of yours, so those kind of things. And this is what will enable high-current multiuse cases as well. In terms of global distribution, we are already on the major regions in Americas and Europe. We want to launch in Asia-Pacific and close some of the other regional gaps that we have. And billing, we want to start offering the annual offers in marketplaces. So I’m going to talk a little bit about that later.

Balaji Palani: 00:10:11.042 All right. So what’s available now? So again, this means that it’s either already launched or it’s going to be made available to you pretty shortly. Right. So notebooks is the first thing. I don’t know whether folks had a chance to catch Russ’s webinar. Russ, my colleague in the product management side. The team is really on teams and collaboration, but we view Notebooks as kind of a single, consistent building experience. That’s the foundation for building experience, for querying any sort of things. You want to create a task or you want to create a Flux script or you want to create a dashboard cell or when we come out with a named function or managed function, you want to do that. So Notebook’s is kind of what we think about as a Lego building block for you. And what you’re seeing today, Notebooks is really the MVP version. Right. So there’s tons of things that we could add. And if you really have a - if you’re passionate about it, if you have a lot of feedback on it, we are open to those feedback. So reach out to me, reach out to Russ, or you might see our design team, Molly, asking for feedback. So throw up your hand and we are happy to talk to you.

Balaji Palani: 00:11:37.461 This is a simple feature, but very powerful in my opinion. Because so far in our UI dashboards were without context, you have to really [inaudible], choose your name carefully, measurement names, or really do some [inaudible] functions for it to show up. And this, in my opinion, really allows you to share the context really quickly. You can click on any line graph and say, “Hey, can you investigate this?” to another colleague. Or you want to say, “Hey, this is a deployment,” and really pass on some powerful context there. And more are coming in this, like range annotations, custom colors, and so on. So we will continue to add stuff to this. This is not available or enabled for everybody yet, but pretty soon. It’s actually making its way. Check is in the mail sort of thing, so. This is tentative. Again, this is also not available yet, but making its way out to everybody. We really started on this journey goal. I mean, to be fair, right. So we do provide usage information for everybody, all the users, you go to usage tab or usage menu, but basically, that was not enough. There was so much feedback that, “Hey, I need to look at hourly,” or, “I want to set up alerts. I want to get things. I want to get notified when I’m close to the cardinality limit and not when it exceeds it.” So all of this feedback was well heard and we started to work on this as how do we provide a self-service access to usage and operations?

Balaji Palani: 00:13:27.071 And I believe that what you would see shortly is - so there’s a set of API that we have enabled. Usage is constantly collected in system buckets behind the scenes and we are providing Flux functions that you can use either in a dashboard cell or in an alert or a task that queries usage directly. And you can do all sorts of things with it. We believe that we want to use this for additional things today. When we launch, I think you will see usage. But soon we want to give you things like - by the way, usage plus, it’ll give you cardinality and your limits. So you can also query for your cardinality limit or your write limit and really know the difference. Right. You can set up alerts saying that, hey, when it approaches 80 percent can you alert me or can you send me a Slack message or something that. We also want to use it as a foundation for write errors, query logs. We receive requests for, hey, I want to see all my queries, our top 10 queries, and those kind of things. So we are planning to make those available in a similar format. So you can query them, you can access them. And this will also be a way for you to access audit logs and access log. Those are a little bit down the line, but I think we have a very good framework now. Now we can really push in a lot of the data observability and visibility, transparency data that developers, real users, are looking for in terms of how they can build better on the platform. Going to be available shortly.

Balaji Palani: 00:15:19.111 Telegraf 1.18 - this was released in March. Basically, a lot of plugins, I’m not going to go through each one of these, but specific mention to some - there are some cool aggregator plugins. So if you want to - if you are using it as an edge device and you want to do some derivative aggregator [inaudible], these are something really cool. And then some of the customer requests were taken care of like directory monitoring input plugin, NFS input plugin. Again, we’re just constantly chugging out releases for Telegraf. If you don’t see something you are interested in or you’ve been opening PR for it or whatever, reach out to us and we’ll work with you to put it into a next release. So what’s next? What’s next is, without further ado, I want to launch into teams, collaborations, multi-account, multi-orgs. You may be aware - you might be one of those lucky ones who have had this problem where you already have a personal account and your team invites you to be part of your corporate account. All of a sudden you’re like, hey, you can’t really have one account - I mean, more than one account associated with your email. That is all going to go away. We’re going to have a single email that can take in as many accounts as you want. Simple example. You see me and then I have my personal account and then I have my InfluxData account. I can also be a part of any customer’s account if they invite me. And they can also restrict me to certain orgs within the account. Again, which enables not only at an account level, now we also are going to be enabling multiple organizations. So you can create an organization for your production. Organization for your staging. Organization for your development.

Balaji Palani: 00:17:15.969 What that allows you to do is really have some separation in terms of resources. Also, it allows you to secure them. I mean, we only have owner role today, but we’re going to be adding soon viewer, member, and those kind of things. So you can actually say, hey, I want this user to be in the dev org, but not in the prod org. Or maybe he wants to be in the prod org, but only with a read-only permission. So all of those will be possible. It’s going to be in stages. The first stage is going to be we will enable the user to be a part of multiple accounts. And then we can enable the user to be part of multiple organizations. And then the last step would be we’ll have at least five or six out-of-the-box roles, things like admin and viewer, billing, some specific major roles that gives you specific access. We think 80% of the use cases would be complete with that. But again, if there are things that we need to add, custom roles will come out later, so more fine-grained permissions. That’s in next, which means that that’s something that we’re working on in the next quarter. So again, these will be in phases. You will see them. Moving on into making platform easier to build. This is what I call is how do we enable the platform to do a lot of the heavy lifting? Right. So you don’t have to do it. You as a developer can just focus on developing your application.

Balaji Palani: 00:19:00.253 Triggers is one example. We are focused on triggers as next. In order to explain triggers, you know that today we can run tasks. Right. Task is anything which is Flux queries, that could be run in a schedule [inaudible]. So you can say, every hour, go fetch my data, and then downsample it, create a min/max mean or whatever, and then push a single or multiple aggregated values into a different bucket. So that could be a scheduled task. Or maybe if you want to do an alert check, which is running every five minutes, it’s also built on top of our task system. So it’s like, every five minutes you’re checking for some threshold and then you’re notifyied. Right. So that’s what is there today on the top right. What we are trying to build with triggers is, hey, there are things which could be - because you only look back certain time interval. Right. So they could be later [inaudible]. Or they could be something which you need to get alerted immediately, before even it hits a storage system or just within seconds or something like that. So you can think about these as events. Right. You can register an event, register a trigger on, hey, any writes to a bucket X, fire off. And then you can also put in some predicate values. You can say measurement, blah, blah, blah, and then tag values, and so on. And then you can say, hey, run this task. What that does is it only fire the task when those conditions are met. That is writing to a bucket and then those predicate values or the filter conditions are met, then you basically run the task and that could trigger like that.

Balaji Palani: 00:20:53.099 You could say, hey, I’m going to downsample this late arrival data differently. Or it could be a use case where, oh, I see the data. I need to do some quick notification. Right. So I mean, it enables some simple use cases. We can’t really go and query another storage and try to map the value and do all that. But you can do a very simple threshold-based [inaudible]. Right. So you can get the data, do a simple comparison, and then notify. So that’s what triggers is for. Some use cases are like the late arrivals and streamlined learning is super excited about this. And we will work with customers on this, so. But if you are interested in this and you want to talk to me about it or talk to us, reach out to me. Really want to hear your feedback. Second thing on the make platform easier to build is the named functions. You can think about named functions as - again, from a task or any kind of dashboard cell, what we’re doing is really Flux code, right, it’s a query. So the idea here is code reusability. We are saying that you can define your Flux code once, you can, say, write down your task logic and then register as a named function. When you register as a named function, you might have to put it in some parameter. You have to indicate that, hey, this is a parameter. Your schedule is a problem. Or your measurement name is a parameter. Or your bucket is a parameter. Once you do that, you can actually use the function and invoke it multiple times. And guess what, you can actually create your task based on that function. You can create your trigger based on the function. You could query your dashboard cells based on the function and passed different parameters. So it’s really a very powerful way of reusing your code. And we have many, many customers who are using multiple alerts with the same kind of - similar kind of logic, but just changes in the threshold. So this will unlock all of those things.

Balaji Palani: 00:22:55.699 You have a central named function. You have a single named function. You can build a bunch of tasks, just change the arguments. And when you have to change your logic, you just change once. Right. So really powerful pattern here that we are trying to build. The first step will obviously be something simple. We’ll say, hey, you put in a function, you put in some option params, and save it. And then you can invoke it in any of these tasks or triggers or dashboard cells. Again, same thing here, this is something we are building on next. If you’re interested in talking to us about this or wanting to learn more, just contact me, set up some time with me, we can go over. Next is managed functions. I know it’s kind of confusing from a name perspective. But you can think about managed functions as, let’s say, we’ve heard from users and customers that InfluxDB worked great for ingestion and queries. But when you’re building your application, you have to format the data before you push it and convert it into simple line protocol. That’s one use case. Or maybe you want to do some pre-aggregation before you push it in. That’s another use case. So any pre-processing that you want to do. Or maybe, hey, you are building an application that expects JSON. We output CSV or table format. Right. So you might want to post-process the data that’s coming from the queries and then change it or convert it into a JSON or some other format that could be readily usable by your applications or your team.

Balaji Palani: 00:24:42.905 I’ve spoken to users who have built API gateways in front of InfluxDB Cloud, which actually does simple things. Right. So you don’t have to worry about Flux, you don’t have to worry about anything. You just pass on some parameters and expect some values in some format. Right. So these two main use cases, we think - again, we’re going to start with an MVP, which is going to be Python, but you can basically write your Python code and run the mass managed function, which will run within their own container space and they could be available as an endpoint. The endpoint is created for you. You could just either use your sensors to directly write to that endpoint or make your users or build your API gateway onto that endpoint. So you can do a lot of things. There’s also another thing that this doesn’t talk about. Sometime before that, hey, the same Python code. I just love Python. I’m going to put it in. But why can’t we run it as a task, making our schedule tasks similar to the Flux task? That’s also a possibility. The first thing is we want to enable this. And obviously, this will be something that you as a Python code will run in that container space that we have to enable in a very secure manner with the secrets, with the tokens, and everything. So again, this is something that we are working on next as well. If you are interested, please talk to me, just book some time with me. We are, again, excited about this.

Balaji Palani: 00:26:22.724 The other thing that we are excited about is making data movement easier. So first thing is data import and export. We know we have users coming in from either open-source - hey, I want to move to Cloud because it’s simple. We have production data. Want to make sure it’s always running and so on. But we have data here. So you have data. Maybe you have line protocol data sitting there or CSV or maybe another open-source or even enterprise, which is TSM files (1.x). You should be able to import that data. Again, these are terabytes of data. You try to import them through the front API, it’s going to be really slow. Right. I mean, you have to first of all write some code to really do it. We can really make this faster. Basically, take this, for example, TSM, push it in into S3, and basically start the process. And there’ll be a background process, which we’ll just import it using your token into your org. And we can just import it into [inaudible]. And once it’s done, you’ll be able to use it. So that’s bulk import. We’re planning for TSM first, line protocols, CSV, and then Parquet is other things that we are looking at. Bucket backup. These are things which you just want to - you have a set of buckets or you have a set of data, you want to periodically back up for archival or for maybe disaster recovery purposes. All of these are something that we want to enable. Again, the next portion is this bulk import. And then bucket backup and bulk export will come later. Meaning this is what we’re working on this quarter and then the next one is six months down the line.

Balaji Palani: 00:28:17.846 The other one is on making Cloud integration really easier. We’ve heard from users, they have data sitting in either AWS Kinesis, Azure event hub, or IoT hub - what have you. Today, if you want to get them in, you have to at least use a Telegraf. There are other ways to do it, but one of the ways would be install Telegraf, use the write plugin, fetch the data, and then push it into InfluxDB. Telegraf is a great tool, but again, if you are a Cloud-native - if everything you do is Cloud-native, you’re just using serverless, it’s really hard to find, or just for this, you have to create an EC2 or find a compute resource to run your Telegraf. So this just allows you to connect directly to your Cloud-native messaging service and ingest that periodically. So you can directly ingest it using supported formats into specific buckets. I mean, that’s what we’re shooting for. Last but not least, Telegraf 1.19 is something that we are working on next. Focus is going to be on SQL. Check SQL input plugin, PostgreSQL server. And then there’s also a bunch of parsers and some input plugins. Again, Telegraf, constant releases that we do. Global distribution, run where you are, pay how you want. I mean, our vision is really to enable this to be made available everywhere. On all three major cloud providers, that we already done over the past two years. We have established in at least the super regions, what we call North America and Europe. APAC is next. So for example, we just launched to EU, Belgium on GCP. So folks, if you’re looking for that, we have been in beta for the past four weeks. I think it’s made GA just yesterday. So this is new. If you’re looking for that to run your production [inaudible], you can.

Balaji Palani: 00:30:18.155 And the other thing is we’re looking for AWS Sydney. That’s going to be a first APAC region. There are other APAC also that we plan to do. It’s going to be based on some business decisions. We’re going to look at demand and really look at the use cases work with customers, but definitely AWS [inaudible] Sydney is something that’s next, coming up pretty soon. And Azure Europe node, Ireland, a bunch of companies and customers are in Ireland, so those will serve that. Last but not least, marketplace. So over the past year, we have been enabling a product to be on the marketplaces so that you can have a single bill, which is your Cloud bill that will include your InfluxDB usage spend on all AWS, GCP, and Azure. What we would be working on later this year is allowing annual private offers to be offered over all of these marketplaces. Annual deals today are offered, but you got to become a direct. Meaning you got to provide credit card authorization directly. What’s not available is on the marketplaces, we only have pay as you go. So would allow us to provide you the option in case you decide to say, hey, I’m willing to commit to some spend, then you can choose an annual offer and you can start with that.

Balaji Palani: 00:31:54.199 In summary, we really feel like - I mean, we strive to really ensure that our roadmap is closely shaped by our customer input, and the project, Lighthouse, that I mentioned earlier was key to that. We still have that in place. We onboard a lot of customers and use that as an opportunity to talk to them about their use case, get some iterative feedback, give them early access. Design constantly want to talk to folks, users. So again, we are hungry for feedback. Right. So if you want, currently right now, I’m running a feedback session on [inaudible]. If that’s interesting to you, use [inaudible] book your session. Don’t worry about managed functions. If you feel like you just want to talk to me about anything, about feedback in general, I’m happy to talk to you about that. Just find 60 minutes on my calendar and we can do that. All right. That’s my slide. Thank you very much for listening. And again, these are all the ways how you can contact me. One more thing, I think again, I’m going to mention that, please look at InfluxDays - it’s super, super cool. There are some cool sessions coming on. So look for that.

Caitlin Croft: 00:33:26.457 Awesome. Thank you, Balaji. Yes. Would love to see you all at InfluxDays. It was really cool last year at InfluxDays, our European edition. We had people from around the world. We had attendees from six out of the seven continents, which was really cool. So hopefully we can do that again this year. So we’ll just give everyone a couple of minutes. Any questions that you have for Balaji related to InfluxDB Cloud or just anything in general, feel free to post them in the Q&A. This session, once again, has been recorded and will be made available later today. And probably by tomorrow morning, definitely it will be available. Balaji, if someone wanted to get started with InfluxDB Cloud, what would be the best way to get started?

Balaji Palani: 00:34:14.874 The best way to just go to Cloud 2.0 at influxdata.com and sign up for a free tier account. It’s completely free, we just ask you for your email, and maybe a name, company, and you can just sign up. And there pretty cool content available to really on board you. And we’re always here to reach out if you have any questions, so.

Caitlin Croft: 00:34:42.936 Cool. And are there any tips and tricks for anyone who is just getting started with cloud? Is there something that you’ve seen customers and users have issues with when they’re signing up or getting data in?

Balaji Palani: 00:34:57.673 Yeah. Typically, the data part is the most - depending on your use case, right, it’s probably where you probably need the most help. I mentioned Telegraf. So Telegraf supports more than 200-plus plugins. It’s super easy to set up. It really does a lot of the heavy lifting that you want to do. The other thing I want to mention is InfluxDB is great with APIs as well. If you’re building your applications, we have some really - I think 8 or 10 client libraries in the language of your choice, Python client library, Rust, and so on, so. Where you can just use it and include the package and start pushing data in, querying it. We make it easier for you. We strive to even do it better. Again, always open to, hey, if their issues or things that we can be doing better - really want to do that.

Caitlin Croft: 00:35:57.362 Cool. Awesome. Well, it doesn’t look like anyone else has any more questions, but feel free to -

Balaji Palani: 00:36:02.381 Quiet group.

Caitlin Croft: 00:36:03.631 Pardon?

Balaji Palani: 00:36:05.272 I said quiet group.

Caitlin Croft: 00:36:07.223 They’re just so shocked by everything that you talked about. Maybe they’ll have questions later on. Feel free to ping Balaji. You can always ping me. Everyone should have my email address. So if you have any questions, feel free to email me and I’m happy to loop in Balaji. Thank you, everyone, for joining today’s webinar. It was great seeing you on here, and it will be made available later today.

Balaji Palani: 00:36:31.720 Thank you. Thanks, everyone.

[/et_pb_toggle]