InfluxDB 3.0 Product Roadmap and Update

Session date: Sep 24, 2024 08:00am (Pacific Time)

InfluxDB is the purpose-built time series database. We’ve been busy adding more features and functionalities to the InfluxDB 3.0 product line—InfluxDB Cloud Dedicated, InfluxDB Cloud Serverless, and InfluxDB Clustered. InfluxDB 3.0 is built in Rust and sits on top of Apache Arrow and DataFusion. Apache Parquet is an open source columnar data file format chosen as the persistent format.

Whether you’re an InfluxDB pro or just learning about the time series database, this webinar will provide an overview of 3.0, introduce new features, and preview what’s next.

In this webinar, Gary Fowler, VP of Products, will dive into:

• An InfluxDB 3.0 product line overview – learn which version is best for your needs
• Key features and improvements
• A product roadmap – your guide to what’s coming up next
• Plus, a live Q&A

View the Presentation

Watch the Webinar

Watch the webinar “InfluxDB 3.0 Product Roadmap and Update” by filling out the form and clicking on the Watch Webinar button on the right. This will open the recording.

[et_pb_toggle _builder_version=”3.17.6” title=”Transcript” title_font_size=”26” border_width_all=”0px” border_width_bottom=”1px” module_class=”transcript-toggle” closed_toggle_background_color=”rgba(255,255,255,0)”]

Here is an unedited transcript of the webinar “InfluxDB 3.0 Product Roadmap and Update.” This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors. Speakers:

• Anais Dotis-Georgiou: Developer Advocate, InfluxData
• Gary Fowler: VP of Products, InfluxData

ANAIS DOTIS-GEORGIOU: 00:00
Well, welcome, everyone. Okay. I do see the numbers kind of stabilizing here, so I’ll give Gary a chance to get started. So, today’s webinar is on InfluxDB product roadmap and updates. I’m really excited for the presentation today. And just one more time, a copy of these slides and a recording of the webinar will be made available to you after. So, I hope you enjoy the webinar today. And Gary, go ahead and take it away. Thank you so much.

GARY FOWLER: 00:30
All right. Thanks, everyone. Good morning, good afternoon, good evening, wherever you are. Really appreciate you joining us today and giving us a chance to tell you a little bit about what we’ve been working on and what we’re working on next. So, here’s what I’m going to talk about. First, I’m going to take you through some of the things we’ve done in 2024 so far, in case you missed it. We’ll talk a little bit about some Enterprise and Telegraf releases. Then we’ll move on to where a lot of our development is focused right now is on our 3.0-based products. We’ll talk about those. We’ll talk about some short-term roadmap items. Some things we expect to land fairly soon. We’ll talk about some midterm roadmap items. Things we expect to land shortly after that. And then we’ll also talk about some long-term roadmap items.

GARY FOWLER: 01:15
So first, well, a lot of our new feature development time has gone into 3.0-based products. I just wanted to make sure everyone knows; we’re still very much selling and supporting our previous product line. No plans to stop that anytime soon. We have new customers on Enterprise all the time. Of course, Enterprise or Telegraf is popular across the board. Even if people aren’t using InfluxDB, Telegraf is widely used. We continue to make releases on both of them. So as an example, for our Enterprise product, we’ve had two maintenance releases that included a whole number of bug fixes. Too many to list on this slide. But if you go to our release notes pages, you can see the specifics. So, we’ve had two releases so far this year. We’re now working on another release that’s going to include some bug fixes, but possibly a new feature or two. So, we haven’t decided on the release number yet, whether that’s going to be V1.11.7 or if it’s going to be 1.12, but that’s planned for either later this year or early 2025. We’ve had nine new Telegraf releases this year, incorporating several bug fixes and dependency updates, but also several new plugins. So, you can see here there’s a few new plugins. Some contributed by the community, some contributed by our own developers. So quite a lot of new things happening in Telegraf. Now, I don’t have a slide for it, but we also continually maintain our Cloud 1.0, Cloud 2.0, and Cloud Serverless platforms. As an example, we recently made a change in Cloud Serverless to better handle partial writes. So, I just wrote a blog about that a few days ago and published it. So, you see, we’re continuously making changes to those platforms as well.

GARY FOWLER: 03:03
So, let’s talk about what we’ve done so far in 2024 with our 3.0-based products. First, we’ve added some management and token APIs. If you haven’t seen it for Cloud Dedicated product, we’ve released new token and database management APIs. This lets our customers automate the creation of databases and tokens and build it into their own system. You can still, of course, do this with the Influx Cuddle command—the command line tool—like you always have, but now you can also do it programmatically. We think that was a nice addition. We’ve made quite a lot of performance improvements this year. So, a big focus of our development in 2024 so far has been in performance. We’ve probably weighed a little heavier towards performance than new features. I think we’re going to start getting back to a little bit more balance here, but we’ll continue to improve performance, continue to add new features.

GARY FOWLER: 04:03
We’ve seen big improvements this year in query performance when scaling large numbers of simultaneous queries. This work that made a dramatic improvement there also improved our query consistency. So, if you were one of our first customers to take a look at 3.0 and you saw some inconsistent query performance, we think you’ll see a lot better performance now. You’ll see more consistent numbers, and you’ll see us being able to scale as intended. Where you have a whole great number of simultaneous users, you still have that great query performance. We’ve reduced query latency across the board, especially when using custom partitioning. So, if you’re using custom partitioning, we’ve gotten most types of queries to be low latency. In some cases, we’re down 20, 30, 40 milliseconds. Very low latency queries. We’ve had several other miscellaneous optimizations that’s helped along the way. This has happened both in our own code but also, we have a whole team that contributes to the Apache DataFusion project and helps make performance improvements there as well.

GARY FOWLER: 05:16
Specific to our InfluxDB Clustered product, this is our self-managed version of our distributed 3.0 platform. We did add Helm chart support for those of you using Helm. I know for some of you that use Helm as your deployment method, this streamlined things and made it a lot easier for you. So that is available now. You can download our Helm chart. We continue to make other enhancements here and look for ways to make our Kubernetes-based deployment of InfluxDB cluster simpler and faster. Generally, the feedback so far in InfluxDB Clustered is once people get it up and running that they really like it, it’s worked well for them, but getting it set up is taking a little longer than they would like. It’s a little bit more cumbersome. So, we’re looking at ways to streamline that. We’ve also stopped throttling POCs for Clustered and removed the limited availability designation we had on it. It’s now fully GA. Anyone that wants it can order it. So, we’re excited to see that.

GARY FOWLER: 06:20
For Cloud Dedicated, we added operational dashboards this year to provide a way for our customers to view the health and activity of their clusters. It allows you to see things like CPU and memory utilization statistics on your workload. We can make that data available to Clustered customers as well if you have a running Grafana instance that can access the data. For Cloud Dedicated customers, we spin up an instance of Grafana and allow you to use it right from Cloud Dedicated. For Clustered customers, we can make the data available to you if you have a running Grafana instance that can access it. We made some security improvements as well. We added single sign-on support for Cloud Dedicated, which allows you to use your own identity provider for a login to Cloud Dedicated. We also added parameterized query support for SQL. So, two important features there.

GARY FOWLER: 07:20
All right. So that’s just some of the highlights of what we’ve done so far this year. Now I’m going to dive into what we’re working on right now and what we think is going to land in the next quarter or two, which is what I think everyone is interested in that joined today. So, first couple of items, here are some much-needed features that many of you have been waiting on. While you do have the ability to delete a database with 3.0 right now, we’ve had this really odd limitation where you can’t turn around and create a new database with the same name. That certainly was not our intended design and a weird quirk we’re embarrassed about, but we’ll soon be correcting that behavior. Secondly, we’re adding the ability to delete tables as well. We want to make sure we allow— when we do add the delete table feature; we want to make sure that we do allow it to be rebuilt with the same name as well and not have the same issue that we have, the database deletion. So, when we release it will have that ability as well. It’s certainly taken a lot longer to get to this than we previously thought. There are a few reasons that I won’t go into here, but good progress is being made on this, and we expect this to land in Q4, and hopefully early in Q4. So not too much longer to wait on this since we’re nearing the end of Q3.

GARY FOWLER: 08:40
We’re also working on an administrative graphical user interface, a GUI, for a management interface for InfluxDB. So as many of you know, we’ve had GUIs for previous versions of InfluxDB, but we haven’t really had one for Cloud Dedicated or Clustered yet. You might ask, “Hey, why are you building this since we already have a CLI and API ways to administer InfluxDB?” That’s a fair question. There’s a number of reasons for it, but the main two reasons really first reason has been a common ask for our customers simply for ease of use and ability to get started. Second, we have some features planned that will have a little bit more complex configuration, like ABAC, attribute-based access control, which I’ll talk about in the next slide. It’s going to be much easier to configure with a GUI than just a command line interface. So, we’re excited about this UI. We’re in the middle of it right now. We have some pages created already others still to do. This is likely going to land for Cloud Dedicated first, but then we expect it to be available for Clustered customers shortly thereafter. Eventually, it’ll be available in all our 3.0-based products. I mentioned a slide ago, attribute-based access control. Look for ABAC-related features to start coming out over the next couple of quarters. Many of our Enterprise customers have been asking about it. We’ve been working on it for quite a while already, and now we’re starting to see it come together. So, I’m excited about it. I was getting a demo of some of it from our engineering team just a couple of days ago, and it looks really promising. Very excited about it. So many of our Enterprise customers have been asking about it, and it is coming.

GARY FOWLER: 10:31
All right. So now let’s talk about things on our mid-term roadmap. When I say mid-term, we think these are going to land in the next two, three quarters, right? So, they’re not necessarily imminent, not necessarily Q4 items, but we think shortly thereafter we’ll start seeing some of these things land. Let’s first start with our 3.0-based open source offering, as I know a number of you may be here to learn about the status of open source on 3.0. So, we are committed to this effort. We’ve had a team working on it full time in 2024, and they’re making really great progress. We did increase the scope of the project slightly. We wanted to incorporate some of the learnings that we’ve had with our commercial products on 3.0 so far. So, with this increase in scope, we’re probably not going to get this out in Q4 like we had originally planned, but we think that we’re going to start getting it into the hands of the community to play with in early January. So, a little bit of a delay, but not too much. And I think everybody will be happy with the increased scope. Really excited about this project. I’ve been running dev builds on my laptop, playing around with it, and I’m very impressed. Very simple to install and get running. Just takes a few seconds. Just very easy to work with. Very streamlined. Very fast. So really excited about it.

GARY FOWLER: 11:52
Another thing that we’re going to be working on, or are already working on, that you’re going to see some features released in the next couple of quarters is kind of a generic thing that we’re referring to as customer-focused observability. So, as I mentioned earlier in the year, we released operational dashboards to give insight into things like cluster help and things like memory and CPU utilization, but we also recognize that we still have a lot more to do in this area. We want to give you a lot better visibility of the database level, not just the cluster level. We want a quick way to show you storage stats like number of bytes, number of tables, last data written in date, first data date. Just a few things to help you size and plan for the future. I don’t expect this to be a big bang feature per se. Meaning there’s not some feature we’re going to plop down and say, “Hey, customer-focused observability is available.” This is going to be a collection of miscellaneous improvements over the next two to three quarters. It’ll happen, probably in three different areas. There’ll be enhancements to the operational dashboards that are already in place, some additional dashboards or changes to those dashboards, or there’ll be features in the admin UI that I talked about a couple of slides ago, or they’ll be part of our Influx Cuddle command set. So, we certainly recognize we need to do better in this area, and I’m excited to tell you we’re on our way.

GARY FOWLER: 13:22
Another thing that I’m really excited about and would like to talk to you about is what we’re doing around Apache Iceberg integration with third-party tools. So, for the past several months, we’ve been running a successful POC with one of our great customers in EMEA, which allows them to query InfluxDB directly from Snowflake. So, we have a number of customers that are using lakehouse type products, and they would like to be able to— from the tools that they use with those lakehouse products, be able to access information that’s in InfluxDB without having to make a big copy of the data or replicate all that data into a Snowflake or something like that. So that was what this POC was about is helping that customer do that. We’ve been now working to officially productize this and make it available to all our Cloud Dedicated and Clustered customers. The initial target for this is Snowflake. That’s where we’re seeing the most demand. We expect that to be released as early as Q4, so we’re excited to offer that to folks. But we also plan to support a little bit more generic Iceberg API implementation, which will allow it to work with many different products but also support Iceberg. So, we’ll be working on that shortly after the beginning of the year and we get this Snowflake support out. Delta Sharing, for those of you that are Databrick shops, is also in our plans. We’ll likely start that work after we complete the Iceberg project. But something that a number of you have been asking about and are looking forward to, so we’re excited to say we’re getting pretty close on it.

GARY FOWLER: 14:58
All right. So next, I’d like to talk about some things that are on the longer-term roadmap. They are planned or already in progress, but we know it’s still going to take a little while before they make it to the release. Two, three, four, five quarters away. We don’t have exact dates for some of these things yet but just want to make sure that you know that they’re planned or we’re working on them. So, first up here—and these are just highlights—there’s a lot more things than what I’m listing here, but here are some of the highlights. So first up here is our processing engine. This is going to give you the ability to run Python code at the database level so that you don’t have to ingress, egress data out of the database to do any kind of transformation or other processing on it. So, we’re excited about this. It’s going to replace some of the functionality that came with Kapacitor in our V1 Flux tasks and version 2.0. But we think it’s even going to be more flexible. We’re going to add more hooks and different things that you can do with it to make it even more robust. So really excited about that.

GARY FOWLER: 16:03
Row-level data eviction is work that has started. Just this week, I heard it might make it into Q1. Don’t know that for sure yet, but there’s at least a chance this moves along from— that it moves from long-term to mid-term. So excited about that. Something that everyone wants, along with the table and database deletion that we talked about earlier. We’re also planning on some self-service backup and restore tools. So, this will give you the ability to control, even with Cloud Dedicated. If you want to take a backup of your data and you want to store it away somewhere and then restore it later, we’ll have some tools to be able to do that. We’ve also heard that there may be a Flight SQL ODBC driver coming from the Apache community. We’re excited about that because that would give us Power BI support. We know some of you that are Microsoft shops are interested in Power BI. We already support Tableau, and certainly for visualizations and things, we support Grafana through our partnership with them and Apache Superset. There’s a number of products that we already work with. There is a JDBC Flight SQL driver out there, so we can work with most JDBC-compatible packages. But for Power BI, we need ODBC. We’ve heard that there is a Flight SQL ODBC driver coming, so as soon as that becomes available, we think that we’ll be able to offer Power BI support.

GARY FOWLER: 17:35
We know we need enhancements to auditing and log viewing, so that’s something that’s planned. Give you even more visibility. This kind of also falls into the observability item that I mentioned earlier. We have some rate limiting at the API token level plan. So that’s something that many of you have asked for, is to be able to specify a certain rate that a specific API token can use so that you can prioritize some users over others if you want to. So, look for that next year. And then we have a new V3 Write API and data model queued up. I’m really excited about it. We started work on it. We have it paused a little bit to get our initial OSS 3.0 version out, but then we’ll dive right back into it. I’m excited about it. I think it’s going to be something that you really like. It’s going to be some new data types included with that including arrays. So, I think it’s going to open some more use cases for you and what you can use InfluxDB for.

GARY FOWLER: 18:40
And then, last and certainly not least, something that our Cloud Dedicated and Clustered customers would like. And that is both auto-scaling and alerting around that auto-scaling. That is on our roadmap. We’re doing the specs for it right now. We expect work to start on that soon. And the way that we think that it’s going to work right now—this could change—is that you’ll be able to set minimum, maximum thresholds that you want for scaling. But when you set those, then we’ll be able to scale up or scale down up to those limits. So, I think that’s going to be something that you really like. And with that, that is the end of my presentation. Thank you very much for taking the time to spend with us today and hear about everything that we’re working on. Really appreciate it.

ANAIS DOTIS-GEORGIOU: 19:35
Thank you so much, Gary, for that presentation. I’m really excited for a lot of those features that are coming out and products. So yeah, I’m excited to see where Influx is headed and all the different features and capabilities that it’ll soon have. So, if anyone has any questions, I encourage you to ask them either in the chat or the Q&A. We do have one question right now, which is can we anticipate open source version 3.0 by Q1 2025? Also, is there any migration tool or methods available to migrate from 1.x, 1.7, or 1.8 open source to V 3.0? And then someone else asked—

GARY FOWLER: 20:12
Great—

ANAIS DOTIS-GEORGIOU: 20:13
—why so much time before OSS? [laughter]

GARY FOWLER: 20:16
Yeah. So, I’ll take the second one first. And I tried to answer that a little bit. We do have a— we are completely dedicated to open source. We’ve had a whole team working on it. It’s just taken quite a while to get put together. We increased the scope a lot. We released our commercial products in the past year. One of the things we found with commercial products was they were great. They delivered the SQL support that we wanted. They delivered the unlimited cardinality that we wanted. They delivered the interoperability—the third-party interoperability—that we wanted. And in most cases, they delivered the performance that we wanted. But there were a few cases where the performance wasn’t where we wanted it to be, especially compared to previous products. So, we knew that we needed to make some improvements there. And so, we want to make those improvements as well in the open source version, so we increased the scope to include those. And so that’s really why it’s going from Q4 to Q1. We do expect that to be early in Q1. We expect it—at least a preview—to be in people’s hands early Q1.

ANAIS DOTIS-GEORGIOU: 21:28
And then we have some more questions following that. Can you say more about what’s included in the increased scope? And will the OSS version support S3 storage, local storage, or both?

GARY FOWLER: 21:38
Yeah. So, we’ll announce, probably in early January, exactly what’s going to be in the open source version compared to our commercial versions. So, we’re not really prepared today to talk in too much detail because some things are still being finalized. But we do think that— we do, at least right now, anticipate that the OSS version is going to support object storage. So, I think that’s going to be great news for everyone.

ANAIS DOTIS-GEORGIOU: 22:09
And then [crosstalk]—

GARY FOWLER: 22:09
I think there was a question about migration tooling as well. We are working on migration tooling as well. Mainly right now, it is for to go from our version 1.0-based products, including open source, and our Enterprise product in our Cloud 1.0. We’re working on migration tooling to go to version 3.0. I don’t know if that will be available for the open source version on day one, but it is something that we plan on doing.

ANAIS DOTIS-GEORGIOU: 22:37
Very cool. And then someone else asks, will V 3.0 OSS be able to run alongside 1.8? Yeah, we’ll start with that.

GARY FOWLER: 22:48
I don’t see why they wouldn’t be able to run alongside each other. I don’t know that for sure, I guess. But I don’t know why they wouldn’t be able to run side by side. You might have to use different port numbers, but yeah, they should be able to run side by side.

ANAIS DOTIS-GEORGIOU: 23:04
And then we also have, when can we expect the Iceberg integrations for AWS data catalog and Athena?

GARY FOWLER: 23:13
Yeah. So, there’s a possibility that you’ll see those in Q4, but I think that we may not have that testing and everything completed until Q1. But I think relatively short-term after we release the Snowflake support, you should see support for those.

ANAIS DOTIS-GEORGIOU: 23:32
And then what are the plans regarding Flux? What will it be replaced with?

GARY FOWLER: 23:37
So, for the processing engine in Flux— so first let me just state Flux is not going away. We’re still supporting Flux. We’re not necessarily being aggressive with adding new features to Flux, but we do maintenance work on Flux all the time. We plan on Flux being around for a long time still. For Flux tasks, what we plan to kind of replace that with is the processing engine that I talked about earlier. So that’s going to give you the ability to run Python at the database level.

ANAIS DOTIS-GEORGIOU: 24:10
I’m super excited for that. [laughter] And then someone else asks, is the plan to still have an Edge version and a Community version in the OSS?

GARY FOWLER: 24:20
I’m not ready to make announcements about exactly what we’re going to have. The plan is still to have an open source version and a commercial version, and there may be some various flavors of that, but you’ll hear more about that in January.

ANAIS DOTIS-GEORGIOU: 24:37
And then V3, will there be full support? Full SQL standard support?

GARY FOWLER: 24:44
Good question. I guess I would say I thought we already had full V3, full SQL support. So, I need to know a little bit more about what you’re asking about. We get SQL support via our integration, our use of the Apache ecosystem. The DataFusion and Flight products. So, we do have developers that are contributing to those projects all the time. And so, if there are SQL things that are missing, let us know and we can work on those.

ANAIS DOTIS-GEORGIOU: 25:21
And we also have a question. Is there any plan to support multi-tenancies similar to Grafana Mimir?

GARY FOWLER: 25:28
Not in the short term. There’s not a short-term plan for multi-tenancy. Not saying that we wouldn’t look at it in the long term, but right now, no.

ANAIS DOTIS-GEORGIOU: 25:37
And then how will InfluxDB V3 OSS be packaged? Will it be a single binary or multiple binaries for each subcomponent?

GARY FOWLER: 25:46
Yeah. So, I’ll answer this as of right now. As of right now, it’s a single binary. I love it. Really quick to install, get running. Literally, within seconds you download, install, running. You have a working InfluxDB version. If you do a PS, you’ll see one particular process being done. It’s possible for new features in the future that could increase, right? There may be something else, like when the processing engine comes along, it could be another process. Not sure yet on whether it’ll be part of it or if it’ll be another process. But at least as of right now, it’s single process and single executable.

ANAIS DOTIS-GEORGIOU: 26:30
And then the very last question we have is, if I want to run a dev build of V3, is there documentation available on how to build this? Build and run it?

GARY FOWLER: 26:38
Not yet. I think early January you’ll see all that data become available.

ANAIS DOTIS-GEORGIOU: 26:46
Sweet. Well, that looks like all the questions that we have so far. So, thank you, everyone, for all your questions and engaged participation. I really appreciate it. And thank you so much, Gary, for answering all these questions and for giving us an insight and detail into the future for Influx data. I know I’m looking forward to a lot of those features. And thank you so much, everybody. And if we don’t have any more questions, then I’ll go ahead and end this webinar. And just as a reminder one more time, a copy of this webinar will be made available to you very shortly via email. So, thanks again, everyone, for joining, and I look forward to seeing you in future webinars. All right. Bye.

GARY FOWLER: 27:25
Thanks, everyone.

[/et_pb_toggle]