Ingest your Cloud Foundry Firehose Metrics into InfluxDB Cloud!
In this webinar, Jack Zampolin will show you how to ingest your Cloud Foundry Firehose metrics into InfluxDB Cloud!
Watch the Webinar
Watch the webinar “Ingest your Cloud Foundry Firehose metrics into InfluxDB Cloud” by clicking on the download button on the right. This will open the recording.
[et_pb_toggle _builder_version="3.17.6" title="Transcript" title_font_size="26" border_width_all="0px" border_width_bottom="1px" module_class="transcript-toggle" closed_toggle_background_color="rgba(255,255,255,0)"]
Here is an unedited transcript of the webinar “Ingest your Cloud Foundry Firehose metrics into InfluxDB Cloud.” This is provided for those who prefer to read than watch the webinar. Please note that the transcript is raw. We apologize for any transcribing errors.
Speakers: - Chris Churilo: Director Product Marketing, InfluxData - Jack Zampolin: Developer Evangelist, InfluxData
Jack Zampolin 00:03.074 All right. Thank you, Chris. So this is going to be a relatively short webinar. There is a lot of supporting material here though and I’m happy to dig into anything in more depth. I’m going to go ahead and drop a few links in the chat bar here, repositories that are kind of necessary for this demo as well as some CloudFoundry documentation that might help you understand the concepts involved here.
Jack Zampolin 00:34.563 So what are we going over today? I’m just going to give you a brief intro to Influx, talk about Firehoses and CloudFoundry for a quick second. Talk about spinning up in instance InfluxDB Cloud and then we’re going to go through how to connect CloudFoundry in InfluxDB. Okay. And here are the links that I dropped in the chat, we’ll go ahead and share these slides after the webinar as well.
Jack Zampolin 01:12.196 So what is InfluxDB? It’s a Time Series Database that’s written in Go. It’s a single compiled binary that has no external dependencies so it’s easy to get started. There’s a SQL-like query language. It allows for events as well as more regular time series like metrics. It’s horizontally scalable, and there’s a full stack of products around it. Kapacitor for learning, Telegraf for collection, and Chronograf for visualization. We’re a complete Time Series Platform. Now, what are people most commonly using Influx for? Most of the time it’s used as a monitoring solution for your large groups in servers or large groups in sensors. There’s a few people who use it for different soft-there’s some trading applications and you can imagine, other time series data like business analytic information as well, so there’s really quite a few use cases. We really shine when the data needs to be in real time and when it’s extremely high volume, so that’s InfluxDB.
Jack Zampolin 02:18.644 What is the CloudFoundry Firehose? So the other piece of technology that we’re connecting here is the CloudFoundry Firehose. This is the diagram from their website, and essentially what the Firehose does is it takes all of the metrics that come out of your VMs that you spin up with BOSH, any application containers, any runner VMs, there’s a lot of counters and other metrics that come out of those and they all get aggregated via the Doppler process along with logs, system logs, and then they get forwarded to a wide variety of services through gRPC through the Firehose. And the way that you connect to that is with a nozzle. So what I’ve done is we’ve got an InfluxDB nozzle here. In this diagram, it says Datadog and they also have a nozzle. But we do as well. Just to note, the Datadog nozzle emits StatsD points. The Influx platform can natively adjust StatsD points through Telegraf. So if you wanted to take the Datadog nozzle, point it through a Telegraf instance, and then persist those metrics in InfluxDB instead of paying Datadog 15 bucks a host, you can easily do that as well.
Jack Zampolin 03:47.365 So the first part of this demo, you’re going to need an InfluxDB instance. So you could download InfluxDB. And run one locally, that’s very easy. In this demo, I’m going to show you how to do-we’re going to run it through the InfluxDB Cloud. And I’ve gone ahead and spun up a free trial instance. Your trial lasts for 14 days. And all you need is an email and you do need to put down some credit card information, but we won’t charge you until the trial’s over. So if you want to follow along, you can go ahead and do that. And then, if you have a CloudFoundry instance, you can run this against your CloudFoundry instance. The only thing that’s going to be different between this and running it on your personal CloudFoundry or your company’s is that the CloudFoundry domain is going to be different throughout this presentation. So I’m using PCF-Dev, which is an entire CloudFoundry installed in the sandbox. The environment on a local VM. It’s pretty easy to install. It does require a few gigs of downloads. So it is pretty network-intensive. And it’s a little slow to spin up. But once you have it spun up, it is very quick to work with. So one of the links I’ve dropped in here is the GitHub page for that PCF-Dev. And to install it, just pop that open. That’s in the links as well. Download it. I think you’re going to need VMware or one of the virtual machine providers on your local machine. And then, you’re good to go. And then, just run cf dev start, log into it, and you’re good to go. You can run cf dev trust. And that would allow you to take that skip-ssl-validation off there. It basically just tells your computer to trust your local PCF-Dev install.
Jack Zampolin 06:09.159 So this, for me, one of the more difficult parts was figuring out the UAA, which is the authentication for CloudFoundry. So I think this is universal authentication and authorization or something like that. The easiest way I’ve found to add these users, a lot of tutorials will tell you to edit a manifest file, which you can do as well. There’s instructions for that in the Read Me, in the influxdata/influxdb-firehose-nozzle repo. But I found that downloading the UAAC client was the easiest way to add these users and these permissions to the different users. So, what you’re going to want to do is configure it, so target it at your local PCF-Dev instance. That’s what that first command does. And then you need to log in, and that’s going to save a config file for your UAA client. Here we’re logging in with the admin user so that we can create other users. Just to note, when you spin up PCF-Dev, the username and password to get in is either user/pass or admin/admin. A lot of the, especially, administrative functions are going to require that you have a client secret. The client secret for the admin user is admin-client-secret. So that is how you’re going to log in with your admin user. Once you’ve done that you can go ahead and add your influxdb-firehose-nozzle user. It does require a few permissions. I’m pretty sure that’s the minimum amount you need. You might be able to pull out that UAA admin permission and just have the scim.write. I tried testing that. I’m not quite sure, but you just need permissions to use the Firehose essentially, which is doppler.firehose, and then you’ll be able to log in.
Jack Zampolin 08:29.387 After that, we’re going to configure the Firehose Nozzle. There’s a couple of files that you need to edit in order to configure that. So it’s at the root of the influxdata/influxdb-firehose-nozzle repo. You’re going to go config, and then firehosenozzle.comf. There’s a few sections in there that we’re going to change, I’ll write over that. The manifest, which describes how you’re going to deploy that in your CloudFoundry instance. There’s a couple of config in there and then there’s a service that scrapes your older applications for their state and the Firehose Nozzle will ingest those state changes and various things through that. So you’re also going to need to make sure that your creds are correct in there. So quite a few configuration things. And then after that, we’re going to deploy it.
Jack Zampolin 09:30.809 So, I’m going to go ahead and share my screen here and just run through this real quick. Right. Okay. So here’s the repo, influx data/influxdb-firehose-nozzle and here is the UAA config I was talking about earlier, that’s one way to do it. I am going to-first, UAAC, target, okay. And then we’re going to login token client. Awesome. So we’re now logged in and you can check that with UAAC contexts. You can see there that we’ve logged in as the admin user. I’m going to go ahead and copy-paste this command because it’s a long one. And the user and adding is the influxdb-firehose-nozzle user, you can name whatever if you have an existing user that you’re-that is configured to read from the Firehose. You could also use that user. This is just if you’re starting from scratch and you can see there that that user has been created. Next, we’re going to need to go in and edit a few things. So here’s the root of the repo. The config file is in config so there’s the influxdb-firehose-nozzle.json. You can see the manifest.yml at the root of the repo, and then there’s another manifest file at API example that we’re going to need to edit. So I’ve got Adam open, so let’s go ahead and do that. This is written in Go, so we’ll be using the Go build pack and then the only thing that you might need to change is your domain. If you want to change the name of this application, maybe something more descriptive for your personal environment. You can change the name of the host. Not super important.
Jack Zampolin 12:18.758 This is the config for the Firehose Nozzle so there’s a lot of stuff you’re going to change in here. If you were not using the PCF-Dev environment, you’re going to need to change all of those addresses there to your domain. You can check what your CloudFoundry domain is by typing cf domains. cf login -u admin, okay. So, you can see my domains there. So, that’s correct for our UAA URL. The Firehose URL is correct as well. And then our app info API URL that is correct. That’s where it’s going to deploy once we do deploy it. You can change the events that come through that you pull from the Firehose. That’s how you change that there. If you’re going to be using HTTP start stop or HTTP start or HTTP stop, you’re going to want to deploy this app info, API. There are some InfluxDB related config here as well. So, we’ve got our InfluxDB URL. That’s actually going to be our URL for the InfluxDB Cloud. So, once you have spun up your cloud instance, this is the screen that you’ll get taken to. There’s about three different configurations screens and then you’re ready to roll. The URL for my InfluxDB Cloud instance is here. So, you’re going to need to do HTTPS and then pop in your URL there and then make sure the port is 8086. The username and password that you input when spinning up the instance, mine are very easy. One thing that you’re going to need to do is, go ahead and use that influx command to log in to the instance. Show what database is. Okay. So we are going to need to create that CloudFoundry database. If you notice here that InfluxDB database, make sure you have a database that matches that name in the Influx instance that you’re targeting. So we’re going to create database. Easy. Awesome. So that’s all set up there.
Jack Zampolin 15:36.954 The client and client secret here, that’s the UAA user we created earlier. If you’re using a different one, you’ll probably want to modify that. So this looks all good. So now let’s go ahead and deploy it from the root of the repo. You just type cf push. So it’s going to go ahead and upload it, build it. This will take just a second. And then, we’ll be able to check our logs here and make sure it’s pushing data properly.
[silence]
Jack Zampolin 16:31.313 While in there, are there any questions while I’m going through all of this? Could I make anything more clear? Am I covering things too much? Any feedback is appreciated.
[silence]
Jack Zampolin 17:14.227 Okay. And let’s go on ahead and start properly. Let’s go ahead and check the logs. Boom. So you can see here it has started collecting metrics. It’s posting those metrics. So we do need that app API example. So the manifest there-you’re going to need the CF user and the CF password. Now, that is the password, not the client secret. So it’s going to be admin, admin here, not admin, admin, client secret. That’s really important. Otherwise, this won’t deploy properly. And then, you do need to pass the API address for your CloudFoundry instance. So if you got a different domain, you’re going to need to configure that there. And then, we’re going to say it’s cf push here.
Jack Zampolin 18:17.394 And while we’re doing this-while that’s pushing, let’s go ahead and pop into our InfluxDB Cloud instance and see if that has come up properly.
[silence]
Jack Zampolin 18:49.441 Awesome. And we’ve started getting some metrics in there. Now you’ll get container level metrics for all of your applications, as well as some host level stuff because I’m running this on a VM and I have no other applications to deploy in this environment, we’re not really getting any metrics besides how many it’s scraping. And as you can see there, that cfslowConsumerAlert measurement contains whether or not the queue is building up in the Firehose. And then, messages received and sent. But again, if you have some apps deployed, you will be getting those metrics now. So and then, the cfapp API, we should see some logs for that. It came up okay.
[silence]
Jack Zampolin 20:03.722 There it goes. So it’s good to go. And that is how you ingest the influxdb-firehose-nozzle using CloudFoundry. I’m going to stick around for a little while for any questions. If you have any questions, be they about this tutorial or about the rest of the TICK stack, I’m happy to answer those now. Drop them in either the chat or the Q&A. Thank you all very much for your time. And we’ll be posting this video and the slides afterwards as well.
[/et_pb_toggle]
