Running InfluxDB on AWS with Cloud Formation

By Anais Dotis-Georgiou / Product, Use Cases, Developer
Jul 11, 2019

Coding bootcamps are great. I learned a lot, and it enabled me to land a Developer Advocacy role. However, despite an extensive curriculum and excellent instructors, I still came out lacking a lot of basic knowledge. I took a Data Science bootcamp, which meant that I never focused on back-end, and like many bootcamp grads, only ever used Heroku. Creating and running a cloud instance through a major cloud provider was one glaring gap. Today, we will fill that gap by running the Influx 2.0 Docker image on AWS with Cloud Formation. The repo that accompanies this cookbook can be found here.

While my desire to fill a gap in my knowledge motivated me to get InfluxDB running on an EC2 instance, there are more practical advantages to doing so. You can send all of your DevOps monitoring, Application Monitoring, and Amazon CloudWatch metrics to InfluxDB with telegraf. With InfluxDB, you can get real-time views into your AWS spend to cut costs. If you’re looking for an example, I encourage you to read this whitepaper to learn about how Houghton Mifflin Harcourt achieves that.

Setup for an EC2 instance

To get started, you will need to create an AWS account. Next, you will need to generate a key pair and download the private key which will allow you to ssh on to your EC2 instance.

First, I make sure that I am creating my instance in the correct region. I select my region, us-east-1 or N. Virginia, from the dropdown menu on the top right of the AWS Management Console landing page.

Next, I select EC2 from the Services menu on the left side nav.

Now I can select Key Pairs from the navigation menu on the left, and create a Key Pair.

Selecting Key Pairs from the nav menu brings us to the Key Pair page, where we can find all of the Key Pairs we have already created and create new keys.

Now, I will move my keys into the same directory where I will specify my Cloud Formation yaml configuration.

cp ~/Downloads/Influx20.pem ~/cloudformation/deploy-aws
touch stack.yml

Remember to change the file permission of this key wilth:

chmod 400 Influx20.pem

Next, we will need an access key ID and a secret for the AWS CLI. Navigate to IAM from the Services menu under Security and create a user and grant the user all of the permissions.

We can install the AWS CLI with pip3 install awscli. Configure the aws profile with aws configure --profile and provide the access key ID, secret, and region (us-east-1) as prompted. You can ignore the output format.

Creating the stack definition to install InfluxDB

The stack definition starts with the Resources which will be an EC2 instance type and we’ll name it AppNode, the logical ID. Then we include the Properties of the instance. We select t2.micro, the smallest (and cheapest!) instance class, and find the correct AMI ID, Amazon Machine Image, from searching for Ubuntu 18.04 from the AMIs page. The AMI ID changes based on your region, so make sure that your region is correctly displayed in the top right corner of the nav bar. Finally, we also need to include the name of the key pair we generated, Influx20.

We need to add a Security Group, a virtual firewall that controls the traffic. We do so by using the intrinsic Ref function which returns the resources properties.

Resources:
 AppNode:
   Type: AWS::EC2::Instance
   Properties:
     InstanceType: t2.micro
     ImageId: ami-024a64a6685d05041
     KeyName: Influx20
     SecurityGroups:
       - !Ref AppNodeSG

We define AppNodeSG as having a type ECS::AWS::SecurityGroup. It will have two properties. The description just tells us what this resource does. The SecurityGroupIngress is just an array of inbound connection properties. We will expose the HTTP port 80 using tcp protocol and make it accessible anywhere.

AppNodeSG:
   Type: AWS::EC2::SecurityGroup
   Properties:
     GroupDescription: for the app nodes that allow ssh
     SecurityGroupIngress:
       - IpProtocol: tcp
         FromPort: '80'
         ToPort: '80'
         CidrIp: 0.0.0.0/0

Finally we create a script that runs on the bash shell. It will install docker and pull the InfluxDB 2.0.0 alpha image.

UserData: !Base64 |
         #!/bin/bash
         apt-get update -qq
         apt-get install -y apt-transport-https ca-certificates
         apt-key adv apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
         apt-get update -qq apt-get purge lxc-docker || true
         curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
         add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
         apt-get -y install linux-image-extra-$(uname -r) linux-image-extra-virtual
         apt-get -y install docker-ce
         usermod -aG docker unbuntu
         docker image pull quay.io/influxdb/influxdb:2.0.0-alpha
         docker container run -p 80:9999 quay.io/influxdb/influxdb:2.0.0-alpha

We include this as UserData for our App Node resource.

Using Cloud Formation to create the stack

We’re finally ready to create the stack We create the stack using:

We create the stack using:

aws cloudformation create-stack --stack-name demo --region us-east-1 --template-body file://$PWD/stack.yml

We verify that we successfully created a stack and obtain the Public IP by checking our Instances.

After we ssh into the instance with ssh -l ubuntu -i Influx20.pem 52.71.129.176, we wget and install telegraf with

wget <a href="https://dl.influxdata.com/telegraf/releases/telegraf_1.10.4-1_amd64.deb">https://dl.influxdata.com/telegraf/releases/telegraf_1.10.4-1_amd64.deb</a>
sudo dpkg -i telegraf_1.10.4-1_amd64.deb

Now we’re ready to go through the 2.0 setup steps to create a telegraf config. Once completed, we can finally view our system stats or docker metrics (depending on how you set up telegraf).

Conclusion

I hope this cookbook helps you check another skill off your list. If you have any questions please post them on our community site or slack channel. Finally, as always, here is another brain break: