Telegraf Socket Listener Input Plugin

By Noah Crowley / Use Cases, Product, Developer
Aug 30, 2019

One of Telegraf’s biggest strengths is the large collection of plugins it offers that can be used to immediately to start collecting data from a variety of applications. This covers a lot of common infrastructure components like databases and web servers, but if you want to get data into Telegraf using your own scripts or applications, there are also a variety of more general-purpose input plugins that you can use. There’s a file plugin, which will read from or tail a file on disk, an exec plugin which will execute your own scripts or commands and parse their output, and plugins to listen for data via HTTP or poll external endpoints (http_listener_v2 and http).

One of the plugins I end up using most often is the socket_listener plugin, which allows you to send data to Telegraf using UDP, TCP, or Unix sockets. The simplicity of InfluxDB’s line protocol and the ease with which you can write to a socket using most programming languages makes this a powerful tool for both quick prototyping and more long-term solutions.

Let’s take a look at some examples of the plugin in action. We’ll use a host system running Ubuntu, with Telegraf and InfluxDB installed from the official InfluxData repository.

UDP: User Datagram Protocol

For this first example, let’s configure the socket_listener plugin to accept UDP packets. The UDP protocol is a connectionless, best-effort service: messages are sent to their recipient with no guarantee that they will be received. One benefit of this is that it means there is less processing overhead for UDP relative to something like TCP, which is more reliable. That makes UDP well-suited for things like interprocess communication or situations in which some loss of data is acceptable.

In order to set up Telegraf to receive UDP messages, first configure this block in your telegraf.conf:

[[inputs.socket_listener]]
  service_address = "udp://:8094"
  data_format = "influx"

This configures Telegraf to listen for data on UDP port 8094 in InfluxDB line protocol format. A quick restart of Telegraf and it should be ready to receive data:

sudo systemctl restart telegraf

Now we can start writing some data to Telegraf via UDP. To send the data, we’ll use two common command line utilities, echo and nc, or netcat. Enter the following commands into your terminal:

$ echo "m1,tag1=tag_value value=1" | nc -u -4 -q localhost 8094
$ echo "m1,tag1=tag_value value=2" | nc -u -4 -q localhost 8094
$ echo "m1,tag1=tag_value value=3" | nc -u -4 -q localhost 8094

The InfluxDB Line Protocol is a plain text format, which means we can use the echo command to print messages in that format directly to stdout, and use the | character to “pipe” that text into nc, which will transmit the data over a network connection (in our case, on the same machine via the loopback interface). We provide several arguments to nc: -u, -4, and -q. The first argument tells nc to send data using UDP; the second argument says that we should use IPv4, and the third argument tells nc to quit once the data has been sent.

Because UDP doesn’t have any guarantees about whether the messages will be delivered, let’s verify that the data has actually arrived in the database using the Influx CLI:

$ influx
Connected to http://localhost:8086 version 1.7.4
InfluxDB shell version: unknown
Enter an InfluxQL query
> use database telegraf
> SELECT * FROM mymeasurement WHERE time > now()-5m
name: socketwriter
time                host           tag1       value
----                ----           ----       --------
1562602580414360000 noah-mbp.local tag_value  1
1562602581418350000 noah-mbp.local tag_value  2
1562602582420300000 noah-mbp.local tag_value  3

Success! Our three datapoints have been written to the database.

Unix socket

While UDP and TCP are network protocols, the socket_listener plugin also has the ability to listen on a Unix socket, which provides a method for exchanging data between two processes on the same host via the kernel. Even when communicating on the same host, IP sockets like TCP and UDP will send data through the loopback networking interface. A Unix socket, on the other hand, can avoid this extra work and send data directly to the receiving socket buffer, which gives it a performance edge when sending data locally. You can read more about the differences between Unix sockets and network sockets in this mailing list post titled unix domain sockets vs. internet sockets.

This time, instead of using command line tools, we’ll write a small Python script that will send one-dimensional random walk data over a Unix socket to Telegraf. First, we’ll configure Telegraf as follows (full config here):

[[inputs.socket_listener]]
  service_address = "unixgram:///tmp/telegraf.sock"

Then we’ll run Telegraf with the new config:

$ telegraf --config socket-telegraf.conf --debug

We can see that Telegraf has created the socket by listing the contents of the /tmp directory:

$ ls /tmp
telegraf.sock

Next we’ll create our Python script. The full script can be found here. Following the import statements and an interrupt handler which will close the socket when we use Ctrl+C to exit our script, we’ll create a socket object and connect to the socket Telegraf has opened.

# The file handler for the Telegraf process.
telegraf_socket = "/tmp/telegraf.sock"

# Connection to Telegraf, over a network socket.
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)

sock.connect(telegraf_socket)

And then we create a loop that first sends the current data point to InfluxDB before updating the position variable for the next run through the loop, and sleeps for a second in order to rate limit the number of points we send:

while True:
    message = "socketwriter position=%d\n" % x
    print(message)
    sock.send(message.encode('utf8'))
    x = x + random.randint(-1, 1)
    time.sleep(1)

With Telegraf running, let’s start our script:

$ python3 unix_socket_writer.py 
socketwriter position=0

socketwriter position=0

socketwriter position=1

socketwriter position=0

Once again, we can open up the InfluxDB CLI and verify that our data has been written:

$ influx
Connected to http://localhost:8086 version 1.7.4
InfluxDB shell version: unknown
Enter an InfluxQL query
> use database telegraf
> SELECT * FROM socketwriter WHERE time > now()-1m
name: socketwriter
time                host           position
----                ----           --------
1562602580414360000 noah-mbp.local 0
1562602581418350000 noah-mbp.local 0
1562602582420300000 noah-mbp.local 1
1562602583422080000 noah-mbp.local 0

Looks good!

Use it in your own projects!

Whether you’re using Unix sockets for their local performance or TCP sockets for their network robustness, Telegraf’s socket_listener plugin provides one of the easiest ways to get data from your scripts and applications and into Telegraf. And if that doesn’t work for you, check out some of the other general-purpose plugins that Telegraf has to offer — you’re sure to find something that suits your needs.

If you have questions or need help with getting the socket_listener plugin set up, check out some of our community resources, like our public Slack or Discourse forums, or reach out to me directly on Twitter @noahcrowley.