A Guide to MQTT

One protocol that has rapidly taken over the IoT industry is MQTT. This is because IoT application workloads have unique requirements compared to most applications. More common network protocols like HTTP aren’t ideal for IoT environments where network connectivity may be intermittent, hardware may have low processing power, and bandwidth is constrained. Though there are several options for IoT communication, MQTT has become the default network protocol because of the benefits it provides, which will be covered in depth on this webpage.

What is MQTT?

MQTT is a machine-to-machine (M2M)/Internet of Things communication protocol designed as a lightweight publish/subscribe messaging tool. MQTT is useful for connections where a small resource footprint is required and/or network bandwidth is at a premium. MQTT was created by Dr. Andy Stanford-Clark of IBM, and Arlen Nipper of Arcom in 1999. At the time, IBM was working with an oil and gas company that needed to pull data from oil pipelines in remote areas, which required a new protocol to fit those requirements. The result was MQTT.

MQTT was used internally by IBM until they released the MQTT 3.1 specification in 2010, allowing others to create their own MQTT implementations. Developers quickly realized the value MQTT had for IoT-related use cases and adoption grew quickly, with numerous open source brokers and client libraries being created. In 2013 IBM submitted MQTT to OASIS to be maintained and standardized.

MQTT use cases

MQTT is a versatile network protocol that is ideal for any situation where your network might be unreliable, you want to minimize bandwidth consumption, have low-powered hardware, or you have an architecture where you have many client devices that will need access to the same data in near real-time.

Consumer IoT

MQTT is used for communication between many consumer IoT devices. This could include everything from smart home type devices designed for automating things and making your home more efficient.

Industrial IoT

MQTT was designed for industrial use in the oil and gas industry and has rapidly spread to other areas like manufacturing due to the many benefits it has over competing network protocols. MQTT helps companies become more efficient by allowing them to collect more data and act on that data faster.

Logistics

MQTT is ideal for real-time monitoring of assets as they move all around the world. Even if internet connectivity is lost, MQTT is designed to ensure delivery of data once it is regained. In a world with increasingly complex supply chains, MQTT is becoming even more important.

Mobile application development

One app that uses MQTT that you might be familiar with is Facebook Messenger. The Facebook Messenger team chose MQTT over HTTP because mobile applications are fairly similar to IoT workloads and for group chats especially MQTT’s pub/sub architecture was very natural and made it easy to add members, with each group chat being its own topic that clients could subscribe to.

For general mobile apps, MQTT is ideal because it allows for persistent connections to receive new messages without consuming large amounts of bandwidth or burning through the phone’s battery. If you are building any mobile app that will require network communication frequently, MQTT may be a solid choice.

How MQTT Works

MQTT uses a publish/subscribe message pattern which means that client devices connect to a broker which acts as a middleman between those devices. Devices that are connected to the broker can publish messages that will be forwarded to other connected devices via the broker. Each message must have a topic and the broker will only pass messages to devices that have explicitly subscribed to a topic.

With MQTT any connected client device can send or receive messages once connected to the MQTT broker. Each device also has the option to simply publish messages without subscribing to topics or only subscribe to topics and never publish any messages.

Using a smart home as an example, each room of the house could be designated as a topic. It wouldn’t make sense and would be inefficient for every device to receive and process messages that aren’t relevant to it. For this reason, MQTT allows developers to define topics so that each device only receives the data it needs.

MQTT strengths and weaknesses

No network protocol is perfect and MQTT by definition had to make some tradeoffs to be optimized for IoT use cases. In this section you’ll learn some more details about the strengths and weaknesses of MQTT so you can determine if it makes sense to use for your project.

MQTT strengths

The primary strengths of MQTT are:

• Efficiency
• Reliability
• Flexibility

MQTT is optimized to use as little data and energy as possible for each message being sent. MQTT is able to reuse connections to the broker for sending multiple messages which saves resource overhead compared to a protocol like HTTP. You can think of HTTP being like calling somebody on the phone and then hanging up after each sentence you talk about and then calling them again to continue the conversation, which obviously isn’t as efficient as a normal phone call. The result is that once the MQTT connection is established each message sent via MQTT uses 10x less bandwidth compared to using HTTP, based on tests performed by Google.

MQTT is also more efficient in terms of message size. As an example, MQTT’s header is 2 bytes of data while HTTP headers are at minimum 16 bytes for a GET request, and with HTTP that header has to be sent on every single message, while MQTT only sends the header when creating the connection to the broker. The pub/sub architecture is also efficient because it removes the need for every device to check for new messages at certain intervals via polling. Instead, devices wait to receive the messages as they are delivered directly by the broker, which not only ensures they get new messages faster but also more efficiently.

MQTT is reliable due to its pub/sub architecture which allows for the broker to buffer messages and hold them until a subscribed device is connected again. MQTT also provides quality of service levels for messages to ensure delivery for critical data or save resources by allowing some messages to be dropped if they don’t contain mission-critical data.

Flexibility is another main strength of MQTT. As mentioned above, the user can control the service level requirements for messages depending on the situation. MQTT is also data agnostic in terms of what messages contain, meaning that message payloads can hold binary data, ASCII text, or any other data type. MQTT’s decoupled pub/sub architecture means it can be scaled up easily by adding more brokers to handle more devices as they are added or message frequency increases.

MQTT weaknesses

The primary weaknesses of MQTT are:

• Latency
• Architecture complexity
• Resource requirements on low power devices
• Security

For use cases that require extremely low latency, MQTT may be a problem due to the latency created by messages having to pass through the MQTT broker as an extra step before reaching their destination, rather than being able to go directly from one device to another.

The management of the MQTT broker can also be a source of complexity in terms of architecture, because the broker is another service that has to be managed and monitored to ensure the entire system is working. For larger workloads, this might require multiple broker instances which requires load balancing to make sure messages are being distributed equally among them.

While MQTT is a lightweight protocol, the number of features it provides does make it more resource intensive than some alternative options. Generally this won’t be an issue except for the lowest powered hardware, but it is something that might be relevant if you are working with this type of hardware.

Security can be considered another general weakness of MQTT. MQTT by default doesn’t provide any security features — any form of encryption needs to be implemented using the underlying TCP protocol that MQTT is built on top of. While SSL/TLS are great options for security, it is still something that has to be managed and implemented to make sure your data is secure. The newest release of MQTT, MQTT v5 does add some additional security features as well which will be covered later.

MQTT key concepts and features

In this section, you will learn about some of the key concepts you need to know if you want to work with MQTT. You’ll also learn about some basic features provided by MQTT.

MQTT broker

MQTT brokers work by passing messages between connected devices. Rather than each device trying to communicate with each other, which would be extremely complex and resource intensive, each device just needs to create a single connection to the broker and can receive data from any other device.

MQTT brokers can handle millions of connected MQTT clients and provide a number of the features that will be covered later in this section. The broker is what makes these features possible and is why MQTT is so popular with IoT developers.

MQTT client

An MQTT client is simply any device that is running MQTT client software that allows it to connect to an MQTT broker. MQTT clients can publish messages to the broker and also subscribe to topics to receive messages. An MQTT client is anything that is able to run an MQTT client library implementation and connect to an MQTT broker.

MQTT protocol details

At its core MQTT is built on top of the TCP/IP network stack. While this is slightly less performant and lightweight compared to UDP, TCP allows for guaranteed delivery which is vital for many IoT use cases. Another benefit of using TCP is that MQTT can encrypt data using SSL/TLS.

Topics

Topics are the key way that MQTT organizes messages being sent to the broker. Each message being sent requires a defined topic and connected devices can subscribe to topics to receive messages when they are sent. Topics are UTF-8 strings that are divided using forward slashes. Here is an example of a basic MQTT topic:

top/middle/bottom

These messages can be anything, but ideally you will have some basic structure to make your topic hierarchy logical and easy to understand for developers as your application grows over time and more topics are added. Topics do not need to be pre-defined before messages are published or clients subscribe to the topic, you can add new topics on the fly without having to worry about it.

Here’s a few examples of how you might structure your topics if you were doing something with a smart home.

• Humidity data for the basement - home/basement/mainroom/humidity
• Temperature data from living room - home/mainfloor/livingroom/temperature
• Messages for a specific device in your bedroom - home/mainfloor/bedroom/device

Beyond basic static text strings like the above topics, MQTT provides support for dynamic topics by using wildcard operators by using + between slashes. The plus sign only gives you single- level wildcard matching. If you want multi-level wildcard support, you can use a hashtag and the topic will match for anything after the hashtag. For example, if you wanted to subscribe to every message that your smart home is generating, you could use the following topic:

home/#

These wildcard operators are very useful but should be used carefully — overusing wildcards when they aren’t needed will result in poor performance of your broker as it is using resources to send messages to devices that don’t need them. Always make sure to determine whether a device needs to be subscribed to a MQTT topic — if not, you will be wasting bandwidth and processing power delivering MQTT messages that aren’t needed by the clients receiving them.

Sessions

An MQTT session is considered to be everything that happens from when the client device requests to connect to the MQTT broker until the connection is interrupted. Based on that a session could only be the initial connection request failing, or it could be a long-lasting session with many messages being published and received by the client.

Sessions are important because they will have an associated state depending on the Quality of Service (QoS) level set for the messages involved. The broker needs to be able to determine if certain client devices have been delivered the messages they need and the client needs to respond in some cases to confirm the message has been delivered as well.

Message expiration

MQTT allows clients to set an expiration interval for published messages. The interval is set with seconds as the unit and tells the broker how long to store the message before deleting it. If the message expires while a client isn’t connected, it simply won’t receive the message.

Keep Alive function

MQTT provides a way to ensure that the established connection between broker and client is still alive by setting an interval to confirm the connection. This is known as a Keep Alive function and the frequency can be defined by the developer using MQTT. If the client has not sent or received a message from the broker during the set interval, the broker will ping the client and expects a response. If the client doesn’t respond, the broker will register the device as disconnected.

Retained messages

Retained messages are simply normal MQTT messages that have been flagged so they are stored by the broker. By using retained messages, any client that subscribes to a topic will be sent the most recent retained message so the client can get the most recent update rather than waiting for a message to be sent.

Quality of Service

Quality of Service(QoS) is how MQTT controls delivery guarantees for messages. MQTT QoS is essentially an agreement between the sender and receiver of the message on how to define what is required to consider a message “delivered”. In the case of MQTT, there are three Quality of Service levels available:

• At most once(0) - The minimum QoS level, which is considered to be “best effort” delivery. There is no guarantee that the message will reach subscribed clients and no acknowledgement is sent to the publishing device. The broker simply confirms the subscribed client device is connected and then sends the message to it.
• At least once(1) - QoS level 1 does guarantee delivery of the message but can result in the message being sent or even delivered multiple times. Having multiple copies of the same message being acted on could result in issues if you don’t account for this in your application.
• Exactly once(2) - QoS level 2 is the highest service level provided by MQTT. It not only guarantees delivery to subscribed clients but makes sure they only receive the message once. MQTT does this by requiring a 4-part handshake to first confirm the client received the message and then another request and response using a flag to make sure there have been no duplicate messages received.

During message delivery there are two parts involved with getting a message to a destination. First, the message is created and published by a client and sent to the MQTT broker, and then the broker needs to deliver that message to subscribing clients.

QoS is one of the most important features provided by MQTT because it allows developers to set certain guarantees depending on the importance of their data. This means that even on unreliable networks MQTT can be counted on to deliver data. At the same time it gives developers flexibility; if certain data isn’t important, MQTT can let those messages be lost and not waste bandwidth in certain situations.

Last will and testament

Last will and testament is a feature provided by MQTT that allows clients to specify a message that will be sent to other client devices if it ends its connection due to an unplanned error. When the client device connects to the MQTT broker, the last will and testament is sent and stored on the broker. The message will be sent when the broker detects that the client has disconnected without giving a standard DISCONNECT message used for normal disconnections.

MQTT version 5 features

While the majority of MQTT deployments are still using MQTT 3.1.1, MQTT version 5 is now available with many powerful new features. The majority of new projects will be using MQTT 5, and over time, existing projects will upgrade to MQTT version 5 as well. Here are some of the best new features provided by MQTT version 5:

• Reason codes - Reason codes in MQTT version 5 allow more information to be sent to MQTT clients when certain events happen between the MQTT broker and client. For example, custom reason codes can be sent to acknowledge certain quality of service levels, failed connections, different errors, invalid topics, rate limits, and many more.
• Disconnect messages - MQTT v5 allows the MQTT broker to send messages to client devices for why the connection was closed so devices can take action based on the reason for the disconnection.
• User properties - Collection of key value pairs can be attached to messages to provide additional data or context.
• Server redirect - When a client tries to connect to a broker, it can be redirected to a different MQTT broker. This feature could be used for load balancing by sending devices to a new broker if the current broker has exceeded a certain threshold for connections.
• Topic aliases - Topic aliases are a feature pulled from MQTT-SN that help save bandwidth through reducing message size by allowing long topic names to be mapped to a shorter alternative alias.
• Request/Response pattern support - MQTT v5 gives developers the ability to simulate a more traditional request/response pattern for use cases where it works better than MQTT’s standard publish/subscribe model. For these cases MQTT v5 has added the ability to add response topics to messages which clients can subscribe to and the messages will be treated more like a standard request/response message, although it will still be asynchronous due to MQTT’s underlying pub/sub architecture. This feature is merely an abstraction over the top to make it easier to create these types of requests without extra work for developers.
• Shared subscriptions - Shared subscriptions are basically a form of client load balancing where client devices can all subscribe to the same topic and the messages will be sent to only a single device rather than the same message being sent to every device.
• Retained message control - MQTT v5 gives developers more control over how to handle retained messages for situations like whether they should be sent to a device if the client has just subscribed for the first time or if it has reconnected.

MQTT security

Security is a challenge with any type of software, but internet of things applications bring a whole other level of security challenges. Because IoT devices often have limitations when it comes to compute and memory resources, developers have to make tradeoffs with what security tools, like encryption algorithms, they can use. Another challenge is dealing with updates to devices in the field efficiently. If devices can’t be remotely accessed, software updates become more expensive and will be less frequent.

At the application level MQTT can be set up to require a basic username and password for authentication. Different MQTT broker implementations may also provide additional security features. Because it is built on TCP, MQTT also gives the option of transport-layer level security using SSL/TLS to encrypt MQTT message data. At the network level, developers also have the option of trying to physically secure hardware devices and network components, as well as using Virtual Private Networks (VPNs) for communication between clients and brokers.

MQTT resources and tools

In this section you will learn about the broader MQTT ecosystem and some of the tools available for developers looking to use MQTT for their applications.

MQTT broker implementations

There are a number of available open source and commercial MQTT brokers available for you to get started working with MQTT. Here are a few of the most popular options.

Mosquitto Mosquitto is the most popular open source MQTT broker and is supported by the Eclipse foundation. Mosquitto is probably the best MQTT broker available if you don’t want to be reliant on a broker implementation supported by a vendor that has somewhat conflicting interests due to offering commercial alternatives to their open source implementations.

EMQX EMQX is an open source, cloud native MQTT broker created by the company EMQ, which is a foundational sponsor of OASIS alongside IBM. EMQX was the first MQTT broker to fully support MQTT version 5 functionality and EMQ claims a cluster of EMQX brokers can handle 100 million subscribers.

EMQ also provides a number of other useful open source tools for working with MQTT like NanoMQ and Neuron which work together with EMQX to help connect IoT devices from edge to cloud.

HiveMQ HiveMQ is another popular vendor in the MQTT ecosystem. HiveMQ provides an open source broker that supports MQTT version 3 and 5. HiveMQ also provides a hosted MQTT broker that can be used for scaling your MQTT application and integrates well with InfluxDB.

MQTT client libraries

There are numerous client libraries available for working with MQTT in your language of choice. Most vendors provide their own implementations, but the most popular client libraries are the Paho MQTT libraries supported by the Eclipse Foundation.

Data storage for MQTT

MQTT messages often contain time series data. This data is streaming in very fast from potentially a large number of devices. For many IoT-related workloads, that data also needs to be able to be analyzed shortly after being ingested — long delays are unacceptable for monitoring situations where near real-time access to data is needed. A time series database like InfluxDB is designed for these requirements and provides a number of benefits in terms of pure performance when it comes to writing and querying data, as well as benefits like faster development time due to a number of built-in features that are provided out of the box like downsampling and common analysis and forecasting functions required for time series data.

MQTT Sparkplug

MQTT Sparkplug is a specification created to define a more standardized architecture for how Edge of Network gateways and devices should communicate. While Sparkplug can be used for any MQTT application, it is optimized for systems that involve SCADA and traditional industrial IoT use cases and to make these systems more interoperable with MQTT. MQTT Sparkplug was created by Eclipse, the organization that also maintains the Paho MQTT client libraries. There are three main components to the Sparkplug specification:

• MQTT topic namespaces - By default MQTT doesn’t enforce any required topic conventions or namespaces, Sparkplug intends to create a more standardized convention for structuring topics
• MQTT state management - Sparkplug defines a number of new message types which can help with state management like handling connection states and dealing with metric data. These message types include messages for the connection and disconnection of both edge of network MQTT nodes and the devices themselves. There are also message types for node vs device messages and the ability to define how important messages are.
• MQTT payload definitions - Sparkplug features Google Protobufs-based encoding for messages. This reduces bandwidth usage and also allows for easier integration with other protocols used for industrial IoT applications like Modbus. Protobufs also make it possible to create data models for payloads to define historical data, file data, datasets, complex data via templates, metadata for metrics, and metric aliases.

Node-RED

Node-RED is an open source low-code framework for creating event-driven applications. It provides a visual builder to integrate hardware and software components and was designed by IBM for IoT projects. Node-RED is a great option for rapidly creating MQTT-based applications for both prototyping and production.

MQTT FAQs

What does MQTT stand for?

MQTT originally stood for MQ Telemetry Transport, but MQTT is no longer an acronym and is just the name of the protocol.

What is MQTT-SN?

MQTT-SN stands for MQTT for Sensor Networks. MQTT-SN is designed for working on wireless networks and is optimized for working with low-powered sensors that generally communicate over wireless networks. The main difference between MQTT-SN vs standard MQTT is that MQTT-SN doesn’t require TCP and can use serial communication protocols like Zigbee or Thread. Some other differences are that MQTT-SN uses topic IDs rather than topic names, allows for pre-defined topics, and has a keep-alive feature for sleeping client devices.

MQTT vs AMQP

MQTT and AMQP are both asynchronous message protocols used for sending and receiving messages across systems. The main difference lies in the fact that MQTT is highly optimized for IoT use cases and is a smaller and simpler protocol as a result. AMQP is designed to cover much more general use cases and as a result has more features available but is also more complicated and less efficient.

When it comes to features, AMQP can be considered to win in terms of security options by including things like SASL out of the box, while with MQTT you would need to add it yourself. AMQP also provides the ability to extend the protocol while maintaining backwards compatibility, while MQTT changes require version updates. AMQP provides several messaging patterns beyond MQTT’s pub/sub pattern. The primary benefits of MQTT in comparison are that it is lighter weight and thus works better for battery and resource constrained hardware. MQTT also consumes less bandwidth and works better on unreliable networks. In short, the primary difference between MQTT and AMQP is that AMQP is a general-purpose messaging protocol while MQTT was designed from the ground up to be a specialized protocol to fit a certain use case. Whether you should use MQTT or AMQP will depend on which protocol fits your project’s requirements.

MQTT vs RabbitMQ

RabbitMQ is an open source message broker while MQTT is a lightweight publish-subscribe network protocol designed for machine to machine communication in IoT environments. RabbitMQ was originally designed solely for working with AMQP but has added protocol support via plugins for MQTT, as well as several other protocols like STOMP and HTTP. As a result RabbitMQ can be used as a general message broker that provides great features like failover and clustering while the protocol being used to pass messages can be changed depending on your use case.

MQTT vs HTTP

HTTP is a request/response protocol primarily used for internet applications. HTTP can be used for IoT use cases but isn’t designed for that. It suffers a number of downsides related to message size and inefficiency when it comes to moving data around in an IoT environment with unreliable connection and dealing with numerous connected devices.

MQTT vs COAP

CoAP is similar to HTTP and uses a request/response model with a binary format to make it more compact. MQTT uses a publish/subscribe model for communicating. The biggest difference between CoAP and MQTT is that MQTT focuses on being a many-to-many communication protocol that passes messages to many client devices through a broker while CoAP is primarily for one-to-one communication from client to server. MQTT can be considered to be for more event-driven applications while CoAP is best used for state transfer.

MQTT vs Zigbee

The primary difference between Zigbee and MQTT is that Zigbee is a physical data layer protocol while MQTT is an application layer protocol. Zigbee is focused on how data is actually transmitted between devices and is intended to be used for mesh network devices as a simpler alternative to things like Bluetooth or WiFi. In many cases Zigbee is used as a gateway protocol before finally converting the data to MQTT so the data can be delivered to the cloud.

MQTT vs Thread

Thread is another protocol intended for low-power and low-latency applications and is very similar to Zigbee in many cases. In this way much of the above information comparing MQTT to Zigbee applies, with MQTT being used for application level communication while Thread is for the lower-level transfer of data.

MQTT over websockets

Some MQTT brokers provide support for websockets, which allow you to connect to an MQTT broker from your web browser using JavaScript.

What is OASIS?

OASIS (Organization for the Advancement of Structured Information Standards) is the consortium in charge of developing and maintaining the MQTT specification as well as a number of other projects. OASIS was founded in 1993 as a trade association that focused on a markup language known as SGML. Over time OASIS expanded and now maintains protocols like AMQP, MQTT, SAML, and Legal XML. OASIS members include Oracle, Red Hat, General Motors, IBM, Microsoft, the American Bar Association, and many others.