Skip to content
gunkslab

Primers

MQTT

MQTT is a small messaging protocol. Programs don’t connect to each other: each connects to one server, the broker, and sends messages through it by name. A program publishes a message on a topic, a name like lab/robot-1/telemetry, and the broker hands it to every program that has subscribed to that topic. The sender doesn’t know who’s listening, and the listeners don’t know who sent it. Either side can come and go.

It was designed in 1999 for watching oil pipelines over satellite links, where every byte cost money and the link dropped often. That’s why it’s small and why it plans for things disappearing. Today it runs in home automation, factories, phones and robots, often on microcontrollers too small for most other protocols.

Pick your level at the top of the page.

High school

A noticeboard, not a phone call

Most programs talk the way a phone call works: one calls the other, and both have to be there. MQTT works more like a noticeboard in a school corridor. Anyone can pin a note under a heading, and anyone who cares about that heading reads it. The person pinning doesn’t need to know who reads it, and the readers don’t need to know who wrote it.

In MQTT the noticeboard is a program called the broker. Every device and program connects to the broker and nothing else.

Topics: the headings

Every message goes under a topic, a name with parts split by /, from general to particular, like a file path:

lab/robot-1/telemetry

That’s “the lab, robot 1, its readings”. Publishing is pinning a message under a topic. Subscribing is asking the broker “send me everything under this topic”.

You can subscribe to more than one topic at once with wildcards. + stands for any one part, so lab/+/status is every robot’s status. # stands for everything below, so lab/# is everything in the lab.

Retained: the pinned note

Most messages go to whoever is subscribed right then, and they’re gone. A retained message stays: the broker keeps the last one on its topic and hands it to anyone who subscribes later. It’s like a note pinned at the top of a group chat. A robot that joins late still sees “the emergency stop is on” straight away.

QoS: how hard to try

QoS, quality of service, is how hard the broker tries to get a message through:

  • 0: like shouting across a room. Sent once. If it’s lost, it’s lost.
  • 1: like a text you keep resending until you get a reply. It gets there, but might arrive twice.
  • 2: like a signed-for parcel. Exactly once, with more back and forth.

A sensor reading sent ten times a second can go at 0: if one is lost, the next is only a tenth of a second behind. An emergency stop should go at 1, because it must arrive.

The last will

When a device connects, it can leave a note with the broker: “if I vanish without saying goodbye, tell everyone I’m offline”. That note is its last will. If the device’s battery dies or its Wi-Fi drops, the broker notices it’s gone and publishes the note for it.

One seat per name

Each connection has a name, its client ID. The broker gives one seat per name: if a second connection uses the same name, the first one is thrown off. So every device needs a name no other device has.

Check yourself

  1. A robot publishes on lab/robot-7/status. Which of these subscriptions get it: lab/#, lab/+/status, +/status?
  2. Why should an emergency stop be retained?
  3. A robot’s battery is pulled out. It never says goodbye. How does everyone else find out?

<details><summary>Answers</summary>

  1. The first two. + matches exactly one part, so +/status would need the status straight after the first part, and there’s a robot’s name in between.
  2. So a robot that connects after the stop was pressed still stops straight away, instead of waiting for someone to press it again.
  3. Its last will. The broker notices the robot has gone and publishes the will for it.

</details>

College

Publish and subscribe

MQTT is a publish/subscribe protocol over TCP. Clients never connect to each other. Each one connects to the broker, publishes messages on topics, and subscribes to topic filters. The broker copies each message to every client whose subscription matches. Senders and receivers are decoupled in three ways: they don’t know about each other, they needn’t be connected at the same time (with retained messages), and either can come and go.

The message itself, the payload, is just bytes. MQTT doesn’t care what’s in it: JSON, plain text, or a packed binary layout are all common. The programs on each end have to agree.

Topics and filters

A topic is a UTF-8 string of levels split by /. A subscription is a filter, which may use wildcards:

Filter Matches Doesn’t match
lab/+/status lab/robot-1/status, lab/robot-2/status lab/robot-1/arm/status
lab/estop/+ lab/estop/sam, lab/estop/alex home/estop/sam
lab/# everything under lab, at any depth home/robot-1/status

+ is exactly one level. # is any number of levels and must come last. Topics aren’t created or declared: a topic exists while something publishes to it. A common convention puts a version near the front (lab/v1/...), so old and new programs can share a broker while they’re updated.

Retained messages

A publish with the retain flag set is stored by the broker, one per topic, replacing the last. A new subscriber gets it at once. Publishing an empty retained message deletes it.

Retained messages make the broker a small store of current state: what each device is, whether it’s online, which stops are on. A program that starts up subscribes and has the whole picture in its first second.

QoS

QoS Promise Cost Suits
0 At most once One packet Fast streams, where the next message replaces a lost one: sensor readings, drive commands
1 At least once Publish, then an acknowledgement; resent until it comes Things that must arrive and are safe to get twice: a stop, a state change, an event
2 Exactly once Four packets Things that must arrive once only, like a count or a payment, and can afford the round trips

Why send drive commands at 0? A command resent late is a stale command. A robot that stops by itself when commands stop arriving handles loss better than a resend does.

The last will and the keepalive

A client registers its will (a topic, a message, its QoS and retain flag) when it connects. If the connection ends without a clean goodbye, the broker publishes the will.

How does the broker know? If the connection closes, it knows at once. But a device whose power is cut can’t close anything: its connection just goes quiet. So the client promises a keepalive, the longest it’ll stay silent, and sends a small ping when it has nothing else to say. The broker gives up after one and a half keepalives. With a 5 s keepalive, a device that loses power is marked gone about 7.5 s later.

Client IDs and sessions

Each connection names itself with a client ID, and the broker allows one connection per ID. A second connection with the same ID throws the first off. Two clients sharing an ID throw each other off in turn, forever. It’s a common bug. A device’s MAC address, or a serial number, makes a good unique ID.

A good order to connect in

  1. Connect with a will, such as offline on the device’s status topic, retained.
  2. Publish who it is, then online on the same status topic, retained.
  3. Subscribe to what it listens to.

The will comes first, so there’s no moment when the device is connected but would vanish without a trace. The online status overwrites any retained offline from last time.

Check yourself

  1. Write a filter for every robot’s telemetry in every lab, for topics shaped <lab>/<robot>/telemetry.
  2. A device with a 10 s keepalive loses power. When does its will go out? What if its program crashes instead, and the operating system closes the connection?
  3. Why retain a status, but not a sensor reading sent ten times a second?
  4. Two devices are given the same client ID by mistake. What do you see?

<details><summary>Answers</summary>

  1. +/+/telemetry.
  2. About 15 s later, one and a half keepalives. After a crash the connection closes, so the broker sends the will straight away.
  3. A newcomer needs the last status (is it online?), and it changes rarely. A reading sent ten times a second would be stale as soon as it was saved, and the next arrives within 100 ms anyway.
  4. They keep throwing each other off. Each flickers between connected and not, and their messages stutter.

</details>

University

The protocol on the wire

MQTT 3.1.1 (an OASIS standard) runs over a reliable, ordered byte stream: TCP, or TLS over TCP, usually on port 1883 or 8883. Every packet has a fixed header of one byte for its type and flags, then the remaining length in one to four bytes (seven bits a byte, the top bit saying more follows). The smallest packet, a ping, is two bytes. A QoS 0 publish of a 60-byte message on a 20-byte topic is about 85 bytes plus TCP/IP’s. That overhead is why MQTT suits a microcontroller.

Delivery guarantees

  • QoS 0: PUBLISH, nothing back. Delivery depends only on the TCP connection holding.
  • QoS 1: PUBLISH with a packet identifier, answered by PUBACK. Without the PUBACK, the sender resends with the DUP flag. The receiver needn’t remember what it has already passed on, so a duplicate can reach the application.
  • QoS 2: PUBLISH, PUBREC, PUBREL, PUBCOMP. The receiver remembers the packet identifier between PUBREC and PUBREL, so it passes the message on exactly once.

These guarantees are per hop: publisher to broker, and broker to each subscriber. A subscriber gets the lower of the publish QoS and its subscription’s QoS. And they hold within a session. A client that connects with a clean session loses anything in flight when it disconnects. “Exactly once” means exactly once over one session’s life, not across a crash.

Ordering

By default, the 3.1.1 standard has a broker pass on one publisher’s messages on one topic, at one QoS, in the order it received them. Nothing is promised across publishers, so two programs commanding one robot on one topic can interleave in any order. A sequence number in each message lets the receiver drop one that’s behind.

Failure detection

The keepalive k bounds how long a silent client goes unnoticed: the broker must give up after 1.5k and then publish the will. Lowering k finds failures sooner, at the price of more pings and more false alarms on a congested network. Note what the will can’t tell you: a client whose network froze and a client whose power was cut look the same. The will says “unreachable”, not “stopped”. A machine that must be safe when its link fails needs its own timeout: stop when commands stop arriving, whether the broker has noticed or not.

A blocking client in a control loop

Many small MQTT libraries are synchronous: a QoS 1 publish waits for its PUBACK inside the program’s main loop. If that loop also checks “has it been 500 ms since the last command?”, a slow broker delays the check. The stop can then come late by as much as the publish waits. Capping the wait bounds the lateness: with a 250 ms cap, the stop comes 750 ms after the last command at worst. Choosing the cap is a real-time budget set by the transport’s worst case.

Retained state is last-writer-wins

A retained topic holds one value: the last one published. Two writers to the same topic race, and there’s no compare-and-set. A shared emergency stop on one topic would let one person’s “off” erase another’s “on”. The fix is to give each writer its own topic (estop/<name>, with the broker allowing each person only their own) and have every robot stop while any of them is on. An OR over single-writer registers needs no coordination.

Security

Out of the box a broker trusts anyone who can reach it. A broker for real use takes TLS, and logins: a username and password, or a client certificate. Mosquitto can use the certificate’s name as the login (use_identity_as_username), and an access list gives each login only its own topics. This puts authentication and authorisation in the transport, so the messages themselves carry no secrets.

MQTT 5 adds features 3.1.1 lacks: reason codes on every acknowledgement, message expiry, shared subscriptions that spread messages over several consumers, and request/response helpers.

Check yourself

  1. A publisher sends at QoS 1 and the subscriber subscribed at QoS 0. What does the subscriber get, and when would that be a problem?
  2. Why would QoS 2 for drive commands make driving worse, not better?
  3. A robot’s Wi-Fi freezes for 3 s and comes back. Its keepalive is 5 s. Does its will go out? What should the robot be doing meanwhile?
  4. A robot’s loop stops it 500 ms after the last command, and a QoS 1 publish in the same loop can wait up to 400 ms. What’s the worst-case stop, and what would you change?

<details><summary>Answers</summary>

  1. The lower of the two, QoS 0: a message lost on the broker-to-subscriber hop isn’t resent. It’s a problem for anything that must arrive, such as a stop. Subscribe to those at QoS 1, and retain them, so a reconnecting subscriber gets the current value anyway.
  2. Every command costs four packets and two round trips, and a late command is a stale one. A robot that stops when commands stop already handles loss safely. Exactly-once delivery of an old command is worse than missing it.
  3. No: 3 s is under the 7.5 s the broker waits. But commands stop arriving, so after its own timeout the robot should stop itself.
  4. 900 ms. Cap the wait lower, publish from a separate task so the loop never blocks, or send that message at QoS 0.

</details>

Try it

You need only a container runtime. Start a shell with the Mosquitto broker and its command-line clients in it:

container

container run --rm -it docker.io/library/eclipse-mosquitto:2.1.2-alpine sh

docker

docker run --rm -it eclipse-mosquitto:2.1.2-alpine sh

podman

podman run --rm -it docker.io/library/eclipse-mosquitto:2.1.2-alpine sh

Inside it, start the broker in the background (it prints one line saying it’s running), and a listener for everything under lab:

mosquitto -d
mosquitto_sub -t 'lab/#' -v &

Publish a reading, and a retained status:

mosquitto_pub -t lab/robot-1/telemetry -m '{"battery": 7.6}'
mosquitto_pub -t lab/robot-1/status -m online -r

The listener prints both, each after its topic:

lab/robot-1/telemetry {"battery": 7.6}
lab/robot-1/status online

Now join late. A new subscriber gets the retained status at once, and not the reading. -W 2 stops it after 2 s:

mosquitto_sub -t 'lab/+/status' -v -W 2
lab/robot-1/status online
Timed out

Last, a will. Connect a second “robot” that leaves a will, then kill it without a goodbye:

mosquitto_sub -i robot-2 -t lab/robot-2/cmd --will-topic lab/robot-2/status --will-payload offline --will-retain &
kill -9 $!

The first listener prints lab/robot-2/status offline: the broker published the will. Type exit to leave. The container goes with it.

Learn more