The main mechanisms for communicating between devices around the house in JET are via serial communication and via Ethernet (LAN or WLAN) - often in the form of USB devices acting as virtual serial interfaces. For simple radio modules used for a Wireless Sensor Network, we can then use a USB or WiFi “bridge”.
Other (less common) options are: I2C and SPI hardware interfaces, and direct I/O via either digital or analog “pins”. For these special-purpose interfaces, there is the WiringPi library, which is often also ported to other boards than the Raspberry Pi for which it was originally conceived. In this case, a small C program can be used as bridge to MQTT.
Network connections are simple in Linux, and in particular in Go, and will not be covered here right now. Besides, with MQTT in the mix, network access is essentially solved if the other end knows how to act as MQTT client. Getting data into JET via the network is easy - even if some massaging is needed, it can be done later by including some custom code for it in a JET pack.
Serial ports are a different story. There are many serial conventions in use,
with all sorts of settings that you need to get just right to send messages
across: baudrate, parity, stop bits, h/w or s/w handshake - it can be quite a
puzzle. Just enter the man stty command on Linux to see how many different
configuration choices have been added over time…
Fortunately, a few common serial interface conventions will cover the majority of today’s cases, when it comes to “hooking up” a serial device such as an USB-to-serial “dongle”. And Linux tends to have excellent support out of the box for all the different brands and vendors out there. All we need is some glue code in the hub, and we should be able to get serial data in and out.
And yet that’s just step one in this story. Welcome to the “interfacing to the real world” puzzle!
We also need to match the serial data-format choices of the device. Is it text? Is each line one message? Or if it’s binary data: how do we know where a message ends and the next one begins? There are so many different “framing” and other protocol conventions, that this is probably best handled in a custom pack - at least for more complex cases. But even then we need a serial driver which is able to pass all the information faithfully across to that JET pack.
Another area of concern is with the I/O pins other than the send and receive lines: do we need to connect any of the RTS, CTS, DTR, DSR pins? Do they need to be in a certain state?
Eventually, many of these use cases will need to be addressed. For now, let’s just focus on a basic subset and aim for the following scenario:
- plugging in a serial USB adapter, or an Arduino / JeeLink based on one
- opening and closing a specific serial port on request, via MQTT
- being able to receive plain text, line by line, and to send arbitrary text
- adjusting the state of the DTR and RTS pins, for reset / upload control
- configuring standard baud rates, from at least 4,800 to 230,400 baud
- inserting brief delays of a few milliseconds, up to perhaps a few seconds
What about the actual serial data I/O?
This is where MQTT’s pub-sub can help a lot: we can subscribe to a fixed/known topic for each interface, and pass each incoming message to the serial port. The advantage over plain serial, is that any number of processes can do so - if more than one send at the same time, the output will get inter-mixed, but that’s fine as long as each message is a self-contained outbound “packet”.
On the incoming side, there are two uses cases:
pick up each message and pass in along to anyone interested - this is the most natural mode for MQTT and matches exactly with its pub-sub semantics
briefly claim the output while a client “takes control”, which it then relinquishes once done - this is useful for an “uploader”: switch the attached device to firmware-upgrade mode, send new firmware, after which normal operation resumes, with all listeners getting new incoming data
Here is a design, currently being implemented, which supports both modes:
- each serial interface listens to a fixed topic, i.e. the driver for interface
“xyz” subscribes to
serial/xyzto receive all incoming requests and data - if the message is a JSON object (i.e.
{...}), it is treated as a new serial port open request - if the message is a JSON array (i.e.
[...]), then it’s a list of interface change requests - a JSON string is parsed (with escapes properly replaced) and sent as is
- everything else will probably be treated as an error or be ignored
Serial port open requests
Request format, in JavaScript notation:
{
device: <path> // the serial device to open
sendto: <topic> // the topic to forward incoming data to
init: [<commands...>] // initialisation commands and settings
}
Example (JSON):
{"device":"/dev/ttyUSB0","init":["-dtr","%57600"],"sendto":"logger/USB0"}
New open requests implicitly close the serial port first, if previously open.
To close the serial port and not re-open it, we can send an empty message. This is not valid JSON, but will be recognised as a special “close-only” request.
Serial interface change requests
Interface change requests are inside a JSON array and get processed in order of appearance:
"%<number>"- set the interface to the specified baud rate"+dtr"- assert the ¬DTR line (i.e. a low “0” logical level)"-dtr"- de-assert the ¬DTR line (i.e. a high “1” logical level)"+rts"- assert the ¬RTS line (i.e. a low “0” logical level)"-rts"- de-assert the ¬RTS line (i.e. a high “1” logical level)"=text"- send some text as-is, for use between the other requests<number>- delay for the specified number of milliseconds (1..1000)
(note: “¬” denotes a pin with inverted logic: “1” is inactive, “0” is acive!)
Additional requests could be added later, e.g. to switch between text and binary mode, to set a receive timeout, and to encode/decode hex, base64, etc.
Power-up behaviour
While the above is sufficient to use serial ports, it does not address what happens on power-up or after a system restart. Ah, but wait… meet MQTT’s “RETAIN” flag:
when the RETAIN flag is set in a message, a copy of the message is stored in the MQTT server, and a duplicate is automatically sent whenever a matching subscription is set up
by setting RETAIN on the serial port-open request, we indicate that this request is to persist across reboots of the system
only one RETAIN message is kept (the last one), it overwrites any older one
an empty message with the RETAIN flag set removes the last one from the server
In other words: to configure our system, we merely need to send the proper open requests for each serial port once - with the RETAIN flag set, i.e. the MQTT server now acts as persistent configuration store for each of these settings.
Other messages, without the RETAIN flag set, pass through as is - they won’t affect the storage of prior retained messages. Normal outbound data should therefore be published without the RETAIN flag. Likewise for interface change requests: they must be processed, but not stored.
To permanently open a serial port in a different way, we can simply send a new open request message, with RETAIN set, and replace the previous one.
Incoming data
The open request includes a sendto: field, which specifies the topic where
every incoming message is sent. In the initial implementation, data is expected
to come in line-by-line, and each line will be re-published to the given topic.
By using open requests without the RETAIN flag, we can play tricks and
briefly re-open a serial port for a special case, with a different sendto
value. Then, once ready, we simply re-open again with the original topic, and
data will start getting dispatched as before.
As already mentioned, the above mechanisms are currently being implemented and will be included in the hub once the software is working and stable. For real code, see GitHub.