Ok, with those (somewhat vaguely-defined) requirements down, how do we even start to move in this direction?
Well, we’re going to have to make a number of decisions. We will need to build at least one program / app, of course, and we’re going to need to run it on some hardware. In a product setting (which this is not!), the steps would involve one or more people planning, designing, implementing, and testing the product-to-be. Iteration, waterfalls, agility, that sort of thing.
JET is different. It’s almost more about the process itself than about the resulting system, because 1) the specs are going to be really fluid (we’ll get new ideas as the project grows and evolves), 2) the timeline is wide open (we’er trying to accomplish something, but the moment we do, we’ll come up with more things to do/add/try/fix/replace), and 3) there is no hard deadline.
Different as it may be, there is a good reason for doing things this way: let’s not constrain what is possible, and leave the door open for unpredictable new ideas, devices, and demands. A home should be a fluid and organic living space. The last thing we want, is to turn our home into an industrial design task. JET will be much better off in permanent research & exploration mode.
And then there’s history: are you going to throw out everything you have and replace it with new technology, just because you could? Is everyone going to dump their fridge because there is now a shiny new “IoT model” for sale? (apart from even wanting such a thing in the first place…)
We need a different model. Some people like to mess with their houses, and are always tinkering with it (even if perhaps not a perfect choice for every spouse…). We need a setup which works yet can evolve while being in use. We don’t want to distinguish production from dev-mode.
If you’re familiar with the Erlang programming language, then you’ll know how some aspects of its design makes it eminently suitable for such a task: in Erlang, every piece of code can be replaced without restarting the system (compare that to going through a “minor” Windows update!). Erlang was designed at Ericsson for its telephone exchanges, i.e. always-on hardware.
But we don’t have to adopt Erlang, a functional programming language which can be quite a challenge to learn (and perhaps also a bit heavy for a Raspberry Pi). What we can do, is to design a minimal core process, in such a way that it doesn’t need to be stopped and restarted during development. There’s a little chicken-and-egg issue here, since obviously we will have to build, and test, and restart that core process first. But the trick is to make the core agnostic about our particular application domain: if the core contains only general-purpose code which has next to nothing to do with JET’s home monitoring and automation, then it also won’t need to change as we start augmenting it with features.
Let’s try and work this out in some more detail:
- JET/Hub is the main core process to rule all the others
- it is launched once and then never ever quits or runs out of memory
- it launches everything else as one or more child processes
- it supervises these processes, taking action when they exit or fail
- it provides some communication mechanism(s) between itself and the rest
- it may contain some more key (but generic) functionality, such as logging
- it may include a (generic and robust!) database, just to simplify the system
- it may also include a (generic and robust!) web server, again to simplify
With JET/Hub running, even before we have any monitoring or automation code, we can then start to design and implement one or more child processes, which will be called JET Packs. The implementation language for these packs need not be the same as the hub, all that matters is that the starting and stopping adheres to a few conventions, and that all communication with the hub is well defined and fully compatible accross every pack that will ever be created.
(This diagrams is slightly dated, but still matches most of the current design)
So how will the hub be implemented then? Answer #1: you couldn’t care less. Answer #2: using a language which is very easy to port and install, and which is robust and able to handle messaging extremely well. Answer #3: using the programming language Go, with MQTT for “pub-sub” messaging, Bolt as key-value store database, and Go’s standard net/http package as web server.
JET/Hub has been implemented in an hour, and can be found in the dev branch on
GitHub. Well… that’s a bit of
an exaggeration: this is mostly a prototype for the actual code. Things haven’t
been tied together yet, making the code so far next to useless. We will need a
way to make it all work together and there’s no supervisor logic to manage the
JET packs yet. But it’s safe to say that the entire hub can probably be built
with under 1,000 lines of custom Go code. Which is not too bad for such a key
architectural component of JET!
With the hub running (forever, on Mars, remember?), we still need a way to set up packs, add them to the mix, replace them, and make them do interesting stuff. But apart from these basic requests (over MQTT), there is nothing else we need in the hub to be able to start thinking about the real task at hand: designing and implementing features as a JET pack, using some language.
Some interesting properties emerge out of this approach:
we can run packs on our own local machine during development (even if that means the hub can’t supervise and launch them for us) - as far as the actual operation is concerned, there will be no difference and all messaging will take place via socket-based MQTT, the same as other packs
there is no preferred programming language - the only thing which matters is the protocol and semantics of all the message exchanges over MQTT
authentication can be enforced via MQTT / HTTPS sessions and SSL certificates
a different MQTT implementation can be used, if the hub’s one is not desired
other services from the hub, i.e. web servers and database storage also need not be used, if not needed or if alternatives are preferred: it’s all optional
So in a way there is no core - there are only conventions: MQTT and choices for its topic names and message formats. If Go or any of the packages used in the hub were to vanish or stop being supported, we can look for alternatives as needed and replace the entire hub with a different implementation. This would not affect the overall architecture of JET.
We have now defined a central infrastructure, yet we haven’t really made any limiting choices. Which was exactly the purpose of this whole exercise…