The “database” built into JET/Hub is set up as a general-purpose key-value store - i.e. persistent data storage which can’t store or retrieve anything other than (arbitrary) data by key. Whereby the keys are treated as a hierarchical structure separated by forward slashes (“/”). This database looks very much like an ordinary file system, and also very much like an MQTT topics hierarchy.
This is by design: it makes it easy to treat this persistent data in the hub as a tree-of-files. Where each “file” is usually a JSON object. The size of these objects can range from one byte to multiple megabytes, since the underlying storage implementation can easily accommodate them all.
Speaking of implementation: the hub’s store is now based on BoltDB, which is widely used in Go, is in active development, and appears to be well-designed and robust. And it’s open source.
BoltDB is an “embedded database” package, which means that its implementation is part of the application using it. As an important consequence, no outside access by any other process is possible while the database is open. In the case of the hub, this won’t be very restrictive, as all its functionality is going to be exposed via MQTT requests anyway: the hub is the database server.
Store semantics
Let’s call this a “store” from now on, since it isn’t a full-scale database (no indexes, no multi-user facilities, no user management, just keys and values - albeit arbitrarily many, and nestable).
To store data, we need a (string) key to specify where to store things, and a value (which can be an arbitrary set of bytes).
All keys must start with a slash. Another constraint is that we can’t store any data at the top level, we need to include at least one extra “directory level” in our keys. Empty key segments should be avoided and will at some point probably be rejected. So these are valid keys:
/abc/def (an "abc" directory, with a "def" entry in it)
/a/b/c/d/e/f (5 directory levels, with an "f" entry in the top one)
… while these are not:
abc/def (not starting with a slash)
abc/ (same, and empty entry name)
/abc (need at least one dir and one entry)
/abc/ (empty entry name)
//def (empty directory name)
In the case of “/abc/def/ghi”, there are two directory levels, “abc” and
“def”, plus the final “file-like” entry, which is where the value gets stored.
It really helps to think of these keys as being very much like file-system
paths, except that there are no “.” and “..” special entries, since there is no
concept of a “current directory”.
The values stored can be any non-empty byte sequence, although other parts of JET might impose some further restrictions, such as being valid JSON.
Empty values cannot be stored - they represent the absence of an entry, as will become clear below. But you can store the empty JSON string, represented as a pair of double quotes (“”).
The hub’s store listens to MQTT topics starting with “!” and “@”, i.e. patterns “!/#” and “@/#”.
Storing a value
To store the value “123” in directory “foo”, entry “bar”, send this message to MQTT:
topic: !/foo/bar
payload: 123
I.e. when using the “jet” utility:
jet pub '!/foo/bar' 123
(the “!” needs to be quoted, because it has special significance in the shell)
In the case of JSON objects as values, you can use quoting to get things across without the shell messing things up. For example:
jet pub '!/foo/bar''{"name":"John","age":21}'In both cases, the “foo” top-level directory level will be automatically created if it doesn’t exist. This applies to all intermediate levels.
Deleting a value
To delete a value, store the empty value in it (i.e. a value of zero bytes
long). There are two ways to do this with “jet“:
jet pub '!/foo/bar'''
jet delete '!/foo/bar'To delete all values under directory “foo”, as well as the directory itself, use either of these:
jet pub '!/foo'''
jet delete '!/foo'This will delete “foo” and everything below it, including all sub-directories. Use with caution.
Fetching a value
Now it gets interesting. How do you fetch a value? Or rather, what do you do with that result?
This is implemented in JET is as a “request/reply” pair: the request to fetch a value includes a topic to which the reply (i.e. the fetched value) should be sent. So to fetch the value of directory “foo”, entry “bar”, we need to set up a listener to a unique topic, and send this MQTT message:
topic: @/foo/bar
payload: "<topic>" (a JSON-formatted string)
For example, if we want to receive the value back on topic “abc”, we can send:
jet pub @/foo/bar '"abc"'Note the extra quotes again here, to get that JSON string across properly.
When this request is picked up by the hub, it’ll obtain the requested value from the store and send a message to topic “abc” with that value. It’s a bit like a “call” with a “return value”, and indeed this is essentially a Remote Procedure Call, mapped on top of MQTT.
If the requested key does not exist, an empty value will be returned instead.
Note that if any of the directories don’t exist, nothing will be returned at all.
Listing the entries
One last function is needed to make the above suitable for general use: the ability to find all the entries in a specific directory. This can be done by “fetching” the value of that directory:
jet pub @/foo '"abc1"'The reply sent to the “abc1” topic might be something like:
{"bar":3,"baz":100,"sub":0,"text":30}
Each entry is a key in the returned JSON object, with as value the number of bytes stored for that entry. In the case of sub-directories, this number will be zero.
So in this example, there are three entries, called “bar”, “baz”, and “text”, as well as one sub-directory, called “sub”. We could then follow-up with a new request:
jet pub @/foo/sub '"abc2"'And a reply would be sent to “abc2”, perhaps this one:
{}
Which indicates that “sub” does exist as directory, and that it doesn’t have any entries.
Current status
As of this writing (end Jan 2016), the above is work-in-progress. An early implementation with slightly different semantics has been implemented and is being adapted to match the above API.
Note that this data store - like all other elements of the hub - is totally optional and independent of the rest of JET. Each pack can choose to use this MQTT-based key-value store, or store data using its own approach. One benefit of this “store” is that it’s always available, to every JET pack.