JeeLabs: Temporary post-HouseMon script

With the original Mac Mini server now decommissioned, this also marks the end of the HouseMon installation here at JeeLabs. In this case, it was the older 0.7.0 release - based on Node.js and written in CoffeeScript - HM 0.7 has served us well, and now it’s gone…

Unfortunately, there is no practical replacement for HouseMon at the moment - yet it sure would be a shame to lose all new readings still coming in from over a dozen wireless sensor nodes. So for the time being, we really should keep something going to collect that data…

HouseMon has always logged all its raw incoming data in the form of daily log files. The format of these files has not changed since the very start: text files, one line per reading, a millisecond time-stamp, the actual data in the raw packet format generated by RF12demo, and with each file rotated at midnight (UTC). Here are a few entries:

L 16:06:58.864 usb-ttyUSB0 OK 9 74 36 58 235 130 127 244 234 170 139 30 24
L 16:07:00.334 usb-ttyUSB0 OK 19 96 15 12 21 17 7 0
L 16:07:02.397 usb-ttyUSB0 OK 3 130 151 12 0 130 102 105

This format is easy to parse, handles any number of incoming sources, and above all: it’s the raw incoming data, so it represents the most direct sensor data - before decoding.

There are now some 8 years of log data at JeeLabs, all nicely stored in future-proof flat text files, one per day, and gzipped. The total disk space used is a very modest 765 MB so far.

The easiest for now would be to simply continue to log all incoming packets in this same format. Even if there is no server-side software running to interpret that data right now. Once a new implementation is ready, we can then “replay” all the raw data and convert it to whatever database format the new setup uses.

Here is a small JavaScript file called posthouse/index.js which does just that:

'use strict'; // allows use of new ES6 syntax and features with node.js

let FTDI = 'ttyUSB0';
const PATH = './logger';

let serial = require('serialport');
let fs = require('fs');

fs.mkdir(PATH, () => {});

let dateFilename = (now) => {
  let y = now.getUTCFullYear();
  let d = now.getUTCDate() + 100 * (now.getUTCMonth() + 1 + 100 * y);
  let path = `${PATH}/${y}`;
  if (!fs.existsSync(path)) {
    fs.mkdirSync(path)
  }
  return `${path}/${d}.txt`;
}

let timeString = (now) => {
  let digits = now.getUTCMilliseconds() + 1000 *
              (now.getUTCSeconds() + 100 *
              (now.getUTCMinutes() + 100 *
              (now.getUTCHours() + 100)));
  return digits.toString().replace(/.(..)(..)(..)(...)/, '$1:$2:$3.$4');
}

let options = { baudRate: 57600, parser: serial.parsers.readline('\n') };
let rf12demo = new serial.SerialPort(`/dev/${FTDI}`, options);

let currFile, currDate;

rf12demo.on('open', () => {
  rf12demo.on('data', (data) => {
    let now = new Date;
    if (now.getUTCDate() != currDate) {
      currDate = now.getUTCDate();
      if (currFile)
        fs.close(currFile);
      let filename = dateFilename(now);
      console.log(`${now} - ${filename}`);
      currFile = fs.openSync(filename, 'a');
    }
    data = data.replace('\r', '');
    fs.write(currFile, `L ${timeString(now)} usb-${FTDI} ${data}\n`);
  });
});

Most of the code comes almost unchanged from the HouseMon 0.7 setup, but there are some nice details worth pointing out:

• this is JavaScript, even though there is not a function keyword in sight
• in fact, this is the new“JavaScript ES6”, supported in new browsers and Node.js 5.x
• ES6 adopts some CoffeeScript’isms: “=>” function notation and string interpolation
• to activate ES6 in Node.js 5.3.0, the "use strict" at the top is all that’s needed
• this only depends on one package, installed as: npm install --save serialport
• switching from Mac OSX to Linux required only a “ttyUSB0” device name change

As for the logic of the code itself: all fairly simple stuff, just opening a serial port and saving incoming lines of text in the proper file, while rotating to a new file on each new day.

The above now captures all sensor data, waiting for a new HouseMon design. Onwards!