I have the feeling that my teenager son spends too much time in front of the TV. We have a house rule that he is allowed to watch TV only after all homework is done, room is tidied up and for no more than couple hours a day. Of course, this rule is hard to control unless time spent in front of the TV is accurately measured, so I came up with an easy way to log TV usage to emoncms and later analyze that. The approach to do this is similar to the method I used to tracking my presence at home i.e. see if the smart TV is registered at home’s WiFi and log that information along with time stamp. The major difference from my previous approach is that I track the device by its MAC address rather than static IP lease. I figured that I want to track few more devices, so setting up a static IP lease for pinging them is going to take me more time than working out another method using MAC address. I use arp-scan to achieve this, to install it run:

 sudo apt-get update
 sudo apt-get install arp-scan

It basically can scan the network for certain MAC address and respond with the corresponding IP address, if it is registered. We use exactly feature this to establish if the TV is on or off by specifying part or all of the MAC address of the device to track:

[Edit July 1st 2014] Increased retries count as the default was giving me unreliable readouts and suppressed vendor decoding to speed things up:

sudo arp-scan  --retry 5 --quiet --localnet --interface=wlan0 | grep -s -i 10:08:C1

So here is all this implemented in Node-RED:

The TV check function is an exec node-red module that runs the arp-scan command:

A simple function checks if the arp-scan returns anything (meaning the device is on) or a null string, meaning device is off:

The result of this check is published to emoncms node 30. I do some post-processing in emoncms to track the time-on:

So with this data collection running for a while, I will be able to know average TV usage per day.

The script can be easily extended with a rule to turn off the TV if daily usage limit is exceeded, but I do not want to be that tough on the kid. The purpose of this project is to learn more about the TV usage and have arguments to support further usage restrictions or relaxing these, also good for self quantification purposes.

The exact same approach can be applied to tracking usage of anything on your home network that connects to the Internet i.e. game consoles, laptops and so forth.

  (226)

I toyed a bit with the Paradox security protocol last year, tapping into the keypad’s interface in an attempt to decode the messages being exchanged between the base station and the keypads. I had some success: I was able to see the keys being pressed, but what was more important to me was the state of the alarm system and each room’s PIR + each door/window’s reed switch. That didn’t work out very well and I quickly abandoned the project.

Until today. I decided to revisit my work and inspect the options at the base station Spectra SP7000. I noticed a 4 pin port labeled ‘Serial’ and hooked my oscilloscope to identify what’s on there. Looked like the port provides ~13V and TX/RX lines at 5V. I hooked my FTDI cable  set to 9600 Baud, 1 start bit, 8 data bits, 1 stop bit and no parity and it started spitting out chunks of 37 bytes. I then remembered that someone sent me a Paradox protocol documentation over the email following my last year’s attempt to hack in, but it seemed quite different from what I was getting at the keypad level and I just ignored it. Looking at it again today showed me exactly what I was looking for: the 37 bytes protocol:

I am interested in the event group, event sub-group and the command byte.

Event codes 00 and 01 are zone closed and zone opened respectively and the event sub-group specifies the zone. There are other event codes as well, but I am only interested in these.

I decided to use a JeeNode v6 to tap into the serial port and transmit the event to my home automation system. The JeeNode runs on 3.3V so I had to create a voltage divider (used 10K:10K, one of the resistors isn’t clearly visible on the picture below) to bring down the 5V to more Jee-friendly levels:

The MCP1702 on the JeeNode will handle the ~13V that the serial port is providing (it is at the limit indeed). Some more pictures of the setup:

Finally here is my code:

#include <JeeLib.h>      // https://github.com/jcw/jeelib
#define myNodeID 3       // RF12 node ID in the range 1-30
#define network 210      // RF12 Network group
#define freq RF12_868MHZ // Frequency of RFM12B module

#define RETRY_PERIOD 250    // How soon to retry (in ms) if ACK didn't come in
#define RETRY_LIMIT 5     // Maximum number of times to retry
#define ACK_TIME 25       // Number of milliseconds to wait for an ack

char inData[38]; // Allocate some space for the string
byte index = 0; // Index into array; where to store the character

#define LED 6


typedef struct {
     byte armstatus;
     byte event;
     byte sub_event;    
     byte dummy;
 } Payload;
 
 Payload paradox;
 
 
void setup() {
    pinMode(LED,OUTPUT);
    blink(100);
    delay(1000);
    
    rf12_initialize(myNodeID,freq,network); // Initialize RFM12 with settings defined above 

    Serial.begin(9600);
    Serial.flush(); // Clean up the serial buffer in case previous junk is there
    Serial.println("Paradox serial monitor is up");

    blink(1000);
    serial_flush_buffer();
}

void readSerial() {
    while (Serial.available()<37 )  {} // wait for a serial packet                                   
    {
        index=0;
        while(index < 37) // Paradox packet is 37 bytes 
        {
            inData[index++]=Serial.read();
        }
            inData[++index]=0x00; // Make it print-friendly
    }

}

void loop()
{
  readSerial();
  
  Serial.println(inData);

  if((inData[0] & 0xF0)==0xE0){ // Does it look like a valid packet?
    paradox.armstatus=inData[0];
    paradox.event=inData[7];
    paradox.sub_event=inData[8]; 
    rfwrite(); // Send data via RF  
    blink(50); 
  } 
  else //re-align buffer
  {
    blink(200);
    serial_flush_buffer();
  }
}

void serial_flush_buffer()
{
  while (Serial.read() >= 0)
   ; // do nothing
}

void blink(int duration) {
  
  digitalWrite(LED,HIGH);
  delay(duration);
  digitalWrite(LED,LOW);

}


// Wait a few milliseconds for proper ACK
  static byte waitForAck() {
   MilliTimer ackTimer;
   while (!ackTimer.poll(ACK_TIME)) {
     if (rf12_recvDone() && rf12_crc == 0 &&
        rf12_hdr == (RF12_HDR_DST | RF12_HDR_CTL | myNodeID))
        return 1;
     }
   return 0;
  }



//--------------------------------------------------------------------------------------------------
// Send payload data via RF
//-------------------------------------------------------------------------------------------------
 static void rfwrite(){

   for (byte i = 0; i <= RETRY_LIMIT; ++i) {  // tx and wait for ack up to RETRY_LIMIT times
      while (!rf12_canSend())
        rf12_recvDone();
      rf12_sendStart(RF12_HDR_ACK, &paradox, sizeof paradox); 
      rf12_sendWait(0);           // Wait for RF to finish sending 
      byte acked = waitForAck();  // Wait for ACK
      if (acked) {       
        return; 
      }      // Return if ACK received
  
   delay(RETRY_PERIOD);     // If no ack received wait and try again
   }
 }

The code is running for few hours already and all is as expected.

I have a Raspberry Pi with RFM2Pi board running a read-only Raspbian that does my home automation, it captures the wireless packets using a node-red flow. With this new data available, my smart house will be able to know where its inhabitants are (PIR) and take various decisions based on that. I can also use the window/door state information (Reed switches) for smarter climate control, i.e. turn off the thermostat in a room that has the window open.

  (182)

I purchased a battery operated smoke/fire alarm few days ago and it showed up today. It runs on 9V and will make a loud sound if smoke is detected. My intention was to hook it up with my home automation system so that I would receive alert if it would go off including SMS, pushbullet notification to my phone, email etc.

The Funky v1 is ideal for the purpose because it is really flat/tiny and would fit inside the alarm. It will tap into the piezo siren and sleep until the siren is activated. Upon activation, it will make a wireless transmission to my home automation system (Raspberry Pi running Node-Red) for further processing and alerting me on my phone.

The piezo siren is activated with a 200ms series of 6.5Khz 50% PWM at 9V so I had to create a small voltage divider + a capacitor to smoothen the signal out and bring it to 3V so the Funky v1 can handle it. A 4.7K:10K + 1 uF ceramic capacitor works well:

I then created a miniature PCB using 0603 sized SMD components for the voltage divider and low pass filter so that all can fit in the fire alarm sensor:

The Funky v1 is equipped with a MCP1703 LDO that will supply 3.3V from the available 9V.

Everything is fit inside the original case.

The Funky is running the following code, basically it sleeps waiting for a pin-change interrupt from the piezo siren to kick in. It will wake every 30 min to provide heartbeat transmission and let the home automation system know it is alive and well.

//--------------------------------------------------------------------------------------
// harizanov.com
// GNU GPL V3
//--------------------------------------------------------------------------------------

#include <avr/sleep.h>

#ifndef cbi
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#endif
#ifndef sbi
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))
#endif

#include <JeeLib.h> // https://github.com/jcw/jeelib
#include "pins_arduino.h"

ISR(WDT_vect) { Sleepy::watchdogEvent(); } // interrupt handler for JeeLabs Sleepy power saving

#define myNodeID 13      // RF12 node ID in the range 1-30
#define network 210      // RF12 Network group
#define freq RF12_868MHZ // Frequency of RFM12B module

#define RETRY_PERIOD 5    // How soon to retry (in seconds) if ACK didn't come in
#define RETRY_LIMIT 5     // Maximum number of times to retry
#define ACK_TIME 10       // Number of milliseconds to wait for an ack


#define LEDpin 10


//########################################################################################################################
//Data Structure to be sent
//########################################################################################################################

 typedef struct {
  	  int state;	// State
 } Payload;

static Payload temptx;
static int gotPCI;
 
 ISR(PCINT0_vect) {
  temptx.state=bitRead(PINA,1);
  gotPCI=true;
 }

void setup() {

  pinMode(LEDpin,OUTPUT);
  digitalWrite(LEDpin,LOW);
  delay(1000);
  digitalWrite(LEDpin,HIGH);
 
  pinMode(9,INPUT);
  
  rf12_initialize(myNodeID,freq,network); // Initialize RFM12 with settings defined above 
  // Adjust low battery voltage to 2.2V
  rf12_control(0xC040);
  rf12_sleep(0);                          // Put the RFM12 to sleep
//  Serial.begin(9600);
  PRR = bit(PRTIM1); // only keep timer 0 going
  gotPCI=false;
}

void loop() {
  
  pinMode(LEDpin,OUTPUT);
  
  digitalWrite(LEDpin,LOW);   // LED on  
  Sleepy::loseSomeTime(100); 
  digitalWrite(LEDpin,HIGH);  // LED off
  
  if(gotPCI) {  // How did we get here? Thru PinChangeInterrupt or wait expired?)
    temptx.state=1;
    rfwrite(); // Send data via RF   
    Sleepy::loseSomeTime(65000); //JeeLabs power save function: enter low power mode for 60 seconds (valid range 16-65000 ms)
  }
  
  temptx.state=0;  // Set the doorbell status to off
  rfwrite(); // Send data via RF 
    
  Sleepy::loseSomeTime(10000);  //sleep some more to debounce
  
  sbi(GIMSK,PCIE0); // Turn on Pin Change interrupt
  sbi(PCMSK0,PCINT1); // Which pins are affected by the interrupt
  
  gotPCI=false;
  for(int j = 1; ((j < 30) && (gotPCI==false)); j++) {    // Sleep for 30 minutes
    Sleepy::loseSomeTime(60000); //JeeLabs power save function: enter low power mode for 60 seconds (valid range 16-65000 ms)
  }
  
//  set_sleep_mode(SLEEP_MODE_PWR_DOWN); // Set sleep mode
//  sleep_mode(); // System sleeps here

  cbi(GIMSK,PCIE0); // Turn off Pin Change interrupt

}

//--------------------------------------------------------------------------------------------------
// Send payload data via RF
//--------------------------------------------------------------------------------------------------
static void rfwrite(){
   bitClear(PRR, PRUSI); // enable USI h/w
   for (byte i = 0; i <= RETRY_LIMIT; ++i) {  // tx and wait for ack up to RETRY_LIMIT times
     rf12_sleep(-1);              // Wake up RF module
      while (!rf12_canSend())
        rf12_recvDone();
      rf12_sendStart(RF12_HDR_ACK, &temptx, sizeof temptx); 
      rf12_sendWait(2);           // Wait for RF to finish sending while in standby mode
      byte acked = waitForAck();
      rf12_sleep(0);              // Put RF module to sleep
      if (acked) { return; }
  
   Sleepy::loseSomeTime(RETRY_PERIOD * 1000);     // If no ack received wait and try again
   }   
   bitSet(PRR, PRUSI); // disable USI h/w
}

// Wait a few milliseconds for proper ACK
 static byte waitForAck() {
   MilliTimer ackTimer;
   while (!ackTimer.poll(ACK_TIME)) {
     if (rf12_recvDone() && rf12_crc == 0 &&
        rf12_hdr == (RF12_HDR_DST | RF12_HDR_CTL | myNodeID))
        return 1;
     }
   return 0;
 }

The receiving end is  Raspberry Pi with RFM2Pi board running a read-only Raspbian. All the business logic is implemented using a Node-Red flow:

The flow consists of the following components
1) Parser that analyzes the raw payload
2) Data validator that makes sure the payload exists and contains the expected range of values
3) Heartbeat monitor to make sure we hear from the fire alarm often enough meaning it is up and running
4) Alarm activated notification
5) logging to EmonCMS

Here is the code for the flow:

[{"id":"ba386057.845d3","type":"mqtt-broker","broker":"localhost","port":"1883"},{"id":"35befc4f.ca4104","type":"mqtt in","name":"Fire Alarm 1 raw","topic":"home/fire/alarm1","broker":"ba386057.845d3","x":97.22221374511719,"y":1891.222255706787,"z":"31c82d7c.ce37d2","wires":[["c0c8d3dc.3f373"]]},{"id":"214c6848.deb398","type":"pushbullet","title":"","name":"","x":1112.2222137451172,"y":1981.222255706787,"z":"31c82d7c.ce37d2","wires":[]},{"id":"c0c8d3dc.3f373","type":"function","name":"Parse fire node payload","func":"var raw= JSON.parse(msg.payload);\nmsg.environment = new Object();\n\nmsg.environment.AlarmState= (raw[0]); \n\nreturn msg;","outputs":1,"x":153.2222137451172,"y":1945.222255706787,"z":"31c82d7c.ce37d2","wires":[["18d0cda9.e72f32"]]},{"id":"8d10e00f.72ef2","type":"delay","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"2","rateUnits":"day","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":980.2222137451172,"y":1926.222255706787,"z":"31c82d7c.ce37d2","wires":[["214c6848.deb398"]]},{"id":"18d0cda9.e72f32","type":"function","name":"Validate data","func":"\nif (\n (msg.environment.AlarmState!=null && !isNaN(msg.environment.AlarmState) )\n && (msg.environment.AlarmState >= 0) \n && (msg.environment.AlarmState < 2)\n )  \n {\n \tmsg.environment.valid=1\n }\n else {\n \tmsg.environment.valid=0\n }\n\nreturn msg;","outputs":1,"x":189.2222137451172,"y":2002.222255706787,"z":"31c82d7c.ce37d2","wires":[["3e25e8b5.c1da18"]]},{"id":"3e25e8b5.c1da18","type":"switch","name":"Is data valid?","property":"environment.valid","rules":[{"t":"eq","v":"1"}],"checkall":"true","outputs":1,"x":192.2222137451172,"y":2050.222255706787,"z":"31c82d7c.ce37d2","wires":[["d5f0b640.2a0f48","fb09e85c.04f618","2df65c73.d209a4"]]},{"id":"d5f0b640.2a0f48","type":"trigger","op1":"","op2":"1","op1type":"nul","op2type":"pay","duration":"45","extend":"true","units":"min","name":"Heartbeat monitor","x":451.2222137451172,"y":1874.222255706787,"z":"31c82d7c.ce37d2","wires":[["ced533d2.312ad"]]},{"id":"b1434385.4ebcc","type":"inject","name":"Initial heartbeat","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"x":258.2222137451172,"y":1856.222255706787,"z":"31c82d7c.ce37d2","wires":[["d5f0b640.2a0f48"]]},{"id":"fb09e85c.04f618","type":"switch","name":"Alarm activated?","property":"environment.AlarmState","rules":[{"t":"eq","v":"1"}],"checkall":"true","outputs":1,"x":445.2222137451172,"y":1940.222255706787,"z":"31c82d7c.ce37d2","wires":[["b7b3e435.484c18"]]},{"id":"ced533d2.312ad","type":"function","name":"Prepare 'node not available' alert","func":"msg.payload= \"Haven't heard from Alarm1 sensor node for a while..\";\nmsg.topic=\"Fire alarm 1 not available!\";\nreturn msg;\n","outputs":1,"x":746.2222137451172,"y":1930.222255706787,"z":"31c82d7c.ce37d2","wires":[["8d10e00f.72ef2"]]},{"id":"b7b3e435.484c18","type":"function","name":"Prepare 'Fire Alarm 1' alert","func":"msg.payload= \"Fire Alarm 1 sensor activated !!!\";\nmsg.topic=\"Fire Alarm 1!!!\";\nreturn msg;","outputs":1,"x":726.2222137451172,"y":2016.222255706787,"z":"31c82d7c.ce37d2","wires":[["6351719c.9cae9"]]},{"id":"2df65c73.d209a4","type":"function","name":"Route messages","func":"// The received message is stored in 'msg'\n// It will have at least a 'payload' property:\n//   console.log(msg.payload);\n// The 'context' object is available to store state\n// between invocations of the function\n//   context = {};\n//create json text\n\nif(msg.environment == null)\n{\n\t//no data - stop here\n\treturn null;\n}\n\njsonText = JSON.stringify(msg.environment);\n \n//var msg1 = {payload:JSON.stringify(msg.environment)};\nvar msg1 = msg.payload;\nvar msg2 = {payload:msg.environment.AlarmState};\nvar msg3 = {payload:Date.now()};\n\nreturn [msg1,msg2,msg3];","outputs":"3","x":440.2222137451172,"y":2078.222255706787,"z":"31c82d7c.ce37d2","wires":[["c616e74e.39e918"],["3526f324.cad90c"],["e4a17c3e.1b5e8"]]},{"id":"c616e74e.39e918","type":"mqtt out","name":"Send to EmonCMS","topic":"/home/emoncms/out/13","broker":"ba386057.845d3","x":712.2222137451172,"y":2090.222255706787,"z":"31c82d7c.ce37d2","wires":[]},{"id":"3526f324.cad90c","type":"mqtt out","name":"","topic":"home/alarm/fire/sensor1/state","broker":"ba386057.845d3","x":745.2222137451172,"y":2136.222255706787,"z":"31c82d7c.ce37d2","wires":[]},{"id":"e4a17c3e.1b5e8","type":"mqtt out","name":"","topic":"home/alarm/fire/sensor1/lastupdate","broker":"ba386057.845d3","x":764.2222137451172,"y":2178.222255706787,"z":"31c82d7c.ce37d2","wires":[]},{"id":"6351719c.9cae9","type":"delay","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","rateUnits":"minute","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":946.2222137451172,"y":2018.222255706787,"z":"31c82d7c.ce37d2","wires":[["214c6848.deb398"]]}]

I then closed up everything, added a ‘hacked’ sticker and tested it above a burning piece of paper. My phone was alerting me a second later.. neat.

And a video of it in action:

As a conclusion this DIY solution costs fraction of the price of similar commercial products (like the Nest Protect for example) and is way more flexible as well.

  (1839)

The road

Voir en plein écran

The electronics road

A research on the electronics industry, its science, its technics, its men and its geography.

How electronic work ?

• where do the raw material come from ?
• How is produce the tin of  the weldings, the copper for the printed circuit card, silicon for processor ?
• What are transistors, diodes? How do they work ?
• Who are the women, men and children working on this industry?

En français, tout le projet

Fiction sociale présentant en plusieurs tableaux (T) une industrie de l’électronique équitable, son usage, sa conception, sa fin de vie. Cette utopie socio-industrielle prend place au Kazakhstan qui a réellement tous les moyens miniers, financiers et scientifiques de devenir pionnier de ce domaine.

T1: Historique d’une utopie : les bases réelles

Bien que fictionnelle et utopique, l’idée d’une industrie de grande complexité, entièrement équitable, prenant place au Kazakhstan trouve de nombreux échos dans la réalité de la région comme dans l’intérêt grandissant au niveau mondial pour les marchandises issues de filières plus respectueuses de l’Homme et de l’environnement.

Fier de son passé turco-mongol, conquérant, guerrier, commerçant et poète, le Kazakhstan est aussi une patrie de scientifiques et d’industries.

Le kazakhstan compte de grands chercheurs comme le géologue Kanysh Satbayev qui fut un acteur incontournable de la géologie et de la métallurgie soviétique. Terre de sciences et d’aventures industrielles, le Kazakhstan accueil le cosmodrome de Baïkonour.

Enfin, le pays connaissant de très graves problèmes de pollution nucléaire suite aux essais soviétiques dans le polygone nucléaire de Semipalatinsk, et ayant vu la mer d’Aral quasiment disparaître sous ses yeux, nous pourrions imaginer que le pays souhaiterai promouvoir à présent des technologies apaisées.

Organisation sociale basée sur une structure clanique donnant priorité au clan, à la famille. La culture Kazakhe, ses rites, ses chants, sa cuisine imprègnent le quotidien de chacun.

La fiction pourrait ainsi présenter toutes ces caractéristiques réelles et dévier sur l’aspect fictif du film:

Face à ses atouts scientifiques, ses richesses minières et industriels et pour ne pas laisser sa jeunesse délaisser la vie du clan et les activités traditionnelles au profit de gadgets chronophages sera instauré une cause nationale : “l’électronique apaisée”

Adieu jeunesse aux yeux et aux cerveaux usés par des vidéos de moutons faisant du skate et de chevaux “trop mignon” (l’équivalent Kazakh fictif de nos LOL cats).

La fiction repose également sur un point réel supplémentaire montrant qu’un sursaut est possible. Le kazakhstan est un pays où des décisions peuvent être prisent sans formalités, par un clan au pouvoir depuis 1990. C’est le plus grand pays du monde où Mc Donalds n’est pas implanté et ne s’implantera jamais. On raconte (ce n’est pas la fiction ici) que cette décision fût prise par le président pour faire plaisir à sa fille et donner une leçon à la firme dont le représentant se serait mal comporté durant son séjour.

Ainsi, on pourrait imaginer le patriarche devenu sage, lancer son pays dans la grande aventure de l’électronique équitable.

T2 : Dans les mines

Le film présenterait des mines artisanales. Les quantités de minerais prélevés sont faibles. La main d’oeuvre est outillée, protégée, massée. Les technologies mécaniques sont motorisées par traction animale et Eolienne

Des moutons très bien traités transportent le minerais en tirant des chariots, des chevaux choyés actionnent des tapis roulant.

Quand les ouvriers ont atteints les quotas de production décidés, ils cultivent les terres alentour faisant du traitement des terres extraites une priorité.

On voit également des gisement de silicates destinés à la production de silicium

T3 : La métalurgie

L’ambiance est au workshop de forgeron.

Flamme et tablier de cuir.

Production de cuivre.

Production d’étain.

Plus technique dans un pan technique, la transformation des silices ou silicate en silicium

T4 : La fonderie electronique

Le tableau s’ouvre sur un laboratoire très technique pour la production de semi-conducteurs.

Production de Wafer de silicium dans l’ambiance d’un laboratoire d’alchimiste.

Production de composants, processeurs..

T5 : Production d’un téléphone mobile apaisé

Un Smartphone apaisé.

Ecran Noir et blanc à encre électronique E-Ink, pas de vidéo.

Du texte, de la géo-localisation, de l’image vectorielle

Un design inspiré du graphisme traditionnel, des matériaux feutres, bois, argent, robustes et dessinés. N’oublions pas que les kazakhs sont bien placé pour dessiner des objets nomades !

La chaîne de production tient plus de l’atelier d’assemblage de montres haut de gamme que de l’usine géante. On suit de postes de travail en postes de travail le montage d’un téléphone.

T6 : Usages

L’usage est centré sur la facilitation lowtech d’une vie apaisée entre technologies et mode de vie traditionnel.  Les contenus intéressants que l’on souhaite partager sont notés pour être partagés plus tard via des réunions dédiés, par exemple dans une yourte commune aux pieds des grands immeubles modernes. C’est facebook en live. Des informations en temps réel pour certaines activités très sélectionnées à trouver.

Les enfants (et les adultes) ne jouent pas a candy crush ou angry bird mais jouent de la musique sans avoir les yeux sur l’écran (Théremine en option, voir le projet Théremine ). Ils complètent leurs “herbiers” en dessinant faunes et flores, participant ainsi à des projets de cartographie du vivant et des choses qu’ils aiment.

Le téléphone apaisé peut être vu comme un “plug” numérique permettant de faciliter le quotidien de manière créative et non aliénante. Sans écran couleur, sans vidéo, il ne peut exercer sa déloyale et faussée concurrence avec les doux contrastes de la réalité. (Avez vous déjà posé, par une fraîche nuit d’été, votre dos sur un rocher encore chaud du soleil de la journée?)

Data Quality is one of the areas I have been neglecting in my home automation system so far. As a result, I seem to be capturing some ‘garbage’ data along with the good quality data.  Bad data can affect downstream logic, resulting in hard to predict behavior of the system. It is of course hard/costly to maintain very high quality of data, there is an balance point that I am aiming for:

Cost of data quality

Source: http://priyamvid.blogspot.com/2012/08/the-costs-of-poor-data-quality.html

While I am running a small scale home automation system, bad data can (and is) still cause troubles; after all I rely on this data to interact with “things” from the physical and virtual worlds and some of the tasks they perform are of critical importance (hot water boiler controller, alarm system controller, fire alarm system, irrigation system, etc)

My plan is to transition from reacting to data quality failures to proactively controlling and limiting introduction of data flaws in the environment.

To achieve this,  I plan to

• Profile the data
• Establish metrics
• Design and implement DQ rules
• Monitor DQ exceptions

Profile the data

This is step number one, I need to learn more about the data I am collecting, understanding the content and structure of the data will help me with the next steps. I have a number of

Physical sensors:

• Room/outdoor/heating system temperature sensors
• Humidity Sensors
• Battery Readings of sensor nodes
• Soil moisture sensor
• Power readings
• Security alarm state sensor
• Fire sensor
• Flood sensor
• Shock movement sensor
• Magnetic contact door sensors
• Bathroom scale sensor

Software data delivery bots:

• Outside freezing conditions detectors
• Presence detectors
• GPS location loggers
• Wind speed readings from remote weather stations
• Air pressure readings from remote weather stations
• Heat flux calculations
• TV/game console usage trackers

..and few more of less importance.

The assumption is that anything that can go wrong will go wrong. Given that, I need to understand what are the possible values, payload structures, frequency of sending that each data delivery system can pass on to the home automation system for processing.

For example room temperature reading cannot be less than -5 degrees C indoors and should not be more than 40 degrees C unless something is very, very wrong. Hot water boiler’s temperature should vary in the range 5-99 degrees C, anything outside that range is potential error. The DS18B20 temperature sensor I use will give a reading of 85 degrees C or 127 if it encounters error, so I must ensure these two particular values (although possible valid reading) are not considered valid.

This sort of analysis and study of historical data provides good knowledge of the data and helps plan the next steps. It must be done for all systems that introduce data into the home automation system.

Establish metrics

The next step is to establish metrics. What Data Quality aspects do I want to monitor? What are my tolerance thresholds? The most commonly monitored DQ dimensions that are applicable to my DQ improvement efforts are:

• Completeness
• Domain Integrity
• Redundancy
• Accuracy
• Validity
• Timeliness
• Availability

Wikipedia’s entry for Data Quality says that there are over 200 such measures. The range of possible measures is only limited by the ability to provide a method for measurement.

I try to find the earliest point where bad data is generated and handle it there, before it reaches other business logic.

Completeness

Completeness refers to making sure that the complete set of data makes it across data handoff points. Easy way to check if we got the complete data is to verify that the payload is of expected, pre-defined size. We know the expected payload sizes from out Data Profiling stage, so the task here is to compare the actual received payload structure/length against that expectation. The reasons for having incomplete data could be many, ranging from hardware error to human error. I observe this when by accident I use duplicate node id and send payload of different size.

Domain Integrity

Next check that I plan to implement is to make sure that the data provided is of the correct type and fits the established in the data profiling stage allowable ranges/valid values/matches predefined pattern. This sort of DQ checks is easily performed by validating with code.

Redundancy

My next effort is to reduce the redundant data; I get redundancy mainly due to remote wireless node re-sending its payload after failing to receive ACK or as a side effect of MQTT’s qos level 0 that can deliver one message more than once.

Accuracy

Accuracy is notoriously hard to measure, your best bet is to validate sensor’s input against another credible source. I performed a series of accuracy checks of the DHT22 temperature/air humidity sensor to check on its accuracy. Alternatively, you can check against readings from similar sensor, for example the outside temperature at your location cannot be +/-5 degrees from a nearby weather station.

Some sensors, like GPS, provide accuracy as part of the measurement, making the task easier. I discard inaccurate location readings, where accuracy is > 100m in my location tracking system.

Validity

Data validity is where you make sense of the data in terms of business rules. After data has passed the domain integrity checks, it makes sense to pass it against business rule validation; Can indoor room temperature be < than -10 or > than 40 degrees C? Can hot water boiler temperature be <5 degrees C or > 99 degrees C? Can atmospheric pressure be < 900 or > 1200 where I live?

Timeliness

Timeliness is important data quality dimension when you make decisions based on data provided by other sensors. Would it make sense for the home automation system to turn on heating, if outside temperature reading is 3 days old? We need a mechanism to track the timeliness of the data and expire it based on business rules.

Availability

I have ran into a situation where a remote node has stopped sending for some reason (battery low for example), and I only find out about this days later. The availability of the data is important for decision making, so tracking that dimension and notifying me of disruptions is important.

Design and implement DQ rules

Now that I have established what aspects of Data Quality I want to measure, it is time to implement these. I have completely switched all of my business logic to Node-Red for my home automation system, so these checks will be performed there. Below is an example of my implementation of these rules for my solar hot water tank:

Completeness

Completeness will be ensured by simple check of payload length; should that fail, the flow does not continue:

Domain Integrity & Validity checks

I combine Domain Integrity and Validity checks in one block, probably not ideal from re-usability point of view:

Redundancy

Redundancy is handled by limiting to2 messages per minute. Excessive messages are dropped:

Accuracy

This particular node does not perform accuracy validation, I use that sort of validation in GPS location tracking node.

Timeliness

Timeliness is ensured by providing a MQTT time stamp topic, so that the consuming process’ business logic can decide if the data is good or bad:

Availability

Availability is tracked by a ‘heartbeat monitor’ mechanism; Failing to receive a heartbeat (valid data) within a pre-set time interval results in a notification being sent to me:

The above workflow is available to try out:

Monitor DQ exceptions

In an ideal world, you’d log data quality exceptions and analyze that information to adjust the processes, however this is a small home automation system and I will limit this task to providing debug information only.

Conclusion

As a conclusion from this my effort, the quality of the data I collect for my home automation purposes has improved materially and my trust in it has increased. Well worth the effort.

  (228)

Data Quality is one of the areas I have been neglecting in my home automation system so far. As a result, I seem to be capturing some ‘garbage’ data along with the good quality data.  Bad data can affect downstream logic, resulting in hard to predict behavior of the system. It is of course hard/costly to maintain very high quality of data, there is an balance point that I am aiming for:

Cost of data quality

Source: http://priyamvid.blogspot.com/2012/08/the-costs-of-poor-data-quality.html

While I am running a small scale home automation system, bad data can (and is) still cause troubles; after all I rely on this data to interact with “things” from the physical and virtual worlds and some of the tasks they perform are of critical importance (hot water boiler controller, alarm system controller, fire alarm system, irrigation system, etc)

My plan is to transition from reacting to data quality failures to proactively controlling and limiting introduction of data flaws in the environment.

To achieve this,  I plan to

• Profile the data
• Establish metrics
• Design and implement DQ rules
• Monitor DQ exceptions

Profile the data

This is step number one, I need to learn more about the data I am collecting, understanding the content and structure of the data will help me with the next steps. I have a number of

Physical sensors:

• Room/outdoor/heating system temperature sensors
• Humidity Sensors
• Battery Readings of sensor nodes
• Soil moisture sensor
• Power readings
• Security alarm state sensor
• Fire sensor
• Flood sensor
• Shock movement sensor
• Magnetic contact door sensors
• Bathroom scale sensor

Software data delivery bots:

• Outside freezing conditions detectors
• Presence detectors
• GPS location loggers
• Wind speed readings from remote weather stations
• Air pressure readings from remote weather stations
• Heat flux calculations
• TV/game console usage trackers

..and few more of less importance.

The assumption is that anything that can go wrong will go wrong. Given that, I need to understand what are the possible values, payload structures, frequency of sending that each data delivery system can pass on to the home automation system for processing.

For example room temperature reading cannot be less than -5 degrees C indoors and should not be more than 40 degrees C unless something is very, very wrong (house on fire). Hot water boiler’s temperature should vary in the range 5-99 degrees C, anything outside that range is potential error. The DS18B20 temperature sensor I use will give a reading of 85 degrees C or 127 if it encounters error, so I must ensure these two particular values (although possible valid reading) are not considered valid.

This sort of analysis and study of historical data provides good knowledge of the data and helps plan the next steps. It must be done for all systems that introduce data into the home automation system.

Establish metrics

The next step is to establish metrics. What Data Quality aspects do I want to monitor? What are my tolerance thresholds? The most commonly monitored DQ dimensions that are applicable to my DQ improvement efforts are:

• Completeness
• Domain Integrity
• Redundancy
• Accuracy
• Validity
• Timeliness
• Availability

Wikipedia’s entry for Data Quality says that there are over 200 such measures. The range of possible measures is only limited by the ability to provide a method for measurement.

I try to find the earliest point where bad data is generated and handle it there, before it reaches other business logic.

Completeness

Completeness refers to making sure that the complete set of data makes it across data handoff points. Easy way to check if we got the complete data is to verify that the payload is of expected, pre-defined size. We know the expected payload sizes from out Data Profiling stage, so the task here is to compare the actual received payload structure/length against that expectation. The reasons for having incomplete data could be many, ranging from hardware error to human error. I observe this when by accident I use duplicate node id and send payload of different size.

Domain Integrity

Next check that I plan to implement is to make sure that the data provided is of the correct type and fits the established in the data profiling stage allowable ranges/valid values/matches predefined pattern. This sort of DQ checks is easily performed by validating with code.

Redundancy

My next effort is to reduce the redundant data; I get redundancy mainly due to remote wireless node re-sending its payload after failing to receive ACK or as a side effect of MQTT’s qos level 0 that can deliver one message more than once.

Accuracy

Accuracy is notoriously hard to measure, your best bet is to validate sensor’s input against another credible source. I performed a series of accuracy checks of the DHT22 temperature/air humidity sensor to check on its accuracy. Alternatively, you can check against readings from similar sensor, for example the outside temperature at your location cannot be +/-5 degrees from a nearby weather station.

Some sensors, like GPS, provide accuracy as part of the measurement, making the task easier. I discard inaccurate location readings, where accuracy is > 100m in my location tracking system.

Validity

Data validity is where you make sense of the data in terms of business rules. After data has passed the domain integrity checks, it makes sense to pass it against business rule validation; Can indoor room temperature be < than -10 or > than 40 degrees C? Can hot water boiler temperature be <5 degrees C or > 99 degrees C? Can atmospheric pressure be < 900 or > 1200 where I live?

Timeliness

Timeliness is important data quality dimension when you make decisions based on data provided by other sensors. Would it make sense for the home automation system to turn on heating, if outside temperature reading is 3 days old? We need a mechanism to track the timeliness of the data and expire it based on business rules.

Availability

I have ran into a situation where a remote node has stopped sending for some reason (battery low for example), and I only find out about this days later. The availability of the data is important for decision making, so tracking that dimension and notifying me of disruptions is important.

Design and implement DQ rules

Now that I have established what aspects of Data Quality I want to measure, it is time to implement these. I have completely switched all of my business logic to Node-Red for my home automation system, so these checks will be performed there. Below is an example of my implementation of these rules for my solar hot water tank:

Completeness

Completeness will be ensured by simple check of payload length; should that fail, the flow does not continue:

Domain Integrity & Validity checks

I combine Domain Integrity and Validity checks in one block, probably not ideal from re-usability point of view:

Redundancy

Redundancy is handled by limiting to max of 2 messages per minute. Excessive messages are dropped:

Accuracy

This particular node does not perform accuracy validation, I use that sort of validation in GPS location tracking node.

Timeliness

Timeliness is ensured by providing a MQTT time stamp topic, so that the consuming process’ business logic can decide if the data is good or bad:

Availability

Availability is tracked by a ‘heartbeat monitor’ mechanism; Failing to receive a heartbeat (valid data) within a pre-set time interval results in a notification being sent to me:

The above workflow is available to try out:

Monitor DQ exceptions

In an ideal world, you’d log data quality exceptions and analyze that information to adjust the processes, however this is a small home automation system and I will limit this task to providing debug information only.

Conclusion

As a conclusion from this my effort, the quality of the data I collect for my home automation purposes has improved materially and my trust in it has increased. Well worth the effort.

  (689)

Dyn and No-IP have been for long time my choice for dynamic DNS, but both have recently made steps towards commercializing their services and implementing monthly confirmation process. I think it is time for me to move away and look for alternative. DuckDNS seems to be exactly what I was looking for, easy to use, reliable and most important – free. They support a number of devices already, but I wanted to implement my updater in Node-RED. The reason for this is so that I can keep all of my setup in one packaged environment for ease of management/transfer.

Here is my Node-RED flow to update DuckDNS, the flow was heavily influenced from Paul’s No-Ip updater:

[
  {
    "id": "a34035d4.5cbfc8",
    "type": "inject",
    "name": "Every 30 minutes",
    "topic": "",
    "payload": "",
    "payloadType": "none",
    "repeat": "1800",
    "crontab": "",
    "once": true,
    "x": 149,
    "y": 55,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "83464437.7cb9b8"
      ]
    ]
  },
  {
    "id": "e06ab7d2.1f9548",
    "type": "http request",
    "name": "Update DuckDNS",
    "method": "GET",
    "url": "https://www.duckdns.org/update?domains=****YOUR-DOMAIN****&token=***YOUR-TOKEN****&ip=",
    "x": 335,
    "y": 273,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "38e88442.c7177c"
      ]
    ]
  },
  {
    "id": "38e88442.c7177c",
    "type": "switch",
    "name": "Was update sucessful?",
    "property": "payload",
    "rules": [
      {
        "t": "eq",
        "v": "OK"
      },
      {
        "t": "neq",
        "v": "OK"
      }
    ],
    "checkall": "true",
    "outputs": 2,
    "x": 316,
    "y": 325,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "78932790.876cd8"
      ],
      [
        "efc4308c.103bd",
        "b78d5573.4872a8"
      ]
    ]
  },
  {
    "id": "ae6e141.f5191e8",
    "type": "delay",
    "name": "One notification/day",
    "pauseType": "rate",
    "timeout": "5",
    "timeoutUnits": "seconds",
    "rate": "1",
    "rateUnits": "day",
    "randomFirst": "1",
    "randomLast": "5",
    "randomUnits": "seconds",
    "drop": true,
    "x": 322,
    "y": 448,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "fb043e44.04fbc"
      ]
    ]
  },
  {
    "id": "fb043e44.04fbc",
    "type": "pushbullet",
    "title": "DuckDNS update failed",
    "name": "DuckDNS update failed",
    "x": 315,
    "y": 502,
    "z": "f810b109.07ef5",
    "wires": []
  },
  {
    "id": "3b8d48f5.c472b8",
    "type": "comment",
    "name": "No",
    "info": "",
    "x": 70,
    "y": 398,
    "z": "f810b109.07ef5",
    "wires": []
  },
  {
    "id": "f16adf9d.0e952",
    "type": "comment",
    "name": "Yes?",
    "info": "",
    "x": 530,
    "y": 320,
    "z": "f810b109.07ef5",
    "wires": []
  },
  {
    "id": "efc4308c.103bd",
    "type": "debug",
    "name": "",
    "active": false,
    "console": false,
    "complete": false,
    "x": 99,
    "y": 444,
    "z": "f810b109.07ef5",
    "wires": []
  },
  {
    "id": "78932790.876cd8",
    "type": "function",
    "name": "Reset fail counter",
    "func": "context.duckdnsfails=0;\nreturn msg;\n",
    "outputs": "0",
    "x": 716,
    "y": 320,
    "z": "f810b109.07ef5",
    "wires": []
  },
  {
    "id": "b78d5573.4872a8",
    "type": "function",
    "name": "Increment fail counter, check if >5",
    "func": "duckdnsfails = context.duckdnsfails || 0;\nduckdnsfails++;\ncontext.duckdnsfails = duckdnsfails;\n\n//Fsiled to update 5+ times?\nif(duckdnsfails <=6) {\n\treturn [msg,null];\n}\nelse {\n    //shout out last known IP\n    msg.payload=context.lastip;\n\treturn [null,msg];\n}",
    "outputs": "2",
    "x": 283,
    "y": 400,
    "z": "f810b109.07ef5",
    "wires": [
      [],
      [
        "ae6e141.f5191e8"
      ]
    ]
  },
  {
    "id": "dd239510.22dc68",
    "type": "function",
    "name": "Has IP Changed?",
    "func": "context.lastip = context.lastip || 'initial';\nvar currentip = msg.payload;\n\nif (context.lastip == 'initial') {\ncontext.lastip = currentip;\n}\nelse if (context.lastip != currentip) {\nmsg.payload = currentip;\ncontext.lastip = currentip;\nreturn msg;\n}\n\n",
    "outputs": "1",
    "x": 327,
    "y": 224.5,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "e06ab7d2.1f9548"
      ]
    ]
  },
  {
    "id": "83464437.7cb9b8",
    "type": "exec",
    "command": "wget -qO- http://bot.whatismyipaddress.com/ ; echo",
    "append": "",
    "useSpawn": "",
    "name": "Check my IP - whatismyipaddress.com",
    "x": 266,
    "y": 119.5,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "652e3ea5.9ad1c"
      ],
      [],
      []
    ]
  },
  {
    "id": "652e3ea5.9ad1c",
    "type": "switch",
    "name": "Integrity check",
    "property": "payload",
    "rules": [
      {
        "t": "regex",
        "v": "\\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\b"
      },
      {
        "t": "else"
      }
    ],
    "checkall": "true",
    "outputs": 2,
    "x": 340,
    "y": 173.5,
    "z": "f810b109.07ef5",
    "wires": [
      [
        "dd239510.22dc68"
      ],
      []
    ]
  }
]

  (467)

I’ve set up a uptime monitor for my home automation system so that I’m notified when it is down. Such event could mean two things: either Internet connectivity issue, or system is down due to power failure, crash etc. The service is provided by UptimeRobot and is free. It can watch over your system by pinging it or issuing a HTTP GET request every now and then. I chose to go for the second option, as by having that request served by Node-RED flow I’ll be able to both verify that Node-RED is up and my system is connected to the Internet to send me notifications should that be necessary.

The Node-RED flow to serve the HTTP request issued by UptimeRobot is extremely simple:

It spits out a message “Raspberry Pi is Up!” then you access pi’ ip address:port/watchdog. I use duckdns for dynamic DNS, so with a bit of port-forwarding on my router the above mentioned page is exposed to the web. I have enabled basic HTTP authentication on Node-RED as a simple security measure against accidental lands on my control page.

My UptimeRobot configuration is as follows:

So it checks my home automation system every 5 minutes and will immediately send me email notification upon detecting it is down.

I have it running of over a week now and my uptime isn’t exactly perfect, but as they say “if you can’t measure it, you can’t improve it”

  (702)

I’ve been unhappy with my old MQTT topic tree structure for quite some time now and did some thinking/research on how to improve it. Google pointed me to Tinkerman’s article on similar subject, and I started planning on improvements.

I will repeat a quote I used in my last post “If you can’t measure it, you can’t improve it”, only this time re-phrase it s “If you can’t visualise it, you can’t improve it”. Yes, my first effort would be to visualise my MQTT topic tree and the rest should be fairly easy.

Ben Hardill  had done some excellent work on the subject and his D3 MQTT topic tree visualizer is exactly what I needed. It was quite tricky to get it to work. I wanted to use the new Mosquitto MQTT support for websockets, so I had to upgrade to that new version. Websocket support must be explicitly enabled when building Mosquitto, thankfully Jeremy Gooch has created *excellent* instructions for doing that, I could not have done it without these.

Finally, I had to modify Ben’s mqttws31.js file to work with Mosquitto 1.4, my fixed version is on Github.

With the visualization working, I did about 1003 iterations to get to the following structure, still lots of nodes to add and in rough shape:

The root contains three folders to group my connected things into the categories “places”, “cloud” and “people”. I plan to add more categories there, for example “objects” to track vehicles.

The “places” will contain places and properties that are part of my IoT home automation system, starting with “our place”:

“Our place” is a house with garden, so I broke this in two: the “house” and “outside”. The “house” is branched into three sub-categories: “common”,”first floor” and “second floor”. The “common” section will contain common components that can’t be grouped otherwise. The “outside” will obviously group anything in the garden like outside conditions,greenhouse, front door etc (all WIP)

The “Common” category currently has these leaf nodes:

The “raspberry pi” branch expanded:

The “rfm2pi” leaf represents the the rfm2pi board that handles the wireless transmissions, it has incoming and outgoing leafs. The “out” topic is used to transmit, I use it to send time packets every 5 minutes for example and also to send control packets to remote nodes. Node-RED subscribes to the different NodeID leafs in the “in” topic to handle the raw packets of data and further process it, feeding the other branches with processed data.

The other leafs in this branch are:

The “power monitor” node data is provided by my sensing shield project, still working nicely.

The “solar” node data is provided by the hot water tank controller I described here, the firmware has changed notably from then, but the hardware is same. Note that it is a bi-directional node with “in” and “out” leafs because it can control the heater element in the water tank as well should the sunshine be less than needed to heat up the water.

The “doorbell” node is fed by my IoT doorbell project.

The “security system” node is fed by my home security system interface project.

The “heat pump” node is still work in progress, I plan to have both “in” and “out” sub-topics as it can both control and provide information about the heat pump state. Using my Daikin controller project there.

The “first floor” branch has the following structure:

My friend Glyn Hudson of OpenEnergyMonitor has sent me an EmonGLCD unit as a gift, this is the data provider for temperature in the “kitchen” node.

I have a fire sensor alarm project that is feeding the “fire alarm” node in the “kitchen” branch.

The house’s door, window and PIR states are fed by my security system interface project. That data helps the central heating system, I plan to stop the room thermostats if a window is open.

Living room temperature is provided by a Funky+DS18b20 project.

TV status is provided by the device tracking project. I also can control its state using the LG TV controller project.

“Second floor” branch has lots of to-dos, but in its current shape looks like this:

Room temperature data providers are  Funky+DS18b20 project.

PS3 game console status is fed by the device tracking project.

“Outside” branch looks like this, again I do have still more branches to add:

I have two software bots to feed WeatherUnderground’s forecast and current weather data, quite practical since I do not have rain/pressure/wind sensors available. I use the forecast data to send myself frost alerts and prepare the garden for such events. Still need to blog about these, as I am doing all this in Node-RED now.

The Funky+DHT22 project is the data provider for outside temperature/humidity. That data is being enriched by adding calculated  dew point as well.

The “cloud” branch is naturally grouping all cloud connected “things” like EmonCMS, Pushbullet, Twitter, email, Twilio etc. Still WIP.

Node-RED flows watch over the emoncms/out topic and send that data to EmonCMS.

The twitter/out topic is what gets published to my house’s Twitter account. Follow it

..and finally the “people” branch:

Presence detection project feeds my and wifey’s presence at home topics.

Weight is being tracked with my keypad project.

My location is being tracked with OwnTracks, still need to blog about that project.

That’s roughly where I stand now on my MQTT topic tree restructuring project. I enjoy looking at the D3 visualizer, it is animated and data flies in as it arrives, very beautiful indeed. I like to think of it as my house’s central nervous system visualised.

  (444)

I needed to clone my current Raspberry Pi SD card image to use for another headless project (on another network), but then realized the network settings were my home ones. I had access to a PC only, so could not (easily) modify the settings for that other Pi to connect to the other network. To avoid that for the future, I’ll just move the network setting files to /boot (which is accessible on a PC) and make a Symbolic Link to these in the original location:

cd /etc/network
sudo cp interfaces /boot/interfaces
sudo mv interfaces interfaces.bak
sudo ln -s /boot/interfaces interfaces

cd /etc/wpa_supplicant/
sudo cp wpa_supplicant.conf /boot/wpa_supplicant.conf
sudo mv wpa_supplicant.conf wpa_supplicant.conf.bak
sudo ln -s /boot/wpa_supplicant.conf wpa_supplicant.conf

With this change applied, I’ll be able to plug in the SD card into a PC, modify the network settings and get the Pi connected in no time. Security-wise this is probably not a good idea, but I am taking the risk anyway. (436)

It’s been a year since the last post on this weblog. A year is a long time. The world has changed in many ways, and technology has advanced in just as many, but completely different ways. I have also progressed, in the sense that I’ve been exploring and learning about lots of new things in the world of electronics, software, and physical computing.

Some things have solidified, such as my main laptop, which is still the same as three years ago (an 11″ MBA), because the shiny big fast one went to my daughter Myra, who has a far bigger need for that sort of hardware – for her photography and video work. Another solidifying trend has been my touch typing, which is now at the point where I do so 95% of the time (editing code still makes me go for the hunt-and-peck mode, occasionally).

Other things have stagnated, such as most notably the work on writing The Jee Book. There are pages and pages with words and images on my laptop, but I don’t like them one bit, and will not publish these as it is today. There is not enough direction, passion, focus, and fun in those draft pages. It would diminish the excitement and joy this deserves.

I’ve given a few presentations and workshops in the past year, but nothing of substance has come out of it all with respect to the JeeLabs site(s). My goal for this month is to get back into higher gear in public. Writing has always been very fulfilling and its own reward for me – I’m looking forward to finding my voice on the web again, in some form or other.

Now the hard part… I could use your help.

A year in solitary confinement (just kidding!) has made it harder for me to understand what you’d like most from JeeLabs. Just to get this clear: I don’t think I can restart the daily schedule of the weblog, as it was up to a year ago. This isn’t only about the effort and energy involved, or the lack of material, but the fact that the resulting stream-of-conscience website that it leads to is a bit hard to navigate through. Also, the resulting collection of articles is really not very practical as a resource – there are too many bits and pieces of information in there which are outdated and at times even misleading by now.

What sort of topics would you wish to see covered? My own interests still tend to gravitate towards long-lasting autonomous wireless sensor nodes. What frequency and size of posts / articles do you like? Should the topics be spread out broadly, or rather focus on some very specific problems? How simple or deep-diving should the information be? Do you want more science and maths, or rather some detailed construction plans? Do you prefer a personal and conversational style (such as this), or more a factual information source?

As always, I will make my own independent choices, but I promise to listen carefully and respectfully to each and every comment you send my way (email to jc@wippler.nl).

Thank you for many dozens of kind, encouraging, honest, and very helpful responses and suggestions over the past few weeks. I started replying to each one individually, but had to give up at some point. Still, thank you for every single word you wrote – it’s not something to ask for all the time, but it sure helps to occasionally hear where you’re coming from, what interests you, what topics you’d like to read more about, and of course also that the fun and delight came across loud and clear.

I’m proud to see that the JeeLabs weblog has touched and inspired such a large and diverse group of people. There really is a huge and happy family out there, spread out across the entire planet, interested in physical computing and willing to learn, share insights, and create new stuff. Good vibes make the world go round!

Somewhere next week, I’ll restart the weblog. It won’t be exactly the same as it was, but it will be on a regular weekly schedule (and sometimes more). It’ll take some time to “find my voice” and get back into a regular and consistent mode again, but I’m really looking forward to it. There is definitely no lack of directions to go into, both deep and broad!

Sooo… let’s see where this renewed adventure will take us, eh?

PS. Comments on this weblog will not be re-enabled, but there’s a new area on the forum.

Ok, here’s the deal: once a week there will be a post like this one, with links to one or more “articles”. These articles will cover some common topic, in small doses. There will be weeks with just one article, and there will be weeks with more than one – depending on the topic, energy levels, complexity, moon phases, and of course the generosity of the muses.

On the day when a post is published, only the first article listed will be available. Each day after that, the next one will be ready. This keeps each daily post nicely limited to a short, and hopefully enjoyable and informative experience.

This first episode contains entries for today, Saturday, and Sunday, just to get started:

• Brutally Bare Basics
• Welcome to ’32
• Just an ARM and a LED

If everything works out, all the articles will add up to form chapters, and the chapters will join forces to produce a book. But that’s for the future to tell, for now the articles will only exist on this weblog (and stay there forever, BTW).

These weekly posts will rarely change once published. Articles on the other hand, can and will most likely get updated, extended, and even rearranged from time to time – to fix errors and omissions, or to include any relevant new information.

For comments, please use the new weblog section of the forum at http://jeelabs.net/.

Enjoy,
-jcw

The articles this week continue on the path just started, hooking up a bare ARM chip, and making it do something silly. No soldering and (almost) no software installation involved.

As before, keep in mind that only the first of the following articles is available right away – the others will be published one at a time, a day apart:

• Long live the breadboard
• Minimal code, the basics
• The upload conundrum
• Modifying the USB BUB

A thought occurred to me recently, and it got stuck in my head:

We can send an autonomous vehicle to Mars using a mighty/majestic Saturn V rocket, or we can launch a fragile/fickle weather balloon and enjoy the view on earth from above.

I’m a guy who prefers the latter. Simple, clean, quiet, and with a journey which is at least as interesting as the result. Which is why you’ll find many posts on this weblog which do things differently, and even repeatedly. Not to be different, but because life is as much about going someplace as it is about getting there.

And yes, I’d love to figure out how to launch a weather balloon some day. There’s probably also a point to be made about the difference in energy consumption…

(for comments, please use the weblog section of the jeelabs.net discussion forum)

En étudiant les objets souvenirs proposés aux touristes des pays du Sud, en Afrique, en Asie et en Amérique du Sud, nous retrouvons beaucoup d’objets réalisés à partir d’une matière première récupérée dans les décharges. Pneus, fer blanc, semelles de tong, capsules, fils électriques … mais aucun n’est produit à partir des millions de tonnes de déchets électroniques.

Il semble donc intéressant de dessiner des “jouets” ou objets souvenirs électroniques basiques utilisant des composants électroniques fonctionnels. Pour la fabrication, il serait possible d’organiser des temps de formation aux rudiments de l’électronique destinés aux personnes vivant déjà de la récupération.

Nous pourrions voir apparaître une nouvelle catégorie d’objets recyclés sur les marchés touristiques du monde entier et observer le développement  d’un savoir mi savant, mi populaire sur la mise en oeuvre d’objets électroniques.

Goûte d’eau dans l’océan, le projet, en affichant la provenance des matériaux via une étiquette intégrée à l’objet permet d’aborder le problème des DEEE, les Déchets d’équipements électriques et électroniques. D’une manière concrète.

Ainsi l’étiquette pourrait mentionner

1/ la localisation de la décharge où ont été collecté les composants

2/ le type et la marque des appareils démantelés pour récupérer les composants (TV, ordi, imprimantes…)

3/ la provenance, si cela est possible, des appareils

“Made from Waste” model #1

Le premier objet pourrait être un petit jouet musicale de type sampler-séquenceur.

Cahier de charges

* Produire un rythme

* Jouer le rythme en boucle

* Enregistrer des sons via un micro

* Jouer en boucle les sons enregistrés

* Jouer en direct des sons via le micro

Du plus simple au plus complexe, le fabricant pourra proposer des modèles selon sa créativité.

Les savoir-faires “classiques” de récupérateur (pneu, fer blanc, capsule…) seront également mis en pratique sur ce projet, tant pour réaliser le support, la structure, la boite de ces objets que pour produire des sons, des effets, des commandes manuelles…

Une interface sans contact pour téléphone portable

Le thérémine est un instrument de musique, inventé en URSS, par Lev Termen, en 1919. Il se joue sans contact.

Le principe est simple. Un son de base est produit par un dispositif électronique. Le joueur de Thérémine module ce son en hauteur et en volume en variant la distance de ses mains vis a vis de deux antennes. Le phénomène de “capacitance” permettant ces variations est connu de toutes personnes se déplaçant à proximité d’une radio FM en faisant des “interférences”

Sur ce principe de capacitance, nous pourrions imaginer un dispositif simple, robuste et original permettant d’interagir avec nos téléphones, tablettes, ordinateurs? Qu’ils soient apaisés ou de dernière génération.

Objetcifs :

• Proposer une interface alternative au tactile pour interagir avec nos machines.
• Imaginer des applications ne nécessitant pas l’usage de l’écran
• Proposer une nouvelle interface, c’est proposer au créateur d’application de nouveaux scénario d’usages pour nos équipements et ouvrir une voie pour de nouveaux projets.
• Si un ensemble de 2 antennes peut convertir n’importe quel smartphone/tablette en thérémine, il est évident que l’interaction sans contact devrait permettre d’imaginer des usages totalement en rupture avec ce qui est proposé actuellement pour l’ensemble de nos équipement.

Avantages

• Redonner une place aux corps et aux mouvements amples dans la manipulation d’applications sur téléphone
• Permettre des applications ne faisant pas appel au tactile et à l’écran.
• Sortir de la standardisation des usages de l’électronique grand public, (ici du tout tactil sur écran minuscule) pour repenser les usages, les besoins, c’est ouvrir une porte à la réflexion sur ces objets monolithiques omniprésents que sont nos smartphones.

–

Bouts de ficelles et Peer to Peer. Communication old school ou apaisée

Dans l’idée de proposer un smartphone apaisé, capable de se connecter à des réseaux alternatifs, capable également de faciliter le partage de fichiers volumineux ou d’objets réels, il semble important de dresser un état des lieux documenté des moyens de communication “à la marge”, alternatifs, de secours ou simplement “à l’ancienne”.

• Dans les prisons du monde entier les détenus parviennent à échanger informations et objets autant entre les murs de la prisons qu’avec l’extérieur via des système de relais humains, de transmission par bout de ficelles (le yoyo) de cellule en cellule.
• Espions et brigands communiquent vie des cachettes, des “boites à lettre morte”.
• En 1941, radio Londres organisa l’opération”V” et demanda aux français de tracer sur les murs de France des “V” de la victoire. L’opération fut un succès, en une nuit la France fut recouverte du signe de la victoire : la résistance avait enfin un “visage” public.
• Alors que des systèmes de télécommunication modernes étaient largement disponibles durant la seconde guerre mondiale des pigeons voyageurs furent utilisés car plus difficiles à intercepter (la nuit) que des ondes radio. Aujourd’hui encore, en Syrie, les opposants au régime de Bachar el Assad utilisent ce dispositif quand les réseaux classiques (Web et téléphones) sont coupés.

Objectifs

* Croiser moyens de communication alternatifs avec le téléphone portable peut permettre d’imaginer des dispositifs puissants et inventifs de communication

* Alors que les opérateurs téléphoniques préparent l’arrivée de la 5G (mille fois plus rapide que la 4G) il est bon de se souvenir que les messages les plus importants ne passe pas forcement par les réseaux classiques et tiennent parfois en quelques mots ou objets précieux.

The ESP8266 System-on-chip (SoC) has recently came out of nowhere and has been taking by storm the IoT DIY world. It is a $4.50 Wi-Fi capable chip that has remarkable specs, obsoleting overnight a number of similar products that are out there. The price factor, availability of SDK and a fast growing community make this chip quite attractive.

The most common usage for the chip so far has been to interface it to MCU and utilize the AT commands that are available in the default firmware, however that is quite an abuse of the power under the hood of the ESP8266. I’ve decided not to use external MCU and utilize the chip itself for a proof-of-concept project:

The project shall be able to

• Run a HTTP daemon
• Be configurable through web interface
• Provide web UI for switching on/off a LED connected to GPIO pin
• Provide web UI for reading a temperature+humidity sensor (DHT22)

There are few different versions of the breakouts available, most popular is the DIP version . However that version only has GPIO0 and GPIO2 routed to the header, so I also purchased the SMD version that has all the pins available. I have made couple interface strip-board PCBs so that I can interface the modules with a “pure” 3.3V FTDI cable:

The capacitor is to provide extra juice during active WiFi operations as the FTDI cable can only provide ~50mA while bursts of Wifi may be in the area of ~200mA. I used a 1000uF capacitor on the PCBs, but a 470uF also worked with no issues.

The SMD version is more interesting, as more of the GPIO pins are available:

To program new firmware on the ESP-03 the following pin connections must be done:

1. CH_PD to VCC
2. GPIO02 to VCC
3. GPIO00 to GND
4. GPIO15 to GND

For normal start CH_PD to VCC and GPIO15 to GND.

Thanks to the great community at esp8266.com that project is possible with minimal effort. The esp8266 httpd project published by Sprite_tm is an incredible piece of art that allows you to run a HTTP server and simple CGI on the chip. One of the challenges with embedded systems is the difficulty in connection/misc configuration. The project overcomes this by providing a web UI that one can use to manage the settings.

If upon startup the chip isn’t able to connect to a WiFi hotspot using the saved credentials, it would automatically activate Access Point mode and you’ll be able to see  “ESP_XXXXXX” where XXXXXX are the last 6 digits of the esp8266’s MAC address:

You can connect to that open AP and navigate to http://192.168.4.1 to scan for wifi networks and enter the connection password:

Main page

WiFi settings page:

The password will be saved and from now on the module will automatically connect to that network. You don’t need to do that, all the other functions are fully accessible without the module being connected to the Internet. I can think of at least dozen use cases where that could be useful. However for my particular project, I’d need the module to be available over the Internet.

Once connected to a network, you’ll probably be wondering what the IP address of the module is. The module uses DHCP, so the address of the module will vary. You can set up a static IP lease if your router allows it, or find the ip address every time you need to use it.

I use the following linux command to find the IP address of the ESP8266 modules connected to my network:

sudo arp-scan –retry 7 –quiet –localnet –interface=wlan0 | grep -s -i 18:fe:34

Navigating to the IP address of the module opens the same UI we were seeing before, same functionality as well. Below are the LED control and DHT22 sensor reading pages:

I have a LED connected to GPIO13, but that could be a relay for example.

The DHT22 page is a simple html page, but it could just be a JSON string that can be periodically polled by http://freeboard.io/ dashboard or Node-RED flow for example. The module can also be set to push the readings to a pre-configured URL at some interval; a UI interface could be set to configure destination URL, push frequency and other settings: all in my to-do list. DHT22 code by fasmide.

MQTT support on the esp8266 is a matter of time, so the web configurable settings and MQTT support will make the module excellent choice for a number of home automation tasks.

On the weaknesses side is the power consumption. During active WiFi operations it can draw up to 250mA, making battery mode quite challenging. I’ll probably stick to my ATTiny84+RFM12B nodes for low power battery operated tasks, those last more than a year on single AAA battery with a boost regulator.

The application source code is available here: app, includes the binaries so you can directly flash those without having to set up the SDK environment:

sudo ./esptool.py–port /dev/ttyUSB0 write_flash 0x00000 firmware/0x00000.bin

sudo ./esptool.py –port /dev/ttyUSB0 write_flash 0x40000 firmware/0x40000.bin

sudo ./esptool.py –port /dev/ttyUSB0 write_flash 0x12000 webpages.espfs

As a conclusion I’d say that this chip is a game changer. I am loving it and will be using it in my next home automation projects a lot.

  (3313)

The idea of starting out with the 8-DIP LPC810 ARM µC occurred to me not very long ago, when I discovered a simple upload mechanism based on the modified FTDI interface. It’s quite an intriguing idea that you can put some pretty advanced decision and timing logic into such a small chip and do so entirely through free open source tools, with all the details fully exposed.

Making an LPC810 do stuff feels like creating our own custom chips: we can upload any software of our own design into the chip, then place it in a project as “control centre” to do fun stuff. Protocol decoders / encoders / converters, LED drivers (e.g. the WS2812 “neopixel”), even a small interpreter or a wireless radio driver, these are all feasible – despite having just 4 KB of flash memory.

Small programmable chips such as the LPC810 demand a relentless drive for simplicity, which is an excellent skill to develop IMO – for whatever physical computing projects you may have in mind.

Anyway. The hardware side is now completely done, with something like this ready to go:

Unfortunately, that’s only half of the story. We still need to address uploads + compilation.

Check out the next set of articles, to be published from Wednesday through Saturday:

• Uploading from Mac OSX
• Uploading from Linux
• Uploading from Windows
• Life after uploads

With this out of the way, we can make an LED blink or fade. Trivial stuff, but note that we’re setting up the infrastructure for embedded software development on ARM chips.

Oh, and if this is too basic for you, see if you can figure out this JeePuzzle… any takers?

(For comments, check the forum area)