Project Update — Let’s Build a RHDS System from Scratch (part II)


In this tutorial, we will conclude the series on how we built our “Energy Modelling” system. The working RHDS system can be seen here. The best browser to use is Safari. Here’s an example of what you will see.


We will cover the following areas in this tutorial:

  • Extract, Transform and Load (ETL) aspects
  • Data Warehousing
  • User Interface (UI)
  • Sensor Placement

To recap, the system use case is to help homeowners self-diagnose and optimize energy consumption related to Heating and Cooling. The main purpose of RHDS is to measure conduction heat loss through the building envelope and report on Green House Gas/Carbon output, Energy consumption and home comfort aspects.

The system is implemented as a “Service Platform” with Device management capabilities to allow third parties to provide support to the homeowner.

Let’s begin with a brief overview of the system and data flow model.

preliminary_architectureSystem Description: Things have progressed since the last update. We were able to simplify the data flow architecture thereby improving system integrity.The base hardware platform is still the Raspberry Pi 2. The IoT gateway application is Eclipse Kura. To measure the indoor environment conditions such as temperature and humidity we use the TI Sensortag. We continue to use Mongodb as a local data store primarily for data staging and data cleansing. To measure external outdoor conditions we use Internet available weather data. The End user and Service application is Freeboard. The cloud platform is Amazon Web Services (AWS)

Note: Device Management of the Gateway and HVAC (Heat, Ventilation, Air Conditioning system) is done thru an NXP microcontroller, Eclipse Leshan and Wakaama. As Device management was a secondary priority, we decided not to go beyond basic connectivity aspects already covered in Part 1. We were advised against interfacing with a live Heating System w/o professional input. As a proof-of-concept, basic connectivity between Leshan and Wakaama is sufficient toward demonstrating that Device Management is feasible in respect to managing a HVAC unit.

data-flowDataflow Model: The heart of the system is Kura an IOT Gateway. Its primary job is to (1) connect to multiple TI Sensortags, via BLE, and (2) packetize the sensor data into a JSON format and (3) create an MQTT data stream back to Amazon AWS for cloud storage (Data Warehouse). Kura’s secondary responsibility is to push a copy of the sensor data, JSON format, to a local Data Store running on Mongodb. The fronted UI application is Freeboard, “a damn-sexy, open source real-time dashboard builder”. Freeboard is served with JSON data from Amazon Dynamodb thru the Amazon API Gateway REST interface. Device Management is provided by Leshan and Wakaama utilizing LWM2M and CoAP protocols.

Connect the Senor Data to the Cloud

etl_2Establishing a robust ETL mechanism to collect sensor data and push it to the cloud was the most complex part of the project. Since a major objective was to implement RHDS as a Service Platform, reliability and scalability were paramount consideration. Our biggest challenge turned out to be working in an OSGI scenario. None of us on the team had any experience with OSGI. Creating the Kura bundle is not really the challenge. It’s setting up the project manifest correctly. Once that’s understood, it’s pretty much pure JAVA coding. We are very grateful for the assistance provided by the folks working the Eclipse Kura Forum to help us sort out our OSGI issues.

We were very fortunate to be able to use an example Kura project (org.eclipse. kura.example.ble.tisensortag) as the code base to collect and forward the sensor data. It had most of what we needed. It provided a seamless BLE connection process to the TI Sensortag and solid integration points for our valued modules and a simple to implement MQTT payload interface.

After we got the “example.ble.tisensortag” project working the way we wanted it was just a matter of adding various value added software components and integrating them. We will post the entire project on Github as soon. In the meantime here are a few examples of some of the components.

To connect Kura to Mongo we use the Mongo API (java-java-driver-3.4.1.jar) To get it to work in an OSGI fashion we constructed the project manifest like this. Here is an example of the java class we created to connect and work with Mongodb.

To collect the sensor data from Mongo using Kura we did this.

To collect weather data from Environment Canada we did this.

We use Python scripts to do data cleansing, staging and to create value added data.

Data Warehouse

data-warehouseWe implemented two data warehouses for “lack of time sake”. We use Mongodb which runs on the Pi2 and Dynamodb which is part of the AWS cloud service. Mongodb was used for two purposes. The first is to act as “data-pick-up” staging area for Kura. All data forward by Kura via MQTT to Amazon is first staged. The second purpose was to provide a platform to allow for data cleaning prior to forwarding the data to AWS, and enable a way to add value data processing outside of Kura. We recognize that the second function could have been accomplished ALL in the cloud (AWS). However we felt the learning curve to implement unfamiliar AWS services would jeopardize our ability to deliver the project on time.

The interface to Mongodb was implemented using the Mongodb Java driver as explained in the previous section and pymongo API for scripting. All valued data processing was done using Python scripting. Fast and easy!

Connecting the MQTT data stream to AWS Dynamodb is amazingly straight forward. The trick is twofold. 1) Ensure the JSON Key value pairs are all at the root level of the JSON string. If this consideration is acceptable the Dynamodb will automatically load the MQTT payload in a table of your choice. Dynamodb Table selection is achieved thru setting of a AWS rule to match the MQTT Topic. Use the “Split Message – DynamodbV2” option under rule setup for automatically loading MQTT payloads in columns. Also, adding a new column to Dynamodb is a breeze. Just add the new key value pair to the MQTT payload. There is one note of caution. Ensure there are no JSON Key value pairs that contain empty strings like: {“foo“: “”}. If so, the entire MQTT packet will be not be loaded into Dynamodb. That was fun to figure out!

REST API Gateway (AWS)

rest-apiTo serve the User Interface (UI) with JSON data we implemented the AWS “API Gateway service”  Although at first the learning curve was high, in hindsight it’s not that bad. Implementing a new REST interface can be done in minutes. There is an excellent video here which discuss the process here:

In the end it comes down to (1) making sure that the API Gateway service has the proper access privileges to AWS Dynamodb. This is all explained in the YouTube example video, (2) setting up the db query properly (Integration Request, see diagram) and (3) transforming the data (Integration Response, see diagram) from Dynamodb format to JSON. Code snippets for step (2) and (3) are here.

User Interface (URL)

The public User Interface (UI) is served up using Freeboard: It’s very easy to use and ideal for prototyping scenarios. To get a “dashboard like” view going simply connect an external “datasource” to a FB view object (pane). In our case we are serving up JSON data via an AWS REST interface which is front ending the AWS Dynamodb data warehouse containing the sensor and value added data.

You can see our public AWS REST API example URL here:

screen-shot-exampleThe following is an example of the current FB Dashboard UI which is operating on a “real-time” basis.




These are the main UI components:

  • Sensor Connectivity Panel
    • Show the status of connected sensors; very handy for troubleshooting
  • External Environmental Conditions
    • These is basically a mini weather station, the data is required to perform most of our calculations; a weather station is nice side benefit from the project
  • Internal Environmental Conditions
    • Sensor data; Temperature and Humidity; excellent for assessing temperature variances in the home. Now my wife can prove her office is the coldest room!
  • Building Envelope Energy Heat Loss (BTU / kWh)
    • Equivalent to Energy Audit Design Heat Loss; this number represent how much energy is used to heat/cool the home per hour
  • Green House Gas Carbon Output
    • Equivalent to Energy Audit Estimated Green House Gas Emissions; this number presents the yearly estimated Carbon Output (Footprint) of the home
  • Energy Audit data vs. Empirical comparison chart
    • The chart shows the delta between the Energy Audit Heat Load Assessment and RHDS derived values. There are actually three lines on the chart. The EA assessment and RDHS line up exactly and are shown one on top of the other. The third line, which show a minor deviation represent where we started. We call the Graph Line — the “Text Book” assessment.
  • Air Changes per Hour (ACH)
    • Equivalent to Energy Audit Estimated ACH; we were able to calculate the ACH rate and the deviation between EA and RHDS is absolutely negligible. Although we show ACH with a decimal point value, in practical terms it’s not really relevant to assessing the number of air changes per hour.
  • HVAC cycles per hour
    • A representation of how often the HVAC unit is cycling per hour; this number can be used to asses how efficiently the HVAC unit is performing and how well the home building envelope is performing. The shorter the HVAC cycles On and the longer is stay Off translates to lower energy bills and CARBON output. Now the home owner can see whats going on!
  • Indoor Temperature Hysteresis
    • Shows the hi-low range of temperatures in the house; this can be used to optimize the Home Thermostat setting to the appropriate comfort level. It also indicates whether or not the Thermostat is operating correctly, as they do fail. The homeowner should not see more then a 1C temperature swing. Anything more means it’s time to replace the Thermostat.
  • Indoor Comfort level / Humidity Analysis
    • Indicates the comfort level in the home; typically homes should operate between 40% and 60%. Below 40% is typically too dry and above 60% is typically too moist.

Sensor Placement

sensor-orientWhere to place the sensors was an iterative process. In the end we went with four (4) sensors to measure the Temperature and Humidity. The ideas was to “ZONE” off the home into four (4) quadrants, North, South, East and West. This gave us (1) the most consistent results and (2) provides a real-life view of how the homes Building Envelope performs relative to its geographical orientation ( “compass points”). Now the homeowner can see the hot and cold zones and make the necessary adjustments to improve heating and cooling efficiency by re-purposing the hot/cold air to where it’s needed or not.

That concludes this tutorial. We will provide more information related to RHDS system Application and Use in the final wrap-up Video-Blog.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s