As I'm almost ready to start building the libraries and code to talk to various devices under development I thought this would be a good time to describe what I am hoping to build.
When I first embarked on this project my main focus was to save energy. My first project (after the Tweet-a-Watt) was to design and build a smart outlet/socket that would transmit current load and voltage data to a central 'coordinator' via a ZigBee wireless network and also allow me (through the coordinator) to switch the load on and off - a fancy programmable outlet really. I'm still working on this project and will report progress in future blogs. The initial aim of the smart outlet was to enable me to save money by switching loads off when not in use, as even in standby mode there can still be a noticeable energy draw to the device. However, it was pointed out to me that the ability to do this would save no more than perhaps a few $ per year - hardly worth the investment in time and materials. The area of major savings is in areas such as heating, air-conditioning, pool pump, etc. In these areas 100s, perhaps 1000s, of $ can potentially be saved annually through clever energy usage. This data point seemed to undermine the whole idea of a smart socket being used as part of an energy saving system. Why spend $20 on a device that will save you $1 per year?
It was something my wife said that made me realize that the focus on energy savings was too narrow. Being a Brit my wife (and therefore myself) is a Tea addict, and the only point at which she took interest in my Smart Energy System (SES) project was when she realized she might be able to program the kettle to switch on from her Blackberry shortly before getting up in the morning. Of course, attaching the kettle to a timer socket would nearly achieve the same thing as long as you got up at the same time every day! However, it made me realize that 'control' was the correct paradigm for the SES, NOT 'energy saving'. Of course you need some level of control to achieve energy savings, but the control paradigm is much broader than the energy saving one - it includes, for example, being able to switch the kettle on from your phone, which does not save energy at all, but offers some level of convenience (assuming you remembered to fill the kettle the night before!). This 'boiling the kettle' example may actually use more energy as I'm sure they'll be times when you reach over to your phone in the morning, switch the kettle on, and then fall back to sleep, and so the water might get boiled several times before you actually drag yourself out of the sack!
So the 'control' paradigm is my new focus. I want to be able to enable and disable any socket / outlet in my house from a central computer, or from a remote device like my iPhone (I'd like the ability to switch lighting on and off when I'm away from home - even better, I'd like a 'pretend-someone-is-at-home' autonomous mode!). I want to be able to collect data, such as current load, voltage, temperature, light levels, movement detection (which can be used for security as well as a useful input for developing energy-efficient configurations), etc. The method of communication between all the 'nodes' on this network will be via a ZigBee mesh network. I want to be able to view the status of my device network at any point locally and remotely. With all this data I will then be able to develop algorithms to, for example, minimize my heating and cooling needs, and so I want to be able to implement programmable energy schedules for any node on the network - so the network has to have a certain level of autonomy.
With such a network I will have the convenience of control, and be able to realize the benefits of energy saving configurations. So my wife will be happy as she'll be able to start the kettle boiling as she leaves the kids' school for home, and I will be happy because our bank account balance should look healthier (and I'd've built something pretty cool!).
So that's the challenge - stay tuned for progress reports... and if you have any thoughts on functionality you would like to see with such a network then feel free to comment. One thought I had was to use the control software and data set to develop algorithms that would aim to save enough money in a year to pay for a weekend away... once you have the control a whole new world of applications opens up...
Saturday, February 20, 2010
Installing MySQL on the TonidoPlug
An important element of the Smart Energy System that I am building is a local and remote database to store telemetry packets, processed data, and control / status strings. For remote storage I plan to use Amazon's SimpleDB service (which I have tested previously on the PlugPC), and for local storage I plan to use a MySQL database structure. So my next job is to install MySQL on my Plug.
I first tried installing MySQL 5.0, but this was a complete disaster and the installation kept finding it's way into a infinite error loop... this was removed tout de suite. The problems I saw during the installation of 5.0 might be why version 5.1 emerged! Anyway, I tried again with MySQL 5.1. After SSHing in to my Plug, to install 5.1 I executed:
apt-get install mysql-client-5.1 mysql-server-5.1 (these just happened to be the versions available on the repository I am using - as of writing version 5.5 is available)
The installation process ran without incident, but for some reason no password was created during installation and I could not gain access to the MySQL interactive mode. The following sequence of commands fixed this particular problem.
First, try to stop the MySQL daemon:
/etc/init.d/mysql stop
Followed by:
mysqld_safe --skip-grant-tables & (which is the recommended way to start MySQL)
mysql -u root -p (to enter as 'root' - you will be prompted for your system 'root' password. If successful you will now be in MySQL interactive mode)
mysql client - mysql - root
use mysql
update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root' (should probably choose a MySQL 'root' password that is different from your system 'root' password)
Now, we're ready to start creating tables and fields, and storing and quering data... I'll save this treat for another day when I have had time to decide what table and fields I need for my SES.
To launch MySQL from the command line:
mysql -u root -p (you will then be prompted for your MySQL 'root' password)
The MySQL documentation for getting started is available if you want to start playing right away.
I first tried installing MySQL 5.0, but this was a complete disaster and the installation kept finding it's way into a infinite error loop... this was removed tout de suite. The problems I saw during the installation of 5.0 might be why version 5.1 emerged! Anyway, I tried again with MySQL 5.1. After SSHing in to my Plug, to install 5.1 I executed:
apt-get install mysql-client-5.1 mysql-server-5.1 (these just happened to be the versions available on the repository I am using - as of writing version 5.5 is available)
The installation process ran without incident, but for some reason no password was created during installation and I could not gain access to the MySQL interactive mode. The following sequence of commands fixed this particular problem.
First, try to stop the MySQL daemon:
/etc/init.d/mysql stop
Followed by:
mysqld_safe --skip-grant-tables & (which is the recommended way to start MySQL)
mysql -u root -p (to enter as 'root' - you will be prompted for your system 'root' password. If successful you will now be in MySQL interactive mode)
mysql client - mysql - root
use mysql
update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root' (should probably choose a MySQL 'root' password that is different from your system 'root' password)
Now, we're ready to start creating tables and fields, and storing and quering data... I'll save this treat for another day when I have had time to decide what table and fields I need for my SES.
To launch MySQL from the command line:
mysql -u root -p (you will then be prompted for your MySQL 'root' password)
The MySQL documentation for getting started is available if you want to start playing right away.
Sunday, February 14, 2010
My new PlugPC and Mono
Earlier in the week I received my Tonido PlugPC, and I have immediately fallen in love! What a fantastic little device. Within a few hours I'd got it set up to the point of being able to replace the Media Center PC that I was using for all our media streaming, central storage, and email server. It runs a full Linux kernel and so there is little you can't do with it that you can't do with a desktop... and it run off 9Watts instead of 400! Mine's running a little 'hot' as I have a USB hub plugged in to it. Off the hub I have a 16Gb USB drive which I'm booting the Tonido off of, and 3 x 0.5Tb and 1 x 1Tb drives for storage. I won't bore you with all the set-up details for my Tonido as it is not particularly relevant to this blog. My plan is to use a SheevaPlug (the TonidoPlug is really just a SheevaPlug with some web applications running on it) for development, but for some reason there is a 2-3 week delay with my SheevaPlug order, so I'm doing some playing with my TonidoPlug in the meantime. A few of pickies of my setup:
The most signifiant difference between a PlugPC and a desktop is that you have no screen or mouse or keyboard to interact directly with the device. So once you have your PlugPC connected to an ethernet network, the first thing you have to do in find a route in. I chose SSH, and in particular PuTTY. I haven't had much experience with command line work, but it really isn't that hard. Run PuTTY, give it the IP of your PlugPC, select SSH mode (I used Telnet mode to test my email server when I got it installed), ignore the security warning, login as 'root', and you're in.
Given that the SheevaPlug currently only has a 500Mb flash drive, the chances are that you're going to want more drive space for development. I very much doubt we'll have to wait very long before 16+Gb is available onboard. I followed the directions provided here, to boot from a 16Gb USB drive I got on sale from Staples for $29.99. The directions worked to the letter and I had no trouble whatsoever... very straightforward. My Tonido now boots off the USB drive - and certainly much faster than my Windows PC boots off a hard drive... not bad considering my Windows PC is a water-cooled overclocked (3.6GHz) Quadcore, and the Tonido has a single 1.2GHz core!
My next step was to make sure the packages on my Plug were uptodate. To do this I attempted to run:
apt-get upgrade
This bombed out for reasons I can't quite remember, but was someting to do with not being able to write to a particular location. After some Googling the following fixed things:
mkdir -p /var/cache/apt/archives/partial
apt-get autoclean
This seemed to fix 'apt-get' and re-executing 'apt-get upgrade' worked just fine.
At this point I'm keen to start going through the various tests outlined in previous posts (Amazon SimplDB, SerialPort, obviously the WinForms test is pointless on the Plug!), so I need Mono and IronPython installed. First I tried to install Mono with:
apt-get install mono-2.0-devel
The installation seemed to work, but even the simplest 'Hello World' test failed with a 'Segmentation Fault' response. I messed around some more, but kept running into the same error. Eventually I decided to try and compile Mono from source and install it that way. In doing this I found the article here very useful. There are some packages that the Mono configuration process depends on and need to be available before you attempt your Mono compilation. The following 'apt-get' installed what I needed to move forward:
apt-get install bison gcc gettext pkg-config glib-2.0
Next I downloaded the Mono 2.6.1 source to my root folder, and unzipped it:
cd /
wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.6.1.tar.bz2
bzip2 -cd mono-2.6.1.tar.bz2 | tar xvf -
(NB Edit 11/24/2012: Mono source is now at http://download.mono-project.com/sources/mono/ with the latest version today being 2.10.9). This created a 'mono-2.6.1' folder that I CDed into ('cd mono-2.6.1'). The compilation and install was remarkably smooth, if not time-consuming:
./configure --prefix=/usr (if this step fails it is probably due to a missing package. Install the indicated package and try again)
make (this process took several hours)
make install
After the install process completed I executed 'mono -V' to check the currently installed version and received the following:
Mono JIT compiler version 2.6.1 (tarball Thu Feb 11 17:56:14 MST 2010)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www-mono-project.com
TLS: __thread
GC: Included Boehm (with typed GC and Parallel Mark)
SIGSEGV: normal
Notifications: epoll
Architecture: armel, soft-float
Disabled: none
So far, so good. I then installed IronPython 2.6 as detailed in my previous posting, and performed the same Amazon SimpleDB access that I had previously (including the 'mozroots' command). So a far more challenging test than a simple 'Hello World' - the test ran as expected and so I am now hoping that I can claim that Mono 2.6.1 and IronPython 2.6 are installed and running OK on my PlugPC... how cool is that!
The most signifiant difference between a PlugPC and a desktop is that you have no screen or mouse or keyboard to interact directly with the device. So once you have your PlugPC connected to an ethernet network, the first thing you have to do in find a route in. I chose SSH, and in particular PuTTY. I haven't had much experience with command line work, but it really isn't that hard. Run PuTTY, give it the IP of your PlugPC, select SSH mode (I used Telnet mode to test my email server when I got it installed), ignore the security warning, login as 'root', and you're in.Given that the SheevaPlug currently only has a 500Mb flash drive, the chances are that you're going to want more drive space for development. I very much doubt we'll have to wait very long before 16+Gb is available onboard. I followed the directions provided here, to boot from a 16Gb USB drive I got on sale from Staples for $29.99. The directions worked to the letter and I had no trouble whatsoever... very straightforward. My Tonido now boots off the USB drive - and certainly much faster than my Windows PC boots off a hard drive... not bad considering my Windows PC is a water-cooled overclocked (3.6GHz) Quadcore, and the Tonido has a single 1.2GHz core!
My next step was to make sure the packages on my Plug were uptodate. To do this I attempted to run:
apt-get upgrade
This bombed out for reasons I can't quite remember, but was someting to do with not being able to write to a particular location. After some Googling the following fixed things:
mkdir -p /var/cache/apt/archives/partial
apt-get autoclean
This seemed to fix 'apt-get' and re-executing 'apt-get upgrade' worked just fine.
At this point I'm keen to start going through the various tests outlined in previous posts (Amazon SimplDB, SerialPort, obviously the WinForms test is pointless on the Plug!), so I need Mono and IronPython installed. First I tried to install Mono with:
apt-get install mono-2.0-devel
The installation seemed to work, but even the simplest 'Hello World' test failed with a 'Segmentation Fault' response. I messed around some more, but kept running into the same error. Eventually I decided to try and compile Mono from source and install it that way. In doing this I found the article here very useful. There are some packages that the Mono configuration process depends on and need to be available before you attempt your Mono compilation. The following 'apt-get' installed what I needed to move forward:
apt-get install bison gcc gettext pkg-config glib-2.0
Next I downloaded the Mono 2.6.1 source to my root folder, and unzipped it:
cd /
wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.6.1.tar.bz2
bzip2 -cd mono-2.6.1.tar.bz2 | tar xvf -
(NB Edit 11/24/2012: Mono source is now at http://download.mono-project.com/sources/mono/ with the latest version today being 2.10.9). This created a 'mono-2.6.1' folder that I CDed into ('cd mono-2.6.1'). The compilation and install was remarkably smooth, if not time-consuming:
./configure --prefix=/usr (if this step fails it is probably due to a missing package. Install the indicated package and try again)
make (this process took several hours)
make install
After the install process completed I executed 'mono -V' to check the currently installed version and received the following:
Mono JIT compiler version 2.6.1 (tarball Thu Feb 11 17:56:14 MST 2010)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www-mono-project.com
TLS: __thread
GC: Included Boehm (with typed GC and Parallel Mark)
SIGSEGV: normal
Notifications: epoll
Architecture: armel, soft-float
Disabled: none
So far, so good. I then installed IronPython 2.6 as detailed in my previous posting, and performed the same Amazon SimpleDB access that I had previously (including the 'mozroots' command). So a far more challenging test than a simple 'Hello World' - the test ran as expected and so I am now hoping that I can claim that Mono 2.6.1 and IronPython 2.6 are installed and running OK on my PlugPC... how cool is that!
Thursday, February 11, 2010
Accessing Amazon's SimpleDB
The next test in my 'proof of concept' phase of development is to see if I can access Amazon's SimpleDB cloud service. The reason for picking Amazon's SimpleDB is threefold:
The first step is to get hold of a library to do all the work for me. For no particular reason I went for the C# library I found here. The version I downloaded was released on May 20, 2009. I also found a page in the IronPython Cookbook that talks about using an older version of this library in IronPython. The steps for creating the DLL are still valid, but the test scripts themselves are out of date so not much use to anyone (other than to waste some time wondering why one's scrips keep failing!). I have posted the version of the DLL I compiled here.
Boto is another more comprehensive library written in Python that is for all the Amazon cloud services, not just SimpleDB.
Next, you've got to get yourself a SimpleDB account from here. Once you have registered you'll be given a unique AccessKey ID, and a Secret Access Key, both of which are needed to access SimpleDB service.
One more step require: I had performed a few tests in IronPython that worked just fine and then failed when running under Mono. After some googling I found that the issue was that Mono doesn't trust anyone, and so any SSL web accesses fail by default... so of course my Mono IronPython SimpleDB scripts failed! To save you the hassle, all you need to do is install a pile of root certificates. This is done by executing the following at the command line:
mozroots --import --ask-remove --machine
More details about Mozroots can be found here. If you have the latest version of Mono then Mozroots should already be installed.
OK, now we're ready to go. I'd already added some data to SimpleDB and so we'll just look at a simple query in this posting. To run this for yourselves you're going to have to setup you AWS account with Amazon, and stick some data in SimpleDB. I'll add some data entry snippets another time.
The following code is for a simple SimpleDB test that I saved as simpleDB_examples.py:
import clr
clr.AddReference('Amazon.SimpleDB')
from Amazon.SimpleDB import AmazonSimpleDBClient
from Amazon.SimpleDB.Model import ListDomainsRequest, Select Request
service = AmazonSimpleDBClient('[Access Key ID]','[Secret AccessKey]')
# list domains
listDomainsRequest = ListDomainsRequest()
listDomainsRequest.MaxNumberOfDomains = 10
listDomainsResponse = service.ListDomains(listDomainsRequest)
print listDomainsResponse.ListDomainsResult.DomainName
# select items request
selectRequest = SelectRequest()
selectRequest.SelectExpression = 'Select * From [domain]'
selectResponse = service.Select(selectRequest)
# print first attribute namd and value for first returned item
print selectResponse.SelectResult.Item[0].Attribute[0].Name
print selectResponse.SelectResult.Item[0].Attribute[0].Value
Once saved, the test script can be run by executing:
mono ipy.exe simpleDB_examples.py
As I had already created a couple of domains (equivalent to 'tables'), the following was printed to screen on my Ubuntu machine:
List[str]([dmg_data', 'testDomain'])
node_type
end node
Another success! Now we have WinForms, SerialPort access and SimpleDB partially proven on Ubuntu with Mono and IronPython. With a Multi-Threading/Processing example I think I'm nearly ready to get stuck into the main application.
Side note: My first PlugPC (TonidoPlug) was delivered this week, so plenty of fun ahead!
- IF (big 'if' I know) my efforts to develop a Home Smart Energy System turn out to be a resounded (financial) success, I don't want to have to put together all the hardware needed for the huge database that is going to be needed to store all the telemetry related to time-dependent energy usage and sensor configuration / control for potentially thousands (and more if I'm real lucky!) of homes. Cloud services offer a scaleable approach that gets billed in proportion to one's usage;
- The Amazon SimpleDB has a particularly simple API;
- And the Amazon cloud was the first of the big three (Amazon, Google, Microsoft) I believe so I'll go with the one that has years of postings to support forums to help with my own development.
The first step is to get hold of a library to do all the work for me. For no particular reason I went for the C# library I found here. The version I downloaded was released on May 20, 2009. I also found a page in the IronPython Cookbook that talks about using an older version of this library in IronPython. The steps for creating the DLL are still valid, but the test scripts themselves are out of date so not much use to anyone (other than to waste some time wondering why one's scrips keep failing!). I have posted the version of the DLL I compiled here.
Boto is another more comprehensive library written in Python that is for all the Amazon cloud services, not just SimpleDB.
Next, you've got to get yourself a SimpleDB account from here. Once you have registered you'll be given a unique AccessKey ID, and a Secret Access Key, both of which are needed to access SimpleDB service.
One more step require: I had performed a few tests in IronPython that worked just fine and then failed when running under Mono. After some googling I found that the issue was that Mono doesn't trust anyone, and so any SSL web accesses fail by default... so of course my Mono IronPython SimpleDB scripts failed! To save you the hassle, all you need to do is install a pile of root certificates. This is done by executing the following at the command line:
mozroots --import --ask-remove --machine
More details about Mozroots can be found here. If you have the latest version of Mono then Mozroots should already be installed.
OK, now we're ready to go. I'd already added some data to SimpleDB and so we'll just look at a simple query in this posting. To run this for yourselves you're going to have to setup you AWS account with Amazon, and stick some data in SimpleDB. I'll add some data entry snippets another time.
The following code is for a simple SimpleDB test that I saved as simpleDB_examples.py:
import clr
clr.AddReference('Amazon.SimpleDB')
from Amazon.SimpleDB import AmazonSimpleDBClient
from Amazon.SimpleDB.Model import ListDomainsRequest, Select Request
service = AmazonSimpleDBClient('[Access Key ID]','[Secret AccessKey]')
# list domains
listDomainsRequest = ListDomainsRequest()
listDomainsRequest.MaxNumberOfDomains = 10
listDomainsResponse = service.ListDomains(listDomainsRequest)
print listDomainsResponse.ListDomainsResult.DomainName
# select items request
selectRequest = SelectRequest()
selectRequest.SelectExpression = 'Select * From [domain]'
selectResponse = service.Select(selectRequest)
# print first attribute namd and value for first returned item
print selectResponse.SelectResult.Item[0].Attribute[0].Name
print selectResponse.SelectResult.Item[0].Attribute[0].Value
Once saved, the test script can be run by executing:
mono ipy.exe simpleDB_examples.py
As I had already created a couple of domains (equivalent to 'tables'), the following was printed to screen on my Ubuntu machine:
List[str]([dmg_data', 'testDomain'])
node_type
end node
Another success! Now we have WinForms, SerialPort access and SimpleDB partially proven on Ubuntu with Mono and IronPython. With a Multi-Threading/Processing example I think I'm nearly ready to get stuck into the main application.
Side note: My first PlugPC (TonidoPlug) was delivered this week, so plenty of fun ahead!
Monday, February 8, 2010
Connecting to the Serial Port
So I can now build a WinForms based application and have that form appear pretty much the same in Windows (Vista) and Linux (Ubuntu) without making any code changes whatsoever, thanks to Mono. Another 'feature' I need to test run is the ability to access a serial port to take data from the XBee receiver/transmitter that will be the hub of my Smart Energy System.
As I have never undertaken hardware development before (other than in FPGAs - which is pretty much the same a writing software), I needed to get some hand-ons experience with building some actual hardware. I chose to build a Tweet-a-Watt. This is the perfect project for the wannabee home smart grid engineer. It gives us experience soldering and building-up a real circuit board, as well as playing around with the XBee receivers/transmitters - a new(ish) product especially designed for sensor networks. Given that I had no circuit building equipment I also had a fun couple of days at Fry's Electronics putting together my new home electronics lab!
When I built the Tweet-a-Watt I also wrote a simple terminal program to communicate through the USB port to the XBee via a serial adapter. The .NET System.IO.Ports.SerialPort library provides everything needed for such a project. I was pleasantly surprised as to how easy is was to build my simple terminal application... we'll see some of the commands used later on.
For this particular test case I used the XBee adapter that is built as part of the Tweet-a-Watt project. This plugs directly into any USB port. Here's a photo of mine, with my thumb for scale:
As I have never undertaken hardware development before (other than in FPGAs - which is pretty much the same a writing software), I needed to get some hand-ons experience with building some actual hardware. I chose to build a Tweet-a-Watt. This is the perfect project for the wannabee home smart grid engineer. It gives us experience soldering and building-up a real circuit board, as well as playing around with the XBee receivers/transmitters - a new(ish) product especially designed for sensor networks. Given that I had no circuit building equipment I also had a fun couple of days at Fry's Electronics putting together my new home electronics lab!
When I built the Tweet-a-Watt I also wrote a simple terminal program to communicate through the USB port to the XBee via a serial adapter. The .NET System.IO.Ports.SerialPort library provides everything needed for such a project. I was pleasantly surprised as to how easy is was to build my simple terminal application... we'll see some of the commands used later on.
For this particular test case I used the XBee adapter that is built as part of the Tweet-a-Watt project. This plugs directly into any USB port. Here's a photo of mine, with my thumb for scale:
On Windows Vista the drivers were installed automatically and it was also recognized as a serial port communications link... couldn't be easier. The device was invariably to be found at "COM4" on my Vista machine.
My new Tweet-a-Watt was also active and so the XBee receiver / transmitter (referred to as a 'coordinator' as it is intended to manage, or coordinate, the soon to be built sensor network) connected to my PC was receiving telemetry updates every 2 seconds.
This test was particularly easy.
On my Vista machine (where the 'coordinator' was connected via "COM4", I opened-up an IronPython interactive session (by entering 'ipy' at a command prompt). The following commands were then executed:
>>> import clr
>>> clr.AddReference('System')
>>> from System import *
>>> serialPort = IO.Ports.SerialPort("COM4")
>>> serialPort.BaudRate = 9600
>>> serialPort.DataBits = 8
>>> serialPort.Open()
>>> serialPort.ReadLine()
after entering the last command the screen filled with a load of garbage-looking data which represent a number of data packets from the Tweet-a-Watt. I expected this test to be the easier of the two, although doing this through IronPython was still pretty cool. The second test involved repeating the same trick on my Ubuntu system.
Firstly I realized that I could not do this on my Virtual Ubuntu because Microsoft's Virtual PC 2007 does not support Ubuntu and would not make "COM4" available to it :-( So I had to install a full version of Ubuntu on a 'real' (as opposed to to 'virtual') PC (VMware may work where Virtual PC failed)... I just installed it next to my current Windows 7 installation and had myself a nice dual boot system... Ubuntu is growing on me so I expect I will use it more and more anyhow... and it's installation process is a doddle. I had some trouble configuring my Nvidia drivers (most likely an Nvidia issue rather than Ubuntu's), but I easily fixed this with some Googling, and my three screens sprung into life.
Once in Ubuntu, to install the USB/Serial adapter with my XBee at the end of it I:
My new Tweet-a-Watt was also active and so the XBee receiver / transmitter (referred to as a 'coordinator' as it is intended to manage, or coordinate, the soon to be built sensor network) connected to my PC was receiving telemetry updates every 2 seconds.
This test was particularly easy.
On my Vista machine (where the 'coordinator' was connected via "COM4", I opened-up an IronPython interactive session (by entering 'ipy' at a command prompt). The following commands were then executed:
>>> import clr
>>> clr.AddReference('System')
>>> from System import *
>>> serialPort = IO.Ports.SerialPort("COM4")
>>> serialPort.BaudRate = 9600
>>> serialPort.DataBits = 8
>>> serialPort.Open()
>>> serialPort.ReadLine()
after entering the last command the screen filled with a load of garbage-looking data which represent a number of data packets from the Tweet-a-Watt. I expected this test to be the easier of the two, although doing this through IronPython was still pretty cool. The second test involved repeating the same trick on my Ubuntu system.
Firstly I realized that I could not do this on my Virtual Ubuntu because Microsoft's Virtual PC 2007 does not support Ubuntu and would not make "COM4" available to it :-( So I had to install a full version of Ubuntu on a 'real' (as opposed to to 'virtual') PC (VMware may work where Virtual PC failed)... I just installed it next to my current Windows 7 installation and had myself a nice dual boot system... Ubuntu is growing on me so I expect I will use it more and more anyhow... and it's installation process is a doddle. I had some trouble configuring my Nvidia drivers (most likely an Nvidia issue rather than Ubuntu's), but I easily fixed this with some Googling, and my three screens sprung into life.
Once in Ubuntu, to install the USB/Serial adapter with my XBee at the end of it I:
- unplugged and then re-plugged the USB connector;
- opened a terminal window;
- executed 'dmesg', which prints out the kernel ring buffer(?). This enabled me to confirm that the adapter had been seen;
- executed 'lsusb', which lists all the USB devices. This enabled me to determine the bus number, device number, and device ID of my XBee adapter. It was located at '/dev/ttyUSB0' (needed later);
- executed 'sudo modprobe usbserial vendor=0x0403 product=0x6001' to install the USB/serial adapter driver. I assume that the vendor and product number may be system specific, but are lifted from the appropriate line when the USB devices were listed in step 4.
That's it! Not quite as convenient as the automatic install in Vista, but not particularly painful either.
Now for the test. Once opening up a terminal and launching the IronPython interactive mode ('mono ipy.exe'), the ONLY difference in commands required from what I used above on my Vista machine was the line where the serial port is first instantiated. So,
>>> serialPort = IO.Ports.SerialPort('COM4')
just became
>>> serialPort = IO.Ports.SerialPort('/dev/ttyUSB0')
And after the final 'ReadLine()' command was executed similar looking garbage data was dumped to the screen.
So now we have WinForms and Serial Port access working on Linux (Ubuntu) via IronPython and Mono. Great stuff!
Now if I can only get communication with Amazon's SimpleDB up and running I'll be in business...
Sunday, February 7, 2010
Using Visual Studio to generate a Forms Class
In my last posting I illustrated how I could use System.Windows.Forms in a simple IronPython application that would run on both Windows (Vista) and Linux (Ubuntu). Given that I'm used to building solutions in Visual Studio I was keen to explore if I could continue to do this and then import the Form Class into IronPython. Well I figured it out and this is what I did...
First I created a new project in Visual Studio 2008. Under project types I selected Visual Basic -> Windows -> Class Library. I called my project VSWinFormsTest. Without changing any of the code in the auto-generated Class1.vb file, I added a new form to the project (Project -> Add Windows Form...), and built a simple test form as below:
Once the form was created I built the project and moved the resulting VSWinFormsTest.dll file (in the /bin/Debug/ folder), over to my Ubuntu VPC (I actually just emailed it to my hotmail account and then downloaded via Firefox).
Rather than write and save a script this time, I performed a simple test using IronPython's interactive mode. So after executing:
mono ipy.exe
I entered the following commands directly into the terminal whilst in interactive mode:
>>> import clr>>> clr.AddReference('System.Windows.Forms')
>>> clr.AddReference('VSWinFormsTest')
>>> from System.Windows.Forms import *
>>> from VSWinFormsTest import *
>>> form = VSWinForm()
>>> Application.Run(form)
A few moments after entering the last command the following window popped-up:
Mission accomplished!
And of course now that my form is imported (or subclassed) I can access its various properties in the usual way. For example, if I wanted to change the content of the single line text box I'd just write:
VSWinForm.txtSingleLineTextBox.Text = 'My new text'
As my project progresses I'll need to start adding event handlers to the various form events. My quest to using the best of Python and .NET and have it run on Linux-based system as well as Microsoft Windows systems is coming together... and in theory, the same should hold of Mac OSX!
In the considerable research I have undertaken to get this far it seems clear that if I had set off on this project just six months ago, I would've had a much harder time progressing as far as I have in such a short time. By complete fluke I seem to have set off on this at just the right time as Mono and Iron Python have matured enough for idiots like me to make use of it, and forums are slowly filling up with useful tips and advice. Thanks to all those who have made this possible!
First I created a new project in Visual Studio 2008. Under project types I selected Visual Basic -> Windows -> Class Library. I called my project VSWinFormsTest. Without changing any of the code in the auto-generated Class1.vb file, I added a new form to the project (Project -> Add Windows Form...), and built a simple test form as below:
Once the form was created I built the project and moved the resulting VSWinFormsTest.dll file (in the /bin/Debug/ folder), over to my Ubuntu VPC (I actually just emailed it to my hotmail account and then downloaded via Firefox).
Rather than write and save a script this time, I performed a simple test using IronPython's interactive mode. So after executing:
mono ipy.exe
I entered the following commands directly into the terminal whilst in interactive mode:
>>> import clr>>> clr.AddReference('System.Windows.Forms')
>>> clr.AddReference('VSWinFormsTest')
>>> from System.Windows.Forms import *
>>> from VSWinFormsTest import *
>>> form = VSWinForm()
>>> Application.Run(form)
A few moments after entering the last command the following window popped-up:
Mission accomplished!
And of course now that my form is imported (or subclassed) I can access its various properties in the usual way. For example, if I wanted to change the content of the single line text box I'd just write:
VSWinForm.txtSingleLineTextBox.Text = 'My new text'
As my project progresses I'll need to start adding event handlers to the various form events. My quest to using the best of Python and .NET and have it run on Linux-based system as well as Microsoft Windows systems is coming together... and in theory, the same should hold of Mac OSX!
In the considerable research I have undertaken to get this far it seems clear that if I had set off on this project just six months ago, I would've had a much harder time progressing as far as I have in such a short time. By complete fluke I seem to have set off on this at just the right time as Mono and Iron Python have matured enough for idiots like me to make use of it, and forums are slowly filling up with useful tips and advice. Thanks to all those who have made this possible!
WinForms test
One of my first challenges is to write some IronPython script to create a simple GUI using the WinForms library and get this to run on both my Vista VPC and Ubuntu VPC. After poking around I settled for the simple script below:
import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import Application, Form
class IForm(Form):
def __init__(self):
self.Text = 'Simple'
self.Width = 250
self.Height = 200
self.CenterToScreen()
Application.Run(IForm())
I used the a version of Notepad called the Programmer's Notepad to enter this script and saved it as "WinFormsTest.py".
To run this script in Vista I opened a command window (cmd.exe) and executed:
ipy WinFormsTest.py
(note that I'd already CDed to the directory where WinFormsTest.py had been saved, and I'd also added the folder for IronPython 2.6 to the environmental variable called "path"). On execution the following simple window appeared in the middle of my screen:
So far so good. But what about running the same script on my Ubuntu VPC under mono? I created a text file containing the same script on my Ubuntu VPC and saved it in the same directory as I had unzipped IronPython 2.6. Then, in a terminal window I executed:
mono ipy.exe WinFormsTest.py
import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import Application, Form
class IForm(Form):
def __init__(self):
self.Text = 'Simple'
self.Width = 250
self.Height = 200
self.CenterToScreen()
Application.Run(IForm())
I used the a version of Notepad called the Programmer's Notepad to enter this script and saved it as "WinFormsTest.py".
To run this script in Vista I opened a command window (cmd.exe) and executed:
ipy WinFormsTest.py
(note that I'd already CDed to the directory where WinFormsTest.py had been saved, and I'd also added the folder for IronPython 2.6 to the environmental variable called "path"). On execution the following simple window appeared in the middle of my screen:
So far so good. But what about running the same script on my Ubuntu VPC under mono? I created a text file containing the same script on my Ubuntu VPC and saved it in the same directory as I had unzipped IronPython 2.6. Then, in a terminal window I executed:mono ipy.exe WinFormsTest.py
and low and behold the window below popped up:
Success!
So it seems a pretty straightforward matter to create WinForms with IronPython and run it in both Windows and Linux environments... that is very cool indeed! However, I'm used to building my GUIs in Visual Studio and so I wondered if I could somehow continue to do that, rather than build my forms up control by control at the code level - the topic of my next posting (if I can get it to work that is!).
By the way, if you don't want to use WinForms then GTK# is a perfectly decent alternative.
Mono warning addressed
In the last posting I mentioned that when I ran ipy.exe from mono I got a warning about Microsoft.Dynamic.dll not being found. The full warning was actually:
** (ipy.exe:1774): WARNING **: The following assembly referenced from /home/bertie/Downloads/IronPython-2.6/Microsoft.Dynamic.dll could not be loaded:
Assembly: Sytem.Runtime.Remoting (assemblyref_index=4)
Version: 2.0.0.0
Public Key: b77a5c561934e089
The assembly was not found in the Global Assembly Cache, a path listed in the MONO_PATH environment variable, or in the location of the executing assembly (/hom/bertie/Downloads/IronPython-2.6/).
** (ipy.exe:1774): WARNING **: Could not load file or assembly 'System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
This warning went away after I installed the libmono-system-runtime2.0-cil package via the Synaptic Package Manager. Now executing 'mono ipy.exe' results in no warnings at all.
** (ipy.exe:1774): WARNING **: The following assembly referenced from /home/bertie/Downloads/IronPython-2.6/Microsoft.Dynamic.dll could not be loaded:
Assembly: Sytem.Runtime.Remoting (assemblyref_index=4)
Version: 2.0.0.0
Public Key: b77a5c561934e089
The assembly was not found in the Global Assembly Cache, a path listed in the MONO_PATH environment variable, or in the location of the executing assembly (/hom/bertie/Downloads/IronPython-2.6/).
** (ipy.exe:1774): WARNING **: Could not load file or assembly 'System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
This warning went away after I installed the libmono-system-runtime2.0-cil package via the Synaptic Package Manager. Now executing 'mono ipy.exe' results in no warnings at all.
System setup
Below are the details of my system setup and the software I needed to install before I got going:
System: XPS 720 H2C Quad Core Extreme overclocked to 3.6MHz, 4Gb Ram
OS: Windows 7 64-bit
Software: Visual Studio 2008 Professional, Virtual PC, IronPython 2.6, Python 2.6, Mono 2.6.1.
I setup two virtual computers for Virtual PC:
1. Vista 32-bit - I all too often run into troubles using 64bit windows so decided to use a 32-bit version of Vista. All my Windows development will be done on this virtual machine.
2. Ubuntu 9.10 - I actually installed Ubuntu 9.09 from an image I found on torrentz.com, once installed it automatically upgraded to 9.10. I'd tried setting up a Linux system a couple of times in the past - not a fun experience, as even the various partitions needed by Linux had to be created manually... Life was very different with my Ubuntu install - stuck the disk in and it automatically installed with no trouble whatsoever.
The plan is to take any of the IronPython code I develop on the Vista VPC, and run it under Mono on the Ubuntu VPC.
So on the Ubuntu VPC I used the Synaptic Package Manager (System -> Administration) to upgrade to the latest version of Mono.. surprisingly easy task. Only IronPython 1.1 was available through the Synaptic Package Manager so I went to the IronPython site to download the latest version which was 2.6 at the time of writing.
I downloaded the IronPython-2.6-Bin.zip to /home/bertie/Downloads and then unzipped it with (by first opening a terminal window and CDing to the appropriate directory):
unzip IronPython-2.6-Bin.zip
To check that things were setup correctly I CDed to the IronPython (IPY) directory:
cd IronPython-2.6
and attempted to run ipy.exe under mono:
mono ipy.exe
What I got back was:
IronPython 2.6 (2.6.10920.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
following by a warning about not being able to load Microsoft.Dynamic.dll. Even with the warning this is pretty cool... a complex Windows application running on Linux with the help of Mono. I'll figure out the warning another day.
Some tests I need to run before really getting stuck in are:
1. Perform a WinForms test to make sure my planned GUI will display OK in Ubuntu;
2. Make sure I can access the Serial Port in Ubuntu using the System.IO.Ports namespace in .NET;
3. Make sure I can send some data to Amazon's Simple DB;
4. Demonstrate a simple multi-threading/processing example;
5. Host a simple ASP.NET page on my soon-to-be-delivered PlugPC;
6. Setup a mySQL server for local data storage.
If I can get all these to work then I'll be in pretty good shape to move forward.
System: XPS 720 H2C Quad Core Extreme overclocked to 3.6MHz, 4Gb Ram
OS: Windows 7 64-bit
Software: Visual Studio 2008 Professional, Virtual PC, IronPython 2.6, Python 2.6, Mono 2.6.1.
I setup two virtual computers for Virtual PC:
1. Vista 32-bit - I all too often run into troubles using 64bit windows so decided to use a 32-bit version of Vista. All my Windows development will be done on this virtual machine.
2. Ubuntu 9.10 - I actually installed Ubuntu 9.09 from an image I found on torrentz.com, once installed it automatically upgraded to 9.10. I'd tried setting up a Linux system a couple of times in the past - not a fun experience, as even the various partitions needed by Linux had to be created manually... Life was very different with my Ubuntu install - stuck the disk in and it automatically installed with no trouble whatsoever.
The plan is to take any of the IronPython code I develop on the Vista VPC, and run it under Mono on the Ubuntu VPC.
So on the Ubuntu VPC I used the Synaptic Package Manager (System -> Administration) to upgrade to the latest version of Mono.. surprisingly easy task. Only IronPython 1.1 was available through the Synaptic Package Manager so I went to the IronPython site to download the latest version which was 2.6 at the time of writing.
I downloaded the IronPython-2.6-Bin.zip to /home/bertie/Downloads and then unzipped it with (by first opening a terminal window and CDing to the appropriate directory):
unzip IronPython-2.6-Bin.zip
To check that things were setup correctly I CDed to the IronPython (IPY) directory:
cd IronPython-2.6
and attempted to run ipy.exe under mono:
mono ipy.exe
What I got back was:
IronPython 2.6 (2.6.10920.0) on .NET 2.0.50727.1433
Type "help", "copyright", "credits" or "license" for more information.
following by a warning about not being able to load Microsoft.Dynamic.dll. Even with the warning this is pretty cool... a complex Windows application running on Linux with the help of Mono. I'll figure out the warning another day.
Some tests I need to run before really getting stuck in are:
1. Perform a WinForms test to make sure my planned GUI will display OK in Ubuntu;
2. Make sure I can access the Serial Port in Ubuntu using the System.IO.Ports namespace in .NET;
3. Make sure I can send some data to Amazon's Simple DB;
4. Demonstrate a simple multi-threading/processing example;
5. Host a simple ASP.NET page on my soon-to-be-delivered PlugPC;
6. Setup a mySQL server for local data storage.
If I can get all these to work then I'll be in pretty good shape to move forward.
Saturday, February 6, 2010
Some set-up justifications
So why have I chosen IronPython as my development language? When I started in my quest to build my home SES I was loaned a Digi ConnectPort X4 (by some parties sharing my interests and needing some software) to explore the possibilities of communicating with a variety of sensors around my house. These sensors would monitor things like light levels, temperature, current and voltage loads at various points, movement sensors, etc., none of which have been built yet, but we know they are going to be communicating through XBee Digi Mesh transmitters/receivers. There would also be some simple control like, for example, being able to control whether or not aany particular outlet in my house is powered or not... a kinda of fancy programmable socket, as well as a fully programmable thermostat that would use other sensor data to optimize the HVAC system. Anyway, to cut a long story short the X4 runs on Python and the plan was to develop Python scripts to achieve all our communications/control requirements. So I read a number of books on Python (Python: Visual Quickstart Guide, Python for Dummies, Programming in Python 3, Beginning Python Visualization, Rapid GUI Programming with Python and Qt).
Armed with these books I developed some simple scripts for the X4 to query a test modem and collect a bit of data. I also wrote some simple scripts to dump the data on Amazon's Simple DB, as my longer term vision has my local control system dumping sensor and network configuration data into SimpleDB so that remote control and monitoring software can be developed. Anyway, when trying to load my SimpleDB Python scripts onto the X4 I realized that the X4 has little or no support for the Python Standard Library, and that simple things were going to be rather more complicated on the X4. I haven't the time for such inefficiencies as this is all being done is spare time (and why would you consciously choose a setup that would force you to reinvent the wheel a hundred times over? Most issues that I expect to come up have already been solved by cleverer developers then me, and they have been kind enough to make their libraries available for free!), and decided to dump the X4 in favor of something that runs a full blown version of Python and perhaps a full operating system. I'd gotten to like Python so I decided to stay with it as my main development language.
Then Andy suggested I look at IronPython, i.e., Python that can make use of the .NET libraries. Given that I have done considerable development using .NET this got me excited. Then to further discover that IronPython was compatible with Mono was just too much to bear. I could develop my IronPython SES software on my Windows PC and it would most likely run on Linux (the OS of the PlugPC I plan to use to replace the X4) and Mac OSX... seems to good to be true... so the journey begins. First job: install all the relevant packages and run a few test cases to convince me that this might actually work!
Armed with these books I developed some simple scripts for the X4 to query a test modem and collect a bit of data. I also wrote some simple scripts to dump the data on Amazon's Simple DB, as my longer term vision has my local control system dumping sensor and network configuration data into SimpleDB so that remote control and monitoring software can be developed. Anyway, when trying to load my SimpleDB Python scripts onto the X4 I realized that the X4 has little or no support for the Python Standard Library, and that simple things were going to be rather more complicated on the X4. I haven't the time for such inefficiencies as this is all being done is spare time (and why would you consciously choose a setup that would force you to reinvent the wheel a hundred times over? Most issues that I expect to come up have already been solved by cleverer developers then me, and they have been kind enough to make their libraries available for free!), and decided to dump the X4 in favor of something that runs a full blown version of Python and perhaps a full operating system. I'd gotten to like Python so I decided to stay with it as my main development language.
Then Andy suggested I look at IronPython, i.e., Python that can make use of the .NET libraries. Given that I have done considerable development using .NET this got me excited. Then to further discover that IronPython was compatible with Mono was just too much to bear. I could develop my IronPython SES software on my Windows PC and it would most likely run on Linux (the OS of the PlugPC I plan to use to replace the X4) and Mac OSX... seems to good to be true... so the journey begins. First job: install all the relevant packages and run a few test cases to convince me that this might actually work!
My first blog post
This isn't really a blog for the masses - it is a diary for my own purposes; it is a record of my efforts to develop a Smart Energy System (SES) for my home. It will include notes on the hardware development, focused mainly on the use of XBee transmitters and receivers, as well as notes on my software development, focused mainly on my efforts to write a Mono-compatible IronPython control system for a PlugPC. Given that I've never built any hardware before, and have no experience with Linux, IronPython or Mono (I hadn't heard of the latter two before last week - thanks Andy for the tips), then perhaps this diary may contain some value of others that want to delve into these areas. I won't include much in-depth detail... all I will include is what I did to get things working with little extra discussion - I could care less for the differences between CLR or DLR, all I want is a working system with minimal effort.
Subscribe to:
Posts (Atom)