My development PlugPC (SheevaPlug) arrived last week, and going back over previous postings I attempted to follow my own guidelines and setup my development environment (boot from USB drive, setup Mono, IronPython etc.). The first difference between the TonidoPlug and the SheevaPlug is that the SheevaPlug really is a developers tool and so you have to do some more basic things that you don't need to do on the TonidoPlug.
My first job was to get the SheevaPlug (SP) to boot from an external drive (as it only has 500Mb onboard). Actually, this time I tried to set it up to boot from an SDHC card, which the TonidoPlug has no built-in support for. To cut a long story short, the installation went smoothly, but then the SP died a few days later - removing the SDHC card did not cause the SP to boot from the internal drive (because I did not know how to reconfigure UBoot - see below - back to the correct configuration) - it was officially 'bricked'. The SDHC did seem to get very hot so maybe it literally got fried. But I also remembered that SDHC cards only have a limited number of rewrite cycles, something like 1000000. This seems like a huge number, but maybe given how active your average OS is, SDHC just isn't a good choice for running an OS from... or maybe I just had a faulty card. Anyway, OfficeMax gave me my money back and I took advantage of a sale they were having in USB storage. For the price of the 16GB SanDisk SDHC Card I got 48Gb (16, 16, 8, 4) worth of Toshiba Transmemory USB flash storage - a great deal (I thought). However, after a 4-5 attempts at building an OS on the Toshiba flash storage I still could not get the SP to boot from it... from the lack of LED activity on the Toshiba I guess it was something to do with power management... but whatever it was, it didn't work. My last attempt was with a SanDisk Cruzer 16Gb USB Flash drive; this is what I use without trouble on my TonidoPlug. As with my TonidoPlug, the process to install Ubuntu Jaunty onto the Cruzer drive went flawlessly. So, save yourself the trouble an get one of these right off the get go.
The SP comes with a serial port (via a micro-USB connection), and so initial configuration is done through this port. Again I used PuTTY, but selected the Serial Port option rather than SSH which I've been using to talk to my TonidoPlug. Once you have your SP connected to your desktop, launch a PuTTY serial connection (the default port for my SP was 'COM4', but yours might be different. If you're using Windows just open your Device Manager (under Control Panel -> System) and check to see was Ports are available - your SP should be connected via a USB Serial Port), and start pressing the space bar. You have to get your timing right, as what your trying to do is interrupt the SP boot process with the keyboard presses. If you're successful you'll see 'Marvell>' at the prompt. As an excercise, enter 'reset' and then you get to watch the whole boot process in your PuTTY serial window - again press the spacebar at the appropraite moment to stop the SP booting into Ubuntu.
The boot process is directed by UBoot, so one of your first chores is to make sure you have the latest version of UBoot installed. The instructions here, worked perfectly for me. I went the USB route rather than the TFTP one, and got my Uboot upgraded to version 3.4.27.
The next stage was to build a new OS on my Cruzer drive and configure UBoot to boot from it. The instructions here, again, worked fine for me... although they failed a number of times when I was attempting to do the same thing with the SDHC ccard or the Toshiba flash drive.
So unbricking my bricked SP was not too much trouble as long as I used a compatible USB flash drive.
In all my messing around I also realized that I did not need to go through the process of building Mono and IronPythong from scratch. So, once I had my SP working again, I connected to it via SSH and executed the following commands:
>>apt-get update (this updated the repositories)
>>apt-get dist-upgrade (to make sure my OS was up-to-date)
>>apt-get install ironpython
This last stage installs Mono version 1.9.1 and IronPython 1.1.1. These are earlier version than those I was using on my TonidoPlug, but I'm hoping they're good enough to be going on with. The advantage is that I can rely on apt-get to update them whenever updates are available in the repositories, rather than mess around maintaining them myself (unless I find I have problems and am forced to - but let's not do extra work unless we have to). The other advantage is that I can run IronPython with 'ipy' from any directory rather than 'mono ipy.exe' from the IronPython-2.6 directory like before.
The first thing I noticed is that the IronPython 1.1.1 interactive mode is rather fickle (don't go using the up and down cursors thinking you can scroll through past entries - it doesn't work, and just corrupts the current line, so even if you then enter a legit command it fails), but you get used to it... just be very precise with your typing!
Anyway, I now have a SheevaPlug set up and ready to go. The previous tests regarding Serial Port access, Amazon SimpleDB access, etc., all worked fine, ALTHOUGH you must use a POWERED USB hub to connect the XBee coordinator and the USB drive hosting the OS.
We're ready to move forward once again... keep on pluggin'
Wednesday, March 10, 2010
Saturday, February 20, 2010
My Vision of a Home Smart Energy System
As I'm almost ready to start building the libraries and code to talk to various devices under development I thought this would be a good time to describe what I am hoping to build.
When I first embarked on this project my main focus was to save energy. My first project (after the Tweet-a-Watt) was to design and build a smart outlet/socket that would transmit current load and voltage data to a central 'coordinator' via a ZigBee wireless network and also allow me (through the coordinator) to switch the load on and off - a fancy programmable outlet really. I'm still working on this project and will report progress in future blogs. The initial aim of the smart outlet was to enable me to save money by switching loads off when not in use, as even in standby mode there can still be a noticeable energy draw to the device. However, it was pointed out to me that the ability to do this would save no more than perhaps a few $ per year - hardly worth the investment in time and materials. The area of major savings is in areas such as heating, air-conditioning, pool pump, etc. In these areas 100s, perhaps 1000s, of $ can potentially be saved annually through clever energy usage. This data point seemed to undermine the whole idea of a smart socket being used as part of an energy saving system. Why spend $20 on a device that will save you $1 per year?
It was something my wife said that made me realize that the focus on energy savings was too narrow. Being a Brit my wife (and therefore myself) is a Tea addict, and the only point at which she took interest in my Smart Energy System (SES) project was when she realized she might be able to program the kettle to switch on from her Blackberry shortly before getting up in the morning. Of course, attaching the kettle to a timer socket would nearly achieve the same thing as long as you got up at the same time every day! However, it made me realize that 'control' was the correct paradigm for the SES, NOT 'energy saving'. Of course you need some level of control to achieve energy savings, but the control paradigm is much broader than the energy saving one - it includes, for example, being able to switch the kettle on from your phone, which does not save energy at all, but offers some level of convenience (assuming you remembered to fill the kettle the night before!). This 'boiling the kettle' example may actually use more energy as I'm sure they'll be times when you reach over to your phone in the morning, switch the kettle on, and then fall back to sleep, and so the water might get boiled several times before you actually drag yourself out of the sack!
So the 'control' paradigm is my new focus. I want to be able to enable and disable any socket / outlet in my house from a central computer, or from a remote device like my iPhone (I'd like the ability to switch lighting on and off when I'm away from home - even better, I'd like a 'pretend-someone-is-at-home' autonomous mode!). I want to be able to collect data, such as current load, voltage, temperature, light levels, movement detection (which can be used for security as well as a useful input for developing energy-efficient configurations), etc. The method of communication between all the 'nodes' on this network will be via a ZigBee mesh network. I want to be able to view the status of my device network at any point locally and remotely. With all this data I will then be able to develop algorithms to, for example, minimize my heating and cooling needs, and so I want to be able to implement programmable energy schedules for any node on the network - so the network has to have a certain level of autonomy.
With such a network I will have the convenience of control, and be able to realize the benefits of energy saving configurations. So my wife will be happy as she'll be able to start the kettle boiling as she leaves the kids' school for home, and I will be happy because our bank account balance should look healthier (and I'd've built something pretty cool!).
So that's the challenge - stay tuned for progress reports... and if you have any thoughts on functionality you would like to see with such a network then feel free to comment. One thought I had was to use the control software and data set to develop algorithms that would aim to save enough money in a year to pay for a weekend away... once you have the control a whole new world of applications opens up...
When I first embarked on this project my main focus was to save energy. My first project (after the Tweet-a-Watt) was to design and build a smart outlet/socket that would transmit current load and voltage data to a central 'coordinator' via a ZigBee wireless network and also allow me (through the coordinator) to switch the load on and off - a fancy programmable outlet really. I'm still working on this project and will report progress in future blogs. The initial aim of the smart outlet was to enable me to save money by switching loads off when not in use, as even in standby mode there can still be a noticeable energy draw to the device. However, it was pointed out to me that the ability to do this would save no more than perhaps a few $ per year - hardly worth the investment in time and materials. The area of major savings is in areas such as heating, air-conditioning, pool pump, etc. In these areas 100s, perhaps 1000s, of $ can potentially be saved annually through clever energy usage. This data point seemed to undermine the whole idea of a smart socket being used as part of an energy saving system. Why spend $20 on a device that will save you $1 per year?
It was something my wife said that made me realize that the focus on energy savings was too narrow. Being a Brit my wife (and therefore myself) is a Tea addict, and the only point at which she took interest in my Smart Energy System (SES) project was when she realized she might be able to program the kettle to switch on from her Blackberry shortly before getting up in the morning. Of course, attaching the kettle to a timer socket would nearly achieve the same thing as long as you got up at the same time every day! However, it made me realize that 'control' was the correct paradigm for the SES, NOT 'energy saving'. Of course you need some level of control to achieve energy savings, but the control paradigm is much broader than the energy saving one - it includes, for example, being able to switch the kettle on from your phone, which does not save energy at all, but offers some level of convenience (assuming you remembered to fill the kettle the night before!). This 'boiling the kettle' example may actually use more energy as I'm sure they'll be times when you reach over to your phone in the morning, switch the kettle on, and then fall back to sleep, and so the water might get boiled several times before you actually drag yourself out of the sack!
So the 'control' paradigm is my new focus. I want to be able to enable and disable any socket / outlet in my house from a central computer, or from a remote device like my iPhone (I'd like the ability to switch lighting on and off when I'm away from home - even better, I'd like a 'pretend-someone-is-at-home' autonomous mode!). I want to be able to collect data, such as current load, voltage, temperature, light levels, movement detection (which can be used for security as well as a useful input for developing energy-efficient configurations), etc. The method of communication between all the 'nodes' on this network will be via a ZigBee mesh network. I want to be able to view the status of my device network at any point locally and remotely. With all this data I will then be able to develop algorithms to, for example, minimize my heating and cooling needs, and so I want to be able to implement programmable energy schedules for any node on the network - so the network has to have a certain level of autonomy.
With such a network I will have the convenience of control, and be able to realize the benefits of energy saving configurations. So my wife will be happy as she'll be able to start the kettle boiling as she leaves the kids' school for home, and I will be happy because our bank account balance should look healthier (and I'd've built something pretty cool!).
So that's the challenge - stay tuned for progress reports... and if you have any thoughts on functionality you would like to see with such a network then feel free to comment. One thought I had was to use the control software and data set to develop algorithms that would aim to save enough money in a year to pay for a weekend away... once you have the control a whole new world of applications opens up...
Installing MySQL on the TonidoPlug
An important element of the Smart Energy System that I am building is a local and remote database to store telemetry packets, processed data, and control / status strings. For remote storage I plan to use Amazon's SimpleDB service (which I have tested previously on the PlugPC), and for local storage I plan to use a MySQL database structure. So my next job is to install MySQL on my Plug.
I first tried installing MySQL 5.0, but this was a complete disaster and the installation kept finding it's way into a infinite error loop... this was removed tout de suite. The problems I saw during the installation of 5.0 might be why version 5.1 emerged! Anyway, I tried again with MySQL 5.1. After SSHing in to my Plug, to install 5.1 I executed:
apt-get install mysql-client-5.1 mysql-server-5.1 (these just happened to be the versions available on the repository I am using - as of writing version 5.5 is available)
The installation process ran without incident, but for some reason no password was created during installation and I could not gain access to the MySQL interactive mode. The following sequence of commands fixed this particular problem.
First, try to stop the MySQL daemon:
/etc/init.d/mysql stop
Followed by:
mysqld_safe --skip-grant-tables & (which is the recommended way to start MySQL)
mysql -u root -p (to enter as 'root' - you will be prompted for your system 'root' password. If successful you will now be in MySQL interactive mode)
mysql client - mysql - root
use mysql
update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root' (should probably choose a MySQL 'root' password that is different from your system 'root' password)
Now, we're ready to start creating tables and fields, and storing and quering data... I'll save this treat for another day when I have had time to decide what table and fields I need for my SES.
To launch MySQL from the command line:
mysql -u root -p (you will then be prompted for your MySQL 'root' password)
The MySQL documentation for getting started is available if you want to start playing right away.
I first tried installing MySQL 5.0, but this was a complete disaster and the installation kept finding it's way into a infinite error loop... this was removed tout de suite. The problems I saw during the installation of 5.0 might be why version 5.1 emerged! Anyway, I tried again with MySQL 5.1. After SSHing in to my Plug, to install 5.1 I executed:
apt-get install mysql-client-5.1 mysql-server-5.1 (these just happened to be the versions available on the repository I am using - as of writing version 5.5 is available)
The installation process ran without incident, but for some reason no password was created during installation and I could not gain access to the MySQL interactive mode. The following sequence of commands fixed this particular problem.
First, try to stop the MySQL daemon:
/etc/init.d/mysql stop
Followed by:
mysqld_safe --skip-grant-tables & (which is the recommended way to start MySQL)
mysql -u root -p (to enter as 'root' - you will be prompted for your system 'root' password. If successful you will now be in MySQL interactive mode)
mysql client - mysql - root
use mysql
update user set password=PASSWORD("NEW-ROOT-PASSWORD") where User='root' (should probably choose a MySQL 'root' password that is different from your system 'root' password)
Now, we're ready to start creating tables and fields, and storing and quering data... I'll save this treat for another day when I have had time to decide what table and fields I need for my SES.
To launch MySQL from the command line:
mysql -u root -p (you will then be prompted for your MySQL 'root' password)
The MySQL documentation for getting started is available if you want to start playing right away.
Sunday, February 14, 2010
My new PlugPC and Mono
Earlier in the week I received my Tonido PlugPC, and I have immediately fallen in love! What a fantastic little device. Within a few hours I'd got it set up to the point of being able to replace the Media Center PC that I was using for all our media streaming, central storage, and email server. It runs a full Linux kernel and so there is little you can't do with it that you can't do with a desktop... and it run off 9Watts instead of 400! Mine's running a little 'hot' as I have a USB hub plugged in to it. Off the hub I have a 16Gb USB drive which I'm booting the Tonido off of, and 3 x 0.5Tb and 1 x 1Tb drives for storage. I won't bore you with all the set-up details for my Tonido as it is not particularly relevant to this blog. My plan is to use a SheevaPlug (the TonidoPlug is really just a SheevaPlug with some web applications running on it) for development, but for some reason there is a 2-3 week delay with my SheevaPlug order, so I'm doing some playing with my TonidoPlug in the meantime. A few of pickies of my setup:
The most signifiant difference between a PlugPC and a desktop is that you have no screen or mouse or keyboard to interact directly with the device. So once you have your PlugPC connected to an ethernet network, the first thing you have to do in find a route in. I chose SSH, and in particular PuTTY. I haven't had much experience with command line work, but it really isn't that hard. Run PuTTY, give it the IP of your PlugPC, select SSH mode (I used Telnet mode to test my email server when I got it installed), ignore the security warning, login as 'root', and you're in.
Given that the SheevaPlug currently only has a 500Mb flash drive, the chances are that you're going to want more drive space for development. I very much doubt we'll have to wait very long before 16+Gb is available onboard. I followed the directions provided here, to boot from a 16Gb USB drive I got on sale from Staples for $29.99. The directions worked to the letter and I had no trouble whatsoever... very straightforward. My Tonido now boots off the USB drive - and certainly much faster than my Windows PC boots off a hard drive... not bad considering my Windows PC is a water-cooled overclocked (3.6GHz) Quadcore, and the Tonido has a single 1.2GHz core!
My next step was to make sure the packages on my Plug were uptodate. To do this I attempted to run:
apt-get upgrade
This bombed out for reasons I can't quite remember, but was someting to do with not being able to write to a particular location. After some Googling the following fixed things:
mkdir -p /var/cache/apt/archives/partial
apt-get autoclean
This seemed to fix 'apt-get' and re-executing 'apt-get upgrade' worked just fine.
At this point I'm keen to start going through the various tests outlined in previous posts (Amazon SimplDB, SerialPort, obviously the WinForms test is pointless on the Plug!), so I need Mono and IronPython installed. First I tried to install Mono with:
apt-get install mono-2.0-devel
The installation seemed to work, but even the simplest 'Hello World' test failed with a 'Segmentation Fault' response. I messed around some more, but kept running into the same error. Eventually I decided to try and compile Mono from source and install it that way. In doing this I found the article here very useful. There are some packages that the Mono configuration process depends on and need to be available before you attempt your Mono compilation. The following 'apt-get' installed what I needed to move forward:
apt-get install bison gcc gettext pkg-config glib-2.0
Next I downloaded the Mono 2.6.1 source to my root folder, and unzipped it:
cd /
wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.6.1.tar.bz2
bzip2 -cd mono-2.6.1.tar.bz2 | tar xvf -
(NB Edit 11/24/2012: Mono source is now at http://download.mono-project.com/sources/mono/ with the latest version today being 2.10.9). This created a 'mono-2.6.1' folder that I CDed into ('cd mono-2.6.1'). The compilation and install was remarkably smooth, if not time-consuming:
./configure --prefix=/usr (if this step fails it is probably due to a missing package. Install the indicated package and try again)
make (this process took several hours)
make install
After the install process completed I executed 'mono -V' to check the currently installed version and received the following:
Mono JIT compiler version 2.6.1 (tarball Thu Feb 11 17:56:14 MST 2010)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www-mono-project.com
TLS: __thread
GC: Included Boehm (with typed GC and Parallel Mark)
SIGSEGV: normal
Notifications: epoll
Architecture: armel, soft-float
Disabled: none
So far, so good. I then installed IronPython 2.6 as detailed in my previous posting, and performed the same Amazon SimpleDB access that I had previously (including the 'mozroots' command). So a far more challenging test than a simple 'Hello World' - the test ran as expected and so I am now hoping that I can claim that Mono 2.6.1 and IronPython 2.6 are installed and running OK on my PlugPC... how cool is that!
The most signifiant difference between a PlugPC and a desktop is that you have no screen or mouse or keyboard to interact directly with the device. So once you have your PlugPC connected to an ethernet network, the first thing you have to do in find a route in. I chose SSH, and in particular PuTTY. I haven't had much experience with command line work, but it really isn't that hard. Run PuTTY, give it the IP of your PlugPC, select SSH mode (I used Telnet mode to test my email server when I got it installed), ignore the security warning, login as 'root', and you're in.Given that the SheevaPlug currently only has a 500Mb flash drive, the chances are that you're going to want more drive space for development. I very much doubt we'll have to wait very long before 16+Gb is available onboard. I followed the directions provided here, to boot from a 16Gb USB drive I got on sale from Staples for $29.99. The directions worked to the letter and I had no trouble whatsoever... very straightforward. My Tonido now boots off the USB drive - and certainly much faster than my Windows PC boots off a hard drive... not bad considering my Windows PC is a water-cooled overclocked (3.6GHz) Quadcore, and the Tonido has a single 1.2GHz core!
My next step was to make sure the packages on my Plug were uptodate. To do this I attempted to run:
apt-get upgrade
This bombed out for reasons I can't quite remember, but was someting to do with not being able to write to a particular location. After some Googling the following fixed things:
mkdir -p /var/cache/apt/archives/partial
apt-get autoclean
This seemed to fix 'apt-get' and re-executing 'apt-get upgrade' worked just fine.
At this point I'm keen to start going through the various tests outlined in previous posts (Amazon SimplDB, SerialPort, obviously the WinForms test is pointless on the Plug!), so I need Mono and IronPython installed. First I tried to install Mono with:
apt-get install mono-2.0-devel
The installation seemed to work, but even the simplest 'Hello World' test failed with a 'Segmentation Fault' response. I messed around some more, but kept running into the same error. Eventually I decided to try and compile Mono from source and install it that way. In doing this I found the article here very useful. There are some packages that the Mono configuration process depends on and need to be available before you attempt your Mono compilation. The following 'apt-get' installed what I needed to move forward:
apt-get install bison gcc gettext pkg-config glib-2.0
Next I downloaded the Mono 2.6.1 source to my root folder, and unzipped it:
cd /
wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.6.1.tar.bz2
bzip2 -cd mono-2.6.1.tar.bz2 | tar xvf -
(NB Edit 11/24/2012: Mono source is now at http://download.mono-project.com/sources/mono/ with the latest version today being 2.10.9). This created a 'mono-2.6.1' folder that I CDed into ('cd mono-2.6.1'). The compilation and install was remarkably smooth, if not time-consuming:
./configure --prefix=/usr (if this step fails it is probably due to a missing package. Install the indicated package and try again)
make (this process took several hours)
make install
After the install process completed I executed 'mono -V' to check the currently installed version and received the following:
Mono JIT compiler version 2.6.1 (tarball Thu Feb 11 17:56:14 MST 2010)
Copyright (C) 2002-2008 Novell, Inc and Contributors. www-mono-project.com
TLS: __thread
GC: Included Boehm (with typed GC and Parallel Mark)
SIGSEGV: normal
Notifications: epoll
Architecture: armel, soft-float
Disabled: none
So far, so good. I then installed IronPython 2.6 as detailed in my previous posting, and performed the same Amazon SimpleDB access that I had previously (including the 'mozroots' command). So a far more challenging test than a simple 'Hello World' - the test ran as expected and so I am now hoping that I can claim that Mono 2.6.1 and IronPython 2.6 are installed and running OK on my PlugPC... how cool is that!
Thursday, February 11, 2010
Accessing Amazon's SimpleDB
The next test in my 'proof of concept' phase of development is to see if I can access Amazon's SimpleDB cloud service. The reason for picking Amazon's SimpleDB is threefold:
The first step is to get hold of a library to do all the work for me. For no particular reason I went for the C# library I found here. The version I downloaded was released on May 20, 2009. I also found a page in the IronPython Cookbook that talks about using an older version of this library in IronPython. The steps for creating the DLL are still valid, but the test scripts themselves are out of date so not much use to anyone (other than to waste some time wondering why one's scrips keep failing!). I have posted the version of the DLL I compiled here.
Boto is another more comprehensive library written in Python that is for all the Amazon cloud services, not just SimpleDB.
Next, you've got to get yourself a SimpleDB account from here. Once you have registered you'll be given a unique AccessKey ID, and a Secret Access Key, both of which are needed to access SimpleDB service.
One more step require: I had performed a few tests in IronPython that worked just fine and then failed when running under Mono. After some googling I found that the issue was that Mono doesn't trust anyone, and so any SSL web accesses fail by default... so of course my Mono IronPython SimpleDB scripts failed! To save you the hassle, all you need to do is install a pile of root certificates. This is done by executing the following at the command line:
mozroots --import --ask-remove --machine
More details about Mozroots can be found here. If you have the latest version of Mono then Mozroots should already be installed.
OK, now we're ready to go. I'd already added some data to SimpleDB and so we'll just look at a simple query in this posting. To run this for yourselves you're going to have to setup you AWS account with Amazon, and stick some data in SimpleDB. I'll add some data entry snippets another time.
The following code is for a simple SimpleDB test that I saved as simpleDB_examples.py:
import clr
clr.AddReference('Amazon.SimpleDB')
from Amazon.SimpleDB import AmazonSimpleDBClient
from Amazon.SimpleDB.Model import ListDomainsRequest, Select Request
service = AmazonSimpleDBClient('[Access Key ID]','[Secret AccessKey]')
# list domains
listDomainsRequest = ListDomainsRequest()
listDomainsRequest.MaxNumberOfDomains = 10
listDomainsResponse = service.ListDomains(listDomainsRequest)
print listDomainsResponse.ListDomainsResult.DomainName
# select items request
selectRequest = SelectRequest()
selectRequest.SelectExpression = 'Select * From [domain]'
selectResponse = service.Select(selectRequest)
# print first attribute namd and value for first returned item
print selectResponse.SelectResult.Item[0].Attribute[0].Name
print selectResponse.SelectResult.Item[0].Attribute[0].Value
Once saved, the test script can be run by executing:
mono ipy.exe simpleDB_examples.py
As I had already created a couple of domains (equivalent to 'tables'), the following was printed to screen on my Ubuntu machine:
List[str]([dmg_data', 'testDomain'])
node_type
end node
Another success! Now we have WinForms, SerialPort access and SimpleDB partially proven on Ubuntu with Mono and IronPython. With a Multi-Threading/Processing example I think I'm nearly ready to get stuck into the main application.
Side note: My first PlugPC (TonidoPlug) was delivered this week, so plenty of fun ahead!
- IF (big 'if' I know) my efforts to develop a Home Smart Energy System turn out to be a resounded (financial) success, I don't want to have to put together all the hardware needed for the huge database that is going to be needed to store all the telemetry related to time-dependent energy usage and sensor configuration / control for potentially thousands (and more if I'm real lucky!) of homes. Cloud services offer a scaleable approach that gets billed in proportion to one's usage;
- The Amazon SimpleDB has a particularly simple API;
- And the Amazon cloud was the first of the big three (Amazon, Google, Microsoft) I believe so I'll go with the one that has years of postings to support forums to help with my own development.
The first step is to get hold of a library to do all the work for me. For no particular reason I went for the C# library I found here. The version I downloaded was released on May 20, 2009. I also found a page in the IronPython Cookbook that talks about using an older version of this library in IronPython. The steps for creating the DLL are still valid, but the test scripts themselves are out of date so not much use to anyone (other than to waste some time wondering why one's scrips keep failing!). I have posted the version of the DLL I compiled here.
Boto is another more comprehensive library written in Python that is for all the Amazon cloud services, not just SimpleDB.
Next, you've got to get yourself a SimpleDB account from here. Once you have registered you'll be given a unique AccessKey ID, and a Secret Access Key, both of which are needed to access SimpleDB service.
One more step require: I had performed a few tests in IronPython that worked just fine and then failed when running under Mono. After some googling I found that the issue was that Mono doesn't trust anyone, and so any SSL web accesses fail by default... so of course my Mono IronPython SimpleDB scripts failed! To save you the hassle, all you need to do is install a pile of root certificates. This is done by executing the following at the command line:
mozroots --import --ask-remove --machine
More details about Mozroots can be found here. If you have the latest version of Mono then Mozroots should already be installed.
OK, now we're ready to go. I'd already added some data to SimpleDB and so we'll just look at a simple query in this posting. To run this for yourselves you're going to have to setup you AWS account with Amazon, and stick some data in SimpleDB. I'll add some data entry snippets another time.
The following code is for a simple SimpleDB test that I saved as simpleDB_examples.py:
import clr
clr.AddReference('Amazon.SimpleDB')
from Amazon.SimpleDB import AmazonSimpleDBClient
from Amazon.SimpleDB.Model import ListDomainsRequest, Select Request
service = AmazonSimpleDBClient('[Access Key ID]','[Secret AccessKey]')
# list domains
listDomainsRequest = ListDomainsRequest()
listDomainsRequest.MaxNumberOfDomains = 10
listDomainsResponse = service.ListDomains(listDomainsRequest)
print listDomainsResponse.ListDomainsResult.DomainName
# select items request
selectRequest = SelectRequest()
selectRequest.SelectExpression = 'Select * From [domain]'
selectResponse = service.Select(selectRequest)
# print first attribute namd and value for first returned item
print selectResponse.SelectResult.Item[0].Attribute[0].Name
print selectResponse.SelectResult.Item[0].Attribute[0].Value
Once saved, the test script can be run by executing:
mono ipy.exe simpleDB_examples.py
As I had already created a couple of domains (equivalent to 'tables'), the following was printed to screen on my Ubuntu machine:
List[str]([dmg_data', 'testDomain'])
node_type
end node
Another success! Now we have WinForms, SerialPort access and SimpleDB partially proven on Ubuntu with Mono and IronPython. With a Multi-Threading/Processing example I think I'm nearly ready to get stuck into the main application.
Side note: My first PlugPC (TonidoPlug) was delivered this week, so plenty of fun ahead!
Monday, February 8, 2010
Connecting to the Serial Port
So I can now build a WinForms based application and have that form appear pretty much the same in Windows (Vista) and Linux (Ubuntu) without making any code changes whatsoever, thanks to Mono. Another 'feature' I need to test run is the ability to access a serial port to take data from the XBee receiver/transmitter that will be the hub of my Smart Energy System.
As I have never undertaken hardware development before (other than in FPGAs - which is pretty much the same a writing software), I needed to get some hand-ons experience with building some actual hardware. I chose to build a Tweet-a-Watt. This is the perfect project for the wannabee home smart grid engineer. It gives us experience soldering and building-up a real circuit board, as well as playing around with the XBee receivers/transmitters - a new(ish) product especially designed for sensor networks. Given that I had no circuit building equipment I also had a fun couple of days at Fry's Electronics putting together my new home electronics lab!
When I built the Tweet-a-Watt I also wrote a simple terminal program to communicate through the USB port to the XBee via a serial adapter. The .NET System.IO.Ports.SerialPort library provides everything needed for such a project. I was pleasantly surprised as to how easy is was to build my simple terminal application... we'll see some of the commands used later on.
For this particular test case I used the XBee adapter that is built as part of the Tweet-a-Watt project. This plugs directly into any USB port. Here's a photo of mine, with my thumb for scale:
As I have never undertaken hardware development before (other than in FPGAs - which is pretty much the same a writing software), I needed to get some hand-ons experience with building some actual hardware. I chose to build a Tweet-a-Watt. This is the perfect project for the wannabee home smart grid engineer. It gives us experience soldering and building-up a real circuit board, as well as playing around with the XBee receivers/transmitters - a new(ish) product especially designed for sensor networks. Given that I had no circuit building equipment I also had a fun couple of days at Fry's Electronics putting together my new home electronics lab!
When I built the Tweet-a-Watt I also wrote a simple terminal program to communicate through the USB port to the XBee via a serial adapter. The .NET System.IO.Ports.SerialPort library provides everything needed for such a project. I was pleasantly surprised as to how easy is was to build my simple terminal application... we'll see some of the commands used later on.
For this particular test case I used the XBee adapter that is built as part of the Tweet-a-Watt project. This plugs directly into any USB port. Here's a photo of mine, with my thumb for scale:
On Windows Vista the drivers were installed automatically and it was also recognized as a serial port communications link... couldn't be easier. The device was invariably to be found at "COM4" on my Vista machine.
My new Tweet-a-Watt was also active and so the XBee receiver / transmitter (referred to as a 'coordinator' as it is intended to manage, or coordinate, the soon to be built sensor network) connected to my PC was receiving telemetry updates every 2 seconds.
This test was particularly easy.
On my Vista machine (where the 'coordinator' was connected via "COM4", I opened-up an IronPython interactive session (by entering 'ipy' at a command prompt). The following commands were then executed:
>>> import clr
>>> clr.AddReference('System')
>>> from System import *
>>> serialPort = IO.Ports.SerialPort("COM4")
>>> serialPort.BaudRate = 9600
>>> serialPort.DataBits = 8
>>> serialPort.Open()
>>> serialPort.ReadLine()
after entering the last command the screen filled with a load of garbage-looking data which represent a number of data packets from the Tweet-a-Watt. I expected this test to be the easier of the two, although doing this through IronPython was still pretty cool. The second test involved repeating the same trick on my Ubuntu system.
Firstly I realized that I could not do this on my Virtual Ubuntu because Microsoft's Virtual PC 2007 does not support Ubuntu and would not make "COM4" available to it :-( So I had to install a full version of Ubuntu on a 'real' (as opposed to to 'virtual') PC (VMware may work where Virtual PC failed)... I just installed it next to my current Windows 7 installation and had myself a nice dual boot system... Ubuntu is growing on me so I expect I will use it more and more anyhow... and it's installation process is a doddle. I had some trouble configuring my Nvidia drivers (most likely an Nvidia issue rather than Ubuntu's), but I easily fixed this with some Googling, and my three screens sprung into life.
Once in Ubuntu, to install the USB/Serial adapter with my XBee at the end of it I:
My new Tweet-a-Watt was also active and so the XBee receiver / transmitter (referred to as a 'coordinator' as it is intended to manage, or coordinate, the soon to be built sensor network) connected to my PC was receiving telemetry updates every 2 seconds.
This test was particularly easy.
On my Vista machine (where the 'coordinator' was connected via "COM4", I opened-up an IronPython interactive session (by entering 'ipy' at a command prompt). The following commands were then executed:
>>> import clr
>>> clr.AddReference('System')
>>> from System import *
>>> serialPort = IO.Ports.SerialPort("COM4")
>>> serialPort.BaudRate = 9600
>>> serialPort.DataBits = 8
>>> serialPort.Open()
>>> serialPort.ReadLine()
after entering the last command the screen filled with a load of garbage-looking data which represent a number of data packets from the Tweet-a-Watt. I expected this test to be the easier of the two, although doing this through IronPython was still pretty cool. The second test involved repeating the same trick on my Ubuntu system.
Firstly I realized that I could not do this on my Virtual Ubuntu because Microsoft's Virtual PC 2007 does not support Ubuntu and would not make "COM4" available to it :-( So I had to install a full version of Ubuntu on a 'real' (as opposed to to 'virtual') PC (VMware may work where Virtual PC failed)... I just installed it next to my current Windows 7 installation and had myself a nice dual boot system... Ubuntu is growing on me so I expect I will use it more and more anyhow... and it's installation process is a doddle. I had some trouble configuring my Nvidia drivers (most likely an Nvidia issue rather than Ubuntu's), but I easily fixed this with some Googling, and my three screens sprung into life.
Once in Ubuntu, to install the USB/Serial adapter with my XBee at the end of it I:
- unplugged and then re-plugged the USB connector;
- opened a terminal window;
- executed 'dmesg', which prints out the kernel ring buffer(?). This enabled me to confirm that the adapter had been seen;
- executed 'lsusb', which lists all the USB devices. This enabled me to determine the bus number, device number, and device ID of my XBee adapter. It was located at '/dev/ttyUSB0' (needed later);
- executed 'sudo modprobe usbserial vendor=0x0403 product=0x6001' to install the USB/serial adapter driver. I assume that the vendor and product number may be system specific, but are lifted from the appropriate line when the USB devices were listed in step 4.
That's it! Not quite as convenient as the automatic install in Vista, but not particularly painful either.
Now for the test. Once opening up a terminal and launching the IronPython interactive mode ('mono ipy.exe'), the ONLY difference in commands required from what I used above on my Vista machine was the line where the serial port is first instantiated. So,
>>> serialPort = IO.Ports.SerialPort('COM4')
just became
>>> serialPort = IO.Ports.SerialPort('/dev/ttyUSB0')
And after the final 'ReadLine()' command was executed similar looking garbage data was dumped to the screen.
So now we have WinForms and Serial Port access working on Linux (Ubuntu) via IronPython and Mono. Great stuff!
Now if I can only get communication with Amazon's SimpleDB up and running I'll be in business...
Sunday, February 7, 2010
Using Visual Studio to generate a Forms Class
In my last posting I illustrated how I could use System.Windows.Forms in a simple IronPython application that would run on both Windows (Vista) and Linux (Ubuntu). Given that I'm used to building solutions in Visual Studio I was keen to explore if I could continue to do this and then import the Form Class into IronPython. Well I figured it out and this is what I did...
First I created a new project in Visual Studio 2008. Under project types I selected Visual Basic -> Windows -> Class Library. I called my project VSWinFormsTest. Without changing any of the code in the auto-generated Class1.vb file, I added a new form to the project (Project -> Add Windows Form...), and built a simple test form as below:
Once the form was created I built the project and moved the resulting VSWinFormsTest.dll file (in the /bin/Debug/ folder), over to my Ubuntu VPC (I actually just emailed it to my hotmail account and then downloaded via Firefox).
Rather than write and save a script this time, I performed a simple test using IronPython's interactive mode. So after executing:
mono ipy.exe
I entered the following commands directly into the terminal whilst in interactive mode:
>>> import clr>>> clr.AddReference('System.Windows.Forms')
>>> clr.AddReference('VSWinFormsTest')
>>> from System.Windows.Forms import *
>>> from VSWinFormsTest import *
>>> form = VSWinForm()
>>> Application.Run(form)
A few moments after entering the last command the following window popped-up:
Mission accomplished!
And of course now that my form is imported (or subclassed) I can access its various properties in the usual way. For example, if I wanted to change the content of the single line text box I'd just write:
VSWinForm.txtSingleLineTextBox.Text = 'My new text'
As my project progresses I'll need to start adding event handlers to the various form events. My quest to using the best of Python and .NET and have it run on Linux-based system as well as Microsoft Windows systems is coming together... and in theory, the same should hold of Mac OSX!
In the considerable research I have undertaken to get this far it seems clear that if I had set off on this project just six months ago, I would've had a much harder time progressing as far as I have in such a short time. By complete fluke I seem to have set off on this at just the right time as Mono and Iron Python have matured enough for idiots like me to make use of it, and forums are slowly filling up with useful tips and advice. Thanks to all those who have made this possible!
First I created a new project in Visual Studio 2008. Under project types I selected Visual Basic -> Windows -> Class Library. I called my project VSWinFormsTest. Without changing any of the code in the auto-generated Class1.vb file, I added a new form to the project (Project -> Add Windows Form...), and built a simple test form as below:
Once the form was created I built the project and moved the resulting VSWinFormsTest.dll file (in the /bin/Debug/ folder), over to my Ubuntu VPC (I actually just emailed it to my hotmail account and then downloaded via Firefox).
Rather than write and save a script this time, I performed a simple test using IronPython's interactive mode. So after executing:
mono ipy.exe
I entered the following commands directly into the terminal whilst in interactive mode:
>>> import clr>>> clr.AddReference('System.Windows.Forms')
>>> clr.AddReference('VSWinFormsTest')
>>> from System.Windows.Forms import *
>>> from VSWinFormsTest import *
>>> form = VSWinForm()
>>> Application.Run(form)
A few moments after entering the last command the following window popped-up:
Mission accomplished!
And of course now that my form is imported (or subclassed) I can access its various properties in the usual way. For example, if I wanted to change the content of the single line text box I'd just write:
VSWinForm.txtSingleLineTextBox.Text = 'My new text'
As my project progresses I'll need to start adding event handlers to the various form events. My quest to using the best of Python and .NET and have it run on Linux-based system as well as Microsoft Windows systems is coming together... and in theory, the same should hold of Mac OSX!
In the considerable research I have undertaken to get this far it seems clear that if I had set off on this project just six months ago, I would've had a much harder time progressing as far as I have in such a short time. By complete fluke I seem to have set off on this at just the right time as Mono and Iron Python have matured enough for idiots like me to make use of it, and forums are slowly filling up with useful tips and advice. Thanks to all those who have made this possible!
Subscribe to:
Posts (Atom)