Tuesday, 12 February 2013

Green Deal or No Deal?

Green Deal Basics

As detailed in a blog post last year, my home energy use and a thermal imaging survey I had done show very clearly that my house is in need of serious energy efficiency work. I was intending to do this last year but working out how to finance the work proved to be a bit of a problem as I had depleted my savings to minimise the size of the mortgage. So when the Green Deal came along, I thought it would be the ideal opportunity to get the work done.

The way the Green Deal work is meant to be extremely simple:

  1. Get an assessment done via an accredited Green Deal Assessor,
  2. Find a Green Deal Provider,
  3. Get the work done via a the provider and a Green Deal Installer,
  4. Pay back the money via your electricity bill.

As an aside on the last step, considering that the Green Deal is meant to improve energy efficiency and that most of the housing stock in the UK suffers from poor insulation, the energy use that is most likely to be affected is heating, which in turn would primarily affect gas bills. Therefore paying back via your electricity bill sounds slightly counter-intuitive.

As another aside on the last step, you pay back the money with interest (you didn't think it'd be free credit, did you?) and the interest rate on Green Deal loans is around 7.5% as explained by Which? in their guide. This is not a great rate and you may have cheaper options at your disposal, as explained by Which? as you can choose to finance the works differently. The main advantage of Green Deal loans compared to any other option is that they are attached to the property rather than yourself. The way I understand it, it means that if you sell the property, the next owners will repay the remainder of the loan on their electricity bill as they benefit from the improvements and the loan will not be subject to any credit checks against you. However, don't take my word as gospel, check with an accredited adviser, which brings me to the next step.

Finding an Assessor

The first step in the Green Deal is to find an assessor. The best way to do this is to peruse the search facility on the Green Deal Oversight & Registration Body's web site: you can search by name or postcode, residential, non-residential or both. If you search by postcode, results will be ordered based on proximity, which is very handy.

I did just that and am going down the list, calling each one of them in turn based on the assumption that I will get a much better feel for the company if I have them on the phone than if fill in an online form. I added a small tweak to this, I will not call British Gas for two reasons: I called them last year so I know what their offering is like and will actually use them as a benchmark to compare the other guys against, and I am all for giving my money to smaller businesses rather than large incumbents. So far I called five of them and got fairly varied answers, little of it being promising.

  1. When I called the first company, I got to speak to a nice lady who had an accent that was quite hard to understand. She understood what I was looking for but failed to adequately answer my questions. In particular when I asked her how they could justify charging me £150 for an assessment when British Gas was charging £99, she told me that they would not require me to go forward with them as a Provider, contrary to British Gas that would. This sounded a bit off as I seemed to remember there wasn't any condition to the British Gas assessment. Anyway, the whole conversation felt a bit dodgy and she completely failed to take any of my details or to tell me how to organise an assessment with them. In fact, I positively had the impression she wanted me off the phone.
  2. The second company was a bit more promising as the lady I spoke to knew what I wanted and said that I had to fill in their online survey and that someone would call me back. I felt that their survey was more geared towards working out how much cashback I could get on the deal than actually taking meaningful information from me but I filled it in nonetheless. Nobody has called back yet. Note that even though the cashback option is nice, this is not the primary reason I want to go through the Green Deal so I'm not sensitive to any advertising that focuses on that. Others may be.
  3. The third company I called was the first one that is a generic green energy company and that does not focus solely on the Green Deal. To the point that not only does their web site fail to display the Green Deal Approved logo, it also fails to mention the Green Deal altogether and the lady I had on the phone had no idea what I was talking about but promised to forward my details to the man who did know about this. He hasn't called me back yet. This is a shame because as far as I am concerned I may take the opportunity to have other improvements done that are not covered by the Green Deal so working with a company that has a wider scope could have been good.
  4. The fourth company I called knew what I was talking about but said that they couldn't take any booking for now because they were still waiting for some software from the government. Now, I do understand that if the government promised assessors some software or other to do their work and they are late to provide it, it will put a fly in the ointment but my understanding of the assessment is that the first thing to do is a site visit that doesn't need any special software, just someone who knows what they're talking about. Besides, based on past experience, basing your ability to take on new customers on having some software delivered by the government is not a predictable way to run your business so if you don't have a plan B it doesn't inspire much confidence: conditions are never perfect in business, deal with it or you won't have a business very long.
  5. The last company I called so far was by far the best of the lot. The lady on the phone knew what I wanted, was very knowledgeable about the Green Deal, acknowledged that I could finance the deal in different ways and that even after I had done the assessment with them I could go to a different provider. She took all my details and said that an assessor would call me to arrange a convenient time for a site visit. Nobody has called yet but it's still a lot more than any of the others were able to do.

The conclusion of all this is that if the Green Deal is to take off, the first thing would be to have companies ready to initiate new business when it comes knocking on their door. Apart from British Gas who were already eager to book me for an assessment back in November, only one of the smaller companies I called came remotely close to doing the same thing. Note to companies out there: being Green Deal Approved does not mean you can be completely devoid of business sense, get your act together and train your sales team!

Next Step

Call the next 5 companies on the list.

Sunday, 20 January 2013

Identifying spam

Refugees United

Yesterday, I participated in the Refugees United Modding Day 2013 organised by Rewired State for Refugees United. I worked on a hack that tried to implement and algorithm to identify spam and eventually also identify spammers. Our presentation was OK but not fantastic as time was short. The short of it was that by the end of the day I wasn't convinced the algorithm as coded worked so today I decided to get some statistics out of it and tweak it a little bit.

Bayesian Filters

The idea was to implement a Bayesian filtering algorithm to detect spam. This type of algorithms requires initial training using a body of existing messages comprising both spam and non-spam where you tell it which is which. By doing that, it will build a corpus of know good data and a corpus of known spam data. It will then use that stored information to score incoming messages. The resulting score is a probability between 0 and 1 of the message being spam (in practice the probability is between 0.01 and 0.99 because you can never be completely sure). After that, each time the algorithm gets its decision wrong, the user has the ability to correct it and it will learn from the correction. This is exactly the sort of algorithms that is implemented in popular email clients like Mozilla Thunderbird.

In order to train the filter, we had a file containing 5500 SMS messages, some spam, some ham. So in order to see how good the filter was, I decided to do the following:

  • Train the filter on the first few hundred messages in the file;
  • The score the remainder of the file and see for each score whether it correctly identified the message as ham or spam.


The main factor that influences how such a filter works is the way the message is broken into individual tokens. The simplest algorithm is to break the message into words. Some words will be used mainly in spam messages, others mainly in innocent messages and yet others in both. Using 5% of the file for training and the remainder for scoring and applying that simple tokenizing algorithm, I got the following statistics:

  • Correct guesses: 72.03%
  • Neutral score (doesn't know if it's spam or ham): 25.68%
  • False positives (identified as spam when it is ham): 1.74%
  • False negatives (identified as ham when it is spam): 0.55%

That's not bad for a basic algorithm! So I decided to apply three additional tokenizing techniques on top of the core one and see what would happen:

  • The first technique consists in adding word pairs in addition to individual words. The logic behind this is that words like fantastic and offer may not be typical of spam on their own but the combination fantastic offer may be.
  • The second technique consists in adding the lowercase version of every word to the list of tokens in an attempt to see if weird capitalisation would catch spammers out.
  • The third one is a combination of the first two.

The results show that word pairs seem to be better than capitalisation but both together are even better.

Algorithm comparison: overall

However, this is not the whole story. The important metric is the bad stuff that ends up in the user's inbox as well as the good stuff that gets rejected. The former is composed of two parts: messages that the filter considered good when in fact they should have been caught as spam (false negatives) and messages that had a neutral score that are in fact spam (bad neutral). The latter is messages that the filter marked as spam when in fact they are good (false positives). Looking at those metrics, the result is not great.

Algorithm comparison: mistakes

False positives increase slightly with word pairs and quite significantly when introducing capitalisation variation. This may mean that a lot of legit users actually capitalise their messages like spammers. So not such a good algorithm after all.

Introducing Bias

One aspect of spam is that it is better to have false negatives than false positives. That is, it is better to have a bit of spam in the user's inbox but not block legit messages by mistake. So one typical way to do this is to introduce some bias into the algorithm to give a bit more weight to positive probabilities. Compared to the basic algorithm, it definitely cuts into false positives.

Algorithm bias comparison: overall

Algorithm bias comparison: mistakes

The Impact of Training

An interesting statistic is to see how much training the algorithm makes a difference. So I decided to compare the baseline where 5% of the file is used for training to a version where 10 of the file is used for training.

Algorithm training comparison: overall

Algorithm training comparison: mistakes

And of course, a well trained biased algorithm returns some decent numbers too when compared to the baseline.

Algorithm training and bias comparison: overall

Algorithm training and bias comparison: mistakes


The strength of statistical algorithms like Bayesian filtering is that it learns from the mistakes it does and it doesn't matter if spammers change their wording, the algorithm will adapt. This is demonstrated by the reasonably high rate of success reached by a very basic implementation of the technique with training on a few hundred messages.

In order to improve the efficiency of such an algorithm, there are a few options:

  • Include message meta-data such as sender and recipient details, in addition to the message text: this would also be a first step to identify spammers in addition to spam, which was one of the aims of the hack.
  • Use advanced tokenization techniques such as sparse binary polynomial hashing.
  • Use Markovian discrimination rather than Bayesian filtering.

All those improvements would come at the cost of increased complexity and processing power requirements so may not all be practical.

Sunday, 13 January 2013

for and while constructs in bash

The for construct

When you want to iterate over a list in bash, the first thing that comes to mind is to use a for loop, like this:

for f in "abc def"; do
    echo $f

Simple for loop

This works great when the list to iterate over is short and is composed of items that do not contain any white space. When they do, or the list is long, this construct will get into trouble. Let's demonstrate with a simple example. If I create a file with one item per line, 3 lines like this:

third line

Simple file called test

Then the first attempt at using a for loop would be:

for f in $(cat test); do
    echo $f

Simple for loop to read the file

The result is not quite what was expected:


Output of the for loop

You can put double quote in different places, this will not solve the problem. This is because the for construct splits items against white space and as far as it's concerned, an actual space character or a carriage return are the same and count as separators. Another limitation of the for construct is that the sub-command contained in $(...) needs to be fully executed before for can even start. If the output is large, it can run out of memory or just take a long time to get started.

The while construct

Fortunately, bash has another construct that can bypass those limitations, the while construct. It works slightly differently and needs the help of the read command.

cat test | while read f; do
    echo $f

A simple while example

And the result is:

third line

Output of the while loop

This works because the read command reads a full line and does not split on white space. Therefore the value that f is set to is a complete line in the file. The other advantage is that the pipe actually streams the output of the cat command to while and read, meaning that there is no need to wait until it's finished to handle its output. One typical use of that construct is when using the find command: with modern operating systems, file names can have spaces in them and even with a tight condition, find can return hundreds of lines of output.

Use the right tool for the job

So when should you use which construct?

  • If you are dealing with a list that can be large or where each item can contain space characters, use while;
  • If you are dealing with a short list where no item can contain a space character, you can use for.