Tell Them What You Think

TTWYT’s development blog and related musings

A few months ago, I responded to a couple of government consultations and, in the process, discovered there was no way to search all live consultations, or to be alerted when a new one was published.

This struck me as more than a little mad: consultations are one of the government’s main methods for engaging in substantive public debate. I am not aware of any other process by which Joe Public can express their views on a particular topic, in detail, and can be reasonably certain that they will be read, and justifiably hopeful that they will be taken into account. That said, the entire process is worse than useless if no one finds out about them in time to contribute.

I thought that there must be a solution to this problem. My first port of call was a quick survey of government websites to determine which ones provided their data as RSS feeds. To my utter astonishment, only the Welsh Assembly Government did. With this came the realisation that the problem was a bigger one than a mere aggregator could solve (call me naive, but I had hope!), so I set about writing a screen-scraper for the DCMS as an experiment. I subsequently discovered that I had unwittingly picked one of the worst departments to scrape: their code is particularly unstructured. After implementing scrapers for ten or so departments, I had a look at the horrendous spaggetified ascii pudding that my code had become, and started again.

Over a couple of weeks, my little collection of scrapers grew until I had twenty or so departments. At that point, the data was only available as RSS feeds, and I had no plans to do anything more with it. After showing it to a few people, however, its potential became clear, so I started on a proper website to do all the things that people would want. In particular, Tom Steinberg’s reaction was effusive, and his enthusiasm infectious. I was also much encouraged by the numerous splendid people I met at the inaugural UKGovWeb Barcamp, who all said very useful things. Kudos in particular to MySociety for offering their hosting services.

In the weeks following Barcamp, I continued to beaver away at the scrapers and the site, until it went live on 11th Febuary, 2008. I have been very pleasantly surprised, to say the least, by the level of interest TellThemWhatYouThink has received. I have already had a few meetings with civil servants to discuss changes to their sites which would improve the quality of the data TTWYT’s scrapers can gather, and several more are planned. Lots of people have emailed me great feedback on improving the site, and I and others have lots of ideas about where it could go in the future. To all involved: many thanks.

Watch this space!