Tell Them What You Think

TTWYT’s development blog and related musings

See here for the background — today, there are two more DRM’d PDFs from Defra, and another one from Ofgem. What’s going on?

I’ve been working with Mark Horrell and Steph Gray at DIUS over the last couple of months to improve TellThemWhatYouThink’s support for their consultations.

It has been a splendid experience. DIUS’s consultations are now provided in an XML feed — lovely, structured ATOM goodness with custom elements galore, available to all. DIUS are the first (hopefully of many!) to provide their data in such a useful format, for which they deserve some serious kudos.

Not all has gone perfectly, though. I discovered that the library I was using to parse this feed did not understand custom ATOM elements at all, and as a result, the consultations appearing on the site were pretty garbled. I’ve fixed the problem now, but the consultations I had gathered before today were pretty messed up, so I’ve removed them from the site. The consultations which were currently live were gathered from the ATOM feed — correctly, this time — during the scrape last night and are now all present and correct. Apologies to anyone who got a duplicate email alert or who was inconvenienced by the sudden loss of content!

Thanks very much to Justin Kerr-Stevens for meeting me, to the Open Rights Group for the hook-up, and to Mark & Steph for all their work.

The MCGA have recently deployed a new version of their website. It has been significantly overhauled, and is much improved. Unfortunately, this has created some problems:

First, all links from TellThemWhatYouThink to content on the MCGA website have broken. Many government departments do not bother to ensure that their URIs remain alive after a consultation is completed. Often, they disappear quite quickly after a consultation concludes, or they are changed from something like:

www.department.gov.uk/open/someconsultation

to:

www.department.gov.uk/closed/someconsultation

This is really quite annoying. To deal with this problem, or at least, to lessen its impact, TellThemWhatYouThink checks all outgoing links to ensure that they are still alive. If they aren’t, a page is displayed with some (hopefully) useful suggestions — search for it on Google, and similar.

Unfortunately, this isn’t working with the MCGA, because they’re not returning the correct error code (404) when someone tries to access a dead link. Instead, they return a code indicating that the page has moved (302), and provide the URI of a custom error page as the new location. This is really broken: a 302 redirect should be used when content at a particular URI has moved to a new one, not when it has been removed completely.

Second, their new consultations does not exclusively contain consultations. It also contains awful, incomprehensible mess. Doubtless this is useful and meaningful to some people, but it is certainly not a consultation in the normal sense of the word: it it presumably a response. Whether or not the response itself is open to further comment, I do not know. The page doesn’t say.

Of the links on the consultation page,  only one looks to me like an actual consultation. Its structure does not bode well.

I shall review the MCGA website every so often to see if a new consultation has emerged to which the current one could be compared, and to see if any useful structure is present in the document. For now, though, I think it is broken, so I’m removing it from TellThemWhatYouThink until it can be supported again.

I had the pleasure of speaking to the assembled great and good at the Tower08 Transformational Government conference on Monday this week. I hope that video will be available at some point, and I’ll link to it if it is.

I talked, reasonably predictably, about the resusability of public data, and about why it’s important to embrace the idea that data should be made available in ways that allow people to use it, reuse it, combine it in new and clever ways and produce new, useful tools.

I also pointed out that there is an incredible amount of value to be generated from this data if it can be published in ways that allow more collaboration, and that it’ll be much cheaper in the long run if Government doesn’t try to solve all the problems. I drew a comparison between DirectGov’s fairly awful search facilities and the results produced by DirectionlessGov, which drew both heckles and laughs — an odd response. I am rather surprised to find that there actually are people out there who think that DirectGov’s search is better than Google’s. It’s a strange world we live in!

Being fairly new to the scene, I was most struck by the huge differences in people’s interpretations of what transformational government should be about. In fairness, this shouldn’t have been that surprising: everyone is interpreting it according to their vested interests, which is predictable enough.

At one end, there are people saying that everyone should own their own data, that public data is public property and should be disseminated in ways that make it as useful as possible, that massive data sharing and joined-up delivery of public services through one site is a dangerous folly.

At the other, you have people saying that we need to make identity card systems to share everyone’s data throughout government, that we should make public services usable online by having ultra-secure identification methods, that we need one place to find everything anyone might want from government, and that web 2.0, sharing and mass collaboration are merely the whimsical trends du jour.

I think it’s probably easy to tell where I stand! I’m happy to say that there is a cadre of people in government who also tend towards the former view, and that it is larger than one might think. These ideas are gaining some traction, at least, and that is quite something.

The National Institute for Clinical Excellence and the Legal Services Commission have been added today.

Unfortunately, support for NICE is pretty patchy. It’s the same old story: When their website is better,  TTWYT will be able to display more!

…but you probably won’t notice. Unusually, this is entirely a good thing!

I’ve been working with MoJ over the last few weeks to make some small changes to the way they publish consultations. These won’t be visible to the casual observer, but they will make it much easier for TellThemWhatYouThink to gather the details of consultations from MoJ’s webpages.

This has two consequences:

• The scraper that gathers data from MoJ is less likely to pseudorandomly break and require me to fix it;
• One day, I’ll be able to implement some new features.

I’m going to stay schtum about new features for now, mostly because I can’t launch them until more departments have updated their sites in similar ways.

In any event, I have been very pleasantly surprised by the painlessness of the whole process. Many thanks to Jeremy Gould and his colleagues. These changes happened very quickly and are definitely for the better. Kudos!

The recent redesign of the DoH website is, as has been noted by others, a great improvement. I was happy to discover that they do now have RSS feeds for consultations, and, more impressively, for consultation responses. This is something people have wanted for years, so it’s great to see that it has finally happened.

Unfortunately, they don’t work. The current live consultations feed, erroneously, contains no items.

I shall let them know…

The site will now display all the Ofsted consultations it can find. I hope this means that, along with the DCSF, the site will now find all government consultations relating to education.

That said, a little bird told me that schools regularly receive consultation documents, often directed at specific heads of department, with requests that they participate. I don’t know the details yet. If they are “private” consultations then I suppose that’s all well and good, but one does wonder if they might just be public ones that no one knows about!

Hopefully I’ll have some more information about that soon — watch this space!

One of the things I hope that TTWYT might accomplish is to stimulate more “blogospheric” debate when consultations are released. It would be marvellous if the publication of an important consultation led to lots of excellent blog posts followed by the submission of lots of independent* responses!

To that end, each consultation’s details page now displays a list of pages which have linked to the consultation, as reported by Technorati.
* I say independent because I am reliably informed that when a response to a consultation is submitted in the names of multiple people, it is common practice to count it as a response from one individual, rather than several. Argh!

Just did a quick spot on You & Yours about the lack of central consultation publishing, the difficulty this creates in finding interesting consultations, and the consequent narrowing of public debate that occurs.

Unfortunately, they asked me not to mention the site’s URL on the air. Oh well!