Tell Them What You Think

TTWYT’s development blog and related musings

TellThemWhatYouThink tries to get a list of documents for each consultation it finds. When it finds documents, it indexes them, so that they can also be checked when you perform a search, or when new consultations are matched up against email alerts.

These documents come in a variety of formats. One of those formats — PDF — allows its creator to set a “no-copy” bit. This prevents users from selecting and copying text from the document, and also prevents TellThemWhatYouThink’s indexing tool from converting the document into a form it can read. This is a crude form of Digital Rights Management.

I settled down this morning to figure out why all of the documents from one of Defra’s current consultations were unable to be indexed. Imagine my surprise when I discovered that all of them have the no-copy bit set. I can’t index them. You can’t copy & paste any of the text into your response as a quote. You can’t copy an excerpt to a discussion board.

This is all fantastically irritating, of course, but more to the point: what on earth is government doing copy-protecting its consultation documents?

I added comments to TellThemWhatYouThink over the weekend. As an anti-spam measure, I included a text box asking people to complete a simple sum to prove they were a person. This was for two reasons:

• It’s very quick to implement
• It’s more accessible than an image captcha — the sum will be read out by screen readers

Unfortunately, though, it doesn’t work. Be warned! Spam comments started appearing on the site within hours of the new functionality going live.

I’ve replaced that system with a reCAPTCHA test. It has an audio test for blind users — I hope it’s sufficient. We shall see how it fares!

Some splendid people in the US have released a paper recommending that Government should focus on releasing its data in reusable ways, rather than designing better user experience — their thesis being that third parties will do this for them if the data is available. Funnily enough, I find myself agreeing.

The paper is a little US-centric but the principles are sound and entirely applicable to the UK. Ed Felton has posted an excerpt if you don’t fancy reading the whole thing.

The paper hits all the right buttons: let innovative private bodies come up with the best ways to display government data and compete for each other’s audiences. Make structured data available first, and then produce a government site to display it, if required — the data should be a priority, not an afterthought. Make sure that such data feeds exist in known, permanent locations. Make government sites operate on the same data they provide to others — TellThemWhatYouThink is powered by its own API. We eat our own dogfood. I wonder how many government departments can say the same^*?

I made a lot of these points when I spoke at Tower08 earlier in the year. It’s definitely true that there are people in Government who agree, but there’re still plenty that don’t — hence the unfortunate need for campaigns like Free Our Bills.

Hopefully, reports like these will continue to be written, good examples of the reuse of public data will continue to be found, and Government will eventually see the light.

¹ Ok, that’s not quite true. I know the answer, as does all of sentient life.

I’ve been working with Mark Horrell and Steph Gray at DIUS over the last couple of months to improve TellThemWhatYouThink’s support for their consultations.

It has been a splendid experience. DIUS’s consultations are now provided in an XML feed — lovely, structured ATOM goodness with custom elements galore, available to all. DIUS are the first (hopefully of many!) to provide their data in such a useful format, for which they deserve some serious kudos.

Not all has gone perfectly, though. I discovered that the library I was using to parse this feed did not understand custom ATOM elements at all, and as a result, the consultations appearing on the site were pretty garbled. I’ve fixed the problem now, but the consultations I had gathered before today were pretty messed up, so I’ve removed them from the site. The consultations which were currently live were gathered from the ATOM feed — correctly, this time — during the scrape last night and are now all present and correct. Apologies to anyone who got a duplicate email alert or who was inconvenienced by the sudden loss of content!

Thanks very much to Justin Kerr-Stevens for meeting me, to the Open Rights Group for the hook-up, and to Mark & Steph for all their work.

The MCGA have recently deployed a new version of their website. It has been significantly overhauled, and is much improved. Unfortunately, this has created some problems:

First, all links from TellThemWhatYouThink to content on the MCGA website have broken. Many government departments do not bother to ensure that their URIs remain alive after a consultation is completed. Often, they disappear quite quickly after a consultation concludes, or they are changed from something like:

www.department.gov.uk/open/someconsultation

to:

www.department.gov.uk/closed/someconsultation

This is really quite annoying. To deal with this problem, or at least, to lessen its impact, TellThemWhatYouThink checks all outgoing links to ensure that they are still alive. If they aren’t, a page is displayed with some (hopefully) useful suggestions — search for it on Google, and similar.

Unfortunately, this isn’t working with the MCGA, because they’re not returning the correct error code (404) when someone tries to access a dead link. Instead, they return a code indicating that the page has moved (302), and provide the URI of a custom error page as the new location. This is really broken: a 302 redirect should be used when content at a particular URI has moved to a new one, not when it has been removed completely.

Second, their new consultations does not exclusively contain consultations. It also contains awful, incomprehensible mess. Doubtless this is useful and meaningful to some people, but it is certainly not a consultation in the normal sense of the word: it it presumably a response. Whether or not the response itself is open to further comment, I do not know. The page doesn’t say.

Of the links on the consultation page,  only one looks to me like an actual consultation. Its structure does not bode well.

I shall review the MCGA website every so often to see if a new consultation has emerged to which the current one could be compared, and to see if any useful structure is present in the document. For now, though, I think it is broken, so I’m removing it from TellThemWhatYouThink until it can be supported again.

I had the pleasure of speaking to the assembled great and good at the Tower08 Transformational Government conference on Monday this week. I hope that video will be available at some point, and I’ll link to it if it is.

I talked, reasonably predictably, about the resusability of public data, and about why it’s important to embrace the idea that data should be made available in ways that allow people to use it, reuse it, combine it in new and clever ways and produce new, useful tools.

I also pointed out that there is an incredible amount of value to be generated from this data if it can be published in ways that allow more collaboration, and that it’ll be much cheaper in the long run if Government doesn’t try to solve all the problems. I drew a comparison between DirectGov’s fairly awful search facilities and the results produced by DirectionlessGov, which drew both heckles and laughs — an odd response. I am rather surprised to find that there actually are people out there who think that DirectGov’s search is better than Google’s. It’s a strange world we live in!

Being fairly new to the scene, I was most struck by the huge differences in people’s interpretations of what transformational government should be about. In fairness, this shouldn’t have been that surprising: everyone is interpreting it according to their vested interests, which is predictable enough.

At one end, there are people saying that everyone should own their own data, that public data is public property and should be disseminated in ways that make it as useful as possible, that massive data sharing and joined-up delivery of public services through one site is a dangerous folly.

At the other, you have people saying that we need to make identity card systems to share everyone’s data throughout government, that we should make public services usable online by having ultra-secure identification methods, that we need one place to find everything anyone might want from government, and that web 2.0, sharing and mass collaboration are merely the whimsical trends du jour.

I think it’s probably easy to tell where I stand! I’m happy to say that there is a cadre of people in government who also tend towards the former view, and that it is larger than one might think. These ideas are gaining some traction, at least, and that is quite something.

The National Institute for Clinical Excellence and the Legal Services Commission have been added today.

Unfortunately, support for NICE is pretty patchy. It’s the same old story: When their website is better,  TTWYT will be able to display more!

…but you probably won’t notice. Unusually, this is entirely a good thing!

I’ve been working with MoJ over the last few weeks to make some small changes to the way they publish consultations. These won’t be visible to the casual observer, but they will make it much easier for TellThemWhatYouThink to gather the details of consultations from MoJ’s webpages.

This has two consequences:

• The scraper that gathers data from MoJ is less likely to pseudorandomly break and require me to fix it;
• One day, I’ll be able to implement some new features.

I’m going to stay schtum about new features for now, mostly because I can’t launch them until more departments have updated their sites in similar ways.

In any event, I have been very pleasantly surprised by the painlessness of the whole process. Many thanks to Jeremy Gould and his colleagues. These changes happened very quickly and are definitely for the better. Kudos!

The recent redesign of the DoH website is, as has been noted by others, a great improvement. I was happy to discover that they do now have RSS feeds for consultations, and, more impressively, for consultation responses. This is something people have wanted for years, so it’s great to see that it has finally happened.

Unfortunately, they don’t work. The current live consultations feed, erroneously, contains no items.

I shall let them know…

The site will now display all the Ofsted consultations it can find. I hope this means that, along with the DCSF, the site will now find all government consultations relating to education.

That said, a little bird told me that schools regularly receive consultation documents, often directed at specific heads of department, with requests that they participate. I don’t know the details yet. If they are “private” consultations then I suppose that’s all well and good, but one does wonder if they might just be public ones that no one knows about!

Hopefully I’ll have some more information about that soon — watch this space!