Big data and accountability

Posted on 27/11/2017 by Ed Hammond. Tags: ,

Before I went off on paternity leave I saw an interesting blog about how New Orleans is using “big data” to improve performance and accountability. A few years ago, big data was the big thing in public services – the idea that professionals gathering information from a wide range of sources to give them as good an idea as possible about how services are experienced by people on the ground – and how we can find novel solutions to the complex challenges they face.

The blog is a good read (you can find it at It left me a bit troubled though, and I resolved that when I got back from leave I’d return to it and write something about why it doesn’t quite sit right with me.

There are, I think, two fundamental problems.

Firstly, big data seems to me to be too much about seeing services and issues from the perspectives of ourselves, as professionals. It is about experts gathering information around themselves, which then gives them the evidence they need to inform the decision-making process. It is sold as disruptive, whereas actually it can entrench existing power imbalances. In a world where data is currency, big data can accentuate the divide between those who have data – and the tools to understand and act on it – and those who don’t.

Many municipalities in the USA have put in place public data portals to try to tackle this challenge (it’s notable that – other than some steps taken by the Government Digital Service with, this practice isn’t widespread this side of the pond). But managed incorrectly, this kind of infodumping just serves to exacerbate this problem – not least because of the way that sloppy analysis of large amounts of data can encourage people to identify correlations where none exist (this, and other criticisms of big data, are neatly summarised in this New York Times piece –

This is why I worry about big data being seen as an accountability mechanism, and being talked about in the same breath as lofty aspirations for more government transparency. We can use data to open up local conversations about priorities – about where challenges lie and how we’re going to work together (and with local people!) to address those issues. But big data as it is usually used – and understood – is about professionals and their decision-making duties – not about giving the public the tools to take part in the conversation.

This brings me onto my second point. This is perhaps illustrated best by a quote in the blog referenced above from the Director of New Orleans’ “Office of Performance and Accountability” (and yes, despite all the above, I do experience a pang that English councils don’t have such offices):

“Operational issues are what keep mayors and their staff up at night. Ideology doesn’t do a whole lot in helping solve those problems. It takes pragmatism and a lot of hard work, as well as an honest look at the data to tell you what works and what doesn’t.”

Perhaps something has been lost in translation here (two countries separated by a common language and all that). Because my second problem about big data is that it purports to push out politics. Politics is about trade-offs; it is about personal worldview, about motivation, about the way we want the world to be in the future. Much as it would be lovely to live in a technocratic dictatorship run by a cabal of benevolent bureaucrats, I’m troubled by any approach to decision-making that suggests that it is replacing politics with pragmatism – as if the two are somehow, by definition, in opposition. Our interpretation of evidence is, by definition, subjective – it is coloured by our worldview, by our personal and political preferences. Assuming that we can somehow divorce ourselves and our biases from our decision-making duties is dangerous. Data should be in service of politics and these hard choices – not their replacement.

Where does this leave us? The important of evidence-informed decision-making is not at issue (indeed, we published an excellent report on it quite recently – There is a need to take the next step, to shake up our cosy, traditional decision-making systems and open them out, to allow evidence to inform debate and discussion within the wider community – but led by politicians. Only then will big data mean big accountability.

About the Author: Ed Hammond

Ed leads CfGS's work on devolution, transformation and on support to councils and other public bodies on governance and accountability.