The Internet has fallen apart

The internet used to be a social network.

You used to be able to email people you didn’t know: you’d see an article about a topic in which you have some expertise, and you might want to email its author.

Basically, you can no longer do this: you might be able to leave a comment on the article, which is no good if you want to write in private, and in any case this nowadays involves signing up with some awful third-party identity integration service like Disqus or WordPress or Gravatar or whatever, and your message can get lost in the fray of utter bilge in the comments section, whereas it might have got read and led to a discussion if it had gone by email.

Your other option is sending a message on Twitter, but that’s got all the same problems as the above, but limited to 140 characters.

We’ve certainly lost some of the original benefit of the Internet as a facilitator of discussion, at least between strangers.

If you could bear reading this far, you should follow me on Twitter:

The Crown Prosecution Service and the black market for personal data

The Crown Prosecution Service is not my favourite state agency.

Their leadership is too closely linked to News Corp and the Labour Party, and they don’t seem to take seriously enough the business of prosecuting people for data snooping. The fewer prosecutions, the lower the risk of punishment, the greater the long-run rewards for ransacking personal data holdings and flogging them off on the black market.

The Bank of England by its actions regulates the supply of credit within the economy. By failing to prosecute in sufficient numbers those who abuse their authorised access to personal data and then sell it to newspapers, private detectives, political extremists, criminals, the KGB, etc, etc, the CPS regulates the size of the black market in personal data. I have submitted a FOIA request to ascertain whether the CPS does any economic modelling to work out how harmful these crimes are when deciding how many of them to prosecute. They ignored some of the questions in a previous request on this subject, so I’ve effectively resubmitted one of the questions they pretended omitted to answer.

When I submitted this, I got an automatic courtesy response, advising me of the deadline they have to meet, which is on Monday. I’ve waited until today, the preceding Friday, before responding “thank you for doing me the courtesy of a holding response about my recent FOIA request. I was delighted to read that you are required to respond on or before Monday. Have a good weekend!

I expect they’ll again fail to answer the question, and I’ll go to my MP.

If you could bear reading this far, you should follow me on Twitter:

Are Dark Forces redefining trolling?

Has the word “troll” been redefined deliberately?

Around the end of 2010, I started to notice mainstream news articles using the word “troll”, but it did not retain its then-conventional acceptation which was “person who tries to provoke a negative reaction in an online discussion”. People started to use the word to mean “stalker”, or “critic”. My inference was that our folkwardens were readying the law looms to legislate against free speech online, and that this conflation of criticism, playful satire and already criminal intimidation was part of the softening-up exercise to prepare the public for being censored.

The term “troll” was problematic even in the 1990s, as even when it had a commonly-accepted meaning, it was regarded unreflectively by some as an inherently illegitimate activity. The argument ran that deliberately diverting a discussion was against the forum rules and ipso facto wrong. A better approach was to recognise that posting some absurd provocation is sometimes the best way of puncturing pomposity and luring the pretentious into unforced error and exposing their arguments, but that this consumes finite attention bandwidth, so there is a need for a variety of forums which trade off these concerns against each other to different extents.

The redefinition, or “ambiguisation” of the word “troll” allows someone to attack the activities described by the old sense of the word, whilst purporting only to attack those covered by the new sense. This may in fact now be a general problem; if there’s something you don’t like, start using the word which describes it to refer to something else which no-one likes, and then attack the former by attacking the latter. Your opponents will be left trying to explain semantics.

In the future it may be possible to detect these attempts to redefine and “own” the vocabulary, which could give an early warning of planned repressive legislation.

(The Wikipedia page on trolling primarily documents the old sense of the word, of provocation.)

If you could bear reading this far, you should follow me on Twitter:

WordPress is much more dreadful than I thought

A few months ago I went to considerable effort to convert my blog, which had run on software I’d written myself, to WordPress.

It turned that WordPress is utter rubbish for what I want it to do.

I want to compose blog posts in a text editor, and then publish them. To a first approximation, you can’t do this in WordPress without corrupting the blog post.

WordPress inherits PHP’s inability to process text without messing something up. In this case, the XMLRPC interface declines to cope with embedded HTML in the submitted text. I have therefore been reduced to trying to cut and paste the article into WordPress’s web-based interface. This interface allows you to edit rendered and raw HTML. It overrides the OS’s default cut-and-paste method to prevent you pasting text into the rendered editor, and strips the HTML out of the raw editor. You can upload files, but only media files, not text.

I adopted WordPress to get support for captchas once my homegrown one got beaten by spam bots, and to implement Trackback/Pingback. It looks like it’s time to move on again.

Updated: I managed to post a new article by, get this, using Chrome to edit the raw HTML of the editor window in WordPress’s interface, and pasting in the text from Emacs. Justice.

If you could bear reading this far, you should follow me on Twitter:

Is international democracy possible? Or EU democracy?

Today David Aaronivitch has an article behind the Times paywall titled “But what if Europe follows a different map?”. In it he discusses the possibility of the UK’s terms of EU membership changing, and the establishment of democratic institutions at EU level.

Quite rightly, he questions whether the rest ot the EU would be prepared to accept the sort of changes in UK terms of membership which (e.g.) the UK Prime Minister wants to achieve. He’s probably wrong about this: the EU already accommodates “differential geometry”, in that some countries are not in the Eurozone or Schengen system, and that it is in the economic interests of the rest of the EU to maximise British economic involvement.

As a parallel, we should look at another undemocratic international institution, the ITU. Most ITU member states had government postal / telephone monopolies, whereas the United States basically did not, and the US and a few similar countries therefore constituted an awkward minority. However, the ITU accepted US partial membership on the grounds that this was better than non-membership as a means of advancing the ITU’s core objectives of rent-seeking, surveillance and preventing technological change and competition in the telecommunications sector.

The EU is not going to turn around and tell Britain to take its appalling trade deficit somewhere else. If the UK government asked for minimal EU membership, the other countries would prefer this to UK withdrawal. Aaronovitch doesn’t really want his readers thinking about this.

He also talks about “supra-national democratic institutions” that “can hold this process to account, somehow make it legitimate for the Hanoverian to help out the Thessalonikan, as the Texan pays for the Vermonter”. Why does he question whether a European acceptance of UK disengagement can ever happen, but not question whether this suprademocratic democracy can ever exist? It’s a big blind spot for the Europhiles.

I don’t believe a supra-national democracy can exist. It’s a contradiction in terms.

The existence of a demos is a constraint, not a policy. It’s a social network of people who generally agree that they’re going to govern themselves. The UK shows that separate countries can both come together and form a single country (England, Scotland and Ireland), and that sometimes people want to go off and govern themselves (the Republic of Ireland).

Over the centuries, Greeks and Germans may gradually come to think of themselves as Europeans, as Welsh, English and Scottish people largely came to think of themselves as British; the process can go the other way too, as people come to believe that being Irish meant being not British. Whether a particular group of people generally think they’re part of a group is up to the people themselves; you can’t make people feel a common bond.

The unchanging “essentialist” conception of what a nation is, favoured by far too many British Eurosceptics and occasionally Germany’s judiciary, is wrong. Nations can split and merge. But this is a split or a merger of a collection of social relationships the members choose to make between themselves. It is these relationships which mean that people consent to be governed by other members of the group. I take “consent” broadly, in a way which would encompass mediaeval feudal Europe and include adherence to a long tradition of being governed together, even if not very democratically, which offers some hope for Europhiles who can accept just leaving the people to grow together over centuries.

But to be clear: if the people of two nations consent to be governed as one, then they’re already the same nation; the social network is merging. Supranational democracy is as unnecessary as it is impossible.

It is simply untrue to say that the current population of the EU generally want to be governed as a single polity. In the UK there’s majority support for reducing the powers already exercised at EU-level. The EU constitution in practice already goes vastly beyond what people anywhere in the EU would consent to, with the exception of a statistically insignificant extremist Europhile minority.

So, if we give the words their conventional acceptation, the only way there’ll be a democratic Europe is if the generality of the public want it.

Criticically though, the rational constructivists have, without the consent of the peoples of Europe, created a constitutional system for the EU in which responsibilities are gradually and unilaterally transferred to the EU by the Luxembourg Court, but the democratic institutions to constrain the exercise of those powers can only be established very gradually and require a consent that may take centuries to evolve.

The more that power is exercised by the EU, the more it affects people’s lives and the more they’ll try to have their say.

The smart Europhile would prioritise measures to prevent this situation getting any worse. The European Court of Justice is a sham and may be beyond reform, but that’s where the smart Europhile should focus.

But I’m not a Europhile …

If you could bear reading this far, you should follow me on Twitter:

Sikorski on European integration

Jon Worth asks why Radek Sikorski’s speech about the EU has been ignored. Why wouldn’t it be?

Sikorski’s latest effusion is not really a contribution to any debate about the EU, but a ritual performance of Europeanism which should be counted alongside the secular hagiography of Spinelli, Charlemagne, etc undertaken by the likes of Chris Patten. It is motivated by an ambition to be thought of as strongly pro-EU by Eurocrats, in the hope of receiving some honour or sinecure from them, not as a practical proposal to effect actual change in our constitutional arrangements.

If you could bear reading this far, you should follow me on Twitter:

Project Ronald, source code available

The source code for Project Ronald is now available on GitHub.

If you could bear reading this far, you should follow me on Twitter:

Project Ronald, an example

The first objective of Project Ronald is to make it easy to connect tabular datasets quickly: given two openly-licensed tabular datasets containing a common field, but published by different organisations, it ought to be possible to get them downloaded and joined together in a few seconds. The approach is to identify the components of a system which would do this, implement a minimal version of each, check that the system works as a whole, and then go about replacing each component with better tools, preferably ones already written and matured by someone else.

The example research question to motivate my example is about a minor local political controversy: is there a correlation between ethnicity and voting behaviour in London mayoral elections? What I have done with the data is not statistically rigorous at all; it is only to demonstrate the data acquisition and linking tools.

We start with two datasets: the London mayoral election results by ward, and the census ethnicity breakdown by ward. As it happens, I had to wrangle these both to be in the form of percentages, but they contained a standardised code identifying the ward, which helped considerably. Two pieces of metadata are needed for each tabular dataset in CSV form: the CSV dialect and the CSV schema.

The CSV file, and its two metadata objects are registered with a Data Management System the CKAN DMS allows me to attach arbitrary attributes to a dataset, so I use this facility to record which resource is which, in a manner understandable by an automatic tool. It is then trivial to create a tool which knows this convention and can download a CSV with dialect and schema metadata given a dataset name. This information suffices to create a table in a relational database (I used MySQL) and the statistics system, R, has a MySQL driver; both MySQL and R can efficiently join two datasets using a common field.

With such a system, you can get the data into your statistics package and start running regressions in seconds, which is much harder where the data must be licensed or the tools are not coupled together. In addition to open licensing and tool connectivity, the following steps need to have been taken, not necessarily by the same person:

  • the dialect of the CSV file must be determined
  • the schema of the CSV file must be determined
  • the joinable fields in the CSV table must be consistently coded
  • the CSV file needs to be clean, in various senses

Much of this work is automatable, which is the topic of a later post

If you could bear reading this far, you should follow me on Twitter:

Project Ronald, an introduction

I live in a society in which the Government is restrained from taking any action which can be denounced in a 200 word article in the The Sun. How British democracy has attained this pitch of excellence need not concern us here, rather we must ask: how can we increase the explanatory power of articles in The Sun?. To do otherwise is to favour the obstruction of public policies exclusively on grounds of simplicity rather than demerit, with the result that any bad policy may be adopted if its proponents first trouble to complicate it unnecessarily.

Newspaper articles and narratives are but the finished product in a long supply chain of knowledge. The Open Knowledge movement should conceive of itself as like unto the resourceful colonial railway companies of old, which opened up vast new territories and connected raw materials with factories.

Project Ronald, named for Ronalds Coase and McDonald, is my effort to connect up all the world’s tabular data. The aim is to enable speedy answers to the question “are these things correlated”, where this involves extracting information from public datasets published by different organisations. I take “speedy” to mean “in a couple of seconds”, and I have created a short video demonstrating the process from when someone selects successful search terms to find a dataset, to being able to do regressions in R, a statistics system; the video is heavily padded to show I’m not cheating.

The overall approach of the project is the New Jersey / UNIX philosophy, of getting a simple architecture of loosely connected components to work together, and worrying about obtaining the best implementation of each component later. Subsequent blog posts will relate the current implementation and future challenges.

If you could bear reading this far, you should follow me on Twitter:

Interdisciplinary humour

Did you hear the one about the Berkeleyan idealist who tried to reduce all of quantum mechanics to mutually recursive lambda expressions?

He literally disappeared up his own arse!

If you could bear reading this far, you should follow me on Twitter: