Open data in the United Kingdom

Crown Copyright has been a long-standing copyright protection applied to official works, and at times artistic works, produced under royal or official supervision. In 2006, The Guardian newspaper's Technology section began a "Free Our Data" campaign, calling for data gathered by authorities at public expense to be made freely available for reuse by individuals. In 2010 with the creation of the Open Government Licence and the Data.gov.uk site it appeared that the campaign had been mostly successful, and since 2013 the UK has been consistently named one of the leaders in the open data space.^[1][2][3]

Despite the UK's leadership position in the open data space, many issues (such as uneven data quality and data literacy) still need to be addressed, but as of 2019 progress has been slow.^[4][5] Some of these issues are addressed in the UK Government's 2017–2020 Government Transformation Strategy.^[6]

In 2010 the UK Government created the Open Government Licence, and public bodies can now opt to publish their Crown Copyright material under this licence.^[7] Material marked in this way is available under a free, perpetual licence without restrictions beyond attribution. This new licence was based on, and designed to work with the Creative Commons licences.^[8] Version 2.0 of the licence was released in June 2013 and it was accompanied by a new logo which "at a glance, shows that information can be used and re-used under open licensing".^[9]

Crown Copyright is the default copyright applied to all government department published documents.^[10]

The Met Office is the national weather service. Its main role is to produce forecast models by gathering all the information from weather satellites in space and observations on earth. The principal weather products for UK customers are 36-hour forecasts from the newly-operational 1.5 km resolution UKV model covering the UK and surroundings (replacing the 4 km model), 48-hour forecasts from the 12 km resolution NAE model covering Europe and the North Atlantic, and 144-hour forecasts from the 25 km resolution global model (replacing the 40 km global model).^[11] A wide range of other products for other regions of the globe are sold to customers abroad, provided for MOD operations abroad or provided free to developing countries in Africa. This main bulk of data are then passed on to companies who acquire it. Data are stored in the Met Office's own PP-format. The Met Office held the base data involved in the Climatic Research Unit hacking incident and released information to the public alongside the Climatic Research Unit when pressed.^[12]

Ordnance Survey has been subject to criticisms. Most criticism centres on the point that Ordnance Survey possesses a virtual government monopoly on geographic data in the UK,^[13] while, although a government agency, since 1999 it has been required to act as a Trading Fund or commercial entity. This means that it is supposed to be totally self-funding from the commercial sale of its data and derived products – whilst at the same time it is supposed to be the public supplier of geographical information. In 1985 the "Committee of Enquiry into the Handling of Geographic Information" was set up in order to "advise the Secretary of State for the Environment within two years on the future handling of geographic information in the UK, taking account of modern developments in information technology and market needs". The committee's final report was published under the name of its chairman, Roger Chorley, in 1987.^[14] The report stressed the importance of widely available geographic information to the UK and recommended a loosening of government policies on distribution and cost recovery.

Ordnance Survey historical works are generally available, as the agency is covered by Crown Copyright: works more than fifty years old, including historic surveys of Britain and Ireland and much of the New Popular Edition, are in the public domain. However, finding suitable originals remains an issue as Ordnance Survey does not provide historical mapping on 'free' terms, instead marketing commercially 'enhanced' reproductions in partnership with Landmark Information Group.^[15] This can be contrasted with, for example, the approach in the Republic of Ireland in more recent times, where Ordnance Survey Ireland claims regular copyright over its mapping (and over digital copies of the public domain historical mapping).

Transport Direct is a division of the UK Department for Transport to develop better information technology systems to support public transport. Traveline has created a comprehensive database of all public transport movements in the United Kingdom and it is used to power the Transport Direct Portal. Since the launch of the Portal in December 2004 there have been growing calls for the public transport schedule data, which is generally owned by the transport operator, to be made available as Open data.

The development of Traveline and Transport Direct has created a number of comprehensive national datasets to support the Transport Direct Portal. The assessment of accessibility to services by public transport was made a requirement for the Local Transport Plans in 2004. Measuring the level of accessibility required the use of the data collected by Traveline.^[16]

This data are being used by an increasing number of 3rd parties, including Google Transit (South East Traveline Region)^[17] and East Anglia / East Midlands Traveline Regions.^[18] mySociety's Mapumental accessibility product.^[19] During 2010 increasing amounts of data has become available as Open Data including NaPTAN and NPTDR.

Environment Agency provides Open data are through its Spatial Data Catalogue. Lidar and other survey datasets are also available through its Survey Open Data web portal.

The UK Hydrographic Office is another data holder in the UK who has been listed by Free Our Data as a government office which is charging for the use of its data that Free Our Data believes should be opened for public use.^[20] The UK Hydrographic Office has data on tidal patterns and the position of astronomical bodies used in navigation, as well as being the primary holder of data for nautical charts, not just in the UK but also in many foreign locations, covering features such as bathymetry (depth), wrecks, underwater cables and pipelines, navigation buoys and lighthouses, and coastal features of interest to the mariner. Some of the hydrographic survey data is now being made available through the MEDIN portal as a result of the EU's INSPIRE initiative.

Before publicly available data can be analyzed, it needs to be downloaded and imported into a data analysis tool. Furthermore, an analyst will usually need to combine multiple datasets in order to generate a useful analysis. This can be time-consuming if the datasets do not already come in a well-structured machine-readable format, or if they need to be sourced across different organizations. Since this problem is subject to active work in multiple open-data-related projects, UK Government data can be found on platforms other than data.gov.uk, sometimes in a better shape, where users can download it alongside data from other sources. For example, UK Data Service addresses the needs of research users by offering surveys, studies and other relevant data, including surveys and research provided by UK government departments.^[21] As another example, Datahub Core^[22] offers pre-cleaned small-scale reference and indicator data, including data originally published by the Office for National Statistics and others.

The UK Government itself has recently started addressing this problem by making well-formatted, cleaned datasets available through its data registers service.

In March 2006 The Guardian newspaper's Technology section began a "Free Our Data" campaign, calling for the raw data gathered by Ordnance Survey (not to mention data gathered on its behalf by local authorities at public expense) to be made freely available for reuse by individuals and companies, as happens, for example, with such data in the USA. However, Ordnance Survey claimed that the quality of the freely available government mapping in the USA is low or variable because of funding issues.^[23]

On the 7 April 2006 the Office of Public Sector Information (OPSI) received a complaint from the data management company Intelligent Addressing.^[24] Many, although not all, complaints were upheld by the OPSI, one of the conclusions being that Ordnance Survey "is offering licence terms which unnecessarily restrict competition". Negotiations between Ordnance Survey and interested parties are ongoing with regard to the issues raised by the OPSI report, Ordnance Survey being under no obligation to comply with the report's recommendations.

In June 2006 the Free our data campaign asked "In an age when it is not seen as appropriate for the public sector to run power stations or railways, why is it running nationalised industries in what should be the most dynamic sector of all, the web-based knowledge economy?" And said that "The question lies at the heart of our campaign, which argues that government's role should be to collect and administer high-quality raw data, but make it freely available to everyone to create innovative services". Transport Direct's chief executive, Nick Ilsley, said that research by the department prior to Transport Direct's launch had showed the private sector was not interested in providing a one-stop all-purpose site.^[25]

In April 2007, Cabinet Office Minister Hilary Armstrong commissioned Ed Mayo and MySociety director Tom Steinberg to draft a "Power of Information Review" on how the government could serve the public's information needs better.^[26] The resulting report led Cabinet Minister Tom Watson, MP to create a 'Power of Information Task Force.^[27]

Since August 2007, Ordnance Survey has contracted the political lobbying company Mandate Communications^[28] to help campaign against the free data movement and discover which politicians and advisers support Open Data or have questioned Ordnance Survey policies.^[29]

In 2008 MySociety requested to OPSI that the dataset should be made available for re-use as Open Data rather than at a charge of £7,500.^[30]

In April 2008 Tom Watson MP, Parliamentary Secretary at the Cabinet Office said that pledged to turn upside down the British public's "recipient relationship" with official information.^[31]

The 'Power of Information Taskforce Report' was published by the Cabinet Office in February 2009.^[32] It reported that although the National Public Transport Data Repository, described itself as ‘Crown Copyright’ investigations showed this database is not actually government data and that the NPTDR charges significant fees for use. The report suggested that there should be a presumption in favour of information which has been created by public sector bodies being available for re-use. They also suggested that there should be a clear and consistent copyright and licensing rules applied making it easy to work with data from multiple sources in the public sector. They recommended a 'Crown Commons' style approach with "highly permissive licensing scheme that is transparent, easy to understand and easy to use, modelled on the 'Click Use' licence".^[33]

Also in February 2009 Tom Watson MP from the Cabinet Office advised that the UK government would put Open-source software on equal footing for procurement; stating that "Procurement decisions will be made on the basis on the best value for money solution to the business requirement, taking account of total lifetime cost of ownership of the solution, including exit and transition costs, after ensuring that solutions fulfil minimum and essential capability, security, scalability, transferability, support and manageability requirements." and "Where there is no significant overall cost difference between open and non-open source products, open source will be selected on the basis of its additional inherent flexibility".^[34]

In August 2009 Tom Watson MP mused on his website under the title "Wacky terms and conditions" about why the DirectGov version of the transport direct portal included this condition: "you may retrieve and display Content from the Website on an electronic visual display device, print individual pages on paper (but not photocopy them)".^[35]

In September 2009 Sir Tim Berners Lee and Professor Nigel Shadbolt who had been appointed by Gordon Brown to advise the Government on how to make data more accessible to the public were invited to Number 10 Downing Street to present their early findings to the cabinet.^[36]

In October 2009 Tom Watson MP questioned why railway timetables were the train operators' private intellectual property. He asked Junior transport minister Chris Mole: "Does not my hon. Friend think that the timetable data belong to the people, and that we should make them available for free?".^[37] He also proposed an Early Day Motion that read "That this House calls on the Government to ensure that transport timetables for rail and bus operators are made freely available for re-use, using an open standards format, thus enabling voluntary and commercial organisations to publish the data on their own digital platforms". It was signed by 32 MPs.^[38]

In November 2009 the then Prime Minister, Gordon Brown announced that "mid-scale" data from Ordnance Survey would be made available for free reuse, including commercial applications,^[39][40] from April 2010. A consultation was held inviting responses between 23 December and 17 March by the Department of Communities and Local Government.

In December 2009 Local Transport Today reported that 'Data release could spark transport IT innovations". The magazine reported that "Among the data to be made accessible through the site are trunk road traffic volumes, the National Public Transport Access Node (NaPTAN) database and the National Public Transport Data Repository (NPTDR)". A DfT spokeswoman was reported to say: "We will seek to embrace new technology to enable partners to provide exciting, user-focused services such as recently seen with satellite navigation and iPhone applications,"^[41]

In January 2010 a new government web-service, data.gov.uk was unveiled which would "offer reams of public sector data, ranging from traffic statistics to crime figures, for private or commercial use". The aim of the project was to "kickstart a new wave of services that find novel ways to make use of the information".^[42] The US had recently launched data.gov^[43] and London Mayor Boris Johnson had created the London Data Store the same month.^[44]

In March 2010 the Prime Minister Gordon Brown announced that the NaPTAN dataset was to be immediately made available from the data.gov.uk site as Open Data, as well as confirming an impending release of Ordnance Survey data. In his speech he observed that at present public transport timetables and real-time running information was owned by the operating companies but that the government would work to free it up. He also said that "from today we will make it a condition of future franchises that this data will be made freely available".^[45]

Also in March 2010 a Freedom of Information request was made to the Department for Transport inquiring about arrangements that had been made for the release of cycling data that was being collected for the portal at public expense. It requested information about any claims the Ordnance Survey would have over the data and if so the terms, and prices charged for making the data available to Ordnance Survey customers. Also the cost to a Local Authority for adding their area to the planner and for notes of any meetings of project boards and working parties.^[46] A response was published on 1 April 2010.^[47]

On 1 April 2010 Ordnance Survey released^[48] the brand OS OpenData, under an attribution-only licence compatible with CC-by.^[49] Various groups and individuals had campaigned for this release of data, but some were disappointed when some of the profitable datasets were not included – withheld for the counter-argument that if licensees do not pay for OS data collection then the government would have to be willing to foot a £30m p.a. bill, to obtain the future economic benefit of sharing the highly detailed mapping produced by the UK's national agency.^[50] The files have been cached on the MySociety website for convenient download.^[51] They can also be found on the OS website alongside their commercial data.^[52]

June 2010 saw Transport for London release a substantial amount of its data for free reuse, even in commercial settings. This did not include the roundel or Underground schedules, though the Underground data was released in the following months. Following the release of this data numerous "apps" for phones and online facilities appeared that made use of it.^[53]

On 12 January 2011 the Coalition Government revealed that it was planning to establish a Public Data Corporation (PDC). The goal being to make the UK Government data provided in a much more consistent fashion as well as freeing more data for public and commercial use.^[54] The idea of the PDC was later dropped in favour of grouping a number of government data providing organisations to form the Public Data Group.^[55]

While the UK didn't have a predefined process for keeping data it publishes up to date as late as 2017, at the end of 2017 such a process was set.^[5]

In 2018, the responsibility for data policy, including open data, has been moved from the Government Digital Service (GDS) to the Department for Digital, Culture, Media and Sport.^[56]

• Open Data Institute
• MySociety
• Digital 9
• Open data in the United States

• Data.gov.uk - The UK national data store
• Copyright status of work by the U.S. government, by contrast, most works of the United States government are legally considered to be in the public domain
• Free Our Data - Guardian Campaign
• Ordnance Survey
• Open Rights Group Open Data wiki page
• Inventories of open data for UK local government authorities
• Inventory of open research data in the UK in the Registry of Research Data Repositories