“Our state government possesses vast treasure troves of valuable information and reports: from health, business and public safety data to information on parks, recreation, labor, and transportation … The Open New York web portal will allow researchers, citizens, business and the media direct access to high-value data, which will be continually added to and expanded, so these groups can use the data to innovate for the benefit of all New Yorkers.” – Governor Cuomo, State of the State Address, January 9, 2013

I. INTRODUCTION

Open Data – data.ny.gov

Quality by Design

New York State is committed to proactively releasing publishable state data.
New York State is committed to making publishable state data openly and freely available in accessible formats for the public to reuse and consume.
New York State is committed to publishing high quality data with comprehensive metadata and documentation to foster interoperability and maximize citizen understanding of the data.
New York State is committed to ongoing and continuous publication of publishable state data.

Government is the public’s business and the public should have access to the records of government. New technologies have dramatically changed both the way government conducts business and the public’s expectations about access to government information. As part of this transformation, New York State launched Open.ny.gov, on March 11, 2013, dedicated to increasing public access to one of the State’s most valuable assets – data. Its goals are to spark innovation, promote research and economic opportunities, engage public participation in government, increase transparency, and inform decision-making.

Concurrent with the launch of Open NY, Governor Andrew Cuomo issued Executive Order No. 95, “Using Technology to Promote Transparency, Improve Government Performance and Enhance Citizen Engagement.” This unprecedented Executive Order directed covered state entities, for the first time, to identify and catalogue their data, and make publishable state data available on the new transparency website.

The concept of “Open Data” describes data that is freely available, machine readable, and formatted according to uniform technical standards to facilitate visibility and re-use of published data. New York’s open data platform is a quality by design web-based public data portal that catalogues data and enables data to be discoverable. The portal offers access to standardized data that can be easily retrieved, downloaded, sorted, searched, analyzed, and re-used by citizens, business, researchers, journalists, developers, and government to process, trend, and innovate utilizing a singular dataset or combinations of datasets.

OpenNY puts tools for transparency, accountability, and innovation directly into the hands of New Yorkers and people all around the world through a centralized, user-friendly interface. This increased visibility provides derivative value as the public is able to analyze and utilize government data, and better understand what is happening in government on all levels – federal, state, and local.

The Open Data Handbook

This Open Data Handbook is intended as a general guide for government entities participating in data.ny.gov , as well as the general public. The Handbook provides guidelines for identifying, reviewing, and prioritizing publishable state data for publication – with a foundational emphasis on high quality, and metadata and documentation requirements. These guidelines are intended for use by both covered state entities and other government entities not covered by Executive Order 95 (including localities). These guidelines are also intended for use by the public in order to understand how New York State makes its publishable data sets available.

The breadth of data and agency participation are continually being enhanced and expanded on data.ny.gov, making it a dynamic, living platform. This Handbook, providing guidelines for publication of publishable State data onto data.ny.gov, is the first step in a major shift in the way New York State government agencies share information to promote efficiency, accessibility and transparency; a major shift in the way New York State government engages citizens and fosters innovation and discovery in the scientific and business communities. It begins the process of standardizing the state’s data, which will make it easier for both government workers and the public to discover and use the data. This, in turn, advances “interoperability” so the data can be more easily shared and analyzed.

Working in collaboration with state agencies, this Handbook will be supplemented, as needed, with technical and working documents addressing specific formatting, data preparation, and data refresh and data submission requirements. We welcome input from academics, researchers, developers, businesses, entrepreneurs, and the general public to help identify data that would be useful to them and the best ways of providing it.

Timeline

In advancing a transparency agenda, New York State is committed to continuous publication of publishable State data. Data discovery is an iterative and on-going process. The following is a high-level timeline for the implementation of Executive Order 95.

March 11, 2013: Executive Order No. 95 issued
March 11, 2013: Open.ny.gov website launched
April 10, 2013: Agency Data Coordinators Appointed
April 25, 2013: Data Working Group established
June 10, 2013: ITS issues Provisional Open Data Handbook
September 9, 2013: Covered State entities create catalogues of publishable data; covered State entities’ propose schedule to ITS for making publishable state data publicly available.
November 6, 2013: ITS issues Final Open Data Handbook
April 1, 2014: ITS begins publishing quarterly reports
December 15, 2019: Open Data and compliance with Executive Order No. 95 is fully incorporated into state covered entities' on-going core business planning and strategies.

Definitions

Throughout this handbook, some terms are used as defined in Executive Order 95. These definitions are included below.

Covered State Entity:

any State agency or department, or any office, division, bureau, or board of such State agency or department, except where the head of such agency or department is not appointed by the Governor,
any State board, committee, or commission, at least one of whose members is appointed by the Governor, and
all public-benefit corporations, public authorities and commissions, for which the Governor appoints the Chair, the Chief Executive, or the majority of Board Members, except for the Port Authority of New York and New Jersey.

Chief Data Officer (CDO): The New York State Chief Data Officer in the Office of Information Technology Services or a designee thereof.

Data: Final versions of statistical or factual information that:

are in alphanumeric form reflected in a list, table, graph, chart or other non-narrative form, that can be digitally transmitted or processed;
are regularly created or maintained by or on behalf of a covered State entity and are controlled by such entity; and
record a measurement, transaction or determination related to the mission of the covered State entity.

The term “data” shall not include image files, such as designs, drawings, photos or scanned copies of original documents; provided, however, that the term “data” shall include statistical or factual information about image files and geographic information system data.

Data set: A named collection of related records maintained on a storage device, with the collection containing data organized or formatted in a specific or prescribed way, often in tabular form.

ITS: The New York State Office of Information Technology Services.

Publishable State data: Data that is collected by a covered State entity where the entity is permitted, required or able to make the data available to the public, consistent with any and all applicable laws, rules, regulations, ordinances, resolutions, policies or other restrictions, requirements or rights associated with the State data, including but not limited to contractual or other legal orders, restrictions or requirements. Data shall not be Publishable State data if making such data available on the Open Data Website would violate statute or regulation (e.g., disclosure that would constitute an unwarranted invasion of personal privacy), endanger the public health, safety or welfare, hinder the operation of government, including criminal and civil investigations, or impose an undue financial, operational or administrative burden on the covered State entity or State.