Preview of Open Data

Open Data Badge

Location provides a common context through which we can compare different domains of data in order to understand complex relationships. Government agencies gather this data as part of their mission and daily operations – but no one cares as much about their neighborhood as the people who live there. By making this data available to the public, we have a unique opportunity to leverage the local expertise to understand, analyze, and act on the issues that affect communities.

Growing over the past 7 years, open data has become a goal with increasing support and validation. Most recently the US Congress has passed laws directing the government to standardize and publish data for increased transparency. Internationally there is also increasing government adoption.

Nuances of Open Data

There are common principles and patterns that have been developed by the community to clarify the important aspects of open data. Accessible, Complete, Authoritative, and Timely are a few of these common principles.

While there is a tremendous opportunity, there are still many challenges for government agencies to sustainably and effectively make their data accessible to the public. Historically these sites are custom built, require burdensome data migration and subsequently have a limited lifetime resulting in lack of utilization and validation.

Preview of ArcGIS Open Data

Our community works across nearly all levels of government, edges of the globe and domains of knowledge. They currently leverage ArcGIS to manage, update and publish their authoritative data both internally as well as to the web. In a recent discussion with a local government, they pointed out that 70% of their open data catalog came directly from the GIS department. So why should they have to export and migrate their data in order to make it available to citizens?

Instead, we considered how to keep data at its source, within existing workflows and make it easy – indeed transparent – to make this data open and accessible. This is required if we are to accomplish the principles of open data where the data are authoritative and also up-to-date. By connecting directly to the source we ensure that the open data continues just to be the data, made open.

Today at the FedGIS conference we previewed Esri’s new open data initiative. We are adding capabilities to ArcGIS Online that enable any organization to quickly configure a custom view of their public items and open data groups. We built on the existing web services hosted online, on-premise, or through other data services to provide an easy to use interface that is designed for the open data community. Open Data has been part of Esri for years – this initiative means to make it easier and focused on the community.

Based on our experience, both as citizens, and open data developers, we focused on three key capabilities:

  • Discoverable users need simple search and recommendations for finding and following relevant data
  • Explorable right away people should be able to see a linked map and table view with the ability to filter geographically or by any attribute
  • Accessible one-click download as common data formats and developer API’s

These capabilities are coming this spring and will be included for ArcGIS Online. You can start today by publishing your web services as public items and creating open data groups in preparation for launching your own open data sites.

ArcGIS Open Data

Future of Open Data

These new capabilities aim to remove, or at least mitigate, the technology question of open data publishing. Data owners can leverage their existing infrastructure and web services without having to migrate and update data. This makes initiatives far more sustainable.

However the policy and processes remain difficult questions to resolve. Many agencies are still dealing with historic policies of cost recovery or concerns of releasing their data for public consumption. These are improving, and as more government datasets come online the validation and examples will accelerate adoption.

We believe that through this initiative and working with you that open data will increase by an order of magnitude, dramatically improving citizen engagement and government collaboration.

This entry was posted in ArcGIS Online, Geodata, Web and tagged . Bookmark the permalink.

Leave a Reply

9 Comments

  1. rgrichards says:

    The open data concept and portal should have been the focus of AGOL since its exception several years ago. This effort is exciting but is a bit too late for municipalities already on this path. Other vendors are further ahead in the delivery of open data portals and include charts, tables and aggregations. Maps although a critical information component most users will gravitate towards table and chart data deliveries.

    Nothing in the article mentions tables or charts so the new functionality will likely be missing and a shortfall compared to other open data vendors.

  2. dewright_ca says:

    If you really want to make this successful as well as used by bodies to make data open and public then you need to include it in all levels of ArcGIS Server at NO ADDITIONAL COST! To expect many organisations to pay for tools to share data free and openly is not realistic.

    As rgrichards says many bodies have had to go there own way to implement this kind of functionality that ESRI wasn’t supporting fully. Examples such as the horribly broken Metadata tools at 10.0 meant we had to go our own directions. So expecting us to pay for functionality that you want your users to use to replace exiting workflows is not realistic.

  3. Andrew Turner says:

    thanks for the input. These capabilities are included with ArcGIS Online organizations and have no additional cost or credit usage.

    @rgrichards you should see in the screenshot there is in fact table support. It is automatically generated, dynamic based on filters and interactive with the map.

    We have a full roadmap planned such as charts, and I agree with the concept that usable tools need to be close to the data to make it easy for users to work directly with the information. We wanted to first focus on make the data accessible in formats so that people can quickly fulfill their mandates and keep data up to date. The best part of a web platform is that we can progressively add capabilities based on user feedback and continually roll out new capabilities. Please share specifics you would like to see on charts and aggregations.
    So specifically – what kinds of charts do you think are the most useful? How much control is sufficient in styling without being too complex? There are tremendous chart driven tools out there people can use for customization, but it would be good to provide lightweight options.

    • ashtonj says:

      Government and Open data – I was not at the ESRI FED conference, so I base these comments on screen shots and existing challenges with ESRI products, and Open data. I am a big fan of Commercial Off the Shelf (COTS) solutions. I also think SAAS has great advantages for organizations with challenging IT environments. However something is in-congruent with ESRI’s solutions to open data. Most governments are mandated to share via accessibility, metadata, and interoperability standards such as:

      • Accessibility – (W3C, WCAG 2.0),
      • Metadata, (ISO, NAP, CSW),
      • Provision of interoperable web services (WMS, WFS, WMTS).

      To utilize ESRI products to be “open” we have to:
      1. Customize the interface with the JSAPI to be conformant with accessibility
      2. Install Geoportal Server because ArcGIS Online does not understand or accommodate ISO NAP or CSW
      3. Use ArcGIS Server because you cannot serve WMS, WFS or WMTS from ArcGIS Online

      Government uptake would be easier if ESRI had a unified approach to products, formats and interoperability. This latest release is just another variable confusing group of products we already have to implement and customize to present data as open and as compliant. Linking this to AGOL and SAAS makes uptake “Off the Shelf” impossible and just sends people to the APIs and custom work integrating many ESRI products. A better way would be to deliver a ready-made solution that was compliant with many of the basic rules of being “open” in the federal government . (List above)

      Maybe we don’t have the mandates and policy right, or they evolving quicker than we can react… ISO, NAP, CSW are all metadata driven, and metadata portal centric, which does not seem to be working in many places. I will be interested to read the architecture and kick the tires on this latest “ArcGIS for Open Data”. We know that metadata portals are not the best for search engine integration, and maybe standards like WMS WFS WMTS are all on the up with OGC but other formats like GeoJSON are emerging as interoperable…

      • lsc4dev says:

        I really like the initiative and am convinced that the ArcGIS platform has a lot to offer in this context. One comment on interoperability. The support of OGC standards plays a big role here in Germany. I agree with @ashtonj, that other technologies like GeoJSON can play a supporting role, but at the end a lot of German authorities will require at least the support of WMS, WFS and CSW.

  4. nrschlj2 says:

    ashtonj has touched on what I think has been a major shorcoming of AGSO and one of the reasons I am not a fan – METADATA. Why is there no way to expose the metadata that NEEDS to accompany any and all AUTHORITATIVE data? As a GISP I can not in good conscience use or recommend AGSO because of this. Seems like this new initiative offers a perfect opportunity to address this.

  5. rsmandell says:

    We’re a communicable diseases/sexually transmitted diseases/public health medical practice. Just read this email from ESRI concerning “open sourced” data now (soon) available on the web from various (all?) ESRI government customers. Sounds a bit far fetched, since our experience getting MAPPED PUBLIC HEALTH/COMMUNICABLE DISEASE IMPACT DATA from various government agencies–in map-with-data formats–has been frustrating and dismal.
    What’s all this hullaballoo about “open source” U.S. GOVERNMENT data available at some website by just turning on a spigot??!!
    A few months back SAMSHA (Substance Abuse and Mental Health Services Administration) a sub- sub- entity of CMS which is a sub- sub- entity of U.S.Dept. of Health, ran a webinar on it’s NEW NEW ATLAS. The link they sent us didn’t work, and the work they touted (while possibly of use) disappeared into an electronic filing cabinet somewhere in a web cavern. Ain’t seen their ATLAS YET!!
    Can any brain tell me where’s the promised land website put up by ESRI and their Government Customers where you can plug in a PLAIN LANGUAGE OR SQL SEARCH TERM AND OUTPUT A MAP (OR LIST OF MAPS/RESOURCES) WHICH LEADS YOU ON TO THE PROMISED LAND.
    Sounds like a “nice idea” to me–without much buiilt-up brickwork to support it.

    Any advice here? Or email me direct at: electronystagmography@Yahoo.com (don’t ask what it is. just be happy you don’t need it).

  6. shaunrconway says:

    Is there an update on when this will be available? We are in the planning stages of producing an Open Data portal/page for our locality, and would like to consider esri’s Open Data solution as an option.