Support for OGC GeoPackage specification in ArcGIS.

Have you heard about the OGC GeoPackage specification (http://www.opengeospatial.org/standards/geopackage)?  It is a newly minted OGC spec that defines GeoPackages for exchange and GeoPackage SQLite Extensions for direct use of vector geospatial features and/or tile matrix sets. Esri has actively participated in the spec activity from the very beginning. To this end, we were one of the very early adopters of the specification ( early support even before the spec was approved by OGC membership).

If you are curious about GeoPackages, here’s what you can do. At 10.2.1 or with 10.2.2 ArcGIS desktop, you can create an empty GeoPackage and populate the GeoPackage by copying feature data into it. At 10.2.1, we supported the draft version of the specification and at 10.2.2, the final version of the spec is supported. Currently we support only vector features, but with 10.3 we expect to extend support for raster tiles. One of the primary uses cases driving GeoPackage use is mobile support. Expect to see support for GeoPackage in runtime later this year.

So if you are a sqllite database aficionado and would like to test the waters with GeoPackage, here’s what you can do today with 10.2.1 or 10.2.2. You can use the included script to create a sample empty GeoPackage and then populate it with vector features. Use this GeoPackage as you would any other dataset. We have noticed that in some cases when navigating to a directory that contains GeoPackage (.gpkg) data, ArcCatalog/ArcMap does not display the file. Please review this KB article if you run into this issue. http://support.esri.com/en/knowledgebase/techarticles/detail/42348

Lance Shipman on the database team has been actively involved with this effort from the very beginning. Lance and I would welcome your feedback, as we at Esri continue to improve and extend GeoPackage support in 10.3.

Sample python script to create a GeoPackage.

import arcpy

# Set local variables
sqlite_database_path = ‘C:\data\example.gpkg’

# Execute CreateSQLiteDatabase
arcpy.gp.CreateSQLiteDatabase(sqlite_database_path, “GEOPACKAGE”)

This entry was posted in Analysis & Geoprocessing, Defense, Developer, Geodata, Mapping, Open Data, Python and tagged , , , . Bookmark the permalink.

Leave a Reply

6 Comments

  1. rsarwas says:

    Use this GeoPackage as you would any other dataset.
    I’m using Desktop 10.2.1, and I’ve successfully created a GeoPackage and imported data (very cool!), but I cannot edit this data in ArcMap – The workspace containing this data cannot be edited. Is editing supported? Will it be supported in 10.2.2, or runtime?

    • lshipman says:

      Currently in ArcGIS Desktop, GeoPackage editing is supported via code. You can open an update or insert cursor and update data using ArcObjects. Edit sessions will be supported in a later release.

      Runtime will support GeoPackaging editing as part of the summer release.

  2. hawatson says:

    I’ve been able to use your code to create a new GeoPackage, and after removing my ArcCatalog cache files I can see it in ArcCatalog.

    Can you tell us what feature types are supported for copying to the new GeoPackage Geodatabase? Using default inputs to arcpy.ListFeatureClasses() I may get any of these below returned. I’m assuming only simple feature classes, but if you can clarify what is accepted, that would be great.

    Annotation —Only annotation feature classes are returned.
    Arc —Only arc (or line) feature classes are returned.
    Dimension —Only dimension feature classes are returned.
    Edge —Only edge feature classes are returned.
    Junction —Only junction feature classes are returned.
    Label — Only label feature classes are returned.
    Line —Only line (or arc) feature classes are returned.
    Multipatch —Only multipatch feature classes are returned.
    Node —Only node feature classes are returned.
    Point —Only point feature classes are returned.
    Polygon —Only polygon feature classes are returned.
    Polyline —Only line (or arc) feature classes are returned.
    Region —Only region feature classes are returned.
    Route —Only route feature classes are returned.
    Tic —Only tic feature classes are returned.
    All — All datasets in the workspace. This is the default value.

  3. kimo says:

    I have had great success in loading data into sqlite directly with a Python script. It is very fast and more flexible than a file geodatabase. For example one cursor will handle all the tables in a single database, no need to have a cursor per table. It also accepts compatible field types like a spreadsheet eg a floating point into a nominal integer field.
    Here is my script

    #-------------------------------------------------------------------------------
    # Name: GroupToSqlite.py
    # Purpose: Load Stats Group CSV dump into sqlite
    # groups: Dwelling, Family, Household,
    # Individual-part-1,2,3a,3b
    # Author: kimo
    #
    # Created: 05/04/2014, tidy up 24/04/2014
    # Copyright: (c) Kim Ollivier 2014
    # Licence: Creative Commons 3.0 NZ
    #-------------------------------------------------------------------------------
    import sqlite3
    import os, sys, csv, re
    import datetime

    def load_metadata(metaTab,lstItem):
    '''Take the first line of the CSV
    write out original field names to metadata file
    for later reference by column number
    '''
    print '{} {} fields (zero based)'.format(metaTab,len(lstItem))
    c = conn.cursor()
    c.execute('''DROP TABLE IF EXISTS '''+ metaTab)
    c.execute('''CREATE TABLE '''+metaTab+''' (
    colID text NOT NULL,
    colValue text NOT NULL
    )''')
    cmd = "INSERT INTO "+metaTab+"(colID,colValue) VALUES (?,?)"
    n = 0
    for fld in lstItem:
    trow = ('col'+str(n),unicode(fld,errors='ignore'))
    c.execute(cmd,trow)
    # print trow
    n+=1
    conn.commit()
    c.close()
    # print metaTab,"created"
    return

    def create_dwelling_table(conn,templateTable):
    '''
    Create a table from scratch better than using
    a template, according to manual
    '''
    for lstItem in csv.reader(open(fileCSV, "rb")):
    break
    n = len(lstItem)
    # print n,"fields, zero based"
    c = conn.cursor()
    c.execute('''DROP TABLE IF EXISTS '''+ templateTable)
    # floating point will be accepted into integer fields yay
    # because Dwellings have some mean values
    # might cause a problem later with other databases
    lstCol = []
    for id in range(3,len(lstItem)):
    fn = 'col'+str(id)
    lstCol.append(" {} integer,\n".format(fn))
    schemaCol = ' '.join(lstCol)[0:-2]
    c.execute('''CREATE TABLE '''+templateTable+''' (
    col0 text NOT NULL,
    col1 text NOT NULL,
    col2 text ,'''+
    schemaCol+')')
    conn.commit()
    c.close()
    return

    def create_comment_table(conn,commentTable):
    # make up a comment template
    c = conn.cursor()
    c.execute('''DROP TABLE IF EXISTS '''+ commentTable)
    c.execute('''CREATE TABLE '''+commentTable+''' (
    col0 text NOT NULL,
    col1 text NOT NULL,
    col2 text )''')
    conn.commit()
    c.close()
    return

    def insert_record(outTab,lstItem):
    '''load any table
    requires a database to be connected
    and a cursor to be open
    but the cool thing is the cursor is for all tables'''
    buf = [None if i in ["..C","*"] else i for i in lstItem]
    try:
    cmd = "INSERT INTO " + outTab + " VALUES (" + (",?"*(len(lstItem)))[1:] + ")"
    # implied transaction opened
    c.execute(cmd,buf)
    except Exception,msg:
    print msg
    print buf
    print cmd
    conn.rollback()
    conn.close()
    sys.exit()
    return

    def insert_comment_row(outTab,row):
    '''load comment table
    requires a database to be connected
    and a cursor to be open
    but the cool thing is the cursor is for any tables'''
    try:
    cmd = "INSERT INTO " + outTab + " VALUES (" + (",?"*(len(row)))[1:] + ")"
    # implied transaction opened
    c.execute(cmd,row)
    except Exception,msg:
    print msg
    print row
    print cmd
    conn.rollback()
    conn.close()
    sys.exit()

    return
    # ========================== main ===================================
    if __name__ == '__main__':
    try:
    folder = sys.argv[1]
    group = sys.argv[2]
    except :
    folder = "F:/data/census2013"
    group = "Dwelling"
    dGroup = {
    "Dwelling":"Dwelling",
    "Family":"Family",
    "Household":"Household",
    "IndividualP1":"Individual-part-1",
    "IndividualP2":"Individual-part-2",
    "IndividualP3a":"Individual-part-3a",
    "IndividualP3b":"Individual-part-3b"
    }
    fileCSV = folder+"/2013-mb-dataset-Total-New-Zealand-"+dGroup[group]+".csv"

    start = datetime.datetime.now()
    os.chdir(folder)
    if not os.path.exists(fileCSV):
    print "Source not found",fileCSV
    sys.exit()
    conn = sqlite3.connect(group+'.sqlite')
    print group,"sqlite database successfully opened"
    # regular expressions to separate groups, note included space in expression
    # precompiled patterns for reliable separation
    p = {}
    p['MB'] = re.compile("^MB ") # for 2 digit MB code at start of a line
    p['AU'] = re.compile("^[0-9]{6} ") # for 6 digit AU at start of a line
    p['WA'] = re.compile("^[0-9]{5} ") # for 5 digit WARD at start of a line
    p['TLA'] = re.compile("^[0-9]{3} ") # for 3 digit TLA at start of a line
    p['REG'] = re.compile("^[0-9]{2} ") # for 2 digit REG at start of a line
    p['CMB'] = re.compile("^CMB ") # for CMB flag at start of a line
    p['Total'] = re.compile("^Total ") # for Total at start of a line
    p['Comment'] = re.compile("^Footnotes|Symbols|Source") # comments at end
    r = 0
    e = 0
    # create separate output table for each grouping
    lstTab = ['MB','AU','WA','REG','TLA','CMB','Total']
    for tab in lstTab:
    outTab = group+tab
    create_dwelling_table(conn,outTab)
    create_comment_table(conn,"Comment"+outTab)
    # load all records in one pass into respective group table
    c = conn.cursor() # cursor applies to ALL tables!
    for lstItem in csv.reader(open(fileCSV, "rb")):
    r+=1
    if r == 1:
    # write header to Metadata table
    load_metadata(group+'Metadata',lstItem)
    else :
    # group filter for each record type
    if p['MB'].match(lstItem[0]):
    insert_record(group+'MB',lstItem)
    elif p['AU'].match(lstItem[0]):
    insert_record(group+'AU',lstItem)
    elif p['WA'].match(lstItem[0]):
    insert_record(group+'WA',lstItem)
    elif p['TLA'].match(lstItem[0]):
    insert_record(group+'TLA',lstItem)
    elif p['REG'].match(lstItem[0]):
    insert_record(group+'REG',lstItem)
    elif p['CMB'].match(lstItem[0]):
    insert_record(group+'CMB',lstItem)
    elif p['Total'].match(lstItem[0]):
    insert_record(group+'Total',lstItem)
    elif p['Comment'].match(lstItem[0]):
    row = []
    # oops, one strange char corruption in one of the comments
    # but no 255 char field width limits in sqlite
    row.append(unicode(lstItem[0],errors='ignore'))
    row.append(unicode(lstItem[1],errors='ignore'))
    row.append(unicode(lstItem[2],errors='ignore'))
    insert_comment_row(group+'Comment',row)
    else :
    e+=1
    # save inserts, close cursor, close database
    conn.commit()
    c.close()
    conn.close()
    report2 = '{} total rows {} skipped'.format(r,e)
    print report2
    print "Elapsed",datetime.datetime.now() - start
    print "Well Done"

    • kimo says:

      Aagh, even though I wrapped the script in code tags the formatter dropped the indents.
      Never mind, I will post a link when I publish my toolbox next week.