Updating ArcGIS.com Hosted Feature Services with Python

Note: A this workflow has been updated for ArcGIS 10.2/10.2.1. Please read that post depending on your version. [January 24, 2014]

More organizations are moving towards using ArcGIS.com hosted feature services to serve data. One common task that has become a popular question of late is, “How do I automatically update the data within this hosted service?” For example, your organization may want to push nightly updates to keep synchronized with the daily changes made by your Desktop users. One of the easiest ways this can be done as this blog describes, is by overwriting the feature service completely with an updated one.

The following Python script demonstrates how to:

To get started, copy and paste the code into a script file (.py) on your machine. Update the variables with your service name, path to the MXD and your ArcGIS.com account information. The script will create temporary drafts in the same location you saved the script to. Read the comments in line to customize the script to your needs. Once your script is configured, run it on any machine with Desktop, Engine or Server installed.

import arcpy, os, sys
import xml.dom.minidom as DOM

arcpy.env.overwriteOutput = True

# Update these variables
# The tempPath variable is a relative path which is the same directory
# this script is saved to. You can modify this value to a path on your
# system to hold the temporary files.
serviceName = "importantPoints"
tempPath = sys.path[0]
path2MXD = r"C:\path2MXD\pts.mxd"
userName = " "
passWord = " "

# All paths are built by joining names to the tempPath
SDdraft = os.path.join(tempPath, "tempdraft.sddraft")
newSDdraft = os.path.join(tempPath, "updatedDraft.sddraft")
SD = os.path.join(tempPath, serviceName + ".sd")

arcpy.SignInToPortal_server(userName, passWord, "http://www.arcgis.com/")
mxd = arcpy.mapping.MapDocument(path2MXD)
arcpy.mapping.CreateMapSDDraft(mxd, SDdraft, serviceName, "MY_HOSTED_SERVICES")

# Read the contents of the original SDDraft into an xml parser
doc = DOM.parse(SDdraft)

# The follow 5 code pieces modify the SDDraft from a new MapService
# with caching capabilities to a FeatureService with Query,Create,
# Update,Delete,Uploads,Editing capabilities. The first two code
# pieces handle overwriting an existing service. The last three pieces
# change Map to Feature Service, disable caching and set appropriate
# capabilities. You can customize the capabilities by removing items.
# Note you cannot disable Query from a Feature Service.
tagsType = doc.getElementsByTagName('Type')
for tagType in tagsType:
    if tagType.parentNode.tagName == 'SVCManifest':
        if tagType.hasChildNodes():
            tagType.firstChild.data = "esriServiceDefinitionType_Replacement"

tagsState = doc.getElementsByTagName('State')
for tagState in tagsState:
    if tagState.parentNode.tagName == 'SVCManifest':
        if tagState.hasChildNodes():
            tagState.firstChild.data = "esriSDState_Published"

# Change service type from map service to feature service
typeNames = doc.getElementsByTagName('TypeName')
for typeName in typeNames:
    if typeName.firstChild.data == "MapServer":
        typeName.firstChild.data = "FeatureServer"

#Turn off caching
configProps = doc.getElementsByTagName('ConfigurationProperties')[0]
propArray = configProps.firstChild
propSets = propArray.childNodes
for propSet in propSets:
    keyValues = propSet.childNodes
    for keyValue in keyValues:
        if keyValue.tagName == 'Key':
            if keyValue.firstChild.data == "isCached":
                keyValue.nextSibling.firstChild.data = "false"

#Turn on feature access capabilities
configProps = doc.getElementsByTagName('Info')[0]
propArray = configProps.firstChild
propSets = propArray.childNodes
for propSet in propSets:
    keyValues = propSet.childNodes
    for keyValue in keyValues:
        if keyValue.tagName == 'Key':
            if keyValue.firstChild.data == "WebCapabilities":
                keyValue.nextSibling.firstChild.data = "Query,Create,Update,Delete,Uploads,Editing"

# Write the new draft to disk
f = open(newSDdraft, 'w')
doc.writexml( f )
f.close()

# Analyze the service
analysis = arcpy.mapping.AnalyzeForSD(newSDdraft)

if analysis['errors'] == {}:
    # Stage the service
    arcpy.StageService_server(newSDdraft, SD)

    # Upload the service. The OVERRIDE_DEFINITION parameter allows you to override the
    # sharing properties set in the service definition with new values. In this case,
    # the feature service will be shared to everyone on ArcGIS.com by specifying the
    # SHARE_ONLINE and PUBLIC parameters. Optionally you can share to specific groups
    # using the last parameter, in_groups.
    arcpy.UploadServiceDefinition_server(SD, "My Hosted Services", serviceName,
                                         "", "", "", "", "OVERRIDE_DEFINITION","SHARE_ONLINE",
                                         "PUBLIC","SHARE_ORGANIZATION", "")

    print "Uploaded and overwrote service"

else:
    # If the sddraft analysis contained errors, display them and quit.
    print analysis['errors']

You can take this one step further by creating a scheduled task to run the code nightly. By having an MXD which references geodatabase layers updated throughout the day, the automated task will push those changes up to ArcGIS.com when the task is run.

We will be updating the CreateMapSDDraft help topic with this example. Check it out for numerous other map publishing examples.

This entry was posted in Analysis & Geoprocessing, ArcGIS Online, Python, Services, Web and tagged , , , , . Bookmark the permalink.

Leave a Reply

25 Comments

  1. tribeiro says:

    Thank you for this.
    I really hope that at 10.2 you add some kind of parameters/options to CreateMapSDDraft so that it creates a SDDraft for feature service without having to take all this steps. Why not make it avaible as tool (like the stage service and upload)?
    Anyway, thank you for publishing this workaround!

  2. cbuscagl says:

    Nice work guys, this helped out quite a bit with something that I was working one recently.

  3. matt wilkie says:

    thank you for this

  4. matt wilkie says:

    UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 414: ordinal not in range(128)

    I had to apply unicode encoding before I could get this to complete successfully with our data. Insert at very top of file:

    # -*- coding: utf-8 -*-
    import codecs

    then change

    # Write the new draft to disk
    f = open(newSDdraft, 'w')

    to

    with codecs.open(newSDdraft, 'w','utf-8') as f:
    doc.writexml(f)
    f.close()

    Please put this (and all) code samples on github so to facilitate contributing patches that fix them in place, plus all the other wholesome goodness that comes from using good source code management tools.

  5. matt wilkie says:

    forgot to say this is where I learned how to fix the encoding error:
    http://stackoverflow.com/questions/4484228/writing-xml-to-file-corrupts-files-in-python

  6. sherwinaquino says:

    I have tried this and It is making a new Feature service instead of updating it. I have modified the required parts of the code but it still makes a new Feature service. I have tried this 3 times on my existing data. Is there another part of the code that I must modify.

  7. ih says:

    Could this script be manipulated to update a .csv in ArcGIS Online?

  8. riverside says:

    It would be nice to have some sort of reference specification for the sd draft online rather than having to deconstruct your own. Also, wow, impressive, amazing stuff, you guys did a great job. This would have made a great tech workshop or just a demo in the demo island for UC.

    Having issues with 10.2 with this code, but really just signing into Portal and Uploading SD. Everything else is great.

    • eendrulat says:

      I’m also having trouble with the Portal Sign in function in 10.2, works great in 10.1 and i’m able to publish when signed in to ArcGIS Online from Desktop. See post on ArcGIS Forums here.

  9. Jeff Moulds says:

    At 10.2, significant enhancements were made to server security. To run this code at 10.2, comment out the arcpy.SignInToPortal_server() line (line 21). At 10.2, sign in information is obtained from the ArcMap File > Sign In dialog. More information can be found here: http://resources.arcgis.com/en/help/main/10.2/index.html#//00sp00000023000000. After signing in through ArcMap, run your code.

    Moreover, at bug was introduced at 10.2 where arcpy.UploadServiceDefinition_server() would occasionally fail when overwriting a hosted feature service. The tracking number for the bug is NIM095154. We are hoping to fix this in the next release (10.2.1).

  10. kiki_242 says:

    This is great! Is there a also a possibilty to turn on “enable attachments” for a feature in a feature service using python code to publish a feature servide?

    Thanks Kerstin

  11. nathanenge says:

    Hi
    I’m using 10.2 and am trying to run the script. I do get an error:
    Traceback (most recent call last):
    File "D:\f\AGOLDEMO_CANFOR\Toolbox.py", line 94, in
    "PUBLIC","SHARE_ORGANIZATION", "")
    File "C:\Program Files (x86)\ArcGIS\Desktop10.2\arcpy\arcpy\server.py", line 1293, in UploadServiceDefinition
    raise e
    ExecuteError: ERROR 001566: Service overwrite error: failed to delete the service.
    ERROR: code:400, Item '89a8828f239144e9923ca277813d3e81' does not exist or is inaccessible., Bad syntax in request.
    Failed to execute (UploadServiceDefinition).

    I’ve checked the ArcGIS Online to see that the item does exist. However when I run the script, it deletes the item, but does not replace it, then I get this error.

    I have commented out the # arcpy.SignInToPortal_server(userName, passWord, “http://www.arcgis.com/”)
    I haven’t touched the UploadServiceDefinition code…
    arcpy.UploadServiceDefinition_server(SD, "My Hosted Services", serviceName,
    "", "", "", "", "OVERRIDE_DEFINITION","SHARE_ONLINE",
    "PUBLIC","SHARE_ORGANIZATION", "")

    print "Uploaded and overwrote service"

  12. dspolans says:

    where can i find the full updated script for 10.2.1?

  13. dspolans says:

    I get an error:
    ERROR 001269: Compressing the service definition failed
    failed to execute

  14. iyarbrough says:

    Does anyone know where I can find (or if I can modify) this script to do the reverse? In most cases, my edits will be made on AGOL through field data collection, so I’m more interested in pulling down edits to update a local gdb.

  15. ldhartman says:

    I’m still using 10.1. Everything in this script seems to work for me with no errors. When I go to AGOL though, the feature service has the current date on it, but the .sd does not. So I’m not getting the updated features in my maps. Any ideas why the .sd is not updating?