Updating your hosted feature services with ArcGIS Pro and the ArcGIS API for Python

Many organizations share public maps on ArcGIS online and have defined a process to update and synchronize the feature layers from their local data. One method is well defined using ArcMap; you select the option to overwrite the hosted feature layers when publishing your updates back to ArcGIS online. In 2013 and 2014 we explained how you could update your hosted feature services automatically, at a prescribed time, using Python. In the spirit of this incredibly popular workflow to schedule updates, this blog will provide simpler Python code to update services using ArcGIS Pro 1.4 and the new ArcGIS API for Python.

Publish web map from ArcGIS Pro

This post assumes you’ve already shared a web map from ArcGIS Pro to ArcGIS online. You’ll need to make note of the user name and password of the publisher account, the service name as well as the original project file (.aprx). When publishing a web map from ArcGIS Pro, the underlying feature layers may have _WFL1 appended to the name to ensure they are unique within organization. Investigate your hosted content and be certain of the service definition (.sd) and feature layer names before proceeding.

The first step is to install the ArcGIS API for Python; from within ArcGIS Pro, find and add the arcgis package using the Python Package Manager. Alternatively, open a Python command prompt from: Start > ArcGIS > ArcGIS Pro > Python Command Prompt and use the following command, agreeing to update packages (if necessary):

 conda install -c esri arcgis

After installing the ArcGIS Python API you need to define the steps of your update process. Typically, this will involve one or more authors working on the local datasets that were shared to ArcGIS online. The source documents and data will be used to create a service definition (.sd) file using Python. At a prescribed time, perhaps every night at midnight, the local .sd file with your new data will be uploaded to ArcGIS online where it will be used to replace the hosted service.

Process of updating a web map

Copy the following code and save it locally as a Python file (updatewebmap.py). This code creates the .sd file and upload/publishes it on ArcGIS online. A number of variables will need to be set within the script; the path to the project, service name username, password, metadata and sharing options when you create the file.

import arcpy
import os, sys
from arcgis.gis import GIS

### Start setting variables
# Set the path to the project
prjPath = r"C:\PROJECTS\NightlyUpdates\NightlyUpdates.aprx"

# Update the following variables to match:
#  Feature service/SD name in arcgis.com, user/password of the owner account
sd_fs_name = "MyPublicMap"
portal = "http://www.arcgis.com" # Can also reference a local portal
user = "UserName"
password = "p@sswOrd"

# Set sharing options
shrOrg = True
shrEveryone = False
shrGroups = ""

### End setting variables

# Local paths to create temporary content
relPath = sys.path[0]
sddraft = os.path.join(relPath, "WebUpdate.sddraft")
sd = os.path.join(relPath, "WebUpdate.sd")

# Create a new SDDraft and stage to SD
print("Creating SD file")
arcpy.env.overwriteOutput = True
prj = arcpy.mp.ArcGISProject(prjPath)
mp = prj.listMaps()[0]
arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, sd_fs_name, 'MY_HOSTED_SERVICES', 'FEATURE_ACCESS','', True, True)
arcpy.StageService_server(sddraft, sd)

print("Connecting to {}".format(portal))
gis = GIS(portal, user, password)

# Find the SD, update it, publish /w overwrite and set sharing and metadata
print("Search for original SD on portal...")
sdItem = gis.content.search("{} AND owner:{}".format(sd_fs_name, user), item_type="Service Definition")[0]
print("Found SD: {}, ID: {} \n Uploading and overwriting...".format(sdItem.title, sdItem.id))
sdItem.update(data=sd)
print("Overwriting existing feature service...")
fs = sdItem.publish(overwrite=True)

if shrOrg or shrEveryone or shrGroups:
  print("Setting sharing options...")
  fs.share(org=shrOrg, everyone=shrEveryone, groups=shrGroups)

print("Finished updating: {} - ID: {}".format(fs.title, fs.id))

You can run the code from command line with the following command.

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe c:\mycode\updatewebmap.py

Once you’re sure the script has properly updated your hosted feature service, you can use the command in a scheduled task. This blog has more information about creating a scheduled task.

Important Notes:

  • The Python script needs to be run on a machine with a valid ArcGIS Pro license. The script uses arcpy to create and stage the data to be published.
  • The Python script assumes the first map found within a project will be the one to publish. If you’re working in a project with more than one map, update the Python code or create a new project with only the map you wish to share.
  • The associated service definition (.sd) file must exist on ArcGIS online, along with the hosted feature layers and web map. If either of these items are missing, you’ll need to publish a web map from ArcGIS Pro independent of this script before implementing the update workflow.

The ArcGIS API for Python provides many options and settings for working with your hosted content. Explore the options on publish and content for more fine grain control on your update process.

This entry was posted in Analysis & Geoprocessing, ArcGIS Online, ArcGIS Pro, Portal for ArcGIS, Python. Bookmark the permalink.

Leave a Reply

14 Comments

  1. Is it possible to overwrite a feature service that has offline capabilities and users have offline replicas of that feature service in Collector?

  2. jrflannery says:

    Kevin: My AGOL UN/PW is using enterprise credentials. I am getting the error that you see, below. Is this due to my enterprise login to AGOL? Thank you for sharing your ArcGIS Pro Python script.

    C:\Users\Joe.Flannery>”C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe” M:\Staff\Flannery\ArcGIS_Pro\PublishingScriptTesting\updatewebmap.py
    Creating SD file
    Connecting to http://www.arcgis.com
    Unable to generate token.
    Invalid username or password.
    Traceback (most recent call last):
    File “M:\Staff\Flannery\ArcGIS_Pro\PublishingScriptTesting\updatewebmap.py”, line 47, in
    gis = GIS(portal, user, password)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis.py”, line 93, in __init__
    verify_cert=self._verify_cert)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\portalpy.py”, line 159, in __init__
    verify_cert=verify_cert)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py”, line 274, in __init__
    self.login(username, password, expiration)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py”, line 429, in login
    newtoken = self.generate_token(username, password, expiration)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py”, line 383, in generate_token
    ssl=True, add_token=False)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py”, line 956, in post
    self._handle_json_error(resp_json['error'], errorcode)
    File “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\_impl\connection.py”, line 976, in _handle_json_error
    raise RuntimeError(errormessage)
    RuntimeError: Unable to generate token.
    Invalid username or password.
    (Error Code: 400)

    • Kevin Hibma says:

      Joe – is your org set to use OAUTH? If so, that isn’t supported with the current version of the ArcGIS API for Python. (However, that is something the team is working on). If you’re using IWA/LDAP then it should work.

  3. jrflannery says:

    Kevin – We are using SAML / ADFS. Is that the same as IWA/LDAP?

    James Tedrick’s python code for “Migrating data from the Survey Feature Service to an Enterprise Geodatabase” uses my enterprise login correctly when I open ArcMap and sign into AGOL prior to running the script. For your python script, I had Pro 1.4 open and signed in at the time of running the script.

    Tedrick Code is from here: https://geonet.esri.com/groups/survey123/blog/2016/12/14/migrating-data-from-the-survey-feature-service-to-an-enterprise-geodatabase

    Thank you for your time.
    Joe

    • Kevin Hibma says:

      Joe, the code you reference for the survey feature service – it works because it tests if you’re logged in using the app (arcgis pro/arcmap). If you are, it authenticates that way. Otherwise it would depend on your credentials, and would fail the same way you’re using this script. If you change line 37 to: gis = GIS(‘pro’) and make sure you’re logged into ArcGIS Pro, it’ll work just the same. Thats your only work around until the Python API has been updated to support this authentication method.

  4. whuyp says:

    I got an error when I run the code :
    arcpy.mp.CreateWebLayerSDDraft(mp, sddraft, sd_fs_name, ‘MY_HOSTED_SERVICES’, ‘FEATURE_ACCESS’,”, True, True)

    The error is:
    Runtime error
    Traceback (most recent call last):
    File “”, line 1, in
    File “c:\program files\arcgis\pro\Resources\arcpy\arcpy\utils.py”, line 191, in fn_
    return fn(*args, **kw)
    File “c:\program files\arcgis\pro\Resources\arcpy\arcpy\mp.py”, line 107, in CreateWebLayerSDDraft
    return convertArcObjectToPythonObject(arcgisscripting._mapping.CreateWebLayerSDDraft(*gp_fixargs([map_or_layers, out_sddraft, service_name, server_type, service_type, folder_name, overwrite_existing_service, copy_data_to_server, enable_editing, allow_exporting, enable_sync, summary, tags, description, credits, use_limitations], True)))
    RuntimeError

    I have no idea how to fix this, it says the “convertArcObjectToPythonObject” which I thought doesn’t make sense, do you have some idea about this?

    Thanks,
    Peter

  5. ae2s_gis says:

    Great post! Unfortunately I’m getting the same error as whuyp

  6. forusdomingo1 says:

    Great post!
    Only one question.I copy your post into spyder, I update a Feature Service in my AGO but the table does not get update. What I am missing?
    Thank for your time!!

  7. salvaleonrp says:

    How important is the sd file? If I use Pro to Share a Web Map or Web layer, I typically delete the SD file. In ArcGIS server, I favor publishing from my MXD, instead of using Server Manager and publishing with an SD file. What is best practice in ArcGIS Pro and AndGIS Server and what are the pros and cons of one over the other?

    For the server environment, I have a colleague who keeps his SD file as backup. My backup is my MXD and a copy.

  8. eatoncountyss says:

    This script bombs for me at the end with:

    Creating SD file
    Connecting to http://www.arcgis.com
    Search for original SD on portal…
    Found SD: 417, ID: 700bea4ae0714b8a8119180cce28d8b9
    Uploading and overwriting…
    Overwriting existing feature service…
    User cant overwrite this service, using this data, as this data is already referring to another service….

    Done manually in Pro using “Overwrite Web Layer” it works without issue.

    Any ideas?

    • szelonis_mcpd says:

      I’m getting the same error when I try and update a feature service that has a similar name as another that has been updated more recently. For example… if I run the script to update a feature service called “MCStat_Dash1″ it works fine. When I run the script to update a feature service called “MCStat”, it doesn’t work and I get the same error as the poster above. Instead of finding the service called MCStat, the script for some reason finds the one called MCStat_Dash1 and tries to update it; then throws the error. I’m wondering if it has something to do with the naming??

  9. alex.gole_eldoradocounty says:

    I keep getting this error:
    Traceback (most recent call last):
    File “Z:\Agole\OverwritePython.py”, line 1, in
    import arcpy
    File “C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py”, line 66, in
    from arcpy.geoprocessing import gp
    File “C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\__init__.py”, line 14, in
    from ._base import *
    File “C:\Program Files\ArcGIS\Pro\Resources\ArcPy\arcpy\geoprocessing\_base.py”, line 14, in
    import arcgisscripting
    RuntimeError: Not signed into Portal.

    Any idea why?