What’s new with hosted feature layers: Getting to know Append Part 1 (December 2017)

Managing your hosted feature layers is an important aspect of managing data in ArcGIS Online.  For many, when it comes to adding new data in bulk to existing hosted feature layers, the typical workflow is: download a copy of the data, reconcile the new additions (and any additional edits), and overwrite the layer in ArcGIS Online. In many cases that workflow works; for others, it creates extra steps for an otherwise simple task.

One of the best examples is when working with data that has periodic updates. That’s where the new Append Data to Layer tool (Append) really shines.  With Append, you can simply insert the new data into the layer instead of exporting the layer, merging it with the new data, then overwriting.

Append allows you to upload and append new data to an existing layer from a file.  That file can be a feature class in a geodatabase, a shapefile, or in the case of point layers; CSV(be sure to include x,y fields in lat/long format; don’t worry, we’ll be working toward other coordinate formats in a future release).   Another scenario where it would make sense to append vs overwrite is when your feature layer is large.  Overwriting the entire layer when it’s very large (say millions of features) can take time.  If the number of features you want to append into the layer is proportionally smaller, it is much faster to use Append instead of overwriting.

To get started, go to the item page for your hosted feature layer.  From the Overview tab, you’ll notice a new menu button on the right called Update Data. Click it and you’ll find.  Append is also accessible from the Data tab.

After clicking Append Data to Layer, you’ll be prompted to upload the file that contains the data you want to append.  Supported formats include:

  • File Geodatabase (.zip)
  • Shapefile (.zip)
  • CSV (.csv)

One quick note at this time you can’t append data to a hosted feature layer that has synchronization or change tracking (Keep track of created and updated features) enabled.

To save time and to reduce errors, it is strongly recommended that the schema of the uploaded file matches the schema of the hosted feature service.  Let’s look at an example.  NASA publishes a daily active fires dataset, which can be crucial in mapping fire perimeters and visualizing the day-over-day change.   By using Append instead of publishing new layers, you can store all that information in one layer, with the bonus of being able to use the time slider to visualize daily changes.  You can just download the file and append the new data daily (in the case of the active fires dataset, make sure you take a few extra seconds to define the projection before uploading). By default, ArcGIS Online will attempt to match fields to your feature layer. If you want to change that, just click Show Field Mapping and change the defaults, or unmatch fields you don’t want used.  When you’re happy with the selected options, click Apply to append the data.

You might have noticed a little check box that was skipped that said, “Update existing features”.  While Append is primarily about adding new features into a layer, it also can append updates into existing features.  I’ll cover that option in more detail in a separate blog article (Part 2). So, stay tuned for more details and examples.  In the meantime if you’re looking to read more on managing hosted feature layers you can find additional reading here.

This entry was posted in ArcGIS Online and tagged , . Bookmark the permalink.

Leave a Reply

12 Comments

  1. tig.asset.management says:

    The Update Data is not showing up on my Hosted layers on AGOL. Is there something I need to enable to make that appear? The layer is already set as editable.

    • Paul Barker says:

      tig.asset.management

      The most common reason for the menu to be hidden is if the hosted feature layer has “sync” or “keep track of created and updated features” turned on. We’re working to remove that restriction in a future release. The other scenarios are: if the hosted feature layer is a view or if you do not own the service. Check those, and if none of those apply please reach out to support and we can investigate further.

      • wiscrapids says:

        Can these types of details that restrict applicability be included in blog posts by default? A guess at expansion of a new feature (weeks vs. years) would also be useful. As an example, GIS administrators may want to delay staff training on the current process because a new method available in Hosted Layers would simplify the process considerably and would be easier for end users.

        • wiscrapids says:

          Noting that the restriction is included, could those restrictions be bolded or otherwise emphasized (preferably in a consistent manner across blog posts advertising new functionality)?

      • tig.asset.management says:

        Gotcha. Working now. Thanks

  2. mstarry_mnpower says:

    Can this be scripted?

  3. kulasdn says:

    Thanks for this post (and for Part 2 as well), and for adding this functionality! Just so I’m clear, “One quick note at this time you can’t append data to a hosted feature layer that has synchronization or change tracking (Keep track of created and updated features) enabled.”

    To reiterate your answer to a previous poster, ESRI is planning to remove the sync restriction in a future release… can you give any indication as to how soon/not soon this will be? It would be nice to have an idea for workflow planning purposes. And once this sync restriction is lifted for appending data, would that also apply to the overwrite capability as well? I’d love to one day be able to tell my staff they don’t have to delete and re-download maps in collector every time I update reference data quarterly. Any insights would be greatly appreciated!

    • Paul Barker says:

      That’s certainly where we want to get to. I can’t give you a timeline at the moment unfortunately but removing restriction is something we’re actively working on.

      • kulasdn says:

        Great, thanks! One other question… for the overwriting portion, I see in the help that the ‘Overwrite Entire Layer’ option is not available if the service was published by ArcMap originally. Is this going to change at any point? I published most of my layers via ArcMap in the early days because I felt I had greater control of the data, but it would be nice to overwrite them directly through ArcGIS Online now that they’re hosted and configured properly. Will more flexibility be coming with the overwrite option as well as the append function?

        • Paul Barker says:

          kulasdn,

          That’s something that has been discussed some but no firm plans at this time to change. I would be interested in learning more about your ideal workflow for this though. If you have time can you write your workflow up and share it with support as an enhancement request.

  4. heisenbarth says:

    Is replacement of features no longer an option?