Parallel Geoprocessing in ArcGIS Pro

ArcGIS Pro is a powerful multithreaded application with unparalleled (no pun intended) spatial analysis and geoprocessing capabilities. Over the last year, including the 2.0 and 2.1 Pro releases, nearly 300 new geoprocessing tools were added which solve new spatial problems, make your work easier, and enable processing of new data types. Some of the new tools include:

  • New spatial statistics tools that use machine learning to find spatial and attribute clusters in your data
  • New tools for adding and calculating multiple attribute fields
  • A new extension toolbox for interpreting, exploiting, and analyzing imagery
  • A rich set of tools for creating, managing and tracing the new utility network, and generating network diagrams
  • Many, many other new tools, and countless tool enhancements

Keep checking What’s New in ArcGIS Pro in the coming weeks to see the full list of new analysis and geoprocessing capabilities added to Pro 2.1 and previous releases.

While the quantity of new functionality is undeniably impressive, you might be wondering “What is ArcGIS Pro doing to make my analysis faster?”

Each release a number of tools are enhanced to provide better performance. The Calculate Field tool was enhanced in Pro 2.1 to support Arcade expressions, which are considerably faster compared to calculations in ArcMap. For example, a date-time calculation on 20,000 features in ArcMap took nearly one minute, while Pro with Arcade takes less than 3 seconds.

Calculate Field with Arcade

Calculate Field in ArcGIS Pro 2.1 using Arcade


 

 

♫ Shave and a haircut, 64-bits! ♫

As discussed in this FAQ, geoprocessing in ArcGIS Pro is 64-bit and occurs on a dedicated geoprocessing thread. For most geoprocessing tools performance is roughly equivalent between ArcGIS Pro and ArcMap*, while many tools have been improved and are faster. *ArcMap comparisons use the 64-bit background geoprocessing setup

64-bit geoprocessing does not automatically make tools faster, only able to use more memory for data processing. 64-bit geoprocessing is more robust, results will be more accurate, and processes that previously hung, crashed, or ran out of memory may be able to complete successfully. Some ArcGIS Pro users on twitter recently shared their success with running a large geoprocessing job.

 

 

Listen to this yarn about multiple threads

Likewise, multithreading does not make geoprocessing faster. The multithreading in ArcGIS Pro makes it possible for the geoprocessing thread to run independently from other threads. This means that when a tool is running you can do other things in Pro such as add new data to a map, navigate and make selections in the map, change a layer’s symbology or labels, create and modify a layout, georeference an image, and even find and fill in another geoprocessing tool.

While multithreading is important and will make your work in Pro more productive, many people incorrectly assume it is a performance enhancing drug that will make every tool run in world record time.

 

 

Meanwhile, in a parallel universe…

One technique geoprocessing uses to improve performance is parallel processing. Parallel processing is a computing technique which splits up a big job into many smaller jobs and allows multiple CPUs, cores, or processes to work on the big job at the same time, often resulting in faster processing time. More and more geoprocessing tools leverage parallel processing in each new Pro release. In ArcGIS Pro 2.1, 70 geoprocessing tools can run using parallel processing to split a job across multiple cores or processes on your desktop machine. This is more than twice the number of tools that use parallel processing in ArcMap 10.6.

Parallel processing in ArcGIS Pro is controlled by the Parallel Processing Factor. Some tools will use parallel processing by default, while others require you to set the parallel processing factor to turn it on. Refer to each tool’s help documentation to determine how to control parallel processing for the selected tool.

While there are many geoprocessing tools that do not support parallel processing, we’ve worked on some high priority analysis and data crunching tools to ensure that these processes can leverage modern hardware and have the best possible performance with improved scalability.

The following geoprocessing tools support parallel processing in ArcGIS Pro 2.1.

 

Analysis Tools

                    • Intersect
                    • Pairwise Intersect
                    • Union
                    • Update
                    • Identity
                    • Clip
                    • Erase
                    • Graphic Buffer
                    • Pairwise Buffer
                    • Pairwise Dissolve
                    • Split


3D Analyst Tools

                    • Intervisibility
                    • Regularize Building Footprint
                    • Classify Las Building
                    • Reclassify
                    • Viewshed 2


Conversion Tools

                    • Multipatch To Raster


Spatial Statistics and Space Time Pattern Mining

                    • Spatially Constrained Multivariate Clustering (Spatial Statistics)
                    • Local Outlier Analysis (Space Time Pattern Mining)


Spatial Analyst Tools

                    • Viewshed 2
                    • Weighted Overlay
                    • Weighted Sum
                    • Zonal Statistics
                    • Zonal Statistics As Table
                    • Reclassify


Geostatistical Analyst Tools

                    • Areal Interpolation Layer To Polygons
                    • Cross Validation
                    • Densify Sampling Network
                    • Diffusion Interpolation With Barriers
                    • EBK Regression Prediction
                    • Empirical Bayesian Kriging
                    • Create Geostatistical Layer
                    • GA Layer To Contour
                    • GA Layer To Grid
                    • GA Layer To Points
                    • GA Layer To Rasters
                    • Moving Window Kriging
                    • Gaussian Geostatistical Simulations
                    • Generate Subset Polygons
                    • Global Polynomial Interpolation
                    • IDW
                    • Kernel Interpolation With Barriers
                    • Local Polynomial Interpolation
                    • Radial Basis Functions


Data Management Tools (Raster and Mosaic Dataset tools)

                    • Add Rasters To Mosaic Dataset
                    • Analyze Mosaic Dataset
                    • Build Footprints
                    • Build Mosaic Dataset Item Cache
                    • Build Overviews
                    • Build Pyramids and Statistics
                    • Build Seamlines
                    • Color Balance Mosaic Dataset
                    • Compute Camera  Model
                    • Compute Control Points
                    • Compute Tie Points
                    • Copy Raster
                    • Export Mosaic Dataset Items
                    • Export Tile Cache
                    • Generate Point Cloud
                    • Generate Raster Collection
                    • Generate Raster From Raster Function
                    • Import Tile Cache
                    • Interpolate From Point Cloud
                    • Manage Tile Cache
                    • Register Raster
                    • Split Raster
                    • Synchronize Mosaic Dataset


Image Analyst Tools

                    • Weighted Sum
                    • Zonal Statistics


GeoAnalytics Tools

In addition to the ArcGIS Desktop tools listed above, ArcGIS Pro can also run tools on ArcGIS Enterprise servers. The ArcGIS GeoAnalytics Server provides a number of tools that perform feature and big data analysis using distributed computing on one or more machines. You can run and use these tools in Pro just like any other geoprocessing tools, using either your local or enterprise data sources as input, as well as big data file shares and feature layers in your portal. The GeoAnalytics tools are available in Pro if you are connected to an ArcGIS Enterprise that has a GeoAnalytics server, and your user account has sufficient privileges.

                    • Calculate Density
                    • Create Space Time Cube
                    • Find Hot Spots
                    • Detect Incidents
                    • Find Similar Locations
                    • Calculate Field
                    • Copy To Data Store
                    • Aggregate Points
                    • Join Features
                    • Reconstruct Tracks
                    • Summarize Attributes
                    • Summarize Within
                    • Create Buffers


Example

I was recently running a Dissolve operation on 20,000 overlapping polygons. While this isn’t a huge process, it still took several minutes to complete. I was wishing it could run faster by splitting the operation up and using all 4 cores on my desktop machine. While the Dissolve tool does not support parallel processing, the Pairwise Dissolve tool does. Using the Pairwise Dissolve tool with it’s default Parallel Processing Factor setting (full parallel processing–the tool distributes the work to all machine cores), the same dissolve process took less than 3 seconds.

pairwise dissolve

Pairwise Dissolve completed in under 3 seconds

Now that’s unparalleled performance!

This entry was posted in Analysis & Geoprocessing, ArcGIS Pro. Bookmark the permalink.

Leave a Reply