Bringing it all together – distributed analysis with distributed data

Recently, we were contacted by a Bureau of Antiquities and Archeology (BAA) of a Mediterranean country about issues they were having solving a classic GIS overlay problem. The BAA is responsible for the inventory and protection of archeological sites within their country. Many of these sites are on private land parcels and the BAA needs basic information about these parcels on a daily basis, such as ownership and status (whether the parcel is being sold, subdivided, or permitted for construction). The parcel database is maintained by the Land Survey department.

Both the BAA and the Land Survey use ArcGIS to maintain their databases, and both use ArcGIS for Server to disseminate information. There are approximately 25,000 archeology sites and 800,000 parcels.

To illustrate the problem, I concocted some data, illustrated below. The Sites layer is the BAA data. Each site has unique primary key (for example, ‘Ar_8496’) used to relate to other tables maintained by the BAA. The vast majority of sites are simple rectangles as illustrated below. Parcels also have a unique primary key (for example, ‘p844’) and an additional STATUS attribute that describes the current state of the parcel. The BAA is particularly interested in any parcel that is for sale, in escrow, or has a request for subdivision. As mentioned, I concocted this data and attributes—the actual attributes are far more complex, but, in essence, they boil down to a primary key and a parcel status.

Figure 1: Map showing parcels and archeological sites.

The daily task is to produce a simple table as shown in figure 2. This table tabulates the intersection of sites and parcels. SID is the primary site key, and PID is the primary parcel key. AREA is the area of overlap in hectares, and PERCENTAGE sums to 100.0 for each unique SID.

Figure 2: Cross-tabulation table

This is a classic overlay problem, where one dataset is overlaid on another to create a new feature class or table of the geometric intersections of the input features. There are many ways to perform overlay in ArcGIS for Desktop using geoprocessing tools, including Tabulate Intersection, Intersect, Identity, Union, Spatial Join, and Select Layer By Location.

Currently, the Land Survey hosts a web service that accepts a single polygon feature then executes an SQL query against their parcels using the ST_Intersects function.  The output is a feature class of intersecting parcels and their (public) attributes. This service is meant for general public requests. However, when the BAA uses this service, they send their 25,000 sites to the service—one at a time—and the Land Survey server interprets this as an attack and shuts down the connection.

In the remainder of this post, I’ll show how geoprocessing services in ArcGIS 10.1 Desktop and Server can be used to solve this problem.

• No web programming involved; everything is accomplished using Desktop and a couple of simple services. Basic knowledge of geoprocessing, ModelBuilder, and publishing services on Desktop is all that’s needed.
• Two geoprocessing services are created:
  • A geoprocessing service to serve data from the BAA server
  • A geoprocessing service to performs the analysis on the Land Survey server and sends the result table back to the BAA
• Finally, we’ll use the Land Survey geoprocessing service in a client-side model to do further processing and analysis

Solving the problem with local data

Figure 3: Tabulate Intersection Dialog.

Let’s assume for the moment that both Sites and Parcels are feature classes on local disk—no web services involved.  How would we create the table shown above in figure 2?  The Tabulate Intersection tool, new at Desktop 10.1, does the job perfectly.  Figure 3 shows the Tabulate Intersection tool dialog and parameters.

• Input Zone Features: the BAA Sites layer
• Zone Fields: one or more fields whose values uniquely define input zone polygons.  In this case, it’s the unique site key SID.
• Input Class Features: the Land Survey’s Parcel layer.
• Output Table: table to contain the cross-tabulation.
• Class Fields: one or more fields whose values uniquely define a parcel.  In this case, only PID is needed to uniquely define a parcel, but I also want report the STATUS field on the output table.
• Sum Fields: No fields need to be totaled, so this is left blank.
• XY Tolerance: Tolerance to use when comparing coordinates.  In this case, I’ll let the value default to the tolerance stored with the input data.
• Output Units: reporting units for the output AREA field.

Running Tabulate Intersection with these parameters creates the table as shown in figure 2 above.

Using a geoprocessing service to distribute the data

The problem with using Tabulate Intersection above is that it assumes that all the data is local when in fact it’s not.  There are two choices; we can ship the 800,000 plus parcels to the BAA and have them execute Tabulate Intersection, or we can ship the 25,000 archeology sites to the Land Survey and have them do the processing.  The latter makes the most sense since the Sites feature class has the fewest features, and the feature geometry—mostly simple rectangles—further reduces the size.

To send the site data to the Land Survey, we’ll create a geoprocessing service on the BAA server. The service is built using ModelBuilder using one tool—Make Feature Layer.  Here’s the model:

Figure 4: Get Archeology Sites model

In the Make Feature Layer tool dialog, shown in figure 4, the Input Features is the Sites feature class located in the file geodatabase E:DataArcheologyArcheology.gdb, but it could just as easily be a shapefile or a versioned feature class stored in an enterprise ArcSDE database. Note the list of fields in the Field Info parameter. TYPE is a field on the Sites feature class that we don’t want to serve with the data due to security concerns.  To remove this field on the output layer, TYPE has Visible unchecked, meaning it will not be included on the output feature layer.

Finally, the output variable Sites_Layer must be made a Model Parameter, as shown in figure 5. Model Parameters have a ‘P’ displayed next to the variable in ModelBuilder.

Figure 5: Making the output a model parameter.

That’s it—no other tools or processes are needed. Once the model has been saved to a custom toolbox, all that’s necessary is to execute it using its tool dialog, as shown in figure 6. The output layer name, Sites_Layer, can be any name. For the purposes of publishing a geoprocessing service, we can just use the default name.

Figure 6: Get Archeology Sites tool dialog.

After running the Get Archeology Sites model using the tool dialog, a result is created in the Results window .  To publish the service, right-click the result and choose Share As > Geoprocessing Service as shown in figure 7.  This opens the publishing wizard where you choose a server and give the service a name.  The Service Editor opens, as shown in figure 8, where you define properties of the service and tasks within the service.

Figure 7: Sharing a result as a geoprocessing service

The Service Editor window is where details about the service are defined.  One important setting is the “Maximum number of records returned by the server” found on the Parameters tab, shown in figure 8. The default is 1000, but for this service, we need to bump it up to 25000, the number of features in the BAA Sites feature class.

Figure 8: Setting Maximum number of records returned by the server.

About serving features using Make Feature Layer

A geoprocessing service based on Make Feature Layer is a very simple and direct way to serve features from any feature class.  The service’s output layer can be used directly by geoprocessing tools in Desktop.

Alternatively, you could publish a Map service that includes a Feature service.  A Feature service allows for web-based editing, but it also requires the feature classes reside in an enterprise database (such as SQL Server Express that can be installed with ArcGIS for Server).

Using a data store

Figure 9: Setting up a data store

When publishing a service, one of the decisions you need to make is how data used by the service is accessed by the service.  You have two basic choices:

• Let the publishing process make a copy of the data.  The service will use the copied data.
• Register a data store with the server.  When the service runs, it will use the data from the data store rather than making a copy of the data and using the copy.  The assumption is that the data in the data store is live data.

For this service, we want to use a data store that contains the most recent site data, not a static copy made a publication time.  Figure 9 shows using ArcGIS Server Manager to set up a data store named Archeology Data Store.  This data store is any data found in the E:DataArcheology folder, such as the E:DataArcheologyArcheology.gdbSites feature class used in the Make Feature Layer tool shown in figure 4 above.  In this example, the Get Archeology Sites model was built on the server machine, so the Publisher Folder Path and the Server Folder Path are the same.

Meanwhile, back at the Land Survey

With the Get Archeology Sites task up and running, the Land Survey can now create a service that uses this task.  Figure 10 shows building a model at the Land Survey.  The Land Survey makes a connection to Archeology_Server and drags and drops the Get Archeology Sites task into ModelBuilder.

Figure 10: Adding the Get Archeology Sites task to a model

The completed model, Tabulate Intersection of Sites and Parcels, is shown below.  The Get Archeology Sites task is used to fetch the Sites features. Sites_Layer, the output of Get Archeology Sites, is used as the Input Zone Features for Tabulate Intersection.  Other than that, the parameter values are the same as shown in Figure 2.

Figure 11: Tabulate Intersection of Sites and Parcels.

Just like the Get Archeology Sites model, the output variable, Parcels_TabulateIntersection, is made a model parameter.  The model is saved and then run using its tool dialog, as illustrated below, to create a result in the Results window.

Figure 12: The Tabulate Intersection of Sites and Parcels tool dialog.

The result created by running Tabulate Intersection of Sites and Parcels is published as a service to the Land Survey’s ArcGIS Server.  As with the GetArcheologicSites service, there are a couple of considerations:

• The maximum number of records returned by the server needs to be set.  Since there may be many parcels per archeology site, the initial setting should be in the hundreds of thousands.  This isn’t much of an issue because each record is small—just a few fields—and there are no features to return.  The maximum number of records can be fine-tuned as the service is used and monitored.
• The Parcels feature class should reside in a data store so that Tabulate Intersection uses the most recent data.

Back to the BAA

Figure 13: The two services displayed in the Catalog window

To tabulate the intersection of archeology sites and land parcels, the BAA connects to the Land Survey server and executes the Tabulate Intersection of Sites and Parcels task.  This task, in turn, runs the BAA’s Get Archeology Sites task to fetch the most current archeology sites.  The output of the Tabulate Intersection of Sites and Parcels task is the table shown in Figure 2.

Preserving the output table

When the BAA runs the Tabulate Intersection of Sites and Parcels task in ArcMap, the table is added to the table of contents.  However, certain operations, such as joins and field calculations, are not available for the outputs of server tasks. The best practice is write the output to a known location and do further analysis on the local copy. The model in figure 14 shows how  the Copy Rows tool is used to copy the table to a known location on local disk.

Figure 14: A model to copy the output to disk.

The model in figure 14 is what the BAA will execute on a daily basis.  The model can be enhanced to join data from other BAA datasets, or to automatically generate output names based on the current date, for example.

An optimization

Recall that the parcel features contain a STATUS attribute that may contain the value of NoChange.  In most cases, the BAA is not interested in parcels that have this value. To speed things up considerably, the Tabulate Intersection of Sites and Parcels task can be modified to only select those parcels that have a STATUS of anything but NoChange.  This model is shown in figure 15, where the Parcels feature class is input to Make Feature Layer.  Make Feature Layer has an optional Expression parameter which is used here to  select parcels that do not have a status of NoChange.

Summary

An easy way to distribute data is to create a geoprocessing service task that uses the Make Feature Layer tool to return a layer and its features. The output of the task is a layer that can be used by any client, but is especially useful for geoprocessing on ArcGIS for Desktop clients. Geoprocessing models and scripts that use geoprocessing service tasks within can themselves be published as geoprocessing services, thereby distributing the data and analysis among different servers.  The main concerns are the number of features that will be passed between clients and whether those features are a static copy (copied to the server when published) or come from a data store that contains the most recent data.

For more information about the Tabulate Intersection tool  and other overlay tools, see the next blog post in this series: More adventures in overlay: comparing commonly used overlay tools