Recently, we were contacted by a Bureau of Antiquities and Archeology (BAA) of a Mediterranean country about issues they were having solving a classic GIS overlay problem. The BAA is responsible for the inventory and protection of archeological sites within their country. Many of these sites are on private land parcels and the BAA needs basic information about these parcels on a daily basis, such as ownership and status (whether the parcel is being sold, subdivided, or permitted for construction). The parcel database is maintained by the Land Survey department.
Both the BAA and the Land Survey use ArcGIS to maintain their databases, and both use ArcGIS for Server to disseminate information. There are approximately 25,000 archeology sites and 800,000 parcels.
To illustrate the problem, I concocted some data, illustrated below. The Sites layer is the BAA data. Each site has unique primary key (for example, ‘Ar_8496’) used to relate to other tables maintained by the BAA. The vast majority of sites are simple rectangles as illustrated below. Parcels also have a unique primary key (for example, ‘p844’) and an additional STATUS attribute that describes the current state of the parcel. The BAA is particularly interested in any parcel that is for sale, in escrow, or has a request for subdivision. As mentioned, I concocted this data and attributes—the actual attributes are far more complex, but, in essence, they boil down to a primary key and a parcel status.
The daily task is to produce a simple table as shown in figure 2. This table tabulates the intersection of sites and parcels.
SID is the primary site key, and
PID is the primary parcel key.
AREA is the area of overlap in hectares, and
PERCENTAGE sums to 100.0 for each unique SID.
This is a classic overlay problem, where one dataset is overlaid on another to create a new feature class or table of the geometric intersections of the input features. There are many ways to perform overlay in ArcGIS for Desktop using geoprocessing tools, including Tabulate Intersection, Intersect, Identity, Union, Spatial Join, and Select Layer By Location.
Currently, the Land Survey hosts a web service that accepts a single polygon feature then executes an SQL query against their parcels using the ST_Intersects function. The output is a feature class of intersecting parcels and their (public) attributes. This service is meant for general public requests. However, when the BAA uses this service, they send their 25,000 sites to the service—one at a time—and the Land Survey server interprets this as an attack and shuts down the connection.
In the remainder of this post, I’ll show how geoprocessing services in ArcGIS 10.1 Desktop and Server can be used to solve this problem.
- No web programming involved; everything is accomplished using Desktop and a couple of simple services. Basic knowledge of geoprocessing, ModelBuilder, and publishing services on Desktop is all that’s needed.
- Two geoprocessing services are created:
- A geoprocessing service to serve data from the BAA server
- A geoprocessing service to performs the analysis on the Land Survey server and sends the result table back to the BAA
- Finally, we’ll use the Land Survey geoprocessing service in a client-side model to do further processing and analysis
Solving the problem with local data
Let’s assume for the moment that both
Parcels are feature classes on local disk—no web services involved. How would we create the table shown above in figure 2? The
Tabulate Intersection tool, new at Desktop 10.1, does the job perfectly. Figure 3 shows the
Tabulate Intersection tool dialog and parameters.
- Input Zone Features: the BAA
- Zone Fields: one or more fields whose values uniquely define input zone polygons. In this case, it’s the unique site key
- Input Class Features: the Land Survey’s
- Output Table: table to contain the cross-tabulation.
- Class Fields: one or more fields whose values uniquely define a parcel. In this case, only
PIDis needed to uniquely define a parcel, but I also want report the
STATUSfield on the output table.
- Sum Fields: No fields need to be totaled, so this is left blank.
- XY Tolerance: Tolerance to use when comparing coordinates. In this case, I’ll let the value default to the tolerance stored with the input data.
- Output Units: reporting units for the output
Tabulate Intersection with these parameters creates the table as shown in figure 2 above.
Using a geoprocessing service to distribute the data
The problem with using
Tabulate Intersection above is that it assumes that all the data is local when in fact it’s not. There are two choices; we can ship the 800,000 plus parcels to the BAA and have them execute
Tabulate Intersection, or we can ship the 25,000 archeology sites to the Land Survey and have them do the processing. The latter makes the most sense since the Sites feature class has the fewest features, and the feature geometry—mostly simple rectangles—further reduces the size.
To send the site data to the Land Survey, we’ll create a geoprocessing service on the BAA server. The service is built using ModelBuilder using one tool—
Make Feature Layer. Here’s the model:
Make Feature Layer tool dialog, shown in figure 4, the Input Features is the
Sites feature class located in the file geodatabase
E:DataArcheologyArcheology.gdb, but it could just as easily be a shapefile or a versioned feature class stored in an enterprise ArcSDE database. Note the list of fields in the Field Info parameter.
TYPE is a field on the Sites feature class that we don’t want to serve with the data due to security concerns. To remove this field on the output layer,
TYPE has Visible unchecked, meaning it will not be included on the output feature layer.
Finally, the output variable
Sites_Layer must be made a Model Parameter, as shown in figure 5. Model Parameters have a ‘
P’ displayed next to the variable in ModelBuilder.
That’s it—no other tools or processes are needed. Once the model has been saved to a custom toolbox, all that’s necessary is to execute it using its tool dialog, as shown in figure 6. The output layer name,
Sites_Layer, can be any name. For the purposes of publishing a geoprocessing service, we can just use the default name.
After running the
Get Archeology Sites model using the tool dialog, a result is created in the Results window . To publish the service, right-click the result and choose Share As > Geoprocessing Service as shown in figure 7. This opens the publishing wizard where you choose a server and give the service a name. The Service Editor opens, as shown in figure 8, where you define properties of the service and tasks within the service.
The Service Editor window is where details about the service are defined. One important setting is the “Maximum number of records returned by the server” found on the Parameters tab, shown in figure 8. The default is 1000, but for this service, we need to bump it up to 25000, the number of features in the BAA Sites feature class.
About serving features using Make Feature Layer
A geoprocessing service based on
Make Feature Layer is a very simple and direct way to serve features from any feature class. The service’s output layer can be used directly by geoprocessing tools in Desktop.
Alternatively, you could publish a Map service that includes a Feature service. A Feature service allows for web-based editing, but it also requires the feature classes reside in an enterprise database (such as SQL Server Express that can be installed with ArcGIS for Server).
Using a data store
When publishing a service, one of the decisions you need to make is how data used by the service is accessed by the service. You have two basic choices:
- Let the publishing process make a copy of the data. The service will use the copied data.
- Register a data store with the server. When the service runs, it will use the data from the data store rather than making a copy of the data and using the copy. The assumption is that the data in the data store is live data.
For this service, we want to use a data store that contains the most recent site data, not a static copy made a publication time. Figure 9 shows using ArcGIS Server Manager to set up a data store named Archeology Data Store. This data store is any data found in the
E:DataArcheology folder, such as the
E:DataArcheologyArcheology.gdbSites feature class used in the
Make Feature Layer tool shown in figure 4 above. In this example, the
Get Archeology Sites model was built on the server machine, so the Publisher Folder Path and the Server Folder Path are the same.
Meanwhile, back at the Land Survey
Get Archeology Sites task up and running, the Land Survey can now create a service that uses this task. Figure 10 shows building a model at the Land Survey. The Land Survey makes a connection to Archeology_Server and drags and drops the
Get Archeology Sites task into ModelBuilder.
The completed model,
Tabulate Intersection of Sites and Parcels, is shown below. The
Get Archeology Sites task is used to fetch the
Sites_Layer, the output of
Get Archeology Sites, is used as the Input Zone Features for
Tabulate Intersection. Other than that, the parameter values are the same as shown in Figure 2.
Just like the
Get Archeology Sites model, the output variable,
Parcels_TabulateIntersection, is made a model parameter. The model is saved and then run using its tool dialog, as illustrated below, to create a result in the Results window.
The result created by running
Tabulate Intersection of Sites and Parcels is published as a service to the Land Survey’s ArcGIS Server. As with the
GetArcheologicSites service, there are a couple of considerations:
- The maximum number of records returned by the server needs to be set. Since there may be many parcels per archeology site, the initial setting should be in the hundreds of thousands. This isn’t much of an issue because each record is small—just a few fields—and there are no features to return. The maximum number of records can be fine-tuned as the service is used and monitored.
Parcelsfeature class should reside in a data store so that
Tabulate Intersectionuses the most recent data.
Back to the BAA
To tabulate the intersection of archeology sites and land parcels, the BAA connects to the Land Survey server and executes the
Tabulate Intersection of Sites and Parcels task. This task, in turn, runs the BAA’s
Get Archeology Sites task to fetch the most current archeology sites. The output of the
Tabulate Intersection of Sites and Parcels task is the table shown in Figure 2.
Preserving the output table
When the BAA runs the
Tabulate Intersection of Sites and Parcels task in ArcMap, the table is added to the table of contents. However, certain operations, such as joins and field calculations, are not available for the outputs of server tasks. The best practice is write the output to a known location and do further analysis on the local copy. The model in figure 14 shows how the
Copy Rows tool is used to copy the table to a known location on local disk.
The model in figure 14 is what the BAA will execute on a daily basis. The model can be enhanced to join data from other BAA datasets, or to automatically generate output names based on the current date, for example.
Recall that the parcel features contain a
STATUS attribute that may contain the value of
NoChange. In most cases, the BAA is not interested in parcels that have this value. To speed things up considerably, the
Tabulate Intersection of Sites and Parcels task can be modified to only select those parcels that have a
STATUS of anything but
NoChange. This model is shown in figure 15, where the
Parcels feature class is input to
Make Feature Layer.
Make Feature Layer has an optional Expression parameter which is used here to select parcels that do not have a status of
An easy way to distribute data is to create a geoprocessing service task that uses the Make Feature Layer tool to return a layer and its features. The output of the task is a layer that can be used by any client, but is especially useful for geoprocessing on ArcGIS for Desktop clients. Geoprocessing models and scripts that use geoprocessing service tasks within can themselves be published as geoprocessing services, thereby distributing the data and analysis among different servers. The main concerns are the number of features that will be passed between clients and whether those features are a static copy (copied to the server when published) or come from a data store that contains the most recent data.
For more information about the Tabulate Intersection tool and other overlay tools, see the next blog post in this series: More adventures in overlay: comparing commonly used overlay tools