Tag Archives: ModelBuilder

More adventures in overlay: comparing commonly used overlay tools

In the blog about distributing data and analysis, I showed using the Tabulate Intersection tool to overlay archeological sites and land parcels to create a table containing information about each parcel within an archeological site. Figure 1 shows the data and the results of Tabulate Intersection. This “what sites overlay what parcels?” question is a classic overlay query. These types of queries are the bread-and-butter of geographic analysis.
Continue reading

Posted in Analysis & Geoprocessing, Uncategorized | Tagged , , , , , , | Leave a comment

Cleaning up line data with geoprocessing

Editing and data compilation are less commonly thought of as operations that can be automated through geoprocessing. However, ArcGIS 10 introduced the Editing toolbox, which contains a set of geoprocessing tools to perform bulk edits. These tools combined with others in the geoprocessing environment can automate data import and maintenance work. Automated data compilation tools are especially useful for importing data into a geodatabase but can also be employed on a regular schedule to perform routine quality assurance (QA) checks. In this entry, I will discuss the use of geoprocessing to clean CAD line data as part of the import process.

Importing data with geoprocessing
Lines that are created without the use of spatial integrity measures, such as snapping or topology, almost always contain some inconsistencies. These errors are likely in data that originated in formats such as CAD, shapefile, or KML.  Fortunately, many common topological issues can be resolved in an automated manner by using ModelBuilder to link together tools that will import data into a geodatabase and perform standard data cleanup techniques.

I have a CAD file for a new subdivision that needs to be integrated with my existing GIS parcel data. The GIS data must be kept to stringent accuracy standards, so I need to fix any issues where lines do not connect to each other, overlap, or are duplicated. Rather than risk reducing the quality of the main parcels geodatabase, I can create a local temporary geodatabase where I can preprocess the CAD lines before introducing the features into the production geodatabase. Although the CAD file contains buildings, roads, text, registration tic marks, and other features, I plan to use only the parcel lot lines.

I have built a model that imports the CAD lines into a temporary scratch workspace, cleans and processes the lines, and then copies the corrected lines into an output file geodatabase. When importing CAD data into a geodatabase, I can choose from several available tools, including CAD to Geodatabase or Feature Class to Feature Class. The CAD to Geodatabase tool converts all the geometries in a drawing to individual feature classes, such as a line feature class for the parcel lines, annotation feature class for CAD text, and so on. In my case, I am using Feature Class to Feature Class tool because I need only the lot line geometry from the CAD file. This tool makes the model reusable because it can import many different formats and not simply CAD. In addition, the Feature Class to Feature Class tool allows for an SQL expression so I can further refine the import to include only the CAD features that satisfy an attribute query for lot lines (in this case, “Layer” = ‘LOT-L’).

Performing automated quality assurance on lines
Once the CAD parcel lot lines are imported into a geodatabase feature class, I can begin running tools to perform automated QA processes. Many tools are found in the Editing toolbox, although other toolboxes can be purposed for data compilation QA tasks. For example, I can start by using the Integrate tool in the Data Management toolbox to address minor inconsistencies in the line work. Integrate makes features coincident if they fall within the specified x,y tolerance. By using a small tolerance on Integrate (and other similar tools), I can avoid editing the data beyond the level of cleanup I intended. In addition, since I am running the tools on a copy of the data outside my production database, I can run the tools repeatedly to refine tolerance values to fix more issues in an automated manner. The intermediate data created as the model runs is maintained and can be reviewed in the scratch geodatabase.

After the dataset is integrated, I check for duplicated lines with the Delete Identical tool (Data Management toolbox). The dashed lines connecting to this tool represent preconditions, which are used to control the order of operations in a model. For example, the Integrated Lines output is a precondition to the Delete Identical tool. This way, the Delete Identical tool will not execute until the lines have been integrated.

The next part of the model identifies lines that are dangles. With the Feature Vertices to Points tool in the Data Management toolbox, I create a new point feature class containing the line endpoints that are not connected to any other lines. I can then use Select Layer By Location to identify the lines that intersect these dangling endpoints. The resulting selection represents lines with dangles.

Many of these dangle errors can be fixed by running the Editing toolbox’s Trim Line, Extend Line, and Snap tools. Effective use of the Editing toolbox geoprocessing tools can improve productivity because the tools apply edits in bulk, such as to all features or all selected features. In most cases, the similar editing function applies to only one feature at a time. Because I exposed the tolerances as variables and model parameters, I can easily run the model with different values because the tolerance settings appear as input boxes on the tool’s dialog box. For example, I am willing to extend or trim the lines from this CAD dataset initially up to a maximum length of five feet. After that procedure, I want to inspect the lines visually to see how many issues remain to ensure that I will not be making incorrect edits if I increase the tolerance value. I can change the tolerance as needed depending on the accuracy of the lines I am importing.

In addition, since my organization’s spatial integrity rules indicate the parcel lines should be split and not intersect themselves, I can use a sequence of spatial and attribute queries to find the locations where lines have intersecting endpoints. Lines are often split so that each length can be attributed separately.

Once these processes have run, the lines are output into a feature dataset in a geodatabase and are much cleaner topologically. After the model completes, I can run the Feature Vertices to Points tool again on the cleaned output to see the remaining dangles and compare the current number of endpoints that are dangles (the yellow circles in the graphic) to the number in the original CAD lines (the red circles). While there may be a few remaining issues, there are less than before running the model. At this point, I can build a geodatabase topology to check for and repair any other errors. When I am satisfied that the lines meet the standards for our spatial data, I can import it into the production database.

For more information:
The sample tools and data can be downloaded from the Editing Labs group on ArcGIS.com. An ArcInfo license in required to run the tools.

Posted in Analysis & Geoprocessing, Editing | Tagged , , , , , , , , , | 2 Comments

Understanding which "if" troubles you

Question

Which If?

Use which tool?

Additional Help

Feature
How do I change the field values based on a condition?

Example:

Select rows of field1 with value X, and for those selected rows add value Y to the field 2.

If field value… Calculate Field, Select Layer By Attribute, Select
How do I continue running/stop the tool from running in a model if the condition is true/false? If condition is true/false… While, Stop
How do I run the model based on if the input feature type is a point/polyline/polygon? 

Example:

Do a conditional check if the input feature class is a polygon or a point feature type. If it is point type, run the next tool – Add Location. If it is a polygon feature, convert the polygon feature to a point feature by using the Feature To Point tool before running the Add Location tool.

If feature type… Calculate Value, Script tool If-then-else blog post 1 

If-then-else blog post 2

How do I run the model only if the input has projection similar to another feature class or a projection system specified by me? If projection is/isNot… Calculate Value, Script tool If-then-else blog post 5
How do I run the model only if the sql query resulted in any selected feature? 

Example:

If a common sql query is used in the Select Layer By Attribute tool to make a selection for all the input layers. Sometimes the SQL query results in selected features in one layer and no selected features for another layer. Do a conditional check if any of the features are selected from one layer to another.

If selection exists/ Does not exist… Select Layer By Attribute, Select
+
Get Count, Script tool
If-then-else blog post 4 

If-then-else blog post 2

How do I know if a file exists in a particular directory or not? 

Example:

Do a conditional check if a particular feature class exists in a particular workspace, and if it does, continue running the next tool Add Field in the model. If it does not exists, copy the feature class from a backup location before adding a field.

If feature class exists/Does not exist… Calculate Value, Script tool If-then-else blog post 1 

If-then-else blog post 2

Raster
How do I to perform a conditional if/else evaluation on each cell of the input raster? 

Example:

If the value is greater than 2000 set the new value to be the same as the input data; else set the remaining cells to NoData. Leaving the Input false raster parameter in the Con tool empty sets the value to NoData. Optionally you can provide another raster or a constant value.

If cell value… Con, Raster Calculator
How do I extract the cells of a raster based on a query? 

Example:

Create a new raster layer with cell value that are greater than 2000.

If cell value… Extract By Attribute, Make Raster Layer, Select Layer By Attribute
How do I perform a true/false evaluation of the input raster using an expression? 

Example:

If the cell value is greater than 2000, then set those cell values to 1 (true); else set the output cell value to 0.

If cell value… Test, Con, Raster Calculator
How do I select cells based on an SQL query and set them to NoData? 

Example:

If the cell value is greater than 2000, then set those cell values to NoData; else use the input data values for the remaining cells.

If cell value… Reclassify, Set Null, Raster Calculator
How do I set the NoData values in my input raster to something else? 

Example:

If the cell value is NoData, then set those cell values to be 0 or use values from another raster for those cells. Set the remaining cell values to be same as the input data.

If cell value is NoData… Reclassify, Is Null + Con
How do I check the properties of a raster dataset and run the model only if the property value matches a particular condition?
 

Example:

Check if the mean of all cell values is greater than 100.

If raster property… Get Raster Properties Examples of using the Calculate Value tool to set conditions
How do I check what is the raster format of the input data and run a model if it is of a particular type? 

Example:

Check if the raster dataset format is type Grid, Img, Tif, etc. then branch your model to run tools based on the condition.

If raster format… Calculate Value, Script tool using the Describe function Raster dataset formats
+
Examples of using the Calculate Value tool to set conditions
Posted in Analysis & Geoprocessing | Tagged , , , , , | 2 Comments

Automate your Business Analyst workflows with ModelBuilder

  by Kyle Watson

 

At the 2011 User Conference I spoke with countless clients about how they were using Business Analyst.  It really is a full spectrum of users…some uber GIS power users, some casual map enthusiasts with an eye for statistics.  But I did hear from more than a few that they were “looking into creating models but don’t really know where to get started.”

So I wanted to take a moment and show an example how you can use ModelBuilder to automate your everyday workflows. Continue reading

Posted in Location Analytics | Tagged , , , , , , | Leave a comment

Tutorial on Iterators in ModelBuilder

Shitij Mehta has uploaded a tutorial on working with Iterators in ModelBuilder.  Click here to download the tutorial.

Posted in Analysis & Geoprocessing | Tagged , , , | Leave a comment

Calculating Geodesic Distance Between Points

Going back to the very early days of ArcGIS there have been geoprocessing tools for calculating distances between features. Tools like Near, Point Distance, and Buffer have been around for many releases, and perform key analysis in a number of common GIS workflows. These distance-measuring tools have always worked well and calculated very accurate distances for features concentrated in a relatively small area (a city, state, or single UTM zone) with an appropriate projected coordinate system that minimizes distance distortion. Unfortunately, for groups of features spread over larger areas (regions, countries, or the world!), or for datasets with a geographic coordinate system, these tools have historically produced results that were less than perfect. Continue reading

Posted in Analysis & Geoprocessing | Tagged , , , , | 6 Comments

Geodatabase Modeling and Design SIG at the User Conference

This Special Interest Group will discuss available and forthcoming GDB design tools, patterns, and emerging  issues. The group will provide participants with an opportunity to communicate their  database design and modeling requirements to Esri staff who will be present.

Here’s the link to the online agenda: http://events.esri.com/uc/2011/infoWeb/OnlineAgenda/?fa=ofg_details_form&ScheduleID=2755

For more information visit: http://spatialdba.com/

Please take a minute to fill out the following survey: http://spatialdba.com/limesurvey/index.php?sid=72916&lang=en

Posted in Uncategorized | Tagged , , , , , | Leave a comment

Considerations when exporting a model to a Python script

Since the release of ArcGIS 9, you have been able to export models created in ModelBuilder to a Python script. While this option is valuable in helping you see how tools and environments are used in a scripting environment, there are many instances when your exported model will not work as expected.

This article explains why the option to export models to Python is intended to be used as a learning aid for understanding how tools and environments are accessed in Python. As models become more sophisticated to include techniques such as iteration or in-line variable substitution, this export option becomes less useful.

Continue reading

Posted in Analysis & Geoprocessing | Tagged , , , | 4 Comments

If you are stuck at "if" – Part 1

If-then-else logic is a simple yet powerful concept for performing different actions based on different conditions. If-then-else logic can be explained like this:

IF some condition is true, THEN
   perform an action;
ELSE the condition is false,
   perform a different action.

This is a three part blog which talks about different examples of if-then-else conditions and ways to implement it in ArcGIS.

Part 1 – Gives examples of quick and dirty ways of using the Calculate Value tool to create branches using if statements in a model.

Part 2 – Gives an example of creating and connecting an if-then-else script tool for branching in a model. The If Product license exists example of Calculate value
from part 1 has been converted into script tools to explain the concepts.

Part 3 –Gives three case examples of using script tools in a model. This part highlights the use of various capabilities of ModelBuilder such as:

  • using a branching script in a model
  • merging two branches
  • creating model parameters
  • using Value list filter in a model
  • setting preconditions
  • using an iterator in a model
  • creating a nested model or a model within a model

————————————————————————————————————————————————————————–

Examples of using the Calculate Value tool to create branches using if-else logic
In this blog part we cover five examples of using if-then-else logic code in the Calculate Value tool. There are various reasons to use either Calculate Value or a script tool for branching such as:

Reasons to use Calculate Value tool:

  • It is easy to use.
  • The code remains inside the model as compared to a script tool which reference an external Python file.
  • It has a built-in list of output data types.

Or there are reasons to use a Script tool:

  • It is easier to manage and debug large code examples using the Python IDE than in the Calculate Value tool.
  • If the code has many if and else conditions or too many possibilities of what input value could be. For example, if you are creating an “if” tool for any type of input dataset. There are more than 20 possible dataset types and it will make the code block in the Calculate Value tool too big to manage and debug.
  • The Calculate Value tool only has one output Boolean parameter (either true or false) so if you want to check for multiple conditions you have to add the Calculate Value tool multiple times with the logic to test against. For Example – Suppose you want to check if the input is a shapefile or a feature class from a geodatabase. The only way to do this is to add two Calculate Value tools and add the logic for checking just for the shapefile or just the feature class separately. In comparison the script tool can have two output Boolean parameters from a single tool and the tool redundancy can be avoided by adding the logic for both the checks in a single script tool.
  • If scripts are your style and comfort preference (Oh yes! I have heard this from many)

Where to start?
Just in case you are new to ModelBuilder you might want to start with the following topic and tutorials:

Now that you are comfortable with the ModelBuilder interface and vocabulary you can get started with this blog by downloading the models from here.

You can access the Calculate Value tool in ModelBuilder from the Insert > Model Only tools menu or right click side menu. Calculate Value tool returns a value based on a specified Python expression. Learn more about the tool by clicking here.

————————————————————————————————————————————————————————–

Example 1 – If Product License Exists
Tools have different license level requirements. Some tools run at ArcView level while others need an ArcInfo license. If you want to run tools based on product license levels in a model a quick way to check is by using if logic in Calculate Value tool.

The following example uses the Erase tool if the ArcInfo license is available and uses a combination of multiple tools to execute the same simple task as the Erase tool if the license is not available. You need to have some knowledge of the license level requirement for each tool. You can easily check this from the individual tool documentation.

The check is especially important, if you are sharing the models with your peers as the product level could be different from one user to another, and using this license level check can significantly minimizes the processing time, especially if it is a big model or large data.

For all examples the if-then-else code in Calculate Value tool is written in Python using ArcPy, a Python site package that encompasses and further enhances the arcgisscripting module. This example uses the ArcPy ProductInfo function to check the availability of the product license and sets the Boolean output parameter to true or false based on the if condition.

If Product License Exists Condition

Expression 

x("%ProductToCheck%")

Code Block

def x(ProductToCheck):
import arcpy
if arcpy.ProductInfo() == ProductToCheck:
    return "true"
else:
    return "false"
If Product License Does Not Exist Condition

Expression

x("%ProductToCheck%")

Code Block

def x(ProductToCheck):
import arcpy
if arcpy.ProductInfo() == ProductToCheck:
    return "false"
else:
    return "true"

————————————————————————————————————————————————————————–

Example 2 – If Extension Exists
Esri offers a wide range of optional extensions that can dramatically expand the capabilities of ArcGIS. Some of these ArcGIS tools are packaged in different extensions. The following example executes the Geographically Weighted Regression tool from Spatial Statistics toolbox. Geographically Weighted Regression requires an ArcInfo license or the Spatial Analyst extension or the Geostatistical Analyst extension. The example model checks to see if an ArcInfo license is available just as in the previous example and if the license is not available checks to see if either the Spatial or Geostats extension is available using the Calculate Value tool. The Boolean outputs (True and False) are passed to the Merge Branch, a tool often used in if-then-else logic. Merge Branch takes any number of inputs and these inputs can be of any data type (feature layer, Boolean, String, Double, and so on). When Merge Branch executes, it examines the list of inputs, and outputs the first valid value it finds. In the case of Booleans, it examines the list and outputs the True if any of the inputs are true. If none of the inputs are true, the output of Merge Branch is
false.

If neither the product license (ArcInfo) nor any of the two extensions are available the tool does not execute.

The example uses the CheckExtension acrpy function to check the availability of the extension and CheckOutExtension function to check out the extension for use.

If Extension Exists Condition

Expression

getExtension()

Code Block

def getExtension():
import arcpy
if arcpy.CheckExtension("Spatial") == "Available":
    arcpy.CheckOutExtension("Spatial")
        return "true"
elif arcpy.CheckExtension("Geostats") == "Available":
    arcpy.CheckOutExtension("Geostats")
        return "true"
else:
    return "false"

————————————————————————————————————————————————————————–

Example 3 – If Feature Class Exists
In this example the Calculate Value tool checks if a feature class exists in a workspace and if it exists, adds a field, adds a string value of “Existing “using the Calculate Field tool. If it does not exists the model copies the feature class from a backup location, adds a field, calculates that field with values of the date and time that it was copied and then the two feature classes are input to Merge Branch tool. Merge Branch examines which one has been updated with a valid value and outputs the feature class to merge with the other two feature classes.

The example uses the Exists arcpy function to check if the feature class exists.

If Feature Class Exists Condition

Expression

x("%FeatureClassToCheck%", "%Workspace%")

Code Block

def x(FeatureClassToCheck, Workspace):
import arcpy
arcpy.env.workspace = Workspace
if arcpy.Exists(FeatureClassToCheck):
    return "true"
else:
    return "false"
If Feature Class Does Not Exist Condition

Expression

x("%FeatureClassToCheck%", "%Workspace%")

Code Block

def x(FeatureClassToCheck, Workspace):
import arcpy
arcpy.env.workspace = Workspace
if arcpy.Exists(FeatureClassToCheck):
    return "false"
else:
    return "true"

————————————————————————————————————————————————————————–

Example 4 – If the input is Point, Polyline or Polygon Features Class
If you want to find whether the input feature class is a point, line or polygon you can use a simple if logic in the Calculate Value tool. For example if the input feature class is a polygon then the Feature To Point tool converts the input into a point features class and uses it for a network analysis. If the input is points, it uses the point feature class directly into the network analysis. You can check if the input is a Line or a Point in the similar way as shown in the example below.

The example uses the Describe function to describe the shape type of the data.

If Feature Type Exists Condition

Expression

x(r"%InputFeatureClass%")

Code Block

def x(InputFeatureClass):
import arcpy
desc = arcpy.Describe(InputFeatureClass)
type = desc.shapeType
if type == "Polygon":
    return "true"
else:
    return "false"

————————————————————————————————————————————————————————–

Example 5 – If Feature Type Exists
You can check whether the input to your model is a shapefile or a feature class from a geodatabase using the Calculate Value tool as shown in this example. It checks to see if the input is a shapefile then adds a field and calculates area before calculating population density for the block groups. If it is a feature class from a geodatabase the area field already exists and the model can directly proceed to calculate population density. You can extend the same logic to find whether the input is a layer file or a feature layer in the map’s table of content.

The example uses the Describe to describe the
data type of the data.

If input is a Shapefile Condition

Expression

x(r"%InputData%")

Code Block

def x(InputData):
import arcpy
desc = arcpy.Describe(InputData)
type = desc.datatype
if type == "ShapeFile":
    return "true"
else:
    return "false"
If input is a Feature Class Condition

Expression

x(r"%InputData%")

Code Block

def x(InputData):
import arcpy
desc = arcpy.Describe(InputData)
type = desc.datatype
if type == "FeatureClass":
    return "true"
else:
    return "false"
Posted in Analysis & Geoprocessing | Tagged , , , , , , | 6 Comments

Building a Knowledge Infrastructure Article

For those who recieve the monthly ArcWatch e-magazine you’ll know that they contain vast amounts of information bridging across all industries that use GIS.  This month, the Director of Utility Solutions, Bill Meehan wrote a great article outlining what it takes to capture the large amount of information the aging workforce has in a way that’s defendable, repeatable, and editable.  You can check it out here.

Posted in Uncategorized | Tagged , , , , , , | Leave a comment