Geoprocessing : advanced

Lidar Solutions in ArcGIS_part6: Updating a portion of a terrain dataset with new measurements

bbicking1 — Thu, 02 Jul 2009 17:36:00 GMT

This is the sixth blog in a series on Lidar Solutions in ArcGIS. The author of this blog is Clayton Crawford, lead Product Engineer on ESRI's 3D Analyst team in the Software Products group in Redlands.

Updating a portion of a terrain dataset with new measurements

The ability to update a surface is important to people responsible for providing accurate, up to date surfaces and people performing analysis on those surfaces. Updates come in different forms:

Adding ancillary data (e.g., breaklines)
Removing or replacing bad data
Using newer or more accurate data
Increasing extent with additional data
Inserting design/modeled data to perform ‘what-if’ analysis

These kinds of updates are best performed on the measurements used to construct a surface rather than on derivatives like raster DEMs. Those can be recreated as needed after the measurement edits have taken place. Terrain datasets support this editing model because they maintain a direct link to the source measurement data. When you modify the measurements, you are automatically modifying the terrain in the same process.

How terrains are edited

Editing a terrain dataset is really about editing measurements. Using standard feature edit tools, you can manipulate the measurements that reside in feature classes that participate in a terrain.

Terrains are made from one or more feature classes with some simple rules for each that control how they get used to shape the terrain surface. For example, a multipoint feature class containing lidar points can get added as mass points, a line feature class containing streams and lake shores is used as a source of breaklines, and a polygon feature class can control the data area boundary.

Most feature classes used to define a terrain are what we call referenced. This means that the terrain maintains a pointer, or handle, to them. The terrain prevents its referenced feature classes from being deleted, and pays attention to any edits that occur on them including the addition, deletion, or modification of feature geometry. You can use the feature editor in ArcMap as well as geoprocessing tools to modify these feature classes. A terrain will automatically flag itself as ‘dirty’ in areas where edits were made. Then the terrain can be rebuilt to bring its pyramid in sync with the updated features. It does this based on the dirty areas so it’s a local, or partial, process; the entire terrain does not need to be reconstructed.

Multipoint feature classes have the option of being embedded. When multipoints are embedded the terrain build process copies the points into pyramid tables held private by the terrain and it becomes the container for the points. The terrain does not reference the source feature class. That can be deleted, allowing you to retrieve what is typically a substantial amount of disk space; approximately 1GB per 150 million points. Terrain specific tools, Add Terrain Points (which can both add and replace) and Remove Terrain Points, are used to edit the embedded points based on an area of interest. These tools also offer the benefit of being BLOB attribute aware so if you have any LAS attributes stored with those multipoints (for more on this topic see blog on creating intensity images) the tools know how to keep the BLOB based values in sync with the points relative to the edits. For example, if a few vertices of a multipoint are deleted from an embedded feature class, the terrain will delete the corresponding BLOB based attribute values for those points.

Appending measurements

Measurements can be added to a terrain via the Append and Add Terrain Points geoprocessing tools. The Append tool operates on regular (referenced) feature classes. The Add Terrain Points tool is used to add or replace points in embedded feature classes. You can also add a feature class to an existing terrain via the Add Feature Class To Terrain tool but be aware that this is treated as a schema edit that invalidates the entire terrain, requiring a full rebuild. If data is to be added incrementally, it’s best to append it to a feature class that already participates in the terrain than add a new feature class to the terrain for each new set of data.

Let’s take a scenario where data is provided in phases. In this example the bare earth lidar points are made available first. The breaklines come later in several deliveries. Knowing this schedule, you can create a terrain referencing the lidar multipoint feature class plus an empty line feature class held in anticipation of the breaklines. See Figure 1 with a zoomed-in view of a terrain made solely with bare earth lidar points.

When some of the breaklines are made available they are added to the terrain by adding them to the line feature class referenced by the terrain. This is done using the geoprocessing Append tool (Figure 2).

After running Append, the terrain will become ‘dirty’ in the areas where the lines were added. To see the dirty areas you add a dirty area renderer from the Symbology tab of the terrain layer Properties dialog (Figures 3a and 3b).

At this point the terrain needs to be rebuilt. This is done using either the Build Terrain geoprocessing tool or the Build button on the Update tab of the terrain Properties dialog in ArcCatalog. Once the terrain is re-built the improvement made by the breaklines is evident (Figure 4).

Replace

With lines and polygons in referenced feature classes the replacement of measurements is a two step process. First you delete the old then append the new. If you’re only dealing with a handful of features then you can select and delete them using the Editor in ArcMap. For larger collections rely on geoprocessing tools. For example, use Select By Location followed by Delete Features and Append.

It’s easiest to replace lidar points if they’re embedded. There’s a geoprocessing tool called Add Terrain Points that has a Replace option. This will replace all the points within a given area. So, if you discover something was wrong with a few source point files that were used to build a terrain you can replace them without needing to rebuild the entire terrain from scratch. Figure 5 shows an example where one area, in an otherwise bare earth model, that was inadvertently loaded with first return data.

To fix this problem load the replacement data into a new multipoint feature class. Then run the Add Terrain Points tool with the Replace option. By default, the replacement area will come from the extent of the input feature class (Figure 6).

Once the points have been replaced the terrain needs to be rebuilt to update the affected area. Run the Build Terrain geoprocessing tool or use the Build Terrain button on the Update tab of the terrain Properties dialog in ArcCatalog – either one fixes the terrain.

Conclusion

There are times when updates to a surface model are needed, be it for quality improvement or what-if scenario analysis. It’s hard to make these types of updates to derivative products like raster DEMs without ending up with some anomalies around the update area. It’s more appropriate to modify the source measurement data from which the surface model is derived. For larger datasets, like those coming from lidar, it’s also desirable that datasets only be reprocessed where the updates occur rather than rebuilding everything. Terrain datasets support this by maintaining links to their source measurements in the geodatabase and their use of dirty areas.

That concludes part 6 of Lidar Solutions in ArcGIS. Subscribe to this blog or check back in a month or so for a discussion on how to minimize noise from lidar for contouring and slope analysis.

Lidar Solutions in ArcGIS_part5: Creating Intensity Images from Lidar

bbicking1 — Tue, 12 May 2009 20:50:00 GMT

This is the fifth blog in a series on Lidar Solutions in ArcGIS. It was written by Clayton Crawford, Product Engineer on the 3D Analyst team in ESRI’s Software Products group in Redlands.

Creating Intensity Images from Lidar

Intensity is a measure, collected for every point, of the return strength of the laser pulse that generated the point. It’s based, in part, on the reflectivity of the object struck by the pulse. Other descriptions for intensity include ‘return pulse amplitude’ and ‘backscattered intensity of reflection’. Keep in mind, the reflectivity is a function of the wavelength used which is most commonly in the near infrared. Intensity is used as an aid in feature detection and extraction, lidar point classification, and as a substitute for aerial imagery when none is available. If your lidar points include intensity values you can make images from them that look something like black and white aerial photos (Figure 1).

Loading points with intensity

You'll need source lidar in either LAS or ASCII XYZI format. It’s common to use first returns. If your data are in LAS format use the LAS To Multipoint tool with Intensity selected as an attribute to import (Figure 2). If your data are in ASCII XYZI format use the ASCII 3D To Feature Class tool.

The data loading process results in a multipoint feature class with an attribute field containing the intensity values (Figure 3). You can't read these directly because they're packed into BLOBs (Binary Large OBjects). Every vertex of a multipoint has its intensity mapped into the BLOB stored for that multipoint record.

Substituting intensity for z

A custom VBA script is used to replace every multipoint vertex z with intensity. It does this on a copy of the data. Find the script here; to use it follow the instructions included in the script.

For the ArcObjects geeks out there, take a look at the code to see what it’s doing. Note, there’s a TerrainBlobReader object used to decode the BLOBs. See also that the multipoint vertex ID’s (automatically assigned during import) are used to map into the BLOB value arrays. For the adventurous, note that ArcObjects also provides a TerrainBlobWriter object. If you have a lot of points that need to be stored as multipoints for the sake of efficiency, but also need per-point attribution, these utility objects come in handy.

Another potentially useful script converts, or explodes, multipoints for an area of interest (AOI) into points. Its real utility is that it decodes the lidar attributes stored in BLOBs into readable/usable numeric values placed in the output point feature class attribute table. A word of warning: make sure your AOI is not too large. Keep it small enough so it contains fewer than roughly three million points (adjust according to how patient you consider yourself to be). Find this script here.

Note: Starting with ArcGIS 9.4, the portion of the workflow involving the substitution of intensity for z will not be necessary. The Point to Raster tool will be able to rasterize using the intensity BLOB field directly.

Rasterize the points

Use the Point to Raster tool on the feature class created by the script that swaps intensity with z and set the Value field to Shape (Figure 4). This tells the tool to rasterize using the shape z's which we know are actually intensity. Select MEAN as a rasterization method. The resulting raster is the intensity image. [As a side note: you may find one of the other options is useful for analysis (e.g., using RANGE of intensity as a variable used in feature detection)].

Before moving on, review the presence of NoData. Significant numbers of NoData cells will result if the output cellsize you specified is too small relative to the density of the lidar points. You can see NoData by assigning a color to it on the Symbology tab of the raster layer Properties dialog. If there’s too much, the easiest thing to do is go back and re-run Point to Raster with a larger cellsize. Alternately, you can use a method described in a previous post to fill in missing values using an expression in the Spatial Analyst calculator (see Lidar Solutions in ArcGIS_part2).

Image Display

The range of values in the image is hard to predict without knowing details about how your data was collected and processed. For one thing, the original intensity values are sensor dependent. Secondly, the values may have been adjusted by the vendor (e.g., normalized to a range of 0-255). Because of this it’s hard to say what the best display options are. You will need to experiment with the raster layer stretch type and contrast. Turning on bilinear re-sampling is probably a good idea. If you’re looking for more display possibilities consider combining intensity with another variable such as hillshade (Figure 5).

Conclusion

The return intensity collected for every lidar point can be used to make images. These images have a variety of uses in GIS applications including feature detection and extraction. ArcGIS provides tools to make these images.

That concludes this part of Lidar Solutions in ArcGIS. Subscribe to this blog or check back in a month or so for a discussion on how to update terrain datasets with new measurements.

2009 Developers Summit

dmhoneycutt — Wed, 25 Mar 2009 23:54:00 GMT

The 2009 Developers Summit has come and gone. Thanks to all who attended! We enjoyed meeting with you. We've uploaded the geoprocessing presentations to the Model and Script tool gallery. Session videos have been posted to the Media Gallery.

Slides, scripts, and data from the two workshops on Designing and Building geoprocessing tools are available in the Model and Script tool gallery, or by clicking here.

Slides and other materials from the Building Geoprocessing Services session are available here.

Slides and scripts for the Python Scripting Advanced Techniques session are available here.

Send us your feedback

dmhoneycutt — Wed, 25 Mar 2009 23:47:00 GMT

You may not have noticed, but you can send us feedback about this site. There's a feedback link at the bottom right of the home page, as illustrated at the bottom of this post.

We're particularly interested in hearing from you about blog post topics you'd like to see and suggestions on how we could make this site more useful.

Tips and tricks - accessing feature shape in Calculate Field

dmhoneycutt — Fri, 06 Mar 2009 23:44:00 GMT

With the Calculate Field tool, you can easily create expressions that use some property of a feature's shape, such as length or area. You can also convert length and area to different units. The illustration below shows calculating the MILES field to the length (in miles) of each line feature.

Details about the Expression syntax

The expression syntax -- !shape.length@miles! -- starts and ends with the special 'decode field' character. This decode field character is different depending on the value of the Expression Type parameter. For Expression Type of PYTHON or PYTHON_9.3, the character is an exclamation point ("!"), as shown in the illustration above. For VB, the decode field characters are "[" and "]".

Inside the decode field characters, you can put the name of any field found on the input table. You can also use the reserved field name "shape" followed by a period and an attribute of the shape. In this case, we're using the length attribute: !shape.length!

More things you can do with the shape object and its attributes

The shape field is actually a geometry object with the following properties.

If, for example, you wanted the X-coordinate of the first point of a line, you would use !shape.firstpoint.x! To find the minimum X-coordinate of the line, use !shape.extent.xmin!

Simple equations

The expression can contain simple equations. For example, to calculate minutes it would take to traverse a street segment going 30 miles per hour, use:

!shape.length@miles! / (30.0 / 60.0)

Note that:

!shape.length@miles! / (30 / 60)

won't work - when doing real arithmetic you need the decimal point after 30 and 60.

Code blocks

See http://www.esri.com/news/arcuser/0507/files/pythonscript.pdf for information on using code blocks in Calculate Field

ModelBuilder posting at the Mapping Center

dmhoneycutt — Fri, 06 Mar 2009 00:57:00 GMT

Here's a great post on the mapping center about ModelBuilder:

What I wish I had known about ModelBuilder before I started using it

Tips and tricks - Unix, Python, and Geoprocessing

dmhoneycutt — Mon, 27 Oct 2008 22:41:00 GMT

This posting was written by Mark Zollinger, a Product Engineer and long time UNIX user with the Geoprocessing team.

Do you like Solaris or Linux? Do you "do GIS" on one of those platforms? Maybe you've been asked to convert some ArcGIS Workstation AML applications over to ArcGIS Engine. Are you a command-line junkie? (There are more of us than you might think)

If, for one of the above or any other reason, you find yourself wondering if you should try developing GIS on Solaris or Linux, then it's time to stop wondering. Join in with others who include a flavor of *nix in their list of working development platforms.

There are several approaches to GIS development that work very well on Solaris and Linux.

If you are into low-level programming, you should consider Java or C++, both of which can access ArcGIS's ArcObjects API. I've used Java quite a bit over the years, and can tell you that the doors are wide open on what you can do with it. If this sound like the thing for you, then you can move along now, there's nothing to see here. This post isn't about ArcObjects at all.

If you lean towards scripting and higher level abstractions, then geoprocessing's more coarse-grained object model, then you are in the right place. The rest of this post will show you how to set up an IDE for Python development and write a very simple geoprocessing script.

I mentioned a moment ago that I've written my share of Java ArcObjects code. I've got to confess though, that I've always been a fan of scripting languages. Python and GP is where I spend most of my time these days.

Setting up

"Okay", you ask, "How do I start writing GP code in Python on non-Windows platforms?"

You can do Python and GP from the command line, and I often prefer the focus this brings. I've found, however, that most folks prefer to work in a graphic IDE (Integrated Development Environment).

Setting up an IDE that you can use

Probably your best choice is the Eclipse IDE with the Pydev plugin. Eclipse is an Open Source IDE, intended to work with a wide variety of programming languages. Pydev is the plugin that specifically enables Python development in Eclipse.

The latest version of Eclipse, named Ganymede, has not been fully tested by ESRI for use with ArcGIS 9.3, and is not available for some older UNIX releases. We recommend using version 3.3.2, which you can find in the archive at:

http://archive.eclipse.org/eclipse/downloads/

To install Eclipse, follow the instructions in the "Installing Eclipse" paragraph at:

http://wiki.eclipse.org/FAQ_Where_do_I_get_and_install_Eclipse%3F

(These instructions are for Ganymede, but they basically the same for version 3.3.2)

Once you have Eclipse installed and running, you need to add the Pydev plugin. Just follow the "Getting Started" link on the left side of the Pydev homepage.

http://pydev.sourceforge.net/

Note that you need the Open Source Pydev plugin. The Pydev Extensions plugin is commercial trialware, and is optional.

When you get to the "Configuring the interpreter" page, you are told to choose the Python interpreter that Pydev should use to run Python programs. Use the Python installation you use for Engine development/testing. In most cases, this is the one that got installed with the Engine Runtime:

$ARCGISHOME/python25/bin/python

Note: you might get an error message while trying to choose the interpreter. If you do, click the error dialog's Details button. The message probably includes something like this:

ld.so.1: python: fatal: libpython2.5.so.1.0: open failed: No such file or directory

This is an indication that the python interpreter is not correctly set up. The most likely cause is that your ArcGIS Engine environment has not yet been established. Exit Eclipse and source the init_engine shell script that applies to the shell you are using. When you start Eclipse the problem should go away.

Also, item 3 on the "Configuring the interpreter" page asks you to set the SYSTEM PYTHONPATH. When you do this, make sure to include $ARCGISHOME/bin. This matches the PYTHONPATH that gets set when you source init_engine.

Writing a Python script that does GP stuff

Now that you have a development environment, how do you use all this great GP stuff? Let's create a very simple Python script to get you started on the road to GP development. In most Python scripts, the first thing to do is to import the modules that you will need. For GP, we need to import the arcgisscripting module. Let's also import the os and system modules:

import arcgisscripting
import os, sys

To access GP, we need to create a Geoprocessor object. This is the object through which you will call any GP functions or objects

gp = arcgisscripting.create(9.3)

By passing in the 9.3 argument, you are getting the newer version of the Geoprocessor object, which is a little more Python friendly. For this example we won't be using any of the Python friendly features, but it is never too early to start a good habit.

Now that you have a GP object, what do you do with it? Let's set GP's workspace property and print it out, just to prove to ourselves that it really worked. For this example, let's set the workspace to a "data" directory just underneath the current Python directory.

gp.workspace = os.path.join (sys.path[0], "data")
print "GP Workspace = " + gp.workspace

(Of course you can set the workspace to any accessible directory, not just the one returned by sys.path[0]. I'm just using sys.path[0] because I know it's accessible.)

sys.path is a list of strings where Python looks for modules. The first item in the list is the current Python directory. Calling os.path.join is a great way to build up filesystem paths, since it correctly handles all the path separator issues. (More on path separators later.) When you run this script (Run -> Run As -> Python Run), you should see "GP Workspace= /some/path/data" printed in the Console tab near the bottom of the Eclipse window.

You can use Eclipse or the operating system to create the data directory, since it probably doesn't exist yet.

For this example, let's suppose you have point data representing hawk sitings in a national forest. Your study area is a subset of the forest where a particular plant in known to grow. What you need to do is clip your point data using the polygon of your study area.

Once you have copied your data into the aforementioned data directory, your script could execute the clip tool like this:

gp.clip_analysis("nat_for.gdb/hawk_sightings", "study_area.shp", "nat_for.gdb/study_sightings")

Just to be sure it worked, you can print out the messages that the tool generated like this:

print (gp.getmessages())

Notice that in calling the clip tool, we used forward slashes (/). Back slashes (\) and forward slashes work equally well in ArcGIS on ALL platforms, so you don't need to worry about switching back and forth when you change to Windows. You can even use back slashes on Linux and Solaris. The problem is that in Python, back slashes are a special character, so to use them, you have to treat them differently than most other characters.

If you stick to forward slashes on all platforms, you improve the look of your code and reduce the effort it takes to type it.

So now you have a script that looks like this:

import arcgisscripting
import os, sys

gp = arcgisscripting.create(9.3)

gp.workspace = os.path.join (sys.path[0], "data")
print "GP Workspace = " + gp.workspace

gp.clip_analysis("nat_for.gdb/hawk_sightings", "study_area.shp", "nat_for.gdb/study_sightings")
print (gp.getmessages())

Run it, and it works, creating a new featureclass in the file geodatabase. Now wasn't that easy? To get more complicated, you just string together as many of those tools as you need to get the job done.

Welcome to the world of Geoprocessing with Python!

Links

Automating workflows with scripts - Introduction to writing Python scripts for geoprocessing
Techniques for sharing Python scripts - Useful techniques for sharing Python scripts
The Python Tutorial - If you're new to Python, this is a good tutorial to start
How to duplicate AML functions in Python - For AML programmers
How to duplicate AML directives in Python- For AML programmers

Troubleshooting

Color issues

Upon using Eclipse on Solaris for the first time, you may see some color problems. For example, the text insertion caret might be white on a white background, making it invisible. Another common symptom is that you cannot see the outline of text input fields. If you run into this or something similar, check the color depth of your display. If it is set to 8 bits, change it to 24.

init_engine

You should have your Engine environment already established in the shell in which you start Eclipse. (source init_engine.sh or source init_engine.csh) If you see inexplicable error messages, this is the most likely cause.

Tips and Tricks - Creating ArcSDE connection files on the fly using Python and ArcObjects

dmhoneycutt — Wed, 24 Sep 2008 22:57:00 GMT

ArcGIS 9.3 does not provide a geoprocessing tool for creating ArcSDE connection files (.sde files). Some of you have requested such a tool for security reasons - typically because you do not want .sde connection files hanging around on machines. Rather, you want to automatically create a connection file in a temporary location, use it in your script or model, and then delete it within the process you are running. Others need the ability to create .sde files on the fly because the connection properties, such as what version to connect to, are not known at the start of the process. Having to stop the process for someone to manually create the .sde file with the correct properties really isn't an option anyone wants to entertain.

The attached .zip file (download .zip file) contains a java class (CreateSDEConnFile) and a python script (CreateSDEConnFile.py) to create ArcSDE connection files on the fly. Download and unzip the file. Be sure to unzip the file into a folder that does not contain spaces. For example:

E:\SDE Conn sample (incorrect - contains spaces)
E:\SDEConnectionSample (correct - no spaces)

Java must be installed

You must have Java installed in order to use the files included in the .zip file. Most likely, Java is installed on your machine -- it gets installed by many websites. You can check by examining your installed programs list. You can install java by visiting java.com.

Setting JAVA_HOME

In order for CreateSDEConnFile.py file to execute, you must have the JAVA_HOME system environment variable set to the location of the java install directory.

On Windows XP, right-click "My Computer" on your desktop, click the Advanced tab, and click the "Environment Variables" button. If the JAVA_HOME environment variable doesn't exist, create it by clicking the "New" button. (On my machine, JAVA_HOME = C:\Program Files\Java\jre1.6.0_07)

About CreateSDEConnFile.py Python script

CreateSDEConnFile.py is a sample script. You will need to modify it to include details about your ArcSDE configuration and security needs. There is no associated geoprocessing script tool for CreateSDEConnFile.py -- you will have to create your own script tool for use in ModelBuilder (See An overview of creating script tools in the web help for more information about creating script tools.)

Arguments to the CreateSDEConnFile java class

String sConFileDir = args[0]; //Folder path to store the .sde file.
String sConFile = args[1]; //.sde file name. include the .sde extension.
pConnProps.setProperty("SERVER", args[2]); //ArcSDE Server name.
pConnProps.setProperty("INSTANCE", args[3]); //ArcSDE port number.
pConnProps.setProperty("USER", args[4]); //ArcSDE user name.
pConnProps.setProperty("PASSWORD", args[5]); //ArcSDE password.
pConnProps.setProperty("VERSION", args[6]); //Version to connect to.
pConnProps.setProperty("DATABASE", args[7]); //Database to connect to.

zip file

Download zip file

Watch ESRI TV

dmhoneycutt — Tue, 09 Sep 2008 17:17:00 GMT

Check out http://www.youtube.com/esritv. It contains over 50 videos on a variety of ArcGIS subjects. Content is updated periodically. You can also access the videos on Viddler: http://www.viddler.com/explore/esri

Digging deeper - Troubleshooting geoprocessing errors when using ArcSDE data

dmhoneycutt — Fri, 05 Sep 2008 23:13:00 GMT

Geoprocessing error reporting has significantly improved in the ArcGIS 9.3 release. All tractable errors now have a unique error code and each error code has a detailed explanation in the help system. See Understanding geoprocessing tool errors and warnings for more details.

However, at this time, geoprocessing errors that occur when reading or writing ArcSDE/DBMS data receive a generic 'catch-all' error message, such as error 00210 when writing output:

ERROR 000210: Cannot create output E:\Infra\ToolData\Connection to ArcSDE.sde\ken.cities

We are working hard to improve reporting of ArcSDE/DBMS geoprocessing errors in future service packs and releases. In the meantime, here are some techniques you can use to determine the cause of your error. Note that the cause of the error could be anything from not being able to connect to ArcSDE to any number of DBMS or network related issues.

Test the connection

The first thing you should test is whether your ArcSDE/DBMS connection properties are correct. Open ArcCatalog and navigate to the location containing the .sde connection file ("E:\Infra\ToolData" in the above example) and double-click the connection, causing a connection to be made to the database. If a connection cannot be made, you'll receive an error message with details on the cause. If you can make a connection, then your connection properties may be invalid. Right-click the connection and select Connection Properties. Visually inspect the connection properties. Make sure that your script or model is using the correct user name when fully qualifying object names. For example, you may have connected to the database as the user 'ken' in the connection file, but the data paths in the script or model has a different user for the qualified feature class name, as illustrated below.

If everything seems correct with your connection file, the next investigation step is to set up a client-side SDEINTERCEPT.

Client-side SDEINTERCEPT

You may need to dig deeper by instructing ArcGIS to log information about the communication between your computer and the ArcSDE server. To instruct ArcGIS to log information, and where to write the logs, you need to set certain system environment variables. The exact technique for setting these variables depends on whether you are using Windows or UNIX. For a full explanation of the variables and how to set them, see the web help topic The dbinit.sde file

(For a more general introduction to configuring ArcSDE geodatabases, see An Overview of configuring an ArcSDE geodatabase .)

A SDEINTERCEPT file contains all the calls to ArcSDE along with their return values. These return values often contain the failure message received from ArcSDE or the error message ArcSDE received from the database. On Windows, the general steps are:

1. Set the following system environment variables:

<folder name> can be any existing folder on your computer. Just be sure to add the slash at the end of the pathname (i.e.: C:\temp\logs\)

On Windows XP, system environment variables are accessed by right-clicking My Computer and clicking Properties. On the Advanced Tab, click Environment Variables, as illustrated below.

2. Restart the application that will run the geoprocessing tool (such as ArcMap) so that it inherits the new system environment.

3. Run the geoprocessing tool or script that caused the failure.

After the task is run, a file named se_intercept.<nnn> is created in the folder specified in the SDEINTERCEPTLOC variable (where <nnn> is a unique number starting at 001). This file will contain error and warning messages. Not all of these errors and warnings are issues you need to worry about; some are a normal part of using ArcSDE and are just warnings that are automatically handled by ArcSDE. It will take some time to learn how to read these files. If necessary, you can get help interpreting the file from ESRI support. A good strategy is to search the file from the bottom up since it's more likely that the real error is near the bottom.

ArcSDE Server error logs

Each entry in an ArcSDE Server error log contains three pieces of information: the time of the message, the name of the client machine, and the message. If you can identify the time your error occurred, you can search the log for any obvious error messages around the time the error occurred on your client machine.

ArcSDE Server error logs are written to the location specified by your SDE administrator, so you will need to contact your SDE administrator to gain access to these logs. (They are written to SDEHOME\etc\<sde_instance_name>.log, such as sde_esri_sde.log).

To have more detailed information in the ArcSDE Server error logs, see the SDEVERBOSE setting in the topic The dbinit.sde file

DBMS logs and trace files

Most database management systems have the ability to log detailed information about database activities and queries. Investigating these logs usually require assistance from your database administrator as a certain amount of database specific expertise is needed to gain useful information from their contents.

DBMS logs contain information on the state of the database and any database related issues that may have occurred. Things such as the database running out of space to store your data, dead locks, database system errors and much more are contained in these files.

DBMS trace files, on the other hand, are files that contain all the SQL statements and their return values for a session in the database. When you request ArcSDE to perform an action, a lot of what is done is the running of SQL in the database. The database trace file will contain all the SQL used by ArcSDE and the messages the database returned when running this SQL. Any unexpected errors can be found in these files.

Tips and Tricks - Debugging Python script tools

dmhoneycutt — Mon, 01 Sep 2008 22:02:00 GMT

If you create your own script tools using Python, you'll appreciate these tips and tricks on debugging.

Coding errors are inevitable and there are two basic ways to find out where the error occurred:

Using print statements to discover bugs is an obvious and common method. Since script tools have access to the tool progress dialog, you can edit your script to include calls to AddMessage(), AddWarning(), or AddError() to print values and checkpoint messages to the progress dialog. Another variation is to use an independent method of returning messages, like the win32ui module's MessageBox method. This method displays a popup dialog. Since you have to click OK on the dialog to continue execution, this method allows you to pace the execution of the script. Here's an example using both methods:

import arcgisscripting, win32ui, win32con
gp = arcgisscripting.create()
n = 5

# Print message to progress dialog
#
gp.AddMessage("Value of n = " + str(n))

# Issue a popup dialog with OK and Cancel button
#
val = win32ui.MessageBox("Value of n = " + str(n), "title", win32con.MB_OKCANCEL)

# Based on the button clicked, you can branch execution
#
if val == 1:
    gp.AddMessage("You clicked OK")
else:
    gp.AddError("You clicked Cancel")
    raise arcgisscripting.ExecuteError, "Execution stops due to Cancel button click"

gp.AddMessage("This statement reached")

The other method is to use a Python IDE (Integrated Development Environment) that supports debugging, such as:

https://sourceforge.net/projects/pywin32

http://wingware.com

Debuggers allow you to set breakpoints, step in, out, and over individual lines of code, and examine the contents of variables, all without modifying your code. Compared to inserting print statements, debuggers are much more efficient and usually allow you to quickly isolate your bug. The illustration below shows how to use debug mode in PythonWin.

One way to use a debugger is to open your script directly in your IDE, modify it so that all parameters have values, and then proceed with debugging. This works reasonably well in simple cases. However, if your script uses layer or table view parameters, these variables need created on the fly. Complex parameters like a Field Map or Spatial Reference are hard to create as variables.

Ideally, you'd like to be able to open your script tool dialog, enter parameters, and then have the Python IDE launch with your code ready to be debugged. You can do this with a few simple changes, described below.

Use GetParameterAsText()

The first step is to modify your script so that it used GetParameterAsText() instead of sys.argv[], as discussed in Understanding script tool parameters

This is a modification you can keep - there is no need to change your code back to using sys.argv[].

Create a .bat file that launches your IDE with your .py as an argument

The next step is to create a one-line batch file that will be the source of your script tool (instead of your .py file). This single line contains the pathname to your IDE plus the full path to the .py file, as illustrated below.

Below are 3 examples of how to set your batch (.bat) file for commonly used IDEs. You'll need to confirm that the path is correct for your computer. Other debuggers are of course valid options, and can be called in a similar fashion.

Python IDLE
"C:\Python25\pythonw.exe" "C:\Python25\Lib\idlelib\idle.pyw" "C:\mytools\myscript.py"

PythonWin
"C:\Python25\Lib\site-packages\pythonwin\Pythonwin.exe" "C:\mytools\myscript.py"

Wing IDE 2.1
"C:\Program Files\Wing IDE 2.1\bin\wing.exe" "C:\mytools\myscript.py"

Reset script tool path to the .bat file

The next step is to change your script tool properties to use this new batch file. In the ArcToolbox window, right-click your script tool and click Properties. Click the Source tab and change the script file to be the new .bat file you created, as illustrated below. Click OK.

Execute and debug your tool

Now you can open the script tool from the ArcToolbox window, enter the parameters, and click OK. Shortly afterwards, the debugging application specified in your batch file will open with your Python script displayed. At this point, you are free to interact with the debugging application as you would normally. You can set a breakpoint and let the script run to the breakpoint and use any other options the debugger allows. Parameter values you entered in the dialog will get picked up by GetParameterAsText().

When running the script, any interaction you would normally expect between the script and the application will still occur. So, if you're using methods like AddMessage(), AddWarning(), or AddError(), these messages will show up in the application. If you are using progressor methods (new at 9.3), the progressor will be updated in the tool dialog as you walk through your script. Once the script is finished, you can return back to the application by closing your debugger.

Reset the script tool source

Once you have fixed your bugs, you can reset the source to the script file from the .bat file back to the .py file.

Looking ahead to future releases, we will have integrated options that will make this process smoother. But for now, and with a few little tricks, you can easily perform your script tool debugging.

New at 9.3 - Top Ten

dmhoneycutt — Tue, 29 Jul 2008 16:44:00 GMT

We asked the team to give us their Top 10 New Geoprocessing Features in ArcGIS 9.3. The list quickly turned into the top 20, then the top 30, and then we had to stop. It's by no means exhaustive, but we think it includes the major improvements we made that will make your geoprocessing work a whole lot faster and easier.

In no particular order, here's the first top 10 list.

Join Field tool

At last year's UC a common question was how to permanently add fields from one table to another. The solution at the time involved a combination of the Add Join, Add Field, and Calculate Field tools. With version 9.3, the new Join Field tool does all of this in one step, using a fraction of the time. As with Add Join, Join Field accepts two tables, each with a common attribute field. The tool joins the two tables based on this field and you have the option of selecting which additional fields you want to have included in the final output. The Join Field output is a permanent update of the input table or feature class.

See the Join Field tool reference page for more information.

Error messages

We standardized warning and error messages and gave each one a unique number. So now, when a warning or error comes up on a tool dialog, progress dialog, or command line window, you can click the number to open a detailed description. This description lists probable causes of the error or warning, suggestions to remedy the situation, and links to relevant help topics.

See Understanding geoprocessing tool errors and warnings for more information

Near with lines and polygons

Have you ever been required to find the nearest distances between a mixture of point, line, and polygon features? This would seem to be a very easy operation. However, until ArcGIS 9.3, it required building a script or program. With ArcGIS 9.3, the Near tool is improved to support all feature types. In addition, multiple near feature classes can be supplied.

Generate Near Table tool

The Near tool modifies the attributes of the input feature class. This often caused problems when the input dataset was read only. With ArcGIS 9.3, there is a new Generate Near Table tool. Instead of modifying the input feature class as Near does, Generate Near Table creates a new table containing the proximity information. This tool also contains an option to find the distances to all features, not just the nearest.

Performance improvement using ArcSDE data with GpTools

We've made three major improvements when using ArcSDE data with geoprocessing tools:

By far the most dramatic improvement is the result of better management of ArcSDE connections. Instead of creating multiple connections to ArcSDE for one execution of a tool, we now create only one connection. (This issue is also resolved in version 9.2 service pack 5).

The performance of the Append tool has improved because the spatial index of the output dataset is created only once instead of many times.

We made many improvements to our internal topology analysis engine used by overlay tools. The performance of tools in the Analysis toolbox has greatly improved when using large ArcSDE datasets.

Python scripts run in process

In 9.2 all Python [.py] script tools created a separate process and executed outside the application process. The way you noticed this is that it took as much as 5 to 10 seconds for a script tool to start up. In 9.3, we're taking advantage of the application already running. This greatly improves script tool performance in two major areas: Initial start-up of a script tool and the use of cursors within the script tool. It also greatly improves the start-up performance of scripts tools using smaller datasets as inputs.

See Running a script in process for more information.

Geodesic buffers

In 9.2, the Buffer tool created buffer polygons assuming that all coordinates represented a flat Euclidean space instead of a spherical space, like the Earth.

In 9.3, the Buffer tool will create geodesic buffers if:

The features to be buffered are points or multipoints.

The input feature are in geographic coordinates (i.e., longitude/latitude).

The buffer distance is in Euclidean linear units (i.e, feet, meters).

The output buffer polygons will take into account that longitudinal distance varies varies with latitude, as illustrated below.

See How Buffer works for more information on geodesic buffers.

Script tool validation

You can now have your script tools behave like system tools (those that come with ArcGIS). You can enable (gray-out) parameters based on values in other parameters, provide keyword lists that dynamically change, put parameters into different categories, calculate default values on-the-fly, and update the description of output data for use in ModelBuilder. These capabilities are all found on the new Validation tab you see on a script tool's property page.

See Customizing script tool behavior for more information

Spatial Statistics: Ordinary Least Square Regression and

Geographically Weighted Regression Tools

The Spatial Statistics toolbox has powerful pattern analysis tools to help answer "Where?" questions like:

Where are people persistently dying young in the United States?

Where are our kids consistently turning in high test scores?

Where do we seen an unexpectedly high number of traffic accidents?

With ArcGIS 9.3, new regression analysis tools are added that allow you to answer the next logical set of questions relating to "Why?" like:

Why are people persistently dying young in particular places across the United States?

What are the factors contributing to consistently high test scores?

Why are there so many traffic accidents in particular hot spot locations?

Ordinary Least Squares Regression (OLS) is a commonly used global linear regression method used to generate predictions, or to model a dependent variable or process, in terms of its relationships to a set of explanatory variables. It creates a single equation to represent those relationships. OLS is the starting point for all regression analysis (including all spatial regression analysis). It often breaks down when used with spatial data, however, in the case where the relationships being modeled change across the study area. An education variable, for example, might be a good predictor of crime rates in one part of the study area, but may not be significant in another part of the study area. Geographically Weighted Regression (GWR), on the other hand, allows data relationships to vary. GWR is one of several spatial regression techniques increasingly used in geography and other disciplines. It provides a local model of the variable or process you are trying to understand/predict by fitting a regression equation to every feature in the dataset. GWR constructs these separate equations by incorporating the dependent and explanatory variables of features failing with a particular user-specified bandwidth of each target feature. ArcGIS 9.3 includes "Regression Analysis Basics" and "Result Interpretation" documentation with strategies and graphics aimed at helping the user apply these methods effectively.

See An overview of the Modeling Spatial Relationships toolset for more information.

Batch Grid: deferred validation

If you use a tool in batch mode, you may have noticed that filling in the batch grid became slow if more than 10 rows or so were entered. This was due to the fact that every value in every row was validated whenever you clicked the mouse in a cell.

In 9.3, you control when validation occurs by click the Check Values button in the lower right of the batch grid, as illustrated below.

Validation always occurs when you click the OK button.

See Using the batch grid control for more information. You can find the discussion of the Check Values button at the bottom of the topic.

Guidelines for submitting tools to the Model and Script Tool Gallery

dmhoneycutt — Tue, 29 Jul 2008 16:42:00 GMT

The link below is to a PDF document containing guidelines for making a submittal to the Model and Script Tool Gallery. This document discusses:

The contents of the ZIP file you upload when making a submission.

How to ensure that your submission is portable. That is, when someone downloads your submission, all your tools work without modification.

Advice on creating a submission that gets good user ratings because it's well documented and robust.

Guidelines for submitting Model and Script Tools [PDF]

Welcome to the geoprocessing resource center

dmhoneycutt — Tue, 29 Jul 2008 16:41:00 GMT

We're all excited about the launch of this new geoprocessing blog.

The scope of this blog will range from introductory information regarding general geoprocessing functionality to some more advanced topics and developer related material. We’ll be blogging on things like best practices, tips and tricks, new and existing functionality, and example workflows.

Be sure to visit the Model and Script tool gallery where you can download model and script tools developed by the geoprocessing community and submit your own tools.

This blog is written by the Geoprocessing Development Team and we’re hoping that it grows into a valuable resource for you. Stay tuned…