I recently had a project that involved creating lots of Python script tools. Since other people were going to use these tools, I wanted my error handling to be robust and informative. This post is about some tips and tricks I discovered during this project.

Error handling basics

The basics of handling Python errors are covered in the 9.3 help topic Error handling with Python. The Python.org document Errors and Exceptions has more detailed information.

Tip #1 - Use the arcgisscripting.ExecuteError exception

In version 9.2, we introduced the arcgisscripting object along with a new exception class, arcgisscripting.ExecuteError. (This arcgisscripting.ExecuteError exception class wasn't documented in version 9.2, so few people knew about it.)  The arcgisscripting.ExecuteError exception is thrown whenever a geoprocessing tool or geoprocessing function encounters an error. What this means is that you can divide errors into two groups, geoprocessing errors (those that throw the arcgisscripting.ExecuteError exception) and everything else.  You can then handle the errors differently, as demonstrated in the code below:

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:/blah.shp")
    # x = y
        
# Return GEOPROCESSING specific errors
#
except arcgisscripting.ExecuteError:
    gp.AddError(gp.GetMessages(2))
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

The code above is used as the source for a script tool. To keep things simple, the script tool has no parameters. When the script tool is executed, the call to getcount produces an error because the dataset "C:/blah.shp" doesn't exist. As a result, the progress dialog looks as follows:

To prove how geoprocessing errors and Python errors are handled differently, change the two lines of code as follows:

    # result = gp.getcount("C:/blah.shp")
    x = y

Now when the script is executed, an error will occur because the variable 'y' is undefined, and the progress dialog will display as follows:

Tip #2 - Beware of getting error messages from a result object

Before moving on, a quick word about the result object, shown below:

    result = gp.getcount("C:/blah.shp")

If the call to getcount above raises an exception, the result object is null. This means you cannot retrieve error messages from the result object. For example:

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:/blah.shp")
        
# Return GEOPROCESSING specific errors
# (this method is INCORRECT!)
except arcgisscripting.ExecuteError:
    gp.AddError(result.GetMessages(2))
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

The above code will fail with the message "name 'result' is not defined". This is due to the fact that the result object is null. Since the result object is null, a Python error will be raised since a null object doesn't have a GetMessages() method. Note -- a result object created by calling a geoprocessing service on an ArcGIS Server is never null. Null result objects a created only when a tool is run locally and it raises an error.  For more information about using the result object, see Getting results from a geoprocessing tool.

Tip #3 - Use AddReturnMessage() to preserve links to error codes

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:\\blah.shp")
        
# Return GEOPROCESSING specific errors
#
except arcgisscripting.ExecuteError:
    for msg in range(0, gp.MessageCount):
        if gp.GetSeverity(msg) == 2:
            gp.AddReturnMessage(msg)
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

Now when the script is executed, the progress dialog will look as follows:

Tip #4 -- Use traceback to return more information about the error

The help system topic Error handling with Python shows you how to use Python's traceback to retrieve the line number causing the error, the name of the .py file, and, for Python errors, a description of the error.

At version 9.3, you can run script tools in-process, as described in Running a script in process. Scripts run in-process execute much faster than scripts run out-of-process, so you always want to run the script in-process. When a script is run in-process, ArcGIS reads the .py file into memory and then executes it. Because ArcGIS puts the .py files into memory, the Python traceback module doesn't know the name of the .py file and will report the file name as "<string>". If you want to report the name of the .py file causing the error (very useful for debugging), you'll have to provide it yourself.

The script below has a local routine, trace(), that uses traceback to return the line number, the file name, and the error description. The routine also contains the the hard-coded file name (shown in bold) which you must change for your script.

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    def trace():
        import os, sys, traceback
        tb = sys.exc_info()[2]
        tbinfo = traceback.format_tb(tb)[0]  # script name + line number
        line = tbinfo.split(", ")[1]
        
        # Construct the pathname to this script.  Get the pathname
        #  to the folder containing the script using sys.path[0]. 
        #  Modify line below and replace 'myscript.py' with the name of
        #  this script. 
        #
        filename = sys.path[0] + os.sep + "myscript.py"
        
        # Get Python syntax error
        #
        synerror = traceback.format_exc().splitlines()[-1]
        return line, filename, synerror
    if __name__ == '__main__':  
        # y = x
        result = gp.getcount("C:/blah.shp")
        
except arcgisscripting.ExecuteError:
    # Call our trace routine to retrieve line number and filename.
    #  (returned variable 'err' is ignored since this is a gp error.
    #
    line, filename, err = trace()
    gp.AddError("Geoprocessing error on " + line + " of " + filename + " :")
    for msg in range(0, gp.MessageCount):
        if gp.GetSeverity(msg) == 2:
            gp.AddReturnMessage(msg)
    
except:
    line, filename, err = trace()
    gp.AddError("Python error on " + line + " of " + filename)
    gp.AddError(err)

Below are illustrations of the progress dialog showing the line number and file name for both a geoprocessing error and a Python syntax error (by commenting out the "y = x" statement).

Tip #5 -- Use Finally statement to clean up cursors

Something I discovered by reading the Python doc Errors and Exceptions is the 'finally' statement. No matter what happens in your Python script, code in a 'finally:' block always gets executed. It's a great place to put clean-up actions, such as deleting cursors, as shown in the code below.

import arcgisscripting
gp = arcgisscripting.create(9.3)

def trace():
    import os, sys, traceback
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0] 
    line = tbinfo.split(", ")[1]
    filename = sys.path[0] + os.sep + "testscript.py"
    synerror = traceback.format_exc().splitlines()[-1]
    return line, filename, synerror

if __name__ == '__main__':
    # Define variables for a cursor and its row
    #
    cursor, row = None, None
    try:
        cursor = gp.searchcursor("E:/Data/CityOfSanFrancisco.gdb/FireStations")
        row = cursor.Next()
        while row:
          # Code here...
          # for demonstration...this next statement will cause an error
          y = x
          row = cursor.Next()
    except arcgisscripting.ExecuteError:
        line, filename, err = trace()
        gp.AddError("Geoprocessing error on " + line + " of " + filename + " :")
        for msg in range(0, gp.MessageCount):
            if gp.GetSeverity(msg) == 2:
                gp.AddReturnMessage(msg)
    except:
        line, filename, err = trace()
        gp.AddError("Python error on " + line + " of " + filename)
        gp.AddError(err)
    finally:
        # Close cursor and row objects.
        #
        del row, cursor
            
        # Print a message for demo purposes
        #
        gp.AddMessage("\n   ** Cursor and row have been deleted ** \n")

As illustrated below, the cursor gets deleted even if there is an error in the script.

In the main body of the script, the cursor and row object variables are declared and initialized to None (shown in bold in the above code).  This ensures that these two variables exist when the del statement in the finally clause is executed; if you try to delete a non-existent variable, an exception will be thrown.  

About cursors and the del statement

The Python del statement deletes a variable. You can use it to delete any Python variable, such as a list or dictionary. However, you rarely need to delete native Python variables.  The del statement is mainly used for deleting non-native variables, such as geoprocessing cursors and rows obtained from the cursor. When you delete a cursor and its row, all pending changes to the row (caused by an update or insert cursor) are flushed and all locks on the dataset are removed.  You should always delete cursors and rows obtained from the cursor.  When deleting a cursor, delete the row object first, then the cursor.

This posting was written by Mark Zollinger, a Product Engineer and long time UNIX user with the Geoprocessing team.

Do you like Solaris or Linux? Do you "do GIS" on one of those platforms? Maybe you've been asked to convert some ArcGIS Workstation AML applications over to ArcGIS Engine. Are you a command-line junkie? (There are more of us than you might think)

If, for one of the above or any other reason, you find yourself wondering if you should try developing GIS on Solaris or Linux, then it's time to stop wondering. Join in with others who include a flavor of *nix in their list of working development platforms.

There are several approaches to GIS development that work very well on Solaris and Linux.

If you are into low-level programming, you should consider Java or C++, both of which can access ArcGIS's ArcObjects API. I've used Java quite a bit over the years, and can tell you that the doors are wide open on what you can do with it. If this sound like the thing for you, then you can move along now, there's nothing to see here. This post isn't about ArcObjects at all.

If you lean towards scripting and higher level abstractions, then geoprocessing's more coarse-grained object model, then you are in the right place. The rest of this post will show you how to set up an IDE for Python development and write a very simple geoprocessing script.

I mentioned a moment ago that I've written my share of Java ArcObjects code. I've got to confess though, that I've always been a fan of scripting languages. Python and GP is where I spend most of my time these days.

Setting up

"Okay", you ask, "How do I start writing GP code in Python on non-Windows platforms?"

You can do Python and GP from the command line, and I often prefer the focus this brings. I've found, however, that most folks prefer to work in a graphic IDE (Integrated Development Environment).

Setting up an IDE that you can use

Probably your best choice is the Eclipse IDE with the Pydev plugin. Eclipse is an Open Source IDE, intended to work with a wide variety of programming languages. Pydev is the plugin that specifically enables Python development in Eclipse.

The latest version of Eclipse, named Ganymede, has not been fully tested by ESRI for use with ArcGIS 9.3, and is not available for some older UNIX releases. We recommend using version 3.3.2, which you can find in the archive at:

To install Eclipse, follow the instructions in the "Installing Eclipse" paragraph at:

(These instructions are for Ganymede, but they basically the same for version 3.3.2)

Once you have Eclipse installed and running, you need to add the Pydev plugin. Just follow the "Getting Started" link on the left side of the Pydev homepage.

Note that you need the Open Source Pydev plugin. The Pydev Extensions plugin is commercial trialware, and is optional.

When you get to the "Configuring the interpreter" page, you are told to choose the Python interpreter that Pydev should use to run Python programs. Use the Python installation you use for Engine development/testing. In most cases, this is the one that got installed with the Engine Runtime:

Note: you might get an error message while trying to choose the interpreter. If you do, click the error dialog's Details button. The message probably includes something like this:

This is an indication that the python interpreter is not correctly set up. The most likely cause is that your ArcGIS Engine environment has not yet been established. Exit Eclipse and source the init_engine shell script that applies to the shell you are using. When you start Eclipse the problem should go away.

Also, item 3 on the "Configuring the interpreter" page asks you to set the SYSTEM PYTHONPATH. When you do this, make sure to include $ARCGISHOME/bin. This matches the PYTHONPATH that gets set when you source init_engine.

Writing a Python script that does GP stuff

Now that you have a development environment, how do you use all this great GP stuff? Let's create a very simple Python script to get you started on the road to GP development. In most Python scripts, the first thing to do is to import the modules that you will need. For GP, we need to import the arcgisscripting module. Let's also import the os and system modules:

To access GP, we need to create a Geoprocessor object. This is the object through which you will call any GP functions or objects

By passing in the 9.3 argument, you are getting the newer version of the Geoprocessor object, which is a little more Python friendly. For this example we won't be using any of the Python friendly features, but it is never too early to start a good habit.

Now that you have a GP object, what do you do with it? Let's set GP's workspace property and print it out, just to prove to ourselves that it really worked. For this example, let's set the workspace to a "data" directory just underneath the current Python directory.

(Of course you can set the workspace to any accessible directory, not just the one returned by sys.path[0]. I'm just using sys.path[0] because I know it's accessible.)

sys.path is a list of strings where Python looks for modules. The first item in the list is the current Python directory. Calling os.path.join is a great way to build up filesystem paths, since it correctly handles all the path separator issues. (More on path separators later.) When you run this script (Run -> Run As -> Python Run), you should see "GP Workspace= /some/path/data" printed in the Console tab near the bottom of the Eclipse window.

You can use Eclipse or the operating system to create the data directory, since it probably doesn't exist yet.

For this example, let's suppose you have point data representing hawk sitings in a national forest. Your study area is a subset of the forest where a particular plant in known to grow. What you need to do is clip your point data using the polygon of your study area.

Once you have copied your data into the aforementioned data directory, your script could execute the clip tool like this:

gp.clip_analysis("nat_for.gdb/hawk_sightings", "study_area.shp", "nat_for.gdb/study_sightings")

Just to be sure it worked, you can print out the messages that the tool generated like this:

Notice that in calling the clip tool, we used forward slashes (/). Back slashes (\) and forward slashes work equally well in ArcGIS on ALL platforms, so you don't need to worry about switching back and forth when you change to Windows. You can even use back slashes on Linux and Solaris. The problem is that in Python, back slashes are a special character, so to use them, you have to treat them differently than most other characters.

- You can escape them: "\\some\\path"
- You can use raw strings: r"\some\path"
- You can use unicode strings: u"\some\path"

If you stick to forward slashes on all platforms, you improve the look of your code and reduce the effort it takes to type it.

So now you have a script that looks like this:

import arcgisscripting
import os, sys

gp = arcgisscripting.create(9.3)

gp.workspace = os.path.join (sys.path[0], "data")
print "GP Workspace = " + gp.workspace

gp.clip_analysis("nat_for.gdb/hawk_sightings", "study_area.shp", "nat_for.gdb/study_sightings")
print (gp.getmessages())

Run it, and it works, creating a new featureclass in the file geodatabase. Now wasn't that easy? To get more complicated, you just string together as many of those tools as you need to get the job done.

Welcome to the world of Geoprocessing with Python!

Links

Troubleshooting

Color issues

Upon using Eclipse on Solaris for the first time, you may see some color problems. For example, the text insertion caret might be white on a white background, making it invisible. Another common symptom is that you cannot see the outline of text input fields. If you run into this or something similar, check the color depth of your display. If it is set to 8 bits, change it to 24.

init_engine

You should have your Engine environment already established in the shell in which you start Eclipse. (source init_engine.sh or source init_engine.csh) If you see inexplicable error messages, this is the most likely cause.

The Spatial Analyst at 9.3 has several new capabilities, improvements, and fixes. This post highlights some of the important changes; for a full description, see the What's New pdf document (Once open, in the Extensions section click on Spatial Analyst. This will take you to the right location in the document.)

New Functionality

Zonal Histogram - Helps you to investigate the distribution of cells from one dataset within classes of another. Example applications include rainfall within watersheds or crime distribution by police beat.

Contour with Barriers - This tool allows you to get better results when trying to contour discontinuous surfaces. In addition to its superior handling of barriers, Contour with Barriers creates smoother contour lines, generate much larger output datasets (no 2.14 GB shapefile size limit), and optionally to attribute primary and index contour lines.

The Snap Raster environment has moved from being part of the Extent environment to being its own separate environment setting. This gives you flexibility to set the snap raster dynamically from scripting or ModelBuilder.

Usability Improvements

The Con tool now allows the input conditional raster to be a floating point raster.

The Create Constant Raster, Create Normal Raster, Create Random Raster,  Extract by Rectangle, and Topo to Raster tools now allow you to specify the extent of the output raster to be that of an existing dataset. The main benefit of this is the extent can be set dynamically from scripting or ModelBuilder. While you can still enter the extent values manually (but who wants to type in 370803.47263702 3978856.7539477 443868.87614745 4032133.0258861 ?), being able to specify a dataset is both easier and helps ensure the proper precision is maintained .

All Spatial Analyst tools have new Geoprocessing tool errors and warnings and show execution percent complete using the new progress bar.

Performance and Data Improvements

Combine and Cut/Fill are significantly faster (up to 40 times).

Viewshed now handles much larger inputs (no more MSEEK fatal errors!).

IDW now works with many more points (up to 50 million).

Combine now works on millions of combinations.

ArcGIS 9.3 does not provide a geoprocessing tool for creating ArcSDE connection files (.sde files). Some of you have requested such a tool for security reasons - typically because you do not want .sde connection files hanging around on machines. Rather, you want to automatically create a connection file in a temporary location, use it in your script or model, and then delete it within the process you are running. Others need the ability to create .sde files on the fly because the connection properties, such as what version to connect to, are not known at the start of the process. Having to stop the process for someone to manually create the .sde file with the correct properties really isn't an option anyone wants to entertain.

The attached .zip file (download .zip file) contains a java class (CreateSDEConnFile) and a python script (CreateSDEConnFile.py) to create ArcSDE connection files on the fly. Download and unzip the file. Be sure to unzip the file into a folder that does not contain spaces. For example:

E:\SDE Conn sample (incorrect - contains spaces)
E:\SDEConnectionSample (correct - no spaces)

Java must be installed

You must have Java installed in order to use the files included in the .zip file. Most likely, Java is installed on your machine -- it gets installed by many websites. You can check by examining your installed programs list. You can install java by visiting java.com.

Setting JAVA_HOME

In order for CreateSDEConnFile.py file to execute, you must have the JAVA_HOME system environment variable set to the location of the java install directory.

On Windows XP, right-click "My Computer" on your desktop, click the Advanced tab, and click the "Environment Variables" button. If the JAVA_HOME environment variable doesn't exist, create it by clicking the "New" button. (On my machine, JAVA_HOME = C:\Program Files\Java\jre1.6.0_07)

About CreateSDEConnFile.py Python script

CreateSDEConnFile.py is a sample script. You will need to modify it to include details about your ArcSDE configuration and security needs. There is no associated geoprocessing script tool for CreateSDEConnFile.py -- you will have to create your own script tool for use in ModelBuilder (See An overview of creating script tools in the web help for more information about creating script tools.)

Arguments to the CreateSDEConnFile java class

zip file

Download zip file

If you create your own script tools using Python, you'll appreciate these tips and tricks on debugging.

Coding errors are inevitable and there are two basic ways to find out where the error occurred:

- Add some form of print statements to your code that help you isolate the problem.
- Use an interactive debugger.

Using print statements to discover bugs is an obvious and common method. Since script tools have access to the tool progress dialog, you can edit your script to include calls to AddMessage(), AddWarning(), or AddError() to print values and checkpoint messages to the progress dialog. Another variation is to use an independent method of returning messages, like the win32ui module's MessageBox method. This method displays a popup dialog. Since you have to click OK on the dialog to continue execution, this method allows you to pace the execution of the script. Here's an example using both methods:

import arcgisscripting, win32ui, win32con
gp = arcgisscripting.create()
n = 5

# Print message to progress dialog
#
gp.AddMessage("Value of n = " + str(n))

# Issue a popup dialog with OK and Cancel button
#
val = win32ui.MessageBox("Value of n = " + str(n), "title", win32con.MB_OKCANCEL)

# Based on the button clicked, you can branch execution
#
if val == 1:
    gp.AddMessage("You clicked OK")
else:
    gp.AddError("You clicked Cancel")
    raise arcgisscripting.ExecuteError, "Execution stops due to Cancel button click"

gp.AddMessage("This statement reached")

The other method is to use a Python IDE (Integrated Development Environment) that supports debugging, such as:

- Python IDE that is installed with Python
- PythonWin, available on the web at https://sourceforge.net/projects/pywin32
- Commercial systems, such as Wing IDE (http://wingware.com)

Debuggers allow you to set breakpoints, step in, out, and over individual lines of code, and examine the contents of variables, all without modifying your code. Compared to inserting print statements, debuggers are much more efficient and usually allow you to quickly isolate your bug.  The illustration below shows how to use debug mode in PythonWin.

One way to use a debugger is to open your script directly in your IDE, modify it so that all parameters have values, and then proceed with debugging. This works reasonably well in simple cases. However, if your script uses layer or table view parameters, these variables need created on the fly. Complex parameters like a Field Map or Spatial Reference are hard to create as variables.

Ideally, you'd like to be able to open your script tool dialog, enter parameters, and then have the Python IDE launch with your code ready to be debugged. You can do this with a few simple changes,  described below.

Use GetParameterAsText()

The first step is to modify your script so that it used GetParameterAsText() instead of sys.argv[], as discussed in Understanding script tool parameters

This is a modification you can keep - there is no need to change your code back to using sys.argv[].

Create a .bat file that launches your IDE with your .py as an argument

The next step is to create a one-line batch file that will be the source of your script tool (instead of your .py file). This single line contains the pathname to your IDE plus the full path to the .py file, as illustrated below.

Below are 3 examples of how to set your batch (.bat) file for commonly used IDEs. You'll need to confirm that the path is correct for your computer. Other debuggers are of course valid options, and can be called in a similar fashion.

Reset script tool path to the .bat file

The next step is to change your script tool properties to use this new batch file. In the ArcToolbox window, right-click your script tool and click Properties. Click the Source tab and change the script file to be the new .bat file you created, as illustrated below. Click OK.

Execute and debug your tool

Now you can open the script tool from the ArcToolbox window, enter the parameters, and click OK. Shortly afterwards, the debugging application specified in your batch file will open with your Python script displayed. At this point, you are free to interact with the debugging application as you would normally. You can set a breakpoint and let the script run to the breakpoint and use any other options the debugger allows. Parameter values you entered in the dialog will get picked up by GetParameterAsText().

When running the script, any interaction you would normally expect between the script and the application will still occur. So, if you're using methods like AddMessage(), AddWarning(), or AddError(), these messages will show up in the application. If you are using progressor methods (new at 9.3), the progressor will be updated in the tool dialog as you walk through your script. Once the script is finished, you can return back to the application by closing your debugger.

Reset the script tool source

Once you have fixed your bugs, you can reset the source to the script file from the .bat file back to the .py file.

Looking ahead to future releases, we will have integrated options that will make this process smoother. But for now, and with a few little tricks, you can easily perform your script tool debugging.

...to Everyone who visited us at the Spatial Analysis Island and attended our Tech Workshops and Island Demos. We do hope you recovered from San Diego and are excited about the new things you learned, the answers you got to your questions, and together all will make your work much easier and more fun.

As always, we also learned a lot from you, about what you like, or don’t like, or want us to do in the future. This is a very special time for us to live vicariously through the great work you have done on top of our work, and understand how to better support you.

Since you’re here, you heard about the just launched Resource Centers, the new gateway to all ESRI online resources. Here’s the link to ours http://resources.esri.com/Geoprocessing/ . Add it to your Favorites and visit the site often, especially the Geoprocessing Blog and the Model and Scripts Gallery [both under Community]. The Model and Script Gallery on the Geoprocessing Resource Center is a bit different from the other resource centers, in that it's not just us posting things, we're inviting you to post your work as well. Whether that be generic utilities or complete workflows for a specific task, if you think it can be useful for someone else, feel free to share it. I talked with several of the more prominent ArcScripts contributors from the past, who are pretty excited about having a more interactive place to post their work and communicate with their users. We have guidelines for submission, which include documentation of your model and script. When putting the documentation together, make sure you include any use limitations/restrictions – such as if it relies on an ArcInfo or extension licensed tool.

Regarding writing scripts, we saw noticeable increase in the number and depth of Python questions at the island this year. Actually, there were a lot of great questions and interesting problems to solve at the island, that have us pretty excited about some things we’ll be doing in future releases that will create a richer Python experience in ArcGIS for you (more on that in the future).

On the other end of the spectrum, we met a lot of new people at the Spatial Analysis Island this year who were attending their first International User Conference, and some who were quite new to the software. For many of them, the team enjoyed demoing ModelBuilder and showing them how to automate their daily routine. It was especially rewarding for those of us who had the chance to introduce many of you to ModelBuilder and hear you say: “Wow, ... I’m gonna use that every day.”

From the more experienced ModelBuilder users, there were quite a few posters with some very sophisticated models. We’ll be reprinting a few of these to decorate the halls around our new offices. Thanks for the nice work.

From the analytical side of the conference, all the interest seemed to be on statistics. Our sessions on spatial statistics, regression, and statistical analysis with Spatial Analyst were generally packed, and lots of in depth discussions at the island on applications of these tools as well as Geostatistical Analyst, and integration with 3^rd party statistical packages. We’ll be posting some samples to the Model and Script Gallery in the near future and working on some webcasts to share more about current capabilities and applications.

We hope you got as much out of the conference as we did, and look forward to interacting with you via the Resource Center until we meet again next year.