Geoprocessing : script tools

2009 Developers Summit

dmhoneycutt — Wed, 25 Mar 2009 23:54:00 GMT

The 2009 Developers Summit has come and gone. Thanks to all who attended! We enjoyed meeting with you. We've uploaded the geoprocessing presentations to the Model and Script tool gallery. Session videos have been posted to the Media Gallery.

Slides, scripts, and data from the two workshops on Designing and Building geoprocessing tools are available in the Model and Script tool gallery, or by clicking here.

Slides and other materials from the Building Geoprocessing Services session are available here.

Slides and scripts for the Python Scripting Advanced Techniques session are available here.

Send us your feedback

dmhoneycutt — Wed, 25 Mar 2009 23:47:00 GMT

You may not have noticed, but you can send us feedback about this site. There's a feedback link at the bottom right of the home page, as illustrated at the bottom of this post.

We're particularly interested in hearing from you about blog post topics you'd like to see and suggestions on how we could make this site more useful.

Tips and tricks - Error handling in Python script tools

dmhoneycutt — Mon, 01 Dec 2008 17:43:00 GMT

I recently had a project that involved creating lots of Python script tools. Since other people were going to use these tools, I wanted my error handling to be robust and informative. This post is about some tips and tricks I discovered during this project.

Error handling basics

The basics of handling Python errors are covered in the 9.3 help topic Error handling with Python. The Python.org document Errors and Exceptions has more detailed information.

Tip #1 - Use the arcgisscripting.ExecuteError exception

In version 9.2, we introduced the arcgisscripting object along with a new exception class, arcgisscripting.ExecuteError. (This arcgisscripting.ExecuteError exception class wasn't documented in version 9.2, so few people knew about it.) The arcgisscripting.ExecuteError exception is thrown whenever a geoprocessing tool or geoprocessing function encounters an error. What this means is that you can divide errors into two groups, geoprocessing errors (those that throw the arcgisscripting.ExecuteError exception) and everything else. You can then handle the errors differently, as demonstrated in the code below:

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:/blah.shp")
    # x = y
        
# Return GEOPROCESSING specific errors
#
except arcgisscripting.ExecuteError:
    gp.AddError(gp.GetMessages(2))
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

The code above is used as the source for a script tool. To keep things simple, the script tool has no parameters. When the script tool is executed, the call to getcount produces an error because the dataset "C:/blah.shp" doesn't exist. As a result, the progress dialog looks as follows:

To prove how geoprocessing errors and Python errors are handled differently, change the two lines of code as follows:

    # result = gp.getcount("C:/blah.shp")
    x = y

Now when the script is executed, an error will occur because the variable 'y' is undefined, and the progress dialog will display as follows:

Tip #2 - Beware of getting error messages from a result object

Before moving on, a quick word about the result object, shown below:

    result = gp.getcount("C:/blah.shp")

If the call to getcount above raises an exception, the result object is null. This means you cannot retrieve error messages from the result object. For example:

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:/blah.shp")
        
# Return GEOPROCESSING specific errors
# (this method is INCORRECT!)
except arcgisscripting.ExecuteError:
    gp.AddError(result.GetMessages(2))
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

The above code will fail with the message "name 'result' is not defined". This is due to the fact that the result object is null. Since the result object is null, a Python error will be raised since a null object doesn't have a GetMessages() method. Note -- a result object created by calling a geoprocessing service on an ArcGIS Server is never null. Null result objects a created only when a tool is run locally and it raises an error. For more information about using the result object, see Getting results from a geoprocessing tool.

Tip #3 - Use AddReturnMessage() to preserve links to error codes

In version 9.3, geoprocessing error numbers shown in the progress dialog are hyperlinks to a help page that further describes the error. To enable hyperlinks for errors in your script, you need to use the AddReturnMessage() method instead of the AddError method, as follows:

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    result = gp.getcount("C:\\blah.shp")
        
# Return GEOPROCESSING specific errors
#
except arcgisscripting.ExecuteError:
    for msg in range(0, gp.MessageCount):
        if gp.GetSeverity(msg) == 2:
            gp.AddReturnMessage(msg)
    
# Return any PYTHON or system specific errors
#
except:
    gp.AddError("Python or system error occurred")

Now when the script is executed, the progress dialog will look as follows:

Tip #4 -- Use traceback to return more information about the error

The help system topic Error handling with Python shows you how to use Python's traceback to retrieve the line number causing the error, the name of the .py file, and, for Python errors, a description of the error.

At version 9.3, you can run script tools in-process, as described in Running a script in process. Scripts run in-process execute much faster than scripts run out-of-process, so you always want to run the script in-process. When a script is run in-process, ArcGIS reads the .py file into memory and then executes it. Because ArcGIS puts the .py files into memory, the Python traceback module doesn't know the name of the .py file and will report the file name as "<string>". If you want to report the name of the .py file causing the error (very useful for debugging), you'll have to provide it yourself.

The script below has a local routine, trace(), that uses traceback to return the line number, the file name, and the error description. The routine also contains the the hard-coded file name (shown in bold) which you must change for your script.

import arcgisscripting
gp = arcgisscripting.create(9.3)
try:
    def trace():
        import os, sys, traceback
        tb = sys.exc_info()[2]
        tbinfo = traceback.format_tb(tb)[0]  # script name + line number
        line = tbinfo.split(", ")[1]
        
        # Construct the pathname to this script.  Get the pathname
        #  to the folder containing the script using sys.path[0]. 
        #  Modify line below and replace 'myscript.py' with the name of
        #  this script. 
        #
        filename = sys.path[0] + os.sep + "myscript.py"
        
        # Get Python syntax error
        #
        synerror = traceback.format_exc().splitlines()[-1]
        return line, filename, synerror
    if __name__ == '__main__':  
        # y = x
        result = gp.getcount("C:/blah.shp")
        
except arcgisscripting.ExecuteError:
    # Call our trace routine to retrieve line number and filename.
    #  (returned variable 'err' is ignored since this is a gp error.
    #
    line, filename, err = trace()
    gp.AddError("Geoprocessing error on " + line + " of " + filename + " :")
    for msg in range(0, gp.MessageCount):
        if gp.GetSeverity(msg) == 2:
            gp.AddReturnMessage(msg)
    
except:
    line, filename, err = trace()
    gp.AddError("Python error on " + line + " of " + filename)
    gp.AddError(err)

Below are illustrations of the progress dialog showing the line number and file name for both a geoprocessing error and a Python syntax error (by commenting out the "y = x" statement).

Tip #5 -- Use Finally statement to clean up cursors

Something I discovered by reading the Python doc Errors and Exceptions is the 'finally' statement. No matter what happens in your Python script, code in a 'finally:' block always gets executed. It's a great place to put clean-up actions, such as deleting cursors, as shown in the code below.

import arcgisscripting
gp = arcgisscripting.create(9.3)

def trace():
    import os, sys, traceback
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0] 
    line = tbinfo.split(", ")[1]
    filename = sys.path[0] + os.sep + "testscript.py"
    synerror = traceback.format_exc().splitlines()[-1]
    return line, filename, synerror

if __name__ == '__main__':
    # Define variables for a cursor and its row
    #
    cursor, row = None, None
    try:
        cursor = gp.searchcursor("E:/Data/CityOfSanFrancisco.gdb/FireStations")
        row = cursor.Next()
        while row:
          # Code here...
          # for demonstration...this next statement will cause an error
          y = x
          row = cursor.Next()
    except arcgisscripting.ExecuteError:
        line, filename, err = trace()
        gp.AddError("Geoprocessing error on " + line + " of " + filename + " :")
        for msg in range(0, gp.MessageCount):
            if gp.GetSeverity(msg) == 2:
                gp.AddReturnMessage(msg)
    except:
        line, filename, err = trace()
        gp.AddError("Python error on " + line + " of " + filename)
        gp.AddError(err)
    finally:
        # Close cursor and row objects.
        #
        del row, cursor
            
        # Print a message for demo purposes
        #
        gp.AddMessage("\n   ** Cursor and row have been deleted ** \n")

As illustrated below, the cursor gets deleted even if there is an error in the script.

In the main body of the script, the cursor and row object variables are declared and initialized to None (shown in bold in the above code). This ensures that these two variables exist when the del statement in the finally clause is executed; if you try to delete a non-existent variable, an exception will be thrown.

About cursors and the del statement

The Python del statement deletes a variable. You can use it to delete any Python variable, such as a list or dictionary. However, you rarely need to delete native Python variables. The del statement is mainly used for deleting non-native variables, such as geoprocessing cursors and rows obtained from the cursor. When you delete a cursor and its row, all pending changes to the row (caused by an update or insert cursor) are flushed and all locks on the dataset are removed. You should always delete cursors and rows obtained from the cursor. When deleting a cursor, delete the row object first, then the cursor.

Digging deeper - Troubleshooting geoprocessing errors when using ArcSDE data

dmhoneycutt — Fri, 05 Sep 2008 23:13:00 GMT

Geoprocessing error reporting has significantly improved in the ArcGIS 9.3 release. All tractable errors now have a unique error code and each error code has a detailed explanation in the help system. See Understanding geoprocessing tool errors and warnings for more details.

However, at this time, geoprocessing errors that occur when reading or writing ArcSDE/DBMS data receive a generic 'catch-all' error message, such as error 00210 when writing output:

ERROR 000210: Cannot create output E:\Infra\ToolData\Connection to ArcSDE.sde\ken.cities

We are working hard to improve reporting of ArcSDE/DBMS geoprocessing errors in future service packs and releases. In the meantime, here are some techniques you can use to determine the cause of your error. Note that the cause of the error could be anything from not being able to connect to ArcSDE to any number of DBMS or network related issues.

Test the connection

The first thing you should test is whether your ArcSDE/DBMS connection properties are correct. Open ArcCatalog and navigate to the location containing the .sde connection file ("E:\Infra\ToolData" in the above example) and double-click the connection, causing a connection to be made to the database. If a connection cannot be made, you'll receive an error message with details on the cause. If you can make a connection, then your connection properties may be invalid. Right-click the connection and select Connection Properties. Visually inspect the connection properties. Make sure that your script or model is using the correct user name when fully qualifying object names. For example, you may have connected to the database as the user 'ken' in the connection file, but the data paths in the script or model has a different user for the qualified feature class name, as illustrated below.

If everything seems correct with your connection file, the next investigation step is to set up a client-side SDEINTERCEPT.

Client-side SDEINTERCEPT

You may need to dig deeper by instructing ArcGIS to log information about the communication between your computer and the ArcSDE server. To instruct ArcGIS to log information, and where to write the logs, you need to set certain system environment variables. The exact technique for setting these variables depends on whether you are using Windows or UNIX. For a full explanation of the variables and how to set them, see the web help topic The dbinit.sde file

(For a more general introduction to configuring ArcSDE geodatabases, see An Overview of configuring an ArcSDE geodatabase .)

A SDEINTERCEPT file contains all the calls to ArcSDE along with their return values. These return values often contain the failure message received from ArcSDE or the error message ArcSDE received from the database. On Windows, the general steps are:

1. Set the following system environment variables:

<folder name> can be any existing folder on your computer. Just be sure to add the slash at the end of the pathname (i.e.: C:\temp\logs\)

On Windows XP, system environment variables are accessed by right-clicking My Computer and clicking Properties. On the Advanced Tab, click Environment Variables, as illustrated below.

2. Restart the application that will run the geoprocessing tool (such as ArcMap) so that it inherits the new system environment.

3. Run the geoprocessing tool or script that caused the failure.

After the task is run, a file named se_intercept.<nnn> is created in the folder specified in the SDEINTERCEPTLOC variable (where <nnn> is a unique number starting at 001). This file will contain error and warning messages. Not all of these errors and warnings are issues you need to worry about; some are a normal part of using ArcSDE and are just warnings that are automatically handled by ArcSDE. It will take some time to learn how to read these files. If necessary, you can get help interpreting the file from ESRI support. A good strategy is to search the file from the bottom up since it's more likely that the real error is near the bottom.

ArcSDE Server error logs

Each entry in an ArcSDE Server error log contains three pieces of information: the time of the message, the name of the client machine, and the message. If you can identify the time your error occurred, you can search the log for any obvious error messages around the time the error occurred on your client machine.

ArcSDE Server error logs are written to the location specified by your SDE administrator, so you will need to contact your SDE administrator to gain access to these logs. (They are written to SDEHOME\etc\<sde_instance_name>.log, such as sde_esri_sde.log).

To have more detailed information in the ArcSDE Server error logs, see the SDEVERBOSE setting in the topic The dbinit.sde file

DBMS logs and trace files

Most database management systems have the ability to log detailed information about database activities and queries. Investigating these logs usually require assistance from your database administrator as a certain amount of database specific expertise is needed to gain useful information from their contents.

DBMS logs contain information on the state of the database and any database related issues that may have occurred. Things such as the database running out of space to store your data, dead locks, database system errors and much more are contained in these files.

DBMS trace files, on the other hand, are files that contain all the SQL statements and their return values for a session in the database. When you request ArcSDE to perform an action, a lot of what is done is the running of SQL in the database. The database trace file will contain all the SQL used by ArcSDE and the messages the database returned when running this SQL. Any unexpected errors can be found in these files.

Tips and Tricks - Debugging Python script tools

dmhoneycutt — Mon, 01 Sep 2008 22:02:00 GMT

If you create your own script tools using Python, you'll appreciate these tips and tricks on debugging.

Coding errors are inevitable and there are two basic ways to find out where the error occurred:

Using print statements to discover bugs is an obvious and common method. Since script tools have access to the tool progress dialog, you can edit your script to include calls to AddMessage(), AddWarning(), or AddError() to print values and checkpoint messages to the progress dialog. Another variation is to use an independent method of returning messages, like the win32ui module's MessageBox method. This method displays a popup dialog. Since you have to click OK on the dialog to continue execution, this method allows you to pace the execution of the script. Here's an example using both methods:

import arcgisscripting, win32ui, win32con
gp = arcgisscripting.create()
n = 5

# Print message to progress dialog
#
gp.AddMessage("Value of n = " + str(n))

# Issue a popup dialog with OK and Cancel button
#
val = win32ui.MessageBox("Value of n = " + str(n), "title", win32con.MB_OKCANCEL)

# Based on the button clicked, you can branch execution
#
if val == 1:
    gp.AddMessage("You clicked OK")
else:
    gp.AddError("You clicked Cancel")
    raise arcgisscripting.ExecuteError, "Execution stops due to Cancel button click"

gp.AddMessage("This statement reached")

The other method is to use a Python IDE (Integrated Development Environment) that supports debugging, such as:

https://sourceforge.net/projects/pywin32

http://wingware.com

Debuggers allow you to set breakpoints, step in, out, and over individual lines of code, and examine the contents of variables, all without modifying your code. Compared to inserting print statements, debuggers are much more efficient and usually allow you to quickly isolate your bug. The illustration below shows how to use debug mode in PythonWin.

One way to use a debugger is to open your script directly in your IDE, modify it so that all parameters have values, and then proceed with debugging. This works reasonably well in simple cases. However, if your script uses layer or table view parameters, these variables need created on the fly. Complex parameters like a Field Map or Spatial Reference are hard to create as variables.

Ideally, you'd like to be able to open your script tool dialog, enter parameters, and then have the Python IDE launch with your code ready to be debugged. You can do this with a few simple changes, described below.

Use GetParameterAsText()

The first step is to modify your script so that it used GetParameterAsText() instead of sys.argv[], as discussed in Understanding script tool parameters

This is a modification you can keep - there is no need to change your code back to using sys.argv[].

Create a .bat file that launches your IDE with your .py as an argument

The next step is to create a one-line batch file that will be the source of your script tool (instead of your .py file). This single line contains the pathname to your IDE plus the full path to the .py file, as illustrated below.

Below are 3 examples of how to set your batch (.bat) file for commonly used IDEs. You'll need to confirm that the path is correct for your computer. Other debuggers are of course valid options, and can be called in a similar fashion.

Python IDLE
"C:\Python25\pythonw.exe" "C:\Python25\Lib\idlelib\idle.pyw" "C:\mytools\myscript.py"

PythonWin
"C:\Python25\Lib\site-packages\pythonwin\Pythonwin.exe" "C:\mytools\myscript.py"

Wing IDE 2.1
"C:\Program Files\Wing IDE 2.1\bin\wing.exe" "C:\mytools\myscript.py"

Reset script tool path to the .bat file

The next step is to change your script tool properties to use this new batch file. In the ArcToolbox window, right-click your script tool and click Properties. Click the Source tab and change the script file to be the new .bat file you created, as illustrated below. Click OK.

Execute and debug your tool

Now you can open the script tool from the ArcToolbox window, enter the parameters, and click OK. Shortly afterwards, the debugging application specified in your batch file will open with your Python script displayed. At this point, you are free to interact with the debugging application as you would normally. You can set a breakpoint and let the script run to the breakpoint and use any other options the debugger allows. Parameter values you entered in the dialog will get picked up by GetParameterAsText().

When running the script, any interaction you would normally expect between the script and the application will still occur. So, if you're using methods like AddMessage(), AddWarning(), or AddError(), these messages will show up in the application. If you are using progressor methods (new at 9.3), the progressor will be updated in the tool dialog as you walk through your script. Once the script is finished, you can return back to the application by closing your debugger.

Reset the script tool source

Once you have fixed your bugs, you can reset the source to the script file from the .bat file back to the .py file.

Looking ahead to future releases, we will have integrated options that will make this process smoother. But for now, and with a few little tricks, you can easily perform your script tool debugging.