Tips and tricks – Unix, Python, and Geoprocessing

This posting was written by Mark Zollinger, a Product Engineer and long time
UNIX user with the Geoprocessing team.

Do you like Solaris or Linux? Do you “do GIS” on one of those platforms?  Maybe you’ve been asked to convert some ArcGIS Workstation AML applications over to ArcGIS Engine. Are you a command-line junkie? (There are more of us than you might think)

If, for one of the above or any other reason, you find yourself wondering if you should try developing GIS on Solaris or Linux, then it’s time to stop wondering. Join in with others who include a flavor of *nix in their list of working development platforms.

There are several approaches to GIS development that work very well on Solaris and Linux.

If you are into low-level programming, you should consider Java or C++, both of which can access ArcGIS’s ArcObjects API. I’ve used Java quite a bit over the years, and can tell you that the doors are wide open on what you can do with it. If this sound like the thing for you, then you can move along now, there’s nothing to see here. This post isn’t about ArcObjects at all.

If you lean towards scripting and higher level abstractions, then geoprocessing’s more coarse-grained object model, then you are in the right place. The rest of this post will show you how to set up an IDE for Python development and write a very simple geoprocessing script.

I mentioned a moment ago that I’ve written my share of Java ArcObjects code. I’ve got to confess though, that I’ve always been a fan of scripting languages. Python and GP is where I spend most of my time these days.

Setting up

“Okay”, you ask, “How do I start writing GP code in Python on non-Windows platforms?”

You can do Python and GP from the command line, and I often prefer the focus this brings. I’ve found, however, that most folks prefer to work in a graphic
DE (Integrated Development Environment).

Setting up an IDE that you can use

Probably your best choice is the Eclipse IDE with the Pydev plugin. Eclipse is an Open Source IDE, intended to work with a wide variety of programming languages. Pydev is the plugin that specifically enables Python development in Eclipse.

The latest version of Eclipse, named Ganymede, has not been fully tested by ESRI for use with ArcGIS 9.3, and is not available for some older UNIX releases. We recommend using version 3.3.2, which you can find in the archive at:

http://archive.eclipse.org/eclipse/downloads/

To install Eclipse, follow the instructions in the “Installing Eclipse” paragraph at:

http://wiki.eclipse.org/FAQ_Where_do_I_get_and_install_Eclipse%3F

(These instructions are for Ganymede, but they basically the same for version 3.3.2)

Once you have Eclipse installed and running, you need to add the Pydev plugin. Just follow the “Getting Started” link on the left side of the Pydev homepage.

http://pydev.sourceforge.net/

Note that you need the Open Source Pydev plugin. The Pydev Extensions plugin is commercial trialware, and is optional.

When you get to the “Configuring the interpreter” page, you are told to choose the Python interpreter that Pydev should use to run Python programs. Use the Python installation you use for Engine development/testing. In most cases, this is the one that got installed with the Engine Runtime:

$ARCGISHOME/python25/bin/python

Note: you might get an error message while trying to choose the interpreter. If you do, click the error dialog’s Details button. The message probably includes something like this:

ld.so.1: python: fatal: libpython2.5.so.1.0: open failed: No such file or directory

This is an indication that the python interpreter is not correctly set up. The most likely cause is that your ArcGIS Engine environment has not yet been established. Exit Eclipse and source the init_engine shell script that applies to the shell you are using. When you start Eclipse the problem should go away.

Also, item 3 on the “Configuring the interpreter” page asks you to set the SYSTEM PYTHONPATH. When you do this, make sure to include $ARCGISHOME/bin. This matches the PYTHONPATH that gets set when you source init_engine.

Writing a Python script that does GP stuff

Now that you have a development environment, how do you use all this great GP stuff? Let’s create a very simple Python script to get you started on the road to GP development. In most Python scripts, the first thing to do is to import the modules that you will need. For GP, we need to import the arcgisscripting module. Let’s also import the os and system modules:

import arcgisscripting
import os, sys

To access GP, we need to create a Geoprocessor object. This is the object through which you will call any GP functions or objects


gp = arcgisscripting.create(9.3)

By passing in the 9.3 argument, you are getting the newer version of the Geoprocessor object, which is a little more Python friendly. For this example we won’t be using any of the Python friendly features, but it is never too early to start a good habit.

Now that you have a GP object, what do you do with it? Let’s set GP’s workspace property and print it out, just to prove to ourselves that it really worked. For this example, let’s set the workspace to a “data” directory just underneath the current Python directory.

gp.workspace = os.path.join (sys.path[0], “data”)
print “GP Workspace = ” + gp.workspace

(Of course you can set the workspace to any accessible directory, not just the one returned by sys.path[0]. I’m just using sys.path[0] because I know it’s accessible.)

sys.path is a list of strings where Python looks for modules. The first item in the list is the current Python directory. Calling os.path.join is a great way to build up filesystem paths, since it correctly handles all the path separator issues. (More on path separators later.) When you run this script (Run -> Run As -> Python Run), you should see “GP Workspace=/some/path/data” printed in the Console tab near the bottom of the Eclipse window.

You can use Eclipse or the operating system to create the data directory, since it probably doesn’t exist yet.

For this example, let’s suppose you have point data representing hawk sitings in a national forest. Your study area is a subset of the forest where a particular plant in known to grow. What you need to do is clip your point data using the polygon of your study area.

Once you have copied your data into the aforementioned data directory, your script could execute the clip tool like this:

gp.clip_analysis("nat_for.gdb/hawk_sightings", "study_area.shp", "nat_for.gdb/study_sightings")

Just to be sure it worked, you can print out the messages that the tool generated like this:


print (gp.getmessages())

Notice that in calling the clip tool, we used forward slashes (/). Back slashes () and forward slashes work equally well in ArcGIS on ALL platforms, so you don’t need to worry about switching back and forth when you change to Windows. You can even use back slashes on Linux and Solaris. The problem is that in Python, back slashes are a special character, so to use them, you have to treat them differently than most other characters.

      - You can escape them: “\some\path”
      - You can use raw strings: r”somepath”
    - You can use unicode strings: u”somepath”

If you stick to forward slashes on all platforms, you improve the look of your code
and reduce the effort it takes to type it.

So now you have a script that looks like this:


import arcgisscripting
import os, sys

gp = arcgisscripting.create(9.3)

gp.workspace = os.path.join (sys.path[0], “data”)
print “GP Workspace = ” + gp.workspace

gp.clip_analysis(“nat_for.gdb/hawk_sightings”, “study_area.shp”, “nat_for.gdb/study_sightings”)
print (gp.getmessages())

Run it, and it works, creating a new featureclass in the file geodatabase. Now wasn’t that easy? To get more complicated, you just string together as many of those tools as you need to get the job done.

Welcome to the world of Geoprocessing with Python!

Links

Automating workflows with scripts – Introduction to writing Python scripts for geoprocessing
Techniques for sharing Python scripts – Useful techniques for sharing Python scripts
The Python Tutorial – If you’re new to Python, this is a good tutorial to start
How to duplicate AML functions in Python – For AML programmers
How to duplicate AML directives in Python- For AML programmers

Troubleshooting

Color issues

Upon using Eclipse on Solaris for the first time, you may see some color problems.
For example, the text insertion caret might be white on a white background, making it invisible.
Another common symptom is that you cannot see the outline of text input fields.
If you run into this or something similar, check the color depth of your display.
If it is set to 8 bits, change it to 24.

init_engine

You should have your Engine environment already established in the shell in
which you start Eclipse. (source init_engine.sh
or source init_engine.csh)
If you see inexplicable error messages, this is the most likely cause.

This entry was posted in Analysis & Geoprocessing and tagged , , , , . Bookmark the permalink.

Leave a Reply

7 Comments

  1. reuter says:

    hmm, how to recreate the scratchunique function from aml in python.. see output below… the counter is not increasing (9.2 SPlatest)
    >>> tmp12345 = gp.createscratchname(“tmp”,”",”raster”,str(gp.workspace))
    >>> print tmp12345
    S:workhelgatmp0
    >>> tmp12346 = gp.createscratchname(“tmp”,”",”raster”,str(gp.workspace))
    >>> print tmp12346
    S:workhelgatmp0
    >>> tmp12346 = gp.createscratchname(“tmp”,”",”raster”,”")
    >>> print tmp12345
    S:workhelgatmp0
    >>> tmp12346 = gp.createscratchname(“tmp”,”",”raster”,”")
    >>> print tmp12346
    S:workhelgatmp0

  2. TomGeo says:

    Hi Mark,

    I am using Python and Eclipse under Windows. ArcGIS Desktop is a local installation on this machine. When I found this topic here I was curious how to get ArcGIS to use while on a *nix machine…

    Since the ArcGIS Desktop is installed on Windows, how do I get access from my Linux machine to the geoprocessor on the Windows machine? Is this possible?

    Thomas

  3. BC99 says:

    I’ve been trying to this to work on my Windows box with PyDev the plugin for Aptana. However, I’m not seeing all the gp methods after setting the PYTHNONPATH and External Src.
    Ex:
    import os, arcgisscripting
    Geoprocessor object
    gp = arcgisscripting.create(9.3)

    My scripts run but I’m not able to get intellisense on the gp. Any ideas?

  4. markz says:

    regarding gp.createscratchname not incrementing.

    Each time you call that function, it looks in the directory and starts trying filenames until it finds one that does NOT exist yet. It returns the FIRST filename that it finds that does not yet exist.

    To get it to increment, you call the function, THEN create a file with the name you get back.
    Once the file has been created, then the next time you call the function, it will increment.

    If you never create any files, the filenames will never increment.

  5. markz says:

    Thomas,

    You asked:
    “how do I get access from my Linux machine to the geoprocessor on the Windows machine?”

    I don’t think you can do that, apart from running ArcGIS Server on your Windows machine.

  6. markz says:

    BC99,

    You asked about intellisense for gp tools and functions in Eclipse with Pydev.

    The arcgisscripting module is late binding, which has some neat advantages, but precludes a useful intellisense experience.

    Look for the arcpy site package in the upcoming ArcGIS 10 release. It is early bound and in addition to supporting intellisense, has some pretty neat features.

  7. roy_hewitt says:

    markz,

    I’ve tried just about everything to get ArcGIS 10 arcpy intellisense working with Eclipse (+ pydev). I’ve added the arcpy library to the interpreter, but still no luck.

    Suggestions?