Geoprocessing : ArcSDE

Send us your feedback

dmhoneycutt — Wed, 25 Mar 2009 23:47:00 GMT

You may not have noticed, but you can send us feedback about this site. There's a feedback link at the bottom right of the home page, as illustrated at the bottom of this post.

We're particularly interested in hearing from you about blog post topics you'd like to see and suggestions on how we could make this site more useful.

Tips and Tricks - Creating ArcSDE connection files on the fly using Python and ArcObjects

dmhoneycutt — Wed, 24 Sep 2008 22:57:00 GMT

ArcGIS 9.3 does not provide a geoprocessing tool for creating ArcSDE connection files (.sde files). Some of you have requested such a tool for security reasons - typically because you do not want .sde connection files hanging around on machines. Rather, you want to automatically create a connection file in a temporary location, use it in your script or model, and then delete it within the process you are running. Others need the ability to create .sde files on the fly because the connection properties, such as what version to connect to, are not known at the start of the process. Having to stop the process for someone to manually create the .sde file with the correct properties really isn't an option anyone wants to entertain.

The attached .zip file (download .zip file) contains a java class (CreateSDEConnFile) and a python script (CreateSDEConnFile.py) to create ArcSDE connection files on the fly. Download and unzip the file. Be sure to unzip the file into a folder that does not contain spaces. For example:

E:\SDE Conn sample (incorrect - contains spaces)
E:\SDEConnectionSample (correct - no spaces)

Java must be installed

You must have Java installed in order to use the files included in the .zip file. Most likely, Java is installed on your machine -- it gets installed by many websites. You can check by examining your installed programs list. You can install java by visiting java.com.

Setting JAVA_HOME

In order for CreateSDEConnFile.py file to execute, you must have the JAVA_HOME system environment variable set to the location of the java install directory.

On Windows XP, right-click "My Computer" on your desktop, click the Advanced tab, and click the "Environment Variables" button. If the JAVA_HOME environment variable doesn't exist, create it by clicking the "New" button. (On my machine, JAVA_HOME = C:\Program Files\Java\jre1.6.0_07)

About CreateSDEConnFile.py Python script

CreateSDEConnFile.py is a sample script. You will need to modify it to include details about your ArcSDE configuration and security needs. There is no associated geoprocessing script tool for CreateSDEConnFile.py -- you will have to create your own script tool for use in ModelBuilder (See An overview of creating script tools in the web help for more information about creating script tools.)

Arguments to the CreateSDEConnFile java class

String sConFileDir = args[0]; //Folder path to store the .sde file.
String sConFile = args[1]; //.sde file name. include the .sde extension.
pConnProps.setProperty("SERVER", args[2]); //ArcSDE Server name.
pConnProps.setProperty("INSTANCE", args[3]); //ArcSDE port number.
pConnProps.setProperty("USER", args[4]); //ArcSDE user name.
pConnProps.setProperty("PASSWORD", args[5]); //ArcSDE password.
pConnProps.setProperty("VERSION", args[6]); //Version to connect to.
pConnProps.setProperty("DATABASE", args[7]); //Database to connect to.

zip file

Download zip file

Digging deeper - Troubleshooting geoprocessing errors when using ArcSDE data

dmhoneycutt — Fri, 05 Sep 2008 23:13:00 GMT

Geoprocessing error reporting has significantly improved in the ArcGIS 9.3 release. All tractable errors now have a unique error code and each error code has a detailed explanation in the help system. See Understanding geoprocessing tool errors and warnings for more details.

However, at this time, geoprocessing errors that occur when reading or writing ArcSDE/DBMS data receive a generic 'catch-all' error message, such as error 00210 when writing output:

ERROR 000210: Cannot create output E:\Infra\ToolData\Connection to ArcSDE.sde\ken.cities

We are working hard to improve reporting of ArcSDE/DBMS geoprocessing errors in future service packs and releases. In the meantime, here are some techniques you can use to determine the cause of your error. Note that the cause of the error could be anything from not being able to connect to ArcSDE to any number of DBMS or network related issues.

Test the connection

The first thing you should test is whether your ArcSDE/DBMS connection properties are correct. Open ArcCatalog and navigate to the location containing the .sde connection file ("E:\Infra\ToolData" in the above example) and double-click the connection, causing a connection to be made to the database. If a connection cannot be made, you'll receive an error message with details on the cause. If you can make a connection, then your connection properties may be invalid. Right-click the connection and select Connection Properties. Visually inspect the connection properties. Make sure that your script or model is using the correct user name when fully qualifying object names. For example, you may have connected to the database as the user 'ken' in the connection file, but the data paths in the script or model has a different user for the qualified feature class name, as illustrated below.

If everything seems correct with your connection file, the next investigation step is to set up a client-side SDEINTERCEPT.

Client-side SDEINTERCEPT

You may need to dig deeper by instructing ArcGIS to log information about the communication between your computer and the ArcSDE server. To instruct ArcGIS to log information, and where to write the logs, you need to set certain system environment variables. The exact technique for setting these variables depends on whether you are using Windows or UNIX. For a full explanation of the variables and how to set them, see the web help topic The dbinit.sde file

(For a more general introduction to configuring ArcSDE geodatabases, see An Overview of configuring an ArcSDE geodatabase .)

A SDEINTERCEPT file contains all the calls to ArcSDE along with their return values. These return values often contain the failure message received from ArcSDE or the error message ArcSDE received from the database. On Windows, the general steps are:

1. Set the following system environment variables:

<folder name> can be any existing folder on your computer. Just be sure to add the slash at the end of the pathname (i.e.: C:\temp\logs\)

On Windows XP, system environment variables are accessed by right-clicking My Computer and clicking Properties. On the Advanced Tab, click Environment Variables, as illustrated below.

2. Restart the application that will run the geoprocessing tool (such as ArcMap) so that it inherits the new system environment.

3. Run the geoprocessing tool or script that caused the failure.

After the task is run, a file named se_intercept.<nnn> is created in the folder specified in the SDEINTERCEPTLOC variable (where <nnn> is a unique number starting at 001). This file will contain error and warning messages. Not all of these errors and warnings are issues you need to worry about; some are a normal part of using ArcSDE and are just warnings that are automatically handled by ArcSDE. It will take some time to learn how to read these files. If necessary, you can get help interpreting the file from ESRI support. A good strategy is to search the file from the bottom up since it's more likely that the real error is near the bottom.

ArcSDE Server error logs

Each entry in an ArcSDE Server error log contains three pieces of information: the time of the message, the name of the client machine, and the message. If you can identify the time your error occurred, you can search the log for any obvious error messages around the time the error occurred on your client machine.

ArcSDE Server error logs are written to the location specified by your SDE administrator, so you will need to contact your SDE administrator to gain access to these logs. (They are written to SDEHOME\etc\<sde_instance_name>.log, such as sde_esri_sde.log).

To have more detailed information in the ArcSDE Server error logs, see the SDEVERBOSE setting in the topic The dbinit.sde file

DBMS logs and trace files

Most database management systems have the ability to log detailed information about database activities and queries. Investigating these logs usually require assistance from your database administrator as a certain amount of database specific expertise is needed to gain useful information from their contents.

DBMS logs contain information on the state of the database and any database related issues that may have occurred. Things such as the database running out of space to store your data, dead locks, database system errors and much more are contained in these files.

DBMS trace files, on the other hand, are files that contain all the SQL statements and their return values for a session in the database. When you request ArcSDE to perform an action, a lot of what is done is the running of SQL in the database. The database trace file will contain all the SQL used by ArcSDE and the messages the database returned when running this SQL. Any unexpected errors can be found in these files.