Geoprocessing : New at 9.3

Using arcgisscripting's "9.3" switch

bbicking1 — Tue, 28 Jul 2009 21:46:00 GMT

This blog post was written by Ghislain Prince, Product Engineer on the Geoprocessing Team in Redlands.

Hi Everybody!

We learned in the Advanced Python Scripting Technical Workshops at UC that many of you who use ArcGIS 9.3 do not yet use the changes we made to the geoprocessor. We just wanted you guys to be aware of this option when writing your Python scripts at 9.3. The 9.3 switch option is specified when the geoprocessor is created; in code it looks like this

import arcgisscripting

gp = arcgisscripting.create(9.3)

Existing (pre-9.3) scripts will obviously not have this option as it didn’t exist before the 9.3 release. We’re sure you’ve already noticed that they continue to work unchanged. To get the 9.3 geoprocessor behavior, it is not enough to simply change gp = arcgisscripting.create() to gp = arcgisscripting.create(9.3) - that does not work, because your code needs to be updated elsewhere in your script to accommodate the changed behavior.

The main goal for this switch is that when you create new scripts in ArcGIS 9.3, you will use the “9.3” geoprocessor and get a much nicer coding experience. This “nicer” experience is achieved because the geoprocessor uses more of Python’s native structures as inputs and outputs.

It’s obvious that the improvements to the geoprocessor make it easier for you to write code and resulting code is easier to read and maintain. Of course the changes are not limited to the List functions; here’s the complete list of differences and a comparison with several code examples.

Spatial Statistics: Free Virtual Campus Web Seminar on Regression Analysis Basics

bbicking1 — Mon, 04 May 2009 15:33:00 GMT

This blog post was written by Lauren Scott, Geoprocessing/Spatial Statistics Product Engineer in the Software Products Group at ESRI in Redlands.

There are a number of good resources where you can learn about the new regression analysis tools in ArcGIS 9.3. The newest resource is a free one-hour Web seminar, Regression Analysis Basics in ArcGIS 9.3, available for download from the ESRI Virtual Campus.

Other Regression Analysis resources include:

Regression Analysis Basics Documentation
Ordinary Least Squares Regression Tool Help and Learn more about how OLS works
Geographically Weighted Regression Tool Help and Learn more about how GWR works
Interpreting OLS tool results
Interpreting GWR tool results
Several articles can be found in the Spring 2009 issue of ArcUser magazine [Try the Flip Page view of ArcUser!]:

Answering Why Questions: An introduction to using regression analysis with spatial data;
Regression Analysis Components; and
Applying Geographically Weighted Regression to a Real Estate Problem.

GEOconnexion International Magazine, December/January 2009, includes an article titled Regression Analysis Tools for GIS Modeling
Chapter 5, Analyzing Geographic Relationships, in The ESRI Guide to GIS Analysis, Volume 2 discusses regression analysis and some of the challenges when it is applied to spatial data.

Enjoy!

Send us your feedback

dmhoneycutt — Wed, 25 Mar 2009 23:47:00 GMT

You may not have noticed, but you can send us feedback about this site. There's a feedback link at the bottom right of the home page, as illustrated at the bottom of this post.

We're particularly interested in hearing from you about blog post topics you'd like to see and suggestions on how we could make this site more useful.

New at 9.3 - Spatial Analyst improvements

dmhoneycutt — Thu, 02 Oct 2008 23:08:00 GMT

The Spatial Analyst at 9.3 has several new capabilities, improvements, and fixes. This post highlights some of the important changes; for a full description, see the What's New pdf document (Once open, in the Extensions section click on Spatial Analyst. This will take you to the right location in the document.)

New Functionality

Zonal Histogram - Helps you to investigate the distribution of cells from one dataset within classes of another. Example applications include rainfall within watersheds or crime distribution by police beat.

Contour with Barriers - This tool allows you to get better results when trying to contour discontinuous surfaces. In addition to its superior handling of barriers, Contour with Barriers creates smoother contour lines, generate much larger output datasets (no 2.14 GB shapefile size limit), and optionally to attribute primary and index contour lines.

The Snap Raster environment has moved from being part of the Extent environment to being its own separate environment setting. This gives you flexibility to set the snap raster dynamically from scripting or ModelBuilder.

Usability Improvements

The Con tool now allows the input conditional raster to be a floating point raster.

The Create Constant Raster, Create Normal Raster, Create Random Raster, Extract by Rectangle, and Topo to Raster tools now allow you to specify the extent of the output raster to be that of an existing dataset. The main benefit of this is the extent can be set dynamically from scripting or ModelBuilder. While you can still enter the extent values manually (but who wants to type in 370803.47263702 3978856.7539477 443868.87614745 4032133.0258861 ?), being able to specify a dataset is both easier and helps ensure the proper precision is maintained .

All Spatial Analyst tools have new Geoprocessing tool errors and warnings and show execution percent complete using the new progress bar.

Performance and Data Improvements

Combine and Cut/Fill are significantly faster (up to 40 times).

Viewshed now handles much larger inputs (no more MSEEK fatal errors!).

IDW now works with many more points (up to 50 million).

Combine now works on millions of combinations.

Watch ESRI TV

dmhoneycutt — Tue, 09 Sep 2008 17:17:00 GMT

Check out http://www.youtube.com/esritv. It contains over 50 videos on a variety of ArcGIS subjects. Content is updated periodically. You can also access the videos on Viddler: http://www.viddler.com/explore/esri

New at 9.3 - Error reporting

dmhoneycutt — Mon, 08 Sep 2008 22:41:00 GMT

Check out this Geodatabase blog entry on error reporting. It describes how we use error reports you submit to improve ArcGIS. The blog entry has a link to a short video from the 2008 User Conference that is worth watching.

New at 9.3 - Top Ten

dmhoneycutt — Tue, 29 Jul 2008 16:44:00 GMT

We asked the team to give us their Top 10 New Geoprocessing Features in ArcGIS 9.3. The list quickly turned into the top 20, then the top 30, and then we had to stop. It's by no means exhaustive, but we think it includes the major improvements we made that will make your geoprocessing work a whole lot faster and easier.

In no particular order, here's the first top 10 list.

Join Field tool

At last year's UC a common question was how to permanently add fields from one table to another. The solution at the time involved a combination of the Add Join, Add Field, and Calculate Field tools. With version 9.3, the new Join Field tool does all of this in one step, using a fraction of the time. As with Add Join, Join Field accepts two tables, each with a common attribute field. The tool joins the two tables based on this field and you have the option of selecting which additional fields you want to have included in the final output. The Join Field output is a permanent update of the input table or feature class.

See the Join Field tool reference page for more information.

Error messages

We standardized warning and error messages and gave each one a unique number. So now, when a warning or error comes up on a tool dialog, progress dialog, or command line window, you can click the number to open a detailed description. This description lists probable causes of the error or warning, suggestions to remedy the situation, and links to relevant help topics.

See Understanding geoprocessing tool errors and warnings for more information

Near with lines and polygons

Have you ever been required to find the nearest distances between a mixture of point, line, and polygon features? This would seem to be a very easy operation. However, until ArcGIS 9.3, it required building a script or program. With ArcGIS 9.3, the Near tool is improved to support all feature types. In addition, multiple near feature classes can be supplied.

Generate Near Table tool

The Near tool modifies the attributes of the input feature class. This often caused problems when the input dataset was read only. With ArcGIS 9.3, there is a new Generate Near Table tool. Instead of modifying the input feature class as Near does, Generate Near Table creates a new table containing the proximity information. This tool also contains an option to find the distances to all features, not just the nearest.

Performance improvement using ArcSDE data with GpTools

We've made three major improvements when using ArcSDE data with geoprocessing tools:

By far the most dramatic improvement is the result of better management of ArcSDE connections. Instead of creating multiple connections to ArcSDE for one execution of a tool, we now create only one connection. (This issue is also resolved in version 9.2 service pack 5).

The performance of the Append tool has improved because the spatial index of the output dataset is created only once instead of many times.

We made many improvements to our internal topology analysis engine used by overlay tools. The performance of tools in the Analysis toolbox has greatly improved when using large ArcSDE datasets.

Python scripts run in process

In 9.2 all Python [.py] script tools created a separate process and executed outside the application process. The way you noticed this is that it took as much as 5 to 10 seconds for a script tool to start up. In 9.3, we're taking advantage of the application already running. This greatly improves script tool performance in two major areas: Initial start-up of a script tool and the use of cursors within the script tool. It also greatly improves the start-up performance of scripts tools using smaller datasets as inputs.

See Running a script in process for more information.

Geodesic buffers

In 9.2, the Buffer tool created buffer polygons assuming that all coordinates represented a flat Euclidean space instead of a spherical space, like the Earth.

In 9.3, the Buffer tool will create geodesic buffers if:

The features to be buffered are points or multipoints.

The input feature are in geographic coordinates (i.e., longitude/latitude).

The buffer distance is in Euclidean linear units (i.e, feet, meters).

The output buffer polygons will take into account that longitudinal distance varies varies with latitude, as illustrated below.

See How Buffer works for more information on geodesic buffers.

Script tool validation

You can now have your script tools behave like system tools (those that come with ArcGIS). You can enable (gray-out) parameters based on values in other parameters, provide keyword lists that dynamically change, put parameters into different categories, calculate default values on-the-fly, and update the description of output data for use in ModelBuilder. These capabilities are all found on the new Validation tab you see on a script tool's property page.

See Customizing script tool behavior for more information

Spatial Statistics: Ordinary Least Square Regression and

Geographically Weighted Regression Tools

The Spatial Statistics toolbox has powerful pattern analysis tools to help answer "Where?" questions like:

Where are people persistently dying young in the United States?

Where are our kids consistently turning in high test scores?

Where do we seen an unexpectedly high number of traffic accidents?

With ArcGIS 9.3, new regression analysis tools are added that allow you to answer the next logical set of questions relating to "Why?" like:

Why are people persistently dying young in particular places across the United States?

What are the factors contributing to consistently high test scores?

Why are there so many traffic accidents in particular hot spot locations?

Ordinary Least Squares Regression (OLS) is a commonly used global linear regression method used to generate predictions, or to model a dependent variable or process, in terms of its relationships to a set of explanatory variables. It creates a single equation to represent those relationships. OLS is the starting point for all regression analysis (including all spatial regression analysis). It often breaks down when used with spatial data, however, in the case where the relationships being modeled change across the study area. An education variable, for example, might be a good predictor of crime rates in one part of the study area, but may not be significant in another part of the study area. Geographically Weighted Regression (GWR), on the other hand, allows data relationships to vary. GWR is one of several spatial regression techniques increasingly used in geography and other disciplines. It provides a local model of the variable or process you are trying to understand/predict by fitting a regression equation to every feature in the dataset. GWR constructs these separate equations by incorporating the dependent and explanatory variables of features failing with a particular user-specified bandwidth of each target feature. ArcGIS 9.3 includes "Regression Analysis Basics" and "Result Interpretation" documentation with strategies and graphics aimed at helping the user apply these methods effectively.

See An overview of the Modeling Spatial Relationships toolset for more information.

Batch Grid: deferred validation

If you use a tool in batch mode, you may have noticed that filling in the batch grid became slow if more than 10 rows or so were entered. This was due to the fact that every value in every row was validated whenever you clicked the mouse in a cell.

In 9.3, you control when validation occurs by click the Check Values button in the lower right of the batch grid, as illustrated below.

Validation always occurs when you click the OK button.

See Using the batch grid control for more information. You can find the discussion of the Check Values button at the bottom of the topic.