Geoprocessing

Lidar Solutions in ArcGIS: New Web-based Lidar Courses

bbicking1 — Thu, 19 Nov 2009 17:35:00 GMT

This post was written by Clayton Crawford, Product Engineer on the 3D Analyst team in ESRI's Software Products group in Redlands

Due to the demand for lidar training, ESRI’s Training Center now offers three web-based courses on lidar. First is a free training seminar that provides an overview of lidar capabilities in ArcGIS and introduces high level concepts. The other two include hands-on exercises and are geared toward data managers and analysts. For more information see each course’s description...

Getting Started with Lidar in ArcGIS

Managing Lidar Data in ArcGIS

Using Lidar Data in ArcGIS

Lidar Solutions In ArcGIS_part8: Business Partner Solutions for Lidar in ArcGIS

bbicking1 — Thu, 15 Oct 2009 15:41:00 GMT

This is the eighth blog in a series on Lidar Solutions in ArcGIS. It was written by Clayton Crawford, Product Engineer on the 3D Analyst team in ESRI’s Software Products group in Redlands.

Business Partner Solutions for Lidar in ArcGIS

ArcGIS provides lidar capabilities that are designed to answer the needs of most GIS end users. This is accomplished by providing generic, scalable tools that facilitate the use of lidar for high quality surface construction and analysis. That said, application and data specific tools can improve capability and efficiency for certain workflows. To this end, ESRI often relies on business partners for solutions.

We’ve partnered with several software solutions providers that offer products designed specifically for lidar. Some of these solutions are fully integrated in ArcGIS. Others are standalone applications that are complimentary to ArcGIS. In both cases these partners and their software are referenced below. Other partners that provide only lidar services, are not included because, depending on how you define service, their numbers can grow too large.

Merrick

Merrick sells a standalone application designed to handle lidar called Merrick Advanced Remote Sensing (MARS). This software “includes a modular tool suite that is used to manage field collection, data analysis, quality assurance, production, and client deliverable workflows”. Evaluation copies and a free viewer are available. Find out more about MARS here.

Overwatch Systems, Visual Learning Systems

Overwatch VLS makes Lidar Analyst which “simplifies the process of extracting 3D features directly from lidar data and provides tools for accelerated extraction of bare earth (terrain), buildings, and trees”. A free trial version, including tutorial, is available for download. Find out more about Lidar Analyst here.

QCoherent

QCoherent makes LP360 for ArcGIS as well as several other standalone solutions. The feature rich LP360 “integrates LiDAR point cloud datasets into ArcGIS without requiring an import or conversion process”. Optional modules include Classify and Extractor. Free evaluation copies are available. Find out more about solutions from QCoherent here.

Ultra Electronics, Prologic

Prologic makes Lidar Explorer for ArcGIS. Included in their solution is a “framework for managing, visualizing, analyzing, and processing an entire project of LAS files (any number of files, any number of points) as a single layer within ArcMap”. They offer a free evaluation of the software as well as a free viewer. Find out more about Lidar Explorer here.

Conclusion

Business Partners provide a valuable service to many ArcGIS users. With their domain expertise they craft solutions that answer the needs of specific applications and workflows. Lidar is no exception and ESRI is thankful to have several partners that offer some very useful tools.

Read more about the ESRI Business Partner Program.

If you’re an ESRI Business Partner with a lidar solution, and feel you ought to be included in this post, please let us know.

Lidar Solutions In ArcGIS_part7: Minimizing noise from lidar for contouring and slope analysis

bbicking1 — Wed, 02 Sep 2009 16:12:00 GMT

This is the seventh blog in a series on Lidar Solutions in ArcGIS. It was written by Clayton Crawford, Product Engineer on the 3D Analyst team in ESRI’s Software Products group in Redlands.

Lidar is promoted as an accurate form of elevation data. To a large degree, this comes from its dense sample interval. Some refer to lidar as ‘painting the ground with measurements’. Indeed, it’s commonplace to get sub-meter sampling. This facilitates canopy penetration which improves the accuracy of ground models in forested areas. The high sample density also improves results for certain applications such as floodplain delineation. That said, lidar is not a silver bullet for all surface modeling activities. Two areas that tend to be problematic are contour derivation and slope analysis.

Messy contours and steep slopes

Getting a computer to produce nice looking contours from traditional data sources is hard enough. It's even harder to produce them from lidar. Contours produced from full resolution lidar tend to be jagged with many twists and turns and isolated closed rings (Figure 1).

Slope assessment with lidar is also problematic. If one examines the average slope from full resolution lidar they will find it’s unusually high. This is true even over relatively flat ground as shown in Figure 2.

Some people misinterpret these issues with contours and slope as simply a result of lidar being more accurate than other forms of surface data. At large enough scales almost any surface is rough, right? While there’s some truth to this a significant part of the problem really stems from the relationship between horizontal sample density and vertical accuracy. As with any measurement technology lidar is not perfectly accurate. Its vertical accuracy generally ranges from 12 to 15cm. That provides a random height difference between two adjacent points of 24 to 30cm. When the points are only one meter or less apart horizontally that height differential becomes significant. From a signal processing perspective this is called high frequency noise.

Techniques to reduce noise

In order to get less jagged contours and more reasonable slope estimates you have to remove the noise from the lidar and lose as little real information as possible while doing so. While there’s no way to prevent any loss, harm can be minimized. The terrain dataset offers a couple of tools to facilitate this. One is intelligent point thinning; the other is a high quality interpolator.

Point thinning occurs during the terrain pyramid building process. Some people think pyramids are strictly a visualization tool, used solely to speed up drawing. While this is true for raster pyramids, the same does not apply to terrains. Terrains were designed knowing that lidar is better suited for some applications when it’s been generalized.

There are two terrain point thinning algorithms: z-tolerance and windowsize. Both have something to offer and are arguably better than random n^thpoint filters used by other solutions that, while fast and reasonable for visualization, are not what you’d want to use for analysis.

The z-tolerance filter employs a TIN based algorithm to find a subset of points sufficient to create a surface that’s within a given vertical distance to the full resolution surface. TerraScan, an industry leading lidar processing package, uses a similar technique for the selection of what it refers to as ‘model key’ points. The use of this filter is recommended when a quantifiable measure of vertical accuracy is needed.

The windowsize filter selects points within a given horizontal sample distance. Every so many units in XY (i.e., the sample window size) the points within that area are examined and one or two are selected depending on which option you chose: the one closest to the mean of the other points in the sample window, the highest or lowest of those points, or both the highest and lowest. As stated earlier, much of the noise problem results from a bad ratio of high horizontal sample density to vertical accuracy. The windowsize filter enables you to improve that ratio. While it’s hard to quantify the accuracy of the thinned data, empirical evidence has shown this approach works well.

Thinning the data

Lidar points are thinned when a terrain dataset is built. The filter algorithm applied is based on the type of pyramid selected for the terrain. Fortunately, the same name is used for pyramid type as filter algorithm so there’s no ambiguity. The pyramid type is selected via the terrain wizard found on the feature dataset context menu in ArcCatalog (Figure 3).

The next step is to specify the pyramid levels. For the sake of noise reduction we’re interested in that first pyramid level which is one step removed from full resolution (Figure 4). A z-tolerance equal to the vertical accuracy of the data is reasonable for that level. This will eliminate as much noise as possible while decreasing the accuracy of the result as little as possible. If you start out with 15cm vertical accuracy, you end up with approximately 30cm accuracy and if you go on to produce contours, set their interval to at least double this, namely 60cm or approximately 2 feet. When building a windowsize pyramid a sample distance equal to twice the nominal point spacing is reasonable.

After having built a terrain, the thinned points reside in its pyramid. You can extract a TIN made from these thinned points via the Terrain to TIN tool and from this generate contours and slope. The results will be less noisy than if made from the un-thinned points. One drawback to this approach is that TINs have a size restriction. Ten million points per TIN is about as large as you can go. With lidar projects this can mean that you have to extract multiple TINs from a single terrain, even when using thinned data, if you’re working on a large area. Another drawback is that contours and slope estimates made from these TINs will still be more angular and discontinuous than necessary. A recommended alternative is to go through a raster.

Note, in ArcGIS 9.4 you will be able to derive contour and slope output directly from terrains without needing to go through TINs.

Raster Interpolation

Contour and slope output can be improved by deriving them from a raster that’s made from a terrain. Engineers have a traditional bias towards working directly on TINs, so they may not like the idea of going through a raster, but when dealing specifically with contour and slope production from lidar this bias is unwarranted. For one thing, lidar points are essentially collected in a random distribution and the resulting triangles are not handpicked or guaranteed to fit some mathematical model made inside a CAD package (e.g., for road design). Additionally, the linear piecewise surface defined by planar triangle faces is not smooth. A smoother surface can be made using the Terrain To Raster tool with the natural neighbors interpolation option. An added benefit of using Terrain To Raster is that you can rasterize an entire terrain in one shot and avoid the size constraint associated with TIN extraction (Figure 5).

You then use the Contour and Slope tools on the derived raster. Figures 6 and 7 illustrate the difference between full resolution and generalized contour and slope derivatives.

Conclusion

Lidar is noisy because of a low vertical accuracy relative to horizontal sample distance. This noise translates into poor quality contours and excessively high average slope rates. Noise can be reduced via point thinning and smoothing without also losing much of the accuracy and detail. The terrain dataset provides these tools via pyramiding and natural neighbor interpolation.

That’s it for this installment of the Lidar Solutions in ArcGIS series. Subscribe to this blog or check back in a month or so for a look at business partner offerings.

Using arcgisscripting's "9.3" switch

bbicking1 — Tue, 28 Jul 2009 21:46:00 GMT

This blog post was written by Ghislain Prince, Product Engineer on the Geoprocessing Team in Redlands.

Hi Everybody!

We learned in the Advanced Python Scripting Technical Workshops at UC that many of you who use ArcGIS 9.3 do not yet use the changes we made to the geoprocessor. We just wanted you guys to be aware of this option when writing your Python scripts at 9.3. The 9.3 switch option is specified when the geoprocessor is created; in code it looks like this

import arcgisscripting

gp = arcgisscripting.create(9.3)

Existing (pre-9.3) scripts will obviously not have this option as it didn’t exist before the 9.3 release. We’re sure you’ve already noticed that they continue to work unchanged. To get the 9.3 geoprocessor behavior, it is not enough to simply change gp = arcgisscripting.create() to gp = arcgisscripting.create(9.3) - that does not work, because your code needs to be updated elsewhere in your script to accommodate the changed behavior.

The main goal for this switch is that when you create new scripts in ArcGIS 9.3, you will use the “9.3” geoprocessor and get a much nicer coding experience. This “nicer” experience is achieved because the geoprocessor uses more of Python’s native structures as inputs and outputs.

It’s obvious that the improvements to the geoprocessor make it easier for you to write code and resulting code is easier to read and maintain. Of course the changes are not limited to the List functions; here’s the complete list of differences and a comparison with several code examples.

2009 User Conference

dmhoneycutt — Fri, 17 Jul 2009 18:59:00 GMT

The 2009 User Conference has just ended. We had a great time and it was wonderful to meet with everyone and talk about geoprocessing and analysis. We had some great sessions and really interesting questions at the geoprocessing and analysis island.

We'll be bundling up our tech session slides, models, data, and demo scripts and submitting them to the Model and Script tool gallery. You should start seeing submissions in a few days -- they'll be tagged with '2009 User Conference'. In addition, some session videos will become available in a few weeks as our vendor processes them. Check the Media Gallery for these session videos.

ArcInfo Workstation to ArcGIS Desktop Command Glossaries

ashl2524 — Sat, 11 Jul 2009 00:37:00 GMT

Have you ever wondered what is the equivalent ArcGIS desktop command for an ArcInfo workstation command? Have you ever wondered what the equivalent python command is for an AML function or Directive?

The Geoprocessing team has created the following glossaries to allow workstation users find corresponding ArcGIS Desktop commands. In many cases the mapping of functionality and commands between Workstation and ArCGIS Desktop is straight forward and obvious. Sometimes, however, finding the corresponding Geoprocessing tool in Desktop is not straight forward and obvious, and these glossaries are a helpful resource for those of you that are using workstation in conjunction with Desktop.

Workstation to Desktop Glossaries:

In addition to these glossaries, in 9.4, you will be able to use the new search functionality with Arcmap to search for Workstation commands and the corresponding Geoprocessing tool will be returned if there is a match. These glossaries will be used as a source for populating the keywords to ensure the search mechanism is as accurate as possible.

Keep in mind; we recognize that these glossaries do not show every alternative for workstation functionality within Desktop. As a result, we will continue to update these documents and if you have suggestions or alternatives, please let us know.

Lidar Solutions in ArcGIS_part6: Updating a portion of a terrain dataset with new measurements

bbicking1 — Thu, 02 Jul 2009 17:36:00 GMT

This is the sixth blog in a series on Lidar Solutions in ArcGIS. The author of this blog is Clayton Crawford, lead Product Engineer on ESRI's 3D Analyst team in the Software Products group in Redlands.

Updating a portion of a terrain dataset with new measurements

The ability to update a surface is important to people responsible for providing accurate, up to date surfaces and people performing analysis on those surfaces. Updates come in different forms:

Adding ancillary data (e.g., breaklines)
Removing or replacing bad data
Using newer or more accurate data
Increasing extent with additional data
Inserting design/modeled data to perform ‘what-if’ analysis

These kinds of updates are best performed on the measurements used to construct a surface rather than on derivatives like raster DEMs. Those can be recreated as needed after the measurement edits have taken place. Terrain datasets support this editing model because they maintain a direct link to the source measurement data. When you modify the measurements, you are automatically modifying the terrain in the same process.

How terrains are edited

Editing a terrain dataset is really about editing measurements. Using standard feature edit tools, you can manipulate the measurements that reside in feature classes that participate in a terrain.

Terrains are made from one or more feature classes with some simple rules for each that control how they get used to shape the terrain surface. For example, a multipoint feature class containing lidar points can get added as mass points, a line feature class containing streams and lake shores is used as a source of breaklines, and a polygon feature class can control the data area boundary.

Most feature classes used to define a terrain are what we call referenced. This means that the terrain maintains a pointer, or handle, to them. The terrain prevents its referenced feature classes from being deleted, and pays attention to any edits that occur on them including the addition, deletion, or modification of feature geometry. You can use the feature editor in ArcMap as well as geoprocessing tools to modify these feature classes. A terrain will automatically flag itself as ‘dirty’ in areas where edits were made. Then the terrain can be rebuilt to bring its pyramid in sync with the updated features. It does this based on the dirty areas so it’s a local, or partial, process; the entire terrain does not need to be reconstructed.

Multipoint feature classes have the option of being embedded. When multipoints are embedded the terrain build process copies the points into pyramid tables held private by the terrain and it becomes the container for the points. The terrain does not reference the source feature class. That can be deleted, allowing you to retrieve what is typically a substantial amount of disk space; approximately 1GB per 150 million points. Terrain specific tools, Add Terrain Points (which can both add and replace) and Remove Terrain Points, are used to edit the embedded points based on an area of interest. These tools also offer the benefit of being BLOB attribute aware so if you have any LAS attributes stored with those multipoints (for more on this topic see blog on creating intensity images) the tools know how to keep the BLOB based values in sync with the points relative to the edits. For example, if a few vertices of a multipoint are deleted from an embedded feature class, the terrain will delete the corresponding BLOB based attribute values for those points.

Appending measurements

Measurements can be added to a terrain via the Append and Add Terrain Points geoprocessing tools. The Append tool operates on regular (referenced) feature classes. The Add Terrain Points tool is used to add or replace points in embedded feature classes. You can also add a feature class to an existing terrain via the Add Feature Class To Terrain tool but be aware that this is treated as a schema edit that invalidates the entire terrain, requiring a full rebuild. If data is to be added incrementally, it’s best to append it to a feature class that already participates in the terrain than add a new feature class to the terrain for each new set of data.

Let’s take a scenario where data is provided in phases. In this example the bare earth lidar points are made available first. The breaklines come later in several deliveries. Knowing this schedule, you can create a terrain referencing the lidar multipoint feature class plus an empty line feature class held in anticipation of the breaklines. See Figure 1 with a zoomed-in view of a terrain made solely with bare earth lidar points.

When some of the breaklines are made available they are added to the terrain by adding them to the line feature class referenced by the terrain. This is done using the geoprocessing Append tool (Figure 2).

After running Append, the terrain will become ‘dirty’ in the areas where the lines were added. To see the dirty areas you add a dirty area renderer from the Symbology tab of the terrain layer Properties dialog (Figures 3a and 3b).

At this point the terrain needs to be rebuilt. This is done using either the Build Terrain geoprocessing tool or the Build button on the Update tab of the terrain Properties dialog in ArcCatalog. Once the terrain is re-built the improvement made by the breaklines is evident (Figure 4).

Replace

With lines and polygons in referenced feature classes the replacement of measurements is a two step process. First you delete the old then append the new. If you’re only dealing with a handful of features then you can select and delete them using the Editor in ArcMap. For larger collections rely on geoprocessing tools. For example, use Select By Location followed by Delete Features and Append.

It’s easiest to replace lidar points if they’re embedded. There’s a geoprocessing tool called Add Terrain Points that has a Replace option. This will replace all the points within a given area. So, if you discover something was wrong with a few source point files that were used to build a terrain you can replace them without needing to rebuild the entire terrain from scratch. Figure 5 shows an example where one area, in an otherwise bare earth model, that was inadvertently loaded with first return data.

To fix this problem load the replacement data into a new multipoint feature class. Then run the Add Terrain Points tool with the Replace option. By default, the replacement area will come from the extent of the input feature class (Figure 6).

Once the points have been replaced the terrain needs to be rebuilt to update the affected area. Run the Build Terrain geoprocessing tool or use the Build Terrain button on the Update tab of the terrain Properties dialog in ArcCatalog – either one fixes the terrain.

Conclusion

There are times when updates to a surface model are needed, be it for quality improvement or what-if scenario analysis. It’s hard to make these types of updates to derivative products like raster DEMs without ending up with some anomalies around the update area. It’s more appropriate to modify the source measurement data from which the surface model is derived. For larger datasets, like those coming from lidar, it’s also desirable that datasets only be reprocessed where the updates occur rather than rebuilding everything. Terrain datasets support this by maintaining links to their source measurements in the geodatabase and their use of dirty areas.

That concludes part 6 of Lidar Solutions in ArcGIS. Subscribe to this blog or check back in a month or so for a discussion on how to minimize noise from lidar for contouring and slope analysis.

Lidar Solutions in ArcGIS_part5: Creating Intensity Images from Lidar

bbicking1 — Tue, 12 May 2009 20:50:00 GMT

This is the fifth blog in a series on Lidar Solutions in ArcGIS. It was written by Clayton Crawford, Product Engineer on the 3D Analyst team in ESRI’s Software Products group in Redlands.

Creating Intensity Images from Lidar

Intensity is a measure, collected for every point, of the return strength of the laser pulse that generated the point. It’s based, in part, on the reflectivity of the object struck by the pulse. Other descriptions for intensity include ‘return pulse amplitude’ and ‘backscattered intensity of reflection’. Keep in mind, the reflectivity is a function of the wavelength used which is most commonly in the near infrared. Intensity is used as an aid in feature detection and extraction, lidar point classification, and as a substitute for aerial imagery when none is available. If your lidar points include intensity values you can make images from them that look something like black and white aerial photos (Figure 1).

Loading points with intensity

You'll need source lidar in either LAS or ASCII XYZI format. It’s common to use first returns. If your data are in LAS format use the LAS To Multipoint tool with Intensity selected as an attribute to import (Figure 2). If your data are in ASCII XYZI format use the ASCII 3D To Feature Class tool.

The data loading process results in a multipoint feature class with an attribute field containing the intensity values (Figure 3). You can't read these directly because they're packed into BLOBs (Binary Large OBjects). Every vertex of a multipoint has its intensity mapped into the BLOB stored for that multipoint record.

Substituting intensity for z

A custom VBA script is used to replace every multipoint vertex z with intensity. It does this on a copy of the data. Find the script here; to use it follow the instructions included in the script.

For the ArcObjects geeks out there, take a look at the code to see what it’s doing. Note, there’s a TerrainBlobReader object used to decode the BLOBs. See also that the multipoint vertex ID’s (automatically assigned during import) are used to map into the BLOB value arrays. For the adventurous, note that ArcObjects also provides a TerrainBlobWriter object. If you have a lot of points that need to be stored as multipoints for the sake of efficiency, but also need per-point attribution, these utility objects come in handy.

Another potentially useful script converts, or explodes, multipoints for an area of interest (AOI) into points. Its real utility is that it decodes the lidar attributes stored in BLOBs into readable/usable numeric values placed in the output point feature class attribute table. A word of warning: make sure your AOI is not too large. Keep it small enough so it contains fewer than roughly three million points (adjust according to how patient you consider yourself to be). Find this script here.

Note: Starting with ArcGIS 9.4, the portion of the workflow involving the substitution of intensity for z will not be necessary. The Point to Raster tool will be able to rasterize using the intensity BLOB field directly.

Rasterize the points

Use the Point to Raster tool on the feature class created by the script that swaps intensity with z and set the Value field to Shape (Figure 4). This tells the tool to rasterize using the shape z's which we know are actually intensity. Select MEAN as a rasterization method. The resulting raster is the intensity image. [As a side note: you may find one of the other options is useful for analysis (e.g., using RANGE of intensity as a variable used in feature detection)].

Before moving on, review the presence of NoData. Significant numbers of NoData cells will result if the output cellsize you specified is too small relative to the density of the lidar points. You can see NoData by assigning a color to it on the Symbology tab of the raster layer Properties dialog. If there’s too much, the easiest thing to do is go back and re-run Point to Raster with a larger cellsize. Alternately, you can use a method described in a previous post to fill in missing values using an expression in the Spatial Analyst calculator (see Lidar Solutions in ArcGIS_part2).

Image Display

The range of values in the image is hard to predict without knowing details about how your data was collected and processed. For one thing, the original intensity values are sensor dependent. Secondly, the values may have been adjusted by the vendor (e.g., normalized to a range of 0-255). Because of this it’s hard to say what the best display options are. You will need to experiment with the raster layer stretch type and contrast. Turning on bilinear re-sampling is probably a good idea. If you’re looking for more display possibilities consider combining intensity with another variable such as hillshade (Figure 5).

Conclusion

The return intensity collected for every lidar point can be used to make images. These images have a variety of uses in GIS applications including feature detection and extraction. ArcGIS provides tools to make these images.

That concludes this part of Lidar Solutions in ArcGIS. Subscribe to this blog or check back in a month or so for a discussion on how to update terrain datasets with new measurements.

Using Python 2.5.4 with ArcGIS 9.3.1

bbicking1 — Tue, 12 May 2009 14:44:00 GMT

This blog post was written by David Wynne, Product Engineer on the Geoprocessing Team in ESRI’s Software Products group in Redlands.

The Python Community announced the release of Python 2.5.4 late in December of 2008. Python 2.5.4 is a bug fixes release.

While ArcGIS 9.3.1 was shipped with Python 2.5.1, Python 2.5.4 has now been certified and is supported. In limited testing with ArcGIS 9.3, no issues were encountered. To take advantage of fixes in Python 2.5.4, it can simply be installed on top of Python 2.5.1. That means, it is not necessary to uninstall Python 2.5.1 first.*

To see and to download the release notes go here.

* If you uninstalled Python 2.5.1 and you work with the Spatial Statistics geoprocessing tools, you must reinstall Numerical Python (NumPy) from the ArcGIS installation DVDs.

For more information, see FAQ: Can Python 2.5.4 be used with ArcGIS?

Spatial Statistics: What’s so HOT about Spatial Pattern Analysis?

bbicking1 — Mon, 04 May 2009 16:08:00 GMT

This blog post was written by Lauren Scott, Geoprocessing/Spatial Statistics Product Engineer in the Software Products Group at ESRI in Redlands.

Hot Spot Analysis is just one of the pattern analysis tools in the Spatial Statistics Toolbox.You can use these tools to explore spatial patterns in order to answer questions like:

Where are crime rates unexpectedly high?
Are there regions in the country where people live longer
Where do we find anomalous spending patterns?
Are there sharp boundaries between affluence and poverty?
Is the disease remaining geographically fixed or is it spreading?
Which features are most concentrated?
Does the spatial pattern of the virus mirror the spatial pattern of the population at risk?
Which site is most accessible?
Where is the population center?
Which species has the broadest territory?

To learn more about spatial pattern analysis, check out some of these resources:

- The ESRI Guide to GIS Analysis, Volume 2
- Understanding Spatial Statistics in ArcGIS 9, a free one-hour Web seminar
- The Spatial Statistics Toolbox online documentation
- View a five-minute video showing a hot-spot analysis of 911 emergency call data (click on “Using Spatial Statistics Tools”)
- Download a hot-spot analysis model from the Geoprocessing Resource Center.
- Extend Crime Analysis with ArcGIS Spatial Statistics Tools , Spatial Statistics Provide New Insights, or Spatial Patterns of Disease Inspire New Ideas on Possible Causes in ArcUser Online
- Spatial Pattern Analysis concepts are discussed in the ArcGIS 9.3 Web help and include Modeling Spatial Relations, What is a Z Score? What is a P value?, and Spatial Weights.
- Technical workshop slides are available from the ESRI Public Health and Homeland Security Conferences.

Have fun!

Spatial Statistics: Free Virtual Campus Web Seminar on Regression Analysis Basics

bbicking1 — Mon, 04 May 2009 15:33:00 GMT

This blog post was written by Lauren Scott, Geoprocessing/Spatial Statistics Product Engineer in the Software Products Group at ESRI in Redlands.

There are a number of good resources where you can learn about the new regression analysis tools in ArcGIS 9.3. The newest resource is a free one-hour Web seminar, Regression Analysis Basics in ArcGIS 9.3, available for download from the ESRI Virtual Campus.

Other Regression Analysis resources include:

Regression Analysis Basics Documentation
Ordinary Least Squares Regression Tool Help and Learn more about how OLS works
Geographically Weighted Regression Tool Help and Learn more about how GWR works
Interpreting OLS tool results
Interpreting GWR tool results
Several articles can be found in the Spring 2009 issue of ArcUser magazine [Try the Flip Page view of ArcUser!]:

Answering Why Questions: An introduction to using regression analysis with spatial data;
Regression Analysis Components; and
Applying Geographically Weighted Regression to a Real Estate Problem.

GEOconnexion International Magazine, December/January 2009, includes an article titled Regression Analysis Tools for GIS Modeling
Chapter 5, Analyzing Geographic Relationships, in The ESRI Guide to GIS Analysis, Volume 2 discusses regression analysis and some of the challenges when it is applied to spatial data.

Enjoy!

2009 Developers Summit

dmhoneycutt — Wed, 25 Mar 2009 23:54:00 GMT

The 2009 Developers Summit has come and gone. Thanks to all who attended! We enjoyed meeting with you. We've uploaded the geoprocessing presentations to the Model and Script tool gallery. Session videos have been posted to the Media Gallery.

Slides, scripts, and data from the two workshops on Designing and Building geoprocessing tools are available in the Model and Script tool gallery, or by clicking here.

Slides and other materials from the Building Geoprocessing Services session are available here.

Slides and scripts for the Python Scripting Advanced Techniques session are available here.

Send us your feedback

dmhoneycutt — Wed, 25 Mar 2009 23:47:00 GMT

You may not have noticed, but you can send us feedback about this site. There's a feedback link at the bottom right of the home page, as illustrated at the bottom of this post.

We're particularly interested in hearing from you about blog post topics you'd like to see and suggestions on how we could make this site more useful.

Lidar Solutions in ArcGIS_part4: Estimating Forest Density and Height

bbicking1 — Tue, 17 Mar 2009 15:44:00 GMT

This blog post is written by Clayton Crawford, a Product Engineer in the Software Products Group's 3D Team in Redlands.

This is the fourth in a series on Lidar Solutions in ArcGIS.

Estimating Forest Canopy Density and Height

Canopy density and height are used as variables in a number of applications. These include the estimation of biomass, forest extent and condition, and biodiversity. Canopy density, or canopy cover, is the ratio of vegetation to ground as seen from the air. Canopy height measures how far above the ground the top of the canopy is. Lidar can be used to determine both.

What follows are steps to calculate canopy density and height from lidar points. First, you need lidar that's been classified into ground hits (bare earth) vs. non-ground hits. This type of classification is usually performed by your data provider. Secondly, you need to consider when the lidar was collected and the type of vegetation in the study area. If there are a lot of deciduous trees and the data collection was performed during leaf off, then the density calculation is not going to work.

Loading points into the Geodatabase
To calculate canopy density load the ground, or bare earth, lidar points into one multipoint feature class and above ground points into another. Assuming your data are in LAS format, you do this with the LAS To Multipoint tool. Specify the proper class codes to filter on. Here are the LAS class codes as defined in the LAS 1.1 Standard:

Any LAS file made in the last few years should use these codes if the points have been classified. Unfortunately, there’s still some ambiguity in the standard. For example, we know class ‘2’ is ground but class ‘8’ is ground as well. Class ‘8’, or model key, points are a special set of ground points used for contouring or other application requiring a thinned set of ground points. Whether you have them depends on how the data was processed. If you don’t know specify both classes. If it turns out there aren’t any model key points it won’t hurt. Vegetation has a similar issue. Sometimes vendors place everything that’s above ground into class ‘1’ because they haven’t performed a more detailed classification on them. So, if you’re unsure of the specifics of your data’s classification, load non-ground points using classes 1, 3, 4, and 5; that’s a reasonable catch-all to get your vegetation points. Note: If buildings or other manmade non-ground features are in class ‘1’ you’ll get them too and they’ll skew the results somewhat.

Calculating the density
The most effective way to determine the canopy density is to divide the study area into many small equal sized units. Do this through rasterization. In each raster cell you compare the number of above ground hits to total hits. The trick is to figure out an appropriate cell size. It needs to be at least 4 times the average point spacing. You can go larger but not smaller.

1. Use the Point To Raster tool on the above ground points with the COUNT option.

2. Convert any resulting NoData cells to 0 so that subsequent operations treat zero points in a cell as 0. This is accomplished using the Is Null tool followed by Con.

Next use the Con Tool

3. Repeat steps 1 and 2 with the ground points.

4. Add the above ground and ground rasters to get a total count per cell using the Plus tool.

5. All the rasters we’ve made so far are longs. We need one to be floating point in order to get floating point output from the Divide function that we’ll use in step 6. Do this by sending the output from Plus through the Float tool.

6. Now use the Divide tool between the above ground count raster and the floating point total count raster. This gives us the ratio from 0.0 to 1.0 where 0.0 represents no canopy and 1.0 very dense canopy.

The following image represents canopy density. The lightest areas have little to no vegetation. These are areas where a large percentage of lidar shots could ‘see’ the ground. The dark green areas, where lidar could not penetrate to ground as well, indicate denser vegetation canopy.

Calculating the height
To determine canopy height you'll need to subtract the bare earth surface (DEM) from the 1st return surface (DSM). Take a look at a previous blog to learn how to make these surfaces. Find it at this link: Creating raster DEMs and DSMs from large lidar point collections.

Once you have your first return and bare earth rasters the Minus tool gives you the difference which, over forest, represents the canopy height.

The following image represents canopy height above ground. It ranges from blue for little to no height, to orange which is the tallest.

Conclusion
Lidar can be used to calculate the density and height of vegetation. This is useful for a variety of purposes including biomass and carbon estimates as well as forest management.

That concludes part four of Lidar Solutions in ArcGIS. Subscribe to this blog or check back in a month or so for a discussion on the creation of intensity images from lidar.

Tips and tricks - accessing feature shape in Calculate Field

dmhoneycutt — Fri, 06 Mar 2009 23:44:00 GMT

With the Calculate Field tool, you can easily create expressions that use some property of a feature's shape, such as length or area. You can also convert length and area to different units. The illustration below shows calculating the MILES field to the length (in miles) of each line feature.

Details about the Expression syntax

The expression syntax -- !shape.length@miles! -- starts and ends with the special 'decode field' character. This decode field character is different depending on the value of the Expression Type parameter. For Expression Type of PYTHON or PYTHON_9.3, the character is an exclamation point ("!"), as shown in the illustration above. For VB, the decode field characters are "[" and "]".

Inside the decode field characters, you can put the name of any field found on the input table. You can also use the reserved field name "shape" followed by a period and an attribute of the shape. In this case, we're using the length attribute: !shape.length!

More things you can do with the shape object and its attributes

The shape field is actually a geometry object with the following properties.

If, for example, you wanted the X-coordinate of the first point of a line, you would use !shape.firstpoint.x! To find the minimum X-coordinate of the line, use !shape.extent.xmin!

Simple equations

The expression can contain simple equations. For example, to calculate minutes it would take to traverse a street segment going 30 miles per hour, use:

!shape.length@miles! / (30.0 / 60.0)

Note that:

!shape.length@miles! / (30 / 60)

won't work - when doing real arithmetic you need the decimal point after 30 and 60.

Code blocks

See http://www.esri.com/news/arcuser/0507/files/pythonscript.pdf for information on using code blocks in Calculate Field