Data access tools

Describe Multidimensional Dataset
This tool accesses a local or remote (via the OPeNDAP protocol) multidimensional file and reports the names and attributes of the variables contained in the file.  Global information about the file origin and processing history are also reported, if available.  This tool is the ArcGIS geoprocessing equivalent of the popular netCDF command line tool ncdump.

• OPeNDAP to NetCDF
  This tool will subset and download data from web-based servers which support the OPeNDAP protocol. You can constrain which data are downloaded by specifying specific variables, a spatial extent, and starting and ending values of the dimensions which define a variable.

Layer creation tools

The core geoprocessing tools Make NetCDF Feature Layer, Make NetCDF Raster Layer, and Make NetCDF Table View were designed to provide you with the maximum flexibility for reading a broad range of netCDF formatted files.  However, providing this level of flexibility meant that the tools had to have many parameters.  These new tools are designed to provide a simpler way to access netCDF files that contain well-known geometries, such as station and trajectory data.

• Make NetCDF Regular Points Layer
  This tool makes a point feature layer from a netCDF file which contains regularly gridded points.
• Make NetCDF Station Points Layer
  This tool makes a point feature layer from a netCDF file which contains data from observation stations such as weather stations or fixed ocean buoys.
• Make NetCDF Trajectory Points Layer
  This tool makes a point feature layer from a netCDF file which contains observations from moving platforms such as planes or ships.

Downloading and installing
The Spatial Analyst Supplemental Toolbox is available for download here:
http://esriurl.com/MultidimensionSupplementalTools

Once you’ve downloaded the file to your machine, extract the contents of the zip file.  Then, in Catalog, navigate to the location you extracted the files to. You should see installation instructions and the Python toolbox containing the Supplemental tools.
(Note that the samples are geared for ArcGIS version 10.2 and beyond.)

NITF for ArcGIS for Linux Server requires ArcGIS 10.1 Service Pack 1 (SP1).

Last week, the maps were updated with 2012 demographic estimates from Esri and selected Census 2010 data.  The maps are updated with fresh cartography and improved popups to better support each map’s theme.  For example, the Median Household Income map has a popup that gives details about how many households fall into various income groups.  As in previous years, each map is multi-scale, giving more and more detail as the viewer explores their state, county, metro area and neighborhood.

Here are some other examples:

Households by Net Worth in California

Click this link to view a web map

Population Density in New York City

Click this link to view a web map

As noted in this previous blog, application developers will want to review the changes to the underlying map service fields.  Most of the commonly-used field IDs are unchanged.  Some services have additional or changed fields to better support popups and charts.

If you have questions, please post them in the ArcGIS Online forums.

Business Rule:

I have a business rule that states “All single family parcels must have only one address associated with it.”  Therefore, I want to find single family parcels that have more than one associated address, that way I can address the errors in those parcels. To illustrate, I am using features defined in Esri’s Local Government Information Model but you can leverage these concepts in any data model you choose.

The closest Data Reviewer automated check to perform this type of a spatial validation is the Geometry on Geometry check.  It can find parcel polygons that are missing address points as well as parcel polygons that contain address points.  However, in its current design, the check cannot identify how many address points are in a parcel polygon.

Solution:

Shown below is a GP model that finds all single family parcels that have more than one associated address and writes the records to the Data Reviewer table to be addressed as part of the error correction process.

Here’s how the model has been set up:

1. I used two feature classes: Owner Parcels that have records related to Owner Type (Condo, Single Family Home, etc.) and Address Points that have records related to addresses associated with Owner Parcels. In the model, I configured the Spatial Join GP tool by choosing Intersect (Match Option) between Owner Parcels (Target Features) and Address Points (Join Features).

2. The output is a polygon feature class (OwnerParcel_SpatialJoin) having records from Owner Parcels with associated address. This output feature class is passed as input to Select GP tool.

3. I then used the Select GP tool to find single family homes that have multiple address points by using the following expression: Join_Count >1 AND “OWNTYPE” = ‘Single Family Home’. Join_Count is a field that indicates how many join features match each target feature. The output of Select GP tool is a polygon feature class (OwnerParcel_SpatialJoin_Selected) that has single family homes with more than one address points.

4. This output feature class is passed to the Write to Reviewer Table GP tool as Input Features parameter.

5. I then configured the Create Reviewer Session GP tool to create a new Reviewer session. This is passed as Session input parameter to the Write to Reviewer Table GP tool.

6. Finally I configured the Write to Reviewer Table GP tool and the Owner Parcels that have “OWNTYPE” = ‘Single Family Home’ and more than one associated addresses are written to the newly created session in the Reviewer table.

All parcels with multiple addresses will be written to the Reviewer workspace upon running this GP model. An editor can then go through each result individually and address the problem to see if the parcel was miscoded as a single family or if there may have been a misplaced address point.

By leveraging geoprocessing either in model builder or Python you can easily extend your data validation to support situations when one of Data Reviewer’s out-of-the-box checks cannot be used to validate specific business rules.  Further, by using the Write to Reviewer Table GP tool you can store all your results in a centralized location, namely the Reviewer workspace.

Happy validating!

Content contributed by Chandan Banerjee

We are participating in several interesting presentations and panels:

• Streaming Big Data Analysis with Open Source highlights our recently released GIS Tools for Hadoop, and we will also be announcing a new open-source processing engine. These are core technologies that enable anyone to quickly, and freely, perform highly scaleable geoprocessing.
• Panel: State of the Portals is an open-discussion with several popular data sharing portals. We will share our experience developing GeoPortal Server, specifically regarding metadata, data catalogs, open API’s and usability.
• Lastly, The Business of Open Source will be a fascinating discussion with several prominent companies and platform developers about the various concepts and aspects of Open, and how this benefits both customers and communities.

If you are at the conference, please stop by our booth and introduce yourself. We would love to hear your thoughts and feedback on our large and growing open source library of applications, tools, and libraries. We will also update this post throughout and after the conference with links to our presentations, notes and new code repositories.

Cairo, Egypt, at 1:9k

Dubai, United Arab Emirates, at 1:9k

Lexington, Massachusetts, at 1:2k

Here’s a list of all the community contributors for this release; click this link to view a presentation of their work. The presentation uses ArcGIS Explorer Online, so you can zoom and pan the images.

• Jordan (1:288k to 1:1k)
• Kingdom of Saudi Arabia (1:288k to 1:1k)
• Carson, California, United States (1:9k to 1:1k)
• Commonwealth of Virginia, United States (1:577k to 1:1k)
• Keller, Texas, United States (1:9k to 1:1k)
• Lexington, Massachusetts, United States (1:9k to 1:1k)
• Spokane, Washington, United States (1:9k to 1:1k)
• Wahklakum County, Washington, United States (1:9k to 1:1k)
• Yakima County, Washington, United States (1:9k to 1:1k)

Many of these contributions were made through the Community Maps Program. For more information visit the Community Maps Program Resource Center or view the live map of current and forthcoming contributors.

Spokane, Washington, at 1:9k

Virginia at 1:9k

The service was updated on the following servers: services.arcgisonline.com and server.arcgisonline.com. If you have previously used the World_Topo_Map, you may need to clear your cache in order to see the updates.

If you have questions, please post them in the ArcGIS Online forums.

I often talk with people using ArcGIS Geoprocessing who find themselves surprised about the amount of time the Intersect tool takes to run, or confused about the output results.  Much of the time this confusion comes from a misperception about what tool to use for the analysis, or from a lack of understanding of what the Intersect tool does.

Here’s an example… A user contacted me about the Intersect tool because the tool would run for hours and then fail. They perceived their data as small and really thought Intersect would only take a few minutes.

My first question, and one you should ask too is, “What do you really want for a result?”

The answer in this, and many cases was “I want to know how much of each feature in input 1 is in each feature in input 2.”

Cool!  That means Intersect is not the right tool to use!

Intersect does not compare each feature in the first input to each feature in the second input.  It works in a much more complex, thorough way. The intersect tool is for creating a topologically correct fabric across the entire extent of the dataset and return those newly created features that were created due to overlap from a feature in each input.  The Intersect tool throws all the features from all the inputs into one bucket, and passes this to the underlying topology engine so it can create this topological fabric. This means all relationships, between all features, including those relationships within each input are determined and new features are created for every interaction. The Intersect tool then selects just those features from this topologically correct fabric.

So, for the Intersect tool if you have 2 inputs the output will only contain those features that were created where a feature from each of the 2 inputs overlapped.  If you have 100 inputs the output will only contain those features that were created where a feature from each of the 100 inputs overlapped.

Here is what looks like a very simple example of intersecting one polygon layer with 10 overlapping polygons with a single polygon from a second feature class (features are labeled with their OID):

Input 1 (10 input polys that overlap)

Input2 (One polygon)

Intersect output will contain new features created from the intersection of the overlapping polygons in Input1 and their intersection with the polygon in Input2:

As you can see, there are now a lot more polygons in the output (167!) than there were in the inputs combined.  Imagine if Input1 was actually complex:

Ouch!

Using the output of Intersect you now have to perform further analysis to put together polygons that were broken apart in Input1 due to their intersections with one another. Seems like a lot of work to figure out how much of the polygon with OID 9 in Input1 intersects with the polygon in Input2 right?

So what should they, and you, be doing if you want to know how much of each input feature intersects with each feature in the second input?  Here are two possible workflows, one good, and one much better, that will give you exactly what you want, and may get you the results of your analysis much faster.

Good… Clip and Transfer Attributes workflow

This is a simple approach that uses existing tools and can be setup by anyone familiar with scripting. The only ‘advanced’ part, is the use of a cursor, but it should be fairly easy to understand.

import arcpy, os, sys, time

arcpy.env.workspace = os.getcwd()

arcpy.MakeFeatureLayer_management(r"Data.gdb\input1","input1")
arcpy.MakeFeatureLayer_management(r"Data.gdb\input2","input2")

arcpy.env.overwriteOutput = True

arcpy.CreateFileGDB_management(os.getcwd(),"output.gdb")

try:
    #For each input2 feature in the poly feature class, select it, select the input1 features that intersect it, then clip it with the selected input2 feature
    input2Count = int(arcpy.GetCount_management('input2')[0])
    progressCounter = 1
    startTime = time.time()
    with arcpy.da.SearchCursor("input2", "ID") as cursor:
        for row in cursor:
            print "\n\ninput2 # " + str(progressCounter) + " of " + str(input2Count)
            print "Processing ID = " + str(row[0])
            arcpy.SelectLayerByAttribute_management('input2', "", "ID = " + str(row[0]))

            print "Selecting the input1 features..."
            arcpy.SelectLayerByLocation_management("input1", "INTERSECT", "input2", "", "")

            print "Clipping the selected input1 features with the selected input2 feature..."
            arcpy.Clip_analysis("input1", "input2", r"output.gdb/input1InID" + str(row[0]), "0.001")

            print "Add identification field..."
            arcpy.AddField_management(r"output.gdb/input1InID" + str(row[0]), "ID", "LONG")

            print "Assign input2 ID field value to the output..."
            arcpy.CalculateField_management(r"output.gdb/input1InID" + str(row[0]), "ID", str(row[0]), "PYTHON_9.3")

            progressCounter +=1

    print "Merge all the outputs..."
    arcpy.env.workspace = os.path.join(os.getcwd(), r"output.gdb")
    fcList = arcpy.ListFeatureClasses()
    arcpy.Merge_management(fcList, r"output.gdb/mergedFC")
    stopTime = time.time()
    print "Total time in seconds = " + str(int(stopTime-startTime)) + " and in minutes = " + str(int(stopTime-startTime)/60)
    print "DONE"

except Exception:
    print "FAILED"
    print arcpy.GetMessages()

You add code to transfer as many of the attributes from the second input as you want. If you want this method to go a little faster, you could play with sending the intermediate data to in_memory featureclasses.  Trouble is, you’ll have to manage how many outputs you send to in_memory if your data is very large.  You may use up all your memory storing the intermediate data and cause the workflow to fail.

There are other possible problems with this workflow. The main ones are SelectLayerByLocation ‘INTERSECT’ doesn’t use the exact same rules to select features as Intersect does (see here) and creating and managing all that intermediate data is painful and slow.

So, here’s a much better way to do this…

Better… Intersect geometry method

We can get rid of the creation and maintenance of all that intermediate data, use a method that mimics the Intersect tool, and make this all run tons faster on larger, more complex data.

Let’s use a geometry method instead!

Here is a simplified code snippet explaining the core basics of the workflow:

inCursor = arcpy.da.InsertCursor(r"output.gdb/IntersectResult", fldsOutput)
with arcpy.da.SearchCursor("input1", fldsInput1) as cursor:
    for row in cursor: # For each feature in input1
        # for each feature, select the features in input2 that it intersects
        arcpy.SelectLayerByLocation_management("input2", "INTERSECT", row[-1])
        with arcpy.da.SearchCursor("input2", fldsInput2) as cursor2:
            for row2 in cursor2: # for each selected feature from input2
                # intersect it with the feature from input1
                clippedFeature = row2[-1].intersect(row[-1], 4)
                # setup the insert cursor (for the input you previously created)
                flds2Insert[-1] = clippedFeature
                inCursor.insertRow(flds2Insert) # Insert the resulting geometry from using INTERSECT geometry method

For a fully functional (I hope!) script tool and script that you can try out, download the Pairwise Intersect Tool here.

Taking a look at the same example I used above except this time I use the PairWiseIntersect tool:

That’s more like it!

In testing we’ve done, the “Better” method is multitudes faster with large and/or complex data and it also gets through cases the Intersect Overlay tool appears to have trouble with (and for the most part these ‘trouble’ cases do not make sense to process using the Intersect tool).

Many thanks to Jason Pardy for reviewing the code and blog contents.

We have some new MVP badges in our list!* Welcome to the team, folks! MVP badges do not need to be renewed. Once an MVP, always an MVP. We also give them additional tools like the ability to move a thread from one forum to the other in order to give your questions a better chance of being answered. They can also mark an answer as being the best one by clicking the green check mark. See more about the MVP Program here.

Not only are the MVPs badged so you can find them, but they’re also eligible for prizes like software, books, training courses, and conference seats as a collective ‘thank you’ from us for all of their great effort. First we have our top three Annual Esri MVP Forums users who are being awarded from May 2012 – April 2013:

Robert Scheitlin            rscheitlin

Anthony Giles                ad_giles@hotmail.com

Mathew Coyle               mzcoyle

Next, we are awarding the following Semi-Annual Esri MVP Forums users who are being recognized for their contributions from October 2012 – April 2013:

Robert Scheitlin                    rscheitlin

Anthony Giles                       ad_giles@hotmail.com

Wayne Whitley*                    Wayne_Whitley

Mathew Coyle                        mzcoyle

Curtis Price                             curtvprice

Shaning Yu*                           shaningesri

Richard Watson                     rlwatson

Rhett Zufelt*                          rzufelt

Caleb Mackey*                       Caleb1987

V Stuart Foote                        vsfoote

Again, thank you so much to all of you who help to continue building a great system that supports our Esri community.

Streets in Riyadh, Saudi Arabia, at 1:36k

Click this link to view a web map

Streets in Tel Aviv, Israel, at 1:4k

Click this link to view a web map

World Street Map coverage map

Additionally, the cache for DeLorme World Basemap was updated with more recent data.

The services were updated on the following servers: services.arcgisonline.com and server.arcgisonline.com. If you have previously used these services, and you don’t see anything new, try clearing your cache in order to see the updates.

If you have feedback or comments, please post them to our forum at http://forums.arcgis.com/forums/30-ArcGIS-Online.

A few key enhancements:

• Support for secure map services using basic or digest HTTP security
• Improved feature editing control
• Support for Danish and Portugal Portuguese languages
• Improved Popups

See What’s new for all the enhancements and a list of issues that are addressed with this release.

Apps Team