We are participating in several interesting presentations and panels:

• Streaming Big Data Analysis with Open Source highlights our recently released GIS Tools for Hadoop, and we will also be announcing a new open-source processing engine. These are core technologies that enable anyone to quickly, and freely, perform highly scaleable geoprocessing.
• Panel: State of the Portals is an open-discussion with several popular data sharing portals. We will share our experience developing GeoPortal Server, specifically regarding metadata, data catalogs, open API’s and usability.
• Lastly, The Business of Open Source will be a fascinating discussion with several prominent companies and platform developers about the various concepts and aspects of Open, and how this benefits both customers and communities.

If you are at the conference, please stop by our booth and introduce yourself. We would love to hear your thoughts and feedback on our large and growing open source library of applications, tools, and libraries. We will also update this post throughout and after the conference with links to our presentations, notes and new code repositories.

Cairo, Egypt, at 1:9k

Dubai, United Arab Emirates, at 1:9k

Lexington, Massachusetts, at 1:2k

Here’s a list of all the community contributors for this release; click this link to view a presentation of their work. The presentation uses ArcGIS Explorer Online, so you can zoom and pan the images.

• Jordan (1:288k to 1:1k)
• Kingdom of Saudi Arabia (1:288k to 1:1k)
• Carson, California, United States (1:9k to 1:1k)
• Commonwealth of Virginia, United States (1:577k to 1:1k)
• Keller, Texas, United States (1:9k to 1:1k)
• Lexington, Massachusetts, United States (1:9k to 1:1k)
• Spokane, Washington, United States (1:9k to 1:1k)
• Wahklakum County, Washington, United States (1:9k to 1:1k)
• Yakima County, Washington, United States (1:9k to 1:1k)

Many of these contributions were made through the Community Maps Program. For more information visit the Community Maps Program Resource Center or view the live map of current and forthcoming contributors.

Spokane, Washington, at 1:9k

Virginia at 1:9k

The service was updated on the following servers: services.arcgisonline.com and server.arcgisonline.com. If you have previously used the World_Topo_Map, you may need to clear your cache in order to see the updates.

If you have questions, please post them in the ArcGIS Online forums.

I often talk with people using ArcGIS Geoprocessing who find themselves surprised about the amount of time the Intersect tool takes to run, or confused about the output results.  Much of the time this confusion comes from a misperception about what tool to use for the analysis, or from a lack of understanding of what the Intersect tool does.

Here’s an example… A user contacted me about the Intersect tool because the tool would run for hours and then fail. They perceived their data as small and really thought Intersect would only take a few minutes.

My first question, and one you should ask too is, “What do you really want for a result?”

The answer in this, and many cases was “I want to know how much of each feature in input 1 is in each feature in input 2.”

Cool!  That means Intersect is not the right tool to use!

Intersect does not compare each feature in the first input to each feature in the second input.  It works in a much more complex, thorough way. The intersect tool is for creating a topologically correct fabric across the entire extent of the dataset and return those newly created features that were created due to overlap from a feature in each input.  The Intersect tool throws all the features from all the inputs into one bucket, and passes this to the underlying topology engine so it can create this topological fabric. This means all relationships, between all features, including those relationships within each input are determined and new features are created for every interaction. The Intersect tool then selects just those features from this topologically correct fabric.

So, for the Intersect tool if you have 2 inputs the output will only contain those features that were created where a feature from each of the 2 inputs overlapped.  If you have 100 inputs the output will only contain those features that were created where a feature from each of the 100 inputs overlapped.

Here is what looks like a very simple example of intersecting one polygon layer with 10 overlapping polygons with a single polygon from a second feature class (features are labeled with their OID):

Input 1 (10 input polys that overlap)

Input2 (One polygon)

Intersect output will contain new features created from the intersection of the overlapping polygons in Input1 and their intersection with the polygon in Input2:

As you can see, there are now a lot more polygons in the output (167!) than there were in the inputs combined.  Imagine if Input1 was actually complex:

Ouch!

Using the output of Intersect you now have to perform further analysis to put together polygons that were broken apart in Input1 due to their intersections with one another. Seems like a lot of work to figure out how much of the polygon with OID 9 in Input1 intersects with the polygon in Input2 right?

So what should they, and you, be doing if you want to know how much of each input feature intersects with each feature in the second input?  Here are two possible workflows, one good, and one much better, that will give you exactly what you want, and may get you the results of your analysis much faster.

Good… Clip and Transfer Attributes workflow

This is a simple approach that uses existing tools and can be setup by anyone familiar with scripting. The only ‘advanced’ part, is the use of a cursor, but it should be fairly easy to understand.

import arcpy, os, sys, time

arcpy.env.workspace = os.getcwd()

arcpy.MakeFeatureLayer_management(r"Data.gdb\input1","input1")
arcpy.MakeFeatureLayer_management(r"Data.gdb\input2","input2")

arcpy.env.overwriteOutput = True

arcpy.CreateFileGDB_management(os.getcwd(),"output.gdb")

try:
    #For each input2 feature in the poly feature class, select it, select the input1 features that intersect it, then clip it with the selected input2 feature
    input2Count = int(arcpy.GetCount_management('input2')[0])
    progressCounter = 1
    startTime = time.time()
    with arcpy.da.SearchCursor("input2", "ID") as cursor:
        for row in cursor:
            print "\n\ninput2 # " + str(progressCounter) + " of " + str(input2Count)
            print "Processing ID = " + str(row[0])
            arcpy.SelectLayerByAttribute_management('input2', "", "ID = " + str(row[0]))

            print "Selecting the input1 features..."
            arcpy.SelectLayerByLocation_management("input1", "INTERSECT", "input2", "", "")

            print "Clipping the selected input1 features with the selected input2 feature..."
            arcpy.Clip_analysis("input1", "input2", r"output.gdb/input1InID" + str(row[0]), "0.001")

            print "Add identification field..."
            arcpy.AddField_management(r"output.gdb/input1InID" + str(row[0]), "ID", "LONG")

            print "Assign input2 ID field value to the output..."
            arcpy.CalculateField_management(r"output.gdb/input1InID" + str(row[0]), "ID", str(row[0]), "PYTHON_9.3")

            progressCounter +=1

    print "Merge all the outputs..."
    arcpy.env.workspace = os.path.join(os.getcwd(), r"output.gdb")
    fcList = arcpy.ListFeatureClasses()
    arcpy.Merge_management(fcList, r"output.gdb/mergedFC")
    stopTime = time.time()
    print "Total time in seconds = " + str(int(stopTime-startTime)) + " and in minutes = " + str(int(stopTime-startTime)/60)
    print "DONE"

except Exception:
    print "FAILED"
    print arcpy.GetMessages()

You add code to transfer as many of the attributes from the second input as you want. If you want this method to go a little faster, you could play with sending the intermediate data to in_memory featureclasses.  Trouble is, you’ll have to manage how many outputs you send to in_memory if your data is very large.  You may use up all your memory storing the intermediate data and cause the workflow to fail.

There are other possible problems with this workflow. The main ones are SelectLayerByLocation ‘INTERSECT’ doesn’t use the exact same rules to select features as Intersect does (see here) and creating and managing all that intermediate data is painful and slow.

So, here’s a much better way to do this…

Better… Intersect geometry method

We can get rid of the creation and maintenance of all that intermediate data, use a method that mimics the Intersect tool, and make this all run tons faster on larger, more complex data.

Let’s use a geometry method instead!

Here is a simplified code snippet explaining the core basics of the workflow:

inCursor = arcpy.da.InsertCursor(r"output.gdb/IntersectResult", fldsOutput)
with arcpy.da.SearchCursor("input1", fldsInput1) as cursor:
    for row in cursor: # For each feature in input1
        # for each feature, select the features in input2 that it intersects
        arcpy.SelectLayerByLocation_management("input2", "INTERSECT", row[-1])
        with arcpy.da.SearchCursor("input2", fldsInput2) as cursor2:
            for row2 in cursor2: # for each selected feature from input2
                # intersect it with the feature from input1
                clippedFeature = row2[-1].intersect(row[-1], 4)
                # setup the insert cursor (for the input you previously created)
                flds2Insert[-1] = clippedFeature
                inCursor.insertRow(flds2Insert) # Insert the resulting geometry from using INTERSECT geometry method

For a fully functional (I hope!) script tool and script that you can try out, download the Pairwise Intersect Tool here.

Taking a look at the same example I used above except this time I use the PairWiseIntersect tool:

That’s more like it!

In testing we’ve done, the “Better” method is multitudes faster with large and/or complex data and it also gets through cases the Intersect Overlay tool appears to have trouble with (and for the most part these ‘trouble’ cases do not make sense to process using the Intersect tool).

Many thanks to Jason Pardy for reviewing the code and blog contents.

We have some new MVP badges in our list!* Welcome to the team, folks! MVP badges do not need to be renewed. Once an MVP, always an MVP. We also give them additional tools like the ability to move a thread from one forum to the other in order to give your questions a better chance of being answered. They can also mark an answer as being the best one by clicking the green check mark. See more about the MVP Program here.

Not only are the MVPs badged so you can find them, but they’re also eligible for prizes like software, books, training courses, and conference seats as a collective ‘thank you’ from us for all of their great effort. First we have our top three Annual Esri MVP Forums users who are being awarded from May 2012 – April 2013:

Robert Scheitlin            rscheitlin

Mathew Coyle               mzcoyle

Anthony Giles                ad_giles@hotmail.com

Next, we are awarding the following Semi-Annual Esri MVP Forums users who are being recognized for their contributions from October 2012 – April 2013:

Robert Scheitlin                    rscheitlin

Anthony Giles                       ad_giles@hotmail.com

Wayne Whitley*                    Wayne_Whitley

Mathew Coyle                        mzcoyle

Curtis Price                             curtvprice

Shaning Yu*                           shaningesri

Richard Watson                     rlwatson

Rhett Zufelt*                          rzufelt

Caleb Mackey*                       Caleb1987

V Stuart Foote                        vsfoote

Again, thank you so much to all of you who help to continue building a great system that supports our Esri community.

Streets in Riyadh, Saudi Arabia, at 1:36k

Click this link to view a web map

Streets in Tel Aviv, Israel, at 1:4k

Click this link to view a web map

World Street Map coverage map

Additionally, the cache for DeLorme World Basemap was updated with more recent data.

The services were updated on the following servers: services.arcgisonline.com and server.arcgisonline.com. If you have previously used these services, and you don’t see anything new, try clearing your cache in order to see the updates.

If you have feedback or comments, please post them to our forum at http://forums.arcgis.com/forums/30-ArcGIS-Online.

A few key enhancements:

• Support for secure map services using basic or digest HTTP security
• Improved feature editing control
• Support for Danish and Portugal Portuguese languages
• Improved Popups

See What’s new for all the enhancements and a list of issues that are addressed with this release.

Apps Team

Several classes in the API have been enhanced to support .on() style events, also referred to as evented. Specifically, esri/map as well as all classes in esri/layers and all esri/tasks now support class.on() style events. Classes in esri/dijit will have support for this style of event management at a future release.

Some samples have been updated to demonstrate the new .on() style event listening (as well as re-written to use AMD):

• Feature layer and a dgrid
• Feature layer from feature collection
• Gas price data
• Map with a popup
• Point clustering
• Show map properties

Please refer to the full What’s New in 3.5 for additional details on class.on() style events.

More samples that use Asynchronous Modules Definition(AMD)

Additional samples that have been re-written to use AMD:

• Customize navigation animation
• Manage multiple query results
• Non-Web Mercator tiled layer
• Popup with a chart
• Vector features in any projection
• Vector features with no basemap

The full What’s New in 3.5 also has a list of bug fixes and enhancements that we added at 3.5.

The download-able version of version of the ArcGIS API for JavaScript will be available soon.

Preconference Seminar

The week begins with the Building the Basemap for the Community Maps Program preconference seminar on July 7th. This one day seminar is a perfect choice for organizations that are stewards of basemap content and are interested in contributing to the Esri Community Maps.  We will go through all of the details of the basemap creation and publishing process and have a fun time doing it.

Technical Workshops

Technical workshops are always one of the most popular types of events at the UC and this year there are two important sessions dealing with Community Maps.

• Getting to Know Esri Community Maps is a perfect choice for users from organizations that are not currently participating in the Community Maps Program, but are interested in joining.  This 75-minute workshop is scheduled on both Tuesday and Wednesday of the conference, so if you miss the first offering you can always make the second.
• Ocean Basemap – An Introduction will be of particular interest to anyone considering contributing bathymetric data to this resource, and those who just want to find out how the map service is produced and what they can do with it.  This workshop is only offered once on Thursday, so if you have an interest in this unique basemap, make sure to save a spot on your agenda.

Demo Theater

In addition to the standard technical workshops, the Community Maps team is also presenting a series of 30-minute Demo Theater presentations at the ArcGIS Online Content Island in the Esri Showcase area. These quick demonstrations provide a great opportunity to see Community Maps in action and include:

• Migrating data for Community Maps with ETL Tools
• Using ArcGIS Data Reviewer to Validate the Quality of Your Community Maps  Content
• Using Community Maps in Simple Web Applications
• Managing Your Community Maps Contribution Online

Moderated Paper Session

Esri staff will not be the only ones conducting Community Maps presentations at the UC.  This year we are also hosting a moderated paper session on Thursday titled Community Maps Contributors Share Their Best Practices.  This is a great opportunity for both new and veteran Esri Community Maps contributors to find out how others have leveraged the program and learn from their experience.

Map Gallery and Software Application Fair

Of course attending presentations is just one way to participate at the UC.  We encourage users who want to take a more active role to enter their maps and apps in the Map Gallery and User Software Application Fair.  The Map Gallery even has a specific Community Maps category – Best Product Utilizing Community Maps.  While the Application Fair does not have a separate category for Community Maps, it is a great opportunity for contributors and others to showcase their interesting and useful web applications that take advantage of the unique, authoritative and richly detailed content in the World Topographic Map.  The Community Maps team will be reviewing the entries to both of these contests and selecting our own favorites for special recognition.  Remember, both of these contest have deadlines for entering, so don’t delay:

• Map Gallery submission deadline – May 31^st.
• User Software Applications Fair submission deadline – June 21^st.

Community Maps Data Health Check

Data contributed to Community Maps is required to conform to the Local Government Information Model schema and meet certain quality criteria.  This year at the UC, we’re introducing 45-minute “health checks” where a Community Maps team member can perform a diagnostic on your data to help you assess its overall readiness for contribution. Using the ArcGIS Data Reviewer extension, we will run various automated checks on your basemap data in file or personal geodatabase format. These checks will reveal whether your data is in the correct schema for contribution and if it has any issues that will slow publishing into the Community Maps online basemap services.  We will also provide you recommendations for addressing any issues found.  Just one 45-minute appointment can ensure that your data contribution is swiftly and smoothly published.  Sign-up for an appointment by emailing datareviewer@esri.com with your name, organization, contact details, and preferred date/time slot.

ArcGIS Online Content Island

Finally, in addition to all of these presentations, you will want to block out some time on your UC agenda to visit the Community Maps booth. Whether you are just curious about the program or you are a current contributor, our team is looking forward to meeting you and discussing any questions you may have about Community Maps. So please stop by.

We look forward to seeing you at the UC!

• Get Your Campus on the Map! Introducing the Community Maps Program – This 75-minute technical workshop will go over all the details of how campus organizations can benefit from participating in the Community Maps Program.
• GIS Solutions EXPO – The Community Maps team will be staffing a booth in the Ed UC showcase area for the Saturday (July 6^th) reception and all day Sunday (July 7^th).  Don’t miss this opportunity to see demonstrations and get your questions about Community Maps answered.
• Hands-on Learning Lab – For anyone wanting self-paced, applied instruction on Community Maps, there is a short tutorial offered in the Hands-on Learning Lab.  This quick lesson will give you details on how to contribute your content to Esri Community Maps, and Esri staff will be available to answer questions.  No registration is required!

We are looking forward to seeing you at the Esri Education Conference, and if you are staying for the Esri International User Conference, don’t forget to check out all of the Community Maps events there.

On the iPhone, we’ve leveraged the native geofencing APIs to offer improved battery life compared to using the GPS directly. On Android, there have been minimal location services available from the SDK, essentially limited to turning the raw GPS device on or off.

Three new location APIs were announced today at Google I/O. This is great news for anyone building location-based applications. We’ve been eager for Google to improve the Android location services.

Fused Location Provider

The Fused Location Provider utilizes all of the communication sensors in the phone including wifi, GPS, and cell network, while using only 1% of the battery that previous APIs used.

This is a new service that will greatly improve any application that uses location services. We will improve the ArcGIS Geotrigger SDK for Android significantly by taking advantage of the new Location Provider. Expect to see much better battery life on the Android Geotrigger SDK in the future!

Geofencing

The new Google Geofencing APIs allow apps to respond to the user entering or exiting a configured geofence. The API allows each app to define 100 geofences simultaneously.

This is great news for the ArcGIS Geotrigger SDK, since we will be able to leverage this service to provide better battery life and performance in the SDK. Previously we were limited to turning on and off the GPS device. Now we will be able to use the geofencing API in combination with our optimization algorithms to better provide our Geotrigger service.

Activity Recognition

The new Activity Recognition API uses machine learning classifiers to determine whether the user is walking, cycling or driving. Apps will be able to adjust their behavior depending on the user’s mode of transport. This is done in a very battery efficient way, no GPS is required.

We will leverage this new API to make further improvements to managing the native location services, and it should open up possibilities for some interesting new apps to be built.

The ArcGIS Geotrigger Service

The ArcGIS Geotrigger Service allows developers to easily manage Geotrigger rules with a simple cloud-based API, while also supporting geofences defined as polygons. You can configure an essentially unlimited number of Geotrigger rules in the API, and only the nearby ones will be synced to the phone, making it easy to handle the limitations of the native APIs.

Many of the battery life issues people experience are due to poor implementations of using the Apple and Android location services. It’s easy for a developer to leave the GPS on all the time, but that drains battery quickly. The Geotrigger SDK efficiently manages the native location services and optimizes the server communication required to send location data, saving battery life wherever possible.

The improvements to the Android location APIs are fantastic news and will result in improved Geotrigger services and a better experience for developers!

Aaron Parecki
CTO, Esri R&D Center, Portland

Amber Case
Director, Esri R&D Center, Portland