Category: Python

Downloadable tools for ArcGIS Server administration

With the release of version 10.1, automating administrative tasks on ArcGIS Server got a whole lot easier. The new ArcGIS Server Administrator API allows you to perform the same administrative tasks you can do in ArcGIS Desktop or ArcGIS Server Manager. Because it’s a REST-ful API, you can programmatically build up URLs to make calls to the server to perform tasks.

Using this API, I’ve published a sample that puts server admin tasks into an ArcGIS toolbox and ModelBuilder. The ArcGIS Server Administration Toolkit (download), written with Python, provides a variety of tools that you can use to administer the server. For example, there are tools for starting and stopping services, modifying log settings, registering server object extensions (SOEs), and publishing services from service definition files. You can chain these tools together in ModelBuilder to complete a workflow, or you can run them in a one-off fashion directly from the toolbox.

Continue reading

Posted in Python, Services | Tagged , , | Leave a comment

Be successful overlaying large, complex datasets in Geoprocessing

Spatial datasets often have large numbers of features.  Furthermore, these datasets may also contain individual features such as road casings that are very large and complex.

As a result of such large datasets, massive amounts of feature overlap (number of features and complexity of the overlapping features) within feature classes can result in a geoprocessing tool failure or poor performance.  These failures are often due to system resources limitations, mismanagement of resources, “bad” data, or massive overlap in an area that cannot be processed within the available resources.  Failure messages such as “Invalid Topology [4GB_FILE_LIMIT] “, or “Out of memory” can occur.  In rare cases, an application crash is possible.

At 10.1, significant work has been done for managing the processing of large complex data.  We have removed internal limitations and overhauled our overlay engine’s memory management.  Some overlay processes that previously failed can now complete successfully, and with reasonable performance.  However, following these guidelines and best practices should ensure your large overlay processes complete successfully.

Continue reading

Posted in Analysis & Geoprocessing, Python | Tagged , , , , , , , | 6 Comments

ArcGIS 10.1 Available Soon

The ArcGIS 10.1 Pre-release program has come to a close, and the final release of ArcGIS 10.1 is expected to be available in the next few weeks. Thanks for all the great feedback during beta and pre-release. The 10.1 release has lots of new and exciting capabilities. For more information on ArcGIS 10.1 visit http://www.esri.com/whatscoming. And for details on what’s new, check out this PDF that list all the new functionality.

Posted in 3D GIS, Analysis & Geoprocessing, ArcGIS Online, Community Maps, Defense, Developer, Editing, Geodata, Imagery, Local Government, Mobile, Python, Services, Spatial Statistics, Web | 5 Comments

It’s coming – 6/12/12!

New Resource Center Home page

This Resource Center is getting a facelift and a fresh new look. We’re updating the color and styling of the website, adding tutorials for you to try ArcGIS, improving the gallery and video experiences, and much more.

We’ve been testing the website improvements as part of ArcGIS 10.1 Beta and Prerelease and now, in conjunction with its release, we’re ready to share the website improvements with everyone.

Click here to preview what’s coming. Then, on the evening of 6/12/12, the improvements will be applied to this website: http://resources.arcgis.com. All your bookmarks and page share will continue to work; some pages will just look a little different, and there will be lots of great new content.

Continue reading

Posted in 3D GIS, Analysis & Geoprocessing, ArcGIS Online, Community Maps, Defense, Developer, Editing, Electric & Gas, Geodata, Hydro, Imagery, Local Government, Location Analytics, Mapping, Mobile, National Government, Public Safety, Python, Services, Spatial Statistics, State Government, Telecommunications, Transportation, Water Utilities, Web | Tagged , | Leave a comment

Aggregated Live Feed methodology: The next evolution

Have you ever wanted to add live weather, recent earthquakes, or perhaps current fire locations to your applications without writing any code?  Many of the projects we address in Esri Technical Marketing have this very requirement.

We tackled this challenge by using  what we call the Aggregated Live Feed methodology. This process downloads data from live sources such as NOAA and the USGS and aggregates it into a geodatabase, which is then served through ArcGIS Server as map services. You can see some of these feeds in action by visiting any of the ‘latest incident maps’ on the Esri Disaster Response site.

Recently we’ve developed a much simpler approach called ALF-Lite that doesn’t require specialized knowledge of the enterprise geodatabase or third-party components. This methodology can be deployed to any large or small environment that supports Esri’s ArcPy site package. Continue reading

Posted in Electric & Gas, Public Safety, Python, Services, Telecommunications, Web | Tagged , | Leave a comment

ArcGIS 10.0 Service Pack 4 (SP4) for English

By Aileen Buckley, Mapping Center Lead

ArcGIS Online web site

ArcGIS 10.0 Service Pack 4 (SP4) for English, is now available to our users for download via the ArcGIS Resource Center. This Service Pack contains performance improvements and maintenance fixes. Here are links to the downloads:

ArcGIS 10 SP4

ArcIMS 10 SP4

ArcSDE 10 SP4

Release Note: ArcGIS 10.0 Service Pack 4 for the other five languages (French, German, Japanese, Simplified Chinese and Spanish) will be released in the next two weeks. A follow up blog entry will be posted to announce the availability once they are ready.

Posted in 3D GIS, Analysis & Geoprocessing, Community Maps, Defense, Developer, Editing, Electric & Gas, Geodata, Hydro, Imagery, Local Government, Mapping, Mobile, National Government, Public Safety, Python, Services, Spatial Statistics, Telecommunications, Water Utilities, Web | Tagged , , , | 21 Comments

Welcome to the new unified ArcGIS Blog

For the past few years the various ArcGIS product development teams have hosted a few dozen separate blogs covering the width of the ArcGIS system. Now we have pulled those together into a single ArcGIS Blog so that you can more easily browse, subscribe to, learn from, and stay up-to-speed on the latest information from all our engineers and developers. In addition, the single blog reflects ArcGIS as a system and allows us to better tell big picture implementation stories that we couldn’t in the fragmented system.

If you do want to narrow your focus, just click a Category (under Technical Communities and Industry Communities) or Tag to see the posts for that particular area. You should find a category or tag that matches up with the subject area for each of the old separate blogs. A few may have been overlooked or thought to be unnecessary. We appreciate your feedback on specific categories or tags that should be added to help focus in on subject matter that is of interest to you. Continue reading

Posted in 3D GIS, Analysis & Geoprocessing, ArcGIS Online, Defense, Developer, Editing, Hydro, Imagery, Local Government, Mapping, Mobile, National Government, Public Safety, Python, Services, Spatial Statistics, Telecommunications, Water Utilities, Web | Tagged , , | 5 Comments

The New ArcGIS Discussion Forums MVP Program

Have a question?  Want to talk GIS? Looking for some good ideas?  Jump into the conversation!

A few weeks back we rolled out some new features that we hope will improve the forums’ usefulness.  One of the new features is the ability for those who post questions in the forums to rate the replies.  In addition, any logged in user can also join in and vote on the various replies and conversations in order to help us identify the best information, helping you find good information more quickly.  Another huge benefit to the new voting tools is that it helps us all identify, recognize, and reward those forums users who contribute the most helpful information.

Until June 2010, our old discussion forums (now archived) supported what we called the “Forums MVP Program”.  Once every 6 months we would identify the Top 10 members of the community and make them eligible for MVP recognition and some valuable prizes. Continue reading

Posted in 3D GIS, Analysis & Geoprocessing, Developer, Editing, Geodata, Imagery, Mobile, Python, Services, Spatial Statistics, Web | Leave a comment

New Features for the ArcGIS Discussion Forums

Have a question?  Want to talk GIS? Looking for some good ideas?  Jump into the conversation!

A few weeks back we rolled out some features that we hope will improve the forums’ usefulness.  To be candid, we’re sure many would say that some of these features should have been there since day one, or at least long overdue, but we listened to what you wanted and found a way to get them in there for you to use.

Let’s start with the big one…

Community Voting
If you read something and you like it, give it a thumbs up!

Now any forum user can let the community know where the good information is. How you do it is simple. If you read a post or a reply and you think it contains some really good information, click the “up” arrow on the right. This is similar to the “Like” button in Facebook.

If we all do this, then the best posts will bubble to the top. If you want to read the entire thread you can, but if you need to most quickly find the good stuff, look at the posts that have a high score.

There is also a “down” arrow, but you can only use that to “Unlike” something you previously Liked.

If you find “The Answer”, give it a check!

If you started a thread with a question, then whichever reply you think is the best, give it a check. That will mark your thread as “answered” for all to see, and it will give some MVP points to the person who wrote it.

Two birds with one stone
Clicking the checks and arrows has two benefits:
1.    You are helping everyone find the best information.
2.    You are helping everyone recognize the best contributors.

One of the great things about any community is the trust earned by those folks who share their experience and help others. Some pros out there are always going to stick out, but these new voting tools are going to help find others who are just as helpful and useful who you might not yet know.

Discussions versus Q&A

When you start a new thread, you can let everyone know if you’re starting a discussion or if you’re asking a specific question. Discussions show up in the thread list with a yellow “D” icon and questions with a red “Q”.

When the original poster or a moderator identifies one of the replies as the best answer, the “Q” icon turns into a green “A”.

This helps you find answers more quickly, and if you want to jump in and let everyone know what you think, this helps you more quickly find questions that haven’t been answered yet.

Private Messages
Most of the time an open public discussion is a great way to get the best information, but sometimes you may want to take it off-line.

Up on the menu is a “Private Messages” link. Click that to see your Inbox or to send direct messages to other users. Also, clicking their name anywhere in the forums provides a pop-up you can use to send a message to them if they’ve activated it.

You can use the “Forum Actions > User Control Panel” menu to control who can send you messages.  You can turn it completely on, completely off, or limited to just those users in your Contacts list.

New Badges
Badges are a great way to find those folks who have been around the block a time or two. Anyone with more than 200 posts in the forums becomes a “Senior Member”, and of course anyone on the forums who works for Esri is badged as “esri” with a globe logo.

And when you see someone with a “Forums MVP” badge, you know you’re talking with someone the community has said has the best information and is most dedicated to your success. These are folks who have been voted by the community to be in the Top 10 of all contributors during any previous six-month MVP rating period. And once you’re an MVP, you’re always an MVP.

And more importantly, now that the new community voting tools have been included, it’s time to roll out the new MVP program. Watch this blog post next week for a description of the new rules, new standards, and a list of awards you can earn through all of your good effort. So jump in and help us figure out who the players are; maybe it’s you!

What’s Next?
The Advanced Search page gives you a lot of flexibility to build a complex search.  What we’re working on next is giving you the ability to save that search definition. This will be good for bookmarking and sharing. We’ve also heard that most users participate in some forums a lot, some forums a little and others not at all. We are going to improve your “What’s New” page so that it only includes those forums you want to browse. If there are any other improvements you’d like to see, reply here, or jump into the conversation on the Resource Center Site Feedback forum.

Content for this post provided by Jim Barry

Posted in 3D GIS, Analysis & Geoprocessing, ArcGIS Online, Developer, Editing, Geodata, Imagery, Mobile, Python, Services, Spatial Statistics, Web | Tagged , | 1 Comment

Implementing topology through Python scripting

Geoprocessing can be used to automate many aspects of data compilation, including implementing a geodatabase topology. In my previous post, I created a model that imports data into a file geodatabase and performs initial data cleanup processing. Now, I will use geoprocessing with Python scripting to build a topology on the imported data. Although many spatial integrity issues were resolved by running the model, a geodatabase topology can help check for and repair any remaining errors.

The dataset I am using is a feature class of parcel lot line boundaries that began as a CAD file full of topological inconsistencies, such as lines that overlapped or did not connect to other lines. Once I create the topology and am sure the lines are correct, I will build parcel polygons from the lines and introduce all the features into my production enterprise geodatabase.

Creating a topology in a script
To build topology in an automated manner, I am going to write a simple Python script, add it to a toolbox, and run it as a regular geoprocessing tool. Essentially, the script just performs the functions of the New Topology wizard, but without requiring my intervention. In fact, this script could be developed without any programming by creating a model to perform these tasks and exporting it from ModelBuilder as a Python script.

When scripting, the ArcPy site package allows Python to access and run any of the geoprocessing tools in the ArcGIS system toolboxes, including the topology tools. The Topology toolset in the Data Management toolbox contains all the tools I need to add a geodatabase topology to the line feature class. The import arcpy statement adds ArcPy to a script and is at the top of every ArcGIS Python script.

Before I start adding the tools to the script, I define variables for the paths to the feature dataset, feature class, and topology. Because I created folders to store my tools and data following the recommendations in A structure for sharing tools, I can make the script more portable by setting these paths in relation to the folder containing the script. Since the folder locations are not hard-coded to match the C: absolute path of my hard drive, the script should run on someone else’s machine regardless of the machine’s directory structure.

With the paths defined, I first use the Create Topology tool to add a topology to the feature dataset. The syntax for the tool in my script is arcpy.CreateTopology_management(featureDataset, topologyName, “”), where CreateTopology is the name of the Create Topology tool,  _management is the toolbox in which it resides, featureDataset is a variable I defined earlier representing the path to the feature dataset, and topologyName is a variable for the name of the topology. When working with topology, it is recommended to use the default cluster tolerance, which is the distance in which vertices are determined to be coincident. Since I want ArcGIS to calculate the cluster tolerance, I left the value blank as “”instead of supplying one.

Although that function creates a topology, it is currently empty and has no classes or rules in it. I can use the Add Feature Class To Topology tool to add my line feature class with the statement: arcpy.AddFeatureClassToTopology_management(topology, featureClass, “1″, “1″). If the topology contained multiple feature classes, I can set ranks so the feature class with the highest accuracy is not adjusted to match vertices in a feature class which is known to be less accurate. However, since I am using only one feature class, I leave the rank parameters as “1″.

Next, I set which topology rules to include by calling the Add Rule To Topology tool. When choosing which rules to add to a topology, there are a few rules that many editors commonly add to every topology, such as the line rule for Must Not Have Dangles. In the script, this rule is coded as arcpy.AddRuleToTopology_management(topology, “Must Not Have Dangles (Line)”, featureClass, “”, “”, “”). Because the rule only applies to one feature class that does not have subtypes and is not a rule between two feature classes or subtypes, the other parameters are left blank. I also want to make sure that the lines do not overlap or intersect themselves, so I include the Must Not Overlap and Must Not Self-Intersect rules as well. If I need to add more rules later, I can run the Add Rule to Topology tool or use the topology’s Properties dialog box in the Catalog window.

Once all the feature classes and rules have been added, my script validates the topology. The Validate Topology tool identifies features that share geometry, inserts common vertices into features that share geometry, then performs integrity checks to identify any violations of the rules that I defined for the topology. Since the topology has never been validated before, I am going to validate the entire extent of the data with the syntax, arcpy.ValidateTopology_management(topology, “Full_Extent”). However, when working in ArcMap, validating the visible extent of the map instead of the full extent limits the area to be validated and can be useful for very large datasets that take a long time to validate.

Finding and fixing topology errors
After the script runs, I add to ArcMap the resulting topology so I can inspect the results and fix any errors using the ArcMap editing tools. However, with all the previous automated QA work from my Import and Clean Lines model, the remaining manual edits are minimal in comparison to what they could have been without running it first.

The errors identified with topology are indicated by the orange-colored squares. Almost all of these are dangle errors that could not be fixed by the original model since they exceeded the tolerance value for the Extend, Trim, or Snap tools. The topology found only two Must Not Overlap errors, which I can fix by deleting one of the overlapping features. There are no violations of the Must Not Self-Intersect rule, indicating that the lines were split properly in the model. Using the editing tools on the Topology toolbar, such as the Error Inspector and Fix Topology Error tool, I can review each error to determine if the built-in topology fixes can be used or if the lines should be edited manually to resolve the topology error.

In some cases, the topology error may need to be marked as an exception, which is a valid violation of a topology rule. One of the most common examples of exceptions to the Must Not Have Dangles rule is a cul-de-sac road, which are dead ends that do not connect to other roads. However, when working with parcel lot lines, there are fewer scenarios that are valid exceptions. I do have some lines at the edges of the dataset that do not connect to other lines. I can either mark these as exceptions or choose to delete the features, depending on whether these features are supposed to connect to existing features in my enterprise database.

If an edit is made to correct a topology error, I have to validate the topology again to make sure the error no longer exists. After I perform a visual inspection and fix all the remaining topology errors, I can create new polygons representing landownership parcels using the geometry of the lines. If I attempted to create polygons from lines that do not connect to each other properly, either no polygons would be created or one large polygon would result where there should actually be two polygons. After creating the new polygons, I add the polygon feature class to the topology and check for any gaps or overlaps and make sure the parcel polygon boundaries are always coincident with the lot lines.

After implementing the topology and making edits in ArcMap, the lines and the polygons created from them meet standards for our spatial data. I can now introduce the features into the production enterprise geodatabase.

While a script tool or model may take time to set up initially, in the long run, it is quicker to automate data compilation tasks through geoprocessing whenever possible. I can run a tool as needed and re-run it later with different parameters and tolerances or apply it to other datasets. Scripts are particularly useful because they can be run at specified times as a scheduled task in Windows. For example, I could combine the tools presented in these blog entries into a Python script that imports a dataset into a geodatabase, processes it, and implements topology. If I set the script to run automatically in the evening after working hours, I am ready to start editing on clean data when I come into the office the next morning.

For more information:
The sample tools and data can be downloaded from the Editing Labs group on ArcGIS.com. An ArcInfo license in required to run the tools.

Content provided by Rhonda (Editing Team)

Posted in Analysis & Geoprocessing, Editing, Python | Tagged , , , , , , , , , , | Comments Off