Monthly Archives: November 2011
By Jim Herries, Cartographic Product Engineer
“I’m a VERY novice GISer. I work for a fire department. My Chief wants me to be able to add demographic layers to our district map, but when I look around, the maps are all “Google Earth” types or PDF files, which I can’t do anything with. Where and how do I get demographic data I can add to my map as a layer?”
by Sooria Jeyaraman
I discussed accessing and adding of ArcGIS Online maps in Business Analyst Online in my previous blog. This was very useful in adding maps from ArcGIS Online and for further analysis in Business Analyst Online. As always, we want to take it a step further and make it even more useful. How do we do it? Continue reading
Geoprocessing can be used to automate many aspects of data compilation, including implementing a geodatabase topology. In my previous post, I created a model that imports data into a file geodatabase and performs initial data cleanup processing. Now, I will use geoprocessing with Python scripting to build a topology on the imported data. Although many spatial integrity issues were resolved by running the model, a geodatabase topology can help check for and repair any remaining errors.
The dataset I am using is a feature class of parcel lot line boundaries that began as a CAD file full of topological inconsistencies, such as lines that overlapped or did not connect to other lines. Once I create the topology and am sure the lines are correct, I will build parcel polygons from the lines and introduce all the features into my production enterprise geodatabase.
Creating a topology in a script
To build topology in an automated manner, I am going to write a simple Python script, add it to a toolbox, and run it as a regular geoprocessing tool. Essentially, the script just performs the functions of the New Topology wizard, but without requiring my intervention. In fact, this script could be developed without any programming by creating a model to perform these tasks and exporting it from ModelBuilder as a Python script.
When scripting, the ArcPy site package allows Python to access and run any of the geoprocessing tools in the ArcGIS system toolboxes, including the topology tools. The Topology toolset in the Data Management toolbox contains all the tools I need to add a geodatabase topology to the line feature class. The import arcpy statement adds ArcPy to a script and is at the top of every ArcGIS Python script.
Before I start adding the tools to the script, I define variables for the paths to the feature dataset, feature class, and topology. Because I created folders to store my tools and data following the recommendations in A structure for sharing tools, I can make the script more portable by setting these paths in relation to the folder containing the script. Since the folder locations are not hard-coded to match the C: absolute path of my hard drive, the script should run on someone else’s machine regardless of the machine’s directory structure.
With the paths defined, I first use the Create Topology tool to add a topology to the feature dataset. The syntax for the tool in my script is arcpy.CreateTopology_management(featureDataset, topologyName, “”), where CreateTopology is the name of the Create Topology tool, _management is the toolbox in which it resides, featureDataset is a variable I defined earlier representing the path to the feature dataset, and topologyName is a variable for the name of the topology. When working with topology, it is recommended to use the default cluster tolerance, which is the distance in which vertices are determined to be coincident. Since I want ArcGIS to calculate the cluster tolerance, I left the value blank as “”instead of supplying one.
Although that function creates a topology, it is currently empty and has no classes or rules in it. I can use the Add Feature Class To Topology tool to add my line feature class with the statement: arcpy.AddFeatureClassToTopology_management(topology, featureClass, “1″, “1″). If the topology contained multiple feature classes, I can set ranks so the feature class with the highest accuracy is not adjusted to match vertices in a feature class which is known to be less accurate. However, since I am using only one feature class, I leave the rank parameters as “1″.
Next, I set which topology rules to include by calling the Add Rule To Topology tool. When choosing which rules to add to a topology, there are a few rules that many editors commonly add to every topology, such as the line rule for Must Not Have Dangles. In the script, this rule is coded as arcpy.AddRuleToTopology_management(topology, “Must Not Have Dangles (Line)”, featureClass, “”, “”, “”). Because the rule only applies to one feature class that does not have subtypes and is not a rule between two feature classes or subtypes, the other parameters are left blank. I also want to make sure that the lines do not overlap or intersect themselves, so I include the Must Not Overlap and Must Not Self-Intersect rules as well. If I need to add more rules later, I can run the Add Rule to Topology tool or use the topology’s Properties dialog box in the Catalog window.
Once all the feature classes and rules have been added, my script validates the topology. The Validate Topology tool identifies features that share geometry, inserts common vertices into features that share geometry, then performs integrity checks to identify any violations of the rules that I defined for the topology. Since the topology has never been validated before, I am going to validate the entire extent of the data with the syntax, arcpy.ValidateTopology_management(topology, “Full_Extent”). However, when working in ArcMap, validating the visible extent of the map instead of the full extent limits the area to be validated and can be useful for very large datasets that take a long time to validate.
Finding and fixing topology errors
After the script runs, I add to ArcMap the resulting topology so I can inspect the results and fix any errors using the ArcMap editing tools. However, with all the previous automated QA work from my Import and Clean Lines model, the remaining manual edits are minimal in comparison to what they could have been without running it first.
The errors identified with topology are indicated by the orange-colored squares. Almost all of these are dangle errors that could not be fixed by the original model since they exceeded the tolerance value for the Extend, Trim, or Snap tools. The topology found only two Must Not Overlap errors, which I can fix by deleting one of the overlapping features. There are no violations of the Must Not Self-Intersect rule, indicating that the lines were split properly in the model. Using the editing tools on the Topology toolbar, such as the Error Inspector and Fix Topology Error tool, I can review each error to determine if the built-in topology fixes can be used or if the lines should be edited manually to resolve the topology error.
In some cases, the topology error may need to be marked as an exception, which is a valid violation of a topology rule. One of the most common examples of exceptions to the Must Not Have Dangles rule is a cul-de-sac road, which are dead ends that do not connect to other roads. However, when working with parcel lot lines, there are fewer scenarios that are valid exceptions. I do have some lines at the edges of the dataset that do not connect to other lines. I can either mark these as exceptions or choose to delete the features, depending on whether these features are supposed to connect to existing features in my enterprise database.
If an edit is made to correct a topology error, I have to validate the topology again to make sure the error no longer exists. After I perform a visual inspection and fix all the remaining topology errors, I can create new polygons representing landownership parcels using the geometry of the lines. If I attempted to create polygons from lines that do not connect to each other properly, either no polygons would be created or one large polygon would result where there should actually be two polygons. After creating the new polygons, I add the polygon feature class to the topology and check for any gaps or overlaps and make sure the parcel polygon boundaries are always coincident with the lot lines.
After implementing the topology and making edits in ArcMap, the lines and the polygons created from them meet standards for our spatial data. I can now introduce the features into the production enterprise geodatabase.
While a script tool or model may take time to set up initially, in the long run, it is quicker to automate data compilation tasks through geoprocessing whenever possible. I can run a tool as needed and re-run it later with different parameters and tolerances or apply it to other datasets. Scripts are particularly useful because they can be run at specified times as a scheduled task in Windows. For example, I could combine the tools presented in these blog entries into a Python script that imports a dataset into a geodatabase, processes it, and implements topology. If I set the script to run automatically in the evening after working hours, I am ready to start editing on clean data when I come into the office the next morning.
For more information:
The sample tools and data can be downloaded from the Editing Labs group on ArcGIS.com. An ArcInfo license in required to run the tools.
Content provided by Rhonda (Editing Team)
by Sooria Jeyaraman
I discussed accessing and adding of ArcGIS online maps in Community Analyst in my previous blog. This was very useful in adding maps from ArcGIS Online and for further analysis in Community Analyst. As always, we want to take it a step further and make it even more useful. How do we do it? Continue reading
A few weeks ago we posted a blog regarding how to fix your mosaic dataset when the paths to the source data are broken. Like many other tasks in ArcGIS, there is a geoprocessing method of performing this task.
Find the broken paths
The first step is to find the broken paths.
1) Open up the Export Raster Catalog Paths geoprocessing tool
2) Input Data = your mosaic dataset
3) Export Mode = BROKEN
4) Output Table = a new table (either to a geodatabase or to a .DBF file)
5) Click OK to run the tool
Review the broken paths
1) Add the table into ArcMap
2) In the Table of Contents, right-click on the table and choose Open
3) Review the table to see how many different paths need to be fixed.
Remember, only the parent folder holding the images need to be fixed, not each image.
4) Right-click the first broken path, and choose Copy
This will actually copy the entire path to the image, but we can delete the file name in the next section.
5) You can either Hide or Close the table so you can see the rest of your ArcMap window.
Fix the paths
Now that we know the paths that are broken, we need to fix the paths.
1) Open up the Repair Raster Catalog Paths geoprocessing tool
2) Input Data: choose your mosaic dataset
3) Repair Mode = FIX
4) Original Path = paste your path, but remove the file name and last
You will get a warning that the path does not exist – this is normal
5) New Path = browse or type in the new path
6) Click OK to run the tool
You have now repaired the first broken path. If all your images are in the same folder, then you will have hopefully fixed all your paths. You may need to run through all the steps again until all your paths are fixed. All your paths are fixed when the output table from the Export Raster Catalog Paths tool no longer has any records.
A lot of us have been using the Business Analyst (BA) line of products in a variety of different ways. Business Analyst gives us access to business, consumer and demographic datasets along with capabilities to allow us to analyze our various markets, customers or competition. These interactions have usually been through focused applications:
- BA for Desktop
- BA Online
- Web applications utilizing the BA Online APIs
- BA Server
- Community Analyst
These are typically all forms of client side driven analysis where we ask a question of the system and we get an answer back immediately. In some cases though we need to use the information & analysis that BA can provide as part of a larger business process e.g. tax reporting, outage management reporting…
Such business processes are likely to be running as part of a server side process, wont necessarily be initiated through any client side interactions, and may even take many hours to complete. The processes may just happen as a result of new data coming into the system or at a given time of the day/month/year e.g. a new network outage area, a quarterly reporting cycle is initiated.
Groundwater simulation models are mathematical models that help us understand and predict how water behaves in subsurface aquifers. Groundwater models, such as MODFLOW, are commonly used in the industry to predict different aspects of groundwater management including water availability, effects of pumping, contaminant transport, and more.
One of the challenges facing organizations is the communication of information regarding groundwater systems from professional modelers to others within the organization or to the public. Usually, visualization of groundwater models is achieved with specialized software packages. The model input and output files are a set of text or binary files that are difficult to interpret and map without specialized tools.
The workflow presented enables organizations to quickly create maps and GIS layers of MODFLOW models and to publish those on ArcGIS Online. By posting on ArcGIS Online you can share MODFLOW inputs and outputs with anyone within your organization, or share the model results with the public.
Step 1 – Use MODFLOW Analyst tools to import your model into a geodatabase and create maps of your model inputs and outputs
MODFLOW Analyst tools were used to import a MODFLOW model of the Cache valley in northern Utah into a geodatabase. After importing the model a set of map layers were prepared representing model inputs and outputs such as simulated heads, recharge, bottom elevation of layers, and starting heads.
Map of model inputs and outputs from a MODFLOW model of the Cache valley in northern Utah. The model was imported into ArcGIS using the MODFLOW Analyst import tools.
Step 2 – Create map and layer packages and share them on ArcGIS Online
In ArcMap, Map and Layer Packages were prepared. The Map Package enables the whole map content to be packed as a single file and the Layer Package enables exporting a set of selected layers as single file. The exported layers contain the data and layer symbology.
Map and Layer Packages can be automatically uploaded to your account in ArcGIS Online. The process includes entering a description of the map/layers, selecting appropriate tags, and specifying who you want to share the packages with. In this case the packages are shared with everyone.
Adding Map and Layer Packages to ArcGIS Online.
Once the packages are uploaded anyone can search for the data on ArcGIS Online and can open the packages in ArcGIS applications (ArcMap, ArcGlobe, ArcScene) or using the free ArcGIS Explorer viewer. The following figure shows the search results when searching for “MODFLOW” in ArcGIS Online.
Search results in ArcGIS Online when searching for MODFLOW.
The map and layer packages can be downloaded to your local computer or can be opened directly in ArcMap, ArcScene, and ArcGlobe.
Step 3 – Visualizing in ArcGIS Explorer
One of the interesting options is to open the packaged layers in ArcGIS Explorer. This allows anyone that doesn’t have ArcGIS for Desktop to visualize the results. Using ArcGIS Explorer increases the potential for sharing model information with a large number of users and clients that do not have professional GIS software.
Location of the Cache valley MODFLOW model shown in ArcGIS Explorer.
In addition, ArcGIS Explorer enables you to query the data, change base maps, and add notes and drawings on top of the model data. For example, one can use the Query Builder to display simulated heads for different layers of the model.
ArcGIS Explorer Query Builder is used to display simulated heads for layer 5 of the MODFLOW model.
The resulting layer from this query is a set of cells displaying only the simulated heads for layer 5. Notice, that the base map has been changed to show the world topographic map with a transparency on top of the world imagery layer. You can be creative and design your own base maps to display your data on.
View of simulated heads for layer 5 in the Cache MODFLOW model. The background base map was customized to show the world topographic map with transparency on top of the world imagery layer.
If you are interested in inputs/results at specific locations you can zoom and select individual cells and view their values. The following figure shows the bottom elevation value at a selected cell in layer 5 of the model.
Bottom elevation value at a selected cell in layer 5 of the MODFLOW model.
One of the interesting features of ArcGIS explorer is the ability to create presentations. With this feature you can capture different snapshots of your maps and add text. It is an easy way to communicate your model results. To access the Cache model presentation see the details at the end of the blog.
Presentation created in ArcGIS Explorer to show different inputs/outputs of the Cache valley MODFLOW model.
Finally, the ArcGIS Explorer project can be posted on ArcGIS Online. By simply selecting the Share command you have the option to share the map via email or to upload it to your ArcGIS Online account. Once uploaded, the ArcGIS Explorer project can be downloaded and opened by anyone with access to ArcGIS Online.
Sharing ArcGIS Explorer maps. You can automatically load the ArcGIS Explorer project t
o ArcGIS Online for sharing.
The workflow presented is an easy and cost effective way to publish and share information from groundwater simulation models with a wide range of users. Map and Layer Packages can be posted on ArcGIS online and these can be viewed in any of the ArcGIS for Desktop applications. Another option is to view the data in ArcGIS Explorer, so even users and clients that don’t have the professional GIS software can view and explore your models. With ArcGIS Explorer you also have the capability to create presentations that highlight different views of the model.
To view model described in this post you can find the files on ArcGIS Online by searching for “MODFLOW”. Or just follow this link:
To view the model in ArcMap, just select to open the Map Package or Layer Package using ArcGIS Desktop.
To view the ArcGIS Explorer project:
- Make sure to first download the Layers Package (.lpk file) and the Explorer Project (.nmf file) to the same folder on your desktop and only then open the project.
- You also need to have ArcGIS Explorer build 1750 or later.Thanks to Gil Strassberg for providing this post.
Thanks to Gil Strassberg for providing the post. For more information contact Gil (email@example.com). For more information on the Arc Hydro Groundwater tools visit the Aquaveo website (www.aquaveo.com/archydro-groundwater).
By Aileen Buckley, Mapping Center Lead
Variable-depth masking is an ArcMap drawing technique (available at the ArcInfo level) for hiding parts of one or more layers. One common use for masking is to clarify the legibility of a map that’s packed with text and features.
In the contour map example below, contour lines and annotation features for the contour labels overlap each other making it difficult to read the contour values.