Tips for hillshade data management

By Charlie Frye, Esri Chief Cartographer

Hillshade Thumbnail

We’ve blogged about symbolizing hillshades (rasters that are derived from elevation raster datasets, like DEMs, via the Hillshade tool), but never really covered the basics of the data used to create hillshades, so we wanted to take a minute and share a few best practices we’ve been adopting.

Before getting started, though, it’s worth noting that we’ve been storing our rasters in file geodatabases. For us, these included some rather large hillshade datasets, ranging between 5Gb and 60Gb.

Continue reading

Posted in Mapping | Tagged , | 3 Comments

PolylineZ and floating layers

Recently we had a user report that they had trouble with a shapefile they had added disappearing below the imagery layer in the default globe. We took a closer look, and discovered that the shapefile had 3D geometries, captured at sub-foot resolution, and that’s why the data seemed to disappear (though it really didn’t disappear). Here’s an overview of 3D geometries and how they’re handled in ArcGIS Explorer.

In ArcGIS a shape can be defined as various types of features – points, lines, and polygons are some of the feature types that we work with. Internally lines will be stored as a type called polyline.

Posted in Uncategorized | Leave a comment

Building and Maintaining Water Utility Geodatabases – Part 3

 Copy and Paste

You probably want to bring in some of your datasets and merge them with the projected schema. Copy and paste is the easiest way to do this, but you may get some error messages about differences in the source/target spatial reference. For example, when you try to copy and paste in ArcCatalog you may get messages like this:

This is due to some subtle differences in the properties of the target feature dataset/spatial reference and should not be a concern. The best way to handle this is to use the Import Feature Class tool in ArcCatalog:

You can also use the equivalent Geoprocessing tool: “Feature Class To Feature Class”.

 

Geoprocessing and Data Interoperability

In ArcGIS, Geoprocessing (GP) tools and scripts can be used to do the data manipulation and loading tasks. At a high level, the process involves using GP to massage the source data until it matches the data model of the target geodatabase, and then using commands like Append to get the features into your target geodatabase. This works well for most data loading situations, especially if you using Python or other scripting tools for automation. Once you figure out the pattern you can copy/paste scripts and blocks of code and it is generally easier to manage than ModelBuilder Models for data loading.

Example:

Another option is the ArcGIS Data Interoperability Extension. This extension provides a visual workbench to connect source and target datasets, and has a useful set of tools called “transformers” that can be used to perform calculations between source and target (for example, LifecycleStatus should be a new field called ACTIVEFLAG and LifecycleStatus=”Active” should be ACTIVEFLAG=”1″). This approach is preferred by many specialist users but does have an associated cost and learning curve.

Examples:

Simple Example for wFitting, similar to the Python script for wCasing.

 

Portion of more complex Spatial ETL tool example for wMain

 

Sample Tools

Attached to this post are sample tools used to Load Data, Creating Reporting Layers and Create HTML Inventory Reports from your Geodatabase.  These tools are a working example of how to use Geoprocessing tools to build a water utility database. These tools can be used to build part of a geodatabase that matches the Fort Pierce template data model. The tools are designed to be used by GIS specialists building GIS Servers. Using this sample is easy, but implementing these tools on your project can be a large project effort. This template is designed to help you get started and to show you how we loaded the Fort Pierce data.  The basic principle is to “cook” or prepare feature classes to simplify application development and improve the performance and scalability of your applications. As an initial step, we suggest you watch the online video named How to Load Data into the Template Geodatabase and How to Build Reporting Layers found on the Water Utility Resource Center. Then, you can follow the instructions below to install and use the template on your own.

 

Posted in Uncategorized | Tagged , , , , , , , , , , , , | 3 Comments

Building and Maintaining Water Utility Geodatabases – Part 2

 

Projecting the Template Geodatabase

The Template.mdb has the spatial reference that Fort Pierce, FL uses. You will almost always need to project the template geodatabase to your target spatial reference. There are some geoprocessing tools you can use to accomplish this task. The simplest way to find those tools is to search for “project” in ArcToolbox.

 

Project and Batch Project are both useful. You can use the spatial reference of your source data to select the correct spatial reference. If you are not familiar with spatial references and are not sure what to do, spend some time researching this because it is an important decision for your system.

A Few Notes on Database Platforms

At this point it is good practice to do your data work in a local File or Personal/Workgroup Geodatabase. Often the overhead of setting up a larger database system presents more work by IT/DBA staff and it is ok to wait until you have the data loaded to take that step, especially if you will not have privileges to manage the database schema (add/remove tables for example). It is also easier to work with local databases in the early stages since you will likely end up loading the data multiple times and it is more convenient to do all of this work on the same computer.

Beyond that, data loading to a Geodatabase is the same regardless of the underlying database management system (i.e., Oracle or SQL Server).

File Geodatabases are very fast to work with, but they can be slow to copy over networks and to external devices because there are many files – Template.mdb for example would have about 480 files so we provided an empty Template.mdb instead. For your data loading we recommend that you do not use a personal geodatabase (.mdb) because the performance degrades quickly with even small amounts of data.

Posted in Uncategorized | Leave a comment

The Insane Amounts of Data in ArcGIS Business Analyst…

   by James Killick

When I joined the ArcGIS Business Analyst team about a year ago I was astounded by the depth and breadth of data that we ship with the Business Analyst products — over 11,000 variables on current year and five year forecast demographics, thousands of variables on consumer spending habits, over 12,000,000 businesses, crime data, traffic congestion data, banking data — this list just goes on and on and on.

If you want to see just how insane it gets take a look at this:

 

Households that used three or more packages of dog biscuits in the last month??  You’re kidding me?

It turns out we’re not just creating all this data for fun. It’s actually vital for learning where a business can be successful or determining ways in which a business can be made more successful. You can use it to laser focus your marketing … or to help you decide which product lines to carry in a store … or to help determine whether a particular location is still viable for business.

If you want to see more drill down to the Esri data pages on esri.com at http://www.esri.com/data/esri_data

Have fun exploring… :-)

Posted in Location Analytics | Tagged , , | Leave a comment

ArcGIS Online System Updated to Resolve Connection Issue

3/6/09—ArcGIS Online was updated Friday night, March 6, to address an issue that was reported recently when connecting to ArcGIS Online map services in web applications. Common web application errors encountered were:

  • “An error occurred while processing your request.” or
  • “Client found response content type of ‘text/html’, but expected ‘text/xml’.”

If you experience any ongoing service availability or performance issues related to this change, please post a description of them to this blog or contact ESRI Support.

Posted in ArcGIS Online, Services | Tagged | Comments Off

Tips and tricks – accessing feature shape in Calculate Field

With the Calculate Field tool, you can easily create expressions that use some property of a feature’s shape, such as length or area. You can also convert length and area to different units. The illustration below shows calculating the MILES field to the length (in miles) of each line feature.
Continue reading

Posted in Analysis & Geoprocessing | Tagged , , , , , | Leave a comment

The Geodatabase Team at the 2009 ESRI Developer Summit

We’re just two weeks away from the 2009 ESRI Developer Summit in Palm Springs. A lot of geodatabase team members are going to be at the Wyndham to give technical sessions. We’ll also be on hand in the showcase area to answer any questions you may have and talk about the things you’re working on.

The best time to get any questions addressed and mingle with geodatabase team members will be at the ‘Meet the Development Team’ session help on Wednesday from 10:30 – 11:00.

Here is a list of presentations our team will be giving at the conference:

Demo Theatre

Geometric Networks for Developers
Alan Hatakeyama, Craig Gillgrass
Wednesday – 12:00pm, Oasis 1 (PSCC)

Presummit Seminar

Developer’s Guide to the Geodatabase
Craig Gillgrass, Colin Zwicker, James MacKay
Monday – 1:00pm – 4:00pm, Catalina/Madera (Wyndham)

Technical Sessions

Effective Geodatabase Programming
Erik Hoel, Brent Pierce
Tuesday – 1:00pm – 2:15pm, Pasadena/Ventura/Sierra (Wyndham)

Developing with ArcGIS Raster APIs
Hong XU, Peng Gao, Robert Berger
Wednesday – 1:00pm – 2:15pm, Smoketree A – E (PSCC)

Distributed Geodatabase Development
Gary MacDougal, Khaled Hassen
Wednesday – 2:45pm – 4:00pm, Smoketree A – E (PSCC)

Implementing Enterprise Applications with the Geodatabase
Brent Pierce, Tom Brown, Forrest Jones
Wednesday – 4:30pm – 5:45pm, Smoketree A – E (PSCC)

Working Effectively with the Geodatabase Using SQL
Tom Brown, Kevin Watt
Thursday – 8:30am – 9:45am, Primrose C/D (PSCC)

 

Posted in Geodata | Tagged , | Leave a comment

Quick tint bands

By Aileen Buckley, Mapping Center Lead

Quick Tint Bands

Have you wondered how you can get a graded boundary effect for your polygons layer, which is a nice way to add a subtle emphasis to boundaries? This effect uses what are called “tint bands”.  We posted a blog entry on this a while back. If you are in a rush and want to get a similar effect, you can create what we call “quick tint bands”. It’s pretty easy, really – you just need two layers – one for the fill, one for Continue reading

Posted in Mapping | Tagged , , | 3 Comments

Building and Maintaining Water Utility Geodatabases – Part 1


Part 1: Explanation of the Sample and Template GDB


The Water Utility Templates provide sample datasets and also a template Geodatabase for your use. There are 4 main parts to the Template Geodatabase:

    ReferenceData – Landbase data typically acquired from other organizations

  • WaterDistribution – Water Utility Network Assets
  • PlanningAndOperations – Long Range Planning and Utility Administrative/Engineering Area Boundaries
  • Field Operations – Feature classes include redlines/markups and workorders assigned to field crews
    We would like to once again thank the Fort Pierce Utilities Authority and St Lucie County, Florida for allowing us to include a sample of their data in the templates. You should be aware that we took a copy of the data at a point in time and built additional content on top of those datasets. As a result, the data in the templates has been significantly altered from the original database and also does not reflect real conditions in Fort Pierce Florida. That will be obvious to many readers of this blog, but we want to clarify that for people who are just getting started.


    ReferenceData



    The Sample.gdb is quite similar to the Template.mdb, but it has more content in the ReferenceData feature dataset. The reason for the difference is that most water utilities do not control the landbase data they use, and we expect that (like Fort Pierce) most of the data loading from partners will essentially be a copy and paste into the target geodatabase. As a result, we only included a few ReferenceData feature classes in the Template.mdb, and depending on your system it is ok to remove those feature classes if you have something different to use.
    Below is a detailed explanation of each Feature Dataset in the Sample/Template geodatabase.


    WaterDistribution



    This feature dataset contains the water network feature classes. ESRI has built template water network data models for about 10 years and most water utility projects have used those designs as a starting point for implementation. This design should look familiar to most longtime users, but there are some significant design changes here that are described briefly later in this document.
    To load data into these feature classes, you should first drop the geometric network. Data loading tools will run faster without the network, and you will also be able to use all of the data loading tools available in ArcGIS. Once you have all of the data loaded you can rebuild the network using the wizard in ArcCatalog.
    The loading process is a bit different than the ReferenceData because you have a target database design to load into. Of course it is ok to use a different design or to use the design you already have, but that will mean more work if you want to use the map documents provided with the templates. You might also want to use this latest design if you have existing data because it improves on previous examples.
    Generally speaking you will need to do more than copy/paste your water network features into a target geodatabase. You will likely need to calculate new values, combine or split datasets into new feature classes, and you should also have a plan for doing QA/QC on the results of the data loading process. This can be a time consuming exercise that might be more practical to contract out to an organization that specializes in this type of work, and even for simple examples many weeks of work can be involved in getting existing data loaded and validated.


    PlanningAndOperations



    This feature dataset contains administrative and planning content for water utilities. The reason for putting these features in their own feature dataset is that they are edited on a different cycle than the asset data and they may have different permissions/editors.
    The simple part of this feature dataset is the feature classes for EngineeringGrid, Map Sheet boundaries, and other administrative areas for water utilities. This part of the database also contains planned replacement mains – wReplacementProject (Capital Improvement Projects) and also proposed mains – wProposedMain (New Development).
    This feature dataset also contains content that will be new to most users. What we built in the Fort Pierce database was a set of reporting layers that allow us to organize and view asset inventory, consumption, asset condition, and other data in the context of operational management units like map sheets, districts, and political jurisdictions.



    These feature classes were created by spatially joining/intersecting the water network features to sets of polygons like the EngineeringGrid example shown above, and then further calculations for leaks/mile, consumption, and inventory information were performed.
    There are 2 main benefits we see from this work:

  • 1. The ability to provide more granular reporting. For example, consumption and number of valves by map sheet or smaller area rather than company-wide reports.
    2. Applications such as the Operations Dashboard require that we prepare data so that rather than users running general queries on the data they can just click a few times and get the information they are looking for. As a general principle, we want to “cook” the answers to the common questions to improve performance and reduce load on web servers.


     
    FieldOperations



    Last but not least the FieldOperations feature dataset contains a number of feature classes that are used in the Mobile and Dashboard Water Utility Templates. These include leaks, redlines, and workorders. While the Mobile Map example only uses the redlines capability in the database, the design includes the ability to manage workorders and inspection information to/from the field as well.
    The one thing people notice right away is that workorders and inspections in this data model are Point feature classes. Yes, we know that most workorder/CMMS systems do not store point features because they are tabular systems, but part of simplifying the data/application for field users is finding the geographic location for work. In addition, ArcGIS Mobile works with simple feature classes so this approach works well from an implementation standpoint.
    The content for this feature dataset will be more dynamic, and will likely only contain assigned/current work that needs to be pushed to/from the field. For example, today’s work can be loaded into the mobile feature classes and updates can be pushed to the field. Similarly, as work happens in the field, data can be pushed back into the office through a mobile data service.
    Ultimately this approach requires some back-end work to get the data from one or more enterprise systems to the field application, and then from the field application back to the enterprise systems. This is not particularly difficult work, but the data loading and management activities will be different than the other feature datasets in the template geodatabase.


    Data Model Notes


    The data model available with this template has some changes from those data models that reflect the evolution of implementation models for geodatabases. These changes include:

    • Uppercase, short field names to ensure that field names are not altered/truncated when moved between different database platforms (and yes shapefiles are still being used for data exchange)
    • Longer, more meaningful alias names on fields
    • Descriptions on fields and feature classes (FGDC stored as FGDC metadata)

    Example:

    Feature class aliases are plural but table names are singular. When you add a feature class to ArcMap you will get a layer with the plural name (the convention GIS users prefer), but it also works for database people that insist on singular names for tables.

    Example:

     

    • LifecycleStatus field replaced by ACTIVEFLAG. Previous designs had proposed, active and abandoned features in the same network feature classes. Over time most utilities have separated proposed and abandoned features into different feature classes so they are not accidentally included in network traces or asset inventory reports. Once we added those feature classes to this design, Lifecycle status just became a flag to indicate if the features are active or not. For example, there are temporary/seasonal services that are only used for part of the year, and there are other assets in the ground that are not active because they are still under construction or are temporarily inactive.
    • OWNEDBY/MANAGEDBY fields indicate the owner of each asset and also the maintenance responsibility. While most domains/list of permitted values in this data model are strings, these domains are integers. The plan here is that “Company owned” assets that should be counted for inventory purposes have a value > 0 and things that should not be counted have a value < 0. In many cases there are multiple companies involved in county/regional scale systems and this strategy will make asset reporting simpler. Of course in those situations people should extend the list/domain of values for their specific situation.

      Example:

       

      • You may notice that we also dropped AdministrativeArea and OperatingArea fields because these are better managed as part of the reporting layers we built for the PlanningAndOperations data – which is described in the next section of this document.
        More Information


        In each of the Templates, you will see the sample geodatabase and the template geodatabase in the “Maps and GDBs” folder. If you look in the subfolder “Documentation” there are html documents that provide reports for the geodatabase and map documents.
        These documents will help you to understand the details of the geodatabases and maps, and also help you to plan out how to load data into your Geodatabase and make changes to map documents to work with your data.
        Most projects will start with a source-target matrix spreadsheet that describes the available data and the target datasets for their new Geodatabase. This is a good place to start and it will help you to assess the suitability of the template design as well as the level of effort required to build your Geodatabase.

        Again, this can be a large part of your project, so keep in mind that ESRI and Business Partners are here to help you if you need us. Network with your peers to get their recommendations on who to work with, or email us at ArcGISTeamWater@esri.com and we can get you in touch with someone local to help you to get started.
         


         

    Posted in Water Utilities | Tagged , , , , | Leave a comment