It’s dev time of year again!
Members of the geodatabase team are carpooling down to Palm Springs with the beaksnake this week for the 2012 Esri Develper Summit.
We’re set up in the Esri Showcase area of the Palm Springs Convention Center, so swing by and meet the team, ask some questions, and find the right developer to talk to about your projects.
Craig Gillgrass will be ‘handing’ out complimentary high fives too, so be on the look out.
Esri Showcase hours:
Monday 11:00 am – 7:30 pm
Tuesday 12:30 pm – 6:00 pm
Wednesday 10:00 am – 6:00 pm
Meet The Team
Be sure to swing poolside on Tuesday between 6 – 9pm to chat with the geodatabase team at our Meet the Team session. Continue reading
(Update: You can find the most recent release of the File Gdb API right HERE)
The updated File Geodatabase API version 1.2 is HERE!
There is now support for Intel-based Mac!
The minimum supported OS is Snow Leopard 10.6.
The supported compiler is gcc 4.2.1.
What else is new at version 1.2:
- Structure definitions for curves (CircularArcCurve, BezierCurve, EllipticArcCurve)
- Macros for error values allowing for more readable code. (NIM077630)
- HRESULT IFDEFs have been updated to define if they are not set. (NIM071619)
- File Geodatabase API function parameters are const ref’s where possible. (NIM071620)
- Coordinate systems supported are updated to match ArcGIS 10.1. (NIM078034)
Previous blog posts have introduced you to data driven pages and product library but referred to them as separate, standalone tools. In this blog I’d like to show you how to use product library to manage your data driven pages. Now you might be asking why I would want to store my data driven pages in product library. Remember, product library helps you enforce and standardize your map production processes by centrally managing all of your production related information such as business rules, documents, workflows, and spatial information. By storing your data driven pages in product library you can take advantage of product library’s capabilities such as search, history tracking, check-in and check-out capabilities, permissions, and so forth.
In the following steps I’ll walk you through a basic workflow for importing and managing a US State map book in product library.
1. In ArcCatalog, create a new file geodatabase.
2. In ArcMap, add the Production Cartography toolbar and click the Product Library window button.
Some folks have reported not being able to load a file geodatabase into CityEngine. This can be related to the version of the Microsoft Visual C++ redistributables on the machine.
The latest redistributables can be downloaded here:
The upcoming CityEngine service release in January 2012 will automatically installed the redistributables.
In part 1 of this blog series I wrote about interactive geoprocessing (GP) steps in ArcGIS Workflow Manager. Now, in part 2, we’ll move onto automated GP steps and wrap up the discussion from there.
At the end of part 1, you’ll recall we had pre-populated all of the required parameters for a GP tool.
And I was wondering why, with all of a tool’s arguments pre-populated, do we still need to show a dialog box to the user?
Right. And the answer is that we don’t. If all of the parameters have been correctly pre-populated, there are a number of alternate ways that you can run a GP step without prompting a user for anything. These include:
In my previous blog I introduced you to the different types of custom steps available in ArcGIS Workflow Manager including those that make use of Geoprocessing (GP) tools. Today, we’ll explore using GP steps in a Workflow Manager workflow.
Okay. How do I get started?
Geoprocessing steps use the same tools, models, and scripts that you can access through an ArcGIS toolbox. The easiest way to set up a GP step is with the out-of-the-box JTXDesktopSteps.LaunchGPTool custom step. Refer to the online help for more information about how to create step types.
Editing and data compilation are less commonly thought of as operations that can be automated through geoprocessing. However, ArcGIS 10 introduced the Editing toolbox, which contains a set of geoprocessing tools to perform bulk edits. These tools combined with others in the geoprocessing environment can automate data import and maintenance work. Automated data compilation tools are especially useful for importing data into a geodatabase but can also be employed on a regular schedule to perform routine quality assurance (QA) checks. In this entry, I will discuss the use of geoprocessing to clean CAD line data as part of the import process.
Importing data with geoprocessing
Lines that are created without the use of spatial integrity measures, such as snapping or topology, almost always contain some inconsistencies. These errors are likely in data that originated in formats such as CAD, shapefile, or KML. Fortunately, many common topological issues can be resolved in an automated manner by using ModelBuilder to link together tools that will import data into a geodatabase and perform standard data cleanup techniques.
I have a CAD file for a new subdivision that needs to be integrated with my existing GIS parcel data. The GIS data must be kept to stringent accuracy standards, so I need to fix any issues where lines do not connect to each other, overlap, or are duplicated. Rather than risk reducing the quality of the main parcels geodatabase, I can create a local temporary geodatabase where I can preprocess the CAD lines before introducing the features into the production geodatabase. Although the CAD file contains buildings, roads, text, registration tic marks, and other features, I plan to use only the parcel lot lines.
I have built a model that imports the CAD lines into a temporary scratch workspace, cleans and processes the lines, and then copies the corrected lines into an output file geodatabase. When importing CAD data into a geodatabase, I can choose from several available tools, including CAD to Geodatabase or Feature Class to Feature Class. The CAD to Geodatabase tool converts all the geometries in a drawing to individual feature classes, such as a line feature class for the parcel lines, annotation feature class for CAD text, and so on. In my case, I am using Feature Class to Feature Class tool because I need only the lot line geometry from the CAD file. This tool makes the model reusable because it can import many different formats and not simply CAD. In addition, the Feature Class to Feature Class tool allows for an SQL expression so I can further refine the import to include only the CAD features that satisfy an attribute query for lot lines (in this case, “Layer” = ‘LOT-L’).
Performing automated quality assurance on lines
Once the CAD parcel lot lines are imported into a geodatabase feature class, I can begin running tools to perform automated QA processes. Many tools are found in the Editing toolbox, although other toolboxes can be purposed for data compilation QA tasks. For example, I can start by using the Integrate tool in the Data Management toolbox to address minor inconsistencies in the line work. Integrate makes features coincident if they fall within the specified x,y tolerance. By using a small tolerance on Integrate (and other similar tools), I can avoid editing the data beyond the level of cleanup I intended. In addition, since I am running the tools on a copy of the data outside my production database, I can run the tools repeatedly to refine tolerance values to fix more issues in an automated manner. The intermediate data created as the model runs is maintained and can be reviewed in the scratch geodatabase.
After the dataset is integrated, I check for duplicated lines with the Delete Identical tool (Data Management toolbox). The dashed lines connecting to this tool represent preconditions, which are used to control the order of operations in a model. For example, the Integrated Lines output is a precondition to the Delete Identical tool. This way, the Delete Identical tool will not execute until the lines have been integrated.
The next part of the model identifies lines that are dangles. With the Feature Vertices to Points tool in the Data Management toolbox, I create a new point feature class containing the line endpoints that are not connected to any other lines. I can then use Select Layer By Location to identify the lines that intersect these dangling endpoints. The resulting selection represents lines with dangles.
Many of these dangle errors can be fixed by running the Editing toolbox’s Trim Line, Extend Line, and Snap tools. Effective use of the Editing toolbox geoprocessing tools can improve productivity because the tools apply edits in bulk, such as to all features or all selected features. In most cases, the similar editing function applies to only one feature at a time. Because I exposed the tolerances as variables and model parameters, I can easily run the model with different values because the tolerance settings appear as input boxes on the tool’s dialog box. For example, I am willing to extend or trim the lines from this CAD dataset initially up to a maximum length of five feet. After that procedure, I want to inspect the lines visually to see how many issues remain to ensure that I will not be making incorrect edits if I increase the tolerance value. I can change the tolerance as needed depending on the accuracy of the lines I am importing.
In addition, since my organization’s spatial integrity rules indicate the parcel lines should be split and not intersect themselves, I can use a sequence of spatial and attribute queries to find the locations where lines have intersecting endpoints. Lines are often split so that each length can be attributed separately.
Once these processes have run, the lines are output into a feature dataset in a geodatabase and are much cleaner topologically. After the model completes, I can run the Feature Vertices to Points tool again on the cleaned output to see the remaining dangles and compare the current number of endpoints that are dangles (the yellow circles in the graphic) to the number in the original CAD lines (the red circles). While there may be a few remaining issues, there are less than before running the model. At this point, I can build a geodatabase topology to check for and repair any other errors. When I am satisfied that the lines meet the standards for our spatial data, I can import it into the production database.
For more information:
The sample tools and data can be downloaded from the Editing Labs group on ArcGIS.com. An ArcInfo license in required to run the tools.
Why should a water, wastewater or stormwater utility adopt the Local Government Information Model?
One of the biggest benefits of a water utility adopting the Local Government Information Model is that it makes deploying the ArcGIS for Water Utilities maps and apps easier, faster and cheaper. The further you deviate from the Local Government Information Model, and in particular it’s geodatabase schema, the harder it will be for you to implement the maps and apps that are part of ArcGIS for Water Utilities. It will also be hard and time consuming to upgrade your ArcGIS for Water Utilities implementation when we release updates.
Changes you make to the Local Government Information Model schema may necessitate extensive modifications of the maps documents, and changes to apps (web apps, mobile apps, ArcGIS Desktop, etc.) that are part of ArcGIS for Water Utilities. So the closer you stay to the core Local Government Information Model, the easier your initial deployment will be and the easier it will be to migrate your ArcGIS implementation to new releases or to deploy updates to the maps and apps.
It’s also important to note that when we say “adopt” the Local Government Information Model we don’t mean that you necessarily have to use it as is (or more appropriately – as downloaded). You probably will need to configure the Local Government Information to meet the needs of your organization. But the key thing to keep in mind is you should only be making changes to accommodate the true organizational needs of your utility. For example, instead of changing the field names to the field names you’d like to use in your organization, modify field and map layer aliases. Bottom line, don’t reinvent the wheel, just make changes that are required to meet specific business needs in your organization.
At the very least you need to change the projection to the appropriate coordinate system and set up the domains to reflect the assets in use at your utility. Small utilities or utilities that are new to GIS may choose to take the Local Government Information Model as is, while larger utilities, mature GIS implementations, or GIS implementations that are integrated with other enterprise system will undoubtedly need to make more significant configurations or extensions to the schema to reflect their organizational needs.
Water, Sewer and Stormwater Data Modeling Best Practices
The Local Government Information Model incorporates many best practices for water utility GIS. One of the most important best practices is how to represent a water, sewer or stormwater system in GIS.
For years Esri had downloadable data models for water, wastewater and stormwater utility networks. Those data models were the first freely available water utility GIS data models. They were stewarded by Esri, but built by the user community and became the industry standard. Globally thousands of water utilities have built their GIS around Esri’s free data models.
The Local Government Informational Model is the next iteration of Esri’s water, sewer and stormwater data models. In essence we’ve modernized the data models to reflect how water utilities have been deploying GIS over the past few years and we’ve also modified the schema to fit the requirements of the ArcGIS for Water Utilities maps and apps. As water utility GIS continues to evolve Esri will regularly maintain the Local Government Information Model to keep introducing new best practices into the user community and functionality into our apps.
Comprehensive Data Model
There is no doubt Esri’s water, wastewater and stormwater data models were an incredibly valuable starting point for water utilities to get their utility networks into GIS. Since the original data models focused primarily on a data structure for the assets that comprise utility networks, we received feedback that many utilities wanted more guidance on how to model operational data (workorders, service requests, customer complaints, main breaks, capital improvement projects, etc.) and base data (roads edge of pavement, road centerlines, elevation data, parcels, etc.) in their GIS. The Local Government Data Model solves this problem because it includes a complete schema for typical water utility base data and operational data.
Over the years, an observation we’ve made is that water utilities struggle with how to model and manage schemas for datasets that aren’t their utility networks or operational data – simply put managing base data can be a challenge for water utilities. For example we’ve seen a lot of utilities struggle with managing roads, parcel, buildings, etc. in their enterprise GIS, especially when these datasets are coming from other organizations or departments.
This is a particular issue for water utilities that serve multiple units of local government such as authorities, county wide utilities, state wide utilities and private companies. A good example of this is a water authority whose service territory includes three counties. The water authority needs parcel data that is maintained by the counties. County A, County B and County C all use different schemas for their parcels. So the water utility had two choices – leave the parcels in 3 different data layers and use them as is – which makes analysis, map creation and integration with other systems at the utility that need parcel data (such as a customer information system) difficult. Or invest time to extract, transfer and load (ETL) the parcels into a common schema so they can be used as a single seamless layer across the service area. The Local Government Information Model can now serve as the common schema in this example.
Easier Data Sharing
We describe the Local Government Information as a harmonized information model – meaning designed to accommodate typical GIS needs across local government. If organizations that commonly share data all adopt the Local Government Information Model, it will greatly reduce the time and resources spent establishing a common schema and migrating data to these schemas – thus allowing water utilities to focus on the maintenance and management of their authoritative data.
For example a private water utility may serve two municipalities. If the water utility and both municipalities all adopt the Local Government Information Model then they can all very easily exchange data. When the water utility needs road centerline and edge of pavement layers from the municipalities than the utility can just import the new data without having to manipulate the schema and will have seamless layers for their service areas. The same logic applies to the water utility sharing data with the municipalities – when the water utility updates the location of their upcoming capital projects, the utility can share that data back with the municipalities and the municipalities can use it without any schema manipulation.
Best Cartographic Practices for Water Utility Maps
As we’ve discussed in a previous blog, the Local Government Information Model includes geodatabase schema, map documents and specification for services necessary to deploy the ArcGIS for Water Utilities and ArcGIS for Local Government maps and apps.
The map documents highlight
best practices for displaying water, wastewater and stormwater data in the context that each map is designed to be used. For example the map documents included with the Mobile Map Template have best practice cartography for displaying water utility GIS data in the field in both a day and night time use map. The same goes for the map document included with the Infrastructure Editing Template – this is a best practice map document for editing water utility data with ArcGIS Desktop.
Looking to the Future
The specification for the services (map, feature, geoprocessing, etc) necessary for the ArcGIS Water Utilities maps and apps are also part of the Local Government Information Model. So if other local government entities in the service area of water utility embrace the Local Government Information Model, ArcGIS for Local Government and start to publish services, then water utilities can consume those services for their maps and apps. In this scenario the water utility may no longer have to import some data into their own geodatabase and can just consume the services right from the organization that is the steward of the data.
We hope you’ve found this exploration of some of the benefits water, wastewater and stormwater utilities will experience when adopting the Local Government Information Model helpful. We encourage your feedback on the information in this blog, the Local Government Information Model or ArcGIS for Water Utilities.
The maps and apps that comprise ArcGIS for Water Utilities are built around the Local Government Information Model, so we thought it would beneficial to explain what the Local Government Information Model is in this blog and in a follow up blog explore its benefits for water, wastewater and stormwater utilities.
We’ll start by breaking down the term Local Government Information Model into two parts “Local Government” and “Information Model”.
ArcGIS for Water Utilities is a module of ArcGIS for Local Government. ArcGIS for Local Government is organized around typical services or functions of a local government – water, sewer and stormwater utilities, public safety, land records, public works, etc. From a GIS perspective, no matter what type of entity a water utility is – a municipal department, an authority, part of a county, part of national government, private or a public private partnership, the scale of the data necessary to effectively map and manage your utility is similar to the scale of GIS data used by other local government entities. Simply put, water utilities need local scale GIS data.
The feature classes and feature datasets in the Local Government Information Model are “harmonized” meaning that they are designed to work across and support typical functions of local governments without duplication and redundancies. This enables municipal departments, functions within an organization or entire organizations to manage data that is specific to their domain and utilize data from other domains within local government as base data.
If you look across a typical city enterprise GIS implementation that encompasses all of a city’s departments, what you’ll notice is that the “operational data” of one department is often “base data” for another. For example the parcels maintained by a city’s land records department are typically base data for the city’s water utility. The water utility may use the attributes of the parcels for analysis or may join a table to the parcels with water utility specific information. Another example is the GIS features of a city’s storm water system that are maintained by the water department but are used as base data for the city’s 411 system that is managed by the city’s public works department. It’s important to point out that when we say “base data” we don’t just mean the data is used as a base mapping layer, it can be used for analysis or can be extended by the utility (in the example of joining a table of utility created data to a parcel data). Data not maintained by the utility is used as base data to provide perspective but can also blur the lines and become operational data when used for analysis or joined to information maintained by the utility
The same logic applies to water utilities that are authorities or private companies. Some of the layers they use for base data typically come from other local units of government (cities, towns, counties) within their service territory.
We are using the term Information Model because this is more than just a data schema. In the GIS realm the term “data model” has commonly implied a schema or database structure only. The Local Government Information does include a schema, but we consider things like the Map Documents for our maps and apps and specifications for services to be part of the information model as well.
We are including Map Documents in the Local Government Information Model for two main reasons. First the Map Documents for our maps and apps are built upon best practices for each particular type of map. For example in the Mobile Map Template the .MXD documents have been designed to show best practices for building an interactive water utilities map for field crew use.
Secondly the maps and apps are built specifically for the geodatabase schema that is part of the Local Government Information Model. What this means is if you change the underlying schema to better reflect the true organizational needs of your utility than depending on the changes made you may have to modify any map documents that use that layer. Since you use the Map Documents to publish services to ArcGIS Server than the same logic applies for including services in the Information Model. The schema, the map documents and the services are intertwined.
(Update: You can find the most recent release of the File Gdb API right HERE)
We’ve updated the File Geodatabase API download on the resource center – version 1.1 is now available!
The main bonus to this new download is that it includes a .NET wrapper for VS2010. This allows the use of .NET enabled languages, like C# and VB.NET, to access the File Geodatabase. So, all of the functionality of the C++ API is now available to the .NET developer too. Woot!
Documentation and C# samples are included in the download.
There are also several bug fixes with this new download, including a fix to a domain bug where you couldn’t assign a domain to a field. Color it Fixed!
So rush on over to the resource center and nab the new download then hit up the File GDB API forum and collaborate.