Monthly Archives: March 2008

Geodatabase Essentials – Part I: What is the Geodatabase?

Now that the dev summit is over we can tone down the amateur journalism and get into some real topics. Seeing as how this blog is titled “Inside the Geodatabase”, I thought a good place to start would be an introductory topic on the geodatabase. So here is the first in a series of posts we’re calling “Geodatabase Essentials”. Future posts tagged with this title will contain introductory information laying the foundations for essential geodatabase topics.

What is the Geodatabase?

The geodatabase is the native data storage and data management framework for ArcGIS. Why would you want to use a geodatabase? Because it acts as an organizational tool to store and manage your data, and is also the gateway into advanced GIS capabilities.

The geodatabase is a container which houses a collection of various geographic datasets.

Geodatabases support all of the different types of data that can be used by ArcGIS. Also, there is a complete set of conversion tools available so you can easily migrate existing geospatial data into the geodatabase.

At face value the fundamental ArcGIS datasets are tables, feature classes, and rasters. These and other more complex datasets, such as topologies and geometric networks, are all contained within the geodatabase. The geodatabase can also add advanced capabilities to these datasets and model behavior. Some examples of this are:

  • Data Validation using domains and subtypes
  • Multiuser editing environment through versioning
  • Topologies to enforce the integrity of your spatial data
  • Networks to model and analyze flows
  • Terrains for modeling surfaces and Lidar data management
  • Distributed geodatabase replication
  • Managing historical archives of your data

There are 3 types of geodatabases: Personal, File, and ArcSDE.

Personal Geodatabases

Personal geodatabases were first introduced in ArcGIS 8.0 and are designed for a single user working with smaller datasets. They are stored and managed in Microsoft® Access™, which ties them to the windows platform.

One thing a lot of users like about the personal geodatabase is the ability to manage the tabular data using Access.

Access based personal geodatabases work well for small datasets and they support all the features of the geodatabase model such as topologies, raster catalogs, network datasets, address locators, etc…. They are single user and therefore do not support versioning and long transactions.

File Geodatabases

File geodatabases, introduced at ArcGIS 9.2, store datasets in a file system folder and are portable across operating systems. They are suitable for single user projects and small workgroups with one editor and multiple readers. Although they do not support versioning, it is possible to have multiple editors with a file geodatabase, providing they aren’t editing the same feature datasets, feature classes or tables.

The file geodatabase is optimized for use in ArcGIS so it provides very fast data use and storage, and can scale to over 1 terabyte in size. Also, the File geodatabase allows you to optionally compress your vector data, reducing the memory footprint used by its storage without affecting performance.

ArcSDE Geodatabases

ArcSDE geodatabases manage spatial data within an RDBMS such as DB2, Informix, Oracle, SQL Server, PostgreSQL and SQL Server Express. Through this architecture, ArcSDE offers a multi-user editing environment and can manage extremely large datasets. ArcSDE geodatabases also support version-based workflows such as geodatabase replication and archiving that are not supported with file and personal geodatabases.

Organizations requiring the full suite of geodatabase functionality and a geodatabase with the capacity for extremely large, continuous GIS datasets that can be edited and accessed by many users should use an ArcSDE geodatabase.

9.3 Beta users can find this information and more on the Geodatabase Resource Center page.

Also for beta users, a few useful topics from the help system covering this information in more detail are: An overview of the geodatabase, Essential readings about the geodatabase, and Types of Geodatabases.

Users not in the beta program can access similar topics covering this information in more detail from the 9.2 web help. See An overview of the geodatabase and Types of geodatabases.

Posted in Geodata | Tagged , | Leave a comment

Get to know your organization's developers

By Charlie Frye, Esri Chief Cartographer

Early last month Chris Sergent forwarded me an email via Mapping Center’s submit a blog entry feature.  Chris, who is a developer, was looking to connect with other Esri developers. The spirit of Chris’s email struck me as something we could all do with a little more of, so we put Chris’s post up on the Esri Developer Summit blog. Continue reading

Posted in Mapping | Tagged , , | Leave a comment

AIRS Curtain Plots Custom Task

One of the custom tasks shown during the Developer Summit plenary was authored by John Grayson, a veteran of ESRI’s Applications Prototype Laboratory. Here’s his description of the task.

This ArcGIS Explorer task has been implemented to visualize data collected by the AIRS satellite, a project of NASA’s Jet Propulsion Laboratory.  The AIRS site provides the following description of the technology:

The Atmospheric Infrared Sounder (AIRS) looks down at the Earth and measures the infrared brightness coming up through and from the atmosphere. The AIRS optical system splits infrared radiation into its constituent “colors”. The effect (but not the technique) is similar to rain drops splitting sunlight into a rainbow. The wavelengths used are sensitive to temperature and water vapor over a range of heights in the atmosphere, from the surface up into the stratosphere. The term “sounder” in the instrument’s name refers to the fact that temperature and water vapor are measured as functions of height. AIRS also measures clouds, abundances of trace components in the atmosphere including ozone, carbon monoxide, carbon dioxide, methane, and sulfur dioxide, and detects suspended dust particles.

The data was preprocessed from its original HDF format into a geodatabase where information about when the data was collected and to which swath it pertained is maintained.  Additionally, curtain plots (vertical rasters) for each of the five attributes were generated as a server-side process.  The task starts out by reading the geodatabase and enabling the user to specify which date and swatch to visualize. The screen capture below shows the custom task interface, which allows the user to load the data and choose from different types of information to display. 

 

Footprints of each swath are drawn, and below they can be seen as blue rectangles on the globe. 

 

Using the custom task UI, the user can choose from a variety of types. The task uses OpenGL to load the curtain plots for the specified type, date, and swath onto the display.  Shown below is the complete set of curtain plots showing carbon monoxide levels for one of the swaths.

A slider control allows the user to quickly visualize one curtain plot at a time, providing a simple exploratory tool to visualize the data in the display.  Shown below is one of the curtain plots, and using the task slider the user can move forwards and backwards in the curtain plot sequence.

Here is another example, this one showing ozone levels. You can compare this to the earlier screen capture above in this post showing carbon monoxide levels.

The task leverages the ArcGIS Explorer SDK and it’s support for integrating custom OpenGL implementations. The task makes use of exposed events that tell us when the application has finished drawing and when OpenGL calls can be made.  OpenGL allows the developer to create tasks that perform visualizations that are not otherwise possible via the Explorer API, and provides a wealth of potential possibilities for the Explorer developer.

Posted in Uncategorized | Tagged , , , | Leave a comment

Video: Creating a web application with ArcGIS Server Manager

This video demonstrates how to use ArcGIS Server Manager to create a web application. You’ll find it useful if you’re getting started with ArcGIS Server, or if you just want to see what ArcGIS Server can do.

If you want to follow the steps in this video, you’ll need to install ArcGIS Server, create at least one map service, and log into Manager.

What other video topics would you find interesting for the blog or for the ArcGIS Server Help? Leave us a comment with your ideas.

Posted in Services | Tagged , , , , | 3 Comments

Search Mapping Center content

By Aileen Buckley, Mapping Center Lead

We’ve added new site-wide search functionality to Mapping Center.  As our content and audience continue to grow, we will continue our efforts to make it easy to quickly find the information you need.  Continue reading

Posted in Mapping | Tagged , | Leave a comment

BPC and Dev Summit – Some Custom Tasks

Last week was a busy week for the entire Explorer team, with lots of interesting things happening and lots of great discussions in the Showcase area and elsewhere at Palm Springs. We appreciate all of your comments and feedback. We’ll be highlighting some of the presentations and those discussions in a series of posts beginning this week.

First, here’s a few things that were shown during the ESRI Business Partner Conference and Developer Summit plenaries. We’ve covered some of the general things that we typically demonstrate in previous posts on the ESRI Federal User Conference, Petroleum User Group meeting, and GITA, so we’ll focus on some of the different things that were shown last week.

One of the custom tasks demonstrated is one you’ll find on the Explorer Resource Center. Just choose File > Resource Center and choose the Tasks tab.

 

The one we chose is the Weather Finder task authored by Michael Waltuch, one of the lead designers on the ArcGIS Explorer development team. Just click to open the NMF file, which will add the task to your current session. To learn more about this task click the Description link on the Resource Center entry, or choose About Weather Finder on the task UI. You’ll learn the following:

Weather Finder locates the nearest weather station to a specified location and reports its most recent weather observations. The task is based on a reverse geocoding web service located at GeoNames.org. To learn more about GeoNames go to their web site and see the service description.

The popup window associated with the weather station includes a link to the National Weather Service’s Telecommunication Operations Center which lists more information and a 24 hour summary for the weather station.

This is a good example of a custom task which has been written using the ArcGIS Explorer SDK to work against an existing Web service – In other words it’s a mashup.

Using one of the Portland bookmarks, we right-clicked the pushpin to use the Send To capability to use this location as the starting point for the Weather Finder task. The bookmark itself is the result of a task (the Create Notes task in this case), and when you take the result of one task and send it to another we refer to that as task chaining, and you can read more about that in the Help.

Here’s the result of the Weather Finder task, the closest weather station to our Portland bookmark. In this case it’s located at the airport. If we click the result pushpin you’ll see the latest weather information in the popup, including links to the NOAA site providing more detailed information and historic data for this location.

We also used a task we obtained from the Showcase area on the Explorer Resource Center. The showcase is a place where anyone can contribute tasks, results, or other items they want to share with the Explorer community. We chose the Flickr Task authored by Rob Dunfey, a member of the ArcGIS Explorer SDK team who also authored a blog post with more details about the custom task. This task is another good example of a mashup with other existing Web-based APIs, in this case it uses the Fickr API.

Here we typed in “portland” to return the first 20 Flickr photos tagged with that keyword. Each of the results can be clicked to open a popup displaying the photo.

Another task that was shown was a custom geoprocessing task, authored using the Model Builder in ArcGIS Desktop and published using ArcGIS Server. The task’s UI was automatically generated by ArcGIS Server, based on parameters specified by the task author. The ArcGIS Explorer SDK can be used to modify its UI, but we just used the default, automatically generated one. 

Here we’ve used Send To again to use our Portland bookmark as the input location for a 1 and 3 minute drive time analysis.

In upcoming posts we’ll cover more of what was shown, including some of the technical details of the plenary and workshop presentations, and also cover some of the things we showed that are coming in future releases.

Posted in Uncategorized | Tagged , , , , , | Leave a comment

From the Dev Summit – Thursday March 20, 2008

A few sessions were offered by the geodatabase team on the final day of the Dev Summit. Developing with Rasters in ArcGIS and Distributed Geodatabase Development were given in the morning, and the second offering of Effective Geodatabase Programming was given in the afternoon.

 

The raster session, given by Hong Xu, Joe Roubal, and Peng Gao, discussed typical developing programming patterns for creating raster centric applications. The main topics were: how to create and visualize your raster data, how to create custom geodata transformations, creating custom pixel filters, and how to use the new APIs in 9.3 to work with image services and WCS services.

 

 

Here are some sample slides from the presentation: the following slide shows how developers can create custom geodata transformations based on their own image formats and plug these into ArcGIS. This way their proprietary images can be used by ArcGIS.

 

 

The talk also looked into how developers can access image services (shown below) and WCS services to get a raster from the layer and use it for spatial analysis operations.

 

 

In the distributed data session Gary MacDougal and Khaled Hassen started with an overview of distributing data techniques before presenting the key elements of geodatabase replication and the replication API.

 

 

The following slide shows some common use cases for distributing geodatabases which can be accommodated through geodatabase replication.

 

 

As a sample of some of the code shown during the session, this next slide describes how a developer can extend replica creation through custom behavior.

 

 

After the morning presentations, Jim McKinney MCed the Closing Session while everyone was finishing a sit down lunch in the Oasis room. He went over some feedback from attendees of the conference, basically describing what people thought we did well and some things that could be done better. In raise-your-hand survey fashion Jim gathered some feedback in the room about conference specifics such as: The length of the conference, the time and length of sessions, session topics, should we offer “twilight sessions” (a handful of hardcore developers who crave night-time sessions raised their hands for this), should we have sessions based on a user track, etc…

 

The overall consensus was that we struck a sweet spot this year as far as the length of the conference, staffing, and session content. This was our biggest and probably our best Developer Summit so far.

 

From our teams perspective, we gathered a lot of useful information and feedback from users and felt as though we were able to help a lot of developers find answers to the questions that they had. Our Meet the Development Team session on Wednesday was a good indication of the value of personal interaction with the developer community.

 

Thanks to everyone that attended and to everyone that came to the Geodatabase Island to talk with the development team. We hope the Developer Summit was an interesting and beneficial experience for you – It certainly was for us.

Posted in Geodata | Tagged , | Leave a comment

Python script tool for selecting polygons for maps at smaller scales

By Charlie Frye, Esri Chief Cartographer

Refresh/reload the page  (F5) to refresh the animation

Which are the 30 or 500 most important of Minnesota’s famous 10,000 lakes?  Next, think about that question from the point of view of making a zoom-able map of the U.S. and how many lakes need to be shown as you zoom in.  That question boils down to how to decide which polygons to show.  Not all polygons are created equally, some are roughly round, some are banana shaped (like ox-bow lakes) some have lots of prongs (like a reservoir formed by U.S. Army Corps of Engineers dam), some are long and twisty like a pretzel.  Continue reading

Posted in Mapping | Tagged , | 1 Comment

From the Dev Summit – Wednesday March 19, 2008

Wednesday morning started with a talk from Alan Cooper, this years keynote speaker. The “Father of Visual Basic” delivered an insightful presentation titled Post Industrial Management. The talk compared past management strategies of the atom based industrial age to management strategies of today’s knowledge based era. He delved into a structured approach to managing developers by segregating them into three focused job descriptors which he labeled Interaction Designer, Design Engineer and Production Engineer.

During the talk Cooper threw out many shrewd nuggets of wisdom as well as humorous and accurate observations, such as “Building software is like walking through a minefield. If you don’t hit a mine it’s really quick”. His knack for pairing key concepts and ideas with aptly chosen metaphors made for a lighthearted and instructive presentation. All in all it was a very interesting talk that inspired a great deal of discussion throughout the day.

We aren’t allowed to distribute the actual presentation, but many of the concepts are covered in a similar article on Cooper’s site titled Design Engineering: The Next Step

In the afternoon, Forrest Jones and Brent Pierce gave the Implementing Enterprise Applications with the Geodatabase session.

 

 

  

This session was designed to take an enterprise centric view on common APIs that enterprise developers regularly need to work with. The session also explored many different tips and tricks to improve overall application performance:

 

 

It finished on a practical note with a group of slides covering database tuning and tracing of the complete enterprise Geodatabase stack.

 

Posted in Geodata | Tagged , | Leave a comment

From the ESRI Developer Summit – Tuesday March 18, 2008

The morning of the Dev Summit was spent in the Plenary Session. The Plenary acts as a presentation to look at some new product functionality and more recent projects that have been developed. This years Plenary went smoothly and the talks and demos did a great job of highlighting some key projects that the development teams at ESRI have been working on lately.

There was a good flow to the presentation as Jim McKinney used the newly launched Resource Center as a staging point to introduce each of the development teams and their respective lead developers.

 

Each team did a good job of not only looking at recent projects from a user perspective, but also delving into the developer perspective, showing code and programming examples of how things work behind the scenes.

Following the Plenary session the technical tracks kicked off. From the geodatabase perspective the “Effective Geodatabase Programming” session was presented by Brent Pierce and Erik Hoel, a senior developer on the geodatabase team. This session dealt with very low level programming patterns that should be followed when programming with the geodatabase API. The presentation tackled the subjects that a developer needs to be knowledgeable of to effectively use the geodatabase API.

 

Here are some slides highlighting the session contents as a teaser for those that might want to grab the whole presentation on EDN after the conference. 

 

The presenters also went into great detail about the inner-workings of the Geodatabase. The next slide explaining the client server cursor buffering model is a good example of this.

 

 

To keep the session up-beat Erik injected his classic sense of humor in some slides including interesting images on slides where he was highlighting bad programming patterns with the geodatabase API (most people got the point…).

 

 

Following this session our in house database guru’s Tom Brown, Kevin Watt and Brijesh Shrivastav gave their “Working with the Geodatabase Effectively Using SQL” session. This popular presentation lead to large crowds and a series of very interesting spill over discussions following the talk.  
 
This session went into detail on working with the geodatabase at the SQL level. The presenters dealt with PostgreSQL, SQL Server 2008 and Oracle DBMS platforms. Here is a slide given by Brijesh concerning working with geometries in the spatial type for PostgreSQL fuctionality being released in ArcGIS 9.3.

Tom also went into detail about how to work with the various spatial types available on the Oracle DBMS platform. Here is a slide highlighting some new operations added at 9.3.

Tomorrow the Geodatabase Team will be giving two sessions: Implementing Enterprise Applications with the Geodatabase and the first of two offerings of the Distributed Geodatabase Development session which delves into geodatabase replication.

Also, tomorrow morning Alan Cooper is giving a keynote speaker address which should make for an interesting talk and generate some buzz and discussion.

Posted in Geodata | Tagged , | Leave a comment