Mapping Center

How fast can you cache?  We keep asking ourselves that and keep finding that the more we know the faster we cache.

If you’re involved in the job of caching maps for online map services, you are already familiar with the need to optimize the process as much as possible so that it takes less time and effort. One way you can do this is to optimize how your maps are displayed – another is to optimize the environment you are caching your map in or the circumstances under which you are caching. This blog post is dedicated to the latter.

Here is what we’ve learned in the past few months:

1. Upgrade to ArcGIS 9.3.1. If you have not upgraded to 9.3.1, do it before caching because the file I/O for writing map cache tile images was vastly improved in the latest version. The result is anywhere from 25% to 2000% faster caching. Basically, the simpler the map, the more improvement you’ll see. That’s because simple maps draw faster and therefore file I/O is a larger proportion of caching time.
2. Use more than one computer to cache. If you have the luxury of using several computers to do your caching, break up the caching job first by map level, and, if needed for the largest scales, geographically. Each portion of the job should be done such that tiles are being written to each of the several computers’ hard drives. When the caching is done, then copy the tiles from each machine into a master cache. One tip here is that we bought a software application called Secure Copy by ScriptLogic Corporation which made the job of copying possible for the largest scales where the potential exists to produce millions of map tiles.
3. Set the number of processes to two. We’ve found that two is the optimum number of processes on each computer doing caching (we’ve been using 2 Duo CPUs E8500s each with 4Gb of RAM). One process just doesn’t take full advantage of available CPU and memory. Three processes pin the CPU, but actually produces fewer tiles per hour than two processes. Our theory is that file I/O is maxed out—we’ve been using 250GB and 1TB internal IDE hard drives.
4. Turn off the option for indexing in your cache folder. In the Windows file manager, right-click on your cache folder and select Properties, then click Advanced (Windows XP), and uncheck the option “For fast searching, allow Indexing Service to index this folder”. This will make cache production 15-20% faster.
5. Check your available memory. During our caching processes, we’ve usually got the Windows Task Manager open because we want to know how much memory is being used by the ArcSOC.exe processes that are being used to create the cache. This is especially important if you’ve got other services running on the server, because these services are taking up memory, too, and you want to make sure the caching process, which will be more demanding of memory usage, is not pushing out of your available physical RAM and into your pagefile.sys (which will result in catastrophically slower caching times!)
6. Relocate the server’s pagefile.sys file. Unfortunately, stability of the caching process can be an issue, especially if you have other services running on your server. We’ve found one particularly good practice to minimize unexpected stoppage of our caching processes, which is to relocate the server’s pagefile.sys file onto a dedicated partition, preferably on a disk that is not primarily involved in caching. Minimally, we used to do this for printing large maps because it kept the pagefile.sys file from becoming fragmented, which limited the size of available memory blocks.
7. Avoid having other services running while you are caching – this is really a summarization of the last few tips together!

Recently, a new version of ColorBrewer called ColorBrewer 2.0 (colorbrewer2.org) was released by Axis Maps. ColorBrewer is a web tool for selecting colors for maps. The original ColorBrewer was released in 2002, and the update incorporates comments that the developers, Dr. Cynthia Brewer of Penn State University and Dr. Mark Harrower of University of Wisconsin Madison (he used to be a grad student at Penn State), have received over the years. Here are what some of the new features are:

• In the original version of ColorBrewer, you were given a set of predefined colors to modify the symbology of roads, cities and boundary lines. In the new version you can select from a wider variety of colors to customize the symbology of map features so they appear more similar to your own to see how a given color scheme works in conjunction with your symbology.

  

• You now have the ability to turn on a hillshade in the background. Transparency settings can be varied to see how the color schemes look when overlaid on a black to white ramp for the hillshade.

  

• Previously, you had to check the suitability of each color scheme individually for colorblind-safe, print-friendly or photocopy-friendly. You can now filter the color schemes based on those factors so only the ones that are suitable for your map’s media and display requirements are shown.

  

• The Export options allow you export colors into Adobe Illustrator or Photoshop, copy and paste color specifications, and/or download the color specifications into an Excel document.

  

• Also on the Export page is an option to download a tool to run ColorBrewer directly inside ArcGIS 9.X. This allows you to see the color schemes directly in ArcMap (instead of manually typing them in). The export page provides this link to the National Cancer Institute GIS where you can get more information on this tool as well as the download: http://gis.cancer.gov/tools/colortool/.

This doesn’t mean that the original version of ColorBrewer isn’t out there -- it still is!

On Thursday, June 25, Charlie Frye (ESRI Mapping Center) and Mark Ho (ESRI Educational Services) will present the Live Training Seminar Getting Started with Map Templates at 9am, 11am, and 3pm Pacific Time.

This seminar will provide an overview of what map templates contain, how to get started, and how to adapt the contents of the templates or evolve your data to your mapping needs. Participants will learn where to find and download map templates. Templates include example map documents, data models, geoprocessing tools, and more—each template is a complete solution for a given kind of map. This seminar will then discuss how you can use your data with map templates to produce professional quality basemaps and publish them.

For more information regarding templates, visit the ESRI Map Templates Resource Center. And for archived Live Training Seminars, visit the ESRI Educational Services homepage.

A couple of weeks ago, Aileen wrote a blog post called "About geographic transformations and how to choose the right one". In it, she described many of the parameters that you can set for map projections.  Two that were not mentioned were angular and linear units, so I thought it might help to describe them here.

Angular units in the projection file

In most Geographic Coordinate System projection files, such as GCS_North_American_1927.prj (shown below), you have a unit of measure described as a "Degree", with the value 0.017453292519943295.

GEOGCS["GCS_North_American_1927",DATUM["D_North_American_1927",SPHEROID ["Clarke_1866",6378206.4,294.9786982]],PRIMEM["Greenwich",0],UNIT ["Degree",0.017453292519943295]]

Trigonometric functions are used in geodesy to calculate coordinates on a datum and spheroid in order to perform geographic transformations between datums and other operations. Standard trigonometric functions, however, do not use units of degrees – these calculations are performed with units of radians.

The circumference of a circle, from basic geometry, is defined as pi times twice the radius of the circle, or C = 2pi r. However, this yields a circumference in linear units, like feet or meters. The trig functions used in operations on a spheroid and datum can only use angular units of radians so these trig functions will not work.

Instead of thinking of the circle in terms of a circumference measured in linear units like feet or meters, ArcGIS internally converts the circumference to radians, a different angular unit.

The circumference of a circle in radians equals 2pi.

The value in the projection file for a Degree is the number of radians in a degree. Here is the calculation:

2pi (radians in a circle) / 360° = 0.017453292519943295769236907684886 radians per degree

Since ArcGIS Desktop calculates coordinate position to 16 significant digits, the value of 0.017453292519943295, rounded to 18 places to the right of the decimal, is sufficient to maintain data accuracy when performing these calculations.

Having converted the angular units of measure from degrees to radians, internal trigonometric calculations can be performed using standard formulas.

(Thanks to Rob Juergens of the ArcGIS Projection Engine Team for explaining this – I’ve wondered about this value for "degree" for several years.)

Linear units in the projection file

A projected coordinate system expresses the location of data using linear units that can be measured on the ground with a ruler. Projected coordinate systems are used when distance or area must be calculated for data. The most commonly used linear units are FEET or METERS, although other linear units can also be used. In the United States, two different definitions of the foot are typically used:

The U.S. Survey Foot, describes as the unit foot_us in ArcGIS Desktop. This unit is equal to "exactly" 1200/3937 of a meter, or 0.30480060960121920243840487680975...

The International Foot is also used in ArcGIS Desktop, expressed as the unit foot. Some states in the US have said that this unit can also be used with the State Plane Coordinate System, since this definition of a foot will convert exactly to a meter. The International Foot is equal to exactly 0.3048 of a meter.

The unit meter has been redefined many times over the years. The standard definition of the meter today is based on the speed of light in a vacuum: exactly 299,792,458 meters per second. The meter is defined as the distance light travels in a vacuum in exactly 1/299,792,458 of a second, and is equal to 3.2808333333333333333333333333333 U.S. Survey Feet (3937/1200) or 39.37 U. S. Survey Inches.

In a blog entry posted last week called Filling and clipping a raster, Aileen described how to fill in holes in a "bad DEM" using data from an existing "good DEM", then clip the filled DEM to the outline of a feature.

The blog post suggested using some ArcGIS geoprocessing tools that are available with the Spatial Analyst extension. As with most GIS operations, there is more than one way to get to the final answer! In this blog post, I describe how Map Algebra can be used to achieve the same results.

We can break this process down into three steps:

• create a "clip raster" from an existing polygon feature,
• fill in the missing cell values in the bad DEM with values in the good DEM to create a filled DEM, and 
• clip the filled DEM to the extent of the clip raster.

To demonstate, I used the following datasets:

• WashingtonState – a polygon feature of the state of Washington,
• badDEM – the DEM that needs to be filled and clipped, and
• goodDEM – the DEM from the ESRI Data and Maps DVD (world_elevation\etopo2\etopo2 GRID) that I used to fill the badDEM.

You can accomplish the entire process using three Map Algebra statements:

So what exactly are these statements doing?

-- Shapegrid is the map algebra equivalent of Feature to Raster, and defines:

• the feature you want to convert (in this case, WashingtonState),
• the name of the column in the <shapefile>.dbf file used to assign grid cell values (I’ve used # to set this to no field), and
• the cell size of the output grid (I’ve set this to 50) – note that the units are the same as for your input features (in this case, feet).

The output is a raster that closely resembles the Washington State boundary (how closely it resembles the polygon is a function of the cell size of the raster). All cells have a value of either 0 or the value from the attribute you selected to assign grid cell values.

-- here I’ve combined two steps into one using the Con and Is Null statements:

1. The Con statement sets up a conditional test: if a cell in the raster passes the test, it is given a certain value (in this case, the value of the cell in the good DEM). If it fails the test, it is given a different value (in this case, the value of the cell in the bad DEM.)
2. The conditional statement uses Is Null to determine whether the cell being processed in the bad DEM is NoData or not. This statement returns a value of 1 if the input value is NoData and 0 for cells that are not.

The conditional statement is then: set up a condition that tests badDEM for NoData cells; if the cell has a NoData value, give it the value of the cell in the goodDEM; otherwise give it the original value in the badDEM. The output is the badDEM with new values for NoData cells from the goodDEM.

-- Selectmask is the Map Algebra equivalent of the Extract by Mask Spatial Analyst geoprocessing tool.  It asks: what is the raster to extract from (DEMfill), and what is the mask to use as the extent (land_grid)?  The output is a raster that contains elevation values only for the cells that fall completely within the clip raster. All other cells will have NoData values.

Note that another way to clip the filled DEM to the WashingtonState clip raster would be to use another Con statement:

-- this yields the same result, but you have to know the grid cell values that the Shapegrid conversion will produce, in this example 0. The conditional statement is: set up a condition that tests if the cells in land_grid equal 0 (note that you need to use "==" or EQ symbol for the "equal to" condition); if they do, give them the value from DEMfill; otherwise, keep the original value (which in the case of land_grid is NoData because anything outside of land_grid has no elevation value).

So your next question might be, "How you can create a ModelBuilder model using this Math Algebra approach?" You can use the Single Output Map Algebra (SOMA) and Multi Output Map Algebra (MOMA) geoprocessing tools. Below is what the ModelBuilder model looked like when I used the three SOMA expressions to fill and clip a raster. Each line of text in a yellow box in the model is a SOMA expression.

This is what the SOMA tool interface looks like with a Map Algebra statement filled in:

Note that the Input raster data shown in ModelBuilder includes DEMfill and land_grid. These are shown as inputs to the SOMA expression in the ModelBuilder interface. The output raster is also shown in ModelBuilder.

You can download this .zip file which contains the the model described above so you can try out the methods for yourself.

Perhaps the best resource to learn more about Map Algebra is the book by Dr. C. Dana Tomlin called Geographic Information Systems and Cartographic Modeling. This book, published in 1990, was based on the PhD dissertation that Dr. Tomlin wrote in 1983. In his book, Dr. Tomlin introduced the the concepts of a simple and an elegant set-based algebraic approach to manipulating geographic data that has since become the basis of much of the work in raster analysis. Read more about Dr. Tomlin and his book here: http://www.design.upenn.edu/new/larp/facultybio.php?fid=121.

Sometimes you want to use raster data, like a digital elevation model (DEM), but it doesn't have the same exact extent as the area you are mapping. For example, if I use gtopo30 and "countries" data (available on the ESRI Data and Maps CD) to create map of the Pacific Northwest, the coastline boundaries do not coincide. In some places the elevation values are missing for inland areas, and in other places, there are elevation values outside the extent of the land area. So we need a way to clean up the data and make them coincide. Just clipping won't work as this won't add the missing elevation values.

I can fix all this if I have some other elevation data to fill in the missing interior elevation values. For example, I can use etopo2 data (also on the Data and Maps CD) to get the elevation values for the pixels that need them.

Although the etopo2 data are coarser resolution, this will be difficult to see at the scale I am mapping, especially when I further modify the data by creating a hillshade. So I can go ahead and use these data. 

Note that before you decide which dataset to use to "fill in the gaps", be sure you check out the z units – in some datasets elevation is in meters and in others it is in feet. Both gtopo30 and etopo3 use meters so there is no problem using these two datasets as they are. (The fix would be to create a new raster elevation dataset by converting one DEM to the units of the other DEM by multiplying by the appropriate conversion factor.)

Once you have the two raster elevation datasets and a polygon feature class for the area you are mapping, these are the steps to filling in the missing elevation values and clip to the land extent:

1. Convert the polygon feature class for the land area to a raster dataset. We'll call it "land_grid".
2. From the raster dataset that has the missing elevation values (in this example, we'll call it "gtopo30"), create an IsNull raster. (Use the Is Null Tool in the ArcToolbox, Spatial Analyst Toobox, Math Toolset, Logical Toolset.) We'll call the resulting raster dataset "gtopo_isnull".
3. Use a conditional statement to create a new raster dataset that adds the elevation values from the other raster dataset (we'll call it "etopo2") to the one with the missing values. (Essentially, this is the conditional statement: if there is an elevation value in the data set with missing values, use it; if not, use the elevation value from the other raster dataset.) We'll call this raster dataset "gtopo30_fill".
             con(gtopo_isnull = 1, etopo2, gtopo30)
4. Use a conditional statement to set the extent of the resulting elevation raster dataset to the same extent as your land area. (Essentially, this is the conditional statement: if the raster dataset that you created from the polygon feature class is >= 1, then use the elevation value from the step above, otherwise, do nothing – which results in a NoData value.)
             con(land_grid >=1, gtopo30_fill)

At this point, the resulting raster dataset should 1) have all the elevation values, and 2) exactly coincide with your land extent.

This post, is to announce the release of a map template for historical GIS called Historical GIS:  Boston 1775. If you’ve never given historical GIS a second, or a first thought, you might find the contents interesting and maybe even applicable to your work. Consider that the vast majority of GIS data is historical, even if it’s only a few minutes old.

Genealogy and history have been hobbies of mine for years now, and I’ve since developed an interest in colonial U.S. history -- in particular, the U.S. Revolutionary War. For me, GIS and mapping provided an obvious way to make sense of the history I found fascinating. As such, I found it more than a little ironic that relatively few of the historically-inclined geographers I’ve met had turned to GIS much less demonstrated GIS-based methods as a sound basis for scholarly historical inquiry. But, I’ve been happy enough to take that opportunity to blaze a trail.

That trail formally began over two years ago when I began to show some of my historical “work” to historians and historical geographers. I got a good reception, constructive criticism, and encouragement. In particular, I was encouraged to share what I had done with others because all too often historical GIS projects in various universities were not communicated. This is an invitation to the dozens of you I know who are doing historical GIS to do the same on the Map Templates Resource Center.

One of the projects I undertook was to create a GIS of Boston in 1775, using only maps published from that period. I eventually included some later source material of reliable historical character to flush out locations for specific or notorious events, structures, and so on. My goal was to create an inventory and therefore as complete a picture as possible of Boston’s environs in 1775. Not only that, I wanted to be able to cite every feature, making it possible to create a map that was in essence a spatial argument for what I think was in Boston in the year 1775.

Using the database model in this template, I think it is now quite possible for anyone with the time and interest to construct a historical GIS of any city at any time. Granted, adaptations would be necessary, but I don’t think the excuses, that made historical research to this point apt to not include maps, hold any longer.

As added encouragement, publishing your work as a template will very likely improve the scholarship of your map research and content. I mostly cleaned up the database and map for this template over a year ago when I presented this work at the Boston Association of American Geographers Conference—in doing so, I found and fixed a number of my own miscues. So revisiting my map and database allowed me to improve the quality of both.

This post is to announce the Map Templates Resource Center. Map templates are useful examples. Each map template is a kit that contains a collection of resources needed to transfer a specific map’s design and ArcGIS implementation to you.

These templates are not the template map documents that you might have saved as an .MXT file. Instead these templates are .ZIP files that typically include:

• An example map document file (.mxd) that you should explore and use to learn how we used ArcMap to make the map.
• Some sample data; sometimes a map is not enough and seeing how we structured the data makes a big difference.
• Documentation ranging from how to get started to descriptions of critical tasks that might not be obvious to even advanced ArcGIS users.

Some templates will include additional resources like:

• Geoprocessing tools or models to help you expedite any required data processing. Sometimes it is necessary to make typical GIS data map-worthy, and when that involves geoprocessing we’ll include the tools or cover the essential steps in the template’s documentation.
• An example web application. This can range from a simple set of HTML code for designing pop-ups that show something informative, to a complete example application written in Flex, JavaScript, or Silverlight.

To get started with the templates, click the Map Template Gallery tab and find a template you might be interested in. Click on the template’s name to go to its webpage. There you can read about the template, download it, and if you’ve got questions as you use the template, use the comments section. The ESRI Cartography Team (cartographers from the Mapping Center and ArcGIS Online Content Teams) will be monitoring the comments.

The Map Templates Resource Center complements the ESRI Mapping Center by providing useful examples, while Mapping Center will continue to provide cartographic guidance and the details of specific cartographic methods. That means you will soon be seeing some more detailed content about these templates on Mapping Center.

Finally, included in the initial set of templates (there are more coming) are a set of three that we developed based on a new upcoming ArcGIS Online Topographic map. That map will be available on ArcGIS Online within the next month or so.

You will often be prompted to select the geographic transformation when you are projecting data or setting the projection of a data frame in a map document. Here are some concepts that might help you understand what this is all about AND how to make the right selection.

First, "geographic coordinates" are expressed in terms of latitude and longitude. "Latitude" is the north-south angular measure from the equator to the point of interest. "Longitude" is the east-west angular measure along the equator from the prime meridian to the point of interest's longitude. Assuming that the earth is a sphere, geographic coordinates are determined relative to the center of the sphere - these coordinates are called "geocentric latitude and longitude". (See the figure at the right; all figures are from Map Use: Reading and Analysis, 6th edition, ESRI Press.)

Second, we know the earth is not a perfect sphere – it is elongated along the equator due to centrifugal force - its "equatorial radius" is longer than its "polar radius". (The figure below is drawn using the real difference in length between the two axes which is not noticable at this scale.)

Therefore, instead of assuming that the earth is a sphere, we can more accurately assume it is an "oblate ellipsoid of revolution" (a sphere that is slighted squashed in at the poles), which we usually just call an "ellipsoid". Rather than use coordinates defined by the spheroidal approximation of the earth, we use geographic coordinates expressed as latitude and longitude that are referenced to a particular ellipsoid of a defined shape and size. These coordinates are called "geodetic latitude and longitude", and they will be slightly different from the geocentric ones. (See the figures below; in the one at the right, the difference between the equatorial and polar axes is greatly exaggerated!)

Different ellipsoids are used to approximate the earth, based on slightly different definitions of polar and equatorial axes of the ellipsoid - that is, the size and shape of the ellipsoid.

Third, imagine that the ellipsoid that is used as the model of the earth is then "positioned" relative to the surface of the earth to fit a specific area or use. This "fitted ellipsoid" becomes the frame of reference for specifying point coordinate locations, and it is called a "datum". You should now be able to see how coordinates are datum dependent!

We have said that the ellipsoid is positioned to fit a specific area or use.  If the ellipsoid is being used as the basis for a national or regional datum (i.e., frame of reference to define coordinates), it will be positioned to best fit the area of interest (like the North American Datum of 1927, or NAD27).  If it is used as the basis for a world datum, the ellipsoid will most likely be positioned relative to the center of the earth's mass (like the World Geodetic System of 1984, or WGS84.) 

Fourth, the datum, along with the prime meridian and angular units comprise a "geographic coordinate system" (GCS) which is referenced to the 3D ellipsoidal approximation to the earth. This is used to specify locations defined by latitude and longitude and heights above or below the ellipsoid at any latitude, longitude location.

Fifth, a "projected coordinate system" (PCS) results when the GCS is flattened down onto a 2D surface (e.g., paper, computer screen, etc.)  A projected coordinate system is always defined relative to the GCS that it comes from, which in turn is based on a specified datum, which is defined in part by its ellipsoid.

SO – when you are asked to select the geographic transformation, you are being asked to choose which mathematical calculation will be used to convert coordinates referenced to one datum to coordinates referenced to another datum. This geographic or datum transformation is often embedded in the procedure to convert between coordinate systems, or in other words, the projection process. This process often involves more than one coordinate transformation. For example, let's say you want to convert between two projected coordinate systems. This is what would happen in the projection process:

1. Define the PCS that your data are currently in. The PCS includes the GCS.
2. Unproject the data to geodetic latitude and longitude using the same GCS.
3. Transform the data to geodetic latitude and longitude using the new GCS.
4. Project the data to the new PCS using the new GCS.

You can see that when you select the PCS you want to use, you also need to select the geographic transformation (step 3 above) because there are multiple mathematical calculations that can be used to define how the coordinates will be converted to the new GCS. Note that even if you are converting from one GCS to another GCS, you still need to define the geographic transformation; however, the will be no need to convert from PCS to GCS and visa versa during this kind of projection process.

So how do you choose the geographic transformation that should be used? Here are two ESRI Knowledge Base articles that can help you:

HowTo: Select the correct geographic (datum) transformation when projecting between datums. This article contains links to downloadable zip files (for different versions of the software) that contain a list of all available datum transformations and their appropriate geographic areas of use.

HowTo: Determine which NAD_1983_To_WGS_1984 transformation to use.  The gist of this article is summarized below:

1. NAD_1983_To_WGS_1984_1 - for the entire North American continent.
2. NAD_1983_To_WGS_1984_2 - for the Aleutian islands.
3. NAD_1983_To_WGS_1984_3 - for Hawai'i.
4. NAD_1983_To_WGS_1984_4 - superseded by _5; this transformation method should no longer be used!
5. NAD_1983_To_WGS_1984_5 - for the 48 contiguous United States.
6. NAD_1983_To_WGS_1984_6 - for the Canadian province of Quebec.
7. NAD_1983_To_WGS_1984_7 - for the Canadian province of Saskatchewan.
8. NAD_1983_To_WGS_1984_8 - for the Canadian province of Alberta.

Note that geographic transformations work in either direction. For example, the transformation listed as NAD_1983_To_WGS_1984_5 transforms from NAD 1983 to WGS 1984, as well as from WGS 1984 to NAD 1983. When using the Project Tool, the geographic transformation is recorded in the metadata.

To learn more about geographic transformation methods, check out:

• ArcGIS online help at: http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?TopicName=Geographic_transformation_methods
• Map Use: Reading and Analysis, 6th edition, by Kimerling, Buckley, Muehrcke and Muehrcke. ESRI Press. See Chapter 1: The Earth and Earth Coordinates.
• The Encyclopedia of Geographic Information Science, edited by Karen Kemp. Sage Publications. See the entries for "Datum", "Transformations, Coordinate" and "Transformations, Datum".

Thanks to Melita Kennedy, David Burrows, and Rob Juergens of the ArcGIS Projection Engine Team for helping me to make sure I said things correctly!

Someone pointed out to me that ESRI's CEO, Jack Dangermond recently did an interview that focused on web mapping. You can read and hear it here: http://radar.oreilly.com/2009/04/jack-dangermond-interview-web-mapping.html.

In fact, he is talking about what we on the Mapping Center team have been helping with recently - map templates.  These map templates are being created to give users a starting point to create their own maps and to share cartographic designs for web maps. We are planning to make a number of them available prior to the User Conference.

Take a moment to read or listen to the interview so you know what's coming soon!

This week, the Federal Recovery Accountability and Transparency Board is hosting an online forum looking for input from the Information Technology (IT) community on how Recovery.gov can be designed to enhance information access and transparency. For more info, see http://www.recovery.gov/.

Here is ESRI's position on the matter, and what Jack will be bringing to the conversation: http://www.esri.com/company/stimulus_recovery.html

The crux of Jack's message is this:

"IDEA: Add Mapping and Geographic Analysis to Recovery.Gov.  Mapping and Geographic Analysis capabilities would enable Recovery.Gov to provide a "full picture" of stimulus spending as envisioned by the American Recovery and Reinvestment Act (ARRA)."

This seems like a no-brainer to us on Mapping Center, but maybe that is because we think about maps all the time. We can all take this opportunity to bring more attention to the power of maps through the online forum.  Why not take a moment to add your own two-cents worth?

There is another blog post on this from an ESRI employee here: http://blogs.esri.com/Dev/blogs/arcobjectsdevelopment/archive/2009/04/24/Recovery.gov-needs-to-hear-from-the-GIS-Community-_2D00_-Next-Week_2100_.aspx

From time to time we've had to create some building footprint data. A colleague was in my office yesterday looking at the map to the right, and remarked that he thought that adding building footprints to maps "humanizes" the map. An interesting observation, and one that I'm not inclined to argue against.

We've learned a few tricks for making haste with little or no waste to produce building footprint data. The premise is that you're starting with a properly rectified ortho-image base (it's also helpful if your elevation base is rectified the same way--it makes the ensuing cartographic products look much better).

First, get your data model sorted out.  For building footprints, we use a polygon feature class that typically has two essential fields:

1. Name: [type = TEXT, length = 64]
   
2. BldgType: [type = LONG INTEGER] We typically have a coded value domain on this field doing something like:  0 = General Case, 1 = Commerical - Retail, etc.  We have categories for: Education, Industrial, Religious, Healthcare, etc.  Many places use their zoning codes as a starting point for setting this up. One tip is to set the default value to your most common type of building, e.g., "General Case".
   

Second, set up ArcMap so you can easily use the tools you'll need most. The key is to quickly produce lots of buildings in order to avoid situations where you have to drag your mouse all over your screen for common tasks, and to minimize mouse clicks for common tasks. To do this create a new custom toolbar, here is a good set of tools to start with:



All of these tools are already in ArcGIS, in these Categories on the Customize dialog's Commands tab: Editor, Advanced Editing, or Pan/Zoom.  The idea is that this toolbar floats inside of the data view of ArcMap, close to where you're editing--minimizing the distance of the cursor to whatever you need next.

Editing Methods & Tips:

1. Since most buildings are rectangles or composed of rectangles, rotate your data frame so that your imagery shows the buildings as horizontally oriented.   Now you can use the rectangle tool to create a new features.  This ensures you have right angles where right angles ought to be.  Don't waste time using the Sketch tool trying to do vertex by vertex data entry.
2. Many buildings are a combination of rectangles.  You can quickly digitize overlapping rectangles that cover a given building; to complete the building, select all of the buildings, and then Click Merge--having the custom toolbar is a big time saver here.
3. Set your snapping environment to snap to building vertexes and edges.  You'll find it easier to create "L" shaped buildings if you can snap to the outside corner of the "L".  You can follow the same logic to create buildings with complex shapes as well.
4. Use the Circle tool to create curves that are parts of circles.  This works well since most curves on buildings are based on circles (the more complicated the curve, the higher the construction cost).  You can either snap to tangent points on circles, or complete circles by snapping to corner vertexes.
   
5. For buildings with angles that the rectangle tool doesn't easily handle, first try using the rectangle tool where you can--rotate your data frame one or more times to make this easy.  Otherwise use the Sketch tool to create triangles that snap to your rectangle's corners.  

Sometimes you will find that some annotation you had thought you produced is missing. You can add this missing annotation into your existing annotation feature class without having to recreate all the annotation. The approach you take will depend on whether you are creating standard annotation or feature-linked annotation.

Standard annotation elements are pieces of geographically placed text that are not formally associated with features in the geodatabase. For example, you might have a piece of standard annotation that represents a mountain range—the annotation simply marks the general area on the map.

Feature-linked annotation is a special type of geodatabase annotation that is directly linked to the features that are being annotated by a geodatabase relationship class.

Here are some tips for working with feature-linked annotation.

Once you have created your annotation, the first step is to check whether or not all of your annotation was created for a given feature class. One way to do this is using the instructions in Step 1 below – or you can simply visually examine your map. Was all of your annotation produced?

If the answer is yes, then you can begin the process of editing the annotation so it is placed exactly where you want and has the exact properties you want (e.g., font characteristics, character spacing, word spacing, offsets, alignment, etc.)

If the answer is no, here are a couple of ways to check and resolve some issues you may find:

• Right click on the annotation feature class in your ArcMap document, click Properties, and click the Symbology tab. Check the box 'Draw Unplaced Annotation'. Any annotation that was not placed will show up in red or any other color you choose to draw your unplaced annotation with.
• If you see that most of your features have either placed or unplaced annotation, but other features within that same feature class do not have any annotation, don’t panic, there is a relatively easy and straightforward way to fix this. First, join your annotation feature class to the original feature class (this only works with feature linked annotation) – join the ObjectID for the feature class to the FeatureID for the annotation feature class. Once you have joined the two feature classes, work through the following steps:
1. Open the attribute table for the annotation feature class. Find all of the features that have a <Null> value in the TextString column by sorting on that column. Select those features using this query in the Select By Attributes dialog: (joined feature class)[TextString] IS NULL. The name of your attribute will depend on the join. For example, when I wrote my query it looked something like this: urban_144K_Anno144447.TextString IS NULL.
2. Once the features are selected, right click on the feature class that you joined your annotation feature class to, click Selection, and choose the option to "Annotate Selected Features."
3. In this new window, select the annotation class in which you want to store the annotation.
4. Check the check box to add unplaced labels to the Overflow window. If any annotation features cannot be placed on the map, they will be listed in the Unplaced Annotation dialog box.






5. Click OK.

You should now have feature linked annotation produced for those features that did not have annotation before.

As you zoom in (or out) of the online maps you see on Virtual Earth (VE) or Google Maps (GM), you are actually seeing a series of different maps with slightly different information displayed at each zoom level. Zoom level is indicated and controlled in an online map by the vertical zoom slider, like the one shown at the left in the image here. Whenever the zoom level is changed, a different map is shown.

Of course, these maps are well designed so that viewers are largely unaware that they are seeing these different maps. The foundation for good design of an “online map” hinges on understanding how to design for each of the zoom level represented in the entire online map. Colors, fonts, number of and types of features, etc. are all seriously considered when each of the maps is created for each of the zoom levels.

There are twenty zoom levels for Virtual Earth or Google Maps. The corresponding map scales that you would design and create your maps at if you wanted them to mash up on VE or GM are:

For some months now, we’ve kept printed versions of the table above taped by our monitors so we could easily remember which map scales matched the zoom levels.

When you create a cached map service, tiled images are created at each map scale from your map document. These images are stored in a pre-defined folder structure on your server, and these folders are named for the corresponding zoom level.

Knowing which map scales match the zoom levels is important because sometimes caching a map takes a long time. If there’s any error in symbology or labeling settings for a certain zoom level, that map scale will need to be re-cached, which could take a lot of time. By browsing into the folders where the cached image tiles are stored, you can preview the tiles and verify that they look good. If they don’t, cancel the caching process, fix the map document, and re-start the caching process.

In order to help with creating maps at these zoom levels easier, use this cachescales.txt file - it can be loaded into ArcMap's scale control when you choose to customise the list (with Version 9.3). Setting your available map scales this way, it is easy to work at exactly the map scales users of your map service will see as they zoom in and out of your online map.
 

We’ve blogged about symbolizing hillshades (rasters that are derived from elevation raster datasets, like DEMs, via the Hillshade tool), but never really covered the basics of the data used to create hillshades, so we wanted to take a minute and share a few best practices we’ve been adopting.

Before getting started, though, it’s worth noting that we’ve been storing our rasters in file geodatabases. For us, these included some rather large hillshade datasets, ranging between 5Gb and 60Gb.

Pyramid resampling technique

First, the default pyramid resampling technique for rasters is Nearest Neighbor. For hillshades, we prefer Bilinear -- it looks smoother and results in fewer abrupt changes in the gradation of shading.

8-bit unsigned integer data

Second, hillshades can be generated from elevation data as well as non-elevation data that represents a statistical surface. You we can take advantage of the fact that the values in a hillshade always range from 0 to 255. The Hillshade tool by default produces output that matches the input data. A typical elevation raster is a 16-bit signed integer, which allows for elevations above and below sea level and for the values to be stored in either of the typical units (meters or feet). Using the Copy Raster tool, these hillshade data can be converted to 8-bit unsigned integer rasters. The advantages of doing this are: 1) reduced file size, and 2) the ability, for large hillshade datasets, to rapidly access the histogram for symbology purposes.

JPEG2000 Compression

Third, the default output from the Hillshade tool uses LZ77 compression; we’ve found the JPEG2000 compression is even better.

Implications for file size

We’ve found that by combining the use of 8-bit unsigned integer data with the JPEG2000 compression, we cut our file sizes nearly in half.

Here’s an example using Oahu:

The default is normally 16 bit signed integer with nearest neighbor pyramids and LZ77 compression. The file size is 2.26Mb.

Converting to 8 bit unsigned integer data with default nearest neighbor pyramids and LZ77 compression, the file size drops to 1.8Mb.

Using all the settings described above, the 8 bit unsigned integer data with bilinear pyramids and JPEG2000 compression results in a file size of only 1.4Mb.

Notice that there is no difference between the first two images (outside of the nearly 25% file size savings). Keep in mind that the above images are showing pyramids, i.e., not the highest resolution; these hillshades look identical at full resolution.