Designing and Optimizing Image Services for High-Performance Analysis

Image services are not only for serving imagery; they can also perform dynamic pixel-level analysis on multiple overlapping raster datasets using chained raster functions.  Image services deliver blazing fast performance with pre-processed source imagery, especially when it is served from a tile cache.  Dynamic image services, on the other hand, may not respond as quickly because of the additional processing demands they place on the server.  High performance is important for dynamic image services because the data is re-processed automatically each time the user pans or zooms the map.  Dynamic image services typically produce sub-second responses for simple types of analysis.  But more complex analysis may take much longer depending on the complexity of the processing chain and the number of input datasets involved.  How can we get better performance in those situations?  To answer that question I ran a series of tests to see how the following factors affect the performance of dynamic image services:

  • Map scale and source data resolution
  • Resampling method
  • Source data format and compression
  • Project on-the-fly
  • Request size

In this article I present the results of those tests along with some suggestions for maximizing performance.  My testing machine is a desktop computer running Windows 7 SP1 and ArcGIS 10.2.1 with 18GB of RAM and a quad-core Intel Xeon W3550 processor running at 3 GHZ.  The test data was stored on an otherwise empty 2 TB SATA hard drive that I defragmented and consolidated prior to testing.   The tests were configured to determine the average response times of services under various conditions.  By “response time” I mean the time it takes a service to retrieve the source data, process it, and transmit an output image.  Transmission time was minimized by running the testing application directly on the server machine.

This information is written with the intermediate to advanced GIS user in mind.  I assume the reader has a general understanding of image services, raster data and analysis, raster functions, geoprocessing, mosaic datasets, map projections, and map service caching.

Map Scale and Source Data Resolution

The pixels that are processed for analysis by dynamic image services are generally not identical to the pixels stored in the source datasets.  Instead, the source data pixels are first resampled on-the-fly to a new size based on the current scale of the map.  This formula shows the relationship between map scale and resampling size when the map units of the data are meters:

Resampled pixel size = map scale * 0.0254/96

The resampled pixel size is analogous to the “Analysis Cell Size” parameter in the Geoprocessing Framework and is sometimes referred to as the “pixel size of the request”.  As you zoom out to smaller map scales, the resampled pixel size increases until eventually the service resamples from the pixels in the pyramids.  Resampling from pyramids helps to keep the performance of the service relatively consistent over a range of map scales.

Chart 1. Performance of an image service that performs a binary overlay analysis over a range of map scales.

Performance still varies depending on map scale and typically looks similar to chart 1.    I generated these results using an application configured to simulate a single user panning the map 100 times in succession at specific map scales.  The chart shows the average time the service took to process and transmit the output images for different map scales.  This particular service was configured with a raster function template to perform a binary overlay analysis on eleven overlapping rasters in a mosaic dataset.  The pixel sizes of the source datasets ranged from 91.67 to 100 meters.  The raster function template was configured to return a binary result, where each output pixel is classified as either “suitable” or “unsuitable” based on the analysis parameters.

Take a look at the three points along the horizontal axis where the response time drops abruptly.  At those map scales the resampled pixel size is the same as the pixel sizes of the pyramids in the source data.  The processing time for resampling is the lowest at those scales because there is nearly a 1:1 match between source data pixels and resampled pixels.  Client applications which use this particular service will see dramatically faster response times if they are limited somehow to only those scales.  One way to do this is to use a tiled basemap layer.  Web mapping applications which use tiled basemaps are generally limited to only those map scales.  The most commonly used tiling scheme is the ArcGIS Online/Bing Maps/Google Maps tiling scheme (referred to hereafter as the “AGOL tiling scheme” for brevity).  The red-dotted vertical lines in the chart indicate the map scales for levels 7 – 12 of this tiling scheme.  Unfortunately those scales are not very close to the scales where this service performs it’s best.  There are two options for aligning source data pixels and tiling scheme scales:

  1. Build a custom basemap with a custom tiling scheme that matches the pixels sizes of the data.
  2. Sample or resample the data to a pixel size that matches the tiling scheme of the basemap.

Chart 2. Performance of the binary overlay analysis service with different source data pixel sizes

The horizontal axis in chart 2 represents the “pixel size of the request” rather than map scale as in chart 1.  The orange graph shows the response times of another service configured identically to the first one in blue, except it uses source datasets that were up-sampled to 38 meter pixels using the Resample geoprocessing tool.  Up-sampling to 38 meters aligned the service’s fastest response times with the AGOL tiling scheme scales, which resulted in a significant decrease in processing time at those scales from approximately 1.5 seconds to about 0.5 seconds.  Furthermore, notice that performance is improved at nearly all scales except for the very largest.  This is most likely due to having all the source data at the same resolution (38m) instead of three (91.67m, 92.5m, 100m), and/or because the source data pixels are also aligned between datasets (accomplished by defining a common origin point for each resampled raster using the “Snap Raster” environment setting).

Admittedly, using the Resample tool to prepare data for analysis is not ideal because it results in second-generation data that is less accurate than the original.  This may be perfectly acceptable for applications intended to provide an initial survey-level analysis;however, it’s best to generate new first-generation data at the desired pixel size whenever possible.  For example, if you have access to land-class polygons, you could use them to generate a new first-generation raster dataset at the desired pixel size using the Polygon to Raster tool, rather than resampling an existing land-class raster dataset.

To determine by how much performance improved with 38 meter pixels, I calculated the percentage change in average response times for each scale and averaged the values over multiple scales.

Up-sampling the source data to 38 meter pixels reduced response times by  63.8% at the poorest-performing -target map scales!  38 meters was not my only option in this example.  I could have chosen a size that corresponded to one of the other tiling scheme scales.  The following table lists all the map scales of the AGOL tiling scheme, and the corresponding pixel sizes in units of meters, feet and Decimal Degrees.   The three columns on the right provide suggested values for sampling raster data.  These suggestions are not set in stone.  It’s not necessary to sample your data to exactly these recommended sizes.  The key is to choose a size that is slightly smaller than one of the sizes of the target tiling scheme scales.

Map Scales and Pixel Sizes for the ArcGIS Online/Bing Maps/Google Maps tiling scheme

By the way, matching the pixel sizes of your data with a basemap tiling scheme is also useful for workflows that involve static imagery overlaid onto a tiled basemap.  For those cases, you can build mosaic dataset overviews for viewing at smaller scales instead of raster pyramids.  One of the great things about mosaic dataset overviews is that you can define the base pixel size of overviews as well as the scale factor to match your target tiling scheme.  This way you don’t have resample the source data to a new base pixel size in order to cater to any particular tiling scheme.

Resampling Method

The resampling method specified for an image service request also has an impact on performance.  The choice of which one to use should be based primarily on the type of data used in the analysis.  Chart 3 shows the performance of the binary overlay analysis service (with 38 meter data) with different resampling methods.

Chart 3. Response times of the binary overlay analysis service with different resampling methods

Bilinear resampling is the default method.  Here is how the response times for the other methods compared to bilinear averaged over the five map scales tested:

Raster Format

The storage format of the data can have a huge impact on performance.  For example, the response time of the binary overlay analysis service averaged over all map scales was 36% lower when the data was stored in the GeoTIFF format versus file geodatabase managed raster.  The Data Sources and Formats section of the  Image Management guide book recommends leaving the data in its original format unless it is in one of the slower-performing formats such as ASCII.  GeoTIFF with internal tiles is the recommended choice for reformatting because it provides fast access to the pixels for rectangular areas that cover only a subset of the entire file.

Pixel Type and Compression

The pixel type determines the precision of the values stored in the data and can have a huge impact on performance.  In general, integer types are faster than floating-point types, and lower-precision types are faster than higher-precision types.  Compression of imagery can potentially increase or reduce performance depending on the situation.   For more information about the affect of compression on file size refer to the Image Management guide book section on Data Sources and Formats.  To assess the impact of pixel type and compression on the performance of data stored on a local hard drive, I tested a group of image services configured to perform an extremely intensive overlay analysis on 15 raster datasets.  The services were configured identically except for the pixel and compression types of the analysis data.  The tests were run at the map scale corresponding to the pixel size of the data.

Charts 5 & 6. Avg. response time and storage size vs. compression type for an image service that performs a complex overlay analysis

The following table shows the percentage change in response times with the reformatted datasets versus the original double-precision floating-point dataset.

On-the-fly Projection

On-the-fly projection is a very important feature of the ArcGIS platform.  It has saved GIS users like me countless hours of work by eliminating the need to ensure that every dataset in a map is stored in same coordinate system.  However, in some cases this flexibility and convenience may be costly when ultra-fast performance is required.   The following chart shows one of those cases.

Chart 8. Performance of a dynamic image service configured to perform a weighted overlay analysis.

Chart 8 shows the performance of a service which performs a weighted overlay analysis  on six datasets in an Albers projection.  The upper graph shows the performance when the output of the service is set to Web Mercator (Auxiliary Sphere).  The lower graph shows the performance when the output of the service is the same coordinate system as the data.  Performance without reprojection to Web Mercator improved by an average of 45% over all map scales.  This is a fairly extreme example.  The performance cost of reprojection is related to the mathematical complexity of the input and output projections.  Equal-area projections such as Albers are mathematically complex compared to cylindrical projections such as Mercator.  I have not run tests to prove this, but I expect that the performance cost of reprojection between two cylindrical projections such as UTM and Web Mercator would be less costly than seen in this example, and that a simple projection from geographic coordinates to Web Mercator would be even less costly.

To avoid on-the-fly projection you must ensure that all of your data is in the same coordinate system, including the basemap.  Most of the basemap services currently available from Esri on ArcGIS Online are in Web Mercator (auxiliary sphere).  So if you are going to use one of those basemaps, you would have to convert your data to the same coordinate system.  This can be an acceptable solution for some situations, but keep in mind that it results in second-generation data with less positional accuracy than the original source data.  Alternatively, you can create your own basemap in the same coordinate system as your data, and either publish it to an ArcGIS Server site or upload it to ArcGIS Online as a hosted map service.  If you take this approach, I recommend caching the basemap using a custom tiling scheme with scale levels that match the pixel sizes of your data.

Request Size

Request size is directly related to the size of the map window in the application and is specified in the REST API as the number of rows and columns of pixels in the output image.  To measure its impact on performance, I ran a series of tests at different request sizes on the weighted overlay analysis service that I used for the reprojection-on-the-fly tests.  I measured the average response times for request sizes ranging from 400×400 to 2200×2200, increasing at 100 pixel increments (e.g. 500×500, 600×600, etc…).  All of the tests were run at the map scale of 1:113386, which corresponds to the 30 meter pixel size of the source raster datasets.

Chart 9. Average response in MP/s for different request sizes for the weighted overlay service.

Chart 10. Average response time at different request sizes for the weighted overlay service.

Chart 9 shows that the throughput for this service levels off at a request size of approximately 1000×1000 pixels to about 1.5 – 1.6 MP/s.  Chart 10 shows that request size has a linear impact on performance.  This service is capable of providing sub-second response times for requests up to about 1,440,000 pixels, or a request size of 1200×1200.

Summary

Raster analysis can involve many stages of data processing and analysis.  Complex on-the-fly processing chains can place heavy processing loads on a server and contribute to sluggish performance.  Huge performance improvements can be achieved in some cases by pre-processing the data into a more efficient format for resampling and on-the-fly processing.

For applications which use tiled basemap layers, the greatest performance improvements are likely to be achieved by aligning the pixel sizes of the data with the scales of the basemap tiling scheme.  The section “Map Scales and Source Data Resolution” describes the theory behind this approach and provides a table with recommended pixel sizes for applications which use basemaps with the ArcGIS Online/Bing Maps/Google Maps tiling scheme.  Alternatively, developers can build basemaps with custom tiling schemes to align with the existing pixel sizes of the analysis data.

Another way to significantly reduce the processing load on a server in some cases is to avoid on-the-fly projection of the analysis data.  This is accomplished by ensuring that the basemap and the analysis data are in the same coordinate system.  The performance impact of on-the-fly projection varies depending on the input and output coordinate systems and is discussed in the section titled “On-the-fly Projection”.

The file format, pixel type, and compression type of the analysis data can also have a huge impact on performance.  GeoTIFF with internal tiles is recommended for situations where it’s necessary to re-format the data from a slower format.  Lower-precision pixel types give better performance than higher-precision types.  Pixel compression has the potential to either increase or decrease performance depending on the how the data is stored and accessed by the server.  These topics are discussed in the sections titled “Raster Format” and “Pixel Type and Compression”.

Client applications can also play a role in dynamic image service performance.  Service response times are the lowest when applications specify nearest neighbor resampling, followed by bilinear resampling.  And there is a direct relationship between service performance and the size of the map window in an application.  These topics are discussed in the sections titled “Resampling Method” and “Request Size”.

Posted in Uncategorized | Tagged , , , | Leave a comment

Landsat Pivot Viewer

YouTube Preview Image

Click here for the live application.

The landsat program recently celebrated its 40th birthday. Since the launch of the first satellite in 1972 the program has amassed more than 3,000,000 images.

The USGS has published this archive as a single ArcGIS Image Service called LandsatLook. The prototype described in this posting uses this service in an HTML5 web mapping application.

While the map view is perfect to identify an area of interest, it is not so useful for sorting through hundreds or thousands of overlapping images. This prototypes uses a control developed by LobsterPot called the PivotViewer to present a sortable collection of imagery. Using the map and pivotviewer together, the presenter in the video above was able to quickly find and download a recent cloud-free image of London.

Contributed by Richie C.

Posted in Uncategorized | Tagged , , | 2 Comments

Thank You – Latest Innovations from APL session @EsriDevsummit

A huge thank you to all of you who attended our sessions yesterday at Esri Developer Summit in Palm Springs. There were many topics and technologies covered during the sessions and we hope that you found our work interesting, informative and worthwhile. For those who didn’t get a chance to attend our sessions, looks like one of the session got recorded so you might get a chance to catch up soon via online videos soon.

We were enthused and thrilled to have such huge attendance for both the sessions, you were a great group and your enthusiasm and comments helped make our time together both productive and fun.

Please join us at the Esri Labs area in the upcoming Esri User Conference this year at San Diego for more awesome stuff we are working on. Follow us on @esrilabs twitter account for more updates and details closer to the event.

Here are a few pictures and tweets about the sessions yesterday, Thanks and see you all soon!


Click here to view tweets on Storify

Posted in Uncategorized | Tagged , | Leave a comment

Proximity Map for Windows Store

YouTube Preview Image

This video demonstrates peer-to-peer collaboration in a mapping application developed using the ArcGIS Runtime SDK for Windows Store apps.  The app uses Windows 8 native proximity capability to establish an ad hoc Wi-Fi Direct connection that is fast, low latency, secure and doesn’t require any existing network.

The first part of the video shows extent synchronization or “shared navigation”.  As a map on one device is panned or zoomed, its extent is immediately broadcast to the other device.

The second part is somewhat more advanced.  We developed a map-friendly overlay, similar to a graphics layer, that could broadcast graphics as they are being digitized.  Graphics, or partial graphics, are serialized to JSON and transmitted to the other device.  Note: the ability to digitize to a graphics layers will be part of the next release of the ArcGIS Runtime (see Map.Editor.RequestShapeAsync()).

The last part of the video shows the ability to offset one screen from the other.  Any extents that are transmitted or received are automatically shifted to give the illusion that the screens are connected.  This would be useful if someone wanted to create a single map display from multiple devices.

We have shown that map collaboration with Windows 8 is viable even in the absence of a local network.  The proximity capability at Windows 8.0 is still rather limited, for example, only two devices can participate in a Wi-Fi Direct connection.  Later this year when Windows 8.1 is released, proximity will be expanded to support ad hoc networks with two or more nodes using either Wi-Fi or Bluetooth.

Cinematography by Mark D.

Posted in Uncategorized | Tagged , | Leave a comment

QR Map for Windows Store

YouTube Preview Image

This video demonstrations a proof-of-concept that uses a dynamic QR code to communicate map information from a presenter’s screen to mobile devices.  Whenever the presenter’s map changes, the map’s coordinate extent is compressed and encoded into a QR code using the ZXing.Net.

The same app used by presenter is also installed on the mobile device, in this instance, a Microsoft Surface.  Rather than a “presenter mode”, the app on the mobile device is placed in a “following” mode, this turns on the devices back facing camera and uses ZXing.net to scan for, and decode, coordinate extents.

We agree that by itself it may seem somewhat unremarkable.  We found it intriguing because the communication, while one way, was established without traditional connectivity such as Bluetooth or Wi-Fi.  This technique illustrated here may be useful in classroom settings, seminars and briefings.

References:

Video filmed by Mark D.

Posted in Uncategorized | Tagged , , , | 1 Comment

Geometric Network Configuration Manager

This post discusses the release of a sample utility called Geometric Network Configuration Manager.  In some instances when it is necessary to temporarily remove a geometric network, this tool can recreate the geometric network from a backed definition file.

The add-in can be downloaded from here.

The source code is available here.

Configuration Manager has a very long lineage.  More than a decade ago the Prototype Lab published Geodatabase Designer for documenting geodatabases and exchanging schema.  Designer is now obsolete but fortunately much of its capabilities are now incorporated into ArcMap or other tools such as ArcGIS Diagrammer and XRay for ArcCatalog.

However the one feature from Designer that has yet to be replicated is the ability to save and restore geometric networks.  This useful if loading large amounts data or performing a schema change like switching a feature class from a simple edge to a complex edge.

Prerequisites

Libraries Used

The following section will walkthrough the steps required to backup geometric network, remove it and then restore it.  The use case for this workflow could be for bulk data loading or transferring a geometric network from a test server to a production server.

Walkthrough

Following the successful installation of the add-in.  Display the Geometric Network Tools toolbar and click the first button to launch the main dialog.  Drag and drop a geometric network into the configuration manager window.

The complete definition of the geometric network will be loaded into the dialog.  The four tabs below the ribbon make it is possible to review and, in some cases, modify classes, weights and connectivity rules.

The geometric network definition is current stored in memory and should really be saved to a file so that it can backed-up or restored at a later time.  Click the save or Save As button to export the definition to a file with a esriGeoNet extension.

Recreating a geometric network is just a matter of loading an esriGeoNet file and clicking Export.  The application will prompt the user for the name and location of the exported geometric network.  If the geometric network already exists, it will be overwritten.

Known Issues:

  • The dialog that appears when the Export button is click may be hidden by the Configuration Manager window.  Either minimize or move the window to the side to continue with the export operation.
  • Add and removing of network classes and weights is currently not supported.
Posted in Uncategorized | Tagged , , | Leave a comment

Augmented Map for Windows Store

YouTube Preview Image

The Applications Prototype Lab is proud to present a preview of “augmented map”, a windows store based augmented reality app.  The purpose of this app is to provide an intuitive alternative to image discovery and interrogation.

The app is running on a Windows Surface and the only other peripheral is a block marker printed on a white piece of paper.  This marker orients the map displayed in the video feed of the augmented reality app.

In the demonstration video, the user starts by zooming to, and requesting imagery for, New Zealand’s capital.  LandSat preview images are downloaded and placed on the map.  Because there are potentially hundreds or thousands of satellite images, the user will use augmented reality to sort through the collection.  The demonstrator first chooses to sorted and stack the imagery based on the date of capture with the newest images located at the top.  Detailed information such as the sun’s position and percentage cloud cover is available by tapping on an image.  To analyze older images, the user applies a filter to progressively hide images from top to bottom.  Lastly, the user sorts images based on percentage cloud so that the clearest images are at the top of the stack.

This project, to a large extent, is derived from the Silverlight-based app completed almost three years ago.

Data used:

Technology used:

Posted in Uncategorized | Tagged , | Leave a comment

Distributive Flow Maps – More Raster, More Faster

NOTE:  The Distributive Flow Lines tool was updated  Jan 10, 2014 .  If you are using ArcGIS 10.1sp1 or 10.2 please download the latest version.

Based on comments and feedback from the first version of the Flow Map tool and blog post we gave it a second look and have released an update.  As you can see in the screen capture above, the output is similar to the original tool but we have switched to an all raster approach which provides more control over the lines and reduces processing time.

A distributive flow map made by Charles Joseph Minard showing coal exports from England in 1864.

For convenience, I have placed the introductory text from Brad Simantel’s original post below to provide a concise introduction to the history and use of Flow Maps.

“Flow maps are used to show the movement of goods or people from one place to another. These maps use lines to symbolize the movement, often varied in width to represent the quantity of the flow, and fall into one of three categories: radial, network, and distributive. Radial flow maps are used to show relationships between one source and many destinations. If there are more than a handful of destinations and you still want to show the quantity of flow, however, the lines overlap too much to discern individual values. Network flow maps are used to show the quantity of flow over some existing network — transportation and communication networks being the most common. Distributive flow maps are similar to radial flow maps, but rather than having individual lines from the source to each destination, lines are joined together, only forking once they get close to their destinations”.

If you would like to try it out as you read this post, you can access the new  Distributive Flow Lines tool (DFLT) on ArcGIS Online. If you do not have data handy, the British Coal export data, used to create the first map in this post, is also available for download on ArcGIS Online.

Tip: The DFLT requires the Distributed Quantity Field to be of type Integer.  If you use the British Coal sample data a new integer field will need to be added and calculated based on the existing CoalTonnage field.  The current field represents thousands of tons so multiplying by 10 and rounding to the nearest integer would represent hundreds of tons rather than thousands.

Like the first Flow Map tool, the Spatial Analyst extension is required to use the new DFLT.   In fact, the new tool is completely raster based until the end when the flow line feature class is created as output.

The DFLT will accept polygon or point features as input for the Source and Destinations, however, point features are recommended.   If polygons feature classes are used as input for either Source or Destinations they will be converted to points for internal use but these intermediate point datasets are not saved as part of the tool output.  After the Destination feature class is selected the user will need to select an integer field from this dataset representing the quantity that will flow to the destinations from the source.

The DFLT provides an option for the user to specify a feature class to use as impassable features.  These features will be buffered by 1.4 times the Cell Size parameter.  The area inside this buffer will be NoData and will be used as a processing mask within the tool.  A second, slightly larger buffer, 3 times the Cell Size parameter, is also created. The area between the inside buffer and this larger buffer is given a very high cost so Flow Direction will not calculate routes that go into the NoData area (most of the time).

The second optional parameter is a Impedance feature class.  These polygons represent  features in your map that you would like flow lines to avoid as much as possible.  How much these features are avoided is controlled by the Impedance Weight slider.  When impedance features are specified, they are given a high cost; but may still be crossed in extreme cases where there is no alternative or where they can be crossed in a narrow location as shown in the example below.

Tip: One common effect of the DFLT is that it will often skirt Impedance features too close.  This can be avoided by buffering the Foreground features and then using this new feature class as the Foreground features in the tool.

Since the primary use case for the DFLT is to create pleasing flow lines, the user will most likely run it several times using different values for the weight sliders to achieve a good starting point for the effect they are looking for.  Using an all raster approach makes this process a bit easier and faster.  In raster processing, cell size has a major effect on processing time. The DFLT calculates a default cell that size works well for the final product.  In most cases it will not be necessary to use a smaller cell size than the default. We recommend using a much larger cell size during initial iterations to reduce processing time until you start getting results close to what you want.

Tip: It is a good idea to make a note of the default cell size for final processing.  For initial runs of the tool,  multiplying the default cell size by 5 or 10  works well to get a feel for how the various weights and optional parameters effect the output flow lines.  More drastic changes in cell size may have a significant effect on how the lines are routed around Impassable and Foreground features.

Tip: If you use the post 10.1 sp1 version of the tool you will notice the tool dialog has been simplified.  Source Weight and Destination Weight sliders have been replaced by a single slider to control how close to the Source or Destinations the flow lines branch.  Also processing extent options have been reduced to “Same as Display” and “Same as Destinations plus Source.”  This was changed to prevent most of the cases where no solution is possible.

 

Cell size is so important in the DFLT because the tool calculates Euclidean distances from source and from all of the destination features. These two distance rasters are then Sliced into an equal number of discrete cost classes.  The number of classes is determined by dividing the maximum processing extent dimension by the cell size.  This acts to normalize the effect of the Source and Destination cost rasters and provide a basis for the cost of the Impedance raster.  By normalizing these costs the tool provides the user more control over the shape of the final flow lines through the Source and Destination Weight sliders.

To understand how cell size affects the total cost surface it helps to see a few examples.  In the following ArcScene screenshots we are using a Source location in the center of the US and the destination features are the centroids of a several countries in Africa. The CostDistance surface is shown floating over the “flat Earth” to provide a means of comparing the effects of the weight parameters and cell size. In each example the perspective and Z exaggeration are the same.

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =8 Cell Size = 100,000 meters

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =1 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 1 Foreground Weight =10 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 30,000 meters

In this last example, it was necessary to reduce the elevation exaggeration and to zoom out so the cost surface would display in a reasonable way.  The only difference between the two results is that the cell size has been reduced from 100,000 meters to just 30,000 meters.  The effect is that the number of “cost slices” is larger and so the CostDistance “elevation” is increased.

As previously stated, the new DFLT processes the Impassable features separate from Impedance features.  Impedance features are treated much the same as Impassable features in the first tool.  They are features the user would like the flow lines to avoid if possible but the flow lines will cross them if necessary to reach a destination location.  Impassable features represent a hard barrier in the cost surface and can be used to control the shape of the flow lines as in the example below.

Flow lines representing distribution of Coal by ship are crossing Central America regardless of Impedance weight

With the red barrier added flow lines now go around the tip of South America

As with the previous Flow Map tool, the output of this new tool will usually be the starting point to produce a final flow map with smoother more organic looking flow lines.  Once the lines are roughly where you want them the next step is to use Graduated Symbols to display the Distributed quantity.  The “GRID_CODE” field will contain values based on the input Distributed quantity field.  The values in this field will be equal to the Distributed quantity field for the smallest lines nearest to the destinations.  Each time a tributary flow line meets another line the combined line closer to the Source feature will have a GRID_CODE value equal to the sum of previous two tributaries. Temporarily placing labels on the lines and destination points makes this clear. In the examples above the lines are represented using 10 classes of line weights ranging from 1 – 10. Using rounded joints and ends on the lines also improves the output. After setting up graduated symbols many users may also prefer to run the Smooth Line tool on the flow lines. This will also add some curve to the flow lines which is often desirable. A good starting point is to use a tolerance value approximately four times the Cell Size used to create the flow lines. It is also recommended to select the PEAK algorithm and FIXED_CLOSED_ENDPOINT options. Even after these steps it may be necessary to make manual adjustments to the shape of the line.

Installation instructions:

After downloading and unzipping the Distributive Flow Lines tool you should see the following in the ArcMap catalog window.

Expanding python toolbox will reveal one script tool as shown below:

By default, python toolboxes do not display the “pyt” file extension. To display this and other known extensions in ArcCatalog (or the catalog window in ArcMap) check the following:

This dialog is accessible in ArcCatalog by clicking Customize > ArcCatalog Options.  Unchecking the highlighted option will show the following:

We hope you find the new tool useful and the tips above help get you started experimenting with the tool.  We look forward to you comments.

NOTE:  The Distributive Flow Lines tool was updated  Oct 22, 2013 to address a backwards compatibility issue.  If you are using ArcGIS 10.1 please download the latest version  here.

NOTE:  The Distributive Flow Lines tool was updated  Jan 10, 2014.  If you are using ArcGIS 10.1sp1 or 10.2 please download the latest version.

Contributed by Bob Gerlt.

Posted in Uncategorized | Tagged , , | 24 Comments

iPad display wall showcases Urban Observatory project

YouTube Preview Image

After helping with the Urban Observatory exhibit at the User Conference, 2013 we were inspired to bring the same concept to mobile devices. This would provide for the same visualization and comparison of different themes across different cities around the world.

The Urban Observatory works in a grid format. All the cities to be compared would be in the bottom most row. Each had the same theme such as housing density, traffic etc,. and the map scale in each device was synchronized. As devices are stacked horizontally they would load a user defined city with the same theme and the same map scale as adjoining iPads.  Likewise, vertically stacked devices would display the same city but with a different theme as defined in that row.

We replicated this concept by using multiple iPads. As soon as we put down a new iPad they would realize their position with respect to the previous iPad and would pull in the correct data. All the devices synchronized the map scale and only the same city iPads synchronized the map extent. You can change the city on one device and all the other ones would update accordingly to show a different city. We also brought in the concept of using the iPhone as a remote control to manage the themes and the cities on all the iPads. The top row of iPads could also serve additional information about the cities. We had them show the fly by video of each city shown in the iPad below.

Technical Details :

This application was written using ArcGIS Runtime SDK for iOS.

All the transactions happened using the Gamekit framework for iOS. This helped us automatically setup a sessionID to which other iPads would connect to. Using CoreLocation, we also knew their relative positions and we could connect a lot of these devices together and they automatically knew what data to pull from ArcGIS Online. They established a peer to peer networking using the bluetooth or the internet when available.

The original content for the Urban Observatory, its goals, objectives, sponsors and contributors, could be accessed at http://www.urbanobservatory.org.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Where Did Water Flow on Mars? Displaying 2D Maps on a 3D Planet

In the 2008 article ‘Where Did Water Flow on Mars? Modeling Mars’ surface in search of ancient rivers and oceans’ Witold Fraczek demonstrated how GIS can furnish support for the theory that at some time in the past, water did flow on the Martian surface.  By utilizing NASA’s available Martian DEM and other supporting data layers, a hydrologic network was created by running a series of hydro functions. For this analysis, a selected section of the Martian DEM was treated in exactly the same way that a DEM from Earth would have been handled. A series of cylindrical projections were then exported from ArcMap and wrapped around 3D spheres to represent Mars. These 3D planet models were then imported into CityEngine as Collada where small selectable domes were added to represent the many probes that have successfully landed on Mars. Finally this model was exported as a 3D Web Scene and uploaded to ArcGIS online to easily share with the public. Since 3D Web Scenes are based on WebGL technology, no plug-in is required for most browsers.

Read more about how GIS helped to derive the Martian Ocean: http://www.esri.com/news/arcuser/0408/mars.html

Exporting to a 3D Web Scene is currently available for CityEngine, ArcGlobe and ArcScene. 3D scenes and the ability to publish directly on the web is revolutionizing the way we share, collaborate, and communicate analysis results or design proposals with decision makers or the public. After all, our world is in 3D.

ArcMap is used to analyze the digital terrain model for Mars’ hydrological network.

The cylindrical projection is then wrapped around a 3D sphere and imported into CityEngine as Collada.

CityEngine was then used to export the layers to a 3D Web Scene which is now hosted on ArcGIS Online.

You can view the Web Application here.

Credits: Dr. Witold Fraczek (APL), Brooks Patrick (City Engine)

Posted in Uncategorized | Tagged , , , | Leave a comment