Tag Archives: Performance
Boosting performance through pre-computing
At Esri conferences, we frequently offer workshops on designing faster GIS web services. Perhaps the most important piece of advice that we give in these workshops is to pre-compute when possible. In this post, we’ll explain what we mean by “pre-computing” and offer some simple ways you can speed up your ArcGIS Server services by following this principle. Continue reading
EA SIG 2011: Real-time Answers To Key Enterprise Issues
Based on last year’s very successful Enterprise Architecture SIG, we are keeping the round table “Birds of a Feather” format for facilitating conversations at the Esri International User Conference EA SIG this year. We are modifying the structure slightly to allow for time to summarize key findings from each group attending, providing you real-time answers. Table topics this year include:
- Virtualization / Cloud Computing
- Security
- Enterprise Integration
- Performance & Scalability
- EA Methodologies
- Application Development Patterns
- Enterprise Workflows – New for 2011
- Mobile – New for 2011
The EA SIG will be bright and early again with a full breakfast! Thanks again to IBM for sponsorship of this SIG. See you there!
Who: Open to all UC Attendees
Date/Time: Wed, Jul 13, 7:30AM – 8:15AM
Location: Room 17A – San Diego Convention Center (The wrong room # is listed in the printed agenda)
High performance web apps with big datasets as feature layers
Dealing with big datasets is a frequent topic of discussion among web mapping app developers. The prevalence of huge, crowdsourced datasets ensures that big data will continue to be a common point of discussion when you get more than two web mapping geeks in the same room. The traditional approaches of generating a raster tile cache or using a dynamic map service work, but if you want any interactivity with the underlying data, it requires a server round trip. This post will discuss some of the key concepts behind building a web app that provides an extensive interactive experience while maintaining responsiveness and performance.
The main factor causing big data to be a tricky topic is that browsers cannot elegantly handle upwards of a few thousand features. Send more than a few thousand points, or a few hundred polygons to a browser and you can watch it grind to a halt, especially if you’re running a legacy version of Internet Explorer. This is a problem because slow performance is a death knell for web apps. If someone using your web app has to wait ten seconds, twenty seconds, or longer for five megs of JSON data to be retrieved and displayed, there’s a good chance they’ll just give up and move on. What good is a web app that no one uses?
At some point, you cross a threshold where it’s not practical to send all of your data across the wire and let the client sort it out. That threshold is an ambiguous one (and was explored in a previous post), but you know when you hit it. If you’re a JavaScript developer, how should you deal with big data? If you’ve read my recent posts, this answer will not surprise you: Feature Layers.
Building on the concepts discussed in the previous feature layer posts (generalizing features and vector tiling) and client side graphics performance post, I’ve put together an application that demonstrates how to use Feature Layers to set up custom scale dependencies and follow best practices to ensure performance does not suffer when querying, retrieving and displaying large datasets in an ArcGIS API for JavaScript application: High Performance Maps with Feature Layers.
The goal was to display appropriate US boundaries (states, counties or census block groups) for the current map scale. For example, trying to display census block groups for the whole US would overwhelm the browser with features. This app makes sure that only the states appear at the small scales, counties at the medium scales, and census block groups at the large scales. This ensures that a manageable number of features are always being transferred and displayed.
The app is simple, but the concepts can be applied to just about any web mapping app that has to deal with big data. The total size of the data used by the app is around a couple of hundred megabytes. This is accomplished through custom scale dependencies. This is as simple as creating a few feature layers, listening for their onLoad event and setting their minScale and maxScale properties. Here’s a simple code sample showing how this is done:
var fl = new esri.layers.FeatureLayer(url, options);
dojo.connect(fl, ‘onLoad’, function() {
fl.minScale = minScale;
fl.maxScale = maxScale;
});
The app includes a couple of other bells and whistles that demonstrate some of tools that are available when you have your features client side. The first is a rich Popup that displays feature attributes, as well as an option to zoom to the feature when you touch a graphic. The second is the ability to filter features based on population using a slider. Neither of these would perform nearly as well if features weren’t available client side.
The final point to drive home is that the app’s performance is kept snappy because only the required data is being sent to the client. For geometries, that means using maxAllowableOffset to generalize geometries on the server before they’re sent to the client (see my previous post on maxAllowableOffset and generalizing features on the fly for more info). For attributes, only the fields required by the Popup are sent across the wire.
Contributed by Derek Swingley of the ArcGIS API for JavaScript development team
Determining limits for map graphics
We’ve recently been talking about some of the benefits of Feature Layers. Since Feature Layers are a subclass of GraphicsLayer, we thought it was appropriate to spend a little time talking more about map graphics, limits on how many can be added to a map, and how to find the appropriate limit for your app/environment.
“How many graphics can I put on my map?” or “How many points can I put on a map?” are questions that continue to be asked. Probably because we keep saying “it depends.” While technically correct, “it depends” is not helpful on its own. So, with the lofty goal of providing some actionable guidelines, I’d like to discuss two scenarios: point graphics and polygon graphics. To aid in this discussion, I’ve put together a simple app for each scenario: Display an Arbitrary Number of Points and Incrementally Add U.S. County Polygons To a Map.
The objective of the point graphics app is to see how many graphics you can add to a map before performance starts to suffer. Since this is dependent on many factors (your browser, your hardware, your definition of “suffer”), it’s probably best to let you see for yourself how a map performs with 100, 1,000 and even 10,000 graphics. I wouldn’t try 10k in IE, but Chrome and even Firefox hold up well (at least on my machine, YMMV).
There is also an option to select symbol type. Not all point symbols are equal in terms of performance so it’s important to realize that 500 picture marker symbols might result in a more sluggish app than one with 1,000 simple marker symbols.
But wait, just because you can put 3,000 points on your map and not see a performance penalty, does that mean you should? My answer, nine times out of ten, is no. Performance is one thing but user experience is another. Does putting a map with thousands of point graphics on a 15” screen provide a meaningful user experience? Again, I would say no. Outside of looking for a general spatial distribution pattern, I don’t think you’re helping a user discover much or answer any particular question by showing thousands of point graphics at one time. If you find yourself in this situation, it’s a good time to consider using a raster surface (like a heatmap), filtering your data through queries, or maybe clustering.
The story for polygon graphics is a bit more complex. Not only do you need to worry about the number of polygons, but the number of polygon vertices is also an important consideration. More detailed geometries take longer to transfer to the client, and once they make it to the client, they require more resources to display.
Like the point graphic app, the polygon app linked above lets you add any number of polygons to a map (but you probably should keep it under 1000), and it also allows you to specify whether or not maxAllowableOffset should be used. If maxAllowableOffset is a new term for you, please read my previous post to catch up.
Let’s look at a couple of reasons to always use maxAllowableOffset. The image below shows geometries for some states in the Northeastern United States:
The geometries on the left were fetched using a query that set maxAllowableOffset appropriately. The geometries on the right are not generalized. The generalized geometries look nearly identical to the non-generalized geometries (maybe even better according to some) but they are less than a quarter of the size. That’s the key takeaway and why we use maxAllowableOffset: transfer the minimum level of detail required for effective display.
One more example:
I think this example really drives the point home. Using maxAllowableOffset, you’re transferring ~44KB to show the county boundaries for Mississippi. Without maxAllowableOffset, you push over 3MB across the wire. If you’re setting maxAllowableOffset correctly, you’re not likely to be sending more than 100KB (and usually a lot less) per query to the client regardless of the level detail present in your geometries.
The total number of polygon graphics on a map is something that you should experiment with but I would recommend to keep it below a few hundred.
We still don’t have a hard and fast rule for the number of graphics on a map (and probably never will), but hopefully this post along with the accompanying apps will make it easier for you to find your own answer.
Contributed by Derek Swingley of the ArcGIS API for JavaScript development team
Multiprocess Geocoding
A common request from the user community is to parallelize geoprocessing tasks to take advantage of available machine CPU resources. A search of the forums will bring up topics like Python’s multiprocessing module and the PyCuda module, both feasible approaches but with their own issues. The multiprocessing module for example requires that data passed to spawned processes be “pickleable” – basically restricting the feasible arguments to strings and numbers and structures built from them – eliminating ArcPy classes like FeatureSets and so on. PyCuda is a whole new level of installation and development challenge too.
Before exploring multiprocessing and GPU programming, don’t forget Python’s subprocess module. This supports multiple independent processes but does not require thread safety – just a simple approach of a master script and a worker script.
Batch geocoding is an “embarrassingly parallel” problem, the input may be broken up into as many jobs as desired. The sample script tool linked here lets you leverage all available CPU cores on your local machine, or all available Server instances on a remote server. Splitting the work into equal sized chunks is a trivial exercise using queries for ranges of ObjectID (which is a very fast query to execute), so workers get started immediately. The biggest overhead delay is recombining the various outputs, but on a well specified machine you should see worthwhile gains nevertheless.
Download the tool even if you’re not a geocoder, you may be able to leverage the logic to parallelize other geoprocessing jobs – and if you do, don’t forget to share your own tools!
Content for this post provided by Bruce H.
Map caching tips and best practices with ArcGIS Server 10
Today we’d like to share a list of techniques, best practices, and little-known facts that can improve your experience building map caches with ArcGIS Server 10.
Configuring local cache directories on the server
ArcGIS Server 10 includes an option to speed performance when caching with multiple SOC machines. When you open the Caching tab of the Service Properties dialog box and check Use local cache directory when generating tiles on the server, each “bundle” of tiles is written into a local directory before being copied into the main, shared cache directory. This is faster than trying to write all the tiles into the main cache directory at once.
Although the local cache directory option is a performance boon and should always be used in multiple-machine deployments, there are some tips you should follow to get the most out of it:
- Uncheck this option if you know you’ll be building your cache with just one SOC machine.
- Make sure you have enough space available in the local cache directories. It’s recommended that you have at least 0.5 GB of available space in this location for each running map service instance (ArcSOC.exe) dedicated to caching. More space may be required for JPEG caches that use a high compression quality, or detailed PNG caches with high bit depth.
- If you use this option and your caching job fails, clean out the local directories before you begin caching again. The default location of the local cache directory is the temp folder for the SOC account unless you define a system variable on the computer named ARCTMPDIR whose value is some other folder.
Using MSDs and MSD-based antialiasing
The faster your map draws, the faster you’ll see tiles created. In ArcMap, use the Analyze Map button on the Map Service Publishing toolbar to identify performance bottlenecks. Fix all of the errors and address as many of the warnings and informational messages as you can. You should then use the toolbar to create a map service definition (MSD) and publish your map service using the MSD.
If you have a layer in the map that is causing an error in the analysis, consider isolating that layer in its own map service so that you can publish the remaining layers in an MSD-based map service.
MSD-based antialiasing is much faster than the antialiasing applied by the caching tools. If you intend on building your caches with antialiasing, enable the antialiasing in the MSD using the Map Service Publishing Options button of the Map Service Publishing toolbar. Then when you set up your tiling scheme, leave the antialiasing checkbox (Smooth line and label edges) unchecked. You’ll still see antialiasing in the cached images because it was applied in the MSD.
Troubleshooting cache failures
Sometimes SOC processes can crash while tiles are being built. These crashes can have various possible causes which are often difficult to detect. Below are some tips for preventing and dealing with crashes.
- Before you start caching, make sure that adequate space is available in your local cache directories and your main cache directory.
- Check if any of the data locations used in your maps, such as ArcSDE connections, have become unavailable during the caching job. Sometimes data located on different drives may fail to be retrieved due to network problems. When this happens a crash can occur, or blank tiles may be generated.
- Attempt to re-run the cache at just the failed extents. These extents are reported in the geoprocessing results messages for the Manage Map Server Cache Tiles tool. There’s a clever utility on the Code Gallery that can parse the failed extents and make a feature class from them. You can then feed that feature class right back into the Manage Map Server Cache Tiles tool to constrain tile creation to just the failed areas. If the caching job happens to work the second time, you will have a completed cache.
- If the failed extents do not get created successfully after a second try, open your source map document and examine the affected extents to make sure they draw properly in ArcMap (if a draw fails in ArcMap, it will certainly fail while caching).
- Using a copy of your original map document, make a second service that has no cache (draws dynamically). Then set your server log level to Info:Detailed and make several dynamic draw requests at the failed extents. You can then scrutinize the log file to find drawing and connection problems. You can read more about this technique in Troubleshooting map service performance with log files.
- If other techniques fail to find the problem, try to create the failed extents using a second map service. If the tiles generate successfully, use the Import Map Server Cache tool to import them into your original cache.
Converting the storage format of a specific data extent
Many of you have asked for better control in the Convert Map Cache Storage Format tool, so that you can convert from compact to exploded (or vice versa) for only a specific extent of cache data. You can actually do this already using the Export Map Server Cache tool. Export Map Server Cache lets you choose a specific area or scale from a given cache map service and convert its storage format, while at the same time exporting the tiles to a location you define.
Contributed by Garima Tiwari and Sterling Quinn of the ArcGIS Server development team
Improving batch geocoding performance
Geocoding performance has always been a top priority when working with large volumes of addresses, and it’s a primary consideration when designing and implementing a geocoding workflow. There are several factors that contribute to geocoding performance, I won’t cover all of these in this article but, let’s look at some simple, high value changes we can make to our locators and workflow to improve performance.
Increasing the run time memory limit:
ArcGIS Locators can be configured to use more system memory. Using system memory (RAM) to work with locators is much faster than reading a locator from disk. ArcGIS supports use of 2GB memory on a 32 bit system and 3GB of memory on a 64 bit system. By default ArcGIS locators are configured to use 512MB of system memory. The default values were chosen to ensure the greatest possible system coverage, without exceeding system limits and possibly destabilizing a system. By increasing the memory allocation for locator use above 512MB we see good performance gains.
ArcGIS locators support a “Run Time Memory Limit” parameter. For large geocoding jobs working against a large locator such as US streets, or US Composite, we’ll want to set this limit near the capabilities of our system. With ArcGIS 10, geocoding performance continues improving as the cache grows beyond 2GB.
Follow the steps below to create or edit an existing Run Time Memory Limit parameter:
- Open the *.loc file in a text editor (note this is the .loc file and not the .loc.xml file).
- Look for the line that starts with RuntimeMemoryLimit =
- If the value does not exist, create one and change the value to represent the number of bytes of system memory you want to allocate for the locator:
RuntimeMemoryLimit = 2048000000 - Save the locator.
- Use the locator and compare performance and system memory use.
You can play with this value to find something that is appropriate for your system. In most cases you will see optimum performance near 2GB of memory. If you’ve allocated too much memory, the system may become unstable or ArcGIS may crash. Reduce the memory allocation to rectify.
Presort reference data for optimal performance:
It’s always much faster to find something when it is organized than when it is chaotic. Geocoding is no different. Sorted addresses can be geocoded faster than unsorted data. Sorted addresses tax the system less, hard disks don’t have to work so hard and the geocoding engine makes less frequent memory allocations.
A typical approach for sorting data would be to sort by the highest level of geography first and then to sort on smaller levels of geography. With a U.S. address for example, you should sort the address data by State, followed by City, and Postal Code. If your data isn’t already sorted, you can sort it using the ArcGIS Sort (Data Management) tool or you can configure a locator to sort data as it’s read into the geocoding process.
Configuring a locator for sorting is similar to changing the run time memory limit, except you need to configure two parameters instead of one. The first parameter specifies the fields to sort by. These fields are the input fields defined by the locator. The second parameter is the number of records to sort in each process. You could sort the entire table, but that would be very memory and CPU intensive so instead you can choose to sort the first n number of records. Defining a higher value can help produce higher performance but again, taxes system resources. A typical value supporting good performance might be 100000 records.
Follow the steps below to create or edit an existing Batch Presort parameter:
- Open the *.loc file in a text editor (note this is the .loc file and not the .loc.xml file) .
- Look for the line that starts with BatchPresortInputs = State
- If the value does not exist, create one and assign the first locator input field to sort by. Repeat the entry for additional fields
BatchPresortInputs = State
BatchPresortInputs = City
BatchPresortInputs = Zip - Look for the line that starts with BatchPresortCacheSize =
- If the value does not exist, create one and assign the first locator input field to sort by. Repeat the entry for additional fields
BatchPresortCacheSize = 100000 - Save the locator.
- Use the locator and compare performance and system memory use.
In this example I geocoded 10,000 unsorted U.S. addresses with a U.S. streets locator and the preceding enhancements. Performance increased from 350,000 recs/hr to 614,000 recs/hr. You can experiment with the batch presort cache size to find optimal performance for your locators and data.
In the future we’ll look at more ways to increase geocoding performance, but these techniques are a good start and should result in some excellent results.
Jeff
Business Analyst 10 SP1 Released
By Lucy Guerra
The Business Analyst team is pleased to announce that Service Pack 1 for Business Analyst 10 Desktop is now available. This service pack contains several performance improvements and maintenance fixes. We recommend that customers using Business Analyst 10 or Business Analyst 10 Premium download and install this service pack at their earliest convenience to ensure the highest-quality experience when working with Business Analyst.
Download the Service Pack from the Esri Resource Center or install it automatically by using the Check for Business Analyst Updates utility from your Start > Programs > ArcGIS > Business Analyst location…
Increase Productivity with ArcGIS Seminar Material
As the ArcGIS 10 seminars are wrapping up, I wanted to make sure that everyone knows that the content from the seminars is available online. Some really good information in the PDFs to help you be more productive with ArcGIS 10. You can also watch the ArcGIS 10 seminar on Esri Video.
