Caching Thumb

We got a question the other day on Ask a Cartographer about how to cache map services with several computers. First, we want to explain why you would even want to do this. If you will be caching a map that covers a large extent and needs to be cached at large scales, it is possible that the caching job could take days or even weeks to process on one computer. For instance the World Topographic Map on ArcGIS Online. It would take about 8-10 weeks to cache completely on one computer. By spreading these large caching jobs across many computers, it will take less time to complete.

The basic idea is to replicate the caching environment on a second computer and subdivide the caching job by either cache scale or geographically. To do this, you need to make sure that ArcGIS Server is running on each additional computer and that you have defined on those computers an identical map service that will be the basis for caching.

We have found through practice there are a number of best practices to follow:

  1. Expect to copy the cache files into a master service's cache to test and verify the complete product.
  2. Run one last caching process on this master service after all the other files have been copied with the intent of caching any empty tiles to ensure all the tiles are present.
  3. Managing disk resources is important--you do not want to have the other computers caching into the same cache location--that will make that location become file I/O-bound, meaning that all processes will be slowed down while competing and waiting to write to that disk.
  4. Similarly, all the data used by the MXD that is being cached should also be replicated for each computer that is caching. Sometimes that is not possible, i.e., some datasets are quite large and it’s more appropriate to consider optimizing how many users or several very active users access that data. DBAs earn their living by solving problems like this.
  5. Another way to avoid slowing down the caching process is to have the data for the MXD on a different physical drive than the \arcgisserver folder where the cache files will be stored. The reason is the same as for #3 above, which is that multiple processes accessing, that is, means reading or writing (for whatever reasons), the same disk are in competition and only one process will ultimately be run at a time.
  6. In most cases, when caching maps, use a formula of 5/4 instances for caching per CPU on each computer. So a two CPU computer would have 2 instances (or ArcSOC.exe processes), while an 8 CPU computer could use 9 or 10 instances.  When we used more than this, we found that we became "CPU-bound". It actually took longer to complete caching a map with three instances per CPU than with two. We have also found that for that last caching run on the master service, using one instance (regardless of the number of CPUs) is the most stable way to guarantee that a complete cache is produced. By stable, we mean that the process actually completes without crashing or failing to produce all tiles.
  7. One other tip for stability is to not have any shared folders on the disks you are caching to or where your map data located—this was particularly an issue with versions of Windows prior to Windows 7. On this issue, problems resulted when others accessed data, i.e., competed for disk resources, while caching was in progress. This sometimes caused the caching process to crash.
  8. At the outset, check your caches within the \arcgiscache folder to verify that you are getting what you expect from each computer that is caching. This allows you to immediately detect caching errors and correct them, rather than wait until the caching job is done, which can sometimes take a bit of time.
  9. To copy the cached files between computers, use a software program. Like SecureCopy or RichCopy (and there are likely others), that is designed to move large numbers of files. We have both SecureCopy and RichCopy in-house, and I cannot advocate one over the other – both are sufficient for the task. The important issue is that the Windows Operating system is not capable of moving a very large number of files around at once through the user interface.

All of these suggestions apply to ArcGIS 9.3.1. or an exploded cache (versus a managed cache) in ArcGIS 10. We have not tried this yet with a managed cache in ArcGIS 10.

A final note – this is not a precise science and it took an accumulated set of experiences for us to learn all of this. You should expect to have some trials (plural) and errors (plural). Your expectations should be that using three computers will cut the caching time in half rather than a third, because you will still need to allow extra time to set up and maintain the additional computers, and copy the files from those extra computers to the master cache.