Tag Archives: raster data
Is my raster catalog managed or unmanaged?
With the arrival of the mosaic dataset, raster catalogs have been quite neglected in the raster blog. There are probably still many users that use raster catalogs, so here is a blog for those that cheer use raster catalogs.
Recently someone asked us, I can’t figure out if I created a managed or unmanaged raster catalog. How can I tell which one I created?
Share your source raster within an image service – Part 1
Image services are a great way to allow others to use your service layers within their GIS projects. Did you know you can actually allow users to download your source files? By default, this option already enabled when you publish a mosaic dataset. Continue reading
Georeferencing – Should I Update or Rectify?
Once you have finished choosing and evaluating your links, you are ready to persist your results. There are 2 options to choose from: Update Georeferencing and Rectify.
Update Georeferencing
- Only updates your files- no redundancy created
- Update are almost immediate
Rectify
- A new raster dataset is created and the original file is left unaltered
- Choose many options: such as file format, cell size, resampling technique, and NoData value.
- This is the only option for some files: such as GRID and other files that cannot be updated (e.g. read-only files)
Depending on your workflow, both Update Georeferencing and Rectify have their advantages.
Overviews and pyramids: Part 2 of 2, Doesn’t my mosaic dataset use both?
This is part 2 in a 2-part blog. Part 1 provided an overview of pyramids and overviews. Part 2 provides you with some guidance on generating them when creating a mosaic dataset.
When you are viewing a mosaic dataset and you’re zoomed in to a small number of images, the pyramids are typically used to generate the image you’re seeing. When you zoom out and your view contains many images—to see the many images, overviews are typically being displayed.
How to know what you’re seeing?
If you want to know what is being displayed, open the attribute table. Then right-click the Image layer and click Selection > Select Visible Rasters. This will select the item in the attribute table (and will highlight the footprint if displayed).
General design
If the raster data contained within the mosaic dataset has pyramids, then generally fewer overviews are needed. In the diagram below there are three source images. They each have a pixel size of 0.5. The pyramids are built for each source image—in this case only two levels. The pyramids are resampled by a factor of 2 (this is the default). The overviews are then generated where the pyramids end, but using a resampling factor of 3 (this is the default). The overviews are a mosaic of the images and are limited by a tile size. In this example, there are only 2 overviews at the first level, even though there were 3 images. Eventually, there is only one overview at the top level.
You can have various combinations. For example the source images above may not have pyramids. In that case the overviews will start with a pixel size of 1.5 (1:5670); which is a factor of 3, and it will continue until it reaches and appropriate overview size to display the entire image quickly.
If pyramids have been generated, then fewer overviews will be needed and it will take less time to generate the overviews.
Guidelines
Generally, overviews will perform faster than displaying the pyramids for each raster within the mosaic dataset. This is because they are often larger in extent than the source files, so fewer files are opened. You may consider building overviews instead of raster pyramids when using
- Preprocessed tiled imagery, such as orthophoto quads
- Edge-joined (non-overlapping) imagery, which will not be effected by changing mosaic methods
- Imagery that will be processed on-the-fly, when the parameters and mosaicking method will not be changed
When building more complex mosaic datasets, especially where you will be taking advantage of the mosaic methods and on-the-fly processing, then it can be advantageous to build pyramids on the source rasters and to build overviews only where they are needed, such as when
- The raster datasets are larger than 5000 columns
- The rasters overlap and the mosaic methods will be used to control the order
- On-the-fly processing will occur on the source rasters at all scales
- Images are not static preprocessed rasters
When building pyramids or overviews, use the appropriate resampling method and compression
- For imagery, bilinear resampling is recommended and a JPEG compression
- For discrete data, nearest neighbor resampling is recommended and a LZW compression
Overviews are generated using the defaults of the mosaic dataset, such as the default mosaic method. If you have multiple overlapping scenes over time, then you won’t see the different dates until you’ve zoomed into a level where the source images (or their pyramids) are displayed. For example, in the diagram above if there were overlapping source images from different dates, you could use the Time Slider to examine each image; however, you would not be able to do this until you were zoomed into approximately 1:22 000.
When your view is zoomed in to view a small number of raster datasets, pyramids are typically involved. When the view zooms out to cover a mosaic comprised of many individual datasets, overviews are typically involved.
Additionally, if you plan to view individual rasters from within the mosaic dataset using Lock Raster (for example), the rendering may be faster at different scales for raster datasets with their own pyramids (and statistics). Previewing the rasters from the mosaic dataset’s attribute table will also be faster.
Submitted by: Melanie Harlow
Overviews and pyramids: Part 1 of 2, What are they and why do I need them?
This is part 1 in a 2-part blog. Part 1 provides an overview of pyramids and overviews. Part 2 will provide you with some guidance on generating them when creating a mosaic dataset.
Basically—overviews are not pyramids and pyramids are not overviews. But pyramids generated by ArcGIS have an .ovr extension (short for overview)…Wait, did I just write that?
Yes, the storage format for a pyramid is an .ovr file. But please don’t confuse this with overviews. Fortunately, overviews are organized in a folder named *.overviews. Both are similar but pyramids are created for raster datasets and overviews are created for mosaic datasets.
|
|
Pyramids |
Overviews |
Description |
Lower resolution (downsampled) images of the original data. |
|
Purpose |
Improve display speed and performance. |
|
Created for |
Raster datasets |
Mosaic datasets |
Format |
Writes .ovr files—with a few exceptions. |
Writes as .tif files. |
Reads pyramids stored externally as *.ovr or *.rrd or internally (e.g. MrSID) |
||
Storage |
In a single file that generally resides next to the source raster dataset and using the same name. |
By default, in a folder next to the geodatabase with a *.overviews extension, or internally for ArcSDE.Storage location is customizable. |
Storage size |
2 to 10% (compared to original raster datasets) |
|
Downsampling factor |
2 |
3 (default) |
Extent |
|
|
Options when building |
|
|
Why are you generating them?
Pyramids aren’t mandatory—but without them, the display speed of your raster dataset can be prohibitively slow, especially if the datasets are very large.
Overviews aren’t mandatory—but they are highly recommended. You do not have to create them or you can generate them in only particular parts of the mosaic dataset, such as a highly visited part of the imagery. However, if you don’t create them you may not see any imagery (you may see a wireframe instead or gray images), since there is a limit to the number of rasters that will be processed at one time (which you can change). Without them, the mosaic dataset may display slowly because of all the processing.
How to create pyramids
If a raster dataset doesn’t have pyramids, then you will often be prompted to create them when you display the data in an ArcGIS application, such as ArcMap. But it’s better to create them before you use the data. Pyramids can be created using geoprocessing tools. There are a few tool choices, since it depends on whether you have one or many datasets to process. To learn about these, see Building pyramids using geoprocessing tools. You can change the properties of the pyramids, such as the resampling method and compression via the geoprocessing environments.
How to create overviews
To create overviews, first they are defined, then generated. When they are defined, the application analyzes the mosaic dataset and using the parameters set for the overviews it defines how many are needed, at what levels, and where. Then they are added as items in the mosaic dataset which appear as new rows in the attribute table. At this point, only the rows have been created to identify the properties and number of the overviews. Next, the overview files are generated. Both defining and generating can be done with one tool—Build Overviews. However, if you need to modify any properties, such as defining a new output location or tile size, then you must run Define Overvews first (to define the properties and add the items to the attribute table), then run Build Overviews to generate the overview files.
When you do generate them, the mosaic dataset keeps track of any changes made to it, such as updating an image, adding or removing images, or altering the footprints. By running the Build Overviews tool or Synchronize Mosaic Dataset tool, with the appropriate options, the overviews will be updated.
Associated blogs
- Oh where oh where have my pyramid files gone?
- Pyramids and overviews, or pyramids or overviews?
- Take a look at raster compression, pyramids, time, and space
Submitted by: Melanie Harlow
Raster XML file contains a histogram in ArcGIS 10
At 10.0, ArcGIS began storing the histogram for a raster
dataset when it generates the statistics. This allows the application to
provide more capabilities, such as adding stretches, like Percent Clip. If you export the statistics file you can take a look all
the information.
Previously to 10.0 the exported XML file would look like…
<?xml version=”1.0″?>
<RasterStatistics xml:lang=”en”>
<DatasetName>dem30</DatasetName>
<Band Name=”dem30″>
<min>2081</min>
<max>3447</max>
<mean>2544.409567056246</mean>
<stddev>208.6604762988481</stddev>
</Band>
</RasterStatistics>
At 10.0 the exported XML file will contain the histogram…
<?xml version=”1.0″?>
<RasterStatistics xml:lang=”en”>
<DatasetName>dem30</DatasetName>
<Band Name=”dem30″>
<min>2081</min>
<max>3447</max>
<mean>2544.409567056246</mean>
<stddev>208.6604762988481</stddev>
<Histogram>
<npixels>2597680</npixels>
<min>2081</min>
<max>3447</max>
<v>23</v>
<v>45</v>
<v>54</v>
. . .
</Histogram>
</Band>
</RasterStatistics>
Submitted by: Melanie Harlow
Orthorectification: To Geoid or not to Geoid
When you ortho-rectify a raster dataset, you are given a choice whether or not to use Geoid correction. Geoid correction should be applied when your data source is using orthometric heights and you want to perform a geoid (EGM96) correction to the z-values. Orthometric heights are measured near sea level – which tends to be the case in many images.
Orthorectification can be performed various ways within ArcGIS:
- Geoprocessing – Create Ortho Corrected Raster Dataset tool
- Image Analysis window – Orthorectify button
- Geomtric Function
- Layer Properties Display tab
Clipping an image or raster in ArcGIS
There are many ways to clip out a part of an image or raster in ArcGIS. What you choose may depend on what you’re doing. For example, if you’re using ArcMap and have a graphic, you can use the Clip button on the Image Analysis window. Or if you want to clip something as part of an automated process you can write a script or model that uses the Clip tool. Continue reading
Are there broken paths in your mosaic dataset?
Did you move the source data for your mosaic dataset? Did you share or copy the mosaic dataset to another machine? If you have done either of these tasks, you may have broken the paths that link to the source data.
To view and repair your mosaic dataset item paths, follow the steps below.
1) Browse the Catalog Window, right-click on the mosaic dataset that has broken paths, and click Repair.
Currently the window shows you all the paths and 0 folder levels.
2) Check on the Only Broken Paths check box, so that only the paths that have broken links are shown. Optionally, you can check the Repair Select Items Only check box, if you only want to repair the subset of your selection.
3) Click the up arrow to increase the Folder Paths Depth. Increase the depth until the first set of paths show up. This is the level that the path is broken.
4) Click the Browse button under the New Path column and navigate to the correct location.
5) Click OK to run the tool.
You may need to run steps 1 – 5 again, to make sure there are not any more broken links.
Color balancing a subset of a mosaic dataset
There are times you probably only need to color balance a portion of your mosaic dataset, rather than the entire thing. If you are in this situation then you simply choose a subset of the mosaic dataset items, and then color balance only those selected.
This can be performed in two different work flows: in the User Interface, and also in the geoprocessing environment.
In the ArcMap user interface:
(1) Open up the Properties of your mosaic dataset
– make sure that you are using the top-most level of you mosaic dataset 
(2) Click on the Definition Query tab
(3) Create a query that will be used to subset your mosaic dataset
(4) Now you can use the Mosaic Color Correction window to color balance the subset
In the geoprocessing framework:
(1) Use the Make Mosaic Layer geoprocessing tool
– specify a SQL Expression in this tool
(2) Use the Color Balance Mosaic Dataset tool
– make sure to specify the input to be the mosaic layer you just created
