Identifying Coincident Service Connections using the Duplicate Geometry Check

The presence of coincident point features in the geometric network, such as duplicate gas meters (service points), impacts analysis since the geometric network will only allow connectivity to one of those points. The second point is disconnected from the network which affects service outage notifications. The Duplicate Geometry check in Data Reviewer can be used to identify where coincident points exist that may negatively impact analysis tasks. In the screenshot below, the check has been configured to look for duplicate meters which may disrupt connectivity and affect network tracing tasks.

Here are the steps to configure and run this check:

1. From the Data Reviewer Toolbar, click on the Select Data Check dropdown.
2. Expand the Duplicate Geometry Checks category and click on Duplicate Geometry Check.
3. For Check Title type Duplicate Meters.
4. Choose Meters for Feature Class 1.
5. Choose Meters for Feature Class 2.
6. Click OK to dismiss the check properties dialog.

The image illustrates duplicate meters found upon running the check.

Finding Missing Features using the Valency Check

Data completeness is an important aspect of data quality when modeling real world infrastructures in the geometric network. Missing features such as fittings and other network devices impact modeling results and limit the ability to maintain accountability in your equipment inventory. Data Reviewer’s Valency check enables you to find features that should be in your data but are actually missing. By using the Valency at nonexistent point location validation method, you can search for points not present in the data. This check is helpful in managing distribution systems and allows you to pinpoint missing fittings or other network features. The steps below describe how to configure the Valency check to locate intersections where a tee fitting should be present.

Here are the steps to configure and run the Valency check:

1. From the Data Reviewer toolbar, click the Select Data Check drop-down arrow.
2. Expand Advanced Checks category and click Valency Check.
3. For Check Title type Missing Tees.
4. Choose the Tees subtype from your Fitting feature class as Feature Class 1. 
5. Choose Mains as Feature Class 2.
6. Set the Validation Method as Valency at nonexistent point location.
7. In the Valency area, set the Operation to  = (equal to) and type 3 in the text box.
8. Click OK to dismiss the check properties dialog.

In this image you can see a tee fitting is missing at the intersection (circled area) where three mains connect.

Checking Geometric Network Connectivity Rules using a Database Validation Check

A geometric network’s connectivity rules define the type of network features that may be connected to one another and the number of features of any particular type that can be connected to features of another type. The Connectivity Rules check in Data Reviewer allows you to use pre-existing geometric network connectivity rules to validate your data. This check will return feature geometries that violate the connectivity rules established in the geometric network properties. For example, if the connectivity rules allow Mains to connect only to NonControllableFittings, the check will return any connection between Mains and another point (or junction) feature class.

Here are the steps to configure and run the Connectivity Rules check:

1. From the Data Reviewer toolbar, click the Select Data Check drop-down arrow.
2. Expand Database Validation Checks and click on Connectivity Rules Check.
3. For Check Title type Connectivity Rules.
4. To configure this check, select your geometric network. 
5. Click OK to dismiss the check properties dialog.

Checking Connectivity using a Feature on Feature Check

For a variety of reasons some organizations may choose not to implement connectivity rules in their geometric network. These reasons may range from not having the necessary resources to invest in defining the rules to known data quality issues which prevent effective data validation. For such organizations, there is another way to validate feature connectivity by using the Geometry on Geometry check. Take the same example as above – NonControllableFittings should only connect to Mains. The Geometry on Geometry check can be configured to find any NonControllableFittings that are not connected to a Main. This includes fittings connected to other line features such as service laterals or fittings that are disconnected.

Here are the steps to configure and run the Geometry on Geometry check:

1. On the Data Reviewer toolbar, click the Select Data Check drop-down arrow.
2. Expand Feature on Feature Checks and click Geometry on Geometry Check.
3. For the Check Title type Disconnected Fittings.
4. Choose NonControllableFittings for Feature Class 1.
5. Choose Mains for Feature Class 2.
6. Set the Spatial Relation Check Type to Intersects.
7. Toggle on the Not – find features not in this relationship check box.
8. Click OK to dismiss the check properties.

In this image you can see a fitting that is disconnected from the network.

Described here are just a few simple examples of Data Reviewer checks that can help in validating features participating in a geometric network. Each are easily configured and can be used for validating both gas distribution and water/wastewater/stormwater datasets. In a  future blog, we’ll focus on examples of how Data Reviewer’s advanced checks can be leveraged for more specific validation of features in the geometric network.

Content contributed by Karen Lowery and Craig Venker

• Cannot create a Reviewer workspace in a SDE database
• Creating grids using Create Polygon Grid was taking longer or would not create at all

 Be sure to download the patch if these issues are critical to your organization.

 The check returns features that meet one of the following conditions:

1. Nothing

2. Empty

3. Has an empty envelope

4. Not simple

Okay, so what do these conditions mean?

 Let me address the first three conditions as they are somewhat related.  You might ask, “if they are related why is Data Reviewer reporting these issue(s) independently?” For each feature the three conditions have different meanings. Furthermore, the issues could have been caused by different workflow errors. As a user, all you really need to know is that the feature has no geometry.  However, I’d like to explore these conditions a bit more and provide a brief description of each.

Nothing

The feature has nothing in the SHAPE field (SHAPE = <Null>). This type of error is returned when Data Reviewer is unable to retrieve the geometry information for that feature.

Empty geometry

The geometry of a feature is considered empty if it has zero or no points and it does not contain geometric information (represented by SHAPE _LEN and/or SHAPE_AREA fields) beyond its original initialization state.

Note:

 
Empty geometries can be introduced when editing or creating data programmatically and when importing bad data into your GIS.

Has an empty envelope 

This condition occurs when a feature’s envelope (bounding rectangle) does not have any geometric information. This is similar to empty geometry.  

Not simple

Geometry is considered non-simple if any of the below are true. 

Short segments seem to be the most common non-simple geometry users find in their data.  This occurs when the vertices in a polyline or polygon boundary are too close; but still are within the database’s XY tolerance.  Depending on the type of data, this may or may not be a critical issue for you. In the case of a utility, for instance, this type of error could have a huge impact. An utility may be basing the rotation of point features (i.e. tees) on the angle of its connected lines (i.e. pipes). In the screenshot on the left you can clearly see the tee is not rotated properly.  What you can’t see, but illustrated in the screenshot on the right, is the extra vertex at the end of the pipe which is affecting the angle of the tee.

 Content contributed by Michelle Johnson

Warning: this blog is not for the faint-hearted as we’ll be delving into complex geometric relationships!!

Custom relationships

Trying to understand when and how to use the custom spatial relation option can get tricky. If the six standard relationships are not meeting your needs, it’s time to start thinking about whether the custom spatial relation option could be the solution. Today, I’d like to share just a few tips and tricks to get you started on using the custom spatial relation. My first and best advice is to begin by reviewing the topic in the Desktop Help here. I always refer back to it whenever I’m configuring a custom spatial relationship.

In order to configure a custom spatial relationship, you’d choose the Relation spatial relationship type and enter a string with 9 characters. Each character describes different aspects of how the features relate to each other. Keep in mind the values you enter for the string are different based on the geometry type. To get you started, the link to the Desktop Help above provides several examples of the relationship strings used for many of the most common relationships including the six standard ones (however, when using a standard relationship you should always select it from the drop-down list instead of configuring its equivalent custom relation). One example is the Crosses operation where there are three different relationship strings depending on the geometries of the input feature classes. When comparing lines to lines, the relationship string is TF*FF****. When comparing polygons to lines, it is TT**F****. And when comparing lines to polygons, it is TT*T****.

Note: The custom relation framework does not use database topology or honor the XY tolerance. Therefore, results returned will not be consistent with the results expected when using the topology engine (which includes the six standard relationships). Additionally, the custom relation framework works on the feature classes, not a topologic graph. Therefore, vertex differences between features will be reported as errors. These differences should be considered when using custom relations. If these are requirements in your quality control workflows you should consider building a geodatabase topology, which is a more advanced way of discovering spatial relationship errors.

Example: Lines that do not follow a polygon boundary

One use case for the custom spatial relation is: lines that do not exactly follow the boundary of a polygon. Two examples are lot lines that do not follow the boundary of a polygon or political boundary lines that do not follow a state or county polygon.

Let’s walk through the process of building a custom spatial relationship to find these types of errors. First, I start by creating some basic features in my map. Some of the features have geometric relationships between features that should pass validation and others do not and should fail. By running the check on these sample geometries I’m able to ensure the expected results are returned before validating my actual data. It also allows me to carefully examine how the features relate to each other which will help when defining each of the nine characters in the string.

Above are the sample features I created to test the custom relation. Lines are illustrated in red and polygons in blue. By looking at the sample features you can begin to identify the lines that should fail validation are inside (3), outside (1), and cross the boundary (6, 21). The lines that should pass validation follow the entire boundary of the polygon (2, 13) and other lines only represent a side of the polygon (16-19).

Using this visual representation of the relationships, it’s time to start identifying what the nine characters are and what they mean for the geometry types I’m working with. When building the custom relationship I first determined how the interior, boundary, and exterior of the first feature class relate to the same parts of the second feature class. The first feature class is a line – interior is all of the feature not including the endpoints, boundary is the line’s endpoints, and exterior is everything else. The second feature class is a polygon – interior is the inside and boundary is the outline.

For this example, I’ll use the Not – find features not in this relationship check box (refer to part 2) and will look for lines that follow the boundary. I need to ensure the interior of the line touches the boundary of the polygon; therefore character 2 is set to True. If just character 2 is set then any feature with part of the interior touching the boundary of the polygon will be considered valid. Features such as line number 6 (see below) that mostly follow the boundary but have portions inside or outside will be considered valid when they should not be.

For the check to return this feature as an error, additional characters need to be added. In order to ensure the interior of the line does not touch the interior or exterior of the polygon, characters 1 and 3 were set to False. Now is a good place to pause and test my custom relationship. I used * for the other characters so they are not included in this initial test.

Then I set up the check properties:

1. Lines as feature class 1
2. Polygons as feature class 2
3. Spatial relation check type set to Relation and custom spatial relation string of FTF******
4. Toggle on the Not option

Upon running the check, the results returned were as expected. The features in error were in fact returned as errors (see below) while those that were correct were not returned as errors.

Configuring custom relationships typically requires some trial and error. You might find yourself building and testing several different custom relations before finding the correct one.

Note: Although this check will find lines not coincident with the polygon boundary, it will not identify where a portion of a polygon boundary may be missing a correspoding line.

What’s coming at 10.1?

Before I wrap up, I’d like to give you a sneak peek at a new property added to the Geo on Geo check at 10.1 that allows you to merge features from feature class 2.

Let’s explore a scenario when you’d use this option; for instance, while comparing building footprints to parcels to identify buildings that are not fully within a parcel. In Data Reviewer 10 the check would return both the blue and red buildings, shown in screenshot below, as errors instead of just the red buildings. The blue building is not an error because it is contained within multiple parcels. However, when the merge option is not used, each building is compared to each parcel and because no single parcel contains the blue building it is flagged as an error.

With the new merge option available at 10.1 all the parcels are merged together and each building will be compared to one large parcel (that is made up of all the parcel geometries). Therefore, only the red buildings will be returned as errors.

And that wraps up our series on the Geometry on Geometry check. Here’s something else to think about - are you aware that you can combine multiple Geo on Geo checks together into one check called a Composite check?  What does this mean? Well, if you’re unable to achieve the results you expect because of false positives you can add additional checks to further filter the results. The Composite check will be discussed in a future blog post.

Content contributed by Amber Bethell

An advanced example

Let’s say I want to compare the street name on buildings to the names of the streets in the roads feature class to ensure the street name associated to the building is in fact the name of a nearby street.  As you can see in the image below, the circular building has an address on “Hyde St” but none of the surrounding streets are named “Hyde St”. The Geo on Geo check  enables you to flag such features as errors.

In order for the check to validate this business rule, we’ll take advantage of almost all the parameters in the Geo on Geo check properties dialog. First enter a Title for the check, Building Not Near Matching Street. Then choose Buildings as the first feature class and Roads as the second feature class.

Note, not all the buildings have the address populated. As you can see in Figure 1 above, the large building in the center has no label indicating it does not have an address associated to it. While not having an address value could also be considered an error, a better option for identifying these types of errors would be to use the Execute SQL check. Since I do not want any building features without an address to be returned by the Geo on Geo check, I’ll exclude them by using a WHERE clause. For the first feature class let’s set the query to validate only those features where the street name is not empty (NULL).

The next parameter is Compare Attributes. We are interested in comparing the street name value on buildings to the street name value on roads to see if they are the same. We’ll use the ‘=‘ (equal to) operator to accomplish this.

Since none of the buildings are touching the roads I’ll choose the Intersects option and take advantage of the tolerance parameter. In this sample data each block is about 100 meters on each side, so I’ll input 50 Meters as the tolerance to compare each building to all the roads that are within 50 meters. Lastly, make sure Not – find features not in this relationship is checked.

Now that the parameters are set, let’s summarize – I’m comparing all the Buildings (that have the StreetName field populated) to the Roads and am looking for any building not within 50 meters of a road with the same StreetName.

Why the “Not” is important

You may be wondering why I didn’t set up the check’s Compare Attributes properties to be “StreetName not equal to StreetName” and the spatial relationship to Intersects with the Not option turned off. Take a look at the following screenshot.

The building with the address of 900 North Point St is on the corner of North Point St and Polk St. That means the building is within 50 meters of both streets. If I changed the parameters to say the building intersects a street with a different name, then the building at 900 North Point St will be returned as an error. When the building is compared to North Point St it will not be identified as an error. But when the check continues processing the rest of the streets it will compare the building to Polk St and determine that the criteria of the check are met, therefore, return the feature as an error.

When using the original parameters, this feature will not be returned as an error. The check will compare the building to Polk St and determine that the feature does not intersect a street with the same name. If this was the only nearby street the building would be returned as an error. However, there is another street nearby and the check will continue comparing the building to the other streets within the given tolerance. When the building is compared to North Point St, the check will determine it does intersect a street with the same name. This means the building failed to meet the specified parameters (not intersecting a street with the same name) and will not be returned as an error. 

The difference is the Not – find features not in this relationship checkbox. If this is unchecked,  any street that intersects a building with a different name is returned as an error. However, if the check box is checked then the check will only return an error if none of the streets the building intersects has the same name.

Now that I’ve have explored all the parameters available in the Geo on Geo check properties dialog, there’s one more key piece of functionality left to look at. So, in part 3 of this blog series I’ll discuss what to do when the six standard geometric relationships – Touches, Contains, Intersects, Within, Crosses, and Overlaps – do not find the features you want. If this is the case, you can then use the custom spatial relation which allows you to build your own geometric relationship.

Content contributed by Amber Bethell

Geo on Geo parameters

To get started, here’s a quick breakdown of the parameters available in the Geo on Geo Check Properties dialog.

A simple example

In many cases you might not use all the above parameters. Let’s explore a simple business rule: buildings should not overlap. I am using a sample dataset where the buildings are quite dense and many of them share edges, and we want to ensure they do not overlap.

To validate this business rule simply choose the buildings feature class for both Feature Class parameters. You do not need to set up any attribute comparisons. For Spatial Relation Check Type choose Overlaps. You can optionally enter a Check Title, Notes, and choose a Severity.

Upon running the check any buildings that overlap are returned. Data Reviewer zooms you to the area of the feature that failed the check – in this case the area of overlap is shown in red.

An intermediate example

Now that you’ve seen how to configure the Geo on Geo check to validate a simple business rule, let’s explore another business rule that compares feature attributes. For this example, we’ll look for features that are connected but have different values on a key attribute field. This scenario is often encountered in the utilities industry where you may have a valve connected to a pipe but the diameters of both are different. In my sample data, there is a 12″ diameter valve that connects to two 8″ diameter water mains. This is obviously an issue and the Geo on Geo check can be used to find such discrepancies.

First, we’ll input a check title. Then choose the two feature classes to compare - SystemValve and Main.

Since our goal is to compare just the diameters of features we will choose the Compare Attributes option in the Attributes section of the check’s dialog.

Note: If you want to compare all of the attributes to ensure they match, use the Compare All Attributes option.

Choose the Diameter field from both feature classes and then choose the operator that will be used to compare the fields. Since we’re looking for features with different diameters, choose the <> (not equal)  operator.

Note: Since the drop-down menus list all the attributes on each feature class you can choose the attributes desired and the names of the fields do not have to match (however, the types do need to match).

Now let’s set the final parameters for the check. Choose the spatial relationship - Intersects. No Tolerance is necessary because we’re looking to compare system valves to the mains only if they are connected. Optionally, enter Notes to describe the issue and input a Severity value.

Note: In this scenario, you could also choose Touches for the spatial relationship. When comparing points to lines, the Intersect option will identify points that intersect the line anywhere along the length of the line. The Touch option will only identify points that touch the ends of the line.

The Geo on Geo check is extremely flexible and can be used to identify many types of potential issues in your GIS data. While we’ve come to the end of this blog, we have not come to the end of what the Geo on Geo check can do. This is only the first of three blogs! Stay tuned for the next one which will focus on the “Not – find features not in this relationship” parameter.

Content contributed by Amber Bethell

The steps include:

1. From the Data Reviewer toolbar, select the Select Data Check dropdown.

2. Expand the Advanced Checks category and select Sampling Check. 

3. In the Sampling Check Properties dialog, select Auto Calculate.

4. Under Auto Calculate, select your Confidence Level and Margin of Error.

Note: The question you’re looking to answer: given a population size (number of features), what sample size do I need so that I’m “X” percent confident the sample size is statistically significant within a “Y” percent margin of error?

Once the sample size is generated you have two options for using it with the PAAT. The first is to use the Browse Features dialog to zoom to the selected features in the sample set and then the PAAT to read feature locations. The second option is to use the Grid Properties dialog in the PAAT to produce a number of grid cells corresponding to the sample size and then read feature locations from within each grid cell. Both options are described in more detail below using a fairly common scenario – evaluating vectors based on a raster. In this case, I am evaluating roads based on imagery for the State of Louisiana.

Browse Features Option with PAAT

The steps below use the Browse Features dialog (which is activated upon running the previously configured Sampling check).

1. From the Data Reviewer toolbar, click the Data Reviewer dropdown, and select Positional Accuracy Assessment.

2. Set up the PAAT to use a single grid cell (1 x 1) that will encompass your entire image. The PAAT always produces a grid when evaluating vector features, but since you won’t be using the grid cells, the minimum number, 1, is sufficient. 

3. Turn the PAAT Auto-pan function off, since you won’t be using it either. 

4. Use the Zoom to Feature option in the Browse Features dialog (circled in red in the image below) to zoom to each feature.

5. Use the PAAT Digitize Points function to collect data (vector) locations and image (reference) locations. 

In this example, the sample size Data Reviewer generated was 62 of the total 713 features. As a result of using the tools from the PAAT, you can see we have a horizontal accuracy of about 477 meters CE90.

Grid Option with PAAT

Below are the steps for using the PAAT grid option.

1. Use the PAAT to set up a sampling grid that contains the same number of grid cells as the sample size generated by the Sampling check.  

2. Once you have the sample size you can dismiss the Browse Features dialog, as you will not be using it.

Note: In order to generate grid cells that intersect your data equal to the sample size for an irregularly shaped area, you may have to create a grid that contains more cells than the sample size. In the case of the State of Louisiana, which has an irregular (non-rectangular) shape, I created a 10 x 10 grid (100 cells) to get 62 cells (the calculated sample size) that actually intersects the data.

3. The PAAT will automatically zoom you to the next grid cell when a feature’s data and reference positions are collected.

The result of using the grid option with PAAT was a horizontal accuracy of approximately 487 meters CE90.

The difference in accuracy between the two options is only about 2 percent. This indicates the results are basically in agreement given the relatively low sample size of features over a large area. 

Data Reviewer’s Sampling check (using Auto Calculate) is a quick way to determine a statistically valid sample size for use with the Positional Accuracy Assessment Tool (PAAT). Neither option is necessarily more valid; it basically comes down to your product specifications for sampling and which option works better for you.

Content contributed by Pete Aniello