Advanced Data Validation Workflows

In a previous blog we discussed how to automate your data validation workflows by integrating ArcGIS Data Reviewer with ArcGIS Workflow Manager. We also discussed some of the basic functions you can perform with the Reviewer
Custom Steps
. For this blog I’d like to take it a step further and describe other ways in which you can use the custom steps. I’ll also introduce additional functionality in Workflow Manager that you might find useful when integrating
with Data Reviewer.

Working with Reviewer sessions

Reviewer sessions are a way of filtering records in the Reviewer workspace to show only those records you are interested in. Last time, we outlined how to use tokens with the Create a Reviewer Session custom step to create unique sessions for each job based on the Job ID. There are other ways you can create sessions depending on your workflows. For example, if you have unique sessions that already exist for each user, you can use tokens like the [JOB:ASSIGNED_TO] in the Run Reviewer Batch Job custom step to write the errors to the existing sessions.

Run Reviewer Batch Job window

Different levels of quality control (QC) can be performed on your GIS data – for example, an initial round of data integrity checks, followed by an automated review, and lastly a visual review (such as, reviewing a sample set). To model these levels of QC in a workflow you can add multiple Run Reviewer Batch Job custom steps, each pointing to a different batch job. In the example workflow below the Run Data Integrity Batch Job step contains a batch job that checks for data integrity errors such as invalid geometry. If errors are found during this step they should be fixed before moving to the next level of validation. That next level is a detailed automated review which is performed by the Run Feature Comparison Batch Job step. This step references a batch job containing checks for feature/attribute comparison. And lastly, to perform a visual review of your data you can add a step in the workflow that creates a sample set of features. This allows you to review only a sample set of features rather than assigning
resources to manually review the entire dataset and is especially useful when reviewing data deliverables from contractors. That way if the sample set fails review, you can reject the deliverable. The Create Sample Set step below creates that sample set by running a batch job containing the Sampling check. The Visual Review step launches ArcMap and allows you to visually review the sample features stored in the Reviewer table against an authoritative source such as imagery.

Advanced validation workflow

Using geoprocessing steps and models to find errors

To validate data in your workflow, you can use more than just the Run Reviewer Batch Job custom step.  Data Reviewer provides a geoprocessing (GP) tool that writes the results of a GP process to the Reviewer table. For
example, if you’ve built a model to find certain types of errors in your data you can store the results in the Reviewer table. Workflow Manager provides a custom step, LaunchGPTool, allowing you to execute GP tools and models within your workflows. 

Displaying Data Reviewer information in Workflow Manager

When linking a Reviewer session to a job, information gets stored in a table in the Workflow Manager database. You can add this table as an extended properties table and display the information in the Workflow Manager interface as well as include it in automatically generated emails.

Reviewer Session Properties 

Generating and attaching Data Reviewer reports to a job

Workflow Manager provides the ability to attach supplementary information to a job. A QC technician might be responsible for finding the errors while his/her manager may be responsible for deciding to accept the data
or fix the errors. For that reason, it’s important for the QC technician to provide supporting information so managers can make sound business decisions.

QC technicians can generate reports about the errors in the Reviewer table and add them as attachments to the job using the Job Information window in Workflow Manager.

Managers making the final approval decisions can easily view the reports within Workflow Manager without having to open ArcMap to mine through the Reviewer table.

Workflow Manager is a powerful tool for managing your organization’s processes. By integrating your data quality control tasks into your workflows and automating them, you’re making your data production that much more efficient. I hope you found this blog useful and it gives you some creative ideas for utilizing these capabilities in your QC workflows.

Content contributed by Amber Bethell

This entry was posted in Editing and tagged , , , , , , , , , , , , , , . Bookmark the permalink.

Leave a Reply