Distributive Flow Maps – More Raster, More Faster

NOTE:  The Distributive Flow Lines tool was updated  Jan 10, 2014 .  If you are using ArcGIS 10.1sp1 or 10.2 please download the latest version.

Based on comments and feedback from the first version of the Flow Map tool and blog post we gave it a second look and have released an update.  As you can see in the screen capture above, the output is similar to the original tool but we have switched to an all raster approach which provides more control over the lines and reduces processing time.

A distributive flow map made by Charles Joseph Minard showing coal exports from England in 1864.

For convenience, I have placed the introductory text from Brad Simantel’s original post below to provide a concise introduction to the history and use of Flow Maps.

“Flow maps are used to show the movement of goods or people from one place to another. These maps use lines to symbolize the movement, often varied in width to represent the quantity of the flow, and fall into one of three categories: radial, network, and distributive. Radial flow maps are used to show relationships between one source and many destinations. If there are more than a handful of destinations and you still want to show the quantity of flow, however, the lines overlap too much to discern individual values. Network flow maps are used to show the quantity of flow over some existing network — transportation and communication networks being the most common. Distributive flow maps are similar to radial flow maps, but rather than having individual lines from the source to each destination, lines are joined together, only forking once they get close to their destinations”.

If you would like to try it out as you read this post, you can access the new  Distributive Flow Lines tool (DFLT) on ArcGIS Online. If you do not have data handy, the British Coal export data, used to create the first map in this post, is also available for download on ArcGIS Online.

Tip: The DFLT requires the Distributed Quantity Field to be of type Integer.  If you use the British Coal sample data a new integer field will need to be added and calculated based on the existing CoalTonnage field.  The current field represents thousands of tons so multiplying by 10 and rounding to the nearest integer would represent hundreds of tons rather than thousands.

Like the first Flow Map tool, the Spatial Analyst extension is required to use the new DFLT.   In fact, the new tool is completely raster based until the end when the flow line feature class is created as output.

The DFLT will accept polygon or point features as input for the Source and Destinations, however, point features are recommended.   If polygons feature classes are used as input for either Source or Destinations they will be converted to points for internal use but these intermediate point datasets are not saved as part of the tool output.  After the Destination feature class is selected the user will need to select an integer field from this dataset representing the quantity that will flow to the destinations from the source.

The DFLT provides an option for the user to specify a feature class to use as impassable features.  These features will be buffered by 1.4 times the Cell Size parameter.  The area inside this buffer will be NoData and will be used as a processing mask within the tool.  A second, slightly larger buffer, 3 times the Cell Size parameter, is also created. The area between the inside buffer and this larger buffer is given a very high cost so Flow Direction will not calculate routes that go into the NoData area (most of the time).

The second optional parameter is a Impedance feature class.  These polygons represent  features in your map that you would like flow lines to avoid as much as possible.  How much these features are avoided is controlled by the Impedance Weight slider.  When impedance features are specified, they are given a high cost; but may still be crossed in extreme cases where there is no alternative or where they can be crossed in a narrow location as shown in the example below.

Tip: One common effect of the DFLT is that it will often skirt Impedance features too close.  This can be avoided by buffering the Foreground features and then using this new feature class as the Foreground features in the tool.

Since the primary use case for the DFLT is to create pleasing flow lines, the user will most likely run it several times using different values for the weight sliders to achieve a good starting point for the effect they are looking for.  Using an all raster approach makes this process a bit easier and faster.  In raster processing, cell size has a major effect on processing time. The DFLT calculates a default cell that size works well for the final product.  In most cases it will not be necessary to use a smaller cell size than the default. We recommend using a much larger cell size during initial iterations to reduce processing time until you start getting results close to what you want.

Tip: It is a good idea to make a note of the default cell size for final processing.  For initial runs of the tool,  multiplying the default cell size by 5 or 10  works well to get a feel for how the various weights and optional parameters effect the output flow lines.  More drastic changes in cell size may have a significant effect on how the lines are routed around Impassable and Foreground features.

Tip: If you use the post 10.1 sp1 version of the tool you will notice the tool dialog has been simplified.  Source Weight and Destination Weight sliders have been replaced by a single slider to control how close to the Source or Destinations the flow lines branch.  Also processing extent options have been reduced to “Same as Display” and “Same as Destinations plus Source.”  This was changed to prevent most of the cases where no solution is possible.

 

Cell size is so important in the DFLT because the tool calculates Euclidean distances from source and from all of the destination features. These two distance rasters are then Sliced into an equal number of discrete cost classes.  The number of classes is determined by dividing the maximum processing extent dimension by the cell size.  This acts to normalize the effect of the Source and Destination cost rasters and provide a basis for the cost of the Impedance raster.  By normalizing these costs the tool provides the user more control over the shape of the final flow lines through the Source and Destination Weight sliders.

To understand how cell size affects the total cost surface it helps to see a few examples.  In the following ArcScene screenshots we are using a Source location in the center of the US and the destination features are the centroids of a several countries in Africa. The CostDistance surface is shown floating over the “flat Earth” to provide a means of comparing the effects of the weight parameters and cell size. In each example the perspective and Z exaggeration are the same.

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =8 Cell Size = 100,000 meters

Source Attraction = 10 Destination Attraction = 1 Foreground Weight =1 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 1 Foreground Weight =10 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 100,000 meters

Source Attraction = 1 Destination Attraction = 10 Foreground Weight =8 Cell Size = 30,000 meters

In this last example, it was necessary to reduce the elevation exaggeration and to zoom out so the cost surface would display in a reasonable way.  The only difference between the two results is that the cell size has been reduced from 100,000 meters to just 30,000 meters.  The effect is that the number of “cost slices” is larger and so the CostDistance “elevation” is increased.

As previously stated, the new DFLT processes the Impassable features separate from Impedance features.  Impedance features are treated much the same as Impassable features in the first tool.  They are features the user would like the flow lines to avoid if possible but the flow lines will cross them if necessary to reach a destination location.  Impassable features represent a hard barrier in the cost surface and can be used to control the shape of the flow lines as in the example below.

Flow lines representing distribution of Coal by ship are crossing Central America regardless of Impedance weight

With the red barrier added flow lines now go around the tip of South America

As with the previous Flow Map tool, the output of this new tool will usually be the starting point to produce a final flow map with smoother more organic looking flow lines.  Once the lines are roughly where you want them the next step is to use Graduated Symbols to display the Distributed quantity.  The “GRID_CODE” field will contain values based on the input Distributed quantity field.  The values in this field will be equal to the Distributed quantity field for the smallest lines nearest to the destinations.  Each time a tributary flow line meets another line the combined line closer to the Source feature will have a GRID_CODE value equal to the sum of previous two tributaries. Temporarily placing labels on the lines and destination points makes this clear. In the examples above the lines are represented using 10 classes of line weights ranging from 1 – 10. Using rounded joints and ends on the lines also improves the output. After setting up graduated symbols many users may also prefer to run the Smooth Line tool on the flow lines. This will also add some curve to the flow lines which is often desirable. A good starting point is to use a tolerance value approximately four times the Cell Size used to create the flow lines. It is also recommended to select the PEAK algorithm and FIXED_CLOSED_ENDPOINT options. Even after these steps it may be necessary to make manual adjustments to the shape of the line.

Installation instructions:

After downloading and unzipping the Distributive Flow Lines tool you should see the following in the ArcMap catalog window.

Expanding python toolbox will reveal one script tool as shown below:

By default, python toolboxes do not display the “pyt” file extension. To display this and other known extensions in ArcCatalog (or the catalog window in ArcMap) check the following:

This dialog is accessible in ArcCatalog by clicking Customize > ArcCatalog Options.  Unchecking the highlighted option will show the following:

We hope you find the new tool useful and the tips above help get you started experimenting with the tool.  We look forward to you comments.

NOTE:  The Distributive Flow Lines tool was updated  Oct 22, 2013 to address a backwards compatibility issue.  If you are using ArcGIS 10.1 please download the latest version  here.

NOTE:  The Distributive Flow Lines tool was updated  Jan 10, 2014.  If you are using ArcGIS 10.1sp1 or 10.2 please download the latest version.

Contributed by Bob Gerlt.

This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

25 Comments

  1. benblatt says:

    Hello, I would like to use this, but am having some trouble. When I download it I think I may be missing files. In the folder there are seven files

    -DistributiveFlowLines_bv2~.pyt
    -DistributiveFlowLines_bv2.DistributiveFlowlinesNew.pyt
    -DistributiveFlowLines_bv2
    -DistributiveFlowLines_bv2.pyt
    -DistributiveFlowLines_bv2bu.DistributiveFlowLinesNew.pyt
    -DistributiveFlowLines_bv2bu.pyt
    and the README

    there is no file ‘FlowMapGenerator’ to click from the catalog.There is a ‘DistributiveFlowLineLinesNew’ that comes up in its place, but that does not function and there is the following error associated with it

    Traceback (most recent call last): File “”, line 85, in getParameterInfo File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\mixins.py”, line 286, in __init__ setattr(self, attrib, attribvalue) File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\_base.py”, line 87, in _set return setattr(self._arc_object, attr_name, cval(val))ValueError: ParameterObject: Invalid input value for DataType property

    Thanks!

  2. bgerlt says:

    Thanks for the note about the number of files. The zip file has been updated and now contains only the required 4 files. I have added a missing link and corrected a typo in the README.txt file. There are two ways to add the Python toolbox to ArcGIS. The simple method is the one described in the README file. The second method is through ArcToolbox. Here is a link describing the process of adding a Toolbox to to the Arctoolbox window. http://resources.arcgis.com/en/help/main/10.1/index.html#//003q0000001m000000

    After adding Flow Map Generator v2 to ArcToolbox click the plus sign to the left of the tool window. This should open the tooldbox and reveal the Flow Map Generator script tool. Double click the tool to open it.

  3. clemgtva says:

    Good morning,

    I m trying to run the tool properly on a 10.1 sp1 win7 64. I use a basic edition but have the Spatial Analyst tools on a trial for 60 days. I have the “tool is not licensed” error (000824). Is that normal ?

    See the code below for errors :

    Executing: FlowMapGenerator Export_Output SM08_Point_joined SM115 # “480675,225163667 4935375,70926774 576719,167251551 5081029,12557457″ 200 100 100 100 20 \\xxxxx….xxxxx\Default.gdb\Export_Output_FlowMapGenerat1
    Start Time: Mon Sep 16 10:30:46 2013
    Running script FlowMapGenerator…

    Traceback (most recent call last):
    File “”, line 186, in execute
    File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\management.py”, line 2376, in FeatureToPoint
    raise e
    ExecuteError: Failed to execute. Parameters are not valid.
    ERROR 000824: The tool is not licensed.
    Failed to execute (FeatureToPoint).

    Failed to execute (FlowMapGenerator).
    Failed at Mon Sep 16 10:30:53 2013 (Elapsed Time: 7,00 seconds).

    Thank you for your time.

    • bgerlt says:

      Thank you for your input. I have updated the Distributive Flow Lines tool (and also changed the name). The tool now checks the input feature type for both Origin and Destination feature classes. If they are both point feature classes the FeatureToPoint GP tool is not called. This allows the tool to be used with a Basic license. If polygon feature classes are used then FeatureToPoint is called which will require an Advanced (ArcInfo) level license. In both cases the tool will still need a Spatial Analyst license.

  4. bnhilton says:

    This looks great, but I’m receiving the same error…any ideas?

    Traceback (most recent call last): File “”, line 85, in getParameterInfo File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\mixins.py”, line 286, in __init__ setattr(self, attrib, attribvalue) File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\_base.py”, line 87, in _set return setattr(self._arc_object, attr_name, cval(val))ValueError: ParameterObject: Invalid input value for DataType property

    • bgerlt says:

      After a couple rounds of testing we found a backwards compatibility issue that has been corrected and the tool has been updated. Please download a fresh install if you are using ArcGIS 10.1 or older. Keep the feedback coming it is always appreciated.

      • straatvaart says:

        Just tested your latest version as suggested three days ago because I was also having the same issue mentioned by clemgtva (“tool is not licensed” error (000824). Now, while using the British Coal data I am stuck because after entering the “Destinations Feature Class” value as “myfolder\British_Coal_Exports.gdb\Destination_Ports” I cant fill in the mandatory field “Distributed quantity field” as no values pop up (blank list) and if I manually enter “Coal” (the field name value from the atribute table) it makes no difference… lacking that mandatory field I cant run the Distribitive Flow Lines script as I got message saying that field “Error: 000728. Field Coal does not exist within the table” :(

        • straatvaart says:

          Ok got it. The Distributed quantity field did not appear because there was no long integer field in the atribute table. Now there is one. Yet if I execute the Distribitive Flow Lines script I still got this error: Traceback (most recent call last): File “, line 150 in UpdateMessages File “c:\program files\arcgis1desktop10.1\arcpy\_init_.py”, line 1190, in Describe return gp.describe(value) File “c:\program files\arcgis1desktop10.1\arcpy\geoprocessing\_base.py”, line 374, in describe self.gp.Describe(*gp_fixargs(args, True)) RuntimeError: Object Describe input value is not valid type Failed to execute (DistributiveFlowLinesNew).

          • straatvaart says:

            Thanks Bob for your assistance. As you immediately notice the problema above was due to the fact that my processing extent parameter was wrongly setup to “as specified below” (wrong) instead of “as display” (good); and that the cell size recommendation was larger than the one I mistakenly used. Many thanks and congrats for this useful tool.

  5. jimju says:

    Hi there!
    I’m using ArcGIS 10;1 and when I open distributiveFlowLines.pyt, the script is with a red ‘x’ over it. its unable to run.
    The problem is Traceback (most recent call last): File “”, line 30, in getParameterInfo File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\mixins.py”, line 286, in __init__ setattr(self, attrib, attribvalue) File “c:\program files (x86)\arcgis\desktop10.1\arcpy\arcpy\arcobjects\_base.py”, line 87, in _set return setattr(self._arc_object, attr_name, cval(val))ValueError: ParameterObject: Valor de entrada inv\xe1lido para propriedade DataType

    I know it has already been posted here, but I still facing this… can you help me?

  6. lomasmuertas says:

    I have been running into errors every time I run this tool..first because I had the old version, next because I was using a column that was a result of a join that “didn’t exist”, now with the current set up everything is projected into world mercator. I have one point feature class and another with multiple point features that has a COUNT short int column I am using for the Z value. Here is what I get, any suggestions??

    Messages
    Executing: DistributiveFlowLinesNew Presidio_LatLng_Project2 Origins_Project1 COUNT World_Countries_Project3 World_Countries_Project3 5 5 5 “-20037507.0672 -30198185.1699 20037507.0672 18386391.7109″ 32389 “C:\Users\Acorn Cache\Documents\ArcGIS\Default.gdb\Presidio_LatLng_Project2DISTTRY200″
    Start Time: Wed Nov 20 13:47:19 2013
    Running script DistributiveFlowLinesNew…
    Temporary processing extent feature class created
    Creating processing mask and background raster with impassable features
    Foreground cost raster created
    Source Feature processed
    Destination features processed
    Executing: Slice “C:\Users\Acorn Cache\Documents\ArcGIS\Default.gdb\EucDist_temp1″ “C:\Users\Acorn Cache\Documents\ArcGIS\Default.gdb\Slice_EucDis2″ 1500 EQUAL_INTERVAL 1
    Start Time: Wed Nov 20 13:50:45 2013
    ERROR 010429: Error in GRID IO: RMPValidate: Invalid Remap data are entered near record 1 and 2 ..
    ERROR 010235: Returned error from SLICE.
    ERROR 010067: Error in executing grid expression.
    Failed to execute (Slice).
    Failed at Wed Nov 20 13:50:55 2013 (Elapsed Time: 10.00 seconds)
    Completed script DistributiveFlowLinesNew…
    Failed to execute (DistributiveFlowLinesNew).
    Failed at Wed Nov 20 13:50:55 2013 (Elapsed Time: 3 minutes 36 seconds)

  7. grantjg12 says:

    Using the new tool on 10.2 with spatial analysis. But I keep getting the same error:

    Traceback (most recent call last):
    File “,string>”, line 189, in execute
    AttributeError: ‘Unicode’ object has no attribute

    failed to execute (DistributiveFlowLinesNew)

    This tool looks great. Just what I need. I tried it on both my data and the download example datasets (and changed the z value to integers). I get this error every time.

    Any ideas???

    • bgerlt says:

      It looks like the issues reported in the previous comments were caused by a change in GP datatype definitions. I have uploaded a new version of the tool that is now compatible with ArcGIS 10.1sp1 and 10.2 Please select the link to the updated Distributive Flow Lines tool above. The links at the very top or bottom of the blog will work.

  8. johnmdye says:

    Shazam Bob!
    I was just perusing through and saw you had sliders in your tool. Totally caught me off guard because I didnt even know that was possible. Pulled down the tool, took a glace at the pyt, and poof, there it is! filter.type = “Range”. Learn something new everyday. No idea how I’ll use it yet, but excited that i can! Thanks!

  9. phildu says:

    Hi, Looks like a useful tool, but I’m finding problems with my dataset. I’ve attempted to find the coal data to replicate your examples, but it doesn’t appear to be accessible on the “ArcGIS Online” interface.
    Anyone know where I can get it?
    Cheers.

    • bgerlt says:

      When you click on the British Coal link above it should take you here:
      http://www.arcgis.com/home/item.html?id=9d48ce9b6837492e982e980402d3ed11

      On the Open drop down menu select “Open in ArcGIS for Desktop.” This should cause an ArcGIS Package Information file to download. Double clicking this file should then spark up ArcGIS for Desktop with the data loaded in the table of contents. It looks like you will need to provide a point for the UK but Destination Ports layer is there and has Coal column. This Coal column should be multiplied by 10 to “intergerize” it so it can be consumed by the DFM tool. After the tool runs divide the Gridcode column by 10 to get the original values back.

  10. mhornfeck says:

    I am having trouble running the tool when I use the impassible feature class. I have arcview and am on the free spatial analyst trial. Basically it says parameters are not valid.

    I am trying to re-create the flow map while trying to avoid having lines go through the continental landmasses.

    Thanks,

    • bgerlt says:

      If your destinations are inside or very close too continental boundaries then you want to use the continents as Impedance features rather than Impassable features. If they are Impassable then the model cannot create a path to the points inside the continents. If you make them Impedance features and set the Impedance weight very high then the continents will be avoided as much as possible but the paths can still reach the destination points.

  11. harbingeriv says:

    This seems great! However i cannot get it to work properly. I am not too experienced with the software, so it could be that; however, there are two things that bugs me as of now:

    1. I cannot find England in the coalmap-dataset; i solved this by creating a new layer, which i georeferenced to the destinationports and created England as a point in that one. That is probably not a right way to do it and might have caused the problem.

    2. When i add the data to the Distributive flow lines-script, it returns the errors:
    “Failed to execute (DistributiveFlowLinesNew)” & “Destination feature class only contains one feature.

    Anyone got any ideas of why this happens? Any help would be much appriciated.

    -AØ

    • bgerlt says:

      You should use England as the source point and the other ports as the destinations. You may be doing that but I just want to make sure.

      • harbingeriv says:

        Yes, i created a point for England, which i gave the same georeference as the rest of the map. I also have turned on the spatial analyst for the task. I have multiplies the dataset with 10 to create the new one for coal. But when i put it in it prompts me with the message “Failed to execute (DistributiveFlowLinesNew)” “Destination feature class only contains one feature.

        Perhaps i did something wrong when i multiplied with 10? i just added the new field and used the field calculator to =10*coal.

  12. rawatson says:

    I cannot seem to log in to get the code – my ERSI log in is not sufficient as I am “not part of the organisation” I do much scientific mapping of fisheries product flow and would like to see if this approach will work. The previous version of this code does not seem to function. Is there any way to get a copy of this code?

  13. pufferfish.tien says:

    Hi there,
    I am using the tool to create a flow map of refugee flow. While executing the tool this error message occurred:

    Traceback(most recent recall last):
    File “”, line 219, in execute
    ZeroDivisionError: float division by zero

    Please let me know what I should do to amend this problem. Thank you so much!