Strengthening the Link between GIS and Science

An interview with Kevin Butler about the integration of ArcGIS and SciPy

Geography is the science of our world, and GIS is a foundational technology for helping us to better understand that science.  To further strengthen the link between GIS and science, today at the Esri Ocean GIS Forum we’re pleased to announce the integration of ArcGIS with SciPy, a Python-based ecosystem of open-source software for mathematics, science, and engineering.

I recently caught up with Kevin Butler, a Product Engineer with the Geoprocessing and Analysis Team, to ask him a few questions about the integration between ArcGIS and SciPy.


Matt Artz
: What is SciPy?

Kevin Butler: SciPy is a software library that helps scientists, engineers, and GIS analysts perform custom scientific and technical computing.  It is an open-source library built using Python—an easy to learn, highly scalable, stable scripting language.  Python is both a programming language, and a collection of modules (small sections of code which perform some useful operations).  For example, there is a Python module called ‘math’ which performs trigonometric functions and angular conversions.  Python ships with a predefined set of modules which meet most of the needs for general scripting.  SciPy extends the basic functionality of Python by adding modules which perform functions useful to the scientific and engineering communities.

SciPy is not just one Python module but a collection of modules and is sometimes referred to as the SciPy Stack.  The SciPy Stack consists of seven core packages:

  • NumPy for numerical computation using arrays
  • SciPy, a collection of numerical algorithms
  • Matplotlib for 2D and 3D plotting
  • Pandas for high-performance data structures
  • SymPy for symbolic mathematics and computer algebra
  • IPython providing an interactive interface for quickly testing scripts and
  • Nose which allows you to test your Python code.

NumPy and Matplotlib have shipped with the ArcGIS platform for the last several releases.  Adding the remaining core packages will fully integrate the SciPy Stack into the ArcGIS platform.

SciPy ecosystem.


Matt:
Why is it being integrated with ArcGIS?

Kevin: SciPy is being integrated with the ArcGIS platform to make scientific and technical computing easier for our users.  While the SciPy Stack is open-source software and can be freely downloaded, it can sometimes be difficult to integrate with ArcGIS.  Why?  The strength of SciPy lies in its integration of many software modules, and installing these modules can be time consuming and error prone. For example, Pandas and Matplotlib may depend on a particular version of NumPy.  If the correct version of NumPy is not loaded on your computer, Pandas and Matplotlib may not function as expected.  Getting the correct versions of all of the components of the SciPy Stack can be challenging.  Since SciPy will be integrated into the ArcGIS platform, all of these interdependencies will be taken care of for you.

Integrating SciPy with ArcGIS will make developing scientific and technical geoprocessing tools and scripts easier and more efficient.  In other words, you won’t have to ‘start from scratch’ or ‘reinvent the wheel’ each time you start to solve a scientific or technical problem.  The SciPy Stacks represents tens of thousands of lines of useful code which has already been thoroughly tested and documented.  You can leverage the work of hundreds of other programmers and simply reuse code from one of the SciPy modules.

A custom Python geoprocessing tool designed to detect edges in an image. This tool leverages SciPy image filtering functions.

Another important reason that SciPy is being integrated with ArcGIS is that it will ensure that any custom scientific or technical geoprocessing tools you create are sharable. One of the fundamental design goals of Esri’s geoprocessing framework is the ability to distribute your tools among the ArcGIS community of users.  If you develop a custom geoprocessing tool which depends on SciPy, you can share that tool as a geoprocessing package or a geoprocessing service and not have to worry whether the person you’re sharing with has the correct versions of SciPy loaded on their computer.


Matt:
Can you give me some examples of what people might do with this?

Kevin: The possibilities of SciPy being integrated with ArcGIS are really only limited by the imagination of the GIS community.  Python, and by extension SciPy, is embedded in the ArcGIS platform in order to make automating repetitive geoprocessing task and extending the geoprocessing framework easier.  For the scientific and engineering communities, extending the geoprocessing framework will be the more important part.  The SciPy stack has so many rich functions that it is hard to image all of the creative spatial problems that GIS users with scientific or engineering domain expertise will be able to tackle—but here are a few that come to mind:

  • An atmospheric scientist may use the image filtering modules to delineate zones of horizontal transport of water vapor.
  • A transportation geographer may use the Markov chain modules to simulate traffic flow.
  • A geoscientist may use the symbolic mathematics routines to trace faults and model crustal movement.
  • An ocean scientist may use the calculus module to calculate ocean dynamics
  • A fisheries scientist or resource manager may use the linear algebra modules to set a harvest quota for a fish stock.


Matt:
Are there future plans for SciPy and ArcGIS?

Kevin: The future plans for SciPy and ArcGIS depend on the scientific and engineering practitioners in the GIS community.  SciPy is open-source software, meaning that it grows and improves by users contributing new and/or improved code or modules.  Newer versions of the SciPy stack will be integrated and supported in future releases of ArcGIS.  Look for the familiar “What new in ArcGIS 10.x” documents on the web for information regarding potential new versions of the SciPy Stack.


Matt:
How can our customers get it?

Kevin: SciPy will be integrated with ArcGIS through a staged release.  It will be available for ArcGIS Pro with the 10.3 release as an optional install. At the 10.3.1 release, SciPy will be automatically installed with ArcGIS for Desktop.

What is ArcGIS Pro and why is SciPy being integrated there first?  ArcGIS Pro is the new, premier application for visualizing, editing, and performing analysis.  It has a 64-bit architecture and leverages threading to keep the user interface responsive and to utilize additional CPU cores on the local machine (a very important feature when running computationally intensive scientific algorithms).

Matt Artz

About Matt Artz

Matt Artz joined Esri in 1989. In his current role as GIS and Science Manager, he helps communicate the value of GIS as a tool for scientific research and understanding. He writes extensively about geospatial technologies and manages the GIS and Science blog. Prior to joining Esri he worked as an Environmental Scientist at a large science and engineering consulting company, on such diverse projects as highway noise modeling, archaeological impact assessment, and chemical weapons disposal. His educational background includes an M.S. degree in Environmental Policy and Planning and a B.S. degree in Anthropology and Geography.
This entry was posted in Technology and tagged , , , , . Bookmark the permalink.

Leave a Reply

17 Comments

  1. Curtis Price says:

    Will the 10.3.1 release remain at Python 2.7.x?

  2. nmiller says:

    We currently run Python(x,y) alongside ArcGIS 10.2. This installs SciPy and other modules and allows us to import ArcPy into these scientific environments when required. With ArcGIS 10.3 and 10.3.1 including SciPy, what would your reccomendation be? Could we still install Python(x,y) and import ArcPy into it or would there be less need to do so now that many of the Python(xy) modules will now be included in ArcGIS?

    • Kevin Butler says:

      I think there will be less of a need to install Python(x,y) since SciPy will be available in ArcGIS. However, if Python(x,y) contains some functionality that is missing from SciPy, there is nothing that would prevent you from installing both.

  3. will_vj says:

    So from 10.3.1 I dont need to install seperately all the other site packages viz Panda, NumPy, I will have them all installed automatically under SciPy stack?
    Thank you.

  4. Curtis Price says:

    Just discovered netCDF4 is included in the 10.3 Python 2.7.8 stack. This is great, as it gives us some built in access right away to resources like OpenDAP from Desktop. There were DLL conflicts that have made it difficult to access netCDF4 from Python before 10.3 so we’re just thrilled about this!

  5. csburris says:

    I am currently using ArcGIS 10.3. It would be nice if I could install scipy (with Pandas, etc) without ArcGIS Pro, since I am not ready to upgrade yet, but am experiencing difficulty building it without messing with the versions of numpy and matplotlib that Desktop 10.3 installs.

  6. gcsrfx says:

    Hi,
    We’re using SciPy for python scripts that we want to publish to ArcGIS for Server as Geoprocessing services. It appears to be only 2.7.x 64-bit python. What is the status or time frame of SciPy for ArcGIS for Server?

    thanks

  7. Curtis Price says:

    Matt, please update the text of this article to reflect the fact that the science “stack” has been postponed until the 10.4 release. Thanks.

    • holisticbynature says:

      This really should be updated. I know for a fact I was hoping to use several signal processing functions in the release.
      To those who read this comment, it seems that this has been post-poned to 10.4 ArcGIS Release .

  8. dan_patterson says:

    Update time! It has been a year and news is slim. Any info would be appreciated.