**An interview with Kevin Butler about the integration of ArcGIS and SciPy**

Geography is the science of our world, and GIS is a foundational technology for helping us to better understand that science. To further strengthen the link between GIS and science, today at the Esri Ocean GIS Forum we’re pleased to announce the integration of ArcGIS with SciPy, a Python-based ecosystem of open-source software for mathematics, science, and engineering.

I recently caught up with Kevin Butler, a Product Engineer with the Geoprocessing and Analysis Team, to ask him a few questions about the integration between ArcGIS and SciPy.

*
Matt Artz*: What is SciPy?

*Kevin Butler*: SciPy is a software library that helps scientists, engineers, and GIS analysts perform custom scientific and technical computing. It is an open-source library built using Python—an easy to learn, highly scalable, stable scripting language. Python is both a programming language, and a collection of modules (small sections of code which perform some useful operations). For example, there is a Python module called ‘math’ which performs trigonometric functions and angular conversions. Python ships with a predefined set of modules which meet most of the needs for general scripting. SciPy extends the basic functionality of Python by adding modules which perform functions useful to the scientific and engineering communities.

SciPy is not just one Python module but a collection of modules and is sometimes referred to as the SciPy Stack. The SciPy Stack consists of seven core packages:

- NumPy for numerical computation using arrays
- SciPy, a collection of numerical algorithms
- Matplotlib for 2D and 3D plotting
- Pandas for high-performance data structures
- SymPy for symbolic mathematics and computer algebra
- IPython providing an interactive interface for quickly testing scripts and
- Nose which allows you to test your Python code.

NumPy and Matplotlib have shipped with the ArcGIS platform for the last several releases. Adding the remaining core packages will fully integrate the SciPy Stack into the ArcGIS platform.

*
Matt:* Why is it being integrated with ArcGIS?

*Kevin:* SciPy is being integrated with the ArcGIS platform to make scientific and technical computing easier for our users. While the SciPy Stack is open-source software and can be freely downloaded, it can sometimes be difficult to integrate with ArcGIS. Why? The strength of SciPy lies in its integration of many software modules, and installing these modules can be time consuming and error prone. For example, Pandas and Matplotlib may depend on a particular version of NumPy. If the correct version of NumPy is not loaded on your computer, Pandas and Matplotlib may not function as expected. Getting the correct versions of all of the components of the SciPy Stack can be challenging. Since SciPy will be integrated into the ArcGIS platform, all of these interdependencies will be taken care of for you.

Integrating SciPy with ArcGIS will make developing scientific and technical geoprocessing tools and scripts easier and more efficient. In other words, you won’t have to ‘start from scratch’ or ‘reinvent the wheel’ each time you start to solve a scientific or technical problem. The SciPy Stacks represents tens of thousands of lines of useful code which has already been thoroughly tested and documented. You can leverage the work of hundreds of other programmers and simply reuse code from one of the SciPy modules.

Another important reason that SciPy is being integrated with ArcGIS is that it will ensure that any custom scientific or technical geoprocessing tools you create are sharable. One of the fundamental design goals of Esri’s geoprocessing framework is the ability to distribute your tools among the ArcGIS community of users. If you develop a custom geoprocessing tool which depends on SciPy, you can share that tool as a geoprocessing package or a geoprocessing service and not have to worry whether the person you’re sharing with has the correct versions of SciPy loaded on their computer.

*
Matt:* Can you give me some examples of what people might do with this?

*Kevin:* The possibilities of SciPy being integrated with ArcGIS are really only limited by the imagination of the GIS community. Python, and by extension SciPy, is embedded in the ArcGIS platform in order to make automating repetitive geoprocessing task and extending the geoprocessing framework easier. For the scientific and engineering communities, extending the geoprocessing framework will be the more important part. The SciPy stack has so many rich functions that it is hard to image all of the creative spatial problems that GIS users with scientific or engineering domain expertise will be able to tackle—but here are a few that come to mind:

- An atmospheric scientist may use the image filtering modules to delineate zones of horizontal transport of water vapor.
- A transportation geographer may use the Markov chain modules to simulate traffic flow.
- A geoscientist may use the symbolic mathematics routines to trace faults and model crustal movement.
- An ocean scientist may use the calculus module to calculate ocean dynamics
- A fisheries scientist or resource manager may use the linear algebra modules to set a harvest quota for a fish stock.

*
Matt:* Are there future plans for SciPy and ArcGIS?

*Kevin:* The future plans for SciPy and ArcGIS depend on the scientific and engineering practitioners in the GIS community. SciPy is open-source software, meaning that it grows and improves by users contributing new and/or improved code or modules. Newer versions of the SciPy stack will be integrated and supported in future releases of ArcGIS. Look for the familiar “What new in ArcGIS 10.x” documents on the web for information regarding potential new versions of the SciPy Stack.

*
Matt:* How can our customers get it?

*Kevin:* SciPy will be integrated with ArcGIS through a staged release. It will be available for ArcGIS Pro with the 10.3 release as an optional install. At the 10.3.1 release, SciPy will be automatically installed with ArcGIS for Desktop.

What is ArcGIS Pro and why is SciPy being integrated there first? ArcGIS Pro is the new, premier application for visualizing, editing, and performing analysis. It has a 64-bit architecture and leverages threading to keep the user interface responsive and to utilize additional CPU cores on the local machine (a very important feature when running computationally intensive scientific algorithms).

Will the 10.3.1 release remain at Python 2.7.x?

Hi Curtis, my understanding is that the 10.3.1 release will continue tracking Python 2.7.x.

We currently run Python(x,y) alongside ArcGIS 10.2. This installs SciPy and other modules and allows us to import ArcPy into these scientific environments when required. With ArcGIS 10.3 and 10.3.1 including SciPy, what would your reccomendation be? Could we still install Python(x,y) and import ArcPy into it or would there be less need to do so now that many of the Python(xy) modules will now be included in ArcGIS?

I think there will be less of a need to install Python(x,y) since SciPy will be available in ArcGIS. However, if Python(x,y) contains some functionality that is missing from SciPy, there is nothing that would prevent you from installing both.

So from 10.3.1 I dont need to install seperately all the other site packages viz Panda, NumPy, I will have them all installed automatically under SciPy stack?

Thank you.

You are correct. At 10.3.1, the ArcGIS install will include Scipy, pandas, Numpy, sympy, and matplotlib.

Just discovered netCDF4 is included in the 10.3 Python 2.7.8 stack. This is great, as it gives us some built in access right away to resources like OpenDAP from Desktop. There were DLL conflicts that have made it difficult to access netCDF4 from Python before 10.3 so we’re just thrilled about this!

I am currently using ArcGIS 10.3. It would be nice if I could install scipy (with Pandas, etc) without ArcGIS Pro, since I am not ready to upgrade yet, but am experiencing difficulty building it without messing with the versions of numpy and matplotlib that Desktop 10.3 installs.

We are looking into using anaconda to set up a virtual environment that plays nice with ArcGIS. Would be nice if Esri teamed up with those guys to make a more flexible solution.

Hi,

We’re using SciPy for python scripts that we want to publish to ArcGIS for Server as Geoprocessing services. It appears to be only 2.7.x 64-bit python. What is the status or time frame of SciPy for ArcGIS for Server?

thanks

The goal is to have SciPy included with the 10.4 release (Server and Desktop) which will tentatively ship fourth quarter of 2015.

So you’ve changed plans? The article above says:

At the 10.3.1 release, SciPy will be automatically installed with ArcGIS for Desktop.

So I am looking at it with the 10.3.1 release – not there! Bummer!

Curtis:

The release strategy changed after the blog was released. The full strategy is outlined in this video from the Developer Summit: : http://video.esri.com/watch/4231/python-for-the-arcgis-platform

So when will it be available? Still not clear to me. Please update the article, we have to read all the comments to understand that it’s giving wrong information.

Matt, please update the text of this article to reflect the fact that the science “stack” has been postponed until the 10.4 release. Thanks.

This really should be updated. I know for a fact I was hoping to use several signal processing functions in the release.

To those who read this comment, it seems that this has been post-poned to

10.4 ArcGIS Release.Update time! It has been a year and news is slim. Any info would be appreciated.