Tag: Big Data

A Fix for the Create Space Time Cube Tool in GeoAnalytics Server

The Create Space Time Cube tool included with GeoAnalytics Server summarizes a set of points into a netCDF file structure by aggregating them into space-time bins. In addition to summarizing the number of features, the tool can also calculate summary statistics for the attribute values of points within each space-time bin. Continue reading

Posted in Analysis & Geoprocessing, ArcGIS Enterprise, Spatial Statistics, Uncategorized | Tagged , , , , , , , , , , , , , , , , , , | Leave a comment

Managing Imagery Data? Start Here!

Enterprise Image Management white paper

When people think about managing data, I can’t imagine anything as intimidating to an IT person as imagery data.  Why you may ask?  Because the file size is large, very large, the formats can be unfamiliar, and after you’ve acquired … Continue reading

Posted in Analysis & Geoprocessing, ArcGIS Enterprise, Imagery, Mapping, National Government, Services | Tagged , , , , , , | 1 Comment

An Introduction to Big Data

ArcGISHadoop

Two years ago the Big Data team released GIS Tools for Hadoop on GitHub. GIS Tools for Hadoop is an open source project that allows users to integrate Hadoop (a distributed big data platform) with big spatial data, complete distributed … Continue reading

Posted in Geodata | Tagged , , , | 5 Comments

New Spatial Aggregation Tutorial for GIS Tools for Hadoop

3.png

The Big Data team is excited to offer a new tutorial on spatial aggregation (sometimes called spatial binning). Spatial aggregation is extremely useful in summarizing big data to gain a meaningful snapshot of patterns in your data. Spatial aggregation works … Continue reading

Posted in Geodata | Tagged , , , , , , | 3 Comments

Setting up a small budget Hadoop Cluster for Big Data Analysis

At the 2014 Esri User Conference, the Big Data team gave several presentations, including two technical workshops entitled: ‘Big Data and Analytics: The Fundamentals’ and ‘Big Data and Analytics with ArcGIS’. We presented our open source GIS Tools for Hadoop (shared on GitHub), as well as some research that we’re currently pursuing (exciting things to come!). We gave demos using both our open source tools as well as the prototype tools being currently researched.

For the demos (source data consisted of > 170 million data points that represent all the taxi cab trips in New York City in 2013), we ran all of our analytics on a Hadoop cluster back in Redlands. A twenty node cluster may seem like a big investment (and it can be); but, it doesn’t have to be. Enter the DREDD cluster… Continue reading

Posted in Geodata | Tagged , , , | 8 Comments

Big Data ST_Geometry Queries up to 20X Faster in Hive

The Big Data development team at Esri is excited to announce a major performance speedup in ST_Geometry for Hive, which is part of Esri’s open-source Spatial Framework for Hadoop.  The amount of performance gain depends on the type of spatial query run and on the size of the table in Hive.  The biggest gain comes with relational operations such as ST_Contains and ST_Overlaps.  In general, the performance gain will be greater with larger tables — exactly where it helps the most.

Continue reading

Posted in Geodata | Tagged , , , , , , | 1 Comment

The NSF EarthCube Initiative and Esri’s Role

d1

This post also appears in Esri Insider. Last update: April 1, 2014. With all the recent excitement and good hopes over the White House Climate Data Initiative, and the ongoing progress of the Group on Earth Observation System of Systems (GEOSS), … Continue reading

Posted in Analysis & Geoprocessing, ArcGIS Online, Hydro, Imagery, Location Analytics, National Government, Oceans & Maritime, Services | Tagged , , , , , , , , , , , | Leave a comment

Ocean pH Modeled with Latest CMIP5 Data

Assessment Report 5 (AR5) offers opportunities to model the marine environment and its wide range of biophysical ocean variables, improving AR4 ocean model shortcomings (Griffies et al., 2010; Mora et al., 2013). More than 40 variables are available from limited GCM runs, and processing of some biogeochemical models is now available for application through Esri ArcGIS tools and namely SimCLIM for ArcGIS/Marine. The currently available variables include sea surface temperature, net primary productivity of carbon by phytoplankton; and dissolved nitrate concentration a; dissolved oxygen concentration; pH; dissolved phosphate concentration; total alkalinity; dissolved iron concentration; and dissolved silicate concentration, all at the surface. For more information see www.climsystems.com

Posted on behalf of and authored by Dr. Peter Urich, Managing Director of CLIMsystems Ltd. Esri’s Silver Business Partner CLIMsystems Ltd., located in Hamilton, New Zealand, has been exhaustively researching and processing the latest ocean data from the Coupled Model … Continue reading

Posted in Analysis & Geoprocessing, Hydro, Oceans & Maritime | Tagged , , , , | Leave a comment

Esri’s New Partnership with GEOSS

geoss

This post also appears in Esri Insider. Last update: January 26, 2014. Over the past week we heard quite a bit about the polar vortex (not a new term, by the way) as North America struggles with some of the … Continue reading

Posted in Analysis & Geoprocessing, ArcGIS Online, Hydro, Imagery, National Government, Oceans & Maritime, Public Safety | Tagged , , , , , , , , , , , , , , , , , | Leave a comment

What is CyberGIS?

A summary graphic of ArcGIS Online, presented by Esri's Steve Kopp at CyberGIS AHM '13, highlighting the full breadth of "cyberGIS" capabilities to date, including various web service types, computing platforms served, API languages used, and the connection of ArcGIS Online to the high-performance, supercomputing environment of the CyberGIS Gateway.

This post also appears in Esri Insider. In an earlier Esri Insider post, I had mentioned Esri’s involvement in the large National Science Foundation-funded project known as CyberGIS, which aims to establish a fundamentally new software framework via a seamless … Continue reading

Posted in Analysis & Geoprocessing, ArcGIS Online | Tagged , , , , , , , , , , , | 2 Comments