Using Yahoo! Pipes to create dynamic map layers

Yahoo! Pipes are a handy way to aggregate web feeds and other information from various sources. One way to think of Pipes is as a way to create data/information mashups, just like we can create ArcGIS Online web maps by mashing up a variety of map layers. An interesting aspect of Pipes is that we can leverage them in the context of our ArcGIS Online web maps, and use them to create dynamic layers using URL-located KML or CSV files as mappable information sources.

If your .csv or .txt file is stored at a public website and contains lat/long, you can reference it as a layer in your web map. The key word is reference; when we open the map or refresh it the layer source is refreshed from its web location. If the source is updated, we automatically see those changes in our web map every time we use it. For more information see our post on Authoring maps that update using CSV files.

Yahoo! Pipes run when we use them, so when the map is opened or refreshed the pipe runs and we see new data in our map.

Our goal is to use Pipes to create web-located CSV and KML files which we will use in our web map. We’ll build upon our previous post covering how to Add Flickr photos to ArcGIS Online web maps and use Pipes to create a dynamic layer displaying geotagged photos from Flickr. You’ll need a Yahoo! account, a Flickr account, a Flickr API key, and a little knowledge about Pipes. You can view a YouTube tutorial (which even uses Flickr as an example) to get up to speed with what you need to know.

First, sign-in to Yahoo and go to Pipes. Click Create Pipe to get started.

You’ll see the pipe layout screen appear. Along the left side under Sources you will find the Flickr source. Click to add it to your pipe layout.

Rather than build a pipe from scratch, we’ll take the easy route and borrow from an existing one. When you clicked the Flickr source, at the bottom left of the layout an information window appears, with a link to About this module

The Flickr module example, handily enough, is one that searches for all geotagged Flickr photos of the Chicago skyline. You can’t run the pipe directly because it requires a Flickr API key. Instead, click View source:

In the pipe layout enter your API key. Note the keywords that are being searched, and that the results of the source stream are being filtered (another tool in the pipe layout dashboard) to ensure that all results have a valid location:

Save a copy of the pipe, and Run Pipe.

You’ll see the results displayed on a Yahoo! map:

We want to use this pipe to create a dynamic layer in our ArcGIS Online web map. There’s a couple of ways to accomplish this – by using KML as the output or by using CSV as the output. First, we’ll use the KML.

In the upper right corner of the map find More options, then right-click Get as KML, then Copy Link Location (note that we’re using Firefox, and your right-click options will differ using another browser).

Open a new ArcGIS Online map, click Add, then Add Layer from Web.

Check the KML radio button and paste the URL to add the layer:

The pipe will run each time we open or refresh the map, and display the Flickr results.

In the pipe URL we can change the render parameter from KML to CSV to generate a CSV file instead. Change render=kml to render=csv.

Use Add, Add Layer from Web to add the CSV to your map. Adding the pipe using a CSV file provides additional opportunities to configure the pop-up for the Flickr results.

For more information see Adding items from the Web and Configuring pop-up windows.

Thanks to my colleague Christophe Tourret of Esri France for the suggestion about using Yahoo! Pipes.

This entry was posted in ArcGIS Online and tagged , , , , . Bookmark the permalink.

Leave a Reply

One Comment

  1. kmayall says:

    I created a web map with two layers, both to the same Flickr pipe, one as kml and one as csv.
    I changed the pipe by adding a keyword filter on the title text. I reloaded the map page several times but the map was not changing dynamically. After several tries, the csv layer updated but the kml layer did not. But the source url for both layers is identical except for the render parameter being csv or kml. Then I removed the keyword filter from the pipe, saved it, ran it, re-grabbed the kml URL, created a new web map, added a web csv layer, pasted in the URL, changed the render parameter to csv, and the map displayed the pipe as if it still had the keyword filter turned on.
    In all cases the Yahoo Pipe id is the same. Is this being cached somewhere…?