Tag: Feature layers

Taking control of transformations with hosted feature layers (September 2017)

2017-09-25 08_59_48-B08 - Taking Control of Transformations.docx - Word

Imagine how simple the world would be if there was only one projection.  In our quest to flatten the earth onto paper and device screens, we’ve created numerous projections to suit our purposes. Each projection has a set of goals, … Continue reading

Posted in ArcGIS Online | Tagged , , , | 2 Comments

ArcGIS Online and ArcGIS Enterprise Early Adopters Community Available Now

OAE

With the June 2017 Release of ArcGIS Online and the 10.5.1 Release of ArcGIS Enterprise, new functionality to collaborate between ArcGIS Online Organizations and ArcGIS Enterprise Deployments is available. This new capability provides the functionality to copy Feature Layers and … Continue reading

Posted in ArcGIS Enterprise, ArcGIS Online | Tagged , , , , , , , | 4 Comments

Who would win? Matching layer types to common workflows

thumb

One of the great things about ArcGIS Online is having the multiple ways of delivering your data, it ensures that you can tailor it different ways and deliver the best experience for your many different audiences.  In a sea of … Continue reading

Posted in ArcGIS Online | Tagged , , , , | Leave a comment

Tips for creating hosted feature layers

Layers5

Feature layers are often the best type of layer to use for visualizing your vector data in web maps. They are easily placed on top of tiled basemaps for geographic context and support interactive queries and selection, on-the-fly styling, and … Continue reading

Posted in ArcGIS Online | Tagged , , , , , | Leave a comment

Esri picture marker symbol generator for JavaScript developers

The ArcGIS.com map viewer provides a nice collection of picture marker symbols that can be used to symbolize features on your map. If you’re a developer, you might wonder how you can use these symbols in your applications. You could potentially use tools like Firebug or Chrome Developer tools to inspect ArcGIS.com and discover the image URL, but to make things easier, we’ve created a sample that allows you to select a symbol and generate the JavaScript Object Notation (JSON) needed to draw the symbol:

When using this sample, just click a symbol and Continue reading

Posted in Web | Tagged , , , | 3 Comments

Visualizing time-aware data in a web map

You may have heard that ArcGIS 10 is “time-aware”. This simply means that you can designate one or more fields in your data as containing date/time values, and then ArcGIS can play back the temporal progression of that data using a “time slider” bar. There are time sliders built into ArcMap and ArcGIS Online, but you can also add them to web apps that you build with JavaScript, Flex, or Silverlight. Showing the temporal progression of data like this was technically possible before ArcGIS 10, but it required a lot more coding.

To demonstrate one way that time-aware GIS data can be put in a web app, I created this JavaScript map of fish migration up the Columbia and Snake Rivers earlier this year. The dams along the river have fish ladders, where each fish is counted as it swims up the river. Visualizing the daily fish counts over time is an excellent way to see migration patterns. Take a few seconds to give the app a try.

Daily Fish Migration at Northwest US Dams

In this post I’ll explain a little about how I built this app and where you can get the code.

Getting the data and making it time-aware

The raw fish count data came from the Fish Passage Center website, www.fpc.org. Here I was able to download comma-separated value (CSV) files showing fish counts for each dam in 2011. These CSV files are similar to what you might be able to export from Microsoft Excel. To make this data work on a map, I also had to have the dam locations, so I created a point feature class and heads-up digitized each dam.

Digitized dams

Sometimes time-aware datasets are structured differently from traditional datasets you might use for a paper map. What I really needed for this web app was not one record for each dam, but rather, one record for each dam for each day. Thus, what looks like a simple dataset with only a handful of points, really contains over 800 records.

With a little bit of Python, I was able to read the fish counts out of the CSV files and write them as attributes for each dam for each day. Here’s a portion of the final table:

Attribute table of dams dataset

In order to make this type of data time-aware in ArcGIS, you have to add it to ArcMap. Then open the Layer Properties and click the Time tab. Here is where you tell ArcGIS about your dataset and how its time fields should be interpreted.

Time tab of Layer Properties dialog box

One of the first settings you need to choose is whether the layer has a single time field, or both start and end time fields. Some phenomena, like fish counts, are taken at one point in time and just have one field. Other things might have a field recording both the start and end time of an event, such as a wildfire that begins on June 9 and ends on June 13.

You can also indicate which field holds the time values, as well as the time extent and intervals that should be displayed in the time slider.

In my particular map, I wanted the same service to expose data about four fish types. So I made four copies of the layer, symbolized them with proportional symbols, and made them all time aware. Below is what the final map looked like. (It’s okay that all layers are turned on and overlap, because my app only requests one layer at a time from the map service.)

Map of dams in ArcMap before publishing

Exposing time-aware layers through a web service

At this point I was ready to publish the map as a service. I saved a map service definition (MSD) file (yes, these support time-aware layers) and published it to ArcGIS Server in the usual way. All of the time information saved in the map is accessible through the map service. You can verify this by looking at a layer in the Services Directory:

Time-aware properties viewed in the Services Directory

One easy way to view a time-aware map service is to add it to the ArcGIS.com map viewer (go to www.ArcGIS.com and click Map to get to this viewer). The viewer recognizes any time-aware services you add and automatically gives you a time slider. In my case, I wanted to go a little further and make my own web app with a time slider. I chose the ArcGIS API for JavaScript because I had a bit of previous experience with it, and my adjacent co-workers happen to know this API very well. (I can’t deny asking for the occasional hint!)

Building a time-aware web map with JavaScript

When you’re starting out with the ArcGIS web APIs, a good strategy is to find some of the online samples that do the things you want to do, combine them into an app, and then modify the result to fit your requirements. I decided I wanted to use elements from these two samples:

  • Overview map sample from the Layouts book. You’ll see this is where I got the overview map, color scheme, styling, and partitioning of the window.
  • Layer Definition sample from the Time book. This is where I got the time slider and the legend.

The fish counts update very quickly because the app uses client-side graphics to display them. In other words, the browser makes a request to the map service saying essentially: “Give me all the point vertices for the dams from April 17 – June 30, and their attributes. I’ll do the work of drawing them.” After the server sends the response, the browser can draw the points and their symbols without ever having to go back to the server for more information. Because I’m dealing with fewer than 1000 records, I don’t have to worry about hitting the map service’s default limit for how many features it will return, but at the same time I am not requesting an unreasonable number of points for the browser to display.

The key to displaying the graphics this easily is to use the FeatureLayer class. Here’s where I create it in my code:

var fishLayer = new esri.layers.FeatureLayer(initialFishLayerUrl, {
  mode: esri.layers.FeatureLayer.MODE_SNAPSHOT,
  outFields: ["*"]
});

When you connect a FeatureLayer to a TimeSlider, the time slider does all the work for you of requesting and displaying the appropriate features for the currently displayed time slice.

The time slider is a widget included with the ArcGIS API for JavaScript. To use it, I had to put the following at the top of my code:

dojo.require("esri.dijit.TimeSlider");

Then, I had to create the time slider and set some properties on it, such as the thumb moving rate (how fast the slider plays) and whether the time slider should replay itself when it gets to the end (accomplished with the setLoop method).

// Create and configure the time slider
var timeSlider = new esri.dijit.TimeSlider({
}, dojo.byId("timeSliderDiv"));
map.setTimeSlider(timeSlider);
timeSlider.setThumbCount(1);
var layerTimeExtent = results[0].layer.timeInfo.timeExtent;
timeSlider.createTimeStopsByTimeInterval(layerTimeExtent, 1, 'esriTimeUnitsDays');
timeSlider.setThumbMovingRate(350);
timeSlider.singleThumbAsTimeInstant(true);
timeSlider.setLoop(false);
timeSlider.startup();

If you use the time slider in the ArcGIS API s for Flex or Silverlight, you’ll see similar properties.

You can handle events on the time slider to control what happens when the “thumb” on the slider moves. The drawing of the fish counts is handled automatically, but I did add some code to change the date label that appears directly below the slider.

// Report the date whenever the time slider moves
dojo.connect(timeSlider, "onTimeExtentChange", function(timeExtent){
  map.infoWindow.hide();
  dojo.byId("timeSliderLabel").innerHTML = dojo.string.substitute("<b>${startTime}, 2011</b>", timeExtent, function(val) {

      return dojo.date.locale.format(dojo.date.add(val, "hour", hourOffset), {
      selector: 'date',
      datePattern: 'MMM dd' // http://www.unicode.org/reports/tr35/tr35-4.html#Date_Format_Patterns
    });
  });
});

The above code results in a label that changes as the slider moves:

Label that changes with the time slider

Conclusion and download link

This post has shown how you can start with raw tabular data and ultimately display it in a time-aware web application. The coding part is optional and can be eliminated if you want to use the time slider in ArcMap or in ArcGIS Online. However, because many of you may have to write similar types of web apps, I’ve made the complete application and a readme file available for download at http://esriurl.com/fish.

Contributed by Sterling Quinn of the ArcGIS Server software development team

Posted in Services | Tagged , , , | 5 Comments

High performance web apps with big datasets as feature layers

High performance feature layer applicationDealing with big datasets is a frequent topic of discussion among web mapping app developers. The prevalence of huge, crowdsourced datasets ensures that big data will continue to be a common point of discussion when you get more than two web mapping geeks in the same room. The traditional approaches of generating a raster tile cache or using a dynamic map service work, but if you want any interactivity with the underlying data, it requires a server round trip. This post will discuss some of the key concepts behind building a web app that provides an extensive interactive experience while maintaining responsiveness and performance.

The main factor causing big data to be a tricky topic is that browsers cannot elegantly handle upwards of a few thousand features. Send more than a few thousand points, or a few hundred polygons to a browser and you can watch it grind to a halt, especially if you’re running a legacy version of Internet Explorer. This is a problem because slow performance is a death knell for web apps. If someone using your web app has to wait ten seconds, twenty seconds, or longer for five megs of JSON data to be retrieved and displayed, there’s a good chance they’ll just give up and move on. What good is a web app that no one uses?

At some point, you cross a threshold where it’s not practical to send all of your data across the wire and let the client sort it out. That threshold is an ambiguous one (and was explored in a previous post), but you know when you hit it. If you’re a JavaScript developer, how should you deal with big data? If you’ve read my recent posts, this answer will not surprise you: Feature Layers.

Building on the concepts discussed in the previous feature layer posts (generalizing features and vector tiling) and client side graphics performance post, I’ve put together an application that demonstrates how to use Feature Layers to set up custom scale dependencies and follow best practices to ensure performance does not suffer when querying, retrieving and displaying large datasets in an ArcGIS API for JavaScript application: High Performance Maps with Feature Layers.

The goal was to display appropriate US boundaries (states, counties or census block groups) for the current map scale. For example, trying to display census block groups for the whole US would overwhelm the browser with features. This app makes sure that only the states appear at the small scales, counties at the medium scales, and census block groups at the large scales. This ensures that a manageable number of features are always being transferred and displayed.

The app is simple, but the concepts can be applied to just about any web mapping app that has to deal with big data. The total size of the data used by the app is around a couple of hundred megabytes. This is accomplished through custom scale dependencies. This is as simple as creating a few feature layers, listening for their onLoad event and setting their minScale and maxScale properties. Here’s a simple code sample showing how this is done:

var fl = new esri.layers.FeatureLayer(url, options);
dojo.connect(fl, ‘onLoad’, function() {
fl.minScale = minScale;
fl.maxScale = maxScale;
});

The app includes a couple of other bells and whistles that demonstrate some of tools that are available when you have your features client side. The first is a rich Popup that displays feature attributes, as well as an option to zoom to the feature when you touch a graphic. The second is the ability to filter features based on population using a slider. Neither of these would perform nearly as well if features weren’t available client side.

The final point to drive home is that the app’s performance is kept snappy because only the required data is being sent to the client. For geometries, that means using maxAllowableOffset to generalize geometries on the server before they’re sent to the client (see my previous post on maxAllowableOffset and generalizing features on the fly for more info). For attributes, only the fields required by the Popup are sent across the wire.

Contributed by Derek Swingley of the ArcGIS API for JavaScript development team

Posted in Services | Tagged , , , , , | 2 Comments

Determining limits for map graphics

We’ve recently been talking about some of the benefits of Feature Layers. Since Feature Layers are a subclass of GraphicsLayer, we thought it was appropriate to spend a little time talking more about map graphics, limits on how many can be added to a map, and how to find the appropriate limit for your app/environment.

“How many graphics can I put on my map?” or “How many points can I put on a map?” are questions that continue to be asked. Probably because we keep saying “it depends.” While technically correct, “it depends” is not helpful on its own. So, with the lofty goal of providing some actionable guidelines, I’d like to discuss two scenarios: point graphics and polygon graphics. To aid in this discussion, I’ve put together a simple app for each scenario: Display an Arbitrary Number of Points and Incrementally Add U.S. County Polygons To a Map.

The objective of the point graphics app is to see how many graphics you can add to a map before performance starts to suffer. Since this is dependent on many factors (your browser, your hardware, your definition of “suffer”), it’s probably best to let you see for yourself how a map performs with 100, 1,000 and even 10,000 graphics. I wouldn’t try 10k in IE, but Chrome and even Firefox hold up well (at least on my machine, YMMV).

There is also an option to select symbol type. Not all point symbols are equal in terms of performance so it’s important to realize that 500 picture marker symbols might result in a more sluggish app than one with 1,000 simple marker symbols.

But wait, just because you can put 3,000 points on your map and not see a performance penalty, does that mean you should? My answer, nine times out of ten, is no. Performance is one thing but user experience is another. Does putting a map with thousands of point graphics on a 15” screen provide a meaningful user experience? Again, I would say no. Outside of looking for a general spatial distribution pattern, I don’t think you’re helping a user discover much or answer any particular question by showing thousands of point graphics at one time. If you find yourself in this situation, it’s a good time to consider using a raster surface (like a heatmap), filtering your data through queries, or maybe clustering.

The story for polygon graphics is a bit more complex. Not only do you need to worry about the number of polygons, but the number of polygon vertices is also an important consideration. More detailed geometries take longer to transfer to the client, and once they make it to the client, they require more resources to display.

Like the point graphic app, the polygon app linked above lets you add any number of polygons to a map (but you probably should keep it under 1000), and it also allows you to specify whether or not maxAllowableOffset should be used. If maxAllowableOffset is a new term for you, please read my previous post to catch up.

Let’s look at a couple of reasons to always use maxAllowableOffset. The image below shows geometries for some states in the Northeastern United States:

Northeast states graphics comparison

The geometries on the left were fetched using a query that set maxAllowableOffset appropriately. The geometries on the right are not generalized. The generalized geometries look nearly identical to the non-generalized geometries (maybe even better according to some) but they are less than a quarter of the size. That’s the key takeaway and why we use maxAllowableOffset: transfer the minimum level of detail required for effective display.

One more example:

Mississippi graphics comparison

I think this example really drives the point home. Using maxAllowableOffset, you’re transferring ~44KB to show the county boundaries for Mississippi. Without maxAllowableOffset, you push over 3MB across the wire. If you’re setting maxAllowableOffset correctly, you’re not likely to be sending more than 100KB (and usually a lot less) per query to the client regardless of the level detail present in your geometries.

The total number of polygon graphics on a map is something that you should experiment with but I would recommend to keep it below a few hundred.

We still don’t have a hard and fast rule for the number of graphics on a map (and probably never will), but hopefully this post along with the accompanying apps will make it easier for you to find your own answer.

Contributed by Derek Swingley of the ArcGIS API for JavaScript development team

Posted in Services | Tagged , , , , | 6 Comments

Feature layers can generalize geometries on the fly

Last week, we discussed vector tiling for improving performance of feature layers. While awesome in its own right, if you’re trying to push twenty megabytes of features over the wire, vector tiling isn’t going to save you. Your app will be slow.

How should complex geometries be handled/served? The intuitive approach is to use multiple layers in a service and, depending on the map’s scale, display different layers with varying levels of geometric detail. And why not? You have the tools and you know the API will let you easily craft a solution; it’ll just take a little time to code the thing.

Well, frankly, that’s a lot of work. Feature generalization is a common enough scenario that you would think the API should just know how to do this type of thing. Thankfully, it does.

When you create a Feature Layer, one of the options is maxAllowableOffset. You might recognize this parameter name from the ArcGIS Desktop generalization tools, where it’s been used for years. For more technical details, please refer to the Wikipedia page for the Douglas–Peucker algorithm. If this is new terminology, go check out the ArcGIS Desktop Help for the Generalize tool.

Another bonus is that the FeatureLayer comes with a method, setMaxAllowableOffset, that you can use to simplify features on the fly. Since the web APIs fire an event when the map’s extent/zoom level changes, you can listen for this event and use setMaxAllowableOffset to indicate an appropriate value.

OK, so now all we have to do is figure out a value for maxAllowableOffset. This can take some trial and error, but let me suggest a method: a feature’s geometry should not display more than one vertex per pixel. After all, a pixel is the smallest unit for our displays, so displaying more than one vertex per pixel is wasted effort. If we want to display no more than one vertex per pixel, we can take the map’s width in coordinate space divided by the map’s width in pixels and we have a reasonable value for maxAllowableOffset. With this method, you are telling your Feature Layer to make sure simplified features do differ from the original by more than the width of a pixel. In fact, this is exactly what happens in the Generalize Features JavaScript API sample.

But wait, there’s more! Because this is such a useful idea, ArcGIS.com does it for you automatically. Whenever you add a feature layer to an ArcGIS Online map, maxAllowableOffset is calculated and set appropriately. This results in a better performing map and feature layers that load faster than if complex geometries were sent across the network.

Update: As of version 2.7 of the JS API, feature layers also automatically set maxAllowableOffset for you.

Finally, one caveat. You can’t specify maxAllowableOffset on layers that you allow users to edit. If you’re editing a feature, you need to see the true representation of that feature, including all its participating vertices. Editing a generalized feature and then sending it back to the server could wipe out detailed information that was meticulously created and maintained. We wouldn’t want that, so you can’t specify maxAllowableOffset on Feature Layers that are editable.

Contributed by Derek Swingley of the ArcGIS API for JavaScript development team

Posted in Services | Tagged , , | 3 Comments

Out of the box vector tiling using feature layers

Feature Layers arrived with ArcGIS 10 and the 2.0 release of the ArcGIS web APIs. They provide a flexible way to retrieve (and optionally edit) vector features from a web service. To highlight some of the advantages and innovative aspects of feature layers, we’ll be sharing a series of blog posts about them in the weeks to come. The focus of this post (if you haven’t already guessed): vector tiling.

Vector tiling is a relatively new topic in the realm of web maps. Perhaps you’ve used the raster tiles that you can create en masse with ArcGIS Server to speed up your web app. Vector tiles are a little different. The underlying idea is that a static grid is used to issue a series of requests to retrieve vector features. The features are added to the map, but they are also stored in your browser cache.

What’s the motivation behind vector tiling? Some benefits include:

  • Dynamic interaction with individual features to change feature symbols, make selections, or work with feature attributes without going back to the server
  • Features are retrieved from the browser cache when the same request (URL) is issued more than once (think HTTP status code 304)
  • Only features that are in the map’s current extent are automatically retrieved
  • Improved user experience and performance by using multiple queries to retrieve features in reasonable numbers rather than a single query to retrieve all features

One key thing to point out, especially for veteran uses of ArcGIS Server that are familiar with traditional map caches, is that vector tiles are generated on the fly and they are not persisted on the server. For vector tiles, there’s no cache building and no additional server maintenance required. Vector tiles are only cached on the client and are used on a session-by-session basis.

As the title of this post says, you get vector tiling out of the box with ArcGIS Server and Feature Layers. To take advantage of a Feature Layer’s vector tiling capabilities, the key is to use ONDEMAND mode. The code to set this up can be as simple as:

var featureLayer = new esri.layers.FeatureLayer(layer_url ,{
  mode: esri.layers.FeatureLayer.MODE_ONDEMAND,
  outFields: ["*"]
});

Let’s look at a sample: JavaScript API Feature Layer ONDEMAND. If you look for it, you can see hydro features being drawn in chunks as you move around the map. These are the tile-based queries doing their thing. To explore further, pop open Firebug or the Chrome Developer Tools and take a look at the Net tab or the Network pane. Pan to get the feature layer to make some requests back to the server. Initially, you’ll see requests to the server to retrieve features. If the map ends up somewhere it’s been before, you’ll start seeing HTTP 304s being returned:

304 errors

What does a 304 mean? In a nutshell, it’s the server saying it’s already responded to this request (URL) and that the browser should use the previous response. When this happens, the server doesn’t send any features over the wire and the browser uses the previous response by pulling it out of the browser cache. Pretty cool. Because the Feature Layer uses a grid to generate query URLs, and the browser caches responses to these URLs, the server doesn’t need to send the same features over the wire multiple times.

One issue that vector tiling doesn’t directly address is generalizing features. As a map scale changes, the amount of detail required for each feature changes. If you’re at a small scale, you need more generalized features. At larger scales, features need to show more detail. Feature Layers can handle this efficiently but that is a topic for another post. For those who can’t wait and want to explore on their own, have a look at the maxAllowableOffset property on the FeatureLayer.

Contributed by Derek Swingley of the ArcGIS API for JavaScript development team

Posted in Services | Tagged , , | 6 Comments