A lot of us have been using the Business Analyst (BA) line of products in a variety of different ways. Business Analyst gives us access to business, consumer and demographic datasets along with capabilities to allow us to analyze our various markets, customers or competition. These interactions have usually been through focused applications:
- BA for Desktop
- BA Online
- Web applications utilizing the BA Online APIs
- BA Server
- Community Analyst
These are typically all forms of client side driven analysis where we ask a question of the system and we get an answer back immediately. In some cases though we need to use the information & analysis that BA can provide as part of a larger business process e.g. tax reporting, outage management reporting…
Such business processes are likely to be running as part of a server side process, wont necessarily be initiated through any client side interactions, and may even take many hours to complete. The processes may just happen as a result of new data coming into the system or at a given time of the day/month/year e.g. a new network outage area, a quarterly reporting cycle is initiated.
So how do we use Business Analyst functionality in a server side manner?
Well BA Online does provide developer level APIs in order to access BA data and services. Check these out here. The options available include Flex, Silverlight, SOAP & REST. The first two of these (Flex and Silverlight) are meant for client side development. The later 2 (SOAP and Rest) are intended for lower level integrations. For example i can run a benchmark report as a REST call like this:
http://baoapi.esri.com/rest/report/BenchmarkReport?
BenchmarkOptions=useNone&
FieldSortType=sortNone&
StandardReportOptions={“ReportFormat”:”PDF”}&
Summarizations=AVGVAL_CY;AVGHINC_CY;AVGHHSZ_CY&
TradeAreas=[{"RecordSet":{"geometryType":"esriGeometryPolygon","spatialReference":{"wkid":4326},"features":[
{"geometry":{"rings":[[[-117.177401,34.049393],[-117.186904,34.043611],[-117.160785,34.042416],[-117.177401,34.049393]]],”spatialReference”:{“wkid”:4326}},
“attributes”:{“AREA_ID”:”custom1″,”STOREID”:”1″}}]}}]&
f=pjson&
Token=ABC123
To run this live (with a valid token) and see the response click here. This is a very simple example of just one operation we can perform through Business Analyst API calls. In this case i just asked for some basic demographic information about a single reporting area. We obviously want to be able to perform this on many areas as part of that larger business process. But how do we construct these many and admittedly complex calls in a server side environment?
Well thats where Python can once again come to the rescue… lets look at how we might make the above call in a script-able fashion. I’m going to use the case of the BA Benchmark Report as the example we wish to execute:
1. Get a token in order to use BA services…
def getBAOToken(username,password):
try:
tokenurl = “https://baoapi.esri.com/rest/authentication?request=getToken&username=” + username + “&password=” + password + “&f=PJSON”
urllib2.urlopen(tokenurl)
data = json.load(urllib2.urlopen(tokenurl))
return (data['results']['token'])
2. Get our geometries (in my case polygons) to report on as a JSON representtion… this is the tricky bit…we need it in a format as follows…
TradeAreas=[{"RecordSet":{"geometryType":"esriGeometryPolygon","spatialReference":{"wkid":4326},"features":[
{"geometry":{"rings":[[[-117.177401,34.049393],[-117.186904,34.043611],[-117.160785,34.042416],[-117.177401,34.049393]]],”spatialReference”:{“wkid”:4326}},
“attributes”:{“AREA_ID”:”custom1″,”STOREID”:”1″}}]}}]
To do this we must iterate over each feature (polygon) in the feature class, get its potentially many sub parts, and the X & Y coordinates of those parts. This whole process also needs to wrap up the geometry information surrounded by the correct JSON tags (spatial reference, Area IDs etc). Not for the faint hearted but once done its not so bad.
def getPolygonGeometriesAsArray(featureClass, wkid, AreaIDFieldName):
# Identify the geometry field
#
desc = arcpy.Describe(featureClass)
shapefieldname = desc.ShapeFieldName
rows = arcpy.SearchCursor(featureClass, “”, “”, ‘;’.join([shapefieldname,AreaIDFieldName]), “”)result = “[{'RecordSet':{'geometryType':'esriGeometryPolygon','spatialReference':{'wkid':"
result = string.join([result,wkid,"},'features':["])
featnum = 0
for row in rows: # for each feature
# Create the geometry object
#
feat = row.getValue(shapefieldname)if(featnum == 0):
result = string.join([result,"{'geometry':{'rings':["])
else:
result = string.join([result,",{'geometry':{'rings':["])# Step through each part of the feature
#
partnum = 0
str_list = []
str_list.append(result)
for part in feat: # for each part
if(partnum == 0):
str_list.append(“[")
else:
str_list.append(",[")
# Step through each vertex in the feature
#
vertexnum = 0
newpart = "true"
for pnt in feat.getPart(partnum):
if pnt:
if(newpart == "true"):
newpart = "false"
out = "%s%s%s%s%s" % ('[', str(pnt.X), ',', str(pnt.Y),']‘)
str_list.append(out)
else:
out = “%s%s%s%s%s” % (‘,[', str(pnt.X), ',', str(pnt.Y),']‘)
str_list.append(out)
else:
newpart = “true”
# If pnt is None, this represents an interior ring
#
str_list.append(“],[")
vertexnum += 1
str_list.append("]“)
partnum += 1
result = string.join(str_list)
result = string.join([result, "],’spatialReference’:{‘wkid’:”,wkid,”}},’attributes’:{‘AREA_ID’:'”,row.getValue(AreaIDFieldName),”‘}}”])
featnum += 1
result = string.join([result,"]}}]&”])
return result
FUTURE: You should be able to use the fact that ArcPy geometry objects implement __geo_interface__ to do a lot of this work once a bug with dough-nut polygons gets fixed. For now we need to do this loop. I’ve optimized the string concatenation as best i know so it works pretty fast.
3. Construct the full benchmark report call and make the request…
This includes fleshing out the details of the BA call to tell it what type of report to run, the fields to report on, and the format of the returned information (JSON, XML etc). Another tricky part we deal with here is that an HTTP POST has limits on the URL length. Since the geometry information we just encoded in the previous steps may be extremely long we cant send this information as part of the URL. To solve this we encode this information as name values pairs on the request DATA.
def runBenchmarkReport(token,geomarray,fields):
try:
print “run benchmark report”
fieldsAsString = “”
count = 0
for field in fields:
if count > 0:
fieldsAsString += “;”
fieldsAsString += field
count += 1
#print fieldsAsStringvalues = {}
benchmarkurl = “http://baoapi.esri.com/rest/report/BenchmarkReport”
values['BenchmarkOptions'] = “useNone”
values['FieldSortType'] = “sortNone”
values['StandardReportOptions'] = “{‘ReportFormat’:'S.XML’}”
values['Summarizations'] = fieldsAsString
values['FieldSortType'] = “sortNone”
values['TradeAreas'] = geomarray
values['f'] = “pjson”
values['token'] = ‘”‘ + token + ‘”‘req = urllib2.Request(benchmarkurl,data)
response = urllib2.urlopen(req)
data = json.load(response)
xmlurl = data['Reports'][0]['ReportURL']
return xmlurl
4. Lastly we need to process the returned information and save it, in this case as attributes tagged on the areas we reported on.
def processBenchmarkReport(featureClass, xmlurl, fields):
try:
print “process benchmark report”
strio = urllib2.urlopen(xmlurl)
dom = parse(strio)
# Create a feature layer so we can easily describe and see what fields exist in the field info.
arcpy.MakeFeatureLayer_management(featureClass,”Outage_Layer”)
desc = arcpy.Describe(“Outage_Layer”)
fieldInfo = desc.fieldInfo
# make sure dataset has appropriate fields
# Assumption is fields are all numeric and floating point
fieldPrecision = 9
for field in fields:
if fieldInfo.findFieldByName(field) == -1: #field not found
print “Creating field: ” + field
arcpy.AddField_management(featureClass, field, “FLOAT”, fieldPrecision, “”, “”, field, “NULLABLE”)
# Now lets push the report numbers to the feature class
reports = dom.getElementsByTagName(“benchmark_report”)
for report in reports:
name = report.getAttribute(“AreaID”)
query = “Name = ‘” + name + “‘”
rows = arcpy.UpdateCursor( featureClass, query, “”, “*”, “”)
for row in rows:
print “found row: ” + row.Name + ” Updating attributes”
for field in fields:
row.setValue(field, report.getAttribute(field))
rows.updateRow(row)
And thats it! What we end up with is a feature class that is tagged with the Business Analyst demographic variables that we requested. This script can be run on an as needed basis e.g as the features come into creation, or at regular timed intervals.
As a result our client applications can simply request attributes from a feature class rather than making more complex calls (via the Flex or Silverlight APIs) to ask for this information. This has the potential added benefit of reducing network traffic and creating more responsive web applications since everything has been pre-calculated server side instead of having multiple clients making multiple expensive calls.
I’ve included a python file that includes the mentioned code. This file will not run as is since it will require you to enter your BA username and password and also some tweaking for the relevant local data sources on which you want it to operate. It is provide as an example of how this type of operation can be done. If you see errors or better or more efficient ways to do things please feel free to email the team @ telecom_templates@esri.com.
Regards,
Team Telecom
