OR Ten Reasons Not to Trust Top Ten Lists
By Data Driven Detroit Staff
Data Driven Detroit’s mission is to provide up-to-date, objective information to inform better decisions. Often that information is negative and our first inclination is to run away from it. Accurate information is purely information that, whether negative or positive, should be accepted/acknowledged for what it is and serve as a basis for action.
However, when the information, or its analysis, is shown to be Inaccurate, it must be challenged! Data literacy is best described as a general understanding of a dataset in respect to how it was gathered and how it was analyzed. Debunking misleading analysis is one approach to improving data literacy.
On October 3rd 2011, Forbes released an article worthy of critique: America’s Most Dangerous Cities lists Detroit as Number 1.
1. The difference between apples and oranges:
Nine out of the ten geographies listed in Forbes’ piece are metropolitan statistical areas (MSA), but the included geography for Detroit is a metropolitan division (MD). The Detroit MSA includes 6 counties while the Detroit MD includes only Wayne County; this is a difference of roughly 2.5 million people. All this underlines a basic lesson from Statistics 101: DO NOT compare apples to oranges. In this instance the apples-to oranges comparison hugely biases the results against Detroit by concentrating its crime. When Detroit is properly included on the list as an MSA, it is actually ranked seventeenth in the nation for crime – not first!
2. Size Matters:
To be fair, a possible reason that Forbes used the Detroit MD instead of the Detroit MSA is because the Detroit MD is closer in size to the other 9 MSAs included in the ranking. The average size of the 9 included MSAs is roughly 650,000 people. In comparison, Detroit’s MSA is 4.35 million whereas its MD is just shy of 2 million. Choosing the Detroit MD is likely Forbes’ attempt to compare apples to apples; however; it is unfortunate that the apples they chose are not the apples recommended by the Census Bureau.
What may be a more relevant question to ask Forbes is “Why are the most dangerous cities in America so small?” Where is Los Angeles? New York? These after all, are the settings for America’s top television crime drama Law & Order. It is likely that this pattern is related to the fact that MSAs are not really cities at all but economic regions. Cities tend to have economies that spill over into the neighboring cities, townships and counties—the larger the city the more spillover that is going to be included in the MSA. The larger MSAs are more likely to pull in more neighboring suburbs than the smaller MSAs. The inclusion of suburbs, which tend to be safer than cities, will help to offset larger cities’ crime stats. Smaller cities will have smaller MSAs which are less likely to include many suburbs, thus isolating their crime similar to the manner in which Forbes’ isolated Detroit’s crime in Point 1.
3. Forbes’ Flip Flops: Last year we were one of the safest cities?
The Forbes headline, “Most Dangerous Cities,” could be described as an incomplete superlative. Their definition of “Dangerous” only includes violent crime, not other obvious dangers like automotive collisions, natural disasters, or environmental health risks. Back in 2010, when Forbes was actually comparing MSA’s to each other instead of MDs, and using multiple measures of danger instead of simply using crime alone, they listed metro Detroit as the twelfth safest city in the nation. Is it possible that in the span of only two years, Detroit went from being the twelfth safest city to the most dangerous? Probably not.
4. Read the Fine Print:
Forbes uses the FBI’s Uniform Crime Report (UCR) data in exactly the way that the FBI advises it should NOT be used: to rank locales. From the FBI UCR site: “Data users should not rank locales because there are many factors that cause the nature and type of crime to vary from place to place. UCR statistics include only jurisdictional population figures along with reported crime, clearance, or arrest data. Rankings ignore the uniqueness of each locale” .
5. Fill in the blanks:
The FBI UCR data Forbes uses is incomplete. Cities report voluntarily to the FBI UCR database and not all cities participate. Chicago, for example, does not report its crime statistics to the FBI and therefore is not included in Forbes’ Most Dangerous Cities ranking. Forbes’ admits this flaw outright in the safest American cities article. However, it is important to reiterate that they voluntarily chose to use a flawed data set for the dangerous cities article.
6. Place matters, and so does perception:
The Forbes’ analysis ignores the fact that crime is often concentrated in certain neighborhoods, not spread evenly across entire cities. It reinforces negative stereotypes about its listed cities in their entirety, making it more difficult for those places to take positive actions to reduce crime and improve their communities’ strength. The New Haven Register reported on this issue last year when Forbes’ list was published: “In Washington, D.C., for example, the city is divided into four districts, but only one is ridden with crime, [Robert McCrie, professor in the Department of Protection Management at The City University of New York,] said… ‘That creates a public perception, which, in turn, has a “measurable effect” on the community, real estate values and choices on where students matriculate for college and other quality of life issues’, McCrie said.” 
7. Data can tell many stories
The story Forbes’ chose is more telling of their perception of Detroit than it is an accurate description of the data. The FBI’s UCR data does not report anything more than how many incidents for each violent crime category. The opening paragraph in the Forbes’ article highlights the brutal and sensational murder of twenty-three-year-old Diana DeMayo in order to grab the reader’s attention. Additionally, as the Detroit News Hub highlighted for us, the reporter had to change his original lead when a comment from a reader pointed out that the “crime” he originally chose to highlight … the death of banker David Widlak …was ruled a suicide and that Widlak did not have an office in downtown Detroit nor did he live or work in Detroit, Livonia or Dearborn.
8. Pictures can Distort the Story
Pictures also help tell a story. The 2nd picture of Detroit in the Forbes’ article is of an former Chrysler factory being demolished. The use of this photo suggests that it is intended as a metaphor for danger. However, not only is there another Chrysler plant opening just down the street , but the demolition of an old abandoned plant could be perceived as a positive action. As Mayor Bing has stated, “Abandoned and dilapidated buildings are hotspots for crime and a living reminder of a time when the City of Detroit turned a blind eye to owners who neglected their properties” . Bing himself sees the demolition of unused buildings as a way to fight crime and spawn redevelopment.
9. Make the Picture Match Your Story
Forbes’ photo of the gas station in “Flint” is not actually in Flint. Any reader can see from the writing on the police car in the photo, that it is hails from Mt. Morris.   What this does suggest however is that the authors did actually have an understanding of what constitutes an MSA , as in this instance they clearly understood that Mt. Morris is in the Flint MSA which aligns with the boundaries of Genesee County. The question then becomes again: “ Why did Forbes choose to isolate Detroit by using its MD to compare it to 9 other MSAs?”
10. Rankings, While Accessible, Often Hide the Details
Lastly, any “top ten” list or ranking should be approached with caution! This post included! For example; though Forbes’ analysis is certainly misleading, Detroit still has a crime issue that still needs to be addressed. What D3 seeks to underline with this ranking is that the ranking of anything is going to involve a level of simplification that may contort facts or simply ignore them. The reader should be especially critical of lists that seek to simplify data sets. Data are highly complex, especially those gathered at the national level such as the FBI’s UCR.
That said, we can’t fault Forbes’ for trying. D3, like Forbes’, has also tried its hand at ranking cities and regions—the OneD Scorecard. However, we also would like to believe that our methodology is a bit more honest than Forbes’. In addition, lists are certainly a handy tool for making complicated things more accessible, something in which D3 definitely believes!
Let us not use this as a reason to hide our heads in the sand when faced with information that portrays us in a negative light. Instead, let us face it head on, understand the information presented, and proactively address what we are doing to turn it to a positive!
Picture is an AP news image of the Former Chrysler Plant at Wyoming & McGraw in Detroit, Michigan
 Picture is taken at the intersection of Detroit St. and Coldwater St in Mt. Morris, Michigan.