Data Driven Detroit (D3) launched the 2014 One D Scorecard in May (read more about that here). Today, we’re writing to share more about our process for making this interactive data tool through a Q & A with NiJeL, a team of data scientists and developers that D3 collaborated with to build out the Scorecard. But first, a little context about the project:
In its third iteration, the new Scorecard makes exciting strides in data management and presentation. Working with NiJeL, we focused our resources and energy on two key components of development: an administrative tool for data management on the back end; and an interface powered by interactive data visualizations. We also revamped our data by updating and curating indicators, creating indices for each of the five Priority Areas, and incorporating a data deep-dive using original Opportunity Mapping research from the Kirwan Institute.
Let’s dig into four questions with Lela Prashad and JD Godchaux of the NiJeL team.
D3: Way back when we first started working together, we warned you that we were managing the One D Scorecard data through Excel workbooks, a single workbook per indicator (over 50 workbooks at one time!). It was a bit of a data nightmare in a few ways, especially when it came time for annual updates or when we needed to compare individual indicators across geographies. How did you sift through our data and come to the new centralized solution we’re using today?
NiJeL: Good question! We realized early on that we needed an easy way for D3 staff to update indicators as these datasets are updated, rather than all at once, say on an annual basis. We also understood that we wanted an automated way to add these new data to the One D Scorecard website as soon as an indicator is updated, and the only reasonable way to make this happen was to build a database to house all these indicators. So, we built a MySQL database and modified Xataface, an open-source software designed to add a simple admin interface for a MySQL database.
Once we had honed in on using these tools, we went through each Excel workbook and added each indicator to the MySQL database, slightly reorienting the data from the Excel sheets to make it easier to use. We then wrote two scripts, one to simply pull all of the relevant indicators for each region and package them up all together so the website could create the visualizations that it does, and another to calculate the five Priority Area index values and the overall One D Index score for each region.
Now, the staff at D3 can update any specific indicator by uploading a CSV (comma-separated value) file with any new data they would like to add. Once these new data are added, the web app will update the site visualizations once there is a critical mass of the data from each year to make the index calculations meaningful. We’re hoping this will be a big improvement!
D3: While older iterations of the Scorecard ranked regions based on their performance in a single indicator, we took that a few steps further this year using indices. An index lets us roll up the individual indicators that comprise a Priority Area into a single summary score, and then roll up each of those five Priority Area scores to create a One D Index Score for each region, making comparisons comprehensive and straightforward. Our favorite feature is how an index calculates on the fly and smartly recognizes when too few indicators for a given index have been updated to update the index itself. Can you share a bit about the process for programming in these analytical features to the highly visual front end?
NiJeL: Of course! As you mentioned, we want to be smart about how we’re calculating the index scores for each of the five Priority Areas and the overall One D Index, and we want to do this in the context of new data being continually added to the database. To accomplish this goal, we programmatically look at the group of indicators in each Priority Area and determine if more than 50% of these indicators have data for the year in question. For instance, the Economy Priority Area has 7 indicators, so if 4 or more of these indicators have data available for 2012, then we would calculate an Economy Index for 2012.
However, for us to go ahead and calculate a One D Index and include the year in our visualizations, each Priority Area would need to surmount the 50% threshold. Once that occurs, the indicators and Priority Area indices are added to the visualizations and the data become available for download.
NiJeL: Well, we had the distinct advantage of working with two individuals, one being Ms. Hartman and the other being D3 Project Manager Jessica McInchak, who both were interested in web interactive design and building interactive visualizations. Both actually contributed to the codebase for the One D Scorecard, which is an extremely rare thing for a designer and a project manager to want to do, but both Ms. Hartman and Ms. McInchak were excited to have the opportunity which made for a fantastic working relationship.
Ms. Hartman’s design for many of the elements in the One D Scorecard were inspired by other designs live on other websites, and so when it came time to build these visualizations, we did have some examples to view, though most were written using other tools like Raphael. We decided to use D3.js mainly because of its flexibility — it allowed us to be able to build the visualizations as closely as possible to what Ms. Hartman designed. The most challenging aspect of building to static wireframes, like the ones Ms. Hartman designed, is understanding the intended interactions and transitions between states within a particular visualization. It’s challenging as a designer to draw out intended interactions and as a developer to follow through on those intentions, but our close collaboration with Ms. Hartman and Ms. McInchak, minimized any differences we had on building the interactions as intended.
It’s tough to pick our favorite chart to build, since they all had their challenges and rewards, but we’d have to say the “array of pinwheels’ visualization (where viewers can see each region’s pinwheel in an array of rows and columns) was our favorite to build.
In this visualization, the pinwheels load such that the region with the highest One D Index value is placed in the upper left corner and the remaining regions are placed in descending order from left to right and top to bottom. Visitors can reorder the pinwheels by selecting a specific priority area to view from the “Organize By” drop down menu. Building this chart required extensive use of D3.js transitions, which allowed us to be creative in how we moved from one state to another. When visitors select a different Priority Area (or a different year of data) to view, we effectively run three separate transitions on each regional pinwheel. First, we change the color of each pinwheel slice, setting the Priority Area selected to its full opacity and setting the opacity of the other slices to almost fully transparent. At the same time, we change the size of the pie slice if a visitor has selected to view a different year with the time slider. Finally, we reorder the pinwheels in descending order based on the index chosen, but that transition only occurs after a 1 second delay to allow the first set of transitions to complete. Building these transitions in an attempt to clearly compare the differences across regions and indices was the most challenging and fun part of the development.
D3: The 2014 One D Scorecard presented a lot of opportunities to collaborate. Not only did we work with your team around development, but we also partnered with the Kirwan Institute to integrate their Southeast Michigan Regional Opportunity Mapping initiative. Kirwan’s original research was presented through static maps of the overall index scores. What was your motivation and method for interactively mapping both the index and individual indicators?
Crosslet is particularly designed to allow visitors to explore one or multiple variables to see how each is connected, and to see representations of those connections on a map and a simple frequency distribution bar chart. For instance, if a visitor selected the median household income variable, and then selected the income range of $0 – $50,000, they would see only the geographies (census tracts) that have median household incomes below $50,001. They would also see the frequency distribution of the Opportunity Index, high school completion rates, and vacant property rates. Clearly, the distribution of each of the other variable is skewed toward the negative end of the spectrum when we select that income range. However, if we click on the selected range and drag it toward a higher income range, we can see the frequency distributions of the other variables shift toward the more positive end of the spectrum along with income, and on the map we can see which census tracts specifically fit these new criteria. One can also select a range of any other variable on the map to further filter these data. We think that gives visitors a great entry point to exploring these data and drawing their own conclusions about the drivers behind opportunity in the Detroit metro region.
The new One D Scorecard is a powerful tool for its users to access data through visualizations, but it’s also a powerful data management system for D3 to maintain and scale these datasets into the future. And we couldn’t have built it without the awesome team at NiJeL. We’re already counting down the months until the newest annual data indicators are released so we can do our first update!
Check out the 2014 One D Scorecard at onedscorecard.datadrivendetroit.org, and NiJeL’s GitHub repository to explore the code driving the interactive data visualizations.
If you’re interested in talking more about code and collaboration for the One D Scorecard or beyond, connect with D3’s Project Lead Jessica McInchak at firstname.lastname@example.org.
In November 2013, the Motor City Mapping project moved forward at a dizzying pace, with the goal of identifying every blighted structure and empty lot in the city of Detroit. Loveland Technologies was in the process of finalizing the Blexting application for the city-wide parcel survey, and the Michigan Nonprofit Association hired more than 100 community surveyors. Meanwhile, in these early stages, Data Driven Detroit (D3) focused its surveying expertise on developing and testing the questions that would drive the survey. During this time, D3 received a number of requests from community groups to integrate additional questions into the survey process, ranging from assessing historic relevance to estimating rehabilitation costs. Unfortunately, with the speed of the project and the complexity of the existing survey, the Motor City Mapping team was forced to delay such modifications until future phases of the project. However, in one of these instances, a stunning example of grass-roots organizing and collaboration between the Michigan Historic Preservation Network, Local Data, and D3 resulted in the Historic Resource Survey, one of the richest datasets that would emerge from the Motor City Mapping project.
Background for the Survey
The early weeks of Motor City Mapping focused on surveying properties located within six areas that the city of Detroit identified for disbursement of the $52 million in Hardest-Hit (HHF) funds. These funds are required to be used for targeted demolitions in stronger-market areas, and the city faced a short timeframe in which to disburse the awarded resources. Due to these caveats – the emphasis on demolition, and the quick turnaround required by the conditions under which the grants were awarded – some groups expressed concerns that the HHF deployment would result in demolitions of structures with considerable historic significance. The Michigan State Historic Preservation Office received determination from the U.S. Department of Treasury that the HHF program was not eligible for preservation oversight, accentuating these reservations.
In response to these worries, the Michigan State Housing Development Authority – the state department charged with disbursing the HHF dollars – exempted designated historic districts from demolition efforts. However, eligible but not yet designated historic districts did not receive this protection. Concerned about the potential loss of historic structures in these areas, the Michigan Historic Preservation Network (MHPN) partnered with Preservation Detroit and the Detroit Land Bank Authority to advocate for bringing a preservation perspective to the process of identifying structures for demolition.
Bringing the Survey to Life
Though every organization involved in Motor City Mapping supported the aims of MHPN and Preservation Detroit, the initial requests to add a historic component to the survey faced the same obstacles as other issues that were brought to the team’s attention. By this time, the survey was in full motion, and adding additional questions into the Blexting application was virtually impossible. Not to be dissuaded, Emilie Evans from MHPN and the National Trust for Historic Preservation reached out to Data Driven Detroit to ask if there was any potential way that the Historic Resource Survey could still be completed. In response, D3 offered up use of its Local Data license. This provided MHPN with access to a mobile surveying application similar to the Blexting platform used by the wider Motor City Mapping survey. D3 and MHPN collaborated further, identifying parcels for the survey based on two criteria – location in an eligible or proposed historic district, and location in one of the six designated HHF deployment areas. Once D3 had determined the survey geographies and delivered them to Local Data, MHPN was ready to commence the Historic Resource Survey…
…from a technical standpoint, that is. MHPN still needed surveyors to collect data on each of the nearly 18,000 properties that were located within the targeted areas. A call for volunteers was met with a tremendous response – nearly 55 individuals from dozens of organizations throughout the city. Offering their evenings and weekends to the project, these volunteers used Local Data to survey 17,500 properties across the City of Detroit in only two weeks.
The Historic Resource Survey answered three questions for each property, focusing on its architectural integrity, how in-keeping it was with neighborhood character, and how well its block remained intact. These questions were then aggregated into an easily-digestible Historic Preservation Score – Very Important, Important, Less Important, or Not Historic – that identified how much of a preservation priority a particular property was for a neighborhood. The survey data makes possible data-driven decisions about both building removal and restoration. Funding sources tied to demolitions, such as HHF, can be focused on properties that are labeled as less important or not historic, while preservation efforts can focus on properties of greater importance – not only measured in their architectural significance, but also by their place as an anchor of some of Detroit’s most vibrant communities.
The full Historic Resource Survey dataset is available for download at parcel level in a variety of formats from D3’s Open Data Portal, http://portal.datadrivendetroit.org.
For more information on the activities of the Michigan Historic Preservation Network and Preservation Detroit, visit the following websites:
For more information on the Local Data application used in the Historic Resource Survey, visit www.localdata.com.
The nation-wide movement toward public data transparency and democratization is continuing to gain support. As cities including Portland, Chicago, New York, Louisville, Ann Arbor and others are embracing Open Data in government by creating web sites for citizens to easily view and download data, the potential for developing useful applications driven by these data is also growing.
On May 31 and June 1, participants in the second annual National Day of Civic Hacking will gather across the country and beyond to leverage new data sets from local and federal agencies in order to create impactful, technology-based tools and services.
The civic hacking initiative aims to illustrate the power of open government, particularly where data is available to support meaningful collaboration between the public and private sectors, and demonstrate how citizens can improve their local communities with data and technology. The promotion of transparency, participation and collaboration is a cause very close to our mission here at Data Driven Detroit.
Detroit’s participation in the National Day of Civic Hacking this year includes local events at The M@dison Building, 1555 Broadway. Data in Detroit, from 2 to 4 p.m., will bring together civic data practitioners in a high-speed presentation format. In six minutes and 40 seconds, each presenter will give an all-filler rundown of their work. Detroit Startup Drinks, from 4 to 5:30 p.m., is a regular monthly meet-up with Code for Detroit. The Open Data Edition will focus on open data challenges facing Detroit, including datasets that could be used to improve communities, and datasets that government entities should open.
For more information about the initiative, visit The National Day of Civic Hacking. For open datasets provided by Data Driven Detroit, visit our recently launched Open Data Site.
We would love to hear if you plan on participating, and what’s on your data wish list. Please take a moment to let us know in the comments!
- What data do you want to access in order to better perform your work?
- What form would you like that data to take – spreadsheets, APIs, other spatial or visual representations?
- How would you prefer to access that data – Internet, mobile app, or other published form?
Data Driven Detroit (D3), an affiliate of Michigan Nonprofit Association (MNA), is excited to participate in the release of the Detroit Blight Removal Task Force’s final report, “Every Neighborhood Has a Future…And It Doesn’t Include Blight.”
Detroit Blight Removal Task Force Co-Chairs
- Glenda Price, President of the Detroit Public Schools Foundation
- Linda Smith, Executive Director of U-SNAP-BAC
- Dan Gilbert, Founder and Chairman of Quicken Loans and Rock Ventures
Eight months ago, the Task Force sought to develop a straightforward and detailed implementation plan to address every blighted structure and vacant lot in the city of Detroit. The three co-chairs organized a team of experts from all levels of government, the private and nonprofit sectors, and the foundation community to provide insight on the topic of blight elimination.
The report uses the data collected from the Motor City Mapping project as a foundation to understand the city’s landscape and create informed recommendations.
As you flip through the pages of the report, you’ll find detailed descriptions and recommendations on how to:
- Define the overall scope of blight in Detroit;
- Focus efforts for greatest geographic impact;
- Choose the appropriate intervention on a structure-by-structure basis;
- Conduct blight elimination in a way that’s sensitive to environmental and public health factors;
- Institute policy reform that will proactively address future blight; and
- Fund blight elimination initiatives across the city.
What is Motor City Mapping?
Motor City Mapping Funders
- Michigan State Housing Development Authority (MSHDA)
- Skillman Foundation
- Kresge Foundation
The Motor City Mapping project was a groundbreaking effort to survey each of the 380,000 parcels in the city of Detroit, implemented by D3, MNA, LOVELAND Technologies, the Quicken Loans family of companies, and a dedicated team of 200 resident surveyors and drivers. For every property, the survey identified condition, occupancy, and use, providing the information necessary to understand the challenge of blight on both a micro and macro scale.
The data from the survey, as well as data from over 20 additional third-party datasets, together created a detailed description of every property in the city.
The survey built on the Detroit Residential Parcel Survey (DRPS), completed in 2009 by D3, in partnership with Community Legal Resources (now Michigan Community Resources), the University of Michigan, and the Detroit Office of Foreclosure Prevention and Response. The DRPS evaluated condition and occupancy for 350,000 residential parcels in Detroit, focusing on single-family homes, duplexes, multi-family structures up to four units, and vacant lots in residential areas.
How We Improved On DRPS
DRPS was an unprecedented effort for Detroit, and the impacts were enormous. Since 2009, the data were a cornerstone of almost every planning process in the city, on both a block and citywide level.
However, as many in Detroit can attest, the landscape in the city has changed in those five years. When given the opportunity by the Detroit Blight Removal Task Force to update the data, D3 used their expertise to improve the process.
In 2009, every record was written with paper and pencil, which an analyst then entered into a database. With support from LOVELAND, the team implemented the most significant process improvement – technology. Rock Ventures donated 200 tablets to the project that came pre-loaded with LOVELAND’s “Blexting” app (“Blight” plus “texting”), enabling the surveyors to evaluate properties with ease. LOVELAND also created an online interface that displayed data records coming in from the field in real time. D3’s quality control associates used this interface to review the records for accuracy and provide feedback to surveyors on how to improve their work when capturing future data.
The Method: Resident Surveyors
With Michigan Nonprofit Association’s experience in community engagement and capacity-building, the project hired over 120 resident surveyors, leveraging a relationship with the Detroit Employment Solutions Corporation.
What started as a simple survey turned into much more – community engagement, education, and buy-in for residents in the neighborhoods. The people who participated in the survey will keep the data alive once it goes public, telling their neighbors and friends about the process and encouraging people to get involved.
- What’s a parcel?
- In real estate terms, a parcel is a plot or tract of land.
- What is blight?
- Generally, blight refers to abandoned structures within an urban area. The Blight Removal Task Force includes a more specific definition of blight in their report.Access the report here.
To provide a more comprehensive picture of property in the city, D3 integrated over 20 additional datasets to create the most comprehensive property database ever produced for Detroit. These datasets include historic designation and eligibility, ownership, current and future land use from Detroit Future City, tax delinquency, and foreclosure status, among many others.
In the spirit of open data, all of the survey data are available for download on the project’s website, http://www.motorcitymapping.org. Check back to the project website as we make more data available for display and for download.
Everyone at D3 is looking forward to the implementation of the Task Force’s recommendations. We are committed to making this information accessible to all and hope to serve as a resource for those interested in bringing about change.
For more information on the Detroit Blight Removal Task Force and their report, go to http://www.timetoendblight.com/.
To visit the Motor City Mapping project website, go to http://www.motorcitymapping.org.
For more information on Data Driven Detroit, go to http://www.datadrivendetroit.org.
Several years ago, One D began as an effort to bring together parallel research and resources aimed at facilitating and evaluating regional development in Southeast Michigan. Although One D disbanded in 2011, D3 continues to be a steward of the One D Scorecard – first crafted by One D partners as a comprehensive blueprint for moving our region – metropolitan Detroit – forward.
Today, D3 is launching the 2014 One D Scorecard. Our newest iteration implements exciting changes in user experience, while building on the Scorecard’s original mission to increase access to key information, inform civic dialogue, and track progress on shared regional goals. We’ve thoughtfully curated new indicators, analyzed them using the “One D Index,” and designed a Scorecard that is powered by interactive data visualizations.
The Scorecard measures metro Detroit’s performance in comparison to over 50 metropolitan regions throughout the U.S. across five priority areas: Economic Prosperity, Educational Preparedness, Quality of Life, Social Equity, and Regional Transit. This year, we’ve created the One D Index to roll up over 30 outcome-based indicators into a single comprehensive score to better understand how metro Detroit stacks up across priority areas and other regions overall.
Among its many new features, the 2014 One D Scorecard uses interactive data visuals to learn about our region. For example, we find that our region – metro Detroit – scores higher on the Social Equity Index than on any of the other four priority area indices in 2011. Using the bar chart, we unpack the index and individually explore the indicators that drive the Social Equity priority area. There, we see that metro Detroit leads all other regions included in the Scorecard in “percent of owner-occupied housing” for Hispanics and ranks second in the same category for White households; therefore, positively boosting our overall Social Equity Index score. And this is only one short chapter of the many data stories to be discovered!
Moreover, the 2014 One D Scorecard is now home to the Kirwan Institute’s Southeast Michigan Regional Opportunity Index mapping initiative, through the One Detroit Portal. While the Scorecard reports on and across regions, the Portal is a unique data deep-dive into Southeast Michigan to explore complementary census tract level indicators that illustrate the diversity within and among the cities that make up metro Detroit.
Why is D3’s Director, Erica Raleigh, excited about the 2014 One D Scorecard?
“Regional thinking is critical,” says Erica. “Understanding how we compare to other regions can lead to localized action, and integrating the Opportunity Index and the One D Scorecard allows just that – we can see exactly where we need to foster greater opportunity within metro Detroit.”
Start exploring at onedscorecard.datadrivendetroit.org and let us know how you’re using this data to drive regional decision-making! Share feedback or questions by commenting or contacting us through AskD3, with subject line “One D Scorecard.”
D3 thanks the New Economy Initiative, Kresge Foundation, and Bosch Community Fund for their generous support of the Scorecard, as well as our research and development partners, the Kirwan Institute for the Study of Race and Ethnicity at Ohio State University and NiJeL.
This post is the next in a series of profiles of partner organizations using data from Data Driven Detroit to successfully support their work.
A shortage of safe and reliable public transportation presents a huge roadblock for many Detroit residents, especially children interested in participating in after-school and summer programs. The Youth Transit Alliance, funded [Read on...]
This Q&A is the sixth in a series of profiles of Data Driven Detroit staff members.
Diana Flora comes to Data Driven Detroit as a Detroit Revitalization Fellow, but was introduced to D3 years ago through former D3 staff and classmates. Since joining the D3 team, Diana has been the D3 lead on the [Read on...]
Data Driven Detroit’s mission to provide accessible high-quality information and analysis to drive informed decision-making just got a little more high-tech. This week, ESRI rolled out a new platform that allows customers to more easily distribute geographic data to the public. D3 has been an avid user of ESRI products like ArcGIS and their [Read on...]
Earlier this month, I had the pleasure of attending the National Neighborhood Indicators Partnership (NNIP) meeting in St. Louis. For readers who don’t know, D3 has been the Detroit NNIP partner since 2009, and we are now one of nearly 40 partner organizations across the country. NNIP is a collaborative effort by the Urban Institute [Read on...]
On February 22, participants in the second annual International Open Data Hackathon will gather around the globe to liberate, analyze and publish data in order to “show support for and encourage the adoption of open data policies by the world’s local, regional and national governments.” (For more information, please visit opendataday.org.)
While there isn’t an [Read on...]