My favorite time of the season! Halloween? Sure, it’s fun, and I’m looking forward to Friday, but how much data can we squeeze out of that day? I’m just returning from my hands down favorite event in the fall – the National Neighborhood Indicators Partnership (NNIP) meeting.
For those who haven’t heard me rave before, NNIP is a national network of organizations that share a commitment to creating and maintaining neighborhood-level data systems and helping their communities use the information to make better decisions. We meet up twice a year to talk about our data shops – new discoveries, advances in analysis and online tools, challenges we’ve encountered, and much, much more. D3 has been a proud partner organization since 2009. Partners take turns hosting the meeting, and this time we went to Denver, Colorado, where the Piton Foundation makes data come alive through their Data Initiative.
As always, I learned so much last week from all of our partners in the network. Here are some of the highlights I took away…
Nonprofit does not mean we can keep going without funds!
(A quick aside: I’m using the term nonprofit here, but I agree with Doug Bitonti Stewart of the Max M. and Marjorie S. Fisher Foundation that we need find a new term for our sector such as the “for impact sector”. Why should we define ourselves by what we are not, versus what we are for?)
As D3 approaches its sixth anniversary, we’ve been doing a lot of thinking here about how we fit into the local environment, how we can best serve our community through the products and services we provide, and how to ensure D3 can continue to serve our community over the long haul. As we grow up, we are also turning our data-driven philosophy inward and discovering some really important information about ourselves.
Taking a data-driven approach on the inside as well as out is certainly not new among the national partners, but it seemed to be a common theme of our meeting this fall. Many of us are working through formulating business models, diversifying funding streams, monitoring internal performance, and understanding external impact. Check back in the near future to read more about what D3 is learning through our investigation and analysis.
Our partners are building awesome online tools
Rhode Island Community Profiles from the Providence Plan show us a creative and expertly crafted use of the new Census Bureau API. And the code is on Github!
Greater Portland Pulse is demonstrating incredible potential for regional collaboration and metric tracking for a region comprised of seven counties in two states.
My hands down favorite illustration of the uncertainty we should feel when using American Community Survey (ACS) data: The Randomizer. The tool generates random values for households in poverty by census tract between the range of possible values given by ACS, and a resulting total value of the households in poverty based on the tract numbers generated. Thanks to John Garvey for sharing his creation with me, and to Urban Strategies Council in Oakland for always pushing the cutting edge.
We can all be powerful data advocates
As John’s tool depicts, American Community Survey (ACS) data can present some challenges when presenting data for small areas. It seems I find myself engaged in multiple discussions at every NNIP meeting about ACS data: to use or not to use? Are the data reliable enough or just garbage? Should we replace ACS with locally sourced data, data which may be more accurate, but less comparable to other places and much more time consuming to produce for more than one jurisdiction?
I didn’t walk away with any clear answers, but I do know one thing: if we appropriately fund the ACS as a nation the data would be more reliable than they are now, which is important not only for the annual $400 billion in federal and state fund disbursement which relies on the data, but also for all the community development and planning work that goes on every day at a local level across the country. Congress appropriates ACS funding – let your congressperson know you care! And I beg you – if you are lucky enough to receive an ACS questionnaire, please fill it out! I’m still waiting for mine…
NNIP data advocacy work extends beyond government or administrative data use, it also includes maintaining open data sites and convening data user groups. D3 only recently launched our Open Data Site, but some partners have been publishing open data in their respective cities for years. Some have even developed sophisticated user groups, trainings, and conferences to more broadly engage their communities in the practical use and application of data. Why is this so important? William Gibson put it best: “The future has already arrived. It’s just not evenly distributed yet.”
Baltimore, Columbus, Indianapolis, and the MacArthur Foundation (on behalf of the Chicago School of Data) shared with us their experiences with local user conferences during the last day of the meeting. I still have much to digest and apply to D3’s trainings and advocacy for improving data literacy, but in the meantime, I’ll suggest readers refer to NNIP’s putting open data to work in communities.
Thanks NNIP for gathering us all together once again!
Just as longer commutes can have detrimental effects on adults, it reasonably follows that longer school commutes may have such effects as an increase in stress, tardiness and obesity rates on our youth. Conversely, in areas where housing patterns concentrate poverty and race in a neighborhood, longer commutes outside those neighborhoods might improve student outcomes. Though it is not yet clear just how school commute distance effects student performance, what is clear is that Detroit students and families are exercising their choice. Given the potential problems and benefits, where do these patterns exist and how might schools and families adapt?
For the first time, in partnership with Excellent Schools Detroit and The Skillman Foundation, Data Driven Detroit was able to perform our student dispersion analysis with data covering all publicly funded schools in Detroit*. The Michigan Center for Educational Performance and Information (CEPI), working with the Michigan Center for Shared Solutions (CSS) provided Data Driven Detroit unidentifiable enrollment data and census block codes approximating student residence location from the fall of 2013. CEPI also provided information for students attending schools in the Cities of Warren, River Rouge, Southfield, Hamtramck and Highland Park. This summary analysis is the most comprehensive student dispersion analysis D3 has done to this point in terms of breadth of schools, and includes data from more than 141,000 students and 300 schools!
Student dispersion map examples: Chrysler Elementary School and Western International High School
The Detroit Context
Educational reforms in Michigan have broadly opened up public grade school options and resulted in a complicated school environment where families have many choices, near and far, in where to send their kids to school.
Detroit’s school context is dynamic, if not chaotic, where several schools may open and close year to year, and where there are a number of different public school systems serving students, including suburban schools of choice, charter schools (Public School Academies), the Detroit Public Schools (DPS) system, and the state-run Educational Achievement Authority (EAA). With more options, inconsistent transportation, school openings and closings, and other reasons, some students may find themselves, either by choice or necessity, with a more difficult commute. Because of this complex environment, it is not well understood just where youth from different parts of the city choose to attend school and how far they go.
The way students in Detroit are dispersed may have many important implications of interest for educators and planners.
- Which schools have the longest commutes for students? Which schools fail to draw students outside their neighborhood, and why?
- Are there parts of the city that have a stronger local draw?
- Are there parts of the city whose students are more mobile?
- Are certain students drawn to DPS schools vs. charter schools vs. the EAA schools?
- How can future transit decisions be made that help students get to the schools that they attend?
- If schools create neighborhood identity and community by causing local youth to attend classes together, are there neighborhoods that might be disproportionately affected because students aren’t attending local schools?
- In schools that have further average commutes, how can educators help mitigate those students’ further commutes?
Data Driven Detroit is now making available a couple of new resources to help explore this data. First, our new interactive map allows users to view the locations of schools by type or level and then download digital maps showing the dispersion patterns of each individual school. In addition, we have calculated the average student commute distance for every public school in Detroit and summarized our results in a brief report.
While these data and map resources will not answer these questions on their own, we hope that these tools are useful to researchers, policy makers and educators as they make plans and formulate policy to improve the educational environment in the city.
Student Dispersion Maps
D3′s student dispersion maps show the patterns of where students who attend certain schools live. Our tool allows users view maps of any publicly funded school from across the city by level, type, or by searching by name. These maps reveal spatial patterns that would be impossible to understand without this type of visualization.
Commute Distance Summary
The commute distance summary report helps to quantify just how far students from different schools are traveling, and how different types of schools compare against each other. This summary is meant to help educators understand the commute burden placed on their students and assist administrators in planning for school location. The summary lists the average distance for each public school. In addition, the school locations from our study, complete with average distance information, are available as a GIS shape file or table from Data Driven Detroit’s Open Data Portal (Search for “commute”).
D3 will be following up this blog post in the near future with two more blogs on this topic. The first will take a deeper look at a few interesting dispersion pattern maps. The second will take a closer look at the average commute distance analysis.
Since this analysis elicits more questions than answers, D3 is hoping to continue its analysis of student location data. Through our partnership with ESD and CEPI, we are expecting to get a more complete data set including not just students that attend Detroit publicly funded schools, but also Detroit residents who attend schools outside the city. As a large percentage of Detroit youth do attend school in the suburbs, this next data set should draw a much more complete picture.
* Ann Arbor Trail Magnet School was mistakenly excluded from the data set delivered to Data Driven Detroit.
Today marks a bittersweet day in Data Driven Detroit’s history as we celebrate the achievements of Detroit’s own data guru and D3’s founding director, Kurt Metzger.
This morning, Kurt was honored at a breakfast at Detroit’s Rattlesnake Club. Friends and colleagues praised his accomplishments and thanked him for his work using data to make nonprofits and governments more efficient and effective. Although Kurt has moved on to new endeavors, he has left an indelible imprint on the city and Data Driven Detroit.
Kurt has spent more than 35 years promoting data accessibility in Southeast Michigan through his work with the Census Bureau, Wayne State University, the United Way for Southeastern Michigan, and founding of Data Driven Detroit.
In 2008, Kurt was serving as the Research Director for the United Way when the philanthropic community in Detroit recognized the need for data to help focus investment and measure outcomes. City Connect was developing a concept for the Detroit-Area Community Information System (D-ACIS). They tapped Kurt to bring the concept to life and serve as director.
Kurt’s vision for D-ACIS was an independent, objective clearinghouse that would provide information to the community at large, working with organizations that held the same beliefs regarding collaboration and information-sharing. In 2010, D-ACIS became Data Driven Detroit (D3). For five years at D3, Kurt was our mentor, leading us through many challenges and opportunities along the way.
Erica Raleigh, D3’s current director, is proud to continue the work that Kurt began. “Kurt’s vision is the reason I wanted to work with him,” she said. “He gathered a talented group of people at D3 who are committed to ensuring the equitable distribution of unbiased and essential information. We will continue our mission of providing high-quality, accessible information to drive informed decision-making.”
Data Driven Detroit (D3) launched the 2014 One D Scorecard in May (read more about that here). Today, we’re writing to share more about our process for making this interactive data tool through a Q & A with NiJeL, a team of data scientists and developers that D3 collaborated with to build out the Scorecard. But first, a little context about the project:
In its third iteration, the new Scorecard makes exciting strides in data management and presentation. Working with NiJeL, we focused our resources and energy on two key components of development: an administrative tool for data management on the back end; and an interface powered by interactive data visualizations. We also revamped our data by updating and curating indicators, creating indices for each of the five Priority Areas, and incorporating a data deep-dive using original Opportunity Mapping research from the Kirwan Institute.
Let’s dig into four questions with Lela Prashad and JD Godchaux of the NiJeL team.
D3: Way back when we first started working together, we warned you that we were managing the One D Scorecard data through Excel workbooks, a single workbook per indicator (over 50 workbooks at one time!). It was a bit of a data nightmare in a few ways, especially when it came time for annual updates or when we needed to compare individual indicators across geographies. How did you sift through our data and come to the new centralized solution we’re using today?
NiJeL: Good question! We realized early on that we needed an easy way for D3 staff to update indicators as these datasets are updated, rather than all at once, say on an annual basis. We also understood that we wanted an automated way to add these new data to the One D Scorecard website as soon as an indicator is updated, and the only reasonable way to make this happen was to build a database to house all these indicators. So, we built a MySQL database and modified Xataface, an open-source software designed to add a simple admin interface for a MySQL database.
Once we had honed in on using these tools, we went through each Excel workbook and added each indicator to the MySQL database, slightly reorienting the data from the Excel sheets to make it easier to use. We then wrote two scripts, one to simply pull all of the relevant indicators for each region and package them up all together so the website could create the visualizations that it does, and another to calculate the five Priority Area index values and the overall One D Index score for each region.
Now, the staff at D3 can update any specific indicator by uploading a CSV (comma-separated value) file with any new data they would like to add. Once these new data are added, the web app will update the site visualizations once there is a critical mass of the data from each year to make the index calculations meaningful. We’re hoping this will be a big improvement!
D3: While older iterations of the Scorecard ranked regions based on their performance in a single indicator, we took that a few steps further this year using indices. An index lets us roll up the individual indicators that comprise a Priority Area into a single summary score, and then roll up each of those five Priority Area scores to create a One D Index Score for each region, making comparisons comprehensive and straightforward. Our favorite feature is how an index calculates on the fly and smartly recognizes when too few indicators for a given index have been updated to update the index itself. Can you share a bit about the process for programming in these analytical features to the highly visual front end?
NiJeL: Of course! As you mentioned, we want to be smart about how we’re calculating the index scores for each of the five Priority Areas and the overall One D Index, and we want to do this in the context of new data being continually added to the database. To accomplish this goal, we programmatically look at the group of indicators in each Priority Area and determine if more than 50% of these indicators have data for the year in question. For instance, the Economy Priority Area has 7 indicators, so if 4 or more of these indicators have data available for 2012, then we would calculate an Economy Index for 2012.
However, for us to go ahead and calculate a One D Index and include the year in our visualizations, each Priority Area would need to surmount the 50% threshold. Once that occurs, the indicators and Priority Area indices are added to the visualizations and the data become available for download.
NiJeL: Well, we had the distinct advantage of working with two individuals, one being Ms. Hartman and the other being D3 Project Manager Jessica McInchak, who both were interested in web interactive design and building interactive visualizations. Both actually contributed to the codebase for the One D Scorecard, which is an extremely rare thing for a designer and a project manager to want to do, but both Ms. Hartman and Ms. McInchak were excited to have the opportunity which made for a fantastic working relationship.
Ms. Hartman’s design for many of the elements in the One D Scorecard were inspired by other designs live on other websites, and so when it came time to build these visualizations, we did have some examples to view, though most were written using other tools like Raphael. We decided to use D3.js mainly because of its flexibility — it allowed us to be able to build the visualizations as closely as possible to what Ms. Hartman designed. The most challenging aspect of building to static wireframes, like the ones Ms. Hartman designed, is understanding the intended interactions and transitions between states within a particular visualization. It’s challenging as a designer to draw out intended interactions and as a developer to follow through on those intentions, but our close collaboration with Ms. Hartman and Ms. McInchak, minimized any differences we had on building the interactions as intended.
It’s tough to pick our favorite chart to build, since they all had their challenges and rewards, but we’d have to say the “array of pinwheels’ visualization (where viewers can see each region’s pinwheel in an array of rows and columns) was our favorite to build.
In this visualization, the pinwheels load such that the region with the highest One D Index value is placed in the upper left corner and the remaining regions are placed in descending order from left to right and top to bottom. Visitors can reorder the pinwheels by selecting a specific priority area to view from the “Organize By” drop down menu. Building this chart required extensive use of D3.js transitions, which allowed us to be creative in how we moved from one state to another. When visitors select a different Priority Area (or a different year of data) to view, we effectively run three separate transitions on each regional pinwheel. First, we change the color of each pinwheel slice, setting the Priority Area selected to its full opacity and setting the opacity of the other slices to almost fully transparent. At the same time, we change the size of the pie slice if a visitor has selected to view a different year with the time slider. Finally, we reorder the pinwheels in descending order based on the index chosen, but that transition only occurs after a 1 second delay to allow the first set of transitions to complete. Building these transitions in an attempt to clearly compare the differences across regions and indices was the most challenging and fun part of the development.
D3: The 2014 One D Scorecard presented a lot of opportunities to collaborate. Not only did we work with your team around development, but we also partnered with the Kirwan Institute to integrate their Southeast Michigan Regional Opportunity Mapping initiative. Kirwan’s original research was presented through static maps of the overall index scores. What was your motivation and method for interactively mapping both the index and individual indicators?
Crosslet is particularly designed to allow visitors to explore one or multiple variables to see how each is connected, and to see representations of those connections on a map and a simple frequency distribution bar chart. For instance, if a visitor selected the median household income variable, and then selected the income range of $0 – $50,000, they would see only the geographies (census tracts) that have median household incomes below $50,001. They would also see the frequency distribution of the Opportunity Index, high school completion rates, and vacant property rates. Clearly, the distribution of each of the other variable is skewed toward the negative end of the spectrum when we select that income range. However, if we click on the selected range and drag it toward a higher income range, we can see the frequency distributions of the other variables shift toward the more positive end of the spectrum along with income, and on the map we can see which census tracts specifically fit these new criteria. One can also select a range of any other variable on the map to further filter these data. We think that gives visitors a great entry point to exploring these data and drawing their own conclusions about the drivers behind opportunity in the Detroit metro region.
The new One D Scorecard is a powerful tool for its users to access data through visualizations, but it’s also a powerful data management system for D3 to maintain and scale these datasets into the future. And we couldn’t have built it without the awesome team at NiJeL. We’re already counting down the months until the newest annual data indicators are released so we can do our first update!
Check out the 2014 One D Scorecard at onedscorecard.datadrivendetroit.org, and NiJeL’s GitHub repository to explore the code driving the interactive data visualizations.
If you’re interested in talking more about code and collaboration for the One D Scorecard or beyond, connect with D3’s Project Lead Jessica McInchak at firstname.lastname@example.org.
In November 2013, the Motor City Mapping project moved forward at a dizzying pace, with the goal of identifying every blighted structure and empty lot in the city of Detroit. Loveland Technologies was in the process of finalizing the Blexting application for the city-wide parcel survey, and the Michigan Nonprofit Association hired more than 100 community surveyors. Meanwhile, in these early stages, Data Driven Detroit (D3) focused its surveying expertise on developing and testing the questions that would drive the survey. During this time, D3 received a number of requests from community groups to integrate additional questions into the survey process, ranging from assessing historic relevance to estimating rehabilitation costs. Unfortunately, with the speed of the project and the complexity of the existing survey, the Motor City Mapping team was forced to delay such modifications until future phases of the project. However, in one of these instances, a stunning example of grass-roots organizing and collaboration between the Michigan Historic Preservation Network, Local Data, and D3 resulted in the Historic Resource Survey, one of the richest datasets that would emerge from the Motor City Mapping project.
Background for the Survey
The early weeks of Motor City Mapping focused on surveying properties located within six areas that the city of Detroit identified for disbursement of the $52 million in Hardest-Hit (HHF) funds. These funds are required to be used for targeted demolitions in stronger-market areas, and the city faced a short timeframe in which to disburse the awarded resources. Due to these caveats – the emphasis on demolition, and the quick turnaround required by the conditions under which the grants were awarded – some groups expressed concerns that the HHF deployment would result in demolitions of structures with considerable historic significance. The Michigan State Historic Preservation Office received determination from the U.S. Department of Treasury that the HHF program was not eligible for preservation oversight, accentuating these reservations.
In response to these worries, the Michigan State Housing Development Authority – the state department charged with disbursing the HHF dollars – exempted designated historic districts from demolition efforts. However, eligible but not yet designated historic districts did not receive this protection. Concerned about the potential loss of historic structures in these areas, the Michigan Historic Preservation Network (MHPN) partnered with Preservation Detroit and the Detroit Land Bank Authority to advocate for bringing a preservation perspective to the process of identifying structures for demolition.
Bringing the Survey to Life
Though every organization involved in Motor City Mapping supported the aims of MHPN and Preservation Detroit, the initial requests to add a historic component to the survey faced the same obstacles as other issues that were brought to the team’s attention. By this time, the survey was in full motion, and adding additional questions into the Blexting application was virtually impossible. Not to be dissuaded, Emilie Evans from MHPN and the National Trust for Historic Preservation reached out to Data Driven Detroit to ask if there was any potential way that the Historic Resource Survey could still be completed. In response, D3 offered up use of its Local Data license. This provided MHPN with access to a mobile surveying application similar to the Blexting platform used by the wider Motor City Mapping survey. D3 and MHPN collaborated further, identifying parcels for the survey based on two criteria – location in an eligible or proposed historic district, and location in one of the six designated HHF deployment areas. Once D3 had determined the survey geographies and delivered them to Local Data, MHPN was ready to commence the Historic Resource Survey…
…from a technical standpoint, that is. MHPN still needed surveyors to collect data on each of the nearly 18,000 properties that were located within the targeted areas. A call for volunteers was met with a tremendous response – nearly 55 individuals from dozens of organizations throughout the city. Offering their evenings and weekends to the project, these volunteers used Local Data to survey 17,500 properties across the City of Detroit in only two weeks.
The Historic Resource Survey answered three questions for each property, focusing on its architectural integrity, how in-keeping it was with neighborhood character, and how well its block remained intact. These questions were then aggregated into an easily-digestible Historic Preservation Score – Very Important, Important, Less Important, or Not Historic – that identified how much of a preservation priority a particular property was for a neighborhood. The survey data makes possible data-driven decisions about both building removal and restoration. Funding sources tied to demolitions, such as HHF, can be focused on properties that are labeled as less important or not historic, while preservation efforts can focus on properties of greater importance – not only measured in their architectural significance, but also by their place as an anchor of some of Detroit’s most vibrant communities.
The full Historic Resource Survey dataset is available for download at parcel level in a variety of formats from D3’s Open Data Portal, http://portal.datadrivendetroit.org.
For more information on the activities of the Michigan Historic Preservation Network and Preservation Detroit, visit the following websites:
For more information on the Local Data application used in the Historic Resource Survey, visit www.localdata.com.
The nation-wide movement toward public data transparency and democratization is continuing to gain support. As cities including Portland, Chicago, New York, Louisville, Ann Arbor and others are embracing Open Data in government by creating web sites for citizens to easily view and download data, the potential for developing useful applications driven by these data [Read on...]
Data Driven Detroit (D3), an affiliate of Michigan Nonprofit Association (MNA), is excited to participate in the release of the Detroit Blight Removal Task Force’s final report, “Every Neighborhood Has a Future…And It Doesn’t Include Blight.”
Detroit Blight Removal Task Force Co-Chairs
Glenda Price, President of the Detroit [Read on...]
Several years ago, One D began as an effort to bring together parallel research and resources aimed at facilitating and evaluating regional development in Southeast Michigan. Although One D disbanded in 2011, D3 continues to be a steward of the One D Scorecard – first crafted by One D partners as a comprehensive blueprint for [Read on...]
This post is the next in a series of profiles of partner organizations using data from Data Driven Detroit to successfully support their work.
A shortage of safe and reliable public transportation presents a huge roadblock for many Detroit residents, especially children interested in participating in after-school and summer programs. The Youth Transit Alliance, funded [Read on...]
This Q&A is the sixth in a series of profiles of Data Driven Detroit staff members.
Diana Flora comes to Data Driven Detroit as a Detroit Revitalization Fellow, but was introduced to D3 years ago through former D3 staff and classmates. Since joining the D3 team, Diana has been the D3 lead on the [Read on...]