As protests for racial justice continue in cities across the U.S., more and more institutions are publicly reckoning with their own roles in perpetuating systemic racism. In the journalism space, a debate about the concept of objectivity is raging: whose view is considered “objective”? Do “both sides” of an issue always demand equal weight? Some journalists are questioning the concept of “objectivity” altogether, and opting for “moral clarity” instead. Within philanthropy, foundations are being called upon to explore and correct their roles in upholding inequity.

But what about the seemingly “objective” practice of impact evaluation, one of the most established ways that foundations measure impact? While evaluation is often used to determine whether or not an organization or effort was successful, it is important to look at whether the evaluation processes themselves are rooted in inequity.

The Equitable Evaluation Initiative (EEI), a five-year project to reimagine evaluation in service of equity, explains:

Evaluation uses the tools of science—systematic inquiry, rigorous data collection, and analysis—to help stakeholders understand the impact of the work they do. This can lead to the assumption that evaluation is “value neutral” because it is “scientific.” However, choosing what data to collect, how it is collected, and who interprets it are all value-laden choices. The understanding of impact will be incomplete, if not outright wrong, if the process is driven only by the interests and values of the most powerful stakeholders.

Questioning evaluation

In implementing evaluation frameworks and practices, funders and evaluators may not be stopping to consider a range of questions about whether evaluation processes are being conducted through an equity lens. For example: 

  • What values are implicit in this process? Do they uphold concepts of equity or negate them? Are they steeped in historically oppressive concepts? 
  • Whose perspective is held up as “objective” or “neutral”? Who is placed in the powerful position to assess others based on notions of “good” and “bad”? 
  • Are evaluations conducted in a collaborative way that distributes power? Or are they conducted in a way that disrespects or even traumatizes people? 
  • Does the evaluation actually reflect the values and experiences of the people doing the work? Who gets to tell the evaluation story, and whose truths are silenced? 
  • How is evaluation data collected, scaled, etc. Is it generalized in such a way that it leaves entire communities out?

Changing evaluation

EEI, led by Jara Dean-Coffey, is inviting funders and other partners to come together to reimagine evaluation in service of equity. Now in year two, EEI is focused on the mindset shifts that must happen in order for true change to take place. (Learn more about EEI’s specific five-year goals here.)If you are committed to equity (in any form),” said Dean-Coffey, “You have a particular mandate to reconceptualize what evaluation is for and about. It is time to interrogate and make explicit the values and assumptions and assure they make sense for the complexity of the 21st century.” 

In our recent conversation, Dean-Coffey was quick to point out that no singular solutions exist; embedding equity in evaluation is not as simple as checking items off a list. That’s why the EEI has focused its early efforts on bringing stakeholders together to reimagine what evaluation could look like in different contexts. The effort began with a particular emphasis on philanthropy, since foundations “have power to shape and resource demand within nonprofits and communities who are not well-served by the prevailing evaluation paradigm.” (EEI is expanding to include other sectors as well over time.)

A core focus of the work is an emerging framework, the Equitable Evaluation Framework, which includes three core principles for equity in evaluation:

  1. Evaluation work is in service of and contributes to equity: Production, consumption, and management of evaluation and evaluative work should hold at its core a responsibility to advance progress towards equity. 
  2. Evaluative work should be designed and implemented in a way that is commensurate with the values underlying equity work: Multi-culturally valid, and oriented toward participant ownership. 
  3. Evaluative work can and should answer critical questions about the: effect of a strategy on different populations, effect of a strategy on the underlying systemic drivers of inequity, and ways in which history and cultural context are tangled up in the structural conditions and the change initiative itself. 

The framework also includes a list of dominant practices within philanthropic evaluation that currently get in the way of advancing equity. For example: 

  • Funders are the ones who set the evaluation criteria and definitions of “success”;
  • Evaluators are positioned as the experts and selected based on traditional notions of qualifications and expertise; 
  • Evaluation funding should go to research and analysis, rather than building relationships and trust.

A vision for equitable evaluation

What will equitable evaluation look like, and how will it be different from what we’ve seen historically? “We do not believe that evaluation should continue to be in service of appraising the value of human worth, motivation or achievement,” Dean-Coffrey said. “That frame assumes there is a ‘good and a bad’ and often that which is deemed good centers whiteness and white dominant framing as the ideal.”

“We envision a field of evaluation practitioners who expand definitions of rigor and validity and embrace 21st-century complexity,” she continued. “Doing so will require practitioners to unlearn white dominant framing, hold the tensions that will be ever present as we shift paradigms and continue to evolve the field of practice to be in service of equity. Building the field ensures we all hold the responsibility and accountability to do better because we are learning how to be better in this space.”

What does this all look like right now, during an unprecedented global pandemic in which many organizations are focused on urgent and immediate responses, without time to deeply reflect on the assumptions behind their processes? It turns out, EEI partners such as the Oregon Community Foundation and the Annie E. Casey Foundation have been able to put these principles into practice during this time of great upheaval. “Our partners engaged in conversation during the pandemic,” said Dean-Coffey. “[EEI is] not about one organization or a specific project or effort. It’s about building a sustainable field of practice where we are pushing the paradigm and boundaries and supporting one another while in the work.”

Go to equitableeval.org to learn more about the Equitable Evaluation Initiative.

About the Author
Katie Donnelly

Katie Donnelly

Research Consultant

Katie is a research consultant for Media Impact Funders and associate director for media strategy and production firm Dot Connector Studio. She formerly served as associate research director at American University’s Center for Social Media (now the Center for Media and Social Impact), and as senior research associate at the University of Rhode Island’s Media Education Lab. Katie has led impact evaluations for many media organizations including PBS, Working Films, and the National Association for Latino Independent Producers. She has conducted extensive impact research, particularly on the power of documentary film, and has written about the power of media to make change for numerous academic and journalistic publications. Katie has created many educational toolkits that use media to dig into social issues, including curricula addressing youth and gender, substance abuse, and gender-based violence.