Do grant-supported media projects incite change, or are they simply an expensive way of preaching to the choir?” writes New York Times media correspondent Michael Cieply. “Ultimately, the answers may help determine which projects get financed, which formats are favored and how stories are structured.”
His July 6 piece, “Participant Index Seeks to Determine Why One Film Spurs Activism, While Others Falter,” brought a high-stakes conversation that has largely been taking place behind closed doors into the light of public scrutiny.

The story, which covered an inaugural report from The Participant Index project, has generated mixed reactions: on the one hand, admiring retweets and on the other, critiques of TPI’s methodology as too instrumental or reductive.
Finneran P photo_2“What is lost when both the storytellers and the social sector start working towards quarterly results reporting, echoing mega-corporations earnings reports?” blogs documentary strategist Patricia Finneran. “Will outcome measurement trump nuanced understanding of the human experience?” TPI has “raised some concerns from the independent film community,” observes Shaady Salehi, the executive director of Active Voice “specifically: how can you assign a numeric value to art?” An overemphasis on short-term actions “could lead to the bankrolling of more didactic narratives about issues that lend themselves to relatively straightforward solutions,” writes Alison Byrne Fields, the president of creative strategy firm Aggregate. Or perhaps, the whole impact discussion has just “jumped the shark,” suggests impact producer Jennifer MacArthur, tongue mostly-in-cheek, in a recent newsletter for her consulting firm, Borderline.
The on-the-record reservations that these seasoned outreach experts express mirror a groundswell of off-the-record concerns expressed by makers and culture-shifters, who say they’re reluctant to critique impact methodologies advanced by the large foundations that support their work.
However, even the critics acknowledge the growing consensus that, as Finneran writes, “we need tools and measurements to help shape our work, and to report back to funders and supporters.” As Salehi observes, “While there is not necessarily agreement on the topic of measurement, this trend illuminates something exciting: If more and more players want to better understand how films contribute to our culture and our communities, there is a growing acknowledgement that film (and art, more broadly) is essential to our social fabric.”
In other words, now is not a moment for closing down the impact dialogue. Instead, it’s an opportunity to further deepen it.
Competing Visions
As we’ve been reporting in our AIM Analysis section, foundations are playing a central role in supporting research to understand the social impact of media. Since our November Assessing the Impact of Media event at the Paley Center, we’ve explored a wide range of perspectives and tools for understanding how public interest productions influence society, including the Public Media Database, ConText, Sparkwise, ImpactSpace, Crimson Hexagon, and others.
Among these, TPI represents an unusual partnership between a grant-funded academic research center and a for-profit entertainment company. Johanna Blakley, who co-heads the Media Impact Project at USC-Annenberg’s Norman Lear Center, reflects on the genesis of this collaboration on her personal blog, concluding “I’m really excited to see for-profit companies like Participant Media devote significant resources and time to measuring the impact of media. Here’s hoping that many others will follow in their path.”
Public-private partnerships are less of a comfortable prospect, however, for those filmmakers whose commitment to the work arises exactly from doubts about the motives and methods of commercial media. What looks like professionalism to some smacks of creeping technocracy to others. What’s more, the past decade’s explosion of media that is both commercial and partisan has shifted expectations about documentaries’ role in public debate.

This ambivalence is reflected in the new Impact Field Guide & Toolkit published by BRITDOC earlier this month.

As we reported in April, they surveyed 200-plus filmmakers about their views on impact evaluation. They found that 76% feel increasingly pressured into becoming campaigners, and 55% agree that “The art and story come first. Social change agendas get in the way and can undermine a work’s integrity.”

The BRITDOC team locates these responses in a persistent tug-of-war between “art, impact, and money,” observing:

There is no point attempting to resolve these tensions simply by dispelling one or other of the three elements involved.  The best results and happiest lives are achieved by figuring out where the best balance lies for each team on each project. The director is the creative leader without whom there would be no film to work with but film teams also contain producers, financiers and impact professionals. They may all have different visions and priorities. It’s going to be key to get everyone on the same page with a shared vision.

Building Tools from the Ground Up
In doing so, it’s helpful to reframe media assessment not as punitive judgment, but rather as one learning tool among many. Filmmakers, funders and allies who do so open the door to inventive new impact approaches.
For example, in crafting the case study for Media Impact Festival selection American Promise, we discovered that producers and co-directors Joe Brewster and Michèle Stephenson have been working hand-in-hand with transmedia producer Andrew DeVigal to prototype a new mobile community engagement tool called Harvis.
Harvis allows audience members to register their real-time reactions to a screening or presentation using a mobile app. Event organizers can then return to the moments where audience members were most engaged, and use those as focal points for a group discussion. Learn more about how the tool evolved here.
Room to Grow
While it might be easy to pigeonhole this as a post-production market research tool — in the same way that TPI focuses on capturing short-term audience responses — both also provide valuable insights for makers and funders seeking to understand how their future productions will resonate.
What’s more, these tools are not mutually exclusive, but mutually reinforcing. For example, Harvis might be just as useful to the production team for A Place at the Table — another one of our Media Impact Festival selections, produced by Participant Media — as it has been for American Promise. Both documentary campaigns have relied on targeted community-based screenings to educate and engage stakeholders and move them to action.
There is both room and demand for varied approaches to analyzing impact and building informed media strategy. In fact, time and again, research into media impact has concluded that no single tool or solution will do.
In a recent report commissioned by the Media Impact ProjectMeasuring Media Impact: An Overview of the Field, Rutgers University professor Philip M. Napoli writes:

Different projects require different methodological approaches to impact assessment. Moreover, a comprehensive approach to impact assessment typically requires the application of multiple methodological approaches that address different levels of analysis that reflect the different spheres of potential impact (e.g., on individual attitudes/behaviors, on media debate/discussion, on public policy.)

Many of the toughest questions about media’s role in society are still wide open. This offers a tremendous creative opportunity for funders, researchers and creators seeking to understand these dynamics together. Let’s keep that conversation flowing.

About the Author
Jessica Clark

Jessica Clark

Research Consultant

Jessica is a research consultant for Media Impact Funders, and the founder and director of media production/strategy firm Dot Connector Studio. She is also currently a senior fellow at the Norman Lear Center’s Media Impact Project. Previously, she served as the media strategist for AIR’s groundbreaking Localore project, the director of the Future of Public Media project at American University’s Center for Media and Social Impact, and a Knight Media Policy Fellow at D.C.-based think tank the New America Foundation. Over the past decade, she has led research and convenings with high-profile universities and national media networks, including NPR, PBS, Harvard’s Berkman Center for Internet and Society, MIT, and USC’s Annenberg School for Communication. She is the co-author of Beyond the Echo Chamber: Reshaping Politics Through Networked Progressive Media (The New Press, 2010), and a longtime independent journalist.