In early September, The Fledgling Fund team released the following open letter in response to ongoing debates about how best to assess media investments:
To Our Community,
Over the past several months, Fledgling has participated in many discussions with fellow funders, filmmakers, practitioners and others who are all wrestling with the question of whether and how to assess the social impact of creative media, and especially documentary film. Emerging tools and platforms, like The Participant Index (TPI), Harmony Institute’s Story Pilot, Sparkwise, and ConTEXT, are attempting to capture social impact in different ways, many using techniques that rely in part on access to big data. We expect others will follow. Like many of you, we have reviewed these platforms and tools as they have evolved and have listened to the robust debate they have sparked, revealing the concerns and push back from many filmmakers and others about this increased focus on “measurement”. This came through loud and clear in Aggregate’s survey of True/False filmmakers released in July in which 62% of respondents answered “no” to the question “Do you think there should be metrics to measure the social change created by a film?” In light of this, we have been thinking a lot about what all of this means for Fledgling, for our grantees, and for the field.
We recognize that not all documentaries have social change goals, rather they exist as works of art to be consumed and enjoyed, not to be assessed in the context of social impact. We are referring in this letter solely to films and filmmakers with social change as a key goal and especially those that have decided to build outreach and engagement campaigns around their films to deepen the connection that audiences have with the social issues presented. As a funder focused on supporting social impact strategies for documentaries, we have a special interest in how these new tools and platforms could help us and our grantees learn. We also are aware that as a funder we have a special responsibility to be thoughtful about our approach to assessing impact.
Overall, we believe that the proliferation of these tools and platforms and the debate they have spurred is healthy and important for the field. There are now more tools in our collective toolbox to document, track, and communicate the social impact of documentaries and their campaigns, and that is a good thing. And, when these insights are used in real-time throughout a campaign’s evolution, they can help shape and strengthen campaigns as they unfold. They also allow project teams to share information with key partners along the way, deepening those relationships. However, we know that not every tool is appropriate for every film and engagement campaign and it is highly unlikely that one tool will capture the complete scope of impact.
When we consider how the various tools apply to our work and the work of our grantees, there are a number of things that are important to keep in mind. First, each tool is developed with a specific purpose and definition of impact by its creators. For example, TPI focuses on the “relationship between social issues, emotional involvement in entertainment, and social actions an audience is inspired to take after consuming entertainment” and ultimately creates a numerical impact score for a project. To understand the value of this Index, it is important to understand the methodology behind it — how social action is defined, what assumptions are made, how information is gathered and over what timeframe. Importantly, we also need to decide whether this methodology captures the kind of social change envisioned for a project before deciding whether or not a project’s score is meaningful and if so how. At Fledgling, we are not interested in the overall numerical score. But we see value in understanding what specific parts of the Index reveal because we can map that against the project’s unique goals, its target audiences and its strategy for change.
Second, many of these emerging tools rely on big data sets that capture online activity. Many use complex algorithms and web crawling tools to determine how a piece of media or a film has influenced a conversation and public dialogue. For some projects, these tools can reveal incredibly interesting information, particularly for those that are seeking to influence a broad national conversation around a particular issue or affect culture change. Ultimately, the more information we have the better and these tools can add value to understanding the impact of film.
However, this style of analysis can capture only very specific aspects of a project’s social impact. It will inevitably miss much of the crucial long-term, off-line, grasstops and/or deeply personal impact that a project can have. For example, a new conversation around a dinner table about an important topic, a moment of personal healing, a quiet donation made to a cause after a viewing and discussion, or a closed-door influencer screening that could have policy impact in the longer term are all incredibly important ways in which documentaries have impact beyond what can be captured by big data. Likewise, a powerful community screening that leads to deep change at the community level or a project that does not have broad reach but has a lasting impact on specific target audiences might be missed. We, the social issue documentary field, often put great pressure on filmmakers to change the world, when changing someone’s world should be considered equally important. This does not mean that tools that rely on big data are not useful, only that we need to be clear about what they can and cannot capture. Indeed, for many outreach and engagement campaigns, these off-line and personal moments of change are the most valuable and most indicative of the deeper impact a film and its campaign can have.
We encourage our grantees to create an evaluation plan that is clearly linked to their own distinct impact goals and strategy and relies on different kinds of data that can help track key indicators of their progress over time. While quantitative or numerical data may be easier to come by and compare, qualitative data, which is more descriptive and observational, in many cases is more appropriate to capture the complexities of social change. With this quantitative and qualitative data, filmmakers can create a story (which we know they can do!) about the impact they have had. This “impact story” allows for deep context that cannot be achieved with numerical data alone. One of the reasons we supported the development of Sparkwise as a platform for understanding and sharing social impact data and subsequently decided to use it for our grantees is its capability to show both quantitative data (number of Twitter followers, Facebook fans, web searches etc.), qualitative reports of impact as well as self-generated stories. Our grantees create Sparkwise boards that are customized to their specific social change goals and they can share those campaign goals as well as rich information about the issues and existing social movements that provide context to the numbers.
We should also note here that it is important that any evaluation methodology distinguish between “attribution” and “contribution”. The vast majority of social issue documentaries and their engagement campaigns are entering into a community (however small or large) of activists, leaders, organizations and coalitions that have laid groundwork long before the films and campaigns were conceived and they will be there for many years continuing to build the movement. This critical work must be acknowledged – just another reason an element of “story” is so important in this evaluation work – so that this relationship can be explained.
Think of the case studies that have been shared by BRITDOC, Media Impact Funders, Active Voice, Harmony Institute, Fledgling and others that rely on both quantitative and qualitative information to create a more complete picture of the impact that a film achieved, sometimes over a long period of time. Often these kinds of “impact stories” offer a more complete and useful tool for understanding the power of social issue documentary. We recognize that creating these case studies can be time-consuming and resource intensive, particularly if done retroactively. However, we believe that filmmakers who are trying to achieve social change, impact producers and importantly funders need to make a commitment to supporting data collection and impact tracking from the moment that campaign planning begins. This front-end quantitative and qualitative data collection will not only inform the campaign but also make impact reporting more efficient and accurate. We are not calling for extensive and expensive evaluation strategies — simply for thoughtful collection of diverse data throughout the life of a campaign that will show broad sample evidence of the ways that the project is affecting the world, communities or individuals, according to stated goals.
We should also point out here that we believe “measuring impact” is not a good description of what we are talking about, as it seems to imply a comparison between other film projects/campaigns or against arbitrary benchmarks. When we think about data collection and assessment we are seeking to understand the success a project has in meeting its stated goals. The importance of the link between a project’s goals and strategy and its impact assessment plan cannot be understated.
Revisiting the Aggregate survey, we wonder how respondents interpreted the word “metrics” in the question noted above. If they thought the question was about finding a set of numbers to measure and compare the social change of films, then Fledgling would stand strongly with the 62% that answered ”No”. But, if the question is about methods for understanding the social change of films and engagement campaigns and those methods include both quantitative and rich qualitative data — relying on big and small data and deep context, then Fledgling would say emphatically “Yes”!
Our goal with this letter is to share our thinking on the subject of if and how to assess the impact of documentaries and their associated outreach and engagement campaigns. In short, we are thrilled that this field is making solid progress in our collective understanding of how films and engagement campaigns make change in the world. The emerging tools increase the options for filmmakers and offer insights into how their creators define impact. Ultimately though, the utility of these tools for individual projects depends in large part on a project’s goals and its strategy for change. We will continue to encourage and support our grantees to tell the stories of the impact they are having — using a mix of data to provide deep context. We are deeply aware that the impact of this work is incredibly complex, often happens over long periods of time and can not all be captured. However, we have seen over and over again that evidence of impact is incredibly useful and continues to move this field forward.
The Fledgling Fund
— Diana, Sheila and Emily

About the Author
Jessica Clark

Jessica Clark

Research Consultant

Jessica is a research consultant for Media Impact Funders, and the founder and director of media production/strategy firm Dot Connector Studio. She is also currently a senior fellow at the Norman Lear Center’s Media Impact Project. Previously, she served as the media strategist for AIR’s groundbreaking Localore project, the director of the Future of Public Media project at American University’s Center for Media and Social Impact, and a Knight Media Policy Fellow at D.C.-based think tank the New America Foundation. Over the past decade, she has led research and convenings with high-profile universities and national media networks, including NPR, PBS, Harvard’s Berkman Center for Internet and Society, MIT, and USC’s Annenberg School for Communication. She is the co-author of Beyond the Echo Chamber: Reshaping Politics Through Networked Progressive Media (The New Press, 2010), and a longtime independent journalist.