The Media Consortium launched our Metrics Impact Project in 2012 with generous support from the Voqal Fund. The goal of the project is to learn if we can quantify the impact that progressive news stories have on audiences by measuring changes in sentiment. In the metrics world, “sentiment” means “how someone thinks about a topic.” In short, if you publish a story about charter schools, does that change how the public thinks about charter schools? The research is being carried out by Gary King, the Albert J. Weatherhead III University Professor and Director of the Institute for Quantitative Social Science at Harvard, and by his two incredibly capable graduate students, Ariel White and Benjamin Schneer. The project is guided by three core assumptions: 1) Change in sentiment is the right metric for measuring the impact of news. Specifically, the researchers are measuring changes in the sentiment of what they call “activated public opinion,” which are the views of people actively trying to change public policy or the views of others (vs. surveys, which measure the sentiment of the average American) 2) Measuring changes in sentiment on Twitter will closely match changes in the sentiment of “activated public opinion” more generally. 3) Editorial collaborations are more likely to produce changes in activated public opinion on a regular basis than individual stories by individual outlets. How the Project Works The way the project actually works is this:
- Back in 2013, the researchers asked us to choose a few evergreen topic areas for this project. We chose: immigration, education, reproductive health, and recently added climate change.
- The researchers have access to the full Twitter “firehose.” They looked back over the past several years of tweets on these four topic areas and established a baseline for different frames in which people were tweeting about them. Their analysis goes beyond keywords, using an algorithm that is modified by human beings in order to pull apart the nuances of positions like pro-charter schools or anti-charter schools.
- TMC staff (Manolia and Jo Ellen) organize collaborations around the project topic areas. Each collaboration ideally includes at least two original pieces, and at least 3-5 outlets posting the pieces.
- The researchers need to randomize the experiment to ensure that changes in sentiment come from your stories and not from world events. So we pick two possible publishing dates, and the researchers randomly choose one date.
- Participants actively try to not publish on the same topic on the week not chosen.
- The stories all publish on the designated date. Participants in the collaboration retweet each other’s stories, using a common hashtag (which lets researchers track the reach of the specific story). TMC also promotes.
- The researchers measure sentiment on the topic, and compare the sentiment in the week after a story runs to the baseline measurement of sentiment to see if the collaboration had an effect.