If you want to get a good idea of Twitters presence in the US, look at the picture above. Those blue and red dots are all tweets, red is happy and blue is sad.
Who would have thought you can measure people’s wellbeing by analyzing Twitter data?
Last week I attended a presentation by Chris Danforth, UVM professor and Associate Director of Vermont Complex Systems Center. Most importantly, Danforth produced compelling research along with Peter Dodds and their team. The data that came with their research allowed them to invent a nifty tool called the Hedonometer.
The Hedonometer is an online tool used to measure the public’s overall happiness through tweets. This is achieved through sentiment analysis, which is basically a fancy term for judging an overall attitude toward something based on the words used.
Click here to read the full story behind the Hedonomter algorithm.
It should be known by now that Twitter has an API that grants access to it’s data in order to be integrated into new software. Twitter will allow software engineers to harness up to 1% of Twitter’s user data using their API. Long story short, the tools are available for anybody to analyze millions of tweets – you just have to know how to use the tools.
When building the Hedonometer, Danforth was given the unique opportunity to access 10% of Twitter’s user data (that’s a lot). When asked how he managed to pull those strings, Danforth said, “I emailed the Twitter guy.”
This serves as a lesson to all: don’t be shy.
How it works
The Hedonometer measures happiness over 100 million tweets. The words within the tweets are each weighted to reflect a particular score between 1 and 10. A higher number means a lower level of happiness, and visa versa. Of course, words like “sunshine” and “love” are labeled happier than “darkness” or “hate”.
Again, they call this sentiment analysis.
“The study’s findings are based on 5 million individual human scores and pave the way for the development of powerful language-based tools for measuring emotion,” says a paper published by Danforth and his colleagues.
But then there’s the inevitable problem we run into: context. If someone tweets, “I don’t hate sunshine on this dark day,” is that a positive or a negative statement?
In order to account for context, the Hedonometer is designed so that each word along with the “sentiment” of hundreds of millions of other tweets affect one another (side note: there are algorithms that can detect sarcasm by analyzing emojis). Luckily, most people aren’t tweeting confusing double negatives all the time.
The sentiment analysis of Twitter uncovered a variety of interesting findings. One of which is that the happiness of certain words can change over time depending on how it’s used. Take “snow” for instance. People referred to snow much more positively in the summer months than they did in the winter (the grass is always greener).
The image above shows the average level of happiness on Twitter since the Hedonometer’s inception in late 2008. Notice a pattern (other than: holidays are happy)? It appears our happiness goes up and down at a pretty even pace over long periods of time. We’re happy until something happens, then we work our way out of the trouble until all is well again – rinse and repeat.
This isn’t far off from how our feelings change on a micro basis either. Take a look:
When we zoom in on our average day-to-day happiness, we find ourselves having a short attention span for sorrow. Not even the Las Vegas shooting could keep people’s online spirits down for more than a couple days. Then again, most people aren’t constantly expressing their emotions on Twitter. I could be feeling down all week and only tweet about it once. So we can’t consider Twitter as an exact replica of our stream of consciousness when measuring surprise events like the Vegas shooting.
But what about more long-term topics? Twitter proved to be a GREAT source for gauging approval rating for Barack Obama. In another paper by the people who built the Hedonometer, they explain how unsolicited public opinion polls correlate very well with their Twitter sentiment analysis. In fact, their program was able to predict President Obama’s job approval three months in advance.
So how can we implement this?
The folks behind the Hedonometer strayed away from Twitter and instead used Instagram to harness metadata of photos posted by users. Also having access to healthcare data, they found that users who were currently diagnosed with depression posted darker and less saturated photos than the average user. What’s particularly interesting is that the same depressed users exemplified “sad” behavior on social media up to a full year an a half before diagnosis.
Healthcare providers could use this technology to provide care to those who are showing very early signs of depression. Our smart homes could integrate this preemptive alert system as well. After all, social media is the go-to source of expression for many people.
Either way, the point is that we can perform big data analysis of social media behavior in order to stop depression in its tracks.
A mayor of a city could use this to tend to the citizen’s. If the data shows a positive correlation between happiness expressed and the amount of trees present, then action could be taken to plant trees in key areas (working commutes, hospitals, hospice homes, etc). This approach could effectively raise the quality of life for those who need it most.
The bottom line is that tools like the Hedonometer offer immense value. With the billions of data points that Twitter collects every day, we’re able to look in the mirror and dissect our own behavior like never before. How would you implement this technology?