Happy? Your smartphone will know

Home Technologist 03 Happiness Happy? Your smartphone will know

New technologies and citizen science offer innovative ways to track and quantify emotions. They are uncovering new ingredients in the recipe for happiness.

Photo of a woman receiving a mild electric shock from a Taser weapon. She looks as if she is laughing.

Why is one person content and another one grumpy? The answer is as elusive as it is intriguing. Even defining happiness is difficult, says Pearl Pu of Switzerland’s École Polytechnique Fédérale de Lausanne (EPFL). “Is it an emotion, an attitude, a mood? What makes us happy today may result in unhappiness tomorrow.”

Below: Chirpy tweeters



While some researchers use new technologies to track emotions, others are turning to smartphone apps to improve everyday lives.

Beam: A prototype developed by students at the University of Michigan School of Information tracks emotions but also helps to improve users’ moods by challenging them to perform random acts of kindness.

MoodKit: Created by psychologist Edrick Dorian, uses a cognitive behavioural therapy approach to track mood, identify unhealthy thinking habits and promote positive activities.

Superbetter: Founded by games designer Jane McGonagal, draws on neuroscience, psychology and medicine to help users develop “personal resilience” to help meet goals and manage stressful times.

An entirely different approach is to steer clear of technology. “One of the vogue things at the moment is mindfulness training, an effort to fully pay attention in the moment and avoid being distracted,” says Paul Dolan of the London School of Economics.

“Of course, you can also do that by turning off your phone, which otherwise you will be unconsciously paying attention to, even if you’re not actively looking for updates.”

Traditionally, happiness studies have looked only at correlations with subjective measures of life satisfaction. They have unearthed positive links to sleep, exercise, social interaction and having a sense of purpose. Today researchers like Pu are trying to unpack what affects emotion on a finer scale, moment-to-moment. For this they rely on new technology and citizen science to track happiness on a far larger scale than any laboratory study would allow.

One method is to combine hard data with what people say. Such apps as Mappiness (from the London School of Economics) and Track Your Happiness (from Harvard University) use data picked up from a phone’s GPS to tag results with location and weather data. But Pu is on a quest to understand even more factors that influence well-being.

Digital lifestyle coach

Participants in her experiments wear sensors that record their physical activities and sleep patterns. The researchers then analyse the sensor data and develop algorithms to identify behaviour, looking for differences between groups of users and also between a person’s current and past behaviour. By identifying healthy routines and habits, the hope is to develop a “lifestyle recommender” that can provide accurate, personalised suggestions on daily activities that would make someone healthier and happier.

Many people are willing to wear sensors, but gathering large amounts of data requires cheap devices that have no contact with a person, says Daniël Lakens, a cognitive psychologist at the Eindhoven University of Technology. Smartphone cameras, for example, can be used to study heartbeats. “Although you can’t see it, with every heartbeat all your veins become slightly wider and your skin becomes slightly redder,” he says. A computer filters and amplifies the signal to detect the rhythm.

In such experiments, a camera was able to measure heart-rate differences associated with different emotional states. It is not a perfect science, says Lakens, in part because some emotions – such as anger and extreme happiness – can look similar physiologically. To differentiate further, heart-rate data might be combined with such other information as breathing rates. Lakens says he plans to work with a clinical psychiatrist to develop a system that gets patients to use their own smartphone cameras to tell them when they are stressed – which they often do not realise. This could help them train themselves to avoid stress.

Photos of three people receiving a mild electric shock from a Taser weapon. All three  look as if they are laughing.

Happiness can take the most surprising form. American photographer Patrick Hall convinced 53 people to let him freeze their reactions with a high-speed camera as they received a mild electric shock from a Taser weapon. Oddly, the pain often seemed to produce more joy than suffering.

The face reader

Another way for a smartphone to read emotions is by studying faces. In the 1970s Paul Ekman, a psychologist at the University of California, San Francisco, devised the Facial Action Coding System to classify facial movements. Using this system, researchers today translate tiny movements of facial muscles into emotions. These “microexpressions”, which are often imperceptible to the human eye, involve sequences that occur in quick succession and are almost impossible to fake. “As individuals we probably don’t feel this sequence, but we can analyse it with a video camera,” says Jean-Philippe Thiran, professor of signal processing at EPFL.

Software developed by Thiran and his team can decode emotions in real environments by tracking microexpressions and eye movements, even in low light or with moving faces. Machine-learning techniques allow the software to adjust for differences between individuals. Thiran is now working with Peugeot-Citroen to install software in cars that can detect a driver’s emotional state. This might detect when a driver is getting sleepy, but it could also look for stress and perhaps adapt the ambience of the car accordingly.

Meanwhile nViso, founded by Thiran’s former PhD student Matteo Sorci, is among a number of start-ups that use facial technology to help market researchers test products with a webcam. Unlike questionnaires, microexpressions cannot lie.

“We take thin slices of behaviour and try to figure out how we feel in a certain context,” says Lakens, “but we never measure huge amounts of data because it’s very costly to follow people for days and weeks.” In the long run, smartphones could well be used as constant behavioural labs that provide researchers with even more information. A Google Glass system, which gathers data on what people are seeing, feeling and doing, could gather unprecedented amounts of data on what affects happiness.

Winning is not always best

Sensors and cameras can help reveal emotions that self- reporting might hide, but scientists are also using smartphones to test general theories about happiness on a wider scale. Robb Rutledge, a neuroscientist at University College London, developed an app called The Great Brain Experiment to better understand subjective feelings.

In his experiment, a small sample of volunteers had to play a game in which their choices between guaranteed cash rewards and risky gambles led to wins and losses. Using reports of how happy they felt at each stage, Rutledge came up with a model that described how a person would feel at any point, based on their winnings and experiences in the game. He found that happiness was not just about success or positive events; it reflected moment-to-moment expectations based on the difference between recent experiences and anticipated rewards. “You could be doing just fine. But if your expectation is high, you might not get any happier when you win,” he says. He tested his model on 18,000 app users, showing that it could accurately predict a player’s happiness.

Rutledge hopes this more complex, quantifiable understanding of happiness will help doctors better understand such mood disorders as depression. “Often patients will have reduced pleasure from things that people usually find pleasurable. We’re hoping we can pin down some of what’s different, for example in their reaction to reward and expectations and give an idea of what treatments might be effective,” he says.

The collection of data on such a large scale carries major responsibility. Who uses the information, and for what protection of end-user privacy versus the benefits these systems can bring to users is really important,” says EPFL’s Pu. She works with experts to protect privacy – for example by scrambling data so that only certain information can be extracted, a process known as obfuscation. The law may need to catch up, too, says Lakens, as there is no real legal difference now between just looking at a video of someone and extracting additional information, such as heart rate.

Engineered content

Daniel Quercia, a computer scientist at the Yahoo Labs based in Cambridge, UK, is harnessing citizen scientists to understand how cities affect emotions. Using a mobile phone app, he asked users which of a random set of two images they considered happier, more beautiful or quieter. Using an algorithm to analyse the pictures, he found which visual cues correlated with preferences. The key to urban happiness seemed to be whatever promoted social interaction. “Green areas, small houses, small streets all made people happy. Negative elements were isolated buildings and moving cars,” he says. Quercia hopes to use the research to build a dictionary of happy elements, accessible to urban designers to help them retrofit cities for happiness. “As a research community, we are increasingly losing the buzz word of ‘smart cities’ in favour or the concept of ‘happy cities’.”

These new insights have already been incorporated in apps that have sprung up to help people live better (see Happy apps, p. 28). Although the changes they inspire may seem small, Paul Dolan, a behavioural scientist at the London School of Economics and author of Happy by Design (2014), says that when it comes to happiness, a nudge in the right direction is powerful. “Listening to French music can make you buy French wine. Citrus smells will inspire you to clean.” Happiness, he says, is no exception: “If we can just become aware of what situations make us happy, we can step back and create environments to make it easier to do what makes us happy.”


The deluge of information that people freely offer up about their mood, circumstances and location is a treasure trove for scientists. Working with Alan Mislove at Northeastern University, Sune Lehmann, a computer scientist at the Technical University of Denmark, analysed 300 million tweets gathered between 2006 and 2009 for words that correlated with happiness.

Californians, known for their sunny temperament, really did seem happier than New Yorkers. Thursday evenings were a low point and Sunday mornings a high. The hedono- meter, a similar tracker by Peter Dodds and Chris Danforth of the University of Vermont, found that… Twitter’s low point came on 29 September 2008, at the height of the international financial crisis, while the standout happy day was 29 April 2011, when Prince William and Kate Middleton were married.

Lehmann cautions against over-interpretation. “Just because tweets contain more happy words does not necessarily mean people are happier,” he says. But the sheer volume of data on Twitter points in a direction that is enormously interesting, he adds.

Pearl Pu, a computer scientist at EPFL, has taken these ideas a step further with a project that charts 20 human emotions, including joy, sadness, surprise, envy and pride. Her algorithm was tested during the 2012 London Olympics and the 2014 Sochi Olympics, pulling keywords from Twitter to create a real-time online visualisation of colour-coded sentiment – red for anger, yellow for joy, blue for worry, etc. – called EmotionWatch. Pu’s team can also use the algorithm in reverse to detect events from social media.

Testing the virtual world But is the online world a trustworthy proxy for reality? To understand how online inter­action relates to daily life, Sune Lehmann launched the “Sensible” study at the Technical University of Denmark. One thousand Smartphones with a preinstalled app were given to undergraduates to understand how they communicate by phone, on social networks and face to face.

Even if people present different versions of themselves on such sites, personality seeps out, Lehmann says. “When you analyse those networks, you analyse how we choose to behave and the conventions of the media. But sometimes they reveal something more. It’s hard to hide who you are,” says Lehman.

By Elizabeth Gibney

More on happiness: What makes people happy; Think yourself healthyThe art of probing emotionsA global guide to gleeReading feelings



Festival hacks, designed by engineering students.

Seducing the heart to repair itself.
Robert West, researcher at EPFLRobert West, researcher at EPFL

A Swiss researcher has created a system that scans Wikipedia for important articles that are missing in other languages. This project…