Measuring Emotions in the Digital Age

Let’s face it — emotions matter. Whether we’d like them to or not, emotions play a role in decision-making, performance, and overall well-being, and it’s impossible (and undesirable) to stop people from experiencing them — even in environments where we’d often rather they didn’t, like the workplace. Given the omnipresence and impact of emotions in our lives, they have become a popular topic among researchers in business and management. Traditionally, many studies have focused on negative emotions in the workplace, given their ability to disrupt. Research has proved that emotions such as anger can be linked to higher levels of incivility among colleagues and that some negative emotions can be contagious.

When it comes to addressing employees’ emotions at work, strategizing the best response is often more difficult than one expects. After all, before addressing an employee’s problem, managers should first understand and clarify what the person is actually experiencing. But what if employees are experiencing multiple negative emotions? How do we know if employees are accurately reporting their feelings? These types of questions have generated a unique field of inquiry into emotion measurement and have fueled our research. One type of measurement is the real-time tracking of facial expressions, which represents a new and promising digital approach to emotion measurement in management.

The majority of prior research has measured emotions using different forms of self-reporting. The most widely used technique currently uses standardized numerical rating scales; for example, a respondent may be asked to rate a feeling from 1 (“not feeling anything at all”) to 7 (“feeling very strongly”). This allows researchers to capture emotional measurements with specificity and to cleanly compare results among different participants. While these self-reported measures are convenient to administer, they are not always problem-free.

One concern is that participants do not always accurately report their true feelings. Inaccuracies from respondents withholding, misidentifying, or misrepresenting emotions stem from a variety of causes, such as feeling embarrassed to report certain emotions. The act of distributing self-reports can also precipitate a negative response, which can cloud results. In a study we have conducted on the link between trust and disgust, we found that participants completing a self-report scale on how they are feeling adversely affects their own emotional state. This is likely because they see the questionnaire as a nuisance, or because the process of reflecting on one’s emotions is aversive.

Facial-expression-based recognition software allows researchers to go further in the study of emotion measurement without the limitations of self-reported methods. This technology provides an immediate, unobtrusive, and objective reading of real-time expressions of anger, fear, anxiety, sadness, and happiness. In psychology, researchers used emotion recognition software to examine the relationship between business leaders’ smiles and the leaders’ cultures. They found that top-ranked American leaders appeared consistently more excited in photos than top-ranked Chinese leaders. Economics researchers have used face-reading software to show that the happier individuals are, the more likely they are to donate money to a charitable organization. There have even been applications in the field of finance, where researchers have illustrated a link between exuberance and asset price bubbles, as well as between fear and low asset prices.

Using this emerging technology in their methodology affords researchers unique insight into human behavior. Emotion recognition software offerings have ballooned in recent years, and products such as Noldus FaceReader, iMotions, and Emotient offer researchers and organizations several advantages over traditional self-reported measures. The Noldus FaceReader uses a series of algorithms to find and analyze 500 key points in the face to detect emotions objectively and unobtrusively, using facial movements from photos and videos recorded in real time. Deep learning and AI play an important role here too, as this software relies on multilayered neural networking to recognize patterns in the face and classify facial expressions from image pixels. It provides users with readings of the six basic universal emotions (sadness, happiness, anger, fear, disgust, and surprise) on a scale from “not present at all” to “present at maximum intensity.” Measuring emotions in such an unobtrusive way can capture what happens in a typical work or social interaction, allowing emotions to be measured as they develop in real time and ensuring that emotions are unaffected by the act of measuring them.

This technology offers new possibilities when it comes to several areas in management, including issues such as trust, negotiations, and deviance. In one of our studies using Noldus FaceReader, we found that individuals experiencing disgust judged other people in the study as less trustworthy and were less likely to risk lending them money. With the rise of virtual work spaces, and meetings and collaborations often happening across applications like Zoom or Slack, researchers can now record an employee’s virtual interaction with a colleague to examine a variety of research questions. Do employees experience anxiety when a colleague criticizes their work? Does this anxiety depend on the critic’s age or gender? In studies that examine deviant behavior, researchers often deploy computer tasks where a participant can lie to earn money. Determining whether certain emotional expressions correlate with greater unethical behavior could provide insights into the relationship between emotions and deviance. Do angry participants lie more? Are participants happy after they cheat and get away with it?

That being said, facial-expression-based emotion recognition software brings up many ethical issues and questions surrounding privacy. There are major questions to consider: What are the ethical boundaries of the data that an observer can gather when it comes to facial expressions? Is it ever inappropriate to measure someone’s facial expressions and use that measure to determine emotional state? Are we always required to tell participants that their expressions are analyzed? What if simply telling them they are being videotaped alters their facial expressions? These are important questions currently left unanswered, and ones that organizations and managers should consider carefully before adopting technology for specific emotion measurement practices.

When we look outside of the company, there are also many possible opportunities to use face-based emotion recognition analysis to improve interaction with clients and customers. For marketers, product owners, and sales teams, an important goal is often getting customers through the funnel from the first point of contact to a concluding transaction. With facial recognition software, organizations could analyze the types of interactions that put customers in more positive emotional states and examine the relationship between emotional state and likelihood to make a purchase.

There are also opportunities here for employee training. For example, customer service employees can use the technology to hone and validate their skills in interpreting emotions of customers, allowing smoother interactions and less misunderstanding.

Understanding the links between specific emotions and behavior is a crucial building block in developing effective emotion regulation techniques. The sheer data supplied by face-based software recognition also allows researchers and practitioners to study how minute changes in emotions might actually yield significant changes over time. With such big data readily available, we are able to identify patterns previously undetectable with self-report emotion questionnaires.

In the digital age, face-based emotion recognition software offers a revolutionary approach to gauging employee emotions. It can do so objectively and unobtrusively, and allow us to tackle novel research questions with significant organizational implications. However, we must be vigilant in using this software responsibly, keeping in mind that the primary goal is to help employees deal with negative emotions in a constructive way, which leads to better outcomes for themselves and the company.