How Emotion-Sensing Technology Can Reshape the Workplace

As companies search for new ways to improve performance, some executives have begun paying attention to developments in emotion-sensing technologies (ESTs) and software fueled by artificial emotional intelligence. Although we are still in the early days, research shows that these technologies, which read such things as eye movements, facial expressions, and skin conductance, can help employees make better decisions, improve concentration, and alleviate stress. While important privacy issues need to be addressed, the opportunities are significant.

Consider the technology developed by Koninklijke Philips Electronics N.V. and ABN AMRO Bank N.V., both based in Amsterdam, to reduce trading risk in financial markets. Research has shown that traders in heightened emotional states will overpay for assets and downplay risk, a condition known as “auction fever” or “bidding frenzy.” To address this problem, the companies jointly developed a tool called the Rationalizer that has two components: a bracelet attached to the trader’s wrist that measures emotions via electrodermal activity (similar to the way a lie detector works) and a display showing the strength of the person’s emotions using light patterns and colors. Researchers have found that when users become aware of their heightened emotional states, they are more likely to rethink their decisions. In addition to helping individuals improve performance, the aggregated data from such settings can help managers understand how internal and external environmental factors influence the risks taken by groups.

Individuals are also more prone to make mistakes when they are not paying enough attention. Although multitasking has become standard in many jobs, there are some activities, such as air-traffic control and fast-paced buying and selling, where maintaining one’s undivided attention is critical. In a high-profile foul-up in 2005, a trader working for Mizuho Securities Co. in Tokyo intended to sell a single share of a stock it owned for about 610,000 yen (which was approximately $5,000). By mistake, he placed an order to sell 610,000 shares for one yen. The company was unable to cancel the sell order, leading to an estimated loss of $224 million.

Although such egregious blunders are rare, the story speaks to how important it is to hold the attention of employees involved in high-stakes activities. ESTs can help people improve their focus, often with relatively minimal technological investment. For example, recent research has found that slow or uneven cursor movements can be an indication of distraction or negative emotions. Detection doesn’t require installing expensive hardware, but rather just some additional code or software to computers or smartphones.

Professional athletes have been early adopters of tools that can help people sharpen their focus to gain a competitive edge. Major League Baseball All-Star Carlos Quentin, National Basketball Association All-Star Kyle Korver, and Olympic gold medal swimmer Eric Shanteau are among those who have used special headsets produced by San Francisco-based SenseLabs Inc. to monitor cognitive performance and develop customized training aimed at shoring up their personal weaknesses. Microsoft Corp. has also conducted research on the use of wearable sensors in an effort to understand, among other things, what work activities are associated with changes in emotion and when people working on certain types of tasks should take breaks.

In settings where employee engagement is critical, the ability of managers to recognize boredom is vital. Assuming that data can be accessed without compromising privacy or anonymity, managers will soon be able to watch for signs of boredom in an underperforming team and take steps to counter it. Indeed, researchers at Telefónica I+D in Barcelona have developed an algorithm that analyzes smartphone activity for such signs. On the basis of a combination of data points — including how often users check their email, whether they log in to Instagram, whether they are adjusting their device settings, and how much battery power they consume — the algorithm correctly identifies user boredom more than 80% of the time. It can tell when employees use their phones to pass the time as opposed to pursuing specific goals.

In light of such discoveries, managers can seek to redesign processes that induce boredom or alternate them with other activities that employees find more engaging. ESTs, moreover, might help managers figure out which work schedules work best for particular teams: Employees in one group may be most productive in the early morning, while another group may do better later in the day. Meeting schedules could be organized to take advantage of this information.

Reducing Stress and Burnout

Although some types of stress can help people focus, research shows that too much stress is detrimental to productivity, creativity, and job satisfaction, not to mention psychological and physical health. What’s more, stress can reach harmful levels long before people are aware of it. In some organizations, human resources departments try to monitor stress levels using surveys. But surveys don’t necessarily capture how employees actually feel, in part because people don’t always know when their stress levels are elevated. Having a tool that provides a quantifiable, objective measure of stress would be extremely helpful.

As with tools to improve decision-making and focus, numerous options are available, including smart watches and fitness trackers that detect stress by measuring changes in heart rate and sweat (through what’s known as electrodermal activity). These measures can identify small changes that users themselves don’t notice. And as with algorithms that monitor smartphone usage for boredom or cursor activity for distraction, stress-related information can also be drawn from the hardware that people are accustomed to using every day. For example, a study by MIT’s Affective Computing Lab found that computer users who were under stress pushed harder on keyboard keys and held the mouse more tightly. Other research has found that it’s possible to detect stress-related surges in heart rates by monitoring the changes in the light reflected off users’ faces with an ordinary webcam.

We have found that there can be important benefits to monitoring stress at both the individual level and across the organization. At the individual level, managers can learn when people are under sustained pressure (and therefore more susceptible to recklessness, burnout, or conflict with others) and take steps to help ameliorate such situations. At an organizational level, measuring physiology (for example, heart rate or electrodermal activity) can help managers identify stress “hot spots” among teams and functions. Using wristbands or webcams, for example, managers can pick up on problems relating to excessive workload or interpersonal conflict and respond to them, often before employees are aware they exist. Employees may be spinning their wheels on frustrating, unproductive activities (for example, arguing over who has responsibility for specific tasks). Having access to this data might allow managers to create a “heat map” indicating where the problem is concentrated.

Addressing the Barriers

As companies become interested in ESTs, they will need to address barriers related to cost, complexity, and issues of privacy. (See “Implementation Barriers for Emotion-Sensing Technologies.”)

The cost- and complexity-related barriers seem to be relatively straightforward — both have been declining, and numerous low-cost/low-complexity options are already available. Allaying the privacy concerns, however, will be trickier. Many employees are highly skeptical of monitoring technology and uneasy about how ESTs might be used. A fundamental issue is who will get to see the data and whether the data will be broken down individually or aggregated across groups. Such concerns are understandable given that much of the value will come from measuring and managing aspects of behavior that people are unable (or perhaps unwilling) to self-report. Even if all parties agree to common rules for consent, anonymity, and personal well-being, there are lingering issues. For example, what happens if ESTs uncover medical issues that individuals aren’t aware of or wish to keep private?

One can speculate that privacy concerns will become less problematic when the people being measured are the beneficiaries and when disclosure is voluntary. But even then, there are dicey issues, such as whether an employee interprets feedback in an unexpected way or overadjusts to correct behaviors. With that in mind, managers can attempt both to maintain oversight and to reduce employee concerns by doing the following:

1. Be sensitive to employee concerns. Prepare your organization for using ESTs through education and transparency. Explain how the tools can benefit employees by reducing stress and risks of burnout. One potentially useful strategy, known as BYOD, involves inviting employees to bring their own devices to work. Under this scenario, individuals maintain a sense of ownership over the deployment of ESTs and the data they are gathering.
2. Develop data governance agreements. Employees should have sole control over their personal emotional data and be able to stipulate what types of usage are permitted (for example, data can be used only on an aggregate level, and no one can drill down into individual data signatures).
3. Similarly, assure employees in written agreements that emotional data will be used only for specific business goals. For technologies that rely on broad-stroke measures, such as webcam-based emotion detection, data gathering and analysis should be directed toward highly specific and well-defined outcomes.

As long as organizations operate responsibly, we believe employees will gradually become comfortable with the gathering and analysis of physiological, behavioral, and emotional data. Although this won’t happen overnight, several trends suggest that trust can be built over time. Millions of individuals already use smart watches and fitness devices like Apple Watches and Fitbits, and many people share their workout and nutrition data openly on social media. Social media itself has conditioned us to accept and even embrace new levels of personal transparency. The challenge will be to introduce new devices and measures into workplaces in a way that empowers performance, mitigates privacy concerns, and generally reassures employees that the benefits are mutual.