Breaking Logjams in Knowledge Work

If you work in an organization, you know what it’s like to have too much to do and not enough resources to do it. Digital tools for communication and collaboration are meant to make it all more manageable, but access to technology often can’t fix the root causes: poor work design and entrenched organizational behaviors.

The costs of overload are well-documented: It makes people less creative, less productive, more prone to illness, less likely to hit deadlines and goals, and more likely to leave their organizations to work elsewhere.2 And it’s been implicated in many major accidents and disasters, from BP’s Texas City Refinery explosion to the more recent U.S. Navy ship collisions.3 But, despite the evidence, many leaders continue to believe that their organizations thrive when overloaded, often both creating pressure and rewarding those who deliver under duress. It’s a popular but pathological approach to management.

U.S. manufacturers suffered mightily under this approach for decades, until many found a better way.

Before the 1980s, plant managers tended to believe that keeping every person and machine busy was the key to success. If everybody was busy, the thinking went, the plant would produce more. But visits to Japanese manufacturers and books like The Goal4 revealed that this approach actually undermined performance. Today, factories are run differently. On the whole, managers have become much more aware of which operations are critical to overall performance — and manufacturing and assembly plants are both more efficient and more flexible than they were in the 1980s.

Nevertheless, the “keep everybody busy” theory remains alive and well in other settings, particularly in knowledge work. Though it hasn’t been studied as extensively in such contexts, evidence suggests that in many types of jobs — for instance, serving bank customers, performing complex surgeries, and developing cutting-edge products — organizations overload their employees in hopes of maximizing the performance of the enterprise.5 They have a lot to learn from manufacturing, where managers have adopted a “pull” system for controlling the number and the rhythm of tasks in a work process.

In this article, we explain how this concept from the world of physical work can be used to improve resource allocation and prevent overload in other settings. We also explore how “visual management,” a technique often used in agile project management, can make it easier to apply pull thinking to an entire development portfolio by rendering nonphysical tasks more tangible. To illustrate, we describe two recent work-design changes at the Broad Institute of MIT and Harvard, a biomedical and genomic research center in Cambridge, Massachusetts, where one of us, Sheila Dodge, oversees the main technology platform. The first intervention streamlined lab operations, a setting similar to manufacturing and assembly operations, and the second improved the flow of R&D and technology development work. Though Broad may seem like a specialized case, our experience suggests that managers in just about any knowledge-based organization struggling with overload can learn from the institute’s past mistakes and process improvements.6

Managing a Lab Like a Factory

Completed in 2001, the sequencing of the first human genome took almost 10 years and cost $2.7 billion.7 A few years later, in 2004, the Broad Institute was launched with the mission of transforming medicine by systematically understanding the genetic underpinnings of disease through cutting-edge analysis and new technologies. The cost of genomic sequencing has dropped more than 100,000-fold and can now be done in a matter of days for about $1,000.

Today, the Broad Institute’s organizational structure comprises two distinct components: (1) a set of research programs that explores how genetic information might provide a window into the origins of diseases like cancer and diabetes, and (2) a set of technology platforms that supports the research by analyzing samples (typically blood or tissue) and identifying DNA sequences. 

But Broad wasn’t always set up that way. It started as a distributed research-focused organization staffed with chemists, biologists, and applied mathematicians, and the genomics technology platform resembled the research labs. Work was done in small batches, often following informal or even improvised processes. Given the highly educated and capable people Broad hired, there was never a shortage of new ideas. This loose configuration of dedicated staffers produced rapid advances in sequencing technology — until that growth revealed the limits of Broad’s approach to managing and doing the work. In 2012, cycle time for processing samples was more than 120 days, leaving Broad unable to keep up with the increased industry demand for sample analysis. Researchers from collaborating institutions began sending samples to other labs.

To address this challenge, Broad changed its approach to scheduling work in its genomics lab from a traditional “push” system to one based on “pull,” which streamlined the flow of samples from chemical manipulation to analysis to sequencing. In a push-based system, tasks are effectively decoupled, and each person “pushes” as much work as she can to the next step in the process, whether or not the next person is ready for it, often creating costly overload. In a pull system, in contrast, the amount of work in the system is carefully controlled, leading to both improved transparency, which enables learning, and greater productivity. Since it’s an approach that could benefit any organization where tasks — whether physical work or knowledge work — tend to pile up between steps in a process, we’ll describe here how the change played out at Broad.

The costs of a “push” process. Broad’s struggle to handle the demand for its services was rooted in the use of a push system to manage the flow of lab samples. When a sample arrived, it would immediately go to the first, preparatory step of the analysis and sequencing process, where it would sit until someone could turn to it. (See “Push Versus Pull: Two Systems for Managing Workflow.”) When that step was completed, the sample would move to the next step, where it would again wait for processing and so on, until it reached the sequencing machines that read the DNA.

The samples that accumulated between steps constituted work-in-process inventory, or WIP. When used properly, WIP can improve overall throughput by decoupling steps — even if the person upstream from me is stuck on a hard task, the WIP inventory allows me to keep working. Scholars in operations management have developed sophisticated models for figuring out exactly how much WIP should be placed between operations in a manufacturing process.8

Unfortunately, practice doesn’t always follow theory. The team at Broad was working hard every day to push samples through the system, and yet performance kept getting worse. The piles of WIP continued to grow, far exceeding any optimum level. When somebody needed a specific sample, it could take two days to find it. Managing the consequent congestion and confusion occupied an increasing portion of the leadership team’s time.

To better appreciate the challenges created by a push system, consider the process from an individual’s perspective: When WIP accumulates at each step, the person executing a particular operation often faces more work than she can complete in a given shift. She may then engage in local reprioritization, meaning she looks at her pile of tasks, determines which ones are most important, and works on those first. Though this is a sensible approach from an individual perspective, when each person (or team) in a process chain prioritizes work differently, the performance of the work system becomes increasingly variable. If a task happens to be given a high priority by everyone in the chain, it gets done quickly. But that means another task has been moved to the bottom of several to-do lists, and it might take weeks or months to move through the system.

Before switching from push to pull, the lab staffers at Broad started each day by prioritizing their own tasks, asking themselves, “Is there a particularly important set of samples that needs to go first? Should I do the next sample in the pile or respond to the angry researcher who just called to complain about not receiving her data?” As thousands of in-progress samples stacked up, not only did the average cycle time grow, but outcomes also became less and less predictable. Some samples were completed relatively quickly while others took six months or more.

The tendency of push systems to produce long and unpredictable cycle times creates another problem. When a really important piece of work comes along, people don’t want to risk having it get stuck in a WIP pile. So they work around the system to ensure that the task gets prioritized at every step. In factories, this is called expediting. Before the lean revolution, it was not unusual for plants to dedicate staff to hand-carry “special” jobs through the production line. But expediting is like a narcotic — the more you use it, the more you need it. When a piece of work is expedited, all the other WIP tasks are deprioritized. Eventually, those tasks will be so late that they also will require expediting, creating a vicious cycle.

At Broad, production team members developed daily schedules but rarely adhered to them for more than a few hours before reshuffling tasks to meet shifting demands. Locally reprioritizing and expediting tasks created an almost constant need for firefighting. When a technician wanted to start preparing a sample for sequencing, the first thing she had to do was find it. Often, halfway through her search, her attention would be directed to another set of samples that suddenly had become a higher priority. Members of the operations team spent their day responding to complaints, leading to an increasingly inefficient allocation of resources. Despite working longer hours, the lab was falling further behind. Morale suffered, and arguments erupted daily as team leaders tried to figure out why yet another sample was about to miss its promised delivery date.

The benefits of a “pull” process. To get operations out of constant overload and firefighting, Broad’s genomics platform switched to a pull system. The key to understanding the difference between push and pull is to recognize that WIP inventory is a double-edged sword. Though it is intended to help mitigate variability in speed and productivity between steps in a process, it also hides information that supervisors and operators could use to manage and do the work more effectively. In a push system with lots of WIP, an operator can focus on her individual task with little regard for what is happening around her. But a pull system forces a broader awareness. It sets clear limits (both upper and lower) on WIP accumulation. When an individual or team hits one of those limits, that’s a sign of an underlying problem. Managers can trade off short-term productivity and long-run learning by adjusting the WIP limits. Tighter limits allow them to identify and fix problems in the system; a wider span leads to fewer hiccups and more short-run throughput.

At Broad, implementing pull began with reconfiguring the inventory holding areas between steps. A simple color-coded system now provides technicians with clear signals about the state of their operation relative to the overall production system. Each operation now has a WIP box that has three sections, colored green, yellow, and red. If the box is completely empty, then the technician should process samples. Once the green area is full, she can slow down, and filling the yellow area means she is nearing the end of the day’s work. A full red section signals that it is time to stop. If a technician is done with her work for the day, she can quickly assess the overall state of the system by looking at other people’s boxes to identify a colleague who might be behind and need some help. By providing clear produce and stop signals, a pull system promotes effective line balancing.

Pull systems also provide a clear set of vital signs for managers to monitor. At Broad, a quick walk through the production area reveals which parts of the operation are moving and which are stuck. A perpetually full pull box means either the downstream task is moving too slowly or the upstream one is moving too quickly. An empty pull box at the end of the day means that something is wrong with the operation that feeds it. With this transparency, the operations team has been able to identify and address a variety of problems that had been previously hidden by the piles of samples in progress. For example, empty pull boxes showed the team how a seemingly small change in shift schedules meant that sequencing machines would often finish their work on a Saturday or a Sunday but would not be reloaded until the following Monday, reducing utilization.

Limiting WIP between steps and allowing reprioritization only at the beginning of the process resulted in a system that was both faster and more reliable. Resisting the temptation to expedite remained a challenge for the Broad team. With time, however, the culture changed and people began holding one another accountable for sticking with the process. In the morning production meeting, it’s not unusual to hear team members call one another “pushers,” a lighthearted reminder that they are falling back into the vicious cycle of expediting. Even the center director has agreed to refrain from reprioritizing midprocess.

Implementing pull produced significant gains at Broad. With the new system, utilization of the sequencing machines — the single biggest capital investment — rose almost immediately and eventually more than doubled. Today, it rarely falls below 90% and often exceeds 95%. Process cycle time eventually fell by more than 85%, and the variance declined dramatically. A faster, more predictable, and more transparent process has created stability and competitive advantage. The lab receives fewer queries from researchers wondering where their data went. Staffers once dedicated to expediting samples can now focus on fixing fundamental problems that prevent the process from functioning as desired. Resources have also been freed up to innovate, enabling the platform to pioneer a variety of industry-leading services, such as clinical genome sequencing, where data is returned to patients, and cell-free blood biopsy interrogation, enabling minimally invasive characterization of tumor metastasis.

Managing Tech Development

The improvements Broad made in its operations would be impressive in any industry. That said, it was a relatively modest leap from a traditional manufacturing and assembly environment to implementing a pull system in the samples lab. Broad’s next intervention was far more novel: adapting the pull approach to managing technology development. While many management scholars have argued that efficiency and innovation are strict trade-offs, Broad created a system that allows it to be a highly efficient processor and an industry leader in generating new technology.

When knowledge work processes are managed via push, it’s difficult to track tasks in process because so many of them reside in individual email in-boxes, project files, and to-do lists. Complicating matters, talented employees — particularly those in innovation-focused environments — have a knack for continually pushing more new ideas into an organization than it’s equipped to process. Studies of new product development organizations in the consumer electronics and motorcycle industries suggest that R&D systems often have three to five times as many projects in progress as they have capacity to complete.9

The R&D processes in Broad’s genomics platform were a case in point. In 2012, the group had many more ideas for tech development under consideration than it could fully investigate and many more projects under way than its overloaded operations team could ever implement. What’s more, the push approach to managing tech development created snags similar to those experienced in the lab. Accustomed to exercising considerable autonomy, the development teams would engage in local reprioritization and regularly switch their focus from one idea to another, both reducing productivity and creating variability.10 When they felt a sense of urgency, often because of a customer requirement or a change in a vendor’s technology, they would drop everything and fight the new fire. Given a slow and unpredictable development process, leaders routinely resorted to expediting. Expediting had become the development process, and the genomics platform was losing the technology leadership position it had worked so hard to gain.

To apply the pull concept that had worked well in the samples lab, the tech development teams used visualization to give their less-tangible work a physical “face.” Managers have done this in risk and crisis management contexts for decades, often capturing the data needed for a pending merger or acquisition in a “war room” or dedicating a room to “incident command” after an accident. Though visualization is less common in “peacetime,” it can be equally effective for managing day-to-day work.

Our physical environment shapes how we perceive and process information.11 In physical work, it’s visually obvious when excess WIP inventory piles up and production lines stall out, so colleagues naturally converge to help. But when work on a key component of an R&D project stops, it doesn’t usually generate a clear signal to the rest of the organization. Visual management makes it easier to see what is moving and what is stuck. Broad’s technology teams made their work more tangible by drawing a simple schematic of the development “funnel” on some unused wall space and creating a separate box for each major stage (feasibility, design, and validation). Working together and drawing from multiple emails, spreadsheets, and project files, they generated a list of all projects under way. They then transferred each one to a Post-it note and placed it on the funnel diagram in the box corresponding to the development phase that it was in.

Though creating a visual representation of an individual project is not novel (agile teams do it all the time), it is fairly rare to create one for an entire development portfolio, as the R&D group at Broad did. The exercise led to two insights. First, there was an obvious lack of common prioritization: Nobody was aware of every project, there was little consensus about which ones mattered most, and many projects overlapped or competed with others. Second, the system had too much work in process. Comparing the number of current projects with recent delivery history showed that employees had at least twice as much work as they could complete in the best of circumstances.

With the unintended consequences of continually pushing new ideas into an overloaded development system now visible, the teams began meeting in front of the funnel board weekly to determine which activities in the portfolio were in trouble and needed to be escalated for consideration by leadership.12 For each project, a set of Post-it notes captured three elements: relevant activities (such as developing a testing protocol), the name of each activity’s “owner,” and target completion dates. During the weekly meeting, people would briefly report on whether activities were completed on time. If the answer was no, a Post-it note in a contrasting color (usually pink) was placed on top of the original entry to signal that something was not going according to plan. Once a “stuck” activity was identified, team leaders could discuss ways to get it moving, whether by adding resources or by removing organizational roadblocks. After all the activities for a particular phase were completed, the group would discuss whether the project was compelling enough to move to the next phase of development. If it was, then new sticky notes, representing the next set of key activities, were created and placed on the board.

This exercise also helped the group identify and cancel low-priority activities. Over the two years, the number of projects in progress was cut by more than half, reducing the time required to complete the projects that survived and increasing the overall throughput of the system.

But paring down the portfolio wasn’t sufficient to sustain improvement. Broad’s scientists and technicians would continue to generate more good ideas than they had the resources to execute. Without rules for managing the portfolio, the overload was almost certain to return, and painful cuts would again be required.

So the genomics platform made a few adjustments to the weekly meeting and supporting visual board. First, a “hopper” was added in front of the funnel to capture and represent all of the proposed projects that hadn’t yet begun. To facilitate shared prioritization, the group ranked each project in the hopper by potential impact and effort required to complete it. Ideas could be added to the hopper at any time, and new suggestions were discussed and ranked each week. Second, an “agreed and ready” column was added between the hopper and the first stage of the development funnel. New ideas deemed worthy of development would go there. Managers and employees would review those together weekly and adjust priorities as needed. Third, they would also review the amount of work in the various stages of the funnel and pull from the “agreed and ready” column only when people agreed that they had enough space to start a new project.

To determine whether there was space for a new project, the R&D group modified the pull approach used for lab samples. In the lab, samples can be reduced easily to a common unit of work, and therefore it is relatively straightforward to quickly specify the resources needed. In contrast, R&D projects come in a variety of shapes and sizes. So the funnel board by itself does not provide enough information to allocate resources. Instead, the group relies on the weekly check-ins, facilitated by the visual system, to assess the amount of work people can handle, thus allowing the teams to capitalize on the experience and expertise of their members. Only someone who is deeply familiar with a task can accurately gauge how much time and effort it will require. Each activity’s target completion date is based on its owner’s assessment of how long it would take to complete in normal circumstances — that is, in a properly loaded development system. The relevant contributors are then asked whether, given their current loads, they think they can meet that date. If they say no, that’s a sign (like a profusion of pink Post-it notes) of too much work in that part of the system. The new activity can’t move forward until others are completed or canceled.

Leaders also use the board to spot bottlenecks and other individual performance management issues. If certain portions of the funnel are moving more slowly than the others, the resources need to be rebalanced or the relevant employees need to engage with their work differently.

Creating a pull system for technology development at Broad has led to significant gains, just as it has in the lab. Having increased the velocity and throughput of its system severalfold, the development group has freed up resources to create new products and services, including sequencing for commercial clients, processing clinical samples for individual patients, and creating a data platform that enables more scientists to do research using genetic data. Employees report deeper engagement in their work and more success with cross-functional collaboration (fewer “turf” battles, for instance, and better-integrated goals).

As the R&D group developed this system, it dutifully kept every Post-it note that came off the board when an activity was completed and placed it on a large spike. After four years of running the system, managers and employees held a small celebration and counted all the sticky notes in that stack — more than 4,000 separate development activities, or about one every two weeks for each member of the extended leadership team.

Designing Work for People

As we have begun to teach this material at MIT, by far the most common question goes something like this: “Sure, this works in your examples.” (We typically discuss manufacturing, genomic sequencing, and oil drilling.) “But will it work in my organization?” We always give the same reply: “The stuff we teach works only in organizations that have people in them.”

Toyota and other Asian manufacturers catalyzed a revolution in physical work, and Western companies spent the better part of two decades mastering quality management and lean production, an effort that continues to this day. But, while the companies using these methods have developed significant capability in manufacturing and supply chain operations, many have missed a larger message. The tools and practices associated with quality management and lean approaches work not because they are better ways of organizing manufacturing activity, but because they are better ways of organizing human activity.

Creating systems that allow people to see their nonphysical work more clearly as well — both when it is moving and when it is stuck — may represent the next frontier of improved organizational performance.