Researchers Decio Coviello, Andrea Ichino and Nicola Persico studied a group of Italian judges who were randomly assigned cases and who had similar workloads, in terms of the quantity and type of cases they were assigned. The researchers' findings? The judges who worked on fewer cases at a time tended to complete more cases per quarter and took less time, on average, to complete a case. (You can read more about the authors' findings in their recent National Bureau of Economic Research working paper, "Don't Spread Yourself Too Thin: The Impact of Task Juggling on Workers' Speed of Job Completion.")
That doesn't mean all multitasking at work is ineffiicient. In earlier research into information-worker productivity in an executive recruiting company, Sinan Aral, Erik Brynjolfsson and Marshall Van Alstyne found that the level of multitasking matters. Their findings in that study suggested that, for the recruiters, working on more projects in one time period at first increased productivity, as measured by revenue generation. But as the level of multitasking increased, the marginal benefits of additional multitasking declined -- and, at a certain point, taking on still more tasks made workers less productive rather than more so.
Aral, Brynjolfsson and Van Alstyne essentially suggested that excessive multitasking may result in the workflow equivalent of a traffic jam, where projects get backed up behind other projects much the way cars get stuck in traffic when there are too many on a highway at once. (You can read a brief summary of their findings in "What Makes Information Workers Productive," an article from the Winter 2008 issue of MIT Sloan Management Review.)
So perhaps the real question to ask yourself is: Am I multitasking so much that it's significantly slowing my completion of tasks?