Business analytics projects are often characterized by uncertain or changing requirements — and a high implementation risk. So it takes a special breed of project manager to execute and deliver them.
Managers have used business analytics to inform their decision making for years. Numerous studies have pointed to its growing importance, not only in analyzing past performance but also in identifying opportunities to improve future performance.1 As business environments become more complex and competitive, managers need to be able to detect or, even better, predict trends and respond to them early.2 Companies are giving business analytics increasingly high priority in hopes of gaining an edge on their competitors. Few companies would yet qualify as being what management innovation and strategy expert Thomas H. Davenport has dubbed “analytic competitors,” but more and more businesses are moving in that direction.3
Against this backdrop, we set out to examine what characterizes the most experienced project managers involved in business analytics projects. Which best practices do they employ, and how would they advise their less experienced peers? Our goal was to fill in gaps in management’s understanding of how project managers involved in analytics projects can contribute to the new intelligent enterprise. (See “About the Research.”) We found that project managers’ most important qualities can be sorted into five areas: (1) having a delivery orientation and a bias toward execution, (2) seeing value in use and value of learning, (3) working to gain commitment, (4) relying on intelligent experimentation and (5) promoting smart use of information technology.
About the Research
This paper assesses issues that were top of mind for experienced project managers involved in business analytics projects, which best practices they used and what advice they had for less experienced peers. We set out to find common denominators and to describe trends relevant to experienced project managers.
We use the term “project management” broadly to refer to a disciplined way of improving a result, product or service, subject to constraints of time and cost. The idea was to highlight the discipline labeled “best practice” by experienced project managers. For “business analytics” we rely on the definition of Thomas H. Davenport and Jeanne G. Harris in “Competing on Analytics: The New Science of Winning” (Harvard Business Press, 2007). They define it as the extensive use of data, statistical and quantitative analysis, explanatory and predictive models and fact-based management to drive decisions and actions.
The findings presented here are based on data gathered in 32 in-depth interviews during 2010 and 2011, conducted in person and lasting one to two hours. The interviews were conducted on the basis of a semistructured questionnaire with open-ended questions. Additional information was obtained by probing the initial responses. All interviews were taped with permission of the interviewees and fully transcribed for further qualitative analysis.
We restricted our sample of interviewees to experienced project managers who had (1) a minimum work experience of 10 years and (2) at least five years of experience in managing analytics projects. We recruited interviewees through the method of snowball sampling: An initial set of eligible project managers referred us to other project managers whom they considered highly competent. The average work experience of our sample was 16.6 years (median = 15, min. = 5, max. = 35), with an average experience of 11.1 years on analytics projects (median = 10, min. = 1.5, max. = 20).
Among the interviewees were both internal and consulting project managers. They were active in a wide range of sectors, including financial services, insurance, manufacturing, transportation, leisure, retail, media, telecommunications and government. One in three project managers interviewed had been, or was still, active as a business analyst.
Beyond the interviews, the paper draws on the findings of six years of dedicated research in the field of business intelligence, undertaken by the Business Intelligence Research Centre at Vlerick Leuven Gent Management School, sponsored since 2008 by SAS Institute and Enqio. The complete report is titled “Business Analytics Project Managers: What Defines Them?”
1. Having a Delivery Orientation and a Bias Toward Execution
As a starting point, it’s important to understand what makes experienced business analytics project managers tick. The vast majority of our interviewees do not consider themselves different from other project managers. Like other focused project managers, they want to deliver their projects on time and on budget, and they have a strong delivery orientation.
But unlike many traditional project managers, they do not have a plan bias. Instead, they have a strong bias toward execution. (See “Learning From Experience.”) Although our interviewees don’t question the importance of initial planning, their focus is on project execution and delivery as opposed to adherence to the plan. In fact, they start with the assumption that the initial plan will have to change as the project progresses. This is what we mean by “a bias toward execution.”
Learning from Experience
Certain practices can contribute to success. Starting project managers may want to take the following recommendations to heart.
Plan, but plan for change. Start with the premise that the initial plan will have to change as the project progresses, and focus on project execution and the delivery of value, rather than on adhering to the plan.
Adapt project execution to the nature of the project. Iterative development encourages and enables a culture of learning and will help to optimize value in use.
Gain commitment. Engage stakeholders as early as possible and throughout the project, set the right expectations from the start and manage them throughout. Avoid implementing “black boxes.”
Experiment whenever possible. Experimentation is fundamental to the learning process. When experimenting, adopt a pragmatic approach — that is, apply sufficient scientific rigor to each step without losing focus on practical relevance, usability and usefulness.
Work with the situation, not against it. Remember that in most organizations, business analytics is still relatively new. Instead of forcing an approach and cutting out the learning curve, try coaching the stakeholders into cultural change.
Work in partnerships. Business analytics projects are business projects, but look for ways to involve IT. Projects conducted as partnerships yield better results.
Why do analytics project managers have this execution bias? Many say it is because of the inherent complexity of the projects themselves, and they cite three reasons. First, analytics projects are typically characterized by uncertain or changing requirements. Project sponsors and users will often have a vision of what they seek to accomplish with analytics — for example, to improve direct marketing response, reduce inventory or increase service quality and customer satisfaction while controlling costs. But how they will achieve those goals is often unclear and involves further exploration.
The Leading Question
How do experienced business analytics project managers approach their projects?
- They start with the assumption that the initial plan will have to change as the project progresses.
- They enable a process of engaging stakeholders, explaining and managing expectations.
- They rely on intelligent experimentation.
Second, the technology or models for meeting the uncertain requirements are often not known; they may be new to the team, or they may not even exist. This adds to the exploratory nature of analytics projects. Third, users of business analytics applications expect responsiveness, so the applications, by nature, should be highly responsive to user interaction. The challenge, then, is to find a balance between responsiveness and robustness.
Traditional project management methods tend to focus primarily on planning or a priori risk management (as opposed to managing and mitigating risk during execution). However, the uncertainty associated with analytics projects calls for a different approach.4 A growing body of literature on project management emphasizes the importance of adapting management and processes to the project characteristics. So while there may be a set of general-purpose tools for managing projects, different projects call for different managerial approaches. On the one hand, production-oriented and specifications-based approaches emphasize detailed early planning and requirements specification with minimal ongoing change and exploration. On the other, experimentation-based approaches emphasize less-specific early planning, good-enough requirements, and experimental and evolutionary design with significant ongoing learning and change.5 The latter, more adaptive approach, interviewees say, is better suited to analytics projects.
2. Seeing Value in Use and Value of Learning
There is increasing awareness in the project management community that sticking to the original plan does not necessarily provide value. Instead, value comes from a focus on execution and delivery. Experienced analytics project managers say they approach return on investment as a process rather than as a control metric. By focusing on execution, they seek to add value throughout the project’s life cycle, not just at the end of the project.
Our interviewees are guided by the concept of “value in use,” which measures value in terms of how a given asset provides benefits to a specific owner under a specific use.6 The idea is that the assets themselves have no inherent value; they generate value only when they offer specific benefits to their owners or users (for example, by allowing them to do their work differently). Consequently, only when an analytical model or application is actually used can its real benefits (and costs) be identified.
We found that project managers involved in analytics projects usually want to assess the value of the project quickly and accurately. Interviewees explained how they try to capture value both early in a project and throughout (for example, by using iterative feature-based delivery or rapid prototyping). Indeed, capturing value early and often can significantly improve a project’s ROI. For the assessment of value to be accurate, it needs to be carried out with a certain degree of rigor — which, as we have noted and will discuss later, is what our interviewees do.
Many project managers have learned through experience that they can’t expect to be right the first time. A bias toward execution is essential, interviewees report, because it is better to attempt to execute good ideas quickly than to attempt to impose the “perfect” plan. This implies that the focus is not on explaining discrepancies between the plan and actual results but on learning something new in the course of implementation that might justify altering the plan. Similarly, the iterative, incremental delivery described by interviewees assumes that each iteration provides learning inputs.
As a result, project benefits can be expressed in terms of “value in use” and “value of learning” that accrue during the project. Many analytics project managers have adopted project management approaches that tie in with the project management methods that are being developed to support highly complex projects. Adaptive methods assume there is a need to gather information and learn as you go along. These methods typically emphasize rapid delivery of prototypes and require that those involved be allowed to experiment during the project.7
The success of an analytics project is a function of the user’s acceptance of the model or the application. Our data make a convincing case for the value of continuous exposure to user feedback. As a project manager at a European financial services group explained, “Ideally, analysts and users are physically in the same room, or in close proximity.” The design environment and the operating environment should be closely linked, with the analytics project managers facilitating continuous interaction between them.
3. Working to Gain Commitment
Experienced project managers are unequivocal about the importance of engaging business users and other stakeholders as much as possible, as opposed to merely informing them after the fact. As an enterprise-business-intelligence architect at an international transport solutions provider put it: “We don’t want to develop a model just like that. If the business processes aren’t aligned with the model, or if the business doesn’t understand the definitions used in the model, then it simply won’t be used.”
The importance of explaining or clarifying the thinking behind a decision — or, in this case, the analytical model or application — cannot be overestimated. Indeed, interviewees say that one of the major risks of analytics projects is that the decision makers won’t be savvy enough to understand the analysis or the model’s underlying assumptions, and they will try to apply it where it isn’t applicable. Explanation is also crucial in gaining trust, as one project manager at a financial institution notes: “Gut feeling and intuition still take precedence over analytics. No matter how transparent analytical models are, they are inevitably statistically complex. That’s why users find it difficult to put their faith in quantitative data and methods.” And this is why analytics project managers should be pedagogical experts and help open up the black box of analytic models.
Ultimately, our interviewees agree that expectation management should not be overlooked. Setting the right expectations at the beginning (for example, regarding the quality of the data and the applicability of the models) and managing them as the project progresses increases both acceptance and the chances that the project will be successful.
The process described above bears a strong resemblance to what W. Chan Kim and Renée Mauborgne, recognized thought-leaders and authorities on business strategy, innovation and wealth creation, have described as “fair process.”8 Process fairness has proven its worth in diverse management contexts as a way to gain stakeholder commitment to decisions and change.
4. Relying on Intelligent Experimentation
A key element that emerges from the interview data is the importance of experimentation. Many of the analytics project managers we spoke to consider experimentation fundamental to the learning process. This is consistent with leading research by Harvard University professor and innovation management
authority Stefan H. Thomke, who defines experimentation as a fundamental innovation-process activity, consisting of iterative trial and error and directed by insight.9 The execution of an experiment, then, follows a four-step cycle: design, build, run and analyze.
The quality of the experimentation process has a strong bearing on the extent to which the project succeeds. Interviewees tend to be strong advocates of “good experimentation,” which is consistent with the scientific method. Well-designed experiments need clear goals and objectives, which is why the first steps of the scientific method devote significant time and effort to observation, to specifying the questions the experiment is intended to answer and to background research. Project managers need to invest time upfront examining the analytics project and setting the objectives. Good experiments need measurable hypotheses about the expected outcomes and controlled testing of these hypotheses. Interviewees reported spending significant amounts of time setting up the experiments and analyzing the results. What they learn forms the basis for improvements and for the next batch of experiments.
However, most interviewees recognize that a laboratory-style scientific approach is neither appropriate nor practical. They take a more pragmatic10 approach to experimentation. Still, their process has enough rigor to allow a sufficiently accurate assessment of a project’s benefits. Interviewees acknowledge that business analytics is still a relatively new area and that some companies are still learning to incorporate analytics into their business. As a business analytics coordinator at an international food retailer put it: “Our policy is clear: Not using a control group is no option. But the reality is sometimes different. Many of our people are still trying to get to grips with analytics. Conflicts won’t help our case.” Successful, experienced project managers will try to advance the learning curve and coach the stakeholders into cultural change.
5. Promoting Smart Use of Information Technology
So far, information technology has been conspicuously absent from this discussion. Yet intelligent use of IT can allow for frequent and faster iterations between the design and operating environments, and this can improve experimentation efficiency. MIT professor and global IT expert Erik Brynjolfsson, who coined the phrase “IT productivity paradox,”11 has noted that leading companies leverage IT to revolutionize the way in which they innovate by playing on four dimensions simultaneously: measurement, experimentation, sharing and replication.12 The big advantage of IT-based experimentation, he argues, is that it can trace causality in a way that would be impossible with pure measurement and observation.
IT capabilities are key to helping companies to explore as well as exploit their full potential in turbulent markets. But while business has embraced IT capabilities, it is often far less positive about the IT department. Many of the analytics project managers we interviewed identify with the business side of their organizations even if they report to the IT department. They often trace their frustration with IT to negative experiences. As one explained: “Whenever IT is involved, business analytics projects cost more and take more time than planned. They are hideously inflexible. It’s virtually impossible to go ahead with anything at short notice. And not only that, they just speak another language. They really lack the sense of urgency and pragmatism you find on the business side.”
IT departments that ignore complaints from the business side risk being circumvented. In fact, some of the analytics teams we encountered built valuable models and applications independently from the IT department. Still, bypassing the IT department altogether can be counterproductive, especially when the focus is on delivering enterprise value rather than locally optimized solutions and functional value. Certainly with enterprise value, the most appropriate modus operandi would be to approach analytics projects as partnerships between the business side and IT.13
Research suggests that best-in-class CIOs have realized that IT and business need to find better ways to work together.14 By proposing pragmatic solutions and pointing out the consequences of infrastructural decisions, CIOs can become constructive partners, enabling their businesses to make smarter choices. This means that IT should, where possible, pursue opportunities to deliver faster implementation cycles, maintaining just enough process and architectural hygiene to ensure quality and professional support.
But what is just enough process and infrastructure? Enterprise infrastructure remains a long-term investment. The big challenge is to develop a process that provides for flexible infrastructure even as the process itself — the way applications and infrastructure are built and modified — remains stable. This will require infrastructure architects with mind-sets much like those of the analytics project managers we interviewed. Indeed, these architects will need to have a strong bias toward execution, so that IT solutions and infrastructure are rooted in the present without mortgaging the future.
Two things are certain. First, the boundaries between functional domains are blurring within organizations, requiring cross-functional collaboration. Second, it will take experience-based negotiation, not theoretical design, to create just enough process and infrastructure. This is a vital area where experienced analytics project managers can put their interpersonal skills to good use.