ComScore: The Art and Science of Big Data, From the Inside

While comScore, Inc. can rightly take its place among the world’s biggest data purveyors — the digital measurement and analytics company has collected about 14 petabytes of online data from around the globe, at a rate of about 20 terabytes a day — it still has to execute on all that data internally to create valuable insights its customers can actually use.

This, it turns out, is where a lot of organizations stumble: Capturing the value in big data.

A recent report by MIT Sloan’s Center for Information Systems Research (CISR) details how comScore does just that. The secret lies in how comScore organizes its assets to create value from big data, so that it can easily generate big data insights for its clients. (Hint: it has as much to do with culture as it does technology and talent.)

ComScore delivers insights on web, mobile and TV consumer behavior that allows its more than 2,000 clients — advertisers and marketers across a range of industries, from CPG to energy to health care — to understand the performance and value of their digital investments. It bases these insights on measurement data it gathers from 172 countries in 43 markets. It captures 50 billion events a day, and well over 1.5 trillion (yes, with a “T”) interactions monthly — equivalent to almost 40% of the monthly page views of the entire Internet.

The company consistently turns its data into insights, releasing about 150,000 reports a day (output that peaks at about 45 reports per second), according to CISR’s research.

Like any analytics (or product or service) company, how well comScore gets the right insights to the right client depends, to no small degree, on how well it functions internally. But it’s also a function of how well the company is aligned around data as a core asset.

That’s what comScore is doing.

CISR’s research details how comScore achieves value creation from big data through three distinct levers: a cost-efficient, scalable platform; an analytics-savvy workforce; and a deep understanding of its clients.

Tech Strategy: Build, Then Buy

Data to fuel comScore’s reports are culled from four main sources: panel data collected from 2 million online users; census data gathered from sensors placed (with permission) on about 90% of the top 100 U.S. digital media properties; survey data; and data obtained from partners, such as in-store purchase data, that can be used to tie together online advertising with offline purchasing behavior.

These data sources, according to CISR, comprise comScore’s data factory. In the late 1990s, when comScore began seeking technology solutions to manage its data factory, there wasn’t a commercial solution available. So the company’s IT team built their own very efficient — and very patent-protected — platform. Then it moved with the times, according to CISR:

By 2013, the technology team had evolved the platform from a proprietary solution into a service-oriented architecture that supported three key systems running on MapR’s Hadoop and Pivotal Greenplum Database, each having unique workload and scaling requirements. ComScore needs to be relentless about staying on top of its platform. In the last 12 months, data volumes have grown by about 80%.

Talent Strategy: Analytics-Savvy Workforce

With a global workforce of 1,200, comScore invests in ensuring each employee has some level of “data scientist” abilities, according to CISR. It does this through a number of initiatives, including hiring analytical people, primarily from university business schools and math programs. It also invests in working with universities to establish a presence in these programs.

Another tactic is through internal training. ComScore University is an ongoing company-wide training program that is attended by the majority of the company’s employees — regardless of their team or position.

The company has also established groups across the company with varying levels of analytic skills, but with “enough skills overlap to ensure that the groups can communicate. Teams that need to communicate regularly are co-located,” reports CISR. On a similar note, people with diverse skill sets are brought together in “scrum teams” that include a business product owner and analytics professionals.

Client Strategy: Deep Domain Expertise

The challenge in delivering its myriad daily reports is “to deliver reports from which clients can not only extract insights, but also incorporate insights into a work task,” writes CISR.

How it accomplishes this is by understanding what its clients need. While that might sound simplistic, what it boils down to is this: employees having deep domain expertise and an understanding of the problems their clients are trying to solve. According to CISR:

Armed with this knowledge, comScore makes big data “consumable” and proactively helps clients identify actionable insights. It recognizes that data can quickly become overwhelming, and encourages clients to focus on answering a few key questions and then to iterate.

In the end, says CSIR, it’s all about the data — and constant adaptation to keep up with the turbulence in the big data space.