Why Big Data (Alone) Can’t Make Your Company Better

Earlier this month, The Wall Street Journal wrote about “How Big Data Is Changing the Whole Equation”, and discussed how companies from every sector are leveraging the sea of data now available thanks to the Internet. Much of the work detailed by the WSJ article focuses on growing efficiency. Numerous projects over the last decade have shown us the wonders of digital automation, from leaps and bounds in data collection to robot scientists that hypothesize, experiment, and interpret data autonomously. But mostly, the examples the WSJ cites merely improve existing processes; they don’t transform the problem being solved. In other words, they aren’t innovative.

Sean Lindsay noted last month, “Rigorous data driven approaches seem best suited for finding local maxima, how do you inject innovation and find taller hills to climb?” The answer: wherever data grows your understanding of those you serve. As that understanding grows, it will transform how you think about the processes, products, and services you use to solve problems, whether your own or your customers’.

When I do data mining, I’m trawling for nuggets of data that illuminate how my users and my customers think, and what drives them to behave the way they do. It’s simple enough to say, “86% of our users are on Pinterest,” but the transformative data answers why they’re on Pinterest. That illuminates motivation, which makes for stronger predictions about user behavior. Even data that merely narrows the list of possibilities gives me testable hypotheses.

Say I discover that people who fit my customer profile all read a specific blog. While that knowledge alone may lead me to place ads on that site, it’s more valuable to discover what about that site attracts my people. Knowing that strengthens the mental model I have about my users, and leads me to generate better content, craft a more tailored product, and otherwise serve my customers more effectively.

Where in your company do you have user data, but lack a predictive model of user behavior? That is where data can drive innovation.