Big Data Todd Goldman October 6, 2017

Big Data Poll 3: Keith Urban is Wrong about “Wasted Time”

Welcome to our third installment of our ongoing Big Data polls. (You can read about our first round of data in our “Carpenters” post and the second round in our “Electric Boogaloo” post.)

We collect this great data through MakeBigDataWork, a special website launched by Waterline in partnership with Arcadia Data, StreamSets and Trifacta to help organizations get the most from their data.

The first question I’d like to delve into is one I’ve been eager to talk about for some time. We asked data professionals how much of their data analysts’ time is typically spent finding data compared to time spent analyzing data. Here’s what they said:

Most data analysts (41-80%), it seems, are devoting about a third of their total time to finding the data they need to analyze. That’s a right smart piece of time. It’s like sales professionals spending a third of their day looking for the phone numbers they have to call.

No offense, Keith Urban, but I don’t think the best days of our lives is “all that wasted time” by any stretch.

Looking for data is just not what you want to be paying your analysts to do. You want them analyzing that data, so you can put it to work for your business. (And if I might add, hunting for data—known and trusted data—is one of the big things data catalogs do, so your analysts don’t have to.) So, if I were to ask you this same question, what would your answer be? How much time are your analysts looking for work vs. doing the work?

Another question we asked: How do you give users access to analyze their data? Turns out most (58%) rely on traditional business intelligence (BI) tools like Tableau, Qlik and Drill. Less than 20% depend on either development tools like Spark, MapReduce, etc. or SQL engines like Hive, Impala, or SparkSQL.

This is line with what everyone in the industry has been noticing, right? Analytics are becoming increasingly democratized, and self-service technology will continue to make it easier for analysts and other business users to see and analyze data—the results and value of which can sometimes only be recognized by business users, because who knows how data can be of use in marketing or finance better than a marketing or finance professional? None of the data professionals we queried have moved to Hadoop-native, distributed BI platforms yet, but experts are saying we will start to move toward Hadoop-native tools for more seamless environments.

So what does this info tell us? Combine the ease with which business users are able to play with data on the analytics side with the automated data cataloging tools that dramatically cut time on the data finding side, and soon we will begin to see Big Data truly take off in infinite ways across all data-driven organizations.

Tune in for more from our ongoing Big Data polling and possibly more about Keith Urban.