Recently, we blogged about the latest installment of the annual NewVantage Partners survey, which found that of the 68% of organizations that haven’t yet succeeded in creating a data-driven organization, 95% point to organizational alignment and agility along with their company’s culture as the biggest obstacles in achieving complete big data adoption.
Was I surprised to see this stat? No. Still, it’s incredibly disheartening all the same when all the ingredients you need to drive success in big data are there for the taking–when all that’s holding organizations back is the ability to put those all those ingredients together.
TechCrunch once published a great piece written by a couple of big data experts on the overarching data culture that’s required to drive big data success. The article talks about requiring a single “blessed” source of clean, high quality data that the entire company has access to. Then, there’s creating a data dictionary–a key area where the entire effort tends to stumble, because of businesses that don’t do a good job of creating glossaries that are populated with clear agreed-upon definitions that everyone can understand.
However, as the article also mentions, it’s not enough to make high quality data available from a central source. People need to be able to access it. This can be a tall order in the era of increased regulation and need for governance. You can’t just sit and wait for IT to get around to responding to a request for access. That access has to already be built in, automatically delivered according to pre-defined user roles. Furthermore, while there are all kinds of great visualization and other tools for analytics that democratize data for everyone within the organization, these users still need to be reasonably data literate to come up with their insights and present their findings in an effective manner. Those findings then need to be factored into the decision making process.
It’s remarkable how much of what’s needed to establish a data-driven culture can be found in an automated data catalog. With it, organizations can check off the boxes for making their data known, governed, trusted, and available. Business analysts can search for data using common business terms. Of course, there are the challenges. Evidence-based decision making requires the sharing of data, which can be a hurdle, because people don’t want to lose control of their data. They also don’t often have the budgets (or the inclination) to convert their data into whatever format they are told to use. And who feels like documenting it? But here again is where Waterline can help, letting people share their metadata while keeping data access native. This allows teams to make their data findable without having to give anyone full access. Plus: the documentation is automatically generated.
As we’ve mentioned before, allowing data scientists and business analysts to spend less time looking for data and more time using it isn’t enough. It must be clear that gut-driven decision making is out and data-driven decision making is in. The big data vision must be effectively illustrated and communicated to everyone. There must be company-wide access. Processes must be developed that keep data teams on point. People need to be trained–another big ask since people can be extremely resistant to behavioral change even when it’s simply a matter of getting them to use a new tool or adopt an official taxonomy. (This is one of the reasons why our AI-driven data catalog looks and feels like a shopping app. Driving greater use of any new tool requires this element of familiarity and usability.) The catalog must also be accurate and up to date, which is why Waterline automatically crawls the data assets, detects changes and automatically updates the catalog.
Finally, data needs to permeate decision making from the top down, throughout every department and every business unit. Leadership needs to lead by both example and mandating that the burden of proof always come down to the actual proof. An anecdote is not research! Once you’re able to put all these elements in place, your organization can start benefiting from the more data-minded culture that’s been established. And then, the NewVantage Partners survey stat that shows most companies aren’t yet data-driven (and won’t be anytime soon) will be good news, because your organization will be.