With an internet stuffed full of 2016 projections, Matt Turck’s stands out as providing clarity on where we have been and where we are going in Big Data. Read the full article on Matt’s blog.

In a tech startup industry that loves its shiny new objects, the term “Big Data” is in the unenviable position of sounding increasingly “3 years ago”.   While Hadoop was created in 2006, interest in the concept of “Big Data” reached fever pitch sometime between 2011 and 2014.  This was the period when, at least in the press and on industry panels, Big Data was the new “black”, “gold” or “oil”.  However, at least in my conversations with people in the industry, there’s an increasing sense of having reached some kind of plateau.  2015 was probably the year when the cool kids in the data world (to the extent there is such a thing) moved on to obsessing over AI and its many related concepts and flavors: machine intelligence, deep learning, etc.

Yes! We’ve seen an industry that is increasingly “over” Big Data, even while only a small fraction of companies have implemented it correctly. The hard truth is that many companies are so process-bound and locked into aging data database architectures that they have been unable to make a change.

Eventually those who do not make the necessary upgrades will be handicapped, while those that jump on the adoption curve will gain a competitive edge.

The funny thing about Big Data is, it wasn’t a very likely candidate for the type of hype it experienced in the first place.

Products and services that receive widespread interest beyond technology circles tend to be those that people can touch and feel, or relate to:  mobile apps, social networks, wearables, virtual reality, etc.

But Big Data, fundamentally, is… plumbing.  Certainly, Big Data powers many consumer or business user experiences, but at its core, it is enterprise technology: databases, analytics, etc: stuff that runs in the back that no one but a few get to see.

At the outset, the idea of Big Data seemed exciting, and the companies that were the faces of the movement were trendy and smart. Marketers who wanted to be trendy and smart jumped in with both feet. However, the reality is that Big Data implementation is the realm of the technologists in the back room, rather than the creative brand management types.

It didn’t help the situation that several truly great taglines were created around the Big Data idea.

In order for Big Data to do all of those breathtaking, dynamic things it promised, some fundamentally boring (to most people!) data processes have to be accomplished. Once these steps are taken, the power of Big Data is truly dazzling, but getting there takes fortitude, determination and a solid master plan.

Another key thing to understand: Big Data success is not about implementing one piece of technology (like Hadoop or anything else), but instead requires putting together an assembly line of technologies, people and processes.  You need to capture data, store data, clean data, query data, analyze data, visualize data.   Some of this will be done by products, and some of it will be done by humans.  Everything needs to be integrated seamlessly. Ultimately, for all of this to work, the entire company, starting from senior management, needs to commit to building a data-driven culture, where Big Data is not “a” thing, but “the” thing.

In today’s world, Big Data implementation is ultimately a decision that leaders make to prioritize long-term capacity-building against short-term investments in time, resources and money. However, a few years from now executives will be facing a situation where the lack of technology infrastructure is a huge liability; the pressure to upgrade will leave executives with fewer options.

The next set of large companies (call them the “early majority” in the traditional technology adoption cycle) has been staying on the sidelines for the most part, and is still looking at this whole Big Data thing with some degree of puzzlement.   Up until recently, they were hoping that a large vendor (e.g., an IBM) would offer a one-stop-shop solution, but it’s starting to look like that may not happen anytime soon.  They look at something like our Big Data Landscape with horror, and wonder whether they seriously need to work with all those startups that often sound the same, and cobble those solutions together.

Bottom line: YES, Big Data is absolutely relevant and will be an essential building block for the way business is done going forward. You want to be on the leading edge of this curve.

Let Us Know in the Comments Section Below!