End-of-year reviews are boring — and everyone does them. Predictions are boring — and they are hard. Of course, this is different — because big data.

How do big data people go about making end-of-year reviews and predictions? Using data is the obvious answer, but there’s a few issues with that approach: there is no synthesis in data alone — you have to find the story behind data, pick an angle and seek meaning. In addition, that approach does not account for subtle hints, industry knowledge, and big ideas.

To paraphrase Carl Sagan, “we wish to find the truth, no matter where it lies. But to find the truth we need imagination and skepticism data both. We will not be afraid to speculate, but we will be careful to distinguish speculation from fact.” In this spirit, let’s keep things equally opinionated and objective in 2017.

Hadoop turned 10 in 2016. It’s come a long way from a pet project named after a toy elephant to the (metaphorical) stampeding beast now in most every CXO’s name-dropping list. The latest Big Data maturity survey showed that 73 percent of respondents are now in production with Hadoop (vs. 65 percent last year). And yet we’re here to tell you Hadoop as we know it is dead. And that’s not even news.

Hadoop has been constantly evolving, expanding, and re-inventing itself throughout its lifetime. A massive ecosystem has been developing around the initial bare-bones offering, and today Hadoop is more of a platform than “just” a storage and compute framework. The introduction of YARN was a game changer, enabling Hadoop to become a Big Data OS and to break away from its batch-oriented MapReduce origins.

Read the full article on ZDNet