By Gene Banman, CEO, DriveScale, Inc.
Recently there have been several articles with headlines like “Hadoop Has Failed Us” or “Is Hadoop Failing” or “Hadoop not living up to the Hype.” Sure its eye catching to write about how a once hyped technology has fallen, but is it true with Hadoop? Not so fast!
The Strata-Hadoop show just happened in San Jose. Over 4,000 attended. There were almost 200 separate presentations about how to use Hadoop: from installation rules of thumb to AI applications, to IoT apps, to Customer 360 degree analysis to predictive analysis. These were presentations from every industry talking about how they use Hadoop to impact business outcomes. There were over 100 exhibiting companies in the exhibition hall.
Hortonworks went public last year. While they are having some growing pains, one thing is clear from their quarterly reports: the customer base is growing rapidly and node count is doubling every year. The other two big players, Cloudera and MapR, while not public, are also communicating triple digit growth rates.
For sure, Hadoop was over hyped. The reality is “merely” 100% growth per year, from a large base. When something is supposed take over the whole world, this may seem like a come down. But really folks, Hadoop is doing just fine, thank you very much.
Another issue is the definition of “Hadoop”. It used to mean HDFS and MapReduce. If you stick to that definition, then the market is moving on. But what are they moving on to? Spark? Beam? Flink? Presto? These run mainly on HDFS. Hadoop is really a bunch of Apache projects, which use parts of each other to run. This makes it a bit of a mess and makes it a steep learning curve to use these technologies. But really, is there any real alternative when you have to store and process huge amounts of data? The Cloud? Yes, but even there you end up using Map Reduce or Spark or other bits of Hadoop, and the Cloud isn’t the end all and be all. (Wait till the pundits start writing articles about how the Cloud isn’t all it was cracked up to be! That is when you will know the Cloud is mainstream!)
The Strata-Hadoop show was rocking and rolling. Lots of activity. Lots of projects. Lots of very smart people moving things forward. Not very indicative of a dying technology! To the contrary, Hadoop is booming!