Is it the end of Big Data Era?

The advent of Multi-cloud and AI (machine learning and deep learning) has brought in real time data processing and analytics, and with that the Big Data era seems to have come to an end. Rest in Peace Big Data Epoch (2006 – 2019)! Big Data is no longer a technological disruption, just an asset supporting the next generation strategies in multi-cloud, machine learning and real time data analytics.

Hadoop and companies like Cloudera, Hortonworks, MapR (to name a few) widely popularised Big Data. Why was it so widely accepted and popular? It was the ability to process large volumes of variety of data types – structured, semi-structured, un-structured. Businesses had collected so much data and were struggling to process them and make sense out of it. So, advent of tools like Hadoop took centerstage and provided a mechanism to process this vast amount of data. In a way it enabled the social media businesses to dominate and brought the idea of using data for business advantage to mainstream. It changed the mindset where organisations realised data to be an important asset.  

However with the decreasing use of Hadoop and crash in the market cap of Cloudera (less than $3B currently, almost half of what it was worth when it merged with Hortonworks),  I believe the Big Data Analytics in the current form (which is mainly tied to Hadoop) has reached a stage of saturation where it’s already well-established and has no opportunities to mature further and reflect infinite growth that we have seen in the last decade or so.  In summary the Big Data era is over!

What does that mean for businesses? Well, certain changes are permanent and so is Digital transformation, organisations continue to move large amount of data and analytics workloads into cloud. They tap into Data science tools in the cloud and experiment with Machine Learning and Deep Learning. The rapid progress in NLP and computer vision processing capabilities in the recent times has been enabled by Big data.  On that note, not only does multi-cloud deliver flexible modern data management and analytics, but also offers enhanced capabilities of workload shifting to the cloud of choice (AWS, Azure, Google Cloud, VMware and others) thereby preventing vendor lock-in. It also builds bridges and eliminates silos while integrating security, metadata and governance across multiple varied ecosystems.

Recent acquisitions in the analytics space by companies such as Google Cloud and Salesforce is a reaction to the rate at which analytics workloads are shifting to the cloud. AI, machine learning, and analytics have become the primary growth engines for the cloud today. This offers lot more options for the businesses to analyse data in real time as they collect it. Big Data is certainly evolving into AI and Cloud.  Hence a well thought out Multi cloud strategy with AI, ML and analytics capability is the future for the Digital World!