Big data open source software (OSS) like Apache Hadoop, Apache Spark, Presto, and others continue to become industry-standard in enterprise data lakes and big data architectures. As enterprises turn to on-prem OSS deployments to avoid vendor lock-in, lower costs, and harness the power of community innovation, they struggle to balance the requirements of the enterprise (security, SLAs, etc.) with the needs of the business. We’re at a moment of change in this industry, and our Dataproc platform offers a way forward to manage, analyze and take full advantage of your data and the OSS systems you’re already using.

It all started with on-prem deployments of Apache Hadoop in 2010, which left much to be desired as big data architects had to work with rigid machines that couldn’t scale independently, and required significant tuning and testing. They couldn’t tell if the next spike in data or the next OSS release would break pipelines or queries. Then the cloud came along and solved many of these issues. With the launch of Google Cloud’s Dataproc in 2015, enterprises could now decouple compute from storage and allow custom OSS clusters to be built with custom machines. Automating the cluster build and scaling avoided common on-prem challenges and helped set our customers up for the next massive transition in OSS: serverless.

Before we examine the serverless future, let’s first discuss where the industry started and where we are today.

#google cloud platform #data analytics #big-data