Dataproc Cookbook

Get up to speed with Dataproc, the fully managed and highly scalable service for running open source big data tools and frameworks, including Hadoop, Spark, Flink, and Presto. This cookbook shows data engineers, data scientists, data analysts, and cloud architects how to use Dataproc, integrated with Google Cloud, for data lake modernization, ETL, and secure data science at a fraction of the cost. Narasimha Sadineni from Google and former Googler Anu Venkataraman show you how to set up and run Hadoop and Spark jobs on Dataproc. You'll learn how to create Dataproc clusters and run data engineering and data science workloads in long-running, ephemeral, and serverless ways. In the process, you'll gain an understanding of Dataproc, orchestration, logging and monitoring, Spark History Server, and migration patterns. This cookbook includes hands-on examples for configuring, logging, securing clusters, and migrating from on-prem to Dataproc. You'll learn how to: Create Dataproc clusters on Compute Engine and Kubernetes Engine Run data science workloads on Dataproc Execute Spark jobs on Dataproc Serverless Optimize Dataproc clusters to be cost effective and performant Monitor Spark jobs in various ways Orchestrate various workloads and activities Use different methods for migrating data and workloads from existing Hadoop clusters to Dataproc