4.53 out of 5
7619 reviews on Udemy

Apache Spark 2 with Scala – Hands On with Big Data!

Dive right in with 20+ hands-on examples of analyzing large data sets with Apache Spark, on your desktop or on Hadoop!
Sundog Education by Frank Kane
37,292 students enrolled
English More
Frame big data analysis problems as Apache Spark scripts
Develop distributed code using the Scala programming language
Optimize Spark jobs through partitioning, caching, and other techniques
Build, deploy, and run Spark scripts on Hadoop clusters
Process continual streams of data with Spark Streaming
Transform structured data using SparkSQL and DataFrames
Traverse and analyze graph structures using GraphX

New! Updated for Spark 2.3.

“Big data” analysis is a hot and highly valuable skill – and this course will teach you the hottest technology in big data: Apache Spark. Employers including AmazonEBayNASA JPL, and Yahoo all use Spark to quickly extract meaning from massive data sets across a fault-tolerant Hadoop cluster. You’ll learn those same techniques, using your own Windows system right at home. It’s easier than you might think, and you’ll be learning from an ex-engineer and senior manager from Amazon and IMDb.

Spark works best when using the Scala programming language, and this course includes a crash-course in Scala to get you up to speed quickly. For those more familiar with Python however, a Python version of this class is also available: “Taming Big Data with Apache Spark and Python – Hands On”.

Learn and master the art of framing data analysis problems as Spark problems through over 20 hands-on examples, and then scale them up to run on cloud computing services in this course.

  • Learn the concepts of Spark’s Resilient Distributed Datastores

  • Get a crash course in the Scala programming language

  • Develop and run Spark jobs quickly using Scala

  • Translate complex analysis problems into iterative or multi-stage Spark scripts

  • Scale up to larger data sets using Amazon’s Elastic MapReduce service

  • Understand how Hadoop YARN distributes Spark across computing clusters

  • Practice using other Spark technologies, like Spark SQL, DataFrames, DataSets, Spark Streaming, and GraphX

By the end of this course, you’ll be running code that analyzes gigabytes worth of information – in the cloud – in a matter of minutes. 

We’ll have some fun along the way. You’ll get warmed up with some simple examples of using Spark to analyze movie ratings data and text in a book. Once you’ve got the basics under your belt, we’ll move to some more complex and interesting tasks. We’ll use a million movie ratings to find movies that are similar to each other, and you might even discover some new movies you might like in the process! We’ll analyze a social graph of superheroes, and learn who the most “popular” superhero is – and develop a system to find “degrees of separation” between superheroes. Are all Marvel superheroes within a few degrees of being connected to SpiderMan? You’ll find the answer.

This course is very hands-on; you’ll spend most of your time following along with the instructor as we write, analyze, and run real code together – both on your own system, and in the cloud using Amazon’s Elastic MapReduce service. 7.5 hours of video content is included, with over 20 real examples of increasing complexity you can build, run and study yourself. Move through them at your own pace, on your own schedule. The course wraps up with an overview of other Spark-based technologies, including Spark SQL, Spark Streaming, and GraphX.

Enroll now, and enjoy the course!

“I studied Spark for the first time using Frank’s course “Apache Spark 2 with Scala – Hands On with Big Data!”. It was a great starting point for me,  gaining knowledge in Scala and most importantly practical examples of Spark applications. It gave me an understanding of all the relevant Spark core concepts,  RDDs, Dataframes & Datasets, Spark Streaming, AWS EMR. Within a few months of completion, I used the knowledge gained from the course to propose in my current company to  work primarily on Spark applications. Since then I have continued to work with Spark. I would highly recommend any of Franks courses as he simplifies concepts well and his teaching manner is easy to follow and continue with!  “ – Joey Faherty

Getting Started

Tip: Apply for a Twitter Developer Account now!
Udemy 101: Getting the Most From This Course
Warning about Java 11 and Spark 2.4!

Be sure to install a JDK for Java 8 for this course, NOT Java 9, 10, or 11 - and install Spark 2.3, not 2.4.0.

Introduction, and Getting Set Up

A brief introduction to the course, and then we'll get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system! Remember - be sure to install Spark 2.2 and Java 8 for this course.

[Activity] Create a Histogram of Real Movie Ratings with Spark!

Let's dive right in! We'll download a data set of 100,000 real movie ratings from real people, and run a Spark script that generates histogram data of the distribution of movie ratings. Some final setup of your Scala development environment and downloading the course materials is also part of this lecture, so be sure not to skip this one.

Scala Crash Course [Optional]

[Activity] Scala Basics, Part 1

We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it. Part 1 of 2.

[Exercise] Scala Basics, Part 2

We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it. Part 2 of 2, with some hands-on practice at the end.

[Exercise] Flow Control in Scala

You'll see how flow control works in Scala (if/then statements, loops, etc.), and practice what you've learned at the end.

[Exercise] Functions in Scala

Scala is a functional programming language, and so functions are central to the language. We'll go over the many ways functions can be declared and used in Scala, and practice what you've learned.

[Exercise] Data Structures in Scala

We'll cover the common data structures in Scala such as Map and List, and put them into practice.

Spark Basics and Simple Examples

Introduction to Spark

What is Apache Spark anyhow? What does it do, and what is it used for?

The Resilient Distributed Dataset

The core object of Spark programming is the Resilient Distributed Dataset, or RDD. Once you know how to use RDD's, you know how to use Spark. We'll go over what they are, and what you can do with them.

Ratings Histogram Walkthrough

Now that we understand Scala and have the theory of Spark behind us, we can revisit the rating counter code from lesson 2 and better understand what's actually going on within it.

Spark Internals

How does Spark convert your script into a Directed Acyclic Graph and figure out how to distribute it on a cluster? Understanding how this process works under the hood can be important in writing optimal Spark driver scripts.

Key / Value RDD's, and the Average Friends by Age example

RDD's that contain a tuple of two values are key/value RDD's, and you can use them much like you might use a NoSQL data store. We'll use key/value RDD's to figure out the average number of friends by age in some fake social network data.

[Activity] Running the Average Friends by Age Example

We'll run the average friends by age example on your desktop, and give you some ideas for further extending this script on your own.

Filtering RDD's, and the Minimum Temperature by Location Example

We'll cover how to filter data out of an RDD efficiently, and illustrate this with a new example that finds the minimum temperature by location using real weather data.

[Activity] Running the Minimum Temperature Example, and Modifying it for Maximum

We'll run our minimum temperature by location example, and modify it to find maximum temperatures as well. Plus, some ideas for extending this script on your own.

[Activity] Counting Word Occurrences using Flatmap()

flatmap() on an RDD can return variable amounts of new entries into the resulting RDD. We'll use this as part of a hands-on example that finds how often each word is used inside a real book's text.

[Activity] Improving the Word Count Script with Regular Expressions

We extend the previous lecture's example by using regular expressions to better extract words from our book.

[Activity] Sorting the Word Count Results

Finally, we sort the final results to see what the most common words in this book really are! And some ideas to extend this script on your own.

[Exercise] Find the Total Amount Spent by Customer

Your assignment: write a script that finds the total amount spent per customer using some fabricated e-commerce data, using what you've learned so far.

[Exercise] Check your Results, and Sort Them by Total Amount Spent

We'll review my solution to the previous lecture's assignment, and challenge you further to sort your results to find the biggest spenders.

Check Your Results and Implementation Against Mine

Check your results for finding the biggest spenders in our e-commerce data against my own solution.

Advanced Examples of Spark Programs

[Activity] Find the Most Popular Movie

We'll revisit our movie ratings data set, and start off with a simple example to find the most-rated movie.

[Activity] Use Broadcast Variables to Display Movie Names

Broadcast variables can be used to share small amounts of data to all of the machines on your cluster. We'll use them to share a lookup table of movie ID's to movie names, and use that to get movie names in our final results.

[Activity] Find the Most Popular Superhero in a Social Graph

We introduce the Marvel superhero social network data set, and write a script to find the most-connected superhero in it. It's not who you might think!

Superhero Degrees of Separation: Introducing Breadth-First Search

As a more complex example, we'll apply a breadth-first-search (BFS) algorithm to the Marvel dataset to compute the degrees of separation between any two superheroes. In this lecture, we go over how BFS works.

Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark

We'll go over our strategy for implementing BFS within a Spark script that can be distributed, and introduce the use of Accumulators to maintain running totals that are synced across a cluster.

Superhero Degrees of Separation: Review the code, and run it!

Finally, we'll review the code for finding the degrees of separation using breadth-first-search, run it, and see the results!

Item-Based Collaborative Filtering in Spark, cache(), and persist()

Back to our movie ratings data - we'll discover movies that are similar to each other just based on user ratings. We'll cover the algorithm, and how to implement it as a Spark script.

[Activity] Running the Similar Movies Script using Spark's Cluster Manager

We'll run our movie similarties script and see the results. In doing so, we'll introduce the process of exporting your Spark script as a JAR file that can be run from the command line using the spark-submit script (instead of running from within the Scala IDE.)

[Exercise] Improve the Quality of Similar Movies

Your challenge: make the movie similarity results even better! Here are some ideas for you to try out.

Running Spark on a Cluster

[Activity] Using spark-submit to run Spark driver scripts

In a production environment, you'll use spark-submit to start your driver scripts from a command line, cron job, or the like. We'll cover the details on what you need to do differently in this case.

[Activity] Packaging driver scripts with SBT

Spark / Scala scripts that have external dependencies can be bundled up into self-contained packages using the SBT tool. We'll use SBT to package up our movie similarities script as an exercise.

Introducing Amazon Elastic MapReduce

Amazon Web Services (AWS) offers the Elastic MapReduce service (EMR,) which gives us a way to rent time on a Hadoop cluster of our choosing - with Spark pre-installed on it. We'll use EMR to illustrate running a Spark script on a real cluster, so let's go over what EMR is and how it works first.

Creating Similar Movies from One Million Ratings on EMR

Let's compute movie similarities on a real cluster in the cloud, using one million user ratings!


Explicitly partitioning your RDD's can be an important optimization; we'll go over when and how to do this.

Best Practices for Running on a Cluster

Other tips and tricks for taking your script to a real cluster and getting it to run as you expect.

Troubleshooting, and Managing Dependencies

How to troubleshoot Spark jobs on a cluster using the Spark UI and logs, and more on managing dependencies of your script and data.

SparkSQL, DataFrames, and DataSets

Introduction to SparkSQL

Understand SparkSQL in Spark 2, and the new DataFrame and DataSet API's used for querying structured data in an efficient, scalable manner.

[Activity] Using SparkSQL

We'll revisit our fabricated social network data, but load it into a DataFrame and analyze it with actual SQL queries!

[Activity] Using DataFrames and DataSets

We'll analyze our social network data another way - this time using SQL-like functions on a DataSet, instead of actual SQL query strings.

[Activity] Using DataSets instead of RDD's

We'll revisit our "most popular movie" exercise, but this time use a DataSet to make getting the answer more straightforward.

Machine Learning with MLLib

Introducing MLLib

MLLib offers several distributed machine learning algorithms that you can run on a Spark cluster. We'll cover what MLLib can do and how it fits in.

[Activity] Using MLLib to Produce Movie Recommendations

We'll use MLLib's Alternating Least Squares recommender algorithm to produce movie recommendations using our MovieLens ratings data. The results are... unexpected!

[Activity] Linear Regression with MLLib

A brief overview of what linear regression is and how it works, followed by a hands-on example of finding a regression and applying it to fabricated page speed vs. revenue data.

[Activity] Using DataFrames with MLLib

Spark 2 makes DataFrames the preferred API for MLLib. Let's re-write our linear regression example, this time using Spark's DataFrame MLLib API.

Intro to Spark Streaming

Spark Streaming Overview

Spark Streaming allows you create Spark driver scripts that run indefinitely, continually processing data as it streams in! We'll cover how it works and what it can do.

[Activity] Set up a Twitter Developer Account, and Stream Tweets

As a hands-on example of using Spark Streaming, we'll set up a Twitter developer account, and run a script that will keep track of the most popular hashtags from the past five minutes in real time! Plus some ideas for extending this script on your own.

Structured Streaming

Spark 2.0 introduced experimental support for Structured Streaming, a new DataFrame-based API for writing continuous applications.

Intro to GraphX

GraphX, Pregel, and Breadth-First-Search with Pregel.

We cover Spark's GraphX library and how it works, followed by a strategy for re-implementing breadth-first-search using GraphX and its Pregel API.

[Activity] Superhero Degrees of Separation using GraphX

We'll use GraphX and Pregel to recreate our earlier results analyzing the superhero social network data - but with a lot less code!

You Made It! Where to Go from Here.

Learning More, and Career Tips

You made it to the end! Here are some book recommendations if you want to learn more, as well as some career advice on landing a job in "big data".

Bonus Lecture: Discounts to continue your journey!
You can view and review the lecture materials indefinitely, like an on-demand channel.
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
4.5 out of 5
7619 Ratings

Detailed Rating

Stars 5
Stars 4
Stars 3
Stars 2
Stars 1
30-Day Money-Back Guarantee


7 hours on-demand video
3 articles
Full lifetime access
Access on mobile and TV
Certificate of Completion