Apache Spark 2 with Scala – Hands On with Big Data!
New! Updated for Spark 2.3.
“Big data” analysis is a hot and highly valuable skill – and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, EBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive data sets across a fault-tolerant Hadoop cluster. You’ll learn those same techniques, using your own Windows system right at home. It’s easier than you might think, and you’ll be learning from an ex-engineer and senior manager from Amazon and IMDb.
Spark works best when using the Scala programming language, and this course includes a crash-course in Scala to get you up to speed quickly. For those more familiar with Python however, a Python version of this class is also available: “Taming Big Data with Apache Spark and Python – Hands On”.
Learn and master the art of framing data analysis problems as Spark problems through over 20 hands-on examples, and then scale them up to run on cloud computing services in this course.
Learn the concepts of Spark’s Resilient Distributed Datastores
Get a crash course in the Scala programming language
Develop and run Spark jobs quickly using Scala
Translate complex analysis problems into iterative or multi-stage Spark scripts
Scale up to larger data sets using Amazon’s Elastic MapReduce service
Understand how Hadoop YARN distributes Spark across computing clusters
Practice using other Spark technologies, like Spark SQL, DataFrames, DataSets, Spark Streaming, and GraphX
By the end of this course, you’ll be running code that analyzes gigabytes worth of information – in the cloud – in a matter of minutes.
We’ll have some fun along the way. You’ll get warmed up with some simple examples of using Spark to analyze movie ratings data and text in a book. Once you’ve got the basics under your belt, we’ll move to some more complex and interesting tasks. We’ll use a million movie ratings to find movies that are similar to each other, and you might even discover some new movies you might like in the process! We’ll analyze a social graph of superheroes, and learn who the most “popular” superhero is – and develop a system to find “degrees of separation” between superheroes. Are all Marvel superheroes within a few degrees of being connected to SpiderMan? You’ll find the answer.
This course is very hands-on; you’ll spend most of your time following along with the instructor as we write, analyze, and run real code together – both on your own system, and in the cloud using Amazon’s Elastic MapReduce service. 7.5 hours of video content is included, with over 20 real examples of increasing complexity you can build, run and study yourself. Move through them at your own pace, on your own schedule. The course wraps up with an overview of other Spark-based technologies, including Spark SQL, Spark Streaming, and GraphX.
Enroll now, and enjoy the course!
“I studied Spark for the first time using Frank’s course “Apache Spark 2 with Scala – Hands On with Big Data!”. It was a great starting point for me, gaining knowledge in Scala and most importantly practical examples of Spark applications. It gave me an understanding of all the relevant Spark core concepts, RDDs, Dataframes & Datasets, Spark Streaming, AWS EMR. Within a few months of completion, I used the knowledge gained from the course to propose in my current company to work primarily on Spark applications. Since then I have continued to work with Spark. I would highly recommend any of Franks courses as he simplifies concepts well and his teaching manner is easy to follow and continue with! “ – Joey Faherty
Be sure to install a JDK for Java 8 for this course, NOT Java 9, 10, or 11 - and install Spark 2.3, not 2.4.0.
A brief introduction to the course, and then we'll get your development environment for Spark and Scala all set up on your desktop. A quick test application will confirm Spark is working on your system! Remember - be sure to install Spark 2.2 and Java 8 for this course.
Let's dive right in! We'll download a data set of 100,000 real movie ratings from real people, and run a Spark script that generates histogram data of the distribution of movie ratings. Some final setup of your Scala development environment and downloading the course materials is also part of this lecture, so be sure not to skip this one.
Scala Crash Course [Optional]
We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it. Part 1 of 2.
We'll go over the basic syntax and structure of Scala code with lots of examples. It's backwards from most other languages, but you quickly get used to it. Part 2 of 2, with some hands-on practice at the end.
You'll see how flow control works in Scala (if/then statements, loops, etc.), and practice what you've learned at the end.
Scala is a functional programming language, and so functions are central to the language. We'll go over the many ways functions can be declared and used in Scala, and practice what you've learned.
We'll cover the common data structures in Scala such as Map and List, and put them into practice.
Spark Basics and Simple Examples
What is Apache Spark anyhow? What does it do, and what is it used for?
The core object of Spark programming is the Resilient Distributed Dataset, or RDD. Once you know how to use RDD's, you know how to use Spark. We'll go over what they are, and what you can do with them.
Now that we understand Scala and have the theory of Spark behind us, we can revisit the rating counter code from lesson 2 and better understand what's actually going on within it.
How does Spark convert your script into a Directed Acyclic Graph and figure out how to distribute it on a cluster? Understanding how this process works under the hood can be important in writing optimal Spark driver scripts.
RDD's that contain a tuple of two values are key/value RDD's, and you can use them much like you might use a NoSQL data store. We'll use key/value RDD's to figure out the average number of friends by age in some fake social network data.
We'll run the average friends by age example on your desktop, and give you some ideas for further extending this script on your own.
We'll cover how to filter data out of an RDD efficiently, and illustrate this with a new example that finds the minimum temperature by location using real weather data.
We'll run our minimum temperature by location example, and modify it to find maximum temperatures as well. Plus, some ideas for extending this script on your own.
flatmap() on an RDD can return variable amounts of new entries into the resulting RDD. We'll use this as part of a hands-on example that finds how often each word is used inside a real book's text.
We extend the previous lecture's example by using regular expressions to better extract words from our book.
Finally, we sort the final results to see what the most common words in this book really are! And some ideas to extend this script on your own.
Your assignment: write a script that finds the total amount spent per customer using some fabricated e-commerce data, using what you've learned so far.
We'll review my solution to the previous lecture's assignment, and challenge you further to sort your results to find the biggest spenders.
Check your results for finding the biggest spenders in our e-commerce data against my own solution.
Advanced Examples of Spark Programs
We'll revisit our movie ratings data set, and start off with a simple example to find the most-rated movie.
Broadcast variables can be used to share small amounts of data to all of the machines on your cluster. We'll use them to share a lookup table of movie ID's to movie names, and use that to get movie names in our final results.
We introduce the Marvel superhero social network data set, and write a script to find the most-connected superhero in it. It's not who you might think!
As a more complex example, we'll apply a breadth-first-search (BFS) algorithm to the Marvel dataset to compute the degrees of separation between any two superheroes. In this lecture, we go over how BFS works.
We'll go over our strategy for implementing BFS within a Spark script that can be distributed, and introduce the use of Accumulators to maintain running totals that are synced across a cluster.
Finally, we'll review the code for finding the degrees of separation using breadth-first-search, run it, and see the results!
Back to our movie ratings data - we'll discover movies that are similar to each other just based on user ratings. We'll cover the algorithm, and how to implement it as a Spark script.
We'll run our movie similarties script and see the results. In doing so, we'll introduce the process of exporting your Spark script as a JAR file that can be run from the command line using the spark-submit script (instead of running from within the Scala IDE.)
Your challenge: make the movie similarity results even better! Here are some ideas for you to try out.
Running Spark on a Cluster
In a production environment, you'll use spark-submit to start your driver scripts from a command line, cron job, or the like. We'll cover the details on what you need to do differently in this case.
Spark / Scala scripts that have external dependencies can be bundled up into self-contained packages using the SBT tool. We'll use SBT to package up our movie similarities script as an exercise.
Amazon Web Services (AWS) offers the Elastic MapReduce service (EMR,) which gives us a way to rent time on a Hadoop cluster of our choosing - with Spark pre-installed on it. We'll use EMR to illustrate running a Spark script on a real cluster, so let's go over what EMR is and how it works first.
Let's compute movie similarities on a real cluster in the cloud, using one million user ratings!
Explicitly partitioning your RDD's can be an important optimization; we'll go over when and how to do this.
Other tips and tricks for taking your script to a real cluster and getting it to run as you expect.
How to troubleshoot Spark jobs on a cluster using the Spark UI and logs, and more on managing dependencies of your script and data.
SparkSQL, DataFrames, and DataSets
Understand SparkSQL in Spark 2, and the new DataFrame and DataSet API's used for querying structured data in an efficient, scalable manner.
We'll revisit our fabricated social network data, but load it into a DataFrame and analyze it with actual SQL queries!
We'll analyze our social network data another way - this time using SQL-like functions on a DataSet, instead of actual SQL query strings.
We'll revisit our "most popular movie" exercise, but this time use a DataSet to make getting the answer more straightforward.
Machine Learning with MLLib
MLLib offers several distributed machine learning algorithms that you can run on a Spark cluster. We'll cover what MLLib can do and how it fits in.
We'll use MLLib's Alternating Least Squares recommender algorithm to produce movie recommendations using our MovieLens ratings data. The results are... unexpected!
A brief overview of what linear regression is and how it works, followed by a hands-on example of finding a regression and applying it to fabricated page speed vs. revenue data.
Spark 2 makes DataFrames the preferred API for MLLib. Let's re-write our linear regression example, this time using Spark's DataFrame MLLib API.
Intro to Spark Streaming
Spark Streaming allows you create Spark driver scripts that run indefinitely, continually processing data as it streams in! We'll cover how it works and what it can do.
As a hands-on example of using Spark Streaming, we'll set up a Twitter developer account, and run a script that will keep track of the most popular hashtags from the past five minutes in real time! Plus some ideas for extending this script on your own.
Spark 2.0 introduced experimental support for Structured Streaming, a new DataFrame-based API for writing continuous applications.
Intro to GraphX
We cover Spark's GraphX library and how it works, followed by a strategy for re-implementing breadth-first-search using GraphX and its Pregel API.
We'll use GraphX and Pregel to recreate our earlier results analyzing the superhero social network data - but with a lot less code!
You Made It! Where to Go from Here.
You made it to the end! Here are some book recommendations if you want to learn more, as well as some career advice on landing a job in "big data".