Scala for Spark – The Most Important Parts
Understanding how to leverage Apache Spark from Scala is becoming both essential for data engineers, architects, and data scientists. But, what if you are new to Scala?
If you are new to Scala, this course will teach you to become productive and confident in Scala. The course focus is Scala relevant for Apache Spark. It will demonstrate through a variety of hands-on examples as well as prepare and configure your own development environment in the free IntelliJ Community Edition.
In the book “High Performance Spark” from O’Reilly and Associates, the authors state “serious performant Spark development is most easily achieved in Scala” and more reasons include “To Be a Spark Expert You Have to Learn a Little Scala Anyway”.
Other noteworthy quotes from the book include “The Spark Scala API is Easier to Use than the Java API” and “Scala is More Performant Than Python”
In this “Scala for Spark” course, you will be taught the core Scala principles needed to build your confidence and become both productive and valuable.
Scala for Spark training
This course will guide you through a wide range of hands-on Scala source code and how it relates to Apache Spark concepts. This is the only course designed specifically what you need to know in Scala in order to be productive with Apache Spark.
This course will teach you how to configure SBT and SBT plugins to make packaging and deploying Scala applications to Spark easier and more efficient.
The Scala for Spark course also includes hands-on examples of configuring the free Community Edition of IntelliJ which is one of the most popular IDEs for Scala development.
All of the source code is available for download, so you can run and modify for yourself. This can save you a tremendous amount of time over reading a book or finding tutorials on the Internet.
The course is organized into the following sections:
Introduction, Environment Setup and Basic Concepts of the Scala
Jumpstart Scala SBT for Apache Spark project
Getting Started with Scala in IntelliJ
Scala key language elements
Scala classes, inheritance, and traits
Scala for Spark
Scala With IntelliJ
SBT with Scala
Sample Spark Examples
Conclusion and Free Resources
Save time and learn from an experienced Scala and Spark data engineer.
Course Overview and Objectives
Let's cover the big picture first. In this section, we'll cover how the course will be conducted and the key deliverables. This should give you an expectation of how you will benefit at the end of the Scala for Spark course.
Setting Up Your Development Environment
One of the key resources of this course is a sample Scala SBT for Spark project. This project covers many of the areas I've learned the hard way over the years. In this section, we'll cover how to download it so you can use it in this course and beyond.
Let's continue setup with Java and IntelliJ. Both of these are free to download and use.
Confirm your IntelliJ setup and begin Scala coding.
Hands-on Scala expressions, vals and vars on how it relates to mutability, functions, anonymous functions, troubleshooting, partially applied functions, curried functions, types
Scala Classes, Traits and Inheritance
Advanced Scala for Spark
Scala data structures including arrays, lists, sets, etc. and one of the most powerful features of Scala, pattern matching.
How does Scala treat null values? Is it different or similar to other languages such as Java? In this lecture, we'll get hands-on with Scala Option, Some and None. Also, we'll continue with Scala pattern matching examples.
The line between Apache Spark and Scala start to become blurred in this section. Scala Functional combinators are covered in this lecture and how they are also found in Apache Spark.
Let's test your knowledge
Scala with IntelliJ - Tricks and Tips
Learn multiple ways to debug Scala and Spark programs from IntelliJ, key shortcuts and documentation resource references
SBT - Building, Deploying and Managing Scala for Spark Projects
What is SBT? Why and how do we use it in Scala Spark projects? What are SBT's key features and deliverables.
Apache Spark with Scala
Let's cover two examples on how to run Scala based Spark programs. We can run Scala Spark in IntelliJ in "standalone" or "local" mode. Or, we can package our code up into a Jar file and deploy to a Spark cluster. In this lecture, we'll review how to do both.
One way to improve your Scala with Spark is to review and even customize Scala with Spark examples. In this lecture, I'll show a great resource for Scala with Spark examples and how to customize and run in your environment.
Scala for Spark Course Conclusion
Suggested resources to continue your Scala for Spark journeys