Spark

Broadcast Joins

• Introduction to Spark Broadcast Joins

Local Setup

• How to Install PySpark on Windows
• Tutorial: Running PySpark inside Docker containers
• How to create a Docker Container with Pyspark ready to work with Elyra

Working with JSON

• Using PySpark to Read and Flatten JSON data with an enforced schema
• JSON in Databricks and PySpark

Performance Optimization

• Performance Optimization in Apache Spark
• Part 3: Cost Efficient Executor Configuration for Apache Spark
• Spark Performance Tuning - Decide Number of executors
• Garbage Collection Tuning Concepts in Spark
• Spark Performance Tuning: Skewness Part 1
• Spark Performance Tuning: Skewness Part 2
• Spark Tuning, Optimization, and Performance Techniques

Basic Concepts

• Apache Spark: RDDs, DataFrames, Datasets
• What are workers, executors, cores in Spark Standalone cluster?
• Understanding the working of Spark Driver and Executor
• Course on Spark Core
• Course on Spark Structured Streaming 3.0

Filter, Projection and Pushdown

• Apache Spark and Predicate Pushdown
• Projection and Filter Pushdown with Apache Spark DataFrames and Datasets
• Important Considerations when filtering in Spark with filter and where

Data Skew

• Data Skew in Apache Spark
• Skewed Data in Spark? Add SALT to Compensate

Read CSV

• PySpark Read CSV file into DataFrame

Spark Metastore

• Metastore in Apache Spark

Pandas on Spark

• Pandas API on Apache Spark - Part 1: Introduction
• Pandas API on Apache Spark - Part 2: Hello World

XGBoost on PySpark

• dmlc - eXtreme Gradient Boosting
• PySpark ML and XGBoost full integration tested on the Kaggle Titanic dataset

XGBoost on Spark

• XGBoost4J-Spark Tutorial

Data Modeling

• Data Modeling in Apache Spark - Part 1 : Date Dimension
• Data Modeling in Apache Spark - Part 2 : Working With Multiple Dates
• Continuous Data Processing with Star Schema Data Warehouse using Apache Spark
• Processing a Slowly Changing Dimension Type 2 Using PySpark in AWS

Jupyter + PySpark Setup on AWS EMR

• Setup Jupyter Notebook with EMR to run spark job in 5 minutes

Accumulators

• Instrumenting PySpark Applications using Spark Accumulators

Spark Issues

Rishabh Pandey(He/Him) on LinkedIn

Different types of Issues in Spark.?

---

1) Out of Memory Exceptions 2) Missing data 3) Data Skewness 4) Spark job repeatedly fails 5) FileAlreadyExistsException in Spark jobs 6) Serialization Issues 7) Inferschema Issues 8) Creating Small Files 9) Too Large Frame error 10) Error when the total size of results is greater than the Spark Driver Max Result Size value. 11) Spark Shell Command failure 12) Join/Shuffle 13) Spark jobs fail because of compilation failures 14) Reading Encoding Files 15) Executor Misconfiguration 16) Broadcasting Large Data 17) Result Exceeds Driver Memory 18) Too Small and Too Large Partitions 19) Optimizing Long Running Jobs 20) Using coalesce() – Creates Uneven Partitions

---

Some challenges occur at the job level; these challenges are shared right across the data team. They include:

1. How many executors should each job use?
2. How much memory should I allocate for each job?
3. How do I find and eliminate data skew?
4. How do I make my pipelines work better?
5. How do I know if a specific job is optimized?

Other challenges come up at the cluster level, or even at the stack level, as you decide what jobs to run on what clusters. These problems tend to be the remit of operations people and data engineers. They include:

1. How do I size my nodes, and match them to the right servers/instance types?
2. How do I see what’s going on across the Spark stack and apps?
3. Is my data partitioned correctly for my SQL queries?
4. When do I take advantage of auto-scaling?
5. How do I get insights into jobs that have problems?