pyspark-tutorial

A short tutorial notebook on PySpark

Prerequisites

• Install Java 7 or newer on your OS.

• Download a pre-built version of Spark from here and unpack it.

• Set the following environment variables, for example in your ~/.bashrc:

  export SPARK_HOME=/PATH/TO/SPARK

  export PYTHONPATH=/PATH/TO/SPARK/python

• If you want Spark to work with a specific Python version/virtualenv, also set this one:

  export PYSPARK_PYTHON=/PATH/TO/PYTHON/INSIDE/VIRTUALENV

• Install Py4j dependency:

  pip install py4j