Interview Questions and Answers

Apache Spark is an open-source distributed computing system that provides fast and general-purpose cluster computing for big data processing and analytics.

Example:

SparkContext sc = new SparkContext("local", "SparkExample");

SparkContext is the entry point for Spark functionality and represents the connection to the Spark cluster. It coordinates the execution of operations on the cluster.

Example:

val sc = new SparkContext("local", "SparkExample")

The Spark Driver is the program that runs the main() function and creates the SparkContext. It coordinates the execution of tasks on the Spark Executors and collects results from them.

Example:

object MyApp {
  def main(args: Array[String]): Unit = {
    val sc = new SparkContext("local", "MyApp")
  }
}

A DataFrame is a distributed collection of data organized into named columns, similar to a relational table. An RDD (Resilient Distributed Dataset) is a low-level abstraction representing a distributed collection of objects.

Example:

val df = spark.read.json("/path/to/data.json")