Interview Questions and Answers

Example:

val mappedRDD = inputRDD.map(x => x * 2)
val result = mappedRDD.reduce((x, y) => x + y)

Example:

val filteredRDD = inputRDD.filter(x => x > 0)
filteredRDD.toDebugString

Example:

val inputRDD = sc.parallelize(Seq(1, 2, 3, 4, 5), 2)

Example:

spark-submit --master yarn --deploy-mode client --num-executors 3 mySparkApp.jar

Example:

val broadcastVar = sc.broadcast(Array(1, 2, 3))

Example:

val df = spark.sql("SELECT * FROM table")

Example:

val cachedRDD = inputRDD.persist(StorageLevel.MEMORY_ONLY)

Example:

Narrow: map, filter
Wide: groupByKey, reduceByKey

Example:

val streamingContext = new StreamingContext(sparkContext, Seconds(1))

Example:

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

Example:

val accumulator = sc.longAccumulator("MyAccumulator")

Example:

val dag = inputRDD.map(x => x * 2).toDebugString

Example:

val taskResult = executor.runTask(taskID, taskInfo)

Example:

val model = new RandomForestClassifier().fit(trainingData)