Apache Spark Questions et reponses d'entretien
Question 16. How does Spark handle data serialization and why is it important?
Spark uses Java's Object Serialization to serialize data between the Spark Driver and Executors. Efficient serialization is crucial for optimizing data transfer and reducing network overhead.
Example:
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
Question 17. What is the purpose of the accumulator in Spark?
An accumulator is a variable that can be added to and is used in Spark to implement counters and sums in a parallel and fault-tolerant manner across distributed tasks.
Example:
val accumulator = sc.longAccumulator("MyAccumulator")
Question 18. Explain the concept of Spark DAG (Directed Acyclic Graph).
The Spark DAG represents the logical execution plan of transformations and actions in a Spark application. It is a graph of stages, where each stage contains a sequence of tasks that can be executed in parallel.
Example:
val dag = inputRDD.map(x => x * 2).toDebugString
Question 19. What is the difference between a DataFrame and an RDD in Spark?
A DataFrame is a distributed collection of data organized into named columns, similar to a relational table. An RDD (Resilient Distributed Dataset) is a low-level abstraction representing a distributed collection of objects.
Example:
val df = spark.read.json("/path/to/data.json")
Question 20. What are the advantages of using Spark over Hadoop MapReduce?
Spark offers in-memory processing, higher-level abstractions like DataFrames, and iterative processing, making it faster and more versatile than Hadoop MapReduce.
Example:
SparkContext sc = new SparkContext("local", "SparkExample")
Les plus utiles selon les utilisateurs :
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.