Apache Spark вопросы и ответы для интервью
Вопрос 16. How does Spark handle data serialization and why is it important?
Spark uses Java's Object Serialization to serialize data between the Spark Driver and Executors. Efficient serialization is crucial for optimizing data transfer and reducing network overhead.
Example:
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
Вопрос 17. What is the purpose of the accumulator in Spark?
An accumulator is a variable that can be added to and is used in Spark to implement counters and sums in a parallel and fault-tolerant manner across distributed tasks.
Example:
val accumulator = sc.longAccumulator("MyAccumulator")
Вопрос 18. Explain the concept of Spark DAG (Directed Acyclic Graph).
The Spark DAG represents the logical execution plan of transformations and actions in a Spark application. It is a graph of stages, where each stage contains a sequence of tasks that can be executed in parallel.
Example:
val dag = inputRDD.map(x => x * 2).toDebugString
Вопрос 19. What is the difference between a DataFrame and an RDD in Spark?
A DataFrame is a distributed collection of data organized into named columns, similar to a relational table. An RDD (Resilient Distributed Dataset) is a low-level abstraction representing a distributed collection of objects.
Example:
val df = spark.read.json("/path/to/data.json")
Вопрос 20. What are the advantages of using Spark over Hadoop MapReduce?
Spark offers in-memory processing, higher-level abstractions like DataFrames, and iterative processing, making it faster and more versatile than Hadoop MapReduce.
Example:
SparkContext sc = new SparkContext("local", "SparkExample")
Самое полезное по оценкам пользователей:
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.