Apache Spark 面接の質問と回答
質問 16. How does Spark handle data serialization and why is it important?
Spark uses Java's Object Serialization to serialize data between the Spark Driver and Executors. Efficient serialization is crucial for optimizing data transfer and reducing network overhead.
Example:
sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
質問 17. What is the purpose of the accumulator in Spark?
An accumulator is a variable that can be added to and is used in Spark to implement counters and sums in a parallel and fault-tolerant manner across distributed tasks.
Example:
val accumulator = sc.longAccumulator("MyAccumulator")
質問 18. Explain the concept of Spark DAG (Directed Acyclic Graph).
The Spark DAG represents the logical execution plan of transformations and actions in a Spark application. It is a graph of stages, where each stage contains a sequence of tasks that can be executed in parallel.
Example:
val dag = inputRDD.map(x => x * 2).toDebugString
質問 19. What is the difference between a DataFrame and an RDD in Spark?
A DataFrame is a distributed collection of data organized into named columns, similar to a relational table. An RDD (Resilient Distributed Dataset) is a low-level abstraction representing a distributed collection of objects.
Example:
val df = spark.read.json("/path/to/data.json")
質問 20. What are the advantages of using Spark over Hadoop MapReduce?
Spark offers in-memory processing, higher-level abstractions like DataFrames, and iterative processing, making it faster and more versatile than Hadoop MapReduce.
Example:
SparkContext sc = new SparkContext("local", "SparkExample")
ユーザー評価で最も役立つ内容:
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.