Apache Spark 면접 질문과 답변
Ques 11. What is the purpose of the Spark SQL module?
Spark SQL is a Spark module for structured data processing. It provides a programming interface for data manipulation using SQL, as well as a DataFrame API for processing structured and semi-structured data.
Example:
val df = spark.sql("SELECT * FROM table")
Ques 12. How can you persist an RDD in Apache Spark? Provide an example.
You can persist an RDD using the persist() or cache() method. It allows you to store the RDD's data in memory or on disk for faster access.
Example:
val cachedRDD = inputRDD.persist(StorageLevel.MEMORY_ONLY)
Ques 13. Explain the difference between narrow and wide transformations in Spark.
Narrow transformations involve operations where each input partition contributes to only one output partition. Wide transformations involve operations where multiple input partitions contribute to multiple output partitions.
Example:
Narrow: map, filter
Wide: groupByKey, reduceByKey
Ques 14. What is the purpose of the Spark Streaming module?
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. It allows processing real-time data using batch processing capabilities of Spark.
Example:
val streamingContext = new StreamingContext(sparkContext, Seconds(1))
Ques 15. What is the significance of the Spark Shuffle operation?
The Spark Shuffle operation redistributes data across partitions during certain transformations, such as groupByKey or reduceByKey. It is a costly operation that involves data exchange and can impact performance.
Example:
val groupedRDD = inputRDD.groupByKey()
Most helpful rated by users:
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.