Apache Spark вопросы и ответы для интервью
Вопрос 11. What is the purpose of the Spark SQL module?
Spark SQL is a Spark module for structured data processing. It provides a programming interface for data manipulation using SQL, as well as a DataFrame API for processing structured and semi-structured data.
Example:
val df = spark.sql("SELECT * FROM table")
Вопрос 12. How can you persist an RDD in Apache Spark? Provide an example.
You can persist an RDD using the persist() or cache() method. It allows you to store the RDD's data in memory or on disk for faster access.
Example:
val cachedRDD = inputRDD.persist(StorageLevel.MEMORY_ONLY)
Вопрос 13. Explain the difference between narrow and wide transformations in Spark.
Narrow transformations involve operations where each input partition contributes to only one output partition. Wide transformations involve operations where multiple input partitions contribute to multiple output partitions.
Example:
Narrow: map, filter
Wide: groupByKey, reduceByKey
Вопрос 14. What is the purpose of the Spark Streaming module?
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. It allows processing real-time data using batch processing capabilities of Spark.
Example:
val streamingContext = new StreamingContext(sparkContext, Seconds(1))
Вопрос 15. What is the significance of the Spark Shuffle operation?
The Spark Shuffle operation redistributes data across partitions during certain transformations, such as groupByKey or reduceByKey. It is a costly operation that involves data exchange and can impact performance.
Example:
val groupedRDD = inputRDD.groupByKey()
Самое полезное по оценкам пользователей:
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.