Apache Spark вопросы и ответы для интервью
Вопрос 21. Explain the concept of a Spark task.
A task is the smallest unit of work in Spark, representing the execution of a transformation or action on a partition of data. Tasks are scheduled by the Spark Scheduler on Spark Executors.
Example:
val taskResult = executor.runTask(taskID, taskInfo)
Вопрос 22. How does Spark handle data skewness in transformations like groupByKey?
Data skewness occurs when certain keys have significantly more data than others. Spark handles it by using techniques like data pre-partitioning or using advanced algorithms like map-side aggregation.
Example:
val skewedData = inputRDD.groupByKey(numPartitions)
Вопрос 23. What is the purpose of the Spark MLlib library?
Spark MLlib is Spark's machine learning library, providing scalable implementations of various machine learning algorithms and tools for building and evaluating machine learning models.
Example:
val model = new RandomForestClassifier().fit(trainingData)
Вопрос 24. How does Spark handle data locality optimization?
Spark aims to schedule tasks on nodes that have a copy of the data to minimize data transfer over the network. This is achieved by using data locality-aware task scheduling.
Example:
sparkConf.set("spark.locality.wait", "2s")
Самое полезное по оценкам пользователей:
- What is the purpose of the Spark SQL module?
- Explain the difference between narrow and wide transformations in Spark.