PySpark Questions et reponses d'entretien
Question 11. Explain the significance of the 'window' function in PySpark.
The 'window' function in PySpark is used for defining windows over data based on partitioning and ordering, often used with aggregation functions.
Example:
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
window_spec = Window.orderBy('column')
result = df.withColumn('row_num', row_number().over(window_spec))
Question 12. What is the purpose of the 'explode' function in PySpark?
The 'explode' function is used to transform a column with arrays or maps into multiple rows, duplicating the values of the other columns.
Example:
from pyspark.sql.functions import explode
exploded_df = df.select('ID', explode('items').alias('item'))
Question 13. Explain the concept of 'checkpointing' in PySpark.
'Checkpointing' is a mechanism in PySpark to truncate the lineage of a RDD or DataFrame by saving it to a reliable distributed file system.
Example:
spark.sparkContext.setCheckpointDir('hdfs://path/to/checkpoint')
df_checkpointed = df.checkpoint()
Question 14. How can you handle skewed data in PySpark?
You can use techniques like salting, bucketing, or using the 'broadcast' hint to handle skewed data in PySpark.
Example:
df.write.option('skew_hint', 'true').parquet('output_path')
Question 15. Explain the purpose of the 'persist' operation in PySpark.
'Persist' is used to persist a DataFrame or RDD in memory or on disk, allowing faster access to the data in subsequent operations.
Example:
df.persist()
Les plus utiles selon les utilisateurs :