PySpark perguntas e respostas de entrevista
Pergunta 11. Explain the significance of the 'window' function in PySpark.
The 'window' function in PySpark is used for defining windows over data based on partitioning and ordering, often used with aggregation functions.
Example:
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
window_spec = Window.orderBy('column')
result = df.withColumn('row_num', row_number().over(window_spec))
Isto e util?
Adicionar comentario
Ver comentarios
Pergunta 12. What is the purpose of the 'explode' function in PySpark?
The 'explode' function is used to transform a column with arrays or maps into multiple rows, duplicating the values of the other columns.
Example:
from pyspark.sql.functions import explode
exploded_df = df.select('ID', explode('items').alias('item'))
Isto e util?
Adicionar comentario
Ver comentarios
Pergunta 13. Explain the concept of 'checkpointing' in PySpark.
'Checkpointing' is a mechanism in PySpark to truncate the lineage of a RDD or DataFrame by saving it to a reliable distributed file system.
Example:
spark.sparkContext.setCheckpointDir('hdfs://path/to/checkpoint')
df_checkpointed = df.checkpoint()
Isto e util?
Adicionar comentario
Ver comentarios
Pergunta 14. How can you handle skewed data in PySpark?
You can use techniques like salting, bucketing, or using the 'broadcast' hint to handle skewed data in PySpark.
Example:
df.write.option('skew_hint', 'true').parquet('output_path')
Isto e util?
Adicionar comentario
Ver comentarios
Pergunta 15. Explain the purpose of the 'persist' operation in PySpark.
'Persist' is used to persist a DataFrame or RDD in memory or on disk, allowing faster access to the data in subsequent operations.
Example:
df.persist()
Isto e util?
Adicionar comentario
Ver comentarios
Mais uteis segundo os usuarios: