PySpark Pertanyaan dan Jawaban Wawancara
Ques 11. Explain the significance of the 'window' function in PySpark.
The 'window' function in PySpark is used for defining windows over data based on partitioning and ordering, often used with aggregation functions.
Example:
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
window_spec = Window.orderBy('column')
result = df.withColumn('row_num', row_number().over(window_spec))
Apakah ini membantu?
Add Comment
View Comments
Ques 12. What is the purpose of the 'explode' function in PySpark?
The 'explode' function is used to transform a column with arrays or maps into multiple rows, duplicating the values of the other columns.
Example:
from pyspark.sql.functions import explode
exploded_df = df.select('ID', explode('items').alias('item'))
Apakah ini membantu?
Add Comment
View Comments
Ques 13. Explain the concept of 'checkpointing' in PySpark.
'Checkpointing' is a mechanism in PySpark to truncate the lineage of a RDD or DataFrame by saving it to a reliable distributed file system.
Example:
spark.sparkContext.setCheckpointDir('hdfs://path/to/checkpoint')
df_checkpointed = df.checkpoint()
Apakah ini membantu?
Add Comment
View Comments
Ques 14. How can you handle skewed data in PySpark?
You can use techniques like salting, bucketing, or using the 'broadcast' hint to handle skewed data in PySpark.
Example:
df.write.option('skew_hint', 'true').parquet('output_path')
Apakah ini membantu?
Add Comment
View Comments
Ques 15. Explain the purpose of the 'persist' operation in PySpark.
'Persist' is used to persist a DataFrame or RDD in memory or on disk, allowing faster access to the data in subsequent operations.
Example:
df.persist()
Apakah ini membantu?
Add Comment
View Comments
Most helpful rated by users: