PySpark 면접 질문과 답변

Question: Explain the concept of 'checkpointing' in PySpark.

Answer: 'Checkpointing' is a mechanism in PySpark to truncate the lineage of a RDD or DataFrame by saving it to a reliable distributed file system.

Example:

spark.sparkContext.setCheckpointDir('hdfs://path/to/checkpoint')
df_checkpointed = df.checkpoint()