PySpark اسئلة واجوبة المقابلات
سؤال 11. Explain the significance of the 'window' function in PySpark.
The 'window' function in PySpark is used for defining windows over data based on partitioning and ordering, often used with aggregation functions.
Example:
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number
window_spec = Window.orderBy('column')
result = df.withColumn('row_num', row_number().over(window_spec))
هل هذا مفيد؟
اضف تعليقا
عرض التعليقات
سؤال 12. What is the purpose of the 'explode' function in PySpark?
The 'explode' function is used to transform a column with arrays or maps into multiple rows, duplicating the values of the other columns.
Example:
from pyspark.sql.functions import explode
exploded_df = df.select('ID', explode('items').alias('item'))
هل هذا مفيد؟
اضف تعليقا
عرض التعليقات
سؤال 13. Explain the concept of 'checkpointing' in PySpark.
'Checkpointing' is a mechanism in PySpark to truncate the lineage of a RDD or DataFrame by saving it to a reliable distributed file system.
Example:
spark.sparkContext.setCheckpointDir('hdfs://path/to/checkpoint')
df_checkpointed = df.checkpoint()
هل هذا مفيد؟
اضف تعليقا
عرض التعليقات
سؤال 14. How can you handle skewed data in PySpark?
You can use techniques like salting, bucketing, or using the 'broadcast' hint to handle skewed data in PySpark.
Example:
df.write.option('skew_hint', 'true').parquet('output_path')
هل هذا مفيد؟
اضف تعليقا
عرض التعليقات
سؤال 15. Explain the purpose of the 'persist' operation in PySpark.
'Persist' is used to persist a DataFrame or RDD in memory or on disk, allowing faster access to the data in subsequent operations.
Example:
df.persist()
هل هذا مفيد؟
اضف تعليقا
عرض التعليقات
الاكثر فائدة حسب تقييم المستخدمين: