PySpark Questions et reponses d'entretien
Question 21. What is the purpose of the 'groupBy' operation in PySpark?
'groupBy' is used to group the data based on one or more columns. It is often followed by aggregation functions to perform operations on each group.
Example:
grouped_data = df.groupBy('Category').agg({'Price': 'mean'})
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 22. Explain the difference between 'cache' and 'persist' operations in PySpark.
'Cache' is a shorthand for 'persist(memory_only=True)', while 'persist' allows more flexibility by specifying storage levels (memory-only, disk-only, etc.).
Example:
df.cache()
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 23. How can you create a temporary view from a PySpark DataFrame?
You can use the 'createOrReplaceTempView' method to create a temporary view from a PySpark DataFrame.
Example:
df.createOrReplaceTempView('temp_view')
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 24. What is the purpose of the 'orderBy' operation in PySpark?
'OrderBy' is used to sort the rows of a DataFrame based on one or more columns.
Example:
result = df.orderBy('column')
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 25. Explain the role of the 'broadcast' variable in PySpark.
A 'broadcast' variable is used to cache a read-only variable in each node of a cluster to enhance the performance of joins.
Example:
from pyspark.sql.functions import broadcast
result = df1.join(broadcast(df2), 'key')
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Les plus utiles selon les utilisateurs :