PySpark Questions et reponses d'entretien
Question 6. Explain the concept of a SparkSession in PySpark.
SparkSession is the entry point to any PySpark functionality. It is used to create DataFrames, register DataFrames as tables, and execute SQL queries.
Example:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('example').getOrCreate()
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 7. What is the purpose of the 'cache' operation in PySpark?
The 'cache' operation is used to persist a DataFrame or RDD in memory, enhancing the performance of iterative algorithms or repeated operations.
Example:
df.cache()
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 8. How can you handle missing or null values in a PySpark DataFrame?
You can use the 'na' functions like 'drop' or 'fill' to handle missing values in a PySpark DataFrame.
Example:
df.na.drop()
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 9. Explain the purpose of the 'collect' action in PySpark.
The 'collect' action retrieves all elements of a distributed dataset (RDD or DataFrame) and brings them to the driver program.
Example:
data = df.collect()
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Question 10. What is the role of the 'broadcast' variable in PySpark?
A 'broadcast' variable is used to cache a read-only variable in each node of a cluster to enhance the performance of joins.
Example:
from pyspark.sql.functions import broadcast
result = df1.join(broadcast(df2), 'key')
Est-ce utile ?
Ajouter un commentaire
Voir les commentaires
Les plus utiles selon les utilisateurs :