PySpark 面接の質問と回答

Question: What is the difference between a DataFrame and an RDD in PySpark?

Answer: DataFrame is a higher-level abstraction on top of RDD, providing a structured and tabular representation of data. It supports various optimizations and operations similar to SQL.

Example:

df = spark.createDataFrame([(1, 'John'), (2, 'Jane')], ['ID', 'Name'])