DataFrame vs Dataset


DataFrame
DataSet (RDD + DF)
Definition
Distributed collection of rows(data) organized into named columns.
Extension of DF API

Distributed collection of strongly typed Domain Specific Objects.
Regeneration of  Domain specific object
Can’t regenerate original domain specific object
Can regenerate original domain specific object from JVM object
Static typing/Compile time safety
DF API trying to access a column which doesn’t exists in the table, doesn’t give compile time error, detects attribute error at runtime
Gives an error at compile time
Individual attribute of an object
To access individual attribute, we need to serialize the entire object
No need to serialize the entire object to access the individual element of an object
Programming language support
Python, R, Scala and Java
Scala and Java.
Optimization
Catalyst optimizer
Catalyst optimizer + Tungsten Encoder
Encoder maps the Domain specific type T to spark’s internal type system.
High level domain specific language methods

E.g. sum, avg, join, select, groupBy etc
Caching

Uses more optimal layout
Aggregation

Faster to perform aggregation on plenty of datasets
Memory usage
Uses off-heap memory for serialization, (reduces the overhead when compared to RDD)
Uses encoder for converting between the JVM objects and tabular form
Operation on serialized data


Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor t...
Read more

DAG in Spark and Usage

DAG - Directed Acyclic Graph DAG is a set of vertices and edges Vertices represent RDD Edges represent Operations that needs to be applied on RDD DAG is a finite graph with no directed cycles, which means it have finite vertices and edges. Each Spark Application is converted into DAG DAG Shows the complete task i.e. Transformation(s) and Action DAG shows different stages of Spark Job. Logical DAG of operations is created implicitly by the Spark Program. or in other words, user code defines DAG of RDDs For each Action, spark creates a DAG and submits to the DAG Scheduler. DAG scheduler divides the operators into stages of tasks.(This is the final result of DAG scheduler) DAG Scheduler pipelines the operators together. e.g. Many map operations can be scheduled on a single stage. DAG helps in achieving Fault tolerance. Lost RDD can be recovered. MR has two levels: map and reduce, DAG can have multiple levels, because of this executing SQL queries is more flexible. DA...
Read more

When to use RDD?

1.        Want to precisely instruct Spark how to do a query i.e., controlling the low-level operations. 2.        Can forgo the code optimization, efficient space utilization and performance benefits available with DF’s and DS’. 3.        If the data is unstructured such as media streams or streams of text. 4.        Not imposing the schema while processing or accessing the attributes by name or column. 5.        Want to manipulate the data with functional programming constructs than domain specific expressions. 6.        Existing dependent third-party package is written using RDD’s.
Read more