DataFrame vs Dataset


DataFrame
DataSet (RDD + DF)
Definition
Distributed collection of rows(data) organized into named columns.
Extension of DF API

Distributed collection of strongly typed Domain Specific Objects.
Regeneration of  Domain specific object
Can’t regenerate original domain specific object
Can regenerate original domain specific object from JVM object
Static typing/Compile time safety
DF API trying to access a column which doesn’t exists in the table, doesn’t give compile time error, detects attribute error at runtime
Gives an error at compile time
Individual attribute of an object
To access individual attribute, we need to serialize the entire object
No need to serialize the entire object to access the individual element of an object
Programming language support
Python, R, Scala and Java
Scala and Java.
Optimization
Catalyst optimizer
Catalyst optimizer + Tungsten Encoder
Encoder maps the Domain specific type T to spark’s internal type system.
High level domain specific language methods

E.g. sum, avg, join, select, groupBy etc
Caching

Uses more optimal layout
Aggregation

Faster to perform aggregation on plenty of datasets
Memory usage
Uses off-heap memory for serialization, (reduces the overhead when compared to RDD)
Uses encoder for converting between the JVM objects and tabular form
Operation on serialized data


Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor to collect. b.        Broadcast Join ¨        The table will be materialized at the driver and then broadcasted to the exectuors. c.        Low driver memory configurati
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the elements of the I
Read more

Spark Persistence(Caching)

Image
Cached RDDs should have modest size so that they can fit in the MEMORY entirely. To identify the size, it is challenging and unclear. Caching strategy that caches the blocks in memory and disk is preferred. The reason is the cached blocks that are evicted will be written to disk. Reading from disk is relatively fast as compared to re-evaluating the RDD. Internals: Internally caching is performed at the block level. That means each RDD consists of multiple blocks, each block is being cached independently of other blocks. Caching is performed on the node that is generated the particular RDD block. Each executor in spark has an associated Block Manager that is used to cache RDD blocks. The memory allocation for the Block Manager is given by the storage memory fraction, which gives the fraction from the memory pool that is allocated for the spark engine. The block manager manages
Read more