GroupByKey vs ReduceByKey

GroupByKey and ReduceByKey => Both are transformations
GroupByKey
ReduceByKey

Groups the dataset based on the key
Grouping + Aggregation
Shuffling
More shuffling
Less shuffling
Combiner
No Combiner
Aggregates the data before shuffling
Which one is better?
All the key,value pairs are shuffled around. This is a lot of unnecessary data transfer on the network.
Works better on larger dataset, because spark knows that it can combine the output with a common key on each partition before shuffling the data.
The function that’s passed to reduceByKey will be called again to reduce all the values from each partition to produce the final result.
Only one output for each key at each partition to send over network.
Partitioner method
Will be called on each and every key.
 will be called once per key in the partition
Disk spill
If there is more data to be shuffled on to a single executor machine than can fit in memory.
 No spill
Out or memory exeption can occur
Single key has more key, value pairs than can fit in memory an out of memory exception can occur.

However, it flushes the out the data to disk one at a time.
 No outOfMemory Exception
Performance impact
When disk spill happens, performance will be severely impacted
 No performance impact because of the operation
Preferred functions to be used
combineByKey – can be used when combining the results, but the return type will differ

foldByKey – merges the values for each key using an associative function
 NA

Example word count:
val input = Array(“a”,”a”,”b”,”b”,”c”,”c”,”c”)
val inputRDD = sc.parallelize(input).map(element => (element,1))
val wordcountUsingGroupByKey = inputRDD.groupByKey.map(words => (words._1, words._2.sum)).collect

val wordcountUsingReduceByKey = inputRDD.reduceByKey(_+_).collect


Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor t...
Read more

When to use RDD?

1.        Want to precisely instruct Spark how to do a query i.e., controlling the low-level operations. 2.        Can forgo the code optimization, efficient space utilization and performance benefits available with DF’s and DS’. 3.        If the data is unstructured such as media streams or streams of text. 4.        Not imposing the schema while processing or accessing the attributes by name or column. 5.        Want to manipulate the data with functional programming constructs than domain specific expressions. 6.        Existing dependent third-party package is written using RDD’s.
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the ele...
Read more