DAG in Spark and Usage

DAG - Directed Acyclic Graph

DAG is a set of vertices and edges
Vertices represent RDD
Edges represent Operations that needs to be applied on RDD
DAG is a finite graph with no directed cycles, which means it have finite vertices and edges.

Each Spark Application is converted into DAG
DAG Shows the complete task i.e. Transformation(s) and Action
DAG shows different stages of Spark Job.
Logical DAG of operations is created implicitly by the Spark Program. or in other words, user code defines DAG of RDDs
For each Action, spark creates a DAG and submits to the DAG Scheduler.
DAG scheduler divides the operators into stages of tasks.(This is the final result of DAG scheduler)
DAG Scheduler pipelines the operators together.
e.g. Many map operations can be scheduled on a single stage.


DAG helps in achieving Fault tolerance.
Lost RDD can be recovered.
MR has two levels: map and reduce, DAG can have multiple levels, because of this executing SQL queries is more flexible.

DAG is an alternate to MapReduce in Hadoop, it is a programming style in distributed systems.

DAG is faster because it doesn't write the intermediate results to the disk.

Action force the translation of RDD into an Execution plan


DAG vs Lineage


Q1) Then what is Lineage Graph  

When a new RDD is derived from existing RDD using transformation, Spark keeps track of all the dependencies between these RDDs called the lineage graph. In case of data loss, this lineage graph is used to rebuild the data.

Q2) Are there any difference between Lineage Graph and Directed Acyclic Graph ? 

Yes, DAG and Lineage graphs are different. 

DAG shows the different stages of a spark job.
Q3) Are they both same, can they both be used interchangeably ? 

No they cannot be used interchangeably, because workings are different. Lineage graph deals with RDDs so it is applicable up-till transformations ,  Whereas, DAG shows the complete task, ie; trasnformation + Action   


From <http://support.edureka.co/support/solutions/articles/4000083868--directed-acyclic-graph-dag-vs-lineage-graph-in-spark>





Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor to collect. b.        Broadcast Join ¨        The table will be materialized at the driver and then broadcasted to the exectuors. c.        Low driver memory configurati
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the elements of the I
Read more

Spark Persistence(Caching)

Image
Cached RDDs should have modest size so that they can fit in the MEMORY entirely. To identify the size, it is challenging and unclear. Caching strategy that caches the blocks in memory and disk is preferred. The reason is the cached blocks that are evicted will be written to disk. Reading from disk is relatively fast as compared to re-evaluating the RDD. Internals: Internally caching is performed at the block level. That means each RDD consists of multiple blocks, each block is being cached independently of other blocks. Caching is performed on the node that is generated the particular RDD block. Each executor in spark has an associated Block Manager that is used to cache RDD blocks. The memory allocation for the Block Manager is given by the storage memory fraction, which gives the fraction from the memory pool that is allocated for the spark engine. The block manager manages
Read more