Concurrency vs Parallel

Concurrency
Parallelism
There are multiple things in progress
Concurrent/Multiple things are processing at the same time.
E.g: Juggler juggling many balls, juggler is throwing one ball per hand at a time.
E.g: Multiple jugglers juggles multiple balls simultaneously
Multiple execution flow with a potential for a shared resource.
E.g. Two threads comparing for a single IO port
Splitting a problem into multiple chunks.
E.g. Parsing a big file by splitting and processing the splits
Two queues accessing one ATM machine
Two queues accessing two ATM machines.
Multiple tasks can be performed in overlapping the time periods with the shared resources.
Task is divided into multiple sub-tasks, which can be run independently.
E.g. Multi tasking on a single-core machine
Running the tasks at the same time on multi-core machine
Context switching happens
No context switching
Shared resources will exists
No shared resources
Composition of independently executing processes
Simultaneous execution
Interruptability exists
Independentability exists

Comments

  1. UnknownAugust 15, 2018 at 2:53 AM

    You should rename this to SparkScalaNotes...rather SparkScalaMyNotes

    ReplyDelete

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor t...
Read more

When to use RDD?

1.        Want to precisely instruct Spark how to do a query i.e., controlling the low-level operations. 2.        Can forgo the code optimization, efficient space utilization and performance benefits available with DF’s and DS’. 3.        If the data is unstructured such as media streams or streams of text. 4.        Not imposing the schema while processing or accessing the attributes by name or column. 5.        Want to manipulate the data with functional programming constructs than domain specific expressions. 6.        Existing dependent third-party package is written using RDD’s.
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the ele...
Read more