Advantages of Immutability in distributed systems and in programming


Specific to Distributed systems:
1.      Performance – simple, easy to share the RDD with multiple processing elements
2.      Fault tolerance – re-creatable on failure
3.      Caching and in-memory processing - References can be cached as they are not going to change. 
4.      Multiple threads can access the partition
5.      Sharing - Safe to share across processes
6.      Processing is easy - makes it easy to parallelize, as there are no conflicts
7.      Replication - internal state will be in consistent even if you have an exception.
8.      Rules out the potential problems due to updates from multiple threads.
9.      Can easily live in memory as on disk, this makes it reasonable to easily move operations that hit disk to instead use data in memory, and again, adding memory is easy than I/O bandwidth.

RDD significant wins, at cost of having to copy the data rather than mutate it in place.

General list of reasons to favor immutability in programming:
  • immutable objects are simpler to construct, test, and use
  • truly immutable objects are always thread-safe
  • they help to avoid temporal coupling
  • their usage is side-effect free (no defensive copies)
  • identity mutability problem is avoided
  • they always have failure atomicity
  • they are much easier to cache
  • they prevent NULL references, which are bad
Note: when you care a lot about performance, e.g. programming game, it may be necessary to use mutable object.

Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor to collect. b.        Broadcast Join ¨        The table will be materialized at the driver and then broadcasted to the exectuors. c.        Low driver memory configurati
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the elements of the I
Read more

Spark Persistence(Caching)

Image
Cached RDDs should have modest size so that they can fit in the MEMORY entirely. To identify the size, it is challenging and unclear. Caching strategy that caches the blocks in memory and disk is preferred. The reason is the cached blocks that are evicted will be written to disk. Reading from disk is relatively fast as compared to re-evaluating the RDD. Internals: Internally caching is performed at the block level. That means each RDD consists of multiple blocks, each block is being cached independently of other blocks. Caching is performed on the node that is generated the particular RDD block. Each executor in spark has an associated Block Manager that is used to cache RDD blocks. The memory allocation for the Block Manager is given by the storage memory fraction, which gives the fraction from the memory pool that is allocated for the spark engine. The block manager manages
Read more