Python vs Scala in Spark


PySpark
Scala
Nativity w.r.to spark
API's provided, not for all features
Spark is developed in Scala, things are more natural - de-facto interface
Learning and use
Comparatively easy to learn and use
Complex to learn and easy to use
Complexity
Less compared to Scala
More complex when compared to PySpark - lot of internal manipulations and conversions will happen
Conciseness
More of imperative styles
More concise - fewer lines of code allows faster development, testing and deployment
Performance
Less compared to Scala - internal conversions are required
10 times faster than PySpark
Effective for?
smaller ad-hoc experiments
for production and engineering applications
Scalability
Not much
Scalable
Refactoring
not good, more bugs gets introduced
Hassle-free and much easier to refactor
Data Scientist tools: ML/NLP/Deep learning
Lot of libraries based on python
Less libraries - Have MLlib in Scala
Streaming support
Not much matured
Best pick for streaming
Visualization tools
lot of them are already existing
very less compared to Python
versioning/compatibility dependencies?
Python 2 or python 3
Latest Scala version
Community
Good community when compared to scala
Community is evolving
Internals
Python is supported by serializing/deserializing between python worker process and the main spark jvm process.
Native serialization/deserialization and well suited for distributed environments
Execution mechanism
Interpreter
Compiler
Type safety
Less compared to Scala - Dynamic typed
More compared to PySpark - Static typing
Concurrency
Does not support true multi threading
Powerful concurrency with Akka Actor model
New/latest features availability?
Depends on third party, may not be available immediately
Immediately available
Overall opinion
easy to learn, use and more libraries  for visualization and much matured community
Concise, Scalable, native to Spark, easy to refactor, fast and moderately easy to use.



Data Scientist preferable approach - Hybrid approach i.e. both Python and Scala.
Bottomline: Scala is faster and moderately easy to use, while Python is slower but very easy to use.


The performance issues can be addressed in python by:

1. Using vectorized UDFs in python https://databricks.com/session/vectorized-udf-scalable-analysis-with-python-and-pyspark) 
2. Convert UDFs to SQL (wherever is possible) 
3. In the worst case we can implement the specific step(in data factory) in Scala notebook.


Comments

Post a Comment

Popular posts from this blog

Out Of Memory in Spark(OOM) - Typical causes and resolutions

   Spark OOM – Can occur at Driver/Executor/node manager. The error that we see is: java.lang.OutOfMermoryError. This will happen when the required memory exceeds the available memory for a particular operation. 1.        Driver OOM - possible causes: a.        Collect For e.g. in the program, we have    val data = inDF.collect()                 Collect operation, collects the data from all executors and send it to the driver. The driver will try to merge into the single object, then there is a possibility that it may be too big to fit into the driver memory. This problem can be solved in two ways: Ø   Setting proper limit using spark.driver.maxResultSize Ø   Repartitioning before saving the result to the output file. §   data.repartition(1).write… §   Repartition uses dedicated executor to collect. b.        Broadcast Join ¨        The table will be materialized at the driver and then broadcasted to the exectuors. c.        Low driver memory configurati
Read more

map vs flatMap in Spark

Image
map flatMap Returns single element based on the function/custom business logic/algorithm Returns zero or more elements based on the function/custom business logic/algorithm Single element as return type Iterator with our return values, but we don’t return an iterator of RDD’s, return RDD that consists of the elements from all the iterators. e.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.map(_.to(3)) List[scala.collection.immutable.Range.Inclusive] = List(Range(1, 2, 3), Range(2, 3), Range(3), Range(), Range()) E.g. val list = 1 to 5 toList list: List[Int] = List(1, 2, 3, 4, 5) list.flatMap(_.to(3)) List[Int] = List(1, 2, 3, 2, 3, 3)   Returns  RDD of lists ; refer to the example Returns  RDD of elements map operation only Equivalent to map followed by flatten Returns the elements of the RDD Returns the elements of the I
Read more

Spark Persistence(Caching)

Image
Cached RDDs should have modest size so that they can fit in the MEMORY entirely. To identify the size, it is challenging and unclear. Caching strategy that caches the blocks in memory and disk is preferred. The reason is the cached blocks that are evicted will be written to disk. Reading from disk is relatively fast as compared to re-evaluating the RDD. Internals: Internally caching is performed at the block level. That means each RDD consists of multiple blocks, each block is being cached independently of other blocks. Caching is performed on the node that is generated the particular RDD block. Each executor in spark has an associated Block Manager that is used to cache RDD blocks. The memory allocation for the Block Manager is given by the storage memory fraction, which gives the fraction from the memory pool that is allocated for the spark engine. The block manager manages
Read more