Nabarun Chakraborti
1 min readJul 1, 2020

--

The basic analysis and behaviors will remain same in terms of applying your logic to optimize the performance. But Spark 3 version is highly optimized and way faster compare to Spark 2.x in few areas. 46% of the deployment were on Spark SQL area to make it more efficient and fast. Few concepts like Dynamic Partition Pruning, Adaptive Query Execution, Optimizer Hints have been implemented which make the overall execution faster. I've prepared a small overview covering these features which you can refer @ https://medium.com/@ch.nabarun/whats-new-in-spark-3-8250a65b3144

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

--

--

Nabarun Chakraborti
Nabarun Chakraborti

Written by Nabarun Chakraborti

Big Data Solution Architect and pySpark Developer

No responses yet

Write a response