You can consider this as one of the reference notebook where will cover the below topics Spark SQL Execution Plan Lineage Vs. Dag Narrow Vs. Wide Dependency How Does Spark Read Large File 1. SPARK SQL EXECUTION PLAN: Whenever we create a dataframe or Spark SQL or a HIVE query, spark will i. Generate an…

We all know that caching is basically keeping the important/most popular data in memory rather than in a disk for faster execution. But someone has to be sensible enough to decide what needs to be cached and how? …

There are scenarios when we are using application password in our code. This is completely an unethical practice. Password should be hidden. Here I’ve used python (cryptography pkg ) to encrypt a password and later decrypt it for further use. > pip install cryptography Mostly, we have seen application passwords…

This is a very basic but powerful python program to track the live position of ISS. My son used to run it in every 30–40 mins interval to see it’s position :-) Prerequisites: python , IDE, install few packages — pandas, plotly, google. Reference: We are going to use the…

The JSON (JavaScript Object Notation) is a lightweight format to store and exchange data. The input JSON may be in different format — simple, multi line with complex format, HTTP link, a CSV which contains JSON column. Below will cover all the above scenarios: 1. Simple JSON: Input JSON file (Simple.json) -

Spark is no doubt a powerful processing engine and a distributed cluster computing framework for faster processing. It is getting enriched day by day with additional cool features. Unfortunately there are few areas where spark is struggling. But if we combine Spark with Delta Lake, it can overcomes all those…

Data is all around and twitter is one of the golden source of data for any kind of sentiment analysis. There are lot of ways we can read twitter live data and process them. In this article I will demonstrate how easily we can create a connection with twitter account…

This is a basic and simple documentation for those who never did any kind of video processing to detect different kind of objects like Car, Human, Bus etc. If you have free time and interested to play around then please follow this documentation. …

Focus here is to analyse few use cases and design ETL pipeline with the help of Spark Structured Streaming and Delta Lake. Why delta lake? Because, this open source storage layer brings ACID transactions to big data workloads. …