PinnedApache Spark Optimization TechniquesBefore discuss various optimization techniques have a quick review how does spark runJun 18, 20202Jun 18, 20202
PinnedFew Spark Concepts :You can consider this as one of the reference notebook where will cover the below topicsJul 1, 20203Jul 1, 20203
Unleashing the Power of LLM with Snowflake CORTEXSnowflake Cortex, now available in private preview, stands as Snowflake’s latest innovation, delivering intelligent data analysis and AI…Mar 26, 2024Mar 26, 2024
Your Personalized Email Reminder Assistant (Windows m/c)— Using PythonIn the hustle and bustle of our daily lives, it’s common to overlook important emails amid the chaos of work and daily tasks. But what if…Feb 21, 20241Feb 21, 20241
Enable ETL Using Snowflake Task and Stream — Super EasyIn this article I will try to explain how task and stream can be easily clubbed together to execute ETL pipeline.Jan 9, 20241Jan 9, 20241
Caching Data using PythonWe all know that caching is basically keeping the important/most popular data in memory rather than in a disk for faster execution. But…Jun 12, 2021Jun 12, 2021
How to Encrypt and Decrypt application password using PythonThere are scenarios when we are using application password in our code. This is completely an unethical practice. Password should be…Jun 1, 20211Jun 1, 20211
Track live position of International Space Station and people in space — using PythonThis is a very basic but powerful python program to track the live position of ISS. My son used to run it in every 30–40 mins interval to…Feb 15, 20211Feb 15, 20211
Read JSON using PySparkThe JSON (JavaScript Object Notation) is a lightweight format to store and exchange data. The input JSON may be in different format —Oct 4, 20202Oct 4, 20202
APACHE SPARK AND DELTA LAKE, A POWERFUL COMBINATIONSpark is no doubt a powerful processing engine and a distributed cluster computing framework for faster processing. It is getting enriched…Sep 23, 20201Sep 23, 20201
Easy to Play with Twitter Data Using Spark Structured StreamingData is all around and twitter is one of the golden source of data for any kind of sentiment analysis. There are lot of ways we can read…Aug 24, 20203Aug 24, 20203
Detect Objects Using Python and OpenCVThis is a basic and simple documentation for those who never did any kind of video processing to detect different kind of objects like Car…Aug 10, 20202Aug 10, 20202
ETL PIPELINE WITH SPARK STRUCTURED STREAMINGFocus here is to analyse few use cases and design ETL pipeline with the help of Spark Structured Streaming and Delta Lake.Jul 8, 2020Jul 8, 2020
What’s New in Spark 3Few new features available in Spark 3.0 which will make it more efficient and faster in executionJun 28, 2020Jun 28, 2020
Process Unstructured Data Using pySparkTo process unstructured data either we can use spark built-in functions or need to create our own functions to transform the unstructured…Jun 17, 20202Jun 17, 20202