Part 1 of 2: This post covers building the foundational data pipeline for a healthcare claims analytics project called claims-pulse. We walk through realistic data generation, uploading to S3 using bo
Introduction In today's world, data is growing at an exponential rate. While this data holds immense value and has the potential to impact various aspects of our world, extracting value from such large datasets is no simple task. This is where data a...
Introduction At present, data exists like a vast ocean, and extracting valuable insights from this is like looking for a small needle in a big haystack. Enter ELT(Extract, Load, Transform) the modern way to process and analyze data, and data lakes, a...
Introduction Hello There 👋🏻. Hope you all are doing well. In this blog, I will elucidate a recent data engineering project I undertook, leveraging the prowess of AWS services. Throughout this endeavor, I harnessed the capabilities of AWS S3, Python...
Introduction Hello There 👋🏻. Hope you all are doing well. In my previous blog, I discussed various AWS services that are commonly used in the data engineering field to store, manage, and process large volumes of data. As a widely popular cloud comp...
Introduction Hello There 👋🏻. Hope you all are doing well. In my previous blog, I discussed the significance of data engineering for data-driven organizations and highlighted several essential components of data engineering. And, In this blog post, ...
Introduction With the rise of the internet and modern technologies, a huge amount of data is being generated every second. As this data brings great value to business organizations, an ability to understand and work with such a vast amount of data is...