DataEng

TL;DR This post will show some tips on how to work efficiently as a Data Engineer 🚀, either navigating throught documentation or using a local LLM to ease your development experience (having a Mac chip will be mandatory for this one). 👨‍💻 Let’s go ! Intro Nowadays, working in limited internet connection can occur and there is a huge gap compared to our workstation setup 🦾 In those situations, connection speed might be very slow, with a very broken bandwith. This makes it very difficult to work in those environments, but with a few preparation you might be as effective than before ! 💥 ...

TL;DR This post will deep dive in the buidling of an end to end data engineering project ⚙️ . The idea will be to retrieve a price list of gas stations in France ⛽, create a job to extract it every day 📅 and craft a dashboard to expose those price to logged user 📊 After reading this blog post, you’ll have fundamentals on how to build data dashboard and scrap your own data sources 🚀 ...

TL;DR This post will expose you how to convert in a very convenient and fast way 🚀 some Apache Parquet files to CSV, and vice-versa, using either DuckDB 🦆 or Pandas 🐍 for a baseline comparison As a quick bonus, we will embedded this tool in a small convient CLI script, easily triggered from your favorite shell 👨‍💻 Let’s go ! Intro Recently, I’ve been working a little bit more on Data Engineering tasks (setup a Datalake, convert data, design pipelines, make cleanup of some data). 📊 ...