Airflow is one of the most popular ETL tools out there. The GCP (Google Cloud Platform) managed service of Airflow — Cloud Composer does a lot of the heavy lifting for us, so we can focus on modeling and building awesome data tools. As I will elaborate further in this post, to support a stable and scalable environment there are still a few things we should keep in mind — such as managing per task dependencies and workloads. This is where Kubernetes Pod Operator comes in handy.

Although I was able to find some info about implementing such an architecture…

Data Engineer @ Resident

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store