Airflow is one of the most popular ETL tools out there. The GCP (Google Cloud Platform) managed service of Airflow — Cloud Composer does a lot of the heavy lifting for us, so we can focus on modeling and building awesome data tools. As I will elaborate further in this post, to support a stable and scalable environment there are still a few things we should keep in mind — such as managing per task dependencies and workloads. This is where Kubernetes Pod Operator comes in handy.
Data Engineer @ Resident