Apache Airflow - one position in our tech stack

One position in our tech stack is Apache Airflow. Our co-founder, Jan Kościałkowski, talked about it in an interview with Polidea

"What type of deployment of Airflow have you used, and why? Was it determined by your current architecture (i.e., the fact that you are cloud-native)?

Our product is currently in the development/PoC stage, so we wanted a semi-production environment. For the sake of cost management, we don’t want any VMs to run continuously, so there exist some Terraform scripts which set up a VM with Airflow running LocalExecutor and a separate MySQL DB. As we are currently exploring possible analyses, the data volume is not too high, and such a setup is sufficient. We will most likely switch to Kubernetes when we start generating analyzes of moderate complexity daily."

read more