#AIRFLOW HELM CHART VALUES INSTALL#
#AIRFLOW HELM CHART VALUES UPDATE#

The application will get the name airflow and the flag –debug allows to check if anything goes wrong during the deployment.Īfter a few minutes, you should be able to see your Pods running, corresponding to the different Airflow components. Finally, you deploy Airflow on Kubernetes with Helm install. You can take a look at the current version with search repo.

#AIRFLOW HELM CHART VALUES UPDATE#

Then you update the repo to make you got the latest version of it. In the order of the commands above, you add the official repository of the Apache Airflow Helm chart is.

#AIRFLOW HELM CHART VALUES INSTALL#

Helm install airflow apache-airflow/airflow -namespace airflow -debug Or, almost magically 😅 helm repo add apache-airflow Then, thanks to Helm, you need to fetch the official Helm of Apache Airflow that will magically get deployed on your cluster. To deploy Airflow on Kuberntes, the first step is to create a namespace.

Once your local Kubernetes cluster is running, you are ready to deploy Airflow on it. You can do some checks kubectl cluster-info Ok, now execute the following command to create your Kubernetes cluster kind create cluster -name airflow-cluster -config kind-cluster.yaml That means, next to your kind-cluster.yaml, you should create a folder data and the logs of your tasks will be stored in /tmp/data within your Kubernetes cluster. Without this, you wouldn’t be able to access your logs once your tasks get completed in Airflow. Why? It allows to mount a hostPath persistent volume to store the logs of your tasks from your pods to your machine. In addition, each work node has the extraMounts field defined. For example, with the KubernetesExecutor you could define a nodeSelector in the executor_config argument of your Operator and run the task on a specific node 😉 It’s useful if you want to run your tasks on a specific node based on the label of that node. Notice that each worker node has a label attached to it. With it, you are going to create 4 nodes, 1 control plane and 3 workers. This file describe your local Kubernetes Cluster. If you take a look into the materials and open the file kind-cluster.yaml, you will obtain the following content: kind: Cluster Create a Kubernetes Cluster with KinDīefore deploying Airflow on Kubernetes, the first step, create and configure the local Kubernetes cluster with KinD. If you want to follow this very hands-on beautiful tutorial, check out the repository here where you will find all the materials needed. Once you’re done, you’re ready to go! Materials To run Airflow on Kubernetes you need 5 tools: Docker, Docker Compose, KinD, Helm and Kubectl. Without Kubectl, you won’t be able to get the logs of your PODs, debug your errors or check your nodes. As you know, Helm allows you deploy applications on Kubernetes whereas Kubectl allows you to run commands against your Kubernetes Cluster. In addition to Docker and KinD, you obviously need to install Helm as well as Kubectl. If you know MiniKube, Kind is pretty similar now (It wasn’t the case a few years ago). It was primarily designed for testing Kubernetes but it is perfect to have a quick local development environment to experiment applications that run on top of Kubernetes. KinD is the second tool you need as it allows you to set up and run a Kubernetes cluster using Docker container “nodes”. Now you may ask: “How can I create a Kubernetes cluster with Docker?” Well, say hello to KinD! You are going to have multiple Docker containers where each container will represent a Kubernetes node. Why? Because the Kubernetes cluster will be created in Docker.

Therefore, there are some prerequisites before jumping into the practice part.įirst, you need to install Docker and Docker Compose.

In this very hands-on focused tutorial you are going to create a local multi-node Kubernetes cluster to deploy and run Airflow on it. Deploy your DAGs on Kubernetes with GitSync.Install dependencies with Airflow on Kubernetes.

Airflow helm chart values