• Home
• Building the image

↑↓ Navigate ⏎ Select Esc Close

Building the image

Before you dive-deeply in the way how the Airflow Image is built, let us first explain why you might need to build the custom container image and we show a few typical ways you can do it.

Quick start scenarios of image extending

The most common scenarios where you want to build your own image are adding a new apt package, adding a new PyPI dependency (either individually or via requirements.txt) and embedding dags into the image.

Example Dockerfiles for those scenarios are below, and you can read further for more complex cases which might involve either extending or customizing the image. You will find more information about more complex scenarios below, but if your goal is to quickly extend the Airflow image with new provider, package, etc. then here is a quick start for you.

Adding new apt package

The following example adds vim to the Airflow image. When adding packages via apt you should switch to the root user when running the apt commands, but do not forget to switch back to the airflow user after installation is complete.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-apt-packages/Dockerfile

FROM apache/airflow:3.3.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow

Adding new PyPI packages individually

The following example adds lxml python package from PyPI to the image. When adding packages via pip you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

Note

In the example below, we also add apache-airflow package to be installed - in the very same version that the image version you used it from. This is not strictly necessary, but it is a good practice to always install the same version of apache-airflow as the one you are using. This way you can be sure that the version you are using is the same as the one you are extending. In some cases where your new packages have conflicting dependencies, pip might decide to downgrade or upgrade apache-airflow for you, so adding it explicitly is a good practice - this way if you have conflicting requirements, you will get an error message with conflict information, rather than a surprise downgrade or upgrade of airflow. If you upgrade Airflow base image, you should also update the version to match the new version of airflow.

Note

Creating custom images means that you need to maintain also a level of automation as you need to re-create the images when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts. Also keep in mind, that in cases when you run pure Python tasks, you can use the Python Virtualenv functions which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages/Dockerfile

FROM apache/airflow:3.3.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml

Adding packages from requirements.txt

The following example adds few python packages from requirements.txt from PyPI to the image. Note that similarly when adding individual packages, you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

Note

In the example below, we also add apache-airflow package to be installed - in the very same version that the image version you used it from. This is not strictly necessary, but it is a good practice to always install the same version of apache-airflow as the one you are using. This way you can be sure that the version you are using is the same as the one you are extending. In some cases where your new packages have conflicting dependencies, pip might decide to downgrade or upgrade apache-airflow for you, so adding it explicitly is a good practice - this way if you have conflicting requirements, you will get an error message with conflict information, rather than a surprise downgrade or upgrade of airflow. If you upgrade Airflow base image, you should also update the version to match the new version of airflow.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/Dockerfile

FROM apache/airflow:3.3.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/requirements.txt

lxml
beautifulsoup4

Embedding dags

The following example adds test_dag.py to your image in the /opt/airflow/dags folder.

/opt/airflow/docker-stack-docs/docker-examples/extending/embedding-dags/Dockerfile

FROM apache/airflow:3.3.0

COPY --chown=airflow:root test_dag.py /opt/airflow/dags

docker-examples/extending/embedding-dags/test_dag.py

import datetime

import pendulum

from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator

now = pendulum.now(tz="UTC")
now_to_the_hour = (now - datetime.timedelta(0, 0, 0, 0, 0, 3)).replace(minute=0, secon