Skip to main content

Migrating an Airflow BashOperator (dbt) to Dagster

info

Airlift v2 is under active development. You may encounter feature gaps, and the APIs may change. To report issues or give feedback, please reach out to your CSM.

The Airflow BashOperator is used to execute bash commands as part of a data pipeline.

In Airflow, you might have a BashOperator that runs a dbt command. For example, you might have a task that runs dbt run to build your dbt models:

from airflow.operators.bash import BashOperator

run_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run")

Dagster equivalent​

The Dagster equivalent to using the BashOperator to run a dbt command is to use the dagster-dbt library to run commands against your dbt project:

import dagster_dbt as dg_dbt

import dagster as dg

project = dg_dbt.DbtProject(project_dir="path/to/dbt_project")


@dg_dbt.dbt_assets(manifest=project.manifest_path)
def my_dbt_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource):
    yield from dbt.cli(["run"], context=context).stream()

Migrating the operator​

To migrate the operator, you will need to:

  1. Make the dbt project available to both your Airflow and Dagster deployments and build the manifest
  2. Write a @dbt_asset-decorated function to run your dbt commands
  3. Use the Dagster Airlift component to proxy execution of the original task to Dagster

Step 1: Make the dbt project available to your Airflow and Dagster deployments and build the manifest​

First, you'll need to make the dbt project available to the Dagster runtime and build the manifest.

Step 2: Write a @dbt_asset-decorated function to run your dbt commands​

Once your dbt project is available to the Dagster runtime, you can write a function that runs your dbt commands using the @dagster_dbt.dbt_assets decorator and DbtCliResource. Most dbt CLI commands and flags are supported; to learn more about using @dbt_assets, check out the dagster-dbt quickstart and reference documentation.

Step 3: Use the Dagster Airlift component to proxy execution of the original task to Dagster​

Finally, you can use the Dagster Airlift component to proxy the execution of the original task to Dagster.