07: Task Parameters — Passing Values Between Tasks

📂 View Source Code

In this module, you will learn how to pass dynamic values from an upstream task to multiple downstream tasks. This is a core pattern for building data-aware pipelines where the output of one step controls the behaviour of the next.

The workflow uses dbutils.jobs.taskValues.set() in Task A to broadcast values, and routes those values to Task B and Task C using the base_parameters mapping syntax in the bundle configuration.

What Are We Building?

DAB Architecture Setup

Prerequisites and Local Setup

Complete the Prerequisites and Local Setup before continuing.

Bundle Structure

  1. databricks.yml — Master control file.
  2. resources/jobs/task_parameters_job.yml — Defines the base_parameters mapping (e.g., {{tasks.task_a.values.target_table}}).
  3. src/task_a.py — Sets values using dbutils.jobs.taskValues.set().
  4. src/task_b.py and src/task_c.py — Retrieve values using dbutils.widgets.get().

How to Deploy and Run

Once authenticated, navigate to this folder (07-task-parameters) in your terminal.

Step 1: Validate and deploy

databricks bundle validate
databricks bundle deploy

Validation and Deployment Success

Step 2: Run the job

databricks bundle run task_parameters_job

Run Workflow Success

Step 3: Verify the output

Task B and Task C execute in parallel. Each receives the values broadcast by Task A.