02: Job with Parameters

📂 View Source Code

In this module, you will learn two core enterprise concepts:

  1. Modular Architecture: Separating DAB configurations into dedicated resource folders (jobs/, pipelines/, etc.).
  2. Dynamic Workflows: Defining job parameters and passing them into a PySpark script at runtime.

What Are We Building?

DAB Architecture Setup

Prerequisites and Local Setup

Complete the Prerequisites and Local Setup before continuing.

Bundle Structure

  1. databricks.yml — The master control file that uses include: to load external YAML files.
  2. resources/jobs/parameterized_job.yml — Contains the job definition and default parameter values.
  3. src/job_parameters.py — A PySpark script that uses argparse to accept command-line arguments.

How to Deploy and Run

Once authenticated, navigate to this folder (02-job-parameters) in your terminal.

Step 1: Validate and deploy

databricks bundle validate
databricks bundle deploy

Validation Deploy Success

Step 2: Run the job with default parameters

This uses the default values defined in resources/jobs/parameterized_job.yml.

databricks bundle run parameterized_job

Job Run Default Values

Workflow UI View

Step 3: Run the job with overridden parameters

You can inject new values at runtime without changing any code.

databricks bundle run parameterized_job --params environment=production,greeting="Hello YouTube"

Job Run New Values

Workflow UI View