You need to develop the task logic for a new job in Lakeflow Jobs that processes telemetry data. Each task must contain only the appropriate logic…

The question requires a modular and maintainable design for a data processing pipeline with distinct stages: ingestion, cleansing, and curation. Creating separate tasks for each stage aligns with best practices for building robust data engineering workflows. This approach adheres to the principle of separation of concerns, where each task has a single, well-defined responsibility. This modularity makes the overall job easier to develop, debug, and maintain. If the logic for one stage needs to be changed (as implied by "support the planned changes"), only the corresponding task needs to be updated, minimizing the risk of introducing errors into other parts of the pipeline.

A. A single notebook task creates a monolithic script, which is difficult to maintain, debug, and modify, directly contradicting the requirements for modularity and support for future changes.

B. Creating three tasks with identical logic is functionally incorrect for a multi-stage pipeline and does not perform the distinct steps of ingestion, cleansing, and curation.

C. A single SQL task, similar to a single notebook, is a monolithic approach that violates the requirement for each task to contain only the logic for its specific step.

1. Microsoft Learn

"Orchestrate data processing with Azure Databricks jobs": This official learning path for Databricks explains the structure of jobs. It states

"A job can consist of a single task

or it can be a large

complex workflow with many tasks and dependencies between them. In a multi-task job

Databricks manages the orchestration of the tasks

creating a directed acyclic graph (DAG) of the task dependencies." This directly supports creating separate

dependent tasks for a multi-stage workflow like the one described.

Source: Microsoft Learn

Module: "Orchestrate data processing with Azure Databricks jobs"

Unit: "Create and run jobs in Azure Databricks".

2. Databricks Documentation

"What is Azure Databricks Jobs?": The official documentation emphasizes the composition of jobs from one or more tasks. It describes how to "create a job that is a complex workflow with multiple tasks with dependencies." This architectural pattern is precisely what is needed to separate ingestion

cleansing

and curation into distinct

manageable units of work.

Source: Databricks Documentation

Section: "Jobs"

Article: "What is Azure Databricks Jobs?".

3. Databricks Documentation

"Create and run Databricks Jobs": This guide provides practical steps for building multi-task jobs

stating

"You can use task dependencies to specify the order of execution of tasks in your job." This feature is essential for implementing a sequential pipeline where the curation task runs only after the cleansing task succeeds

which in turn runs after ingestion.

Source: Databricks Documentation

Section: "Jobs"

Article: "Create and run Databricks Jobs"

Subsection: "Configure task dependencies".