DP-750 practice question 2 of 10
You need to develop the task logic for a new job in Lakeflow Jobs that processes telemetry data. Each task must contain only the appropriate logic for its step in the pipeline. The solution must support the planned changes and meet the data ingestion and processing requirements. What should you do?
Full set: 91 questionsThe question requires a modular and maintainable design for a data processing pipeline with distinct stages: ingestion, cleansing, and curation. Creating separate tasks for each stage aligns with best practices for building robust data engineering workflows. This approach adheres to the principle of separation of concerns, where each task has a single, well-defined responsibility. This modularity makes the overall job easier to develop, debug, and maintain. If the logic for one stage needs to be changed (as implied by "support the planned changes"), only the corresponding task needs to be updated, minimizing the risk of introducing errors into other parts of the pipeline.
A. A single notebook task creates a monolithic script, which is difficult to maintain, debug, and modify, directly contradicting the requirements for modularity and support for future changes.
B. Creating three tasks with identical logic is functionally incorrect for a multi-stage pipeline and does not perform the distinct steps of ingestion, cleansing, and curation.
C. A single SQL task, similar to a single notebook, is a monolithic approach that violates the requirement for each task to contain only the logic for its specific step.
1. Microsoft Learn
"Orchestrate data processing with Azure Databricks jobs": This official learning path for Databricks explains the structure of jobs. It states
"A job can consist of a single task
or it can be a large
complex workflow with many tasks and dependencies between them. In a multi-task job
Databricks manages the orchestration of the tasks
creating a directed acyclic graph (DAG) of the task dependencies." This directly supports creating separate
dependent tasks for a multi-stage workflow like the one described.
Source: Microsoft Learn
Module: "Orchestrate data processing with Azure Databricks jobs"
Unit: "Create and run jobs in Azure Databricks".
2. Databricks Documentation
"What is Azure Databricks Jobs?": The official documentation emphasizes the composition of jobs from one or more tasks. It describes how to "create a job that is a complex workflow with multiple tasks with dependencies." This architectural pattern is precisely what is needed to separate ingestion
cleansing
and curation into distinct
manageable units of work.
Source: Databricks Documentation
Section: "Jobs"
Article: "What is Azure Databricks Jobs?".
3. Databricks Documentation
"Create and run Databricks Jobs": This guide provides practical steps for building multi-task jobs
stating
"You can use task dependencies to specify the order of execution of tasks in your job." This feature is essential for implementing a sequential pipeline where the curation task runs only after the cleansing task succeeds
which in turn runs after ingestion.
Source: Databricks Documentation
Section: "Jobs"
Article: "Create and run Databricks Jobs"
Subsection: "Configure task dependencies".
Keep practicing
Take the free DP-750 practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other DP-750 practice questions
- Question 1You have an Azure Databricks workspace that is enabled for Unity Catalog and contains two…
- Question 3You need to configure compute for the ingestion of telemetry data. The solution must meet…
- Question 4You have an Azure Databricks workspace named Workspace1 that contains a lakehouse and is …
