You need to configure compute for the ingestion of telemetry data. The solution must meet the data ingestion and processing requirements. What should…

The most effective way to meet the performance requirements for telemetry data ingestion and processing is to enable Photon acceleration. Photon is a high-performance, vectorized query engine native to Databricks that significantly speeds up SQL and DataFrame API workloads, which are typical for ETL and ingestion tasks. Configuring this on a job compute cluster is the best practice for automated, production pipelines, as it provides an isolated and cost-effective environment tailored specifically for the job's execution, aligning perfectly with the requirements.

B. Move the ingestion pipelines to shared compute.

Shared (all-purpose) compute is designed for interactive, ad-hoc analysis by multiple users, not for running automated production pipelines, which is less cost-effective and provides less isolation.

C. Increase an all-purpose cluster to a larger fixed node type.

While increasing node size can improve performance, using an all-purpose cluster for a production ingestion job is inefficient. Photon offers performance gains without necessarily increasing hardware costs.

D. Disable autoscaling for a job compute cluster.

Disabling autoscaling for a workload like telemetry, which can have variable volume, is counterproductive. It can lead to either over-provisioning and unnecessary costs or under-provisioning and performance bottlenecks.

---

1. Microsoft Learn

Azure Databricks Documentation

"What is Photon?".

Reference Section: Overview.

Content: "Photon is a high-performance Azure Databricks-native vectorized query engine that runs your SQL workloads and DataFrame API calls faster to reduce your total cost per workload... Photon is enabled by default in clusters running Databricks Runtime 9.1 LTS and above." This directly supports using Photon to accelerate ingestion workloads.

2. Microsoft Learn

Azure Databricks Documentation

"Use job compute to run your Azure Databricks jobs".

Reference Section: "Why use job compute?".

Content: "Job compute is dedicated to the job

providing workload isolation... Using job compute for your scheduled or triggered jobs is a best practice." This confirms that a job cluster is the correct environment

invalidating options B and C which suggest all-purpose/shared compute.

3. Microsoft Learn

Azure Databricks Documentation

"Cluster size and autoscaling".

Reference Section: "Autoscaling".

Content: "Autoscaling makes it easier to achieve high cluster utilization

because you don't need to provision the cluster to match a workload... This can significantly reduce your overall costs compared to a statically-sized cluster." This explains why disabling autoscaling (Option D) is generally not the recommended approach for efficiency.