PROFESSIONAL-DATA-ENGINEER practice question 4 of 10

You need to deploy additional dependencies to all of a Cloud Dataproc cluster at startup using an existing initialization action. Company security policies require that Cloud Dataproc nodes do not have access to the Internet so public initialization actions cannot fetch resources. What should you do?

Full set: 401 questions
Answer options
A: Deploy the Cloud SQL Proxy on the Cloud Dataproc master
B: Use an SSH tunnel to give the Cloud Dataproc cluster access to the Internet
C: Copy all dependencies to a Cloud Storage bucket within your VPC security perimeter Correct answer
D: Use Resource Manager to add the service account used by the Cloud Dataproc cluster to the Network User role
Correct Answer:
C. Copy all dependencies to a Cloud Storage bucket within your VPC security perimeter
Explanation

The primary constraint is that Cloud Dataproc nodes cannot access the public internet, which prevents initialization actions from downloading dependencies from public repositories. The correct and secure approach is to pre-stage all required dependencies in a location accessible from within the Google Cloud private network. A Cloud Storage bucket is the ideal service for this. By copying dependencies to a bucket and ensuring the cluster's subnet has Private Google Access enabled, the initialization action script can use gsutil to securely copy the files from the bucket to the cluster nodes for installation, fully complying with the security policy.

Why Incorrect

A. The Cloud SQL Proxy is a specific tool for connecting securely to Cloud SQL instances; it is not used for managing or deploying general software dependencies on a cluster.
B. Creating an SSH tunnel to the internet directly contravenes the explicit security requirement that nodes must not have internet access. This is a policy violation, not a solution.
D. The Network User role (roles/compute.networkUser) grants permissions to use network resources, such as subnets in a Shared VPC. It does not provide a mechanism for accessing software dependencies.

References

1. Google Cloud Documentation - Dataproc Initialization Actions: "We recommend that you store initialization actions in a Cloud Storage bucket... When you create a cluster, Dataproc copies the scripts from the bucket to the master and worker nodes in the cluster and then runs the scripts." This establishes Cloud Storage as the standard location for initialization action resources. (Source: Google Cloud, "Initialization actions," Section: "Where to store initialization actions").

2. Google Cloud Documentation - Configuring Private Google Access: "With Private Google Access, VMs that only have internal IP addresses (no external IP addresses) can reach the external IP addresses of Google APIs and services," including Cloud Storage. This is the underlying network configuration that enables VMs without internet access to reach the Cloud Storage API. (Source: Google Cloud, "Private Google Access," Section: "Overview").

3. Google Cloud Documentation - Dataproc Cluster Networking: In scenarios without external IP addresses, "To access Google Cloud services, including Cloud Storage where the Dataproc staging bucket is located, you must enable Private Google Access on the cluster subnet." This explicitly links the need for Private Google Access to allow Dataproc to access Cloud Storage in isolated networks. (Source: Google Cloud, "Dataproc Cluster Network Configuration," Section: "Cluster network requirements and considerations").

Scroll to Top