PROFESSIONAL-MACHINE-LEARNING-ENGINEER practice question 9 of 35

You recently joined an enterprise-scale company that has thousands of datasets. You know that there are accurate descriptions for each table in BigQuery, and you are searching for the proper BigQuery table to use for a model you are building on AI Platform. How should you find the data that you need?

Full set: 283 questions
Answer options
A: Use Data Catalog to search the BigQuery datasets by using keywords in the table description. Correct answer
B: Tag each of your model and version resources on AI Platform with the name of the BigQuery table that was used for training.
C: Maintain a lookup table in BigQuery that maps the table descriptions to the table ID. Query the lookup table to find the correct table ID for the data that you need.
D: Execute a query in BigQuery to retrieve all the existing table names in your project using the INFORMATION_SCHEMA metadata tables that are native to BigQuery. Use the result o find the table that you need.
Correct Answer:
A. Use Data Catalog to search the BigQuery datasets by using keywords in the table description.
Explanation

Data Catalog is Google Cloud's fully managed data discovery and metadata management service. It is specifically designed for enterprise-scale scenarios involving numerous datasets. It automatically ingests metadata from sources like BigQuery, including table descriptions, and provides a powerful search interface. This allows users to efficiently find relevant data assets by searching for keywords in their descriptions, making it the ideal tool for the described task of locating the correct table among thousands.

Why Incorrect

B. Tagging AI Platform models is a best practice for tracking data lineage after a model is trained, not for the initial discovery of the training data itself.
C. Maintaining a custom lookup table is inefficient and redundant. It requires manual effort to build and maintain, whereas Data Catalog provides this functionality automatically as a managed service.
D. Querying INFORMATIONSCHEMA is a valid way to get metadata, but it is less scalable and user-friendly than Data Catalog for enterprise-wide discovery, especially across multiple projects.

References

1. Google Cloud, Data Catalog documentation, "Introduction to Data Catalog": "Data Catalog is a fully managed and scalable data discovery and metadata management service. [...] Data Catalog provides a centralized place for organizations to find, understand, and manage data." This establishes Data Catalog as the purpose-built tool for this scenario.

2. Google Cloud, Data Catalog documentation, "Search with Data Catalog": "Data Catalog provides a powerful predicate-based search experience for finding data assets to which you have access. You can search on the metadata of an asset." This section explicitly details how to search using metadata like descriptions, which directly addresses the user's need.

3. Google Cloud, BigQuery documentation, "Introduction to BigQuery INFORMATIONSCHEMA": "BigQuery INFORMATIONSCHEMA is a series of views that provide access to metadata about datasets, routines, tables, views, jobs, reservations, and streaming data." This confirms that option D is technically possible, but the existence of the more specialized Data Catalog service makes it a less optimal choice for enterprise-scale discovery.

Scroll to Top