PROFESSIONAL-MACHINE-LEARNING-ENGINEER practice question 20 of 35

You recently joined a machine learning team that will soon release a new project. As a lead on the project, you are asked to determine the production readiness of the ML components. The team has already tested features and data, model development, and infrastructure. Which additional readiness check should you recommend to the team?

Full set: 283 questions
Answer options
A: Ensure that training is reproducible
B: Ensure that all hyperparameters are tuned
C: Ensure that model performance is monitored Correct answer
D: Ensure that feature expectations are captured in the schema
Correct Answer:
C. Ensure that model performance is monitored
Explanation

Production readiness for a machine learning system encompasses both pre-deployment validation and post-deployment operational capabilities. The team has already addressed key pre-deployment stages: data validation, model development, and infrastructure setup. The critical missing component is a plan for the model's lifecycle after it is deployed. Ensuring that model performance is monitored is a fundamental post-deployment readiness check. This allows the team to track key metrics, detect performance degradation due to concept or data drift, and ensure the model continues to deliver business value in the live production environment.

Why Incorrect

A. Ensuring training is reproducible is a foundational aspect of the "model development" and "infrastructure" testing phases, which were already completed.
B. Ensuring all hyperparameters are tuned is a core activity within the "model development" stage, which the prompt states has been tested.
D. Ensuring feature expectations are captured in a schema is a primary task of the "features and data" testing phase, which has already been covered. ---

References

1. Google Cloud AI & Machine Learning Documentation: In the whitepaper "MLOps: Continuous delivery and automation pipelines in machine learning," the architecture for an MLOps system explicitly includes "Model monitoring" as a distinct, crucial stage that follows model deployment. This stage is responsible for "monitoring the quality of the model," which directly supports the correct answer. (See Figure 1: MLOps automated pipeline, Stage 08: Model monitoring).

2. Zinkevich, M. (2017). "Rules of Machine Learning: Best Practices for ML Engineering." This guide, originating from Google Research, dedicates a section to post-deployment considerations. The section titled "Monitoring" explicitly states, "you should be monitoring your models in production," highlighting the need to track metrics like prediction bias, action bias, and up-to-date performance to catch issues that arise only after launch.

3. Stanford University Courseware (CS 329S: Machine Learning Systems Design): Lecture 11, "MLOps & Tooling," emphasizes that a key part of the ML lifecycle is monitoring for performance degradation and data drift after a model is in production. This is presented as a separate and essential concern from the pre-deployment development and testing phases.

Scroll to Top