NCP-AAI practice question 23 of 25
Which technique is MOST useful for decreasing response latency without retraining the base model?
Full set: 118 questionsTo decrease response latency without retraining the base model, the most effective techniques are inference optimizations. These methods focus on making the model execute faster on the hardware. Quantization reduces the precision of the model's weights (e.g., from FP32 to INT8), which decreases memory footprint and speeds up computation. Serving improvements, such as using NVIDIA Triton Inference Server with a TensorRT-LLM backend, compile the model into a highly optimized engine for the target GPU architecture. These approaches directly reduce the time it takes to generate a response for a given input, addressing the latency requirement without altering the model's learned parameters.
B. Increasing prompt length adds more tokens for the model to process, which directly increases, not decreases, latency.
C. Adding manual review steps introduces human-in-the-loop delays, significantly increasing the total time to get a response.
D. Removing batching can hurt throughput and may not decrease single-request latency; optimized batching often improves overall system performance.
1. NVIDIA. (2024). NVIDIA TensorRT-LLM Documentation. Section: "Features". The documentation explicitly states that TensorRT-LLM provides "in-flight batching" and "INT8 and FP8 quantization" to optimize inference performance and reduce latency for large language models.
2. NVIDIA. (2024). NVIDIA Triton Inference Server Documentation. Section: "Optimization". This guide details how Triton can be configured for performance, including using optimized backends like TensorRT to "minimize latency."
3. Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. This academic paper demonstrates how 8-bit quantization can accelerate inference for large transformer models without significant performance degradation. DOI: https://doi.org/10.48550/arXiv.2208.07339
Keep practicing
Take the free NCP-AAI practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other NCP-AAI practice questions
- Question 1Which two design goals are MOST important for human-in-the-loop agent systems in regulate…
- Question 2A company plans to launch a multi-agent system that must serve thousands of users simulta…
- Question 3Which two orchestration methods are MOST suitable for implementing complex agentic workfl…
- Question 4You are designing an AI agent for summarizing medical documents that include images and t…
- Question 5A company is deploying a multi-agent AI system to handle large-scale customer interaction…
- Question 6A multi-agent workflow is failing because one agent completes tasks before dependent stat…
