DEA-C01 practice question 21 of 25

A data engineer is building a data pipeline. A large data file is uploaded to an Amazon S3 bucket once each day at unpredictable times. An AWS Glue workflow uses hundreds of workers to process the file and load the data into Amazon Redshift. The company wants to process the file as quickly as possible. Which solution will meet these requirements?

Answer options

A: Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Lambda function that runs every 15 minutes to check the S3 bucket for the daily file. Configure the function to start the AWS Glue workflow if the file is present.

B: Create an event-based AWS Glue trigger to start the workflow. Configure Amazon S3 to log events to AWS CloudTrail. Create a rule in Amazon EventBridge to forward PutObject events to the AWS Glue trigger. Correct answer

C: Create a scheduled AWS Glue trigger to start the workflow. Create a cron job that runs the AWS Glue job every 15 minutes. Set up the AWS Glue job to check the S3 bucket for the daily file. Configure the job to stop if the file is not present.

D: Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Database Migration Service (AWS DMS) migration task. Set the DMS source as the S3 bucket. Set the target endpoint as the AWS Glue workflow.

Correct Answer:

B. Create an event-based AWS Glue trigger to start the workflow. Configure Amazon S3 to log events to AWS CloudTrail. Create a rule in Amazon EventBridge to forward PutObject events to the AWS Glue trigger.

Explanation

The requirement is to process a file as quickly as possible after it arrives at an unpredictable time. An event-driven architecture is the most suitable solution. S3 can emit PutObject events to Amazon EventBridge when a new file is uploaded. An EventBridge rule can then filter for these specific events and trigger the AWS Glue workflow via an event-based Glue trigger. This approach minimizes latency by initiating the workflow immediately upon file arrival, directly fulfilling the core requirement without the delay or inefficiency of polling mechanisms.

Why Incorrect

A. A Lambda function polling every 15 minutes introduces an average latency of 7.5 minutes and a maximum of 15 minutes, which does not meet the "as quickly as possible" requirement.
C. A scheduled Glue trigger is a polling mechanism. Running a large Glue job just to check for a file is inefficient, costly, and introduces processing delays.
D. AWS Database Migration Service (AWS DMS) is designed for migrating data between data stores, not for orchestrating or triggering AWS Glue ETL workflows. This is an incorrect use of the service.

References

1. AWS Glue Developer Guide. (2023). Starting an AWS Glue workflow with an Amazon S3 event. Section: "To create a workflow that is triggered by an Amazon S3 event". AWS. Retrieved from https://docs.aws.amazon.com/glue/latest/dg/starting-workflow-eventbridge.html

2. Amazon S3 User Guide. (2023). Using EventBridge. This document explains how S3 bucket events can be delivered to EventBridge, which can then invoke targets like AWS Glue. AWS. Retrieved from https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

3. Amazon EventBridge User Guide. (2023). Amazon EventBridge event patterns. This guide details how to create rules that match events, such as PutObject from S3, to trigger specific targets. AWS. Retrieved from https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-and-event-patterns.html

Keep practicing

Take the free DEA-C01 practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test