A data engineer must orchestrate a data pipeline that consists of one AWS Lambda function and one AWS Glue job. The solution must integrate with AWS services. Which solution will meet these requirements with the LEAST management overhead?

AWS Step Functions is a serverless orchestration service specifically designed to coordinate multiple AWS services into a workflow. It has native integration with both AWS Lambda and AWS Glue, allowing a data engineer to define a state machine visually or with code that executes the Lambda function and then the Glue job. As a fully managed service, it handles state, error handling, and retries automatically, resulting in the least management overhead compared to self-managed or more complex solutions.

B. Deploying and managing Apache Airflow on an EC2 instance involves significant operational overhead for infrastructure and software management.
C. While AWS Glue Workflows can orchestrate Glue jobs and trigger Lambda functions, Step Functions is the more general-purpose and flexible orchestrator for diverse AWS services.
D. Using Apache Airflow on Amazon EKS introduces even more management overhead than on EC2, as it requires managing a Kubernetes cluster.

1. AWS Step Functions Developer Guide. (2023). What is AWS Step Functions?. AWS Documentation. Retrieved from https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html. The guide states, "AWS Step Functions is a serverless orchestration service that lets you integrate with AWS Lambda functions and other AWS services to build business-critical applications."

2. AWS Step Functions Developer Guide. (2023). Managing AWS Glue jobs with Step Functions. AWS Documentation. Retrieved from https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html. This section details the direct, managed integration between Step Functions and AWS Glue.

3. AWS Step Functions Developer Guide. (2023). Invoking Lambda functions. AWS Documentation. Retrieved from https://docs.aws.amazon.com/step-functions/latest/dg/connect-lambda.html. This page describes the native integration for invoking Lambda functions from a Step Functions workflow.