EMR Operator Async¶
To submits jobs to EMR on EKS virtual clusters asynchronously
EmrContainerOperatorAsync
.
run_emr_container_job = EmrContainerOperatorAsync(
task_id="run_emr_container_job",
virtual_cluster_id=str(VIRTUAL_CLUSTER_ID),
execution_role_arn=JOB_ROLE_ARN,
release_label="emr-6.2.0-latest",
job_driver=JOB_DRIVER_ARG,
configuration_overrides=CONFIGURATION_OVERRIDES_ARG,
name="pi.py",
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/amazon/aws/example_dags/example_emr_eks_containers_job.py