:py:mod:`astronomer.providers.apache.livy.operators.livy`
=========================================================

.. py:module:: astronomer.providers.apache.livy.operators.livy

.. autoapi-nested-parse::

   This module contains the Apache Livy operator async.



Module Contents
---------------

Classes
~~~~~~~

.. autoapisummary::

   astronomer.providers.apache.livy.operators.livy.LivyOperatorAsync




.. py:class:: LivyOperatorAsync(*, file, class_name = None, args = None, conf = None, jars = None, py_files = None, files = None, driver_memory = None, driver_cores = None, executor_memory = None, executor_cores = None, num_executors = None, archives = None, queue = None, name = None, proxy_user = None, livy_conn_id = 'livy_default', livy_conn_auth_type = None, polling_interval = 0, extra_options = None, extra_headers = None, retry_args = None, **kwargs)

   Bases: :py:obj:`airflow.providers.apache.livy.operators.livy.LivyOperator`

   This operator wraps the Apache Livy batch REST API, allowing to submit a Spark
   application to the underlying cluster asynchronously.

   :param file: path of the file containing the application to execute (required).
   :param class_name: name of the application Java/Spark main class.
   :param args: application command line arguments.
   :param jars: jars to be used in this sessions.
   :param py_files: python files to be used in this session.
   :param files: files to be used in this session.
   :param driver_memory: amount of memory to use for the driver process.
   :param driver_cores: number of cores to use for the driver process.
   :param executor_memory: amount of memory to use per executor process.
   :param executor_cores: number of cores to use for each executor.
   :param num_executors: number of executors to launch for this session.
   :param archives: archives to be used in this session.
   :param queue: name of the YARN queue to which the application is submitted.
   :param name: name of this session.
   :param conf: Spark configuration properties.
   :param proxy_user: user to impersonate when running the job.
   :param livy_conn_id: reference to a pre-defined Livy Connection.
   :param polling_interval: time in seconds between polling for job completion. If poll_interval=0, in that case
       return the batch_id and if polling_interval > 0, poll the livy job for termination in the polling interval
       defined.
   :param extra_options: Additional option can be passed when creating a request.
           For example, ``run(json=obj)`` is passed as ``aiohttp.ClientSession().get(json=obj)``
   :param extra_headers: A dictionary of headers passed to the HTTP request to livy.
   :param retry_args: Arguments which define the retry behaviour.
           See Tenacity documentation at https://github.com/jd/tenacity

   .. py:method:: execute(context)

      Airflow runs this method on the worker and defers using the trigger.
      Submit the job and get the job_id using which we defer and poll in trigger


   .. py:method:: execute_complete(context, event)

      Callback for when the trigger fires - returns immediately.
      Relies on trigger to throw an exception, otherwise it assumes execution was
      successful.