astronomer.providers.apache.livy.hooks.livy

This module contains the Apache Livy hook async.

Module Contents

Classes

LivyHookAsync

Hook for Apache Livy through the REST API using LivyHookAsync

class astronomer.providers.apache.livy.hooks.livy.LivyHookAsync(livy_conn_id=default_conn_name, extra_options=None, extra_headers=None)

Bases: astronomer.providers.http.hooks.http.HttpHookAsync, airflow.utils.log.logging_mixin.LoggingMixin

Hook for Apache Livy through the REST API using LivyHookAsync

Parameters:

• livy_conn_id (str) – reference to a pre-defined Livy Connection.

• extra_options (Optional[Dict[str, Any]]) – Additional option can be passed when creating a request. For example, run(json=obj) is passed as aiohttp.ClientSession().get(json=obj)

• extra_headers (Optional[Dict[str, Any]]) – A dictionary of headers passed to the HTTP request to livy.

See also

For more details refer to the Apache Livy API reference: Apache Livy API reference

TERMINAL_STATES

conn_name_attr = 'livy_conn_id'

default_conn_name = 'livy_default'

conn_type = 'livy'

hook_name = 'Apache Livy'

async run_method(endpoint, method='GET', data=None, headers=None)

Wrapper for HttpHookAsync, allows to change method on the same HttpHookAsync

Parameters:

• method (str) – http method

• endpoint (str) – endpoint

• data (Optional[Any]) – request payload

• headers (Optional[Dict[str, Any]]) – headers

Returns:

http response

Return type:

requests.Response

async get_batch_state(session_id)

Fetch the state of the specified batch asynchronously.

Parameters:

session_id (Union[int, str]) – identifier of the batch sessions

Returns:

batch state

Return type:

BatchState

async get_batch_logs(session_id, log_start_position, log_batch_size)

Gets the session logs for a specified batch asynchronously.

Parameters:

• session_id (Union[int, str]) – identifier of the batch sessions

• log_start_position (int) – Position from where to pull the logs

• log_batch_size (int) – Number of lines to pull in one batch

Returns:

response body

Return type:

dict

async dump_batch_logs(session_id)

Dumps the session logs for a specified batch asynchronously

Parameters:

session_id (Union[int, str]) – identifier of the batch sessions

Returns:

response body

Return type:

dict

static build_post_batch_body(file, args=None, class_name=None, jars=None, py_files=None, files=None, archives=None, name=None, driver_memory=None, driver_cores=None, executor_memory=None, executor_cores=None, num_executors=None, queue=None, proxy_user=None, conf=None)

Build the post batch request body.

Parameters:

• file (str) – Path of the file containing the application to execute (required).

• proxy_user (Optional[str]) – User to impersonate when running the job.

• class_name (Optional[str]) – Application Java/Spark main class string.

• args (Optional[Sequence[Union[str, int, float]]]) – Command line arguments for the application s.

• jars (Optional[List[str]]) – jars to be used in this sessions.

• py_files (Optional[List[str]]) – Python files to be used in this session.

• files (Optional[List[str]]) – files to be used in this session.

• driver_memory (Optional[str]) – Amount of memory to use for the driver process string.

• driver_cores (Optional[Union[int, str]]) – Number of cores to use for the driver process int.

• executor_memory (Optional[str]) – Amount of memory to use per executor process string.

• executor_cores (Optional[int]) – Number of cores to use for each executor int.

• num_executors (Optional[Union[int, str]]) – Number of executors to launch for this session int.

• archives (Optional[List[str]]) – Archives to be used in this session.

• queue (Optional[str]) – The name of the YARN queue to which submitted string.

• name (Optional[str]) – The name of this session string.

• conf (Optional[Dict[Any, Any]]) – Spark configuration properties.

Returns:

request body

Return type:

dict

For more information about the format refer to .. seealso:: https://livy.apache.org/docs/latest/rest-api.html