astronomer.providers.apache.livy.hooks.livy
¶
This module contains the Apache Livy hook async.
Module Contents¶
Classes¶
Hook for Apache Livy through the REST API using LivyHookAsync |
- class astronomer.providers.apache.livy.hooks.livy.LivyHookAsync(livy_conn_id=default_conn_name, extra_options=None, extra_headers=None)[source]¶
Bases:
astronomer.providers.http.hooks.http.HttpHookAsync
,airflow.utils.log.logging_mixin.LoggingMixin
Hook for Apache Livy through the REST API using LivyHookAsync
- Parameters:
livy_conn_id (str) – reference to a pre-defined Livy Connection.
extra_options (Optional[Dict[str, Any]]) – Additional option can be passed when creating a request. For example,
run(json=obj)
is passed asaiohttp.ClientSession().get(json=obj)
extra_headers (Optional[Dict[str, Any]]) – A dictionary of headers passed to the HTTP request to livy.
See also
For more details refer to the Apache Livy API reference: Apache Livy API reference
- TERMINAL_STATES¶
- conn_name_attr = livy_conn_id¶
- default_conn_name = livy_default¶
- conn_type = livy¶
- hook_name = Apache Livy¶
- async run_method(endpoint, method='GET', data=None, headers=None)[source]¶
Wrapper for HttpHookAsync, allows to change method on the same HttpHookAsync
- Parameters:
method (str) – http method
endpoint (str) – endpoint
data (Optional[Any]) – request payload
headers (Optional[Dict[str, Any]]) – headers
- Returns:
http response
- Return type:
requests.Response
- async get_batch_state(session_id)[source]¶
Fetch the state of the specified batch asynchronously.
- Parameters:
session_id (Union[int, str]) – identifier of the batch sessions
- Returns:
batch state
- Return type:
BatchState
- async get_batch_logs(session_id, log_start_position, log_batch_size)[source]¶
Gets the session logs for a specified batch asynchronously.
- Parameters:
session_id (Union[int, str]) – identifier of the batch sessions
log_start_position (int) – Position from where to pull the logs
log_batch_size (int) – Number of lines to pull in one batch
- Returns:
response body
- Return type:
dict
- async dump_batch_logs(session_id)[source]¶
Dumps the session logs for a specified batch asynchronously
- Parameters:
session_id (Union[int, str]) – identifier of the batch sessions
- Returns:
response body
- Return type:
dict
- static build_post_batch_body(file, args=None, class_name=None, jars=None, py_files=None, files=None, archives=None, name=None, driver_memory=None, driver_cores=None, executor_memory=None, executor_cores=None, num_executors=None, queue=None, proxy_user=None, conf=None)[source]¶
Build the post batch request body.
- Parameters:
file (str) – Path of the file containing the application to execute (required).
proxy_user (Optional[str]) – User to impersonate when running the job.
class_name (Optional[str]) – Application Java/Spark main class string.
args (Optional[Sequence[Union[str, int, float]]]) – Command line arguments for the application s.
jars (Optional[List[str]]) – jars to be used in this sessions.
py_files (Optional[List[str]]) – Python files to be used in this session.
files (Optional[List[str]]) – files to be used in this session.
driver_memory (Optional[str]) – Amount of memory to use for the driver process string.
driver_cores (Optional[Union[int, str]]) – Number of cores to use for the driver process int.
executor_memory (Optional[str]) – Amount of memory to use per executor process string.
executor_cores (Optional[int]) – Number of cores to use for each executor int.
num_executors (Optional[Union[int, str]]) – Number of executors to launch for this session int.
archives (Optional[List[str]]) – Archives to be used in this session.
queue (Optional[str]) – The name of the YARN queue to which submitted string.
name (Optional[str]) – The name of this session string.
conf (Optional[Dict[Any, Any]]) – Spark configuration properties.
- Returns:
request body
- Return type:
dict
For more information about the format refer to .. seealso:: https://livy.apache.org/docs/latest/rest-api.html