astronomer.providers.apache.hive.sensors.named_hive_partition
¶
Module Contents¶
Classes¶
Waits asynchronously for a set of partitions to show up in Hive. |
- class astronomer.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensorAsync(*, partition_names, metastore_conn_id='metastore_default', poke_interval=60 * 3, hook=None, **kwargs)[source]¶
Bases:
airflow.providers.apache.hive.sensors.named_hive_partition.NamedHivePartitionSensor
Waits asynchronously for a set of partitions to show up in Hive.
Note
HivePartitionSensorAsync uses impyla library instead of PyHive. The sync version of this sensor uses PyHive <https://github.com/dropbox/PyHive>.
Since we use impyla library, please set the connection to use the port
10000
instead of9083
. Forauth_mechanism='GSSAPI'
the ticket renewal happens through commandairflow kerberos
in worker/trigger.You may also need to allow traffic from Airflow worker/Triggerer to the Hive instance, depending on where they are running. For example, you might consider adding an entry in the
etc/hosts
file present in the Airflow worker/Triggerer, which maps the EMR Master node Public IP Address to its Private DNS Name to allow the network traffic.The library version of hive and hadoop in
Dockerfile
should match the remote cluster where they are running.- Parameters:
partition_names (List[str]) – List of fully qualified names of the partitions to wait for. A fully qualified name is of the form
schema.table/pk1=pv1/pk2=pv2
, for example, default.users/ds=2016-01-01.metastore_conn_id (str) – Metastore thrift service connection id.