tonglin0325的个人主页

airflow学习笔记——sensor

sensor也是airflow的一种operator,用于检测某个条件是否达到。如果条件满足,sensor将会执行成功;如果条件不满足,sensor将会重试,直到超时,task超时的时候状态就位skipped。

下面是常用的几种sensor:

  • The FileSensor: Waits for a file or folder to land in a filesystem.
  • The S3KeySensor: Waits for a key to be present in a S3 bucket.
  • The SqlSensor: Runs a sql statement repeatedly until a criteria is met.
  • The HivePartitionSensor: Waits for a partition to show up in Hive.
  • The ExternalTaskSensor: Waits for a different DAG or a task in a different DAG to complete for a specific execution date. (Pretty useful that one 🤓 )
  • The DateTimeSensor: Waits until the specified datetime (Useful to add some delay to your DAGs)
  • The TimeDeltaSensor: Waits for a timedelta after the task’s execution_date + schedule interval (Looks similar to the previous one no?)

参考:Airflow Sensors : What you need to know

以及
 How Airbnb Built “Wall” to prevent data bugs