sensor也是airflow的一种operator,用于检测某个条件是否达到。如果条件满足,sensor将会执行成功;如果条件不满足,sensor将会重试,直到超时,task超时的时候状态就位skipped。
下面是常用的几种sensor:
- The FileSensor: Waits for a file or folder to land in a filesystem.
- The S3KeySensor: Waits for a key to be present in a S3 bucket.
- The SqlSensor: Runs a sql statement repeatedly until a criteria is met.
- The HivePartitionSensor: Waits for a partition to show up in Hive.
- The ExternalTaskSensor: Waits for a different DAG or a task in a different DAG to complete for a specific execution date. (Pretty useful that one 🤓 )
- The DateTimeSensor: Waits until the specified datetime (Useful to add some delay to your DAGs)
- The TimeDeltaSensor: Waits for a timedelta after the task’s execution_date + schedule interval (Looks similar to the previous one no?)
参考:Airflow Sensors : What you need to know
以及
How Airbnb Built “Wall” to prevent data bugs