airflow学习笔记——operator

1. BaseOperator
2. BaseSensorOperator
3. MySQLToS3Operator
4. SqlToS3Operator
5. MongoToS3Operator
6. BashOperator：执行bash命令
7. PythonOperator：调用python代码
8. SparkOperators
9. EmailOperator：发送邮件
10. HTTPOperator : 发送 HTTP 请求
11. MySqlOperator : 执行 SQL 命令
12. SSHExecuteOperator，airflow1.9.0之后废除，改成 SSHOperator
13. SparkSubmitOperator：提交spark任务

operator用于产生特定的DAG节点

https://airflow.apache.org/docs/apache-airflow/stable/python-api-ref.html#operators

 

下面是常用的operator及其用法

BaseOperator#

https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/baseoperator.html

用法：

 

BaseSensorOperator#

https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/sensors/base.html#BaseSensorOperator.poke

用法：

 

MySQLToS3Operator#

这个已经过时了，建议使用SqlToS3Operator

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/transfers/mysql_to_s3.html

用法：

 

SqlToS3Operator#

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/transfer/sql_to_s3.html#howto-operator-sqltos3operator

用法：

 

MongoToS3Operator#

https://airflow.apache.org/docs/apache-airflow-providers-amazon/1.2.0/_modules/airflow/providers/amazon/aws/transfers/mongo_to_s3.html

用法：

 

BashOperator：执行bash命令#

https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html

 

PythonOperator：调用python代码#

https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html

 

SparkOperators#

https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/operators.html

 

EmailOperator：发送邮件#

https://bhavaniravi.com/blog/sending-emails-from-airflow/

 

HTTPOperator : 发送 HTTP 请求#

https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html

 

MySqlOperator : 执行 SQL 命令#

https://airflow.apache.org/docs/apache-airflow-providers-mysql/stable/operators.html

 

SSHExecuteOperator，airflow1.9.0之后废除，改成 SSHOperator#

https://newbedev.com/airflow-how-to-ssh-and-run-bashoperator-from-a-different-server

 

SparkSubmitOperator：提交spark任务#

https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/_api/airflow/providers/apache/spark/operators/spark_submit/index.html#airflow.providers.apache.spark.operators.spark_submit.SparkSubmitOperator

　　

除了这些基本的构建块之外，还有更多的特定处理器：DockerOperator，HiveOperator，S3FileTransferOperator，PrestoToMysqlOperator，SlackOperator