https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/batch/

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

readTextFile(path) / TextInputFormat - Reads files line wise and returns them as Strings.

readTextFileWithValue(path) / TextValueInputFormat - Reads files line wise and returns them as StringValues. StringValues are mutable strings.

readCsvFile(path) / CsvInputFormat - Parses files of comma (or another char) delimited fields. Returns a DataSet of tuples or POJOs. Supports the basic java types and their Value counterparts as field types.

readFileOfPrimitives(path, Class) / PrimitiveInputFormat - Parses files of new-line (or another char sequence) delimited primitive data types such as String or Integer.

readFileOfPrimitives(path, delimiter, Class) / PrimitiveInputFormat - Parses files of new-line (or another char sequence) delimited primitive data types such as String or Integer using the given delimiter.

fromCollection(Collection) - Creates a data set from a Java.util.Collection. All elements in the collection must be of the same type.

fromCollection(Iterator, Class) - Creates a data set from an iterator. The class specifies the data type of the elements returned by the iterator.

fromElements(T ...) - Creates a data set from the given sequence of objects. All objects must be of the same type.

fromParallelCollection(SplittableIterator, Class) - Creates a data set from an iterator, in parallel. The class specifies the data type of the elements returned by the iterator.

generateSequence(from, to) - Generates the sequence of numbers in the given interval, in parallel.

readFile(inputFormat, path) / FileInputFormat - Accepts a file input format.

createInput(inputFormat) / InputFormat - Accepts a generic input format.

https://ci.apache.org/projects/flink/flink-docs-stable/zh/dev/datastream_api.html

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

readTextFile(path) - Reads text files, i.e. files that respect the TextInputFormat specification, line-by-line and returns them as Strings.

readFile(fileInputFormat, path) - Reads (once) files as dictated by the specified file input format.

readFile(fileInputFormat, path, watchType, interval, pathFilter, typeInfo) - This is the method called internally by the two previous ones. It reads files in the path based on the given fileInputFormat. Depending on the provided watchType, this source may periodically monitor (every interval ms) the path for new data (FileProcessingMode.PROCESS_CONTINUOUSLY), or process once the data currently in the path and exit (FileProcessingMode.PROCESS_ONCE). Using the pathFilter, the user can further exclude files from being processed.

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

ExecutionEnvironment env = LocalEnvironment.getExecutionEnvironment();

ExecutionEnvironment env = CollectionEnvironment.getExecutionEnvironment();

https://github.com/apache/flink/blob/master/docs/dev/application_parameters.zh.md

String propertiesFilePath = "/home/sam/flink/myjob.properties";
ParameterTool parameter = ParameterTool.fromPropertiesFile(propertiesFilePath);

--input hdfs:///mydata --elements 42

https://vldb.org/pvldb/vol5/p1771_georgelee_vldb2012.pdf
https://www.kdd.org/exploration_files/V14-02-02-Lin.pdf

https://www.slideshare.net/Hadoop_Summit/scaling-big-data-mining-infrastructure-twitter-experience
https://slideplayer.com/slide/12451118/
https://blog.twitter.com/engineering/en_us/topics/insights/2016/discovery-and-consumption-of-analytics-data-at-twitter.html
https://www.slideshare.net/kevinweil/hadoop-at-twitter-hadoop-summit-2010
https://www.slideshare.net/kevinweil/protocol-buffers-and-hadoop-at-twitter/1-Twitter_Open_Source_Coming_soon

https://github.com/7-sevens/Developer-Books/blob/master/Redis/Redis%E5%AE%9E%E6%88%98.pdf

<!-- redis -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

https://archive.apache.org/dist/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

https://docs.aws.amazon.com/zh_cn/athena/latest/ug/serde-about.html

https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java

https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/RegexSerDe.java

https://github.com/apache/hadoop/blob/master/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/OutputFormat.java

https://github.com/apache/hadoop/tree/master/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output

https://github.com/apache/hadoop/blob/master/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/TextOutputFormat.java

https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-input-http_endpoint.html

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.9.2-linux-x86_64.tar.gz

filebeat.inputs:
- type: http_endpoint
  enabled: true
  listen_address: localhost
  listen_port: 19080

#----------------------------------File output--------------------------------#
output.file:
  path: "/tmp/filebeat"
  filename: filebeat