tonglin0325的个人主页

Ubuntu16.04安装flink-1.10.0

本来想cdh集成flink,但是我的cdh版本为5.16.2,参考了下面的issue可能cdh版本太低,至少要cdh6

1
2
https://github.com/pkeropen/flink-parcel/issues

进行独立安装

1
2
wget https://archive.apache.org/dist/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.11.tgz

安装路径

1
2
/home/lintong/software/apache/flink-1.10.0

/etc/profile添加,并source /etc/profile

1
2
3
4
#flink
export FLINK_HOME=/home/lintong/software/apache/flink-1.10.0
export PATH=${FLINK_HOME}/bin:$PATH

下载flink-shaded-hadoop-2-uber-2.7.5-7.0.jar包,放到flink的lib目录下

1
2
wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar

不然flink on yarn的时候会报

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

启动yarn-session

1
2
yarn-session.sh -n 3 -s 5 -jm 1024 -tm 4096 -d

yarn-seesion参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
-n : 指定TaskManager的数量;
-d: 以分离模式运行;
-id:指定yarn的任务ID;
-j:Flink jar文件的路径;
-jm:JobManager容器的内存(默认值:MB);
-nl:为YARN应用程序指定YARN节点标签;
-nm:在YARN上为应用程序设置自定义名称;
-q:显示可用的YARN资源(内存,内核);
-qu:指定YARN队列;
-s:指定TaskManager中slot的数量;
-st:以流模式启动Flink;
-tm:每个TaskManager容器的内存(默认值:MB);
-z:命名空间,用于为高可用性模式创建Zookeeper子路径;

去CDH上查看,第一个是正在运行,第二个是结束

去appliance id进到yarn的app页面

 

再点击ApplicationMaster进到Flink Dashboard页面

再提交Flink任务到yarn上

1
2
lintong@master:~/software/apache/flink-1.10.0$ bin/flink run examples/batch/WordCount.jar

再查看Flink Dashboard

 

输出

 

 

如果页面中的Available task slot一直为0的话,运行任务报,那说明yarn的资源不够

1
2
3
4
5
6
7
8
9
10
11
12
Caused by: org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: Could not allocate the required slot within slot request timeout. Please make sure that the cluster has enough resources.
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:452)
... 45 more
Caused by: java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
... 25 more
Caused by: java.util.concurrent.TimeoutException
... 23 more

去cdh上调整yarn.nodemanager.resource.memory-mb,容器内存,我这边从2G调整成4G就可以运行flink任务