hadoop - yarn HA

zhyilong

546人浏览 · 2022-10-07 18:42:25

zhyilong · 2022-10-07 18:42:25 发布

hadoop - hdfs HA 请参考：dfs HA

服务规划：

hostname	ip	zookeeper	hdfs	journalnode	rm	zkfc
node03	192.168.0.3	master	datanode		*:8088
node04	192.168.0.4	follower	datanode	*:8485
node05	192.168.0.5	follower	datanode	*:8485
node06	192.168.0.6		datanode	*:8485	*:8088
node07	192.168.0.7		namenode:8082(rpc)/50070(http)			*
node08	192.168.0.8		namenode:8082/50070			*

yarn-env.sh

在yarn-env.sh末尾添加一下两句避免 java.lang 反射错误

export YARN_RESOURCEMANAGER_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"
export YARN_NODEMANAGER_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"

mapred-site.xml

<configuration>
        <!-- 指定 mr 框架为 yarn 方式 -->
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

</configuration>

yarn-site.xml

<configuration>

        <!-- 支持mapperreduce-->
	<property>
  		<name>yarn.nodemanager.aux-services</name>
  		<value>mapreduce_shuffle</value>
	</property>
        <!--开启高可用-->
	<property>
  		<name>yarn.resourcemanager.ha.enabled</name>
  		<value>true</value>
	</property>
	<!--集群名称，要和hdfs-site.xml中的集群名称相同-->
	<property>
  		<name>yarn.resourcemanager.cluster-id</name>
  		<value>mycluster</value>
	</property>

	<!--resourcemanager节点在集群中的名称-->
	<property>
  		<name>yarn.resourcemanager.ha.rm-ids</name>
  		<value>rm1,rm2</value>
	</property>
	<!--rm节点-->
	<property>
  		<name>yarn.resourcemanager.hostname.rm1</name>
  		<value>node03</value>
	</property>

	<property>
  		<name>yarn.resourcemanager.hostname.rm2</name>
  		<value>node06</value>
	</property>
        <!--rm对外的http地址-->
	<property>
  		<name>yarn.resourcemanager.webapp.address.rm1</name>
  		<value>node03:8088</value>
	</property>

	<property>
  		<name>yarn.resourcemanager.webapp.address.rm2</name>
  		<value>node06:8088</value>
	</property>

	<!--zookeeper集群-->
	<property>
 		 <name>yarn.resourcemanager.zk-address</name>
  		<value>node03:2181,node04:2181,node05:2181</value>
	</property>
        <!--开启自动恢复-->
	<property>
  		<name>yarn.resourcemanager.recovery.enabled</name>
  		<value>true</value>
	</property>
	<!--rm状态保存在zookeeper-->
	<property>
  		<name>yarn.resourcemanager.store.class</name>
  		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
	</property>

	<property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>86400</value>
	</property>

</configuration>

启动yarn

启动yarn前确保zookeeper和dfs正常启动

在其中一个rescouremanager节点执行如下命令：yarn-start.sh

顺利启动后执行jps：

↵