Hadoop3.1.0完全分布式集群部署超详细记录-JobPlus

Hadoop3.1.0完全分布式集群部署，三台服务器部署结构如下github配置文件源码地址

#部署完成后

root@servera:/opt/hadoop/hadoop-3.1.0# jps

14056 SecondaryNameNode

14633 Jps

13706 NameNode

14317 ResourceManager

root@serverb:~# jps

5288 NodeManager

5162 DataNode

5421 Jps

root@serverc:~# jps

4545 NodeManager

4371 DataNode

4678 Jps

如上图，一共三台机器作为集群，servera作为master，其他两台作为worker。

2.开始部署-前期准备(三台机器都需要进行如下操作)

2.1.配置hosts文件【三台】

vim /etc/hosts

10.80.80.110 servera

10.80.80.111 serverb

10.80.80.112 serverc

2.2.jdk 安装【三台】
查看 Javajava --version
- 下载jdk
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz

- 解压
- mkdir /opt/java
- wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz
- tar -zxf jdk-8u172-linux-x64.tar.gz
- mv jdk1.8.0_172/ /opt/java/

配置JAVA变量

/vim /etc/profile.d/jdk-1.8.sh
#!/bin/sh
# Author:wangxiaolei 王小雷
# Blog: http://blog.csdn.net/dream_an
# Github: https://github.com/wangxiaoleiai
# web: www.xiaolei.wang
# Date: 2018.05
# Path: /etc/profile.d/
export JAVA_HOME=/opt/java/jdk1.8.0_172
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
# 使环境变量生效
source /etc/profile
# 查看
Javajava --version

2.3.pdsh、ssh安装【三台】

root@servera:~# apt install ssh pdsh

echo ssh>/etc/pdsh/rcmd_default

2.4.免密码登录自身【三台】

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

$ chmod 0600 ~/.ssh/authorized_keys

ssh localhost(首次需输入yes)

2.5.servera免密码登录其他机器（master免密码登录worker）【单台，只需在servera上执行】

ssh-copy-id -i ~/.ssh/id_rsa.pub servera

ssh-copy-id -i ~/.ssh/id_rsa.pub serverb

ssh-copy-id -i ~/.ssh/id_rsa.pub serverc

3.hadoop3+配置文件

共需要配置/opt/hadoop/hadoop-3.1.0/etc/hadoop/下的六个个文件，分别是

hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、workers

3.1. hadoop-env.sh 添加如下内容

export JAVA_HOME=/opt/java/jdk1.8.0_172/

export HDFS_NAMENODE_USER="root"

export HDFS_DATANODE_USER="root"

export HDFS_SECONDARYNAMENODE_USER="root"

export YARN_RESOURCEMANAGER_USER="root"

export YARN_NODEMANAGER_USER="root"

3.2. core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://ruizhia:9000</value>

</property>

<name>io.file.buffer.size</name>

</property>

</configuration>

3.3. hdfs-site.xml

<name>dfs.namenode.name.dir</name>

<value>/var/lib/hadoop/hdfs/name/</value>

</property>

<name>dfs.blocksize</name>

</property>

<name>dfs.namenode.handler.count </name>

</property>

<name>dfs.datanode.data.dir</name>

<value>/var/lib/hadoop/hdfs/data/</value>

</property>

<name>dfs.replication</name>

</property>

</configuration>

3.4. yarn-site.xml

<name>yarn.resourcemanager.hostname</name>

<value>servera</value>

</property>

<!-- <property>

<name>yarn.resourcemanager.webapp.address</name>

</property> -->

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

3.5. mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

3.6. workers

serverb

serverc

4. 复制Hadoop文件到其他集群、配置Hadoop环境变量、格式化hdfs、开启集群、查看、关闭、重置集群

4.1. 将步骤3配置好的hadoop文件复制到其他同样位置的机器上
/opt/hadoop/hadoop-3.1.0
4.2 配置Hadoop环境变量【三台机器都操作】

vim /etc/profile.d/hadoop-3.1.0.sh

#!/bin/sh

# Author:wangxiaolei 王小雷

# Blog: http://blog.csdn.net/dream_an

# Github: https://github.com/wangxiaoleiai

# Date: 201805

# web: www.xiaolei.wang

# Path: /etc/profile.d/

export HADOOP_HOME="/opt/hadoop/hadoop-3.1.0"

export PATH="$HADOOP_HOME/bin:$PATH"

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

source /etc/profile

4.3. 格式化HDFS [只有首次部署才可使用]【谨慎操作，只在servera上操作】

/opt/hadoop/hadoop-3.1.0/bin/hdfs namenode -format myClusterName

4.4. 开启【只在servera上操作】

/opt/hadoop/hadoop-3.1.0/sbin/start-dfs.sh

/opt/hadoop/hadoop-3.1.0/sbin/start-yarn.sh

4.5. 查看【三台】

jps

4.6. web端localhost:8088查看【localhost只定servera的localhost，也可以换成外网ip，在详见步骤3.4. yarn-site.xml 】

4.7. 关闭【只在servera上操作】

/opt/hadoop/hadoop-3.1.0/sbin/stop-dfs.sh

/opt/hadoop/hadoop-3.1.0/sbin/stop-yarn.sh

4.8. 重置hadoop环境 [移除hadoop hdfs log文件] 【谨慎操作，只在servera上操作】

rm -rf /opt/hadoop/hadoop-3.1.0/logs/*

rm -rf /var/lib/hadoop/

5.遇到的坑 pdsh@servera: servera: connect: Connection refused

root@servera:/opt/hadoop/hadoop-3.1.0# sbin/start-dfs.sh
Starting namenodes on [servera]
pdsh@servera: servera: connect: Connection refused
Starting datanodes
pdsh@servera: serverc: connect: Connection refused
pdsh@servera: serverb: connect: Connection refused
Starting secondary namenodes [servera]
pdsh@servera: servera: connect: Connection refused

解决方法步骤2.3中

echo ssh>/etc/pdsh/rcmd_default

<h2>Hadoop3.1.0完全分布式集群部署，三台服务器部署结构如下<a href="https://github.com/wangxiaoleiAI/CSDN-CODE/tree/master/Hadoop-3.1.0-Fully-Distributed-Operation" target="_self">github配置文件源码地址</a></h2><img src="https://file.jobplus.com.cn/2018/05/10/70c7f820be2a472fbec4da1eea4a4f98.png" _src="https://file.jobplus.com.cn/2018/05/10/70c7f820be2a472fbec4da1eea4a4f98.png"/>#部署完成后root@servera:/opt/hadoop/hadoop-3.1.0# jps14056 SecondaryNameNode14633 Jps13706 NameNode14317 ResourceManagerroot@serverb:~# jps5288 NodeManager5162 DataNode5421 Jpsroot@serverc:~# jps4545 NodeManager4371 DataNode4678 Jps<ul><li> </li></ul>如上图，一共三台机器作为集群，servera作为master，其他两台作为worker。<h2>2.开始部署-前期准备(三台机器都需要进行如下操作)</h2><ul><li>2.1.配置hosts文件【三台】</li></ul>vim /etc/hosts10.80.80.110    servera10.80.80.111    serverb10.80.80.112    serverc<ul><li> </li></ul><ul><li>2.2.jdk 安装【三台】  查看 Javajava --version</li><li> </li><li><ul><li>下载jdk</li></ul>wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz</li><li> </li><ul><li><ul><li>解压</li><li>mkdir /opt/java</li><li>wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.tar.gz</li><li> tar -zxf jdk-8u172-linux-x64.tar.gz</li><li>mv jdk1.8.0_172/ /opt/java/</li><li> </li></ul></li></ul><ul><li>配置JAVA变量</li></ul><ul><li> /vim /etc/profile.d/jdk-1.8.sh</li><li> </li><li>#!/bin/sh</li><li># Author:wangxiaolei 王小雷</li><li># Blog: http://blog.csdn.net/dream_an</li><li># Github: https://github.com/wangxiaoleiai</li><li># web: www.xiaolei.wang</li><li># Date: 2018.05</li><li># Path: /etc/profile.d/</li><li>export JAVA_HOME=/opt/java/jdk1.8.0_172</li><li>export JRE_HOME=${JAVA_HOME}/jre</li><li>export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib</li><li>export PATH=${JAVA_HOME}/bin:$PATH</li><li> </li><li># 使环境变量生效</li><li>source /etc/profile</li><li># 查看</li><li> Javajava --version</li></ul><ul><li> </li></ul></ul><img src="https://file.jobplus.com.cn/2018/05/10/0e246da338cb4003b7d340c4a8a17d18.png" _src="https://file.jobplus.com.cn/2018/05/10/0e246da338cb4003b7d340c4a8a17d18.png"/><ul><li>2.3.pdsh、ssh安装【三台】</li></ul>root@servera:~# apt install ssh pdsh<ul><li> </li></ul>echo ssh>/etc/pdsh/rcmd_default<ul><li> </li></ul><ul><li>2.4.免密码登录自身【三台】</li></ul>$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys$ chmod 0600 ~/.ssh/authorized_keys ssh localhost(首次需输入yes)<ul><li> </li></ul><ul><li>2.5.servera免密码登录其他机器（master免密码登录worker）【单台，只需在servera上执行】</li></ul>ssh-copy-id -i ~/.ssh/id_rsa.pub servera ssh-copy-id -i ~/.ssh/id_rsa.pub serverb ssh-copy-id -i ~/.ssh/id_rsa.pub serverc<ul><li> </li></ul><h2>3.hadoop3+配置文件</h2>共需要配置/opt/hadoop/hadoop-3.1.0/etc/hadoop/下的六个个文件，分别是hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、workers<ul><li>3.1. hadoop-env.sh 添加如下内容</li></ul>export JAVA_HOME=/opt/java/jdk1.8.0_172/export HDFS_NAMENODE_USER="root"export HDFS_DATANODE_USER="root"export HDFS_SECONDARYNAMENODE_USER="root"export YARN_RESOURCEMANAGER_USER="root"export YARN_NODEMANAGER_USER="root"<ul><li> </li></ul><ul><li>3.2. core-site.xml</li></ul><configuration>    <property>      <name>fs.defaultFS</name>      <value>hdfs://ruizhia:9000</value>  </property>  <property>      <name>io.file.buffer.size</name>      <value>131072</value>  </property></configuration><ul><li> </li></ul><ul><li>3.3. hdfs-site.xml</li></ul><configuration><property>  <name>dfs.namenode.name.dir</name>  <value>/var/lib/hadoop/hdfs/name/</value></property><property>  <name>dfs.blocksize</name>  <value>268435456</value></property><property>  <name>dfs.namenode.handler.count  </name>  <value>100</value></property><property>  <name>dfs.datanode.data.dir</name>  <value>/var/lib/hadoop/hdfs/data/</value></property><property>    <name>dfs.replication</name>    <value>1</value></property></configuration><ul><li> </li></ul><ul><li>3.4. yarn-site.xml</li></ul><configuration>  <property>          <name>yarn.resourcemanager.hostname</name>          <value>servera</value>  </property>      <property>          <name>yarn.nodemanager.aux-services</name>          <value>mapreduce_shuffle</value>  </property></configuration><ul><li> </li></ul><ul><li>3.5. mapred-site.xml</li></ul><configuration>    <property>       <name>mapreduce.framework.name</name>       <value>yarn</value>   </property></configuration><ul><li> </li></ul><ul><li>3.6. workers</li></ul>serverb serverc<ul><li> </li></ul><h2>4. 复制Hadoop文件到其他集群、配置Hadoop环境变量、格式化hdfs、开启集群、查看、关闭、重置集群</h2><ul><li>4.1. 将步骤3配置好的hadoop文件复制到其他同样位置的机器上  /opt/hadoop/hadoop-3.1.0</li><li>4.2 配置Hadoop环境变量【三台机器都操作】</li></ul>vim /etc/profile.d/hadoop-3.1.0.sh<ul><li> </li></ul>#!/bin/sh# Author:wangxiaolei 王小雷# Blog: http://blog.csdn.net/dream_an# Github: https://github.com/wangxiaoleiai# Date: 201805# web: www.xiaolei.wang# Path: /etc/profile.d/ export HADOOP_HOME="/opt/hadoop/hadoop-3.1.0"export PATH="$HADOOP_HOME/bin:$PATH"export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop <ul><li> </li></ul>source /etc/profile<ul><li> </li></ul><ul><li>4.3. 格式化HDFS [只有首次部署才可使用]【谨慎操作，只在servera上操作】</li></ul>/opt/hadoop/hadoop-3.1.0/bin/hdfs namenode -format myClusterName<ul><li> </li></ul><ul><li>4.4. 开启 【只在servera上操作】</li></ul>/opt/hadoop/hadoop-3.1.0/sbin/start-dfs.sh /opt/hadoop/hadoop-3.1.0/sbin/start-yarn.sh<ul><li> </li></ul><ul><li>4.5. 查看 【三台】</li></ul>jps<ul><li> </li></ul><img src="https://file.jobplus.com.cn/2018/05/10/e4e1ca6fb79f4f70adc540f23f1dad46.png" _src="https://file.jobplus.com.cn/2018/05/10/e4e1ca6fb79f4f70adc540f23f1dad46.png"/><ul><li>4.6. web端localhost:8088查看【localhost只定servera的localhost，也可以换成外网ip，在详见步骤3.4. yarn-site.xml 】</li></ul><img src="https://file.jobplus.com.cn/2018/05/10/da2cb88568084445bf13f79e50bdf694.png" _src="https://file.jobplus.com.cn/2018/05/10/da2cb88568084445bf13f79e50bdf694.png"/><ul><li>4.7. 关闭 【只在servera上操作】</li></ul>/opt/hadoop/hadoop-3.1.0/sbin/stop-dfs.sh /opt/hadoop/hadoop-3.1.0/sbin/stop-yarn.sh<ul><li> </li></ul><ul><li>4.8. 重置hadoop环境 [移除hadoop hdfs log文件] 【谨慎操作，只在servera上操作】</li></ul>rm -rf /opt/hadoop/hadoop-3.1.0/logs/* rm -rf /var/lib/hadoop/<ul><li> </li></ul><h2>5.遇到的坑 pdsh@servera: servera: connect: Connection refused</h2>root@servera:/opt/hadoop/hadoop-3.1.0# sbin/start-dfs.sh  Starting namenodes on [servera]  pdsh@servera: servera: connect: Connection refused  Starting datanodes  pdsh@servera: serverc: connect: Connection refused  pdsh@servera: serverb: connect: Connection refused  Starting secondary namenodes [servera]  pdsh@servera: servera: connect: Connection refused<img src="https://file.jobplus.com.cn/2018/05/10/b98451b4763847709c5137705cc26b78.png" _src="https://file.jobplus.com.cn/2018/05/10/b98451b4763847709c5137705cc26b78.png"/><ul><li>解决方法步骤2.3中</li></ul>echo ssh>/etc/pdsh/rcmd_default