Hadoop 全分布环境搭建

首先编辑 hosts文件

vi /etc/hosts

根据你的集群架构进行配置

192.168.101.11 bigdata11
192.168.101.12 bigdata12
192.168.101.13 bigdata13
192.168.101.14 bigdata14
192.168.101.15 bigdata15

参照之前文章在bigdata15上安装hadoop

参见之前文章将每台机器的公钥部署在其他机器上

需要在每一台机器上执行如下命令

ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata11
ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata12
ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata13
ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata14
ssh-copy-id -i /root/.ssh/id_rsa.pub root@bigdata15

进入bigdata15 hadoop配置目录

cd /etc/hadoop/etc/hadoop/

vi hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>3</value>
</property>

将每台机器的冗余设为3

core-site.xml 与伪分布一致如下(NameNode地址)

<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://bigdata15:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/hdfs/tmp</value>
</property>
</configuration>

注意在每台机器上创建该目录
mkdir /hdfs
mkdir /hdfs/tmp

mapred-site.xml 与伪分布一致

创建该文件
cp mapred-site.xml.template mapred-site.xml

编辑内容如下

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

yarn-site.mxl 需更改yarn.resourcemanager.hostname的值为主节点地址

<configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>bigdata15</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
</configuration>

编辑slaves

vi slaves

bigdata11
bigdata12
bigdata13
bigdata14

格式化

hdfs namenode -format

复制配置文件至其他主机

scp -r /etc/hadoop/ root@bigdata11:/etc/
scp -r /etc/hadoop/ root@bigdata12:/etc/
scp -r /etc/hadoop/ root@bigdata13:/etc/
scp -r /etc/hadoop/ root@bigdata14:/etc/

注意更改防火墙，也可以使用远程拷贝

scp -r /etc/sysconfig/iptables bigdata11:/etc/sysconfig/iptables
scp -r /etc/sysconfig/iptables bigdata12:/etc/sysconfig/iptables
scp -r /etc/sysconfig/iptables bigdata13:/etc/sysconfig/iptables
scp -r /etc/sysconfig/iptables bigdata14:/etc/sysconfig/iptables

搞定启动服务

start-all.sh

查看集群状态

hadoop dfsadmin -report

Hadoop 全分布环境搭建

猜你喜欢