版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u012848709/article/details/85135787
楔子
学习 hadoop(基于hadoop3.1.1),学习视频可以参看大数据第二阶段:hadoop体系之离线计算
1 前期准备
1.1 hosts修改
虚拟机4 台 ,分配如下
NN-1 | NN-2 | DN | ZK | ZKFC | JNN | |
---|---|---|---|---|---|---|
had1 | * | * | * | |||
had2 | * | * | * | * | * | |
had3 | * | * | * | |||
had4 | * | * |
整个过程使用的用户为root。
1.2 安装java,配置HADOOP_HOME环境变量
为了方便,我配置了
HADOOP_HOME
到/etc/profile
export HADOOP_HOME=/opt/hadoop-3.1.1
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
2分布式搭建
2.1配置文件
hadoop-env.sh
末尾添加如下内容
export JAVA_HOME=/usr/local/soft/jdk
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://had1:9820</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/data/hadoop</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<!--文件副本数量 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>had2:9868</value>
</property>
</configuration>
workers
此文件中需要配置datanode节点
had2
had3
had4
2.2 格式化namenode,启动
-------格式化namenode----------------------------------
[root@had1 hadoop-3.1.1]# hdfs namenode -format
-------启动--------------------------------------------
[root@had1 hadoop-3.1.1]# start-dfs.sh
Starting namenodes on [had1]
Starting datanodes
had2: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
had4: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
had3: WARNING: /opt/hadoop-3.1.1/logs does not exist. Creating.
Starting secondary namenodes [had2]
2018-12-20 06:25:01,882 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@had1 hadoop-3.1.1]#
2.3查看进程
jps
查看进程