在笔记本电脑安装学习环境,采用all in one的伪分布式,所有都在一台电脑部署。
1.下载 hadoop
wget http://archive-primary.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.16.0.tar.gz
2. 解压 tar -zxvf hadoop-2.6.0-cdh5.16.0.tar.gz
3. 修改环境变量,加入hadoop 和 路径
nano /etc/profile 在文件后面加入如下
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
export HADOOP_HOME=/root/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin
其中如果不知道java的路径,可以用如下方法定位
source /etc/profile 或重启电脑
4. 修改 /etc/hosts
加入 192.168.0.101 centos7 根据自己的主机名称和ip修改
5.设置SSH免密码登录
ssh-keygen -t rsa
cd .ssh
cp id_rsa.pub authorized_keys
6.关闭防火墙
systemctl stop firewalld
7. 修改相关hadoop配置文件
如下5个文件
hadoop-env.sh
yarn-site.xml
mapred-site.xml
hdfs-site.xml
core-site.xml
(1)hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
(2)yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
(3)mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
(4)hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/root/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/root/hadoop/tmp/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
(5) core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://centos7:8020</value>
</property>
</configuration>
8. format hdfs
#hadoop namenode -format
9.到此hadoop配置完成,启动测试
[root@centos7 hadoop]#start-dfs.sh
[root@centos7 hadoop]# start-yarn.sh
[root@centos7 hadoop]# jps
8673 DataNode
9378 ResourceManager
7907 SecondaryNameNode
8549 NameNode
9685 Jps
9502 NodeManager
输入jps 显示如上说明hadoop启动成功
may be must install jps
yum install java-1.8.0-openjdk-devel.x86_64
9. 可以测试hadoop命令了
hadoop fs -ls /
hadoop fs -mkdir /test
hadoop fs -put hello.txt /test //发送当前目录下的hello.txt到hdfs
hadoop fs -txt /test/hello.txt //等于 cat hello.txt
10.可以通过浏览器查看
http://centos7:50070
http://centos7:8088