下载的包有如下几个:
CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel
CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel.sha
manifest.json
cloudera-manager-el6-cm5.7.4_x86_64.tar.gz
mysql-connector-java-5.1.38.jar
下载地址如下:
注意要将CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel.sha1改名为CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel.sha
安装准备(以下操作如果没特别指出,需要在所有的主机上操作)
设置静态ip,hosts
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 HWADDR=00:50:56:8F:B3:52 TYPE=Ethernet UUID=a95295f7-f373-4ce9-970a-dc873db3bec9 ONBOOT=yes NM_CONTROLLED=yes #BOOTPROTO=dhcp IPADDR=192.168.2.224 NETMASK=255.255.255.0 GATEWAY=192.168.2.226 |
vi /etc/sysconfig/network(以hadoop-1为例)
NETWORKING=yes HOSTNAME=hadoop-1 |
修改/etc/hosts 文件配置ip与host的映射
# vi /etc/hosts
添加如下数据:
192.168.2.224 hadoop-1 192.168.2.234 hadoop-2 192.168.2.235 hadoop-3 192.168.2.236 hadoop-4 192.168.2.244 hadoop-5 |
关闭防火墙
# service iptables stop
# chkconfig iptables off
禁用IPv6
# echo "alias net-pf-10 off" >> /etc/modprobe.d/dist.conf
# echo "alias ipv6 off" >> /etc/modprobe.d/dist.conf
禁止Selinux
# vi /etc/sysconfig/selinux
修改内容
SELINUX=disabled
ssh互信
# ssh-keygen -t rsa (一直回车就行)
分发公钥
# ssh-copy-id hadoop-1
# ssh-copy-id hadoop-2
# ssh-copy-id hadoop-3
# ssh-copy-id hadoop-4
# ssh-copy-id hadoop-5
集群时间同步
(如果没办法连接外网,可以先把自己的机器设置成外部ntp服务器,参考文档:"windows设置ntp服务",集群时间正确之后再选在hadoop-1作为集群的时间服务器)
1配置ntp服务
a) 开启 ntp 进程服务
# service ntpd start
# chkconfig ntpd on
b) 修改 ntpd 配置文件(hadoop-1)
# vi /etc/ntp.conf
1) 取消注释 保持同一网段
# Hosts on local network are less restricted.
restrict 172.168.2.0 mask 255.255.255.0 nomodify notrap
(注意 172.168.2.0 是集群所在的网段)
2) 注释服务
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
3) 添加配置
NTPD把本地主机的时钟也看作外部时钟源来处理,分配的地址是127.127.1.0
server 127.127.1.0
fudge 127.127.1.0 startum 10
c) 重启ntp 服务,并将服务设置为开机启动
# service ntpd restart
校准时间(hadoop-1)
1)系统时间
将本机时间与外部ntp服务器进行同步
# ntpdate -u 192.168.2.32 (已经作为ntp服务器的ip,或同网段其他机器)
date -s "2007-08-03 14:15:00"
2)硬件时钟时间
a.电脑系统暂停 系统时间暂停 硬件时钟时间继续执行
# hwclock --localtime
# hwclock --localtime -w (更改 用系统时间更新硬件时钟时间)
# hwclock --localtime
b.用硬件时钟时间更新系统时间
# vi /etc/sysconfig/ntpd
首行 添加
SYNC_HWCLOCK=yes
# vi /etc/sysconfig/ntpdate
SYNC_HWCLOCK=yes (no修改为yes)
2其他机器与集群的ntp服务器进行时间同步(除去hadoop-1外的机器)
a.修改配置文件
# vi /etc/ntp.conf
1) 取消注释 保持同一网段
# Hosts on local network are less restricted.
restrict 172.168.2.0 mask 255.255.255.0 nomodify notrap
(注意 172.168.2.0 是集群所在的网段)
2) 注释服务
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
3) 配置服务源,即hadoop-1的地址
server 192.168.2.224
分发配置:
scp /etc/ntp.conf root@hadoop-5:/etc/
scp /etc/ntp.conf root@hadoop-4:/etc/
scp /etc/ntp.conf root@hadoop-3:/etc/
4) 手动执行
# ntpdate -u hadoop-1
5)硬件时钟时间
a.电脑系统暂停 系统时间暂停 硬件时钟时间继续执行
# hwclock --localtime
# hwclock --localtime -w (更改 用系统时间更新硬件时钟时间)
# hwclock --localtime
b.用硬件时钟时间更新系统时间
# vi /etc/sysconfig/ntpd
首行 添加
SYNC_HWCLOCK=yes
# vi /etc/sysconfig/ntpdate
SYNC_HWCLOCK=yes (no修改为yes)
scp /etc/sysconfig/ntpd root@hadoop-5:/etc/sysconfig
scp /etc/sysconfig/ntpd root@hadoop-4:/etc/sysconfig
scp /etc/sysconfig/ntpd root@hadoop-3:/etc/sysconfig
6) 脚本定时执行
# crontab -e
0-59/3 * * * * /usr/sbin/ntpdate -u hadoop-1
7)开启 ntp 进程服务
# service ntpd start
# chkconfig ntpd on
安装JDK
- 上传 jdk-8u151-linux-x64.tar.gz
2)创建/usr/java目录并解压jdk:
# mkdir /usr/java
# tar -xvf jdk-8u151-linux-x64.tar.gz -C /usr/java/
scp -r jdk1.8.0_151/ root@hadoop-5:/usr/java/
3)配置JAVA环境变量:
# vi /etc/profile
添加如下内容:
export JAVA_HOME=/usr/java/jdk1.8.0_151
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
scp /etc/profile root@hadoop-2:/etc/
4)配置生效:
# source /etc/profile
5)验证是否安装成功:
# java -version
在hadoop-1安装mysql,并将字符集改成utf8
删除原有mysql
# rpm -qa |grep mysql //查看原有mysql
存在类似如下:
qt-mysql-4.6.2-26.el6_4.x86_64
mysql-connector-odbc-5.1.5r1144-7.el6.x86_64
mysql-libs-5.1.71-1.el6.x86_64
mysql-5.1.71-1.el6.x86_64
mysql-devel-5.1.71-1.el6.x86_64
mysql-server-5.1.71-1.el6.x86_64
//强制卸载原先系统自带的mysql包(按照列出的进行删除)
rpm -e mysql-libs --nodeps
rpm -e mysql-connector-odbc --nodeps
rpm -e mysql --nodeps
rpm -e qt-mysql --nodeps
rpm -e mysql-devel --nodeps
rpm -e mysql-server --nodeps
将mysql安装包上传并解压(5.5)
tar -zxvf mysql-5.5.52-linux2.6-x86_64.tar.gz
建立mysql用户组
groupadd mysql
为mysql用户组添加用户mysql
useradd -r -g mysql mysql
复制解压得到的目录到系统的本地软件目录“/usr/local/”,命令如下:
/usr/local/mysql 为本次mysql的安装目录
cp -r mysql-5.5.52-linux2.6-x86_64 /usr/local/mysql
--有效
cd /usr/local/mysql
ls --进入mysql目录,查看复制是否成功
修改当前目录拥有者为刚才新建的mysql用户
chown -R mysql:mysql ./
ls -al
加入mysql配置
vi /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
lower_case_table_names=1
[mysqld_safe]
log-error=/var/log/mysql.log
pid-file=/var/run/mysql/mysql.pid
初始化mysql
./scripts/mysql_install_db
为防有人攻破mysql进程,则不安全,所以将此目录下除data之外的所有文件拥有者改为root用户
chown -R root:root ./ ---将文件权限设置为root用户
chown -R mysql:mysql data ---data目录要使用mysql用户/组
ls -al
为系统增加mysql服务
cp support-files/mysql.server /etc/init.d/mysql
chkconfig --add mysql
chkconfig --list mysql
(看看默认是否会启用,3,4,5是“on”状态即可)
重启后,系统会自动开启mysql服务
启动mysql 服务
service mysql start
如失败继续执行以下命令:
./scripts/mysql_install_db
为mysql用户添加根目录
mkdir /home/mysql
chown -Rv mysql /home/mysql --设置目录mysql的所有权为mysql用户
chown -Rv mysql /home/mysql
changed ownership of `/home/mysql' to mysql
chgrp mysql /home/mysql
重启服务器,验证Mysql安装是否成功
使用service mysql status查看mysql服务是否开启
service mysql status
创建相关软连接
cd /usr/local/bin/
ln -fs /usr/local/mysql/bin/mysql mysql
ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock
cd /usr/local/mysql
./bin/mysqladmin -u root password '123456'
1.在数据库中建表(cm)
mysql -uroot -p123456
create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database hmon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database smon DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database rma DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database nas DEFAULT CHARSET utf8 COLLATE utf8_general_ci; create database nms DEFAULT CHARSET utf8 COLLATE utf8_general_ci; grant all on scm.* to 'root'@'%' identified by '123456'; grant all on hive.* to 'root'@'%' identified by '123456'; grant all on oozie.* to 'root'@'%' identified by '123456'; grant all on amon.* to 'root'@'%' identified by '123456'; grant all on hmon.* to 'root'@'%' identified by '123456'; grant all on smon.* to 'root'@'%' identified by '123456'; grant all on rma.* to 'root'@'%' identified by '123456'; grant all on nas.* to 'root'@'%' identified by '123456'; grant all on nms.* to 'root'@'%' identified by '123456'; flush privileges; |
2.设置mysql开机启动
# chkconfig mysql on
其他设置
# echo 0 > /proc/sys/vm/swappiness
# echo 'vm.swappiness=0'>> /etc/sysctl.conf
# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
# echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag' >> /etc/rc.local
安装目录
# mkdir /opt/cloudera-manager
将安装包解压到安装目录
cd /opt/cloudera-manager/
scp cloudera-manager-el6-cm5.7.4_x86_64.tar.gz root@hadoop-2:/opt/cloudera-manager/
# tar -xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager
创建CM的默认用户,增加sudo权限
# useradd --system --home=/opt/cloudera-manager/cm-5.7.4/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
# chmod +w /etc/sudoers
# vi /etc/sudoers
添加cloudera-scm ALL=(ALL) NOPASSWD:ALL
# chmod -w /etc/sudoers
创建Cloudera Manager Server 本地存储路径
# mkdir /var/lib/cloudera-scm-server
# chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
日志路径
# mkdir /var/log/cloudera-scm-server
# chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-server
修改配置文件
# vi /opt/cloudera-manager/cm-5.7.4/etc/cloudera-scm-agent/config.ini
server_host= hadoop-1
创建目录
# mkdir -p /var/cm_logs/cloudera-scm-headlamp
# chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp
创建Parcel 目录 并上传需要的parcel包
# mkdir -p /opt/cloudera/parcel-repo
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
# mkdir -p /opt/cloudera/parcels
# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
cd /opt/cloudera/parcel-repo
上传 CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel
CDH-5.7.4-1.cdh5.7.4.p0.2-el6.parcel.sha
manifest.json
将上述三个文件上传到/opt/cloudera/parcel-repo 目录下
scp * root@hadoop-2:/opt/cloudera/parcel-repo/
scp * root@hadoop-3:/opt/cloudera/parcel-repo/
scp * root@hadoop-4:/opt/cloudera/parcel-repo/
scp * root@hadoop-5:/opt/cloudera/parcel-repo/
在hadoop-1节点上配置Cloudera Manager Server 数据库(本集群采用mysql数据库)
建立数据库前要下载 jdbc驱动,放到下面目录(否则创建scm数据会报错):
/opt/cloudera-manager/cm-5.7.4/share/cmf/lib
为cloudera-manager创建数据库scm
执行
cd /opt/cloudera-manager/cm-5.7.4/share/cmf/lib
上传mysql
# /opt/cloudera-manager/cm-5.7.4/share/cmf/schema/scm_prepare_database.sh mysql scm root 123456
在hadoop-1节点上执行
# /opt/cloudera-manager/cm-5.7.4/etc/init.d/cloudera-scm-server start
设置开机启动
#cp /opt/cloudera-manager/cm-5.7.4/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
# chkconfig cloudera-scm-server on
修改
# vi /etc/init.d/cloudera-scm-server
将 CMF_DEFAULTS的值由 ${CMF_DEFAULTS:-/etc/default},设置为CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.4/etc/default
在每个主机上执行如下命令:
# /opt/cloudera-manager/cm-5.7.4/etc/init.d/cloudera-scm-agent start
设置开机启动:
# cp /opt/cloudera-manager/cm-5.7.4/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
# chkconfig cloudera-scm-agent on
修改
# vi /etc/init.d/cloudera-scm-agent
将 CMF_DEFAULTS的值由 ${CMF_DEFAULTS:-/etc/default},设置为CMF_DEFAULTS=/opt/cloudera-manager/cm-5.7.4/etc/default
scp /etc/init.d/cloudera-scm-agent root@hadoop-2:/etc/init.d/cloudera-scm-agent
至此cloudera-manager已经安装完毕
CDH服务安装
登录
http://192.168.2.224:7180 (192.168.2.224是hadoop-1的地址)
用户名/密码 : admin/admin
为CDH安装指定主机
选择所有当前管理的主机
验证必须全部通过才能继续下一步
安装CM
1:点击图标链接到如图的首页
2:点击添加cloudera management 的服务
3.服务全部安装到hadoop-1节点上
4:数据库主机名称:hadoop-1(测试不通过尝试localhost)
5:数据库类型:mysql
7:数据库名称:amon
8:用户名/密码:hadoop-1上的mysql的用户名和密码(root/123456)
9:测试链接通过后才能进行下一步
10:选择默认配置
注意:
如果出现类似如下问题,修改修改指定文件:
Error found before invoking supervisord: dictionary update sequence element #78 has length1; 2 is required
该错误为cm5.7的自带bug,解决方法是修改/opt/cloudera-manager/cm-5.7.4/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.4-py2.6.egg/cmf/util.py,将
pipe = subprocess.Popen(['/bin/bash', '-c', ". %s; %s; env" % (path, command)],
stdout=subprocess.PIPE, env=caller_env)
修改为:
pipe = subprocess.Popen(['/bin/bash', '-c', ". %s; %s; env | grep -v { | grep -v }" % (path, command)],
stdout=subprocess.PIPE, env=caller_env)
将该配置同步到其他机器即可。