系统环境:
四台虚拟主机
操作系统:CentOS 6(8核,16G,512G硬盘)
Cloudera Manager:5.13.3
CDH: 5.13.3
192.168.1.xx bigdata1
192.168.1.xx bigdata2
192.168.1.xx bigdata3
192.168.1.xx bigdata4
相关资源下载:
1.Cloudera Manager13.3下载地址:
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz
其他版本自行寻找:
http://archive.cloudera.com/cm5/cm/5/
2.CDH安装包地址:
http://archive.cloudera.com/cdh5/parcels/5.13.3.2/
同样对应操作系统下载:
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1
manifest.json
一、系统环境搭建(详细步骤略)
大概就是:
1.没改好静态IP的设置主机IP
vi /etc/sysconfig/network-scripts/ifcfg-eth0
里面大概长这样:
DEVICE=eth0
TYPE=Ethernet
UUID=1c81710f-f99e-4f6e-a3fa-7c8bb20b0xxx
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=00:0C:29:28:C5:52
IPADDR=192.168.x.xxx
PREFIX=24
GATEWAY=192.168.x.x
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
2.重启网卡
service network restart
3.修改主机名
vi /etc/sysconfig/network
4.关闭防火墙
chkconfig iptables off
service iptables stop
5.关闭SELinux(centOS需要)
vi /etc/selinux/config
#修改SELINUX=disabled
6.配置 /etc/hosts 把主机都填进去
vi /etc/hosts
192.168.1.xxx bigdata1
192.168.1.xxx bigdata2
192.168.1.xxx bigdata3
192.168.1.xxx bigdata4
7.重启机器
8.打通SSH免密码登录:
生成密钥:ssh-keygen -t rsa
复制秘钥到每台机器:ssh-copy-id -i <用户名>@<主机名>
8.安装配置JDK1.8+
解压到/usr/目录下,配置profile
二、主节点配置
1.在主节点安装配置mysql
2.登录mysql数据库
mysql -uroot -p密码
3.创建所需库
--创建hive数据库
create database hive DEFAULT CHARSET latin1;
--集群监控数据库
create database amon DEFAULT CHARSET utf8;
--hue数据库
create database hue DEFAULT CHARSET utf8;
--oozie数据库
create database oozie default charset utf8;
--授权
grant all privileges on hive.* to 'scm'@'%' identified by 'cdh' with grant option;
grant all privileges on amon.* to 'scm'@'%' identified by 'cdh' with grant option;
grant all privileges on hue.* to 'scm'@'%' identified by 'cdh' with grant option;
grant all privileges on oozie.* to 'scm'@'%' identified by 'cdh' with grant option;
--远程访问->
grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
flush privileges;
4.主节点解压安装cloudera-manager-el6-cm5.13.3_x86_64.tar.gz
cloudera manager的目录默认位置在/opt下,解压:
tar -zxvf cloudera-manager-*.tar.gz;
将解压后的目录放到/opt目录下。
5.为Cloudera Manager 5建立数据库
去MySql的官网下载JDBC驱动,找到mysql-connector-java-5.1.43-bin.jar,放到/opt/cm-5.13.3/share/cmf/lib/中
6.初始化cloudera manager数据库配置
/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh -h <mysql所在主机> mysql cm scm cdh
注意:上行命令更正为:
/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh -h <mysql所在主机> mysql -uroot -p123456 cm scm cdh
三、其他配置
1.在所有节点创建cloudera-scm用户
useradd --system --home=/opt/cm-5.13.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
2.修改/opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。
vim /opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini
3.同步Agent到其他节点
scp -r /opt/cm-5.13.3 bigdata1:/opt/
scp -r /opt/cm-5.13.3 bigdata2:/opt/
scp -r /opt/cm-5.13.3 bigdata3:/opt/
scp -r /opt/cm-5.13.3 bigdata4:/opt/
4.准备Parcels,用以安装CDH5
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中
相关的文件如下:
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1
manifest.json
最后将CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1,重命名为CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha,这点必须注意,否则,系统会重新下载文件。
5.相关启动脚本
/opt/cm-5.13.3/etc/init.d/cloudera-scm-server start
/opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start
<主节点都启动、其他节点只启动agent>
5.哦了,等一会打开浏览器
http://bigdata1:7180
按照步骤开始配置就行了