1、 主机分配
以七台测试虚拟机为例:
七台主机名分别为:cm、master1、master2、master3、node1、node2、node3
对应的ip为:192.168.138.141 ——192.168.138.147
2、修改静态IP
因为除cm外都是克隆虚拟机机,所以首先要解决克隆机的静态IP跟主机名更改问题,更改cm为例:
vi /etc/udev/rules.d/70-persistent-net.rules
删除eth0的驱动,把eth1
的name改为eth0,MAC记住
修改ip配置
vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:50:56:8B:4D:06
TYPE=Ethernet
UUID=5ff79caa-25f8-46a2-aee6-360cc73809fa
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.138.141
GATEWAY=192.168.138.254
NETMASK=255.255.255.0
IPV6INIT=no
DNS=8.8.8.8
标记的为改动添加的地方第一个HWADDR应改为之前记住的MAC号
重启机器(等下面修改主机名操作完成后一起重启验证)
ping www.baidu.com
测试网络连接以及静态IP是否配置成功
查看本机ip命令:ifconfig
3、修改主机名
vi /etc/sysconfig/network
主机名修改为cm
重启机器后验证指令:hostname
4、 添加SSH免密登陆
在添加之前要逐个关闭各主机上面的防火墙先执行永久关闭再执行临时关闭,因为永久关闭不是即时生效,所以需要再次执行临时关闭,然后逐个主机查看状态是否关闭
永久关闭防火墙
Centos6:chkconfig iptables off
Centos7:systemctl disable firewalld.service
关闭防火墙
Centos6:[root@localhost ~]# chkconfig iptables off
Centos7:[root@localhost ~]# systemctl stop firewalld
查防火墙状态
Centos6: [root@localhost ~]# service iptables status
service iptables stop
Centos7: [root@localhost ~]# systemctl status firewalld.service
如果显示状态不是iptables: Firewall is not running.则需要关闭防火墙
4.1、生成公钥
在七台主机下面逐个执行指令:ssh-keygen -t rsa //回车键到结束
在/root/.ssh目录下查看是否生成公钥文件
cd /root/.ssh/
ls
id_rsa id_rsa.pub known_hosts
cat id_rsa.pub >> authorized_keys
ls
authorized_keys id_rsa id_rsa.pub known_hosts
生成公钥集合文件authorized_keys
分发到各子机
scp authorized_keys root@master1:/root/.ssh/
5、 禁用selinux
重启机器才会生效,这里先不重启
vim /etc/sysconfig/selinux
修改配置项:
SELINUX=disabled
并调用scp命令分发到其它主机
6、修改hosts文件
vi /etc/hosts
追加
并调用scp命令分发到其它主机
7、修改主机时间
查看当前时区
date -R;cat /etc/sysconfig/clock
设置东八区时区为当前时区
rm -rf /etc/localtime
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
vim /etc/sysconfig/clock
ZONE="Asia/Shanghai"
** 手动同步下网络时间**
ntpdate -u cn.pool.ntp.org
8、卸载centOS自带OpenJDK
查看自带JDK
rpm -qa|grep java
xxx yyy zzz为你要卸载的插件,插件之间以空格隔开
卸载JDK
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.38-1.13.10.4.el6.x86_64 java-1.7.0-openjdk-1.7.0.99-2.6.5.1.el6.x86_64
9、关闭透明大页
临时关闭透明大页面(立即生效)
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
永久关闭透明大页面(重启生效)
echo ' ' >> /etc/rc.local
echo '# 关闭大透明页面' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled' >> /etc/rc.local
查看是否已经关闭
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
10、修改swappiness
临时修改
sysctl -w vm.swappiness=1
永久修改
echo "vm.swappiness=1" >> /etc/sysctl.conf
然后重启服务器验证之前修改
检查永久生效配置是否生效
11、配置JDK1.8环境
JDK配置调置环境变量,编辑文件/etc/profile添加如下内容
export JAVA_HOME=/opt/jdk1.8
export PATH=$JAVA_HOME/bin:$PATH
拷贝JAVA文件与profile文件至其它节点
scp -r jdk1.8/ root@node4:/opt/
scp /etc/profile root@node1:/etc/
设置全局变量并生效,在五台机器上均执行如下命令
source /etc/profile
echo "JAVA_HOME=/opt/jdk1.8" >> /etc/environment
检查JAVA是否安装成功,执行命令java -version出现如下结果
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)
12、安装Mysql
下载Mysql,官方网址:https://www.mysql.com/
下载:https://dev.mysql.com/downloads/mysql/5.6.html#downloads
mkdir -p /opt/software/mysql
解压tar包
tar -xvf MySQL-5.6.43-1.el6.x86_64.rpm-bundle.tar -C /opt/software/mysql/
添加执行权限
chmod u+x *.rpm
先查看一下是否有系统自带的mysql插件,我这里是有的
rpm -qa | grep -i mysql
先安装MySQL-shared-compat
rpm -ivh MySQL-shared-compat-5.6.43-1.el6.x86_64.rpm
再卸载mysql-libs
yum remove mysql-libs
安装MySQL-server
rpm -ivh MySQL-server-5.6.43-1.el6.x86_64.rpm
安装MySQL-client
rpm -ivh MySQL-client-5.6.43-1.el6.x86_64.rpm
启动Mysql
service mysql start
不用再配置开机启动了,安装包已经设置Mysql开机启动
查看Mysql密码
more ~/.mysql_secret
##```
java version “1.8.0_112”
Java™ SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot™ 64-Bit Server VM (build 25.112-b15, mixed mode)
/usr/bin/mysql_secure_installation --user=mysql
## 开启远程登陆
### 命令行登陆Mysql
mysql -uroot -p111111
## 添加远程登陆权限
Grant all privileges on . to ‘root’@’%’ identified by ‘111111’ with grant option;
## 重新加载权限
flush privileges;
创建数据库及用户并添加权限
其他的数据库实例使用uft8编码,hive和oozie使用默认的编码方式
如下,可直接复制,后面会用到。
```sql
create database if not exists amon default charset utf8 collate utf8_general_ci;
create database if not exists rman default charset utf8 collate utf8_general_ci;
create database if not exists nav default charset utf8 collate utf8_general_ci;
create database if not exists navms default charset utf8 collate utf8_general_ci;
create database if not exists hue default charset utf8 collate utf8_general_ci;
create database if not exists sentry default charset utf8 collate utf8_general_ci;
create database if not exists hive;
create database if not exists oozie;
grant all on hive.* to 'hive'@'%' identified by 'hive' with grant option;
grant all on oozie.* to 'oozie'@'%' identified by 'oozie' with grant option;
grant all on hue.* to 'hue'@'%' identified by 'hue' with grant option;
grant all on amon.* to 'amon'@'%' identified by 'amon' with grant option;
grant all on rman.* to 'rman'@'%' identified by 'rman' with grant option;
grant all on nav.* to 'nav'@'%' identified by 'nav' with grant option;
grant all on navms.* to 'navms'@'%' identified by 'navms' with grant option;
grant all on sentry.* to 'sentry'@'%' identified by 'sentry' with grant option;
flush privileges;
为Cloudera Manager配置Mysql
为Cloudera Manager配置外部的数据库需要对此外部数据库进行一些配置
ClouderaManager官网对Mysql配置的翻译,
大致意思是:
1) 需要配置Mysql的搜索引擎为InnoDB
2) Mysql安装好以后默认分配的缓冲区和内存比较小,ClouderaManager会非常频繁地连接Mysql需要配置一下,具体根据你集群的大小配置最大连接数
3) 二进制日志对于ClouderaManager不是必须的,但是对Mysql有好处,这个你随意
然后官网在下面给了一个my.cnf配置的例子,然后说只要你的大数据集群在50个节点以内都可以使用下面的这个配置
查找my.cnf
my.cnf路径一般是/etc/my.cnf没有的话创建,其他地方有的话就修改。
find / -name my.cnf
我这里在/usr/下
修改/usr/my.cnf文件
[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
# For MySQL version 5.1.8 or later. For older versions, reference MySQL documentation for configuration help.
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
重启Mysql服务
service mysql restart
13、配置MYSQLJDBC驱动包
拷贝驱动包
cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar
分发驱动包
scp /usr/share/java/mysql-connector-java.jar root@node1:/usr/share/java/
14、配置集群时间同步NTP服务
修改cm主机ntp.conf
注释所有server
添加如下配置
server 127.127.1.0
fudge 127.127.1.0 stratum 10
进入其他集群节点修改ntp配置文件
注释所有server 添加一条server改为cm节点的IP
15、CM安装
解压cm包
创建以下三个文件夹
mkdir -p /var/lib/cloudera-scm-server
mkdir -p /var/lib/cloudera-scm-agent
mkdir -p /var/lib/cloudera-scm-server-db/data
MySQL驱动复制到/opt/cm-5.12.1/share/cmf/lib中
cp /usr/share/java/mysql-connector-java.jar /opt/cm-5.11.2/share/cmf/lib/
vi /opt/cm-5.11.2/etc/cloudera-scm-agent/config.ini
修改/opt/cm-5.12.1/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。
将Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中(如没有parcel-repo,需要手动创建)
[root@cm tar]# mv CDH-5.11.2-1.cdh5.11.2.p0.4-el6.parcel manifest.json /opt/cloudera/parcel-repo/
[root@cm tar]# mv CDH-5.11.2-1.cdh5.11.2.p0.4-el6.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.11.2-1.cdh5.11.2.p0.4-el6.parcel.sha
创建cloudera-scm用户(每个节点都要创建)
useradd --system --home=/opt/cm-5.11.2/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
初始化CM的元数据库
/opt/cm-5.11.2/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p111111 --scm-host localhost scm scm scm
主节点启动CM
/opt/cm-5.11.2/etc/init.d/cloudera-scm-server start
agent启动CM
/opt/cm-5.11.2/etc/init.d/cloudera-scm-agent start
登陆192.168.138.141:7180访问 用户名密码admin
没有启动监控一下7180端口:
netstat -apn|grep 7180
16、进入角色分配安装步骤
角色分配参考角色分配文档
17、开启HDFS高可用模式
开启两个HDFS节点