cent0S 7 安装 cdh5.13 笔记之三:
安装 hadoop-hdfs
------------------------------------------------------------------------ pkgname | n1 | n2 | n3 | pkgtype ------------------------------------------------------------------------ #### common #### net-tools (ALL) gcc (ALL) perl (ALL) jdk-8u152-linux-x64.tar.gz (ALL) zookeeper-server Y Y Y yum cyrus-sasl ? ? ? yum cyrus-sasl-devel ? ? ? yum kafka_2.11-1.0.0.tgz Y Y Y #### mysql #### perl Y yum mysql-community-common-5.7.20-1.el7.x86_64.rpm Y rpm mysql-community-libs-5.7.20-1.el7.x86_64.rpm Y rpm mysql-community-client-5.7.20-1.el7.x86_64.rpm Y rpm mysql-community-server-5.7.20-1.el7.x86_64.rpm Y rpm #### hadoop #### hadoop-hdfs-namenode Y Y N yum hadoop-hdfs-secondarynamenode* X X X yum (不要安装) hadoop-hdfs-datanode Y Y Y + yum hadoop-hdfs-journalnode Y Y Y yum (奇数: 3,5,7,..., 资源消耗低) hadoop-mapreduce Y Y Y yum hadoop-hdfs-zkfc Y Y N yum hadoop-yarn-resourcemanager Y Y N yum (最好单独一台) hadoop-yarn-nodemanager Y Y Y + yum hadoop-lzo Y Y Y + yum hadoop-mapreduce-historyserver Y N N yum hadoop-libhdfs Y Y N yum #### hbase #### hbase-master Y Y N yum hbase-regionserver Y Y Y yum #### hive #### hive-metastore Y N N yum hive-server2 Y Y Y yum hive-hbase Y Y Y yum hive-hcatalog Y Y Y yum hive-webhcat Y Y Y yum hive-webhcat-server Y N N yum #### spark #### spark-master Y Y N yum spark-worker Y Y Y yum spark-history-server Y N N yum ------------------------------------------------------------------------
10 安装hadoop (cdh-5.13)组件
如上表,对pkgtype为yum的执行下面的命令:
yum install -y pkgname
例如, 对于n1, n2执行(Y - 安装, N - 不安装):
yum install -y hadoop-hdfs-namenode hadoop-hdfs-datanode
对于n3,执行:
yum install -y hadoop-hdfs-datanode
这样就安装好了。重点是配置的过程。
11 安装mysql-server for hive
hive remote模式:
hive-metastore 安装在n1上, 需要在n3上安装mysql-server,下面是安装mysql的步骤:
1) 下载安装包
wget ftp://pub:[email protected]/rpm/mysql-community-common-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-libs-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-client-5.7.20-1.el7.x86_64.rpm wget ftp://pub:[email protected]/rpm/mysql-community-server-5.7.20-1.el7.x86_64.rpm
2) 安装mysql-server
a.首先卸载mariadb
# rpm -qa |grep mariadb mariadb-libs-5.5.56-2.el7.x86_64 # rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
b. 安装mysql-server
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm # yum install -y perl # rpm -ivh mysql-community-server-5.7.20-1.el7.x86_64.rpm
c. 启动 mysql
# service mysqld start
或
# systemctl start mysqld
d. 查看安装过程的临时密码:
# sudo grep 'temporary password' /var/log/mysqld.log
YVo)b8Er=g<g
d. 更改临时密码
# sudo mysql_secure_installation
New password:
Abc1234!
e. 更改mysql数据库文件路径
查看配置文件路径
# mysql --help|grep my.cnf
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
查看默认数据文件路径:
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
| datadir | /var/lib/mysql/
修改到:/data/mysql
# systemctl stop mysqld.service # mv /var/lib/mysql /data/
编辑: /etc/my.cnf, 内容如下:
[client] port = 3306 socket = /data/mysql/mysql.sock #### (必须) [mysqld] port = 3306 datadir=/data/mysql socket=/data/mysql/mysql.sock
再启动mysqld:
# systemctl start mysqld
再次查看数据文件路径:
# mysqladmin -u root -p'Abc1234!' variables | grep datadir
| datadir | /data/mysql/
这样就把数据库文件改到/data/mysql下面了。
最后设置mysql开机启动(待测试):
# systemctl enable mysqld.service
3) 安装mysql-client
在需要访问mysql服务的机器上安装mysql客户端, 这里是在n1上安装:
首先卸载mariadb
# rpm -qa |grep mariadb mariadb-libs-5.5.56-2.el7.x86_64 # rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
安装mysql-client
# rpm -ivh mysql-community-common-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-libs-5.7.20-1.el7.x86_64.rpm # rpm -ivh mysql-community-client-5.7.20-1.el7.x86_64.rpm
4) 在n1 上访问 n3的mysql
# mysql -u root -pAbc1234!mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1130 (HY000): Host 'cent7-n1.pepstack.com' is not allowed to connect to this MySQL server
需要在n3上设置mysql可以远程访问, 回到n3上,执行:
登录到mysql: # mysql -u root -pAbc1234! 设置在n1上可以访问mysql: mysql > grant all privileges on *.* to 'root'@'cent7-n1.pepstack.com' identified by 'Abc1234!' mysql > flush privileges;
这样就可以从n1上访问n3的mysql了。
5) 配置hive使用的mysql数据库
为hive 创建mysql数据库 hive_metastore (hive-schema-1.1.0.mysql.sql):
在n3 (mysql所在的机器上),执行:
# cd /usr/lib/hive/scripts/metastore/upgrade/mysql/ # mysql -u root -pAbc1234!
为hive创建表:
mysql> create database hive_metastore; mysql> use hive_metastore; mysql> source ./hive-schema-1.1.0.mysql.sql; mysql> quit;
在n3 上,为metastore所在的机器(n1)以hive连接到n3上的mysql做如下配置:
# mysql -u root -pAbc1234!
mysql> create user 'hive'@'cent7-n1.pepstack.com' identified by 'Abc1234!'; mysql> revoke all privileges,grant option from 'hive'@'cent7-n1.pepstack.com'; mysql> grant all privileges on hive_metastore.* to 'hive'@'cent7-n1.pepstack.com'; mysql> flush privileges; mysql> quit;
在n1上,测试访问n3:
# mysql -u hive -pAbc1234! -h cent7-n3.pepstack.com
这样,hive使用的数据库就配置成功了!