版权声明:本文为博主原创文章,转载请注明出处 https://blog.csdn.net/vkingnew/article/details/84565100
规划:
IP地址 角色 端口
192.168.0.151 clickhouse 9000 zookeeper2181
192.168.0.152 clickhouse 9000 zookeeper2181
192.168.0.153 clickhouse 9000 zookeeper2181
下载软件:
# curl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.rpm.sh | sudo bash
--安装软件依赖:依次提供python3和mysql-community-devel(libmysqlclient)
sudo yum install -y epel-release
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
--软件搜索:
# yum search clickhouse
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.ustc.edu.cn
* epel: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.cn99.com
=================== N/S matched: clickhouse =======================
clickhouse-client.x86_64 : clickhouse client binary
clickhouse-common-static.x86_64 : clickhouse common static binaries
clickhouse-compressor.x86_64 : clickhouse compressor binary
clickhouse-debuginfo.x86_64 : Debug information for package clickhouse
clickhouse-mysql.noarch : MySQL to ClickHouse data migrator
clickhouse-odbc.x86_64 : Yandex ClickHouse DBMS ODBC driver
clickhouse-server.x86_64 : Server files for clickhouse
clickhouse-server-common.x86_64 : Common configuration files for clickhouse
clickhouse-test.x86_64 : clickhouse test suite
Name and summary matches only, use "search all" for everything.
--软件安装和保存:
# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
注意:将keepcache=0 修改为keepcache=1即可保存安装的RPM软件。
注释:若只是需要安装clickhouse-mysql则至少需要安装如下软件依赖包:
clickhouse-client
mysql-community-devel
mariadb-devel
gcc
python34-devel
python34-pip
安装clickhouse-mysql需要安装如下python软件依赖包:
#sudo pip3 install clickhouse-mysql
pip3 install mysqlclient
pip3 install mysql-replication
pip3 install clickhouse-driver
通过RPM安装的可以自动安装软件: pycparser, cffi, six, asn1crypto, idna, cryptography, pymysql, mysql-replication
软件安装:
# yum -y install clickhouse-client clickhouse-common-static clickhouse-compressor clickhouse-mysql clickhouse-odbc clickhouse-server clickhouse-server-common
---配置文件说明:
默认配置文件位置:
# rpm -ql clickhouse-server-common
/etc/clickhouse-server/config.xml
/etc/clickhouse-server/users.xml
ClickHouse有几核心的配置文件:
config.xml 端口配置、本地机器名配置、内存设置等
metrika.xml 集群配置、ZK配置、分片配置等
users.xml 权限、配额设置
默认的数据目录、临时目录、日志目录:
/var/lib/clickhouse
/var/lib/clickhouse/tmp/
/var/log/clickhouse-server
# rpm -ql clickhouse-server
/etc/cron.d/clickhouse-server 定时任务脚本
/etc/init.d/clickhouse-server 默认启动脚本
/etc/security/limits.d/clickhouse.conf 安全配置
/usr/bin/clickhouse-clang 提供的二进制命令
/usr/bin/clickhouse-copier
/usr/bin/clickhouse-format
/usr/bin/clickhouse-lld
/usr/bin/clickhouse-obfuscator
/usr/bin/clickhouse-odbc-bridge
/usr/bin/clickhouse-report
/usr/bin/clickhouse-server
/usr/share/clickhouse
/usr/share/clickhouse/bin
/usr/share/clickhouse/headers
/usr/share/clickhouse/headers/18.14.15
# cat /etc/security/limits.d/clickhouse.conf
clickhouse soft nofile 262144
clickhouse hard nofile 262144
说明:由于默认的数据目录不便于管理,需要修改下。
--clickhouse的启动和关闭:
clickhouse-server --config-file=/etc/clickhouse-server/config.xml
或:
systemctl stop clickhouse-server
systemctl start clickhouse-server
--登录:
# clickhouse-client --host=197.255.20.64 --user=default --password=oracle --port=9900
ClickHouse client version 18.14.15.
Connecting to 197.255.20.64:9900 as user default.
Connected to ClickHouse server version 18.14.15 revision 54409.
--登录进入之后clickhouse默认有两个数据库:
production :) show databases;
SHOW DATABASES
┌─name────┐
│ default │
│ system │
└─────────┘
2 rows in set. Elapsed: 0.004 sec.
production :) select * from system.clusters;
SELECT *
FROM system.clusters
┌─cluster─────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────────────┬─host_address──┬─port─┬─is_local─┬─user────┬─default_database─┐
│ bip_ck_cluster │ 1 │ 1 │ 1 │ 197.255.20.62 │ 197.255.20.62 │ 9900 │ 0 │ default │ │
│ bip_ck_cluster │ 2 │ 1 │ 1 │ 197.155.20.64 │ 197.155.20.64 │ 9900 │ 0 │ default │ │
│ bip_ck_cluster │ 3 │ 1 │ 1 │ 197.255.20.65 │ 197.255.20.65 │ 9900 │ 0 │ default │ │
│
└─────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────────────┴───────────────┴──────┴──────────┴─────────┴──────────────────┘
--创建数据库:
roduction :) create database if not exists wuhan;
CREATE DATABASE IF NOT EXISTS wuhan
Ok.
0 rows in set. Elapsed: 0.006 sec.
production :) show databases;
SHOW DATABASES
┌─name────┐
│ default │
│ system │
│ wuhan │
└─────────┘
3 rows in set. Elapsed: 0.005 sec.
--将MySQL的数据实时同步到clickhouse:
curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo bash
安装EPEL和python3 以及mysql的客户端工具:
sudo yum install -y epel-release
sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
或者:
sudo yum install -y https://download.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
sudo yum install -y clickhouse-mysql
--配置mysql:
sudo cp /etc/clickhouse-mysql/clickhouse-mysql-example.conf /etc/clickhouse-mysql/clickhouse-mysql.conf
sudo vim /etc/clickhouse-mysql/clickhouse-mysql.conf
--使用clickhouse-mysql直接同步单个表的命令:
clickhouse-mysql \
--src-server-id=1 \
--src-resume \
--src-wait \
--nice-pause=1 \
--src-host=192.168.3.191 \
--src-user=root \
--src-password=abcd@1234 \
--src-tables=ZHIXIAOQIYE.zhixiao_data \
--dst-host=192.168.3.194 \
--dst-schema=default \ # 指定clickhouse数据库
--dst-table=zhixiao \ # 指定clickhouse数据表
--csvpool \
--csvpool-file-path-prefix=qwe_ \
--mempool-max-flush-interval=60 \
--mempool-max-events-num=10000
---------------------
--可以实时同步MySQL的多个库的数据:
--通过csv文件导入:
--参考:
https://github.com/Altinity/clickhouse-mysql-data-reader#rpm-installation