Thingsboard下Cassandra集群化Docker部署
徐景周
一、环境
- CentOS
- 安装Nano
apt-get update
apt-get install nano
- Oracle JDK:1.8及以上
查找Java安装路径
which java
ls /usr/bin/java (上一步结果路径)
ls -lrt /etc/alternatives/java (上一步结果路径)
ls /usr/lib/jvm/java-8-openjdk-amd64 (最终路径)
- 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
二、单机部署(Docker版)
1. 停止宿主机Cassandra(如有必要)
sudo service cassandra stop
sudo chkconfig cassandra off
sudo service cassandra status
2. 安装Docker
- 安装
请浏览【参考文献 1】。
- 修改Docker默认目录
sudo service docker stop
mv /var/lib/docker /home/docker
ln -s /home/docker /var/lib/docker
sudo yum install -y nano
sudo nano /usr/lib/systemd/system/docker.service
文件修改项如下:
ExecStart=/usr/bin/dockerd --graph /home/docker -H fd:// --containerd=/run/containerd/containerd.soc
sudo systemctl daemon-reload
sudo service docker restart
sudo service docker status
docker info | grep Dir
- Docker默认磁盘空间
df -h
docker info
docker exec -it tb-cassandra df -h
- 自启动
sudo systemctl enable docker
3. Docker中安装Cassandra
1. 操作指令
-
安装
docker stop tb-cassandra
docker rm tb-cassandra
docker run --name tb-cassandra -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 -d cassandra:3.0
- 启动
docker start tb-cassandra
- 检查
docker ps -a
- 自启动
docker update --restart=always 容器ID
2. 实现细节
请浏览【参考文献 2】和【参考文献 3】。
4. 修改Cassandra配置
1. 操作指令
- 进入容器bash
docker exec -it tb-cassandra bash
- 启动cql环境
cqlsh
- cql环境下,修改族名
update system.local set cluster_name = 'Thingsboard Cluster' where key='local';
- cql环境下,查看族名
select cluster_name from system.local;
- cql环境下,退出
exit
- 刷新配置
nodetool flush system
- 修改cassandra.yaml配置文件
nano /etc/cassandra/cassandra.yaml
或者采用文件替换方式(先退出docker bash)
docker exec -it tb-cassandra bash
ls etc/cassandra/cassandra.yaml
exit;
docker ps
docker cp cassandra.yaml 容器id:/etc/cassandra/cassandra.yaml
docker exec -it tb-cassandra bash
cat /etc/cassandra/cassandra.yaml
Cassandra.yaml配置文件修改
创建目录及修改属性
ls /var/lib/cassandra/
mkdir /var/lib/cassandra/data
mkdir /var/lib/cassandra/commitlog
mkdir /var/lib/cassandra/saved_caches
mkdir /var/lib/cassandra/hints
chown 777 /var/lib/cassandra/data
chown 777 /var/lib/cassandra/commitlog
chown 777 /var/lib/cassandra/saved_caches
chown 777 /var/lib/cassandra/hints
修改文件下面的选项
cluster_name: 'Thingsboard Cluster’
hints_directory: /var/lib/cassandra/hints
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
saved_caches_directory: /var/lib/cassandra/saved_caches
seeds: "127.0.0.1”
listen_address: localhost
# broadcast_address: 1.2.3.4
start_rpc: true
rpc_address: 0.0.0.0
broadcast_rpc_address: 1.2.3.4
- 退出docker bash
exit;
- 安装Thingsboard对应数据库及数据表(顺序:schema-ts.cql -> schema-entities.cql)
docker ps
docker cp schema-ts.cql 容器id:/root/schema-ts.cql
docker cp schema-entities.cql 容器id:/root/schema-entities.cql
docker exec -it tb-cassandra bash
ls /root
cqlsh -f /root/schema-entities.cql
cqlsh -f /root/schema-ts.cql
exit;
- 重启Cassandra
docker restart tb-cassandra
- 查看Cassandra是否启动正常
docker logs tb-cassandra
实现细节
请浏览【参考文献4】和【参考文献 5】。
5. 修改Thingsboard配置
- 修改文件thingsboard.conf(/usr/share/thingsboard/conf)
# DB Configuration
export DATABASE_ENTITIES_TYPE=sql
export DATABASE_TS_TYPE=cassandra
export SPRING_JPA_DATABASE_PLATFORM=org.hibernate.dialect.PostgreSQLDialect
export SPRING_DRIVER_CLASS_NAME=org.postgresql.Driver
export SPRING_DATASOURCE_URL=jdbc:postgresql://192.168.1.131:5432/thingsboard
export SPRING_DATASOURCE_USERNAME=postgres
export SPRING_DATASOURCE_PASSWORD=123456
export CASSANDRA_CLUSTER_NAME=Thingsboard Cluster
export CASSANDRA_KEYSPACE_NAME=thingsboard
export CASSANDRA_URL=192.168.1.131:9042
export CASSANDRA_USE_CREDENTIALS=false
export CASSANDRA_USERNAME=
export CASSANDRA_PASSWORD=
- 重启Thingsboard
sudo service thingsboard restart
nano /var/log/thingsboard/thingsboard.log
- 重启Mosquitto(可选)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
- 重启TB-Gateway
sudo service tb-gateway restart
nano /var/log/tb-gateway/tb-gateway.log
6. Cassandra调优
实现细节
请浏览【参考文献7】和【参考文献8】。
7. 测试
- 打开Thingsboard后,运行测试脚本。请浏览【参考文献20】。
node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js
- 查看tb-cassandra容器中Cassandra数据
docker exec -it tb-cassandra bash
cqlsh
select count(1) from thingsboard.ts_kv_latest_cf;
exit
exit;
8. Cassandra查询工具
DataStax Studio
Cassandra Manager
Razorsql
9. 配置示例
192.168.1.131(Cassandra + Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
三、单机部署(非Docker版)
如果不希望基于Docker,直接在机器上安装Cassandra的情况。
1. 停止Docker版Cassandra(如有必要)
sudo service docker stop
sudo systemctl disable docker
docker info
# 杀掉Cassandra端口占用程序
netstat -tunlp
kill PID
2. 实现细节
请浏览【参考文献6】和“单机部署(Docker版)-修改Cassandra配置”章节。
四、集群部署(Docker版)
1. 停止宿主机Cassandra(如有必要)
sudo service cassandra stop
sudo chkconfig cassandra off
sudo service cassandra status
2. 安装Docker
请参考单机部署中对应章节。
3. 安装Ping和IFConfig
apt-get update
apt-get install inetutils-ping
apt-get install net-tools
4. 修改主机名
# 查看主机名
Hostnamectl
# 133机器
hostnamectl set-hostname 133
# 134机器
hostnamectl set-hostname 134
# 135机器
hostnamectl set-hostname 135
# 136机器
hostnamectl set-hostname 136
5. 安装IPTables
sudo yum install -y iptables-services
systemctl enable iptables.service
/usr/libexec/iptables/iptables.init save
systemctl daemon-reload
sudo service docker restart
6. Docker之间跨主机通讯(SWARM方式)
实现细节请浏览【参考文献9】。
例如:将133做为Manager机,其它做为Work机。
① Manager上创建指令(133机器)
docker swarm init --advertise-addr=192.168.1.133
② Work上加入指令(136机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.136 192.168.1.133:2377
③ Work上加入指令(135机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.135 192.168.1.133:2377
④ Work上加入指令(135机器)
docker swarm join --token SWMTKN-1-2gnzi68pon5q23f0x0hwda05q6uc5bhzj8svm0jpfakfek9g28-4xe5gow6kn7bxuqohg91rec9g --advertise-addr=192.168.1.134 192.168.1.133:2377
⑤ Manager上查看集群状态(133机器)
docker node ls
⑥ Manager上创建overlay网络(133机器)
docker network create -d overlay --attachable cassandra1-net
docker service update --network-add cassandra1-net --network-rm cassandranet my-cassandra
docker network create -d overlay --attachable cassandranet
docker network create -d overlay --subnet=172.172.0.0/24 --attachable cassandranet
# Manager机器和其它Work机器上都可查看
docker network ls
⑦ Manager上创建swarm服务(133机器)
docker service rm my-cassandra
docker network rm cassandranet
docker service create --name my-cassandra --network cassandra1-net --replicas 1 --publish published=8080,target=80 cassandra
# 黄色标记为机器数
docker service create --name my-cassandra --network cassandranet --replicas 5 --publish published=8080,target=80 cassandra
docker service ls
⑧ Manager上查看新建服务是否连接到新建overlay网络(133机器)
docker network inspect cassandranet
docker network inspect cassandra1-net
⑨ 查看服务
docker ps -a
⑩ 查看节点情况
docker node ls
docker service ps my-cassandra
7. Docker中安装Cassandra
- 操作指令
# 安装
docker stop tb-cassandra
docker rm tb-cassandra
# 133机器
sudo docker run -d --name tb-cassandra --network=cassandranet --ip 172.172.0.133 --restart always -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 cassandra:3.0
# 134机器
sudo docker run -d --network=cassandranet --ip 172.172.0.134 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 135机器
sudo docker run -d --network=cassandranet --ip 172.172.0.135 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 136机器
sudo docker run -d --network=cassandranet --ip 172.172.0.136 -p 7000:7000 -p 7001:7001 -p 7199:7199 -p 9042:9042 -p 9160:9160 --name tb-cassandra --restart always cassandra:3.0
# 启动
docker start tb-cassandra
# 检查
docker ps -a
# 自启动
docker update --restart=always 容器ID
8. 修改Cassandra配置
① 按单机部署相关章节,执行前6步。
② 在修改后cassandra.yaml 基础上,再修改下面几项内容
1) # 选取一半数量的节点物理IP,做为Seeds
例如:
- seeds: "192.168.1.133,192.168.1.134"
2) # 替换成各节点自己的物理IP地址
例如;
# 容器IP(docker inspect tb-cassandra)
listen_address: 172.172.0.133
# hostIP
broadcast_address: 192.168.1.133
3) # 同步策略
endpoint_snitch: GossipingPropertyFileSnitch
③ 修改文件cassandra-rackdc.properties
1) dc=datacenter1
2) rack=rack1
④ 修改文件cassandra-env.sh
1) #JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=192.168.1.134"
2) LOCAL_JMX=no
3) JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
4) JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file=/etc/cassandra/jmxremote.access"
⑤ 分别替换各个容器节点cassandra中对应的文件:cassandra.yaml、cassandra-rackdc.properties、cassandra-env.sh文件
例如:
docker exec -it tb-cassandra bash
ls etc/cassandra/cassandra.yaml
exit;
docker ps
# 133机器
# cassandra.yaml
docker cp /root/cassandra_133.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-133.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
ls /etc/cassandra/
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 134机器
# cassandra.yaml
docker cp /root/cassandra_134.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-134.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 135机器
# cassandra.yaml
docker cp /root/cassandra_135.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-135.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 136机器
# cassandra.yaml
docker cp /root/cassandra_136.yaml 容器id:/etc/cassandra/cassandra.yaml
# cassandra-rackdc.properties
docker cp /root/cassandra-rackdc.properties 容器id:/etc/cassandra/cassandra-rackdc.properties
# cassandra-env.sh
docker cp /root/cassandra-env-136.sh 容器ID:/etc/cassandra/cassandra-env.sh
docker cp /root/jmxremote.access 容器ID:/etc/cassandra/jmxremote.access
docker cp /root/jmxremote.password 容器ID:/etc/cassandra/jmxremote.password
docker exec -it tb-cassandra bash
# 修改权限
chown cassandra:cassandra /etc/cassandra/jmxremote.password
chmod 400 /etc/cassandra/jmxremote.password
chown cassandra:cassandra /etc/cassandra/jmxremote.access
chmod 400 /etc/cassandra/jmxremote.access
more /etc/cassandra/cassandra-env.sh
more /etc/cassandra/cassandra.yaml
more /etc/cassandra/jmxremote.access
more /etc/cassandra/jmxremote.password
more /etc/cassandra/cassandra-rackdc.properties
# 退出
exit;
⑥ 安装Thingsboard数据库及数据表(顺序:schema-entities.cql -> schema-ts.cql)
docker ps
docker cp /root/schema-entities.cql 容器id:/root/schema-entities.cql
docker cp /root/schema-ts.cql 容器id:/root/schema-ts.cql
docker exec -it tb-cassandra bash
ls /root
cqlsh -f /root/schema-entities.cql
cqlsh -f /root/schema-ts.cql
exit;
⑦ 重启Cassandra
docker stop tb-cassandra
docker start tb-cassandra
⑧ 查看Cassandra是否启动正常
docker logs tb-cassandra
9. 查看集群情况
各节点的cassandra.yaml文件依次替换后,分别启动各个Cassandra节点,查看集群情况。
例如:
# 任一机器
docker exec -it tb-cassandra bash
nodetool -h 192.168.1.133 -u cassandra -pw cassandra status
nodetool -h 192.168.1.134 -u cassandra -pw cassandra status
nodetool -h 192.168.1.135 -u cassandra -pw cassandra status
nodetool -h 192.168.1.136 -u cassandra -pw cassandra status
nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.134 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.135 -u cassandra -pw cassandra ring
nodetool -h 192.168.1.136 -u cassandra -pw cassandra ring
cqlsh 192.168.1.133 -u cassandra -p cassandra
ping 172.172.0.134
ping 172.172.0.135
ping 172.172.0.136
exit;
# Manager(133机器
docker node ls
docker network inspect cassandranet
10. 修改Thingsboard配置
在单机部署修改后thingsboard.conf的基础上,修改下面内容(/usr/share/thingsboard/conf):
1) Export CASSANDRA_URL=192.168.1.133:9042,192.168.1.134:9042,192.168.1.135:9042,192.168.1.136:9042
2) export CASSANDRA_USE_CREDENTIALS=true
3) export CASSANDRA_USERNAME=cassandra
4) export CASSANDRA_PASSWORD=cassandra
重启Thingsboard
sudo service thingsboard restart
nano /var/log/thingsboard/thingsboard.log
重启Mosquitto(可选,MQTT)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
重启TB-Gateway
sudo service tb-gateway restart
nano /var/log/tb-gateway/tb-gateway.log
11. 集群工具
Nodetool
JConsole
例如:
nodetool status
nodetool info
nodetool proxyhistograms
12. 修复无法启动的容器
Docker容器中,由于个别文档(例如:cassandra-env.sh)导致cassandra无法启动。
例如:
cd /home/docker
find ./ -name 'cassandra-env.sh'
对搜索到的文档进行比对修改,再重启容器即可!
13. 配置示例
192.168.1.133(Cassandra)
192.168.1.134(Cassandra)
192.168.1.135(Cassandra)
192.168.1.136(Cassandra)
192.168.1.131(Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
五、集群部署(非Docker版)
如果不希望基于Docker,直接在机器上安装Cassandra集群的情况。
1. 停止Docker版Cassandra(如有必要)
sudo service docker stop
sudo systemctl disable docker
docker info
2. 关闭运行中YUM
rm -f /var/run/yum.pid
3. 安装Cassandra
请浏览【参考文献6】
4. 修改配置文件
① 修改Cassandra默认族名
例如:
cqlsh
update system.local set cluster_name = 'Thingsboard Cluster' where key='local';
select cluster_name from system.local;
exit
nodetool flush system
② 修改配置文件
请浏览【参考文献18】、【参考文献19】,以及“单机部署(Docker版)-修改Cassandra配置”章节内容,进行修改。
例如:
sudo nano /etc/thingsboard/conf/thingsboard.conf
sudo nano /etc/cassandra/default.conf/cassandra.yaml
备注:其中修改的文件(cassandra.yaml和thingsboard.conf)
③ 安装Thingsboard数据库及数据表
备注:数据库及表只需任选一台安装,其它节点会定时自动同步。执行顺序:schema-entities.cql -> schema-ts.cql
例如:
# 安装在133机器
ls /root
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-entities.cql
cqlsh 192.168.1.133 -u cassandra -p cassandra -f /root/schema-ts.cql
# 查看是否创建
cqlsh 192.168.1.133 -u cassandra -p cassandra
select * from thingsboard.user;
select count(1) from thingsboard.ts_kv_latest_cf;
exit
5. 查看集群情况
重启各cassandra节点,查看集群情况。
例如:
# 如果出现前后datacenter不一致问题,执行下面语句
rm -rf /var/lib/cassandra/data/*
ls /var/lib/cassandra/data
# 杀掉Cassandra端口占用程序
netstat -tunlp
kill PID
sudo systemctl stop cassandra
sudo systemctl start cassandra
nano /var/log/cassandra/cassandra.log
sudo systemctl status cassandra
# 查看集群
nodetool status
cqlsh 192.168.1.133 -u cassandra -p cassandra
nodetool -h 192.168.1.133 -u cassandra -pw cassandra status
nodetool -h 192.168.1.133 -u cassandra -pw cassandra ring
# 查看库、表同步状况
# 136机器
cqlsh 192.168.1.136 -u cassandra -p cassandra
select * from thingsboard.user;
select count(1) from thingsboard.ts_kv_latest_cf;
# 135机器
cqlsh 192.168.1.135 -u cassandra -p cassandra
select * from thingsboard.user;
# 查看集群节点
select peer from system.peers;
# Cassandra各节点成功启动后,查看日志(cassandra.log)
6. 修改system_auth表的配置
此步骤可选,可视情况而定。
例如:
# 133机器上,”dc1”为数据中心名(可通过nodetool status获取)
cqlsh 192.168.1.133 -u cassandra -p cassandra
ALTER KEYSPACE system_auth WITH replication = {'class': 'NetworkTopologyStrategy','dc1': '5'};
exit
nodetool repair
# 其它机器上进行同步
nodetool repair
7. 测试
替换配置文件(thingsboard.conf)后,重启Thingsboard和TB-Gateway,运行测试脚本进行测试!
例如:
# 重启Mosquitto(可选)
ps -aux | grep mosquitto
kill -9 PID
mosquitto -c /root/mosquitto-1.6.5/mosquitto.conf -d
# 重启Thingsboard
sudo systemctl restart thingsboard
nano /var/log/thingsboard/thingsboard.log
sudo systemctl status thingsboard
# 重启TB-Gateway
sudo systemctl restart tb-gateway
nano /var/log/tb-gateway/tb-gateway.log
sudo systemctl status tb-gateway
# 运行测试脚本
node /opt/mqtt/mqtt_gateway_mqtt_sz_new.js
# Thingsboard成功运行后,查看日志(thingsboard.log)
3. 查看数据库负载均衡及同步情况
测试成功运行后,可在任一台Cassandra节点查看库状况。
例如:
# 133机器上
cqlsh 192.168.1.133 -u cassandra -p cassandra
select count(1) from thingsboard.ts_kv_latest_cf;
exit
9. 配置示例
192.168.1.133(Cassandra)
192.168.1.134(Cassandra)
192.168.1.135(Cassandra)
192.168.1.136(Cassandra)
192.168.1.131(Postgres)
192.168.1.59(Thingsboard + TB-Gateway)
参考文献
- https://docs.docker.com/install/linux/docker-ce/centos/
- https://hub.docker.com/_/cassandra/
- https://thingsboard.io/docs/user-guide/install/docker/
- https://stackoverflow.com/questions/22006887/cassandra-saved-cluster-name-test-cluster-configured-name
- https://stackoverflow.com/questions/36133127/how-to-configure-cassandra-for-remote-connection
- https://thingsboard.io/docs/user-guide/install/rhel/?rhelThingsboardDatabase=hybrid
- https://docs.datastax.com/en/ddac/doc/datastax_enterprise/config/configRecommendedSettings.html#configRecommendedSettings
- https://www.jianshu.com/p/5bacb06e334b
- https://docs.docker.com/v17.12/network/network-tutorial-overlay/#walkthrough
- https://cloud.tencent.com/developer/article/1110563
- https://www.ibm.com/developerworks/cn/opensource/os-cn-cassandraxu1/index.html
- https://www.jianshu.com/p/b970876ee3cf
- https://blog.csdn.net/chenxingzhen001/article/details/8729623
- https://stackoverflow.com/questions/38405983/apache-cassandra-3-7-snitch-issue-cannot-start-data-center/39150744#39150744
- https://blog.csdn.net/walkerjong/article/details/52221126
- https://www.cnblogs.com/51kata/p/5276407.html
- https://www.cnblogs.com/capcom923/p/deploy_dockerize_cassandra_HA_cluster.html
- https://thingsboard.io/docs/user-guide/install/cluster-setup/
- https://thingsboard.io/docs/reference/performance-tests/