ubuntu 搭建 ceph
首先理解三个概念: 文件存储系统,快存储系统,对象存储系统
准备工作:
- 虚拟机三台:
ip
: 192.168.87.132 ceph-admin
ip
: 192.168.87.133 node0
ip
: 192.168.87.134 node1
系统
: ubuntu 16.04
1.配置节点host
为了方便后边安装,以及 ssh 方式连接各个节点,我们先修改一下各个节点的 Hostname 以及配置 Hosts 如下:
ceph-admin(192.168.87.132)
$ cat /etc/hostname
admin
$ cat /etc/hosts
192.168.87.132 admin
192.168.87.133 node0
192.168.87.134 node1
-----------------------------
node0(192.168.87.133)
$ cat /etc/hostname
node0
$ cat /etc/hosts
192.168.87.132 admin
192.168.87.133 node0
192.168.87.134 node1
-----------------------------
node1(192.168.87.134)
$ cat /etc/hostname
node1
$ cat /etc/hosts
192.168.87.132 admin
192.168.87.133 node0
192.168.87.134 node1
2.更新源
添加 release key
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
添加Ceph软件包源,用Ceph稳定版(如 luminous 等等)替换掉 {ceph-stable-release}
将这条信息写入/etc/apt/sources.list
echo deb https://download.ceph.com/debian-luminous/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
更新系统更新源,3台服务器执行以下命令
sudo apt-get clean
sudp apt-get update
3.安装 ntpdate(解决时区问题) 和 ssh(无密码登录)
务必保证3台服务器的时区是一样的,强制更改时区为上海,执行以下命令
ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
bash -c "echo 'Asia/Shanghai' > /etc/timezone"
安装ntpdate
apt-get install -y ntpdate
使用阿里云的时间服务器更新
ntpdate ntp1.aliyun.com
3台服务器都执行一下,确保时间一致!
后续操作,ceph-deploy 节点需要使用 ssh 方式登录各个节点完成 ceph 安装配置工作,所以要确保各个节点上有可用 SSH 服务。
sudo apt-get install openssh-server -y
4.创建 Ceph 部署应用
ceph-deploy 工具必须以普通用户登录 Ceph 节点,且此用户拥有无密码使用 sudo 的权限,因为它需要在安装软件及配置文件的过程中,不必输入密码。官方建议所有 Ceph 节点上给 ceph-deploy 创建一个特定的用户,而且不要使用 ceph 这个名字。这里为了方便,我们使用 cephd 这个账户作为特定的用户,而且每个节点上(admin-node、node0、node1)上都需要创建该账户,并且拥有 sudo 权限。
(1).建立用户
# 在 Ceph 集群各节点进行如下操作
# 创建 ceph 特定用户
# -d<登入目录>:指定用户登入时的启始目录;
# -m:自动建立用户的登入目录;
$ sudo useradd -d /home/cephd -m cephd
$ sudo passwd cephd
# 添加 sudo 权限
$ echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd
$ sudo chmod 0440 /etc/sudoers.d/cephd
(2).免登陆
接下来在 ceph-deploy 节点(admin-node)上,切换到 cephd 用户,生成 SSH 密钥并把其公钥分发到各 Ceph 节点上,注意使用 cephd 账户生成,且提示输入密码时,直接回车,因为它需要免密码登录到各个节点。
# ceph-deploy (admin-node) 上执行
# 生成 ssh 密钥
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cephd/.ssh/id_rsa):
Created directory '/home/cephd/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cephd/.ssh/id_rsa.
Your public key has been saved in /home/cephd/.ssh/id_rsa.pub.
The key fingerprint is:
...
# 将公钥复制到 node0 节点
$ ssh-copy-id cephd@node0
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
The authenticity of host 'node0 (10.222.77.242)' can't be established.
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephd@node0's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'cephd@node0'"
and check to make sure that only the key(s) you wanted were added.
# 将公钥复制到 node1 节点
$ ssh-copy-id cephd@node1
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
The authenticity of host 'node1 (10.222.77.253)' can't be established.
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephd@node1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'cephd@node1'"
and check to make sure that only the key(s) you wanted were added.
测试没有问题,接下来,修改 ceph-deploy 管理节点上的 ~/.ssh/config 文件,这样无需每次执行 ceph-deploy 都要指定 –username cephd 。这样做同时也简化了 ssh 和 scp 的用法
$ cat ~/.ssh/config
Host node0
Hostname node0
User cephd
Host node1
Hostname node1
User cephd
测试一下在 ceph-deploy 管理节点免密码登录各个节点。
$ ssh node0
Last login: Fri Dec 8 15:50:08 2017 from admin
$ ssh node1
Last login: Fri Dec 8 15:49:27 2017 from admin
(3).安装 ceph-deploy
# admin上执行
sudo apt-get install ceph-deploy -y
5.搭建集群
好了,经过上边一系列的预检设置后,我们就可以开始 Ceph 存储集群搭建了,集群结构为 admin-node (ceph-deploy、Monitor)、node0(osd.0)、node1(osd.1)。首先要提一下的是,如果我们在安装过程中出现了问题,需要重新操作的时候,例如想清理我搭建的这个集群的话,可以使用以下命令。
# ceph-deploy (admin-node) 上执行
# 清理配置
ceph-deploy purgedata admin node0 node1
ceph-deploy forgetkeys
# 清理 Ceph 安装包
ceph-deploy purge admin node0 node1
好了,现在开始搭建。首先 Cephd 用户创建一个目录 ceph-cluster 并进入到该目录执行一系列操作。因为我们设计的 monitor 节点在 admin-node 节点上,所以,执行如下命令。
# 创建执行目录
$ mkdir ~/ceph-cluster && cd ~/ceph-cluster
# 创建集群
$ ceph-deploy new admin
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy new admin
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] func : <function new at 0xf24938>
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xf1f6c8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['admin']
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
...
[ceph_deploy.new][DEBUG ] Resolving host admin
[ceph_deploy.new][DEBUG ] Monitor admin at 10.222.77.213
[ceph_deploy.new][DEBUG ] Monitor initial members are ['admin']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.222.77.213']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
此时,我们会发现 ceph-deploy 会在 ceph-cluster 目录下生成几个文件,ceph.conf 为 ceph 配置文件,ceph-deploy-ceph.log 为 ceph-deploy 日志文件,ceph.mon.keyring 为 ceph monitor 的密钥环。
$ ll ceph-cluster
-rw-rw-r--. 1 cephd cephd 196 12月 7 14:46 ceph.conf
-rw-rw-r--. 1 cephd cephd 3694 12月 7 14:46 ceph-deploy-ceph.log
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
$ cat ceph.conf
[global]
fsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
mon_initial_members = admin
mon_host = 10.222.77.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
接下来,我们需要修改下 ceph.conf 配置文件,增加副本数为 2,因为我们有两个 osd 节点。
$ cat ceph.conf
[global]
fsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
mon_initial_members = admin
mon_host = 10.222.77.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 2 #增加默认副本数为 2
然后,我们需要通过 ceph-deploy 在各个节点安装 ceph。
$ ceph-deploy install admin node0 node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy install admin node0 node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xde1c68>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : True
[ceph_deploy.cli][INFO ] func : <function install at 0xd5b140>
[ceph_deploy.cli][INFO ] install_mgr : False
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['admin', 'node0', 'node1']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] nogpgcheck : False
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts admin node0 node1
[ceph_deploy.install][DEBUG ] Detecting platform for host admin ...
...
此过程需要等待一段时间,因为 ceph-deploy 会 SSH 登录到各 node 上去,依次执行安装 ceph 依赖的组件包。
漫长的等待安装完毕之后,接下来需要初始化 monitor 节点并收集所有密钥。
$ ceph-deploy mon create-initial
...
ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
...
漫长的等待安装完毕之后,接下来需要初始化 monitor 节点并收集所有密钥。
$ ceph-deploy mon create-initial
...
ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
...
不过很遗憾,执行过程中报错了。查看原因应该是已经存在了 /etc/ceph/ceph.conf 配置文件了,解决方案就是加上 --overwrite-conf 参数,覆盖已存在的配置。
$ ceph-deploy --overwrite-conf mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mon create-initial
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : create-initial
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1d9e758>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mon at 0x1d8ea28>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] keyrings : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts admin
[ceph_deploy.mon][DEBUG ] detecting platform for host admin ...
...
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.admin.asok mon_status
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.admin
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mds
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mgr
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-osd
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpLv4mug
执行完毕后,会在当前目录下生成一系列的密钥环,应该是各组件之间访问所需要的认证信息吧。
$ ll ~/ceph-cluster
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-mds.keyring
-rw-------. 1 cephd cephd 71 12月 7 15:13 ceph.bootstrap-mgr.keyring
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-osd.keyring
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-rgw.keyring
-rw-------. 1 cephd cephd 129 12月 7 15:13 ceph.client.admin.keyring
-rw-rw-r--. 1 cephd cephd 222 12月 7 14:47 ceph.conf
-rw-rw-r--. 1 cephd cephd 120207 12月 7 15:13 ceph-deploy-ceph.log
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
到此,ceph monitor 已经成功启动了。接下来需要创建 OSD 了,OSD 是最终数据存储的地方,这里我们准备了两个 OSD 节点,分别为 osd.0 和 osd.1。官方建议为 OSD 及其日志使用独立硬盘或分区作为存储空间,不过本机虚拟机上不具备条件,但是我们可以在虚拟机本地磁盘上创建目录,来作为 OSD 的存储空间。
# ceph-deploy (admin-node) 上执行
$ ssh node0
$ sudo mkdir /var/local/osd0
$ sudo chown -R ceph:ceph /var/local/osd0
$ exit
$ ssh node1
$ sudo mkdir /var/local/osd1
$ sudo chown -R ceph:ceph /var/local/osd1
$ exit
注意:这里执行了 chown -R ceph:ceph
操作,将 osd0 和 osd1 目录的权限赋予 ceph:ceph,否则,接下来执行 ceph-deploy osd activate ...
时会报权限错误。
接下来,我们需要 ceph-deploy 节点执行 prepare
OSD 操作,目的是分别在各个 OSD 节点上创建一些后边激活 OSD 需要的信息。
$ ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] block_db : None
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] block_wal : None
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : prepare
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x117d950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] filestore : None
[ceph_deploy.cli][INFO ] func : <function osd at 0x116a320>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node0
...
[node0][INFO ] checking OSD status...
[node0][DEBUG ] find the location of an executable
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node0 is now ready for osd use.
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
...
[node1][INFO ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.
OK 接下来,我们需要激活 activate OSD。
$ ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : activate
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2045950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x2032320>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host node0 disk /var/local/osd0
[ceph_deploy.osd][DEBUG ] will use init type: systemd
...
[node0][INFO ] checking OSD status...
[node0][DEBUG ] find the location of an executable
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[node0][INFO ] Running command: sudo systemctl enable ceph.target
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host node1 disk /var/local/osd1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
...
[node1][INFO ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[node1][INFO ] Running command: sudo systemctl enable ceph.target
看日志,激活也没有问题,最后一步,通过 ceph-deploy admin
将配置文件和 admin 密钥同步到各个节点,以便在各个 Node 上使用 ceph 命令时,无需指定 monitor 地址和 ceph.client.admin.keyring
密钥。
$ ceph-deploy admin admin node0 node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy admin admin node0 node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15a93b0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['admin', 'node0', 'node1']
[ceph_deploy.cli][INFO ] func : <function admin at 0x1500cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to admin
[admin][DEBUG ] connection detected need for sudo
[admin][DEBUG ] connected to host: admin
[admin][DEBUG ] detect platform information from remote host
[admin][DEBUG ] detect machine type
[admin][DEBUG ] write cluster configuration to /etc/ceph/{
cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node0
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] write cluster configuration to /etc/ceph/{
cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] write cluster configuration to /etc/ceph/{
cluster}.conf
同时为了确保对 ceph.client.admin.keyring 有正确的操作权限,所以还需要增加权限设置。
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
至此,Ceph 存储集群已经搭建完毕了,我们可以查看那一下集群是否启动成功!
# 查看集群状态
$ ceph -s
cluster 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
health HEALTH_OK
monmap e1: 1 mons at {
admin=10.222.77.213:6789/0}
election epoch 6, quorum 0 admin
osdmap e15: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v9276: 64 pgs, 1 pools, 863 MB data, 226 objects
33440 MB used, 33574 MB / 67015 MB avail
64 active+clean
# 或者查看集群健康状况
$ ceph health
HEALTH_OK
# 查看集群 OSD 信息
$ ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.06400 root default
-2 0.03200 host node0
0 0.03200 osd.0 up 1.00000 1.00000
-3 0.03200 host node1
1 0.03200 osd.1 up 1.00000 1.00000