|
Hi Mike,
Thanks a lot for your input. I tried exactly as you have mentioned in your posting. The good news is that “osd prepare” succeeds but now “osd
activate” fails. See below: [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy osd prepare ceph-node2-osd0-centos-6-4:/ceph/osd0 ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy osd prepare ceph-node2-osd0-centos-6-4:/ceph/osd0
ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph-node2-osd0-centos-6-4:/ceph/osd0: ceph-node3-osd1-centos-6-4:/ceph/osd1: [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.osd][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf
to overwrite [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph-node3-osd1-centos-6-4][WARNIN] osd keyring does not exist yet, creating one [ceph-node3-osd1-centos-6-4][DEBUG ] create a keyring file [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory [ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy uninstall ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.cli][INFO
] Invoked (1.3): /usr/bin/ceph-deploy uninstall ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.install][DEBUG ] Uninstalling on cluster ceph hosts ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node2-osd0-centos-6-4 ... [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final [ceph-node2-osd0-centos-6-4][INFO ] uninstalling ceph on ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo yum -y -q remove ceph [ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node3-osd1-centos-6-4 ... [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final [ceph-node3-osd1-centos-6-4][INFO ] uninstalling ceph on ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo yum -y -q remove ceph [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ssh ceph-node2-osd0-centos-6-4 Last login: Tue Nov 5 14:09:56 2013 from 10.12.132.70 [ceph@ceph-node2-osd0-centos-6-4 ~]$sudo rm -rf /etc/ceph [ceph@ceph-node2-osd0-centos-6-4 ~]$ exit logout Connection to ceph-node2-osd0-centos-6-4 closed. [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ssh ceph-node3-osd1-centos-6-4 Last login: Tue Nov 5 14:10:29 2013 from 10.12.132.70 [ceph@ceph-node3-osd1-centos-6-4 ~]$ sudo rm -rf /etc/ceph [ceph@ceph-node3-osd1-centos-6-4 ~]$ exit logout Connection to ceph-node3-osd1-centos-6-4 closed. [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy gatherkeys ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy gatherkeys ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring [ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring [ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-mds.keyring [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy -v install ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.cli][INFO
] Invoked (1.3): /usr/bin/ceph-deploy -v install ceph-node2-osd0-centos-6-4 ceph-node3-osd1-centos-6-4 [ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster ceph hosts ceph-node2-osd0-centos-6-4
ceph-node3-osd1-centos-6-4 [ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node2-osd0-centos-6-4 ... [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final [ceph-node2-osd0-centos-6-4][INFO ] installing ceph on ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][INFO ] adding EPEL repository [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [ceph-node2-osd0-centos-6-4][ERROR ] --2013-11-05 14:20:31-- http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [ceph-node2-osd0-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... connected. [ceph-node2-osd0-centos-6-4][ERROR ] Proxy request sent, awaiting response... 200 OK [ceph-node2-osd0-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] [ceph-node2-osd0-centos-6-4][ERROR ] Saving to: `epel-release-6-8.noarch.rpm.5' [ceph-node2-osd0-centos-6-4][ERROR ] [ceph-node2-osd0-centos-6-4][ERROR ] 0K .......... .... 100% 4.48M=0.003s [ceph-node2-osd0-centos-6-4][ERROR ] [ceph-node2-osd0-centos-6-4][ERROR ] Last-modified header invalid -- time-stamp ignored. [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:21:31 (4.48 MB/s) - `epel-release-6-8.noarch.rpm.5' saved
[14540/14540] [ceph-node2-osd0-centos-6-4][ERROR ] [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh --replacepkgs epel-release-6*.rpm [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... ################################################## [ceph-node2-osd0-centos-6-4][DEBUG ] epel-release ################################################## [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo rpm -Uvh --replacepkgs http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm [ceph-node2-osd0-centos-6-4][DEBUG ] Retrieving http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm [ceph-node2-osd0-centos-6-4][DEBUG ] Preparing... ################################################## [ceph-node2-osd0-centos-6-4][DEBUG ] ceph-release ################################################## [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo yum -y -q install ceph [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo ceph --version [ceph-node2-osd0-centos-6-4][DEBUG ] ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) [ceph_deploy.install][DEBUG ] Detecting platform for host ceph-node3-osd1-centos-6-4 ... [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: CentOS 6.4 Final [ceph-node3-osd1-centos-6-4][INFO ] installing ceph on ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][INFO ] adding EPEL repository [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [ceph-node3-osd1-centos-6-4][ERROR ] --2013-11-05 14:21:41-- http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [ceph-node3-osd1-centos-6-4][ERROR ] Connecting to 10.12.132.208:8080... connected. [ceph-node3-osd1-centos-6-4][ERROR ] Proxy request sent, awaiting response... 200 OK [ceph-node3-osd1-centos-6-4][ERROR ] Length: 14540 (14K) [application/x-rpm] [ceph-node3-osd1-centos-6-4][ERROR ] Saving to: `epel-release-6-8.noarch.rpm.4' [ceph-node3-osd1-centos-6-4][ERROR ] [ceph-node3-osd1-centos-6-4][ERROR ] 0K .......... .... 100% 6.22M=0.002s [ceph-node3-osd1-centos-6-4][ERROR ] [ceph-node3-osd1-centos-6-4][ERROR ] Last-modified header invalid -- time-stamp ignored. [ceph-node3-osd1-centos-6-4][ERROR ] 2013-11-05 14:22:41 (6.22 MB/s) - `epel-release-6-8.noarch.rpm.4' saved
[14540/14540] [ceph-node3-osd1-centos-6-4][ERROR ] [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh --replacepkgs epel-release-6*.rpm [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... ################################################## [ceph-node3-osd1-centos-6-4][DEBUG ] epel-release ################################################## [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo rpm -Uvh --replacepkgs http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm [ceph-node3-osd1-centos-6-4][DEBUG ] Retrieving http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm [ceph-node3-osd1-centos-6-4][DEBUG ] Preparing... ################################################## [ceph-node3-osd1-centos-6-4][DEBUG ] ceph-release ################################################## [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo yum -y -q install ceph [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo ceph --version [ceph-node3-osd1-centos-6-4][DEBUG ] ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy -v osd prepare ceph-node2-osd0-centos-6-4:/ceph/osd0 ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd prepare ceph-node2-osd0-centos-6-4:/ceph/osd0
ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph-node2-osd0-centos-6-4:/ceph/osd0: ceph-node3-osd1-centos-6-4:/ceph/osd1: [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph-node2-osd0-centos-6-4][WARNIN] osd keyring does not exist yet, creating one [ceph-node2-osd0-centos-6-4][DEBUG ] create a keyring file [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host ceph-node2-osd0-centos-6-4 disk /ceph/osd0 journal None activate False [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs --cluster ceph --
/ceph/osd0 [ceph_deploy.osd][DEBUG ] Host ceph-node2-osd0-centos-6-4 is now ready for osd use. [ceph-node3-osd1-centos-6-4][DEBUG ] connected to host: ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node3-osd1-centos-6-4][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] Deploying osd to ceph-node3-osd1-centos-6-4 [ceph-node3-osd1-centos-6-4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph-node3-osd1-centos-6-4][WARNIN] osd keyring does not exist yet, creating one [ceph-node3-osd1-centos-6-4][DEBUG ] create a keyring file [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host ceph-node3-osd1-centos-6-4 disk /ceph/osd1 journal None activate False [ceph-node3-osd1-centos-6-4][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs --cluster ceph --
/ceph/osd1 [ceph_deploy.osd][DEBUG ] Host ceph-node3-osd1-centos-6-4 is now ready for osd use. [ceph@ceph-admin-node-centos-6-4 my-cluster]$ ceph-deploy -v osd activate ceph-node2-osd0-centos-6-4:/ceph/osd0 ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd activate ceph-node2-osd0-centos-6-4:/ceph/osd0
ceph-node3-osd1-centos-6-4:/ceph/osd1 [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph-node2-osd0-centos-6-4:/ceph/osd0: ceph-node3-osd1-centos-6-4:/ceph/osd1: [ceph-node2-osd0-centos-6-4][DEBUG ] connected to host: ceph-node2-osd0-centos-6-4 [ceph-node2-osd0-centos-6-4][DEBUG ] detect platform information from remote host [ceph-node2-osd0-centos-6-4][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.4 Final [ceph_deploy.osd][DEBUG ] activating host ceph-node2-osd0-centos-6-4 disk /ceph/osd0 [ceph_deploy.osd][DEBUG ] will use init type: sysvinit [ceph-node2-osd0-centos-6-4][INFO ] Running command: sudo ceph-disk-activate --mark-init sysvinit --mount /ceph/osd0 [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:36.213550 7f3a547cb700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a50024480 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a500246e0).fault [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:39.209287 7f3a546ca700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a44000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a44000e60).fault [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:42.209632 7f3a547cb700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a44003010 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a44003270).fault [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:46.209217 7f3a546ca700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a44003a70 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a44003cd0).fault [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:49.210193 7f3a547cb700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a44005550 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a440057b0).fault [ceph-node2-osd0-centos-6-4][ERROR ] 2013-11-05 14:24:52.210233 7f3a546ca700 0 -- :/1007531 >> 10.12.0.70:6789/0
pipe(0x7f3a44007050 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3a440072b0).fault What did you get when you octivated OSDs??
Thanks! Narendra From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Operações Servidores Hello, We experienced the same error as reported by Navendra, although we're running Ubuntu Server 12.04. We managed to work around the error (by trial and error). Below are the steps we performed, perhaps this can help you track down the error. Step 1 - This was the error openstack@monitor3:~/cluster1$
ceph-deploy -v osd prepare ceph1:sde:/dev/sdb [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd prepare ceph1:sde:/dev/sdb [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph1:/dev/sde:/dev/sdb [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise [ceph_deploy.osd][DEBUG ] Deploying osd to ceph1 [ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph1][WARNIN] osd keyring does not exist yet, creating one [ceph_deploy.osd][ERROR ] OSError: [Errno 2] No such file or directory [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs Step 2 - We then ran the following commands: Deploy-node : ceph-deploy uninstall ceph1 Ceph1-node : sudo rm –fr /etc/ceph/* Deploy-node : ceph-deploy gatherkeys ceph1 Deploy-node : ceph-deploy -v install ceph1 Step 3 - Apparently the problem was solved: Deploy-node :
ceph-deploy -v osd prepare ceph1:sde:/dev/sdb openstack@monitor3:~/cluster1$ ceph-deploy -v osd prepare ceph1:sde:/dev/sdb [ceph_deploy.cli][INFO ] Invoked (1.3): /usr/bin/ceph-deploy -v osd prepare ceph1:sde:/dev/sdb [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ceph1:/dev/sde:/dev/sdb [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 12.04 precise [ceph_deploy.osd][DEBUG ] Deploying osd to ceph1 [ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph1][WARNIN] osd keyring does not exist yet, creating one [ceph1][INFO ] Running command: sudo udevadm trigger --subsystem-match=block --action=""> [ceph_deploy.osd][DEBUG ] Preparing host ceph1 disk /dev/sde journal /dev/sdb activate False [ceph1][INFO ] Running command: sudo ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sde /dev/sdb [ceph1][ERROR ] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data [ceph1][DEBUG ] Information: Moved requested sector from 34 to 2048 in [ceph1][DEBUG ] order to align on 2048-sector boundaries. [ceph1][DEBUG ] The operation has completed successfully. [ceph1][DEBUG ] Information: Moved requested sector from 34 to 2048 in [ceph1][DEBUG ] order to align on 2048-sector boundaries. [ceph1][DEBUG ] The operation has completed successfully. [ceph1][DEBUG ] meta-data="" isize=2048 agcount=4, agsize=61047597 blks [ceph1][DEBUG ] = sectsz=512 attr=2, projid32bit=0 [ceph1][DEBUG ] data = bsize=4096 blocks=244190385, imaxpct=25 [ceph1][DEBUG ] = sunit=0 swidth=0 blks [ceph1][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 [ceph1][DEBUG ] log =internal log bsize=4096 blocks=119233, version=2 [ceph1][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1 [ceph1][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0 [ceph1][DEBUG ] The operation has completed successfully. [ceph_deploy.osd][DEBUG ] Host ceph1 is now ready for osd use. Thanks! Mike This message contains information which may be confidential and/or privileged. Unless you are the intended recipient (or authorized to receive for the intended recipient), you may not read, use, copy or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail and delete the message and any attachment(s) thereto without retaining any copies. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
