|
Running ceph-create-keys -i cm as ceph results in the following line being repeated forever: admin_socket: exception getting command descriptions: [Errno 13] Permission denied Running sudo ceph-create-keys -i cm results in the following: INFO:ceph-create-keys:Talking to monitor... ... and it goes on forever
[ceph@cm my-cluster]$ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination [ceph@cm my-cluster]$
> > > On Fri, Feb 14, 2014 at 10:14 AM, Ron Gage <ron@xxxxxxxxxxx> wrote: > > Alfredo: > > As near as I can tell, ceph-create-keys is hanging - or taking a huge amount > > ot fime to finish. > > Correct! Do you have iptables enabled by any chance? > > Also it can be useful if you manually execute that create-keys command > and see the output. > > > > Segment from screen output: > > (Make sure that ALL ceph programs/processes are not running) > > > > [ceph@cm my-cluster]$ ceph-deploy mon create-initial cm > > ... > > [cm][DEBUG ] === mon.cm === > > [cm][DEBUG ] Starting Ceph mon.cm on cm... > > [cm][DEBUG ] Starting ceph-create-keys on cm... > > [cm][WARNIN] No data was received after 7 seconds, disconnecting... > > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > > /var/run/ceph/ceph-mon.cm.asok mon_status > > ... > > > > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > > /etc/ceph/ceph.client.admin.keyring > > [cm][DEBUG ] connected to host: cm > > [cm][DEBUG ] detect platform information from remote host > > [cm][DEBUG ] detect machine type > > [cm][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][WARNIN] Unable to find > > /etc/ceph/ceph.client.admin.keyring on ['cm'] > > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring > > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > > /var/lib/ceph/bootstrap-osd/ceph.keyring > > [cm][DEBUG ] connected to host: cm > > [cm][DEBUG ] detect platform information from remote host > > [cm][DEBUG ] detect machine type > > [cm][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][WARNIN] Unable to find > > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm'] > > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > > /var/lib/ceph/bootstrap-mds/ceph.keyring > > [cm][DEBUG ] connected to host: cm > > [cm][DEBUG ] detect platform information from remote host > > [cm][DEBUG ] detect machine type > > [cm][DEBUG ] fetch remote file > > [ceph_deploy.gatherkeys][WARNIN] Unable to find > > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm'] > > > > [ceph@cm my-cluster]$ ps ax | grep -i ceph > > 1298 ? Ss 0:00 sshd: ceph [priv] > > 1302 ? S 0:00 sshd: ceph@pts/0 > > 2058 pts/0 Sl 0:00 /usr/bin/ceph-mon -i cm --pid-file > > /var/run/ceph/mon.cm.pid -c /etc/ceph/ceph.conf > > 2071 pts/0 S 0:00 python /usr/sbin/ceph-create-keys -i cm > > 2081 pts/0 Sl 0:00 python /usr/bin/ceph --cluster=ceph --name=mon. > > --keyring=/var/lib/ceph/mon/ceph-cm/keyring auth get-or-create client.admin > > mon allow * osd allow * mds allow > > 2212 pts/0 S+ 0:00 grep -i ceph > > [ceph@cm my-cluster]$ > > > > > > > >> On February 14, 2014 at 9:41 AM Alfredo Deza <alfredo.deza@xxxxxxxxxxx> > >> wrote: > >> > >> > >> On Fri, Feb 14, 2014 at 9:38 AM, Ron Gage <ron@xxxxxxxxxxx> wrote: > >> > Thanks Alfredo! > >> > Would that be /etc/ceph/ceph.conf or ~/my-cluster/ceph.conf > >> > >> Good question, sorry for the ambiguity. From where you are running > >> ceph-deploy, you should have a ceph.conf there, you can edit > >> that one file and then call 'ceph-deploy config push {nodes}' where > >> {nodes} should be replaced with the server(s) to push the config to. > >> > >> > >> > > >> > Ron Gage > >> > > >> > > >> >> On February 14, 2014 at 9:31 AM Alfredo Deza <alfredo.deza@xxxxxxxxxxx> > >> >> wrote: > >> >> > >> >> > >> >> On Fri, Feb 14, 2014 at 9:08 AM, Ron Gage <ron@xxxxxxxxxxx> wrote: > >> >> > Please see the post you replied to. The admin keyring is not being > >> >> > created. > >> >> > > >> >> > Thanks! > >> >> > > >> >> > Ron > >> >> > >> >> There is no need to run other commands instead of create-initial. That > >> >> is the best way to get started with monitor deployment. > >> >> > >> >> From ceph-deploy's perspective, everything has been called correctly > >> >> and the output from the node looks correct. The next thing I would > >> >> suggest is to bump the log levels of the monitors in ceph.conf and > >> >> restart them so that you can inspect a bit better what is going on > >> >> there. > >> >> > >> >> In the [global] section of ceph.conf: > >> >> > >> >> mon debug = 10 > >> >> > >> >> > >> >> > >> >> > > >> >> > > >> >> > On February 14, 2014 at 8:54 AM Srinivasa Rao Ragolu > >> >> > <sragolu@xxxxxxxxxx> > >> >> > wrote: > >> >> > > >> >> > Follow the below link as it is > >> >> > > >> >> > http://karan-mj.blogspot.fi/2013/12/ceph-installation-part-2.html > >> >> > > >> >> > Thanks, > >> >> > Srinivas. > >> >> > > >> >> > > >> >> > On Fri, Feb 14, 2014 at 7:21 PM, Ron Gage <ron@xxxxxxxxxxx> wrote: > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > [ceph@cm my-cluster]$ ceph-deploy mon create cm > >> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon > >> >> > create > >> >> > cm > >> >> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm > >> >> > [ceph_deploy.mon][DEBUG ] detecting platform for host cm ... > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final > >> >> > [cm][DEBUG ] determining if provided host has same hostname in remote > >> >> > [cm][DEBUG ] get remote short hostname > >> >> > [cm][DEBUG ] deploying mon to cm > >> >> > [cm][DEBUG ] get remote short hostname > >> >> > [cm][DEBUG ] remote hostname: cm > >> >> > [cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf > >> >> > [cm][DEBUG ] create the mon path if it does not exist > >> >> > [cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done > >> >> > [cm][DEBUG ] create a done file to avoid re-doing the mon deployment > >> >> > [cm][DEBUG ] create the init path if it does not exist > >> >> > [cm][DEBUG ] locating the `service` executable... > >> >> > [cm][INFO ] Running command: sudo /sbin/service ceph -c > >> >> > /etc/ceph/ceph.conf > >> >> > start mon.cm > >> >> > [cm][DEBUG ] === mon.cm === > >> >> > [cm][DEBUG ] Starting Ceph mon.cm on cm... > >> >> > [cm][DEBUG ] Starting ceph-create-keys on cm... > >> >> > [cm][WARNIN] No data was received after 7 seconds, disconnecting... > >> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > >> >> > /var/run/ceph/ceph-mon.cm.asok mon_status > >> >> > [cm][DEBUG ] > >> >> > > >> >> > > >> >> > ******************************************************************************** > >> >> > [cm][DEBUG ] status for monitor: mon.cm > >> >> > [cm][DEBUG ] { > >> >> > [cm][DEBUG ] "election_epoch": 1, > >> >> > [cm][DEBUG ] "extra_probe_peers": [], > >> >> > [cm][DEBUG ] "monmap": { > >> >> > [cm][DEBUG ] "created": "0.000000", > >> >> > [cm][DEBUG ] "epoch": 1, > >> >> > [cm][DEBUG ] "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5", > >> >> > [cm][DEBUG ] "modified": "0.000000", > >> >> > [cm][DEBUG ] "mons": [ > >> >> > [cm][DEBUG ] { > >> >> > [cm][DEBUG ] "addr": " 172.24.12.91:6789/0", > >> >> > [cm][DEBUG ] "name": "cm", > >> >> > [cm][DEBUG ] "rank": 0 > >> >> > [cm][DEBUG ] } > >> >> > [cm][DEBUG ] ] > >> >> > [cm][DEBUG ] }, > >> >> > [cm][DEBUG ] "name": "cm", > >> >> > [cm][DEBUG ] "outside_quorum": [], > >> >> > [cm][DEBUG ] "quorum": [ > >> >> > [cm][DEBUG ] 0 > >> >> > [cm][DEBUG ] ], > >> >> > [cm][DEBUG ] "rank": 0, > >> >> > [cm][DEBUG ] "state": "leader", > >> >> > [cm][DEBUG ] "sync_provider": [] > >> >> > [cm][DEBUG ] } > >> >> > [cm][DEBUG ] > >> >> > > >> >> > > >> >> > ******************************************************************************** > >> >> > [cm][INFO ] monitor: mon.cm is running > >> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > >> >> > /var/run/ceph/ceph-mon.cm.asok mon_status > >> >> > [ceph@cm my-cluster]$ ^C > >> >> > > >> >> > [ceph@cm my-cluster]$ ceph-deploy gatherkeys cm > >> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy > >> >> > gatherkeys > >> >> > cm > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /etc/ceph/ceph.client.admin.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /etc/ceph/ceph.client.admin.keyring on ['cm'] > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm'] > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm'] > >> >> > [ceph@cm my-cluster]$ > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > On February 14, 2014 at 8:20 AM Ashish Chandra < > >> >> > mail.ashishchandra@xxxxxxxxx> wrote: > >> >> > > >> >> > Hi Ron, > >> >> > Can you try running : > >> >> > > >> >> > ceph-deploy mon create {ceph-node} > >> >> > > >> >> > ceph-deploy gatherkeys {ceph-node} > >> >> > > >> >> > > >> >> > as two different commands, instead of running > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > ceph-deploy mon create-initial > >> >> > > >> >> > and let me know if the problem still persists. > >> >> > > >> >> > Actually it is not able to find "client.admin.keyring" file as it has > >> >> > not > >> >> > been created yet. > >> >> > > >> >> > Thanks > >> >> > > >> >> > Ashish Chandra > >> >> > > >> >> > > >> >> > > >> >> > On Fri, Feb 14, 2014 at 6:15 PM, Ron Gage <ron@xxxxxxxxxxx> wrote: > >> >> > > >> >> > Greetings all! > >> >> > I am trying to deploy a proof of concept system for a replicated > >> >> > fault > >> >> > tolerant CIFS server cluster using CEPH as the back end. I am having > >> >> > significant trouble getting the CEPH part to install correctly. > >> >> > Specifically, I can't get the monitor to come up into a useful state > >> >> > due > >> >> > to > >> >> > the apparent lack of a keyring or two. Here is the background: > >> >> > > >> >> > Everything is running on a standalone VMware server with internal > >> >> > storage. > >> >> > The idea here is to see it work, not to see how well it performs. > >> >> > > >> >> > Current configuraton: 4 x OSD, 1 x monitor/manager > >> >> > OSD: 2 x CPU, 8 gig memory, internal only networking, 16 gig sda for > >> >> > boot/root, 160 gig sdb for CEPH store, Centos 6.5 x64 fully up to > >> >> > date, > >> >> > minimal install > >> >> > mon/mgr: 2 x CPU, 2 gig memory, internal and external networking, 16 > >> >> > gig > >> >> > sda, Centos 6.5 x64 fully up to date, minimal install > >> >> > > >> >> > Since I have made several attempts already to get this working, I > >> >> > execute > >> >> > the following commands from the mgr to reset everything (logged in as > >> >> > user > >> >> > ceph) > >> >> > ceph-deploy purgedata c1 c2 c3 c4 cm > >> >> > ceph-deploy forgetkeys > >> >> > ceph-deploy purge c1 c2 c3 c4 cm > >> >> > sudo rm -f /etc/ceph/* > >> >> > sudo rm -f /var/lib/ceph/mon/*.* > >> >> > cd ~/my-cluster > >> >> > rm -f * > >> >> > > >> >> > Now to deploy a straight forward single monitor node: > >> >> > ceph-deploy new cm > >> >> > > >> >> > And to add the public interface to the ceph.conf > >> >> > vi ceph.conf > >> >> > (add to end of global section) public network = 172.24.0.0/16 > >> >> > > >> >> > Now to install the ceph software in the 5 nodes: > >> >> > ceph-deploy install cm c1 c2 c3 c4 > >> >> > (waiting...waiting...waiting...) > >> >> > > >> >> > And here is where the problems begin - creating the initial monitor: > >> >> > [ceph@cm my-cluster]$ ceph-deploy mon create-initial > >> >> > [ceph_deploy.cli][INFO ] Invoked (1.3.5): /usr/bin/ceph-deploy mon > >> >> > create-initial > >> >> > [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cm > >> >> > [ceph_deploy.mon][DEBUG ] detecting platform for host cm ... > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final > >> >> > [cm][DEBUG ] determining if provided host has same hostname in remote > >> >> > [cm][DEBUG ] get remote short hostname > >> >> > [cm][DEBUG ] deploying mon to cm > >> >> > [cm][DEBUG ] get remote short hostname > >> >> > [cm][DEBUG ] remote hostname: cm > >> >> > [cm][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf > >> >> > [cm][DEBUG ] create the mon path if it does not exist > >> >> > [cm][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-cm/done > >> >> > [cm][DEBUG ] create a done file to avoid re-doing the mon deployment > >> >> > [cm][DEBUG ] create the init path if it does not exist > >> >> > [cm][DEBUG ] locating the `service` executable... > >> >> > [cm][INFO ] Running command: sudo /sbin/service ceph -c > >> >> > /etc/ceph/ceph.conf > >> >> > start mon.cm > >> >> > [cm][DEBUG ] === mon.cm === > >> >> > [cm][DEBUG ] Starting Ceph mon.cm on cm... > >> >> > [cm][DEBUG ] Starting ceph-create-keys on cm... > >> >> > [cm][WARNIN] No data was received after 7 seconds, disconnecting... > >> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > >> >> > /var/run/ceph/ceph-mon.cm.asok mon_status > >> >> > [cm][DEBUG ] > >> >> > > >> >> > > >> >> > ******************************************************************************** > >> >> > [cm][DEBUG ] status for monitor: mon.cm > >> >> > [cm][DEBUG ] { > >> >> > [cm][DEBUG ] "election_epoch": 1, > >> >> > [cm][DEBUG ] "extra_probe_peers": [], > >> >> > [cm][DEBUG ] "monmap": { > >> >> > [cm][DEBUG ] "created": "0.000000", > >> >> > [cm][DEBUG ] "epoch": 1, > >> >> > [cm][DEBUG ] "fsid": "b11875f0-e611-40b9-ae81-e7cba7586ed5", > >> >> > [cm][DEBUG ] "modified": "0.000000", > >> >> > [cm][DEBUG ] "mons": [ > >> >> > [cm][DEBUG ] { > >> >> > [cm][DEBUG ] "addr": " 172.24.12.91:6789/0", > >> >> > [cm][DEBUG ] "name": "cm", > >> >> > [cm][DEBUG ] "rank": 0 > >> >> > [cm][DEBUG ] } > >> >> > [cm][DEBUG ] ] > >> >> > [cm][DEBUG ] }, > >> >> > [cm][DEBUG ] "name": "cm", > >> >> > [cm][DEBUG ] "outside_quorum": [], > >> >> > [cm][DEBUG ] "quorum": [ > >> >> > [cm][DEBUG ] 0 > >> >> > [cm][DEBUG ] ], > >> >> > [cm][DEBUG ] "rank": 0, > >> >> > [cm][DEBUG ] "state": "leader", > >> >> > [cm][DEBUG ] "sync_provider": [] > >> >> > [cm][DEBUG ] } > >> >> > [cm][DEBUG ] > >> >> > > >> >> > > >> >> > ******************************************************************************** > >> >> > [cm][INFO ] monitor: mon.cm is running > >> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > >> >> > /var/run/ceph/ceph-mon.cm.asok mon_status > >> >> > [ceph_deploy.mon][INFO ] processing monitor mon.cm > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][INFO ] Running command: sudo ceph --cluster=ceph --admin-daemon > >> >> > /var/run/ceph/ceph-mon.cm.asok mon_status > >> >> > [ceph_deploy.mon][INFO ] mon.cm monitor has reached quorum! > >> >> > [ceph_deploy.mon][INFO ] all initial monitors are running and have > >> >> > formed > >> >> > quorum > >> >> > [ceph_deploy.mon][INFO ] Running gatherkeys... > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /etc/ceph/ceph.client.admin.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /etc/ceph/ceph.client.admin.keyring on ['cm'] > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /var/lib/ceph/bootstrap-osd/ceph.keyring on ['cm'] > >> >> > [ceph_deploy.gatherkeys][DEBUG ] Checking cm for > >> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring > >> >> > [cm][DEBUG ] connected to host: cm > >> >> > [cm][DEBUG ] detect platform information from remote host > >> >> > [cm][DEBUG ] detect machine type > >> >> > [cm][DEBUG ] fetch remote file > >> >> > [ceph_deploy.gatherkeys][WARNIN] Unable to find > >> >> > /var/lib/ceph/bootstrap-mds/ceph.keyring on ['cm'] > >> >> > [ceph@cm my-cluster]$ > >> >> > > >> >> > > >> >> > From my untrained eye, it looks like several key files aren't either > >> >> > being > >> >> > created or copied to the expected place. > >> >> > > >> >> > Contents of ~/my-cluster: > >> >> > [ceph@cm my-cluster]$ ls > >> >> > ceph.conf ceph.log ceph.mon.keyring epel-release-6-8.noarch.rpm > >> >> > > >> >> > Contents of /var/lib/ceph/mon/ceph-cm: > >> >> > > >> >> > [ceph@cm ceph-cm]$ ls -R .: done keyring store.db sysvinit > >> >> > > >> >> > ./store.db: 000016.sst 000021.sst CURRENT LOG 000018.sst 000022.log > >> >> > LOCK MANIFEST-000020 > >> >> > > >> >> > Contents of /etc/ceph: > >> >> > > >> >> > [ceph@cm ceph]$ ls > >> >> > ceph.client.admin.keyring.5704.tmp ceph.conf rbdmap > >> >> > > >> >> > As you can plainly see, I have followed the directions in the "quick > >> >> > start" > >> >> > deployment guide on ceph.com > >> >> > > >> >> > (http://ceph.com/docs/master/start/quick-ceph-deploy/#create-a-cluster). > >> >> > > >> >> > What am I missing? Are the directions incomplete (or wrong)? > >> >> > > >> >> > > >> >> > > >> >> > HELP! > >> >> > > >> >> > > >> >> > > >> >> > Ron Gage > >> >> > Westland, Michigan > >> >> > > >> >> > Please be sure to check out My Photography Website > >> >> > > >> >> > _______________________________________________ > >> >> > ceph-users mailing list > >> >> > ceph-users@xxxxxxxxxxxxxx > >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> >> > > >> >> > > >> >> > > >> >> > Ron Gage > >> >> > Westland, Michigan > >> >> > > >> >> > Please be sure to check out My Photography Website > >> >> > > >> >> > _______________________________________________ > >> >> > ceph-users mailing list > >> >> > ceph-users@xxxxxxxxxxxxxx > >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> >> > > >> >> > > >> >> > > >> >> > Ron Gage > >> >> > Westland, Michigan > >> >> > > >> >> > Please be sure to check out My Photography Website > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > ceph-users mailing list > >> >> > ceph-users@xxxxxxxxxxxxxx > >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> >> > > >> > Ron Gage > >> > Westland, Michigan > >> > > >> > Please be sure to check out My Photography Website > >> > > > Ron Gage > > Westland, Michigan > > > > Please be sure to check out My Photography Website > > |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
