subscribe > 在 2018年4月16日，04:01，ceph-users-request@xxxxxxxxxxxxxx 写道： > > Send ceph-users mailing list submissions to > ceph-users@xxxxxxxxxxxxxx > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > or, via email, send a message with subject or body 'help' to > ceph-users-request@xxxxxxxxxxxxxx > > You can reach the person managing the list at > ceph-users-owner@xxxxxxxxxxxxxx > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > > Today's Topics: > > 1. Re: Error Creating OSD (Rhian Resnick) > 2. ZeroDivisionError: float division by zero in > /usr/lib/ceph/mgr/dashboard/module.py (12.2.4) (Nicolas Huillard) > 3. High TCP retransmission rates, only with Ceph (Robert Stanford) > 4. Re: High TCP retransmission rates, only with Ceph (Pawe? Sadowski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 14 Apr 2018 21:17:32 +0000 > From: Rhian Resnick <rresnick@xxxxxxx> > To: Alfredo Deza <adeza@xxxxxxxxxx> > Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: Error Creating OSD > Message-ID: > <DM5PR0401MB362486BA8476827E92A0ED25CBB20@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> > > Content-Type: text/plain; charset="iso-8859-1" > > Afternoon, > > > Happily, I resolved this issue. > > > Running vgdisplay showed that ceph-volume tried to create a disk on failed disk. (We didn't know we had a bad did so this is information that was new to us) and when the command failed it left three bad volume groups. Since you cannot rename them you need to use the following command to delete them. > > > vgdisplay to find the bad volume groups > > vgremove --select vg_uuid=your uuid -f # -f forces it to be removed > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ________________________________ > From: Rhian Resnick > Sent: Saturday, April 14, 2018 12:47 PM > To: Alfredo Deza > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Error Creating OSD > > > Thanks all, > > > Here is a link to our our command being executed: https://pastebin.com/iy8iSaKH > > > > Here are the results from the command > > > Executed with debug enabled (after a zap with destroy) > > > [root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 664894a8-530a-4557-b2f4-1af5b391f2b7 > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > Traceback (most recent call last): > File "/sbin/ceph-volume", line 6, in <module> > main.Volume() > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in __init__ > self.main(self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 74, in main > self.create(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create > prepare_step.safe_prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 217, in safe_prepare > self.prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 283, in prepare > block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 193, in prepare_device > if api.get_vg(vg_name=vg_name): > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in get_vg > return vgs.get(vg_name=vg_name, vg_tags=vg_tags) > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in get > raise MultipleVGsError(vg_name) > ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ________________________________ > From: Alfredo Deza <adeza@xxxxxxxxxx> > Sent: Saturday, April 14, 2018 8:45 AM > To: Rhian Resnick > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Error Creating OSD > > > > On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick <rresnick@xxxxxxx<mailto:rresnick@xxxxxxx>> wrote: > > Evening, > > When attempting to create an OSD we receive the following error. > > [ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c8cb8cff-dad9-48b8-8d77-6f130a4b629d > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > --> MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > Any hints on what to do? This occurs when we attempt to create osd's on this node. > > Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? Also, if you could try the same command but with: > > CEPH_VOLUME_DEBUG=1 > > I think you are hitting two issues here: > > 1) Somehow `osd new` is not completing and failing > 2) The `purge` command to wipe out the LV is getting multiple LV's and cannot make sure to match the one it used. > > #2 definitely looks like something we are doing wrong, and #1 can have a lot of different causes. The logs would be tremendously helpful! > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx<mailto:ceph-users@xxxxxxxxxxxxxx> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20180414/ea2b5e6c/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Sun, 15 Apr 2018 14:46:46 +0200 > From: Nicolas Huillard <nhuillard@xxxxxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Subject: ZeroDivisionError: float division by zero in > /usr/lib/ceph/mgr/dashboard/module.py (12.2.4) > Message-ID: <1523796406.2801.33.camel@xxxxxxxxxxx> > Content-Type: text/plain; charset="UTF-8" > > Hi, > > I'm not sure if this have been solved since 12.2.4. The same code > occurs in a different file in Github https://github.com/ceph/ceph/blob/ > 50412f7e9c2691ec10132c8bf9310a05a40e9f9d/src/pybind/mgr/status/module.p > y > The ZeroDivisionError occurs when the dashboard is open, and there is a > network outage (link between 2 datacenters is broken). I'm not sure > about the behaviour of the actual UI in the dashboard at the same time. > > Syslog trace: > > ceph-mgr[1324]: [15/Apr/2018:09:47:12] HTTP Traceback (most recent call last): > ceph-mgr[1324]: File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in respond > ceph-mgr[1324]: response.body = self.handler() > ceph-mgr[1324]: File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 217, in __call__ > ceph-mgr[1324]: self.body = self.oldhandler(*args, **kwargs) > ceph-mgr[1324]: File "/usr/lib/python2.7/dist-packages/cherrypy/lib/jsontools.py", line 63, in json_handler > ceph-mgr[1324]: value = cherrypy.serving.request._json_inner_handler(*args, **kwargs) > ceph-mgr[1324]: File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 61, in __call__ > ceph-mgr[1324]: return self.callable(*self.args, **self.kwargs) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 991, in list_data > ceph-mgr[1324]: return self._osds_by_server() > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 1040, in _osds_by_server > ceph-mgr[1324]: osd_map.osds_by_id[osd_id]) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 1007, in _osd_summary > ceph-mgr[1324]: result['stats'][s.split(".")[1]] = global_instance().get_rate('osd', osd_spec, s) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 268, in get_rate > ceph-mgr[1324]: return (data[-1][1] - data[-2][1]) / float(data[-1][0] - data[-2][0]) > ceph-mgr[1324]: ZeroDivisionError: float division by zero > > HTH, > > -- > Nicolas Huillard > > > ------------------------------ > > Message: 3 > Date: Sun, 15 Apr 2018 13:18:37 -0500 > From: Robert Stanford <rstanford8896@xxxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Subject: High TCP retransmission rates, only with Ceph > Message-ID: > <CAAca39mo+840Q2JSkn-gKLOFcwWR_GvCDPnzfmxVt44+9sm9Ng@xxxxxxxxxxxxxx> > Content-Type: text/plain; charset="utf-8" > > Iperf gives about 7Gb/s between a radosgw host and one of my OSD hosts (8 > disks, 8 OSD daemons, one of 3 OSD hosts). When I benchmark radosgw with > cosbench I see high TCP retransmission rates (from sar -n ETCP 1). I don't > see this with iperf. Why would Ceph, but not iperf, cause high TCP > retransmission rates? > > Thank you > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20180415/07b492bd/attachment-0001.html> > > ------------------------------ > > Message: 4 > Date: Sun, 15 Apr 2018 20:48:40 +0200 > From: Pawe? Sadowski <ceph@xxxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Subject: Re: High TCP retransmission rates, only with > Ceph > Message-ID: <fef55a30-f13e-c8db-942d-3490a141666a@xxxxxxxxx> > Content-Type: text/plain; charset=utf-8; format=flowed > > On 04/15/2018 08:18 PM, Robert Stanford wrote: >> >> ?Iperf gives about 7Gb/s between a radosgw host and one of my OSD >> hosts (8 disks, 8 OSD daemons, one of 3 OSD hosts).? When I benchmark >> radosgw with cosbench I see high TCP retransmission rates (from sar -n >> ETCP 1).? I don't see this with iperf.? Why would Ceph, but not iperf, >> cause high TCP retransmission rates? > > Most probably your application (radosgw in this case) is not able to > process requests fast enough and some packets are dropped. > > -- > PS > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ------------------------------ > > End of ceph-users Digest, Vol 63, Issue 15 > ****************************************** _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |