>-----Original Message----- >From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx] >Sent: Wednesday, November 20, 2013 7:17 AM >To: Gruher, Joseph R >Cc: ceph-users@xxxxxxxxxxxxxx >Subject: Re: ceph-deploy disk zap fails but succeeds on retry > >On Mon, Nov 18, 2013 at 1:12 PM, Gruher, Joseph R ><joseph.r.gruher@xxxxxxxxx> wrote: >> >>>-----Original Message----- >>>From: Alfredo Deza [mailto:alfredo.deza@xxxxxxxxxxx] >>>Sent: Monday, November 18, 2013 6:34 AM >>>To: Gruher, Joseph R >>>Cc: ceph-users@xxxxxxxxxxxxxx >>>Subject: Re: ceph-deploy disk zap fails but succeeds on >>>retry >>> >>>I went ahead and created a ticket to track this, if you have any new >>>input, please make sure you add to the actual ticket: >>>http://tracker.ceph.com/issues/6793 >>> >>>Thanks for reporting the problem! >>> >> >> Will do! I should be bringing up a few different cluster configurations on this >hardware (we're doing some Ceph performance testing) so I may be able to >reproduce again and get more details. > >I am trying to replicate this but somehow failing... in what state where the >drives, e.g. did you have any partitions before starting? or was this like a new >drive out of the box that was put in there? > >So far I can only see it sometimes in 13.04 and not anywhere else Looking at the ceph-deploy disk list output I captured at the time (see below) it seems to be reporting partition data on the drives (/dev/sdd1 exists in this example). The last thing I did with the drives prior to deploying Emperor was some tests to baseline their performance with FIO, including a fair amount of write activity to the raw devices. As a result I would expect their initial state to have basically been junk data. Prior to that FIO testing the disks would have been OSD disks in a Dumpling cluster. ceph@joceph-admin01:/etc/ceph$ ceph-deploy disk list joceph02 [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk list joceph02 [joceph02][DEBUG ] connected to host: joceph02 [joceph02][DEBUG ] detect platform information from remote host [joceph02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring [ceph_deploy.osd][DEBUG ] Listing disks on joceph02... [joceph02][INFO ] Running command: sudo ceph-disk list [joceph02][DEBUG ] /dev/sda : [joceph02][DEBUG ] /dev/sda1 other, ext4, mounted on / [joceph02][DEBUG ] /dev/sda2 other [joceph02][DEBUG ] /dev/sda5 swap, swap [joceph02][DEBUG ] /dev/sdb other, unknown [joceph02][DEBUG ] /dev/sdc other, unknown [joceph02][DEBUG ] /dev/sdd : [joceph02][DEBUG ] /dev/sdd1 other [joceph02][DEBUG ] /dev/sde : [joceph02][DEBUG ] /dev/sde1 other [joceph02][DEBUG ] /dev/sdf : [joceph02][DEBUG ] /dev/sdf1 other ceph@joceph-admin01:/etc/ceph$ ceph-deploy disk zap joceph02:/dev/sdd [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk zap joceph02:/dev/sdd [ceph_deploy.osd][DEBUG ] zapping /dev/sdd on joceph02 [joceph02][DEBUG ] connected to host: joceph02 [joceph02][DEBUG ] detect platform information from remote host [joceph02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring [joceph02][DEBUG ] zeroing last few blocks of device [joceph02][INFO ] Running command: sudo sgdisk --zap-all --clear --mbrtogpt -- /dev/sdd [joceph02][ERROR ] Caution: invalid main GPT header, but valid backup; regenerating main header [joceph02][ERROR ] from backup! [joceph02][ERROR ] [joceph02][ERROR ] Warning! Main partition table CRC mismatch! Loaded backup partition table [joceph02][ERROR ] instead of main partition table! [joceph02][ERROR ] [joceph02][ERROR ] Warning! One or more CRCs don't match. You should repair the disk! [joceph02][ERROR ] [joceph02][ERROR ] Invalid partition data! [joceph02][DEBUG ] Caution! After loading partitions, the CRC doesn't check out! [joceph02][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or [joceph02][DEBUG ] other utilities. [joceph02][DEBUG ] Information: Creating fresh partition table; will override earlier problems! [joceph02][DEBUG ] Non-GPT disk; not saving changes. Use -g to override. [joceph02][ERROR ] Traceback (most recent call last): [joceph02][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/process.py", line 68, in run [joceph02][ERROR ] reporting(conn, result, timeout) [joceph02][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/log.py", line 13, in reporting [joceph02][ERROR ] received = result.receive(timeout) [joceph02][ERROR ] File "/usr/lib/python2.7/dist-packages/ceph_deploy/lib/remoto/lib/execnet/gateway_base.py", line 455, in receive [joceph02][ERROR ] raise self._getremoteerror() or EOFError() [joceph02][ERROR ] RemoteError: Traceback (most recent call last): [joceph02][ERROR ] File "<string>", line 806, in executetask [joceph02][ERROR ] File "", line 35, in _remote_run [joceph02][ERROR ] RuntimeError: command returned non-zero exit status: 3 [joceph02][ERROR ] [joceph02][ERROR ] [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: sgdisk --zap-all --clear --mbrtogpt -- /dev/sdd ceph@joceph-admin01:/etc/ceph$ ceph-deploy disk list joceph02 [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk list joceph02 [joceph02][DEBUG ] connected to host: joceph02 [joceph02][DEBUG ] detect platform information from remote host [joceph02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring [ceph_deploy.osd][DEBUG ] Listing disks on joceph02... [joceph02][INFO ] Running command: sudo ceph-disk list [joceph02][DEBUG ] /dev/sda : [joceph02][DEBUG ] /dev/sda1 other, ext4, mounted on / [joceph02][DEBUG ] /dev/sda2 other [joceph02][DEBUG ] /dev/sda5 swap, swap [joceph02][DEBUG ] /dev/sdb other, unknown [joceph02][DEBUG ] /dev/sdc other, unknown [joceph02][DEBUG ] /dev/sdd other, unknown [joceph02][DEBUG ] /dev/sde : [joceph02][DEBUG ] /dev/sde1 other [joceph02][DEBUG ] /dev/sdf : [joceph02][DEBUG ] /dev/sdf1 other ceph@joceph-admin01:/etc/ceph$ ceph-deploy disk zap joceph02:/dev/sdd [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk zap joceph02:/dev/sdd [ceph_deploy.osd][DEBUG ] zapping /dev/sdd on joceph02 [joceph02][DEBUG ] connected to host: joceph02 [joceph02][DEBUG ] detect platform information from remote host [joceph02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring [joceph02][DEBUG ] zeroing last few blocks of device [joceph02][INFO ] Running command: sudo sgdisk --zap-all --clear --mbrtogpt -- /dev/sdd [joceph02][DEBUG ] Creating new GPT entries. [joceph02][DEBUG ] GPT data structures destroyed! You may now partition the disk using fdisk or [joceph02][DEBUG ] other utilities. [joceph02][DEBUG ] The operation has completed successfully. ceph@joceph-admin01:/etc/ceph$ ceph-deploy disk list joceph02 [ceph_deploy.cli][INFO ] Invoked (1.3.2): /usr/bin/ceph-deploy disk list joceph02 [joceph02][DEBUG ] connected to host: joceph02 [joceph02][DEBUG ] detect platform information from remote host [joceph02][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: Ubuntu 13.04 raring [ceph_deploy.osd][DEBUG ] Listing disks on joceph02... [joceph02][INFO ] Running command: sudo ceph-disk list [joceph02][DEBUG ] /dev/sda : [joceph02][DEBUG ] /dev/sda1 other, ext4, mounted on / [joceph02][DEBUG ] /dev/sda2 other [joceph02][DEBUG ] /dev/sda5 swap, swap [joceph02][DEBUG ] /dev/sdb other, unknown [joceph02][DEBUG ] /dev/sdc other, unknown [joceph02][DEBUG ] /dev/sdd other, unknown [joceph02][DEBUG ] /dev/sde : [joceph02][DEBUG ] /dev/sde1 other [joceph02][DEBUG ] /dev/sdf : [joceph02][DEBUG ] /dev/sdf1 other ceph@joceph-admin01:/etc/ceph$ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |