Thanks Dan, I have raised the tracker for this issue http://tracker.ceph.com/issues/15176
On Thu, Mar 17, 2016 at 10:47 AM, Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
Hi,
It's true, partprobe works intermittently. I extracted the key
commands to show the problem:
[18:44]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph
journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d
--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt --
/dev/sdc
The operation has completed successfully.
[18:44]# partprobe /dev/sdc
Error: Error informing the kernel about modifications to partition
/dev/sdc2 -- Device or resource busy. This means Linux won't know
about any changes you made to /dev/sdc2 until you reboot -- so you
shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 2 (Device or resource busy)
[18:44]# partprobe /dev/sdc
[18:44]# partprobe /dev/sdc
Error: Error informing the kernel about modifications to partition
/dev/sdc2 -- Device or resource busy. This means Linux won't know
about any changes you made to /dev/sdc2 until you reboot -- so you
shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 2 (Device or resource busy)
[18:44]# partprobe /dev/sdc
Error: Error informing the kernel about modifications to partition
/dev/sdc2 -- Device or resource busy. This means Linux won't know
about any changes you made to /dev/sdc2 until you reboot -- so you
shouldn't mount it or use it in any way before rebooting.
Error: Failed to add partition 2 (Device or resource busy)
But partx works every time:
[18:46]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph
journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d
--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt --
/dev/sdd
The operation has completed successfully.
[18:46]# partx -u /dev/sdd
[18:46]# partx -u /dev/sdd
[18:46]# partx -u /dev/sdd
[18:46]#
-- Dan
On Thu, Mar 17, 2016 at 6:31 PM, Vasu Kulkarni <vakulkar@xxxxxxxxxx> wrote:
> I can raise a tracker for this issue since it looks like an intermittent
> issue and mostly dependent on specific hardware or it would be better if you
> add all the hardware/os details in tracker.ceph.com, also from your logs it
> looks like you have
> Resource busy issue: Error: Failed to add partition 2 (Device or resource
> busy)
>
> From my test run logs on centos 7.2 , 10.0.5 (
> http://qa-proxy.ceph.com/teuthology/vasu-2016-03-15_15:34:41-selinux-master---basic-mira/62626/teuthology.log
> )
>
> 2016-03-15T18:49:56.305
> INFO:teuthology.orchestra.run.mira041.stderr:[ceph_deploy.osd][DEBUG ]
> Preparing host mira041 disk /dev/sdb journal None activate True
> 2016-03-15T18:49:56.305
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] find the
> location of an executable
> 2016-03-15T18:49:56.309
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][INFO ] Running
> command: sudo /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs --
> /dev/sdb
> 2016-03-15T18:49:56.546
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
> 2016-03-15T18:49:56.611
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster
> ceph
> 2016-03-15T18:49:56.643
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph
> 2016-03-15T18:49:56.708
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph
> 2016-03-15T18:49:56.708
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.709
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] set_type:
> Will colocate journal with data on /dev/sdb
> 2016-03-15T18:49:56.709
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-osd --cluster=ceph
> --show-config-value=osd_journal_size
> 2016-03-15T18:49:56.774
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.774
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.775
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.775
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
> osd_mkfs_options_xfs
> 2016-03-15T18:49:56.777
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
> osd_fs_mkfs_options_xfs
> 2016-03-15T18:49:56.809
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
> osd_mount_options_xfs
> 2016-03-15T18:49:56.841
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup
> osd_fs_mount_options_xfs
> 2016-03-15T18:49:56.857
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.858
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.858
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> ptype_tobe_for_name: name = journal
> 2016-03-15T18:49:56.859
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:56.859
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> create_partition: Creating journal partition num 2 size 5120 on /dev/sdb
> 2016-03-15T18:49:56.859
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M
> --change-name=2:ceph journal
> --partition-guid=2:d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198
> --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb
> 2016-03-15T18:49:57.927
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] The operation
> has completed successfully.
> 2016-03-15T18:49:57.927
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> update_partition: Calling partprobe on created device /dev/sdb
> 2016-03-15T18:49:57.928
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
> 2016-03-15T18:49:58.393
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command:
> Running command: /sbin/partprobe /dev/sdb
> 2016-03-15T18:49:58.393
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> command_check_call: Running command: /usr/bin/udevadm settle --timeout=600
> 2016-03-15T18:49:59.109
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:59.203
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:59.203
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid
> 2016-03-15T18:49:59.204
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> prepare_device: Journal is GPT partition
> /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198
> 2016-03-15T18:49:59.204
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> prepare_device: Journal is GPT partition
> /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198
> 2016-03-15T18:49:59.204
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:59.205
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> set_data_partition: Creating osd partition on /dev/sdb
> 2016-03-15T18:49:59.205
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:59.205
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> ptype_tobe_for_name: name = data
> 2016-03-15T18:49:59.206
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] get_dm_uuid:
> get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid
> 2016-03-15T18:49:59.206
> INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING]
> create_partition: Creating data partition num 1 size 0 on /dev/sdb
>
>
>
> On Thu, Mar 17, 2016 at 8:06 AM, Dan van der Ster <dan@xxxxxxxxxxxxxx>
> wrote:
>>
>> Hi,
>>
>> Is there a tracker for this? We just hit the same problem on 10.0.5.
>>
>> Cheers, Dan
>>
>> # rpm -q ceph
>> ceph-10.0.5-0.el7.x86_64
>>
>> # cat /etc/redhat-release
>> CentOS Linux release 7.2.1511 (Core)
>>
>> # ceph-disk -v prepare /dev/sdc
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph
>> --show-config-value=fsid
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_mkfs_type
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_mkfs_options_xfs
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_fs_mkfs_options_xfs
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_mount_options_xfs
>> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph
>> --show-config-value=osd_journal_size
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_cryptsetup_parameters
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_dmcrypt_key_size
>> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph
>> --name=osd. --lookup osd_dmcrypt_type
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> INFO:ceph-disk:Will colocate journal with data on /dev/sdc
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is
>> /sys/dev/block/8:32/dm/uuid
>> DEBUG:ceph-disk:Creating journal partition num 2 size 20480 on /dev/sdc
>> INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:20480M
>> --change-name=2:ceph journal
>> --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d
>> --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt --
>> /dev/sdc
>> The operation has completed successfully.
>> DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdc
>> INFO:ceph-disk:Running command: /usr/bin/udevadm settle
>> INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/sdc
>> Error: Error informing the kernel about modifications to partition
>> /dev/sdc2 -- Device or resource busy. This means Linux won't know
>> about any changes you made to /dev/sdc2 until you reboot -- so you
>> shouldn't mount it or use it in any way before rebooting.
>> Error: Failed to add partition 2 (Device or resource busy)
>> Traceback (most recent call last):
>> File "/usr/sbin/ceph-disk", line 3528, in <module>
>> main(sys.argv[1:])
>> File "/usr/sbin/ceph-disk", line 3482, in main
>> args.func(args)
>> File "/usr/sbin/ceph-disk", line 1817, in main_prepare
>> luks=luks
>> File "/usr/sbin/ceph-disk", line 1447, in prepare_journal
>> return prepare_journal_dev(data, journal, journal_size,
>> journal_uuid, journal_dm_keypath, cryptsetup_parameters, luks)
>> File "/usr/sbin/ceph-disk", line 1401, in prepare_journal_dev
>> raise Error(e)
>> __main__.Error: Error: Command '['/usr/sbin/partprobe', '/dev/sdc']'
>> returned non-zero exit status 1
>>
>> On Tue, Mar 15, 2016 at 8:38 PM, Vasu Kulkarni <vakulkar@xxxxxxxxxx>
>> wrote:
>> > Thanks for the steps that should be enough to test it out, I hope you
>> > got
>> > the latest ceph-deploy either from pip or throught github.
>> >
>> > On Tue, Mar 15, 2016 at 12:29 PM, Stephen Lord <Steve.Lord@xxxxxxxxxxx>
>> > wrote:
>> >>
>> >> I would have to nuke my cluster right now, and I do not have a spare
>> >> one..
>> >>
>> >> The procedure though is literally this, given a 3 node redhat 7.2
>> >> cluster,
>> >> ceph00, ceph01 and ceph02
>> >>
>> >> ceph-deploy install --testing ceph00 ceph01 ceph02
>> >> ceph-deploy new ceph00 ceph01 ceph02
>> >>
>> >> ceph-deploy mon create ceph00 ceph01 ceph02
>> >> ceph-deploy gatherkeys ceph00
>> >>
>> >> ceph-deploy osd create ceph00:sdb:/dev/sdi
>> >> ceph-deploy osd create ceph00:sdc:/dev/sdi
>> >>
>> >> All devices have their partition tables wiped before this. They are all
>> >> just SATA devices, no special devices in the way.
>> >>
>> >> sdi is an ssd and it is being carved up for journals. The first osd
>> >> create
>> >> works, the second one gets stuck in a loop in the update_partition call
>> >> in
>> >> ceph_disk for the 5 iterations before it gives up. When I look in
>> >> /sys/block/sdi the partition for the first osd is visible, the one for
>> >> the
>> >> second is not. However looking at /proc/partitions it sees the correct
>> >> thing. So something about partprobe is not kicking udev into doing the
>> >> right
>> >> thing when the second partition is added I suspect.
>> >>
>> >> If I do not use the separate journal device then it usually works, but
>> >> occasionally I see a single retry in that same loop.
>> >>
>> >> There is code in ceph_deploy which uses partprobe or partx depending on
>> >> which distro it detects, that is how I worked out what to change here.
>> >>
>> >> If I have to tear things down again I will reproduce and post here.
>> >>
>> >> Steve
>> >>
>> >> > On Mar 15, 2016, at 2:12 PM, Vasu Kulkarni <vakulkar@xxxxxxxxxx>
>> >> > wrote:
>> >> >
>> >> > Do you mind giving the full failed logs somewhere in fpaste.org along
>> >> > with some os version details?
>> >> > There are some known issues on RHEL, If you use 'osd prepare' and
>> >> > 'osd
>> >> > activate'(specifying just the journal partition here) it might work
>> >> > better.
>> >> >
>> >> > On Tue, Mar 15, 2016 at 12:05 PM, Stephen Lord
>> >> > <Steve.Lord@xxxxxxxxxxx>
>> >> > wrote:
>> >> > Not multipath if you mean using the multipath driver, just trying to
>> >> > setup OSDs which use a data disk and a journal ssd. If I run just a
>> >> > disk
>> >> > based OSD and only specify one device to ceph-deploy then it usually
>> >> > works
>> >> > although sometimes has to retry. In the case where I am using it to
>> >> > carve an
>> >> > SSD into several partitions for journals it fails on the second one.
>> >> >
>> >> > Steve
>> >> >
>> >>
>> >>
>> >> ----------------------------------------------------------------------
>> >> The information contained in this transmission may be confidential. Any
>> >> disclosure, copying, or further distribution of confidential
>> >> information is
>> >> not permitted unless such privilege is explicitly granted in writing by
>> >> Quantum. Quantum reserves the right to have electronic communications,
>> >> including email and attachments, sent across its networks filtered
>> >> through
>> >> anti virus and spam software programs and retain such messages in order
>> >> to
>> >> comply with applicable data security and retention requirements.
>> >> Quantum is
>> >> not responsible for the proper and complete transmission of the
>> >> substance of
>> >> this communication or for any delay in its receipt.
>> >
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
