Re: Issue with Ceph File System and LIO

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



With cephfs.patch and cephfs1.patch applied and I am now seeing:

[Thu Dec 17 14:27:59 2015] ------------[ cut here ]------------
[Thu Dec 17 14:27:59 2015] WARNING: CPU: 0 PID: 3036 at
fs/ceph/addr.c:1171 ceph_write_begin+0xfb/0x120 [ceph]()
[Thu Dec 17 14:27:59 2015] Modules linked in: iscsi_target_mod
vhost_scsi tcm_qla2xxx ib_srpt tcm_fc tcm_usb_gadget tcm_loop
target_core_file target_core_iblock target_core_pscsi target_core_user
target_core_mod ipmi_devintf vhost qla2xxx ib_cm ib_sa ib_mad ib_core
ib_addr libfc scsi_transport_fc libcomposite udc_core uio configfs ttm
drm_kms_helper drm ipmi_ssif coretemp gpio_ich i2c_algo_bit kvm
fb_sys_fops syscopyarea sysfillrect sysimgblt shpchp input_leds ceph
irqbypass i7core_edac serio_raw hpilo edac_core ipmi_si
ipmi_msghandler 8250_fintek lpc_ich acpi_power_meter libceph mac_hid
libcrc32c fscache bonding lp parport mlx4_en vxlan ip6_udp_tunnel
udp_tunnel ptp pps_core hid_generic usbhid hid mlx4_core hpsa psmouse
bnx2 fjes scsi_transport_sas [last unloaded: target_core_mod]
[Thu Dec 17 14:27:59 2015] CPU: 0 PID: 3036 Comm: iscsi_trx Tainted: G
       W I     4.4.0-rc4-ede2 #1
[Thu Dec 17 14:27:59 2015] Hardware name: HP ProLiant DL360 G6, BIOS
P64 01/22/2015
[Thu Dec 17 14:27:59 2015]  ffffffffc02b2e37 ffff880c0289b958
ffffffff813ad644 0000000000000000
[Thu Dec 17 14:27:59 2015]  ffff880c0289b990 ffffffff81079702
ffff880c0289ba50 0000000846c21000
[Thu Dec 17 14:27:59 2015]  ffff880c009ea200 0000000000001000
ffffea00122ed700 ffff880c0289b9a0
[Thu Dec 17 14:27:59 2015] Call Trace:
[Thu Dec 17 14:27:59 2015]  [<ffffffff813ad644>] dump_stack+0x44/0x60
[Thu Dec 17 14:27:59 2015]  [<ffffffff81079702>] warn_slowpath_common+0x82/0xc0
[Thu Dec 17 14:27:59 2015]  [<ffffffff810797fa>] warn_slowpath_null+0x1a/0x20
[Thu Dec 17 14:27:59 2015]  [<ffffffffc028b41b>]
ceph_write_begin+0xfb/0x120 [ceph]
[Thu Dec 17 14:27:59 2015]  [<ffffffff8117c8df>]
generic_perform_write+0xbf/0x1a0
[Thu Dec 17 14:27:59 2015]  [<ffffffffc0285f9c>]
ceph_write_iter+0xf5c/0x1010 [ceph]
[Thu Dec 17 14:27:59 2015]  [<ffffffff817c3396>] ? __schedule+0x386/0x9c0
[Thu Dec 17 14:27:59 2015]  [<ffffffff817c3a05>] ? schedule+0x35/0x80
[Thu Dec 17 14:27:59 2015]  [<ffffffff811d7c65>] ? __slab_free+0xb5/0x290
[Thu Dec 17 14:27:59 2015]  [<ffffffff813c0003>] ?
iov_iter_get_pages+0x113/0x210
[Thu Dec 17 14:27:59 2015]  [<ffffffff811f6c93>] vfs_iter_write+0x63/0xa0
[Thu Dec 17 14:27:59 2015]  [<ffffffffc02d2f29>]
fd_do_rw.isra.5+0xc9/0x1b0 [target_core_file]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc02d30d5>]
fd_execute_rw+0xc5/0x2a0 [target_core_file]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc0430e72>]
sbc_execute_rw+0x22/0x30 [target_core_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc042f9cf>]
__target_execute_cmd+0x1f/0x70 [target_core_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc0430525>]
target_execute_cmd+0x195/0x2a0 [target_core_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc05b778a>]
iscsit_execute_cmd+0x20a/0x270 [iscsi_target_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc05c08da>]
iscsit_sequence_cmd+0xda/0x190 [iscsi_target_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffffc05c6c4d>]
iscsi_target_rx_thread+0x51d/0xe30 [iscsi_target_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffff8101463d>] ? __switch_to+0x1cd/0x570
[Thu Dec 17 14:27:59 2015]  [<ffffffffc05c6730>] ?
iscsi_target_tx_thread+0x1c0/0x1c0 [iscsi_target_mod]
[Thu Dec 17 14:27:59 2015]  [<ffffffff810974c9>] kthread+0xc9/0xe0
[Thu Dec 17 14:27:59 2015]  [<ffffffff81097400>] ?
kthread_create_on_node+0x180/0x180
[Thu Dec 17 14:27:59 2015]  [<ffffffff817c794f>] ret_from_fork+0x3f/0x70
[Thu Dec 17 14:27:59 2015]  [<ffffffff81097400>] ?
kthread_create_on_node+0x180/0x180
[Thu Dec 17 14:27:59 2015] ---[ end trace 8346192e3f29ed5d ]---

Each of the WARNING on line 1171 is followed by a WARNING on line 125.
The dmesg output is attached to the tracker ticket 14086

Regards,
Eric

On Thu, Dec 17, 2015 at 2:38 AM, Yan, Zheng <ukernel@xxxxxxxxx> wrote:
> On Thu, Dec 17, 2015 at 4:56 PM, Eric Eastman
> <eric.eastman@xxxxxxxxxxxxxx> wrote:
>> I patched the 4.4rc4 kernel source and restarted the test.  Shortly
>> after starting it, this showed up in dmesg:
>>
>> [Thu Dec 17 03:29:55 2015] WARNING: CPU: 0 PID: 2547 at
>> fs/ceph/addr.c:1162 ceph_write_begin+0xfb/0x120 [ceph]()
>> [Thu Dec 17 03:29:55 2015] Modules linked in: iscsi_target_mod
>> vhost_scsi tcm_qla2xxx ib_srpt tcm_fc tcm_usb_gadget tcm_loop
>> target_core_file target_core_iblock target_core_pscsi target_core_user
>> target_core_mod ipmi_devintf vhost qla2xxx ib_cm ib_sa ib_mad ib_core
>> ib_addr libfc scsi_transport_fc libcomposite udc_core uio configfs ttm
>> ipmi_ssif drm_kms_helper drm coretemp kvm gpio_ich i2c_algo_bit
>> i7core_edac fb_sys_fops syscopyarea edac_core sysfillrect sysimgblt
>> ipmi_si input_leds hpilo ipmi_msghandler shpchp acpi_power_meter
>> irqbypass serio_raw 8250_fintek lpc_ich mac_hid ceph bonding libceph
>> lp parport libcrc32c fscache mlx4_en vxlan ip6_udp_tunnel udp_tunnel
>> ptp pps_core hid_generic usbhid hid mlx4_core hpsa psmouse bnx2 fjes
>> scsi_transport_sas [last unloaded: target_core_mod]
>> [Thu Dec 17 03:29:55 2015] CPU: 0 PID: 2547 Comm: iscsi_trx Tainted: G
>>        W I     4.4.0-rc4-ede1 #1
>> [Thu Dec 17 03:29:55 2015] Hardware name: HP ProLiant DL360 G6, BIOS
>> P64 01/22/2015
>> [Thu Dec 17 03:29:55 2015]  ffffffffc020cd47 ffff8805f1e97958
>> ffffffff813ad644 0000000000000000
>> [Thu Dec 17 03:29:55 2015]  ffff8805f1e97990 ffffffff81079702
>> ffff8805f1e97a50 00000000015dd000
>> [Thu Dec 17 03:29:55 2015]  ffff880c034df800 0000000000000200
>> ffffea0000b26a80 ffff8805f1e979a0
>> [Thu Dec 17 03:29:55 2015] Call Trace:
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff813ad644>] dump_stack+0x44/0x60
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff81079702>] warn_slowpath_common+0x82/0xc0
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff810797fa>] warn_slowpath_null+0x1a/0x20
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc01e53bb>]
>> ceph_write_begin+0xfb/0x120 [ceph]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff8117c8df>]
>> generic_perform_write+0xbf/0x1a0
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc01dff9c>]
>> ceph_write_iter+0xf5c/0x1010 [ceph]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff810a888c>] ? __enqueue_entity+0x6c/0x70
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff813c0003>] ?
>> iov_iter_get_pages+0x113/0x210
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff816b6802>] ?
>> skb_copy_datagram_iter+0x122/0x250
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff811f6c93>] vfs_iter_write+0x63/0xa0
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc03c3f29>]
>> fd_do_rw.isra.5+0xc9/0x1b0 [target_core_file]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc03c40d5>]
>> fd_execute_rw+0xc5/0x2a0 [target_core_file]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc0445e72>]
>> sbc_execute_rw+0x22/0x30 [target_core_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc04449cf>]
>> __target_execute_cmd+0x1f/0x70 [target_core_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc0445525>]
>> target_execute_cmd+0x195/0x2a0 [target_core_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc05c978a>]
>> iscsit_execute_cmd+0x20a/0x270 [iscsi_target_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc05d28da>]
>> iscsit_sequence_cmd+0xda/0x190 [iscsi_target_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc05d8c4d>]
>> iscsi_target_rx_thread+0x51d/0xe30 [iscsi_target_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff8101463d>] ? __switch_to+0x1cd/0x570
>> [Thu Dec 17 03:29:55 2015]  [<ffffffffc05d8730>] ?
>> iscsi_target_tx_thread+0x1c0/0x1c0 [iscsi_target_mod]
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff810974c9>] kthread+0xc9/0xe0
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff81097400>] ?
>> kthread_create_on_node+0x180/0x180
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff817c794f>] ret_from_fork+0x3f/0x70
>> [Thu Dec 17 03:29:55 2015]  [<ffffffff81097400>] ?
>> kthread_create_on_node+0x180/0x180
>> [Thu Dec 17 03:29:55 2015] ---[ end trace 382a45986961da4e ]---
>
>
> Could you please try the apply the new incremental patch and try again.
>
>
> Regards
> Yan, Zheng
>
>
>>
>> There are WARNINGs on both line 125 and 1162. I will attached the
>> whole set of dmesg output to the tracker ticket 14086
>>
>> I wanted to note that file system snapshots are enabled and being used
>> on this file system.
>>
>> Thanks
>> Eric
>>
>> On Wed, Dec 16, 2015 at 8:15 AM, Eric Eastman
>> <eric.eastman@xxxxxxxxxxxxxx> wrote:
>>>>>
>>>> This warning is really strange. Could you try the attached debug patch.
>>>>
>>>> Regards
>>>> Yan, Zheng
>>>
>>> I will try the patch and get back to the list.
>>>
>>> Eric
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux