On Wed, Nov 30, 2016 at 11:02 AM, Yan, Zheng <zyan@xxxxxxxxxx> wrote: > >> On 30 Nov 2016, at 17:06, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: >> >> On Wed, Nov 30, 2016 at 1:30 AM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: >>> On Tue, Nov 29, 2016 at 11:29 PM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: >>>> On Tue, Nov 29, 2016 at 4:03 PM, Yan, Zheng <zyan@xxxxxxxxxx> wrote: >>>>> >>>>>> On 29 Nov 2016, at 22:58, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: >>>>>> >>>>>> On Tue, Nov 29, 2016 at 2:59 PM, Yan, Zheng <zyan@xxxxxxxxxx> wrote: >>>>>>> >>>>>>>> On 29 Nov 2016, at 21:54, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: >>>>>>>> >>>>>>>> On Tue, Nov 29, 2016 at 12:49 PM, Yan, Zheng <zyan@xxxxxxxxxx> wrote: >>>>>>>>> Current code does not wake up sync waiter if osd replies error >>>>>>>>> or the request does not want ondisk ack. >>>>>>>> >>>>>>>> Hi Zheng, >>>>>>>> >>>>>>>> Did you mean to write "and" instead of "or" - "if osd replies error >>>>>>>> *and* the request does not want ondisk ack”? >>>>>>> >>>>>>> I mean ‘or’. The code does not wake up sync waiter in either of these two cases. >>>>>> >>>>>> It does wake the sync waiter (r_safe_completion) on error, but only >>>>>> if ondisk ack was requested. If ondisk ack wasn't requested, it's not >>>>>> woken up, success or error. >>>>>> >>>>>> By not setting CEPH_OSD_FLAG_ONDISK, the submitter is indicating that >>>>>> *any* reply is fine. Blocking on the safe reply instead of any reply >>>>>> in the !CEPH_OSD_FLAG_ONDISK case sounds wrong. >>>>>> >>>>>> What is the motivation for this patch? >>>>> >>>>> I saw fsstress hang at ceph_osdc_sync several times. >>>>> >>>>> [52310.533483] sysrq: SysRq : Show Blocked State >>>>> [52310.534057] task PC stack pid father >>>>> [52310.534653] fsstress D14656 2836 2834 0x00000000 >>>>> [52310.535568] ffff880036cff200 ffff88003e2dac80 0000000000002fa3 ffff88003e3142c0 >>>>> [52310.536529] ffff88003fd18318 ffffc900038b7d48 ffffffff819354cc ffffffff00000000 >>>>> [52310.537464] 000000013e314ae0 ffff88003e314890 ffff88003fd18318 ffff88003fd18300 >>>>> [52310.538396] Call Trace: >>>>> [52310.538762] [<ffffffff819354cc>] ? __schedule+0x6ac/0x880 >>>>> [52310.539326] [<ffffffff81935713>] schedule+0x73/0x90 >>>>> [52310.539840] [<ffffffff81939d21>] schedule_timeout+0x31/0x5d0 >>>>> [52310.540425] [<ffffffff810ac4a6>] ? mark_held_locks+0x76/0x90 >>>>> [52310.540980] [<ffffffff8193b447>] ? _raw_spin_unlock_irq+0x27/0x50 >>>>> [52310.541582] [<ffffffff810ac64d>] ? trace_hardirqs_on_caller+0x18d/0x1c0 >>>>> [52310.542217] [<ffffffff81936a1b>] wait_for_completion+0x8b/0x100 >>>>> [52310.542788] [<ffffffff8108dc10>] ? wake_up_q+0x70/0x70 >>>>> [52310.543380] [<ffffffffa00116b8>] ceph_osdc_sync+0xc8/0x150 [libceph] >>>>> [52310.543990] [<ffffffff811eaed0>] ? SyS_tee+0x3b0/0x3b0 >>>>> [52310.544569] [<ffffffffa005c3ea>] ceph_sync_fs+0x3a/0xb0 [ceph] >>>>> [52310.545158] [<ffffffff811eaeeb>] sync_fs_one_sb+0x1b/0x20 >>>>> [52310.545699] [<ffffffff811b931e>] iterate_supers+0x7e/0xe0 >>>>> [52310.546258] [<ffffffff811eb1e0>] sys_sync+0x50/0x80 >>>>> [52310.546849] [<ffffffff8193bb6a>] entry_SYSCALL_64_fastpath+0x18/0xad >>>> >>>> Did you happen to capture /osdc? It's probably the pool perm check >>>> thing >>> >>> osdc is complete empty. >> >> If you haven't already done so while investigating, could you please >> dump the offending @req fields? I want to make sure it's the pool perm >> check request… > > Sorry, I have already rebooted that machine. Hi Zheng, I took it a step further and also fixed libceph cancellation code: https://github.com/ceph/ceph-client/commit/c297eb42690b904fb5b78dd9ad001bafe25f49ec I squashed your patch to reduce rename churn. Let me know if that's OK with you. Thanks, Ilya -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
