Am 29.06.2016 um 04:30 schrieb Alex Gorbachev: > Hi Stefan, > > On Tue, Jun 28, 2016 at 1:46 PM, Stefan Priebe - Profihost AG > <s.priebe@xxxxxxxxxxxx> wrote: >> Please be aware that you may need even more patches. Overall this needs 3 >> patches. Where the first two try to fix a bug and the 3rd one fixes the >> fixes + even more bugs related to the scheduler. I've no idea on which patch >> level Ubuntu is. > > Stefan, would you be able to please point to the other two patches > beside https://lkml.org/lkml/diff/2016/6/22/102/1 ? Sorry sure yes: 1. 2b8c41daba32 ("sched/fair: Initiate a new task's util avg to a bounded value") 2.) 40ed9cba24bb7e01cc380a02d3f04065b8afae1d ("sched/fair: Fix post_init_entity_util_avg() serialization") 3.) the one listed at lkml. Stefan > > Thank you, > Alex > >> >> Stefan >> >> Excuse my typo sent from my mobile phone. >> >> Am 28.06.2016 um 17:59 schrieb Tim Bishop <tim-lists@xxxxxxxxxxx>: >> >> Yes - I noticed this today on Ubuntu 16.04 with the default kernel. No >> useful information to add other than it's not just you. >> >> Tim. >> >> On Tue, Jun 28, 2016 at 11:05:40AM -0400, Alex Gorbachev wrote: >> >> After upgrading to kernel 4.4.13 on Ubuntu, we are seeing a few of >> >> these issues where an OSD would fail with the stack below. I logged a >> >> bug at https://bugzilla.kernel.org/show_bug.cgi?id=121101 and there is >> >> a similar description at https://lkml.org/lkml/2016/6/22/102, but the >> >> odd part is we have turned off CFQ and blk-mq/scsi-mq and are using >> >> just the noop scheduler. >> >> >> Does the ceph kernel code somehow use the fair scheduler code block? >> >> >> Thanks >> >> -- >> >> Alex Gorbachev >> >> Storcium >> >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.684974] CPU: 30 PID: >> >> 10403 Comm: ceph-osd Not tainted 4.4.13-040413-generic #201606072354 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.684991] Hardware name: >> >> Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.2 >> >> 03/04/2015 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685009] task: >> >> ffff880f79df8000 ti: ffff880f79fb8000 task.ti: ffff880f79fb8000 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685024] RIP: >> >> 0010:[<ffffffff810b416e>] [<ffffffff810b416e>] >> >> task_numa_find_cpu+0x22e/0x6f0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685051] RSP: >> >> 0018:ffff880f79fbb818 EFLAGS: 00010206 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685063] RAX: >> >> 0000000000000000 RBX: ffff880f79fbb8b8 RCX: 0000000000000000 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685076] RDX: >> >> 0000000000000000 RSI: 0000000000000000 RDI: ffff8810352d4800 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685107] RBP: >> >> ffff880f79fbb880 R08: 00000001020cf87c R09: 0000000000ff00ff >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685150] R10: >> >> 0000000000000009 R11: 0000000000000006 R12: ffff8807c3adc4c0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685194] R13: >> >> 0000000000000006 R14: 000000000000033e R15: fffffffffffffec7 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685238] FS: >> >> 00007f30e46b8700(0000) GS:ffff88105f580000(0000) >> >> knlGS:0000000000000000 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685283] CS: 0010 DS: >> >> 0000 ES: 0000 CR0: 0000000080050033 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685310] CR2: >> >> 000000001321a000 CR3: 0000000853598000 CR4: 00000000000406e0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685354] Stack: >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685374] >> >> ffffffff813d050f 000000000000000d 0000000000000045 ffff880f79df8000 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685426] >> >> 000000000000033f 0000000000000000 0000000000016b00 000000000000033f >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685477] >> >> ffff880f79df8000 ffff880f79fbb8b8 00000000000001f4 0000000000000054 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685528] Call Trace: >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685555] >> >> [<ffffffff813d050f>] ? cpumask_next_and+0x2f/0x40 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685584] >> >> [<ffffffff810b4a6e>] task_numa_migrate+0x43e/0x9b0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685613] >> >> [<ffffffff810b3acc>] ? update_cfs_shares+0xbc/0x100 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685642] >> >> [<ffffffff810b5059>] numa_migrate_preferred+0x79/0x80 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685672] >> >> [<ffffffff810b9b94>] task_numa_fault+0x7f4/0xd40 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685700] >> >> [<ffffffff813d9634>] ? timerqueue_del+0x24/0x70 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685729] >> >> [<ffffffff810b9205>] ? should_numa_migrate_memory+0x55/0x130 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685762] >> >> [<ffffffff811bd590>] handle_mm_fault+0xbc0/0x1820 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685793] >> >> [<ffffffff810edc00>] ? __hrtimer_init+0x90/0x90 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685822] >> >> [<ffffffff810c211d>] ? remove_wait_queue+0x4d/0x60 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685853] >> >> [<ffffffff8121e20a>] ? poll_freewait+0x4a/0xa0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685882] >> >> [<ffffffff8106a537>] __do_page_fault+0x197/0x400 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685910] >> >> [<ffffffff8106a7c2>] do_page_fault+0x22/0x30 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685939] >> >> [<ffffffff8180a878>] page_fault+0x28/0x30 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685967] >> >> [<ffffffff813e4c5f>] ? copy_page_to_iter_iovec+0x5f/0x300 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.685997] >> >> [<ffffffff810b2795>] ? select_task_rq_fair+0x625/0x700 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686026] >> >> [<ffffffff813e4f16>] copy_page_to_iter+0x16/0xa0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686056] >> >> [<ffffffff816f02ad>] skb_copy_datagram_iter+0x14d/0x280 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686087] >> >> [<ffffffff8174a503>] tcp_recvmsg+0x613/0xbe0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686117] >> >> [<ffffffff8177844e>] inet_recvmsg+0x7e/0xb0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686146] >> >> [<ffffffff816e0d3b>] sock_recvmsg+0x3b/0x50 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686173] >> >> [<ffffffff816e0f91>] SYSC_recvfrom+0xe1/0x160 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686202] >> >> [<ffffffff810f36f5>] ? ktime_get_ts64+0x45/0xf0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686230] >> >> [<ffffffff816e239e>] SyS_recvfrom+0xe/0x10 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686259] >> >> [<ffffffff818086f2>] entry_SYSCALL_64_fastpath+0x16/0x71 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686287] Code: 55 b0 4c >> >> 89 f7 e8 53 cd ff ff 48 8b 55 b0 49 8b 4e 78 48 8b 82 d8 01 00 00 48 >> >> 83 c1 01 31 d2 49 0f af 86 b0 00 00 00 4c 8b 73 78 <48> f7 f1 48 8b 4b >> >> 20 49 89 c0 48 29 c1 48 8b 45 d0 4c 03 43 48 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686512] RIP >> >> [<ffffffff810b416e>] task_numa_find_cpu+0x22e/0x6f0 >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686544] RSP >> <ffff880f79fbb818> >> >> Jun 28 09:46:41 roc04r-sca090 kernel: [137912.686896] ---[ end trace >> >> 544cb9f68cb55c93 ]--- >> >> Jun 28 09:52:15 roc04r-sca090 kernel: [138246.669713] mpt2sas_cm0: >> >> log_info(0x30030101): originator(IOP), code(0x03), sub_code(0x0101) >> >> Jun 28 09:55:01 roc0 >> >> _______________________________________________ >> >> ceph-users mailing list >> >> ceph-users@xxxxxxxxxxxxxx >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> Tim. >> >> -- >> Tim Bishop >> http://www.bishnet.net/tim/ >> PGP Key: 0x6C226B37FDF38D55 >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |