On 2018/03/12 19:22, Tetsuo Handa wrote: > On 2018/03/12 3:11, Mikhail Gavrilov wrote: >> $ uname -a >> Linux localhost.localdomain 4.15.7-300.fc27.x86_64+debug #1 SMP Wed >> Feb 28 17:32:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >> >> >> How reproduce: >> 1. start virtual machine >> 2. open https://oom.sy24.ru/ in Firefox which will helps occurred OOM. >> Sorry I can't attach here html page because my message will rejected >> as message would contained HTML subpart. >> >> Actual result virtual machine hang and even couldn't be force off. >> >> Expected result virtual machine continue work. > > Looks like similar problem with http://lkml.kernel.org/r/20180212124359.GB3443@xxxxxxxxxxxxxx . > We are waiting for Michal Hocko to come back. > > Since Michal Hocko is still busy, I made a bit of debug patch. Can you try SysRq-i when you noticed io_schedule() deadlock? The intent of doing so is to test whether the reason of sleeping is that someone who is responsible to wake up failed to wake up or not. If someone failed to wake up, io_schedule() deadlock situation will continue because of uninterruptible sleep. If the reason is out of memory, terminating all processes should reclaim memory and allow someone who is responsible to wake up to wake up and io_schedule() deadlock situation should be solved. Also, can you try a debug patch shown below? This patch calls dump_page() in order to help understand state of the page in question. If someone failed to unlock the page, this patch should report forever. If someone failed to wake up, this patch should report only once (because of sleep with timeout). ---------------------------------------- diff --git a/mm/filemap.c b/mm/filemap.c index 693f622..192ed67f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -36,6 +36,9 @@ #include <linux/cleancache.h> #include <linux/shmem_fs.h> #include <linux/rmap.h> +#include <linux/oom.h> /* oom_lock */ +#include <linux/sched/debug.h> /* sched_show_task() */ +#include <linux/sched/sysctl.h> /* sysctl_hung_task_timeout_secs */ #include "internal.h" #define CREATE_TRACE_POINTS @@ -1076,6 +1079,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; int ret = 0; + unsigned long start = jiffies; init_wait(wait); wait->flags = lock ? WQ_FLAG_EXCLUSIVE : 0; @@ -1084,6 +1088,10 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, wait_page.bit_nr = bit_nr; for (;;) { + unsigned long timeout = sysctl_hung_task_timeout_secs; + if (!timeout) + timeout = 60; + spin_lock_irq(&q->lock); if (likely(list_empty(&wait->entry))) { @@ -1095,8 +1103,17 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, spin_unlock_irq(&q->lock); - if (likely(test_bit(bit_nr, &page->flags))) { - io_schedule(); + if (likely(test_bit(bit_nr, &page->flags)) && + io_schedule_timeout(timeout * HZ) == 0) { + char buf[32]; + struct task_struct *t = current; + snprintf(buf, sizeof(buf), "stuck on bit %u", bit_nr); + mutex_lock(&oom_lock); + pr_err("INFO: task %s:%d blocked for %lu seconds.\n", + t->comm, t->pid, (jiffies - start) / HZ); + sched_show_task(t); + dump_page(page, buf); + mutex_unlock(&oom_lock); } if (lock) { ---------------------------------------- I tried to reproduce your problem using plain idle Fedora 27 guest (RAM 4GB + SWAP 4GB). While the system apparently hang after OOM killer, the system seemed to be just under swap memory thrashing situation, for heavy disk I/O had been observed from the host side. Thus, my case might be different from your case. ---------------------------------------- # yum install yum-utils # yumdownloader --source kernel kernel-4.15.8-300.fc27.src.rpm # rpm --checksig kernel-4.15.8-300.fc27.src.rpm # yum-builddep kernel-4.15.8-300.fc27.src.rpm # rpm -ivh kernel-4.15.8-300.fc27.src.rpm # rngd -r /dev/urandom # rpmbuild -bp ~/rpmbuild/SPECS/kernel.spec # cd ~/rpmbuild/BUILD/kernel-4.15.fc27/linux-4.15.8-300.fc27.x86_64/ # make -s localmodconfig # patch -p1 # make -sj8 && make -s modules_install install ---------------------------------------- ---------------------------------------- [ 0.000000] Linux version 4.15.8 (root@localhost.localdomain) (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Thu Mar 15 14:56:32 JST 2018 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.15.8 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap console=ttyS0,115200n8 console=tty0 LANG=en_US.UTF-8 (...snipped...) Fedora 27 (Workstation Edition) Kernel 4.15.8 on an x86_64 (ttyS0) localhost login: [ 209.887120] INFO: task pool:2315 blocked for 63 seconds. [ 209.892531] pool R running task 0 2315 1520 0x00000000 [ 209.899481] Call Trace: [ 209.902185] sched_show_task+0xf9/0x130 [ 209.906256] __lock_page_or_retry+0x423/0x450 [ 209.910528] ? page_cache_tree_insert+0xe0/0xe0 [ 209.915676] ? pagecache_get_page+0x30/0x320 [ 209.920382] filemap_fault+0x44b/0x8c0 [ 209.921914] ? page_add_file_rmap+0xbf/0x210 [ 209.923568] ? alloc_set_pte+0x45c/0x570 [ 209.925098] ? filemap_map_pages+0x425/0x4e0 [ 209.926743] ext4_filemap_fault+0x2c/0x3b [ 209.928297] __do_fault+0x1f/0x120 [ 209.929631] __handle_mm_fault+0xd78/0x1280 [ 209.931270] handle_mm_fault+0xaa/0x1e0 [ 209.932848] __do_page_fault+0x25d/0x4e0 [ 209.934615] do_page_fault+0x32/0x110 [ 209.936051] ? page_fault+0x65/0x80 [ 209.937421] page_fault+0x7b/0x80 [ 209.938728] RIP: 0033:0x7f1fb961b50b [ 209.940139] RSP: 002b:00007f1fa65ded80 EFLAGS: 00010246 [ 209.942150] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 209.944870] RDX: 000056326eab1d01 RSI: 0000000000000001 RDI: 000056326eab1cf0 [ 209.947747] RBP: 0000000000000000 R08: 0000000000000083 R09: 0000000000000083 [ 209.950405] R10: 00007f1fa65deca0 R11: 0000000000000246 R12: 00007ffd19418d2e [ 209.952478] R13: 00007ffd19418d2f R14: 000056326eaad630 R15: 00007ffd19418db0 [ 209.954440] page:ffffe61cc47d7600 count:4 mapcount:0 mapping:ffff93eaaccadfe0 index:0x7 [ 209.956668] flags: 0x17fffe000100a1(locked|waiters|lru|mappedtodisk) [ 209.958808] raw: 0017fffe000100a1 ffff93eaaccadfe0 0000000000000007 00000004ffffffff [ 209.961296] raw: ffffe61cc47d73e0 ffffe61cc47d7660 0000000000000000 ffff93eab51ec000 [ 209.963796] page dumped because: stuck on bit 0 [ 209.965368] page->mem_cgroup:ffff93eab51ec000 [ 286.262807] sysrq: SysRq : Changing Loglevel [ 286.267231] sysrq: Loglevel set to 9 [ 316.232499] hrtimer: interrupt took 3062939 ns [ 617.319540] Web Content invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0 [ 617.323056] Web Content cpuset=/ mems_allowed=0 [ 617.324305] CPU: 6 PID: 2279 Comm: Web Content Not tainted 4.15.8 #1 [ 617.326054] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017 [ 617.329287] Call Trace: [ 617.330071] dump_stack+0x5c/0x79 [ 617.331032] dump_header+0x6e/0x289 [ 617.332063] oom_kill_process+0x218/0x420 [ 617.333247] out_of_memory+0x2ea/0x4f0 [ 617.334331] __alloc_pages_slowpath+0xa44/0xde0 [ 617.335570] ? pagecache_get_page+0x30/0x320 [ 617.336758] __alloc_pages_nodemask+0x28e/0x2b0 [ 617.338277] alloc_pages_vma+0x7c/0x200 [ 617.339724] __handle_mm_fault+0x98b/0x1280 [ 617.341401] handle_mm_fault+0xaa/0x1e0 [ 617.342548] __do_page_fault+0x25d/0x4e0 [ 617.343714] do_page_fault+0x32/0x110 [ 617.344806] ? page_fault+0x65/0x80 [ 617.345848] page_fault+0x7b/0x80 [ 617.346949] RIP: 0033:0x7fe3dc0631d0 [ 617.348039] RSP: 002b:00007ffcdf8fbce8 EFLAGS: 00010287 [ 617.349659] RAX: 00007fe2d3b46cb0 RBX: 0000000000002710 RCX: 00007fe2d3400000 [ 617.351786] RDX: 00000000000009c4 RSI: 0000000000000000 RDI: 00007fe2d3b45000 [ 617.353849] RBP: 00007ffcdf8fcad0 R08: 0000000000000000 R09: 00007ffcdf8fcad0 [ 617.355944] R10: 0000000000000000 R11: 00000000000009c4 R12: 0000000000002710 [ 617.358001] R13: 00007fe2d3b46cb0 R14: 00007fe3dbf5aaa0 R15: 00000000000006c8 [ 617.360216] Mem-Info: [ 617.360959] active_anon:671116 inactive_anon:135614 isolated_anon:95 [ 617.360959] active_file:0 inactive_file:0 isolated_file:112 [ 617.360959] unevictable:0 dirty:0 writeback:109 unstable:0 [ 617.360959] slab_reclaimable:9349 slab_unreclaimable:24588 [ 617.360959] mapped:0 shmem:3 pagetables:18513 bounce:0 [ 617.360959] free:21550 free_pcp:956 free_cma:0 [ 617.370418] Node 0 active_anon:2684464kB inactive_anon:542456kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):380kB isolated(file):448kB mapped:0kB dirty:0kB writeback:436kB shmem:12kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 617.378204] Node 0 DMA free:15716kB min:268kB low:332kB high:396kB active_anon:24kB inactive_anon:96kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 617.385902] lowmem_reserve[]: 0 2954 3864 3864 3864 [ 617.387310] Node 0 DMA32 free:54944kB min:51460kB low:64324kB high:77188kB active_anon:2107576kB inactive_anon:451104kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:1088kB present:3129152kB managed:3063584kB mlocked:0kB kernel_stack:7408kB pagetables:46044kB bounce:0kB free_pcp:1676kB local_pcp:0kB free_cma:0kB [ 617.395670] lowmem_reserve[]: 0 0 909 909 909 [ 617.396971] Node 0 Normal free:15540kB min:15848kB low:19808kB high:23768kB active_anon:576960kB inactive_anon:90176kB active_file:0kB inactive_file:696kB unevictable:0kB writepending:24kB present:1048576kB managed:936284kB mlocked:0kB kernel_stack:7776kB pagetables:28008kB bounce:0kB free_pcp:2216kB local_pcp:152kB free_cma:0kB [ 617.405373] lowmem_reserve[]: 0 0 0 0 0 [ 617.406519] Node 0 DMA: 1*4kB (M) 0*8kB 0*16kB 1*32kB (U) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15716kB [ 617.410339] Node 0 DMA32: 310*4kB (M) 34*8kB (UME) 127*16kB (UE) 73*32kB (UE) 24*64kB (UE) 7*128kB (UE) 2*256kB (ME) 1*512kB (U) 2*1024kB (ME) 1*2048kB (U) 10*4096kB (UM) = 54392kB [ 617.415091] Node 0 Normal: 181*4kB (UEH) 359*8kB (UMEH) 95*16kB (UEH) 117*32kB (UMEH) 33*64kB (UMEH) 9*128kB (UME) 5*256kB (UEH) 3*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 14940kB [ 617.419688] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 617.422199] 602 total pagecache pages [ 617.423339] 500 pages in swap cache [ 617.424403] Swap cache stats: add 1078498, delete 1077986, find 19251/34948 [ 617.426459] Free swap = 0kB [ 617.427297] Total swap = 4157436kB [ 617.428277] 1048429 pages RAM [ 617.429142] 0 pages HighMem/MovableOnly [ 617.430292] 44486 pages reserved [ 617.431294] 0 pages cma reserved [ 617.432278] 0 pages hwpoisoned [ 617.433214] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 617.435749] [ 613] 0 613 28285 74 233472 192 0 systemd-journal [ 617.438780] [ 639] 0 639 25230 11 204800 1529 -1000 systemd-udevd [ 617.441368] [ 716] 0 716 43953 13 114688 490 0 lvmetad [ 617.443784] [ 762] 0 762 16049 25 135168 87 -1000 auditd [ 617.446265] [ 764] 0 764 21122 0 73728 40 0 audispd [ 617.448778] [ 772] 0 772 9652 21 118784 61 0 sedispatch [ 617.451264] [ 787] 0 787 104417 0 323584 445 0 ModemManager [ 617.453756] [ 788] 0 788 130671 1 389120 490 0 udisksd [ 617.456203] [ 789] 172 789 46481 0 131072 84 0 rtkit-daemon [ 617.458733] [ 791] 81 791 15408 306 147456 211 -900 dbus-daemon [ 617.461234] [ 797] 0 797 51276 0 180224 129 0 gssproxy [ 617.463645] [ 808] 0 808 24562 0 176128 374 0 VGAuthService [ 617.466202] [ 809] 0 809 78151 98 241664 231 0 vmtoolsd [ 617.468636] [ 810] 0 810 87021 1 364544 4873 0 firewalld [ 617.471107] [ 813] 70 813 18576 0 159744 106 0 avahi-daemon [ 617.473627] [ 815] 0 815 70715 2 352256 263 0 sssd [ 617.476034] [ 816] 0 816 4198 1 69632 33 0 mcelog [ 617.478486] [ 820] 0 820 116604 0 315392 402 0 abrtd [ 617.480720] [ 827] 995 827 29457 17 122880 97 0 chronyd [ 617.483115] [ 828] 70 828 18544 0 139264 86 0 avahi-daemon [ 617.485653] [ 831] 997 831 141547 0 294912 1532 0 polkitd [ 617.488124] [ 835] 0 835 72503 20 380928 537 0 sssd_be [ 617.490532] [ 849] 0 849 69785 32 331776 340 0 abrt-dump-journ [ 617.493129] [ 850] 0 850 67925 28 307200 337 0 abrt-dump-journ [ 617.495749] [ 851] 0 851 67925 32 331776 336 0 abrt-dump-journ [ 617.498495] [ 857] 0 857 70131 21 372736 328 0 sssd_nss [ 617.500905] [ 864] 0 864 19719 40 188416 525 0 systemd-logind [ 617.503393] [ 865] 0 865 106036 0 221184 239 0 accounts-daemon [ 617.505865] [ 908] 0 908 184194 1 446464 743 0 NetworkManager [ 617.508433] [ 927] 0 927 440572 1 823296 2167 0 libvirtd [ 617.510892] [ 928] 0 928 22276 0 184320 207 -1000 sshd [ 617.513213] [ 941] 0 941 35864 21 106496 140 0 crond [ 617.515454] [ 943] 0 943 106017 0 241664 288 0 gdm [ 617.517651] [ 960] 0 960 96490 1 299008 301 0 gdm-session-wor [ 617.520313] [ 1012] 42 1012 22369 1 217088 312 0 systemd [ 617.522712] [ 1022] 42 1022 34344 0 258048 1172 0 (sd-pam) [ 617.525142] [ 1034] 42 1034 105310 1 356352 325 0 gdm-wayland-ses [ 617.527658] [ 1062] 42 1062 18142 1 147456 242 0 dbus-daemon [ 617.530305] [ 1070] 42 1070 166336 0 421888 602 0 gnome-session-b [ 617.532902] [ 1099] 0 1099 19834 1 188416 475 0 dhclient [ 617.535344] [ 1101] 42 1101 953274 1221 1609728 14656 0 gnome-shell [ 617.537777] [ 1141] 0 1141 36365 1 290816 275 0 sshd [ 617.540211] [ 1153] 0 1153 104695 0 241664 322 0 upowerd [ 617.542637] [ 1161] 0 1161 19770 0 196608 286 0 systemd [ 617.545038] [ 1163] 0 1163 31225 0 253952 1169 0 (sd-pam) [ 617.547350] [ 1168] 0 1168 36371 32 278528 251 0 sshd [ 617.549655] [ 1169] 0 1169 33591 9 94208 535 0 bash [ 617.552040] [ 1177] 42 1177 208216 44 516096 3820 0 Xwayland [ 617.554492] [ 1213] 42 1213 88943 0 192512 207 0 at-spi-bus-laun [ 617.557103] [ 1218] 42 1218 18027 1 143360 142 0 dbus-daemon [ 617.559661] [ 1221] 42 1221 55770 1 204800 209 0 at-spi2-registr [ 617.562309] [ 1227] 42 1227 292888 1 344064 285 0 pulseaudio [ 617.564800] [ 1247] 42 1247 122023 0 356352 427 0 xdg-permission- [ 617.567274] [ 1254] 42 1254 118810 21 208896 595 0 ibus-daemon [ 617.569682] [ 1257] 42 1257 97414 0 184320 194 0 ibus-dconf [ 617.572168] [ 1260] 42 1260 149149 0 688128 2344 0 ibus-x11 [ 617.574598] [ 1265] 42 1265 97381 0 184320 186 0 ibus-portal [ 617.576991] [ 1284] 0 1284 12545 15 143360 135 0 wpa_supplicant [ 617.579492] [ 1286] 42 1286 170296 0 724992 2465 0 gsd-wacom [ 617.582029] [ 1287] 0 1287 139149 14 491520 799 0 packagekitd [ 617.584523] [ 1289] 42 1289 186251 0 704512 2486 0 gsd-xsettings [ 617.587082] [ 1290] 42 1290 170801 0 716800 2393 0 gsd-a11y-keyboa [ 617.589693] [ 1291] 42 1291 127616 0 401408 388 0 gsd-a11y-settin [ 617.592388] [ 1292] 42 1292 149054 0 679936 2347 0 gsd-clipboard [ 617.594948] [ 1295] 42 1295 208380 26 753664 2372 0 gsd-color [ 617.597320] [ 1296] 42 1296 117670 1 434176 460 0 gsd-datetime [ 617.599838] [ 1297] 42 1297 127536 1 393216 369 0 gsd-housekeepin [ 617.602592] [ 1302] 42 1302 188713 0 741376 2407 0 gsd-keyboard [ 617.605138] [ 1310] 42 1310 337263 0 831488 2544 0 gsd-media-keys [ 617.607710] [ 1311] 42 1311 108575 0 376832 361 0 gsd-mouse [ 617.610175] [ 1312] 42 1312 192424 24 745472 2446 0 gsd-power [ 617.612625] [ 1316] 42 1316 118069 0 454656 460 0 gsd-print-notif [ 617.615262] [ 1317] 42 1317 108579 1 393216 361 0 gsd-rfkill [ 617.617698] [ 1318] 42 1318 127008 1 393216 370 0 gsd-screensaver [ 617.620247] [ 1323] 42 1323 104122 0 237568 250 0 gsd-sharing [ 617.622831] [ 1326] 42 1326 153276 0 458752 494 0 gsd-smartcard [ 617.625414] [ 1331] 42 1331 138208 0 483328 473 0 gsd-sound [ 617.627875] [ 1349] 42 1349 78964 0 176128 189 0 ibus-engine-sim [ 617.630500] [ 1405] 991 1405 108150 0 258048 535 0 colord [ 617.632914] [ 1434] 0 1434 30765 0 69632 31 0 agetty [ 617.635357] [ 1467] 0 1467 102707 1 344064 330 0 gdm-session-wor [ 617.637956] [ 1483] 1000 1483 22369 1 204800 321 0 systemd [ 617.640294] [ 1488] 1000 1488 71210 0 274432 1183 0 (sd-pam) [ 617.642631] [ 1499] 1000 1499 98795 1 192512 268 0 gnome-keyring-d [ 617.645292] [ 1503] 1000 1503 105342 0 352256 324 0 gdm-wayland-ses [ 617.647893] [ 1517] 1000 1517 18261 181 147456 199 0 dbus-daemon [ 617.650408] [ 1520] 1000 1520 166420 151 442368 482 0 gnome-session-b [ 617.653041] [ 1571] 1000 1571 100256 1 208896 247 0 gvfsd [ 617.655528] [ 1586] 1000 1586 1016566 2154 1720320 21173 0 gnome-shell [ 617.658035] [ 1603] 1000 1603 213032 405 602112 3953 0 Xwayland [ 617.660472] [ 1614] 1000 1614 88942 1 188416 207 0 at-spi-bus-laun [ 617.663117] [ 1619] 1000 1619 18060 11 151552 137 0 dbus-daemon [ 617.665695] [ 1622] 1000 1622 55770 0 196608 219 0 at-spi2-registr [ 617.668312] [ 1628] 1000 1628 497904 0 430080 392 0 pulseaudio [ 617.670800] [ 1645] 1000 1645 122044 0 348160 428 0 xdg-permission- [ 617.673317] [ 1647] 1000 1647 247911 0 974848 1921 0 gnome-shell-cal [ 617.675909] [ 1651] 1000 1651 137313 26 212992 662 0 ibus-daemon [ 617.678437] [ 1655] 1000 1655 97434 0 196608 209 0 ibus-dconf [ 617.680876] [ 1657] 1000 1657 149206 0 700416 2359 0 ibus-x11 [ 617.683214] [ 1659] 1000 1659 97435 0 188416 192 0 ibus-portal [ 617.685669] [ 1679] 1000 1679 109553 0 278528 380 0 gvfs-udisks2-vo [ 617.688332] [ 1687] 1000 1687 253549 0 1093632 2003 0 evolution-sourc [ 617.690953] [ 1689] 1000 1689 99386 0 200704 214 0 gvfs-gphoto2-vo [ 617.693582] [ 1693] 1000 1693 96270 0 180224 185 0 gvfs-mtp-volume [ 617.696255] [ 1701] 1000 1701 95830 0 176128 194 0 gvfs-goa-volume [ 617.698913] [ 1705] 1000 1705 219480 0 688128 1403 0 goa-daemon [ 617.701400] [ 1713] 1000 1713 130446 60 425984 365 0 goa-identity-se [ 617.703970] [ 1715] 1000 1715 122018 0 241664 287 0 gvfs-afc-volume [ 617.706664] [ 1732] 1000 1732 127662 0 401408 401 0 gsd-mouse [ 617.709200] [ 1733] 1000 1733 192483 127 741376 2367 0 gsd-power [ 617.711660] [ 1734] 1000 1734 139152 0 479232 505 0 gsd-print-notif [ 617.714214] [ 1735] 1000 1735 163957 0 409600 407 0 gsd-rfkill [ 617.716588] [ 1736] 1000 1736 127054 0 385024 378 0 gsd-screensaver [ 617.719309] [ 1737] 1000 1737 141099 0 245760 384 0 gsd-sharing [ 617.721786] [ 1738] 1000 1738 153321 0 454656 495 0 gsd-smartcard [ 617.724221] [ 1739] 1000 1739 170365 0 716800 2485 0 gsd-wacom [ 617.726563] [ 1741] 1000 1741 186326 0 708608 2502 0 gsd-xsettings [ 617.729184] [ 1743] 1000 1743 138253 0 487424 486 0 gsd-sound [ 617.731642] [ 1772] 1000 1772 127662 0 401408 389 0 gsd-a11y-settin [ 617.734215] [ 1773] 1000 1773 170874 0 720896 2406 0 gsd-a11y-keyboa [ 617.736687] [ 1774] 1000 1774 226896 105 757760 2376 0 gsd-color [ 617.739050] [ 1775] 1000 1775 149130 0 692224 2356 0 gsd-clipboard [ 617.741694] [ 1778] 1000 1778 148277 53 438272 342 0 gsd-housekeepin [ 617.744302] [ 1780] 1000 1780 136758 0 479232 486 0 gsd-datetime [ 617.746771] [ 1782] 1000 1782 320983 113 823296 2443 0 gsd-media-keys [ 617.749310] [ 1785] 1000 1785 188787 0 720896 2417 0 gsd-keyboard [ 617.751874] [ 1799] 0 1799 57809 12 270336 356 0 cupsd [ 617.754243] [ 1835] 1000 1835 318486 1 1208320 10872 0 evolution-calen [ 617.756850] [ 1837] 1000 1837 78985 0 180224 207 0 ibus-engine-sim [ 617.759392] [ 1903] 1000 1903 154442 0 462848 469 0 gsd-printer [ 617.761954] [ 1919] 1000 1919 366545 1 839680 54447 0 tracker-miner-a [ 617.764613] [ 1925] 1000 1925 183657 4 413696 1748 0 tracker-miner-f [ 617.767215] [ 1927] 1000 1927 140518 209 589824 1562 0 vmtoolsd [ 617.769679] [ 1929] 1000 1929 161051 1 540672 1348 0 abrt-applet [ 617.772266] [ 1930] 1000 1930 69563 0 180224 253 0 gsd-disk-utilit [ 617.774883] [ 1950] 1000 1950 191060 1 356352 3292 0 tracker-store [ 617.777357] [ 1952] 1000 1952 298180 0 1077248 2683 0 evolution-alarm [ 617.779821] [ 1956] 1000 1956 215285 2108 937984 8547 0 gnome-software [ 617.782472] [ 1968] 1000 1968 139879 1 516096 3785 0 seapplet [ 617.784914] [ 1972] 1000 1972 395485 1 831488 2963 0 tracker-extract [ 617.787415] [ 1999] 1000 1999 340260 0 1097728 10842 0 evolution-calen [ 617.789943] [ 2009] 0 2009 89135 0 311296 431 -900 abrt-dbus [ 617.792465] [ 2032] 1000 2032 48069 1 147456 158 0 dconf-service [ 617.795025] [ 2034] 1000 2034 358685 0 1101824 10856 0 evolution-calen [ 617.797530] [ 2064] 1000 2064 288193 1 1069056 1956 0 evolution-addre [ 617.799992] [ 2077] 1000 2077 349890 0 1064960 2036 0 evolution-addre [ 617.802640] [ 2093] 0 2093 159752 34 524288 740 0 fwupd [ 617.805037] [ 2114] 1000 2114 579875 2029 1630208 29183 0 firefox [ 617.808215] [ 2204] 1000 2204 1198321 369608 6414336 297145 0 Web Content [ 617.810717] [ 2255] 1000 2255 47181 26 196608 112 0 gconfd-2 [ 617.813148] [ 2279] 1000 2279 1401628 423539 7999488 445698 0 Web Content [ 617.815741] [ 2317] 1000 2317 519868 490 983040 4881 0 Web Content [ 617.818209] [ 2389] 0 2389 32947 128 282624 0 0 sssd_kcm [ 617.820631] Out of memory: Kill process 2279 (Web Content) score 426 or sacrifice child [ 617.823206] Killed process 2279 (Web Content) total-vm:5606512kB, anon-rss:1694156kB, file-rss:0kB, shmem-rss:0kB [ 618.020277] oom_reaper: reaped process 2279 (Web Content), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [ 754.652953] INFO: task gdbus:1963 blocked for 62 seconds. [ 754.655359] gdbus R running task 0 1963 1483 0x00000000 [ 754.658282] Call Trace: [ 754.659282] sched_show_task+0xf9/0x130 [ 754.660812] __lock_page_or_retry+0x423/0x450 [ 754.663615] ? page_cache_tree_insert+0xe0/0xe0 [ 754.665607] do_swap_page+0x5b6/0xa40 [ 754.667166] __handle_mm_fault+0x87c/0x1280 [ 754.668812] handle_mm_fault+0xaa/0x1e0 [ 754.670563] __do_page_fault+0x25d/0x4e0 [ 754.672318] do_page_fault+0x32/0x110 [ 754.674069] ? page_fault+0x65/0x80 [ 754.675443] page_fault+0x7b/0x80 [ 754.676789] RIP: 0033:0x7ff81f922f67 [ 754.678247] RSP: 002b:00007ff8151976d0 EFLAGS: 00010293 [ 754.680264] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00007ff80c0008d0 [ 754.682975] RDX: 0000564d62a0b9e0 RSI: 00007ff80c0008d0 RDI: 0000000000000001 [ 754.686356] RBP: ffffffffffffff60 R08: 00007ff815197888 R09: 0000000000000011 [ 754.699292] R10: 0000000000000010 R11: 0000000000000033 R12: 00007ff815197888 [ 754.724044] R13: 0000000000000000 R14: 00007ff815197a80 R15: 00007ff815197888 [ 754.726411] page:ffffe61cc435ac00 count:2 mapcount:0 mapping:0000000000000000 index:0x0 [ 754.728747] flags: 0x17fffe000402a1(locked|waiters|lru|owner_priv_1|swapbacked) [ 754.730877] raw: 0017fffe000402a1 0000000000000000 0000000000000000 00000002ffffffff [ 754.733260] raw: ffffe61cc435ac60 ffffe61cc411d0e0 000000000002ecb7 0000000000000000 [ 754.735494] page dumped because: stuck on bit 0 [ 816.092318] INFO: task gdbus:1963 blocked for 124 seconds. [ 816.097742] gdbus R running task 0 1963 1483 0x00000000 [ 816.104631] Call Trace: [ 816.107092] sched_show_task+0xf9/0x130 [ 816.110971] __lock_page_or_retry+0x423/0x450 [ 816.115267] ? page_cache_tree_insert+0xe0/0xe0 [ 816.120709] do_swap_page+0x5b6/0xa40 [ 816.124465] __handle_mm_fault+0x87c/0x1280 [ 816.129135] handle_mm_fault+0xaa/0x1e0 [ 816.131533] __do_page_fault+0x25d/0x4e0 [ 816.133243] do_page_fault+0x32/0x110 [ 816.134839] ? page_fault+0x65/0x80 [ 816.136378] page_fault+0x7b/0x80 [ 816.137835] RIP: 0033:0x7ff81f922f67 [ 816.139383] RSP: 002b:00007ff8151976d0 EFLAGS: 00010293 [ 816.141621] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 00007ff80c0008d0 [ 816.144630] RDX: 0000564d62a0b9e0 RSI: 00007ff80c0008d0 RDI: 0000000000000001 [ 816.147653] RBP: ffffffffffffff60 R08: 00007ff815197888 R09: 0000000000000011 [ 816.150673] R10: 0000000000000010 R11: 0000000000000033 R12: 00007ff815197888 [ 816.153692] R13: 0000000000000000 R14: 00007ff815197a80 R15: 00007ff815197888 [ 816.156716] page:ffffe61cc435ac00 count:2 mapcount:0 mapping:0000000000000000 index:0x0 [ 816.160093] flags: 0x17fffe000402a1(locked|waiters|lru|owner_priv_1|swapbacked) [ 816.162318] raw: 0017fffe000402a1 0000000000000000 0000000000000000 00000002ffffffff [ 816.164586] raw: ffffe61cc435ac60 ffffe61cc43720a0 000000000002ecb7 0000000000000000 [ 816.166868] page dumped because: stuck on bit 0 [ 1336.841310] sysrq: SysRq : Show Memory [ 1336.843919] Mem-Info: [ 1336.845313] active_anon:666669 inactive_anon:135943 isolated_anon:0 [ 1336.845313] active_file:416 inactive_file:591 isolated_file:0 [ 1336.845313] unevictable:0 dirty:0 writeback:0 unstable:0 [ 1336.845313] slab_reclaimable:9555 slab_unreclaimable:25072 [ 1336.845313] mapped:246 shmem:13 pagetables:18544 bounce:0 [ 1336.845313] free:24109 free_pcp:984 free_cma:0 [ 1336.866984] Node 0 active_anon:2666676kB inactive_anon:543772kB active_file:1816kB inactive_file:2340kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1572kB dirty:0kB writeback:0kB shmem:52kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1336.886741] Node 0 DMA free:15720kB min:268kB low:332kB high:396kB active_anon:120kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1336.901945] lowmem_reserve[]: 0 2954 3864 3864 3864 [ 1336.903380] Node 0 DMA32 free:63972kB min:51460kB low:64324kB high:77188kB active_anon:2038972kB inactive_anon:510968kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129152kB managed:3063584kB mlocked:0kB kernel_stack:6880kB pagetables:45196kB bounce:0kB free_pcp:2076kB local_pcp:0kB free_cma:0kB [ 1336.911514] lowmem_reserve[]: 0 0 909 909 909 [ 1336.912793] Node 0 Normal free:17116kB min:15848kB low:19808kB high:23768kB active_anon:627584kB inactive_anon:32804kB active_file:448kB inactive_file:2004kB unevictable:0kB writepending:0kB present:1048576kB managed:936284kB mlocked:0kB kernel_stack:7744kB pagetables:28980kB bounce:0kB free_pcp:1932kB local_pcp:32kB free_cma:0kB [ 1336.922050] lowmem_reserve[]: 0 0 0 0 0 [ 1336.923182] Node 0 DMA: 2*4kB (M) 0*8kB 0*16kB 1*32kB (U) 1*64kB (U) 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 2*1024kB (UM) 0*2048kB 3*4096kB (M) = 15720kB [ 1336.927603] Node 0 DMA32: 1164*4kB (UME) 489*8kB (UME) 127*16kB (UME) 50*32kB (UME) 43*64kB (UME) 12*128kB (UME) 3*256kB (UME) 3*512kB (UM) 6*1024kB (ME) 1*2048kB (U) 9*4096kB (UM) = 63848kB [ 1336.932590] Node 0 Normal: 256*4kB (UMEH) 181*8kB (UME) 100*16kB (UMEH) 104*32kB (UMEH) 34*64kB (UMEH) 12*128kB (UME) 16*256kB (UME) 3*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 16744kB [ 1336.937741] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1336.940400] 990 total pagecache pages [ 1336.941483] 54 pages in swap cache [ 1336.942611] Swap cache stats: add 1564554, delete 1564488, find 37795/66721 [ 1336.944869] Free swap = 0kB [ 1336.945720] Total swap = 4157436kB [ 1336.946725] 1048429 pages RAM [ 1336.947563] 0 pages HighMem/MovableOnly [ 1336.948643] 44486 pages reserved [ 1336.949549] 0 pages cma reserved [ 1336.950465] 0 pages hwpoisoned [ 1339.038124] sysrq: SysRq : Kill All Tasks [ 1345.219353] kauditd_printk_skb: 4 callbacks suppressed [ 1345.219355] audit: type=1131 audit(1521101183.830:268): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1345.435207] audit: type=1131 audit(1521101184.046:269): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1345.895461] audit: type=1131 audit(1521101184.506:270): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=lvm2-lvmetad comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.111947] audit: type=1131 audit(1521101184.723:271): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=auditd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.163635] audit: type=1131 audit(1521101184.774:272): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ModemManager comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.197119] audit: type=1131 audit(1521101184.808:273): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=udisks2 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.213456] audit: type=1131 audit(1521101184.824:274): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rtkit-daemon comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.405816] audit: type=1131 audit(1521101185.017:275): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.543337] audit: type=1131 audit(1521101185.154:276): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=gssproxy comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1346.619190] audit: type=1131 audit(1521101185.230:277): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=vgauthd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 1350.617393] kauditd_printk_skb: 54 callbacks suppressed [ 1350.617397] audit: type=1130 audit(1521101189.228:332): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-quit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1350.643554] audit: type=1131 audit(1521101189.228:333): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=plymouth-quit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1351.532747] audit: type=1305 audit(1521101190.143:334): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1 [ 1351.982065] systemd-journald[2542]: File /var/log/journal/c34a3e77044e48009188cc8510309231/system.journal corrupted or uncleanly shut down, renaming and replacing. Fedora 27 (Workstation Edition) Kernel 4.15.8 on an x86_64 (ttyS0) localhost login: [ 1356.402863] audit: type=1130 audit(1521101195.013:335): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1356.574118] audit: type=1130 audit(1521101195.185:336): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1356.595716] audit: type=1130 audit(1521101195.190:337): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udevd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1356.818119] audit: type=1130 audit(1521101195.429:338): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sssd-secrets comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1356.832245] audit: type=1130 audit(1521101195.435:339): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-logind comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1357.053961] audit: type=1130 audit(1521101195.665:340): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1357.071389] audit: type=1130 audit(1521101195.682:341): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=getty@tty2 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1357.583456] audit: type=1130 audit(1521101196.194:342): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ModemManager comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1357.919797] audit: type=1130 audit(1521101196.530:343): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=libvirtd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1358.029038] audit: type=1130 audit(1521101196.640:344): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1361.428472] kauditd_printk_skb: 3 callbacks suppressed [ 1361.428477] audit: type=1130 audit(1521101200.039:348): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=polkit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1368.440467] audit: type=1130 audit(1521101207.051:349): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-wait-online comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1379.011292] audit: type=1131 audit(1521101217.622:350): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1389.091306] audit: type=1131 audit(1521101227.702:351): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1389.113601] audit: type=1130 audit(1521101227.707:352): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1389.132071] audit: type=1131 audit(1521101227.707:353): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1389.335759] audit: type=1130 audit(1521101227.946:354): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1391.048021] audit: type=1130 audit(1521101229.659:355): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1401.010558] audit: type=1131 audit(1521101239.621:356): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 1513.479243] sysrq: SysRq : Show Blocked State [ 1513.486357] task PC stack pid father ----------------------------------------
