On Wed 23-11-16 09:44:45, Nikolay Borisov wrote:
>
>
> On 11/22/2016 07:02 PM, Michal Hocko wrote:
> > On Tue 22-11-16 16:35:38, Nikolay Borisov wrote:
> >>
> >>
> >> On 11/22/2016 04:30 PM, Michal Hocko wrote:
> >>> On Tue 22-11-16 10:56:51, Nikolay Borisov wrote:
> >>>>
> >>>>
> >>>> On 11/21/2016 07:31 AM, Michal Hocko wrote:
> >>>>> Hi,
> >>>>> I am sorry for a late response, but I was offline until this weekend. I
> >>>>> will try to get to this email ASAP but it might take some time.
> >>>>
> >>>> No worries. I did some further digging up and here is what I got, which
> >>>> I believe is rather strange:
> >>>>
> >>>> struct scan_control {
> >>>> nr_to_reclaim = 32,
> >>>> gfp_mask = 37880010,
> >>>> order = 0,
> >>>> nodemask = 0x0,
> >>>> target_mem_cgroup = 0xffff8823990d1400,
> >>>> priority = 7,
> >>>> may_writepage = 1,
> >>>> may_unmap = 1,
> >>>> may_swap = 0,
> >>>> may_thrash = 1,
> >>>> hibernation_mode = 0,
> >>>> compaction_ready = 0,
> >>>> nr_scanned = 0,
> >>>> nr_reclaimed = 0
> >>>> }
> >>>>
> >>>> Parsing: 37880010
> >>>> #define ___GFP_HIGHMEM 0x02
> >>>> #define ___GFP_MOVABLE 0x08
> >>>> #define ___GFP_IO 0x40
> >>>> #define ___GFP_FS 0x80
> >>>> #define ___GFP_HARDWALL 0x20000
> >>>> #define ___GFP_DIRECT_RECLAIM 0x400000
> >>>> #define ___GFP_KSWAPD_RECLAIM 0x2000000
> >>>>
> >>>> And initial_priority is 12 (DEF_PRIORITY). Given that nr_scanned is 0
> >>>> and priority is 7 this means we've gone 5 times through the do {} while
> >>>> in do_try_to_free_pages. Also total_scanned seems to be 0. Here is the
> >>>> zone which was being reclaimed :
> >>
> >> This is also very strange that total_scanned is 0.
> >>
> >>
> >>>>
> >>>> http://sprunge.us/hQBi
> >>>
> >>> LRUs on that zones seem to be empty from a quick glance. kmem -z in the
> >>> crash can give you per zone counters much more nicely.
> >>>
> >>
> >> So here are the populated zones:
> > [...]
> >> NODE: 0 ZONE: 2 ADDR: ffff88207fffcf00 NAME: "Normal"
> >> SIZE: 33030144 MIN/LOW/HIGH: 22209/27761/33313
> >> VM_STAT:
> >> NR_FREE_PAGES: 62436
> >> NR_ALLOC_BATCH: 2024
> >> NR_INACTIVE_ANON: 8177867
> >> NR_ACTIVE_ANON: 5407176
> >> NR_INACTIVE_FILE: 5804642
> >> NR_ACTIVE_FILE: 9694170
> >
> > So your LRUs are definitely not empty as I have thought. Having
> > 0 pages scanned is indeed very strange. We do reset sc->nr_scanned
> > for each priority but my understanding was that you are looking at a
> > state where we are somwhere in the middle of shrink_zones. Moreover
> > total_scanned should be cumulative.
>
> So the server began acting wonky. People logged on it and saw the
> softlockup as per my initial email. They then initiated a crashdump via
> sysrq since most commands weren't going through (e.g. forking) so
> crashing it was a last resort measure. After that I start looking at the
> crashdump and observe that prior to the crash machine seems to have
> locked up judging from the dmesg logs. However, when I manually inspect
> the *current* (and current being at the time the crash was actually
> initiated) state of the processes reported as softlock up they seem to
> have made progress are now in
> shrink_zone->shrink_lruvec->shrink_inactive_list->_cond_resched->__schedule
OK, I see.
> And the softlockup was being shown to be in mem_cgroup_iter. So it's
> mystery how come this function can softlockup and after the softlockup
> apparently got resolved reclaim is not making any progress.
This might be just a coincidence and the lockup might really mean that
we couldn't isolate (thus scan) any pages at the time the lockup was
detected. mem_cgroup_iter shouldn't itself loop without any bounds to
trigger the lockup on its own. There is a loop around
css_next_descendant_pre but this should take only few iterations in case
we are racing with cgroup removal AFAIR. So to me it sounds more like a
problem with the state of LRU lists rather than anything else.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>