crash: struct command can read irrelevant pages.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



>Hello Atsushi,
>
>I've committed a SLAB/SLUB kmem_cache-specific fix for this issue:
>
>  https://github.com/crash-utility/crash/commit/c0b7a74fc13121203810d06d163550436b2d5476
>
>which is queued for crash-7.0.6.

Thanks Dave, I made sure that this patch solved my problem.

>> This is a "known" issue has been discussed on the crash-utility list in the
>> past, at least with respect to the kmem_cache data structure.  But for any random
>> data structure that has such a construct, I'm not sure what can be done.

I also have no ideas how to solve it, but it seems that it hasn't been a
practical problem yet. So I think your patch is enough for now.

>> By any chance can you make the 32-bit vmlinux/vmcore pair available for
>> me to download?  Reply to me off-list if you can.

Sure, I'll send another mail.


Thanks
Atsushi Kumagai

>>
>>
>> ----- Original Message -----
>> > Hello,
>> >
>> > Finally, I've found the cause of the issue I mentioned as below
>> > when makedumpfile v1.5.5 was released:
>> >
>> > > 2. At first, the supported kernel will be updated to 3.12, but I
>> > > found an issue while testing for v1.5.5, which seems that the page
>> > > filtering works wrongly on kernel 3.12. I couldn't investigate this
>> > > yet and it will take some time to finish it.
>> > > Therefore, the latest supported kernel version is 3.11 in v1.5.5.
>> >
>> > This is neither a kernel issue nor a makedumpfile issue, it's a crash's bug.
>> > It can happen when a slab cache is stored at almost end of a page.
>> >
>> > == Description ==
>> >
>> > At the beginning, I found the error message below when I used crash for
>> > a dumpfile generated by makedumpfile -d2:
>> >
>> >     please wait... (gathering kmem slab cache data)
>> >     crash: page excluded: kernel virtual address: f4e87000  type:
>> >     "kmem_cache
>> >     buffer"
>> >
>> >     crash: unable to initialize kmem slab cache subsystem
>> >
>> > This message indicated that crash failed to get a slab cache during
>> > kmem_cache_init(), and according to the below, crash failed to get
>> > the slab cache stored at f4e86f40:
>> >
>> >     crash> p kmem_cache
>> >     kmem_cache = $1 = (struct kmem_cache *) 0xc0b1cbc0 <kmem_cache_boot>
>> >     crash>
>> >     crash> list kmem_cache.list -s kmem_cache.name -h 0xc0b1cbc0
>> >     ...
>> >     f4d37840
>> >       name = 0xf4edf540 "uid_cache"
>> >     f4e86f40
>> >     list: page excluded: kernel virtual address: f4e87000  type:
>> >     "gdb_readmem_callback"
>> >
>> > It seems that the slab cache covered two pages, [f4e86000- f4e87000] and
>> > [f4e87000- f4e88000]. Well, let's confirm the *real* size of it.
>> >
>> > Since slab caches except kmem_cache_boot are allocated as slab objects,
>> > we can confirm the size like below:
>> >
>> >   crash> p kmem_cache
>> >   kmem_cache = $2 = (struct kmem_cache *) 0xc0b1cbc0 <kmem_cache_boot>
>> >   crash> struct kmem_cache.object_size 0xc0b1cbc0
>> >     object_size = 104
>> >   crash>
>> >
>> > In my environment, the size was 104 bytes. Therefore, the slab cache
>> > stored at f4e86f40 fits in the single page([f4e86000- f4e87000]) and
>> > the excluded page([f4e87000- f4e88000]) isn't a related page.
>> >
>> > On the other hand, crash get the size from vmlinux by using gdb,
>> > it was 216 bytes:
>> >
>> >     crash> struct kmem_cache
>> >     struct kmem_cache {
>> >         unsigned int batchcount;
>> >         unsigned int limit;
>> >         ...
>> >         struct kmem_cache_node **node;
>> >         struct array_cache *array[33];
>> >     }
>> >     SIZE: 216
>> >     crash>
>> >
>> > So crash mistook the correlative pages of the slab cache as
>> > [f4e86000- f4e87000] and [f4e87000- f4e88000] even though the latter
>> > was a irrelevant page.
>> >
>> > This gap came from the fact that the size of slab cache is variable.
>> >
>> >     struct kmem_cache {
>> >     ...
>> >             struct kmem_cache_node **node;
>> >             struct array_cache *array[NR_CPUS + MAX_NUMNODES];
>> >             /*
>> >              * Do not add fields after array[]
>> >              */
>> >     };
>> >
>> > The size of "array" is the variable factor of kmem_cache.
>> > When building vmlinux, the size of kmem_cache will be calculated with
>> > NR_CPUS and MAX_NUMNODES, and put it into vmlinux as a debug information.
>> > (Sorry, I don't know gcc well. I may misunderstand this.)
>> > However, the actual size will be smaller than the defined size because
>> > the actual size will be decided based on the actual number of CPUs and
>> > NODEs.
>> >
>> > void __init kmem_cache_init(void)::
>> > ...
>> >         /*
>> >          * struct kmem_cache size depends on nr_node_ids & nr_cpu_ids
>> >          */
>> >         create_boot_cache(kmem_cache, "kmem_cache",
>> >                 offsetof(struct kmem_cache, array[nr_cpu_ids]) +
>> >                                   nr_node_ids * sizeof(struct
>> >                                   kmem_cache_node
>> >                                   *),  // object_size
>> >                                   SLAB_HWCACHE_ALIGN);
>> >         list_add(&kmem_cache->list, &slab_caches);
>> >
>> >
>> > As for kmem_cache, we can get the actual size of it from kmem_cache_boot,
>> > but I suppose that kmem_cache is not the only struct in kernel whose size
>> > is variable. So I think we should discuss how to address such issues like
>> > this.
>> >
>> > By the way, I mentioned the case of *SLAB* in this mail,
>> > but SLUB seems have the same issue.
>> >
>> >
>> > Thanks
>> > Atsushi Kumagai
>>
>>
>> This is a "known" issue has been discussed on the crash-utility list in the
>> past,
>> at least with respect to the kmem_cache data structure.  But for any random
>> data
>> structure that has such a construct, I'm not sure what can be done.
>>
>> In the case of the CONFIG_SLAB kmem_cache data structure, there is a function
>> that is supposed to "downsize" the size value of the kmem_cache data
>> structure
>> that is returned by gdb.  It is called here in kmem_cache_init(), just
>> prior to cycling through all of the kmem_cache structures, where the
>> page excluded error shown above occurred:
>>
>>    8561         if (!(pc->flags & RUNTIME))
>>    8562                 kmem_cache_downsize();
>>    8563
>>    8564         cache_buf = GETBUF(SIZE(kmem_cache_s));
>>    8565         hq_open();
>>    8566
>>    8567         do {
>>    8568                 cache_count++;
>>    8569
>>    8570                 if (!readmem(cache, KVADDR, cache_buf,
>>    SIZE(kmem_cache_s),
>>    8571                         "kmem_cache buffer", RETURN_ON_ERROR)) {
>>    8572                         FREEBUF(cache_buf);
>>    8573                         vt->flags |= KMEM_CACHE_UNAVAIL;
>>    8574                         error(INFO,
>>    8575                           "%sunable to initialize kmem slab cache
>>    subsystem\n\n",
>>    8576                                 DUMPFILE() ? "\n" : "");
>>    8577                         hq_close();
>>    8578                         return;
>>    8579                 }
>>
>> The SIZE(kmem_cache_s) value should have been downsized by that function,
>> but presumably it did not work.  If CRASHDEBUG(1) was turned on during
>> initialization,
>> you would have seen either of these two messages from kmem_cache_downsize():
>>
>>                 if (CRASHDEBUG(1))
>>                         fprintf(fp, "kmem_cache_downsize: %ld to %ld\n",
>>                                 STRUCT_SIZE("kmem_cache"),
>>                                 SIZE(kmem_cache_s));
>>
>> or:
>>
>>                 if (CRASHDEBUG(1)) {
>>                         fprintf(fp,
>>                             "\nkmem_cache_downsize: SIZE(kmem_cache_s): %ld "
>>                             "cache_cache.buffer_size: %d\n",
>>                                 STRUCT_SIZE("kmem_cache"), buffer_size);
>>                         fprintf(fp,
>>                             "kmem_cache_downsize: nr_node_ids: %ld\n",
>>                                 vt->kmem_cache_len_nodes);
>>                 }
>>
>> The function failed probably failed due to some kernel change.  In fact,
>> I just checked a 3.13 CONFIG_SLAB kernel, and I see that
>> kmem_cache_downsize()
>> no longer works for that kernel.
>>
>> I see that kmem_cache_boot would be a good alternative for determining
>> the size on CONFIG_SLAB kernels, at least on 3.7 and later kernels where
>> it was introduced.  And for CONFIG_SLUB, which doesn't currently have a
>> "downsize" function, it looks like its "kmem_cache" cache also has size
>> fields that could be used.
>>
>> By any chance can you make the 32-bit vmlinux/vmcore pair available for
>> me to download?  Reply to me off-list if you can.
>>
>> Thanks,
>>   Dave



[Index of Archives]     [LM Sensors]     [Linux Sound]     [ALSA Users]     [ALSA Devel]     [Linux Audio Users]     [Linux Media]     [Kernel]     [Gimp]     [Yosemite News]     [Linux Media]

  Powered by Linux