On Thu, May 5, 2011 at 11:21 PM, Balbir Singh <balbir@xxxxxxxxxxxxxxxxxx> wrote: > * Ying Han <yinghan@xxxxxxxxxx> [2011-05-05 10:17:36]: > >> On Thu, May 5, 2011 at 2:21 AM, Balbir Singh <balbir@xxxxxxxxxxxxxxxxxx> wrote: >> > * Ying Han <yinghan@xxxxxxxxxx> [2011-05-04 10:14:04]: >> > >> >> The problem with small dmesg ring buffer like 512k is that only limited number >> >> of task traces will be logged. Sometimes we lose important information only >> >> because of too many duplicated stack traces. >> >> >> >> This patch tries to reduce the duplication of task stack trace in the dump >> >> message by hashing the task stack. The hashtable is a 32k pre-allocated buffer >> >> during bootup. Then we hash the task stack with stack_depth 32 for each stack >> >> entry. Each time if we find the identical task trace in the task stack, we dump >> >> only the pid of the task which has the task trace dumped. So it is easy to back >> >> track to the full stack with the pid. >> >> >> >> [ 58.469730] kworker/0:0 S 0000000000000000 0 4 2 0x00000000 >> >> [ 58.469735] ffff88082fcfde80 0000000000000046 ffff88082e9d8000 ffff88082fcfc010 >> >> [ 58.469739] ffff88082fce9860 0000000000011440 ffff88082fcfdfd8 ffff88082fcfdfd8 >> >> [ 58.469743] 0000000000011440 0000000000000000 ffff88082fcee180 ffff88082fce9860 >> >> [ 58.469747] Call Trace: >> >> [ 58.469751] [<ffffffff8108525a>] worker_thread+0x24b/0x250 >> >> [ 58.469754] [<ffffffff8108500f>] ? manage_workers+0x192/0x192 >> >> [ 58.469757] [<ffffffff810885bd>] kthread+0x82/0x8a >> >> [ 58.469760] [<ffffffff8141aed4>] kernel_thread_helper+0x4/0x10 >> >> [ 58.469763] [<ffffffff8108853b>] ? kthread_worker_fn+0x112/0x112 >> >> [ 58.469765] [<ffffffff8141aed0>] ? gs_change+0xb/0xb >> >> [ 58.469768] kworker/u:0 S 0000000000000004 0 5 2 0x00000000 >> >> [ 58.469773] ffff88082fcffe80 0000000000000046 ffff880800000000 ffff88082fcfe010 >> >> [ 58.469777] ffff88082fcea080 0000000000011440 ffff88082fcfffd8 ffff88082fcfffd8 >> >> [ 58.469781] 0000000000011440 0000000000000000 ffff88082fd4e9a0 ffff88082fcea080 >> >> [ 58.469785] Call Trace: >> >> [ 58.469786] <Same stack as pid 4> >> >> [ 58.470235] kworker/0:1 S 0000000000000000 0 13 2 0x00000000 >> >> [ 58.470255] ffff88082fd3fe80 0000000000000046 ffff880800000000 ffff88082fd3e010 >> >> [ 58.470279] ffff88082fcee180 0000000000011440 ffff88082fd3ffd8 ffff88082fd3ffd8 >> >> [ 58.470301] 0000000000011440 0000000000000000 ffffffff8180b020 ffff88082fcee180 >> >> [ 58.470325] Call Trace: >> >> [ 58.470332] <Same stack as pid 4> >> > >> > Given that pid's can be reused, I wonder if in a large time window the >> > output can be confusing? The dmesg ring buffer can be scaled with a >> > config option .. (CONFIG_LOG_BUF_LEN??) >> >> yes, we can always configure it to a larger value. however, it depends >> how much duplications you might have and the buffer can be easily >> filled up. this patch is useful which we can keep the same ring buffer >> size but get more information out of by only doing >> dedup of task stacks. >> > > The dedup has a cost, given large memory systems and the ability > to process some of these things in user space, does it make sense to > do it there? Just a thought, since I've not seen a whole lot of > duplication. When the system reach the state, the cost of getting enough information out of it for further debugging is worthy it. We've been using this with the netdump driver to get kernel dumps which proves works very well and saved lots of duplications. Any feedbacks on the current version is welcomed and I would like to get this included in Andrew's tree. Thank you --Ying > > -- > Three Cheers, > Balbir > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxxx For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href