Hello everyone, TL;DR: In Linux RT scheduler, how can rt_nr_running be non-zero AND active-bitmap NOT have any valid bit set? Details: Recently i encountered the following BUG() within the realtime scheduler (sched_rt.c) on 3.1.10 kernel. [101640.492840] kernel BUG at kernel/sched_rt.c:1126! This turns out to be 1126 BUG_ON(idx >= MAX_RT_PRIO); within the function pick_next_rt_entity() as shown here: http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/kernel/sched_rt.c?h=linux-3.1.y#n1115 What this means is that the scheduler failed to find a valid bit within the bitmap containing a prioritised list of active tasks. However before attempting to parse the bitmap, there is a check for a non-zero rt_nr_running. (i.e. parsing the bitmap should find atleast 1 bit of the active running rt task) So how could rt_nr_running be non-zero AND active-bitmap NOT have any valid bit set? The issue is observed on - a quad-core Cortex A9 SMP embedded system. - running an userspace app with ~25 RT threads (FIFO and RR) - typical ubuntu-core rootfs This issue consistently reproduces within 24-48hours on continuously running the system. Searching the net/lkml i could not find this issue reported before, though there are a few memory corruption bugs in scheduler. I have already backported the patches to fix know memory corruption issue from upstream kernel version and still encounter the above BUG(). Is anyone aware of this issue? Also including the kernel OOPS below. Do you see any tell-tale signs in the register-dump/backtrace that can point me in the right direction? [101640.488133] ------------[ cut here ]------------ [101640.492840] kernel BUG at kernel/sched_rt.c:1126! [101640.497621] Internal error: Oops - undefined instruction: 0 [#1] PREEMPT SMP [101640.504742] Modules linked in: misc_arz(P) audio_sta3_2(P) audio_sta3_1(P) audio_sta3(P) lamp_tlc8116_2(P) lamp_tlc8116_1(P) lamp_tlc8116(P) i2c_master_pcu9669(P) tegra_gpio_helper(P) outport_timer(P) intTimer(P) nvidia(P) [101640.524618] CPU: 0 Tainted: P (3.1.10 #1) [101640.530015] PC is at pick_next_task_rt+0x138/0x140 [101640.534888] LR is at __schedule+0x63c/0x858 [101640.539150] pc : [<c0042900>] lr : [<c0435190>] psr: 200f0093 [101640.539154] sp : e2d7fcc8 ip : e2d7fce8 fp : e2d7fce4 [101640.550785] r10: c05a4e60 r9 : e21f6d4c r8 : c05c8ab0 [101640.556084] r7 : e2d7e000 r6 : 00000001 r5 : c186fe60 r4 : c043f928 [101640.562685] r3 : c186ff60 r2 : 00000064 r1 : fffffff0 r0 : c186fe60 [101640.569288] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user [101640.576582] Control: 10c5387d Table: a335804a DAC: 00000015 [101640.582400] [101640.582403] PC: 0xc0042880: [101640.586839] 2880 eaffffeb e5932008 e3520000 0a000015 e2621000 e0012002 e16f2f12 e262205f [101640.595143] 28a0 eaffffe3 e30034b8 e2406f5a e18320d5 e3560000 e1c625f8 0affffc9 e2854d11 [101640.603449] 28c0 e2800070 e2844008 e1a01004 eb06b771 e5953448 e1a00006 e0534004 13a04001 [101640.611753] 28e0 e58544d4 e89da878 e593200c e2621000 e0012002 e16f2f12 e262207f eaffffcc [101640.620057] 2900 e7f001f2 e7f001f2 e1a0c00d e92ddff0 e24cb004 e24dd01c e2914038 e1a08002 [101640.628361] 2920 0a000074 e5913054 e3530000 1a000068 e30ba720 e3a09e4b e34ca05e e50ba038 [101640.636665] 2940 e2083005 e5945124 e3530001 0a000004 e1c423d0 e1c501d8 e0922000 e0a33001 [101640.644969] 2960 e1c423f0 e1a00005 ebffee88 e3a01000 e1a00005 ebfff0b7 e5943000 e3a01000 [101640.653276] [101640.653278] LR: 0xc0435110: [101640.657713] 5110 e1932f9f e2822001 e1831f92 e3310000 1afffffa eaffff5d e1a00008 eb000947 [101640.666017] 5130 e51b004c eb0009bc e1a00008 eb0009ba eaffffde e1a00008 e1a01009 e3a02001 [101640.674323] 5150 ebf032e2 eaffffdf e59f3248 e1a08007 e51ba070 e1a07006 e1a06005 e1a05004 [101640.682627] 5170 e1a04003 ea000002 e5944000 e3540000 0a000086 e5943018 e1a00005 e12fff33 [101640.690931] 5190 e3500000 0afffff7 e1a04005 e50ba070 e1a05006 e1a0a000 e1a06007 e1a07008 [101640.699235] 51b0 eafffeb5 e1a00004 eb0008fc eaffff48 e3000518 e3071ff4 e18420d0 e3a00e4b [101640.707539] 51d0 e34c105c e591104c e14b26fc e18420d0 e1a00001 e3a01000 e14b23fc e3083080 [101640.715845] 51f0 e34c305a e50b304c e593c000 e14b26dc e1530001 01520000 e14b03dc e3a03e51 [101640.724152] [101640.724154] SP: 0xe2d7fc48: [101640.728589] fc48 00000000 e21f6aa0 2c5bae94 00000000 e2d7fc74 e2d7fc68 c0042904 200f0093 [101640.736893] fc68 c000e394 00000000 e2d7fce4 e2d7fc80 c000e0c8 c00081a0 c186fe60 fffffff0 [101640.745197] fc88 00000064 c186ff60 c043f928 c186fe60 00000001 e2d7e000 c05c8ab0 e21f6d4c [101640.753501] fca8 c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8 c0435190 c0042900 200f0093 ffffffff [101640.761807] fcc8 c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64 e2d7fce8 c0435190 c00427d4 [101640.770111] fce8 e39de688 00000031 00000001 c05a4e60 00062904 00000000 e2d7fd54 e2d7fd10 [101640.778415] fd08 c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080 00005c71 c05a40c4 c05a4e60 [101640.786721] fd28 05303bf8 00005c71 00000000 e39de520 00000001 e2d7fe44 00000000 e2d7e038 [101640.795027] [101640.795030] IP: 0xe2d7fc68: [101640.799465] fc68 c000e394 00000000 e2d7fce4 e2d7fc80 c000e0c8 c00081a0 c186fe60 fffffff0 [101640.807770] fc88 00000064 c186ff60 c043f928 c186fe60 00000001 e2d7e000 c05c8ab0 e21f6d4c [101640.816076] fca8 c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8 c0435190 c0042900 200f0093 ffffffff [101640.824382] fcc8 c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64 e2d7fce8 c0435190 c00427d4 [101640.832686] fce8 e39de688 00000031 00000001 c05a4e60 00062904 00000000 e2d7fd54 e2d7fd10 [101640.840990] fd08 c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080 00005c71 c05a40c4 c05a4e60 [101640.849296] fd28 05303bf8 00005c71 00000000 e39de520 00000001 e2d7fe44 00000000 e2d7e038 [101640.857600] fd48 e2d7fe44 c064b020 e2d7fe7c e2d7e000 e2d7fd74 e2d7fd68 c04354fc c0434b60 [101640.865904] [101640.865907] FP: 0xe2d7fc64: [101640.870342] fc64 200f0093 c000e394 00000000 e2d7fce4 e2d7fc80 c000e0c8 c00081a0 c186fe60 [101640.878646] fc84 fffffff0 00000064 c186ff60 c043f928 c186fe60 00000001 e2d7e000 c05c8ab0 [101640.886952] fca4 e21f6d4c c05a4e60 e2d7fce4 e2d7fce8 e2d7fcc8 c0435190 c0042900 200f0093 [101640.895256] fcc4 ffffffff c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64 e2d7fce8 c0435190 [101640.903560] fce4 c00427d4 e39de688 00000031 00000001 c05a4e60 00062904 00000000 e2d7fd54 [101640.911866] fd04 e2d7fd10 c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080 00005c71 c05a40c4 [101640.920171] fd24 c05a4e60 05303bf8 00005c71 00000000 e39de520 00000001 e2d7fe44 00000000 [101640.928475] fd44 e2d7e038 e2d7fe44 c064b020 e2d7fe7c e2d7e000 e2d7fd74 e2d7fd68 c04354fc [101640.936781] [101640.936784] R0: 0xc186fde0: [101640.941219] fde0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101640.949521] fe00 00000000 00000000 00000000 00000000 0001125e 0000008d 00000000 00000000 [101640.957827] fe20 00000091 00000000 00000000 0000370a 00000004 00000000 00000000 00000000 [101640.966130] fe40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101640.974436] fe60 00000001 00000001 00000000 00000000 00000060 000000bd 000000a8 0000009c [101640.982741] fe80 009aa1e9 00000000 00000000 00000000 00000000 00000000 05303bf8 00005c71 [101640.991045] fea0 00000776 1f39f5f4 00000000 00000000 009a2769 00000000 0bf2a4f2 00000000 [101640.999350] fec0 00000000 00000000 00000000 00000000 4d4a2fa0 00000093 f116833c 00000091 [101641.007657] [101641.007659] R3: 0xc186fee0: [101641.012094] fee0 f116833c 00000091 00000000 00000000 c186fef0 c186fef0 00000000 00000000 [101641.020397] ff00 00000000 00000000 00000000 00000288 c186fe60 00000001 c18702e8 c18702e8 [101641.028701] ff20 c0617a50 00000000 00000400 00000000 00000000 00000000 00000000 00000000 [101641.037003] ff40 00000001 00000000 00000000 00000000 4d4a2fa0 00000093 00000000 00000000 [101641.045307] ff60 00000000 00000000 00000000 00000010 c186ff70 c186ff70 c186ff78 c186ff78 [101641.053611] ff80 c186ff80 c186ff80 c186ff88 c186ff88 c186ff90 c186ff90 c186ff98 c186ff98 [101641.061915] ffa0 c186ffa0 c186ffa0 c186ffa8 c186ffa8 c186ffb0 c186ffb0 c186ffb8 c186ffb8 [101641.070219] ffc0 c186ffc0 c186ffc0 c186ffc8 c186ffc8 c186ffd0 c186ffd0 c186ffd8 c186ffd8 [101641.078525] [101641.078527] R4: 0xc043f8a8: [101641.082963] f8a8 0004624c 00057ff5 0006dd25 000899bc 000abe5d 000d5d21 0010c73e 0014ff97 [101641.091267] f8c8 001a3434 0020ea87 00295252 003351fe 00400000 004fec05 00640e12 007c97d9 [101641.099573] f8e8 009aee73 00c3a13e 00f0f0f1 0130d190 017d05f4 01de5d6e 0253c825 02f14990 [101641.107877] f908 03a83a84 04924924 05b05b05 071c71c7 08d3dcb0 0b21642c 0e38e38e 11111111 [101641.116183] f928 c043f728 c0040f1c c00409e8 c0042dd8 00000000 c0042df4 c00427c8 c0040e94 [101641.124486] f948 c003f9d8 c00484ac c0048878 00000000 c0048800 c0040fa8 c003fb3c c0040094 [101641.132790] f968 c003fb0c c0042e98 00000000 c0048480 c004873c c00483ec c003c1e0 00000000 [101641.141094] f988 00000000 00000000 c050e51c c04f9d3c c04f9d48 00000000 00000063 00000063 [101641.149398] [101641.149401] R5: 0xc186fde0: [101641.153835] fde0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101641.162140] fe00 00000000 00000000 00000000 00000000 0001125e 0000008d 00000000 00000000 [101641.170443] fe20 00000091 00000000 00000000 0000370a 00000004 00000000 00000000 00000000 [101641.178747] fe40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101641.187047] fe60 00000001 00000001 00000000 00000000 00000060 000000bd 000000a8 0000009c [101641.195352] fe80 009aa1e9 00000000 00000000 00000000 00000000 00000000 05303bf8 00005c71 [101641.203655] fea0 00000776 1f39f5f4 00000000 00000000 009a2769 00000000 0bf2a4f2 00000000 [101641.211958] fec0 00000000 00000000 00000000 00000000 4d4a2fa0 00000093 f116833c 00000091 [101641.220262] [101641.220264] R7: 0xe2d7df80: [101641.224697] df80 1afffd0b e1a00004 eb178742 e3500000 059f3428 05853498 eafffd05 e1a00004 [101641.233001] dfa0 eb1785c4 e3500000 1afffcfd e1a00004 eb1785d2 e3500000 1afffcf9 e1a00004 [101641.241307] dfc0 eb1785ff e3500000 1afffcf5 e1a00004 eb178616 e3500000 1afffcf1 e1a00004 [101641.249611] dfe0 eb178624 e3500000 1afffced e1a00004 eb17866c e3500000 1afffce9 e1a00004 [101641.257915] e000 00000000 00000004 00000000 e21f6aa0 c05ebdd4 00000000 00000015 c186fe60 [101641.266220] e020 e21f6aa0 e2d7e000 c05c8ab0 e3221580 00000000 c05cb3f0 e2d7fd64 e2d7fce8 [101641.274526] e040 c0434ec0 00000000 00000000 00000000 00000000 00000000 01010000 00000000 [101641.282831] e060 48b19900 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101641.291135] [101641.291137] R8: 0xc05c8a30: [101641.295572] 8a30 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101641.303878] 8a50 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [101641.312184] 8a70 00000064 00008000 0000000f c188e020 00000008 00000004 c188e000 c1886000 [101641.320490] 8a90 c186f000 0000002c c1876000 c187e000 00000001 00001000 00000000 00000003 [101641.328796] 8ab0 012cb000 012d3000 012db000 012e3000 0005af00 00000001 000a3ffb 000ffeff [101641.337102] 8ad0 00000000 00000032 0000fffa 00000001 00000000 00000001 e3800f60 e381d580 [101641.345406] 8af0 e392ec60 00000011 c1892120 0001ffff e3800960 e38009c0 00000064 00000010 [101641.353710] 8b10 c1912120 0000ffff e3800900 00100000 e381a000 00000001 e3800f00 00000001 [101641.362016] [101641.362019] R9: 0xe21f6ccc: [101641.366455] 6ccc e339e1e0 e339e1e0 e21f6cd4 e21f6cd4 e21f6cdc e21f6cdc e300f1a0 e21f6ce8 [101641.374759] 6cec e21f6ce8 e21f6cf0 e21f6cf0 00000000 e39e3328 e39e3320 00000000 e38297ec [101641.383065] 6d0c e38297e0 e339e450 e39e34b0 e39e34a0 e339f8fc e21f755c 00000000 00000000 [101641.391371] 6d2c 48b194a8 0008834d 000178b7 0008834d 000178b7 00000000 00000000 00000000 [101641.399675] 6d4c 016b5f5b 002bbe0a 00000078 37e1e0b2 00000078 37e1e0b2 0000001e 00000000 [101641.407979] 6d6c 00000000 00000000 00000000 00000000 00000000 e21f6d80 e21f6d80 e21f6d88 [101641.416285] 6d8c e21f6d88 e21f6d90 e21f6d90 e320f200 e320f200 00000000 64616f4c 652e7265 [101641.424589] 6dac 0000666c 00000000 00000000 00000000 e31ab280 00000000 00000000 00000000 [101641.432893] [101641.432896] R10: 0xc05a4de0: [101641.437417] 4de0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.445723] 4e00 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.454027] 4e20 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.462333] 4e40 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.470639] 4e60 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.478945] 4e80 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.487251] 4ea0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.495555] 4ec0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 e7fddef0 [101641.503864] Process Loader.elf (pid: 571, stack limit = 0xe2d7e2f0) [101641.510205] Stack: (0xe2d7fcc8 to 0xe2d80000) [101641.514642] fcc0: c00427c8 c043f928 c186fe60 e21f6aa0 e2d7fd64 e2d7fce8 [101641.522895] fce0: c0435190 c00427d4 e39de688 00000031 00000001 c05a4e60 00062904 00000000 [101641.531146] fd00: e2d7fd54 e2d7fd10 c00404f0 c003e308 c05a4e60 c05a4e60 c05a8080 00005c71 [101641.539397] fd20: c05a40c4 c05a4e60 05303bf8 00005c71 00000000 e39de520 00000001 e2d7fe44 [101641.547648] fd40: 00000000 e2d7e038 e2d7fe44 c064b020 e2d7fe7c e2d7e000 e2d7fd74 e2d7fd68 [101641.555899] fd60: c04354fc c0434b60 e2d7fdc4 e2d7fd78 c007dccc c04354b8 e2d7fdc4 e2d7fd88 [101641.564150] fd80: c007e710 e2d7fe50 c003e048 00000002 00000001 00000002 e2d7fdb4 e2d7e000 [101641.572400] fda0: 00000000 00000000 e2d7fe44 00000000 e2d7fe7c 481df054 e2d7feac e2d7fdc8 [101641.580650] fdc0: c007e934 c007dbe4 e2d7fe7c c0094c4c 00000000 c186fe60 00000000 00000002 [101641.588900] fde0: c187fe60 c05c8ab0 e39de8b8 00000002 00000000 c05a4e60 e2d7fe44 e2d7fe08 [101641.597152] fe00: c004c24c c043760c c007dd90 c05a4e60 600f0013 c05a4e60 00000000 e39de520 [101641.605405] fe20: e39de528 e3191e44 c064b01c c064b028 e2d7e000 c064b020 e2d7fe74 e2d7fe48 [101641.613656] fe40: c007e57c 00000031 e2d7fe48 e2d7fe48 e2d7fe50 e2d7fe50 e21f6aa0 00000000 [101641.621906] fe60: 481df000 e3221580 00000054 00000000 00000000 00000000 ffffffff c064b020 [101641.630158] fe80: e3221580 00000000 481df054 00000002 00000002 00000000 e2d7e000 481df054 [101641.638408] fea0: e2d7ff44 e2d7feb0 c00801c8 c007e78c ffffffff e2d7ff50 00000000 c061404c [101641.646658] fec0: ffffffff e10a4000 ffffffff 00000000 e2d7ff04 e2d7fee0 c0072a40 c0094c4c [101641.654910] fee0: 00000000 e21f6c78 e10a401c e21f6aa0 e2d7e000 c05c8ab0 e2d7ff1c e2d7ff08 [101641.663160] ff00: c0072a6c c0072a04 0126113e 00000000 3800038a 00000000 e2d7ff74 00000000 [101641.671412] ff20: 00000080 481df054 00000002 00000000 e2d7e000 00000000 e2d7ffa4 e2d7ff48 [101641.679662] ff40: c0080be0 c00800e8 481df054 00000000 ffffffff 6553f400 00000107 c000e604 [101641.687912] ff60: e2d7e000 481f1b8c 6553f400 00000107 c000e604 48b18d08 00000008 481df054 [101641.696162] ff80: 00000002 00000001 000000f0 c000e604 e2d7e000 00000000 00000000 e2d7ffa8 [101641.704414] ffa0: c000e480 c0080b5c 481df054 00000002 481df054 00000080 00000002 00000000 [101641.712664] ffc0: 481df054 00000002 00000001 000000f0 00000080 00000001 00000000 48b19674 [101641.720914] ffe0: 000000f0 48b18cd0 40103f53 40106386 600f0030 481df054 00000000 00000000 [101641.729161] Backtrace: [101641.731703] [<c00427c8>] (pick_next_task_rt+0x0/0x140) from [<c0435190>] (__schedule+0x63c/0x858) [101641.740643] r6:e21f6aa0 r5:c186fe60 r4:c043f928 r3:c00427c8 [101641.746409] [<c0434b54>] (__schedule+0x0/0x858) from [<c04354fc>] (schedule+0x50/0x68) [101641.754405] [<c04354ac>] (schedule+0x0/0x68) from [<c007dccc>] (futex_wait_queue_me+0xf4/0x108) [101641.763177] [<c007dbd8>] (futex_wait_queue_me+0x0/0x108) from [<c007e934>] (futex_wait+0x1b4/0x27c) [101641.772298] [<c007e780>] (futex_wait+0x0/0x27c) from [<c00801c8>] (do_futex+0xec/0xa74) [101641.780376] [<c00800dc>] (do_futex+0x0/0xa74) from [<c0080be0>] (sys_futex+0x90/0x188) [101641.788372] [<c0080b50>] (sys_futex+0x0/0x188) from [<c000e480>] (ret_fast_syscall+0x0/0x30) [101641.796883] Code: e0012002 e16f2f12 e262207f eaffffcc (e7f001f2) regards CVS -- To unsubscribe from this list: send the line "unsubscribe stable-rt" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
