On Mon, Oct 24, 2011 at 10:07:19AM -0700, Eric W. Biederman wrote: > Am?rico Wang <xiyou.wangcong at gmail.com> writes: > > > On Mon, Oct 24, 2011 at 11:14 PM, Eric W. Biederman > > <ebiederm at xmission.com> wrote: > >> Michael Holzheu <holzheu at linux.vnet.ibm.com> writes: > >> > >>> Hello Vivek, > >>> > >>> In our tests we ran into the following scenario: > >>> > >>> Two CPUs have called panic at the same time. The first CPU called > >>> crash_kexec() and the second CPU called smp_send_stop() in panic() > >>> before crash_kexec() finished on the first CPU. So the second CPU > >>> stopped the first CPU and therefore kdump failed. > >>> > >>> 1st CPU: > >>> panic()->crash_kexec()->mutex_trylock(&kexec_mutex)-> do kdump > >>> > >>> 2nd CPU: > >>> panic()->crash_kexec()->kexec_mutex already held by 1st CPU > >>> ? ? ? ?->smp_send_stop()-> stop CPU 1 (stop kdump) > >>> > >>> How should we fix this problem? One possibility could be to do > >>> smp_send_stop() before we call crash_kexec(). > >>> > >>> What do you think? > >> > >> smp_send_stop is insufficiently reliable to be used before crash_kexec. > >> > >> My first reaction would be to test oops_in_progress and wait until > >> oops_in_progress == 1 before calling smp_send_stop. > >> > > > > +1 > > > > One of my colleague mentioned the same problem with me inside > > RH, given the fact that the race condition window is small, it would > > not be easy to reproduce this scenario. > > As for reproducing it I have a hunch you could hack up something > horrible with smp_call_function and kprobes. > > > On a little more reflection we can't wait until oops_in_progress goes > to 1 before calling smp_send_stop. Because if crash_kexec is not > involved nothing we will never call smp_send_stop. > > So my second thought is to introduce another atomic variable > panic_in_progress, visible only in panic. The cpu that sets > increments panic_in_progress can call smp_send_stop. The rest of > the cpus can just go into a busy wait. That should stop nasty > fights about who is going to come out of smp_send_stop first. Introducing panic_on_oops atomic sounds good. Thanks Vivek
