Hi Martijn, Martijn de Gouw <martijn.de.gouw@xxxxxxxxxxxxxxxxxxxxxxxxx> writes: > One of the symptoms is that our monitoring system complains about not > being able to stat() every now and then, the next scraping cycle, stat() > works again. Even when the mounts are not accesses at all. > > Also, lot of applications get stuck on either accessing data on the > mounts, or performing stat() like operations on the mounts. > > For us, the worst part is that applications end up in 'D'. The number of > 'D' processes pile up really quickly, blocking users from performing > their work. Gah, sorry to hear. Thanks for the report. If you could pin down a specific way to reproduce some of those hangs that would be of great help. > We are running Linux 4.20.17 SMP PREEMPT on all machines. We tried > upgrading to > 5.x, but caused even more problems and kernel hangs. 5.0 and 5.1 really fixed a lot of issues with credits and reconnection. > I do not really have a clue where to start debugging. I enabled kernel > debug options suggested on the wiki, but the amount of logging is > immense now. Yes that is normal. > Can you provide any pointers where to look or start debugging? > Or any help on how to kill those D processes and get our Linux servers > stable again? Are there any kernel oops/panic with stack traces and register dumps in the log? You can inspect the kernel stack trace of the hung processes (to see where they are stuck) by printing the file /proc/<pid>/stack. Cheers, -- Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)
