Am 06.08.2018 um 16:21 schrieb Coly Li: > On 2018/8/6 9:33 PM, Stefan Priebe - Profihost AG wrote: >> Hi Coly, >> Am 06.08.2018 um 15:06 schrieb Coly Li: >>> On 2018/8/6 2:33 PM, Stefan Priebe - Profihost AG wrote: >>>> Hi Coly, >>>> >>>> while running the SLES15 kernel i observed a workqueue lockup and a >>>> totally crashed system today. >>>> >>>> dmesg output is about 3,5mb but it seems it just repeats the >>>> bch_data_insert_keys msg. >>>> >>> >>> Hi Stefan, >>> >>> Thanks for your information! >>> >>> Could you please to give me any hint on how to reproduce it ? Even it is >>> not stable reproducible, a detailed procedure may help me a lot. >> >> i'm sorry but i can't reproduce it. It happens just out of nothing in >> our ceph production cluster. > > I see. Could you please share the configuration information. E.g. sure. > - How many CPU cores 12 > - How many physical memory 64GB > - How large the SSD size, NVMe or SATA bcache cache size: 250GB SATA SSD > - How many SSDs 1x > - How large (many) the backing hard drives are 2x 1TB > I try to simulate similar workload with fio, see how lucky I am. Thanks! Generally the workload at the timeframe was mostly read, fssync inside guests and fstrim. Stefan > > Thanks. > > Coly Li > >>>> The beginning is: >>>> >>>> 2018-08-06 02:08:06 BUG: workqueue lockup - pool cpus=1 node=0 >>>> flags=0x1 nice=0 stuck for 51s! >>>> 2018-08-06 02:08:06 pending: memcg_kmem_cache_create_func >>>> 2018-08-06 02:08:06 delayed: memcg_kmem_cache_create_func >>>> 2018-08-06 02:08:06 workqueue bcache: flags=0x8 >>>> 2018-08-06 02:08:06 pwq 22: cpus=11 node=0 flags=0x0 nice=0 active=1/256 >>>> 2018-08-06 02:08:06 in-flight: 1764369:bch_data_insert_keys [bcache] >>>> 2018-08-06 02:08:06 pwq 18: cpus=9 node=0 flags=0x1 nice=0 >>>> active=256/256 MAYDAY >>>> 2018-08-06 02:08:06 in-flight: 1765894:bch_data_insert_keys >>>> [bcache], 1765908:bch_data_insert_keys [bcache], >>>> 1765931:bch_data_insert_keys [bcache], 1765984:bch_data_insert_keys >>>> [bcache], 1765815:bch_data_insert_keys [bcache], >>>> 1765893:bch_data_insert_keys [bcache], 1765981:bch_data_insert_keys >>>> [bcache], 1765875:bch_data_insert_keys [bcache], >>>> 1765963:bch_data_insert_keys [bcache], 1765960:bch_data_insert_keys >>>> [bcache], 1765889:bch_data_insert_keys [bcache], >>>> 1765989:bch_data_insert_keys [bcache], 1765897:bch_data_insert_keys >>>> [bcache], 1765911:bch_data_insert_keys [bcache], >>>> 1765924:bch_data_insert_keys [bcache], 1765808:bch_data_insert_keys >>>> [bcache], 1765879:bch_data_insert_keys [bcache], >>>> 1765948:bch_data_insert_keys [bcache], 1765970:bch_data_insert_keys >>>> [bcache], 1765859:bch_data_insert_keys [bcache], >>>> 1765884:bch_data_insert_keys [bcache] >>>> 2018-08-06 02:08:06 , 1765952:bch_data_insert_keys [bcache], >>>> 1765990:bch_data_insert_keys [bcache], 1765817:bch_data_insert_keys >>>> [bcache], 1765858:bch_data_insert_keys [bcache], >>>> 1765928:bch_data_insert_keys [bcache], 1765936:bch_data_insert_keys >>>> [bcache], 1762396:bch_data_insert_keys [bcache], >>>> 1765831:bch_data_insert_keys [bcache], 1765847:bch_data_insert_keys >>>> [bcache], 1765895:bch_data_insert_keys [bcache], >>>> 1765925:bch_data_insert_keys [bcache], 1765967:bch_data_insert_keys >>>> [bcache], 1765798:bch_data_insert_keys [bcache], >>>> 1765827:bch_data_insert_keys [bcache], 1765857:bch_data_insert_keys >>>> [bcache], 1765979:bch_data_insert_keys [bcache], >>>> 1765809:bch_data_insert_keys [bcache], 1765856:bch_data_insert_keys >>>> [bcache], 1765878:bch_data_insert_keys [bcache], >>>> 1765918:bch_data_insert_keys [bcache], 1765934:bch_data_insert_keys [bcache] >>>> 2018-08-06 02:08:06 , 1765982:bch_data_insert_keys [bcache], >>>> 1765813:bch_data_insert_keys [bcache], 1765883:bch_data_insert_keys >>>> [bcache], 1765993:bch_data_insert_keys [bcache], >>>> 1765834:bch_data_insert_keys [bcache], 1765920:bch_data_insert_keys >>>> [bcache], 1765962:bch_data_insert_keys [bcache], >>>> 1765788:bch_data_insert_keys [bcache], 1765882:bch_data_insert_keys >>>> [bcache], 1765942:bch_data_insert_keys [bcache], >>>> 1765825:bch_data_insert_keys [bcache], 1765854:bch_data_insert_keys >>>> [bcache], 1765902:bch_data_insert_keys [bcache], >>>> 1765838:bch_data_insert_keys [bcache], 1765868:bch_data_insert_keys >>>> [bcache], 1765932:bch_data_insert_keys [bcache], >>>> 1765944:bch_data_insert_keys [bcache], 1765975:bch_data_insert_keys >>>> [bcache], 1765983:bch_data_insert_keys [bcache], >>>> 1765810:bch_data_insert_keys [bcache], 1765863:bch_data_insert_keys [bcache] > -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
