On 2018/8/6 9:33 PM, Stefan Priebe - Profihost AG wrote: > Hi Coly, > Am 06.08.2018 um 15:06 schrieb Coly Li: >> On 2018/8/6 2:33 PM, Stefan Priebe - Profihost AG wrote: >>> Hi Coly, >>> >>> while running the SLES15 kernel i observed a workqueue lockup and a >>> totally crashed system today. >>> >>> dmesg output is about 3,5mb but it seems it just repeats the >>> bch_data_insert_keys msg. >>> >> >> Hi Stefan, >> >> Thanks for your information! >> >> Could you please to give me any hint on how to reproduce it ? Even it is >> not stable reproducible, a detailed procedure may help me a lot. > > i'm sorry but i can't reproduce it. It happens just out of nothing in > our ceph production cluster. I see. Could you please share the configuration information. E.g. - How many CPU cores - How many physical memory - How large the SSD size, NVMe or SATA - How many SSDs - How large (many) the backing hard drives are I try to simulate similar workload with fio, see how lucky I am. Thanks. Coly Li >>> The beginning is: >>> >>> 2018-08-06 02:08:06 BUG: workqueue lockup - pool cpus=1 node=0 >>> flags=0x1 nice=0 stuck for 51s! >>> 2018-08-06 02:08:06 pending: memcg_kmem_cache_create_func >>> 2018-08-06 02:08:06 delayed: memcg_kmem_cache_create_func >>> 2018-08-06 02:08:06 workqueue bcache: flags=0x8 >>> 2018-08-06 02:08:06 pwq 22: cpus=11 node=0 flags=0x0 nice=0 active=1/256 >>> 2018-08-06 02:08:06 in-flight: 1764369:bch_data_insert_keys [bcache] >>> 2018-08-06 02:08:06 pwq 18: cpus=9 node=0 flags=0x1 nice=0 >>> active=256/256 MAYDAY >>> 2018-08-06 02:08:06 in-flight: 1765894:bch_data_insert_keys >>> [bcache], 1765908:bch_data_insert_keys [bcache], >>> 1765931:bch_data_insert_keys [bcache], 1765984:bch_data_insert_keys >>> [bcache], 1765815:bch_data_insert_keys [bcache], >>> 1765893:bch_data_insert_keys [bcache], 1765981:bch_data_insert_keys >>> [bcache], 1765875:bch_data_insert_keys [bcache], >>> 1765963:bch_data_insert_keys [bcache], 1765960:bch_data_insert_keys >>> [bcache], 1765889:bch_data_insert_keys [bcache], >>> 1765989:bch_data_insert_keys [bcache], 1765897:bch_data_insert_keys >>> [bcache], 1765911:bch_data_insert_keys [bcache], >>> 1765924:bch_data_insert_keys [bcache], 1765808:bch_data_insert_keys >>> [bcache], 1765879:bch_data_insert_keys [bcache], >>> 1765948:bch_data_insert_keys [bcache], 1765970:bch_data_insert_keys >>> [bcache], 1765859:bch_data_insert_keys [bcache], >>> 1765884:bch_data_insert_keys [bcache] >>> 2018-08-06 02:08:06 , 1765952:bch_data_insert_keys [bcache], >>> 1765990:bch_data_insert_keys [bcache], 1765817:bch_data_insert_keys >>> [bcache], 1765858:bch_data_insert_keys [bcache], >>> 1765928:bch_data_insert_keys [bcache], 1765936:bch_data_insert_keys >>> [bcache], 1762396:bch_data_insert_keys [bcache], >>> 1765831:bch_data_insert_keys [bcache], 1765847:bch_data_insert_keys >>> [bcache], 1765895:bch_data_insert_keys [bcache], >>> 1765925:bch_data_insert_keys [bcache], 1765967:bch_data_insert_keys >>> [bcache], 1765798:bch_data_insert_keys [bcache], >>> 1765827:bch_data_insert_keys [bcache], 1765857:bch_data_insert_keys >>> [bcache], 1765979:bch_data_insert_keys [bcache], >>> 1765809:bch_data_insert_keys [bcache], 1765856:bch_data_insert_keys >>> [bcache], 1765878:bch_data_insert_keys [bcache], >>> 1765918:bch_data_insert_keys [bcache], 1765934:bch_data_insert_keys [bcache] >>> 2018-08-06 02:08:06 , 1765982:bch_data_insert_keys [bcache], >>> 1765813:bch_data_insert_keys [bcache], 1765883:bch_data_insert_keys >>> [bcache], 1765993:bch_data_insert_keys [bcache], >>> 1765834:bch_data_insert_keys [bcache], 1765920:bch_data_insert_keys >>> [bcache], 1765962:bch_data_insert_keys [bcache], >>> 1765788:bch_data_insert_keys [bcache], 1765882:bch_data_insert_keys >>> [bcache], 1765942:bch_data_insert_keys [bcache], >>> 1765825:bch_data_insert_keys [bcache], 1765854:bch_data_insert_keys >>> [bcache], 1765902:bch_data_insert_keys [bcache], >>> 1765838:bch_data_insert_keys [bcache], 1765868:bch_data_insert_keys >>> [bcache], 1765932:bch_data_insert_keys [bcache], >>> 1765944:bch_data_insert_keys [bcache], 1765975:bch_data_insert_keys >>> [bcache], 1765983:bch_data_insert_keys [bcache], >>> 1765810:bch_data_insert_keys [bcache], 1765863:bch_data_insert_keys [bcache] -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
