IO hang when cache do not have enough buckets on small SSD

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello, Mr. Li.
Recently I was experiencing frequent io hang when testing with fio
with 4K random write. Fio iops dropped  to 0 for about 20 seconds
every several minutes.
After some debugging, I discovered that it is the incremental gc that
cause this problem.
My cache disk is relatively small (375GiB with 4K block size and 512K
bucket size), backing hdds are 4 x 1 TiB. I cannot reproduce this on
another environment with bigger cache disk.
When running 4K random write fio bench, the buckets are consumed  very
quickly and soon it has to invalidate some bucket (this happens quite
often). Since the cache disk is small, a lot of write io will soon
reach sectors_to_gc and trigger gc thread. Write io will also increase
search_inflight, which cause gc thread to sleep for 100ms. This will
cause gc procedure to execute for a long time, and invalidating bucket
for the write io will wait for the whole gc procedure.
After removing the 100ms sleep from the incremental gc patch,  the io
never hang any more.
I think for small ssd, sleeping for 100ms seems too long or maybe
write io should not trigger gc thread to sleep for 100ms?
Thank you very much.



[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM Kernel]     [Linux Filesystem Development]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux