Recently I was experiencing frequent io hang when testing with fio with 4K random write. Fio iops dropped to 0 for about 20 seconds every several minutes. After some debugging, I discovered that it is the incremental gc that cause this problem. My cache disk is relatively small (375GiB with 4K block size and 512K bucket size), backing hdds are 4 x 1 TiB. I cannot reproduce this on another environment with bigger cache disk. When running 4K random write fio bench, the buckets are consumed very quickly and soon it has to invalidate some bucket (this happens quite often). Since the cache disk is small, a lot of write io will soon reach sectors_to_gc and trigger gc thread. Write io will also increase search_inflight, which cause gc thread to sleep for 100ms. This will cause gc procedure to execute for a long time, and invalidating bucket for the write io will wait for the whole gc procedure. After removing the 100ms sleep from the incremental gc patch, the io never hang any more. I think for small ssd, sleeping for 100ms seems too long or maybe write io should not trigger gc thread to sleep for 100ms?
