[Sorry for the Spam detection ... ] Hi Aleksei, This is a very interesting finding, I understand that ceph blustore will issue fdatasync requests when it tries to flush data or metadata (via bluefs) to the OSD device. But I'm surprised to see so much pressure it can bring to the backing device. May I know how do you measure the number of flush requests to the backing device per second that is sent from the bcache with the REQ_PREFLUSH flag? (ftrace to some bcache tracepoint ?) My understanding is that the bcache doesn't need to wait for the flush requests to be completed from the backing device in order to finish the write request, since it used a new bio "flush" for the backing device. So I don't think this will increase the fdatasync latency as long as the write can be performed in a writeback mode. It does increase the read latency if the read io missed the cache. Or maybe I am missing something, let me know how did you observe the latency increasing from bcache layer , I would want to do some experiments as well? Regards, Dongdong On Fri, Nov 5, 2021 at 7:21 PM Aleksei Zakharov <zakharov.a.g@xxxxxxxxx> wrote: > > Hi all, > > I've used bcache a lot for the last three years, mostly in writeback mode with ceph, and I faced a strange behavior. When there's a heavy write load on the bcache device with a lot of fsync()/fdatasync() requests, the bcache device issues a lot of flush requests to the backing device. If the writeback rate is low, then there might be hundreds of flush requests per second issued to the backing device. > > If the writeback rate growths, then latency of the flush requests increases. And latency of the bcache device increases as a result and the application experiences higher disk latency. So, this behavior of bcache slows the application in it's I/O requests when writeback rate becomes high. > > This workload pattern with a lot of fsync()/fdatasync() requests is a common for a latency-sensitive applications. And it seems that this bcache behavior slows down this type of workloads. > > As I understand, if a write request with REQ_PREFLUSH is issued to bcache device, then bcache issues new empty write request with REQ_PREFLUSH to the backing device. What is the purpose of this behavior? It looks like it might be eliminated for the better performance. > > -- > Regards, > Aleksei Zakharov > alexzzz.ru
