> [Sorry for the Spam detection ... ] > > Hi Aleksei, > > This is a very interesting finding, I understand that ceph blustore > will issue fdatasync requests when it tries to flush data or metadata > (via bluefs) to the OSD device. But I'm surprised to see so much > pressure it can bring to the backing device. > May I know how do you measure the number of flush requests to the > backing device per second that is sent from the bcache with the > REQ_PREFLUSH flag? (ftrace to some bcache tracepoint ?) That was easy: the writeback rate was minimal and there were a lot of write requests to the backing device in iostat -xtd 1 output and bytes/s was too small for that number of writes. It was relatively old kernel, so flushes were not separated in the block layer stats yet. > > My understanding is that the bcache doesn't need to wait for the flush > requests to be completed from the backing device in order to finish > the write request, since it used a new bio "flush" for the backing > device. > So I don't think this will increase the fdatasync latency as long as > the write can be performed in a writeback mode. It does increase the > read latency if the read io missed the cache. Hm, that might be truth for the reads, i'll do some experiments. But, I don't see any reason to send flush request to the backing device if there's nothing to flush. > Or maybe I am missing something, let me know how did you observe the > latency increasing from bcache layer , I would want to do some > experiments as well? I'll do some experiments and come back with more details on the issue in a week! Already quit that job and don't work with ceph anymore, but still thinking about this interesting issue. > > Regards, > Dongdong > > On Fri, Nov 5, 2021 at 7:21 PM Aleksei Zakharov <zakharov.a.g@xxxxxxxxx> wrote: > >> Hi all, >> >> I've used bcache a lot for the last three years, mostly in writeback mode with ceph, and I faced a strange behavior. When there's a heavy write load on the bcache device with a lot of fsync()/fdatasync() requests, the bcache device issues a lot of flush requests to the backing device. If the writeback rate is low, then there might be hundreds of flush requests per second issued to the backing device. >> >> If the writeback rate growths, then latency of the flush requests increases. And latency of the bcache device increases as a result and the application experiences higher disk latency. So, this behavior of bcache slows the application in it's I/O requests when writeback rate becomes high. >> >> This workload pattern with a lot of fsync()/fdatasync() requests is a common for a latency-sensitive applications. And it seems that this bcache behavior slows down this type of workloads. >> >> As I understand, if a write request with REQ_PREFLUSH is issued to bcache device, then bcache issues new empty write request with REQ_PREFLUSH to the backing device. What is the purpose of this behavior? It looks like it might be eliminated for the better performance. >> >> -- >> Regards, >> Aleksei Zakharov >> alexzzz.ru -- Regards, Aleksei Zakharov alexzzz.ru
