Hi. I am removing linux-fsdevel from the CC, because it's not really the quorum for it. On Wed, Sep 25, 2019 at 03:51:27PM -0700, Jianshen Liu wrote: > Hi, > > I am working on a project trying to evaluate the performance of a > workload running on a storage device. I don't want the benchmark > result depends on a specific platform (e.g., a platform with X GiB of > physical memory). Well, this does not sound realistic to me. Memory is just only one of the variables in how IO throughput will perform. You bypass memory, then, what about IO Controller, Disks, Storage cache, etc etc etc? All of these are 'platform specific'. > Because it prevents people from reproducing the same > result on a different platform configuration. Every platform will behave different, memory is only one factor. > Think about you are > benchmarking a read-heavy workload, with data caching enabled you may > end up with just testing the performance of the system memory. > Not really true. It all depends on what kind of workload you are talking about. And what you are trying to measure. A read-heavy workload, may well use a lot of page cache, but it all depends on the IO patterns, and exactly what statistics you care about. Are you trying to measure how well a storage solution will perform on a random workload? On a sequential workload? Are you trying to measure how well an application will perform? If that's the case, removing the page cache from the equation really matters? I.e. will it give you realistic results? If you are benchmarking systems with 'random' kinds of workloads, there are several tools around there which can help, and you can configure to use DIO. > Currently, I'm thinking how to eliminate the cache effects created by > the page cache. Direct I/O is a good option for testing with a single > application but is not good for testing with unknown > applications/workloads. You can use many tools for that purpose, which can 'emulate' different workloads, without needing to modify a specific application. But if you are trying to create benchmarks for a specific application, if your benchmarks uses DIO or not, will depend on if the application uses DIO or not. > Therefore, it is not feasible to ask people to > modify the application source code before running the benchmark. Well, IMHO, your approach is wrong. First, if you are benchmarking how an application will perform, you need to use the same IO patterns the application is using, i.e. you won't need to modify it. If it does not use direct IO, benchmarking a system using direct IO will bring you something very wrong data. And the opposite is true, if the application uses direct IO, you don't want to benchmark a system by using the page cache, because one of the things you really want to measure is how well the application's cache is performing. Also, direct IO is also not a good option to use when you 'don't know how to issue I/O requests'. All I/O requests submitted using direct IO must be aligned. So, if the application does not issue aligned requests, the IO requests will fail. I remember some filesystems to had an option to 'open all files with O_DIRECT by default', and many problems being created because IO requests to such files were not all sector aligned. > > Making changes within the kernel may only be the option because it is > transparent to all user-space applications. I will hit the same point again :) and my question is: Why? :) Will you be using a custom kernel? With this modification? If not, you will not be gathering trustable data anyway. > The problem is I don't > know how to modify the kernel so that it does not use the page cache > for any IOs to a specific storage device. I have tried to append a > fadvise64() call with POSIX_FADV_DONTNEED to the end of each > read/write system calls. The performance of this approach is far from > using Direct I/O. It is also unable to eliminate the caching effects > under concurrent I/Os. I'm looking for any advice here to point me an > efficient way to remove the cache effects from the page cache. > > Thanks, > Jianshen Benchmarking systems is an 'art', and I am certainly not an expert on it, but at first, it looks like you are trying to create a 'generic benchmark' to some generic random system. And I will tell you, this is not going to work well. We have tons of cases and stories about people running benchmark X on system Z, and it performing 'well', but when running their real workload, everything starts to perform poorly, exactly because they did not use the correct benchmark at first. You have several layers in a storage stack, which starts from how the application handles its own IO requests. And each layer which will behave differently on each type of workload. Apologies to be repeating myself: If you are trying to measure only a storage solution, there are several tools around which can create different kinds of workload. If you are trying to measure an application performance on solution X, well, it is pointless to measure direct IO if the application does not use it or vice-versa, so, modifying an application, again, is not what you will want to do for benchmarking, for sure. Hope to have helped (and not created more questions :) Cheers -- Carlos
