Hi Johannes and Greg
On 2/12/2022 12:35 AM, Abhinav Kumar wrote:
Hi Johannes
On 2/12/2022 12:24 AM, Johannes Berg wrote:
On Fri, 2022-02-11 at 23:52 -0800, Abhinav Kumar wrote:
The thread is writing the data to a file in local storage. From our
profiling, the read is the one taking the time not the write.
That seems kind of hard to believe, let's say it's a 4/3 split (4
minutes reading, 3 minutes writing, to make read > write as you say),
and 3MiB size, that'd mean you get 12.8KiB/sec? That seems implausibly
low, unless you're reading with really tiny buffers?
Can you strace this somehow? (with timestamp info)
Yes, like I have already mentioned in earlier comments, we continue to
check whats taking that long.
Once we find something from our analysis and also have the trace, will
update the thread.
Just doubling what we have currently. I am not sure how the current 5
mins timeout came from.
To be honest it came out of thin air, and wasn't really meant as a limit
on how fast you can read (feels like even if it's tens of MiB you should
read it in milliseconds into userspace), but more of a maximum time that
we're willing to waste kernel memory if nobody is around to read the
data.
I thought it'd be better if we could somehow pin it while the userspace
is reading it, but OTOH maybe that's actually bad, since that means
userspace (though suitably privileged) could pin this kernel memory
indefinitely.
johannes
So, we were able to narrow down the bottle-neck further. The tiny
buffers which Johannes was referring to is coming from the sysfs method
below.
It defaults to a PAGE_SIZE worth of data which results in taking a lot
of time due to many number of reads.
If we increase the length to match the size of our data like below we
are able to finish the read in almost no-time.
--- a/fs/kernfs/file.c
+++ b/fs/kernfs/file.c
@@ -184,10 +184,11 @@ static const struct seq_operations kernfs_seq_ops = {
static ssize_t kernfs_file_read_iter(struct kiocb *iocb, struct
iov_iter *iter)
{
struct kernfs_open_file *of = kernfs_of(iocb->ki_filp);
- ssize_t len = min_t(size_t, iov_iter_count(iter), PAGE_SIZE);
+ ssize_t len = min_t(size_t, iov_iter_count(iter), (PAGE_SIZE *
768));
const struct kernfs_ops *ops;
char *buf;
+ pr_err("[hbc debug] %s, len:%d\n", __func__, len);
buf = of->prealloc_buf;
if (buf)
mutex_lock(&of->prealloc_mutex);
( 768 because the size of our dump was ~3MB so that would be ~ 768 * 4kB
block sizes )
We also did some profiling around how much increasing the block size
helps and here is the data:
Block size cost
4KB 229s
8KB 86s
3MB 2s
So looks like 2 * block size OR 4 * block size can help quite a bit.
Hence, while we are exploring some options like reducing the size of the
dump etc, I wanted to also check if increasing the block size to like 4
* 4Kb could be a solution because it will bring down the read times
drastically based on our profiling.
Thanks
Abhinav
