Re: [PATCH] devcoredump: increase the device delete timeout to 10 mins

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Johannes and Greg

On 2/12/2022 12:35 AM, Abhinav Kumar wrote:
Hi Johannes

On 2/12/2022 12:24 AM, Johannes Berg wrote:
On Fri, 2022-02-11 at 23:52 -0800, Abhinav Kumar wrote:

The thread is writing the data to a file in local storage. From our
profiling, the read is the one taking the time not the write.


That seems kind of hard to believe, let's say it's a 4/3 split (4
minutes reading, 3 minutes writing, to make read > write as you say),
and 3MiB size, that'd mean you get 12.8KiB/sec? That seems implausibly
low, unless you're reading with really tiny buffers?

Can you strace this somehow? (with timestamp info)


Yes, like I have already mentioned in earlier comments, we continue to check whats taking that long.

Once we find something from our analysis and also have the trace, will update the thread.

Just doubling what we have currently. I am not sure how the current 5
mins timeout came from.


To be honest it came out of thin air, and wasn't really meant as a limit
on how fast you can read (feels like even if it's tens of MiB you should
read it in milliseconds into userspace), but more of a maximum time that
we're willing to waste kernel memory if nobody is around to read the
data.

I thought it'd be better if we could somehow pin it while the userspace
is reading it, but OTOH maybe that's actually bad, since that means
userspace (though suitably privileged) could pin this kernel memory
indefinitely.

johannes

So, we were able to narrow down the bottle-neck further. The tiny buffers which Johannes was referring to is coming from the sysfs method below.

It defaults to a PAGE_SIZE worth of data which results in taking a lot of time due to many number of reads.

If we increase the length to match the size of our data like below we are able to finish the read in almost no-time.

--- a/fs/kernfs/file.c
+++ b/fs/kernfs/file.c
@@ -184,10 +184,11 @@ static const struct seq_operations kernfs_seq_ops = {
static ssize_t kernfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
 {
        struct kernfs_open_file *of = kernfs_of(iocb->ki_filp);
-       ssize_t len = min_t(size_t, iov_iter_count(iter), PAGE_SIZE);
+ ssize_t len = min_t(size_t, iov_iter_count(iter), (PAGE_SIZE * 768));
        const struct kernfs_ops *ops;
        char *buf;

+       pr_err("[hbc debug] %s, len:%d\n", __func__, len);
        buf = of->prealloc_buf;
        if (buf)
                mutex_lock(&of->prealloc_mutex);

( 768 because the size of our dump was ~3MB so that would be ~ 768 * 4kB block sizes )

We also did some profiling around how much increasing the block size helps and here is the data:

Block size	cost

4KB	        229s
8KB	         86s
3MB	          2s

So looks like 2 * block size OR 4 * block size can help quite a bit.

Hence, while we are exploring some options like reducing the size of the dump etc, I wanted to also check if increasing the block size to like 4 * 4Kb could be a solution because it will bring down the read times drastically based on our profiling.

Thanks

Abhinav



[Index of Archives]     [Linux DRI Users]     [Linux Intel Graphics]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux