Re: patch for "limited performance for disperse volumes"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Here's a quote from a paper titled: Non-blocking Writes to Files
https://www.usenix.org/conference/fast15/technical-sessions/presentation/campello

-----
Ordering of Page Updates.
Non-blocking writes may alter the sequence in which patches to
different pages get applied since the page fetches may complete
out-of-order. Non-blocking writes only replace writes that are
to memory that are not guaranteed to be reflected to persistent
storage in any particular sequence. Thus, ordering violations in
updates of in-memory pages are crash-safe.

Page Persistence and Syncs.
If an application would like explicit disk ordering for memory
page updates, it would execute a blocking flush operation
(e.g., fsync ) subsequent to each operation. The flush operation
causes the OS to force the fetch of any page indexed as NBW even
if it has not been allocated yet. The OS then obtains the page
lock, waits for the page fetch, and applies any outstanding
patches, before flushing the page and returning control to the
application. Ordering of disk writes are thus preserved with
non-blocking writes.
-----

Milind

On 02/10/2017 01:37 PM, Xavier Hernandez wrote:
Hi Raghavendra,

On 10/02/17 04:51, Raghavendra Gowdappa wrote:
+gluster-devel

----- Original Message -----
From: "Milind Changire" <mchangir@xxxxxxxxxx>
To: "Raghavendra Gowdappa" <rgowdapp@xxxxxxxxxx>
Cc: "rhs-zteam" <rhs-zteam@xxxxxxxxxx>
Sent: Thursday, February 9, 2017 11:00:18 PM
Subject: patch for "limited performance for disperse volumes"

My first comment was:
looks like patch for "limited performance for disperse volume" [1] is
going
to be helpful for all other types of volumes as well; but how do we
guarantee ordering for writes over the same fd for the same offset and
length in the file ?

then thinking over a bit and in case you missed my comment over IRC:
I was thinking about network multi-pathing and rpc requests(two writes)
being routed through different interfaces to gluster nodes which might
lead to a non-increasing transaction ID sequence and hence might lead
to incorrect final value if the older write is committed to the same
offset+length

then it dawned on me that for blocking operations the write() call
wont return until the data is safe on the disk across the network or
the intermediate translators have cached it appropriately to be
written behind.

so would the patch work for two non-blocking writes originating for the
same fd from the same thread for the same offset+length and being
routed over multi-pathing and write #2 getting routed quicker than
write #1 ?

To be honest I've not considered the case of asynchronous writes from
application till now. What is the ordering guarantee the
OS/filesystems provide for two async writes? For eg., if there are two
writes w1 and w2, when is w2 issued?
* After cbk of w1 is called or
* parallely just after async_write (w1) returns (cbk of w1 is not
invoked yet)?

What do POSIX or other standards (or expectation from OS) say about
ordering in case 2 above?

I'm not an expert on POSIX. But I've found this [1]:

    2.9.7 Thread Interactions with Regular File Operations

    All of the following functions shall be atomic with respect to
    each other in the effects specified in POSIX.1-2008 when they
    operate on regular files or symbolic links: [...] write [...]

    If two threads each call one of these functions, each call shall
    either see all of the specified effects of the other call, or none
    of them. The requirement on the close() function shall also apply
    whenever a file descriptor is successfully closed, however caused
    (for example, as a consequence of calling close(), calling dup2(),
    or of process termination).

Not sure if this also applies to write requests issued asynchronously
from the same thread, but this would be the worst case (if the OS
already orders it, we won't have any problem).

As I see it, this is already satisfied by EC because it doesn't allow
two concurrent writes to happen at the same time. They can be reordered
if the second one arrives before the first one, but they are executed
atomically as POSIX requires. Not sure if AFR also satisfies this
condition, but I think so.

From the point of view of EC it's irrelevant if the write comes from the
same thread or from different processes on different clients. They are
handled in the same way.

However a thing to be aware of (from the man page of write):

    [...] among the effects that should be atomic across threads (and
    processes) are updates of the file offset. However, on Linux before
    version 3.14, this was not the case: if two processes that share an
    open file description (see open(2)) perform a write() (or
    writev(2)) at the same time, then the I/O operations were not atomic
    with respect updating the file offset, with the result that the
    blocks of data output by the two processes might (incorrectly)
    overlap. This problem was fixed in Linux 3.14.

Xavi

[1]
http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09_07




[1] https://review.gluster.org/15036


just thinking aloud

--
Milind

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel



[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux