Re: OSD write op out of order

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Dec 27, 2021 at 9:12 AM gyfelectric <gyfelectric@xxxxxxxxx> wrote:

>
> Hi all,
>
> Recently, the problem of OSD disorder has often appeared in my
> environment(14.2.5) and my Fuse Client borken
> due to "FAILED assert(ob->last_commit_tid < tid)”. My application can’t
> work normally now.
>
> The time series that triggered this problem is like this:
> note:
> a. my datapool is: EC 4+2
> b. osd(osd.x) of pg_1 is down
>
> Event Sequences:
> t1: op_1(write) send to OSD and send 5 shards to 5 osds. only return 4
> shards except primary osd because there is osd(osd.x) down.
> t2: many other operations have occurred in this pg and record in pg_log
> t3: op_n(write) send to OSD and send 5 shards to 5 osds. only return 4
> shards except primary osd because there is osd(osd.x) down.
> t4: the peer osd report osd.x timeout to monitor and osd.x is marked down
> t5: pg_1 start canceling and requeueing op_1, op_2 … op_n to osd op_wq
> t6: pg_1 start peering and op_1 is trimmed from pg_log and dup map in this
> process
>

Unless I’m misunderstanding, either you have more ops that haven’t been
committed+acked than the length of the pg log dup tracking, or else there’s
a bug here and it’s trimming farther than it should.

Can you clarify which case? Because if you’re sending more ops than the pg
log length, this is an expected failure and not one that’s feasible to
resolve. You just need to spend the money to have enough memory for longer
logs and dup detection.

-Greg

t7: pg_1 become active and start reprocessing the op_1, op_2 … op_n
> t8: op_1 is not found in pg_log and dup map, so redo it.
> t9: op_n is found in pg_log or dup map and be considered completed, so
> return osd reply to client directly with tid_op_n
> t10: op_1 complete and return to client with tid_op_1. client will break
> down due to "assert(ob->last_commit_tid < tid)”
>
> I found some relative issues in https://tracker.ceph.com/issues/23827
> <https://tracker.ceph.com/issues/23827.> which have some discussions
> about this problem.
> But i didn’t find an effective method to avoid this problem.
>
> I think the current mechanism to prevent non-idempotent op from being
> repeated is flawed, may be we should redesign it.
> How do you think about it? And if my idea is wrong, what should i do to
> avoid this problem?
>
> Any response is very grateful, thank you!
>
> gyfelectric
> gyfelectric@xxxxxxxxx
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=gyfelectric&uid=gyfelectric%40gmail.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22gyfelectric%40gmail.com%22%5D>
> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux