On Mon, Jul 9, 2018 at 5:28 PM, Benjamin Naber
<benjamin.naber@xxxxxxxxxxxxxx> wrote:
> Hi @all,
>
> Problem seems to be solved, afther downgrading from Kernel 4.17.2 to 3.10.0-862.
> Anyone other have issues with newer Kernels and osd nodes?
I'd suggest you pursue that with whoever supports the kernel
exhibiting the problem.
>
> kind regards
>
> Ben
>
>> Brad Hubbard <bhubbard@xxxxxxxxxx> hat am 5. Juli 2018 um 01:16 geschrieben:
>>
>>
>> On Wed, Jul 4, 2018 at 6:26 PM, Benjamin Naber <der-coder@xxxxxxxxxxxxxx> wrote:
>> > Hi @all,
>> >
>> > im currently in testing for setup an production environment based on the following OSD Nodes:
>> >
>> > CEPH Version: luminous 12.2.5
>> >
>> > 5x OSD Nodes with following specs:
>> >
>> > - 8 Core Intel Xeon 2,0 GHZ
>> >
>> > - 96GB Ram
>> >
>> > - 10x 1,92 TB Intel DC S4500 connectet via SATA
>> >
>> > - 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP for Backend Network.
>> >
>> > if i run some fio benchmark via a VM that ist running on a RBD Device on a KVM testing Host. the cluster always runs into slow request warning. Also the performance goes heavy down.
>> >
>> > If i dump the osd that stucks, i get the following output:
>> >
>> > {
>> > "ops": [
>> > {
>> > "description": "osd_op(client.141944.0:359346834 13.1da 13:5b8b7fd3:::rbd_data.170a3238e1f29.00000000000000be:head [write 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.475879",
>> > "age": 287.180328,
>> > "duration": 287.180355,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346834
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.475879",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.476935",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477547",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477578",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477614",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484679",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484681",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.485588",
>> > "event": "sub_op_commit_rec from 5"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346835 13.1da 13:5b8b7fd3:::rbd_data.170a3238e1f29.00000000000000be:head [write 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.477065",
>> > "age": 287.179143,
>> > "duration": 287.179221,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346835
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.477065",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478116",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478178",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478201",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478232",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484695",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484696",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.485621",
>> > "event": "sub_op_commit_rec from 5"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346440 13.11d 13:b8afbe4a:::rbd_data.170a3238e1f29.000000000000005c:head [write 0~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.091127",
>> > "age": 287.565080,
>> > "duration": 287.565196,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346440
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.091127",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.092764",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.092787",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093062",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093107",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.099496",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.099502",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.100772",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346441 13.11d 13:b8afbe4a:::rbd_data.170a3238e1f29.000000000000005c:head [write 1048576~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.092772",
>> > "age": 287.563436,
>> > "duration": 287.563588,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346441
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.092772",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093766",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093865",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093895",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.093937",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.099505",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.099506",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.100788",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346442 13.11d 13:b8afbe4a:::rbd_data.170a3238e1f29.000000000000005c:head [write 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.093773",
>> > "age": 287.562434,
>> > "duration": 287.562622,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346442
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.093773",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.095388",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.095403",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.095428",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.095466",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.103411",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.103413",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.106365",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346443 13.11d 13:b8afbe4a:::rbd_data.170a3238e1f29.000000000000005c:head [write 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.095395",
>> > "age": 287.560813,
>> > "duration": 287.561037,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346443
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.095395",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.097194",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.098851",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.098878",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.098914",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.103704",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.103706",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.107064",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346172 13.11d 13:b88a4b5e:::rbd_data.170a3238e1f29.000000000000001a:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:48.836391",
>> > "age": 287.819817,
>> > "duration": 287.820078,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346172
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:48.836391",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.837419",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.837438",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.837548",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.837605",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847008",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847013",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.862005",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346372 13.bc 13:3d131f8b:::rbd_data.170a3238e1f29.000000000000004b:head [write 0~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.023513",
>> > "age": 287.632694,
>> > "duration": 287.632990,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346372
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.023513",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.025442",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.025464",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.025796",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.025844",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033594",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033600",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.036086",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346173 13.11d 13:b88a4b5e:::rbd_data.170a3238e1f29.000000000000001a:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1048576~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:48.837425",
>> > "age": 287.818782,
>> > "duration": 287.819113,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346173
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:48.837425",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.838409",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.838427",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.838463",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.838505",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847328",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847330",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.862049",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346373 13.bc 13:3d131f8b:::rbd_data.170a3238e1f29.000000000000004b:head [write 1048576~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.025449",
>> > "age": 287.630759,
>> > "duration": 287.631126,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346373
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.025449",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027010",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027030",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027071",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027126",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033603",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033605",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.036143",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346174 13.11d 13:b88a4b5e:::rbd_data.170a3238e1f29.000000000000001a:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:48.838414",
>> > "age": 287.817793,
>> > "duration": 287.818196,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346174
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:48.838414",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.839187",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.839203",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.839229",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.839269",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847345",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847347",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.862060",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346374 13.bc 13:3d131f8b:::rbd_data.170a3238e1f29.000000000000004b:head [write 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.027016",
>> > "age": 287.629192,
>> > "duration": 287.629634,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346374
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.027016",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027972",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.027987",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028017",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028053",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033606",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.033609",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.036159",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346175 13.11d 13:b88a4b5e:::rbd_data.170a3238e1f29.000000000000001a:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:48.839192",
>> > "age": 287.817015,
>> > "duration": 287.817491,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346175
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:48.839192",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.840612",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.840626",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.840658",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.840693",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847361",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.847363",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:48.862089",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346375 13.bc 13:3d131f8b:::rbd_data.170a3238e1f29.000000000000004b:head [write 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.027978",
>> > "age": 287.628229,
>> > "duration": 287.628739,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346375
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.027978",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028846",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028863",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028889",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.028924",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.035948",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.035950",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.038542",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346792 13.bc 13:3d313592:::rbd_data.170a3238e1f29.00000000000000b4:head [write 0~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.423998",
>> > "age": 287.232209,
>> > "duration": 287.232753,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346792
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.423998",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.425331",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.425350",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.425655",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.425715",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434856",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434869",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.436410",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346793 13.bc 13:3d313592:::rbd_data.170a3238e1f29.00000000000000b4:head [write 1048576~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.425349",
>> > "age": 287.230859,
>> > "duration": 287.231436,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346793
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.425349",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.426349",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.426877",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427022",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427112",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434876",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434877",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.436661",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346794 13.bc 13:3d313592:::rbd_data.170a3238e1f29.00000000000000b4:head [write 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.426356",
>> > "age": 287.229852,
>> > "duration": 287.230461,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346794
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.426356",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427126",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427729",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427771",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.427812",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434878",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.434879",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.436686",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346795 13.bc 13:3d313592:::rbd_data.170a3238e1f29.00000000000000b4:head [write 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.427141",
>> > "age": 287.229066,
>> > "duration": 287.229715,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346795
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.427141",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.428212",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.431693",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.431730",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.431794",
>> > "event": "waiting for subops from 21,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.437279",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.437282",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.442448",
>> > "event": "sub_op_commit_rec from 21"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346832 13.1da 13:5b8b7fd3:::rbd_data.170a3238e1f29.00000000000000be:head [write 0~901120] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.473234",
>> > "age": 287.182974,
>> > "duration": 287.183657,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346832
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.473234",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.474116",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.474135",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.474444",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.474489",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478163",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478170",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.485418",
>> > "event": "sub_op_commit_rec from 5"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346833 13.1da 13:5b8b7fd3:::rbd_data.170a3238e1f29.00000000000000be:head [write 901120~1196032] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.474651",
>> > "age": 287.181557,
>> > "duration": 287.182274,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346833
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.474651",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.475829",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.475845",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.475881",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.475923",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484417",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484420",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.485539",
>> > "event": "sub_op_commit_rec from 5"
>> > }
>> > ]
>> > }
>> > }
>> > ],
>> > "num_ops": 20
>> > }
>> >
>> > can someone explain me how i can find the source of the Problem ?
>>
>> All of these are waiting for subops to complete on osd 26.
>>
>> >
>> > kind regards
>> >
>> > Ben
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Cheers,
>> Brad
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Cheers,
Brad
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
