Answering the question myself, here are the contents of xattr for the object user.cephos.spill_out: 0000 30 00 0. user.ceph._: 0000 0F 08 05 01 00 00 04 03 41 00 00 00 00 00 00 00 ........A....... 0010 20 00 00 00 72 62 2E 30 2E 31 62 61 37 30 2E 32 ...rb.0.1ba70.2 0020 33 38 65 31 66 32 39 2E 30 30 30 30 30 30 30 31 38e1f29.00000001 0030 31 34 37 37 FE FF FF FF FF FF FF FF 28 25 31 8B 1477........(%1. 0040 00 00 00 00 00 00 00 00 00 00 00 00 00 06 03 1C ................ 0050 00 00 00 00 00 00 00 00 00 00 00 FF FF FF FF 00 ................ 0060 00 00 00 00 00 00 00 FF FF FF FF FF FF FF FF 00 ................ 0070 00 00 00 CB FF 11 00 00 00 00 00 70 08 00 00 8C ...........p.... 0080 B1 00 00 00 00 00 00 0C 04 00 00 02 02 15 00 00 ................ 0090 00 04 01 00 00 00 00 00 00 00 74 FE 28 00 00 00 ..........t.(... 00A0 00 00 00 00 00 00 00 00 10 00 00 00 00 00 5D 04 ..............]. 00B0 07 55 00 9C 84 39 02 02 15 00 00 00 00 00 00 00 .U...9.......... 00C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00E0 00 00 00 00 00 00 8C B1 00 00 00 00 00 00 00 00 ................ 00F0 00 00 00 00 00 00 00 34 00 00 00 5D 04 07 55 B8 .......4...]..U. 0100 E3 F1 3A 00 5B 0A A5 FF FF FF FF ..:.[...... user.ceph.snapset: 0000 02 02 19 00 00 00 00 00 00 00 00 00 00 00 01 00 ................ 0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ............... > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Nick Fisk > Sent: 30 June 2015 10:51 > To: 'Somnath Roy' > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Old vs New pool on same OSDs - Performance > Difference > > > -----Original Message----- > > From: Somnath Roy [mailto:Somnath.Roy@xxxxxxxxxxx] > > Sent: 29 June 2015 23:29 > > To: Nick Fisk > > Cc: ceph-users@xxxxxxxxxxxxxx > > Subject: RE: Old vs New pool on same OSDs - Performance > > Difference > > > > Nick, > > I think you are probably hitting the issue of crossing the xattr size > > limit that XFS can inline (255 bytes). In your case "_" xattr size is 267 bytes. > > Sage talked about that in one of his earlier mails..You can try to > > apply the following patch (not backported to hammer yet) and see if it > > is improving anything. > > > > c6cdb4081e366f471b372102905a1192910ab2da > > Ok I will see if this is something I can apply, I haven't really got the facility to > rebuild Ceph at the moment, so I will look into getting a VM set up to build > some debs. > > > > > But, I am not sure why this will impact one pool but not other ! > > In the slow pool do you have lot of snaps/clones/watchers ? > > I don't think this is related to particular pools, I think the problem relates to > RBD's that haven't been written to in a while. Overwriting the RBD's contents > with a fio run seems to restore performance. > > No, that’s the weird thing. I have a few pools with maybe 8-10 RBD's on > them, nothing special is being done. Can there be a case where xattr's can > grow larger than 255 bytes if I'm not using any special features? Is there a > way to "dump" the xattrs to see why they are taking up so much space? > > > > > > > Thanks & Regards > > Somnath > > > > > > -----Original Message----- > > From: Nick Fisk [mailto:nick@xxxxxxxxxx] > > Sent: Monday, June 29, 2015 3:05 PM > > To: Somnath Roy > > Cc: ceph-users@xxxxxxxxxxxxxx > > Subject: RE: Old vs New pool on same OSDs - Performance > > Difference > > > > Sorry, forgot to enable that. Here is another capture with it on and I > > think you are spot on as I can see a 100ms delay doing the getattr > > request. Any ideas how to debug further? Thanks for the help by the way, > really appreciated. > > > > 2015-06-29 22:48:50.851645 7fd8a2a1e700 15 osd.1 26349 enqueue_op > > 0x522bf00 prio 63 cost 0 latency 0.000288 osd_op(client.2796502.0:136 > > rb.0.1ba70.238e1f29.000000011477 [read 65536~65536] 0.8b312528 > > ack+read+known_if_redirected e26349) v5 > > 2015-06-29 22:48:50.851735 7fd8b026e700 10 osd.1 26349 dequeue_op > > 0x522bf00 prio 63 cost 0 latency 0.000378 osd_op(client.2796502.0:136 > > rb.0.1ba70.238e1f29.000000011477 [read 65536~65536] 0.8b312528 > > ack+read+known_if_redirected e26349) v5 pg pg[0.128( v 26335'8141858 > > (26331'8138777,26335'8141858] local-les=26276 n=5243 ec=1 les/c > > 26276/26305 26264/26272/26272) [1,21,31] r=0 lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > 2015-06-29 22:48:50.852076 7fd8b026e700 20 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > op_has_sufficient_caps pool=0 (rbd ) owner=0 need_read_cap=1 > > need_write_cap=0 need_class_read_cap=0 need_class_write_cap=0 -> > yes > > 2015-06-29 22:48:50.852252 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > handle_message: 0x522bf00 > > 2015-06-29 22:48:50.852471 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > do_op osd_op(client.2796502.0:136 rb.0.1ba70.238e1f29.000000011477 > > [read 65536~65536] 0.8b312528 ack+read+known_if_redirected e26349) v5 > > may_read -> read-ordered flags ack+read+known_if_redirected > > 2015-06-29 22:48:50.852960 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > get_object_context: obc NOT found in cache: > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 2015-06-29 22:48:50.853016 7fd8b026e700 15 > > filestore(/var/lib/ceph/osd/ceph-1) getattr > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 '_' > > 2015-06-29 22:48:50.953748 7fd8b026e700 10 > > filestore(/var/lib/ceph/osd/ceph-1) getattr > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 '_' = 267 > > 2015-06-29 22:48:50.953951 7fd8b026e700 15 > > filestore(/var/lib/ceph/osd/ceph-1) getattr > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > 'snapset' > > 2015-06-29 22:48:50.954148 7fd8b026e700 10 > > filestore(/var/lib/ceph/osd/ceph-1) getattr > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > 'snapset' > > = 31 > > 2015-06-29 22:48:50.954379 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > populate_obc_watchers > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 2015-06-29 22:48:50.954557 7fd8b026e700 20 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > ReplicatedPG::check_blacklisted_obc_watchers for obc > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 2015-06-29 22:48:50.954581 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > get_object_context: creating obc from disk: 0x5ee4a3c0 > > 2015-06-29 22:48:50.954597 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > get_object_context: 0x5ee4a3c0 > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 rwstate(none n=0 > > w=0) oi: > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0(2160'1179595 > > osd.1.0:2686580 wrlock_by=unknown.0.0:0 dirty|data_digest|omap_digest > > s > > 1048576 uv 45452 dd a50a5b00 od ffffffff) ssc: 0xee72780 snapset: > > 0=[]:[]+head > > 2015-06-29 22:48:50.954735 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > find_object_context 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > @head > > oi=8b312528/rb.0.1ba70.238e1f29.000000011477/head//0(2160'1179595 > > osd.1.0:2686580 wrlock_by=unknown.0.0:0 dirty|data_digest|omap_digest > > s > > 1048576 uv 45452 dd a50a5b00 od ffffffff) > > 2015-06-29 22:48:50.955142 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > execute_ctx 0x684fd000 > > 2015-06-29 22:48:50.955310 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > do_op 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 [read > > 65536~65536] ov 2160'1179595 > > 2015-06-29 22:48:50.955371 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > taking ondisk_read_lock > > 2015-06-29 22:48:50.955475 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > do_osd_op 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 [read > > 65536~65536] > > 2015-06-29 22:48:50.955493 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > do_osd_op read 65536~65536 > > 2015-06-29 22:48:50.955511 7fd8b026e700 15 > > filestore(/var/lib/ceph/osd/ceph-1) read > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 65536~65536 > > 2015-06-29 22:48:50.965854 7fd8b026e700 10 > > filestore(/var/lib/ceph/osd/ceph-1) FileStore::read > > 0.128_head/8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 65536~65536/65536 > > 2015-06-29 22:48:50.965879 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > read got 65536 / 65536 bytes from obj > > 8b312528/rb.0.1ba70.238e1f29.000000011477/head//0 > > 2015-06-29 22:48:50.966149 7fd8b026e700 10 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > dropping ondisk_read_lock > > 2015-06-29 22:48:50.966285 7fd8b026e700 15 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > do_osd_op_effects client.2796502 con 0x514a5600 > > 2015-06-29 22:48:50.966331 7fd8b026e700 15 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > log_op_stats osd_op(client.2796502.0:136 > > rb.0.1ba70.238e1f29.000000011477 [read 65536~65536] 0.8b312528 > > ack+read+known_if_redirected e26349) v5 inb 0 outb 65536 rlat 0.000000 > > lat 0.114972 > > 2015-06-29 22:48:50.966769 7fd8b026e700 15 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > publish_stats_to_osd 26349:8019961 > > 2015-06-29 22:48:50.966961 7fd8b026e700 15 osd.1 pg_epoch: 26349 > > pg[0.128( v 26335'8141858 (26331'8138777,26335'8141858] > > local-les=26276 > > n=5243 ec=1 les/c 26276/26305 26264/26272/26272) [1,21,31] r=0 > > lpr=26272 > > crt=26335'8141855 lcod 26335'8141857 mlcod 26335'8141857 active+clean] > > requeue_ops > > 2015-06-29 22:48:50.967127 7fd8b026e700 10 osd.1 26349 dequeue_op > > 0x522bf00 finish > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |