I used writeback mode for cache pool :
ceph osd tier add sas ssd
ceph osd tier add sas ssd
ceph osd tier cache-mode ssd writeback
ceph osd tier set-overlay sas ssd
and i also set dirty ratio and full ratio:
ceph osd pool set ssd cache_target_full_ratio .8
the capacity of ssd cache pool is 4T.
I used fio to test performance:
fio -filename=/dev/rbd0 -direct=1 -iodepth 32 -thread -rw=randwrite -ioengine=libaio -bs=16M -size=2000G -group_reporting -name=mytest
at the begin, the performance is very good, but after half a hour, I find when the hot cache pool begin flushing dirty objects, the performance of rados is instability. from 87851 kB/s to 860 MB/s.
Do have any tunning parameters to get more stable performance?
Thanks.
2014-12-23 22:46:24.844730 mon.0 [INF] pgmap v24101: 6144 pgs: 6144 active+clean; 1246 GB data, 4012 GB used, 45109 GB / 49121 GB avail; 680 MB/s wr, 1007 op/s
2014-12-23 22:46:27.851431 mon.0 [INF] pgmap v24102: 6144 pgs: 6144 active+clean; 1246 GB data, 4012 GB used, 45109 GB / 49121 GB avail; 161 MB/s wr, 299 op/s
2014-12-23 22:46:28.883866 mon.0 [INF] pgmap v24103: 6144 pgs: 6144 active+clean; 1247 GB data, 4015 GB used, 45106 GB / 49121 GB avail; 308 MB/s wr, 1065 op/s
2014-12-23 22:46:29.885914 mon.0 [INF] pgmap v24104: 6144 pgs: 6144 active+clean; 1247 GB data, 4016 GB used, 45105 GB / 49121 GB avail; 701 MB/s wr, 1621 op/s
2014-12-23 22:46:32.842955 mon.0 [INF] pgmap v24105: 6144 pgs: 6144 active+clean; 1247 GB data, 4016 GB used, 45105 GB / 49121 GB avail; 116 MB/s wr, 160 op/s
2014-12-23 22:46:33.863964 mon.0 [INF] pgmap v24106: 6144 pgs: 6144 active+clean; 1248 GB data, 4021 GB used, 45100 GB / 49121 GB avail; 344 MB/s wr, 923 op/s
2014-12-23 22:46:34.861011 mon.0 [INF] pgmap v24107: 6144 pgs: 6144 active+clean; 1248 GB data, 4021 GB used, 45100 GB / 49121 GB avail; 706 MB/s wr, 1564 op/s
2014-12-23 22:46:38.176885 mon.0 [INF] pgmap v24108: 6144 pgs: 6144 active+clean; 1249 GB data, 4024 GB used, 45097 GB / 49121 GB avail; 222 MB/s wr, 938 op/s
2014-12-23 22:46:39.177233 mon.0 [INF] pgmap v24109: 6144 pgs: 6144 active+clean; 1250 GB data, 4026 GB used, 45095 GB / 49121 GB avail; 427 MB/s wr, 1292 op/s
2014-12-23 22:46:42.842279 mon.0 [INF] pgmap v24110: 6144 pgs: 6144 active+clean; 1250 GB data, 4026 GB used, 45095 GB / 49121 GB avail; 320 MB/s wr, 570 op/s
2014-12-23 22:46:43.872017 mon.0 [INF] pgmap v24111: 6144 pgs: 6144 active+clean; 1251 GB data, 4030 GB used, 45090 GB / 49121 GB avail; 405 MB/s wr, 992 op/s
2014-12-23 22:46:44.862873 mon.0 [INF] pgmap v24112: 6144 pgs: 6144 active+clean; 1251 GB data, 4030 GB used, 45090 GB / 49121 GB avail; 729 MB/s wr, 1755 op/s
2014-12-23 22:46:47.847813 mon.0 [INF] pgmap v24113: 6144 pgs: 6144 active+clean; 1251 GB data, 4031 GB used, 45090 GB / 49121 GB avail; 2053 kB/s wr, 135 op/s
2014-12-23 22:46:48.857285 mon.0 [INF] pgmap v24114: 6144 pgs: 6144 active+clean; 1252 GB data, 4033 GB used, 45087 GB / 49121 GB avail; 272 MB/s wr, 433 op/s
2014-12-23 22:46:49.871775 mon.0 [INF] pgmap v24115: 6144 pgs: 6144 active+clean; 1252 GB data, 4034 GB used, 45087 GB / 49121 GB avail; 535 MB/s wr, 586 op/s
2014-12-23 22:46:52.842098 mon.0 [INF] pgmap v24116: 6144 pgs: 6144 active+clean; 1252 GB data, 4033 GB used, 45088 GB / 49121 GB avail; 3074 kB/s wr, 113 op/s
2014-12-23 22:46:53.845398 mon.0 [INF] pgmap v24117: 6144 pgs: 6144 active+clean; 1254 GB data, 4037 GB used, 45084 GB / 49121 GB avail; 342 MB/s wr, 571 op/s
2014-12-23 22:46:57.844137 mon.0 [INF] pgmap v24118: 6144 pgs: 6144 active+clean; 1254 GB data, 4037 GB used, 45084 GB / 49121 GB avail; 302 MB/s wr, 577 op/s
2014-12-23 22:46:58.848028 mon.0 [INF] pgmap v24119: 6144 pgs: 6144 active+clean; 1255 GB data, 4039 GB used, 45082 GB / 49121 GB avail; 319 MB/s wr, 897 op/s
2014-12-23 22:47:02.844724 mon.0 [INF] pgmap v24120: 6144 pgs: 6144 active+clean; 1255 GB data, 4039 GB used, 45082 GB / 49121 GB avail; 327 MB/s wr, 856 op/s
2014-12-23 22:47:03.850795 mon.0 [INF] pgmap v24121: 6144 pgs: 6144 active+clean; 1256 GB data, 4043 GB used, 45078 GB / 49121 GB avail; 297 MB/s wr, 887 op/s
2014-12-23 22:47:08.169046 mon.0 [INF] pgmap v24122: 6144 pgs: 6144 active+clean; 1256 GB data, 4045 GB used, 45076 GB / 49121 GB avail; 318 MB/s wr, 830 op/s
2014-12-23 22:47:09.169302 mon.0 [INF] pgmap v24123: 6144 pgs: 6144 active+clean; 1257 GB data, 4046 GB used, 45075 GB / 49121 GB avail; 133 MB/s wr, 257 op/s
2014-12-23 22:47:12.844073 mon.0 [INF] pgmap v24124: 6144 pgs: 6144 active+clean; 1257 GB data, 4046 GB used, 45075 GB / 49121 GB avail; 65702 kB/s wr, 124 op/s
2014-12-23 22:47:13.845286 mon.0 [INF] pgmap v24125: 6144 pgs: 6144 active+clean; 1257 GB data, 4047 GB used, 45074 GB / 49121 GB avail; 142 MB/s wr, 284 op/s
2014-12-23 22:47:14.846753 mon.0 [INF] pgmap v24126: 6144 pgs: 6144 active+clean; 1257 GB data, 4047 GB used, 45074 GB / 49121 GB avail; 461 MB/s wr, 663 op/s
2014-12-23 22:47:18.170362 mon.0 [INF] pgmap v24127: 6144 pgs: 6144 active+clean; 1258 GB data, 4048 GB used, 45073 GB / 49121 GB avail; 127 MB/s wr, 119 op/s
2014-12-23 22:47:19.171722 mon.0 [INF] pgmap v24128: 6144 pgs: 6144 active+clean; 1258 GB data, 4050 GB used, 45071 GB / 49121 GB avail; 155 MB/s wr, 273 op/s
2014-12-23 22:47:20.175130 mon.0 [INF] pgmap v24129: 6144 pgs: 6144 active+clean; 1258 GB data, 4050 GB used, 45071 GB / 49121 GB avail; 213 MB/s wr, 372 op/s
2014-12-23 22:47:22.837606 mon.0 [INF] pgmap v24130: 6144 pgs: 6144 active+clean; 1258 GB data, 4051 GB used, 45070 GB / 49121 GB avail; 101 MB/s wr, 447 op/s
2014-12-23 22:47:23.843802 mon.0 [INF] pgmap v24131: 6144 pgs: 6144 active+clean; 1259 GB data, 4057 GB used, 45064 GB / 49121 GB avail; 463 MB/s wr, 1858 op/s
2014-12-23 22:47:24.840098 mon.0 [INF] pgmap v24132: 6144 pgs: 6144 active+clean; 1259 GB data, 4057 GB used, 45063 GB / 49121 GB avail; 765 MB/s wr, 2603 op/s
2014-12-23 22:47:27.847831 mon.0 [INF] pgmap v24133: 6144 pgs: 6144 active+clean; 1259 GB data, 4058 GB used, 45063 GB / 49121 GB avail; 87851 kB/s wr, 293 op/s
2014-12-23 22:47:28.853847 mon.0 [INF] pgmap v24134: 6144 pgs: 6144 active+clean; 1261 GB data, 4064 GB used, 45057 GB / 49121 GB avail; 453 MB/s wr, 1418 op/s
2014-12-23 22:47:29.850868 mon.0 [INF] pgmap v24135: 6144 pgs: 6144 active+clean; 1261 GB data, 4065 GB used, 45056 GB / 49121 GB avail; 860 MB/s wr, 2281 op/s
2014-12-23 22:47:32.836600 mon.0 [INF] pgmap v24136: 6144 pgs: 6144 active+clean; 1261 GB data, 4065 GB used, 45056 GB / 49121 GB avail; 81164 kB/s wr, 360 op/s
2014-12-23 22:47:33.839495 mon.0 [INF] pgmap v24137: 6144 pgs: 6144 active+clean; 1262 GB data, 4067 GB used, 45054 GB / 49121 GB avail; 383 MB/s wr, 1281 op/s
2014-12-23 22:47:34.840910 mon.0 [INF] pgmap v24138: 6144 pgs: 6144 active+clean; 1262 GB data, 4067 GB used, 45054 GB / 49121 GB avail; 698 MB/s wr, 1853 op/s
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
