Well I think I got it! The issue was with pools that were created but then their OSD were pulled from under them via crush rools (I created SSD and regular disk pools and move the OSD into these). After I deleted these pools, all the bad PGs dissipated, which made perfect sense, since these were referencing non-existent OSDs.
On Sun, Jun 7, 2015 at 2:00 PM, Marek Dohojda <mdohojda@xxxxxxxxxxxxxxxxxxx> wrote:
Unfortunately nothing. It done its thing, re-balanced it, and left with same thing in the end. BTW Thank you very much for the time and suggestion, I really appreciate it.ceph health detailHEALTH_WARN 65 pgs stale; 22 pgs stuck inactive; 65 pgs stuck stale; 22 pgs stuck unclean; too many PGs per OSD (456 > max 300)pg 0.21 is stuck inactive since forever, current state creating, last acting []pg 0.7 is stuck inactive since forever, current state creating, last acting []pg 5.2 is stuck inactive since forever, current state creating, last acting []pg 1.7 is stuck inactive since forever, current state creating, last acting []pg 0.34 is stuck inactive since forever, current state creating, last acting []pg 0.33 is stuck inactive since forever, current state creating, last acting []pg 5.1 is stuck inactive since forever, current state creating, last acting []pg 0.1b is stuck inactive since forever, current state creating, last acting []pg 0.32 is stuck inactive since forever, current state creating, last acting []pg 1.2 is stuck inactive since forever, current state creating, last acting []pg 0.31 is stuck inactive since forever, current state creating, last acting []pg 2.0 is stuck inactive since forever, current state creating, last acting []pg 5.7 is stuck inactive since forever, current state creating, last acting []pg 1.0 is stuck inactive since forever, current state creating, last acting []pg 2.2 is stuck inactive since forever, current state creating, last acting []pg 0.16 is stuck inactive since forever, current state creating, last acting []pg 0.15 is stuck inactive since forever, current state creating, last acting []pg 0.2b is stuck inactive since forever, current state creating, last acting []pg 0.3f is stuck inactive since forever, current state creating, last acting []pg 0.27 is stuck inactive since forever, current state creating, last acting []pg 0.3c is stuck inactive since forever, current state creating, last acting []pg 0.3a is stuck inactive since forever, current state creating, last acting []pg 0.21 is stuck unclean since forever, current state creating, last acting []pg 0.7 is stuck unclean since forever, current state creating, last acting []pg 5.2 is stuck unclean since forever, current state creating, last acting []pg 1.7 is stuck unclean since forever, current state creating, last acting []pg 0.34 is stuck unclean since forever, current state creating, last acting []pg 0.33 is stuck unclean since forever, current state creating, last acting []pg 5.1 is stuck unclean since forever, current state creating, last acting []pg 0.1b is stuck unclean since forever, current state creating, last acting []pg 0.32 is stuck unclean since forever, current state creating, last acting []pg 1.2 is stuck unclean since forever, current state creating, last acting []pg 0.31 is stuck unclean since forever, current state creating, last acting []pg 2.0 is stuck unclean since forever, current state creating, last acting []pg 5.7 is stuck unclean since forever, current state creating, last acting []pg 1.0 is stuck unclean since forever, current state creating, last acting []pg 2.2 is stuck unclean since forever, current state creating, last acting []pg 0.16 is stuck unclean since forever, current state creating, last acting []pg 0.15 is stuck unclean since forever, current state creating, last acting []pg 0.2b is stuck unclean since forever, current state creating, last acting []pg 0.3f is stuck unclean since forever, current state creating, last acting []pg 0.27 is stuck unclean since forever, current state creating, last acting []pg 0.3c is stuck unclean since forever, current state creating, last acting []pg 0.3a is stuck unclean since forever, current state creating, last acting []pg 0.22 is stuck stale for 321498.474316, current state stale+active+clean, last acting [5,1,6]pg 0.1f is stuck stale for 322077.806428, current state stale+active+clean, last acting [2,0,4]pg 0.1e is stuck stale for 321075.078151, current state stale+active+clean, last acting [6,5,4]pg 0.1d is stuck stale for 1473417.920791, current state stale+active+clean, last acting [4,5,2]pg 0.19 is stuck stale for 322454.627532, current state stale+active+clean, last acting [0,2,6]pg 0.18 is stuck stale for 322077.806431, current state stale+active+clean, last acting [2,0,3]pg 0.17 is stuck stale for 1473417.920792, current state stale+active+clean, last acting [4,3,6]pg 0.14 is stuck stale for 1473417.920791, current state stale+active+clean, last acting [4,3,2]pg 0.13 is stuck stale for 321075.078158, current state stale+active+clean, last acting [6,4,0]pg 0.12 is stuck stale for 321075.078159, current state stale+active+clean, last acting [6,4,5]pg 0.10 is stuck stale for 321075.078159, current state stale+active+clean, last acting [6,0,2]pg 0.f is stuck stale for 321498.474329, current state stale+active+clean, last acting [5,6,3]pg 0.e is stuck stale for 321075.078160, current state stale+active+clean, last acting [6,4,1]pg 0.d is stuck stale for 322077.806441, current state stale+active+clean, last acting [2,5,3]pg 0.b is stuck stale for 322454.627537, current state stale+active+clean, last acting [0,1,4]pg 0.a is stuck stale for 322077.806442, current state stale+active+clean, last acting [2,6,1]pg 0.9 is stuck stale for 321075.078166, current state stale+active+clean, last acting [6,0,3]pg 1.6 is stuck stale for 321498.474328, current state stale+active+clean, last acting [5,3,2]pg 2.5 is stuck stale for 321498.474332, current state stale+active+clean, last acting [5,2,1]pg 4.3 is stuck stale for 321075.078170, current state stale+active+clean, last acting [6,5,3]pg 0.6 is stuck stale for 322454.627543, current state stale+active+clean, last acting [0,6,5]pg 3.5 is stuck stale for 1473417.920811, current state stale+active+clean, last acting [4,1,2]pg 4.2 is stuck stale for 1473417.920815, current state stale+active+clean, last acting [4,5,2]pg 0.5 is stuck stale for 321075.078175, current state stale+active+clean, last acting [6,4,0]pg 1.4 is stuck stale for 1473417.920812, current state stale+active+clean, last acting [4,5,6]pg 2.7 is stuck stale for 321075.078181, current state stale+active+clean, last acting [6,0,2]pg 3.6 is stuck stale for 1473417.920818, current state stale+active+clean, last acting [4,2,6]pg 4.1 is stuck stale for 321075.078187, current state stale+active+clean, last acting [6,3,2]pg 5.0 is stuck stale for 1473417.920825, current state stale+active+clean, last acting [4,0,3]pg 4.0 is stuck stale for 322077.806472, current state stale+active+clean, last acting [2,4,6]pg 3.7 is stuck stale for 1473417.920821, current state stale+active+clean, last acting [4,5,6]pg 2.6 is stuck stale for 322077.806475, current state stale+active+clean, last acting [2,5,1]pg 1.5 is stuck stale for 321498.474356, current state stale+active+clean, last acting [5,2,3]pg 4.7 is stuck stale for 321075.078195, current state stale+active+clean, last acting [6,4,1]pg 3.0 is stuck stale for 321498.474354, current state stale+active+clean, last acting [5,2,4]pg 2.1 is stuck stale for 322454.627571, current state stale+active+clean, last acting [0,6,1]pg 4.6 is stuck stale for 322454.627569, current state stale+active+clean, last acting [0,1,2]pg 3.1 is stuck stale for 1473417.920834, current state stale+active+clean, last acting [4,0,5]pg 1.3 is stuck stale for 1473417.920835, current state stale+active+clean, last acting [4,2,0]pg 0.2 is stuck stale for 321075.078206, current state stale+active+clean, last acting [6,1,5]pg 4.5 is stuck stale for 321075.078207, current state stale+active+clean, last acting [6,4,3]pg 3.2 is stuck stale for 322454.627577, current state stale+active+clean, last acting [0,5,3]pg 2.3 is stuck stale for 321075.078210, current state stale+active+clean, last acting [6,2,3]pg 5.4 is stuck stale for 321075.078211, current state stale+active+clean, last acting [6,1,2]pg 3.3 is stuck stale for 322077.806499, current state stale+active+clean, last acting [2,4,1]pg 1.1 is stuck stale for 321498.474372, current state stale+active+clean, last acting [5,2,6]pg 0.0 is stuck stale for 321498.474372, current state stale+active+clean, last acting [5,4,3]pg 5.5 is stuck stale for 1473417.920850, current state stale+active+clean, last acting [4,6,3]pg 0.3e is stuck stale for 322077.806554, current state stale+active+clean, last acting [2,6,5]pg 0.3d is stuck stale for 1473417.920921, current state stale+active+clean, last acting [4,2,6]pg 0.3b is stuck stale for 322454.627651, current state stale+active+clean, last acting [0,2,4]pg 0.39 is stuck stale for 321498.474468, current state stale+active+clean, last acting [5,4,3]pg 0.37 is stuck stale for 1473417.920922, current state stale+active+clean, last acting [4,6,5]pg 0.30 is stuck stale for 321075.078279, current state stale+active+clean, last acting [6,5,1]pg 0.2f is stuck stale for 321075.078280, current state stale+active+clean, last acting [6,2,4]pg 0.2e is stuck stale for 322454.627655, current state stale+active+clean, last acting [0,1,3]pg 0.2d is stuck stale for 321075.078282, current state stale+active+clean, last acting [6,4,1]pg 0.2c is stuck stale for 322077.806564, current state stale+active+clean, last acting [2,0,3]pg 0.2a is stuck stale for 321498.474465, current state stale+active+clean, last acting [5,1,2]pg 0.29 is stuck stale for 1473417.920927, current state stale+active+clean, last acting [4,6,0]pg 0.28 is stuck stale for 322454.627664, current state stale+active+clean, last acting [0,1,2]pg 0.26 is stuck stale for 321498.474467, current state stale+active+clean, last acting [5,1,3]pg 0.25 is stuck stale for 321498.474468, current state stale+active+clean, last acting [5,1,2]pg 0.24 is stuck stale for 1473417.920931, current state stale+active+clean, last acting [4,2,3]pg 0.23 is stuck stale for 321498.474467, current state stale+active+clean, last acting [5,2,4]too many PGs per OSD (456 > max 300)On Sun, Jun 7, 2015 at 11:12 AM, Alex Muntada <alexm@xxxxxxxxx> wrote:You can try moving osd.5 out and see what happens next.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
