Re: pool distribution quality report script

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi David,

Mind sending me the output of "ceph pg dump -f json"?

thanks!
Mark

On 03/05/2015 12:52 PM, David Burley wrote:
Mark,

It worked for me earlier this morning but the new rev is throwing a
traceback:

$ ceph pg dump -f json | python ./readpgdump.py > pgdump_analysis.txt
dumped all in format json
Traceback (most recent call last):
   File "./readpgdump.py", line 294, in <module>
     parse_json(data)
   File "./readpgdump.py", line 263, in parse_json
     print_report(pool_counts, total_counts, "JSON")
   File "./readpgdump.py", line 119, in print_report
     print_data(data, pool_weights, total_weights)
   File "./readpgdump.py", line 161, in print_data
     print format_line("Efficiency score using optimal weights for pool
%s: %.1f%%" % (pool, efficiency_score(data[name],
weights['acting_totals'])))
   File "./readpgdump.py", line 71, in efficiency_score
     if weights and weights[osd]:
KeyError: 0

On Thu, Mar 5, 2015 at 1:46 PM, Mark Nelson <mnelson@xxxxxxxxxx
<mailto:mnelson@xxxxxxxxxx>> wrote:

    Hi Blair,

    I've updated the script and it now (theoretically) computes optimal
    crush weights based on both primary and secondary acting set OSDs.
    It also attempts to show you the efficiency of equal weights vs
    using weights optimized for different pools (or all pools).  This is
    done by looking at the way weights would be assigned and how they
    would affect the current pool distribution, then looking at the skew
    for the heaviest weighted OSD vs the average.

    Unfortunately the output has now become beastly and complex (granted
    this is a large cluster with many pools!).  I think the trick now is
    how to make the interface for this more manageable.  For instance
    perhaps it's not interesting to know how one pool's weights affect
    the efficiency of the acting primary OSDs for another pool.

    I've included sample output, but it's huge (15K lines long!)

    Mark


    On 03/05/2015 01:52 AM, Blair Bethwaite wrote:

        Hi Mark,

        Cool, that looks handy. Though it'd be even better if it could go a
        step further and recommend re-weighting values to balance things out
        (or increased PG counts where needed).

        Cheers,

        On 5 March 2015 at 15:11, Mark Nelson <mnelson@xxxxxxxxxx
        <mailto:mnelson@xxxxxxxxxx>> wrote:

            Hi All,

            Recently some folks showed interest in gathering pool
            distribution
            statistics and I remembered I wrote a script to do that a
            while back. It was
            broken due to a change in the ceph pg dump output format
            that was committed
            a while back, so I cleaned the script up, added detection of
            header fields,
            automatic json support, and also added in calculation of
            expected max and
            min PGs per OSD and std deviation.

            The script is available here:

            https://github.com/ceph/ceph-__tools/blob/master/cbt/tools/__readpgdump.py
            <https://github.com/ceph/ceph-tools/blob/master/cbt/tools/readpgdump.py>

            Some general comments:

            1) Expected numbers are derived by treating PGs and OSDs as a
            balls-in-buckets problem ala Raab & Steger:

            http://www14.in.tum.de/__personen/raab/publ/balls.pdf
            <http://www14.in.tum.de/personen/raab/publ/balls.pdf>

            2) You can invoke it either by passing it a file or stdout, ie:

            ceph pg dump -f json | ./readpgdump.py

            or

            ./readpgdump.py ~/pgdump.out


            3) Here's a snippet of some of some sample output from a 210
            OSD cluster.
            Does this output make sense to people?  Is it useful?

                [nhm@burnupiX tools]$ ./readpgdump.py ~/pgdump.out

                +-----------------------------__------------------------------__-----------------+
                | Detected input as plain
                |

                +-----------------------------__------------------------------__-----------------+


                +-----------------------------__------------------------------__-----------------+
                | Pool ID: 681
                |

                +-----------------------------__------------------------------__-----------------+
                | Participating OSDs: 210
                |
                | Participating PGs: 4096
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Primary Role (Acting)
                |
                | Expected PGs Per OSD: Min: 4, Max: 33, Mean: 19.5, Std
                Dev: 7.2
                |
                | Actual PGs Per OSD: Min: 7, Max: 43, Mean: 19.5, Std
                Dev: 6.5
                |
                | 5 Most Subscribed OSDs: 199(43), 175(36), 149(34),
                167(32), 20(31)
                |
                | 5 Least Subscribed OSDs: 121(7), 46(7), 70(8), 94(9),
                122(9)
                |
                | Avg Deviation from Most Subscribed OSD: 54.6%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Secondary Role (Acting)
                |
                | Expected PGs Per OSD: Min: 18, Max: 59, Mean: 39.0,
                Std Dev: 10.2
                |
                | Actual PGs Per OSD: Min: 17, Max: 61, Mean: 39.0, Std
                Dev: 9.7
                |
                | 5 Most Subscribed OSDs: 44(61), 14(60), 2(59),
                167(59), 164(57)
                |
                | 5 Least Subscribed OSDs: 35(17), 31(20), 37(20),
                145(20), 16(20)
                |
                | Avg Deviation from Most Subscribed OSD: 36.0%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in All Roles (Acting)
                |
                | Expected PGs Per OSD: Min: 32, Max: 83, Mean: 58.5,
                Std Dev: 12.5
                |
                | Actual PGs Per OSD: Min: 29, Max: 93, Mean: 58.5, Std
                Dev: 14.6
                |
                | 5 Most Subscribed OSDs: 199(93), 175(92), 44(92),
                167(91), 14(91)
                |
                | 5 Least Subscribed OSDs: 121(29), 35(30), 47(30),
                131(32), 145(32)
                |
                | Avg Deviation from Most Subscribed OSD: 37.1%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Primary Role (Up)
                |
                | Expected PGs Per OSD: Min: 4, Max: 33, Mean: 19.5, Std
                Dev: 7.2
                |
                | Actual PGs Per OSD: Min: 7, Max: 43, Mean: 19.5, Std
                Dev: 6.5
                |
                | 5 Most Subscribed OSDs: 199(43), 175(36), 149(34),
                167(32), 20(31)
                |
                | 5 Least Subscribed OSDs: 121(7), 46(7), 70(8), 94(9),
                122(9)
                |
                | Avg Deviation from Most Subscribed OSD: 54.6%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Secondary Role (Up)
                |
                | Expected PGs Per OSD: Min: 18, Max: 59, Mean: 39.0,
                Std Dev: 10.2
                |
                | Actual PGs Per OSD: Min: 17, Max: 61, Mean: 39.0, Std
                Dev: 9.7
                |
                | 5 Most Subscribed OSDs: 44(61), 14(60), 2(59),
                167(59), 164(57)
                |
                | 5 Least Subscribed OSDs: 35(17), 31(20), 37(20),
                145(20), 16(20)
                |
                | Avg Deviation from Most Subscribed OSD: 36.0%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in All Roles (Up)
                |
                | Expected PGs Per OSD: Min: 32, Max: 83, Mean: 58.5,
                Std Dev: 12.5
                |
                | Actual PGs Per OSD: Min: 29, Max: 93, Mean: 58.5, Std
                Dev: 14.6
                |
                | 5 Most Subscribed OSDs: 199(93), 175(92), 44(92),
                167(91), 14(91)
                |
                | 5 Least Subscribed OSDs: 121(29), 35(30), 47(30),
                131(32), 145(32)
                |
                | Avg Deviation from Most Subscribed OSD: 37.1%
                |

                +-----------------------------__------------------------------__-----------------+



            This is shown for all the pools, followed by the totals:


                +-----------------------------__------------------------------__-----------------+
                | Pool ID: Totals (All Pools)
                |

                +-----------------------------__------------------------------__-----------------+
                | Participating OSDs: 210
                |
                | Participating PGs: 131072
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Primary Role (Acting)
                |
                | Expected PGs Per OSD: Min: 542, Max: 705, Mean: 624.0,
                Std Dev: 40.8
                |
                | Actual PGs Per OSD: Min: 356, Max: 917, Mean: 624.0,
                Std Dev: 176.2
                |
                | 5 Most Subscribed OSDs: 167(917), 123(876), 23(873),
                194(869), 60(868)
                |
                | 5 Least Subscribed OSDs: 117(356), 155(358), 46(361),
                78(361), 70(362)
                |
                | Avg Deviation from Most Subscribed OSD: 32.0%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Secondary Role (Acting)
                |
                | Expected PGs Per OSD: Min: 1132, Max: 1363, Mean:
                1248.0, Std Dev: 57.8
                |
                | Actual PGs Per OSD: Min: 864, Max: 1554, Mean: 1248.0,
                Std Dev: 229.9
                |
                | 5 Most Subscribed OSDs: 182(1554), 142(1552), 4(1548),
                123(1544),
                100(1540)|
                | 5 Least Subscribed OSDs: 16(864), 55(885), 18(885),
                42(885), 85(891)
                |
                | Avg Deviation from Most Subscribed OSD: 19.7%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in All Roles (Acting)
                |
                | Expected PGs Per OSD: Min: 1730, Max: 2013, Mean:
                1872.0, Std Dev: 70.7
                |
                | Actual PGs Per OSD: Min: 1252, Max: 2420, Mean:
                1872.0, Std Dev: 402.0
                |
                | 5 Most Subscribed OSDs: 123(2420), 142(2403),
                195(2390), 100(2389),
                182(2380)|
                | 5 Least Subscribed OSDs: 55(1252), 42(1268), 16(1271),
                85(1271),
                155(1275) |
                | Avg Deviation from Most Subscribed OSD: 22.6%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Primary Role (Up)
                |
                | Expected PGs Per OSD: Min: 542, Max: 705, Mean: 624.0,
                Std Dev: 40.8
                |
                | Actual PGs Per OSD: Min: 356, Max: 917, Mean: 624.0,
                Std Dev: 176.2
                |
                | 5 Most Subscribed OSDs: 167(917), 123(876), 23(873),
                194(869), 60(868)
                |
                | 5 Least Subscribed OSDs: 117(356), 155(358), 46(361),
                78(361), 70(362)
                |
                | Avg Deviation from Most Subscribed OSD: 32.0%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in Secondary Role (Up)
                |
                | Expected PGs Per OSD: Min: 1132, Max: 1363, Mean:
                1248.0, Std Dev: 57.8
                |
                | Actual PGs Per OSD: Min: 864, Max: 1554, Mean: 1248.0,
                Std Dev: 229.9
                |
                | 5 Most Subscribed OSDs: 182(1554), 142(1552), 4(1548),
                123(1544),
                100(1540)|
                | 5 Least Subscribed OSDs: 16(864), 55(885), 18(885),
                42(885), 85(891)
                |
                | Avg Deviation from Most Subscribed OSD: 19.7%
                |

                +-----------------------------__------------------------------__-----------------+
                | OSDs in All Roles (Up)
                |
                | Expected PGs Per OSD: Min: 1730, Max: 2013, Mean:
                1872.0, Std Dev: 70.7
                |
                | Actual PGs Per OSD: Min: 1252, Max: 2420, Mean:
                1872.0, Std Dev: 402.0
                |
                | 5 Most Subscribed OSDs: 123(2420), 142(2403),
                195(2390), 100(2389),
                182(2380)|
                | 5 Least Subscribed OSDs: 55(1252), 42(1268), 16(1271),
                85(1271),
                155(1275) |
                | Avg Deviation from Most Subscribed OSD: 22.6%
                |

                +-----------------------------__------------------------------__-----------------+



            --
            To unsubscribe from this list: send the line "unsubscribe
            ceph-devel" in
            the body of a message to majordomo@xxxxxxxxxxxxxxx
            <mailto:majordomo@xxxxxxxxxxxxxxx>
            More majordomo info at
            http://vger.kernel.org/__majordomo-info.html
            <http://vger.kernel.org/majordomo-info.html>





    _______________________________________________
    ceph-users mailing list
    ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
David Burley
NOC Manager, Sr. Systems Programmer/Analyst
Slashdot Media

e: david@xxxxxxxxxxxxxxxxx <mailto:david@xxxxxxxxxxxxxxxxx>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux