Re: Understanding filesystem size

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

First of all, thank you so much again for the time you spend in trying to help me, it's much appreciated. 

Then:
- here's the dump of the CRUSH rules:

# ceph osd crush rule dump
[
    {
        "rule_id": 0,
        "rule_name": "replicated_rule",
        "type": 1,
        "steps": [
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "host"
            },
            {
                "op": "emit"
            }
        ]
    },
    {
        "rule_id": 1,
        "rule_name": "wizard_data",
        "type": 3,
        "steps": [
            {
                "op": "set_chooseleaf_tries",
                "num": 5
            },
            {
                "op": "set_choose_tries",
                "num": 250
            },
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "chooseleaf_indep",
                "num": 0,
                "type": "host"
            },
            {
                "op": "emit"
            }
        ]
    }
]


- Running `ceph osd metadata | grep min_alloc_size` shows:

   "bluestore_min_alloc_size": "4096"

for all the OSDs
- About PG count: currently I have two OSDs destroyed waiting for disk upgrade; anyway this doesn't change the overall picture. That said, here's the info you requested: 

# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 4 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 53 MiB 3.4 GiB 1.2 TiB 32.22 0.99 24 up 11 hdd 1.81940 1.00000 1.8 TiB 624 GiB 622 GiB 58 MiB 2.1 GiB 1.2 TiB 33.50 1.03 25 up 18 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 80 MiB 3.4 GiB 1.2 TiB 36.36 1.12 27 up 26 hdd 1.81940 1.00000 1.8 TiB 625 GiB 623 GiB 60 MiB 2.5 GiB 1.2 TiB 33.55 1.03 24 up 32 hdd 1.81940 1.00000 1.8 TiB 625 GiB 623 GiB 2 KiB 1.7 GiB 1.2 TiB 33.53 1.03 24 up 41 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 9 KiB 8.6 GiB 4.8 TiB 33.51 1.03 97 up 48 hdd 1.81940 1.00000 1.8 TiB 625 GiB 622 GiB 59 MiB 2.8 GiB 1.2 TiB 33.52 1.03 25 up 55 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 52 MiB 7.8 GiB 4.8 TiB 33.51 1.03 97 up 0 hdd 1.81940 1.00000 1.8 TiB 626 GiB 622 GiB 2 KiB 3.6 GiB 1.2 TiB 33.59 1.03 24 up 8 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 59 MiB 6.0 GiB 4.8 TiB 33.49 1.03 97 up 15 hdd 1.81940 1.00000 1.8 TiB 572 GiB 569 GiB 16 KiB 3.4 GiB 1.3 TiB 30.71 0.94 22 up 22 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 56 MiB 2.1 GiB 1.2 TiB 36.29 1.12 27 up 29 hdd 1.81940 1.00000 1.8 TiB 573 GiB 570 GiB 59 MiB 2.3 GiB 1.3 TiB 30.74 0.95 23 up 34 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 113 MiB 7.3 GiB 4.9 TiB 32.15 0.99 94 up 43 hdd 1.81940 1.00000 1.8 TiB 572 GiB 570 GiB 15 KiB 2.7 GiB 1.3 TiB 30.73 0.95 22 up 49 hdd 1.81940 1.00000 1.8 TiB 547 GiB 544 GiB 4 KiB 2.9 GiB 1.3 TiB 29.38 0.90 21 up 3 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 7 KiB 3.8 GiB 2.5 TiB 30.73 0.95 44 up 14 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 51 MiB 4.5 GiB 2.4 TiB 34.92 1.07 51 up 27 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 15 KiB 3.1 GiB 2.4 TiB 34.91 1.07 50 up 37 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 51 MiB 3.3 GiB 2.5 TiB 29.99 0.92 44 up 50 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 22 KiB 3.7 GiB 2.4 TiB 34.88 1.07 50 up 59 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 58 MiB 3.7 GiB 2.5 TiB 30.02 0.92 44 up 64 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 28 KiB 3.9 GiB 2.5 TiB 30.02 0.92 43 up 69 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 53 MiB 3.4 GiB 2.4 TiB 34.19 1.05 50 up 2 hdd 3.63869 1.00000 3.6 TiB 1.3 TiB 1.3 TiB 59 MiB 4.2 GiB 2.4 TiB 34.90 1.07 51 up 21 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 56 MiB 3.5 GiB 2.5 TiB 30.01 0.92 44 up 39 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 49 MiB 4.3 GiB 2.4 TiB 34.20 1.05 50 up 57 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 15 MiB 4.2 GiB 2.4 TiB 34.26 1.05 49 up 66 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 148 KiB 3.5 GiB 2.5 TiB 31.40 0.97 45 up 72 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 50 MiB 3.9 GiB 2.5 TiB 30.75 0.95 45 up 76 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 19 KiB 4.1 GiB 2.5 TiB 30.05 0.92 43 up 79 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 47 KiB 3.3 GiB 2.5 TiB 32.10 0.99 46 up 88 hdd 1.81897 1.00000 1.8 TiB 650 GiB 648 GiB 54 KiB 2.8 GiB 1.2 TiB 34.92 1.07 25 up 89 hdd 1.81898 1.00000 1.8 TiB 573 GiB 570 GiB 54 MiB 2.4 GiB 1.3 TiB 30.76 0.95 23 up 90 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 19 KiB 1.8 GiB 1.2 TiB 34.91 1.07 25 up 91 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 24 KiB 2.3 GiB 1.2 TiB 34.92 1.07 25 up 92 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 11 KiB 1.6 GiB 1.2 TiB 34.90 1.07 25 up 93 hdd 1.81898 1.00000 1.8 TiB 548 GiB 545 GiB 129 MiB 2.8 GiB 1.3 TiB 29.42 0.91 21 up 94 hdd 1.81897 1.00000 1.8 TiB 548 GiB 545 GiB 29 MiB 3.2 GiB 1.3 TiB 29.43 0.91 21 up 95 hdd 1.81898 1.00000 1.8 TiB 677 GiB 674 GiB 7 KiB 2.6 GiB 1.2 TiB 36.33 1.12 26 up 96 hdd 1.81898 1.00000 1.8 TiB 573 GiB 570 GiB 1.8 MiB 2.8 GiB 1.3 TiB 30.75 0.95 22 up 97 hdd 1.81898 1.00000 1.8 TiB 546 GiB 544 GiB 20 KiB 1.9 GiB 1.3 TiB 29.32 0.90 21 up 98 hdd 1.81898 1.00000 1.8 TiB 547 GiB 545 GiB 24 KiB 1.9 GiB 1.3 TiB 29.34 0.90 21 up 99 hdd 1.81898 1.00000 1.8 TiB 677 GiB 674 GiB 64 MiB 2.4 GiB 1.2 TiB 36.33 1.12 27 up 100 hdd 1.81898 1.00000 1.8 TiB 650 GiB 648 GiB 13 KiB 1.7 GiB 1.2 TiB 34.90 1.07 25 up 101 hdd 1.81898 1.00000 1.8 TiB 547 GiB 545 GiB 104 MiB 1.9 GiB 1.3 TiB 29.38 0.90 23 up 102 hdd 1.81898 1.00000 1.8 TiB 599 GiB 596 GiB 315 KiB 2.0 GiB 1.2 TiB 32.13 0.99 23 up 103 hdd 1.81897 1.00000 1.8 TiB 599 GiB 596 GiB 21 KiB 2.6 GiB 1.2 TiB 32.14 0.99 23 up 80 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 9 KiB 3.0 GiB 1.2 TiB 36.32 1.12 26 up 106 hdd 1.81940 1.00000 1.8 TiB 677 GiB 674 GiB 2 KiB 3.0 GiB 1.2 TiB 36.31 1.12 26 up 107 hdd 1.81940 1.00000 1.8 TiB 547 GiB 545 GiB 27 KiB 2.5 GiB 1.3 TiB 29.38 0.90 21 up 108 hdd 1.81940 1.00000 1.8 TiB 625 GiB 622 GiB 109 MiB 3.0 GiB 1.2 TiB 33.57 1.03 25 up 109 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 4 KiB 1.8 GiB 1.2 TiB 36.27 1.12 26 up 110 hdd 1.81940 1.00000 1.8 TiB 572 GiB 571 GiB 5 KiB 1.6 GiB 1.3 TiB 30.72 0.95 22 up 111 hdd 1.81940 1.00000 1.8 TiB 651 GiB 648 GiB 59 MiB 2.7 GiB 1.2 TiB 34.93 1.07 26 up 112 hdd 1.81940 1.00000 1.8 TiB 598 GiB 595 GiB 59 MiB 2.3 GiB 1.2 TiB 32.08 0.99 24 up 113 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 4 KiB 2.4 GiB 1.2 TiB 36.29 1.12 26 up 114 hdd 1.81940 1.00000 1.8 TiB 624 GiB 622 GiB 2 KiB 1.8 GiB 1.2 TiB 33.49 1.03 24 up 115 hdd 1.81940 1.00000 1.8 TiB 521 GiB 519 GiB 4 KiB 2.3 GiB 1.3 TiB 27.96 0.86 20 up 116 hdd 1.81940 1.00000 1.8 TiB 650 GiB 647 GiB 2 KiB 2.9 GiB 1.2 TiB 34.91 1.07 25 up 62 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 103 MiB 3.7 GiB 2.4 TiB 34.17 1.05 51 up 117 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 60 MiB 4.1 GiB 2.4 TiB 34.20 1.05 50 up 118 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 139 KiB 4.3 GiB 2.5 TiB 31.41 0.97 45 up 119 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 52 MiB 4.2 GiB 2.4 TiB 34.23 1.05 50 up 120 hdd 3.63869 1.00000 3.6 TiB 1.2 TiB 1.2 TiB 2 KiB 4.2 GiB 2.4 TiB 32.83 1.01 47 up 121 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 2 KiB 3.8 GiB 2.6 TiB 29.34 0.90 42 up 122 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 10 KiB 3.9 GiB 2.5 TiB 31.41 0.97 45 up 123 hdd 3.63869 1.00000 3.6 TiB 1.1 TiB 1.1 TiB 85 MiB 3.9 GiB 2.5 TiB 30.71 0.94 45 up 7 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 10 KiB 5.5 GiB 5.0 TiB 31.02 0.95 90 up 16 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 115 MiB 6.0 GiB 5.0 TiB 31.05 0.96 91 up 23 hdd 1.81940 1.00000 1.8 TiB 598 GiB 596 GiB 9 KiB 2.3 GiB 1.2 TiB 32.11 0.99 23 up 33 hdd 1.81940 1.00000 1.8 TiB 652 GiB 650 GiB 14 KiB 2.8 GiB 1.2 TiB 35.01 1.08 25 up 40 hdd 1.81940 1.00000 1.8 TiB 600 GiB 598 GiB 1.0 MiB 2.4 GiB 1.2 TiB 32.21 0.99 23 up 47 hdd 1.81940 1.00000 1.8 TiB 521 GiB 519 GiB 14 KiB 2.4 GiB 1.3 TiB 27.98 0.86 20 up 54 hdd 1.81940 1.00000 1.8 TiB 522 GiB 519 GiB 4.5 MiB 2.8 GiB 1.3 TiB 28.00 0.86 21 up 61 hdd 1.81940 1.00000 1.8 TiB 548 GiB 545 GiB 12 KiB 2.9 GiB 1.3 TiB 29.39 0.90 21 up 81 hdd 1.81940 1.00000 1.8 TiB 522 GiB 518 GiB 24 KiB 3.6 GiB 1.3 TiB 28.02 0.86 20 up 82 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 7 KiB 6.7 GiB 4.8 TiB 33.49 1.03 96 up 83 hdd 1.81940 1.00000 1.8 TiB 521 GiB 518 GiB 2 KiB 2.8 GiB 1.3 TiB 27.98 0.86 20 up 84 hdd 1.81940 1.00000 1.8 TiB 625 GiB 621 GiB 229 KiB 3.7 GiB 1.2 TiB 33.54 1.03 24 up 85 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 56 MiB 7.5 GiB 5.0 TiB 31.45 0.97 91 up 86 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 3 KiB 2.6 GiB 1.2 TiB 34.91 1.07 25 up 87 hdd 1.81940 1.00000 1.8 TiB 522 GiB 519 GiB 55 MiB 2.7 GiB 1.3 TiB 27.99 0.86 21 up 104 hdd 1.81940 1.00000 1.8 TiB 520 GiB 518 GiB 1.0 MiB 1.7 GiB 1.3 TiB 27.89 0.86 20 up 6 hdd 1.81897 1.00000 1.8 TiB 598 GiB 595 GiB 26 KiB 2.3 GiB 1.2 TiB 32.08 0.99 23 up 12 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 54 MiB 2.8 GiB 1.2 TiB 32.22 0.99 24 up 19 hdd 1.81897 1.00000 1.8 TiB 599 GiB 597 GiB 4 KiB 2.1 GiB 1.2 TiB 32.15 0.99 23 up 28 hdd 1.81897 1.00000 1.8 TiB 678 GiB 675 GiB 51 MiB 2.9 GiB 1.2 TiB 36.37 1.12 27 up 35 hdd 1.81897 1.00000 1.8 TiB 572 GiB 571 GiB 10 KiB 1.5 GiB 1.3 TiB 30.72 0.95 22 up 44 hdd 1.81897 1.00000 1.8 TiB 651 GiB 649 GiB 55 MiB 2.1 GiB 1.2 TiB 34.94 1.07 26 up 52 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 21 KiB 2.2 GiB 1.2 TiB 36.29 1.12 26 up 58 hdd 1.81897 1.00000 1.8 TiB 677 GiB 674 GiB 10 KiB 2.9 GiB 1.2 TiB 36.36 1.12 26 up 63 hdd 1.81897 1.00000 1.8 TiB 572 GiB 571 GiB 17 KiB 1.7 GiB 1.3 TiB 30.73 0.95 22 up 67 hdd 1.81897 1.00000 1.8 TiB 652 GiB 649 GiB 55 MiB 2.8 GiB 1.2 TiB 34.99 1.08 26 up 71 hdd 1.81897 1.00000 1.8 TiB 625 GiB 623 GiB 2 KiB 2.0 GiB 1.2 TiB 33.55 1.03 24 up 73 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 14 KiB 2.8 GiB 1.2 TiB 34.92 1.07 25 up 74 hdd 1.81897 1.00000 1.8 TiB 600 GiB 597 GiB 20 KiB 3.0 GiB 1.2 TiB 32.20 0.99 23 up 75 hdd 1.81897 1.00000 1.8 TiB 572 GiB 570 GiB 29 KiB 2.1 GiB 1.3 TiB 30.71 0.94 22 up 77 hdd 1.81897 1.00000 1.8 TiB 600 GiB 597 GiB 51 MiB 3.0 GiB 1.2 TiB 32.20 0.99 24 up 78 hdd 1.81940 1.00000 1.8 TiB 650 GiB 648 GiB 21 KiB 2.1 GiB 1.2 TiB 34.90 1.07 25 up 1 hdd 1.81940 1.00000 1.8 TiB 678 GiB 675 GiB 58 MiB 3.3 GiB 1.2 TiB 36.39 1.12 27 up 10 hdd 7.29999 1.00000 7.3 TiB 2.4 TiB 2.4 TiB 52 MiB 7.1 GiB 4.9 TiB 33.15 1.02 97 up 17 hdd 1.81940 1.00000 1.8 TiB 495 GiB 493 GiB 51 MiB 2.0 GiB 1.3 TiB 26.55 0.82 20 up 24 hdd 1.81940 1.00000 1.8 TiB 548 GiB 544 GiB 3.4 MiB 3.8 GiB 1.3 TiB 29.40 0.90 21 up 31 hdd 7.29999 1.00000 7.3 TiB 2.3 TiB 2.3 TiB 60 MiB 8.4 GiB 5.0 TiB 31.44 0.97 91 up 38 hdd 1.81940 1.00000 1.8 TiB 571 GiB 570 GiB 26 KiB 1.6 GiB 1.3 TiB 30.67 0.94 22 up 46 hdd 1.81940 1.00000 1.8 TiB 651 GiB 649 GiB 2 KiB 2.3 GiB 1.2 TiB 34.95 1.08 25 up 53 hdd 1.81940 1.00000 1.8 TiB 600 GiB 597 GiB 9 KiB 2.8 GiB 1.2 TiB 32.18 0.99 23 up 5 hdd 1.81940 1.00000 1.8 TiB 676 GiB 673 GiB 55 MiB 2.3 GiB 1.2 TiB 36.27 1.12 27 up 9 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 15 KiB 3.0 GiB 1.2 TiB 36.31 1.12 26 up 13 hdd 1.81940 1.00000 1.8 TiB 676 GiB 674 GiB 59 MiB 2.5 GiB 1.2 TiB 36.30 1.12 27 up 20 hdd 1.81940 1.00000 1.8 TiB 573 GiB 571 GiB 9 KiB 2.7 GiB 1.3 TiB 30.77 0.95 22 up 25 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 2 KiB 3.0 GiB 1.2 TiB 36.33 1.12 26 up 30 hdd 1.81940 1.00000 1.8 TiB 573 GiB 571 GiB 49 MiB 1.5 GiB 1.3 TiB 30.76 0.95 23 up 36 hdd 1.81999 1.00000 1.8 TiB 677 GiB 674 GiB 15 KiB 2.4 GiB 1.2 TiB 36.32 1.12 26 up 42 hdd 1.81940 1.00000 1.8 TiB 598 GiB 597 GiB 2 KiB 1.4 GiB 1.2 TiB 32.09 0.99 23 up 45 hdd 1.81999 1.00000 1.8 TiB 599 GiB 597 GiB 14 KiB 2.2 GiB 1.2 TiB 32.15 0.99 23 up 51 hdd 0.90970 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 destroyed 56 hdd 1.81940 1.00000 1.8 TiB 574 GiB 572 GiB 9 KiB 1.7 GiB 1.3 TiB 30.79 0.95 22 up 60 hdd 1.81940 1.00000 1.8 TiB 572 GiB 570 GiB 12 KiB 1.4 GiB 1.3 TiB 30.69 0.94 22 up 65 hdd 1.81940 1.00000 1.8 TiB 675 GiB 673 GiB 61 MiB 2.3 GiB 1.2 TiB 36.26 1.12 27 up 68 hdd 1.81940 1.00000 1.8 TiB 571 GiB 570 GiB 16 KiB 1.4 GiB 1.3 TiB 30.68 0.94 22 up 70 hdd 1.81940 1.00000 1.8 TiB 571 GiB 569 GiB 14 KiB 1.3 GiB 1.3 TiB 30.63 0.94 22 up 105 hdd 0.90700 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 destroyed TOTAL 320 TiB 104 TiB 104 TiB 2.9 GiB 378 GiB 216 TiB 32.51 
MIN/MAX VAR: 0.82/1.12  STDDEV: 2.53

# ceph osd dump | grep pool
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 191543 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 150.00 pool 2 'wizard_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 191543 lfor 0/8092/8090 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 7.86 pool 3 'wizard_data' erasure profile k6_m2_host size 8 min_size 7 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 191543 lfor 0/0/41881 flags hashpspool,ec_overwrites stripe_width 24576 application cephfs
- All the REWEIGHT entries of `ceph osd tree` are currently set to 1 (except for the two destroyed OSDs which are at 0) 

Thanks again,

Nicola

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux