Which host did you reboot to cause the incomplete pgs? Do you happen to have the output of ceph -s or, even better, ceph pg dump from the period with the incomplete pgs? From what i can see, the pgs should not have gone incomplete (at least, not for long). -Sam On Mon, Apr 15, 2013 at 10:17 AM, Jeppesen, Nelson <Nelson.Jeppesen@xxxxxxxxxx> wrote: > OSD tree and OSD map inline. Thanks for the help. OSD 32 was removed due to a bad drive. > > # id weight type name up/down reweight > -1 96 root default > -3 48 rack rack_aa12 > -2 4 host osdhost001 > 0 1 osd.0 up 1 > 1 1 osd.1 up 1 > 2 1 osd.2 up 1 > 3 1 osd.3 up 1 > -4 4 host osdhost003 > 10 1 osd.10 up 1 > 11 1 osd.11 up 1 > 8 1 osd.8 up 1 > 9 1 osd.9 up 1 > -5 4 host osdhost004 > 12 1 osd.12 up 1 > 13 1 osd.13 up 1 > 14 1 osd.14 up 1 > 15 1 osd.15 up 1 > -6 4 host osdhost005 > 16 1 osd.16 up 1 > 17 1 osd.17 up 1 > 18 1 osd.18 up 1 > 19 1 osd.19 up 1 > -7 4 host osdhost006 > 20 1 osd.20 up 1 > 21 1 osd.21 up 1 > 22 1 osd.22 up 1 > 23 1 osd.23 up 1 > -8 4 host osdhost007 > 24 1 osd.24 up 1 > 25 1 osd.25 up 1 > 26 1 osd.26 up 1 > 27 1 osd.27 up 1 > -9 4 host osdhost008 > 28 1 osd.28 up 1 > 29 1 osd.29 up 1 > 30 1 osd.30 up 1 > 31 1 osd.31 up 1 > -10 4 host osdhost009 > 33 1 osd.33 up 1 > 34 1 osd.34 up 1 > 35 1 osd.35 up 1 > -11 4 host osdhost010 > 36 1 osd.36 up 1 > 37 1 osd.37 up 1 > 38 1 osd.38 up 1 > 39 1 osd.39 up 1 > -12 4 host osdhost002 > 4 1 osd.4 up 1 > 5 1 osd.5 up 1 > 6 1 osd.6 up 1 > 7 1 osd.7 up 1 > -13 4 host osdhost011 > 40 1 osd.40 up 1 > 41 1 osd.41 up 1 > 42 1 osd.42 up 1 > 43 1 osd.43 up 1 > -14 4 host osdhost012 > 44 1 osd.44 up 1 > 45 1 osd.45 up 1 > 46 1 osd.46 up 1 > 47 1 osd.47 up 1 > -27 48 rack rack_aa20 > -15 4 host osdhost013 > 48 1 osd.48 up 1 > 49 1 osd.49 up 1 > 50 1 osd.50 up 1 > 51 1 osd.51 up 1 > -16 4 host osdhost014 > 52 1 osd.52 up 1 > 53 1 osd.53 up 1 > 54 1 osd.54 up 1 > 55 1 osd.55 up 1 > -17 4 host osdhost015 > 56 1 osd.56 up 1 > 57 1 osd.57 up 1 > 58 1 osd.58 up 1 > 59 1 osd.59 up 1 > -18 4 host osdhost016 > 60 1 osd.60 up 1 > 61 1 osd.61 up 1 > 62 1 osd.62 up 1 > 63 1 osd.63 up 1 > -19 4 host osdhost017 > 64 1 osd.64 up 1 > 65 1 osd.65 up 1 > 66 1 osd.66 up 1 > 67 1 osd.67 up 1 > -20 4 host osdhost018 > 68 1 osd.68 up 1 > 69 1 osd.69 up 1 > 70 1 osd.70 up 1 > 71 1 osd.71 up 1 > -21 4 host osdhost019 > 72 1 osd.72 up 1 > 73 1 osd.73 up 1 > 74 1 osd.74 up 1 > 75 1 osd.75 up 1 > -22 4 host osdhost020 > 76 1 osd.76 up 1 > 77 1 osd.77 up 1 > 78 1 osd.78 up 1 > 79 1 osd.79 up 1 > -23 4 host osdhost021 > 80 1 osd.80 up 1 > 81 1 osd.81 up 1 > 82 1 osd.82 up 1 > 83 1 osd.83 up 1 > -24 4 host osdhost022 > 84 1 osd.84 up 1 > 85 1 osd.85 up 1 > 86 1 osd.86 up 1 > 87 1 osd.87 up 1 > -25 4 host osdhost023 > 88 1 osd.88 up 1 > 89 1 osd.89 up 1 > 90 1 osd.90 up 1 > 91 1 osd.91 up 1 > -26 4 host osdhost024 > 92 1 osd.92 up 1 > 93 1 osd.93 up 1 > 94 1 osd.94 up 1 > 95 1 osd.95 up 1 > > > root@osdhost001:~# osdmaptool /tmp/map --print > osdmaptool: osdmap file '/tmp/map' > epoch 2028 > fsid 969f28c3-5ee1-4451-9b5b-97c52b724a06 > created 2013-01-26 15:31:21.670594 > modifed 2013-04-15 10:13:03.045973 > flags > > pool 13 '.rgw.buckets' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 4800 pgp_num 4800 last_change 1198 owner 0 > pool 14 '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 242 owner 18446744073709551615 > pool 15 '.rgw.gc' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 243 owner 18446744073709551615 > pool 16 '.rgw.control' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 244 owner 18446744073709551615 > pool 17 '.users.uid' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 246 owner 0 > pool 18 '.users.email' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 248 owner 0 > pool 19 '.users' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 250 owner 0 > pool 20 '.usage' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 256 owner 18446744073709551615 > pool 21 '.users.swift' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 1138 owner 0 > > max_osd 96 > osd.0 up in weight 1 up_from 1770 up_thru 1994 down_at 1766 last_clean_interval [940,1765) 10.198.141.21:6800/1689 10.198.142.21:6800/1689 10.198.142.21:6801/1689 exists,up 60fdecb6-fb58-4742-91f9-bc21265bcb0b > osd.1 up in weight 1 up_from 1771 up_thru 1994 down_at 1766 last_clean_interval [941,1765) 10.198.141.21:6801/1786 10.198.142.21:6802/1786 10.198.142.21:6803/1786 exists,up 2bb3ac97-ddae-4782-89b2-51e216e39923 > osd.2 up in weight 1 up_from 1772 up_thru 1992 down_at 1768 last_clean_interval [941,1765) 10.198.141.21:6802/2015 10.198.142.21:6804/2015 10.198.142.21:6805/2015 exists,up 6f5671f3-503d-41fe-8b9e-895cf8ddb287 > osd.3 up in weight 1 up_from 1772 up_thru 1994 down_at 1768 last_clean_interval [941,1765) 10.198.141.21:6803/2254 10.198.142.21:6806/2254 10.198.142.21:6807/2254 exists,up 123d88f1-901d-40ed-92cc-1cdb7e7ac530 > osd.4 up in weight 1 up_from 1780 up_thru 1994 down_at 1774 last_clean_interval [1589,1773) 10.198.141.22:6800/2063 10.198.142.22:6800/2063 10.198.142.22:6801/2063 exists,up ffb8ba24-80a8-4716-be06-5a3dc7274bd3 > osd.5 up in weight 1 up_from 1781 up_thru 1994 down_at 1774 last_clean_interval [1588,1773) 10.198.141.22:6801/2277 10.198.142.22:6802/2277 10.198.142.22:6803/2277 exists,up 00b83261-1262-4c56-af1c-13393459ca9a > osd.6 up in weight 1 up_from 1781 up_thru 1994 down_at 1775 last_clean_interval [1589,1773) 10.198.141.22:6802/2516 10.198.142.22:6804/2516 10.198.142.22:6805/2516 exists,up cf4f3f79-96b4-40fc-b724-d04ffb03a737 > osd.7 up in weight 1 up_from 1783 up_thru 1994 down_at 1777 last_clean_interval [1589,1773) 10.198.141.22:6803/2746 10.198.142.22:6806/2746 10.198.142.22:6807/2746 exists,up 30ae4d62-516e-4548-8c9f-9aee632e8ef5 > osd.8 up in weight 1 up_from 1793 up_thru 1994 down_at 1789 last_clean_interval [1596,1785) 10.198.141.23:6802/2765 10.198.142.23:6804/2765 10.198.142.23:6805/2765 exists,up be1553cf-ee60-4bc2-8853-cf30920521b0 > osd.9 up in weight 1 up_from 1792 up_thru 1992 down_at 1789 last_clean_interval [1596,1785) 10.198.141.23:6803/2977 10.198.142.23:6806/2977 10.198.142.23:6807/2977 exists,up bb15ac76-7b01-45a6-8ca4-20a238d1449d > osd.10 up in weight 1 up_from 1791 up_thru 1994 down_at 1786 last_clean_interval [1594,1785) 10.198.141.23:6800/1637 10.198.142.23:6800/1637 10.198.142.23:6801/1637 exists,up 3e4ecf43-9de3-4c90-86d9-50d18e71d8de > osd.11 up in weight 1 up_from 1792 up_thru 1992 down_at 1786 last_clean_interval [1595,1785) 10.198.141.23:6801/1731 10.198.142.23:6802/1731 10.198.142.23:6803/1731 exists,up 00ec7fdd-cf91-44be-862c-0e1054e3f5b5 > osd.12 up in weight 1 up_from 1798 up_thru 1994 down_at 1796 last_clean_interval [1601,1795) 10.198.141.24:6800/1656 10.198.142.24:6800/1656 10.198.142.24:6801/1656 exists,up cc39f155-a3c6-4499-ae4d-03eb5bafb338 > osd.13 up in weight 1 up_from 1800 up_thru 1992 down_at 1796 last_clean_interval [1602,1795) 10.198.141.24:6801/1750 10.198.142.24:6802/1750 10.198.142.24:6803/1750 exists,up dbf0a788-bfba-417c-aec9-a14791ad5096 > osd.14 up in weight 1 up_from 1798 up_thru 1992 down_at 1796 last_clean_interval [1601,1795) 10.198.141.24:6802/1843 10.198.142.24:6804/1843 10.198.142.24:6805/1843 exists,up ff4aa630-e637-43e6-97b8-de7033f6a271 > osd.15 up in weight 1 up_from 1800 up_thru 1994 down_at 1796 last_clean_interval [1602,1795) 10.198.141.24:6803/1953 10.198.142.24:6806/1953 10.198.142.24:6807/1953 exists,up 04a2cfb7-a9ec-4701-8ba1-e97e3d78a321 > osd.16 up in weight 1 up_from 1808 up_thru 1992 down_at 1803 last_clean_interval [1609,1802) 10.198.141.25:6800/1618 10.198.142.25:6800/1618 10.198.142.25:6801/1618 exists,up 6099188a-073d-4180-ba64-b12bf975986a > osd.17 up in weight 1 up_from 1807 up_thru 1992 down_at 1803 last_clean_interval [1610,1802) 10.198.141.25:6801/1712 10.198.142.25:6802/1712 10.198.142.25:6803/1712 exists,up 838f20dc-147a-47e7-ac63-4016d4801bdb > osd.18 up in weight 1 up_from 1808 up_thru 1982 down_at 1805 last_clean_interval [1610,1802) 10.198.141.25:6802/1804 10.198.142.25:6804/1804 10.198.142.25:6805/1804 exists,up dcff4b5c-ff48-4ff4-9e46-953fa185d6d8 > osd.19 up in weight 1 up_from 1808 up_thru 1994 down_at 1805 last_clean_interval [1611,1802) 10.198.141.25:6803/1913 10.198.142.25:6806/1913 10.198.142.25:6807/1913 exists,up c29b0f83-fe81-4eac-afa5-e3b60bf910fe > osd.20 up in weight 1 up_from 1815 up_thru 1992 down_at 1811 last_clean_interval [1616,1810) 10.198.141.26:6800/1683 10.198.142.26:6800/1683 10.198.142.26:6801/1683 exists,up 9550a366-d28e-45d6-ab83-20825d0531a9 > osd.21 up in weight 1 up_from 1816 up_thru 1992 down_at 1811 last_clean_interval [1615,1810) 10.198.141.26:6801/1777 10.198.142.26:6802/1777 10.198.142.26:6803/1777 exists,up aabc75c5-2d98-4ff5-a437-fdfa92e93ade > osd.22 up in weight 1 up_from 1817 up_thru 1994 down_at 1813 last_clean_interval [1617,1810) 10.198.141.26:6802/1869 10.198.142.26:6804/1869 10.198.142.26:6805/1869 exists,up 0a88db42-9427-4980-8d63-ed5f7adc9663 > osd.23 up in weight 1 up_from 1816 up_thru 1994 down_at 1813 last_clean_interval [1616,1810) 10.198.141.26:6803/1990 10.198.142.26:6806/1990 10.198.142.26:6807/1990 exists,up f7aac20c-7153-498e-b076-87f69ad93c2e > osd.24 up in weight 1 up_from 1825 up_thru 1994 down_at 1820 last_clean_interval [1625,1819) 10.198.141.27:6800/1763 10.198.142.27:6800/1763 10.198.142.27:6801/1763 exists,up e971ae57-687b-465f-9b08-4adb12a7cf42 > osd.25 up in weight 1 up_from 1825 up_thru 1992 down_at 1820 last_clean_interval [1624,1819) 10.198.141.27:6801/1858 10.198.142.27:6802/1858 10.198.142.27:6803/1858 exists,up 3dd19146-1fb1-4626-bfde-c50c7923e7b2 > osd.26 up in weight 1 up_from 1826 up_thru 1994 down_at 1821 last_clean_interval [1625,1819) 10.198.141.27:6802/1950 10.198.142.27:6804/1950 10.198.142.27:6805/1950 exists,up 3adc9c6c-1d61-43e2-9c20-b277703965f7 > osd.27 up in weight 1 up_from 1828 up_thru 1992 down_at 1823 last_clean_interval [1625,1819) 10.198.141.27:6803/2059 10.198.142.27:6806/2059 10.198.142.27:6807/2059 exists,up 2dc425b4-6d19-4624-a677-68f2be87280a > osd.28 up in weight 1 up_from 1835 up_thru 1992 down_at 1831 last_clean_interval [1632,1830) 10.198.141.28:6800/1786 10.198.142.28:6800/1786 10.198.142.28:6801/1786 exists,up b0c33e64-8b24-4227-b752-aec1a1cbdaa5 > osd.29 up in weight 1 up_from 1836 up_thru 1994 down_at 1831 last_clean_interval [1633,1830) 10.198.141.28:6801/1881 10.198.142.28:6802/1881 10.198.142.28:6803/1881 exists,up 44b3a173-3749-416c-9b54-dd1020cddce0 > osd.30 up in weight 1 up_from 1836 up_thru 1994 down_at 1831 last_clean_interval [1631,1830) 10.198.141.28:6802/1985 10.198.142.28:6804/1985 10.198.142.28:6805/1985 exists,up 2fdd18c3-370b-48a2-8567-e532a0601421 > osd.31 up in weight 1 up_from 1836 up_thru 1992 down_at 1833 last_clean_interval [1633,1830) 10.198.141.28:6803/2095 10.198.142.28:6806/2095 10.198.142.28:6807/2095 exists,up a50985ac-2f05-488d-b119-079b1a90a38f > osd.33 up in weight 1 up_from 1872 up_thru 2019 down_at 1860 last_clean_interval [1845,1859) 10.198.141.29:6801/1960 10.198.142.29:6802/1960 10.198.142.29:6803/1960 exists,up 4dd53408-3b0b-4ca8-a834-eca30a152e82 > osd.34 up in weight 1 up_from 1872 up_thru 2025 down_at 1860 last_clean_interval [1846,1859) 10.198.141.29:6802/2082 10.198.142.29:6804/2082 10.198.142.29:6805/2082 exists,up 365a4f9f-adf9-44e4-8132-71bccb313b65 > osd.35 up in weight 1 up_from 1875 up_thru 2005 down_at 1860 last_clean_interval [1848,1859) 10.198.141.29:6803/2175 10.198.142.29:6806/2175 10.198.142.29:6807/2175 exists,up c507dd1a-8de3-4fe5-804b-f1271eab3959 > osd.36 up in weight 1 up_from 1856 up_thru 1994 down_at 1851 last_clean_interval [1649,1850) 10.198.141.30:6800/1914 10.198.142.30:6800/1914 10.198.142.30:6801/1914 exists,up 5e536ea7-9478-4022-8f8a-954cbe299fd2 > osd.37 up in weight 1 up_from 1857 up_thru 1991 down_at 1851 last_clean_interval [1648,1850) 10.198.141.30:6801/2008 10.198.142.30:6802/2008 10.198.142.30:6803/2008 exists,up 89f03a60-7a00-4e4c-8302-8a9defbad9b4 > osd.38 up in weight 1 up_from 1857 up_thru 1994 down_at 1853 last_clean_interval [1649,1850) 10.198.141.30:6802/2101 10.198.142.30:6804/2101 10.198.142.30:6805/2101 exists,up ffce38bd-4b60-4620-a9db-00613a0fbd48 > osd.39 up in weight 1 up_from 1857 up_thru 1994 down_at 1853 last_clean_interval [1650,1850) 10.198.141.30:6803/2227 10.198.142.30:6806/2227 10.198.142.30:6807/2227 exists,up 525f1028-9b4c-4751-95a3-8e4e41c0489c > osd.40 up in weight 1 up_from 1884 up_thru 1992 down_at 1866 last_clean_interval [1656,1865) 10.198.141.31:6800/1971 10.198.142.31:6800/1971 10.198.142.31:6801/1971 exists,up f7882b62-ae97-45e9-83de-170df085a82c > osd.41 up in weight 1 up_from 1884 up_thru 1994 down_at 1866 last_clean_interval [1657,1865) 10.198.141.31:6801/2066 10.198.142.31:6802/2066 10.198.142.31:6803/2066 exists,up 4a4a41bc-1be9-4e21-992f-2c9ed7c55c15 > osd.42 up in weight 1 up_from 1887 up_thru 1992 down_at 1868 last_clean_interval [1658,1865) 10.198.141.31:6802/2175 10.198.142.31:6804/2175 10.198.142.31:6805/2175 exists,up 0cfde622-b84b-4d28-a6af-899dbc9f1a67 > osd.43 up in weight 1 up_from 1884 up_thru 1992 down_at 1868 last_clean_interval [1657,1865) 10.198.141.31:6803/2277 10.198.142.31:6806/2277 10.198.142.31:6807/2277 exists,up dbeb8a3e-b359-4e28-bc7f-fa7677fb8e46 > osd.44 up in weight 1 up_from 1901 up_thru 1994 down_at 1895 last_clean_interval [1667,1894) 10.198.141.32:6800/2005 10.198.142.32:6800/2005 10.198.142.32:6801/2005 exists,up 96b2f069-3628-4aed-af4e-63ef42442620 > osd.45 up in weight 1 up_from 1900 up_thru 1994 down_at 1895 last_clean_interval [1666,1894) 10.198.141.32:6801/2099 10.198.142.32:6802/2099 10.198.142.32:6803/2099 exists,up c7e68723-df40-42fe-85a4-649ceeed8c4e > osd.46 up in weight 1 up_from 1901 up_thru 1994 down_at 1896 last_clean_interval [1668,1894) 10.198.141.32:6802/2192 10.198.142.32:6804/2192 10.198.142.32:6805/2192 exists,up 787ccacb-9b86-4e5f-9f5f-e381b5229647 > osd.47 up in weight 1 up_from 1901 up_thru 1992 down_at 1898 last_clean_interval [1668,1894) 10.198.141.32:6803/2328 10.198.142.32:6806/2328 10.198.142.32:6807/2328 exists,up 2b76b5ef-61fe-45d9-b48b-e165bebe9d83 > osd.48 up in weight 1 up_from 1910 up_thru 1927 down_at 1904 last_clean_interval [1676,1903) 10.198.141.33:6800/2101 10.198.142.33:6800/2101 10.198.142.33:6801/2101 exists,up 09bff29e-f07f-4239-942a-1ce4531a00a9 > osd.49 up in weight 1 up_from 1908 up_thru 1927 down_at 1904 last_clean_interval [1677,1903) 10.198.141.33:6801/2195 10.198.142.33:6802/2195 10.198.142.33:6803/2195 exists,up 4fc6d620-5801-4000-a37a-debcbbafb55b > osd.50 up in weight 1 up_from 1911 up_thru 2012 down_at 1906 last_clean_interval [1679,1903) 10.198.141.33:6802/2299 10.198.142.33:6804/2299 10.198.142.33:6805/2299 exists,up 9158613e-449c-484f-9f07-23e776f5f7e4 > osd.51 up in weight 1 up_from 1912 up_thru 1927 down_at 1906 last_clean_interval [1677,1903) 10.198.141.33:6803/2410 10.198.142.33:6806/2410 10.198.142.33:6807/2410 exists,up 834e9c58-f956-4edb-b5be-478e78ed3b7a > osd.52 up in weight 1 up_from 1920 up_thru 2020 down_at 1914 last_clean_interval [1685,1913) 10.198.141.34:6800/2108 10.198.142.34:6800/2108 10.198.142.34:6801/2108 exists,up e8d66816-8ce6-4a7c-8b0a-e535fa5c12ad > osd.53 up in weight 1 up_from 1920 up_thru 1928 down_at 1914 last_clean_interval [1685,1913) 10.198.141.34:6801/2202 10.198.142.34:6802/2202 10.198.142.34:6803/2202 exists,up 25dca904-a617-4504-8857-c13c8f31b758 > osd.54 up in weight 1 up_from 1919 up_thru 1919 down_at 1914 last_clean_interval [1684,1913) 10.198.141.34:6802/2312 10.198.142.34:6804/2312 10.198.142.34:6805/2312 exists,up ad65b25d-2fb9-471e-9cd2-8c08ff86f6d3 > osd.55 up in weight 1 up_from 1919 up_thru 1926 down_at 1917 last_clean_interval [1685,1913) 10.198.141.34:6803/2414 10.198.142.34:6806/2414 10.198.142.34:6807/2414 exists,up ff0b8f74-8f4d-47eb-a008-8229981f01f7 > osd.56 up in weight 1 up_from 1935 up_thru 1940 down_at 1922 last_clean_interval [1692,1921) 10.198.141.35:6800/2234 10.198.142.35:6800/2234 10.198.142.35:6801/2234 exists,up 88f2a234-1273-4519-9598-56a071250231 > osd.57 up in weight 1 up_from 1931 up_thru 1931 down_at 1922 last_clean_interval [1691,1921) 10.198.141.35:6801/2328 10.198.142.35:6802/2328 10.198.142.35:6803/2328 exists,up 2c8f0562-24d8-4bb7-87a6-f58cc125f824 > osd.58 up in weight 1 up_from 1936 up_thru 1942 down_at 1922 last_clean_interval [1693,1921) 10.198.141.35:6802/2421 10.198.142.35:6804/2421 10.198.142.35:6805/2421 exists,up 687d3a7d-a0f5-4854-acf4-d66695f58e66 > osd.59 up in weight 1 up_from 1933 up_thru 1952 down_at 1924 last_clean_interval [1692,1921) 10.198.141.35:6803/2557 10.198.142.35:6806/2557 10.198.142.35:6807/2557 exists,up 794d2daf-3f3c-4dda-93c8-4ca4d4369aac > osd.60 up in weight 1 up_from 1967 up_thru 1967 down_at 1955 last_clean_interval [1700,1954) 10.198.141.36:6800/2283 10.198.142.36:6800/2283 10.198.142.36:6801/2283 exists,up bbed85bb-7096-4ea6-8df9-04c1a48a0021 > osd.61 up in weight 1 up_from 1968 up_thru 1968 down_at 1955 last_clean_interval [1701,1954) 10.198.141.36:6801/2378 10.198.142.36:6802/2378 10.198.142.36:6803/2378 exists,up 2e3fc7a7-71b0-477d-a258-e4f7646a6e32 > osd.62 up in weight 1 up_from 1967 up_thru 1967 down_at 1956 last_clean_interval [1701,1954) 10.198.141.36:6802/2471 10.198.142.36:6804/2471 10.198.142.36:6805/2471 exists,up e2b88a19-da9d-4bc4-b89b-4d467a5da927 > osd.63 up in weight 1 up_from 1968 up_thru 1977 down_at 1956 last_clean_interval [1701,1954) 10.198.141.36:6803/2580 10.198.142.36:6806/2580 10.198.142.36:6807/2580 exists,up 892ed9bb-a87e-49ce-a6ea-ef434283b9a0 > osd.64 up in weight 1 up_from 1991 up_thru 1991 down_at 1979 last_clean_interval [1708,1978) 10.198.141.37:6800/2317 10.198.142.37:6800/2317 10.198.142.37:6801/2317 exists,up 0028d026-0c49-470a-921e-237a820ebbf6 > osd.65 up in weight 1 up_from 1992 up_thru 1992 down_at 1979 last_clean_interval [1708,1978) 10.198.141.37:6801/2411 10.198.142.37:6802/2411 10.198.142.37:6803/2411 exists,up 357c43bb-bb3e-4ff8-b747-0047c1a03996 > osd.66 up in weight 1 up_from 1992 up_thru 1992 down_at 1982 last_clean_interval [1708,1978) 10.198.141.37:6802/2503 10.198.142.37:6804/2503 10.198.142.37:6805/2503 exists,up 1b45e8fb-ecc5-4fd1-a24a-9b5a7921bb8c > osd.67 up in weight 1 up_from 1994 up_thru 1994 down_at 1982 last_clean_interval [1707,1978) 10.198.141.37:6803/2622 10.198.142.37:6806/2622 10.198.142.37:6807/2622 exists,up df199549-ccec-467e-b1a5-635dbca53497 > osd.68 up in weight 1 up_from 1717 up_thru 1928 down_at 1710 last_clean_interval [1540,1709) 10.198.141.38:6800/2305 10.198.142.38:6800/2305 10.198.142.38:6801/2305 exists,up 3374c038-cbbe-4989-82d0-d0d438e11da9 > osd.69 up in weight 1 up_from 1717 up_thru 2014 down_at 1711 last_clean_interval [1540,1709) 10.198.141.38:6801/2400 10.198.142.38:6802/2400 10.198.142.38:6803/2400 exists,up 387c4522-f15b-4692-aee8-7317e68b0cb8 > osd.70 up in weight 1 up_from 1715 up_thru 1926 down_at 1711 last_clean_interval [1539,1709) 10.198.141.38:6802/2504 10.198.142.38:6804/2504 10.198.142.38:6805/2504 exists,up fc362184-4b8e-4680-b189-151a030370b4 > osd.71 up in weight 1 up_from 1717 up_thru 2024 down_at 1713 last_clean_interval [1540,1709) 10.198.141.38:6803/2623 10.198.142.38:6806/2623 10.198.142.38:6807/2623 exists,up e32596ae-9021-4b12-8a59-86e190d939db > osd.72 up in weight 1 up_from 1725 up_thru 1927 down_at 1719 last_clean_interval [1546,1718) 10.198.141.39:6800/2310 10.198.142.39:6800/2310 10.198.142.39:6801/2310 exists,up a8279024-9b81-4824-bc7d-52afa2db4516 > osd.73 up in weight 1 up_from 1724 up_thru 1901 down_at 1719 last_clean_interval [1545,1718) 10.198.141.39:6801/2405 10.198.142.39:6802/2405 10.198.142.39:6803/2405 exists,up 95502db5-4c4b-4b77-bf76-9e2ef6e6fb9a > osd.74 up in weight 1 up_from 1725 up_thru 1927 down_at 1720 last_clean_interval [1547,1718) 10.198.141.39:6802/2497 10.198.142.39:6804/2497 10.198.142.39:6805/2497 exists,up 9aad6a95-a671-4f6f-8489-d4b4a62ba27f > osd.75 up in weight 1 up_from 1726 up_thru 1927 down_at 1722 last_clean_interval [1547,1718) 10.198.141.39:6803/2606 10.198.142.39:6806/2606 10.198.142.39:6807/2606 exists,up fe3d0639-824f-49b2-9126-47b881471b2d > osd.76 up in weight 1 up_from 1732 up_thru 2017 down_at 1728 last_clean_interval [1556,1727) 10.198.141.40:6800/2409 10.198.142.40:6800/2409 10.198.142.40:6801/2409 exists,up 0eb886e3-8080-4d2c-b45c-02d1bfcde894 > osd.77 up in weight 1 up_from 1733 up_thru 1927 down_at 1728 last_clean_interval [1555,1727) 10.198.141.40:6801/2503 10.198.142.40:6802/2503 10.198.142.40:6803/2503 exists,up 1cc21081-ca09-4875-865e-5938da044096 > osd.78 up in weight 1 up_from 1733 up_thru 2008 down_at 1728 last_clean_interval [1556,1727) 10.198.141.40:6802/2596 10.198.142.40:6804/2596 10.198.142.40:6805/2596 exists,up 87910bf8-85f5-4632-a6bc-f540cbeb3699 > osd.79 up in weight 1 up_from 1733 up_thru 1901 down_at 1730 last_clean_interval [1554,1727) 10.198.141.40:6803/2714 10.198.142.40:6806/2714 10.198.142.40:6807/2714 exists,up 9fb83231-b14f-4a8e-9958-33ad47138050 > osd.80 up in weight 1 up_from 1738 up_thru 2027 down_at 1735 last_clean_interval [1565,1734) 10.198.141.41:6800/2425 10.198.142.41:6800/2425 10.198.142.41:6801/2425 exists,up a582d3d5-49f7-4f72-9c50-20b940ff3ce0 > osd.81 up in weight 1 up_from 1738 up_thru 1927 down_at 1735 last_clean_interval [1563,1734) 10.198.141.41:6801/2519 10.198.142.41:6802/2519 10.198.142.41:6803/2519 exists,up fe1f41cd-eb44-4532-9e14-bdb84bb0707d > osd.82 up in weight 1 up_from 1739 up_thru 1926 down_at 1736 last_clean_interval [1564,1734) 10.198.141.41:6802/2629 10.198.142.41:6804/2629 10.198.142.41:6805/2629 exists,up d708e2bd-4413-45b4-bb0d-ff06492db571 > osd.83 up in weight 1 up_from 1741 up_thru 1927 down_at 1736 last_clean_interval [1565,1734) 10.198.141.41:6803/2732 10.198.142.41:6806/2732 10.198.142.41:6807/2732 exists,up 8d0508a1-7fe7-43e4-a679-1a0a552d3da3 > osd.84 up in weight 1 up_from 1748 up_thru 1901 down_at 1743 last_clean_interval [1572,1742) 10.198.141.42:6800/2528 10.198.142.42:6800/2528 10.198.142.42:6801/2528 exists,up 18bf317c-7b34-4028-9d4e-7100c72a5308 > osd.85 up in weight 1 up_from 1749 up_thru 1926 down_at 1743 last_clean_interval [1573,1742) 10.198.141.42:6801/2622 10.198.142.42:6802/2622 10.198.142.42:6803/2622 exists,up 440a4b77-59a0-484b-9f29-ea798860e8d5 > osd.86 up in weight 1 up_from 1749 up_thru 1926 down_at 1744 last_clean_interval [1574,1742) 10.198.141.42:6802/2715 10.198.142.42:6804/2715 10.198.142.42:6805/2715 exists,up f506fbd4-b41c-4bc8-9cfd-4909f02c1ffd > osd.87 up in weight 1 up_from 1749 up_thru 2023 down_at 1746 last_clean_interval [1572,1742) 10.198.141.42:6803/2834 10.198.142.42:6806/2834 10.198.142.42:6807/2834 exists,up 922794f2-827b-467c-a18d-685716deb9c1 > osd.88 up in weight 1 up_from 1755 up_thru 1927 down_at 1751 last_clean_interval [1579,1750) 10.198.141.43:6800/2561 10.198.142.43:6800/2561 10.198.142.43:6801/2561 exists,up f2931004-62c5-4ba4-8fd6-c38ad9c97459 > osd.89 up in weight 1 up_from 1755 up_thru 2010 down_at 1751 last_clean_interval [1578,1750) 10.198.141.43:6801/2675 10.198.142.43:6802/2675 10.198.142.43:6803/2675 exists,up 79fd1a48-00d2-4655-8334-0e67196a5f27 > osd.90 up in weight 1 up_from 1755 up_thru 1927 down_at 1753 last_clean_interval [1581,1750) 10.198.141.43:6802/2791 10.198.142.43:6804/2791 10.198.142.43:6805/2791 exists,up 372fb901-4f16-42c6-8e56-cd3c6bd4bac1 > osd.91 up in weight 1 up_from 1756 up_thru 1927 down_at 1753 last_clean_interval [1580,1750) 10.198.141.43:6803/2883 10.198.142.43:6806/2883 10.198.142.43:6807/2883 exists,up 13c8cd96-45ea-42ed-a7ac-64dcee08d9c2 > osd.92 up in weight 1 up_from 1762 up_thru 1927 down_at 1758 last_clean_interval [1134,1757) 10.198.141.44:6800/2670 10.198.142.44:6800/2670 10.198.142.44:6801/2670 exists,up b9515f45-f827-4961-8c33-b0970a50828e > osd.93 up in weight 1 up_from 1764 up_thru 1927 down_at 1758 last_clean_interval [1135,1757) 10.198.141.44:6801/2763 10.198.142.44:6802/2763 10.198.142.44:6803/2763 exists,up 7f712a96-063e-429b-aa53-303c0f4618de > osd.94 up in weight 1 up_from 1763 up_thru 2018 down_at 1758 last_clean_interval [1135,1757) 10.198.141.44:6802/2856 10.198.142.44:6804/2856 10.198.142.44:6805/2856 exists,up b3daae82-6451-4fc0-8fbf-9212b774bac0 > osd.95 up in weight 1 up_from 1764 up_thru 1927 down_at 1760 last_clean_interval [1135,1757) 10.198.141.44:6803/2948 10.198.142.44:6806/2948 10.198.142.44:6807/2948 exists,up 63911b22-01c4-45b1-95de-97a462f8581f > > pg_temp 13.57 [75,33,35] > pg_temp 13.1b8 [94,33,35] > pg_temp 13.234 [69,35,34] > pg_temp 13.2d5 [63,34,33] > pg_temp 13.2db [51,35,34] > pg_temp 13.30a [81,35,34] > pg_temp 13.3ce [35,33,60] > pg_temp 13.400 [74,34,35] > pg_temp 13.46b [35,34,59] > pg_temp 13.47c [90,35,34] > pg_temp 13.4a9 [91,35,34] > pg_temp 13.533 [91,33,34] > pg_temp 13.535 [95,34,33] > pg_temp 13.694 [34,35,66] > pg_temp 13.6d6 [51,33,35] > pg_temp 13.6e0 [74,33,34] > pg_temp 13.6f2 [83,34,35] > pg_temp 13.71c [35,34,65] > pg_temp 13.7e6 [87,33,34] > pg_temp 13.812 [63,35,33] > pg_temp 13.84b [68,34,33] > pg_temp 13.8e0 [88,34,35] > pg_temp 13.8fb [48,35,34] > pg_temp 13.917 [91,33,35] > pg_temp 13.978 [77,35,34] > pg_temp 13.a1b [49,33,35] > pg_temp 13.a57 [51,35,33] > pg_temp 13.a6e [87,35,33] > pg_temp 13.bfb [33,34,56] > pg_temp 13.bff [95,33,35] > pg_temp 13.c33 [81,35,34] > pg_temp 13.c62 [88,33,35] > pg_temp 13.cf0 [94,35,34] > pg_temp 13.d34 [93,35,33] > pg_temp 13.d3b [83,33,35] > pg_temp 13.d53 [95,34,35] > pg_temp 13.dd5 [93,33,35] > pg_temp 13.df0 [68,33,34] > pg_temp 13.e58 [50,33,35] > pg_temp 13.eba [94,35,33] > pg_temp 13.f3d [53,34,35] > pg_temp 13.f5b [34,33,57] > pg_temp 13.f73 [49,33,34] > pg_temp 13.fa7 [72,35,34] > pg_temp 13.10d4 [63,35,33] > pg_temp 13.1107 [53,35,34] > pg_temp 13.1115 [91,33,34] > pg_temp 13.1138 [35,34,66] > pg_temp 13.1191 [34,35,59] > pg_temp 13.1199 [34,33,58] > pg_temp 13.123d [50,35,33] > > Nelson Jeppesen > Disney Technology Solutions and Services > Phone 206-588-5001 > > > -----Original Message----- > From: Samuel Just [mailto:sam.just@xxxxxxxxxxx] > Sent: Monday, April 15, 2013 10:11 AM > To: Jeppesen, Nelson > Cc: Gregory Farnum; ceph-users@xxxxxxxxxxxxxx > Subject: Re: ceph -w question > > Also the ceph osdmap. (ceph osd getmap -o /tmp/map will put the osdmap in /tmp/map). > -Sam > > On Mon, Apr 15, 2013 at 10:09 AM, Samuel Just <sam.just@xxxxxxxxxxx> wrote: >> Can you post the output of ceph osd tree? >> -Sam >> >> On Mon, Apr 15, 2013 at 9:52 AM, Jeppesen, Nelson >> <Nelson.Jeppesen@xxxxxxxxxx> wrote: >>> Thanks for the help but how do I track down this issue? If data is inaccessible, that's a very bad thing given this is production. >>> >>> # ceph osd dump | grep pool >>> pool 13 '.rgw.buckets' rep size 2 crush_ruleset 0 object_hash >>> rjenkins pg_num 4800 pgp_num 4800 last_change 1198 owner 0 pool 14 >>> '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 >>> pgp_num 8 last_change 242 owner 18446744073709551615 pool 15 >>> '.rgw.gc' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 >>> pgp_num 8 last_change 243 owner 18446744073709551615 pool 16 >>> '.rgw.control' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num >>> 8 pgp_num 8 last_change 244 owner 18446744073709551615 pool 17 >>> '.users.uid' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 >>> pgp_num 8 last_change 246 owner 0 pool 18 '.users.email' rep size 2 >>> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change >>> 248 owner 0 pool 19 '.users' rep size 2 crush_ruleset 0 object_hash >>> rjenkins pg_num 8 pgp_num 8 last_change 250 owner 0 pool 20 '.usage' >>> rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 >>> last_change 256 owner 18446744073709551615 pool 21 '.users.swift' rep >>> size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 >>> last_change 1138 owner 0 >>> >>> Nelson Jeppesen >>> Disney Technology Solutions and Services >>> Phone 206-588-5001 >>> >>> -----Original Message----- >>> From: Gregory Farnum [mailto:greg@xxxxxxxxxxx] >>> Sent: Monday, April 15, 2013 9:34 AM >>> To: Jeppesen, Nelson >>> Cc: ceph-users@xxxxxxxxxxxxxx >>> Subject: Re: ceph -w question >>> >>> "Incomplete" means that there are fewer than the minimum copies of the placement group (by default, half of the requested size, rounded up). >>> In general rebooting one node shouldn't do that unless you've changed your minimum size on the pool, and it does mean that data in those PGs is unaccessible. >>> -Greg >>> Software Engineer #42 @ http://inktank.com | http://ceph.com >>> >>> >>> On Mon, Apr 15, 2013 at 9:01 AM, Jeppesen, Nelson <Nelson.Jeppesen@xxxxxxxxxx> wrote: >>>> When I reboot any node in my prod environment with no activity I see >>>> incomplete pgs. Is that a concern? Does that mean some data is unavailable? >>>> Thank you. >>>> >>>> >>>> >>>> # ceph -v >>>> >>>> ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca) >>>> >>>> >>>> >>>> # ceph -w >>>> >>>> 2013-04-15 08:57:27.712065 mon.0 [INF] pgmap v585220: 4864 pgs: 4443 >>>> active+clean, 1 active+degraded, 420 incomplete; 3177 GB data, 6504 >>>> active+GB used, >>>> 38186 GB / 44691 GB avail; 252/8168154 degraded (0.003%) >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |