> first of all, I'd still recommend to use the orchestrator to deploy OSDs. Building > OSDs manually and then adopt them is redundant. Or do you have issues with > the drivegroups? I am having to do it this way because I couldn't find any doco on how to specify a separate DB/WAL device when deploying OSDs using the orchestrator. If there is such a command I agree it would be a better choice. Ideally I would use the commands to simply move the DB of my existing orchestrator deployed ODSs to the SSD, but when I tried that command it broke my OSD and I had to delete it and leave he cluster in a degraded state until it had recovered. I find it very stressful when I get out of my depth with problems like that, so I gave up on that idea and am doing the remove, redploy, adopt method, which is working, but VERY slow. > I don't have *the* solution but you could try to disable the mclock scheduler > [1] which is the default since Quincy. Maybe that will speed up things? There > have been reports in the list about some unwanted or at least unexpected > behavior. I did try this to try and speed up my rebalances, but it didn't seem to make much difference. I haven't tried it to see what difference it makes to scrubbing. > As for the "not (deep-)scrubbed in time" messages, there seems to be > progress (in your ceph status), but depending on the drive utilization you > could increase the number of scrubs per OSD (osd_max_scrubs). There are lot of scrubs running, this morning after my rebalance finally completed it has 22 scrubbing, 6 deep scrubbing (across 28 OSDs). This has fallen from the number it was running yesterday when the rebalance was still happening (38/9). I believe if I kick the cluster by taking a host into maintenance and back the numbers will jump up again. The trouble is I don't know how to tell if a scrub is actually achieving something, stuck or restarting over and over. My current ceph pg dump is: https://pastebin.com/AQhNKSBN and if I run it again a few minutes later: https://pastebin.com/yfREzJ4s I see evidence of scrubs not working because some of my OSD logs look like this: 2023-11-01T20:51:08.668+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:11.658+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:19.565+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:20.516+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:22.463+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:24.488+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:29.474+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:31.484+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:34.455+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:39.444+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:51:42.473+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:51:44.510+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:51:46.491+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:51:47.465+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:51:49.443+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:51:51.439+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:51:53.388+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:00.345+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:02.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:03.452+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:10.421+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:11.436+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 deep-scrub starts 2023-11-01T20:52:12.465+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:13.470+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:14.468+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d deep-scrub starts 2023-11-01T20:52:17.512+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:20.507+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:22.428+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:23.438+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:52:24.444+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:28.461+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:45.551+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:52:46.593+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:52:48.595+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:52:52.488+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b deep-scrub starts 2023-11-01T20:52:55.504+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:52:58.519+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:52:59.477+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts 2023-11-01T20:53:01.505+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.17 scrub starts 2023-11-01T20:53:03.467+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1d9 scrub starts 2023-11-01T20:53:06.406+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 5.9b scrub starts 2023-11-01T20:53:10.446+0000 7f3be0b27700 0 log_channel(cluster) log [DBG] : 5.65 scrub starts 2023-11-01T20:53:13.470+0000 7f3be1328700 0 log_channel(cluster) log [DBG] : 6.2d scrub starts 2023-11-01T20:53:15.521+0000 7f3bdfb25700 0 log_channel(cluster) log [DBG] : 5.1ac scrub starts There is no scrub OK type messages. I notice these logs were from yesterday, and currently there is nothing being logged for these OSDs. If I was to "kick the cluster" these scrub logs would probably start showing up again. When this is happening the PGs mentioned seem to stay in a "scrub queued" state. Any light you could shine my way would be appreciated. Thanks, Martin _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |