http://tracker.ceph.com/issues/13990 was created by a colleague of mine from an issue that was affecting us in production. When 0.94.8 was released
with the fix, I immediately deployed a test cluster on 0.94.7, reproduced this issue, upgraded to 0.94.8, and tested the fix. It worked beautifully.
I suspect the issue you're seeing is that the clean-up only occurs when new osdmaps are generated, so as long as nothing is changing you'll continue to see lots of stale maps cached. We delete RBD snapshots all the time in our production use case, which updates the osdmap, so I did that in my test cluster and watched the map cache on one of the OSDs. Sure enough, after a while the cache was pruned down to the expected size.
Over time I imagine you'll see things settle, but it may take a while if you don't update the osdmap frequently.
I suspect the issue you're seeing is that the clean-up only occurs when new osdmaps are generated, so as long as nothing is changing you'll continue to see lots of stale maps cached. We delete RBD snapshots all the time in our production use case, which updates the osdmap, so I did that in my test cluster and watched the map cache on one of the OSDs. Sure enough, after a while the cache was pruned down to the expected size.
Over time I imagine you'll see things settle, but it may take a while if you don't update the osdmap frequently.
 |
Steve Taylor |
Senior Software Engineer |
StorageCraft
Technology Corporation 380 Data Drive Suite 300 | Draper | Utah | 84020 Office: 801.871.2799 | |
| If you are not the intended recipient of this message or received it erroneously, please notify the sender and delete it, together with any attachments, and be advised that any dissemination or copying of this message is prohibited. |
________________________________________
From: ceph-users [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Dan Van Der Ster [daniel.vanderster@xxxxxxx]
Sent: Wednesday, September 14, 2016 3:45 AM
To: ceph-users@xxxxxxxx
Subject: Cleanup old osdmaps after #13990 fix applied
Hi,
We've just upgraded to 0.94.9, so I believe this issue is fixed:
http://tracker.ceph.com/issues/13990
AFAICT "resolved" means the number of osdmaps saved on each OSD will not grow unboundedly anymore.
However, we have many OSDs with loads of old osdmaps, e.g.:
# pwd
/var/lib/ceph/osd/ceph-257/current/meta
# find . -name 'osdmap*' | wc -l
112810
(And our maps are ~1MB, so this is >100GB per OSD).
Is there a solution to remove these old maps?
Cheers,
Dan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
From: ceph-users [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Dan Van Der Ster [daniel.vanderster@xxxxxxx]
Sent: Wednesday, September 14, 2016 3:45 AM
To: ceph-users@xxxxxxxx
Subject: Cleanup old osdmaps after #13990 fix applied
Hi,
We've just upgraded to 0.94.9, so I believe this issue is fixed:
http://tracker.ceph.com/issues/13990
AFAICT "resolved" means the number of osdmaps saved on each OSD will not grow unboundedly anymore.
However, we have many OSDs with loads of old osdmaps, e.g.:
# pwd
/var/lib/ceph/osd/ceph-257/current/meta
# find . -name 'osdmap*' | wc -l
112810
(And our maps are ~1MB, so this is >100GB per OSD).
Is there a solution to remove these old maps?
Cheers,
Dan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
