Hi Jesper & Lars,
Thnx for your answer.
c) is indeed the option with the devices being used for block.db
Adding an extra NVMe doesn't really seem necessary, since there is no space issue... (Contrary... The only advantage is the impact of a defect NVMe disk). The performance of the NVMe's shouldn't be a problem too as they should be more than fast...
What Lars is saying might be an option too:
Going for option c) and using the remaining space for RGW metadata pool.
This would result in +/- 500GB/host being used as fast storage for the OSDs and 2.7TB being used for the RGW metadata pool...
I don't find any calculations for the size of the RGW metadata pool yet, but we're storing very limited metadata information, so I can't imaging that 27TB of NVMe disk space wouldn't be sufficient for that pool...
So, the final pick will be a) combined with c). I'm just going to search further on the metadata size calculation to be sure that there is no issue related to the sizing.
Regards,
Kristof
Op ma 18 nov. 2019 om 08:14 schreef <jesper@xxxxxxxx>:
Is c) the bcache solution?real life experience - unless you are really beating an enterprise ssd with writes - they last very,very long and even when failure happens- you can typically see it by the wear levels in smart months before.I would go for c) but if possible add one more nvme to each host - we have a 9-hdd+3-ssd scenario here.Jesper
Sent from myMail for iOS
Monday, 18 November 2019, 07.49 +0100 from kristof.coucke@xxxxxxxxx <kristof.coucke@xxxxxxxxx>:Hi all,Thanks for the feedback.Though, just to be sure:1. There is no 30GB limit if I understand correctly for the RocksDB size. If metadata crosses that barrier, the L4 part will spillover to the primary device? Or will it just move the RocksDB completely? Or will it just stop and indicate it's full?2. Since the WAL will also be written to that device, I assume a few additional GB's is still usefull...With my setup (13x 14TB + 2 NVMe of 1.6TB / host, 10 hosts) I have multiple possible scenario's:- Assigning 35GB of space of the NVMe disk (30GB for DB, 5 spare) would result in only 455GB being used (13 x 35GB). This is a pity, since I have 3.2TB of NVMe disk space...Options line-up:Option a: Not using the NVMe for block.db storage, but as RGW metadata pool.Advantages:- Impact of 1 defect NVMe is limited.- Fast storage for the metadata pool.Disadvantage:- RocksDB for each OSD is on the primary disk, resulting in slower performance of each OSD.Option b: Hardware mirror of the NVMe driveAdvantages:- Impact of 1 defect NVMe is limited- Fast KV lookup for each OSDDisadvantage:- I/O to NVMe is serialized for all OSDs on 1 host. Though the NVMe are fast, I imagine that there still is an impact.- 1 TB of NVMe is not used / hostOption c: Split the NVMe's accross the OSDAdvantages:- Fast RockDB access - up to L3 (assuming spillover does it job)Disadvantage:- 1 defect NVMe impacts max 7 OSDs (1 NVMe assigned to 7 or 6 OSD daemons per host)- 2.7TB of NVMe space not used per hostOption d: 1 NVMe disk for OSDs, 1 for RGW metadata poolAdvantages:- Fast RockDB access - up to L3- Fast RGW metadata pool (though limited to 5,3TB (raw pool size will be 16TB, divided by 3 due to replication) I assume this already gives some possibilitiesDisadvantages:- 1 defect NVMe might impact a complete host (all OSDs might be using it for the RockDB storage)- 1 TB of NVMe is not usedThough menu to choose from, each with it possibilities... The initial idea was too assign 200GB per OSD of the NVMe space per OSD, but this would result in a lot of unused space. I don't know if there is anything on the roadmap to adapt the RocksDB sizing to make better use of the available NVMe disk space.With all the information, I would assume that the best option would be option A. Since we will be using erasure coding for the RGW data pool (k=6, m=3), the impact of a defect NVMe would be too significant. The other alternative would be option b, but then again we would be dealing with HW raid which is against all Ceph design rules.Any other options or (dis)advantages I missed? Or any other opinions to choose another option?Regards,KristofOp vr 15 nov. 2019 om 18:22 schreef <vitalif@xxxxxxxxxx>:Use 30 GB for all OSDs. Other values are pointless, because
https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing
You can use the rest of free NVMe space for bcache - it's much better
than just allocating it for block.db.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
