Ok, I understand.
And the same configuration has worked on your NVMe servers? If yes it’s strange, but I think that Ceph developers can tell you why better than me for this
part :-)
Regards,
___________________________________________________________________
PSA Groupe
Loïc Devulder (loic.devulder@xxxxxxxx)
Senior Linux System Engineer / Linux HPC Specialist
___________________________________________________________________
Hi Loic,
i want that kind of setup for the NVMe SSD drives, these drives gives better performance when you use more than one OSD per NVMe SSD drive.
I was just testing the setup on virtual machines, because i installed the same setup on my server having NVMe SSD's.
On Fri, Dec 16, 2016 at 2:07 PM, LOIC DEVULDER <loic.devulder@xxxxxxxx> wrote:
Hi,
I’m not sure that having multiple OSD on one drive is supported.
And also: why do you want this? It’s not good for perfomance and more important for data redundancy.
Regards,
___________________________________________________________________
PSA Groupe
Loïc Devulder (loic.devulder@xxxxxxxx)
Senior Linux System Engineer / Linux HPC Specialist
___________________________________________________________________
Hi,
I was trying the scenario where i have partitioned my drive (/dev/sdb) into 4 (sdb1, sdb2 , sdb3, sdb4) using the sgdisk utility:
# sgdisk -n 1:0:+1024 /dev/sdb -c 1:"ceph journal"
# sgdisk -n 1:0:+1024 /dev/sdb -c 2:"ceph journal"
# sgdisk -n 1:0:+4096 /dev/sdb -c 3:"ceph data"
# sgdisk -n 1:0:+4096 /dev/sdb -c 3:"ceph data"
checked the partition with lsblk and it has created the partitions as expected.
im using the ceph-disk command to create the osd's:
# ceph-disk prepare --cluster ceph /dev/sdb3 /dev/sdb1
prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
prepare_device: Journal /dev/sdb1 was not prepared with ceph-disk. Symlinking directly.
set_data_partition: incorrect partition UUID: 0fc63daf-8483-4772-8e79-3d69d8477de4, expected ['4fbd7e29-9d25-41b8-afd0-5ec00ceff05d', '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', '4fbd7e29-8ae0-4982-bf9d-5a8d867af560', '4fbd7e29-9d25-41b8-afd0-35865ceff05d']
meta-data="" isize=2048 agcount=4, agsize=261760 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1047040, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=65536 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# chown ceph:ceph /dev/sdb*
# ceph-disk activate /dev/sdb3
got monmap epoch 1
added key for osd.2
Created symlink from
/etc/systemd/system/ceph-osd.target.wants/ceph-osd@2.service to
/usr/lib/systemd/system/ceph-osd@.service.
# lsblk
sdb 8:16 0 10G 0 disk
├─sdb1 8:17 0 1G 0 part
├─sdb2 8:18 0 1G 0 part
├─sdb3 8:19 0 4G 0 part /var/lib/ceph/osd/ceph-2
└─sdb4 8:20 0 4G 0 part
# systemctl status ceph-osd@2.service
● ceph-osd@2.service - Ceph object storage daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2016-12-16 13:44:44 IST; 1min 38s ago
Process: 4599 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
Main PID: 4650 (ceph-osd)
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@2.service
└─4650 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph -...
Dec 16 13:46:19 admin ceph-osd[4650]: 2016-12-16 13:46:19.816627 7f1a7cd127...4)
Dec 16 13:46:19 admin ceph-osd[4650]: 2016-12-16 13:46:19.816652 7f1a7cd127...4)
Dec 16 13:46:20 admin ceph-osd[4650]: 2016-12-16 13:46:20.534610 7f1a54b2c7...9)
Dec 16 13:46:20 admin ceph-osd[4650]: 2016-12-16 13:46:20.534638 7f1a54b2c7...9)
Dec 16 13:46:20 admin ceph-osd[4650]: 2016-12-16 13:46:20.816934 7f1a7cd127...1)
Dec 16 13:46:20 admin ceph-osd[4650]: 2016-12-16 13:46:20.816979 7f1a7cd127...1)
Dec 16 13:46:21 admin ceph-osd[4650]: 2016-12-16 13:46:21.817323 7f1a7cd127...8)
Dec 16 13:46:21 admin ceph-osd[4650]: 2016-12-16 13:46:21.817436 7f1a7cd127...8)
Dec 16 13:46:22 admin ceph-osd[4650]: 2016-12-16 13:46:22.826281 7f1a7cd127...7)
Dec 16 13:46:22 admin ceph-osd[4650]: 2016-12-16 13:46:22.826334 7f1a7cd127...7)
Hint: Some lines were ellipsized, use -l to show in full.
but when i reboot the node , the osd doesn't comes up automatically!!
sdb 8:16 0 10G 0 disk
├─sdb1 8:17 0 1G 0 part
├─sdb2 8:18 0 1G 0 part
├─sdb3 8:19 0 4G 0 part
└─sdb4 8:20 0 4G 0 part
]# systemctl status ceph-osd@2.service
● ceph-osd@2.service - Ceph object storage daemon
Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2016-12-16 13:48:52 IST; 2min 6s ago
Process: 2491 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Process: 2446 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
Main PID: 2491 (code=exited, status=1/FAILURE)
Dec 16 13:48:52 admin systemd[1]:
ceph-osd@2.service: main process exited, code=exited, status=1/FAILURE
Dec 16 13:48:52 admin systemd[1]: Unit
ceph-osd@2.service entered failed state.
Dec 16 13:48:52 admin systemd[1]:
ceph-osd@2.service failed.
Dec 16 13:48:52 admin systemd[1]:
ceph-osd@2.service holdoff time over, scheduling restart.
Dec 16 13:48:52 admin systemd[1]: start request repeated too quickly for
ceph-osd@2.service
Dec 16 13:48:52 admin systemd[1]: Failed to start Ceph object storage daemon.
Dec 16 13:48:52 admin systemd[1]: Unit
ceph-osd@2.service entered failed state.
Dec 16 13:48:52 admin systemd[1]:
ceph-osd@2.service failed.
# systemctl start ceph-osd@2.service
Job for ceph-osd@2.service failed because start of the service was attempted too often. See "systemctl status
ceph-osd@2.service" and "journalctl -xe" for details.
To force a start use "systemctl reset-failed
ceph-osd@2.service" followed by "systemctl start
ceph-osd@2.service" again.
But if i do it with the single osd per drive, it works fine....
Anyone else faced the same issue anytime??
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
