I have a fairly vanilla ceph nautilus setup. One node that is the mgr,
mds, and primary mon. Four nodes with 12 8TB osds each, two of which are
backup mons. I am configured for 3 replicas and 2048 pgs, per the
calculator. I recently added a new node with 12 10TB osds. Because of my
3 replicas, I assumed I would get about 40TB available added out of the
120TB raw total. After rebalancing, I only gained about 20TB. Can
anybody tell me why this might be, or if there's anything else I can do
to gain more available space out of my setup?
This information may be helpful to explain my situation. I use this
cluster as storage for bareos backups. Average space used on all 60 osds
is 63% each, with a range of 47%-86%. Only one is at 86% and that one is
reporting nearfull. I have two pools: data and metadata. Both are
sharing the same osds. Each osd node has a separate dedicated SSD for
storing journal.
Output from ceph fs status:
cephfs - 1 clients
======
+------+--------+-------+---------------+-------+-------+
| Rank | State | MDS | Activity | dns | inos |
+------+--------+-------+---------------+-------+-------+
| 0 | active | cfsb0 | Reqs: 0 /s | 997 | 1000 |
+------+--------+-------+---------------+-------+-------+
+-----------------+----------+-------+-------+
| Pool | type | used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 521M | 12.9T |
| cephfs_data | data | 288T | 12.9T |
+-----------------+----------+-------+-------+
+-------------+
| Standby MDS |
+-------------+
+-------------+
Output from ceph -s:
HEALTH_WARN 1 nearfull osd(s); 2 pool(s) nearfull
OSD_NEARFULL 1 nearfull osd(s)
osd.31 is near full
POOL_NEARFULL 2 pool(s) nearfull
pool 'cephfs_data' is nearfull
pool 'cephfs_metadata' is nearfull
[root@cfsb0 ~]# ceph -s
cluster:
id: a5acca4a-22bc-45bd-b0a6-f45e713101ff
health: HEALTH_WARN
1 nearfull osd(s)
2 pool(s) nearfull
services:
mon: 3 daemons, quorum cfsb0,cfsb1,cfsb2 (age 3w)
mgr: cfsb0(active, since 3w)
mds: cephfs:1 {0=cfsb0=up:active}
osd: 60 osds: 60 up (since 2w), 60 in (since 2w)
data:
pools: 2 pools, 2176 pgs
objects: 25.21M objects, 96 TiB
usage: 291 TiB used, 170 TiB / 461 TiB avail
pgs: 2175 active+clean
1 active+clean+scrubbing+deep
Output from df on the backup server as the ceph client:
$ df -h
Filesystem Size Used Avail Use% Mounted on
<snip>
<ceph.mgr.ip>:/backup 110T 97T 13T 89% /mnt/backup
So I'm confused. Do I have 461TB, 300TB, or 110TB usable space out of
504TB raw? Why did I only gain 20TB from adding 120TB of raw osds? Is
there anything I can tune to get more usable space?
I'm happy to provide additional information if it would be helpful.
Thanks.
Seth
--
Seth Galitzer
IT Administrator
Computer Science Department
Carl R. Ice College of Engineering
Kansas State University
http://www.cs.ksu.edu/~sgsax
sgsax@xxxxxxx
785-532-7790
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
