On Wed, Sep 19, 2012 at 2:45 PM, Tren Blackburn <tren@xxxxxxxxxxxxxxx> wrote:
> On Wed, Sep 19, 2012 at 2:33 PM, Sage Weil <sage@xxxxxxxxxxx> wrote:
>> On Wed, 19 Sep 2012, Tren Blackburn wrote:
>>> On Wed, Sep 19, 2012 at 2:12 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
>>> > On Wed, Sep 19, 2012 at 2:05 PM, Tren Blackburn <tren@xxxxxxxxxxxxxxx> wrote:
>>> >
>>> >> Greg: It's difficult to tell you that. I'm rsyncing 2 volumes from our
>>> >> filers. Each base directory on each filer mount has approximate 213
>>> >> directories, and then each directory under that has approximately
>>> >> anywhere from 3000 - 5000 directories (very loose approximation here,
>>> >> 850,000 directories per filer mount), and then each of those
>>> >> directories contains files.
>>> >
>>> > Ah, directories are larger ? Sage, do you think they're enough bigger
>>> > to make up that much extra memory usage?
>>> >
>>> >
>>> >> We have many many files here. We're doing this to see how CephFS
>>> >> handles lots of files. We are coming from MooseFS which its master
>>> >> metalogger process eats lots of ram, so we're hoping that Ceph is a
>>> >> bit lighter on us.
>>> >>
>>> >> Sage: The memory the MDS is using is only a cache? There should be no
>>> >> problem restarting the MDS server while activity is going on? I should
>>> >> probably change the limit for the non-active MDS servers first, and
>>> >> then the active one and hope it fails over cleanly?
>>> > Yep, that should work fine, with the obvious caveat that your
>>> > filesystem will become inaccessible if the MDS is down long enough for
>>> > clients to exceed their timeouts (no metadata loss though, if all
>>> > clients remain active until the MDS comes back up).
>>>
>>> I have 3 MDS's (active/standby setup). Shouldn't the MDS fail over to
>>> the other node when I restart the process? I'm not sure what the best
>>> method for just restarting the MDS is, and can it be done without
>>> forcing a fail over?
>>
>> Any running standby ceph-mds daemon will take over when the first one is
>> shut down. Just stop the daemons on the other nodes too if for some
>> reason you care which machine the daemon runs on (Ceph certainly
>> doesn't!).
>>
>> You can restart with
>>
>> /etc/init.d/ceph restart mds
>
> This does not work on gentoo. However "/usr/lib64/ceph/ceph_init.sh -c
> /etc/ceph/ceph.conf restart mds" works fine. I have restarted the
> MDS's, saving the active one for last. I restarted it, and now my
> cluster seems locked.
>
> sap ceph # ceph -s
> health HEALTH_OK
> monmap e1: 3 mons at
> {0=10.87.1.87:6789/0,1=10.87.1.88:6789/0,2=10.87.1.104:6789/0},
> election epoch 38, quorum 0,1,2 0,1,2
> osdmap e25: 192 osds: 192 up, 192 in
> pgmap v10025: 73728 pgs: 73728 active+clean; 48355 MB data, 148 GB
> used, 280 TB / 286 TB avail
> mdsmap e17: 1/1/1 up {0=0=up:clientreplay}, 2 up:standby
>
> What is clientreplay? All IO to ceph has frozen. The mds.0.log shows:
>
> 2012-09-19 14:39:16.315311 7f9d9ad33700 1 mds.0.3 reconnect_done
> 2012-09-19 14:39:17.077926 7f9d9ad33700 1 mds.0.3 handle_mds_map i am
> now mds.0.3
> 2012-09-19 14:39:17.077931 7f9d9ad33700 1 mds.0.3 handle_mds_map
> state change up:reconnect --> up:rejoin
> 2012-09-19 14:39:17.077935 7f9d9ad33700 1 mds.0.3 rejoin_joint_start
> 2012-09-19 14:39:17.354120 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.91:6833/29579
> 2012-09-19 14:39:17.371475 7f9d9ad33700 1 mds.0.3 rejoin_done
> 2012-09-19 14:39:17.736378 7f9d9ad33700 1 mds.0.3 handle_mds_map i am
> now mds.0.3
> 2012-09-19 14:39:17.736383 7f9d9ad33700 1 mds.0.3 handle_mds_map
> state change up:rejoin --> up:clientreplay
> 2012-09-19 14:39:17.736385 7f9d9ad33700 1 mds.0.3 recovery_done --
> successful recovery!
> 2012-09-19 14:39:17.748784 7f9d9ad33700 1 mds.0.3 clientreplay_start
> 2012-09-19 14:39:17.761751 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.104:6831/11000
> 2012-09-19 14:39:17.763888 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.95:6818/18116
> 2012-09-19 14:39:17.775943 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.98:6812/7539
> 2012-09-19 14:39:17.786640 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.104:6819/10452
> 2012-09-19 14:39:17.801893 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.98:6821/7893
> 2012-09-19 14:39:17.827436 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.93:6827/3894
> 2012-09-19 14:39:17.837971 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.89:6809/28294
> 2012-09-19 14:39:17.839187 7f9d9ad33700 0 mds.0.3 ms_handle_connect
> on 10.87.1.99:6833/23283
>
> How long does this "clientreplay"stage take? It doesn't seem like the
> process is actually doing anything.
>
Woo! Love responding to my own posts. This was pretty exciting. I
ended up restarting the mon/mds services on the node that was the
active mds. The mds restarted, and came out of the "clientreplay"
state, but the mon crashed on restart. I was able to start just the
mon process and now my cluster is happy.
Here is the log of the mon crash I received while restarting it.
http://pastebin.com/fGriuEDQ
Thanks!
t.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
