On Mon, Jul 25, 2016 at 7:38 PM, Oliver Dzombic <info@xxxxxxxxxxxxxxxxx> wrote:
> Hi,
>
> currently some productive stuff is down, because it can not be accessed
> through cephfs.
>
> Client server restart, did not help.
> Cluster restart, did not help.
>
> Only ONE directory inside cephfs has this issue.
>
> All other directories are working fine.
What's the full output of "ceph -s"?
>
>
> MDS Server: Kernel 4.5.4
> client server: Kernel 4.5.4
> ceph version 10.2.2
>
> # ceph fs dump
> dumped fsmap epoch 92
> e92
> enable_multiple, ever_enabled_multiple: 0,0
> compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file
> layout v2}
>
> Filesystem 'ceph-gen2' (2)
> fs_name ceph-gen2
> epoch 92
> flags 0
> created 2016-06-11 21:53:02.142649
> modified 2016-06-14 11:09:16.783356
> tableserver 0
> root 0
> session_timeout 60
> session_autoclose 300
> max_file_size 1099511627776
> last_failure 0
> last_failure_osd_epoch 2164
> compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=file
> layout v2}
> max_mds 1
> in 0
> up {0=234109}
> failed
> damaged
> stopped
> data_pools 4
> metadata_pool 5
> inline_data disabled
> 234109: 10.0.0.11:6801/22255 'cephmon1' mds.0.89 up:active seq 250
>
>
> Standby daemons:
>
> 204171: 10.0.0.13:6800/19434 'cephmon3' mds.-1.0 up:standby seq 1
>
>
> ceph --admin-daemon ceph-mds.cephmon1.asok dump_ops_in_flight
> {
> "ops": [
> {
> "description": "client_request(client.204153:432 getattr
> pAsLsXsFs #10000001432 2016-07-25 21:57:30.697894 RETRY=2)",
> "initiated_at": "2016-07-26 04:24:05.528832",
> "age": 816.092461,
> "duration": 816.092528,
> "type_data": [
> "failed to rdlock, waiting",
> "client.204153:432",
> "client_request",
> {
> "client": "client.204153",
> "tid": 432
> },
> [
> {
> "time": "2016-07-26 04:24:05.528832",
> "event": "initiated"
> },
> {
> "time": "2016-07-26 04:24:07.613779",
> "event": "failed to rdlock, waiting"
> }
> ]
> ]
> }
> ],
> "num_ops": 1
> }
>
>
> 2016-07-26 04:32:09.355503 7ffb331ca700 0 log_channel(cluster) log
> [WRN] : 1 slow requests, 1 included below; oldest blocked for >
> 483.826590 secs
>
> 2016-07-26 04:32:09.355531 7ffb331ca700 0 log_channel(cluster) log
> [WRN] : slow request 483.826590 seconds old, received at 2016-07-26
> 04:24:05.528832: client_request(client.204153:432 getattr pAsLsXsFs
> #10000001432 2016-07-25 21:57:30.697894 RETRY=2) currently failed to
> rdlock, waiting
>
>
> Any idea ? :(
>
> --
> Mit freundlichen Gruessen / Best regards
>
> Oliver Dzombic
> IP-Interactive
>
> mailto:info@xxxxxxxxxxxxxxxxx
>
> Anschrift:
>
> IP Interactive UG ( haftungsbeschraenkt )
> Zum Sonnenberg 1-3
> 63571 Gelnhausen
>
> HRB 93402 beim Amtsgericht Hanau
> Geschäftsführung: Oliver Dzombic
>
> Steuer Nr.: 35 236 3622 1
> UST ID: DE274086107
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
