Re: cephfs snapshot mirror peer_bootstrap import hung

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Venky,

Thank you very much.

Anantha

-----Original Message-----
From: Venky Shankar <vshankar@xxxxxxxxxx> 
Sent: Monday, August 7, 2023 5:23 PM
To: Adiga, Anantha <anantha.adiga@xxxxxxxxx>
Cc: ceph-users@xxxxxxx
Subject: Re:  Re: cephfs snapshot mirror peer_bootstrap import hung

Hi Anantha,

On Tue, Aug 8, 2023 at 1:59 AM Adiga, Anantha <anantha.adiga@xxxxxxxxx> wrote:
>
> Hi Venky,
>
>
>
> Could this be the reason that the peer-bootstrap import is hanging?  how do I upgrade cephfs-mirror to Quincy?

I was on leave yesterday -- will have a look at the log and update.

>
> root@fl31ca104ja0201:/# cephfs-mirror --version
>
> ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)
>
> root@fl31ca104ja0201:/# ceph version
>
> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
>
> root@fl31ca104ja0201:/#
>
>
>
>
>
> Thank you,
>
> Anantha
>
> From: Adiga, Anantha
> Sent: Monday, August 7, 2023 11:21 AM
> To: 'Venky Shankar' <vshankar@xxxxxxxxxx>; 'ceph-users@xxxxxxx' <ceph-users@xxxxxxx>
> Subject: RE:  Re: cephfs snapshot mirror peer_bootstrap import hung
>
>
>
> Hi Venky,
>
>
>
> I tried on another secondary Quincy cluster and it is the same problem. The peer_bootstrap mport  command hangs.
>
>
>
>
>
> root@fl31ca104ja0201:/# ceph fs  snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ==
>
> ……
>
> …….
>
> ..command does not complete..waits here
>
> ^C  to exit.
>
> Thereafter some commands do not complete…
>
> root@fl31ca104ja0201:/# ceph -s
>
>   cluster:
>
>     id:     d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
>     health: HEALTH_OK
>
>
>
>   services:
>
>     mon:           3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 2d)
>
>     mgr:           fl31ca104ja0201.kkoono(active, since 3d), standbys: fl31ca104ja0202, fl31ca104ja0203
>
>     mds:           1/1 daemons up, 2 standby
>
>     osd:           44 osds: 44 up (since 2d), 44 in (since 5w)
>
>     cephfs-mirror: 1 daemon active (1 hosts)
>
>     rgw:           3 daemons active (3 hosts, 1 zones)
>
>
>
>   data:
>
>     volumes: 1/1 healthy
>
>     pools:   25 pools, 769 pgs
>
>     objects: 614.40k objects, 1.9 TiB
>
>     usage:   2.9 TiB used, 292 TiB / 295 TiB avail
>
>     pgs:     769 active+clean
>
>
>
>   io:
>
>     client:   32 KiB/s rd, 0 B/s wr, 33 op/s rd, 1 op/s wr
>
>
>
> root@fl31ca104ja0201:/#
>
> root@fl31ca104ja0201:/# ceph fs status cephfs
>
> This command also waits. ……
>
>
>
> I have attached the mgr log
>
> root@fl31ca104ja0201:/# ceph service status
>
> {
>
>     "cephfs-mirror": {
>
>         "5306346": {
>
>             "status_stamp": "2023-08-07T17:35:56.884907+0000",
>
>             "last_beacon": "2023-08-07T17:45:01.903540+0000",
>
>             "status": {
>
>                 "status_json": "{\"1\":{\"name\":\"cephfs\",\"directory_count\":0,\"peers\":{}}}"
>
>             }
>
>         }
>
>
>
> Quincy secondary cluster
>
>
>
> root@a001s008-zz14l47008:/# ceph mgr module enable mirroring
>
> root@a001s008-zz14l47008:/# ceph fs authorize cephfs client.mirror_remote / rwps
>
> [client.mirror_remote]
>
>         key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==
>
> root@a001s008-zz14l47008:/# ceph auth get client.mirror_remote
>
> [client.mirror_remote]
>
>         key = AQCIFtFkI+SLNhAAWmez2DJpH9eGrbxA9efdoA==
>
>         caps mds = "allow rwps fsname=cephfs"
>
>         caps mon = "allow r fsname=cephfs"
>
>         caps osd = "allow rw tag cephfs data=cephfs"
>
> root@a001s008-zz14l47008:/#
>
> root@a001s008-zz14l47008:/# ceph fs snapshot mirror peer_bootstrap create cephfs client.mirror_remote shgR-site
>
> {"token": "eyJmc2lkIjogIjJlYWMwZWEwLTYwNDgtNDQ0Zi04NGIyLThjZWVmZWQyN2E1YiIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJzaGdSLXNpdGUiLCAia2V5IjogIkFRQ0lGdEZrSStTTE5oQUFXbWV6MkRKcEg5ZUdyYnhBOWVmZG9BPT0iLCAibW9uX2hvc3QiOiAiW3YyOjEwLjIzOS4xNTUuMTg6MzMwMC8wLHYxOjEwLjIzOS4xNTUuMTg6Njc4OS8wXSBbdjI6MTAuMjM5LjE1NS4xOTozMzAwLzAsdjE6MTAuMjM5LjE1NS4xOTo2Nzg5LzBdIFt2MjoxMC4yMzkuMTU1LjIwOjMzMDAvMCx2MToxMC4yMzkuMTU1LjIwOjY3ODkvMF0ifQ=="}
>
> root@a001s008-zz14l47008:/#
>
>
>
> Thank you,
>
> Anantha
>
>
>
> From: Adiga, Anantha
> Sent: Friday, August 4, 2023 11:55 AM
> To: Venky Shankar <vshankar@xxxxxxxxxx>; ceph-users@xxxxxxx
> Subject: RE:  Re: cephfs snapshot mirror peer_bootstrap import hung
>
>
>
> Hi Venky,
>
>
>
> Thank you so much for the guidance. Attached is the mgr log.
>
>
>
> Note: the 4th node in the primary cluster has smaller capacity  drives, the other 3 nodes have the larger capacity drives.
>
> 32    ssd    6.98630   1.00000  7.0 TiB   44 GiB   44 GiB   183 KiB  148 MiB  6.9 TiB  0.62  0.64   40      up          osd.32
>
> -7          76.84927         -   77 TiB  652 GiB  648 GiB    20 MiB  3.0 GiB   76 TiB  0.83  0.86    -              host fl31ca104ja0203
>
>   1    ssd    6.98630   1.00000  7.0 TiB   73 GiB   73 GiB   8.0 MiB  333 MiB  6.9 TiB  1.02  1.06   54      up          osd.1
>
>   4    ssd    6.98630   1.00000  7.0 TiB   77 GiB   77 GiB   1.1 MiB  174 MiB  6.9 TiB  1.07  1.11   55      up          osd.4
>
>   7    ssd    6.98630   1.00000  7.0 TiB   47 GiB   47 GiB   140 KiB  288 MiB  6.9 TiB  0.66  0.68   51      up          osd.7
>
> 10    ssd    6.98630   1.00000  7.0 TiB   75 GiB   75 GiB   299 KiB  278 MiB  6.9 TiB  1.05  1.09   44      up          osd.10
>
> 13    ssd    6.98630   1.00000  7.0 TiB   94 GiB   94 GiB  1018 KiB  291 MiB  6.9 TiB  1.31  1.36   72      up          osd.13
>
> 16    ssd    6.98630   1.00000  7.0 TiB   31 GiB   31 GiB   163 KiB  267 MiB  7.0 TiB  0.43  0.45   49      up          osd.16
>
> 19    ssd    6.98630   1.00000  7.0 TiB   14 GiB   14 GiB   756 KiB  333 MiB  7.0 TiB  0.20  0.21   50      up          osd.19
>
> 22    ssd    6.98630   1.00000  7.0 TiB  105 GiB  104 GiB   1.3 MiB  313 MiB  6.9 TiB  1.46  1.51   48      up          osd.22
>
> 25    ssd    6.98630   1.00000  7.0 TiB   17 GiB   16 GiB   257 KiB  272 MiB  7.0 TiB  0.23  0.24   45      up          osd.25
>
> 28    ssd    6.98630   1.00000  7.0 TiB   72 GiB   72 GiB   6.1 MiB  180 MiB  6.9 TiB  1.01  1.05   43      up          osd.28
>
> 31    ssd    6.98630   1.00000  7.0 TiB   47 GiB   46 GiB   592 KiB  358 MiB  6.9 TiB  0.65  0.68   56      up          osd.31
>
> -9          64.04089         -   64 TiB  728 GiB  726 GiB    17 MiB  1.8 GiB   63 TiB  1.11  1.15    -              host fl31ca104ja0302
>
> 33    ssd    5.82190   1.00000  5.8 TiB   65 GiB   65 GiB   245 KiB  144 MiB  5.8 TiB  1.09  1.13   47      up          osd.33
>
> 34    ssd    5.82190   1.00000  5.8 TiB   14 GiB   14 GiB   815 KiB   83 MiB  5.8 TiB  0.24  0.25   55      up          osd.34
>
> 35    ssd    5.82190   1.00000  5.8 TiB   77 GiB   77 GiB   224 KiB  213 MiB  5.7 TiB  1.30  1.34   44      up          osd.35
>
> 36    ssd    5.82190   1.00000  5.8 TiB  117 GiB  117 GiB   8.5 MiB  284 MiB  5.7 TiB  1.96  2.03   52      up          osd.36
>
> 37    ssd    5.82190   1.00000  5.8 TiB   58 GiB   58 GiB   501 KiB  132 MiB  5.8 TiB  0.98  1.01   40      up          osd.37
>
> 38    ssd    5.82190   1.00000  5.8 TiB  123 GiB  123 GiB   691 KiB  266 MiB  5.7 TiB  2.07  2.14   73      up          osd.38
>
> 39    ssd    5.82190   1.00000  5.8 TiB   77 GiB   77 GiB   609 KiB  193 MiB  5.7 TiB  1.30  1.34   62      up          osd.39
>
> 40    ssd    5.82190   1.00000  5.8 TiB   77 GiB   77 GiB   262 KiB  148 MiB  5.7 TiB  1.29  1.34   55      up          osd.40
>
> 41    ssd    5.82190   1.00000  5.8 TiB   44 GiB   44 GiB   4.4 MiB  140 MiB  5.8 TiB  0.75  0.77   44      up          osd.41
>
> 42    ssd    5.82190   1.00000  5.8 TiB   45 GiB   45 GiB   886 KiB  135 MiB  5.8 TiB  0.75  0.78   47      up          osd.42
>
> 43    ssd    5.82190   1.00000  5.8 TiB   28 GiB   28 GiB   187 KiB  104 MiB  5.8 TiB  0.48  0.49   58      up          osd.43
>
>
>
> [Also: Yesterday I had two cfs-mirror running one on fl31ca104ja0201 and fl31ca104ja0302. The cfs-mirror on fl31ca104ja0201 was stopped. When the  import token was run on fl31ca104ja0302, the cfs-mirror log was active. Just in case it is useful I have attached that log (cfsmirror-container.log) as well. ]
>
>
>
> How can I list the token on the target cluster after running the create peer_bootstrap command?
>
>
>
> Here is today’s status with your suggestion:
>
> There is only one cfs-mirror daemon running now. It is on fl31ca104ja0201 node.
>
>
>
> root@fl31ca104ja0201:/# ceph -s
>
>   cluster:
>
>     id:     d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
>     health: HEALTH_OK
>
>
>
>   services:
>
>     mon:           3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 7m)
>
>     mgr:           fl31ca104ja0201.kkoono(active, since 13m), standbys: fl31ca104ja0202, fl31ca104ja0203
>
>     mds:           1/1 daemons up, 2 standby
>
>     osd:           44 osds: 44 up (since 7m), 44 in (since 4w)
>
>     cephfs-mirror: 1 daemon active (1 hosts)
>
>     rgw:           3 daemons active (3 hosts, 1 zones)
>
>
>
>   data:
>
>     volumes: 1/1 healthy
>
>     pools:   25 pools, 769 pgs
>
>     objects: 614.40k objects, 1.9 TiB
>
>     usage:   2.8 TiB used, 292 TiB / 295 TiB avail
>
>     pgs:     769 active+clean
>
>
>
>   io:
>
>     client:   32 MiB/s rd, 0 B/s wr, 57 op/s rd, 1 op/s wr
>
>
>
> root@fl31ca104ja0201:/#
>
> root@fl31ca104ja0201:/#
>
> root@fl31ca104ja0201:/# ceph tell mgr.fl31ca104ja0201.kkoono config set debug_mgr 20
>
> {
>
>     "success": ""
>
> }
>
> root@fl31ca104ja0201:/# ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaWxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwgInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUuNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6MzMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0=
>
> ^CInterrupted
>
>
>
> Ctrl-C after 15  min. Once the command is run, the health status goes to WARN .
>
>
>
> root@fl31ca104ja0201:/# ceph -s
>
>   cluster:
>
>     id:     d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
>     health: HEALTH_WARN
>
>             6 slow ops, oldest one blocked for 1095 sec, mon.fl31ca104ja0203 has slow ops
>
>
>
>   services:
>
>     mon:           3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 30m)
>
>     mgr:           fl31ca104ja0201.kkoono(active, since 35m), standbys: fl31ca104ja0202, fl31ca104ja0203
>
>     mds:           1/1 daemons up, 2 standby
>
>     osd:           44 osds: 44 up (since 29m), 44 in (since 4w)
>
>     cephfs-mirror: 1 daemon active (1 hosts)
>
>     rgw:           3 daemons active (3 hosts, 1 zones)
>
>
>
>   data:
>
>     volumes: 1/1 healthy
>
>     pools:   25 pools, 769 pgs
>
>     objects: 614.40k objects, 1.9 TiB
>
>     usage:   2.8 TiB used, 292 TiB / 295 TiB avail
>
>     pgs:     769 active+clean
>
>
>
>   io:
>
>     client:   67 KiB/s rd, 0 B/s wr, 68 op/s rd, 21 op/s wr
>
>
>
>
>
> -----Original Message-----
> From: Venky Shankar <vshankar@xxxxxxxxxx>
> Sent: Thursday, August 3, 2023 11:03 PM
> To: Adiga, Anantha <anantha.adiga@xxxxxxxxx>
> Cc: ceph-users@xxxxxxx
> Subject:  Re: cephfs snapshot mirror peer_bootstrap import hung
>
>
>
> Hi Anantha,
>
>
>
> On Fri, Aug 4, 2023 at 2:27 AM Adiga, Anantha <anantha.adiga@xxxxxxxxx> wrote:
>
> >
>
> > Hi
>
> >
>
> > Could you please  provide guidance on how to diagnose this issue:
>
> >
>
> > In this case, there are two  Ceph clusters: cluster A, 4 nodes and cluster B, 3 node, in different locations.  Both are already running RGW multi-site,  A is master.
>
> >
>
> > Cephfs snapshot mirroring is being configured on the clusters.  Cluster A  is the primary, cluster B is the peer. Cephfs snapshot mirroring is being configured. The bootstrap import  step on the primary node hangs.
>
> >
>
> > On the target cluster :
>
> > ---------------------------
>
> > "version": "16.2.5",
>
> >     "release": "pacific",
>
> >     "release_type": "stable"
>
> >
>
> > root@cr21meg16ba0101:/# ceph fs snapshot mirror peer_bootstrap create
>
> > cephfs client.mirror_remote flex2-site
>
> > {"token":
>
> > "eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJma
>
> > Wxlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiw
>
> > gInNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd
>
> > 1h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTU
>
> > uNzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6M
>
> > zMwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0="}
>
>
>
> Seems fine uptil here.
>
>
>
> > root@cr21meg16ba0101:/var/run/ceph#
>
> >
>
> > On the source cluster:
>
> > ----------------------------
>
> > "version": "17.2.6",
>
> >     "release": "quincy",
>
> >     "release_type": "stable"
>
> >
>
> > root@fl31ca104ja0201:/# ceph -s
>
> >   cluster:
>
> >     id:     d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e
>
> >     health: HEALTH_OK
>
> >
>
> >   services:
>
> >     mon:           3 daemons, quorum fl31ca104ja0202,fl31ca104ja0203,fl31ca104ja0201 (age 111m)
>
> >     mgr:           fl31ca104ja0201.nwpqlh(active, since 11h), standbys: fl31ca104ja0203, fl31ca104ja0202
>
> >     mds:           1/1 daemons up, 2 standby
>
> >     osd:           44 osds: 44 up (since 111m), 44 in (since 4w)
>
> >     cephfs-mirror: 1 daemon active (1 hosts)
>
> >     rgw:           3 daemons active (3 hosts, 1 zones)
>
> >
>
> >   data:
>
> >     volumes: 1/1 healthy
>
> >     pools:   25 pools, 769 pgs
>
> >     objects: 614.40k objects, 1.9 TiB
>
> >     usage:   2.8 TiB used, 292 TiB / 295 TiB avail
>
> >     pgs:     769 active+clean
>
> >
>
> > root@fl31ca104ja0302:/# ceph mgr module enable mirroring module
>
> > 'mirroring' is already enabled root@fl31ca104ja0302:/# ceph fs
>
> > snapshot mirror peer_bootstrap import cephfs
>
> > eyJmc2lkIjogImE2ZjUyNTk4LWU1Y2QtNGEwOC04NDIyLTdiNmZkYjFkNWRiZSIsICJmaW
>
> > xlc3lzdGVtIjogImNlcGhmcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcmVtb3RlIiwg
>
> > InNpdGVfbmFtZSI6ICJmbGV4Mi1zaXRlIiwgImtleSI6ICJBUUNmd01sa005MHBMQkFBd1
>
> > h0dnBwOGowNEl2Qzh0cXBBRzliQT09IiwgIm1vbl9ob3N0IjogIlt2MjoxNzIuMTguNTUu
>
> > NzE6MzMwMC8wLHYxOjE3Mi4xOC41NS43MTo2Nzg5LzBdIFt2MjoxNzIuMTguNTUuNzM6Mz
>
> > MwMC8wLHYxOjE3Mi4xOC41NS43Mzo2Nzg5LzBdIn0=
>
>
>
> Going by your description, I'm guessing this is the command that hangs? If that's the case, set `debug_mgr=20`, repeat the token import step and share the ceph-mgr log. Also note that you can check the mirror daemon status as detailed in
>
>
>
>         https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#mirror-daemon-status
>
>
>
> >
>
> >
>
> > root@fl31ca104ja0302:/var/run/ceph# ceph --admin-daemon
>
> > /var/run/ceph/ceph-client.cephfs-mirror.fl31ca104ja0302.sypagt.7.94083135960976.asok status {
>
> >     "metadata": {
>
> >         "ceph_sha1": "d7ff0d10654d2280e08f1ab989c7cdf3064446a5",
>
> >         "ceph_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)",
>
> >         "entity_id": "cephfs-mirror.fl31ca104ja0302.sypagt",
>
> >         "hostname": "fl31ca104ja0302",
>
> >         "pid": "7",
>
> >         "root": "/"
>
> >     },
>
> >     "dentry_count": 0,
>
> >     "dentry_pinned_count": 0,
>
> >     "id": 5194553,
>
> >     "inst": {
>
> >         "name": {
>
> >             "type": "client",
>
> >             "num": 5194553
>
> >         },
>
> >         "addr": {
>
> >             "type": "v1",
>
> >             "addr": "10.45.129.5:0",
>
> >             "nonce": 2497002034
>
> >         }
>
> >     },
>
> >     "addr": {
>
> >         "type": "v1",
>
> >         "addr": "10.45.129.5:0",
>
> >         "nonce": 2497002034
>
> >     },
>
> >     "inst_str": "client.5194553 10.45.129.5:0/2497002034",
>
> >     "addr_str": "10.45.129.5:0/2497002034",
>
> >     "inode_count": 1,
>
> >     "mds_epoch": 118,
>
> >     "osd_epoch": 6266,
>
> >     "osd_epoch_barrier": 0,
>
> >     "blocklisted": false,
>
> >     "fs_name": "cephfs"
>
> > }
>
> >
>
> > root@fl31ca104ja0302:/home/general# docker logs
>
> > ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e-cephfs-mirror-fl31ca104ja030
>
> > 2-sypagt --tail  10 debug 2023-08-03T05:24:27.413+0000 7f8eb6fc0280  0
>
> > ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
>
> > (stable), process cephfs-mirror, pid 7 debug
>
> > 2023-08-03T05:24:27.413+0000 7f8eb6fc0280  0 pidfile_write: ignore
>
> > empty --pid-file debug 2023-08-03T05:24:27.445+0000 7f8eb6fc0280  1
>
> > mgrc service_daemon_register cephfs-mirror.5184622 metadata
>
> > {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.6
>
> > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy
>
> > (stable),ceph_version_short=17.2.6,container_hostname=fl31ca104ja0302,
>
> > container_image=quay.io/ceph/ceph@sha256:af79fedafc42237b7612fe2d18a9c
>
> > 64ca62a0b38ab362e614ad671efa4a0547e,cpu=Intel(R) Xeon(R) Gold 6252 CPU
>
> > @ 2.10GHz,distro=centos,distro_description=CentOS Stream
>
> > 8,distro_version=8,hostname=fl31ca104ja0302,id=fl31ca104ja0302.sypagt,
>
> > instance_id=5184622,kernel_description=#82-Ubuntu SMP Tue Jun 6
>
> > 23:10:23 UTC
>
> > 2023,kernel_version=5.15.0-75-generic,mem_swap_kb=8388604,mem_total_kb
>
> > =527946928,os=Linux} debug 2023-08-03T05:27:10.419+0000 7f8ea1b2c700
>
> > 0 client.5194553 ms_handle_reset on v2:10.45.128.141:3300/0 debug
>
> > 2023-08-03T05:50:10.917+0000 7f8ea1b2c700  0 client.5194553
>
> > ms_handle_reset on v2:10.45.128.139:3300/0
>
> >
>
> > Thank you,
>
> > Anantha
>
> > _______________________________________________
>
> > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
>
> > email to ceph-users-leave@xxxxxxx
>
> >
>
>
>
>
>
> --
>
> Cheers,
>
> Venky
>
> _______________________________________________
>
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx



-- 
Cheers,
Venky

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux