Re: Upgrade stalled after upgrading managers

[Torkil Svensgaard <torkil@xxxxxxxx>]

Thanks guys, turning off the balancer seems to have fixed it.

Mvh.

Torkil

On 17/12/2024 12:40, Eugen Block wrote:
> I know 19.2.1 is already in the validation phase, but it would make 
> sense (to me) to add this to the upgrade notes for Squid (https:// 
> docs.ceph.com/en/latest/releases/squid/#v19-2-0-squid) until the fix has 
> been released. Similar to the note about ISCSI users. Adding Zac here 
> directly.
>
> Zitat von Laimis Juzeliūnas <laimis.juzeliunas@xxxxxxxxxx>:
>
> > Hi Torkil,
> >
> > Possible that you are hitting balancer issues on 19.2.0 for clusters 
> > with larger pg numbers: https://tracker.ceph.com/issues/68657
> > Try turning it off with ceph balancer off
> >
> > Best,
> > Laimis J.
> >
> > > On 17 Dec 2024, at 13:15, Torkil Svensgaard <torkil@xxxxxxxx> wrote:
> > >
> > >
> > >
> > > On 17/12/2024 12:05, Torkil Svensgaard wrote:
> > > > Hi
> > > > Running upgrade from 18.2.4 to 19.2.0 and it managed to upgrade the 
> > > > managers but no further progress.
> > >
> > > Now it actually seems to have upgraded 1 MON now then the 
> > > orchestrator crashed again:
> > >
> > > "
> > > {
> > >    "mon": {
> > >     "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) 
> > > reef (stable)": 4,
> > >        "ceph version 19.2.0 
> > > (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)": 1
> > >    },
> > >    "mgr": {
> > >        "ceph version 19.2.0 
> > > (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)": 3
> > >    },
> > >    "osd": {
> > >     "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) 
> > > reef (stable)": 548
> > >    },
> > >    "mds": {
> > >     "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) 
> > > reef (stable)": 3
> > >    },
> > >    "overall": {
> > >        "ceph version 18.2.4 
> > > (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 555,
> > >        "ceph version 19.2.0 
> > > (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)": 4
> > >    }
> > > }
> > > "
> > >
> > > Mvh.
> > >
> > > Torkil
> > >
> > >
> > > > If I fail over the mgr it goes:
> > > > "
> > > > [root@ceph-flash1 ~]# ceph orch upgrade status
> > > > Error ENOTSUP: Module 'orchestrator' is not enabled/loaded (required 
> > > > by command 'orch upgrade status'): use `ceph mgr module enable 
> > > > orchestrator` to enable it
> > > > "
> > > > From mgr log:
> > > > "
> > > > ...
> > > > 2024-12-17T10:43:11.729+0000 7f70efafe640  0 log_channel(audit) log 
> > > > [DBG] : from='client.2110010386 -' entity='client.admin' 
> > > > cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: 
> > > > dispatch
> > > > 2024-12-17T10:43:11.733+0000 7f70ebaf6640  0 [cephadm INFO 
> > > > cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTING
> > > > 2024-12-17T10:43:11.733+0000 7f70ebaf6640  0 log_channel(cephadm) 
> > > > log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTING
> > > > 2024-12-17T10:43:11.811+0000 7f70e7aee640  0 [dashboard INFO 
> > > > dashboard.module] Engine started.
> > > > 2024-12-17T10:43:11.861+0000 7f70ebaf6640  0 [cephadm INFO 
> > > > cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on https:// 
> > > > www.google.com/url?q=https://172.21.15.148:7150&source=gmail- 
> > > > imap&ust=1735039047000000&usg=AOvVaw3LyWY24vMZA-AbVVOsv3Z9
> > > > 2024-12-17T10:43:11.861+0000 7f70ebaf6640  0 log_channel(cephadm) 
> > > > log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on https:// 
> > > > www.google.com/url?q=https://172.21.15.148:7150&source=gmail- 
> > > > imap&ust=1735039047000000&usg=AOvVaw3LyWY24vMZA-AbVVOsv3Z9
> > > > 2024-12-17T10:43:11.864+0000 7f70a2d7a640  0 [cephadm ERROR 
> > > > cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve
> > > > Traceback (most recent call last):
> > > >   File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, 
> > > > in serve
> > > >     self._connections.run(self.expiration_interval)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 203, in run
> > > >     self._run(expiration_interval)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 246, in _run
> > > >     new_conn = self._from_server_socket(self.server.socket)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 300, in _from_server_socket
> > > >     s, ssl_env = self.server.ssl_adapter.wrap(s)
> > > >   File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 
> > > > 277, in wrap
> > > >     s = self.context.wrap_socket(
> > > >   File "/lib64/python3.9/ssl.py", line 501, in wrap_socket
> > > >     return self.sslsocket_class._create(
> > > >   File "/lib64/python3.9/ssl.py", line 1074, in _create
> > > >     self.do_handshake()
> > > >   File "/lib64/python3.9/ssl.py", line 1343, in do_handshake
> > > >     self._sslobj.do_handshake()
> > > > ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) 
> > > > (_ssl.c:1133)
> > > > 2024-12-17T10:43:11.865+0000 7f70a2d7a640 -1 log_channel(cephadm) 
> > > > log [ERR] : [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve
> > > > Traceback (most recent call last):
> > > >   File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, 
> > > > in serve
> > > >     self._connections.run(self.expiration_interval)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 203, in run
> > > >     self._run(expiration_interval)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 246, in _run
> > > >     new_conn = self._from_server_socket(self.server.socket)
> > > >   File "/lib/python3.9/site-packages/cheroot/connections.py", line 
> > > > 300, in _from_server_socket
> > > >     s, ssl_env = self.server.ssl_adapter.wrap(s)
> > > >   File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 
> > > > 277, in wrap
> > > >     s = self.context.wrap_socket(
> > > >   File "/lib64/python3.9/ssl.py", line 501, in wrap_socket
> > > >     return self.sslsocket_class._create(
> > > >   File "/lib64/python3.9/ssl.py", line 1074, in _create
> > > >     self.do_handshake()
> > > >   File "/lib64/python3.9/ssl.py", line 1343, in do_handshake
> > > >     self._sslobj.do_handshake()
> > > > ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) 
> > > > (_ssl.c:1133)
> > > > 2024-12-17T10:43:11.963+0000 7f70ebaf6640  0 [cephadm INFO 
> > > > cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on https:// 
> > > > www.google.com/url?q=http://172.21.15.148:8765&source=gmail- 
> > > > imap&ust=1735039047000000&usg=AOvVaw1D05c8loKEwXnozNdlOMpU
> > > > 2024-12-17T10:43:11.963+0000 7f70ebaf6640  0 log_channel(cephadm) 
> > > > log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on https:// 
> > > > www.google.com/url?q=http://172.21.15.148:8765&source=gmail- 
> > > > imap&ust=1735039047000000&usg=AOvVaw1D05c8loKEwXnozNdlOMpU
> > > > 2024-12-17T10:43:11.963+0000 7f70ebaf6640  0 [cephadm INFO 
> > > > cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTED
> > > > 2024-12-17T10:43:11.964+0000 7f70ebaf6640  0 log_channel(cephadm) 
> > > > log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTED
> > > > ...
> > > > "
> > > > It will recover after some timeout, maybe 5-10 mins, and then just 
> > > > sit there with no upgrade progress.
> > > > Nothing in mgr/cephadm/osd_remove_queue.
> > > > Suggestions?
> > > > Mvh.
> > > > Torkil
> > >
> > > --
> > > Torkil Svensgaard
> > > Sysadmin
> > > MR-Forskningssektionen, afs. 714
> > > DRCMR, Danish Research Centre for Magnetic Resonance
> > > Hvidovre Hospital
> > > Kettegård Allé 30
> > > DK-2650 Hvidovre
> > > Denmark
> > > Tel: +45 386 22828
> > > E-mail: torkil@xxxxxxxx
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: torkil@xxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx