Re: Systemd killed processes of custom services instead of graceful shutdown

[Andrei Borzenkov <arvidjaar@xxxxxxxxx>]

17.06.2020 07:46, Kamal Rathi пишет:
>> Please take at least some efforts to format mail so that it can be
>> properly understood. Otherwise I am afraid your mails will simply be
>> ignored.
> 
>        I am sorry for inconvenience, i will try to follow correct approach
> afterwards while replying
> 
>> How RemainAfterExit affects *startup* of services?
> 
>       The services worked as expected and as i fixed the issue where it was
> using su in script.
>       As in background there was SysVinit script which was getting invoked
> when system boot and makes processes falls in users.slice
>       So the startup is working perfectly after disabling that SysVinit
> script.
> 
>> So I was right. Your programs are running as user services or part of
>> user sessions.
> 
>      You were absolutely right, I tried to dig into that and find out as it
> was using internally su,
>      but now only problem remains is shutdown order.
>      As I have created a dependency that rdbms.service should start after
> grid.service which works perfect at boot.
>     But when I do shutdown / reboot both services stop simultaneously.
>     which i want my grid.service to wait first before rdbms.service will
> stop
>     As Type mentioned in services are oneshot.
> 
>> dmesg output does not contain any systemd log. Anyway, we already know
>> that your services are not really system services.
> 
>     So my service is system services, as let me attach cgls output
> 
> └─system.slice
>  ├─rdbms.service
>  │ └─8039 /sarahn9db/db/11.2.0/bin/tnslsnr sarahn9 -inherit
>  ├─grid.service
>  │ ├─  5316 /grid/gi/11.2.0/bin/ohasd.bin reboot
>  │ ├─  6159 /grid/gi/11.2.0/bin/cssdagent
> 

The obvious possibilities

- ExecStop command returns before application is actually stopped.
ExecStop must be synchronous, systemd assumes service is stopped when
ExecStop command completes

- stopping takes longer then TimeoutStopSec

- your actual database it still started outside of rdbms.service. The
only process shown in your output is listener, but I would expect rather
more of Oracle there

Without logs it is impossible to say anything more specific.


> On Tue, Jun 16, 2020 at 12:23 AM Andrei Borzenkov <arvidjaar@xxxxxxxxx>
> wrote:
> 
>> 16.06.2020 07:36, Kamal Rathi пишет:
>>> The fact that you need RemainAfterExit at all hints that processes that
>>> belong to your service are not running as part of service control group.
>>> Knowing how Oracle has traditionally been managed, I suspect that you
>>> perform "su - oracle_owner" or similar to start them in which case all
>>> actual service processes become part of respective user sessions, and
>>> not part of your system services. There is no way to synchronize
>>> stopping of processes/services belonging to different users. They are
>>> completely independent and shutdown for all sessions is initiated in
>>> parallel.
>>>
>>
>> Please take at least some efforts to format mail so that it can be
>> properly understood. Otherwise I am afraid your mails will simply be
>> ignored.
>>
>>> If my theory is correct, the fix would be to actually run your systemd
>>> services as systemd services. If my theory is wrong, provide full from
>>> system boot to shutdown where it could be seen how your services are
>>> started/stopped. Enabling systemd debug log level when doing it
>>> certainly won't harm.
>>>
>>>
>>>
>>> I have incorporated Type=oneshot and remainAfterExit=yes as it was
>> helpful
>>> in startup of services,
>>
>>
>> How RemainAfterExit affects *startup* of services?
>>
>>> but still grid.service are starting up in
>>> user.slice
>>
>> So I was right. Your programs are running as user services or part of
>> user sessions.
>>
>>> and shutdown was giving the same results
>>> I am attaching debug logs .
>>>
>>
>> dmesg output does not contain any systemd log. Anyway, we already know
>> that your services are not really system services.
>>
>>> On Mon, Jun 15, 2020 at 10:34 AM Andrei Borzenkov <arvidjaar@xxxxxxxxx>
>>> wrote:
>>>
>>>> 15.06.2020 11:01, Kamal Rathi пишет:
>>>>> Hi Team,
>>>>>
>>>>> I have two services which are dependent on each other and are working
>>>> fine
>>>>> at boot up but at shutdown / reboot , the processes get killed as
>>>> shutdown
>>>>> got initated.
>>>>>
>>>>> Services are running fine in particular order but processes got killed
>> .I
>>>>> have enabled lingering on both users and changed confgiuration in
>>>>> logind.conf to KillUserProcesses=no but still issue is same
>>>>>
>>>>
>>>> Lingering/KillUserProcesses are relevant only for user services/sessions
>>>> and so far there was no indication you use either.
>>>>
>>>>> ##############
>>>>> Systemd service files content are below
>>>>>
>>>>> cat /etc/systemd/system/grid.service
>>>>> [Unit]
>>>>> Description=Service to auto start Oracle ASM application
>>>>> Before=rdbms.service
>>>>> After=syslog.target network.target nfs-mountd.service autofs.service
>>>>> systemd-user-sessions.service system.slice
>>>>> [Service]
>>>>> Type=simple
>>>>> TimeoutSec=5min
>>>>> User=grid
>>>>> Group=dba
>>>>> ExecStart=/opt/admin/bin/asm
>>>>> ExecStop=/opt/admin/bin/asm_stop
>>>>> RemainAfterExit=yes
>>>>> [Install]
>>>>> WantedBy=multi-user.target
>>>>>
>>>>>
>>>>>
>>>>> cat /etc/systemd/system/rdbms.service
>>>>> [Unit]
>>>>> Description=Service to auto start Oracle RDBMS application
>>>>> Requires=grid.service
>>>>> After=grid.service syslog.target network.target nfs-mountd.service
>>>>> autofs.service systemd-user-sessions.service system.slice
>>>>> [Service]
>>>>> Type=simple
>>>>> TimeoutSec=5min
>>>>> User=osarahn9
>>>>> Group=dba
>>>>> ExecStart=/opt/admin/bin/rdbms
>>>>> ExecStop=/opt/admin/bin/rdbms_stop
>>>>> RemainAfterExit=yes
>>>>> [Install]
>>>>> WantedBy=multi-user.target grid.service
>>>>>
>>>>>
>>>>> let me know if my configuration is faulty or what I have missed so that
>>>>> shutdown should be graceful for services and processes will be
>>>>> shutdown with systemd custom service?
>>>>>
>>>>
>>>> You do not provide enough information (full logs would be certainly much
>>>> more useful than long description) so I can only give educated guess.
>>>>
>>>>> I want first rdbms.service should be called and get process stopped
>>>> before
>>>>> grid.services (it seems systemd are killing user.slices processes) and
>> in
>>>>> startup-inverse should be followed .
>>>>> Please help .
>>>>>
>>>>
>>>> The fact that you need RemainAfterExit at all hints that processes that
>>>> belong to your service are not running as part of service control group.
>>>> Knowing how Oracle has traditionally been managed, I suspect that you
>>>> perform "su - oracle_owner" or similar to start them in which case all
>>>> actual service processes become part of respective user sessions, and
>>>> not part of your system services. There is no way to synchronize
>>>> stopping of processes/services belonging to different users. They are
>>>> completely independent and shutdown for all sessions is initiated in
>>>> parallel.
>>>>
>>>> If my theory is correct, the fix would be to actually run your systemd
>>>> services as systemd services. If my theory is wrong, provide full from
>>>> system boot to shutdown where it could be seen how your services are
>>>> started/stopped. Enabling systemd debug log level when doing it
>>>> certainly won't harm.
>>>> _______________________________________________
>>>> systemd-devel mailing list
>>>> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
>>>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> systemd-devel mailing list
>>> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
>>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>>
>>
>> _______________________________________________
>> systemd-devel mailing list
>> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>
> 
> 
> _______________________________________________
> systemd-devel mailing list
> systemd-devel@xxxxxxxxxxxxxxxxxxxxx
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> 

_______________________________________________
systemd-devel mailing list
systemd-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/systemd-devel