Amar,
I see debuginfo packages now and have installed them. I'm available via Skype as before, just ping me there.
On Tue, Mar 19, 2019 at 10:46 PM Amar Tumballi Suryanarayan <atumball@xxxxxxxxxx> wrote:
On Wed, Mar 20, 2019 at 9:52 AM Artem Russakovskii <archon810@xxxxxxxxx> wrote:Can I roll back performance.write-behind: off and lru-limit=0 then? I'm
waiting for the debug packages to be available for OpenSUSE, then I can
help Amar with another debug session.
Yes, the write-behind issue is now fixed. You can enable write-behind. Also remove lru-limit=0, so you can also utilize the benefit of garbage collection introduced in 5.4Lets get to fixing the problem once the debuginfo packages are available.In the meantime, have you had time to set up 1x4 replicate testing? I was
told you were only testing 1x3, and it's the 4th brick that may be causing
the crash, which is consistent with this whole time only 1 of 4 bricks
constantly crashing. The other 3 have been rock solid. I'm hoping you could
find the issue without a debug session this way.
That is my gut feeling still. Added a basic test case with 4 bricks, https://review.gluster.org/#/c/glusterfs/+/22328/. But I think this particular issue is happening only on certain pattern of access for 1x4 setup. Lets get to the root of it once we have debuginfo packages for Suse builds.-AmarSincerely,
Artem
--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>
On Tue, Mar 19, 2019 at 8:27 PM Nithya Balachandran <nbalacha@xxxxxxxxxx>
wrote:
> Hi Artem,
>
> I think you are running into a different crash. The ones reported which
> were prevented by turning off write-behind are now fixed.
> We will need to look into the one you are seeing to see why it is
> happening.
>
> Regards,
> Nithya
>
>
> On Tue, 19 Mar 2019 at 20:25, Artem Russakovskii <archon810@xxxxxxxxx>
> wrote:
>
>> The flood is indeed fixed for us on 5.5. However, the crashes are not.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>>
>> On Mon, Mar 18, 2019 at 5:41 AM Hu Bert <revirii@xxxxxxxxxxxxxx> wrote:
>>
>>> Hi Amar,
>>>
>>> if you refer to this bug:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1674225 : in the test
>>> setup i haven't seen those entries, while copying & deleting a few GBs
>>> of data. For a final statement we have to wait until i updated our
>>> live gluster servers - could take place on tuesday or wednesday.
>>>
>>> Maybe other users can do an update to 5.4 as well and report back here.
>>>
>>>
>>> Hubert
>>>
>>>
>>>
>>> Am Mo., 18. März 2019 um 11:36 Uhr schrieb Amar Tumballi Suryanarayan
>>> <atumball@xxxxxxxxxx>:
>>> >
>>> > Hi Hu Bert,
>>> >
>>> > Appreciate the feedback. Also are the other boiling issues related to
>>> logs fixed now?
>>> >
>>> > -Amar
>>> >
>>> > On Mon, Mar 18, 2019 at 3:54 PM Hu Bert <revirii@xxxxxxxxxxxxxx>
>>> wrote:
>>> >>
>>> >> update: upgrade from 5.3 -> 5.5 in a replicate 3 test setup with 2
>>> >> volumes done. In 'gluster peer status' the peers stay connected during
>>> >> the upgrade, no 'peer rejected' messages. No cksum mismatches in the
>>> >> logs. Looks good :-)
>>> >>
>>> >> Am Mo., 18. März 2019 um 09:54 Uhr schrieb Hu Bert <
>>> revirii@xxxxxxxxxxxxxx>:
>>> >> >
>>> >> > Good morning :-)
>>> >> >
>>> >> > for debian the packages are there:
>>> >> >
>>> https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/
>>> >> >
>>> >> > I'll do an upgrade of a test installation 5.3 -> 5.5 and see if
>>> there
>>> >> > are some errors etc. and report back.
>>> >> >
>>> >> > btw: no release notes for 5.4 and 5.5 so far?
>>> >> > https://docs.gluster.org/en/latest/release-notes/ ?
>>> >> >
>>> >> > Am Fr., 15. März 2019 um 14:28 Uhr schrieb Shyam Ranganathan
>>> >> > <srangana@xxxxxxxxxx>:
>>> >> > >
>>> >> > > We created a 5.5 release tag, and it is under packaging now. It
>>> should
>>> >> > > be packaged and ready for testing early next week and should be
>>> released
>>> >> > > close to mid-week next week.
>>> >> > >
>>> >> > > Thanks,
>>> >> > > Shyam
>>> >> > > On 3/13/19 12:34 PM, Artem Russakovskii wrote:
>>> >> > > > Wednesday now with no update :-/
>>> >> > > >
>>> >> > > > Sincerely,
>>> >> > > > Artem
>>> >> > > >
>>> >> > > > --
>>> >> > > > Founder, Android Police <http://www.androidpolice.com>, APK
>>> Mirror
>>> >> > > > <http://www.apkmirror.com/>, Illogical Robot LLC
>>> >> > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii
>>> >> > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> >> > > > <http://twitter.com/ArtemR>
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <
>>> archon810@xxxxxxxxx
>>> >> > > > <mailto:archon810@xxxxxxxxx>> wrote:
>>> >> > > >
>>> >> > > > Hi Amar,
>>> >> > > >
>>> >> > > > Any updates on this? I'm still not seeing it in OpenSUSE
>>> build
>>> >> > > > repos. Maybe later today?
>>> >> > > >
>>> >> > > > Thanks.
>>> >> > > >
>>> >> > > > Sincerely,
>>> >> > > > Artem
>>> >> > > >
>>> >> > > > --
>>> >> > > > Founder, Android Police <http://www.androidpolice.com>,
>>> APK Mirror
>>> >> > > > <http://www.apkmirror.com/>, Illogical Robot LLC
>>> >> > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii
>>> >> > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> >> > > > <http://twitter.com/ArtemR>
>>> >> > > >
>>> >> > > >
>>> >> > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan
>>> >> > > > <atumball@xxxxxxxxxx <mailto:atumball@xxxxxxxxxx>> wrote:
>>> >> > > >
>>> >> > > > We are talking days. Not weeks. Considering already it
>>> is
>>> >> > > > Thursday here. 1 more day for tagging, and packaging.
>>> May be ok
>>> >> > > > to expect it on Monday.
>>> >> > > >
>>> >> > > > -Amar
>>> >> > > >
>>> >> > > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii
>>> >> > > > <archon810@xxxxxxxxx <mailto:archon810@xxxxxxxxx>>
>>> wrote:
>>> >> > > >
>>> >> > > > Is the next release going to be an imminent hotfix,
>>> i.e.
>>> >> > > > something like today/tomorrow, or are we talking
>>> weeks?
>>> >> > > >
>>> >> > > > Sincerely,
>>> >> > > > Artem
>>> >> > > >
>>> >> > > > --
>>> >> > > > Founder, Android Police <
>>> http://www.androidpolice.com>, APK
>>> >> > > > Mirror <http://www.apkmirror.com/>, Illogical
>>> Robot LLC
>>> >> > > > beerpla.net <http://beerpla.net/> |
>>> +ArtemRussakovskii
>>> >> > > > <https://plus.google.com/+ArtemRussakovskii> |
>>> @ArtemR
>>> >> > > > <http://twitter.com/ArtemR>
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii
>>> >> > > > <archon810@xxxxxxxxx <mailto:archon810@xxxxxxxxx>>
>>> wrote:
>>> >> > > >
>>> >> > > > Ended up downgrading to 5.3 just in case. Peer
>>> status
>>> >> > > > and volume status are OK now.
>>> >> > > >
>>> >> > > > zypper install --oldpackage
>>> glusterfs-5.3-lp150.100.1
>>> >> > > > Loading repository data...
>>> >> > > > Reading installed packages...
>>> >> > > > Resolving package dependencies...
>>> >> > > >
>>> >> > > > Problem: glusterfs-5.3-lp150.100.1.x86_64
>>> requires
>>> >> > > > libgfapi0 = 5.3, but this requirement cannot be
>>> provided
>>> >> > > > not installable providers:
>>> >> > > > libgfapi0-5.3-lp150.100.1.x86_64[glusterfs]
>>> >> > > > Solution 1: Following actions will be done:
>>> >> > > > downgrade of libgfapi0-5.4-lp150.100.1.x86_64
>>> to
>>> >> > > > libgfapi0-5.3-lp150.100.1.x86_64
>>> >> > > > downgrade of
>>> libgfchangelog0-5.4-lp150.100.1.x86_64 to
>>> >> > > > libgfchangelog0-5.3-lp150.100.1.x86_64
>>> >> > > > downgrade of libgfrpc0-5.4-lp150.100.1.x86_64
>>> to
>>> >> > > > libgfrpc0-5.3-lp150.100.1.x86_64
>>> >> > > > downgrade of libgfxdr0-5.4-lp150.100.1.x86_64
>>> to
>>> >> > > > libgfxdr0-5.3-lp150.100.1.x86_64
>>> >> > > > downgrade of
>>> libglusterfs0-5.4-lp150.100.1.x86_64 to
>>> >> > > > libglusterfs0-5.3-lp150.100.1.x86_64
>>> >> > > > Solution 2: do not install
>>> glusterfs-5.3-lp150.100.1.x86_64
>>> >> > > > Solution 3: break
>>> glusterfs-5.3-lp150.100.1.x86_64 by
>>> >> > > > ignoring some of its dependencies
>>> >> > > >
>>> >> > > > Choose from above solutions by number or cancel
>>> >> > > > [1/2/3/c] (c): 1
>>> >> > > > Resolving dependencies...
>>> >> > > > Resolving package dependencies...
>>> >> > > >
>>> >> > > > The following 6 packages are going to be
>>> downgraded:
>>> >> > > > glusterfs libgfapi0 libgfchangelog0 libgfrpc0
>>> >> > > > libgfxdr0 libglusterfs0
>>> >> > > >
>>> >> > > > 6 packages to downgrade.
>>> >> > > >
>>> >> > > > Sincerely,
>>> >> > > > Artem
>>> >> > > >
>>> >> > > > --
>>> >> > > > Founder, Android Police
>>> >> > > > <http://www.androidpolice.com>, APK Mirror
>>> >> > > > <http://www.apkmirror.com/>, Illogical Robot
>>> LLC
>>> >> > > > beerpla.net <http://beerpla.net/> |
>>> +ArtemRussakovskii
>>> >> > > > <https://plus.google.com/+ArtemRussakovskii> |
>>> @ArtemR
>>> >> > > > <http://twitter.com/ArtemR>
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Mar 5, 2019 at 10:57 AM Artem
>>> Russakovskii
>>> >> > > > <archon810@xxxxxxxxx <mailto:
>>> archon810@xxxxxxxxx>> wrote:
>>> >> > > >
>>> >> > > > Noticed the same when upgrading from 5.3 to
>>> 5.4, as
>>> >> > > > mentioned.
>>> >> > > >
>>> >> > > > I'm confused though. Is actual replication
>>> affected,
>>> >> > > > because the 5.4 server and the 3x 5.3
>>> servers still
>>> >> > > > show heal info as all 4 connected, and the
>>> files
>>> >> > > > seem to be replicating correctly as well.
>>> >> > > >
>>> >> > > > So what's actually affected - just the
>>> status
>>> >> > > > command, or leaving 5.4 on one of the nodes
>>> is doing
>>> >> > > > some damage to the underlying fs? Is it
>>> fixable by
>>> >> > > > tweaking transport.socket.ssl-enabled? Does
>>> >> > > > upgrading all servers to 5.4 resolve it, or
>>> should
>>> >> > > > we revert back to 5.3?
>>> >> > > >
>>> >> > > > Sincerely,
>>> >> > > > Artem
>>> >> > > >
>>> >> > > > --
>>> >> > > > Founder, Android Police
>>> >> > > > <http://www.androidpolice.com>, APK Mirror
>>> >> > > > <http://www.apkmirror.com/>, Illogical
>>> Robot LLC
>>> >> > > > beerpla.net <http://beerpla.net/> |
>>> >> > > > +ArtemRussakovskii
>>> >> > > > <https://plus.google.com/+ArtemRussakovskii
>>> >
>>> >> > > > | @ArtemR <http://twitter.com/ArtemR>
>>> >> > > >
>>> >> > > >
>>> >> > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert
>>> >> > > > <revirii@xxxxxxxxxxxxxx
>>> >> > > > <mailto:revirii@xxxxxxxxxxxxxx>> wrote:
>>> >> > > >
>>> >> > > > fyi: did a downgrade 5.4 -> 5.3 and it
>>> worked.
>>> >> > > > all replicas are up and
>>> >> > > > running. Awaiting updated v5.4.
>>> >> > > >
>>> >> > > > thx :-)
>>> >> > > >
>>> >> > > > Am Di., 5. März 2019 um 09:26 Uhr
>>> schrieb Hari
>>> >> > > > Gowtham <hgowtham@xxxxxxxxxx
>>> >> > > > <mailto:hgowtham@xxxxxxxxxx>>:
>>> >> > > > >
>>> >> > > > > There are plans to revert the patch
>>> causing
>>> >> > > > this error and rebuilt 5.4.
>>> >> > > > > This should happen faster. the
>>> rebuilt 5.4
>>> >> > > > should be void of this upgrade issue.
>>> >> > > > >
>>> >> > > > > In the meantime, you can use 5.3 for
>>> this cluster.
>>> >> > > > > Downgrading to 5.3 will work if it
>>> was just
>>> >> > > > one node that was upgrade to 5.4
>>> >> > > > > and the other nodes are still in 5.3--Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
