On 09/10/2009 06:25 AM, Stephan von Krawczynski wrote:
> On Wed, 09 Sep 2009 19:43:15 -0400
> Mark Mielke<mark at mark.mielke.cc> wrote:
>
>
>> In this case, there is too many unknowns - but I agree with Anand's
>> logic 100%. Gluster should not be able to cause a CPU lock up. It should
>> be impossible. If it is not impossible - it means a kernel bug, and the
>> best place to have this addressed is the kernel devel list, or, if you
>> have purchased a subscription from a company such as RedHat, than this
>> belongs as a ticket open with RedHat.
>>
> You know, I am really bothered about the way the maintainers are acting since
> I read this list. There is really a lot of ideology going on ("can't be", "is
> impossible for userspace" etc) and very few real debugging.
>
In general, if one didn't understand the architecture (black box), you
would be right. I would not want to here these things either -
However, Anand did not *just* say "can't be", or "is impossible for
userspace". He provided significant explanation which exactly matches my
own prior understanding about the separation between user space and
kernel space. In particular, if you read about the intent of FUSE - the
technology being used to create a file system, I think you will find
that what Anand is saying is the *exact* purpose for this project. Why
have a file system in user space in the first place? It introduces
performance limitations. The "why do it" is precisely to provide the
level of isolation and separation from the kernel. Take into account
that FUSE *encourages* user's to use or write their own file system
layers. That is, you do not need to be admin/root in order to call
fusermount and mount a FUSE file system. If any user of your system can
create it - don't you think it is reasonable to expect the kernel and
FUSE developers to protect "complete system lock up" from occurring?
If FUSE cannot provide separation from kernel space, then FUSE is
garbage and should be thrown away.
GlusterFS folk chose to use FUSE to obtain this separation. If this
separation is not being provided - than the value of FUSE in the first
place is brought into question.
> This application is not the only one in the world. People use heavily file-
> and net-acting applications like firefox, apache, shell-scripts, name-one on
> their boxes. None leads to effects seen if you play with glusterfs. If you
> really think it is a logical way of debugging to go out and simply tell
> "userspace can't do that" while the rest of the application-world does not
> show up with dead-ends like seen on this list, how can I change your mind?
> I hardly believe I can. I can only tell you what I would do: I would try to
> document _first_ that my piece of code really does behave well. But as you may
> have noticed there is no real way to provide this information. And that is
> indeed part of the problem.
Other applications fail all of the time as well - it seems to be a
question, in this case, of how critical the failure is. But, let us say
that the problem is "every time you send too many bytes to the network
device, it dies" - whose responsibility is this to fix? Is it GlusterFS
for being too efficient? Is it the Linux kernel driver for not loading
the data onto the device properly? Is it the device for locking up under
a certain type of load? Do you think the best and only place to look for
an answer is the user space program? Let's say every time you used
Firefox, your CPU + network device locked up - would you go to the
Firefox devel mailing list and demand they fix this? Because, you know
you would get the same response. They would laugh you out - whether you
were right or wrong. The onus is on you, as it is your hardware which is
dying. Now, if you have paid a subscription price - the responsibility
might increase - but if it's free / open source / community-based
support, and the general consensus on the technology is that FUSE is a
user space capability, and user space should not be able to lock up CPU
or network device - which is most definitely true, at least as an ideal
- then yes, the onus is the community member to prove that others should
reconsider their long held beliefs.
I disagree that other user space programs do not trigger kernel bugs.
Read the kernel devel list for a while, or patch the kernel release
notes. I think you will find that most bugs - of which there are
thousands or more - are triggered by user space programs. The kernel
developers fix the problems, because they are kernel problems. Unless
you tell them about the problem - they won't know to look into it. In
this particular case - what do you want Anand or gluster.com to do?
Let's say every time they send one particular packet very quickly after
another particular packet, it locks up. Do you want Anand or gluster.com
to stop sending these packets? Or do you want the Linux developers to
fix the device so this no longer happens again? Which is a better
solution to the problem? Which helps the most people, and prevents the
problem from occurring in the future?
> Wouldn't it be a nice step if you could debug the ongoings of a
> glusterfs-server on the client by simply reading an exported file (something
> like a server-dependant meta-debug-file) that outputs something like strace
> does? Something that enables you to say: "Ok, here you can see what the
> application did, and there you can see what the kernel made of it". As we
> noticed a server-logfile is not sufficient.
>
Sure, that would be awesome - and I think it's provided by such things
as DTrace, or strace. This is a kernel problem. If every user space
application must re-invent this wheel, that's surely a lot of effort on
the part of application developers. Should Firefox provide the same
functionality?
> Is ideology really a prove for anything in todays' world? Do you really think
> it is possible to understand the complete world by seeing half of it and the
> other half painted by ideology? What is wrong about _proving_ being not
> guilty? About acting defensive ?
>
Ideology? It depends. Some basic principles and basic understandings of
how the systems are designed are required to help us triage problems we
face every day. Knowing the system has failed is insufficient. The first
question is - where did the failure occur, and who should I ask for
help? In the case of a CPU lockup and/or network device lockup, most
people would start with the Linux kernel. Now, if this was an NFS
problem - than NFS is part of the kernel, and so the NFS part of the
kernel would be open for consideration. But, we know that GlusterFS does
*not* introduce any code into the kernel. This little bit of information
is important, and should not be ignored.
Now, maybe it's easier to start with GlusterFS, and try and pull on the
community and the GlusterFS support people for *help*, because we can
argue "it's a failure that from a superficial level, seems to be trigger
by your software, so you should be concerned" - but this has a limit to
how effective it is as a means of addressing the problem. First, the
GlusterFS people probably have little or no clue on how to diagnose a
CPU lockup or network device lockup. Since their software is entirely in
user space, they would not require this capability as part of their
employment requirements. If somebody on the staff *happens* to have the
right experience, and happens to know the answer, than you would be
lucky. This is not the sort of thing I would expect however, even if I
had a subscription for support. For them to say - we have looked into
this, and this is not our problem because of such and such, but please
come back if you can prove otherwise such that we can do something about
this - is a fair answer.
> It is important to understand that this application is a kind of core
> technology for data storage. This means people want to be sure that their
> setup does not explode just because they made a kernel update or some other
> change where their experience tells them it should have no influence on the
> glusterfs service. You want to be sure, just like you are when using nfs. It
> just does work (even being in kernel-space!).
> Now, answer for yourself if you think glusterfs is as stable as nfs on the
> same box.
One could say the same thing about every layer between user space and
the hardware. If your disk dies, is this a GlusterFS problem? If your
disk controller dies, is this a GlusterFS problem? If your network
device dies, is this a GlusterFS problem? If your CPU dies, is this a
GlusterFS problem?
I don't agree that NFS just works. NFS has gone through a lot of
evolution and maturity. At our company, I've been aware of numerous
problems with NFS. Coincidentally, I was in a call with one of the
owners of the Linux NFS code from RedHat a few weeks ago to discuss the
subject of file system caching, and whether or not NFS would be a
solution to a problem we were having with another network file system
(ClearCase MVFS). During the call, the subject of NFS development did
come up, and as I recall, he humbly acknowledged that NFS has had a lot
of problems and they have been working hard on it. I told him I thought
they were doing a great job and I meant it. Everything is relative. Yes,
we rely on NFS every day at work - but, for the most part, it works
great. We have problems, but RedHat has been responsive to our problems
*once we have identified them as NFS problems*, and we work together
towards a solution. But then, we also pay RedHat money to support us.
Cheers,
mark
--
Mark Mielke<mark at mielke.cc>
