Hi,
In glusterfs at the time of the process coming up, it creates 5 pthread
keys (for saving THIS, syncop, uuid buf, lkowner buf and syncop ctx).
Even gfapi does the same thing in its glfs_new function. But with User
Serviceable Snapshots (where a glusterfs process spawns multiple gfapi
instances for a snapshot) this will lead to more and more consumption of
pthread keys. In fact the old keys are lost (as same variables are used
for creating the keys) and eventually the process will run out of
pthread keys after 203 snapshots (maximum allowed number is 1024 per
process.). So to avoid it, pthread keys creation can be done only once
(using pthread_once in which, the globals_init function is called).
But now a new problem arises. Say, from snapview-server xlator glfs_new
was called (or glfs_init etc). Now gfapi wants calls THIS for some of
its operations such properly accounting the memory within the xlator
while allocating a new structure. But when gfapi calls THIS it gets
snapview-server's pointer. Since snapview-server does not know about
gfapi's internal structure, it asserts at the time of allocating.
For now, a patch has been sent to handle the issue by turning off
memory-accounting for snapshot daemon.
(http://review.gluster.org/#/c/9430).
But if memory-accounting has to be turned on for snapshot daemon, then
the above problem has to be fixed.
2 ways that can be used for fixing the issue are:
1) Add the datastructures that are used by gfapi to libglusterfs (and
hence their mem-types as well), so that any xlator that is calling gfapi
functions (such as snapview-server as of now) will be aware of the
memory types used by gfapi and hence will not cause problems, when
memory accounting has to be done as part of allocations and frees.
OR
2) Properly manage THIS by introducing a new macro similar to STACK_WIND
(for now it can be called STACK_API_WIND). The macro will be much
simpler than STACK_WIND as it need not create new frames before handing
over the call to the next layer. Before handing over the call to gfapi
(any call, such as glfs_new or fops such as glfs_h_open), saves THIS in
a variable and calls the gfapi function given as an argument. After the
function returns it again sets THIS back the value before the gfapi
function was called.
Ex:
STACK_API_WIND (this, fn, ret, params....)
do {
xlator_t *old_THIS = NULL;
old_THIS = this;
ret = fn (params);
THIS = old_THIS;
} while (0);
a caller (as of now snapview-server xlator) would call the macro like this
STACK_API_WIND (this, glfs_h_open, glfd, fs, object, flags);
Please provide feedback and any suggestions or solutions to handle the
mentioned issue are welcome.
Regards,
Raghavendra Bhat
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
