Re: Simulating some kind of "virtual file"

[David Spisla <david.spisla@xxxxxxxxxxxx>]

Hello Xavier,

 

now adding gluster-devel 😉

 

Von: Xavi Hernandez [mailto:jahernan@xxxxxxxxxx]
Gesendet: Dienstag, 9. Januar 2018 23:02
An: David Spisla <david.spisla@xxxxxxxxxxxx>
Cc: gluster-devel@xxxxxxxxxxx
Betreff: Re: [Gluster-devel] Simulating some kind of "virtual file"

 

Hi David,

 

adding again gluster-devel.

 

On Tue, Jan 9, 2018 at 4:15 PM, David Spisla <david.spisla@xxxxxxxxxxxx> wrote:

Hello Xavi,

 

Von: Xavi Hernandez [mailto:jahernan@xxxxxxxxxx]
Gesendet: Dienstag, 9. Januar 2018 09:48
An: David Spisla <spisla80@xxxxxxxxxxxxxx>
Cc: gluster-devel@xxxxxxxxxxx
Betreff: Re: [Gluster-devel] Simulating some kind of "virtual file"

 

Hi David,

 

On Tue, Jan 9, 2018 at 9:09 AM, David Spisla <spisla80@xxxxxxxxxxxxxx> wrote:

Dear Gluster Devels,

at the moment I do some Xlator stuff and I want to know if there is a way to simulate the existing of a file to the client. It should be a kind of "virtual file". Here are more details.

1. Client lookup for a file e.g. "apple.test". This file does not exist in the backend

$ ls -l apple.test

ls: cannot access apple.test: No such file or directory

 

Normally the system will not find that file

 

2. In the backend I have a real file called  e.g. "apple". Now there is a Xlator which manipulates the lookup request and is looking for the file "apple" instead of "apple.test". Gluster finds the file "apple" and the client will get a message from gluster that there is a file called "apple.test" with the attributes of the file "apple" (maybe we can manipulate that attributes too).

 

Intercepting lookup is fine to be able to manipulate "virtual files", however it's not enough to completely operate on virtual files. You basically need to intercept all file operations that work on a loc or are path based and do the translation. You also need to intercept answers from readdir and readdirp to do the reverse transformation so that the user sees the virtual name and not the real name stored on the bricks.

[David Spisla] Yes, this is a very good hint.

 

 

$ ls -l apple.test
-rw-r--r-- 1 davids davids 0 Jan  5 15:42 apple.test

 

My first idea is, to have a special lookup and lookup_cbk in some Xlator in the server stack. Or it is better to have this Xlator in the Client Stack?

 

That depends on what you are really trying to do. If you don't need any information or coordination with other bricks, you can safely create the xlator in the server stack.

[David Spisla] At the moment I do it in the worm xlator. It seems to be no problem.

 

The lookup function has a parameter called "loc_t *loc". In a first test I tried to manipulate loc->name and loc-path. If I manipulate loc->path I got an error and my volume crashed.

 

You should be able to modify the loc without causing any crash. However there are some details you must be aware of:

• loc->path and/or loc->name can be NULL in some cases.
• If loc->path and loc->name are not NULL, loc->name always points to a substring of loc->path. It's not allocated independently (and so it must not be freed).
• If you change loc->path, you also need to change loc->name (to point to the basename of loc->path)
• You shouldn't change the contents of loc->path directly. It's better to allocate a new string with the modified path and assign it to loc->path (you need to free the old value of loc->path to avoid memory leaks).

[David Spisla] I tried this:

char *new_path = malloc(1+len_path-5);

memcpy(new_path, loc->path, len_path-5);

new_path[strlen(new_path)] = '\0';

loc->name = new_path + (len_path - len_name);

 

First of all, you should always use memory allocation functions from gluster. This includes GF_MALLOC(), gf_strdup(), gf_asprintf() and several other variants. You can look at libglusterfs/src/mem-pool.h to see all available options.

 

The second problem I see is that memcpy() doesn't write a terminating null character, so when you compute strlen() afterwards, it will return invalid length, or even try to access invalid memory, causing a crash.

 

You should do something like this (assuming both loc->path and loc->name are not NULL and skipping many necessary checks):

 

len_path = strlen(loc->path);

len_name = strlen(loc->name);

new_path = GF_MALLOC(len_path - 4, gf_common_mt_char);

memcpy(new_path, loc->path, len_path - 5);

new_path[len_path - 5] = 0;

loc->name = new_path + len_path - len_name;

 

This should work fine.

 

Xavi

[David Spisla] Yes, this worls fine. Thank you 😊. By the way, is there a way inside gluster xlator to get access to xattr or attr of a file. In the lookup function there is only the struct loc, but I am missing there the files gfid. It seems to be null always. I could use syncop_getxattr() with the parameter loc, but the gfid is missing. Can I get the gfid if I have only loc->path and loc-> name? It is like a conversion from files path to files gfid.

 

David

 

So, if I do this command:

$ ls -l /test/dir/test1.txt.test

Sometimes it is working but sometimes I got strange outputs in the lookup_function

[2018-01-09 14:55:33.179568] I [worm.c:564:worm_lookup] 0-gv0-worm: checking if *.test

[2018-01-09 14:55:33.179583] I [worm.c:584:worm_lookup] 0-gv0-worm: is *.test

[2018-01-09 14:55:33.179596] I [worm.c:585:worm_lookup] 0-gv0-worm: /test/dir/test1.txtn_US.�

[2018-01-09 14:55:33.179610] I [worm.c:592:worm_lookup] 0-gv0-worm: Length of path: 24

[2018-01-09 14:55:33.179623] I [worm.c:594:worm_lookup] 0-gv0-worm: Length of name: 14

[2018-01-09 14:55:33.179635] I [worm.c:595:worm_lookup] 0-gv0-worm: /test/dir/test1.txt.test

[2018-01-09 14:55:33.179648] I [worm.c:596:worm_lookup] 0-gv0-worm: test1.txtn_US.�

But it should be test1.txt.

I think this should be enough to correctly change loc->path and loc->name.

[David Spisla] I am hanging around with that. I want to assign a new pointer to loc->path via „loc->path = new_path“, but that causes a memory crash:

 

[2018-01-09 15:14:02.340676] E [mem-pool.c:307:__gf_free] (-->/lib64/libglusterfs.so.0(args_wipe+0x12) [0x7f925414bb72] -->/lib64/libglusterfs.so.0(loc_wipe+0x27) [0x7f92540d2027] -->/lib64/libglusterfs.so.0(__gf_free+0xac) [0x7f92540fc6cc] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC == header->magic

 

What ist he best way to allocate a new pointer to loc->path?

 

Regards

David

 

Xavi

 

 

Any hints?

Regards

David Spisla

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel

 

 

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-devel