On Dec 15, 2014, at 4:10 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > On Mon, Dec 15, 2014 at 12:06 PM, Atchley, Scott <atchleyes@xxxxxxxx> wrote: >> On Dec 15, 2014, at 2:42 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >> >>> On Mon, Dec 15, 2014 at 10:54 AM, Atchley, Scott <atchleyes@xxxxxxxx> wrote: >>>> Hi all, >>>> >>>> For a given file in cephfs, I would like to determine: >>>> >>>> 1) the number of PGs >>>> 2) the PG IDs >>>> 3) the offsets handles by each PG >>>> 4) the stripe unit (i.e. bytes per block of data) >>>> >>>> preferably using a C API. >>>> >>>> I found the getfattr command line tool which provides (1) and (4). >>>> >>>> It appears that cephfs command line tool provides some of the info, but the manpage is not clear as to what is provided by show_layout versus show_location. Does show_layout provide (1) and (4)? Does show_location include (2) and (3)? >>>> >>>> I found a Java CephFileExtent() that provides (1) and a list of OSDs for the stripe unit that contains the provided file extent. >>> >>> What exactly are you trying to accomplish here? I can imagine some >>> limited scenarios where you want to map to PGs, but in general you're >>> a lot more interested in either the objects associated with the file, >>> or the OSDs that host them. I think you can get objects and OSDs out >>> of the latest libcephfs api. >> >> We implemented a file transfer tool for Lustre that transfers data by object rather than by file. The scheduling of the object transfers is done in a way to defer transfers from congested servers in hopes that they will be less congested in the near future (given our bulk-synchronous workload, congestion of a set of servers is generally temporary). >> >> I would rather know OSDs versus PGs, but I did not assume that the info was available. > > Yeah, look at ceph_get_file_extent_osds in libcephfs. :) Ok, thanks for the pointer. It looks like I can get (1) with ceph_get_file_stripe_count(). I can get (2) and (3) with ceph_get_file_extent_osds(), but I need the stripe unit length (4) first. It seems that ceph_get_file_stripe_unit() will prove item (4). Or I can use ceph_get_file_layout() to get items (1) and (4). I think I can get what I need with these. >>>> On the architecture page (http://ceph.com/docs/next/architecture/) under DATA STRIPING, does cephfs client use the same set of PGs used for Object Set 1 for Object Set 2 or does it use a different PGs? >>> >>> Assuming you mean does it use the same PGs for each file's objects, >>> the answer is no, definitely not. >>> -Greg >> >> No, I assume the Object Set 1 on that page indicates four objects on four different PGs. Within Object Set 1, Object 0 holds stripe units 0, 4, 8, and 12. Object 1 holds stripe units 1, 5, 9, and 13 and so on. Each Object 0-3 are stored on different PGs. Is this correct so far? >> >> My question above is about Object Set 2, which holds Objects 4-7? Is Object 4 on the same PG as Object 0? Is Object 5 on the same PG as Object 1? Etc? Or does Object Set 2 (holding Objects 4-7) use different PGs? > > The sets use completely different names which are (generally) going to > be placed completely differently. They're just <inode number>.<object > number>, where the inode number is in hex and the object number is > zero-padded up to some size. > -Greg I am still not fully understanding object sets. Can a file have more than one object set? Scott -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
