It's really cool. Do you prepare to push to upstream? I think it
should be more convenient if we make fio repo as submodule.
On Sat, Sep 12, 2015 at 5:04 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
> I found my problem why segment:
>
> because fio links librbd/librados from my /usr/local/lib but use
> ceph/src/.libs/libfio_ceph_objectstore.so. They are different ceph
> version.
>
> So maybe we need to add check for abi version?
>
> On Sat, Sep 12, 2015 at 4:08 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote:
>> Hi James,
>>
>> I just looked back at the results you posted, and saw that you were using iodepth=1. Setting this higher should help keep the FileStore busy.
>>
>> Casey
>>
>> ----- Original Message -----
>>> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> To: "Casey Bodley" <cbodley@xxxxxxxxxx>
>>> Cc: "Haomai Wang" <haomaiwang@xxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> Sent: Friday, September 11, 2015 1:18:31 PM
>>> Subject: RE: About Fio backend with ObjectStore API
>>>
>>> Hi Casey,
>>> You are right. I think the bottleneck is in fio side rather than in
>>> filestore side in this case. The fio did not issue the io commands faster
>>> enough to saturate the filestore.
>>> Here is one of possible solution for it: Create a async engine which are
>>> normally way faster than sync engine in fio.
>>>
>>> Here is possible framework. This new Objectstore-AIO engine in FIO in
>>> theory will be way faster than sync engine. Once we have FIO which can
>>> saturate newstore, memstore and filestore, we can investigate them in
>>> very details of where the bottleneck in their design.
>>>
>>> .........................
>>> struct objectstore_aio_data {
>>> struct aio_ctx *q_aio_ctx;
>>> struct aio_completion_data *a_data;
>>> aio_ses_ctx_t *p_ses_ctx;
>>> unsigned int entries;
>>> };
>>> ...................................
>>> /*
>>> * Note that the structure is exported, so that fio can get it via
>>> * dlsym(..., "ioengine");
>>> */
>>> struct ioengine_ops us_aio_ioengine = {
>>> .name = "objectstore-aio",
>>> .version = FIO_IOOPS_VERSION,
>>> .init = fio_objectstore_aio_init,
>>> .prep = fio_objectstore_aio_prep,
>>> .queue = fio_objectstore_aio_queue,
>>> .cancel = fio_objectstore_aio_cancel,
>>> .getevents = fio_objectstore_aio_getevents,
>>> .event = fio_objectstore_aio_event,
>>> .cleanup = fio_objectstore_aio_cleanup,
>>> .open_file = fio_objectstore_aio_open,
>>> .close_file = fio_objectstore_aio_close,
>>> };
>>>
>>>
>>> Let me know what you think.
>>>
>>> Regards,
>>> James
>>> ........................
>>> -----Original Message-----
>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxxx]
>>> Sent: Friday, September 11, 2015 7:28 AM
>>> To: James (Fei) Liu-SSI
>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> Subject: Re: About Fio backend with ObjectStore API
>>>
>>> Hi James,
>>>
>>> That's great that you were able to get fio-objectstore running! Thanks to you
>>> and Haomai for all the help with testing.
>>>
>>> In terms of performance, it's possible that we're not handling the
>>> completions optimally. When profiling with MemStore I remember seeing a
>>> significant amount of cpu time spent in polling with
>>> fio_ceph_os_getevents().
>>>
>>> The issue with reads is more of a design issue than a bug. Because the test
>>> starts with a mkfs(), there are no objects to read from initially. You would
>>> just have to add a write job to run before the read job, to make sure that
>>> the objects are initialized. Or perhaps the mkfs() step could be an optional
>>> part of the configuration.
>>>
>>> Casey
>>>
>>> ----- Original Message -----
>>> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley" <cbodley@xxxxxxxxxx>
>>> Cc: ceph-devel@xxxxxxxxxxxxxxx
>>> Sent: Thursday, September 10, 2015 8:08:04 PM
>>> Subject: RE: About Fio backend with ObjectStore API
>>>
>>> Hi Casey and Haomai,
>>>
>>> We finally made the fio-objectstore works in our end . Here is fio data
>>> against filestore with Samsung 850 Pro. It is sequential write and the
>>> performance is very poor which is expected though.
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=524288KB, aggrb=9467KB/s, minb=9467KB/s, maxb=9467KB/s,
>>> mint=55378msec, maxt=55378msec
>>>
>>> But anyway, it works even though still some bugs to fix like read and
>>> filesytem issues. thanks a lot for your great work.
>>>
>>> Regards,
>>> James
>>>
>>> jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_casey/src$ sudo ./fio/fio
>>> ./test/objectstore.fio
>>> filestore: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K,
>>> ioengine=cephobjectstore, iodepth=1 fio-2.2.9-56-g736a Starting 1 process
>>> test1
>>> filestore: Laying out IO file(s) (1 file(s) / 512MB)
>>> 2015-09-10 16:55:40.614494 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> mkfs in /home/jamesliu/fio_ceph
>>> 2015-09-10 16:55:40.614924 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> mkfs generated fsid 5508d58e-dbfc-48a5-9f9c-c639af4fe73a
>>> 2015-09-10 16:55:40.630326 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> write_version_stamp 4
>>> 2015-09-10 16:55:40.673417 7f19d34d1840 0 filestore(/home/jamesliu/fio_ceph)
>>> backend xfs (magic 0x58465342)
>>> 2015-09-10 16:55:40.724097 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> leveldb db exists/created
>>> 2015-09-10 16:55:40.724218 7f19d34d1840 -1 journal FileJournal::_open:
>>> disabling aio for non-block journal. Use journal_force_aio to force use of
>>> aio anyway
>>> 2015-09-10 16:55:40.724226 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 5: 5368709120 bytes, block size 4096 bytes,
>>> directio = 1, aio = 0
>>> 2015-09-10 16:55:40.724468 7f19d34d1840 -1 journal check: ondisk fsid
>>> 7580401a-6863-4863-9873-3adda08c9150 doesn't match expected
>>> 5508d58e-dbfc-48a5-9f9c-c639af4fe73a, invalid (someone else's?) journal
>>> 2015-09-10 16:55:40.724481 7f19d34d1840 1 journal close
>>> /tmp/fio_ceph_filestore1
>>> 2015-09-10 16:55:40.724506 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 5: 5368709120 bytes, block size 4096 bytes,
>>> directio = 1, aio = 0
>>> 2015-09-10 16:55:40.730417 7f19d34d1840 0 filestore(/home/jamesliu/fio_ceph)
>>> mkjournal created journal on /tmp/fio_ceph_filestore1
>>> 2015-09-10 16:55:40.730446 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> mkfs done in /home/jamesliu/fio_ceph
>>> 2015-09-10 16:55:40.730527 7f19d34d1840 0 filestore(/home/jamesliu/fio_ceph)
>>> backend xfs (magic 0x58465342)
>>> 2015-09-10 16:55:40.730773 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features: FIEMAP
>>> ioctl is disabled via 'filestore fiemap' config option
>>> 2015-09-10 16:55:40.730779 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option
>>> 2015-09-10 16:55:40.730793 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features: splice is
>>> supported
>>> 2015-09-10 16:55:40.751951 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features: syncfs(2)
>>> syscall fully supported (by glibc and kernel)
>>> 2015-09-10 16:55:40.752102 7f19d34d1840 0
>>> xfsfilestorebackend(/home/jamesliu/fio_ceph) detect_features: extsize is
>>> supported and your kernel >= 3.5
>>> 2015-09-10 16:55:40.794731 7f19d34d1840 0 filestore(/home/jamesliu/fio_ceph)
>>> mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
>>> 2015-09-10 16:55:40.794906 7f19d34d1840 -1 journal FileJournal::_open:
>>> disabling aio for non-block journal. Use journal_force_aio to force use of
>>> aio anyway
>>> 2015-09-10 16:55:40.794917 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 11: 5368709120 bytes, block size 4096 bytes,
>>> directio = 1, aio = 0
>>> 2015-09-10 16:55:40.795219 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 11: 5368709120 bytes, block size 4096 bytes,
>>> directio = 1, aio = 0
>>> 2015-09-10 16:55:40.795533 7f19d34d1840 1 filestore(/home/jamesliu/fio_ceph)
>>> upgrade
>>> 2015-09-10 16:56:36.420878 7f19d34d1840 1 journal close
>>> /tmp/fio_ceph_filestore10s]
>>>
>>> filestore: (groupid=0, jobs=1): err= 0: pid=16515: Thu Sep 10 16:56:36 2015
>>> write: io=524288KB, bw=9467.5KB/s, iops=73, runt= 55378msec
>>> slat (usec): min=22, max=142, avg=57.93, stdev=11.46
>>> clat (msec): min=6, max=148, avg=13.46, stdev=10.61
>>> lat (msec): min=6, max=148, avg=13.51, stdev=10.61
>>> clat percentiles (msec):
>>> | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10],
>>> | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
>>> | 70.00th=[ 10], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 29],
>>> | 99.00th=[ 43], 99.50th=[ 93], 99.90th=[ 137], 99.95th=[ 143],
>>> | 99.99th=[ 149]
>>> bw (KB /s): min= 1505, max=13750, per=100.00%, avg=9525.38,
>>> stdev=3821.47
>>> lat (msec) : 10=71.73%, 20=11.52%, 50=15.87%, 100=0.46%, 250=0.42%
>>> cpu : usr=1.50%, sys=1.87%, ctx=353850, majf=0, minf=48
>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=4096/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=524288KB, aggrb=9467KB/s, minb=9467KB/s, maxb=9467KB/s,
>>> mint=55378msec, maxt=55378msec
>>>
>>> Disk stats (read/write):
>>> loop0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>>>
>>>
>>> ________________________________________
>>> From: Haomai Wang [haomaiwang@xxxxxxxxx]
>>> Sent: Friday, September 04, 2015 8:43 PM
>>> To: James (Fei) Liu-SSI
>>> Cc: Casey Bodley; ceph-devel@xxxxxxxxxxxxxxx
>>> Subject: Re: About Fio backend with ObjectStore API
>>>
>>> On Sat, Sep 5, 2015 at 4:29 AM, James (Fei) Liu-SSI
>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> > Hi Casey,
>>> > Thanks. I even got a compiling error with fio-objectstore branch.
>>> >
>>> > Here is error message:
>>> >
>>> > make[3]: *** No rule to make target `rbd_fuse/rbd-fuse.c', needed by
>>> > `rbd_fuse/rbd-fuse.o'. Stop.
>>> > make[3]: *** Waiting for unfinished jobs....
>>> > CXX ceph_fuse.o
>>>
>>> This is a old make problem. I nearly forgot the details, but you can run
>>> "make clean" then try again. Or remove this ceph build directory directly
>>> and git clone a clean tree
>>>
>>> > make[3]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make[2]: *** [all-recursive] Error 1
>>> > make[2]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make[1]: *** [all] Error 2
>>> > make[1]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make: *** [all-recursive] Error 1
>>> > jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_fio$ git branch
>>> > * fio-objectstore
>>> > master
>>> >
>>> >
>>> > Regards,
>>> > James
>>> >
>>> > -----Original Message-----
>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxxx]
>>> > Sent: Thursday, September 03, 2015 10:44 AM
>>> > To: James (Fei) Liu-SSI
>>> > Cc: ceph-devel@xxxxxxxxxxxxxxx
>>> > Subject: Re: About Fio backend with ObjectStore API
>>> >
>>> > Hi James,
>>> >
>>> > I'm sorry for not following up on that segfault, but I wasn't ever able to
>>> > reproduce it. I used it recently for memstore testing without any
>>> > problems. I wonder if there's a problem with the autotools build? I've
>>> > only tested it with cmake. When I find some time, I'll rebase it on master
>>> > and do another round of testing.
>>> >
>>> > Casey
>>> >
>>> > ----- Original Message -----
>>> >> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> >> To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley"
>>> >> <cbodley@xxxxxxxxxx>
>>> >> Cc: "Casey Bodley" <cbodley@xxxxxxxxx>, "Matt W. Benjamin"
>>> >> <matt@xxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> >> Sent: Wednesday, September 2, 2015 8:06:14 PM
>>> >> Subject: RE: About Fio backend with ObjectStore API
>>> >>
>>> >> Hi Haomai and Case,
>>> >> Do you have any fixes for that segfault?
>>> >>
>>> >> Thanks,
>>> >> James
>>> >>
>>> >> -----Original Message-----
>>> >> From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx]
>>> >> Sent: Wednesday, July 22, 2015 6:07 PM
>>> >> To: Casey Bodley
>>> >> Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI;
>>> >> ceph-devel@xxxxxxxxxxxxxxx
>>> >> Subject: Re: About Fio backend with ObjectStore API
>>> >>
>>> >> no special
>>> >>
>>> >> [global]
>>> >> #logging
>>> >> #write_iops_log=write_iops_log
>>> >> #write_bw_log=write_bw_log
>>> >> #write_lat_log=write_lat_log
>>> >> ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so
>>> >> invalidate=0 # mandatory
>>> >> rw=write
>>> >> #bs=4k
>>> >>
>>> >> [filestore]
>>> >> iodepth=1
>>> >> # create a journaled filestore
>>> >> objectstore=filestore
>>> >> directory=./osd/
>>> >> filestore_journal=./osd/journal
>>> >>
>>> >> On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote:
>>> >> > Hi Haomai,
>>> >> >
>>> >> > Sorry for the late response, I was out of the office. I'm afraid I
>>> >> > haven't run into that segfault. The io_ops should be set at the
>>> >> > very beginning when it calls get_ioengine(). All I can suggest is
>>> >> > that you verify that your job file is pointing to the correct
>>> >> > fio_ceph_objectstore.so. If you've made any other interesting
>>> >> > changes to the job file, could you share it here?
>>> >> >
>>> >> > Casey
>>> >> >
>>> >> > ----- Original Message -----
>>> >> > From: "Haomai Wang" <haomaiwang@xxxxxxxxx>
>>> >> > To: "Casey Bodley" <cbodley@xxxxxxxxx>
>>> >> > Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI"
>>> >> > <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> >> > Sent: Tuesday, July 21, 2015 7:50:32 AM
>>> >> > Subject: Re: About Fio backend with ObjectStore API
>>> >> >
>>> >> > Hi Casey,
>>> >> >
>>> >> > I check your commits and know what you fixed. I cherry-picked your
>>> >> > new commits but I still met the same problem.
>>> >> >
>>> >> > """
>>> >> > It's strange that it alwasys hit segment fault when entering
>>> >> > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL
>>> >> > but when I up the stack, the "td->io_ops" is not null. Maybe it's
>>> >> > related to dlopen?
>>> >> > """
>>> >> >
>>> >> > Do you have any hint about this?
>>> >> >
>>> >> > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote:
>>> >> >> Hi Haomai,
>>> >> >>
>>> >> >> I was able to run this after a couple changes to the filestore.fio
>>> >> >> job file. Two of the config options were using the wrong names. I
>>> >> >> pushed a fix for the job file, as well as a patch that renames
>>> >> >> everything from filestore to objectstore (thanks James), to
>>> >> >> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore.
>>> >> >>
>>> >> >> I found that the read support doesn't appear to work anymore, so
>>> >> >> give "rw=write" a try. And because it does a mkfs(), make sure
>>> >> >> you're pointing it to an empty xfs directory with the "directory="
>>> >> >> option.
>>> >> >>
>>> >> >> Casey
>>> >> >>
>>> >> >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx>
>>> >> >> wrote:
>>> >> >>> Anyone who have successfully ran the fio with this external io
>>> >> >>> engine ceph_objectstore?
>>> >> >>>
>>> >> >>> It's strange that it alwasys hit segment fault when entering
>>> >> >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL
>>> >> >>> but when I up the stack, the "td->io_ops" is not null. Maybe it's
>>> >> >>> related to dlopen?
>>> >> >>>
>>> >> >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang
>>> >> >>> <haomaiwang@xxxxxxxxx>
>>> >> >>> wrote:
>>> >> >>>> I have rebased the branch with master, and push it to ceph
>>> >> >>>> upstream repo.
>>> >> >>>> https://github.com/ceph/ceph/compare/fio-objectstore?expand=1
>>> >> >>>>
>>> >> >>>> Plz let me know if who is working on this. Otherwise, I would
>>> >> >>>> like to improve this to be merge ready.
>>> >> >>>>
>>> >> >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin
>>> >> >>>> <matt@xxxxxxxxxxxx>
>>> >> >>>> wrote:
>>> >> >>>>> That makes sense.
>>> >> >>>>>
>>> >> >>>>> Matt
>>> >> >>>>>
>>> >> >>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>
>>> >> >>>>>> Hi Casey,
>>> >> >>>>>> Got it. I was directed to the old code base. By the way,
>>> >> >>>>>> Since the testing case was used to exercise all of object stores.
>>> >> >>>>>> Strongly recommend to change the name from
>>> >> >>>>>> fio_ceph_filestore.cc to fio_ceph_objectstore.cc . And the
>>> >> >>>>>> code in fio_ceph_filestore.cc should be refactored to reflect
>>> >> >>>>>> that the whole objectstore will be supported by
>>> >> >>>>>> fio_ceph_objectstore.cc. what you think?
>>> >> >>>>>>
>>> >> >>>>>> Let me know if you need any help from my side.
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> Regards,
>>> >> >>>>>> James
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> -----Original Message-----
>>> >> >>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> Sent: Thursday, July 09, 2015 12:32 PM
>>> >> >>>>>> To: James (Fei) Liu-SSI
>>> >> >>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>>
>>> >> >>>>>> Hi James,
>>> >> >>>>>>
>>> >> >>>>>> Are you looking at the code from
>>> >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore?
>>> >> >>>>>> It uses ObjectStore::create() instead of new FileStore(). This
>>> >> >>>>>> allows us to exercise all of the object stores with the same code.
>>> >> >>>>>>
>>> >> >>>>>> Casey
>>> >> >>>>>>
>>> >> >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> > Hi Casey,
>>> >> >>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically,
>>> >> >>>>>> > it
>>> >> >>>>>> creates a filestore as backend engine for IO exercises. If we
>>> >> >>>>>> got to send IO commands to KeyValue Store or Newstore, we got
>>> >> >>>>>> to change the code accordingly, right? I did not see any
>>> >> >>>>>> other files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc.
>>> >> >>>>>> In my humble opinion, we might need to create other two fio
>>> >> >>>>>> engines for keyvaluestore and newstore if we want to exercise
>>> >> >>>>>> these two, right?
>>> >> >>>>>> >
>>> >> >>>>>> > Regards,
>>> >> >>>>>> > James
>>> >> >>>>>> >
>>> >> >>>>>> > static int fio_ceph_filestore_init(struct thread_data *td)
>>> >> >>>>>> > 209 {
>>> >> >>>>>> > 210 vector<const char*> args;
>>> >> >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct
>>> >> >>>>>> ceph_filestore_data *) td->io_ops->data;
>>> >> >>>>>> > 212 ObjectStore::Transaction ft;
>>> >> >>>>>> > 213
>>> >> >>>>>> >
>>> >> >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD,
>>> >> >>>>>> CODE_ENVIRONMENT_UTILITY, 0);
>>> >> >>>>>> > 215 //g_conf->journal_dio = false;
>>> >> >>>>>> > 216 common_init_finish(g_ceph_context);
>>> >> >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore",
>>> >> >>>>>> > "20");
>>> >> >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20");
>>> >> >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL);
>>> >> >>>>>> > 220
>>> >> >>>>>> >
>>> >> >>>>>> > 221 ceph_filestore_data->osd_path =
>>> >> >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX");
>>> >> >>>>>> > 222 ceph_filestore_data->journal_path =
>>> >> >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"
>>> >> >>>>>> );
>>> >> >>>>>> > 223
>>> >> >>>>>> >
>>> >> >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) {
>>> >> >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) <<
>>> >> >>>>>> std::endl;
>>> >> >>>>>> > 226 return 1;
>>> >> >>>>>> > 227 }
>>> >> >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC
>>> >> >>>>>> > issue
>>> >> >>>>>> > 229
>>> >> >>>>>> >
>>> >> >>>>>> > 230 ObjectStore *fs = new
>>> >> >>>>>> FileStore(ceph_filestore_data->osd_path,
>>> >> >>>>>> ceph_filestore_data->journal_path);
>>> >> >>>>>> > 231 ceph_filestore_data->fs = fs;
>>> >> >>>>>> > 232
>>> >> >>>>>> >
>>> >> >>>>>> > 233 if (fs->mkfs() < 0) {
>>> >> >>>>>> > 234 cout << "mkfs failed" << std::endl;
>>> >> >>>>>> > 235 goto failed;
>>> >> >>>>>> > 236 }
>>> >> >>>>>> > 237
>>> >> >>>>>> > 238 if (fs->mount() < 0) {
>>> >> >>>>>> > 239 cout << "mount failed" << std::endl;
>>> >> >>>>>> > 240 goto failed;
>>> >> >>>>>> > 241 }
>>> >> >>>>>> > 242
>>> >> >>>>>> >
>>> >> >>>>>> > 243 ft.create_collection(coll_t());
>>> >> >>>>>> > 244 fs->apply_transaction(ft);
>>> >> >>>>>> > 245
>>> >> >>>>>> >
>>> >> >>>>>> > 246
>>> >> >>>>>> >
>>> >> >>>>>> > 247 return 0;
>>> >> >>>>>> > 248
>>> >> >>>>>> >
>>> >> >>>>>> > 249 failed:
>>> >> >>>>>> > 250 return 1;
>>> >> >>>>>> > 251
>>> >> >>>>>> >
>>> >> >>>>>> > 252 }
>>> >> >>>>>> > -----Original Message-----
>>> >> >>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM
>>> >> >>>>>> > To: James (Fei) Liu-SSI
>>> >> >>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> > Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>> >
>>> >> >>>>>> > Hi James,
>>> >> >>>>>> >
>>> >> >>>>>> > In the job file src/test/filestore.fio, you can modify the
>>> >> >>>>>> > line "objectstore=filestore" to use any objectstore type
>>> >> >>>>>> > supported by
>>> >> >>>>>> the
>>> >> >>>>>> > ObjectStore::create() factory.
>>> >> >>>>>> >
>>> >> >>>>>> > Casey
>>> >> >>>>>> >
>>> >> >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> >> Hi Casey,
>>> >> >>>>>> >> Quick questions, The code in the trunk only cover the
>>> >> >>>>>> >> test for
>>> >> >>>>>> filestore. I was wondering do you have any plan to cover the
>>> >> >>>>>> test for kvstore and newstore?
>>> >> >>>>>> >>
>>> >> >>>>>> >> Thanks,
>>> >> >>>>>> >> James
>>> >> >>>>>> >>
>>> >> >>>>>> >> -----Original Message-----
>>> >> >>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of
>>> >> >>>>>> >> James
>>> >> >>>>>> >> (Fei)
>>> >> >>>>>>
>>> >> >>>>>> >> Liu-SSI
>>> >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM
>>> >> >>>>>> >> To: Casey Bodley
>>> >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> Subject: RE: About Fio backend with ObjectStore API
>>> >> >>>>>> >>
>>> >> >>>>>> >> Hi Casey,
>>> >> >>>>>> >>
>>> >> >>>>>> >> Thanks a lot.
>>> >> >>>>>> >>
>>> >> >>>>>> >> Regards,
>>> >> >>>>>> >> James
>>> >> >>>>>> >>
>>> >> >>>>>> >> -----Original Message-----
>>> >> >>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM
>>> >> >>>>>> >> To: James (Fei) Liu-SSI
>>> >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>> >>
>>> >> >>>>>> >> Hi,
>>> >> >>>>>> >>
>>> >> >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph
>>> >> >>>>>> >> Performance
>>> >> >>>>>> >> Analysis: fio and RBD" at
>>> >> >>>>>> >>
>>> >> >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performanc
>>> >> >>>>>> e
>>> >> >>>>>> -a
>>> >> >>>>>> naly
>>> >> >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that
>>> >> >>>>>> >> linked directly into ceph's FileStore. I was able to find
>>> >> >>>>>> >> Daniel's branch
>>> >> >>>>>> on
>>> >> >>>>>> >> github at
>>> >> >>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2, and
>>> >> >>>>>> did some more work on it at the time.
>>> >> >>>>>> >>
>>> >> >>>>>> >> I just rebased that work onto the latest ceph master
>>> >> >>>>>> >> branch, and
>>> >> >>>>>> pushed to our github at
>>> >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore.
>>> >> >>>>>> You can find the source in src/test/fio_ceph_filestore.cc, and
>>> >> >>>>>> run fio with the provided example fio job file in
>>> >> >>>>>> src/test/filestore.fio.
>>> >> >>>>>> >>
>>> >> >>>>>> >> I didn't have a chance to confirm that it builds with
>>> >> >>>>>> >> automake, but
>>> >> >>>>>>
>>> >> >>>>>> >> the cmake version built for me. I'm happy to help if you
>>> >> >>>>>> >> run into problems, Casey
>>> >> >>>>>> >>
>>> >> >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> >>> Hi Haomai,
>>> >> >>>>>> >>> What are you trying to ask is to benchmark local
>>> >> >>>>>> objectstore(like kvstore/filestore/newstore) locally with
>>> >> >>>>>> FIO(ObjectStore engine)? You want to purely compare the
>>> >> >>>>>> performance locally for these objectstores, right?
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Regards,
>>> >> >>>>>> >>> James
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> -----Original Message-----
>>> >> >>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>> >> >>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of
>>> >> >>>>>> >>> Haomai
>>> >> >>>>>> Wang
>>> >> >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM
>>> >> >>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >>> Subject: About Fio backend with ObjectStore API
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Hi all,
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Long long ago, is there someone said about fio backend
>>> >> >>>>>> >>> with Ceph
>>> >> >>>>>> ObjectStore API? So we could use the existing mature fio
>>> >> >>>>>> facility to benchmark ceph objectstore.
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> --
>>> >> >>>>>> >>> Best Regards,
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Wheat
>>> >> >>>>>> >>> --
>>> >> >>>>>> >>> To unsubscribe from this list: send the line "unsubscribe
>>> >> >>>>>> ceph-devel"
>>> >> >>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More
>>> >> >>>>>> majordomo
>>> >> >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html
>>> >> >>>>>> >> {.n + +% lzwm b ? r y? ?zX ?} ?z &j:+v
>>> >> >>>>>> >> zZ+
>>> >> >>>>>> +zf h ~ i z w ? & )? f
>>> >> >>>>>> N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w
>>> >> >>>>>> j:+v w j m zZ+ ?j"
>>> >> >>>>>
>>> >> >>>>> --
>>> >> >>>>> Matt Benjamin
>>> >> >>>>> CohortFS, LLC.
>>> >> >>>>> 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
>>> >> >>>>>
>>> >> >>>>> http://cohortfs.com
>>> >> >>>>>
>>> >> >>>>> tel. 734-761-4689
>>> >> >>>>> fax. 734-769-8938
>>> >> >>>>> cel. 734-216-5309
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>> Best Regards,
>>> >> >>>>
>>> >> >>>> Wheat
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> Best Regards,
>>> >> >>>
>>> >> >>> Wheat
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Best Regards,
>>> >> >
>>> >> > Wheat
>>> >> > --
>>> >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> >> > in the body of a message to majordomo@xxxxxxxxxxxxxxx More
>>> >> > majordomo info at http://vger.kernel.org/majordomo-info.html
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >>
>>> >> Wheat
>>> >> N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w j
>>> >> :+v w j m zZ+ ?j"
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>>> body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at
>>> http://vger.kernel.org/majordomo-info.html
>>> N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������j:+v���w�j�m��������zZ+��ݢj"��
>
>
>
> --
> Best Regards,
>
> Wheat
--
Best Regards,
Wheat
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
