Hi Casey,
I also make the fio_objectstore to support kevvaluestore. Attachment is the patch for supporting keyvaluestore(LevelDB). More parameters should be added if we want to support RocksDB with more parameters. It would be great if you can put fio-objectstore into upstream as suggested by Haomai.
Hi Haomai,
The performance with KeyValueStore(LevelDB) is only half of filestore roughly with similar FIO configuration.
Regards,
James
-----Original Message-----
From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx]
Sent: Saturday, September 12, 2015 2:16 AM
To: Casey Bodley
Cc: James (Fei) Liu-SSI; ceph-devel@xxxxxxxxxxxxxxx
Subject: Re: About Fio backend with ObjectStore API
It's really cool. Do you prepare to push to upstream? I think it should be more convenient if we make fio repo as submodule.
On Sat, Sep 12, 2015 at 5:04 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
> I found my problem why segment:
>
> because fio links librbd/librados from my /usr/local/lib but use
> ceph/src/.libs/libfio_ceph_objectstore.so. They are different ceph
> version.
>
> So maybe we need to add check for abi version?
>
> On Sat, Sep 12, 2015 at 4:08 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote:
>> Hi James,
>>
>> I just looked back at the results you posted, and saw that you were using iodepth=1. Setting this higher should help keep the FileStore busy.
>>
>> Casey
>>
>> ----- Original Message -----
>>> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> To: "Casey Bodley" <cbodley@xxxxxxxxxx>
>>> Cc: "Haomai Wang" <haomaiwang@xxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> Sent: Friday, September 11, 2015 1:18:31 PM
>>> Subject: RE: About Fio backend with ObjectStore API
>>>
>>> Hi Casey,
>>> You are right. I think the bottleneck is in fio side rather than in
>>> filestore side in this case. The fio did not issue the io commands faster
>>> enough to saturate the filestore.
>>> Here is one of possible solution for it: Create a async engine which are
>>> normally way faster than sync engine in fio.
>>>
>>> Here is possible framework. This new Objectstore-AIO engine in FIO in
>>> theory will be way faster than sync engine. Once we have FIO which can
>>> saturate newstore, memstore and filestore, we can investigate them in
>>> very details of where the bottleneck in their design.
>>>
>>> .........................
>>> struct objectstore_aio_data {
>>> struct aio_ctx *q_aio_ctx;
>>> struct aio_completion_data *a_data;
>>> aio_ses_ctx_t *p_ses_ctx;
>>> unsigned int entries;
>>> };
>>> ...................................
>>> /*
>>> * Note that the structure is exported, so that fio can get it via
>>> * dlsym(..., "ioengine");
>>> */
>>> struct ioengine_ops us_aio_ioengine = {
>>> .name = "objectstore-aio",
>>> .version = FIO_IOOPS_VERSION,
>>> .init = fio_objectstore_aio_init,
>>> .prep = fio_objectstore_aio_prep,
>>> .queue = fio_objectstore_aio_queue,
>>> .cancel = fio_objectstore_aio_cancel,
>>> .getevents = fio_objectstore_aio_getevents,
>>> .event = fio_objectstore_aio_event,
>>> .cleanup = fio_objectstore_aio_cleanup,
>>> .open_file = fio_objectstore_aio_open,
>>> .close_file = fio_objectstore_aio_close,
>>> };
>>>
>>>
>>> Let me know what you think.
>>>
>>> Regards,
>>> James
>>> ........................
>>> -----Original Message-----
>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxxx]
>>> Sent: Friday, September 11, 2015 7:28 AM
>>> To: James (Fei) Liu-SSI
>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> Subject: Re: About Fio backend with ObjectStore API
>>>
>>> Hi James,
>>>
>>> That's great that you were able to get fio-objectstore running!
>>> Thanks to you and Haomai for all the help with testing.
>>>
>>> In terms of performance, it's possible that we're not handling the
>>> completions optimally. When profiling with MemStore I remember
>>> seeing a significant amount of cpu time spent in polling with
>>> fio_ceph_os_getevents().
>>>
>>> The issue with reads is more of a design issue than a bug. Because
>>> the test starts with a mkfs(), there are no objects to read from
>>> initially. You would just have to add a write job to run before the
>>> read job, to make sure that the objects are initialized. Or perhaps
>>> the mkfs() step could be an optional part of the configuration.
>>>
>>> Casey
>>>
>>> ----- Original Message -----
>>> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley"
>>> <cbodley@xxxxxxxxxx>
>>> Cc: ceph-devel@xxxxxxxxxxxxxxx
>>> Sent: Thursday, September 10, 2015 8:08:04 PM
>>> Subject: RE: About Fio backend with ObjectStore API
>>>
>>> Hi Casey and Haomai,
>>>
>>> We finally made the fio-objectstore works in our end . Here is fio data
>>> against filestore with Samsung 850 Pro. It is sequential write and the
>>> performance is very poor which is expected though.
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=524288KB, aggrb=9467KB/s, minb=9467KB/s, maxb=9467KB/s,
>>> mint=55378msec, maxt=55378msec
>>>
>>> But anyway, it works even though still some bugs to fix like read and
>>> filesytem issues. thanks a lot for your great work.
>>>
>>> Regards,
>>> James
>>>
>>> jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_casey/src$ sudo ./fio/fio
>>> ./test/objectstore.fio
>>> filestore: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K,
>>> ioengine=cephobjectstore, iodepth=1 fio-2.2.9-56-g736a Starting 1
>>> process
>>> test1
>>> filestore: Laying out IO file(s) (1 file(s) / 512MB)
>>> 2015-09-10 16:55:40.614494 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> mkfs in /home/jamesliu/fio_ceph
>>> 2015-09-10 16:55:40.614924 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> mkfs generated fsid 5508d58e-dbfc-48a5-9f9c-c639af4fe73a
>>> 2015-09-10 16:55:40.630326 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> write_version_stamp 4
>>> 2015-09-10 16:55:40.673417 7f19d34d1840 0
>>> filestore(/home/jamesliu/fio_ceph)
>>> backend xfs (magic 0x58465342)
>>> 2015-09-10 16:55:40.724097 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> leveldb db exists/created
>>> 2015-09-10 16:55:40.724218 7f19d34d1840 -1 journal FileJournal::_open:
>>> disabling aio for non-block journal. Use journal_force_aio to force
>>> use of aio anyway
>>> 2015-09-10 16:55:40.724226 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 5: 5368709120 bytes, block size 4096
>>> bytes, directio = 1, aio = 0
>>> 2015-09-10 16:55:40.724468 7f19d34d1840 -1 journal check: ondisk
>>> fsid
>>> 7580401a-6863-4863-9873-3adda08c9150 doesn't match expected
>>> 5508d58e-dbfc-48a5-9f9c-c639af4fe73a, invalid (someone else's?)
>>> journal
>>> 2015-09-10 16:55:40.724481 7f19d34d1840 1 journal close
>>> /tmp/fio_ceph_filestore1
>>> 2015-09-10 16:55:40.724506 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 5: 5368709120 bytes, block size 4096
>>> bytes, directio = 1, aio = 0
>>> 2015-09-10 16:55:40.730417 7f19d34d1840 0
>>> filestore(/home/jamesliu/fio_ceph)
>>> mkjournal created journal on /tmp/fio_ceph_filestore1
>>> 2015-09-10 16:55:40.730446 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> mkfs done in /home/jamesliu/fio_ceph
>>> 2015-09-10 16:55:40.730527 7f19d34d1840 0
>>> filestore(/home/jamesliu/fio_ceph)
>>> backend xfs (magic 0x58465342)
>>> 2015-09-10 16:55:40.730773 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> FIEMAP ioctl is disabled via 'filestore fiemap' config option
>>> 2015-09-10 16:55:40.730779 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole'
>>> config option
>>> 2015-09-10 16:55:40.730793 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> splice is supported
>>> 2015-09-10 16:55:40.751951 7f19d34d1840 0
>>> genericfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> syncfs(2) syscall fully supported (by glibc and kernel)
>>> 2015-09-10 16:55:40.752102 7f19d34d1840 0
>>> xfsfilestorebackend(/home/jamesliu/fio_ceph) detect_features:
>>> extsize is supported and your kernel >= 3.5
>>> 2015-09-10 16:55:40.794731 7f19d34d1840 0
>>> filestore(/home/jamesliu/fio_ceph)
>>> mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
>>> 2015-09-10 16:55:40.794906 7f19d34d1840 -1 journal FileJournal::_open:
>>> disabling aio for non-block journal. Use journal_force_aio to force
>>> use of aio anyway
>>> 2015-09-10 16:55:40.794917 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 11: 5368709120 bytes, block size 4096
>>> bytes, directio = 1, aio = 0
>>> 2015-09-10 16:55:40.795219 7f19d34d1840 1 journal _open
>>> /tmp/fio_ceph_filestore1 fd 11: 5368709120 bytes, block size 4096
>>> bytes, directio = 1, aio = 0
>>> 2015-09-10 16:55:40.795533 7f19d34d1840 1
>>> filestore(/home/jamesliu/fio_ceph)
>>> upgrade
>>> 2015-09-10 16:56:36.420878 7f19d34d1840 1 journal close
>>> /tmp/fio_ceph_filestore10s]
>>>
>>> filestore: (groupid=0, jobs=1): err= 0: pid=16515: Thu Sep 10 16:56:36 2015
>>> write: io=524288KB, bw=9467.5KB/s, iops=73, runt= 55378msec
>>> slat (usec): min=22, max=142, avg=57.93, stdev=11.46
>>> clat (msec): min=6, max=148, avg=13.46, stdev=10.61
>>> lat (msec): min=6, max=148, avg=13.51, stdev=10.61
>>> clat percentiles (msec):
>>> | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10],
>>> | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
>>> | 70.00th=[ 10], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 29],
>>> | 99.00th=[ 43], 99.50th=[ 93], 99.90th=[ 137], 99.95th=[ 143],
>>> | 99.99th=[ 149]
>>> bw (KB /s): min= 1505, max=13750, per=100.00%, avg=9525.38,
>>> stdev=3821.47
>>> lat (msec) : 10=71.73%, 20=11.52%, 50=15.87%, 100=0.46%, 250=0.42%
>>> cpu : usr=1.50%, sys=1.87%, ctx=353850, majf=0, minf=48
>>> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>> >=64=0.0%
>>> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>> >=64=0.0%
>>> issued : total=r=0/w=4096/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>>> latency : target=0, window=0, percentile=100.00%, depth=1
>>>
>>> Run status group 0 (all jobs):
>>> WRITE: io=524288KB, aggrb=9467KB/s, minb=9467KB/s, maxb=9467KB/s,
>>> mint=55378msec, maxt=55378msec
>>>
>>> Disk stats (read/write):
>>> loop0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>>>
>>>
>>> ________________________________________
>>> From: Haomai Wang [haomaiwang@xxxxxxxxx]
>>> Sent: Friday, September 04, 2015 8:43 PM
>>> To: James (Fei) Liu-SSI
>>> Cc: Casey Bodley; ceph-devel@xxxxxxxxxxxxxxx
>>> Subject: Re: About Fio backend with ObjectStore API
>>>
>>> On Sat, Sep 5, 2015 at 4:29 AM, James (Fei) Liu-SSI
>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> > Hi Casey,
>>> > Thanks. I even got a compiling error with fio-objectstore branch.
>>> >
>>> > Here is error message:
>>> >
>>> > make[3]: *** No rule to make target `rbd_fuse/rbd-fuse.c', needed by
>>> > `rbd_fuse/rbd-fuse.o'. Stop.
>>> > make[3]: *** Waiting for unfinished jobs....
>>> > CXX ceph_fuse.o
>>>
>>> This is a old make problem. I nearly forgot the details, but you can
>>> run "make clean" then try again. Or remove this ceph build directory
>>> directly and git clone a clean tree
>>>
>>> > make[3]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make[2]: *** [all-recursive] Error 1
>>> > make[2]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make[1]: *** [all] Error 2
>>> > make[1]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src'
>>> > make: *** [all-recursive] Error 1
>>> > jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_fio$ git branch
>>> > * fio-objectstore
>>> > master
>>> >
>>> >
>>> > Regards,
>>> > James
>>> >
>>> > -----Original Message-----
>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxxx]
>>> > Sent: Thursday, September 03, 2015 10:44 AM
>>> > To: James (Fei) Liu-SSI
>>> > Cc: ceph-devel@xxxxxxxxxxxxxxx
>>> > Subject: Re: About Fio backend with ObjectStore API
>>> >
>>> > Hi James,
>>> >
>>> > I'm sorry for not following up on that segfault, but I wasn't ever
>>> > able to reproduce it. I used it recently for memstore testing
>>> > without any problems. I wonder if there's a problem with the
>>> > autotools build? I've only tested it with cmake. When I find some
>>> > time, I'll rebase it on master and do another round of testing.
>>> >
>>> > Casey
>>> >
>>> > ----- Original Message -----
>>> >> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>
>>> >> To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley"
>>> >> <cbodley@xxxxxxxxxx>
>>> >> Cc: "Casey Bodley" <cbodley@xxxxxxxxx>, "Matt W. Benjamin"
>>> >> <matt@xxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> >> Sent: Wednesday, September 2, 2015 8:06:14 PM
>>> >> Subject: RE: About Fio backend with ObjectStore API
>>> >>
>>> >> Hi Haomai and Case,
>>> >> Do you have any fixes for that segfault?
>>> >>
>>> >> Thanks,
>>> >> James
>>> >>
>>> >> -----Original Message-----
>>> >> From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx]
>>> >> Sent: Wednesday, July 22, 2015 6:07 PM
>>> >> To: Casey Bodley
>>> >> Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI;
>>> >> ceph-devel@xxxxxxxxxxxxxxx
>>> >> Subject: Re: About Fio backend with ObjectStore API
>>> >>
>>> >> no special
>>> >>
>>> >> [global]
>>> >> #logging
>>> >> #write_iops_log=write_iops_log
>>> >> #write_bw_log=write_bw_log
>>> >> #write_lat_log=write_lat_log
>>> >> ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so
>>> >> invalidate=0 # mandatory
>>> >> rw=write
>>> >> #bs=4k
>>> >>
>>> >> [filestore]
>>> >> iodepth=1
>>> >> # create a journaled filestore
>>> >> objectstore=filestore
>>> >> directory=./osd/
>>> >> filestore_journal=./osd/journal
>>> >>
>>> >> On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote:
>>> >> > Hi Haomai,
>>> >> >
>>> >> > Sorry for the late response, I was out of the office. I'm
>>> >> > afraid I haven't run into that segfault. The io_ops should be
>>> >> > set at the very beginning when it calls get_ioengine(). All I
>>> >> > can suggest is that you verify that your job file is pointing
>>> >> > to the correct fio_ceph_objectstore.so. If you've made any
>>> >> > other interesting changes to the job file, could you share it here?
>>> >> >
>>> >> > Casey
>>> >> >
>>> >> > ----- Original Message -----
>>> >> > From: "Haomai Wang" <haomaiwang@xxxxxxxxx>
>>> >> > To: "Casey Bodley" <cbodley@xxxxxxxxx>
>>> >> > Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI"
>>> >> > <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
>>> >> > Sent: Tuesday, July 21, 2015 7:50:32 AM
>>> >> > Subject: Re: About Fio backend with ObjectStore API
>>> >> >
>>> >> > Hi Casey,
>>> >> >
>>> >> > I check your commits and know what you fixed. I cherry-picked
>>> >> > your new commits but I still met the same problem.
>>> >> >
>>> >> > """
>>> >> > It's strange that it alwasys hit segment fault when entering
>>> >> > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is
>>> >> > NULL but when I up the stack, the "td->io_ops" is not null.
>>> >> > Maybe it's related to dlopen?
>>> >> > """
>>> >> >
>>> >> > Do you have any hint about this?
>>> >> >
>>> >> > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote:
>>> >> >> Hi Haomai,
>>> >> >>
>>> >> >> I was able to run this after a couple changes to the
>>> >> >> filestore.fio job file. Two of the config options were using
>>> >> >> the wrong names. I pushed a fix for the job file, as well as a
>>> >> >> patch that renames everything from filestore to objectstore
>>> >> >> (thanks James), to https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore.
>>> >> >>
>>> >> >> I found that the read support doesn't appear to work anymore,
>>> >> >> so give "rw=write" a try. And because it does a mkfs(), make
>>> >> >> sure you're pointing it to an empty xfs directory with the "directory="
>>> >> >> option.
>>> >> >>
>>> >> >> Casey
>>> >> >>
>>> >> >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang
>>> >> >> <haomaiwang@xxxxxxxxx>
>>> >> >> wrote:
>>> >> >>> Anyone who have successfully ran the fio with this external
>>> >> >>> io engine ceph_objectstore?
>>> >> >>>
>>> >> >>> It's strange that it alwasys hit segment fault when entering
>>> >> >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is
>>> >> >>> NULL but when I up the stack, the "td->io_ops" is not null.
>>> >> >>> Maybe it's related to dlopen?
>>> >> >>>
>>> >> >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang
>>> >> >>> <haomaiwang@xxxxxxxxx>
>>> >> >>> wrote:
>>> >> >>>> I have rebased the branch with master, and push it to ceph
>>> >> >>>> upstream repo.
>>> >> >>>> https://github.com/ceph/ceph/compare/fio-objectstore?expand=
>>> >> >>>> 1
>>> >> >>>>
>>> >> >>>> Plz let me know if who is working on this. Otherwise, I
>>> >> >>>> would like to improve this to be merge ready.
>>> >> >>>>
>>> >> >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin
>>> >> >>>> <matt@xxxxxxxxxxxx>
>>> >> >>>> wrote:
>>> >> >>>>> That makes sense.
>>> >> >>>>>
>>> >> >>>>> Matt
>>> >> >>>>>
>>> >> >>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>
>>> >> >>>>>> Hi Casey,
>>> >> >>>>>> Got it. I was directed to the old code base. By the way,
>>> >> >>>>>> Since the testing case was used to exercise all of object stores.
>>> >> >>>>>> Strongly recommend to change the name from
>>> >> >>>>>> fio_ceph_filestore.cc to fio_ceph_objectstore.cc . And the
>>> >> >>>>>> code in fio_ceph_filestore.cc should be refactored to
>>> >> >>>>>> reflect that the whole objectstore will be supported by
>>> >> >>>>>> fio_ceph_objectstore.cc. what you think?
>>> >> >>>>>>
>>> >> >>>>>> Let me know if you need any help from my side.
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> Regards,
>>> >> >>>>>> James
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>>
>>> >> >>>>>> -----Original Message-----
>>> >> >>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> Sent: Thursday, July 09, 2015 12:32 PM
>>> >> >>>>>> To: James (Fei) Liu-SSI
>>> >> >>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>>
>>> >> >>>>>> Hi James,
>>> >> >>>>>>
>>> >> >>>>>> Are you looking at the code from
>>> >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore?
>>> >> >>>>>> It uses ObjectStore::create() instead of new FileStore().
>>> >> >>>>>> This allows us to exercise all of the object stores with the same code.
>>> >> >>>>>>
>>> >> >>>>>> Casey
>>> >> >>>>>>
>>> >> >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> > Hi Casey,
>>> >> >>>>>> > Here is the code in the fio_ceph_filestore.cc.
>>> >> >>>>>> > Basically, it
>>> >> >>>>>> creates a filestore as backend engine for IO exercises. If
>>> >> >>>>>> we got to send IO commands to KeyValue Store or Newstore,
>>> >> >>>>>> we got to change the code accordingly, right? I did not
>>> >> >>>>>> see any other files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc.
>>> >> >>>>>> In my humble opinion, we might need to create other two
>>> >> >>>>>> fio engines for keyvaluestore and newstore if we want to
>>> >> >>>>>> exercise these two, right?
>>> >> >>>>>> >
>>> >> >>>>>> > Regards,
>>> >> >>>>>> > James
>>> >> >>>>>> >
>>> >> >>>>>> > static int fio_ceph_filestore_init(struct thread_data
>>> >> >>>>>> > *td)
>>> >> >>>>>> > 209 {
>>> >> >>>>>> > 210 vector<const char*> args;
>>> >> >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct
>>> >> >>>>>> ceph_filestore_data *) td->io_ops->data;
>>> >> >>>>>> > 212 ObjectStore::Transaction ft;
>>> >> >>>>>> > 213
>>> >> >>>>>> >
>>> >> >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD,
>>> >> >>>>>> CODE_ENVIRONMENT_UTILITY, 0);
>>> >> >>>>>> > 215 //g_conf->journal_dio = false;
>>> >> >>>>>> > 216 common_init_finish(g_ceph_context);
>>> >> >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore",
>>> >> >>>>>> > "20");
>>> >> >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20");
>>> >> >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL);
>>> >> >>>>>> > 220
>>> >> >>>>>> >
>>> >> >>>>>> > 221 ceph_filestore_data->osd_path =
>>> >> >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX");
>>> >> >>>>>> > 222 ceph_filestore_data->journal_path =
>>> >> >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"
>>> >> >>>>>> );
>>> >> >>>>>> > 223
>>> >> >>>>>> >
>>> >> >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) {
>>> >> >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) <<
>>> >> >>>>>> std::endl;
>>> >> >>>>>> > 226 return 1;
>>> >> >>>>>> > 227 }
>>> >> >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC
>>> >> >>>>>> > issue
>>> >> >>>>>> > 229
>>> >> >>>>>> >
>>> >> >>>>>> > 230 ObjectStore *fs = new
>>> >> >>>>>> FileStore(ceph_filestore_data->osd_path,
>>> >> >>>>>> ceph_filestore_data->journal_path);
>>> >> >>>>>> > 231 ceph_filestore_data->fs = fs;
>>> >> >>>>>> > 232
>>> >> >>>>>> >
>>> >> >>>>>> > 233 if (fs->mkfs() < 0) {
>>> >> >>>>>> > 234 cout << "mkfs failed" << std::endl;
>>> >> >>>>>> > 235 goto failed;
>>> >> >>>>>> > 236 }
>>> >> >>>>>> > 237
>>> >> >>>>>> > 238 if (fs->mount() < 0) {
>>> >> >>>>>> > 239 cout << "mount failed" << std::endl;
>>> >> >>>>>> > 240 goto failed;
>>> >> >>>>>> > 241 }
>>> >> >>>>>> > 242
>>> >> >>>>>> >
>>> >> >>>>>> > 243 ft.create_collection(coll_t());
>>> >> >>>>>> > 244 fs->apply_transaction(ft);
>>> >> >>>>>> > 245
>>> >> >>>>>> >
>>> >> >>>>>> > 246
>>> >> >>>>>> >
>>> >> >>>>>> > 247 return 0;
>>> >> >>>>>> > 248
>>> >> >>>>>> >
>>> >> >>>>>> > 249 failed:
>>> >> >>>>>> > 250 return 1;
>>> >> >>>>>> > 251
>>> >> >>>>>> >
>>> >> >>>>>> > 252 }
>>> >> >>>>>> > -----Original Message-----
>>> >> >>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM
>>> >> >>>>>> > To: James (Fei) Liu-SSI
>>> >> >>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> > Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>> >
>>> >> >>>>>> > Hi James,
>>> >> >>>>>> >
>>> >> >>>>>> > In the job file src/test/filestore.fio, you can modify
>>> >> >>>>>> > the line "objectstore=filestore" to use any objectstore
>>> >> >>>>>> > type supported by
>>> >> >>>>>> the
>>> >> >>>>>> > ObjectStore::create() factory.
>>> >> >>>>>> >
>>> >> >>>>>> > Casey
>>> >> >>>>>> >
>>> >> >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> >> Hi Casey,
>>> >> >>>>>> >> Quick questions, The code in the trunk only cover the
>>> >> >>>>>> >> test for
>>> >> >>>>>> filestore. I was wondering do you have any plan to cover
>>> >> >>>>>> the test for kvstore and newstore?
>>> >> >>>>>> >>
>>> >> >>>>>> >> Thanks,
>>> >> >>>>>> >> James
>>> >> >>>>>> >>
>>> >> >>>>>> >> -----Original Message-----
>>> >> >>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of
>>> >> >>>>>> >> James
>>> >> >>>>>> >> (Fei)
>>> >> >>>>>>
>>> >> >>>>>> >> Liu-SSI
>>> >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM
>>> >> >>>>>> >> To: Casey Bodley
>>> >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> Subject: RE: About Fio backend with ObjectStore API
>>> >> >>>>>> >>
>>> >> >>>>>> >> Hi Casey,
>>> >> >>>>>> >>
>>> >> >>>>>> >> Thanks a lot.
>>> >> >>>>>> >>
>>> >> >>>>>> >> Regards,
>>> >> >>>>>> >> James
>>> >> >>>>>> >>
>>> >> >>>>>> >> -----Original Message-----
>>> >> >>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>> >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM
>>> >> >>>>>> >> To: James (Fei) Liu-SSI
>>> >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >> Subject: Re: About Fio backend with ObjectStore API
>>> >> >>>>>> >>
>>> >> >>>>>> >> Hi,
>>> >> >>>>>> >>
>>> >> >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph
>>> >> >>>>>> >> Performance
>>> >> >>>>>> >> Analysis: fio and RBD" at
>>> >> >>>>>> >>
>>> >> >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-perfor
>>> >> >>>>>> manc
>>> >> >>>>>> e
>>> >> >>>>>> -a
>>> >> >>>>>> naly
>>> >> >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine
>>> >> >>>>>> >> that linked directly into ceph's FileStore. I was able
>>> >> >>>>>> >> to find Daniel's branch
>>> >> >>>>>> on
>>> >> >>>>>> >> github at
>>> >> >>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2,
>>> >> >>>>>> >> and
>>> >> >>>>>> did some more work on it at the time.
>>> >> >>>>>> >>
>>> >> >>>>>> >> I just rebased that work onto the latest ceph master
>>> >> >>>>>> >> branch, and
>>> >> >>>>>> pushed to our github at
>>> >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore.
>>> >> >>>>>> You can find the source in src/test/fio_ceph_filestore.cc,
>>> >> >>>>>> and run fio with the provided example fio job file in
>>> >> >>>>>> src/test/filestore.fio.
>>> >> >>>>>> >>
>>> >> >>>>>> >> I didn't have a chance to confirm that it builds with
>>> >> >>>>>> >> automake, but
>>> >> >>>>>>
>>> >> >>>>>> >> the cmake version built for me. I'm happy to help if
>>> >> >>>>>> >> you run into problems, Casey
>>> >> >>>>>> >>
>>> >> >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI
>>> >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>> >> >>>>>> >>> Hi Haomai,
>>> >> >>>>>> >>> What are you trying to ask is to benchmark local
>>> >> >>>>>> objectstore(like kvstore/filestore/newstore) locally with
>>> >> >>>>>> FIO(ObjectStore engine)? You want to purely compare the
>>> >> >>>>>> performance locally for these objectstores, right?
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Regards,
>>> >> >>>>>> >>> James
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> -----Original Message-----
>>> >> >>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>> >> >>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of
>>> >> >>>>>> >>> Haomai
>>> >> >>>>>> Wang
>>> >> >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM
>>> >> >>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx
>>> >> >>>>>> >>> Subject: About Fio backend with ObjectStore API
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Hi all,
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Long long ago, is there someone said about fio backend
>>> >> >>>>>> >>> with Ceph
>>> >> >>>>>> ObjectStore API? So we could use the existing mature fio
>>> >> >>>>>> facility to benchmark ceph objectstore.
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> --
>>> >> >>>>>> >>> Best Regards,
>>> >> >>>>>> >>>
>>> >> >>>>>> >>> Wheat
>>> >> >>>>>> >>> --
>>> >> >>>>>> >>> To unsubscribe from this list: send the line
>>> >> >>>>>> >>> "unsubscribe
>>> >> >>>>>> ceph-devel"
>>> >> >>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx
>>> >> >>>>>> >>> More
>>> >> >>>>>> majordomo
>>> >> >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html
>>> >> >>>>>> >> {.n + +% lzwm b ? r y? ?zX ?} ?z &j:+v
>>> >> >>>>>> >> zZ+
>>> >> >>>>>> +zf h ~ i z w ? & )? f
>>> >> >>>>>> N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w
>>> >> >>>>>> j:+v w j m zZ+ ?j"
>>> >> >>>>>
>>> >> >>>>> --
>>> >> >>>>> Matt Benjamin
>>> >> >>>>> CohortFS, LLC.
>>> >> >>>>> 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
>>> >> >>>>>
>>> >> >>>>> http://cohortfs.com
>>> >> >>>>>
>>> >> >>>>> tel. 734-761-4689
>>> >> >>>>> fax. 734-769-8938
>>> >> >>>>> cel. 734-216-5309
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>> Best Regards,
>>> >> >>>>
>>> >> >>>> Wheat
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> --
>>> >> >>> Best Regards,
>>> >> >>>
>>> >> >>> Wheat
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Best Regards,
>>> >> >
>>> >> > Wheat
>>> >> > --
>>> >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>> >> > in the body of a message to majordomo@xxxxxxxxxxxxxxx More
>>> >> > majordomo info at http://vger.kernel.org/majordomo-info.html
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >>
>>> >> Wheat
>>> >> N r y b X ?v ^ )?{.n + z ]z {ay ?? ,j f h z w j
>>> >> :+v w j m zZ+ ?j"
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w
>>> j:+v w j m zZ+ ݢj"
>
>
>
> --
> Best Regards,
>
> Wheat
--
Best Regards,
Wheat
Attachment:
fio_ceph_objectstore.cc.diff
Description: fio_ceph_objectstore.cc.diff
