Hi Haomai and Casey, The fio-objectstore was compiled after new repo cloned. However, I still got below error with undefined symbol from ObjectStore.h. jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph/src$ ./fio/fio ./test/objectstore.fio filestore: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=cephobjectstore, iodepth=1 fio-2.2.9-56-g736a Starting 1 process ./fio/fio: symbol lookup error: ./.libs/libfio_ceph_objectstore.so: undefined symbol: _ZN11ObjectStore6createEP11CephContextRKSsS3_S3_j Regards, James Run status group 0 (all jobs): Disk stats (read/write): sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% Here if fio file: ###################################################################### # Example test for the external fio ioengine for ObjectStore. # # Runs a 4k random write test against a ObjectStore configuration. # ###################################################################### [global] #logging #write_iops_log=write_iops_log #write_bw_log=write_bw_log #write_lat_log=write_lat_log ioengine=./.libs/libfio_ceph_objectstore.so invalidate=0 # mandatory #rw=randwrite #bs=4k [filestore] iodepth=1 # create a journaled filestore objectstore=filestore directory=/mnt/fio_ceph_filestore filestore_journal=/var/lib/ceph/osd/journal-ram/fio_ceph_filestore1 -----Original Message----- From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] Sent: Tuesday, September 08, 2015 7:28 PM To: James (Fei) Liu-SSI Cc: Casey Bodley Subject: Re: About Fio backend with ObjectStore API On Wed, Sep 9, 2015 at 6:08 AM, James (Fei) Liu-SSI <james.liu@xxxxxxxxxxxxxxx> wrote: > Hi Haomai, > DO you get this compiling error in fio-objectstore branch? This is a build bug and happen when switching from very old commit. The simple way is that git clone new ceph repo and build again. The problem is that some legacy compiled files cause incorrect dependence > > Regards, > James > > -----Original Message----- > From: James (Fei) Liu-SSI > Sent: Friday, September 04, 2015 1:30 PM > To: 'Casey Bodley' > Cc: ceph-devel@xxxxxxxxxxxxxxx > Subject: RE: About Fio backend with ObjectStore API > > Hi Casey, > Thanks. I even got a compiling error with fio-objectstore branch. > > Here is error message: > > make[3]: *** No rule to make target `rbd_fuse/rbd-fuse.c', needed by `rbd_fuse/rbd-fuse.o'. Stop. > make[3]: *** Waiting for unfinished jobs.... > CXX ceph_fuse.o > make[3]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src' > make[2]: *** [all-recursive] Error 1 > make[2]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/home/jamesliu/WorkSpace/ceph_fio/src' > make: *** [all-recursive] Error 1 > jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_fio$ git branch > * fio-objectstore > master > > > Regards, > James > > -----Original Message----- > From: Casey Bodley [mailto:cbodley@xxxxxxxxxx] > Sent: Thursday, September 03, 2015 10:44 AM > To: James (Fei) Liu-SSI > Cc: ceph-devel@xxxxxxxxxxxxxxx > Subject: Re: About Fio backend with ObjectStore API > > Hi James, > > I'm sorry for not following up on that segfault, but I wasn't ever able to reproduce it. I used it recently for memstore testing without any problems. I wonder if there's a problem with the autotools build? I've only tested it with cmake. When I find some time, I'll rebase it on master and do another round of testing. > > Casey > > ----- Original Message ----- >> From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> >> To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley" >> <cbodley@xxxxxxxxxx> >> Cc: "Casey Bodley" <cbodley@xxxxxxxxx>, "Matt W. Benjamin" >> <matt@xxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx >> Sent: Wednesday, September 2, 2015 8:06:14 PM >> Subject: RE: About Fio backend with ObjectStore API >> >> Hi Haomai and Case, >> Do you have any fixes for that segfault? >> >> Thanks, >> James >> >> -----Original Message----- >> From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] >> Sent: Wednesday, July 22, 2015 6:07 PM >> To: Casey Bodley >> Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI; >> ceph-devel@xxxxxxxxxxxxxxx >> Subject: Re: About Fio backend with ObjectStore API >> >> no special >> >> [global] >> #logging >> #write_iops_log=write_iops_log >> #write_bw_log=write_bw_log >> #write_lat_log=write_lat_log >> ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so >> invalidate=0 # mandatory >> rw=write >> #bs=4k >> >> [filestore] >> iodepth=1 >> # create a journaled filestore >> objectstore=filestore >> directory=./osd/ >> filestore_journal=./osd/journal >> >> On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote: >> > Hi Haomai, >> > >> > Sorry for the late response, I was out of the office. I'm afraid I >> > haven't run into that segfault. The io_ops should be set at the >> > very beginning when it calls get_ioengine(). All I can suggest is >> > that you verify that your job file is pointing to the correct >> > fio_ceph_objectstore.so. If you've made any other interesting >> > changes to the job file, could you share it here? >> > >> > Casey >> > >> > ----- Original Message ----- >> > From: "Haomai Wang" <haomaiwang@xxxxxxxxx> >> > To: "Casey Bodley" <cbodley@xxxxxxxxx> >> > Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI" >> > <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx >> > Sent: Tuesday, July 21, 2015 7:50:32 AM >> > Subject: Re: About Fio backend with ObjectStore API >> > >> > Hi Casey, >> > >> > I check your commits and know what you fixed. I cherry-picked your >> > new commits but I still met the same problem. >> > >> > """ >> > It's strange that it alwasys hit segment fault when entering >> > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL >> > but when I up the stack, the "td->io_ops" is not null. Maybe it's >> > related to dlopen? >> > """ >> > >> > Do you have any hint about this? >> > >> > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote: >> >> Hi Haomai, >> >> >> >> I was able to run this after a couple changes to the filestore.fio >> >> job file. Two of the config options were using the wrong names. I >> >> pushed a fix for the job file, as well as a patch that renames >> >> everything from filestore to objectstore (thanks James), to >> >> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore. >> >> >> >> I found that the read support doesn't appear to work anymore, so >> >> give "rw=write" a try. And because it does a mkfs(), make sure >> >> you're pointing it to an empty xfs directory with the "directory=" option. >> >> >> >> Casey >> >> >> >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: >> >>> Anyone who have successfully ran the fio with this external io >> >>> engine ceph_objectstore? >> >>> >> >>> It's strange that it alwasys hit segment fault when entering >> >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL >> >>> but when I up the stack, the "td->io_ops" is not null. Maybe it's >> >>> related to dlopen? >> >>> >> >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang >> >>> <haomaiwang@xxxxxxxxx> >> >>> wrote: >> >>>> I have rebased the branch with master, and push it to ceph >> >>>> upstream repo. >> >>>> https://github.com/ceph/ceph/compare/fio-objectstore?expand=1 >> >>>> >> >>>> Plz let me know if who is working on this. Otherwise, I would >> >>>> like to improve this to be merge ready. >> >>>> >> >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin >> >>>> <matt@xxxxxxxxxxxx> >> >>>> wrote: >> >>>>> That makes sense. >> >>>>> >> >>>>> Matt >> >>>>> >> >>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote: >> >>>>> >> >>>>>> Hi Casey, >> >>>>>> Got it. I was directed to the old code base. By the way, >> >>>>>> Since the testing case was used to exercise all of object stores. >> >>>>>> Strongly recommend to change the name from >> >>>>>> fio_ceph_filestore.cc to fio_ceph_objectstore.cc . And the >> >>>>>> code in fio_ceph_filestore.cc should be refactored to reflect >> >>>>>> that the whole objectstore will be supported by >> >>>>>> fio_ceph_objectstore.cc. what you think? >> >>>>>> >> >>>>>> Let me know if you need any help from my side. >> >>>>>> >> >>>>>> >> >>>>>> Regards, >> >>>>>> James >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -----Original Message----- >> >>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >> >>>>>> Sent: Thursday, July 09, 2015 12:32 PM >> >>>>>> To: James (Fei) Liu-SSI >> >>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >> >>>>>> Subject: Re: About Fio backend with ObjectStore API >> >>>>>> >> >>>>>> Hi James, >> >>>>>> >> >>>>>> Are you looking at the code from >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore? >> >>>>>> It uses ObjectStore::create() instead of new FileStore(). This >> >>>>>> allows us to exercise all of the object stores with the same code. >> >>>>>> >> >>>>>> Casey >> >>>>>> >> >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >> >>>>>> > Hi Casey, >> >>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically, >> >>>>>> > it >> >>>>>> creates a filestore as backend engine for IO exercises. If we >> >>>>>> got to send IO commands to KeyValue Store or Newstore, we got >> >>>>>> to change the code accordingly, right? I did not see any >> >>>>>> other files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc. >> >>>>>> In my humble opinion, we might need to create other two fio >> >>>>>> engines for keyvaluestore and newstore if we want to exercise >> >>>>>> these two, right? >> >>>>>> > >> >>>>>> > Regards, >> >>>>>> > James >> >>>>>> > >> >>>>>> > static int fio_ceph_filestore_init(struct thread_data *td) >> >>>>>> > 209 { >> >>>>>> > 210 vector<const char*> args; >> >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct >> >>>>>> ceph_filestore_data *) td->io_ops->data; >> >>>>>> > 212 ObjectStore::Transaction ft; >> >>>>>> > 213 >> >>>>>> > >> >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD, >> >>>>>> CODE_ENVIRONMENT_UTILITY, 0); >> >>>>>> > 215 //g_conf->journal_dio = false; >> >>>>>> > 216 common_init_finish(g_ceph_context); >> >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20"); >> >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20"); >> >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL); >> >>>>>> > 220 >> >>>>>> > >> >>>>>> > 221 ceph_filestore_data->osd_path = >> >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX"); >> >>>>>> > 222 ceph_filestore_data->journal_path = >> >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX" >> >>>>>> ); >> >>>>>> > 223 >> >>>>>> > >> >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) { >> >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) << >> >>>>>> std::endl; >> >>>>>> > 226 return 1; >> >>>>>> > 227 } >> >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue >> >>>>>> > 229 >> >>>>>> > >> >>>>>> > 230 ObjectStore *fs = new >> >>>>>> FileStore(ceph_filestore_data->osd_path, >> >>>>>> ceph_filestore_data->journal_path); >> >>>>>> > 231 ceph_filestore_data->fs = fs; >> >>>>>> > 232 >> >>>>>> > >> >>>>>> > 233 if (fs->mkfs() < 0) { >> >>>>>> > 234 cout << "mkfs failed" << std::endl; >> >>>>>> > 235 goto failed; >> >>>>>> > 236 } >> >>>>>> > 237 >> >>>>>> > 238 if (fs->mount() < 0) { >> >>>>>> > 239 cout << "mount failed" << std::endl; >> >>>>>> > 240 goto failed; >> >>>>>> > 241 } >> >>>>>> > 242 >> >>>>>> > >> >>>>>> > 243 ft.create_collection(coll_t()); >> >>>>>> > 244 fs->apply_transaction(ft); >> >>>>>> > 245 >> >>>>>> > >> >>>>>> > 246 >> >>>>>> > >> >>>>>> > 247 return 0; >> >>>>>> > 248 >> >>>>>> > >> >>>>>> > 249 failed: >> >>>>>> > 250 return 1; >> >>>>>> > 251 >> >>>>>> > >> >>>>>> > 252 } >> >>>>>> > -----Original Message----- >> >>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >> >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM >> >>>>>> > To: James (Fei) Liu-SSI >> >>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >> >>>>>> > Subject: Re: About Fio backend with ObjectStore API >> >>>>>> > >> >>>>>> > Hi James, >> >>>>>> > >> >>>>>> > In the job file src/test/filestore.fio, you can modify the >> >>>>>> > line "objectstore=filestore" to use any objectstore type >> >>>>>> > supported by >> >>>>>> the >> >>>>>> > ObjectStore::create() factory. >> >>>>>> > >> >>>>>> > Casey >> >>>>>> > >> >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >> >>>>>> >> Hi Casey, >> >>>>>> >> Quick questions, The code in the trunk only cover the >> >>>>>> >> test for >> >>>>>> filestore. I was wondering do you have any plan to cover the >> >>>>>> test for kvstore and newstore? >> >>>>>> >> >> >>>>>> >> Thanks, >> >>>>>> >> James >> >>>>>> >> >> >>>>>> >> -----Original Message----- >> >>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx >> >>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of >> >>>>>> >> James >> >>>>>> >> (Fei) >> >>>>>> >> >>>>>> >> Liu-SSI >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM >> >>>>>> >> To: Casey Bodley >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >> >>>>>> >> Subject: RE: About Fio backend with ObjectStore API >> >>>>>> >> >> >>>>>> >> Hi Casey, >> >>>>>> >> >> >>>>>> >> Thanks a lot. >> >>>>>> >> >> >>>>>> >> Regards, >> >>>>>> >> James >> >>>>>> >> >> >>>>>> >> -----Original Message----- >> >>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >> >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM >> >>>>>> >> To: James (Fei) Liu-SSI >> >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >> >>>>>> >> Subject: Re: About Fio backend with ObjectStore API >> >>>>>> >> >> >>>>>> >> Hi, >> >>>>>> >> >> >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph >> >>>>>> >> Performance >> >>>>>> >> Analysis: fio and RBD" at >> >>>>>> >> >> >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performanc >> >>>>>> e >> >>>>>> -a >> >>>>>> naly >> >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that >> >>>>>> >> linked directly into ceph's FileStore. I was able to find >> >>>>>> >> Daniel's branch >> >>>>>> on >> >>>>>> >> github at >> >>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2, and >> >>>>>> did some more work on it at the time. >> >>>>>> >> >> >>>>>> >> I just rebased that work onto the latest ceph master >> >>>>>> >> branch, and >> >>>>>> pushed to our github at >> >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore. >> >>>>>> You can find the source in src/test/fio_ceph_filestore.cc, and >> >>>>>> run fio with the provided example fio job file in >> >>>>>> src/test/filestore.fio. >> >>>>>> >> >> >>>>>> >> I didn't have a chance to confirm that it builds with >> >>>>>> >> automake, but >> >>>>>> >> >>>>>> >> the cmake version built for me. I'm happy to help if you >> >>>>>> >> run into problems, Casey >> >>>>>> >> >> >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI >> >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >> >>>>>> >>> Hi Haomai, >> >>>>>> >>> What are you trying to ask is to benchmark local >> >>>>>> objectstore(like kvstore/filestore/newstore) locally with >> >>>>>> FIO(ObjectStore engine)? You want to purely compare the >> >>>>>> performance locally for these objectstores, right? >> >>>>>> >>> >> >>>>>> >>> Regards, >> >>>>>> >>> James >> >>>>>> >>> >> >>>>>> >>> -----Original Message----- >> >>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx >> >>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of >> >>>>>> >>> Haomai >> >>>>>> Wang >> >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM >> >>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx >> >>>>>> >>> Subject: About Fio backend with ObjectStore API >> >>>>>> >>> >> >>>>>> >>> Hi all, >> >>>>>> >>> >> >>>>>> >>> Long long ago, is there someone said about fio backend >> >>>>>> >>> with Ceph >> >>>>>> ObjectStore API? So we could use the existing mature fio >> >>>>>> facility to benchmark ceph objectstore. >> >>>>>> >>> >> >>>>>> >>> -- >> >>>>>> >>> Best Regards, >> >>>>>> >>> >> >>>>>> >>> Wheat >> >>>>>> >>> -- >> >>>>>> >>> To unsubscribe from this list: send the line "unsubscribe >> >>>>>> ceph-devel" >> >>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >> >>>>>> majordomo >> >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html >> >>>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+ >> >>>>>> +zf h ~ i z w ? & )ߢ f >> >>>>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w >> >>>>>> j:+v w j m zZ+ ݢj" >> >>>>> >> >>>>> -- >> >>>>> Matt Benjamin >> >>>>> CohortFS, LLC. >> >>>>> 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 >> >>>>> >> >>>>> http://cohortfs.com >> >>>>> >> >>>>> tel. 734-761-4689 >> >>>>> fax. 734-769-8938 >> >>>>> cel. 734-216-5309 >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Best Regards, >> >>>> >> >>>> Wheat >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, >> >>> >> >>> Wheat >> > >> > >> > >> > -- >> > Best Regards, >> > >> > Wheat >> > -- >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> > in the body of a message to majordomo@xxxxxxxxxxxxxxx More >> > majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> -- >> Best Regards, >> >> Wheat >> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j >> :+v w j m zZ+ ݢj" -- Best Regards, Wheat ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
