no special [global] #logging #write_iops_log=write_iops_log #write_bw_log=write_bw_log #write_lat_log=write_lat_log ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so invalidate=0 # mandatory rw=write #bs=4k [filestore] iodepth=1 # create a journaled filestore objectstore=filestore directory=./osd/ filestore_journal=./osd/journal On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote: > Hi Haomai, > > Sorry for the late response, I was out of the office. I'm afraid I haven't run into that segfault. The io_ops should be set at the very beginning when it calls get_ioengine(). All I can suggest is that you verify that your job file is pointing to the correct fio_ceph_objectstore.so. If you've made any other interesting changes to the job file, could you share it here? > > Casey > > ----- Original Message ----- > From: "Haomai Wang" <haomaiwang@xxxxxxxxx> > To: "Casey Bodley" <cbodley@xxxxxxxxx> > Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx > Sent: Tuesday, July 21, 2015 7:50:32 AM > Subject: Re: About Fio backend with ObjectStore API > > Hi Casey, > > I check your commits and know what you fixed. I cherry-picked your new > commits but I still met the same problem. > > """ > It's strange that it alwasys hit segment fault when entering > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but > when I up the stack, the "td->io_ops" is not null. Maybe it's related > to dlopen? > """ > > Do you have any hint about this? > > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote: >> Hi Haomai, >> >> I was able to run this after a couple changes to the filestore.fio job >> file. Two of the config options were using the wrong names. I pushed a >> fix for the job file, as well as a patch that renames everything from >> filestore to objectstore (thanks James), to >> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore. >> >> I found that the read support doesn't appear to work anymore, so give >> "rw=write" a try. And because it does a mkfs(), make sure you're >> pointing it to an empty xfs directory with the "directory=" option. >> >> Casey >> >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: >>> Anyone who have successfully ran the fio with this external io engine >>> ceph_objectstore? >>> >>> It's strange that it alwasys hit segment fault when entering >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but >>> when I up the stack, the "td->io_ops" is not null. Maybe it's related >>> to dlopen? >>> >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: >>>> I have rebased the branch with master, and push it to ceph upstream >>>> repo. https://github.com/ceph/ceph/compare/fio-objectstore?expand=1 >>>> >>>> Plz let me know if who is working on this. Otherwise, I would like to >>>> improve this to be merge ready. >>>> >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin <matt@xxxxxxxxxxxx> wrote: >>>>> That makes sense. >>>>> >>>>> Matt >>>>> >>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>> >>>>>> Hi Casey, >>>>>> Got it. I was directed to the old code base. By the way, Since the >>>>>> testing case was used to exercise all of object stores. Strongly >>>>>> recommend to change the name from fio_ceph_filestore.cc to >>>>>> fio_ceph_objectstore.cc . And the code in fio_ceph_filestore.cc should >>>>>> be refactored to reflect that the whole objectstore will be supported >>>>>> by fio_ceph_objectstore.cc. what you think? >>>>>> >>>>>> Let me know if you need any help from my side. >>>>>> >>>>>> >>>>>> Regards, >>>>>> James >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>>> Sent: Thursday, July 09, 2015 12:32 PM >>>>>> To: James (Fei) Liu-SSI >>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>>> Subject: Re: About Fio backend with ObjectStore API >>>>>> >>>>>> Hi James, >>>>>> >>>>>> Are you looking at the code from >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore? It >>>>>> uses ObjectStore::create() instead of new FileStore(). This allows us >>>>>> to exercise all of the object stores with the same code. >>>>>> >>>>>> Casey >>>>>> >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>>> > Hi Casey, >>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically, it >>>>>> creates a filestore as backend engine for IO exercises. If we got to >>>>>> send IO commands to KeyValue Store or Newstore, we got to change the >>>>>> code accordingly, right? I did not see any other files like >>>>>> fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc. In my humble >>>>>> opinion, we might need to create other two fio engines for >>>>>> keyvaluestore and newstore if we want to exercise these two, right? >>>>>> > >>>>>> > Regards, >>>>>> > James >>>>>> > >>>>>> > static int fio_ceph_filestore_init(struct thread_data *td) >>>>>> > 209 { >>>>>> > 210 vector<const char*> args; >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct >>>>>> ceph_filestore_data *) td->io_ops->data; >>>>>> > 212 ObjectStore::Transaction ft; >>>>>> > 213 >>>>>> > >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD, >>>>>> CODE_ENVIRONMENT_UTILITY, 0); >>>>>> > 215 //g_conf->journal_dio = false; >>>>>> > 216 common_init_finish(g_ceph_context); >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20"); >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20"); >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL); >>>>>> > 220 >>>>>> > >>>>>> > 221 ceph_filestore_data->osd_path = >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX"); >>>>>> > 222 ceph_filestore_data->journal_path = >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"); >>>>>> > 223 >>>>>> > >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) { >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) << >>>>>> std::endl; >>>>>> > 226 return 1; >>>>>> > 227 } >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue >>>>>> > 229 >>>>>> > >>>>>> > 230 ObjectStore *fs = new >>>>>> FileStore(ceph_filestore_data->osd_path, >>>>>> ceph_filestore_data->journal_path); >>>>>> > 231 ceph_filestore_data->fs = fs; >>>>>> > 232 >>>>>> > >>>>>> > 233 if (fs->mkfs() < 0) { >>>>>> > 234 cout << "mkfs failed" << std::endl; >>>>>> > 235 goto failed; >>>>>> > 236 } >>>>>> > 237 >>>>>> > 238 if (fs->mount() < 0) { >>>>>> > 239 cout << "mount failed" << std::endl; >>>>>> > 240 goto failed; >>>>>> > 241 } >>>>>> > 242 >>>>>> > >>>>>> > 243 ft.create_collection(coll_t()); >>>>>> > 244 fs->apply_transaction(ft); >>>>>> > 245 >>>>>> > >>>>>> > 246 >>>>>> > >>>>>> > 247 return 0; >>>>>> > 248 >>>>>> > >>>>>> > 249 failed: >>>>>> > 250 return 1; >>>>>> > 251 >>>>>> > >>>>>> > 252 } >>>>>> > -----Original Message----- >>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM >>>>>> > To: James (Fei) Liu-SSI >>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>>> > Subject: Re: About Fio backend with ObjectStore API >>>>>> > >>>>>> > Hi James, >>>>>> > >>>>>> > In the job file src/test/filestore.fio, you can modify the line >>>>>> > "objectstore=filestore" to use any objectstore type supported by >>>>>> the >>>>>> > ObjectStore::create() factory. >>>>>> > >>>>>> > Casey >>>>>> > >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>>> >> Hi Casey, >>>>>> >> Quick questions, The code in the trunk only cover the test for >>>>>> filestore. I was wondering do you have any plan to cover the test for >>>>>> kvstore and newstore? >>>>>> >> >>>>>> >> Thanks, >>>>>> >> James >>>>>> >> >>>>>> >> -----Original Message----- >>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx >>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of James (Fei) >>>>>> >>>>>> >> Liu-SSI >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM >>>>>> >> To: Casey Bodley >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>>> >> Subject: RE: About Fio backend with ObjectStore API >>>>>> >> >>>>>> >> Hi Casey, >>>>>> >> >>>>>> >> Thanks a lot. >>>>>> >> >>>>>> >> Regards, >>>>>> >> James >>>>>> >> >>>>>> >> -----Original Message----- >>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM >>>>>> >> To: James (Fei) Liu-SSI >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>>> >> Subject: Re: About Fio backend with ObjectStore API >>>>>> >> >>>>>> >> Hi, >>>>>> >> >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph Performance >>>>>> >> Analysis: fio and RBD" at >>>>>> >> >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analy >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that linked >>>>>> >> directly into ceph's FileStore. I was able to find Daniel's branch >>>>>> on >>>>>> >> github at https://github.com/gollub/ceph/tree/fio_filestore_v2, and >>>>>> did some more work on it at the time. >>>>>> >> >>>>>> >> I just rebased that work onto the latest ceph master branch, and >>>>>> pushed to our github at >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore. You >>>>>> can find the source in src/test/fio_ceph_filestore.cc, and run fio >>>>>> with the provided example fio job file in src/test/filestore.fio. >>>>>> >> >>>>>> >> I didn't have a chance to confirm that it builds with automake, but >>>>>> >>>>>> >> the cmake version built for me. I'm happy to help if you run into >>>>>> >> problems, Casey >>>>>> >> >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>>> >>> Hi Haomai, >>>>>> >>> What are you trying to ask is to benchmark local >>>>>> objectstore(like kvstore/filestore/newstore) locally with >>>>>> FIO(ObjectStore engine)? You want to purely compare the performance >>>>>> locally for these objectstores, right? >>>>>> >>> >>>>>> >>> Regards, >>>>>> >>> James >>>>>> >>> >>>>>> >>> -----Original Message----- >>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx >>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Haomai >>>>>> Wang >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM >>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx >>>>>> >>> Subject: About Fio backend with ObjectStore API >>>>>> >>> >>>>>> >>> Hi all, >>>>>> >>> >>>>>> >>> Long long ago, is there someone said about fio backend with Ceph >>>>>> ObjectStore API? So we could use the existing mature fio facility to >>>>>> benchmark ceph objectstore. >>>>>> >>> >>>>>> >>> -- >>>>>> >>> Best Regards, >>>>>> >>> >>>>>> >>> Wheat >>>>>> >>> -- >>>>>> >>> To unsubscribe from this list: send the line "unsubscribe >>>>>> ceph-devel" >>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>>>>> majordomo >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html >>>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+ >>>>>> +zf h ~ i z w ? & )ߢ f >>>>>> N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������j:+v���w�j�m��������zZ+��ݢj"�� >>>>> >>>>> -- >>>>> Matt Benjamin >>>>> CohortFS, LLC. >>>>> 315 West Huron Street, Suite 140A >>>>> Ann Arbor, Michigan 48103 >>>>> >>>>> http://cohortfs.com >>>>> >>>>> tel. 734-761-4689 >>>>> fax. 734-769-8938 >>>>> cel. 734-216-5309 >>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> >>>> Wheat >>> >>> >>> >>> -- >>> Best Regards, >>> >>> Wheat > > > > -- > Best Regards, > > Wheat > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
