Hi Haomai, Sorry for the late response, I was out of the office. I'm afraid I haven't run into that segfault. The io_ops should be set at the very beginning when it calls get_ioengine(). All I can suggest is that you verify that your job file is pointing to the correct fio_ceph_objectstore.so. If you've made any other interesting changes to the job file, could you share it here? Casey ----- Original Message ----- From: "Haomai Wang" <haomaiwang@xxxxxxxxx> To: "Casey Bodley" <cbodley@xxxxxxxxx> Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx Sent: Tuesday, July 21, 2015 7:50:32 AM Subject: Re: About Fio backend with ObjectStore API Hi Casey, I check your commits and know what you fixed. I cherry-picked your new commits but I still met the same problem. """ It's strange that it alwasys hit segment fault when entering "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but when I up the stack, the "td->io_ops" is not null. Maybe it's related to dlopen? """ Do you have any hint about this? On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote: > Hi Haomai, > > I was able to run this after a couple changes to the filestore.fio job > file. Two of the config options were using the wrong names. I pushed a > fix for the job file, as well as a patch that renames everything from > filestore to objectstore (thanks James), to > https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore. > > I found that the read support doesn't appear to work anymore, so give > "rw=write" a try. And because it does a mkfs(), make sure you're > pointing it to an empty xfs directory with the "directory=" option. > > Casey > > On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: >> Anyone who have successfully ran the fio with this external io engine >> ceph_objectstore? >> >> It's strange that it alwasys hit segment fault when entering >> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but >> when I up the stack, the "td->io_ops" is not null. Maybe it's related >> to dlopen? >> >> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: >>> I have rebased the branch with master, and push it to ceph upstream >>> repo. https://github.com/ceph/ceph/compare/fio-objectstore?expand=1 >>> >>> Plz let me know if who is working on this. Otherwise, I would like to >>> improve this to be merge ready. >>> >>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin <matt@xxxxxxxxxxxx> wrote: >>>> That makes sense. >>>> >>>> Matt >>>> >>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote: >>>> >>>>> Hi Casey, >>>>> Got it. I was directed to the old code base. By the way, Since the >>>>> testing case was used to exercise all of object stores. Strongly >>>>> recommend to change the name from fio_ceph_filestore.cc to >>>>> fio_ceph_objectstore.cc . And the code in fio_ceph_filestore.cc should >>>>> be refactored to reflect that the whole objectstore will be supported >>>>> by fio_ceph_objectstore.cc. what you think? >>>>> >>>>> Let me know if you need any help from my side. >>>>> >>>>> >>>>> Regards, >>>>> James >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>> Sent: Thursday, July 09, 2015 12:32 PM >>>>> To: James (Fei) Liu-SSI >>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>> Subject: Re: About Fio backend with ObjectStore API >>>>> >>>>> Hi James, >>>>> >>>>> Are you looking at the code from >>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore? It >>>>> uses ObjectStore::create() instead of new FileStore(). This allows us >>>>> to exercise all of the object stores with the same code. >>>>> >>>>> Casey >>>>> >>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI >>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>> > Hi Casey, >>>>> > Here is the code in the fio_ceph_filestore.cc. Basically, it >>>>> creates a filestore as backend engine for IO exercises. If we got to >>>>> send IO commands to KeyValue Store or Newstore, we got to change the >>>>> code accordingly, right? I did not see any other files like >>>>> fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc. In my humble >>>>> opinion, we might need to create other two fio engines for >>>>> keyvaluestore and newstore if we want to exercise these two, right? >>>>> > >>>>> > Regards, >>>>> > James >>>>> > >>>>> > static int fio_ceph_filestore_init(struct thread_data *td) >>>>> > 209 { >>>>> > 210 vector<const char*> args; >>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct >>>>> ceph_filestore_data *) td->io_ops->data; >>>>> > 212 ObjectStore::Transaction ft; >>>>> > 213 >>>>> > >>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD, >>>>> CODE_ENVIRONMENT_UTILITY, 0); >>>>> > 215 //g_conf->journal_dio = false; >>>>> > 216 common_init_finish(g_ceph_context); >>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20"); >>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20"); >>>>> > 219 g_ceph_context->_conf->apply_changes(NULL); >>>>> > 220 >>>>> > >>>>> > 221 ceph_filestore_data->osd_path = >>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX"); >>>>> > 222 ceph_filestore_data->journal_path = >>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"); >>>>> > 223 >>>>> > >>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) { >>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) << >>>>> std::endl; >>>>> > 226 return 1; >>>>> > 227 } >>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue >>>>> > 229 >>>>> > >>>>> > 230 ObjectStore *fs = new >>>>> FileStore(ceph_filestore_data->osd_path, >>>>> ceph_filestore_data->journal_path); >>>>> > 231 ceph_filestore_data->fs = fs; >>>>> > 232 >>>>> > >>>>> > 233 if (fs->mkfs() < 0) { >>>>> > 234 cout << "mkfs failed" << std::endl; >>>>> > 235 goto failed; >>>>> > 236 } >>>>> > 237 >>>>> > 238 if (fs->mount() < 0) { >>>>> > 239 cout << "mount failed" << std::endl; >>>>> > 240 goto failed; >>>>> > 241 } >>>>> > 242 >>>>> > >>>>> > 243 ft.create_collection(coll_t()); >>>>> > 244 fs->apply_transaction(ft); >>>>> > 245 >>>>> > >>>>> > 246 >>>>> > >>>>> > 247 return 0; >>>>> > 248 >>>>> > >>>>> > 249 failed: >>>>> > 250 return 1; >>>>> > 251 >>>>> > >>>>> > 252 } >>>>> > -----Original Message----- >>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>> > Sent: Thursday, July 09, 2015 9:19 AM >>>>> > To: James (Fei) Liu-SSI >>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>> > Subject: Re: About Fio backend with ObjectStore API >>>>> > >>>>> > Hi James, >>>>> > >>>>> > In the job file src/test/filestore.fio, you can modify the line >>>>> > "objectstore=filestore" to use any objectstore type supported by >>>>> the >>>>> > ObjectStore::create() factory. >>>>> > >>>>> > Casey >>>>> > >>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI >>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>> >> Hi Casey, >>>>> >> Quick questions, The code in the trunk only cover the test for >>>>> filestore. I was wondering do you have any plan to cover the test for >>>>> kvstore and newstore? >>>>> >> >>>>> >> Thanks, >>>>> >> James >>>>> >> >>>>> >> -----Original Message----- >>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx >>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of James (Fei) >>>>> >>>>> >> Liu-SSI >>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM >>>>> >> To: Casey Bodley >>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>> >> Subject: RE: About Fio backend with ObjectStore API >>>>> >> >>>>> >> Hi Casey, >>>>> >> >>>>> >> Thanks a lot. >>>>> >> >>>>> >> Regards, >>>>> >> James >>>>> >> >>>>> >> -----Original Message----- >>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] >>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM >>>>> >> To: James (Fei) Liu-SSI >>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx >>>>> >> Subject: Re: About Fio backend with ObjectStore API >>>>> >> >>>>> >> Hi, >>>>> >> >>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph Performance >>>>> >> Analysis: fio and RBD" at >>>>> >> >>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analy >>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that linked >>>>> >> directly into ceph's FileStore. I was able to find Daniel's branch >>>>> on >>>>> >> github at https://github.com/gollub/ceph/tree/fio_filestore_v2, and >>>>> did some more work on it at the time. >>>>> >> >>>>> >> I just rebased that work onto the latest ceph master branch, and >>>>> pushed to our github at >>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore. You >>>>> can find the source in src/test/fio_ceph_filestore.cc, and run fio >>>>> with the provided example fio job file in src/test/filestore.fio. >>>>> >> >>>>> >> I didn't have a chance to confirm that it builds with automake, but >>>>> >>>>> >> the cmake version built for me. I'm happy to help if you run into >>>>> >> problems, Casey >>>>> >> >>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI >>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: >>>>> >>> Hi Haomai, >>>>> >>> What are you trying to ask is to benchmark local >>>>> objectstore(like kvstore/filestore/newstore) locally with >>>>> FIO(ObjectStore engine)? You want to purely compare the performance >>>>> locally for these objectstores, right? >>>>> >>> >>>>> >>> Regards, >>>>> >>> James >>>>> >>> >>>>> >>> -----Original Message----- >>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx >>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Haomai >>>>> Wang >>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM >>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx >>>>> >>> Subject: About Fio backend with ObjectStore API >>>>> >>> >>>>> >>> Hi all, >>>>> >>> >>>>> >>> Long long ago, is there someone said about fio backend with Ceph >>>>> ObjectStore API? So we could use the existing mature fio facility to >>>>> benchmark ceph objectstore. >>>>> >>> >>>>> >>> -- >>>>> >>> Best Regards, >>>>> >>> >>>>> >>> Wheat >>>>> >>> -- >>>>> >>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" >>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More >>>>> majordomo >>>>> >>> info at http://vger.kernel.org/majordomo-info.html >>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+ >>>>> +zf h ~ i z w ? & )ߢ f >>>>> N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������j:+v���w�j�m��������zZ+��ݢj"�� >>>> >>>> -- >>>> Matt Benjamin >>>> CohortFS, LLC. >>>> 315 West Huron Street, Suite 140A >>>> Ann Arbor, Michigan 48103 >>>> >>>> http://cohortfs.com >>>> >>>> tel. 734-761-4689 >>>> fax. 734-769-8938 >>>> cel. 734-216-5309 >>> >>> >>> >>> -- >>> Best Regards, >>> >>> Wheat >> >> >> >> -- >> Best Regards, >> >> Wheat -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
