Hi James, I'm sorry for not following up on that segfault, but I wasn't ever able to reproduce it. I used it recently for memstore testing without any problems. I wonder if there's a problem with the autotools build? I've only tested it with cmake. When I find some time, I'll rebase it on master and do another round of testing. Casey ----- Original Message ----- > From: "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> > To: "Haomai Wang" <haomaiwang@xxxxxxxxx>, "Casey Bodley" <cbodley@xxxxxxxxxx> > Cc: "Casey Bodley" <cbodley@xxxxxxxxx>, "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx > Sent: Wednesday, September 2, 2015 8:06:14 PM > Subject: RE: About Fio backend with ObjectStore API > > Hi Haomai and Case, > Do you have any fixes for that segfault? > > Thanks, > James > > -----Original Message----- > From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] > Sent: Wednesday, July 22, 2015 6:07 PM > To: Casey Bodley > Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI; > ceph-devel@xxxxxxxxxxxxxxx > Subject: Re: About Fio backend with ObjectStore API > > no special > > [global] > #logging > #write_iops_log=write_iops_log > #write_bw_log=write_bw_log > #write_lat_log=write_lat_log > ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so > invalidate=0 # mandatory > rw=write > #bs=4k > > [filestore] > iodepth=1 > # create a journaled filestore > objectstore=filestore > directory=./osd/ > filestore_journal=./osd/journal > > On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote: > > Hi Haomai, > > > > Sorry for the late response, I was out of the office. I'm afraid I haven't > > run into that segfault. The io_ops should be set at the very beginning > > when it calls get_ioengine(). All I can suggest is that you verify that > > your job file is pointing to the correct fio_ceph_objectstore.so. If > > you've made any other interesting changes to the job file, could you share > > it here? > > > > Casey > > > > ----- Original Message ----- > > From: "Haomai Wang" <haomaiwang@xxxxxxxxx> > > To: "Casey Bodley" <cbodley@xxxxxxxxx> > > Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI" > > <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx > > Sent: Tuesday, July 21, 2015 7:50:32 AM > > Subject: Re: About Fio backend with ObjectStore API > > > > Hi Casey, > > > > I check your commits and know what you fixed. I cherry-picked your new > > commits but I still met the same problem. > > > > """ > > It's strange that it alwasys hit segment fault when entering > > "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but > > when I up the stack, the "td->io_ops" is not null. Maybe it's related > > to dlopen? > > """ > > > > Do you have any hint about this? > > > > On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote: > >> Hi Haomai, > >> > >> I was able to run this after a couple changes to the filestore.fio > >> job file. Two of the config options were using the wrong names. I > >> pushed a fix for the job file, as well as a patch that renames > >> everything from filestore to objectstore (thanks James), to > >> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore. > >> > >> I found that the read support doesn't appear to work anymore, so give > >> "rw=write" a try. And because it does a mkfs(), make sure you're > >> pointing it to an empty xfs directory with the "directory=" option. > >> > >> Casey > >> > >> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: > >>> Anyone who have successfully ran the fio with this external io > >>> engine ceph_objectstore? > >>> > >>> It's strange that it alwasys hit segment fault when entering > >>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but > >>> when I up the stack, the "td->io_ops" is not null. Maybe it's > >>> related to dlopen? > >>> > >>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang <haomaiwang@xxxxxxxxx> > >>> wrote: > >>>> I have rebased the branch with master, and push it to ceph upstream > >>>> repo. https://github.com/ceph/ceph/compare/fio-objectstore?expand=1 > >>>> > >>>> Plz let me know if who is working on this. Otherwise, I would like > >>>> to improve this to be merge ready. > >>>> > >>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin <matt@xxxxxxxxxxxx> > >>>> wrote: > >>>>> That makes sense. > >>>>> > >>>>> Matt > >>>>> > >>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote: > >>>>> > >>>>>> Hi Casey, > >>>>>> Got it. I was directed to the old code base. By the way, Since > >>>>>> the testing case was used to exercise all of object stores. > >>>>>> Strongly recommend to change the name from fio_ceph_filestore.cc > >>>>>> to fio_ceph_objectstore.cc . And the code in > >>>>>> fio_ceph_filestore.cc should be refactored to reflect that the > >>>>>> whole objectstore will be supported by fio_ceph_objectstore.cc. what > >>>>>> you think? > >>>>>> > >>>>>> Let me know if you need any help from my side. > >>>>>> > >>>>>> > >>>>>> Regards, > >>>>>> James > >>>>>> > >>>>>> > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] > >>>>>> Sent: Thursday, July 09, 2015 12:32 PM > >>>>>> To: James (Fei) Liu-SSI > >>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx > >>>>>> Subject: Re: About Fio backend with ObjectStore API > >>>>>> > >>>>>> Hi James, > >>>>>> > >>>>>> Are you looking at the code from > >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore? > >>>>>> It uses ObjectStore::create() instead of new FileStore(). This > >>>>>> allows us to exercise all of the object stores with the same code. > >>>>>> > >>>>>> Casey > >>>>>> > >>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI > >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: > >>>>>> > Hi Casey, > >>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically, it > >>>>>> creates a filestore as backend engine for IO exercises. If we got > >>>>>> to send IO commands to KeyValue Store or Newstore, we got to > >>>>>> change the code accordingly, right? I did not see any other > >>>>>> files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc. In > >>>>>> my humble opinion, we might need to create other two fio engines > >>>>>> for keyvaluestore and newstore if we want to exercise these two, > >>>>>> right? > >>>>>> > > >>>>>> > Regards, > >>>>>> > James > >>>>>> > > >>>>>> > static int fio_ceph_filestore_init(struct thread_data *td) > >>>>>> > 209 { > >>>>>> > 210 vector<const char*> args; > >>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct > >>>>>> ceph_filestore_data *) td->io_ops->data; > >>>>>> > 212 ObjectStore::Transaction ft; > >>>>>> > 213 > >>>>>> > > >>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD, > >>>>>> CODE_ENVIRONMENT_UTILITY, 0); > >>>>>> > 215 //g_conf->journal_dio = false; > >>>>>> > 216 common_init_finish(g_ceph_context); > >>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20"); > >>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20"); > >>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL); > >>>>>> > 220 > >>>>>> > > >>>>>> > 221 ceph_filestore_data->osd_path = > >>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX"); > >>>>>> > 222 ceph_filestore_data->journal_path = > >>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX" > >>>>>> ); > >>>>>> > 223 > >>>>>> > > >>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) { > >>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) << > >>>>>> std::endl; > >>>>>> > 226 return 1; > >>>>>> > 227 } > >>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue > >>>>>> > 229 > >>>>>> > > >>>>>> > 230 ObjectStore *fs = new > >>>>>> FileStore(ceph_filestore_data->osd_path, > >>>>>> ceph_filestore_data->journal_path); > >>>>>> > 231 ceph_filestore_data->fs = fs; > >>>>>> > 232 > >>>>>> > > >>>>>> > 233 if (fs->mkfs() < 0) { > >>>>>> > 234 cout << "mkfs failed" << std::endl; > >>>>>> > 235 goto failed; > >>>>>> > 236 } > >>>>>> > 237 > >>>>>> > 238 if (fs->mount() < 0) { > >>>>>> > 239 cout << "mount failed" << std::endl; > >>>>>> > 240 goto failed; > >>>>>> > 241 } > >>>>>> > 242 > >>>>>> > > >>>>>> > 243 ft.create_collection(coll_t()); > >>>>>> > 244 fs->apply_transaction(ft); > >>>>>> > 245 > >>>>>> > > >>>>>> > 246 > >>>>>> > > >>>>>> > 247 return 0; > >>>>>> > 248 > >>>>>> > > >>>>>> > 249 failed: > >>>>>> > 250 return 1; > >>>>>> > 251 > >>>>>> > > >>>>>> > 252 } > >>>>>> > -----Original Message----- > >>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx] > >>>>>> > Sent: Thursday, July 09, 2015 9:19 AM > >>>>>> > To: James (Fei) Liu-SSI > >>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx > >>>>>> > Subject: Re: About Fio backend with ObjectStore API > >>>>>> > > >>>>>> > Hi James, > >>>>>> > > >>>>>> > In the job file src/test/filestore.fio, you can modify the line > >>>>>> > "objectstore=filestore" to use any objectstore type supported > >>>>>> > by > >>>>>> the > >>>>>> > ObjectStore::create() factory. > >>>>>> > > >>>>>> > Casey > >>>>>> > > >>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI > >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: > >>>>>> >> Hi Casey, > >>>>>> >> Quick questions, The code in the trunk only cover the test > >>>>>> >> for > >>>>>> filestore. I was wondering do you have any plan to cover the test > >>>>>> for kvstore and newstore? > >>>>>> >> > >>>>>> >> Thanks, > >>>>>> >> James > >>>>>> >> > >>>>>> >> -----Original Message----- > >>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of James > >>>>>> >> (Fei) > >>>>>> > >>>>>> >> Liu-SSI > >>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM > >>>>>> >> To: Casey Bodley > >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx > >>>>>> >> Subject: RE: About Fio backend with ObjectStore API > >>>>>> >> > >>>>>> >> Hi Casey, > >>>>>> >> > >>>>>> >> Thanks a lot. > >>>>>> >> > >>>>>> >> Regards, > >>>>>> >> James > >>>>>> >> > >>>>>> >> -----Original Message----- > >>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx] > >>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM > >>>>>> >> To: James (Fei) Liu-SSI > >>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx > >>>>>> >> Subject: Re: About Fio backend with ObjectStore API > >>>>>> >> > >>>>>> >> Hi, > >>>>>> >> > >>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph Performance > >>>>>> >> Analysis: fio and RBD" at > >>>>>> >> > >>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-a > >>>>>> naly > >>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that > >>>>>> >> linked directly into ceph's FileStore. I was able to find > >>>>>> >> Daniel's branch > >>>>>> on > >>>>>> >> github at > >>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2, and > >>>>>> did some more work on it at the time. > >>>>>> >> > >>>>>> >> I just rebased that work onto the latest ceph master branch, > >>>>>> >> and > >>>>>> pushed to our github at > >>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore. > >>>>>> You can find the source in src/test/fio_ceph_filestore.cc, and > >>>>>> run fio with the provided example fio job file in > >>>>>> src/test/filestore.fio. > >>>>>> >> > >>>>>> >> I didn't have a chance to confirm that it builds with > >>>>>> >> automake, but > >>>>>> > >>>>>> >> the cmake version built for me. I'm happy to help if you run > >>>>>> >> into problems, Casey > >>>>>> >> > >>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI > >>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote: > >>>>>> >>> Hi Haomai, > >>>>>> >>> What are you trying to ask is to benchmark local > >>>>>> objectstore(like kvstore/filestore/newstore) locally with > >>>>>> FIO(ObjectStore engine)? You want to purely compare the > >>>>>> performance locally for these objectstores, right? > >>>>>> >>> > >>>>>> >>> Regards, > >>>>>> >>> James > >>>>>> >>> > >>>>>> >>> -----Original Message----- > >>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Haomai > >>>>>> Wang > >>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM > >>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx > >>>>>> >>> Subject: About Fio backend with ObjectStore API > >>>>>> >>> > >>>>>> >>> Hi all, > >>>>>> >>> > >>>>>> >>> Long long ago, is there someone said about fio backend with > >>>>>> >>> Ceph > >>>>>> ObjectStore API? So we could use the existing mature fio facility > >>>>>> to benchmark ceph objectstore. > >>>>>> >>> > >>>>>> >>> -- > >>>>>> >>> Best Regards, > >>>>>> >>> > >>>>>> >>> Wheat > >>>>>> >>> -- > >>>>>> >>> To unsubscribe from this list: send the line "unsubscribe > >>>>>> ceph-devel" > >>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More > >>>>>> majordomo > >>>>>> >>> info at http://vger.kernel.org/majordomo-info.html > >>>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+ > >>>>>> +zf h ~ i z w ? & )ߢ f > >>>>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w > >>>>>> j:+v w j m zZ+ ݢj" > >>>>> > >>>>> -- > >>>>> Matt Benjamin > >>>>> CohortFS, LLC. > >>>>> 315 West Huron Street, Suite 140A > >>>>> Ann Arbor, Michigan 48103 > >>>>> > >>>>> http://cohortfs.com > >>>>> > >>>>> tel. 734-761-4689 > >>>>> fax. 734-769-8938 > >>>>> cel. 734-216-5309 > >>>> > >>>> > >>>> > >>>> -- > >>>> Best Regards, > >>>> > >>>> Wheat > >>> > >>> > >>> > >>> -- > >>> Best Regards, > >>> > >>> Wheat > > > > > > > > -- > > Best Regards, > > > > Wheat > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > > -- > Best Regards, > > Wheat > N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������j:+v���w�j�m��������zZ+��ݢj"�� -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
