Hi Haomai and Case,
Do you have any fixes for that segfault?
Thanks,
James
-----Original Message-----
From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx]
Sent: Wednesday, July 22, 2015 6:07 PM
To: Casey Bodley
Cc: Casey Bodley; Matt W. Benjamin; James (Fei) Liu-SSI; ceph-devel@xxxxxxxxxxxxxxx
Subject: Re: About Fio backend with ObjectStore API
no special
[global]
#logging
#write_iops_log=write_iops_log
#write_bw_log=write_bw_log
#write_lat_log=write_lat_log
ioengine=./ceph-int/src/.libs/libfio_ceph_objectstore.so
invalidate=0 # mandatory
rw=write
#bs=4k
[filestore]
iodepth=1
# create a journaled filestore
objectstore=filestore
directory=./osd/
filestore_journal=./osd/journal
On Thu, Jul 23, 2015 at 4:56 AM, Casey Bodley <cbodley@xxxxxxxxxx> wrote:
> Hi Haomai,
>
> Sorry for the late response, I was out of the office. I'm afraid I haven't run into that segfault. The io_ops should be set at the very beginning when it calls get_ioengine(). All I can suggest is that you verify that your job file is pointing to the correct fio_ceph_objectstore.so. If you've made any other interesting changes to the job file, could you share it here?
>
> Casey
>
> ----- Original Message -----
> From: "Haomai Wang" <haomaiwang@xxxxxxxxx>
> To: "Casey Bodley" <cbodley@xxxxxxxxx>
> Cc: "Matt W. Benjamin" <matt@xxxxxxxxxxxx>, "James (Fei) Liu-SSI"
> <james.liu@xxxxxxxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
> Sent: Tuesday, July 21, 2015 7:50:32 AM
> Subject: Re: About Fio backend with ObjectStore API
>
> Hi Casey,
>
> I check your commits and know what you fixed. I cherry-picked your new
> commits but I still met the same problem.
>
> """
> It's strange that it alwasys hit segment fault when entering
> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but
> when I up the stack, the "td->io_ops" is not null. Maybe it's related
> to dlopen?
> """
>
> Do you have any hint about this?
>
> On Thu, Jul 16, 2015 at 5:23 AM, Casey Bodley <cbodley@xxxxxxxxx> wrote:
>> Hi Haomai,
>>
>> I was able to run this after a couple changes to the filestore.fio
>> job file. Two of the config options were using the wrong names. I
>> pushed a fix for the job file, as well as a patch that renames
>> everything from filestore to objectstore (thanks James), to
>> https://github.com/linuxbox2/linuxbox-ceph/commits/fio-objectstore.
>>
>> I found that the read support doesn't appear to work anymore, so give
>> "rw=write" a try. And because it does a mkfs(), make sure you're
>> pointing it to an empty xfs directory with the "directory=" option.
>>
>> Casey
>>
>> On Tue, Jul 14, 2015 at 2:45 AM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
>>> Anyone who have successfully ran the fio with this external io
>>> engine ceph_objectstore?
>>>
>>> It's strange that it alwasys hit segment fault when entering
>>> "_fio_setup_ceph_filestore_data", gdb tells "td->io_ops" is NULL but
>>> when I up the stack, the "td->io_ops" is not null. Maybe it's
>>> related to dlopen?
>>>
>>> On Fri, Jul 10, 2015 at 3:51 PM, Haomai Wang <haomaiwang@xxxxxxxxx> wrote:
>>>> I have rebased the branch with master, and push it to ceph upstream
>>>> repo. https://github.com/ceph/ceph/compare/fio-objectstore?expand=1
>>>>
>>>> Plz let me know if who is working on this. Otherwise, I would like
>>>> to improve this to be merge ready.
>>>>
>>>> On Fri, Jul 10, 2015 at 4:26 AM, Matt W. Benjamin <matt@xxxxxxxxxxxx> wrote:
>>>>> That makes sense.
>>>>>
>>>>> Matt
>>>>>
>>>>> ----- "James (Fei) Liu-SSI" <james.liu@xxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Casey,
>>>>>> Got it. I was directed to the old code base. By the way, Since
>>>>>> the testing case was used to exercise all of object stores.
>>>>>> Strongly recommend to change the name from fio_ceph_filestore.cc
>>>>>> to fio_ceph_objectstore.cc . And the code in
>>>>>> fio_ceph_filestore.cc should be refactored to reflect that the
>>>>>> whole objectstore will be supported by fio_ceph_objectstore.cc. what you think?
>>>>>>
>>>>>> Let me know if you need any help from my side.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> James
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>>>>> Sent: Thursday, July 09, 2015 12:32 PM
>>>>>> To: James (Fei) Liu-SSI
>>>>>> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>>>>> Subject: Re: About Fio backend with ObjectStore API
>>>>>>
>>>>>> Hi James,
>>>>>>
>>>>>> Are you looking at the code from
>>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore?
>>>>>> It uses ObjectStore::create() instead of new FileStore(). This
>>>>>> allows us to exercise all of the object stores with the same code.
>>>>>>
>>>>>> Casey
>>>>>>
>>>>>> On Thu, Jul 9, 2015 at 2:01 PM, James (Fei) Liu-SSI
>>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>>>>> > Hi Casey,
>>>>>> > Here is the code in the fio_ceph_filestore.cc. Basically, it
>>>>>> creates a filestore as backend engine for IO exercises. If we got
>>>>>> to send IO commands to KeyValue Store or Newstore, we got to
>>>>>> change the code accordingly, right? I did not see any other
>>>>>> files like fio_ceph_keyvaluestore.cc or fio_ceph_newstore.cc. In
>>>>>> my humble opinion, we might need to create other two fio engines
>>>>>> for keyvaluestore and newstore if we want to exercise these two, right?
>>>>>> >
>>>>>> > Regards,
>>>>>> > James
>>>>>> >
>>>>>> > static int fio_ceph_filestore_init(struct thread_data *td)
>>>>>> > 209 {
>>>>>> > 210 vector<const char*> args;
>>>>>> > 211 struct ceph_filestore_data *ceph_filestore_data = (struct
>>>>>> ceph_filestore_data *) td->io_ops->data;
>>>>>> > 212 ObjectStore::Transaction ft;
>>>>>> > 213
>>>>>> >
>>>>>> > 214 global_init(NULL, args, CEPH_ENTITY_TYPE_OSD,
>>>>>> CODE_ENVIRONMENT_UTILITY, 0);
>>>>>> > 215 //g_conf->journal_dio = false;
>>>>>> > 216 common_init_finish(g_ceph_context);
>>>>>> > 217 //g_ceph_context->_conf->set_val("debug_filestore", "20");
>>>>>> > 218 //g_ceph_context->_conf->set_val("debug_throttle", "20");
>>>>>> > 219 g_ceph_context->_conf->apply_changes(NULL);
>>>>>> > 220
>>>>>> >
>>>>>> > 221 ceph_filestore_data->osd_path =
>>>>>> strdup("/mnt/fio_ceph_filestore.XXXXXXX");
>>>>>> > 222 ceph_filestore_data->journal_path =
>>>>>> strdup("/var/lib/ceph/osd/journal-ram/fio_ceph_filestore.XXXXXXX"
>>>>>> );
>>>>>> > 223
>>>>>> >
>>>>>> > 224 if (!mkdtemp(ceph_filestore_data->osd_path)) {
>>>>>> > 225 cout << "mkdtemp failed: " << strerror(errno) <<
>>>>>> std::endl;
>>>>>> > 226 return 1;
>>>>>> > 227 }
>>>>>> > 228 //mktemp(ceph_filestore_data->journal_path); // NOSPC issue
>>>>>> > 229
>>>>>> >
>>>>>> > 230 ObjectStore *fs = new
>>>>>> FileStore(ceph_filestore_data->osd_path,
>>>>>> ceph_filestore_data->journal_path);
>>>>>> > 231 ceph_filestore_data->fs = fs;
>>>>>> > 232
>>>>>> >
>>>>>> > 233 if (fs->mkfs() < 0) {
>>>>>> > 234 cout << "mkfs failed" << std::endl;
>>>>>> > 235 goto failed;
>>>>>> > 236 }
>>>>>> > 237
>>>>>> > 238 if (fs->mount() < 0) {
>>>>>> > 239 cout << "mount failed" << std::endl;
>>>>>> > 240 goto failed;
>>>>>> > 241 }
>>>>>> > 242
>>>>>> >
>>>>>> > 243 ft.create_collection(coll_t());
>>>>>> > 244 fs->apply_transaction(ft);
>>>>>> > 245
>>>>>> >
>>>>>> > 246
>>>>>> >
>>>>>> > 247 return 0;
>>>>>> > 248
>>>>>> >
>>>>>> > 249 failed:
>>>>>> > 250 return 1;
>>>>>> > 251
>>>>>> >
>>>>>> > 252 }
>>>>>> > -----Original Message-----
>>>>>> > From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>>>>> > Sent: Thursday, July 09, 2015 9:19 AM
>>>>>> > To: James (Fei) Liu-SSI
>>>>>> > Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>>>>> > Subject: Re: About Fio backend with ObjectStore API
>>>>>> >
>>>>>> > Hi James,
>>>>>> >
>>>>>> > In the job file src/test/filestore.fio, you can modify the line
>>>>>> > "objectstore=filestore" to use any objectstore type supported
>>>>>> > by
>>>>>> the
>>>>>> > ObjectStore::create() factory.
>>>>>> >
>>>>>> > Casey
>>>>>> >
>>>>>> > On Wed, Jul 8, 2015 at 8:02 PM, James (Fei) Liu-SSI
>>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>>>>> >> Hi Casey,
>>>>>> >> Quick questions, The code in the trunk only cover the test
>>>>>> >> for
>>>>>> filestore. I was wondering do you have any plan to cover the test
>>>>>> for kvstore and newstore?
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> James
>>>>>> >>
>>>>>> >> -----Original Message-----
>>>>>> >> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>>>>> >> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of James
>>>>>> >> (Fei)
>>>>>>
>>>>>> >> Liu-SSI
>>>>>> >> Sent: Tuesday, June 30, 2015 2:19 PM
>>>>>> >> To: Casey Bodley
>>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>>>>> >> Subject: RE: About Fio backend with ObjectStore API
>>>>>> >>
>>>>>> >> Hi Casey,
>>>>>> >>
>>>>>> >> Thanks a lot.
>>>>>> >>
>>>>>> >> Regards,
>>>>>> >> James
>>>>>> >>
>>>>>> >> -----Original Message-----
>>>>>> >> From: Casey Bodley [mailto:cbodley@xxxxxxxxx]
>>>>>> >> Sent: Tuesday, June 30, 2015 2:16 PM
>>>>>> >> To: James (Fei) Liu-SSI
>>>>>> >> Cc: Haomai Wang; ceph-devel@xxxxxxxxxxxxxxx
>>>>>> >> Subject: Re: About Fio backend with ObjectStore API
>>>>>> >>
>>>>>> >> Hi,
>>>>>> >>
>>>>>> >> When Danny Al-Gaaf & Daniel Gollub published "Ceph Performance
>>>>>> >> Analysis: fio and RBD" at
>>>>>> >>
>>>>>> https://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-a
>>>>>> naly
>>>>>> >> s is_fio_rbd.html, they also mentioned a fio engine that
>>>>>> >> linked directly into ceph's FileStore. I was able to find
>>>>>> >> Daniel's branch
>>>>>> on
>>>>>> >> github at
>>>>>> >> https://github.com/gollub/ceph/tree/fio_filestore_v2, and
>>>>>> did some more work on it at the time.
>>>>>> >>
>>>>>> >> I just rebased that work onto the latest ceph master branch,
>>>>>> >> and
>>>>>> pushed to our github at
>>>>>> https://github.com/linuxbox2/linuxbox-ceph/tree/fio-objectstore.
>>>>>> You can find the source in src/test/fio_ceph_filestore.cc, and
>>>>>> run fio with the provided example fio job file in src/test/filestore.fio.
>>>>>> >>
>>>>>> >> I didn't have a chance to confirm that it builds with
>>>>>> >> automake, but
>>>>>>
>>>>>> >> the cmake version built for me. I'm happy to help if you run
>>>>>> >> into problems, Casey
>>>>>> >>
>>>>>> >> On Tue, Jun 30, 2015 at 2:31 PM, James (Fei) Liu-SSI
>>>>>> <james.liu@xxxxxxxxxxxxxxx> wrote:
>>>>>> >>> Hi Haomai,
>>>>>> >>> What are you trying to ask is to benchmark local
>>>>>> objectstore(like kvstore/filestore/newstore) locally with
>>>>>> FIO(ObjectStore engine)? You want to purely compare the
>>>>>> performance locally for these objectstores, right?
>>>>>> >>>
>>>>>> >>> Regards,
>>>>>> >>> James
>>>>>> >>>
>>>>>> >>> -----Original Message-----
>>>>>> >>> From: ceph-devel-owner@xxxxxxxxxxxxxxx
>>>>>> >>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Haomai
>>>>>> Wang
>>>>>> >>> Sent: Tuesday, June 30, 2015 9:06 AM
>>>>>> >>> To: ceph-devel@xxxxxxxxxxxxxxx
>>>>>> >>> Subject: About Fio backend with ObjectStore API
>>>>>> >>>
>>>>>> >>> Hi all,
>>>>>> >>>
>>>>>> >>> Long long ago, is there someone said about fio backend with
>>>>>> >>> Ceph
>>>>>> ObjectStore API? So we could use the existing mature fio facility
>>>>>> to benchmark ceph objectstore.
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Best Regards,
>>>>>> >>>
>>>>>> >>> Wheat
>>>>>> >>> --
>>>>>> >>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> ceph-devel"
>>>>>> >>> in the body of a message to majordomo@xxxxxxxxxxxxxxx More
>>>>>> majordomo
>>>>>> >>> info at http://vger.kernel.org/majordomo-info.html
>>>>>> >> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+
>>>>>> +zf h ~ i z w ? & )ߢ f
>>>>>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w
>>>>>> j:+v w j m zZ+ ݢj"
>>>>>
>>>>> --
>>>>> Matt Benjamin
>>>>> CohortFS, LLC.
>>>>> 315 West Huron Street, Suite 140A
>>>>> Ann Arbor, Michigan 48103
>>>>>
>>>>> http://cohortfs.com
>>>>>
>>>>> tel. 734-761-4689
>>>>> fax. 734-769-8938
>>>>> cel. 734-216-5309
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>>
>>>> Wheat
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>
>
>
> --
> Best Regards,
>
> Wheat
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo
> info at http://vger.kernel.org/majordomo-info.html
--
Best Regards,
Wheat
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
