Hi Xiaoxi, We fixed the problems which from fio-objectstore and Makefile. We are able to enable both rocksdb and newstore at same time with fio-objectstore. However, the fio got below errors while issue the commands to the newstore. 2015-09-18 16:26:29.471099 7ffd86aec700 -1 newstore(/mnt/nvmedevice/) _aio_thread got (14) Bad address 2015-09-18 16:26:29.471106 7ffd86aec700 -1 newstore(/mnt/nvmedevice/) _aio_thread got (14) Bad address 2015-09-18 16:26:29.471112 7ffd86aec700 -1 newstore(/mnt/nvmedevice/) _aio_thread got (14) Bad address We are debugging and trying to fix them. thanks Regards, James -----Original Message----- From: James (Fei) Liu-SSI Sent: Friday, September 18, 2015 2:55 PM To: 'Chen, Xiaoxi' Cc: 'ceph-devel@xxxxxxxxxxxxxxx' Subject: RE: Newstore create failed with fio_objectstore Hi Xiaoxi and Cephers, Thanks for your feedback. I am trying to get newstore works with rocksdb in fio-objectstore plugin. Here are steps I did: 1. Reconfigure the ceph with RocksDB build in ./configure --with-fio-dir=./src/fio/ --with-librocksdb-static 2. sudo ./fio/fio ./test/objectstore.fio I always get error of undefined symbol RocksDBStore. fio: engine ./.libs/libfio_ceph_objectstore.so not loadable fio: failed to load engine ./.libs/libfio_ceph_objectstore.so Bad option <objectstore=newstore> Did you mean directory? Bad option <experimentalfeature=newstore> Did you mean experimental_verify? fio: job newstore dropped fio: file:ioengines.c:99, func=dlopen, error=./.libs/libfio_ceph_objectstore.so: undefined symbol: _ZTV12RocksDBStore libgcc_s.so.1 must be installed for pthread_cancel to work 3. nm .libs/libos.a |grep RocksDBStore I do find that RocksDBStore is undefined after I used nm. U _ZN12RocksDBStore10_test_initERKSs 0000000000000000 W _ZN12RocksDBStore13CompactThread5entryEv 0000000000000000 W _ZN12RocksDBStore13CompactThreadD0Ev 0000000000000000 W _ZN12RocksDBStore13CompactThreadD1Ev 0000000000000000 W _ZN12RocksDBStore13CompactThreadD2Ev 0000000000000000 n _ZN12RocksDBStore13CompactThreadD5Ev U _ZN12RocksDBStore20compact_thread_entryEv 0000000000000000 V _ZTIN12RocksDBStore13CompactThreadE 0000000000000000 V _ZTSN12RocksDBStore13CompactThreadE U _ZTV12RocksDBStore 0000000000000000 V _ZTVN12RocksDBStore13CompactThreadE Do you have any clue why RocksDBStore is undefined even though I already compiled the code with --with-librocksdb-static flags? Adding other Ceph developers just in case they got same issues. Thanks, James From: Chen, Xiaoxi [mailto:xiaoxi.chen@xxxxxxxxx] Sent: Thursday, September 17, 2015 7:09 PM To: James (Fei) Liu-SSI Subject: RE: Newstore create failed with fio_objectstore Hi James, On default , newstore_backend is rocksdb, so is that your ceph build with –with-rocksdb or –with-rocksdb-static? Pls note after you have rocksdb support, then you also need to have rocksdb listed in the experimental feature, otherwise an experimental checking error will present. Since now you don’t set the experimental feature and don’t see the error, so I suspect your ceph is not build with rocksdb. Changing the default to LevelDB may or maynot work as most of the development and testing are based on rocksdb, but it would be very helpful if you would like to help us playing on LevelDB. By the way, great job on fio_objectstore, which is really good for backend bottleneck analysis. --Xiaoxi From: James (Fei) Liu-SSI [mailto:james.liu@xxxxxxxxxxxxxxx] Sent: Friday, September 18, 2015 1:47 AM To: Chen, Xiaoxi Subject: Newstore create failed with fio_objectstore Hi Xiaoxi, I am trying to run fio-objectstore against the new store . Here is my newstore code for supporting newstore with fio-objectstore. Here is fio configure file. ###################################################################### # Example test for the external fio ioengine for ObjectStore. # # Runs a 4k random write test against a ObjectStore configuration. # ###################################################################### [global] #logging #write_iops_log=write_iops_log #write_bw_log=write_bw_log #write_lat_log=write_lat_log ioengine=./.libs/libfio_ceph_objectstore.so invalidate=0 # mandatory rw=write bs=128k size=2G numjobs=1 [newstore] iodepth=16 # create a journaled filestore objectstore=newstore experimentalfeature=newstore directory=/mnt/nvmedevice/ #filestore_journal=/tmp/filestore However, I am always facing below errors while trying to create leveldb for newstore. jamesliu@jamesliu-OptiPlex-7010:~/WorkSpace/ceph_casey/src$ sudo fio/fio ./test/objectstore.fio newstore: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=cephobjectstore, iodepth=16 fio-2.2.9-56-g736a Starting 1 process newstore: Laying out IO file(s) (1 file(s) / 2048MB) 2015-09-17 10:25:28.167283 7f2044733840 -1 WARNING: the following dangerous and experimental features are enabled: newstore 2015-09-17 10:25:28.167308 7f2044733840 -1 WARNING: experimental feature 'newstore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2015-09-17 10:25:28.167732 7f2044733840 1 newstore(/mnt/nvmedevice/) mkfs path /mnt/nvmedevice/ 2015-09-17 10:25:28.167761 7f2044733840 1 newstore(/mnt/nvmedevice/) _open_path using fs driver 'generic' 2015-09-17 10:25:28.168055 7f2044733840 1 newstore(/mnt/nvmedevice/) mkfs generated fsid 0d430a18-b88f-4466-9d40-636ea17543f2 2015-09-17 10:25:28.168751 7f2044733840 1 newstore(/mnt/nvmedevice/) _open_db !!!!!!!error creating db!!!!!! fio: io engine init failed. Perhaps try reducing io depth? fio: pid=18049, err=1/ Thanks for your help in advance. Regards, James From: Haomai Wang [mailto:haomaiwang@xxxxxxxxx] Sent: Wednesday, September 16, 2015 6:51 PM To: James (Fei) Liu-SSI Subject: Re: GDB can not stop in the newstore or any functions in libos.a You can attach running ceph-osd process and breakpoint at any ObjectStore functions. I'm not sure what the problem you met. On Thu, Sep 17, 2015 at 7:46 AM, James (Fei) Liu-SSI <james.liu@xxxxxxxxxxxxxxx> wrote: Hi Haomai, Do you use GDB to debug objectstore code? If yes, how do you set it up? We can not step in the newstore sourcecode. Thanks, James -- Best Regards, Wheat ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
