An update on the port of SCST to run entirely in usermode: I've created a new branch "ceph_rbd" in my SCST-Usermode-Adaptation repository which includes a "shim" between the SCST BLOCKIO code (when running in usermode), and the Ceph librbd API. https://github.com/DavidButterfield/SCST-Usermode-Adaptation/blob/ceph_rbd/scst/src/dev_handlers/scst_vdisk_ceph_rbd.c This enables a purely usermode instance of SCST to serve iSCSI initiators in a TCP/IP network using calls to librbd for access to the backing store. There is no SCST code running in the kernel -- the usermode server uses regular socket(2) calls for access to the network, with the iSCSI protocol interpreter and the core SCSI engine both running in usermode. Running the iSCSI/SCST/RBD server in usermode using regular socket calls opens the possibility of building it for non-Linux systems (e.g. FreeBSD). I just decided to write this shim to Ceph RBD three days ago, and got it working today, so consider it "alpha". It doesn't handle everything perfectly right, but it runs, and read/write seem to be working. So far I've tested it by running an iSCSI session from another Linux machine (open-iscsi) to the usermode SCST/RBD server, exporting a 4GB image I created on the Ceph server: I copied some files in and out of it a few times, and compared the result with the original. That worked. At the moment you have to compile it either to use local aio(7) calls or librbd calls -- making them work side-by-side is TBD. (The idea to connect the usermode iSCSI server with Ceph came to me from Adrian Saul.) On Fri, Apr 21, 2017 at 2:18 PM, David Butterfield <dab21774@xxxxxxxxx> wrote: > This may be of interest, running SCST as an iSCSI server entirely in usermode. > It uses regular socket calls on the network and preadv/pwritev on the > backing store. > It looks like it would be straightforward to adapt it to make direct > calls to librados. > > One thing mentioned in the paper is that with iSCSI/SCST running entirely in > usermode using regular system calls, iSCSI/SCST could be compiled not only on > Linux systems, but on any POSIX-style system with gcc and the other libraries > (e.g. freebsd). > > https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_Usermode.html > > Regards, > David Butterfield > ----------------- > > iSCSI-SCST Storage Server Usermode Adaptation > An adaptation of the iSCSI-SCST storage server software to run > entirely in usermode on an unmodified kernel > David A. Butterfield > > This paper describes an adaptation of the iSCSI-SCST storage server software to > run entirely in usermode on an unmodified Linux kernel; performance > measurements and model; and an experimental algorithm to improve performance > for small Read operations. > > In a standard installation of SCST the iscsi-scstd daemon runs as a > single-threaded Linux usermode process that cooperates with the kernel-resident > SCST datapath implementation using ioctl(2) and netlink(7) for communication. > > In the iSCSI-SCST Usermode Adaptation the iscsi-scstd daemon runs on the main > thread in a multi-threaded process in which other usermode threads are > concurrently providing the services and executing the SCST code that would be > running inside the kernel in a standard installation of SCST. > > The iSCSI server executable program can run as a regular (non-super) user, as > long as it has permission to access the backing storage (file or block device). > Administration is done in the usual SCST way using scstadmin, which accesses > the running server program through a fuse-mounted filesystem implemented using > a shim to connect the SCST procfs calls with the fuse(8) filesystem API. > > The subset of SCST used supports the iSCSI transport type and SCSI Block > Commands (vdisk). It includes the SCST Core, the iSCSI daemon and kernel > logic, the vdisk device, and the /proc interface; comprising about 80,000 lines > of SCST source code. To support running in usermode, around 55 (fifty-five) > lines of executable C code had to be added or changed in SCST source files. > > For a single session over 1 Gb Ethernet being serviced by a single 2.4 GHz CPU: > the described Adaptive Nagle optimization improves peak throughput performance > for 512-Byte Random Read of /dev/zero from around 63,000 IOPS to more than > 100,000 IOPS, with no adverse impact below Queue Depth 17. > > Paper: https://davidbutterfield.github.io/SCST-Usermode-Adaptation/docs/SCST_Usermode.html > Code: https://github.com/DavidButterfield/SCST-Usermode-Adaptation -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
