|
Hi Yuan,
Thanks for the answer. Our main use case is to replace AWS S3 with object storage in private cloud, very preferably with S3 compatible API. But we also know that we want to perform some machine learning and data processing by Spark in not so far future on the data residing in the object storage. The data locality feature would be very nice to have but I was not aware that this is possible with Ceph or Swift. We do not want to use HDFS, mainly because of the cost brought in by replication factor 3x and we also plan to store more/a lot of smaller files. Best regards, Milan > From: dunk007@xxxxxxxxx > Date: Wed, 17 Jun 2015 23:41:48 +0800 > Subject: Re: Accessing Ceph from Spark > To: milan.sladky@xxxxxxxxxxx > CC: ceph-users@xxxxxxxxxxxxxx > > Hi Milan, > > We've done some tests here and our hadoop can talk to RGW successfully > with this SwiftFS plugin. But we haven't tried Spark yet. One thing is > the data locality feature, it actually requires some special > configuration of Swift proxy-server, so RGW is not able to archive the > data locality there. > > Could you please kindly share some deployment consideration of running > Spark on Swift/Ceph? Tachyon seems more promising... > > > Sincerely, Yuan > > > On Wed, Jun 17, 2015 at 9:58 PM, Milan Sladky <milan.sladky@xxxxxxxxxxx> wrote: > > Is it possible to access Ceph from Spark as it is mentioned here for > > Openstack Swift? > > > > https://spark.apache.org/docs/latest/storage-openstack-swift.html > > > > Thanks for help. > > > > Milan Sladky > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
