James, I echo what Christian Balzer says. DO not fixate on CEPH at this stage, we need to look at what the requirements are, There are alternatives such as Spectrum Scale and Minio. Also, depending on how often the videos are to be recalled, looking at a tape based solution.
Regarding hardware, Supermicro is a good place to start. I helped supply 20 Petabytes of CEPH storage to an STFC site based on the 36 bay Supermicro storage servers.
Regarding the requirement for wireless transfer of the data we would have to examine this. I am sure that you have a good reason for this, though there should be a way to do this using a high bandwidth connection - 40 or 100Gbps Ethernet. I would look at basing your solution around a modern high bandwidth network anyway.
Are you based in the UK? If so, we should talk off list.
John Hearns
On Mon, 27 Aug 2018 at 03:39, Christian Balzer <chibi@xxxxxxx> wrote:
Hello,
On Sun, 26 Aug 2018 22:23:53 +0400 James Watson wrote:
> Hi CEPHers,
>
> I need to design an HA CEPH object storage system.
The first question that comes to mind is why?
Why does it need to be Ceph and why object based (RGW)?
>From what's stated below it seems that nobody at your end has in depth
experience with Ceph and related HW and the bit about getting that amount
of data in via WiFi boggles the mind.
HA is implied with Ceph, the bits that feed into it or read out if it will
be your problem/responsibility.
> The scenario is that we
> are recording HD Videos and end of the day we need to copy all these video
> files (each file has approx 15 TB ) to our storage system.
>
How many of these 15TB files, how long do you need to keep them,
estimated future growth, etc?
> 1)Which would be the best tech in storage to transfer these PBs size loads
> of videos to CEPH based object storage wirelessly.
>
As you already figured out, wireless is going to be an issue.
Are those 15TB files accumulated locally (sounds odd for remote cameras to
have such storage)?
Unless you deploy something beefier in the network department OR go to
sneakernet this won't work well.
> 2)How should I design my CEPH in the scale of PBs and make sure its future
> proof.
>
Lots of examples around if you're looking around here and on google.
Dense nodes (lots of HDDs) work if you have enough of them, otherwise
smaller is preferable.
> 3)What are the latest hardware components I might require to accomplish
> this task?
>
You're entering the realm of paid consulting here, which I'm sure some
people present might be willing to do.
Vendors like Supermicro probably, too.
Heck, you can buy it off the shelf even (not the only one out there):
http://www.fujitsu.com/global/products/computing/storage/eternus-cd/s2/
In general your needs sound a lot like large streaming (in and out), which
has less demands on the HW then lots of small IOPS.
>
> I am a software developer and am new to this domain. Kindly request all to
> provide the name of even the most basic of hardware components required for
> the setup so that I can do a cost estimation and compare with other techs.
>
You can do cheap with Ceph, _if_ you know what you're doing.
OTOH Ceph emphasizes redundancy first, so compared with other solutions it
may not be as cheap as it could be, depending on configuration choices and
performance requirements.
> My novice solution so far:
>
> 1. Transmitting module if using WiFi (802.11ac (aka Gigabit Wifi) max 200
> Mbps speed) to transfer a file of size 15 TB to CEPH Storage takes 7 days !!
>
> 2.CEPH needs to be configured with High Availability
> A SAN with FC networking in place (GEN 6 SANS) using NVMe SSD with HBAs
> that support NVMe over Fibre Channel giving a transfer rate of 16 Gbps to
> Host Server.
>
Ceph is all about local storage per node, SAN and FC are anathema and NVMe
is likely not needed in your scenario, at least not for actual storage
space.
Christian
--
Christian Balzer Network/Systems Engineer
chibi@xxxxxxx Rakuten Communications
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
