Do you have any locking which guarantees that nodes don't copy files which are still in the process of being written? CephFS will guarantee any readers see the results of writes which are already reported complete while reading, but I don't see any guarantees about atomicity in https://docs.microsoft.com/en-us/dotnet/api/system.io.file.writeallbytes?view=net-6.0, and those byte counts sound like they may be C# pages which are getting sent out incrementally. -Greg On Thu, May 5, 2022 at 1:30 PM Kiran Ramesh <kirame@xxxxxxxxxxxxx> wrote: > > We have a kubernetes cluster with 18 nodes (1 master + 17 workers) with ceph cluster setup with rook operator. > > There are 3 monitors. Each of the worker nodes expose a disk as OSD, so 17 OSDs. There is a one active MDS and data is set to have 3 replicas. > > A PVC from the ceph file storage is mounted as a volume in 12 of the worker nodes. > > This storage is used as a shared cache. Each of the worker checks for a file in its local disk which if misses, looks for the file in the shared cache. If present, pulls it into the local cache. If the file is not present in the shared cache, file is downloaded from the source into the local disk and then copied over to the shared cache. > > The file read/write operations is done through the C# System.IO : File.ReadAllBytes and File.WriteAllBytes methods. > > When the worker has a shared cache hit and is copied over from the shared cache to local disk, occasionally 0, 512, 1024 or 2048 bytes alone are copied by C# file operations. > > Thanks, > Kiran > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |