On Mon, Oct 1, 2018 at 8:43 PM Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx> wrote: > > Hmmm, did not know that, so it becomes a soft link or so? > > totally new for me, also not what I would expect of a mv on a fs. I know > this is normal to expect coping between pools, also from the s3cmd > client. But I think more people will not expect this behaviour. Can't > the move be implemented as a move? In almost all filesystems, a rename (like "mv") is a pure metadata operation -- it doesn't involve reading all the file's data and re-writing it. It would be very surprising for most users if they found that their "mv" command blocked for a very long time while waiting for a large file's content to be e.g. read out of one pool and written into another. If would be especially surprising if the system was close to being full, and the user found that "mv" operations started getting ENOSPC because there wasn't enough room to write to a new location before clearing data out of the old one. All that said, I would quite like to be able to explicitly change layouts on files and have the data copy happen in the background -- the route of doing a "cp -R" does feel hacky. > How can users even know about what folders have a 'different layout'. > What happens if we export such mixed pool filesystem via smb. How would > smb deal with the 'move' between those directories? Samba would pass a rename though to Ceph, so you'd have the same behaviour whether doing a mv via Samba or doing it via a native CephFS client. John > > > > -----Original Message----- > From: Gregory Farnum [mailto:gfarnum@xxxxxxxxxx] > Sent: maandag 1 oktober 2018 21:28 > To: Marc Roos > Cc: ceph-users; jspray; ukernel > Subject: Re: cephfs issue with moving files between data > pools gives Input/output error > > Moving a file into a directory with a different layout does not, and is > not intended to, copy the underlying file data into a different pool > with the new layout. If you want to do that you have to make it happen > yourself by doing a copy. > > On Mon, Oct 1, 2018 at 12:16 PM Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx> > wrote: > > > > I will explain the test again, I think you might have some bug in > your > cephfs copy between data pools. > > c04 has mounted the root cephfs > /a (has data pool a, ec21) > /test (has data pool b, r1) > > test2 has mounted > /m (nfs mount of cephfs /a) > /m2 (cephfs mount of /a) > > Creating the test file. > [root@c04 test]# echo asdfasdfasdfasdfasdfasdfasdfasdfasdf > > testfile.txt > > Then I am moving on c04 the test file from the test folder(pool b) > to > the a folder/pool > > Now on test2 > [root@test2 m]# ls -arlt > -rw-r--r-- 1 nobody nobody 21 Oct 1 20:48 r1.txt > -rw-r--r-- 1 nobody nobody 21 Oct 1 20:49 r1-copy.txt > -rw-r--r-- 1 nobody nobody 37 Oct 1 21:02 testfile.txt > > [root@test2 /]# cat /mnt/m/testfile.txt > cat: /mnt/m/old/testfile.txt: Input/output error > > [root@test2 /]# cat /mnt/m2/testfile.txt > cat: /mnt/m2/old/testfile.txt: Operation not permitted > > Now I am creating a copy of the test file in the same directory > back on > c04 > > [root@c04 a]# cp testfile.txt testfile-copy.txt > [root@c04 a]# ls -alrt > -rw-r--r-- 1 root root 21 Oct 1 20:49 r1-copy.txt > -rw-r--r-- 1 root root 37 Oct 1 21:02 testfile.txt > -rw-r--r-- 1 root root 37 Oct 1 21:07 testfile-copy.txt > > Now I trying to access the copy of testfile.txt back on test2 > (without > unmounting, or changing permissions) > > [root@test2 /]# cat /mnt/m/testfile-copy.txt > asdfasdfasdfasdfasdfasdfasdfasdfasdf > [root@test2 /]# cat /mnt/m2/testfile-copy.txt > asdfasdfasdfasdfasdfasdfasdfasdfasdf > > > > > > > > -----Original Message----- > From: Yan, Zheng [mailto:ukernel@xxxxxxxxx] > Sent: zaterdag 29 september 2018 6:55 > To: Marc Roos > Subject: Re: cephfs issue with moving files between > data > pools gives Input/output error > > check_pool_perm on pool 30 ns need Fr, but no read perm > > client does not permission to read the pool. ceph-fuse did return > EPERM > for the kernel readpage request. But kernel return -EIO for any > readpage error. > On Fri, Sep 28, 2018 at 10:09 PM Marc Roos > <M.Roos@xxxxxxxxxxxxxxxxx> > wrote: > > > > > > Is this useful? I think this is the section of the client log > when > > > > [@test2 m]$ cat out6 > > cat: out6: Input/output error > > > > 2018-09-28 16:03:39.082200 7f1ad01f1700 10 client.3246756 > fill_statx > > on 0x100010943bc snap/devhead mode 040557 mtime 2018-09-28 > > 14:49:35.349370 ctime 2018-09-28 14:49:35.349370 > > 2018-09-28 16:03:39.082223 7f1ad01f1700 3 client.3246756 > ll_getattrx > > 0x100010943bc.head = 0 > > 2018-09-28 16:03:39.082727 7f1ae813f700 10 client.3246756 > fill_statx > > on > > 0x10001698ac5 snap/devhead mode 0100644 mtime 2018-09-28 > > 14:45:50.323273 ctime 2018-09-28 14:47:47.028679 > > 2018-09-28 16:03:39.082737 7f1ae813f700 3 client.3246756 > ll_getattrx > > 0x10001698ac5.head = 0 > > 2018-09-28 16:03:39.083149 7f1ac07f8700 3 client.3246756 ll_open > > > 0x10001698ac5.head 0 > > 2018-09-28 16:03:39.083160 7f1ac07f8700 10 client.3246756 > _getattr > > mask As issued=1 > > 2018-09-28 16:03:39.083165 7f1ac07f8700 3 client.3246756 > may_open > > 0x7f1a7810ad00 = 0 > > 2018-09-28 16:03:39.083169 7f1ac07f8700 10 break_deleg: breaking > > delegs on 0x10001698ac5.head(faked_ino=0 ref=2 ll_ref=1 > cap_refs={} > > open={1=1} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFs(0=pAsLsXsFs) objectset[0x10001698ac5 ts 0/0 > objects 0 > > dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.083183 7f1ac07f8700 10 delegations_broken: > > delegations empty on 0x10001698ac5.head(faked_ino=0 ref=2 > ll_ref=1 > > cap_refs={} open={1=1} mode=100644 size=17/0 nlink=1 > mtime=2018-09-28 > > 14:45:50.323273 caps=pAsLsXsFs(0=pAsLsXsFs) > objectset[0x10001698ac5 ts > > > 0/0 objects 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 > 0x7f1a7810ad00) > > 2018-09-28 16:03:39.083198 7f1ac07f8700 10 client.3246756 > > choose_target_mds from caps on inode > 0x10001698ac5.head(faked_ino=0 > > ref=3 ll_ref=1 cap_refs={} open={1=1} mode=100644 size=17/0 > nlink=1 > > mtime=2018-09-28 14:45:50.323273 caps=pAsLsXsFs(0=pAsLsXsFs) > > objectset[0x10001698ac5 ts 0/0 objects 0 dirty_or_tx 0] > > parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.083209 7f1ac07f8700 10 client.3246756 > send_request > > > rebuilding request 1911 for mds.0 > > 2018-09-28 16:03:39.083218 7f1ac07f8700 10 client.3246756 > send_request > > client_request(unknown.0:1911 open #0x10001698ac5 2018-09-28 > > 16:03:39.083194 caller_uid=501, caller_gid=501{501,}) v4 to mds.0 > > 2018-09-28 16:03:39.084088 7f1a82ffd700 5 client.3246756 > > set_cap_epoch_barrier epoch = 24093 > > 2018-09-28 16:03:39.084097 7f1a82ffd700 10 client.3246756 mds.0 > seq > > now > > 1 > > 2018-09-28 16:03:39.084108 7f1a82ffd700 5 client.3246756 > > handle_cap_grant on in 0x10001698ac5 mds.0 seq 7 caps now > pAsLsXsFscr > > was pAsLsXsFs > > 2018-09-28 16:03:39.084118 7f1a82ffd700 10 client.3246756 > > update_inode_file_time 0x10001698ac5.head(faked_ino=0 ref=3 > ll_ref=1 > > cap_refs={} open={1=1} mode=100644 size=17/0 nlink=1 > mtime=2018-09-28 > > 14:45:50.323273 caps=pAsLsXsFs(0=pAsLsXsFs) > objectset[0x10001698ac5 ts > > > 0/0 objects 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 > 0x7f1a7810ad00) > > pAsLsXsFs ctime 2018-09-28 14:47:47.028679 mtime 2018-09-28 > > 14:45:50.323273 > > 2018-09-28 16:03:39.084133 7f1a82ffd700 10 client.3246756 > grant, new > > caps are Fcr > > 2018-09-28 16:03:39.084143 7f1a82ffd700 10 client.3246756 > insert_trace > > > from 2018-09-28 16:03:39.083217 mds.0 is_target=1 is_dentry=0 > > 2018-09-28 16:03:39.084147 7f1a82ffd700 10 client.3246756 > features > > 0x3ffddff8eea4fffb > > 2018-09-28 16:03:39.084148 7f1a82ffd700 10 client.3246756 > > update_snap_trace len 48 > > 2018-09-28 16:03:39.084181 7f1a82ffd700 10 client.3246756 > > update_snap_trace snaprealm(0x1 nref=755 c=0 seq=1 parent=0x0 > > my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, > SKIPPING > > 2018-09-28 16:03:39.084186 7f1a82ffd700 10 client.3246756 hrm > > is_target=1 is_dentry=0 > > 2018-09-28 16:03:39.084195 7f1a82ffd700 10 client.3246756 > > add_update_cap issued pAsLsXsFscr -> pAsLsXsFscr from mds.0 on > > 0x10001698ac5.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} > open={1=1} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.084268 7f1ac07f8700 10 client.3246756 > _create_fh > > 0x10001698ac5 mode 1 > > 2018-09-28 16:03:39.084280 7f1ac07f8700 3 client.3246756 ll_open > > > 0x10001698ac5.head 0 = 0 (0x7f1a24028e10) > > 2018-09-28 16:03:39.084373 7f1a82ffd700 10 client.3246756 > put_inode on > > > 0x10001698ac5.head(faked_ino=0 ref=5 ll_ref=1 cap_refs={} > open={1=1} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.084392 7f1a82ffd700 10 client.3246756 > put_inode on > > > 0x10001698ac5.head(faked_ino=0 ref=4 ll_ref=1 cap_refs={} > open={1=1} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.084899 7f1af0161700 3 client.3246756 ll_read > > > 0x7f1a24028e10 0x10001698ac5 0~17 > > 2018-09-28 16:03:39.084911 7f1af0161700 10 client.3246756 > > check_pool_perm on pool 30 ns need Fr, but no read perm > > 2018-09-28 16:03:39.086697 7f1ac02ba700 3 client.3246756 > ll_release > > (fh)0x7f1a24028e10 0x10001698ac5 > > 2018-09-28 16:03:39.086707 7f1ac02ba700 8 client.3246756 > _release_fh > > 0x7f1a24028e10 mode 1 on 0x10001698ac5.head(faked_ino=0 ref=3 > ll_ref=1 > > > cap_refs={} open={1=1} mode=100644 size=17/0 nlink=1 > mtime=2018-09-28 > > 14:45:50.323273 caps=pAsLsXsFscr(0=pAsLsXsFscr) > > objectset[0x10001698ac5 ts 0/0 objects 0 dirty_or_tx 0] > > parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.086721 7f1ac02ba700 10 client.3246756 _flush > > 0x10001698ac5.head(faked_ino=0 ref=4 ll_ref=1 cap_refs={} > open={1=0} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.086729 7f1ac02ba700 10 client.3246756 > nothing to > > flush > > 2018-09-28 16:03:39.086731 7f1ac02ba700 10 client.3246756 > put_inode on > > > 0x10001698ac5.head(faked_ino=0 ref=4 ll_ref=1 cap_refs={} > open={1=0} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.086759 7f1ac02ba700 10 client.3246756 > check_caps > > on 0x10001698ac5.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} > > open={1=0} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) wanted - > used > > - issued pAsLsXsFscr revoking - flags=0 > > 2018-09-28 16:03:39.086772 7f1ac02ba700 10 client.3246756 > > cap_delay_requeue on 0x10001698ac5.head(faked_ino=0 ref=3 > ll_ref=1 > > cap_refs={} open={1=0} mode=100644 size=17/0 nlink=1 > mtime=2018-09-28 > > 14:45:50.323273 caps=pAsLsXsFscr(0=pAsLsXsFscr) > > objectset[0x10001698ac5 ts 0/0 objects 0 dirty_or_tx 0] > > parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:39.086780 7f1ac02ba700 10 client.3246756 cap > mds.0 > > issued pAsLsXsFscr implemented pAsLsXsFscr revoking - > > 2018-09-28 16:03:39.086784 7f1ac02ba700 10 client.3246756 > delaying cap > > > release > > 2018-09-28 16:03:39.086786 7f1ac02ba700 10 client.3246756 > _release_fh > > 0x7f1a24028e10 on inode 0x10001698ac5.head(faked_ino=0 ref=3 > ll_ref=1 > > cap_refs={} open={1=0} mode=100644 size=17/0 nlink=1 > mtime=2018-09-28 > > 14:45:50.323273 caps=pAsLsXsFscr(0=pAsLsXsFscr) > > objectset[0x10001698ac5 ts 0/0 objects 0 dirty_or_tx 0] > > parents=0x7f1a780f1dd0 0x7f1a7810ad00) no async_err state > > 2018-09-28 16:03:39.086796 7f1ac02ba700 10 client.3246756 > put_inode on > > > 0x10001698ac5.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} > open={1=0} > > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 > objects > > > 0 dirty_or_tx 0] parents=0x7f1a780f1dd0 0x7f1a7810ad00) > > 2018-09-28 16:03:41.223087 7f1b0bfff700 10 client.3246741 > renew_caps() > > 2018-09-28 16:03:41.223633 7f1b0affd700 10 client.3246741 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.243573 7f1af3fff700 10 client.3246744 > renew_caps() > > 2018-09-28 16:03:41.243584 7f1af3fff700 10 client.3246744 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.244175 7f1af2ffd700 10 client.3246744 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.265549 7f1adbfff700 10 client.3246747 > renew_caps() > > 2018-09-28 16:03:41.265570 7f1adbfff700 10 client.3246747 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.266197 7f1adaffd700 10 client.3246747 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.284915 7f1ac3fff700 10 client.3246750 > renew_caps() > > 2018-09-28 16:03:41.284923 7f1ac3fff700 10 client.3246750 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.285495 7f1ac2ffd700 10 client.3246750 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.314160 7f1ab3fff700 10 client.3246753 > renew_caps() > > 2018-09-28 16:03:41.314182 7f1ab3fff700 10 client.3246753 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.314717 7f1ab2ffd700 10 client.3246753 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.333642 7f1aa0ff9700 10 client.3246586 > renew_caps() > > 2018-09-28 16:03:41.333670 7f1aa0ff9700 10 client.3246586 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.334151 7f1a9b7fe700 10 client.3246586 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.352189 7f1a88ff9700 10 client.3246756 > renew_caps() > > 2018-09-28 16:03:41.352196 7f1a88ff9700 10 client.3246756 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.352630 7f1a82ffd700 10 client.3246756 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.372079 7f1a6bfff700 10 client.3246759 > renew_caps() > > 2018-09-28 16:03:41.372087 7f1a6bfff700 10 client.3246759 > renew_caps > > mds.0 > > 2018-09-28 16:03:41.372544 7f1a6affd700 10 client.3246759 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > > > > > > > > > -----Original Message----- > > From: John Spray [mailto:jspray@xxxxxxxxxx] > > Sent: vrijdag 28 september 2018 15:45 > > To: Marc Roos > > Cc: ceph-users@xxxxxxxxxxxxxx > > Subject: Re: cephfs issue with moving files between > data > > pools gives Input/output error > > > > On Fri, Sep 28, 2018 at 2:28 PM Marc Roos > <M.Roos@xxxxxxxxxxxxxxxxx> > > wrote: > > > > > > > > > Looks like that if I move files between different data pools of > the > > > cephfs, something is still refering to the 'old location' and > gives > > > an > > > > > Input/output error. I assume this, because I am using different > > > > client > > > > > ids for authentication. > > > > > > With the same user as configured in ganesha, mounting (kernel) > > > erasure > > > > > code cephfs m can create file out4 > > > > > > At nfs4 client, same location m > > > I can read out4 > > > I can create out5 > > > I can read out5 > > > > > > Mounted root cephfs create file in folder t (test replicated 1) > I > > > can create out6 I can move out6 to a the folder m (erasure > coded) I > > > can read out6 > > > > > > At nfs4 client, m location > > > [@m]# cat out6 > > > cat: out6: Input/output error > > > > If it was due to permissions, I would expect to see EPERM rather > than > > EIO. EIO suggests something more fundamentally broken, like a > client > > version that doesn't understand the latest layout format. > > > > Assuming you're using the CephFS FSAL in Ganesha (rather than > > re-exporting a local mount of CephFS), it should be possible to > create > > > an /etc/ceph/ceph.conf file with a "[client]" section that > enables > > debug logging (debug client = 10 or similar), and sets an output > > location ("log file = /tmp/client.log") -- that might give a bit > more > > information about the nature of the error. > > > > John > > > > > > > > > > > > > > [client.cephfs.t] > > > key = xxx== > > > caps mds = "allow rw path=/t" > > > caps mgr = "allow r" > > > caps mon = "allow r" > > > caps osd = "allow rwx pool=fs_meta,allow rwx pool=fs_data, > > > > allow > > > > > rwx pool=fs_data.r1" > > > > > > [client.cephfs.m] > > > key = xxx== > > > caps mds = "allow rw path=/m" > > > caps mgr = "allow r" > > > caps mon = "allow r" > > > caps osd = "allow rwx pool=fs_meta,allow rwx > pool=fs_data.ec" > > > > > > > > > [@ test]# cat /etc/redhat-release > > > CentOS Linux release 7.5.1804 (Core) > > > > > > [@ test]# rpm -qa | grep ceph | sort > > > ceph-12.2.8-0.el7.x86_64 > > > ceph-base-12.2.8-0.el7.x86_64 > > > ceph-common-12.2.8-0.el7.x86_64 > > > ceph-fuse-12.2.8-0.el7.x86_64 > > > ceph-mds-12.2.8-0.el7.x86_64 > > > ceph-mgr-12.2.8-0.el7.x86_64 > > > ceph-mon-12.2.8-0.el7.x86_64 > > > ceph-osd-12.2.8-0.el7.x86_64 > > > ceph-radosgw-12.2.8-0.el7.x86_64 > > > ceph-selinux-12.2.8-0.el7.x86_64 > > > collectd-ceph-5.8.0-2.el7.x86_64 > > > libcephfs2-12.2.8-0.el7.x86_64 > > > python-cephfs-12.2.8-0.el7.x86_64 > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |