Hi Carlos,
Sudden overnight intuition, and I think this will fix the issue -- the problem isn't that the lseek is failing at >2GB; the problem is that its off_t return value is being truncated to an int before being checked to see if it's negative for the error case (so any "I succeeded, the offset is > 2GB" response looks like "negative, it's an error"). Doh!
I think this will fix the immediate issue:
But then I found and fixed some similar ones:
Remains to be seen whether more similar issues pop up -- since no-one's tripped over this previously, I guess you're the first person to try this with a >2GB backup file :(
Cheers,
ellie
On Fri, Jun 14, 2019, at 3:50 PM, ellie timoney wrote:
Hi Carlos,This is quite weird, I'm not sure why a 64bit platform would have any trouble around the 2GB mark??What does the Cyrus ./configure report for your system's integer sizes? e.g. mine shows:checking size of int... 4checking size of long... 8checking size of size_t... 8checking size of off_t... 8checking size of time_t... 8checking size of long long int... 8checking size of unsigned long long int... 8checking whether byte ordering is bigendian... noWhat's your level of comfort with C debugging? It'd be very helpful to see a core file+binary from the time that lseek error occurs?Cheers,ellieOn Fri, Jun 7, 2019, at 1:58 AM, Carlos Larrañaga wrote:Hi Ellie,
Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I said before, sorry) and zlib is also 64bit version:# uname -a
Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux
# yum list installed zlib
Loaded plugins: langpacks, ulninfo
Installed Packages
zlib.x86_64 1.2.7-18.el7 @ol7_latest
# lsof -p 17005 |grep lib |grep -i z
backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7
# file /usr/lib64/libz.so.1.2.7
/usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped
# rpm -qf /usr/lib64/libz.so.1.2.7
zlib-1.2.7-18.el7.x86_64I have summarized here some information about the backupd error reading the file descriptor 15, which is a backup larger than 2GB. The error is logged 1755 times. Instead, there is no error for fd 12, which is a backup of less than 2 GB:#-- CLIENT SIDE-----------------------------------------------# time sync_client -v -o -n backup -AThu Jun 6 17:08:02 CEST 2019USER aaaQUOTA user.aaaUSER cccError from do_user(ccc): bailing out!real 30m16.223suser 0m0.006ssys 0m0.006s#-- BACKUP SERVER SIDE ----------------------------------------# LOGFILE# tail -f /var/log/imapd.logJun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directoryJun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directoryJun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directoryJun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory# lsof of the backupd PROCESS
# lsof -P -p 2584
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
backupd 2584 cyrus cwd DIR 253,0 4096 128 /
backupd 2584 cyrus rtd DIR 253,0 4096 128 /
backupd 2584 cyrus txt REG 253,0 768632 939542214 /usr/local/cyrus/libexec/backupd
backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2
backupd 2584 cyrus mem REG 253,0 31416 134406707 /usr/lib64/libnss_dns-2.17.so
backupd 2584 cyrus mem REG 253,0 61632 134406709 /usr/lib64/libnss_files-2.17.so
backupd 2584 cyrus mem REG 253,0 43336 68814977 /usr/lib64/sasl2/webmail.so.0.0.0
backupd 2584 cyrus mem REG 253,0 57960 68856627 /usr/lib64/sasl2/libdigestmd5.so.3.0.0
backupd 2584 cyrus mem REG 253,0 24232 68797220 /usr/lib64/sasl2/libcrammd5.so.3.0.0
backupd 2584 cyrus mem REG 253,0 20088 68797044 /usr/lib64/sasl2/libplain.so.3.0.0
backupd 2584 cyrus mem REG 253,0 20056 68797036 /usr/lib64/sasl2/liblogin.so.3.0.0
backupd 2584 cyrus mem REG 253,0 1845816 135873512 /usr/lib64/libdb-5.3.so
backupd 2584 cyrus mem REG 253,0 28272 67113426 /usr/lib64/sasl2/libsasldb.so.3.0.0
backupd 2584 cyrus mem REG 253,0 20064 67113423 /usr/lib64/sasl2/libanonymous.so.3.0.0
backupd 2584 cyrus mem REG 253,0 11448 135872733 /usr/lib64/libfreebl3.so
backupd 2584 cyrus mem REG 253,0 155784 134400716 /usr/lib64/libselinux.so.1
backupd 2584 cyrus mem REG 253,0 15464 134401447 /usr/lib64/libkeyutils.so.1.5
backupd 2584 cyrus mem REG 253,0 40672 134399432 /usr/lib64/libcrypt-2.17.so
backupd 2584 cyrus mem REG 253,0 105824 134781819 /usr/lib64/libresolv-2.17.so
backupd 2584 cyrus mem REG 253,0 53944 134406785 /usr/lib64/libjansson.so.4.10.0
backupd 2584 cyrus mem REG 253,0 88776 135352942 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
backupd 2584 cyrus mem REG 253,0 1137032 134400337 /usr/lib64/libm-2.17.so
backupd 2584 cyrus mem REG 253,0 991616 135352396 /usr/lib64/libstdc++.so.6.0.19
backupd 2584 cyrus mem REG 253,0 19296 134399434 /usr/lib64/libdl-2.17.so
backupd 2584 cyrus mem REG 253,0 142008 134722750 /usr/lib64/libpthread-2.17.so
backupd 2584 cyrus mem REG 253,0 2151704 134365659 /usr/lib64/libc-2.17.so
backupd 2584 cyrus mem REG 253,0 753232 134401161 /usr/lib64/libsqlite3.so.0.8.6
backupd 2584 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7
backupd 2584 cyrus mem REG 253,0 11128 134400947 /usr/lib64/libpcreposix.so.0.0.1
backupd 2584 cyrus mem REG 253,0 402384 134400940 /usr/lib64/libpcre.so.1.2.0
backupd 2584 cyrus mem REG 253,0 67104 135269427 /usr/lib64/libkrb5support.so.0.1
backupd 2584 cyrus mem REG 253,0 15920 135873508 /usr/lib64/libcom_err.so.2.1
backupd 2584 cyrus mem REG 253,0 210832 134781960 /usr/lib64/libk5crypto.so.3.1
backupd 2584 cyrus mem REG 253,0 967864 134781967 /usr/lib64/libkrb5.so.3.3
backupd 2584 cyrus mem REG 253,0 320400 134781952 /usr/lib64/libgssapi_krb5.so.2.2
backupd 2584 cyrus mem REG 253,0 115856 134400341 /usr/lib64/libnsl-2.17.so
backupd 2584 cyrus mem REG 253,0 42168 134401419 /usr/lib64/libwrap.so.0.7.6
backupd 2584 cyrus mem REG 253,0 2516640 134400708 /usr/lib64/libcrypto.so.1.0.2k
backupd 2584 cyrus mem REG 253,0 470360 134400711 /usr/lib64/libssl.so.1.0.2k
backupd 2584 cyrus mem REG 253,0 121208 134401258 /usr/lib64/libsasl2.so.3.0.0
backupd 2584 cyrus mem REG 253,0 2898664 838888996 /usr/local/cyrus/lib/libcyrus_imap.so.0.0.0
backupd 2584 cyrus mem REG 253,0 599528 838888999 /usr/local/cyrus/lib/libcyrus_sieve.so.0.0.0
backupd 2584 cyrus mem REG 253,0 20112 134406704 /usr/lib64/libuuid.so.1.3.0
backupd 2584 cyrus mem REG 253,0 20781704 136182553 /usr/lib64/libicudata.so.50.1.2
backupd 2584 cyrus mem REG 253,0 1539544 136197152 /usr/lib64/libicuuc.so.50.1.2
backupd 2584 cyrus mem REG 253,0 553264 838890012 /usr/local/cyrus/lib/libcyrus_min.so.0.0.0
backupd 2584 cyrus mem REG 253,0 1961968 838888994 /usr/local/cyrus/lib/libcyrus.so.0.0.0
backupd 2584 cyrus mem REG 253,0 163408 134355369 /usr/lib64/ld-2.17.so
backupd 2584 cyrus 0u IPv4 2092952 0t0 TCP hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
backupd 2584 cyrus 1u IPv4 2092952 0t0 TCP hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
backupd 2584 cyrus 2u IPv4 2092952 0t0 TCP hostb1.upv.es:2005->host1.upv.es:27107 (ESTABLISHED)
backupd 2584 cyrus 3w FIFO 0,9 0t0 2141764 pipe
backupd 2584 cyrus 4u IPv4 2141763 0t0 TCP hostb1.upv.es:2005 (LISTEN)
backupd 2584 cyrus 5u REG 0,20 0 86644 /run/cyrus-b1/socket/backupd-0.lock
backupd 2584 cyrus 6r FIFO 0,9 0t0 2141754 pipe
backupd 2584 cyrus 7w FIFO 0,9 0t0 2141754 pipe
backupd 2584 cyrus 8r FIFO 0,9 0t0 2141755 pipe
backupd 2584 cyrus 9w FIFO 0,9 0t0 2141755 pipe
backupd 2584 cyrus 10u unix 0xffff9d9df53d4400 0t0 2093417 socket
backupd 2584 cyrus 11w REG 253,5 680 18253611600 /b1/lib/log/admin/backupd-2584
backupd 2584 cyrus 12uW REG 253,5 976281587 13958644353 /b1/bck/b1/a/aaa_ouuUx9
backupd 2584 cyrus 13ur REG 253,5 3573760 13958644354 /b1/bck/b1/a/aaa_ouuUx9.index
backupd 2584 cyrus 14u REG 253,5 2576 13958644355 /b1/bck/b1/a/aaa_ouuUx9.index-journal
backupd 2584 cyrus 15uW REG 253,5 3104341527 12884902595 /b1/bck/b1/c/ccc_Mqmymx
backupd 2584 cyrus 16u REG 253,5 47927296 12884902596 /b1/bck/b1/c/ccc_Mqmymx.index
# BACKUP SIZES
# ls -lh /b1/bck/b1/a/*/b1/bck/b1/c/*-rw------- 1 cyrus mail 932M Jun 6 17:08/b1/bck/b1/a/aaa_ouuUx9-rw------- 1 cyrus mail 3.5M Jun 6 16:46/b1/bck/b1/a/aaa_ouuUx9.index-rw------- 1 cyrus mail 2.6K Jun 6 17:08/b1/bck/b1/a/aaa_ouuUx9.index-journal
-rw------- 1 cyrus mail 2.9G Jun 6 17:01 /b1/bck/b1/c/ccc_Mqmymx
-rw------- 1 cyrus mail 46M Jun 6 17:01 /b1/bck/b1/c/ccc_Mqmymx.index
# IMAP SESSION DEBUG
# cat /b1/lib/log/admin/backupd-2584
---------- admin Thu Jun 6 17:08:02 2019
<1559833682<COMPRESS DEFLATE
>1559833682>OK DEFLATE active
<1559833682<GET USER aaa
>1559833698>* MAILBOX %(UNIQUEID acd014aa-7b1d-43c9-b1d3-f97b06540739 MBOXNAME user.aaa MBOXTYPE NIL LAST_UID 0 HIGHESTMODSEQ 2 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1520512152 PARTITION default ACL "aaa lrswipkxtecdan group:super lrswipkxtecdan " OPTIONS P SYNC_CRC 0 SYNC_CRC_ANNOT 0 QUOTAROOT NIL XCONVMODSEQ 0)
OK Success
<1559833698<APPLY QUOTA %(ROOT user.aaa LIMIT 5120000 STORAGE 5120000)
>1559833698>OK Success
<1559833698<GET USER cccBest regards,CarlosEl 06/06/2019 a las 4:04, ellie timoney escribió:It kinda sounds like your platform might be 32bit? Or your zlib is compiled to use 32bit integer sizes?On Mon, Jun 3, 2019, at 10:07 PM, Carlos Larrañaga wrote:Hi,
We're testing backup feature un cyrus-imapd 3.0.10. There's no problem when backup is created first time, but when the backup already exists and is larger than 2GB, we get the following error from the backupd:cyrus/backupd[xxxx]: IOERROR: gzuc_read: lseek 12: No such file or directoryAs said, it happens only when then backup file already exists and is larger than 2GB. The backupd process keeps reading de compressed backup file til sync_client exit with "Error from sync_do_user(xxx): bailing out!".
We use xfs with latest CentOs. Seems like backupd is unable to read the compressed backup file.
Anyone else with this problem? Any idea how to fix it?
Thanks in advance for your help.
Best regards,
Carlos.----Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/To Unsubscribe:Attachments:
- smime.p7s
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrusAttachments:
- smime.p7s
----Cyrus Home Page: http://www.cyrusimap.org/List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/To Unsubscribe:https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
---- Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
