Re: WebDAV folders internally have hundreds of copies of the same few files

Anatoli <me@xxxxxxxxxx> · Tue, 20 Nov 2018 20:23:43 -0300



      > What
        are you using as the mailbox name?

        
        #drive/shared/test2

        
        Ken, do you know why cyr_expire -u
        and -p don't work on shared DAV
        mailboxes? E.g. cyr_expire -X 3 -u "#drive/shared/test2" doesn't clean
        the folder and with -v shows 0
        for everything, then unexpunge -l "#drive/shared/test2"
        shows a lot of items with Expg
        date older that 3 days, but a general cyr_expire
          -X 3 (i.e. without the -u
        or -p) does clean the entire
        store, including the shared DAV folders.

        
      From:
        Ken Murchison

        Sent: Tuesday, November 20, 2018 10:22

        To: Info-cyrus

        Subject: Re: WebDAV folders internally have hundreds of
        copies of the same few files

      
    What are you using as the mailbox name?  

    
    On 11/20/18 1:01 AM, Anatoli wrote:

    
        >
          Enable the imapmagicplus option in imapd.conf

          
          With this option I could login with imtest specifying "+DAV"
          for the user, both as the admin and as a regular user, but
          then when trying to select a folder, I got "NO Permission
          denied" for the admin and "NO Mailbox does not exist" for a
          common user. This looks like undocumented magic and mbexamine actually provides all
          needed information, so I will get file details with it.

          
          Thanks for you help, Ken.

          
        From:
          Ken Murchison

          Sent: Friday, November 16, 2018 13:17

          To: Info-cyrus

          Subject: Re: WebDAV folders internally have hundreds of
          copies of the same few files

        
      On 11/16/18 1:19 AM, Anatoli wrote:

      
          Ken,

            
            > If you login as an admin, you should be able to SELECT
            the mailbox and use normal IMAP commands.  Alternatively, if
            you add "+DAV" to a regular userid, this will also allow
            SELECTing of DAV collection mailboxes.

            
            When I do imtest -a mailadmin
            and then issue a002 select
              #drive/folder, I get:

            
              a002 NO Invalid mailbox type

            
      It appears that just logging in as an admin won't do the trick.
      

            When I do imtest -a mailadmin+DAV,
            I get (XX replaces the real
            auth value):

            
            C: A01 AUTHENTICATE PLAIN
              XXXXXXXXXXXXXXXXXXX

              S: A01 NO authentication failure

              Authentication failed. generic failure

            
            I've also tried these combinations with the same result: "user@xxxxxxxxxx+DAV"
            and "user+DAV@xxxxxxxxxx".
            Am I doing it wrong?

          
      Enable the imapmagicplus option in imapd.conf
      

            Independently of this, is there a way to obtain the details
            about the flags for each message in a folder directly from
            the db files? My idea is to rsync the Cyrus store where the
            WebDAV is located to some other folder (probably on another
            server), purge there the files marked as deleted and backup
            the remaining files, all this without altering the Cyrus
            store. Is it possible?

            
            > If the append of the resource into the mailbox, or
            updating the DAV db entry fails, the operation should be
            reverted, with partial saving done.  Which version of Cyrus
            are you using?

            
            Currently it is 3.0.5, but I can upgrade if there were any
            related changes in the later versions. What I was observing
            is that if a PC (LibreOffice Calc on Ubuntu 16.04, the
            folder mounted with davfs2) crashes in the middle of saving
            a file on the server, the result was a damaged file: some
            400 bytes instead of 3-4Mb of a real file.

            
            I was thinking that maybe when the TCP stream hangs in the
            middle, Cyrus would interpret it as an end of data and write
            a file, but I've just tried to make some tests killing mount.davfs2 process and/or
            blocking the stream with iptables in the middle of file
            transmission and Cyrus responded correctly. For killing mount.davfs2 it showed in the
            corresponding log file the bytestream followed by:

            
            HTTP/1.1 400 Bad Request

              
              Unable to read body data

            
            And for blocking the stream with iptables it was showing the
            bytestream ending where the transmission stopped, without
            the 400 Bad Request, but in both cases the original file was
            not modified by the partial upload. So it appears the
            problem is elsewhere.

            
            Regards,

            Anatoli

            
          From:
            Ken Murchison

            Sent: Thursday, November 15, 2018 14:09

            To: Info-cyrus

            Subject: Re: WebDAV folders internally have hundreds
            of copies of the same few files

          
        On 11/15/18 1:48 AM, Anatoli wrote:

        
            Hi
              Ken,

              
              Thanks a lot for the clarification, everything makes sense
              now. How can I list the files marked for deletion and
              those that are not marked? unexpunge
              can provide the list of files marked for deletion, is
              there any better way to list them, directly reading the
              DB? How to list those that are not marked?

            
        If you login as an admin, you should be able to SELECT the
          mailbox and use normal IMAP commands.  Alternatively, if you
          add "+DAV" to a regular userid, this will also allow SELECTing
          of DAV collection mailboxes.
        

             One
              more question, related: we got a client who's PC was
              crashing exactly during the modify operation (some issue
              with the PC hardware triggered by Excel save operation,
              probably a RAM spike touching some bad blocks). As a
              result, the file in Cyrus was becoming damaged, i.e.
              partially saved. Is it expected?

              
              Shouldn't Cyrus update the db with the pointer to the new
              file (a new message in the store) only if the operation
              completes successfully (e.g. the WebDAV messages exchange
              completes and the connection is closed at the right time
              or something similar)?

            
        If the append of the resource into the mailbox, or updating
          the DAV db entry fails, the operation should be reverted, with
          partial saving done.  Which version of Cyrus are you using?
        

            From:
              Ken Murchison

              Sent: Wednesday, November 14, 2018 10:54

              To: Info-cyrus

              Subject: Re: WebDAV folders internally have
              hundreds of copies of the same few files

            
          On 11/13/18 10:15 PM, Anatoli
            wrote:

          
                Hi,

                
                I'm not sure this is due to some configuration option,
                bug or feature, but I'm observing some folders on Cyrus
                HTTP WebDAV server having hundreds (995 at this moment
                to be precise) internal files in the format "NNN." that
                correspond to the same file but different versions in
                time.

                
                There are 2-3 files (xls) in the folder that are edited
                constantly during the day and it looks like each save
                operation creates a new file. The files are of some
                3-5Mb each. In the explorer/web view there are only a
                couple of files with a total size of 17.5Mb, but the
                reported disk usage for the folder is of 1.6Gb.

                
                Could someone please shed some light on what's going on
                and how to make each file visible to the users to be
                stored in only one internal file?

                
                Thanks,

                Anatoli
            
          
          Because *DAV is layered on top of an IMAP store, we have to
            abide by IMAP semantics in which messages (in this case DAV
            resources) are immutable.  Therefore, we can NOT overwrite
            an existing message in the mailbox.  Each change MUST result
            in a new message.  However, the server does mark the
            previous version(s) as deleted and expunged, which means
            that they will eventually be removed by cyr_expire.  If you
            aren't running cyr_expire, you should consider adding an
            event to cyrus.conf to remove expunged messages (see -X
            option).

          
          -- 
Ken Murchison
Cyrus Development Team
FastMail US LLC
          

          ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
          

          ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
        
        -- 
Ken Murchison
Cyrus Development Team
FastMail US LLC
        

        ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
        

        ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
      
      -- 
Ken Murchison
Cyrus Development Team
FastMail US LLC
      

      ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
      

      ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
    
    -- 
Ken Murchison
Cyrus Development Team
FastMail US LLC
    

    ----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
    

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus