On Wed, 17 Mar 2010 05:24:38 +0200, Henrik K <hege@xxxxxxx> wrote: > On Tue, Mar 16, 2010 at 08:58:27PM +0100, Henrik Nordström wrote: >> mån 2010-03-15 klockan 18:47 +0200 skrev Henrik K: >> >> > If you don't want this limitation, you can use HAVP. It scans the file >> > while >> > it's being transferred to client, while keeping small part of it >> > buffered >> > (in case of virus, it is not transferred so client can't open >> > incomplete >> > file). It's as close to transparent as you can get. >> >> That's also one of the three modes supported by c-icap clamav service. > > Such comment can only be made when one doesn't understand what HAVP does. > It > is NOT the same thing. > > http://www.server-side.de/documentation.htm > > While one can speculate about the usefulness of scanning huge files at HTTP > level, HAVP with mandatory locking does it much more efficiently. > > C-icap will only call the scanner after a file is completely received, > resulting in additional wait and a load spike. > > HAVP starts scanning the file immediately as it is received from the server > and gradually unlocked. When c-icap has just started scanning the file, > HAVP > has already scanned most (if not all) of it and is sending final bytes to > client. If a virus had happened to be found, HAVP would have already > stopped > the unnecessary download without wasting time on the whole file. This also > works on ZIP files as it first tries to download the header at end of the > file using Range request. So HAVP is designed specifically to send client scanned parts of the file before the entire thing is checked? That explains something that I was wondering about... Consider this scenario which I have seen in the wild: Background: Clients visit website A and fetch a large document PDF file. Unknown to the website author the server has been infected and PDF is one of the files which get a macro virus appended. The CRM system records in a database the upload and change times for efficient if-modified responses. The server admin is alerted and runs a virus scan, cleaning the files some time later. The CRM database gets omitted from the update. Imagining that HAVP was in use in a proxy between this site and a user... During the infected period imaginary-HAVP scans the documents and sends a large "clean" prefix to all visitors. BUT... aborts when the appended infection is detected. Browser is lucky enough to notice the file is incomplete and retires later with a range request for the missing bit. a) during the infected period the fetched ranges will never succeed. b) after the infection is cleaned up the file will pass through imaginary-HAVP and client will get a truncated version. With complete-file being indicated. This is where the problem comes in. Being a macro infection one of the changes to the file was that the virus appended some undetectable jump code at the beginning to go with the virus at the end. We are left with the situation where intermediary proxies are holding corrupted files (first part being original infected with jump, followed by terminal bytes of teh file. the server is left with a pristine and working file. New visitors loading it will be fine, and sill analysts coming along later. However for clients visiting through one of the proxies which cached the file meanwhile ... One of two things will happen to depending on the file viewer used: 1) dumb viewer will try to run the random part of file (now text!) where virus inserted itself as binary code and crash. 2) smart viewer will notice the missing/corrupt macro (its past the end of file maybe) and display the file without running it. However, even then there is a discrepancy in file prefix and some of the content appears corrupted. This type of traffic is the #1 reason for buffering until fully processed. I do like the idea of incremental scanning as it arrives though. That will at least reduce the delays to very little more than the total receiving time. Amos
