Thanks - there's a lot of uploaders out there... I'll look at it more, but I'd really like a solution which allows for us to use any applet language we want. We'd prefer not to use java too :) I don't see anything specific right off the bat that says how it uploads (POST vs PUT) or splitting up of files. I would like to design something that can work without any changes to server settings - so anyone can use it... On 8/26/08, Simcha Younger <simcha.younger@xxxxxxxxx> wrote: > > Is this what you are looking for -- It's a java applet that has most of the > features you mentioned. I have used it for very large files with no problem. > > http://www.javazoom.net/applets/jclientupload/jclientupload.html > > (Sorry, its not free software.) > > -----Original Message----- > From: mike [mailto:mike503@xxxxxxxxx] > Sent: Tuesday, August 26, 2008 10:07 AM > To: PHP General list > Subject: Large/unreliable file uploading over HTTP > > Let's face it - HTTP is not very good for file uploads. It's stateless > nature, slow connections, inability to resume (technically), etc, etc. > > What I've been thinking about is a way to skip all the normal > annoyances with file uploading - multipart form encodings, file upload > tools with specific needs, PUT vs POST, connection resets, ... the > list goes on and on. > > Usenet and BitTorrent and other protocols have the right idea - split > up the workload into smaller sets of data. It's easier to operate on. > Usenet has NZB files. BitTorrent splits everything up into chunks. Not > only would it make working with the data more portable (no need to set > your PHP memory limit or POST limits to insane amounts to support > large files) but it could also support multiple segments of the file > being transferred at once... > > Here's somewhat of a process braindump of what I'm thinking. It still > requires a 'smart' applet (Flash, Java, anything that can split a file > up and send data over HTTP/HTTPS) - no special socket needs, no PUT > support needed, don't even need to use multipart POST encoding (as far > as I know) - just send the data in chunks over the wire and have a PHP > script on the other side collect the data and reassemble it. > > Does this sound insane? I think this is a pretty good approach - no > PUT needed, no large POST configuration required, anything could > upload to it as long as it sends the information properly (I'm > thinking HTTP POST for the header info, and the data could be sent as > another POST field maybe base64 encoded or something that will stay > safe during transit...) > > > > - take input file, checksum it (md5 / sha1) > - calculate number of chunks to split it up based on $chunk configured > size (for example 128k chunks) > - split the file into chunks of $chunk size and create checksums for > all (could use sfv?) > - send request to the server - with the info - use JSON? > action=begin > filename=$filename > filesize=$filesize > checksum=$checksum > segments=list of segments (unique segment id, checksum, bytes) > - server sends back a "server ready" and unique $transaction_id > - start sending to the server, send header with transaction key and > unique chunk identifier in it > action=process > transaction=$transaction_id > segment=$unique_segment_id > data=base64_encode($segment_data) > - when done, server sends back $transaction_id, $segment_id, $status > (okay, fail) > - client compares checksum for identifier, if okay, move to next chunk > - if it does not match, retry uploading to server again > - when all chunks are done, send request with transaction key and > action=finish > transaction=$transaction_id > - when the server receives this, it assembles the file > from the segments and does a final checksum, and reports the checksum > back to the client (warning: on a large file this could take a bit?) > and sends back $transaction_id, $checksum > - client does one last check against the file's original checksum, if > it matches report success, otherwise report failure (would need to > determine why though - if all segments match this should not be able > to happen...) > > I'd appreciate everyone's thoughts. This would also allow for file > upload progress, more or less, as the server and client are constantly > communicating when chunks are done and in progress (but again, that > has to be done with an applet) > > I can't think of any method to do it in-browser, but doing it this way > could open the gates for things like Google Gears to possibly work > too... > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > No virus found in this incoming message. > Checked by AVG - http://www.avg.com > Version: 8.0.138 / Virus Database: 270.6.7/1632 - Release Date: 25/08/2008 > 07:05 > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
