On 16/03/10 00:48, Shridhar Daithankar wrote:
[...]
But as far as file system performance goes, the overhead should be identical
for both the runs, no?
I'm not too sure about that. I'm guessing there is less seeking going on
with Btrfs. Some files systems (reiserfs + reiserfs4 IIRC) are very good
with many small files, better than the ext*fs, this may be another case
of that.
Besides, I need to run the comparison(rather verification of file contents)
many times over during the application life-cycle and I cannot afford to bring
in another copy from disk. The working set is expected to be 30-40GB at a
time, 3GB is just test setup.
With md5sum, I can store it in database and verify it on one copy only.
Fair enough.
And finally, it is terrible on timings. Running md5sum is lot faster, about 3
times in the best case.
[...]
wow, that's slow!
So when the source file system is btrfs, it is still couple of times faster at
least.
I still think you could achieve better times by not calling the external
command that many times.
Since you're already gonna store the checksums in a database, I'd just
write a proper program in python or something.
Or even just a shellscript, but you might wanna refrain from for .. in
`find .. , it's the slowest and that relies on the fact that your
filenames don't have spaces in them.
[[ky] ~]# }} time find /usr/bin -type f -print0 | xargs -0 md5sum > /tmp/1
real 0m3.633s
[[ky] ~]# }} time find /usr/bin -type f -exec md5sum "{}" \; > /tmp/2
real 0m10.196s
[[ky] ~]# }} time for i in `find /usr/bin -type f`;do md5sum "$i";done >
/tmp/3
real 0m11.245s
this last version missed a file because it has spaces in its name and as
result the file 3 was inconsistent with files 1 and 2
[[ky] ~]# }} diff /tmp/{1,2}
[[ky] ~]# }} diff /tmp/{3,2}
3054a3055
> 0c5d8f10aa0731671a00961f059dc46e /usr/bin/New SMB and DCERPC
features in Impacket.pdf
that was a test against just 4008, so you can imagine time savings with
50000+ files.
