Brian Lim wrote:
>
> Hi,
>
> could someone tell me the way to handle large
> file (> 2GB) on linux ?
>
Attached is a short writeup I did on using large files several months
ago. I assume it is still accurate.
Bruce
--
Bruce Blinn 408-615-9100
Mission Critical Linux, Inc. blinn@MissionCriticalLinux.com
www.MissionCriticalLinux.com
Never teach a pig to sing,
it makes you look stupid
and it annoys the pig.
Using Large Files on Linux
Large files are files that are larger than 2 gigabytes. This distinction
comes from the largest number that can be represented in the off_t type, which
is used to specify file offsets in system and library calls. On 32-bit
architectures (such as Intel), an off_t is commonly 32 bits. To avoid
conflicts with negative numbers, the largest offset that can be represented in
a 32-bit off_t variable is 2 gigabytes.
Applications compiled on 64-bit architectures (such as Intel IA64) can access
large files naturally; they do not require any changes. However, applications
compiled for 32-bit architectures, by default, can only access the first 2
gigabytes of a file even if the kernel has support for large files.
This paper describes the steps necessary to access large files on 32-bit
architectures. Once an application is modified to support large files, it will
be able to access any file regardless of size.
Dependencies
The Linux kernel does not support large files prior to version 2.4.x.
An application compiled for large files will run on earlier version of
the Linux kernel, but if you try to access beyond the 2 gigabyte
boundary, you will get an error.
Version 2.1.3 of the C library (/lib/libc-2.1.3.so), which comes with
RedHat 6.2, seems to have most of the large file support needed.
Notably missing (or broken), however, is the fcntl(2) system call,
which fails when trying to lock byte ranges that are beyond the 2
gigabyte boundary of a large file. This problem is corrected in
version 2.2.2 of the C library, which comes with RedHat 7.1.
If you are going to access a large file over NFS, you must use NFS
version 3, and the kernel on both the client and server must be 2.4.x
or later.
Applications Changes
Simplistically, the only change you need to make to your application in
order to use large files is to add the following definition. This
definition must come before any header files are included.
#define _FILE_OFFSET_BITS 64
This will cause the size of the off_t type to be 64 bits instead of 32.
In addition, calls to procedures in libc that have parameters of type
off_t will be redirected to their 64 bit counterparts. For example,
calls to read(2) will be linked to read64(2), calls to lseek(2)
will be linked to lseek64(2), and so on.
It is probably a good precaution to compile with -Wall so you will see
any type conflicts that may have arisen. In particular, if you use
printf(3) or one of its cousins to print a file offset, it will no
longer be correct. To format a 64 bit integer in printf(3) use %lld
or %llx, in printk() use %Ld or %Lx (the L format qualifier does not
work until Linux 2.4.x).
Also, there is a new signal, SIGXFSZ, that can be safely ignored. If
this signal is ignored, the failing system call will return a useful
error code in its place.
signal(SIGXFSZ, SIG_IGN);
References
There are several programs in the MCLX NFS test suite (chklfs,
readfile, and writefile) that exercise large files. These are simple
programs, and therefore, they may be useful as examples.
The following documents are arguably a representative and useful subset
of the billions of documents written on large file support.
Large File Support in Linux http://www.suse.de/~aj/linux_lfs.html
Large File Support in Linux 2.2 http://www.scyld.com/products/beowulf/software/lfs.html
Large File Support in SuSE 7.0 http://sdb.suse.de/en/sdb/html/lfs70.html
Large File HowTo http://www.beowulf.org/pipermail/beowulf/2000-March/008708.html
SIGXFSZ Discussion http://ftp.sas.com/standards/large.file/hmail.compat/0136.html
Large File Summit Report http://ewe3.sas.com/standards/large.file/x_open.20Mar96.html
Data Size Neutrality http://www.UNIX-systems.org/version2/whatsnew/login_64bit.html
Large Files White Paper (HP-UX) http://www.docs.hp.com/hpux/onlinedocs/os/lgfiles4.pdf
