On Tue, Jul 21, 2009 at 9:38 AM, Eric Sandeen<sandeen@xxxxxxxxxx> wrote:
> Curt Wohlgemuth wrote:
>> On Mon, Jul 20, 2009 at 8:41 PM, Eric Sandeen<sandeen@xxxxxxxxxx> wrote:
>>> Xiang Wang wrote:
>
>>>> For comparison, I did the same experiment on an ext2 partition,
>>>> resulting in each file having only 1 extent.
>>> Interestinng, not sure I would have expected that.
>>
>> Same with us; we're looking into more variables to understand it.
>
> To be more clear, I would not have expected ext2 to deal well with it
> either, is more what I meant ;) I'm not terribly surprised that ext4
> gets fragmented.
>
> For the numbers posted, how big were the files (how many 1m chunks were
> written?)
>
> Just FWIW; I did something like:
>
> # for I in `seq 1 16`; do dd if=/dev/zero of=testfile$I bs=1M count=16
> oflag=direct & done
>
> on a rhel5.4 beta kernel and got:
>
> ~5 extents per file on ext4 (per filefrag output)
> between 41 and 234 extents on ext2.
> ~6 extents per file on ext3.
> ~16 extents per file on xfs
>
I repeated this test(bs=1M count=16) by tuning some parameters in my
test program. And I got the following results(per filefrag output):
ext4:
5 extents per file
ext2:
file0: 5 extents found, perfection would be 1 extent
file1: 5 extents found, perfection would be 1 extent
file2: 6 extents found, perfection would be 1 extent
file3: 4 extents found, perfection would be 1 extent
file4: 4 extents found, perfection would be 1 extent
file5: 6 extents found, perfection would be 1 extent
file6: 4 extents found, perfection would be 1 extent
file7: 5 extents found, perfection would be 1 extent
file8: 6 extents found, perfection would be 1 extent
file9: 4 extents found, perfection would be 1 extent
file10: 5 extents found, perfection would be 1 extent
file11: 6 extents found, perfection would be 1 extent
file12: 6 extents found, perfection would be 1 extent
file13: 8 extents found, perfection would be 1 extent
file14: 4 extents found, perfection would be 1 extent
file15: 7 extents found, perfection would be 1 extent
The results on ext4 look comparable to yours while the results on ext2
look very different.
I am attaching the test program I use in case you want to try it. It
is at the end of the message.
I invoked it like: ./mt_writes 16 1 to have 16 threads writing using O_DIRECT.
> if I created a subdir for each file:
>
> # for I in `seq 1 16`; do mkdir dir$I; dd if=/dev/zero
> of=dir$I/testfile$I bs=1M count=16 oflag=direct & done
>
> ~5 extents per file on ext4
> 1 or 2 extents per file on ext2
> 1 or 2 extents per file on ext3
> ~16 extents per file on xfs.
>
> -Eric
>
======
/*
* mt_write.c -- multiple threads extending files concurrently.
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <pthread.h>
#include <sys/stat.h>
#include <fcntl.h>
#define _XOPEN_SOURCE 600
#define O_DIRECT 00040000 /* direct disk access hint */
#define MAX_THREAD 1000
#define BUFSIZE 1048576
#define COUNT 16
typedef struct {
int id;
int odirect;
} parm;
void *expand(void *arg)
{
char *buf;
char fname[16];
int fd;
int i, count;
parm *p = (parm *)arg;
// O_DIRECT needs to work with aligned memory
if (posix_memalign((void *) &buf, 512, BUFSIZE) != 0) {
fprintf(stderr, "cannot allocate aligned mem!\n");
return NULL;
}
sprintf(fname, "file%d", p->id);
if (p->odirect)
fd = open(fname, O_RDWR|O_CREAT|O_APPEND|O_DIRECT);
else
fd = open(fname, O_RDWR|O_CREAT|O_APPEND);
if (fd == -1) {
fprintf(stderr, "Open %s failed!\n", fname);
return NULL;
}
for(i = 0; i < COUNT; i++) {
count = write(fd, buf, BUFSIZE);
if (count == -1) {
fprintf(stderr, "Only able to finish %d blocks
of data\n", i);
return NULL;
}
}
if (!p->odirect) {
fsync(fd);
}
printf("Done with writing %d blocks of data\n", COUNT);
close(fd);
free(buf);
return NULL;
}
int main(int argc, char* argv[]) {
int n,i, odirect;
pthread_t *threads;
pthread_attr_t pthread_custom_attr;
parm *p;
if (argc != 3)
{
printf ("Usage: %s <# of threads> <O_DIRECT? 1:0>\n",argv[0]);
exit(1);
}
n=atoi(argv[1]);
odirect = atoi(argv[2]);
if ((n < 1) || (n > MAX_THREAD))
{
printf ("The # of thread should between 1 and
%d.\n",MAX_THREAD);
exit(1);
}
threads=(pthread_t *)malloc(n*sizeof(*threads));
pthread_attr_init(&pthread_custom_attr);
p=(parm *)malloc(sizeof(parm)*n);
/* Start up thread */
for (i = 0; i < n; i++)
{
p[i].id = i;
p[i].odirect = odirect;
pthread_create(&threads[i], &pthread_custom_attr,
expand, (void *)(p+i));
}
/* Synchronize the completion of each thread. */
for (i=0; i<n; i++)
{
pthread_join(threads[i],NULL);
}
free(p);
return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
