I am running 3.1.3. This application is a java application that simply
spawns threads and writes files. I've broken down the latency into 2
commands:
1) FileOpen
2) File Write including close.
Below is the code:
public boolean writeTextFile(String dir, String fname, String data){
BufferedWriter bufferedOutput = null;
String filename = commandArgs.path + "/" + String.valueOf(dir)
+ "/" + fname;
boolean suc=false;
long lat = System.currentTimeMillis();
long lat_w = 0l;
try {
//Construct the BufferedOutputStream object
log.info("Writing to text file " + filename);
bufferedOutput = new BufferedWriter(new FileWriter(filename));
log.info("FileOpen: write " + (System.currentTimeMillis() - lat));
//Start writing to the output stream
lat_w = System.currentTimeMillis();
bufferedOutput.write(data);
suc=true;
} catch (FileNotFoundException ex) {
log.error("File not found " + filename);
} catch (IOException ex) {
ex.printStackTrace();
} finally {
//Close the BufferedOutputStream
try {
if (bufferedOutput != null) {
// bufferedOutput.flush();
bufferedOutput.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
log.info("File: write " + (System.currentTimeMillis() - lat_w));
return suc;
}
On Tue, May 3, 2011 at 11:00 PM, Pavan <tcp at gluster.com> wrote:
> On Tuesday 03 May 2011 09:52 PM, Mohit Anchlia wrote:
>>
>> Yes I can reproduce everytime I run my tests.
>> This happens always. Not sure if you call 50 threads writing files to
>> 4 X 2 servers high load.
>
> What version of GlusterFS are you using?
>
> What application is seeing the slow open? How are you measuring the open
> latency?
>
>> Application see slower response time and low throughput. For eg: you
>> would expect at least 80MB/s on 1GigE with 4 X 2 when each cluster
>> with "dd" can do 130 MB/s. But because of this I see only 20-30MB/s.
>
> I am not very sure I would dive right away to that conclusion. Lets first
> measure what is taking long to complete.
>
> Pavan
>
>>
>> On Mon, May 2, 2011 at 9:36 PM, Pavan<tcp at gluster.com> ?wrote:
>>>
>>> On Sunday 01 May 2011 03:59 AM, Mohit Anchlia wrote:
>>>>
>>>> Can someone from dev help with this? Is it overhead in fuse or
>>>> something else? I am seeing file open operation taking anywhere
>>>> between 30ms - 2 secs and this is killing the performance.
>>>
>>> * Can you recreate this problem on your setup at will?
>>> * Does the delay happen when you are opening any file or is it under
>>> certain
>>> scenarios - like excessive load? What is application that is facing this
>>> hang?
>>>
>>> To start with, check if the open on the backend (on a single brick) is
>>> taking long, with the same application.
>>>
>>> Pavan
>>>
>>>>
>>>> On Fri, Apr 29, 2011 at 4:27 PM, Mohit Anchlia<mohitanchlia at gmail.com>
>>>> ?wrote:
>>>>>
>>>>> I am not seeing optimal performance out of my 4 X 2 cluster node. When
>>>>> I look at thread state that's writing files I see most of them are
>>>>> waiting on file open.
>>>>>
>>>>> Does anyone know how to further troubleshoot this issue?
>>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>>
>
>
