www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Memory leak with BufferedFile?

reply Leon <leonp gmail.com> writes:
I'm writing a simple program which uses merge sort to sort very large text
files. I split the text file into several temporary files which are sorted and
merged together.

The trouble I'm having is that the BufferedFile class seems to have a memory
leak. The memory usage of the process slowly goes up until it throws an out of
memory exception. I can fix the memory leak by switching to the File class,
but that makes the program much slower.

Calling GC.collect() didn't help, and neither did explicitly deleting the
BufferedFile object.
Aug 27 2011
next sibling parent =?UTF-8?B?Q2hyaXN0aWFuIEvDtnN0bGlu?= <christian.koestlin gmail.com> writes:
On 8/28/11 1:00 , Leon wrote:
 I'm writing a simple program which uses merge sort to sort very large text
 files. I split the text file into several temporary files which are sorted and
 merged together.

 The trouble I'm having is that the BufferedFile class seems to have a memory
 leak. The memory usage of the process slowly goes up until it throws an out of
 memory exception. I can fix the memory leak by switching to the File class,
 but that makes the program much slower.

 Calling GC.collect() didn't help, and neither did explicitly deleting the
 BufferedFile object.

Thats very interesting! Perhaps you can have a look at: https://github.com/gizmomogwai/performance There I ran into performanceproblems using BufferedFile. One of the best solutions I found till now is using mmfile as in https://github.com/gizmomogwai/performance/blob/master/src/d/readbytes5.d. Perhaps this could be of help for you. I dont know how you have to access the data in the files, but perhaps the interface of mmfile is good enough for you! christian
Aug 29 2011
prev sibling parent torhu <no spam.invalid> writes:
On 28.08.2011 01:00, Leon wrote:
 I'm writing a simple program which uses merge sort to sort very large text
 files. I split the text file into several temporary files which are sorted and
 merged together.

 The trouble I'm having is that the BufferedFile class seems to have a memory
 leak. The memory usage of the process slowly goes up until it throws an out of
 memory exception. I can fix the memory leak by switching to the File class,
 but that makes the program much slower.

 Calling GC.collect() didn't help, and neither did explicitly deleting the
 BufferedFile object.

You could try just using the functions in std.stdio instead.
Aug 29 2011