www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.file or stream

reply Kar <akmalxxx gmail.com> writes:
Im trying to index a very large document collections. I need to know which file
access method is faster, either std.file read all data into buffer or use
stream.

i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.

The index file could be more than 10GBs, so which method is better.
Nov 22 2007
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

Well, assuming the machine you're going to be running on doesn't actually have 10 GB of memory, you probably don't want to read it all in at once. Streams work by reading in only what you actually use as you use it. A stream remembers "where" it is, so it doesn't need to read the whole file into memory. -- Daniel
Nov 22 2007
parent "Janice Caron" <caron800 googlemail.com> writes:
Halfway between the two extremes is the BufferedFile, which keeps a
bufferful of data in memory, and the rest of the file on disc.
Nov 23 2007
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

std.mmfile for the win. Try the higher-level interface in std.stream.MmFileStream.
Nov 23 2007
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.

std.mmfile for the win. Try the higher-level interface in std.stream.MmFileStream.

I haven't used memory-mapping before, so excuse my ignorance, but how would you map a 10GB file into the address space on a 32-bit machine?! -- Daniel
Nov 23 2007
parent Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 
 Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.

Try the higher-level interface in std.stream.MmFileStream.

I haven't used memory-mapping before, so excuse my ignorance, but how would you map a 10GB file into the address space on a 32-bit machine?! -- Daniel

Um...good point. Allow me to blush.
Nov 23 2007
prev sibling parent Kar <akmalxxx gmail.com> writes:
Christopher Wright Wrote:

 Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.

std.mmfile for the win. Try the higher-level interface in std.stream.MmFileStream.

Im developing the program under win machine but it will be run as a linux daemon. So mmfile is not an option for me. But i think i understand more about stream now, thanks alot. btw, any other option for buffer file other than mmfile?
Nov 25 2007