• Kar <akmalxxx gmail.com> Nov 22 2007
• Daniel Keep <daniel.keep.lists gmail.com> Nov 22 2007
• "Janice Caron" <caron800 googlemail.com> Nov 23 2007
• Christopher Wright <dhasenan gmail.com> Nov 23 2007
• Daniel Keep <daniel.keep.lists gmail.com> Nov 23 2007
• Christopher Wright <dhasenan gmail.com> Nov 23 2007
• Kar <akmalxxx gmail.com> Nov 25 2007

Kar <akmalxxx gmail.com> writes:

Im trying to index a very large document collections. I need to know which file
access method is faster, either std.file read all data into buffer or use
stream.

i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.

The index file could be more than 10GBs, so which method is better.

Daniel Keep <daniel.keep.lists gmail.com> writes:

Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.


Well, assuming the machine you're going to be running on doesn't
actually have 10 GB of memory, you probably don't want to read it all in
at once.

Streams work by reading in only what you actually use as you use it.  A
stream remembers "where" it is, so it doesn't need to read the whole
file into memory.

	-- Daniel

"Janice Caron" <caron800 googlemail.com> writes:

Halfway between the two extremes is the BufferedFile, which keeps a
bufferful of data in memory, and the rest of the file on disc.

Christopher Wright <dhasenan gmail.com> writes:

Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.


std.mmfile for the win.

Try the higher-level interface in std.stream.MmFileStream.

Daniel Keep <daniel.keep.lists gmail.com> writes:

Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.


 std.mmfile for the win.
 
 Try the higher-level interface in std.stream.MmFileStream.


I haven't used memory-mapping before, so excuse my ignorance, but how
would you map a 10GB file into the address space on a 32-bit machine?!

	-- Daniel

Christopher Wright <dhasenan gmail.com> writes:

Daniel Keep wrote:
 
 Christopher Wright wrote:
 Kar wrote:
 Im trying to index a very large document collections. I need to know
 which file access method is faster, either std.file read all data into
 buffer or use stream.

 i didnt understand stream very well, does it store file strings in
 memory or just a pointer to exact disc location.

 The index file could be more than 10GBs, so which method is better.



 Try the higher-level interface in std.stream.MmFileStream.


 I haven't used memory-mapping before, so excuse my ignorance, but how
 would you map a 10GB file into the address space on a 32-bit machine?!
 
 	-- Daniel


Um...good point. Allow me to blush.

Kar <akmalxxx gmail.com> writes:

Christopher Wright Wrote:

 Kar wrote:
 Im trying to index a very large document collections. I need to know which
file access method is faster, either std.file read all data into buffer or use
stream.
 
 i didnt understand stream very well, does it store file strings in memory or
just a pointer to exact disc location.
 
 The index file could be more than 10GBs, so which method is better.


 std.mmfile for the win.
 
 Try the higher-level interface in std.stream.MmFileStream.


Im developing the program under win machine but it will be run as a linux
daemon. So mmfile is not an option for me. But i think i understand more about
stream now, thanks alot. btw, any other option for buffer file other than
mmfile?