www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Any easy way to extract files to memory buffer?

reply Michelle Long <HappyDance321 gmail.com> writes:
Trying to speed up extracting some files that I first have to 
extract using the command line to files then read those in...

Not sure what is taking so long. I imagine windows caches the 
extraction so maybe it is pointless?
Mar 18
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
 Trying to speed up extracting some files that I first have to 
 extract using the command line to files then read those in...

 Not sure what is taking so long. I imagine windows caches the 
 extraction so maybe it is pointless?
You can speed up such things using a tmpfs or RAM disk. On Linux you just use mount -t tmpfs ... On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever). If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program.
Mar 18
parent reply Michelle Long <HappyDance321 gmail.com> writes:
On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev 
wrote:
 On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
 Trying to speed up extracting some files that I first have to 
 extract using the command line to files then read those in...

 Not sure what is taking so long. I imagine windows caches the 
 extraction so maybe it is pointless?
You can speed up such things using a tmpfs or RAM disk. On Linux you just use mount -t tmpfs ... On Windows it is a little more complicated, there are programs like ImDisk that can create RAM drives (which can then be formatted to whatever). If it's just one file, sometimes you can pipe it from the unpacking program (tar etc.) into the consuming program.
Yeah, but it seems like a lot of work simply to extract the files to memory.
Mar 18
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via Digitalmars-d-learn
wrote:
 On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev wrote:
 On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long wrote:
 Trying to speed up extracting some files that I first have to
 extract using the command line to files then read those in...
 
 Not sure what is taking so long. I imagine windows caches the
 extraction so maybe it is pointless?
[...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T -- Give me some fresh salted fish, please.
Mar 18
parent reply Michelle Long <HappyDance321 gmail.com> writes:
On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote:
 On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via 
 Digitalmars-d-learn wrote:
 On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev 
 wrote:
 On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long 
 wrote:
 Trying to speed up extracting some files that I first have 
 to extract using the command line to files then read those 
 in...
 
 Not sure what is taking so long. I imagine windows caches 
 the extraction so maybe it is pointless?
[...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T
The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't.
Mar 18
parent Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Monday, 18 March 2019 at 23:40:02 UTC, Michelle Long wrote:
 On Monday, 18 March 2019 at 23:01:27 UTC, H. S. Teoh wrote:
 On Mon, Mar 18, 2019 at 10:38:17PM +0000, Michelle Long via 
 Digitalmars-d-learn wrote:
 On Monday, 18 March 2019 at 21:14:05 UTC, Vladimir Panteleev 
 wrote:
 On Monday, 18 March 2019 at 21:09:55 UTC, Michelle Long 
 wrote:
 Trying to speed up extracting some files that I first 
 have to extract using the command line to files then read 
 those in...
 
 Not sure what is taking so long. I imagine windows caches 
 the extraction so maybe it is pointless?
[...] Why not just use std.mmfile to memory-map the file into memory directly? Let the OS take care of actually paging in the file data. T
The files are on disk and there is an external program that read them and converts them and then writes the converted files to disk then my program reads. Ideally the conversion program would take memory instead of disk files but it doesn't.
the file that was written by the first program will be in the file cache. mmap() (and the Windows equivalent of that) syscalls are at the core only giving access to the OS file cache. This means that std.mmfile is the way to go. There will be no reloading from disk if the file sizes are within reason.
Mar 19