www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Coping files and folders

reply "Joel" <joelcnz gmail.com> writes:
How does one copy folders (with files in them) including sub 
folders, and creating any needed folders?

Like:
land\house\cat.d
land\house\rat.exe
land\house\bedroom\ants.txt

to
root\island\house\cat.d
root\island\house\rat.exe
root\island\house\bedroom\ants.txt

One work around is to use 'system' (under std.process).

-joelcnz
Jan 23 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-24 07:28, Joel wrote:
 How does one copy folders (with files in them) including sub folders,
 and creating any needed folders?

 Like:
 land\house\cat.d
 land\house\rat.exe
 land\house\bedroom\ants.txt

 to
 root\island\house\cat.d
 root\island\house\rat.exe
 root\island\house\bedroom\ants.txt

 One work around is to use 'system' (under std.process).
I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove. -- /Jacob Carlborg
Jan 23 2013
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg 
wrote:
 On 2013-01-24 07:28, Joel wrote:
 How does one copy folders (with files in them) including sub 
 folders,
 and creating any needed folders?

 Like:
 land\house\cat.d
 land\house\rat.exe
 land\house\bedroom\ants.txt

 to
 root\island\house\cat.d
 root\island\house\rat.exe
 root\island\house\bedroom\ants.txt

 One work around is to use 'system' (under std.process).
I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.
Mustn't be very hard to manually write "copyDir". The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue? Anyways, I just threw this together. The first version bails on first error. The second version keeps going as much as it can, and returns true on success. The caller is then free to (or not to) call rmdirRecurse in case of failure. //Throws exception on first error. void copyDir(string inDir, string outDir) { if (!exists(outDir)) mkdir(outDir); else if (!isDir(outDir)) throw new FileException(format("Destination path %s is not a folder.", outDir)); foreach (entry; dirEntries(inDir.idup, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) copyDir(entry.name, destName); else copy(entry.name, destName); } } //Silently keeps going as much as it can, then returns true on success, //or false if an error occured. bool copyDirSilent(string inDir, string outDir) { if (!exists(outDir)) { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } else if (!isDir(outDir)) return false; foreach (entry; dirEntries(inDir, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) { bool b = copyDirSilent(entry.name, destName); if (b == false) return false; } else { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } } return true; }
Jan 24 2013
parent "Joel" <joelcnz gmail.com> writes:
Brilliant! Thanks.

I think the silent one has errors. But I've used the first 
function in my program now. With mine, strait after doing the 
copying is was supposed then update a file, but didn't. Your one 
seems to work though.

On Thursday, 24 January 2013 at 08:58:06 UTC, monarch_dodra wrote:
 On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg 
 wrote:
 On 2013-01-24 07:28, Joel wrote:
 How does one copy folders (with files in them) including sub 
 folders,
 and creating any needed folders?

 Like:
 land\house\cat.d
 land\house\rat.exe
 land\house\bedroom\ants.txt

 to
 root\island\house\cat.d
 root\island\house\rat.exe
 root\island\house\bedroom\ants.txt

 One work around is to use 'system' (under std.process).
I don't think Phobos currently has any functions for this. Someone posted code in these newsgroups of a parallel implementation of copy and remove.
Mustn't be very hard to manually write "copyDir". The problem with "copyDir" is its transactional behavior: What should it do in case of a failure mid copy? Bail? Cleanup? Continue? Anyways, I just threw this together. The first version bails on first error. The second version keeps going as much as it can, and returns true on success. The caller is then free to (or not to) call rmdirRecurse in case of failure. //Throws exception on first error. void copyDir(string inDir, string outDir) { if (!exists(outDir)) mkdir(outDir); else if (!isDir(outDir)) throw new FileException(format("Destination path %s is not a folder.", outDir)); foreach (entry; dirEntries(inDir.idup, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) copyDir(entry.name, destName); else copy(entry.name, destName); } } //Silently keeps going as much as it can, then returns true on success, //or false if an error occured. bool copyDirSilent(string inDir, string outDir) { if (!exists(outDir)) { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } else if (!isDir(outDir)) return false; foreach (entry; dirEntries(inDir, SpanMode.shallow)) { auto fileName = baseName(entry.name); auto destName = buildPath(outDir, fileName); if (entry.isDir()) { bool b = copyDirSilent(entry.name, destName); if (b == false) return false; } else { auto e = collectException(mkdir(outDir)); if (e !is null) return false; } } return true; }
Jan 24 2013
prev sibling parent reply "Jay Norwood" <jayn prismnet.com> writes:
On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg 
wrote:

 Someone posted code in these newsgroups of a parallel 
 implementation of copy and remove.
I posted a parallel implementation a while back, and also put it on github. The parallel trick is to create the folder structure first, then populate the folders with the files. Best for ssd drives. I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwood
Jan 24 2013
next sibling parent reply "Joel" <joelcnz gmail.com> writes:
Sounds cool. I've bookmarked your github hub link.

Just wondering with this:

// parallel foreach for regular files
foreach(fn ; taskPool.parallel(files,100)) {
     string dfn = destRoot ~ fn[srcLen..$];
     copy(fn,dfn);
}

What is the 100 number for?

On Friday, 25 January 2013 at 05:30:11 UTC, Jay Norwood wrote:
 On Thursday, 24 January 2013 at 07:41:23 UTC, Jacob Carlborg 
 wrote:

 Someone posted code in these newsgroups of a parallel 
 implementation of copy and remove.
I posted a parallel implementation a while back, and also put it on github. The parallel trick is to create the folder structure first, then populate the folders with the files. Best for ssd drives. I also wrote a copy version that orders file sequence on disk efficiently, using write through, and posted it. This speeds up any subsequent file system operations done in the directory order as if you have done a defrag. Great for hard drives, but not needed for ssd. https://github.com/jnorwood
Jan 24 2013
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 01/24/2013 10:23 PM, Joel wrote:

 // parallel foreach for regular files
 foreach(fn ; taskPool.parallel(files,100)) {
 What is the 100 number for?
It is the work unit size: http://dlang.org/phobos/std_parallelism.html#.TaskPool Quoting: The number of elements processed per work unit is controlled by the workUnitSize parameter. Smaller work units provide better load balancing, but larger work units avoid the overhead of communicating with other threads frequently to fetch the next work unit. Large work units also avoid false sharing in cases where the range is being modified. The less time a single iteration of the loop takes, the larger workUnitSize should be. For very expensive loop bodies, workUnitSize should be 1. An overload that chooses a default work unit size is also available. Ali
Jan 25 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-25 06:30, Jay Norwood wrote:

 I posted a parallel implementation a while back, and also put it on github.

 The parallel trick is to create the folder structure first, then
 populate the folders with the files.  Best for ssd drives.

 I also wrote a copy version that orders file sequence on disk
 efficiently, using write through, and posted it.  This speeds up any
 subsequent file system operations done in the directory order as if you
 have done a defrag. Great for hard drives, but not needed for ssd.

 https://github.com/jnorwood
Any change you could turn this into a pull request and submit to Phobos? -- /Jacob Carlborg
Jan 24 2013
parent "Jay Norwood" <jayn prismnet.com> writes:
 I also wrote a copy version that orders file sequence on disk
 efficiently, using write through, and posted it.  This speeds 
 up any
 subsequent file system operations done in the directory order 
 as if you
 have done a defrag. Great for hard drives, but not needed for 
 ssd.

 https://github.com/jnorwood
Any change you could turn this into a pull request and submit to Phobos?
Looks like I never promoted the write through code on github. I just posted it on the forum.dlang.org, along with some timings. I was only experimenting on ntfs, and so the extension I posted doesn't force write-through on linux. http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org After re-reading my post, I don't think I experimented with write through in copydir. I show the timings in the post for ntfs operations on a 2GB layout after unzipping with a write through version of unzip. I suspect you'd see similar improvement following a copydir that uses write through. The problem I saw with ntfs was that the target directory was very fragmented when allowing ntfs to do its lazy flushes. I tried several experiments, and using the write through was the only solution that resulted in a reasonably defragged target immediately as the result of the unzip.
Jan 25 2013