www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - a pretty exciting result for parallel D lang rmd following defrag by

reply "Jay Norwood" <jayn prismnet.com> writes:
These are measured times to unzip and then delete a 2GB folder in 
Win7. Both are using the parallel rmd to remove the directory on 
a regular hard drive.  The first measurement is for an unzip of 
the archive.  The second is remove of the folder when no defrag 
has been done.  The third is unzip of the same archive.  
Following it, I used a myDefrag script to sort the LCN positions 
of all the files in the folder based on the full path name.  They 
describe this sort by name script on their website.  Following 
that I ran the rmd D program to remove the folder, and it took 
only 3.7  secs ( vs 197 secs the first time). I thought I must 
have done something wrong so I repeated the whole thing, and zipd 
up the folder before deleting it and also looked at its 
properties and poked around in it.  Same 3.7 second delete.   
I'll have to analyze what is happening, but this is a huge 
improvement.   If it is just the sequential LCN order of the 
operations, it may be that I can just pre-sort the delete 
operations by the file lcn number and get similar results.  It 
also makes a case for creating a zip and unzip implementations 
that preserve the sort by filepath order.

G:\>uzp tz.zip tz
unzipping: .\tz.zip
finished! time: 87066 ms

G:\>rmd tz
removing: .\tz
finished! time:197182 ms

G:\>uzp tzip.zip tz
unzipping: .\tzip.zip
finished! time: 86015 ms

G:\>rmd tz
removing: .\tz
finished! time:3654 ms


Below is the simple sortByName defrag script that I ran prior to 
the deletion.


# MyDefrag v4.0 default script: Sort By Name
#
# This is an example script.

Title('Sort By Name tz')
Description('
Sort all the files in G:\tz by name on all the selected disk(s).
')



WriteLogfile("MyDefrag.log","LogHeader")

VolumeSelect
   Name("g:")
VolumeActions

   AppendLogfile("MyDefrag.log","LogBefore")

   FileSelect
     DirectoryName("tz")
   FileActions
     SortByName(Ascending)
   FileEnd

   AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

AppendLogfile("MyDefrag.log","LogFooter")
Apr 07 2012
next sibling parent reply "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:
 in it.  Same 3.7 second delete.   I'll have to analyze what is 
 happening, but this is a huge improvement.   If it is just the 
 sequential LCN order of the operations, it may be that I can 
 just pre-sort the delete operations by the file lcn number and 
 get similar results.

I ran rmd in the debugger to look at the order of entries being returned from the depth first search. The directory entry list returned is sorted alphabetically the same whether or not the sortByName() defrag script has been executed. This article confirms that directory entries are sorted alphabetically. http://msdn.microsoft.com/en-us/library/ms995846.aspx "Directory entries are sorted alphabetically, which explains why NTFS files are always printed alphabetically in directory listings." I'll have to write something to dump the starting lcn for each directory entry and see if the sortByName defrag is matching the DirEntries list exactly.
Apr 08 2012
parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 08/04/2012 09:34, Jay Norwood a écrit :
 On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:
 in it.  Same 3.7 second delete.   I'll have to analyze what is
 happening, but this is a huge improvement.   If it is just the
 sequential LCN order of the operations, it may be that I can just
 pre-sort the delete operations by the file lcn number and get similar
 results.

I ran rmd in the debugger to look at the order of entries being returned from the depth first search. The directory entry list returned is sorted alphabetically the same whether or not the sortByName() defrag script has been executed. This article confirms that directory entries are sorted alphabetically. http://msdn.microsoft.com/en-us/library/ms995846.aspx "Directory entries are sorted alphabetically, which explains why NTFS files are always printed alphabetically in directory listings." I'll have to write something to dump the starting lcn for each directory entry and see if the sortByName defrag is matching the DirEntries list exactly.

Hi, You seem to have done a pretty good job with your parallel unzip. Have you tried a parallel zip as well ? Do you think you could include this in std.zip when you're done ?
Apr 08 2012
parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 08/04/2012 18:14, Jay Norwood a écrit :
 On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:
 Hi,

 You seem to have done a pretty good job with your parallel unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done ?

I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.

version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }
Apr 08 2012
parent Somedude <lovelydear mailmetrash.com> writes:
Le 09/04/2012 00:15, Somedude a écrit :
 Le 08/04/2012 18:14, Jay Norwood a écrit :
 On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:
 Hi,

 You seem to have done a pretty good job with your parallel unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done ?

I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.

version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }

Or rather: // single thread zip ... version (parallel) { import std.parallelism; // multithreaded ... }
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:
 Hi,

 You seem to have done a pretty good job with your parallel 
 unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done 
 ?

I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 16:14:05 UTC, Jay Norwood wrote:

There are signficant improvements also in copy operations as a 
result of defrag by Name.  43 seconds vs 1 min 43 secs for xcopy 
of sorted 2GB vs unsorted.


this is the 2GB folder defragged with sorted LCN by pathname
G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo 
!TIME!"
12:28:15.30
34119 File(s) copied
12:28:58.81


this is the same 2GB folder, but not defragged
G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo 
!TIME!"
12:34:10.58
34119 File(s) copied
12:35:53.14

I think it is probable you would see a large part of this 
improvement if you sorted accesses by LCN,  so you would need to 
have support for looking up the lcn for a directory entry.

This guy has made c# wrappers for some of the ntfs defrag api.  
It includes looking up the lcns for a filename.  We would just 
need the first lcn, and sort accesses by that.  After unzip, 
there were only 300 of the 34000 entries that were fragmented, so 
my guess is just sorting accesses by the start lcn would provide 
most of the benefit that would be achieved by reorganizing the 
files with the defrag by filename.

http://blogs.msdn.com/b/jeffrey_wall/archive/2004/09/13/229137.aspx
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 22:17:43 UTC, Somedude wrote:

version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }

Or rather: // single thread zip ... version (parallel) { import std.parallelism; // multithreaded ... }

ok, I'll look at doing that. Thanks.
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
I was able to achieve similar efficiency to the defrag result on 
ntfs by using a modified version of std.file.write that uses 
FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 
sec vs 161 sec when removing the unzipped layout.  I posted  the 
measurements in D.learn, as well as the modified code.

http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org

This has a big effect on processing files in a folder of a hard 
drive, where the operations are dominated by the seek times.
Apr 21 2012
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 22 Apr 2012 01:10:18 +0200
schrieb "Jay Norwood" <jayn prismnet.com>:

 I was able to achieve similar efficiency to the defrag result on 
 ntfs by using a modified version of std.file.write that uses 
 FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 
 sec vs 161 sec when removing the unzipped layout.  I posted  the 
 measurements in D.learn, as well as the modified code.
 
 http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org
 
 This has a big effect on processing files in a folder of a hard 
 drive, where the operations are dominated by the seek times.

So when you did your first measurements, with 160 seconds for rmd, did you wait for the I/O to complete? Sorry if that's a stupid question :p but that's the obvious difference when using write-through from what the documentation says. -- Marco
Apr 22 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 22 April 2012 at 09:33:59 UTC, Marco Leise wrote:
 So when you did your first measurements, with 160 seconds for 
 rmd, did you wait for the I/O to complete? Sorry if that's a 
 stupid question :p but that's the obvious difference when using 
 write-through from what the documentation says.

You are probably right that the buffers are just not flushed. I unzipped with 7zip, then waited 5 min, then used rmdir below, which still took 144 secs. G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 11:47:32.40 11:49:56.70 If I use 7zip to unzip, then reboot prior to rmdir, the rmdir takes only 27 sec G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 12:47:38.13 12:48:05.60 If I use 7zip to unzip, then run sync on G:, rmdir takes 2:36 http://technet.microsoft.com/en-us/sysinternals/bb897438 C:\Ntutils>sync g: Sync 2.2: Disk Flusher for Windows 9x/Me/NT/2K/XP Copyright (C) 1997-2004 Mark Russinovich Sysinternals - www.sysinternals.com Flushing: G G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 13:10:11.93 13:12:47.75 according to that ntfs document, the buffers should have been scheduled for flush within about 8 secs. http://msdn.microsoft.com/en-us/library/windows/desktop/aa364218(v=vs.85).aspx "To ensure that the right amount of flushing occurs, the cache manager spawns a process every second called a lazy writer. The lazy writer process queues one-eighth of the pages that have not been flushed recently to be written to disk. It constantly reevaluates the amount of data being flushed for optimal system performance, and if more data needs to be written it queues more data. Lazy writers do not flush temporary files, because the assumption is that they will be deleted by the application or system. " This is measurement below is for rmdir immediately following unzip with the writeThrough code ... about 6 secs. G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 12:07:58.23 12:08:04.26 I can't explain why the writeThrough enables faster rmdir operation, but I saw similar improvement following myDefrag sortByName on the folder, which is what I was attempting to duplicate with the sorted unzip. At this time it provides the only reasonable way to get consistent results ... the lazy flushing doesn't seem to work, myDefrag takes too long, the sync utility doesn't seem to improve anything, and reboot isn't really an option.
Apr 22 2012
prev sibling parent "Jay Norwood" <jayn prismnet.com> writes:
Table 5.1 in this article, and some surrounding description, 
indicate that ntfs converts to upper case when doing directory 
inserts, so if you want to optimize the disk order for the order 
processed by directory entry it seems toUpper would be a better 
choice.

http://members.fortunecity.com/clark_kent/exjobb/report3.html

A  $UpCase  Maps lowercase characters to their uppercase version, 
used when inserting files in a directory.

Maybe a more efficient ordering for a folder could be achieved by 
create zero length files for everything, then read back the ntfs 
directory entries, and use that order to expand the files.
Apr 22 2012