www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - a pretty exciting result for parallel D lang rmd following defrag by

reply "Jay Norwood" <jayn prismnet.com> writes:
These are measured times to unzip and then delete a 2GB folder in 
Win7. Both are using the parallel rmd to remove the directory on 
a regular hard drive.  The first measurement is for an unzip of 
the archive.  The second is remove of the folder when no defrag 
has been done.  The third is unzip of the same archive.  
Following it, I used a myDefrag script to sort the LCN positions 
of all the files in the folder based on the full path name.  They 
describe this sort by name script on their website.  Following 
that I ran the rmd D program to remove the folder, and it took 
only 3.7  secs ( vs 197 secs the first time). I thought I must 
have done something wrong so I repeated the whole thing, and zipd 
up the folder before deleting it and also looked at its 
properties and poked around in it.  Same 3.7 second delete.   
I'll have to analyze what is happening, but this is a huge 
improvement.   If it is just the sequential LCN order of the 
operations, it may be that I can just pre-sort the delete 
operations by the file lcn number and get similar results.  It 
also makes a case for creating a zip and unzip implementations 
that preserve the sort by filepath order.

G:\>uzp tz.zip tz
unzipping: .\tz.zip
finished! time: 87066 ms

G:\>rmd tz
removing: .\tz
finished! time:197182 ms

G:\>uzp tzip.zip tz
unzipping: .\tzip.zip
finished! time: 86015 ms

G:\>rmd tz
removing: .\tz
finished! time:3654 ms


Below is the simple sortByName defrag script that I ran prior to 
the deletion.


# MyDefrag v4.0 default script: Sort By Name
#
# This is an example script.

Title('Sort By Name tz')
Description('
Sort all the files in G:\tz by name on all the selected disk(s).
')



WriteLogfile("MyDefrag.log","LogHeader")

VolumeSelect
   Name("g:")
VolumeActions

   AppendLogfile("MyDefrag.log","LogBefore")

   FileSelect
     DirectoryName("tz")
   FileActions
     SortByName(Ascending)
   FileEnd

   AppendLogfile("MyDefrag.log","LogAfter")

VolumeEnd

AppendLogfile("MyDefrag.log","LogFooter")
Apr 07 2012
next sibling parent reply "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:

 in it.  Same 3.7 second delete.   I'll have to analyze what is 
 happening, but this is a huge improvement.   If it is just the 
 sequential LCN order of the operations, it may be that I can 
 just pre-sort the delete operations by the file lcn number and 
 get similar results.



I ran  rmd in the debugger to look at the order of entries being 
returned from the depth first search.  The directory entry list 
returned is sorted alphabetically  the same whether or not the 
sortByName() defrag script has been executed.

This article confirms that directory entries are sorted 
alphabetically.

http://msdn.microsoft.com/en-us/library/ms995846.aspx

"Directory entries are sorted alphabetically, which explains why 
NTFS files are always printed alphabetically in directory 
listings."

I'll have to write something to dump the starting lcn for each 
directory entry and see if  the sortByName defrag is matching the 
DirEntries list exactly.
Apr 08 2012
parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 08/04/2012 09:34, Jay Norwood a écrit :

 On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:

 in it.  Same 3.7 second delete.   I'll have to analyze what is
 happening, but this is a huge improvement.   If it is just the
 sequential LCN order of the operations, it may be that I can just
 pre-sort the delete operations by the file lcn number and get similar
 results.



 I ran  rmd in the debugger to look at the order of entries being
 returned from the depth first search.  The directory entry list returned
 is sorted alphabetically  the same whether or not the sortByName()
 defrag script has been executed.
 
 This article confirms that directory entries are sorted alphabetically.
 
 http://msdn.microsoft.com/en-us/library/ms995846.aspx
 
 "Directory entries are sorted alphabetically, which explains why NTFS
 files are always printed alphabetically in directory listings."
 
 I'll have to write something to dump the starting lcn for each directory
 entry and see if  the sortByName defrag is matching the DirEntries list
 exactly.



Hi,

You seem to have done a pretty good job with your parallel unzip. Have
you tried a parallel zip as well ?
Do you think you could include this in std.zip when you're done ?
Apr 08 2012
parent reply Somedude <lovelydear mailmetrash.com> writes:
Le 08/04/2012 18:14, Jay Norwood a écrit :

 On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:

 Hi,

 You seem to have done a pretty good job with your parallel unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done ?



 I'm going to do a parallel zip as well.  There is already parallel zip
 utility available with 7zip, so I haven't looked closely at D's std.zip.
 
 These parallel implementations all bring in std.parallelism as a
 dependency, and I don't know if that is acceptable.  I'm just putting
 them in my github for now, along with examples.
 
 




version (parallel)
{
  import std.parallelism;
  // multithreaded
  ...
}
else
{
  // single thread
  ...
}
Apr 08 2012
parent Somedude <lovelydear mailmetrash.com> writes:
Le 09/04/2012 00:15, Somedude a écrit :

 Le 08/04/2012 18:14, Jay Norwood a écrit :

 On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:

 Hi,

 You seem to have done a pretty good job with your parallel unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done ?



 I'm going to do a parallel zip as well.  There is already parallel zip
 utility available with 7zip, so I haven't looked closely at D's std.zip.

 These parallel implementations all bring in std.parallelism as a
 dependency, and I don't know if that is acceptable.  I'm just putting
 them in my github for now, along with examples.



 
 version (parallel)
 {
   import std.parallelism;
   // multithreaded
   ...
 }
 else
 {
   // single thread
   ...
 }



Or rather:

// single thread zip
...
version (parallel)
{
   import std.parallelism;
   // multithreaded
   ...
}
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:

 Hi,

 You seem to have done a pretty good job with your parallel 
 unzip. Have
 you tried a parallel zip as well ?
 Do you think you could include this in std.zip when you're done 
 ?



I'm going to do a parallel zip as well.  There is already 
parallel zip utility available with 7zip, so I haven't looked 
closely at D's std.zip.

These parallel implementations all bring in std.parallelism as a 
dependency, and I don't know if that is acceptable.  I'm just 
putting them in my github for now, along with examples.
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 16:14:05 UTC, Jay Norwood wrote:

There are signficant improvements also in copy operations as a 
result of defrag by Name.  43 seconds vs 1 min 43 secs for xcopy 
of sorted 2GB vs unsorted.


this is the 2GB folder defragged with sorted LCN by pathname
G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo 
!TIME!"
12:28:15.30
34119 File(s) copied
12:28:58.81


this is the same 2GB folder, but not defragged
G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo 
!TIME!"
12:34:10.58
34119 File(s) copied
12:35:53.14

I think it is probable you would see a large part of this 
improvement if you sorted accesses by LCN,  so you would need to 
have support for looking up the lcn for a directory entry.

This guy has made c# wrappers for some of the ntfs defrag api.  
It includes looking up the lcns for a filename.  We would just 
need the first lcn, and sort accesses by that.  After unzip, 
there were only 300 of the 34000 entries that were fragmented, so 
my guess is just sorting accesses by the start lcn would provide 
most of the benefit that would be achieved by reorganizing the 
files with the defrag by filename.

http://blogs.msdn.com/b/jeffrey_wall/archive/2004/09/13/229137.aspx
Apr 08 2012
prev sibling next sibling parent "Jay Norwood" <jayn prismnet.com> writes:
On Sunday, 8 April 2012 at 22:17:43 UTC, Somedude wrote:



 
 version (parallel)
 {
   import std.parallelism;
   // multithreaded
   ...
 }
 else
 {
   // single thread
   ...
 }



 Or rather:

 // single thread zip
 ...
 version (parallel)
 {
    import std.parallelism;
    // multithreaded
    ...
 }



ok, I'll look at doing that.  Thanks.



 Apr 08 2012



prev sibling next sibling parent  "Jay Norwood" <jayn prismnet.com>  writes:


I was able to achieve similar efficiency to the defrag result on 
ntfs by using a modified version of std.file.write that uses 
FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 
sec vs 161 sec when removing the unzipped layout.  I posted  the 
measurements in D.learn, as well as the modified code.

http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org

This has a big effect on processing files in a folder of a hard 
drive, where the operations are dominated by the seek times.



 Apr 21 2012



prev sibling next sibling parent  Marco Leise <Marco.Leise gmx.de>  writes:


Am Sun, 22 Apr 2012 01:10:18 +0200
schrieb "Jay Norwood" <jayn prismnet.com>:


 I was able to achieve similar efficiency to the defrag result on 
 ntfs by using a modified version of std.file.write that uses 
 FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 
 sec vs 161 sec when removing the unzipped layout.  I posted  the 
 measurements in D.learn, as well as the modified code.
 
 http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org
 
 This has a big effect on processing files in a folder of a hard 
 drive, where the operations are dominated by the seek times.



So when you did your first measurements, with 160 seconds for rmd, did you wait
for the I/O to complete? Sorry if that's a stupid question :p but that's the
obvious difference when using write-through from what the documentation says.

-- 
Marco



 Apr 22 2012



prev sibling next sibling parent  "Jay Norwood" <jayn prismnet.com>  writes:


On Sunday, 22 April 2012 at 09:33:59 UTC, Marco Leise wrote:

 So when you did your first measurements, with 160 seconds for 
 rmd, did you wait for the I/O to complete? Sorry if that's a 
 stupid question :p but that's the obvious difference when using 
 write-through from what the documentation says.



You are probably right that the buffers are just not flushed. I 
unzipped with 7zip, then waited 5 min, then used rmdir below, 
which still took 144 secs.

G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!"
11:47:32.40
11:49:56.70

If I use 7zip to unzip, then reboot prior to rmdir, the rmdir 
takes only 27 sec

G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!"
12:47:38.13
12:48:05.60

If I use 7zip to unzip, then run sync on G:, rmdir takes 2:36
http://technet.microsoft.com/en-us/sysinternals/bb897438
C:\Ntutils>sync g:

Sync 2.2: Disk Flusher for Windows 9x/Me/NT/2K/XP
Copyright (C) 1997-2004 Mark Russinovich
Sysinternals - www.sysinternals.com

Flushing: G

G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!"
13:10:11.93
13:12:47.75





according to that ntfs document, the buffers should have been 
scheduled for flush within about 8 secs.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa364218(v=vs.85).aspx

"To ensure that the right amount of flushing occurs, the cache 
manager spawns a process every second called a lazy writer. The 
lazy writer process queues one-eighth of the pages that have not 
been flushed recently to be written to disk. It constantly 
reevaluates the amount of data being flushed for optimal system 
performance, and if more data needs to be written it queues more 
data. Lazy writers do not flush temporary files, because the 
assumption is that they will be deleted by the application or 
system.
"

This is measurement below is for rmdir immediately following 
unzip with the writeThrough code  ... about 6 secs.
G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!"
12:07:58.23
12:08:04.26

I can't explain why the writeThrough enables faster rmdir 
operation, but I saw similar improvement following myDefrag 
sortByName on the folder, which is what I was attempting to 
duplicate with the sorted unzip.  At this time it provides the 
only reasonable way to get consistent results ... the lazy 
flushing doesn't seem to work, myDefrag takes too long, the sync 
utility doesn't seem to improve anything, and reboot isn't really 
an option.



 Apr 22 2012



prev sibling  parent  "Jay Norwood" <jayn prismnet.com>  writes:


Table 5.1 in this article, and some surrounding description, 
indicate that ntfs converts to upper case when doing directory 
inserts, so if you want to optimize the disk order for the order 
processed by directory entry it seems toUpper would be a better 
choice.

http://members.fortunecity.com/clark_kent/exjobb/report3.html

A  $UpCase  Maps lowercase characters to their uppercase version, 
used when inserting files in a directory.

Maybe a more efficient ordering for a folder could be achieved by 
create zero length files for everything, then read back the ntfs 
directory entries, and use that order to expand the files.



 Apr 22 2012