digitalmars.D - a pretty exciting result for parallel D lang rmd following defrag by
- Jay Norwood (53/53) Apr 07 2012 These are measured times to unzip and then delete a 2GB folder in
- Jay Norwood (14/19) Apr 08 2012 I ran rmd in the debugger to look at the order of entries being
- Somedude (5/27) Apr 08 2012 Hi,
- Jay Norwood (7/13) Apr 08 2012 I'm going to do a parallel zip as well. There is already
- Jay Norwood (27/27) Apr 08 2012 On Sunday, 8 April 2012 at 16:14:05 UTC, Jay Norwood wrote:
- Somedude (13/28) Apr 08 2012 Well, you can always do something like this:
- Somedude (10/39) Apr 08 2012 Or rather:
- Jay Norwood (2/25) Apr 08 2012 ok, I'll look at doing that. Thanks.
- Jay Norwood (8/8) Apr 21 2012 I was able to achieve similar efficiency to the defrag result on
- Marco Leise (5/15) Apr 22 2012 So when you did your first measurements, with 160 seconds for rmd, did y...
- Jay Norwood (48/52) Apr 22 2012 You are probably right that the buffers are just not flushed. I
- Jay Norwood (11/11) Apr 22 2012 Table 5.1 in this article, and some surrounding description,
These are measured times to unzip and then delete a 2GB folder in Win7. Both are using the parallel rmd to remove the directory on a regular hard drive. The first measurement is for an unzip of the archive. The second is remove of the folder when no defrag has been done. The third is unzip of the same archive. Following it, I used a myDefrag script to sort the LCN positions of all the files in the folder based on the full path name. They describe this sort by name script on their website. Following that I ran the rmd D program to remove the folder, and it took only 3.7 secs ( vs 197 secs the first time). I thought I must have done something wrong so I repeated the whole thing, and zipd up the folder before deleting it and also looked at its properties and poked around in it. Same 3.7 second delete. I'll have to analyze what is happening, but this is a huge improvement. If it is just the sequential LCN order of the operations, it may be that I can just pre-sort the delete operations by the file lcn number and get similar results. It also makes a case for creating a zip and unzip implementations that preserve the sort by filepath order. G:\>uzp tz.zip tz unzipping: .\tz.zip finished! time: 87066 ms G:\>rmd tz removing: .\tz finished! time:197182 ms G:\>uzp tzip.zip tz unzipping: .\tzip.zip finished! time: 86015 ms G:\>rmd tz removing: .\tz finished! time:3654 ms Below is the simple sortByName defrag script that I ran prior to the deletion. Title('Sort By Name tz') Description(' Sort all the files in G:\tz by name on all the selected disk(s). ') WriteLogfile("MyDefrag.log","LogHeader") VolumeSelect Name("g:") VolumeActions AppendLogfile("MyDefrag.log","LogBefore") FileSelect DirectoryName("tz") FileActions SortByName(Ascending) FileEnd AppendLogfile("MyDefrag.log","LogAfter") VolumeEnd AppendLogfile("MyDefrag.log","LogFooter")
Apr 07 2012
On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:in it. Same 3.7 second delete. I'll have to analyze what is happening, but this is a huge improvement. If it is just the sequential LCN order of the operations, it may be that I can just pre-sort the delete operations by the file lcn number and get similar results.I ran rmd in the debugger to look at the order of entries being returned from the depth first search. The directory entry list returned is sorted alphabetically the same whether or not the sortByName() defrag script has been executed. This article confirms that directory entries are sorted alphabetically. http://msdn.microsoft.com/en-us/library/ms995846.aspx "Directory entries are sorted alphabetically, which explains why NTFS files are always printed alphabetically in directory listings." I'll have to write something to dump the starting lcn for each directory entry and see if the sortByName defrag is matching the DirEntries list exactly.
Apr 08 2012
Le 08/04/2012 09:34, Jay Norwood a écrit :On Sunday, 8 April 2012 at 01:18:49 UTC, Jay Norwood wrote:Hi, You seem to have done a pretty good job with your parallel unzip. Have you tried a parallel zip as well ? Do you think you could include this in std.zip when you're done ?in it. Same 3.7 second delete. I'll have to analyze what is happening, but this is a huge improvement. If it is just the sequential LCN order of the operations, it may be that I can just pre-sort the delete operations by the file lcn number and get similar results.I ran rmd in the debugger to look at the order of entries being returned from the depth first search. The directory entry list returned is sorted alphabetically the same whether or not the sortByName() defrag script has been executed. This article confirms that directory entries are sorted alphabetically. http://msdn.microsoft.com/en-us/library/ms995846.aspx "Directory entries are sorted alphabetically, which explains why NTFS files are always printed alphabetically in directory listings." I'll have to write something to dump the starting lcn for each directory entry and see if the sortByName defrag is matching the DirEntries list exactly.
Apr 08 2012
On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:Hi, You seem to have done a pretty good job with your parallel unzip. Have you tried a parallel zip as well ? Do you think you could include this in std.zip when you're done ?I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.
Apr 08 2012
On Sunday, 8 April 2012 at 16:14:05 UTC, Jay Norwood wrote: There are signficant improvements also in copy operations as a result of defrag by Name. 43 seconds vs 1 min 43 secs for xcopy of sorted 2GB vs unsorted. this is the 2GB folder defragged with sorted LCN by pathname G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo !TIME!" 12:28:15.30 34119 File(s) copied 12:28:58.81 this is the same 2GB folder, but not defragged G:\>cmd /v:on /c "echo !TIME! & xcopy /q /e /I tz h:\tz & echo !TIME!" 12:34:10.58 34119 File(s) copied 12:35:53.14 I think it is probable you would see a large part of this improvement if you sorted accesses by LCN, so you would need to have support for looking up the lcn for a directory entry. It includes looking up the lcns for a filename. We would just need the first lcn, and sort accesses by that. After unzip, there were only 300 of the 34000 entries that were fragmented, so my guess is just sorting accesses by the start lcn would provide most of the benefit that would be achieved by reorganizing the files with the defrag by filename. http://blogs.msdn.com/b/jeffrey_wall/archive/2004/09/13/229137.aspx
Apr 08 2012
Le 08/04/2012 18:14, Jay Norwood a écrit :On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:Well, you can always do something like this: version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }Hi, You seem to have done a pretty good job with your parallel unzip. Have you tried a parallel zip as well ? Do you think you could include this in std.zip when you're done ?I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.
Apr 08 2012
Le 09/04/2012 00:15, Somedude a écrit :Le 08/04/2012 18:14, Jay Norwood a écrit :Or rather: // single thread zip ... version (parallel) { import std.parallelism; // multithreaded ... }On Sunday, 8 April 2012 at 09:21:43 UTC, Somedude wrote:Well, you can always do something like this: version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }Hi, You seem to have done a pretty good job with your parallel unzip. Have you tried a parallel zip as well ? Do you think you could include this in std.zip when you're done ?I'm going to do a parallel zip as well. There is already parallel zip utility available with 7zip, so I haven't looked closely at D's std.zip. These parallel implementations all bring in std.parallelism as a dependency, and I don't know if that is acceptable. I'm just putting them in my github for now, along with examples.
Apr 08 2012
On Sunday, 8 April 2012 at 22:17:43 UTC, Somedude wrote:ok, I'll look at doing that. Thanks.Or rather: // single thread zip ... version (parallel) { import std.parallelism; // multithreaded ... }Well, you can always do something like this: version (parallel) { import std.parallelism; // multithreaded ... } else { // single thread ... }
Apr 08 2012
I was able to achieve similar efficiency to the defrag result on ntfs by using a modified version of std.file.write that uses FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 sec vs 161 sec when removing the unzipped layout. I posted the measurements in D.learn, as well as the modified code. http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org This has a big effect on processing files in a folder of a hard drive, where the operations are dominated by the seek times.
Apr 21 2012
Am Sun, 22 Apr 2012 01:10:18 +0200 schrieb "Jay Norwood" <jayn prismnet.com>:I was able to achieve similar efficiency to the defrag result on ntfs by using a modified version of std.file.write that uses FILE_FLAG_WRITE_THROUGH. The ntfs rmdir of the 2GB layout takes 6 sec vs 161 sec when removing the unzipped layout. I posted the measurements in D.learn, as well as the modified code. http://forum.dlang.org/thread/gmkocaqzmlmfbuozhrsj forum.dlang.org This has a big effect on processing files in a folder of a hard drive, where the operations are dominated by the seek times.So when you did your first measurements, with 160 seconds for rmd, did you wait for the I/O to complete? Sorry if that's a stupid question :p but that's the obvious difference when using write-through from what the documentation says. -- Marco
Apr 22 2012
On Sunday, 22 April 2012 at 09:33:59 UTC, Marco Leise wrote:So when you did your first measurements, with 160 seconds for rmd, did you wait for the I/O to complete? Sorry if that's a stupid question :p but that's the obvious difference when using write-through from what the documentation says.You are probably right that the buffers are just not flushed. I unzipped with 7zip, then waited 5 min, then used rmdir below, which still took 144 secs. G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 11:47:32.40 11:49:56.70 If I use 7zip to unzip, then reboot prior to rmdir, the rmdir takes only 27 sec G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 12:47:38.13 12:48:05.60 If I use 7zip to unzip, then run sync on G:, rmdir takes 2:36 http://technet.microsoft.com/en-us/sysinternals/bb897438 C:\Ntutils>sync g: Sync 2.2: Disk Flusher for Windows 9x/Me/NT/2K/XP Copyright (C) 1997-2004 Mark Russinovich Sysinternals - www.sysinternals.com Flushing: G G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 13:10:11.93 13:12:47.75 according to that ntfs document, the buffers should have been scheduled for flush within about 8 secs. http://msdn.microsoft.com/en-us/library/windows/desktop/aa364218(v=vs.85).aspx "To ensure that the right amount of flushing occurs, the cache manager spawns a process every second called a lazy writer. The lazy writer process queues one-eighth of the pages that have not been flushed recently to be written to disk. It constantly reevaluates the amount of data being flushed for optimal system performance, and if more data needs to be written it queues more data. Lazy writers do not flush temporary files, because the assumption is that they will be deleted by the application or system. " This is measurement below is for rmdir immediately following unzip with the writeThrough code ... about 6 secs. G:\>cmd /v:on /c "echo !TIME! & rmdir /q /s tz & echo !TIME!" 12:07:58.23 12:08:04.26 I can't explain why the writeThrough enables faster rmdir operation, but I saw similar improvement following myDefrag sortByName on the folder, which is what I was attempting to duplicate with the sorted unzip. At this time it provides the only reasonable way to get consistent results ... the lazy flushing doesn't seem to work, myDefrag takes too long, the sync utility doesn't seem to improve anything, and reboot isn't really an option.
Apr 22 2012
Table 5.1 in this article, and some surrounding description, indicate that ntfs converts to upper case when doing directory inserts, so if you want to optimize the disk order for the order processed by directory entry it seems toUpper would be a better choice. http://members.fortunecity.com/clark_kent/exjobb/report3.html A $UpCase Maps lowercase characters to their uppercase version, used when inserting files in a directory. Maybe a more efficient ordering for a folder could be achieved by create zero length files for everything, then read back the ntfs directory entries, and use that order to expand the files.
Apr 22 2012