www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Which option is faster...

reply "jicman" <cabrera__SP M__wrc.xerox.com> writes:
Greetings!

I have this code,

foreach (...)
{

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
}

foreach (...)
{
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
}

thanks.

josé
Aug 05 2013
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08/05/2013 03:59 PM, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

    if (std.string.tolower(fext[0]) == "doc" ||
      std.string.tolower(fext[0]) == "docx" ||
      std.string.tolower(fext[0]) == "xls" ||
      std.string.tolower(fext[0]) == "xlsx" ||
      std.string.tolower(fext[0]) == "ppt" ||
      std.string.tolower(fext[0]) == "pptx")
     continue;
 }

 foreach (...)
 {
    if (std.string.tolower(fext[0]) == "doc")
      continue;
    if (std.string.tolower(fext[0]) == "docx")
      continue;
    if (std.string.tolower(fext[0]) == "xls")
      continue;
    if (std.string.tolower(fext[0]) == "xlsx")
      continue;
    if (std.string.tolower(fext[0]) == "ppt")
      continue;
    if (std.string.tolower(fext[0]) == "pptx")
     continue;
    ...
    ...
 }

 thanks.

 josé
They are both equally slow.
Aug 05 2013
parent reply "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 14:13:33 UTC, Timon Gehr wrote:
 On 08/05/2013 03:59 PM, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }

 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }

 thanks.

 josé
They are both equally slow.
How would you make it faster in D1?
Aug 05 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
jicman:

 How would you make it faster in D1?
Compute std.string.tolower(fext[0]) and put it in a temporary variable. And then compare that variable with all your string literals. In most cases that's fast enough. If it's not enough, you could create a little finite state machine that represents your directed acyclic word graph, and uses gotos to jump around states. The amount of strings is small, so perhaps there are not enough code cache misses to nullify this optimization. Bye, bearophile
Aug 05 2013
parent "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 14:50:27 UTC, bearophile wrote:
 jicman:

 How would you make it faster in D1?
Compute std.string.tolower(fext[0]) and put it in a temporary variable. And then compare that variable with all your string literals. In most cases that's fast enough. If it's not enough, you could create a little finite state machine that represents your directed acyclic word graph, and uses gotos to jump around states. The amount of strings is small, so perhaps there are not enough code cache misses to nullify this optimization. Bye, bearophile
Thanks. This is great.
Aug 05 2013
prev sibling next sibling parent "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,
First option...
 foreach (...)
 {

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }
Second option...
 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }

 thanks.

 josé
So, after I saw this post I asked myself, what? So, the question is: which of the two foreach loops options are faster: 1. The concatenated if || 2. The single if I am trying to see if it matters. I have a project with lots of files and if one is faster, then, it will matter to code it the faster way. Thanks.
Aug 05 2013
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
did you benchmarked your current szenario - how do you know that this is 
the slow part - or are you working on an only-extension-compare-tool?

btw: they are both equal and slow - and full of partly code-duplication
std.string.tolower(fext[0]) multiple times, i hope your list isn't going 
much longer

Am 05.08.2013 15:59, schrieb jicman:
 Greetings!

 I have this code,

 foreach (...)
 {

     if (std.string.tolower(fext[0]) == "doc" ||
       std.string.tolower(fext[0]) == "docx" ||
       std.string.tolower(fext[0]) == "xls" ||
       std.string.tolower(fext[0]) == "xlsx" ||
       std.string.tolower(fext[0]) == "ppt" ||
       std.string.tolower(fext[0]) == "pptx")
      continue;
 }

 foreach (...)
 {
     if (std.string.tolower(fext[0]) == "doc")
       continue;
     if (std.string.tolower(fext[0]) == "docx")
       continue;
     if (std.string.tolower(fext[0]) == "xls")
       continue;
     if (std.string.tolower(fext[0]) == "xlsx")
       continue;
     if (std.string.tolower(fext[0]) == "ppt")
       continue;
     if (std.string.tolower(fext[0]) == "pptx")
      continue;
     ...
     ...
 }

 thanks.

 josé
Aug 05 2013
parent reply "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 14:27:43 UTC, dennis luehring wrote:
 did you benchmarked your current szenario - how do you know 
 that this is the slow part - or are you working on an 
 only-extension-compare-tool?

 btw: they are both equal and slow - and full of partly 
 code-duplication
 std.string.tolower(fext[0]) multiple times, i hope your list 
 isn't going much longer
Ok, how would you make it faster?
Aug 05 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
 Ok, how would you make it faster?
i don't see a better solution here - how to reduce ONE lowercase and SOME compares in any way? (i dont think a hash or something will help) but i know that anything like your continue-party is worth nothing (feels a little bit like script-kiddies "do it with assembler that would it make million times faster" blabla) question: is this the slow part in your project? do you know it for sure or just an emotion - HOW do you benchmark? Am 05.08.2013 16:31, schrieb jicman:
 On Monday, 5 August 2013 at 14:27:43 UTC, dennis luehring wrote:
 did you benchmarked your current szenario - how do you know
 that this is the slow part - or are you working on an
 only-extension-compare-tool?

 btw: they are both equal and slow - and full of partly
 code-duplication
 std.string.tolower(fext[0]) multiple times, i hope your list
 isn't going much longer
Ok, how would you make it faster?
Aug 05 2013
next sibling parent reply "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 14:47:37 UTC, dennis luehring wrote:
 Ok, how would you make it faster?
i don't see a better solution here - how to reduce ONE lowercase and SOME compares in any way? (i dont think a hash or something will help) but i know that anything like your continue-party is worth nothing (feels a little bit like script-kiddies "do it with assembler that would it make million times faster" blabla) question: is this the slow part in your project? do you know it for sure or just an emotion - HOW do you benchmark? Am 05.08.2013 16:31, schrieb jicman:
 On Monday, 5 August 2013 at 14:27:43 UTC, dennis luehring 
 wrote:
 did you benchmarked your current szenario - how do you know
 that this is the slow part - or are you working on an
 only-extension-compare-tool?

 btw: they are both equal and slow - and full of partly
 code-duplication
 std.string.tolower(fext[0]) multiple times, i hope your list
 isn't going much longer
Ok, how would you make it faster?
It is a tool that was a script, but I have turned it into do, which now has taken two hours from the last jscript script. I have not benchmarked it, yet. I may. But I see that a great idea has been provided, which I will use. Thanks for the help.
Aug 05 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 05.08.2013 17:18, schrieb jicman:
 It is a tool that was a script, but I have turned it into do,
 which now has taken two hours from the last jscript script.  I
 have not benchmarked it, yet.  I may.  But I see that a great
 idea has been provided, which I will use.  Thanks for the help.
 have not benchmarked it, yet.  I may.
so its totaly unclear if the presented code is your 2h monster, what was the runtime of your jscript?
 But I see that a great idea has been provided
using a local variable for not lowercasing on each if is not an great idea it is default programming style and i don't think you're be able to implement the tree statemachine when doing such simple performance killer like multiple lowercase calls, and try to help youselfe by introducing "continue"...
Aug 05 2013
parent reply "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 15:27:09 UTC, dennis luehring wrote:
 Am 05.08.2013 17:18, schrieb jicman:
 It is a tool that was a script, but I have turned it into do,
 which now has taken two hours from the last jscript script.  I
 have not benchmarked it, yet.  I may.  But I see that a great
 idea has been provided, which I will use.  Thanks for the help.
 have not benchmarked it, yet.  I may.
so its totaly unclear if the presented code is your 2h monster, what was the runtime of your jscript?
The files are in a network drive, so, that has some slowness already involved because of that. The jscript use to take over 8 hours. The new D program has dropped that to under less than six. This is huge to us. But, I know that I can probably fine tune the program to make it a few minutes less. :-)
 But I see that a great idea has been provided
using a local variable for not lowercasing on each if is not an great idea it is default programming style
This will help. If there are 100K files, which I know that there are more than that, it will help a little bit.
 and i don't think you're be able to implement the tree 
 statemachine
 when doing such simple performance killer like multiple 
 lowercase calls, and try to help youselfe by introducing 
 "continue"...
Perhaps, but it's a good idea, nonetheless.
Aug 05 2013
next sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 08/05/2013 10:04 AM, jicman wrote:

 The files are in a network drive, so, that has some slowness already
 involved because of that.
Which may dominate the running time.
 The jscript use to take over 8 hours.  The new D program has
 dropped that to under less than six.
And that is proof that a sizable amount of time is spent during computations. :)
 But, I know that I can probably fine tune the program to make
 it a few minutes less. :-)
You really should profile where the time is spent. If file I/O is still a big issue, there are WAN optimization products that may solve that problem amazingly efficiently: http://en.wikipedia.org/wiki/WAN_optimization Ali
Aug 05 2013
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 05.08.2013 19:04, schrieb jicman:
 so its totaly unclear if the presented code is your 2h monster,
 what was
 the runtime of your jscript?
The files are in a network drive, so, that has some slowness already involved because of that. The jscript use to take over 8 hours. The new D program has dropped that to under less than six. This is huge to us. But, I know that I can probably fine tune the program to make it a few minutes less. :-)
can you describe more detailed what are you doing - are you also reading these files? why not run your tool on the server and only collect the results over network (wouldn't that be much faster)? what is the impact of using the networkdrive? just copy your biggeste szenario on a local machine an run it against this to get a feeling how painfull the networkdrive communication is
 But I see that a great idea has been provided
using a local variable for not lowercasing on each if is not an great idea it is default programming style
This will help. If there are 100K files, which I know that there are more than that, it will help a little bit.
 and i don't think you're be able to implement the tree
 statemachine
 when doing such simple performance killer like multiple
 lowercase calls, and try to help youselfe by introducing
 "continue"...
Perhaps, but it's a good idea, nonetheless.
maybe your program can be better optimized globaly, but we need more information of what is the programing doing - can you give some psuedo code like: 1. read all filenames recursivley -> 100k filenames 2. reduce down to known extensions -> 10k filenames 3. ... 4. ... 5. ... i don't think that your lower-case is the major part of the 6h, there must be other very slow parts in your project - which you don't find without putting in some bechmarking (and i don't understand why you don't start with benchmarking)
Aug 05 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Aug 05, 2013 at 04:47:36PM +0200, dennis luehring wrote:
 Ok, how would you make it faster?
i don't see a better solution here - how to reduce ONE lowercase and SOME compares in any way? (i dont think a hash or something will help) but i know that anything like your continue-party is worth nothing (feels a little bit like script-kiddies "do it with assembler that would it make million times faster" blabla)
If you really want optimal performance, use std.regex: import std.regex; auto reExtMatch = ctRegex!(`doc|docx|exe|...`, "i"); foreach (...) { if (fext[0].match(reExtMatch)) continue; } The regex is set up to ignore case (the "i" flag), and is compiled at compile-time to generate optimal matching code for the given list of extensions. To get any faster than this, you'll have to hand-optimize or write in assembly. :) T -- Computerese Irregular Verb Conjugation: I have preferences. You have biases. He/She has prejudices. -- Gene Wirchenko
Aug 05 2013
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/5/13, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:
 If you really want optimal performance, use std.regex:
Yes and if you also want to bloat your executable to the point that other parts of the system start slowing down.
Aug 06 2013
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }

 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }

 thanks.

 josé
better: foreach (...) { auto tmp = std.string.tolower(fext[0]); if(tmp == "doc" || tmp == "docx" || tmp == "xls" || tmp == "xlsx" || tmp == "ppt" || tmp == "pptx") { continue; } } but still not super-fast as (unless the compiler is very clever) it still means multiple passes over tmp. Also, it converts the whole string to lower case even when it's not necessary. If you have large numbers of possible matches you will probably want to be clever with your data structures / algorithms. E.g. You could create a tree-like structure to quickly eliminate possibilities as you read successive letters. You read one character, follow the appropriate branch, check if there are any further branches, if not then no match and break. Else, read the next character and follow the appropriate branch and so on.... Infeasible for large (or even medium-sized) character-sets without hashing, but might be pretty fast for a-z and a large number of short strings.
Aug 05 2013
next sibling parent "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 15:18:42 UTC, John Colvin wrote:
 On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

  if (std.string.tolower(fext[0]) == "doc" ||
    std.string.tolower(fext[0]) == "docx" ||
    std.string.tolower(fext[0]) == "xls" ||
    std.string.tolower(fext[0]) == "xlsx" ||
    std.string.tolower(fext[0]) == "ppt" ||
    std.string.tolower(fext[0]) == "pptx")
   continue;
 }

 foreach (...)
 {
  if (std.string.tolower(fext[0]) == "doc")
    continue;
  if (std.string.tolower(fext[0]) == "docx")
    continue;
  if (std.string.tolower(fext[0]) == "xls")
    continue;
  if (std.string.tolower(fext[0]) == "xlsx")
    continue;
  if (std.string.tolower(fext[0]) == "ppt")
    continue;
  if (std.string.tolower(fext[0]) == "pptx")
   continue;
  ...
  ...
 }

 thanks.

 josé
better: foreach (...) { auto tmp = std.string.tolower(fext[0]); if(tmp == "doc" || tmp == "docx" || tmp == "xls" || tmp == "xlsx" || tmp == "ppt" || tmp == "pptx") { continue; } } but still not super-fast as (unless the compiler is very clever) it still means multiple passes over tmp. Also, it converts the whole string to lower case even when it's not necessary. If you have large numbers of possible matches you will probably want to be clever with your data structures / algorithms. E.g. You could create a tree-like structure to quickly eliminate possibilities as you read successive letters. You read one character, follow the appropriate branch, check if there are any further branches, if not then no match and break. Else, read the next character and follow the appropriate branch and so on.... Infeasible for large (or even medium-sized) character-sets without hashing, but might be pretty fast for a-z and a large number of short strings.
As the Hispanic speaking community say, muchas gracias.
Aug 05 2013
prev sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 5 August 2013 at 15:18:42 UTC, John Colvin wrote:
 better:

 foreach (...)
 {
     auto tmp = std.string.tolower(fext[0]);
     if(tmp == "doc" || tmp == "docx"
        || tmp == "xls" || tmp == "xlsx"
        || tmp == "ppt" || tmp == "pptx")
     {
         continue;
     }
 }

 but still not super-fast as (unless the compiler is very 
 clever) it still means multiple passes over tmp. Also, it 
 converts the whole string to lower case even when it's not 
 necessary.

 If you have large numbers of possible matches you will probably 
 want to be clever with your data structures / algorithms. E.g.

 You could create a tree-like structure to quickly eliminate 
 possibilities as you read successive letters. You read one 
 character, follow the appropriate branch, check if there are 
 any further branches, if not then no match and break. Else, 
 read the next character and follow the appropriate branch and 
 so on.... Infeasible for large (or even medium-sized) 
 character-sets without hashing, but might be pretty fast for 
 a-z and a large number of short strings.
Arguably, you'd do even better with: foreach (...) { auto tmp = std.string.tolower(fext[0]); switch(tmp) { case "doc", "docx": case "xls", "xlsx": case "ppt", "pptx": continue; default: } } Since it gives the compiler more wiggle room to optimize things as it wishes. For example, it *could* (who knows :D !) implement the switch as a hash table, or a tree. BTW, a very convenient "tree-like" structure that could be used is a "heap": it is a basic binary tree, but stored inside an array. You could build it during compilation, and then simply search it. A possible optimization is to first switch on string length: foreach (...) { auto tmp = std.string.tolower(fext[0]); switch(tmp.length) { case 3: switch(tmp) { case "doc", "xls", "ppt": continue; default: } break; case 4: switch(tmp) { case "docx", "xlsx", "pptx": continue; default: } break; default: } } That said, I'm not even sure this would be faster, so a bench would be called for. Further more, I'd really be tempted to say that at this point, we are in the realm of premature optimization.
Aug 05 2013
parent "jicman" <cabrera_ _wrc.xerox.com> writes:
On Monday, 5 August 2013 at 17:04:36 UTC, monarch_dodra wrote:
 On Monday, 5 August 2013 at 15:18:42 UTC, John Colvin wrote:
 better:

 foreach (...)
 {
    auto tmp = std.string.tolower(fext[0]);
    if(tmp == "doc" || tmp == "docx"
       || tmp == "xls" || tmp == "xlsx"
       || tmp == "ppt" || tmp == "pptx")
    {
        continue;
    }
 }

 but still not super-fast as (unless the compiler is very 
 clever) it still means multiple passes over tmp. Also, it 
 converts the whole string to lower case even when it's not 
 necessary.

 If you have large numbers of possible matches you will 
 probably want to be clever with your data structures / 
 algorithms. E.g.

 You could create a tree-like structure to quickly eliminate 
 possibilities as you read successive letters. You read one 
 character, follow the appropriate branch, check if there are 
 any further branches, if not then no match and break. Else, 
 read the next character and follow the appropriate branch and 
 so on.... Infeasible for large (or even medium-sized) 
 character-sets without hashing, but might be pretty fast for 
 a-z and a large number of short strings.
Arguably, you'd do even better with: foreach (...) { auto tmp = std.string.tolower(fext[0]); switch(tmp) { case "doc", "docx": case "xls", "xlsx": case "ppt", "pptx": continue; default: } } Since it gives the compiler more wiggle room to optimize things as it wishes. For example, it *could* (who knows :D !) implement the switch as a hash table, or a tree. BTW, a very convenient "tree-like" structure that could be used is a "heap": it is a basic binary tree, but stored inside an array. You could build it during compilation, and then simply search it. A possible optimization is to first switch on string length: foreach (...) { auto tmp = std.string.tolower(fext[0]); switch(tmp.length) { case 3: switch(tmp) { case "doc", "xls", "ppt": continue; default: } break; case 4: switch(tmp) { case "docx", "xlsx", "pptx": continue; default: } break; default: } } That said, I'm not even sure this would be faster, so a bench would be called for. Further more, I'd really be tempted to say that at this point, we are in the realm of premature optimization.
:-) Now that's funny. He he he he... By the way, your coding makes sense. For 100K plus files, it may not mean much, but we are thinking that this amount will be only growing.
Aug 05 2013
prev sibling next sibling parent reply David <d dav1d.de> writes:
Am 05.08.2013 15:59, schrieb jicman:
 
 Greetings!
 
 I have this code,
 
 foreach (...)
 {
 
   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }
 
That's how I would do it. if(["doc", "docx", "xls", "xlsx", "ppt", "pptx"].canFind(fext[0].toLower) { continue; } With the array beeing a runtime/compiletime constant with a sensible name. WORD_EXTENSIONS or something.
Aug 05 2013
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 5 August 2013 at 15:21:25 UTC, David wrote:
 Am 05.08.2013 15:59, schrieb jicman:
 
 Greetings!
 
 I have this code,
 
 foreach (...)
 {
 
   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }
 
That's how I would do it. if(["doc", "docx", "xls", "xlsx", "ppt", "pptx"].canFind(fext[0].toLower) { continue; } With the array beeing a runtime/compiletime constant with a sensible name. WORD_EXTENSIONS or something.
In case some of the strings are actually rather long, one could use toLower lazily (either std.ascii.toLower or std.uni.toLower, depending on which you need): if(WORD_EXT.canFind(fext[0].map!toLower()))
Aug 05 2013
prev sibling next sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }

 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }

 thanks.

 josé
Both are slow, those trying to optimize out tolower are missing the boat. A good compiler should optimize that out. The issue is that n compare are done in those cases with each successive compare getting slower and slower. if the match is doc then it happens in one compare in both cases. If it's pptx then it is 6. So if there are many pptx matches then the code will perform slower than for doc. This can be good or bad behavior depending on if you are guarantee that your order is representative of what actually happens(many doc's few pptx's). To get O(1) behavior, you must use some sort of jump list. A hash table can provide such behavior. If there are many comparisons needed then at some point this will be, on average, faster. If performance is critical and the string length is small enough then the fastest method is to use a direct lookup table. For n character of char. It is 256^n. For n = 4, this is 4GB of memory! (This is one reason to use a hash table). Since you are not using the full ascii you could compact this a bit, say 36^n, for n = 4 it is only 1.6MB of memory. The cost is packing the strings for lookup and computing the index. One way to speed up the loop is to also compare for what is not used. In this case you can compare just on the first char. if (fext[0][0]) == "d") { ... } if (fext[0][0]) == "x") { ... } if (fext[0][0]) == "p") { ... } This pares down the results. In this case it is only 3 comparisions + 2 comparisons = 5 for pptx rather than 6. You can try and develop your own hash to get optimal amortized performance.
Aug 05 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Aug 05, 2013 at 07:30:36PM +0200, JS wrote:
 On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
Greetings!

I have this code,

foreach (...)
{

  if (std.string.tolower(fext[0]) == "doc" ||
    std.string.tolower(fext[0]) == "docx" ||
    std.string.tolower(fext[0]) == "xls" ||
    std.string.tolower(fext[0]) == "xlsx" ||
    std.string.tolower(fext[0]) == "ppt" ||
    std.string.tolower(fext[0]) == "pptx")
   continue;
}

foreach (...)
{
  if (std.string.tolower(fext[0]) == "doc")
    continue;
  if (std.string.tolower(fext[0]) == "docx")
    continue;
  if (std.string.tolower(fext[0]) == "xls")
    continue;
  if (std.string.tolower(fext[0]) == "xlsx")
    continue;
  if (std.string.tolower(fext[0]) == "ppt")
    continue;
  if (std.string.tolower(fext[0]) == "pptx")
   continue;
  ...
  ...
}

thanks.

jos
Both are slow, those trying to optimize out tolower are missing the boat. A good compiler should optimize that out. The issue is that n compare are done in those cases with each successive compare getting slower and slower. if the match is doc then it happens in one compare in both cases. If it's pptx then it is 6. So if there are many pptx matches then the code will perform slower than for doc. This can be good or bad behavior depending on if you are guarantee that your order is representative of what actually happens(many doc's few pptx's). To get O(1) behavior, you must use some sort of jump list. A hash table can provide such behavior. If there are many comparisons needed then at some point this will be, on average, faster.
[...] There's no need to reinvent the wheel here. std.regex.ctRegex will create the equivalent of jump tables for you. Besides, I doubt the OP's real performance bottleneck is in this code; sounds like the program is I/O bound. Compared to I/O, even repeatedly calling std.string.tolower repeatedly is just roundoff error in terms of total execution time. Using a profiler will reveal the *real* bottleneck. Based on past experience, I'm expecting that it will reveal that the bottleneck isn't anywhere we're predicting it is. :) T -- VI = Visual Irritation
Aug 05 2013
prev sibling next sibling parent =?UTF-8?B?UmFwaGHDq2wgSmFrc2U=?= <raphael.jakse gmail.com> writes:
Le 05/08/2013 15:59, jicman a écrit :
 Greetings!

 I have this code,

 foreach (...)
 {

    if (std.string.tolower(fext[0]) == "doc" ||
      std.string.tolower(fext[0]) == "docx" ||
      std.string.tolower(fext[0]) == "xls" ||
      std.string.tolower(fext[0]) == "xlsx" ||
      std.string.tolower(fext[0]) == "ppt" ||
      std.string.tolower(fext[0]) == "pptx")
     continue;
 }

 foreach (...)
 {
    if (std.string.tolower(fext[0]) == "doc")
      continue;
    if (std.string.tolower(fext[0]) == "docx")
      continue;
    if (std.string.tolower(fext[0]) == "xls")
      continue;
    if (std.string.tolower(fext[0]) == "xlsx")
      continue;
    if (std.string.tolower(fext[0]) == "ppt")
      continue;
    if (std.string.tolower(fext[0]) == "pptx")
     continue;
    ...
    ...
 }

 thanks.

 josé
Like others said, writing if( ...) continue; if(...) continue; versus if(... || ...) continue; is not the problem in the code. Computing the lower case of the string for each comparison is a greater problem that can be spotted at the first glance. Storing the value in a variable and then using this variables in comparisons would be better. Both versions of your code should be equivalent, though this should be verified. I would suggest another way of doing this: switch(std.string.tolower(fext[0])) { case "doc": case "docx": case "xls": case "xlsx": case "ppt": case "pptx": continue; default: // do something else } This way of writing your code seems far more readable and elegant to me. Even more concise: switch(std.string.tolower(fext[0])) { case "doc","docx","xls","xlsx","ppt","pptx": continue; default: // do something else } Much like what morarch_dodra proposed, I guess. But notice that tolower is deprecated in current D2 version, toLower should be used instead. If you still use D1, maybe a better thing to do before considering such optimization would be opting for D2. I would also suggest avoiding usage of continue if you don't need it. Here, it is likely that you can write something more structured like: import std.string : toLower // ... foreach(...) { switch(fext[0].toLower) { case "doc","docx","xls","xlsx","ppt","pptx": // do what is to be done here with these cases, or nothing break; default: // do something for other cases } } If you need to use fext[0].toLower in any case, I would suggest you write this kind of code instead: import std.string : toLower // ... auto ext = fext[0].toLower; switch(ext) { case "doc","docx","xls","xlsx","ppt","pptx": // do what is to be done here with these cases, or nothing // use ext instead of fext[0].toLower, if needed break; default: // do something for other cases // use ext instead of fext[0].toLower, if needed } auto ext = fext[0].toLower; should be placed before the foreach loop if fext[0] isn't changed in the loop (avoid computing a value more than one time when it is possible, though thinking before applying this rule or any other rule is not forbidden). There might exist solutions that could be faster, like testing the first letter before the second / using the length of the string / using finite state machine to save tests, but beware of the maintainability of your code and if you go into this, triple check that these solutions are indeed more efficient in terms of execution. See morarch_dodra, JS, bearophile and others posts for more precise information. If this part of your code is not known to be a bottleneck, I would opt for readability / elegance over over-optimization and for this particular case, I would trust the compiler for optimizing enough with the switch version of the code (though it is not more than a opinion here).
Aug 05 2013
prev sibling next sibling parent reply "Andre Artus" <andre.artus gmail.com> writes:
On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }

 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }

 thanks.

 josé
What exactly are you trying to do with this? I get the impression that there is an attempt at "local optimization" when broader approach could lead to better results. For instance. Using the OS's facilities to filter (six requests, one each for "*.doc", "*.docx") could actually end up being a lot faster. If you could give more detail about what you are trying to achieve then it could be possible to get better results.
Aug 05 2013
parent reply "jicman" <cabrera_ _wrc.xerox.com> writes:
On Tuesday, 6 August 2013 at 04:10:57 UTC, Andre Artus wrote:
 On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

  if (std.string.tolower(fext[0]) == "doc" ||
    std.string.tolower(fext[0]) == "docx" ||
    std.string.tolower(fext[0]) == "xls" ||
    std.string.tolower(fext[0]) == "xlsx" ||
    std.string.tolower(fext[0]) == "ppt" ||
    std.string.tolower(fext[0]) == "pptx")
   continue;
 }

 foreach (...)
 {
  if (std.string.tolower(fext[0]) == "doc")
    continue;
  if (std.string.tolower(fext[0]) == "docx")
    continue;
  if (std.string.tolower(fext[0]) == "xls")
    continue;
  if (std.string.tolower(fext[0]) == "xlsx")
    continue;
  if (std.string.tolower(fext[0]) == "ppt")
    continue;
  if (std.string.tolower(fext[0]) == "pptx")
   continue;
  ...
  ...
 }

 thanks.

 josé
What exactly are you trying to do with this? I get the impression that there is an attempt at "local optimization" when broader approach could lead to better results. For instance. Using the OS's facilities to filter (six requests, one each for "*.doc", "*.docx") could actually end up being a lot faster. If you could give more detail about what you are trying to achieve then it could be possible to get better results.
The files are in a network drive and doing a list foreach *.doc, *.docx, etc. will be more expensive than getting the list of all the files at once and then processing them accordingly.
Aug 06 2013
next sibling parent reply "Andre Artus" <andre.artus gmail.com> writes:
On Tuesday, 6 August 2013 at 12:32:13 UTC, jicman wrote:
 On Tuesday, 6 August 2013 at 04:10:57 UTC, Andre Artus wrote:
 On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

 if (std.string.tolower(fext[0]) == "doc" ||
   std.string.tolower(fext[0]) == "docx" ||
   std.string.tolower(fext[0]) == "xls" ||
   std.string.tolower(fext[0]) == "xlsx" ||
   std.string.tolower(fext[0]) == "ppt" ||
   std.string.tolower(fext[0]) == "pptx")
  continue;
 }

 foreach (...)
 {
 if (std.string.tolower(fext[0]) == "doc")
   continue;
 if (std.string.tolower(fext[0]) == "docx")
   continue;
 if (std.string.tolower(fext[0]) == "xls")
   continue;
 if (std.string.tolower(fext[0]) == "xlsx")
   continue;
 if (std.string.tolower(fext[0]) == "ppt")
   continue;
 if (std.string.tolower(fext[0]) == "pptx")
  continue;
 ...
 ...
 }

 thanks.

 josé
What exactly are you trying to do with this? I get the impression that there is an attempt at "local optimization" when broader approach could lead to better results. For instance. Using the OS's facilities to filter (six requests, one each for "*.doc", "*.docx") could actually end up being a lot faster. If you could give more detail about what you are trying to achieve then it could be possible to get better results.
The files are in a network drive and doing a list foreach *.doc, *.docx, etc. will be more expensive than getting the list of all the files at once and then processing them accordingly.
Again, what are you trying to achieve? Your statement is not necessarily true, for a myriad of reasons, but it entirely depends on what you want to do. I would reiterate Dennis Luehring's reply, why are you not benching? It seems like you are guessing at what the problems are, that's hardly ever useful. One of the first rules of network optimization is to reduce the amount od data, that normally means filtering.at the server, the next thing is coarse grained is better than fine (BOCTAOE/L).
Aug 06 2013
parent reply "jicman" <cabrera wrc.xerox.com> writes:
On Tuesday, 6 August 2013 at 14:49:42 UTC, Andre Artus wrote:
 On Tuesday, 6 August 2013 at 12:32:13 UTC, jicman wrote:
 On Tuesday, 6 August 2013 at 04:10:57 UTC, Andre Artus wrote:
 On Monday, 5 August 2013 at 13:59:24 UTC, jicman wrote:
 Greetings!

 I have this code,

 foreach (...)
 {

 if (std.string.tolower(fext[0]) == "doc" ||
  std.string.tolower(fext[0]) == "docx" ||
  std.string.tolower(fext[0]) == "xls" ||
  std.string.tolower(fext[0]) == "xlsx" ||
  std.string.tolower(fext[0]) == "ppt" ||
  std.string.tolower(fext[0]) == "pptx")
 continue;
 }

 foreach (...)
 {
 if (std.string.tolower(fext[0]) == "doc")
  continue;
 if (std.string.tolower(fext[0]) == "docx")
  continue;
 if (std.string.tolower(fext[0]) == "xls")
  continue;
 if (std.string.tolower(fext[0]) == "xlsx")
  continue;
 if (std.string.tolower(fext[0]) == "ppt")
  continue;
 if (std.string.tolower(fext[0]) == "pptx")
 continue;
 ...
 ...
 }

 thanks.

 josé
What exactly are you trying to do with this? I get the impression that there is an attempt at "local optimization" when broader approach could lead to better results. For instance. Using the OS's facilities to filter (six requests, one each for "*.doc", "*.docx") could actually end up being a lot faster. If you could give more detail about what you are trying to achieve then it could be possible to get better results.
The files are in a network drive and doing a list foreach *.doc, *.docx, etc. will be more expensive than getting the list of all the files at once and then processing them accordingly.
Again, what are you trying to achieve? Your statement is not necessarily true, for a myriad of reasons, but it entirely depends on what you want to do. I would reiterate Dennis Luehring's reply, why are you not benching? It seems like you are guessing at what the problems are, that's hardly ever useful. One of the first rules of network optimization is to reduce the amount od data, that normally means filtering.at the server, the next thing is coarse grained is better than fine (BOCTAOE/L).
It's a long story and I will return in a few months and give you the whole story, but right now, time is not on my side. I have answers for all the questions you folks have asked, and I appreciate all the input. I have the answer that I was looking for, so in a few months, I will come back and explain the whole story. Thanks for all the response and suggestions. jic
Aug 06 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 07.08.2013 06:30, schrieb jicman:
 Again, what are you trying to achieve?
 Your statement is not necessarily true, for a  myriad of
 reasons, but it entirely depends on what you want to do.
 I would reiterate Dennis Luehring's reply, why are you not
 benching? It seems like you are guessing at what the problems
 are, that's hardly ever useful.
 One of the first rules of network optimization  is to reduce
 the amount od data, that normally means filtering.at the
 server, the next thing is coarse grained is better than fine
 (BOCTAOE/L).
It's a long story and I will return in a few months and give you the whole story, but right now, time is not on my side. I have answers for all the questions you folks have asked, and I appreciate all the input. I have the answer that I was looking for, so in a few months, I will come back and explain the whole story. Thanks for all the response and suggestions.
after makeing us girls all wet to help you - your reply is "no sex on the first date, im a gentlemen... but maybe in a few months" so: you having a jscript doing somehting with files,fileextensions over networkdrive - it runs around 8h you ported that jscript to D - now it runs for 6h you noob-guessed the lowercase-if-party could be evil (btw: it cost more time to guess then to benchmark) you get trivial answers that won't get you very much, the lowercase would not boost your speed that much and the networkdrive latency will kill all the other statemachine ideas you don't answer trivial questions about the big picture - and now you're out of time open questions: -why not collect the data on the server itself - instead of grabbing tiny bits over network? - this is for understanding your environent -how big is the speed drop with your tool on the very same drive localy and over a networkdrive? - this is for understanding the latency -are you also reading this files or just doing filename search (recursively?) and throwing out non office-extensions? this is for getting an idea if buildin OS(operating system) features can help see you in a few months
Aug 06 2013
parent "Andre Artus" <andre.artus gmail.com> writes:
 It's a long story and I will return in a few months and give 
 you
 the whole story, but right now, time is not on my side.  I have
 answers for all the questions you folks have asked, and I
 appreciate all the input.  I have the answer that I was looking
 for, so in a few months, I will come back and explain the whole
 story.  Thanks for all the response and suggestions.
after makeing us girls all wet to help you - your reply is "no sex on the first date, im a gentlemen... but maybe in a few months" so: you having a jscript doing somehting with files,fileextensions over networkdrive - it runs around 8h you ported that jscript to D - now it runs for 6h you noob-guessed the lowercase-if-party could be evil (btw: it cost more time to guess then to benchmark) you get trivial answers that won't get you very much, the lowercase would not boost your speed that much and the networkdrive latency will kill all the other statemachine ideas you don't answer trivial questions about the big picture - and now you're out of time open questions: -why not collect the data on the server itself - instead of grabbing tiny bits over network? - this is for understanding your environent
Just for a sanity check I implemented a quick client-server setup where the daemon takes filespec from the client and returns a line-by-line list compressed into one packet. The total running time on 8 terabytes of files stored over a dozen drives searched recursively: less than 1 minute. Same over slow WiFi, negligible difference (list compresses to a few Kb) with LZMA. I did not even bother to search each physical drive separately, just produced the list sequentially.
 -how big is the speed drop with your tool on the very same 
 drive localy and over a networkdrive? - this is for 
 understanding the latency

 -are you also reading this files or just doing filename search 
 (recursively?) and throwing out non office-extensions?
 this is for getting an idea if buildin OS(operating system) 
 features can help

 see you in a few months
It's impossible to help people who refuse to give basic information.
Aug 06 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Aug 06, 2013 at 02:32:11PM +0200, jicman wrote:
 On Tuesday, 6 August 2013 at 04:10:57 UTC, Andre Artus wrote:
[...]
What exactly are you trying to do with this? I get the impression
that there is an attempt at "local optimization" when broader
approach could lead to better results.

For instance. Using the OS's facilities to filter (six requests,
one each for "*.doc", "*.docx") could actually end up being a lot
faster.

If you could give more detail about what you are trying to achieve
then it could be possible to get better results.
The files are in a network drive and doing a list foreach *.doc, *.docx, etc. will be more expensive than getting the list of all the files at once and then processing them accordingly.
In this case, the bottleneck most likely is in the network latency, so optimizing string comparisons won't get you very far. You should instead be focusing on how to improve the performance of network accesses. What kind of processing are you doing with the filtered list of files? If it's a complicated operation, you might want to consider having a separate thread for fetching the list of files while work is being done on them in the main thread. T -- Computers shouldn't beep through the keyhole.
Aug 06 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Aug 05, 2013 at 03:59:23PM +0200, jicman wrote:
 
 Greetings!
 
 I have this code,
 
 foreach (...)
 {
 
   if (std.string.tolower(fext[0]) == "doc" ||
     std.string.tolower(fext[0]) == "docx" ||
     std.string.tolower(fext[0]) == "xls" ||
     std.string.tolower(fext[0]) == "xlsx" ||
     std.string.tolower(fext[0]) == "ppt" ||
     std.string.tolower(fext[0]) == "pptx")
    continue;
 }
 
 foreach (...)
 {
   if (std.string.tolower(fext[0]) == "doc")
     continue;
   if (std.string.tolower(fext[0]) == "docx")
     continue;
   if (std.string.tolower(fext[0]) == "xls")
     continue;
   if (std.string.tolower(fext[0]) == "xlsx")
     continue;
   if (std.string.tolower(fext[0]) == "ppt")
     continue;
   if (std.string.tolower(fext[0]) == "pptx")
    continue;
   ...
   ...
 }
[...] It would appear that your bottleneck is not in the continue statements, but in the repeated calls to std.string.tolower. It's probably better to write it this way: foreach (...) { auto ext = std.string.tolower(fext[0]); if (ext == "doc" || ext == "docx" || ... ) continue; } This way you save on the overhead of many identical function calls, which probably outweights any benefit you get by optimizing continue. T -- It's amazing how careful choice of punctuation can leave you hanging:
Aug 05 2013
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, August 05, 2013 15:59:23 jicman wrote:
 Greetings!
 
 I have this code,
 
 foreach (...)
 {
 
    if (std.string.tolower(fext[0]) == "doc" ||
      std.string.tolower(fext[0]) == "docx" ||
      std.string.tolower(fext[0]) == "xls" ||
      std.string.tolower(fext[0]) == "xlsx" ||
      std.string.tolower(fext[0]) == "ppt" ||
      std.string.tolower(fext[0]) == "pptx")
     continue;
 }
 
 foreach (...)
 {
    if (std.string.tolower(fext[0]) == "doc")
      continue;
    if (std.string.tolower(fext[0]) == "docx")
      continue;
    if (std.string.tolower(fext[0]) == "xls")
      continue;
    if (std.string.tolower(fext[0]) == "xlsx")
      continue;
    if (std.string.tolower(fext[0]) == "ppt")
      continue;
    if (std.string.tolower(fext[0]) == "pptx")
     continue;
    ...
    ...
 }
As others have pointed out, the two code snippets are semantically identical, and there's a decent chance that the compiler will generate identical code for both. It wouldn't surprise me if the first snippet resulted in more concise assembly code, but there shouldn't be any performance difference between the two. But the first snippet is cleaner, so between those two, you should use that. Of course, as others have pointed out, it would be far more efficient to simply calculate std.string.tolower(fext[0]) once and re-use the result, which should be a very large performance gain here (assuming that the code around it doesn't dwarf it in cost). However, another possibility that I don't think anyone has pointed out is std.string.icmp, which will do case-insensitive comparison, meaning that you could do if(fext[0].icmp("doc") == 0 || fext[0].icmp("docx") == 0 || ...) and that might be more efficient as it avoids having to allocate a new string when fext[0] isn't all lowercase already. However, you would have to benchmark the code to be sure, since depending on the typical input, the code around this code, and how many times this code gets called, calling toLower once could be more efficient or using icmp could be more efficient, since toLower allocates whereas icmp doesn't, but icmp does more comparisons that a simple ==. But if you really care about efficiency, you should try both ways and see which is faster for your use case. - Jonathan M Davis
Aug 05 2013