www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - tolf and detab

reply Walter Bright <newshound2 digitalmars.com> writes:
I wrote these two trivial utilities for the purpose of canonicalizing source 
code before checkins and to deal with FreeBSD's inability to deal with CRLF
line 
endings, and because I can never figure out the right settings for git to make 
it do the canonicalization.

tolf - converts LF, CR, and CRLF line endings to LF.

detab - converts all tabs to the correct number of spaces. Assumes tabs are 8 
column tabs. Removes trailing whitespace from lines.

Posted here just in case someone wonders what they are.
---------------------------------------------------------
/* Replace tabs with spaces, and remove trailing whitespace from lines.
  */

import std.file;
import std.path;

int main(string[] args)
{
     foreach (f; args[1 .. $])
     {
         auto input = cast(char[]) std.file.read(f);
         auto output = filter(input);
         if (output != input)
             std.file.write(f, output);
     }
     return 0;
}


char[] filter(char[] input)
{
     char[] output;
     size_t j;

     int column;
     for (size_t i = 0; i < input.length; i++)
     {
         auto c = input[i];

         switch (c)
         {
             case '\t':
                 while ((column & 7) != 7)
                 {   output ~= ' ';
                     j++;
                     column++;
                 }
                 c = ' ';
                 column++;
                 break;

             case '\r':
             case '\n':
                 while (j && output[j - 1] == ' ')
                     j--;
                 output = output[0 .. j];
                 column = 0;
                 break;

             default:
                 column++;
                 break;
         }
         output ~= c;
         j++;
     }
     while (j && output[j - 1] == ' ')
         j--;
     return output[0 .. j];
}
-----------------------------------------------------
/* Replace line endings with LF
  */

import std.file;
import std.path;

int main(string[] args)
{
     foreach (f; args[1 .. $])
     {
         auto input = cast(char[]) std.file.read(f);
         auto output = filter(input);
         if (output != input)
             std.file.write(f, output);
     }
     return 0;
}


char[] filter(char[] input)
{
     char[] output;
     size_t j;

     for (size_t i = 0; i < input.length; i++)
     {
         auto c = input[i];

         switch (c)
         {
             case '\r':
                 c = '\n';
                 break;

             case '\n':
                 if (i && input[i - 1] == '\r')
                     continue;
                 break;

             case 0:
                 continue;

             default:
                 break;
         }
         output ~= c;
         j++;
     }
     return output[0 .. j];
}
------------------------------------------
Aug 06 2010
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/06/2010 08:34 PM, Walter Bright wrote:
 I wrote these two trivial utilities for the purpose of canonicalizing
 source code before checkins and to deal with FreeBSD's inability to deal
 with CRLF line endings, and because I can never figure out the right
 settings for git to make it do the canonicalization.

 tolf - converts LF, CR, and CRLF line endings to LF.

 detab - converts all tabs to the correct number of spaces. Assumes tabs
 are 8 column tabs. Removes trailing whitespace from lines.

 Posted here just in case someone wonders what they are.

Nice, though they don't account for multiline string literals. A good exercise would be rewriting these tools in idiomatic D2 and assess the differences. Andrei
Aug 06 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Andrej Mitrovic wrote:
 Or improve your google-fu by finding some existing tools that do the job 
 right. :)

Sure, but I suspect it's faster to write the utility! After all, they are trivial.
Aug 06 2010
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Andrei Alexandrescu wrote:
 A good exercise would be rewriting these tools in idiomatic D2 and 
 assess the differences.

Some D2-fu would be cool. Any takers?
Aug 06 2010
prev sibling next sibling parent reply "Yao G." <nospamyao gmail.com> writes:
What does idiomatic D means?

On Fri, 06 Aug 2010 20:50:52 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 08/06/2010 08:34 PM, Walter Bright wrote:
 I wrote these two trivial utilities for the purpose of canonicalizing
 source code before checkins and to deal with FreeBSD's inability to deal
 with CRLF line endings, and because I can never figure out the right
 settings for git to make it do the canonicalization.

 tolf - converts LF, CR, and CRLF line endings to LF.

 detab - converts all tabs to the correct number of spaces. Assumes tabs
 are 8 column tabs. Removes trailing whitespace from lines.

 Posted here just in case someone wonders what they are.

Nice, though they don't account for multiline string literals. A good exercise would be rewriting these tools in idiomatic D2 and assess the differences. Andrei

-- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Aug 06 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/06/2010 09:33 PM, Yao G. wrote:
 What does idiomatic D means?

At a quick glance - I'm thinking two elements would be using string and possibly byLine. Andrei
Aug 06 2010
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"Yao G." <nospamyao gmail.com> wrote in message 
news:op.vg1qpcjfxeuu2f miroslava.gateway.2wire.net...
 What does idiomatic D means?

"idiomatic D" -> "In typical D style"
Aug 06 2010
prev sibling next sibling parent reply Jonathan M Davis <jmdavisprog gmail.com> writes:
On Friday 06 August 2010 18:50:52 Andrei Alexandrescu wrote:
 On 08/06/2010 08:34 PM, Walter Bright wrote:
 I wrote these two trivial utilities for the purpose of canonicalizing
 source code before checkins and to deal with FreeBSD's inability to deal
 with CRLF line endings, and because I can never figure out the right
 settings for git to make it do the canonicalization.
 
 tolf - converts LF, CR, and CRLF line endings to LF.
 
 detab - converts all tabs to the correct number of spaces. Assumes tabs
 are 8 column tabs. Removes trailing whitespace from lines.
 
 Posted here just in case someone wonders what they are.

[snip] Nice, though they don't account for multiline string literals. A good exercise would be rewriting these tools in idiomatic D2 and assess the differences. Andrei

I didn't try and worry about multiline string literals, but here are my more idiomatic solutions: detab: /* Replace tabs with spaces, and remove trailing whitespace from lines. */ import std.conv; import std.file; import std.stdio; import std.string; void main(string[] args) { const int tabSize = to!int(args[1]); foreach(f; args[2 .. $]) removeTabs(tabSize, f); } void removeTabs(int tabSize, string fileName) { auto file = File(fileName); string[] output; foreach(line; file.byLine()) { int lastTab = 0; while(lastTab != -1) { const int tab = line.indexOf('\t'); if(tab == -1) break; const int numSpaces = tabSize - tab % tabSize; line = line[0 .. tab] ~ repeat(" ", numSpaces) ~ line[tab + 1 .. $]; lastTab = tab + numSpaces; } output ~= line.idup; } std.file.write(fileName, output.join("\n")); } ------------------------------------------- The three differences between mine and Walter's are that mine takes the tab size as the first argumen,t it doesn't put a newline at the end of the file, and it writes the file even if it changed (you could test for that, but when using byLine(), it's a bit harder). Interestingly enough, from the few tests that I ran, mine seems to be somewhat faster. I also happen to think that the code is clearer (it's certainly shorter), though that might be up for debate. ------------------------------------------- tolf: /* Replace line endings with LF */ import std.file; import std.string; void main(string[] args) { foreach(f; args[1 .. $]) fixEndLines(f); } void fixEndLines(string fileName) { auto fileStr = std.file.readText(fileName); auto result = fileStr.replace("\r\n", "\n").replace("\r", "\n"); std.file.write(fileName, result); } ------------------------------------------- This version is ludicrously simple. And it was also faster than Walter's in the few tests that I ran. In either case, I think that it is definitely clearer code. I would have thought that being more idomatic would have resulted in slower code than what Walter did, but interestingly enough, both programs are faster with my code. They might take more memory though. I'm not quite sure how to check that. In any cases, you wanted some idiomatic D2 solutions, so there you go. - Jonathan M Davis
Aug 07 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jonathan M Davis:
 I would have thought that being more idomatic would have resulted in slower
code 
 than what Walter did, but interestingly enough, both programs are faster with
my 
 code. They might take more memory though. I'm not quite sure how to check
that. 
 In any cases, you wanted some idiomatic D2 solutions, so there you go.

Your code looks better. My (probably controversial) opinion on this is that the idiomatic D solution for those text "scripts" is to use a scripting language, as Python :-) In this case a Python version is more readable, shorter and probably faster too because reading the lines of a _normal_ text file is faster in Python compared to D (because Python is more optimized for such purposes. I can show benchmarks on request). On the other hand D2 is in its debugging phase, so it's good to use it even for purposes it's not the best language for, to catch bugs or performance bugs. So I think it's positive to write such scripts in D2, even if in a real-world setting I want to use Python to write them. Bye, bearophile
Aug 07 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/07/2010 11:16 PM, bearophile wrote:
 Jonathan M Davis:
 I would have thought that being more idomatic would have resulted
 in slower code than what Walter did, but interestingly enough, both
 programs are faster with my code. They might take more memory
 though. I'm not quite sure how to check that. In any cases, you
 wanted some idiomatic D2 solutions, so there you go.

Your code looks better. My (probably controversial) opinion on this is that the idiomatic D solution for those text "scripts" is to use a scripting language, as Python :-) In this case a Python version is more readable, shorter and probably faster too because reading the lines of a _normal_ text file is faster in Python compared to D (because Python is more optimized for such purposes. I can show benchmarks on request). On the other hand D2 is in its debugging phase, so it's good to use it even for purposes it's not the best language for, to catch bugs or performance bugs. So I think it's positive to write such scripts in D2, even if in a real-world setting I want to use Python to write them.

I think it's worth targeting D2 to tasks that are usually handled by scripting languages. I've done a lot of that and it beats the hell out of rewriting in D a script that's grown out of control Andrei
Aug 07 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/07/2010 11:16 PM, bearophile wrote:
 In this case a Python version is more readable, shorter and probably
 faster too because reading the lines of a _normal_ text file is
 faster in Python compared to D (because Python is more optimized for
 such purposes. I can show benchmarks on request).

That would be great so we can tune our approach. Thanks! Andrei
Aug 07 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 This makes me think we should have a range that detects and replaces 
 patterns lazily and on the fly.

In Python there is a helper module: http://docs.python.org/library/fileinput.html
 I think it's worth targeting D2 to tasks that are usually handled by
 scripting languages. I've done a lot of that and it beats the hell out
 of rewriting in D a script that's grown out of control

Dynamic languages are handy but they require some rigour when you program. Python is probably unfit to write one million lines long programs, but if you train yourself a little and keep your code clean, you usually become able to write clean larghish programs in Python.
 That would be great so we can tune our approach. Thanks!

In my dlibs I have the xio module that read by lines efficiently, it was faster than the iterating on the lines of BufferedFile. There are tons of different benchmarks that you may use, but a simple one to start is better, one that just iterates the file lines. See below. Related: experiments have shown that the (oldish) Java GC improves its performance if it is able to keep strings (that are immutable) in a separate memory pool, _and_ be able to recognize duplicated strings, of course keeping only one string for each equality set. It's positive to do a similar experiment with the D GC, but first you need applications that use the GC to test if this idea is an improvement :-) So I have used a minimal benchmark: -------------------------- # Python 2.7 code from sys import argv def process(file_name): total = 0 for line in open(file_name): total += len(line) return total print "Total:", process(argv[1]) -------------------------- // D2 code import std.stdio: File, writeln; int process(string fileName) { int total = 0; auto file = File(fileName); foreach (rawLine; file.byLine()) { string line = rawLine.idup; total += line.length; } file.close(); return total; } void main(string[] args) { if (args.length == 2) writeln("Total: ", process(args[1])); } -------------------------- In the D code I have added an idup to make the comparison more fair, because in the Python code the "line" is a true newly allocated line, you can safely use it as dictionary key. I have used Python 2.7 with no Psyco JIT (http://psyco.sourceforge.net/ ) to speed up the Python code because it's not available yet for Python 2.7. D code compiled with dmd 2.047, optimized build. As test text data I have used a concatenation of all text files here (they are copyrighted, but freely usable): http://gnosis.cx/TPiP/ The result on Windows is a file of 1_116_552 bytes. I have attached the file to itself, duplicating its length, some times, the result is a file of 71_459_328 bytes (this is not an fully realistic case because you often have many small files to read instead of a very large one). The timings are taken with warm disk cache, so they are essentially read from RAM. This is not fully realistic, but if you want to write a benchmark you have to do this, because for me it's very hard on Windows to make sure that the disk cache is fully empty. So it's better to do the opposite and benchmark a warm file. The output of the Python code is: Total: 69789888 Found in 0.88 seconds (best of 6, the variance is minimal). The output of the D code is: Total: 69789888 Found in 1.28 seconds (best of 6, minimal variance). If in the D2 code I comment out the idup like this: foreach (rawLine; file.byLine()) { total += rawLine.length; } The output of the D code without idup is: Total: 69789888 Found in 0.75 seconds (best of 6, minimal variance). As you see it's a matter of GC efficiency too. Beside the GC the cause of the higher performance of the Python code comes from a tuned design, you can see the function getline_via_fgets here: http://svn.python.org/view/python/trunk/Objects/fileobject.c?revision=81275&view=markup It uses a "stack buffer" (char buf[MAXBUFSIZE]; where MAXBUFSIZE is 300) too. Bye, bearophile
Aug 08 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:
 In the D code I have added an idup to make the comparison more fair, because
 in the Python code the "line" is a true newly allocated line, you can safely
 use it as dictionary key.

So it is with byLine, too. You've burdened D with double the amount of allocations. Also, I object in general to this method of making things "more fair". Using a less efficient approach in X because Y cannot use such an approach is not a legitimate comparison.
Aug 08 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound2 digitalmars.com> wrote in message 
news:i3mpnb$2hcf$1 digitalmars.com...
 bearophile wrote:
 In the D code I have added an idup to make the comparison more fair, 
 because
 in the Python code the "line" is a true newly allocated line, you can 
 safely
 use it as dictionary key.

So it is with byLine, too. You've burdened D with double the amount of allocations.

I thought byLine just re-uses the same buffer each time?
Aug 08 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:
 bearophile wrote:
 In the D code I have added an idup to make the comparison more fair, because
 in the Python code the "line" is a true newly allocated line, you can safely
 use it as dictionary key.

So it is with byLine, too. You've burdened D with double the amount of allocations.

I think you are wrong two times: 1) byLine() doesn't return a newly allocated line, you can see it with this small program: import std.stdio: File, writeln; void main(string[] args) { char[][] lines; auto file = File(args[1]); foreach (rawLine; file.byLine()) { writeln(rawLine.ptr); lines ~= rawLine; } file.close(); } Its output shows that all "strings" (char[]) share the same pointer: 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 ... 2) You can't use the result of rawLine() as string key for an associative array, as you I have said you can in Python. Currently you can, but according to Andrei this is a bug. And if it's not a bug then I'll reopen this closed bug 4474: http://d.puremagic.com/issues/show_bug.cgi?id=4474
 Also, I object in general to this method of making things "more fair". Using a
 less efficient approach in X because Y cannot use such an approach is not a
 legitimate comparison.

I generally agree, but this it not the case. In some situations you indeed don't need a newly allocated string for each loop, because for example you just want to read them and process them and not change/store them. You can't do this in Python, but this is not what I want to test. As I have explained in bug 4474 this behaviour is useful but it is acceptable only if explicitly requested by the programmer, and not as default one. The language is safe, as Andrei explains there, because you are supposed to idup the char[] to use it as key for an associative array (if your associative array is declared as int[char[]] then it can accept such rawLine() as keys, but you can clearly see those aren't strings. This is why I have closed bug 4474). Bye, bearophile
Aug 08 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:
 Walter Bright:
 bearophile wrote:
 In the D code I have added an idup to make the comparison more fair,
 because in the Python code the "line" is a true newly allocated line, you
 can safely use it as dictionary key.

allocations.

I think you are wrong two times: 1) byLine() doesn't return a newly allocated line, you can see it with this small program: import std.stdio: File, writeln; void main(string[] args) { char[][] lines; auto file = File(args[1]); foreach (rawLine; file.byLine()) { writeln(rawLine.ptr); lines ~= rawLine; } file.close(); } Its output shows that all "strings" (char[]) share the same pointer: 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 ...

eh, you're right. the phobos documentation for byLine needs to be fixed.
 You can't do this in Python, but this is not what I want to test.

If you want to conclude that Python is better at processing files, you need to show it using each language doing it a way well suited to that language, rather than burdening one so it uses the same method as the less powerful one.
Aug 08 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:
 If you want to conclude that Python is better at processing files, you need to 
 show it using each language doing it a way well suited to that language,
rather 
 than burdening one so it uses the same method as the less powerful one.

byLine() yields a char[], so if you want to do most kinds of strings processing or you want to store the line (or parts of it), you have to idup it. So in this case Python is not significantly less powerful than D. You can of course use the raw char[], but then you lose the advantages advertised when you have introduced the safer immutable D2 strings. And in many situations you have to dup the char[] anyway, otherwise your have all kinds of bugs, that Python lacks. In D1 to avoid it I used to use dup more often than necessary. I have explained this in the bug 4474. In this newsgroup my purpose it to show D faults, suggest improvements, etc. In this case my purpose was just to show that byLine()+idup is slow. And you have to thankful for my benchmarks. In my dlibs1 for D1 I have a xio module that reads files by line that is faster than iterating on a BufferedFile, so it's not a limit of the language, it's Phobos that has a performance bug that can be improved. Bye, bearophile
Aug 08 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
 so it's not a limit of the language, it's Phobos that has a performance bug
that can be improved.

I don't know where the performance bug is, maybe it's a matter of GC, not a Phobos performance bug. Bye, bearophile
Aug 08 2010
prev sibling next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Yao G.:
 <g> What's next? Will you demand attribution like the time Andrei  
 presented the ranges design?

Of course. In the end all D will be mine <evil laugh with echo effects> :-) Bye, bearophile
Aug 08 2010
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 04:44 PM, bearophile wrote:
 Walter Bright:
 If you want to conclude that Python is better at processing files, you need to
 show it using each language doing it a way well suited to that language, rather
 than burdening one so it uses the same method as the less powerful one.

byLine() yields a char[], so if you want to do most kinds of strings processing or you want to store the line (or parts of it), you have to idup it. So in this case Python is not significantly less powerful than D. You can of course use the raw char[], but then you lose the advantages advertised when you have introduced the safer immutable D2 strings. And in many situations you have to dup the char[] anyway, otherwise your have all kinds of bugs, that Python lacks. In D1 to avoid it I used to use dup more often than necessary. I have explained this in the bug 4474. In this newsgroup my purpose it to show D faults, suggest improvements, etc. In this case my purpose was just to show that byLine()+idup is slow. And you have to thankful for my benchmarks. In my dlibs1 for D1 I have a xio module that reads files by line that is faster than iterating on a BufferedFile, so it's not a limit of the language, it's Phobos that has a performance bug that can be improved.

Thanks for your analysis. Where does xio derive its performance advantage from? Andrei
Aug 08 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei:

Where does xio derive its performance advantage from?<

I'd like to give you a good answer, but I can't. dlibs1 (that you can found online still) has a Python Licence, so to create xio.xfile() I have just translated to D1 the C code of the CPython implementation code of the file object I have already linked here. I think it minimizes heap allocations, the performance is tuned for a line length found to be the "average one" for normal files. So I presume if your text file has very short lines (like 5 chars each) or very long ones (like 1000 chars each) it becomes less efficient. So it's probably a matter of good usage of the C I/O functions and probably a more efficient management by the GC. Phobos is Boost Licence, but I don't think Python devs can get mad if you take a look at how Python reads lines lazily :-) Someone has tried to implement a Python-style associative array in a similar way. Bye, bearophile
Aug 08 2010
parent reply Kagamin <spam here.lot> writes:
bearophile Wrote:

 I think it minimizes heap allocations, the performance is tuned for a line
length found to be the "average one" for normal files. So I presume if your
text file has very short lines (like 5 chars each) or very long ones (like 1000
chars each) it becomes less efficient.
 
 So it's probably a matter of good usage of the C I/O functions and probably a
more efficient management by the GC.
 

Aug 08 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Kagamin:

 Don't you minimize heap allocation etc by reading whole file in one io call?

The whole thread was about lazy read of file lines. If the file is very large it's not wise to load it all in RAM at once. Bye, bearophile
Aug 09 2010
next sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2010-08-09 07:12:38 -0400, bearophile <bearophileHUGS lycos.com> said:

 Kagamin:
 
 Don't you minimize heap allocation etc by reading whole file in one io call?

The whole thread was about lazy read of file lines. If the file is very large it's not wise to load it all in RAM at once.

For non-huge files that can fit in the memory space, I'd just memory-map the whole file and treat it as a giant string that I could then slice and keep the slices around (yeah!). The virtual memory system will take care of loading the file content's as you read from its memory space, so the file isn't loaded all at once. But that's not compatible with the C file IO functions. Does Python uses C file IO calls when reading from a file? If not, perhaps that's why it's faster. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Aug 09 2010
prev sibling parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Monday, August 09, 2010 05:30:33 Michel Fortin wrote:
 On 2010-08-09 07:12:38 -0400, bearophile <bearophileHUGS lycos.com> said:
 Kagamin:
 Don't you minimize heap allocation etc by reading whole file in one io
 call?

The whole thread was about lazy read of file lines. If the file is very large it's not wise to load it all in RAM at once.

For non-huge files that can fit in the memory space, I'd just memory-map the whole file and treat it as a giant string that I could then slice and keep the slices around (yeah!). The virtual memory system will take care of loading the file content's as you read from its memory space, so the file isn't loaded all at once. But that's not compatible with the C file IO functions. Does Python uses C file IO calls when reading from a file? If not, perhaps that's why it's faster.

Well, you can just read the whole file in as a string with readText(), and any slices to that could stick around, but presumably, that's using the C file I/O calls underneath. - Jonathan M Davis
Aug 09 2010
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 04:48 PM, Andrej Mitrovic wrote:
 Andrei used to!string() in an early example in TDPL for some
 line-by-line processing. I'm not sure of the advantages/disadvantages of
 to!type vs .dup.

For example, to!string(someString) does not duplicate the string. Andrei
Aug 08 2010
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 05:17 PM, Yao G. wrote:
 On Sun, 08 Aug 2010 16:44:09 -0500, bearophile
 <bearophileHUGS lycos.com> wrote:

 Walter Bright:
 If you want to conclude that Python is better at processing files,
 you need to
 show it using each language doing it a way well suited to that
 language, rather
 than burdening one so it uses the same method as the less powerful one.

byLine() yields a char[], so if you want to do most kinds of strings processing or you want to store the line (or parts of it), you have to idup it. So in this case Python is not significantly less powerful than D. [snip] And you have to [be] thankful for my benchmarks. [snip] Bye, bearophile

<g> What's next? Will you demand attribution like the time Andrei presented the ranges design?

Well I understand his frustration. I asked him for a comparison and he took the time to write one and play with it. I think the proper answer to that is to see what we can do to improve the situation, not defend the status quo. Whatever the weaknesses of the benchmark are they should be fixed, and then whatever weaknesses the library has they should be addressed. Andrei
Aug 08 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 02:32 PM, bearophile wrote:
 Walter Bright:
 bearophile wrote:
 In the D code I have added an idup to make the comparison more fair, because
 in the Python code the "line" is a true newly allocated line, you can safely
 use it as dictionary key.

So it is with byLine, too. You've burdened D with double the amount of allocations.

I think you are wrong two times: 1) byLine() doesn't return a newly allocated line, you can see it with this small program: import std.stdio: File, writeln; void main(string[] args) { char[][] lines; auto file = File(args[1]); foreach (rawLine; file.byLine()) { writeln(rawLine.ptr); lines ~= rawLine; } file.close(); } Its output shows that all "strings" (char[]) share the same pointer: 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 14E5E00 ... 2) You can't use the result of rawLine() as string key for an associative array, as you I have said you can in Python. Currently you can, but according to Andrei this is a bug. And if it's not a bug then I'll reopen this closed bug 4474: http://d.puremagic.com/issues/show_bug.cgi?id=4474
 Also, I object in general to this method of making things "more fair". Using a
 less efficient approach in X because Y cannot use such an approach is not a
 legitimate comparison.

I generally agree, but this it not the case. In some situations you indeed don't need a newly allocated string for each loop, because for example you just want to read them and process them and not change/store them. You can't do this in Python, but this is not what I want to test. As I have explained in bug 4474 this behaviour is useful but it is acceptable only if explicitly requested by the programmer, and not as default one. The language is safe, as Andrei explains there, because you are supposed to idup the char[] to use it as key for an associative array (if your associative array is declared as int[char[]] then it can accept such rawLine() as keys, but you can clearly see those aren't strings. This is why I have closed bug 4474). Bye, bearophile

I think at the end of the day, regardless the relative possibilities of file reading in the two languages, we should be faster than Python when allocating one new string per line. Andrei
Aug 08 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 I think at the end of the day, regardless the relative possibilities of 
 file reading in the two languages, we should be faster than Python when 
 allocating one new string per line.

For now I suggest you to aim to be just about as fast as Python in this task :-) Beating Python significantly on this task is probably not easy. (Later someday I'd also like D AAs to become about as fast as Python dicts.) Bye, bearophile
Aug 08 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 10:29 PM, bearophile wrote:
 Andrei Alexandrescu:
 I think at the end of the day, regardless the relative
 possibilities of file reading in the two languages, we should be
 faster than Python when allocating one new string per line.

For now I suggest you to aim to be just about as fast as Python in this task :-) Beating Python significantly on this task is probably not easy.

Why? Andrei
Aug 08 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 For now I suggest you to aim to be just about as fast as Python in
 this task :-) Beating Python significantly on this task is probably
 not easy.

Why?

Because it's a core functionality for Python so devs probably have optimized it well, it's written in C, and in this case there is very little interpreter overhead. Bye, bearophile
Aug 09 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
bearophile wrote:
 Andrei Alexandrescu:
 
 For now I suggest you to aim to be just about as fast as Python
 in this task :-) Beating Python significantly on this task is
 probably not easy.


Because it's a core functionality for Python so devs probably have optimized it well, it's written in C, and in this case there is very little interpreter overhead.

Then we can do whatever they've done. It's not like they're using APIs nobody heard of. It seems such a comparison of file I/O speed becomes in fact a comparison of garbage collectors. That's fine, but in that case the notion that D offers the possibility to avoid allocation should come back to the table. Andrei
Aug 09 2010
prev sibling next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from bearophile (bearophileHUGS lycos.com)'s article
 Jonathan M Davis:
 I would have thought that being more idomatic would have resulted in slower
code
 than what Walter did, but interestingly enough, both programs are faster with
my
 code. They might take more memory though. I'm not quite sure how to check that.
 In any cases, you wanted some idiomatic D2 solutions, so there you go.

My (probably controversial) opinion on this is that the idiomatic D solution for

 In this case a Python version is more readable, shorter and probably faster too

D (because Python is more optimized for such purposes. I can show benchmarks on request).
 On the other hand D2 is in its debugging phase, so it's good to use it even for

think it's positive to write such scripts in D2, even if in a real-world setting I want to use Python to write them.
 Bye,
 bearophile

I disagree completely. D is clearly designed from the "simple things should be simple and complicated things should be possible" point of view. If it doesn't work well for these kinds of short scripts then we've failed at making simple things simple and we're just like every other crappy "large scale, industrial strength" language like Java and C++ that's great for megaprojects but makes simple things complicated. That said, I think D does a great job in this regard. I actually use Python as my language of second choice for things D isn't good at. Mostly this means needing Python's huge standard library, needing 64-bit support, or needing to share my code with people who don't know D. Needing to write a very short script tends not to be a reason for me to switch over. It's not that rare for me to start with a short script and then end up adding something that needs performance to it (like monte carlo simulation of a null probability distribution) and I don't find D substantially harder to use for these cases.
Aug 08 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 08/08/2010 14:31, dsimcha wrote:
 I disagree completely.  D is clearly designed from the "simple things should be
 simple and complicated things should be possible" point of view.  If it doesn't
 work well for these kinds of short scripts then we've failed at making simple
 things simple and we're just like every other crappy "large scale, industrial
 strength" language like Java and C++ that's great for megaprojects but makes
 simple things complicated.

dsimcha wrote: "I hate Java and every programming language where a readable hello world takes more than 3 SLOC" That may be your preference, but other people here in the community, me at least, very much want D to be a "large scale, industrial strength" language that's great for megaprojects. I think that medium and large scale projects are simply much more important and interesting than small scale ones. I am hoping this would become an *explicit* point of D design goals, if it isn't already. And I will campaign against (so to speak), people like you who think small scale is more important. No personal animosity intended though. Note: I am not stating that is is not possible to be good, even great, at both things (small and medium/large scale). -- Bruno Medeiros - Software Engineer
Sep 30 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Bruno Medeiros:

 I think that medium and large
 scale projects are simply much more important and interesting than small
 scale ones.

 I am hoping this would become an *explicit* point of D design goals, if
 it isn't already.
 And I will campaign against (so to speak), people like you who think
 small scale is more important. No personal animosity intended though.

 Note: I am not stating that is is not possible to be good, even great,
 at both things (small and medium/large scale).

This is an interesting topic of practical language design, it's a wide problem and I can't have complete answers. D2 design is mostly done, only small parts may be changed now, so those campaigns probably can't change D2 design much. The name of the Scala language means that it is meant to be a scalable language, this means it is designed to be useful and usable for both large and quite small programs. A language like Ada is not a bad language. Programming practice shows that in many situations the debug time is the larger percentage of the development of a program. So minimizing debug time is usually a very good thing. Ada tries hard to avoid many common bugs, much more than D (it has ranged integers, integer overflows, it defines a portable floating point semantics (despite there is a way to use the faster IEEE semantics), it forces to use clear interfaces between modules (much more explicit ones than D ones), it never silently changes variable types, its semantics is fully specified, there are very precise Ada semantics specs, all Ada compilers must pass a very large test suite, and so on and on). In practice the language is able to catch many bugs before they happen. So if you want to write a program in critical situations, like important control systems, Ada is a language better than Perl, and probably better than D too :-) Yet, writing programs in Ada is not handy, if you need to write small programs you need lot of boilerplate code that is useful only in larger programs. And Ada is a Pascal-like language that many modern programmers don't know/like. Ada looks designed for larger, low-bug-count, costly (and often well planned out from the beginning, with no specs that change with time) programs, but it's not handy to write small programs. Probably Ada is not the best language to write web code that has to change all the time. Today Ada is not a dead language, but it smells funny, it's not commonly used. Andrei has expressed the desire to use D2 as a language to write script-like programs too. I think in most cases a language like Python is better than D2 to write small script-like programs, yet I agree with Andrei that it's good to try to make D2 language fit to write small script-like programs too, because to write such programs you need a very handy language, that catches/avoids many simple common bugs quickly, gives you excellent error messages/stack traces, and allows you to do common operations on web/text files/images/sounds/etc in few lines of code. My theory is that later those qualities turn out to be useful even in large programs. I think such qualities may help D avoid the Ada fate. The ability to write small programs with D is also useful to attract programmers to D, because if in your language you need to write 30 lines long programs to write a "hello world" on the screen then newcomers are likely to stop using that language after their first try. Designing a language that is both good for small and large programs is not easy, but it is a worth goal. D module system must be debugged & finished & improved to improve the usage of D for larger programs. Some features of the unittesting and design by contract currently missing are very useful if you want to use D to write large programs. If you want to write large programs reliability becomes an important concern, so integer overflow tests and some system to avoid null-related bugs (not-nullable types and more) become useful or very useful. Bye, bearophile
Sep 30 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 30/09/2010 19:31, bearophile wrote:
 Bruno Medeiros:

 I think that medium and large
 scale projects are simply much more important and interesting than small
 scale ones.

 I am hoping this would become an *explicit* point of D design goals, if
 it isn't already.
 And I will campaign against (so to speak), people like you who think
 small scale is more important. No personal animosity intended though.

 Note: I am not stating that is is not possible to be good, even great,
 at both things (small and medium/large scale).

This is an interesting topic of practical language design, it's a wide problem and I can't have complete answers. D2 design is mostly done, only small parts may be changed now, so those campaigns probably can't change D2 design much.

I'm not so sure about that. Probably backwards-incompatible changes will be very few, if any. But there can be backwards-compatible changes, or changes to stuff that was not mentioned in TDPL. And there may be a D3 eventually (a long way down the road though) But my main worry is not language changes, I actually think it's very unlikely Walter and Andrei would do a language change that intentionally would adversely affect medium/large scale programs in favor of small scale programs. My main issue is with the time and thinking resources that are expended here in the NG when people argue for changes (or against other changes) with the intention of favoring small-scall programs. If this were explicit in the D design goals, it would help save us from these discussions (which affect NG readers, not just posters).
 The name of the Scala language means that it is meant to be a scalable
language, this means it is designed to be useful and usable for both large and
quite small programs.

Whoa wait. From my understanding, Scala is a "scalable language" in the sense that it easy to add new language features, or something similar to that. But lets be clear, that's not what I'm talking about, and neither is scalability of program data/inputs/performance. I'm talking about scalability of source code, software components, developers, teams, requirements, planning changes, project management issues, etc..
 A language like Ada is not a bad language. Programming practice shows that in
many situations the debug time is the larger percentage of the development of a
program. So minimizing debug time is usually a very good thing. Ada tries hard
to avoid many common bugs, much more than D (it has ranged integers, integer
overflows, it defines a portable floating point semantics (despite there is a
way to use the faster IEEE semantics), it forces to use clear interfaces
between modules (much more explicit ones than D ones), it never silently
changes variable types, its semantics is fully specified, there are very
precise Ada semantics specs, all Ada compilers must pass a very large test
suite, and so on and on). In practice the language is able to catch many bugs
before they happen.

 So if you want to write a program in critical situations, like important
control systems, Ada is a language better than Perl, and probably better than D
too :-)

 Yet, writing programs in Ada is not handy, if you need to write small programs
you need lot of boilerplate code that is useful only in larger programs. And
Ada is a Pascal-like language that many modern programmers don't know/like. Ada
looks designed for larger, low-bug-count, costly (and often well planned out
from the beginning, with no specs that change with time) programs, but it's not
handy to write small programs. Probably Ada is not the best language to write
web code that has to change all the time. Today Ada is not a dead language, but
it smells funny, it's not commonly used.

Certainly it's not just web code that can change all the time. But I'm missing your point here, what does Ada have to do with this?
 Andrei has expressed the desire to use D2 as a language to write script-like
programs too. I think in most cases a language like Python is better than D2 to
write small script-like programs, yet I agree with Andrei that it's good to try
to make D2 language fit to write small script-like programs too, because to
write such programs you need a very handy language, that catches/avoids many
simple common bugs quickly, gives you excellent error messages/stack traces,
and allows you to do common operations on web/text files/images/sounds/etc in
few lines of code. My theory is that later those qualities turn out to be
useful even in large programs. I think such qualities may help D avoid the Ada
fate.

 The ability to write small programs with D is also useful to attract
programmers to D, because if in your language you need to write 30 lines long
programs to write a "hello world" on the screen then newcomers are likely to
stop using that language after their first try.

 Designing a language that is both good for small and large programs is not
easy, but it is a worth goal. D module system must be debugged&  finished& 
improved to improve the usage of D for larger programs. Some features of the
unittesting and design by contract currently missing are very useful if you
want to use D to write large programs. If you want to write large programs
reliability becomes an important concern, so integer overflow tests and some
system to avoid null-related bugs (not-nullable types and more) become useful
or very useful.

 Bye,
 bearophile

Yeah, I actually think D (or any other language under design) can be quite good at both things. Maybe something like 90% of features that are good for large-scale programs are also good for small-scale ones. One of the earliest useful programs I wrote in D, was a two-page bash shell script that I converted to D. Even though it was just abut two pages, it was already hard to extend and debug. After converting it to D, with the right shortcut methods and abstractions, the code actually manage to be quite succint and comparable, I suspect, to code in Python or Perl, or languages like that. (I say suspect because I don't actually know much about Python or Perl, but I simply didn't see much language changes that could have made my D more succint, barring crazy stuff like dynamic scoping) -- Bruno Medeiros - Software Engineer
Oct 01 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Bruno Medeiros:

 From my understanding, Scala is a "scalable language" in the sense
 that it easy to add new language features, or something similar to that.

I see. You may be right.
 But I'm missing your point here, what does Ada have to do with this?

Ada has essentially died for several reasons, but in my opinion one of them is the amount of code you have to write to do even small things. If you design a language that is not handy to write small programs, you have a higher risk of seeing your language die.
 but I simply didn't see much language 
 changes that could have made my D more succint,

Making a language more succint is easy, you may take a look at J or K languages. The hard thing is to design a succint language that is also readable and not bug-prone. Python has some features that make the code longer, like the obligatory "self." before class instance names and the optional usage of argument names at the calling point make the code longer. The ternary operator too in Python is longer, as the "and" operator, etc. Such things improve readability, etc. Several Python features help shorten the code, like sequence unpacking syntax and multiple return values:
 def foo():



...
 a, b = foo()
 a



 b



List comprehensions help shorten the code, but I think they also reduce bug count a bit and allow you to think about your code at a bit higher level:
 xs = [2,3,4,5,6,7,8,9,10,11,12,13]
 ps = [x * x for x in xs if x % 2]
 ps



Python has some other features that help shorten the code, like the significant leading white space that avoids some bugs, avoids brace style wars, and removes both some noise and closing brace code lines.
 barring crazy stuff like dynamic scoping)

I don't know what dynamic scoping is, do you mean that crazy nice thing named dynamic typing? :-) Bye, bearophile
Oct 01 2010
next sibling parent Pelle <pelle.mansson gmail.com> writes:
On 10/01/2010 01:54 PM, bearophile wrote:
 Bruno Medeiros:

  From my understanding, Scala is a "scalable language" in the sense
 that it easy to add new language features, or something similar to that.

I see. You may be right.
 But I'm missing your point here, what does Ada have to do with this?

Ada has essentially died for several reasons, but in my opinion one of them is the amount of code you have to write to do even small things. If you design a language that is not handy to write small programs, you have a higher risk of seeing your language die.
 but I simply didn't see much language
 changes that could have made my D more succint,

Making a language more succint is easy, you may take a look at J or K languages. The hard thing is to design a succint language that is also readable and not bug-prone. Python has some features that make the code longer, like the obligatory "self." before class instance names and the optional usage of argument names at the calling point make the code longer. The ternary operator too in Python is longer, as the "and" operator, etc. Such things improve readability, etc. Several Python features help shorten the code, like sequence unpacking syntax and multiple return values:
 def foo():



...
 a, b = foo()
 a



 b



List comprehensions help shorten the code, but I think they also reduce bug count a bit and allow you to think about your code at a bit higher level:
 xs = [2,3,4,5,6,7,8,9,10,11,12,13]
 ps = [x * x for x in xs if x % 2]
 ps



Python has some other features that help shorten the code, like the significant leading white space that avoids some bugs, avoids brace style wars, and removes both some noise and closing brace code lines.
 barring crazy stuff like dynamic scoping)

I don't know what dynamic scoping is, do you mean that crazy nice thing named dynamic typing? :-) Bye, bearophile

No, dynamic scoping is the crazy thing. Perl code: #!/usr/bin/perl $x = 1; sub p { print "$x\n" } sub a { local $x = 2; p; } p; a; p results in: pp ~/perl% perl wat.pl 1 2 1 Crazy. :-)
Oct 01 2010
prev sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 01/10/2010 12:54, bearophile wrote:
 Bruno Medeiros:

  From my understanding, Scala is a "scalable language" in the sense
 that it easy to add new language features, or something similar to that.

I see. You may be right.
 But I'm missing your point here, what does Ada have to do with this?

Ada has essentially died for several reasons, but in my opinion one of them is the amount of code you have to write to do even small things. If you design a language that is not handy to write small programs, you have a higher risk of seeing your language die.

There are a lot of things in a language that, if they make it harder to write small programs, they will also make it harder for larger programs. (sometimes even much harder) I'm no expert in ADA, and there are many things that will affect the success of the language, so I can't comment in detail. But from a cursory look at the language, it looks terribly terse. That "begin" "end <name of block>" syntax is awful. I already think just "begin" "end" syntax is bad, but also having to repeat the name of block/function/procedure/loop at the "end", that's awful. Is it trying to compete with "XML" ? :p
 but I simply didn't see much language
 changes that could have made my D more succint,

Making a language more succint is easy, you may take a look at J or K languages. The hard thing is to design a succint language that is also readable and not bug-prone.

Indeed, I agree. And that was the spirit of that original comment: First of all, I meant succinct not only in character and line count but also syntactical and semantic constructs. And succinct without changes that would impact a lot the readability or safety of the code. (as mentioned in "barring crazy stuff like dynamic scoping")
 barring crazy stuff like dynamic scoping)

I don't know what dynamic scoping is, do you mean that crazy nice thing named dynamic typing? :-)

Like Pete explained, it's indeed exactly "dynamic scoping" that I meant. -- Bruno Medeiros - Software Engineer
Oct 05 2010
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:i3lb30$26vf$1 digitalmars.com...
 Jonathan M Davis:
 I would have thought that being more idomatic would have resulted in 
 slower code
 than what Walter did, but interestingly enough, both programs are faster 
 with my
 code. They might take more memory though. I'm not quite sure how to check 
 that.
 In any cases, you wanted some idiomatic D2 solutions, so there you go.

Your code looks better. My (probably controversial) opinion on this is that the idiomatic D solution for those text "scripts" is to use a scripting language, as Python :-)

I can respect that. Personally, though, I find a lot of value in not needing to switch languages for that sort of thing. Too much "context switch" for my brain ;)
Aug 08 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmail.com> writes:
Jonathan M Davis wrote:

 void removeTabs(int tabSize, string fileName)
 {
     auto file = File(fileName);
     string[] output;
 
     foreach(line; file.byLine())
     {
         int lastTab = 0;
 
         while(lastTab != -1)
         {
             const int tab = line.indexOf('\t');
 
             if(tab == -1)
                 break;
 
             const int numSpaces = tabSize - tab % tabSize;
 
             line = line[0 .. tab] ~ repeat(" ", numSpaces) ~ line[tab + 1
             .. $];
 
             lastTab = tab + numSpaces;
         }
 
         output ~= line.idup;
     }
 
     std.file.write(fileName, output.join("\n"));
 }

Actually, looking at the code again, that while loop really should be while(1) rather than while(lastTab != -1), but it will work the same regardless. - Jonathan M Davis
Aug 07 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/07/2010 11:04 PM, Jonathan M Davis wrote:
 On Friday 06 August 2010 18:50:52 Andrei Alexandrescu wrote:
 A good exercise would be rewriting these tools in idiomatic D2 and
 assess the differences.


 Andrei

I didn't try and worry about multiline string literals, but here are my more idiomatic solutions: detab: /* Replace tabs with spaces, and remove trailing whitespace from lines. */ import std.conv; import std.file; import std.stdio; import std.string; void main(string[] args) { const int tabSize = to!int(args[1]); foreach(f; args[2 .. $]) removeTabs(tabSize, f); } void removeTabs(int tabSize, string fileName) { auto file = File(fileName); string[] output; foreach(line; file.byLine()) { int lastTab = 0; while(lastTab != -1) { const int tab = line.indexOf('\t'); if(tab == -1) break; const int numSpaces = tabSize - tab % tabSize; line = line[0 .. tab] ~ repeat(" ", numSpaces) ~ line[tab + 1 .. $]; lastTab = tab + numSpaces; } output ~= line.idup; } std.file.write(fileName, output.join("\n")); }

Very nice. Here's how I'd improve removeTabs: #!/home/andrei/bin/rdmd import std.conv; import std.file; import std.getopt; import std.stdio; import std.string; void main(string[] args) { uint tabSize = 8; getopt(args, "tabsize|t", &tabSize); foreach(f; args[1 .. $]) removeTabs(tabSize, f); } void removeTabs(int tabSize, string fileName) { auto file = File(fileName); string output; bool changed; foreach(line; file.byLine(File.KeepTerminator.yes)) { int lastTab = 0; while(lastTab != -1) { const tab = line.indexOf('\t'); if(tab == -1) break; const numSpaces = tabSize - tab % tabSize; line = line[0 .. tab] ~ repeat(" ", numSpaces) ~ line[tab + 1 .. $]; lastTab = tab + numSpaces; changed = true; } output ~= line; } file.close(); if (changed) std.file.write(fileName, output); }
 -------------------------------------------

 The three differences between mine and Walter's are that mine takes the tab
size
 as the first argumen,t it doesn't put a newline at the end of the file, and it
 writes the file even if it changed (you could test for that, but when using
 byLine(), it's a bit harder). Interestingly enough, from the few tests that I
 ran, mine seems to be somewhat faster. I also happen to think that the code is
 clearer (it's certainly shorter), though that might be up for debate.

 -------------------------------------------



 tolf:

 /* Replace line endings with LF
    */

 import std.file;
 import std.string;

 void main(string[] args)
 {
      foreach(f; args[1 .. $])
          fixEndLines(f);
 }

 void fixEndLines(string fileName)
 {
      auto fileStr = std.file.readText(fileName);
      auto result = fileStr.replace("\r\n", "\n").replace("\r", "\n");

      std.file.write(fileName, result);
 }

 -------------------------------------------

 This version is ludicrously simple. And it was also faster than Walter's in the
 few tests that I ran. In either case, I think that it is definitely clearer
code.

Very nice! You may as well guard the write with an if (result != fileStr). With control source etc. in the mix it's always polite to not touch files unless you are actually modifying them. This makes me think we should have a range that detects and replaces patterns lazily and on the fly. I've always thought that loading entire files in memory and working on them is "cheating" in some sense, and a range would help with replacing patterns in streams.
 I would have thought that being more idomatic would have resulted in slower
code
 than what Walter did, but interestingly enough, both programs are faster with
my
 code. They might take more memory though. I'm not quite sure how to check that.
 In any cases, you wanted some idiomatic D2 solutions, so there you go.

Looking very good, thanks. I think we should have a feature these and a few others as examples on the website. Andrei
Aug 07 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Jonathan M Davis wrote:
 It would certainly be nice to have a way to reasonably process with ranges 
 without having to load the whole thing into memory at once.

Because of asynchronous I/O, being able to start processing and start writing the new file before the old one is finished reading should speed things up.
Aug 07 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:i3ldk4$2ci0$1 digitalmars.com...
 Very nice! You may as well guard the write with an if (result != fileStr). 
 With control source etc. in the mix it's always polite to not touch files 
 unless you are actually modifying them.

I'm fairly sure SVN doesn't commit touched files unless there are actual changes. (Or maybe it's TortoiseSVN that adds that intelligence?)
Aug 08 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/08/2010 12:28 PM, Nick Sabalausky wrote:
 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
 news:i3ldk4$2ci0$1 digitalmars.com...
 Very nice! You may as well guard the write with an if (result != fileStr).
 With control source etc. in the mix it's always polite to not touch files
 unless you are actually modifying them.

I'm fairly sure SVN doesn't commit touched files unless there are actual changes. (Or maybe it's TortoiseSVN that adds that intelligence?)

It doesn't, but it still shows them as changed etc. Andrei
Aug 08 2010
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
--001636b148d41526a9048d56dc52
Content-Type: text/plain; charset=ISO-8859-1

Andrei used to!string() in an early example in TDPL for some line-by-line
processing. I'm not sure of the advantages/disadvantages of to!type vs .dup.

On Sun, Aug 8, 2010 at 11:44 PM, bearophile <bearophileHUGS lycos.com>wrote:

 Walter Bright:
 If you want to conclude that Python is better at processing files, you

 show it using each language doing it a way well suited to that language,

 than burdening one so it uses the same method as the less powerful one.

byLine() yields a char[], so if you want to do most kinds of strings processing or you want to store the line (or parts of it), you have to idup it. So in this case Python is not significantly less powerful than D. You can of course use the raw char[], but then you lose the advantages advertised when you have introduced the safer immutable D2 strings. And in many situations you have to dup the char[] anyway, otherwise your have all kinds of bugs, that Python lacks. In D1 to avoid it I used to use dup more often than necessary. I have explained this in the bug 4474. In this newsgroup my purpose it to show D faults, suggest improvements, etc. In this case my purpose was just to show that byLine()+idup is slow. And you have to thankful for my benchmarks. In my dlibs1 for D1 I have a xio module that reads files by line that is faster than iterating on a BufferedFile, so it's not a limit of the language, it's Phobos that has a performance bug that can be improved. Bye, bearophile

--001636b148d41526a9048d56dc52 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Andrei used to!string() in an early example in TDPL for some line-by-line p= rocessing. I&#39;m not sure of the advantages/disadvantages of to!type vs .= dup.<br><br><div class=3D"gmail_quote">On Sun, Aug 8, 2010 at 11:44 PM, bea= rophile <span dir=3D"ltr">&lt;<a href=3D"mailto:bearophileHUGS lycos.com">b= earophileHUGS lycos.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde= r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Walter Bright:<br=

cessing files, you need to<br> &gt; show it using each language doing it a way well suited to that languag= e, rather<br> &gt; than burdening one so it uses the same method as the less powerful one= .<br> <br> </div>byLine() yields a char[], so if you want to do most kinds of strings = processing or you want to store the line (or parts of it), you have to idup= it. So in this case Python is not significantly less powerful than D.<br> <br> You can of course use the raw char[], but then you lose the advantages adve= rtised when you have introduced the safer immutable D2 strings. And in many= situations you have to dup the char[] anyway, otherwise your have all kind= s of bugs, that Python lacks. In D1 to avoid it I used to use dup more ofte= n than necessary. I have explained this in the bug 4474.<br> <br> In this newsgroup my purpose it to show D faults, suggest improvements, etc= . In this case my purpose was just to show that byLine()+idup is slow. And = you have to thankful for my benchmarks. In my dlibs1 for D1 I have a xio mo= dule that reads files by line that is faster than iterating on a BufferedFi= le, so it&#39;s not a limit of the language, it&#39;s Phobos that has a per= formance bug that can be improved.<br> <br> Bye,<br> <font color=3D"#888888">bearophile<br> </font></blockquote></div><br> --001636b148d41526a9048d56dc52--
Aug 08 2010
prev sibling next sibling parent "Yao G." <nospamyao gmail.com> writes:
On Sun, 08 Aug 2010 16:44:09 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 Walter Bright:
 If you want to conclude that Python is better at processing files, you  
 need to
 show it using each language doing it a way well suited to that  
 language, rather
 than burdening one so it uses the same method as the less powerful one.

byLine() yields a char[], so if you want to do most kinds of strings processing or you want to store the line (or parts of it), you have to idup it. So in this case Python is not significantly less powerful than D. [snip] And you have to [be] thankful for my benchmarks. [snip] Bye, bearophile

<g> What's next? Will you demand attribution like the time Andrei presented the ranges design?
Aug 08 2010
prev sibling parent "Yao G." <nospamyao gmail.com> writes:
On Sun, 08 Aug 2010 17:27:04 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 Yao G.:
 <g> What's next? Will you demand attribution like the time Andrei
 presented the ranges design?

Of course. In the end all D will be mine <evil laugh with echo effects> :-) Bye, bearophile

:D That was a good comeback. -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Aug 08 2010
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
--e0cb4e88782748c3cd048d325954
Content-Type: text/plain; charset=ISO-8859-1

Or improve your google-fu by finding some existing tools that do the job
right. :)

I'm pretty sure Uncrustify is good at most of these issues, not to mention
it's a very nice source-code "prettifier/indenter". There's a front-end
called UniversalIndentGUI, which has about a dozen integrated versions of
source-code prettifiers (including uncrustify, and for many languages). It
has varios settings on the left, and togglable *Live* preview mode which you
can view on the right.

I invite you guys to try it out sometime:

http://universalindent.sourceforge.net/

(+ you can save different settings which is neat when you're coding for
different projects that have different "code design & look" standards)

On Sat, Aug 7, 2010 at 3:50 AM, Andrei Alexandrescu <
SeeWebsiteForEmail erdani.org> wrote:

 On 08/06/2010 08:34 PM, Walter Bright wrote:

 I wrote these two trivial utilities for the purpose of canonicalizing
 source code before checkins and to deal with FreeBSD's inability to deal
 with CRLF line endings, and because I can never figure out the right
 settings for git to make it do the canonicalization.

 tolf - converts LF, CR, and CRLF line endings to LF.

 detab - converts all tabs to the correct number of spaces. Assumes tabs
 are 8 column tabs. Removes trailing whitespace from lines.

 Posted here just in case someone wonders what they are.

Nice, though they don't account for multiline string literals. A good exercise would be rewriting these tools in idiomatic D2 and assess the differences. Andrei

--e0cb4e88782748c3cd048d325954 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Or improve your google-fu by finding some existing tools that do the job ri= ght. :)<br><br>I&#39;m pretty sure Uncrustify is good at most of these issu= es, not to mention it&#39;s a very nice source-code &quot;prettifier/indent= er&quot;. There&#39;s a front-end called UniversalIndentGUI, which has abou= t a dozen integrated versions of source-code prettifiers (including uncrust= ify, and for many languages). It has varios settings on the left, and toggl= able *Live* preview mode which you can view on the right.<br> <br>I invite you guys to try it out sometime:<br><br><a href=3D"http://univ= ersalindent.sourceforge.net/">http://universalindent.sourceforge.net/</a><b= r><br>(+ you can save different settings which is neat when you&#39;re codi= ng for different projects that have different &quot;code design &amp; look&= quot; standards)<br> <br><div class=3D"gmail_quote">On Sat, Aug 7, 2010 at 3:50 AM, Andrei Alexa= ndrescu <span dir=3D"ltr">&lt;<a href=3D"mailto:SeeWebsiteForEmail erdani.o= rg">SeeWebsiteForEmail erdani.org</a>&gt;</span> wrote:<br><blockquote clas= s=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px soli= d rgb(204, 204, 204); padding-left: 1ex;"> <div class=3D"im">On 08/06/2010 08:34 PM, Walter Bright wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde= r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"> I wrote these two trivial utilities for the purpose of canonicalizing<br> source code before checkins and to deal with FreeBSD&#39;s inability to dea= l<br> with CRLF line endings, and because I can never figure out the right<br> settings for git to make it do the canonicalization.<br> <br> tolf - converts LF, CR, and CRLF line endings to LF.<br> <br> detab - converts all tabs to the correct number of spaces. Assumes tabs<br> are 8 column tabs. Removes trailing whitespace from lines.<br> <br> Posted here just in case someone wonders what they are.<br> </blockquote></div> [snip]<br> <br> Nice, though they don&#39;t account for multiline string literals.<br> <br> A good exercise would be rewriting these tools in idiomatic D2 and assess t= he differences.<br><font color=3D"#888888"> <br> <br> Andrei<br> </font></blockquote></div><br> --e0cb4e88782748c3cd048d325954--
Aug 06 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisprog gmail.com> writes:
On Saturday 07 August 2010 21:59:50 Andrei Alexandrescu wrote:
 Very nice. Here's how I'd improve removeTabs:
 
 #!/home/andrei/bin/rdmd
 import std.conv;
 import std.file;
 import std.getopt;
 import std.stdio;
 import std.string;
 
 void main(string[] args)
 {
      uint tabSize = 8;
      getopt(args, "tabsize|t", &tabSize);
      foreach(f; args[1 .. $])
          removeTabs(tabSize, f);
 }
 
 void removeTabs(int tabSize, string fileName)
 {
      auto file = File(fileName);
      string output;
      bool changed;
 
      foreach(line; file.byLine(File.KeepTerminator.yes))
      {
          int lastTab = 0;
 
          while(lastTab != -1)
          {
              const tab = line.indexOf('\t');
              if(tab == -1)
                  break;
              const numSpaces = tabSize - tab % tabSize;
              line = line[0 .. tab] ~ repeat(" ", numSpaces) ~ line[tab +
 1 .. $];
              lastTab = tab + numSpaces;
              changed = true;
          }
 
          output ~= line;
      }
 
      file.close();
      if (changed)
          std.file.write(fileName, output);
 }

Ah. I needed to close the file. I pretty much always just use readText(), so I didn't catch that. Also, it does look like detecting whether the file changed was a bit simpler than I thought that it would be. Quite simple really. Thanks.
 Very nice! You may as well guard the write with an if (result !=
 fileStr). With control source etc. in the mix it's always polite to not
 touch files unless you are actually modifying them.

Yes. That would be good. It's the kind of thing that I forget - probably because most of the code that I write generates new files rather than updating pre- existing ones.
 
 This makes me think we should have a range that detects and replaces
 patterns lazily and on the fly. I've always thought that loading entire
 files in memory and working on them is "cheating" in some sense, and a
 range would help with replacing patterns in streams.

It would certainly be nice to have a way to reasonably process with ranges without having to load the whole thing into memory at once. Most of the time, I wouldn't care too much, but if you start processing large files, having the whole thing in memory could be a problem (especially if you have multiple versions of it which were created along the way as you were manipulating it). Haskell does lazy loading of files by default and doesn't load the data until you read the appropriate part of the string. It shouldn't be all that hard to do something similar with D and ranges. The hard port would be trying to do all of it in a way that makes it so that all of the processing of the file's data doesn't have to load it all into memory (let alone load it multiple times). I'm not sure that you could do that without explicitly processing a file line by line, writing it to disk after each line is processed, since you could be doing an arbitrary set of operations on the data. It could be interesting to try and find a solution for that though.
 
 Looking very good, thanks. I think we should have a feature these and a
 few others as examples on the website.

Well, I for one, much prefer the ability to program in a manner that's closer to telling the computer to do what I want rather than having to tell it how to do what I want (the replace end-of-line character program being a prime example). It makes life much simpler. Ranges certainly help a lot in that regard too. And having good example code of how to program that way could help encourage people to program that way and use std.range and std.algorithm and their ilk rather than trying more low-level solutions which aren't as easy to understand. - Jonathan M Davis
Aug 07 2010
prev sibling next sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
I usually do the same thing with a shell pipe
	expand | sed 's/ *$//;s/\r$//;s/\r/\n/'


On 07/08/10 02:34, Walter Bright wrote:
 I wrote these two trivial utilities for the purpose of canonicalizing
 source code before checkins and to deal with FreeBSD's inability to deal
 with CRLF line endings, and because I can never figure out the right
 settings for git to make it do the canonicalization.

 tolf - converts LF, CR, and CRLF line endings to LF.

 detab - converts all tabs to the correct number of spaces. Assumes tabs
 are 8 column tabs. Removes trailing whitespace from lines.

 Posted here just in case someone wonders what they are.
 ---------------------------------------------------------
 /* Replace tabs with spaces, and remove trailing whitespace from lines.
 */

 import std.file;
 import std.path;

 int main(string[] args)
 {
 foreach (f; args[1 .. $])
 {
 auto input = cast(char[]) std.file.read(f);
 auto output = filter(input);
 if (output != input)
 std.file.write(f, output);
 }
 return 0;
 }


 char[] filter(char[] input)
 {
 char[] output;
 size_t j;

 int column;
 for (size_t i = 0; i < input.length; i++)
 {
 auto c = input[i];

 switch (c)
 {
 case '\t':
 while ((column & 7) != 7)
 { output ~= ' ';
 j++;
 column++;
 }
 c = ' ';
 column++;
 break;

 case '\r':
 case '\n':
 while (j && output[j - 1] == ' ')
 j--;
 output = output[0 .. j];
 column = 0;
 break;

 default:
 column++;
 break;
 }
 output ~= c;
 j++;
 }
 while (j && output[j - 1] == ' ')
 j--;
 return output[0 .. j];
 }
 -----------------------------------------------------
 /* Replace line endings with LF
 */

 import std.file;
 import std.path;

 int main(string[] args)
 {
 foreach (f; args[1 .. $])
 {
 auto input = cast(char[]) std.file.read(f);
 auto output = filter(input);
 if (output != input)
 std.file.write(f, output);
 }
 return 0;
 }


 char[] filter(char[] input)
 {
 char[] output;
 size_t j;

 for (size_t i = 0; i < input.length; i++)
 {
 auto c = input[i];

 switch (c)
 {
 case '\r':
 c = '\n';
 break;

 case '\n':
 if (i && input[i - 1] == '\r')
 continue;
 break;

 case 0:
 continue;

 default:
 break;
 }
 output ~= c;
 j++;
 }
 return output[0 .. j];
 }
 ------------------------------------------

Aug 08 2010
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Norbert Nemec wrote:
 I usually do the same thing with a shell pipe
     expand | sed 's/ *$//;s/\r$//;s/\r/\n/'

<g>
Aug 08 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Norbert Nemec" <Norbert Nemec-online.de> wrote in message 
news:i3lq17$99u$1 digitalmars.com...
I usually do the same thing with a shell pipe
 expand | sed 's/ *$//;s/\r$//;s/\r/\n/'

Filed under "Why I don't like regex for non-trivial things" ;)
Aug 08 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Leandro Lucarella" <luca llucax.com.ar> wrote in message 
news:20100808212859.GL3360 llucax.com.ar...
 Nick Sabalausky, el  8 de agosto a las 13:31 me escribiste:
 "Norbert Nemec" <Norbert Nemec-online.de> wrote in message
 news:i3lq17$99u$1 digitalmars.com...
I usually do the same thing with a shell pipe
 expand | sed 's/ *$//;s/\r$//;s/\r/\n/'

Filed under "Why I don't like regex for non-trivial things" ;)

Those regex are non-trivial?

IMHO, A task has to be REALLY trivial to be trivial in regex ;)
 Maybe you're confusing sed statements with regex, in that sed program,
 there are 3 trivial regex:

Ahh, I see. I'm not familiar with sed, so my eyes got to the part after "sed" and began bleeding, so I figured it had to be one of three things: - Encrypted data - Hardware crash - Regex ;) Insert other joke about "read-only languages" or "languages that look the same before and after RSA encryption" here. (I'm not genuinely complaining about regexes. They can be very useful. They just tend to get real ugly real fast.)
Aug 08 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Nick Sabalausky wrote:
 (I'm not genuinely complaining about regexes. They can be very useful. They 
 just tend to get real ugly real fast.)

Regexes are like flying airplanes. You have to do them often or you get "rusty" real fast. (Flying is not a natural behavior, it's not like riding a bike.)
Aug 08 2010
prev sibling next sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
Nick Sabalausky, el  8 de agosto a las 13:31 me escribiste:
 "Norbert Nemec" <Norbert Nemec-online.de> wrote in message 
 news:i3lq17$99u$1 digitalmars.com...
I usually do the same thing with a shell pipe
 expand | sed 's/ *$//;s/\r$//;s/\r/\n/'

Filed under "Why I don't like regex for non-trivial things" ;)

Those regex are non-trivial? Maybe you're confusing sed statements with regex, in that sed program, there are 3 trivial regex: regex replace with *$ (nothing) \r$ (nothing) \r \n They are the most trivial regex you'd ever find! =) -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- Vaporeso sostenía a rajacincha la teoría del No-Water, la cual le pertenecía y versaba lo siguiente: "Para darle la otra mejilla al fuego, éste debe ser apagado con alpargatas apenas húmedas".
Aug 08 2010
prev sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
Andrei Alexandrescu, el  8 de agosto a las 14:44 me escribiste:
 On 08/08/2010 12:28 PM, Nick Sabalausky wrote:
"Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
news:i3ldk4$2ci0$1 digitalmars.com...
Very nice! You may as well guard the write with an if (result != fileStr).
With control source etc. in the mix it's always polite to not touch files
unless you are actually modifying them.

I'm fairly sure SVN doesn't commit touched files unless there are actual changes. (Or maybe it's TortoiseSVN that adds that intelligence?)

It doesn't, but it still shows them as changed etc.

Nope, not really: /tmp$ svnadmin create x /tmp$ svn co file:///tmp/x xwc Revisión obtenida: 0 /tmp$ cd xwc/ /tmp/xwc$ echo hello > hello /tmp/xwc$ svn add hello A hello /tmp/xwc$ svn commit -m 'test' Añadiendo hello Transmitiendo contenido de archivos . Commit de la revisión 1. /tmp/xwc$ touch hello /tmp/xwc$ svn status /tmp/xwc$ echo changed > hello /tmp/xwc$ svn status M hello /tmp/xwc$ (sorry about the Spanish messages, I saw them after copying the test and I'm too lazy to repeat them changing the LANG environment variable :) You might want to set the mtime to the same as the original file for build purposes though (you know you're changing the file in a way it doesn't really change its semantics, so you might want to avoid unnecessary recompilation). -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- ... los cuales son susceptibles a una creciente variedad de ataques previsibles, tales como desbordamiento del tampón, falsificación de parámetros, ... -- Stealth - ISS LLC - Seguridad de IT
Aug 08 2010