"Graham Fawcett" <fawcett uwindsor.ca> writes:

Hi folks,

There's an interesting discussion going on at Reddit about 
choosing a replacement language for 0install:

http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

I've tried to do a bit of D advocacy there, but there's more to 
be done. :) If you have a few moments to dispel some D myths, and 
contribute constructively to the discussion, please take a look!

Best,
Graham

"Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:

On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more to 
 be done. :) If you have a few moments to dispel some D myths, 
 and contribute constructively to the discussion, please take a 
 look!

 Best,
 Graham

I don't know how to make this test on Windows (current OS). But 
he uses this to test that failure to print hello correctly 
indicates failure.

./hello 1< /dev/null; echo Exit status: $?


Haskell, OCaml, Python)

"Anthony Goins" <neontotem gmail.com> writes:

On Monday, 10 June 2013 at 20:51:16 UTC, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more 
 to be done. :) If you have a few moments to dispel some D 
 myths, and contribute constructively to the discussion, please 
 take a look!

 Best,
 Graham

 I don't know how to make this test on Windows (current OS). But 
 he uses this to test that failure to print hello correctly 
 indicates failure.

 ./hello 1< /dev/null; echo Exit status: $?


 Haskell, OCaml, Python)

If you want to know what happens on my linux box

   1 module hellotest;
   2
   3 import std.stdio;
   4
   5 void main()
   6 {
   7         writeln("hello world.");
   8 }

anthony LinuxGen12:~/projects/temp$ ./hellotest
hello world.
anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo 
status : $?
status : 0
anthony LinuxGen12:~/projects/temp$

"Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:

On Monday, 10 June 2013 at 22:06:29 UTC, Anthony Goins wrote:
 If you want to know what happens on my linux box

   1 module hellotest;
   2
   3 import std.stdio;
   4
   5 void main()
   6 {
   7         writeln("hello world.");
   8 }

 anthony LinuxGen12:~/projects/temp$ ./hellotest
 hello world.
 anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; 
 echo status : $?
 status : 0
 anthony LinuxGen12:~/projects/temp$

So D fails the test too. But now that I think about it, isn't 
/dev/null where you write data to a black hole... maybe he meant 
/dev/zero

Walter Bright <newshound2 digitalmars.com> writes:

On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 22:06:29 UTC, Anthony Goins wrote:
 If you want to know what happens on my linux box

   1 module hellotest;
   2
   3 import std.stdio;
   4
   5 void main()
   6 {
   7         writeln("hello world.");
   8 }

 anthony LinuxGen12:~/projects/temp$ ./hellotest
 hello world.
 anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo status : $?
 status : 0
 anthony LinuxGen12:~/projects/temp$

 So D fails the test too. But now that I think about it, isn't /dev/null where
 you write data to a black hole... maybe he meant /dev/zero

Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on writing to
that, 
and didn't.

"David Nadlinger" <code klickverbot.at> writes:

On Tuesday, 11 June 2013 at 00:27:28 UTC, Walter Bright wrote:
 On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 So D fails the test too. But now that I think about it, isn't 
 /dev/null where
 you write data to a black hole... maybe he meant /dev/zero

 Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on 
 writing to that, and didn't.

The command actually redirects /dev/null to fd 1 as an *input* 
stream, thus rendering it non-writable.

David

Walter Bright <newshound2 digitalmars.com> writes:

On 6/10/2013 5:35 PM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 00:27:28 UTC, Walter Bright wrote:
 On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 So D fails the test too. But now that I think about it, isn't /dev/null where
 you write data to a black hole... maybe he meant /dev/zero

 Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on writing to
 that, and didn't.

 The command actually redirects /dev/null to fd 1 as an *input* stream, thus
 rendering it non-writable.

Ah, I see. In that case, writeln should have thrown an exception. Should submit 
a bug report.

"Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:

On Tuesday, 11 June 2013 at 00:36:50 UTC, Walter Bright wrote:
 Ah, I see. In that case, writeln should have thrown an 
 exception. Should submit a bug report.

http://d.puremagic.com/issues/show_bug.cgi?id=10328

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/10/13 6:06 PM, Anthony Goins wrote:
 On Monday, 10 June 2013 at 20:51:16 UTC, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about choosing a
 replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/


 I've tried to do a bit of D advocacy there, but there's more to be
 done. :) If you have a few moments to dispel some D myths, and
 contribute constructively to the discussion, please take a look!

 Best,
 Graham

 I don't know how to make this test on Windows (current OS). But he
 uses this to test that failure to print hello correctly indicates
 failure.

 ./hello 1< /dev/null; echo Exit status: $?


 OCaml, Python)

 If you want to know what happens on my linux box

 1 module hellotest;
 2
 3 import std.stdio;
 4
 5 void main()
 6 {
 7 writeln("hello world.");
 8 }

 anthony LinuxGen12:~/projects/temp$ ./hellotest
 hello world.
 anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo status
 : $?
 status : 0
 anthony LinuxGen12:~/projects/temp$

The test program is flawed; writeln() writes to stdout, and the 
redirection is to stdin.

writeln throws upon error.


Andrei

"David Nadlinger" <code klickverbot.at> writes:

On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu 
wrote:
 The test program is flawed; writeln() writes to stdout, and the 
 redirection is to stdin.

Wouldn't stdin be fd 0?

David

"Infiltrator" <Infiltrator d.irc> writes:

On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu 
 wrote:
 The test program is flawed; writeln() writes to stdout, and 
 the redirection is to stdin.

 Wouldn't stdin be fd 0?

 David

What David said.  Here's the same thing done to a cat (poor 
kitty):

$ cat 1</dev/null
test
cat: write error: Bad file descriptor

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 12:16 AM, Infiltrator wrote:
 On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

 Wouldn't stdin be fd 0?

 David

 What David said. Here's the same thing done to a cat (poor kitty):

 $ cat 1</dev/null
 test
 cat: write error: Bad file descriptor

No, this is also wrong. cat reads and writes, "hello world" only writes. 
Consider:

echo meh 1</dev/null

Always succeeds.


Andrei

Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:

On Tue, 11 Jun 2013 01:39:47 -0400
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 12:16 AM, Infiltrator wrote:
 On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu
 wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

 Wouldn't stdin be fd 0?

 David

 What David said. Here's the same thing done to a cat (poor kitty):

 $ cat 1</dev/null
 test
 cat: write error: Bad file descriptor

 
 No, this is also wrong. cat reads and writes, "hello world" only
 writes. Consider:
 
 echo meh 1</dev/null
 
 Always succeeds.
 

I just tried both on Debian 6:

nick debian6:~$ cat 1< /dev/null
cfws
cat: write error: Bad file descriptor
nick debian6:~$ echo meh 1< /dev/null
bash: echo: write error: Bad file descriptor

Maybe OSX behaves differently?

Jacob Carlborg <doob me.com> writes:

On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using 
another shell?

-- 
/Jacob Carlborg

Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:

On Tue, 11 Jun 2013 09:09:43 +0200
Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 
 I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using 
 another shell?
 

I think I remember him saying somewhere that he uses zsh, although I
can't look it up or test it on zsh at the moment.

"estew" <estewh gmail.com> writes:

On Tuesday, 11 June 2013 at 10:55:16 UTC, Nick Sabalausky wrote:
 On Tue, 11 Jun 2013 09:09:43 +0200
 Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 
 I get the same on Mac OS X 10.6.3 using bash. Is Andrei 
 perhaps using another shell?
 

 I think I remember him saying somewhere that he uses zsh, 
 although I
 can't look it up or test it on zsh at the moment.

Just tried zsh, bash, sh..

$ echo meh 1< /dev/null
zsh: no output
bash: Bad file descriptor
sh: Bad file descriptor

$ echo meh 2< /dev/null
zsh: meh
bash: meh
sh: meh

Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:

On Tue, 11 Jun 2013 13:17:13 +0200
"estew" <estewh gmail.com> wrote:

 On Tuesday, 11 June 2013 at 10:55:16 UTC, Nick Sabalausky wrote:
 On Tue, 11 Jun 2013 09:09:43 +0200
 Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 
 I get the same on Mac OS X 10.6.3 using bash. Is Andrei 
 perhaps using another shell?
 

 I think I remember him saying somewhere that he uses zsh, 
 although I
 can't look it up or test it on zsh at the moment.

 
 Just tried zsh, bash, sh..
 
 $ echo meh 1< /dev/null
 zsh: no output
 bash: Bad file descriptor
 sh: Bad file descriptor
 
 $ echo meh 2< /dev/null
 zsh: meh
 bash: meh
 sh: meh
 

Hmm, yea, sounds like it may not have been the best test in the first
place after all if even the shells and stdout-vs-stderr don't all agree
with each other.

I'm no unix expert, but wouldn't 1</dev/null just mean "If the program
tries to read input from stdout, feed it input from /dev/null instead"?
Or is that really a known way to force stdout to be read-only?

Jacob Carlborg <doob me.com> writes:

On 2013-06-11 12:55, Nick Sabalausky wrote:

 I think I remember him saying somewhere that he uses zsh, although I
 can't look it up or test it on zsh at the moment.

With zsh and the first one I get:

cat: stdout: Bad file descriptor

With the second one I get nothing and 0 as the exit code.

-- 
/Jacob Carlborg

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using
 another shell?

I use zsh. Indeed under bash I can reproduce the failure. But if I run 
the printf-based test, it exits successfully.

Andrei

"Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:

On Tuesday, 11 June 2013 at 12:37:53 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 I get the same on Mac OS X 10.6.3 using bash. Is Andrei 
 perhaps using
 another shell?

 I use zsh. Indeed under bash I can reproduce the failure. But 
 if I run the printf-based test, it exits successfully.

 Andrei

This is a good point, I kind of wondered why bash was the one 
reporting echo failed.

=?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:

Andrei Alexandrescu wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using
 another shell?

=20
 I use zsh. Indeed under bash I can reproduce the failure. But if I run =

 the printf-based test, it exits successfully.
=20

	Note that for zsh, "echo" is a builtin command, whereas bash calls
the /usr/bin/echo executable...

		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 14:44:17 -0400, J=C3=A9r=C3=B4me M. Berger <jeberger=
 free.fr>  =

wrote:

 Andrei Alexandrescu wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

 I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps usin=



g
 another shell?

 I use zsh. Indeed under bash I can reproduce the failure. But if I ru=


n
 the printf-based test, it exits successfully.

 	Note that for zsh, "echo" is a builtin command, whereas bash calls
 the /usr/bin/echo executable...

This is so not true :)

Stevens-MacBook-Pro:testd steves$ echo $SHELL
/bin/bash
Stevens-MacBook-Pro:testd steves$ type echo
echo is a shell builtin

echo has been a shell builtin since bourne shell (/bin/sh)

But your point is a good one, built-in commands very much will behave  =

differently than separate programs.  For one, echo has to clean up after=
  =

itself since it doesn't spawn a new process.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 12:04 AM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

 Wouldn't stdin be fd 0?

 David

Oh indeed my bad. But the test is still flawed. Consider:

import std.stdio;
int main()
{
     return printf("test\n") < 0;
}

This should return 1 if printf fails. It succeeds for 1</dev/null.


Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 01:37:52 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 12:04 AM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

 Wouldn't stdin be fd 0?

 David

 Oh indeed my bad. But the test is still flawed. Consider:

 import std.stdio;
 int main()
 {
      return printf("test\n") < 0;
 }

 This should return 1 if printf fails. It succeeds for 1</dev/null.

This actually gave me an idea of why it doesn't fail.

This code DOES fail:

import std.stdio;

int main()
{
     writeln("hello");
     std.stdio.stdout.flush();
     return 0;
}

Here is the (interesting) issue:

FILE *, and likewise std.stdio.File, which uses FILE * as it's  
implementation, is a buffered stream.

An interesting problem to solve for buffered streams is when to flush the  
buffer.  The most efficient thing to do is to wait until the entire buffer  
is full, then flush it (call the syscall to write the data to the  
handle/descriptor).  However, in an INTERACTIVE console, you want to flush  
more frequently, in case the data is coming out sporadically.

HOWEVER, the C runtime first checks to see if the file descriptor is a  
console.  If it is, it sets the flag indicating that it should flush on  
newlines, otherwise, it does not.  Instead of seeing 4K (or whatever  
buffer size is) of data at a time, the user sees 1 line at a time, much  
more parseable by a human.  This is why cat'ing a file to another file is  
much more efficient than cat'ing it to the console.

Important to note is that /dev/null is NOT a console :)

So what is happening is, writeln("hello") is writing "hello\n" to the FILE  
*, which sits in the buffer, no flushing occurs, because /dev/null is not  
a console.  However, upon calling flush, it fails, because the file  
descriptor is invalid.

Upon exit, std.stdio.stdout relies on core.stdc.stdio.stdout to flush the  
data (in fact, I don't even think stdout's dtor is called).  And C's  
stdout does not throw an exception or alter the exit code.  This is why  
you have no visible errors.

If D did NOT rely on C's FILE *, it may have a chance to throw an  
exception on program exit (but likely wouldn't because the buffer is  
likely GC based and therefore can't be accessed on the dtor :)

Re: different shells behaving differently, depending on how the individual  
shells are implemented can affect how this plays out.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; 
people wanted to eliminate it for efficiency reasons.)

We could introduce a flush() with throw in std.stdiobase.


Andrei

"Lars T. Kyllingstad" <public kyllingen.net> writes:

On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do 
 a flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.

As in a module destructor?  Isn't it better to let the error pass 
silently rather than throwing an exception that can't be caught?

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 13:01:41 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a  
 flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.

 As in a module destructor?  Isn't it better to let the error pass  
 silently rather than throwing an exception that can't be caught?

I agree.  This should not be a failure at that point.

If you want to induce and catch failure, put std.stdio.stdout.flush() at  
the end of your main function, like I did.

-Steve

"QAston" <qaston gmail.com> writes:

On Tuesday, 11 June 2013 at 17:01:43 UTC, Lars T. Kyllingstad 
wrote:
 Ah, I suspected so. (At a point in D's history writeln() did 
 do a flush; people wanted to eliminate it for efficiency 
 reasons.)

 We could introduce a flush() with throw in std.stdiobase.

 As in a module destructor?  Isn't it better to let the error 
 pass silently rather than throwing an exception that can't be 
 caught?

Passing silently on errors would contradict with several talks 
Andrei and Walter have given about how D's error handling is much 
better than C/C++ one (there are some HelloWorld program 
comparisions from those languages available on YT), I think they 
want that claim to remain valid. To me it is important to have 
error reported whenever something fails and have it as fast as 
possible - image a db backup system which fails silently.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a
 flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.

 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a 
diagnostic and exit with a nonzero error code, which is useful.

Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/11/2013 11:38 AM, Andrei Alexandrescu wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a
 flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.

 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

 It will not be caught but will cause the entire program to print a diagnostic
 and exit with a nonzero error code, which is useful.

The flush should be in the termination code (not in writeln), where it should
throw.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

 It will not be caught but will cause the entire program to print a  
 diagnostic and exit with a nonzero error code, which is useful.

How exactly is this useful?  What if you returned a different error code  
and you prefer that to come through?

Also note, if stderr isn't working, "printing" anything diagnostic will be  
gone (I wonder what happens in that case?)

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 3:25 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

 It will not be caught but will cause the entire program to print a
 diagnostic and exit with a nonzero error code, which is useful.

 How exactly is this useful? What if you returned a different error code
 and you prefer that to come through?

I think that can be very useful.

Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/11/2013 12:25 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

 It will not be caught but will cause the entire program to print a diagnostic
 and exit with a nonzero error code, which is useful.

 How exactly is this useful?

I used to entertain myself and friends by setting up unix so the disk is nearly 
full, and then trying various commands that produced output, and sending the 
output to the disk.

Because so many C programs fail to check for failed output, all kinds of 
entertaining random things would happen.

Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:

On Tue, 11 Jun 2013 16:42:43 -0700
Walter Bright <newshound2 digitalmars.com> wrote:
 
 I used to entertain myself and friends by setting up unix so the disk
 is nearly full, and then trying various commands that produced
 output, and sending the output to the disk.
 
 Because so many C programs fail to check for failed output, all kinds
 of entertaining random things would happen.
 

http://xkcd.com/1084/

Unix's design is amazingly powerful, but it's also interesting
how much potential that creates for things to get seriously screwed
up. :)

"Lars T. Kyllingstad" <public kyllingen.net> writes:

On Tuesday, 11 June 2013 at 18:38:51 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error 
 pass
 silently rather than throwing an exception that can't be 
 caught?

 It will not be caught but will cause the entire program to 
 print a diagnostic and exit with a nonzero error code, which is 
 useful.

Then we should also provide a hook so the user has the option of 
handling it.  For example, core.exception could provide

   void setStaticDestructorExceptionHandler(
       void function(string module, Throwable t) handler);

or similar.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 3:36 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 18:38:51 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

 It will not be caught but will cause the entire program to print a
 diagnostic and exit with a nonzero error code, which is useful.

 Then we should also provide a hook so the user has the option of
 handling it. For example, core.exception could provide

 void setStaticDestructorExceptionHandler(
 void function(string module, Throwable t) handler);

 or similar.

That's nice but but seems overdesign. I'd say let's fix the bug for now.

Andrei

"Denis Koroskin" <2korden gmail.com> writes:

On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do 
 a flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.


 Andrei

The best solution would be for writeln() to throw on use, and I 
think it's fairly easy to implement: just flush once after using 
the file descriptor for the first time, and throw if it fails.

While it doesn't cover a case where file descriptor becomes 
non-writable during the program lifetime, it covers the most 
common case of file descriptor not being writable at all.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>  
wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a  
 flush; people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.


 Andrei

 The best solution would be for writeln() to throw on use, and I think  
 it's fairly easy to implement: just flush once after using the file  
 descriptor for the first time, and throw if it fails.

This is a good idea.

I think you meant "just flush once after using the FILE * for the first  
time"

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a flush;
 people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.


 Andrei

 The best solution would be for writeln() to throw on use, and I think it's
 fairly easy to implement: just flush once after using the file descriptor for
 the first time, and throw if it fails.

 This is a good idea.

 I think you meant "just flush once after using the FILE * for the first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

The fix is when main() returns to do the flush there.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 11 Jun 2013 23:42:33 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>  
 wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a  
 flush;
 people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.


 Andrei

 The best solution would be for writeln() to throw on use, and I think  
 it's
 fairly easy to implement: just flush once after using the file  
 descriptor for
 the first time, and throw if it fails.

 This is a good idea.

 I think you meant "just flush once after using the FILE * for the first  
 time"

 I think it's a bad idea, it'll muck up the buffering (i.e. make it  
 slower).

Note, it's only done once, the very first time anything is written.  The  
rest of the time, flushing follows normal procedure.

In effect, the first write confirms the FD is valid, then all writes after  
assume it stays valid.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/11/2013 8:55 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 23:42:33 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

 Ah, I suspected so. (At a point in D's history writeln() did do a flush;
 people wanted to eliminate it for efficiency reasons.)

 We could introduce a flush() with throw in std.stdiobase.


 Andrei

 The best solution would be for writeln() to throw on use, and I think it's
 fairly easy to implement: just flush once after using the file descriptor for
 the first time, and throw if it fails.

 This is a good idea.

 I think you meant "just flush once after using the FILE * for the first time"

 I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

 Note, it's only done once, the very first time anything is written.  The rest
of
 the time, flushing follows normal procedure.

 In effect, the first write confirms the FD is valid, then all writes after
 assume it stays valid.

I don't agree. Buffering is often done on page size boundaries - throwing out a 
random number of characters and then flushing will get it all wonky.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 00:23:36 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 8:55 PM, Steven Schveighoffer wrote:

 Note, it's only done once, the very first time anything is written.   
 The rest of
 the time, flushing follows normal procedure.

 In effect, the first write confirms the FD is valid, then all writes  
 after
 assume it stays valid.

 I don't agree. Buffering is often done on page size boundaries -  
 throwing out a random number of characters and then flushing will get it  
 all wonky.

What?  You only flush the 'random number' of characters once, the rest of  
the time you are flushing full buffers.  Where is the issue?

Or do you have a specific application in mind?

-Steve

"deadalnix" <deadalnix gmail.com> writes:

On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries 
 - throwing out a random number of characters and then flushing 
 will get it all wonky.

You clearly missed something in the discussion here. The proposal 
is to flush once at first use, so an Exception is thrown. Nothing 
change after that first flush at initialization, other flushes 
stay where they are.

Walter Bright <newshound2 digitalmars.com> writes:

On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries - throwing out
 a random number of characters and then flushing will get it all wonky.

 You clearly missed something in the discussion here. The proposal is to flush
 once at first use, so an Exception is thrown. Nothing change after that first
 flush at initialization, other flushes stay where they are.

Not at all. A flush forces a write to the disk - that's the point of it. Disks 
are not at all well tuned to writing a few bytes, they like to be written in 
aligned blocks of block sizes, and the I/O subsystem is designed for that.

This is why stdout has a flag in it saying if it is a "block oriented" or 
"character oriented" device. It makes a big difference. This proposal attempts 
to treat a block device like a character device. It will work, but it will 
perform poorly.

P.S. I've written device drivers for disks.

P.P.S. The solution is simple, as I said earlier. Just do a flush after main() 
exits. It happens anyway - done by the C stdio subsystem - I just propose doing 
it in the D code before it hands things back to the C runtime. This will entail 
no performance degradation.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 02:48:44 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries -  
 throwing out
 a random number of characters and then flushing will get it all wonky.

 You clearly missed something in the discussion here. The proposal is to  
 flush
 once at first use, so an Exception is thrown. Nothing change after that  
 first
 flush at initialization, other flushes stay where they are.

 Not at all. A flush forces a write to the disk - that's the point of it.  
 Disks are not at all well tuned to writing a few bytes, they like to be  
 written in aligned blocks of block sizes, and the I/O subsystem is  
 designed for that.

 This is why stdout has a flag in it saying if it is a "block oriented"  
 or "character oriented" device. It makes a big difference. This proposal  
 attempts to treat a block device like a character device. It will work,  
 but it will perform poorly.

No, it does perform well.  You are still not understanding the proposal.

Here is a test:

Stevens-MacBook-Pro:testd steves$ cat testwrite.d
import std.stdio;

void main(string[] args)
{
     writeln("hello walter");
     if(args.length > 1 && args[1] == "flush")
         stdout.flush();
     foreach(i; 0..10_000_000)
         writeln("hello walter");
}

Stevens-MacBook-Pro:testd steves$ ~/dmd-2.063/osx/bin/dmd testwrite.d
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m2.466s
user	0m1.323s
sys	0m0.230s
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m2.077s
user	0m1.296s
sys	0m0.202s
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m2.121s
user	0m1.289s
sys	0m0.203s
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m1.997s
user	0m1.297s
sys	0m0.202s
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m2.495s
user	0m1.330s
sys	0m0.210s
Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt

real	0m2.316s
user	0m1.309s
sys	0m0.207s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m2.024s
user	0m1.291s
sys	0m0.221s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m1.943s
user	0m1.287s
sys	0m0.219s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m1.792s
user	0m1.274s
sys	0m0.217s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m2.026s
user	0m1.286s
sys	0m0.219s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m1.971s
user	0m1.285s
sys	0m0.222s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m2.252s
user	0m1.288s
sys	0m0.219s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m2.243s
user	0m1.293s
sys	0m0.218s
Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt

real	0m2.290s
user	0m1.303s
sys	0m0.219s

I see no significant difference between the version that flushes after the  
first write of 13 bytes, and the version that does its first flush on  
buffer full.  In fact, the best performing test was when the flush occurs  
on the first write.  It doesn't mean flushing is better, it just means it  
has no impact.

 P.S. I've written device drivers for disks.

That's nice :)  I understand the concept of buffered disk output quite  
well too.

 P.P.S. The solution is simple, as I said earlier. Just do a flush after  
 main() exits. It happens anyway - done by the C stdio subsystem - I just  
 propose doing it in the D code before it hands things back to the C  
 runtime. This will entail no performance degradation.

This is not ideal.  Sometimes you will catch the error during the program  
(maybe at different locations), sometimes you exhibit an uncatchable  
error.  With the proposed solution, you catch it deterministically, at the  
very first write.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 11:18 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 02:48:44 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries -
 throwing out
 a random number of characters and then flushing will get it all wonky.

 You clearly missed something in the discussion here. The proposal is
 to flush
 once at first use, so an Exception is thrown. Nothing change after
 that first
 flush at initialization, other flushes stay where they are.

 Not at all. A flush forces a write to the disk - that's the point of
 it. Disks are not at all well tuned to writing a few bytes, they like
 to be written in aligned blocks of block sizes, and the I/O subsystem
 is designed for that.

 This is why stdout has a flag in it saying if it is a "block oriented"
 or "character oriented" device. It makes a big difference. This
 proposal attempts to treat a block device like a character device. It
 will work, but it will perform poorly.

 No, it does perform well. You are still not understanding the proposal.

I agree performance is not a problem. But it's just weird behavior. We 
should flush stdout termination, anything else would have to be 
carefully justified - and this is not.

Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We  
 should flush stdout termination, anything else would have to be  
 carefully justified - and this is not.

stdout is flushed on termination.  Your code just doesn't puke a stack  
trace if there is an issue after main exits.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We should
 flush stdout termination, anything else would have to be carefully justified -
 and this is not.

 stdout is flushed on termination.  Your code just doesn't puke a stack trace if
 there is an issue after main exits.

There is main(), and then there is _Dmain(). Take a look at
druntime/src/rt/dmain2.d

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 11:56:59 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We  
 should
 flush stdout termination, anything else would have to be carefully  
 justified -
 and this is not.

 stdout is flushed on termination.  Your code just doesn't puke a stack  
 trace if
 there is an issue after main exits.

 There is main(), and then there is _Dmain(). Take a look at  
 druntime/src/rt/dmain2.d

I don't get the point.

Upon process termination, the C runtime flushes stdout/stderr.

There is no reason to reinvent this wheel.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 2:36 PM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:56:59 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior.
 We should
 flush stdout termination, anything else would have to be carefully
 justified -
 and this is not.

 stdout is flushed on termination. Your code just doesn't puke a stack
 trace if
 there is an issue after main exits.

 There is main(), and then there is _Dmain(). Take a look at
 druntime/src/rt/dmain2.d

 I don't get the point.

 Upon process termination, the C runtime flushes stdout/stderr.

 There is no reason to reinvent this wheel.

Flushes but doesn't propagate the potential error to the return code.

Let's take further discussion to 
http://d.puremagic.com/issues/show_bug.cgi?id=10344. I think what we 
need to do is cut and dried.


Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

Yes, I understand your proposal quite well.

Your benchmark is seriously flawed. Most modern systems cache writes in memory, 
and have a delayed write to the media. This hides the problem.

Try writing to a floppy disk system with mem caching turned off.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

 Yes, I understand your proposal quite well.

 Your benchmark is seriously flawed. Most modern systems cache writes in  
 memory, and have a delayed write to the media. This hides the problem.

Sorry, but this is just bunk.  One uneven flush does not mess up all  
future aligned writes.  I/O is not that fragile.

If you can demonstrate your theory, I will concede.  Until you do, don't  
expect any more responses, it's no use arguing when you don't understand  
the problem.

-Steve

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 12:02 PM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well. You are still not understanding the proposal.

 Yes, I understand your proposal quite well.

 Your benchmark is seriously flawed. Most modern systems cache writes
 in memory, and have a delayed write to the media. This hides the problem.

 Sorry, but this is just bunk. One uneven flush does not mess up all
 future aligned writes. I/O is not that fragile.

 If you can demonstrate your theory, I will concede. Until you do, don't
 expect any more responses, it's no use arguing when you don't understand
 the problem.

You're right that calling fflush() only once upon the first call will 
not cause a performance problem. My argument is different - it's not a 
good thing to do.

Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

 Yes, I understand your proposal quite well.

 Your benchmark is seriously flawed. Most modern systems cache writes in
 memory, and have a delayed write to the media. This hides the problem.

 Sorry, but this is just bunk.  One uneven flush does not mess up all future
 aligned writes.  I/O is not that fragile.

 If you can demonstrate your theory, I will concede.  Until you do, don't expect
 any more responses, it's no use arguing when you don't understand the problem.

May I present source code?
-----------------------------
int fflush(FILE *fp)
{       int length;
         int result= 0;

         /* if fflush(NULL) flush all buffers */
         if (fp == NULL)
         {
                 if (flushall() >= 0)
                         result = 0;
         }
         else
         {
           /* don't flush buffer if we are not writing   */
         __fp_lock(fp);
         if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT &&
             (fp->_base
#ifdef BIGBUF
                 || fp->_seg
#endif
                         ))

#ifdef BIGBUF
                 if (length && _writex(fp->_file,fp->_base,length,fp->_seg)
                         != length)
                         fp->_flag |= _IOERR;
#else
                 if (length)
                 {   int nwritten = write(fp->_file,fp->_base,length);
                     /* The following check for isatty() is because:
                      *  #define WIN32_LEAN_AND_MEAN
                      *  #include <windows.h>
                      *  #include <stdio.h>
                      *  void main()
                      *  {
                      *      // Set console output to UTF-8 (one can use 'chcp 
65001' instead)
                      *      SetConsoleOutputCP( 65001 );
                      *      // Latin small letter e with acute
                      *      fputs( "Output utf-8 accented char \xc3\xa9\n... 
and the rest is cut of
                      *  }
                      * fails because WriteFile() apparently treats UTF-8
                      * sequences as 1 byte when writing to the console.
                      */
                     if (!nwritten || (nwritten != length &&
!isatty(fp->_file)))
                         fp->_flag |= _IOERR;
                 }
#endif
                 fp->_cnt = 0;
                 fp->_ptr = fp->_base;
         }
         else
                 fp->_cnt = 0;
         result = (ferror(fp)) ? EOF : 0;
         __fp_unlock(fp);
         }
         return result;
}
-----------------------------------
This does exactly what I said it does. It's from the Digital Mars C runtime 
library. Now you could argue that this code sux, and you might even be right. 
But there are a lot of implementations of C fflush out there - are you sure
that 
all of them will realign after a misaligned write?

And even if they do realign, you cannot write 10 bytes to a disk. You can only 
write a block or a sector. Which means the next write, even if aligned, has to 
write that block or sector again.

You are proposing that this repeated write of the first sector be done for all 
file output. You probably won't notice a speed difference if write caching is 
done, but that doesn't make it a good idea.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 12:40 PM, Walter Bright wrote:
 You are proposing that this repeated write of the first sector be done
 for all file output.

No, he's proposing that only the very first write flushes. Not every 
write. I'd argue it's a bizarre thing to do, but not that it's 
necessarily slow.

Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 10:07 AM, Andrei Alexandrescu wrote:
 On 6/12/13 12:40 PM, Walter Bright wrote:
 You are proposing that this repeated write of the first sector be done
 for all file output.

 No, he's proposing that only the very first write flushes. Not every write. I'd
 argue it's a bizarre thing to do, but not that it's necessarily slow.

I meant for every file being output.

And, again, it is forcing two writes of the first block, at a minimum. The perf 
cost can be hidden by write caching, but to rely on that is bad design and 
undermines how the I/O system is set up and tuned.

I predict a WTF bug report on this in the future, as someone with an unusual 
device attached is looking for the cause of why his D program writing to it 
behaves badly. Or he just writes it off as "D sux".

There's a *reason* why stdio distinguishes between block devices and character 
devices.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the  
 proposal.

 Yes, I understand your proposal quite well.

 Your benchmark is seriously flawed. Most modern systems cache writes in
 memory, and have a delayed write to the media. This hides the problem.

 Sorry, but this is just bunk.  One uneven flush does not mess up all  
 future
 aligned writes.  I/O is not that fragile.

 If you can demonstrate your theory, I will concede.  Until you do,  
 don't expect
 any more responses, it's no use arguing when you don't understand the  
 problem.

 May I present source code?
 -----------------------------
 int fflush(FILE *fp)
 {       int length;
          int result= 0;

          /* if fflush(NULL) flush all buffers */
          if (fp == NULL)
          {
                  if (flushall() >= 0)
                          result = 0;
          }
          else
          {
            /* don't flush buffer if we are not writing   */
          __fp_lock(fp);
          if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT &&
              (fp->_base
 #ifdef BIGBUF
                  || fp->_seg
 #endif
                          ))

 */
 #ifdef BIGBUF
                  if (length &&  
 _writex(fp->_file,fp->_base,length,fp->_seg)
                          != length)
                          fp->_flag |= _IOERR;
 #else
                  if (length)
                  {   int nwritten = write(fp->_file,fp->_base,length);
                      /* The following check for isatty() is because:
                       *  #define WIN32_LEAN_AND_MEAN
                       *  #include <windows.h>
                       *  #include <stdio.h>
                       *  void main()
                       *  {
                       *      // Set console output to UTF-8 (one can use  
 'chcp 65001' instead)
                       *      SetConsoleOutputCP( 65001 );
                       *      // Latin small letter e with acute
                       *      fputs( "Output utf-8 accented char  
 \xc3\xa9\n... and the rest is cut of
                       *  }
                       * fails because WriteFile() apparently treats UTF-8
                       * sequences as 1 byte when writing to the console.
                       */
                      if (!nwritten || (nwritten != length &&  
 !isatty(fp->_file)))
                          fp->_flag |= _IOERR;
                  }
 #endif
                  fp->_cnt = 0;
                  fp->_ptr = fp->_base;
          }
          else
                  fp->_cnt = 0;
          result = (ferror(fp)) ? EOF : 0;
          __fp_unlock(fp);
          }
          return result;
 }
 -----------------------------------
 This does exactly what I said it does. It's from the Digital Mars C  
 runtime library. Now you could argue that this code sux, and you might  
 even be right. But there are a lot of implementations of C fflush out  
 there - are you sure that all of them will realign after a misaligned  
 write?

If I understand correctly, the buffer is flushed, and ptr is reset to the  
base, the count is reset to 0.  I don't see any code in there that does  
any kind of realignment.  Are you suggesting that other fflush code  
doesn't do this?

 And even if they do realign, you cannot write 10 bytes to a disk. You  
 can only write a block or a sector. Which means the next write, even if  
 aligned, has to write that block or sector again.

That is one block or sector.  The write cache of the OS or the drive will  
probably absorb this hit anyway.

The performance hit is extremely negligible.  Not only that, but the  
hardware can differ from file system to file system.  You are going  
through the file system driver, through the disk driver, through the  
disk.  All of those pieces are written to optimize writes that SPECIFIC  
hardware.  There isn't much you can do to make this perform poorly.

The only performance hit you can really affect is the system call  
penalty.  And that is done by allowing the normal flushing routine to  
continue for subsequent writes.

 You are proposing that this repeated write of the first sector be done  
 for all file output. You probably won't notice a speed difference if  
 write caching is done, but that doesn't make it a good idea.

No, this is incorrect.  fflush is not called on subsequent writeln.

Not only that, but we only have to do this on FILE * that were initialized  
with unknown file descriptors (such as stdin/stdout/stderr).  If we use  
fopen, you don't have to do this.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 11:36 AM, Steven Schveighoffer wrote:
 If I understand correctly, the buffer is flushed, and ptr is reset to the base,
 the count is reset to 0.  I don't see any code in there that does any kind of
 realignment.

You write 7 bytes, fflush. Now, every 4K buffered write from then on is going
to 
write from 7..4096+7, 4096+7..8192+7, ...


 That is one block or sector.  The write cache of the OS or the drive will
 probably absorb this hit anyway.

Write caches are not always there.


 The performance hit is extremely negligible.

Only if there is a write cache (which can be turned off, and are often turned 
off for removable drives).


 Not only that, but the hardware can differ from file system to file system. 
You are going through the file
 system driver, through the disk driver, through the disk.  All of those pieces
 are written to optimize writes that SPECIFIC hardware.  There isn't much you
can
 do to make this perform poorly.

Embedded systems and custom devices often do not have sophisticated hardware or 
software drivers.


 You are proposing that this repeated write of the first sector be done for all
 file output. You probably won't notice a speed difference if write caching is
 done, but that doesn't make it a good idea.

 No, this is incorrect.  fflush is not called on subsequent writeln.

I didn't say it was. See my other reply. In any case, if fflush is not called
on 
subsequent writelns, it is still possible to have the first writeln succeed,
the 
subsequent writelns fail and have main() return success.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 3:40 PM, Walter Bright wrote:
 I didn't say it was. See my other reply. In any case, if fflush is not
 called on subsequent writelns, it is still possible to have the first
 writeln succeed, the subsequent writelns fail and have main() return
 success.

I'm a bit surprised that there's any other response to this than: "Oh 
okay, then flushing only after the first write or checking whether the 
file is writable in the beginning doesn't work".

Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 15:40:10 -0400, Walter Bright  =

<newshound2 digitalmars.com> wrote:

 On 6/12/2013 11:36 AM, Steven Schveighoffer wrote:
 If I understand correctly, the buffer is flushed, and ptr is reset to=


  =

 the base,
 the count is reset to 0.  I don't see any code in there that does any=


  =

 kind of
 realignment.

 You write 7 bytes, fflush. Now, every 4K buffered write from then on i=

s  =

 going to write from 7..4096+7, 4096+7..8192+7, ...

If this was such a big deal, then FILE * would guarantee the alignment  =

went back.  It's not that hard.

 That is one block or sector.  The write cache of the OS or the drive =


 =

 will
 probably absorb this hit anyway.

 Write caches are not always there.


 The performance hit is extremely negligible.

 Only if there is a write cache (which can be turned off, and are often=

  =

 turned off for removable drives).

The write cache is irrelevant.  The filesystem driver will dictate where=
  =

writes are done, and will likely cache.

And I have experience with turning them off -- some of my clients  =

requested that to avoid losing data that was in the write cache.  It doe=
s  =

not all of a sudden slow down the performance of the drive.

 Not only that, but the hardware can differ from file system to file  =


 system.  You are going through the file
 system driver, through the disk driver, through the disk.  All of tho=


se  =

 pieces
 are written to optimize writes that SPECIFIC hardware.  There isn't  =


 much you can
 do to make this perform poorly.

 Embedded systems and custom devices often do not have sophisticated  =

 hardware or software drivers.

The day that D runs on an embedded system, with no drivers or OS, BUT  =

provides a C runtime that D builds on, is the day I'll agree with you ;)=


But this whole discussion is academic at this point, J=C3=A9r=C3=B4me id=
entified  =

that we can simply check to see if the file descriptor is valid, without=
  =

flushing.

 I didn't say it was. See my other reply. In any case, if fflush is not=

  =

 called on subsequent writelns, it is still possible to have the first =

 =

 writeln succeed, the subsequent writelns fail and have main() return  =

 success.

This isn't the problem that was presented.  The problem that was present=
ed  =

is that given an invalid file descriptor, writeln happily works (as long=
  =

as you don't cause a flush) and does not throw.  This is unintuitive to =
 =

someone who is expecting writeln to choke on an invalid descriptor.

Subsequent writelns that fail throw an exception.  Any writeln that caus=
es  =

a flush will create an exception.

The rare corner case the "throw at end" will catch in addition is when t=
he  =

file descriptor becomes invalid on the final flush only.

I think we can probably implement both checks.  The 'throw at end' metho=
d  =

will catch both, but it would be nice to catch an invalid descriptor at =
 =

the call the causes the problem, not at the end when the program is most=
ly  =

gone.  It's like catching a memory error far away from the cause.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:
 If this was such a big deal, then FILE * would guarantee the alignment went
 back.  It's not that hard.

I know it isn't hard. But D's charter is not to go and patch the C runtime 
library, nor is that remotely practical across all the thousands of C 
implementations out there.


 And I have experience with turning them off -- some of my clients requested
that
 to avoid losing data that was in the write cache.  It does not all of a sudden
 slow down the performance of the drive.

I ran some experiments a while back writing to a slow SD card. There was a big 
difference in speed when turning write caching on and off. I bet you'll see 
quite a whopper of a difference if you try that with a floppy drive!

In other words, the difference shows up when you've got a fairly slow device. 
Write caching would not have been invented if it had no effect.


 But this whole discussion is academic at this point, Jérôme identified that
we
 can simply check to see if the file descriptor is valid, without flushing.

His fix is insufficient for other reasons as already pointed out.


 This isn't the problem that was presented.  The problem that was presented is
 that given an invalid file descriptor, writeln happily works (as long as you
 don't cause a flush) and does not throw.  This is unintuitive to someone who is
 expecting writeln to choke on an invalid descriptor.

That behavior is normal to anyone used to working with C stdio.

What is wrong and needs fixing is the program exiting with "success" indication 
when the output has actually failed.

"H. S. Teoh" <hsteoh quickfur.ath.cx> writes:

On Thu, Jun 13, 2013 at 11:11:06AM -0700, Walter Bright wrote:
 On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:

[...]
This isn't the problem that was presented.  The problem that was
presented is that given an invalid file descriptor, writeln happily
works (as long as you don't cause a flush) and does not throw.  This
is unintuitive to someone who is expecting writeln to choke on an
invalid descriptor.

 
 That behavior is normal to anyone used to working with C stdio.
 
 What is wrong and needs fixing is the program exiting with "success"
 indication when the output has actually failed.

Wouldn't an easy fix be for DMD to insert this into main()?

	scope(exit) if (fflush(stdout) != 0)
		throw new Exception(...);

	// ... rinse, repeat for whatever other fd's we care about

Of course, instead of hardcoding stdio into DMD, the scope(exit) can
call some kind of library cleanup hook, maybe something like:

	scope(exit) _druntimeCleanup();

and the throw can happen inside _druntimeCleanup, which should cause the
exit status to be non-zero.

Not sure how to make this exception user-catchable, though.


T

-- 
Ph.D. = Permanent head Damage

Jacob Carlborg <doob me.com> writes:

On 2013-06-13 20:11, Walter Bright wrote:

 I know it isn't hard. But D's charter is not to go and patch the C
 runtime library, nor is that remotely practical across all the thousands
 of C implementations out there.

Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't 
rely on the C runtime library in this case, if possible.

 That behavior is normal to anyone used to working with C stdio.

But we're using D here, not C. You have to stop assume that everyone 
using D knows C or C++. Just because you do doesn't mean everyone else does.

-- 
/Jacob Carlborg

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 12:10 AM, Jacob Carlborg wrote:
 On 2013-06-13 20:11, Walter Bright wrote:

 I know it isn't hard. But D's charter is not to go and patch the C
 runtime library, nor is that remotely practical across all the thousands
 of C implementations out there.

 Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't rely on
 the C runtime library in this case, if possible.

 That behavior is normal to anyone used to working with C stdio.

 But we're using D here, not C. You have to stop assume that everyone using D
 knows C or C++. Just because you do doesn't mean everyone else does.


D is set up to work hand-in-glove with C. This isn't going to change, as such 
compatibility is a big feature of D.

Part of that is seamlessly working with existing C I/O. D code that does 
writeln's interleaved with C code that does printf's should work as expected.

Since this particular issue can be easily fixed without "fixing" the C runtime, 
there is no motivation to do so, either.

"Denis Koroskin" <2korden gmail.com> writes:

On Thursday, 13 June 2013 at 18:11:12 UTC, Walter Bright wrote:
 What is wrong and needs fixing is the program exiting with 
 "success" indication when the output has actually failed.

What's wrong is a program existing with a status code *different* 
from one that was supplied by a programmer.

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

12-Jun-2013 22:36, Steven Schveighoffer пишет:
 On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright

...
 No, this is incorrect.  fflush is not called on subsequent writeln.

 Not only that, but we only have to do this on FILE * that were
 initialized with unknown file descriptors (such as
 stdin/stdout/stderr).  If we use fopen, you don't have to do this.

Steve, the only unchecked place right now is in the _last_ flush. Even 
if you flush on first call it won't help some other forms of errors 
(other then it's not writable to begin with). The only thing it will do 
is surprise certain kinds of applications that use fdopen with "passed 
in" sockets and whatnot. Basically a year later we'll see puzzled folks 
with strace printouts.

The only thing that's broken here is that C-runtime doesn't care for 
errors on file descriptors it flushes at termination.

Now since it's D we may as flush global files at end of main (in static 
dtor) while still in the "D world" then throw. This however leaves 
flushes on uncollected Files that were allocated with new or as a part 
of GC arrays. Given that these cases are bogus to begin with I'd say go 
ahead with

shared static ~this()
{
	stdout.flush();
	stderr.flush();
}


-- 
Dmitry Olshansky

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 16:41:03 -0400, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:

 12-Jun-2013 22:36, Steven Schveighoffer =D0=BF=D0=B8=D1=88=D0=B5=D1=82=

:
 On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright

 ...
 No, this is incorrect.  fflush is not called on subsequent writeln.

 Not only that, but we only have to do this on FILE * that were
 initialized with unknown file descriptors (such as
 stdin/stdout/stderr).  If we use fopen, you don't have to do this.

 Steve, the only unchecked place right now is in the _last_ flush. Even=

  =

 if you flush on first call it won't help some other forms of errors  =

 (other then it's not writable to begin with). The only thing it will d=

o  =

 is surprise certain kinds of applications that use fdopen with "passed=

  =

 in" sockets and whatnot. Basically a year later we'll see puzzled folk=

s  =

 with strace printouts.

The two problems described are separate problems.  Both can be addressed=
  =

differently.

I think it's worth putting the flush at the end, and generating an  =

exception.  But if we can catch the error on intialization, or at least =
on  =

first use, it would be beneficial.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 If you can demonstrate your theory, I will concede.  Until you do, don't expect
 any more responses, it's no use arguing when you don't understand the problem.

There's another reason it won't work. If there are 4 bytes of free space left
on 
the drive:

void main() {
    writeln("foo");   // succeeds
    writeln("bar");   // silently fails
}

"Denis Koroskin" <2korden gmail.com> writes:

On Wednesday, 12 June 2013 at 06:48:48 UTC, Walter Bright wrote:
 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright 
 wrote:
 I don't agree. Buffering is often done on page size 
 boundaries - throwing out
 a random number of characters and then flushing will get it 
 all wonky.

 You clearly missed something in the discussion here. The 
 proposal is to flush
 once at first use, so an Exception is thrown. Nothing change 
 after that first
 flush at initialization, other flushes stay where they are.

 Not at all. A flush forces a write to the disk - that's the 
 point of it. Disks are not at all well tuned to writing a few 
 bytes, they like to be written in aligned blocks of block 
 sizes, and the I/O subsystem is designed for that.

 This is why stdout has a flag in it saying if it is a "block 
 oriented" or "character oriented" device. It makes a big 
 difference. This proposal attempts to treat a block device like 
 a character device. It will work, but it will perform poorly.

 P.S. I've written device drivers for disks.

 P.P.S. The solution is simple, as I said earlier. Just do a 
 flush after main() exits. It happens anyway - done by the C 
 stdio subsystem - I just propose doing it in the D code before 
 it hands things back to the C runtime. This will entail no 
 performance degradation.

How about we test it before making any claims?

I wrote a simple test program that writes 4gigs of '!' to stdout, 
and measured it in 2 ways:

time ./a.out > /dev/null
time ./a.out > file.txt

Here are the test runs (a warmup followed by 3 runs of each):

Output to /dev/null W/O FLUSH
real	0m0.188s 0m0.186s 0m0.190s
user	0m0.166s 0m0.165s 0m0.168s
sys	0m0.021s 0m0.020s 0m0.021s

/dev/null WITH FLUSH
real	0m0.168s 0m0.166s 0m0.163s
user	0m0.145s 0m0.144s 0m0.142s
sys	0m0.022s 0m0.021s 0m0.021s

Output to file.txt W/O FLUSH
real	0m10.131s 0m11.766s 0m9.900s
user	0m0.444s 0m0.443s 0m0.451s
sys	0m5.352s 0m5.324s 0m5.308s

real	0m9.474s 0m9.971s 0m10.115s
user	0m0.233s 0m0.241s 0m0.233s
sys	0m5.588s 0m5.616s 0m5.591s

Does it look like it's slower? To me it looks like it's actually 
FASTER with a flush, although I don't know why. Try it yourself. 
The code that I tested with is below (apologies for using C).

#include <stdio.h>
#include <string.h>

int main()
{
     char buffer[4096];
     memset(buffer, '!', 4096);
     fwrite(buffer, 1, 100, stdout);
     //fflush(stdout);
     for (int i = 0; i < 1024*1024; ++i) {
       fwrite(buffer, 1, 4096, stdout);
     }
     return 0;
}

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's actually FASTER with a
 flush, although I don't know why.

Because of write caching, you can get very different results from one run to
the 
next of the same program.

"Denis Koroskin" <2korden gmail.com> writes:

On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's 
 actually FASTER with a
 flush, although I don't know why.

 Because of write caching, you can get very different results 
 from one run to the next of the same program.

I re-ran them a few times obviously. The results were consistent.

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 14 June 2013 at 08:24:54 UTC, Denis Koroskin wrote:
 On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's 
 actually FASTER with a
 flush, although I don't know why.

 Because of write caching, you can get very different results 
 from one run to the next of the same program.

 I re-ran them a few times obviously. The results were 
 consistent.

That is super weird and unexpected to me, but I trust you.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 14 Jun 2013 04:24:53 -0400, Denis Koroskin <2korden gmail.com>  
wrote:

 On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's actually  
 FASTER with a
 flush, although I don't know why.

 Because of write caching, you can get very different results from one  
 run to the next of the same program.

 I re-ran them a few times obviously. The results were consistent.

Walter is right to a point -- caching can effect this behavior.

There are several levels of caching at play here, and all of them are  
working hard to optimize the writing to disk.  That likely is more of a  
cause than the flushing of the first element.

But in that sense, his argument is also broken -- you have to be in a very  
special situation (no caching, no drivers, etc.) in order for this to  
matter.

I had a similar test, and my results were not consistently one better than  
the other, but basically flushing had no discernible effect.

And quite honestly, if disk performance were dependent on user-land C  
buffering schemes, C runtime writers would pay more attention to the C  
buffering scheme and make sure it performs well!  i.e. if a single  
misaligned write can mess up gigabytes of output, then the next write  
would correct it.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 8:22 AM, Steven Schveighoffer wrote:
 And quite honestly, if disk performance were dependent on user-land C buffering
 schemes, C runtime writers would pay more attention to the C buffering scheme
 and make sure it performs well!

Actually, this has been a common oversight of C runtime writers. My compiler
was 
able to consistently beat others in I/O back in the 80's because I used a
better 
buffering scheme in the runtime.

It's not the only anecdote I have about that, either.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 14 Jun 2013 13:58:33 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 8:22 AM, Steven Schveighoffer wrote:
 And quite honestly, if disk performance were dependent on user-land C  
 buffering
 schemes, C runtime writers would pay more attention to the C buffering  
 scheme
 and make sure it performs well!

 Actually, this has been a common oversight of C runtime writers. My  
 compiler was able to consistently beat others in I/O back in the 80's  
 because I used a better buffering scheme in the runtime.

The 80's are a long time ago.  Plus, your posting of the source code  
pretty much refutes that your buffering scheme takes into account how  
important this should be.  It ignores alignment of writes if you add an  
fflush in between writes.

 It's not the only anecdote I have about that, either.

That's good, because the floppy DOS days are pretty much over :)

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 11:43 AM, Steven Schveighoffer wrote:
 The 80's are a long time ago.

But old code can live on in surprising ways.


 Plus, your posting of the source code pretty much
 refutes that your buffering scheme takes into account how important this should
 be.  It ignores alignment of writes if you add an fflush in between writes.

One aspect of its buffering scheme being inferior doesn't mean the rest of it 
is. There are a rather large number of issues with doing good I/O.


 It's not the only anecdote I have about that, either.

 That's good, because the floppy DOS days are pretty much over :)

You're overlooking that there are a LOT of C runtimes in use out there, and 
testing on one of them in one system doesn't say anything about other systems, 
and many of them (such as for embedded systems) are fairly primitive.

I've never seen two C runtimes use quite the same buffering logic.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 14 Jun 2013 15:17:00 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 11:43 AM, Steven Schveighoffer wrote:
 The 80's are a long time ago.

 But old code can live on in surprising ways.

The code living on is not what I'm talking about.  The *assumptions*  
living on is the problem :)  Old code can be written very carefully to  
avoid situations that are not present anymore.

 Plus, your posting of the source code pretty much
 refutes that your buffering scheme takes into account how important  
 this should
 be.  It ignores alignment of writes if you add an fflush in between  
 writes.

 One aspect of its buffering scheme being inferior doesn't mean the rest  
 of it is. There are a rather large number of issues with doing good I/O.

I'm not saying it's inferior, just that it's not as big a deal as you say  
it is.  At least not any more.  I can see that it might have been very  
important with an OS like DOS.

 It's not the only anecdote I have about that, either.

 That's good, because the floppy DOS days are pretty much over :)

 You're overlooking that there are a LOT of C runtimes in use out there,  
 and testing on one of them in one system doesn't say anything about  
 other systems, and many of them (such as for embedded systems) are  
 fairly primitive.

The same can be said for your runtime.  That is, your choices of buffering  
may not do well on other systems for which other buffering schemes may be  
tuned.

I think in the end, we are optimizing here in the wrong place.  If a  
specific hardware/software combination requires specific buffering, the  
place to handle it is in the runtime, not code on top of it.  If the C  
runtime that D uses isn't up to snuff, let's use a different scheme, or  
abandon it all together *for that specific device*.

Not that this is the situation we currently have, where D only runs on  
full-blown PCs...

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.  If a specific
 hardware/software combination requires specific buffering, the place to handle
 it is in the runtime, not code on top of it.  If the C runtime that D uses
isn't
 up to snuff, let's use a different scheme, or abandon it all together *for that
 specific device*.

There's simply no reason to do that. There has been a fix proposed that not
only 
solves the problem correctly (your solution is incomplete), it does not require 
any improvements to the underlying C runtime.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Fri, 14 Jun 2013 16:31:35 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.  If a  
 specific
 hardware/software combination requires specific buffering, the place to  
 handle
 it is in the runtime, not code on top of it.  If the C runtime that D  
 uses isn't
 up to snuff, let's use a different scheme, or abandon it all together  
 *for that
 specific device*.

 There's simply no reason to do that. There has been a fix proposed that  
 not only solves the problem correctly (your solution is incomplete), it  
 does not require any improvements to the underlying C runtime.

This doesn't fix the "problem" of when someone calls

fflush(stdout);

In their code somewhere.  Your contention is this messes up all the writes  
beyond that.  If that is such a problem, we should avoid using such a  
problematic library.  We are after all, in control of the D runtime.

But my argument is purposely ad absurdum, because it clearly is not a  
problem for any systems that exist today (as evidence shows).  You are  
arguing unproven hypotheses about systems that are long dead with no hope  
of running D code.  I respectfully bow out to work on more important  
things.

-Steve

"deadalnix" <deadalnix gmail.com> writes:

On Friday, 14 June 2013 at 20:31:39 UTC, Walter Bright wrote:
 On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.
  If a specific
 hardware/software combination requires specific buffering, the 
 place to handle
 it is in the runtime, not code on top of it.  If the C runtime 
 that D uses isn't
 up to snuff, let's use a different scheme, or abandon it all 
 together *for that
 specific device*.

 There's simply no reason to do that. There has been a fix 
 proposed that not only solves the problem correctly (your 
 solution is incomplete), it does not require any improvements 
 to the underlying C runtime.

I don't think any of the proposed fixes is a good idea. With he 
one you are pushing for, the program can fail behind the feet of 
the programmer, which is not better.

Walter Bright <newshound2 digitalmars.com> writes:

On 6/14/2013 10:45 PM, deadalnix wrote:
 I don't think any of the proposed fixes is a good idea. With he one you are
 pushing for, the program can fail behind the feet of the programmer, which is
 not better.

Please explain.

"deadalnix" <deadalnix gmail.com> writes:

On Saturday, 15 June 2013 at 06:49:00 UTC, Walter Bright wrote:
 On 6/14/2013 10:45 PM, deadalnix wrote:
 I don't think any of the proposed fixes is a good idea. With 
 he one you are
 pushing for, the program can fail behind the feet of the 
 programmer, which is
 not better.

 Please explain.

The solution that consist in flushing at first use is a 
surprising behavior, and look more like a weird hack than a 
solution.

The solution that consist into flushing in main at the end of the 
program is problematic as well. At this point, from programmer 
perspective, the program ran fun and is terminated successfully. 
Still the whole stuff will explode under its feet, in the 
runtime. That isn't something we should promote.

Walter Bright <newshound2 digitalmars.com> writes:

On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the program
ran
 fun and is terminated successfully. Still the whole stuff will explode under
its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved 
as intended?

"deadalnix" <deadalnix gmail.com> writes:

On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of 
 the program is
 problematic as well. At this point, from programmer 
 perspective, the program ran
 fun and is terminated successfully. Still the whole stuff will 
 explode under its
 feet, in the runtime. That isn't something we should promote.

 If the output failed to happen, how could the program have 
 successfully behaved as intended?

 From programmer's perspective.

Walter Bright <newshound2 digitalmars.com> writes:

On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the program
ran
 fun and is terminated successfully. Still the whole stuff will explode under
its
 feet, in the runtime. That isn't something we should promote.

 If the output failed to happen, how could the program have successfully
 behaved as intended?

  From programmer's perspective.

I'm sorry, that makes no sense to me. In fact, this thread started out
reporting 
an article by a programmer who was unhappy that the program exited normally but 
the output didn't happen.

Defaulting to ignoring errors and blithely proceeding is not usually considered 
a best practice.

"deadalnix" <deadalnix gmail.com> writes:

On Saturday, 15 June 2013 at 10:08:38 UTC, Walter Bright wrote:
 On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end 
 of the program is
 problematic as well. At this point, from programmer 
 perspective, the program ran
 fun and is terminated successfully. Still the whole stuff 
 will explode under its
 feet, in the runtime. That isn't something we should promote.

 If the output failed to happen, how could the program have 
 successfully
 behaved as intended?

 From programmer's perspective.

 I'm sorry, that makes no sense to me.

That is because you aren't reading what is written. See 2/

 Defaulting to ignoring errors and blithely proceeding is not 
 usually considered a best practice.

I have never written that. That explains 1/

Walter Bright <newshound2 digitalmars.com> writes:

On 6/15/2013 3:24 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 10:08:38 UTC, Walter Bright wrote:
 On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the
 program ran
 fun and is terminated successfully. Still the whole stuff will explode
 under its
 feet, in the runtime. That isn't something we should promote.

 If the output failed to happen, how could the program have successfully
 behaved as intended?

 From programmer's perspective.

 I'm sorry, that makes no sense to me.

 That is because you aren't reading what is written. See 2/

 Defaulting to ignoring errors and blithely proceeding is not usually
 considered a best practice.

 I have never written that. That explains 1/

I have no idea what you're talking about.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/15/13 12:04 PM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the
 program is
 problematic as well. At this point, from programmer perspective, the
 program ran
 fun and is terminated successfully. Still the whole stuff will
 explode under its
 feet, in the runtime. That isn't something we should promote.

 If the output failed to happen, how could the program have
 successfully behaved as intended?

  From programmer's perspective.

You may want to discuss alternative solutions in the bug report: 
http://d.puremagic.com/issues/show_bug.cgi?id=10344

Thanks,

Andrei

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

 I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

 The fix is when main() returns to do the flush there.

I agree. One question is, should we flush all buffers upon termination? 
fflush(null) would do so. 
http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html

Andrei

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 5:30 AM, Andrei Alexandrescu wrote:
 On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

 I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

 The fix is when main() returns to do the flush there.

 I agree. One question is, should we flush all buffers upon termination?
 fflush(null) would do so.
 http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html

I haven't heard about the fflush(null) thing. That may be gnu-specific. I don't 
think that works on Windows.

But yes, all open files should be flushed on termination. It's what C's stdio 
does already.

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 8:59 AM, Walter Bright wrote:
 I haven't heard about the fflush(null) thing. That may be gnu-specific. I don't
 think that works on Windows.

Followup: I have indeed heard about the fflush(null) thing, as it's in the 
fflush code I wrote several eons ago!

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 11:59 AM, Walter Bright wrote:
 On 6/12/2013 5:30 AM, Andrei Alexandrescu wrote:
 On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

 I think it's a bad idea, it'll muck up the buffering (i.e. make it
 slower).

 The fix is when main() returns to do the flush there.

 I agree. One question is, should we flush all buffers upon termination?
 fflush(null) would do so.
 http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html

 I haven't heard about the fflush(null) thing. That may be gnu-specific.
 I don't think that works on Windows.

It's quite an old standard feature albeit obscure, I found it and forgot 
about it a couple of times already. Works everywhere, see 
http://www.cplusplus.com/reference/cstdio/ferror/ and 
http://msdn.microsoft.com/en-us/library/9yky46tz(v=vs.80).aspx

 But yes, all open files should be flushed on termination. It's what C's
 stdio does already.

http://d.puremagic.com/issues/show_bug.cgi?id=10344


Andrei

"deadalnix" <deadalnix gmail.com> writes:

On Wednesday, 12 June 2013 at 16:34:08 UTC, Andrei Alexandrescu 
wrote:
 It's quite an old standard feature albeit obscure, I found it 
 and forgot about it a couple of times already. Works 
 everywhere, see 
 http://www.cplusplus.com/reference/cstdio/ferror/ and 
 http://msdn.microsoft.com/en-us/library/9yky46tz(v=vs.80).aspx

 But yes, all open files should be flushed on termination. It's 
 what C's
 stdio does already.

 http://d.puremagic.com/issues/show_bug.cgi?id=10344

How come that in a first place, we have a file descriptor that 
isn't working ?

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/11/13 10:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>
 wrote:
 The best solution would be for writeln() to throw on use, and I think
 it's fairly easy to implement: just flush once after using the file
 descriptor for the first time, and throw if it fails.

 This is a good idea.

 I think you meant "just flush once after using the FILE * for the first
 time"

 -Steve

Yah, but now we're looking at extra state associated with each file etc. 
Also it's kind of unexpected.

Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 08:26:43 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 10:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>
 wrote:
 The best solution would be for writeln() to throw on use, and I think
 it's fairly easy to implement: just flush once after using the file
 descriptor for the first time, and throw if it fails.

 This is a good idea.

 I think you meant "just flush once after using the FILE * for the first
 time"

 -Steve

 Yah, but now we're looking at extra state associated with each file etc.

There is room.  Each File is allocated on the heap.

 Also it's kind of unexpected.

According to this whole conversation, it is expected :)

-Steve

=?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:

Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I=20
 think it's fairly easy to implement: just flush once after using=20
 the file descriptor for the first time, and throw if it fails.
=20
 While it doesn't cover a case where file descriptor becomes=20
 non-writable during the program lifetime, it covers the most=20
 common case of file descriptor not being writable at all.

	Actually, the best solution is to check if the file is writable
before writing and *without flushing*. On Posix systems, this can be
done with:

fcntl (fd, F_GETFL) & W_OK

		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 10:36 AM, "Jérôme M. Berger" wrote:
 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL) & W_OK

Sure, but that doesn't cover the case of a write failure because the disk is is 
near full and then is full.

The write has to be tested to see if it succeeded, not assumed to succeed 
because only one of the possible failure conditions is not present.

"Adam D. Ruppe" <destructionator gmail.com> writes:

Here's a radical thought: should stdout even be global?

With any other file, this wouldn't be an issue because File's 
dtor calls close() which (I'm pretty sure) flushes the file, and 
thus would detect the error at the latest, when it goes out of 
scope.

I think if stdout wasn't global, I'd be among those complaining 
about the code it breaks and the hassle it brings, but it seems 
to me that this is the real problem here. It doesn't go out of 
scope until the program is terminating.

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 13:58:45 -0400, Walter Bright  =

<newshound2 digitalmars.com> wrote:

 On 6/12/2013 10:36 AM, "J=C3=A9r=C3=B4me M. Berger" wrote:
 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL) & W_OK

 Sure, but that doesn't cover the case of a write failure because the  =

 disk is is near full and then is full.

 The write has to be tested to see if it succeeded, not assumed to  =

 succeed because only one of the possible failure conditions is not  =

 present.

It is already.  Why would you think it's not?  The specific case being  =

discussed is when the file descriptor is invalid from the start.

-Steve

Walter Bright <newshound2 digitalmars.com> writes:

On 6/12/2013 11:41 AM, Steven Schveighoffer wrote:
 It is already.  Why would you think it's not?  The specific case being
discussed
 is when the file descriptor is invalid from the start.

The problem is more general than the specific case, as I pointed out in another 
post here.

It is not reliably detected by only checking the first write.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/12/13 1:36 PM, "Jérôme M. Berger" wrote:
 Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I
 think it's fairly easy to implement: just flush once after using
 the file descriptor for the first time, and throw if it fails.

 While it doesn't cover a case where file descriptor becomes
 non-writable during the program lifetime, it covers the most
 common case of file descriptor not being writable at all.

 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL)&  W_OK

 		Jerome

When do you check and how frequently? I think the right thing to do is 
as the bug report indicates.

Andrei

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Wed, 12 Jun 2013 13:36:17 -0400, J=C3=A9r=C3=B4me M. Berger <jeberger=
 free.fr>
wrote:

 Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I
 think it's fairly easy to implement: just flush once after using
 the file descriptor for the first time, and throw if it fails.

 While it doesn't cover a case where file descriptor becomes
 non-writable during the program lifetime, it covers the most
 common case of file descriptor not being writable at all.

 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL) & W_OK

This is probably the better answer.

You only have to do that with FILE * initialized with fdopen.

-Steve

"bearophile" <bearophileHUGS lycos.com> writes:

Graham Fawcett:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

I was about to link that Reddit thread here myself :-)

The original article proposes to translate to your language this 
little piece of Python+libs and measure its speed, safety in 
presence of errors, etc:



import os, sys, json
envname = os.path.basename(sys.argv[0])
args = json.loads(os.environ["0install-runenv-" + envname])
os.execv(args[0], args + sys.argv[1:])


Later he proposes other means to measure a language quality. 
Overall the comparison is quite interesting, despite several 
methodological problems.

Bye,
bearophile

"ixid" <nuaccount gmail.com> writes:

On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more to 
 be done. :) If you have a few moments to dispel some D myths, 
 and contribute constructively to the discussion, please take a 
 look!

 Best,
 Graham

Be careful with advocating D too much, I think what with the very 
positive but visible D conference coverage we don't want to seem 
like zealous fanboys. It may be better to let discussion of D 
cool off for a while, then return when there is something new and 
cool to show rather than get into arguments over Rust vs D. Those 
will only degenerate and make D users look bad.

This is important- it reflects on the users and the language and 
can very easily turn people away while single misinformed 
comments will not be noticed by all that many people.

Don't rubbish Rust. Comparisons quickly and easily turn negative.

"bearophile" <bearophileHUGS lycos.com> writes:

ixid:

 Be careful with advocating D too much, I think what with the 
 very positive but visible D conference coverage we don't want 
 to seem like zealous fanboys.

Right.
A better idea is to port the proposed code to D, count how many 
points it gets, and show all of it in the Reddit thread.

Bye,
bearophile