www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - reddit discussion on replacing Python in 0install

reply "Graham Fawcett" <fawcett uwindsor.ca> writes:
Hi folks,

There's an interesting discussion going on at Reddit about 
choosing a replacement language for 0install:

http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

I've tried to do a bit of D advocacy there, but there's more to 
be done. :) If you have a few moments to dispel some D myths, and 
contribute constructively to the discussion, please take a look!

Best,
Graham
Jun 10 2013
next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more to 
 be done. :) If you have a few moments to dispel some D myths, 
 and contribute constructively to the discussion, please take a 
 look!

 Best,
 Graham

I don't know how to make this test on Windows (current OS). But he uses this to test that failure to print hello correctly indicates failure. ./hello 1< /dev/null; echo Exit status: $? And Rust is the only one to pass in his list (ATS, C#, Go, Haskell, OCaml, Python)
Jun 10 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 22:06:29 UTC, Anthony Goins wrote:
 If you want to know what happens on my linux box

   1 module hellotest;
   2
   3 import std.stdio;
   4
   5 void main()
   6 {
   7         writeln("hello world.");
   8 }

 anthony LinuxGen12:~/projects/temp$ ./hellotest
 hello world.
 anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo status : $?
 status : 0
 anthony LinuxGen12:~/projects/temp$

So D fails the test too. But now that I think about it, isn't /dev/null where you write data to a black hole... maybe he meant /dev/zero

Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on writing to that, and didn't.
Jun 10 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2013 5:35 PM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 00:27:28 UTC, Walter Bright wrote:
 On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 So D fails the test too. But now that I think about it, isn't /dev/null where
 you write data to a black hole... maybe he meant /dev/zero

Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on writing to that, and didn't.

The command actually redirects /dev/null to fd 1 as an *input* stream, thus rendering it non-writable.

Ah, I see. In that case, writeln should have thrown an exception. Should submit a bug report.
Jun 10 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/13 6:06 PM, Anthony Goins wrote:
 On Monday, 10 June 2013 at 20:51:16 UTC, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about choosing a
 replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/


 I've tried to do a bit of D advocacy there, but there's more to be
 done. :) If you have a few moments to dispel some D myths, and
 contribute constructively to the discussion, please take a look!

 Best,
 Graham

I don't know how to make this test on Windows (current OS). But he uses this to test that failure to print hello correctly indicates failure. ./hello 1< /dev/null; echo Exit status: $? And Rust is the only one to pass in his list (ATS, C#, Go, Haskell, OCaml, Python)

If you want to know what happens on my linux box 1 module hellotest; 2 3 import std.stdio; 4 5 void main() 6 { 7 writeln("hello world."); 8 } anthony LinuxGen12:~/projects/temp$ ./hellotest hello world. anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo status : $? status : 0 anthony LinuxGen12:~/projects/temp$

The test program is flawed; writeln() writes to stdout, and the redirection is to stdin. writeln throws upon error. Andrei
Jun 10 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 12:04 AM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

Wouldn't stdin be fd 0? David

Oh indeed my bad. But the test is still flawed. Consider: import std.stdio; int main() { return printf("test\n") < 0; } This should return 1 if printf fails. It succeeds for 1</dev/null. Andrei
Jun 10 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei
Jun 11 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase.

As in a module destructor? Isn't it better to let the error pass silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful. Andrei
Jun 11 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2013 11:38 AM, Andrei Alexandrescu wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase.

As in a module destructor? Isn't it better to let the error pass silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

The flush should be in the termination code (not in writeln), where it should throw.
Jun 11 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 3:25 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

How exactly is this useful? What if you returned a different error code and you prefer that to come through?

I think that can be very useful. Andrei
Jun 11 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 3:36 PM, Lars T. Kyllingstad wrote:
 On Tuesday, 11 June 2013 at 18:38:51 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

Then we should also provide a hook so the user has the option of handling it. For example, core.exception could provide void setStaticDestructorExceptionHandler( void function(string module, Throwable t) handler); or similar.

That's nice but but seems overdesign. I'd say let's fix the bug for now. Andrei
Jun 11 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2013 12:25 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

How exactly is this useful?

I used to entertain myself and friends by setting up unix so the disk is nearly full, and then trying various commands that produced output, and sending the output to the disk. Because so many C programs fail to check for failed output, all kinds of entertaining random things would happen.
Jun 11 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei

The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower). The fix is when main() returns to do the flush there.
Jun 11 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2013 8:55 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 23:42:33 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei

The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

Note, it's only done once, the very first time anything is written. The rest of the time, flushing follows normal procedure. In effect, the first write confirms the FD is valid, then all writes after assume it stays valid.

I don't agree. Buffering is often done on page size boundaries - throwing out a random number of characters and then flushing will get it all wonky.
Jun 11 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries - throwing out
 a random number of characters and then flushing will get it all wonky.

You clearly missed something in the discussion here. The proposal is to flush once at first use, so an Exception is thrown. Nothing change after that first flush at initialization, other flushes stay where they are.

Not at all. A flush forces a write to the disk - that's the point of it. Disks are not at all well tuned to writing a few bytes, they like to be written in aligned blocks of block sizes, and the I/O subsystem is designed for that. This is why stdout has a flag in it saying if it is a "block oriented" or "character oriented" device. It makes a big difference. This proposal attempts to treat a block device like a character device. It will work, but it will perform poorly. P.S. I've written device drivers for disks. P.P.S. The solution is simple, as I said earlier. Just do a flush after main() exits. It happens anyway - done by the C stdio subsystem - I just propose doing it in the D code before it hands things back to the C runtime. This will entail no performance degradation.
Jun 11 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 11:18 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 02:48:44 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries -
 throwing out
 a random number of characters and then flushing will get it all wonky.

You clearly missed something in the discussion here. The proposal is to flush once at first use, so an Exception is thrown. Nothing change after that first flush at initialization, other flushes stay where they are.

Not at all. A flush forces a write to the disk - that's the point of it. Disks are not at all well tuned to writing a few bytes, they like to be written in aligned blocks of block sizes, and the I/O subsystem is designed for that. This is why stdout has a flag in it saying if it is a "block oriented" or "character oriented" device. It makes a big difference. This proposal attempts to treat a block device like a character device. It will work, but it will perform poorly.

No, it does perform well. You are still not understanding the proposal.

I agree performance is not a problem. But it's just weird behavior. We should flush stdout termination, anything else would have to be carefully justified - and this is not. Andrei
Jun 12 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We should
 flush stdout termination, anything else would have to be carefully justified -
 and this is not.

stdout is flushed on termination. Your code just doesn't puke a stack trace if there is an issue after main exits.

There is main(), and then there is _Dmain(). Take a look at druntime/src/rt/dmain2.d
Jun 12 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 2:36 PM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:56:59 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior.
 We should
 flush stdout termination, anything else would have to be carefully
 justified -
 and this is not.

stdout is flushed on termination. Your code just doesn't puke a stack trace if there is an issue after main exits.

There is main(), and then there is _Dmain(). Take a look at druntime/src/rt/dmain2.d

I don't get the point. Upon process termination, the C runtime flushes stdout/stderr. There is no reason to reinvent this wheel.

Flushes but doesn't propagate the potential error to the return code. Let's take further discussion to http://d.puremagic.com/issues/show_bug.cgi?id=10344. I think what we need to do is cut and dried. Andrei
Jun 12 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

Yes, I understand your proposal quite well. Your benchmark is seriously flawed. Most modern systems cache writes in memory, and have a delayed write to the media. This hides the problem. Try writing to a floppy disk system with mem caching turned off.
Jun 12 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 12:02 PM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright
 <newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well. You are still not understanding the proposal.

Yes, I understand your proposal quite well. Your benchmark is seriously flawed. Most modern systems cache writes in memory, and have a delayed write to the media. This hides the problem.

Sorry, but this is just bunk. One uneven flush does not mess up all future aligned writes. I/O is not that fragile. If you can demonstrate your theory, I will concede. Until you do, don't expect any more responses, it's no use arguing when you don't understand the problem.

You're right that calling fflush() only once upon the first call will not cause a performance problem. My argument is different - it's not a good thing to do. Andrei
Jun 12 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright <newshound2 digitalmars.com>
 wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

Yes, I understand your proposal quite well. Your benchmark is seriously flawed. Most modern systems cache writes in memory, and have a delayed write to the media. This hides the problem.

Sorry, but this is just bunk. One uneven flush does not mess up all future aligned writes. I/O is not that fragile. If you can demonstrate your theory, I will concede. Until you do, don't expect any more responses, it's no use arguing when you don't understand the problem.

May I present source code? ----------------------------- int fflush(FILE *fp) { int length; int result= 0; /* if fflush(NULL) flush all buffers */ if (fp == NULL) { if (flushall() >= 0) result = 0; } else { /* don't flush buffer if we are not writing */ __fp_lock(fp); if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT && (fp->_base #ifdef BIGBUF || fp->_seg #endif )) { length = fp->_ptr - fp->_base; /* # of bytes in buffer */ #ifdef BIGBUF if (length && _writex(fp->_file,fp->_base,length,fp->_seg) != length) fp->_flag |= _IOERR; #else if (length) { int nwritten = write(fp->_file,fp->_base,length); /* The following check for isatty() is because: * #define WIN32_LEAN_AND_MEAN * #include <windows.h> * #include <stdio.h> * void main() * { * // Set console output to UTF-8 (one can use 'chcp 65001' instead) * SetConsoleOutputCP( 65001 ); * // Latin small letter e with acute * fputs( "Output utf-8 accented char \xc3\xa9\n... and the rest is cut of * } * fails because WriteFile() apparently treats UTF-8 * sequences as 1 byte when writing to the console. */ if (!nwritten || (nwritten != length && !isatty(fp->_file))) fp->_flag |= _IOERR; } #endif fp->_cnt = 0; fp->_ptr = fp->_base; } else fp->_cnt = 0; result = (ferror(fp)) ? EOF : 0; __fp_unlock(fp); } return result; } ----------------------------------- This does exactly what I said it does. It's from the Digital Mars C runtime library. Now you could argue that this code sux, and you might even be right. But there are a lot of implementations of C fflush out there - are you sure that all of them will realign after a misaligned write? And even if they do realign, you cannot write 10 bytes to a disk. You can only write a block or a sector. Which means the next write, even if aligned, has to write that block or sector again. You are proposing that this repeated write of the first sector be done for all file output. You probably won't notice a speed difference if write caching is done, but that doesn't make it a good idea.
Jun 12 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 12:40 PM, Walter Bright wrote:
 You are proposing that this repeated write of the first sector be done
 for all file output.

No, he's proposing that only the very first write flushes. Not every write. I'd argue it's a bizarre thing to do, but not that it's necessarily slow. Andrei
Jun 12 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 10:07 AM, Andrei Alexandrescu wrote:
 On 6/12/13 12:40 PM, Walter Bright wrote:
 You are proposing that this repeated write of the first sector be done
 for all file output.

No, he's proposing that only the very first write flushes. Not every write. I'd argue it's a bizarre thing to do, but not that it's necessarily slow.

I meant for every file being output. And, again, it is forcing two writes of the first block, at a minimum. The perf cost can be hidden by write caching, but to rely on that is bad design and undermines how the I/O system is set up and tuned. I predict a WTF bug report on this in the future, as someone with an unusual device attached is looking for the cause of why his D program writing to it behaves badly. Or he just writes it off as "D sux". There's a *reason* why stdio distinguishes between block devices and character devices.
Jun 12 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 11:36 AM, Steven Schveighoffer wrote:
 If I understand correctly, the buffer is flushed, and ptr is reset to the base,
 the count is reset to 0.  I don't see any code in there that does any kind of
 realignment.

You write 7 bytes, fflush. Now, every 4K buffered write from then on is going to write from 7..4096+7, 4096+7..8192+7, ...
 That is one block or sector.  The write cache of the OS or the drive will
 probably absorb this hit anyway.

Write caches are not always there.
 The performance hit is extremely negligible.

Only if there is a write cache (which can be turned off, and are often turned off for removable drives).
 Not only that, but the hardware can differ from file system to file system. 
You are going through the file
 system driver, through the disk driver, through the disk.  All of those pieces
 are written to optimize writes that SPECIFIC hardware.  There isn't much you
can
 do to make this perform poorly.

Embedded systems and custom devices often do not have sophisticated hardware or software drivers.
 You are proposing that this repeated write of the first sector be done for all
 file output. You probably won't notice a speed difference if write caching is
 done, but that doesn't make it a good idea.

No, this is incorrect. fflush is not called on subsequent writeln.

I didn't say it was. See my other reply. In any case, if fflush is not called on subsequent writelns, it is still possible to have the first writeln succeed, the subsequent writelns fail and have main() return success.
Jun 12 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 3:40 PM, Walter Bright wrote:
 I didn't say it was. See my other reply. In any case, if fflush is not
 called on subsequent writelns, it is still possible to have the first
 writeln succeed, the subsequent writelns fail and have main() return
 success.

I'm a bit surprised that there's any other response to this than: "Oh okay, then flushing only after the first write or checking whether the file is writable in the beginning doesn't work". Andrei
Jun 12 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:
 If this was such a big deal, then FILE * would guarantee the alignment went
 back.  It's not that hard.

I know it isn't hard. But D's charter is not to go and patch the C runtime library, nor is that remotely practical across all the thousands of C implementations out there.
 And I have experience with turning them off -- some of my clients requested
that
 to avoid losing data that was in the write cache.  It does not all of a sudden
 slow down the performance of the drive.

I ran some experiments a while back writing to a slow SD card. There was a big difference in speed when turning write caching on and off. I bet you'll see quite a whopper of a difference if you try that with a floppy drive! In other words, the difference shows up when you've got a fairly slow device. Write caching would not have been invented if it had no effect.
 But this whole discussion is academic at this point, Jérôme identified that
we
 can simply check to see if the file descriptor is valid, without flushing.

His fix is insufficient for other reasons as already pointed out.
 This isn't the problem that was presented.  The problem that was presented is
 that given an invalid file descriptor, writeln happily works (as long as you
 don't cause a flush) and does not throw.  This is unintuitive to someone who is
 expecting writeln to choke on an invalid descriptor.

That behavior is normal to anyone used to working with C stdio. What is wrong and needs fixing is the program exiting with "success" indication when the output has actually failed.
Jun 13 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-06-13 20:11, Walter Bright wrote:

 I know it isn't hard. But D's charter is not to go and patch the C
 runtime library, nor is that remotely practical across all the thousands
 of C implementations out there.

Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't rely on the C runtime library in this case, if possible.
 That behavior is normal to anyone used to working with C stdio.

But we're using D here, not C. You have to stop assume that everyone using D knows C or C++. Just because you do doesn't mean everyone else does. -- /Jacob Carlborg
Jun 14 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 12:10 AM, Jacob Carlborg wrote:
 On 2013-06-13 20:11, Walter Bright wrote:

 I know it isn't hard. But D's charter is not to go and patch the C
 runtime library, nor is that remotely practical across all the thousands
 of C implementations out there.

Why not? Why shouldn't we try to be better than C? Perhaps we shouldn't rely on the C runtime library in this case, if possible.
 That behavior is normal to anyone used to working with C stdio.

But we're using D here, not C. You have to stop assume that everyone using D knows C or C++. Just because you do doesn't mean everyone else does.

D is set up to work hand-in-glove with C. This isn't going to change, as such compatibility is a big feature of D. Part of that is seamlessly working with existing C I/O. D code that does writeln's interleaved with C code that does printf's should work as expected. Since this particular issue can be easily fixed without "fixing" the C runtime, there is no motivation to do so, either.
Jun 14 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Jun-2013 22:36, Steven Schveighoffer пишет:
 On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright

 No, this is incorrect.  fflush is not called on subsequent writeln.

 Not only that, but we only have to do this on FILE * that were
 initialized with unknown file descriptors (such as
 stdin/stdout/stderr).  If we use fopen, you don't have to do this.

Steve, the only unchecked place right now is in the _last_ flush. Even if you flush on first call it won't help some other forms of errors (other then it's not writable to begin with). The only thing it will do is surprise certain kinds of applications that use fdopen with "passed in" sockets and whatnot. Basically a year later we'll see puzzled folks with strace printouts. The only thing that's broken here is that C-runtime doesn't care for errors on file descriptors it flushes at termination. Now since it's D we may as flush global files at end of main (in static dtor) while still in the "D world" then throw. This however leaves flushes on uncollected Files that were allocated with new or as a part of GC arrays. Given that these cases are bogus to begin with I'd say go ahead with shared static ~this() { stdout.flush(); stderr.flush(); } -- Dmitry Olshansky
Jun 12 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 If you can demonstrate your theory, I will concede.  Until you do, don't expect
 any more responses, it's no use arguing when you don't understand the problem.

There's another reason it won't work. If there are 4 bytes of free space left on the drive: void main() { writeln("foo"); // succeeds writeln("bar"); // silently fails }
Jun 12 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's actually FASTER with a
 flush, although I don't know why.

Because of write caching, you can get very different results from one run to the next of the same program.
Jun 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 8:22 AM, Steven Schveighoffer wrote:
 And quite honestly, if disk performance were dependent on user-land C buffering
 schemes, C runtime writers would pay more attention to the C buffering scheme
 and make sure it performs well!

Actually, this has been a common oversight of C runtime writers. My compiler was able to consistently beat others in I/O back in the 80's because I used a better buffering scheme in the runtime. It's not the only anecdote I have about that, either.
Jun 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 11:43 AM, Steven Schveighoffer wrote:
 The 80's are a long time ago.

But old code can live on in surprising ways.
 Plus, your posting of the source code pretty much
 refutes that your buffering scheme takes into account how important this should
 be.  It ignores alignment of writes if you add an fflush in between writes.

One aspect of its buffering scheme being inferior doesn't mean the rest of it is. There are a rather large number of issues with doing good I/O.
 It's not the only anecdote I have about that, either.


You're overlooking that there are a LOT of C runtimes in use out there, and testing on one of them in one system doesn't say anything about other systems, and many of them (such as for embedded systems) are fairly primitive. I've never seen two C runtimes use quite the same buffering logic.
Jun 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.  If a specific
 hardware/software combination requires specific buffering, the place to handle
 it is in the runtime, not code on top of it.  If the C runtime that D uses
isn't
 up to snuff, let's use a different scheme, or abandon it all together *for that
 specific device*.

There's simply no reason to do that. There has been a fix proposed that not only solves the problem correctly (your solution is incomplete), it does not require any improvements to the underlying C runtime.
Jun 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2013 10:45 PM, deadalnix wrote:
 I don't think any of the proposed fixes is a good idea. With he one you are
 pushing for, the program can fail behind the feet of the programmer, which is
 not better.

Please explain.
Jun 14 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the program
ran
 fun and is terminated successfully. Still the whole stuff will explode under
its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?
Jun 15 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the program
ran
 fun and is terminated successfully. Still the whole stuff will explode under
its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?

From programmer's perspective.

I'm sorry, that makes no sense to me. In fact, this thread started out reporting an article by a programmer who was unhappy that the program exited normally but the output didn't happen. Defaulting to ignoring errors and blithely proceeding is not usually considered a best practice.
Jun 15 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/15/2013 3:24 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 10:08:38 UTC, Walter Bright wrote:
 On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the program is
 problematic as well. At this point, from programmer perspective, the
 program ran
 fun and is terminated successfully. Still the whole stuff will explode
 under its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?

From programmer's perspective.

I'm sorry, that makes no sense to me.

That is because you aren't reading what is written. See 2/
 Defaulting to ignoring errors and blithely proceeding is not usually
 considered a best practice.

I have never written that. That explains 1/

I have no idea what you're talking about.
Jun 15 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/13 12:04 PM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of the
 program is
 problematic as well. At this point, from programmer perspective, the
 program ran
 fun and is terminated successfully. Still the whole stuff will
 explode under its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?

From programmer's perspective.

You may want to discuss alternative solutions in the bug report: http://d.puremagic.com/issues/show_bug.cgi?id=10344 Thanks, Andrei
Jun 15 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower). The fix is when main() returns to do the flush there.

I agree. One question is, should we flush all buffers upon termination? fflush(null) would do so. http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html Andrei
Jun 12 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 5:30 AM, Andrei Alexandrescu wrote:
 On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower). The fix is when main() returns to do the flush there.

I agree. One question is, should we flush all buffers upon termination? fflush(null) would do so. http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html

I haven't heard about the fflush(null) thing. That may be gnu-specific. I don't think that works on Windows. But yes, all open files should be flushed on termination. It's what C's stdio does already.
Jun 12 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 8:59 AM, Walter Bright wrote:
 I haven't heard about the fflush(null) thing. That may be gnu-specific. I don't
 think that works on Windows.

Followup: I have indeed heard about the fflush(null) thing, as it's in the fflush code I wrote several eons ago!
Jun 12 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 11:59 AM, Walter Bright wrote:
 On 6/12/2013 5:30 AM, Andrei Alexandrescu wrote:
 On 6/11/13 11:42 PM, Walter Bright wrote:
 I think you meant "just flush once after using the FILE * for the
 first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower). The fix is when main() returns to do the flush there.

I agree. One question is, should we flush all buffers upon termination? fflush(null) would do so. http://www.gnu.org/software/libc/manual/html_node/Flushing-Buffers.html

I haven't heard about the fflush(null) thing. That may be gnu-specific. I don't think that works on Windows.

It's quite an old standard feature albeit obscure, I found it and forgot about it a couple of times already. Works everywhere, see http://www.cplusplus.com/reference/cstdio/ferror/ and http://msdn.microsoft.com/en-us/library/9yky46tz(v=vs.80).aspx
 But yes, all open files should be flushed on termination. It's what C's
 stdio does already.

http://d.puremagic.com/issues/show_bug.cgi?id=10344 Andrei
Jun 12 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 10:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>
 wrote:
 The best solution would be for writeln() to throw on use, and I think
 it's fairly easy to implement: just flush once after using the file
 descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time" -Steve

Yah, but now we're looking at extra state associated with each file etc. Also it's kind of unexpected. Andrei
Jun 12 2013
prev sibling parent reply =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I=20
 think it's fairly easy to implement: just flush once after using=20
 the file descriptor for the first time, and throw if it fails.
=20
 While it doesn't cover a case where file descriptor becomes=20
 non-writable during the program lifetime, it covers the most=20
 common case of file descriptor not being writable at all.

Actually, the best solution is to check if the file is writable before writing and *without flushing*. On Posix systems, this can be done with: fcntl (fd, F_GETFL) & W_OK Jerome --=20 mailto:jeberger free.fr http://jeberger.free.fr Jabber: jeberger jabber.fr
Jun 12 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 10:36 AM, "Jérôme M. Berger" wrote:
 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL) & W_OK

Sure, but that doesn't cover the case of a write failure because the disk is is near full and then is full. The write has to be tested to see if it succeeded, not assumed to succeed because only one of the possible failure conditions is not present.
Jun 12 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2013 11:41 AM, Steven Schveighoffer wrote:
 It is already.  Why would you think it's not?  The specific case being
discussed
 is when the file descriptor is invalid from the start.

The problem is more general than the specific case, as I pointed out in another post here. It is not reliably detected by only checking the first write.
Jun 12 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/13 1:36 PM, "Jérôme M. Berger" wrote:
 Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I
 think it's fairly easy to implement: just flush once after using
 the file descriptor for the first time, and throw if it fails.

 While it doesn't cover a case where file descriptor becomes
 non-writable during the program lifetime, it covers the most
 common case of file descriptor not being writable at all.

Actually, the best solution is to check if the file is writable before writing and *without flushing*. On Posix systems, this can be done with: fcntl (fd, F_GETFL)& W_OK Jerome

When do you check and how frequently? I think the right thing to do is as the bug report indicates. Andrei
Jun 12 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 12:16 AM, Infiltrator wrote:
 On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

Wouldn't stdin be fd 0? David

What David said. Here's the same thing done to a cat (poor kitty): $ cat 1</dev/null test cat: write error: Bad file descriptor

No, this is also wrong. cat reads and writes, "hello world" only writes. Consider: echo meh 1</dev/null Always succeeds. Andrei
Jun 10 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell? -- /Jacob Carlborg
Jun 11 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-06-11 12:55, Nick Sabalausky wrote:

 I think I remember him saying somewhere that he uses zsh, although I
 can't look it up or test it on zsh at the moment.

With zsh and the first one I get: cat: stdout: Bad file descriptor With the second one I get nothing and 0 as the exit code. -- /Jacob Carlborg
Jun 11 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell?

I use zsh. Indeed under bash I can reproduce the failure. But if I run the printf-based test, it exits successfully. Andrei
Jun 11 2013
parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Andrei Alexandrescu wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

another shell?

I use zsh. Indeed under bash I can reproduce the failure. But if I run =

 the printf-based test, it exits successfully.
=20

the /usr/bin/echo executable... Jerome --=20 mailto:jeberger free.fr http://jeberger.free.fr Jabber: jeberger jabber.fr
Jun 11 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Graham Fawcett:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

I was about to link that Reddit thread here myself :-) The original article proposes to translate to your language this little piece of Python+libs and measure its speed, safety in presence of errors, etc: #!/usr/bin/env python import os, sys, json envname = os.path.basename(sys.argv[0]) args = json.loads(os.environ["0install-runenv-" + envname]) os.execv(args[0], args + sys.argv[1:]) Later he proposes other means to measure a language quality. Overall the comparison is quite interesting, despite several methodological problems. Bye, bearophile
Jun 10 2013
prev sibling next sibling parent "Anthony Goins" <neontotem gmail.com> writes:
On Monday, 10 June 2013 at 20:51:16 UTC, Jesse Phillips wrote:
 On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more 
 to be done. :) If you have a few moments to dispel some D 
 myths, and contribute constructively to the discussion, please 
 take a look!

 Best,
 Graham

I don't know how to make this test on Windows (current OS). But he uses this to test that failure to print hello correctly indicates failure. ./hello 1< /dev/null; echo Exit status: $? And Rust is the only one to pass in his list (ATS, C#, Go, Haskell, OCaml, Python)

If you want to know what happens on my linux box 1 module hellotest; 2 3 import std.stdio; 4 5 void main() 6 { 7 writeln("hello world."); 8 } anthony LinuxGen12:~/projects/temp$ ./hellotest hello world. anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; echo status : $? status : 0 anthony LinuxGen12:~/projects/temp$
Jun 10 2013
prev sibling next sibling parent "ixid" <nuaccount gmail.com> writes:
On Monday, 10 June 2013 at 18:25:05 UTC, Graham Fawcett wrote:
 Hi folks,

 There's an interesting discussion going on at Reddit about 
 choosing a replacement language for 0install:

 http://www.reddit.com/r/programming/comments/1g1fhf/case_study_for_replacing_python_in_0install/

 I've tried to do a bit of D advocacy there, but there's more to 
 be done. :) If you have a few moments to dispel some D myths, 
 and contribute constructively to the discussion, please take a 
 look!

 Best,
 Graham

Be careful with advocating D too much, I think what with the very positive but visible D conference coverage we don't want to seem like zealous fanboys. It may be better to let discussion of D cool off for a while, then return when there is something new and cool to show rather than get into arguments over Rust vs D. Those will only degenerate and make D users look bad. This is important- it reflects on the users and the language and can very easily turn people away while single misinformed comments will not be noticed by all that many people. Don't rubbish Rust. Comparisons quickly and easily turn negative.
Jun 10 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 10 June 2013 at 22:06:29 UTC, Anthony Goins wrote:
 If you want to know what happens on my linux box

   1 module hellotest;
   2
   3 import std.stdio;
   4
   5 void main()
   6 {
   7         writeln("hello world.");
   8 }

 anthony LinuxGen12:~/projects/temp$ ./hellotest
 hello world.
 anthony LinuxGen12:~/projects/temp$ ./hellotest 1</dev/null; 
 echo status : $?
 status : 0
 anthony LinuxGen12:~/projects/temp$

So D fails the test too. But now that I think about it, isn't /dev/null where you write data to a black hole... maybe he meant /dev/zero
Jun 10 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
ixid:

 Be careful with advocating D too much, I think what with the 
 very positive but visible D conference coverage we don't want 
 to seem like zealous fanboys.

Right. A better idea is to port the proposed code to D, count how many points it gets, and show all of it in the Reddit thread. Bye, bearophile
Jun 10 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 11 June 2013 at 00:27:28 UTC, Walter Bright wrote:
 On 6/10/2013 4:26 PM, Jesse Phillips wrote:
 So D fails the test too. But now that I think about it, isn't 
 /dev/null where
 you write data to a black hole... maybe he meant /dev/zero

Yeah, I thought /dev/null was a bit bucket. D shouldn't fail on writing to that, and didn't.

The command actually redirects /dev/null to fd 1 as an *input* stream, thus rendering it non-writable. David
Jun 10 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 11 June 2013 at 00:36:50 UTC, Walter Bright wrote:
 Ah, I see. In that case, writeln should have thrown an 
 exception. Should submit a bug report.

http://d.puremagic.com/issues/show_bug.cgi?id=10328
Jun 10 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu 
wrote:
 The test program is flawed; writeln() writes to stdout, and the 
 redirection is to stdin.

Wouldn't stdin be fd 0? David
Jun 10 2013
prev sibling next sibling parent "Infiltrator" <Infiltrator d.irc> writes:
On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu 
 wrote:
 The test program is flawed; writeln() writes to stdout, and 
 the redirection is to stdin.

Wouldn't stdin be fd 0? David

What David said. Here's the same thing done to a cat (poor kitty): $ cat 1</dev/null test cat: write error: Bad file descriptor
Jun 10 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 11 Jun 2013 01:39:47 -0400
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 12:16 AM, Infiltrator wrote:
 On Tuesday, 11 June 2013 at 04:04:03 UTC, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu
 wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

Wouldn't stdin be fd 0? David

What David said. Here's the same thing done to a cat (poor kitty): $ cat 1</dev/null test cat: write error: Bad file descriptor

No, this is also wrong. cat reads and writes, "hello world" only writes. Consider: echo meh 1</dev/null Always succeeds.

I just tried both on Debian 6: nick debian6:~$ cat 1< /dev/null cfws cat: write error: Bad file descriptor nick debian6:~$ echo meh 1< /dev/null bash: echo: write error: Bad file descriptor Maybe OSX behaves differently?
Jun 10 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 11 Jun 2013 09:09:43 +0200
Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell?

I think I remember him saying somewhere that he uses zsh, although I can't look it up or test it on zsh at the moment.
Jun 11 2013
prev sibling next sibling parent "estew" <estewh gmail.com> writes:
On Tuesday, 11 June 2013 at 10:55:16 UTC, Nick Sabalausky wrote:
 On Tue, 11 Jun 2013 09:09:43 +0200
 Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell?

I think I remember him saying somewhere that he uses zsh, although I can't look it up or test it on zsh at the moment.

Just tried zsh, bash, sh.. $ echo meh 1< /dev/null zsh: no output bash: Bad file descriptor sh: Bad file descriptor $ echo meh 2< /dev/null zsh: meh bash: meh sh: meh
Jun 11 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 11 Jun 2013 13:17:13 +0200
"estew" <estewh gmail.com> wrote:

 On Tuesday, 11 June 2013 at 10:55:16 UTC, Nick Sabalausky wrote:
 On Tue, 11 Jun 2013 09:09:43 +0200
 Jacob Carlborg <doob me.com> wrote:

 On 2013-06-11 08:38, Nick Sabalausky wrote:
 
 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell?

I think I remember him saying somewhere that he uses zsh, although I can't look it up or test it on zsh at the moment.

Just tried zsh, bash, sh.. $ echo meh 1< /dev/null zsh: no output bash: Bad file descriptor sh: Bad file descriptor $ echo meh 2< /dev/null zsh: meh bash: meh sh: meh

Hmm, yea, sounds like it may not have been the best test in the first place after all if even the shells and stdout-vs-stderr don't all agree with each other. I'm no unix expert, but wouldn't 1</dev/null just mean "If the program tries to read input from stdout, feed it input from /dev/null instead"? Or is that really a known way to force stdout to be read-only?
Jun 11 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 11 June 2013 at 12:37:53 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?

I get the same on Mac OS X 10.6.3 using bash. Is Andrei perhaps using another shell?

I use zsh. Indeed under bash I can reproduce the failure. But if I run the printf-based test, it exits successfully. Andrei

This is a good point, I kind of wondered why bash was the one reporting echo failed.
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 01:37:52 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 12:04 AM, David Nadlinger wrote:
 On Tuesday, 11 June 2013 at 04:02:59 UTC, Andrei Alexandrescu wrote:
 The test program is flawed; writeln() writes to stdout, and the
 redirection is to stdin.

Wouldn't stdin be fd 0? David

Oh indeed my bad. But the test is still flawed. Consider: import std.stdio; int main() { return printf("test\n") < 0; } This should return 1 if printf fails. It succeeds for 1</dev/null.

This actually gave me an idea of why it doesn't fail. This code DOES fail: import std.stdio; int main() { writeln("hello"); std.stdio.stdout.flush(); return 0; } Here is the (interesting) issue: FILE *, and likewise std.stdio.File, which uses FILE * as it's implementation, is a buffered stream. An interesting problem to solve for buffered streams is when to flush the buffer. The most efficient thing to do is to wait until the entire buffer is full, then flush it (call the syscall to write the data to the handle/descriptor). However, in an INTERACTIVE console, you want to flush more frequently, in case the data is coming out sporadically. HOWEVER, the C runtime first checks to see if the file descriptor is a console. If it is, it sets the flag indicating that it should flush on newlines, otherwise, it does not. Instead of seeing 4K (or whatever buffer size is) of data at a time, the user sees 1 line at a time, much more parseable by a human. This is why cat'ing a file to another file is much more efficient than cat'ing it to the console. Important to note is that /dev/null is NOT a console :) So what is happening is, writeln("hello") is writing "hello\n" to the FILE *, which sits in the buffer, no flushing occurs, because /dev/null is not a console. However, upon calling flush, it fails, because the file descriptor is invalid. Upon exit, std.stdio.stdout relies on core.stdc.stdio.stdout to flush the data (in fact, I don't even think stdout's dtor is called). And C's stdout does not throw an exception or alter the exit code. This is why you have no visible errors. If D did NOT rely on C's FILE *, it may have a chance to throw an exception on program exit (but likely wouldn't because the buffer is likely GC based and therefore can't be accessed on the dtor :) Re: different shells behaving differently, depending on how the individual shells are implemented can affect how this plays out. -Steve
Jun 11 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase.

As in a module destructor? Isn't it better to let the error pass silently rather than throwing an exception that can't be caught?
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 13:01:41 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase.

As in a module destructor? Isn't it better to let the error pass silently rather than throwing an exception that can't be caught?

I agree. This should not be a failure at that point. If you want to induce and catch failure, put std.stdio.stdout.flush() at the end of your main function, like I did. -Steve
Jun 11 2013
prev sibling next sibling parent "QAston" <qaston gmail.com> writes:
On Tuesday, 11 June 2013 at 17:01:43 UTC, Lars T. Kyllingstad 
wrote:
 Ah, I suspected so. (At a point in D's history writeln() did 
 do a flush; people wanted to eliminate it for efficiency 
 reasons.)

 We could introduce a flush() with throw in std.stdiobase.

As in a module destructor? Isn't it better to let the error pass silently rather than throwing an exception that can't be caught?

Passing silently on errors would contradict with several talks Andrei and Walter have given about how D's error handling is much better than C/C++ one (there are some HelloWorld program comparisions from those languages available on YT), I think they want that claim to remain valid. To me it is important to have error reported whenever something fails and have it as fast as possible - image a db backup system which fails silently.
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 14:44:17 -0400, J=C3=A9r=C3=B4me M. Berger <jeberger=
 free.fr>  =

wrote:

 Andrei Alexandrescu wrote:
 On 6/11/13 3:09 AM, Jacob Carlborg wrote:
 On 2013-06-11 08:38, Nick Sabalausky wrote:

 I just tried both on Debian 6:

 nick debian6:~$ cat 1< /dev/null
 cfws
 cat: write error: Bad file descriptor
 nick debian6:~$ echo meh 1< /dev/null
 bash: echo: write error: Bad file descriptor

 Maybe OSX behaves differently?




 another shell?

I use zsh. Indeed under bash I can reproduce the failure. But if I ru=


 the printf-based test, it exits successfully.

the /usr/bin/echo executable...

This is so not true :) Stevens-MacBook-Pro:testd steves$ echo $SHELL /bin/bash Stevens-MacBook-Pro:testd steves$ type echo echo is a shell builtin echo has been a shell builtin since bourne shell (/bin/sh) But your point is a good one, built-in commands very much will behave = differently than separate programs. For one, echo has to clean up after= = itself since it doesn't spawn a new process. -Steve
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 14:38:51 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error pass
 silently rather than throwing an exception that can't be caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

How exactly is this useful? What if you returned a different error code and you prefer that to come through? Also note, if stderr isn't working, "printing" anything diagnostic will be gone (I wonder what happens in that case?) -Steve
Jun 11 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Tuesday, 11 June 2013 at 18:38:51 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 1:01 PM, Lars T. Kyllingstad wrote:
 As in a module destructor? Isn't it better to let the error 
 pass
 silently rather than throwing an exception that can't be 
 caught?

It will not be caught but will cause the entire program to print a diagnostic and exit with a nonzero error code, which is useful.

Then we should also provide a hook so the user has the option of handling it. For example, core.exception could provide void setStaticDestructorExceptionHandler( void function(string module, Throwable t) handler); or similar.
Jun 11 2013
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu 
wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei

The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails. While it doesn't cover a case where file descriptor becomes non-writable during the program lifetime, it covers the most common case of file descriptor not being writable at all.
Jun 11 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 11 Jun 2013 16:42:43 -0700
Walter Bright <newshound2 digitalmars.com> wrote:
 
 I used to entertain myself and friends by setting up unix so the disk
 is nearly full, and then trying various commands that produced
 output, and sending the output to the disk.
 
 Because so many C programs fail to check for failed output, all kinds
 of entertaining random things would happen.
 

http://xkcd.com/1084/ Unix's design is amazingly powerful, but it's also interesting how much potential that creates for things to get seriously screwed up. :)
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>  
wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei

The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time" -Steve
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 23:42:33 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 7:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>  
 wrote:

 On Tuesday, 11 June 2013 at 16:50:50 UTC, Andrei Alexandrescu wrote:
 On 6/11/13 11:57 AM, Steven Schveighoffer wrote:
 This code DOES fail:

 import std.stdio;

 int main()
 {
 writeln("hello");
 std.stdio.stdout.flush();
 return 0;
 }

Ah, I suspected so. (At a point in D's history writeln() did do a flush; people wanted to eliminate it for efficiency reasons.) We could introduce a flush() with throw in std.stdiobase. Andrei

The best solution would be for writeln() to throw on use, and I think it's fairly easy to implement: just flush once after using the file descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time"

I think it's a bad idea, it'll muck up the buffering (i.e. make it slower).

Note, it's only done once, the very first time anything is written. The rest of the time, flushing follows normal procedure. In effect, the first write confirms the FD is valid, then all writes after assume it stays valid. -Steve
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 00:23:36 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 8:55 PM, Steven Schveighoffer wrote:

 Note, it's only done once, the very first time anything is written.   
 The rest of
 the time, flushing follows normal procedure.

 In effect, the first write confirms the FD is valid, then all writes  
 after
 assume it stays valid.

I don't agree. Buffering is often done on page size boundaries - throwing out a random number of characters and then flushing will get it all wonky.

What? You only flush the 'random number' of characters once, the rest of the time you are flushing full buffers. Where is the issue? Or do you have a specific application in mind? -Steve
Jun 11 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries 
 - throwing out a random number of characters and then flushing 
 will get it all wonky.

You clearly missed something in the discussion here. The proposal is to flush once at first use, so an Exception is thrown. Nothing change after that first flush at initialization, other flushes stay where they are.
Jun 11 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jun 13, 2013 at 11:11:06AM -0700, Walter Bright wrote:
 On 6/13/2013 6:24 AM, Steven Schveighoffer wrote:

This isn't the problem that was presented.  The problem that was
presented is that given an invalid file descriptor, writeln happily
works (as long as you don't cause a flush) and does not throw.  This
is unintuitive to someone who is expecting writeln to choke on an
invalid descriptor.

That behavior is normal to anyone used to working with C stdio. What is wrong and needs fixing is the program exiting with "success" indication when the output has actually failed.

Wouldn't an easy fix be for DMD to insert this into main()? scope(exit) if (fflush(stdout) != 0) throw new Exception(...); // ... rinse, repeat for whatever other fd's we care about Of course, instead of hardcoding stdio into DMD, the scope(exit) can call some kind of library cleanup hook, maybe something like: scope(exit) _druntimeCleanup(); and the throw can happen inside _druntimeCleanup, which should cause the exit status to be non-zero. Not sure how to make this exception user-catchable, though. T -- Ph.D. = Permanent head Damage
Jun 13 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 02:48:44 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright wrote:
 I don't agree. Buffering is often done on page size boundaries -  
 throwing out
 a random number of characters and then flushing will get it all wonky.

You clearly missed something in the discussion here. The proposal is to flush once at first use, so an Exception is thrown. Nothing change after that first flush at initialization, other flushes stay where they are.

Not at all. A flush forces a write to the disk - that's the point of it. Disks are not at all well tuned to writing a few bytes, they like to be written in aligned blocks of block sizes, and the I/O subsystem is designed for that. This is why stdout has a flag in it saying if it is a "block oriented" or "character oriented" device. It makes a big difference. This proposal attempts to treat a block device like a character device. It will work, but it will perform poorly.

No, it does perform well. You are still not understanding the proposal. Here is a test: Stevens-MacBook-Pro:testd steves$ cat testwrite.d import std.stdio; void main(string[] args) { writeln("hello walter"); if(args.length > 1 && args[1] == "flush") stdout.flush(); foreach(i; 0..10_000_000) writeln("hello walter"); } Stevens-MacBook-Pro:testd steves$ ~/dmd-2.063/osx/bin/dmd testwrite.d Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m2.466s user 0m1.323s sys 0m0.230s Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m2.077s user 0m1.296s sys 0m0.202s Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m2.121s user 0m1.289s sys 0m0.203s Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m1.997s user 0m1.297s sys 0m0.202s Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m2.495s user 0m1.330s sys 0m0.210s Stevens-MacBook-Pro:testd steves$ time ./testwrite > walter.txt real 0m2.316s user 0m1.309s sys 0m0.207s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m2.024s user 0m1.291s sys 0m0.221s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m1.943s user 0m1.287s sys 0m0.219s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m1.792s user 0m1.274s sys 0m0.217s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m2.026s user 0m1.286s sys 0m0.219s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m1.971s user 0m1.285s sys 0m0.222s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m2.252s user 0m1.288s sys 0m0.219s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m2.243s user 0m1.293s sys 0m0.218s Stevens-MacBook-Pro:testd steves$ time ./testwrite flush > walter.txt real 0m2.290s user 0m1.303s sys 0m0.219s I see no significant difference between the version that flushes after the first write of 13 bytes, and the version that does its first flush on buffer full. In fact, the best performing test was when the flush occurs on the first write. It doesn't mean flushing is better, it just means it has no impact.
 P.S. I've written device drivers for disks.

That's nice :) I understand the concept of buffered disk output quite well too.
 P.P.S. The solution is simple, as I said earlier. Just do a flush after  
 main() exits. It happens anyway - done by the C stdio subsystem - I just  
 propose doing it in the D code before it hands things back to the C  
 runtime. This will entail no performance degradation.

This is not ideal. Sometimes you will catch the error during the program (maybe at different locations), sometimes you exhibit an uncatchable error. With the proposed solution, you catch it deterministically, at the very first write. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 08:26:43 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 6/11/13 10:07 PM, Steven Schveighoffer wrote:
 On Tue, 11 Jun 2013 17:36:24 -0400, Denis Koroskin <2korden gmail.com>
 wrote:
 The best solution would be for writeln() to throw on use, and I think
 it's fairly easy to implement: just flush once after using the file
 descriptor for the first time, and throw if it fails.

This is a good idea. I think you meant "just flush once after using the FILE * for the first time" -Steve

Yah, but now we're looking at extra state associated with each file etc.

There is room. Each File is allocated on the heap.
 Also it's kind of unexpected.

According to this whole conversation, it is expected :) -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We  
 should flush stdout termination, anything else would have to be  
 carefully justified - and this is not.

stdout is flushed on termination. Your code just doesn't puke a stack trace if there is an issue after main exits. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the proposal.

Yes, I understand your proposal quite well. Your benchmark is seriously flawed. Most modern systems cache writes in memory, and have a delayed write to the media. This hides the problem.

Sorry, but this is just bunk. One uneven flush does not mess up all future aligned writes. I/O is not that fragile. If you can demonstrate your theory, I will concede. Until you do, don't expect any more responses, it's no use arguing when you don't understand the problem. -Steve
Jun 12 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 12 June 2013 at 16:34:08 UTC, Andrei Alexandrescu 
wrote:
 It's quite an old standard feature albeit obscure, I found it 
 and forgot about it a couple of times already. Works 
 everywhere, see 
 http://www.cplusplus.com/reference/cstdio/ferror/ and 
 http://msdn.microsoft.com/en-us/library/9yky46tz(v=vs.80).aspx

 But yes, all open files should be flushed on termination. It's 
 what C's
 stdio does already.

http://d.puremagic.com/issues/show_bug.cgi?id=10344

How come that in a first place, we have a file descriptor that isn't working ?
Jun 12 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
Here's a radical thought: should stdout even be global?

With any other file, this wouldn't be an issue because File's 
dtor calls close() which (I'm pretty sure) flushes the file, and 
thus would detect the error at the latest, when it goes out of 
scope.

I think if stdout wasn't global, I'd be among those complaining 
about the code it breaks and the hassle it brings, but it seems 
to me that this is the real problem here. It doesn't go out of 
scope until the program is terminating.
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 9:02 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:55:39 -0400, Walter Bright  
 <newshound2 digitalmars.com>
 wrote:

 On 6/12/2013 8:18 AM, Steven Schveighoffer wrote:
 No, it does perform well.  You are still not understanding the  
 proposal.

Yes, I understand your proposal quite well. Your benchmark is seriously flawed. Most modern systems cache writes in memory, and have a delayed write to the media. This hides the problem.

Sorry, but this is just bunk. One uneven flush does not mess up all future aligned writes. I/O is not that fragile. If you can demonstrate your theory, I will concede. Until you do, don't expect any more responses, it's no use arguing when you don't understand the problem.

May I present source code? ----------------------------- int fflush(FILE *fp) { int length; int result= 0; /* if fflush(NULL) flush all buffers */ if (fp == NULL) { if (flushall() >= 0) result = 0; } else { /* don't flush buffer if we are not writing */ __fp_lock(fp); if ((fp->_flag & (_IOWRT | _IONBF | _IOERR)) == _IOWRT && (fp->_base #ifdef BIGBUF || fp->_seg #endif )) { length = fp->_ptr - fp->_base; /* # of bytes in buffer */ #ifdef BIGBUF if (length && _writex(fp->_file,fp->_base,length,fp->_seg) != length) fp->_flag |= _IOERR; #else if (length) { int nwritten = write(fp->_file,fp->_base,length); /* The following check for isatty() is because: * #define WIN32_LEAN_AND_MEAN * #include <windows.h> * #include <stdio.h> * void main() * { * // Set console output to UTF-8 (one can use 'chcp 65001' instead) * SetConsoleOutputCP( 65001 ); * // Latin small letter e with acute * fputs( "Output utf-8 accented char \xc3\xa9\n... and the rest is cut of * } * fails because WriteFile() apparently treats UTF-8 * sequences as 1 byte when writing to the console. */ if (!nwritten || (nwritten != length && !isatty(fp->_file))) fp->_flag |= _IOERR; } #endif fp->_cnt = 0; fp->_ptr = fp->_base; } else fp->_cnt = 0; result = (ferror(fp)) ? EOF : 0; __fp_unlock(fp); } return result; } ----------------------------------- This does exactly what I said it does. It's from the Digital Mars C runtime library. Now you could argue that this code sux, and you might even be right. But there are a lot of implementations of C fflush out there - are you sure that all of them will realign after a misaligned write?

If I understand correctly, the buffer is flushed, and ptr is reset to the base, the count is reset to 0. I don't see any code in there that does any kind of realignment. Are you suggesting that other fflush code doesn't do this?
 And even if they do realign, you cannot write 10 bytes to a disk. You  
 can only write a block or a sector. Which means the next write, even if  
 aligned, has to write that block or sector again.

That is one block or sector. The write cache of the OS or the drive will probably absorb this hit anyway. The performance hit is extremely negligible. Not only that, but the hardware can differ from file system to file system. You are going through the file system driver, through the disk driver, through the disk. All of those pieces are written to optimize writes that SPECIFIC hardware. There isn't much you can do to make this perform poorly. The only performance hit you can really affect is the system call penalty. And that is done by allowing the normal flushing routine to continue for subsequent writes.
 You are proposing that this repeated write of the first sector be done  
 for all file output. You probably won't notice a speed difference if  
 write caching is done, but that doesn't make it a good idea.

No, this is incorrect. fflush is not called on subsequent writeln. Not only that, but we only have to do this on FILE * that were initialized with unknown file descriptors (such as stdin/stdout/stderr). If we use fopen, you don't have to do this. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 11:56:59 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/12/2013 8:34 AM, Steven Schveighoffer wrote:
 On Wed, 12 Jun 2013 11:32:58 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 I agree performance is not a problem. But it's just weird behavior. We  
 should
 flush stdout termination, anything else would have to be carefully  
 justified -
 and this is not.

stdout is flushed on termination. Your code just doesn't puke a stack trace if there is an issue after main exits.

There is main(), and then there is _Dmain(). Take a look at druntime/src/rt/dmain2.d

I don't get the point. Upon process termination, the C runtime flushes stdout/stderr. There is no reason to reinvent this wheel. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 13:58:45 -0400, Walter Bright  =

<newshound2 digitalmars.com> wrote:

 On 6/12/2013 10:36 AM, "J=C3=A9r=C3=B4me M. Berger" wrote:
 	Actually, the best solution is to check if the file is writable
 before writing and *without flushing*. On Posix systems, this can be
 done with:

 fcntl (fd, F_GETFL) & W_OK

Sure, but that doesn't cover the case of a write failure because the =

 disk is is near full and then is full.

 The write has to be tested to see if it succeeded, not assumed to  =

 succeed because only one of the possible failure conditions is not  =

 present.

It is already. Why would you think it's not? The specific case being = discussed is when the file descriptor is invalid from the start. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 13:36:17 -0400, J=C3=A9r=C3=B4me M. Berger <jeberger=
 free.fr>
wrote:

 Denis Koroskin wrote:
 The best solution would be for writeln() to throw on use, and I
 think it's fairly easy to implement: just flush once after using
 the file descriptor for the first time, and throw if it fails.

 While it doesn't cover a case where file descriptor becomes
 non-writable during the program lifetime, it covers the most
 common case of file descriptor not being writable at all.

Actually, the best solution is to check if the file is writable before writing and *without flushing*. On Posix systems, this can be done with: fcntl (fd, F_GETFL) & W_OK

This is probably the better answer. You only have to do that with FILE * initialized with fdopen. -Steve
Jun 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 15:40:10 -0400, Walter Bright  =

<newshound2 digitalmars.com> wrote:

 On 6/12/2013 11:36 AM, Steven Schveighoffer wrote:
 If I understand correctly, the buffer is flushed, and ptr is reset to=


 the base,
 the count is reset to 0.  I don't see any code in there that does any=


 kind of
 realignment.

You write 7 bytes, fflush. Now, every 4K buffered write from then on i=

 going to write from 7..4096+7, 4096+7..8192+7, ...

If this was such a big deal, then FILE * would guarantee the alignment = went back. It's not that hard.
 That is one block or sector.  The write cache of the OS or the drive =


 will
 probably absorb this hit anyway.

Write caches are not always there.
 The performance hit is extremely negligible.

Only if there is a write cache (which can be turned off, and are often=

 turned off for removable drives).

The write cache is irrelevant. The filesystem driver will dictate where= = writes are done, and will likely cache. And I have experience with turning them off -- some of my clients = requested that to avoid losing data that was in the write cache. It doe= s = not all of a sudden slow down the performance of the drive.
 Not only that, but the hardware can differ from file system to file  =


 system.  You are going through the file
 system driver, through the disk driver, through the disk.  All of tho=


 pieces
 are written to optimize writes that SPECIFIC hardware.  There isn't  =


 much you can
 do to make this perform poorly.

Embedded systems and custom devices often do not have sophisticated =

 hardware or software drivers.

The day that D runs on an embedded system, with no drivers or OS, BUT = provides a C runtime that D builds on, is the day I'll agree with you ;)= But this whole discussion is academic at this point, J=C3=A9r=C3=B4me id= entified = that we can simply check to see if the file descriptor is valid, without= = flushing.
 I didn't say it was. See my other reply. In any case, if fflush is not=

 called on subsequent writelns, it is still possible to have the first =

 writeln succeed, the subsequent writelns fail and have main() return  =

 success.

This isn't the problem that was presented. The problem that was present= ed = is that given an invalid file descriptor, writeln happily works (as long= = as you don't cause a flush) and does not throw. This is unintuitive to = = someone who is expecting writeln to choke on an invalid descriptor. Subsequent writelns that fail throw an exception. Any writeln that caus= es = a flush will create an exception. The rare corner case the "throw at end" will catch in addition is when t= he = file descriptor becomes invalid on the final flush only. I think we can probably implement both checks. The 'throw at end' metho= d = will catch both, but it would be nice to catch an invalid descriptor at = = the call the causes the problem, not at the end when the program is most= ly = gone. It's like catching a memory error far away from the cause. -Steve
Jun 13 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 12 Jun 2013 16:41:03 -0400, Dmitry Olshansky  =

<dmitry.olsh gmail.com> wrote:

 12-Jun-2013 22:36, Steven Schveighoffer =D0=BF=D0=B8=D1=88=D0=B5=D1=82=

 On Wed, 12 Jun 2013 12:40:53 -0400, Walter Bright

 No, this is incorrect.  fflush is not called on subsequent writeln.

 Not only that, but we only have to do this on FILE * that were
 initialized with unknown file descriptors (such as
 stdin/stdout/stderr).  If we use fopen, you don't have to do this.

Steve, the only unchecked place right now is in the _last_ flush. Even=

 if you flush on first call it won't help some other forms of errors  =

 (other then it's not writable to begin with). The only thing it will d=

 is surprise certain kinds of applications that use fdopen with "passed=

 in" sockets and whatnot. Basically a year later we'll see puzzled folk=

 with strace printouts.

The two problems described are separate problems. Both can be addressed= = differently. I think it's worth putting the flush at the end, and generating an = exception. But if we can catch the error on intialization, or at least = on = first use, it would be beneficial. -Steve
Jun 13 2013
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Wednesday, 12 June 2013 at 06:48:48 UTC, Walter Bright wrote:
 On 6/11/2013 10:15 PM, deadalnix wrote:
 On Wednesday, 12 June 2013 at 04:23:39 UTC, Walter Bright 
 wrote:
 I don't agree. Buffering is often done on page size 
 boundaries - throwing out
 a random number of characters and then flushing will get it 
 all wonky.

You clearly missed something in the discussion here. The proposal is to flush once at first use, so an Exception is thrown. Nothing change after that first flush at initialization, other flushes stay where they are.

Not at all. A flush forces a write to the disk - that's the point of it. Disks are not at all well tuned to writing a few bytes, they like to be written in aligned blocks of block sizes, and the I/O subsystem is designed for that. This is why stdout has a flag in it saying if it is a "block oriented" or "character oriented" device. It makes a big difference. This proposal attempts to treat a block device like a character device. It will work, but it will perform poorly. P.S. I've written device drivers for disks. P.P.S. The solution is simple, as I said earlier. Just do a flush after main() exits. It happens anyway - done by the C stdio subsystem - I just propose doing it in the D code before it hands things back to the C runtime. This will entail no performance degradation.

How about we test it before making any claims? I wrote a simple test program that writes 4gigs of '!' to stdout, and measured it in 2 ways: time ./a.out > /dev/null time ./a.out > file.txt Here are the test runs (a warmup followed by 3 runs of each): Output to /dev/null W/O FLUSH real 0m0.188s 0m0.186s 0m0.190s user 0m0.166s 0m0.165s 0m0.168s sys 0m0.021s 0m0.020s 0m0.021s /dev/null WITH FLUSH real 0m0.168s 0m0.166s 0m0.163s user 0m0.145s 0m0.144s 0m0.142s sys 0m0.022s 0m0.021s 0m0.021s Output to file.txt W/O FLUSH real 0m10.131s 0m11.766s 0m9.900s user 0m0.444s 0m0.443s 0m0.451s sys 0m5.352s 0m5.324s 0m5.308s real 0m9.474s 0m9.971s 0m10.115s user 0m0.233s 0m0.241s 0m0.233s sys 0m5.588s 0m5.616s 0m5.591s Does it look like it's slower? To me it looks like it's actually FASTER with a flush, although I don't know why. Try it yourself. The code that I tested with is below (apologies for using C). #include <stdio.h> #include <string.h> int main() { char buffer[4096]; memset(buffer, '!', 4096); fwrite(buffer, 1, 100, stdout); //fflush(stdout); for (int i = 0; i < 1024*1024; ++i) { fwrite(buffer, 1, 4096, stdout); } return 0; }
Jun 14 2013
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Thursday, 13 June 2013 at 18:11:12 UTC, Walter Bright wrote:
 What is wrong and needs fixing is the program exiting with 
 "success" indication when the output has actually failed.

What's wrong is a program existing with a status code *different* from one that was supplied by a programmer.
Jun 14 2013
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's 
 actually FASTER with a
 flush, although I don't know why.

Because of write caching, you can get very different results from one run to the next of the same program.

I re-ran them a few times obviously. The results were consistent.
Jun 14 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 14 June 2013 at 08:24:54 UTC, Denis Koroskin wrote:
 On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's 
 actually FASTER with a
 flush, although I don't know why.

Because of write caching, you can get very different results from one run to the next of the same program.

I re-ran them a few times obviously. The results were consistent.

That is super weird and unexpected to me, but I trust you.
Jun 14 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jun 2013 04:24:53 -0400, Denis Koroskin <2korden gmail.com>  
wrote:

 On Friday, 14 June 2013 at 08:17:02 UTC, Walter Bright wrote:
 On 6/14/2013 1:11 AM, Denis Koroskin wrote:
 Does it look like it's slower? To me it looks like it's actually  
 FASTER with a
 flush, although I don't know why.

Because of write caching, you can get very different results from one run to the next of the same program.

I re-ran them a few times obviously. The results were consistent.

Walter is right to a point -- caching can effect this behavior. There are several levels of caching at play here, and all of them are working hard to optimize the writing to disk. That likely is more of a cause than the flushing of the first element. But in that sense, his argument is also broken -- you have to be in a very special situation (no caching, no drivers, etc.) in order for this to matter. I had a similar test, and my results were not consistently one better than the other, but basically flushing had no discernible effect. And quite honestly, if disk performance were dependent on user-land C buffering schemes, C runtime writers would pay more attention to the C buffering scheme and make sure it performs well! i.e. if a single misaligned write can mess up gigabytes of output, then the next write would correct it. -Steve
Jun 14 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jun 2013 13:58:33 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 8:22 AM, Steven Schveighoffer wrote:
 And quite honestly, if disk performance were dependent on user-land C  
 buffering
 schemes, C runtime writers would pay more attention to the C buffering  
 scheme
 and make sure it performs well!

Actually, this has been a common oversight of C runtime writers. My compiler was able to consistently beat others in I/O back in the 80's because I used a better buffering scheme in the runtime.

The 80's are a long time ago. Plus, your posting of the source code pretty much refutes that your buffering scheme takes into account how important this should be. It ignores alignment of writes if you add an fflush in between writes.
 It's not the only anecdote I have about that, either.

That's good, because the floppy DOS days are pretty much over :) -Steve
Jun 14 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jun 2013 15:17:00 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 11:43 AM, Steven Schveighoffer wrote:
 The 80's are a long time ago.

But old code can live on in surprising ways.

The code living on is not what I'm talking about. The *assumptions* living on is the problem :) Old code can be written very carefully to avoid situations that are not present anymore.
 Plus, your posting of the source code pretty much
 refutes that your buffering scheme takes into account how important  
 this should
 be.  It ignores alignment of writes if you add an fflush in between  
 writes.

One aspect of its buffering scheme being inferior doesn't mean the rest of it is. There are a rather large number of issues with doing good I/O.

I'm not saying it's inferior, just that it's not as big a deal as you say it is. At least not any more. I can see that it might have been very important with an OS like DOS.
 It's not the only anecdote I have about that, either.


You're overlooking that there are a LOT of C runtimes in use out there, and testing on one of them in one system doesn't say anything about other systems, and many of them (such as for embedded systems) are fairly primitive.

The same can be said for your runtime. That is, your choices of buffering may not do well on other systems for which other buffering schemes may be tuned. I think in the end, we are optimizing here in the wrong place. If a specific hardware/software combination requires specific buffering, the place to handle it is in the runtime, not code on top of it. If the C runtime that D uses isn't up to snuff, let's use a different scheme, or abandon it all together *for that specific device*. Not that this is the situation we currently have, where D only runs on full-blown PCs... -Steve
Jun 14 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Jun 2013 16:31:35 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.  If a  
 specific
 hardware/software combination requires specific buffering, the place to  
 handle
 it is in the runtime, not code on top of it.  If the C runtime that D  
 uses isn't
 up to snuff, let's use a different scheme, or abandon it all together  
 *for that
 specific device*.

There's simply no reason to do that. There has been a fix proposed that not only solves the problem correctly (your solution is incomplete), it does not require any improvements to the underlying C runtime.

This doesn't fix the "problem" of when someone calls fflush(stdout); In their code somewhere. Your contention is this messes up all the writes beyond that. If that is such a problem, we should avoid using such a problematic library. We are after all, in control of the D runtime. But my argument is purposely ad absurdum, because it clearly is not a problem for any systems that exist today (as evidence shows). You are arguing unproven hypotheses about systems that are long dead with no hope of running D code. I respectfully bow out to work on more important things. -Steve
Jun 14 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 14 June 2013 at 20:31:39 UTC, Walter Bright wrote:
 On 6/14/2013 1:02 PM, Steven Schveighoffer wrote:
 I think in the end, we are optimizing here in the wrong place.
  If a specific
 hardware/software combination requires specific buffering, the 
 place to handle
 it is in the runtime, not code on top of it.  If the C runtime 
 that D uses isn't
 up to snuff, let's use a different scheme, or abandon it all 
 together *for that
 specific device*.

There's simply no reason to do that. There has been a fix proposed that not only solves the problem correctly (your solution is incomplete), it does not require any improvements to the underlying C runtime.

I don't think any of the proposed fixes is a good idea. With he one you are pushing for, the program can fail behind the feet of the programmer, which is not better.
Jun 14 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 15 June 2013 at 06:49:00 UTC, Walter Bright wrote:
 On 6/14/2013 10:45 PM, deadalnix wrote:
 I don't think any of the proposed fixes is a good idea. With 
 he one you are
 pushing for, the program can fail behind the feet of the 
 programmer, which is
 not better.

Please explain.

The solution that consist in flushing at first use is a surprising behavior, and look more like a weird hack than a solution. The solution that consist into flushing in main at the end of the program is problematic as well. At this point, from programmer perspective, the program ran fun and is terminated successfully. Still the whole stuff will explode under its feet, in the runtime. That isn't something we should promote.
Jun 15 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end of 
 the program is
 problematic as well. At this point, from programmer 
 perspective, the program ran
 fun and is terminated successfully. Still the whole stuff will 
 explode under its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?

From programmer's perspective.
Jun 15 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 15 June 2013 at 10:08:38 UTC, Walter Bright wrote:
 On 6/15/2013 3:04 AM, deadalnix wrote:
 On Saturday, 15 June 2013 at 09:48:59 UTC, Walter Bright wrote:
 On 6/15/2013 1:36 AM, deadalnix wrote:
 The solution that consist into flushing in main at the end 
 of the program is
 problematic as well. At this point, from programmer 
 perspective, the program ran
 fun and is terminated successfully. Still the whole stuff 
 will explode under its
 feet, in the runtime. That isn't something we should promote.

If the output failed to happen, how could the program have successfully behaved as intended?

From programmer's perspective.

I'm sorry, that makes no sense to me.

That is because you aren't reading what is written. See 2/
 Defaulting to ignoring errors and blithely proceeding is not 
 usually considered a best practice.

I have never written that. That explains 1/
Jun 15 2013