www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Can we get rid of non-raw write?

reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
Since *at least* as far back as XP, Windows has handled "\n" newlines
perfectly fine. The command line displays them properly, .BAT scripts
handle them properly, every code editor in existence handles them
properly. The *only* thing I've found that doesn't is Windows Notepad,
but really, whoTF uses that anyway?

So, why are silently and forcefully converting "\n" to "\r\n" on
windows by default? All it does is cause bugs. For example:
https://github.com/repeatedly/mustache-d/issues/3

And that's definitely not the first time I've run into problems due
using the "write*" functions instead of rawWrite.

Consider this straightforward code:

--------------------------------------
import std.file;
import std.stdio;

void transform(string str)
{
	/+ ...perform some modification of 'str'... +/
	return str;
}

void main()
{
	auto str = cast(string) read(args[1]);
	str = transform(str);
	write(str);
}
--------------------------------------

That simple code is *wrong*:

It works correctly for all input on Unix: Output newlines match input
newlines. Always. The code never asks for newlines to be messed with,
and therefore they never are.

But on Windows the behavior is just plain weird: Unix-style newlines in
the input are silently and forcefully converted to Windows-style
newlines behind the user's back. And even worse yet, *Windows*-style
newlines on the input are converted to a bizarre "Mac9 plus
Windows-style" combination of "\r\r\n" (not only is that wrong period,
but this sequence is often interpreted as two newlines which makes it
even worse). Using rawWrite fixes the problem, and creates *no* problem.
I feel like the current "write*" behavior is a design Steve Jobs would
have come up with.

So how is this useful to anyone, and how is it worthy of being the
default output behavior?

Even with Windows as my primary system, there hasn't been one single
time "write*"'s output-altering "feature" has helped me by doing
something more correctly than rawWrite would have done. And that's just
when I'm dealing with pure text. I'm literally better off just using
rawWrite everywhere, which is exactly what I intend to do from now on
(and I'll be viewing uses of "write*" with suspicion as latent bugs) -
unlike "write*", rawWrite *never* does the wrong thing.

Can we please get rid of this "text-mode output"? Or at least eliminate
it as the default?
Mar 20 2013
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
20-Mar-2013 18:34, Nick Sabalausky пишет:
 Since *at least* as far back as XP, Windows has handled "\n" newlines
 perfectly fine. The command line displays them properly, .BAT scripts
 handle them properly, every code editor in existence handles them
 properly. The *only* thing I've found that doesn't is Windows Notepad,
 but really, whoTF uses that anyway?

 So, why are silently and forcefully converting "\n" to "\r\n" on
 windows by default? All it does is cause bugs. For example:
 https://github.com/repeatedly/mustache-d/issues/3

 And that's definitely not the first time I've run into problems due
 using the "write*" functions instead of rawWrite.

 Consider this straightforward code:

 --------------------------------------
 import std.file;
 import std.stdio;

 void transform(string str)
 {
 	/+ ...perform some modification of 'str'... +/
 	return str;
 }

 void main()
 {
 	auto str = cast(string) read(args[1]);
 	str = transform(str);
 	write(str);
 }
 --------------------------------------

 That simple code is *wrong*:

 It works correctly for all input on Unix: Output newlines match input
 newlines. Always. The code never asks for newlines to be messed with,
 and therefore they never are.

 But on Windows the behavior is just plain weird: Unix-style newlines in
 the input are silently and forcefully converted to Windows-style
 newlines behind the user's back. And even worse yet, *Windows*-style
 newlines on the input are converted to a bizarre "Mac9 plus
 Windows-style" combination of "\r\r\n" (not only is that wrong period,
 but this sequence is often interpreted as two newlines which makes it
 even worse). Using rawWrite fixes the problem, and creates *no* problem.
 I feel like the current "write*" behavior is a design Steve Jobs would
 have come up with.

 So how is this useful to anyone, and how is it worthy of being the
 default output behavior?

No. Let's obliterate it. The only problem is compatibility and the fact that Windows has had this "text mode" for I/O in e.g. MSVCRT for a long time. It never helped me in any sensible way except that "yay! I can view it in notepad!" but broken many things far too often.
 Even with Windows as my primary system, there hasn't been one single
 time "write*"'s output-altering "feature" has helped me by doing
 something more correctly than rawWrite would have done. And that's just
 when I'm dealing with pure text. I'm literally better off just using
 rawWrite everywhere, which is exactly what I intend to do from now on
 (and I'll be viewing uses of "write*" with suspicion as latent bugs) -
 unlike "write*", rawWrite *never* does the wrong thing.

 Can we please get rid of this "text-mode output"? Or at least eliminate
 it as the default?

-- Dmitry Olshansky
Mar 20 2013
prev sibling next sibling parent reply "Chris Cain" <clcain uncg.edu> writes:
On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
wrote:
 Can we please get rid of this "text-mode output"? Or at least 
 eliminate
 it as the default?

+1. I've had to work around this issue several times by doing strange things like stripping the line ending and putting a \n at the end. I figured it was something I was just missing but seeing someone else mention the problem makes me think this is erroneous behavior. Is this in bugzilla?
Mar 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-20 22:08, Nick Sabalausky wrote:

 I doubt there's a bugzilla entry for this since it is, unfortunately,
 the intended behavior. I didn't want to go posting a 'zilla issue for
 it before discussing here because I figured that might just end up
 "INVALID" or "WONTFIX".

If you make it an enhancement request it shouldn't at least end up as "INVALID". -- /Jacob Carlborg
Mar 21 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 20 Mar 2013 21:13:54 +0100
"Chris Cain" <clcain uncg.edu> wrote:

 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
 wrote:
 Can we please get rid of this "text-mode output"? Or at least 
 eliminate
 it as the default?

+1. I've had to work around this issue several times by doing strange things like stripping the line ending and putting a \n at the end. I figured it was something I was just missing but seeing someone else mention the problem makes me think this is erroneous behavior. Is this in bugzilla?

There *is* "stdout.rawWrite(...)" (ya gotta include the "stdout." part), but unfortunately it doesn't come in "*ln", "*f" or "*fln" varieties. And it should be the default. I doubt there's a bugzilla entry for this since it is, unfortunately, the intended behavior. I didn't want to go posting a 'zilla issue for it before discussing here because I figured that might just end up "INVALID" or "WONTFIX".
Mar 20 2013
prev sibling next sibling parent reply "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
wrote:
 Since *at least* as far back as XP, Windows has handled "\n" 
 newlines
 perfectly fine. The command line displays them properly, .BAT 
 scripts
 handle them properly, every code editor in existence handles 
 them
 properly. The *only* thing I've found that doesn't is Windows 
 Notepad,
 but really, whoTF uses that anyway?
 ....

 Can we please get rid of this "text-mode output"? Or at least 
 eliminate
 it as the default?

+1. Leave an option in their for "ancient Windows support" if necessary, but take it out as the default. Graham
Mar 20 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-03-20 23:00, Graham Fawcett wrote:

 Is anyone still using MacOS earlier than version 10 (OSX)? Mac OS 9 was
 discontinued in 2002.

 On OSX, there's certainly no problem with Unix line endings. But I guess
 if we include "ancient Windows support" as an option, then "ancient Mac
 support" should in be there too.

 But writeln/writefln should emit '\n' as a line terminator, by default,
 on all platforms. Ancient terminators should be always opt-in,
 regardless of platform.

I agree. -- /Jacob Carlborg
Mar 21 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Mar 20, 2013 at 10:20:08PM +0100, Graham Fawcett wrote:
 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky wrote:
Since *at least* as far back as XP, Windows has handled "\n" newlines
perfectly fine. The command line displays them properly, .BAT scripts
handle them properly, every code editor in existence handles them
properly. The *only* thing I've found that doesn't is Windows
Notepad, but really, whoTF uses that anyway?  ....

Can we please get rid of this "text-mode output"? Or at least
eliminate it as the default?

+1. Leave an option in their for "ancient Windows support" if necessary, but take it out as the default.

What about MacOS? T -- He who does not appreciate the beauty of language is not worthy to bemoan its flaws.
Mar 20 2013
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Wednesday, 20 March 2013 at 21:33:48 UTC, H. S. Teoh wrote:
 On Wed, Mar 20, 2013 at 10:20:08PM +0100, Graham Fawcett wrote:
 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
 wrote:
Since *at least* as far back as XP, Windows has handled "\n" 
newlines
perfectly fine. The command line displays them properly, .BAT 
scripts
handle them properly, every code editor in existence handles 
them
properly. The *only* thing I've found that doesn't is Windows
Notepad, but really, whoTF uses that anyway?  ....

Can we please get rid of this "text-mode output"? Or at least
eliminate it as the default?

+1. Leave an option in their for "ancient Windows support" if necessary, but take it out as the default.

What about MacOS?

Is anyone still using MacOS earlier than version 10 (OSX)? Mac OS 9 was discontinued in 2002. On OSX, there's certainly no problem with Unix line endings. But I guess if we include "ancient Windows support" as an option, then "ancient Mac support" should in be there too. But writeln/writefln should emit '\n' as a line terminator, by default, on all platforms. Ancient terminators should be always opt-in, regardless of platform. Graham
 T

Mar 20 2013
prev sibling next sibling parent "Chris Cain" <clcain uncg.edu> writes:
On Wednesday, 20 March 2013 at 21:09:03 UTC, Nick Sabalausky 
wrote:
 There *is* "stdout.rawWrite(...)"

Sure, I saw that in your post. Thanks for the heads up. That'll be sometimes helpful for when I want to use it later. But still, rawWrite isn't exactly a replacement for write (just like, as you noted it's not a replacement for writef and ln varieties). In particular, it doesn't handle things like ranges as seemlessly as you'd want. Also, things such as write("my range = ", myRange) or writeln("This thing is ", myDesc, "!") are more difficult to do appropriately (though, std.string.format is pretty helpful here) Furthermore, lockingTextWriter displays the same sort of behavior and rawWrite doesn't help at all there because the use-case is completely different for an OutputRange. That's why the default behavior should be the way that works on modern systems, as you suggest.
Mar 20 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Wed, 20 Mar 2013 23:00:18 +0100
"Graham Fawcett" <fawcett uwindsor.ca> wrote:

 On Wednesday, 20 March 2013 at 21:33:48 UTC, H. S. Teoh wrote:
 On Wed, Mar 20, 2013 at 10:20:08PM +0100, Graham Fawcett wrote:
 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
 wrote:
Since *at least* as far back as XP, Windows has handled "\n" 
newlines
perfectly fine. The command line displays them properly, .BAT 
scripts
handle them properly, every code editor in existence handles 
them
properly. The *only* thing I've found that doesn't is Windows
Notepad, but really, whoTF uses that anyway?  ....

Can we please get rid of this "text-mode output"? Or at least
eliminate it as the default?

+1. Leave an option in their for "ancient Windows support" if necessary, but take it out as the default.

What about MacOS?

Is anyone still using MacOS earlier than version 10 (OSX)? Mac OS 9 was discontinued in 2002.

It was more than discontinued, it was more or less obliterated. Even OSX 10.3 and below are basically unusable anymore (unless you don't expect to be able to install anything). Probably 10.4, too. \r as a line ending is long dead. Might be worth supporting *reading* it in certain cases (old text files can live on for a long time), but not writing.
 On OSX, there's certainly no problem with Unix line endings. But 
 I guess if we include "ancient Windows support" as an option, 
 then "ancient Mac support" should in be there too.
 
 But writeln/writefln should emit '\n' as a line terminator, by 
 default, on all platforms. Ancient terminators should be always 
 opt-in, regardless of platform.
 

Yea, and by "Ancient" it's not as if we're even calling XP ancient. We're talking circa-Win9x line here. DMD and Phobos don't even try to support those anyway, and yet that's exactly what "write*" are essentially catering to. I know Walter has said in the past that "there are places" where Windows still expects \r\n, but if even if that's true, such places are rare and are best handled as special cases as-needed.
Mar 20 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 21 Mar 2013 00:44:56 -0400
Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> wrote:
 
 I know Walter has said in the past that "there are places" where
 Windows still expects \r\n...

Actually, I might be confusing that with "forward-slashes vs backslashes" in filepaths...
Mar 20 2013
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
wrote:
 Since *at least* as far back as XP, Windows has handled "\n" 
 newlines
 perfectly fine. The command line displays them properly, .BAT 
 scripts
 handle them properly, every code editor in existence handles 
 them
 properly. The *only* thing I've found that doesn't is Windows 
 Notepad,
 but really, whoTF uses that anyway?

Notepad is used as a log viewer. When you're presented a machine you don't run, you usually don't have time to setup your preferred work environment, notepad comes in handy to just view the log and diagnose an error.
Mar 20 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Mar-2013 09:01, Kagamin пишет:
 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky wrote:
 Since *at least* as far back as XP, Windows has handled "\n" newlines
 perfectly fine. The command line displays them properly, .BAT scripts
 handle them properly, every code editor in existence handles them
 properly. The *only* thing I've found that doesn't is Windows Notepad,
 but really, whoTF uses that anyway?

Notepad is used as a log viewer.

One word - Wordpad. Any relatively interesting log file is measured in MBs thus hanging the notepad on as said random machine.
 When you're presented a machine you
 don't run, you usually don't have time to setup your preferred work
 environment, notepad comes in handy to just view the log and diagnose an
 error.

-- Dmitry Olshansky
Mar 21 2013
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 21 March 2013 at 05:01:06 UTC, Kagamin wrote:
 On Wednesday, 20 March 2013 at 14:34:21 UTC, Nick Sabalausky 
 wrote:
 Since *at least* as far back as XP, Windows has handled "\n" 
 newlines
 perfectly fine. The command line displays them properly, .BAT 
 scripts
 handle them properly, every code editor in existence handles 
 them
 properly. The *only* thing I've found that doesn't is Windows 
 Notepad,
 but really, whoTF uses that anyway?

Notepad is used as a log viewer. When you're presented a machine you don't run, you usually don't have time to setup your preferred work environment, notepad comes in handy to just view the log and diagnose an error.

My workaround in such cases is to use Write, which is also installed by default and handles the files correctly. I think Notepad still uses the same code from Windows 3.x days! :) -- Paulo
Mar 21 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 21 Mar 2013 10:41:35 +0100
Jacob Carlborg <doob me.com> wrote:

 On 2013-03-20 22:08, Nick Sabalausky wrote:
 
 I doubt there's a bugzilla entry for this since it is,
 unfortunately, the intended behavior. I didn't want to go posting a
 'zilla issue for it before discussing here because I figured that
 might just end up "INVALID" or "WONTFIX".

If you make it an enhancement request it shouldn't at least end up as "INVALID".

http://d.puremagic.com/issues/show_bug.cgi?id=9776
Mar 21 2013
prev sibling next sibling parent reply torhu <no spam.invalid> writes:
On 20.03.2013 15:34, Nick Sabalausky wrote:
 Since *at least* as far back as XP, Windows has handled "\n" newlines
 perfectly fine. The command line displays them properly, .BAT scripts
 handle them properly, every code editor in existence handles them
 properly. The *only* thing I've found that doesn't is Windows Notepad,
 but really, whoTF uses that anyway?

 So, why are silently and forcefully converting "\n" to "\r\n" on
 windows by default? All it does is cause bugs. For example:
 https://github.com/repeatedly/mustache-d/issues/3

 And that's definitely not the first time I've run into problems due
 using the "write*" functions instead of rawWrite.

 Consider this straightforward code:

 --------------------------------------
 import std.file;
 import std.stdio;

 void transform(string str)
 {
 	/+ ...perform some modification of 'str'... +/
 	return str;
 }

 void main()
 {
 	auto str = cast(string) read(args[1]);
 	str = transform(str);
 	write(str);
 }
 --------------------------------------

 That simple code is *wrong*:

 It works correctly for all input on Unix: Output newlines match input
 newlines. Always. The code never asks for newlines to be messed with,
 and therefore they never are.

You're mixing binary and text mode functions. read() is binary, stdout.write() is text mode. And yes, you are asking for newlines to be messed with, as File.write is documented to write in text mode. But I agree that the docs need improvement. And maybe the API.
Mar 21 2013
parent torhu <no spam.invalid> writes:
On 22.03.2013 00:36, Nick Sabalausky wrote:
 On Thu, 21 Mar 2013 23:37:06 +0100
 torhu <no spam.invalid> wrote:
 You're mixing binary and text mode functions.  read() is binary,
 stdout.write() is text mode.  And yes, you are asking for newlines to
 be messed with, as File.write is documented to write in text mode.

 But I agree that the docs need improvement.  And maybe the API.

You're missing the point. The point is that the "text mode" is bug-prone, grossly obsolete, and completely useless and therefore should absolutely not be the default, *if* it has any reason to even exist at all.

Text mode isn't going away, I don't know where you'd get that idea from. I'm sure Microsoft wants Linux to go away, too. If you want a to finance a FUD campaign to hurt the reputation of text mode, be my guest. Or keep on ranting, who's gonna stop you.
Mar 21 2013
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 21 Mar 2013 23:37:06 +0100
torhu <no spam.invalid> wrote:
 
 You're mixing binary and text mode functions.  read() is binary, 
 stdout.write() is text mode.  And yes, you are asking for newlines to
 be messed with, as File.write is documented to write in text mode.
 
 But I agree that the docs need improvement.  And maybe the API.

You're missing the point. The point is that the "text mode" is bug-prone, grossly obsolete, and completely useless and therefore should absolutely not be the default, *if* it has any reason to even exist at all. We could toss a function "output" into Phobos and document it as being "fubar mode, which converts every third word into 'DERP'", but obviously just because its behavior matches the description doesn't justify its existence or its usage of such a generic name. The current "write" function is every bit as useless as this hypothetical "output" function, but it's more dangerous because it's harder to notice you're getting the wrong result.
Mar 21 2013