www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Tango I/O bug?

reply Jason House <jason.james.house gmail.com> writes:
My friend using a mac has had severe issues getting my project (recently ported
to Tango) to run on his machine.

Looking at debug output and doing some tests, it appears that Stdout is not
flushing.  

Typical offending code:
auto outStream = new Print!(char)(new Layout!(char),Stdout);
outStream.formatln("= {}\n", valueToReturn); // adds extra \n and flushes


Sometimes I get a non-response too, but restarting the program usually fixes
it.  It appears that on the mac this is not the case.  Any ideas what could be
wrong?  I do not know which version of tango my friend is using.  I think I'm
using Tango 0.99 rc1 with dmd 1.020.
Sep 17 2007
parent reply Sean Kelly <sean f4.ca> writes:
Jason House wrote:
 My friend using a mac has had severe issues getting my project (recently
ported to Tango) to run on his machine.
 
 Looking at debug output and doing some tests, it appears that Stdout is not
flushing.  
 
 Typical offending code:
 auto outStream = new Print!(char)(new Layout!(char),Stdout);

Is this really the line used? The second parameter is supposed to be an OutputStream, and Stdout is not.
 outStream.formatln("= {}\n", valueToReturn); // adds extra \n and flushes

Seems okay, though it's difficult to say from just that line. Is output redirected? Are objects being re-used? Is .close() perhaps not being called if output is to a file? Sean
Sep 17 2007
parent reply Jason House <jason.james.house gmail.com> writes:
Sean Kelly Wrote:

 Jason House wrote:
 Typical offending code:
 auto outStream = new Print!(char)(new Layout!(char),Stdout);

Is this really the line used? The second parameter is supposed to be an OutputStream, and Stdout is not.

No, I did a quick hack to make it short. The code was more generic with the last input parameter being passed in from elsewhere. I took a quick swag at it without verifying with other code. The real code compiles ;) Here's the key lines from the class: Print!(char) outStream; this(InputStream _inStream=Cin.stream, OutputStream _outStream=Cout.stream, double _pollPeriod=0.1){ outStream = new Print!(char)(new Layout!(char),_outStream); ...
 outStream.formatln("= {}\n", valueToReturn); // adds extra \n and flushes

Seems okay, though it's difficult to say from just that line. Is output redirected? Are objects being re-used? Is .close() perhaps not being called if output is to a file?

Output is going to Cout.stream and is never closed. The code relies on formatln to flush the output. It does real time communication in plain text with a 3rd party controlling application. This line really was from the program. In this particular example, valueToReturn is of type char[] and is literally "HouseBot"
Sep 17 2007
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Jason House wrote:
 Sean Kelly Wrote:
 
 Jason House wrote:
 outStream.formatln("= {}\n", valueToReturn); // adds extra \n and flushes

redirected? Are objects being re-used? Is .close() perhaps not being called if output is to a file?

Output is going to Cout.stream and is never closed. The code relies on formatln to flush the output. It does real time communication in plain text with a 3rd party controlling application. This line really was from the program. In this particular example, valueToReturn is of type char[] and is literally "HouseBot"

This part is likely your problem: "It does real time communication in plain text with a 3rd party controlling application" That probably means the output is redirected and not sent to an actual console, right? The default behavior for Stdout is to auto-flush only if the output is sent to an actual console (i.e. isn't redirected). To make sure flushing occurs, execute "Stdout.flush = true;" somewhere at the start of your program (main(), a static this(), or just right before the first output).
Sep 17 2007
next sibling parent reply kris <foo bar.com> writes:
Frits van Bommel wrote:
 Jason House wrote:
 Sean Kelly Wrote:

 Jason House wrote:
 outStream.formatln("= {}\n", valueToReturn); // adds extra \n and 
 flushes

output redirected? Are objects being re-used? Is .close() perhaps not being called if output is to a file?

Output is going to Cout.stream and is never closed. The code relies on formatln to flush the output. It does real time communication in plain text with a 3rd party controlling application. This line really was from the program. In this particular example, valueToReturn is of type char[] and is literally "HouseBot"

This part is likely your problem: "It does real time communication in plain text with a 3rd party controlling application" That probably means the output is redirected and not sent to an actual console, right? The default behavior for Stdout is to auto-flush only if the output is sent to an actual console (i.e. isn't redirected). To make sure flushing occurs, execute "Stdout.flush = true;" somewhere at the start of your program (main(), a static this(), or just right before the first output).

Yeah, Stdout and Cout inhibit *automatic* flush when the console is redirected. If you think about it, the whole automatic flush thing is a 'shortcut' so that output to the real console doesn't need an explicit call to flush() each time. It's a royal PITA to support that :) Anyway, the most effective route here is simply to invoke flush() explicitly, whenever you actually need to. For example: # outStream.format("blah blah").flush;
Sep 17 2007
parent reply Jason House <jason.james.house gmail.com> writes:
kris wrote:
 Anyway, the most effective route here is simply to invoke flush() 
 explicitly, whenever you actually need to. For example:
 
 # outStream.format("blah blah").flush;

I thought I checked to verify that formatln calls newline which calls flush. I therefore concluded that an explicit flush call was useless. My friend had the issue, so I couldn't really test. When I hear more about testing fixes, I'll post back on the list.
Sep 17 2007
parent kris <foo bar.com> writes:
Jason House wrote:
 
 kris wrote:
 Anyway, the most effective route here is simply to invoke flush() 
 explicitly, whenever you actually need to. For example:

 # outStream.format("blah blah").flush;

I thought I checked to verify that formatln calls newline which calls flush. I therefore concluded that an explicit flush call was useless. My friend had the issue, so I couldn't really test. When I hear more about testing fixes, I'll post back on the list.

Well, newline *does* call flush, but only if the console has not been redirected. This was not the case in some earlier Tango releases, but the change was made to accommodate requests for faster throughput (flush is very expensive on some platforms). In short, we may have caused your app to break. Please accept my apologies, and I hope it didn't cause too much trouble for you
Sep 17 2007
prev sibling parent reply Jason House <jason.james.house gmail.com> writes:
Sadly, this simple fix isn't the solution.  Here's what I got back (via
e-mail)...

 This part is likely your problem: "It does real time communication in
 plain text with a 3rd party controlling application"
 That probably means the output is redirected and not sent to an actual
 console, right?
 The default behavior for Stdout is to auto-flush only if the output is
 sent to an actual console (i.e. isn't redirected).

Well, it is sent to an actual console. I mean I just start housebot by hand and interact with it.
 To make sure flushing occurs, execute "Stdout.flush = true;" somewhere
 at the start of your program (main(), a static this(), or just right
 before the first output).

I tried that. I added the import for tango.io.Stdout to housebot.d and added "Stdout.flush = true;" at the beginning of main(), just after the variable declarations. But nothing changed.
Sep 18 2007
parent reply Jason House <jason.james.house gmail.com> writes:
Given that enabling a flush doesn't solve the problem, what is the path forward
for figuring out how to fix this?  It works on all other systems tested with
dmd, but doesn't work on mac (with gdc).  (I've had big problems getting gdc
and tango to play nicely with each other in a cygwin environment, but that's
probably another thread)

Jason House Wrote:

 Sadly, this simple fix isn't the solution.  Here's what I got back (via
e-mail)...
 
 This part is likely your problem: "It does real time communication in
 plain text with a 3rd party controlling application"
 That probably means the output is redirected and not sent to an actual
 console, right?
 The default behavior for Stdout is to auto-flush only if the output is
 sent to an actual console (i.e. isn't redirected).

Well, it is sent to an actual console. I mean I just start housebot by hand and interact with it.
 To make sure flushing occurs, execute "Stdout.flush = true;" somewhere
 at the start of your program (main(), a static this(), or just right
 before the first output).

I tried that. I added the import for tango.io.Stdout to housebot.d and added "Stdout.flush = true;" at the beginning of main(), just after the variable declarations. But nothing changed.

Sep 18 2007
parent reply Sean Kelly <sean f4.ca> writes:
For what it's worth, flush() merely writes all data buffered by Tango to 
the underlying device.  On non-Win32 platforms, there is no actual 
kernel call performed to force an immediate flush of the data to 
wherever.  Is it possible that OSX does buffering within the kernel and 
the data is lingering in there?  And if so, do you know of a routine 
that could be called to perform a hard flush?  Typically, the commit() 
method is used for this purpose, but it's empty on Posix systems right now.

Jason House wrote:
 Given that enabling a flush doesn't solve the problem, what is the path
forward for figuring out how to fix this?  It works on all other systems tested
with dmd, but doesn't work on mac (with gdc).  (I've had big problems getting
gdc and tango to play nicely with each other in a cygwin environment, but
that's probably another thread)
 
 Jason House Wrote:
 
 Sadly, this simple fix isn't the solution.  Here's what I got back (via
e-mail)...

 This part is likely your problem: "It does real time communication in
 plain text with a 3rd party controlling application"
 That probably means the output is redirected and not sent to an actual
 console, right?
 The default behavior for Stdout is to auto-flush only if the output is
 sent to an actual console (i.e. isn't redirected).

by hand and interact with it.
 To make sure flushing occurs, execute "Stdout.flush = true;" somewhere
 at the start of your program (main(), a static this(), or just right
 before the first output).

and added "Stdout.flush = true;" at the beginning of main(), just after the variable declarations. But nothing changed.


Sep 18 2007
parent reply Jason House <jason.james.house gmail.com> writes:
You already know more than I do... even after a while trying to google 
on the topic.  What would be the right forum to even ask questions like 
that?

Sean Kelly wrote:
 For what it's worth, flush() merely writes all data buffered by Tango to 
 the underlying device.  On non-Win32 platforms, there is no actual 
 kernel call performed to force an immediate flush of the data to 
 wherever.  Is it possible that OSX does buffering within the kernel and 
 the data is lingering in there?  And if so, do you know of a routine 
 that could be called to perform a hard flush?  Typically, the commit() 
 method is used for this purpose, but it's empty on Posix systems right now.
 
 Jason House wrote:
 Given that enabling a flush doesn't solve the problem, what is the 
 path forward for figuring out how to fix this?  It works on all other 
 systems tested with dmd, but doesn't work on mac (with gdc).  (I've 
 had big problems getting gdc and tango to play nicely with each other 
 in a cygwin environment, but that's probably another thread)

 Jason House Wrote:

 Sadly, this simple fix isn't the solution.  Here's what I got back 
 (via e-mail)...

 This part is likely your problem: "It does real time communication in
 plain text with a 3rd party controlling application"
 That probably means the output is redirected and not sent to an actual
 console, right?
 The default behavior for Stdout is to auto-flush only if the output is
 sent to an actual console (i.e. isn't redirected).

by hand and interact with it.
 To make sure flushing occurs, execute "Stdout.flush = true;" somewhere
 at the start of your program (main(), a static this(), or just right
 before the first output).

and added "Stdout.flush = true;" at the beginning of main(), just after the variable declarations. But nothing changed.



Sep 18 2007
parent reply kris <foo bar.com> writes:
Jason,

This is trivial to test, if you'd be prepared to do that? switch to the 
tango examples dir, and then the console dir (examples/console/)

Compile hello.d

execute it

What happens?
Sep 18 2007
parent reply Urban Hafner <urban bettong.net> writes:
kris wrote:
 Jason,

I guess it's easier if I start answering instead of having Jason relay messages.
 This is trivial to test, if you'd be prepared to do that? switch to the 
 tango examples dir, and then the console dir (examples/console/)
 
 Compile hello.d

That didn't quite work for me, with all the include and library files, so I instead did a "dsss build" in the examples directory.
 execute it
 
 What happens?

u koala:~/Programmieren/D/tango/example/console$ ./hello hello, sweetheart ☺ u koala:~/Programmieren/D/tango/example/console$ Is that what should happen? Urban
Sep 19 2007
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
Urban Hafner wrote:
 kris wrote:
 Jason,

I guess it's easier if I start answering instead of having Jason relay messages.
 This is trivial to test, if you'd be prepared to do that? switch to 
 the tango examples dir, and then the console dir (examples/console/)

 Compile hello.d

That didn't quite work for me, with all the include and library files, so I instead did a "dsss build" in the examples directory.
 execute it

 What happens?

u koala:~/Programmieren/D/tango/example/console$ ./hello hello, sweetheart ☺ u koala:~/Programmieren/D/tango/example/console$ Is that what should happen? Urban

If I understand things correctly, Urban does see program output once the program exits... He just doesn't see any while it's running. When I look at Hello.d, it looks like it prints and then exits... potentially hiding this particular problem.
Sep 19 2007
parent kris <foo bar.com> writes:
Jason House wrote:

 
 If I understand things correctly, Urban does see program output once the 
 program exits...  He just doesn't see any while it's running.  When I 
 look at Hello.d, it looks like it prints and then exits... potentially 
 hiding this particular problem.

Well, the code does exactly the same thing when running or exiting. All the happens at exit is a flush() call on Stdout, just in case the user hadn't done one already (courtesy of Frits). This, Any flush() should operate just fine. If the example had done nothing, it would have been more suspicious
Sep 19 2007
prev sibling parent kris <foo bar.com> writes:
Urban Hafner wrote:
 kris wrote:
 Jason,

I guess it's easier if I start answering instead of having Jason relay messages.
 This is trivial to test, if you'd be prepared to do that? switch to 
 the tango examples dir, and then the console dir (examples/console/)

 Compile hello.d

That didn't quite work for me, with all the include and library files, so I instead did a "dsss build" in the examples directory.
 execute it

 What happens?

u koala:~/Programmieren/D/tango/example/console$ ./hello hello, sweetheart ☺ u koala:~/Programmieren/D/tango/example/console$ Is that what should happen? Urban

Yes, that's exactly what should happen :)
Sep 19 2007