Timothee Cour <thelastmammoth gmail.com> writes:

I'd like to do the following:

auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

while(true){
version(A1)
  string line=pipes.stdout.readln;
version(A2)
  auto line=pipes.stdout.readChunk(10);
version(A3)
  auto line=pipes.stdout.readChar();

  // do something with line

  if(tryWait(pipes.pid).terminated)
    break;
}


The problem is that 'string line=pipes.stdout.readln;' seems to block until
the process is terminated, ie if the command is a long running command that
prints a line every 1 second for 10 seconds, this program will wait 10
seconds before starting the processing.
I also tried with rawRead, readf, fgetc but couldn't make it work.
I'm on OSX, if that matters.

Is there any way to achieve this?

"Steven Schveighoffer" <schveiguy yahoo.com> writes:

On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour  
<thelastmammoth gmail.com> wrote:

 I'd like to do the following:

 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block  
 until
 the process is terminated, ie if the command is a long running command  
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?

I think the issue is on the child process side.  If you are using buffered  
I/O you have to flush the buffer.

For instance, if the child is using D writeln or C printf, and you are  
using stdout, then it will only flush after writing 4096 bytes.  You can  
flush early by calling flush on stdout, or fflush in C.  Note that C will  
auto-detect if it is an interactive console, and flush via newlines  
instead.  So running the same program from the console will flush every  
line!

Alternatively, you can set the flush policy to flush after every line.   
See here:

https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html

And here:



-Steve

Timothee Cour <thelastmammoth gmail.com> writes:

On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
<schveiguy yahoo.com>wrote:

 On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
 thelastmammoth gmail.com> wrote:

  I'd like to do the following:
 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(**10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).**terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block
 until
 the process is terminated, ie if the command is a long running command
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?

 I think the issue is on the child process side.  If you are using buffered
 I/O you have to flush the buffer.

yes


 For instance, if the child is using D writeln or C printf, and you are
 using stdout, then it will only flush after writing 4096 bytes.  You can
 flush early by calling flush on stdout, or fflush in C.  Note that C will
 auto-detect if it is an interactive console, and flush via newlines
 instead.  So running the same program from the console will flush every
 line!

 Alternatively, you can set the flush policy to flush after every line.
  See here:

 https://developer.apple.com/**library/ios/#documentation/**
 System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>

 And here:



 -Steve

Thanks, that does indeed work if I have source code for the child program
and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main.
However I want to make it work without modifying source of child program.

I tried
http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipewith
script:
auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
Redirect.stderr);
that works but has issues : only buffers stdout, not stderr; and I may not
want to redirect stderr to stdout; also it won't work in more complex
cases, eg if program contains '|' etc, and it requires replacing \r\n by \n.

I tried replacing fork with forkpty inside std.process. That doesn't seem
to work: calling isatty(1),isatty(2)   on child process still returns 0.
Not sure why.

I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
creation in std.process (after the fork with case 0); doesn't work either.

Kevin Lamonte <kevin.lamonte gmail.com> writes:

Timothee Cour wrote:

 On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 
 On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
 thelastmammoth gmail.com> wrote:

  I'd like to do the following:
 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(**10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).**terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block
 until
 the process is terminated, ie if the command is a long running command
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?

 I think the issue is on the child process side.  If you are using
 buffered I/O you have to flush the buffer.

 
 yes
 
 
 For instance, if the child is using D writeln or C printf, and you are
 using stdout, then it will only flush after writing 4096 bytes.  You can
 flush early by calling flush on stdout, or fflush in C.  Note that C will
 auto-detect if it is an interactive console, and flush via newlines
 instead.  So running the same program from the console will flush every
 line!

 Alternatively, you can set the flush policy to flush after every line.
  See here:

 https://developer.apple.com/**library/ios/#documentation/**
 


System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
 And here:

 



 -Steve

 
 Thanks, that does indeed work if I have source code for the child program
 and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after
 main. However I want to make it work without modifying source of child
 program.
 
 I tried
 http://stackoverflow.com/questions/1401002/trick-an-application-into-

thinking-its-stdin-is-interactive-not-a-pipewith
 script:
 auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
 Redirect.stderr);
 that works but has issues : only buffers stdout, not stderr; and I may not
 want to redirect stderr to stdout; also it won't work in more complex
 cases, eg if program contains '|' etc, and it requires replacing \r\n by
 \n.
 
 I tried replacing fork with forkpty inside std.process. That doesn't seem
 to work: calling isatty(1),isatty(2)   on child process still returns 0.
 Not sure why.
 
 I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
 creation in std.process (after the fork with case 0); doesn't work either.

I ran into this also.  My solution is the makeShell() function at 
https://github.com/klamonte/d-tui/blob/master/tterminal.d .  I'd like to see 
forkpty in phobos myself, so I just opened bug 10464 for that request.