www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - std.process: how to process stdout chunk by chunk without waiting for

reply Timothee Cour <thelastmammoth gmail.com> writes:
I'd like to do the following:

auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

while(true){
version(A1)
  string line=pipes.stdout.readln;
version(A2)
  auto line=pipes.stdout.readChunk(10);
version(A3)
  auto line=pipes.stdout.readChar();

  // do something with line

  if(tryWait(pipes.pid).terminated)
    break;
}


The problem is that 'string line=pipes.stdout.readln;' seems to block until
the process is terminated, ie if the command is a long running command that
prints a line every 1 second for 10 seconds, this program will wait 10
seconds before starting the processing.
I also tried with rawRead, readf, fgetc but couldn't make it work.
I'm on OSX, if that matters.

Is there any way to achieve this?
Jun 18 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour  
<thelastmammoth gmail.com> wrote:

 I'd like to do the following:

 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block  
 until
 the process is terminated, ie if the command is a long running command  
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?
I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer. For instance, if the child is using D writeln or C printf, and you are using stdout, then it will only flush after writing 4096 bytes. You can flush early by calling flush on stdout, or fflush in C. Note that C will auto-detect if it is an interactive console, and flush via newlines instead. So running the same program from the console will flush every line! Alternatively, you can set the flush policy to flush after every line. See here: https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html And here: http://dlang.org/phobos/std_stdio.html#.File.setvbuf -Steve
Jun 18 2013
parent reply Timothee Cour <thelastmammoth gmail.com> writes:
On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
<schveiguy yahoo.com>wrote:

 On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
 thelastmammoth gmail.com> wrote:

  I'd like to do the following:
 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(**10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).**terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block
 until
 the process is terminated, ie if the command is a long running command
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?
I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer.
yes
 For instance, if the child is using D writeln or C printf, and you are
 using stdout, then it will only flush after writing 4096 bytes.  You can
 flush early by calling flush on stdout, or fflush in C.  Note that C will
 auto-detect if it is an interactive console, and flush via newlines
 instead.  So running the same program from the console will flush every
 line!

 Alternatively, you can set the flush policy to flush after every line.
  See here:

 https://developer.apple.com/**library/ios/#documentation/**
 System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>

 And here:

 http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>

 -Steve
Thanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program. I tried http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipewith script: auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout | Redirect.stderr); that works but has issues : only buffers stdout, not stderr; and I may not want to redirect stderr to stdout; also it won't work in more complex cases, eg if program contains '|' etc, and it requires replacing \r\n by \n. I tried replacing fork with forkpty inside std.process. That doesn't seem to work: calling isatty(1),isatty(2) on child process still returns 0. Not sure why. I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process creation in std.process (after the fork with case 0); doesn't work either.
Jun 19 2013
parent Kevin Lamonte <kevin.lamonte gmail.com> writes:
Timothee Cour wrote:

 On Tue, Jun 18, 2013 at 3:00 PM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 
 On Tue, 18 Jun 2013 17:41:57 -0400, Timothee Cour <
 thelastmammoth gmail.com> wrote:

  I'd like to do the following:
 auto pipes = pipeShell(command, Redirect.stdout | Redirect.stderr);

 while(true){
 version(A1)
   string line=pipes.stdout.readln;
 version(A2)
   auto line=pipes.stdout.readChunk(**10);
 version(A3)
   auto line=pipes.stdout.readChar();

   // do something with line

   if(tryWait(pipes.pid).**terminated)
     break;
 }


 The problem is that 'string line=pipes.stdout.readln;' seems to block
 until
 the process is terminated, ie if the command is a long running command
 that
 prints a line every 1 second for 10 seconds, this program will wait 10
 seconds before starting the processing.
 I also tried with rawRead, readf, fgetc but couldn't make it work.
 I'm on OSX, if that matters.

 Is there any way to achieve this?
I think the issue is on the child process side. If you are using buffered I/O you have to flush the buffer.
yes
 For instance, if the child is using D writeln or C printf, and you are
 using stdout, then it will only flush after writing 4096 bytes.  You can
 flush early by calling flush on stdout, or fflush in C.  Note that C will
 auto-detect if it is an interactive console, and flush via newlines
 instead.  So running the same program from the console will flush every
 line!

 Alternatively, you can set the flush policy to flush after every line.
  See here:

 https://developer.apple.com/**library/ios/#documentation/**
 
System/Conceptual/ManPages_**iPhoneOS/man3/setvbuf.3.html<https://developer.apple.com/library/ios/#documentation/System/Conceptual/ManPages_iPhoneOS/man3/setvbuf.3.html>
 And here:

 
http://dlang.org/phobos/std_**stdio.html#.File.setvbuf<http://dlang.org/phobos/std_stdio.html#.File.setvbuf>
 -Steve
Thanks, that does indeed work if I have source code for the child program and I can run 'std.stdio.stdout.setvbuf(buffer, _IOLBF);' right after main. However I want to make it work without modifying source of child program. I tried http://stackoverflow.com/questions/1401002/trick-an-application-into-
thinking-its-stdin-is-interactive-not-a-pipewith
 script:
 auto pipes = pipeShell("script -q /dev/null program", Redirect.stdout |
 Redirect.stderr);
 that works but has issues : only buffers stdout, not stderr; and I may not
 want to redirect stderr to stdout; also it won't work in more complex
 cases, eg if program contains '|' etc, and it requires replacing \r\n by
 \n.
 
 I tried replacing fork with forkpty inside std.process. That doesn't seem
 to work: calling isatty(1),isatty(2)   on child process still returns 0.
 Not sure why.
 
 I tried calling stdout.setvbuf(buffer, _IOLBF); right after child process
 creation in std.process (after the fork with case 0); doesn't work either.
I ran into this also. My solution is the makeShell() function at https://github.com/klamonte/d-tui/blob/master/tterminal.d . I'd like to see forkpty in phobos myself, so I just opened bug 10464 for that request.
Jun 24 2013