www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How do I redirect stderr of a spawned process into an internal buffer?

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've talked about this before, there's a problem with spawning
multiple processes and letting them write to stderr asynchronously It
seems like this might be a Windows-only problem, I couldn't recreate
on Ubuntu but maybe that's because it was virtualized.

Take buggy.d:

import std.concurrency;
import std.process;

void test(int i)
{
    system("dmd buggy.d buggy.d");
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }
}

Run it via rdmd: rdmd buggy.d

The output is messy (ran via cmd.exe): http://paste.pocoo.org/show/513763/

So it just gets worse the more processes are spawned. One workaround
is to redirect each process' stderr to a unique file. Example:

import core.thread;
import std.concurrency;
import std.process;
import std.string;
import std.stdio;
import std.file : readText;

void test(int i)
{
    // redirect each process stderr to its own file
    system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
}

void main()
{
    foreach (i; 0 .. 100)
    {
        spawn(&test, i);
    }

    thread_joinAll();

    foreach (i; 0 .. 100)
    {
        writeln(format("Output %s = %s", i,
readText(format("error_%s.txt", i))));
    }
}

So now the output is perfect: http://paste.pocoo.org/show/513771/

But writing to files is rather slow. Is there anything in Phobos I can
use to spawn a process and redirect stderr to an internal buffer?

Tango had this sort of thing for D1, but I'm not seeing anything in
Phobos. Since I can only recreate this on Windows I'm ok with using a
WinAPI function if it provides this feature.
Nov 27 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 27 Nov 2011 17:37:24 -0500, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 I've talked about this before, there's a problem with spawning
 multiple processes and letting them write to stderr asynchronously It
 seems like this might be a Windows-only problem, I couldn't recreate
 on Ubuntu but maybe that's because it was virtualized.

 Take buggy.d:

 import std.concurrency;
 import std.process;

 void test(int i)
 {
     system("dmd buggy.d buggy.d");
 }

 void main()
 {
     foreach (i; 0 .. 100)
     {
         spawn(&test, i);
     }
 }

 Run it via rdmd: rdmd buggy.d

 The output is messy (ran via cmd.exe):  
 http://paste.pocoo.org/show/513763/

 So it just gets worse the more processes are spawned. One workaround
 is to redirect each process' stderr to a unique file. Example:

 import core.thread;
 import std.concurrency;
 import std.process;
 import std.string;
 import std.stdio;
 import std.file : readText;

 void test(int i)
 {
     // redirect each process stderr to its own file
     system(format("dmd workaround.d workaround.d 2> error_%s.txt", i));
 }

 void main()
 {
     foreach (i; 0 .. 100)
     {
         spawn(&test, i);
     }

     thread_joinAll();

     foreach (i; 0 .. 100)
     {
         writeln(format("Output %s = %s", i,
 readText(format("error_%s.txt", i))));
     }
 }

 So now the output is perfect: http://paste.pocoo.org/show/513771/

 But writing to files is rather slow. Is there anything in Phobos I can
 use to spawn a process and redirect stderr to an internal buffer?

 Tango had this sort of thing for D1, but I'm not seeing anything in
 Phobos. Since I can only recreate this on Windows I'm ok with using a
 WinAPI function if it provides this feature.


the yet-to-be-reviewed std.process upgrade should fix all your problems.   
However, I need to get Walter to include a fix in dmc so reading past the  
end of a pipe produces EOF instead of an invalid handle error.

I'll work on getting that pull request to Walter, this keeps coming up...

-Steve
Nov 28 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Cool stuff Steve, I'll be eager to review it when the time comes.
Nov 28 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Btw, is the new std.process hosted anywhere? My solution is
windows-specific and probably doesn't handle all the edge-cases. Even
if the new std.process isn't ready yet I'd love to look at your
implementation.
Nov 28 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 28 Nov 2011 17:42:54 -0500, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 Btw, is the new std.process hosted anywhere? My solution is
 windows-specific and probably doesn't handle all the edge-cases. Even
 if the new std.process isn't ready yet I'd love to look at your
 implementation.


Yes,but in order for it to work you need a new dmc runtime.

it's on my github account.  I'd have to send you the dmc runtime  
privately, send me a private email.

-Steve
Nov 29 2011