www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - forks/pipes and std.socket

reply "Nick Sabalausky" <a a.a> writes:
This may be a stupid question:

Does std.socket encorporate or replace pipe usage? Ie, if I'm going to do 
something along the lines of (psuedo-code):

auto parentToChild = pipe();
auto childToParent = pipe();
if(fork())
{
    // talk to other process
}
else
{
    // talk to other process
}

Is there anything in std.socket, or any way of using it that would aid in 
this sort of thing (ideally in a cross-platform way), or is it entirely 
orthogonal to this?

Does anyone who's done this sort of thing in D before, on Win or Lin, know 
of anything else in particular to be aware of?
Sep 05 2010
next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Mon, 06 Sep 2010 05:32:28 +0900, Nick Sabalausky <a a.a> wrote:


 This may be a stupid question:

 Does std.socket encorporate or replace pipe usage? Ie, if I'm going to do
 something along the lines of (psuedo-code):

 auto parentToChild = pipe();
 auto childToParent = pipe();
 if(fork())
 {
     // talk to other process
 }
 else
 {
     // talk to other process
 }

 Is there anything in std.socket, or any way of using it that would aid in
 this sort of thing (ideally in a cross-platform way), or is it entirely
 orthogonal to this?

 Does anyone who's done this sort of thing in D before, on Win or Lin,  
 know
 of anything else in particular to be aware of?


Unix domain socket?

http://lists.puremagic.com/pipermail/phobos/2010-July/001171.html

This socket I rewrote can communicate with other processes using  
LocalEndpoint.
(Sorry, I don't know Windows environment).
Sep 07 2010
prev sibling parent reply Kagamin <spam here.lot> writes:
Nick Sabalausky Wrote:


 Does anyone who's done this sort of thing in D before, on Win or Lin, know 
 of anything else in particular to be aware of? 


There's no fork on windows. If you want a multithreaded server, it's usually
implemented with threads on windows.
Sep 07 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Kagamin" <spam here.lot> wrote in message 
news:i660qi$nud$1 digitalmars.com...

 Nick Sabalausky Wrote:


 Does anyone who's done this sort of thing in D before, on Win or Lin, 
 know
 of anything else in particular to be aware of?


 There's no fork on windows. If you want a multithreaded server, it's 
 usually implemented with threads on windows.


That was just an example. CreateProcess/spawn/system are fine for my 
purposes. I'm just talking about creating a child process and communicating 
with it via pipes.
Sep 07 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 07 Sep 2010 14:54:46 -0400, Nick Sabalausky <a a.a> wrote:


 "Kagamin" <spam here.lot> wrote in message
 news:i660qi$nud$1 digitalmars.com...

 Nick Sabalausky Wrote:


 Does anyone who's done this sort of thing in D before, on Win or Lin,
 know
 of anything else in particular to be aware of?


 There's no fork on windows. If you want a multithreaded server, it's
 usually implemented with threads on windows.


 That was just an example. CreateProcess/spawn/system are fine for my
 purposes. I'm just talking about creating a child process and  
 communicating
 with it via pipes.


The upcoming std.process changes will make this easy.

-Steve
Sep 07 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.vioibdr2eav7ka localhost.localdomain...

 On Tue, 07 Sep 2010 14:54:46 -0400, Nick Sabalausky <a a.a> wrote:


 "Kagamin" <spam here.lot> wrote in message
 news:i660qi$nud$1 digitalmars.com...

 Nick Sabalausky Wrote:


 Does anyone who's done this sort of thing in D before, on Win or Lin,
 know
 of anything else in particular to be aware of?


 There's no fork on windows. If you want a multithreaded server, it's
 usually implemented with threads on windows.


 That was just an example. CreateProcess/spawn/system are fine for my
 purposes. I'm just talking about creating a child process and 
 communicating
 with it via pipes.


 The upcoming std.process changes will make this easy.


Ah cool, looking forward to it. I was just about ready to launch into a 
bunch of std.process improvements myself ;) In the meantime, it seems that 
File has an undocumented constructor that takes a file handle instead of a 
filename, so that should work for me.

BTW, do any of the upcoming std.process changes do anything to help work 
around Windows's exec being broken? By that I mean, exec is supposed to 
*reuse* the current process when launching the new one, but on Windows (and 
I *don't* believe this is D-specific) it creates a *new* process and kills 
the old one, which fucks up anything that waits on the original process to 
finish (such as the command-line). spawn(OVERLAY) has the same problem.
Sep 07 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 07 Sep 2010 16:51:48 -0400, Nick Sabalausky <a a.a> wrote:


 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.vioibdr2eav7ka localhost.localdomain...

 On Tue, 07 Sep 2010 14:54:46 -0400, Nick Sabalausky <a a.a> wrote:


 "Kagamin" <spam here.lot> wrote in message
 news:i660qi$nud$1 digitalmars.com...

 Nick Sabalausky Wrote:


 Does anyone who's done this sort of thing in D before, on Win or Lin,
 know
 of anything else in particular to be aware of?


 There's no fork on windows. If you want a multithreaded server, it's
 usually implemented with threads on windows.


 That was just an example. CreateProcess/spawn/system are fine for my
 purposes. I'm just talking about creating a child process and
 communicating
 with it via pipes.


 The upcoming std.process changes will make this easy.


 Ah cool, looking forward to it. I was just about ready to launch into a
 bunch of std.process improvements myself ;) In the meantime, it seems  
 that
 File has an undocumented constructor that takes a file handle instead of  
 a
 filename, so that should work for me.


Beware, it takes a file *descriptor*, which is different from a *HANDLE*  
on windows.  In writing the updated std.process I had to write D code that  
mimics the internals of DMC's runtime in order to translate between file  
descriptors and handles, not a pretty sight...

The updates are being held back right now by bug 3979.


 BTW, do any of the upcoming std.process changes do anything to help work
 around Windows's exec being broken? By that I mean, exec is supposed to
 *reuse* the current process when launching the new one, but on Windows  
 (and
 I *don't* believe this is D-specific) it creates a *new* process and  
 kills
 the old one, which fucks up anything that waits on the original process  
 to
 finish (such as the command-line). spawn(OVERLAY) has the same problem.


Windows' exec is a hack.  std.process will not use it, instead, we'll use  
CreateProcess directly.

FWIW, we are not going to support fork or exec individually in  
std.process.  We will only support creating a new process via fork+exec or  
CreateProcess.  This is the common functionality among OSes.

If you want fork or exec individually, you have to use them on posix  
systems only, and call them directly.

-Steve
Sep 08 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.vipr20xfeav7ka localhost.localdomain...

 On Tue, 07 Sep 2010 16:51:48 -0400, Nick Sabalausky <a a.a> wrote:


 Ah cool, looking forward to it. I was just about ready to launch into a
 bunch of std.process improvements myself ;) In the meantime, it seems 
 that
 File has an undocumented constructor that takes a file handle instead of 
 a
 filename, so that should work for me.


 Beware, it takes a file *descriptor*, which is different from a *HANDLE* 
 on windows.  In writing the updated std.process I had to write D code that 
 mimics the internals of DMC's runtime in order to translate between file 
 descriptors and handles, not a pretty sight...


I'm just feeding it the values I get from CreatePipe:
http://msdn.microsoft.com/en-us/library/aa365152(VS.85).aspx

Seems to be working fine, though...


 The updates are being held back right now by bug 3979.


 BTW, do any of the upcoming std.process changes do anything to help work
 around Windows's exec being broken? By that I mean, exec is supposed to
 *reuse* the current process when launching the new one, but on Windows 
 (and
 I *don't* believe this is D-specific) it creates a *new* process and 
 kills
 the old one, which fucks up anything that waits on the original process 
 to
 finish (such as the command-line). spawn(OVERLAY) has the same problem.


 Windows' exec is a hack.  std.process will not use it, instead, we'll use 
 CreateProcess directly.


Yea, I've come to the conclusion, too, that exec and spawn(OVERLAY) are 
worthless on Windows. I was going to create and submit a patch that avoids 
them and uses system instead (which *does* maintain the current pid) and 
then exits. But if someone's already been doing this, and using 
CreateProcess directly, then all the better :)


 FWIW, we are not going to support fork or exec individually in 
 std.process.  We will only support creating a new process via fork+exec or 
 CreateProcess.  This is the common functionality among OSes.

 If you want fork or exec individually, you have to use them on posix 
 systems only, and call them directly.


Sounds good.
Sep 09 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 09 Sep 2010 16:25:44 -0400, Nick Sabalausky <a a.a> wrote:


 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.vipr20xfeav7ka localhost.localdomain...

 On Tue, 07 Sep 2010 16:51:48 -0400, Nick Sabalausky <a a.a> wrote:


 Ah cool, looking forward to it. I was just about ready to launch into a
 bunch of std.process improvements myself ;) In the meantime, it seems
 that
 File has an undocumented constructor that takes a file handle instead  
 of
 a
 filename, so that should work for me.


 Beware, it takes a file *descriptor*, which is different from a *HANDLE*
 on windows.  In writing the updated std.process I had to write D code  
 that
 mimics the internals of DMC's runtime in order to translate between file
 descriptors and handles, not a pretty sight...


 I'm just feeding it the values I get from CreatePipe:
 http://msdn.microsoft.com/en-us/library/aa365152(VS.85).aspx

 Seems to be working fine, though...


That won't work.  I'm surprised it works at all.  DMC maintains the list  
of handles in an array, with the index of the array being the file  
descriptor, and the value at that element being the HANDLE.  If it's  
working, you're extremely lucky that the unallocated slot at the index  
that happens to be that HANDLE value has the same value as the HANDLE  
itself.

Or maybe I'm misunderstanding something...  Walter?

-Steve
Sep 09 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.vir82cineav7ka localhost.localdomain...

 On Thu, 09 Sep 2010 16:25:44 -0400, Nick Sabalausky <a a.a> wrote:


 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.vipr20xfeav7ka localhost.localdomain...

 On Tue, 07 Sep 2010 16:51:48 -0400, Nick Sabalausky <a a.a> wrote:


 Ah cool, looking forward to it. I was just about ready to launch into a
 bunch of std.process improvements myself ;) In the meantime, it seems
 that
 File has an undocumented constructor that takes a file handle instead 
 of
 a
 filename, so that should work for me.


 Beware, it takes a file *descriptor*, which is different from a *HANDLE*
 on windows.  In writing the updated std.process I had to write D code 
 that
 mimics the internals of DMC's runtime in order to translate between file
 descriptors and handles, not a pretty sight...


 I'm just feeding it the values I get from CreatePipe:
 http://msdn.microsoft.com/en-us/library/aa365152(VS.85).aspx

 Seems to be working fine, though...


 That won't work.  I'm surprised it works at all.  DMC maintains the list 
 of handles in an array, with the index of the array being the file 
 descriptor, and the value at that element being the HANDLE.  If it's 
 working, you're extremely lucky that the unallocated slot at the index 
 that happens to be that HANDLE value has the same value as the HANDLE 
 itself.

 Or maybe I'm misunderstanding something...  Walter?


Here's a simplified example of what I'm doing:

--------------------------------

 type mainapp.d


import std.conv;
import std.process;
import std.stdio;
import std.stream;
import core.sys.windows.windows;

// From here: http://msdn.microsoft.com/en-us/library/aa365152(VS.85).aspx
extern(Windows) int CreatePipe(
    HANDLE* hReadPipe,
    HANDLE* hWritePipe,
    SECURITY_ATTRIBUTES* lpPipeAttributes,
    uint nSize);

void main()
{
    // Set up pipe
    HANDLE readHandle;
    HANDLE writeHandle;
    auto secAttr = SECURITY_ATTRIBUTES(SECURITY_ATTRIBUTES.sizeof, null, 
true);
    if(!CreatePipe(&readHandle, &writeHandle, &secAttr, 0))
        throw new Exception("Couldn't create pipe");

    auto pipeReader = new std.stream.File(readHandle, FileMode.In);

    // Run subapp
    system("subapp "~to!string(cast(size_t)writeHandle));

    // Retrieve value from pipe
    int valueFromSubApp;
    pipeReader.read(valueFromSubApp);
    writeln("Received: ", valueFromSubApp);
}



 type subapp.d


import std.conv;
import std.stdio;
import std.stream;
import std.c.windows.windows;

void main(string[] args)
{
    auto writeHandle = cast(HANDLE)std.conv.to!size_t(args[1]);
    auto pipeWriter = new std.stream.File(writeHandle, FileMode.Out);

    int value = 777;
    writeln("Sending: ", value);
    pipeWriter.write(value);
}


 dmd subapp.d
 dmd mainapp.d
 mainapp


Sending: 777
Received: 777
--------------------------------


Maybe the OS is just allowing me to use the wrong file descriptor?
Sep 09 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 09 Sep 2010 18:29:55 -0400, Nick Sabalausky <a a.a> wrote:



 Maybe the OS is just allowing me to use the wrong file descriptor?


OK, after downloading the latest dmd, testing your code, and perplexing a  
while, then reading the source of phobos, I realized that you are using  
std.stream.File, not std.stdio.File.  The former is being deprecated for  
the latter.

Yes, std.stream.File will work.  But std.stdio.File is what is going to be  
supported.  std.stream is scheduled for deprecation.

-Steve
Sep 10 2010
parent "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.vitle0kgeav7ka localhost.localdomain...

 On Thu, 09 Sep 2010 18:29:55 -0400, Nick Sabalausky <a a.a> wrote:



 Maybe the OS is just allowing me to use the wrong file descriptor?


 OK, after downloading the latest dmd, testing your code, and perplexing a 
 while, then reading the source of phobos, I realized that you are using 
 std.stream.File, not std.stdio.File.  The former is being deprecated for 
 the latter.

 Yes, std.stream.File will work.  But std.stdio.File is what is going to be 
 supported.  std.stream is scheduled for deprecation.


Ahh ok, I thought it seemed strange that there were both. std.stdio.File is 
going to allow using a file descriptor, right? It looked like all it 
currently supports in its public interface was a filename, but maybe I'm 
remembering wrong.

Supposing I did need to convert the handle to a file descriptor, is there 
already a way to do that, or is that in-the-works?
Sep 10 2010