www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.process - POSIX specific callback

reply "nazriel" <spam dzfl.pl> writes:
Would it be possible to add to std.process.Config POSIX specific 
callback which would be called after fork()?

It is currently main blocker in switching dpaste-be from handmade 
process handling module to std.process.

It could look something like this.

struct Config {
      // current fields
      void delegate() posixCallback;
}

// ...

int i = fork();
if (i > 0)
{
//...
    if (config.posixCallback !is null)
         config.posixCallback();
//...
}

Such construct would allow for various child process 
manipulation, for instance
dropping root privileges or setting limits via setrmlimit.

Example:

config.posixCallback = {
     setguid(ourGUID);
     setgroups(ourGROUPS);
     setuid(ourUID);

     setrmlimit(NFORK, 123);
};


AFAIK we already have Windows specific flag related to spawning 
console for GUI apps.

I can make pull request ASAP when I get reasonable name for field.

Lars? ;)
Jun 06 2013
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 06 Jun 2013 12:05:14 -0400, nazriel <spam dzfl.pl> wrote:

 Would it be possible to add to std.process.Config POSIX specific  
 callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from handmade  
 process handling module to std.process.

 It could look something like this.

 struct Config {
       // current fields
       void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
     if (config.posixCallback !is null)
          config.posixCallback();
 //...
 }

 Such construct would allow for various child process manipulation, for  
 instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
      setguid(ourGUID);
      setgroups(ourGROUPS);
      setuid(ourUID);

      setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to spawning console  
 for GUI apps.

 I can make pull request ASAP when I get reasonable name for field.

 Lars? ;)

I agree with the ability, but not with the interface. If this is to be done, it should be at a lower level, not inside config. Keep in mind that std.process is generalized for both Windows and Posix, with very minor differences. -Steve
Jun 06 2013
parent dennis luehring <dl.soluz gmx.net> writes:
Am 07.06.2013 10:21, schrieb Lars T. Kyllingstad:
 On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:
 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because
 you are the experts and I really like your work on new
 std.process

It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess().

maybe http://sourceware.org/pthreads-win32/ or this nice fork https://github.com/GerHobbelt/pthread-win32 can give a hint how pthread_atfork can "work" under windosw and show a way how to unifie the idea for both worlds
Jun 07 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Thursday, 6 June 2013 at 17:18:20 UTC, Steven Schveighoffer 
wrote:
 On Thu, 06 Jun 2013 12:05:14 -0400, nazriel <spam dzfl.pl> 
 wrote:

 Would it be possible to add to std.process.Config POSIX 
 specific callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from 
 handmade process handling module to std.process.

 It could look something like this.

 struct Config {
      // current fields
      void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
    if (config.posixCallback !is null)
         config.posixCallback();
 //...
 }

 Such construct would allow for various child process 
 manipulation, for instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
     setguid(ourGUID);
     setgroups(ourGROUPS);
     setuid(ourUID);

     setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to 
 spawning console for GUI apps.

 I can make pull request ASAP when I get reasonable name for 
 field.

 Lars? ;)

I agree with the ability, but not with the interface. If this is to be done, it should be at a lower level, not inside config. Keep in mind that std.process is generalized for both Windows and Posix, with very minor differences. -Steve

I am aware that std.process is generalized but I doubt such useful functionality which is usable on various Posixen is more disturbing than Windows-only suprpressConsole https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954 But I was mistaken. Config is an enum not struct, so yeah, not worth changing it only for sake of posix callback. So maybe module level variable? module std.process; // ... void delegate() posixPostFork = null; // ... I would *really* love to see this implemented. It is really basic stuff for posixen. Thanks a lot for responding Steven.
Jun 06 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:
 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is more 
 disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954

I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.)
 But I was mistaken. Config is an enum not struct, so yeah, not 
 worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...

Global state? Don't want to go there...
 I would *really* love to see this implemented. It is really 
 basic stuff for posixen.

It needs a good API and community support. I don't think we should introduce new functionality, that looks like it was bolted on, because one person said they really needed it. Is it possible to abstract the things you would like to do in such a callback? You mention privilege lowering as a use case. Can we make an API that does this, and which modifies the process' security context in Windows in an equivalent/similar way, for instance?
Jun 06 2013
prev sibling next sibling parent dennis luehring <dl.soluz gmx.net> writes:
are you talking about http://linux.die.net/man/3/pthread_atfork 
funktionality?

Am 06.06.2013 18:05, schrieb nazriel:
 Would it be possible to add to std.process.Config POSIX specific
 callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from handmade
 process handling module to std.process.

 It could look something like this.

 struct Config {
        // current fields
        void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
      if (config.posixCallback !is null)
           config.posixCallback();
 //...
 }

 Such construct would allow for various child process
 manipulation, for instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
       setguid(ourGUID);
       setgroups(ourGROUPS);
       setuid(ourUID);

       setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to spawning
 console for GUI apps.

 I can make pull request ASAP when I get reasonable name for field.

 Lars? ;)

Jun 06 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 05:59:24 UTC, Lars T. Kyllingstad wrote:
 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:
 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is 
 more disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954

I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.)

Depends on the point of view. In my opinion both suppressConsole and posixCallack defines process details after process space is created. The only difference is that suppressConsole is an exceptional switch to define behavior on exactly one platform (Windows) while posixCallback allows defying behavior on multiple POSIX compatible systems.
 But I was mistaken. Config is an enum not struct, so yeah, not 
 worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...

Global state? Don't want to go there...

 I would *really* love to see this implemented. It is really 
 basic stuff for posixen.

It needs a good API and community support. I don't think we should introduce new functionality, that looks like it was bolted on, because one person said they really needed it.

I understand. I just didn't except that this will be so controversial. I thought that chopping off a lot of potential functionality on POSIX was just oversight, given that Windows has its own specific flag. If such functionality isn't needed and I am the only one whining about it, let's just forget about whole topic. I can live with my own more specialized implementation, I just thought it may be useful for others.
 Is it possible to abstract the things you would like to do in
 such a callback?  You mention privilege lowering as a use case.

My use cases are privilege lowering and setting up process limits (on per process basics). But my point was that post-fork callback opens the window of additional process tuning and other possibilities, like dumping memory map or whatever user may want to do.
 Can we make an API that does this, and which modifies the
 process' security context in Windows in an equivalent/similar
 way, for instance?

I think we can, but then it closes various opportunities on POSIX. I'm not that much experienced with Windows to be honest, but I think it is possible to switch user, but not possible to set per process limits (at least not possible on non-Server versions of Windows). Again, I am not forcing anything on you or Steven. I am asking you for opinion on something I find useful because you are the experts and I really like your work on new std.process Best regards, Damian Ziemba
Jun 07 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:
 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because 
 you are the experts and I really like your work on new 
 std.process

It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess().
Jun 07 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 06:27:32 UTC, dennis luehring wrote:
 are you talking about http://linux.die.net/man/3/pthread_atfork 
 funktionality?

Very interesting. That may be exactly what I need. I will try this out and see how does it play with std.process. Thanks a lot!
Jun 07 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 08:21:32 UTC, Lars T. Kyllingstad wrote:
 On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:
 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because 
 you are the experts and I really like your work on new 
 std.process

It's not up to us either. If the community wants it, and it can be implemented in a seamless manner, it should be. By the way, did you look at Dennis Luehring's suggestion? http://linux.die.net/man/3/pthread_atfork This looks like just what you need and more, it is part of the POSIX standard, and as far as I can see it can be used together with std.process. You just call pthread_atfork() before you call spawnProcess().

Yeah, now I noticed Dennis response. It may be the solution to my issue yes. Lovely POSIX, you can always count on it *G* Thanks Dennis, Lars for help. Sorry for the noise I caused.
Jun 07 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad
<public kyllingen.net> wrote:

 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:
 I am aware that std.process is generalized but I doubt such useful  
 functionality which is usable on various Posixen is more disturbing  
 than Windows-only suprpressConsole  
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954

I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.)

First, suppressConsole is a simple flag, basically passed through to the CreateProcess function, so even though it's Windows-specific, so is the behavior we are suppressing. Consider that when you specify suppressConsole on Posix, the flag works! No console window is created :) But what to do with arbitrary "run this code between fork and exec" on windows? It's not possible. It doesn't belong in the generalized API.
 But I was mistaken. Config is an enum not struct, so yeah, not worth  
 changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...

Global state? Don't want to go there...

I agree that the global state is a bad idea, ideally you want to specify PER CALL what happens on a fork/exec, not PER THREAD (or PER PROCESS). But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows) Then if someone wants to actually take advantage of OS-specific features, they can call on the appropriate object. It shouldn't compile where it's not implemented (e.g. windows spawnProcess shouldn't be callable on Linux). Does this make sense? I think it can be done without breaking any code. May be a lot of boilerplate :) -Steve
Jun 10 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:
 But I think we need some way to hook this.  To give up all the 
 niceties of std.process just so you can hook the fork/exec 
 sequence seems overly burdensome.

But with the pthread_atfork() solution you don't have to. Call that function before you call spawnProcess() or any of the other std.process functions, and it should Just Work (TM). The nice thing here is that it is already part of the POSIX standard, and thus should be available on all relevant systems, and we don't have to adapt our cross-platform API at all.
Jun 10 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 10 Jun 2013 16:05:15 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:

 On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote:
 But I think we need some way to hook this.  To give up all the niceties  
 of std.process just so you can hook the fork/exec sequence seems overly  
 burdensome.

But with the pthread_atfork() solution you don't have to. Call that function before you call spawnProcess() or any of the other std.process functions, and it should Just Work (TM). The nice thing here is that it is already part of the POSIX standard, and thus should be available on all relevant systems, and we don't have to adapt our cross-platform API at all.

This is not a good solution. It deals with the idea that when forking, only the calling thread is alive, all other threads are dead, and one of those dead threads may hold a lock. Also note that the function pointers are function pointers, not delegates. The idea is that prior to fork, you lock all mutexes you want to be unlocked. Then after fork is called, you unlock those mutexes (thus ensuring no dead threads hold the locks). I don't think it makes for a very good generalized solution to "I want to run this arbitrary code". Also, according to SO, it doesn't even do what it means to do, since the newly created process thread can't unlock the mutexes: http://stackoverflow.com/questions/2620313/how-to-use-pthread-atfork-and-pthread-once-to-reinitialize-mutexes-in-child Not only that, but it seems to be permanent -- there is no "unregister pthread_atfork" call. So this has to be a one-time *process-wide* and permanent solution. If you wanted to run code for this specific call to spawnProcess, and not others, then you are SOL. And finally, if your ultimate purpose is to call exec right after fork (as it is in the general case), you are penalized by having to wait for some mutex to be unlocked in order to fork. -Steve
Jun 10 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 20:26:59 UTC, Steven Schveighoffer 
wrote:
 This is not a good solution.
 [...]

Ok, you make many good points that I hadn't even thought about. I admit, I didn't look too hard at the phtread_atfork() documentation after I'd decided it was just the thing. ;) So maybe we have to do something about this after all. I'll comment on your earlier post.
Jun 11 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:
 What I am thinking of is possibly to expose the OS-specific 
 spawnProcess implementation as an object with the API defined 
 by it, similar to how writeln simply forwards to 
 stdout.writeln.  We could have spawnProcess simply forward to 
 posixProcessImpl.spawnProcess (or 
 windowsProcessImpl.spawnProcess on windows)

 Then if someone wants to actually take advantage of OS-specific 
 features, they can call on the appropriate object.  It 
 shouldn't compile where it's not implemented (e.g. windows 
 spawnProcess shouldn't be callable on Linux).

Why should we add an object? Why not expose the OS-specific spawnProcess() implementation as it is -- a free function -- with an additional parameter which specifies a callback delegate? Can you think of any other places we'd want to hook besides post-fork-pre-exec?
Jun 11 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 12:31:19 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:

 On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote:
 What I am thinking of is possibly to expose the OS-specific  
 spawnProcess implementation as an object with the API defined by it,  
 similar to how writeln simply forwards to stdout.writeln.  We could  
 have spawnProcess simply forward to posixProcessImpl.spawnProcess (or  
 windowsProcessImpl.spawnProcess on windows)

 Then if someone wants to actually take advantage of OS-specific  
 features, they can call on the appropriate object.  It shouldn't  
 compile where it's not implemented (e.g. windows spawnProcess shouldn't  
 be callable on Linux).

Why should we add an object? Why not expose the OS-specific spawnProcess() implementation as it is -- a free function -- with an additional parameter which specifies a callback delegate?

We could do that too. The only thing is that the implementation function is more rough -- it doesn't have all the nice overloads. I was thinking of simply moving the overloads to an object, and then calling on the object's overloads. But thinking about it now, it doesn't make sense. The object's overloads would all have to support this callback parameter. It's probably best to simply expose the underlying implementation. The documentation should explicitly warn about this... From reading the pthread_atfork man page, it is clear that there are very significant problems that can arise from running arbitrary code at that time.
 Can you think of any other places we'd want to hook besides  
 post-fork-pre-exec?

I can't think of anything else at the moment. If we ever support async events in Phobos, we should add an event for "child exited". -Steve
Jun 11 2013
prev sibling parent "nazriel" <spam dzfl.pl> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:
 On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad
 <public kyllingen.net> wrote:

 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:
 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is 
 more disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954

I think there is a huge difference between a simple flag and the ability to execute arbitrary code on one OS but not on another. (When set, suppressConsole actually *eliminates* a difference in the default behaviour of the two OS families.)

First, suppressConsole is a simple flag, basically passed through to the CreateProcess function, so even though it's Windows-specific, so is the behavior we are suppressing. Consider that when you specify suppressConsole on Posix, the flag works! No console window is created :) But what to do with arbitrary "run this code between fork and exec" on windows? It's not possible. It doesn't belong in the generalized API.
 But I was mistaken. Config is an enum not struct, so yeah, 
 not worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...

Global state? Don't want to go there...

I agree that the global state is a bad idea, ideally you want to specify PER CALL what happens on a fork/exec, not PER THREAD (or PER PROCESS). But I think we need some way to hook this. To give up all the niceties of std.process just so you can hook the fork/exec sequence seems overly burdensome. What I am thinking of is possibly to expose the OS-specific spawnProcess implementation as an object with the API defined by it, similar to how writeln simply forwards to stdout.writeln. We could have spawnProcess simply forward to posixProcessImpl.spawnProcess (or windowsProcessImpl.spawnProcess on windows)

I had no time yet to check out the pthread_atfork approach but I see you found some issue with that. posixProcessImpl.spawnProcess and windowsProcessImpl.spawnProcess sounds very good.
 Then if someone wants to actually take advantage of OS-specific 
 features, they can call on the appropriate object.  It 
 shouldn't compile where it's not implemented (e.g. windows 
 spawnProcess shouldn't be callable on Linux).

That would be great. Being able to use the easy, simple functions to get the work done and encourage to use them (by Docs and examples) but also allow access to more specialized, "less visible" functions as you proposed.
 Does this make sense?  I think it can be done without breaking 
 any code.  May be a lot of boilerplate :)

 -Steve

Thank you very much for looking into this.
Jun 11 2013