www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.process - POSIX specific callback

reply "nazriel" <spam dzfl.pl> writes:
Would it be possible to add to std.process.Config POSIX specific 
callback which would be called after fork()?

It is currently main blocker in switching dpaste-be from handmade 
process handling module to std.process.

It could look something like this.

struct Config {
      // current fields
      void delegate() posixCallback;
}

// ...

int i = fork();
if (i > 0)
{
//...
    if (config.posixCallback !is null)
         config.posixCallback();
//...
}

Such construct would allow for various child process 
manipulation, for instance
dropping root privileges or setting limits via setrmlimit.

Example:

config.posixCallback = {
     setguid(ourGUID);
     setgroups(ourGROUPS);
     setuid(ourUID);

     setrmlimit(NFORK, 123);
};


AFAIK we already have Windows specific flag related to spawning 
console for GUI apps.

I can make pull request ASAP when I get reasonable name for field.

Lars? ;)
Jun 06 2013
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 06 Jun 2013 12:05:14 -0400, nazriel <spam dzfl.pl> wrote:


 Would it be possible to add to std.process.Config POSIX specific  
 callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from handmade  
 process handling module to std.process.

 It could look something like this.

 struct Config {
       // current fields
       void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
     if (config.posixCallback !is null)
          config.posixCallback();
 //...
 }

 Such construct would allow for various child process manipulation, for  
 instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
      setguid(ourGUID);
      setgroups(ourGROUPS);
      setuid(ourUID);

      setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to spawning console  
 for GUI apps.

 I can make pull request ASAP when I get reasonable name for field.

 Lars? ;)


I agree with the ability, but not with the interface.  If this is to be  
done, it should be at a lower level, not inside config.  Keep in mind that  
std.process is generalized for both Windows and Posix, with very minor  
differences.

-Steve
Jun 06 2013
parent reply "nazriel" <spam dzfl.pl> writes:
On Thursday, 6 June 2013 at 17:18:20 UTC, Steven Schveighoffer 
wrote:

 On Thu, 06 Jun 2013 12:05:14 -0400, nazriel <spam dzfl.pl> 
 wrote:


 Would it be possible to add to std.process.Config POSIX 
 specific callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from 
 handmade process handling module to std.process.

 It could look something like this.

 struct Config {
      // current fields
      void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
    if (config.posixCallback !is null)
         config.posixCallback();
 //...
 }

 Such construct would allow for various child process 
 manipulation, for instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
     setguid(ourGUID);
     setgroups(ourGROUPS);
     setuid(ourUID);

     setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to 
 spawning console for GUI apps.

 I can make pull request ASAP when I get reasonable name for 
 field.

 Lars? ;)


 I agree with the ability, but not with the interface.  If this 
 is to be done, it should be at a lower level, not inside 
 config.  Keep in mind that std.process is generalized for both 
 Windows and Posix, with very minor differences.

 -Steve


I am aware that std.process is generalized but I doubt such 
useful functionality which is usable on various Posixen is more 
disturbing than Windows-only suprpressConsole 
https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954

But I was mistaken. Config is an enum not struct, so yeah, not 
worth changing it only for sake of posix callback.

So maybe module level variable?

module std.process;

// ...
void delegate() posixPostFork = null;
// ...


I would *really* love to see this implemented. It is really basic 
stuff for posixen.

Thanks a lot for responding Steven.
Jun 06 2013
parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:

 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is more 
 disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954


I think there is a huge difference between a simple flag and the
ability to execute arbitrary code on one OS but not on another.
(When set, suppressConsole actually *eliminates* a difference in
the default behaviour of the two OS families.)


 But I was mistaken. Config is an enum not struct, so yeah, not 
 worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...


Global state?  Don't want to go there...



 I would *really* love to see this implemented. It is really 
 basic stuff for posixen.


It needs a good API and community support.  I don't think we
should introduce new functionality, that looks like it was bolted
on, because one person said they really needed it.

Is it possible to abstract the things you would like to do in
such a callback?  You mention privilege lowering as a use case.
Can we make an API that does this, and which modifies the
process' security context in Windows in an equivalent/similar
way, for instance?
Jun 06 2013
next sibling parent reply "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 05:59:24 UTC, Lars T. Kyllingstad wrote:

 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:

 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is 
 more disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954


 I think there is a huge difference between a simple flag and the
 ability to execute arbitrary code on one OS but not on another.
 (When set, suppressConsole actually *eliminates* a difference in
 the default behaviour of the two OS families.)


Depends on the point of view.
In my opinion both suppressConsole and posixCallack defines 
process details after process space is created.
The only difference is that suppressConsole is an exceptional 
switch to define behavior on exactly one platform (Windows) while 
posixCallback allows defying behavior on multiple POSIX 
compatible systems.



 But I was mistaken. Config is an enum not struct, so yeah, not 
 worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...


 Global state?  Don't want to go there...


Just proposition. I don't know what way you and Steven prefer.


 I would *really* love to see this implemented. It is really 
 basic stuff for posixen.


 It needs a good API and community support.  I don't think we
 should introduce new functionality, that looks like it was 
 bolted
 on, because one person said they really needed it.


I understand. I just didn't except that this will be so 
controversial.
I thought that chopping off a lot of potential functionality on 
POSIX was just oversight, given that Windows has its own specific 
flag.

If such functionality isn't needed and I am the only one whining 
about it, let's just forget about whole topic.
I can live with my own more specialized implementation, I just 
thought it may be useful for others.


 Is it possible to abstract the things you would like to do in
 such a callback?  You mention privilege lowering as a use case.


My use cases are privilege lowering and setting up process limits 
(on per process basics).
But my point was that post-fork callback opens the window of 
additional process tuning and other possibilities, like dumping 
memory map or whatever user may want to do.


 Can we make an API that does this, and which modifies the
 process' security context in Windows in an equivalent/similar
 way, for instance?


I think we can, but then it closes various opportunities on POSIX.
I'm not that much experienced with Windows to be honest, but I 
think it is possible to switch user, but not possible to set per 
process limits (at least not possible on non-Server versions of 
Windows).

Again, I am not forcing anything on you or Steven.
I am asking you for opinion on something I find useful because 
you are the experts and I really like your work on new std.process

Best regards,
Damian Ziemba
Jun 07 2013
parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:

 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because 
 you are the experts and I really like your work on new 
 std.process


It's not up to us either.  If the community wants it, and it can 
be implemented in a seamless manner, it should be.

By the way, did you look at Dennis Luehring's suggestion?

   http://linux.die.net/man/3/pthread_atfork

This looks like just what you need and more, it is part of the 
POSIX standard, and as far as I can see it can be used together 
with std.process.  You just call pthread_atfork() before you call 
spawnProcess().
Jun 07 2013
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 07.06.2013 10:21, schrieb Lars T. Kyllingstad:

 On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:

 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because
 you are the experts and I really like your work on new
 std.process


 It's not up to us either.  If the community wants it, and it can
 be implemented in a seamless manner, it should be.

 By the way, did you look at Dennis Luehring's suggestion?

     http://linux.die.net/man/3/pthread_atfork

 This looks like just what you need and more, it is part of the
 POSIX standard, and as far as I can see it can be used together
 with std.process.  You just call pthread_atfork() before you call
 spawnProcess().


maybe

http://sourceware.org/pthreads-win32/

or this nice fork
https://github.com/GerHobbelt/pthread-win32

can give a hint how pthread_atfork can "work" under windosw
and show a way how to unifie the idea for both worlds
Jun 07 2013
prev sibling parent "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 08:21:32 UTC, Lars T. Kyllingstad wrote:

 On Friday, 7 June 2013 at 07:57:07 UTC, nazriel wrote:

 Again, I am not forcing anything on you or Steven.
 I am asking you for opinion on something I find useful because 
 you are the experts and I really like your work on new 
 std.process


 It's not up to us either.  If the community wants it, and it 
 can be implemented in a seamless manner, it should be.

 By the way, did you look at Dennis Luehring's suggestion?

   http://linux.die.net/man/3/pthread_atfork

 This looks like just what you need and more, it is part of the 
 POSIX standard, and as far as I can see it can be used together 
 with std.process.  You just call pthread_atfork() before you 
 call spawnProcess().


Yeah, now I noticed Dennis response.
It may be the solution to my issue yes.

Lovely POSIX, you can always count on it *G*

Thanks Dennis, Lars for help.
Sorry for the noise I caused.
Jun 07 2013
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad
<public kyllingen.net> wrote:


 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:

 I am aware that std.process is generalized but I doubt such useful  
 functionality which is usable on various Posixen is more disturbing  
 than Windows-only suprpressConsole  
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954


 I think there is a huge difference between a simple flag and the
 ability to execute arbitrary code on one OS but not on another.
 (When set, suppressConsole actually *eliminates* a difference in
 the default behaviour of the two OS families.)


First, suppressConsole is a simple flag, basically passed through to the
CreateProcess function, so even though it's Windows-specific, so is the
behavior we are suppressing.  Consider that when you specify
suppressConsole on Posix, the flag works!  No console window is created :)

But what to do with arbitrary "run this code between fork and exec" on
windows?  It's not possible.  It doesn't belong in the generalized API.


 But I was mistaken. Config is an enum not struct, so yeah, not worth  
 changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...


 Global state?  Don't want to go there...


I agree that the global state is a bad idea, ideally you want to specify
PER CALL what happens on a fork/exec, not PER THREAD (or PER PROCESS).

But I think we need some way to hook this.  To give up all the niceties of  
std.process just so you can hook the fork/exec sequence seems overly  
burdensome.

What I am thinking of is possibly to expose the OS-specific spawnProcess  
implementation as an object with the API defined by it, similar to how  
writeln simply forwards to stdout.writeln.  We could have spawnProcess  
simply forward to posixProcessImpl.spawnProcess (or  
windowsProcessImpl.spawnProcess on windows)

Then if someone wants to actually take advantage of OS-specific features,  
they can call on the appropriate object.  It shouldn't compile where it's  
not implemented (e.g. windows spawnProcess shouldn't be callable on Linux).

Does this make sense?  I think it can be done without breaking any code.   
May be a lot of boilerplate :)

-Steve
Jun 10 2013
next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:

 But I think we need some way to hook this.  To give up all the 
 niceties of std.process just so you can hook the fork/exec 
 sequence seems overly burdensome.


But with the pthread_atfork() solution you don't have to.  Call 
that function before you call spawnProcess() or any of the other 
std.process functions, and it should Just Work (TM).  The nice 
thing here is that it is already part of the POSIX standard, and 
thus should be available on all relevant systems, and we don't 
have to adapt our cross-platform API at all.
Jun 10 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 10 Jun 2013 16:05:15 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:


 On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote:

 But I think we need some way to hook this.  To give up all the niceties  
 of std.process just so you can hook the fork/exec sequence seems overly  
 burdensome.


 But with the pthread_atfork() solution you don't have to.  Call that  
 function before you call spawnProcess() or any of the other std.process  
 functions, and it should Just Work (TM).  The nice thing here is that it  
 is already part of the POSIX standard, and thus should be available on  
 all relevant systems, and we don't have to adapt our cross-platform API  
 at all.


This is not a good solution.  It deals with the idea that when forking,  
only the calling thread is alive, all other threads are dead, and one of  
those dead threads may hold a lock.  Also note that the function pointers  
are function pointers, not delegates.

The idea is that prior to fork, you lock all mutexes you want to be  
unlocked.  Then after fork is called, you unlock those mutexes (thus  
ensuring no dead threads hold the locks).

I don't think it makes for a very good generalized solution to "I want to  
run this arbitrary code".

Also, according to SO, it doesn't even do what it means to do, since the  
newly created process thread can't unlock the mutexes:

http://stackoverflow.com/questions/2620313/how-to-use-pthread-atfork-and-pthread-once-to-reinitialize-mutexes-in-child

Not only that, but it seems to be permanent -- there is no "unregister  
pthread_atfork" call.  So this has to be a one-time *process-wide* and  
permanent solution.  If you wanted to run code for this specific call to  
spawnProcess, and not others, then you are SOL.

And finally, if your ultimate purpose is to call exec right after fork (as  
it is in the general case), you are penalized by having to wait for some  
mutex to be unlocked in order to fork.

-Steve
Jun 10 2013
parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 20:26:59 UTC, Steven Schveighoffer 
wrote:

 This is not a good solution.
 [...]


Ok, you make many good points that I hadn't even thought about.  
I admit, I didn't look too hard at the phtread_atfork() 
documentation after I'd decided it was just the thing. ;)  So 
maybe we have to do something about this after all.

I'll comment on your earlier post.
Jun 11 2013
prev sibling next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:

 What I am thinking of is possibly to expose the OS-specific 
 spawnProcess implementation as an object with the API defined 
 by it, similar to how writeln simply forwards to 
 stdout.writeln.  We could have spawnProcess simply forward to 
 posixProcessImpl.spawnProcess (or 
 windowsProcessImpl.spawnProcess on windows)

 Then if someone wants to actually take advantage of OS-specific 
 features, they can call on the appropriate object.  It 
 shouldn't compile where it's not implemented (e.g. windows 
 spawnProcess shouldn't be callable on Linux).


Why should we add an object?  Why not expose the OS-specific 
spawnProcess() implementation as it is -- a free function -- with 
an additional parameter which specifies a callback delegate?

Can you think of any other places we'd want to hook besides 
post-fork-pre-exec?
Jun 11 2013
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 11 Jun 2013 12:31:19 -0400, Lars T. Kyllingstad  
<public kyllingen.net> wrote:


 On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer wrote:

 What I am thinking of is possibly to expose the OS-specific  
 spawnProcess implementation as an object with the API defined by it,  
 similar to how writeln simply forwards to stdout.writeln.  We could  
 have spawnProcess simply forward to posixProcessImpl.spawnProcess (or  
 windowsProcessImpl.spawnProcess on windows)

 Then if someone wants to actually take advantage of OS-specific  
 features, they can call on the appropriate object.  It shouldn't  
 compile where it's not implemented (e.g. windows spawnProcess shouldn't  
 be callable on Linux).


 Why should we add an object?  Why not expose the OS-specific  
 spawnProcess() implementation as it is -- a free function -- with an  
 additional parameter which specifies a callback delegate?


We could do that too.  The only thing is that the implementation function  
is more rough -- it doesn't have all the nice overloads.  I was thinking  
of simply moving the overloads to an object, and then calling on the  
object's overloads.

But thinking about it now, it doesn't make sense.  The object's overloads  
would all have to support this callback parameter.  It's probably best to  
simply expose the underlying implementation.

The documentation should explicitly warn about this...  From reading the  
pthread_atfork man page, it is clear that there are very significant  
problems that can arise from running arbitrary code at that time.


 Can you think of any other places we'd want to hook besides  
 post-fork-pre-exec?


I can't think of anything else at the moment.  If we ever support async  
events in Phobos, we should add an event for "child exited".

-Steve
Jun 11 2013
prev sibling parent "nazriel" <spam dzfl.pl> writes:
On Monday, 10 June 2013 at 16:20:53 UTC, Steven Schveighoffer 
wrote:

 On Fri, 07 Jun 2013 01:59:22 -0400, Lars T. Kyllingstad
 <public kyllingen.net> wrote:


 On Thursday, 6 June 2013 at 17:32:25 UTC, nazriel wrote:

 I am aware that std.process is generalized but I doubt such 
 useful functionality which is usable on various Posixen is 
 more disturbing than Windows-only suprpressConsole 
 https://github.com/D-Programming-Language/phobos/blob/master/std/process.d#L954


 I think there is a huge difference between a simple flag and 
 the
 ability to execute arbitrary code on one OS but not on another.
 (When set, suppressConsole actually *eliminates* a difference 
 in
 the default behaviour of the two OS families.)


 First, suppressConsole is a simple flag, basically passed 
 through to the
 CreateProcess function, so even though it's Windows-specific, 
 so is the
 behavior we are suppressing.  Consider that when you specify
 suppressConsole on Posix, the flag works!  No console window is 
 created :)

 But what to do with arbitrary "run this code between fork and 
 exec" on
 windows?  It's not possible.  It doesn't belong in the 
 generalized API.


 But I was mistaken. Config is an enum not struct, so yeah, 
 not worth changing it only for sake of posix callback.

 So maybe module level variable?

 module std.process;

 // ...
 void delegate() posixPostFork = null;
 // ...


 Global state?  Don't want to go there...


 I agree that the global state is a bad idea, ideally you want 
 to specify
 PER CALL what happens on a fork/exec, not PER THREAD (or PER 
 PROCESS).

 But I think we need some way to hook this.  To give up all the 
 niceties of std.process just so you can hook the fork/exec 
 sequence seems overly burdensome.

 What I am thinking of is possibly to expose the OS-specific 
 spawnProcess implementation as an object with the API defined 
 by it, similar to how writeln simply forwards to 
 stdout.writeln.  We could have spawnProcess simply forward to 
 posixProcessImpl.spawnProcess (or 
 windowsProcessImpl.spawnProcess on windows)


I had no time yet to check out the pthread_atfork approach but I 
see you found some issue with that.

posixProcessImpl.spawnProcess and windowsProcessImpl.spawnProcess 
sounds very good.


 Then if someone wants to actually take advantage of OS-specific 
 features, they can call on the appropriate object.  It 
 shouldn't compile where it's not implemented (e.g. windows 
 spawnProcess shouldn't be callable on Linux).


That would be great.
Being able to use the easy, simple functions to get the work done 
and encourage to use them (by Docs and examples) but also allow 
access to more specialized, "less visible" functions as you 
proposed.


 Does this make sense?  I think it can be done without breaking 
 any code.  May be a lot of boilerplate :)

 -Steve


Thank you very much for looking into this.
Jun 11 2013
prev sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
are you talking about http://linux.die.net/man/3/pthread_atfork 
funktionality?

Am 06.06.2013 18:05, schrieb nazriel:

 Would it be possible to add to std.process.Config POSIX specific
 callback which would be called after fork()?

 It is currently main blocker in switching dpaste-be from handmade
 process handling module to std.process.

 It could look something like this.

 struct Config {
        // current fields
        void delegate() posixCallback;
 }

 // ...

 int i = fork();
 if (i > 0)
 {
 //...
      if (config.posixCallback !is null)
           config.posixCallback();
 //...
 }

 Such construct would allow for various child process
 manipulation, for instance
 dropping root privileges or setting limits via setrmlimit.

 Example:

 config.posixCallback = {
       setguid(ourGUID);
       setgroups(ourGROUPS);
       setuid(ourUID);

       setrmlimit(NFORK, 123);
 };


 AFAIK we already have Windows specific flag related to spawning
 console for GUI apps.

 I can make pull request ASAP when I get reasonable name for field.

 Lars? ;)
Jun 06 2013
parent "nazriel" <spam dzfl.pl> writes:
On Friday, 7 June 2013 at 06:27:32 UTC, dennis luehring wrote:

 are you talking about http://linux.die.net/man/3/pthread_atfork 
 funktionality?


Very interesting. That may be exactly what I need.
I will try this out and see how does it play with std.process.

Thanks a lot!
Jun 07 2013