www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - What is safe to do in an extern (C) function and how can I test this?

reply Jaime <benjamin.i.mccann gmail.com> writes:
**The lede**:

Can I, for instance, safely call Fiber.yield in a C callback that 
I know will be run in a Fiber?

The stack will look like:
Thread
|- Fiber in D runtime
| |- Call into a C API (stays on same thread)
| | |- Extern (C) callback (stays on same thread)
| | | |- Fiber.yield <-- Is this OK?

Also, in general, is there a convenient way to know or test if 
particular C callbacks are safe? Or is it just case by case?

**The context I was gonna bury the lede in but thought better of 
it**:

So, I am a big lame novice. Don't worry, I know this already.

I'm trying to use gtk-d, but I don't even know how to use gtk, so 
I'm learning them both at once. Probably not a good idea, I know.

 From what I can tell so far, it seems like the way gtk works is 
you write your whole application in terms of gtk's event loop. I 
really don't want to do that, so -- and I realize this is 
probably an even worse idea, but I don't know the right thing to 
do -- I decided I'd try writing a worker thread class -- "Rope" 
-- and run gtk's event loop in a Rope.

A Rope runs its given task in a new Fiber in a new thread, and 
that thread repeatedly resumes the fiber and terminates when the 
fiber terminates, but, in the meantime, takes some time to act as 
an executor each time the fiber suspends, accepting work in 
std.concurrency messages and running it on the same thread as its 
main fiber. My thought was that this way, I can call gtk-d's API 
from any thread, and still have all the calls happen on the same 
thread that the event loop is running on, as the API requires.

All I needed, then, was some way to make gtk-d suspend the fiber 
it's running on, to give its rope some time to service external 
work. My first thought was to use glib.Timeout.Timeout.add and 
put Fiber.yield in the timeout callback. The potential problem: 
the timeout callback, of course, has to be extern (C).

Now, I could just drop the more general concept of Rope and 
rewrite it specifically for gtk, and drop the whole Fiber part, 
and instead directly put the calls to 
std.concurrency.receiveTimeout in the glib.Timeout.Timeout.add 
callback. Assuming, of course, receiveTimeout would be any safer 
to call from extern (C) than Fiber.yield would. But that's just 
the trouble, I can't guess which would be safer, and even a 
minimal test for this question seems daunting.
Jan 24 2022
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/24/22 8:31 PM, Jaime wrote:

 **The lede**:
 
 Can I, for instance, safely call Fiber.yield in a C callback that I know 
 will be run in a Fiber?
 
 The stack will look like:
 Thread
 |- Fiber in D runtime
 | |- Call into a C API (stays on same thread)
 | | |- Extern (C) callback (stays on same thread)
 | | | |- Fiber.yield <-- Is this OK?
 
 Also, in general, is there a convenient way to know or test if 
 particular C callbacks are safe? Or is it just case by case?
 
 **The context I was gonna bury the lede in but thought better of it**:
 
 So, I am a big lame novice. Don't worry, I know this already.
 
 I'm trying to use gtk-d, but I don't even know how to use gtk, so I'm 
 learning them both at once. Probably not a good idea, I know.
 
  From what I can tell so far, it seems like the way gtk works is you 
 write your whole application in terms of gtk's event loop. I really 
 don't want to do that, so -- and I realize this is probably an even 
 worse idea, but I don't know the right thing to do -- I decided I'd try 
 writing a worker thread class -- "Rope" -- and run gtk's event loop in a 
 Rope.
 
 A Rope runs its given task in a new Fiber in a new thread, and that 
 thread repeatedly resumes the fiber and terminates when the fiber 
 terminates, but, in the meantime, takes some time to act as an executor 
 each time the fiber suspends, accepting work in std.concurrency messages 
 and running it on the same thread as its main fiber. My thought was that 
 this way, I can call gtk-d's API from any thread, and still have all the 
 calls happen on the same thread that the event loop is running on, as 
 the API requires.
 
 All I needed, then, was some way to make gtk-d suspend the fiber it's 
 running on, to give its rope some time to service external work. My 
 first thought was to use glib.Timeout.Timeout.add and put Fiber.yield in 
 the timeout callback. The potential problem: the timeout callback, of 
 course, has to be extern (C).
 
 Now, I could just drop the more general concept of Rope and rewrite it 
 specifically for gtk, and drop the whole Fiber part, and instead 
 directly put the calls to std.concurrency.receiveTimeout in the 
 glib.Timeout.Timeout.add callback. Assuming, of course, receiveTimeout 
 would be any safer to call from extern (C) than Fiber.yield would. But 
 that's just the trouble, I can't guess which would be safer, and even a 
 minimal test for this question seems daunting.


I would *imagine* it's fine, all the fiber context switch is doing (WRT 
the stack) is swapping out the fiber portion of the stack (i.e. back to 
the `Fiber.call` invokation), which should include the C stack portions 
as well.

There's a lot of low-level details in the Fiber.d file itself:

https://github.com/dlang/druntime/blob/e390ba7e0a1f80f15e72ca773fca7252057ba4c5/src/core/thread/fiber.d#L387

-Steve
Jan 24 2022
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 25, 2022 at 01:31:29AM +0000, Jaime via Digitalmars-d-learn wrote:

 **The lede**:
 
 Can I, for instance, safely call Fiber.yield in a C callback that I
 know will be run in a Fiber?
 
 The stack will look like:
 Thread
 |- Fiber in D runtime
 | |- Call into a C API (stays on same thread)
 | | |- Extern (C) callback (stays on same thread)
 | | | |- Fiber.yield <-- Is this OK?


I haven't tested this myself, but I *think* it should be OK.

One thing to watch out for, though, is the size of the Fiber's stack
(this can be specified when you first create the Fiber).  If the C part
of the code uses up too much stack space (e.g. if somewhere in the C
code it tries to allocate a large object on the stack), you may
inadvertently overflow the Fiber's stack and cause a crash or abort.
Increasing the stack size of the Fiber when it is created should fix
this problem.


T

-- 
Those who don't understand D are condemned to reinvent it, poorly. -- Daniel N
Jan 24 2022
parent Jaime <benjamin.i.mccann gmail.com> writes:
On Tuesday, 25 January 2022 at 01:41:03 UTC, Steven Schveighoffer 
wrote:

 On 1/24/22 8:31 PM, Jaime wrote:

 Can I, for instance, safely call Fiber.yield in a C callback 
 that I know will be run in a Fiber?


 I would *imagine* it's fine, all the fiber context switch is 
 doing (WRT the stack) is swapping out the fiber portion of the 
 stack (i.e. back to the `Fiber.call` invokation), which should 
 include the C stack portions as well.

 There's a lot of low-level details in the Fiber.d file itself:

 https://github.com/dlang/druntime/blob/e390ba7e0a1f80f15e72ca773fca7252057ba4c5/src/core/thread/fiber.d#L387

 -Steve


On Tuesday, 25 January 2022 at 02:02:27 UTC, H. S. Teoh wrote:

 I haven't tested this myself, but I *think* it should be OK.

 One thing to watch out for, though, is the size of the Fiber's 
 stack (this can be specified when you first create the Fiber).  
 If the C part of the code uses up too much stack space (e.g. if 
 somewhere in the C code it tries to allocate a large object on 
 the stack), you may inadvertently overflow the Fiber's stack 
 and cause a crash or abort. Increasing the stack size of the 
 Fiber when it is created should fix this problem.


 T


The insight and vote of confidence are much appreciated, you two. 
Steven, good call to check out the implementation of Fiber, I'll 
do that.
Jan 24 2022