www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - nothrow function callbacks in extern(C) code - solution

reply Walter Bright <newshound2 digitalmars.com> writes:
With nothrow and  nogc annotations, we've been motivated to add these 
annotations to C system API functions, because obviously such functions aren't 
going to throw D exceptions or call the D garbage collector.

But this exposed a problem - functions like C's qsort() take a pointer to a 
callback function. The callback function, being supplied by the D programmer, 
may throw and may call the garbage collector. By requiring the callback
function 
to be also nothrow  nogc, this is an unreasonable requirement besides breaking 
most existing D code that uses qsort().

This problem applies as well to the Windows APIs and the Posix APIs with
callbacks.

The solution is to use overloading so that if your callback is nothrow, it will 
call the nothrow version of qsort, if it is throwable, it calls the throwable 
version of qsort.

Never mind that those two versions of qsort are actually the same function (!), 
even though D's type system regards them as different. Although this looks like 
an usafe hack, it actually is quite safe, presuming that the rest of the qsort 
code itself does not throw. This technique relies on the fact that extern(C) 
functions do not get their types mangled into the names.

Some example code:

   extern (C)         { alias int function() fp_t; }
   extern (C) nothrow { alias int function() fpnothrow_t; }

   extern (C)         int foo(int a, fp_t fp);
   extern (C) nothrow int foo(int a, fpnothrow_t fp);

   extern (C)         int bar();
   extern (C) nothrow int barnothrow();

   void test() {
     foo(1, &bar);         // calls the 'throwing' foo()
     foo(1, &barnothrow);  // calls the 'nothrow' foo()
   }
Jun 19 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jun 19, 2014 at 12:59:00PM -0700, Walter Bright via Digitalmars-d wrote:
 With nothrow and  nogc annotations, we've been motivated to add these
 annotations to C system API functions, because obviously such
 functions aren't going to throw D exceptions or call the D garbage
 collector.
 
 But this exposed a problem - functions like C's qsort() take a pointer
 to a callback function. The callback function, being supplied by the D
 programmer, may throw and may call the garbage collector. By requiring
 the callback function to be also nothrow  nogc, this is an
 unreasonable requirement besides breaking most existing D code that
 uses qsort().
 
 This problem applies as well to the Windows APIs and the Posix APIs
 with callbacks.
 
 The solution is to use overloading so that if your callback is
 nothrow, it will call the nothrow version of qsort, if it is
 throwable, it calls the throwable version of qsort.
 
 Never mind that those two versions of qsort are actually the same
 function (!), even though D's type system regards them as different.
 Although this looks like an usafe hack, it actually is quite safe,
 presuming that the rest of the qsort code itself does not throw. This
 technique relies on the fact that extern(C) functions do not get their
 types mangled into the names.
 
 Some example code:
 
   extern (C)         { alias int function() fp_t; }
   extern (C) nothrow { alias int function() fpnothrow_t; }
 
   extern (C)         int foo(int a, fp_t fp);
   extern (C) nothrow int foo(int a, fpnothrow_t fp);
 
   extern (C)         int bar();
   extern (C) nothrow int barnothrow();
 
   void test() {
     foo(1, &bar);         // calls the 'throwing' foo()
     foo(1, &barnothrow);  // calls the 'nothrow' foo()
   }

This is a clever hack to work around the type system, but it introduces boilerplate, and doesn't ultimately fix the underlying problem. I agree that it's "good enough" for now -- to get those system APIs working without massive breakage of existing code -- but I think for the long run, we should face the root issue: functions that call callbacks have attributes that *depend* on an input argument. So we really should have the equivalent of "inout" for other attributes than const. For example, a function can be "dependently nothrow", meaning that the body of the function is nothrow, except for that call to user-supplied delegate: // This is hypothetical syntax. int dgCaller(scope int delegate(int) dg inout(nothrow)) inout(nothrow) { //throw new Exception(...); // illegal: body of function must not throw int result = dg(1); // OK: we inherit nothrow-ness from dg(). return result; } void f1() nothrow { // OK: the delegate is nothrow, so dgCaller is nothrow, // so it's permitted to call it from a nothrow function. auto x = dgCaller((x) => x+1); } void f2() { // OK: the delegate is throwing, which makes dgCaller // nothrowing w.r.t. this call. So this function becomes // throwing. auto x = dgCaller((int) { throw new Exception(...); }); } void f3() nothrow { // ILLEGAL: the delegate throws, so dgCaller may throw, // so we can't call it from a nothrow function. auto x = dgCaller((int) { throw new Exception(...); }); } This capability will solve a lot of attribute-related issues especially in generic code. For example, a container that implements opApply() currently cannot be marked pure, because it would impose purity on the delegate passed to it, which greatly limits its applicability. But accepting a non-pure delegate makes it uncallable from pure code, even if opApply() itself doesn't do anything impure. So we end up needing to write two identical versions of opApply: one pure, and one non-pure. And this has to be multiplied for each attribute we wish to support (pure, nothrow, safe, nogc -- that's 2^4 = 16 copies of the same function), which is clearly untenable. With inout(nothrow), inout(pure), etc., we can collapse all of these variants into a single function, and at the same time have the compiler statically verify that the result does not violate any attribute. T -- "I'm not childish; I'm just in touch with the child within!" - RL
Jun 19 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/19/2014 10:29 PM, Dicebot wrote:
 +1
 I have always wondered why `inout` is limited to const when problem is
 almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solve this kind of problem: Parametric (i.e. not query-able at either runtime or compile time inside the function) compile-time arguments do it better. I.e. instead of: inout(int)[] foo(inout(int)[] arg){ return arg; } do: T foo![T <: const(int)[]](T arg){ return arg; } this can be extended to other attributes, for example in the following way (this is just an example): void evaluate![transitive_attributes a](void delegate() a dg) a{ dg(); } {void foo() safe{} evaluate(&foo);} // evaluate is safe {void foo()pure{} evaluate(&foo);} // evaluate is pure {void foo()pure safe{} evaluate(&foo); // evaluate is pure safe evaluate![pure](&foo);} // argument explicitly specified, system pure
Jun 19 2014
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
22-Jun-2014 14:39, bearophile пишет:
 H. S. Teoh:

 Pretty soon, we need an attribute algebra to express these complicated
 relationships.

I think problems come from refusing to have formalized features in D. Having an attribute algebra is the lesser problem. Having formalized features causes problems, but no having them formalized is worse, as visible with D safety, D pureness, D uniqueness, that are a growing mess. D design needs more mathematicians and less piling of patches on patches.

+1 I often find myself having second thoughts about all of the 'implicit' rules esp. with regard to value-range propagation, and half-formalized such as inout. -- Dmitry Olshansky
Jun 22 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
+1
I have always wondered why `inout` is limited to const when 
problem is almost identical with all other restrictive attributes.
Jun 19 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Thursday, 19 June 2014 at 20:29:42 UTC, Dicebot wrote:
 +1
 I have always wondered why `inout` is limited to const when 
 problem is almost identical with all other restrictive 
 attributes.

I think the most common function this kind of thing could be useful for would be opApply functions. I haven't yet figured out a good way to make opApply implementations get all of the nice qualifiers without writing a bunch of overloads. Of course, in my own code I often just enforce the qualifiers to the exclusion of code without them, but that's no good for a standard library.
Jun 19 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jun 19, 2014 at 10:22:32PM +0000, w0rp via Digitalmars-d wrote:
 On Thursday, 19 June 2014 at 20:29:42 UTC, Dicebot wrote:
+1
I have always wondered why `inout` is limited to const when problem
is almost identical with all other restrictive attributes.

I think the most common function this kind of thing could be useful for would be opApply functions. I haven't yet figured out a good way to make opApply implementations get all of the nice qualifiers without writing a bunch of overloads. Of course, in my own code I often just enforce the qualifiers to the exclusion of code without them, but that's no good for a standard library.

Yes I mention opApply in my post. ;-) This isn't the only issue with opApply, though. The other issue is that you can't (easily) control the ref-ness of the loop index. And another the (lack of) inlining of loop bodies when opApply is involved. This makes them slightly less attractive in performance-sensitive situations, which is unfortunate. T -- It is impossible to make anything foolproof because fools are so ingenious. -- Sammy
Jun 19 2014
prev sibling next sibling parent "anonymous" <anonymous example.com> writes:
On Thursday, 19 June 2014 at 22:22:33 UTC, w0rp wrote:

 I haven't yet figured out a good way to make opApply 
 implementations get all of the nice qualifiers without writing 
 a bunch of overloads.

I don't know how practical this is, but since attributes are inferred for templated methods ... struct S { int opApply(Dg : int delegate(ref int))(Dg dg) { int e; return dg(e); } } void f1() pure nothrow nogc safe { S s; foreach(int x; s) {} } void f2() { S s; import std.stdio; foreach(int x; s) writeln(x); } void main() { f1(); f2(); }
Jun 19 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jun 20, 2014 at 12:36:53AM +0200, Timon Gehr via Digitalmars-d wrote:
 On 06/19/2014 10:29 PM, Dicebot wrote:
+1
I have always wondered why `inout` is limited to const when problem
is almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solve this kind of problem: Parametric (i.e. not query-able at either runtime or compile time inside the function) compile-time arguments do it better. I.e. instead of: inout(int)[] foo(inout(int)[] arg){ return arg; } do: T foo![T <: const(int)[]](T arg){ return arg; } this can be extended to other attributes, for example in the following way (this is just an example): void evaluate![transitive_attributes a](void delegate() a dg) a{ dg(); }

What if there are multiple delegate arguments? void evalTwo(void delegate() a dg1, void delegate() b dg2) // how to express union of a and b? { dg1(); dg2(); } What if the delegate arguments themselves take delegate arguments? void eval(void delegate(void delegate() a) b dg1, void delegate() c dg2) d // how to express that b depends on a, and // d depends on b and c? { dg1(dg2); } Pretty soon, we need an attribute algebra to express these complicated relationships. It would be nice to have a solution that can handle all of these cases without exploding complexity in the syntax. T -- "The number you have dialed is imaginary. Please rotate your phone 90 degrees and try again."
Jun 19 2014
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.06.2014 21:59, Walter Bright wrote:
 With nothrow and  nogc annotations, we've been motivated to add these
 annotations to C system API functions, because obviously such functions
 aren't going to throw D exceptions or call the D garbage collector.

 But this exposed a problem - functions like C's qsort() take a pointer
 to a callback function. The callback function, being supplied by the D
 programmer, may throw and may call the garbage collector. By requiring
 the callback function to be also nothrow  nogc, this is an unreasonable
 requirement besides breaking most existing D code that uses qsort().

 This problem applies as well to the Windows APIs and the Posix APIs with
 callbacks.

 The solution is to use overloading so that if your callback is nothrow,
 it will call the nothrow version of qsort, if it is throwable, it calls
 the throwable version of qsort.

 Never mind that those two versions of qsort are actually the same
 function (!), even though D's type system regards them as different.
 Although this looks like an usafe hack, it actually is quite safe,
 presuming that the rest of the qsort code itself does not throw. This
 technique relies on the fact that extern(C) functions do not get their
 types mangled into the names.

 Some example code:

    extern (C)         { alias int function() fp_t; }
    extern (C) nothrow { alias int function() fpnothrow_t; }

    extern (C)         int foo(int a, fp_t fp);
    extern (C) nothrow int foo(int a, fpnothrow_t fp);

    extern (C)         int bar();
    extern (C) nothrow int barnothrow();

    void test() {
      foo(1, &bar);         // calls the 'throwing' foo()
      foo(1, &barnothrow);  // calls the 'nothrow' foo()
    }

This only works for those functions that call the callback function directly. OS function do not always work this way. They register callbacks for later use like a windows procedure or a signal handler. This causes innocent looking functions to not behave as annotated because they internally use the callback functions. E.g. a lot of the Windows API functions might use message sending/dispatching internally, which might execute both throwing or GC allocating callbacks. These are currently not meeting the promise of their annotations. We either have to be more conservative with annotating OS functions or relax the guarantees of nothrow or nogc. Both alternatives are not very compelling.
Jun 20 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/14, 3:14 AM, Rainer Schuetze wrote:
 We either have to be more conservative with annotating OS functions or
 relax the guarantees of nothrow or  nogc. Both alternatives are not very
 compelling.

Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei
Jun 21 2014
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 22.06.2014 02:23, Andrei Alexandrescu wrote:
 On 6/20/14, 3:14 AM, Rainer Schuetze wrote:
 We either have to be more conservative with annotating OS functions or
 relax the guarantees of nothrow or  nogc. Both alternatives are not very
 compelling.

Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei

For nothrow that restriction makes sense, though there are functions in the Windows "C API" that actually may throw, e.g. HeapAlloc [1] and MmProbeAndLockPages [2]. No callbacks are involved in these examples, though. Under unix, signal handlers that throw exceptions seem to be used, though this might be non-standard. We even have one in druntime [3]. I'm ok with the current treatment of nothrow, but what can we do about nogc? [1] http://msdn.microsoft.com/en-us/library/windows/desktop/aa366597%28v=vs.85%29.aspx [2] http://msdn.microsoft.com/en-us/library/windows/hardware/ff554664%28v=vs.85%29.aspx [3] https://github.com/D-Programming-Language/druntime/blob/master/src/etc/linux/memoryerror.d BTW: The D exception handling is compatible with SEH for Win32, but not for Win64. You cannot handle exceptions by type across languages, though, only unwinding and "catch(Throwable)" works here.
Jun 22 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Sunday, 22 June 2014 at 00:23:06 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/14, 3:14 AM, Rainer Schuetze wrote:
 We either have to be more conservative with annotating OS 
 functions or
 relax the guarantees of nothrow or  nogc. Both alternatives 
 are not very
 compelling.

Callbacks passed into OS/clib functions are never supposed to throw so we must annotate them all with nothrow. C functions are never designed under the assumption callbacks may throw. -- Andrei

Yeah. I think the signature should communicate, "Woah, hold on there. That isn't going to work."
Jun 22 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 22 June 2014 at 08:43:20 UTC, Rainer Schuetze wrote:
 On 22.06.2014 02:23, Andrei Alexandrescu wrote:
 Callbacks passed into OS/clib functions are never supposed to 
 throw so
 we must annotate them all with nothrow. C functions are never 
 designed
 under the assumption callbacks may throw. -- Andrei

For nothrow that restriction makes sense, though there are functions in the Windows "C API" that actually may throw, e.g. HeapAlloc [1] and MmProbeAndLockPages [2]. No callbacks are involved in these examples, though. Under unix, signal handlers that throw exceptions seem to be used, though this might be non-standard. We even have one in druntime [3].

The SEGV handles throws Errors, not Exceptions, it's ok to be nothrow.
Jun 23 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Thursday, 19 June 2014 at 19:58:58 UTC, Walter Bright wrote:

<snip>

 The callback function, being supplied by the D programmer, may 
 throw and may call the garbage collector. By requiring the 
 callback function to be also nothrow  nogc, this is an 
 unreasonable requirement besides breaking most existing D code 
 that uses qsort().

<d.learn> I'm missing something, as I'm annotating all my C/API/etc callback function with nothrow: when the callback throws, what happens? I was thinking that this will mess-up the stack once the unwind will proceed... What's the use-case for having such a callback 'throwable'? Thanks! </d.learn> --- Paolo
Jun 20 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 20 June 2014 at 11:07:48 UTC, Paolo Invernizzi wrote:
 On Thursday, 19 June 2014 at 19:58:58 UTC, Walter Bright wrote:

 <snip>

 The callback function, being supplied by the D programmer, may 
 throw and may call the garbage collector. By requiring the 
 callback function to be also nothrow  nogc, this is an 
 unreasonable requirement besides breaking most existing D code 
 that uses qsort().

<d.learn> I'm missing something, as I'm annotating all my C/API/etc callback function with nothrow: when the callback throws, what happens? I was thinking that this will mess-up the stack once the unwind will proceed... What's the use-case for having such a callback 'throwable'? Thanks! </d.learn> --- Paolo

This is actually a really good point. How can a callback in C code expect to throw exceptions? Surely it should be nothrow anyway, because it's just not going to work otherwise. Maybe we should just strengthen the constraints for that, and make people update their code which isn't likely to work anyway. You can make any throwing function nothrow by catching Exceptions and throwing Errors instead at least. Ideally you wouldn't even throw Errors in C callbacks.
Jun 20 2014
prev sibling next sibling parent Artur Skawina via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 06/20/14 01:39, H. S. Teoh via Digitalmars-d wrote:
 On Fri, Jun 20, 2014 at 12:36:53AM +0200, Timon Gehr via Digitalmars-d wrote:
 On 06/19/2014 10:29 PM, Dicebot wrote:
 I have always wondered why `inout` is limited to const when problem
 is almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one `inout' wildcard in scope. This is not the best existing way to solve


 T foo![T <: const(int)[]](T arg){ return arg; }

 this can be extended to other attributes, for example in the following way
 (this is just an example):

 void evaluate![transitive_attributes a](void delegate() a dg) a{
     dg();
 }


 What if there are multiple delegate arguments?

 What if the delegate arguments themselves take delegate arguments?

 Pretty soon, we need an attribute algebra to express these complicated
 relationships.

Simple propagation, from just one source, would be enough for almost all cases - there's no need to over-complicate this. The remaining cases could be handled via introspection and ctfe; this way allows for more options, not just using some pre-defined algebra subset, which happens to be supported by a particular compiler (-version).
 It would be nice to have a solution that can handle all of these cases
 without exploding complexity in the syntax.

Actually supporting parametrized attributes, is something that I think everybody agrees on in principle (hence the lack of discussions when this topic is mentioned, every few weeks or so). The required semantics are pretty clear; what I still haven't seen is a good enough syntax proposal. One syntax that might have worked for built-in attributes could have been "const!A" etc, but I'm not sure if the parameter inference would be intuitive enough, and it would appear, at least superficially, to potentially clash with user defined attributes, especially once those become more powerful. artur
Jun 20 2014
prev sibling next sibling parent "Mason McGill" <mmcgill caltech.edu> writes:
On Thursday, 19 June 2014 at 23:41:25 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 Pretty soon, we need an attribute algebra to express these 
 complicated
 relationships.

 It would be nice to have a solution that can handle all of 
 these cases
 without exploding complexity in the syntax.

Here's one idea: Attributes can be thought of as "declarative" compile-time parameters--they don't directly define a computation, but they feel like compile-time parameters in that - They are variable in number. - They may take on many possible values. - They may be parameterized (like `extern`). - They play a role in overload resolution. - Their combinations may lead to combinatorial explosions that can hopefully be mitigated with metaprogramming. My point is, many of the problems we're encountering with attributes have already been solved (via templates, tuples, CTFE, `static if`, etc.) Attribute propagation would be simple (not trivial, but appropriately simple) for user-defined attributes, because they can be arbitrary expressions. If built-in attributes were just treated as pre-defined symbols, they could be manipulated with templates and CTFE: enum mayThrow(alias func) = staticIndexOf!(nothrow, functionAttributes!func) == -1; template throwsLike(alias func) { static if (mayThrow!func) enum throwsLike = tuple().expand; else enum throwsLike = nothrow; } void call(alias func)() throwsLike!func { func(); } (This thread may be relevant: http://forum.dlang.org/thread/hfmulninvghjntqkpguk forum.dlang.org)
Jun 22 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
H. S. Teoh:

 Pretty soon, we need an attribute algebra to express these 
 complicated relationships.

I think problems come from refusing to have formalized features in D. Having an attribute algebra is the lesser problem. Having formalized features causes problems, but no having them formalized is worse, as visible with D safety, D pureness, D uniqueness, that are a growing mess. D design needs more mathematicians and less piling of patches on patches. Bye, bearophile
Jun 22 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 19 Jun 2014 12:59:00 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 With nothrow and  nogc annotations, we've been motivated to add these 
 annotations to C system API functions, because obviously such functions aren't 
 going to throw D exceptions or call the D garbage collector.
 
 But this exposed a problem - functions like C's qsort() take a pointer to a 
 callback function. The callback function, being supplied by the D programmer, 
 may throw and may call the garbage collector. By requiring the callback
function 
 to be also nothrow  nogc, this is an unreasonable requirement besides breaking 
 most existing D code that uses qsort().
 
 This problem applies as well to the Windows APIs and the Posix APIs with
callbacks.

I just stumbled upon this thread now... In general you cannot throw exceptions unless you know how the call stack outside of your language's barrier is constructed. What I mean is that while DMD on 64-bit Linux uses frame pointers that druntime uses to unwind, GCC omits them. So druntime bails out once it reaches the C part of the call stack. That makes for two options: 1) D callbacks passed to other languages must not throw, like Andrei proposes if I understood that right. 2) Druntime must adapt to the system's C compiler's ABI. (by the use of libunwind) -- Marco
Jul 08 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 8 July 2014 at 17:16:27 UTC, Marco Leise wrote:
 2) Druntime must adapt to the system's C compiler's ABI.
    (by the use of libunwind)

Yes, longjmp() could happen in a callback and may be expected to unwind the stack (depending on the dev environment).
Jul 08 2014