www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Custom calling conventions

reply Manu <turkeyman gmail.com> writes:
So I was thinking about this extern(language) thing, the obvious ones are
supported, but it would be really nice to be able to implement custom
conventions for other languages/scripting languages.

For instance, I'm thinking about Android, I have JNI binding code
everywhere, it's really ugly.
I'd love to be able to declare:
  extern(Java) int someJavaFunc(int x, float y)

And then use my function like any regular function, with the 'extern(Java)'
bit handling the JNI business behind the scenes.

all be implemented the same way.

I'm imaging some mechanism to declare a calling convention (which would be
resolved within the extern(...) statement), and define it with a template,
something like:

callconv Java
{
  R call(T...)
  {
    // process tuple of args, make the call, return something?
  }

  R thisCall(Class, T...)
  {
    // also need a way to implementing methods... this might be enough.
  }
}

Some fancy code in there could conceivably call into any foreign language,
and this would be great!
Now when I: import java.jni;
I have the jni interface, but I also have access to extern(Java), and
that's awesome! :)

The main benefit over using a template, for
instance: jniCall!"functionName"(args...), would be the function name is
not a string, or require custom code construct (facilitating later
re-factoring or delegation to script without changing masses of existing
code, something I have done often), and if it was seen by the language as a
regular function call, you can mark it with all the usual stuff,
const/pure/etc, and apply the expected set of optimisations to the call.

I'm sure this has been discussed before... so go on, tear it apart :)
Feb 21 2012
next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
Why can't you do that with existing language features?

alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);

Where the templates generate the wrapper code/calling convention arg shuffle 
for each function.
Feb 21 2012
parent reply Manu <turkeyman gmail.com> writes:
On 21 February 2012 14:13, Daniel Murphy <yebblies nospamgmail.com> wrote:


 Why can't you do that with existing language features?

 alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
 mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);

 Where the templates generate the wrapper code/calling convention arg
 shuffle
 for each function.


You could, but that's really ugly. I did make the point in my OP that it
can be hacked right now, but I also listed the benefits of making it a
proper feature:
 1. It would be very neat and tidy. Clarity and readability is important
for code longevity.
 2. Ease of refactoring/delegation to scripts without refactoring all the
calling code throughout your app (this is real, I have done this on many
occasions)
 3. Declaring functions in the regular way, and annotating them with
various properties (const/pure/etc) allows all the usual type safety and
optimisation potentials available to the language.
     As far as the compiler is concerned, it is a real and normal function
call. All surrounding code gen/optimisations are still applicable, the
language will simply substitute the marshalling function at the final step
of the call.
 4-ish. It actually seems relatively simple to implement... and it's not a
breaking change. I can't really see any down sides.
Feb 21 2012
parent reply Paulo Pinto <pjmlp progtools.org> writes:
I think this fails in the same trap as extern "language" in C++.

Besides C and C++, I think the only variation I saw so far for the 
language part has been fortran, and cannot recall anylonger in what
compiler it was.

In C++'s case, since it is implementation defined, most compiler vendors
did not bother to support more than what the standard specifies (C and C++).

In you proposal, how to keep the feature portable across D implementations?

If you make it part of D language specification, which languages would 
be the chosen ones to be made available in any D compiler?

How to garantee correct interoperability (ABI) with other languages? 
After deciding the set of chosen languages, which compilers/runtimes 
would be the lucky ones?

--
Paulo

Am 21.02.2012 13:29, schrieb Manu:

 On 21 February 2012 14:13, Daniel Murphy <yebblies nospamgmail.com
 <mailto:yebblies nospamgmail.com>> wrote:

     Why can't you do that with existing language features?

     alias JniExternFunc!(void function(int)) someJNIFuncYouWantToCallFromD;
     mixin JniExportFunc!(&dFuncYouWantToCallUsingJNI);

     Where the templates generate the wrapper code/calling convention arg
     shuffle
     for each function.


 You could, but that's really ugly. I did make the point in my OP that it
 can be hacked right now, but I also listed the benefits of making it a
 proper feature:
   1. It would be very neat and tidy. Clarity and readability is
 important for code longevity.
   2. Ease of refactoring/delegation to scripts without refactoring all
 the calling code throughout your app (this is real, I have done this on
 many occasions)
   3. Declaring functions in the regular way, and annotating them with
 various properties (const/pure/etc) allows all the usual type safety and
 optimisation potentials available to the language.
       As far as the compiler is concerned, it is a real and normal
 function call. All surrounding code gen/optimisations are still
 applicable, the language will simply substitute the marshalling function
 at the final step of the call.
   4-ish. It actually seems relatively simple to implement... and it's
 not a breaking change. I can't really see any down sides.
Feb 21 2012
parent Manu <turkeyman gmail.com> writes:
On 21 February 2012 21:03, Paulo Pinto <pjmlp progtools.org> wrote:


 I think this fails in the same trap as extern "language" in C++.

 Besides C and C++, I think the only variation I saw so far for the
 language part has been fortran, and cannot recall anylonger in what
 compiler it was.

 In C++'s case, since it is implementation defined, most compiler vendors
 did not bother to support more than what the standard specifies (C and
 C++).

 In you proposal, how to keep the feature portable across D implementations?


Well since the languages I'm using as examples are VM based, the
interaction is through a standardised API, which is most likely part of the
same module that defines the custom extern type, both would be presented in
the same package.


If you make it part of D language specification, which languages would be

 the chosen ones to be made available in any D compiler?


I'm not making any language part of the D specification, I'm suggesting a
system to integrate custom calling code, which can be introduced in
libraries. No intrinsic support in D for any particular language.


How to garantee correct interoperability (ABI) with other languages? After

 deciding the set of chosen languages, which compilers/runtimes would be the
 lucky ones?


VM based languages have a strictly defined API, this is guaranteed, no
problem.
Extern to hard-linking languages requires the extra bits I detailed in a
prior post (name mangler, call receiver), and probably some version() mess
to support different compilers just like C/C++ does all over the place.
Many C compilers can't link against each others binaries, but with a mature
lib, you'd be able to support the calling convention of popular compilers
for your language of interest I would think.
t's probably quite do-able, but I'm not proposing to take the system that
far initially. Custom calling conventions to integrate with VM's seems more
useful, and much simpler to start with.
Feb 21 2012
prev sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2012-02-21 11:03:09 +0000, Manu <turkeyman gmail.com> said:


 
 So I was thinking about this extern(language) thing, the obvious ones are
 supported, but it would be really nice to be able to implement custom
 conventions for other languages/scripting languages.
 
 For instance, I'm thinking about Android, I have JNI binding code
 everywhere, it's really ugly.
 I'd love to be able to declare:
   extern(Java) int someJavaFunc(int x, float y)
 
 And then use my function like any regular function, with the 'extern(Java)'
 bit handling the JNI business behind the scenes.

 all be implemented the same way.
 
 I'm imaging some mechanism to declare a calling convention (which would be
 resolved within the extern(...) statement), and define it with a template,
 something like:
 
 callconv Java
 {
   R call(T...)
   {
     // process tuple of args, make the call, return something?
   }
 
   R thisCall(Class, T...)
   {
     // also need a way to implementing methods... this might be enough.
   }
 }
 
 Some fancy code in there could conceivably call into any foreign language,
 and this would be great!
 Now when I: import java.jni;
 I have the jni interface, but I also have access to extern(Java), and
 that's awesome! :)


In all currently existing cases, extern(Lang) offers not only a way to 
declare and call foreign-language functions, but also a way to define 
functions to be called from the foreign language. Basically, you have 
two-way compatibility. How would that work with Java?

 - - -

Also, I see you're thinking about methods, which means you're thinking 
about declaring Java classes. How is that supposed to work exactly?

I have some experience bridging Objective-C and D. I once built a 
complete wrapper system for Objective-C objects, each object was 
wrapped by a D one. It worked very well, but it generated so much bloat 
that it became unusable as soon as I started defining enough classes 
for it to be useful. See the D/Objective-C bridge: 
<http://michelf.com/projects/d-objc-bridge/>.

Maybe you'll have more luck, but for me wrappers just couldn't work. So 
I decided making the compiler understand the Objective-C runtime and 
object model was a much better solution. It was fun to do and wasn't 
that hard actually. Here's the result: 
<http://michelf.com/projects/d-objc/>. Unfortunately I don't have much 
time to work on it anymore.

More background:
<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Feb 21 2012
parent reply Manu <turkeyman gmail.com> writes:
On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com> wrote:


 On 2012-02-21 11:03:09 +0000, Manu <turkeyman gmail.com> said:



 So I was thinking about this extern(language) thing, the obvious ones are
 supported, but it would be really nice to be able to implement custom
 conventions for other languages/scripting languages.

 For instance, I'm thinking about Android, I have JNI binding code
 everywhere, it's really ugly.
 I'd love to be able to declare:
  extern(Java) int someJavaFunc(int x, float y)

 And then use my function like any regular function, with the
 'extern(Java)'
 bit handling the JNI business behind the scenes.

 all be implemented the same way.

 I'm imaging some mechanism to declare a calling convention (which would be
 resolved within the extern(...) statement), and define it with a template,
 something like:

 callconv Java
 {
  R call(T...)
  {
    // process tuple of args, make the call, return something?
  }

  R thisCall(Class, T...)
  {
    // also need a way to implementing methods... this might be enough.
  }
 }

 Some fancy code in there could conceivably call into any foreign language,
 and this would be great!
 Now when I: import java.jni;
 I have the jni interface, but I also have access to extern(Java), and
 that's awesome! :)


 In all currently existing cases, extern(Lang) offers not only a way to
 declare and call foreign-language functions, but also a way to define
 functions to be called from the foreign language. Basically, you have
 two-way compatibility. How would that work with Java?


Interesting point.
Java (or any VM based language) can't hard link to the binary, so it never
could try and call back in through an extern-ed function directly, it must
be explicitly supplied the pointer in the first place. In that case, the
API that supplies the callback to Java can produce the appropriate incoming
call wrapper.

I can imagine a way to doing a hard-extern under my proposal if you were
hard linking against an unsupported language language... you might need to
provide a name mangler in the definition block I describe to match the
foreign language, and an incoming call template would also need to be
defined as a naked function with the responsibility of managing the
args/stack appropriately, and calling the externed function as a regular
D-call (this would probably only be some light register/stack management, I
don't think it would bloat a lot). This would allow D code to also call the
externed function, since the physical definition is still a D-call, you
just call it directly internally.

Anyway, while I can imagine a solution, I don't think it would be needed
immediately. It'll just open a further can of worms like foreign language
struct layout as people try to use it in complex scenarios ;)

I don't think this reduces the usefulness of my proposal at all though, ie.
being able to apply it to outgoing calls for any foreign language.

- - -

 Also, I see you're thinking about methods, which means you're thinking
 about declaring Java classes. How is that supposed to work exactly?


Well I was thinking about that with respect to calling functions on classes
that I have imported from a foreign language, not declared locally for
export to a foreign language.
I clearly didn't think that through though, I'll try and clarify my
thoughts to that end... I get the feeling that idea was a dead-end though.



 I have some experience bridging Objective-C and D. I once built a complete
 wrapper system for Objective-C objects, each object was wrapped by a D one.
 It worked very well, but it generated so much bloat that it became unusable
 as soon as I started defining enough classes for it to be useful. See the
 D/Objective-C bridge:
<http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>

.




What was the primary cause of the bloat? I can't imagine my proposal
causing any more bloat than the explicit jni call (or equivalent) woudl
have otherwise.



 Maybe you'll have more luck, but for me wrappers just couldn't work. So I
 decided making the compiler understand the Objective-C runtime and object
 model was a much better solution. It was fun to do and wasn't that hard
 actually. Here's the result:
<http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.
 Unfortunately I don't have much time to work on it anymore.


That sounds appropriate for Obj-C, but I don't think it applies to VM based
languages however you slice it, since there's not actually a direct binary
linkage. There's no way to know about the Java/.NET/Lua/etc object model
internally, the VM owns that entirely, so you're forced to marshal through
API's. That's okay though, you have to do it anyway. I'm proposing a way to
unify the concept of calling functions without obscuring the actual call
with junk, and still allowing the D language to annotate and optimise to
its fullest extent.

More background:

 <http://michelf.com/weblog/**2010/dobjc-dead-end-start-**anew/<http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>



 --
 Michel Fortin
 michel.fortin michelf.com
 http://michelf.com/





 Feb 21 2012



 next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com>  writes:


On 21-02-2012 18:03, Manu wrote:

 On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com
 <mailto:michel.fortin michelf.com>> wrote:

     On 2012-02-21 11:03:09 +0000, Manu <turkeyman gmail.com
     <mailto:turkeyman gmail.com>> said:


         So I was thinking about this extern(language) thing, the obvious
         ones are
         supported, but it would be really nice to be able to implement
         custom
         conventions for other languages/scripting languages.

         For instance, I'm thinking about Android, I have JNI binding code
         everywhere, it's really ugly.
         I'd love to be able to declare:
           extern(Java) int someJavaFunc(int x, float y)

         And then use my function like any regular function, with the
         'extern(Java)'
         bit handling the JNI business behind the scenes.

         these could
         all be implemented the same way.

         I'm imaging some mechanism to declare a calling convention
         (which would be
         resolved within the extern(...) statement), and define it with a
         template,
         something like:

         callconv Java
         {
           R call(T...)
           {
             // process tuple of args, make the call, return something?
           }

           R thisCall(Class, T...)
           {
             // also need a way to implementing methods... this might be
         enough.
           }
         }

         Some fancy code in there could conceivably call into any foreign
         language,
         and this would be great!
         Now when I: import java.jni;
         I have the jni interface, but I also have access to
         extern(Java), and
         that's awesome! :)


     In all currently existing cases, extern(Lang) offers not only a way
     to declare and call foreign-language functions, but also a way to
     define functions to be called from the foreign language. Basically,
     you have two-way compatibility. How would that work with Java?


 Interesting point.
 Java (or any VM based language) can't hard link to the binary, so it
 never could try and call back in through an extern-ed function directly,


Sure it could. Why wouldn't it be able to? This is how extern + 



 it must be explicitly supplied the pointer in the first place. In that
 case, the API that supplies the callback to Java can produce the
 appropriate incoming call wrapper.

 I can imagine a way to doing a hard-extern under my proposal if you were
 hard linking against an unsupported language language... you might need
 to provide a name mangler in the definition block I describe to match
 the foreign language, and an incoming call template would also need to
 be defined as a naked function with the responsibility of managing the
 args/stack appropriately, and calling the externed function as a regular
 D-call (this would probably only be some light register/stack
 management, I don't think it would bloat a lot). This would allow D code
 to also call the externed function, since the physical definition is
 still a D-call, you just call it directly internally.

 Anyway, while I can imagine a solution, I don't think it would be needed
 immediately. It'll just open a further can of worms like foreign
 language struct layout as people try to use it in complex scenarios ;)

 I don't think this reduces the usefulness of my proposal at all though,
 ie. being able to apply it to outgoing calls for any foreign language.

     - - -

     Also, I see you're thinking about methods, which means you're
     thinking about declaring Java classes. How is that supposed to work
     exactly?


 Well I was thinking about that with respect to calling functions on
 classes that I have imported from a foreign language, not declared
 locally for export to a foreign language.
 I clearly didn't think that through though, I'll try and clarify my
 thoughts to that end... I get the feeling that idea was a dead-end though.

     I have some experience bridging Objective-C and D. I once built a
     complete wrapper system for Objective-C objects, each object was
     wrapped by a D one. It worked very well, but it generated so much
     bloat that it became unusable as soon as I started defining enough
     classes for it to be useful. See the D/Objective-C bridge:
     <http://michelf.com/projects/__d-objc-bridge/
     <http://michelf.com/projects/d-objc-bridge/>>.


 What was the primary cause of the bloat? I can't imagine my proposal
 causing any more bloat than the explicit jni call (or equivalent) woudl
 have otherwise.

     Maybe you'll have more luck, but for me wrappers just couldn't
     work.. So I decided making the compiler understand the Objective-C
     runtime and object model was a much better solution. It was fun to
     do and wasn't that hard actually. Here's the result:
     <http://michelf.com/projects/__d-objc/
     <http://michelf.com/projects/d-objc/>>. Unfortunately I don't have
     much time to work on it anymore.


 That sounds appropriate for Obj-C, but I don't think it applies to VM
 based languages however you slice it, since there's not actually a
 direct binary linkage. There's no way to know about the
 Java/.NET/Lua/etc object model internally, the VM owns that entirely, so
 you're forced to marshal through API's. That's okay though, you have to
 do it anyway. I'm proposing a way to unify the concept of calling
 functions without obscuring the actual call with junk, and still
 allowing the D language to annotate and optimise to its fullest extent.

     More background:
     <http://michelf.com/weblog/__2010/dobjc-dead-end-start-__anew/
     <http://michelf.com/weblog/2010/dobjc-dead-end-start-anew/>>


     --
     Michel Fortin
     michel.fortin michelf.com <mailto:michel.fortin michelf.com>
     http://michelf.com/


-- 
- Alex



 Feb 21 2012



  parent reply Manu <turkeyman gmail.com>  writes:


On 21 February 2012 19:10, Alex R=C3=B8nne Petersen <xtzgzorex gmail.com> w=
rote:


 On 21-02-2012 18:03, Manu wrote:


 On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com
 <mailto:michel.fortin michelf.**com <michel.fortin michelf.com>>> wrote:

    On 2012-02-21 11:03:09 +0000, Manu <turkeyman gmail.com
    <mailto:turkeyman gmail.com>> said:

 Interesting point.
 Java (or any VM based language) can't hard link to the binary, so it
 never could try and call back in through an extern-ed function directly,


 Sure it could. Why wouldn't it be able to? This is how extern + DllImport




calling convention, so you just need to export an extern(C) in that case,
like any regular DLL.



 Feb 21 2012



  parent  =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com>  writes:


On 21-02-2012 20:43, Manu wrote:

 On 21 February 2012 19:10, Alex Rønne Petersen <xtzgzorex gmail.com
 <mailto:xtzgzorex gmail.com>> wrote:

     On 21-02-2012 18:03, Manu wrote:

         On 21 February 2012 16:59, Michel Fortin
         <michel.fortin michelf.com <mailto:michel.fortin michelf.com>
         <mailto:michel.fortin michelf.__com
         <mailto:michel.fortin michelf.com>>> wrote:

             On 2012-02-21 11:03:09 +0000, Manu <turkeyman gmail.com
         <mailto:turkeyman gmail.com>
         <mailto:turkeyman gmail.com <mailto:turkeyman gmail.com>>> said:

         Interesting point.
         Java (or any VM based language) can't hard link to the binary, so it
         never could try and call back in through an extern-ed function
         directly,


     Sure it could. Why wouldn't it be able to? This is how extern +




 calling convention, so you just need to export an extern(C) in that
 case, like any regular DLL.


Surely Java 'native' methods require the native functions to conform to 
some calling convention?

-- 
- Alex



 Feb 21 2012



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-02-21 18:03, Manu wrote:

 On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com
     I have some experience bridging Objective-C and D. I once built a
     complete wrapper system for Objective-C objects, each object was
     wrapped by a D one. It worked very well, but it generated so much
     bloat that it became unusable as soon as I started defining enough
     classes for it to be useful. See the D/Objective-C bridge:
     <http://michelf.com/projects/__d-objc-bridge/
     <http://michelf.com/projects/d-objc-bridge/>>.


 What was the primary cause of the bloat? I can't imagine my proposal
 causing any more bloat than the explicit jni call (or equivalent) woudl
 have otherwise.


Template bloat. Every call bridging D/Objective-C is made throw a series 
of templates. This is for making it possible (less verbose) to create 
bindings.

It might be possible to decrease the template bloat by having a tool 
that automatically generates the bindings and outputs what the templates 
do inline.

-- 
/Jacob Carlborg



 Feb 21 2012



  parent reply Manu <turkeyman gmail.com>  writes:


On 21 February 2012 20:12, Jacob Carlborg <doob me.com> wrote:


 On 2012-02-21 18:03, Manu wrote:


 On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com
    I have some experience bridging Objective-C and D. I once built a
    complete wrapper system for Objective-C objects, each object was
    wrapped by a D one. It worked very well, but it generated so much
    bloat that it became unusable as soon as I started defining enough
    classes for it to be useful. See the D/Objective-C bridge:
    <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
    <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>

.





 What was the primary cause of the bloat? I can't imagine my proposal
 causing any more bloat than the explicit jni call (or equivalent) woudl
 have otherwise.


 Template bloat. Every call bridging D/Objective-C is made throw a series
 of templates. This is for making it possible (less verbose) to create
 bindings.

 It might be possible to decrease the template bloat by having a tool that
 automatically generates the bindings and outputs what the templates do
 inline.



Why aren't the templates inline themselves? Although if the templates do a
lot of work, wouldn't that INCREASE the code volume?
I can't really imagine how Obj-C linkage could bloat so much, what was
involved? What did you have to do in addition to what a regular Obj-C
function call would have done?



 Feb 21 2012



 next sibling parent  Michel Fortin <michel.fortin michelf.com>  writes:


On 2012-02-21 19:45:37 +0000, Manu <turkeyman gmail.com> said:


 On 21 February 2012 20:12, Jacob Carlborg <doob me.com> wrote:
 

 On 2012-02-21 18:03, Manu wrote:
 

 On 21 February 2012 16:59, Michel Fortin <michel.fortin michelf.com
 What was the primary cause of the bloat? I can't imagine my proposal
 causing any more bloat than the explicit jni call (or equivalent) woudl
 have otherwise.


 
 Template bloat. Every call bridging D/Objective-C is made throw a series
 of templates. This is for making it possible (less verbose) to create
 bindings.
 
 It might be possible to decrease the template bloat by having a tool that
 automatically generates the bindings and outputs what the templates do
 inline.


 
 Why aren't the templates inline themselves? Although if the templates do a
 lot of work, wouldn't that INCREASE the code volume?
 I can't really imagine how Obj-C linkage could bloat so much, what was
 involved? What did you have to do in addition to what a regular Obj-C
 function call would have done?


I answered in my other post. In short: virtual functions you can 
override to override Objective-C methods.


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/



 Feb 21 2012



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-02-21 20:45, Manu wrote:

 On 21 February 2012 20:12, Jacob Carlborg <doob me.com
 <mailto:doob me.com>> wrote:

     On 2012-02-21 18:03, Manu wrote:

         On 21 February 2012 16:59, Michel Fortin
         <michel.fortin michelf.com <mailto:michel.fortin michelf.com>
             I have some experience bridging Objective-C and D. I once
         built a
             complete wrapper system for Objective-C objects, each object was
             wrapped by a D one. It worked very well, but it generated so
         much
             bloat that it became unusable as soon as I started defining
         enough
             classes for it to be useful. See the D/Objective-C bridge:
         <http://michelf.com/projects/____d-objc-bridge/
         <http://michelf.com/projects/__d-objc-bridge/>
         <http://michelf.com/projects/__d-objc-bridge/
         <http://michelf.com/projects/d-objc-bridge/>>>.


         What was the primary cause of the bloat? I can't imagine my proposal
         causing any more bloat than the explicit jni call (or
         equivalent) woudl
         have otherwise.


     Template bloat. Every call bridging D/Objective-C is made throw a
     series of templates. This is for making it possible (less verbose)
     to create bindings.

     It might be possible to decrease the template bloat by having a tool
     that automatically generates the bindings and outputs what the
     templates do inline.


 Why aren't the templates inline themselves? Although if the templates do
 a lot of work, wouldn't that INCREASE the code volume?
 I can't really imagine how Obj-C linkage could bloat so much, what was
 involved? What did you have to do in addition to what a regular Obj-C
 function call would have done?


Michel Fortin explained this better in two of his answers. You can also 
read the documentation of my implementation, very similar to Michel's:

http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals

But to give a quick example:

class Foo : NSObject
{		
     Foo foo ()
     {
         return invokeObjcSelf!(Foo, "foo");
     }

     Foo bar ()
     {
         return invokeObjcSelf!(Foo, "bar");
     }
}

"invokeObjcSelf" is a template function that calls an Objective-C 
method. Basically each time "invokeObjcSelf" is called a new 
instantiation of the template is created and that is put in the symbol 
table. "invokeObjcSelf" then calls several more template functions 
making the template bloat increase exponentially.

-- 
/Jacob Carlborg



 Feb 21 2012



  parent reply Manu <turkeyman gmail.com>  writes:


On 21 February 2012 22:35, Jacob Carlborg <doob me.com> wrote:


 On 2012-02-21 20:45, Manu wrote:


 On 21 February 2012 20:12, Jacob Carlborg <doob me.com
 <mailto:doob me.com>> wrote:

    On 2012-02-21 18:03, Manu wrote:

        On 21 February 2012 16:59, Michel Fortin
        <michel.fortin michelf.com
<mailto:michel.fortin michelf.**com<michel.fortin michelf.com>


            I have some experience bridging Objective-C and D. I once
        built a
            complete wrapper system for Objective-C objects, each object
 was
            wrapped by a D one. It worked very well, but it generated so
        much
            bloat that it became unusable as soon as I started defining
        enough
            classes for it to be useful. See the D/Objective-C bridge:
        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>


        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
        <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>

.







        What was the primary cause of the bloat? I can't imagine my
 proposal
        causing any more bloat than the explicit jni call (or
        equivalent) woudl
        have otherwise.


    Template bloat. Every call bridging D/Objective-C is made throw a
    series of templates. This is for making it possible (less verbose)
    to create bindings.

    It might be possible to decrease the template bloat by having a tool
    that automatically generates the bindings and outputs what the
    templates do inline.


 Why aren't the templates inline themselves? Although if the templates do
 a lot of work, wouldn't that INCREASE the code volume?
 I can't really imagine how Obj-C linkage could bloat so much, what was
 involved? What did you have to do in addition to what a regular Obj-C
 function call would have done?


 Michel Fortin explained this better in two of his answers. You can also
 read the documentation of my implementation, very similar to Michel's:

 http://www.dsource.org/**projects/dstep/wiki/**ObjcBridge/BridgeInternals<http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>

 But to give a quick example:

 class Foo : NSObject
 {
    Foo foo ()
    {
        return invokeObjcSelf!(Foo, "foo");
    }

    Foo bar ()
    {
        return invokeObjcSelf!(Foo, "bar");
    }
 }

 "invokeObjcSelf" is a template function that calls an Objective-C method.
 Basically each time "invokeObjcSelf" is called a new instantiation of the
 template is created and that is put in the symbol table. "invokeObjcSelf"
 then calls several more template functions making the template bloat
 increase exponentially.



But they should all be inlined, and the symbol table should be stripped,
which shouldn't leave anything in the end other than the inlined function
calling code, and in my examples, this will be basically the exact same
code that you'd have to write anyway to call through some vm's API...



 Feb 21 2012



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-02-21 22:01, Manu wrote:

 On 21 February 2012 22:35, Jacob Carlborg <doob me.com
 <mailto:doob me.com>> wrote:

     On 2012-02-21 20:45, Manu wrote:

         On 21 February 2012 20:12, Jacob Carlborg <doob me.com
         <mailto:doob me.com>
         <mailto:doob me.com <mailto:doob me.com>>> wrote:

             On 2012-02-21 18:03, Manu wrote:

                 On 21 February 2012 16:59, Michel Fortin
         <michel.fortin michelf.com <mailto:michel.fortin michelf.com>
         <mailto:michel.fortin michelf.__com
         <mailto:michel.fortin michelf.com>>

                     I have some experience bridging Objective-C and D. I
         once
                 built a
                     complete wrapper system for Objective-C objects,
         each object was
                     wrapped by a D one. It worked very well, but it
         generated so
                 much
                     bloat that it became unusable as soon as I started
         defining
                 enough
                     classes for it to be useful. See the D/Objective-C
         bridge:
         <http://michelf.com/projects/______d-objc-bridge/
         <http://michelf.com/projects/____d-objc-bridge/>
         <http://michelf.com/projects/____d-objc-bridge/
         <http://michelf.com/projects/__d-objc-bridge/>>

         <http://michelf.com/projects/____d-objc-bridge/
         <http://michelf.com/projects/__d-objc-bridge/>
         <http://michelf.com/projects/__d-objc-bridge/
         <http://michelf.com/projects/d-objc-bridge/>>>>.


                 What was the primary cause of the bloat? I can't imagine
         my proposal
                 causing any more bloat than the explicit jni call (or
                 equivalent) woudl
                 have otherwise.


             Template bloat. Every call bridging D/Objective-C is made
         throw a
             series of templates. This is for making it possible (less
         verbose)
             to create bindings.

             It might be possible to decrease the template bloat by
         having a tool
             that automatically generates the bindings and outputs what the
             templates do inline.


         Why aren't the templates inline themselves? Although if the
         templates do
         a lot of work, wouldn't that INCREASE the code volume?
         I can't really imagine how Obj-C linkage could bloat so much,
         what was
         involved? What did you have to do in addition to what a regular
         Obj-C
         function call would have done?


     Michel Fortin explained this better in two of his answers. You can
     also read the documentation of my implementation, very similar to
     Michel's:

     http://www.dsource.org/__projects/dstep/wiki/__ObjcBridge/BridgeInternals
     <http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>

     But to give a quick example:

     class Foo : NSObject
     {
         Foo foo ()
         {
             return invokeObjcSelf!(Foo, "foo");
         }

         Foo bar ()
         {
             return invokeObjcSelf!(Foo, "bar");
         }
     }

     "invokeObjcSelf" is a template function that calls an Objective-C
     method. Basically each time "invokeObjcSelf" is called a new
     instantiation of the template is created and that is put in the
     symbol table. "invokeObjcSelf" then calls several more template
     functions making the template bloat increase exponentially.


 But they should all be inlined, and the symbol table should be stripped,
 which shouldn't leave anything in the end other than the inlined
 function calling code, and in my examples, this will be basically the
 exact same code that you'd have to write anyway to call through some
 vm's API...


Well, that's not what happen with templates.

-- 
/Jacob Carlborg



 Feb 21 2012



  parent reply Manu <turkeyman gmail.com>  writes:


On 21 February 2012 23:06, Jacob Carlborg <doob me.com> wrote:


 On 2012-02-21 22:01, Manu wrote:


 On 21 February 2012 22:35, Jacob Carlborg <doob me.com

 <mailto:doob me.com>> wrote:

    On 2012-02-21 20:45, Manu wrote:

        On 21 February 2012 20:12, Jacob Carlborg <doob me.com
        <mailto:doob me.com>
        <mailto:doob me.com <mailto:doob me.com>>> wrote:

            On 2012-02-21 18:03, Manu wrote:

                On 21 February 2012 16:59, Michel Fortin
        <michel.fortin michelf.com
<mailto:michel.fortin michelf.**com<michel.fortin michelf.com>


        <mailto:michel.fortin michelf.**__com

        <mailto:michel.fortin michelf.**com <michel.fortin michelf.com>>>

                    I have some experience bridging Objective-C and D. I
        once
                built a
                    complete wrapper system for Objective-C objects,
        each object was
                    wrapped by a D one. It worked very well, but it
        generated so
                much
                    bloat that it became unusable as soon as I started
        defining
                enough
                    classes for it to be useful. See the D/Objective-C
        bridge:
        <http://michelf.com/projects/_**_____d-objc-bridge/<http://michelf.com/projects/______d-objc-bridge/>
        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>


        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>




        <http://michelf.com/projects/_**___d-objc-bridge/<http://michelf.com/projects/____d-objc-bridge/>
        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>


        <http://michelf.com/projects/_**_d-objc-bridge/<http://michelf.com/projects/__d-objc-bridge/>
        <http://michelf.com/projects/**d-objc-bridge/<http://michelf.com/projects/d-objc-bridge/>

.









                What was the primary cause of the bloat? I can't imagine
        my proposal
                causing any more bloat than the explicit jni call (or
                equivalent) woudl
                have otherwise.


            Template bloat. Every call bridging D/Objective-C is made
        throw a
            series of templates. This is for making it possible (less
        verbose)
            to create bindings.

            It might be possible to decrease the template bloat by
        having a tool
            that automatically generates the bindings and outputs what the
            templates do inline.


        Why aren't the templates inline themselves? Although if the
        templates do
        a lot of work, wouldn't that INCREASE the code volume?
        I can't really imagine how Obj-C linkage could bloat so much,
        what was
        involved? What did you have to do in addition to what a regular
        Obj-C
        function call would have done?


    Michel Fortin explained this better in two of his answers. You can
    also read the documentation of my implementation, very similar to
    Michel's:

    http://www.dsource.org/__**projects/dstep/wiki/__**
 ObjcBridge/BridgeInternals<http://www.dsource.org/__projects/dstep/wiki/__ObjcBridge/BridgeInternals>

    <http://www.dsource.org/**projects/dstep/wiki/**
 ObjcBridge/BridgeInternals<http://www.dsource.org/projects/dstep/wiki/ObjcBridge/BridgeInternals>


    But to give a quick example:

    class Foo : NSObject
    {
        Foo foo ()
        {
            return invokeObjcSelf!(Foo, "foo");
        }

        Foo bar ()
        {
            return invokeObjcSelf!(Foo, "bar");
        }
    }

    "invokeObjcSelf" is a template function that calls an Objective-C
    method. Basically each time "invokeObjcSelf" is called a new
    instantiation of the template is created and that is put in the
    symbol table. "invokeObjcSelf" then calls several more template
    functions making the template bloat increase exponentially.


 But they should all be inlined, and the symbol table should be stripped,
 which shouldn't leave anything in the end other than the inlined
 function calling code, and in my examples, this will be basically the
 exact same code that you'd have to write anyway to call through some
 vm's API...


 Well, that's not what happen with templates.



... really? why?
You've just made me very very scared for my simd module :/

I've had a serious concern about D's lack of a force-inline for quite a
while...



 Feb 21 2012



 next sibling parent reply Michel Fortin <michel.fortin michelf.com>  writes:


On 2012-02-21 23:13:31 +0000, Manu <turkeyman gmail.com> said:


 
 On 21 February 2012 23:06, Jacob Carlborg <doob me.com> wrote:
 

 On 2012-02-21 22:01, Manu wrote:
 

 On 21 February 2012 22:35, Jacob Carlborg <doob me.com

 But to give a quick example:
 
 class Foo : NSObject
 {
 Foo foo ()
 {
 return invokeObjcSelf!(Foo, "foo");
 }
 
 Foo bar ()
 {
 return invokeObjcSelf!(Foo, "bar");
 }
 }
 
 "invokeObjcSelf" is a template function that calls an Objective-C
 method. Basically each time "invokeObjcSelf" is called a new
 instantiation of the template is created and that is put in the
 symbol table. "invokeObjcSelf" then calls several more template
 functions making the template bloat increase exponentially.


 
 But they should all be inlined, and the symbol table should be stripped,
 which shouldn't leave anything in the end other than the inlined
 function calling code, and in my examples, this will be basically the
 exact same code that you'd have to write anyway to call through some
 vm's API...


 
 Well, that's not what happen with templates.


 
 ... really? why?
 You've just made me very very scared for my simd module :/
 
 I've had a serious concern about D's lack of a force-inline for quite a
 while...


I'm not sure what Jacob is referring to.

But here's what I can tell: Foo's methods can't be stripped because 
they're referenced by the virtual table. invokeObjcSelf could be 
inlined in theory, but it probably isn't because it is not that small 
considering it actually does exception bridging and argument/return 
value conversions as needed. Also, for each method there's also a 
trampoline function generated (by a mixin which is not in his example) 
which is referenced by the corresponding Objective-C reverse wrapper 
class (which is added dynamically on demand), so the Objective-C side 
can call the D method if you subclass the D wrapper and override one of 
its methods.

The problem is that if you write bindings for a significant part of 
Cocoa, you end up with thousands of those functions, and none can be 
stripped because they're all referenced somehow because they're needed 
for dynamic dispatch. Sure, you could inline what is inside the virtual 
function (invokeObjcSelf), but beside saving a few function prologues I 
don't expect it'll have a very noticeable effect. Even if you saved 25% 
of the symbols it won't bring things to a reasonable size or a 
reasonable compilation time.


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/



 Feb 21 2012



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-02-22 05:38, Michel Fortin wrote:

 On 2012-02-21 23:13:31 +0000, Manu <turkeyman gmail.com> said:


 On 21 February 2012 23:06, Jacob Carlborg <doob me.com> wrote:


 On 2012-02-21 22:01, Manu wrote:


 On 21 February 2012 22:35, Jacob Carlborg <doob me.com

 But to give a quick example:

 class Foo : NSObject
 {
 Foo foo ()
 {
 return invokeObjcSelf!(Foo, "foo");
 }

 Foo bar ()
 {
 return invokeObjcSelf!(Foo, "bar");
 }
 }

 "invokeObjcSelf" is a template function that calls an Objective-C
 method. Basically each time "invokeObjcSelf" is called a new
 instantiation of the template is created and that is put in the
 symbol table. "invokeObjcSelf" then calls several more template
 functions making the template bloat increase exponentially.


 But they should all be inlined, and the symbol table should be
 stripped,
 which shouldn't leave anything in the end other than the inlined
 function calling code, and in my examples, this will be basically the
 exact same code that you'd have to write anyway to call through some
 vm's API...


 Well, that's not what happen with templates.


 ... really? why?
 You've just made me very very scared for my simd module :/

 I've had a serious concern about D's lack of a force-inline for quite a
 while...


 I'm not sure what Jacob is referring to.


I referring to that every instantiation of a template function is put in 
the symbol table. If this is an implementation issue with DMD not being 
able to properly optimize this I don't know. I thought that this was one 
of the major reason for the executable size when using the Objective-C/D 
bridge. For example:

void foo () {}
void main () {}

$ dmd -inline -O -release main.d
$ nm main | grep foo

00000001000012b4 T _D4main3fooFZv

Outputs one symbol as expected

---------------------------------------

void foo (T) () {}
void main () {}

$ dmd -inline -O -release main.d
$ nm main | grep foo

Doesn't output anything since the template functions is not 
instantiated, it won't even be fully compiled (only lexed and perhaps 
parsed).

--------------------------------------------------------------

void foo (T) () {}
void main ()
{
     foo!(int);
     foo!(char);
}

$ dmd -inline -O -release main.d
$ nm main | grep foo

00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
00000001000012b0 T _D4main10__T3fooTiZ3fooFZv

Outputs two symbols as expected, one for each instantiation.

-- 
/Jacob Carlborg



 Feb 21 2012



  parent reply Michel Fortin <michel.fortin michelf.com>  writes:


On 2012-02-22 07:41:21 +0000, Jacob Carlborg <doob me.com> said:


 void foo (T) () {}
 void main ()
 {
      foo!(int);
      foo!(char);
 }
 
 $ dmd -inline -O -release main.d
 $ nm main | grep foo
 
 00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
 00000001000012b0 T _D4main10__T3fooTiZ3fooFZv
 
 Outputs two symbols as expected, one for each instantiation.


That's expected indeed.

This doesn't mean the inliner will not inline the templates. In fact, 
if you correct the example and look at the assembler output you'll see 
it will (in your example there is nothing to inline since you're just 
instantiating the template without calling it).

What doesn't happen is stripping the unreferenced symbols from the 
executable. On OS X, try adding "-L-dead_strip" to DMD's argument to 
instruct the linker to do so.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/



 Feb 22 2012



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-02-22 13:53, Michel Fortin wrote:

 On 2012-02-22 07:41:21 +0000, Jacob Carlborg <doob me.com> said:


 void foo (T) () {}
 void main ()
 {
 foo!(int);
 foo!(char);
 }

 $ dmd -inline -O -release main.d
 $ nm main | grep foo

 00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
 00000001000012b0 T _D4main10__T3fooTiZ3fooFZv

 Outputs two symbols as expected, one for each instantiation.


 That's expected indeed.

 This doesn't mean the inliner will not inline the templates. In fact, if
 you correct the example and look at the assembler output you'll see it
 will (in your example there is nothing to inline since you're just
 instantiating the template without calling it).

 What doesn't happen is stripping the unreferenced symbols from the
 executable. On OS X, try adding "-L-dead_strip" to DMD's argument to
 instruct the linker to do so.


1. The example is correct, you can call a method without parentheses
2. Adding -L-dead_strip does not seem to strip the "foo" symbols
3. Adding -L-dead_strip causes a segmentation fault with this example 
(DMD 2.057)

import std.stdio;

void foo (T) () { writeln("asd"); }
void main ()
{
      foo!(int);
      foo!(char);
}

4. I thought the reason for the big executable using the Objective-C/D 
bridge was due to all these template symbols.

-- 
/Jacob Carlborg



 Feb 22 2012



 next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Wed, Feb 22, 2012 at 06:32:14PM +0100, Jacob Carlborg wrote:
[...]

 1. The example is correct, you can call a method without parentheses


[...]

This is not directly related, but isn't calling methods without
parentheses deprecated, unless it's marked  property? I thought we will
soon enforce  property.


T

-- 
Talk is cheap. Whining is actually free. -- Lars Wirzenius



 Feb 22 2012



  parent  Jacob Carlborg <doob me.com>  writes:


On 2012-02-22 18:40, H. S. Teoh wrote:

 On Wed, Feb 22, 2012 at 06:32:14PM +0100, Jacob Carlborg wrote:
 [...]

 1. The example is correct, you can call a method without parentheses


 [...]

 This is not directly related, but isn't calling methods without
 parentheses deprecated, unless it's marked  property? I thought we will
 soon enforce  property.


As I understand it, that will most likely be where we're going. But 
we're not there yet.

-- 
/Jacob Carlborg



 Feb 22 2012



prev sibling  parent reply Michel Fortin <michel.fortin michelf.com>  writes:


On 2012-02-22 17:32:14 +0000, Jacob Carlborg <doob me.com> said:


 On 2012-02-22 13:53, Michel Fortin wrote:

 On 2012-02-22 07:41:21 +0000, Jacob Carlborg <doob me.com> said:
 

 void foo (T) () {}
 void main ()
 {
 foo!(int);
 foo!(char);
 }
 
 $ dmd -inline -O -release main.d
 $ nm main | grep foo
 
 00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
 00000001000012b0 T _D4main10__T3fooTiZ3fooFZv
 
 Outputs two symbols as expected, one for each instantiation.


 
 That's expected indeed.
 
 This doesn't mean the inliner will not inline the templates. In fact, if
 you correct the example and look at the assembler output you'll see it
 will (in your example there is nothing to inline since you're just
 instantiating the template without calling it).
 
 What doesn't happen is stripping the unreferenced symbols from the
 executable. On OS X, try adding "-L-dead_strip" to DMD's argument to
 instruct the linker to do so.
 


 
 1. The example is correct, you can call a method without parentheses


Ah, seems I forgot this.


 2. Adding -L-dead_strip does not seem to strip the "foo" symbols
 3. Adding -L-dead_strip causes a segmentation fault with this example 
 (DMD 2.057)
 
 import std.stdio;
 
 void foo (T) () { writeln("asd"); }
 void main ()
 {
       foo!(int);
       foo!(char);
 }


I'm still on v2.055. And I can confirm -L-dead_strip worked for me, but 
I did a change similar to yours so the template was calling printf 
(that makes it easier to see if it was inlined when looking at the 
assembly). Everything worked as expected with printf. Calling writeln, 
which is a template itself, which could be inlined into foo, could 
cause foo to not be inlined, although it doesn't explain the segfault.

I'm starting to suspect the empty version of foo doesn't get inlined 
for some reason.



 4. I thought the reason for the big executable using the Objective-C/D 
 bridge was due to all these template symbols.


I might have been, but I'd be surprised I didn't try striping the 
executable to fix the problem.

In the end, mostly all the instantiated code has to be there for the 
virtual tables. There's almost one instanciation per method, and that 
is a lot of bloat. Whether the template instanciation is inlined or not 
just changes under which symbol the code resides, the code doesn't 
magically disappear, it still has to be there somewhere.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/



 Feb 22 2012



  parent  Jacob Carlborg <doob me.com>  writes:


On 2012-02-22 19:39, Michel Fortin wrote:

 On 2012-02-22 17:32:14 +0000, Jacob Carlborg <doob me.com> said:


 On 2012-02-22 13:53, Michel Fortin wrote:

 On 2012-02-22 07:41:21 +0000, Jacob Carlborg <doob me.com> said:


 void foo (T) () {}
 void main ()
 {
 foo!(int);
 foo!(char);
 }

 $ dmd -inline -O -release main.d
 $ nm main | grep foo

 00000001000012b8 T _D4main10__T3fooTaZ3fooFZv
 00000001000012b0 T _D4main10__T3fooTiZ3fooFZv

 Outputs two symbols as expected, one for each instantiation.


 That's expected indeed.

 This doesn't mean the inliner will not inline the templates. In fact, if
 you correct the example and look at the assembler output you'll see it
 will (in your example there is nothing to inline since you're just
 instantiating the template without calling it).

 What doesn't happen is stripping the unreferenced symbols from the
 executable. On OS X, try adding "-L-dead_strip" to DMD's argument to
 instruct the linker to do so.


 1. The example is correct, you can call a method without parentheses


 Ah, seems I forgot this.


 2. Adding -L-dead_strip does not seem to strip the "foo" symbols
 3. Adding -L-dead_strip causes a segmentation fault with this example
 (DMD 2.057)

 import std.stdio;

 void foo (T) () { writeln("asd"); }
 void main ()
 {
 foo!(int);
 foo!(char);
 }


 I'm still on v2.055. And I can confirm -L-dead_strip worked for me, but
 I did a change similar to yours so the template was calling printf (that
 makes it easier to see if it was inlined when looking at the assembly).
 Everything worked as expected with printf. Calling writeln, which is a
 template itself, which could be inlined into foo, could cause foo to not
 be inlined, although it doesn't explain the segfault.

 I'm starting to suspect the empty version of foo doesn't get inlined for
 some reason.



 4. I thought the reason for the big executable using the Objective-C/D
 bridge was due to all these template symbols.


 I might have been, but I'd be surprised I didn't try striping the
 executable to fix the problem.

 In the end, mostly all the instantiated code has to be there for the
 virtual tables. There's almost one instanciation per method, and that is
 a lot of bloat. Whether the template instanciation is inlined or not
 just changes under which symbol the code resides, the code doesn't
 magically disappear, it still has to be there somewhere.


Yes, exactly.

-- 
/Jacob Carlborg



 Feb 22 2012



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2012-02-22 00:13, Manu wrote:

 On 21 February 2012 23:06, Jacob Carlborg <doob me.com
     Well, that's not what happen with templates.


 ... really? why?
 You've just made me very very scared for my simd module :/


I've answered to this in an answer to Michel's answer.


 I've had a serious concern about D's lack of a force-inline for quite a
 while...


-- 
/Jacob Carlborg



 Feb 21 2012



prev sibling  parent  Michel Fortin <michel.fortin michelf.com>  writes:


On 2012-02-21 17:03:56 +0000, Manu <turkeyman gmail.com> said:


 What was the primary cause of the bloat? I can't imagine my proposal
 causing any more bloat than the explicit jni call (or equivalent) woudl
 have otherwise.


I had one D class for every corresponding Objective-C class, each 
method wrapped in both direction, each wrapper doing automatic mapping 
of D and Objective-C wrapper objects and translating exceptions. I 
could actually override a method in a derived class of the D wrapper 
and it'd effectively override the method on the Objective-C side. But 
with all those D classes having virtual tables referring to all these 
wrapper functions, even if you use only 1% of all the bindings nothing 
can be stripped of the final executable.

Quoting from a blog post of mine:
"""
But the real show stopper is the size of the generated code for the 
bindings. The system I created needs to generate a lot of code for each 
and every one of the functions and classes you define bindings for. 
This makes the final executable awfully big (and might also explain in 
part the slowness while compiling). The small test application included 
with the bridge takes about 60 Mb of code, thats for something that’d 
be only a few kilobytes when written in Objective-C.

This wasn’t the case at first. This bloat appeared as I added bindings 
for more and more classes in Cocoa. It is a real problem because no one 
in his right mind will want to ship a 60 Mb application that contains 
so much binding code that stay unused most of the time. A solution 
could be that everyone would have to write bindings that only contains 
the classes and methods he needs, but this is of rather limited 
interest.
"""


 Maybe you'll have more luck, but for me wrappers just couldn't work. So I
 decided making the compiler understand the Objective-C runtime and object
 model was a much better solution. It was fun to do and wasn't that hard
 actually. Here's the result: 
 <http://michelf.com/projects/**d-objc/<http://michelf.com/projects/d-objc/>>.




Unfortunately 

 
 I don't have much time to work on it anymore.


 
 That sounds appropriate for Obj-C, but I don't think it applies to VM based
 languages however you slice it, since there's not actually a direct binary
 linkage. There's no way to know about the Java/.NET/Lua/etc object model
 internally, the VM owns that entirely, so you're forced to marshal through
 API's.


That's not actually very different from what I'm doing for Objective-C. 
Call to Objective-C methods are all done through low-level API 
functions in the Objective-C runtime. Class definitions are static 
data, but could be done dynamically through API calls in module 
constructors.

The most interesting bit however is that class *declarations* require 
nothing to be compiled, the compiler will just emit the right code when 
calling a method, using objc_msgSend(). This has a major impact on 
compilation time if you have thousands of such functions.



 That's okay though, you have to do it anyway. I'm proposing a way to
 unify the concept of calling functions without obscuring the actual call
 with junk, and still allowing the D language to annotate and optimise to
 its fullest extent.


The goal is certainly worthwhile. I think your proposal has merit if 
you goal is just to call functions and not define function or classes, 
although I'm skeptical overloading extern() is the right way to do that 
(perhaps a pragma?). But I'd suggest you define your goals more 
throughly before deciding on an implementation.

Some questions you should try to have an answer for regarding OO: if 
you're interacting with classes, how should it work in an ideal world? 
Then, do you need to define new classes that'd be available in the 
foreign language? Do you need to override functions on existing 
classes? How do you want code using foreign objects to look? Then ask 
yourself what would be an acceptable compromise, and start implementing 
it.

For my part, I came to the conclusion that you couldn't really use 
Objective-C APIs in a meaningful way without the ability to create 
subclasses and override functions… which led me to the current design 
where you can use Objective-C classes as if they were D classes.


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/



 Feb 21 2012