www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [WIP] Embed .NET (Core) in your D app

reply evilrat <evilrat666 gmail.com> writes:
Hi everyone,
recently I was working on .NET embedding, and I have done simple 
library for hosting .NET Core in D, it is accompanied with small 

implementation regarding native delegates.
It allows to embed .NET runtime in your program and call .NET 
code, for example it might be some important library implemented 

use it as scripting* engine like Unity does.
*Oh, and this is also interesting because .NET Core 3.1 size is 
roughly 70 MB with libs, which is nearly the same size as 
CPython, but has advantage of performance AND security features.

It is still in quite early stage and lacks some important 
features, so I don't recommend using it in production.
However I think some might find it useful, others might like to 
evaluate the possibilities.


I have quite fuzzy roadmap for future related projects ATM and I 
can't give any precise dates, but anyway here is 4 projects that 
I have in mind:

     1)  host utility library (WIP, already has some crucial 
features, prerequisite for all future work on this list)
     2)  D .NET host library (WIP, .NET Core low level stuff done, 
high level .NET wrapper generator is still not designed though)
     3)  utility tool that outputs proper declarations to use in D 
(planned)
     4)  D to .NET compiler, this ultimately requires all previous 
steps done, and I still haven't sorted out what D features could 
be problematic to implement (not yet started)

Wrt. to compiler my biggest concern is actually not the language 
features itself, since .NET has almost everything D can do, but 
rather D runtime and standard library. This is however is so 
distant future that I can't even say I will ever do this.
The good thing is that I can work on compiler itself thanks to 
dmd-as-a-library dub package.


Now of course the question is why I am even bother doing this?
- .NET 5 is going to release later in 2020 finally unifying both 
MS .NET and Mono VM's
- it will have many improvements, features and stuff, all this is 
cross platform (you can find more info on the net) (oh crap, 
sounds like some BS ad)
- AOT/JIT (D lacks this, can benefit REPL's, and can also in some 
cases give you that last drops of performance)
- so far no other "serious" language have targeted .NET so this 
niche is quite empty, this could give D enormous advantage and 
recognition especially in enterprise (don't quote me on this 
though)
- ???
- profit (oh, I hope so)


source code
https://github.com/Superbelko/dotnethost-d
Feb 10 2020
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 05:21:20 UTC, evilrat wrote:

 Hi everyone,
 recently I was working on .NET embedding, and I have done 
 simple library for hosting .NET Core in D, it is accompanied 

 current .NET implementation regarding native delegates.
 It allows to embed .NET runtime in your program and call .NET 
 code, for example it might be some important library 

 engine and use it as scripting* engine like Unity does.
 *Oh, and this is also interesting because .NET Core 3.1 size is 
 roughly 70 MB with libs, which is nearly the same size as 
 CPython, but has advantage of performance AND security features.

 [...]


I started a project just over a month ago that fills the same 
role:

https://github.com/marler8997/clrbridge

I started with the dflat project found here 
(https://github.com/thewilsonator/dflat) and eventually changed 
the design to create this new one.  I'll have to take a look at 
your project and see if we can share/mutually benefit from our 
approaches.
Feb 10 2020
parent reply evilrat <evilrat666 gmail.com> writes:
On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
wrote:

 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually changed 
 the design to create this new one.  I'll have to take a look at 
 your project and see if we can share/mutually benefit from our 
 approaches.


Yes, when I started it I was looking at dflat first, but 
instantly rejected it because of complexity and extra steps 
involved.

Your project seems to aim to be a complete solution, it looks far 
more robust but also much more complex.

While my library is basically a .NET Core API wrapper over 5 

tools are required, no patching, no intermediate steps. Just 
provide type declarations in D and that's it.
The most complex part is IL generator that weaves in custom 
marshaling logic that avoids current CLR implementation 
limitations (which may or may not be resolved in .NET 5), but it 
also means delegates generated on demand, and the next biggest 
chunk is huge D template to glue this together and hide memory 
management details.
I also keep it pretty independent, one is free to make their own 
wrapper generator, also feel free to use delegate emitter library 
in other projects or even languages.

When designing it I tried to keep it minimalistic as possible, I 
also think it nicely fits the position as a temporary solution 
until I do real compiler (if ever). Meanwhile it might be useful 
on occasion to call few methods here and there.
Feb 10 2020
next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:

 On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
 wrote:

 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually 
 changed the design to create this new one.  I'll have to take 
 a look at your project and see if we can share/mutually 
 benefit from our approaches.


 Yes, when I started it I was looking at dflat first, but 
 instantly rejected it because of complexity and extra steps 
 involved.


After spending some time working on dflat I came to the same 
conclusion.  Since coreclr only supports static functions, Dflat 
was using Cecil to generate wrapper assemblies in order to access 
non-static functions.  However, it's simpler to have just one 
library that can do this for any assembly rather than generating 
a new unique wrapper assembly for each one.


 Your project seems to aim to be a complete solution, it looks 
 far more robust but also much more complex.

 While my library is basically a .NET Core API wrapper over 5 

 tools are required, no patching, no intermediate steps. Just 
 provide type declarations in D and that's it.


The code generation side adds complexity.  However, all the 
generated code ends up calling into a small simple library called 
ClrBridge.dll.

https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs

It defines a handful static functions to load assemblies, get 
types, methods, call methods, marshal values, box values and 
create arrays.  The bare minimum you need to have access to the 
entire Clr if you're only able to call static methods through the 
coreclr library.

You can have access to the full CLR with this library without any 
code generation as well.  Here's an example that doesn't use any 
code generation:

https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d



 The most complex part is IL generator that weaves in custom 
 marshaling logic that avoids current CLR implementation 
 limitations (which may or may not be resolved in .NET 5), but 
 it also means delegates generated on demand, and the next 
 biggest chunk is huge D template to glue this together and hide 
 memory management details.


Interesting.  I haven't found a case yet where I've needed to 
generate IL code to marshal types.  I don't anticipate needing to 
either as I already have a design that should work for marshaling 
any type.  What types are you needing to generate IL to marshal 
for?
Feb 11 2020
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:

 On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
 wrote:

 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually 
 changed the design to create this new one.  I'll have to take 
 a look at your project and see if we can share/mutually 
 benefit from our approaches.


 Yes, when I started it I was looking at dflat first, but 
 instantly rejected it because of complexity and extra steps 
 involved.


After spending some time working on dflat I came to the same 
conclusion.  Since coreclr only supports static functions, Dflat 
was using Cecil to generate wrapper assemblies in order to access 
non-static functions.  However, it's simpler to have just one 
library that can do this for any assembly rather than generating 
a new unique wrapper assembly for each one.


 Your project seems to aim to be a complete solution, it looks 
 far more robust but also much more complex.

 While my library is basically a .NET Core API wrapper over 5 

 tools are required, no patching, no intermediate steps. Just 
 provide type declarations in D and that's it.


The code generation side adds complexity.  However, all the 
generated code ends up calling into a small simple library called 
ClrBridge.dll.

https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs

It defines a handful static functions to load assemblies, get 
types, methods, call methods, marshal values, box values and 
create arrays.  The bare minimum you need to have access to the 
entire Clr if you're only able to call static methods through the 
coreclr library.

You can have access to the full CLR with this library without any 
code generation as well.  Here's an example that doesn't use any 
code generation:

https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d



 The most complex part is IL generator that weaves in custom 
 marshaling logic that avoids current CLR implementation 
 limitations (which may or may not be resolved in .NET 5), but 
 it also means delegates generated on demand, and the next 
 biggest chunk is huge D template to glue this together and hide 
 memory management details.


Interesting.  I haven't found a case yet where I've needed to 
generate IL code to marshal types.  I don't anticipate needing to 
either as I already have a design that should work for marshaling 
any type.  What types are you needing to generate IL to marshal 
for?
Feb 11 2020
parent evilrat <evilrat666 gmail.com> writes:
On Tuesday, 11 February 2020 at 08:47:48 UTC, Jonathan Marler 
wrote:

 The code generation side adds complexity.  However, all the 
 generated code ends up calling into a small simple library 
 called ClrBridge.dll.

 https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs


Yes, but it is moved from user to library maintainer. That way 
users are happy, they just don't care about these details. And it 
only adds single DLL with few dependencies to redistribute. In 
total 12 files less than 1 MB in size, including 4 files for 
Mono.Cecil which is a leftover from my compiler experiments that 
I forgot to exclude.


 It defines a handful static functions to load assemblies, get 
 types, methods, call methods, marshal values, box values and 
 create arrays.  The bare minimum you need to have access to the 
 entire Clr if you're only able to call static methods through 
 the coreclr library.


Most of this is accessible without extra helpers.
That's why I did delegate handling first. It creates static 
delegate at runtime that can be called as normal function 
pointer, and rewrites incompatible types to pass handles instead, 
then before calling former method it obtains actual objects using 
that handles.

System.Activator - is 'new' operator.
System.Array - all things for arrays.
System.Type - basically all reflection
System.Delegate - call methods, etc..

 From that list I only need static helpers for Type.GetType(), 
Type.GetMethod(), Delegate.CreateDelegate() and 
Marshal.GetFunctionPointerForDelegate(), this provides access to 
specific types and methods and after creating delegate and then 
obtaining callable function pointer with these helpers the rest 
should be accessible.



 You can have access to the full CLR with this library without 
 any code generation as well.  Here's an example that doesn't 
 use any code generation:

 https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d


ok, will take a second look.



 Interesting.  I haven't found a case yet where I've needed to 
 generate IL code to marshal types.  I don't anticipate needing 
 to either as I already have a design that should work for 
 marshaling any type.  What types are you needing to generate IL 
 to marshal for?


There is not much going on.
My conclusion is that they just not yet implemented full handling 
for delegates so far.
So all I did was just take reference types in method 
params/return instead to be rewritten as IntPtr and handled using 
GCHandle, simply it just calls 
GCHandle.Alloc(ret)/GCHandle.FromIntPtr(param).Target on return 
value and parameters.
(hamburger analogy where meat is the real method and bread is 
that extra handling)

Though this can change later as it is still at the very early 
stage right now. This does not account for structs with reference 
types, etc...
Also CLR does marshal arrays just fine, while my current solution 
currently takes the route of using object handle for that(which 
might be slower or even incorrect in some cases?).
Feb 11 2020