www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [WIP] Embed .NET (Core) in your D app

reply evilrat <evilrat666 gmail.com> writes:
Hi everyone,
recently I was working on .NET embedding, and I have done simple 
library for hosting .NET Core in D, it is accompanied with small 

implementation regarding native delegates.
It allows to embed .NET runtime in your program and call .NET 
code, for example it might be some important library implemented 

use it as scripting* engine like Unity does.
*Oh, and this is also interesting because .NET Core 3.1 size is 
roughly 70 MB with libs, which is nearly the same size as 
CPython, but has advantage of performance AND security features.

It is still in quite early stage and lacks some important 
features, so I don't recommend using it in production.
However I think some might find it useful, others might like to 
evaluate the possibilities.


I have quite fuzzy roadmap for future related projects ATM and I 
can't give any precise dates, but anyway here is 4 projects that 
I have in mind:

     1)  host utility library (WIP, already has some crucial 
features, prerequisite for all future work on this list)
     2)  D .NET host library (WIP, .NET Core low level stuff done, 
high level .NET wrapper generator is still not designed though)
     3)  utility tool that outputs proper declarations to use in D 
(planned)
     4)  D to .NET compiler, this ultimately requires all previous 
steps done, and I still haven't sorted out what D features could 
be problematic to implement (not yet started)

Wrt. to compiler my biggest concern is actually not the language 
features itself, since .NET has almost everything D can do, but 
rather D runtime and standard library. This is however is so 
distant future that I can't even say I will ever do this.
The good thing is that I can work on compiler itself thanks to 
dmd-as-a-library dub package.


Now of course the question is why I am even bother doing this?
- .NET 5 is going to release later in 2020 finally unifying both 
MS .NET and Mono VM's
- it will have many improvements, features and stuff, all this is 
cross platform (you can find more info on the net) (oh crap, 
sounds like some BS ad)
- AOT/JIT (D lacks this, can benefit REPL's, and can also in some 
cases give you that last drops of performance)
- so far no other "serious" language have targeted .NET so this 
niche is quite empty, this could give D enormous advantage and 
recognition especially in enterprise (don't quote me on this 
though)
- ???
- profit (oh, I hope so)


source code
https://github.com/Superbelko/dotnethost-d
Feb 10
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 05:21:20 UTC, evilrat wrote:
 Hi everyone,
 recently I was working on .NET embedding, and I have done 
 simple library for hosting .NET Core in D, it is accompanied 

 current .NET implementation regarding native delegates.
 It allows to embed .NET runtime in your program and call .NET 
 code, for example it might be some important library 

 engine and use it as scripting* engine like Unity does.
 *Oh, and this is also interesting because .NET Core 3.1 size is 
 roughly 70 MB with libs, which is nearly the same size as 
 CPython, but has advantage of performance AND security features.

 [...]
I started a project just over a month ago that fills the same role: https://github.com/marler8997/clrbridge I started with the dflat project found here (https://github.com/thewilsonator/dflat) and eventually changed the design to create this new one. I'll have to take a look at your project and see if we can share/mutually benefit from our approaches.
Feb 10
parent reply evilrat <evilrat666 gmail.com> writes:
On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
wrote:
 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually changed 
 the design to create this new one.  I'll have to take a look at 
 your project and see if we can share/mutually benefit from our 
 approaches.
Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved. Your project seems to aim to be a complete solution, it looks far more robust but also much more complex. While my library is basically a .NET Core API wrapper over 5 tools are required, no patching, no intermediate steps. Just provide type declarations in D and that's it. The most complex part is IL generator that weaves in custom marshaling logic that avoids current CLR implementation limitations (which may or may not be resolved in .NET 5), but it also means delegates generated on demand, and the next biggest chunk is huge D template to glue this together and hide memory management details. I also keep it pretty independent, one is free to make their own wrapper generator, also feel free to use delegate emitter library in other projects or even languages. When designing it I tried to keep it minimalistic as possible, I also think it nicely fits the position as a temporary solution until I do real compiler (if ever). Meanwhile it might be useful on occasion to call few methods here and there.
Feb 10
next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:
 On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
 wrote:
 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually 
 changed the design to create this new one.  I'll have to take 
 a look at your project and see if we can share/mutually 
 benefit from our approaches.
Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved.
After spending some time working on dflat I came to the same conclusion. Since coreclr only supports static functions, Dflat was using Cecil to generate wrapper assemblies in order to access non-static functions. However, it's simpler to have just one library that can do this for any assembly rather than generating a new unique wrapper assembly for each one.
 Your project seems to aim to be a complete solution, it looks 
 far more robust but also much more complex.

 While my library is basically a .NET Core API wrapper over 5 

 tools are required, no patching, no intermediate steps. Just 
 provide type declarations in D and that's it.
The code generation side adds complexity. However, all the generated code ends up calling into a small simple library called ClrBridge.dll. https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs It defines a handful static functions to load assemblies, get types, methods, call methods, marshal values, box values and create arrays. The bare minimum you need to have access to the entire Clr if you're only able to call static methods through the coreclr library. You can have access to the full CLR with this library without any code generation as well. Here's an example that doesn't use any code generation: https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d
 The most complex part is IL generator that weaves in custom 
 marshaling logic that avoids current CLR implementation 
 limitations (which may or may not be resolved in .NET 5), but 
 it also means delegates generated on demand, and the next 
 biggest chunk is huge D template to glue this together and hide 
 memory management details.
Interesting. I haven't found a case yet where I've needed to generate IL code to marshal types. I don't anticipate needing to either as I already have a design that should work for marshaling any type. What types are you needing to generate IL to marshal for?
Feb 11
prev sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 11 February 2020 at 07:05:28 UTC, evilrat wrote:
 On Tuesday, 11 February 2020 at 05:41:21 UTC, Jonathan Marler 
 wrote:
 I started a project just over a month ago that fills the same 
 role:

 https://github.com/marler8997/clrbridge

 I started with the dflat project found here 
 (https://github.com/thewilsonator/dflat) and eventually 
 changed the design to create this new one.  I'll have to take 
 a look at your project and see if we can share/mutually 
 benefit from our approaches.
Yes, when I started it I was looking at dflat first, but instantly rejected it because of complexity and extra steps involved.
After spending some time working on dflat I came to the same conclusion. Since coreclr only supports static functions, Dflat was using Cecil to generate wrapper assemblies in order to access non-static functions. However, it's simpler to have just one library that can do this for any assembly rather than generating a new unique wrapper assembly for each one.
 Your project seems to aim to be a complete solution, it looks 
 far more robust but also much more complex.

 While my library is basically a .NET Core API wrapper over 5 

 tools are required, no patching, no intermediate steps. Just 
 provide type declarations in D and that's it.
The code generation side adds complexity. However, all the generated code ends up calling into a small simple library called ClrBridge.dll. https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs It defines a handful static functions to load assemblies, get types, methods, call methods, marshal values, box values and create arrays. The bare minimum you need to have access to the entire Clr if you're only able to call static methods through the coreclr library. You can have access to the full CLR with this library without any code generation as well. Here's an example that doesn't use any code generation: https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d
 The most complex part is IL generator that weaves in custom 
 marshaling logic that avoids current CLR implementation 
 limitations (which may or may not be resolved in .NET 5), but 
 it also means delegates generated on demand, and the next 
 biggest chunk is huge D template to glue this together and hide 
 memory management details.
Interesting. I haven't found a case yet where I've needed to generate IL code to marshal types. I don't anticipate needing to either as I already have a design that should work for marshaling any type. What types are you needing to generate IL to marshal for?
Feb 11
parent evilrat <evilrat666 gmail.com> writes:
On Tuesday, 11 February 2020 at 08:47:48 UTC, Jonathan Marler 
wrote:
 The code generation side adds complexity.  However, all the 
 generated code ends up calling into a small simple library 
 called ClrBridge.dll.

 https://github.com/marler8997/clrbridge/blob/master/dotnetlib/ClrBridge.cs
Yes, but it is moved from user to library maintainer. That way users are happy, they just don't care about these details. And it only adds single DLL with few dependencies to redistribute. In total 12 files less than 1 MB in size, including 4 files for Mono.Cecil which is a leftover from my compiler experiments that I forgot to exclude.
 It defines a handful static functions to load assemblies, get 
 types, methods, call methods, marshal values, box values and 
 create arrays.  The bare minimum you need to have access to the 
 entire Clr if you're only able to call static methods through 
 the coreclr library.
Most of this is accessible without extra helpers. That's why I did delegate handling first. It creates static delegate at runtime that can be called as normal function pointer, and rewrites incompatible types to pass handles instead, then before calling former method it obtains actual objects using that handles. System.Activator - is 'new' operator. System.Array - all things for arrays. System.Type - basically all reflection System.Delegate - call methods, etc.. From that list I only need static helpers for Type.GetType(), Type.GetMethod(), Delegate.CreateDelegate() and Marshal.GetFunctionPointerForDelegate(), this provides access to specific types and methods and after creating delegate and then obtaining callable function pointer with these helpers the rest should be accessible.
 You can have access to the full CLR with this library without 
 any code generation as well.  Here's an example that doesn't 
 use any code generation:

 https://github.com/marler8997/clrbridge/blob/master/dlang/noCodegenExample.d
ok, will take a second look.
 Interesting.  I haven't found a case yet where I've needed to 
 generate IL code to marshal types.  I don't anticipate needing 
 to either as I already have a design that should work for 
 marshaling any type.  What types are you needing to generate IL 
 to marshal for?
There is not much going on. My conclusion is that they just not yet implemented full handling for delegates so far. So all I did was just take reference types in method params/return instead to be rewritten as IntPtr and handled using GCHandle, simply it just calls GCHandle.Alloc(ret)/GCHandle.FromIntPtr(param).Target on return value and parameters. (hamburger analogy where meat is the real method and bread is that extra handling) Though this can change later as it is still at the very early stage right now. This does not account for structs with reference types, etc... Also CLR does marshal arrays just fine, while my current solution currently takes the route of using object handle for that(which might be slower or even incorrect in some cases?).
Feb 11