www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - C# Interop

reply Eelco Hoogendoorn <hoogendoorn.eelco gmail.com> writes:
Hi all,

At work I currently develop in C++ with a C# front-end, using CLI interop.

Ive been using D for quite a while for hobby projects, and needless to say, it
makes C++ feel like masochism. Id like to convince my coworkers that adding D
in the mix is more than just another complication.

But im not quite sure what would be the best way to do C# and D interop. CLI
will no longer do me any good I fear.

Do I just create a D DLL with a bunch of free extern(C) function for
communication? What about marshalling? Is using unsafe pointers back and forth
the best I can do? C# can read from a pointer allocated by D in unsafe mode,
and D can read from a pinned C# pointer, right?

(no, im not looking for D.NET; what I miss in C# is to-the-metal /
compilation.)
Jan 31 2011
next sibling parent David Gileadi <gileadis NSPMgmail.com> writes:
On 1/31/11 2:25 PM, Eelco Hoogendoorn wrote:
 Hi all,

 At work I currently develop in C++ with a C# front-end, using CLI interop.

 Ive been using D for quite a while for hobby projects, and needless to say, it
 makes C++ feel like masochism. Id like to convince my coworkers that adding D
 in the mix is more than just another complication.

 But im not quite sure what would be the best way to do C# and D interop. CLI
 will no longer do me any good I fear.

 Do I just create a D DLL with a bunch of free extern(C) function for
 communication? What about marshalling? Is using unsafe pointers back and forth
 the best I can do? C# can read from a pointer allocated by D in unsafe mode,
 and D can read from a pinned C# pointer, right?

 (no, im not looking for D.NET; what I miss in C# is to-the-metal /
 compilation.)

It's been a number of years and it was in D1, so take this for the little value it may have, but the last time I tried making a D DLL used by a C# client I got hangs. I suspected it was the interaction of the garbage collectors but didn't have the time to debug it (it was for a school project). I'd be interested to hear if someone else has gotten this working. -Dave
Jan 31 2011
prev sibling next sibling parent Michal Minich <michal.minich gmail.com> writes:
On Mon, 31 Jan 2011 21:25:11 +0000, Eelco Hoogendoorn wrote:

 Do I just create a D DLL with a bunch of free extern(C) function for
 communication?

Yes, why not. You can pass struts in extern(C) functions...
 What about marshalling? Is using unsafe pointers back and
 forth the best I can do? 

No.
 C# can read from a pointer allocated by D in
 unsafe mode, and D can read from a pinned C# pointer, right?

Yes. Just an idea, but have you considered using COM. C# interoperates with it seamlessly and heard of some D projects using it. one starter link: http://www.dsource.org/projects/juno/wiki/ComProgramming
Jan 31 2011
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn  
<hoogendoorn.eelco gmail.com> wrote:

 Hi all,

 At work I currently develop in C++ with a C# front-end, using CLI  
 interop.

 Ive been using D for quite a while for hobby projects, and needless to  
 say, it
 makes C++ feel like masochism. Id like to convince my coworkers that  
 adding D
 in the mix is more than just another complication.

 But im not quite sure what would be the best way to do C# and D interop.  
 CLI
 will no longer do me any good I fear.

 Do I just create a D DLL with a bunch of free extern(C) function for
 communication?

Basically, yes. I've been doing this for a project and it works reasonably well. I have one module based on From D's/bugzilla's public domain example code to properly enable the dll and then declare the interop functions as export extern(C) {}. There's also some general helper functions you'll want to write. Although exceptions can propagate to .NET, they just turn into a System.Runtime.InteropServices.SEHException exception, so you'll want to wrap all your export function in a try-catch block and save the error message to a global variable. Then you can call a lastError function to get the actual error string. Oh, and remember .NET defaults to wstrings, not strings. The other helper function you'll want is a way to pin and unpin objects as D's GC can't see C#'s memory.
 What about marshalling?

C# marshaling, though I'm glad it's there, involves pulling teeth to do anything other then calling basic C system calls. Lucky, you really only need it for arrays and structs. Objects can be be treated as handles inside a proxy C# object. And then you can handle methods as free functions whose first argument is the object's handle. But you'll also have to write a C# proxy object + pin/unpin the D object. If you're doing a lot of this, I'd recommend writing a mixin to generate all the free functions on the D side, and looking into the dynamic language features of C# to write an auto-wrapping proxy object. (I haven't needed to do this yet)
 Is using unsafe pointers back and forth
 the best I can do? C# can read from a pointer allocated by D in unsafe  
 mode,
 and D can read from a pinned C# pointer, right?

I don't know if it's the best you can do, but it does work.
 (no, im not looking for D.NET; what I miss in C# is to-the-metal /
 compilation.)

Lastly, D DLLs will only work on Vista/Windows 7/later. They will not work on XP. This is due to a long known bug with DLLs and thread local storage in general on XP. Also, you'll have to use 32-bit C# currently, as DMD isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit version of DMD, but it will be Linux only at first, with Windows following sometime later) I've listed some example code from my project below: // Written in the D Programming Language (www.digitalmars.com/d) ///Basic DLL setup and teardown code. From D's/bugzilla's public domain example code. module dll; import std.c.windows.windows; import std.c.stdlib; import core.runtime; import core.memory; extern (Windows) BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved) { switch (ulReason) { case DLL_PROCESS_ATTACH: Runtime.initialize(); break; case DLL_PROCESS_DETACH: Runtime.terminate(); break; case DLL_THREAD_ATTACH: case DLL_THREAD_DETACH: return false; } return true; } D code: private { wstring last_error_msg = "No error"; /// The last encountered by the dose server __gshared Object[Object] pinned; /// Hash of pinned objects void pin(Object value) { pinned[ value ] = value; } /// Pin an object /// Stores a string as the last error void lastError(string str) { wstring err; auto app = appender(&err); foreach (dchar c; str) app.put(c); last_error_msg = err; enforce(false); } } export extern(C) { ref wstring lastError() { return last_error_msg; } /// returns: the last error message, used for exception marshaling } C# code: [DllImport(gpu_dll)] [return: MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(DString))] private static extern string lastError(); /// <summary> /// A custom marshaller for D's strings (char[]) passed by ref /// </summary> private class DString : ICustomMarshaler { //[ThreadStatic] //private static ICustomMarshaler _instance = new DString(); private IntPtr ptr = IntPtr.Zero; /// <summary> /// Factory method /// </summary> /// <param name="pstrCookie"></param> /// <returns></returns> static ICustomMarshaler GetInstance(String pstrCookie) { return new DString(); } /// <summary> /// Convert a pointer to a D array to a managed T[] /// </summary> /// <param name="pNativeData"></param> /// <returns></returns> public Object MarshalNativeToManaged(IntPtr pNativeData) { Int32 length = Marshal.ReadInt32(pNativeData, 0); IntPtr data = Marshal.ReadIntPtr(pNativeData, 4); char[] result = new char[length]; Marshal.Copy(data, result, 0, length); return new string(result); } /// <summary> /// Convert a managed T[] to a pointer to a D array /// </summary> /// <param name="ManagedObj"></param> /// <returns></returns> public IntPtr MarshalManagedToNative(Object ManagedObj) { char[] managed = ((string)ManagedObj).ToCharArray(); ; ptr = Marshal.AllocHGlobal(8 + managed.Length * 2); // unicode = 16 bytes, sigh IntPtr data = (IntPtr)((UInt32)ptr + 8); Marshal.Copy(managed, 0, data, managed.Length); Marshal.WriteInt32(ptr, 0, managed.Length); Marshal.WriteIntPtr(ptr, 4, data); return ptr; } /// <summary> /// Delete the marshaled D array /// </summary> /// <param name="pNativeData"></param> public void CleanUpNativeData(IntPtr pNativeData) { if (ptr == pNativeData) Marshal.FreeHGlobal(pNativeData); } /// <summary> /// Clean up managed data (i.e. do nothing) /// </summary> /// <param name="ManagedObj"></param> public void CleanUpManagedData(Object ManagedObj) { } /// <returns>The size of a D array struct (8)</returns> public int GetNativeDataSize() { return 8; } }
Jan 31 2011
next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 31 Jan 2011 22:32:35 -0500, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:

 On 2/1/11, Robert Jacques <sandford jhu.edu> wrote:
 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not  
 work
 on XP. This is due to a long known bug with DLLs and thread local  
 storage
 in general on XP.

Is there a bugzilla link for this?

This isn't a D issue. It's a well documented problem with thread local storage with all DLLs on Windows XP. I haven't seen a specific bugzilla on this issue, but there are several issues with writting/loading DLLs in D, which I assume will be addressed at/about the time Linex .SOs are, which are listed as next/in progress after 64-bit support.
Jan 31 2011
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Robert Jacques wrote:
 On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn 
 <hoogendoorn.eelco gmail.com> wrote:
 [...]
 
 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not 
 work on XP. This is due to a long known bug with DLLs and thread local 
 storage in general on XP. Also, you'll have to use 32-bit C# currently, 
 as DMD isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit 
 version of DMD, but it will be Linux only at first, with Windows 
 following sometime later)
 

XP TLS support with dynamically loaded DLLs is fixed for some time now with a workaround implemented in druntime. Also, DLLs can be used in multi-threading environments.
 I've listed some example code from my project below:

 // Written in the D Programming Language (www.digitalmars.com/d)
 ///Basic DLL setup and teardown code. From D's/bugzilla's public domain
 example code.
 module dll;

 import std.c.windows.windows;
 import std.c.stdlib;
 import core.runtime;
 import core.memory;

 extern (Windows)
 BOOL DllMain(HINSTANCE hInstance, ULONG ulReason, LPVOID pvReserved) {
     switch (ulReason) {
         case DLL_PROCESS_ATTACH:
             Runtime.initialize();
             break;
         case DLL_PROCESS_DETACH:
             Runtime.terminate();
             break;
         case DLL_THREAD_ATTACH:
         case DLL_THREAD_DETACH:
             return false;
     }
     return true;
 }

This DLLMain code is a bit outdated (is it D1?), the current proposed version is here: http://www.digitalmars.com/d/2.0/dll.html Unfortunately, there is a regression in the latest dmd release (2.051): http://d.puremagic.com/issues/show_bug.cgi?id=5382 that causes TLS not to be initialized for new threads created by the host application. Rainer
Feb 01 2011
next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
Robert Jacques wrote:
 On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze <r.sagitario gmx.de> 
 wrote:
 XP TLS support with dynamically loaded DLLs is fixed for some time now 
 with a workaround implemented in druntime. Also, DLLs can be used in 
 multi-threading environments.

Yes, I pointed out in another thread that D loading D DLLs can work around this issue, but the original post was about calling a D DLL from another language, specifically C#, where the limitation in XP still exists. (Of course, you might be able to port the work around to C#. Hmm...)

The workaround is not about D loading a D DLL. Visual D lives happily in the C++/C# world of Visual Studio, even on XP. It's the magic inside dll_process_attach() that sets up TLS for existing threads and patches the loader structures to make XP think the DLL was loaded at startup (where implicite TLS works). The downside is that the DLL cannot be unloaded, though.
 
  > I've listed some example code from my project below:

[snip]
 This DLLMain code is a bit outdated (is it D1?), the current proposed 
 version is here: http://www.digitalmars.com/d/2.0/dll.html

Thanks. It was D2, but it was forked a while ago. Given that the recommended way of doing this might change in the future, a string mixin in core.dll_helper might be appropriate.

I don't like mixins too much, but a standard_DllMain that you can forward to from DllMain, might be a good idea to include into the runtime library.
Feb 01 2011
prev sibling parent reply Richard Webb <webby beardmouse.org.uk> writes:
 Also, DLLs can be used in multi-threading environments.


On a related note, i'm having a bit of a problem with a D dll at the moment. I have an Outlook COM Addin that is written in D2 using the Juno library and that is running ok, but i'm now trying to use a D dll from a COM addin written in C++ and i'm getting a crash inside dll_process_attach. The only thing i get in the call stack is gcx.mark, but it appears that the crash is due to the GC running midway through the thread setup code. Disabling the GC during the thread_moduleTlsCtor() calls avoids the crash, but that might just be covering the problem up. This is with the latest druntime/phobos code from a couple of days ago plus your patch from #5382, running on Windows 2008. Any ideas?
Feb 02 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Richard Webb wrote:
 Also, DLLs can be used in multi-threading environments.


On a related note, i'm having a bit of a problem with a D dll at the moment. I have an Outlook COM Addin that is written in D2 using the Juno library and that is running ok, but i'm now trying to use a D dll from a COM addin written in C++ and i'm getting a crash inside dll_process_attach. The only thing i get in the call stack is gcx.mark, but it appears that the crash is due to the GC running midway through the thread setup code. Disabling the GC during the thread_moduleTlsCtor() calls avoids the crash, but that might just be covering the problem up.

If there is a garbage collection going on, I assume that you do quite a lot of initialization for TLS. Are you using callbacks into the C++ DLL from the module ctors? This might be dangerous... You might want to try this version of impersonate thread(), that might work better if the C++ part also uses thread local storage: // execute function on the TLS for the given thread static void impersonate_thread( uint id, void function() fn ) { if( id == GetCurrentThreadId() ) { fn(); return; } // temporarily set current TLS array pointer to the array pointer of the referenced thread void** curteb = getTEB(); void** teb = getTEB( id ); assert( teb && curteb ); void** curtlsarray = cast(void**) curteb[11]; void** tlsarray = cast(void**) teb[11]; if( !curtlsarray || !tlsarray ) return; curteb[11] = tlsarray; fn(); curteb[11] = curtlsarray; } Especially loading more DLLs from the TLS init code will still be troublesome... Rainer
Feb 02 2011
parent reply Richard Webb <webby beardmouse.org.uk> writes:
 I assume that you do quite a lot of initialization for TLS


I'm not directly, but Juno has a number of static class constructors that allocate static members, and Outlook creates lots of threads (for example, i see the static constructor of the module juno.com.core get called 20+ times on startup). The Juno lib predates the TLS changes in D2 so i guess that some of these static variables don't need to be thread local, but i haven't had chance to investigate that.
Are you using callbacks into the C++ DLL from the module ctors


The D dll just exports a couple of 'C' functions that the C++ dll calls. There are no calls in the other direction.
 that might work better if the C++ part also uses thread local storage


The C++ dll uses ATL for it's COM stuff, and i'm not sure what that does with TLS in the background. I'll try your suggested change tommorow. Thanks, Richard Webb
Feb 02 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
I forgot one more thing: I don't know how juno deals with it, but I 
consider the default memory management of COM objects in phobos/druntime 
rather unusable. It leaks memory and can easily cause garbage collection 
on objects still referenced by COM objects. That's why I use 
http://d.puremagic.com/issues/show_bug.cgi?id=4092

Rainer

Richard Webb wrote:
 I assume that you do quite a lot of initialization for TLS


I'm not directly, but Juno has a number of static class constructors that allocate static members, and Outlook creates lots of threads (for example, i see the static constructor of the module juno.com.core get called 20+ times on startup). The Juno lib predates the TLS changes in D2 so i guess that some of these static variables don't need to be thread local, but i haven't had chance to investigate that.
 Are you using callbacks into the C++ DLL from the module ctors


The D dll just exports a couple of 'C' functions that the C++ dll calls. There are no calls in the other direction.
 that might work better if the C++ part also uses thread local storage


The C++ dll uses ATL for it's COM stuff, and i'm not sure what that does with TLS in the background. I'll try your suggested change tommorow. Thanks, Richard Webb

Feb 03 2011
parent Richard Webb <webby beardmouse.org.uk> writes:
It overrides new and allocates class instances using malloc. The official
version
calls addRange from new, and removeRange/free from Release when the reference
count is 0, but i changed my local version to use add/remove(Root) instead of
range at some point.
I'm only currently trying to use external com objects rather than creating my
own,
so this isn't an issue here.


fwiw, i just marked some static variables in juno as __gshared to work around
the
problem for now, and it seems to be loading ok. Juno could really do with some
official maintainance work to keep up with D2 changes, but the author seems to
have left a while back  :-(
Feb 06 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:
 Robert Jacques wrote:
 On Mon, 31 Jan 2011 16:25:11 -0500, Eelco Hoogendoorn  
 <hoogendoorn.eelco gmail.com> wrote:
 [...]
  Lastly, D DLLs will only work on Vista/Windows 7/later. They will not  
 work on XP. This is due to a long known bug with DLLs and thread local  
 storage in general on XP. Also, you'll have to use 32-bit C# currently,  
 as DMD isn't 64-bit compatible yet. (Walter is hard at work on a 64-bit  
 version of DMD, but it will be Linux only at first, with Windows  
 following sometime later)

XP TLS support with dynamically loaded DLLs is fixed for some time now with a workaround implemented in druntime. Also, DLLs can be used in multi-threading environments.

Yes, I pointed out in another thread that D loading D DLLs can work around this issue, but the original post was about calling a D DLL from another language, specifically C#, where the limitation in XP still exists. (Of course, you might be able to port the work around to C#. Hmm...)
  > I've listed some example code from my project below:

[snip]
 This DLLMain code is a bit outdated (is it D1?), the current proposed  
 version is here: http://www.digitalmars.com/d/2.0/dll.html

Thanks. It was D2, but it was forked a while ago. Given that the recommended way of doing this might change in the future, a string mixin in core.dll_helper might be appropriate.
Feb 01 2011
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 01 Feb 2011 13:33:20 -0500, Rainer Schuetze <r.sagitario gmx.de>  
wrote:

 Robert Jacques wrote:
 On Tue, 01 Feb 2011 03:05:13 -0500, Rainer Schuetze  
 <r.sagitario gmx.de> wrote:
 XP TLS support with dynamically loaded DLLs is fixed for some time now  
 with a workaround implemented in druntime. Also, DLLs can be used in  
 multi-threading environments.

around this issue, but the original post was about calling a D DLL from another language, specifically C#, where the limitation in XP still exists. (Of course, you might be able to port the work around to C#. Hmm...)

The workaround is not about D loading a D DLL. Visual D lives happily in the C++/C# world of Visual Studio, even on XP. It's the magic inside dll_process_attach() that sets up TLS for existing threads and patches the loader structures to make XP think the DLL was loaded at startup (where implicite TLS works). The downside is that the DLL cannot be unloaded, though.

Thanks, again. Though the pros and cons of this should be listed in the docs somewhere.
  > I've listed some example code from my project below:

 This DLLMain code is a bit outdated (is it D1?), the current proposed  
 version is here: http://www.digitalmars.com/d/2.0/dll.html

recommended way of doing this might change in the future, a string mixin in core.dll_helper might be appropriate.

I don't like mixins too much, but a standard_DllMain that you can forward to from DllMain, might be a good idea to include into the runtime library.

Yes, on second thought, a standard_DllMain is the better solution.
Feb 01 2011
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/1/11, Robert Jacques <sandford jhu.edu> wrote:
 Lastly, D DLLs will only work on Vista/Windows 7/later. They will not work
 on XP. This is due to a long known bug with DLLs and thread local storage
 in general on XP.

Is there a bugzilla link for this?
Jan 31 2011