www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How to make D resolve C++ symbols by mangling symbols with the Itanium

reply thumbgun <thumbgun07 gmail.com> writes:
I'm currently trying to call some C++ functions that were 
compiled by g++ (mingw). However g++ uses the Itanium ABI name 
mangling rules. dmd on Windows tries to link functions based on 
the MSVC name mangling rules. For example:

```
// add.cpp
int add(int lhs, int rhs)
{
     return lhs + rhs;
}


// main.d
import std.stdio;

extern(C++) int add(int lhs, int rhs);

void main()
{
     auto x = add(5, 6);
     writeln(x);
}


// Command line
g++ add.cpp -c -o add.obj
dmd main.d add.obj


// Output
main.obj : error LNK2019: unresolved external symbol "int __cdecl 
add(int,int)"
(?add  YAHHH Z) referenced in function _Dmain

```
The prefix for MSVC symbols is '?' while the Itanium ABI's prefix 
is '_Z'. Is there any way I can make dmd link to symbols mangled 
according to the Itanium ABI's rules on Windows?
Feb 26
next sibling parent reply thumbgun <thumbgun07 gmail.com> writes:
The title wasn't very accurate as it isn't dmd that mangles the 
names but my question still stands.
Feb 26
parent reply user1234 <user1234 12.de> writes:
On Monday, 26 February 2024 at 13:40:56 UTC, thumbgun wrote:

 The title wasn't very accurate as it isn't dmd that mangles the 
 names but my question still stands.


Have you tried [pragma 
mangle](https://dlang.org/spec/pragma.html#mangle) ?
Feb 26
parent reply thumbgun <thumbgun07 gmail.com> writes:
On Monday, 26 February 2024 at 13:58:46 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 13:40:56 UTC, thumbgun wrote:

 The title wasn't very accurate as it isn't dmd that mangles 
 the names but my question still stands.


 Have you tried [pragma 
 mangle](https://dlang.org/spec/pragma.html#mangle) ?


I have seen it but it's not really what I need. If I don't find a 
solution I might have to build all of the C++ libraries I need 
with MSVC.
Feb 26
parent reply user1234 <user1234 12.de> writes:
On Monday, 26 February 2024 at 14:41:26 UTC, thumbgun wrote:

 On Monday, 26 February 2024 at 13:58:46 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 13:40:56 UTC, thumbgun wrote:

 The title wasn't very accurate as it isn't dmd that mangles 
 the names but my question still stands.


 Have you tried [pragma 
 mangle](https://dlang.org/spec/pragma.html#mangle) ?


 I have seen it but it's not really what I need.
 [...]


Not even

```d
pragma(mangle, "?add  YAHHH Z")
extern(C++) int add(int lhs, int rhs);
```

?
Feb 26
parent reply user1234 <user1234 12.de> writes:
On Monday, 26 February 2024 at 15:46:34 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 14:41:26 UTC, thumbgun wrote:

 On Monday, 26 February 2024 at 13:58:46 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 13:40:56 UTC, thumbgun wrote:

 The title wasn't very accurate as it isn't dmd that mangles 
 the names but my question still stands.


 Have you tried [pragma 
 mangle](https://dlang.org/spec/pragma.html#mangle) ?


 I have seen it but it's not really what I need.
 [...]


 Not even

 ```d
 pragma(mangle, "?add  YAHHH Z")
 extern(C++) int add(int lhs, int rhs);
 ```

 ?


Sorry, the one starting with a qmark is already what the D obj 
thinks it has to be linked to. I meant

```d
pragma(mangle, "_Zadd  YAHHH Z")
extern(C++) int add(int lhs, int rhs);
```
Feb 26
parent reply thumbgun <thumbgun07 gmail.com> writes:
On Monday, 26 February 2024 at 15:49:40 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 15:46:34 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 14:41:26 UTC, thumbgun wrote:

 On Monday, 26 February 2024 at 13:58:46 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 13:40:56 UTC, thumbgun wrote:

 The title wasn't very accurate as it isn't dmd that mangles 
 the names but my question still stands.


 Have you tried [pragma 
 mangle](https://dlang.org/spec/pragma.html#mangle) ?


 I have seen it but it's not really what I need.
 [...]


 Not even

 ```d
 pragma(mangle, "?add  YAHHH Z")
 extern(C++) int add(int lhs, int rhs);
 ```

 ?


 Sorry, the one starting with a qmark is already what the D obj 
 thinks it has to be linked to. I meant

 ```d
 pragma(mangle, "_Zadd  YAHHH Z")
 extern(C++) int add(int lhs, int rhs);
 ```


The name mangling differences between the Itanium ABI and the 
MSVC ABI are more significant than just the prefix. I only 
mentioned that the prefixes are different to show that dmd was 
actually trying to link against symbols mangled by the MSVC ABI, 
not the Itanium one. If I used pragma mangle then I would have to 
manually translate each C++ function (signature) and struct/class 
to what the Itanium ABI would mangle them to which is very error 
prone and time consuming. I'm going to read the dmd source code - 
and maybe make a fork - because it apparently only tries to link 
against MSVC mangled symbols on Windows and not other platforms.
Feb 26
parent reply user1234 <user1234 12.de> writes:
On Monday, 26 February 2024 at 17:23:32 UTC, thumbgun wrote:

 [...]
 The name mangling differences between the Itanium ABI and the 
 MSVC ABI are more significant than just the prefix. I only 
 mentioned that the prefixes are different to show that dmd was 
 actually trying to link against symbols mangled by the MSVC 
 ABI, not the Itanium one. If I used pragma mangle then I would 
 have to manually translate each C++ function (signature) and 
 struct/class to what the Itanium ABI would mangle them to which 
 is very error prone and time consuming. I'm going to read the 
 dmd source code - and maybe make a fork - because it apparently 
 only tries to link against MSVC mangled symbols on Windows and 
 not other platforms.


I see. That would be extremely painful. Could CTFE help here ?

```d
string getRightMangling(alias T)()
{
     return "todo" // ;)
}

pragma(mangle, getRightMangling!(add))
extern(C++) int add(int lhs, int rhs);
```

That's not a serious suggestion but I'm still curious to see how 
that would scale.

Also that looks weird that the pragma refers to what it is 
supposed to affect, not sure that would work, although in theory 
D manages forward refs.
Feb 26
next sibling parent reply user1234 <user1234 12.de> writes:
On Monday, 26 February 2024 at 19:50:31 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 17:23:32 UTC, thumbgun wrote:

 [...]


 I see. That would be extremely painful. Could CTFE help here ?

 ```d
 string getRightMangling(alias T)()
 {
     return "todo" // ;)
 }

 pragma(mangle, getRightMangling!(add))
 extern(C++) int add(int lhs, int rhs);
 ```

 That's not a serious suggestion but I'm still curious to see 
 how that would scale.

 Also that looks weird that the pragma refers to what it is 
 supposed to affect, not sure that would work, although in 
 theory D manages forward refs.


Turns out that the forward ref is not a problem. Remains the 
question of writing a mangler working fast at compile time ;)
Feb 26
parent thumbgun <thumbgun07 gmail.com> writes:
On Monday, 26 February 2024 at 20:07:55 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 19:50:31 UTC, user1234 wrote:

 On Monday, 26 February 2024 at 17:23:32 UTC, thumbgun wrote:

 [...]


 I see. That would be extremely painful. Could CTFE help here ?

 ```d
 string getRightMangling(alias T)()
 {
     return "todo" // ;)
 }

 pragma(mangle, getRightMangling!(add))
 extern(C++) int add(int lhs, int rhs);
 ```

 That's not a serious suggestion but I'm still curious to see 
 how that would scale.

 Also that looks weird that the pragma refers to what it is 
 supposed to affect, not sure that would work, although in 
 theory D manages forward refs.


 Turns out that the forward ref is not a problem. Remains the 
 question of writing a mangler working fast at compile time ;)


Although I'm not quite sure what available features are permitted 
for CTFE, I'll definitely have a look at that. Johan raised an 
important point though about calling conventions as they are 
different between the Itanium and the MSVC ABIs. So even if DMD 
can link against Itanium mangled symbols, it won't follow the 
same calling convention.

Another alternative apart from compiling libraries from source 
with MSVC would be to create a C API to wrap the C++ library but 
that's also time consuming.
Feb 26
prev sibling parent Mathias LANG <geod24 gmail.com> writes:
On Monday, 26 February 2024 at 19:50:31 UTC, user1234 wrote:

 That's not a serious suggestion but I'm still curious to see 
 how that would scale.


Terribly.
We have a full implementation of the Itanium ABI scheme and the 
MSVC++ one in the frontend. If there is a case that requires a 
different mangling than the default, that should be handled by 
*providing a triple to the compiler*, not replicating the whole 
mangling logic in user code. I've worked on some of that code and 
getting it right is very time-consuming.
Mar 01
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Monday, 26 February 2024 at 13:36:42 UTC, thumbgun wrote:

 I'm currently trying to call some C++ functions that were 
 compiled by g++ (mingw). However g++ uses the Itanium ABI name 
 mangling rules. dmd on Windows tries to link functions based on 
 the MSVC name mangling rules.


...

 Is there any way I can make dmd link to symbols mangled 
 according to the Itanium ABI's rules on Windows?


Rather than the mangling names of functions, I'd be more worried 
about calling conventions (something that mismatched name 
mangling is supposed to help you with! ;-)).

You can try with LDC and `-mtriple=x86_64-windows-gnu`.

-Johan
Feb 26
parent thumbgun <thumbgun07 gmail.com> writes:
On Monday, 26 February 2024 at 20:05:40 UTC, Johan wrote:

 On Monday, 26 February 2024 at 13:36:42 UTC, thumbgun wrote:

 I'm currently trying to call some C++ functions that were 
 compiled by g++ (mingw). However g++ uses the Itanium ABI name 
 mangling rules. dmd on Windows tries to link functions based 
 on the MSVC name mangling rules.


 ...

 Is there any way I can make dmd link to symbols mangled 
 according to the Itanium ABI's rules on Windows?


 Rather than the mangling names of functions, I'd be more 
 worried about calling conventions (something that mismatched 
 name mangling is supposed to help you with! ;-)).

 You can try with LDC and `-mtriple=x86_64-windows-gnu`.

 -Johan


The problem is that I have a library that was compiled by g++ 
(mingw). When I try to wrap it with extern (C++) DMD on Windows 
will try to link with symbols that are supposed to be mangled 
with MSVC.

Your point about calling conventions is right as the Itanium ABI 
specifies a different calling convention than MSVC. So I need a 
way for DMD to link against symbols mangled according to the 
Itanium ABI and then call the functions according to the Itanium 
ABI's calling convention. DMD currently supports this for other 
platforms but not for Windows.

LDC also uses DMD as a frontend so the problem could be the same 
but I'll try your suggestion anyway.
Feb 26
prev sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Monday, 26 February 2024 at 13:36:42 UTC, thumbgun wrote:

 I'm currently trying to call some C++ functions that were 
 compiled by g++ (mingw). However g++ uses the Itanium ABI name 
 mangling rules. dmd on Windows tries to link functions based on 
 the MSVC name mangling rules.
 [...]
 Is there any way I can make dmd link to symbols mangled 
 according to the Itanium ABI's rules on Windows?


Here's a simple way to do this with *no change to source code* in 
either the dynamic library or the DMD project that is supposed to 
dynamically link to it. Of course this doesn't not resolve 
potential C++ calling convention issues (that don't exist for C), 
but now anyone is in a position to investigate when they exist.

I made a tiny proof of concept and it works. For concreteness, 
suppose the dynamic library is `libx.dll`, built with the 
(mingw64) `gcc` installed with the latest 
[MSYS2](https://www.msys2.org/) as that led to all the utilities 
I needed and a bash command line.

Suppose also the DMD project executable when build will be 
`main.exe` compiled from `main.d` and `other.d` and a D interface 
file `header.di` containing the necessary declarations for using 
`libx.dll`. I'll state the obvious below to make the explanation 
complete and for snag free experimentation.

Suppose for a moment there's no mangling problem because 
`libx.dll` is compiled from C source, not C++. I'll describe the 
exact context and then show how to fix it up for C++ with the 
mangling problem solved.

To dynamically link to `libx.dll` from a DMD executable 
`main.exe` DMD needs to link an *implib* ([import 
library](https://en.wikipedia.org/wiki/Dynamic-link_library#Import_libraries))
during the build of `main.exe` and such can be made from a *def file* (module
definition file) — which is a text file — using a library manager that
knows about the MSVC world that DMD inhabits. Let `libx.def` be a def file for
`libx.dll`, and let `libx.lib` be an implib for `libx.dll`.

The def file would usually be created by `gcc` given 
`-Wl,--output-def=libx.def` when `libx.dll` is linked. And an 
implib can be created from it using `dlltool` which is 
distributed with that mingw64 `gcc`.[¹](#one)
```
$ dlltool -D libx.dll -d libx.def -l libx.lib -m i386:x86-64
```
Alternatively, the MS librarian `lib` can be used.[²](#two) 
[³](#three)
```
$ lib -nologo -machine:x64 -def:libx.def -out:libx.lib
```
Now when main.exe is built, it just needs to link to that import 
library and we're in business.
```
$ dmd main.d other.d header.di libx.lib
$ ./main #works
```
Now suppose we move to C++. If we make an import library as 
above, then a build of `main.exe` will not link, because the 
`gcc`-mangled names in the implib `libx.lib` do not match the 
MSVC-mangled names supplied by DMD.

*We can fix this by modifying the def file and producing an 
implib containing the MSVC-mangled names in place of the 
corresponding `gcc`-mangled names!*

An implib contains each name to link to paired with the 
corresponding location of the function in the dynamic library 
that name refers to. Concretely `libx.lib` contains each 
`gcc`-mangled name paired with the location in `libx.dll` of the 
corresponding function. So the problem is solved if the 
`gcc`-mangled names are replaced by the corresponding 
MSVC-mangled names in the implib `libx.lib`.

There are many ways to do this! However, there's a 
[mechanism](https://learn.microsoft.com/en-us/cpp/build/reference/exports?vi
w=msvc-170#remarks) in a def file to do just that.

Here's the def file `libx.def` for my toy `libx.dll` generated by 
`g++ -shared libx.o -o libx.dll -Wl,--output-def=libx.def`.
```
EXPORTS
     _Z11complicatedi  1
```
Here `_Z11complicatedi  1` is the `gcc`-mangle of `int 
complicated(int)`. Unfortunately, `other.d` expects this function 
to be mangled as `?complicated  YAHH Z`, as this is the 
MSVC-mangle of `int __cdecl complicated(int)`[⁴](#four) and comes 
from `extern(C++) int complicated(int);` in `header.di`.

Editing `libx.def` into
```
EXPORTS
     ?complicated  YAHH Z=_Z11complicatedi  1
```
substitutes the MSVC-mangled name on the left for the 
`gcc`-mangled name on the right when generating the implib 
`libx.lib`. Using the MS librarian as before and building 
`main.exe` removes the linking error and the result just works.

*However while using `dlltool` or `llvm-dlltool` as before 
produces implibs that satify the linker, the resulting `main.exe` 
when run did nothing in my toy example, simply returning to the 
prompt with no output as of 2024-02-29.*

A `libx.def` and hence `libx.lib` for any `main.exe` and 
`libx.dll` with many substitution lines placed in the def file 
could be mechanically generated for once and for all. Or 
`libx.def` and `libx.lib` could be rebuilt on the fly as new 
symbols are used while the DMD project is being written.

Using the MS `dumpbin` tool produces text from which MSVC-mangled 
symbols can be extracted, along with their demanglings. So if the 
DMD project is compiled to a lib using the -lib option so that it 
builds when linkage would be broken then a table of 
(unmangled,MSVC-mangled) name pairs for linkage can automatically 
constructed from running `dumpbin` on the resulting `main.lib` 
and tearing up the resulting text. Similarly, the utility `nm` 
can be used to produce a table of (unmangled,`gcc`-mangled) pairs 
from `libx.dll` and that combined with the text of `libx.def` to 
produce the modified `libx.def` with the necessary additional 
qualifiers as in the example above.

A script could do this and then `lib` run to build the import 
library on the fly during a build. Or, if the library's bindings 
are all in a D header file already, say `header.di` then that 
could be used to produce the pairs containing unmangled and the 
MSVC-mangled names once and for all, and the corresponding 
`libx.def` file then used to produce the implib `libx.lib` that 
could be endlessly used with `libx.dll`.

Lots of possibilities here!

There is a library distributed with mingw64 `gcc` to [demangle 
MSVC-mangled 
names](https://mingw-w64.sourceforge.net/libmangle/index.html), 
though I did not use it. So in principle the substitutive def 
file could be made using just nm to dump the MSVC-mangled 
binaries, so no MS tools are needed to make it.

Of course what we really need to know is the extent to which 
cross calling actually works for various C++ constructs. I'd be 
grateful if anyone finds this out that they'd post it here. I'm 
not a C++ fan, so I'm not the person to do this.

___

online that dlltool is unreliable, in which case 
[llvm-dlltool](https://github.com/ldc-developers/llvm-project/releases/download/ldc-v14.0.0/llvm-14
0.0-windows-x64.7z) might be better. They both have the same command line, and
I could distinguish no difference between them in my toy examples.


resident in at the front of your MSYS2 path before executing it 
is handy, so as to avoid polluting that deliberately isolated 
path with MS related executables. This technique can be used for 
other MS tools mentioned above. So e.g. in `~/bin/lib` made 
executable could be the following with `VCBIN` appropriately set 
in `~/.bash_profile` as an MSYS2 path obtained from the windows 
path to `lib.exe`'s directory using the `cygpath` utility that 
comes with MSYS2. Note that it says `lib.exe` in the script, not 
`lib` to avoid accidental recursion.
```

PATH="$VCBIN:$PATH"
lib.exe "$ "
```


favor of unix style options like `-nologo` because MSYS2 tries to 
helpfully modify command lines and regards `/nologo` as a an 
MSYS2 path which will be converted to a Windows path before 
executing the command.


calling convention for (mingw64) `gcc` and DMD's `extern(C++)` 
assumes this, and MSVC-mangling always includes the calling 
convention in the signature being mangled, even though 
`gcc`-mangling does not if it is the default of `__cdecl`.
Feb 29
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 29 February 2024 at 19:54:24 UTC, Carl Sturtivant 
wrote:

 There is a library distributed with mingw64 `gcc` to [demangle 
 MSVC-mangled 
 names](https://mingw-w64.sourceforge.net/libmangle/index.html), 
 though I did not use it. So in principle the substitutive def 
 file could be made using just nm to dump the MSVC-mangled 
 binaries, so no MS tools are needed to make it.


I just found that DMD comes with ddemangle.exe for MSVC-mangled 
names.
Feb 29
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Friday, 1 March 2024 at 00:02:44 UTC, Carl Sturtivant wrote:

 On Thursday, 29 February 2024 at 19:54:24 UTC, Carl Sturtivant 
 wrote:

 There is a library distributed with mingw64 `gcc` to [demangle 
 MSVC-mangled 
 names](https://mingw-w64.sourceforge.net/libmangle/index.html), though I did
not use it. So in principle the substitutive def file could be made using just
nm to dump the MSVC-mangled binaries, so no MS tools are needed to make it.


 I just found that DMD comes with ddemangle.exe for MSVC-mangled 
 names.


Cancel that: ddemangle only demangles according to how D mangles 
names, not MSVC.
Feb 29
next sibling parent thumbgun <thumbgun07 gmail.com> writes:
On Friday, 1 March 2024 at 05:24:55 UTC, Carl Sturtivant wrote:

 On Friday, 1 March 2024 at 00:02:44 UTC, Carl Sturtivant wrote:

 On Thursday, 29 February 2024 at 19:54:24 UTC, Carl Sturtivant 
 wrote:

 There is a library distributed with mingw64 `gcc` to 
 [demangle MSVC-mangled 
 names](https://mingw-w64.sourceforge.net/libmangle/index.html), though I did
not use it. So in principle the substitutive def file could be made using just
nm to dump the MSVC-mangled binaries, so no MS tools are needed to make it.


 I just found that DMD comes with ddemangle.exe for 
 MSVC-mangled names.


 Cancel that: ddemangle only demangles according to how D 
 mangles names, not MSVC.


Thanks a lot for your reply! I'll try it out now.
Mar 01
prev sibling parent reply thumbgun <thumbgun07 gmail.com> writes:
On Thursday, 29 February 2024 at 19:54:24 UTC, Carl Sturtivant 
wrote:

Of course what we really need to know is the extent to which 
cross calling actually works >for various C++ constructs. I'd be 
grateful if anyone finds this out that they'd post it >here. I'm 
not a C++ fan, so I'm not the person to do this.


Your solution works for the difference in name mangling but 
another problem is that on Windows DMD attempts to follow the 
MSVC ABI which is different from the Itanium ABI. So when calling 
a function compiled by g++ (mingw), it might not work.

The library I need to compile is Skia and after I looked a bit 
more into the docs earlier, it turns out that the library is much 
faster when compiled with clang. I looked into clang's 
compatibility with the MSVC ABI and it pretty has what I need 
apart from the name mangling support which is not fully ready yet.

I'll compile Skia with clang and use your solution to fix the 
name mangling issue so I can call the functions. Once again 
thanks for your reply.
Mar 01
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Friday, 1 March 2024 at 11:02:14 UTC, thumbgun wrote:

 I'll compile Skia with clang and use your solution to fix the 
 name mangling issue so I can call the functions. Once again 
 thanks for your reply.


Sounds good! It'd be interesting to hear how that goes.
Mar 02