www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - core.sys.windows.com.ComObject apparently has wrongly laid out Vtable

reply Carl Sturtivant <sturtivant gmail.com> writes:
Any insight appreciated. Details:
```

dmd --version


DMD64 D Compiler v2.107.0
Copyright (C) 1999-2024 by The D Language Foundation, All Rights 
Reserved written by Walter Bright
```
References:
[class ComObject](https://wiki.dlang.org/COM_Programming)
[interface 
IUnknown](https://dlang.org/spec/interface.html#com-interfaces)

I only used class ComObject and implicitly the IUnknown interface 
it inherits from as a test of some COM code I was writing using 
`comheaders.c` containing the following, compiled with ImportC.
```C
#define WINVER 0x0A00
#define _WIN32_WINNT 0x0A00
#define _WIN32_DCOM
#include <wtypes.h>
#include <oleauto.h>
#include <oaidl.h>
```
The file `main.d` compiled with it is as follows.
```D

import std.stdio;
import comheaders;
static import com = core.sys.windows.com;

pragma(lib, "onecore"); //to fix linkage of two irrelevant symbols

void main() {
     auto COMobject = new com.ComObject();
     //auto COMobject = new ComObject();
     IUnknown* ip = cast(IUnknown*)COMobject;
     writeln(COMobject.count);
     writeln("       ip vtable: ", ip.lpVtbl);
     auto vtable = COMobject.__vptr;
     writeln("COMobject vtable: ", vtable);
     writeln("ip &AddRef: ", &ip.lpVtbl.AddRef);
     writeln("ip offset: ", cast(void*)&ip.lpVtbl.AddRef - 
cast(void*)ip.lpVtbl);
     auto ipaddref = cast(void*)ip.lpVtbl.AddRef;
     writeln("       ip AddRef: ", ipaddref);
     auto addref = cast(void*)(&COMobject.AddRef).funcptr;
     writeln("COMobject AddRef: ", addref);
     writeln("COMobject AddRef : ip AddRef offset: ", addref - 
ipaddref);
     COMobject.AddRef();
     writeln(COMobject.count);
     ip.lpVtbl.AddRef(ip);
     writeln(COMobject.count);
}
```
Here I make a `ComObject` from the statically imported 
`core.sys.windows.com` but avoid using anything else from the D 
windows libs. The object contains a reference count, that should 
be incremented with a call of AddRef. The output was as follows.
```
0
        ip vtable: 7FF756091A30
COMobject vtable: 7FF756091A30
ip &AddRef: 7FF756091A38
ip offset: 8
        ip AddRef: 7FF756027970
COMobject AddRef: 7FF756022EC0
COMobject AddRef : ip AddRef offset: -19120
1
1
```
This shows that the call of AddRef with the correct offset does 
nothing, and is a different function pointer to that of AddRef in 
the com.ComObject. So that object will apparently not work 
correctly with outside world code as claimed (see the above 
reference links). This was compiled with
```
dmd main.d comheaders.c vcintrinsics.lib -P/wd5105
```
where vcintrinsics.lib is a library I constructed to fix a 
problem with DMD not knowing of a series of MSVC intrinsics, i.e. 
to satify the linker as per 
[here](https://forum.dlang.org/post/lbvjpwqgcgducgoxwsxe forum.dlang.org), and
-P/wd5105 is to suppress a warning from MSVC when it is used by ImportC for as
a C preprocessor.

I copied the source of the inconveniently named `interface 
IUnknown` in `unknwn.d` and of `class ComObject` from `com.d`, 
both in `C:\D\dmd2\src\druntime\src\core\sys\windows\`
into the bottom of main.d and experimented. I made a local 
ComObject with the commented out line above active. One change 
fixed the problem: using extern(C++) --- no other linking 
attribute worked. Here's the working code. I had to edit 
`IUnknown` which was an interface but is now a struct in 
`comheaders.c`, into `IUnknown*` inside QueryInterface, and 
`E_NOINTERFACE` into `com.E_NOINTERFACE` and define the D 
interface with a name different to the struct `IUnknown` so I 
made it `_IUnknown_` but otherwise the source is unchanged apart 
from not using extern(Windows) and prefixing it all with 
extern(C++). Here is the rest of `main.d`.
```D
import core.atomic;

extern(C++):

interface _IUnknown_ {
     HRESULT QueryInterface(IID* riid, void** pvObject);
     ULONG AddRef();
     ULONG Release();
}


class ComObject : _IUnknown_
{
     HRESULT QueryInterface(const(IID)* riid, void** ppv)
     {
         if (*riid == IID_IUnknown)
         {
             *ppv = cast(void*)cast(IUnknown*)this;
             AddRef();
             return S_OK;
         }
         else
         {   *ppv = null;
             return com.E_NOINTERFACE;
         }
     }

     ULONG AddRef()
     {
         return atomicOp!"+="(*cast(shared)&count, 1);
     }

     ULONG Release()
     {
         LONG lRef = atomicOp!"-="(*cast(shared)&count, 1);
         if (lRef == 0)
         {
             // free object

             // If we delete this object, then the postinvariant 
called upon
             // return from Release() will fail.
             // Just let the GC reap it.
             //delete this;

             return 0;
         }
         return cast(ULONG)lRef;
     }

     LONG count = 0;             // object reference count
}
```
The output is now as follows.
```
0
        ip vtable: 7FF76B9C0360
COMobject vtable: 7FF76B9C0360
ip &AddRef: 7FF76B9C0368
ip offset: 8
        ip AddRef: 7FF76B951580
COMobject AddRef: 7FF76B951580
COMobject AddRef : ip AddRef offset: 0
1
2
```
showing that finding `AddRef` in the `ComObject`s Vtable produces 
the same function pointer as that through the COM Interface, and 
both work.

My working hypothesis: both `ComObject` and `IUnknown` brought in 
by importing `core.sys.windows.com` are broken which is all of 
the support for COM in Phobos. Please confirm or deny.
Mar 18
next sibling parent reply Kagamin <spam here.lot> writes:
You incorrectly declared the interface variable. Try this:
```
     auto COMobject = new com.ComObject();
     //auto COMobject = new ComObject();
     IUnknown ip = COMobject;
     writeln(COMobject.count);
     writeln("       ip vtable: ", ip.lpVtbl);
     auto vtable = COMobject.__vptr;
     writeln("COMobject vtable: ", vtable);
     writeln("ip &AddRef: ", &ip.lpVtbl.AddRef);
     writeln("ip offset: ", cast(void*)&ip.lpVtbl.AddRef - 
cast(void*)ip.lpVtbl);
     auto ipaddref = cast(void*)ip.lpVtbl.AddRef;
     writeln("       ip AddRef: ", ipaddref);
     auto addref = cast(void*)(&COMobject.AddRef).funcptr;
     writeln("COMobject AddRef: ", addref);
     writeln("COMobject AddRef : ip AddRef offset: ", addref - 
ipaddref);
     COMobject.AddRef();
     writeln(COMobject.count);
     ip.lpVtbl.AddRef(ip);
     writeln(COMobject.count);
```
Mar 19
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Tuesday, 19 March 2024 at 09:22:09 UTC, Kagamin wrote:

 You incorrectly declared the interface variable. Try this:
 ```
     auto COMobject = new com.ComObject();
     //auto COMobject = new ComObject();
     IUnknown ip = COMobject;


**Nonsense**. Of course that doesn't work! It seems you didn't 
read or didn't understand my post.

All you've done is written circular code that confirms 
compatibility with itself by remaining entirely within its own 
world of D only, and only using `core.sys.windows.com`. To make 
this "work" you had to `import core.sys.windows.com;` which I do 
not.

You have NOT confirmed the compatibility of 
`core.sys.windows.com.IUnknown` and 
`core.sys.windows.com.ComObject` with the outside world of actual 
COM.

Supposedly things defined using those can be passed through COM 
to outside world and will work correctly there. This is exactly 
what is NOT happening here.

As mentioned in my original post `core.sys.windows.com` is 
*statically* included so `core.sys.windows.com.IUnknown` is NOT 
`IUnknown` in `main.d` and in fact, apart from making a 
`ComObject` none of `core.sys.windows.com` is used at all in the 
original run.

`IUnknown` in my code is the usual COM struct defined in a 
Windows header and imported from `comheaders.c`. It's called an 
interface in COM terminology but it is NOT a D interface, and you 
can't inherit from it.

So if I just change `main.d` to the code you quoted above it 
won't compile of course.
```
main.d(11): Error: cannot implicitly convert expression 
`COMobject` of type `core.sys.windows.com.ComObject` to `IUnknown`
```
The outside world in C would work with an object implementing 
IUnknown (or another COM interface) as a pointer to a struct of 
type IUnknown (or similar), and main.d works with COM in exactly 
that way.

The documentation of the D interface IUnknown at the quoted link 
in my original post
says that
*A COM interface is defined as one that derives from the 
interface core.sys.win­dows.com.IUnknown.*
and
*A COM interface is designed to map directly onto a Windows COM 
object. Any COM object can be represented by a COM interface, and 
any D object with a COM interface can be used by external COM 
clients.*

This is D's *internal* definition, how a COM interface (outside 
world definition independent of programming language) is 
represented inside D. Conflating the inside D meanings of *COM 
interface* and *IUnknown* with the outside world COM definitions 
of those terms is the source of confusion.

In effect the documentation deliberately conflates the two by 
tacitly assuming that the D representation is correct so we need 
not make that distinction.

The D class `core.sys.win­dows.com.ComObject` inherits from 
`core.sys.win­dows.com.IUnknown` and so should follow those 
statements. My code gets one of those and behaves as an *external 
COM client*, but the result does not work.

If I fix the source of `core.sys.win­dows.com.ComObject` and 
`core.sys.win­dows.com.IUnknown` to have `extern(C++)` linkage 
then the self-same code works, showing that I wrote it correctly.

In order to work with COM the way that C does, when given a 
ComObject from the world of D, I need to treat it as a pointer to 
an IUnknown struct. The ComObject variable COMobject  is already 
a reference so I need to turn it into a pointer of type IUnknown* 
that points to the actual ComObject that the COMobject variable 
refers to, all without knowing the D type of the object.

This is just simulating what would happen if such an object was 
passed by D though COM to e.g. an arbitrary C program. COM 
clients do not know the native type of the objects passed them, 
only the COM interface (NOT D interface) they are supposed to 
implement.

All the recipient knows is that the pointer is to a struct that 
contains a properly laid out Vtable with `QueryInterface`, 
`AddRef` and `Release` as the first three entries in that order 
in accordance with the standard layout of an IUnknown struct. The 
following code correctly achieves that outcome.
```
IUnknown* ip = cast(IUnknown*)COMobject
```
Then *ip is used as the (COM not D) interface to the externally 
provided COM object and it doesn't work unless the D library code 
is repaired.

Then it DOES work. So IUnknown and ComObject in druntime are 
broken. They do not lead to Vtables laid out according to the COM 
structs defined by Windows.
Mar 19
prev sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
I have gone ahead and ported the C style interface over to D and 
rewritten your test code.

Here the output it prints:

```
initial count 0
                ip vtable: 7FF76EFFA240
                ip AddRef: 7FF76EF9D210
    iunknownObject vtable: 7FF76EFFA240
    iunknownObject AddRef: 7FF76EF9D210
After D com object count: 1
After C com object count: 2
```

The code:

```d
import std.stdio;
import drtcom = core.sys.windows.com;
import windows = core.sys.windows.windows;

void main() {
     auto comObject = new drtcom.ComObject;
     auto iunknownObject = cast(drtcom.IUnknown)comObject;

     writeln("initial count ", comObject.count);

      {
          C_IUnknown* ip = cast(C_IUnknown*)iunknownObject;
          writeln("               ip vtable: ", ip.lpVtbl);

          auto ipaddref = cast(void*)ip.lpVtbl.AddRef;
          writeln("               ip AddRef: ", ipaddref);
      }

      {
          auto vtable = iunknownObject.__vptr;
          writeln("   iunknownObject vtable: ", vtable);

          auto addref = cast(void*)(&iunknownObject.AddRef).funcptr;
          writeln("   iunknownObject AddRef: ", addref);
      }

      {
          comObject.AddRef();
          writeln("After D com object count: ", comObject.count);

          C_IUnknown* ip = cast(C_IUnknown*)iunknownObject;
          ip.lpVtbl.AddRef(ip);
          writeln("After C com object count: ", comObject.count);
      }
}

struct C_IUnknown {
     C_IUnknownVtbl* lpVtbl;
}

struct C_IUnknownVtbl {
     extern(Windows) {
         windows.HRESULT function(C_IUnknown* This, windows.IID* riid, 
void** ppvObject) QueryInterface;
         windows.ULONG function(C_IUnknown* This) AddRef;
         windows.ULONG function(C_IUnknown* This) Release;
     }
}
```
Mar 19
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
Very interesting, and thank you.

It's the cast I am using! In the notation of your code, by 
casting comObject directly to C_IUnknown* I am getting a 
different pointer than you do by first casting to drtcom.IUnknown 
and then casting to C_IUnknown*. There are various other subtle 
differences going on here too. In my code the Vtables are the 
same despite the cast I use. But you get your Vtable from the 
drtcom.IUnknown variable, not the comObject variable. The results 
are different. Here's a modification of your main function that 
shows what's going on

```D
void main() {
     auto comObject = new drtcom.ComObject;
     auto iunknownObject = cast(drtcom.IUnknown)comObject;

     writeln("initial count ", comObject.count);

      {
          C_IUnknown* ip1 = cast(C_IUnknown*)iunknownObject;
          writeln("               ip1 vtable: ", ip1.lpVtbl);

          auto ipaddref = cast(void*)ip1.lpVtbl.AddRef;
          writeln("               ip1 AddRef: ", ipaddref);
      }

      {
          C_IUnknown* ip2 = cast(C_IUnknown*)comObject;
          writeln("               ip2 vtable: ", ip2.lpVtbl);

          auto ipaddref = cast(void*)ip2.lpVtbl.AddRef;
          writeln("               ip2 AddRef: ", ipaddref);
      }


      {
          auto vtable = iunknownObject.__vptr;
          writeln("   iunknownObject vtable: ", vtable);

          auto addref = 
cast(void*)(&iunknownObject.AddRef).funcptr;
          writeln("   iunknownObject AddRef: ", addref);
      }

      {
          auto vtable = comObject.__vptr;
          writeln("        comObject vtable: ", vtable);

          auto addref = cast(void*)(&comObject.AddRef).funcptr;
          writeln("        comObject AddRef: ", addref);
      }

      {
          comObject.AddRef();
          writeln("After D com object count: ", comObject.count);

          C_IUnknown* ip_old = cast(C_IUnknown*)iunknownObject;
          C_IUnknown* ip = cast(C_IUnknown*)comObject;
          writefln("ip     = %s ", ip);
          writefln("ip_old = %s", ip_old);
          ip.lpVtbl.AddRef(ip);
          writeln("After C com object count: ", comObject.count);
      }
}
```
The output is as follows.
```
initial count 0
                ip1 vtable: 7FF7EA4318A8
                ip1 AddRef: 7FF7EA3C2C90
                ip2 vtable: 7FF7EA4318E0
                ip2 AddRef: 7FF7EA3C7800
    iunknownObject vtable: 7FF7EA4318A8
    iunknownObject AddRef: 7FF7EA3C2C90
         comObject vtable: 7FF7EA4318E0
         comObject AddRef: 7FF7EA3C2D50
After D com object count: 1
ip     = 20227481000
ip_old = 20227481010
After C com object count: 1
```
I am confident that casting a D interface variable to a pointer 
works at the binary level in the expected way, especially for 
drtcom.IUnknown because the only reasonably simple assumption the 
compiler can make is that it is a valid COM object, not something 
that came from D.

The cast of a ComObject to a pointer produces something 16 bytes 
earlier than the actual start of the COM object according to the 
working cast, suggesting that the compiler is using some trickery 
in it's internal representation of COM objects made from D 
classes that inherit from the D interface IUnknown.

Intuitively, D may make a "normal" object with the "COM" object 
inside it offset by 16 bytes: two words at 64 bits. We might 
guess this is the two usual words in a class object, a Vtable 
pointer and a monitor. We might guess that a "COM" Vtable pointer 
follows that inside the "COM" part and it points to an offset 
inside the "normal" vtable so as to start with the necessary 
QueryInterface, AddRef, Release.

However, when casting such an object to a pointer, a pointer to 
the normal object is produced, NOT a pointer to the internal 
"COM" object. This is possible because D knows the type of the 
object because it is manufactured by D. This is responsible for 
the bad behavior shown in my initial post.

If the ComObject is created with extern(C++) we might guess that 
it is then behaves exactly as what it purports to be so as to be 
entirely compatible with C++. COM in Windows has both a C++ API 
and a binary compatible C API, so now everything works as 
expected. extern(C++) object pointers/references need to just 
work, so the compiler won't produce a bogus offset, whether or 
not the extern(C++) object is secretly embedded inside a "normal" 
object, which we might guess it is. There's just a different 
convention for pointers and references.
Mar 19
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
Basically what you are seeing is normal C++ class behavior.

The only difference between a C++ class reference and a COM class 
reference, is that the methods in the vtable is stdcall for COM.

It's worth noting that for every class and interface in a given class 
hierarchy, each have different vtables for a given object.
This is why you must cast the D object to IUnknown first before 
inspecting its reference.

Which reflects what the compiler is doing when you inherit from 
IUnknown, set the default calling convention and convert to a C++ 
class/interface.
Mar 19
next sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Tuesday, 19 March 2024 at 18:35:38 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 It's worth noting that for every class and interface in a given 
 class hierarchy, each have different vtables for a given object.


Nevertheless, a given virtual method may occur with the same 
index in the Vtable of a class and in the Vtable of a class that 
inherits from it and overrides that method, so dynamic lookup 
finds a given method in a given position in a Vtable irrespective 
of the exact class in the hierarchy it occurs in: statically 
determined code can then look it up in the Vtable by its fixed 
index at the point in source that a method call occurs. And the 
case of COM this position is determined by the COM interface 
being implemented, where AddRef is always in position 1 after 
QueryInterface for example.


 This is why you must cast the D object to IUnknown first before 
 inspecting its reference.


I do not think this follows, either for a regular D object or a 
COM object. For the first the class hierarchy can determine the 
order of the virtual methods and overriding doesn't change the 
index of a method with a given signature. For a COM object the 
order of the methods in the Vtable is determined by the COM 
interface that the object is implementing. In the case of this 
thread, it's determined by the definition of IUnknown in COM.

Specifically, the ComObject class in D inherits from D's IUnknown 
interface and this is modeling implementing the COM interface 
IUnknown with an object with methods in positions 0,1,2 in the 
Vtable implementing QueryInterface, AddRef and Release.

It should be possible to cast a ComObject which by definition 
implements the COM interface IUnknown into a pointer to its first 
word, which external COM expects to contain a pointer to its 
Vtable, which external COM expects to contain pointers to those 
three methods in positions 0,1,2.

However this casting doesn't produce a pointer to the COM object 
needed in any external COM context. It produces a pointer 16 
bytes short of that object, which makes no sense semantically. 
What is it a pointer to? Something hidden by the implementation 
that does not function as a COM object externally.

Casting to a pointer is producing an implementation dependent 
pointer to an object NOT defined in source that happens to seem 
to contain offset down two words the actual object that we have 
defined in source, and the object pointed to seems to have a 
bogus Vtable for COM purposes that doesn't work. We have no 
reason whatsoever to be given access to this outer object. D does 
not define what it is! And D is not producing a pointer to the 
COM object that we defined.

This is a semantic bug: it defines casting to a pointer to have 
silly undefined semantics.

Perhaps the reference to a COM object defined by a class that 
inherits IUnknown is secretly represented by a pointer to this 
outer object. That's no reason to have casting such a reference 
to a pointer not be to compute a pointer to the COM object inside 
that outer object, which is the only cast that has any meaning.
Mar 19
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 20/03/2024 12:36 PM, Carl Sturtivant wrote:

     This is why you must cast the D object to IUnknown first before
     inspecting its reference.
 
 I do not think this follows, either for a regular D object or a COM 
 object. For the first the class hierarchy can determine the order of the 
 virtual methods and overriding doesn't change the index of a method with 
 a given signature. For a COM object the order of the methods in the 
 Vtable is determined by the COM interface that the object is 
 implementing. In the case of this thread, it's determined by the 
 definition of IUnknown in COM.


I suspect you have misunderstood another aspect of classes in general.

First an example, a class and an interface WILL have a different vtable 
entries even if what the vtable represents is the same thing.

```d
import std.stdio : writeln;

void main() {
     C c = new C;
     I i = cast(I)c;

     printVtable(c.__vptr, 3);
     printVtable(i.__vptr, 3);

     printMember!"query"(i, c);
     printMember!"add"(i, c);
     printMember!"sub"(i, c);
}

void printVtable(immutable(void)* vtable, size_t members) {
     writeln("vtable ", vtable);

     void** ptr = cast(void**)vtable;

     foreach(i; 0 .. members) {
         writeln("- ", ptr[i]);
     }
}

void printMember(string member, T, U)(T first, U second) {
     writeln(
         member, " ",
         "First: ", (&__traits(getMember, first, member)).funcptr,
         " Second: ", (&__traits(getMember, second, member)).funcptr
     );
}

extern(C++) interface I {
     void query();
     void add();
     void sub();
}

extern(C++) class C : I {
     override void query() {}
     override void add() {}
     override void sub() {}
}
```

Will output:

```
vtable 562927D04210
- 562927C9577C
- 562927C9578C
- 562927C9579C
vtable 562927D041E8
- 562927C9566C
- 562927C9567C
- 562927C9568C
query First: 562927C9566C Second: 562927C9577C
add First: 562927C9567C Second: 562927C9578C
sub First: 562927C9568C Second: 562927C9579C
```

The reason for this is something called a thunk.

```asm
.text	segment
	assume	CS:.text
:
_THUNK0:
		sub	RDI,8
		jmp	  _ZN1C5queryEv PLT32
		0f1f
		add	byte ptr [RAX],0
		add	[RAX],AL
_THUNK1:
		sub	RDI,8
		jmp	  _ZN1C3addEv PLT32
		0f1f
		add	byte ptr [RAX],0
		add	[RAX],AL
_THUNK2:
		sub	RDI,8
		jmp	  _ZN1C3subEv PLT32
		add	[RAX],AL
		add	[RAX],AL
.text	ends
.data	segment
```

These are functions that alter in this case the this pointer to align 
with what the actual function expects.

https://dlang.org/spec/abi.html#classes

Without it the interface will not understand how to call the class and 
all the pointers will be a little bit off, in this case the size of a 
pointer aka the extra vtable entry.
Mar 19
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 04:14:44 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 These are functions that alter in this case the this pointer to 
 align with what the actual function expects.

 https://dlang.org/spec/abi.html#classes


A useful reference! Should have factored interfaces and not just 
single inheritance into my speculation on inheritance and 
Vtables. Thanks for the detailed counterargument.

However, in the world of COM the situation can be considered 
simpler by having COM objects implement only one COM interface. 
Perhaps that cannot be reflected in how COM is represented in D 
because of the potentially more complicated possibilities for an 
object while it still represents a regular COM object.


 Without it the interface will not understand how to call the 
 class and all the pointers will be a little bit off, in this 
 case the size of a pointer aka the extra vtable entry.


So the layout is as I concretely suspected in this case, the 16 
bytes being exactly the offset mentioned abstractly here in the 
concrete case of IUnknown and ComObject. I inspected the 
reference to a ComObject from druntime and it is indeed a pointer 
offset upward by those 16 bytes from a reference cast to a 
pointer when it is first cast to IUnknown.

This being the case, how do C++ objects in D escape this 
constraint when cast to a pointer? Remember the top post? 
Redefining IUnknown and ComObject to have C++ linkage eliminated 
the 16 byte offset when casting directly and not via IUnknown, 
and the code worked.

Is there some reason why this arrangement is not used for COM 
objects in D?
Mar 19
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 05:26:19 UTC, Carl Sturtivant 
wrote:

 This being the case, how do C++ objects in D escape this 
 constraint when cast to a pointer? Remember the top post? 
 Redefining IUnknown and ComObject to have C++ linkage 
 eliminated the 16 byte offset when casting directly and not via 
 IUnknown, and the code worked.


Ah, I am jumping to conclusions above. The code worked. So a call 
to a working AddRef was found in the ComObject Vtable. Exactly 
the way it wasn't for the library code of a ComObject, where the 
IUnknown interface Vtable had to be found in order for a working 
AddRef to be called.

And your reply uses extern(C++) --- should have noticed that.


 Is there some reason why this arrangement is not used for COM 
 objects in D?


What are the Vtable differences that cause an extern(C++) 
ComObject to work when calling AddRef when the druntime ComObject 
with extern(Windows) does not?
Mar 19
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 05:42:29 UTC, Carl Sturtivant 
wrote:

 What are the Vtable differences that cause an extern(C++) 
 ComObject to work when calling AddRef when the druntime 
 ComObject with extern(Windows) does not?


I mean this of course in the context of the code in my original 
post in this thread.
There casting a druntime ComObject to a pointer leads to a 
non-working AddRef, but casting a extern(C++) ComObject to a 
pointer leads to a working AddRef. I could speculate, but what 
are the actual rules operating that lead to this?
Mar 19
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 20/03/2024 6:56 PM, Carl Sturtivant wrote:

 On Wednesday, 20 March 2024 at 05:42:29 UTC, Carl Sturtivant wrote:

 What are the Vtable differences that cause an extern(C++) ComObject to 
 work when calling AddRef when the druntime ComObject with 
 extern(Windows) does not?


 
 I mean this of course in the context of the code in my original post in 
 this thread.
 There casting a druntime ComObject to a pointer leads to a non-working 
 AddRef, but casting a extern(C++) ComObject to a pointer leads to a 
 working AddRef. I could speculate, but what are the actual rules 
 operating that lead to this?


The only difference I'm aware of between a class that inherits from 
IUnknown and extern(C++) should be the calling convention used for the 
functions in the vtable.

But your original code had a number of assumptions in it that weren't 
correct, so you should probably write a replacement first and work off 
of that.
Mar 19
next sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 06:16:01 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 The only difference I'm aware of between a class that inherits 
 from IUnknown and extern(C++) should be the calling convention 
 used for the functions in the vtable.


Here's the interesting situation I mentioned, isolated, but with 
the C/Windows definition of the struct IUnknown placed in C 
source compiled with ImportC.

`comdef.c`
```C
#include <wtypes.h>

struct C_IUnknownVtbl {
         HRESULT (*QueryInterface)(C_IUnknown* This, IID* riid, 
void** ppvObject);
         ULONG (*AddRef)(C_IUnknown* This);
         ULONG (*Release)(C_IUnknown* This);
};

struct C_IUnknown {
     C_IUnknownVtbl* lpVtbl;
};
```
Mar 20
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
Now here's the new ComObject class, identical to that in druntime 
apart from the necessary `com.` prefix due to the static import 
of `core.sys.windows.com`, except that I declared the class 
extern(C++).
```D
import com = core.sys.windows.com;
import windows = core.sys.windows.windows;

import core.atomic;


extern(C++) class ComObject : com.IUnknown
{
     HRESULT QueryInterface(const(com.IID)* riid, void** ppv)
     {
         if (*riid == com.IID_IUnknown)
         {
             *ppv = cast(void*)cast(com.IUnknown)this;
             AddRef();
             return S_OK;
         }
         else
         {   *ppv = null;
             return com.E_NOINTERFACE;
         }
     }

     ULONG AddRef()
     {
         return atomicOp!"+="(*cast(shared)&count, 1);
     }

     ULONG Release()
     {
         LONG lRef = atomicOp!"-="(*cast(shared)&count, 1);
         if (lRef == 0)
         {
             // free object

             // If we delete this object, then the postinvariant 
called upon
             // return from Release() will fail.
             // Just let the GC reap it.
             //delete this;

             return 0;
         }
         return cast(ULONG)lRef;
     }

     LONG count = 0;             // object reference count
}
```
So this ComObject inherits from the druntime interface IUnknown 
and according to the docs therefore has extern(System) as its 
default for methods, which is extern(Windows) here.
Mar 20
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
Here's the test of that extern(C++) ComObject.
```D
import std.stdio;
import com = core.sys.windows.com;
import windows = core.sys.windows.windows;

import comdef;

pragma(lib,"onecore");

void main() {
     //auto COMobject = new com.ComObject();
     auto COMobject = new ComObject();
     IUnknown* ip = cast(IUnknown*)COMobject;
     writeln(COMobject.count);
     writeln("       ip vtable: ", ip.lpVtbl);
     auto vtable = COMobject.__vptr;
     writeln("COMobject vtable: ", vtable);
     writeln("ip &AddRef: ", &ip.lpVtbl.AddRef);
     writeln("ip offset: ", cast(void*)&ip.lpVtbl.AddRef - 
cast(void*)ip.lpVtbl);
     auto ipaddref = cast(void*)ip.lpVtbl.AddRef;
     writeln("       ip AddRef: ", ipaddref);
     auto addref = cast(void*)(&COMobject.AddRef).funcptr;
     writeln("COMobject AddRef: ", addref);
     writeln("COMobject AddRef : ip AddRef offset: ", addref - 
ipaddref);
     COMobject.AddRef();
     writeln(COMobject.count);
     ip.lpVtbl.AddRef(ip);
     writeln(COMobject.count);
}
```
Note that I cast COMobject (which is a new ComObject) directly to 
a IUnknown*. (Here IUnknown is the struct defined in `comdef.c` 
just the way Windows defines it.) This code works! The result:
```
0
        ip vtable: 7FF6AFEB0360
COMobject vtable: 7FF6AFEB0360
ip &AddRef: 7FF6AFEB0368
ip offset: 8
        ip AddRef: 7FF6AFE31580
COMobject AddRef: 7FF6AFE31580
COMobject AddRef : ip AddRef offset: 0
1
2
```
This shows that the Vtable of the extern(C++) ComObject has COM 
interface layout. However, if I remove `extern(C++)` from the 
definition of the new ComObject class, now this does NOT work. 
Result without extern(C++) is:
```
0
        ip vtable: 7FF6C74D02B0
COMobject vtable: 7FF6C74D02B0
ip &AddRef: 7FF6C74D02B8
ip offset: 8
        ip AddRef: 7FF6C74619E0
COMobject AddRef: 7FF6C7451580
COMobject AddRef : ip AddRef offset: -66656
1
1
```
This shows the Vtable of the ComObject does not have COM 
interface layout.
Mar 20
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 14:26:24 UTC, Carl Sturtivant 
wrote:

 Note that I cast COMobject (which is a new ComObject) directly 
 to a IUnknown*. (Here IUnknown is the struct defined in 
 `comdef.c` just the way Windows defines it.) This code works! 
 This shows that the Vtable of the extern(C++) ComObject has COM 
 interface layout. However, if I remove `extern(C++)` from the 
 definition of the new ComObject class, now this does NOT work.


So ComObject in core.sys.windows.com (in druntime) is actually 
NOT a COM object in the external world of COM sense. But if was 
to be declared extern(C++) it would be a COM object in that 
sense, directly castable to an external IUnknown* that works 
externally, or so it seems.

The only possible snag might be calling conventions for its 
methods. However, it inherits from the druntime D interface 
IUnknown which supposedly guarantees the correct convention for 
Windows/COM.

So why isn't ComObject declared extern(C++) in druntime?
Mar 20
next sibling parent reply Dom DiSc <dominikus scherkl.de> writes:
On Wednesday, 20 March 2024 at 14:38:11 UTC, Carl Sturtivant 
wrote:

 So why isn't ComObject declared extern(C++) in druntime?


Maybe because that was not available at the time druntime was 
written?
Write a bug report and suggest to change that.
But be aware: There is likely code out there that relies on this 
bug :-(
Mar 20
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 14:46:43 UTC, Dom DiSc wrote:

 On Wednesday, 20 March 2024 at 14:38:11 UTC, Carl Sturtivant 
 wrote:

 So why isn't ComObject declared extern(C++) in druntime?


 Maybe because that was not available at the time druntime was 
 written?
 Write a bug report and suggest to change that.
 But be aware: There is likely code out there that relies on 
 this bug :-(


Perhaps you already knew what's in my proof-of-concept, but I've 
just now made it experimentally clear just above in this thread 
that this may very reasonably be regarded as a bug.
Mar 23
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/20/2024 7:38 AM, Carl Sturtivant wrote:

 So why isn't ComObject declared extern(C++) in druntime?


Because COM objects need to follow the extern(Windows) calling convention. It's 
a mistake to try to cast it to extern(C++).

I didn't invent this. Microsoft did. COM was invented around 1990 or so. It's 
cast in concrete.
Mar 20
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 03:48:04 UTC, Walter Bright wrote:

 On 3/20/2024 7:38 AM, Carl Sturtivant wrote:

 So why isn't ComObject declared extern(C++) in druntime?


 Because COM objects need to follow the extern(Windows) calling 
 convention. It's a mistake to try to cast it to extern(C++).

 I didn't invent this. Microsoft did. COM was invented around 
 1990 or so. It's cast in concrete.


I was hoping that the special treatment by the D language of 
IUnknown in core.sys.windows.com would apply to any class that 
inherited from it, even an extern(C++) class, so I could have my 
cake and eat it, and

```D
extern(C++) class ComObject : IUnknown { /* ... */ }
```
would produce a class with extern(Windows) calling and an 
extern(C++) COM compatible vtable.

This does compile. Does the extern(C++) remove the special 
treatment of IUnknown and revert the class to the wrong calling 
convention?
Mar 21
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2024 6:22 AM, Carl Sturtivant wrote:

 On Thursday, 21 March 2024 at 03:48:04 UTC, Walter Bright wrote:

 On 3/20/2024 7:38 AM, Carl Sturtivant wrote:

 So why isn't ComObject declared extern(C++) in druntime?


 Because COM objects need to follow the extern(Windows) calling convention. 
 It's a mistake to try to cast it to extern(C++).

 I didn't invent this. Microsoft did. COM was invented around 1990 or so. It's 
 cast in concrete.


 
 I was hoping that the special treatment by the D language of IUnknown in 
 core.sys.windows.com would apply to any class that inherited from it, even an 
 extern(C++) class, so I could have my cake and eat it, and
 
 ```D
 extern(C++) class ComObject : IUnknown { /* ... */ }
 ```
 would produce a class with extern(Windows) calling and an extern(C++) COM 
 compatible vtable.
 
 This does compile. Does the extern(C++) remove the special treatment of
IUnknown 
 and revert the class to the wrong calling convention?
 
 


The C++ and Windows calling convention are different. COM code will not be able 
to successfully call C++ functions. You can derive a C++ class from a COM 
interface, but you can't cast a COM interface to a C++ class.
Mar 21
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 16:06:32 UTC, Walter Bright wrote:

 The C++ and Windows calling convention are different. COM code 
 will not be able to successfully call C++ functions. You can 
 derive a C++ class from a COM interface, but you can't cast a 
 COM interface to a C++ class.


OK, thanks for the clarification.
Mar 21
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2024 6:22 AM, Carl Sturtivant wrote:

 would produce a class with extern(Windows) calling and an extern(C++) COM 
 compatible vtable.


There are two vtables in that example - one for the C++ class, and one for the 
COM interface. They are not interchangeable with a cast, because the layouts
are 
different.
Mar 21
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 16:07:46 UTC, Walter Bright wrote:

 On 3/21/2024 6:22 AM, Carl Sturtivant wrote:

 would produce a class with extern(Windows) calling and an 
 extern(C++) COM compatible vtable.


 There are two vtables in that example - one for the C++ class, 
 and one for the COM interface. They are not interchangeable 
 with a cast, because the layouts are different.


Good to know! Thank you.
Mar 21
prev sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 06:16:01 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 The only difference I'm aware of between a class that inherits 
 from IUnknown and extern(C++) should be the calling convention 
 used for the functions in the vtable.


I just read-read the entire thread and this jumped out at me. My 
latest proof of concept above endorses this position. It would be 
very helpful if you investigated and fully confirmed that the 
difference you stated is indeed the only difference? Then this 
thread will have reached its conclusion.
Mar 24
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 25/03/2024 3:39 AM, Carl Sturtivant wrote:

 On Wednesday, 20 March 2024 at 06:16:01 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 The only difference I'm aware of between a class that inherits from 
 IUnknown and extern(C++) should be the calling convention used for the 
 functions in the vtable.


 
 I just read-read the entire thread and this jumped out at me. My latest 
 proof of concept above endorses this position. It would be very helpful 
 if you investigated and fully confirmed that the difference you stated 
 is indeed the only difference? Then this thread will have reached its 
 conclusion.


Have a closer look at the header file for IUnknown and compare the 
interface to the c-style struct.

IUnknown is a regular C++ interface.

The c-style struct uses ``extern(Windows)`` function pointers in a 
vtable struct.

This doesn't mean that there are no other changes, but as far as this 
thread is concerned I think that confirms what you wanted to know.
Mar 24
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/19/2024 10:56 PM, Carl Sturtivant wrote:

 On Wednesday, 20 March 2024 at 05:42:29 UTC, Carl Sturtivant wrote:

 What are the Vtable differences that cause an extern(C++) ComObject to work 
 when calling AddRef when the druntime ComObject with extern(Windows) does not?


 
 I mean this of course in the context of the code in my original post in this 
 thread.
 There casting a druntime ComObject to a pointer leads to a non-working AddRef, 
 but casting a extern(C++) ComObject to a pointer leads to a working AddRef. I 
 could speculate, but what are the actual rules operating that lead to this?
 


A com object is an interface, not a class. All com objects must derive from 
core.sys.windows.unknwn.IUknown, or from an interface that derives from
IUnknown.

IUnknown uses the extern (Windows) calling convention.

For com objects, slot 0 in the vtbl[] is for QueryInterface(), not the
classinfo.
Mar 20
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 03:38:18 UTC, Walter Bright wrote:

 A com object is an interface, not a class. All com objects must 
 derive from core.sys.windows.unknwn.IUknown, or from an 
 interface that derives from IUnknown.

 IUnknown uses the extern (Windows) calling convention.

 For com objects, slot 0 in the vtbl[] is for QueryInterface(), 
 not the classinfo.


It's nice that the D language itself makes this happen for 
`core.sys.windows.unknwn.IUnknown` specially so as to make 
writing a COM client using a class possible and having it pass on 
its extern(Windows) calling convention is nice too.

As you say, all COM objects must inherit from this --- otherwise 
they don't get the twin benefits of extern(Windows) calling and a 
COM compatible vtbl[] for their IUnknown interface.

Can we get those benefits out here without 
`core.sys.windows.unknwn.IUnknown` by definining a class 
extern(C++) to get a COM compatible vtbl[] and then qualifying 
each of its methods as extern(Windows) to get COM compatible 
calling, and writing our methods in the correct COM order, 
starting with QueryInterface, AddRef, Release, and relying on D 
to place the methods in the vtbl[] in that order?

(Of course, this won't then bless the descendants of such an 
interface or class, the way the descendents of 
`core.sys.windows.unknwn.IUnknown` are blessed by a special 
dispensation in the D language definition, but still. It would be 
very nice if we could so bless an interface or class ourselves 
with `extern(COM)`, fixing the vtbl[], calling convention, and 
method ordering in the vtable rules, for self and any descendant.)

I ask if we can get these benefits ourselves as above because 
there's an external world difficulty using 
`core.sys.windows.unknwn.IUnknown` with the rest of Windows 
programming outside of just using `core.sys.windows.windows` and 
suchlike.

`core.sys.windows.unknwn.IUnknown` forces the definition of 
`core.sys.windows.basetyps.GUID` into any COM code written using 
it because the first parameter of 
`core.sys.windows.unknwn.IUnknown.QueryInterface` is defined to 
be of type `IID*` and `IID` is defined to be `const(GUID)`, all 
in `core.sys.windows.basetyps`.

Now if ImportC is used to bring in Windows COM related headers so 
as to use the wide collection COM machinery there, a 
binary-identical GUID type is defined in `guidef.h` and used 
throughout. Inheriting from `core.sys.windows.unknwn.IUnknown` to 
make COM objects of type that interface means that QueryInterface 
will have the wrong type for its first parameter to work with all 
of Windows COM machinery. Working around this is awkward.
Mar 21
next sibling parent Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 14:30:24 UTC, Carl Sturtivant wrote:

 It's nice that the D language itself makes this happen for 
 `core.sys.windows.unknwn.IUnknown` specially so as to make 
 writing a COM ~~client~~ **server** using a class possible and 
 having it pass       on its extern(Windows) calling convention 
 is nice too.
Mar 21
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/21/2024 7:30 AM, Carl Sturtivant wrote:

 On Thursday, 21 March 2024 at 03:38:18 UTC, Walter Bright wrote:

 A com object is an interface, not a class. All com objects must derive from 
 core.sys.windows.unknwn.IUknown, or from an interface that derives from
IUnknown.

 IUnknown uses the extern (Windows) calling convention.

 For com objects, slot 0 in the vtbl[] is for QueryInterface(), not the
classinfo.


 
 It's nice that the D language itself makes this happen for 
 `core.sys.windows.unknwn.IUnknown` specially so as to make writing a COM
client 
 using a class possible and having it pass on its extern(Windows) calling 
 convention is nice too.
 
 As you say, all COM objects must inherit from this --- otherwise they don't
get 
 the twin benefits of extern(Windows) calling and a COM compatible vtbl[] for 
 their IUnknown interface.
 
 Can we get those benefits out here without `core.sys.windows.unknwn.IUnknown`
by 
 definining a class extern(C++) to get a COM compatible vtbl[] and then 
 qualifying each of its methods as extern(Windows) to get COM compatible
calling, 
 and writing our methods in the correct COM order, starting with
QueryInterface, 
 AddRef, Release, and relying on D to place the methods in the vtbl[] in that
order?


No, because the vtbl[] layout is different.



 (Of course, this won't then bless the descendants of such an interface or
class, 
 the way the descendents of `core.sys.windows.unknwn.IUnknown` are blessed by a 
 special dispensation in the D language definition, but still. It would be very 
 nice if we could so bless an interface or class ourselves with `extern(COM)`, 
 fixing the vtbl[], calling convention, and method ordering in the vtable
rules, 
 for self and any descendant.)
 
 I ask if we can get these benefits ourselves as above because there's an 
 external world difficulty using `core.sys.windows.unknwn.IUnknown` with the
rest 
 of Windows programming outside of just using `core.sys.windows.windows` and 
 suchlike.
 
 `core.sys.windows.unknwn.IUnknown` forces the definition of 
 `core.sys.windows.basetyps.GUID` into any COM code written using it because
the 
 first parameter of `core.sys.windows.unknwn.IUnknown.QueryInterface` is
defined 
 to be of type `IID*` and `IID` is defined to be `const(GUID)`, all in 
 `core.sys.windows.basetyps`.
 
 Now if ImportC is used to bring in Windows COM related headers so as to use
the 
 wide collection COM machinery there, a binary-identical GUID type is defined
in 
 `guidef.h` and used throughout. Inheriting from 
 `core.sys.windows.unknwn.IUnknown` to make COM objects of type that interface 
 means that QueryInterface will have the wrong type for its first parameter to 
 work with all of Windows COM machinery. Working around this is awkward.


You can invent your own COM-like system in D by using interfaces, but they
won't 
work with Windows COM code.
Mar 21
next sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 16:10:25 UTC, Walter Bright wrote:

 On 3/21/2024 7:30 AM, Carl Sturtivant wrote:

 Can we get those benefits out here without 
 `core.sys.windows.unknwn.IUnknown` by definining a class 
 extern(C++) to get a COM compatible vtbl[] and then qualifying 
 each of its methods as extern(Windows) to get COM compatible 
 calling, and writing our methods in the correct COM order, 
 starting with QueryInterface, AddRef, Release, and relying on 
 D to place the methods in the vtbl[] in that order?


 No, because the vtbl[] layout is different.


Thank you for clarifying!
Mar 21
parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 16:10:25 UTC, Walter Bright wrote:

 On 3/21/2024 7:30 AM, Carl Sturtivant wrote:

 Can we get those benefits out here without 
 `core.sys.windows.unknwn.IUnknown` by definining a class 
 extern(C++) to get a COM compatible vtbl[] and then qualifying 
 each of its methods as extern(Windows) to get COM compatible 
 calling, and writing our methods in the correct COM order, 
 starting with QueryInterface, AddRef, Release, and relying on 
 D to place the methods in the vtbl[] in that order?


 No, because the vtbl[] layout is different.


I don't agree: this works in a proof of concept compiled at both 
32 and 64 bits. Details below.
Mar 23
parent Carl Sturtivant <sturtivant gmail.com> writes:
Here is a proof of concept that a genuine COM object can be 
created in D that works in the outside word, yet NOT using 
`core.sys.windows.unknwn.IUnknown`, following a recipe I 
suggested above.

The vtbl[] of the COM object is the vtbl[] of the D class object 
and is NOT the vtbl[] of a D interface inside it, the way it 
would have been had the class been implemented conventionally by 
inheriting from `core.sys.windows.unknwn.IUnknown`.
```D
import core.sys.windows.windows;
import core.sys.windows.com;

import std.stdio;

extern(C++)
class COMclassObject {
     extern(Windows):
   //Does NOT inherit from D interface IUnknown
   //Specifies extern(C++) for the class:
   //    in hope of a COM compatible vtbl[]
   //    with QueryInterface in slot 0
   //Specifies extern(Windows) for the methods:
   //    in hope of COM compatible calling (__stdcall)
   //Hope that methods are added to vtbl[] for COM:
   //    in the order they are written
   //Cast directly to void* :
   //    to get a pointer to its vtbl[] as per ABI docs here:
   // https://dlang.org/spec/abi.html#classes

     HRESULT QueryInterface(REFIID riid, void** ppvObject) {
writeln("Called QueryInterface.");
         if( *riid == IID_IUnknown ) {
             *ppvObject = cast(void*)this; //cast directly to void*
             AddRef();
             return S_OK;
         }
         *ppvObject = null;
         return E_NOINTERFACE;
     }
     import core.atomic;

     ULONG AddRef() {
writeln("Called AddRef.");
         return atomicOp!"+="(*cast(shared)&count, 1);
     }
     ULONG Release() { //e.g. GC version, does not destroy
writeln("Called Release.");
         LONG lRef = atomicOp!"-="(*cast(shared)&count, 1);
         return cast(ULONG)lRef;
     }
     LONG count = 0;
}

void main() {
     //act as COM server
         auto comClassObject = new COMclassObject();
     //cast to void* to get COM interface
         auto pInterface = cast(void*)comClassObject;

     //instead of sending pInterface to an outside client
     //simulate being that COM client here

     //test as COM client conventionally here
     //for convenience, using D's interface IUnknown
     // https://dlang.org/spec/interface.html#com-interfaces
         auto unknown = cast(IUnknown)pInterface;
     //get a COM interface using QueryInterface
         void* pUnk;
     write("Calling QueryInterface: ");
         HRESULT hr = unknown.QueryInterface(&IID_IUnknown, &pUnk);
         assert(SUCCEEDED(hr));
    //use the resulting interface conventionally as test
         auto unk = cast(IUnknown)pUnk;
     write("Calling AddRef: ");
         unk.AddRef();
   //check that the actual pointers involved are identical
     writefln("comClassObject: %x", pointer(comClassObject));
     writefln("    pInterface: %x", pointer(pInterface));
     writefln("       unknown: %x", pointer(unknown));
     writefln("          pUnk: %x", pointer(comClassObject));
     writefln("           unk: %x", pointer(comClassObject));
}

     //examine a class or interface
     //as its actual pointer (no cast)
union vunion(REF) {
     REF refvar;
     void* ptr;
     this(REF r) { refvar = r; }
}
void* pointer(REF)(REF refvar) {
     return vunion!REF(refvar).ptr;
}
```
Output:
```
Calling QueryInterface: Called QueryInterface.
Called AddRef.
Calling AddRef: Called AddRef.
comClassObject: 21de9ef0010
     pInterface: 21de9ef0010
        unknown: 21de9ef0010
           pUnk: 21de9ef0010
            unk: 21de9ef0010
```
Without `extern(C++)` this produces a crash because the vtbl[] 
has QueryInterface in slot 1 according to the D ABI docs linked 
in the above code, with type information in slot 0 where COM 
expects QueryInterface to be.

However, I wondered whether the `extern(Windows):` qualification 
was operating inside an `extern(C++):` class, or were those calls 
accidentally successful with mismatched calling conventions, and 
the stack/registers were in some way silently corrupted. So I 
commented `extern(Windows):` out and recompiled and it still 
worked!

So then I found out about this. [Windows x64 Calling 
Conventions](https://stackoverflow.com/questions/66398953/writing-a-microsoft-fastcall-64-bit-assembly-function

convention  and that apparently includes win32 API calls. So `extern(Windows):`
is unnecessary and makes no difference!

It seems that if you only care about 64-bits, the annotation 
`extern(C++):` fixes up a COM compatible vtbl[] and you're in 
business provided you take care with method order-of-declaration 
so you're compatible with the COM interface you're implementing.

I then compiled and ran at 32-bits where there *is* a well-known 
difference. With `extern(Windows):` for the methods the program 
again worked, with output
```
Calling QueryInterface: Called QueryInterface.
Called AddRef.
Calling AddRef: Called AddRef.
comClassObject: 2640010
     pInterface: 2640010
        unknown: 2640010
           pUnk: 2640010
            unk: 2640010
```
whereas with `extern(Windows):` commented out, the output was
```
Calling QueryInterface: Called QueryInterface.

object.Error (0): Access Violation
----------------
0x00A91045
0x00A910ED
0x00A9FAF7
0x00A9FA57
0x00A9F8C6
0x00A9B2EC
0x00A912A7
0x76A4FCC9 in BaseThreadInitThunk
0x770E7C5E in RtlGetAppContainerNamedObjectPath
0x770E7C2E in RtlGetAppContainerNamedObjectPath
```
suggesting a calling convention mismatch.

Either way, COM can apparently be implemented with classes and 
not interfaces. It seems this behavior of `extern(C++):` classes 
in D comes from the way the [C++ single inheritance interface in 
D](https://dlang.org/spec/abi.html#interfaces) works. We might 
expect that the vtbl[] of a C++ interface in D starts without D's 
type information which is irrelevant to C++, and in fact by the 
above proof-of-concept conforms to the COM standard. Apparently 
the vtbl[] of an `extern(C++):` class in D counts as a C++ 
interface in D.

Is there any text in the documentation which when juxtaposed 
would enable me to overtly infer all of this? I can't find such.

Please confirm or deny that these conclusions are correct in 
general.
Mar 23
prev sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 16:10:25 UTC, Walter Bright wrote:

 You can invent your own COM-like system in D by using 
 interfaces, but they won't work with Windows COM code.


I'm not sure we understand each other. I am not interested in 
anything COM-like that doesn't interoperate with the outside 
world of COM, but perhaps you were making the point that other 
suggestions in this thread simply won't work.

I'm trying to write some useful abstractions for COM so as to 
lower the high administrative friction of writing COM code. I am 
working in the ImportC/Use-Windows-Headers situation because that 
provides an environment rich in existing definitions in which to 
do that, and I want my machinery to work with any new COM header 
files in the future.

I definitely will be using your D interface IUnknown from 
Druntime because the D language has given it special status that 
as you have clarified above in this thread cannot be simulated in 
another way with D classes. Thank you for making it clear in 
detail that COM can only be approached with classes by using 
Druntime's IUnknown.

I'll just have to work around there being two GUID type 
definitions that are essentially identical when I do that, one 
from Windows' `guiddef.h` and one from Druntime's `basetyps.d`, 
the second used in Druntime's IUnknown.QueryInterface, the first 
used everywhere by Windows.
Mar 21
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Thursday, 21 March 2024 at 17:51:17 UTC, Carl Sturtivant wrote:

 I definitely will be using your D interface IUnknown from 
 Druntime because the D language has given it special status 
 that as you have clarified above in this thread cannot be 
 simulated in another way with D classes.


It seems I spoke too soon. I have a proof of concept, not using 
core.sys.windows.unknwn.IUnknown that works compiled at both 32 
and 64 bits for Windows. I will continue with details in this 
thread.
Mar 23
prev sibling parent reply Carl Sturtivant <sturtivant gmail.com> writes:
On Tuesday, 19 March 2024 at 18:35:38 UTC, Richard (Rikki) Andrew 
Cattermole wrote:

 Basically what you are seeing is normal C++ class behavior.


Yes, C++ classes in D got it right as far as casting a D 
reference to a pointer goes, so as to fit in neatly with the 
external C++ world.

That's a reason for COM classes in D to do the same thing and not 
produce a meaningless bogus result. The same mechanism can be 
used as is already implemented for C++ classes.

As you point out, the only difference is that the functions 
pointed to in the Vtable are stdcall for COM.
Mar 19
parent reply Zoadian <no no.no> writes:
i've also run into problems with IUnknown.

https://issues.dlang.org/show_bug.cgi?id=23819

there's some magic happening that's messing up the vtable.
Mar 20
parent Carl Sturtivant <sturtivant gmail.com> writes:
On Wednesday, 20 March 2024 at 11:33:32 UTC, Zoadian wrote:

 i've also run into problems with IUnknown.

 https://issues.dlang.org/show_bug.cgi?id=23819

 there's some magic happening that's messing up the vtable.


Yes, and it would be nice if a short very direct summary of the 
situation with all of these interacting binary features was 
written. When I have found out everything I'm going to do that.
Mar 20